Repository: mistralai/mistral-inference Branch: main Commit: 2557e12d0e18 Files: 25 Total size: 232.0 KB Directory structure: gitextract_yufqghio/ ├── .github/ │ └── ISSUE_TEMPLATE/ │ ├── bug_report.yml │ └── config.yml ├── .gitignore ├── LICENSE ├── README.md ├── deploy/ │ ├── .dockerignore │ ├── Dockerfile │ └── entrypoint.sh ├── pyproject.toml ├── src/ │ └── mistral_inference/ │ ├── __init__.py │ ├── args.py │ ├── cache.py │ ├── generate.py │ ├── lora.py │ ├── main.py │ ├── mamba.py │ ├── model.py │ ├── moe.py │ ├── rope.py │ ├── transformer.py │ ├── transformer_layers.py │ └── vision_encoder.py ├── tests/ │ └── test_generate.py └── tutorials/ ├── classifier.ipynb └── getting_started.ipynb ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/ISSUE_TEMPLATE/bug_report.yml ================================================ name: Bug report related to mistral-inference description: Submit a bug report that's related to mistral-inference title: '[BUG: ' labels: ['bug', 'triage'] body: - type: markdown attributes: value: | Thanks for taking the time to fill out this bug report! - type: textarea id: python-vv attributes: label: Python -VV description: Run `python -VV` from your virtual environment placeholder: Copy-paste the output (no need for backticks, will be formatted into code automatically) render: shell validations: required: true - type: textarea id: pip-freeze attributes: label: Pip Freeze description: Run `pip freeze` from your virtual environment placeholder: Copy-paste the output (no need for backticks, will be formatted into code automatically) render: shell validations: required: true - type: textarea id: reproduction-steps attributes: label: Reproduction Steps description: Provide a clear and concise description of the steps that lead to your issue. placeholder: | 1. First step... 2. Step 2... ... validations: required: true - type: textarea id: expected-behavior attributes: label: Expected Behavior description: Explain briefly what you expected to happen. validations: required: true - type: textarea id: additional-context attributes: label: Additional Context description: Add any context about your problem that you deem relevant. - type: textarea id: suggested-solutions attributes: label: Suggested Solutions description: Please list any solutions you recommend we consider. ================================================ FILE: .github/ISSUE_TEMPLATE/config.yml ================================================ blank_issues_enabled: false contact_links: - name: Documentation url: https://docs.mistral.ai about: Developer documentation for the Mistral AI platform - name: Discord url: https://discord.com/invite/mistralai) about: Chat with the Mistral community ================================================ FILE: .gitignore ================================================ # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py,cover .hypothesis/ .pytest_cache/ cover/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder .pybuilder/ target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv # For a library or package, you might want to ignore these files since the code is # intended to run in multiple environments; otherwise, check them in: # .python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock # poetry # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. # This is especially recommended for binary packages to ensure reproducibility, and is more # commonly ignored for libraries. # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control #poetry.lock # pdm # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. #pdm.lock # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it # in version control. # https://pdm.fming.dev/#use-with-ide .pdm.toml # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # pytype static type analyzer .pytype/ # Cython debug symbols cython_debug/ # PyCharm # JetBrains specific template is maintained in a separate JetBrains.gitignore that can # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. #.idea/ ================================================ FILE: LICENSE ================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: README.md ================================================ # Mistral Inference Open In Colab This repository contains minimal code to run Mistral models. Blog 7B: [https://mistral.ai/news/announcing-mistral-7b/](https://mistral.ai/news/announcing-mistral-7b/)\ Blog 8x7B: [https://mistral.ai/news/mixtral-of-experts/](https://mistral.ai/news/mixtral-of-experts/)\ Blog 8x22B: [https://mistral.ai/news/mixtral-8x22b/](https://mistral.ai/news/mixtral-8x22b/)\ Blog Codestral 22B: [https://mistral.ai/news/codestral](https://mistral.ai/news/codestral/) \ Blog Codestral Mamba 7B: [https://mistral.ai/news/codestral-mamba/](https://mistral.ai/news/codestral-mamba/) \ Blog Mathstral 7B: [https://mistral.ai/news/mathstral/](https://mistral.ai/news/mathstral/) \ Blog Nemo: [https://mistral.ai/news/mistral-nemo/](https://mistral.ai/news/mistral-nemo/) \ Blog Mistral Large 2: [https://mistral.ai/news/mistral-large-2407/](https://mistral.ai/news/mistral-large-2407/) \ Blog Pixtral 12B: [https://mistral.ai/news/pixtral-12b/](https://mistral.ai/news/pixtral-12b/) Blog Mistral Small 3.1: [https://mistral.ai/news/mistral-small-3-1/](https://mistral.ai/news/mistral-small-3-1/) Discord: [https://discord.com/invite/mistralai](https://discord.com/invite/mistralai)\ Documentation: [https://docs.mistral.ai/](https://docs.mistral.ai/)\ Guardrailing: [https://docs.mistral.ai/usage/guardrailing](https://docs.mistral.ai/usage/guardrailing) ## Installation Note: You will use a GPU to install `mistral-inference`, as it currently requires `xformers` to be installed and `xformers` itself needs a GPU for installation. ### PyPI ``` pip install mistral-inference ``` ### Local ``` cd $HOME && git clone https://github.com/mistralai/mistral-inference cd $HOME/mistral-inference && poetry install . ``` ## Model download ### Direct links | Name | Download | md5sum | |-------------|-------|-------| | 7B Instruct | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-Instruct-v0.3.tar | `80b71fcb6416085bcb4efad86dfb4d52` | | 8x7B Instruct | https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar (**Updated model coming soon!**) | `8e2d3930145dc43d3084396f49d38a3f` | | 8x22 Instruct | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-Instruct-v0.3.tar | `471a02a6902706a2f1e44a693813855b` | | 7B Base | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-v0.3.tar | `0663b293810d7571dad25dae2f2a5806` | | 8x7B | **Updated model coming soon!** | - | | 8x22B | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-v0.3.tar | `a2fa75117174f87d1197e3a4eb50371a` | | Codestral 22B | https://models.mistralcdn.com/codestral-22b-v0-1/codestral-22B-v0.1.tar | `1ea95d474a1d374b1d1b20a8e0159de3` | | Mathstral 7B | https://models.mistralcdn.com/mathstral-7b-v0-1/mathstral-7B-v0.1.tar | `5f05443e94489c261462794b1016f10b` | | Codestral-Mamba 7B | https://models.mistralcdn.com/codestral-mamba-7b-v0-1/codestral-mamba-7B-v0.1.tar | `d3993e4024d1395910c55db0d11db163` | | Nemo Base | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-base-2407.tar | `c5d079ac4b55fc1ae35f51f0a3c0eb83` | | Nemo Instruct | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar | `296fbdf911cb88e6f0be74cd04827fe7` | | Mistral Large 2 | https://models.mistralcdn.com/mistral-large-2407/mistral-large-instruct-2407.tar | `fc602155f9e39151fba81fcaab2fa7c4` | Note: - **Important**: - `mixtral-8x22B-Instruct-v0.3.tar` is exactly the same as [Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), only stored in `.safetensors` format - `mixtral-8x22B-v0.3.tar` is the same as [Mixtral-8x22B-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1), but has an extended vocabulary of 32768 tokens. - `codestral-22B-v0.1.tar` has a custom non-commercial license, called [Mistral AI Non-Production (MNPL) License](https://mistral.ai/licenses/MNPL-0.1.md) - `mistral-large-instruct-2407.tar` has a custom non-commercial license, called [Mistral AI Research (MRL) License](https://mistral.ai/licenses/MRL-0.1.md) - All of the listed models above support function calling. For example, Mistral 7B Base/Instruct v3 is a minor update to Mistral 7B Base/Instruct v2, with the addition of function calling capabilities. - The "coming soon" models will include function calling as well. - You can download the previous versions of our models from our [docs](https://docs.mistral.ai/getting-started/open_weight_models/#downloading). ### From Hugging Face Hub | Name | ID | URL | |-------------|-------|-------| | Pixtral Large Instruct | mistralai/Pixtral-Large-Instruct-2411 | https://huggingface.co/mistralai/Pixtral-Large-Instruct-2411 | | Pixtral 12B Base | mistralai/Pixtral-12B-Base-2409 | https://huggingface.co/mistralai/Pixtral-12B-Base-2409 | | Pixtral 12B | mistralai/Pixtral-12B-2409 | https://huggingface.co/mistralai/Pixtral-12B-2409 | | Mistral Small 3.1 24B Base | mistralai/Mistral-Small-3.1-24B-Base-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503 | Mistral Small 3.1 24B Instruct | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503 | ### Usage **News!!!**: Mistral Large 2 is out. Read more about its capabilities [here](https://mistral.ai/news/mistral-large-2407/). Create a local folder to store models ```sh export MISTRAL_MODEL=$HOME/mistral_models mkdir -p $MISTRAL_MODEL ``` Download any of the above links and extract the content, *e.g.*: ```sh export 12B_DIR=$MISTRAL_MODEL/12B_Nemo wget https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar mkdir -p $12B_DIR tar -xf mistral-nemo-instruct-2407.tar -C $12B_DIR ``` or ```sh export M8x7B_DIR=$MISTRAL_MODEL/8x7b_instruct wget https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar mkdir -p $M8x7B_DIR tar -xf Mixtral-8x7B-v0.1-Instruct.tar -C $M8x7B_DIR ``` For Hugging Face models' weights, here is an example to download [Mistral Small 3.1 24B Instruct](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503): ```python from pathlib import Path from huggingface_hub import snapshot_download mistral_models_path = Path.home().joinpath("mistral_models") model_path = mistral_models_path / "mistral-small-3.1-instruct" model_path.mkdir(parents=True, exist_ok=True) repo_id = "mistralai/Mistral-Small-3.1-24B-Instruct-2503" snapshot_download( repo_id=repo_id, allow_patterns=["params.json", "consolidated.safetensors", "tekken.json"], local_dir=model_path, ) ``` ## Usage The following sections give an overview of how to run the model from the Command-line interface (CLI) or directly within Python. ### CLI - **Demo** To test that a model works in your setup, you can run the `mistral-demo` command. *E.g.* the 12B Mistral-Nemo model can be tested on a single GPU as follows: ```sh mistral-demo $12B_DIR ``` Large models, such **8x7B** and **8x22B** have to be run in a multi-GPU setup. For these models, you can use the following command: ```sh torchrun --nproc-per-node 2 --no-python mistral-demo $M8x7B_DIR ``` *Note*: Change `--nproc-per-node` to more GPUs if available. - **Chat** To interactively chat with the models, you can make use of the `mistral-chat` command. ```sh mistral-chat $12B_DIR --instruct --max_tokens 1024 --temperature 0.35 ``` For large models, you can make use of `torchrun`. ```sh torchrun --nproc-per-node 2 --no-python mistral-chat $M8x7B_DIR --instruct ``` *Note*: Change `--nproc-per-node` to more GPUs if necessary (*e.g.* for 8x22B). - **Chat with Codestral** To use [Codestral](https://mistral.ai/news/codestral/) as a coding assistant you can run the following command using `mistral-chat`. Make sure `$M22B_CODESTRAL` is set to a valid path to the downloaded codestral folder, e.g. `$HOME/mistral_models/Codestral-22B-v0.1` ```sh mistral-chat $M22B_CODESTRAL --instruct --max_tokens 256 ``` If you prompt it with *"Write me a function that computes fibonacci in Rust"*, the model should generate something along the following lines: ```sh Sure, here's a simple implementation of a function that computes the Fibonacci sequence in Rust. This function takes an integer `n` as an argument and returns the `n`th Fibonacci number. fn fibonacci(n: u32) -> u32 { match n { 0 => 0, 1 => 1, _ => fibonacci(n - 1) + fibonacci(n - 2), } } fn main() { let n = 10; println!("The {}th Fibonacci number is: {}", n, fibonacci(n)); } This function uses recursion to calculate the Fibonacci number. However, it's not the most efficient solution because it performs a lot of redundant calculations. A more efficient solution would use a loop to iteratively calculate the Fibonacci numbers. ``` You can continue chatting afterwards, *e.g.* with *"Translate it to Python"*. - **Chat with Codestral-Mamba** To use [Codestral-Mamba](https://mistral.ai/news/codestral-mamba/) as a coding assistant you can run the following command using `mistral-chat`. Make sure `$7B_CODESTRAL_MAMBA` is set to a valid path to the downloaded codestral-mamba folder, e.g. `$HOME/mistral_models/mamba-codestral-7B-v0.1`. You then need to additionally install the following packages: ``` pip install packaging mamba-ssm causal-conv1d transformers ``` before you can start chatting: ```sh mistral-chat $7B_CODESTRAL_MAMBA --instruct --max_tokens 256 ``` - **Chat with Mathstral** To use [Mathstral](https://mistral.ai/news/mathstral/) as an assistant you can run the following command using `mistral-chat`. Make sure `$7B_MATHSTRAL` is set to a valid path to the downloaded codestral folder, e.g. `$HOME/mistral_models/mathstral-7B-v0.1` ```sh mistral-chat $7B_MATHSTRAL --instruct --max_tokens 256 ``` If you prompt it with *"Albert likes to surf every week. Each surfing session lasts for 4 hours and costs $20 per hour. How much would Albert spend in 5 weeks?"*, the model should answer with the correct calculation. You can then continue chatting afterwards, *e.g.* with *"How much would he spend in a year?"*. - **Chat with Mistral Small 3.1 24B Instruct** To use [Mistral Small 3.1 24B Instruct](https://mistral.ai/news/mistral-small-3-1/) as an assistant you can run the following command using `mistral-chat`. Make sure `$MISTRAL_SMALL_3_1_INSTRUCT` is set to a valid path to the downloaded mistral small folder, e.g. `$HOME/mistral_models/mistral-small-3.1-instruct` ```sh mistral-chat $MISTRAL_SMALL_3_1_INSTRUCT --instruct --max_tokens 256 ``` If you prompt it with *"The above image presents an image of which park ? Please give the hints to identify the park."* with the following image URL *https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png*, the model should answer with the Yosemite park and give hints to identify it. You can then continue chatting afterwards, *e.g.* with *"What is the name of the lake in the image?"*. The model should respond that it is not a lake but a river. ### Python - *Instruction Following*: ```py from mistral_inference.transformer import Transformer from mistral_inference.generate import generate from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.protocol.instruct.messages import UserMessage from mistral_common.protocol.instruct.request import ChatCompletionRequest tokenizer = MistralTokenizer.from_file("./mistral-nemo-instruct-v0.1/tekken.json") # change to extracted tokenizer file model = Transformer.from_folder("./mistral-nemo-instruct-v0.1") # change to extracted model dir prompt = "How expensive would it be to ask a window cleaner to clean all windows in Paris. Make a reasonable guess in US Dollar." completion_request = ChatCompletionRequest(messages=[UserMessage(content=prompt)]) tokens = tokenizer.encode_chat_completion(completion_request).tokens out_tokens, _ = generate([tokens], model, max_tokens=1024, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id) result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0]) print(result) ``` - *Multimodal Instruction Following*: ```python from pathlib import Path from huggingface_hub import snapshot_download from mistral_common.protocol.instruct.messages import ImageURLChunk, TextChunk from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_inference.generate import generate from mistral_inference.transformer import Transformer model_path = Path.home().joinpath("mistral_models") / "mistral-small-3.1-instruct" # change to extracted model tokenizer = MistralTokenizer.from_file(model_path / "tekken.json") model = Transformer.from_folder(model_path) url = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png" prompt = "The above image presents an image of which park ? Please give the hints to identify the park." user_content = [ImageURLChunk(image_url=url), TextChunk(text=prompt)] tokens, images = tokenizer.instruct_tokenizer.encode_user_content(user_content, False) out_tokens, _ = generate( [tokens], model, images=[images], max_tokens=256, temperature=0.15, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id, ) result = tokenizer.decode(out_tokens[0]) print("Prompt:", prompt) print("Completion:", result) ``` - *Function Calling*: ```py from mistral_common.protocol.instruct.tool_calls import Function, Tool completion_request = ChatCompletionRequest( tools=[ Tool( function=Function( name="get_current_weather", description="Get the current weather", parameters={ "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "format": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location.", }, }, "required": ["location", "format"], }, ) ) ], messages=[ UserMessage(content="What's the weather like today in Paris?"), ], ) tokens = tokenizer.encode_chat_completion(completion_request).tokens out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id) result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0]) print(result) ``` - *Fill-in-the-middle (FIM)*: Make sure to have `mistral-common >= 1.2.0` installed: ``` pip install --upgrade mistral-common ``` You can simulate a code completion in-filling as follows. ```py from mistral_inference.transformer import Transformer from mistral_inference.generate import generate from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.tokens.instruct.request import FIMRequest tokenizer = MistralTokenizer.from_model("codestral-22b") model = Transformer.from_folder("./mistral_22b_codestral") prefix = """def add(""" suffix = """ return sum""" request = FIMRequest(prompt=prefix, suffix=suffix) tokens = tokenizer.encode_fim(request).tokens out_tokens, _ = generate([tokens], model, max_tokens=256, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id) result = tokenizer.decode(out_tokens[0]) middle = result.split(suffix)[0].strip() print(middle) ``` ### Test To run logits equivalence: ``` python -m pytest tests ``` ## Deployment The `deploy` folder contains code to build a [vLLM](https://M7B_DIR.com/vllm-project/vllm) image with the required dependencies to serve the Mistral AI model. In the image, the [transformers](https://github.com/huggingface/transformers/) library is used instead of the reference implementation. To build it: ```bash docker build deploy --build-arg MAX_JOBS=8 ``` Instructions to run the image can be found in the [official documentation](https://docs.mistral.ai/quickstart). ## Model platforms - Use Mistral models on [Mistral AI official API](https://console.mistral.ai/) (La Plateforme) - Use Mistral models via [cloud providers](https://docs.mistral.ai/deployment/cloud/overview/) ## References [1]: [LoRA](https://arxiv.org/abs/2106.09685): Low-Rank Adaptation of Large Language Models, Hu et al. 2021 ## License This library is licensed under the Apache 2.0 License. See the [LICENCE](./LICENCE) file for more information. *You must not use this library or our models in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.* ================================================ FILE: deploy/.dockerignore ================================================ * !entrypoint.sh ================================================ FILE: deploy/Dockerfile ================================================ FROM --platform=amd64 nvcr.io/nvidia/cuda:12.1.0-devel-ubuntu22.04 as base WORKDIR /workspace RUN apt update && \ apt install -y python3-pip python3-packaging \ git ninja-build && \ pip3 install -U pip # Tweak this list to reduce build time # https://developer.nvidia.com/cuda-gpus ENV TORCH_CUDA_ARCH_LIST "7.0;7.2;7.5;8.0;8.6;8.9;9.0" RUN pip3 install "torch==2.1.1" # This build is slow but NVIDIA does not provide binaries. Increase MAX_JOBS as needed. RUN pip3 install "git+https://github.com/stanford-futuredata/megablocks.git" RUN pip3 install "git+https://github.com/vllm-project/vllm.git" RUN pip3 install "xformers==0.0.23" "transformers==4.36.0" "fschat[model_worker]==0.2.34" RUN git clone https://github.com/NVIDIA/apex && \ cd apex && git checkout 2386a912164b0c5cfcd8be7a2b890fbac5607c82 && \ sed -i '/check_cuda_torch_binary_vs_bare_metal(CUDA_HOME)/d' setup.py && \ python3 setup.py install --cpp_ext --cuda_ext COPY entrypoint.sh . RUN chmod +x /workspace/entrypoint.sh ENTRYPOINT ["/workspace/entrypoint.sh"] ================================================ FILE: deploy/entrypoint.sh ================================================ #!/bin/bash if [[ ! -z "${HF_TOKEN}" ]]; then echo "The HF_TOKEN environment variable is set, logging to Hugging Face." python3 -c "import huggingface_hub; huggingface_hub.login('${HF_TOKEN}')" else echo "The HF_TOKEN environment variable is not set or empty, not logging to Hugging Face." fi # Run the provided command exec python3 -u -m vllm.entrypoints.openai.api_server "$@" ================================================ FILE: pyproject.toml ================================================ [tool.poetry] name = "mistral_inference" version = "1.6.0" description = "" authors = ["bam4d "] readme = "README.md" packages = [{ include = "mistral_inference", from = "src" }] [tool.ruff] lint.select = ["E", "F", "W", "Q", "I"] lint.ignore = ["E203"] lint.fixable = ["ALL"] lint.unfixable = [] line-length = 120 exclude = ["docs", "build", "tutorials"] [tool.mypy] disallow_untyped_defs = true show_error_codes = true no_implicit_optional = true warn_return_any = true warn_unused_ignores = true exclude = ["docs", "tools", "build"] [tool.poetry.dependencies] python = "^3.9.10" xformers = ">=0.0.24" simple-parsing = ">=0.1.5" fire = ">=0.6.0" mistral_common = ">=1.5.4" safetensors = ">=0.4.0" pillow = ">=10.3.0" [tool.poetry.group.dev.dependencies] types-protobuf = "4.24.0.20240129" mypy-protobuf = "^3.5.0" pytest = "7.4.4" ruff = "^0.2.2" mypy = "^1.8.0" [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api" [tool.pytest.ini_options] testpaths = ["./tests"] [tool.poetry.scripts] mistral-chat = "mistral_inference.main:mistral_chat" mistral-demo = "mistral_inference.main:mistral_demo" ================================================ FILE: src/mistral_inference/__init__.py ================================================ __version__ = "1.6.0" ================================================ FILE: src/mistral_inference/args.py ================================================ from dataclasses import dataclass from typing import List, Optional from simple_parsing.helpers import Serializable from mistral_inference.lora import LoraArgs from mistral_inference.moe import MoeArgs PATCH_MERGE = "patch_merge" @dataclass class VisionEncoderArgs: hidden_size: int num_channels: int image_size: int patch_size: int intermediate_size: int num_hidden_layers: int num_attention_heads: int rope_theta: float = 1e4 # for rope-2D image_token_id: int = 10 adapter_bias: bool = True spatial_merge_size: int = 1 add_pre_mm_projector_layer_norm: bool = False mm_projector_id: str = "" @dataclass class TransformerArgs(Serializable): dim: int n_layers: int head_dim: int hidden_dim: int n_heads: int n_kv_heads: int norm_eps: float vocab_size: int max_batch_size: int = 0 # For rotary embeddings. If not set, will be inferred rope_theta: Optional[float] = None # If this is set, we will use MoE layers instead of dense layers. moe: Optional[MoeArgs] = None # If this is set, we will load LoRA linear layers instead of linear layers. lora: Optional[LoraArgs] = None sliding_window: Optional[int] | Optional[List[int]] = None _sliding_window: Optional[int] | Optional[List[int]] = None model_type: str = "transformer" vision_encoder: Optional[VisionEncoderArgs] = None def __post_init__(self) -> None: assert self.model_type == "transformer", self.model_type assert self.sliding_window is None or self._sliding_window is None # hack for now so that vLLM is supported correctly self.sliding_window = self.sliding_window if self.sliding_window is not None else self._sliding_window @dataclass class MambaArgs(Serializable): dim: int n_layers: int vocab_size: int n_groups: int rms_norm: bool residual_in_fp32: bool fused_add_norm: bool pad_vocab_size_multiple: int tie_embeddings: bool model_type: str = "mamba" def __post_init__(self) -> None: assert self.model_type == "mamba", self.model_type ================================================ FILE: src/mistral_inference/cache.py ================================================ from dataclasses import dataclass from typing import List, Optional, Tuple import torch from xformers.ops.fmha.attn_bias import ( # type: ignore AttentionBias, BlockDiagonalCausalMask, BlockDiagonalCausalWithOffsetPaddedKeysMask, BlockDiagonalMask, ) def get_cache_sizes(n_layers: int, max_seq_len: int, sliding_window: Optional[int] | Optional[List[int]]) -> List[int]: if sliding_window is None: return n_layers * [max_seq_len] elif isinstance(sliding_window, int): return n_layers * [sliding_window] else: assert isinstance(sliding_window, list), f"Expected list, got {type(sliding_window)}" assert ( n_layers % len(sliding_window) == 0 ), f"Expected n_layers % len(sliding_window) == 0, got {n_layers} % {len(sliding_window)}" num_repeats = n_layers // len(sliding_window) return num_repeats * [w if w is not None else max_seq_len for w in sliding_window] @dataclass class CacheInputMetadata: # # rope absolute positions # positions: torch.Tensor # # where tokens should go in the cache # cache_positions: torch.Tensor # # if prefill, use block diagonal causal mask # # else use causal with padded key mask # prefill: bool # mask: AttentionBias # seqlens: List[int] # rope absolute positions positions: torch.Tensor # which elements in the sequences need to be cached to_cache_mask: torch.Tensor # how many elements are cached per sequence cached_elements: torch.Tensor # where tokens should go in the cache cache_positions: torch.Tensor # if prefill, use block diagonal causal mask # else use causal with padded key mask prefill: bool mask: AttentionBias seqlens: List[int] def interleave_list(l1: List[torch.Tensor], l2: List[torch.Tensor]) -> List[torch.Tensor]: assert len(l1) == len(l2) return [v for pair in zip(l1, l2) for v in pair] def unrotate(cache: torch.Tensor, seqlen: int) -> torch.Tensor: assert cache.ndim == 3 # (W, H, D) position = seqlen % cache.shape[0] if seqlen < cache.shape[0]: return cache[:seqlen] elif position == 0: return cache else: return torch.cat([cache[position:], cache[:position]], dim=0) class CacheView: def __init__( self, cache_k: torch.Tensor, cache_v: torch.Tensor, metadata: CacheInputMetadata, kv_seqlens: torch.Tensor, ): self.cache_k = cache_k self.cache_v = cache_v self.kv_seqlens = kv_seqlens self.metadata = metadata def update(self, xk: torch.Tensor, xv: torch.Tensor) -> None: """ to_cache_mask masks the last [max_seq_len] tokens in each sequence """ n_kv_heads, head_dim = self.cache_k.shape[-2:] flat_cache_k = self.cache_k.view(-1, n_kv_heads, head_dim) flat_cache_v = self.cache_v.view(-1, n_kv_heads, head_dim) flat_cache_k.index_copy_(0, self.metadata.cache_positions, xk[self.metadata.to_cache_mask]) flat_cache_v.index_copy_(0, self.metadata.cache_positions, xv[self.metadata.to_cache_mask]) def interleave_kv(self, xk: torch.Tensor, xv: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: """ This is a naive implementation and not optimized for speed. """ assert xk.ndim == xv.ndim == 3 # (B * T, H, D) assert xk.shape == xv.shape if all([s == 0 for s in self.metadata.seqlens]): # No cache to interleave return xk, xv # Make it a list of [(T, H, D)] xk: Tuple[torch.Tensor] = torch.split(xk, self.metadata.seqlens) # type: ignore xv: Tuple[torch.Tensor] = torch.split(xv, self.metadata.seqlens) # type: ignore assert len(xk) == len(self.kv_seqlens), f"Batch size is {len(self.kv_seqlens)}, got {len(xk)}" # Order elements in cache by position by unrotating cache_k = [unrotate(t, s) for t, s in zip(self.cache_k, self.kv_seqlens)] cache_v = [unrotate(t, s) for t, s in zip(self.cache_v, self.kv_seqlens)] interleaved_k = interleave_list(cache_k, list(xk)) interleaved_v = interleave_list(cache_v, list(xv)) return torch.cat(interleaved_k, dim=0), torch.cat(interleaved_v, dim=0) @property def max_seq_len(self) -> int: return self.cache_k.shape[1] @property def key(self) -> torch.Tensor: return self.cache_k[: len(self.kv_seqlens)] @property def value(self) -> torch.Tensor: return self.cache_v[: len(self.kv_seqlens)] @property def prefill(self) -> bool: return self.metadata.prefill @property def mask(self) -> AttentionBias: return self.metadata.mask class BufferCache: """ This is an example that implements a buffer cache, allowing for variable length sequences. Allocated cache is rectangular which is wasteful (see PagedAttention for better mechanisms) """ def __init__( self, n_layers: int, max_batch_size: int, max_seq_len: int, n_kv_heads: int, head_dim: int, sliding_window: Optional[int] | Optional[List[int]] = None, ): self.max_seq_len = max_seq_len self.n_kv_heads = n_kv_heads self.head_dim = head_dim self.n_layers = n_layers self.cache_sizes: List[int] = get_cache_sizes(n_layers, max_seq_len, sliding_window) assert len(self.cache_sizes) == n_layers, f"Expected {n_layers} cache sizes, got {len(self.cache_sizes)}" self.cache_k = {} self.cache_v = {} for i, cache_size in enumerate(self.cache_sizes): self.cache_k[i] = torch.empty((max_batch_size, cache_size, n_kv_heads, head_dim)) self.cache_v[i] = torch.empty((max_batch_size, cache_size, n_kv_heads, head_dim)) # holds the valid length for each batch element in the cache self.kv_seqlens: Optional[torch.Tensor] = None def get_view(self, layer_id: int, metadata: CacheInputMetadata) -> CacheView: assert self.kv_seqlens is not None return CacheView(self.cache_k[layer_id], self.cache_v[layer_id], metadata, self.kv_seqlens) def reset(self) -> None: self.kv_seqlens = None def init_kvseqlens(self, batch_size: int) -> None: self.kv_seqlens = torch.zeros((batch_size,), device=self.device, dtype=torch.long) @property def device(self) -> torch.device: return self.cache_k[0].device def to(self, device: torch.device, dtype: torch.dtype) -> "BufferCache": for i in range(self.n_layers): self.cache_k[i] = self.cache_k[i].to(device=device, dtype=dtype) self.cache_v[i] = self.cache_v[i].to(device=device, dtype=dtype) return self def update_seqlens(self, seqlens: List[int]) -> None: assert self.kv_seqlens is not None self.kv_seqlens += torch.tensor(seqlens, device=self.device, dtype=torch.long) def get_input_metadata(self, seqlens: List[int]) -> List[CacheInputMetadata]: """ input = seqlens [5,7,2] // seqpos [0, 1, 3] // sliding_window 3 --> only cache last 3 tokens in each sequence - to_cache_mask = [0 0 1 1 1 | 0 0 0 0 1 1 1 | 1 1] - cached_elements = [3 | 3 | 2] --> absolute positions are used for rope - positions = [0 1 2 3 4 | 1 2 3 4 5 6 7 | 3 4] --> cache positions are positions cache_masked, modulo sliding_window + batch_idx * sliding_window - cache_positions = [2 0 1 | 5 3 4 | 6 7] """ metadata: List[CacheInputMetadata] = [] if self.kv_seqlens is None: self.init_kvseqlens(len(seqlens)) assert self.kv_seqlens is not None assert len(seqlens) == len( self.kv_seqlens ), f"Batch size is {len(self.kv_seqlens)}, got {len(seqlens)}, did you forget to reset cache?" seqpos = self.kv_seqlens.tolist() assert len(seqlens) > 0, seqlens for cache_size in self.cache_sizes: metadata.append(self._get_input_metadata_layer(cache_size, seqlens, seqpos)) return metadata def _get_input_metadata_layer(self, cache_size: int, seqlens: List[int], seqpos: List[int]) -> CacheInputMetadata: masks = [[x >= seqlen - cache_size for x in range(seqlen)] for seqlen in seqlens] to_cache_mask = torch.tensor(sum(masks, []), device=self.device, dtype=torch.bool) cached_elements = torch.tensor([sum(mask) for mask in masks], device=self.device, dtype=torch.long) positions = torch.cat([torch.arange(pos, pos + seqlen) for pos, seqlen in zip(seqpos, seqlens)]).to( device=self.device, dtype=torch.long ) batch_idx = torch.tensor( sum([[i] * seqlen for i, seqlen in enumerate(seqlens)], []), device=self.device, dtype=torch.long ) cache_positions = positions % cache_size + batch_idx * cache_size first_prefill = seqpos[0] == 0 subsequent_prefill = any(seqlen > 1 for seqlen in seqlens) if first_prefill: assert all([pos == 0 for pos in seqpos]), seqpos mask = BlockDiagonalCausalMask.from_seqlens(seqlens).make_local_attention(cache_size) elif subsequent_prefill: assert self.kv_seqlens is not None mask = BlockDiagonalMask.from_seqlens( q_seqlen=seqlens, kv_seqlen=[ s + cached_s.clamp(max=cache_size).item() for (s, cached_s) in zip(seqlens, self.kv_seqlens) ], ).make_local_attention_from_bottomright(cache_size) else: mask = BlockDiagonalCausalWithOffsetPaddedKeysMask.from_seqlens( q_seqlen=seqlens, kv_padding=cache_size, kv_seqlen=(self.kv_seqlens + cached_elements).clamp(max=cache_size).tolist(), ) return CacheInputMetadata( positions=positions, to_cache_mask=to_cache_mask, cached_elements=cached_elements, cache_positions=cache_positions[to_cache_mask], prefill=first_prefill or subsequent_prefill, mask=mask, seqlens=seqlens, ) ================================================ FILE: src/mistral_inference/generate.py ================================================ from typing import List, Optional, Tuple import numpy as np import torch from mistral_inference.cache import BufferCache from mistral_inference.mamba import Mamba from mistral_inference.transformer import Transformer @torch.inference_mode() def generate_mamba( encoded_prompts: List[List[int]], model: Mamba, *, max_tokens: int, temperature: float, chunk_size: Optional[int] = None, eos_id: Optional[int] = None, ) -> Tuple[List[List[int]], List[List[float]]]: input_ids = torch.tensor(encoded_prompts, device=model.device) output = model.model.generate( input_ids=input_ids, max_length=input_ids.shape[-1] + max_tokens, cg=True, return_dict_in_generate=True, output_scores=True, enable_timing=False, eos_token_id=eos_id, temperature=temperature, top_p=0.8, ) generated_tokens = output.sequences[:, input_ids.shape[-1] :].tolist() _logprobs: List[List[float]] = [[] for _ in range(len(generated_tokens))] for seq_idx, batch_score in enumerate(output.scores): for batch_idx, score in enumerate(batch_score.tolist()): _logprobs[batch_idx].append(score[generated_tokens[batch_idx][seq_idx]]) return generated_tokens, _logprobs @torch.inference_mode() def generate( encoded_prompts: List[List[int]], model: Transformer, images: List[List[np.ndarray]] = [], *, max_tokens: int, temperature: float, chunk_size: Optional[int] = None, eos_id: Optional[int] = None, ) -> Tuple[List[List[int]], List[List[float]]]: images_torch: List[List[torch.Tensor]] = [] if images: assert chunk_size is None images_torch = [ [torch.tensor(im, device=model.device, dtype=model.dtype) for im in images_for_sample] for images_for_sample in images ] model = model.eval() B, V = len(encoded_prompts), model.args.vocab_size seqlens = [len(x) for x in encoded_prompts] # Cache cache_window = max(seqlens) + max_tokens cache = BufferCache( model.n_local_layers, model.args.max_batch_size, cache_window, model.args.n_kv_heads, model.args.head_dim, model.args.sliding_window, ) cache.to(device=model.device, dtype=model.dtype) cache.reset() # Bookkeeping logprobs: List[List[float]] = [[] for _ in range(B)] last_token_prelogits = None # One chunk if size not specified max_prompt_len = max(seqlens) if chunk_size is None: chunk_size = max_prompt_len flattened_images: List[torch.Tensor] = sum(images_torch, []) # Encode prompt by chunks for s in range(0, max_prompt_len, chunk_size): prompt_chunks = [p[s : s + chunk_size] for p in encoded_prompts] assert all(len(p) > 0 for p in prompt_chunks) prelogits = model.forward( torch.tensor(sum(prompt_chunks, []), device=model.device, dtype=torch.long), images=flattened_images, seqlens=[len(p) for p in prompt_chunks], cache=cache, ) logits = torch.log_softmax(prelogits, dim=-1) if last_token_prelogits is not None: # Pass > 1 last_token_logits = torch.log_softmax(last_token_prelogits, dim=-1) for i_seq in range(B): logprobs[i_seq].append(last_token_logits[i_seq, prompt_chunks[i_seq][0]].item()) offset = 0 for i_seq, sequence in enumerate(prompt_chunks): logprobs[i_seq].extend([logits[offset + i, sequence[i + 1]].item() for i in range(len(sequence) - 1)]) offset += len(sequence) last_token_prelogits = prelogits.index_select( 0, torch.tensor([len(p) for p in prompt_chunks], device=prelogits.device).cumsum(dim=0) - 1, ) assert last_token_prelogits.shape == (B, V) # decode generated_tensors = [] is_finished = torch.tensor([False for _ in range(B)]) assert last_token_prelogits is not None for _ in range(max_tokens): next_token = sample(last_token_prelogits, temperature=temperature, top_p=0.8) if eos_id is not None: is_finished = is_finished | (next_token == eos_id).cpu() if is_finished.all(): break last_token_logits = torch.log_softmax(last_token_prelogits, dim=-1) for i in range(B): logprobs[i].append(last_token_logits[i, next_token[i]].item()) generated_tensors.append(next_token[:, None]) last_token_prelogits = model.forward(next_token, seqlens=[1] * B, cache=cache) assert last_token_prelogits.shape == (B, V) generated_tokens: List[List[int]] if generated_tensors: generated_tokens = torch.cat(generated_tensors, 1).tolist() else: generated_tokens = [] return generated_tokens, logprobs def sample(logits: torch.Tensor, temperature: float, top_p: float) -> torch.Tensor: if temperature > 0: probs = torch.softmax(logits / temperature, dim=-1) next_token = sample_top_p(probs, top_p) else: next_token = torch.argmax(logits, dim=-1).unsqueeze(0) return next_token.reshape(-1) def sample_top_p(probs: torch.Tensor, p: float) -> torch.Tensor: assert 0 <= p <= 1 probs_sort, probs_idx = torch.sort(probs, dim=-1, descending=True) probs_sum = torch.cumsum(probs_sort, dim=-1) mask = probs_sum - probs_sort > p probs_sort[mask] = 0.0 probs_sort.div_(probs_sort.sum(dim=-1, keepdim=True)) next_token = torch.multinomial(probs_sort, num_samples=1) return torch.gather(probs_idx, -1, next_token) ================================================ FILE: src/mistral_inference/lora.py ================================================ import logging from dataclasses import dataclass from pathlib import Path from typing import Any, Dict, NamedTuple, Union import safetensors.torch import torch import torch.nn as nn from simple_parsing.helpers import Serializable @dataclass class LoraArgs(Serializable): rank: int scaling: float def __post_init__(self) -> None: assert self.rank > 0 assert self.scaling > 0.0 class LoRALinear(nn.Module): """ Implementation of: - LoRA: https://arxiv.org/abs/2106.09685 Notes: - Freezing is handled at network level, not layer level. - Scaling factor controls relative importance of LoRA skip connection versus original frozen weight. General guidance is to keep it to 2.0 and sweep over learning rate when changing the rank. """ def __init__( self, in_features: int, out_features: int, rank: int, scaling: float, bias: bool = False, ): super().__init__() self.in_features = in_features self.out_features = out_features assert not bias self.bias = bias self.rank = rank self.scaling = scaling self.lora_A = nn.Linear( self.in_features, self.rank, bias=self.bias, ) self.lora_B = nn.Linear( self.rank, self.out_features, bias=self.bias, ) self.linear = nn.Linear(self.in_features, self.out_features, bias=self.bias) # make sure no LoRA weights are marked as "missing" in load_state_dict def ignore_missing_keys(m: nn.Module, incompatible_keys: NamedTuple) -> None: incompatible_keys.missing_keys[:] = [] # type: ignore self.register_load_state_dict_post_hook(ignore_missing_keys) def forward(self, x: torch.Tensor) -> torch.Tensor: lora = self.lora_B(self.lora_A(x)) result: torch.Tensor = self.linear(x) + lora * self.scaling return result def _load_from_state_dict(self, state_dict: Dict[str, Any], prefix: str, *args, **kwargs) -> None: # type: ignore[no-untyped-def] key_name = prefix + "weight" # full checkpoint if key_name in state_dict: w_ref = state_dict[key_name] # load frozen weights state_dict = { "linear.weight": w_ref, "lora_A.weight": torch.zeros_like(self.lora_A.weight, device=w_ref.device, dtype=w_ref.dtype), "lora_B.weight": torch.zeros_like(self.lora_B.weight, device=w_ref.device, dtype=w_ref.dtype), } self.load_state_dict(state_dict, assign=True, strict=True) class LoRALoaderMixin: def load_lora(self, lora_path: Union[Path, str], scaling: float = 2.0) -> None: """Loads LoRA checkpoint""" lora_path = Path(lora_path) assert lora_path.is_file(), f"{lora_path} does not exist or is not a file" state_dict = safetensors.torch.load_file(lora_path) self._load_lora_state_dict(state_dict, scaling=scaling) def _load_lora_state_dict(self, lora_state_dict: Dict[str, torch.Tensor], scaling: float = 2.0) -> None: """Loads LoRA state_dict""" lora_dtypes = set([p.dtype for p in lora_state_dict.values()]) assert ( len(lora_dtypes) == 1 ), f"LoRA weights have multiple different dtypes {lora_dtypes}. All weights need to have the same dtype" lora_dtype = lora_dtypes.pop() assert lora_dtype == self.dtype, f"LoRA weights dtype differs from model's dtype {lora_dtype} != {self.dtype}" # type: ignore[attr-defined] assert all("lora" in key for key in lora_state_dict.keys()) # move tensors to device lora_state_dict = {k: v.to(self.device) for k, v in lora_state_dict.items()} # type: ignore[attr-defined] state_dict = self.state_dict() # type: ignore[attr-defined] if self.args.lora is None: # type: ignore[attr-defined] logging.info("Loading and merging LoRA weights...") # replace every nn.Linear with a LoRALinear with 'meta' device except the output layer named_modules = dict(self.named_modules()) # type: ignore[attr-defined] for name, module in named_modules.items(): if isinstance(module, nn.Linear) and name != "output": layer_id = name.split(".")[1] if layer_id not in self.layers: # type: ignore[attr-defined] logging.debug( "Skipping parameter %s at pipeline rank %d", name, self.pipeline_rank, # type: ignore[attr-defined] ) elif (name + ".lora_B.weight") in lora_state_dict: weight = ( module.weight + (lora_state_dict[name + ".lora_B.weight"] @ lora_state_dict[name + ".lora_A.weight"]) * scaling ) state_dict[name + ".weight"] = weight else: logging.info("Loading LoRA weights...") for k, v in lora_state_dict.items(): state_dict.update(lora_state_dict) layer_id = k.split(".")[1] if layer_id in self.layers: # type: ignore[attr-defined] state_dict[k] = v else: logging.debug( "Skipping parameter %s at pipeline rank %d", k, self.pipeline_rank, # type: ignore[attr-defined] ) self.load_state_dict(state_dict, strict=True) # type: ignore[attr-defined] ================================================ FILE: src/mistral_inference/main.py ================================================ import json import logging import os import warnings from pathlib import Path from typing import List, Optional, Tuple, Type, Union import fire # type: ignore import torch import torch.distributed as dist from mistral_common.protocol.instruct.messages import ( AssistantMessage, ContentChunk, ImageChunk, ImageURLChunk, TextChunk, UserMessage, ) from mistral_common.protocol.instruct.request import ChatCompletionRequest from mistral_common.tokens.tokenizers.base import Tokenizer from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.tokens.tokenizers.sentencepiece import is_sentencepiece from mistral_common.tokens.tokenizers.tekken import ( SpecialTokenPolicy, Tekkenizer, is_tekken, ) from PIL import Image from mistral_inference.args import TransformerArgs from mistral_inference.generate import generate, generate_mamba from mistral_inference.mamba import Mamba from mistral_inference.transformer import Transformer def is_torchrun() -> bool: required_vars = ["MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE"] return all(var in os.environ for var in required_vars) def load_tokenizer(model_path: Path) -> MistralTokenizer: tokenizer = [f for f in os.listdir(model_path) if is_tekken(model_path / f) or is_sentencepiece(model_path / f)] assert len(tokenizer) > 0, ( f"No tokenizer in {model_path}, place a `tokenizer.model.[v1,v2,v3]` or `tekken.json` file in {model_path}." ) assert len(tokenizer) == 1, ( f"Multiple tokenizers {', '.join(tokenizer)} found in `model_path`, make sure to only have one tokenizer" ) mistral_tokenizer = MistralTokenizer.from_file(str(model_path / tokenizer[0])) if isinstance(mistral_tokenizer.instruct_tokenizer.tokenizer, Tekkenizer): mistral_tokenizer.instruct_tokenizer.tokenizer.special_token_policy = SpecialTokenPolicy.KEEP logging.info(f"Loaded tokenizer of type {mistral_tokenizer.instruct_tokenizer.__class__}") return mistral_tokenizer def get_model_cls(model_path: str) -> Union[Type[Mamba], Type[Transformer]]: with open(Path(model_path) / "params.json", "r") as f: args_dict = json.load(f) return {"mamba": Mamba, "transformer": Transformer}[args_dict.get("model_type", "transformer")] # type: ignore[return-value] def pad_and_convert_to_tensor(list_of_lists: List[List[int]], pad_id: int) -> List[List[int]]: # Determine the length of the longest list max_len = max(len(lst) for lst in list_of_lists) # Left pad each list to the maximum length padded_lists = [[pad_id] * (max_len - len(lst)) + lst for lst in list_of_lists] return padded_lists def _get_multimodal_input() -> Tuple[UserMessage, bool]: chunks: List[ContentChunk] = [] response = input("Text prompt: ") if response: chunks.append(TextChunk(text=response)) print("[You can input zero, one or more images now.]") while True: did_something = False response = input("Image path or url [Leave empty and press enter to finish image input]: ") if response: if Path(response).is_file(): chunks.append(ImageChunk(image=Image.open(response))) else: assert response.startswith("http"), f"{response} does not seem to be a valid url." chunks.append(ImageURLChunk(image_url=response)) did_something = True if not did_something: break return UserMessage(content=chunks), not chunks def interactive( model_path: str, max_tokens: int = 35, temperature: float = 0.7, num_pipeline_ranks: int = 1, instruct: bool = False, lora_path: Optional[str] = None, ) -> None: if is_torchrun(): torch.distributed.init_process_group() torch.cuda.set_device(torch.distributed.get_rank()) should_print = torch.distributed.get_rank() == 0 num_pipeline_ranks = torch.distributed.get_world_size() else: should_print = True num_pipeline_ranks = 1 mistral_tokenizer: MistralTokenizer = load_tokenizer(Path(model_path)) tokenizer: Tokenizer = mistral_tokenizer.instruct_tokenizer.tokenizer model_cls = get_model_cls(model_path) model = model_cls.from_folder(Path(model_path), max_batch_size=3, num_pipeline_ranks=num_pipeline_ranks) is_multimodal = isinstance(model.args, TransformerArgs) and model.args.vision_encoder is not None if is_multimodal: assert instruct, "Multimodal models should only be used in instruct mode" # load LoRA if lora_path is not None: model.load_lora(Path(lora_path)) prompt: str = "" messages: List[UserMessage | AssistantMessage] = [] while True: if should_print: if not is_multimodal: user_input = input("Prompt: ") if instruct: if is_multimodal: mm_input, finished = _get_multimodal_input() if finished: break messages += [mm_input] else: messages += [UserMessage(content=user_input)] chat_completion_request = ChatCompletionRequest(messages=messages) tokenized = mistral_tokenizer.encode_chat_completion(chat_completion_request) tokens = tokenized.tokens images = tokenized.images else: prompt += user_input tokens = tokenizer.encode(prompt, bos=True, eos=False) images = [] length_tensor = torch.tensor([len(tokens)], dtype=torch.int) else: length_tensor = torch.tensor([0], dtype=torch.int) images = [] if is_torchrun(): dist.broadcast(length_tensor, src=0) if not should_print: tokens = int(length_tensor.item()) * [0] if isinstance(model, Transformer): generated_tokens, _ = generate( [tokens], model, [images], max_tokens=max_tokens, temperature=temperature, eos_id=tokenizer.eos_id, ) else: # Mamba models don't support images generated_tokens, _ = generate_mamba( [tokens], model, max_tokens=max_tokens, temperature=temperature, eos_id=tokenizer.eos_id, ) answer = tokenizer.decode(generated_tokens[0]) if should_print: print(answer) print("=====================") if instruct: messages += [AssistantMessage(content=answer)] else: prompt += answer def demo( model_path: str, max_tokens: int = 35, temperature: float = 0, lora_path: Optional[str] = None, ) -> None: if is_torchrun(): torch.distributed.init_process_group() torch.cuda.set_device(torch.distributed.get_rank()) should_print = torch.distributed.get_rank() == 0 num_pipeline_ranks = torch.distributed.get_world_size() else: should_print = True num_pipeline_ranks = 1 model_cls = get_model_cls(model_path) model = model_cls.from_folder(Path(model_path), max_batch_size=3, num_pipeline_ranks=num_pipeline_ranks) # load LoRA if lora_path is not None: model.load_lora(Path(lora_path)) mistral_tokenizer: MistralTokenizer = load_tokenizer(Path(model_path)) tokenizer: Tokenizer = mistral_tokenizer.instruct_tokenizer.tokenizer prompts = [ "This is a test", "This is another great test", "This is a third test, mistral AI is very good at testing. ", ] encoded_prompts = [tokenizer.encode(prompt, bos=True, eos=False) for prompt in prompts] if isinstance(model, Transformer): generate_fn = generate else: generate_fn = generate_mamba # type: ignore[assignment] warnings.warn( "Batched generation is not correctly supported at the moment and therefore might lead to worse results " "as compared to non-batched generation. " "See https://github.com/state-spaces/mamba/issues/66#issuecomment-1862349718 for more information." ) encoded_prompts = pad_and_convert_to_tensor(encoded_prompts, mistral_tokenizer.instruct_tokenizer.BOS) # type: ignore[attr-defined] generated_tokens, _logprobs = generate_fn( encoded_prompts, model, # type: ignore[arg-type] max_tokens=max_tokens, temperature=temperature, eos_id=tokenizer.eos_id, ) generated_words = [] for i, x in enumerate(generated_tokens): generated_words.append(tokenizer.decode(encoded_prompts[i] + x)) res = generated_words if should_print: for w, logprob in zip(res, _logprobs): print(w) logging.debug("Logprobs: %s", logprob) print("=====================") def mistral_chat() -> None: fire.Fire(interactive) def mistral_demo() -> None: fire.Fire(demo) if __name__ == "__main__": logging.basicConfig(level=logging.INFO) fire.Fire( { "interactive": interactive, "demo": demo, } ) ================================================ FILE: src/mistral_inference/mamba.py ================================================ import json from pathlib import Path from typing import List, Optional, Union import safetensors import torch import torch.nn as nn from mistral_inference.args import MambaArgs from mistral_inference.cache import BufferCache from mistral_inference.model import ModelBase _is_mamba_installed = False try: from mamba_ssm.models.config_mamba import MambaConfig from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel _is_mamba_installed = True except ImportError: _is_mamba_installed = False class Mamba(ModelBase, nn.Module): def __init__(self, args: MambaArgs): super().__init__() self.args = args assert _is_mamba_installed, "Mamba is not installed. Please install it using `pip install mamba-ssm`." # make sure naming is consistent with `mamba_ssm` config = MambaConfig( d_model=args.dim, n_layer=args.n_layers, vocab_size=args.vocab_size, ssm_cfg={"ngroups": args.n_groups, "layer": "Mamba2"}, attn_layer_idx=[], attn_cfg={}, rms_norm=args.rms_norm, residual_in_fp32=args.residual_in_fp32, fused_add_norm=args.fused_add_norm, pad_vocab_size_multiple=args.pad_vocab_size_multiple, tie_embeddings=args.tie_embeddings, ) self.model = MambaLMHeadModel(config) @property def dtype(self) -> torch.dtype: return next(self.parameters()).dtype @property def device(self) -> torch.device: return next(self.parameters()).device def forward( self, input_ids: torch.Tensor, seqlens: List[int], # not supported for now cache: Optional[BufferCache] = None, # not supported for now ) -> torch.Tensor: lm_output = self.model(input_ids) result: torch.Tensor = lm_output.logits return result @staticmethod def from_folder( folder: Union[Path, str], max_batch_size: int = 1, num_pipeline_ranks: int = 1, device: Union[torch.device, str] = "cuda", dtype: Optional[torch.dtype] = None, ) -> "Mamba": with open(Path(folder) / "params.json", "r") as f: model_args = MambaArgs.from_dict(json.load(f)) with torch.device("meta"): model = Mamba(model_args) model_file = Path(folder) / "consolidated.safetensors" assert model_file.exists(), f"Make sure {model_file} exists." loaded = safetensors.torch.load_file(str(model_file)) model.load_state_dict(loaded, assign=True, strict=True) return model.to(device=device, dtype=dtype) ================================================ FILE: src/mistral_inference/model.py ================================================ from abc import ABC, abstractmethod from pathlib import Path from typing import List, Optional, Union import torch import torch.nn as nn from mistral_inference.cache import BufferCache class ModelBase(nn.Module, ABC): def __init__(self) -> None: super().__init__() @property @abstractmethod def dtype(self) -> torch.dtype: pass @property @abstractmethod def device(self) -> torch.device: pass @abstractmethod def forward( self, input_ids: torch.Tensor, seqlens: List[int], # not supported for now cache: Optional[BufferCache] = None, # not supported for now ) -> torch.Tensor: pass @staticmethod @abstractmethod def from_folder( folder: Union[Path, str], max_batch_size: int = 1, num_pipeline_ranks: int = 1, device: Union[torch.device, str] = "cuda", dtype: Optional[torch.dtype] = None, ) -> "ModelBase": pass ================================================ FILE: src/mistral_inference/moe.py ================================================ import dataclasses from typing import List import torch import torch.nn.functional as F from simple_parsing.helpers import Serializable from torch import nn @dataclasses.dataclass class MoeArgs(Serializable): num_experts: int num_experts_per_tok: int class MoeLayer(nn.Module): def __init__(self, experts: List[nn.Module], gate: nn.Module, moe_args: MoeArgs): super().__init__() assert len(experts) > 0 self.experts = nn.ModuleList(experts) self.gate = gate self.args = moe_args def forward(self, inputs: torch.Tensor) -> torch.Tensor: gate_logits = self.gate(inputs) weights, selected_experts = torch.topk(gate_logits, self.args.num_experts_per_tok) weights = F.softmax(weights, dim=1, dtype=torch.float).to(inputs.dtype) results = torch.zeros_like(inputs) for i, expert in enumerate(self.experts): batch_idx, nth_expert = torch.where(selected_experts == i) results[batch_idx] += weights[batch_idx, nth_expert, None] * expert(inputs[batch_idx]) return results ================================================ FILE: src/mistral_inference/rope.py ================================================ from typing import Tuple import torch def precompute_freqs_cis(dim: int, end: int, theta: float) -> torch.Tensor: freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim)) t = torch.arange(end, device=freqs.device) freqs = torch.outer(t, freqs).float() return torch.polar(torch.ones_like(freqs), freqs) # complex64 def apply_rotary_emb( xq: torch.Tensor, xk: torch.Tensor, freqs_cis: torch.Tensor, ) -> Tuple[torch.Tensor, torch.Tensor]: xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2)) xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2)) freqs_cis = freqs_cis[:, None, :] xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(-2) xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(-2) return xq_out.type_as(xq), xk_out.type_as(xk) def precompute_freqs_cis_2d( dim: int, height: int, width: int, theta: float, ) -> torch.Tensor: """ freqs_cis: 2D complex tensor of shape (height, width, dim // 2) to be indexed by (height, width) position tuples """ # (dim / 2) frequency bases freqs = 1.0 / (theta ** (torch.arange(0, dim, 2).float() / dim)) h = torch.arange(height, device=freqs.device) w = torch.arange(width, device=freqs.device) freqs_h = torch.outer(h, freqs[::2]).float() freqs_w = torch.outer(w, freqs[1::2]).float() freqs_2d = torch.cat( [ freqs_h[:, None, :].repeat(1, width, 1), freqs_w[None, :, :].repeat(height, 1, 1), ], dim=-1, ) return torch.polar(torch.ones_like(freqs_2d), freqs_2d) ================================================ FILE: src/mistral_inference/transformer.py ================================================ import json import logging import math from dataclasses import dataclass from pathlib import Path from typing import Any, List, Mapping, Optional, Union import safetensors.torch import torch from torch import nn from mistral_inference.args import PATCH_MERGE, TransformerArgs from mistral_inference.cache import BufferCache, CacheInputMetadata from mistral_inference.lora import LoRALoaderMixin from mistral_inference.model import ModelBase from mistral_inference.rope import precompute_freqs_cis from mistral_inference.transformer_layers import RMSNorm, TransformerBlock from mistral_inference.vision_encoder import PatchMerger, VisionLanguageAdapter, VisionTransformer @dataclass class SimpleInputMetadata: # rope absolute positions positions: torch.Tensor @staticmethod def from_seqlens(seqlens: List[int], device: torch.device) -> "SimpleInputMetadata": return SimpleInputMetadata( positions=torch.cat([torch.arange(0, seqlen) for seqlen in seqlens]).to(device=device, dtype=torch.long) ) class Transformer(ModelBase, LoRALoaderMixin): def __init__( self, args: TransformerArgs, pipeline_rank: int = 0, num_pipeline_ranks: int = 1, softmax_fp32: bool = True, ): super().__init__() self.args = args self.vocab_size = args.vocab_size self.n_layers = args.n_layers self._precomputed_freqs_cis: Optional[torch.Tensor] = None assert self.vocab_size > 0 assert pipeline_rank < num_pipeline_ranks, (pipeline_rank, num_pipeline_ranks) self.pipeline_rank = pipeline_rank self.num_pipeline_ranks = num_pipeline_ranks self.softmax_fp32 = softmax_fp32 # Modules specific to some ranks: self.tok_embeddings: Optional[nn.Embedding] = None self.norm: Optional[RMSNorm] = None self.output: Optional[nn.Linear] = None if pipeline_rank == 0: self.tok_embeddings = nn.Embedding(args.vocab_size, args.dim) self.vision_encoder: Optional[VisionTransformer] = None self.vision_language_adapter: Optional[VisionLanguageAdapter] = None if args.vision_encoder is not None: self.vision_encoder = VisionTransformer(args.vision_encoder) self.vision_language_adapter = VisionLanguageAdapter( args.vision_encoder.hidden_size, args.dim, args.vision_encoder.adapter_bias ) if args.vision_encoder.add_pre_mm_projector_layer_norm: self.pre_mm_projector_norm = RMSNorm(args.vision_encoder.hidden_size, eps=1e-5) if args.vision_encoder.mm_projector_id == PATCH_MERGE: self.patch_merger = PatchMerger( vision_encoder_dim=args.vision_encoder.hidden_size, spatial_merge_size=args.vision_encoder.spatial_merge_size, ) if pipeline_rank == num_pipeline_ranks - 1: self.norm = RMSNorm(args.dim, eps=args.norm_eps) self.output = nn.Linear(args.dim, args.vocab_size, bias=False) # Initialize all layers but slice off those not of this rank. layers = [ TransformerBlock( dim=args.dim, hidden_dim=args.hidden_dim, n_heads=args.n_heads, n_kv_heads=args.n_kv_heads, head_dim=args.head_dim, norm_eps=args.norm_eps, lora=args.lora, moe=args.moe, ) for _ in range(args.n_layers) ] num_layers_per_rank = math.ceil(self.n_layers / self.num_pipeline_ranks) offset = self.pipeline_rank * num_layers_per_rank end = min(self.n_layers, offset + num_layers_per_rank) self.layers = nn.ModuleDict({str(i): layers[i] for i in range(offset, end)}) self.n_local_layers = len(self.layers) @property def dtype(self) -> torch.dtype: return next(self.parameters()).dtype @property def device(self) -> torch.device: return next(self.parameters()).device @property def freqs_cis(self) -> torch.Tensor: # We cache freqs_cis but need to take care that it is on the right device # and has the right dtype (complex64). The fact that the dtype is different # from the module's dtype means we cannot register it as a buffer if self._precomputed_freqs_cis is None: # default to 10**6 theta = self.args.rope_theta or 1000000.0 self._precomputed_freqs_cis = precompute_freqs_cis(self.args.head_dim, 128_000, theta) if self._precomputed_freqs_cis.device != self.device: self._precomputed_freqs_cis = self._precomputed_freqs_cis.to(device=self.device) return self._precomputed_freqs_cis def embed_vision_language_features(self, input_ids: torch.Tensor, images: List[torch.Tensor]) -> torch.Tensor: assert self.tok_embeddings is not None assert self.vision_encoder is not None assert self.vision_language_adapter is not None assert self.args.vision_encoder is not None text_locations = input_ids != self.args.vision_encoder.image_token_id image_locations = input_ids == self.args.vision_encoder.image_token_id text_features = self.tok_embeddings(input_ids[text_locations]) image_features = self.vision_encoder(images) if self.args.vision_encoder.add_pre_mm_projector_layer_norm: image_features = self.pre_mm_projector_norm(image_features) if self.args.vision_encoder.mm_projector_id == PATCH_MERGE: patch_size = self.args.vision_encoder.patch_size img_patch_dims = [(img.shape[1] // patch_size, img.shape[2] // patch_size) for img in images] image_features = self.patch_merger(image_features, image_sizes=img_patch_dims) image_features = self.vision_language_adapter(image_features) N_txt, D_txt = text_features.shape N_img, D_img = image_features.shape seq_len = input_ids.shape[0] assert D_txt == D_img, f"Text features dim {D_txt} should be equal to image features dim {D_img}" assert seq_len == N_txt + N_img, ( f"seq_len {seq_len} should be equal to N_txt + N_img {(N_txt, N_img, image_locations.sum().item())}" ) combined_features = torch.empty( (seq_len, D_txt), dtype=text_features.dtype, device=text_features.device, ) combined_features[text_locations, :] = text_features combined_features[image_locations, :] = image_features return combined_features def forward_partial( self, input_ids: torch.Tensor, seqlens: List[int], cache: Optional[BufferCache] = None, images: Optional[List[torch.Tensor]] = None, ) -> torch.Tensor: """Local forward pass. If doing pipeline parallelism, this will return the activations of the last layer of this stage. For the last stage, this will return the normalized final embeddings. """ assert len(seqlens) <= self.args.max_batch_size, ( f"Max batch size is {self.args.max_batch_size}, got batch size of {len(seqlens)}" ) (num_toks,) = input_ids.shape assert sum(seqlens) == num_toks, (sum(seqlens), num_toks) input_metadata: List[CacheInputMetadata] | List[SimpleInputMetadata] if cache is not None: input_metadata = cache.get_input_metadata(seqlens) else: input_metadata = [SimpleInputMetadata.from_seqlens(seqlens, self.device) for _ in range(len(self.layers))] if self.pipeline_rank == 0: assert self.tok_embeddings is not None if self.vision_encoder is not None and images: h = self.embed_vision_language_features(input_ids, images) else: h = self.tok_embeddings(input_ids) else: h = torch.empty(num_toks, self.args.dim, device=self.device, dtype=self.dtype) torch.distributed.recv(h, src=self.pipeline_rank - 1) # freqs_cis is always the same for every layer freqs_cis = self.freqs_cis[input_metadata[0].positions] for local_layer_id, layer in enumerate(self.layers.values()): if cache is not None: assert input_metadata is not None cache_metadata = input_metadata[local_layer_id] assert isinstance(cache_metadata, CacheInputMetadata) cache_view = cache.get_view(local_layer_id, cache_metadata) else: cache_view = None h = layer(h, freqs_cis, cache_view) if cache is not None: cache.update_seqlens(seqlens) if self.pipeline_rank < self.num_pipeline_ranks - 1: torch.distributed.send(h, dst=self.pipeline_rank + 1) return h else: # Last rank has a final normalization step. assert self.norm is not None return self.norm(h) # type: ignore def forward( self, input_ids: torch.Tensor, seqlens: List[int], cache: Optional[BufferCache] = None, images: Optional[List[torch.Tensor]] = None, ) -> torch.Tensor: h = self.forward_partial(input_ids, seqlens, cache=cache, images=images) if self.pipeline_rank < self.num_pipeline_ranks - 1: # ignore the intermediate activations as we'll get the final output from # the last stage outs = torch.empty(h.shape[0], self.vocab_size, device=h.device, dtype=h.dtype) else: assert self.output is not None outs = self.output(h) if self.num_pipeline_ranks > 1: torch.distributed.broadcast(outs, src=self.num_pipeline_ranks - 1) if self.softmax_fp32: return outs.float() else: return outs def load_state_dict(self, state_dict: Mapping[str, Any], strict: bool = True, assign: bool = False) -> None: state_to_load = {} skipped = set([]) for k, v in state_dict.items(): if k.startswith("tok_embeddings"): if self.pipeline_rank == 0: state_to_load[k] = v else: logging.debug( "Skipping parameter %s at pipeline rank %d", k, self.pipeline_rank, ) skipped.add(k) elif k.startswith("norm") or k.startswith("output"): if self.pipeline_rank == self.num_pipeline_ranks - 1: state_to_load[k] = v else: logging.debug( "Skipping parameter %s at pipeline rank %d", k, self.pipeline_rank, ) skipped.add(k) elif k.startswith("layers"): layer_id = k.split(".")[1] if layer_id in self.layers: state_to_load[k] = v else: logging.debug( "Skipping parameter %s at pipeline rank %d", k, self.pipeline_rank, ) skipped.add(k) elif any( k.startswith(key) for key in ["vision_encoder", "vision_language_adapter", "patch_merger", "pre_mm_projector_norm"] ): if self.pipeline_rank == 0: state_to_load[k] = v else: logging.debug( "Skipping parameter %s at pipeline rank %d", k, self.pipeline_rank, ) skipped.add(k) else: raise ValueError(f"Unexpected key {k}") assert set(state_dict.keys()) == skipped.union(set(state_to_load.keys())) super().load_state_dict(state_to_load, strict=strict, assign=assign) @staticmethod def from_folder( folder: Union[Path, str], max_batch_size: int = 1, num_pipeline_ranks: int = 1, device: Union[torch.device, str] = "cuda", dtype: Optional[torch.dtype] = None, softmax_fp32: bool = True, ) -> "Transformer": with open(Path(folder) / "params.json", "r") as f: model_args = TransformerArgs.from_dict(json.load(f)) model_args.max_batch_size = max_batch_size if num_pipeline_ranks > 1: pipeline_rank = torch.distributed.get_rank() else: pipeline_rank = 0 with torch.device("meta"): model = Transformer( model_args, pipeline_rank=pipeline_rank, num_pipeline_ranks=num_pipeline_ranks, softmax_fp32=softmax_fp32, ) pt_model_file = Path(folder) / "consolidated.00.pth" safetensors_model_file = Path(folder) / "consolidated.safetensors" assert pt_model_file.exists() or safetensors_model_file.exists(), ( f"Make sure either {pt_model_file} or {safetensors_model_file} exists" ) assert not (pt_model_file.exists() and safetensors_model_file.exists()), ( f"Both {pt_model_file} and {safetensors_model_file} cannot exist" ) if pt_model_file.exists(): loaded = torch.load(str(pt_model_file), mmap=True) else: loaded = safetensors.torch.load_file(str(safetensors_model_file)) model.load_state_dict(loaded, assign=True, strict=True) return model.to(device=device, dtype=dtype) ================================================ FILE: src/mistral_inference/transformer_layers.py ================================================ from functools import partial from typing import Optional, Tuple, Type, Union import torch from torch import nn from xformers.ops.fmha import memory_efficient_attention # type: ignore from xformers.ops.fmha.attn_bias import BlockDiagonalMask from mistral_inference.args import LoraArgs from mistral_inference.cache import CacheView from mistral_inference.lora import LoRALinear from mistral_inference.moe import MoeArgs, MoeLayer from mistral_inference.rope import apply_rotary_emb def repeat_kv(keys: torch.Tensor, values: torch.Tensor, repeats: int, dim: int) -> Tuple[torch.Tensor, torch.Tensor]: keys = torch.repeat_interleave(keys, repeats=repeats, dim=dim) values = torch.repeat_interleave(values, repeats=repeats, dim=dim) return keys, values def maybe_lora( lora_args: Optional[LoraArgs], ) -> Union[Type[nn.Linear], partial[LoRALinear]]: if lora_args is None: return nn.Linear else: return partial(LoRALinear, rank=lora_args.rank, scaling=lora_args.scaling) class Attention(nn.Module): def __init__( self, dim: int, n_heads: int, head_dim: int, n_kv_heads: int, lora: Optional[LoraArgs] = None, ): super().__init__() self.n_heads: int = n_heads self.head_dim: int = head_dim self.n_kv_heads: int = n_kv_heads self.repeats = self.n_heads // self.n_kv_heads self.scale = self.head_dim**-0.5 MaybeLora = maybe_lora(lora) self.wq = MaybeLora(dim, n_heads * head_dim, bias=False) self.wk = MaybeLora(dim, n_kv_heads * head_dim, bias=False) self.wv = MaybeLora(dim, n_kv_heads * head_dim, bias=False) self.wo = MaybeLora(n_heads * head_dim, dim, bias=False) def forward( self, x: torch.Tensor, freqs_cis: torch.Tensor, cache: Optional[CacheView] = None, mask: Optional[BlockDiagonalMask] = None, ) -> torch.Tensor: assert mask is None or cache is None seqlen_sum, _ = x.shape xq, xk, xv = self.wq(x), self.wk(x), self.wv(x) xq = xq.view(seqlen_sum, self.n_heads, self.head_dim) xk = xk.view(seqlen_sum, self.n_kv_heads, self.head_dim) xv = xv.view(seqlen_sum, self.n_kv_heads, self.head_dim) xq, xk = apply_rotary_emb(xq, xk, freqs_cis=freqs_cis) if cache is None: key, val = xk, xv elif cache.prefill: key, val = cache.interleave_kv(xk, xv) cache.update(xk, xv) else: cache.update(xk, xv) key, val = cache.key, cache.value key = key.view(seqlen_sum * cache.max_seq_len, self.n_kv_heads, self.head_dim) val = val.view(seqlen_sum * cache.max_seq_len, self.n_kv_heads, self.head_dim) # Repeat keys and values to match number of query heads key, val = repeat_kv(key, val, self.repeats, dim=1) # xformers requires (B=1, S, H, D) xq, key, val = xq[None, ...], key[None, ...], val[None, ...] output = memory_efficient_attention(xq, key, val, mask if cache is None else cache.mask) output = output.view(seqlen_sum, self.n_heads * self.head_dim) assert isinstance(output, torch.Tensor) return self.wo(output) # type: ignore class FeedForward(nn.Module): def __init__(self, dim: int, hidden_dim: int, lora: Optional[LoraArgs] = None): super().__init__() MaybeLora = maybe_lora(lora) self.w1 = MaybeLora(dim, hidden_dim, bias=False) self.w2 = MaybeLora(hidden_dim, dim, bias=False) self.w3 = MaybeLora(dim, hidden_dim, bias=False) def forward(self, x: torch.Tensor) -> torch.Tensor: return self.w2(nn.functional.silu(self.w1(x)) * self.w3(x)) # type: ignore class RMSNorm(torch.nn.Module): def __init__(self, dim: int, eps: float = 1e-6): super().__init__() self.eps = eps self.weight = nn.Parameter(torch.ones(dim)) def _norm(self, x: torch.Tensor) -> torch.Tensor: return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) def forward(self, x: torch.Tensor) -> torch.Tensor: output = self._norm(x.float()).type_as(x) return output * self.weight class TransformerBlock(nn.Module): def __init__( self, dim: int, hidden_dim: int, n_heads: int, n_kv_heads: int, head_dim: int, norm_eps: float, lora: Optional[LoraArgs] = None, moe: Optional[MoeArgs] = None, ): super().__init__() self.n_heads = n_heads self.dim = dim self.attention = Attention( dim=dim, n_heads=n_heads, head_dim=head_dim, n_kv_heads=n_kv_heads, lora=lora, ) self.attention_norm = RMSNorm(dim, eps=norm_eps) self.ffn_norm = RMSNorm(dim, eps=norm_eps) self.feed_forward: nn.Module if moe is not None: self.feed_forward = MoeLayer( experts=[FeedForward(dim=dim, hidden_dim=hidden_dim, lora=lora) for _ in range(moe.num_experts)], gate=nn.Linear(dim, moe.num_experts, bias=False), moe_args=moe, ) else: self.feed_forward = FeedForward(dim=dim, hidden_dim=hidden_dim, lora=lora) def forward( self, x: torch.Tensor, freqs_cis: torch.Tensor, cache: Optional[CacheView] = None, mask: Optional[BlockDiagonalMask] = None, ) -> torch.Tensor: r = self.attention.forward(self.attention_norm(x), freqs_cis, cache) h = x + r r = self.feed_forward.forward(self.ffn_norm(h)) out = h + r return out ================================================ FILE: src/mistral_inference/vision_encoder.py ================================================ from typing import List, Optional import torch import torch.nn as nn from xformers.ops.fmha.attn_bias import BlockDiagonalMask from mistral_inference.args import VisionEncoderArgs from mistral_inference.rope import precompute_freqs_cis_2d from mistral_inference.transformer_layers import RMSNorm, TransformerBlock def position_meshgrid( patch_embeds_list: list[torch.Tensor], ) -> torch.Tensor: positions = torch.cat( [ torch.stack( torch.meshgrid( torch.arange(p.shape[-2]), torch.arange(p.shape[-1]), indexing="ij", ), dim=-1, ).reshape(-1, 2) for p in patch_embeds_list ] ) return positions class VisionTransformer(nn.Module): def __init__(self, args: VisionEncoderArgs): super().__init__() self.args = args self.patch_conv = nn.Conv2d( in_channels=args.num_channels, out_channels=args.hidden_size, kernel_size=args.patch_size, stride=args.patch_size, bias=False, ) self.ln_pre = RMSNorm(args.hidden_size, eps=1e-5) self.transformer = VisionTransformerBlocks(args) head_dim = self.args.hidden_size // self.args.num_attention_heads assert head_dim % 2 == 0, "ROPE requires even head_dim" self._freqs_cis: Optional[torch.Tensor] = None @property def max_patches_per_side(self) -> int: return self.args.image_size // self.args.patch_size @property def device(self) -> torch.device: return next(self.parameters()).device @property def freqs_cis(self) -> torch.Tensor: if self._freqs_cis is None: self._freqs_cis = precompute_freqs_cis_2d( dim=self.args.hidden_size // self.args.num_attention_heads, height=self.max_patches_per_side, width=self.max_patches_per_side, theta=self.args.rope_theta, ) if self._freqs_cis.device != self.device: self._freqs_cis = self._freqs_cis.to(device=self.device) return self._freqs_cis def forward( self, images: List[torch.Tensor], ) -> torch.Tensor: """ Args: images: list of N_img images of variable sizes, each of shape (C, H, W) Returns: image_features: tensor of token features for all tokens of all images of shape (N_toks, D) """ # pass images through initial convolution independently patch_embeds_list = [self.patch_conv(img.unsqueeze(0)).squeeze(0) for img in images] # flatten to a single sequence patch_embeds = torch.cat([p.flatten(1).permute(1, 0) for p in patch_embeds_list], dim=0) patch_embeds = self.ln_pre(patch_embeds) # positional embeddings positions = position_meshgrid(patch_embeds_list).to(self.device) freqs_cis = self.freqs_cis[positions[:, 0], positions[:, 1]] # pass through Transformer with a block diagonal mask delimiting images mask = BlockDiagonalMask.from_seqlens( [p.shape[-2] * p.shape[-1] for p in patch_embeds_list], ) out = self.transformer(patch_embeds, mask=mask, freqs_cis=freqs_cis) # remove batch dimension of the single sequence return out # type: ignore[no-any-return] class VisionLanguageAdapter(nn.Module): def __init__(self, in_dim: int, out_dim: int, bias: bool = True): super().__init__() self.w_in = nn.Linear( in_dim, out_dim, bias=bias, ) self.gelu = nn.GELU() self.w_out = nn.Linear(out_dim, out_dim, bias=bias) def forward(self, x: torch.Tensor) -> torch.Tensor: return self.w_out(self.gelu(self.w_in(x))) # type: ignore[no-any-return] class VisionTransformerBlocks(nn.Module): def __init__(self, args: VisionEncoderArgs): super().__init__() self.layers = torch.nn.ModuleList() for _ in range(args.num_hidden_layers): self.layers.append( TransformerBlock( dim=args.hidden_size, hidden_dim=args.intermediate_size, n_heads=args.num_attention_heads, n_kv_heads=args.num_attention_heads, head_dim=args.hidden_size // args.num_attention_heads, norm_eps=1e-5, ) ) def forward( self, x: torch.Tensor, mask: BlockDiagonalMask, freqs_cis: Optional[torch.Tensor], ) -> torch.Tensor: for layer in self.layers: x = layer(x, mask=mask, freqs_cis=freqs_cis) return x class PatchMerger(nn.Module): """ Learned merging of spatial_merge_size ** 2 patches """ def __init__( self, vision_encoder_dim: int, spatial_merge_size: int, ) -> None: super().__init__() mlp_input_dim = vision_encoder_dim * (spatial_merge_size**2) self.spatial_merge_size = spatial_merge_size self.mlp_input_dim = mlp_input_dim self.merging_layer = nn.Linear(mlp_input_dim, vision_encoder_dim, bias=False) def forward(self, x: torch.Tensor, image_sizes: list[tuple[int, int]]) -> torch.Tensor: # image_sizes specified in tokens assert sum([h * w for h, w in image_sizes]) == len(x), f"{sum([h * w for h, w in image_sizes])} != {len(x)}" # x is (N, vision_encoder_dim) x = self.permute(x, image_sizes) # x is (N / spatial_merge_size ** 2, # vision_encoder_dim * spatial_merge_size ** 2) x = self.merging_layer(x) # x is (N / spatial_merge_size ** 2, vision_encoder_dim) return x def permute( self, x: torch.Tensor, image_sizes: list[tuple[int, int]], ) -> torch.Tensor: """ Args: x: (N, D) where N is flattened and concatenated patch tokens for all images image_sizes: list of tuple of (height, width) in tokens for each image Returns: image_features: reorders patch tokens so each grid of (spatial_merge_size, spatial_merge_size) is contiguous. now (N / spatial_merge_size ** 2, D * spatial_merge_size ** 2) """ sub_grids = get_sub_grids( x=x, image_sizes=image_sizes, spatial_merge_size=self.spatial_merge_size ) # list of [d x sub_grid_size x sub_grid_size x n_patches] permuted_tensor = [ grid.view(-1, grid.shape[-1]).t() for grid in sub_grids ] # n_patches x d * sub_grid_size * sub_grid_size return torch.cat(permuted_tensor, dim=0) # (N / spatial_merge_size ** 2, d * spatial_merge_size ** 2) def get_sub_grids( x: torch.Tensor, image_sizes: list[tuple[int, int]], spatial_merge_size: int, ) -> list[torch.Tensor]: # image_sizes specified in tokens tokens_per_image = [h * w for h, w in image_sizes] d = x.shape[-1] all_img_sub_grids: list[torch.Tensor] = [] sub_grid_size = spatial_merge_size for image_index, image_tokens in enumerate(x.split(tokens_per_image)): # Reshape image_tokens into a 2D grid h, w = image_sizes[image_index] image_grid = image_tokens.view(h, w, d).permute(2, 0, 1)[None, :, :, :] # 1 x d x h x w sub_grids = torch.nn.functional.unfold(image_grid, kernel_size=sub_grid_size, stride=sub_grid_size) sub_grids = sub_grids.view( 1, d, sub_grid_size, sub_grid_size, -1 ) # 1 x d x sub_grid_size x sub_grid_size x n_patches all_img_sub_grids.append(sub_grids[0]) return all_img_sub_grids ================================================ FILE: tests/test_generate.py ================================================ from typing import List import numpy as np import torch from mistral_inference.args import VisionEncoderArgs from mistral_inference.generate import generate_mamba from mistral_inference.main import generate from mistral_inference.mamba import Mamba, MambaArgs from mistral_inference.transformer import Transformer, TransformerArgs class DebugTokenizer: @property def bos_id(self) -> int: return 0 @property def eos_id(self) -> int: return 1 @property def pad_id(self) -> int: return -1 def encode(self, s: str, bos: bool = True) -> List[int]: assert isinstance(s, str) t = [int(x) for x in s.split()] if bos: t = [self.bos_id, *t] return t def decode(self, t: List[int]) -> str: return " ".join([str(x) for x in t]) def test_generation_transformer() -> None: torch.manual_seed(42) sequences = ["1 2 3 4 5 6 7", "0 1 2", "12 13 14", "2 4 34"] args = TransformerArgs( dim=512, n_layers=1, head_dim=128, hidden_dim=2048, n_heads=4, n_kv_heads=2, norm_eps=1e-5, vocab_size=32_000, max_batch_size=len(sequences), ) model = Transformer(args).to("cuda", dtype=torch.float32) tokenizer = DebugTokenizer() encoded = [tokenizer.encode(s, bos=True) for s in sequences] toks, all_logprobs_old = generate(encoded, model, temperature=0.0, max_tokens=7) # concat generated and prompt encoded = [e + t for e, t in zip(encoded, toks)] generated, all_logprobs_new = generate(encoded, model, temperature=0.0, max_tokens=0) assert generated == [] # Verify that logprobs are the same assert len(sequences) == len(all_logprobs_old) == len(all_logprobs_new) for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new): assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f"\n{lp_old}\n{lp_new}" print("All tests passed.") def test_generation_pixtral() -> None: torch.manual_seed(42) gen = np.random.default_rng(seed=42) sequences = ["1 2 2 2 2 4 5 6 7", "12 13 14", "2 2 2 2 7 8 9"] images = [[gen.normal(size=(3, 4, 4))], [], [gen.normal(size=(3, 4, 4))]] args = TransformerArgs( dim=512, n_layers=1, head_dim=128, hidden_dim=2048, n_heads=4, n_kv_heads=2, norm_eps=1e-5, vocab_size=32_000, max_batch_size=len(sequences), vision_encoder=VisionEncoderArgs( hidden_size=128, num_channels=3, image_size=4, patch_size=2, intermediate_size=256, num_hidden_layers=1, num_attention_heads=2, rope_theta=10000, image_token_id=2, ), ) model = Transformer(args).to("cuda", dtype=torch.float32) tokenizer = DebugTokenizer() encoded = [tokenizer.encode(s, bos=True) for s in sequences] toks, all_logprobs_old = generate(encoded, model, images=images, temperature=0.0, max_tokens=7) # concat generated and prompt encoded = [e + t for e, t in zip(encoded, toks)] generated, all_logprobs_new = generate(encoded, model, images=images, temperature=0.0, max_tokens=0) assert generated == [] # Verify that logprobs are the same assert len(sequences) == len(all_logprobs_old) == len(all_logprobs_new) for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new): assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f"\n{lp_old}\n{lp_new}" print("All tests passed.") def test_generation_pixtral_patch_merger() -> None: torch.manual_seed(42) gen = np.random.default_rng(seed=42) sequences = ["1 2 2 2 2 4 5 6 7", "12 13 14", "2 2 2 2 7 8 9"] images = [[gen.normal(size=(3, 8, 8))], [], [gen.normal(size=(3, 8, 8))]] args = TransformerArgs( dim=512, n_layers=1, head_dim=128, hidden_dim=2048, n_heads=4, n_kv_heads=2, norm_eps=1e-5, vocab_size=32_000, max_batch_size=len(sequences), vision_encoder=VisionEncoderArgs( hidden_size=128, num_channels=3, image_size=8, patch_size=2, intermediate_size=256, num_hidden_layers=1, num_attention_heads=2, rope_theta=10000, image_token_id=2, adapter_bias=False, spatial_merge_size=2, add_pre_mm_projector_layer_norm=True, mm_projector_id="patch_merge", ), ) model = Transformer(args).to("cuda", dtype=torch.float32) tokenizer = DebugTokenizer() encoded = [tokenizer.encode(s, bos=True) for s in sequences] toks, all_logprobs_old = generate(encoded, model, images=images, temperature=0.0, max_tokens=7) # concat generated and prompt encoded = [e + t for e, t in zip(encoded, toks)] generated, all_logprobs_new = generate(encoded, model, images=images, temperature=0.0, max_tokens=0) assert generated == [] # Verify that logprobs are the same assert len(sequences) == len(all_logprobs_old) == len(all_logprobs_new) for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new): assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f"\n{lp_old}\n{lp_new}" print("All tests passed.") def test_generation_mamba() -> None: torch.manual_seed(42) sequences = ["1 2 3 4 5 6 7"] args = MambaArgs( dim=512, n_layers=1, n_groups=1, rms_norm=True, residual_in_fp32=True, fused_add_norm=True, pad_vocab_size_multiple=1, tie_embeddings=False, vocab_size=32768, ) model = Mamba(args).to("cuda", dtype=torch.float32) tokenizer = DebugTokenizer() encoded = [tokenizer.encode(s, bos=True) for s in sequences] toks, all_logprobs_old = generate_mamba(encoded, model, temperature=0.0, max_tokens=7) assert len(toks[0]) == 7 assert toks == [[25574, 14821, 11843, 23698, 12735, 23522, 27542]] def test_chunks_transformer() -> None: torch.manual_seed(42) sequences = [ " ".join([str(i) for i in range(7)]), " ".join([str(i) for i in range(9, 0, -1)]), ] args = TransformerArgs( dim=512, n_layers=1, head_dim=128, hidden_dim=2048, n_heads=4, n_kv_heads=2, norm_eps=1e-5, vocab_size=32_000, max_batch_size=3, ) model = Transformer(args).to("cuda", dtype=torch.float32) tokenizer = DebugTokenizer() encoded = [tokenizer.encode(s, bos=True) for s in sequences] toks, all_logprobs_old = generate(encoded, model, temperature=0.0, max_tokens=8) # concat generated and prompt encoded = [e + t for e, t in zip(encoded, toks)] generated, all_logprobs_new = generate(encoded, model, temperature=0.0, max_tokens=0, chunk_size=5) assert len(generated) == 0 for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new): assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f"\n{lp_old}\n{lp_new}" ================================================ FILE: tutorials/classifier.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "id": "0f47fc4b-7cdc-48ce-b42c-e0b73d902ece", "metadata": {}, "source": [ "# Train a classifier with Mistral 7B" ] }, { "cell_type": "markdown", "id": "bfd6bdfb-c8cc-4fb6-8969-153a97dfb576", "metadata": {}, "source": [ "In this tutorial, we will see how we can leverage Mistral 7B to train a classifier. We need:\n", "- The mistral codebase: `https://github.com/mistralai/mistral-inference`\n", "- The pretrained model and its tokenizer: `https://docs.mistral.ai/llm/mistral-v0.1`\n", "- A dataset to train our classifier\n", "\n", "We will train and evaluate the classifier on `Symptom2Disease`, a public classification dataset from Kaggle provided in a CSV format: https://www.kaggle.com/datasets/niyarrbarman/symptom2disease/" ] }, { "cell_type": "markdown", "id": "225f88da-96ab-4262-88f8-8f1d6d5224bd", "metadata": {}, "source": [ "## Set paths" ] }, { "cell_type": "code", "execution_count": 1, "id": "ebe4bc9b-d0f7-42a6-9e41-bb3d042e0e07", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "code_path = \"/codebase_path/mistral-inference\" # codebase\n", "data_path = Path(\"/dataset_path/Symptom2Disease.csv\") # dataset downloaded from Kaggle\n", "model_path = Path(\"/model_path/\") # model and tokenizer location" ] }, { "cell_type": "markdown", "id": "3fcc420b-2ac7-42e1-a3b1-8a80f87ccc10", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": 2, "id": "5e8227fe-d2fb-4a2f-a136-96ee2d32287f", "metadata": {}, "outputs": [], "source": [ "import csv\n", "import tqdm\n", "import torch\n", "import numpy as np\n", "\n", "import sys\n", "sys.path.append(code_path) # append the path where mistral-inference was cloned\n", "\n", "from mistral.model import Transformer\n", "from mistral_common.tokens.tokenizers.mistral import MistralTokenizer" ] }, { "cell_type": "markdown", "id": "3b9c8408-1bc1-4e79-8b6d-391e07d78eb4", "metadata": {}, "source": [ "## Load the model and tokenizer" ] }, { "cell_type": "code", "execution_count": 3, "id": "806ff308-5909-46f7-b10e-8b1f1c7b7edc", "metadata": {}, "outputs": [], "source": [ "model = Transformer.from_folder(model_path, dtype=torch.bfloat16)\n", "mistral_tokenizer = MistralTokenizer.from_file(str(model_path / \"tokenizer.model\"))\n", "tokenizer = mistral_tokenizer.instruct_tokenizer.tokenizer" ] }, { "cell_type": "markdown", "id": "4a9aa34b-cbe3-49a6-948b-17f6c84d93c6", "metadata": {}, "source": [ "The tokenizer is a critical component of the model that segments input text into word-pieces.\n", "\n", "For instance, the sentence `\"Roasted barramundi fish\"` is encoded as `['▁Ro', 'asted', '▁barr', 'am', 'und', 'i', '▁fish']`.\n", "\n", "The number of subwords in the model is set to `32000`, and each word is decomposed into known word pieces to avoid unknown words." ] }, { "cell_type": "markdown", "id": "ce0e03ef-439f-4078-a848-d69bf4e339ec", "metadata": {}, "source": [ "## Load the dataset" ] }, { "cell_type": "markdown", "id": "65d1f696-138d-467d-bb53-032e08d47bf1", "metadata": {}, "source": [ "We reload the Kaggle dataset from the disk. Each sample is composed of a sentence and a label. The dataset contains 24 different labels." ] }, { "cell_type": "code", "execution_count": 4, "id": "cc2bf18b-d99c-4c31-9988-1675d019e04b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Reloaded 1200 samples with 24 labels.\n" ] } ], "source": [ "data = [] # list of (text, label)\n", "with open(data_path, newline='') as csvfile:\n", " reader = csv.reader(csvfile)\n", " for i, row in enumerate(reader):\n", " if i == 0: # skip csv header\n", " continue\n", " data.append((row[2], row[1]))\n", "\n", "# label text to label ID\n", "labels = sorted({x[1] for x in data})\n", "txt_to_label = {x: i for i, x in enumerate(labels)}\n", "print(f\"Reloaded {len(data)} samples with {len(labels)} labels.\")\n", "\n", "# integer class for each datapoint\n", "data_class = [txt_to_label[x[1]] for x in data]" ] }, { "cell_type": "markdown", "id": "c5434265-fad8-4daa-b54b-438d420bcb4d", "metadata": {}, "source": [ "The task is to classify a symptom, for instance `\"I have a dry cough that never stops.\"` to one of the 24 disease labels (`Acne, Arthritis, Bronchial Asthma, Cervical spondylosis, Chicken pox, Common Cold, etc.`)." ] }, { "cell_type": "markdown", "id": "1540b537-c3cc-4851-80e3-91a9b9e8cc7c", "metadata": {}, "source": [ "## Embed data points in the dataset" ] }, { "cell_type": "markdown", "id": "c73d5d75-55eb-4210-8ffe-d0db3e9f8735", "metadata": {}, "source": [ "We will now learn a linear classifier on frozen features provided by Mistral 7B.\n", "In particular, each sentence in the dataset will be tokenized, and provided to the model.\n", "\n", "If the input sentence is composed of `N` tokens, the model output will be a list of `N` vectors of dimension `d=4096`, where `d` is the dimensionality of the model.\n", "These vectors are then averaged along the dimension 0 to get a vector of size `d`.\n", "\n", "Finally, the vectors of all sentences in the dataset are concatenated into a matrix of shape `(D, d)` where `D` is the number of samples in the dataset (in particular, D=1200)." ] }, { "cell_type": "code", "execution_count": 5, "id": "78019021-01d7-499f-94ea-a3c88bbcb42c", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "1200it [00:25, 46.65it/s]\n" ] } ], "source": [ "with torch.no_grad():\n", " featurized_x = []\n", " # compute an embedding for each sentence\n", " for i, (x, y) in tqdm.tqdm(enumerate(data)):\n", " tokens = tokenizer.encode(x, bos=True)\n", " tensor = torch.tensor(tokens).to(model.device)\n", " features = model.forward_partial(tensor, [len(tokens)]) # (n_tokens, model_dim)\n", " featurized_x.append(features.float().mean(0).cpu().detach().numpy())\n", "\n", "# concatenate sentence embeddings\n", "X = np.concatenate([x[None] for x in featurized_x], axis=0) # (n_points, model_dim)" ] }, { "cell_type": "markdown", "id": "7a997101-7844-4928-bdf4-d7a3051eca75", "metadata": {}, "source": [ "## Plot t-SNE embeddings" ] }, { "cell_type": "markdown", "id": "61bbc6b4-dfc5-42e0-b99b-1ed281d929cb", "metadata": {}, "source": [ "Now that we have one vector/embedding for each sample in our dataset, we can visualize them using t-SNE.\n", "t-SNE is a powerful tool for reducing the dimensionality of high-dimensional data, while preserving the underlying structure and relationships between the data points, which can help uncover patterns and insights that would be difficult to discern in the high-dimensional space.\n", "\n", "In the graph below, we assign a different color to each class in the dataset. It is apparent that several distinct clusters are visible in the data." ] }, { "cell_type": "code", "execution_count": 6, "id": "8e3cd894-ac30-4f34-a4c8-ef34293b587d", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOyddXhcVfrHP+fekbg2TdI2dTcqlFKglBaKS6HI4ostC8V3F1tBFn4sK8guLgss7lasUKe01N09tUjjNnbP74+bNEkzcmcysXI+zzNP03vPPedMMnPP977nFSGllCgUCoVCoVB0ELS2noBCoVAoFApFOCjxolAoFAqFokOhxItCoVAoFIoOhRIvCoVCoVAoOhRKvCgUCoVCoehQKPGiUCgUCoWiQ6HEi0KhUCgUig6FEi8KhUKhUCg6FLa2nkBzMQyDffv2kZiYiBCiraejUCgUCoXCAlJKysvL6dKlC5oWni2lw4uXffv2kZOT09bTUCgUCoVCEQG5ubl069YtrGs6vHhJTEwEzDeflJTUxrNRKBQKhUJhhbKyMnJycg6t4+HQ4cVL3VZRUlKSEi8KhUKhUHQwInH5UA67CoVCoVAoOhRKvCgUCoVCoehQKPGiUCgUCoWiQ6HEi0KhUCgUig6FEi8KhUKhUCg6FEq8KBQKhUKh6FAo8aJQKBQKhaJDocSLQqFQKBSKDkWHT1KnUASiQlaQL/MB6Cw6kyAS2nhGCoVCoYgGSrwojjhqZA0LfQvJlbmNjncX3Rmnj8MpnG00M4VCoVBEA7VtpDii8EovM7wz2CP3NDmXK3OZ4Z2BT/raYGYKhUKhiBbK8qI4otgut1NCid9zEkkxxeyQO+hJT3bKnRQYBQghyBJZ5IgcdKG37oQVCoVCETZKvCiOKLb6toZss9q3mqUsxY0bgQAJm9lMHHGcYjuFFJHS8hNVKBQKRcQo8aI4oqimOmSbCioO/SyRja79xvsNXUQXSmUpNmz01/rTW/RG09QOq0KhULQXlHhRHFHEEUcllRFdK5F48LBL7jp0rNAoZAlLOIdzSNCCRytJKSmUheySu/DgIZFEuoquJIgE7MIe0ZwUCoVC0RQlXhRHFP30fhT4CqLapwcPX/q+5BIuCWiB8UgPc3xz2C/3IxCHLDrLWQ5AjshhuD6cdJEe1bkpFArFLxFlC1ccUfQSvUgn3fRliSIePGyRWwKen+ebxwF5AGi8FVVHrszlG+83HDAORHVeCoVC8UtEWV4URxS60Jlsm8xi32K2y+1R7XudsY7+Wn9qqGGrsZUiWYSGRqpIZa/cG/J6A4M5vjkk+ZIopxw7dnppvRigDSBOxEV1rgqFQnEkI6SUTR8TOxBlZWUkJydTWlpKUlJSW09H0Y7Y6dvJPGNeVPtMIYVSSg9ZVxpuEUWCQGDHzqm2U0kTadGapkKhULR7mrN+K8uL4ojFX6K65nJ4DpnmCJe66924me6djh07/bR+jBKjVHSTQqFQBEHdIRVHLFVUtfUUwsKDh/XGej7xfYLX8Lb1dBQKhaLdosSL4ogljrioO+62BlVURX27S6FQKI4klHhRHLH01fo2e1unrdgr92IYRltPQ6FQKNolSrwojlgyRSY5IqetpxERdXWYFAqFQtEUJV4URyxCCE7UT2SAGNAm20c55NBf64+OKvaoUCgU0USJF8URjS50xtrGcpF+EbHEturYueTSmc4M1gaHfa1AkEpqC8xKoVAoOj5KvCh+EcRoMYzWR7f6uIuNxQwUA8MWTj1FTxUurVAoFAFQd0fFL4beWm8yRWaz+xkihpBBhqW2btxUUMFwbbjl/lNI4Xjt+Einp1AoFEc8Kkmd4hfDZt9m8mRes/roIXow2jYaj/TwifcTXLhCXlMhK/DitZSNd4AYwBhtjLK6KBQKRRCUeFH8YlhhrGjW9XXi4xvPNxzkIAbWQpmTRBIGhqWw7QH6ADShhItCoVAEQ4kXxS+CCllhyUoSDIlkt9wd9nWa0MghBw0tqOBJIYVkkpszRYUiKFJKDC/o9o6XvFGhaIgSL4pfBDWyps3G/tr7NafbTme4NpyVxsqA7UbpoxBCLSqK6FO8XzLzFVj4MdSUQ1yK5PiL4eTrILGT+swpOh5KvCh+EcSJuDYb24ePOd45TNGngAarjdUYGIe2oZw4OVY/lm5atxYZ3zAMlhpL2Sq34sWsmZRAAqP10fTQerTImIr2Q94OyRMXQ1UpGD7zWFUJzHwVlnwBv/9QktpFCRhFx0KJF8UvAh0dG7ZDi3drU0klP/h+YLJtMgO1geTKXFzSRYJIoKvoii5aJpGdYRh85vuMCioaHa+ggrm+uQyRQ9okhFzRerzxu8bCpQ7DB6V58Oa9cNv/2mZuCkWkKM9AxRGPIQ1memdGLFwyaX54NUAeeWyVW3EIB320PgzWB9Nd695iwgVgkbGoiXBpyDpjHaVGaYuNr2hb9myQ7FrVVLjUISVsWgDvPygxjI5ZB0zxy0SJF8URzz65j0IKI74+jzycOKMylw2+DVHpxyrb5faQbZYaS1thJoq2IHedtXbz3oSvn27ZuSgU0USJF8URz05jZ7NrGzU3UqmOUkqRsnWecF0+l6Vw7mKpCkAeqdQENro1YcaLUFmirC+KjoESL4ojHjduSzlWQhEN64uO3moRRXvkHkvt2qJopaJ12L3WelufB5Z/1XJzUSiiiRIviiOeRJEYlQXahYuJ2kR6iB7EE08ssWSKTAYwAM3CV0kgyBE5zZ6HVaxsGQF0EV1aeCaKtsDwSVZ8Hd41H/4Vln6prC+K9o+KNlIc8fTT+rHBiI6vSaaW6VeAHCWPYrOxmVXGqqBWniH6kKjMIxRSSst+PqM1FW10JOKuBk+Yu50+D7x2J8QmSYZMUBY5RftFWV4URzwpIoUhWvNFQzLJOITD77kYEcNwfThTbFNIJBEwLS11Fh8dnQn6BNJFerPnYYVCWYgHT8h28cTj0Py/J0XHxhEHzgjSGwlg+pNRn45CEVWU5UXxi2CUNooEElhjrKGKKqDe18OqP4wVq0miSOQ823nskXvYY+zBwCBdpNNH6xNQ+LQEW42tltoN1Ya28EwUbYWmCcZdJJnzRnjXSQm718DBPZL0bsr6omifKPGi+EUghGCAPoD+Wn9KKcWHjySSANOxtUbW4MPHamM1PnyHBE1dFtwBYgB9RB9LY2lCo7voTnete4u9n1DUCbRQ9NGsvSdFx2TyjTDvrcB5XoJRXR79+SgU0UKJF8UvCiEEKaQ0OtZL9Dr0c2+tN5uNzewyduHFS5pIY4A2gGyR3aHqDsUQc0h4BcKBA5tQt4AjmZRMQUYPSZ413+1DaDqkZrfMnBSKaKDuXApFA+JEHCP0EYzQR7T1VJpFH60P23zbAp4XCPpp/VpxRoq2YvTZ8M0zIEOn/AFM4TLyDIhP6ThiXfHLQznsKhRHIJkik66iq98QcYEghhgGa4PbYGaK1uaES8ERC1ayBWg6xKfAlHtaelYKRfNQ4kWhiAKGNFjrW8vX3q/52vs1K30r8RptUwQSzO2xk/ST6Cv6NhEwnejE6bbTiRWxbTQ7RWuS3Flwy+sQlxi8XZ3F5e7PIE1VmVa0c4RsrVzlLURZWRnJycmUlpaSlJTU1tNR/ALZb+xnpm9mk1T8AsEEfUKbOu4CVMtq9sv9hyKfUkVqm85H0Ta4qiRLv4DNP4O7CpI6w6gzoFN30zk3NVttFSlal+as30q8KBTNoNKo5GPfx0HbnKOfQ6qmBINCoVA0pDnrd6ttG/3tb39DCMEdd9xx6FhNTQ3Tpk0jPT2dhIQEpk6dSl5eXmtNSaFoNkuMJVFpo1C0BLnrJG/8TvLARMnvRkjuHyd56x5J7roO/cyqULROtNGSJUt48cUXGT58eKPjd955J1999RUffvghycnJ3HLLLVxwwQUsWLCgNaalUDSbfXJfyDb5Mr8VZqJoL1SVSZZ8Dge2mRluR54OPYa37naMzyt5535YdJhRsKYcFn5kvi55WHLi5WqbSNExaXHxUlFRweWXX87LL7/MI488cuh4aWkpr776Ku+88w6TJk0C4LXXXmPQoEEsWrSIY489tqWnplA0m8P9XCJtozgyWPy55O37wOsGXTez1X7/Igw8QXLdf2DfJvjpAyjcDQlpMOY8GH4y6PboioivnoJFnwRv8/5foLJEUrgLfD7oNQKOmQKxiUrQKNo/LS5epk2bxllnncUpp5zSSLwsW7YMj8fDKaeccujYwIED6d69OwsXLgwoXlwuFy5XfbWxsrKylpu8QhGCGGJCZrO1Y2+l2Sjako0LJG/8DuryAvoaBJtt+gn+ehqU5ZtRPYYPhAarZkD3YXDL65L4FEFNhWThR7DoIyg/CGld4bhLYMy5YHcGFhVSSrb8bGbT3b0GDu6tn0cwpj9hzgdgyefw2d/hhmclg09UAkbRvmlR8fLee++xfPlylixpuud/4MABHA4HKSkpjY5nZmZy4MCBgH0+9thjPPTQQ9GeqkIREQO1gSw3lgdt01f0baXZKNqSr/8DQpjWlsORhilcoD5Vf13SuD3r4aWb4Lw/SP73ByjYVXcRlBXCjhWw4D249X+SmPimokJKySePwaxX64VRODRs766C52+A+76QdBmgBIyi/dJiDru5ubncfvvtvP3228TExESt3/vuu4/S0tJDr9zc3Kj1rVCEy2Ax+FAVaX/EEstIbWQrzkjRFlSWSLYtsZ7FtiGGD7Yuhn9dBAU7MS0mtQKorr9dq+Gjv4LP01QZLfnCFC51fTUXwwtPXQ4VRcqpV9F+aTHxsmzZMvLz8xk1ahQ2mw2bzcbcuXP597//jc1mIzMzE7fbTUlJSaPr8vLyyMrKCtiv0+kkKSmp0UuhaCs0TeNc/VxyRE6TZHDZZDNFn4JNU1U4jnTc1upgRow0YOGHcNtAuH+c5O37JDtXSaSUzHzFtPhEk8pieO766PapUESTFsvzUl5ezq5duxodu+aaaxg4cCD33HMPOTk5ZGRk8O677zJ16lQANm3axMCBA4P6vByOyvOiaC94DS8FFCCRZJCBXevYvi4bdxqs2CgRAiaNEXROUwm5A+F1S+4ZAzUVrTtuVl84sLXl+u81Cqb9VznxKlqG5qzfLfZImJiYyNChQxsdi4+PJz09/dDx6667jrvuuou0tDSSkpK49dZbGTdunIo0UnRIbJqNbNpfKd6qGklRGSTFQ1K8QErJ7gNQUiHpnCrI7tR4Ydq1X3L3v70cOFh/7PmPoU83H0/dpZOUoETM4dgcgmMvlMx5vXXHbUnhArBzBbxwA9zxruxQVdUVRz5tas9+8skn0TSNqVOn4nK5OO2003juuefackoKxRHD/kLJq5/7mLVE4qv1nejZBaprIK+ovl33LLjnap2hfTTyiyQ3POLF5Wna37Y9cNmffHz0OMQ4lYBpiJSSU66j1cVLSyMlbF1ivvod09azUSjqUeUBFIojkD35kpse81JZzSHhEooHb9T5aZXBjEXBbwkXTBLc/ivlxyNLd2DsnIEsWAWGDxnflT/ffjflxQ4slXDuIGg2szL1JQ8eOe9J0T7oEOUBFApF6/H0uz4qwhAuAI+84mPm4lDPMpJPF3j5U+U2fvSU4OvYzz4RYxxYgu/nvyHzV5jhOUhE5V5OmvCVpetFR7rzSrNwo0LRnuhIXyGFQmGBvIOSxeskRphhu16fFbEjkC6dFZ4K/q96F3+p2o47kvjgDox0lWGseQ2Qh8VGS06aNJOcHrsIliHu6n/BQ7MhJqGlZxodpITMXm09C4WiMUq8KBRHGHvyW9gaoklk7Z1jpa+CN1yBk0oeich9P4H0n1DFZvdy++/+wUmnLcLmaHwutQv85gU4ZoogvZvgtjfB5myFCUeBcRe29QwUisaojWuFIoq4pIEEYsLcFyirlKzaLPH4YEB3QdfO4fkXuDySr380+Hyuwb6CsC5tRGIclIfIWaINqt9DkMDX7kKucGYSK/TIB+5AyLJdQc/b7V7On/I6U/8zlvwdGqX5kJRhhjU3jNjpMVww4jTJ0i9aesaRUZct+Px7ISVL+bso2hdKvCgUzURKyfvufD5x5VNRW4QxFo0zHelc68wOGmLq9kie+9DHl/Ml3gYP88cMEdx9lU5GqsAwJNv3QmW1JDEOtu6BGpekRxfB8L6Cahfc9aSXDTua/14mj4Uv5tFoLg3eKQhwntG4SrYLyVZfNcNsHWQfpLlotsB1AA4hEJogq68gK0h1iHEX0W7FS1ZfOONWGH2WEi6K9ocSLwpFM3mwagdLfI09Gqsx+NhdwBpvBU/G9/MrYKSUPPiSj59Wyybr4LINkml/93LlmRpvf2Owv9D/2DmZ0CmVqAgXAVxyqo2zT5Dc9i8fFYdbYOwGzmty0bLczR+sAyM6DUPu/zlICw06DUFYsL4NPE6QkiUpaQc7b0KD7kPhsv+D2CRI64LK7aJotyjxolA0g3me4ibCpSGbjWo+cOdziTOzybkVmyQLVvl/evcZkH8Q/vlmcGfY3DzzFQ3+cJVOVrqAdMH0JwU/rjL44mcvyzzl6IPKsY0sR9iazteBoI8eG51JdABE5ijYkg6u4gDFjAz0nqda7u/mV+Bv50WnLpHQIquvJATEJUP/cZDVx0y6p1C0Z5TDrkLRDN6uCa0cPnf5d0L5dqGBHuQbGE2327qlSDtsTRICjhkKT/9O56wTNLxeicttjjx+hM4/bnQy+poSnGPK/AqXFFHNOc4CZnu/5Vvvt6zzraNG1kRx5u0PodnQR98BzpS6Iw3+1dCGXIVIG2C5v66DBHe8E9lcYhIguVYXO2Jh2CSwOayFYg+dVP+zlGY9o+9fhLuGwZaQIfMKRduiLC8KRTPIk6G3UErx/0hdUBxeHpbmcP0UjcR4yC+C1CSYeLRGSqIpZoQQLFlncOcTXlZslEigWyZceLLGuSdq3BPbg7srt5In3YcElQb01QsY5diHQHAQCRLyZT5rjDVMtk0mXaS3zptrA0R8Jvrxf0XmLUMWrAbDA4k5aN3GI2JSw+6vz9GCiddIZr8Wxhw06DEcbntTYPgkQoOHJ4PPG9z6ktwZRp4ZOBuwzwv/vgIeWyxJSFEWGEX7RIkXhaIZaAhC2UgC3f4zUkHTJIbR8gvE8SM0enXxP86nc3w89Y6BptW/kz158NQ7ZmHGB35j45mE/sz0FPO9u4hS6aWPXkVP+z4A5GHv34OHH7w/cIHtAuyiYxenDIbQ7Ygux0KX6NRiO/9ecFfDgvdA0wFJ0Fw90jAz3wJoumDrEkm+Bd+nO96BF38bvI3hg88fh8sfszx9haJVUdtGCkUzGKTHhWzTXcQ0OeaVkrKRha0iXAb0IKBw2V8oefpdc4X0t1DOXS759idJnNA5VaQzZUNfTpk/AMeiWKrLmr4vMMWMCxc7ZBS8iH9B6DbBZY8KHpwFZ94KJ1wOXQbgV/0KAUMnwojT6o9ZLdK4coa1tmtmWetPoWgLlOVFoWgGv3F24aaqzUHbXBOT1eTYSzV7Wd3rIPogB76NCSAPX6Ek0aqPc8elgZ9RvpxnBI36FQI+me0jNgb+9ZYZgaRr4JP9EB/0ZchJmxhz/io0rWkH+4399Nf6R+U9/JLI6CE441bzZ59XMuMFmP266ZMCEJcCE66EM6aZFpc6nPHW+v/879baeX/ZQWWKdo4SLwpFM+hhi+XWmG78p2aP3/OXOTIZY09udKzE8PC156CZM+WqPbi/yMS7OBV8DcVK9Cwy//1C8s87/J/bkhu8jICUsH0vPPSS79CMTD8dgZSCtbMGAjB26sqm10bV5fiXiW4TnHELnHqj5MB2QEJmb//RQJ1yojt2Wtfo9qdQRBO1baRQNJMzHOm8ET+IE2zJpAobyULnaD2B5+P7c4Ufq8tSbzl1ekHYJc6pB4j7y2ZEVg2I6C/4S9ZLzrvLw8Mve1m7rbFSiXGY1pVgSFmbk83vWcG6OQP8biFliIyI56xojG4XdB0g6DpQ+BUuBbskz/w6umOefUd0+1MooomyvCgUUSBDd3B/XE9LbWtoauoQ8T5ksd3P9lF0KKmA2UslM5f4uOlCya9ONVP5H3+UxrwVwROMhCocLaVgx4ocBk/YcuiYjk5fLUhqWUVU+eppcIUo6xAOg8bD8FNUpJGi/aIsLwpFK9ND8+/o2tLfRqNWhDz/kcGqzaaAmni0oHNq0/wv4SCExFVlViEUCDQ0TtJPwik6SNXBDo6rSrLsq8iS0x2OIxZOvh5ueV0JF0X7RokXhaKVGarHk0XTEGJ9UDn4cXyNNroGH80yVzqnQ/Dk72zozaipKA2N1PQaYmQs/bX+nGM7h65avcPEAcPFNl815dLb3Kkr/FBRDEYUfrVT7oEn1wouuE8JF0X7R20bKRStjAuJXdM4fPfIPqEI34pkohlp5A+fASs3mSJpxz7Ju9/68DRj8dN1mP/2Mcx6w0xud8FEjXNOlKyQ5fzPtZ/thplxVwPG21K4NiabDM0RhXeiAIhPjrwsQEM+exwqSyTn/UHVNFK0f5TlRaFoZb5yF7LHcDU5rnetwXn5XtBAawHH3YYIYP4Kg+se9vJ9hKng65Y3wwC315Rce/Lg3+8Z3PB0DQ+U72SHUV8qwADme0u4vXILBYaKw40WMQmCoybXJrZrJt+/CD++2/x+FIqWRokXhaKV+cp9MGAQsW1EGfH3b2HIaVUcPVhw7DDBKcdE9ylY12BYP3joZR8+I3gW16DUTquhQ6+sfe3YrOOZm97kfRpAmfTyes3+CAdV+OOs282aRtHgu+fBMFSYu6J9o7aNFEckbmkwy1PMDHcRLgx6a7Fc5swkW28dJ1KflJRJL14MDhge7LWVl20IDoSqh5TipdNpRTwYnwKA2yPZV+Bl4856p9tmzc2AtCSBz9e8zoJGIUnw/piKfWJhkyKBBjDPW8LNshvxIgrmgtbE54EN82H1DKgoMqsijjgD+o2NjukjQroMENzxjuSpy8wSA82heB/kbYPsftGZm0LREijxojji2Oer4fbKLVQ2cCrZYdQw01vMVEcG18V0abGxy6WX9135fOMupPowu0MiOhc4M3ACTTeN6tGAeK3+q+mwC/51p41n3vfx3SKJN3hkc0B0zRQuv7tCZ8Eqo9lCSNOCWW0EstyOrNARSU0n7AMKDQ/xzfEUbm2qy+Gde+BAfUg4BTtg6yLI7g9XPRk980cE9BguuOKJEv57U3LQdsEyKtfh9URxYrX4PJJVP8DWxeYc+o2FYSebifgUinBR4kVxROGTsolwacjH7gIyhYOznZ2iPu48TwnP1ewJOHY5Pt5wHQj5pTOAE20pjY7FxQjuvtrGjVMlG3dKhICNOw1e/TzwKjRiAAzsIVi6XlJWCfFx0LuLICEOBNJCScnmI2yBR4g/3CTT3pn+r8BFgfZvhrfvhqufatUp1VHp2ktuyQwYtJ9hl57EmnfHITQDaZi/Y003Bcuka2HmK8H7csRCRo/ozm/PBsmz10BZAWg2c8dxzhtmhetfPyXpP1YJGEV4KPGiOKKY4TkYUDzU8ZbrQFTFy37DxZ8rt7Mv1HZQLaECe5LROdqW6P9cgmDsUPNGP2awoNpl8M63/t/vxh1w2liNHxb7KCwBrRR2HzAT1cXFNE+4JMVDWWWQBkKidatGxPlJyAcM0OPo1JEijkr2w+YFwdvsWWeaFfoe0zpzqqXStZfNBW8hMetUHfPbOWSPyGXdh0eTv74rus3GsIk6k66BboNh+VdQkuc/OknTYdyFEBMfPTGx+WfJM1eBr/aD3zCsuzQfnr4MugyQnHU7jDhNiRiFNZR4URxRfOcuCtmmDB9Fhoc0rWmulXCpkQb3Vm7joIyend2NRLcQqupym+n9A87NDY//r37LpuEWT3XtvpUmIvOjGTlAsCdfsnNfXa2jw5ACx8kHA15/pbNp2YR2za7V1trNernVxUtuyfdIDBrK0Zxjt5Fz7DYAdC2WYdm3odX6F13/rOTpK8DrAqPBjp7QzCrW5/wuenNbP0/y3HWhw7j3bYKXb4YL/yyZ+GslYBSh6WB2W4UiONUhrC51lERJbMzzFFMgPRZHtYbLQm9fzjc4/w9e/vtFZCPX+TwIEVlGmX0FksdvtdGltnyRptX/K4DfXig4c6Qdgfn/Os+WODTuje3ByACWpXZLKCeROgp2Qu7aFp1KQ2o8B6ny7COYHc1nVFNWU7/d1fMowX1fwnGX1FeiTusC5/4O7nofYhOjIx7WzZU8e214+Wc++isU5qpIJ0VolOVFcUTRVTjJDeoOa7LJW8VBw8sIWwL2Zvhe/OgpjbrvSJII/rX8bqHBP9+M0Gv3MPxaTSyws8bNnbbNdPtDDGPXZ7B2pc5+lxct00W/cdX06JLCxbZu/MqZyQJPCVXSoIvm5Hh7Ms6O5usC0G2Q9bazXoWrn2y5uTTA5S211M7tLWv0/849BZf+FS79K0gpo56U7uBeyYs3EtEX4/U74fcfRXU6iiMQJV4URxSXOTNZVFUWst1/XHsBMwLo1zHZnOFIj2i8GoyoO70O0eMDnvMZkpc+iY5wiRhNYmRXUyS9FFPB6sEVMNi0sHiBDcC66hKOsSXyx9ieXODs3LbzjQadephh0aV5odvuWWu2S85s0SlJKSmoXGaprU2PC3iuJbLp/vhO4y2pcNi5EiqKJAlpavtIEZgO+AikUDRFSsnnrgLurtoW1nXl+PhPzR4eq9rJqzX7+NCVT14Y2V97ajFEO9j3XHsnpJQcMFzs8FVTKetXgbXbJIXWHrZbDkNgP64YaPxgXfdznTFnibect10WFvuOwtS/WG9bWdJi06gjv+Jnymq2hGynCTvJMa2btGXtrMjLFUgJa2ZFdz6KIw9leVEcEXzhLuRF176Ir5/vLUXHXIBfc+3nTHs6N8V0Dek4e4YjnemewI6p4dJDOCmXXqZVbmZnbWp9G4IJthSuicmmrKIN86IICVJgO/EgWp+qkM0lMN1dyKXOzI65VXQ42f1h1Dmw/MvQbRMis+RZRUofeeWLLLXNTjoRvZUjuyK1ugAgwBX646X4hXME3FEUv3RqpMH/XAea3Y+PeqvBN56DvFITWgz10mO5wmluDzTHyC2AZKFzoj2FR2p2satBTSAvktneYu6s3EJcWtttGWlZLhyX7sVxTh5WdxqqMNjha2bK1/bEpOtADyIEhAa9RkFSdPMIHU6N9yBeI/QKn+TsQ+eE1o1+AjPgKuKEwxKy+kR1OoojECVeFB2epd4yy1FGVpHAl55Cio3QUUmXObO4N7YHaSEcbQ8nUejYEaQJGxc5OvP3uL68584/NH5DDOCg9LAg4wB9u2FZPDSHuiHuvkrjsscLSfjdduyjS1tl7JAYPti/FvauBE9NyOZRwxkPJ9/g/5zQQLfBpADno4i0FP0kiHd2bZMK0SdeEXnNrLSu0H9cdOejOPJQ20aKDk+pDJX2LTIMYJG3zJIz7xhbIk/K8Kwi59o7cXlMfb6TL92FeIO4/xrALG8xf76sC3f/y8AwolPrKBDZGXDLRTrHj9BY5o3n8whM+TFo9NRjojcpw4D5/4FdCxs7VXQeCCffC47AjqlRY8wUcMTAnNfM+kZ1ZPWF028z/21hYuxpaMKBETQxoiTB0a3F5+KPrgMFFz8g+eDB8K7T7XD1v0DT2oNCVrRnlHhRdHg6iZbZz9eAKouCZLG3DFeYcUcjbAmN/r/PcKFhbl8Fwo0ks5eH//zBwbMf+lgbnn+yZTQB6clw/AjTODtST6Sb5mSfYSULjYkAznKkExPN4otf3QtFO5sez98IH0+DC58HexTFUiCOOh2GTTaz6tZUQEo2dO7V8uPWogk7GfGjyKv4Gf/xyAKnLZUEZ5Tz/IfBhCsFOYMlHzwEuetCt+82CC7/G3QfqoSLIjRq20jR4RltSyS5BaoTG0BXzVoV6jLpC8vnJVXYGHxYSLQmgwuXOv5SuZ19XUs46vZ8hp4TOiw8EgwJa7bCpl3mwqgJwYNxvUgVtkbv098NpO7YUXpCdDPpbpntX7jU4a6ERSEK90QTTYfuw6H/ca0qXOrITh5PgiPHzxmBrsXQO31qm2wZNaT3aME9n8N5fzB/XUJr7Atjj4GjToM73oX7pgslXBSWUZYXRYfHJgRXO7P5d82eqPVpOtDaGGNLstS+s7BbtrvYEDwa27vJwlJj0aZxAA9PuHIB0I4TMDsWKmw0z2W4KZoGi9cZDOhhrjZdNCcvJgzkB08RczwlVEkf3bUYJtlT2W+4mOkppkz6yNYcnOFIZ7wtxVKZA8us/Sx0m12LYPwt0RuzHaMJO30zLqWoag2FFStweYvRNSdpcUPJSBiNXU8I3UkrIITg1N/CuAslS76Aon2QkAZjzoX0bkqsKCJDiRfFEcE6b0XU+qqzHNwVm3No8a2WPn72lFEhfQzU4+hra+xbMdqWRLLQKQ2xzXScLZkbnNlk6k0tOku95WHP1bBLYm7YTc2LPaCq7pE2OguCAHyHvZ04oXOuI4NzHRlN2p/f0snoqopDt/FZz9FzJKAJnU7xI+gUP6KtpxKSxE6CSde29SwURwpKvCg6NFJK3nIdYKa3JGp9Dq3d7hhii8eQkieqdzPbW9LIspIqbPwlticDbObWj00IpsV04/+qdzUpF6BhCoFz7J3I0ByU48Nf7tWKMB1+69C7uoi7ZxuuzzLxrUiJqA9/+AwY2LMdPRnrDvCGiixqR/NVKBQthhIvig7NdM9B3q0NL24OnYWdv8T0IE13kNKg2vSfqraz0tfUqlMsvfyuaiv/ju9H79rU6yfYU3gQjf+69rHbqK+v5ESjGoPPPYUASBcM1OO4L7YHGQ2Sh2VpDnYaNRGVGxDxPpwX76dqQyLUNN//R9MkGSmCMUPakRjofgxs+SF4m/TW9z1RKBStj3LYVXRYvFLyThSS0wHkSw87pKuRcNnsrfQrXOowgH9W7W507Bh7Es/HD+DZ+P48ENuTDOyHqkRL6i0ym31V3F25rVE005mO9ObVSRJgG1dEc8tECs3A7vDxyE029PYUsjr6cgjlmH2M2pdQKH4JKPGi6LBs8lWG9DEJh48Os+BYydq7U7qoPiwXuhCCXnosBdJDAR6/brgGkC/dfO+uzxNyqj2NQXpcRBsfngWpVD3UH+/sDMLbOjFI7FROQno5us1HTEINQ07azHn3f0Vm95aJZIoYZzyc+Yj/DLdCgxNugc79W39eCoWi1VHbRooOSzhZdTsLO4XSv5CoY5fh4oDPRVatM22+hey6AHmGm55abKNjXil5qya4+JHAD55iznOazq92ofFoXG/ecB3gG3chVl1PPfPTcH8eWUiy0KDvMTsZddbaJucKZAHJIjmifluMTn3gsv/B5u9hxwIzUV3WEBg+FWytW79HoVC0HUq8KDos3SzmYBmtJ5Am7Mzyho5WublyE4/F9WGALZ54oVnagUnVmn6N/lu9lzILWVvKDssOHCN0bozpylXOLHb6aljpLedjdwFVAWSXrNFwfx15lI80BP3G7vB7TjTD+bXIKGKP3IOGRm/RmzgtiplvNQ0Gnma+FArFLxIlXhQdlizNyUg9gVW+Cr9LuwAyhJ2H43ozy1PM9xbESw2Su6q28khcb85zZPD3mt1B26dhI7mBnwxAmeHhc2/oStMCyA5Q7TdW6AyyxTPIFs/Zzk68UrOP2Z6SQ+UDbIAX8K5LBE/kImPoyRtJ7FTp91ymyKRK+qg2PNRQhCF8JItkkkTg3DclRgk/+H6givpaAstZTidfJybrk7Ef9rtSKBSKSFDiRdGhmRbbjbsqt1ApfY3sHBpmMrh7Y3sghGC8PYVna/ZaSgQngUerdvJBwhBeRqc4iAXlupguTY595C6w5DIrwVLdpERh487Y7lwf04Vcnwu7EPTSYvFi8K7L4H9a+EXwhM3HqDPWctSp65ueQ5BIFo9U7ccldjLYlo9T1P8OskQWx+rHNhExlUYl033TMfz8jgsp5DPfZ1woLmzzrK8KhaLjoxx2FR2aLpqTp+P7M8meSl3iegGMsyXzZHw/BtbmYXEIjbMsCIU6qjBY5qvguYQBZAj/1oJrndlMdKQ2Ob7MYrK5XloMJ9hSLM8pUdgYbIunnx6HTQhihE63VD2i6r2xt24na/JuhFZf3LGuULFNJvFadTpCbGOkfX8j4QKQJ/P42vs1FbJxJNZC30K/wqWOaqrZaGwMf7IKhUJxGMryoujwZGoO7oztzs0x3SiTXhKETqyfkNqzHOl87C6w3O8Kbzlj7Em8njCIld5ypnsOUiMN+utxTHV2JiFA2O7hfiyBuDu2e7PT558wQhDjgBrLiWUlWu8qRFcPc9296aKV0dt2kHjhIYZYjtcGcX91BQ7hYaDNf/4cicSDh9W+1RxnO+7Q8f3sDzn6BmMDg/RBVierUCgUflHiRXHE4BQaGUEqTGdpTk6wJfOjt9RSf7HCNEwKIRhpT2KkvX6bxJCBN4aShY2DIQRMLBo99NigbSzN0Sm46UKNJ98JZX4x5yuSvMRcuaf2iGCvkcxedzI6cJo9nY048FJJf70ISeCga4lku9zOWDkWXehIKZEWNstcuEK2USgUilAo8aL4RXFHbA6FFW42yuqQbU+wpzT6f7Hh5pWa/Sz1llOODyeCifZULnB0pluDWkUn2VPZ4dofdCk/O4wtrFBMOUnHbhO89KmPkgY7VkLUbwXZYyVibBG20woQ9qYz8wET7Sl85ylCA2JF6DBxAwMXLuKIs+zHYkc57CoUiuajxIviF0Wc0PlXQj9urNjEHhnYCtBbi6FXrWWkrn7Se+78RoLEhWSGp4jZnmL+L64Pg2r9a05zpPGpu4BS6W3iAeLbHIfv20w+LIjlE83DiP6Cmy/SyO7UPPezs07QOO1YweqtktIKyO4E/XKgqkbgsMNBm5tbKwpxIZvMSQNG6AkM1uOZ6ykBoEaGFhkCgYN6S1cGGRQQfFuun9YvzHemUCgUTRFSBrF/dwDKyspITk6mtLSUpKTAIZwKRUPchsEtlZvYI5s6i2QIO0/H9ztUKuCNmv28H6R+ksAs1PhGwmA8GKz0VrDXcPGFu5AC6UHHtIBUvd0V38pk8LMhc8/VGmceb70mUbHh4R1XHlt8VTjRGG1L5DRHOsl+cs7UscVXxeNVu9gn3WjUlyuYYEvh9thuxAidld5y7q/aTqKo4ayYTUHes6CH6MGJthMbzKmYL31fBrzGjp2L9YvRtebXXurQHDwA370Ba340w8T6HAVTboLOOW09M4WiVWnO+q3Ei+IXi5SSHz2lfOjOp1h6SBY2znN04iR7KvZaf5ciw8NVFest5fI9yZbCIm9Zo3DsLOFguB7PntmJLPk8Iej1b/1VJyfTvwWmWvpY7i2n0vDxs7eMhb6mqft14O7YHow/bLurIVJKVvkq2O6rwSEEY2xJZDbINSOl5M7KLWw1qhllz6WPXoS/HSEbNs6yndUkA+8B4wCzfLPw0tjnJ554ztDPiG6yuo7I0h/gpfvwm/3wglvh9KtbfUoKRVuhxIsSL4oW4nNXAS+59kVc6lAD7FLD9cAAKquC+4VMGCV4+LeNLSduw+CJmlx+9JZYElACeCK+HwP0yEVCqeHlweodbPZVcpRtP/1sheii/jeQQgrH244nXQT229lp7GSPYWbY7af1I0PLiHg+RwwFe+CP5xM0bfPt/4Eh41ptSgpFW9Kc9Vv5vCgUQSiRXjSwkOjfPwbgKtVCCheAZRsaL2r7DRe3VWymMowaThL42JXP/XE9w5toA5I1G0/E9WWtr5KF3gwqfC6y9QoG6k4yRRqdRKeQDro9tZ701CKfwxHJZ88Tst7EB0/CQ0q8KBShUOJFoQhCumYPQzr4xzCsReIYDdY1tzS4p3JbWMKljoXeUqSUzcpkK4RgmC2BYbbgW12KMFi/MHSb/dtNBymVhVihCIoSLwpFEE60pfAS+w7VFIoEkewBmwHe4BFFfbrV/zzfU0KhtFbV+nB8+HMJDk5hvmTGl5KffwS3G3r3g9PPFQwdiUrnHy181pIX4nWD3VrRUYXil4oSLwpFEJI0G1c7s3jVFTh7bAo2SoPIG6GDfUwJnoVpQcf6zfn14maRt6lDrlW6CQdaGIJj41rJI/dJPO76GklFB2HxAsnZU+GqG48sAWNIyax9Pmbu9eIxJCPTdc7vZSfO1sLv0eaABgUrg7dTKBTBUOJFoQjBVGdnYoXOm64DlDbInJuEzuXOTMbakrimMnDNHg2YOLWaDdshN89/myknCYb3qw8hdsnIN6umODtbbltdLXnsTxK3qz6hHYBR6+Qz/WPoOwCOnxjxdNoVeyoMLvqhik2lBnVa5b/Swx+X1PDGxDgmZLfgLdFqbIS7BpzNz76sUBzJKPGiUFjgTEc6p9rT2OCrpFL66KI56a7HHDp/saOz31wwGuBE46r4LLIfsPHyZz6+nCeprDHPZ3eCa8/VOPXYxrlPeumxLPeVh+3xMkKP5zR7cAtPQ36cCVWVwdt88aHk+Ikdw/KSX23w4gY328oMusQJLu3rYFia+bt1+STnzahkd4UpIrwNtESFBy75oYq558QzIKWF8tAYFt2+O3YAqELRKijxolBYxFbrxOqPq5xZxAmd9115VDWQHH20WO6IzSGnVujcdKGNmy4MPdYZjjQ+CpIY73CcCM53ZHC5MyusYo8b1ko0jaCVqbdvAY/bwO5oP0Xo1xT5mL3PiyHh6Ayd4zN17lpUwxubG/sJvbDBw8BkQaxNsKPcoCRAAUsD02H6+fVunjquhaweiWlQXRG8jc0BMb/wXDgKhQWUeFEoooAQgoscGZyXl8fBTfMxPDU4O/eh07DTIIICjFmak9/GdOX5mr0ImgbYJqDxW2dXUjQbnTUH3RpYgcLFyoP+1s0waGjEQ0SN/GqDa+dW81OeD02YTsk+CSkOAgqTjaV1uYSD45Xw6U5Py4mXUy6Fdx4P3mbESS0ztkJxhKHEi0IRDarK4MO/4NizjmxNBwQY38GsV+Hce2Dg+LC7PMfRiWzNwUeufFb7zL2dVGFjsj2NK8O0sARiyFGC+TNDL+zffdH24sXlk5z3XRVby0wzUcPQ8kDCJVyqvfDqRjez9nnxGpKjM2xc0c9OdlwUrE7HnQPfvQkH9/k/74yDK+5r/jgKxS8AlWFX0f4pOwjzPoWfv4HyYkhMhZET4aQLYfF3sHsj2Oww7ATzuK2VKxdLCa/fCvs3+zFjCDNnx1VPQLchEQ/hkQZeJDFoUY38qak2uPLc0O0SEuG1T9p22+iDbW5++2NNi46hi3pRJAFNmMdeGh/LeT2j8LkqL4bn/wBbVzY+nt0Lbn8G0jKbP4ZC0UFQGXYVRy65m+Bfv4Wq8vpjVWXw7evmC0Bo5v7Boq8hPRvufK51i9ytnwP7AhUxrM24suBduOSRiIewC42WkGQxsRoxMQY1ITRBRTns3yPJ7tZ2jrsfbvegQbOTBgbDkI03mAxp6tHr51XTO0k75PwbMYmpcPcrsH8HbFoGSOg7ArqpatsKRTi0Hw88heJwvB749+1QFcLJURr1HqfF+fDETeBxtfz8wFzZvns2RBsDti42k4+1QwYOs9Zu984WnUZIilyyRYUL+PeMqTv2wvoofqaye5mWw5MuUsJFoYgAJV4U7ZcVs6G0ECvOlocwfFB0AJbNbLFpNSJ3LVSXWmgoW09Qhckxx1tr52jjpK99kjT0FjD8ZMcJeicG79gn4dtcixlyFQpFi6PEyy8Frwe2rIWNq6A6RGKP9sKWFeaWkFVi7ZDghHgnrJzdcvOqQ0qY+ZK1tjGJEBPfsvOJkOMmiJBuQjGxMNiihaaluLq/A59FHRtTIcjeZiN7q4240sDCJFaH78+MIzM29OfM09JmH4VCYZkWFS+PPfYYY8aMITExkc6dOzNlyhQ2bWrsG1BTU8O0adNIT08nISGBqVOnkpcXIA2pInwMA957AX51HNx8Htx6AVx0DDz7UAcRMRZWqzg7dEuGzgmQFgud4uDgalj8Scsm/Fo9A/YFzqzbiMEnhSfEWpH4BMGZU4LXAjz3IoEzpm0T1R2XqXNJb5vfmk2agDQn2NwwbG4Mx30Wx+CFMQxeFMO4L+MZ+X0MDj+VvV0GvL3Vy+gMPahVRxcwslMLJa9TKBRh06J307lz5zJt2jQWLVrE999/j8fj4dRTT6Wysn7RvPPOO/nyyy/58MMPmTt3Lvv27eOCCy5oyWn9cpASnrgPXv0nlBbVH3fVwBdvwd1Xgrt9bmUA0H90aPERa4dO8ebqBQ1WYAO+fx6WfNoyc5MSFn1ora3Q4KRrgve1ZwusWwR7t7ZJhtXLrhOcdKr5s66DVvsCOO1cmHp5q0+pCUIInjk+lj+PctLJWa804m1w4yAHy6ckcPmyBDrv1RGHSZyUfJ3RM2LRa92OkhzVnNl3Hb8+aiE1toVc2r8Eh+4hNaYKu9Z0e8gnzTEUCkX7oFVDpQsKCujcuTNz587lxBNPpLS0lIyMDN555x0uvNBMO7px40YGDRrEwoULOfbYY0P2qUKlg7B2Kdx5SeDzQsDNf4EpV7XenMLB64X7zq71ewlAlySwaYHNBvZYuON9cEQ58VhNJfxrirW24y6BSdc3Pe5xwawPYPYHUNSg8GO3/nDxXTDw6LCnJaWkwrWLgsrlVHvy0YWT1LhBpMcfhU0L/TvYvUMy9wdJSRGkdYKTJgu6dm/70gCF3lx2u9bgw0uqLZscfQSbyzS8BgxI0YizCRbMljz1f4FvZxLJ1lFuBpy1hiuGL0ETEkMKBBJdMzWjEOAzBOVuB3bNoMZr5+e9PUkTQ3lwZEbbFKisKoK9K8HnhrSekDEguJlMoeggdJhQ6dJS07ExLc2svbJs2TI8Hg+nnHLKoTYDBw6ke/fuAcWLy+XC5aq3FpSVRV5994jnmw/Mx2hfkJoqX73bfsWLzQa3/wcevRJ8fpwl7br5Coan2oz0GTwhypOzqPkTM2DC1Y2PGQZ88gzMfMf/+9q7BZ6aBrf9GwaPtT4jKckt+ZbCyhXQIKi4qnQ/eeWL6JdxObH2jKB9dO8luPKG9rMw1hiV/FT5AS5Zb60t9O1mC4sZGn8SOc763DlzZkiEZgZ3BaL3LsmvRyw+9H9N1P8d6/SArkmSnS6EgHiHhzP6bsAmNlPiO49UW3az35OUEi9uNDR0EcTZyOeGRf+FbXMav6nkbnDibaaQUSh+obSaeDEMgzvuuIPjjz+eoUPNVJ0HDhzA4XCQkpLSqG1mZiYHDhzw289jjz3GQw891NLTPTLYuzO4cJESDuS22nQiols/uOt5+McNTc9ZDT2pLI7unACc8ZDeHQ7mElTITLwW9AYLlM8LD18G+7cHvkZKwIB3H4eHP7b8lF1YuaJWuMDh2VC8RjXbCj9gSNZNiHbqe3M4hmEwv+IdvPjb2pSsdc3GqSXQ2d4DgOLi4MJFIIhxmblbtBC/0oa/ciEkPrwsqvqYDL0HPZ0jsAkHue61lBtF2HCQbe9LF3v/oGLEJ33sdK9kl3v1ITFmJ4ZkvTO9naNI0NKpNIoRCJK0DPS5T0PuUpp8vsr2wbcPwNmPQ1JW8DeiUByhtJp4mTZtGmvXruXHH39sVj/33Xcfd91116H/l5WVkZPTignJOgquGkhMImTVvYTk1ptTpPQbCUOOg/ULI/MHsbdAjK8QcOyF8NUTAc5rEJcEg05sfPzJm4MLlzqkhLzdsGMt9A4d5iOlJL/852AtcPtKKa3ZSkps/9DjtwCGIVm1DDatkwgBQ0cIBg8n4FbMbs+aAMKlng018+hsvxKAjM6QuyPwx10iSexUHlK4BKPAt4uCql2AKYZkrbA46Mtlm2spx8SfT5zW1PxtSB/Lqr7koG9Po+Meaij07aawanej4+lF1RyTuyTAGzHA64J1X8C430T+ZhSKDkyrPILdcsstTJ8+ndmzZ9OtW7dDx7OysnC73ZSUlDRqn5eXR1aW/ycKp9NJUlJSo1eHxTBMv5R538C6ZcFFhhWkhO8/gRvPgrOHwKLZwfvUNDjl/OaN2Vrc8Gj9Iu6wQWYCZCZau3bpFy3jBHvU6TDqbPNnrcFXSWimj83Fj5hVguv44R3YvDy8MQ7uD90G8BqVuHz1FqatBZks292L4qr6CsUCjXLXrvDGjxI7txnc9mvJ/90v+ew9+PRdePD3kt/fKMk/4P9vs9uzLmS/VbIUo/YzfvIZIujHXQADJq+PZPp+kYdZRGpkBcuqvsSfG+Eu95omwiUYnXP3YgSzuEkDts0LbmpSKI5gWtTyIqXk1ltv5dNPP2XOnDn06tWr0fnRo0djt9uZOXMmU6dOBWDTpk3s3r2bcePGteTU2p4FM+C5v0J+gyJt2Tkw7QEYOzH8/qSE/zwAX75tbZtB003LTHv1dzmcuET4wyuw7Dv4/t8gw0gYlrcV9qyDnChXFhQCTr8NBhwPy76EvG1gjzH9a0aeBQlp9W3LS+Cjp8IfIzEtdJsGfLjiWL5edzRuX932hSQrsZg7Jk6ne2oxYSX8iwCXS7JlA3g80L2XZNM6wWfvS3ZsqW/TcCdzzy5TxPzrZYiNbfy59UlrGYm9uHEQw6ixMOJoWLWsqVbVNOjeGwZM3OK/kyggkVQYxez1bMQr3fjwkqR1opMth13u1WH15XB5Qgtunxt8HrC1cfZAhaINaFHxMm3aNN555x0+//xzEhMTD/mxJCcnExsbS3JyMtdddx133XUXaWlpJCUlceuttzJu3DhLkUYdlrlfwyO3Nj2+Pxf+fAM8/BIcOym8PpfMNYULBL7pCWFaBQwfdOsFf3kW0juHN05bomlQsAmEDG8NFhpsXxZ98QLm77T30eYrGAu/DN+yltwJ+o2w1NSmxfPqwjOZtXkAjX85ggPlqdz/5RU8ds7/6JnWPbw5WMTnk3zyDnz5kaS6quGZ4H8ow4CCPHj4D5JhIyWVVVBWDEkpkHpcDnH9NoTU4jZM65amCf7wELz1suSHr8FTq310XXLcRMl1t+hs1/qy17OhidUkmqypMbM7120rOYnHRXg5lWrinKapKNg0HfGgq/BtxS+TFhUvzz//PAAnnXRSo+OvvfYav/71rwF48skn0TSNqVOn4nK5OO2003juuedaclpti8cDf7sz8Hkp4dmH4ZiTGm9FhOLzt0xrihHEQbdnfzj5PBg4AoYf0/HCLX1eWP198PfoF9n25vVt4T15A3DBraBb+4ruK5XM2tyfQ4UgGyEwJPxn7nmcM7Rlqha/9JRk1reRX791k/mqQ9fB9+VEcsb0ZOJdM7A5/P/NU7VstAbfE4dDcO00wZSrKpi/Zgv57j2k98kjNrmGjfSkp20ERd69VMuyFhUwUL+tFK5wAdjbPZPeW4JsMwkN+p3c8b7DCkWUaPFto1DExMTw7LPP8uyzIYrbHSncf62ZvyQYB3JhwwoYMjp4ux2b4NPXTavLwYLQC/S+3XDJjWFNt13hqjRN5eEiJXQbHP35WGXZTFg5x3r7mHgzz8u4syxf8sRsD01FS0MEe0pTKK4WpMUFaRYB2zY3T7j4o25rac/Snix8eTzjp81p0kagMTxucpPj1UY5y+R7OIa66NbgeL5vJ/nVO8mxD0HDxh7Penx4ojvxKFGRFM/OPl3puW1v05NCg7g0GHpu609MoWgndIyYySOFvTth5U/W2haGKJEw+0v47dkw42OzrRXLgqsadrXcnn+L44yzbIlohGYLva3TUuzdCi/fb91h+Owb4F8z4ITzwhpmx0Er/Qu25kfX2uDzST74n2wxA4CUgq1zBlFV3FhxxYlkxsdf6jeyZ0XVN0GjlHI966jwFTEp4RrS9C5NsvG2FzYM683GIb1w2xt+5gWlmaN5NPsvXPGTjdt+qmbefq+lB0WF4kiiVZPU/eL59iPrbdOCJBPbnwuP/y6y6KSFM6FHv/Cvaw/odhgyCdbODG/rKCW7Ptd9eTHM/Qjmf2o60TpioOdgmHo75LTA72Xme9bbXvdXGHtGRMMkWPTZTI+P3kK95CfJS09JSlogjU5DpCFI23gVAyfuwCvdpNu7Eaf5D/Gv8VVQauSH7POgkctuz1qKfPtpaSfmhmjoGFj87ArBjv457OzblZSiCuK8PXhzz0m8mpuITYBXerEJeGuLh16JgjNy7AxM0ZjS006CvX0KMoUiWijLS2tyYHfoNmBu+AfbMpr+TuT323efh+Ig6fbbOydcbkb0hJNorXNP89+83fDQJfDFi1CcD143VJXB+kXw10vhvX9GP6R69XxrQuvXD0YsXACuPib0c0iiE/pkROcrv3KJ5B8PtrxwqWPWtxr5a/rS1T44oHAB2O+1blnc6lpCawoXG05GxZ4Z9nVS0ziYnsRjeX15NddMD+CtnXbdvzvKJc+vd3PbTzUM/KCcj3e0z+0whSJaKPHSqlh8Ghp/pn9nXSnN1+rFETit1lJdWR+V1BFJ7QJXPQmde1u/5qjTzN/bC3eboSyBmPUe/BDl342Vv1O3/nDc2c0a5uQBtpBpb6aNj46hVUrJmy+37jbF5vXwyL2SP94uyc8z2LdHUlrcdA5erIfQt66/iyDHMYTdnjURXW1IjQV7egVtU/fbqPLCb+ZVM2tvGOkEFIoOhhIvrUVZMSydH7qdboNpf2587OfZcPvFcFo/87V7a+TzkNL0k+nIdO4F1z8P1z4Hp98O8Wn+LTFCg54joc8Y2LrK9D8J9aT99euhHarDofew+i0rf2g69B8VlaE+u95JVgABc82xNq4eG6SOThjs3Q27d7Ru8eu6HdKtm2DalXD7NZLrL5b85S6DdavqJ5Kqtb90+QJBnEikp+Mo8r07w7q27nf87tpRVLhjwhgT/rayHVeMVyiaifJ5aS2+/gAqSkO3e/AFOJgPr/zdFDtlJWb14YZUVTRvLmUlzbu+vZDdz3wNOB6++hdsbZAeX9Nh+Glw6s2miNm2ygwrDbXiVpbA7g2WUvJbYtIl5tZRIKQBEy6MylApcRqzb4tl/jYvb/zso8It6Z+hccdEG2lx0XtOKbPwMfbHoGGmBUXKZiSTPuzPt3EdPPQHye8fgGOOF6TbumLDgZcIotJaiCStM0fFTuaAJ/yHjsKqOD7aMJIfc/uEdZ0BLC30sb/KIDuKf3uFor2gxEtrMfuL0G269jSdSP/yG7NyXLCiihEjILNrC/TbhiSkwiWPQMkB2LfJ3HLLGQbxKfVtglk/DudwsdgcBh8LZ1wD37zWuM5UXU6ey++D7J7RGw8Y38fG+D4t99VOD16YugmaBseOhzv/pLFjq+Td/0pWBCjbEy51QXbP/kMyYgw4HBrDYiexojrKsdvNoNTI48fKd6076gIgeGnZsczb3RfZjGioco+k+XWwFYr2hxIvrUWlBWuJ2wX/vNu8I0eiWzTdvDaUdeGsX0XQeS011TB7OpQVQb+hMGJceMn0WpKULPPlj0FjrO1zCAHZwX0Lwub8adDnKJj5jpmsTmgw5Fg45TLoOyK6Y0WZ8jLJ7O9gxWKJzwf9B8HkswWDhsGmdaEtKJoGyalw1Y3mAtyrr+D+/xOUlki++0Ly4ZvRmWdVJSz+EU6YBFn2vozkdNZVz8VNdXQGaCbhCBeBzoH8c5m722LtrgA4NJTVRXHEosRLa9GjLxTsD+zAqem19vQwHQmcsWbfPp+ZNffsy+CtZ2DOl/4LvPQdAmdcEv78fV742+9g7leN+41PhN89DuNPC7/P1iRnAHTtW+v3EoTRp0BSevTHH36C+epAbFpvFlKsrqr/k29aB198JLnwcti2yXQPCiRg7A4YPwmGj4KtG6GqUpLT0xQxySmCCZOjJ150HfbtMbMLl5dJtiztg1Hdm/Tu+cT12USlLEKTOom2TmTaevNz1cctnmE3MgTHxJ3Lb3emENkTjIku4KLedhIPD5mWEgwPCFv7eehQKCJAiZfW4uzLYPGcwOcNHxQeCL9fZww8+1njY/f8A7r1hE9eg8py85jdAZMvgN/ca14TDlLCXb+C9Suanqssh4dvhgeehxNODX/+rYWUZmh0KC68reXn0gGoKK8VLtWNtWqdUPnoLbjp9/D9dNiyof58ZjZceCUMHg4L58Kn71GbfdfspO9AyW/vEvToJcjM1sjqYnCgQW3SSDEMcDrhtecMvvvS1Nomncnp0Zlb7xX06lu/kPd1jmWLa1HzB446knU1c6jyNC97bpId/jiyQfKfigJY8DwcWMchx6GkrjD2GugyvFljKRRtgZLercXYiTCxeeGwfuk7pOkx3QZX3Q4f/AxPfwhPvGf+fOejpqUkHHxemP6uf+HSkKf+2AwvzFZgzxYzz0sodm8K3eYXwJwZmBaXAH9SocGqpfB//9Z45GmYeBp0yTEf5lcsho/ehLdertfOdWzfDH+6XbJ3t7mA3v9/kSVNPhwpzUikbz5rKFxM9ubCX+6StZYZkz6O0XS1DwxrDIFAIwzfqQipMIrJTt7drLy/VT6ItdX2ULoPPrkNDqylkcdz2V74/hHYMqcZIykUbYOyvLQWmgb3PGGKjY9fg6LaLKA2O3ibkW9i2gOBzzmcMDjCMFzDgOcfgW8/MP1cQlFaZK0eU1tRUmCxXQdO4BdFViyRQV2EDJ/pY7LsZ4NnHjdduura5+0PrGMNw/w43Xm9xOGUjDga7vgjfP0JbDhsbQ2HMcfBogBBXYYBbjd8/Lbk1nvMBV0IQbqew17PRstjSCRd7P3Z69kQunEz8BmCzORdSHIi7sPtg/e2ublxkBO+exBkkC2ohS9Ar3Fgs5imWaFoByjx0hLs3AIv/h9srk1INWAY/PaP0L0vXPwbmHod5O2B3B3wp+siH2fYGOgeRrI2q1SWw3WnwcEQ9ZUO52DotOxtRkona+2SLbZr7+zbBj9/C5Wl0KkLHHu29d8BTa0X/vB44G9/anrcigFOSnDVwOIF8POP8Ns7BeddAn/7U/jq5ejjILtr8KLqhg8WzIHf3imxO2r9bvTOlscQCJL1TIY4J1BjVHDQl2vpqkjUmCYkDr15CfR0AeuKDdj+I1SXBG8sDVj/NQw/v1ljKhStido2ijZvPws3nA5L55mJ6cqKYck8Uwy8+7zZRtehS4/wxUFDBo+Cf74TnTkfzr1XRza3YPWY2ppu/SG7N4cqCAogKQYyE6BzAsTYIC4Rhh7XptNsNh43vPInePAS+O5/sOAL+PQ5uPdMM1zbIgOGtI4/Z5215sWnJIlJEj2CXZnrbxWUl4fOX+3zQlVV/f8T9FTS9W6WCjMmaZ0ZHXsWumbj6LizGRoziTjRuEyBQCdJyyBR60Sy1pnejlGMiTsPG46AYzhcbuIqqtG99arLkIK95Skh5xSKGA1YbvEekbe+2eMpFK2JsrxEk1U/w+tPBD7/33/C0NEw7Bjz/+HkHgHI6WPmaLnmLugfpSRqh7NtA2xcFf51CUmRb1G1BkLAxXfBv2+DBDukxdGoFHKs3aw+XXEQUjtwZoz3/g5LvjN/bmiGkMCnz0JiKpwwJWQ3p5wl+Oz91ovGkRJ+mmOGOs/7wXpUe7fukJYO6Z1CX+NwQHxC42PDY09hYeXH1MgKDreS6NjJsvWlq2MAaXpXRO3nRRM61UY5VbJxtj6JjzKjgFiRxLj4C3FqZiXscfEXssm1kHzvjkNtu+48QP/1O4hxeWqvhaJOyaw8eiA1MU5m72xekVCvhAvSC2GTxW1QPTrZlxWK1kJZXqLJ038O3eaFx+p/Hjmu8QIajAuvh//OgMdeaznhArBgRmTXTXug/YdeDjkWLroJ0uL9/94NLzx/DbhrWn9u0aA4H378Ivgq/uXLlvZ1MjoLbrlbIER4NTCbw/KfTd8Uq2UHpIQzLzD9VyZMFkHflqbBSaeBzdb47x6jJXB8wiX0cx5DrEhEw0acSKa/cxwTE3/NoNjjKfcdZF3NHDbUzKfIu49drjVscwfOslcty9hU89Oh/yfoaYyOO4tJCddxfPwlnLwlgWErNuN01W8NCSC9sJQJ3y/h6+UDKKxK8NOzNXQBw1I1jkkosX5R/8kRj6dQtAXK8hItfvgMcreFbretgXk2syuccJopGPzdeX0GVLghqxfkFsCs72DCKTSxrXu9Zg6Z+ERISmnOu4CaqtBtDmfaA3DKlOaN21qs+y74/oL0wfR/wgV+nDnaO6vnBQ4PqqM4D/Zshu6hI23GTxJ0zYEP/idZ1gpRxQcLYP/e8K55/w0YPkqS1UVw7sWSLz5o2kbTICERLrg0wNaNiKGvcwx9nWMaHd/r3sTamlmNEsztdFuzSu7zbiazYDybVzswDOg3CLp0i8VZVQNrv8bMSNMU3Wdw1e6ZvMtYS+M0RMMsCzAwReODyXForlRrF9pjoeuIsMdTKNoSJV6iwf7d8I+7rbU9/LHyd38z87tsWGneZb0+U7CU1YC7diEq2QSbNsO7r0Pf/vDaR9ClGxQXwiO3wpol9f0mp5lh0udeEdl76R6Bufrk8yIbq7XxecxtoVBs/il0m/aIq8Y0k4QSMC7rWWd79xPceg9cd6FsmWoVDXBHUI6opAj+8aDkHy8IrrhekJQEn74vG4VoDx0Jv7ldkJ5hPfi40JvL6prvw58Q4Km2s+DFE/nvjzZkg+/7sJGSeya/iZPA+lkAx7CdRKopJ9bymH2TBMdl2jiru42Tu9rQhICYrpDaA4p3E9Rx+KzHrFuAFYp2ghIv0eDLd7AcVdAps/H/4xPhyfdh4Ux471X4fk7TqsYNV40d2+Cq8+HdL+CayU0XotIi+M8DsHcX3PTHcN8JTDgTnn3IugVG08NPetdWeCxuB/maF+nRZmT3Ci1chAaZ3cPqNj5BcNxJkgWz22cqn53bYOsmSd8BZsTSmefDpvXgcpk+MZnZtQuzlOCpNhdqe3BhsM0VWfElwyeY8chZ5G/OQsrGgmDdKijsvosuicGNfwLoTR6r6BlyvBgNxmfrdIvXOD7LxoTsWuEC5vsccxV8/2jt7cnPPWrM1ZDcxeK7UyjaD0q8NJf9u+GzN6zf1S+9qekx3QaDRsMPVzcVLofj85kC5pZLgz9Bf/JfmHoNdA7zxhQbB/f8Ex6eFtr5QNfhuMlmPpmOgNOiH0G4jtTthSHHQmpnM1eNPxGj6TBiQtjlD3Ztl/QfZPqkVFVa90lpTTashr4DzJ/tDsHQEQ1OluXBolcgb53p1wSQ1AWOuhB6Hd/E6uCRLop8kaX93b20J3kb/X/nDANq3NZuuaXEWWpXY8D3e33o+Hhts4esWMGHp8QxJK32M5w9DE6+Fxa+BJUNnHcd8TDyUhjYjrNiKxRBUOKlOZSVwG/OMsNTrTDyODjrUv/nrr/Eej+6Dhs3Quf44O1eegz+9B9rfTbkhNPMrLxv/huWL/DfRmjm67Kbw++/rRAC0rpB0Z7g7fqNa/5YNZWwvzZbb3Z/iIncAROAyoOw6GXYt9pcgDU75BwNY6+D2CSzjabDdY/AU9NM54eGglrTzUiji++yPGTuTsmz/5Bs29ygG619ipcmpgxPDWyZBeunN1606yjbB/P/DSW7YdRljU75ZOMHCHeVnW3z+nNwewaazSBn9C66jtiNpjf9RWydPQChGUjDv5fz3O3j6ZO+w++5OqQths/P680H2z08usJahfM622xBjeS8GVX8PCWe9JjaOXQdAVOfgbwNZpkAZ6JZEkBFGCk6MEq8NIdnHrS+vdJnEDz+P/97y7t2wOoQ6fcb4vNBgJtjI376wbQMZYe3TQDA0KPN+R7Mh3/dC0vmmiuX0MyEGUkpcP9T/ssTtGcuehhevDbweaHBZD/WsVBUl0NZvmlFW/I5rPym/ilfaDD0ZDj9VnBY92M4RNEOmH5fY2uK4YFdC2HPUjjvSUisTbjWfxTc+xp89SqsnGteY3fCcWfDWddDirVcPAf2Sf50h2ySXLlODw0cBh4XjYRNW9LI0lJTBt8+AKUWvH/XfGaKwIz+hw45RCw2nHhxsXtpD+Y8cSpelw2hm29+43dDSe5WxGl/mk5CRuNq8VXF8QGFC8D3W07l8pHv4bQFflARQ84mJ0GjZ0L4YV4+CSUuydtbPdw2tIFFVGiQ1cG+qwpFEISU7fI5yjJlZWUkJydTWlpKUlJS6w5+1mBwW3gyEho880ngEOcXnoJ/PGx9XCEg2QlpFhbC9Ex46StIshh5EIjc7fDT9+b77dkfxp1sljboiOxaDe/e0zSNrM0JV/4Lugyw1o/XDcu/gmVfQNFeQvo9pXWF37wSfjGfd68Bd2Xg8/EZcOGzTY+7qqG6AuKTzcKcYfDM3w3mzwy+G5qYZJYFaGs/mMxseOZ/DRb6WX+HPctD+//U0edEOOGWRoc21SxkycZdfHHvVFOMHOa/IjSDxMwyzn/yPXS7OY6GjcX/upS1ixKD/k7GDNzJ3cf82RSgh9PzOJhwBwDlHsnA98upjsBRemS6xsyzm2ntUyhamOas38ryEilVFdaEC8Af/h48N0t5mbUokTqkhNgYMyrJqYM9iI9GUQF8/T6MmQRvvmKGW/t8MHosXHUDHH2stTFzesMlN1pr297pMRzu/QbWzYZ1s8zf/ZBJMGi89aQmu9fAh3+BmorQbeso2gvz3oCJYZSE2LsyuHABqCwwI0pSD7OwOWPNV5i43ZIfLTjnlpeF3XVEpHeCgwFyrdns8Ke/NThQUQC5S8MboGhnk0N9nKN5/fMMU7TIptZSaWiU7U/Bs/xUBp5YTLyWQmdbbzLO1Fj9U2ARKwQMmdgLznkV1k+nZvM8ql0uNsss/q5PZWtBL7K/ruCUrnau7G/n90c5+etyi/eZBlRaKO+gUHRklHiJFKsGq7QMmByiZkivPtaFSx37S+p/jrNBRjzofhZeacA7r8IDfzHnXBe59O0X8NWn8Ie/wG/vCG/sI4UhE81XuBTtgXfvA2/4iwrLp4cnXnZZTLCyc2FT8RIhVZXWahu1Fnc+XM2imbF8+0XjGqb9B8Pv/wKp6Q0+9wVbwh/A1jRaziYc7Pq5F9IIHBckNMmuJb05f3L9+CkDDYxePsQOrUlJAE0zK29POgOwx/Af/RweqDoVTYAhMf2UPJLcSsmSAhf/WuPijQkx3D/CyeOrXPgs3nJ0AUNT23nCSIWimSjxEilxCeaWTKgaQBPOCt3XmVPgoXvNVSMSqrywrxy6JoF22M3Wa8D6bYBoLLjqRMw/Hobho+C4EyMb+5fI4k8iD6euqTC3m2wWt3GsWoKiGCEVHw82W+jAtzri4s22Vg2RlhEGnXoXkpv5JRfccAEXX5XG1o3mR7d3f0hK9iMsIkkH3KOp9VFKiccbPPeJNESj97y/yuDMbyspHifp5XDQdYsdrVb8SCSOQQYPP2wjNlYwd7+XB5aZFxt+RIkEXD64cnYNi89PYGxnnfNmWPOv80m4dkB424QKRUdDyfNIEQKuuCV4G5sdrvld6L7i4uHRp5qXKMpjQJmf1aPcZd4JA1mKdB0evQ9uPAsuOwHuuQrmfUOLZyTryKyfG76lrCHeMIRPn5OstesdPfFpdwhOmGRdB1RVwl+fhN/9RXDjndGahYGmScZe+yMe3Cyv/oaYWBg2SjBijPAvXAAyB4YnYOxx0PekJoeFEHTpRtCELJoGOT3r///0WjcHXRKPBpvHuJk/tZKVE6tZNaGaHy+o4quR1eT6DKSUPLq8xkI5SPBIeGqNi/HZNvolWbs/nN5VY1xmBw33VygsosRLczjzV3DqVP/nbHb4+//MvCmBkBK2bISli+CYcfDfD+Co0f7bOmNCi5tyP+Kl2kLemM0bYftGs8TAyoXw11vgoZvCW2R/SVhNducPTYfvn4eti60JoBirTmzR9bu/8ApBnLVUIwgBq5cJjh0vOOVMjZ59mp+wNblrCac/9DmZAw8AkiqjhCKfheih2BTobdF3yR4HZzwcMP/PmVOC15uWEk4+w2xhSMnbW9yNtna8TjjY1Udhjg93nMQm4J+rXBz9aQVLCw3Lf7F3t3mQUjKlpzUH+RsGOQ4VkVQojlTUtlFz0DT4/eNmXpTP/gdb14HTCSedA1Ouapogbusm+PBt2JcLlZXm//fmmueEgPGT4B/PmpaYinLI2wvvvww7N8H2faH9bPxtisfFQ01p0+OBqPPSXDQL3nnOLDWgaExaN8jfQUSCwfDB2h9g9XeQ1Q9+9X8QnxK4fdF2a/0e3AaJmaHbWSQzW/DXp+DhuyUlRcHbahpUV9dX6zn3IsG//xaZmBLCIKN/Hmc9+mkTAZTv2Um6rVvoTsZeB+X5kL/BjyO8gITO0O8kGHJe0Mivk8+EpQth1bLGXz1NM78m19ws6JxlTvJgjQzpJGtI+HxX+M5EbgPm7veR7LAmSBz+fN8UiiMMJV6aixBm2PC4kwO3kRIe+zO8+py5TWMYTYWIlLBgDlwwGT75HlYtgBf/r/68lfuRftjNTQjQDfPfYMIn1s/HQEozc/CvfttxMui2FkefB18/Gfn1Ru2WXN42+PABuPqpwKYKYdH8L6L/Vc7pIfjni5LrLyaoTvP5oPggrFgiGT4KTpgE27fA9I/DG09oBvYYD+Onzfb76yj07bbUT77Xwcc595DsWMaxZfPpJopxJHaCvhMhZ3RI/yC3T1Ltg0Q73PNXwdefwtefSg4WmOf7D4bzfyUYNdac5O4Kg3O/C+2vFulGowBm7/NyQlboz4IAeiUq8aI48lHipTX473OmcIHgviQ+H1RXwR9ugvKdjc8lOqAqxDZO4mEiQ0qI02sfiIMImASHee7wFaO8FPbsgN6hKxD/ohh+KmycDzuWBf6dxqdBZQiThTRg73rz1S1AArHMweZiawT53Gg2yBpsbe5hkpyiMeEUg3k/BNe/s7+D2d9JUlLhhtsFV90oyNsvWWK5xqWk+5gdHH3FIpK7+LcUVhhFuI1qHJr/8G8pJU+ucfPYSheGBE0chSGPAuCmrg4e7u7EY8BHW9y8ucXD3iqD7FiNy/rZOa+HjQ+3e/hgm4eVRQaGhGQ7XD3Awe3nOjnnQkFFubkbHBsrGo156cwq9la2XLosTYBXwqQuNrJiBXnV0q+W1AWc3EWnS3yUxIvXA5uXm7mCsnpA177R6VehiAJKvLQ0Hg88/5T19j4frFoB3ZPB1uAmFGeHGBvUBDA72zVI8mMh0YVZRqCwJnD4SF6leYdMdkJyTOOIJbV33hTdBhc/DAs/gKWfQ2WxeTwuGXqNMkOhkzOhtACeuSx4X5oOWxYFFi8xidB3Emz+Af/mDwH9Jlmv2xQMaZj9HfY3v+IGwbpVkqLC0LlfSool/3xI8ruHPZxyppMlQXKe1CEEDD99O6Ov/y5kW5eswuOJ4aMdHhbn+9A1mJBl4+weNt7c4uGRBun0G+6iPrfejSElP+f7WHHQQMO0hOyr9LG00MfvFza1jJR64Nl1bqbv8vDtmfF0SmoqCuYd8LGhpGWz9PkkjMnQ0TXB8+NjufiHKgzZ+P3pAlIcgr+NjSCD8+FICbPfhy9fhsoGQrLHYLjyfuiuHmYUbY8SLy3N2pWmTT1cvEZj8SIEZCXAwSooPyy1eJwdMuKahkk3PH90L9i0FSrd4DOa+scYEoproNQFqTGmFSclDbr1Cn/uvwR0O5xwORz3K6goMq0f8SmHLfxWFjVhhk67K81Ec0KD9F6gNwh1PeZqsz7P3hX1Phx1/3YdYVYOjhQpzVwy66ZD4VbTSpc5BIacA91GApCSKnjsGfjgf5I5M0KV4BJIJC+/WME1T31FdvezydtjDyh6hDB92s+40EW+hemuKrBz5exySt3mgi2At7Z4yFxCyEy0L2zwHPqK1E2n7lsQ6C/lk7CrQvLg0hqeOaGpB/PcfV5stZaRlkAT0MkpOKu7eauekG3j69Pj+dvKGmbt8yExn1vO72njjyNjyImgpEATvn0dPvWTsXn3Rvj79XDf642tMF4v7FwLrhrI6gnpWc2fg0IRAiVeWhpXhMkvDvdfAfNOlhFvlgWo8Zp33hhbY5ETCOmrFSUO2B3EgdeQcLAaiqrhpClhp5VvFVbPhy9egML95vxGTIDzb4G4xNafi6ZDUoB6QQnppuWkpjzw9YYXynbD+zfU10Kyx8HgM2H4VLN/3WFWBj6wFrbOgaoiiEszQ3yzhkZuHZMSlv4P1n9V24c0P1N5682xRl0Gw6YApoD5ze2Cq38rmfWN5L9+1rb6fgWle9LYsdPD+Pvf5fuHplK8P97sH9HgX0jrBHc/JOjarRezKuYS2LlGEE9XLvkBXLVKo6H+LqixJhX95VQJhU/Ce9u9jM9yc0FvO/YGDwlWE8fVvetw2ukCYnV4a1JsozFHZ+h8ODmeohqDEjdkxAoS7VGykJYXw+cv+D8nDXDXwOsPwX1vmJ+Z2e+bdbTKi+vfwdDj4LK7oVPX6MxJofCDEi8tzZowCi6CeUOwa8FT/usaxIcpKvoMhqJ8U5RYQQLvvgdX3wbde4Y3VkshJTx9K6w/LOvs3I9h/mfw+5eh7/A2mZpfdBuMOhsWvh8gLFqYoSvl20E0WNo8VbDqIyg7AONvNT8TQkD2MPMVLfatMoULNHZoqZvr8negy1GmJagWp9NcJIWQIYPfakpjSe9VyLlPvsmun3uzc1FvqoriQUi69XZz8rE5jD7Ghq4LIJ6etqPY6V3pty+B4Kddo3Ab/gVIS5dXMiTctKCGB5a5eHVCLMdnmbfOMRk6/7GgSqxqpkEpGrsrDRJsggt62blxkCOgNSUtRiOtaXLg5rFkRugQ/l0b4I6J0HMIbFx82Elpfj8fuwb+9CakRi8CTqFoiBIvLcmLT8PfHwrvGikDF1w0pLntU1679WPTzO2deHvwp+/EZLjwWlg0EzxhJJ/zeuCtV+D+R8J7Dy3Fh082FS51GD544kb497z2VTDy+Eth5wrYv6mxQKjLQ5IW21i4NGTHj9D/5JarBrzx2+A1tYQGm2bAcY1rWqVnWKuOEd/JrPuk2w16n7CV3idsbXS+FAcVnE8yGayvns9u7xq//cSKJIbFnswfNibha+M6soUuyYXfVzHz7HgGp+qcnmOjS5xgX1XoecXo4Pb5F1oCODFb59NT46M+57AoKTCtfaHqQ9RU+hEutRg+01fmq1fhivujP0eFApWkruWYPSN84aLVOtfG+Vl8vQbsLYOCKnPLyGOYCejyK81tIHeQm81v7oPhY+Hk88LbYvD54Nsvw3sPLYWUpoUlGF4PfPe/1pmPVRyxcMU/4KRr67eXhIBYu7kFGBNEaAkNtsxqubkVbg/+lC0NKGxaK2jkGEgIskMnhEGnPnmkdCsO3Ajw4mZJ5ecsrZzOLs8qZIBl/ei4c0i3daWmHSR9NqTp3/L0WnM72KYJ3p4UhzNEFLMu4OLedvoma00S3wkgJ0Hw3AlRcLZtLrHxwSPbrGL4YOFXoRykFIqIUeKlpfjPP8Jrn5wMvVLNsGV/7C83BYs/fBL2lpu5xA9Re4s853I47UIzTGTqtTAhSD4af7iakU02muTvAY8F/6FlM1t+LuFijzEde294EQYMhK6pkBYHjhCGT2lAeYjaWc0hSIK2+jZNP492h+DaW+qW4MMsDsJAaJKx1y6wNAUPNRT4dgZpIVlVPQOAoWmaX1ewQ0NbGrH5+CR8ttOL12d+H49K15l5VjzOAHdTAdgE3DrUwfdnxfPQ0U76Jmkk2KBXouBPo5zMOTuB7Lg2vB0bBkx/2YwwipZ1y+OCqlYqPa74xaG2jVqCg4VmWk6rCAHDhkDe5qZ5YKSEvIrAwuVQO0BLhvQYcLuh/1A47yoYOxG+eR/eesZM/y8lOHSzv1A3KV2HYSOtv4+WxGoFZ287ftJb/TFUFWDZA0IIiEluufl0P8bcOgpofRHQfYzfM+MnCex2+N+LUNBAX6X1OMi4G+bXpvWPDmVGAV7Dw/UDHczdH9hnSwKZsZBn0a2rOXgMSPrhIKdmOPhdz1jGpzn47sx4LvqhioIaiV6bVkliFn1/c2IcfZJM88wtQ5zcMqSdJX787Fn49o3o9qnpEBuFEH6Fwg9KvLQENWHePQcOgZGj4NumJnpKXWbVaCvs2gXvroPM7Ppjb/4H/vdU/f/rQq4PVJgb8MHw+eCK66yN3dJk9sBSzIbTYkGe1sbnNbeADB8UV8OBMqjxmI7ZnROgU4LpiN0QKaHPeDMKKXcpFO0yq1F3Gw2p3Zs/p4Gnw+bva0NmDregaGCLMXPMBODY8YK+Y/czY81SyksFCZ0qSOsZQVoAC1QbZZyZk8Zlfey8s83T6JOgCXM75+6jHFw/0MG9P9fw2U7voU2oZAeM62zj2z3hp+YPhKFJPMB3B918XejmhcEJXNstlpVTE/hkh4d5B3z4DMnYzjYu6WMnyWJq/zahpBBmvBndPjUdRp8Mjmh7FCsUJkq8tAQZmaZTQEWQENk64mxQvQd+mtHUSU5KKA1z22bP7nrxcmBPY+FSh02Drom1MacJsG9f4xICdcVbrrweJpwS3vgthc0OCSlQEdyPgmoLv/O2oKYM3NWwKQ+KquqPV3ugrAZyS2B4l/qtJKFBWk+wx8OHN5n1qYQOSFj+LnQdCSfeDo5miLWkLJh0D8z+x2EWK2mGa59yP8QGtvyU+PJYWvMZqf0MUiOfhSUcWixCCP59fAzHdNZ5br2bzaWmPBmRrnHrECfn1RYufGVCHI8dY7C+xMCpwchOOm9s9vDdHm9UyldKJJ44n5mip7bDm9ZXcHK6gx6xOpf3c3B5vygM1Fos+S66dT2FZn5fz2onDz6KIxIlXloChwN+dTX89/ngzm9pMZBS66RXXND0vMdPMrlQJNUuNlLC338fuJ0QZvhDvA3+/Ca8+qxZ3Rpg8DC4bhqcM7V9ZdhNzQgtXipKWmUqYWOPgdzixsKlIS4vrNgDY3qYpoSuI2DoFJjxV5C1olY2+CztWwU/PApjrjGz6yZFmBisy3C48Hkzf0zBJkBA9lDofaI55yBsrlmI/0T10cVBLE7NFGmaEFzV38FV/R1UeCSagDhb089oRqzGhNh6S9boTnrUhIuhSdzxjb/XAnh1Tw0P92vjaKFIKC82H1h84Qac14b6Q+19rtYeltEVrv0rdOkT5YkqFPUo8dJS3PJ7s9Dipg3+BUx6rJmKPxCaFjiENhDJKdB3gPnzJ6/BmiWhr6ksh3EnwKlnmZkypQR7Owo1bkh6F9i7LYggFJDSuVWnZB3N3KoLhteAfAf89h+Q3AXm/dsULP58k6QBBVvg69pQ1Lg0GPNr6Hls+FNzJsCQs4GzLV/iMqo46NsT/liHMch5Iltdi/EQ2MLY3znO7/GEMBKzfbC9+b5QEomhS6o6eZqEOviA5WUhao+1V9IyrUcYCc18oDF8EJ8EN//LrHu0ZoFpWczuDf1Hta+HHsURiRIvLUViErz3Fbz0b3jnNSguMr/QmRmguwgYmqDrcNQ4GDkOykrhqaesZ+k9arQ5htsFbwdLgXrYeDG11h9bO/84HH8urJwTvM3481tlKkGpLIOF02HTMkBC3xHQtV/g2lIN2b4FbPHmtt2uhaEThtVRVQRzn4Dqa2DQGc2ZvSXcsvlesel6N3o6h5Nt78OCyvdxyaZWqX6OseQ4/ReddPskRS6DJIfm1/pSx8fbPby8MXJhUVcHybBJqtKbChcwbQ6OQOU52jtHT4b3nwBfgN+RpsOwE+DEC8w8Sz4v9B5m+rTYax2Pj7MufBWKaNDOV6sOTkIi3PVHuP1eU4jExsI1p0BhkEgMnw8K9sGvfmv+P78UXn/J2iJ2Wu0NZN0yKC8J3V4IOPHM9lkCwB/DjofBx8KGn5taIzTdrKty/LltMrVDbFkB/7kdXNX1c1w936yFZAVpwIbFMOLEyPJtLH7dLNRoa9loFjvNd8SM1ZIAcGrxTEq8ljzPTna7V+GTXhL1TvR3Hotda/o+lhV4uf2nGtY3KIg4MFnjqeNiOKZz41uaISV/XR55uH//JI0eiYJCu4+fXIEFkATOzgjjeyQN2LEANnxtOmJrOnQ7GoaeA+m9I55vUAwD9q+BinzT2tZ1pLk1mJAC598MHz3d9BpNMwXKBbdAdi/zO6hQtAOUeGkNdB1S08yf4yyEDjas0XPXH2HFUli5NHB7IUzrSdVB+M+DUBGkdlFDNB0uvcla2/aApsO0f8GHT8GPn5lJ6cC8wY6aBJffCzFtGG1UUgD/vs20fDVKty/DC+F2u8z8KjFJpqNvWEhY/SmM+lWY1zWmWBZTLIuxYSNLZOEQjRfmfd5NzeofoLOtZ6P/Z9p7kmnv6bdtHT/s8XDJzOom/isbSw3O+KaK906OZXK3eqG4vthgd2Xk3i7/OSGGMRk2lpZ6OOFnl994Nx3IcGhckmVR0EkDfnwOts+rd5Q3vKalbddCOPE26HlcxHP2y54VsPBF00JXh80JR11kFuE89UqIiYcvXoSyBhFjvYaZ36tsVaBV0b5Q4qWl2bsTlswzF6SkFBh6NOzeGvyaSefU/xwXD+98AW/9F554tGkYtq6bN78kDd5/HhDWn9in/QV6DQjjzbQD7E647B447ybYttp8rz2HQEqntp4ZzPukVrg0s9JOTn9zURtwKqz+JPz+CiIXFiWyhAXeBRykfgHT0RmkDWKENgKttqzBLrf/VP5WiRPJZBwmXkJhSMmv5zYVLnVI4Jo51ey6zIZeu4WTXx3538ImoE+i+X6PTrbz5vBErl5Tjk/Wl5Y0gM5OjW9GJxNvVMGG+WZhS90BnQdAr+ObRoRt/N4ULuC/ptT8/0DmYIhNiXjujdi/FmY93tRa6XXBsrdMv6ph55vbQsefa36vaiohIweye0ZnDgpFlFHipaWoLId/3A0LZoR/7cjj4at3Ycs6SEmHKVfBdTfD5dfAGy/Dmy/D/r1mhtQB/aE0F5w20yxsBSEgpw+cfVn4c2svxCfB8BPaehaNWTW3ecJFaNBjkCleAAafDbt+htK94fWrRfa1LpflfOv9Fg+Nt0d8+FhrrKVG1nCczbQIeJrh8xIrkhgTf94hIWSVL3Z5QqY8qvLBV7lezu1hWl+eWxeZo64u4IJeNtJi6ud4UVYMJ6Y6eG1vNUvLvDiE4PRODi7qbCd21buwfnrjv9P2ebD4NRh7LfQ/xRQPPz5bL1wCYRhmTqDhF0Q0d8D0S8ldDIXbYNu8WuESQPat/MgUyo54857Sf1Tk4yoUrYQSLy2BYcCfb4C1YWTZPYSA357VWIi8/QwcOwkefAFuvM18uWpMB9BLx5nCxXL3GjiccPc/VERAtGlOHRehmduF1zaoh+WIgzMehuXvwdbZ4LPYf9+JEU1hjW8NHjwBw5+3yq0MloNJESno2PER2gF5bNxU8j3bKfcVYdPsZNp7kWnriy5CFAPyw7z91iyKPx0wxcuaIh+zLF7TEF1At3jBw0c33QbKdGrc2/uwcOhFr5gFLP1heGHhS+BIgJ2120IhkaboiJT8TTD7n7W5gYIU3jw0Rw/sWgz9IvvcKBRtgRIvLcGy+dbClP0izXShh7NoFtz3a/h7bSZMZwysmAXVAfKGBKL3ALj3CejZP8L5KQLSZzjk51rftrM5TF8YRwwcdw6cdjWkH5avxREPx14HOUebeV1CYY+LKFzakAY75I6geVsEgm3GNkbro+niGMBO98qgfSZpnUmzZZNmy/Z7fnOJjz8vrWFjiYFdg7O727l3hIMYm3+LTLCIoobUhVB/vtODrUEiOSsk2eHKfg7uGOYgPcaCZahsf2Dh0pBlb0OF1TpVwlrdKb/zOQAzHqkXulYsdkIzhU4YSCkRbfTwI6U0BZdmb7M5KNoeJV5agllfmM6l0ajO2pAVP8HrT0D/4TDmRKiuDO963QZjJijh0lKcdBEs+MJ6+2PPhEt+b1rCQt2E11noV7PBmY+Yi1GYePDgI/Tntbp2u6ifYyy73WsxglhfhscGzs5896JqXtnUeHvq3+vcPL/BzfTT4hjTuemt6cq+dp5bH9r6dFU/c8uozC3DKtY49+x4Bqdqh/xlLLF9PpbKVlgWLph9dY2wptj6r8yFPZyUfNKA+NA+Y9LnRubOwcidA9WFoDsRWWPQep6KiI8wSWIYyJpijJ3fIfcuAJ/LHL/r8Wg9T0PEtHSOZ0V7Q4mXlqC0OPrCpY66/C3JqXDxjeFdaxiQlmFGLr3/P9i+1eznnAvgtHPMzMDNobwMPnoHvv4MCgvMm2J1lRmKffwEuOo3MGR488aINlLWlhSQEJvUvK20HoPMkNJPnrHWvnCvaUELhafGDHENRe8TIaWbtbEPw44dHT2kgIkTpvOpTbMzPv5SFlZ+hJvG/i8aNo6OO5tEPc1vH69tcjURLnV4DDj7uyp2/iqecq/AkNA5VqAJwYBUnSGpGuuKA1sTRnfS6J5obkn1TtIsWV0EMKWnjWHp4W9lUR2excISsamRRxvtWBC+35UtJmABzjqk14Vv2RNQupNDwsjnQu77Cd/+n9FH34FIbbmaCLIqH9/PfwNPZePxd8/Gd2AJ+jH3IOLaa4JKRUugxEtLkJVjRgEdXiE6mpQWw8uPQ3Z3yNtrTSwJAYuWwu/vMq0wPq8ZZjzzG+g3EN76DDpFeAPYsQ0uPwfy8/xnhP34XVPY/N9TcMlVkY0RTaSEld/Aog+hqDZTbFo3GHshjDwzchFz+q8hLRte+WPotmmZ1vp0W9warPRTYsIimtDoo/Vhi7El4NaRRNJHq0/5Hqcnc3LSdRR6drPHswGJJMPWk672AUHN+Y+vDJ500WPAkI8qKak1snSLF9w02MFvBjqYflocJ35ZSa6f8Oc+SYIvT6uP7Lm4t50Hl7lwh1jLJ2TrPH1cbPBGgYjzL9AixhEPk/9kbilGgjeCnDYjLgqZF8jYPr2xcKlDmtXpfateQD/xcUSEzuKh8K1+BTz+MlRLcJfjW/s6tmPubpGxFe2T8O3LitCccVHLCpc6BKZocThNsRSKPqNNAQH1RSDrHIO3b4Gbr45sHj4fXH9JrbUlwKNu3fH774CN6yIbJ1pICd/+G75+Eor21h8v2gPfPGW+Ar0PKxw9GeKSQrfra3FrwGMxsqeZIdpDtaE4cCACbLb0F/1JFk0LNXayd2dE3GmMjDudbo6BQYWL22uQb2F9LWmwO7SnUvLHJS6mfl9Fgh1WTE3gxRNiGJam0TVOMCJd4/UJsSw5P7GRv0xajMbjY03L1uE3OgEk2OCtibF8PDnOeqkBaZiWsLrPR58J1q5zWKl5JGDqc5CaY61PfyR1Cf+anBBWF8OD3DOPwFtRpoCQ+avCH9sCRvk+KNsZvFHJVmTF/hYZX9E+UeKlJeg3NPww5ORUM/w3HKQ0rS73PWFmyg0kYOIS4Pq7YUWQm4vPB8t+htUrwpsDwLxZsHO7NcEmBLz5SvhjRJOdK2D59Nr/+Lkhr/gadkQSKVaLpsHpIYRgTDyMOc1af1az86Y1L5FYgkjgDNsZZIiMRsdt2BiuDWesPrZZ/QPUNENfzTvgY/hHlWwsMbioj4O55ySw5qJEZp2dwLk9/f+Oru7v4K2JsQxKrb/VOTS4oq+dZRckcGZ3i06fZfthwfPw1hXwzlXw3rWmE67NCcPOC36tI96aZTShEzgitADVMfDUMC8Qoa1H1QfBG0JACx1ZtivMsa0hD1gLfjDKc1tkfEX7RG0btRS3PgRZ3eC9FxtnvPUZUO6GmtoiiDE2SHbClF+b7R//XfhjOWPh/qfgzkehvBQSkiB3u5kgLy4BRh4Hu3dC/n3B+9F1+HE2DA/TWXDhPLMukpXaPVLC91/Do0+GN0Y0WfYFQZ0shYBlX0LvoyMf45TLYfNyWLvgsL41c8vu5n9a83cBSMgwXxUhtoW6HxPZXBuQJJI43XY6pbKUElmCjk6myMQuolOsM8mhhR0B1JD91ZKzv61k3rkJdIu39ux1Znc7Z+TY2FMpqfBIuiVoJIZR1JGD2+HbB82kbnWfGXclrPvSDH8+46/gTISVHzbdtuk2yszz8tX9obd0EixuIwaj7yTYuQgOrCOwpaQBSVmht6gsbQXJiPMLhey5Mkg5lQaIUAJLcUShxEtLoWlwyY0w9Vp4/lH44k2ocEP+YRFC1V4oroHcfLjsZjN/y7MPhZczpHOtqTg23nwBDBhuvuqwahXx+nekDE6YK9HBAtO5NzFMS1O02LuRoHOWEg5sad4YNptZyuCn6TD7A9i/w9zeO3oynHJZeOnWhTAzoC58KfD5tF5mRtcokSyS/W4RRYPTc3Sm7458W7XcA8+vd/PoGOv1lYQQ5CRE4MckDZj1d//CQxqmoFzyBky4AwaebuZYqSwAZzJkDqjfLup+jJl4LuDWnoDuzRDLdeg2OOU+WPOpGXkUastxzK9D9xmTDnGZUBUkYkoaaBkt5IxvVZQkqxIGvyTUtlFLY7PDLQ+YhfYOFy4NeeUZeOgeOOtX8OVauPufcN6VwZ/ONQ0GjYAcC4XcevaG+BB1lbxeszJ1uIwea83q0pDPPwx/nGjhseB0EZGIOwzdBuOnwF/egecXwtNz4Mo/RlYnpt/JZsZdaBAKXbsYJ2bBxLs7TNLBF8bHkt6MupE+Ce9vi8LfxwrrvmpcD6gJEnYtMqOOdDtkDzWTBOaMauznMiiIE7jQzEKJVv1nQqHbYcTFcMkr0CPIVt+AU6FbaCurEAKtV5BK5UKDlL6I5J7hzzUE0vBB6Y7QDTU72OOQJduRVflRn4ei/aHES2sghPm4GGpxeetV05lV12Hy+XDLg/CHvwOi6bVa3fbDX6zNISYWLv21eZ0/dB269YDxk6z115CTz4CsLtYXT90GW5tf2C9ivMGjXQCzhlJ7QggYcxWc/TezanTnQeaWxPhb4dx/QnyUo15akDibxqoLEzgrR6dh3rm4MCKVS93NcKi2iqsClr8Tup00zBIOwUjpBhP/YNY8QpivOhHqTIRT/2zRqTcMdDtMuBOOvd4UuHUkd4Pjb4ax11nuSnQZh+h9uHiu/TehK/qI6Bd4lTXFZni0z4LlRXdizL8f3+K/4fvxT3jn/A5jT4gyDIoOjdo2ag0MA2bNsBDBIuDDt+DPj9UfmnCWmSfl5cdhT4MnkP7DTOEy8Chrc1i1DH6a57/+ka6bVpkX3gwsboJhs8HL78Jl55jbQaEQQGwbVX92VdVHWgUjgvT1rUJ6bxj3m7aeRbOJs2m8OclcrItqDOJs4DYEYz+rIL86WJ5fky5xrWBl2vC1WbTQCvmbIGtw8DbdRsFFL8DWOVCw2Uxk2eUoM6eLH78Tn6eCmrINSMOFLaYzzoS+iHATEArNtLD0n2yKMSFMkRSmlU4Igd73XGT2MRh75ptbSLZYROZoRMbw8OcVAumpxLf4cagptnbB4WHU7nKM9W9h5K9CH3mLysR7BKLES2vg81lbMJGwzk8ysuMmw7hTYNt6KCkyfVy692naLhCrV8ClZ4MngKn9uAlm/pUukSU4A2DwMJixCJ5+HN57I3hbrxdOPyd4m5bAVQmvTrPY1l9OCUVLUFf8MAb44rQ4pnxXyf4gD9sacM2AZiZUtMK2+dbbrngXYpNNq1gwnAkw5OygTaT0Ubr3SyoLF2LWrTady3VHKqndL8WZEMG2oxAQkxj+dYd3E5+FPuCiZvcTCrl7dq1waaaFrXANxs4Z6L0sRvYpOgxq26g1sNutZ69dvthM+HY4QkDfIXD0+PCEC8D//Rk83sBVp5cvhtQobDt0zjKjiF54K3AbXYdjT4DhbVC5dv6bULzPWtvq8padi8Iv/ZJ1Vl+YyMld/Fu+dAEDUjSuH2jh+7R9GXzwF3jlRnj3flj1LXgsbBnWYTU5YB2LX7PmTxWCktyPqSxcgClcoG4B97lLKNz2Eu6qEFtURwDG3h9ptnCpRe74GmmE6ZOnaPco8dIa5O0Ht8XoIcMH998OVWHWLQrEnt2w5KfgeSYqK+DLT6IzHsDkM+Gfz9fX7LHZzBfA0cfC82+2vnOp123mb7F6Q4y0MJ6i2eia4P1T4nhwtJN0Z/3nxK7BpX3sfHV6fPCkcqV58Nyv4d17YctCyNsO25fA9H/Bi9dBicVkZklhhi57XabzbjPw1BRQVRQor4kEaVCe932zxugQuKP48OCthvI90etP0S5Qd+jWoDqM/ANSwuKf4Lgh8OwbZk2g5nDAoqXhT3eaETaXXdO88eo4/xKYdBp89oHpnBsXb24VjTi6baJiygvBHcbfof+4lpvLEYghJTvJYwd5lFFFKgn0JoscOkXkb6AJwW1Dnfx2kIM1RQYeQzIwRSfFGaSvskIza/K2xYHblObDe3+EG18JXcCy/ylQuNX6pIUOlYXW2/uhungF5jNloJBqg5Lyzcz2rWGddhAvkmScTKY3R4mWL47YajiTzOR40UJZXo44lHhpDbK7mIt3ONaU8nK4/lcwfS70aUYV6PTQ1WIB0y/nz7+DkmK4+a7Ix2tIcgpc3U6cS8OqFSNg3CUtNpUjjT2ykC9ZQiWNt0yWspVU4pkix5EuIvO3cOiC0RkWnKd3LDctLSGd4iUczDW3lPoET4tP7xNh61zI32BtstIwfVqageEz7xE+BDuSMjgYE4/d8NGnrIBkdzX7Y5P4sO8xGFp94rYavLzFGubL3dzM0WhRdp5tC7Su4zG2fhat3iAhO0p9KdoLHf9T3hFwxsAlV1qrP3QIaTr5vvZC88bu1ReGjrAeRfTEo7D3CEyzndgJMvtYsPoIOP+PkN0MwWiBvCKDPz/v5aw7PJx+q4drHvawYGXzahNZxZAGy+U23pfzeV/OZ7HcjBFhXaQCWcqHLGgiXOooppL3mEelDMPXJFzc1aY1xWo9KiFgm4WU87oNJt8Pg84w84hY6bfHsdbmEGhIRyq7E1J5ZfCJTO81gp+z+jC/ywBeG3gCX+cM5aO+YzACfIZ3UcrHbGzW+G2BlBJZU4ysKcLweTB2z8HYuyD0hZYQZpi3Pcph6Io2R0jZnAp0bU9ZWRnJycmUlpaSlNRGGVutUF4GF59hbqEEcpz1R3IqLPfjwBsOP82Dq6ceqgAbkvMugidebN6Y7ZGNP8LHDwU+n9QZrn0G4lNbdBqzlxo89JLPr/fN8UcJ/m9aaIOolJL9FLOTfCSSbFLpSSZaCHG2Tx7kA37Ee9i2hAC6kk4icfQggwF0w24hXPwLuZgt7A3pSTSOgRwvBoXsLyLmvA4L3g7jAgGjz4HTb7V+iafaDHFe/FrgNkPPhdFXhDGPpuzy7Oc5bbUpUA7/W0oZUnzrCP6PSR3C+iKlRO6Zh7FzBlTXlr7QbNHd4onLQh97jxIv7ZTmrN/t4hP+7LPP0rNnT2JiYhg7diyLFwfZs+6oJCbBB9/AjXdYr2kDZrmA5nLcifDyO9bT8c85Qh0CB54Ap04zfR2EZvooaLULdLehcP0LLS5cSssNHnrZv3ABWLBK8sb04LlF9shCXuV73mEuP7GBn9nEJyzkFWaQL0sDXlcha3iX+U2EC5huzHs4yEZy+ZblvMJ37JUHWSN3slBuZLXcSY1s7HTulT62sM+SC/QyttBiz0lbfgrzAgldwxRS9ljTAjPxboip/R7VCQTNBkOnwKgwi7E2wCcNlsi9vGTbgCE0/yLFgu+QD8k+2n+knJQSY+O7GBverhcuED3h4khC63se+rH3KeFyhNLmlpf333+fq666ihdeeIGxY8fy1FNP8eGHH7Jp0yY6d+4c8voOY3lpiLd2O+jpv0F1kHBMTYPBw+HzWdEZd/5M+LWFHA0OJ2w4gsvLlx+E1d+Zvg+OOBh0InQf3iqOxI+/4eXrBcG/cgmx8NXTTbcpimUF37GcPfh3ZBSAAzu/5mQSRdPqxNPlYjYSfpithsBAoqNxPIMYQz+EEFRJF8/xteV+htCdNBLpSWcyRUrY8wjIi9dDYRgVjZ3xcMcHYfpBNcDwwp4VUJ5nfn5yxvjPoVKRD7nLzCik1O7QZYTf7VuX9PIvFlIcYOutEZJDVSECMY2j6SlaVoQ3F1m8Bd+Sf7TsILEZaN0nIXImIFqoaKSieTRn/W7zv+gTTzzBDTfcwDXXmFEuL7zwAl999RX//e9/uffee9t4di2EzQY33ALjJ8JZ4wO3Mwy48vrojXvcSWZm22CCCaDHEV7gLDEdjo/8Kbk5LNsQ+lmhohqqaiRxMfWrVJms4h3mUkPgkHsJuPGygu2cyJAm57cTpLBeEIy6PCMYzGMdNnRG0QcnduzoeLCWhXYduxEI5rOOrjKNcxlLvAjDChmI7sOsixehwcV/jVy4gGlp6R7E2dfrhoUvwvb5HCrtIQ2IS4cTb4PMxlaf51lqTbhASOEigC40PxldS2PkzjX/FhH6WlmiugBj0wdQuBZ95DQlYI4w2nTbyO12s2zZMk455ZRDxzRN45RTTmHhwoV+r3G5XJSVlTV6dVgGDoHf/9n8+fAnMiFg8llmyHG00HW46/7Q7S79dfTGVFDmK2BV1Q98X/YyVb4KrOSaMQxwSQ/L5TY+lQt5l3lU4w55pUSyHv8O1/62iyJhARvwSh+60BhGz1DraZP5AeyjmPf5Ea/V9PvBGHextXYZveA3L5lipyWZ9zTs+LH2P7J+ga4qgu8fgaJ6oVUgK9kbxW2ePqThEO1/kZYVe1tWuNSPBAfXI3PntMJYitakTcVLYWEhPp+PzMzGyaAyMzM5cOCA32see+wxkpOTD71ycnJaY6otx013wrOvmxFBdXTtDvf/FZ55LcwIJQtcc5OZf8UfQjPzsPzqquiO+Qtmv2crCyo/YL93E15cZHUrCHGFBE3y6LYDPG98xyxWs40DlGM9R40b/2UgYrEQMWMBFx52Y76PsfQnnvCtJxJJEeVsjmAbqwkp2TA5SGHAmASY9rYpXDr1aP54wSjcBrlLAjjGSzNZ5Or6hJDz2R21oeOwczUWa521Nbam25oth8TYPbPlfK4UbUK7cNgNh/vuu4/S0tJDr9zcIyCs9/Rz4dMfYOVOWLoV5q6Aa2+uz0obTYSAF9+Gex9qXBIgNhauuh7e/DQ8h2Ir+HxmtJUvCk/ZHQiXUcmq6hmYS7V545x0Tp1FMdCNVFCRU0V8/yUBRUgoUvDvoDiSMMtKBKEUMx9JDA6SiHwh2kCUMp8ecwFc9jhk9a0/5oyHsRfCnR9CSmj/uaiwY0Hwop7SgN0/g8/c/qshPAfVs+nHbxhFNvX5ZGxojCaLP3ICMR3A6gKgZYXIsRNtqg+CYTHLuaJD0Kaf9E6dOqHrOnl5jffi8/LyyMryny3S6XTidDpbY3qtj9VooOaiaXDDraYVZstG04G4d1+zsnQ02bcHXngKPn4HamogJhYu+JVpbWpOEciDB2DzUnNvpfdwyO4ZrRlHlVzPeg6vj5zaqZwTTl3KjzOOpqH3pUQiENTEGGSemovD7kOL0H+4L/4Tch1NH35iwyEfluYwk9UUyQoySWEfFiv/+qGaKOZ/6TUKrnu+1uohQ2fQbQncFrYFpWHWQNIddCOJFfi3MjckGSdn0I/Rwvzb3kXHzgAtuoyDHd+Auyz49lGdX4wzFdF9InJLpGVMRPutFK+IiDYVLw6Hg9GjRzNz5kymTJkCgGEYzJw5k1tuuaUtp/bLwGaDQUNbpu8dW2HKyVDRYD+/phrefR2++hQ+nmEm0AuHqgp48W7YcFgofY9BcPO/ILUZT9fF+TD/U9i+xkxONmQcHHsWxEUu6Eq8efhbyE44dTlZ3QqZPX0sB/NTkYBPh+IsDwe6exmbUxiWH8nh/MRGdsp8skjBh0EKCQymOzNZFRXhUscKthOPs7bmcWSkEmXBDLVRY35+g3tXwbK3oSQXpM90oB1yDgw4BfQoValOsPAZtMWAw7SOjaEL09kc9PcXi40/Mj5kmQWvNPjZKGShUUgxbpKwM0pL43gtg7hWsshIKS2VgxC2GPQxv8e3/D9QlVcrNIX5d3EkIzJHmcJGsyMyhiM6j0Dmr4r80xuTgtz5HXQZh4iJQhFaRZvTLkKlr776al588UWOOeYYnnrqKT744AM2btzYxBfGHx0yVPqXwMRRsHtn4PMjx8BH31nvz+uFP18ABwPUaopPgkc/h7gIIi1+/hZee6Bp8crYRLjzWeg5OPw+geVVX5Pn3R60jWFoXPPFFXgbfAv/n73zDo+juvrwe2e2r3q3ZNmWe8dgDDa9995L6CHUJKQR+EhIIyQkJIGQhBICoRMg9N6rMcYG995kq/e2fWfu98dIVtsyu1rJNtnXjx5Zu3dm7u7Ozj1zyu9ccvyXlBWmJhG9ZxnZXaP953Iw5cJkC4tkqV8Ln9wTve9Q4WQ45tahVSD14GmC564j6jsuFEMvZt4lOx/6SG7jVTYOHtvtmLuauUwQsRfcDXoHd2vr8ESo/LKjcIM6lWlKdgIvxDxS6siaRejb3zMaICoqonA2yrhjEdkV6B1VyLovQA8j8mcgCqYjhIL0NqBtehnqlrCzl5M1E0oXoFYch7D1N2xl22a0xXckOcteg0qMPxFlwslJ9dxKk1r26FLpc889l8bGRm699Vbq6uqYM2cOb775pinDJc1uSFMj/P7W2IYLwNdfwoa1MNmkWNj7T0c3XAA8HfDSvXD+jaanCsC2tfCvn0V+ztcJf74Wfv9KUkZRgWVMTONFIGjzjmZ8aRPzZlTisIVp7XTQ3O6iJL8TVRm6ybG7Gi0ARWQzmvzhPcjy52DZM7HHNG6AFc8NSWRuJ13t4BwHlSuM8JWqgNMKLpshiOjKh1mn99vkUDEORVN5SW5EqL3GR5vPxtYt4wmPzyJKGhMANdLLH7U1hKJ82gF0/qyt5fdib/JFakPuUuroKx5E1i+BHh+cHkY2LEOr/xqsGRDq9b7K7e+BxYmYdCZy438NDZy+VXChTqh8G237eyizLu+fG5M9HpyF4Gsi8TO7d7zc8irSlokYc3gSrzjN7sIu97wMlbTnZTdi3Wq48BSjuaMZ7vonnHymubE/PAq62mKPsdrh7wn2RPntxVC5JvaYc34ER52f2H6BsAzyYdejhGSASBfbMLDZWUHYEoxYnPJNvjFUEHyHY8mIIKaXEP4O2LoQfG3gyoFxB/Qq4FYvg3dvN7cfqxPO/ZcRMkwGTys8fxtsXxH5eVWBOYfCodeCM6ffU1JKzljWwRuNAYqyPditOu1eG61eByqQYxUsWZDLaIc6aDtvCJ4Qm/mcxpjLuQBOUEo5W01ttZVe9Sn6mkeT3DpewFGg7ncjIqc30VxvXIn+9d+6/xrC0mXLQj3kDoSSzoPZlezx7QHSfAMIheDyc4yqIrM4Eli4PNFl73vnkGDyp65DpYmOwV+YV5Hti0XYmOc6FSv973ZFtwu7OmMcYdWogOhpZdP3B8z3GtxTEICKwhkcMDTDxd8Jn/8T/vMdWPwQrH4JvngYnr0KVr5gvHFrXiOuqlsPIR944pWxR8DbDp89Bf+4NLrhAoZzYeMqsLoGPfVZW5hXG4NoCGrbM9jWlEWr16j404C2sOSuyt5SeV9Icu+nIQ79q599/+jj01CTCf0fWKq3JPrq4qJvfw/T7/Eg4p/c2tY3+/2tFM5C2ft6sA8IowkVLIPf26gEO6Bjm/nxaXY7dnnYKM03hHdfh/oEWwrMP8j82OFYxVtqMXX31pH8RT9bLeTQzIuoDq2jIbQNHY0ctRibrZzVYmnU7fZko6XnfnoUuexFBVU0UUsrCoIJjGI248gSCSw0AIEuaK+BoAc2fQSVn9Pvs+vJV9I1+OopsNiNXJdE7s4TVWDdsBBe+C2EQ/GPI3XoaIT1n8KMI/o99XiNH4ugX95TXzQJ/672c+eUDLxByaWPB1hdJ9G7c2IUq7nXGEqRSKFs24y+/X1k83oIDadIqITGlUipI/pUjgl7Vr9QlDFUg3Ac5fCBe9cCQ0qMT7NrSRsvaVLD4s+N6qVwAroVXo/58nBnppGDEgtbgnfyZrt75xQmtt8BWIWdcba9GGfrFRB7SX4Rc5s9NWQ0nyk4sVFOIUXCSBCdyRBCFYEuWPKYIbWfSNO+Zc8mdhxXHrgTSBxu2Ar//fXgJO9YCAU2LR5kvDQG9aiGSw/tYYkuJQ98Fu41XACkINBhxZYZinvOVIihV3bp295B3/Ds8Ev79x7RMEy6jRcpdbQlf06JZotwR5YUSLNnkA4bpUkNiS62QsBzT5off4QJCfjDzkpsDlkmSyYPOjWx/ZoglKA42Z7CXCYyV0zcabgMiaAX3rgVNn+UeLfhoMdohmhW62XOOYlZjIv/m9h8AJBG36MBjHYoWOIcutAq0HR46qtwr+HSTf3X5oyuo5TI2llmkW2bDcMFRshwAZwFCKVXGVpu/xDC5tWmIyIUKJiFcOzezSvTxCZtvKRJDfsfmJjXRVFg62bz44+5CDJiXGzc2XBSgk0sX/+3uXEHnpLYfk1QiInFfQ+KHQlgNPk4RYr0UgDWvgHtQ+iBU7qXuW0nHm78JMKGzxPzuuyc05RBD11S5ojpeVGBb4920uyBjgj9G+u/KsBT74x5upyklDF1iKXSWuU7Q9o+cQTKmP5eKr1m4RB3qYA1A3XarmnMmiZ1pI2XNKnhyONhVJn5XkxCQEYC5cfODPjZYzB68uDnRk+Cnz8Bjhj1pJH46l1z48yGlxJgAVNjD5ASeyiIIxTcI4wYiYnXlCirX2FIFSVj5sG+PX26Irg23IVwxE1w4DWJx+m0JFo3KCrMHtxXbJ8sK5eW2iM6Ly0CyhwK3xvrxB4lyK+HVNY+PZG6pYXo4f57KcbB9epkzlLHJD7fgTStNjnQzHspoDhOi4D8aYjyw/o/FmgzOYcI81AsiLKDUOffgnAOc4l+mmEnnfOSJjVYLPCv/xil0u1t8Rf8cBhOOj32mIHklcCtT8K21bDhK+OxyfvAuBlJTdlItDRBKACW1DQ17MEmLBwop/EZEaqduo2VQ7etI8vv473x02l1Z6YsEWYoiri9+zBqpnQkKgrHsjdjRQr7B2lBCCWWgNk7OQVyyiGvAvLHQ8kMWPcWNK43qlJGz4Epx0LGEHKZSibCjtXmvUJCgVNvAndOxKfvm5HJaIfK3ZU+OjXj0xHACQU27pmWQYFNARvsVSpYWSsHhY70oMr298vY8fEo7r5IMqtEpRA76sBu9Ukipb6zH1NMFCuicBbkTERWvg/+CMKAihWxzw2oeZOQ3tPQt7yKbF4HgW6JBdUOihW8jehrn0IU7QVdNciwz2QYUKDM/YFRTSQUyJmEcOaC1d0vBJVmzyZtvKRJHVOmw9tfwDOPwXNPRA8LqSrsuwDm7p/cccbNSN5g6UvxGGiJ01dGUQ2vzzCwQEzFLR18LFfiJ2QYJ1KSGfCxoHIj+X6j+eFxm1fx+uS9aHe4k69K7YMLB5Mp5Wtiq//GIo8MyikgnyymU45dpHhRaIygOGsKYVQNHXBVr7GXXwEHXp2yqQGw72mwfWX8cYoKkw+ABedGDBn1oArBLya6+UmFi8/bQgR0yexMyyBtl6sOsnLtM5GNCFXA7BKVo0bZU68eG2jHlMmbUYoy+yq0L34PgShVenoI4a2DvEkIVyHqzMsA0GoWIVc9DHoItACEupDVjcjqT4zthGok78ZDCETuJER+aj2BnmAtDZ1f0O7fhETHbR1FYeY8chxT0mq9u4C08ZImteTlw9U3GD8fvwc3XGl4YixW4y5V0+Cgw+HuB3d9Sc1ZN8Bv4sS+Zx04rFOYLcYxqcPL9qon8NrsZPl8OCIkpx62eQ2vzdiP8BB8Jrm4WcA0JlOKRahMlKN4naV4iJBIEYcOvBwl5iQ9l7gEPMltV7aXoZSbNy6l0xnE1INhzvGw7I2dRifQW4Uz/2w4+KKEK+BcquDI/Oh5Q4dPUrn1OCu3vdXrNRQCNB2mlwj+fvYwGC5gvqlh5hho3wIdW2MO0ze9jCg7aOdcpbcBufrfgIweJjVjuABIHdmyFlGQur5tjV1L2dHWv51JV3AHXc07KHDPpTznmLQBM8KkjZc0w8chR8Lna+GtV2HDGrA74OgTYGoKvCapoHwyHHAyLHwl8vOuTLjiN8M+DYEgIxQkIxTdLe/SQlzcVsZHuWG2Up9wg0WBYBKlTBflADTINpaxNSnDBSCExjPyUxYwdXh6E2Um2B6kfB7sd+nQQkGJIASc8AMYMwsWvwB1m4zHKvYxDJeKfYbt0OfPtXDkZJXnl4fZ0ixxWeGYaSrzxykow7WA2jIhYzR0VcUcppTMQ29aFb+UOtiOvvF51MlnIn3NaJtfTW1ulz/5TucD6QrWDDJc+tLkWUqmfSy5rhTnfKWJSdp4STO82O1wypmAyTYAI82lv4BRFfDGv8HbLbglFKOr9HduTzwJOAms7jEmXOKCrIxJnC5yANgia3meRaaPIZFMoxyPDPAqi9lBlCaFCbCDJnbwCSfJ/Zgiyoa8v37kjTVyVlpi38GTPRqOuBGyIpQBBzzQWmu0jcgbnXpPnxAw62jjR+qAGDFvYlGm4OqDRi5/QwiBMv549BX/jDYCMsoQeVOQLWswE9+U294ivONj0IZY+hwJe+qaUFY2vxx3TEPXl2njZYRJGy9p0hx7sfHjaYeAH3IKjFyFEUK1uHHl7Yu3eTGR8woEzpzZWGw5Ox8ZL0ZhlRZTejECmEwZuWTwOB/STByxP5PI7rm+yVIqZDE2keLLyYLvwJu/6K7sifC+zDjZCBEN/Ky87fDBv2Dlu71VQZkFcNjlMPvo1M6xB7N6MnswSsk8I4l204t9PCvd6d+uItR9vosQApFVgTQb4hkOwwVhhK9SgCdYR0CLr7DtDVan5HhpzJM2XtKk6cGdbfzsArLLTiEcaCLYtZneeiDjt9VZRk75YM/VcezDKyyOuV8BzGAsR7EXG6mhidTLuYfQWEcVsxmX2h0XTIATboOlT0DN8t7Hi6fD3AugMELZvK8D/v19aK2hn8HT2QSv/AFq1sFx303tPP+HUMafgCiei171MbKrBmFxIIrnIormILrbK4jC2YbnI2CiH9kwIbe/C5MTFK2MwI7WN0yOTOe7jDRp4yXN7k04aOQT6BoUjjPaBHwDURQbBROuxN++Bk/zYrRQK6o1G1fePJzZM3cuDH2ZIspQ5f68wzI89DaltKJSQQljKGQCJWR2N0BcK6tSUiY9aO4ImofBKAKMxNujbwFvK/hawZEN7hgaHZ//Z7Dh0pelL8OkBTBh3+GY7f8Ewl2MOuXs6M8rKupe16AtvoPUn21mkMiqT5CTzujXEylRAuE2vCFz/doc1hHKtUqzk7Txkmb3RNdg4dPwxX+NDsIAqgVmHglHXQ2O4Slf3pUIoeLMmYUzZ5bpbSaKUiZSSrPsoIlOcnBT3J0XMxAfgWFZSiRgHe5LiSvX+ImFrsHSV4m7YL79d7jm4cjPdTRB1SojeXT0DMhOoXbN/xAiZzyi/DDkjg92zQTCPmSgE9myBr32Cwh5Ee5ilLKDIXeSqcogbyiOjEIfRmUdPJTZpkmCtPGSZvdDSnjtz7Di7f6Pa2FY/jZUrjAWlpp1xoJVPAH2Ox3KZ+368utdRL7IIp/YTS5zyaCOtp25KqlCIplMaUr3mRQBr9EPKR4tVUZ4ydnn/fJ74I27YO1HfapeBExeACf+CFwmG4im2YlScRxa9afdfamSOOdU5xByYgT6kjvBW09P+FV2bker/QJRegDKjIvjemUUzOW92dU8cpyTkpxnmmT55meZpdnzqF472HDZiYS2Wlj1rrEItdXC+k/hsR/Bs7dGbHyXxmA241JuuAgEFRRTFMXbM6LYHJjOPfD2ycfQQvDkT2HtxwPKdSVsXASP/whCyZWUf9OR3kb0hq/RG1ciw/3fI+HIRdn7uiSS3wVkjUUZfwJJ5ZIIBawu8DX2zLL7l1G6LWsWIivjtwZx28sRJgyYSYXpPkm7grTnJc3ux7I3jAteoo3vNi6Cd+6F478/PPNKAl3z4W1Ziq91Gbrux+oYhTt/PraM8SMualVGPjMZwyq2p2yfYyjkZOL0qBkpVCsUVUCDCeVgV07v/9d8BLXrI4+TOjRugxXvwNyTUzHLbwTS14y25jFoXtP7oGpDjDkKZeIpO70aSv505JRzkWufSGTvKBNOQbiKYGOi3buF8ROKLXKob3sbMeZIRAzDyqI4KMzYh4auJUTzHBW498ZmSXvldgVpz0ua3Y+2uuQ69oJh+Hh3XZVDX8KBRurX3kl79UsEvZWE/fX42lbQtPk+2qpeQI5ww0UhBMeyDwcyLWX7PJ652FLdGmAoHGWiDcDkA/onfq94K06ps4Dlbw55at8UZKAdbfHvoXld/ye0IHLr6+irH+33sLA4zO9ctaPMugKlcBbCXYwonkt874vo/fzsWYjSBcRd2oId3SGl2JRmH0GOoyckJPr9znJMYHTOMJXep4lL2vOSZvfDlU3S7QN1DbZ9DdMPS/GkEkNKnaYtD6GHuwY8Y7iuvc2fY3OOwl2wYETnJYRgAVNpkx5Wp8ADs9vd/VTsDbOPiR52tNgNvZe+dDbHabAojTFpAMNrQbCTnnN5ILJmIXLMkYgsQ81ZZI83t2OLC2XBz1CcvYrNouxgZNvmyN2kFRuU7Idw5BjVeBmliIJZyK1vIM1cPkxo0ShCpSL/TLqCO2jxrCCodWBVM8l3zSLDPjbdEmAXsttde9KkYeaRDKnEUjPZLRqgegfcdzccvT9MHwUT82H2GPjVTVATWwo9FoHODWiBJqJd4AE6Gz4cce9LDwcznQwSuCOOwqes3WWvISon/RiOuBKsA17fqMlwyV1QOLb/41mF8T0vmcPQAmEPREqJrP4strEnFPSahb1/ugqhYGZ8IT/Nj6x8b+dxtPXPon91V3S9GD0INQuROz5AFO+DUjTHCAPljI/f7Vt1gMtcCwohBJn2MYzNO4lJhRcwLu9kMh3j0obLLibteUmz+zFxPyieCPWbktu+JIJ4GRhNIT94GxZ/Bu3tsHYlrF4xeJynCx59AJ5/Cp56FaabL13uIdC5kXjeIy3Ygh5qR+2jnDtSZAgnF8rDeJ8VbKQm6f2sYBszGEMZMbRXRhohYME5MO802L7CqELKGw3FUTwAex0LW7+KsUMJe58wHDPd85BhCMep6JJyUG8hdcalaAt/ETsXRerI6s+Qk89Er/4MWflOzxMxDqZD2Ie24p+o839mKPzmTTUME28jkW8eBKL8UIQavQFmmt2ftOclze6HosK37uxfymqW8lmD76wBNq6DI+bCVRfCIw/Ac09ENlz60tUJ114Cepy7uAgY3oj4HolUV/8kQqZwcqrYn8s5knIKknJ2CQQriNN/KIU0dErW1uk0dpmYrMUG4/eFaYdEN1wAph4Co6dH9gwIxSjFn3lk8pP+JiEshtci5hgB9v7fXWHPgoJZxM1f0fxoX/8Nue5p83OSOnTugI5t3YdXUOdcA1Yn/Ze47mPnTqKx8CTeWhvm861hgtpu5jlMY4q05yXN7sPWrfDEE1BfD2VlcOJP4I3bIBTE1MrqzIZTfjL48dYWuOAUaO++GwzH7we0kx3b4LMP4eAjzG8DKBaniVECRR2+xo/hQAtasBXF4sTiGBXVzZ3hbePIuo20+rewtnAUW/KKCKvmLg0SSQsD83ris6Q9xL07fHzaGkIVghMKbVxd7mSiK3L1x6oanT9/EOLzbYYhKYADxiv86HAr00qGeA+mWuC838Fbf4NV7/fmQgjFMHyO+57R3DGN4dkoOwC548PooRmpo5QeMHhbexZSiPjdo/tWMJmfGbJ9GyK7wvgroxT1gF+i7/gQWbPI8Ba5itjkPpbvfzKD7W/r9HhlLEqI8+eq3HS0dfi6cqdJOWnjJc2uR9PgBz+Av/0NFMX40XW4RcItN8A0O2z5sne8Mwt8new0aFSr0XDvkEshI4IK6zOPGQZMvDh4JBQFln+VsPFizo0h0YLNKM5Ric8rBiFfHW3VLxHs6g27WewFZI06YZB6r79jHc1bDLVZFzpza7Yxp7aSusxcAo5clpeU4RexjT0Hibnf76708uP1HiwCwt1v09+2+/jHdh/PzsnixML+hsJXOzQueyJIuM/HJ4FFW3XO3x7g0W/ZmV02RAPG7oJTbjRyZapWG0com5bOdYmAMu5YtNrFhkEw6DslEMX7ILLHDd6uZB7atmj6TUNFwoAWGsKejTrxVJh4KgAbGnTOejDAQEdLWIfHvtSo7ZDcc1baSN1TSBsvaXY9v/iFYbhIaRgyWp8qgN/8Gf70J/jeU4Z0uysbckcZSbn+LrC54t8Vv/pCcoYLGHOyJhMbVzBXMZXaO72Qr47Gjfcg9f5Jy+FAEy3bHiV3zHm48uYCIPUQLZVPMjAvQJWSso4W6GjD5y5iZZYSM7w1jdGm5/dpa5AfrzfyHsJ9dqlJYxbnLu9gw0F5lDoMD4yUkp+9FiKsgz5gCpo0nCS/eD3IC1cOPfkYMIzfqQelZl/fUIQjF3W/n6Ktegja+4QMhYoYfShi/PHoLesQKJA1FmExvp8iaywU7gWNK0h9zyOByJ8ec8QN/x1suPTl3fU6a+o0ppeMXEf5NMmTNl7S7Fo6OgzjJJYr+be/heuug7I+d8GqFdxxet304OlMfn5SwmFHJbyZPWMCnXEu0IrqxuJIbUO39ppXkHp0leG2qhdw5sxGKFZ87auQMeXXdSqqlrFh2n4ERHiQASMQ5JLBZMpMz++vlT5UIFKRqgRCOjxY5efWiUY4bXmNZGtz9PdRl7CuwciDGXL4KI1phLsYy/43IzurkJ07QLFCdgVyyxvoH99sJPYCqHZE+eGGcJ1iQZ39bfTVjyLrvox9gMRmY3S2dkZPGm/olGxtib+nf3wS5m9nD7/xIqVknezgPb2ObdKDDYV9lTwOV0rIFb03SyHNg6b7sapuVCVFBvo3hPS3PU1iaBosXQoffwy15jquxuStt8AfR3q9pQU+/TT5Y0yeBmqSF6QDDoUpse/oImFzj8PiLCXWV0zXvHTUvjPIS5IsWrCNQOeGmGOkHsDXvgqAsL8e4sifu0I+TmxSyNKNC6ro/gcwilzO5SAswvx7+05zMKLh0oMOfNjS+35UtpjzmP3tk1A68XIXIDJHo5QugLzp6Ev+hKz+uNdwAdACyG1voS2/Hyl1hGpHnX0lytTzUnDw7u9W7iSUGRfHHLq12dx5tL11+M8hKSX/0Su5Q1vDV7KFJgLU4OMVvZqbw1+zSe/EE6hmY+OTrKy9mzX197O85i9sbX6RQLg1/gH+R0h7XtKY58EH4Ve/gqpu/RNFgVNOgbvvhjFjkttnl8lkz46O5PYPcOHl8M7riW83aQr89V9JHVIIQf64S2jadC9aqC3KKElXw3v4O9ZQOOlalHhVHHEIRz1Of7SgcQEUio1YOjQ92Go+4PgaqM8uoqt4LnZnGWMpYpQw6fnqZmFrkK4EhZOz7ObCau9v0Pnus0H+cY4NVRnZpEspJVVtEk8QSrMFWQ7j+J6gZOEWjWXVOjlOwV5lCvuOUb5xSaHS24T+xW9jlEFLaFyObFqNKOzOuXIOweOo2CC7AuHIRYzaH5E/LW6TRbfN3HueMQIpL4tkE2/qxo1f32+fBALo3BlezQmtX1Mc7sDa59lW31o6AluYUngpDmve8E90NydtvKQxx+9/Dzff3P8xXYdXX4UvvoAlS6A0ic7C5eXmxnUOIfRz0OFwzrfgmceNMs5YISpFhYoJ8J3vwmnngiX5r4jFnkd2+Rm0bHko5riwv5b26lfIHXN20scCkNLcBbon/OPInkFH7RumthFASXsDtL9B5qjjySqekvD8zloW3wAVwOH5vZfsBRUKGXboCsTf/8ebdd7boHPMVPOeoDaf5LMtGv4QTCpUmFUqEhIfe2+9xl8/DrGhwXhPLQocO02QbRf8Z5mONsA2HJUluONUK/PGfDPyKqTU0b66O24vIVDQqz9F6TZeRO5kUG2gJdhINWcS6uwrEI7EFu9pJQKnFXxxnJyX7Df8S+KbWk3UbDgJ+IXk+fw5WPUws3w1zO/aigUdkGh6gKq2t5lYmALP1R5O2nhJE5/aWvjZzyI/Fw5DYyP8+tdw332J79ttslT4ssvgzTfhxhthzpzEjiEE/PYumD4bHvw7VFUaj5eVw+HHGMZKRiYcfizkp7a6xN++GjOJu97WpWSXnohicQ3hWHF0a3Zi3KWq1lxTcxtIZ+0buHJmYbGbv3v+tDVIYyj+cRQBV5T1eqAcVsG1B1n4w3vxy9sVAc98HTZlvIR1yZ/eD/HEEo1QH2/QlCLB70+xMbU4fkT9v8vC/Oy1UL+U67AOr62OrvFT2yG54okgT1xiZ1bpnh+1l02rTPUIAr1Pl2cQFjti7DHILa/G3EpMPhthdYPUEDkTEBlJ3CABqiK46kALd30Y/TwqyiAhwzcZglKnkjgif92EFAtfu8pptGRwatsKFIzzqiOwhWC443++IWTaeEkTn0cfje2tCIfhkUeM8JE9Qb+r1WRTP12HZ5+F556DV16BY4+FzZvh7bchGIR582DBAsNQiYSiwEXfNkJIDXXG6ykeZTw+jOhhL6aMA6kR9FXjyJwUf2ykzaWOt2WJ2UkB4GtbZm5uEfA0f0F26Ummx7/WYO4O+4Bs685Kox4u3d+CPwx//Si2AaNLqDSZ23DrayFeXKENevUbGyUXPRrguSvsjM2Lfm50+iW/ecu4jU/0HQzrcPdHIR48f88vy5VNq4zcEzPVfLb+i60y4ST0UJehGTPQiHbko8y6HCU3ue9DJL5zgIXGLskTSwbHLkuz4b9X2Hc7yX8pBDvseWx0FDLF37Dz8UC4JW287OoJ/C/SENJoCesUWRXyLIlZ+lJKGroFL4osysh82bZti24U9OD3Q1OTIS6XCLNnQ3a2Idcfj3DYmMcppxjhps2bjb+FMIybmTPhmWdgWoyuyYoCJcndvSWDxZaHWe+GGELZdKBzPVIzd0dncxsKxCFvFYYXJvEy8lCfC6kZrCZtxFmZg78PQgiuOcjKm2vDbIhz2DY/BMMSmyX6e/llpcYLKyIn3+jSCC3c/1mY20+OXiL/2mqNYAJah32RwMItOs0eSb57ZBfLdp9kyXYdTcKMEkFZzhCN9wS6vyul/ZuQCqGgTrsAOeZw9JrPkf5WgqqOr2AMSkY52Y5xQ5vbAIQQ/OxYG99eoHPvp2E2NkoybHDpfJUDKkZmKbQJhXG4qcRj2ugVUmeVs7Sf8aIqe77hO1TSxssIstQT4I7adj7rMu5CFeCYbAc3j8pmoiO2B0KXkkebPdzX0MmOoHHBKLepXF2YycUFbprDOn5dZ5RVxZJqb0JOTn/tlWhkZw9+rLkZHnrICPmEw3DAAXDVVTBunPH8G28kls8ipeFp2by59+8er9DatXDQQbBiReJG1DDhyt+PrsaP4o4TworVZV4vZSDe1mWYMpKEFXuPdyeBKqGBSC1IZ8OHCKFiz5yC1VEUc/yCHCsQqyzb4Lox0ZWJz9zLyu/eiZ204AkY5a43HD74+xQIS377VojnlsU+lzUJr67W+OUJEpsa2bjY3ipRFfoJ5yWCBFq9I2e8BMOSP7wX4tmvNYI9AsLAIRMVfn2CjaLM5OYhsschqz+JPzCjDFG8T+R9uEfRVjqZqra30aQfgk3QshSBlTz3DKTU0GUQu5qPzZJFu28DAa0dq+Iizz2bPNd0FGHSgwuUZCn86oRd19foOLWU+7SNpsdLodCm9n4vrGoWTmvJcExtj2LPD7ruIXza6eeMjY0s6up1n+vAO+1+TtzQwPoYmWRSSn6yo5VbqtqoCvZeeKuCGrdUt1GxvJo5q2uZv7ae8StquGRLE56BmYJDYbTJRfWyy/pXD332GYwfDz/9Kbz/vlFe/cc/wsSJ8Pjj0NYG552XVO+giGia4cG5++7U7C8FWB1FZBQeGmeUwFUwf0jVRlLzY8a7k1Vy3E5vnSN7Gsl4XQCCns101LxOe/UrNKz7I81bHkIPRzdOjimwMSpOxce+WRYmu6PfT52xl0pBnBQpCTy5NEww3P+9kFLyg+eD/Hf54FBRJEIaHPZXP/d8FKLdN3iLTEd8lftYKAIKMkbGcJFScsPzQZ5c2mu4gPFefbpZ58JHA7R6Q7R619BU9wZtde8SDraZ2rcomdfd6yjGa3GXoO77I4QS2cBo9a6lsvVlw3DZOTOQhGj2LKPFu4o233rquxayo+1NOgJbCISb6QpWsb31NdbVP0zYpNdxd2B/kc9eiVTqSYmzj5xCadahu114a1eQNl6GkbCUSCnRpeSH21vRGSzOpQE+XXJLVfT6/Y86AzzdYnw5+14ve/7f13utAe92+Jm/pg5fqgyYL00KSj3/PBxzjOEZaWyEI480Spz7XuV7FHQvuQS+8534Gi+Jomnw73+ndp9DJKv0RDJLjiPa182eOZnsUUPrWqza86Puvweh2MksPqT3uBkTsThGxd0uOr2Jqf6OtTRuuhcZJfdBFYIX9snGHeVQ5Q6Fd+ZG8Nz1IcMuOHlmfG9RZwC2tfS3LJZV63ywUR+k0huLVi/c91mYcx4O0Ozpv+Fx09SYaq2xEMCRkxVynCOzAC3aZrz2SMaWJqGmXef+19/Fsegecla8QMaKZ9A/+imdy/+EHoxtFAiLA2XO1YYXL1K5cu4U1AW/QNgyIm4vpaS6/b04ryDaG9197oWb2dbycpx9DB9S6rT61lHd9j7V7R/SGajsbswaGSEE9TK+F7Iv0/x1CFRGZx9FvjvxLvffRNLGS4ppCWv8vradWStrGLu8mikra7hiazPVIS3qPa4GfO4Jsi0wOIjeEda5aUfiwkQtms6Pk9huED6fkUdiBl2Hzz+Hp5+GCy+EQIz6ViGM5NvhoHX3EnISQpBVciSjZv2azJJjsTpHY7EX4ciaTl7FZeSPvxyhDC2C687fn9heFEFmcf/+TEIoFIy/Aou9YOeY/r8TI+yvpXHDPeha5M99bpaVZQfm8d0xTnIshjxemV3h95NcrD4wjwwTiTFZDkGUSE4/Bt6YvrRCQ03iaqdLqG6T/Pat/gnHFfkKJ89U4qaCRcJphRsOMx/mGCovrNBivme6hLe3zsYS7r21UqTEUb+ewOJfI8Ox69SV/OmoC36GKD0ALE6j83RmOcr0i1Hn3oBQohucnmA1QW0IGk5ATwWOP9Q8xP0kjidYw6rav7G1+XkauhZT37mIjY1PsK7hIYLhyK9ri95FHeZu2hQJ2VJylG0as0q/R1Hmfqmc/h5NOuclhTSENE7d2EB1UNvpYfHoknc7zJ2o2wJhxtl7P5J/1Hdwe21H0l1AXm9PzLqPyAsvGAaMWRTF6FMUz1tjJocmWcxqx4wwimonq+QoskrMtxuQUsPfvgZ/53qQOjZXOc7cvQeFmKyOYjKKDqer4YMIexFYnaNwFwzu2aPasima+kP87avxta1E6kEsjmLDGJI6XY2fEg61YrHm4G39CqnHXshCvipatj1OwYQrIj4/zqny56kZ/Hlq5DvxeMyvULk7TtVRvgsq8vuv1s1emXR0UpPw1jqdpi7ZL9Rz24k2FBHi5ZUaQhihoLAOGTaYVSpYtG1wR6gZowS3n2RjfMHI3TfWd8o4XiJBcyhzkMkqAIu3BW37O1jGx64sExmlqDMuhjhKtwMJ66kL93QGKnFYo7cISDWBcDsbG59Elz1VZ70nmC/UwMamJ5hWfCWK6L/MvqJXmT7GWOHmOusUChKt4vwfIG28JMkmf4itgTDZqsJctw1VCG6pau1nuPRg9pqZ0ef26NVWL7+tHdodSVBCSEqsQ4mPbtliCLWFTZZW6DpsiC1RP6woipEQHA1dh65OcDjBtuuS9qKha36Cnq1IXUNYnLRtfwYt2NuUxdvyJe01r5JXcQmOzMn9ts0adTwWWy4dde+i973rEyo293ik7jeEwQYghIozZzbOnNmDnsspP33n/z3NX5h6DYHOdQS9O7C5Um9E7lUqmDlKsLZeDhKA6+GS/S1YBqjsjsoSKApRt4mHLmFjo05BRq8XwWYxNGGuP0TnnXUaXQHDaDp6qordImjqkny2JczmJklJlmBBhUpFfmqNlkbpp0p6saEwSWQRCgpeXa2xqlbHqsLBE1QK3RJVENOAsYowIV3Bqgx+g7QdH/YzXqSUtPnW0di1BG+oHiFUcpxTKMqYh9OamHKuTY0dKtydaez6sttwiSw3Fwi30upd2y/ME5Q6y6Q5z/BJopSzLGNTM9lvIGnjJUFWeYP8X1UbS729buQii8L5eS7eaPcn7SWxCXiksYtcVWGCw8pvatpSMt8hf8C5uYl5SYQwjAJFSU0irs1m5NCYQVVh6lS49trBz3W0wwN/hScfhvY2Y36HHwvf/ynMGLxojzRSanTUvIGnaSFSxq6okXqQ5s0PUjT1R1gdxTsfF0LgyJ5FZ8NH9Ks8kmE8TZ/ha1tG4aTr+oSJetHDXkKBBqPqyTkqoty6asvpZ0hFR8HXtnJYjBchBPecZeeSxwNsb5U7X2XP4nzyTIXL5w8+60+fbeGxL4fm7bNF+TKNzlG4bP7g96sgQ3Dq7OEJDzVKP49oW1gleyUGbLrKjs+L2PZZEapiFN4/tVSjJDO24QLg0e38aPPl3DXxXyiid7AAlEAHMuxDWJxIKdne+jrN3uXsPMckNHuW0+JZwfj8s8h2TjT9OpzWIhyWQvzhxviD45BhT75aLxlavKuJnSQvaPWt6W+8YC5hHGC+klrBzG8a6ZyXBFjrC3Hapka+9vZfTBvCOnc3dA2pyXtQwvNtPg5ZV88p6+upCg194S+3qkPPSj/zzMS3mTdvaMfsobDQvOFitcJFF8Enn0BmZv/n2lrhzKPh3rsMwwUMw+q9N+CUw+DXN6eu4ikJpJS0VD5JV+PHcQ2XPlvRVvXizr+0YBtdDR/TtOkf3QbGwLNRooe9tGx7st+jWriLlsqnqF39a5o2/p3GDXdRt/q3dDV+Nijp0J0/H7P5MEb10/BQkiV48Uo7vz7ByryxClOKBEdPVfjXBTbuOCVyb6NpJQpnzkm+NDzDBrNG7R6Xy1YZ5DfhVazpY7gABBWNogNqGXtENZreW8bd2IWJPCGFD9pm8X7r4GRQAWjvf5/wF7+jY8dL3YYLDCwfkOhsaX6esG4+zCyEINsxVCE6hQzbGJzW2OX6qUaP+1015Pz7YpEC1eR36BM5dIPum0za85IAt9e0E9RlksWl5lkarwGHSW4uTYFLNi/PUM01UxVkscCkSXDbbfB6Eo0QBzJrFnz0UXzPz7PPwhFHGHONxK9vgi2bom//yP2w9Au44lo46nhwmWxZkCKCnm3428xK+/fZrmszuh6mo/oVPM2fE79UWifk20HQW43NVYYe9tK48e9ogRb6Bjf1cAft1S+ihdrJLu2tgnIXHIC39WvC/njdxGVCrQPMUuXXeLDKz4ctQQSCw/Os/O4sB6UOlS1eje1+jZVdklkZkY32Xx1vJaTByyuT88C0+aAoM/644eY1vZouQhGvQ0JAyb5N1H9dgL/VyIsyWxWloPNs4wHsl7WRl5r2493WvfDqdqa5dnBu0WfMYBvu9q0UFOfRlB9Z3VUSptmzguLM/U2/Hl/YTHuB6NjULMblnzro8Q7/Vhq6luANVgMKOc6JFCYR2oqG3ZKLLxRr7goOq+E9qZQeXtaqWCrNeC4NvtSbOV8dN7RJfoNJGy8maQhpfNCZfFhopPl+UQan5ibfJ2cn77xjvpz5qKOMNgE5ObDXXrB8edxN+iGEEc6R0tCDMWO4AJxxRnSZ//Z2eNlEVdOqZfCD7xiGy09uhYuvTGjqQ8GQ9U9G6VbSXvUi3hZzuSg9hHxV2FxldDV+jBZoJprR09XwAa68eVgdxsVeUe0UTrqGxo3/IOyvi34AoeDMiyxI1hddlyz6WLJ6BTgccPRJUBKl388rDQHOW96BJnvlBha2hfj9Vi9T3Cqr+rSrnuJW+d0kNycX9U9yVBXBHafYGJsb5J6PNRSB6dJpTxB++EKQxy/etYmTupR8ojfEPFOkDgUzW6j6JDElaR2FTb4STl55C61hN0ZQTrDRO4oXmhZwVembXF/2BqPqW+jIdBK0RQqJCTzBmoSOq+nmri8ClWznFIJaG6FwBxbVRb5rNvnuvQYpzta0f0hd50L6hk+bPCto8iynIv90cp1TE5pjJAoz5rK9NdZNmk6Be2/W6u38SVuLnuDqERz22+Q9m7TxYpKGkPlY5e7ACTkpMFwA6k3eFf3970auia7D6acbKreJ4HTCCScY0v5XXGEo8G7dGj9ZuKQkdn+il55JTE3M64Ff/RTefAW+fR0cepSRSzOMaKE2khWLS9RwAUCoSCnxNC0iXsy+o+5NMvIXYHWPQVFsKKqTwonX0LDhbrRg64DtjYUiZ/TpqJbY3quli3T+clv/avqXn4UJk3V+8Udwuno/042eMOcu7yAs+x9NxzA++houABs8Gmcu6+CxWZmcO2qw8N+1B9s4YLzOk0tCLN0hqWmPf35IYOkOnTV1OtNLdl34yIdGwMS5Ys9KxnsraQllIhHIPhkFGsb5f3/NcUx01nJs3jLyWjupK47s6RQJZiM4LAV4gtVxx5VmH0px5vy449p9m7oNFxh8xsC25hdxj7oOmzo0N1q+axat3rV0BrYR6XtUlLEfDmsJ94WXojG48iwWAigXiV/D1wR83N/RQrOuUahauDIzj+n25MUvd2d2jyDuHkCeZeTfqqEcsTLZxisDKTV599aT5/Laa/Dyy4kZDIoC++4L//ynkSB8wQVGz6OtW2MbLqoau7IIjHBQMnzxKVx5PhyzP1RuTW4fJlGtWST3aSeTzySM5o8yjK554oyV+NtW0LT5fupW/ZqO2jeRUkOxuCia/H3cBQsQfWTZrc4y8iou69acic7aVTq//3lkGaDNG+DGa+iXb3PvDj865hsg9kjnfXdtF4EorpU5ZQp/ONXOe9c7KDRZta0IWFw5jCX+MVhcqfGD5wOcfm8QPRz7c5cSQp7k7kvDWHYaKwNR0Hm49kgE4PBHM44kWY6KhI5Z4J4Td4xAJd9lLrG+oWsxsb4bEkmzZ5m5ycWak1CZUHA2JZkHoiq98v02NYcxOcdTln0kK2Qr7YQSvvGVwFGK+RYAIV3nkvrtXNRYxacBL2tDAT72e7iocQdXNOwgNBQ56N2UtOfFJKU2C/PdNhZ7Rs6Zd0aOk42BMMuTyIHJSUaRKxL7mRBFEgImTDD+/8ADhlGRSIWSrsNJJ8GUKUZzx769iqKhqlBRAd//fux9D1UfYUclXHgKvL1o2HJhXLlzzXeE7keiFySBM3cfVGu2YRwIC0hzRq7UA3TWv0c40Ezu2AtQLC5yRp9OVumJaMF2FNWGajWXY3X/n2M/X1cDixdK9j/QWIBebwwkpWbbGpa81hjkjOLY58AZe1n458KwqRDSSK8BUkrufD/MQ4vCqApousCyJpfCmS0RBW0BFBWaVkfJ/4qBwyII6dHLyXUU1njH4NFs6BGSokFgUVzkuiI3RpVS0hnYhj/chCpsZDkmYlXduO1lFLr3pdET7TsgmFBwNhZ1sCfCE6imoWsJXcEdCATZjol0BaqI/d2QdAV2xHjePIqwUJp9CKOyDuwW21OwqVk7c66qdG9SAeGDRCH7CPOf4XVN1awKRdZgWhb086OmGv5auHv0e0sVac9LAtxcmo3CyL1pb3X4aAlrzHFY+HaBm7NyHKbutXNVhf0zUhSb/+yz+GOkhA+6xdE2b07McFEUI1fmr3+Flpb+q0Nfr4urz4VLVeGss4y55cbpETJjL/NziYSmQW21ubyZJLFlTMCeOZVklW3jY+zXnjmJnPIzjEeEwJW7N4mezb62ZQQ923b+rSg2rI5C04ZLMKBTbWLdePHp3v8nK8OvAHdu9XLOsnZu2tDFek9kQ+2ieRbyTdiluoR9yoc3hDiQ11ZrPLTImHePUVGzqBgtqBCpE4OU0LQ6F29j9AaXkXBY4bBJ5s4FiaAjy8XA89WiOJlUeH7EJoldge2srvs7m5qeoqrtHSpbX2Nl7T3saH0bKXVG5xxNec6xqINCJYIc51Rsas6gfdZ3fsH6xkdo9a0lpHUQ1Npp9HyFxIxBntqruBAqdksudkt2v2RxG2pCtxhOFC5SKrhcnWC6UrQ2HGJpMHbe0GcBL01airzxuwlp4yUB9nXbeWJCAWW2/hcwm2BYjJpOHXaEdJb7wzzY5OHjrqCpL8L1RRlYUtW4K5bEf6RxBQWxc1D6oqpw8slGQ8fq6uhGjxAwapRRtn3EEXD99UZFU5GJ0sjTzzUE6YaCEPDGS0PbR8zdC/IrLsaVN4+BZ5FqH1r3WEV148rdh/wJ3yF//LdRlF6Ruoyiw7vbEiRyrih4W0z2uopAq8lii84+VcAH5VqxJHE668CXHWFeaAhy1zYfMz9r5WcbuwaVgOe7BU9dYqcoRvhIFTCjRDC7dGQb4j20KMxAJ0egzc6aJyfhbeqfy6BrgrqlBWx5Y0zCxynNgv3HKjFF/AQ6FY563O4sSid+j8KMfXHbRpNpr2B0zjHMKLk6YrmyN1jHxsanCGoDu8frNHqWsL31TYQQ5LlmYrMMzEMxBPHWNfwLT6A3L6YzsL1PTyS933gzmAlt6VKjw7+VVu9avMG6mP2KorGPkht3Ri5UzhFjuF2dw73W/TlSLUFJ4Pr9VGebqXEve9rjD9qDSIeNEuSgTAcLp5WwsCtAZTBMlqpwRKaDprDOv5u6eLzJgzfFvuWevTWH4zsfK2wqVxdHLmNMijlzzI3be2/jd4/Wihl0HV56CbZvj52YK6Xh0amsNP7/0UdG5+if/hR+97vBjWz6kpUNf3kArr/UOF4yn42U4OmKP24ICMVK7pizyRp1PIGuTSDDWJyltFY+HX/jiCgI1UHh5OsjitIBWB2FFEy8mpZtj3drw/QRtouKTtiUUF1kskxW72fl9P7/2jFOHq81aURHoccsvmOrj3KHylXl/Q3ashyFt69zcPFjAVbUyH7vhAAKM+DuM20j2s3XH5KsrY/8efianKz69xTcJT5chT70sEL71kzC/uQu6VubwRuCLAd0BSJXYUkEF435Csu8H2K151Nu0rCu6fikWzo/8mtp9i6jOHM+DV1fRCk9lugyxJbm/zJz1PUIodDY+SXJVegZuG2xBe0au76ipuMjtD6aNQ5LIWNzT8BtNx9+KRQO5ot8vpDNUb9ZF6jjOEhJXqOmXZrzdLfvQi2r4SDteUkCRQgOynRwYX4GJ+e4cKsKY+0WflGWQ5GJ5nLJEusUFRiN7u4am3isOyYTJxrdoWNV3IwZYyjbgtGQcZJJ0akeQ+Lrr82JxIXDvV2pAe64A+65J/52x5wIz70Nx59qGEkAzm63t5nFSLXAlBnxx6UA1ZqBK3cOrrx9CftqTWiqdG9n6/u5CxzZMyia/L2ohksPNlc5RVNvJGvUCdgzJxrdgWMikJqf1h3P0Vb9MoGurQndkTpdCqUmhFDPvKD3//OyrfxhshHXMdOUMR6/3+JFizBnu0Xw+MV2fn+Klb1HCwozYFKh4EdHWHjxSgdlOSN7uYz/rgo8dS4aV+bTvDY3acOlh2e/1vj72TZslv7vsyqM7+Zpkzs458xTEU7z/YM03U+HfyPxXk2zdwVNcZJoQ3oX7X5Dr6kzUEmyhgsItrW8GLUDen3nYna0vdnPcAHwh5vY0Pg43qC572QPl6sT2FsY4W2luxZLdP+crYwZkuECMM1mLkVglu2bVXUkZDK+sN2Ijo4OsrOzaW9vJysrhR4HEzQEw/ykqo0PO/yEMSxBmwD/ML+jxRaFDk3i6/PRldtU7izP5aDMYThBKysND0xbW/Qx3/8+3HWX8f877oCbbjK/fyGSz4QsLoaqql6jJB5SGsaPxQJLFsFtt8DKr+Nv99IHMHOI+TMJ0rDhHkLeHcRfxhRKZtyClDpS86JYs+OWKvcQDjTSvOURwoF6jDO4p1YnHj0LuY7NXUF+xaUoFnOlnatX6PzyR9GfHz0W/vLgYEPhF5u6+N0W36DZ9dx/m/Eb9fD1glxmZu7+jufTH/SzoUGa1qMZCoqA1f/npLpN54klGm+uDRMIw+QihQv3tXDkZCVhz1Mg3M7qur/HHZdpraAzFL+qrzjzAMqyD2N5zV8GGReJMj7/THKcU/o9pul+VtTcjYx6qyjItI9hUuGFCR9vm+xikd6ER4YpFA4OVgrJFUPPTfRLnUOqN8e8uXUg+KRsQkLhqJFgKOt32vOSJNv8YfZfW8e73YYLGBfQ4TZcAOrDOidkO7h/XB5/LM/lmQkFLJxWMjyGC8Do0fHzWO6+G2q6xalefNGcR6OHodjP9fXw1VfmxwvRa+jsOx9efA/eWGj8f9DY7td83Y9G1HAJerbTtPlfhLzbMbMc29zjCHq2oagOrM5S04aLobB7L+FAjwx5IgXJOj13vkHPNpq3PmLaAzNjtsJPfw2R5CemzIA//GPw45+3hSIaLj3kWwVT3eYvZ/6RsAZSwOXzLSNiuAA4u/Nsy3IUbjzKyvvfdfLZD5w8fKGdo6Yk12rEqpgzaP1as6lxobCHrsAO7GoOQ0twV+jwDzaWWn3rYhguAJLOQCXBvo1PTTJOZHCeOo4L1QqOVkrIYWiNYfUwbP8Utr+mcH19ccyxv8gr3u0Ml6Gy+9967Kact7mB4C68/r3Q5uPOMbnYzCbHDoWFC41KoHj8/Ofwr39BQ8PI1pT6hnYHxuSp8MTLRpuAh+8zqot6Hr/6BjjlrCFP0Sz+jvU0b/kXiZRBBz1baPFsQQgrGcWHk1l8ZMTGigPxNH+BHu5K6FiRkQQ9Wwh6K7G7xxHyN9LV+DG+1mVIGcJiLyCj4EBcefO6E4Rh3wUKj74kWfSJZO0qcDnhmJMhvzDyvP+01YsqIBxhqjrQHJKMcZj7LtgVmOwe2aqhZDlphsqqGp1Hv9TidoYeiNrdRbvnd7yxJ81I/XsSv/+PQdikym5HYCMtvgSVu6My+E0JaV2YyaUJ6V3YSMxT8IXexOtaDZUY+kolODhWKeVQpShhw+KDX8DCP0J456Uvi3OmuPnsnhqqp/W+l4WKys9yiznIObItT0aCtPGSBOt9QXakoHHiUNCBp1u8XFxgUmFrKJgxXAAWLzZ+jxsH27aNTLPDnk7SQ8VigSuug8uugZYmI88lJzcxD9IQkXqYlm2Pk6wxIWWIzrq3kXqY7NLj4473tn6d9LEGI/C3rQKp07T5n4ZGffcCEPbX01b1PN62FRSMv2KnAaMoggMOFRxwaOw9Syl5vSkY0XDpIZHUzYtKHWTtAtHJZBBCcNPRVg6bpPLAZyEWVZr7vO4+08qS7To17ZJsp+CE6Qq/fyfE1ubBBpAiwKrAJfunfjnoCJgTeDQb8kuk6WNsdNy2wV3PrYobM2eSMc48/9W284pe3c9XVIefR/QtbJadXJFAafTzF8HKxwc/7l+vMu/Ecn61OIRvSoBxFjul1uhdzWuWwPbPwF0AM84FZQ+zBvaw6e4evNNh7i4hRxHYFEGHpg9LOOnjDv/IGC/jx5sbp2lw441GKMes4TKUfBeLxehrVBzbZZoQigIFI9udtgdP82KkyTvQWHQ1fEBG4YHdyr3R0TXvkI/Vi0TX/bRsfQSkRqSlKNi1iYYNf8WRNRVX3r5YHebeZ0lkj0tfdKDUrrDZq+HRoi+EORbBkvYQsz5rYVaGUXV0SK415VVEmpS83RTk1cYgPl0yO9PCxaUO8pJI6BdCsKBCZXOT5IvK+GqtY3PhmKkWjum26aWUCCF46EKV6581qqlUxTAYwjrkOI1Kqor81Bt00ZJiB2K1ZBEIN5nZ49AmBIBAVRwRxfRyXFPZ0fZ2zJyXDFs5NovJsjlgi97JK7rhzY00+09lI3vLPOb2EaVr2wZfPwQtG8GeA+OPhrwJ4G+NbLj0oIdg6dVWvr0outGy9X147jzw9mla/cIlsPflcPIDpl/WLidtvCSB1eR1rk2XoEv2clhY7k+9QNDnngCalKjD7R2YNcvonBevQePatbB+fWIel2QNF1WFsrLeJOFvAN7WpSnak8TXupyMooNjjrLYCwmGOkiV90UPd8U1iML+Wrr89UbTx/wF5Iw+LW6ISxGCmRkqq7u0qPfECrB/tpUfj3Nx8lft+PXB989WAe1hybJOY2Ha5NV4tj7IdeUO/jI1I2UGTI1f48Sv2lnVpWERxrv7eE2An2308PDMTM4uGd6qj0v2s6BLyUsrNB77Msz6BolFgUMnKdx0tLGofbRJJ6TBzFGCI6eo2IZQxhUIt9Hk+RpPsBoFC1mOCeS5Z2FRHLhs5m4sRmUeRFXb24RlKg3qyCjCwoT8s1HE4OXPojgZlXUINR0fRNhSIBCUZR+e0PE+0OtjegYV4D29jrmKYbx8cju8/zMj5U5KY8Ol95k/XvUX4GkAd4R7g+2fwqNHMegrLzX46p/gaYTzXjB/rF3JnuE73c04M8FuzWYMl2QuHW2a5EtPMIktEyQcNh8+GYlQUU4O/OAHsGSJIV4Xi1AIOjpGZl5DJOxLrAQzFr72NXHHZBQsIHVho55+RGYuKcZn4W3+nM66d03t+7tjnTGd+YqAy8ocHJJnY/VBedw03sXMDJWJToXzS2wU2wS67P9qe7w5f9/h5+HqoXu8wPC4HL+0nXUebecxtO7jBnX41opOFrYl0zQR9hurxP207BY4eZbKjS8F+b9XQ6zvrlQKavD+Bp1vPRqkpl1yw2FWfnKkleOnW4ZkuDR7lrO67l7qOxfRFdhOR2ALVe3vsLruXrzBWpzWIlzW2P3RrEoGua5pTCw8HyXJ6huBgtNqTnemIv9MMuzR6/WLM+czOvso1AFzsVtymVh4QUI6LwCV0hPz3NWBHdLIg1n+KLx/CyC7HZhJXrY6a43voz7g5vCFi4n5lV//IjRvTO6YI03aeEmCAquFua6hZYoPJNklpDk8Ao3i/vjHoSfFHnjg0PNHVq6Ezk5objbmVBBDw2TFCjj/fKOtQHY2FBbCLbfELvfexUgzYlPCnCJu0LMJX8f62INUJ4jUncdmq5z60tX4EboW3wC/pNTB2cXGXPtetCzCeDcenJFJqcNIOB3tUPnVRDdfH5DH2oPzOaPYQX0wViAA/rTNl5SC6kDeaAyyxqNFDHNJDCPrj1uT8y5MLlLYt1yJqnWjCLhgroW31+m8ttpY9fpWKmm68fePXwwx63c+pv/WxwF/8fGHd0MEowhgSqkT0jxo+uDPqCtQRWXra0Qqr9d0PxubnkbTA1Tkn4olStWRImxMLrrEaFdhK2ZmyTWUZh2G01qM3ZJHjnMa4/POJt45L5E4rYUxx/QQCLfS0LmYmvYPaez6alCysBCCosz9mFX6fSbkn83Y3FOYXHgx04uvItOeuHqxPUqjy77YUJASPr6NlHQJ+aNax/zqTcyr3sQ5dZU839VOW7WkzUQK0ns3G78DnbD47/DQwXDvbCPUtPWDke/vFY102ChJnp5YwKFr66jZxYm7o23D/BGGw4Zuy1BQFCOcFO+sF8IYG6lNwK23wsyZ8Y+l6/CnPxk6M329LS0txut4/nmjJ1JeisX8UoDFnt+nbDkydncFQrXhb18dd38dNa/gzJoS8Tlv20patz0adVtn7r74O1YjNXNGq9U5GlfuPgm3DpB6kKBnM46syM38elCE4LHZWRxV7eee7T5Wd4dkji+w8cNxLg7MjR7j/7A1hFVAKMrpJ4ENXo0PW0Icnj80Y+7lxiAq0QUlwxJeawwS1iWWiM0NY/On021c9FiAHa3Gi+kxiHQJC8YpfP8wC+c9HIibANtjq7R64eEvwjy3LMzz37YzuluIT9OD1HcuotGzdKeeSqZ9HCWZB5LpGAtAQ+cXRE+1lWi6jxbvKgoz5jKt+NvUdy6m2bMMTfpRhZ0C994UZe6PVe01ei2qi5KsAyjJOqDf3nJ8U2jzrY/6qkRII299E0r9FkKZDjomliCjNKetanvb2AaBRKeq7R3Kso+gKHNev3GKsJDtNCm4GYN9lXw26Z1RPw8F2E8poGWTkeMyNCT+Ao1Pcjp3noNbwkF+29bA4kVhnMQXGGzbZnhf/rUAfH0q2BtXw+r/wD5Xwkn3EbUx6EiRNl6SxKUofDGthIeaPdxb30mdCen+VCKAiXYLs53RL9op4a23oH2IPTGkNBe2ycyEyZONcFBfhIA1a+DLL2HevMjbgtFf6ayz4NVXIz+vabBxI9x8M9x/v/n5jxDuggNor47dQymr9ESEYjFlvIT9kaTWDXdya+WTMbf1tS5BseaZNF4E2WUnYXOPx+IY1X1c898HqZsLo6hCcPloJ5ePdqJLQ8LfTJ6KWY/KMUvbGeNQeGhmJofmJWfEdIb1mCohYCy/IZncxbcoU/D8t+28vFLjxRVhWr1Qnis4e28LR01RUASsb5AJe3I7A3DhowHeu96BIMTGxsfxhurpayx0BirpDGxjXN6p5Llm0BHYsvN5TQo+6yrns84xhKTCVGczJ2RvoN2/mcKMuVjVDEbnHMHonCN2JhADEArAsvegrREqZsD4WQD4Q800dH1Jq28tUoawqTkI1O5E2v6vrmDJFka/vw7F7yeT7kRkp40dx+1F66zBFUU928udvzWq2t9BVezku2cn+M7F5yClkFf1KjyEB30rBGBF4UilpE/Zc7IYr2jNTU39zsGed2thYQdHmjBeQl64f47xu9/euyf/1T+haBbs/92hzndopI2XCOi6zpMtXt5oN0Sxjs92cmGeC2WApoqiKHy7MJMzcl3MWVUb96KVKnqaQN4+Omf4e62sX28kxybSKToSs2cbPY+i7cdiMUqsl0ZIWpUSnnvO+FFV2G8/+N734Nxz+4eibroJXnst9jw0DR59FO680zCWEmHb1/D+g9C8w7jtKJsGR10NhWMT208U3Pn7421dFlWczl14MDZXd7xd2EDGD7f0Wyi68bYsBRk/D0sPdZfIC7U7AB8ZoTqw2AsQQlAw/gqaNt8f14PUD2WwoVAb0HivOURQl8zNtrLXADXcRHQxDsy18o8d5nJatvt1jlrSzk/HObltcuKVfGZ0WKyAHpb9NPjrOiRPfxXmw40aQQ32Hq1wwVwLM0YNvr112wTnz7Vw/tzBl28pjUqiZO6lGjrh4006U4s/H2S4dO8dgMqWV8lyjN9ZSVQbzOD724+nMpiD2r08v94+mb/Vz+PPFWuYOCC6K4Qwbmae+Qt8+Azofc4tVya+865m3ejKfoaKP9wMSFThQJN9dEy+qqL89WW9++7+rfqCVLzwJVIVtE030Y8CqGn/iDzXTFMaSYngFhZusszgzvBaWgnuDHtalrjJfaKYqU15fD3GyoxzQHWANoT0q2COjmd85JsB74Qw4VwNS2vsMFbT2vjH+fxO2O+6Xet9SbcHGMCXXQHO29w4qLTZIeDJ8QXsP0DFdlsgzFeeIP9u6uRrbyjpbhuxsAsI9JnPLKeVX5blMD9j6NLScXnwQbjyyqHtQ1UNDZj5840w1FBPuZ7y6u98B+67z/h7xw6jp5LZLtjLlsFeCajmvnonLH8r8nPHfhf2PcX8vmKga0E6697C0/wFUjdei2rNIaPoMNwFB+w0ROrX/Ymwvy72zoSFsr1+N+jhlsr/4GtdEmGDZFFwFx5ITpnxHkg9jLdtBZ3176KZMGKE6qJo8vex2PPwapLvru3kiZpAv5uB/bMtPDIriwmuxIXUgrpkwsctNAbje0X68sisTC4YlVhl0PnLO3iuPv45OGO5m3PnWLjhMCsrqnWu/k+QoNabo9IjLPeTIy1cPj8x7+p1zwT4aJOekKBdD2fPUThz73/Eld4fnX00rb61tAVqOW/TmdSGMtEGpFAKJKqAhfvnsnfWgNfwz1vgy8Hfp54pbz1jP9pmRjY6ijL2J8NejhoWZPz8OwjvwG7VvfsKZrtY/b1jTefbTS68OGYy71AIS52lsoXVvnaaLinG998MFItE6gKhGIq59hwItCV/DIkEAZ8/Wk3TYYM/wwl/z2HG7wtIRWLN97dCzrih7WMo63fa89KH2mCYMzc1RrzA+SWcvbmJz6eXUGazUB/S+OH2Fj7sHFq3WzPiTG9PLiKEoDGsUWJVmewY5lBRX045Ba65JnrHZzNYrYYBc8EF8NhjxoWkJ4zU0026rAxqa82Fl3qMnwcegAkTDAXgl19OzChyOuOP6WHZm9ENF4C37oGKuZCfWBVCJBTVRnbZyWSNOs7wXghLt1ej/8KQVXIMLTFyVgBcOXOiHiO16HibF+PInIan+XOC3mp0zQO6ue+G1Px01r9HTvlZnL2snXebB98ELOkIc+jiVpYsyKPEntjtnk0RvLRPFscsaacrHFv8vS8/Xd/FeSX2hLw8dsWcYJ4vBI9/qbFsh8bGJgho/U/fHkXcP74XZlqxwoIK80bbFQssfLAxuSpEX0gz0TNIwR9upChjHi80LqcqFFnzRHYvkH/Z5uPR2X2uWbVbIxou0Hs9HPvKUtqmlxkJPQNo6FqMECpFm/1RDZeefdnbvbirWvCUxw+XVJPFQo8f3dfEOKuNo50ZOFOoYG4RCvuLAhp/VEBVdzmyHjZeX09IZiiGC3Tn8UjJvGtG8cbaLf2eszeqTHgod2gH6INJCZ9hI11t1Idbq9tiXtg04OdVbXRoOqdvbOCTJAyXnq9itiL4blEGN5ZEtzZV4OgsBxOdNqY5rRyS6RhZwwWgqAiuu25olUJ+v9HY8bHHjL913TBaiovhnHPg9dehujrxcmZFMUJFiRouEyea73wN8Mlj8ce8e6/5/ZlAKFaszlKsjqKIbmxnzixs7ujigYrqJqvstIjPuQsOStU0dyL1AM1bHsDfvtoIN5k0XAx0vK1f8X6zn7cjGC5ghGOagpK7K720eCSeBHtz7JNlZeWBudw83kW+SaGmuqBkcXtiRvvBudbYhosEm0dBkUbp9opa8Iein76qgH9/kdgc9ilXuf1kK6oSce2PyfQSc0uCIqzkOKfyhW9vlBivOCzh+foBcZB3noi5bwGoIY3MzZFztkBS3/k5tdUvm5qrxdP/XNSB1RTzNHvxBHvzIeP5GwfwS47lEa/kia5WftVazzG1W3jf12XqGGbpqoev/zW8C79AYO1SKXrPqPBy1KoUv+Nm1q2F2JtUUuF1sWdBduKFVykl7XnpwwcmlHM/6vTzZLOH7UEtqfJmAZya4+Tv44w7ASklLZrOPxu7dlYp9Pye67bx17G7QVXMnXcaein33ju0kE9f40RK8HrhV78Ce5Lhr2S1W37+88SMsY6G+GOq4uuqpJqCiVfTXv0inubFfXJYBPbMSeSO/RaqJfL7anUUdifWpk5XppckPxMZ5rEaL5Yo/YvA+E7ctcnHi88aXoj54xSuOtDC/HHmvBKj7Cq/mOimzKFwzRpzi1JrgtWE55U4uHmDh/awjPxOCMit638DEusbpUn4ojLx9/S02cb78tyyMGvqdASSjzfLmLkwVgVO38tOQ+dYuoKxmoLqZDunGCFMpRid2F6eoIS6ji8oydrfeKBhh6nX4GjqoHNSNO0WSSDLnAcxmN3rZW3GyV85mBqyUdARgNZnMe9tagFeKflpcy33F45mH3sCntoYbH7LCA+NBOPvzWHs49mUvOdGyNTmRs65fNe3E0gbL30wczMXkPBksydpXRYdeLXNx21hnVyL0WL+l2U5nJvn5qlmD9uDYXJUhdNzXRycmZjLetiwWOCGGwwF3Q8iKU8mgaYZ2jF33gn33GOULpvtoTQUfvc7uPji1O93F/hQhRDkjD6d7LJTCfkbQA9hcRShqPGNwcJJ11G/7k/oodYRmGkvrbqDx31784RvL5p0N1p3weoktQXVEb8NQLiPnbK4UueLbUHuONXKyTPNX8rOLLbzvbVdUUun+1LRnWPzdUeIJ2oD1Ad0yhwKF5c6mJ4x+Jhui+DFfbI5cWkbPq1PybQEBGTXWchoSeyyKyVoukRN0I1SkiW4/pBeQ2lZlcZFjwUjGjBCwB9Ps5HjFKjiQDY2VUbZq8BtKyWjuy/QrEwrbzSFYmjoSMba2qjpeI9c11TslmzIMndDprlin8edFYUEMx1YO/0RfQkS8Bdm4SvJAaAdO7dxFF0Y+9VNBB4k8M+OZu4tTE0eTChVrZlMUPiF4XkRqRCO6YOrAI75Y0p3mRTphN0+TFxehS+eFAmQoUDnENeqtyYXMTPFQnfDxubNRolyR8fQq44G4nZDV5fhDbntttTuuy/Z2fDmm0bScKL8+UzwdcQeM3oGXHJXUlPbVUgp8bUuo7Ph/RjJvwKhWLFnTsbfvmpIx6vWMjmr9ULq9IzufIi+F1WJGXe2GhRULO8vhmdT4aPvO8hxmr9I/36Lh59vii4WJ4DJLoXD82280xRks083pP6lsdCHJVxT7uCuqRkRbzCq/BoP7PDzj01+PGGJ3auQ3WDF2aEmtZgowJFTFL69wMrsst5FV0rJYtnM21otW+lCQTBT5HCcMoqpyuBclDavzi/fCPHxZh1fyAgrHTRe4dqDrezVZ7/NnlVsb30NvxBstRfgU6xkaj5mSAdT8s/CohoLY6VPY/InLTH8bZIbSz7jzLx1lGQeQGn2obB1NfzukpivVwpY8ZOT0Byxr5GO9Y1Me+ZTQ++mz1ImjTp6Nn7rILrGFbKKIv7GQWgmBOMi8WLJWMotQ79eVy0y9FP2ZC75CMYdkpp9DWX9ThsvfThiXR3rTUj5ZynQMUTj5bNpJYyz7yGOr7PPhhdeSL3h0kM4DMEgTJsGldHu+JKkqMgoq77mmuSF6T55DD6OnRzLBX+Air2T2/9uQMCzjeYt/0JqfnqNCIlQHOSPvwybeywdtW/S1fipqTLrSJzTej5LQ2WDqlJMIyGv2kZebf9FRAA3HW3l4v3Mf5+klFy7posHI7QF6H318RPqfznBxS0ToisLv7IqzI0vRdexERipW7puTmVbAH89y8ZRU1SklDymbeV9Wd9vnj0Jw5co4zlcjd5bSNMlioAG/LTIIJnCShlOhDCSPl/RKnlZryUsQEiJFAI3Khep45mv9NY//2O7h++v86Kg9/FmGObpAvcO/jjmbSwCcpxTGJ9/hvH07ZfAtshaRRJonjeV7cdPj/ledCp2ns3bhwlrNnHe8y8wqqE3vLujrJT6Y2bQUV7Ic8ziC8YylFyP8RYrTxaPxTpET7iUcN9ehuDbrk54TQoFFvwAjrkzNbvb7aqNtm3bxm9+8xvef/996urqKC0t5Vvf+ha33HILNlvvhWfFihVcd911fPnllxQWFvLd736XG2+8cTimZIoyi0IcQXVgaIaLAkx1WBlrS+4OYMRpbk7ecJkxA9ati71taalRifTAA+YNF1XtvtqbuNx/8YWhHzMUDvoWfPwYMZeXTx/bo40Xu3scJdNvwdf6NYEuQ3zMljEeV+4+KKpRLpxdeiIZhYdQt/b3EEEuPhYbwvksDkUSDDOJBEtAkN0wOGFdVWBTY2JfSiEE987I5IxiG99f18VGb+/2DsXoQxS5N3Z//rTNxw/HuXBG0ew/dqrKPxeG2dIkB5UuqwKynXDz0VZueTWEpsfXiZHAT14M8skNDtZbW3lf1u98vIeeV/KovoXpShbFInK+xnY8PBHeyiZ683/KcHKuOpYq6eV5WbtzvZfdi7YHjfu0jVhRdjYSvKbcBd7neaRpNl95jT5GJdYuzs1bzTl5q7AICSj9ewXd+CD85TrY+NWgeYkDTyHvWz9Fetewo/1Non0KH2ROxqPYWD5rJstnzmDsjiqyOjtozcmlprSUnLCFZS122nEw1CTVLeEQH/u6ONKVoDZUH4Ie+PIf4GvdQw0XAB1qB39ku4RhMV7WrVuHruvcf//9TJw4kVWrVnHllVfi8Xi4807DZOvo6OCYY47hqKOO4r777mPlypVcfvnl5OTk8J3vfGc4phWTsJQs9SbXMC0RdOAno7KGX1wuVVRXJ2e4CAFnngm//nX0MYoCV11lKOfecEPs/U2ZAhUVxv/3288wjM49N/Y2RUUwNgUCcivfJe4ytn0l+LvAkbiw2e6CojpwFyzAXRDdr60FWxM2XAC+DCaWM+BSYKc9IcHdqlJYaUfVBn9vJJJt1hDHLfGyrDOMXRGcVmTj+jFOJrmNS1xdQGdJewhVwIIcKzlWw0NwdIGdNQfZ2erVaAjqfNke4gfrPabn2alJPm0NcXRB5JCCzSJ4+EI7P3w+yOLtOkp3LyZNwtg8wT1n2RhfoDC7TOGppRpPLgkTjPN184fh5ZUaO+bUxizLFhgdjc9Txw16bqXeyt3aegaWHdTg48/aOixxFvuHtc3sTQ6KYuTtHV+YyfyM1/HrCmGp4FZCA3LidXKcU3v/tFjhJw9AQxXyjYcItm7Dl2en9aB5uAtnky9AJ0S0712nYmebPb838V4IKsf0N46/6nLiwUYqqmsU4A1vZ9LGi78dHjkM6lfswYYLgABranKXh8ywGC/HHXccxx133M6/x48fz/r167n33nt3Gi9PPPEEwWCQhx56CJvNxowZM1i2bBl//vOfd4nx8n6Hn3Y99RE0BaN6KAzYheD20Tkck72bfPpmSDbUIiV861tGUu4fI2R3qaphkNxwA1x/fa/eSyRUFQ47zBCk67v/3/7WaNYYzQPzu98NvRkkwNqPzI3buAhmHTX04+3GyCRCRpoUPOLbJ6Ft1h2Ux8cNIX7ycgi7V8ESihxqkkhqywM8RRi1pSdBVvJAlZ9/Vft5fFYmz9YF+W99r+idXYEryhzcMTkDR7fHpMKlUuFS+dM2rymdlr7441w38t2CRy6ys65eZ+FWDU2HOaONJos9NzFjchUu21+YKos2WgDo1MuuuN2Kt8j+VVWb9E6e17ezRkbO4ep5JeE4xnoXYR7Wt3CFMhGA4swFtPnW4VA0Bnd2EjitxWQ5Bpf2d2QG2HKUE11O7H6kjtb2Oqra30VVogsENlvcMb/bIU2hM5gawwWM97JNTz5s/t7/Qf3KPdxw6WZyavQ4h8yI6by0t7eT12ch/PzzzznkkEP6hZGOPfZY1q9fT2tr9AqIQCBAR0dHv59UsCUQTjKVKzpZiuDHJVl8pyiTO8pzWD5zFOfmJ955d5cyejQccIDhJTGLqsJJJxlaKnfcYVQTlZb2Pm+3w6WXwqefQlYWfPxxbBE8TYOPBhgQQhg9jCZO7P0beud5001w2WXm5xwLs2lh34QrUwwCXVtoqXw64e0+DI5ng2au468A9suyMMqhUr1Jxd1uiWq4AHTmh+koMs6dvktLWBqhn3OWd/YzXAACOty3w8+Zy9rRB3y2XVqUEucYzIhQdTQQTZc0dBo5JtkOwdhcZZD31ax2jZSgOkIETczU0ucSv0pv43faatZGMVwS5RPZSKs0NFRctmImFJyzMzQkdjYxAbetlIkF5w5+vYEaNjX/B11G8njraHr0hGo1jnHlD6dGz6T3eDAmSsKuT9d51dPBve3NPN7ZSm24/+sJdMKyh2J22NgjECpkFMPsC3f1TAxGJGN006ZN3HPPPTu9LgB1dXVU9IQBuikuLt75XG5uZCXA3/3ud/zqV79K+RwzFJFyaf8OXXJSjpMJIy0sl2p++1s48sheWf6B9Dzek3k4Z47RP6jnueuvNxJmV60y5PunTDGqf3qwmDgN1QimZXk5rFhh9Dx65hmjgeT06UbbgL1TmH8y+QDYvDj+uAn7pe6YuxlB7w6aNj+QlIH2jG8mKrqpRF0J3DjeqGRp8fapHkHiy9JoLwoRcOkIHTJaLXTlhBGyu7okwr4gcodnHXi7OcRbTUGOL+zNxZiZYeGDllDckm0wclaOzLMyPk7LgsWVGje+FKS+0/CaSAnizRBHT1G4dH8L00oU7BZBUabAphI3bIRFo+aAtaauV3sL4zqqSck/tU3oJN60MRaf602coBrK0lmO8cwq/R6t3nX4QnUIYSHbMQm3rSximHx76+tJH3dUsB2bHiYYRWxEpPRVGufQ6e4sNocCbAkFcQqFuXYnH/q7uL21Aa+UWDDOq7vamzjDncVPcoqwCkHzeggPoV/RLkUY/YukBhklcNHbYNtNIuMJGS833XQTd9xxR8wxa9euZerU3thmdXU1xx13HGeffTZXDrVHDnDzzTfzwx/+cOffHR0dlJcPIRGwm2OynfxfVWyF3WRQ95Tcllgcdhi8+CJcfjk0NfUmzKoqnHqq8buyEgoLDQ2V004zWgL0RVWj9xI6/ngjJBQrbHT88ZGfcziM8NS3vpXkizPB3ifAO/+AcIxcj1GTwZ0zfHPYxXTUvNFtuCS+KNTqWaYrjH470YVXkxz5ZRsr7WE80wWZzSpBh6SzKNyvorrVERrSzbUKPFTt72e8XDHawV2V8cU4VAGFVoW/T4+dA7G6VufbTwV3yv33RJikhLfW6by1LkimHS6Ya+HaQyycMkvl+eUasSJRs4+txa/GDy+5sXCAYni8VshW2kltTp8Adsj+3hFFWMl3zwJmxdw2GG7HFzYh/hgFCzr7eLezyD0+4jnQGUitDMWRDjd/am9iZbDXCrEDfbV7+34iz3s6EAhuzi0i4W4cAlQb6KFd58wVVljwQ2jfBqoVJhwH08+CKLqXu4SEjJcf/ehHXHrppTHHjB/fG9esqanh8MMP54ADDuCBBx7oN66kpIT6+v7yzz1/l5REU1UEu92OPVlF1hgUWVUuKXDzcFPyAnQDsQFj9pSqonicfDLU1Bihmk2bDM/JaacZSbFD5frrexssDvTsCGEYL9deO/TjJIsQcO7t8OSNka8m9gw4b3ADxGHjo+fgtYegrfvin5UPx14MRw+PP1cLdRHo2pj09iVKJ6spjmnA5Kjw7rwcfrLBwwctffJOXJJml967QPVdqIZ4X6ABW739b1emuC3cPsnNzRs9UXNfMlS4rMzJTyqcjLLH/n7/7eMQuk5MY6QzAA98HmZtvc6vT7Ty2RaNus7ITs6SLMiZ3oqZG/mfWKbhFt0Jy9KfUC5Pj8p3LCTwuWwiQ7NwgTIuoSKEkDZU2X3BQYEOHJlFfCgbUOi1a7tCFpp8qckrzBAKYy1WPvZ7GGguxmqAIYH/etq5PDOXwhlWMkuhs8bkQSVoASNMk1Li1f33nUIIlv0LfpJAc/iRJiHjpbCwkMJCc7Hr6upqDj/8cObOncvDDz+MMiBnYsGCBdxyyy2EQiGs3Xfp77zzDlOmTIkaMhpuflGWQ1BKnmj27rzMDsUTEwQCUuL8JnhfwPCmnH566vc7ZQr85z9w3nmGR6enuklVjZDSc8/1VhrtKsbtBVc/BG//HbZ+bWh8W50w8wg48jtgd43MPO6/CZa+2/+xjmZ49i+wZhF8/56UH1LXzFffROJM5yreDk6O+rwAbp7g5vHaAB+1GN4Bve+TYFbDLmF8EayKH1e4mOhS+cNWL192GEtWmV3hunIHV4x2kGNVTClfd/olH23STa0XUsLHm3W+2qHzzGUO/vpRiJdWajtDSHkuuHBfC1cdaOHbJqr/LAjGiV7/vgM1obB4OW4siH5l1NF4R68jDzvHq6Vxx+6cn2ou90+gYrTS7Fl5jd82NYtJBecx05LLkbKET/VGmmWALGFlkdfCVgJDSgMQgEsIuqTOmlAg6VYw7/m6uCAzlwN/Cm9+P7HtU54jk+CL8DbB5ndgwtEpnkeKGBaRuurqag477DDGjh3LI488gtonX6HHq9Le3s6UKVM45phj+OlPf8qqVau4/PLL+ctf/pJQtVEqRep62B4I81Kbl1favKz1hYf0JVg/q5QMNd3/0hSVlXD//fD++0b+zJFHGqXUo4enRf0ex5J34YGbYo+58GY49MyUHlYPe6ld9UuSCRmBUW10fuu5LA6XE8kCUYBX987i7BUdeEY4qTHbAk1HRL8hawvpBHUosImEW3VUt+sc9bfEmrdOLBC8cpVRZeMJGkm+GXZB2KrzcLWfL9vDrKCFUXlexhV3YlEjfyYOVO6z9uZgtckgPwwvNX0tsyD4mTqT27VVBE187plY+ItlLpYITUSjsbb+IXyhaMrOBoUZC1imTuE/nc1s0y3YkBxmF1ycM5Zx1sE3DLqULKjeNMhLkgyJVp0NxAJckZXHd7LykRJevQa+uj8FExtB5lwKpz48fPvf7UTq3nnnHTZt2sSmTZsYPWDh6bGVsrOzefvtt7nuuuuYO3cuBQUF3HrrrbukTHogY+wWvlucRYems94XuxwxGgKosFtwJ9rW9X+ZsWPh9tt39Sx2X1420bn6jYdSbrwoFheO7On429eSzOVcFZLj7BtZHI7chlYCZ63owLsLqjHawxDUJbYo39MePZhkyHeZTMDtw6YmyRtrwhwxWcVtE1TkC56rC3Dxyg402fPuu9ne6GbZljyOnFNDbsbgXKyZZFDXsZDOgCH8mGkfyyHOAj6i2ZQJGkaSKazcYpnF3eF1tMRpvthJmO3Sw3hhXgelPOdoNjQ+TjSjWBWZ/D08mXe6PCg40AEf8GoA3qyv4e6CUuY5+hswGqTEcIGhGS5gzGO0akQVhIC5Vw6/8aI6QIsXU1SMnJq442BYvJ2pYlhcApdeeilSyog/fZk9ezaffPIJfr+fqqoqfvrTnw7HdJLm2Gxn0l8ECXy7MGPPEaNLs/vTaCJo3pp8EmQsskadgFCsRLuaWZzRQwZSwiP+faJWgEjYJYYLGGq61mH6ijqsglNmqSTqeP3hCyHm3OHnqqcDPLUxwLdWdBCWfRdToy9UMKTy3rJSQuH+L0BImN3wNjUdH9EZ2EpnYCs1HR8xs+El5uvmckEE4MLCWOHmWKXU1BoWStAzl2EvZ2LBeViUwSGkDPs41mSezzt+IyG4ryGhAUEkP26uxTegs7xVCMrUeBJ7sSlWhp5sIgC3UDjC1Ru666ga8m7jHjRvYvxh6DD5RHO7nLWblEVHIh3PiMFcl4393Lak3qTTcpxctKdpuqTZvTFjCCdjLEtptHFYuNBQVI6A1VFE4aTrsbnH9XtcsWSSU34WhROvwWKPHH5pkU4qtdzubje7DxYB55fYh/UG47qDreQ4jeqkRPlsi851S4wCgkhmgUQQCKlsre/1dqjAaa0ryJDeAVtJFBliv8a3uIHIHrAeFGAfkYezO2N0jHDFNUsEkIWFd7Ra/qNV8oZWQ4uMHzLLclQwa9T3mFR4IaNzjiE3+0RKir/PuPzz+I+nK+oZI4EuqfOmr7Pf47qUlFusQyq68AyxxKdnzj/LLcLRN4w23F0EJbRthdzBWoCDcORCwdTYY1yFMP7I1ExtONhDOgPuGoQQPFRRwLmbGlntN1dmONtp5aqiTE7JcSYcI0+TJiZjpsCWlbHHlCSY2Pzaa4ag36rujtFCwDHHwF/+YjTK7IPVWULhpGsJ+RvRAk0I1Y7NPRbRvcgVTLyGutW3MdDhnopvQU+6pkVgSoMl0rZ9UTGUdn9UMbyJ1iVZgv9cZud3b4f4YKMes+poIJoET7YWN3zR0pRFRZnG3iKPGV3raQ+1RB0r0SnybuAw9xg+lJG9dAqC09TecP9UkUU21pil1hK4RVtudHdGoCN5Rq/kWGUU5yhjY14LhRCslYU86FVZGvRBew0uBN44q70KrAz4Od3dqxn1p7ZGFgXil7rHwiulqWqrHgbmxriFwg+yCzhmQCsBl7lalyFhywRXEbRuiT2ufpnRHfpvkyAQQbNQtcNlnwzLFFNG2vMSh1yLwltTihgXp+TZCnwxvZg3phRzWq4rbbikST1n3RB/zJnfM7+///zHKIFf3ae7r5Tw7ruw//6wZk3EzayOQhzZ07BnjN9puACo1kzchQcOGp8rfFSoLQkJh/VcmET3zwSXwnNzMjmz2M5ou5JQqOfAHMvOfVq6tyt1KLw1N4cp7uG/fyvLVvjb2XY++K6Di/dLLCQRSXyvP4IxZPIL62xOsYzG799I7Ft8SbtvIxep41kgCqKMgI/1Bvzd5S5CCNwm7nN7VIC0biE8Cbyp13JneA23hpZzS2gZ/wptomNAb6y3vJ1c21TN18FeoyOe4QLGeWERRh7lYr+XW1vqeNrTHnc7M68jkSjmQOPSJ3Xuam9ixwBdKEv0bgepQTHUbzUTOeKtWyCjCH5YC/OuNzwxisUwfmZfDD+sgYIpwzzfIZL2vJhACMEfynM5f3PTzi/oQH49OofRtj1cSTfN7s3EveDUa+ClKIm7x14CswYbDxHx+w3VYxgsKKJp4PXCD38Ib76Z0BSzS09ED3fha/0autU3hBBc5VrMTZ3Hxdt8J9PcKgEJhVbBt0odXFgA7h2fMr9hNe+2hPjQNon/OPfFJyIrgKkYi8rfpmXwnXIn6z1hXm8MEtAle2VaOKbANuICkkWZgh8ebuWF5RqdURYYXZF0FIboKAijWSQ7rYAoU1WBedm9l3EpdToUB+ucxXgUOy49yBR/HTl9sjMlGltlF0tkc8R9akje1evYpHdys2UGNqFSb0pZJjJr6L21r8bHJ1ojB+qFXGmZSKeu8evW+qihsViEgRlWB5c27GBVKLGqrngkIIkyCA3wSp272pr4U0FvLthHv2LoJUwxUG2w//eM5o9CiS1w58gxfttccMI9xs+exrCUSo8kw1EqHY2PO/3ctKOVyj7lA3mqwv+VZnN+Or9lj2SxrOYdNtPWLTnlwsohjOVIsYt1ZWKxfb2h67J9nWF4jJ5keGXGzzS/j2eeid+VWwjYvj2pUvWgtwpvy1L0cCeKNQtX7r78uDKTe3eYWwTf3TebQ/O6DZOGdfDe7yHoJdztk1HRaRZuTs67jiW2/p3D98pQOTjPxlXlDqaOgGclGqtqNVZU6+S7BAdNNKqHAF5bHeYnLxohmL4X37BVp3qqj5C9+9G+0iZRUDCaWFa4VKSU/NPzHgttrm6PlUR2+7tm+6o5tHMjAoVc1wz+lTmGKqL3Duo5/HnKWI5VS7kq9MUQlVMGc5RSgt2Xyx1tjQkbCipQqKjkqRbWhVI9s9SgAG+PqiBXteBvhz/kDa9i7qST4IJX4OuH4eXLo48TChzyczjsl8M3F7PsdqXS31QOyXTw2bQSvvQEqQ5p5KoKB2basaZDRHsk/5GrWEJtv8e8hHiTTayXTVwr5u2imcVhzBT40X3xx8Vi69bYnbzBMIwqK5MyXmyu0dhc/be7e6rktCI75y3voDVK4ooCjHEqHJzb7cX0NME7v93ZmsHSZ5nKkV7eaPkrMwp/QYNqXPgEcP1YF5eWDbePPjqfbdH4yYtBWvukXigixLl7q/zsOCsnzrDgsgr+9EGIzU2970N9RcAwXEyoCFuEkRNz7/QMKrp7K72m17DQbtxEGZ6M3o1XOMtw6GHme7biz5hF1YDzPhISeF+v51i1lDkily9lc0qNhPf0OqaEXKjEL2/uyUHpsedyFZWrswv4ZWt97A13ITpQp4UN46VteA0XxQLZ5cZXtmVT9HFCBWce7HvN8M1lpEjnvCSIEIL9MuycnuvisCxH2nDZQ1kvmwYZLn3ZShsfyq0jOKMRJi+vV8k43rgUIYTgiHwbb+6bjVNhUBd3FaNx4QPTM3tzxta9DVqISE58C5JMGeBy78LefQhoDO66+/CFW8N8+6n+hgsY7QGe+krjxy8YRtjhk1Ve+Y6dl79j58dHWAg6dHzZWkwvi8C4YNsVOKHAxvvzcrh8tFH6HJQar+mRK8WMjQVfucspyD6CTot5LZbmbo/kcUppyotlJNCKOfXaE11ZzLY52M/u5P9yinihZBzrQoHdfgHL7i67dheSeI+jBNDDMO0MWPhH+DSGVJYrHy79yOgOvaezu3/2adIMC28Q4/akm4/YPgIz2UWcfnrsbt5CwMyZMDVOPWUS7JNl5dP9czky39pvrT4gx8p783I4PL/PVX77FzFvWRUkZ/i/3vl3WEKZY9dd1m58MXZV4htrdbY09SbCTipUuGKBlWMXxN+3BF7ZJ4uuowr5797ZHJTbm2O3Tnbgi5NmGhYqje5JuAeZjdFxdI8dK9y4hsFRX+GInRyrAHvbHPwir5iHi8r5R+FozszIxqUo+KW51guRONLh5vrM1BnmAxHADKudUovxGVldMOuC+P2KkulnJFQo3RdGz4ePb4s9VguZK6XeE0gbL2n+J2kgfr+erjiqons0BQXw4x9Hfq7H6/H73yenG2OC2ZkWXpubQ+WheSyan8OWQ/J4f78cDsgZkPQeq5M3xiLhlL0GQ4YKpxXtmta36xs0mmOnkQDwr88HL9f7jjF3KY5WxWg268OPzhSRRYYJQ0SBnV2p18p2BrcmHDoH2jLZz+6MuhDpwLezIhsZEyy2pHsOTbHaOcxl3gOVDNdm5/f7+7BfgTM3uoHiKoCJx0L2uMSOIzWYdCJsfR+CnbHH+luh8qPE9r+7kjZe0vxPophQH/nGBwRvuw1uvtlouClErycmJweefBJONCnDOQRG2VXmZlkpd0S5ouePNzIMoxBCYam1V3TtD1MycJlVhNM18LRBcGi6ID2srjW3lFa2DjY0diYnx8CuwLysyEZHqTCnnFsmnFiEwhlKecxxArCjcowyCoCmmD2U+29nFivwrl6LJ6sKly24c3tL928bgt/kFjPfEbkY4iR3VlK+IAXYFg5SHQ4xShmaGi/0lvP3nMHZisIf8kcNmnf2GPj2FzDhWPq9UZllcMxfIKcCNr4O7ZWJz+GjX8HWD82NjaTrsieSTthN8z9JOVlsojXmmHzMLQh7LIpi9JL64Q/h+eehpcXo3n3qqeDYdQmv/Zh6LOz4MurTVnTudx8CwCE5Fs4vMeF1CXhg4dPw1Wvg775VHb8vHHQhlCdQsTWAXJN6d3muwcvl0o74IpiXlTnIjtJrqUy4mEQmm+mM6INRgNG4dnaaPkItIYjOc3plRH9KPna+a5lCgTDez0zMyUD8QJlKgeKgXvr4q7Y+pmckDKykHUWBitwOvCELHX4bVmnhTFsJZ7pzyIwh1Z+lqPwit5ifJ5i0K4H3/B5e98XvmB0PBTjVlcU0m4NOXWO0xcohTje2KAZ37ni48DVo3wHNG8CWYYR8njgear/qM8EEESpsfTf+OIC8SYnvf3ckbbyk+Z/DJ0METchQZWLnTbmJuRh3nyF08nDiEN+wr01BAewGDVEjMmoWTDsB1r6OROwUugsjsCC5PeM4PrcZQfyP28Lkv9/MpWV2bqpw76zC6UfAA4/+ABor++fSbP0Kti6FM34OUw9OaqoLxqlYlBDhOBGcqw4cfP78eZsvrrbIvlG8Lj1cZhnPbeFV+OmvyqsANhS+benf+OY4tZSDlSKWyGYq9S7aCTNKOJkispghsvuFqGaKbGyImB2mc7EyU8lBEYJS4eT/mMl92gaaB4RfHd0O/wB6v3m6rGFcVqMY3i9sZCr9wy6ROMGdhU/q/L6t0XQllA4EUqQQogNv+Tq5JbcooTYT2eXGDxi6LFveGdo8pAYNKw3J/+YNkdPEhArFs6Fkr6Eda3fhG3YVTpMmPo+xgh3E951W0s422niP3qojFcFcOYoTmIQ7ikBamhQiBMy7hJX2cYRXvcLe4R0ALLWO5c/uI3neuU+/4Trw7+oA/60P8uG8HGZmDrjEffL4YMMFev9++Q+GF8aWuNfNYRVcsp/KvxZFN4xnlAhmjOpvVHnCkqUdsfNJVOCztjCXlEUfUypc/NIym5e0HSySzWhIVATzRD6nqqMZFSG05BYWDhXFoMQuPzEk/2NTRP+WKJOUTP6kzKVO+liutxKUGjOVHFoIcY+2Pup+dOBL2cy35DgyRHyPz5kZOSz1e3nLHz+PbTjwSkkQiT3JANTGNwzDQqagOenx98BTJxuJuX33J1Sj2unkfw79GLsLaeMlzf8UO2Q7G4ne+6UvkWoZNCRLqGULbXxX7ofLxMU1zdCQwIVds1lXOANrd3JuMMb7rgNdYcmlqzpYsqBPsqcWgq9fjy24EfLD6g9g7xOSmuuPjrDS6oPnlw9eiWaWCJ68dLDBa8YHIASm+iIVCQdXWiZxiRyPhzAuLNiTKWHpJix1Vso2FmlNhOPMdCMdaLqOqvQPmZQIJyVqr+G0SquKKzSrIdmidzJbNVcR9HUweQXgoZIhFGxDyJzRgsbnOyRfkICccVBxJFy+EN6/BTa9yU6Rw0nHwxG/NTwv3xTSxkua/ylW0bCzcVyy6Eia8fI+WzmJyRHHhKWOjo7tmxZiGkm2bobPPqTGEyRLTkRW7BXTaOmLBizv1FjSHmLf7O5tulogGKccSCjQuC3pKQsh+O1JNq49SOehRWG2NktyXXDlARamFkc2IjIsgqlulfUeLepZGZZwQI75c8kmVGwJlERH4mu9hYe1LXTEaMjYFx1YTTuzyY05zo5q6tv3V30DN4ipzFRy4o717CKheBU41Z01pM7kpXMNnZahMv8GwwgatTdc+Dp4Gowfd7GhM/NNI31lTfONJyDDrKcZP2Ea8aakikgCX1DNCXJSP1f5m3ITn7KdQHdOjVUqzKWU05mCEqNqZsQJheDFF+GRR6CqCsrKjF5HJ544bOXRpulohx9dDe+/BUJQiuBTqbN89FTO//ZdbCw237phZVe413ixmEjmlXpKVpKyHIWfHxc5rKhLSTtBBIJsrAghuGGsk6vXRE4gVYAsi+C8USOXRL1Gb4+bcBsJr4nYx95KLk/q2+KOCyO5S1vHHWJv8kXszy5XUfBosQNbNkip+IEK5CgqF2XGNtbiMeFYoxKpozpG6ChKQlRPD6Opp8G8a/s/5y4yfr6ppI2XNN9YpJS8zzbeZ6upBN1E8RPGTxhXdyXGA3LpoJBUCJ1FVLGNVn4g5+8eBszmzXD44bBjR+9jy5fD66/DmDHw3nswcWL07YeTcBguPQtWLTP+lnJnku70mo18eOcF7POzl6nPNncr6VT6GGLuHLA6jNBQLELDo++jS8nbei1v6bW0di+jxTg4Xi3lstJCFreHeKg6gNot/Q+GYrBDgRf2zjZfAp4CntWSqNcFxoj4JVeFwsGBopCFMn5PozCSt7RaLrCMiznuDHc2f+2I3GiyhyMcGWwOB9kcDqakzcEsq4Nf55dQqA5tGVVUOOe/8MgREPYNsJ0F5E+CkjnQtB5Uq5G7Ur/KCDcVTYf9vguzLzL2879E2nhJM3Q0Dd55HVqbYd4BkF8At/0fvP0a+H1G2e3RJ8LPboe8+BUEqeItNvdLtk01hhaFccX4WtbGzKWpw8O7bOEYdpFR0IPfP9hw6cv27bBgAaxbB/kj91nt5L03YfnSiE9ZdY08TxvXfvg4vzj1B3F3ZRVwdP4A70cc0TvACC+lGF1K7tU28uWATs71+Pm3toUq4eW+6eM4udDOvTt8LO8M41QFZxXbubrcyVjnyK1MjdLPVhMijgMpxE6pYq5e/BJ1PCFNZ3GUztZ9+Vw2cgHjYo45IyObRztbaYuSz+RA8IPcQraHQ1zdGD/nxgynZWRTZklNzlvpvnD1Mvj8z7DicUNsLnsczLvG8KhYTZbh/y+RNl7SDI1bfgDPPgFaDFe71wsvPQtvvAyvfwIVw7+Ad8gAH7BtWI8xkyIs3Z6Ud9gSd/xnVO164+XZZ6MbLj00NcE//wk33TQyc+rLS88a+jN65KXFInUuXvRCXONFAFeNdpBvG+DpslghFEtwTRhjEmCbT+OlhgBdYclkt8opRXbsSn8vyRLZMshw6cu7so79ZT4nFWVxUgSF4C4ZYo3sIIzOWOGmzISHI1k6ZeJhMwW4To2c/xUJm1C4Rp3EknD8Zo9dhNGljKouDJCpqDxUVM71jdXUDAj75QmFvxWOpkC1UKBauK9wNH9obWCjGUM2CgrQqafWm5s7Hk74m/GTJj5p4yVN8nznAuNO2SzBAJx3yi+KVAAAJ2FJREFUEnyxbvjm1M0K6pEpbyXXHwE7L6qtxK928JpMfBxWXn7Z3LjHHts1xktLU1TDpYc8T/vO/9sFBCSD7qT3y7Zw26T+CqeesKSubD/GVn6GJWrFkYRJ801NNaBLrlvTyaM1AaNpojASa/OsXdw1w4Erv5UGGcAlVDbonTHv9hWMDs6TlKx+j4elztNaJR/Ievqm804WmXxbnUiRSH0eTF6CEgDjcHO5Op4xSkZC222SXaa8HxIIoOGMs1yNtdp4cdQ4Fvq9fBkwErPn2J0c4nBj6WP47GN38lTxGDaFglSGgzzd1cbXQT8qxnfajOmmA+WWtFTCriRtvKRJjnWrEzNcemhqgC8+g/0PTP2c+tDZnRA5nAbMChrIZSMnMXnPaTfgNdF8B6CtbVinEZXysfDV4qgdr6UQiLLRPLNXFoU2wWSXymFftrHR27sMCuCL9jDHLW3njbk5ZFgEq7vCHLukjdGuY/lUfoZOhN4oQgFXDsw4fNBxK1t0nv5KY9FWY17zK1S+zPHySosh2ybpLWVuDelcsszLMXs3UZTjBxk/RKEDVbL/ZyO7Q01fyZZBZ/Em2clt4VX82jKbnBTrDeUIG7NFDqtkW9R52xD8SJ1GkXCS2+f4Phnmdb2Gz/UmgugUYudMdQzTlexB+/hQrzM1HxVMV06pQnCw083BzsgtBXoIIslSFBY43BzlymRjKMC97U0sCfgIx6lcEkC+orLAkY7l7Ep2g+zBNHskd/4m+W3feCl184hCNvZh97wAfMp2PDJIBTlxx45ieBvBmWLOHHPjJu0iDfFzLopquEB3I8ZvXcbpxXYOyrVx9Zoutvj6L7E9n/qX7WF+sK4TryY5bkk7TUHJUvd4LppwPWGhonULr4V7LoPuXLjwD0ZSbx/eXKtx4v0BHlscZl2DZF2D5MEVQV5qiZz4KbvN1GVb89Axn1vhGLBAb5JdLI1guNC9zy5CvKnXmNx7YpyrjsWGEnWB+JY6nilKdj/DpVLv4nvhJbyiV9NEgA5CbKaLP2hruDO8ZtA+aqW5nlJzyENNUQVcoxbm9tZ6Dq/ewgl12zi0ZjM/bKrhxa52PvJ745ZcKxjn4K25xf28OWlGnrTxkiY5qrYnv+0IfOn3otiUN2SoaEhW04DdxJ3hSewGTUWuvNLc+3/VVcM/l0jsOx9OPSfyHFUVps2Ecy8GjFyTVxuDOytzBqIBj9cGeKjKR11Q31lv9lzefCr2+hu3lp3DSznzeD5vP+6efT1c+wgUju23j63NOj9+MYim0+84HbnhmKpiEkF9m4tAyPwldv8BcvifycaYF2gd+ERvML3/RCgTLn5umcUU0T+MVYSda9XJHKL0r8ENS53btdWEorwpq2Q7j2n988JcJjWQ/CmqFKwLh/hW/XZe9HQQ6J6nBD7xe3i6TygyFnvZHNxfOJoD43h20gw/6bBRmuTIGuwGNs2Jp6duHlFwCxvHyYm8xsaktp9ADptpiztOAG+zmfY4ChInMpFJYhdU7wxk3Dj429/guuuijznuODj77BGbUj+EgD/+HcZVwEP3Qmd3GwerDU4/F/7vN+AyFo5PW6Mtlb2EJTxXHxgkk9FgzeYPpafu/Nsq4HuWwTqpTy2NbKRoqtypXhqLYFjBbo3te1GALKwcqPQv/26X8Ut6PWhxk1mTpUy4+KllBo3ST5MM4BIWxuCKKMj2kd5AIM5sP9IbuFCp2DnX/ZUCVmnxjYY1tNMpQ2QOUc36T22NtOraIFPIjGfMhuDZ4jGMtqbzXHYX0p6XNMlx5XeT266oxLi7HgEOE+PYl9KktjVjuICxfsUzXA5lLIcJ88Jqw8611xqaLuPH93/c6YSbb4aXXgLLLryvUVX43k+NxO5n3oCnXjX+/7u7ITMr/vYD2O6Pr6esyciOlE826xE9O9aAiGu4KIqO0zZgqezRb0Ggdu+gAAc3WWbgHOCJyBW2uBfoDCzDYrj0pVA4mKZkM1a4oyrJLtQb4+4njGSz7Nz59/4iH7cJj6UEWoYoL9eshfnQ70nahxNE0qZrLPZ7WR30o+0iRd80vaQ9L2mS4+gToHQ01FSZ38bhhP+8PnxzisCZTKOKDurpGpYMmHidgAGWUhu1jcAu4/jjDbG6ri5YvRrsdpg61dDk2V2wO2Du/lGfPjDHGvf9V4Ed/tj31gKYnakmZARkNltpLg8io2wikFQUd2JR5cAnOF4pRevOjJkmspktciIe+yBRyPvUR52DAhyq7B4SqtEbG/TH36fKyyZUjlZG8aIe/xriGmKrgx3h0JB1XS5p7J1nkWrhmqw8TnEPwQOdZkikPS9pkuf1T2HchMGPCwFz9jVCSxYrZGbD+ZfA52tgzLgRnWIXQaZSQAbD4+4VJvJqPCkVJU8xGRmw//5GIu/uZLiYoMKlcmKhjWjCsyowPUONu+xJ4HtjB1eOVPo0WsoDVM70sG2Wh7rxPnxu495d1QQF2+29O+iDQOKwacyu6C92J3UIdVk4JlTOBeo4zlfHMUfJjWo0VYgM5ou8iNaZAmRj42hRTIt3NZubnmFdwyNsbX6ZzkAlcoQ9AxOEuTLp8gEidocqRTG/QQKowE3hEEvCnSlWtm7QwvyqtYHHOltTut805kl7XtIkT2YWvPclfL0E/vV38HTBrDlw7Y92i4XwQ7mN17tzXobDsW5HRUfGbfI4EonD/6s8OCOTo5a0sapL26mj0vN732wLhTaFlV3xgwX7Z6lU+TXK7ApCCN5vDnLq1+0ErCC77d4um0ZXvo+8Kht5tTayG62oYYG/IkBbj4dFQlmOl32nNeCy9x5X1wAp2PjKOJqOh1wHhDUvHYGt6DKMy1qEyzZq53gpJQ1di9mvYyEh9yiWu0YT7tMderrI5mJKqW18HF+ogR4foJcaWn2ryHPNZGzuSYgRakdxhlLOe1p0LxHAGFyDyrpzhZ0jRQnvyuhl02eqY4Y8v0lWG6NUC7WxxDST4G/tTZzkyiR3iC0C0iSOkCNtoqeYjo4OsrOzaW9vJysr8Xh4mm8mX8tanmTVsB5jPqNpwcuGGG0BACrI4Voxb1jn8r+MT5M8Wevn39V+agM6Yxwql492cE6JnatWd/JUbcB0rsN0t8p3xzj58YYuvFr0kNSoDQ4cXSqaVVLklPz7Yjs3L/HxdmuIgEtHCElprpfp41opzPTTsiGHmkXFeBudvP9dK2HtfZo8XyH7BDOc1mLG5Z2C01pITfuH1HUu3PlcUKhUW3PQhEJxOMj8wgvY3voGnYFKos2yNOtQSrKGV0+pL29qNTytR+6JZEVwuzqHQmXwTY0mJU9r23hP1nXnQBtd312oXKpOYD8lNYnuL3s6+FVrbAMrGQoVlcuz8jjZlYVTSQczEmEo63faeEnzjUNKyR9ZSCMmBdmSQAC3ciidBPgzi2KOLcLFeHKZTzllYjfQevmm4+2ERa/BtjVUhhRutM7m5dIFhJX4d8c9OTQxc2kkKCHQLewMvBdZBQ2hARVIuvH/okobWY02FAGzSwW3n/wWrb7VEY+uCCtu62g6g7HaTQiyHBPo8G+K+VpUxcmsUd9DESPXF2mp3szTWiWNGC0YBIaX6HJlAvlK7M7QbTLIEr0FL2GKhIN9RB62FHuOHu1s5Z72JsD46CRGSf0Cu4vPA8lfLwQwwWLjgaLRZP+vdUgcAmnjJW28pOlDk/RyB58N+3H2oogddNJCfLEtpftu8jgmcuTuVHn0TUJKePFeePNh4//02hJb3cUcf/DtbM5Irvps8LEwF4vsvrqOW+HCGlT4+9ldZLv+mYIJmEkVh6lFl/ULR40UAanhRSMDC9bdoZN6H5q0MK97O6kOh8hWVI5zZeAUCifVbRvSflXgCGcGv88f+fd7T2Uo63c6UJfmG0coRaJW8ViOeYGwnryYN9lEsXQzU+weVSLfGKSEh39peFz60GNfjPE08PZHN7HX8f8koNoJDfWWzWwaU7eN4S0JcVG2k398qtEVuJDxBXUcOXkF4/LjlxhHxtwLkFF7OA0vdqGaEm7cFRSoFi7OzB30+Cybg9VBf9JVSRrwnq+LRi1MYToHZtjZvUziNGlSQB5OLLvpqS2AD4e52/X/JKsWDjJc+qIiGeNr5EHfIq4tH+FkcgF+l87/t3fn8U1WacPHf+dOmnRPW7pjy1opawHLJpsIQ9EZXx0d3ICR0QcE66u4jIo+is5nFMTtVUYF5hmVRxlhUD84ODLAI9vjWASK7JRlAGVrK0sXoGty3j9CA6VtmtCUNu319ZM/ct/nvnN6DMmVs1zn7zsc7DwRzuHTsazd353nvhrP0u39r/ym9ZVQZgIDoq/w/q3Pw+FtGryc2gHsKK9/k1bRcM3zE16IBrAqM+kkNss3twZ+pJDKS34RF+kyjuhCTnu414uoxZrFHhW748QGXk8NI+hqvjk0VLoS9TqDDod29kos+WEwm3/qWOeltVNEBHYh0BxN3UGMIjqkN6Z65pmIi66zBhHugyGu5vi50xJJ35ZokW6iMwc5zc+cvwrbM3rvW35isz7OaUq4NH1Wkg7nZlLorKLc3+DcWVj2OezZCRYLjMiAQUOvyr5RzdJBD1eWXfhVPDbeyl9PlFF5ld4cIYW1f9Qq5eAfu9JJT3Y3QbfaFRjKTIJtKArFvp8/odJRwuXDSMGWBBLDb2hQnVubHWWlFDVwmE0B8R5MDBcNJ60sWqRgFcDDuj9rOcwGjnGeiqauUjV17bl0hCLmk819Oo3udc2LWbMKHn3AGcBUpfH/4H3o3gv+a5FzC4bWxtPtDK5xbo75RPtg/pZbhkN7vuvzFdFgqlSEnaq9flob7MtvS6XdwGyquyZGWQW2A3lYykxEW3tgzZsLhafobovkdFp7cmOKsOsyLCYbMaF9aROShuHJxof7N8DGL+Dobmfg2zEdBtwBST2c84gqy8FsaRVBcU5FWYPvoYEpJ4/xaVwyCeaG7cUk3JPgRbRYQSqAm0ghQ3emjEq+IIft5NWbVK6paeAz9pCqozFd1o1t35aF8eC94HA4BwwqL0m6lbML7rsDlq1r2r2JmkKvofCvZc40tu4Mux2AbqFmvupr485tRZyu0AQoZ7v7pCfmknuYKhWJewMxHO6//B11Df9oTcK6PcR9tw+jsupvy7p4f8NEzLd2YtJHw/1/8O7/+5q/wHeLQBkX221/Fuz9FuJT4OfDYK8Aawj0HgMD74TQenoE/ZjVRwFasXbw+MnjfBrfrv7C4orJ8Jxo8QylCFIB6Av/eUoBnai5KuFqOEs5ezhZ7Vhx/lrK3vi/oB2o2jIc2O2wbw+sXXWVatmMjLyn/t6B2x6C2CTX0+FRFn4a3ob/7hnG9JgSvixZzj9P/Jn/99MCBhfnuJZbXwnLeUXsYSvttgdjLal71Y3CQXLkz1hMta+QS/xmF/Hrcy4JXC7juHDd5lWw9F3PK3hwszNwgeoBX9X9cvc7AxeAsnPw/Rfwl6lQ6Pskb83FjUEhPrvXvspy1pw/67P7iZokeBGtRls8zyNgoAgigLvoTg9iapyv+prsQzzxXNzXxeKjzQAUVMsfc+7URoqOfUXQhsMoh5svVZMJ/rnMBzXwM207w4OzwBRAjUmshgnGPgY331/jMquhuOfYCp5fPpnROxcy4th6Juf9D2tz/sDKvS8TXnkFicuUZvSQwwy+5QAJaScxAupeuq8xGNNtS63nAopLiMva5+H7ScOav0HpOc/quGkpeJW8TsPZAvj6LS+u8S9hJjNtfbjE+ZOzsu9RY2plfcuiNetHIis44NEOuLGEMJ6eRKogxuteZHGUb/mJUxcCivZEMIL2dFUxaK0poRIHmnzO8j7ZDa6rBsov5KvR2kFx7ipwaJS9nmERhwNa6y++PiNg1jL4din8ezugILUfDLkNguvYODDnf2HFn1xPDS7+ohtanMNf//0Ov+ryjBeVcO4mbYtw9lq0G3WM+H4/s+vja6ksMVEVWBnKgUMb3HjtdoZ12l3rnSJ3erFjO0BFGez7AXoNqb/ssT2gvc2H5ICD2XDmBES2zERsvwkO5+1i99t9eGqPLJluVBK8iFYjVFm4R/dkIdtd+6fAxVyliYTSlwSSiaA9NtSFYQiTMhhCMoN1EmXYMaEIuORXq1KKYJyT80J0JP1py0aONbi+33CQ63US5pI87BUFYDKobBOC6dS5un+NKwNSUhv82n7LFg2//A/PymoN//uJc7ipliEiMw4yiraTdv4w24Lbe3TLmPBS+ne5mHhOKbDaykm57SB7Pr2WUOt5FNA+Kp9fpG6jb9LBGqNdGsgNCMdREUSCYWCqL2C9lN3DiekNSWGff7DFBi+jQ3wXvNS1W7jwDQleRKuSpuKI0v1Zy4/s5mfsOIgnlCEkO3PDuPnAUUoRWM8/GaUUd+iuaDSbON6gulaiWcQOxjlsABhFpRjn61kRoR1w128b9LqtRvFJ5xexGw5l8JeAnXxwTSrvHa3vl7TmhrQTmE3VAyGlIOya84zs8wO/67nW7dScUmXmHxE9OGaJZFRMMX0dmzz8Yy5IutazcikDYes/8TRTbzVmS/1l/FS8OYCxITY+O1fYoGn9BtDXGuSraolaSPAiWp0kZWMCvQDnJo7Kh7+QHFpzgNNswzcTG3M4Ram1HaAIWbEHVVLpfg7ErWMh8RqfvHaL58HSWEMZpJX/yJvH3mHyzz/zkymCj4IG8WVgGvZqc0Y0nRKKsJhr7yVRCnoOPIhyM4VGA/+I6MHxgAgANqSnc/cXSzHsHgzvGCboNgCi29ZfFqDfr2Hrcs/KXspkdS6jbiobvoZvPoW8n8AcAF0HOCdrd/RdnZ6MiMEMLL4QwBg4U/97tpuUkwMYF9o0k/1bCwleRKvmy8ClRFfwIVs5RIHP7qmBXDNEh3cleO3f3JdVCnX6lM9eu8ULj4GAQKhw06PiqIQTP2AKDaK7dtCl8gQ3le1mfUBnbol6iPMXMthGhZWRnnKy7vsAZcp93o/TQR04Zrn4hXc2NJTPb/kVdy390v3fYRjO4bIJz7kvd6nYDtCpP/x7o+fXAAy8AyxN0KOwdzO8+0TNCcmbVjof46e7lsE3lFkpnoyMZWJ4FGtKznLW4SDJHMBLp3M93qf+EVsbBgQG+6Q+onay2kgIH/mUnRz2YeBSxYTC1vZWjOIyt70uSms47f4LtKWz2zX7cxx8s9zBlu8dnC1281s5wAq9b3LOE6qLUhBscS0nNl/47T244t+8V7yEtFAT07tpMvoerTFcdLnEigK353PDetX4QF7+i1F8dPddFIaFVTuuq+ashEfBTb+D/1wIkXFu71/DjR7ODarSvjcMa4IhyZxN8MaUOlZSaefjk5lw7IBPXzbaZGZsaAS/C49iVHAY8WZLvSu/EgwzH8UkcV9Yy82H01xIz4sQPpCnz9bIy+ILFkwkEY7Zaka364zes7v2HC8AJjMkd/B5HfyB1pqVyzR//QDOV/uO0wQEaPoPgdvuUrTvdNnXz7DfwqEtcOpI9XwnVYnbokJqzR9jQjOuZCPj+j5AuTWCqZUH617FpjVm7aBDed29YhrYxblas/2uHTqE9dcPovOhQwSVlpIXE0tc/LU8ZkptWObb2A7QLg1+3FZ/2SETYPhVDlwqK+DgDpj7VP1lFbD2Mxjnzcow79weYuONQve7gL8QFUdP61Xe+LOVkp4XIXxgDyd9kt/lcoNJwnohzbu69wGUux/39kq4a0Ij1KL5W/Kx5r/mXB64OFVUwL/WwNMPaf619rIGDAyF+96GQXdB4IXeDWVA22shNgyC3Az1aDvk52BRBmON5LrLKcUNxXvdvj92BiWSq+qe2+IwmdjXuTPbevQgNy6WbbqAMl9sbHDL792fVwo6D7y6gYvWsGoh/H4MvD4Zzhd7ds2+hqcocOe2kHCuDbDW+qWpgJFBofSTSbpXjQQvQviAHQfKx+FLEGYy6HTxwO33QJ/02pe5KgU33waDb/BpHfxBfq5mycf1l3M4YM4szcn8ywOYEBhxPzy2BB7/HJ5aBsPuBosHHdPlJXA2nzE6mnuNdgRc9h6waMWowj10K617ArcGtoR4n0reJ8GLLQ4G/Kb2c8pwJv27YWLDX8cbS9+DJW/BuULvrmvI8m8PBBkG82La8qvgsGpDFkFKcV9YJC9Hxft0Dp1wT4aNhPCBRMJ8vmfSIK6pvreR1Qr//QXM/gMs+QRKL2TgDbfBxAch88lWsYHe5dau9LzdtYb/+Vpz98Ra2skwQdCFLMwxHi45/teFlPymAEZ3Gs7wXrezNQiKdAVRykKqw8ze0tVub3HWsFJo8m6oIRgTob76+B45CSzBsGFx9RVYUW3hlichrlPd1/raqRPwz4+u7Nqeg31aldqEGSZmRMXzqC2GfRVlmBR0CwgkyJB+gKtNghchfKAL0URgpZAyn4QwJhSDSKp5IjgEXnwVnvxP2LvbuR1A1x7QisfZ83M9L+twwJ4dHhQMiYaweCj28Ob2Cti/GuvRLQy4+RUIuZDEzYDwwBSKSg9Q10Jbw/BuVYoBDFRtWOz4kYP6LBYMhhux9FNtruyXvzJg2ATnbtKHsp17GbVJhrZdr34wnPWPOpMGumWYYHgdPUiNIMJkor9JVhM1JQlehPABQykm6DTmkU0ljivuhVE491X6LWlEKDcBSWgYXDfgyirbwoSFe/d959H2NQfWeB64VNEOKCmA7E9g2COuw8kRo8nJP06l4zzVAxiFwkRa5K+wkUch9WfHNYAQzKzW+dVutcteSCgHed7UkzjjCuddWIMhdeiVXesrZ/IurP7yckgs8w1o0zKz/oraSV+XED6SrGxMYwD9SCTgwj+tcCyMoiPtsNV7vQnFEJL5PdfTTdXcDFLUbsgI5XHgohSkXVdPb4LWzgDkSmgHHM6Csov7S1nMNlLj7ic6pA9KVUVOBhFBqaTGTSTcmsRoI6HeGVNWDFKxUUxlrefPYucF+3ZKdO3n/UJohHe9LmFR8Mel0NOD/ZxEiyI9L0L4UIwK4Td04w7dFQfaNWdlhG7PUnJq3TJAAUEE8BDpxKk6NhAUderURdHves2m79yXUwoCA+HGMfXcsOBoteDDa9oOxXlgvWS3cVMYyZFjuCbiF9gdJZgMK8YlSevGGIkc1MVk6zPVMrkaOIOWyaYUumHj9/Yf3L50GQ6+sh9jrNn7CcDNwsCbYfmHbgooCIuAzn3g5t9Bu65Xq2aimZHgRYhGoJTCdMlvaYsycSfdGa07soqD7OYk56kgEDPpJDKUZPfDRMKtac8p3n9d8+2a2s8r5ZwWNP0VRVh4PX0cxz3Ie1KfgNqHbgxlwjDVDFBNSpFp6sImfYpvHLkc1yUEYWKgEc2NRjyRysIJXUKRB0NL63U+Y/HT4CWhAwz+P/Ddspo9MMpwJhZ8cr6znGjVJHgR4iqKUEGMpXtTV6PFsVgUjz6rGPcfmjUrNHt2QMEZsNshPBz6DlTcOAZsEZ5MQG3gaLotEcK9n39hKMUAFc0AI7rW8+UezgMp9cUS6qY0/lkICoU1f3PmLqoSlwz3vySBiwAkeBFCtCDRsYqxExq4QsYW37Dre9/ZKKt0YrF6tDlguL9/rJvMcOfjcPP9sPM7KCuBxE7QOa1VpgIQtfPzd7kQQvhYYhoE2qDUTZK0gGCoOO8cylDKuQbbMKDffdD++kapVpAy0w0bu3CfvC1DtZBVN6ERzjkwQtRCghchhLiUYYLBU+GbV6m1n8NkgTEvOcsd/g7Kz0FoHHQcenGLgUbyoKkzT9p/qHMIKRoLw01ebtAohB9SWnubDah5KSoqwmazUVhYSHh4eFNXRwjRUpzYCVv+Cicv2a04MQ3SJ0Ckm72MGlmBo4w37Tn8xPlqx7tjY6o5hVDlZj8mIZqRhnx/S/AihBDuFOdDaRGEREFwVFPXxqXIUc5uXYiBQYoRRqSyNHWVhPBKQ76/ZdhICCHcCYt1PpqZcMPCQCSZoWidJMOuEEIIIfyKBC9CCCGE8CsSvAghhBDCr0jwIoQQQgi/IsGLEEIIIfyKBC9CCCGE8CuNHryUlZXRu3dvlFJs3bq12rnt27czdOhQAgMDSUpKYvbs2Y1dHSGEEEL4uUYPXp566ikSExNrHC8qKmL06NG0a9eO7OxsXnvtNV588UXmz5/f2FUSQgghhB9r1CR1y5cvZ+XKlXz++ecsX7682rmFCxdSXl7OBx98gMVioXv37mzdupU333yTyZMnN2a1hBBCCOHHGi14ycvLY9KkSSxdupTg4OAa57Oyshg2bBgWy8WU1hkZGbz66qucOXOGyMjIWu9bVlZGWVmZ63lhoXOH1aKiIh//BUIIIYRoLFXf21eyS1GjBC9aayZOnMiUKVNIT0/n8OHDNcrk5ubSoUOHasfi4uJc5+oKXmbOnMlLL71U43hSUlLDKy6EEEKIq6q4uBibzebVNV4FL8888wyvvvqq2zJ79uxh5cqVFBcXM336dK8q44np06fz+OOPu547HA5Onz5NmzZtUEr5/PX8TVFREUlJSRw5ckQ2qvQBaU/fkzb1PWlT35M29a3a2lNrTXFxca3zYuvjVfDyxBNPMHHiRLdlOnbsyOrVq8nKysJqtVY7l56ezrhx41iwYAHx8fHk5eVVO1/1PD4+vs77W63WGveNiIjw/I9oJcLDw+UfnA9Je/qetKnvSZv6nrSpb13ent72uFTxKniJiYkhJqb+XUzfeecd/vjHP7qeHz9+nIyMDBYvXsyAAQMAGDRoEM899xwVFRUEBAQAsGrVKrp06VLnkJEQQgghRKPMeUlOTq72PDQ0FIBOnTpxzTXXAHDvvffy0ksv8cADD/D000+zc+dO3n77bd56663GqJIQQgghWohGXSrtjs1mY+XKlWRmZnLdddcRHR3NCy+8IMukG8hqtTJjxowaQ2viykh7+p60qe9Jm/qetKlv+bo9lb6SNUpCCCGEEE1E9jYSQgghhF+R4EUIIYQQfkWCFyGEEEL4FQlehBBCCOFXJHhpgcrKyujduzdKKbZu3Vrt3Pbt2xk6dCiBgYEkJSUxe/bspqlkM3f48GEeeOABOnToQFBQEJ06dWLGjBmUl5dXKyft6b13332X9u3bExgYyIABA9i4cWNTV8kvzJw5k379+hEWFkZsbCy33XYbe/furVamtLSUzMxM2rRpQ2hoKHfccUeNZKCibrNmzUIpxbRp01zHpE29d+zYMcaPH0+bNm0ICgqiZ8+ebN682XVea80LL7xAQkICQUFBjBo1iv3793v1GhK8tEBPPfVUremWi4qKGD16NO3atSM7O5vXXnuNF198kfnz5zdBLZu3nJwcHA4H8+bNY9euXbz11lvMnTuXZ5991lVG2tN7ixcv5vHHH2fGjBls2bKFtLQ0MjIyyM/Pb+qqNXvr1q0jMzOTDRs2sGrVKioqKhg9ejTnzp1zlXnsscdYtmwZS5YsYd26dRw/fpzbb7+9CWvtPzZt2sS8efPo1atXtePSpt45c+YMgwcPJiAggOXLl7N7927eeOONaslnZ8+ezTvvvMPcuXP5/vvvCQkJISMjg9LSUs9fSIsW5euvv9apqal6165dGtA//PCD69x7772nIyMjdVlZmevY008/rbt06dIENfU/s2fP1h06dHA9l/b0Xv/+/XVmZqbrud1u14mJiXrmzJlNWCv/lJ+frwG9bt06rbXWBQUFOiAgQC9ZssRVZs+ePRrQWVlZTVVNv1BcXKxTUlL0qlWr9PDhw/Wjjz6qtZY2vRJPP/20HjJkSJ3nHQ6Hjo+P16+99prrWEFBgbZarfrTTz/1+HWk56UFycvLY9KkSXz88ccEBwfXOJ+VlcWwYcOwWCyuYxkZGezdu5czZ85czar6pcLCQqKiolzPpT29U15eTnZ2NqNGjXIdMwyDUaNGkZWV1YQ180+FhYUArvdkdnY2FRUV1do3NTWV5ORkad96ZGZm8stf/rJa24G06ZX4+9//Tnp6OmPHjiU2NpY+ffrw5z//2XX+0KFD5ObmVmtTm83GgAEDvGpTCV5aCK01EydOZMqUKaSnp9daJjc3l7i4uGrHqp7n5uY2eh392YEDB5gzZw4PPvig65i0p3dOnjyJ3W6vtc2kvbzjcDiYNm0agwcPpkePHoDzPWexWGpsVCvt696iRYvYsmULM2fOrHFO2tR7Bw8e5P333yclJYUVK1YwdepUHnnkERYsWABc/Gxs6OeABC/N3DPPPINSyu0jJyeHOXPmUFxczPTp05u6ys2ap+15qWPHjjFmzBjGjh3LpEmTmqjmQlyUmZnJzp07WbRoUVNXxa8dOXKERx99lIULFxIYGNjU1WkRHA4Hffv25ZVXXqFPnz5MnjyZSZMmMXfuXJ++TpPtbSQ888QTTzBx4kS3ZTp27Mjq1avJysqqsW9Eeno648aNY8GCBcTHx9eYJV/1PD4+3qf1bq48bc8qx48fZ8SIEVx//fU1JuJKe3onOjoak8lUa5tJe3nu4Ycf5quvvmL9+vWujW7B+Z4rLy+noKCgWk+BtG/dsrOzyc/Pp2/fvq5jdrud9evX86c//YkVK1ZIm3opISGBbt26VTvWtWtXPv/8c+DiZ2NeXh4JCQmuMnl5efTu3dvzF2rIxBzRfPz44496x44drseKFSs0oD/77DN95MgRrfXFCabl5eWu66ZPny4TTOtw9OhRnZKSou+++25dWVlZ47y0p/f69++vH374Yddzu92u27ZtKxN2PeBwOHRmZqZOTEzU+/btq3G+anLpZ5995jqWk5Mjk0vdKCoqqva5uWPHDp2enq7Hjx+vd+zYIW16Be65554aE3anTZumBw0apLW+OGH39ddfd50vLCz0esKuBC8t1KFDh2qsNiooKNBxcXF6woQJeufOnXrRokU6ODhYz5s3r+kq2kwdPXpUd+7cWY8cOVIfPXpUnzhxwvWoIu3pvUWLFmmr1ao/+ugjvXv3bj158mQdERGhc3Nzm7pqzd7UqVO1zWbTa9eurfZ+PH/+vKvMlClTdHJysl69erXevHmzHjRokOtLQ3jm0tVGWkubemvjxo3abDbrl19+We/fv18vXLhQBwcH608++cRVZtasWToiIkJ/+eWXevv27frWW2/VHTp00CUlJR6/jgQvLVRtwYvWWm/btk0PGTJEW61W3bZtWz1r1qymqWAz9+GHH2qg1selpD29N2fOHJ2cnKwtFovu37+/3rBhQ1NXyS/U9X788MMPXWVKSkr0Qw89pCMjI3VwcLD+9a9/XS3gFvW7PHiRNvXesmXLdI8ePbTVatWpqal6/vz51c47HA79/PPP67i4OG21WvXIkSP13r17vXoNpbXW3o5pCSGEEEI0FVltJIQQQgi/IsGLEEIIIfyKBC9CCCGE8CsSvAghhBDCr0jwIoQQQgi/IsGLEEIIIfyKBC9CCCGE8CsSvAghhBDCr0jwIoQQQgi/IsGLEEIIIfyKBC9CCCGE8CsSvAghhBDCr/x/FtV4i9z+YloAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import sklearn\n", "from sklearn.manifold import TSNE\n", "from matplotlib import pyplot as plt\n", "from matplotlib.cm import rainbow\n", "\n", "# This is a sanity check that our data clusters nicely\n", "# map our feature space to 2D, plot it and color it by its labels\n", "\n", "reduced = sklearn.manifold.TSNE(n_components=2, random_state=0).fit_transform(X)\n", "plt.scatter(reduced[:, 0], reduced[:, 1], c=data_class, cmap='rainbow')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "d3a6b699-b006-4ea5-8bcd-91dd69405b1b", "metadata": {}, "source": [ "## Create a train / test split" ] }, { "cell_type": "markdown", "id": "cbdd2624-0f99-450a-93e5-55c009c7e535", "metadata": {}, "source": [ "For the sake of demonstration, we will shuffle the dataset and split it into two train and test splits composed of 80% and 20% of the data." ] }, { "cell_type": "code", "execution_count": 7, "id": "21a9b3f4-a564-49f4-ba8a-fde55c636fc3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train set : 960 samples\n", "Test set : 240 samples\n" ] } ], "source": [ "# make things reproducible\n", "rng = np.random.default_rng(seed=0)\n", "\n", "# shuffle the data\n", "permuted = rng.permutation(len(X))\n", "shuffled_x, shuffled_class = X[permuted], np.array(data_class)[permuted]\n", "\n", "# create a train / test split\n", "train_prop = 0.8\n", "n_train = int(len(shuffled_x) * 0.8)\n", "train_x, train_y = shuffled_x[:n_train], shuffled_class[:n_train]\n", "test_x, test_y = shuffled_x[n_train:], shuffled_class[n_train:]\n", "\n", "# summary\n", "print(f\"Train set : {len(train_x)} samples\")\n", "print(f\"Test set : {len(test_x)} samples\")" ] }, { "cell_type": "markdown", "id": "e25720eb-e499-43b6-a19a-b82b100af32b", "metadata": {}, "source": [ "## Normalize features" ] }, { "cell_type": "markdown", "id": "5ecee2d7-5f23-49df-8c6a-066c38ef01bf", "metadata": {}, "source": [ "It is usually recommended to normalize features to improve the performance and stability of the classifier.\n", "In particular, we ensure that each feature in the training set has a mean of 0 and a standard deviation of 1.\n", "This can be done with [`StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) of Scikit-learn:" ] }, { "cell_type": "code", "execution_count": 8, "id": "a28bbe9b-0be4-4e68-b9ed-977fb20f29e3", "metadata": {}, "outputs": [], "source": [ "from sklearn.preprocessing import StandardScaler\n", "\n", "scaler = StandardScaler()\n", "train_x = scaler.fit_transform(train_x)\n", "test_x = scaler.transform(test_x)" ] }, { "cell_type": "markdown", "id": "91116a77-b188-4c1c-95ac-e6e8872308fb", "metadata": {}, "source": [ "## Train a classifier and compute the test accuracy" ] }, { "cell_type": "markdown", "id": "ad69a0bd-e57d-45e6-80da-2a49a64fe9a3", "metadata": {}, "source": [ "We can now train the classifier. We will use the [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) object of Scikit-learn.\n", "\n", "The default learning algorithm is `lbfgs`, which should give an accuracy of 98.33% for this particular train/test split.\n", "\n", "You can try different algorithms, and hyper-parameters. In a real-life scenario, it is possible to obtain better results by using k-fold cross-validation methods to validate accuracy and choice of the hyperparameters." ] }, { "cell_type": "code", "execution_count": 9, "id": "b496a585-018b-471e-8b26-4bd1fc0cd0b0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Precision: 98.33%\n" ] } ], "source": [ "from sklearn.linear_model import LogisticRegression\n", "\n", "# For a real problem, C should be properly cross validated and the confusion matrix analyzed\n", "clf = LogisticRegression(random_state=0, C=1.0, max_iter=500).fit(train_x, train_y) # 98.33%\n", "\n", "# you can also try the sag algorithm:\n", "# clf = LogisticRegression(random_state=0, C=1.0, max_iter=1000, solver='sag').fit(train_x, train_y)\n", "\n", "print(f\"Precision: {100*np.mean(clf.predict(test_x) == test_y):.2f}%\")" ] }, { "cell_type": "markdown", "id": "db9a2ab7-c354-4273-8f07-76687b10f7e2", "metadata": {}, "source": [ "## Classify a single example" ] }, { "cell_type": "markdown", "id": "62c0c8ed-e56a-46fb-9754-b48bba907914", "metadata": {}, "source": [ "Below is an example showing how to classify new samples, and how to print the probabilities of the top-5 predicted labels." ] }, { "cell_type": "code", "execution_count": 10, "id": "c5e2cb04-2326-46d9-b080-20b464d6ad08", "metadata": {}, "outputs": [], "source": [ "@torch.no_grad()\n", "def classify(s: str):\n", " tokens = tokenizer.encode(s, bos=True)\n", " tensor = torch.tensor(tokens).to(model.device)\n", " features = model.forward_partial(tensor, [len(tokens)]) # (n_tokens, model_dim)\n", " embedding = features.float().mean(0).cpu().detach().numpy()\n", " probas = clf.predict_proba(embedding[None])\n", " assert probas.shape == (1, len(labels))\n", " return probas[0]" ] }, { "cell_type": "code", "execution_count": 11, "id": "41246dbf-ea83-4140-bb57-510c958c26df", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 99.65% Migraine\n", " 0.20% Dengue\n", " 0.08% Cervical spondylosis\n", " 0.05% Jaundice\n", " 0.02% allergy\n" ] } ], "source": [ "sentence = \"I have been feeling excessively hungry, even after eating, and have had a stiff neck.\"\n", "probas = classify(sentence)\n", "\n", "for i in np.argsort(probas)[::-1][:5]:\n", " print(f\"{100*probas[i]:6.2f}% {labels[i]}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "51284fe5-da9b-4176-9468-72d5041e0ae4", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "52e440c1-d377-4580-bd6b-84ee1e5686f1", "metadata": {}, "source": [ "## Second implementation: Zero shot" ] }, { "cell_type": "markdown", "id": "a454b5ae-f68e-4f0d-9274-7f401420a882", "metadata": {}, "source": [ "Below is another method to do classification which does not require any training set.\n", "It also does not require to train a classifier on top of Mistral.\n", "However, it is much slower to predict, and it is not expected to work as well as the previous method in the case where you have a training dataset available.\n", "\n", "The method consists in prompting the model with the following text:\n", "\n", "```\n", "Symptoms: {symptom}\n", "Disease:\n", "```\n", "\n", "and to evaluate the probability that the model predicts each of the possible labels.\n", "The method is quite slower, because you need to compute the probability of all labels." ] }, { "cell_type": "code", "execution_count": 12, "id": "12546a21-6b0c-4df2-800b-43cb0b753e91", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "240it [00:14, 16.82it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Accuracy: 30.42%\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "model.args.max_batch_size = len(labels)\n", "\n", "test_data = [data[i] for i in permuted[n_train:]]\n", "\n", "good = []\n", "\n", "with torch.no_grad():\n", "\n", " for i, (symptom, disease) in tqdm.tqdm(enumerate(test_data)):\n", " \n", " # given an input, predict the probability of each output (one output corresponds to one label)\n", " input_ids, seq_lengths, output_lengths = [], [], []\n", " for label in labels:\n", " prefix = f\"Symptoms: {symptom}\\nDisease:\"\n", " tokens = tokenizer.encode(f\"{prefix} {label}\", bos=True)\n", " input_ids.extend(tokens)\n", " seq_lengths.append(len(tokens))\n", " output_lengths.append(len(tokens) - len(tokenizer.encode(prefix, bos=True)))\n", " tensor = torch.tensor(input_ids).to(model.device)\n", " logprobs = torch.log_softmax(model.forward(tensor, seq_lengths), dim=-1) # sum_slens * vocab_size\n", " \n", " # select logprobs for each output\n", " offset = 0\n", " avg_logprobs = []\n", " for slen, output_len in zip(seq_lengths, output_lengths):\n", " output_tokens = input_ids[offset + slen - output_len:offset + slen]\n", " output_logprobs = torch.gather(\n", " logprobs[offset + slen - output_len - 1:offset + slen - 1],\n", " dim=1,\n", " index=torch.tensor(output_tokens).to(model.device)[:, None],\n", " ).mean().item()\n", " avg_logprobs.append(output_logprobs)\n", " offset += slen\n", "\n", " good.append(np.argmax(avg_logprobs) == txt_to_label[disease])\n", " \n", "print(f\"Accuracy: {100 * np.mean(good):.2f}%\")" ] }, { "cell_type": "markdown", "id": "cc4157a9-236d-4ef5-a02d-0731ae8e89eb", "metadata": {}, "source": [ "Even though this approach did not require any training set, the performance is only of 30.4%, compared to 98.3% for the previous, faster one." ] }, { "cell_type": "code", "execution_count": null, "id": "7a42ece8-c879-414c-a21a-25843f7d2bd1", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.18" } }, "nbformat": 4, "nbformat_minor": 5 } ================================================ FILE: tutorials/getting_started.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Getting Started with `mistral-inference`\n", "\n", "This notebook will guide you through the process of running Mistral models locally. We will cover the following: \n", "- How to chat with Mistral 7B Instruct\n", "- How to run Mistral 7B Instruct with function calling capabilities\n", "\n", "We recommend using a GPU such as the A100 to run this notebook. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "G6tXvIsQenpI" }, "outputs": [], "source": [ "!pip install mistral-inference" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download Mistral 7B Instruct" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "background_save": true }, "id": "4ytmRt0WQeMW" }, "outputs": [], "source": [ "!wget https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-Instruct-v0.3.tar" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "eRZg_8wvs5A6" }, "outputs": [], "source": [ "!DIR=$HOME/mistral_7b_instruct_v3 && mkdir -p $DIR && tar -xf mistral-7B-Instruct-v0.3.tar -C $DIR" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7CN8gShDf65M" }, "outputs": [], "source": [ "!ls mistral_7b_instruct_v3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Chat with the model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os \n", "\n", "from mistral_inference.transformer import Transformer\n", "from mistral_inference.generate import generate\n", "\n", "from mistral_common.tokens.tokenizers.mistral import MistralTokenizer\n", "from mistral_common.protocol.instruct.messages import UserMessage\n", "from mistral_common.protocol.instruct.request import ChatCompletionRequest\n", "\n", "# load tokenizer\n", "mistral_tokenizer = MistralTokenizer.from_file(os.path.expanduser(\"~\")+\"/mistral_7b_instruct_v3/tokenizer.model.v3\")\n", "# chat completion request\n", "completion_request = ChatCompletionRequest(messages=[UserMessage(content=\"Explain Machine Learning to me in a nutshell.\")])\n", "# encode message\n", "tokens = mistral_tokenizer.encode_chat_completion(completion_request).tokens\n", "# load model\n", "model = Transformer.from_folder(os.path.expanduser(\"~\")+\"/mistral_7b_instruct_v3\")\n", "# generate results\n", "out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=mistral_tokenizer.instruct_tokenizer.tokenizer.eos_id)\n", "# decode generated tokens\n", "result = mistral_tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])\n", "print(result)" ] }, { "cell_type": "markdown", "metadata": { "id": "ce4woS3LkgZ9" }, "source": [ "## Function calling\n", "\n", "Mistral 7B Instruct v3 also supports function calling!" ] }, { "cell_type": "markdown", "metadata": { "id": "TKfPiEwNk1kh" }, "source": [ "Let's start by creating a function calling example" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "0PJdwvDEk3dl" }, "outputs": [], "source": [ "from mistral_common.protocol.instruct.messages import UserMessage\n", "from mistral_common.protocol.instruct.request import ChatCompletionRequest\n", "from mistral_common.protocol.instruct.tool_calls import Function, Tool\n", "\n", "completion_request = ChatCompletionRequest(\n", " tools=[\n", " Tool(\n", " function=Function(\n", " name=\"get_current_weather\",\n", " description=\"Get the current weather\",\n", " parameters={\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"location\": {\n", " \"type\": \"string\",\n", " \"description\": \"The city and state, e.g. San Francisco, CA\",\n", " },\n", " \"format\": {\n", " \"type\": \"string\",\n", " \"enum\": [\"celsius\", \"fahrenheit\"],\n", " \"description\": \"The temperature unit to use. Infer this from the users location.\",\n", " },\n", " },\n", " \"required\": [\"location\", \"format\"],\n", " },\n", " )\n", " )\n", " ],\n", " messages=[\n", " UserMessage(content=\"What's the weather like today in Paris?\"),\n", " ],\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "bG6ZeZUylpBW" }, "source": [ "Since we have already loaded the tokenizer and the model in the example above. We will skip these steps here. \n", "\n", "Now we can encode the message with our tokenizer using `MistralTokenizer`. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Ii8q-JNClwiq" }, "outputs": [], "source": [ "from mistral_common.tokens.tokenizers.mistral import MistralTokenizer\n", "\n", "tokens = mistral_tokenizer.encode_chat_completion(completion_request).tokens" ] }, { "cell_type": "markdown", "metadata": { "id": "NrueDujkmJT4" }, "source": [ "and run `generate` to get a response. Don't forget to pass the EOS id!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "GWJYO43rl0V8" }, "outputs": [], "source": [ "from mistral_inference.generate import generate\n", "\n", "out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=mistral_tokenizer.instruct_tokenizer.tokenizer.eos_id)" ] }, { "cell_type": "markdown", "metadata": { "id": "v7baJ1msmPMv" }, "source": [ "Finally, we can decode the generated tokens." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "RKhryfBWmHon" }, "outputs": [], "source": [ "result = mistral_tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens)[0]\n", "result" ] } ], "metadata": { "accelerator": "GPU", "colab": { "gpuType": "L4", "machine_shape": "hm", "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" } }, "nbformat": 4, "nbformat_minor": 4 }