[
  {
    "path": ".github/ISSUE_TEMPLATE/bug_report.yml",
    "content": "name: Bug report related to mistral-inference\ndescription: Submit a bug report that's related to mistral-inference\ntitle: '[BUG: '\nlabels: ['bug', 'triage']\nbody:\n  - type: markdown\n    attributes:\n      value: |\n        Thanks for taking the time to fill out this bug report!\n  - type: textarea\n    id: python-vv\n    attributes:\n      label: Python -VV\n      description: Run `python -VV` from your virtual environment\n      placeholder: Copy-paste the output (no need for backticks, will be formatted into code automatically)\n      render: shell\n    validations:\n      required: true\n  - type: textarea\n    id: pip-freeze\n    attributes:\n      label: Pip Freeze\n      description: Run `pip freeze` from your virtual environment\n      placeholder: Copy-paste the output (no need for backticks, will be formatted into code automatically)\n      render: shell\n    validations:\n      required: true\n  - type: textarea\n    id: reproduction-steps\n    attributes:\n      label: Reproduction Steps\n      description: Provide a clear and concise description of the steps that lead to your issue.\n      placeholder: |\n        1. First step...\n        2. Step 2...\n        ...\n    validations:\n      required: true\n  - type: textarea\n    id: expected-behavior\n    attributes:\n      label: Expected Behavior\n      description: Explain briefly what you expected to happen.\n    validations:\n      required: true\n  - type: textarea\n    id: additional-context\n    attributes:\n      label: Additional Context\n      description: Add any context about your problem that you deem relevant.\n  - type: textarea\n    id: suggested-solutions\n    attributes:\n      label: Suggested Solutions\n      description: Please list any solutions you recommend we consider.\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/config.yml",
    "content": "blank_issues_enabled: false\ncontact_links:\n  - name: Documentation\n    url: https://docs.mistral.ai\n    about: Developer documentation for the Mistral AI platform\n  - name: Discord\n    url: https://discord.com/invite/mistralai)\n    about: Chat with the Mistral community\n"
  },
  {
    "path": ".gitignore",
    "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n*.py,cover\n.hypothesis/\n.pytest_cache/\ncover/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\ndb.sqlite3-journal\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\n.pybuilder/\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n#   For a library or package, you might want to ignore these files since the code is\n#   intended to run in multiple environments; otherwise, check them in:\n# .python-version\n\n# pipenv\n#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n#   However, in case of collaboration, if having platform-specific dependencies or dependencies\n#   having no cross-platform support, pipenv may install dependencies that don't work, or not\n#   install all needed dependencies.\n#Pipfile.lock\n\n# poetry\n#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.\n#   This is especially recommended for binary packages to ensure reproducibility, and is more\n#   commonly ignored for libraries.\n#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control\n#poetry.lock\n\n# pdm\n#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.\n#pdm.lock\n#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it\n#   in version control.\n#   https://pdm.fming.dev/#use-with-ide\n.pdm.toml\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm\n__pypackages__/\n\n# Celery stuff\ncelerybeat-schedule\ncelerybeat.pid\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n\n# pytype static type analyzer\n.pytype/\n\n# Cython debug symbols\ncython_debug/\n\n# PyCharm\n#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can\n#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore\n#  and can be added to the global gitignore or merged into this file.  For a more nuclear\n#  option (not recommended) you can uncomment the following to ignore the entire idea folder.\n#.idea/\n"
  },
  {
    "path": "LICENSE",
    "content": "                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "README.md",
    "content": "# Mistral Inference\n<a target=\"_blank\" href=\"https://colab.research.google.com/github/mistralai/mistral-inference/blob/main/tutorials/getting_started.ipynb\">\n  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n</a>\n\n\nThis repository contains minimal code to run Mistral models.\n\nBlog 7B: [https://mistral.ai/news/announcing-mistral-7b/](https://mistral.ai/news/announcing-mistral-7b/)\\\nBlog 8x7B: [https://mistral.ai/news/mixtral-of-experts/](https://mistral.ai/news/mixtral-of-experts/)\\\nBlog 8x22B: [https://mistral.ai/news/mixtral-8x22b/](https://mistral.ai/news/mixtral-8x22b/)\\\nBlog Codestral 22B: [https://mistral.ai/news/codestral](https://mistral.ai/news/codestral/) \\\nBlog Codestral Mamba 7B: [https://mistral.ai/news/codestral-mamba/](https://mistral.ai/news/codestral-mamba/) \\\nBlog Mathstral 7B: [https://mistral.ai/news/mathstral/](https://mistral.ai/news/mathstral/) \\\nBlog Nemo: [https://mistral.ai/news/mistral-nemo/](https://mistral.ai/news/mistral-nemo/) \\\nBlog Mistral Large 2: [https://mistral.ai/news/mistral-large-2407/](https://mistral.ai/news/mistral-large-2407/) \\\nBlog Pixtral 12B: [https://mistral.ai/news/pixtral-12b/](https://mistral.ai/news/pixtral-12b/)\nBlog Mistral Small 3.1: [https://mistral.ai/news/mistral-small-3-1/](https://mistral.ai/news/mistral-small-3-1/)\n\nDiscord: [https://discord.com/invite/mistralai](https://discord.com/invite/mistralai)\\\nDocumentation: [https://docs.mistral.ai/](https://docs.mistral.ai/)\\\nGuardrailing: [https://docs.mistral.ai/usage/guardrailing](https://docs.mistral.ai/usage/guardrailing)\n\n## Installation\n\nNote: You will use a GPU to install `mistral-inference`, as it currently requires `xformers` to be installed and `xformers` itself needs a GPU for installation.\n\n### PyPI\n\n```\npip install mistral-inference\n```\n\n### Local\n\n```\ncd $HOME && git clone https://github.com/mistralai/mistral-inference\ncd $HOME/mistral-inference && poetry install .\n```\n\n## Model download\n\n### Direct links\n\n| Name        | Download | md5sum |\n|-------------|-------|-------|\n| 7B Instruct | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-Instruct-v0.3.tar | `80b71fcb6416085bcb4efad86dfb4d52` |\n| 8x7B Instruct | https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar (**Updated model coming soon!**) | `8e2d3930145dc43d3084396f49d38a3f` |\n| 8x22 Instruct | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-Instruct-v0.3.tar | `471a02a6902706a2f1e44a693813855b` |\n| 7B Base | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-v0.3.tar | `0663b293810d7571dad25dae2f2a5806` |\n| 8x7B |     **Updated model coming soon!**       | - |\n| 8x22B | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-v0.3.tar | `a2fa75117174f87d1197e3a4eb50371a` |\n| Codestral 22B | https://models.mistralcdn.com/codestral-22b-v0-1/codestral-22B-v0.1.tar | `1ea95d474a1d374b1d1b20a8e0159de3` |\n| Mathstral 7B | https://models.mistralcdn.com/mathstral-7b-v0-1/mathstral-7B-v0.1.tar | `5f05443e94489c261462794b1016f10b` |\n| Codestral-Mamba 7B | https://models.mistralcdn.com/codestral-mamba-7b-v0-1/codestral-mamba-7B-v0.1.tar | `d3993e4024d1395910c55db0d11db163` |\n| Nemo Base | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-base-2407.tar | `c5d079ac4b55fc1ae35f51f0a3c0eb83` |\n| Nemo Instruct | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar | `296fbdf911cb88e6f0be74cd04827fe7` |\n| Mistral Large 2 | https://models.mistralcdn.com/mistral-large-2407/mistral-large-instruct-2407.tar | `fc602155f9e39151fba81fcaab2fa7c4` |\n\nNote:\n- **Important**:\n  - `mixtral-8x22B-Instruct-v0.3.tar` is exactly the same as [Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), only stored in `.safetensors` format\n  - `mixtral-8x22B-v0.3.tar` is the same as [Mixtral-8x22B-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1), but has an extended vocabulary of 32768 tokens.\n  - `codestral-22B-v0.1.tar` has a custom non-commercial license, called [Mistral AI Non-Production (MNPL) License](https://mistral.ai/licenses/MNPL-0.1.md)\n  - `mistral-large-instruct-2407.tar` has a custom non-commercial license, called [Mistral AI Research (MRL) License](https://mistral.ai/licenses/MRL-0.1.md)\n- All of the listed models above support function calling. For example, Mistral 7B Base/Instruct v3 is a minor update to Mistral 7B Base/Instruct v2,  with the addition of function calling capabilities.\n- The \"coming soon\" models will include function calling as well.\n- You can download the previous versions of our models from our [docs](https://docs.mistral.ai/getting-started/open_weight_models/#downloading).\n\n### From Hugging Face Hub\n\n| Name        | ID | URL |\n|-------------|-------|-------|\n| Pixtral Large Instruct | mistralai/Pixtral-Large-Instruct-2411 | https://huggingface.co/mistralai/Pixtral-Large-Instruct-2411 |\n| Pixtral 12B Base | mistralai/Pixtral-12B-Base-2409 | https://huggingface.co/mistralai/Pixtral-12B-Base-2409 |\n| Pixtral 12B | mistralai/Pixtral-12B-2409 | https://huggingface.co/mistralai/Pixtral-12B-2409 |\n| Mistral Small 3.1 24B Base | mistralai/Mistral-Small-3.1-24B-Base-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503\n| Mistral Small 3.1 24B Instruct | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503 |\n\n\n### Usage\n\n**News!!!**: Mistral Large 2 is out. Read more about its capabilities [here](https://mistral.ai/news/mistral-large-2407/).\n\nCreate a local folder to store models\n```sh\nexport MISTRAL_MODEL=$HOME/mistral_models\nmkdir -p $MISTRAL_MODEL\n```\n\nDownload any of the above links and extract the content, *e.g.*:\n\n```sh\nexport 12B_DIR=$MISTRAL_MODEL/12B_Nemo\nwget https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar\nmkdir -p $12B_DIR\ntar -xf mistral-nemo-instruct-2407.tar -C $12B_DIR\n```\n\nor\n\n```sh\nexport M8x7B_DIR=$MISTRAL_MODEL/8x7b_instruct\nwget https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar\nmkdir -p $M8x7B_DIR\ntar -xf Mixtral-8x7B-v0.1-Instruct.tar -C $M8x7B_DIR\n```\n\nFor Hugging Face models' weights, here is an example to download [Mistral Small 3.1 24B Instruct](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503):\n\n```python\nfrom pathlib import Path\nfrom huggingface_hub import snapshot_download\n\n\nmistral_models_path = Path.home().joinpath(\"mistral_models\")\n\nmodel_path = mistral_models_path / \"mistral-small-3.1-instruct\"\nmodel_path.mkdir(parents=True, exist_ok=True)\n\nrepo_id = \"mistralai/Mistral-Small-3.1-24B-Instruct-2503\"\n\nsnapshot_download(\n    repo_id=repo_id,\n    allow_patterns=[\"params.json\", \"consolidated.safetensors\", \"tekken.json\"],\n    local_dir=model_path,\n)\n```\n\n## Usage\n\nThe following sections give an overview of how to run the model from the Command-line interface (CLI) or directly within Python.\n\n### CLI\n\n- **Demo**\n\nTo test that a model works in your setup, you can run the `mistral-demo` command.\n*E.g.* the 12B Mistral-Nemo model can be tested on a single GPU as follows:\n\n```sh\nmistral-demo $12B_DIR\n```\n\nLarge models, such **8x7B** and **8x22B** have to be run in a multi-GPU setup.\nFor these models, you can use the following command:\n\n```sh\ntorchrun --nproc-per-node 2 --no-python mistral-demo $M8x7B_DIR\n```\n\n*Note*: Change `--nproc-per-node` to more GPUs if available.\n\n- **Chat**\n\nTo interactively chat with the models, you can make use of the `mistral-chat` command.\n\n```sh\nmistral-chat $12B_DIR --instruct --max_tokens 1024 --temperature 0.35\n```\n\nFor large models, you can make use of `torchrun`.\n\n```sh\ntorchrun --nproc-per-node 2 --no-python mistral-chat $M8x7B_DIR --instruct\n```\n\n*Note*: Change `--nproc-per-node` to more GPUs if necessary (*e.g.* for 8x22B).\n\n- **Chat with Codestral**\n\nTo use [Codestral](https://mistral.ai/news/codestral/) as a coding assistant you can run the following command using `mistral-chat`.\nMake sure `$M22B_CODESTRAL` is set to a valid path to the downloaded codestral folder, e.g. `$HOME/mistral_models/Codestral-22B-v0.1`\n\n```sh\nmistral-chat $M22B_CODESTRAL --instruct --max_tokens 256\n```\n\nIf you prompt it with *\"Write me a function that computes fibonacci in Rust\"*, the model should generate something along the following lines:\n\n```sh\nSure, here's a simple implementation of a function that computes the Fibonacci sequence in Rust. This function takes an integer `n` as an argument and returns the `n`th Fibonacci number.\n\nfn fibonacci(n: u32) -> u32 {\n    match n {\n        0 => 0,\n        1 => 1,\n        _ => fibonacci(n - 1) + fibonacci(n - 2),\n    }\n}\n\nfn main() {\n    let n = 10;\n    println!(\"The {}th Fibonacci number is: {}\", n, fibonacci(n));\n}\n\nThis function uses recursion to calculate the Fibonacci number. However, it's not the most efficient solution because it performs a lot of redundant calculations. A more efficient solution would use a loop to iteratively calculate the Fibonacci numbers.\n```\n\nYou can continue chatting afterwards, *e.g.* with *\"Translate it to Python\"*.\n\n- **Chat with Codestral-Mamba**\n\nTo use [Codestral-Mamba](https://mistral.ai/news/codestral-mamba/) as a coding assistant you can run the following command using `mistral-chat`.\nMake sure `$7B_CODESTRAL_MAMBA` is set to a valid path to the downloaded codestral-mamba folder, e.g. `$HOME/mistral_models/mamba-codestral-7B-v0.1`.\n\nYou then need to additionally install the following packages:\n\n```\npip install packaging mamba-ssm causal-conv1d transformers\n```\n\nbefore you can start chatting:\n\n```sh\nmistral-chat $7B_CODESTRAL_MAMBA --instruct --max_tokens 256\n```\n\n- **Chat with Mathstral**\n\nTo use [Mathstral](https://mistral.ai/news/mathstral/) as an assistant you can run the following command using `mistral-chat`.\nMake sure `$7B_MATHSTRAL` is set to a valid path to the downloaded codestral folder, e.g. `$HOME/mistral_models/mathstral-7B-v0.1`\n\n```sh\nmistral-chat $7B_MATHSTRAL --instruct --max_tokens 256\n```\n\nIf you prompt it with *\"Albert likes to surf every week. Each surfing session lasts for 4 hours and costs $20 per hour. How much would Albert spend in 5 weeks?\"*, the model should answer with the correct calculation.\n\nYou can then continue chatting afterwards, *e.g.* with *\"How much would he spend in a year?\"*.\n\n- **Chat with Mistral Small 3.1 24B Instruct**\n\nTo use [Mistral Small 3.1 24B Instruct](https://mistral.ai/news/mistral-small-3-1/) as an assistant you can run the following command using `mistral-chat`.\nMake sure `$MISTRAL_SMALL_3_1_INSTRUCT` is set to a valid path to the downloaded mistral small folder, e.g. `$HOME/mistral_models/mistral-small-3.1-instruct`\n\n```sh\n    mistral-chat $MISTRAL_SMALL_3_1_INSTRUCT --instruct --max_tokens 256\n```\n\nIf you prompt it with *\"The above image presents an image of which park ? Please give the hints to identify the park.\"* with the following image URL *https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png*, the model should answer with the Yosemite park and give hints to identify it.\n\nYou can then continue chatting afterwards, *e.g.* with *\"What is the name of the lake in the image?\"*. The model should respond that it is not a lake but a river.\n\n### Python\n\n- *Instruction Following*:\n\n```py\nfrom mistral_inference.transformer import Transformer\nfrom mistral_inference.generate import generate\n\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_common.protocol.instruct.messages import UserMessage\nfrom mistral_common.protocol.instruct.request import ChatCompletionRequest\n\n\ntokenizer = MistralTokenizer.from_file(\"./mistral-nemo-instruct-v0.1/tekken.json\")  # change to extracted tokenizer file\nmodel = Transformer.from_folder(\"./mistral-nemo-instruct-v0.1\")  # change to extracted model dir\n\nprompt = \"How expensive would it be to ask a window cleaner to clean all windows in Paris. Make a reasonable guess in US Dollar.\"\n\ncompletion_request = ChatCompletionRequest(messages=[UserMessage(content=prompt)])\n\ntokens = tokenizer.encode_chat_completion(completion_request).tokens\n\nout_tokens, _ = generate([tokens], model, max_tokens=1024, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\nresult = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])\n\nprint(result)\n```\n\n- *Multimodal Instruction Following*:\n\n\n```python\nfrom pathlib import Path\n\nfrom huggingface_hub import snapshot_download\nfrom mistral_common.protocol.instruct.messages import ImageURLChunk, TextChunk\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_inference.generate import generate\nfrom mistral_inference.transformer import Transformer\n\nmodel_path = Path.home().joinpath(\"mistral_models\") / \"mistral-small-3.1-instruct\" # change to extracted model\n\ntokenizer = MistralTokenizer.from_file(model_path / \"tekken.json\")\nmodel = Transformer.from_folder(model_path)\n\nurl = \"https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png\"\nprompt = \"The above image presents an image of which park ? Please give the hints to identify the park.\"\n\nuser_content = [ImageURLChunk(image_url=url), TextChunk(text=prompt)]\n\ntokens, images = tokenizer.instruct_tokenizer.encode_user_content(user_content, False)\n\nout_tokens, _ = generate(\n    [tokens],\n    model,\n    images=[images],\n    max_tokens=256,\n    temperature=0.15,\n    eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id,\n)\nresult = tokenizer.decode(out_tokens[0])\n\nprint(\"Prompt:\", prompt)\nprint(\"Completion:\", result)\n```\n\n- *Function Calling*:\n\n```py\nfrom mistral_common.protocol.instruct.tool_calls import Function, Tool\n\ncompletion_request = ChatCompletionRequest(\n    tools=[\n        Tool(\n            function=Function(\n                name=\"get_current_weather\",\n                description=\"Get the current weather\",\n                parameters={\n                    \"type\": \"object\",\n                    \"properties\": {\n                        \"location\": {\n                            \"type\": \"string\",\n                            \"description\": \"The city and state, e.g. San Francisco, CA\",\n                        },\n                        \"format\": {\n                            \"type\": \"string\",\n                            \"enum\": [\"celsius\", \"fahrenheit\"],\n                            \"description\": \"The temperature unit to use. Infer this from the users location.\",\n                        },\n                    },\n                    \"required\": [\"location\", \"format\"],\n                },\n            )\n        )\n    ],\n    messages=[\n        UserMessage(content=\"What's the weather like today in Paris?\"),\n        ],\n)\n\ntokens = tokenizer.encode_chat_completion(completion_request).tokens\n\nout_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\nresult = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])\n\nprint(result)\n```\n\n- *Fill-in-the-middle (FIM)*:\n\nMake sure to have `mistral-common >= 1.2.0` installed:\n```\npip install --upgrade mistral-common\n```\n\nYou can simulate a code completion in-filling as follows.\n\n```py\nfrom mistral_inference.transformer import Transformer\nfrom mistral_inference.generate import generate\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_common.tokens.instruct.request import FIMRequest\n\ntokenizer = MistralTokenizer.from_model(\"codestral-22b\")\nmodel = Transformer.from_folder(\"./mistral_22b_codestral\")\n\nprefix = \"\"\"def add(\"\"\"\nsuffix = \"\"\"    return sum\"\"\"\n\nrequest = FIMRequest(prompt=prefix, suffix=suffix)\n\ntokens = tokenizer.encode_fim(request).tokens\n\nout_tokens, _ = generate([tokens], model, max_tokens=256, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\nresult = tokenizer.decode(out_tokens[0])\n\nmiddle = result.split(suffix)[0].strip()\nprint(middle)\n```\n\n### Test\n\nTo run logits equivalence:\n```\npython -m pytest tests\n```\n\n## Deployment\n\nThe `deploy` folder contains code to build a [vLLM](https://M7B_DIR.com/vllm-project/vllm) image with the required dependencies to serve the Mistral AI model. In the image, the [transformers](https://github.com/huggingface/transformers/) library is used instead of the reference implementation. To build it:\n\n```bash\ndocker build deploy --build-arg MAX_JOBS=8\n```\n\nInstructions to run the image can be found in the [official documentation](https://docs.mistral.ai/quickstart).\n\n\n## Model platforms\n\n- Use Mistral models on [Mistral AI official API](https://console.mistral.ai/) (La Plateforme)\n- Use Mistral models via [cloud providers](https://docs.mistral.ai/deployment/cloud/overview/)\n\n## References\n\n[1]: [LoRA](https://arxiv.org/abs/2106.09685): Low-Rank Adaptation of Large Language Models, Hu et al. 2021\n\n## License\n\nThis library is licensed under the Apache 2.0 License. See the [LICENCE](./LICENCE) file for more information.\n\n*You must not use this library or our models in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.*\n"
  },
  {
    "path": "deploy/.dockerignore",
    "content": "*\n!entrypoint.sh\n"
  },
  {
    "path": "deploy/Dockerfile",
    "content": "FROM --platform=amd64 nvcr.io/nvidia/cuda:12.1.0-devel-ubuntu22.04 as base\n\nWORKDIR /workspace\n\nRUN apt update && \\\n    apt install -y python3-pip python3-packaging \\\n    git ninja-build && \\\n    pip3 install -U pip\n\n# Tweak this list to reduce build time\n# https://developer.nvidia.com/cuda-gpus\nENV TORCH_CUDA_ARCH_LIST \"7.0;7.2;7.5;8.0;8.6;8.9;9.0\"\n\nRUN pip3 install \"torch==2.1.1\"\n\n# This build is slow but NVIDIA does not provide binaries. Increase MAX_JOBS as needed.\nRUN pip3 install \"git+https://github.com/stanford-futuredata/megablocks.git\"\nRUN pip3 install \"git+https://github.com/vllm-project/vllm.git\"\nRUN pip3 install \"xformers==0.0.23\" \"transformers==4.36.0\" \"fschat[model_worker]==0.2.34\"\n\nRUN git clone https://github.com/NVIDIA/apex && \\\n    cd apex && git checkout 2386a912164b0c5cfcd8be7a2b890fbac5607c82 && \\\n    sed -i '/check_cuda_torch_binary_vs_bare_metal(CUDA_HOME)/d' setup.py && \\\n    python3 setup.py install --cpp_ext --cuda_ext\n\n\nCOPY entrypoint.sh .\n\nRUN chmod +x /workspace/entrypoint.sh\n\nENTRYPOINT [\"/workspace/entrypoint.sh\"]"
  },
  {
    "path": "deploy/entrypoint.sh",
    "content": "#!/bin/bash\n\nif [[ ! -z \"${HF_TOKEN}\" ]]; then\n    echo \"The HF_TOKEN environment variable is set, logging to Hugging Face.\"\n    python3 -c \"import huggingface_hub; huggingface_hub.login('${HF_TOKEN}')\"\nelse\n    echo \"The HF_TOKEN environment variable is not set or empty, not logging to Hugging Face.\"\nfi\n\n# Run the provided command\nexec python3 -u -m vllm.entrypoints.openai.api_server \"$@\"\n"
  },
  {
    "path": "pyproject.toml",
    "content": "[tool.poetry]\nname = \"mistral_inference\"\nversion = \"1.6.0\"\ndescription = \"\"\nauthors = [\"bam4d <bam4d@mistral.ai>\"]\nreadme = \"README.md\"\npackages = [{ include = \"mistral_inference\", from = \"src\" }]\n\n[tool.ruff]\nlint.select = [\"E\", \"F\", \"W\", \"Q\", \"I\"]\nlint.ignore = [\"E203\"]\nlint.fixable = [\"ALL\"]\nlint.unfixable = []\nline-length = 120\nexclude = [\"docs\", \"build\", \"tutorials\"]\n\n[tool.mypy]\ndisallow_untyped_defs = true\nshow_error_codes = true\nno_implicit_optional = true\nwarn_return_any = true\nwarn_unused_ignores = true\nexclude = [\"docs\", \"tools\", \"build\"]\n\n[tool.poetry.dependencies]\npython = \"^3.9.10\"\nxformers = \">=0.0.24\"\nsimple-parsing = \">=0.1.5\"\nfire = \">=0.6.0\"\nmistral_common = \">=1.5.4\"\nsafetensors = \">=0.4.0\"\npillow = \">=10.3.0\"\n\n[tool.poetry.group.dev.dependencies]\ntypes-protobuf = \"4.24.0.20240129\"\nmypy-protobuf = \"^3.5.0\"\npytest = \"7.4.4\"\nruff = \"^0.2.2\"\nmypy = \"^1.8.0\"\n\n[build-system]\nrequires = [\"poetry-core>=1.0.0\"]\nbuild-backend = \"poetry.core.masonry.api\"\n\n[tool.pytest.ini_options]\ntestpaths = [\"./tests\"]\n\n[tool.poetry.scripts]\nmistral-chat = \"mistral_inference.main:mistral_chat\"\nmistral-demo = \"mistral_inference.main:mistral_demo\"\n"
  },
  {
    "path": "src/mistral_inference/__init__.py",
    "content": "__version__ = \"1.6.0\"\n"
  },
  {
    "path": "src/mistral_inference/args.py",
    "content": "from dataclasses import dataclass\nfrom typing import List, Optional\n\nfrom simple_parsing.helpers import Serializable\n\nfrom mistral_inference.lora import LoraArgs\nfrom mistral_inference.moe import MoeArgs\n\nPATCH_MERGE = \"patch_merge\"\n\n\n@dataclass\nclass VisionEncoderArgs:\n    hidden_size: int\n    num_channels: int\n    image_size: int\n    patch_size: int\n    intermediate_size: int\n    num_hidden_layers: int\n    num_attention_heads: int\n    rope_theta: float = 1e4  # for rope-2D\n    image_token_id: int = 10\n    adapter_bias: bool = True\n    spatial_merge_size: int = 1\n    add_pre_mm_projector_layer_norm: bool = False\n    mm_projector_id: str = \"\"\n\n\n@dataclass\nclass TransformerArgs(Serializable):\n    dim: int\n    n_layers: int\n    head_dim: int\n    hidden_dim: int\n    n_heads: int\n    n_kv_heads: int\n    norm_eps: float\n    vocab_size: int\n\n    max_batch_size: int = 0\n\n    # For rotary embeddings. If not set, will be inferred\n    rope_theta: Optional[float] = None\n    # If this is set, we will use MoE layers instead of dense layers.\n    moe: Optional[MoeArgs] = None\n    # If this is set, we will load LoRA linear layers instead of linear layers.\n    lora: Optional[LoraArgs] = None\n    sliding_window: Optional[int] | Optional[List[int]] = None\n    _sliding_window: Optional[int] | Optional[List[int]] = None\n    model_type: str = \"transformer\"\n\n    vision_encoder: Optional[VisionEncoderArgs] = None\n\n    def __post_init__(self) -> None:\n        assert self.model_type == \"transformer\", self.model_type\n        assert self.sliding_window is None or self._sliding_window is None\n\n        # hack for now so that vLLM is supported correctly\n        self.sliding_window = self.sliding_window if self.sliding_window is not None else self._sliding_window\n\n\n@dataclass\nclass MambaArgs(Serializable):\n    dim: int\n    n_layers: int\n    vocab_size: int\n    n_groups: int\n    rms_norm: bool\n    residual_in_fp32: bool\n    fused_add_norm: bool\n    pad_vocab_size_multiple: int\n    tie_embeddings: bool\n    model_type: str = \"mamba\"\n\n    def __post_init__(self) -> None:\n        assert self.model_type == \"mamba\", self.model_type\n"
  },
  {
    "path": "src/mistral_inference/cache.py",
    "content": "from dataclasses import dataclass\nfrom typing import List, Optional, Tuple\n\nimport torch\nfrom xformers.ops.fmha.attn_bias import (  # type: ignore\n    AttentionBias,\n    BlockDiagonalCausalMask,\n    BlockDiagonalCausalWithOffsetPaddedKeysMask,\n    BlockDiagonalMask,\n)\n\n\ndef get_cache_sizes(n_layers: int, max_seq_len: int, sliding_window: Optional[int] | Optional[List[int]]) -> List[int]:\n    if sliding_window is None:\n        return n_layers * [max_seq_len]\n    elif isinstance(sliding_window, int):\n        return n_layers * [sliding_window]\n    else:\n        assert isinstance(sliding_window, list), f\"Expected list, got {type(sliding_window)}\"\n        assert (\n            n_layers % len(sliding_window) == 0\n        ), f\"Expected n_layers % len(sliding_window) == 0, got {n_layers} % {len(sliding_window)}\"\n        num_repeats = n_layers // len(sliding_window)\n        return num_repeats * [w if w is not None else max_seq_len for w in sliding_window]\n\n\n@dataclass\nclass CacheInputMetadata:\n    # # rope absolute positions\n    # positions: torch.Tensor\n    # # where tokens should go in the cache\n    # cache_positions: torch.Tensor\n\n    # # if prefill, use block diagonal causal mask\n    # # else use causal with padded key mask\n    # prefill: bool\n    # mask: AttentionBias\n    # seqlens: List[int]\n    # rope absolute positions\n    positions: torch.Tensor\n    # which elements in the sequences need to be cached\n    to_cache_mask: torch.Tensor\n    # how many elements are cached per sequence\n    cached_elements: torch.Tensor\n    # where tokens should go in the cache\n    cache_positions: torch.Tensor\n    # if prefill, use block diagonal causal mask\n    # else use causal with padded key mask\n    prefill: bool\n    mask: AttentionBias\n    seqlens: List[int]\n\n\ndef interleave_list(l1: List[torch.Tensor], l2: List[torch.Tensor]) -> List[torch.Tensor]:\n    assert len(l1) == len(l2)\n    return [v for pair in zip(l1, l2) for v in pair]\n\n\ndef unrotate(cache: torch.Tensor, seqlen: int) -> torch.Tensor:\n    assert cache.ndim == 3  # (W, H, D)\n    position = seqlen % cache.shape[0]\n    if seqlen < cache.shape[0]:\n        return cache[:seqlen]\n    elif position == 0:\n        return cache\n    else:\n        return torch.cat([cache[position:], cache[:position]], dim=0)\n\n\nclass CacheView:\n    def __init__(\n        self,\n        cache_k: torch.Tensor,\n        cache_v: torch.Tensor,\n        metadata: CacheInputMetadata,\n        kv_seqlens: torch.Tensor,\n    ):\n        self.cache_k = cache_k\n        self.cache_v = cache_v\n        self.kv_seqlens = kv_seqlens\n        self.metadata = metadata\n\n    def update(self, xk: torch.Tensor, xv: torch.Tensor) -> None:\n        \"\"\"\n        to_cache_mask masks the last [max_seq_len] tokens in each sequence\n        \"\"\"\n        n_kv_heads, head_dim = self.cache_k.shape[-2:]\n        flat_cache_k = self.cache_k.view(-1, n_kv_heads, head_dim)\n        flat_cache_v = self.cache_v.view(-1, n_kv_heads, head_dim)\n\n        flat_cache_k.index_copy_(0, self.metadata.cache_positions, xk[self.metadata.to_cache_mask])\n        flat_cache_v.index_copy_(0, self.metadata.cache_positions, xv[self.metadata.to_cache_mask])\n\n    def interleave_kv(self, xk: torch.Tensor, xv: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:\n        \"\"\"\n        This is a naive implementation and not optimized for speed.\n        \"\"\"\n        assert xk.ndim == xv.ndim == 3  # (B * T, H, D)\n        assert xk.shape == xv.shape\n\n        if all([s == 0 for s in self.metadata.seqlens]):\n            # No cache to interleave\n            return xk, xv\n\n        # Make it a list of [(T, H, D)]\n        xk: Tuple[torch.Tensor] = torch.split(xk, self.metadata.seqlens)  # type: ignore\n        xv: Tuple[torch.Tensor] = torch.split(xv, self.metadata.seqlens)  # type: ignore\n        assert len(xk) == len(self.kv_seqlens), f\"Batch size is {len(self.kv_seqlens)}, got {len(xk)}\"\n\n        # Order elements in cache by position by unrotating\n        cache_k = [unrotate(t, s) for t, s in zip(self.cache_k, self.kv_seqlens)]\n        cache_v = [unrotate(t, s) for t, s in zip(self.cache_v, self.kv_seqlens)]\n\n        interleaved_k = interleave_list(cache_k, list(xk))\n        interleaved_v = interleave_list(cache_v, list(xv))\n\n        return torch.cat(interleaved_k, dim=0), torch.cat(interleaved_v, dim=0)\n\n    @property\n    def max_seq_len(self) -> int:\n        return self.cache_k.shape[1]\n\n    @property\n    def key(self) -> torch.Tensor:\n        return self.cache_k[: len(self.kv_seqlens)]\n\n    @property\n    def value(self) -> torch.Tensor:\n        return self.cache_v[: len(self.kv_seqlens)]\n\n    @property\n    def prefill(self) -> bool:\n        return self.metadata.prefill\n\n    @property\n    def mask(self) -> AttentionBias:\n        return self.metadata.mask\n\n\nclass BufferCache:\n    \"\"\"\n    This is an example that implements a buffer cache, allowing for variable length sequences.\n    Allocated cache is rectangular which is wasteful (see PagedAttention for better mechanisms)\n    \"\"\"\n\n    def __init__(\n        self,\n        n_layers: int,\n        max_batch_size: int,\n        max_seq_len: int,\n        n_kv_heads: int,\n        head_dim: int,\n        sliding_window: Optional[int] | Optional[List[int]] = None,\n    ):\n        self.max_seq_len = max_seq_len\n        self.n_kv_heads = n_kv_heads\n        self.head_dim = head_dim\n        self.n_layers = n_layers\n\n        self.cache_sizes: List[int] = get_cache_sizes(n_layers, max_seq_len, sliding_window)\n        assert len(self.cache_sizes) == n_layers, f\"Expected {n_layers} cache sizes, got {len(self.cache_sizes)}\"\n\n        self.cache_k = {}\n        self.cache_v = {}\n        for i, cache_size in enumerate(self.cache_sizes):\n            self.cache_k[i] = torch.empty((max_batch_size, cache_size, n_kv_heads, head_dim))\n            self.cache_v[i] = torch.empty((max_batch_size, cache_size, n_kv_heads, head_dim))\n\n        # holds the valid length for each batch element in the cache\n        self.kv_seqlens: Optional[torch.Tensor] = None\n\n    def get_view(self, layer_id: int, metadata: CacheInputMetadata) -> CacheView:\n        assert self.kv_seqlens is not None\n        return CacheView(self.cache_k[layer_id], self.cache_v[layer_id], metadata, self.kv_seqlens)\n\n    def reset(self) -> None:\n        self.kv_seqlens = None\n\n    def init_kvseqlens(self, batch_size: int) -> None:\n        self.kv_seqlens = torch.zeros((batch_size,), device=self.device, dtype=torch.long)\n\n    @property\n    def device(self) -> torch.device:\n        return self.cache_k[0].device\n\n    def to(self, device: torch.device, dtype: torch.dtype) -> \"BufferCache\":\n        for i in range(self.n_layers):\n            self.cache_k[i] = self.cache_k[i].to(device=device, dtype=dtype)\n            self.cache_v[i] = self.cache_v[i].to(device=device, dtype=dtype)\n\n        return self\n\n    def update_seqlens(self, seqlens: List[int]) -> None:\n        assert self.kv_seqlens is not None\n        self.kv_seqlens += torch.tensor(seqlens, device=self.device, dtype=torch.long)\n\n    def get_input_metadata(self, seqlens: List[int]) -> List[CacheInputMetadata]:\n        \"\"\"\n        input = seqlens [5,7,2] // seqpos [0, 1, 3] // sliding_window 3\n        --> only cache last 3 tokens in each sequence\n        - to_cache_mask = [0 0 1 1 1 | 0 0 0 0 1 1 1 | 1 1]\n        - cached_elements = [3 | 3 | 2]\n        --> absolute positions are used for rope\n        - positions = [0 1 2 3 4 | 1 2 3 4 5 6 7 | 3 4]\n        --> cache positions are positions cache_masked, modulo sliding_window + batch_idx * sliding_window\n        - cache_positions = [2 0 1 | 5 3 4 | 6 7]\n        \"\"\"\n        metadata: List[CacheInputMetadata] = []\n\n        if self.kv_seqlens is None:\n            self.init_kvseqlens(len(seqlens))\n\n        assert self.kv_seqlens is not None\n        assert len(seqlens) == len(\n            self.kv_seqlens\n        ), f\"Batch size is {len(self.kv_seqlens)}, got {len(seqlens)}, did you forget to reset cache?\"\n        seqpos = self.kv_seqlens.tolist()\n        assert len(seqlens) > 0, seqlens\n\n        for cache_size in self.cache_sizes:\n            metadata.append(self._get_input_metadata_layer(cache_size, seqlens, seqpos))\n\n        return metadata\n\n    def _get_input_metadata_layer(self, cache_size: int, seqlens: List[int], seqpos: List[int]) -> CacheInputMetadata:\n        masks = [[x >= seqlen - cache_size for x in range(seqlen)] for seqlen in seqlens]\n        to_cache_mask = torch.tensor(sum(masks, []), device=self.device, dtype=torch.bool)\n        cached_elements = torch.tensor([sum(mask) for mask in masks], device=self.device, dtype=torch.long)\n        positions = torch.cat([torch.arange(pos, pos + seqlen) for pos, seqlen in zip(seqpos, seqlens)]).to(\n            device=self.device, dtype=torch.long\n        )\n        batch_idx = torch.tensor(\n            sum([[i] * seqlen for i, seqlen in enumerate(seqlens)], []), device=self.device, dtype=torch.long\n        )\n        cache_positions = positions % cache_size + batch_idx * cache_size\n        first_prefill = seqpos[0] == 0\n        subsequent_prefill = any(seqlen > 1 for seqlen in seqlens)\n        if first_prefill:\n            assert all([pos == 0 for pos in seqpos]), seqpos\n            mask = BlockDiagonalCausalMask.from_seqlens(seqlens).make_local_attention(cache_size)\n        elif subsequent_prefill:\n            assert self.kv_seqlens is not None\n            mask = BlockDiagonalMask.from_seqlens(\n                q_seqlen=seqlens,\n                kv_seqlen=[\n                    s + cached_s.clamp(max=cache_size).item() for (s, cached_s) in zip(seqlens, self.kv_seqlens)\n                ],\n            ).make_local_attention_from_bottomright(cache_size)\n        else:\n            mask = BlockDiagonalCausalWithOffsetPaddedKeysMask.from_seqlens(\n                q_seqlen=seqlens,\n                kv_padding=cache_size,\n                kv_seqlen=(self.kv_seqlens + cached_elements).clamp(max=cache_size).tolist(),\n            )\n        return CacheInputMetadata(\n            positions=positions,\n            to_cache_mask=to_cache_mask,\n            cached_elements=cached_elements,\n            cache_positions=cache_positions[to_cache_mask],\n            prefill=first_prefill or subsequent_prefill,\n            mask=mask,\n            seqlens=seqlens,\n        )\n"
  },
  {
    "path": "src/mistral_inference/generate.py",
    "content": "from typing import List, Optional, Tuple\n\nimport numpy as np\nimport torch\n\nfrom mistral_inference.cache import BufferCache\nfrom mistral_inference.mamba import Mamba\nfrom mistral_inference.transformer import Transformer\n\n\n@torch.inference_mode()\ndef generate_mamba(\n    encoded_prompts: List[List[int]],\n    model: Mamba,\n    *,\n    max_tokens: int,\n    temperature: float,\n    chunk_size: Optional[int] = None,\n    eos_id: Optional[int] = None,\n) -> Tuple[List[List[int]], List[List[float]]]:\n    input_ids = torch.tensor(encoded_prompts, device=model.device)\n    output = model.model.generate(\n        input_ids=input_ids,\n        max_length=input_ids.shape[-1] + max_tokens,\n        cg=True,\n        return_dict_in_generate=True,\n        output_scores=True,\n        enable_timing=False,\n        eos_token_id=eos_id,\n        temperature=temperature,\n        top_p=0.8,\n    )\n    generated_tokens = output.sequences[:, input_ids.shape[-1] :].tolist()\n\n    _logprobs: List[List[float]] = [[] for _ in range(len(generated_tokens))]\n    for seq_idx, batch_score in enumerate(output.scores):\n        for batch_idx, score in enumerate(batch_score.tolist()):\n            _logprobs[batch_idx].append(score[generated_tokens[batch_idx][seq_idx]])\n\n    return generated_tokens, _logprobs\n\n\n@torch.inference_mode()\ndef generate(\n    encoded_prompts: List[List[int]],\n    model: Transformer,\n    images: List[List[np.ndarray]] = [],\n    *,\n    max_tokens: int,\n    temperature: float,\n    chunk_size: Optional[int] = None,\n    eos_id: Optional[int] = None,\n) -> Tuple[List[List[int]], List[List[float]]]:\n    images_torch: List[List[torch.Tensor]] = []\n    if images:\n        assert chunk_size is None\n        images_torch = [\n            [torch.tensor(im, device=model.device, dtype=model.dtype) for im in images_for_sample]\n            for images_for_sample in images\n        ]\n\n    model = model.eval()\n    B, V = len(encoded_prompts), model.args.vocab_size\n\n    seqlens = [len(x) for x in encoded_prompts]\n\n    # Cache\n    cache_window = max(seqlens) + max_tokens\n    cache = BufferCache(\n        model.n_local_layers,\n        model.args.max_batch_size,\n        cache_window,\n        model.args.n_kv_heads,\n        model.args.head_dim,\n        model.args.sliding_window,\n    )\n    cache.to(device=model.device, dtype=model.dtype)\n    cache.reset()\n\n    # Bookkeeping\n    logprobs: List[List[float]] = [[] for _ in range(B)]\n    last_token_prelogits = None\n\n    # One chunk if size not specified\n    max_prompt_len = max(seqlens)\n    if chunk_size is None:\n        chunk_size = max_prompt_len\n\n    flattened_images: List[torch.Tensor] = sum(images_torch, [])\n\n    # Encode prompt by chunks\n    for s in range(0, max_prompt_len, chunk_size):\n        prompt_chunks = [p[s : s + chunk_size] for p in encoded_prompts]\n        assert all(len(p) > 0 for p in prompt_chunks)\n        prelogits = model.forward(\n            torch.tensor(sum(prompt_chunks, []), device=model.device, dtype=torch.long),\n            images=flattened_images,\n            seqlens=[len(p) for p in prompt_chunks],\n            cache=cache,\n        )\n        logits = torch.log_softmax(prelogits, dim=-1)\n\n        if last_token_prelogits is not None:\n            # Pass > 1\n            last_token_logits = torch.log_softmax(last_token_prelogits, dim=-1)\n            for i_seq in range(B):\n                logprobs[i_seq].append(last_token_logits[i_seq, prompt_chunks[i_seq][0]].item())\n\n        offset = 0\n        for i_seq, sequence in enumerate(prompt_chunks):\n            logprobs[i_seq].extend([logits[offset + i, sequence[i + 1]].item() for i in range(len(sequence) - 1)])\n            offset += len(sequence)\n\n        last_token_prelogits = prelogits.index_select(\n            0,\n            torch.tensor([len(p) for p in prompt_chunks], device=prelogits.device).cumsum(dim=0) - 1,\n        )\n        assert last_token_prelogits.shape == (B, V)\n\n    # decode\n    generated_tensors = []\n    is_finished = torch.tensor([False for _ in range(B)])\n\n    assert last_token_prelogits is not None\n    for _ in range(max_tokens):\n        next_token = sample(last_token_prelogits, temperature=temperature, top_p=0.8)\n\n        if eos_id is not None:\n            is_finished = is_finished | (next_token == eos_id).cpu()\n\n        if is_finished.all():\n            break\n\n        last_token_logits = torch.log_softmax(last_token_prelogits, dim=-1)\n        for i in range(B):\n            logprobs[i].append(last_token_logits[i, next_token[i]].item())\n\n        generated_tensors.append(next_token[:, None])\n        last_token_prelogits = model.forward(next_token, seqlens=[1] * B, cache=cache)\n        assert last_token_prelogits.shape == (B, V)\n\n    generated_tokens: List[List[int]]\n    if generated_tensors:\n        generated_tokens = torch.cat(generated_tensors, 1).tolist()\n    else:\n        generated_tokens = []\n\n    return generated_tokens, logprobs\n\n\ndef sample(logits: torch.Tensor, temperature: float, top_p: float) -> torch.Tensor:\n    if temperature > 0:\n        probs = torch.softmax(logits / temperature, dim=-1)\n        next_token = sample_top_p(probs, top_p)\n    else:\n        next_token = torch.argmax(logits, dim=-1).unsqueeze(0)\n\n    return next_token.reshape(-1)\n\n\ndef sample_top_p(probs: torch.Tensor, p: float) -> torch.Tensor:\n    assert 0 <= p <= 1\n\n    probs_sort, probs_idx = torch.sort(probs, dim=-1, descending=True)\n    probs_sum = torch.cumsum(probs_sort, dim=-1)\n    mask = probs_sum - probs_sort > p\n    probs_sort[mask] = 0.0\n    probs_sort.div_(probs_sort.sum(dim=-1, keepdim=True))\n    next_token = torch.multinomial(probs_sort, num_samples=1)\n    return torch.gather(probs_idx, -1, next_token)\n"
  },
  {
    "path": "src/mistral_inference/lora.py",
    "content": "import logging\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Any, Dict, NamedTuple, Union\n\nimport safetensors.torch\nimport torch\nimport torch.nn as nn\nfrom simple_parsing.helpers import Serializable\n\n\n@dataclass\nclass LoraArgs(Serializable):\n    rank: int\n    scaling: float\n\n    def __post_init__(self) -> None:\n        assert self.rank > 0\n        assert self.scaling > 0.0\n\n\nclass LoRALinear(nn.Module):\n    \"\"\"\n    Implementation of:\n        - LoRA: https://arxiv.org/abs/2106.09685\n\n    Notes:\n        - Freezing is handled at network level, not layer level.\n        - Scaling factor controls relative importance of LoRA skip\n          connection versus original frozen weight. General guidance is\n          to keep it to 2.0 and sweep over learning rate when changing\n          the rank.\n    \"\"\"\n\n    def __init__(\n        self,\n        in_features: int,\n        out_features: int,\n        rank: int,\n        scaling: float,\n        bias: bool = False,\n    ):\n        super().__init__()\n\n        self.in_features = in_features\n        self.out_features = out_features\n        assert not bias\n        self.bias = bias\n        self.rank = rank\n        self.scaling = scaling\n\n        self.lora_A = nn.Linear(\n            self.in_features,\n            self.rank,\n            bias=self.bias,\n        )\n        self.lora_B = nn.Linear(\n            self.rank,\n            self.out_features,\n            bias=self.bias,\n        )\n\n        self.linear = nn.Linear(self.in_features, self.out_features, bias=self.bias)\n\n        # make sure no LoRA weights are marked as \"missing\" in load_state_dict\n        def ignore_missing_keys(m: nn.Module, incompatible_keys: NamedTuple) -> None:\n            incompatible_keys.missing_keys[:] = []  # type: ignore\n\n        self.register_load_state_dict_post_hook(ignore_missing_keys)\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        lora = self.lora_B(self.lora_A(x))\n        result: torch.Tensor = self.linear(x) + lora * self.scaling\n        return result\n\n    def _load_from_state_dict(self, state_dict: Dict[str, Any], prefix: str, *args, **kwargs) -> None:  # type: ignore[no-untyped-def]\n        key_name = prefix + \"weight\"\n\n        # full checkpoint\n        if key_name in state_dict:\n            w_ref = state_dict[key_name]\n\n            # load frozen weights\n            state_dict = {\n                \"linear.weight\": w_ref,\n                \"lora_A.weight\": torch.zeros_like(self.lora_A.weight, device=w_ref.device, dtype=w_ref.dtype),\n                \"lora_B.weight\": torch.zeros_like(self.lora_B.weight, device=w_ref.device, dtype=w_ref.dtype),\n            }\n            self.load_state_dict(state_dict, assign=True, strict=True)\n\n\nclass LoRALoaderMixin:\n    def load_lora(self, lora_path: Union[Path, str], scaling: float = 2.0) -> None:\n        \"\"\"Loads LoRA checkpoint\"\"\"\n\n        lora_path = Path(lora_path)\n        assert lora_path.is_file(), f\"{lora_path} does not exist or is not a file\"\n\n        state_dict = safetensors.torch.load_file(lora_path)\n\n        self._load_lora_state_dict(state_dict, scaling=scaling)\n\n    def _load_lora_state_dict(self, lora_state_dict: Dict[str, torch.Tensor], scaling: float = 2.0) -> None:\n        \"\"\"Loads LoRA state_dict\"\"\"\n        lora_dtypes = set([p.dtype for p in lora_state_dict.values()])\n        assert (\n            len(lora_dtypes) == 1\n        ), f\"LoRA weights have multiple different dtypes {lora_dtypes}. All weights need to have the same dtype\"\n        lora_dtype = lora_dtypes.pop()\n        assert lora_dtype == self.dtype, f\"LoRA weights dtype differs from model's dtype {lora_dtype} != {self.dtype}\"  # type: ignore[attr-defined]\n        assert all(\"lora\" in key for key in lora_state_dict.keys())\n\n        # move tensors to device\n        lora_state_dict = {k: v.to(self.device) for k, v in lora_state_dict.items()}  # type: ignore[attr-defined]\n\n        state_dict = self.state_dict()  # type: ignore[attr-defined]\n\n        if self.args.lora is None:  # type: ignore[attr-defined]\n            logging.info(\"Loading and merging LoRA weights...\")\n\n            # replace every nn.Linear with a LoRALinear with 'meta' device except the output layer\n            named_modules = dict(self.named_modules())  # type: ignore[attr-defined]\n            for name, module in named_modules.items():\n                if isinstance(module, nn.Linear) and name != \"output\":\n                    layer_id = name.split(\".\")[1]\n                    if layer_id not in self.layers:  # type: ignore[attr-defined]\n                        logging.debug(\n                            \"Skipping parameter %s at pipeline rank %d\",\n                            name,\n                            self.pipeline_rank,  # type: ignore[attr-defined]\n                        )\n                    elif (name + \".lora_B.weight\") in lora_state_dict:\n                        weight = (\n                            module.weight\n                            + (lora_state_dict[name + \".lora_B.weight\"] @ lora_state_dict[name + \".lora_A.weight\"])\n                            * scaling\n                        )\n\n                        state_dict[name + \".weight\"] = weight\n        else:\n            logging.info(\"Loading LoRA weights...\")\n            for k, v in lora_state_dict.items():\n                state_dict.update(lora_state_dict)\n\n                layer_id = k.split(\".\")[1]\n                if layer_id in self.layers:  # type: ignore[attr-defined]\n                    state_dict[k] = v\n                else:\n                    logging.debug(\n                        \"Skipping parameter %s at pipeline rank %d\",\n                        k,\n                        self.pipeline_rank,  # type: ignore[attr-defined]\n                    )\n\n        self.load_state_dict(state_dict, strict=True)  # type: ignore[attr-defined]\n"
  },
  {
    "path": "src/mistral_inference/main.py",
    "content": "import json\nimport logging\nimport os\nimport warnings\nfrom pathlib import Path\nfrom typing import List, Optional, Tuple, Type, Union\n\nimport fire  # type: ignore\nimport torch\nimport torch.distributed as dist\nfrom mistral_common.protocol.instruct.messages import (\n    AssistantMessage,\n    ContentChunk,\n    ImageChunk,\n    ImageURLChunk,\n    TextChunk,\n    UserMessage,\n)\nfrom mistral_common.protocol.instruct.request import ChatCompletionRequest\nfrom mistral_common.tokens.tokenizers.base import Tokenizer\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_common.tokens.tokenizers.sentencepiece import is_sentencepiece\nfrom mistral_common.tokens.tokenizers.tekken import (\n    SpecialTokenPolicy,\n    Tekkenizer,\n    is_tekken,\n)\nfrom PIL import Image\n\nfrom mistral_inference.args import TransformerArgs\nfrom mistral_inference.generate import generate, generate_mamba\nfrom mistral_inference.mamba import Mamba\nfrom mistral_inference.transformer import Transformer\n\n\ndef is_torchrun() -> bool:\n    required_vars = [\"MASTER_ADDR\", \"MASTER_PORT\", \"RANK\", \"WORLD_SIZE\"]\n    return all(var in os.environ for var in required_vars)\n\n\ndef load_tokenizer(model_path: Path) -> MistralTokenizer:\n    tokenizer = [f for f in os.listdir(model_path) if is_tekken(model_path / f) or is_sentencepiece(model_path / f)]\n    assert len(tokenizer) > 0, (\n        f\"No tokenizer in {model_path}, place a `tokenizer.model.[v1,v2,v3]` or `tekken.json` file in {model_path}.\"\n    )\n    assert len(tokenizer) == 1, (\n        f\"Multiple tokenizers {', '.join(tokenizer)} found in `model_path`, make sure to only have one tokenizer\"\n    )\n\n    mistral_tokenizer = MistralTokenizer.from_file(str(model_path / tokenizer[0]))\n\n    if isinstance(mistral_tokenizer.instruct_tokenizer.tokenizer, Tekkenizer):\n        mistral_tokenizer.instruct_tokenizer.tokenizer.special_token_policy = SpecialTokenPolicy.KEEP\n\n    logging.info(f\"Loaded tokenizer of type {mistral_tokenizer.instruct_tokenizer.__class__}\")\n\n    return mistral_tokenizer\n\n\ndef get_model_cls(model_path: str) -> Union[Type[Mamba], Type[Transformer]]:\n    with open(Path(model_path) / \"params.json\", \"r\") as f:\n        args_dict = json.load(f)\n\n    return {\"mamba\": Mamba, \"transformer\": Transformer}[args_dict.get(\"model_type\", \"transformer\")]  # type: ignore[return-value]\n\n\ndef pad_and_convert_to_tensor(list_of_lists: List[List[int]], pad_id: int) -> List[List[int]]:\n    # Determine the length of the longest list\n    max_len = max(len(lst) for lst in list_of_lists)\n\n    # Left pad each list to the maximum length\n    padded_lists = [[pad_id] * (max_len - len(lst)) + lst for lst in list_of_lists]\n\n    return padded_lists\n\n\ndef _get_multimodal_input() -> Tuple[UserMessage, bool]:\n    chunks: List[ContentChunk] = []\n\n    response = input(\"Text prompt: \")\n    if response:\n        chunks.append(TextChunk(text=response))\n\n    print(\"[You can input zero, one or more images now.]\")\n    while True:\n        did_something = False\n        response = input(\"Image path or url [Leave empty and press enter to finish image input]: \")\n        if response:\n            if Path(response).is_file():\n                chunks.append(ImageChunk(image=Image.open(response)))\n            else:\n                assert response.startswith(\"http\"), f\"{response} does not seem to be a valid url.\"\n                chunks.append(ImageURLChunk(image_url=response))\n            did_something = True\n\n        if not did_something:\n            break\n\n    return UserMessage(content=chunks), not chunks\n\n\ndef interactive(\n    model_path: str,\n    max_tokens: int = 35,\n    temperature: float = 0.7,\n    num_pipeline_ranks: int = 1,\n    instruct: bool = False,\n    lora_path: Optional[str] = None,\n) -> None:\n    if is_torchrun():\n        torch.distributed.init_process_group()\n        torch.cuda.set_device(torch.distributed.get_rank())\n        should_print = torch.distributed.get_rank() == 0\n\n        num_pipeline_ranks = torch.distributed.get_world_size()\n    else:\n        should_print = True\n        num_pipeline_ranks = 1\n\n    mistral_tokenizer: MistralTokenizer = load_tokenizer(Path(model_path))\n    tokenizer: Tokenizer = mistral_tokenizer.instruct_tokenizer.tokenizer\n\n    model_cls = get_model_cls(model_path)\n    model = model_cls.from_folder(Path(model_path), max_batch_size=3, num_pipeline_ranks=num_pipeline_ranks)\n    is_multimodal = isinstance(model.args, TransformerArgs) and model.args.vision_encoder is not None\n\n    if is_multimodal:\n        assert instruct, \"Multimodal models should only be used in instruct mode\"\n\n    # load LoRA\n    if lora_path is not None:\n        model.load_lora(Path(lora_path))\n\n    prompt: str = \"\"\n    messages: List[UserMessage | AssistantMessage] = []\n\n    while True:\n        if should_print:\n            if not is_multimodal:\n                user_input = input(\"Prompt: \")\n\n            if instruct:\n                if is_multimodal:\n                    mm_input, finished = _get_multimodal_input()\n                    if finished:\n                        break\n                    messages += [mm_input]\n                else:\n                    messages += [UserMessage(content=user_input)]\n                chat_completion_request = ChatCompletionRequest(messages=messages)\n\n                tokenized = mistral_tokenizer.encode_chat_completion(chat_completion_request)\n                tokens = tokenized.tokens\n                images = tokenized.images\n            else:\n                prompt += user_input\n\n                tokens = tokenizer.encode(prompt, bos=True, eos=False)\n                images = []\n\n            length_tensor = torch.tensor([len(tokens)], dtype=torch.int)\n        else:\n            length_tensor = torch.tensor([0], dtype=torch.int)\n            images = []\n\n        if is_torchrun():\n            dist.broadcast(length_tensor, src=0)\n\n        if not should_print:\n            tokens = int(length_tensor.item()) * [0]\n\n        if isinstance(model, Transformer):\n            generated_tokens, _ = generate(\n                [tokens],\n                model,\n                [images],\n                max_tokens=max_tokens,\n                temperature=temperature,\n                eos_id=tokenizer.eos_id,\n            )\n        else:\n            # Mamba models don't support images\n            generated_tokens, _ = generate_mamba(\n                [tokens],\n                model,\n                max_tokens=max_tokens,\n                temperature=temperature,\n                eos_id=tokenizer.eos_id,\n            )\n\n        answer = tokenizer.decode(generated_tokens[0])\n\n        if should_print:\n            print(answer)\n            print(\"=====================\")\n\n        if instruct:\n            messages += [AssistantMessage(content=answer)]\n        else:\n            prompt += answer\n\n\ndef demo(\n    model_path: str,\n    max_tokens: int = 35,\n    temperature: float = 0,\n    lora_path: Optional[str] = None,\n) -> None:\n    if is_torchrun():\n        torch.distributed.init_process_group()\n        torch.cuda.set_device(torch.distributed.get_rank())\n        should_print = torch.distributed.get_rank() == 0\n\n        num_pipeline_ranks = torch.distributed.get_world_size()\n    else:\n        should_print = True\n        num_pipeline_ranks = 1\n\n    model_cls = get_model_cls(model_path)\n    model = model_cls.from_folder(Path(model_path), max_batch_size=3, num_pipeline_ranks=num_pipeline_ranks)\n    # load LoRA\n    if lora_path is not None:\n        model.load_lora(Path(lora_path))\n\n    mistral_tokenizer: MistralTokenizer = load_tokenizer(Path(model_path))\n    tokenizer: Tokenizer = mistral_tokenizer.instruct_tokenizer.tokenizer\n\n    prompts = [\n        \"This is a test\",\n        \"This is another great test\",\n        \"This is a third test, mistral AI is very good at testing. \",\n    ]\n\n    encoded_prompts = [tokenizer.encode(prompt, bos=True, eos=False) for prompt in prompts]\n\n    if isinstance(model, Transformer):\n        generate_fn = generate\n    else:\n        generate_fn = generate_mamba  # type: ignore[assignment]\n        warnings.warn(\n            \"Batched generation is not correctly supported at the moment and therefore might lead to worse results \"\n            \"as compared to non-batched generation. \"\n            \"See https://github.com/state-spaces/mamba/issues/66#issuecomment-1862349718 for more information.\"\n        )\n        encoded_prompts = pad_and_convert_to_tensor(encoded_prompts, mistral_tokenizer.instruct_tokenizer.BOS)  # type: ignore[attr-defined]\n\n    generated_tokens, _logprobs = generate_fn(\n        encoded_prompts,\n        model,  # type: ignore[arg-type]\n        max_tokens=max_tokens,\n        temperature=temperature,\n        eos_id=tokenizer.eos_id,\n    )\n\n    generated_words = []\n    for i, x in enumerate(generated_tokens):\n        generated_words.append(tokenizer.decode(encoded_prompts[i] + x))\n\n    res = generated_words\n\n    if should_print:\n        for w, logprob in zip(res, _logprobs):\n            print(w)\n            logging.debug(\"Logprobs: %s\", logprob)\n            print(\"=====================\")\n\n\ndef mistral_chat() -> None:\n    fire.Fire(interactive)\n\n\ndef mistral_demo() -> None:\n    fire.Fire(demo)\n\n\nif __name__ == \"__main__\":\n    logging.basicConfig(level=logging.INFO)\n    fire.Fire(\n        {\n            \"interactive\": interactive,\n            \"demo\": demo,\n        }\n    )\n"
  },
  {
    "path": "src/mistral_inference/mamba.py",
    "content": "import json\nfrom pathlib import Path\nfrom typing import List, Optional, Union\n\nimport safetensors\nimport torch\nimport torch.nn as nn\n\nfrom mistral_inference.args import MambaArgs\nfrom mistral_inference.cache import BufferCache\nfrom mistral_inference.model import ModelBase\n\n_is_mamba_installed = False\ntry:\n    from mamba_ssm.models.config_mamba import MambaConfig\n    from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel\n\n    _is_mamba_installed = True\nexcept ImportError:\n    _is_mamba_installed = False\n\n\nclass Mamba(ModelBase, nn.Module):\n    def __init__(self, args: MambaArgs):\n        super().__init__()\n        self.args = args\n        assert _is_mamba_installed, \"Mamba is not installed. Please install it using `pip install mamba-ssm`.\"\n\n        # make sure naming is consistent with `mamba_ssm`\n        config = MambaConfig(\n            d_model=args.dim,\n            n_layer=args.n_layers,\n            vocab_size=args.vocab_size,\n            ssm_cfg={\"ngroups\": args.n_groups, \"layer\": \"Mamba2\"},\n            attn_layer_idx=[],\n            attn_cfg={},\n            rms_norm=args.rms_norm,\n            residual_in_fp32=args.residual_in_fp32,\n            fused_add_norm=args.fused_add_norm,\n            pad_vocab_size_multiple=args.pad_vocab_size_multiple,\n            tie_embeddings=args.tie_embeddings,\n        )\n        self.model = MambaLMHeadModel(config)\n\n    @property\n    def dtype(self) -> torch.dtype:\n        return next(self.parameters()).dtype\n\n    @property\n    def device(self) -> torch.device:\n        return next(self.parameters()).device\n\n    def forward(\n        self,\n        input_ids: torch.Tensor,\n        seqlens: List[int],  # not supported for now\n        cache: Optional[BufferCache] = None,  # not supported for now\n    ) -> torch.Tensor:\n        lm_output = self.model(input_ids)\n        result: torch.Tensor = lm_output.logits\n        return result\n\n    @staticmethod\n    def from_folder(\n        folder: Union[Path, str],\n        max_batch_size: int = 1,\n        num_pipeline_ranks: int = 1,\n        device: Union[torch.device, str] = \"cuda\",\n        dtype: Optional[torch.dtype] = None,\n    ) -> \"Mamba\":\n        with open(Path(folder) / \"params.json\", \"r\") as f:\n            model_args = MambaArgs.from_dict(json.load(f))\n\n        with torch.device(\"meta\"):\n            model = Mamba(model_args)\n\n        model_file = Path(folder) / \"consolidated.safetensors\"\n\n        assert model_file.exists(), f\"Make sure {model_file} exists.\"\n        loaded = safetensors.torch.load_file(str(model_file))\n\n        model.load_state_dict(loaded, assign=True, strict=True)\n        return model.to(device=device, dtype=dtype)\n"
  },
  {
    "path": "src/mistral_inference/model.py",
    "content": "from abc import ABC, abstractmethod\nfrom pathlib import Path\nfrom typing import List, Optional, Union\n\nimport torch\nimport torch.nn as nn\n\nfrom mistral_inference.cache import BufferCache\n\n\nclass ModelBase(nn.Module, ABC):\n    def __init__(self) -> None:\n        super().__init__()\n\n    @property\n    @abstractmethod\n    def dtype(self) -> torch.dtype:\n        pass\n\n    @property\n    @abstractmethod\n    def device(self) -> torch.device:\n        pass\n\n    @abstractmethod\n    def forward(\n        self,\n        input_ids: torch.Tensor,\n        seqlens: List[int],  # not supported for now\n        cache: Optional[BufferCache] = None,  # not supported for now\n    ) -> torch.Tensor:\n        pass\n\n    @staticmethod\n    @abstractmethod\n    def from_folder(\n        folder: Union[Path, str],\n        max_batch_size: int = 1,\n        num_pipeline_ranks: int = 1,\n        device: Union[torch.device, str] = \"cuda\",\n        dtype: Optional[torch.dtype] = None,\n    ) -> \"ModelBase\":\n        pass\n"
  },
  {
    "path": "src/mistral_inference/moe.py",
    "content": "import dataclasses\nfrom typing import List\n\nimport torch\nimport torch.nn.functional as F\nfrom simple_parsing.helpers import Serializable\nfrom torch import nn\n\n\n@dataclasses.dataclass\nclass MoeArgs(Serializable):\n    num_experts: int\n    num_experts_per_tok: int\n\n\nclass MoeLayer(nn.Module):\n    def __init__(self, experts: List[nn.Module], gate: nn.Module, moe_args: MoeArgs):\n        super().__init__()\n        assert len(experts) > 0\n        self.experts = nn.ModuleList(experts)\n        self.gate = gate\n        self.args = moe_args\n\n    def forward(self, inputs: torch.Tensor) -> torch.Tensor:\n        gate_logits = self.gate(inputs)\n        weights, selected_experts = torch.topk(gate_logits, self.args.num_experts_per_tok)\n        weights = F.softmax(weights, dim=1, dtype=torch.float).to(inputs.dtype)\n        results = torch.zeros_like(inputs)\n        for i, expert in enumerate(self.experts):\n            batch_idx, nth_expert = torch.where(selected_experts == i)\n            results[batch_idx] += weights[batch_idx, nth_expert, None] * expert(inputs[batch_idx])\n        return results\n"
  },
  {
    "path": "src/mistral_inference/rope.py",
    "content": "from typing import Tuple\n\nimport torch\n\n\ndef precompute_freqs_cis(dim: int, end: int, theta: float) -> torch.Tensor:\n    freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))\n    t = torch.arange(end, device=freqs.device)\n    freqs = torch.outer(t, freqs).float()\n    return torch.polar(torch.ones_like(freqs), freqs)  # complex64\n\n\ndef apply_rotary_emb(\n    xq: torch.Tensor,\n    xk: torch.Tensor,\n    freqs_cis: torch.Tensor,\n) -> Tuple[torch.Tensor, torch.Tensor]:\n    xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2))\n    xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2))\n    freqs_cis = freqs_cis[:, None, :]\n    xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(-2)\n    xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(-2)\n    return xq_out.type_as(xq), xk_out.type_as(xk)\n\n\ndef precompute_freqs_cis_2d(\n    dim: int,\n    height: int,\n    width: int,\n    theta: float,\n) -> torch.Tensor:\n    \"\"\"\n    freqs_cis: 2D complex tensor of shape (height, width, dim // 2) to be indexed by\n        (height, width) position tuples\n    \"\"\"\n    # (dim / 2) frequency bases\n    freqs = 1.0 / (theta ** (torch.arange(0, dim, 2).float() / dim))\n\n    h = torch.arange(height, device=freqs.device)\n    w = torch.arange(width, device=freqs.device)\n\n    freqs_h = torch.outer(h, freqs[::2]).float()\n    freqs_w = torch.outer(w, freqs[1::2]).float()\n    freqs_2d = torch.cat(\n        [\n            freqs_h[:, None, :].repeat(1, width, 1),\n            freqs_w[None, :, :].repeat(height, 1, 1),\n        ],\n        dim=-1,\n    )\n    return torch.polar(torch.ones_like(freqs_2d), freqs_2d)\n"
  },
  {
    "path": "src/mistral_inference/transformer.py",
    "content": "import json\nimport logging\nimport math\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Any, List, Mapping, Optional, Union\n\nimport safetensors.torch\nimport torch\nfrom torch import nn\n\nfrom mistral_inference.args import PATCH_MERGE, TransformerArgs\nfrom mistral_inference.cache import BufferCache, CacheInputMetadata\nfrom mistral_inference.lora import LoRALoaderMixin\nfrom mistral_inference.model import ModelBase\nfrom mistral_inference.rope import precompute_freqs_cis\nfrom mistral_inference.transformer_layers import RMSNorm, TransformerBlock\nfrom mistral_inference.vision_encoder import PatchMerger, VisionLanguageAdapter, VisionTransformer\n\n\n@dataclass\nclass SimpleInputMetadata:\n    # rope absolute positions\n    positions: torch.Tensor\n\n    @staticmethod\n    def from_seqlens(seqlens: List[int], device: torch.device) -> \"SimpleInputMetadata\":\n        return SimpleInputMetadata(\n            positions=torch.cat([torch.arange(0, seqlen) for seqlen in seqlens]).to(device=device, dtype=torch.long)\n        )\n\n\nclass Transformer(ModelBase, LoRALoaderMixin):\n    def __init__(\n        self,\n        args: TransformerArgs,\n        pipeline_rank: int = 0,\n        num_pipeline_ranks: int = 1,\n        softmax_fp32: bool = True,\n    ):\n        super().__init__()\n        self.args = args\n        self.vocab_size = args.vocab_size\n        self.n_layers = args.n_layers\n        self._precomputed_freqs_cis: Optional[torch.Tensor] = None\n        assert self.vocab_size > 0\n        assert pipeline_rank < num_pipeline_ranks, (pipeline_rank, num_pipeline_ranks)\n        self.pipeline_rank = pipeline_rank\n        self.num_pipeline_ranks = num_pipeline_ranks\n        self.softmax_fp32 = softmax_fp32\n\n        # Modules specific to some ranks:\n        self.tok_embeddings: Optional[nn.Embedding] = None\n        self.norm: Optional[RMSNorm] = None\n        self.output: Optional[nn.Linear] = None\n        if pipeline_rank == 0:\n            self.tok_embeddings = nn.Embedding(args.vocab_size, args.dim)\n\n            self.vision_encoder: Optional[VisionTransformer] = None\n            self.vision_language_adapter: Optional[VisionLanguageAdapter] = None\n\n            if args.vision_encoder is not None:\n                self.vision_encoder = VisionTransformer(args.vision_encoder)\n                self.vision_language_adapter = VisionLanguageAdapter(\n                    args.vision_encoder.hidden_size, args.dim, args.vision_encoder.adapter_bias\n                )\n\n                if args.vision_encoder.add_pre_mm_projector_layer_norm:\n                    self.pre_mm_projector_norm = RMSNorm(args.vision_encoder.hidden_size, eps=1e-5)\n\n                if args.vision_encoder.mm_projector_id == PATCH_MERGE:\n                    self.patch_merger = PatchMerger(\n                        vision_encoder_dim=args.vision_encoder.hidden_size,\n                        spatial_merge_size=args.vision_encoder.spatial_merge_size,\n                    )\n\n        if pipeline_rank == num_pipeline_ranks - 1:\n            self.norm = RMSNorm(args.dim, eps=args.norm_eps)\n            self.output = nn.Linear(args.dim, args.vocab_size, bias=False)\n        # Initialize all layers but slice off those not of this rank.\n        layers = [\n            TransformerBlock(\n                dim=args.dim,\n                hidden_dim=args.hidden_dim,\n                n_heads=args.n_heads,\n                n_kv_heads=args.n_kv_heads,\n                head_dim=args.head_dim,\n                norm_eps=args.norm_eps,\n                lora=args.lora,\n                moe=args.moe,\n            )\n            for _ in range(args.n_layers)\n        ]\n        num_layers_per_rank = math.ceil(self.n_layers / self.num_pipeline_ranks)\n        offset = self.pipeline_rank * num_layers_per_rank\n        end = min(self.n_layers, offset + num_layers_per_rank)\n        self.layers = nn.ModuleDict({str(i): layers[i] for i in range(offset, end)})\n        self.n_local_layers = len(self.layers)\n\n    @property\n    def dtype(self) -> torch.dtype:\n        return next(self.parameters()).dtype\n\n    @property\n    def device(self) -> torch.device:\n        return next(self.parameters()).device\n\n    @property\n    def freqs_cis(self) -> torch.Tensor:\n        # We cache freqs_cis but need to take care that it is on the right device\n        # and has the right dtype (complex64). The fact that the dtype is different\n        # from the module's  dtype means we cannot register it as a buffer\n        if self._precomputed_freqs_cis is None:\n            # default to 10**6\n            theta = self.args.rope_theta or 1000000.0\n            self._precomputed_freqs_cis = precompute_freqs_cis(self.args.head_dim, 128_000, theta)\n\n        if self._precomputed_freqs_cis.device != self.device:\n            self._precomputed_freqs_cis = self._precomputed_freqs_cis.to(device=self.device)\n        return self._precomputed_freqs_cis\n\n    def embed_vision_language_features(self, input_ids: torch.Tensor, images: List[torch.Tensor]) -> torch.Tensor:\n        assert self.tok_embeddings is not None\n        assert self.vision_encoder is not None\n        assert self.vision_language_adapter is not None\n        assert self.args.vision_encoder is not None\n\n        text_locations = input_ids != self.args.vision_encoder.image_token_id\n        image_locations = input_ids == self.args.vision_encoder.image_token_id\n        text_features = self.tok_embeddings(input_ids[text_locations])\n\n        image_features = self.vision_encoder(images)\n\n        if self.args.vision_encoder.add_pre_mm_projector_layer_norm:\n            image_features = self.pre_mm_projector_norm(image_features)\n\n        if self.args.vision_encoder.mm_projector_id == PATCH_MERGE:\n            patch_size = self.args.vision_encoder.patch_size\n            img_patch_dims = [(img.shape[1] // patch_size, img.shape[2] // patch_size) for img in images]\n            image_features = self.patch_merger(image_features, image_sizes=img_patch_dims)\n\n        image_features = self.vision_language_adapter(image_features)\n\n        N_txt, D_txt = text_features.shape\n        N_img, D_img = image_features.shape\n\n        seq_len = input_ids.shape[0]\n\n        assert D_txt == D_img, f\"Text features dim {D_txt} should be equal to image features dim {D_img}\"\n        assert seq_len == N_txt + N_img, (\n            f\"seq_len {seq_len} should be equal to N_txt + N_img {(N_txt, N_img, image_locations.sum().item())}\"\n        )\n\n        combined_features = torch.empty(\n            (seq_len, D_txt),\n            dtype=text_features.dtype,\n            device=text_features.device,\n        )\n        combined_features[text_locations, :] = text_features\n        combined_features[image_locations, :] = image_features\n        return combined_features\n\n    def forward_partial(\n        self,\n        input_ids: torch.Tensor,\n        seqlens: List[int],\n        cache: Optional[BufferCache] = None,\n        images: Optional[List[torch.Tensor]] = None,\n    ) -> torch.Tensor:\n        \"\"\"Local forward pass.\n\n        If doing pipeline parallelism, this will return the activations of the last layer of this stage.\n        For the last stage, this will return the normalized final embeddings.\n        \"\"\"\n        assert len(seqlens) <= self.args.max_batch_size, (\n            f\"Max batch size is {self.args.max_batch_size}, got batch size of {len(seqlens)}\"\n        )\n        (num_toks,) = input_ids.shape\n        assert sum(seqlens) == num_toks, (sum(seqlens), num_toks)\n\n        input_metadata: List[CacheInputMetadata] | List[SimpleInputMetadata]\n\n        if cache is not None:\n            input_metadata = cache.get_input_metadata(seqlens)\n        else:\n            input_metadata = [SimpleInputMetadata.from_seqlens(seqlens, self.device) for _ in range(len(self.layers))]\n\n        if self.pipeline_rank == 0:\n            assert self.tok_embeddings is not None\n            if self.vision_encoder is not None and images:\n                h = self.embed_vision_language_features(input_ids, images)\n            else:\n                h = self.tok_embeddings(input_ids)\n        else:\n            h = torch.empty(num_toks, self.args.dim, device=self.device, dtype=self.dtype)\n            torch.distributed.recv(h, src=self.pipeline_rank - 1)\n\n        # freqs_cis is always the same for every layer\n        freqs_cis = self.freqs_cis[input_metadata[0].positions]\n\n        for local_layer_id, layer in enumerate(self.layers.values()):\n            if cache is not None:\n                assert input_metadata is not None\n                cache_metadata = input_metadata[local_layer_id]\n                assert isinstance(cache_metadata, CacheInputMetadata)\n                cache_view = cache.get_view(local_layer_id, cache_metadata)\n            else:\n                cache_view = None\n            h = layer(h, freqs_cis, cache_view)\n\n        if cache is not None:\n            cache.update_seqlens(seqlens)\n        if self.pipeline_rank < self.num_pipeline_ranks - 1:\n            torch.distributed.send(h, dst=self.pipeline_rank + 1)\n            return h\n        else:\n            # Last rank has a final normalization step.\n            assert self.norm is not None\n            return self.norm(h)  # type: ignore\n\n    def forward(\n        self,\n        input_ids: torch.Tensor,\n        seqlens: List[int],\n        cache: Optional[BufferCache] = None,\n        images: Optional[List[torch.Tensor]] = None,\n    ) -> torch.Tensor:\n        h = self.forward_partial(input_ids, seqlens, cache=cache, images=images)\n        if self.pipeline_rank < self.num_pipeline_ranks - 1:\n            # ignore the intermediate activations as we'll get the final output from\n            # the last stage\n            outs = torch.empty(h.shape[0], self.vocab_size, device=h.device, dtype=h.dtype)\n        else:\n            assert self.output is not None\n            outs = self.output(h)\n        if self.num_pipeline_ranks > 1:\n            torch.distributed.broadcast(outs, src=self.num_pipeline_ranks - 1)\n\n        if self.softmax_fp32:\n            return outs.float()\n        else:\n            return outs\n\n    def load_state_dict(self, state_dict: Mapping[str, Any], strict: bool = True, assign: bool = False) -> None:\n        state_to_load = {}\n        skipped = set([])\n        for k, v in state_dict.items():\n            if k.startswith(\"tok_embeddings\"):\n                if self.pipeline_rank == 0:\n                    state_to_load[k] = v\n                else:\n                    logging.debug(\n                        \"Skipping parameter %s at pipeline rank %d\",\n                        k,\n                        self.pipeline_rank,\n                    )\n                    skipped.add(k)\n            elif k.startswith(\"norm\") or k.startswith(\"output\"):\n                if self.pipeline_rank == self.num_pipeline_ranks - 1:\n                    state_to_load[k] = v\n                else:\n                    logging.debug(\n                        \"Skipping parameter %s at pipeline rank %d\",\n                        k,\n                        self.pipeline_rank,\n                    )\n                    skipped.add(k)\n            elif k.startswith(\"layers\"):\n                layer_id = k.split(\".\")[1]\n                if layer_id in self.layers:\n                    state_to_load[k] = v\n                else:\n                    logging.debug(\n                        \"Skipping parameter %s at pipeline rank %d\",\n                        k,\n                        self.pipeline_rank,\n                    )\n                    skipped.add(k)\n            elif any(\n                k.startswith(key)\n                for key in [\"vision_encoder\", \"vision_language_adapter\", \"patch_merger\", \"pre_mm_projector_norm\"]\n            ):\n                if self.pipeline_rank == 0:\n                    state_to_load[k] = v\n                else:\n                    logging.debug(\n                        \"Skipping parameter %s at pipeline rank %d\",\n                        k,\n                        self.pipeline_rank,\n                    )\n                    skipped.add(k)\n            else:\n                raise ValueError(f\"Unexpected key {k}\")\n        assert set(state_dict.keys()) == skipped.union(set(state_to_load.keys()))\n        super().load_state_dict(state_to_load, strict=strict, assign=assign)\n\n    @staticmethod\n    def from_folder(\n        folder: Union[Path, str],\n        max_batch_size: int = 1,\n        num_pipeline_ranks: int = 1,\n        device: Union[torch.device, str] = \"cuda\",\n        dtype: Optional[torch.dtype] = None,\n        softmax_fp32: bool = True,\n    ) -> \"Transformer\":\n        with open(Path(folder) / \"params.json\", \"r\") as f:\n            model_args = TransformerArgs.from_dict(json.load(f))\n        model_args.max_batch_size = max_batch_size\n        if num_pipeline_ranks > 1:\n            pipeline_rank = torch.distributed.get_rank()\n        else:\n            pipeline_rank = 0\n        with torch.device(\"meta\"):\n            model = Transformer(\n                model_args,\n                pipeline_rank=pipeline_rank,\n                num_pipeline_ranks=num_pipeline_ranks,\n                softmax_fp32=softmax_fp32,\n            )\n\n        pt_model_file = Path(folder) / \"consolidated.00.pth\"\n        safetensors_model_file = Path(folder) / \"consolidated.safetensors\"\n\n        assert pt_model_file.exists() or safetensors_model_file.exists(), (\n            f\"Make sure either {pt_model_file} or {safetensors_model_file} exists\"\n        )\n        assert not (pt_model_file.exists() and safetensors_model_file.exists()), (\n            f\"Both {pt_model_file} and {safetensors_model_file} cannot exist\"\n        )\n\n        if pt_model_file.exists():\n            loaded = torch.load(str(pt_model_file), mmap=True)\n        else:\n            loaded = safetensors.torch.load_file(str(safetensors_model_file))\n\n        model.load_state_dict(loaded, assign=True, strict=True)\n\n        return model.to(device=device, dtype=dtype)\n"
  },
  {
    "path": "src/mistral_inference/transformer_layers.py",
    "content": "from functools import partial\nfrom typing import Optional, Tuple, Type, Union\n\nimport torch\nfrom torch import nn\nfrom xformers.ops.fmha import memory_efficient_attention  # type: ignore\nfrom xformers.ops.fmha.attn_bias import BlockDiagonalMask\n\nfrom mistral_inference.args import LoraArgs\nfrom mistral_inference.cache import CacheView\nfrom mistral_inference.lora import LoRALinear\nfrom mistral_inference.moe import MoeArgs, MoeLayer\nfrom mistral_inference.rope import apply_rotary_emb\n\n\ndef repeat_kv(keys: torch.Tensor, values: torch.Tensor, repeats: int, dim: int) -> Tuple[torch.Tensor, torch.Tensor]:\n    keys = torch.repeat_interleave(keys, repeats=repeats, dim=dim)\n    values = torch.repeat_interleave(values, repeats=repeats, dim=dim)\n    return keys, values\n\n\ndef maybe_lora(\n    lora_args: Optional[LoraArgs],\n) -> Union[Type[nn.Linear], partial[LoRALinear]]:\n    if lora_args is None:\n        return nn.Linear\n    else:\n        return partial(LoRALinear, rank=lora_args.rank, scaling=lora_args.scaling)\n\n\nclass Attention(nn.Module):\n    def __init__(\n        self,\n        dim: int,\n        n_heads: int,\n        head_dim: int,\n        n_kv_heads: int,\n        lora: Optional[LoraArgs] = None,\n    ):\n        super().__init__()\n\n        self.n_heads: int = n_heads\n        self.head_dim: int = head_dim\n        self.n_kv_heads: int = n_kv_heads\n\n        self.repeats = self.n_heads // self.n_kv_heads\n\n        self.scale = self.head_dim**-0.5\n\n        MaybeLora = maybe_lora(lora)\n        self.wq = MaybeLora(dim, n_heads * head_dim, bias=False)\n        self.wk = MaybeLora(dim, n_kv_heads * head_dim, bias=False)\n        self.wv = MaybeLora(dim, n_kv_heads * head_dim, bias=False)\n        self.wo = MaybeLora(n_heads * head_dim, dim, bias=False)\n\n    def forward(\n        self,\n        x: torch.Tensor,\n        freqs_cis: torch.Tensor,\n        cache: Optional[CacheView] = None,\n        mask: Optional[BlockDiagonalMask] = None,\n    ) -> torch.Tensor:\n        assert mask is None or cache is None\n        seqlen_sum, _ = x.shape\n\n        xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)\n        xq = xq.view(seqlen_sum, self.n_heads, self.head_dim)\n        xk = xk.view(seqlen_sum, self.n_kv_heads, self.head_dim)\n        xv = xv.view(seqlen_sum, self.n_kv_heads, self.head_dim)\n        xq, xk = apply_rotary_emb(xq, xk, freqs_cis=freqs_cis)\n\n        if cache is None:\n            key, val = xk, xv\n        elif cache.prefill:\n            key, val = cache.interleave_kv(xk, xv)\n            cache.update(xk, xv)\n        else:\n            cache.update(xk, xv)\n            key, val = cache.key, cache.value\n            key = key.view(seqlen_sum * cache.max_seq_len, self.n_kv_heads, self.head_dim)\n            val = val.view(seqlen_sum * cache.max_seq_len, self.n_kv_heads, self.head_dim)\n\n        # Repeat keys and values to match number of query heads\n        key, val = repeat_kv(key, val, self.repeats, dim=1)\n\n        # xformers requires (B=1, S, H, D)\n        xq, key, val = xq[None, ...], key[None, ...], val[None, ...]\n        output = memory_efficient_attention(xq, key, val, mask if cache is None else cache.mask)\n        output = output.view(seqlen_sum, self.n_heads * self.head_dim)\n\n        assert isinstance(output, torch.Tensor)\n\n        return self.wo(output)  # type: ignore\n\n\nclass FeedForward(nn.Module):\n    def __init__(self, dim: int, hidden_dim: int, lora: Optional[LoraArgs] = None):\n        super().__init__()\n\n        MaybeLora = maybe_lora(lora)\n        self.w1 = MaybeLora(dim, hidden_dim, bias=False)\n        self.w2 = MaybeLora(hidden_dim, dim, bias=False)\n        self.w3 = MaybeLora(dim, hidden_dim, bias=False)\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        return self.w2(nn.functional.silu(self.w1(x)) * self.w3(x))  # type: ignore\n\n\nclass RMSNorm(torch.nn.Module):\n    def __init__(self, dim: int, eps: float = 1e-6):\n        super().__init__()\n        self.eps = eps\n        self.weight = nn.Parameter(torch.ones(dim))\n\n    def _norm(self, x: torch.Tensor) -> torch.Tensor:\n        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        output = self._norm(x.float()).type_as(x)\n        return output * self.weight\n\n\nclass TransformerBlock(nn.Module):\n    def __init__(\n        self,\n        dim: int,\n        hidden_dim: int,\n        n_heads: int,\n        n_kv_heads: int,\n        head_dim: int,\n        norm_eps: float,\n        lora: Optional[LoraArgs] = None,\n        moe: Optional[MoeArgs] = None,\n    ):\n        super().__init__()\n        self.n_heads = n_heads\n        self.dim = dim\n        self.attention = Attention(\n            dim=dim,\n            n_heads=n_heads,\n            head_dim=head_dim,\n            n_kv_heads=n_kv_heads,\n            lora=lora,\n        )\n        self.attention_norm = RMSNorm(dim, eps=norm_eps)\n        self.ffn_norm = RMSNorm(dim, eps=norm_eps)\n\n        self.feed_forward: nn.Module\n        if moe is not None:\n            self.feed_forward = MoeLayer(\n                experts=[FeedForward(dim=dim, hidden_dim=hidden_dim, lora=lora) for _ in range(moe.num_experts)],\n                gate=nn.Linear(dim, moe.num_experts, bias=False),\n                moe_args=moe,\n            )\n        else:\n            self.feed_forward = FeedForward(dim=dim, hidden_dim=hidden_dim, lora=lora)\n\n    def forward(\n        self,\n        x: torch.Tensor,\n        freqs_cis: torch.Tensor,\n        cache: Optional[CacheView] = None,\n        mask: Optional[BlockDiagonalMask] = None,\n    ) -> torch.Tensor:\n        r = self.attention.forward(self.attention_norm(x), freqs_cis, cache)\n        h = x + r\n        r = self.feed_forward.forward(self.ffn_norm(h))\n        out = h + r\n        return out\n"
  },
  {
    "path": "src/mistral_inference/vision_encoder.py",
    "content": "from typing import List, Optional\n\nimport torch\nimport torch.nn as nn\nfrom xformers.ops.fmha.attn_bias import BlockDiagonalMask\n\nfrom mistral_inference.args import VisionEncoderArgs\nfrom mistral_inference.rope import precompute_freqs_cis_2d\nfrom mistral_inference.transformer_layers import RMSNorm, TransformerBlock\n\n\ndef position_meshgrid(\n    patch_embeds_list: list[torch.Tensor],\n) -> torch.Tensor:\n    positions = torch.cat(\n        [\n            torch.stack(\n                torch.meshgrid(\n                    torch.arange(p.shape[-2]),\n                    torch.arange(p.shape[-1]),\n                    indexing=\"ij\",\n                ),\n                dim=-1,\n            ).reshape(-1, 2)\n            for p in patch_embeds_list\n        ]\n    )\n    return positions\n\n\nclass VisionTransformer(nn.Module):\n    def __init__(self, args: VisionEncoderArgs):\n        super().__init__()\n        self.args = args\n        self.patch_conv = nn.Conv2d(\n            in_channels=args.num_channels,\n            out_channels=args.hidden_size,\n            kernel_size=args.patch_size,\n            stride=args.patch_size,\n            bias=False,\n        )\n        self.ln_pre = RMSNorm(args.hidden_size, eps=1e-5)\n        self.transformer = VisionTransformerBlocks(args)\n\n        head_dim = self.args.hidden_size // self.args.num_attention_heads\n        assert head_dim % 2 == 0, \"ROPE requires even head_dim\"\n        self._freqs_cis: Optional[torch.Tensor] = None\n\n    @property\n    def max_patches_per_side(self) -> int:\n        return self.args.image_size // self.args.patch_size\n\n    @property\n    def device(self) -> torch.device:\n        return next(self.parameters()).device\n\n    @property\n    def freqs_cis(self) -> torch.Tensor:\n        if self._freqs_cis is None:\n            self._freqs_cis = precompute_freqs_cis_2d(\n                dim=self.args.hidden_size // self.args.num_attention_heads,\n                height=self.max_patches_per_side,\n                width=self.max_patches_per_side,\n                theta=self.args.rope_theta,\n            )\n\n        if self._freqs_cis.device != self.device:\n            self._freqs_cis = self._freqs_cis.to(device=self.device)\n\n        return self._freqs_cis\n\n    def forward(\n        self,\n        images: List[torch.Tensor],\n    ) -> torch.Tensor:\n        \"\"\"\n        Args:\n            images: list of N_img images of variable sizes, each of shape (C, H, W)\n\n        Returns:\n            image_features: tensor of token features for all tokens of all images of\n                shape (N_toks, D)\n        \"\"\"\n        # pass images through initial convolution independently\n        patch_embeds_list = [self.patch_conv(img.unsqueeze(0)).squeeze(0) for img in images]\n\n        # flatten to a single sequence\n        patch_embeds = torch.cat([p.flatten(1).permute(1, 0) for p in patch_embeds_list], dim=0)\n        patch_embeds = self.ln_pre(patch_embeds)\n\n        # positional embeddings\n        positions = position_meshgrid(patch_embeds_list).to(self.device)\n        freqs_cis = self.freqs_cis[positions[:, 0], positions[:, 1]]\n\n        # pass through Transformer with a block diagonal mask delimiting images\n        mask = BlockDiagonalMask.from_seqlens(\n            [p.shape[-2] * p.shape[-1] for p in patch_embeds_list],\n        )\n        out = self.transformer(patch_embeds, mask=mask, freqs_cis=freqs_cis)\n\n        # remove batch dimension of the single sequence\n        return out  # type: ignore[no-any-return]\n\n\nclass VisionLanguageAdapter(nn.Module):\n    def __init__(self, in_dim: int, out_dim: int, bias: bool = True):\n        super().__init__()\n        self.w_in = nn.Linear(\n            in_dim,\n            out_dim,\n            bias=bias,\n        )\n        self.gelu = nn.GELU()\n        self.w_out = nn.Linear(out_dim, out_dim, bias=bias)\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        return self.w_out(self.gelu(self.w_in(x)))  # type: ignore[no-any-return]\n\n\nclass VisionTransformerBlocks(nn.Module):\n    def __init__(self, args: VisionEncoderArgs):\n        super().__init__()\n        self.layers = torch.nn.ModuleList()\n        for _ in range(args.num_hidden_layers):\n            self.layers.append(\n                TransformerBlock(\n                    dim=args.hidden_size,\n                    hidden_dim=args.intermediate_size,\n                    n_heads=args.num_attention_heads,\n                    n_kv_heads=args.num_attention_heads,\n                    head_dim=args.hidden_size // args.num_attention_heads,\n                    norm_eps=1e-5,\n                )\n            )\n\n    def forward(\n        self,\n        x: torch.Tensor,\n        mask: BlockDiagonalMask,\n        freqs_cis: Optional[torch.Tensor],\n    ) -> torch.Tensor:\n        for layer in self.layers:\n            x = layer(x, mask=mask, freqs_cis=freqs_cis)\n        return x\n\n\nclass PatchMerger(nn.Module):\n    \"\"\"\n    Learned merging of spatial_merge_size ** 2 patches\n    \"\"\"\n\n    def __init__(\n        self,\n        vision_encoder_dim: int,\n        spatial_merge_size: int,\n    ) -> None:\n        super().__init__()\n\n        mlp_input_dim = vision_encoder_dim * (spatial_merge_size**2)\n\n        self.spatial_merge_size = spatial_merge_size\n        self.mlp_input_dim = mlp_input_dim\n\n        self.merging_layer = nn.Linear(mlp_input_dim, vision_encoder_dim, bias=False)\n\n    def forward(self, x: torch.Tensor, image_sizes: list[tuple[int, int]]) -> torch.Tensor:\n        # image_sizes specified in tokens\n        assert sum([h * w for h, w in image_sizes]) == len(x), f\"{sum([h * w for h, w in image_sizes])} != {len(x)}\"\n\n        # x is (N, vision_encoder_dim)\n        x = self.permute(x, image_sizes)\n\n        # x is (N / spatial_merge_size ** 2,\n        #       vision_encoder_dim * spatial_merge_size ** 2)\n        x = self.merging_layer(x)\n\n        # x is (N / spatial_merge_size ** 2, vision_encoder_dim)\n        return x\n\n    def permute(\n        self,\n        x: torch.Tensor,\n        image_sizes: list[tuple[int, int]],\n    ) -> torch.Tensor:\n        \"\"\"\n        Args:\n            x: (N, D) where N is flattened and concatenated patch tokens\n                for all images\n            image_sizes: list of tuple of (height, width) in tokens for\n                each image\n        Returns:\n            image_features: reorders patch tokens so each grid of\n                (spatial_merge_size, spatial_merge_size) is contiguous.\n                now (N / spatial_merge_size ** 2, D * spatial_merge_size ** 2)\n        \"\"\"\n\n        sub_grids = get_sub_grids(\n            x=x, image_sizes=image_sizes, spatial_merge_size=self.spatial_merge_size\n        )  # list of [d x sub_grid_size x sub_grid_size x n_patches]\n        permuted_tensor = [\n            grid.view(-1, grid.shape[-1]).t() for grid in sub_grids\n        ]  # n_patches x d * sub_grid_size * sub_grid_size\n        return torch.cat(permuted_tensor, dim=0)  # (N / spatial_merge_size ** 2, d * spatial_merge_size ** 2)\n\n\ndef get_sub_grids(\n    x: torch.Tensor,\n    image_sizes: list[tuple[int, int]],\n    spatial_merge_size: int,\n) -> list[torch.Tensor]:\n    # image_sizes specified in tokens\n    tokens_per_image = [h * w for h, w in image_sizes]\n    d = x.shape[-1]\n    all_img_sub_grids: list[torch.Tensor] = []\n    sub_grid_size = spatial_merge_size\n\n    for image_index, image_tokens in enumerate(x.split(tokens_per_image)):\n        # Reshape image_tokens into a 2D grid\n        h, w = image_sizes[image_index]\n        image_grid = image_tokens.view(h, w, d).permute(2, 0, 1)[None, :, :, :]  # 1 x d x h x w\n        sub_grids = torch.nn.functional.unfold(image_grid, kernel_size=sub_grid_size, stride=sub_grid_size)\n        sub_grids = sub_grids.view(\n            1, d, sub_grid_size, sub_grid_size, -1\n        )  # 1 x d x sub_grid_size x sub_grid_size x n_patches\n\n        all_img_sub_grids.append(sub_grids[0])\n\n    return all_img_sub_grids\n"
  },
  {
    "path": "tests/test_generate.py",
    "content": "from typing import List\n\nimport numpy as np\nimport torch\nfrom mistral_inference.args import VisionEncoderArgs\nfrom mistral_inference.generate import generate_mamba\nfrom mistral_inference.main import generate\nfrom mistral_inference.mamba import Mamba, MambaArgs\nfrom mistral_inference.transformer import Transformer, TransformerArgs\n\n\nclass DebugTokenizer:\n    @property\n    def bos_id(self) -> int:\n        return 0\n\n    @property\n    def eos_id(self) -> int:\n        return 1\n\n    @property\n    def pad_id(self) -> int:\n        return -1\n\n    def encode(self, s: str, bos: bool = True) -> List[int]:\n        assert isinstance(s, str)\n        t = [int(x) for x in s.split()]\n        if bos:\n            t = [self.bos_id, *t]\n        return t\n\n    def decode(self, t: List[int]) -> str:\n        return \" \".join([str(x) for x in t])\n\n\ndef test_generation_transformer() -> None:\n    torch.manual_seed(42)\n\n    sequences = [\"1 2 3 4 5 6 7\", \"0 1 2\", \"12 13 14\", \"2 4 34\"]\n    args = TransformerArgs(\n        dim=512,\n        n_layers=1,\n        head_dim=128,\n        hidden_dim=2048,\n        n_heads=4,\n        n_kv_heads=2,\n        norm_eps=1e-5,\n        vocab_size=32_000,\n        max_batch_size=len(sequences),\n    )\n    model = Transformer(args).to(\"cuda\", dtype=torch.float32)\n    tokenizer = DebugTokenizer()\n\n    encoded = [tokenizer.encode(s, bos=True) for s in sequences]\n    toks, all_logprobs_old = generate(encoded, model, temperature=0.0, max_tokens=7)\n\n    # concat generated and prompt\n    encoded = [e + t for e, t in zip(encoded, toks)]\n\n    generated, all_logprobs_new = generate(encoded, model, temperature=0.0, max_tokens=0)\n\n    assert generated == []\n\n    # Verify that logprobs are the same\n    assert len(sequences) == len(all_logprobs_old) == len(all_logprobs_new)\n    for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new):\n        assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f\"\\n{lp_old}\\n{lp_new}\"\n\n    print(\"All tests passed.\")\n\n\ndef test_generation_pixtral() -> None:\n    torch.manual_seed(42)\n    gen = np.random.default_rng(seed=42)\n\n    sequences = [\"1 2 2 2 2 4 5 6 7\", \"12 13 14\", \"2 2 2 2 7 8 9\"]\n    images = [[gen.normal(size=(3, 4, 4))], [], [gen.normal(size=(3, 4, 4))]]\n    args = TransformerArgs(\n        dim=512,\n        n_layers=1,\n        head_dim=128,\n        hidden_dim=2048,\n        n_heads=4,\n        n_kv_heads=2,\n        norm_eps=1e-5,\n        vocab_size=32_000,\n        max_batch_size=len(sequences),\n        vision_encoder=VisionEncoderArgs(\n            hidden_size=128,\n            num_channels=3,\n            image_size=4,\n            patch_size=2,\n            intermediate_size=256,\n            num_hidden_layers=1,\n            num_attention_heads=2,\n            rope_theta=10000,\n            image_token_id=2,\n        ),\n    )\n    model = Transformer(args).to(\"cuda\", dtype=torch.float32)\n    tokenizer = DebugTokenizer()\n\n    encoded = [tokenizer.encode(s, bos=True) for s in sequences]\n    toks, all_logprobs_old = generate(encoded, model, images=images, temperature=0.0, max_tokens=7)\n\n    # concat generated and prompt\n    encoded = [e + t for e, t in zip(encoded, toks)]\n\n    generated, all_logprobs_new = generate(encoded, model, images=images, temperature=0.0, max_tokens=0)\n\n    assert generated == []\n\n    # Verify that logprobs are the same\n    assert len(sequences) == len(all_logprobs_old) == len(all_logprobs_new)\n    for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new):\n        assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f\"\\n{lp_old}\\n{lp_new}\"\n\n    print(\"All tests passed.\")\n\n\ndef test_generation_pixtral_patch_merger() -> None:\n    torch.manual_seed(42)\n    gen = np.random.default_rng(seed=42)\n\n    sequences = [\"1 2 2 2 2 4 5 6 7\", \"12 13 14\", \"2 2 2 2 7 8 9\"]\n    images = [[gen.normal(size=(3, 8, 8))], [], [gen.normal(size=(3, 8, 8))]]\n    args = TransformerArgs(\n        dim=512,\n        n_layers=1,\n        head_dim=128,\n        hidden_dim=2048,\n        n_heads=4,\n        n_kv_heads=2,\n        norm_eps=1e-5,\n        vocab_size=32_000,\n        max_batch_size=len(sequences),\n        vision_encoder=VisionEncoderArgs(\n            hidden_size=128,\n            num_channels=3,\n            image_size=8,\n            patch_size=2,\n            intermediate_size=256,\n            num_hidden_layers=1,\n            num_attention_heads=2,\n            rope_theta=10000,\n            image_token_id=2,\n            adapter_bias=False,\n            spatial_merge_size=2,\n            add_pre_mm_projector_layer_norm=True,\n            mm_projector_id=\"patch_merge\",\n        ),\n    )\n    model = Transformer(args).to(\"cuda\", dtype=torch.float32)\n    tokenizer = DebugTokenizer()\n\n    encoded = [tokenizer.encode(s, bos=True) for s in sequences]\n    toks, all_logprobs_old = generate(encoded, model, images=images, temperature=0.0, max_tokens=7)\n\n    # concat generated and prompt\n    encoded = [e + t for e, t in zip(encoded, toks)]\n\n    generated, all_logprobs_new = generate(encoded, model, images=images, temperature=0.0, max_tokens=0)\n\n    assert generated == []\n\n    # Verify that logprobs are the same\n    assert len(sequences) == len(all_logprobs_old) == len(all_logprobs_new)\n    for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new):\n        assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f\"\\n{lp_old}\\n{lp_new}\"\n\n    print(\"All tests passed.\")\n\n\ndef test_generation_mamba() -> None:\n    torch.manual_seed(42)\n\n    sequences = [\"1 2 3 4 5 6 7\"]\n    args = MambaArgs(\n        dim=512,\n        n_layers=1,\n        n_groups=1,\n        rms_norm=True,\n        residual_in_fp32=True,\n        fused_add_norm=True,\n        pad_vocab_size_multiple=1,\n        tie_embeddings=False,\n        vocab_size=32768,\n    )\n    model = Mamba(args).to(\"cuda\", dtype=torch.float32)\n    tokenizer = DebugTokenizer()\n\n    encoded = [tokenizer.encode(s, bos=True) for s in sequences]\n    toks, all_logprobs_old = generate_mamba(encoded, model, temperature=0.0, max_tokens=7)\n\n    assert len(toks[0]) == 7\n    assert toks == [[25574, 14821, 11843, 23698, 12735, 23522, 27542]]\n\n\ndef test_chunks_transformer() -> None:\n    torch.manual_seed(42)\n\n    sequences = [\n        \" \".join([str(i) for i in range(7)]),\n        \" \".join([str(i) for i in range(9, 0, -1)]),\n    ]\n    args = TransformerArgs(\n        dim=512,\n        n_layers=1,\n        head_dim=128,\n        hidden_dim=2048,\n        n_heads=4,\n        n_kv_heads=2,\n        norm_eps=1e-5,\n        vocab_size=32_000,\n        max_batch_size=3,\n    )\n    model = Transformer(args).to(\"cuda\", dtype=torch.float32)\n    tokenizer = DebugTokenizer()\n\n    encoded = [tokenizer.encode(s, bos=True) for s in sequences]\n    toks, all_logprobs_old = generate(encoded, model, temperature=0.0, max_tokens=8)\n\n    # concat generated and prompt\n    encoded = [e + t for e, t in zip(encoded, toks)]\n\n    generated, all_logprobs_new = generate(encoded, model, temperature=0.0, max_tokens=0, chunk_size=5)\n    assert len(generated) == 0\n\n    for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new):\n        assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f\"\\n{lp_old}\\n{lp_new}\"\n"
  },
  {
    "path": "tutorials/classifier.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"0f47fc4b-7cdc-48ce-b42c-e0b73d902ece\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Train a classifier with Mistral 7B\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"bfd6bdfb-c8cc-4fb6-8969-153a97dfb576\",\n   \"metadata\": {},\n   \"source\": [\n    \"In this tutorial, we will see how we can leverage Mistral 7B to train a classifier. We need:\\n\",\n    \"- The mistral codebase: `https://github.com/mistralai/mistral-inference`\\n\",\n    \"- The pretrained model and its tokenizer: `https://docs.mistral.ai/llm/mistral-v0.1`\\n\",\n    \"- A dataset to train our classifier\\n\",\n    \"\\n\",\n    \"We will train and evaluate the classifier on `Symptom2Disease`, a public classification dataset from Kaggle provided in a CSV format: https://www.kaggle.com/datasets/niyarrbarman/symptom2disease/\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"225f88da-96ab-4262-88f8-8f1d6d5224bd\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Set paths\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"id\": \"ebe4bc9b-d0f7-42a6-9e41-bb3d042e0e07\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"from pathlib import Path\\n\",\n    \"\\n\",\n    \"code_path = \\\"/codebase_path/mistral-inference\\\"  # codebase\\n\",\n    \"data_path = Path(\\\"/dataset_path/Symptom2Disease.csv\\\")  # dataset downloaded from Kaggle\\n\",\n    \"model_path = Path(\\\"/model_path/\\\")  # model and tokenizer location\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"3fcc420b-2ac7-42e1-a3b1-8a80f87ccc10\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Imports\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"id\": \"5e8227fe-d2fb-4a2f-a136-96ee2d32287f\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"import csv\\n\",\n    \"import tqdm\\n\",\n    \"import torch\\n\",\n    \"import numpy as np\\n\",\n    \"\\n\",\n    \"import sys\\n\",\n    \"sys.path.append(code_path)  # append the path where mistral-inference was cloned\\n\",\n    \"\\n\",\n    \"from mistral.model import Transformer\\n\",\n    \"from mistral_common.tokens.tokenizers.mistral import MistralTokenizer\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"3b9c8408-1bc1-4e79-8b6d-391e07d78eb4\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Load the model and tokenizer\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"id\": \"806ff308-5909-46f7-b10e-8b1f1c7b7edc\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"model = Transformer.from_folder(model_path, dtype=torch.bfloat16)\\n\",\n    \"mistral_tokenizer = MistralTokenizer.from_file(str(model_path / \\\"tokenizer.model\\\"))\\n\",\n    \"tokenizer = mistral_tokenizer.instruct_tokenizer.tokenizer\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"4a9aa34b-cbe3-49a6-948b-17f6c84d93c6\",\n   \"metadata\": {},\n   \"source\": [\n    \"The tokenizer is a critical component of the model that segments input text into word-pieces.\\n\",\n    \"\\n\",\n    \"For instance, the sentence `\\\"Roasted barramundi fish\\\"` is encoded as `['▁Ro', 'asted', '▁barr', 'am', 'und', 'i', '▁fish']`.\\n\",\n    \"\\n\",\n    \"The number of subwords in the model is set to `32000`, and each word is decomposed into known word pieces to avoid unknown words.\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"ce0e03ef-439f-4078-a848-d69bf4e339ec\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Load the dataset\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"65d1f696-138d-467d-bb53-032e08d47bf1\",\n   \"metadata\": {},\n   \"source\": [\n    \"We reload the Kaggle dataset from the disk. Each sample is composed of a sentence and a label. The dataset contains 24 different labels.\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"id\": \"cc2bf18b-d99c-4c31-9988-1675d019e04b\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Reloaded 1200 samples with 24 labels.\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"data = []  # list of (text, label)\\n\",\n    \"with open(data_path, newline='') as csvfile:\\n\",\n    \"    reader = csv.reader(csvfile)\\n\",\n    \"    for i, row in enumerate(reader):\\n\",\n    \"        if i == 0:  # skip csv header\\n\",\n    \"            continue\\n\",\n    \"        data.append((row[2], row[1]))\\n\",\n    \"\\n\",\n    \"# label text to label ID\\n\",\n    \"labels = sorted({x[1] for x in data})\\n\",\n    \"txt_to_label = {x: i for i, x in enumerate(labels)}\\n\",\n    \"print(f\\\"Reloaded {len(data)} samples with {len(labels)} labels.\\\")\\n\",\n    \"\\n\",\n    \"# integer class for each datapoint\\n\",\n    \"data_class = [txt_to_label[x[1]] for x in data]\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"c5434265-fad8-4daa-b54b-438d420bcb4d\",\n   \"metadata\": {},\n   \"source\": [\n    \"The task is to classify a symptom, for instance `\\\"I have a dry cough that never stops.\\\"` to one of the 24 disease labels (`Acne, Arthritis, Bronchial Asthma, Cervical spondylosis, Chicken pox, Common Cold, etc.`).\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"1540b537-c3cc-4851-80e3-91a9b9e8cc7c\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Embed data points in the dataset\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"c73d5d75-55eb-4210-8ffe-d0db3e9f8735\",\n   \"metadata\": {},\n   \"source\": [\n    \"We will now learn a linear classifier on frozen features provided by Mistral 7B.\\n\",\n    \"In particular, each sentence in the dataset will be tokenized, and provided to the model.\\n\",\n    \"\\n\",\n    \"If the input sentence is composed of `N` tokens, the model output will be a list of `N` vectors of dimension `d=4096`, where `d` is the dimensionality of the model.\\n\",\n    \"These vectors are then averaged along the dimension 0 to get a vector of size `d`.\\n\",\n    \"\\n\",\n    \"Finally, the vectors of all sentences in the dataset are concatenated into a matrix of shape `(D, d)` where `D` is the number of samples in the dataset (in particular, D=1200).\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"id\": \"78019021-01d7-499f-94ea-a3c88bbcb42c\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1200it [00:25, 46.65it/s]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"with torch.no_grad():\\n\",\n    \"    featurized_x = []\\n\",\n    \"    # compute an embedding for each sentence\\n\",\n    \"    for i, (x, y) in tqdm.tqdm(enumerate(data)):\\n\",\n    \"        tokens = tokenizer.encode(x, bos=True)\\n\",\n    \"        tensor = torch.tensor(tokens).to(model.device)\\n\",\n    \"        features = model.forward_partial(tensor, [len(tokens)])  # (n_tokens, model_dim)\\n\",\n    \"        featurized_x.append(features.float().mean(0).cpu().detach().numpy())\\n\",\n    \"\\n\",\n    \"# concatenate sentence embeddings\\n\",\n    \"X = np.concatenate([x[None] for x in featurized_x], axis=0)  # (n_points, model_dim)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"7a997101-7844-4928-bdf4-d7a3051eca75\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Plot t-SNE embeddings\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"61bbc6b4-dfc5-42e0-b99b-1ed281d929cb\",\n   \"metadata\": {},\n   \"source\": [\n    \"Now that we have one vector/embedding for each sample in our dataset, we can visualize them using t-SNE.\\n\",\n    \"t-SNE is a powerful tool for reducing the dimensionality of high-dimensional data, while preserving the underlying structure and relationships between the data points, which can help uncover patterns and insights that would be difficult to discern in the high-dimensional space.\\n\",\n    \"\\n\",\n    \"In the graph below, we assign a different color to each class in the dataset. It is apparent that several distinct clusters are visible in the data.\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"id\": \"8e3cd894-ac30-4f34-a4c8-ef34293b587d\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOyddXhcVfrHP+fekbg2TdI2dTcqlFKglBaKS6HI4ostC8V3F1tBFn4sK8guLgss7lasUKe01N09tUjjNnbP74+bNEkzcmcysXI+zzNP03vPPedMMnPP977nFSGllCgUCoVCoVB0ELS2noBCoVAoFApFOCjxolAoFAqFokOhxItCoVAoFIoOhRIvCoVCoVAoOhRKvCgUCoVCoehQKPGiUCgUCoWiQ6HEi0KhUCgUig6FEi8KhUKhUCg6FLa2nkBzMQyDffv2kZiYiBCiraejUCgUCoXCAlJKysvL6dKlC5oWni2lw4uXffv2kZOT09bTUCgUCoVCEQG5ubl069YtrGs6vHhJTEwEzDeflJTUxrNRKBQKhUJhhbKyMnJycg6t4+HQ4cVL3VZRUlKSEi8KhUKhUHQwInH5UA67CoVCoVAoOhRKvCgUCoVCoehQKPGiUCgUCoWiQ6HEi0KhUCgUig6FEi8KhUKhUCg6FEq8KBQKhUKh6FAo8aJQKBQKhaJDocSLQqFQKBSKDkWHT1KnUASiQlaQL/MB6Cw6kyAS2nhGCoVCoYgGSrwojjhqZA0LfQvJlbmNjncX3Rmnj8MpnG00M4VCoVBEA7VtpDii8EovM7wz2CP3NDmXK3OZ4Z2BT/raYGYKhUKhiBbK8qI4otgut1NCid9zEkkxxeyQO+hJT3bKnRQYBQghyBJZ5IgcdKG37oQVCoVCETZKvCiOKLb6toZss9q3mqUsxY0bgQAJm9lMHHGcYjuFFJHS8hNVKBQKRcQo8aI4oqimOmSbCioO/SyRja79xvsNXUQXSmUpNmz01/rTW/RG09QOq0KhULQXlHhRHFHEEUcllRFdK5F48LBL7jp0rNAoZAlLOIdzSNCCRytJKSmUheySu/DgIZFEuoquJIgE7MIe0ZwUCoVC0RQlXhRHFP30fhT4CqLapwcPX/q+5BIuCWiB8UgPc3xz2C/3IxCHLDrLWQ5AjshhuD6cdJEe1bkpFArFLxFlC1ccUfQSvUgn3fRliSIePGyRWwKen+ebxwF5AGi8FVVHrszlG+83HDAORHVeCoVC8UtEWV4URxS60Jlsm8xi32K2y+1R7XudsY7+Wn9qqGGrsZUiWYSGRqpIZa/cG/J6A4M5vjkk+ZIopxw7dnppvRigDSBOxEV1rgqFQnEkI6SUTR8TOxBlZWUkJydTWlpKUlJSW09H0Y7Y6dvJPGNeVPtMIYVSSg9ZVxpuEUWCQGDHzqm2U0kTadGapkKhULR7mrN+K8uL4ojFX6K65nJ4DpnmCJe66924me6djh07/bR+jBKjVHSTQqFQBEHdIRVHLFVUtfUUwsKDh/XGej7xfYLX8Lb1dBQKhaLdosSL4ogljrioO+62BlVURX27S6FQKI4klHhRHLH01fo2e1unrdgr92IYRltPQ6FQKNolSrwojlgyRSY5IqetpxERdXWYFAqFQtEUJV4URyxCCE7UT2SAGNAm20c55NBf64+OKvaoUCgU0USJF8URjS50xtrGcpF+EbHEturYueTSmc4M1gaHfa1AkEpqC8xKoVAoOj5KvCh+EcRoMYzWR7f6uIuNxQwUA8MWTj1FTxUurVAoFAFQd0fFL4beWm8yRWaz+xkihpBBhqW2btxUUMFwbbjl/lNI4Xjt+Einp1AoFEc8Kkmd4hfDZt9m8mRes/roIXow2jYaj/TwifcTXLhCXlMhK/DitZSNd4AYwBhtjLK6KBQKRRCUeFH8YlhhrGjW9XXi4xvPNxzkIAbWQpmTRBIGhqWw7QH6ADShhItCoVAEQ4kXxS+CCllhyUoSDIlkt9wd9nWa0MghBw0tqOBJIYVkkpszRYUiKFJKDC/o9o6XvFGhaIgSL4pfBDWyps3G/tr7NafbTme4NpyVxsqA7UbpoxBCLSqK6FO8XzLzFVj4MdSUQ1yK5PiL4eTrILGT+swpOh5KvCh+EcSJuDYb24ePOd45TNGngAarjdUYGIe2oZw4OVY/lm5atxYZ3zAMlhpL2Sq34sWsmZRAAqP10fTQerTImIr2Q94OyRMXQ1UpGD7zWFUJzHwVlnwBv/9QktpFCRhFx0KJF8UvAh0dG7ZDi3drU0klP/h+YLJtMgO1geTKXFzSRYJIoKvoii5aJpGdYRh85vuMCioaHa+ggrm+uQyRQ9okhFzRerzxu8bCpQ7DB6V58Oa9cNv/2mZuCkWkKM9AxRGPIQ1memdGLFwyaX54NUAeeWyVW3EIB320PgzWB9Nd695iwgVgkbGoiXBpyDpjHaVGaYuNr2hb9myQ7FrVVLjUISVsWgDvPygxjI5ZB0zxy0SJF8URzz65j0IKI74+jzycOKMylw2+DVHpxyrb5faQbZYaS1thJoq2IHedtXbz3oSvn27ZuSgU0USJF8URz05jZ7NrGzU3UqmOUkqRsnWecF0+l6Vw7mKpCkAeqdQENro1YcaLUFmirC+KjoESL4ojHjduSzlWQhEN64uO3moRRXvkHkvt2qJopaJ12L3WelufB5Z/1XJzUSiiiRIviiOeRJEYlQXahYuJ2kR6iB7EE08ssWSKTAYwAM3CV0kgyBE5zZ6HVaxsGQF0EV1aeCaKtsDwSVZ8Hd41H/4Vln6prC+K9o+KNlIc8fTT+rHBiI6vSaaW6VeAHCWPYrOxmVXGqqBWniH6kKjMIxRSSst+PqM1FW10JOKuBk+Yu50+D7x2J8QmSYZMUBY5RftFWV4URzwpIoUhWvNFQzLJOITD77kYEcNwfThTbFNIJBEwLS11Fh8dnQn6BNJFerPnYYVCWYgHT8h28cTj0Py/J0XHxhEHzgjSGwlg+pNRn45CEVWU5UXxi2CUNooEElhjrKGKKqDe18OqP4wVq0miSOQ823nskXvYY+zBwCBdpNNH6xNQ+LQEW42tltoN1Ya28EwUbYWmCcZdJJnzRnjXSQm718DBPZL0bsr6omifKPGi+EUghGCAPoD+Wn9KKcWHjySSANOxtUbW4MPHamM1PnyHBE1dFtwBYgB9RB9LY2lCo7voTnete4u9n1DUCbRQ9NGsvSdFx2TyjTDvrcB5XoJRXR79+SgU0UKJF8UvCiEEKaQ0OtZL9Dr0c2+tN5uNzewyduHFS5pIY4A2gGyR3aHqDsUQc0h4BcKBA5tQt4AjmZRMQUYPSZ413+1DaDqkZrfMnBSKaKDuXApFA+JEHCP0EYzQR7T1VJpFH60P23zbAp4XCPpp/VpxRoq2YvTZ8M0zIEOn/AFM4TLyDIhP6ThiXfHLQznsKhRHIJkik66iq98QcYEghhgGa4PbYGaK1uaES8ERC1ayBWg6xKfAlHtaelYKRfNQ4kWhiAKGNFjrW8vX3q/52vs1K30r8RptUwQSzO2xk/ST6Cv6NhEwnejE6bbTiRWxbTQ7RWuS3Flwy+sQlxi8XZ3F5e7PIE1VmVa0c4RsrVzlLURZWRnJycmUlpaSlJTU1tNR/ALZb+xnpm9mk1T8AsEEfUKbOu4CVMtq9sv9hyKfUkVqm85H0Ta4qiRLv4DNP4O7CpI6w6gzoFN30zk3NVttFSlal+as30q8KBTNoNKo5GPfx0HbnKOfQ6qmBINCoVA0pDnrd6ttG/3tb39DCMEdd9xx6FhNTQ3Tpk0jPT2dhIQEpk6dSl5eXmtNSaFoNkuMJVFpo1C0BLnrJG/8TvLARMnvRkjuHyd56x5J7roO/cyqULROtNGSJUt48cUXGT58eKPjd955J1999RUffvghycnJ3HLLLVxwwQUsWLCgNaalUDSbfXJfyDb5Mr8VZqJoL1SVSZZ8Dge2mRluR54OPYa37naMzyt5535YdJhRsKYcFn5kvi55WHLi5WqbSNExaXHxUlFRweWXX87LL7/MI488cuh4aWkpr776Ku+88w6TJk0C4LXXXmPQoEEsWrSIY489tqWnplA0m8P9XCJtozgyWPy55O37wOsGXTez1X7/Igw8QXLdf2DfJvjpAyjcDQlpMOY8GH4y6PboioivnoJFnwRv8/5foLJEUrgLfD7oNQKOmQKxiUrQKNo/LS5epk2bxllnncUpp5zSSLwsW7YMj8fDKaeccujYwIED6d69OwsXLgwoXlwuFy5XfbWxsrKylpu8QhGCGGJCZrO1Y2+l2Sjako0LJG/8DuryAvoaBJtt+gn+ehqU5ZtRPYYPhAarZkD3YXDL65L4FEFNhWThR7DoIyg/CGld4bhLYMy5YHcGFhVSSrb8bGbT3b0GDu6tn0cwpj9hzgdgyefw2d/hhmclg09UAkbRvmlR8fLee++xfPlylixpuud/4MABHA4HKSkpjY5nZmZy4MCBgH0+9thjPPTQQ9GeqkIREQO1gSw3lgdt01f0baXZKNqSr/8DQpjWlsORhilcoD5Vf13SuD3r4aWb4Lw/SP73ByjYVXcRlBXCjhWw4D249X+SmPimokJKySePwaxX64VRODRs766C52+A+76QdBmgBIyi/dJiDru5ubncfvvtvP3228TExESt3/vuu4/S0tJDr9zc3Kj1rVCEy2Ax+FAVaX/EEstIbWQrzkjRFlSWSLYtsZ7FtiGGD7Yuhn9dBAU7MS0mtQKorr9dq+Gjv4LP01QZLfnCFC51fTUXwwtPXQ4VRcqpV9F+aTHxsmzZMvLz8xk1ahQ2mw2bzcbcuXP597//jc1mIzMzE7fbTUlJSaPr8vLyyMrKCtiv0+kkKSmp0UuhaCs0TeNc/VxyRE6TZHDZZDNFn4JNU1U4jnTc1upgRow0YOGHcNtAuH+c5O37JDtXSaSUzHzFtPhEk8pieO766PapUESTFsvzUl5ezq5duxodu+aaaxg4cCD33HMPOTk5ZGRk8O677zJ16lQANm3axMCBA4P6vByOyvOiaC94DS8FFCCRZJCBXevYvi4bdxqs2CgRAiaNEXROUwm5A+F1S+4ZAzUVrTtuVl84sLXl+u81Cqb9VznxKlqG5qzfLfZImJiYyNChQxsdi4+PJz09/dDx6667jrvuuou0tDSSkpK49dZbGTdunIo0UnRIbJqNbNpfKd6qGklRGSTFQ1K8QErJ7gNQUiHpnCrI7tR4Ydq1X3L3v70cOFh/7PmPoU83H0/dpZOUoETM4dgcgmMvlMx5vXXHbUnhArBzBbxwA9zxruxQVdUVRz5tas9+8skn0TSNqVOn4nK5OO2003juuefackoKxRHD/kLJq5/7mLVE4qv1nejZBaprIK+ovl33LLjnap2hfTTyiyQ3POLF5Wna37Y9cNmffHz0OMQ4lYBpiJSSU66j1cVLSyMlbF1ivvod09azUSjqUeUBFIojkD35kpse81JZzSHhEooHb9T5aZXBjEXBbwkXTBLc/ivlxyNLd2DsnIEsWAWGDxnflT/ffjflxQ4slXDuIGg2szL1JQ8eOe9J0T7oEOUBFApF6/H0uz4qwhAuAI+84mPm4lDPMpJPF3j5U+U2fvSU4OvYzz4RYxxYgu/nvyHzV5jhOUhE5V5OmvCVpetFR7rzSrNwo0LRnuhIXyGFQmGBvIOSxeskRphhu16fFbEjkC6dFZ4K/q96F3+p2o47kvjgDox0lWGseQ2Qh8VGS06aNJOcHrsIliHu6n/BQ7MhJqGlZxodpITMXm09C4WiMUq8KBRHGHvyW9gaoklk7Z1jpa+CN1yBk0oeich9P4H0n1DFZvdy++/+wUmnLcLmaHwutQv85gU4ZoogvZvgtjfB5myFCUeBcRe29QwUisaojWuFIoq4pIEEYsLcFyirlKzaLPH4YEB3QdfO4fkXuDySr380+Hyuwb6CsC5tRGIclIfIWaINqt9DkMDX7kKucGYSK/TIB+5AyLJdQc/b7V7On/I6U/8zlvwdGqX5kJRhhjU3jNjpMVww4jTJ0i9aesaRUZct+Px7ISVL+bso2hdKvCgUzURKyfvufD5x5VNRW4QxFo0zHelc68wOGmLq9kie+9DHl/Ml3gYP88cMEdx9lU5GqsAwJNv3QmW1JDEOtu6BGpekRxfB8L6Cahfc9aSXDTua/14mj4Uv5tFoLg3eKQhwntG4SrYLyVZfNcNsHWQfpLlotsB1AA4hEJogq68gK0h1iHEX0W7FS1ZfOONWGH2WEi6K9ocSLwpFM3mwagdLfI09Gqsx+NhdwBpvBU/G9/MrYKSUPPiSj59Wyybr4LINkml/93LlmRpvf2Owv9D/2DmZ0CmVqAgXAVxyqo2zT5Dc9i8fFYdbYOwGzmty0bLczR+sAyM6DUPu/zlICw06DUFYsL4NPE6QkiUpaQc7b0KD7kPhsv+D2CRI64LK7aJotyjxolA0g3me4ibCpSGbjWo+cOdziTOzybkVmyQLVvl/evcZkH8Q/vlmcGfY3DzzFQ3+cJVOVrqAdMH0JwU/rjL44mcvyzzl6IPKsY0sR9iazteBoI8eG51JdABE5ijYkg6u4gDFjAz0nqda7u/mV+Bv50WnLpHQIquvJATEJUP/cZDVx0y6p1C0Z5TDrkLRDN6uCa0cPnf5d0L5dqGBHuQbGE2327qlSDtsTRICjhkKT/9O56wTNLxeicttjjx+hM4/bnQy+poSnGPK/AqXFFHNOc4CZnu/5Vvvt6zzraNG1kRx5u0PodnQR98BzpS6Iw3+1dCGXIVIG2C5v66DBHe8E9lcYhIguVYXO2Jh2CSwOayFYg+dVP+zlGY9o+9fhLuGwZaQIfMKRduiLC8KRTPIk6G3UErx/0hdUBxeHpbmcP0UjcR4yC+C1CSYeLRGSqIpZoQQLFlncOcTXlZslEigWyZceLLGuSdq3BPbg7srt5In3YcElQb01QsY5diHQHAQCRLyZT5rjDVMtk0mXaS3zptrA0R8Jvrxf0XmLUMWrAbDA4k5aN3GI2JSw+6vz9GCiddIZr8Wxhw06DEcbntTYPgkQoOHJ4PPG9z6ktwZRp4ZOBuwzwv/vgIeWyxJSFEWGEX7RIkXhaIZaAhC2UgC3f4zUkHTJIbR8gvE8SM0enXxP86nc3w89Y6BptW/kz158NQ7ZmHGB35j45mE/sz0FPO9u4hS6aWPXkVP+z4A5GHv34OHH7w/cIHtAuyiYxenDIbQ7Ygux0KX6NRiO/9ecFfDgvdA0wFJ0Fw90jAz3wJoumDrEkm+Bd+nO96BF38bvI3hg88fh8sfszx9haJVUdtGCkUzGKTHhWzTXcQ0OeaVkrKRha0iXAb0IKBw2V8oefpdc4X0t1DOXS759idJnNA5VaQzZUNfTpk/AMeiWKrLmr4vMMWMCxc7ZBS8iH9B6DbBZY8KHpwFZ94KJ1wOXQbgV/0KAUMnwojT6o9ZLdK4coa1tmtmWetPoWgLlOVFoWgGv3F24aaqzUHbXBOT1eTYSzV7Wd3rIPogB76NCSAPX6Ek0aqPc8elgZ9RvpxnBI36FQI+me0jNgb+9ZYZgaRr4JP9EB/0ZchJmxhz/io0rWkH+4399Nf6R+U9/JLI6CE441bzZ59XMuMFmP266ZMCEJcCE66EM6aZFpc6nPHW+v/879baeX/ZQWWKdo4SLwpFM+hhi+XWmG78p2aP3/OXOTIZY09udKzE8PC156CZM+WqPbi/yMS7OBV8DcVK9Cwy//1C8s87/J/bkhu8jICUsH0vPPSS79CMTD8dgZSCtbMGAjB26sqm10bV5fiXiW4TnHELnHqj5MB2QEJmb//RQJ1yojt2Wtfo9qdQRBO1baRQNJMzHOm8ET+IE2zJpAobyULnaD2B5+P7c4Ufq8tSbzl1ekHYJc6pB4j7y2ZEVg2I6C/4S9ZLzrvLw8Mve1m7rbFSiXGY1pVgSFmbk83vWcG6OQP8biFliIyI56xojG4XdB0g6DpQ+BUuBbskz/w6umOefUd0+1MooomyvCgUUSBDd3B/XE9LbWtoauoQ8T5ksd3P9lF0KKmA2UslM5f4uOlCya9ONVP5H3+UxrwVwROMhCocLaVgx4ocBk/YcuiYjk5fLUhqWUVU+eppcIUo6xAOg8bD8FNUpJGi/aIsLwpFK9ND8+/o2tLfRqNWhDz/kcGqzaaAmni0oHNq0/wv4SCExFVlViEUCDQ0TtJPwik6SNXBDo6rSrLsq8iS0x2OIxZOvh5ueV0JF0X7RokXhaKVGarHk0XTEGJ9UDn4cXyNNroGH80yVzqnQ/Dk72zozaipKA2N1PQaYmQs/bX+nGM7h65avcPEAcPFNl815dLb3Kkr/FBRDEYUfrVT7oEn1wouuE8JF0X7R20bKRStjAuJXdM4fPfIPqEI34pkohlp5A+fASs3mSJpxz7Ju9/68DRj8dN1mP/2Mcx6w0xud8FEjXNOlKyQ5fzPtZ/thplxVwPG21K4NiabDM0RhXeiAIhPjrwsQEM+exwqSyTn/UHVNFK0f5TlRaFoZb5yF7LHcDU5rnetwXn5XtBAawHH3YYIYP4Kg+se9vJ9hKng65Y3wwC315Rce/Lg3+8Z3PB0DQ+U72SHUV8qwADme0u4vXILBYaKw40WMQmCoybXJrZrJt+/CD++2/x+FIqWRokXhaKV+cp9MGAQsW1EGfH3b2HIaVUcPVhw7DDBKcdE9ylY12BYP3joZR8+I3gW16DUTquhQ6+sfe3YrOOZm97kfRpAmfTyes3+CAdV+OOs282aRtHgu+fBMFSYu6J9o7aNFEckbmkwy1PMDHcRLgx6a7Fc5swkW28dJ1KflJRJL14MDhge7LWVl20IDoSqh5TipdNpRTwYnwKA2yPZV+Bl4856p9tmzc2AtCSBz9e8zoJGIUnw/piKfWJhkyKBBjDPW8LNshvxIgrmgtbE54EN82H1DKgoMqsijjgD+o2NjukjQroMENzxjuSpy8wSA82heB/kbYPsftGZm0LREijxojji2Oer4fbKLVQ2cCrZYdQw01vMVEcG18V0abGxy6WX9135fOMupPowu0MiOhc4M3ACTTeN6tGAeK3+q+mwC/51p41n3vfx3SKJN3hkc0B0zRQuv7tCZ8Eqo9lCSNOCWW0EstyOrNARSU0n7AMKDQ/xzfEUbm2qy+Gde+BAfUg4BTtg6yLI7g9XPRk980cE9BguuOKJEv57U3LQdsEyKtfh9URxYrX4PJJVP8DWxeYc+o2FYSebifgUinBR4kVxROGTsolwacjH7gIyhYOznZ2iPu48TwnP1ewJOHY5Pt5wHQj5pTOAE20pjY7FxQjuvtrGjVMlG3dKhICNOw1e/TzwKjRiAAzsIVi6XlJWCfFx0LuLICEOBNJCScnmI2yBR4g/3CTT3pn+r8BFgfZvhrfvhqufatUp1VHp2ktuyQwYtJ9hl57EmnfHITQDaZi/Y003Bcuka2HmK8H7csRCRo/ozm/PBsmz10BZAWg2c8dxzhtmhetfPyXpP1YJGEV4KPGiOKKY4TkYUDzU8ZbrQFTFy37DxZ8rt7Mv1HZQLaECe5LROdqW6P9cgmDsUPNGP2awoNpl8M63/t/vxh1w2liNHxb7KCwBrRR2HzAT1cXFNE+4JMVDWWWQBkKidatGxPlJyAcM0OPo1JEijkr2w+YFwdvsWWeaFfoe0zpzqqXStZfNBW8hMetUHfPbOWSPyGXdh0eTv74rus3GsIk6k66BboNh+VdQkuc/OknTYdyFEBMfPTGx+WfJM1eBr/aD3zCsuzQfnr4MugyQnHU7jDhNiRiFNZR4URxRfOcuCtmmDB9Fhoc0rWmulXCpkQb3Vm7joIyend2NRLcQqupym+n9A87NDY//r37LpuEWT3XtvpUmIvOjGTlAsCdfsnNfXa2jw5ACx8kHA15/pbNp2YR2za7V1trNernVxUtuyfdIDBrK0Zxjt5Fz7DYAdC2WYdm3odX6F13/rOTpK8DrAqPBjp7QzCrW5/wuenNbP0/y3HWhw7j3bYKXb4YL/yyZ+GslYBSh6WB2W4UiONUhrC51lERJbMzzFFMgPRZHtYbLQm9fzjc4/w9e/vtFZCPX+TwIEVlGmX0FksdvtdGltnyRptX/K4DfXig4c6Qdgfn/Os+WODTuje3ByACWpXZLKCeROgp2Qu7aFp1KQ2o8B6ny7COYHc1nVFNWU7/d1fMowX1fwnGX1FeiTusC5/4O7nofYhOjIx7WzZU8e214+Wc++isU5qpIJ0VolOVFcUTRVTjJDeoOa7LJW8VBw8sIWwL2Zvhe/OgpjbrvSJII/rX8bqHBP9+M0Gv3MPxaTSyws8bNnbbNdPtDDGPXZ7B2pc5+lxct00W/cdX06JLCxbZu/MqZyQJPCVXSoIvm5Hh7Ms6O5usC0G2Q9bazXoWrn2y5uTTA5S211M7tLWv0/849BZf+FS79K0gpo56U7uBeyYs3EtEX4/U74fcfRXU6iiMQJV4URxSXOTNZVFUWst1/XHsBMwLo1zHZnOFIj2i8GoyoO70O0eMDnvMZkpc+iY5wiRhNYmRXUyS9FFPB6sEVMNi0sHiBDcC66hKOsSXyx9ieXODs3LbzjQadephh0aV5odvuWWu2S85s0SlJKSmoXGaprU2PC3iuJbLp/vhO4y2pcNi5EiqKJAlpavtIEZgO+AikUDRFSsnnrgLurtoW1nXl+PhPzR4eq9rJqzX7+NCVT14Y2V97ajFEO9j3XHsnpJQcMFzs8FVTKetXgbXbJIXWHrZbDkNgP64YaPxgXfdznTFnibect10WFvuOwtS/WG9bWdJi06gjv+Jnymq2hGynCTvJMa2btGXtrMjLFUgJa2ZFdz6KIw9leVEcEXzhLuRF176Ir5/vLUXHXIBfc+3nTHs6N8V0Dek4e4YjnemewI6p4dJDOCmXXqZVbmZnbWp9G4IJthSuicmmrKIN86IICVJgO/EgWp+qkM0lMN1dyKXOzI65VXQ42f1h1Dmw/MvQbRMis+RZRUofeeWLLLXNTjoRvZUjuyK1ugAgwBX646X4hXME3FEUv3RqpMH/XAea3Y+PeqvBN56DvFITWgz10mO5wmluDzTHyC2AZKFzoj2FR2p2satBTSAvktneYu6s3EJcWtttGWlZLhyX7sVxTh5WdxqqMNjha2bK1/bEpOtADyIEhAa9RkFSdPMIHU6N9yBeI/QKn+TsQ+eE1o1+AjPgKuKEwxKy+kR1OoojECVeFB2epd4yy1FGVpHAl55Cio3QUUmXObO4N7YHaSEcbQ8nUejYEaQJGxc5OvP3uL68584/NH5DDOCg9LAg4wB9u2FZPDSHuiHuvkrjsscLSfjdduyjS1tl7JAYPti/FvauBE9NyOZRwxkPJ9/g/5zQQLfBpADno4i0FP0kiHd2bZMK0SdeEXnNrLSu0H9cdOejOPJQ20aKDk+pDJX2LTIMYJG3zJIz7xhbIk/K8Kwi59o7cXlMfb6TL92FeIO4/xrALG8xf76sC3f/y8AwolPrKBDZGXDLRTrHj9BY5o3n8whM+TFo9NRjojcpw4D5/4FdCxs7VXQeCCffC47AjqlRY8wUcMTAnNfM+kZ1ZPWF028z/21hYuxpaMKBETQxoiTB0a3F5+KPrgMFFz8g+eDB8K7T7XD1v0DT2oNCVrRnlHhRdHg6iZbZz9eAKouCZLG3DFeYcUcjbAmN/r/PcKFhbl8Fwo0ks5eH//zBwbMf+lgbnn+yZTQB6clw/AjTODtST6Sb5mSfYSULjYkAznKkExPN4otf3QtFO5sez98IH0+DC58HexTFUiCOOh2GTTaz6tZUQEo2dO7V8uPWogk7GfGjyKv4Gf/xyAKnLZUEZ5Tz/IfBhCsFOYMlHzwEuetCt+82CC7/G3QfqoSLIjRq20jR4RltSyS5BaoTG0BXzVoV6jLpC8vnJVXYGHxYSLQmgwuXOv5SuZ19XUs46vZ8hp4TOiw8EgwJa7bCpl3mwqgJwYNxvUgVtkbv098NpO7YUXpCdDPpbpntX7jU4a6ERSEK90QTTYfuw6H/ca0qXOrITh5PgiPHzxmBrsXQO31qm2wZNaT3aME9n8N5fzB/XUJr7Atjj4GjToM73oX7pgslXBSWUZYXRYfHJgRXO7P5d82eqPVpOtDaGGNLstS+s7BbtrvYEDwa27vJwlJj0aZxAA9PuHIB0I4TMDsWKmw0z2W4KZoGi9cZDOhhrjZdNCcvJgzkB08RczwlVEkf3bUYJtlT2W+4mOkppkz6yNYcnOFIZ7wtxVKZA8us/Sx0m12LYPwt0RuzHaMJO30zLqWoag2FFStweYvRNSdpcUPJSBiNXU8I3UkrIITg1N/CuAslS76Aon2QkAZjzoX0bkqsKCJDiRfFEcE6b0XU+qqzHNwVm3No8a2WPn72lFEhfQzU4+hra+xbMdqWRLLQKQ2xzXScLZkbnNlk6k0tOku95WHP1bBLYm7YTc2LPaCq7pE2OguCAHyHvZ04oXOuI4NzHRlN2p/f0snoqopDt/FZz9FzJKAJnU7xI+gUP6KtpxKSxE6CSde29SwURwpKvCg6NFJK3nIdYKa3JGp9Dq3d7hhii8eQkieqdzPbW9LIspIqbPwlticDbObWj00IpsV04/+qdzUpF6BhCoFz7J3I0ByU48Nf7tWKMB1+69C7uoi7ZxuuzzLxrUiJqA9/+AwY2LMdPRnrDvCGiixqR/NVKBQthhIvig7NdM9B3q0NL24OnYWdv8T0IE13kNKg2vSfqraz0tfUqlMsvfyuaiv/ju9H79rU6yfYU3gQjf+69rHbqK+v5ESjGoPPPYUASBcM1OO4L7YHGQ2Sh2VpDnYaNRGVGxDxPpwX76dqQyLUNN//R9MkGSmCMUPakRjofgxs+SF4m/TW9z1RKBStj3LYVXRYvFLyThSS0wHkSw87pKuRcNnsrfQrXOowgH9W7W507Bh7Es/HD+DZ+P48ENuTDOyHqkRL6i0ym31V3F25rVE005mO9ObVSRJgG1dEc8tECs3A7vDxyE029PYUsjr6cgjlmH2M2pdQKH4JKPGi6LBs8lWG9DEJh48Os+BYydq7U7qoPiwXuhCCXnosBdJDAR6/brgGkC/dfO+uzxNyqj2NQXpcRBsfngWpVD3UH+/sDMLbOjFI7FROQno5us1HTEINQ07azHn3f0Vm95aJZIoYZzyc+Yj/DLdCgxNugc79W39eCoWi1VHbRooOSzhZdTsLO4XSv5CoY5fh4oDPRVatM22+hey6AHmGm55abKNjXil5qya4+JHAD55iznOazq92ofFoXG/ecB3gG3chVl1PPfPTcH8eWUiy0KDvMTsZddbaJucKZAHJIjmifluMTn3gsv/B5u9hxwIzUV3WEBg+FWytW79HoVC0HUq8KDos3SzmYBmtJ5Am7Mzyho5WublyE4/F9WGALZ54oVnagUnVmn6N/lu9lzILWVvKDssOHCN0bozpylXOLHb6aljpLedjdwFVAWSXrNFwfx15lI80BP3G7vB7TjTD+bXIKGKP3IOGRm/RmzgtiplvNQ0Gnma+FArFLxIlXhQdlizNyUg9gVW+Cr9LuwAyhJ2H43ozy1PM9xbESw2Su6q28khcb85zZPD3mt1B26dhI7mBnwxAmeHhc2/oStMCyA5Q7TdW6AyyxTPIFs/Zzk68UrOP2Z6SQ+UDbIAX8K5LBE/kImPoyRtJ7FTp91ymyKRK+qg2PNRQhCF8JItkkkTg3DclRgk/+H6givpaAstZTidfJybrk7Ef9rtSKBSKSFDiRdGhmRbbjbsqt1ApfY3sHBpmMrh7Y3sghGC8PYVna/ZaSgQngUerdvJBwhBeRqc4iAXlupguTY595C6w5DIrwVLdpERh487Y7lwf04Vcnwu7EPTSYvFi8K7L4H9a+EXwhM3HqDPWctSp65ueQ5BIFo9U7ccldjLYlo9T1P8OskQWx+rHNhExlUYl033TMfz8jgsp5DPfZ1woLmzzrK8KhaLjoxx2FR2aLpqTp+P7M8meSl3iegGMsyXzZHw/BtbmYXEIjbMsCIU6qjBY5qvguYQBZAj/1oJrndlMdKQ2Ob7MYrK5XloMJ9hSLM8pUdgYbIunnx6HTQhihE63VD2i6r2xt24na/JuhFZf3LGuULFNJvFadTpCbGOkfX8j4QKQJ/P42vs1FbJxJNZC30K/wqWOaqrZaGwMf7IKhUJxGMryoujwZGoO7oztzs0x3SiTXhKETqyfkNqzHOl87C6w3O8Kbzlj7Em8njCIld5ypnsOUiMN+utxTHV2JiFA2O7hfiyBuDu2e7PT558wQhDjgBrLiWUlWu8qRFcPc9296aKV0dt2kHjhIYZYjtcGcX91BQ7hYaDNf/4cicSDh9W+1RxnO+7Q8f3sDzn6BmMDg/RBVierUCgUflHiRXHE4BQaGUEqTGdpTk6wJfOjt9RSf7HCNEwKIRhpT2KkvX6bxJCBN4aShY2DIQRMLBo99NigbSzN0Sm46UKNJ98JZX4x5yuSvMRcuaf2iGCvkcxedzI6cJo9nY048FJJf70ISeCga4lku9zOWDkWXehIKZEWNstcuEK2USgUilAo8aL4RXFHbA6FFW42yuqQbU+wpzT6f7Hh5pWa/Sz1llOODyeCifZULnB0pluDWkUn2VPZ4dofdCk/O4wtrFBMOUnHbhO89KmPkgY7VkLUbwXZYyVibBG20woQ9qYz8wET7Sl85ylCA2JF6DBxAwMXLuKIs+zHYkc57CoUiuajxIviF0Wc0PlXQj9urNjEHhnYCtBbi6FXrWWkrn7Se+78RoLEhWSGp4jZnmL+L64Pg2r9a05zpPGpu4BS6W3iAeLbHIfv20w+LIjlE83DiP6Cmy/SyO7UPPezs07QOO1YweqtktIKyO4E/XKgqkbgsMNBm5tbKwpxIZvMSQNG6AkM1uOZ6ykBoEaGFhkCgYN6S1cGGRQQfFuun9YvzHemUCgUTRFSBrF/dwDKyspITk6mtLSUpKTAIZwKRUPchsEtlZvYI5s6i2QIO0/H9ztUKuCNmv28H6R+ksAs1PhGwmA8GKz0VrDXcPGFu5AC6UHHtIBUvd0V38pk8LMhc8/VGmceb70mUbHh4R1XHlt8VTjRGG1L5DRHOsl+cs7UscVXxeNVu9gn3WjUlyuYYEvh9thuxAidld5y7q/aTqKo4ayYTUHes6CH6MGJthMbzKmYL31fBrzGjp2L9YvRtebXXurQHDwA370Ba340w8T6HAVTboLOOW09M4WiVWnO+q3Ei+IXi5SSHz2lfOjOp1h6SBY2znN04iR7KvZaf5ciw8NVFest5fI9yZbCIm9Zo3DsLOFguB7PntmJLPk8Iej1b/1VJyfTvwWmWvpY7i2n0vDxs7eMhb6mqft14O7YHow/bLurIVJKVvkq2O6rwSEEY2xJZDbINSOl5M7KLWw1qhllz6WPXoS/HSEbNs6yndUkA+8B4wCzfLPw0tjnJ554ztDPiG6yuo7I0h/gpfvwm/3wglvh9KtbfUoKRVuhxIsSL4oW4nNXAS+59kVc6lAD7FLD9cAAKquC+4VMGCV4+LeNLSduw+CJmlx+9JZYElACeCK+HwP0yEVCqeHlweodbPZVcpRtP/1sheii/jeQQgrH244nXQT229lp7GSPYWbY7af1I0PLiHg+RwwFe+CP5xM0bfPt/4Eh41ptSgpFW9Kc9Vv5vCgUQSiRXjSwkOjfPwbgKtVCCheAZRsaL2r7DRe3VWymMowaThL42JXP/XE9w5toA5I1G0/E9WWtr5KF3gwqfC6y9QoG6k4yRRqdRKeQDro9tZ701CKfwxHJZ88Tst7EB0/CQ0q8KBShUOJFoQhCumYPQzr4xzCsReIYDdY1tzS4p3JbWMKljoXeUqSUzcpkK4RgmC2BYbbgW12KMFi/MHSb/dtNBymVhVihCIoSLwpFEE60pfAS+w7VFIoEkewBmwHe4BFFfbrV/zzfU0KhtFbV+nB8+HMJDk5hvmTGl5KffwS3G3r3g9PPFQwdiUrnHy181pIX4nWD3VrRUYXil4oSLwpFEJI0G1c7s3jVFTh7bAo2SoPIG6GDfUwJnoVpQcf6zfn14maRt6lDrlW6CQdaGIJj41rJI/dJPO76GklFB2HxAsnZU+GqG48sAWNIyax9Pmbu9eIxJCPTdc7vZSfO1sLv0eaABgUrg7dTKBTBUOJFoQjBVGdnYoXOm64DlDbInJuEzuXOTMbakrimMnDNHg2YOLWaDdshN89/myknCYb3qw8hdsnIN6umODtbbltdLXnsTxK3qz6hHYBR6+Qz/WPoOwCOnxjxdNoVeyoMLvqhik2lBnVa5b/Swx+X1PDGxDgmZLfgLdFqbIS7BpzNz76sUBzJKPGiUFjgTEc6p9rT2OCrpFL66KI56a7HHDp/saOz31wwGuBE46r4LLIfsPHyZz6+nCeprDHPZ3eCa8/VOPXYxrlPeumxLPeVh+3xMkKP5zR7cAtPQ36cCVWVwdt88aHk+Ikdw/KSX23w4gY328oMusQJLu3rYFia+bt1+STnzahkd4UpIrwNtESFBy75oYq558QzIKWF8tAYFt2+O3YAqELRKijxolBYxFbrxOqPq5xZxAmd9115VDWQHH20WO6IzSGnVujcdKGNmy4MPdYZjjQ+CpIY73CcCM53ZHC5MyusYo8b1ko0jaCVqbdvAY/bwO5oP0Xo1xT5mL3PiyHh6Ayd4zN17lpUwxubG/sJvbDBw8BkQaxNsKPcoCRAAUsD02H6+fVunjquhaweiWlQXRG8jc0BMb/wXDgKhQWUeFEoooAQgoscGZyXl8fBTfMxPDU4O/eh07DTIIICjFmak9/GdOX5mr0ImgbYJqDxW2dXUjQbnTUH3RpYgcLFyoP+1s0waGjEQ0SN/GqDa+dW81OeD02YTsk+CSkOAgqTjaV1uYSD45Xw6U5Py4mXUy6Fdx4P3mbESS0ztkJxhKHEi0IRDarK4MO/4NizjmxNBwQY38GsV+Hce2Dg+LC7PMfRiWzNwUeufFb7zL2dVGFjsj2NK8O0sARiyFGC+TNDL+zffdH24sXlk5z3XRVby0wzUcPQ8kDCJVyqvfDqRjez9nnxGpKjM2xc0c9OdlwUrE7HnQPfvQkH9/k/74yDK+5r/jgKxS8AlWFX0f4pOwjzPoWfv4HyYkhMhZET4aQLYfF3sHsj2Oww7ATzuK2VKxdLCa/fCvs3+zFjCDNnx1VPQLchEQ/hkQZeJDFoUY38qak2uPLc0O0SEuG1T9p22+iDbW5++2NNi46hi3pRJAFNmMdeGh/LeT2j8LkqL4bn/wBbVzY+nt0Lbn8G0jKbP4ZC0UFQGXYVRy65m+Bfv4Wq8vpjVWXw7evmC0Bo5v7Boq8hPRvufK51i9ytnwP7AhUxrM24suBduOSRiIewC42WkGQxsRoxMQY1ITRBRTns3yPJ7tZ2jrsfbvegQbOTBgbDkI03mAxp6tHr51XTO0k75PwbMYmpcPcrsH8HbFoGSOg7ArqpatsKRTi0Hw88heJwvB749+1QFcLJURr1HqfF+fDETeBxtfz8wFzZvns2RBsDti42k4+1QwYOs9Zu984WnUZIilyyRYUL+PeMqTv2wvoofqaye5mWw5MuUsJFoYgAJV4U7ZcVs6G0ECvOlocwfFB0AJbNbLFpNSJ3LVSXWmgoW09Qhckxx1tr52jjpK99kjT0FjD8ZMcJeicG79gn4dtcixlyFQpFi6PEyy8Frwe2rIWNq6A6RGKP9sKWFeaWkFVi7ZDghHgnrJzdcvOqQ0qY+ZK1tjGJEBPfsvOJkOMmiJBuQjGxMNiihaaluLq/A59FHRtTIcjeZiN7q4240sDCJFaH78+MIzM29OfM09JmH4VCYZkWFS+PPfYYY8aMITExkc6dOzNlyhQ2bWrsG1BTU8O0adNIT08nISGBqVOnkpcXIA2pInwMA957AX51HNx8Htx6AVx0DDz7UAcRMRZWqzg7dEuGzgmQFgud4uDgalj8Scsm/Fo9A/YFzqzbiMEnhSfEWpH4BMGZU4LXAjz3IoEzpm0T1R2XqXNJb5vfmk2agDQn2NwwbG4Mx30Wx+CFMQxeFMO4L+MZ+X0MDj+VvV0GvL3Vy+gMPahVRxcwslMLJa9TKBRh06J307lz5zJt2jQWLVrE999/j8fj4dRTT6Wysn7RvPPOO/nyyy/58MMPmTt3Lvv27eOCCy5oyWn9cpASnrgPXv0nlBbVH3fVwBdvwd1Xgrt9bmUA0H90aPERa4dO8ebqBQ1WYAO+fx6WfNoyc5MSFn1ora3Q4KRrgve1ZwusWwR7t7ZJhtXLrhOcdKr5s66DVvsCOO1cmHp5q0+pCUIInjk+lj+PctLJWa804m1w4yAHy6ckcPmyBDrv1RGHSZyUfJ3RM2LRa92OkhzVnNl3Hb8+aiE1toVc2r8Eh+4hNaYKu9Z0e8gnzTEUCkX7oFVDpQsKCujcuTNz587lxBNPpLS0lIyMDN555x0uvNBMO7px40YGDRrEwoULOfbYY0P2qUKlg7B2Kdx5SeDzQsDNf4EpV7XenMLB64X7zq71ewlAlySwaYHNBvZYuON9cEQ58VhNJfxrirW24y6BSdc3Pe5xwawPYPYHUNSg8GO3/nDxXTDw6LCnJaWkwrWLgsrlVHvy0YWT1LhBpMcfhU0L/TvYvUMy9wdJSRGkdYKTJgu6dm/70gCF3lx2u9bgw0uqLZscfQSbyzS8BgxI0YizCRbMljz1f4FvZxLJ1lFuBpy1hiuGL0ETEkMKBBJdMzWjEOAzBOVuB3bNoMZr5+e9PUkTQ3lwZEbbFKisKoK9K8HnhrSekDEguJlMoeggdJhQ6dJS07ExLc2svbJs2TI8Hg+nnHLKoTYDBw6ke/fuAcWLy+XC5aq3FpSVRV5994jnmw/Mx2hfkJoqX73bfsWLzQa3/wcevRJ8fpwl7br5Coan2oz0GTwhypOzqPkTM2DC1Y2PGQZ88gzMfMf/+9q7BZ6aBrf9GwaPtT4jKckt+ZbCyhXQIKi4qnQ/eeWL6JdxObH2jKB9dO8luPKG9rMw1hiV/FT5AS5Zb60t9O1mC4sZGn8SOc763DlzZkiEZgZ3BaL3LsmvRyw+9H9N1P8d6/SArkmSnS6EgHiHhzP6bsAmNlPiO49UW3az35OUEi9uNDR0EcTZyOeGRf+FbXMav6nkbnDibaaQUSh+obSaeDEMgzvuuIPjjz+eoUPNVJ0HDhzA4XCQkpLSqG1mZiYHDhzw289jjz3GQw891NLTPTLYuzO4cJESDuS22nQiols/uOt5+McNTc9ZDT2pLI7unACc8ZDeHQ7mElTITLwW9AYLlM8LD18G+7cHvkZKwIB3H4eHP7b8lF1YuaJWuMDh2VC8RjXbCj9gSNZNiHbqe3M4hmEwv+IdvPjb2pSsdc3GqSXQ2d4DgOLi4MJFIIhxmblbtBC/0oa/ciEkPrwsqvqYDL0HPZ0jsAkHue61lBtF2HCQbe9LF3v/oGLEJ33sdK9kl3v1ITFmJ4ZkvTO9naNI0NKpNIoRCJK0DPS5T0PuUpp8vsr2wbcPwNmPQ1JW8DeiUByhtJp4mTZtGmvXruXHH39sVj/33Xcfd91116H/l5WVkZPTignJOgquGkhMImTVvYTk1ptTpPQbCUOOg/ULI/MHsbdAjK8QcOyF8NUTAc5rEJcEg05sfPzJm4MLlzqkhLzdsGMt9A4d5iOlJL/852AtcPtKKa3ZSkps/9DjtwCGIVm1DDatkwgBQ0cIBg8n4FbMbs+aAMKlng018+hsvxKAjM6QuyPwx10iSexUHlK4BKPAt4uCql2AKYZkrbA46Mtlm2spx8SfT5zW1PxtSB/Lqr7koG9Po+Meaij07aawanej4+lF1RyTuyTAGzHA64J1X8C430T+ZhSKDkyrPILdcsstTJ8+ndmzZ9OtW7dDx7OysnC73ZSUlDRqn5eXR1aW/ycKp9NJUlJSo1eHxTBMv5R538C6ZcFFhhWkhO8/gRvPgrOHwKLZwfvUNDjl/OaN2Vrc8Gj9Iu6wQWYCZCZau3bpFy3jBHvU6TDqbPNnrcFXSWimj83Fj5hVguv44R3YvDy8MQ7uD90G8BqVuHz1FqatBZks292L4qr6CsUCjXLXrvDGjxI7txnc9mvJ/90v+ew9+PRdePD3kt/fKMk/4P9vs9uzLmS/VbIUo/YzfvIZIujHXQADJq+PZPp+kYdZRGpkBcuqvsSfG+Eu95omwiUYnXP3YgSzuEkDts0LbmpSKI5gWtTyIqXk1ltv5dNPP2XOnDn06tWr0fnRo0djt9uZOXMmU6dOBWDTpk3s3r2bcePGteTU2p4FM+C5v0J+gyJt2Tkw7QEYOzH8/qSE/zwAX75tbZtB003LTHv1dzmcuET4wyuw7Dv4/t8gw0gYlrcV9qyDnChXFhQCTr8NBhwPy76EvG1gjzH9a0aeBQlp9W3LS+Cjp8IfIzEtdJsGfLjiWL5edzRuX932hSQrsZg7Jk6ne2oxYSX8iwCXS7JlA3g80L2XZNM6wWfvS3ZsqW/TcCdzzy5TxPzrZYiNbfy59UlrGYm9uHEQw6ixMOJoWLWsqVbVNOjeGwZM3OK/kyggkVQYxez1bMQr3fjwkqR1opMth13u1WH15XB5Qgtunxt8HrC1cfZAhaINaFHxMm3aNN555x0+//xzEhMTD/mxJCcnExsbS3JyMtdddx133XUXaWlpJCUlceuttzJu3DhLkUYdlrlfwyO3Nj2+Pxf+fAM8/BIcOym8PpfMNYULBL7pCWFaBQwfdOsFf3kW0juHN05bomlQsAmEDG8NFhpsXxZ98QLm77T30eYrGAu/DN+yltwJ+o2w1NSmxfPqwjOZtXkAjX85ggPlqdz/5RU8ds7/6JnWPbw5WMTnk3zyDnz5kaS6quGZ4H8ow4CCPHj4D5JhIyWVVVBWDEkpkHpcDnH9NoTU4jZM65amCf7wELz1suSHr8FTq310XXLcRMl1t+hs1/qy17OhidUkmqypMbM7120rOYnHRXg5lWrinKapKNg0HfGgq/BtxS+TFhUvzz//PAAnnXRSo+OvvfYav/71rwF48skn0TSNqVOn4nK5OO2003juuedaclpti8cDf7sz8Hkp4dmH4ZiTGm9FhOLzt0xrihHEQbdnfzj5PBg4AoYf0/HCLX1eWP198PfoF9n25vVt4T15A3DBraBb+4ruK5XM2tyfQ4UgGyEwJPxn7nmcM7Rlqha/9JRk1reRX791k/mqQ9fB9+VEcsb0ZOJdM7A5/P/NU7VstAbfE4dDcO00wZSrKpi/Zgv57j2k98kjNrmGjfSkp20ERd69VMuyFhUwUL+tFK5wAdjbPZPeW4JsMwkN+p3c8b7DCkWUaPFto1DExMTw7LPP8uyzIYrbHSncf62ZvyQYB3JhwwoYMjp4ux2b4NPXTavLwYLQC/S+3XDJjWFNt13hqjRN5eEiJXQbHP35WGXZTFg5x3r7mHgzz8u4syxf8sRsD01FS0MEe0pTKK4WpMUFaRYB2zY3T7j4o25rac/Snix8eTzjp81p0kagMTxucpPj1UY5y+R7OIa66NbgeL5vJ/nVO8mxD0HDxh7Penx4ojvxKFGRFM/OPl3puW1v05NCg7g0GHpu609MoWgndIyYySOFvTth5U/W2haGKJEw+0v47dkw42OzrRXLgqsadrXcnn+L44yzbIlohGYLva3TUuzdCi/fb91h+Owb4F8z4ITzwhpmx0Er/Qu25kfX2uDzST74n2wxA4CUgq1zBlFV3FhxxYlkxsdf6jeyZ0XVN0GjlHI966jwFTEp4RrS9C5NsvG2FzYM683GIb1w2xt+5gWlmaN5NPsvXPGTjdt+qmbefq+lB0WF4kiiVZPU/eL59iPrbdOCJBPbnwuP/y6y6KSFM6FHv/Cvaw/odhgyCdbODG/rKCW7Ptd9eTHM/Qjmf2o60TpioOdgmHo75LTA72Xme9bbXvdXGHtGRMMkWPTZTI+P3kK95CfJS09JSlogjU5DpCFI23gVAyfuwCvdpNu7Eaf5D/Gv8VVQauSH7POgkctuz1qKfPtpaSfmhmjoGFj87ArBjv457OzblZSiCuK8PXhzz0m8mpuITYBXerEJeGuLh16JgjNy7AxM0ZjS006CvX0KMoUiWijLS2tyYHfoNmBu+AfbMpr+TuT323efh+Ig6fbbOydcbkb0hJNorXNP89+83fDQJfDFi1CcD143VJXB+kXw10vhvX9GP6R69XxrQuvXD0YsXACuPib0c0iiE/pkROcrv3KJ5B8PtrxwqWPWtxr5a/rS1T44oHAB2O+1blnc6lpCawoXG05GxZ4Z9nVS0ziYnsRjeX15NddMD+CtnXbdvzvKJc+vd3PbTzUM/KCcj3e0z+0whSJaKPHSqlh8Ghp/pn9nXSnN1+rFETit1lJdWR+V1BFJ7QJXPQmde1u/5qjTzN/bC3eboSyBmPUe/BDl342Vv1O3/nDc2c0a5uQBtpBpb6aNj46hVUrJmy+37jbF5vXwyL2SP94uyc8z2LdHUlrcdA5erIfQt66/iyDHMYTdnjURXW1IjQV7egVtU/fbqPLCb+ZVM2tvGOkEFIoOhhIvrUVZMSydH7qdboNpf2587OfZcPvFcFo/87V7a+TzkNL0k+nIdO4F1z8P1z4Hp98O8Wn+LTFCg54joc8Y2LrK9D8J9aT99euhHarDofew+i0rf2g69B8VlaE+u95JVgABc82xNq4eG6SOThjs3Q27d7Ru8eu6HdKtm2DalXD7NZLrL5b85S6DdavqJ5Kqtb90+QJBnEikp+Mo8r07w7q27nf87tpRVLhjwhgT/rayHVeMVyiaifJ5aS2+/gAqSkO3e/AFOJgPr/zdFDtlJWb14YZUVTRvLmUlzbu+vZDdz3wNOB6++hdsbZAeX9Nh+Glw6s2miNm2ygwrDbXiVpbA7g2WUvJbYtIl5tZRIKQBEy6MylApcRqzb4tl/jYvb/zso8It6Z+hccdEG2lx0XtOKbPwMfbHoGGmBUXKZiSTPuzPt3EdPPQHye8fgGOOF6TbumLDgZcIotJaiCStM0fFTuaAJ/yHjsKqOD7aMJIfc/uEdZ0BLC30sb/KIDuKf3uFor2gxEtrMfuL0G269jSdSP/yG7NyXLCiihEjILNrC/TbhiSkwiWPQMkB2LfJ3HLLGQbxKfVtglk/DudwsdgcBh8LZ1wD37zWuM5UXU6ey++D7J7RGw8Y38fG+D4t99VOD16YugmaBseOhzv/pLFjq+Td/0pWBCjbEy51QXbP/kMyYgw4HBrDYiexojrKsdvNoNTI48fKd6076gIgeGnZsczb3RfZjGioco+k+XWwFYr2hxIvrUWlBWuJ2wX/vNu8I0eiWzTdvDaUdeGsX0XQeS011TB7OpQVQb+hMGJceMn0WpKULPPlj0FjrO1zCAHZwX0Lwub8adDnKJj5jpmsTmgw5Fg45TLoOyK6Y0WZ8jLJ7O9gxWKJzwf9B8HkswWDhsGmdaEtKJoGyalw1Y3mAtyrr+D+/xOUlki++0Ly4ZvRmWdVJSz+EU6YBFn2vozkdNZVz8VNdXQGaCbhCBeBzoH8c5m722LtrgA4NJTVRXHEosRLa9GjLxTsD+zAqem19vQwHQmcsWbfPp+ZNffsy+CtZ2DOl/4LvPQdAmdcEv78fV742+9g7leN+41PhN89DuNPC7/P1iRnAHTtW+v3EoTRp0BSevTHH36C+epAbFpvFlKsrqr/k29aB198JLnwcti2yXQPCiRg7A4YPwmGj4KtG6GqUpLT0xQxySmCCZOjJ150HfbtMbMLl5dJtiztg1Hdm/Tu+cT12USlLEKTOom2TmTaevNz1cctnmE3MgTHxJ3Lb3emENkTjIku4KLedhIPD5mWEgwPCFv7eehQKCJAiZfW4uzLYPGcwOcNHxQeCL9fZww8+1njY/f8A7r1hE9eg8py85jdAZMvgN/ca14TDlLCXb+C9Suanqssh4dvhgeehxNODX/+rYWUZmh0KC68reXn0gGoKK8VLtWNtWqdUPnoLbjp9/D9dNiyof58ZjZceCUMHg4L58Kn71GbfdfspO9AyW/vEvToJcjM1sjqYnCgQW3SSDEMcDrhtecMvvvS1Nomncnp0Zlb7xX06lu/kPd1jmWLa1HzB446knU1c6jyNC97bpId/jiyQfKfigJY8DwcWMchx6GkrjD2GugyvFljKRRtgZLercXYiTCxeeGwfuk7pOkx3QZX3Q4f/AxPfwhPvGf+fOejpqUkHHxemP6uf+HSkKf+2AwvzFZgzxYzz0sodm8K3eYXwJwZmBaXAH9SocGqpfB//9Z45GmYeBp0yTEf5lcsho/ehLdertfOdWzfDH+6XbJ3t7mA3v9/kSVNPhwpzUikbz5rKFxM9ubCX+6StZYZkz6O0XS1DwxrDIFAIwzfqQipMIrJTt7drLy/VT6ItdX2ULoPPrkNDqylkcdz2V74/hHYMqcZIykUbYOyvLQWmgb3PGGKjY9fg6LaLKA2O3ibkW9i2gOBzzmcMDjCMFzDgOcfgW8/MP1cQlFaZK0eU1tRUmCxXQdO4BdFViyRQV2EDJ/pY7LsZ4NnHjdduura5+0PrGMNw/w43Xm9xOGUjDga7vgjfP0JbDhsbQ2HMcfBogBBXYYBbjd8/Lbk1nvMBV0IQbqew17PRstjSCRd7P3Z69kQunEz8BmCzORdSHIi7sPtg/e2ublxkBO+exBkkC2ohS9Ar3Fgs5imWaFoByjx0hLs3AIv/h9srk1INWAY/PaP0L0vXPwbmHod5O2B3B3wp+siH2fYGOgeRrI2q1SWw3WnwcEQ9ZUO52DotOxtRkona+2SLbZr7+zbBj9/C5Wl0KkLHHu29d8BTa0X/vB44G9/anrcigFOSnDVwOIF8POP8Ns7BeddAn/7U/jq5ejjILtr8KLqhg8WzIHf3imxO2r9bvTOlscQCJL1TIY4J1BjVHDQl2vpqkjUmCYkDr15CfR0AeuKDdj+I1SXBG8sDVj/NQw/v1ljKhStido2ijZvPws3nA5L55mJ6cqKYck8Uwy8+7zZRtehS4/wxUFDBo+Cf74TnTkfzr1XRza3YPWY2ppu/SG7N4cqCAogKQYyE6BzAsTYIC4Rhh7XptNsNh43vPInePAS+O5/sOAL+PQ5uPdMM1zbIgOGtI4/Z5215sWnJIlJEj2CXZnrbxWUl4fOX+3zQlVV/f8T9FTS9W6WCjMmaZ0ZHXsWumbj6LizGRoziTjRuEyBQCdJyyBR60Sy1pnejlGMiTsPG46AYzhcbuIqqtG99arLkIK95Skh5xSKGA1YbvEekbe+2eMpFK2JsrxEk1U/w+tPBD7/33/C0NEw7Bjz/+HkHgHI6WPmaLnmLugfpSRqh7NtA2xcFf51CUmRb1G1BkLAxXfBv2+DBDukxdGoFHKs3aw+XXEQUjtwZoz3/g5LvjN/bmiGkMCnz0JiKpwwJWQ3p5wl+Oz91ovGkRJ+mmOGOs/7wXpUe7fukJYO6Z1CX+NwQHxC42PDY09hYeXH1MgKDreS6NjJsvWlq2MAaXpXRO3nRRM61UY5VbJxtj6JjzKjgFiRxLj4C3FqZiXscfEXssm1kHzvjkNtu+48QP/1O4hxeWqvhaJOyaw8eiA1MU5m72xekVCvhAvSC2GTxW1QPTrZlxWK1kJZXqLJ038O3eaFx+p/Hjmu8QIajAuvh//OgMdeaznhArBgRmTXTXug/YdeDjkWLroJ0uL9/94NLzx/DbhrWn9u0aA4H378Ivgq/uXLlvZ1MjoLbrlbIER4NTCbw/KfTd8Uq2UHpIQzLzD9VyZMFkHflqbBSaeBzdb47x6jJXB8wiX0cx5DrEhEw0acSKa/cxwTE3/NoNjjKfcdZF3NHDbUzKfIu49drjVscwfOslcty9hU89Oh/yfoaYyOO4tJCddxfPwlnLwlgWErNuN01W8NCSC9sJQJ3y/h6+UDKKxK8NOzNXQBw1I1jkkosX5R/8kRj6dQtAXK8hItfvgMcreFbretgXk2syuccJopGPzdeX0GVLghqxfkFsCs72DCKTSxrXu9Zg6Z+ERISmnOu4CaqtBtDmfaA3DKlOaN21qs+y74/oL0wfR/wgV+nDnaO6vnBQ4PqqM4D/Zshu6hI23GTxJ0zYEP/idZ1gpRxQcLYP/e8K55/w0YPkqS1UVw7sWSLz5o2kbTICERLrg0wNaNiKGvcwx9nWMaHd/r3sTamlmNEsztdFuzSu7zbiazYDybVzswDOg3CLp0i8VZVQNrv8bMSNMU3Wdw1e6ZvMtYS+M0RMMsCzAwReODyXForlRrF9pjoeuIsMdTKNoSJV6iwf7d8I+7rbU9/LHyd38z87tsWGneZb0+U7CU1YC7diEq2QSbNsO7r0Pf/vDaR9ClGxQXwiO3wpol9f0mp5lh0udeEdl76R6Bufrk8yIbq7XxecxtoVBs/il0m/aIq8Y0k4QSMC7rWWd79xPceg9cd6FsmWoVDXBHUI6opAj+8aDkHy8IrrhekJQEn74vG4VoDx0Jv7ldkJ5hPfi40JvL6prvw58Q4Km2s+DFE/nvjzZkg+/7sJGSeya/iZPA+lkAx7CdRKopJ9bymH2TBMdl2jiru42Tu9rQhICYrpDaA4p3E9Rx+KzHrFuAFYp2ghIv0eDLd7AcVdAps/H/4xPhyfdh4Ux471X4fk7TqsYNV40d2+Cq8+HdL+CayU0XotIi+M8DsHcX3PTHcN8JTDgTnn3IugVG08NPetdWeCxuB/maF+nRZmT3Ci1chAaZ3cPqNj5BcNxJkgWz22cqn53bYOsmSd8BZsTSmefDpvXgcpk+MZnZtQuzlOCpNhdqe3BhsM0VWfElwyeY8chZ5G/OQsrGgmDdKijsvosuicGNfwLoTR6r6BlyvBgNxmfrdIvXOD7LxoTsWuEC5vsccxV8/2jt7cnPPWrM1ZDcxeK7UyjaD0q8NJf9u+GzN6zf1S+9qekx3QaDRsMPVzcVLofj85kC5pZLgz9Bf/JfmHoNdA7zxhQbB/f8Ex6eFtr5QNfhuMlmPpmOgNOiH0G4jtTthSHHQmpnM1eNPxGj6TBiQtjlD3Ztl/QfZPqkVFVa90lpTTashr4DzJ/tDsHQEQ1OluXBolcgb53p1wSQ1AWOuhB6Hd/E6uCRLop8kaX93b20J3kb/X/nDANq3NZuuaXEWWpXY8D3e33o+Hhts4esWMGHp8QxJK32M5w9DE6+Fxa+BJUNnHcd8TDyUhjYjrNiKxRBUOKlOZSVwG/OMsNTrTDyODjrUv/nrr/Eej+6Dhs3Quf44O1eegz+9B9rfTbkhNPMrLxv/huWL/DfRmjm67Kbw++/rRAC0rpB0Z7g7fqNa/5YNZWwvzZbb3Z/iIncAROAyoOw6GXYt9pcgDU75BwNY6+D2CSzjabDdY/AU9NM54eGglrTzUiji++yPGTuTsmz/5Bs29ygG619ipcmpgxPDWyZBeunN1606yjbB/P/DSW7YdRljU75ZOMHCHeVnW3z+nNwewaazSBn9C66jtiNpjf9RWydPQChGUjDv5fz3O3j6ZO+w++5OqQths/P680H2z08usJahfM622xBjeS8GVX8PCWe9JjaOXQdAVOfgbwNZpkAZ6JZEkBFGCk6MEq8NIdnHrS+vdJnEDz+P/97y7t2wOoQ6fcb4vNBgJtjI376wbQMZYe3TQDA0KPN+R7Mh3/dC0vmmiuX0MyEGUkpcP9T/ssTtGcuehhevDbweaHBZD/WsVBUl0NZvmlFW/I5rPym/ilfaDD0ZDj9VnBY92M4RNEOmH5fY2uK4YFdC2HPUjjvSUisTbjWfxTc+xp89SqsnGteY3fCcWfDWddDirVcPAf2Sf50h2ySXLlODw0cBh4XjYRNW9LI0lJTBt8+AKUWvH/XfGaKwIz+hw45RCw2nHhxsXtpD+Y8cSpelw2hm29+43dDSe5WxGl/mk5CRuNq8VXF8QGFC8D3W07l8pHv4bQFflARQ84mJ0GjZ0L4YV4+CSUuydtbPdw2tIFFVGiQ1cG+qwpFEISU7fI5yjJlZWUkJydTWlpKUlJS6w5+1mBwW3gyEho880ngEOcXnoJ/PGx9XCEg2QlpFhbC9Ex46StIshh5EIjc7fDT9+b77dkfxp1sljboiOxaDe/e0zSNrM0JV/4Lugyw1o/XDcu/gmVfQNFeQvo9pXWF37wSfjGfd68Bd2Xg8/EZcOGzTY+7qqG6AuKTzcKcYfDM3w3mzwy+G5qYZJYFaGs/mMxseOZ/DRb6WX+HPctD+//U0edEOOGWRoc21SxkycZdfHHvVFOMHOa/IjSDxMwyzn/yPXS7OY6GjcX/upS1ixKD/k7GDNzJ3cf82RSgh9PzOJhwBwDlHsnA98upjsBRemS6xsyzm2ntUyhamOas38ryEilVFdaEC8Af/h48N0t5mbUokTqkhNgYMyrJqYM9iI9GUQF8/T6MmQRvvmKGW/t8MHosXHUDHH2stTFzesMlN1pr297pMRzu/QbWzYZ1s8zf/ZBJMGi89aQmu9fAh3+BmorQbeso2gvz3oCJYZSE2LsyuHABqCwwI0pSD7OwOWPNV5i43ZIfLTjnlpeF3XVEpHeCgwFyrdns8Ke/NThQUQC5S8MboGhnk0N9nKN5/fMMU7TIptZSaWiU7U/Bs/xUBp5YTLyWQmdbbzLO1Fj9U2ARKwQMmdgLznkV1k+nZvM8ql0uNsss/q5PZWtBL7K/ruCUrnau7G/n90c5+etyi/eZBlRaKO+gUHRklHiJFKsGq7QMmByiZkivPtaFSx37S+p/jrNBRjzofhZeacA7r8IDfzHnXBe59O0X8NWn8Ie/wG/vCG/sI4UhE81XuBTtgXfvA2/4iwrLp4cnXnZZTLCyc2FT8RIhVZXWahu1Fnc+XM2imbF8+0XjGqb9B8Pv/wKp6Q0+9wVbwh/A1jRaziYc7Pq5F9IIHBckNMmuJb05f3L9+CkDDYxePsQOrUlJAE0zK29POgOwx/Af/RweqDoVTYAhMf2UPJLcSsmSAhf/WuPijQkx3D/CyeOrXPgs3nJ0AUNT23nCSIWimSjxEilxCeaWTKgaQBPOCt3XmVPgoXvNVSMSqrywrxy6JoF22M3Wa8D6bYBoLLjqRMw/Hobho+C4EyMb+5fI4k8iD6euqTC3m2wWt3GsWoKiGCEVHw82W+jAtzri4s22Vg2RlhEGnXoXkpv5JRfccAEXX5XG1o3mR7d3f0hK9iMsIkkH3KOp9VFKiccbPPeJNESj97y/yuDMbyspHifp5XDQdYsdrVb8SCSOQQYPP2wjNlYwd7+XB5aZFxt+RIkEXD64cnYNi89PYGxnnfNmWPOv80m4dkB424QKRUdDyfNIEQKuuCV4G5sdrvld6L7i4uHRp5qXKMpjQJmf1aPcZd4JA1mKdB0evQ9uPAsuOwHuuQrmfUOLZyTryKyfG76lrCHeMIRPn5OstesdPfFpdwhOmGRdB1RVwl+fhN/9RXDjndGahYGmScZe+yMe3Cyv/oaYWBg2SjBijPAvXAAyB4YnYOxx0PekJoeFEHTpRtCELJoGOT3r///0WjcHXRKPBpvHuJk/tZKVE6tZNaGaHy+o4quR1eT6DKSUPLq8xkI5SPBIeGqNi/HZNvolWbs/nN5VY1xmBw33VygsosRLczjzV3DqVP/nbHb4+//MvCmBkBK2bISli+CYcfDfD+Co0f7bOmNCi5tyP+Kl2kLemM0bYftGs8TAyoXw11vgoZvCW2R/SVhNducPTYfvn4eti60JoBirTmzR9bu/8ApBnLVUIwgBq5cJjh0vOOVMjZ59mp+wNblrCac/9DmZAw8AkiqjhCKfheih2BTobdF3yR4HZzwcMP/PmVOC15uWEk4+w2xhSMnbW9yNtna8TjjY1Udhjg93nMQm4J+rXBz9aQVLCw3Lf7F3t3mQUjKlpzUH+RsGOQ4VkVQojlTUtlFz0DT4/eNmXpTP/gdb14HTCSedA1Ouapogbusm+PBt2JcLlZXm//fmmueEgPGT4B/PmpaYinLI2wvvvww7N8H2faH9bPxtisfFQ01p0+OBqPPSXDQL3nnOLDWgaExaN8jfQUSCwfDB2h9g9XeQ1Q9+9X8QnxK4fdF2a/0e3AaJmaHbWSQzW/DXp+DhuyUlRcHbahpUV9dX6zn3IsG//xaZmBLCIKN/Hmc9+mkTAZTv2Um6rVvoTsZeB+X5kL/BjyO8gITO0O8kGHJe0Mivk8+EpQth1bLGXz1NM78m19ws6JxlTvJgjQzpJGtI+HxX+M5EbgPm7veR7LAmSBz+fN8UiiMMJV6aixBm2PC4kwO3kRIe+zO8+py5TWMYTYWIlLBgDlwwGT75HlYtgBf/r/68lfuRftjNTQjQDfPfYMIn1s/HQEozc/CvfttxMui2FkefB18/Gfn1Ru2WXN42+PABuPqpwKYKYdH8L6L/Vc7pIfjni5LrLyaoTvP5oPggrFgiGT4KTpgE27fA9I/DG09oBvYYD+Onzfb76yj07bbUT77Xwcc595DsWMaxZfPpJopxJHaCvhMhZ3RI/yC3T1Ltg0Q73PNXwdefwtefSg4WmOf7D4bzfyUYNdac5O4Kg3O/C+2vFulGowBm7/NyQlboz4IAeiUq8aI48lHipTX473OmcIHgviQ+H1RXwR9ugvKdjc8lOqAqxDZO4mEiQ0qI02sfiIMImASHee7wFaO8FPbsgN6hKxD/ohh+KmycDzuWBf6dxqdBZQiThTRg73rz1S1AArHMweZiawT53Gg2yBpsbe5hkpyiMeEUg3k/BNe/s7+D2d9JUlLhhtsFV90oyNsvWWK5xqWk+5gdHH3FIpK7+LcUVhhFuI1qHJr/8G8pJU+ucfPYSheGBE0chSGPAuCmrg4e7u7EY8BHW9y8ucXD3iqD7FiNy/rZOa+HjQ+3e/hgm4eVRQaGhGQ7XD3Awe3nOjnnQkFFubkbHBsrGo156cwq9la2XLosTYBXwqQuNrJiBXnV0q+W1AWc3EWnS3yUxIvXA5uXm7mCsnpA177R6VehiAJKvLQ0Hg88/5T19j4frFoB3ZPB1uAmFGeHGBvUBDA72zVI8mMh0YVZRqCwJnD4SF6leYdMdkJyTOOIJbV33hTdBhc/DAs/gKWfQ2WxeTwuGXqNMkOhkzOhtACeuSx4X5oOWxYFFi8xidB3Emz+Af/mDwH9Jlmv2xQMaZj9HfY3v+IGwbpVkqLC0LlfSool/3xI8ruHPZxyppMlQXKe1CEEDD99O6Ov/y5kW5eswuOJ4aMdHhbn+9A1mJBl4+weNt7c4uGRBun0G+6iPrfejSElP+f7WHHQQMO0hOyr9LG00MfvFza1jJR64Nl1bqbv8vDtmfF0SmoqCuYd8LGhpGWz9PkkjMnQ0TXB8+NjufiHKgzZ+P3pAlIcgr+NjSCD8+FICbPfhy9fhsoGQrLHYLjyfuiuHmYUbY8SLy3N2pWmTT1cvEZj8SIEZCXAwSooPyy1eJwdMuKahkk3PH90L9i0FSrd4DOa+scYEoproNQFqTGmFSclDbr1Cn/uvwR0O5xwORz3K6goMq0f8SmHLfxWFjVhhk67K81Ec0KD9F6gNwh1PeZqsz7P3hX1Phx1/3YdYVYOjhQpzVwy66ZD4VbTSpc5BIacA91GApCSKnjsGfjgf5I5M0KV4BJIJC+/WME1T31FdvezydtjDyh6hDB92s+40EW+hemuKrBz5exySt3mgi2At7Z4yFxCyEy0L2zwHPqK1E2n7lsQ6C/lk7CrQvLg0hqeOaGpB/PcfV5stZaRlkAT0MkpOKu7eauekG3j69Pj+dvKGmbt8yExn1vO72njjyNjyImgpEATvn0dPvWTsXn3Rvj79XDf642tMF4v7FwLrhrI6gnpWc2fg0IRAiVeWhpXhMkvDvdfAfNOlhFvlgWo8Zp33hhbY5ETCOmrFSUO2B3EgdeQcLAaiqrhpClhp5VvFVbPhy9egML95vxGTIDzb4G4xNafi6ZDUoB6QQnppuWkpjzw9YYXynbD+zfU10Kyx8HgM2H4VLN/3WFWBj6wFrbOgaoiiEszQ3yzhkZuHZMSlv4P1n9V24c0P1N5682xRl0Gw6YApoD5ze2Cq38rmfWN5L9+1rb6fgWle9LYsdPD+Pvf5fuHplK8P97sH9HgX0jrBHc/JOjarRezKuYS2LlGEE9XLvkBXLVKo6H+LqixJhX95VQJhU/Ce9u9jM9yc0FvO/YGDwlWE8fVvetw2ukCYnV4a1JsozFHZ+h8ODmeohqDEjdkxAoS7VGykJYXw+cv+D8nDXDXwOsPwX1vmJ+Z2e+bdbTKi+vfwdDj4LK7oVPX6MxJofCDEi8tzZowCi6CeUOwa8FT/usaxIcpKvoMhqJ8U5RYQQLvvgdX3wbde4Y3VkshJTx9K6w/LOvs3I9h/mfw+5eh7/A2mZpfdBuMOhsWvh8gLFqYoSvl20E0WNo8VbDqIyg7AONvNT8TQkD2MPMVLfatMoULNHZoqZvr8negy1GmJagWp9NcJIWQIYPfakpjSe9VyLlPvsmun3uzc1FvqoriQUi69XZz8rE5jD7Ghq4LIJ6etqPY6V3pty+B4Kddo3Ab/gVIS5dXMiTctKCGB5a5eHVCLMdnmbfOMRk6/7GgSqxqpkEpGrsrDRJsggt62blxkCOgNSUtRiOtaXLg5rFkRugQ/l0b4I6J0HMIbFx82Elpfj8fuwb+9CakRi8CTqFoiBIvLcmLT8PfHwrvGikDF1w0pLntU1679WPTzO2deHvwp+/EZLjwWlg0EzxhJJ/zeuCtV+D+R8J7Dy3Fh082FS51GD544kb497z2VTDy+Eth5wrYv6mxQKjLQ5IW21i4NGTHj9D/5JarBrzx2+A1tYQGm2bAcY1rWqVnWKuOEd/JrPuk2w16n7CV3idsbXS+FAcVnE8yGayvns9u7xq//cSKJIbFnswfNibha+M6soUuyYXfVzHz7HgGp+qcnmOjS5xgX1XoecXo4Pb5F1oCODFb59NT46M+57AoKTCtfaHqQ9RU+hEutRg+01fmq1fhivujP0eFApWkruWYPSN84aLVOtfG+Vl8vQbsLYOCKnPLyGOYCejyK81tIHeQm81v7oPhY+Hk88LbYvD54Nsvw3sPLYWUpoUlGF4PfPe/1pmPVRyxcMU/4KRr67eXhIBYu7kFGBNEaAkNtsxqubkVbg/+lC0NKGxaK2jkGEgIskMnhEGnPnmkdCsO3Ajw4mZJ5ecsrZzOLs8qZIBl/ei4c0i3daWmHSR9NqTp3/L0WnM72KYJ3p4UhzNEFLMu4OLedvoma00S3wkgJ0Hw3AlRcLZtLrHxwSPbrGL4YOFXoRykFIqIUeKlpfjPP8Jrn5wMvVLNsGV/7C83BYs/fBL2lpu5xA9Re4s853I47UIzTGTqtTAhSD4af7iakU02muTvAY8F/6FlM1t+LuFijzEde294EQYMhK6pkBYHjhCGT2lAeYjaWc0hSIK2+jZNP492h+DaW+qW4MMsDsJAaJKx1y6wNAUPNRT4dgZpIVlVPQOAoWmaX1ewQ0NbGrH5+CR8ttOL12d+H49K15l5VjzOAHdTAdgE3DrUwfdnxfPQ0U76Jmkk2KBXouBPo5zMOTuB7Lg2vB0bBkx/2YwwipZ1y+OCqlYqPa74xaG2jVqCg4VmWk6rCAHDhkDe5qZ5YKSEvIrAwuVQO0BLhvQYcLuh/1A47yoYOxG+eR/eesZM/y8lOHSzv1A3KV2HYSOtv4+WxGoFZ287ftJb/TFUFWDZA0IIiEluufl0P8bcOgpofRHQfYzfM+MnCex2+N+LUNBAX6X1OMi4G+bXpvWPDmVGAV7Dw/UDHczdH9hnSwKZsZBn0a2rOXgMSPrhIKdmOPhdz1jGpzn47sx4LvqhioIaiV6bVkliFn1/c2IcfZJM88wtQ5zcMqSdJX787Fn49o3o9qnpEBuFEH6Fwg9KvLQENWHePQcOgZGj4NumJnpKXWbVaCvs2gXvroPM7Ppjb/4H/vdU/f/rQq4PVJgb8MHw+eCK66yN3dJk9sBSzIbTYkGe1sbnNbeADB8UV8OBMqjxmI7ZnROgU4LpiN0QKaHPeDMKKXcpFO0yq1F3Gw2p3Zs/p4Gnw+bva0NmDregaGCLMXPMBODY8YK+Y/czY81SyksFCZ0qSOsZQVoAC1QbZZyZk8Zlfey8s83T6JOgCXM75+6jHFw/0MG9P9fw2U7voU2oZAeM62zj2z3hp+YPhKFJPMB3B918XejmhcEJXNstlpVTE/hkh4d5B3z4DMnYzjYu6WMnyWJq/zahpBBmvBndPjUdRp8Mjmh7FCsUJkq8tAQZmaZTQEWQENk64mxQvQd+mtHUSU5KKA1z22bP7nrxcmBPY+FSh02Drom1MacJsG9f4xICdcVbrrweJpwS3vgthc0OCSlQEdyPgmoLv/O2oKYM3NWwKQ+KquqPV3ugrAZyS2B4l/qtJKFBWk+wx8OHN5n1qYQOSFj+LnQdCSfeDo5miLWkLJh0D8z+x2EWK2mGa59yP8QGtvyU+PJYWvMZqf0MUiOfhSUcWixCCP59fAzHdNZ5br2bzaWmPBmRrnHrECfn1RYufGVCHI8dY7C+xMCpwchOOm9s9vDdHm9UyldKJJ44n5mip7bDm9ZXcHK6gx6xOpf3c3B5vygM1Fos+S66dT2FZn5fz2onDz6KIxIlXloChwN+dTX89/ngzm9pMZBS66RXXND0vMdPMrlQJNUuNlLC338fuJ0QZvhDvA3+/Ca8+qxZ3Rpg8DC4bhqcM7V9ZdhNzQgtXipKWmUqYWOPgdzixsKlIS4vrNgDY3qYpoSuI2DoFJjxV5C1olY2+CztWwU/PApjrjGz6yZFmBisy3C48Hkzf0zBJkBA9lDofaI55yBsrlmI/0T10cVBLE7NFGmaEFzV38FV/R1UeCSagDhb089oRqzGhNh6S9boTnrUhIuhSdzxjb/XAnh1Tw0P92vjaKFIKC82H1h84Qac14b6Q+19rtYeltEVrv0rdOkT5YkqFPUo8dJS3PJ7s9Dipg3+BUx6rJmKPxCaFjiENhDJKdB3gPnzJ6/BmiWhr6ksh3EnwKlnmZkypQR7Owo1bkh6F9i7LYggFJDSuVWnZB3N3KoLhteAfAf89h+Q3AXm/dsULP58k6QBBVvg69pQ1Lg0GPNr6Hls+FNzJsCQs4GzLV/iMqo46NsT/liHMch5Iltdi/EQ2MLY3znO7/GEMBKzfbC9+b5QEomhS6o6eZqEOviA5WUhao+1V9IyrUcYCc18oDF8EJ8EN//LrHu0ZoFpWczuDf1Hta+HHsURiRIvLUViErz3Fbz0b3jnNSguMr/QmRmguwgYmqDrcNQ4GDkOykrhqaesZ+k9arQ5htsFbwdLgXrYeDG11h9bO/84HH8urJwTvM3481tlKkGpLIOF02HTMkBC3xHQtV/g2lIN2b4FbPHmtt2uhaEThtVRVQRzn4Dqa2DQGc2ZvSXcsvlesel6N3o6h5Nt78OCyvdxyaZWqX6OseQ4/ReddPskRS6DJIfm1/pSx8fbPby8MXJhUVcHybBJqtKbChcwbQ6OQOU52jtHT4b3nwBfgN+RpsOwE+DEC8w8Sz4v9B5m+rTYax2Pj7MufBWKaNDOV6sOTkIi3PVHuP1eU4jExsI1p0BhkEgMnw8K9sGvfmv+P78UXn/J2iJ2Wu0NZN0yKC8J3V4IOPHM9lkCwB/DjofBx8KGn5taIzTdrKty/LltMrVDbFkB/7kdXNX1c1w936yFZAVpwIbFMOLEyPJtLH7dLNRoa9loFjvNd8SM1ZIAcGrxTEq8ljzPTna7V+GTXhL1TvR3Hotda/o+lhV4uf2nGtY3KIg4MFnjqeNiOKZz41uaISV/XR55uH//JI0eiYJCu4+fXIEFkATOzgjjeyQN2LEANnxtOmJrOnQ7GoaeA+m9I55vUAwD9q+BinzT2tZ1pLk1mJAC598MHz3d9BpNMwXKBbdAdi/zO6hQtAOUeGkNdB1S08yf4yyEDjas0XPXH2HFUli5NHB7IUzrSdVB+M+DUBGkdlFDNB0uvcla2/aApsO0f8GHT8GPn5lJ6cC8wY6aBJffCzFtGG1UUgD/vs20fDVKty/DC+F2u8z8KjFJpqNvWEhY/SmM+lWY1zWmWBZTLIuxYSNLZOEQjRfmfd5NzeofoLOtZ6P/Z9p7kmnv6bdtHT/s8XDJzOom/isbSw3O+KaK906OZXK3eqG4vthgd2Xk3i7/OSGGMRk2lpZ6OOFnl994Nx3IcGhckmVR0EkDfnwOts+rd5Q3vKalbddCOPE26HlcxHP2y54VsPBF00JXh80JR11kFuE89UqIiYcvXoSyBhFjvYaZ36tsVaBV0b5Q4qWl2bsTlswzF6SkFBh6NOzeGvyaSefU/xwXD+98AW/9F554tGkYtq6bN78kDd5/HhDWn9in/QV6DQjjzbQD7E647B447ybYttp8rz2HQEqntp4ZzPukVrg0s9JOTn9zURtwKqz+JPz+CiIXFiWyhAXeBRykfgHT0RmkDWKENgKttqzBLrf/VP5WiRPJZBwmXkJhSMmv5zYVLnVI4Jo51ey6zIZeu4WTXx3538ImoE+i+X6PTrbz5vBErl5Tjk/Wl5Y0gM5OjW9GJxNvVMGG+WZhS90BnQdAr+ObRoRt/N4ULuC/ptT8/0DmYIhNiXjujdi/FmY93tRa6XXBsrdMv6ph55vbQsefa36vaiohIweye0ZnDgpFlFHipaWoLId/3A0LZoR/7cjj4at3Ycs6SEmHKVfBdTfD5dfAGy/Dmy/D/r1mhtQB/aE0F5w20yxsBSEgpw+cfVn4c2svxCfB8BPaehaNWTW3ecJFaNBjkCleAAafDbt+htK94fWrRfa1LpflfOv9Fg+Nt0d8+FhrrKVG1nCczbQIeJrh8xIrkhgTf94hIWSVL3Z5QqY8qvLBV7lezu1hWl+eWxeZo64u4IJeNtJi6ud4UVYMJ6Y6eG1vNUvLvDiE4PRODi7qbCd21buwfnrjv9P2ebD4NRh7LfQ/xRQPPz5bL1wCYRhmTqDhF0Q0d8D0S8ldDIXbYNu8WuESQPat/MgUyo54857Sf1Tk4yoUrYQSLy2BYcCfb4C1YWTZPYSA357VWIi8/QwcOwkefAFuvM18uWpMB9BLx5nCxXL3GjiccPc/VERAtGlOHRehmduF1zaoh+WIgzMehuXvwdbZ4LPYf9+JEU1hjW8NHjwBw5+3yq0MloNJESno2PER2gF5bNxU8j3bKfcVYdPsZNp7kWnriy5CFAPyw7z91iyKPx0wxcuaIh+zLF7TEF1At3jBw0c33QbKdGrc2/uwcOhFr5gFLP1heGHhS+BIgJ2120IhkaboiJT8TTD7n7W5gYIU3jw0Rw/sWgz9IvvcKBRtgRIvLcGy+dbClP0izXShh7NoFtz3a/h7bSZMZwysmAXVAfKGBKL3ALj3CejZP8L5KQLSZzjk51rftrM5TF8YRwwcdw6cdjWkH5avxREPx14HOUebeV1CYY+LKFzakAY75I6geVsEgm3GNkbro+niGMBO98qgfSZpnUmzZZNmy/Z7fnOJjz8vrWFjiYFdg7O727l3hIMYm3+LTLCIoobUhVB/vtODrUEiOSsk2eHKfg7uGOYgPcaCZahsf2Dh0pBlb0OF1TpVwlrdKb/zOQAzHqkXulYsdkIzhU4YSCkRbfTwI6U0BZdmb7M5KNoeJV5agllfmM6l0ajO2pAVP8HrT0D/4TDmRKiuDO963QZjJijh0lKcdBEs+MJ6+2PPhEt+b1rCQt2E11noV7PBmY+Yi1GYePDgI/Tntbp2u6ifYyy73WsxglhfhscGzs5896JqXtnUeHvq3+vcPL/BzfTT4hjTuemt6cq+dp5bH9r6dFU/c8uozC3DKtY49+x4Bqdqh/xlLLF9PpbKVlgWLph9dY2wptj6r8yFPZyUfNKA+NA+Y9LnRubOwcidA9WFoDsRWWPQep6KiI8wSWIYyJpijJ3fIfcuAJ/LHL/r8Wg9T0PEtHSOZ0V7Q4mXlqC0OPrCpY66/C3JqXDxjeFdaxiQlmFGLr3/P9i+1eznnAvgtHPMzMDNobwMPnoHvv4MCgvMm2J1lRmKffwEuOo3MGR488aINlLWlhSQEJvUvK20HoPMkNJPnrHWvnCvaUELhafGDHENRe8TIaWbtbEPw44dHT2kgIkTpvOpTbMzPv5SFlZ+hJvG/i8aNo6OO5tEPc1vH69tcjURLnV4DDj7uyp2/iqecq/AkNA5VqAJwYBUnSGpGuuKA1sTRnfS6J5obkn1TtIsWV0EMKWnjWHp4W9lUR2excISsamRRxvtWBC+35UtJmABzjqk14Vv2RNQupNDwsjnQu77Cd/+n9FH34FIbbmaCLIqH9/PfwNPZePxd8/Gd2AJ+jH3IOLaa4JKRUugxEtLkJVjRgEdXiE6mpQWw8uPQ3Z3yNtrTSwJAYuWwu/vMq0wPq8ZZjzzG+g3EN76DDpFeAPYsQ0uPwfy8/xnhP34XVPY/N9TcMlVkY0RTaSEld/Aog+hqDZTbFo3GHshjDwzchFz+q8hLRte+WPotmmZ1vp0W9warPRTYsIimtDoo/Vhi7El4NaRRNJHq0/5Hqcnc3LSdRR6drPHswGJJMPWk672AUHN+Y+vDJ500WPAkI8qKak1snSLF9w02MFvBjqYflocJ35ZSa6f8Oc+SYIvT6uP7Lm4t50Hl7lwh1jLJ2TrPH1cbPBGgYjzL9AixhEPk/9kbilGgjeCnDYjLgqZF8jYPr2xcKlDmtXpfateQD/xcUSEzuKh8K1+BTz+MlRLcJfjW/s6tmPubpGxFe2T8O3LitCccVHLCpc6BKZocThNsRSKPqNNAQH1RSDrHIO3b4Gbr45sHj4fXH9JrbUlwKNu3fH774CN6yIbJ1pICd/+G75+Eor21h8v2gPfPGW+Ar0PKxw9GeKSQrfra3FrwGMxsqeZIdpDtaE4cCACbLb0F/1JFk0LNXayd2dE3GmMjDudbo6BQYWL22uQb2F9LWmwO7SnUvLHJS6mfl9Fgh1WTE3gxRNiGJam0TVOMCJd4/UJsSw5P7GRv0xajMbjY03L1uE3OgEk2OCtibF8PDnOeqkBaZiWsLrPR58J1q5zWKl5JGDqc5CaY61PfyR1Cf+anBBWF8OD3DOPwFtRpoCQ+avCH9sCRvk+KNsZvFHJVmTF/hYZX9E+UeKlJeg3NPww5ORUM/w3HKQ0rS73PWFmyg0kYOIS4Pq7YUWQm4vPB8t+htUrwpsDwLxZsHO7NcEmBLz5SvhjRJOdK2D59Nr/+Lkhr/gadkQSKVaLpsHpIYRgTDyMOc1af1az86Y1L5FYgkjgDNsZZIiMRsdt2BiuDWesPrZZ/QPUNENfzTvgY/hHlWwsMbioj4O55ySw5qJEZp2dwLk9/f+Oru7v4K2JsQxKrb/VOTS4oq+dZRckcGZ3i06fZfthwfPw1hXwzlXw3rWmE67NCcPOC36tI96aZTShEzgitADVMfDUMC8Qoa1H1QfBG0JACx1ZtivMsa0hD1gLfjDKc1tkfEX7RG0btRS3PgRZ3eC9FxtnvPUZUO6GmtoiiDE2SHbClF+b7R//XfhjOWPh/qfgzkehvBQSkiB3u5kgLy4BRh4Hu3dC/n3B+9F1+HE2DA/TWXDhPLMukpXaPVLC91/Do0+GN0Y0WfYFQZ0shYBlX0LvoyMf45TLYfNyWLvgsL41c8vu5n9a83cBSMgwXxUhtoW6HxPZXBuQJJI43XY6pbKUElmCjk6myMQuolOsM8mhhR0B1JD91ZKzv61k3rkJdIu39ux1Znc7Z+TY2FMpqfBIuiVoJIZR1JGD2+HbB82kbnWfGXclrPvSDH8+46/gTISVHzbdtuk2yszz8tX9obd0EixuIwaj7yTYuQgOrCOwpaQBSVmht6gsbQXJiPMLhey5Mkg5lQaIUAJLcUShxEtLoWlwyY0w9Vp4/lH44k2ocEP+YRFC1V4oroHcfLjsZjN/y7MPhZczpHOtqTg23nwBDBhuvuqwahXx+nekDE6YK9HBAtO5NzFMS1O02LuRoHOWEg5sad4YNptZyuCn6TD7A9i/w9zeO3oynHJZeOnWhTAzoC58KfD5tF5mRtcokSyS/W4RRYPTc3Sm7458W7XcA8+vd/PoGOv1lYQQ5CRE4MckDZj1d//CQxqmoFzyBky4AwaebuZYqSwAZzJkDqjfLup+jJl4LuDWnoDuzRDLdeg2OOU+WPOpGXkUastxzK9D9xmTDnGZUBUkYkoaaBkt5IxvVZQkqxIGvyTUtlFLY7PDLQ+YhfYOFy4NeeUZeOgeOOtX8OVauPufcN6VwZ/ONQ0GjYAcC4XcevaG+BB1lbxeszJ1uIwea83q0pDPPwx/nGjhseB0EZGIOwzdBuOnwF/egecXwtNz4Mo/RlYnpt/JZsZdaBAKXbsYJ2bBxLs7TNLBF8bHkt6MupE+Ce9vi8LfxwrrvmpcD6gJEnYtMqOOdDtkDzWTBOaMauznMiiIE7jQzEKJVv1nQqHbYcTFcMkr0CPIVt+AU6FbaCurEAKtV5BK5UKDlL6I5J7hzzUE0vBB6Y7QDTU72OOQJduRVflRn4ei/aHES2sghPm4GGpxeetV05lV12Hy+XDLg/CHvwOi6bVa3fbDX6zNISYWLv21eZ0/dB269YDxk6z115CTz4CsLtYXT90GW5tf2C9ivMGjXQCzhlJ7QggYcxWc/TezanTnQeaWxPhb4dx/QnyUo15akDibxqoLEzgrR6dh3rm4MCKVS93NcKi2iqsClr8Tup00zBIOwUjpBhP/YNY8QpivOhHqTIRT/2zRqTcMdDtMuBOOvd4UuHUkd4Pjb4ax11nuSnQZh+h9uHiu/TehK/qI6Bd4lTXFZni0z4LlRXdizL8f3+K/4fvxT3jn/A5jT4gyDIoOjdo2ag0MA2bNsBDBIuDDt+DPj9UfmnCWmSfl5cdhT4MnkP7DTOEy8Chrc1i1DH6a57/+ka6bVpkX3gwsboJhs8HL78Jl55jbQaEQQGwbVX92VdVHWgUjgvT1rUJ6bxj3m7aeRbOJs2m8OclcrItqDOJs4DYEYz+rIL86WJ5fky5xrWBl2vC1WbTQCvmbIGtw8DbdRsFFL8DWOVCw2Uxk2eUoM6eLH78Tn6eCmrINSMOFLaYzzoS+iHATEArNtLD0n2yKMSFMkRSmlU4Igd73XGT2MRh75ptbSLZYROZoRMbw8OcVAumpxLf4cagptnbB4WHU7nKM9W9h5K9CH3mLysR7BKLES2vg81lbMJGwzk8ysuMmw7hTYNt6KCkyfVy692naLhCrV8ClZ4MngKn9uAlm/pUukSU4A2DwMJixCJ5+HN57I3hbrxdOPyd4m5bAVQmvTrPY1l9OCUVLUFf8MAb44rQ4pnxXyf4gD9sacM2AZiZUtMK2+dbbrngXYpNNq1gwnAkw5OygTaT0Ubr3SyoLF2LWrTady3VHKqndL8WZEMG2oxAQkxj+dYd3E5+FPuCiZvcTCrl7dq1waaaFrXANxs4Z6L0sRvYpOgxq26g1sNutZ69dvthM+HY4QkDfIXD0+PCEC8D//Rk83sBVp5cvhtQobDt0zjKjiF54K3AbXYdjT4DhbVC5dv6bULzPWtvq8padi8Iv/ZJ1Vl+YyMld/Fu+dAEDUjSuH2jh+7R9GXzwF3jlRnj3flj1LXgsbBnWYTU5YB2LX7PmTxWCktyPqSxcgClcoG4B97lLKNz2Eu6qEFtURwDG3h9ptnCpRe74GmmE6ZOnaPco8dIa5O0Ht8XoIcMH998OVWHWLQrEnt2w5KfgeSYqK+DLT6IzHsDkM+Gfz9fX7LHZzBfA0cfC82+2vnOp123mb7F6Q4y0MJ6i2eia4P1T4nhwtJN0Z/3nxK7BpX3sfHV6fPCkcqV58Nyv4d17YctCyNsO25fA9H/Bi9dBicVkZklhhi57XabzbjPw1BRQVRQor4kEaVCe932zxugQuKP48OCthvI90etP0S5Qd+jWoDqM/ANSwuKf4Lgh8OwbZk2g5nDAoqXhT3eaETaXXdO88eo4/xKYdBp89oHpnBsXb24VjTi6baJiygvBHcbfof+4lpvLEYghJTvJYwd5lFFFKgn0JoscOkXkb6AJwW1Dnfx2kIM1RQYeQzIwRSfFGaSvskIza/K2xYHblObDe3+EG18JXcCy/ylQuNX6pIUOlYXW2/uhungF5jNloJBqg5Lyzcz2rWGddhAvkmScTKY3R4mWL47YajiTzOR40UJZXo44lHhpDbK7mIt3ONaU8nK4/lcwfS70aUYV6PTQ1WIB0y/nz7+DkmK4+a7Ix2tIcgpc3U6cS8OqFSNg3CUtNpUjjT2ykC9ZQiWNt0yWspVU4pkix5EuIvO3cOiC0RkWnKd3LDctLSGd4iUczDW3lPoET4tP7xNh61zI32BtstIwfVqageEz7xE+BDuSMjgYE4/d8NGnrIBkdzX7Y5P4sO8xGFp94rYavLzFGubL3dzM0WhRdp5tC7Su4zG2fhat3iAhO0p9KdoLHf9T3hFwxsAlV1qrP3QIaTr5vvZC88bu1ReGjrAeRfTEo7D3CEyzndgJMvtYsPoIOP+PkN0MwWiBvCKDPz/v5aw7PJx+q4drHvawYGXzahNZxZAGy+U23pfzeV/OZ7HcjBFhXaQCWcqHLGgiXOooppL3mEelDMPXJFzc1aY1xWo9KiFgm4WU87oNJt8Pg84w84hY6bfHsdbmEGhIRyq7E1J5ZfCJTO81gp+z+jC/ywBeG3gCX+cM5aO+YzACfIZ3UcrHbGzW+G2BlBJZU4ysKcLweTB2z8HYuyD0hZYQZpi3Pcph6Io2R0jZnAp0bU9ZWRnJycmUlpaSlNRGGVutUF4GF59hbqEEcpz1R3IqLPfjwBsOP82Dq6ceqgAbkvMugidebN6Y7ZGNP8LHDwU+n9QZrn0G4lNbdBqzlxo89JLPr/fN8UcJ/m9aaIOolJL9FLOTfCSSbFLpSSZaCHG2Tx7kA37Ee9i2hAC6kk4icfQggwF0w24hXPwLuZgt7A3pSTSOgRwvBoXsLyLmvA4L3g7jAgGjz4HTb7V+iafaDHFe/FrgNkPPhdFXhDGPpuzy7Oc5bbUpUA7/W0oZUnzrCP6PSR3C+iKlRO6Zh7FzBlTXlr7QbNHd4onLQh97jxIv7ZTmrN/t4hP+7LPP0rNnT2JiYhg7diyLFwfZs+6oJCbBB9/AjXdYr2kDZrmA5nLcifDyO9bT8c85Qh0CB54Ap04zfR2EZvooaLULdLehcP0LLS5cSssNHnrZv3ABWLBK8sb04LlF9shCXuV73mEuP7GBn9nEJyzkFWaQL0sDXlcha3iX+U2EC5huzHs4yEZy+ZblvMJ37JUHWSN3slBuZLXcSY1s7HTulT62sM+SC/QyttBiz0lbfgrzAgldwxRS9ljTAjPxboip/R7VCQTNBkOnwKgwi7E2wCcNlsi9vGTbgCE0/yLFgu+QD8k+2n+knJQSY+O7GBverhcuED3h4khC63se+rH3KeFyhNLmlpf333+fq666ihdeeIGxY8fy1FNP8eGHH7Jp0yY6d+4c8voOY3lpiLd2O+jpv0F1kHBMTYPBw+HzWdEZd/5M+LWFHA0OJ2w4gsvLlx+E1d+Zvg+OOBh0InQf3iqOxI+/4eXrBcG/cgmx8NXTTbcpimUF37GcPfh3ZBSAAzu/5mQSRdPqxNPlYjYSfpithsBAoqNxPIMYQz+EEFRJF8/xteV+htCdNBLpSWcyRUrY8wjIi9dDYRgVjZ3xcMcHYfpBNcDwwp4VUJ5nfn5yxvjPoVKRD7nLzCik1O7QZYTf7VuX9PIvFlIcYOutEZJDVSECMY2j6SlaVoQ3F1m8Bd+Sf7TsILEZaN0nIXImIFqoaKSieTRn/W7zv+gTTzzBDTfcwDXXmFEuL7zwAl999RX//e9/uffee9t4di2EzQY33ALjJ8JZ4wO3Mwy48vrojXvcSWZm22CCCaDHEV7gLDEdjo/8Kbk5LNsQ+lmhohqqaiRxMfWrVJms4h3mUkPgkHsJuPGygu2cyJAm57cTpLBeEIy6PCMYzGMdNnRG0QcnduzoeLCWhXYduxEI5rOOrjKNcxlLvAjDChmI7sOsixehwcV/jVy4gGlp6R7E2dfrhoUvwvb5HCrtIQ2IS4cTb4PMxlaf51lqTbhASOEigC40PxldS2PkzjX/FhH6WlmiugBj0wdQuBZ95DQlYI4w2nTbyO12s2zZMk455ZRDxzRN45RTTmHhwoV+r3G5XJSVlTV6dVgGDoHf/9n8+fAnMiFg8llmyHG00HW46/7Q7S79dfTGVFDmK2BV1Q98X/YyVb4KrOSaMQxwSQ/L5TY+lQt5l3lU4w55pUSyHv8O1/62iyJhARvwSh+60BhGz1DraZP5AeyjmPf5Ea/V9PvBGHextXYZveA3L5lipyWZ9zTs+LH2P7J+ga4qgu8fgaJ6oVUgK9kbxW2ePqThEO1/kZYVe1tWuNSPBAfXI3PntMJYitakTcVLYWEhPp+PzMzGyaAyMzM5cOCA32see+wxkpOTD71ycnJaY6otx013wrOvmxFBdXTtDvf/FZ55LcwIJQtcc5OZf8UfQjPzsPzqquiO+Qtmv2crCyo/YL93E15cZHUrCHGFBE3y6LYDPG98xyxWs40DlGM9R40b/2UgYrEQMWMBFx52Y76PsfQnnvCtJxJJEeVsjmAbqwkp2TA5SGHAmASY9rYpXDr1aP54wSjcBrlLAjjGSzNZ5Or6hJDz2R21oeOwczUWa521Nbam25oth8TYPbPlfK4UbUK7cNgNh/vuu4/S0tJDr9zcIyCs9/Rz4dMfYOVOWLoV5q6Aa2+uz0obTYSAF9+Gex9qXBIgNhauuh7e/DQ8h2Ir+HxmtJUvCk/ZHQiXUcmq6hmYS7V545x0Tp1FMdCNVFCRU0V8/yUBRUgoUvDvoDiSMMtKBKEUMx9JDA6SiHwh2kCUMp8ecwFc9jhk9a0/5oyHsRfCnR9CSmj/uaiwY0Hwop7SgN0/g8/c/qshPAfVs+nHbxhFNvX5ZGxojCaLP3ICMR3A6gKgZYXIsRNtqg+CYTHLuaJD0Kaf9E6dOqHrOnl5jffi8/LyyMryny3S6XTidDpbY3qtj9VooOaiaXDDraYVZstG04G4d1+zsnQ02bcHXngKPn4HamogJhYu+JVpbWpOEciDB2DzUnNvpfdwyO4ZrRlHlVzPeg6vj5zaqZwTTl3KjzOOpqH3pUQiENTEGGSemovD7kOL0H+4L/4Tch1NH35iwyEfluYwk9UUyQoySWEfFiv/+qGaKOZ/6TUKrnu+1uohQ2fQbQncFrYFpWHWQNIddCOJFfi3MjckGSdn0I/Rwvzb3kXHzgAtuoyDHd+Auyz49lGdX4wzFdF9InJLpGVMRPutFK+IiDYVLw6Hg9GjRzNz5kymTJkCgGEYzJw5k1tuuaUtp/bLwGaDQUNbpu8dW2HKyVDRYD+/phrefR2++hQ+nmEm0AuHqgp48W7YcFgofY9BcPO/ILUZT9fF+TD/U9i+xkxONmQcHHsWxEUu6Eq8efhbyE44dTlZ3QqZPX0sB/NTkYBPh+IsDwe6exmbUxiWH8nh/MRGdsp8skjBh0EKCQymOzNZFRXhUscKthOPs7bmcWSkEmXBDLVRY35+g3tXwbK3oSQXpM90oB1yDgw4BfQoValOsPAZtMWAw7SOjaEL09kc9PcXi40/Mj5kmQWvNPjZKGShUUgxbpKwM0pL43gtg7hWsshIKS2VgxC2GPQxv8e3/D9QlVcrNIX5d3EkIzJHmcJGsyMyhiM6j0Dmr4r80xuTgtz5HXQZh4iJQhFaRZvTLkKlr776al588UWOOeYYnnrqKT744AM2btzYxBfGHx0yVPqXwMRRsHtn4PMjx8BH31nvz+uFP18ABwPUaopPgkc/h7gIIi1+/hZee6Bp8crYRLjzWeg5OPw+geVVX5Pn3R60jWFoXPPFFXgbfAv/n73zDo+juvrwe2e2r3q3ZNmWe8dgDDa9995L6CHUJKQR+EhIIyQkJIGQhBICoRMg9N6rMcYG995kq/e2fWfu98dIVtsyu1rJNtnXjx5Zu3dm7u7Ozj1zyu9ccvyXlBWmJhG9ZxnZXaP953Iw5cJkC4tkqV8Ln9wTve9Q4WQ45tahVSD14GmC564j6jsuFEMvZt4lOx/6SG7jVTYOHtvtmLuauUwQsRfcDXoHd2vr8ESo/LKjcIM6lWlKdgIvxDxS6siaRejb3zMaICoqonA2yrhjEdkV6B1VyLovQA8j8mcgCqYjhIL0NqBtehnqlrCzl5M1E0oXoFYch7D1N2xl22a0xXckOcteg0qMPxFlwslJ9dxKk1r26FLpc889l8bGRm699Vbq6uqYM2cOb775pinDJc1uSFMj/P7W2IYLwNdfwoa1MNmkWNj7T0c3XAA8HfDSvXD+jaanCsC2tfCvn0V+ztcJf74Wfv9KUkZRgWVMTONFIGjzjmZ8aRPzZlTisIVp7XTQ3O6iJL8TVRm6ybG7Gi0ARWQzmvzhPcjy52DZM7HHNG6AFc8NSWRuJ13t4BwHlSuM8JWqgNMKLpshiOjKh1mn99vkUDEORVN5SW5EqL3GR5vPxtYt4wmPzyJKGhMANdLLH7U1hKJ82gF0/qyt5fdib/JFakPuUuroKx5E1i+BHh+cHkY2LEOr/xqsGRDq9b7K7e+BxYmYdCZy438NDZy+VXChTqh8G237eyizLu+fG5M9HpyF4Gsi8TO7d7zc8irSlokYc3gSrzjN7sIu97wMlbTnZTdi3Wq48BSjuaMZ7vonnHymubE/PAq62mKPsdrh7wn2RPntxVC5JvaYc34ER52f2H6BsAzyYdejhGSASBfbMLDZWUHYEoxYnPJNvjFUEHyHY8mIIKaXEP4O2LoQfG3gyoFxB/Qq4FYvg3dvN7cfqxPO/ZcRMkwGTys8fxtsXxH5eVWBOYfCodeCM6ffU1JKzljWwRuNAYqyPditOu1eG61eByqQYxUsWZDLaIc6aDtvCJ4Qm/mcxpjLuQBOUEo5W01ttZVe9Sn6mkeT3DpewFGg7ncjIqc30VxvXIn+9d+6/xrC0mXLQj3kDoSSzoPZlezx7QHSfAMIheDyc4yqIrM4Eli4PNFl73vnkGDyp65DpYmOwV+YV5Hti0XYmOc6FSv973ZFtwu7OmMcYdWogOhpZdP3B8z3GtxTEICKwhkcMDTDxd8Jn/8T/vMdWPwQrH4JvngYnr0KVr5gvHFrXiOuqlsPIR944pWxR8DbDp89Bf+4NLrhAoZzYeMqsLoGPfVZW5hXG4NoCGrbM9jWlEWr16j404C2sOSuyt5SeV9Icu+nIQ79q599/+jj01CTCf0fWKq3JPrq4qJvfw/T7/Eg4p/c2tY3+/2tFM5C2ft6sA8IowkVLIPf26gEO6Bjm/nxaXY7dnnYKM03hHdfh/oEWwrMP8j82OFYxVtqMXX31pH8RT9bLeTQzIuoDq2jIbQNHY0ctRibrZzVYmnU7fZko6XnfnoUuexFBVU0UUsrCoIJjGI248gSCSw0AIEuaK+BoAc2fQSVn9Pvs+vJV9I1+OopsNiNXJdE7s4TVWDdsBBe+C2EQ/GPI3XoaIT1n8KMI/o99XiNH4ugX95TXzQJ/672c+eUDLxByaWPB1hdJ9G7c2IUq7nXGEqRSKFs24y+/X1k83oIDadIqITGlUipI/pUjgl7Vr9QlDFUg3Ac5fCBe9cCQ0qMT7NrSRsvaVLD4s+N6qVwAroVXo/58nBnppGDEgtbgnfyZrt75xQmtt8BWIWdcba9GGfrFRB7SX4Rc5s9NWQ0nyk4sVFOIUXCSBCdyRBCFYEuWPKYIbWfSNO+Zc8mdhxXHrgTSBxu2Ar//fXgJO9YCAU2LR5kvDQG9aiGSw/tYYkuJQ98Fu41XACkINBhxZYZinvOVIihV3bp295B3/Ds8Ev79x7RMEy6jRcpdbQlf06JZotwR5YUSLNnkA4bpUkNiS62QsBzT5off4QJCfjDzkpsDlkmSyYPOjWx/ZoglKA42Z7CXCYyV0zcabgMiaAX3rgVNn+UeLfhoMdohmhW62XOOYlZjIv/m9h8AJBG36MBjHYoWOIcutAq0HR46qtwr+HSTf3X5oyuo5TI2llmkW2bDcMFRshwAZwFCKVXGVpu/xDC5tWmIyIUKJiFcOzezSvTxCZtvKRJDfsfmJjXRVFg62bz44+5CDJiXGzc2XBSgk0sX/+3uXEHnpLYfk1QiInFfQ+KHQlgNPk4RYr0UgDWvgHtQ+iBU7qXuW0nHm78JMKGzxPzuuyc05RBD11S5ojpeVGBb4920uyBjgj9G+u/KsBT74x5upyklDF1iKXSWuU7Q9o+cQTKmP5eKr1m4RB3qYA1A3XarmnMmiZ1pI2XNKnhyONhVJn5XkxCQEYC5cfODPjZYzB68uDnRk+Cnz8Bjhj1pJH46l1z48yGlxJgAVNjD5ASeyiIIxTcI4wYiYnXlCirX2FIFSVj5sG+PX26Irg23IVwxE1w4DWJx+m0JFo3KCrMHtxXbJ8sK5eW2iM6Ly0CyhwK3xvrxB4lyK+HVNY+PZG6pYXo4f57KcbB9epkzlLHJD7fgTStNjnQzHspoDhOi4D8aYjyw/o/FmgzOYcI81AsiLKDUOffgnAOc4l+mmEnnfOSJjVYLPCv/xil0u1t8Rf8cBhOOj32mIHklcCtT8K21bDhK+OxyfvAuBlJTdlItDRBKACW1DQ17MEmLBwop/EZEaqduo2VQ7etI8vv473x02l1Z6YsEWYoiri9+zBqpnQkKgrHsjdjRQr7B2lBCCWWgNk7OQVyyiGvAvLHQ8kMWPcWNK43qlJGz4Epx0LGEHKZSibCjtXmvUJCgVNvAndOxKfvm5HJaIfK3ZU+OjXj0xHACQU27pmWQYFNARvsVSpYWSsHhY70oMr298vY8fEo7r5IMqtEpRA76sBu9Ukipb6zH1NMFCuicBbkTERWvg/+CMKAihWxzw2oeZOQ3tPQt7yKbF4HgW6JBdUOihW8jehrn0IU7QVdNciwz2QYUKDM/YFRTSQUyJmEcOaC1d0vBJVmzyZtvKRJHVOmw9tfwDOPwXNPRA8LqSrsuwDm7p/cccbNSN5g6UvxGGiJ01dGUQ2vzzCwQEzFLR18LFfiJ2QYJ1KSGfCxoHIj+X6j+eFxm1fx+uS9aHe4k69K7YMLB5Mp5Wtiq//GIo8MyikgnyymU45dpHhRaIygOGsKYVQNHXBVr7GXXwEHXp2yqQGw72mwfWX8cYoKkw+ABedGDBn1oArBLya6+UmFi8/bQgR0yexMyyBtl6sOsnLtM5GNCFXA7BKVo0bZU68eG2jHlMmbUYoy+yq0L34PgShVenoI4a2DvEkIVyHqzMsA0GoWIVc9DHoItACEupDVjcjqT4zthGok78ZDCETuJER+aj2BnmAtDZ1f0O7fhETHbR1FYeY8chxT0mq9u4C08ZImteTlw9U3GD8fvwc3XGl4YixW4y5V0+Cgw+HuB3d9Sc1ZN8Bv4sS+Zx04rFOYLcYxqcPL9qon8NrsZPl8OCIkpx62eQ2vzdiP8BB8Jrm4WcA0JlOKRahMlKN4naV4iJBIEYcOvBwl5iQ9l7gEPMltV7aXoZSbNy6l0xnE1INhzvGw7I2dRifQW4Uz/2w4+KKEK+BcquDI/Oh5Q4dPUrn1OCu3vdXrNRQCNB2mlwj+fvYwGC5gvqlh5hho3wIdW2MO0ze9jCg7aOdcpbcBufrfgIweJjVjuABIHdmyFlGQur5tjV1L2dHWv51JV3AHXc07KHDPpTznmLQBM8KkjZc0w8chR8Lna+GtV2HDGrA74OgTYGoKvCapoHwyHHAyLHwl8vOuTLjiN8M+DYEgIxQkIxTdLe/SQlzcVsZHuWG2Up9wg0WBYBKlTBflADTINpaxNSnDBSCExjPyUxYwdXh6E2Um2B6kfB7sd+nQQkGJIASc8AMYMwsWvwB1m4zHKvYxDJeKfYbt0OfPtXDkZJXnl4fZ0ixxWeGYaSrzxykow7WA2jIhYzR0VcUcppTMQ29aFb+UOtiOvvF51MlnIn3NaJtfTW1ulz/5TucD6QrWDDJc+tLkWUqmfSy5rhTnfKWJSdp4STO82O1wypmAyTYAI82lv4BRFfDGv8HbLbglFKOr9HduTzwJOAms7jEmXOKCrIxJnC5yANgia3meRaaPIZFMoxyPDPAqi9lBlCaFCbCDJnbwCSfJ/Zgiyoa8v37kjTVyVlpi38GTPRqOuBGyIpQBBzzQWmu0jcgbnXpPnxAw62jjR+qAGDFvYlGm4OqDRi5/QwiBMv549BX/jDYCMsoQeVOQLWswE9+U294ivONj0IZY+hwJe+qaUFY2vxx3TEPXl2njZYRJGy9p0hx7sfHjaYeAH3IKjFyFEUK1uHHl7Yu3eTGR8woEzpzZWGw5Ox8ZL0ZhlRZTejECmEwZuWTwOB/STByxP5PI7rm+yVIqZDE2keLLyYLvwJu/6K7sifC+zDjZCBEN/Ky87fDBv2Dlu71VQZkFcNjlMPvo1M6xB7N6MnswSsk8I4l204t9PCvd6d+uItR9vosQApFVgTQb4hkOwwVhhK9SgCdYR0CLr7DtDVan5HhpzJM2XtKk6cGdbfzsArLLTiEcaCLYtZneeiDjt9VZRk75YM/VcezDKyyOuV8BzGAsR7EXG6mhidTLuYfQWEcVsxmX2h0XTIATboOlT0DN8t7Hi6fD3AugMELZvK8D/v19aK2hn8HT2QSv/AFq1sFx303tPP+HUMafgCiei171MbKrBmFxIIrnIormILrbK4jC2YbnI2CiH9kwIbe/C5MTFK2MwI7WN0yOTOe7jDRp4yXN7k04aOQT6BoUjjPaBHwDURQbBROuxN++Bk/zYrRQK6o1G1fePJzZM3cuDH2ZIspQ5f68wzI89DaltKJSQQljKGQCJWR2N0BcK6tSUiY9aO4ImofBKAKMxNujbwFvK/hawZEN7hgaHZ//Z7Dh0pelL8OkBTBh3+GY7f8Ewl2MOuXs6M8rKupe16AtvoPUn21mkMiqT5CTzujXEylRAuE2vCFz/doc1hHKtUqzk7Txkmb3RNdg4dPwxX+NDsIAqgVmHglHXQ2O4Slf3pUIoeLMmYUzZ5bpbSaKUiZSSrPsoIlOcnBT3J0XMxAfgWFZSiRgHe5LiSvX+ImFrsHSV4m7YL79d7jm4cjPdTRB1SojeXT0DMhOoXbN/xAiZzyi/DDkjg92zQTCPmSgE9myBr32Cwh5Ee5ilLKDIXeSqcogbyiOjEIfRmUdPJTZpkmCtPGSZvdDSnjtz7Di7f6Pa2FY/jZUrjAWlpp1xoJVPAH2Ox3KZ+368utdRL7IIp/YTS5zyaCOtp25KqlCIplMaUr3mRQBr9EPKR4tVUZ4ydnn/fJ74I27YO1HfapeBExeACf+CFwmG4im2YlScRxa9afdfamSOOdU5xByYgT6kjvBW09P+FV2bker/QJRegDKjIvjemUUzOW92dU8cpyTkpxnmmT55meZpdnzqF472HDZiYS2Wlj1rrEItdXC+k/hsR/Bs7dGbHyXxmA241JuuAgEFRRTFMXbM6LYHJjOPfD2ycfQQvDkT2HtxwPKdSVsXASP/whCyZWUf9OR3kb0hq/RG1ciw/3fI+HIRdn7uiSS3wVkjUUZfwJJ5ZIIBawu8DX2zLL7l1G6LWsWIivjtwZx28sRJgyYSYXpPkm7grTnJc3ux7I3jAteoo3vNi6Cd+6F478/PPNKAl3z4W1Ziq91Gbrux+oYhTt/PraM8SMualVGPjMZwyq2p2yfYyjkZOL0qBkpVCsUVUCDCeVgV07v/9d8BLXrI4+TOjRugxXvwNyTUzHLbwTS14y25jFoXtP7oGpDjDkKZeIpO70aSv505JRzkWufSGTvKBNOQbiKYGOi3buF8ROKLXKob3sbMeZIRAzDyqI4KMzYh4auJUTzHBW498ZmSXvldgVpz0ua3Y+2uuQ69oJh+Hh3XZVDX8KBRurX3kl79UsEvZWE/fX42lbQtPk+2qpeQI5ww0UhBMeyDwcyLWX7PJ652FLdGmAoHGWiDcDkA/onfq94K06ps4Dlbw55at8UZKAdbfHvoXld/ye0IHLr6+irH+33sLA4zO9ctaPMugKlcBbCXYwonkt874vo/fzsWYjSBcRd2oId3SGl2JRmH0GOoyckJPr9znJMYHTOMJXep4lL2vOSZvfDlU3S7QN1DbZ9DdMPS/GkEkNKnaYtD6GHuwY8Y7iuvc2fY3OOwl2wYETnJYRgAVNpkx5Wp8ADs9vd/VTsDbOPiR52tNgNvZe+dDbHabAojTFpAMNrQbCTnnN5ILJmIXLMkYgsQ81ZZI83t2OLC2XBz1CcvYrNouxgZNvmyN2kFRuU7Idw5BjVeBmliIJZyK1vIM1cPkxo0ShCpSL/TLqCO2jxrCCodWBVM8l3zSLDPjbdEmAXsttde9KkYeaRDKnEUjPZLRqgegfcdzccvT9MHwUT82H2GPjVTVATWwo9FoHODWiBJqJd4AE6Gz4cce9LDwcznQwSuCOOwqes3WWvISon/RiOuBKsA17fqMlwyV1QOLb/41mF8T0vmcPQAmEPREqJrP4strEnFPSahb1/ugqhYGZ8IT/Nj6x8b+dxtPXPon91V3S9GD0INQuROz5AFO+DUjTHCAPljI/f7Vt1gMtcCwohBJn2MYzNO4lJhRcwLu9kMh3j0obLLibteUmz+zFxPyieCPWbktu+JIJ4GRhNIT94GxZ/Bu3tsHYlrF4xeJynCx59AJ5/Cp56FaabL13uIdC5kXjeIy3Ygh5qR+2jnDtSZAgnF8rDeJ8VbKQm6f2sYBszGEMZMbRXRhohYME5MO802L7CqELKGw3FUTwAex0LW7+KsUMJe58wHDPd85BhCMep6JJyUG8hdcalaAt/ETsXRerI6s+Qk89Er/4MWflOzxMxDqZD2Ie24p+o839mKPzmTTUME28jkW8eBKL8UIQavQFmmt2ftOclze6HosK37uxfymqW8lmD76wBNq6DI+bCVRfCIw/Ac09ENlz60tUJ114Cepy7uAgY3oj4HolUV/8kQqZwcqrYn8s5knIKknJ2CQQriNN/KIU0dErW1uk0dpmYrMUG4/eFaYdEN1wAph4Co6dH9gwIxSjFn3lk8pP+JiEshtci5hgB9v7fXWHPgoJZxM1f0fxoX/8Nue5p83OSOnTugI5t3YdXUOdcA1Yn/Ze47mPnTqKx8CTeWhvm861hgtpu5jlMY4q05yXN7sPWrfDEE1BfD2VlcOJP4I3bIBTE1MrqzIZTfjL48dYWuOAUaO++GwzH7we0kx3b4LMP4eAjzG8DKBaniVECRR2+xo/hQAtasBXF4sTiGBXVzZ3hbePIuo20+rewtnAUW/KKCKvmLg0SSQsD83ris6Q9xL07fHzaGkIVghMKbVxd7mSiK3L1x6oanT9/EOLzbYYhKYADxiv86HAr00qGeA+mWuC838Fbf4NV7/fmQgjFMHyO+57R3DGN4dkoOwC548PooRmpo5QeMHhbexZSiPjdo/tWMJmfGbJ9GyK7wvgroxT1gF+i7/gQWbPI8Ba5itjkPpbvfzKD7W/r9HhlLEqI8+eq3HS0dfi6cqdJOWnjJc2uR9PgBz+Av/0NFMX40XW4RcItN8A0O2z5sne8Mwt8new0aFSr0XDvkEshI4IK6zOPGQZMvDh4JBQFln+VsPFizo0h0YLNKM5Ric8rBiFfHW3VLxHs6g27WewFZI06YZB6r79jHc1bDLVZFzpza7Yxp7aSusxcAo5clpeU4RexjT0Hibnf76708uP1HiwCwt1v09+2+/jHdh/PzsnixML+hsJXOzQueyJIuM/HJ4FFW3XO3x7g0W/ZmV02RAPG7oJTbjRyZapWG0com5bOdYmAMu5YtNrFhkEw6DslEMX7ILLHDd6uZB7atmj6TUNFwoAWGsKejTrxVJh4KgAbGnTOejDAQEdLWIfHvtSo7ZDcc1baSN1TSBsvaXY9v/iFYbhIaRgyWp8qgN/8Gf70J/jeU4Z0uysbckcZSbn+LrC54t8Vv/pCcoYLGHOyJhMbVzBXMZXaO72Qr47Gjfcg9f5Jy+FAEy3bHiV3zHm48uYCIPUQLZVPMjAvQJWSso4W6GjD5y5iZZYSM7w1jdGm5/dpa5AfrzfyHsJ9dqlJYxbnLu9gw0F5lDoMD4yUkp+9FiKsgz5gCpo0nCS/eD3IC1cOPfkYMIzfqQelZl/fUIQjF3W/n6Ktegja+4QMhYoYfShi/PHoLesQKJA1FmExvp8iaywU7gWNK0h9zyOByJ8ec8QN/x1suPTl3fU6a+o0ppeMXEf5NMmTNl7S7Fo6OgzjJJYr+be/heuug7I+d8GqFdxxet304OlMfn5SwmFHJbyZPWMCnXEu0IrqxuJIbUO39ppXkHp0leG2qhdw5sxGKFZ87auQMeXXdSqqlrFh2n4ERHiQASMQ5JLBZMpMz++vlT5UIFKRqgRCOjxY5efWiUY4bXmNZGtz9PdRl7CuwciDGXL4KI1phLsYy/43IzurkJ07QLFCdgVyyxvoH99sJPYCqHZE+eGGcJ1iQZ39bfTVjyLrvox9gMRmY3S2dkZPGm/olGxtib+nf3wS5m9nD7/xIqVknezgPb2ObdKDDYV9lTwOV0rIFb03SyHNg6b7sapuVCVFBvo3hPS3PU1iaBosXQoffwy15jquxuStt8AfR3q9pQU+/TT5Y0yeBmqSF6QDDoUpse/oImFzj8PiLCXWV0zXvHTUvjPIS5IsWrCNQOeGmGOkHsDXvgqAsL8e4sifu0I+TmxSyNKNC6ro/gcwilzO5SAswvx7+05zMKLh0oMOfNjS+35UtpjzmP3tk1A68XIXIDJHo5QugLzp6Ev+hKz+uNdwAdACyG1voS2/Hyl1hGpHnX0lytTzUnDw7u9W7iSUGRfHHLq12dx5tL11+M8hKSX/0Su5Q1vDV7KFJgLU4OMVvZqbw1+zSe/EE6hmY+OTrKy9mzX197O85i9sbX6RQLg1/gH+R0h7XtKY58EH4Ve/gqpu/RNFgVNOgbvvhjFjkttnl8lkz46O5PYPcOHl8M7riW83aQr89V9JHVIIQf64S2jadC9aqC3KKElXw3v4O9ZQOOlalHhVHHEIRz1Of7SgcQEUio1YOjQ92Go+4PgaqM8uoqt4LnZnGWMpYpQw6fnqZmFrkK4EhZOz7ObCau9v0Pnus0H+cY4NVRnZpEspJVVtEk8QSrMFWQ7j+J6gZOEWjWXVOjlOwV5lCvuOUb5xSaHS24T+xW9jlEFLaFyObFqNKOzOuXIOweOo2CC7AuHIRYzaH5E/LW6TRbfN3HueMQIpL4tkE2/qxo1f32+fBALo3BlezQmtX1Mc7sDa59lW31o6AluYUngpDmve8E90NydtvKQxx+9/Dzff3P8xXYdXX4UvvoAlS6A0ic7C5eXmxnUOIfRz0OFwzrfgmceNMs5YISpFhYoJ8J3vwmnngiX5r4jFnkd2+Rm0bHko5riwv5b26lfIHXN20scCkNLcBbon/OPInkFH7RumthFASXsDtL9B5qjjySqekvD8zloW3wAVwOH5vZfsBRUKGXboCsTf/8ebdd7boHPMVPOeoDaf5LMtGv4QTCpUmFUqEhIfe2+9xl8/DrGhwXhPLQocO02QbRf8Z5mONsA2HJUluONUK/PGfDPyKqTU0b66O24vIVDQqz9F6TZeRO5kUG2gJdhINWcS6uwrEI7EFu9pJQKnFXxxnJyX7Df8S+KbWk3UbDgJ+IXk+fw5WPUws3w1zO/aigUdkGh6gKq2t5lYmALP1R5O2nhJE5/aWvjZzyI/Fw5DYyP8+tdw332J79ttslT4ssvgzTfhxhthzpzEjiEE/PYumD4bHvw7VFUaj5eVw+HHGMZKRiYcfizkp7a6xN++GjOJu97WpWSXnohicQ3hWHF0a3Zi3KWq1lxTcxtIZ+0buHJmYbGbv3v+tDVIYyj+cRQBV5T1eqAcVsG1B1n4w3vxy9sVAc98HTZlvIR1yZ/eD/HEEo1QH2/QlCLB70+xMbU4fkT9v8vC/Oy1UL+U67AOr62OrvFT2yG54okgT1xiZ1bpnh+1l02rTPUIAr1Pl2cQFjti7DHILa/G3EpMPhthdYPUEDkTEBlJ3CABqiK46kALd30Y/TwqyiAhwzcZglKnkjgif92EFAtfu8pptGRwatsKFIzzqiOwhWC443++IWTaeEkTn0cfje2tCIfhkUeM8JE9Qb+r1WRTP12HZ5+F556DV16BY4+FzZvh7bchGIR582DBAsNQiYSiwEXfNkJIDXXG6ykeZTw+jOhhL6aMA6kR9FXjyJwUf2ykzaWOt2WJ2UkB4GtbZm5uEfA0f0F26Ummx7/WYO4O+4Bs685Kox4u3d+CPwx//Si2AaNLqDSZ23DrayFeXKENevUbGyUXPRrguSvsjM2Lfm50+iW/ecu4jU/0HQzrcPdHIR48f88vy5VNq4zcEzPVfLb+i60y4ST0UJehGTPQiHbko8y6HCU3ue9DJL5zgIXGLskTSwbHLkuz4b9X2Hc7yX8pBDvseWx0FDLF37Dz8UC4JW287OoJ/C/SENJoCesUWRXyLIlZ+lJKGroFL4osysh82bZti24U9OD3Q1OTIS6XCLNnQ3a2Idcfj3DYmMcppxjhps2bjb+FMIybmTPhmWdgWoyuyYoCJcndvSWDxZaHWe+GGELZdKBzPVIzd0dncxsKxCFvFYYXJvEy8lCfC6kZrCZtxFmZg78PQgiuOcjKm2vDbIhz2DY/BMMSmyX6e/llpcYLKyIn3+jSCC3c/1mY20+OXiL/2mqNYAJah32RwMItOs0eSb57ZBfLdp9kyXYdTcKMEkFZzhCN9wS6vyul/ZuQCqGgTrsAOeZw9JrPkf5WgqqOr2AMSkY52Y5xQ5vbAIQQ/OxYG99eoHPvp2E2NkoybHDpfJUDKkZmKbQJhXG4qcRj2ugVUmeVs7Sf8aIqe77hO1TSxssIstQT4I7adj7rMu5CFeCYbAc3j8pmoiO2B0KXkkebPdzX0MmOoHHBKLepXF2YycUFbprDOn5dZ5RVxZJqb0JOTn/tlWhkZw9+rLkZHnrICPmEw3DAAXDVVTBunPH8G28kls8ipeFp2by59+8er9DatXDQQbBiReJG1DDhyt+PrsaP4o4TworVZV4vZSDe1mWYMpKEFXuPdyeBKqGBSC1IZ8OHCKFiz5yC1VEUc/yCHCsQqyzb4Lox0ZWJz9zLyu/eiZ204AkY5a43HD74+xQIS377VojnlsU+lzUJr67W+OUJEpsa2bjY3ipRFfoJ5yWCBFq9I2e8BMOSP7wX4tmvNYI9AsLAIRMVfn2CjaLM5OYhsschqz+JPzCjDFG8T+R9uEfRVjqZqra30aQfgk3QshSBlTz3DKTU0GUQu5qPzZJFu28DAa0dq+Iizz2bPNd0FGHSgwuUZCn86oRd19foOLWU+7SNpsdLodCm9n4vrGoWTmvJcExtj2LPD7ruIXza6eeMjY0s6up1n+vAO+1+TtzQwPoYmWRSSn6yo5VbqtqoCvZeeKuCGrdUt1GxvJo5q2uZv7ae8StquGRLE56BmYJDYbTJRfWyy/pXD332GYwfDz/9Kbz/vlFe/cc/wsSJ8Pjj0NYG552XVO+giGia4cG5++7U7C8FWB1FZBQeGmeUwFUwf0jVRlLzY8a7k1Vy3E5vnSN7Gsl4XQCCns101LxOe/UrNKz7I81bHkIPRzdOjimwMSpOxce+WRYmu6PfT52xl0pBnBQpCTy5NEww3P+9kFLyg+eD/Hf54FBRJEIaHPZXP/d8FKLdN3iLTEd8lftYKAIKMkbGcJFScsPzQZ5c2mu4gPFefbpZ58JHA7R6Q7R619BU9wZtde8SDraZ2rcomdfd6yjGa3GXoO77I4QS2cBo9a6lsvVlw3DZOTOQhGj2LKPFu4o233rquxayo+1NOgJbCISb6QpWsb31NdbVP0zYpNdxd2B/kc9eiVTqSYmzj5xCadahu114a1eQNl6GkbCUSCnRpeSH21vRGSzOpQE+XXJLVfT6/Y86AzzdYnw5+14ve/7f13utAe92+Jm/pg5fqgyYL00KSj3/PBxzjOEZaWyEI480Spz7XuV7FHQvuQS+8534Gi+Jomnw73+ndp9DJKv0RDJLjiPa182eOZnsUUPrWqza86Puvweh2MksPqT3uBkTsThGxd0uOr2Jqf6OtTRuuhcZJfdBFYIX9snGHeVQ5Q6Fd+ZG8Nz1IcMuOHlmfG9RZwC2tfS3LJZV63ywUR+k0huLVi/c91mYcx4O0Ozpv+Fx09SYaq2xEMCRkxVynCOzAC3aZrz2SMaWJqGmXef+19/Fsegecla8QMaKZ9A/+imdy/+EHoxtFAiLA2XO1YYXL1K5cu4U1AW/QNgyIm4vpaS6/b04ryDaG9197oWb2dbycpx9DB9S6rT61lHd9j7V7R/SGajsbswaGSEE9TK+F7Iv0/x1CFRGZx9FvjvxLvffRNLGS4ppCWv8vradWStrGLu8mikra7hiazPVIS3qPa4GfO4Jsi0wOIjeEda5aUfiwkQtms6Pk9huED6fkUdiBl2Hzz+Hp5+GCy+EQIz6ViGM5NvhoHX3EnISQpBVciSjZv2azJJjsTpHY7EX4ciaTl7FZeSPvxyhDC2C687fn9heFEFmcf/+TEIoFIy/Aou9YOeY/r8TI+yvpXHDPeha5M99bpaVZQfm8d0xTnIshjxemV3h95NcrD4wjwwTiTFZDkGUSE4/Bt6YvrRCQ03iaqdLqG6T/Pat/gnHFfkKJ89U4qaCRcJphRsOMx/mGCovrNBivme6hLe3zsYS7r21UqTEUb+ewOJfI8Ox69SV/OmoC36GKD0ALE6j83RmOcr0i1Hn3oBQohucnmA1QW0IGk5ATwWOP9Q8xP0kjidYw6rav7G1+XkauhZT37mIjY1PsK7hIYLhyK9ri95FHeZu2hQJ2VJylG0as0q/R1Hmfqmc/h5NOuclhTSENE7d2EB1UNvpYfHoknc7zJ2o2wJhxtl7P5J/1Hdwe21H0l1AXm9PzLqPyAsvGAaMWRTF6FMUz1tjJocmWcxqx4wwimonq+QoskrMtxuQUsPfvgZ/53qQOjZXOc7cvQeFmKyOYjKKDqer4YMIexFYnaNwFwzu2aPasima+kP87avxta1E6kEsjmLDGJI6XY2fEg61YrHm4G39CqnHXshCvipatj1OwYQrIj4/zqny56kZ/Hlq5DvxeMyvULk7TtVRvgsq8vuv1s1emXR0UpPw1jqdpi7ZL9Rz24k2FBHi5ZUaQhihoLAOGTaYVSpYtG1wR6gZowS3n2RjfMHI3TfWd8o4XiJBcyhzkMkqAIu3BW37O1jGx64sExmlqDMuhjhKtwMJ66kL93QGKnFYo7cISDWBcDsbG59Elz1VZ70nmC/UwMamJ5hWfCWK6L/MvqJXmT7GWOHmOusUChKt4vwfIG28JMkmf4itgTDZqsJctw1VCG6pau1nuPRg9pqZ0ef26NVWL7+tHdodSVBCSEqsQ4mPbtliCLWFTZZW6DpsiC1RP6woipEQHA1dh65OcDjBtuuS9qKha36Cnq1IXUNYnLRtfwYt2NuUxdvyJe01r5JXcQmOzMn9ts0adTwWWy4dde+i973rEyo293ik7jeEwQYghIozZzbOnNmDnsspP33n/z3NX5h6DYHOdQS9O7C5Um9E7lUqmDlKsLZeDhKA6+GS/S1YBqjsjsoSKApRt4mHLmFjo05BRq8XwWYxNGGuP0TnnXUaXQHDaDp6qordImjqkny2JczmJklJlmBBhUpFfmqNlkbpp0p6saEwSWQRCgpeXa2xqlbHqsLBE1QK3RJVENOAsYowIV3Bqgx+g7QdH/YzXqSUtPnW0di1BG+oHiFUcpxTKMqYh9OamHKuTY0dKtydaez6sttwiSw3Fwi30upd2y/ME5Q6y6Q5z/BJopSzLGNTM9lvIGnjJUFWeYP8X1UbS729buQii8L5eS7eaPcn7SWxCXiksYtcVWGCw8pvatpSMt8hf8C5uYl5SYQwjAJFSU0irs1m5NCYQVVh6lS49trBz3W0wwN/hScfhvY2Y36HHwvf/ynMGLxojzRSanTUvIGnaSFSxq6okXqQ5s0PUjT1R1gdxTsfF0LgyJ5FZ8NH9Ks8kmE8TZ/ha1tG4aTr+oSJetHDXkKBBqPqyTkqoty6asvpZ0hFR8HXtnJYjBchBPecZeeSxwNsb5U7X2XP4nzyTIXL5w8+60+fbeGxL4fm7bNF+TKNzlG4bP7g96sgQ3Dq7OEJDzVKP49oW1gleyUGbLrKjs+L2PZZEapiFN4/tVSjJDO24QLg0e38aPPl3DXxXyiid7AAlEAHMuxDWJxIKdne+jrN3uXsPMckNHuW0+JZwfj8s8h2TjT9OpzWIhyWQvzhxviD45BhT75aLxlavKuJnSQvaPWt6W+8YC5hHGC+klrBzG8a6ZyXBFjrC3Hapka+9vZfTBvCOnc3dA2pyXtQwvNtPg5ZV88p6+upCg194S+3qkPPSj/zzMS3mTdvaMfsobDQvOFitcJFF8Enn0BmZv/n2lrhzKPh3rsMwwUMw+q9N+CUw+DXN6eu4ikJpJS0VD5JV+PHcQ2XPlvRVvXizr+0YBtdDR/TtOkf3QbGwLNRooe9tGx7st+jWriLlsqnqF39a5o2/p3GDXdRt/q3dDV+Nijp0J0/H7P5MEb10/BQkiV48Uo7vz7ByryxClOKBEdPVfjXBTbuOCVyb6NpJQpnzkm+NDzDBrNG7R6Xy1YZ5DfhVazpY7gABBWNogNqGXtENZreW8bd2IWJPCGFD9pm8X7r4GRQAWjvf5/wF7+jY8dL3YYLDCwfkOhsaX6esG4+zCyEINsxVCE6hQzbGJzW2OX6qUaP+1015Pz7YpEC1eR36BM5dIPum0za85IAt9e0E9RlksWl5lkarwGHSW4uTYFLNi/PUM01UxVkscCkSXDbbfB6Eo0QBzJrFnz0UXzPz7PPwhFHGHONxK9vgi2bom//yP2w9Au44lo46nhwmWxZkCKCnm3428xK+/fZrmszuh6mo/oVPM2fE79UWifk20HQW43NVYYe9tK48e9ogRb6Bjf1cAft1S+ihdrJLu2tgnIXHIC39WvC/njdxGVCrQPMUuXXeLDKz4ctQQSCw/Os/O4sB6UOlS1eje1+jZVdklkZkY32Xx1vJaTByyuT88C0+aAoM/644eY1vZouQhGvQ0JAyb5N1H9dgL/VyIsyWxWloPNs4wHsl7WRl5r2493WvfDqdqa5dnBu0WfMYBvu9q0UFOfRlB9Z3VUSptmzguLM/U2/Hl/YTHuB6NjULMblnzro8Q7/Vhq6luANVgMKOc6JFCYR2oqG3ZKLLxRr7goOq+E9qZQeXtaqWCrNeC4NvtSbOV8dN7RJfoNJGy8maQhpfNCZfFhopPl+UQan5ibfJ2cn77xjvpz5qKOMNgE5ObDXXrB8edxN+iGEEc6R0tCDMWO4AJxxRnSZ//Z2eNlEVdOqZfCD7xiGy09uhYuvTGjqQ8GQ9U9G6VbSXvUi3hZzuSg9hHxV2FxldDV+jBZoJprR09XwAa68eVgdxsVeUe0UTrqGxo3/IOyvi34AoeDMiyxI1hddlyz6WLJ6BTgccPRJUBKl388rDQHOW96BJnvlBha2hfj9Vi9T3Cqr+rSrnuJW+d0kNycX9U9yVBXBHafYGJsb5J6PNRSB6dJpTxB++EKQxy/etYmTupR8ojfEPFOkDgUzW6j6JDElaR2FTb4STl55C61hN0ZQTrDRO4oXmhZwVembXF/2BqPqW+jIdBK0RQqJCTzBmoSOq+nmri8ClWznFIJaG6FwBxbVRb5rNvnuvQYpzta0f0hd50L6hk+bPCto8iynIv90cp1TE5pjJAoz5rK9NdZNmk6Be2/W6u38SVuLnuDqERz22+Q9m7TxYpKGkPlY5e7ACTkpMFwA6k3eFf3970auia7D6acbKreJ4HTCCScY0v5XXGEo8G7dGj9ZuKQkdn+il55JTE3M64Ff/RTefAW+fR0cepSRSzOMaKE2khWLS9RwAUCoSCnxNC0iXsy+o+5NMvIXYHWPQVFsKKqTwonX0LDhbrRg64DtjYUiZ/TpqJbY3quli3T+clv/avqXn4UJk3V+8Udwuno/042eMOcu7yAs+x9NxzA++houABs8Gmcu6+CxWZmcO2qw8N+1B9s4YLzOk0tCLN0hqWmPf35IYOkOnTV1OtNLdl34yIdGwMS5Ys9KxnsraQllIhHIPhkFGsb5f3/NcUx01nJs3jLyWjupK47s6RQJZiM4LAV4gtVxx5VmH0px5vy449p9m7oNFxh8xsC25hdxj7oOmzo0N1q+axat3rV0BrYR6XtUlLEfDmsJ94WXojG48iwWAigXiV/D1wR83N/RQrOuUahauDIzj+n25MUvd2d2jyDuHkCeZeTfqqEcsTLZxisDKTV599aT5/Laa/Dyy4kZDIoC++4L//ynkSB8wQVGz6OtW2MbLqoau7IIjHBQMnzxKVx5PhyzP1RuTW4fJlGtWST3aSeTzySM5o8yjK554oyV+NtW0LT5fupW/ZqO2jeRUkOxuCia/H3cBQsQfWTZrc4y8iou69acic7aVTq//3lkGaDNG+DGa+iXb3PvDj865hsg9kjnfXdtF4EorpU5ZQp/ONXOe9c7KDRZta0IWFw5jCX+MVhcqfGD5wOcfm8QPRz7c5cSQp7k7kvDWHYaKwNR0Hm49kgE4PBHM44kWY6KhI5Z4J4Td4xAJd9lLrG+oWsxsb4bEkmzZ5m5ycWak1CZUHA2JZkHoiq98v02NYcxOcdTln0kK2Qr7YQSvvGVwFGK+RYAIV3nkvrtXNRYxacBL2tDAT72e7iocQdXNOwgNBQ56N2UtOfFJKU2C/PdNhZ7Rs6Zd0aOk42BMMuTyIHJSUaRKxL7mRBFEgImTDD+/8ADhlGRSIWSrsNJJ8GUKUZzx769iqKhqlBRAd//fux9D1UfYUclXHgKvL1o2HJhXLlzzXeE7keiFySBM3cfVGu2YRwIC0hzRq7UA3TWv0c40Ezu2AtQLC5yRp9OVumJaMF2FNWGajWXY3X/n2M/X1cDixdK9j/QWIBebwwkpWbbGpa81hjkjOLY58AZe1n458KwqRDSSK8BUkrufD/MQ4vCqApousCyJpfCmS0RBW0BFBWaVkfJ/4qBwyII6dHLyXUU1njH4NFs6BGSokFgUVzkuiI3RpVS0hnYhj/chCpsZDkmYlXduO1lFLr3pdET7TsgmFBwNhZ1sCfCE6imoWsJXcEdCATZjol0BaqI/d2QdAV2xHjePIqwUJp9CKOyDuwW21OwqVk7c66qdG9SAeGDRCH7CPOf4XVN1awKRdZgWhb086OmGv5auHv0e0sVac9LAtxcmo3CyL1pb3X4aAlrzHFY+HaBm7NyHKbutXNVhf0zUhSb/+yz+GOkhA+6xdE2b07McFEUI1fmr3+Flpb+q0Nfr4urz4VLVeGss4y55cbpETJjL/NziYSmQW21ubyZJLFlTMCeOZVklW3jY+zXnjmJnPIzjEeEwJW7N4mezb62ZQQ923b+rSg2rI5C04ZLMKBTbWLdePHp3v8nK8OvAHdu9XLOsnZu2tDFek9kQ+2ieRbyTdiluoR9yoc3hDiQ11ZrPLTImHePUVGzqBgtqBCpE4OU0LQ6F29j9AaXkXBY4bBJ5s4FiaAjy8XA89WiOJlUeH7EJoldge2srvs7m5qeoqrtHSpbX2Nl7T3saH0bKXVG5xxNec6xqINCJYIc51Rsas6gfdZ3fsH6xkdo9a0lpHUQ1Npp9HyFxIxBntqruBAqdksudkt2v2RxG2pCtxhOFC5SKrhcnWC6UrQ2HGJpMHbe0GcBL01airzxuwlp4yUB9nXbeWJCAWW2/hcwm2BYjJpOHXaEdJb7wzzY5OHjrqCpL8L1RRlYUtW4K5bEf6RxBQWxc1D6oqpw8slGQ8fq6uhGjxAwapRRtn3EEXD99UZFU5GJ0sjTzzUE6YaCEPDGS0PbR8zdC/IrLsaVN4+BZ5FqH1r3WEV148rdh/wJ3yF//LdRlF6Ruoyiw7vbEiRyrih4W0z2uopAq8lii84+VcAH5VqxJHE668CXHWFeaAhy1zYfMz9r5WcbuwaVgOe7BU9dYqcoRvhIFTCjRDC7dGQb4j20KMxAJ0egzc6aJyfhbeqfy6BrgrqlBWx5Y0zCxynNgv3HKjFF/AQ6FY563O4sSid+j8KMfXHbRpNpr2B0zjHMKLk6YrmyN1jHxsanCGoDu8frNHqWsL31TYQQ5LlmYrMMzEMxBPHWNfwLT6A3L6YzsL1PTyS933gzmAlt6VKjw7+VVu9avMG6mP2KorGPkht3Ri5UzhFjuF2dw73W/TlSLUFJ4Pr9VGebqXEve9rjD9qDSIeNEuSgTAcLp5WwsCtAZTBMlqpwRKaDprDOv5u6eLzJgzfFvuWevTWH4zsfK2wqVxdHLmNMijlzzI3be2/jd4/Wihl0HV56CbZvj52YK6Xh0amsNP7/0UdG5+if/hR+97vBjWz6kpUNf3kArr/UOF4yn42U4OmKP24ICMVK7pizyRp1PIGuTSDDWJyltFY+HX/jiCgI1UHh5OsjitIBWB2FFEy8mpZtj3drw/QRtouKTtiUUF1kskxW72fl9P7/2jFOHq81aURHoccsvmOrj3KHylXl/Q3ashyFt69zcPFjAVbUyH7vhAAKM+DuM20j2s3XH5KsrY/8efianKz69xTcJT5chT70sEL71kzC/uQu6VubwRuCLAd0BSJXYUkEF435Csu8H2K151Nu0rCu6fikWzo/8mtp9i6jOHM+DV1fRCk9lugyxJbm/zJz1PUIodDY+SXJVegZuG2xBe0au76ipuMjtD6aNQ5LIWNzT8BtNx9+KRQO5ot8vpDNUb9ZF6jjOEhJXqOmXZrzdLfvQi2r4SDteUkCRQgOynRwYX4GJ+e4cKsKY+0WflGWQ5GJ5nLJEusUFRiN7u4am3isOyYTJxrdoWNV3IwZYyjbgtGQcZJJ0akeQ+Lrr82JxIXDvV2pAe64A+65J/52x5wIz70Nx59qGEkAzm63t5nFSLXAlBnxx6UA1ZqBK3cOrrx9CftqTWiqdG9n6/u5CxzZMyia/L2ohksPNlc5RVNvJGvUCdgzJxrdgWMikJqf1h3P0Vb9MoGurQndkTpdCqUmhFDPvKD3//OyrfxhshHXMdOUMR6/3+JFizBnu0Xw+MV2fn+Klb1HCwozYFKh4EdHWHjxSgdlOSN7uYz/rgo8dS4aV+bTvDY3acOlh2e/1vj72TZslv7vsyqM7+Zpkzs458xTEU7z/YM03U+HfyPxXk2zdwVNcZJoQ3oX7X5Dr6kzUEmyhgsItrW8GLUDen3nYna0vdnPcAHwh5vY0Pg43qC572QPl6sT2FsY4W2luxZLdP+crYwZkuECMM1mLkVglu2bVXUkZDK+sN2Ijo4OsrOzaW9vJysrhR4HEzQEw/ykqo0PO/yEMSxBmwD/ML+jxRaFDk3i6/PRldtU7izP5aDMYThBKysND0xbW/Qx3/8+3HWX8f877oCbbjK/fyGSz4QsLoaqql6jJB5SGsaPxQJLFsFtt8DKr+Nv99IHMHOI+TMJ0rDhHkLeHcRfxhRKZtyClDpS86JYs+OWKvcQDjTSvOURwoF6jDO4p1YnHj0LuY7NXUF+xaUoFnOlnatX6PzyR9GfHz0W/vLgYEPhF5u6+N0W36DZ9dx/m/Eb9fD1glxmZu7+jufTH/SzoUGa1qMZCoqA1f/npLpN54klGm+uDRMIw+QihQv3tXDkZCVhz1Mg3M7qur/HHZdpraAzFL+qrzjzAMqyD2N5zV8GGReJMj7/THKcU/o9pul+VtTcjYx6qyjItI9hUuGFCR9vm+xikd6ER4YpFA4OVgrJFUPPTfRLnUOqN8e8uXUg+KRsQkLhqJFgKOt32vOSJNv8YfZfW8e73YYLGBfQ4TZcAOrDOidkO7h/XB5/LM/lmQkFLJxWMjyGC8Do0fHzWO6+G2q6xalefNGcR6OHodjP9fXw1VfmxwvRa+jsOx9efA/eWGj8f9DY7td83Y9G1HAJerbTtPlfhLzbMbMc29zjCHq2oagOrM5S04aLobB7L+FAjwx5IgXJOj13vkHPNpq3PmLaAzNjtsJPfw2R5CemzIA//GPw45+3hSIaLj3kWwVT3eYvZ/6RsAZSwOXzLSNiuAA4u/Nsy3IUbjzKyvvfdfLZD5w8fKGdo6Yk12rEqpgzaP1as6lxobCHrsAO7GoOQ0twV+jwDzaWWn3rYhguAJLOQCXBvo1PTTJOZHCeOo4L1QqOVkrIYWiNYfUwbP8Utr+mcH19ccyxv8gr3u0Ml6Gy+9967Kact7mB4C68/r3Q5uPOMbnYzCbHDoWFC41KoHj8/Ofwr39BQ8PI1pT6hnYHxuSp8MTLRpuAh+8zqot6Hr/6BjjlrCFP0Sz+jvU0b/kXiZRBBz1baPFsQQgrGcWHk1l8ZMTGigPxNH+BHu5K6FiRkQQ9Wwh6K7G7xxHyN9LV+DG+1mVIGcJiLyCj4EBcefO6E4Rh3wUKj74kWfSJZO0qcDnhmJMhvzDyvP+01YsqIBxhqjrQHJKMcZj7LtgVmOwe2aqhZDlphsqqGp1Hv9TidoYeiNrdRbvnd7yxJ81I/XsSv/+PQdikym5HYCMtvgSVu6My+E0JaV2YyaUJ6V3YSMxT8IXexOtaDZUY+kolODhWKeVQpShhw+KDX8DCP0J456Uvi3OmuPnsnhqqp/W+l4WKys9yiznIObItT0aCtPGSBOt9QXakoHHiUNCBp1u8XFxgUmFrKJgxXAAWLzZ+jxsH27aNTLPDnk7SQ8VigSuug8uugZYmI88lJzcxD9IQkXqYlm2Pk6wxIWWIzrq3kXqY7NLj4473tn6d9LEGI/C3rQKp07T5n4ZGffcCEPbX01b1PN62FRSMv2KnAaMoggMOFRxwaOw9Syl5vSkY0XDpIZHUzYtKHWTtAtHJZBBCcNPRVg6bpPLAZyEWVZr7vO4+08qS7To17ZJsp+CE6Qq/fyfE1ubBBpAiwKrAJfunfjnoCJgTeDQb8kuk6WNsdNy2wV3PrYobM2eSMc48/9W284pe3c9XVIefR/QtbJadXJFAafTzF8HKxwc/7l+vMu/Ecn61OIRvSoBxFjul1uhdzWuWwPbPwF0AM84FZQ+zBvaw6e4evNNh7i4hRxHYFEGHpg9LOOnjDv/IGC/jx5sbp2lw441GKMes4TKUfBeLxehrVBzbZZoQigIFI9udtgdP82KkyTvQWHQ1fEBG4YHdyr3R0TXvkI/Vi0TX/bRsfQSkRqSlKNi1iYYNf8WRNRVX3r5YHebeZ0lkj0tfdKDUrrDZq+HRoi+EORbBkvYQsz5rYVaGUXV0SK415VVEmpS83RTk1cYgPl0yO9PCxaUO8pJI6BdCsKBCZXOT5IvK+GqtY3PhmKkWjum26aWUCCF46EKV6581qqlUxTAYwjrkOI1Kqor81Bt00ZJiB2K1ZBEIN5nZ49AmBIBAVRwRxfRyXFPZ0fZ2zJyXDFs5NovJsjlgi97JK7rhzY00+09lI3vLPOb2EaVr2wZfPwQtG8GeA+OPhrwJ4G+NbLj0oIdg6dVWvr0outGy9X147jzw9mla/cIlsPflcPIDpl/WLidtvCSB1eR1rk2XoEv2clhY7k+9QNDnngCalKjD7R2YNcvonBevQePatbB+fWIel2QNF1WFsrLeJOFvAN7WpSnak8TXupyMooNjjrLYCwmGOkiV90UPd8U1iML+Wrr89UbTx/wF5Iw+LW6ISxGCmRkqq7u0qPfECrB/tpUfj3Nx8lft+PXB989WAe1hybJOY2Ha5NV4tj7IdeUO/jI1I2UGTI1f48Sv2lnVpWERxrv7eE2An2308PDMTM4uGd6qj0v2s6BLyUsrNB77Msz6BolFgUMnKdx0tLGofbRJJ6TBzFGCI6eo2IZQxhUIt9Hk+RpPsBoFC1mOCeS5Z2FRHLhs5m4sRmUeRFXb24RlKg3qyCjCwoT8s1HE4OXPojgZlXUINR0fRNhSIBCUZR+e0PE+0OtjegYV4D29jrmKYbx8cju8/zMj5U5KY8Ol95k/XvUX4GkAd4R7g+2fwqNHMegrLzX46p/gaYTzXjB/rF3JnuE73c04M8FuzWYMl2QuHW2a5EtPMIktEyQcNh8+GYlQUU4O/OAHsGSJIV4Xi1AIOjpGZl5DJOxLrAQzFr72NXHHZBQsIHVho55+RGYuKcZn4W3+nM66d03t+7tjnTGd+YqAy8ocHJJnY/VBedw03sXMDJWJToXzS2wU2wS67P9qe7w5f9/h5+HqoXu8wPC4HL+0nXUebecxtO7jBnX41opOFrYl0zQR9hurxP207BY4eZbKjS8F+b9XQ6zvrlQKavD+Bp1vPRqkpl1yw2FWfnKkleOnW4ZkuDR7lrO67l7qOxfRFdhOR2ALVe3vsLruXrzBWpzWIlzW2P3RrEoGua5pTCw8HyXJ6huBgtNqTnemIv9MMuzR6/WLM+czOvso1AFzsVtymVh4QUI6LwCV0hPz3NWBHdLIg1n+KLx/CyC7HZhJXrY6a43voz7g5vCFi4n5lV//IjRvTO6YI03aeEmCAquFua6hZYoPJNklpDk8Ao3i/vjHoSfFHnjg0PNHVq6Ezk5objbmVBBDw2TFCjj/fKOtQHY2FBbCLbfELvfexUgzYlPCnCJu0LMJX8f62INUJ4jUncdmq5z60tX4EboW3wC/pNTB2cXGXPtetCzCeDcenJFJqcNIOB3tUPnVRDdfH5DH2oPzOaPYQX0wViAA/rTNl5SC6kDeaAyyxqNFDHNJDCPrj1uT8y5MLlLYt1yJqnWjCLhgroW31+m8ttpY9fpWKmm68fePXwwx63c+pv/WxwF/8fGHd0MEowhgSqkT0jxo+uDPqCtQRWXra0Qqr9d0PxubnkbTA1Tkn4olStWRImxMLrrEaFdhK2ZmyTWUZh2G01qM3ZJHjnMa4/POJt45L5E4rYUxx/QQCLfS0LmYmvYPaez6alCysBCCosz9mFX6fSbkn83Y3FOYXHgx04uvItOeuHqxPUqjy77YUJASPr6NlHQJ+aNax/zqTcyr3sQ5dZU839VOW7WkzUQK0ns3G78DnbD47/DQwXDvbCPUtPWDke/vFY102ChJnp5YwKFr66jZxYm7o23D/BGGw4Zuy1BQFCOcFO+sF8IYG6lNwK23wsyZ8Y+l6/CnPxk6M329LS0txut4/nmjJ1JeisX8UoDFnt+nbDkydncFQrXhb18dd38dNa/gzJoS8Tlv20patz0adVtn7r74O1YjNXNGq9U5GlfuPgm3DpB6kKBnM46syM38elCE4LHZWRxV7eee7T5Wd4dkji+w8cNxLg7MjR7j/7A1hFVAKMrpJ4ENXo0PW0Icnj80Y+7lxiAq0QUlwxJeawwS1iWWiM0NY/On021c9FiAHa3Gi+kxiHQJC8YpfP8wC+c9HIibANtjq7R64eEvwjy3LMzz37YzuluIT9OD1HcuotGzdKeeSqZ9HCWZB5LpGAtAQ+cXRE+1lWi6jxbvKgoz5jKt+NvUdy6m2bMMTfpRhZ0C994UZe6PVe01ei2qi5KsAyjJOqDf3nJ8U2jzrY/6qkRII299E0r9FkKZDjomliCjNKetanvb2AaBRKeq7R3Kso+gKHNev3GKsJDtNCm4GYN9lXw26Z1RPw8F2E8poGWTkeMyNCT+Ao1Pcjp3noNbwkF+29bA4kVhnMQXGGzbZnhf/rUAfH0q2BtXw+r/wD5Xwkn3EbUx6EiRNl6SxKUofDGthIeaPdxb30mdCen+VCKAiXYLs53RL9op4a23oH2IPTGkNBe2ycyEyZONcFBfhIA1a+DLL2HevMjbgtFf6ayz4NVXIz+vabBxI9x8M9x/v/n5jxDuggNor47dQymr9ESEYjFlvIT9kaTWDXdya+WTMbf1tS5BseaZNF4E2WUnYXOPx+IY1X1c898HqZsLo6hCcPloJ5ePdqJLQ8LfTJ6KWY/KMUvbGeNQeGhmJofmJWfEdIb1mCohYCy/IZncxbcoU/D8t+28vFLjxRVhWr1Qnis4e28LR01RUASsb5AJe3I7A3DhowHeu96BIMTGxsfxhurpayx0BirpDGxjXN6p5Llm0BHYsvN5TQo+6yrns84xhKTCVGczJ2RvoN2/mcKMuVjVDEbnHMHonCN2JhADEArAsvegrREqZsD4WQD4Q800dH1Jq28tUoawqTkI1O5E2v6vrmDJFka/vw7F7yeT7kRkp40dx+1F66zBFUU928udvzWq2t9BVezku2cn+M7F5yClkFf1KjyEB30rBGBF4UilpE/Zc7IYr2jNTU39zsGed2thYQdHmjBeQl64f47xu9/euyf/1T+haBbs/92hzndopI2XCOi6zpMtXt5oN0Sxjs92cmGeC2WApoqiKHy7MJMzcl3MWVUb96KVKnqaQN4+Omf4e62sX28kxybSKToSs2cbPY+i7cdiMUqsl0ZIWpUSnnvO+FFV2G8/+N734Nxz+4eibroJXnst9jw0DR59FO680zCWEmHb1/D+g9C8w7jtKJsGR10NhWMT208U3Pn7421dFlWczl14MDZXd7xd2EDGD7f0Wyi68bYsBRk/D0sPdZfIC7U7AB8ZoTqw2AsQQlAw/gqaNt8f14PUD2WwoVAb0HivOURQl8zNtrLXADXcRHQxDsy18o8d5nJatvt1jlrSzk/HObltcuKVfGZ0WKyAHpb9NPjrOiRPfxXmw40aQQ32Hq1wwVwLM0YNvr112wTnz7Vw/tzBl28pjUqiZO6lGjrh4006U4s/H2S4dO8dgMqWV8lyjN9ZSVQbzOD724+nMpiD2r08v94+mb/Vz+PPFWuYOCC6K4Qwbmae+Qt8+Azofc4tVya+865m3ejKfoaKP9wMSFThQJN9dEy+qqL89WW9++7+rfqCVLzwJVIVtE030Y8CqGn/iDzXTFMaSYngFhZusszgzvBaWgnuDHtalrjJfaKYqU15fD3GyoxzQHWANoT0q2COjmd85JsB74Qw4VwNS2vsMFbT2vjH+fxO2O+6Xet9SbcHGMCXXQHO29w4qLTZIeDJ8QXsP0DFdlsgzFeeIP9u6uRrbyjpbhuxsAsI9JnPLKeVX5blMD9j6NLScXnwQbjyyqHtQ1UNDZj5840w1FBPuZ7y6u98B+67z/h7xw6jp5LZLtjLlsFeCajmvnonLH8r8nPHfhf2PcX8vmKga0E6697C0/wFUjdei2rNIaPoMNwFB+w0ROrX/Ymwvy72zoSFsr1+N+jhlsr/4GtdEmGDZFFwFx5ITpnxHkg9jLdtBZ3176KZMGKE6qJo8vex2PPwapLvru3kiZpAv5uB/bMtPDIriwmuxIXUgrpkwsctNAbje0X68sisTC4YlVhl0PnLO3iuPv45OGO5m3PnWLjhMCsrqnWu/k+QoNabo9IjLPeTIy1cPj8x7+p1zwT4aJOekKBdD2fPUThz73/Eld4fnX00rb61tAVqOW/TmdSGMtEGpFAKJKqAhfvnsnfWgNfwz1vgy8Hfp54pbz1jP9pmRjY6ijL2J8NejhoWZPz8OwjvwG7VvfsKZrtY/b1jTefbTS68OGYy71AIS52lsoXVvnaaLinG998MFItE6gKhGIq59hwItCV/DIkEAZ8/Wk3TYYM/wwl/z2HG7wtIRWLN97dCzrih7WMo63fa89KH2mCYMzc1RrzA+SWcvbmJz6eXUGazUB/S+OH2Fj7sHFq3WzPiTG9PLiKEoDGsUWJVmewY5lBRX045Ba65JnrHZzNYrYYBc8EF8NhjxoWkJ4zU0026rAxqa82Fl3qMnwcegAkTDAXgl19OzChyOuOP6WHZm9ENF4C37oGKuZCfWBVCJBTVRnbZyWSNOs7wXghLt1ej/8KQVXIMLTFyVgBcOXOiHiO16HibF+PInIan+XOC3mp0zQO6ue+G1Px01r9HTvlZnL2snXebB98ELOkIc+jiVpYsyKPEntjtnk0RvLRPFscsaacrHFv8vS8/Xd/FeSX2hLw8dsWcYJ4vBI9/qbFsh8bGJgho/U/fHkXcP74XZlqxwoIK80bbFQssfLAxuSpEX0gz0TNIwR9upChjHi80LqcqFFnzRHYvkH/Z5uPR2X2uWbVbIxou0Hs9HPvKUtqmlxkJPQNo6FqMECpFm/1RDZeefdnbvbirWvCUxw+XVJPFQo8f3dfEOKuNo50ZOFOoYG4RCvuLAhp/VEBVdzmyHjZeX09IZiiGC3Tn8UjJvGtG8cbaLf2eszeqTHgod2gH6INJCZ9hI11t1Idbq9tiXtg04OdVbXRoOqdvbOCTJAyXnq9itiL4blEGN5ZEtzZV4OgsBxOdNqY5rRyS6RhZwwWgqAiuu25olUJ+v9HY8bHHjL913TBaiovhnHPg9dehujrxcmZFMUJFiRouEyea73wN8Mlj8ce8e6/5/ZlAKFaszlKsjqKIbmxnzixs7ujigYrqJqvstIjPuQsOStU0dyL1AM1bHsDfvtoIN5k0XAx0vK1f8X6zn7cjGC5ghGOagpK7K720eCSeBHtz7JNlZeWBudw83kW+SaGmuqBkcXtiRvvBudbYhosEm0dBkUbp9opa8Iein76qgH9/kdgc9ilXuf1kK6oSce2PyfQSc0uCIqzkOKfyhW9vlBivOCzh+foBcZB3noi5bwGoIY3MzZFztkBS3/k5tdUvm5qrxdP/XNSB1RTzNHvxBHvzIeP5GwfwS47lEa/kia5WftVazzG1W3jf12XqGGbpqoev/zW8C79AYO1SKXrPqPBy1KoUv+Nm1q2F2JtUUuF1sWdBduKFVykl7XnpwwcmlHM/6vTzZLOH7UEtqfJmAZya4+Tv44w7ASklLZrOPxu7dlYp9Pye67bx17G7QVXMnXcaein33ju0kE9f40RK8HrhV78Ce5Lhr2S1W37+88SMsY6G+GOq4uuqpJqCiVfTXv0inubFfXJYBPbMSeSO/RaqJfL7anUUdifWpk5XppckPxMZ5rEaL5Yo/YvA+E7ctcnHi88aXoj54xSuOtDC/HHmvBKj7Cq/mOimzKFwzRpzi1JrgtWE55U4uHmDh/awjPxOCMit638DEusbpUn4ojLx9/S02cb78tyyMGvqdASSjzfLmLkwVgVO38tOQ+dYuoKxmoLqZDunGCFMpRid2F6eoIS6ji8oydrfeKBhh6nX4GjqoHNSNO0WSSDLnAcxmN3rZW3GyV85mBqyUdARgNZnMe9tagFeKflpcy33F45mH3sCntoYbH7LCA+NBOPvzWHs49mUvOdGyNTmRs65fNe3E0gbL30wczMXkPBksydpXRYdeLXNx21hnVyL0WL+l2U5nJvn5qlmD9uDYXJUhdNzXRycmZjLetiwWOCGGwwF3Q8iKU8mgaYZ2jF33gn33GOULpvtoTQUfvc7uPji1O93F/hQhRDkjD6d7LJTCfkbQA9hcRShqPGNwcJJ11G/7k/oodYRmGkvrbqDx31784RvL5p0N1p3weoktQXVEb8NQLiPnbK4UueLbUHuONXKyTPNX8rOLLbzvbVdUUun+1LRnWPzdUeIJ2oD1Ad0yhwKF5c6mJ4x+Jhui+DFfbI5cWkbPq1PybQEBGTXWchoSeyyKyVoukRN0I1SkiW4/pBeQ2lZlcZFjwUjGjBCwB9Ps5HjFKjiQDY2VUbZq8BtKyWjuy/QrEwrbzSFYmjoSMba2qjpeI9c11TslmzIMndDprlin8edFYUEMx1YO/0RfQkS8Bdm4SvJAaAdO7dxFF0Y+9VNBB4k8M+OZu4tTE0eTChVrZlMUPiF4XkRqRCO6YOrAI75Y0p3mRTphN0+TFxehS+eFAmQoUDnENeqtyYXMTPFQnfDxubNRolyR8fQq44G4nZDV5fhDbntttTuuy/Z2fDmm0bScKL8+UzwdcQeM3oGXHJXUlPbVUgp8bUuo7Ph/RjJvwKhWLFnTsbfvmpIx6vWMjmr9ULq9IzufIi+F1WJGXe2GhRULO8vhmdT4aPvO8hxmr9I/36Lh59vii4WJ4DJLoXD82280xRks083pP6lsdCHJVxT7uCuqRkRbzCq/BoP7PDzj01+PGGJ3auQ3WDF2aEmtZgowJFTFL69wMrsst5FV0rJYtnM21otW+lCQTBT5HCcMoqpyuBclDavzi/fCPHxZh1fyAgrHTRe4dqDrezVZ7/NnlVsb30NvxBstRfgU6xkaj5mSAdT8s/CohoLY6VPY/InLTH8bZIbSz7jzLx1lGQeQGn2obB1NfzukpivVwpY8ZOT0Byxr5GO9Y1Me+ZTQ++mz1ImjTp6Nn7rILrGFbKKIv7GQWgmBOMi8WLJWMotQ79eVy0y9FP2ZC75CMYdkpp9DWX9ThsvfThiXR3rTUj5ZynQMUTj5bNpJYyz7yGOr7PPhhdeSL3h0kM4DMEgTJsGldHu+JKkqMgoq77mmuSF6T55DD6OnRzLBX+Air2T2/9uQMCzjeYt/0JqfnqNCIlQHOSPvwybeywdtW/S1fipqTLrSJzTej5LQ2WDqlJMIyGv2kZebf9FRAA3HW3l4v3Mf5+klFy7posHI7QF6H318RPqfznBxS0ToisLv7IqzI0vRdexERipW7puTmVbAH89y8ZRU1SklDymbeV9Wd9vnj0Jw5co4zlcjd5bSNMlioAG/LTIIJnCShlOhDCSPl/RKnlZryUsQEiJFAI3Khep45mv9NY//2O7h++v86Kg9/FmGObpAvcO/jjmbSwCcpxTGJ9/hvH07ZfAtshaRRJonjeV7cdPj/ledCp2ns3bhwlrNnHe8y8wqqE3vLujrJT6Y2bQUV7Ic8ziC8YylFyP8RYrTxaPxTpET7iUcN9ehuDbrk54TQoFFvwAjrkzNbvb7aqNtm3bxm9+8xvef/996urqKC0t5Vvf+ha33HILNlvvhWfFihVcd911fPnllxQWFvLd736XG2+8cTimZIoyi0IcQXVgaIaLAkx1WBlrS+4OYMRpbk7ecJkxA9ati71taalRifTAA+YNF1XtvtqbuNx/8YWhHzMUDvoWfPwYMZeXTx/bo40Xu3scJdNvwdf6NYEuQ3zMljEeV+4+KKpRLpxdeiIZhYdQt/b3EEEuPhYbwvksDkUSDDOJBEtAkN0wOGFdVWBTY2JfSiEE987I5IxiG99f18VGb+/2DsXoQxS5N3Z//rTNxw/HuXBG0ew/dqrKPxeG2dIkB5UuqwKynXDz0VZueTWEpsfXiZHAT14M8skNDtZbW3lf1u98vIeeV/KovoXpShbFInK+xnY8PBHeyiZ683/KcHKuOpYq6eV5WbtzvZfdi7YHjfu0jVhRdjYSvKbcBd7neaRpNl95jT5GJdYuzs1bzTl5q7AICSj9ewXd+CD85TrY+NWgeYkDTyHvWz9Fetewo/1Non0KH2ROxqPYWD5rJstnzmDsjiqyOjtozcmlprSUnLCFZS122nEw1CTVLeEQH/u6ONKVoDZUH4Ie+PIf4GvdQw0XAB1qB39ku4RhMV7WrVuHruvcf//9TJw4kVWrVnHllVfi8Xi4807DZOvo6OCYY47hqKOO4r777mPlypVcfvnl5OTk8J3vfGc4phWTsJQs9SbXMC0RdOAno7KGX1wuVVRXJ2e4CAFnngm//nX0MYoCV11lKOfecEPs/U2ZAhUVxv/3288wjM49N/Y2RUUwNgUCcivfJe4ytn0l+LvAkbiw2e6CojpwFyzAXRDdr60FWxM2XAC+DCaWM+BSYKc9IcHdqlJYaUfVBn9vJJJt1hDHLfGyrDOMXRGcVmTj+jFOJrmNS1xdQGdJewhVwIIcKzlWw0NwdIGdNQfZ2erVaAjqfNke4gfrPabn2alJPm0NcXRB5JCCzSJ4+EI7P3w+yOLtOkp3LyZNwtg8wT1n2RhfoDC7TOGppRpPLgkTjPN184fh5ZUaO+bUxizLFhgdjc9Txw16bqXeyt3aegaWHdTg48/aOixxFvuHtc3sTQ6KYuTtHV+YyfyM1/HrCmGp4FZCA3LidXKcU3v/tFjhJw9AQxXyjYcItm7Dl2en9aB5uAtnky9AJ0S0712nYmebPb838V4IKsf0N46/6nLiwUYqqmsU4A1vZ9LGi78dHjkM6lfswYYLgABranKXh8ywGC/HHXccxx133M6/x48fz/r167n33nt3Gi9PPPEEwWCQhx56CJvNxowZM1i2bBl//vOfd4nx8n6Hn3Y99RE0BaN6KAzYheD20Tkck72bfPpmSDbUIiV861tGUu4fI2R3qaphkNxwA1x/fa/eSyRUFQ47zBCk67v/3/7WaNYYzQPzu98NvRkkwNqPzI3buAhmHTX04+3GyCRCRpoUPOLbJ6Ft1h2Ux8cNIX7ycgi7V8ESihxqkkhqywM8RRi1pSdBVvJAlZ9/Vft5fFYmz9YF+W99r+idXYEryhzcMTkDR7fHpMKlUuFS+dM2rymdlr7441w38t2CRy6ys65eZ+FWDU2HOaONJos9NzFjchUu21+YKos2WgDo1MuuuN2Kt8j+VVWb9E6e17ezRkbO4ep5JeE4xnoXYR7Wt3CFMhGA4swFtPnW4VA0Bnd2EjitxWQ5Bpf2d2QG2HKUE11O7H6kjtb2Oqra30VVogsENlvcMb/bIU2hM5gawwWM97JNTz5s/t7/Qf3KPdxw6WZyavQ4h8yI6by0t7eT12ch/PzzzznkkEP6hZGOPfZY1q9fT2tr9AqIQCBAR0dHv59UsCUQTjKVKzpZiuDHJVl8pyiTO8pzWD5zFOfmJ955d5cyejQccIDhJTGLqsJJJxlaKnfcYVQTlZb2Pm+3w6WXwqefQlYWfPxxbBE8TYOPBhgQQhg9jCZO7P0beud5001w2WXm5xwLs2lh34QrUwwCXVtoqXw64e0+DI5ng2au468A9suyMMqhUr1Jxd1uiWq4AHTmh+koMs6dvktLWBqhn3OWd/YzXAACOty3w8+Zy9rRB3y2XVqUEucYzIhQdTQQTZc0dBo5JtkOwdhcZZD31ax2jZSgOkIETczU0ucSv0pv43faatZGMVwS5RPZSKs0NFRctmImFJyzMzQkdjYxAbetlIkF5w5+vYEaNjX/B11G8njraHr0hGo1jnHlD6dGz6T3eDAmSsKuT9d51dPBve3NPN7ZSm24/+sJdMKyh2J22NgjECpkFMPsC3f1TAxGJGN006ZN3HPPPTu9LgB1dXVU9IQBuikuLt75XG5uZCXA3/3ud/zqV79K+RwzFJFyaf8OXXJSjpMJIy0sl2p++1s48sheWf6B9Dzek3k4Z47RP6jnueuvNxJmV60y5PunTDGqf3qwmDgN1QimZXk5rFhh9Dx65hmjgeT06UbbgL1TmH8y+QDYvDj+uAn7pe6YuxlB7w6aNj+QlIH2jG8mKrqpRF0J3DjeqGRp8fapHkHiy9JoLwoRcOkIHTJaLXTlhBGyu7okwr4gcodnHXi7OcRbTUGOL+zNxZiZYeGDllDckm0wclaOzLMyPk7LgsWVGje+FKS+0/CaSAnizRBHT1G4dH8L00oU7BZBUabAphI3bIRFo+aAtaauV3sL4zqqSck/tU3oJN60MRaf602coBrK0lmO8cwq/R6t3nX4QnUIYSHbMQm3rSximHx76+tJH3dUsB2bHiYYRWxEpPRVGufQ6e4sNocCbAkFcQqFuXYnH/q7uL21Aa+UWDDOq7vamzjDncVPcoqwCkHzeggPoV/RLkUY/YukBhklcNHbYNtNIuMJGS833XQTd9xxR8wxa9euZerU3thmdXU1xx13HGeffTZXDrVHDnDzzTfzwx/+cOffHR0dlJcPIRGwm2OynfxfVWyF3WRQ95Tcllgcdhi8+CJcfjk0NfUmzKoqnHqq8buyEgoLDQ2V004zWgL0RVWj9xI6/ngjJBQrbHT88ZGfcziM8NS3vpXkizPB3ifAO/+AcIxcj1GTwZ0zfHPYxXTUvNFtuCS+KNTqWaYrjH470YVXkxz5ZRsr7WE80wWZzSpBh6SzKNyvorrVERrSzbUKPFTt72e8XDHawV2V8cU4VAGFVoW/T4+dA7G6VufbTwV3yv33RJikhLfW6by1LkimHS6Ya+HaQyycMkvl+eUasSJRs4+txa/GDy+5sXCAYni8VshW2kltTp8Adsj+3hFFWMl3zwJmxdw2GG7HFzYh/hgFCzr7eLezyD0+4jnQGUitDMWRDjd/am9iZbDXCrEDfbV7+34iz3s6EAhuzi0i4W4cAlQb6KFd58wVVljwQ2jfBqoVJhwH08+CKLqXu4SEjJcf/ehHXHrppTHHjB/fG9esqanh8MMP54ADDuCBBx7oN66kpIT6+v7yzz1/l5REU1UEu92OPVlF1hgUWVUuKXDzcFPyAnQDsQFj9pSqonicfDLU1Bihmk2bDM/JaacZSbFD5frrexssDvTsCGEYL9deO/TjJIsQcO7t8OSNka8m9gw4b3ADxGHjo+fgtYegrfvin5UPx14MRw+PP1cLdRHo2pj09iVKJ6spjmnA5Kjw7rwcfrLBwwctffJOXJJml967QPVdqIZ4X6ABW739b1emuC3cPsnNzRs9UXNfMlS4rMzJTyqcjLLH/n7/7eMQuk5MY6QzAA98HmZtvc6vT7Ty2RaNus7ITs6SLMiZ3oqZG/mfWKbhFt0Jy9KfUC5Pj8p3LCTwuWwiQ7NwgTIuoSKEkDZU2X3BQYEOHJlFfCgbUOi1a7tCFpp8qckrzBAKYy1WPvZ7GGguxmqAIYH/etq5PDOXwhlWMkuhs8bkQSVoASNMk1Li1f33nUIIlv0LfpJAc/iRJiHjpbCwkMJCc7Hr6upqDj/8cObOncvDDz+MMiBnYsGCBdxyyy2EQiGs3Xfp77zzDlOmTIkaMhpuflGWQ1BKnmj27rzMDsUTEwQCUuL8JnhfwPCmnH566vc7ZQr85z9w3nmGR6enuklVjZDSc8/1VhrtKsbtBVc/BG//HbZ+bWh8W50w8wg48jtgd43MPO6/CZa+2/+xjmZ49i+wZhF8/56UH1LXzFffROJM5yreDk6O+rwAbp7g5vHaAB+1GN4Bve+TYFbDLmF8EayKH1e4mOhS+cNWL192GEtWmV3hunIHV4x2kGNVTClfd/olH23STa0XUsLHm3W+2qHzzGUO/vpRiJdWajtDSHkuuHBfC1cdaOHbJqr/LAjGiV7/vgM1obB4OW4siH5l1NF4R68jDzvHq6Vxx+6cn2ou90+gYrTS7Fl5jd82NYtJBecx05LLkbKET/VGmmWALGFlkdfCVgJDSgMQgEsIuqTOmlAg6VYw7/m6uCAzlwN/Cm9+P7HtU54jk+CL8DbB5ndgwtEpnkeKGBaRuurqag477DDGjh3LI488gtonX6HHq9Le3s6UKVM45phj+OlPf8qqVau4/PLL+ctf/pJQtVEqRep62B4I81Kbl1favKz1hYf0JVg/q5QMNd3/0hSVlXD//fD++0b+zJFHGqXUo4enRf0ex5J34YGbYo+58GY49MyUHlYPe6ld9UuSCRmBUW10fuu5LA6XE8kCUYBX987i7BUdeEY4qTHbAk1HRL8hawvpBHUosImEW3VUt+sc9bfEmrdOLBC8cpVRZeMJGkm+GXZB2KrzcLWfL9vDrKCFUXlexhV3YlEjfyYOVO6z9uZgtckgPwwvNX0tsyD4mTqT27VVBE187plY+ItlLpYITUSjsbb+IXyhaMrOBoUZC1imTuE/nc1s0y3YkBxmF1ycM5Zx1sE3DLqULKjeNMhLkgyJVp0NxAJckZXHd7LykRJevQa+uj8FExtB5lwKpz48fPvf7UTq3nnnHTZt2sSmTZsYPWDh6bGVsrOzefvtt7nuuuuYO3cuBQUF3HrrrbukTHogY+wWvlucRYems94XuxwxGgKosFtwJ9rW9X+ZsWPh9tt39Sx2X1420bn6jYdSbrwoFheO7On429eSzOVcFZLj7BtZHI7chlYCZ63owLsLqjHawxDUJbYo39MePZhkyHeZTMDtw6YmyRtrwhwxWcVtE1TkC56rC3Dxyg402fPuu9ne6GbZljyOnFNDbsbgXKyZZFDXsZDOgCH8mGkfyyHOAj6i2ZQJGkaSKazcYpnF3eF1tMRpvthJmO3Sw3hhXgelPOdoNjQ+TjSjWBWZ/D08mXe6PCg40AEf8GoA3qyv4e6CUuY5+hswGqTEcIGhGS5gzGO0akQVhIC5Vw6/8aI6QIsXU1SMnJq442BYvJ2pYlhcApdeeilSyog/fZk9ezaffPIJfr+fqqoqfvrTnw7HdJLm2Gxn0l8ECXy7MGPPEaNLs/vTaCJo3pp8EmQsskadgFCsRLuaWZzRQwZSwiP+faJWgEjYJYYLGGq61mH6ijqsglNmqSTqeP3hCyHm3OHnqqcDPLUxwLdWdBCWfRdToy9UMKTy3rJSQuH+L0BImN3wNjUdH9EZ2EpnYCs1HR8xs+El5uvmckEE4MLCWOHmWKXU1BoWStAzl2EvZ2LBeViUwSGkDPs41mSezzt+IyG4ryGhAUEkP26uxTegs7xVCMrUeBJ7sSlWhp5sIgC3UDjC1Ru666ga8m7jHjRvYvxh6DD5RHO7nLWblEVHIh3PiMFcl4393Lak3qTTcpxctKdpuqTZvTFjCCdjLEtptHFYuNBQVI6A1VFE4aTrsbnH9XtcsWSSU34WhROvwWKPHH5pkU4qtdzubje7DxYB55fYh/UG47qDreQ4jeqkRPlsi851S4wCgkhmgUQQCKlsre/1dqjAaa0ryJDeAVtJFBliv8a3uIHIHrAeFGAfkYezO2N0jHDFNUsEkIWFd7Ra/qNV8oZWQ4uMHzLLclQwa9T3mFR4IaNzjiE3+0RKir/PuPzz+I+nK+oZI4EuqfOmr7Pf47qUlFusQyq68AyxxKdnzj/LLcLRN4w23F0EJbRthdzBWoCDcORCwdTYY1yFMP7I1ExtONhDOgPuGoQQPFRRwLmbGlntN1dmONtp5aqiTE7JcSYcI0+TJiZjpsCWlbHHlCSY2Pzaa4ag36rujtFCwDHHwF/+YjTK7IPVWULhpGsJ+RvRAk0I1Y7NPRbRvcgVTLyGutW3MdDhnopvQU+6pkVgSoMl0rZ9UTGUdn9UMbyJ1iVZgv9cZud3b4f4YKMes+poIJoET7YWN3zR0pRFRZnG3iKPGV3raQ+1RB0r0SnybuAw9xg+lJG9dAqC09TecP9UkUU21pil1hK4RVtudHdGoCN5Rq/kWGUU5yhjY14LhRCslYU86FVZGvRBew0uBN44q70KrAz4Od3dqxn1p7ZGFgXil7rHwiulqWqrHgbmxriFwg+yCzhmQCsBl7lalyFhywRXEbRuiT2ufpnRHfpvkyAQQbNQtcNlnwzLFFNG2vMSh1yLwltTihgXp+TZCnwxvZg3phRzWq4rbbikST1n3RB/zJnfM7+///zHKIFf3ae7r5Tw7ruw//6wZk3EzayOQhzZ07BnjN9puACo1kzchQcOGp8rfFSoLQkJh/VcmET3zwSXwnNzMjmz2M5ou5JQqOfAHMvOfVq6tyt1KLw1N4cp7uG/fyvLVvjb2XY++K6Di/dLLCQRSXyvP4IxZPIL62xOsYzG799I7Ft8SbtvIxep41kgCqKMgI/1Bvzd5S5CCNwm7nN7VIC0biE8Cbyp13JneA23hpZzS2gZ/wptomNAb6y3vJ1c21TN18FeoyOe4QLGeWERRh7lYr+XW1vqeNrTHnc7M68jkSjmQOPSJ3Xuam9ixwBdKEv0bgepQTHUbzUTOeKtWyCjCH5YC/OuNzwxisUwfmZfDD+sgYIpwzzfIZL2vJhACMEfynM5f3PTzi/oQH49OofRtj1cSTfN7s3EveDUa+ClKIm7x14CswYbDxHx+w3VYxgsKKJp4PXCD38Ib76Z0BSzS09ED3fha/0autU3hBBc5VrMTZ3Hxdt8J9PcKgEJhVbBt0odXFgA7h2fMr9hNe+2hPjQNon/OPfFJyIrgKkYi8rfpmXwnXIn6z1hXm8MEtAle2VaOKbANuICkkWZgh8ebuWF5RqdURYYXZF0FIboKAijWSQ7rYAoU1WBedm9l3EpdToUB+ucxXgUOy49yBR/HTl9sjMlGltlF0tkc8R9akje1evYpHdys2UGNqFSb0pZJjJr6L21r8bHJ1ojB+qFXGmZSKeu8evW+qihsViEgRlWB5c27GBVKLGqrngkIIkyCA3wSp272pr4U0FvLthHv2LoJUwxUG2w//eM5o9CiS1w58gxfttccMI9xs+exrCUSo8kw1EqHY2PO/3ctKOVyj7lA3mqwv+VZnN+Or9lj2SxrOYdNtPWLTnlwsohjOVIsYt1ZWKxfb2h67J9nWF4jJ5keGXGzzS/j2eeid+VWwjYvj2pUvWgtwpvy1L0cCeKNQtX7r78uDKTe3eYWwTf3TebQ/O6DZOGdfDe7yHoJdztk1HRaRZuTs67jiW2/p3D98pQOTjPxlXlDqaOgGclGqtqNVZU6+S7BAdNNKqHAF5bHeYnLxohmL4X37BVp3qqj5C9+9G+0iZRUDCaWFa4VKSU/NPzHgttrm6PlUR2+7tm+6o5tHMjAoVc1wz+lTmGKqL3Duo5/HnKWI5VS7kq9MUQlVMGc5RSgt2Xyx1tjQkbCipQqKjkqRbWhVI9s9SgAG+PqiBXteBvhz/kDa9i7qST4IJX4OuH4eXLo48TChzyczjsl8M3F7PsdqXS31QOyXTw2bQSvvQEqQ5p5KoKB2basaZDRHsk/5GrWEJtv8e8hHiTTayXTVwr5u2imcVhzBT40X3xx8Vi69bYnbzBMIwqK5MyXmyu0dhc/be7e6rktCI75y3voDVK4ooCjHEqHJzb7cX0NME7v93ZmsHSZ5nKkV7eaPkrMwp/QYNqXPgEcP1YF5eWDbePPjqfbdH4yYtBWvukXigixLl7q/zsOCsnzrDgsgr+9EGIzU2970N9RcAwXEyoCFuEkRNz7/QMKrp7K72m17DQbtxEGZ6M3o1XOMtw6GHme7biz5hF1YDzPhISeF+v51i1lDkily9lc0qNhPf0OqaEXKjEL2/uyUHpsedyFZWrswv4ZWt97A13ITpQp4UN46VteA0XxQLZ5cZXtmVT9HFCBWce7HvN8M1lpEjnvCSIEIL9MuycnuvisCxH2nDZQ1kvmwYZLn3ZShsfyq0jOKMRJi+vV8k43rgUIYTgiHwbb+6bjVNhUBd3FaNx4QPTM3tzxta9DVqISE58C5JMGeBy78LefQhoDO66+/CFW8N8+6n+hgsY7QGe+krjxy8YRtjhk1Ve+Y6dl79j58dHWAg6dHzZWkwvi8C4YNsVOKHAxvvzcrh8tFH6HJQar+mRK8WMjQVfucspyD6CTot5LZbmbo/kcUppyotlJNCKOfXaE11ZzLY52M/u5P9yinihZBzrQoHdfgHL7i67dheSeI+jBNDDMO0MWPhH+DSGVJYrHy79yOgOvaezu3/2adIMC28Q4/akm4/YPgIz2UWcfnrsbt5CwMyZMDVOPWUS7JNl5dP9czky39pvrT4gx8p783I4PL/PVX77FzFvWRUkZ/i/3vl3WEKZY9dd1m58MXZV4htrdbY09SbCTipUuGKBlWMXxN+3BF7ZJ4uuowr5797ZHJTbm2O3Tnbgi5NmGhYqje5JuAeZjdFxdI8dK9y4hsFRX+GInRyrAHvbHPwir5iHi8r5R+FozszIxqUo+KW51guRONLh5vrM1BnmAxHADKudUovxGVldMOuC+P2KkulnJFQo3RdGz4ePb4s9VguZK6XeE0gbL2n+J2kgfr+erjiqons0BQXw4x9Hfq7H6/H73yenG2OC2ZkWXpubQ+WheSyan8OWQ/J4f78cDsgZkPQeq5M3xiLhlL0GQ4YKpxXtmta36xs0mmOnkQDwr88HL9f7jjF3KY5WxWg268OPzhSRRYYJQ0SBnV2p18p2BrcmHDoH2jLZz+6MuhDpwLezIhsZEyy2pHsOTbHaOcxl3gOVDNdm5/f7+7BfgTM3uoHiKoCJx0L2uMSOIzWYdCJsfR+CnbHH+luh8qPE9r+7kjZe0vxPophQH/nGBwRvuw1uvtlouClErycmJweefBJONCnDOQRG2VXmZlkpd0S5ouePNzIMoxBCYam1V3TtD1MycJlVhNM18LRBcGi6ID2srjW3lFa2DjY0diYnx8CuwLysyEZHqTCnnFsmnFiEwhlKecxxArCjcowyCoCmmD2U+29nFivwrl6LJ6sKly24c3tL928bgt/kFjPfEbkY4iR3VlK+IAXYFg5SHQ4xShmaGi/0lvP3nMHZisIf8kcNmnf2GPj2FzDhWPq9UZllcMxfIKcCNr4O7ZWJz+GjX8HWD82NjaTrsieSTthN8z9JOVlsojXmmHzMLQh7LIpi9JL64Q/h+eehpcXo3n3qqeDYdQmv/Zh6LOz4MurTVnTudx8CwCE5Fs4vMeF1CXhg4dPw1Wvg775VHb8vHHQhlCdQsTWAXJN6d3muwcvl0o74IpiXlTnIjtJrqUy4mEQmm+mM6INRgNG4dnaaPkItIYjOc3plRH9KPna+a5lCgTDez0zMyUD8QJlKgeKgXvr4q7Y+pmckDKykHUWBitwOvCELHX4bVmnhTFsJZ7pzyIwh1Z+lqPwit5ifJ5i0K4H3/B5e98XvmB0PBTjVlcU0m4NOXWO0xcohTje2KAZ37ni48DVo3wHNG8CWYYR8njgear/qM8EEESpsfTf+OIC8SYnvf3ckbbyk+Z/DJ0METchQZWLnTbmJuRh3nyF08nDiEN+wr01BAewGDVEjMmoWTDsB1r6OROwUugsjsCC5PeM4PrcZQfyP28Lkv9/MpWV2bqpw76zC6UfAA4/+ABor++fSbP0Kti6FM34OUw9OaqoLxqlYlBDhOBGcqw4cfP78eZsvrrbIvlG8Lj1cZhnPbeFV+OmvyqsANhS+benf+OY4tZSDlSKWyGYq9S7aCTNKOJkispghsvuFqGaKbGyImB2mc7EyU8lBEYJS4eT/mMl92gaaB4RfHd0O/wB6v3m6rGFcVqMY3i9sZCr9wy6ROMGdhU/q/L6t0XQllA4EUqQQogNv+Tq5JbcooTYT2eXGDxi6LFveGdo8pAYNKw3J/+YNkdPEhArFs6Fkr6Eda3fhG3YVTpMmPo+xgh3E951W0s422niP3qojFcFcOYoTmIQ7ikBamhQiBMy7hJX2cYRXvcLe4R0ALLWO5c/uI3neuU+/4Trw7+oA/60P8uG8HGZmDrjEffL4YMMFev9++Q+GF8aWuNfNYRVcsp/KvxZFN4xnlAhmjOpvVHnCkqUdsfNJVOCztjCXlEUfUypc/NIym5e0HSySzWhIVATzRD6nqqMZFSG05BYWDhXFoMQuPzEk/2NTRP+WKJOUTP6kzKVO+liutxKUGjOVHFoIcY+2Pup+dOBL2cy35DgyRHyPz5kZOSz1e3nLHz+PbTjwSkkQiT3JANTGNwzDQqagOenx98BTJxuJuX33J1Sj2unkfw79GLsLaeMlzf8UO2Q7G4ne+6UvkWoZNCRLqGULbXxX7ofLxMU1zdCQwIVds1lXOANrd3JuMMb7rgNdYcmlqzpYsqBPsqcWgq9fjy24EfLD6g9g7xOSmuuPjrDS6oPnlw9eiWaWCJ68dLDBa8YHIASm+iIVCQdXWiZxiRyPhzAuLNiTKWHpJix1Vso2FmlNhOPMdCMdaLqOqvQPmZQIJyVqr+G0SquKKzSrIdmidzJbNVcR9HUweQXgoZIhFGxDyJzRgsbnOyRfkICccVBxJFy+EN6/BTa9yU6Rw0nHwxG/NTwv3xTSxkua/ylW0bCzcVyy6Eia8fI+WzmJyRHHhKWOjo7tmxZiGkm2bobPPqTGEyRLTkRW7BXTaOmLBizv1FjSHmLf7O5tulogGKccSCjQuC3pKQsh+O1JNq49SOehRWG2NktyXXDlARamFkc2IjIsgqlulfUeLepZGZZwQI75c8kmVGwJlERH4mu9hYe1LXTEaMjYFx1YTTuzyY05zo5q6tv3V30DN4ipzFRy4o717CKheBU41Z01pM7kpXMNnZahMv8GwwgatTdc+Dp4Gowfd7GhM/NNI31lTfONJyDDrKcZP2Ea8aakikgCX1DNCXJSP1f5m3ITn7KdQHdOjVUqzKWU05mCEqNqZsQJheDFF+GRR6CqCsrKjF5HJ544bOXRpulohx9dDe+/BUJQiuBTqbN89FTO//ZdbCw237phZVe413ixmEjmlXpKVpKyHIWfHxc5rKhLSTtBBIJsrAghuGGsk6vXRE4gVYAsi+C8USOXRL1Gb4+bcBsJr4nYx95KLk/q2+KOCyO5S1vHHWJv8kXszy5XUfBosQNbNkip+IEK5CgqF2XGNtbiMeFYoxKpozpG6ChKQlRPD6Opp8G8a/s/5y4yfr6ppI2XNN9YpJS8zzbeZ6upBN1E8RPGTxhXdyXGA3LpoJBUCJ1FVLGNVn4g5+8eBszmzXD44bBjR+9jy5fD66/DmDHw3nswcWL07YeTcBguPQtWLTP+lnJnku70mo18eOcF7POzl6nPNncr6VT6GGLuHLA6jNBQLELDo++jS8nbei1v6bW0di+jxTg4Xi3lstJCFreHeKg6gNot/Q+GYrBDgRf2zjZfAp4CntWSqNcFxoj4JVeFwsGBopCFMn5PozCSt7RaLrCMiznuDHc2f+2I3GiyhyMcGWwOB9kcDqakzcEsq4Nf55dQqA5tGVVUOOe/8MgREPYNsJ0F5E+CkjnQtB5Uq5G7Ur/KCDcVTYf9vguzLzL2879E2nhJM3Q0Dd55HVqbYd4BkF8At/0fvP0a+H1G2e3RJ8LPboe8+BUEqeItNvdLtk01hhaFccX4WtbGzKWpw8O7bOEYdpFR0IPfP9hw6cv27bBgAaxbB/kj91nt5L03YfnSiE9ZdY08TxvXfvg4vzj1B3F3ZRVwdP4A70cc0TvACC+lGF1K7tU28uWATs71+Pm3toUq4eW+6eM4udDOvTt8LO8M41QFZxXbubrcyVjnyK1MjdLPVhMijgMpxE6pYq5e/BJ1PCFNZ3GUztZ9+Vw2cgHjYo45IyObRztbaYuSz+RA8IPcQraHQ1zdGD/nxgynZWRTZklNzlvpvnD1Mvj8z7DicUNsLnsczLvG8KhYTZbh/y+RNl7SDI1bfgDPPgFaDFe71wsvPQtvvAyvfwIVw7+Ad8gAH7BtWI8xkyIs3Z6Ud9gSd/xnVO164+XZZ6MbLj00NcE//wk33TQyc+rLS88a+jN65KXFInUuXvRCXONFAFeNdpBvG+DpslghFEtwTRhjEmCbT+OlhgBdYclkt8opRXbsSn8vyRLZMshw6cu7so79ZT4nFWVxUgSF4C4ZYo3sIIzOWOGmzISHI1k6ZeJhMwW4To2c/xUJm1C4Rp3EknD8Zo9dhNGljKouDJCpqDxUVM71jdXUDAj75QmFvxWOpkC1UKBauK9wNH9obWCjGUM2CgrQqafWm5s7Hk74m/GTJj5p4yVN8nznAuNO2SzBAJx3yi+KVAAAJ2FJREFUEnyxbvjm1M0K6pEpbyXXHwE7L6qtxK928JpMfBxWXn7Z3LjHHts1xktLU1TDpYc8T/vO/9sFBCSD7qT3y7Zw26T+CqeesKSubD/GVn6GJWrFkYRJ801NNaBLrlvTyaM1AaNpojASa/OsXdw1w4Erv5UGGcAlVDbonTHv9hWMDs6TlKx+j4elztNaJR/Ievqm804WmXxbnUiRSH0eTF6CEgDjcHO5Op4xSkZC222SXaa8HxIIoOGMs1yNtdp4cdQ4Fvq9fBkwErPn2J0c4nBj6WP47GN38lTxGDaFglSGgzzd1cbXQT8qxnfajOmmA+WWtFTCriRtvKRJjnWrEzNcemhqgC8+g/0PTP2c+tDZnRA5nAbMChrIZSMnMXnPaTfgNdF8B6CtbVinEZXysfDV4qgdr6UQiLLRPLNXFoU2wWSXymFftrHR27sMCuCL9jDHLW3njbk5ZFgEq7vCHLukjdGuY/lUfoZOhN4oQgFXDsw4fNBxK1t0nv5KY9FWY17zK1S+zPHySosh2ybpLWVuDelcsszLMXs3UZTjBxk/RKEDVbL/ZyO7Q01fyZZBZ/Em2clt4VX82jKbnBTrDeUIG7NFDqtkW9R52xD8SJ1GkXCS2+f4Phnmdb2Gz/UmgugUYudMdQzTlexB+/hQrzM1HxVMV06pQnCw083BzsgtBXoIIslSFBY43BzlymRjKMC97U0sCfgIx6lcEkC+orLAkY7l7Ep2g+zBNHskd/4m+W3feCl184hCNvZh97wAfMp2PDJIBTlxx45ieBvBmWLOHHPjJu0iDfFzLopquEB3I8ZvXcbpxXYOyrVx9Zoutvj6L7E9n/qX7WF+sK4TryY5bkk7TUHJUvd4LppwPWGhonULr4V7LoPuXLjwD0ZSbx/eXKtx4v0BHlscZl2DZF2D5MEVQV5qiZz4KbvN1GVb89Axn1vhGLBAb5JdLI1guNC9zy5CvKnXmNx7YpyrjsWGEnWB+JY6nilKdj/DpVLv4nvhJbyiV9NEgA5CbKaLP2hruDO8ZtA+aqW5nlJzyENNUQVcoxbm9tZ6Dq/ewgl12zi0ZjM/bKrhxa52PvJ745ZcKxjn4K25xf28OWlGnrTxkiY5qrYnv+0IfOn3otiUN2SoaEhW04DdxJ3hSewGTUWuvNLc+3/VVcM/l0jsOx9OPSfyHFUVps2Ecy8GjFyTVxuDOytzBqIBj9cGeKjKR11Q31lv9lzefCr2+hu3lp3DSznzeD5vP+6efT1c+wgUju23j63NOj9+MYim0+84HbnhmKpiEkF9m4tAyPwldv8BcvifycaYF2gd+ERvML3/RCgTLn5umcUU0T+MVYSda9XJHKL0r8ENS53btdWEorwpq2Q7j2n988JcJjWQ/CmqFKwLh/hW/XZe9HQQ6J6nBD7xe3i6TygyFnvZHNxfOJoD43h20gw/6bBRmuTIGuwGNs2Jp6duHlFwCxvHyYm8xsaktp9ADptpiztOAG+zmfY4ChInMpFJYhdU7wxk3Dj429/guuuijznuODj77BGbUj+EgD/+HcZVwEP3Qmd3GwerDU4/F/7vN+AyFo5PW6Mtlb2EJTxXHxgkk9FgzeYPpafu/Nsq4HuWwTqpTy2NbKRoqtypXhqLYFjBbo3te1GALKwcqPQv/26X8Ut6PWhxk1mTpUy4+KllBo3ST5MM4BIWxuCKKMj2kd5AIM5sP9IbuFCp2DnX/ZUCVmnxjYY1tNMpQ2QOUc36T22NtOraIFPIjGfMhuDZ4jGMtqbzXHYX0p6XNMlx5XeT266oxLi7HgEOE+PYl9KktjVjuICxfsUzXA5lLIcJ88Jqw8611xqaLuPH93/c6YSbb4aXXgLLLryvUVX43k+NxO5n3oCnXjX+/7u7ITMr/vYD2O6Pr6esyciOlE826xE9O9aAiGu4KIqO0zZgqezRb0Ggdu+gAAc3WWbgHOCJyBW2uBfoDCzDYrj0pVA4mKZkM1a4oyrJLtQb4+4njGSz7Nz59/4iH7cJj6UEWoYoL9eshfnQ70nahxNE0qZrLPZ7WR30o+0iRd80vaQ9L2mS4+gToHQ01FSZ38bhhP+8PnxzisCZTKOKDurpGpYMmHidgAGWUhu1jcAu4/jjDbG6ri5YvRrsdpg61dDk2V2wO2Du/lGfPjDHGvf9V4Ed/tj31gKYnakmZARkNltpLg8io2wikFQUd2JR5cAnOF4pRevOjJkmspktciIe+yBRyPvUR52DAhyq7B4SqtEbG/TH36fKyyZUjlZG8aIe/xriGmKrgx3h0JB1XS5p7J1nkWrhmqw8TnEPwQOdZkikPS9pkuf1T2HchMGPCwFz9jVCSxYrZGbD+ZfA52tgzLgRnWIXQaZSQAbD4+4VJvJqPCkVJU8xGRmw//5GIu/uZLiYoMKlcmKhjWjCsyowPUONu+xJ4HtjB1eOVPo0WsoDVM70sG2Wh7rxPnxu495d1QQF2+29O+iDQOKwacyu6C92J3UIdVk4JlTOBeo4zlfHMUfJjWo0VYgM5ou8iNaZAmRj42hRTIt3NZubnmFdwyNsbX6ZzkAlcoQ9AxOEuTLp8gEidocqRTG/QQKowE3hEEvCnSlWtm7QwvyqtYHHOltTut805kl7XtIkT2YWvPclfL0E/vV38HTBrDlw7Y92i4XwQ7mN17tzXobDsW5HRUfGbfI4EonD/6s8OCOTo5a0sapL26mj0vN732wLhTaFlV3xgwX7Z6lU+TXK7ApCCN5vDnLq1+0ErCC77d4um0ZXvo+8Kht5tTayG62oYYG/IkBbj4dFQlmOl32nNeCy9x5X1wAp2PjKOJqOh1wHhDUvHYGt6DKMy1qEyzZq53gpJQ1di9mvYyEh9yiWu0YT7tMderrI5mJKqW18HF+ogR4foJcaWn2ryHPNZGzuSYgRakdxhlLOe1p0LxHAGFyDyrpzhZ0jRQnvyuhl02eqY4Y8v0lWG6NUC7WxxDST4G/tTZzkyiR3iC0C0iSOkCNtoqeYjo4OsrOzaW9vJysr8Xh4mm8mX8tanmTVsB5jPqNpwcuGGG0BACrI4Voxb1jn8r+MT5M8Wevn39V+agM6Yxwql492cE6JnatWd/JUbcB0rsN0t8p3xzj58YYuvFr0kNSoDQ4cXSqaVVLklPz7Yjs3L/HxdmuIgEtHCElprpfp41opzPTTsiGHmkXFeBudvP9dK2HtfZo8XyH7BDOc1mLG5Z2C01pITfuH1HUu3PlcUKhUW3PQhEJxOMj8wgvY3voGnYFKos2yNOtQSrKGV0+pL29qNTytR+6JZEVwuzqHQmXwTY0mJU9r23hP1nXnQBtd312oXKpOYD8lNYnuL3s6+FVrbAMrGQoVlcuz8jjZlYVTSQczEmEo63faeEnzjUNKyR9ZSCMmBdmSQAC3ciidBPgzi2KOLcLFeHKZTzllYjfQevmm4+2ERa/BtjVUhhRutM7m5dIFhJX4d8c9OTQxc2kkKCHQLewMvBdZBQ2hARVIuvH/okobWY02FAGzSwW3n/wWrb7VEY+uCCtu62g6g7HaTQiyHBPo8G+K+VpUxcmsUd9DESPXF2mp3szTWiWNGC0YBIaX6HJlAvlK7M7QbTLIEr0FL2GKhIN9RB62FHuOHu1s5Z72JsD46CRGSf0Cu4vPA8lfLwQwwWLjgaLRZP+vdUgcAmnjJW28pOlDk/RyB58N+3H2oogddNJCfLEtpftu8jgmcuTuVHn0TUJKePFeePNh4//02hJb3cUcf/DtbM5Irvps8LEwF4vsvrqOW+HCGlT4+9ldZLv+mYIJmEkVh6lFl/ULR40UAanhRSMDC9bdoZN6H5q0MK97O6kOh8hWVI5zZeAUCifVbRvSflXgCGcGv88f+fd7T2Uo63c6UJfmG0coRaJW8ViOeYGwnryYN9lEsXQzU+weVSLfGKSEh39peFz60GNfjPE08PZHN7HX8f8koNoJDfWWzWwaU7eN4S0JcVG2k398qtEVuJDxBXUcOXkF4/LjlxhHxtwLkFF7OA0vdqGaEm7cFRSoFi7OzB30+Cybg9VBf9JVSRrwnq+LRi1MYToHZtjZvUziNGlSQB5OLLvpqS2AD4e52/X/JKsWDjJc+qIiGeNr5EHfIq4tH+FkcgF+l87/t3fn8U1WacPHf+dOmnRPW7pjy1opawHLJpsIQ9EZXx0d3ICR0QcE66u4jIo+is5nFMTtVUYF5hmVRxlhUD84ODLAI9vjWASK7JRlAGVrK0sXoGty3j9CA6VtmtCUNu319ZM/ct/nvnN6DMmVs1zn7zsc7DwRzuHTsazd353nvhrP0u39r/ym9ZVQZgIDoq/w/q3Pw+FtGryc2gHsKK9/k1bRcM3zE16IBrAqM+kkNss3twZ+pJDKS34RF+kyjuhCTnu414uoxZrFHhW748QGXk8NI+hqvjk0VLoS9TqDDod29kos+WEwm3/qWOeltVNEBHYh0BxN3UGMIjqkN6Z65pmIi66zBhHugyGu5vi50xJJ35ZokW6iMwc5zc+cvwrbM3rvW35isz7OaUq4NH1Wkg7nZlLorKLc3+DcWVj2OezZCRYLjMiAQUOvyr5RzdJBD1eWXfhVPDbeyl9PlFF5ld4cIYW1f9Qq5eAfu9JJT3Y3QbfaFRjKTIJtKArFvp8/odJRwuXDSMGWBBLDb2hQnVubHWWlFDVwmE0B8R5MDBcNJ60sWqRgFcDDuj9rOcwGjnGeiqauUjV17bl0hCLmk819Oo3udc2LWbMKHn3AGcBUpfH/4H3o3gv+a5FzC4bWxtPtDK5xbo75RPtg/pZbhkN7vuvzFdFgqlSEnaq9flob7MtvS6XdwGyquyZGWQW2A3lYykxEW3tgzZsLhafobovkdFp7cmOKsOsyLCYbMaF9aROShuHJxof7N8DGL+Dobmfg2zEdBtwBST2c84gqy8FsaRVBcU5FWYPvoYEpJ4/xaVwyCeaG7cUk3JPgRbRYQSqAm0ghQ3emjEq+IIft5NWbVK6paeAz9pCqozFd1o1t35aF8eC94HA4BwwqL0m6lbML7rsDlq1r2r2JmkKvofCvZc40tu4Mux2AbqFmvupr485tRZyu0AQoZ7v7pCfmknuYKhWJewMxHO6//B11Df9oTcK6PcR9tw+jsupvy7p4f8NEzLd2YtJHw/1/8O7/+5q/wHeLQBkX221/Fuz9FuJT4OfDYK8Aawj0HgMD74TQenoE/ZjVRwFasXbw+MnjfBrfrv7C4orJ8Jxo8QylCFIB6Av/eUoBnai5KuFqOEs5ezhZ7Vhx/lrK3vi/oB2o2jIc2O2wbw+sXXWVatmMjLyn/t6B2x6C2CTX0+FRFn4a3ob/7hnG9JgSvixZzj9P/Jn/99MCBhfnuJZbXwnLeUXsYSvttgdjLal71Y3CQXLkz1hMta+QS/xmF/Hrcy4JXC7juHDd5lWw9F3PK3hwszNwgeoBX9X9cvc7AxeAsnPw/Rfwl6lQ6Pskb83FjUEhPrvXvspy1pw/67P7iZokeBGtRls8zyNgoAgigLvoTg9iapyv+prsQzzxXNzXxeKjzQAUVMsfc+7URoqOfUXQhsMoh5svVZMJ/rnMBzXwM207w4OzwBRAjUmshgnGPgY331/jMquhuOfYCp5fPpnROxcy4th6Juf9D2tz/sDKvS8TXnkFicuUZvSQwwy+5QAJaScxAupeuq8xGNNtS63nAopLiMva5+H7ScOav0HpOc/quGkpeJW8TsPZAvj6LS+u8S9hJjNtfbjE+ZOzsu9RY2plfcuiNetHIis44NEOuLGEMJ6eRKogxuteZHGUb/mJUxcCivZEMIL2dFUxaK0poRIHmnzO8j7ZDa6rBsov5KvR2kFx7ipwaJS9nmERhwNa6y++PiNg1jL4din8ezugILUfDLkNguvYODDnf2HFn1xPDS7+ohtanMNf//0Ov+ryjBeVcO4mbYtw9lq0G3WM+H4/s+vja6ksMVEVWBnKgUMb3HjtdoZ12l3rnSJ3erFjO0BFGez7AXoNqb/ssT2gvc2H5ICD2XDmBES2zERsvwkO5+1i99t9eGqPLJluVBK8iFYjVFm4R/dkIdtd+6fAxVyliYTSlwSSiaA9NtSFYQiTMhhCMoN1EmXYMaEIuORXq1KKYJyT80J0JP1py0aONbi+33CQ63US5pI87BUFYDKobBOC6dS5un+NKwNSUhv82n7LFg2//A/PymoN//uJc7ipliEiMw4yiraTdv4w24Lbe3TLmPBS+ne5mHhOKbDaykm57SB7Pr2WUOt5FNA+Kp9fpG6jb9LBGqNdGsgNCMdREUSCYWCqL2C9lN3DiekNSWGff7DFBi+jQ3wXvNS1W7jwDQleRKuSpuKI0v1Zy4/s5mfsOIgnlCEkO3PDuPnAUUoRWM8/GaUUd+iuaDSbON6gulaiWcQOxjlsABhFpRjn61kRoR1w128b9LqtRvFJ5xexGw5l8JeAnXxwTSrvHa3vl7TmhrQTmE3VAyGlIOya84zs8wO/67nW7dScUmXmHxE9OGaJZFRMMX0dmzz8Yy5IutazcikDYes/8TRTbzVmS/1l/FS8OYCxITY+O1fYoGn9BtDXGuSraolaSPAiWp0kZWMCvQDnJo7Kh7+QHFpzgNNswzcTG3M4Ram1HaAIWbEHVVLpfg7ErWMh8RqfvHaL58HSWEMZpJX/yJvH3mHyzz/zkymCj4IG8WVgGvZqc0Y0nRKKsJhr7yVRCnoOPIhyM4VGA/+I6MHxgAgANqSnc/cXSzHsHgzvGCboNgCi29ZfFqDfr2Hrcs/KXspkdS6jbiobvoZvPoW8n8AcAF0HOCdrd/RdnZ6MiMEMLL4QwBg4U/97tpuUkwMYF9o0k/1bCwleRKvmy8ClRFfwIVs5RIHP7qmBXDNEh3cleO3f3JdVCnX6lM9eu8ULj4GAQKhw06PiqIQTP2AKDaK7dtCl8gQ3le1mfUBnbol6iPMXMthGhZWRnnKy7vsAZcp93o/TQR04Zrn4hXc2NJTPb/kVdy390v3fYRjO4bIJz7kvd6nYDtCpP/x7o+fXAAy8AyxN0KOwdzO8+0TNCcmbVjof46e7lsE3lFkpnoyMZWJ4FGtKznLW4SDJHMBLp3M93qf+EVsbBgQG+6Q+onay2kgIH/mUnRz2YeBSxYTC1vZWjOIyt70uSms47f4LtKWz2zX7cxx8s9zBlu8dnC1281s5wAq9b3LOE6qLUhBscS0nNl/47T244t+8V7yEtFAT07tpMvoerTFcdLnEigK353PDetX4QF7+i1F8dPddFIaFVTuuq+ashEfBTb+D/1wIkXFu71/DjR7ODarSvjcMa4IhyZxN8MaUOlZSaefjk5lw7IBPXzbaZGZsaAS/C49iVHAY8WZLvSu/EgwzH8UkcV9Yy82H01xIz4sQPpCnz9bIy+ILFkwkEY7Zaka364zes7v2HC8AJjMkd/B5HfyB1pqVyzR//QDOV/uO0wQEaPoPgdvuUrTvdNnXz7DfwqEtcOpI9XwnVYnbokJqzR9jQjOuZCPj+j5AuTWCqZUH617FpjVm7aBDed29YhrYxblas/2uHTqE9dcPovOhQwSVlpIXE0tc/LU8ZkptWObb2A7QLg1+3FZ/2SETYPhVDlwqK+DgDpj7VP1lFbD2Mxjnzcow79weYuONQve7gL8QFUdP61Xe+LOVkp4XIXxgDyd9kt/lcoNJwnohzbu69wGUux/39kq4a0Ij1KL5W/Kx5r/mXB64OFVUwL/WwNMPaf619rIGDAyF+96GQXdB4IXeDWVA22shNgyC3Az1aDvk52BRBmON5LrLKcUNxXvdvj92BiWSq+qe2+IwmdjXuTPbevQgNy6WbbqAMl9sbHDL792fVwo6D7y6gYvWsGoh/H4MvD4Zzhd7ds2+hqcocOe2kHCuDbDW+qWpgJFBofSTSbpXjQQvQviAHQfKx+FLEGYy6HTxwO33QJ/02pe5KgU33waDb/BpHfxBfq5mycf1l3M4YM4szcn8ywOYEBhxPzy2BB7/HJ5aBsPuBosHHdPlJXA2nzE6mnuNdgRc9h6waMWowj10K617ArcGtoR4n0reJ8GLLQ4G/Kb2c8pwJv27YWLDX8cbS9+DJW/BuULvrmvI8m8PBBkG82La8qvgsGpDFkFKcV9YJC9Hxft0Dp1wT4aNhPCBRMJ8vmfSIK6pvreR1Qr//QXM/gMs+QRKL2TgDbfBxAch88lWsYHe5dau9LzdtYb/+Vpz98Ra2skwQdCFLMwxHi45/teFlPymAEZ3Gs7wXrezNQiKdAVRykKqw8ze0tVub3HWsFJo8m6oIRgTob76+B45CSzBsGFx9RVYUW3hlichrlPd1/raqRPwz4+u7Nqeg31aldqEGSZmRMXzqC2GfRVlmBR0CwgkyJB+gKtNghchfKAL0URgpZAyn4QwJhSDSKp5IjgEXnwVnvxP2LvbuR1A1x7QisfZ83M9L+twwJ4dHhQMiYaweCj28Ob2Cti/GuvRLQy4+RUIuZDEzYDwwBSKSg9Q10Jbw/BuVYoBDFRtWOz4kYP6LBYMhhux9FNtruyXvzJg2ATnbtKHsp17GbVJhrZdr34wnPWPOpMGumWYYHgdPUiNIMJkor9JVhM1JQlehPABQykm6DTmkU0ljivuhVE491X6LWlEKDcBSWgYXDfgyirbwoSFe/d959H2NQfWeB64VNEOKCmA7E9g2COuw8kRo8nJP06l4zzVAxiFwkRa5K+wkUch9WfHNYAQzKzW+dVutcteSCgHed7UkzjjCuddWIMhdeiVXesrZ/IurP7yckgs8w1o0zKz/oraSV+XED6SrGxMYwD9SCTgwj+tcCyMoiPtsNV7vQnFEJL5PdfTTdXcDFLUbsgI5XHgohSkXVdPb4LWzgDkSmgHHM6Csov7S1nMNlLj7ic6pA9KVUVOBhFBqaTGTSTcmsRoI6HeGVNWDFKxUUxlrefPYucF+3ZKdO3n/UJohHe9LmFR8Mel0NOD/ZxEiyI9L0L4UIwK4Td04w7dFQfaNWdlhG7PUnJq3TJAAUEE8BDpxKk6NhAUderURdHves2m79yXUwoCA+HGMfXcsOBoteDDa9oOxXlgvWS3cVMYyZFjuCbiF9gdJZgMK8YlSevGGIkc1MVk6zPVMrkaOIOWyaYUumHj9/Yf3L50GQ6+sh9jrNn7CcDNwsCbYfmHbgooCIuAzn3g5t9Bu65Xq2aimZHgRYhGoJTCdMlvaYsycSfdGa07soqD7OYk56kgEDPpJDKUZPfDRMKtac8p3n9d8+2a2s8r5ZwWNP0VRVh4PX0cxz3Ie1KfgNqHbgxlwjDVDFBNSpFp6sImfYpvHLkc1yUEYWKgEc2NRjyRysIJXUKRB0NL63U+Y/HT4CWhAwz+P/Ddspo9MMpwJhZ8cr6znGjVJHgR4iqKUEGMpXtTV6PFsVgUjz6rGPcfmjUrNHt2QMEZsNshPBz6DlTcOAZsEZ5MQG3gaLotEcK9n39hKMUAFc0AI7rW8+UezgMp9cUS6qY0/lkICoU1f3PmLqoSlwz3vySBiwAkeBFCtCDRsYqxExq4QsYW37Dre9/ZKKt0YrF6tDlguL9/rJvMcOfjcPP9sPM7KCuBxE7QOa1VpgIQtfPzd7kQQvhYYhoE2qDUTZK0gGCoOO8cylDKuQbbMKDffdD++kapVpAy0w0bu3CfvC1DtZBVN6ERzjkwQtRCghchhLiUYYLBU+GbV6m1n8NkgTEvOcsd/g7Kz0FoHHQcenGLgUbyoKkzT9p/qHMIKRoLw01ebtAohB9SWnubDah5KSoqwmazUVhYSHh4eFNXRwjRUpzYCVv+Cicv2a04MQ3SJ0Ckm72MGlmBo4w37Tn8xPlqx7tjY6o5hVDlZj8mIZqRhnx/S/AihBDuFOdDaRGEREFwVFPXxqXIUc5uXYiBQYoRRqSyNHWVhPBKQ76/ZdhICCHcCYt1PpqZcMPCQCSZoWidJMOuEEIIIfyKBC9CCCGE8CsSvAghhBDCr0jwIoQQQgi/IsGLEEIIIfyKBC9CCCGE8CuNHryUlZXRu3dvlFJs3bq12rnt27czdOhQAgMDSUpKYvbs2Y1dHSGEEEL4uUYPXp566ikSExNrHC8qKmL06NG0a9eO7OxsXnvtNV588UXmz5/f2FUSQgghhB9r1CR1y5cvZ+XKlXz++ecsX7682rmFCxdSXl7OBx98gMVioXv37mzdupU333yTyZMnN2a1hBBCCOHHGi14ycvLY9KkSSxdupTg4OAa57Oyshg2bBgWy8WU1hkZGbz66qucOXOGyMjIWu9bVlZGWVmZ63lhoXOH1aKiIh//BUIIIYRoLFXf21eyS1GjBC9aayZOnMiUKVNIT0/n8OHDNcrk5ubSoUOHasfi4uJc5+oKXmbOnMlLL71U43hSUlLDKy6EEEKIq6q4uBibzebVNV4FL8888wyvvvqq2zJ79uxh5cqVFBcXM336dK8q44np06fz+OOPu547HA5Onz5NmzZtUEr5/PX8TVFREUlJSRw5ckQ2qvQBaU/fkzb1PWlT35M29a3a2lNrTXFxca3zYuvjVfDyxBNPMHHiRLdlOnbsyOrVq8nKysJqtVY7l56ezrhx41iwYAHx8fHk5eVVO1/1PD4+vs77W63WGveNiIjw/I9oJcLDw+UfnA9Je/qetKnvSZv6nrSpb13ent72uFTxKniJiYkhJqb+XUzfeecd/vjHP7qeHz9+nIyMDBYvXsyAAQMAGDRoEM899xwVFRUEBAQAsGrVKrp06VLnkJEQQgghRKPMeUlOTq72PDQ0FIBOnTpxzTXXAHDvvffy0ksv8cADD/D000+zc+dO3n77bd56663GqJIQQgghWohGXSrtjs1mY+XKlWRmZnLdddcRHR3NCy+8IMukG8hqtTJjxowaQ2viykh7+p60qe9Jm/qetKlv+bo9lb6SNUpCCCGEEE1E9jYSQgghhF+R4EUIIYQQfkWCFyGEEEL4FQlehBBCCOFXJHhpgcrKyujduzdKKbZu3Vrt3Pbt2xk6dCiBgYEkJSUxe/bspqlkM3f48GEeeOABOnToQFBQEJ06dWLGjBmUl5dXKyft6b13332X9u3bExgYyIABA9i4cWNTV8kvzJw5k379+hEWFkZsbCy33XYbe/furVamtLSUzMxM2rRpQ2hoKHfccUeNZKCibrNmzUIpxbRp01zHpE29d+zYMcaPH0+bNm0ICgqiZ8+ebN682XVea80LL7xAQkICQUFBjBo1iv3793v1GhK8tEBPPfVUremWi4qKGD16NO3atSM7O5vXXnuNF198kfnz5zdBLZu3nJwcHA4H8+bNY9euXbz11lvMnTuXZ5991lVG2tN7ixcv5vHHH2fGjBls2bKFtLQ0MjIyyM/Pb+qqNXvr1q0jMzOTDRs2sGrVKioqKhg9ejTnzp1zlXnsscdYtmwZS5YsYd26dRw/fpzbb7+9CWvtPzZt2sS8efPo1atXtePSpt45c+YMgwcPJiAggOXLl7N7927eeOONaslnZ8+ezTvvvMPcuXP5/vvvCQkJISMjg9LSUs9fSIsW5euvv9apqal6165dGtA//PCD69x7772nIyMjdVlZmevY008/rbt06dIENfU/s2fP1h06dHA9l/b0Xv/+/XVmZqbrud1u14mJiXrmzJlNWCv/lJ+frwG9bt06rbXWBQUFOiAgQC9ZssRVZs+ePRrQWVlZTVVNv1BcXKxTUlL0qlWr9PDhw/Wjjz6qtZY2vRJPP/20HjJkSJ3nHQ6Hjo+P16+99prrWEFBgbZarfrTTz/1+HWk56UFycvLY9KkSXz88ccEBwfXOJ+VlcWwYcOwWCyuYxkZGezdu5czZ85czar6pcLCQqKiolzPpT29U15eTnZ2NqNGjXIdMwyDUaNGkZWV1YQ180+FhYUArvdkdnY2FRUV1do3NTWV5ORkad96ZGZm8stf/rJa24G06ZX4+9//Tnp6OmPHjiU2NpY+ffrw5z//2XX+0KFD5ObmVmtTm83GgAEDvGpTCV5aCK01EydOZMqUKaSnp9daJjc3l7i4uGrHqp7n5uY2eh392YEDB5gzZw4PPvig65i0p3dOnjyJ3W6vtc2kvbzjcDiYNm0agwcPpkePHoDzPWexWGpsVCvt696iRYvYsmULM2fOrHFO2tR7Bw8e5P333yclJYUVK1YwdepUHnnkERYsWABc/Gxs6OeABC/N3DPPPINSyu0jJyeHOXPmUFxczPTp05u6ys2ap+15qWPHjjFmzBjGjh3LpEmTmqjmQlyUmZnJzp07WbRoUVNXxa8dOXKERx99lIULFxIYGNjU1WkRHA4Hffv25ZVXXqFPnz5MnjyZSZMmMXfuXJ++TpPtbSQ888QTTzBx4kS3ZTp27Mjq1avJysqqsW9Eeno648aNY8GCBcTHx9eYJV/1PD4+3qf1bq48bc8qx48fZ8SIEVx//fU1JuJKe3onOjoak8lUa5tJe3nu4Ycf5quvvmL9+vWujW7B+Z4rLy+noKCgWk+BtG/dsrOzyc/Pp2/fvq5jdrud9evX86c//YkVK1ZIm3opISGBbt26VTvWtWtXPv/8c+DiZ2NeXh4JCQmuMnl5efTu3dvzF2rIxBzRfPz44496x44drseKFSs0oD/77DN95MgRrfXFCabl5eWu66ZPny4TTOtw9OhRnZKSou+++25dWVlZ47y0p/f69++vH374Yddzu92u27ZtKxN2PeBwOHRmZqZOTEzU+/btq3G+anLpZ5995jqWk5Mjk0vdKCoqqva5uWPHDp2enq7Hjx+vd+zYIW16Be65554aE3anTZumBw0apLW+OGH39ddfd50vLCz0esKuBC8t1KFDh2qsNiooKNBxcXF6woQJeufOnXrRokU6ODhYz5s3r+kq2kwdPXpUd+7cWY8cOVIfPXpUnzhxwvWoIu3pvUWLFmmr1ao/+ugjvXv3bj158mQdERGhc3Nzm7pqzd7UqVO1zWbTa9eurfZ+PH/+vKvMlClTdHJysl69erXevHmzHjRokOtLQ3jm0tVGWkubemvjxo3abDbrl19+We/fv18vXLhQBwcH608++cRVZtasWToiIkJ/+eWXevv27frWW2/VHTp00CUlJR6/jgQvLVRtwYvWWm/btk0PGTJEW61W3bZtWz1r1qymqWAz9+GHH2qg1selpD29N2fOHJ2cnKwtFovu37+/3rBhQ1NXyS/U9X788MMPXWVKSkr0Qw89pCMjI3VwcLD+9a9/XS3gFvW7PHiRNvXesmXLdI8ePbTVatWpqal6/vz51c47HA79/PPP67i4OG21WvXIkSP13r17vXoNpbXW3o5pCSGEEEI0FVltJIQQQgi/IsGLEEIIIfyKBC9CCCGE8CsSvAghhBDCr0jwIoQQQgi/IsGLEEIIIfyKBC9CCCGE8CsSvAghhBDCr0jwIoQQQgi/IsGLEEIIIfyKBC9CCCGE8CsSvAghhBDCr/x/FtV4i9z+YloAAAAASUVORK5CYII=\",\n      \"text/plain\": [\n       \"<Figure size 640x480 with 1 Axes>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"import sklearn\\n\",\n    \"from sklearn.manifold import TSNE\\n\",\n    \"from matplotlib import pyplot as plt\\n\",\n    \"from matplotlib.cm import rainbow\\n\",\n    \"\\n\",\n    \"# This is a sanity check that our data clusters nicely\\n\",\n    \"# map our feature space to 2D, plot it and color it by its labels\\n\",\n    \"\\n\",\n    \"reduced = sklearn.manifold.TSNE(n_components=2, random_state=0).fit_transform(X)\\n\",\n    \"plt.scatter(reduced[:, 0], reduced[:, 1], c=data_class, cmap='rainbow')\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"d3a6b699-b006-4ea5-8bcd-91dd69405b1b\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Create a train / test split\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"cbdd2624-0f99-450a-93e5-55c009c7e535\",\n   \"metadata\": {},\n   \"source\": [\n    \"For the sake of demonstration, we will shuffle the dataset and split it into two train and test splits composed of 80% and 20% of the data.\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"id\": \"21a9b3f4-a564-49f4-ba8a-fde55c636fc3\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Train set : 960 samples\\n\",\n      \"Test set  : 240 samples\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# make things reproducible\\n\",\n    \"rng = np.random.default_rng(seed=0)\\n\",\n    \"\\n\",\n    \"# shuffle the data\\n\",\n    \"permuted = rng.permutation(len(X))\\n\",\n    \"shuffled_x, shuffled_class = X[permuted], np.array(data_class)[permuted]\\n\",\n    \"\\n\",\n    \"# create a train / test split\\n\",\n    \"train_prop = 0.8\\n\",\n    \"n_train = int(len(shuffled_x) * 0.8)\\n\",\n    \"train_x, train_y = shuffled_x[:n_train], shuffled_class[:n_train]\\n\",\n    \"test_x, test_y = shuffled_x[n_train:], shuffled_class[n_train:]\\n\",\n    \"\\n\",\n    \"# summary\\n\",\n    \"print(f\\\"Train set : {len(train_x)} samples\\\")\\n\",\n    \"print(f\\\"Test set  : {len(test_x)} samples\\\")\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"e25720eb-e499-43b6-a19a-b82b100af32b\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Normalize features\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"5ecee2d7-5f23-49df-8c6a-066c38ef01bf\",\n   \"metadata\": {},\n   \"source\": [\n    \"It is usually recommended to normalize features to improve the performance and stability of the classifier.\\n\",\n    \"In particular, we ensure that each feature in the training set has a mean of 0 and a standard deviation of 1.\\n\",\n    \"This can be done with [`StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) of Scikit-learn:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"id\": \"a28bbe9b-0be4-4e68-b9ed-977fb20f29e3\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"from sklearn.preprocessing import StandardScaler\\n\",\n    \"\\n\",\n    \"scaler = StandardScaler()\\n\",\n    \"train_x = scaler.fit_transform(train_x)\\n\",\n    \"test_x = scaler.transform(test_x)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"91116a77-b188-4c1c-95ac-e6e8872308fb\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Train a classifier and compute the test accuracy\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"ad69a0bd-e57d-45e6-80da-2a49a64fe9a3\",\n   \"metadata\": {},\n   \"source\": [\n    \"We can now train the classifier. We will use the [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) object of Scikit-learn.\\n\",\n    \"\\n\",\n    \"The default learning algorithm is `lbfgs`, which should give an accuracy of 98.33% for this particular train/test split.\\n\",\n    \"\\n\",\n    \"You can try different algorithms, and hyper-parameters. In a real-life scenario, it is possible to obtain better results by using k-fold cross-validation methods to validate accuracy and choice of the hyperparameters.\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"id\": \"b496a585-018b-471e-8b26-4bd1fc0cd0b0\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Precision: 98.33%\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"from sklearn.linear_model import LogisticRegression\\n\",\n    \"\\n\",\n    \"# For a real problem, C should be properly cross validated and the confusion matrix analyzed\\n\",\n    \"clf = LogisticRegression(random_state=0, C=1.0, max_iter=500).fit(train_x, train_y)  # 98.33%\\n\",\n    \"\\n\",\n    \"# you can also try the sag algorithm:\\n\",\n    \"# clf = LogisticRegression(random_state=0, C=1.0, max_iter=1000, solver='sag').fit(train_x, train_y)\\n\",\n    \"\\n\",\n    \"print(f\\\"Precision: {100*np.mean(clf.predict(test_x) == test_y):.2f}%\\\")\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"db9a2ab7-c354-4273-8f07-76687b10f7e2\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Classify a single example\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"62c0c8ed-e56a-46fb-9754-b48bba907914\",\n   \"metadata\": {},\n   \"source\": [\n    \"Below is an example showing how to classify new samples, and how to print the probabilities of the top-5 predicted labels.\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"id\": \"c5e2cb04-2326-46d9-b080-20b464d6ad08\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"@torch.no_grad()\\n\",\n    \"def classify(s: str):\\n\",\n    \"    tokens = tokenizer.encode(s, bos=True)\\n\",\n    \"    tensor = torch.tensor(tokens).to(model.device)\\n\",\n    \"    features = model.forward_partial(tensor, [len(tokens)])  # (n_tokens, model_dim)\\n\",\n    \"    embedding = features.float().mean(0).cpu().detach().numpy()\\n\",\n    \"    probas = clf.predict_proba(embedding[None])\\n\",\n    \"    assert probas.shape == (1, len(labels))\\n\",\n    \"    return probas[0]\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"id\": \"41246dbf-ea83-4140-bb57-510c958c26df\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \" 99.65%  Migraine\\n\",\n      \"  0.20%  Dengue\\n\",\n      \"  0.08%  Cervical spondylosis\\n\",\n      \"  0.05%  Jaundice\\n\",\n      \"  0.02%  allergy\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"sentence = \\\"I have been feeling excessively hungry, even after eating, and have had a stiff neck.\\\"\\n\",\n    \"probas = classify(sentence)\\n\",\n    \"\\n\",\n    \"for i in np.argsort(probas)[::-1][:5]:\\n\",\n    \"    print(f\\\"{100*probas[i]:6.2f}%  {labels[i]}\\\")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"51284fe5-da9b-4176-9468-72d5041e0ae4\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"52e440c1-d377-4580-bd6b-84ee1e5686f1\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Second implementation: Zero shot\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"a454b5ae-f68e-4f0d-9274-7f401420a882\",\n   \"metadata\": {},\n   \"source\": [\n    \"Below is another method to do classification which does not require any training set.\\n\",\n    \"It also does not require to train a classifier on top of Mistral.\\n\",\n    \"However, it is much slower to predict, and it is not expected to work as well as the previous method in the case where you have a training dataset available.\\n\",\n    \"\\n\",\n    \"The method consists in prompting the model with the following text:\\n\",\n    \"\\n\",\n    \"```\\n\",\n    \"Symptoms: {symptom}\\n\",\n    \"Disease:\\n\",\n    \"```\\n\",\n    \"\\n\",\n    \"and to evaluate the probability that the model predicts each of the possible labels.\\n\",\n    \"The method is quite slower, because you need to compute the probability of all labels.\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 12,\n   \"id\": \"12546a21-6b0c-4df2-800b-43cb0b753e91\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"240it [00:14, 16.82it/s]\"\n     ]\n    },\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Accuracy: 30.42%\\n\"\n     ]\n    },\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"model.args.max_batch_size = len(labels)\\n\",\n    \"\\n\",\n    \"test_data = [data[i] for i in permuted[n_train:]]\\n\",\n    \"\\n\",\n    \"good = []\\n\",\n    \"\\n\",\n    \"with torch.no_grad():\\n\",\n    \"\\n\",\n    \"    for i, (symptom, disease) in tqdm.tqdm(enumerate(test_data)):\\n\",\n    \"    \\n\",\n    \"        # given an input, predict the probability of each output (one output corresponds to one label)\\n\",\n    \"        input_ids, seq_lengths, output_lengths = [], [], []\\n\",\n    \"        for label in labels:\\n\",\n    \"            prefix = f\\\"Symptoms: {symptom}\\\\nDisease:\\\"\\n\",\n    \"            tokens = tokenizer.encode(f\\\"{prefix} {label}\\\", bos=True)\\n\",\n    \"            input_ids.extend(tokens)\\n\",\n    \"            seq_lengths.append(len(tokens))\\n\",\n    \"            output_lengths.append(len(tokens) - len(tokenizer.encode(prefix, bos=True)))\\n\",\n    \"        tensor = torch.tensor(input_ids).to(model.device)\\n\",\n    \"        logprobs = torch.log_softmax(model.forward(tensor, seq_lengths), dim=-1)  # sum_slens * vocab_size\\n\",\n    \"        \\n\",\n    \"        # select logprobs for each output\\n\",\n    \"        offset = 0\\n\",\n    \"        avg_logprobs = []\\n\",\n    \"        for slen, output_len in zip(seq_lengths, output_lengths):\\n\",\n    \"            output_tokens = input_ids[offset + slen - output_len:offset + slen]\\n\",\n    \"            output_logprobs = torch.gather(\\n\",\n    \"                logprobs[offset + slen - output_len - 1:offset + slen - 1],\\n\",\n    \"                dim=1,\\n\",\n    \"                index=torch.tensor(output_tokens).to(model.device)[:, None],\\n\",\n    \"            ).mean().item()\\n\",\n    \"            avg_logprobs.append(output_logprobs)\\n\",\n    \"            offset += slen\\n\",\n    \"\\n\",\n    \"        good.append(np.argmax(avg_logprobs) == txt_to_label[disease])\\n\",\n    \"    \\n\",\n    \"print(f\\\"Accuracy: {100 * np.mean(good):.2f}%\\\")\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"cc4157a9-236d-4ef5-a02d-0731ae8e89eb\",\n   \"metadata\": {},\n   \"source\": [\n    \"Even though this approach did not require any training set, the performance is only of 30.4%, compared to 98.3% for the previous, faster one.\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"7a42ece8-c879-414c-a21a-25843f7d2bd1\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.18\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
  },
  {
    "path": "tutorials/getting_started.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Getting Started with `mistral-inference`\\n\",\n    \"\\n\",\n    \"This notebook will guide you through the process of running Mistral models locally. We will cover the following: \\n\",\n    \"- How to chat with Mistral 7B Instruct\\n\",\n    \"- How to run Mistral 7B Instruct with function calling capabilities\\n\",\n    \"\\n\",\n    \"We recommend using a GPU such as the A100 to run this notebook. \"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"G6tXvIsQenpI\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install mistral-inference\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Download Mistral 7B Instruct\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"colab\": {\n     \"background_save\": true\n    },\n    \"id\": \"4ytmRt0WQeMW\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!wget https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-Instruct-v0.3.tar\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"eRZg_8wvs5A6\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!DIR=$HOME/mistral_7b_instruct_v3 && mkdir -p $DIR && tar -xf mistral-7B-Instruct-v0.3.tar -C $DIR\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"7CN8gShDf65M\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!ls mistral_7b_instruct_v3\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## Chat with the model\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"import os \\n\",\n    \"\\n\",\n    \"from mistral_inference.transformer import Transformer\\n\",\n    \"from mistral_inference.generate import generate\\n\",\n    \"\\n\",\n    \"from mistral_common.tokens.tokenizers.mistral import MistralTokenizer\\n\",\n    \"from mistral_common.protocol.instruct.messages import UserMessage\\n\",\n    \"from mistral_common.protocol.instruct.request import ChatCompletionRequest\\n\",\n    \"\\n\",\n    \"# load tokenizer\\n\",\n    \"mistral_tokenizer = MistralTokenizer.from_file(os.path.expanduser(\\\"~\\\")+\\\"/mistral_7b_instruct_v3/tokenizer.model.v3\\\")\\n\",\n    \"# chat completion request\\n\",\n    \"completion_request = ChatCompletionRequest(messages=[UserMessage(content=\\\"Explain Machine Learning to me in a nutshell.\\\")])\\n\",\n    \"# encode message\\n\",\n    \"tokens = mistral_tokenizer.encode_chat_completion(completion_request).tokens\\n\",\n    \"# load model\\n\",\n    \"model = Transformer.from_folder(os.path.expanduser(\\\"~\\\")+\\\"/mistral_7b_instruct_v3\\\")\\n\",\n    \"# generate results\\n\",\n    \"out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=mistral_tokenizer.instruct_tokenizer.tokenizer.eos_id)\\n\",\n    \"# decode generated tokens\\n\",\n    \"result = mistral_tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])\\n\",\n    \"print(result)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"ce4woS3LkgZ9\"\n   },\n   \"source\": [\n    \"## Function calling\\n\",\n    \"\\n\",\n    \"Mistral 7B Instruct v3 also supports function calling!\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"TKfPiEwNk1kh\"\n   },\n   \"source\": [\n    \"Let's start by creating a function calling example\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"0PJdwvDEk3dl\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from mistral_common.protocol.instruct.messages import UserMessage\\n\",\n    \"from mistral_common.protocol.instruct.request import ChatCompletionRequest\\n\",\n    \"from mistral_common.protocol.instruct.tool_calls import Function, Tool\\n\",\n    \"\\n\",\n    \"completion_request = ChatCompletionRequest(\\n\",\n    \"    tools=[\\n\",\n    \"        Tool(\\n\",\n    \"            function=Function(\\n\",\n    \"                name=\\\"get_current_weather\\\",\\n\",\n    \"                description=\\\"Get the current weather\\\",\\n\",\n    \"                parameters={\\n\",\n    \"                    \\\"type\\\": \\\"object\\\",\\n\",\n    \"                    \\\"properties\\\": {\\n\",\n    \"                        \\\"location\\\": {\\n\",\n    \"                            \\\"type\\\": \\\"string\\\",\\n\",\n    \"                            \\\"description\\\": \\\"The city and state, e.g. San Francisco, CA\\\",\\n\",\n    \"                        },\\n\",\n    \"                        \\\"format\\\": {\\n\",\n    \"                            \\\"type\\\": \\\"string\\\",\\n\",\n    \"                            \\\"enum\\\": [\\\"celsius\\\", \\\"fahrenheit\\\"],\\n\",\n    \"                            \\\"description\\\": \\\"The temperature unit to use. Infer this from the users location.\\\",\\n\",\n    \"                        },\\n\",\n    \"                    },\\n\",\n    \"                    \\\"required\\\": [\\\"location\\\", \\\"format\\\"],\\n\",\n    \"                },\\n\",\n    \"            )\\n\",\n    \"        )\\n\",\n    \"    ],\\n\",\n    \"    messages=[\\n\",\n    \"        UserMessage(content=\\\"What's the weather like today in Paris?\\\"),\\n\",\n    \"        ],\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"bG6ZeZUylpBW\"\n   },\n   \"source\": [\n    \"Since we have already loaded the tokenizer and the model in the example above. We will skip these steps here. \\n\",\n    \"\\n\",\n    \"Now we can encode the message with our tokenizer using `MistralTokenizer`. \"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"Ii8q-JNClwiq\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from mistral_common.tokens.tokenizers.mistral import MistralTokenizer\\n\",\n    \"\\n\",\n    \"tokens = mistral_tokenizer.encode_chat_completion(completion_request).tokens\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"NrueDujkmJT4\"\n   },\n   \"source\": [\n    \"and run `generate` to get a response. Don't forget to pass the EOS id!\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"GWJYO43rl0V8\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from mistral_inference.generate import generate\\n\",\n    \"\\n\",\n    \"out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=mistral_tokenizer.instruct_tokenizer.tokenizer.eos_id)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"v7baJ1msmPMv\"\n   },\n   \"source\": [\n    \"Finally, we can decode the generated tokens.\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"RKhryfBWmHon\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"result = mistral_tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens)[0]\\n\",\n    \"result\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"gpuType\": \"L4\",\n   \"machine_shape\": \"hm\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.11.8\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n"
  }
]