[ { "path": ".github/ISSUE_TEMPLATE/bug-report.yml", "content": "name: \"\\U0001F41B Bug Report\"\ndescription: Submit a bug report to help us improve TRL\nlabels: [ \"bug\" ]\nbody:\n - type: markdown\n attributes:\n value: |\n Thanks for taking the time to fill out this bug report! ๐Ÿค—\n\n ๐Ÿšฉ If it is your first time submitting, be sure to check our [bug report guidelines](https://github.com/huggingface/trl/blob/main/CONTRIBUTING.md#did-you-find-a-bug)\n\n - type: textarea\n id: reproduction\n validations:\n required: true\n attributes:\n label: Reproduction\n description: |\n Please provide a code sample that reproduces the problem you ran into. It can be a Colab link or just a code snippet.\n If you have code snippets, error messages, stack traces please provide them here as well.\n Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting\n Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.\n\n value: |\n ```python\n from trl import ...\n\n ```\n\n outputs:\n\n ```\n Traceback (most recent call last):\n File \"example.py\", line 42, in \n ...\n ```\n\n - type: textarea\n id: system-info\n attributes:\n label: System Info\n description: |\n Please provide information about your system: platform, Python version, PyTorch version, Transformers version, devices, TRL version, ...\n You can get this information by running `trl env` in your terminal.\n\n placeholder: Copy-paste the output of `trl env`\n validations:\n required: true\n\n - type: checkboxes\n id: terms\n attributes:\n label: Checklist\n description: |\n Before submitting, please confirm that you've completed each of the following.\n If an item doesn't apply to your issue, check it anyway to show you've reviewed it.\n options:\n - label: \"I have checked that my issue isn't already filed (see [open issues](https://github.com/huggingface/trl/issues?q=is%3Aissue))\"\n required: true\n - label: \"I have included my system information\"\n required: true\n - label: \"Any code provided is minimal, complete, and reproducible ([more on MREs](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))\"\n required: true\n - label: \"Any code provided is properly formatted in code blocks, (no screenshot, [more on code blocks](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))\"\n required: true\n - label: \"Any traceback provided is complete\"\n required: true\n" }, { "path": ".github/ISSUE_TEMPLATE/feature-request.yml", "content": "name: \"\\U0001F680 Feature request\"\ndescription: Submit a proposal/request for a new TRL feature\nlabels: [ \"Feature request\" ]\nbody:\n - type: textarea\n id: feature-request\n validations:\n required: true\n attributes:\n label: Feature request\n description: |\n A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist.\n\n - type: textarea\n id: motivation\n validations:\n required: true\n attributes:\n label: Motivation\n description: |\n Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too.\n\n\n - type: textarea\n id: contribution\n validations:\n required: true\n attributes:\n label: Your contribution\n description: |\n Is there any way that you could help, e.g. by submitting a PR? Make sure to read the CONTRIBUTING.MD [readme](https://github.com/huggingface/trl/blob/main/CONTRIBUTING.md)\n" }, { "path": ".github/ISSUE_TEMPLATE/new-trainer-addition.yml", "content": "name: \"\\U0001F31F New trainer addition\"\ndescription: Submit a proposal/request to implement a new trainer for a post-training method \nlabels: [ \"New trainer\" ]\n\nbody:\n - type: textarea\n id: description-request\n validations:\n required: true\n attributes:\n label: Method description\n description: |\n Put any and all important information relative to the method\n\n - type: checkboxes\n id: information-tasks\n attributes:\n label: Open source status\n description: |\n Please note that if the method implementation isn't available or model weights with training datasets aren't available, we are less likely to implement it in `trl`.\n options:\n - label: \"The method implementation is available\"\n - label: \"The model weights are available\"\n - label: \"The training datasets are available\"\n\n - type: textarea\n id: additional-info\n attributes:\n label: Provide useful links for the implementation\n description: |\n Please provide information regarding the implementation, the weights, and the authors.\n Please mention the authors by @gh-username if you're aware of their usernames.\n" }, { "path": ".github/PULL_REQUEST_TEMPLATE.md", "content": "# What does this PR do?\n\n\n\n\n\nFixes # (issue)\n\n\n## Before submitting\n- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).\n- [ ] Did you read the [contributor guideline](https://github.com/huggingface/trl/blob/main/CONTRIBUTING.md#create-a-pull-request),\n Pull Request section?\n- [ ] Was this discussed/approved via a GitHub issue? Please add a link\n to it if that's the case.\n- [ ] Did you make sure to update the documentation with your changes?\n- [ ] Did you write any new necessary tests?\n\n\n## Who can review?\n\nAnyone in the community is free to review the PR once the tests have passed. Feel free to tag\nmembers/contributors who may be interested in your PR." }, { "path": ".github/codeql/custom-queries.qls", "content": "import codeql\n\nfrom WorkflowString interpolation, Workflow workflow\nwhere \n interpolation.getStringValue().matches(\"${{ github.event.issue.title }}\") or\n interpolation.getStringValue().matches(\"${{ github.event.issue.body }}\") or\n interpolation.getStringValue().matches(\"${{ github.event.pull_request.title }}\") or\n interpolation.getStringValue().matches(\"${{ github.event.pull_request.body }}\") or\n interpolation.getStringValue().matches(\"${{ github.event.review.body }}\") or\n interpolation.getStringValue().matches(\"${{ github.event.comment.body }}\") or\n interpolation.getStringValue().matches(\"${{ github.event.inputs.* }}\") or\n interpolation.getStringValue().matches(\"${{ github.event.head_commit.message }}\")\n interpolation.getStringValue().matches(\"${{ github.event.* }}\") and\n (\n step.getKey() = \"run\" or // Injection in run\n step.getKey() = \"env\" or // Injection via env\n step.getKey() = \"with\" // Injection via with\n )\nselect workflow, \"๐Ÿšจ Do not use directly as input of action\"\n" }, { "path": ".github/workflows/build_documentation.yml", "content": "name: Build documentation\n\non:\n push:\n branches:\n - main\n - doc-builder*\n - v*-release\n\nenv:\n TRL_EXPERIMENTAL_SILENCE: 1\n\njobs:\n build:\n uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@main\n with:\n commit_sha: ${{ github.sha }}\n package: trl\n version_tag_suffix: \"\"\n secrets:\n hf_token: ${{ secrets.HF_DOC_BUILD_PUSH }}\n" }, { "path": ".github/workflows/build_pr_documentation.yml", "content": "name: Build PR Documentation\n\non:\n pull_request:\n\nenv:\n TRL_EXPERIMENTAL_SILENCE: 1\n\nconcurrency:\n group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}\n cancel-in-progress: true\n\njobs:\n build:\n if: github.event.pull_request.draft == false\n uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@main\n with:\n commit_sha: ${{ github.event.pull_request.head.sha }}\n pr_number: ${{ github.event.number }}\n package: trl\n version_tag_suffix: \"\"\n" }, { "path": ".github/workflows/clear_cache.yml", "content": "name: \"Cleanup Cache\"\n\non:\n workflow_dispatch:\n schedule:\n - cron: \"0 0 * * *\"\n \njobs:\n cleanup:\n runs-on: ubuntu-latest\n steps:\n - name: Check out code\n uses: actions/checkout@v6\n \n - name: Cleanup\n run: |\n gh extension install actions/gh-actions-cache\n \n REPO=${{ github.repository }}\n\n echo \"Fetching list of cache key\"\n cacheKeysForPR=$(gh actions-cache list -R $REPO | cut -f 1 )\n\n ## Setting this to not fail the workflow while deleting cache keys. \n set +e\n echo \"Deleting caches...\"\n for cacheKey in $cacheKeysForPR\n do\n gh actions-cache delete $cacheKey -R $REPO --confirm\n done\n echo \"Done\"\n env:\n GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n" }, { "path": ".github/workflows/codeQL.yml", "content": "name: \"CodeQL Analysis - Workflows\"\n\non:\n workflow_dispatch:\n\njobs:\n analyze:\n name: \"Analyze GitHub Workflows\"\n runs-on: ubuntu-latest\n permissions:\n security-events: write\n actions: read\n contents: read\n\n steps:\n - name: \"Checkout repository\"\n uses: actions/checkout@v6\n\n - name: \"Initialize CodeQL\"\n uses: github/codeql-action/init@v2\n with:\n languages: \"yaml\"\n queries: +security-and-quality, ./.github/codeql/custom-queries.qls\n\n - name: \"Perform CodeQL Analysis\"\n uses: github/codeql-action/analyze@v2\n" }, { "path": ".github/workflows/docker-build.yml", "content": "name: Build TRL Docker image\n\non:\n push:\n branches:\n - main\n workflow_dispatch:\n\nconcurrency:\n group: docker-image-builds\n cancel-in-progress: false\n\njobs:\n trl:\n name: \"Build and push TRL Docker image\"\n runs-on:\n group: aws-general-8-plus\n steps:\n - name: Checkout code\n uses: actions/checkout@v6\n\n - name: Get TRL version from PyPI\n run: |\n VERSION=$(curl -s https://pypi.org/pypi/trl/json | jq -r .info.version)\n echo \"VERSION=$VERSION\" >> $GITHUB_ENV\n\n - name: Set up Docker Buildx\n uses: docker/setup-buildx-action@v3\n\n - name: Login to DockerHub\n uses: docker/login-action@v3\n with:\n username: ${{ secrets.DOCKERHUB_USERNAME }}\n password: ${{ secrets.DOCKERHUB_PASSWORD }}\n\n - name: Build and Push\n uses: docker/build-push-action@v6\n with:\n context: docker/trl\n push: true\n tags: |\n huggingface/trl:${{ env.VERSION }}\n huggingface/trl\n\n - name: Post to Slack\n if: always()\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ secrets.CI_DOCKER_CHANNEL }}\n title: ๐Ÿค— Results of the TRL Dev Docker Image build\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n\n trl-dev:\n name: \"Build and push TRL Dev Docker image\"\n runs-on:\n group: aws-general-8-plus\n steps:\n - name: Checkout code\n uses: actions/checkout@v6\n\n - name: Set up Docker Buildx\n uses: docker/setup-buildx-action@v3\n\n - name: Login to DockerHub\n uses: docker/login-action@v3\n with:\n username: ${{ secrets.DOCKERHUB_USERNAME }}\n password: ${{ secrets.DOCKERHUB_PASSWORD }}\n\n - name: Build and Push\n uses: docker/build-push-action@v6\n with:\n context: docker/trl-dev\n push: true\n tags: |\n huggingface/trl:dev\n\n - name: Post to Slack\n if: always()\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ secrets.CI_DOCKER_CHANNEL }}\n title: ๐Ÿค— Results of the TRL Dev Docker Image build\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n" }, { "path": ".github/workflows/issue_auto_labeller.yml", "content": "name: \"Hugging Face Issue Labeler\"\non:\n issues:\n types: opened\n\njobs:\n triage:\n runs-on: ubuntu-latest\n permissions:\n issues: write\n steps:\n - uses: actions/checkout@v6\n - uses: August-murr/auto-labeler@0.0.1\n with:\n hf-api-key: ${{ secrets.CI_HF_API_TOKEN }}\n" }, { "path": ".github/workflows/pr_style_bot.yml", "content": "name: PR Style Bot\n\non:\n workflow_dispatch:\n\n\npermissions:\n contents: write\n pull-requests: write\n\njobs:\n run-style-bot:\n if: >\n contains(github.event.comment.body, '@bot /style') &&\n github.event.issue.pull_request != null\n runs-on: ubuntu-latest\n\n steps:\n - name: Extract PR details\n id: pr_info\n uses: actions/github-script@v8\n with:\n script: |\n const prNumber = context.payload.issue.number;\n const { data: pr } = await github.rest.pulls.get({\n owner: context.repo.owner,\n repo: context.repo.repo,\n pull_number: prNumber\n });\n \n // We capture both the branch ref and the \"full_name\" of the head repo\n // so that we can check out the correct repository & branch (including forks).\n core.setOutput(\"prNumber\", prNumber);\n core.setOutput(\"headRef\", pr.head.ref);\n core.setOutput(\"headRepoFullName\", pr.head.repo.full_name);\n\n - name: Check out PR branch\n uses: actions/checkout@v6\n env: \n HEADREPOFULLNAME: ${{ steps.pr_info.outputs.headRepoFullName }}\n HEADREF: ${{ steps.pr_info.outputs.headRef }}\n with:\n # Instead of checking out the base repo, use the contributor's repo name\n repository: ${{ env.HEADREPOFULLNAME }}\n ref: ${{ env.HEADREF }}\n # You may need fetch-depth: 0 for being able to push\n fetch-depth: 0\n token: ${{ secrets.GITHUB_TOKEN }}\n \n - name: Debug\n env: \n HEADREPOFULLNAME: ${{ steps.pr_info.outputs.headRepoFullName }}\n HEADREF: ${{ steps.pr_info.outputs.headRef }}\n PRNUMBER: ${{ steps.pr_info.outputs.prNumber }}\n run: |\n echo \"PR number: ${{ env.PRNUMBER }}\"\n echo \"Head Ref: ${{ env.HEADREF }}\"\n echo \"Head Repo Full Name: ${{ env.HEADREPOFULLNAME }}\"\n\n - name: Set up Python\n uses: actions/setup-python@v6\n\n - name: Install dependencies\n run: |\n pip install ruff pre-commit\n\n - name: Download Makefile from main branch\n run: |\n curl -o main_Makefile https://raw.githubusercontent.com/huggingface/trl/main/Makefile\n \n - name: Compare Makefiles\n run: |\n if ! diff -q main_Makefile Makefile; then\n echo \"Error: The Makefile has changed. Please ensure it matches the main branch.\"\n exit 1\n fi\n echo \"No changes in Makefile. Proceeding...\"\n rm -rf main_Makefile\n\n - name: Run make style and make quality\n run: |\n make precommit || true\n\n - name: Commit and push changes\n id: commit_and_push\n env: \n HEADREPOFULLNAME: ${{ steps.pr_info.outputs.headRepoFullName }}\n HEADREF: ${{ steps.pr_info.outputs.headRef }}\n PRNUMBER: ${{ steps.pr_info.outputs.prNumber }}\n GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n run: |\n echo \"HEADREPOFULLNAME: ${{ env.HEADREPOFULLNAME }}, HEADREF: ${{ env.HEADREF }}\"\n # Configure git with the Actions bot user\n git config user.name \"github-actions[bot]\"\n git config user.email \"github-actions[bot]@users.noreply.github.com\"\n\n # Make sure your 'origin' remote is set to the contributor's fork\n git remote set-url origin \"https://x-access-token:${GITHUB_TOKEN}@github.com/${{ env.HEADREPOFULLNAME }}.git\"\n\n # If there are changes after running style/quality, commit them\n if [ -n \"$(git status --porcelain)\" ]; then\n git add .\n git commit -m \"Apply style fixes\"\n # Push to the original contributor's forked branch\n git push origin HEAD:${{ env.HEADREF }}\n echo \"changes_pushed=true\" >> $GITHUB_OUTPUT\n else\n echo \"No changes to commit.\"\n echo \"changes_pushed=false\" >> $GITHUB_OUTPUT\n fi\n\n - name: Comment on PR with workflow run link\n if: steps.commit_and_push.outputs.changes_pushed == 'true'\n uses: actions/github-script@v8\n with:\n script: |\n const prNumber = parseInt(process.env.prNumber, 10);\n const runUrl = `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`\n\n await github.rest.issues.createComment({\n owner: context.repo.owner,\n repo: context.repo.repo,\n issue_number: prNumber,\n body: `Style fixes have been applied. [View the workflow run here](${runUrl}).`\n });\n env:\n prNumber: ${{ steps.pr_info.outputs.prNumber }}\n" }, { "path": ".github/workflows/publish.yml", "content": "name: Publish to PyPI\n\non:\n push:\n branches:\n - main\n - v*-release\n paths:\n - \"VERSION\"\n\njobs:\n publish:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n\n - name: Read version\n id: get_version\n run: echo \"version=$(cat VERSION)\" >> $GITHUB_OUTPUT\n\n - name: Debug - Show version.txt content\n run: echo \"Version is ${{ steps.get_version.outputs.version }}\"\n\n - name: Set up Python\n uses: actions/setup-python@v6\n with:\n python-version: \"3.x\"\n\n - name: Install dependencies\n run: |\n python -m pip install --upgrade pip\n pip install build twine\n\n - name: Build package\n run: python -m build\n\n - name: Publish to PyPI\n if: ${{ !contains(steps.get_version.outputs.version, 'dev') }}\n env:\n TWINE_USERNAME: __token__\n TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}\n run: |\n python -m twine upload dist/*\n" }, { "path": ".github/workflows/slow-tests.yml", "content": "name: Slow tests (on push)\n\non:\n push:\n branches: [main]\n paths:\n # Run only when python files are modified\n - \"trl/**.py\"\n - \"examples/**.py\"\nenv:\n RUN_SLOW: \"yes\"\n IS_GITHUB_CI: \"1\"\n SLACK_API_TOKEN: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n TRL_EXPERIMENTAL_SILENCE: 1\n\njobs:\n run_all_tests_single_gpu:\n runs-on:\n group: aws-g4dn-2xlarge\n env:\n CUDA_VISIBLE_DEVICES: \"0\"\n TEST_TYPE: \"single_gpu\"\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all --shm-size \"16gb\"\n defaults:\n run:\n shell: bash\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Install system dependencies\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n uv pip install pytest-reportlog\n\n - name: Run slow SFT tests on single GPU\n if: always()\n run: |\n source .venv/bin/activate\n make slow_tests\n\n - name: Generate Report\n if: always()\n run: |\n source .venv/bin/activate\n uv pip install slack_sdk tabulate\n python scripts/log_reports.py >> $GITHUB_STEP_SUMMARY\n\n run_all_tests_multi_gpu:\n runs-on:\n group: aws-g4dn-2xlarge\n env:\n CUDA_VISIBLE_DEVICES: \"0,1\"\n TEST_TYPE: \"multi_gpu\"\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all --shm-size \"16gb\"\n defaults:\n run:\n shell: bash\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Install system dependencies\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n uv pip install pytest-reportlog\n\n - name: Run slow SFT tests on Multi GPU\n if: always()\n run: |\n source .venv/bin/activate\n make slow_tests\n\n - name: Generate Reports\n if: always()\n run: |\n source .venv/bin/activate\n uv pip install slack_sdk tabulate\n python scripts/log_reports.py >> $GITHUB_STEP_SUMMARY\n rm *.txt" }, { "path": ".github/workflows/tests-experimental.yml", "content": "name: Tests (experimental)\n\non:\n pull_request:\n paths:\n # Run only when relevant files are modified\n - \"trl/experimental/**\"\n - \"tests/experimental/**\"\n\nenv:\n TQDM_DISABLE: 1\n PYTORCH_CUDA_ALLOC_CONF: \"expandable_segments:True\"\n PYTORCH_ALLOC_CONF: \"expandable_segments:True\"\n TRL_EXPERIMENTAL_SILENCE: 1\n\njobs:\n check_code_quality:\n name: Check code quality\n runs-on: ubuntu-latest\n if: github.event.pull_request.draft == false\n steps:\n - uses: actions/checkout@v6\n - name: Set up Python 3.13\n uses: actions/setup-python@v6\n with:\n python-version: 3.13\n - uses: pre-commit/action@v3.0.1\n with:\n extra_args: --all-files\n\n tests:\n name: Tests (experimental)\n runs-on:\n group: aws-g4dn-2xlarge\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all\n defaults:\n run:\n shell: bash\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Set up Python 3.13\n uses: actions/setup-python@v6\n with:\n python-version: 3.13\n\n - name: Install Make and Git\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n\n - name: Test with pytest\n run: |\n source .venv/bin/activate\n make test_experimental\n" }, { "path": ".github/workflows/tests.yml", "content": "name: Tests\n\non:\n push:\n branches:\n - main\n - ci-*\n pull_request:\n paths:\n # Run only when relevant files are modified\n - \".github/**.yml\"\n - \"examples/**.py\"\n - \"scripts/**.py\"\n - \"tests/**.py\"\n - \"trl/**.py\"\n - \"pyproject.toml\"\n # Exclude if only experimental code/tests\n - \"!trl/experimental/**\"\n - \"!tests/experimental/**\"\n\nenv:\n TQDM_DISABLE: 1\n CI_SLACK_CHANNEL: ${{ secrets.CI_PUSH_MAIN_CHANNEL }}\n PYTORCH_CUDA_ALLOC_CONF: \"expandable_segments:True\"\n PYTORCH_ALLOC_CONF: \"expandable_segments:True\"\n\njobs:\n check_code_quality:\n name: Check code quality\n runs-on: ubuntu-latest\n if: github.event.pull_request.draft == false\n steps:\n - uses: actions/checkout@v6\n - name: Set up Python 3.12\n uses: actions/setup-python@v6\n with:\n python-version: 3.12\n - uses: pre-commit/action@v3.0.1\n with:\n extra_args: --all-files\n\n tests:\n name: Tests\n strategy:\n matrix:\n python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']\n fail-fast: false\n runs-on:\n group: aws-g4dn-2xlarge\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all\n defaults:\n run:\n shell: bash\n if: github.event.pull_request.draft == false\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Set up Python ${{ matrix.python-version }}\n uses: actions/setup-python@v6\n with:\n python-version: ${{ matrix.python-version }}\n\n - name: Install Make and Git\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n\n - name: Test with pytest\n run: |\n source .venv/bin/activate\n make test\n\n - name: Post to Slack\n if: github.ref == 'refs/heads/main' && always() # Check if the branch is main\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ env.CI_SLACK_CHANNEL }}\n title: Results with Python ${{ matrix.python-version }} and latest dependencies\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n\n tests_dev:\n name: Tests with dev dependencies\n runs-on:\n group: aws-g4dn-2xlarge\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all\n defaults:\n run:\n shell: bash\n if: github.event.pull_request.draft == false\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Set up Python 3.12\n uses: actions/setup-python@v6\n with:\n python-version: '3.12'\n\n - name: Install Make and Git\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n uv pip install -U git+https://github.com/huggingface/accelerate.git\n uv pip install -U git+https://github.com/huggingface/datasets.git\n uv pip install -U git+https://github.com/huggingface/transformers.git\n uv pip install -U git+https://github.com/huggingface/peft.git\n\n - name: Test with pytest\n run: |\n source .venv/bin/activate\n make test\n\n - name: Post to Slack\n if: github.ref == 'refs/heads/main' && always() # Check if the branch is main\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ env.CI_SLACK_CHANNEL }}\n title: Results with Python 3.12 and dev dependencies\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n\n tests_wo_optional_deps:\n name: Tests without optional dependencies\n runs-on:\n group: aws-g4dn-2xlarge\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all\n defaults:\n run:\n shell: bash\n if: github.event.pull_request.draft == false\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Set up Python 3.12\n uses: actions/setup-python@v6\n with:\n python-version: '3.12'\n\n - name: Install Make and Git\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[test]\"\n\n - name: Test with pytest\n run: |\n source .venv/bin/activate\n make test\n\n - name: Post to Slack\n if: github.ref == 'refs/heads/main' && always() # Check if the branch is main\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ env.CI_SLACK_CHANNEL }}\n title: Results with Python 3.12 without optional dependencies\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n\n tests_min_versions:\n name: Tests with minimum versions\n runs-on:\n group: aws-g4dn-2xlarge\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all\n defaults:\n run:\n shell: bash\n if: github.event.pull_request.draft == false\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Set up Python 3.12\n uses: actions/setup-python@v6\n with:\n python-version: '3.12'\n\n - name: Install Make and Git\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n uv pip install accelerate==1.4.0\n uv pip install datasets==3.0.0\n uv pip install transformers==4.56.2\n\n - name: Test with pytest\n run: |\n source .venv/bin/activate\n make test\n\n - name: Post to Slack\n if: github.ref == 'refs/heads/main' && always() # Check if the branch is main\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ env.CI_SLACK_CHANNEL }}\n title: Results with Python 3.12 and minimum dependencies versions\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n\n distributed_smoke:\n name: Distributed smoke tests\n runs-on:\n group: aws-g5-12xlarge-cache\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all\n defaults:\n run:\n shell: bash\n if: github.event.pull_request.draft == false\n env:\n CUDA_VISIBLE_DEVICES: \"0,1\"\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Set up Python 3.12\n uses: actions/setup-python@v6\n with:\n python-version: '3.12'\n\n - name: Install Make and Git\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n\n - name: Run distributed smoke tests\n run: |\n source .venv/bin/activate\n pytest -v tests/distributed/test_distributed.py\n\n - name: Post to Slack\n if: github.ref == 'refs/heads/main' && always() # Check if the branch is main\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ env.CI_SLACK_CHANNEL }}\n title: Results of distributed smoke tests\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n" }, { "path": ".github/workflows/tests_latest.yml", "content": "name: Tests latest TRL release with dev dependencies\n\non:\n schedule:\n - cron: '0 0 * * *' # Runs daily at midnight UTC\n\n workflow_dispatch:\n\nenv:\n TQDM_DISABLE: 1\n CI_SLACK_CHANNEL: ${{ secrets.CI_PUSH_MAIN_CHANNEL }}\n TRL_EXPERIMENTAL_SILENCE: 1\n\njobs:\n tests:\n name: Tests latest TRL release with dev dependencies\n runs-on:\n group: aws-g4dn-2xlarge\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all\n defaults:\n run:\n shell: bash\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n with: { ref: v0.29-release }\n\n - name: Set up Python 3.12\n uses: actions/setup-python@v6\n with:\n python-version: '3.12'\n\n - name: Install Make and Git\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n uv pip install -U git+https://github.com/huggingface/accelerate.git\n uv pip install -U git+https://github.com/huggingface/datasets.git\n uv pip install -U git+https://github.com/huggingface/transformers.git\n\n - name: Test with pytest\n run: |\n source .venv/bin/activate\n make test\n\n - name: Post to Slack\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ env.CI_SLACK_CHANNEL }}\n title: Results of latest TRL with Python 3.12 and dev dependencies\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n" }, { "path": ".github/workflows/tests_transformers_branch.yml", "content": "name: Tests against Transformers branch\n\non:\n workflow_dispatch:\n inputs:\n transformers_ref:\n description: \"Transformers git ref (branch, tag, or commit SHA)\"\n required: true\n default: \"main\"\n\nenv:\n TQDM_DISABLE: 1\n CI_SLACK_CHANNEL: ${{ secrets.CI_PUSH_MAIN_CHANNEL }}\n PYTORCH_CUDA_ALLOC_CONF: \"expandable_segments:True\"\n PYTORCH_ALLOC_CONF: \"expandable_segments:True\"\n\njobs:\n tests_transformers_branch:\n name: Tests with Transformers ${{ inputs.transformers_ref }}\n runs-on:\n group: aws-g4dn-2xlarge\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all\n defaults:\n run:\n shell: bash\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Set up Python 3.12\n uses: actions/setup-python@v6\n with:\n python-version: '3.12'\n\n - name: Install Make and Git\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n uv pip install -U git+https://github.com/huggingface/transformers.git@${{ inputs.transformers_ref }}\n\n - name: Test with pytest\n run: |\n source .venv/bin/activate\n make test\n\n - name: Post to Slack\n if: github.ref == 'refs/heads/main' && always()\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ env.CI_SLACK_CHANNEL }}\n title: Results with Transformers ${{ inputs.transformers_ref }}\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n\n distributed_smoke:\n name: Distributed smoke tests with Transformers ${{ inputs.transformers_ref }}\n runs-on:\n group: aws-g5-12xlarge-cache\n container:\n image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\n options: --gpus all\n defaults:\n run:\n shell: bash\n env:\n CUDA_VISIBLE_DEVICES: \"0,1\"\n steps:\n - name: Git checkout\n uses: actions/checkout@v6\n\n - name: Set up Python 3.12\n uses: actions/setup-python@v6\n with:\n python-version: '3.12'\n\n - name: Install Make and Git\n run: |\n apt-get update && apt-get install -y make git curl\n\n - name: Install uv\n run: |\n curl -LsSf https://astral.sh/uv/install.sh | sh\n\n - name: Create Python virtual environment\n run: |\n uv venv\n uv pip install --upgrade setuptools wheel\n\n - name: Install dependencies\n run: |\n source .venv/bin/activate\n uv pip install \".[dev]\"\n uv pip install -U git+https://github.com/huggingface/transformers.git@${{ inputs.transformers_ref }}\n\n - name: Run distributed smoke tests\n run: |\n source .venv/bin/activate\n pytest -v tests/distributed/test_distributed.py\n\n - name: Post to Slack\n if: github.ref == 'refs/heads/main' && always()\n uses: huggingface/hf-workflows/.github/actions/post-slack@main\n with:\n slack_channel: ${{ env.CI_SLACK_CHANNEL }}\n title: Results of distributed smoke tests with Transformers ${{ inputs.transformers_ref }}\n status: ${{ job.status }}\n slack_token: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}\n" }, { "path": ".github/workflows/trufflehog.yml", "content": "on:\n push:\n\nname: Secret Leaks\n\njobs:\n trufflehog:\n runs-on: ubuntu-latest\n steps:\n - name: Checkout code\n uses: actions/checkout@v6\n with:\n fetch-depth: 0\n - name: Secret Scanning\n uses: trufflesecurity/trufflehog@v3.93.1\n with:\n # exclude buggy postgres detector that is causing false positives and not relevant to our codebase\n extra_args: --results=verified,unknown --exclude-detectors=postgres\n" }, { "path": ".github/workflows/upload_pr_documentation.yml", "content": "name: Upload PR Documentation\n\non:\n workflow_run:\n workflows: [\"Build PR Documentation\"]\n types:\n - completed\n\njobs:\n build:\n uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@main\n with:\n package_name: trl\n secrets:\n hf_token: ${{ secrets.HF_DOC_BUILD_PUSH }}\n comment_bot_token: ${{ secrets.COMMENT_BOT_TOKEN }}" }, { "path": ".gitignore", "content": "*.bak\n.gitattributes\n.last_checked\n.gitconfig\n*.bak\n*.log\n*~\n~*\n_tmp*\ntmp*\ntags\n\n# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nenv/\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# dotenv\n.env\n\n# virtualenv\n.venv\nvenv/\nENV/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n\n.vscode\n*.swp\n\n# osx generated files\n.DS_Store\n.DS_Store?\n.Trashes\nehthumbs.db\nThumbs.db\n.idea\n\n# pytest\n.pytest_cache\n\n# tools/trust-doc-nbs\ndocs_src/.last_checked\n\n# symlinks to fastai\ndocs_src/fastai\ntools/fastai\n\n# link checker\nchecklink/cookies.txt\n\n# .gitconfig is now autogenerated\n.gitconfig\n\n# wandb files\nnbs/wandb/\nexamples/notebooks/wandb/\nwandb/\n\n# uv\nuv.lock\n" }, { "path": ".pre-commit-config.yaml", "content": "repos:\n - repo: https://github.com/astral-sh/ruff-pre-commit\n rev: v0.13.3\n hooks:\n - id: ruff-check\n types_or: [ python, pyi ]\n args: [ --fix ]\n - id: ruff-format\n types_or: [ python, pyi ]\n\n # - repo: https://github.com/codespell-project/codespell\n # rev: v2.1.0\n # hooks:\n # - id: codespell\n # args:\n # - --ignore-words-list=nd,reacher,thist,ths,magent,ba\n # - --skip=docs/css/termynal.css,docs/js/termynal.js\n" }, { "path": "AGENTS.md", "content": "# AGENTS.md\n\n## Repository-specific guidance\n\n### Main code vs experimental code\n\nThe repository is separated into **main code** and **experimental code**.\n\n* **Main code** should remain stable, consistent, and well-tested.\n* **Experimental code** may be less stable and may contain inconsistent patterns or limited testing.\n\nSmall non-invasive improvements that make experimental code more consistent with the main codebase are encouraged, but avoid large refactors.\n\n### Paper implementations\n\nIf a PR implements a method, algorithm, or training approach from a research paper, it must also add a corresponding subsection to `paper_index.md`.\n\nWhen reviewing such PRs, ensure that `paper_index.md` was updated.\n\n### Code duplication and consistency\n\nTrainers in this repository are **self-contained by design**. Shared logic (generation, reward computation, metric logging, weight syncing, etc.) is deliberately duplicated across trainers rather than abstracted into a shared base class.\n\nThis is intentional: each trainer must be readable, modifiable, and evolvable in isolation. The base class (`_BaseTrainer`) provides only minimal utilities (model card generation). Everything else โ€” vLLM generation paths, `_get_per_token_logps_and_entropies`, `_calculate_rewards`, `_prepare_inputs`, metric logging โ€” is copied in full.\n\n**The tradeoff**: duplication is accepted, but **consistency is mandatory**. When the same logic appears in multiple trainers, the duplicated blocks must stay aligned:\n\n- Same variable names (`self._last_loaded_step`, `self._metrics[mode]`, โ€ฆ)\n- Same control flow structure (if/elif/else branches in the same order)\n- Same comments (word-for-word when the logic is identical)\n- Divergences only where the trainer's semantics require it (e.g., GRPO extracts logprobs from vLLM, RLOO discards them)\n\n**Consistency over correctness**: this is a strong requirement. When duplicating code, reproduce it exactly โ€” even if you believe the original has a bug. Do not silently fix the issue in your copy. Instead, keep your copy consistent with the source and report the problem so it can be fixed across all trainers in a dedicated PR. A correct-but-inconsistent codebase is harder to maintain than a consistently-wrong one that can be fixed in a single sweep.\n\n**When modifying duplicated code**: if you change a pattern that exists in multiple trainers (e.g., the vLLM generation path in `_generate_single_turn`), apply the same change to all other trainers. A fix in GRPO often implies the same fix in RLOO, and vice versa. Not propagating a change is a bug.\n\n**When reviewing**: if a PR touches duplicated logic, verify that all copies are updated consistently. A common mistake is fixing one trainer and forgetting the others.\n\n### Simplicity\n\nThis codebase values **leanness and simplicity above all**. Prefer straightforward, inline code over abstractions, helpers, or utilities โ€” even at the cost of some robustness or generality.\n\nConcretely:\n\n- Do not add layers of indirection (registries, factory patterns, plugin systems). A contributor should be able to read a trainer top to bottom and understand the full flow.\n- Prefer a simple implementation that covers 90% of cases over a complex one that covers 100%. A function that handles the common path in 20 lines is better than a catch-all that handles every edge case in 80.\n- Do not add defensive code, fallback paths, or configuration options \"just in case\". Only handle cases that actually exist today.\n- Avoid `hasattr` and `getattr`. Their use is almost always a symptom of overly defensive programming or a disguised version check (e.g., \"this attribute was added in version X\"). Instead, either drop the conditional entirely or express the version check explicitly with a version comparison. There is nearly always a cleaner alternative.\n- When in doubt, prefer less code. Every new function, parameter, or branch is maintenance burden. The best abstraction is often no abstraction.\n\n## Documentation\n\n### Docstrings\n\nDocstrings must follow the repository format below. Do **not** convert docstrings to other styles (Google, NumPy, etc.).\n\nRules:\n\n* Types appear in backticks inside parentheses: (`str`)\n* Optional parameters are marked with `*optional*`\n* Defaults are written as: `defaults to `\n* When the default is `None`, prefer ```(`str`, *optional*)``` instead of ```(`str` or `None`, *optional*, defaults to `None`)```\n* Union types use `or`: `str` or `None`\n* References to classes use the format: [`~transformers.PreTrainedModel`]\n* Class docstrings may group parameters using headers such as: `> Parameters for X:`\n\nExample:\n\n````python\ndef method(self, param1: str, param2: int = 1, param3: float | None = None):\n \"\"\"\n Brief one-line description of what this does.\n\n Args:\n param1 (`str`):\n Description of required param.\n param2 (`int`, *optional*, defaults to `1`):\n Description of optional param with default.\n param3 (`float`, *optional*):\n Description of optional param without explicit default.\n\n Returns:\n `dict` with keys:\n - `key1` (`list[int]`):\n Description of this key.\n\n Examples:\n\n ```python\n >>> my_func(\"hello\")\n ```\n \"\"\"\n````\n\n### Links to papers\n\nWhen linking to papers, use `https://huggingface.co/papers/` instead of `https://arxiv.org/abs/` (same ID suffix system).\n" }, { "path": "CITATION.cff", "content": "cff-version: 1.2.0\ntitle: 'TRL: Transformers Reinforcement Learning'\nmessage: >-\n If you use this software, please cite it using the\n metadata from this file.\ntype: software\nauthors:\n - given-names: Leandro\n family-names: von Werra\n - given-names: Younes\n family-names: Belkada\n - given-names: Lewis\n family-names: Tunstall\n - given-names: Edward\n family-names: Beeching\n - given-names: Tristan\n family-names: Thrush\n - given-names: Nathan\n family-names: Lambert\n - given-names: Shengyi\n family-names: Huang\n - given-names: Kashif\n family-names: Rasul\n - given-names: Quentin\n family-names: Gallouรฉdec\nrepository-code: 'https://github.com/huggingface/trl'\nabstract: >-\n TRL (Transformers Reinforcement Learning) is an\n open-source toolkit for aligning transformer models via\n post-training. It provides practical, scalable\n implementations of SFT, reward modeling, DPO, and GRPO\n within the Hugging Face ecosystem.\nkeywords:\n - transformers\n - reinforcement learning\n - preference optimization\n - language model alignment\n - post-training\nlicense: Apache-2.0\nversion: '0.29'\ndate-released: '2020-03-27'\n" }, { "path": "CODE_OF_CONDUCT.md", "content": "\n# Contributor Covenant Code of Conduct\n\n## Our Pledge\n\nWe as members, contributors, and leaders pledge to make participation in our\ncommunity a harassment-free experience for everyone, regardless of age, body\nsize, visible or invisible disability, ethnicity, sex characteristics, gender\nidentity and expression, level of experience, education, socio-economic status,\nnationality, personal appearance, race, caste, color, religion, or sexual\nidentity and orientation.\n\nWe pledge to act and interact in ways that contribute to an open, welcoming,\ndiverse, inclusive, and healthy community.\n\n## Our Standards\n\nExamples of behavior that contributes to a positive environment for our\ncommunity include:\n\n* Demonstrating empathy and kindness toward other people\n* Being respectful of differing opinions, viewpoints, and experiences\n* Giving and gracefully accepting constructive feedback\n* Accepting responsibility and apologizing to those affected by our mistakes,\n and learning from the experience\n* Focusing on what is best not just for us as individuals, but for the overall\n community\n\nExamples of unacceptable behavior include:\n\n* The use of sexualized language or imagery, and sexual attention or advances of\n any kind\n* Trolling, insulting or derogatory comments, and personal or political attacks\n* Public or private harassment\n* Publishing others' private information, such as a physical or email address,\n without their explicit permission\n* Other conduct which could reasonably be considered inappropriate in a\n professional setting\n\n## Enforcement Responsibilities\n\nCommunity leaders are responsible for clarifying and enforcing our standards of\nacceptable behavior and will take appropriate and fair corrective action in\nresponse to any behavior that they deem inappropriate, threatening, offensive,\nor harmful.\n\nCommunity leaders have the right and responsibility to remove, edit, or reject\ncomments, commits, code, wiki edits, issues, and other contributions that are\nnot aligned to this Code of Conduct, and will communicate reasons for moderation\ndecisions when appropriate.\n\n## Scope\n\nThis Code of Conduct applies within all community spaces, and also applies when\nan individual is officially representing the community in public spaces.\nExamples of representing our community include using an official e-mail address,\nposting via an official social media account, or acting as an appointed\nrepresentative at an online or offline event.\n\n## Enforcement\n\nInstances of abusive, harassing, or otherwise unacceptable behavior may be\nreported to the community leaders responsible for enforcement at\nfeedback@huggingface.co.\nAll complaints will be reviewed and investigated promptly and fairly.\n\nAll community leaders are obligated to respect the privacy and security of the\nreporter of any incident.\n\n## Enforcement Guidelines\n\nCommunity leaders will follow these Community Impact Guidelines in determining\nthe consequences for any action they deem in violation of this Code of Conduct:\n\n### 1. Correction\n\n**Community Impact**: Use of inappropriate language or other behavior deemed\nunprofessional or unwelcome in the community.\n\n**Consequence**: A private, written warning from community leaders, providing\nclarity around the nature of the violation and an explanation of why the\nbehavior was inappropriate. A public apology may be requested.\n\n### 2. Warning\n\n**Community Impact**: A violation through a single incident or series of\nactions.\n\n**Consequence**: A warning with consequences for continued behavior. No\ninteraction with the people involved, including unsolicited interaction with\nthose enforcing the Code of Conduct, for a specified period of time. This\nincludes avoiding interactions in community spaces as well as external channels\nlike social media. Violating these terms may lead to a temporary or permanent\nban.\n\n### 3. Temporary Ban\n\n**Community Impact**: A serious violation of community standards, including\nsustained inappropriate behavior.\n\n**Consequence**: A temporary ban from any sort of interaction or public\ncommunication with the community for a specified period of time. No public or\nprivate interaction with the people involved, including unsolicited interaction\nwith those enforcing the Code of Conduct, is allowed during this period.\nViolating these terms may lead to a permanent ban.\n\n### 4. Permanent Ban\n\n**Community Impact**: Demonstrating a pattern of violation of community\nstandards, including sustained inappropriate behavior, harassment of an\nindividual, or aggression toward or disparagement of classes of individuals.\n\n**Consequence**: A permanent ban from any sort of public interaction within the\ncommunity.\n\n## Attribution\n\nThis Code of Conduct is adapted from the [Contributor Covenant][homepage],\nversion 2.1, available at\n[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].\n\nCommunity Impact Guidelines were inspired by\n[Mozilla's code of conduct enforcement ladder][Mozilla CoC].\n\nFor answers to common questions about this code of conduct, see the FAQ at\n[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at\n[https://www.contributor-covenant.org/translations][translations].\n\n[homepage]: https://www.contributor-covenant.org\n[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html\n[Mozilla CoC]: https://github.com/mozilla/diversity\n[FAQ]: https://www.contributor-covenant.org/faq\n[translations]: https://www.contributor-covenant.org/translations" }, { "path": "CONTRIBUTING.md", "content": "# How to contribute to TRL?\n\nEveryone is welcome to contribute, and we value everybody's contribution. Code contributions are not the only way to help the community. Answering questions, helping others, and improving the documentation are also immensely valuable.\n\nIt also helps us if you spread the word! Reference the library in blog posts about the awesome projects it made possible, shout out on Twitter every time it has helped you, or simply โญ๏ธ the repository to say thank you.\n\nHowever you choose to contribute, please be mindful and respect our [code of conduct](https://github.com/huggingface/trl/blob/main/CODE_OF_CONDUCT.md).\n\n**This guide was heavily inspired by the awesome [scikit-learn guide to contributing](https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md).**\n\n## Ways to contribute\n\nThere are several ways you can contribute to TRL:\n\n* Fix outstanding issues with the existing code.\n* Submit issues related to bugs or desired new features.\n* Implement trainers for new post-training algorithms.\n* Contribute to the examples or the documentation.\n\nIf you don't know where to start, there is a special [Good First Issue](https://github.com/huggingface/trl/labels/%F0%9F%91%B6%20good%20first%20issue) listing. It will give you a list of open issues that are beginner-friendly and help you start contributing to open-source. The best way to do that is to open a Pull Request and link it to the issue that you'd like to work on. We try to give priority to opened PRs as we can easily track the progress of the fix, and if the contributor does not have time anymore, someone else can take the PR over.\n\nFor something slightly more challenging, you can also take a look at the [Good Second Issue](https://github.com/huggingface/trl/labels/%F0%9F%A7%92%20good%20second%20issue) list. In general though, if you feel like you know what you're doing, go for it and we'll help you get there! ๐Ÿš€\n\n> All contributions are equally valuable to the community. ๐Ÿฅฐ\n\nBefore you start contributing make sure you have installed all the dev tools:\n\n```bash\npip install -e .[dev]\n```\n\n## Fixing outstanding issues\n\nIf you notice an issue with the existing code and have a fix in mind, feel free to [start contributing](#submitting-a-pull-request-pr) and open a Pull Request!\n\n## Submitting a bug-related issue or feature request\n\nDo your best to follow these guidelines when submitting a bug-related issue or a feature request. It will make it easier for us to come back to you quickly and with good feedback.\n\n### Did you find a bug?\n\nThe TRL library is robust and reliable thanks to users who report the problems they encounter.\n\nBefore you report an issue, we would really appreciate it if you could **make sure the bug was not already reported** (use the search bar on GitHub under Issues). Your issue should also be related to bugs in the library itself, and not your code.\n\nOnce you've confirmed the bug hasn't already been reported, please include the following information in your issue so we can quickly resolve it:\n\n* Your **OS type and version**, **Python**, **PyTorch**, **TRL** and **Transformers** versions.\n* A short, self-contained, code snippet that allows us to reproduce the bug in less than 30s.\n* The *full* traceback if an exception is raised.\n* Attach any other additional information, like screenshots, you think may help.\n\nTo get the OS and software versions automatically, run the following command:\n\n```bash\ntrl env\n```\n\n### Do you want a new feature?\n\nIf there is a new feature you'd like to see in TRL, please open an issue and describe:\n\n1. What is the *motivation* behind this feature? Is it related to a problem or frustration with the library? Is it a feature related to something you need for a project? Is it something you worked on and think it could benefit the community?\n\n Whatever it is, we'd love to hear about it!\n\n2. Describe your requested feature in as much detail as possible. The more you can tell us about it, the better we'll be able to help you.\n3. Provide a *code snippet* that demonstrates the feature's usage.\n4. If the feature is related to a paper, please include a link.\n\nIf your issue is well written we're already 80% of the way there by the time you create it.\n\n## Do you want to implement a new trainer?\n\nNew post-training methods are published frequently and those that satisfy the following criteria are good candidates to be integrated into TRL:\n\n* **Simplicity:** Does the new method achieve similar performance as prior methods, but with less complexity? A good example is Direct Preference Optimization (DPO) [[Rafailov et al, 2023]](https://huggingface.co/papers/2305.18290), which provided a simpler and compelling alternative to RLHF methods.\n* **Efficiency:** Does the new method provide a significant improvement in training efficiency? A good example is Odds Ratio Preference Optimization (ORPO) [[Hong et al, 2023]](https://huggingface.co/papers/2403.07691), which utilizes a similar objective as DPO but requires half the GPU VRAM.\n\nMethods that only provide incremental improvements at the expense of added complexity or compute costs are unlikely to be included in TRL.\n\nIf you want to implement a trainer for a new post-training method, first open an issue and provide the following information:\n\n* A short description of the method and a link to the paper.\n* Link to the implementation if it is open-sourced.\n* Link to model weights trained with the method if they are available.\n\nBased on the community and maintainer feedback, the next step will be to implement the trainer and config classes. See the following examples for inspiration:\n\n* Paired preference optimisation: [`dpo_trainer.py`](./trl/trainer/dpo_trainer.py) and [`dpo_config.py`](./trl/trainer/dpo_config.py)\n* RL-based optimisation: [`rloo_trainer.py`](./trl/trainer/rloo_trainer.py) and [`rloo_config.py`](./trl/trainer/rloo_config.py)\n* Online optimisation: [`online_dpo_trainer.py`](./trl/trainer/online_dpo_trainer.py) and [`online_dpo_config.py`](./trl/trainer/online_dpo_config.py)\n\n## Do you want to add documentation?\n\nWe're always looking for improvements to the documentation that make it more clear and accurate. Please let us know how the documentation can be improved, such as typos, dead links, and any missing, unclear, or inaccurate content... We'll be happy to make the changes or help you contribute if you're interested!\n\n## Submitting a pull request (PR)\n\nBefore writing code, we strongly advise you to search through the existing PRs or issues to make sure that nobody is already working on the same thing. If you are unsure, it is always a good idea to open an issue to get some feedback.\n\nYou will need basic `git` proficiency to be able to contribute to TRL. `git` is not the easiest tool to use but it has the greatest manual. Type `git --help` in a shell and enjoy. If you prefer books, [Pro Git](https://git-scm.com/book/en/v2) is a very good reference.\n\nFollow these steps to start contributing:\n\n1. Fork the [repository](https://github.com/huggingface/trl) by clicking on the 'Fork' button on the repository's page. This creates a copy of the code under your GitHub user account.\n\n2. Clone your fork to your local disk, and add the base repository as a remote. The following command assumes you have your public SSH key uploaded to GitHub. See the following guide for more [information](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository).\n\n ```bash\n git clone git@github.com:/trl.git\n cd trl\n git remote add upstream https://github.com/huggingface/trl.git\n ```\n\n3. Create a new branch to hold your development changes, and do this for every new PR you work on.\n\n Start by synchronizing your `main` branch with the `upstream/main` branch (more details in the [GitHub Docs](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/syncing-a-fork)):\n\n ```bash\n git checkout main\n git fetch upstream\n git merge upstream/main\n ```\n\n Once your `main` branch is synchronized, create a new branch from it:\n\n ```bash\n git checkout -b a-descriptive-name-for-my-changes\n ```\n\n **Do not** work on the `main` branch.\n\n4. Set up a development environment by running the following command in a conda or a virtual environment you've created for working on this library:\n\n ```bash\n pip install -e .[dev]\n ```\n\n (If TRL was already installed in the virtual environment, remove it with `pip uninstall trl` before reinstalling it.)\n\n Alternatively, if you are using [Visual Studio Code](https://code.visualstudio.com/Download), the fastest way to get set up is by using the provided Dev Container. Check [the documentation on how to get started with dev containers](https://code.visualstudio.com/docs/remote/containers).\n\n5. Develop the features on your branch.\n\n As you work on the features, you should make sure that the test suite passes. You should run the tests impacted by your changes like this (see below an explanation regarding the environment variable):\n\n ```bash\n pytest tests/.py\n ```\n\n > For the following commands leveraging the `make` utility.\n\n You can also run the full suite with the following command.\n\n ```bash\n make test\n ```\n\n TRL relies on `ruff` for maintaining consistent code formatting across its source files. Before submitting any PR, you should apply automatic style corrections and run code verification checks.\n\n We provide a `precommit` target in the `Makefile` that simplifies this process by running all required checks and optimizations on only the files modified by your PR.\n\n To apply these checks and corrections in one step, use:\n\n ```bash\n make precommit\n ```\n\n This command runs the following:\n\n * Executes `pre-commit` hooks to automatically fix style issues with `ruff` and other tools.\n * Runs additional scripts such as adding copyright information.\n\n If you prefer to apply the style corrections separately or review them individually, the `pre-commit` hook will handle the formatting for the files in question.\n\n Once you're happy with your changes, add changed files using `git add` and make a commit with `git commit` to record your changes locally:\n\n ```bash\n git add modified_file.py\n git commit\n ```\n\n Please write [good commit messages](https://chris.beams.io/posts/git-commit/).\n\n It is a good idea to sync your copy of the code with the original\n repository regularly. This way you can quickly account for changes:\n\n ```bash\n git fetch upstream\n git rebase upstream/main\n ```\n\n Push the changes to your account using:\n\n ```bash\n git push -u origin a-descriptive-name-for-my-changes\n ```\n\n6. Once you are satisfied (**and the checklist below is happy too**), go to the webpage of your fork on GitHub. Click on 'Pull request' to send your changes to the project maintainers for review.\n\n7. It's ok if maintainers ask you for changes. It happens to core contributors too! To ensure everyone can review your changes in the pull request, work on your local branch and push the updates to your fork. They will automatically appear in the pull request.\n\n### Checklist\n\n1. The title of your pull request should be a summary of its contribution;\n2. If your pull request addresses an issue, please mention the issue number in the pull request description to make sure they are linked (and people consulting the issue know you are working on it);\n3. To indicate a work in progress please prefix the title with `[WIP]`, or mark the PR as a draft PR. These are useful to avoid duplicated work, and to differentiate it from PRs ready to be merged;\n4. Make sure existing tests pass;\n5. Add high-coverage tests. No quality testing = no merge.\n\n### Tests\n\nAn extensive test suite is included to test the library behavior and several examples. Library tests can be found in\nthe [tests folder](https://github.com/huggingface/trl/tree/main/tests).\n\nWe use `pytest` to run the tests. From the root of the\nrepository here's how to run tests with `pytest` for the library:\n\n```bash\npython -m pytest -sv ./tests\n```\n\nThat's how `make test` is implemented (without the `pip install` line)!\n\nYou can specify a smaller set of tests to test only the feature\nyou're working on.\n\n### Default values guidelines\n\n1. **Use defaults when appropriate**: \n\n Provide default values unless the parameter's value varies significantly by use case. For example, datasets or models should not have defaults, but parameters like `learning_rate` should.\n\n2. **Prioritize proven defaults**: \n\n Default values should align with those recommended in the original paper or method. Alternatives require strong evidence of superior performance in most cases.\n\n3. **Ensure safety and predictability**: \n\n Defaults must be safe, expected and reliable. Avoid settings that could lead to surprising outcomes, such as excessive memory usage or poor performance in edge cases.\n\n4. **Balance consistency and flexibility**: \n\n Aim for consistent defaults across similar functions or methods. However, consistency should not be preferred to point 2 or 3.\n\n5. **Opt-in for new features**: \n\n Do not enable new features or improvements (e.g., novel loss functions) by default. Users should explicitly opt-in to use these.\n\n### Writing documentation\n\nHigh-quality documentation is crucial for maintaining a project that is easy to use, understand, and extend. When adding new features, ensure they are thoroughly documented to maintain consistency and clarity throughout the project.\n\nTo illustrate what good documentation looks like, hereโ€™s an example of a well-documented function:\n\n````python\ndef replicate_str(string: str, n: int, sep: str = \" \") -> str:\n r\"\"\"\n Replicate a string `n` times with a separator.\n\n Args:\n string (`str`):\n String to replicate.\n n (`int`):\n Number of times to replicate the string.\n sep (`str`, *optional*, defaults to `\" \"`):\n Separator to use between each replication.\n \n Returns:\n `str`: The replicated string.\n \n Examples:\n ```python\n >>> replicate_str(\"hello\", 3)\n \"hello hello hello\"\n >>> replicate_str(\"hello\", 3, sep=\", \")\n \"hello, hello, hello\"\n ```\n \"\"\"\n return sep.join([string] * n)\n````\n\n* **Line Wrapping:** Applied a consistent line wrap at column 120 to improve readability.\n* **Definite Articles:** Removed definite articles where possible to streamline language. (Eg: Changed \"The string to replicate\" to \"String to replicate\")\n* **Type Annotations:**\n * Always include type definitions, indicating if a parameter is optional and specifying the default value.\n\n* **String Defaults:**\n * Ensured that default string values are wrapped in double quotes:\n\n ```txt\n defaults to `\"foo\"`\n ```\n\n* **Dictionary Typing:**\n * Replaced generic `dict` type hints with more explicit `dict[str, Any]` to clarify expected key-value pairs.\n* **Default Value Formatting:**\n * Consistently surrounded default values with backticks for improved formatting:\n\n ```txt\n defaults to `4`\n ```\n\n* **Sub-sectioning:** When the number of arguments is large, consider breaking them into sub-sections for better readability.\n\n ```python\n def calculate_statistics(data: list[float], precision: int = 2, include_variance: bool = False) -> dict[str, float]:\n r\"\"\"\n Calculates basic statistics for a given dataset.\n \n Args:\n > Data inputs\n \n data (`list[float]`):\n A list of numerical values to analyze.\n \n > Configuration parameters\n \n precision (`int`, *optional*, defaults to `2`):\n Number of decimal places to round the results.\n include_variance (`bool`, *optional*, defaults to `False`):\n Whether to include the variance of the dataset in the results.\n \n Returns:\n `dict[str, float]`:\n A dictionary containing calculated statistics such as mean, median, and optionally variance.\n \"\"\"\n ...\n ```\n\n### Deprecation and backward compatibility\n\nOur approach to deprecation and backward compatibility is flexible and based on the featureโ€™s usage and impact. Each deprecation is carefully evaluated, aiming to balance innovation with user needs.\n\nWhen a feature or component is marked for deprecation, its use will emit a warning message. This warning will include:\n\n* **Transition Guidance**: Instructions on how to migrate to the alternative solution or replacement.\n* **Removal Version**: The target version when the feature will be removed, providing users with a clear timeframe to transition.\n\nExample:\n\n ```python\n warnings.warn(\n \"The `Trainer.foo` method is deprecated and will be removed in version 0.14.0. \"\n \"Please use the `Trainer.bar` class instead.\",\n FutureWarning,\n stacklevel=2,\n )\n ```\n\nThe deprecation and removal schedule is based on each feature's usage and impact, with examples at two extremes:\n\n* **Experimental or Low-Use Features**: For a feature that is experimental or has limited usage, backward compatibility may not be maintained between releases. Users should therefore anticipate potential breaking changes from one version to the next.\n\n* **Widely-Used Components**: For a feature with high usage, we aim for a more gradual transition period of approximately **5 months**, generally scheduling deprecation around **5 minor releases** after the initial warning.\n\nThese examples represent the two ends of a continuum. The specific timeline for each feature will be determined individually, balancing innovation with user stability needs.\n\n### Working with warnings\n\nWarnings play a critical role in guiding users toward resolving potential issues, but they should be used thoughtfully to avoid unnecessary noise. Unlike logging, which provides informational context or operational details, warnings signal conditions that require attention and action. Overusing warnings can dilute their importance, leading users to ignore them entirely.\n\n#### Definitions\n\n* **Correct**: An operation is correct if it is valid, follows the intended approach, and aligns with the current best practices or guidelines within the codebase. This is the recommended or intended way to perform the operation.\n* **Supported**: An operation is supported if it is technically valid and works within the current codebase, but it may not be the most efficient, optimal, or recommended way to perform the task. This includes deprecated features or legacy approaches that still work but may be phased out in the future.\n\n#### Choosing the right message\n\n* **Correct โ†’ No warning**: \n If the operation is fully valid and expected, no message should be issued. The system is working as intended, so no warning is necessary. \n\n* **Correct but deserves attention โ†’ No warning, possibly a log message**:\n When an operation is correct but uncommon or requires special attention, providing an informational message can be helpful. This keeps users informed without implying any issue. If available, use the logger to output this message. Example: \n\n ```python\n logger.info(\"This is an informational message about a rare but correct operation.\")\n ```\n\n* **Correct but very likely a mistake โ†’ Warning with option to disable**: \n In rare cases, you may want to issue a warning for a correct operation thatโ€™s very likely a mistake. In such cases, you must provide an option to suppress the warning. This can be done with a flag in the function. Example: \n\n ```python\n def my_function(foo, bar, _warn=True):\n if foo == bar:\n if _warn:\n logger.warning(\"foo and bar are the same, this is likely a mistake. Ignore this warning by setting `_warn=False`.\")\n # Do something\n ```\n\n* **Supported but not correct โ†’ Warning**: \n If the operation is technically supported but is deprecated, suboptimal, or could cause future issues (e.g., conflicting arguments), a warning should be raised. This message should be actionable, meaning it must explain how to resolve the issue. Example: \n\n ```python\n def my_function(foo, bar):\n if foo and bar:\n logger.warning(\"Both `foo` and `bar` were provided, but only one is allowed. Ignoring `foo`. Please pass only one of these arguments.\")\n # Do something\n ```\n\n* **Not supported โ†’ Exception**: \n If the operation is invalid or unsupported, raise an exception. This indicates that the operation cannot be performed and requires immediate attention. Example: \n\n ```python\n def my_function(foo, bar):\n if foo and bar:\n raise ValueError(\"Both `foo` and `bar` were provided, but only one is allowed. Please pass only one of these arguments.\")\n ```\n\nBy following this classification, you ensure that warnings, information, and exceptions are used appropriately, providing clear guidance to the user without cluttering the system with unnecessary messages.\n" }, { "path": "LICENSE", "content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright 2020-2026 The HuggingFace Team\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "MANIFEST.in", "content": "include LICENSE\ninclude CONTRIBUTING.md\ninclude README.md\ninclude trl/accelerate_configs/*.yaml\ninclude trl/templates/*.md\ninclude trl/skills/**/*.md\nrecursive-exclude * __pycache__\nprune tests\n" }, { "path": "MIGRATION.md", "content": "# Migrating from TRL v0 to v1\n\nThis guide covers the breaking changes introduced in TRL v1 and how to update your code. Most structural changes (trainers moved to experimental, removed model classes, etc.) already shipped in v0.29 โ€” if you're already on v0.29, this migration is minimal.\n\n## Changed defaults\n\n| Config | Parameter | v0 default | v1 default | Action needed |\n| --- | --- | --- | --- | --- |\n| `GRPOConfig` | `vllm_mode` | `\"server\"` | `\"colocate\"` | If you use `use_vllm=True` without specifying `vllm_mode`, vLLM will now run in the same process instead of connecting to a separate server. Set `vllm_mode=\"server\"` explicitly if you rely on server mode. |\n| `RLOOConfig` | `vllm_mode` | `\"server\"` | `\"colocate\"` | Same as above. |\n\n## Renamed options\n\n| Config | Parameter | v0 value | v1 value | Action needed |\n| --- | --- | --- | --- | --- |\n| `SFTConfig` | `packing` | `\"bfd-requeue\"` | `\"bfd_split\"` | Replace `packing=\"bfd-requeue\"` with `packing=\"bfd_split\"`. The old value will still be accepted for a few versions but will be removed in a future release. |\n\n## Migrating from an earlier version\n\nDepending on which version you're migrating from, refer to the [release notes](https://github.com/huggingface/trl/releases) for v0.29 and earlier for version-specific changes.\n" }, { "path": "Makefile", "content": ".PHONY: test precommit common_tests slow_tests tests_gpu test_experimental\n\ncheck_dirs := examples tests trl\n\nACCELERATE_CONFIG_PATH = `pwd`/examples/accelerate_configs\n\ntest:\n\tpytest -n auto -m \"not slow and not low_priority\" -s -v --reruns 5 --reruns-delay 1 --only-rerun '(OSError|Timeout|HTTPError.*502|HTTPError.*504||not less than or equal to 0.01)' tests\n\nprecommit:\n\tpython scripts/add_copyrights.py\n\tpre-commit run --all-files\n\tdoc-builder style trl tests docs/source --max_len 119\n\nslow_tests:\n\tpytest -m \"slow\" tests/ $(if $(IS_GITHUB_CI),--report-log \"slow_tests.log\",)\n\ntest_experimental:\n\tpytest -n auto -s -v tests/experimental\n" }, { "path": "README.md", "content": "# TRL - Transformers Reinforcement Learning\n\n
\n \n \n \"TRL\n \n
\n\n

\n\n

\n

A comprehensive library to post-train foundation models

\n

\n\n

\n \"License\"\n \"Documentation\"\n \"GitHub\n \"Hugging\n

\n\n## ๐ŸŽ‰ What's New\n\n**OpenEnv Integration:** TRL now supports **[OpenEnv](https://huggingface.co/blog/openenv)**, the open-source framework from Meta for defining, deploying, and interacting with environments in reinforcement learning and agentic workflows.\n\nExplore how to seamlessly integrate TRL with OpenEnv in our [dedicated documentation](https://huggingface.co/docs/trl/openenv).\n\n## Overview\n\nTRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the [๐Ÿค— Transformers](https://github.com/huggingface/transformers) ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.\n\n## Highlights\n\n- **Trainers**: Various fine-tuning methods are easily accessible via trainers like [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer), [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer), [`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer), [`RewardTrainer`](https://huggingface.co/docs/trl/reward_trainer) and more.\n\n- **Efficient and scalable**:\n - Leverages [๐Ÿค— Accelerate](https://github.com/huggingface/accelerate) to scale from single GPU to multi-node clusters using methods like [DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) and [DeepSpeed](https://github.com/deepspeedai/DeepSpeed).\n - Full integration with [๐Ÿค— PEFT](https://github.com/huggingface/peft) enables training on large models with modest hardware via quantization and LoRA/QLoRA.\n - Integrates [๐Ÿฆฅ Unsloth](https://github.com/unslothai/unsloth) for accelerating training using optimized kernels.\n\n- **Command Line Interface (CLI)**: A simple interface lets you fine-tune with models without needing to write code.\n\n## Installation\n\n### Python Package\n\nInstall the library using `pip`:\n\n```bash\npip install trl\n```\n\n### From source\n\nIf you want to use the latest features before an official release, you can install TRL from source:\n\n```bash\npip install git+https://github.com/huggingface/trl.git\n```\n\n### Repository\n\nIf you want to use the examples you can clone the repository with the following command:\n\n```bash\ngit clone https://github.com/huggingface/trl.git\n```\n\n## Quick Start\n\nFor more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL is a light wrapper around the ๐Ÿค— Transformers trainer and natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP.\n\n### `SFTTrainer`\n\nHere is a basic example of how to use the [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer):\n\n```python\nfrom trl import SFTTrainer\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib/Capybara\", split=\"train\")\n\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen2.5-0.5B\",\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\n### `GRPOTrainer`\n\n[`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer) implements the [Group Relative Policy Optimization (GRPO) algorithm](https://huggingface.co/papers/2402.03300) that is more memory-efficient than PPO and was used to train [Deepseek AI's R1](https://huggingface.co/deepseek-ai/DeepSeek-R1).\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\nfrom trl.rewards import accuracy_reward\n\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\n> [!NOTE]\n> For reasoning models, use the `reasoning_accuracy_reward()` function for better results.\n\n### `DPOTrainer`\n\n[`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer) implements the popular [Direct Preference Optimization (DPO) algorithm](https://huggingface.co/papers/2305.18290) that was used to post-train [Llama 3](https://huggingface.co/papers/2407.21783) and many other models. Here is a basic example of how to use the `DPOTrainer`:\n\n```python\nfrom datasets import load_dataset\nfrom trl import DPOTrainer\n\ndataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\ntrainer = DPOTrainer(\n model=\"Qwen3/Qwen-0.6B\",\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\n### `RewardTrainer`\n\nHere is a basic example of how to use the [`RewardTrainer`](https://huggingface.co/docs/trl/reward_trainer):\n\n```python\nfrom trl import RewardTrainer\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\ntrainer = RewardTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\n## Command Line Interface (CLI)\n\nYou can use the TRL Command Line Interface (CLI) to quickly get started with post-training methods like Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO):\n\n**SFT:**\n\n```bash\ntrl sft --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --output_dir Qwen2.5-0.5B-SFT\n```\n\n**DPO:**\n\n```bash\ntrl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \\\n --dataset_name argilla/Capybara-Preferences \\\n --output_dir Qwen2.5-0.5B-DPO \n```\n\nRead more about CLI in the [relevant documentation section](https://huggingface.co/docs/trl/clis) or use `--help` for more details.\n\n## Development\n\nIf you want to contribute to `trl` or customize it to your needs make sure to read the [contribution guide](https://github.com/huggingface/trl/blob/main/CONTRIBUTING.md) and make sure you make a dev install:\n\n```bash\ngit clone https://github.com/huggingface/trl.git\ncd trl/\npip install -e .[dev]\n```\n\n## Experimental\n\nA minimal incubation area is available under `trl.experimental` for unstable / fast-evolving features. Anything there may change or be removed in any release without notice.\n\nExample:\n\n```python\nfrom trl.experimental.new_trainer import NewTrainer\n```\n\nRead more in the [Experimental docs](https://huggingface.co/docs/trl/experimental_overview).\n\n## Citation\n\n```bibtex\n@software{vonwerra2020trl,\n title = {{TRL: Transformers Reinforcement Learning}},\n author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouรฉdec, Quentin},\n license = {Apache-2.0},\n url = {https://github.com/huggingface/trl},\n year = {2020}\n}\n```\n\n## License\n\nThis repository's source code is available under the [Apache-2.0 License](LICENSE).\n" }, { "path": "RELEASE.md", "content": "# Making a release\n\n> [!NOTE]\n> VERSION needs to be formatted following the `v{major}.{minor}.{patch}` convention. We need to follow this convention to be able to retrieve versioned scripts.\n\n## Major/Minor Release\n\n### 1. Ensure your local repository is up to date with the upstream repository\n\n```bash\ngit checkout main\ngit pull origin main\n```\n\n> [!WARNING]\n> Do not merge other pull requests into `main` until the release is done. This is to ensure that the release is stable and does not include any untested changes. Announce internally (#trl-internal) to other maintainers that you are doing a release and that they must not merge PRs until the release is done.\n\n### 2. Create a release branch from main\n\n```bash\ngit checkout -b release-v{major}.{minor}\n```\n\n### 3. Change the version in the following files\n\n- `.github/workflows/tests_latest.yml`:\n \n ```diff\n - with: { ref: v{major}.{minor-1}-release }\n + with: { ref: v{major}.{minor}-release }\n ```\n\n- `CITATION.cff`\n\n ```diff\n - version: '{major}.{minor-1}'\n + version: '{major}.{minor}'\n ```\n\n- `VERSION`\n\n ```diff\n - {major}.{minor}.0.dev0\n + {major}.{minor}.0\n ```\n\n### 4. Commit and push these changes\n\n```shell\ngit add .github/workflows/tests_latest.yml CITATION.cff VERSION\ngit commit -m 'Release: {major}.{minor}'\ngit push origin release-v{major}.{minor}\n```\n\n### 5. Create a pull request\n\nfrom `release-v{major}.{minor}` to `main`, named `Release: v{major}.{minor}`, wait for tests to pass, and request a review.\n\n### 6. Once the pull request is approved, merge it into `main`\n\nIt will automatically publish the new version of the package on PyPI.\n\n### 7. Add a tag in git to mark the release\n\n```shell\ngit checkout main\ngit pull origin main\ngit tag -a v{major}.{minor}.0 -m 'Adds tag v{major}.{minor}.0 for PyPI'\ngit push origin v{major}.{minor}.0\n```\n\n### 8. Create a branch `v{major}.{minor}-release` for future patch releases\n\n```shell\ngit checkout -b v{major}.{minor}-release\ngit push origin v{major}.{minor}-release\n```\n\nThis ensures that future patch releases (`v{major}.{minor}.1`, `v{major}.{minor}.2`, etc.) can be made separately from `main`.\n\n### 9. Create a GitHub Release\n\n1. Go to the repoโ€™s [releases section](https://github.com/huggingface/trl/releases) on GitHub.\n2. Click **Draft a new release**.\n3. Select the `v{major}.{minor}.0` tag you just created in step 7.\n4. Add a title (`v{major}.{minor}.0`) and a short description of whatโ€™s new.\n5. Click **Publish Release**.\n\n### 10. Bump to dev version\n\n1. Create a branch `bump-dev-version-{major}.{minor+1}` from `main` and checkout to it.\n\n ```shell\n git checkout -b bump-dev-version-{major}.{minor+1}\n ```\n\n2. Change the version in file `VERSION`:\n\n ```diff\n - {major}.{minor}.0\n + {major}.{minor+1}.0.dev0\n ```\n\n3. Commit and push these changes\n\n ```shell\n git add VERSION\n git commit -m 'โฌ†๏ธ Bump dev version'\n git push origin bump-dev-version-{major}.{minor+1}\n ```\n\n4. Create a pull request from `bump-dev-version-{major}.{minor+1}` to `main`, named `โฌ†๏ธ Bump dev version`, and request urgent review.\n\n5. Once the pull request is approved, merge it into `main`.\n\n6. The codebase is now ready for the next development cycle, inform the team in the #trl-internal channel.\n\n## Making a patch release\n\n### 1. Ensure your local repository is up to date with the upstream repository\n\n```bash\ngit checkout v{major}.{minor}-release\ngit pull origin main\n```\n\n### 2. Cherry-pick the changes you want to include in the patch release\n\n```bash\ngit cherry-pick \ngit cherry-pick \n...\n```\n\n### 3. Change the version in the file `VERSION`\n\n```diff\n- {major}.{minor}.{patch-1}\n+ {major}.{minor}.{patch}\n```\n\n### 4. Commit and push these changes\n\n```shell\ngit add VERSION\ngit commit -m 'Release: {major}.{minor}.{patch}'\ngit push origin v{major}.{minor}-release\n```\n\n### 5. Wait for the CI to pass\n\nThe CI will automatically publish the new version of the package on PyPI.\n\n### 6. Add a tag in git to mark the release\n\n```shell\ngit tag -a v{major}.{minor}.{patch} -m 'Adds tag v{major}.{minor}.{patch} for PyPI'\ngit push origin v{major}.{minor}.{patch}\n```\n\n#### 7. Create a GitHub Release\n\n1. Go to the repoโ€™s [releases section](https://github.com/huggingface/trl/releases) on GitHub.\n2. Click **Draft a new release**.\n3. Select the `v{major}.{minor}.{patch}` tag you just created in step 7.\n4. Add a title (`v{major}.{minor}.{patch}`) and a short description of whatโ€™s new.\n5. Click **Publish Release**.\n" }, { "path": "VERSION", "content": "1.0.0.dev0" }, { "path": "docker/trl/Dockerfile", "content": "FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\nRUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*\nRUN pip install --upgrade pip uv\nRUN uv pip install --system trl[liger,peft,vlm] kernels trackio" }, { "path": "docker/trl-dev/Dockerfile", "content": "FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel\nRUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*\nRUN pip install --upgrade pip uv\nRUN uv pip install --system --no-cache \"git+https://github.com/huggingface/trl.git#egg=trl[liger,peft,vlm]\"\nRUN uv pip install --system kernels liger_kernel peft trackio" }, { "path": "docs/source/_toctree.yml", "content": "- sections:\n - local: index\n title: TRL\n - local: installation\n title: Installation\n - local: quickstart\n title: Quickstart\n title: Getting started\n- sections:\n - local: dataset_formats\n title: Dataset Formats\n - local: paper_index\n title: Paper Index\n title: Conceptual Guides\n- sections: # Sorted alphabetically\n - local: dpo_trainer\n title: DPO\n - local: grpo_trainer\n title: GRPO\n - local: reward_trainer\n title: Reward\n - local: rloo_trainer\n title: RLOO\n - local: sft_trainer\n title: SFT\n title: Trainers\n- sections:\n - local: clis\n title: Command Line Interface (CLI)\n - local: jobs_training\n title: Training using Jobs\n - local: customization\n title: Customizing the Training\n - local: reducing_memory_usage\n title: Reducing Memory Usage\n - local: speeding_up_training\n title: Speeding Up Training\n - local: distributing_training\n title: Distributing Training\n - local: use_model\n title: Using Trained Models\n title: How-to guides\n- sections:\n - local: deepspeed_integration\n title: DeepSpeed\n - local: kernels_hub\n title: Kernels Hub\n - local: liger_kernel_integration\n title: Liger Kernel\n - local: peft_integration\n title: PEFT\n - local: ptt_integration\n title: Post Training Toolkit\n - local: rapidfire_integration\n title: RapidFire AI\n - local: trackio_integration\n title: Trackio\n - local: unsloth_integration\n title: Unsloth\n - local: vllm_integration\n title: vLLM\n title: Integrations\n- sections:\n - local: example_overview\n title: Example Overview\n - local: community_tutorials\n title: Community Tutorials\n - local: lora_without_regret\n title: LoRA Without Regret\n title: Examples\n- sections:\n - sections:\n - local: chat_template_utils\n title: Chat Template Utilities\n - local: data_utils\n title: Data Utilities\n - local: script_utils\n title: Script Utilities\n title: Utilities\n - local: callbacks\n title: Callbacks\n - local: rewards\n title: Reward Functions\n title: API\n- sections:\n - local: experimental_overview\n title: Experimental Overview\n - local: openenv\n title: OpenEnv Integration\n - local: async_grpo_trainer # Sorted alphabetically\n title: Asynchronous GRPO\n - local: bema_for_reference_model\n title: BEMA for Reference Model\n - local: bco_trainer\n title: BCO\n - local: cpo_trainer\n title: CPO\n - local: gfpo\n title: GFPO\n - local: gkd_trainer\n title: GKD\n - local: gold_trainer\n title: GOLD\n - local: grpo_with_replay_buffer\n title: GRPO With Replay Buffer\n - local: gspo_token\n title: GSPO-token\n - local: judges\n title: Judges\n - local: kto_trainer\n title: KTO \n - local: merge_model_callback\n title: MergeModelCallback\n - local: minillm_trainer\n title: MiniLLM\n - local: nash_md_trainer\n title: Nash-MD\n - local: nemo_gym\n title: NeMo Gym\n - local: online_dpo_trainer\n title: Online DPO\n - local: orpo_trainer\n title: ORPO\n - local: papo_trainer\n title: PAPO\n - local: ppo_trainer\n title: PPO\n - local: prm_trainer\n title: PRM\n - local: winrate_callback\n title: WinRateCallback\n - local: xpo_trainer\n title: XPO\n title: Experimental\n" }, { "path": "docs/source/async_grpo_trainer.md", "content": "# Asynchronous GRPO\n\n> [!IMPORTANT]\n> This trainer requires `vllm>=0.17.1` and `transformers>=5.2.0`. For distributed training, only FSDP2 is supported (DeepSpeed ZeRO is not).\n>\n> Currently, `vllm` and `transformers` have conflicting dependency constraints. To work around this, install vLLM first and then force-install transformers:\n>\n> ```bash\n> pip install 'vllm>=0.17.1'\n> pip install 'transformers>=5.2.0' --no-deps\n> ```\n\n## Overview\n\n[`AsyncGRPOTrainer`] implements the same [GRPO](grpo_trainer) algorithm but decouples rollout generation from training. A background worker continuously streams completions from a vLLM server while the training loop consumes them, so generation and gradient updates overlap instead of alternating. The API mirrors [`GRPOTrainer`] โ€” for full details on the GRPO method itself (advantage computation, KL estimation, loss formulation, reward functions, etc.), see the [GRPO Trainer](grpo_trainer) documentation. Not all features from [`GRPOTrainer`] are available; refer to [`AsyncGRPOConfig`] for the supported parameters.\n\nThis trainer was contributed by [Quentin Gallouรฉdec](https://huggingface.co/qgallouedec) and [Amine Dirhoussi](https://huggingface.co/aminediroHF).\n\n## How it differs from [`GRPOTrainer`]\n\nIn the standard [`GRPOTrainer`], generation and training are sequential: generate a batch, compute the loss, update weights, repeat. Even in [vLLM colocate mode](grpo_trainer#speed-up-training-with-vllm), where generation runs on the same GPUs, one phase must finish before the other begins.\n\n[`AsyncGRPOTrainer`] separates these two concerns:\n\n- **Rollout worker** (background thread) โ€” sends prompts to a vLLM server, scores completions with reward functions, computes advantages, and pushes ready-to-train samples into a queue.\n- **Training loop** (main process) โ€” pulls samples from the queue, computes the clipped surrogate loss, and updates the model weights.\n\nAfter every `weight_sync_steps` training steps, the updated weights are transferred to the vLLM server via NCCL so that subsequent generations reflect the latest policy.\n\nBecause generation and training run concurrently, the training samples may have been generated by a slightly older version of the model. The `max_staleness` parameter controls how many weight updates a sample can lag behind before being discarded.\n\nThe number of concurrent requests sent to the vLLM server is controlled by `max_inflight_tasks`. By default it is set automatically to `max_staleness ร— per_device_train_batch_size ร— gradient_accumulation_steps ร— num_processes` โ€” the maximum number of samples the trainer can consume before they become stale. Generating more than this is wasteful since the excess samples will be discarded.\n\n## Quick start\n\n```python\n# train_async_grpo.py\nfrom datasets import load_dataset\nfrom trl.experimental.async_grpo import AsyncGRPOTrainer\nfrom trl.rewards import accuracy_reward\n\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\ntrainer = AsyncGRPOTrainer(\n model=\"Qwen/Qwen3-4B\",\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\nThe vLLM server and the trainer must run on **separate GPUs**. Use `CUDA_VISIBLE_DEVICES` to partition your GPUs. For example, with 2 GPUs, you can run the vLLM server on GPU 0 and the trainer on GPU 1 as follows:\n\n```bash\n# Terminal 1: vLLM server on GPU 0 (dev mode + NCCL weight transfer are required)\nCUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3-4B \\\n --max-model-len 4096 \\\n --logprobs-mode processed_logprobs \\\n --weight-transfer-config '{\"backend\":\"nccl\"}'\n```\n\n> [!TIP]\n> Set `--max-model-len` to the maximum total sequence length (prompt + completion) you expect. A lower value reduces GPU memory usage on the server, freeing more memory for the KV cache and increasing throughput. A good starting point is the prompt length plus `max_completion_length` from your config.\n\n```bash\n# Terminal 2: training on GPU 1\nCUDA_VISIBLE_DEVICES=1 accelerate launch train_async_grpo.py\n```\n\n## Design philosophy\n\nThis trainer is intentionally kept minimal and is not meant to grow into a general-purpose solution. If you need a feature that is not supported, we recommend cloning the repository and adapting the trainer to your needs directly. New features will only be considered when there is significant community demand.\n\n## AsyncGRPOConfig\n\n[[autodoc]] trl.experimental.async_grpo.AsyncGRPOConfig\n\n## AsyncGRPOTrainer\n\n[[autodoc]] trl.experimental.async_grpo.AsyncGRPOTrainer\n" }, { "path": "docs/source/bco_trainer.md", "content": "# BCO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-BCO-blue)](https://huggingface.co/models?other=bco,trl)\n\nTRL supports the Binary Classifier Optimization (BCO).\nThe [BCO](https://huggingface.co/papers/2404.04656) authors train a binary classifier whose logit serves as a reward so that the classifier maps {prompt, chosen completion} pairs to 1 and {prompt, rejected completion} pairs to 0.\nFor a full example have a look at [`examples/scripts/bco.py`].\n\n## Expected dataset type\n\nThe [`experimental.bco.BCOTrainer`] requires an [unpaired preference dataset](dataset_formats#unpaired-preference).\nThe [`experimental.bco.BCOTrainer`] supports both [conversational](dataset_formats#conversational) and [standard](dataset_formats#standard) dataset formats. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\n## Expected model format\n\nThe BCO trainer expects a model of `AutoModelForCausalLM`, compared to PPO that expects `AutoModelForCausalLMWithValueHead` for the value function.\n\n## Using the `BCOTrainer`\n\nFor a detailed example have a look at the `examples/scripts/bco.py` script. At a high level we need to initialize the `BCOTrainer` with a `model` we wish to train and a reference `ref_model` which we will use to calculate the implicit rewards of the preferred and rejected response.\n\nThe `beta` refers to the hyperparameter of the implicit reward, and the dataset contains the 3 entries listed above. Note that the `model` and `ref_model` need to have the same architecture (ie decoder only or encoder-decoder).\n\n```python\nfrom trl.experimental.bco import BCOConfig, BCOTrainer\n\ntraining_args = BCOConfig(\n beta=0.1,\n)\n\nbco_trainer = BCOTrainer(\n model,\n model_ref,\n args=training_args,\n train_dataset=train_dataset,\n processing_class=tokenizer,\n)\n```\n\nAfter this one can then call:\n\n```python\nbco_trainer.train()\n```\n\n## Underlying Distribution matching (UDM)\n\nIn practical scenarios, the thumbs-up and thumbs-down datasets are likely to have divergent underlying distributions of prompts.\nConsider an LLM deployed for user feedback: if the model excels in writing tasks but underperforms in coding, the thumbs-up dataset will be dominated by writing-related prompts, while the thumbs-down dataset will contain mostly coding-related prompts. \nIf the prompts in your desired and undesired datasets differ a lot, it is useful to enable UDM. \n\nChoose an embedding model and tokenizer:\n\n```python\nembedding_model = AutoModel.from_pretrained(your_model_id)\nembedding_tokenizer = AutoTokenizer.from_pretrained(your_model_id)\n\n# customize this function depending on your embedding model\ndef embed_prompt(input_ids, attention_mask, model):\n outputs = model(input_ids=input_ids, attention_mask=attention_mask)\n return outputs.last_hidden_state.mean(dim=1)\n\nembedding_model = Accelerator().prepare_model(self.embedding_model)\nembedding_func = partial(embed_prompt, model=embedding_model)\n```\n\nSet `prompt_sample_size` to define how many prompts are selected to train the UDM classifier and start the training with the provided embedding function:\n\n```python\ntraining_args = BCOConfig(\n beta=0.1,\n prompt_sample_size=512,\n)\n\nbco_trainer = BCOTrainer(\n model,\n model_ref,\n args=training_args,\n train_dataset=train_dataset,\n processing_class=tokenizer,\n embedding_func=embedding_func,\n embedding_tokenizer=self.embedding_tokenizer,\n)\n\nbco_trainer.train()\n```\n\n### For Mixture of Experts Models: Enabling the auxiliary loss\n\nMOEs are the most efficient if the load is about equally distributed between experts. \nTo ensure that we train MOEs similarly during preference-tuning, it is beneficial to add the auxiliary loss from the load balancer to the final loss. \n\nThis option is enabled by setting `output_router_logits=True` in the model config (e.g. MixtralConfig). \nTo scale how much the auxiliary loss contributes to the total loss, use the hyperparameter `router_aux_loss_coef=...` (default: 0.001).\n\n## BCOTrainer\n\n[[autodoc]] experimental.bco.BCOTrainer\n - train\n - save_model\n - push_to_hub\n\n## BCOConfig\n\n[[autodoc]] experimental.bco.BCOConfig\n" }, { "path": "docs/source/bema_for_reference_model.md", "content": "# BEMA for Reference Model\n\nThis feature implements the BEMA algorithm to update the reference model during DPO training.\n\n## Usage\n\n```python\nfrom trl.experimental.bema_for_ref_model import BEMACallback, DPOTrainer\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\nbema_callback = BEMACallback(update_ref_model=True)\n\ntrainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n train_dataset=dataset,\n callbacks=[bema_callback],\n)\ntrainer.train()\n```\n\n## DPOTrainer\n\n[[autodoc]] experimental.bema_for_ref_model.DPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## BEMACallback\n\n[[autodoc]] experimental.bema_for_ref_model.BEMACallback\n" }, { "path": "docs/source/callbacks.md", "content": "# Callbacks\n\n## RichProgressCallback\n\n[[autodoc]] RichProgressCallback\n\n## LogCompletionsCallback\n\n[[autodoc]] LogCompletionsCallback\n\n## BEMACallback\n\n[[autodoc]] BEMACallback\n\n## WeaveCallback\n\n[[autodoc]] WeaveCallback\n" }, { "path": "docs/source/chat_template_utils.md", "content": "# Chat template utilities\n\n## clone_chat_template\n\n[[autodoc]] clone_chat_template\n\n## is_chat_template_prefix_preserving\n\n[[autodoc]] chat_template_utils.is_chat_template_prefix_preserving\n\n## get_training_chat_template\n\n[[autodoc]] chat_template_utils.get_training_chat_template\n" }, { "path": "docs/source/clis.md", "content": "# Command Line Interfaces (CLIs)\n\nTRL provides a powerful command-line interface (CLI) to fine-tune large language models (LLMs) using methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and more. The CLI abstracts away much of the boilerplate, letting you launch training jobs quickly and reproducibly.\n\n## Commands\n\nCurrently supported commands are:\n\n### Training Commands\n\n- `trl dpo`: fine-tune a LLM with DPO\n- `trl grpo`: fine-tune a LLM with GRPO\n- `trl kto`: fine-tune a LLM with KTO\n- `trl reward`: train a Reward Model\n- `trl rloo`: fine-tune a LLM with RLOO\n- `trl sft`: fine-tune a LLM with SFT\n\n### Other Commands\n\n- `trl env`: get the system information\n- `trl vllm-serve`: serve a model with vLLM\n\n## Fine-Tuning with the TRL CLI\n\n### Basic Usage\n\nYou can launch training directly from the CLI by specifying required arguments like the model and dataset:\n\n\n\n\n```bash\ntrl sft \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name stanfordnlp/imdb\n```\n\n\n\n\n```bash\ntrl dpo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name anthropic/hh-rlhf\n```\n\n\n\n\n```bash\ntrl reward \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/ultrafeedback_binarized\n```\n\n\n\n\n```bash\ntrl grpo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name HuggingFaceH4/Polaris-Dataset-53K \\\n --reward_funcs accuracy_reward\n```\n\n\n\n\n```bash\ntrl rloo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name HuggingFaceH4/Polaris-Dataset-53K \\\n --reward_funcs accuracy_reward\n```\n\n\n\n\n```bash\ntrl kto \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/kto-mix-14k\n```\n\n\n\n\n### Using Configuration Files\n\nTo keep your CLI commands clean and reproducible, you can define all training arguments in a YAML configuration file:\n\n\n\n\n```yaml\n# sft_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: stanfordnlp/imdb\n```\n\nLaunch with:\n\n```bash\ntrl sft --config sft_config.yaml\n```\n\n\n\n\n```yaml\n# dpo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: anthropic/hh-rlhf\n```\n\nLaunch with:\n\n```bash\ntrl dpo --config dpo_config.yaml\n```\n\n\n\n\n```yaml\n# reward_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: trl-lib/ultrafeedback_binarized\n```\n\nLaunch with:\n\n```bash\ntrl reward --config reward_config.yaml\n```\n\n\n\n\n```yaml\n# grpo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: HuggingFaceH4/Polaris-Dataset-53K\nreward_funcs:\n - accuracy_reward\n```\n\nLaunch with:\n\n```bash\ntrl grpo --config grpo_config.yaml\n```\n\n\n\n\n```yaml\n# rloo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: HuggingFaceH4/Polaris-Dataset-53K\nreward_funcs:\n - accuracy_reward\n```\n\nLaunch with:\n\n```bash\ntrl rloo --config rloo_config.yaml\n```\n\n\n\n\n```yaml\n# kto_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: trl-lib/kto-mix-14k\n```\n\nLaunch with:\n\n```bash\ntrl kto --config kto_config.yaml\n```\n\n\n\n\n### Scaling Up with Accelerate\n\nTRL CLI natively supports [๐Ÿค— Accelerate](https://huggingface.co/docs/accelerate), making it easy to scale training across multiple GPUs, machines, or use advanced setups like DeepSpeed โ€” all from the same CLI.\n\nYou can pass any `accelerate launch` arguments directly to `trl`, such as `--num_processes`. For more information see [Using accelerate launch](https://huggingface.co/docs/accelerate/en/basic_tutorials/launch#using-accelerate-launch).\n\n\n\n\n```bash\ntrl sft \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name stanfordnlp/imdb \\\n --num_processes 4\n```\n\nor, with a config file:\n\n```yaml\n# sft_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: stanfordnlp/imdb\nnum_processes: 4\n```\n\nLaunch with:\n\n```bash\ntrl sft --config sft_config.yaml\n```\n\n\n\n\n```bash\ntrl dpo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name anthropic/hh-rlhf \\\n --num_processes 4\n```\n\nor, with a config file:\n\n```yaml\n# dpo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: anthropic/hh-rlhf\nnum_processes: 4\n```\n\nLaunch with:\n\n```bash\ntrl dpo --config dpo_config.yaml\n```\n\n\n\n\n```bash\ntrl reward \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --num_processes 4\n```\n\nor, with a config file:\n\n```yaml\n# reward_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: trl-lib/ultrafeedback_binarized\nnum_processes: 4\n```\n\nLaunch with:\n\n```bash\ntrl reward --config reward_config.yaml\n```\n\n\n\n\n```bash\ntrl grpo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name HuggingFaceH4/Polaris-Dataset-53K \\\n --reward_funcs accuracy_reward \\\n --num_processes 4\n```\n\nor, with a config file:\n\n```yaml\n# grpo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: HuggingFaceH4/Polaris-Dataset-53K\nreward_funcs:\n - accuracy_reward\nnum_processes: 4\n```\n\nLaunch with:\n\n```bash\ntrl grpo --config grpo_config.yaml\n```\n\n\n\n\n```bash\ntrl rloo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name HuggingFaceH4/Polaris-Dataset-53K \\\n --reward_funcs accuracy_reward \\\n --num_processes 4\n```\n\nor, with a config file:\n\n```yaml\n# rloo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: HuggingFaceH4/Polaris-Dataset-53K\nreward_funcs:\n - accuracy_reward\nnum_processes: 4\n```\n\nLaunch with:\n\n```bash\ntrl rloo --config rloo_config.yaml\n```\n\n\n\n\n```bash\ntrl kto \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/kto-mix-14k \\\n --num_processes 4\n```\n\nor, with a config file:\n\n```yaml\n# kto_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: trl-lib/kto-mix-14k\nnum_processes: 4\n```\n\nLaunch with:\n\n```bash\ntrl kto --config kto_config.yaml\n```\n\n\n\n\n### Using `--accelerate_config` for Accelerate Configuration\n\nThe `--accelerate_config` flag lets you easily configure distributed training with [๐Ÿค— Accelerate](https://github.com/huggingface/accelerate). This flag accepts either:\n\n- the name of a predefined config profile (built into TRL), or\n- a path to a custom Accelerate YAML config file.\n\n#### Predefined Config Profiles\n\nTRL provides several ready-to-use Accelerate configs to simplify common training setups:\n\n| Name | Description |\n| --- | --- |\n| `fsdp1` | Fully Sharded Data Parallel Stage 1 |\n| `fsdp2` | Fully Sharded Data Parallel Stage 2 |\n| `zero1` | DeepSpeed ZeRO Stage 1 |\n| `zero2` | DeepSpeed ZeRO Stage 2 |\n| `zero3` | DeepSpeed ZeRO Stage 3 |\n| `multi_gpu` | Multi-GPU training |\n| `single_gpu` | Single-GPU training |\n\nTo use one of these, just pass the name to `--accelerate_config`. TRL will automatically load the corresponding config file from `trl/accelerate_config/`.\n\n#### Example Usage\n\n\n\n\n```bash\ntrl sft \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name stanfordnlp/imdb \\\n --accelerate_config zero2 # or path/to/my/accelerate/config.yaml\n```\n\nor, with a config file:\n\n```yaml\n# sft_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: stanfordnlp/imdb\naccelerate_config: zero2 # or path/to/my/accelerate/config.yaml\n```\n\nLaunch with:\n\n```bash\ntrl sft --config sft_config.yaml\n```\n\n\n\n\n```bash\ntrl dpo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name anthropic/hh-rlhf \\\n --accelerate_config zero2 # or path/to/my/accelerate/config.yaml\n```\n\nor, with a config file:\n\n```yaml\n# dpo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: anthropic/hh-rlhf\naccelerate_config: zero2 # or path/to/my/accelerate/config.yaml\n```\n\nLaunch with:\n\n```bash\ntrl dpo --config dpo_config.yaml\n```\n\n\n\n\n```bash\ntrl reward \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --accelerate_config zero2 # or path/to/my/accelerate/config.yaml\n```\n\nor, with a config file:\n\n```yaml\n# reward_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: trl-lib/ultrafeedback_binarized\naccelerate_config: zero2 # or path/to/my/accelerate/config.yaml\n```\n\nLaunch with:\n\n```bash\ntrl reward --config reward_config.yaml\n```\n\n\n\n\n```bash\ntrl grpo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name HuggingFaceH4/Polaris-Dataset-53K \\\n --reward_funcs accuracy_reward \\\n --accelerate_config zero2 # or path/to/my/accelerate/config.yaml\n```\n\nor, with a config file:\n\n```yaml\n# grpo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: HuggingFaceH4/Polaris-Dataset-53K\nreward_funcs:\n - accuracy_reward\naccelerate_config: zero2 # or path/to/my/accelerate/config.yaml\n```\n\nLaunch with:\n\n```bash\ntrl grpo --config grpo_config.yaml\n```\n\n\n\n\n```bash\ntrl rloo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name HuggingFaceH4/Polaris-Dataset-53K \\\n --reward_funcs accuracy_reward \\\n --accelerate_config zero2 # or path/to/my/accelerate/config.yaml\n```\n\nor, with a config file:\n\n```yaml\n# rloo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: HuggingFaceH4/Polaris-Dataset-53K\nreward_funcs:\n - accuracy_reward\naccelerate_config: zero2 # or path/to/my/accelerate/config.yaml\n```\n\nLaunch with:\n\n```bash\ntrl rloo --config rloo_config.yaml\n```\n\n\n\n\n```bash\ntrl kto \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/kto-mix-14k \\\n --accelerate_config zero2 # or path/to/my/accelerate/config.yaml\n```\n\nor, with a config file:\n\n```yaml\n# kto_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: trl-lib/kto-mix-14k\naccelerate_config: zero2 # or path/to/my/accelerate/config.yaml\n```\n\nLaunch with:\n\n```bash\ntrl kto --config kto_config.yaml\n```\n\n\n\n\n### Using dataset mixtures\n\nYou can use dataset mixtures to combine multiple datasets into a single training dataset. This is useful for training on diverse data sources or when you want to mix different types of data.\n\n\n\n\n```yaml\n# sft_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndatasets:\n - path: stanfordnlp/imdb\n - path: roneneldan/TinyStories\n```\n\nLaunch with:\n\n```bash\ntrl sft --config sft_config.yaml\n```\n\n\n\n\n```yaml\n# dpo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndatasets:\n - path: BAAI/Infinity-Preference\n - path: argilla/Capybara-Preferences\n```\n\nLaunch with:\n\n```bash\ntrl dpo --config dpo_config.yaml\n```\n\n\n\n\n```yaml\n# reward_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndatasets:\n - path: trl-lib/tldr-preference\n - path: trl-lib/lm-human-preferences-sentiment\n```\n\nLaunch with:\n\n```bash\ntrl reward --config reward_config.yaml\n```\n\n\n\n\n```yaml\n# grpo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndatasets:\n - path: HuggingFaceH4/Polaris-Dataset-53K\n - path: trl-lib/DeepMath-103K\nreward_funcs:\n - accuracy_reward\n```\n\nLaunch with:\n\n```bash\ntrl grpo --config grpo_config.yaml\n```\n\n\n\n\n```yaml\n# rloo_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndatasets:\n - path: HuggingFaceH4/Polaris-Dataset-53K\n - path: trl-lib/DeepMath-103K\nreward_funcs:\n - accuracy_reward\n```\n\nLaunch with:\n\n```bash\ntrl rloo --config rloo_config.yaml\n```\n\n\n\n\n```yaml\n# kto_config.yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndatasets:\n - path: trl-lib/kto-mix-14k\n - path: argilla/ultrafeedback-binarized-preferences-cleaned\n```\n\nLaunch with:\n\n```bash\ntrl kto --config kto_config.yaml\n```\n\n\n\n\nTo see all the available keywords for defining dataset mixtures, refer to the [`scripts.utils.DatasetConfig`] and [`DatasetMixtureConfig`] classes.\n\n## Getting the System Information\n\nYou can get the system information by running the following command:\n\n```bash\ntrl env\n```\n\nThis will print out the system information, including the GPU information, the CUDA version, the PyTorch version, the transformers version, the TRL version, and any optional dependencies that are installed.\n\n```txt\nCopy-paste the following information when reporting an issue:\n\n- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.31\n- Python version: 3.11.9\n- PyTorch version: 2.4.1\n- accelerator(s): NVIDIA H100 80GB HBM3\n- Transformers version: 4.45.0.dev0\n- Accelerate version: 0.34.2\n- Accelerate config: \n - compute_environment: LOCAL_MACHINE\n - distributed_type: DEEPSPEED\n - mixed_precision: no\n - use_cpu: False\n - debug: False\n - num_processes: 4\n - machine_rank: 0\n - num_machines: 1\n - rdzv_backend: static\n - same_network: True\n - main_training_function: main\n - enable_cpu_affinity: False\n - deepspeed_config: {'gradient_accumulation_steps': 4, 'offload_optimizer_device': 'none', 'offload_param_device': 'none', 'zero3_init_flag': False, 'zero_stage': 2}\n - downcast_bf16: no\n - tpu_use_cluster: False\n - tpu_use_sudo: False\n - tpu_env: []\n- Datasets version: 3.0.0\n- HF Hub version: 0.24.7\n- TRL version: 0.12.0.dev0+acb4d70\n- bitsandbytes version: 0.41.1\n- DeepSpeed version: 0.15.1\n- Diffusers version: 0.30.3\n- Liger-Kernel version: 0.3.0\n- LLM-Blender version: 0.0.2\n- OpenAI version: 1.46.0\n- PEFT version: 0.12.0\n- vLLM version: not installed\n```\n\nThis information is required when reporting an issue.\n" }, { "path": "docs/source/community_tutorials.md", "content": "# Community Tutorials\n\nCommunity tutorials are made by active members of the Hugging Face community who want to share their knowledge and expertise with others. They are a great way to learn about the library and its features, and to get started with core classes and modalities.\n\n## Language Models\n\n### Tutorials\n\n| Task | Class | Description | Author | Tutorial | Colab |\n| --- | --- | --- | --- | --- | --- |\n| Reinforcement Learning | [`GRPOTrainer`] | Efficient Online Training with GRPO and vLLM in TRL | [Sergio Paniego](https://huggingface.co/sergiopaniego) | [Link](https://huggingface.co/learn/cookbook/grpo_vllm_online_training) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/grpo_vllm_online_training.ipynb) |\n| Reinforcement Learning | [`GRPOTrainer`] | Post training an LLM for reasoning with GRPO in TRL | [Sergio Paniego](https://huggingface.co/sergiopaniego) | [Link](https://huggingface.co/learn/cookbook/fine_tuning_llm_grpo_trl) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_llm_grpo_trl.ipynb) |\n| Reinforcement Learning | [`GRPOTrainer`] | Mini-R1: Reproduce Deepseek R1 โ€žaha momentโ€œ a RL tutorial | [Philipp Schmid](https://huggingface.co/philschmid) | [Link](https://www.philschmid.de/mini-deepseek-r1) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/philschmid/deep-learning-pytorch-huggingface/blob/main/training/mini-deepseek-r1-aha-grpo.ipynb) |\n| Reinforcement Learning | [`GRPOTrainer`] | RL on LLaMA 3.1-8B with GRPO and Unsloth optimizations | [Andrea Manzoni](https://huggingface.co/AManzoni) | [Link](https://colab.research.google.com/github/amanzoni1/fine_tuning/blob/main/RL_LLama3_1_8B_GRPO.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/amanzoni1/fine_tuning/blob/main/RL_LLama3_1_8B_GRPO.ipynb) | \n| Instruction tuning | [`SFTTrainer`] | Fine-tuning Google Gemma LLMs using ChatML format with QLoRA | [Philipp Schmid](https://huggingface.co/philschmid) | [Link](https://www.philschmid.de/fine-tune-google-gemma) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/philschmid/deep-learning-pytorch-huggingface/blob/main/training/gemma-lora-example.ipynb) |\n| Structured Generation | [`SFTTrainer`] | Fine-tuning Llama-2-7B to generate Persian product catalogs in JSON using QLoRA and PEFT | [Mohammadreza Esmaeilian](https://huggingface.co/Mohammadreza) | [Link](https://huggingface.co/learn/cookbook/en/fine_tuning_llm_to_generate_persian_product_catalogs_in_json_format) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_llm_to_generate_persian_product_catalogs_in_json_format.ipynb) |\n| Preference Optimization | [`DPOTrainer`] | Align Mistral-7b using Direct Preference Optimization for human preference alignment | [Maxime Labonne](https://huggingface.co/mlabonne) | [Link](https://mlabonne.github.io/blog/posts/Fine_tune_Mistral_7b_with_DPO.html) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mlabonne/llm-course/blob/main/Fine_tune_a_Mistral_7b_model_with_DPO.ipynb) |\n| Preference Optimization | [`experimental.orpo.ORPOTrainer`] | Fine-tuning Llama 3 with ORPO combining instruction tuning and preference alignment | [Maxime Labonne](https://huggingface.co/mlabonne) | [Link](https://mlabonne.github.io/blog/posts/2024-04-19_Fine_tune_Llama_3_with_ORPO.html) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1eHNWg9gnaXErdAa8_mcvjMupbSS6rDvi) |\n| Instruction tuning | [`SFTTrainer`] | How to fine-tune open LLMs in 2025 with Hugging Face | [Philipp Schmid](https://huggingface.co/philschmid) | [Link](https://www.philschmid.de/fine-tune-llms-in-2025) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/philschmid/deep-learning-pytorch-huggingface/blob/main/training/fine-tune-llms-in-2025.ipynb) |\n| Step-Level Reasoning | [`GRPOTrainer`] | Supervised Reinforcement Learning (SRL) for step-by-step reasoning with vLLM | [Deepak Swaminathan](https://huggingface.co/s23deepak) | [Link](https://github.com/s23deepak/Supervised-Reinforcement-Learning) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/s23deepak/Supervised-Reinforcement-Learning/blob/main/notebooks/srl_grpo_tutorial.ipynb) |\n\n### Videos\n\n| Task | Title | Author | Video |\n| --- | --- | --- | --- |\n| Instruction tuning | Fine-tuning open AI models using Hugging Face TRL | [Wietse Venema](https://huggingface.co/wietsevenema) | [](https://youtu.be/cnGyyM0vOes) |\n| Instruction tuning | How to fine-tune a smol-LM with Hugging Face, TRL, and the smoltalk Dataset | [Mayurji](https://huggingface.co/iammayur) | [](https://youtu.be/jKdXv3BiLu0) |\n\n\n
\nโš ๏ธ Deprecated features notice for \"How to fine-tune a smol-LM with Hugging Face, TRL, and the smoltalk Dataset\" (click to expand)\n\n> [!WARNING]\n> The tutorial uses two deprecated features:\n>\n> - `SFTTrainer(..., tokenizer=tokenizer)`: Use `SFTTrainer(..., processing_class=tokenizer)` instead, or simply omit it (it will be inferred from the model).\n> - `setup_chat_format(model, tokenizer)`: Use `SFTConfig(..., chat_template_path=\"Qwen/Qwen3-0.6B\")`, where `chat_template_path` specifies the model whose chat template you want to copy.\n\n
\n\n## Vision Language Models\n\n### Tutorials\n\n| Task | Class | Description | Author | Tutorial | Colab |\n| --- | --- | --- | --- | --- | --- |\n| Visual QA | [`SFTTrainer`] | Fine-tuning Qwen2-VL-7B for visual question answering on ChartQA dataset | [Sergio Paniego](https://huggingface.co/sergiopaniego) | [Link](https://huggingface.co/learn/cookbook/fine_tuning_vlm_trl) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_vlm_trl.ipynb) |\n| Visual QA | [`SFTTrainer`] | Fine-tuning SmolVLM with TRL on a consumer GPU | [Sergio Paniego](https://huggingface.co/sergiopaniego) | [Link](https://huggingface.co/learn/cookbook/fine_tuning_smol_vlm_sft_trl) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_smol_vlm_sft_trl.ipynb) |\n| SEO Description | [`SFTTrainer`] | Fine-tuning Qwen2-VL-7B for generating SEO-friendly descriptions from images | [Philipp Schmid](https://huggingface.co/philschmid) | [Link](https://www.philschmid.de/fine-tune-multimodal-llms-with-trl) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/philschmid/deep-learning-pytorch-huggingface/blob/main/training/fine-tune-multimodal-llms-with-trl.ipynb) |\n| Visual QA | [`DPOTrainer`] | PaliGemma ๐Ÿค Direct Preference Optimization | [Merve Noyan](https://huggingface.co/merve) | [Link](https://github.com/merveenoyan/smol-vision/blob/main/PaliGemma_DPO.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/merveenoyan/smol-vision/blob/main/PaliGemma_DPO.ipynb) |\n| Visual QA | [`DPOTrainer`] | Fine-tuning SmolVLM using direct preference optimization (DPO) with TRL on a consumer GPU | [Sergio Paniego](https://huggingface.co/sergiopaniego) | [Link](https://huggingface.co/learn/cookbook/fine_tuning_vlm_dpo_smolvlm_instruct) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_vlm_dpo_smolvlm_instruct.ipynb) |\n| Object Detection Grounding | [`SFTTrainer`] | Fine tuning a VLM for Object Detection Grounding using TRL | [Sergio Paniego](https://huggingface.co/sergiopaniego) | [Link](https://huggingface.co/learn/cookbook/fine_tuning_vlm_object_detection_grounding) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_vlm_object_detection_grounding.ipynb) |\n| Visual QA | [`DPOTrainer`] | Fine-Tuning a Vision Language Model with TRL using MPO | [Sergio Paniego](https://huggingface.co/sergiopaniego) | [Link](https://huggingface.co/learn/cookbook/fine_tuning_vlm_mpo) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_vlm_mpo.ipynb) |\n| Reinforcement Learning | [`GRPOTrainer`] | Post training a VLM for reasoning with GRPO using TRL | [Sergio Paniego](https://huggingface.co/sergiopaniego) | [Link](https://huggingface.co/learn/cookbook/fine_tuning_vlm_grpo_trl) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_vlm_grpo_trl.ipynb) |\n\n## Speech Language Models\n\n### Tutorials\n\n| Task | Class | Description | Author | Tutorial |\n| --- | --- | --- | --- | --- |\n| Text-to-Speech | [`GRPOTrainer`] | Post training a Speech Language Model with GRPO using TRL | [Steven Zheng](https://huggingface.co/Steveeeeeeen) | [Link](https://huggingface.co/blog/Steveeeeeeen/llasa-grpo) |\n\n## Contributing\n\nIf you have a tutorial that you would like to add to this list, please open a PR to add it. We will review it and merge it if it is relevant to the community.\n" }, { "path": "docs/source/cpo_trainer.md", "content": "# CPO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-CPO-blue)](https://huggingface.co/models?other=cpo,trl)\n\n## Overview\n\nContrastive Preference Optimization (CPO) as introduced in the paper [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://huggingface.co/papers/2401.08417) by [Haoran Xu](https://huggingface.co/haoranxu), [Amr Sharaf](https://huggingface.co/amrsharaf), [Yunmo Chen](https://huggingface.co/yunmochen), Weiting Tan, Lingfeng Shen, Benjamin Van Durme, [Kenton Murray](https://huggingface.co/Kenton), and [Young Jin Kim](https://huggingface.co/ykim362). At a high level, CPO trains models to avoid generating adequate, but not perfect, translations in Machine Translation (MT) tasks. However, CPO is a general approximation of the DPO loss and can be applied to other domains, such as chat.\n\nCPO aims to mitigate two fundamental shortcomings of SFT. First, SFTโ€™s methodology of minimizing the discrepancy between predicted outputs and gold-standard references inherently caps model performance at the quality level of the training data. Secondly, SFT lacks a mechanism to prevent the model from rejecting mistakes in translations. The CPO objective is derived from the DPO objective.\n\n## Quick start\n\nThis example demonstrates how to train a model using the CPO method. We use the [Qwen 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) as the base model. We use the preference data from the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback). You can view the data in the dataset here:\n\n\n\nBelow is the script to train the model:\n\n```python\n# train_cpo.py\nfrom datasets import load_dataset\nfrom trl.experimental.cpo import CPOConfig, CPOTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\ntrain_dataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\ntraining_args = CPOConfig(output_dir=\"Qwen2-0.5B-CPO\")\ntrainer = CPOTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset)\ntrainer.train()\n```\n\nExecute the script using the following command:\n\n```bash\naccelerate launch train_cpo.py\n```\n\n## Expected dataset type\n\nCPO requires a [preference dataset](dataset_formats#preference). The [`experimental.cpo.CPOTrainer`] supports both [conversational](dataset_formats#conversational) and [standard](dataset_formats#standard) dataset formats. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\n## Example script\n\nWe provide an example script to train a model using the CPO method. The script is available in [`examples/scripts/cpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/cpo.py)\n\nTo test the CPO script with the [Qwen2 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) on the [UltraFeedback dataset](https://huggingface.co/datasets/trl-lib/ultrafeedback_binarized), run the following command:\n\n```bash\naccelerate launch examples/scripts/cpo.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --num_train_epochs 1 \\\n --output_dir Qwen2-0.5B-CPO\n```\n\n## Logged metrics\n\nWhile training and evaluating, we record the following reward metrics:\n\n* `rewards/chosen`: the mean log probabilities of the policy model for the chosen responses scaled by beta\n* `rewards/rejected`: the mean log probabilities of the policy model for the rejected responses scaled by beta\n* `rewards/accuracies`: mean of how often the chosen rewards are > than the corresponding rejected rewards\n* `rewards/margins`: the mean difference between the chosen and corresponding rejected rewards\n* `nll_loss`: the mean negative log likelihood loss of the policy model for the chosen responses\n\n## CPO variants\n\n### Simple Preference Optimization (SimPO)\n\n[Simple Preference Optimization](https://huggingface.co/papers/2405.14734) (SimPO) by [Yu Meng](https://huggingface.co/yumeng5), [Mengzhou Xia](https://huggingface.co/mengzhouxia), and [Danqi Chen](https://huggingface.co/cdq10131) proposes a simpler and more effective preference optimization algorithm than DPO without using a reference model. The key designs in SimPO are (1) using length-normalized log likelihood as the implicit reward, and (2) incorporating a target reward margin in the Bradley-Terry ranking objective. The official code can be found at [princeton-nlp/SimPO](https://github.com/princeton-nlp/SimPO).\n\nThe abstract from the paper is the following:\n\n> Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm that reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability. In this work, we propose SimPO, a simpler yet more effective approach. The effectiveness of SimPO is attributed to a key design: using the average log probability of a sequence as the implicit reward. This reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient. Additionally, we introduce a target reward margin to the Bradley-Terry objective to encourage a larger margin between the winning and losing responses, further enhancing the algorithm's performance. We compare SimPO to DPO and its latest variants across various state-of-the-art training setups, including both base and instruction-tuned models like Mistral and Llama3. We evaluated on extensive instruction-following benchmarks, including AlpacaEval 2, MT-Bench, and the recent challenging Arena-Hard benchmark. Our results demonstrate that SimPO consistently and significantly outperforms existing approaches without substantially increasing response length. Specifically, SimPO outperforms DPO by up to 6.4 points on AlpacaEval 2 and by up to 7.5 points on Arena-Hard. Our top-performing model, built on Llama3-8B-Instruct, achieves a remarkable 44.7 length-controlled win rate on AlpacaEval 2 -- surpassing Claude 3 Opus on the leaderboard, and a 33.8 win rate on Arena-Hard -- making it the strongest 8B open-source model.\n\nThe SimPO loss is integrated in the [`experimental.cpo.CPOTrainer`], as it's an alternative loss that adds a reward margin, allows for length normalization, and does not use BC regularization. To use this loss, just turn on `loss_type=\"simpo\"` and `cpo_alpha=0.0` in the [`experimental.cpo.CPOConfig`] and set the `simpo_gamma` to a recommended value.\n\n### CPO-SimPO\n\nWe also offer the combined use of CPO and SimPO, which enables more stable training and improved performance. Learn more details at [CPO-SimPO GitHub](https://github.com/fe1ixxu/CPO_SIMPO). To use this method, simply enable SimPO by setting `loss_type=\"simpo\"` and a non-zero `cpo_alpha` in the [`experimental.cpo.CPOConfig`].\n\n### AlphaPO\n\nThe [AlphaPO -- Reward shape matters for LLM alignment](https://huggingface.co/papers/2501.03884) (AlphaPO) method by Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, [Jiwoo Hong](https://huggingface.co/JW17), Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Jason Zhu, Natesh Pillai, and S. Sathiya Keerthi is also implemented in the [`experimental.cpo.CPOTrainer`]. AlphaPO is an alternative method that applies a transformation to the reward function shape in the context of SimPO loss. The abstract from the paper is the following:\n\n> Reinforcement Learning with Human Feedback (RLHF) and its variants have made huge strides toward the effective alignment of large language models (LLMs) to follow instructions and reflect human values. More recently, Direct Alignment Algorithms (DAAs) have emerged in which the reward modeling stage of RLHF is skipped by characterizing the reward directly as a function of the policy being learned. Some popular examples of DAAs include Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO). These methods often suffer from likelihood displacement, a phenomenon by which the probabilities of preferred responses are often reduced undesirably. In this paper, we argue that, for DAAs the reward (function) shape matters. We introduce AlphaPO, a new DAA method that leverages an ฮฑ-parameter to help change the shape of the reward function beyond the standard log reward. AlphaPO helps maintain fine-grained control over likelihood displacement and overoptimization. Compared to SimPO, one of the best performing DAAs, AlphaPO leads to about 7% to 10% relative improvement in alignment performance for the instruct versions of Mistral-7B and Llama3-8B while achieving 15% to 50% relative improvement over DPO on the same models. The analysis and results presented highlight the importance of the reward shape and how one can systematically change it to affect training dynamics, as well as improve alignment performance.\n\nTo use this loss as described in the paper, we can set the `loss_type=\"alphapo\"` which automatically sets `loss_type=\"simpo\"` and `cpo_alpha=0.0`, together with `alpha` and `simpo_gamma` to recommended values in the [`experimental.cpo.CPOConfig`]. Alternatively, you can manually set `loss_type=\"simpo\"`, `cpo_alpha=0.0`, together with `alpha` and `simpo_gamma` to recommended values. Other variants of this method are also possible, such as setting `loss_type=\"ipo\"` and `alpha` to any non-zero value.\n\n## Loss functions\n\nThe CPO algorithm supports several loss functions. The loss function can be set using the `loss_type` parameter in the [`experimental.cpo.CPOConfig`]. The following loss functions are supported:\n\n| `loss_type=` | Description |\n| --- | --- |\n| `\"sigmoid\"` (default) | Given the preference data, we can fit a binary classifier according to the Bradley-Terry model, and in fact, the [DPO](https://huggingface.co/papers/2305.18290) authors propose the sigmoid loss on the normalized likelihood via the `logsigmoid` to fit a logistic regression. |\n| `\"hinge\"` | The [RSO](https://huggingface.co/papers/2309.06657) authors propose to use a hinge loss on the normalized likelihood from the [SLiC](https://huggingface.co/papers/2305.10425) paper. In this case, the `beta` is the reciprocal of the margin. |\n| `\"ipo\"` | The [IPO](https://huggingface.co/papers/2310.12036) authors provide a deeper theoretical understanding of the DPO algorithms and identify an issue with overfitting and propose an alternative loss. In this case, the `beta` is the reciprocal of the gap between the log-likelihood ratios of the chosen vs the rejected completion pair, and thus the smaller the `beta`, the larger this gap is. As per the paper, the loss is averaged over log-likelihoods of the completion (unlike DPO, which is summed only). |\n| `\"simpo\"` | The [SimPO](https://huggingface.co/papers/2405.14734) method is also implemented in the [`experimental.cpo.CPOTrainer`]. SimPO is an alternative loss that adds a reward margin, allows for length normalization, and does not use BC regularization. To use this loss, simply set `loss_type=\"simpo\"` and `cpo_alpha=0.0` in the [`experimental.cpo.CPOConfig`] and `simpo_gamma` to a recommended value. |\n| `\"alphapo\"` | The [AlphaPO](https://huggingface.co/papers/2501.03884) method is also implemented in the [`experimental.cpo.CPOTrainer`]. This is syntactic sugar that automatically sets `loss_type=\"simpo\"` and `cpo_alpha=0.0`. AlphaPO applies a transformation to the reward function shape in the context of SimPO loss when the `alpha` parameter is non-zero. |\n\n### For Mixture of Experts Models: Enabling the auxiliary loss\n\nMOEs are the most efficient if the load is about equally distributed between experts. \nTo ensure that we train MOEs similarly during preference-tuning, it is beneficial to add the auxiliary loss from the load balancer to the final loss.\n\nThis option is enabled by setting `output_router_logits=True` in the model config (e.g., [`~transformers.MixtralConfig`]). \nTo scale how much the auxiliary loss contributes to the total loss, use the hyperparameter `router_aux_loss_coef=...` (default: `0.001`) in the model config.\n\n## CPOTrainer\n\n[[autodoc]] experimental.cpo.CPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## CPOConfig\n\n[[autodoc]] experimental.cpo.CPOConfig\n" }, { "path": "docs/source/customization.md", "content": "# Training customization\n\nTRL is designed with modularity in mind so that users are able to efficiently customize the training loop for their needs. Below are examples on how you can apply and test different techniques.\n\n> [!NOTE]\n> Although these examples use the [`DPOTrainer`], these customization methods apply to most (if not all) trainers in TRL.\n\n## Use different optimizers and schedulers\n\nBy default, the [`DPOTrainer`] creates a `torch.optim.AdamW` optimizer. You can create and define a different optimizer and pass it to [`DPOTrainer`] as follows:\n\n```python\nfrom datasets import load_dataset\nfrom torch import optim\nfrom transformers import AutoModelForCausalLM\nfrom trl import DPOTrainer\n\ndataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2.5-0.5B-Instruct\")\noptimizer = optim.SGD(model.parameters(), lr=1e-6)\n\ntrainer = DPOTrainer(\n model=model,\n train_dataset=dataset,\n optimizers=(optimizer, None),\n)\ntrainer.train()\n```\n\n### Add a learning rate scheduler\n\nYou can also add learning rate schedulers by passing both optimizer and scheduler:\n\n```python\nfrom torch import optim\n\noptimizer = optim.AdamW(model.parameters(), lr=1e-6)\nlr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)\n\ntrainer = DPOTrainer(..., optimizers=(optimizer, lr_scheduler))\n```\n\n## Pass 8-bit reference models\n\nSince `trl` supports all keyword arguments when loading a model from `transformers` using `from_pretrained`, you can also leverage `load_in_8bit` from `transformers` for more memory efficient fine-tuning.\n\nRead more about 8-bit model loading in `transformers` [Load in 8bit or 4bit](https://huggingface.co/docs/transformers/en/peft).\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)\nref_model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2.5-0.5B-Instruct\", quantization_config=quantization_config)\n\ntrainer = DPOTrainer(..., ref_model=ref_model)\n```\n\n## Add custom callbacks\n\nYou can customize the training loop by adding callbacks for logging, monitoring, or early stopping. Callbacks allow you to execute custom code at specific points during training.\n\n```python\nfrom transformers import TrainerCallback\n\n\nclass CustomLoggingCallback(TrainerCallback):\n def on_log(self, args, state, control, logs=None, **kwargs):\n if logs is not None:\n print(f\"Step {state.global_step}: {logs}\")\n\n\ntrainer = DPOTrainer(..., callbacks=[CustomLoggingCallback()])\n```\n\n## Add custom evaluation metrics\n\nYou can define custom evaluation metrics to track during training. This is useful for monitoring model performance on specific tasks.\n\n```python\ndef compute_metrics(eval_preds):\n logits, labels = eval_preds\n # Add your metric computation here\n return {\"custom_metric\": 0.0}\n\n\ntraining_args = DPOConfig(..., eval_strategy=\"steps\", eval_steps=100)\n\ntrainer = DPOTrainer(..., eval_dataset=eval_dataset, compute_metrics=compute_metrics)\n```\n\n## Use mixed precision training\n\nMixed precision training can significantly speed up training and reduce memory usage. You can enable it by setting `bf16=True` or `fp16=True` in the training config.\n\n```python\n# Use bfloat16 precision (recommended for modern GPUs)\ntraining_args = DPOConfig(..., bf16=True)\n```\n\nNote: Use `bf16=True` for Ampere GPUs (A100, RTX 30xx) or newer, and `fp16=True` for older GPUs.\n\n## Use gradient accumulation\n\nWhen training with limited GPU memory, gradient accumulation allows you to simulate larger batch sizes by accumulating gradients over multiple steps before updating weights.\n\n```python\n# Simulate a batch size of 32 with per_device_train_batch_size=4 and gradient_accumulation_steps=8\ntraining_args = DPOConfig(\n ...,\n per_device_train_batch_size=4,\n gradient_accumulation_steps=8,\n)\n```\n" }, { "path": "docs/source/data_utils.md", "content": "# Data Utilities\n\n## is_conversational\n\n[[autodoc]] is_conversational\n\n## maybe_convert_to_chatml\n\n[[autodoc]] maybe_convert_to_chatml\n\n## extract_prompt\n\n[[autodoc]] extract_prompt\n\n## unpair_preference_dataset\n\n[[autodoc]] unpair_preference_dataset\n" }, { "path": "docs/source/dataset_formats.md", "content": "# Dataset formats and types\n\nThis guide provides an overview of the dataset formats and types supported by each trainer in TRL.\n\n## Overview of the dataset formats and types\n\n- The *format* of a dataset refers to how the data is structured, typically categorized as either *standard* or *conversational*.\n- The *type* is associated with the specific task the dataset is designed for, such as *prompt-only* or *preference*. Each type is characterized by its columns, which vary according to the task, as shown in the table.\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Type \\ FormatStandardConversational
Language modeling\n 
{\"text\": \"The sky is blue.\"}
\n 		\n 
{\"messages\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n              {\"role\": \"assistant\", \"content\": \"It is blue.\"}]}
\n
Prompt-only\n 
{\"prompt\": \"The sky is\"}
\n 		\n 
{\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}]}
\n
Prompt-completion\n 
{\"prompt\": \"The sky is\",\n \"completion\": \" blue.\"}
\n 		\n 
{\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}]}
\n
Preference\n 
{\"prompt\": \"The sky is\",\n \"chosen\": \" blue.\",\n \"rejected\": \" green.\"}
\n or, with implicit prompt:\n 
{\"chosen\": \"The sky is blue.\",\n \"rejected\": \"The sky is green.\"}
\n 		\n 
{\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": \"It is green.\"}]}
\n or, with implicit prompt:\n 
{\"chosen\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n              {\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n                {\"role\": \"assistant\", \"content\": \"It is green.\"}]}
\n
Unpaired preference\n 
{\"prompt\": \"The sky is\",\n \"completion\": \" blue.\",\n \"label\": True}
\n 		\n 
{\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is green.\"}],\n \"label\": False}
\n
Stepwise supervision\n 
{\"prompt\": \"Which number is larger, 9.8 or 9.11?\",\n \"completions\": [\"The fractional part of 9.8 is 0.8.\",\n                 \"The fractional part of 9.11 is 0.11.\",\n                 \"0.11 is greater than 0.8.\",\n                 \"Hence, 9.11 > 9.8.\"],\n \"labels\": [True, True, False, False]}
\n
\n\n### Formats\n\n#### Standard\n\nThe standard dataset format typically consists of plain text strings. The columns in the dataset vary depending on the task. This is the format expected by TRL trainers. Below are examples of standard dataset formats for different tasks:\n\n```python\n# Language modeling\nlanguage_modeling_example = {\"text\": \"The sky is blue.\"}\n# Preference\npreference_example = {\"prompt\": \"The sky is\", \"chosen\": \" blue.\", \"rejected\": \" green.\"}\n# Unpaired preference\nunpaired_preference_example = {\"prompt\": \"The sky is\", \"completion\": \" blue.\", \"label\": True}\n```\n\n#### Conversational\n\nConversational datasets are used for tasks involving dialogues or chat interactions between users and assistants. Unlike standard dataset formats, these contain sequences of messages where each message has a `role` (e.g., `\"user\"` or `\"assistant\"`) and `content` (the message text).\n\n```python\nmessages = [\n {\"role\": \"user\", \"content\": \"Hello, how are you?\"},\n {\"role\": \"assistant\", \"content\": \"I'm doing great. How can I help you today?\"},\n {\"role\": \"user\", \"content\": \"I'd like to show off how chat templating works!\"},\n]\n```\n\nJust like standard datasets, the columns in conversational datasets vary depending on the task. Below are examples of conversational dataset formats for different tasks:\n\n```python\n# Prompt-completion\nprompt_completion_example = {\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}]}\n# Preference\npreference_example = {\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": \"It is green.\"}],\n}\n```\n\n#### Tool Calling\n\nSome chat templates support *tool calling*, which allows the model to interact with external functionsโ€”referred to as **tools**โ€”during generation. This extends the conversational capabilities of the model by enabling it to output a `\"tool_calls\"` field instead of a standard `\"content\"` message whenever it decides to invoke a tool.\n\nAfter the assistant initiates a tool call, the tool executes and returns its output. The assistant can then process this output and continue the conversation accordingly.\n\nHereโ€™s a simple example of a tool-calling interaction:\n\n```python\nmessages = [\n {\"role\": \"user\", \"content\": \"Turn on the living room lights.\"},\n {\"role\": \"assistant\", \"tool_calls\": [\n {\"type\": \"function\", \"function\": {\n \"name\": \"control_light\",\n \"arguments\": {\"room\": \"living room\", \"state\": \"on\"}\n }}]\n },\n {\"role\": \"tool\", \"name\": \"control_light\", \"content\": \"The lights in the living room are now on.\"},\n {\"role\": \"assistant\", \"content\": \"Done!\"}\n]\n```\n\nWhen preparing datasets for Supervised Fine-Tuning (SFT) with tool calling, it is important that your dataset includes an additional column named `tools`. This column contains the list of available tools for the model, which is usually used by the chat template to construct the system prompt.\n\nThe tools must be specified in a codified JSON schema format. You can automatically generate this schema from Python function signatures using the [`~transformers.utils.get_json_schema`] utility:\n\n```python\nimport json\nfrom transformers.utils import get_json_schema\n\ndef control_light(room: str, state: str) -> str:\n \"\"\"\n Controls the lights in a room.\n\n Args:\n room: The name of the room.\n state: The desired state of the light (\"on\" or \"off\").\n\n Returns:\n str: A message indicating the new state of the lights.\n \"\"\"\n return f\"The lights in {room} are now {state}.\"\n\n# Generate JSON schema\njson_schema = get_json_schema(control_light)\n```\n\nThe generated schema would look like:\n\n```python\n{\"type\": \"function\", \"function\": {\"name\": \"control_light\", \"description\": \"Controls the lights in a room.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"room\": {\"type\": \"string\", \"description\": \"The name of the room.\"}, \"state\": {\"type\": \"string\", \"description\": \"The desired state of the light (\\\"on\\\" or \\\"off\\\").\"}}, \"required\": [\"room\", \"state\"]}, \"return\": {\"type\": \"string\", \"description\": \"str: A message indicating the new state of the lights.\"}}}\n```\n\nA complete dataset entry for SFT might look like:\n\n```python\n{\"messages\": messages, \"tools\": [json_schema]}\n```\n\nTo get a `Dataset` you need to use the `Json()` type for tool arguments since they are arbitrary JSON objects, and not dictionaries with fixed fields and types:\n\n```python\nfrom datasets import Dataset\n\ndata = [\n {\"messages\": messages1, \"tools\": [json_schema1]},\n {\"messages\": messages2, \"tools\": [json_schema2]},\n]\n# auto-apply the Json() type\ndataset = Dataset.from_list(data, on_mixed_types=\"use_json\")\n\n# or specify the features manually\nfrom datasets import Features, Json, List, Value\n\nfeatures = Features(\n {\n \"messages\": List({\"role\": Value(\"string\"), \"content\": Value(\"string\"), \"tool_calls\": List(Json())}),\n \"tools\": List(Json()),\n }\n)\ndataset = Dataset.from_list(data, features=features)\n```\n\nOn older versions of `datasets` (<4.7.0) that don't have the `Json()` type, you should store `tools` as a JSON `str` (with `json.dumps([...])`):\n\n```python\ndataset = Dataset.from_list(\n [{\"messages\": messages1, \"tools\": json.dumps([json_schema1])},\n {\"messages\": messages2, \"tools\": json.dumps([json_schema2])}]\n)\n```\n\nFor more detailed information on tool calling, refer to the [Tool Calling section in the `transformers` documentation](https://huggingface.co/docs/transformers/chat_extras#tools-and-rag) and the blog post [Tool Use, Unified](https://huggingface.co/blog/unified-tool-use).\n\n### Harmony\n\nThe [Harmony response format](https://cookbook.openai.com/articles/openai-harmony) was introduced with the [OpenAI GPT OSS models](https://huggingface.co/collections/openai/gpt-oss-68911959590a1634ba11c7a4). It extends the conversational format by adding richer structure for reasoning, function calls, and metadata about the modelโ€™s behavior. Key features include:\n\n- **Developer role** โ€“ Provides high level instructions (similar to a system prompt) and lists available tools.\n- **Channels** โ€“ Separate types of assistant output into distinct streams:\n\n - `analysis` โ€“ for internal reasoning, from the key `\"thinking\"`\n - `final` โ€“ for the user-facing answer, from the key `\"content\"`\n - `commentary` โ€“ for tool calls or meta notes\n\n- **Reasoning effort** โ€“ Signals how much thinking the model should show (e.g., `\"low\"`, `\"medium\"`, `\"high\"`).\n- **Model identity** โ€“ Explicitly defines the assistantโ€™s persona.\n\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"openai/gpt-oss-20b\")\n\nmessages = [\n {\"role\": \"developer\", \"content\": \"Use a friendly tone.\"},\n {\"role\": \"user\", \"content\": \"What is the meaning of life?\"},\n {\"role\": \"assistant\", \"thinking\": \"Deep reflection...\", \"content\": \"The final answer is...\"},\n]\n\nprint(\n tokenizer.apply_chat_template(\n messages,\n tokenize=False,\n reasoning_effort=\"low\",\n model_identity=\"You are HuggingGPT, a large language model trained by Hugging Face.\",\n )\n)\n```\n\nThis produces:\n\n```txt\n<|start|>system<|message|>You are HuggingGPT, a large language model trained by Hugging Face.\nKnowledge cutoff: 2024-06\nCurrent date: 2025-08-03\n\nReasoning: low\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions\n\nUse a friendly tone.<|end|><|start|>user<|message|>What is the meaning of life?<|end|><|start|>assistant<|channel|>analysis<|message|>Deep reflection...<|end|><|start|>assistant<|channel|>final<|message|>The final answer is...<|return|>\n```\n\nFor full details on message structure, supported fields, and advanced usage, see the [Harmony documentation](https://cookbook.openai.com/articles/openai-harmony).\n\n### Types\n\n#### Language modeling\n\nA language modeling dataset consists of a column `\"text\"` (or `\"messages\"` for conversational datasets) containing a full sequence of text.\n\n```python\n# Standard format\nlanguage_modeling_example = {\"text\": \"The sky is blue.\"}\n# Conversational format\nlanguage_modeling_example = {\"messages\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"}\n]}\n```\n\n#### Prompt-only\n\nIn a prompt-only dataset, only the initial prompt (the question or partial sentence) is provided under the key `\"prompt\"`. The training typically involves generating completion based on this prompt, where the model learns to continue or complete the given input.\n\n```python\n# Standard format\nprompt_only_example = {\"prompt\": \"The sky is\"}\n# Conversational format\nprompt_only_example = {\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}]}\n```\n\nFor examples of prompt-only datasets, refer to the [Prompt-only datasets collection](https://huggingface.co/collections/trl-lib/prompt-only-datasets-677ea25245d20252cea00368).\n\n> [!TIP]\n> While both the prompt-only and language modeling types are similar, they differ in how the input is handled. In the prompt-only type, the prompt represents a partial input that expects the model to complete or continue, while in the language modeling type, the input is treated as a complete sentence or sequence. These two types are processed differently by TRL. Below is an example showing the difference in the output of the `apply_chat_template` function for each type:\n>\n> ```python\n> from transformers import AutoTokenizer\n> from trl import apply_chat_template\n>\n> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-128k-instruct\")\n>\n> # Example for prompt-only type\n> prompt_only_example = {\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}]}\n> apply_chat_template(prompt_only_example, tokenizer)\n> # Output: {'prompt': '<|user|>\\nWhat color is the sky?<|end|>\\n<|assistant|>\\n'}\n>\n> # Example for language modeling type\n> lm_example = {\"messages\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}]}\n> apply_chat_template(lm_example, tokenizer)\n> # Output: {'text': '<|user|>\\nWhat color is the sky?<|end|>\\n<|endoftext|>'}\n> ```\n>\n> - The prompt-only output includes a `'<|assistant|>\\n'`, indicating the beginning of the assistantโ€™s turn and expecting the model to generate a completion.\n> - In contrast, the language modeling output treats the input as a complete sequence and terminates it with `'<|endoftext|>'`, signaling the end of the text and not expecting any additional content.\n\n#### Prompt-completion\n\nA prompt-completion dataset includes a `\"prompt\"` and a `\"completion\"`.\n\n```python\n# Standard format\nprompt_completion_example = {\"prompt\": \"The sky is\", \"completion\": \" blue.\"}\n# Conversational format\nprompt_completion_example = {\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}]}\n```\n\nFor examples of prompt-completion datasets, refer to the [Prompt-completion datasets collection](https://huggingface.co/collections/trl-lib/prompt-completion-datasets-677ea2bb20bbb6bdccada216).\n\n#### Preference\n\nA preference dataset is used for tasks where the model is trained to choose between two or more possible completions to the same prompt. This dataset includes a `\"prompt\"`, a `\"chosen\"` completion, and a `\"rejected\"` completion. The model is trained to select the `\"chosen\"` response over the `\"rejected\"` response.\nSome datasets may not include the `\"prompt\"` column, in which case the prompt is implicit and directly included in the `\"chosen\"` and `\"rejected\"` completions. We recommend using explicit prompts whenever possible.\n\n```python\n# Standard format\n## Explicit prompt (recommended)\npreference_example = {\"prompt\": \"The sky is\", \"chosen\": \" blue.\", \"rejected\": \" green.\"}\n# Implicit prompt\npreference_example = {\"chosen\": \"The sky is blue.\", \"rejected\": \"The sky is green.\"}\n\n# Conversational format\n## Explicit prompt (recommended)\npreference_example = {\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": \"It is green.\"}]}\n## Implicit prompt\npreference_example = {\"chosen\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is green.\"}]}\n```\n\nFor examples of preference datasets, refer to the [Preference datasets collection](https://huggingface.co/collections/trl-lib/preference-datasets-677e99b581018fcad9abd82c).\n\nSome preference datasets can be found with [the tag `dpo` on Hugging Face Hub](https://huggingface.co/datasets?other=dpo). You can also explore the [librarian-bots' DPO Collections](https://huggingface.co/collections/librarian-bots/direct-preference-optimization-datasets-66964b12835f46289b6ef2fc) to identify preference datasets.\n\n#### Unpaired preference\n\nAn unpaired preference dataset is similar to a preference dataset but instead of having `\"chosen\"` and `\"rejected\"` completions for the same prompt, it includes a single `\"completion\"` and a `\"label\"` indicating whether the completion is preferred or not.\n\n```python\n# Standard format\nunpaired_preference_example = {\"prompt\": \"The sky is\", \"completion\": \" blue.\", \"label\": True}\n# Conversational format\nunpaired_preference_example = {\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"label\": True}\n```\n\nFor examples of unpaired preference datasets, refer to the [Unpaired preference datasets collection](https://huggingface.co/collections/trl-lib/unpaired-preference-datasets-677ea22bf5f528c125b0bcdf).\n\n#### Stepwise supervision\n\nA stepwise (or process) supervision dataset is similar to an [unpaired preference](#unpaired-preference) dataset but includes multiple steps of completions, each with its own label. This structure is useful for tasks that need detailed, step-by-step labeling, such as reasoning tasks. By evaluating each step separately and providing targeted labels, this approach helps identify precisely where the reasoning is correct and where errors occur, allowing for targeted feedback on each part of the reasoning process.\n\n```python\nstepwise_example = {\n \"prompt\": \"Which number is larger, 9.8 or 9.11?\",\n \"completions\": [\"The fractional part of 9.8 is 0.8, while the fractional part of 9.11 is 0.11.\", \"Since 0.11 is greater than 0.8, the number 9.11 is larger than 9.8.\"],\n \"labels\": [True, False]\n}\n```\n\nFor examples of stepwise supervision datasets, refer to the [Stepwise supervision datasets collection](https://huggingface.co/collections/trl-lib/stepwise-supervision-datasets-677ea27fd4c5941beed7a96e).\n\n## Which dataset type to use?\n\nChoosing the right dataset type depends on the task you are working on and the specific requirements of the TRL trainer you are using. Below is a brief overview of the dataset types supported by each TRL trainer.\n\n| Trainer | Expected dataset type |\n| --- | --- |\n| [`DPOTrainer`] | [Preference (explicit prompt recommended)](#preference) |\n| [`GRPOTrainer`] | [Prompt-only](#prompt-only) |\n| [`RewardTrainer`] | [Preference (implicit prompt recommended)](#preference) |\n| [`RLOOTrainer`] | [Prompt-only](#prompt-only) |\n| [`SFTTrainer`] | [Language modeling](#language-modeling) or [Prompt-completion](#prompt-completion) |\n| [`experimental.bco.BCOTrainer`] | [Unpaired preference](#unpaired-preference) or [Preference (explicit prompt recommended)](#preference) |\n| [`experimental.cpo.CPOTrainer`] | [Preference (explicit prompt recommended)](#preference) |\n| [`experimental.gkd.GKDTrainer`] | [Prompt-completion](#prompt-completion) |\n| [`experimental.kto.KTOTrainer`] | [Unpaired preference](#unpaired-preference) or [Preference (explicit prompt recommended)](#preference) |\n| [`experimental.nash_md.NashMDTrainer`] | [Prompt-only](#prompt-only) |\n| [`experimental.online_dpo.OnlineDPOTrainer`] | [Prompt-only](#prompt-only) |\n| [`experimental.orpo.ORPOTrainer`] | [Preference (explicit prompt recommended)](#preference) |\n| [`experimental.ppo.PPOTrainer`] | Tokenized language modeling |\n| [`experimental.prm.PRMTrainer`] | [Stepwise supervision](#stepwise-supervision) |\n| [`experimental.xpo.XPOTrainer`] | [Prompt-only](#prompt-only) |\n\n## Using any dataset with TRL: preprocessing and conversion\n\nMany datasets come in formats tailored to specific tasks, which might not be directly compatible with TRL. To use such datasets with TRL, you may need to preprocess and convert them into the required format.\n\nTo make this easier, we provide a set of [example scripts](https://github.com/huggingface/trl/tree/main/examples/datasets) that cover common dataset conversions.\n\n### Example: UltraFeedback dataset\n\nLetโ€™s take the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback) as an example. Here's a preview of the dataset:\n\n\n\nAs shown above, the dataset format does not match the expected structure. Itโ€™s not in a conversational format, the column names differ, and the results pertain to different models (e.g., Bard, GPT-4) and aspects (e.g., \"helpfulness\", \"honesty\").\n\nBy using the provided conversion script [`examples/datasets/ultrafeedback.py`](https://github.com/huggingface/trl/tree/main/examples/datasets/ultrafeedback.py), you can transform this dataset into an unpaired preference type, and push it to the Hub:\n\n```sh\npython examples/datasets/ultrafeedback.py --push_to_hub --repo_id trl-lib/ultrafeedback-gpt-3.5-turbo-helpfulness\n```\n\nOnce converted, the dataset will look like this:\n\n\n\nNow, you can use this dataset with TRL!\n\nBy adapting the provided scripts or creating your own, you can convert any dataset into a format compatible with TRL.\n\n## Utilities for converting dataset types\n\nThis section provides example code to help you convert between different dataset types. While some conversions can be performed after applying the chat template (i.e., in the standard format), we recommend performing the conversion before applying the chat template to ensure it works consistently.\n\nFor simplicity, some of the examples below do not follow this recommendation and use the standard format. However, the conversions can be applied directly to the conversational format without modification.\n\n| From \\ To | Language modeling | Prompt-completion | Prompt-only | Preference with implicit prompt | Preference | Unpaired preference | Stepwise supervision |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Language modeling | N/A | N/A | N/A | N/A | N/A | N/A | N/A |\n| Prompt-completion | [๐Ÿ”—](#from-prompt-completion-to-language-modeling-dataset) | N/A | [๐Ÿ”—](#from-prompt-completion-to-prompt-only-dataset) | N/A | N/A | N/A | N/A |\n| Prompt-only | N/A | N/A | N/A | N/A | N/A | N/A | N/A |\n| Preference with implicit prompt | [๐Ÿ”—](#from-preference-with-implicit-prompt-to-language-modeling-dataset) | [๐Ÿ”—](#from-preference-with-implicit-prompt-to-prompt-completion-dataset) | [๐Ÿ”—](#from-preference-with-implicit-prompt-to-prompt-only-dataset) | N/A | [๐Ÿ”—](#from-implicit-to-explicit-prompt-preference-dataset) | [๐Ÿ”—](#from-preference-with-implicit-prompt-to-unpaired-preference-dataset) | N/A |\n| Preference | [๐Ÿ”—](#from-preference-to-language-modeling-dataset) | [๐Ÿ”—](#from-preference-to-prompt-completion-dataset) | [๐Ÿ”—](#from-preference-to-prompt-only-dataset) | [๐Ÿ”—](#from-explicit-to-implicit-prompt-preference-dataset) | N/A | [๐Ÿ”—](#from-preference-to-unpaired-preference-dataset) | N/A |\n| Unpaired preference | [๐Ÿ”—](#from-unpaired-preference-to-language-modeling-dataset) | [๐Ÿ”—](#from-unpaired-preference-to-prompt-completion-dataset) | [๐Ÿ”—](#from-unpaired-preference-to-prompt-only-dataset) | N/A | N/A | N/A | N/A |\n| Stepwise supervision | [๐Ÿ”—](#from-stepwise-supervision-to-language-modeling-dataset) | [๐Ÿ”—](#from-stepwise-supervision-to-prompt-completion-dataset) | [๐Ÿ”—](#from-stepwise-supervision-to-prompt-only-dataset) | N/A | N/A | [๐Ÿ”—](#from-stepwise-supervision-to-unpaired-preference-dataset) | N/A |\n\n### From prompt-completion to language modeling dataset\n\nTo convert a prompt-completion dataset into a language modeling dataset, concatenate the prompt and the completion.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"The sky is\", \"The sun is\"],\n \"completion\": [\" blue.\", \" in the sky.\"],\n})\n\ndef concat_prompt_completion(example):\n return {\"text\": example[\"prompt\"] + example[\"completion\"]}\n\ndataset = dataset.map(concat_prompt_completion, remove_columns=[\"prompt\", \"completion\"])\n```\n\n```python\n>>> dataset[0]\n{'text': 'The sky is blue.'}\n```\n\n### From prompt-completion to prompt-only dataset\n\nTo convert a prompt-completion dataset into a prompt-only dataset, remove the completion.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"The sky is\", \"The sun is\"],\n \"completion\": [\" blue.\", \" in the sky.\"],\n})\n\ndataset = dataset.remove_columns(\"completion\")\n```\n\n```python\n>>> dataset[0]\n{'prompt': 'The sky is'}\n```\n\n### From preference with implicit prompt to language modeling dataset\n\nTo convert a preference with implicit prompt dataset into a language modeling dataset, remove the rejected, and rename the column `\"chosen\"` to `\"text\"`.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"chosen\": [\"The sky is blue.\", \"The sun is in the sky.\"],\n \"rejected\": [\"The sky is green.\", \"The sun is in the sea.\"],\n})\n\ndataset = dataset.rename_column(\"chosen\", \"text\").remove_columns(\"rejected\")\n```\n\n```python\n>>> dataset[0]\n{'text': 'The sky is blue.'}\n```\n\n### From preference with implicit prompt to prompt-completion dataset\n\nTo convert a preference dataset with implicit prompt into a prompt-completion dataset, extract the prompt with [`extract_prompt`], remove the rejected, and rename the column `\"chosen\"` to `\"completion\"`.\n\n```python\nfrom datasets import Dataset\nfrom trl import extract_prompt\n\ndataset = Dataset.from_dict({\n \"chosen\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}, {\"role\": \"assistant\", \"content\": \"In the sky.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is green.\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}, {\"role\": \"assistant\", \"content\": \"In the sea.\"}],\n ],\n})\ndataset = dataset.map(extract_prompt).remove_columns(\"rejected\").rename_column(\"chosen\", \"completion\")\n```\n\n```python\n>>> dataset[0]\n{'prompt': [{'role': 'user', 'content': 'What color is the sky?'}], 'completion': [{'role': 'assistant', 'content': 'It is blue.'}]}\n```\n\n### From preference with implicit prompt to prompt-only dataset\n\nTo convert a preference dataset with implicit prompt into a prompt-only dataset, extract the prompt with [`extract_prompt`], and remove the rejected and the chosen.\n\n```python\nfrom datasets import Dataset\nfrom trl import extract_prompt\n\ndataset = Dataset.from_dict({\n \"chosen\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}, {\"role\": \"assistant\", \"content\": \"In the sky.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is green.\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}, {\"role\": \"assistant\", \"content\": \"In the sea.\"}],\n ],\n})\ndataset = dataset.map(extract_prompt).remove_columns([\"chosen\", \"rejected\"])\n```\n\n```python\n>>> dataset[0]\n{'prompt': [{'role': 'user', 'content': 'What color is the sky?'}]}\n```\n\n### From implicit to explicit prompt preference dataset\n\nTo convert a preference dataset with implicit prompt into a preference dataset with explicit prompt, extract the prompt with [`extract_prompt`].\n\n```python\nfrom datasets import Dataset\nfrom trl import extract_prompt\n\ndataset = Dataset.from_dict({\n \"chosen\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}, {\"role\": \"assistant\", \"content\": \"In the sky.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is green.\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}, {\"role\": \"assistant\", \"content\": \"In the sea.\"}],\n ],\n})\n\ndataset = dataset.map(extract_prompt)\n```\n\n```python\n>>> dataset[0]\n{'prompt': [{'role': 'user', 'content': 'What color is the sky?'}],\n 'chosen': [{'role': 'assistant', 'content': 'It is blue.'}],\n 'rejected': [{'role': 'assistant', 'content': 'It is green.'}]}\n```\n\n### From preference with implicit prompt to unpaired preference dataset\n\nTo convert a preference dataset with implicit prompt into an unpaired preference dataset, extract the prompt with [`extract_prompt`], and unpair the dataset with [`unpair_preference_dataset`].\n\n```python\nfrom datasets import Dataset\nfrom trl import extract_prompt, unpair_preference_dataset\n\ndataset = Dataset.from_dict({\n \"chosen\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}, {\"role\": \"assistant\", \"content\": \"In the sky.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is green.\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}, {\"role\": \"assistant\", \"content\": \"In the sea.\"}],\n ],\n})\n\ndataset = dataset.map(extract_prompt)\ndataset = unpair_preference_dataset(dataset)\n```\n\n```python\n>>> dataset[0]\n{'prompt': [{'role': 'user', 'content': 'What color is the sky?'}],\n 'completion': [{'role': 'assistant', 'content': 'It is blue.'}],\n 'label': True}\n```\n\n> [!WARNING]\n> Keep in mind that the `\"chosen\"` and `\"rejected\"` completions in a preference dataset can be both good or bad.\n> Before applying [`unpair_preference_dataset`], please ensure that all `\"chosen\"` completions can be labeled as good and all `\"rejected\"` completions as bad.\n> This can be ensured by checking absolute rating of each completion, e.g. from a reward model.\n\n### From preference to language modeling dataset\n\nTo convert a preference dataset into a language modeling dataset, remove the rejected, concatenate the prompt and the chosen into the `\"text\"` column.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"The sky is\", \"The sun is\"],\n \"chosen\": [\" blue.\", \" in the sky.\"],\n \"rejected\": [\" green.\", \" in the sea.\"],\n})\n\ndef concat_prompt_chosen(example):\n return {\"text\": example[\"prompt\"] + example[\"chosen\"]}\n\ndataset = dataset.map(concat_prompt_chosen, remove_columns=[\"prompt\", \"chosen\", \"rejected\"])\n```\n\n```python\n>>> dataset[0]\n{'text': 'The sky is blue.'}\n```\n\n### From preference to prompt-completion dataset\n\nTo convert a preference dataset into a prompt-completion dataset, remove the rejected, and rename the column `\"chosen\"` to `\"completion\"`.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"The sky is\", \"The sun is\"],\n \"chosen\": [\" blue.\", \" in the sky.\"],\n \"rejected\": [\" green.\", \" in the sea.\"],\n})\n\ndataset = dataset.remove_columns(\"rejected\").rename_column(\"chosen\", \"completion\")\n```\n\n```python\n>>> dataset[0]\n{'prompt': 'The sky is', 'completion': ' blue.'}\n```\n\n### From preference to prompt-only dataset\n\nTo convert a preference dataset into a prompt-only dataset, remove the rejected and the chosen.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"The sky is\", \"The sun is\"],\n \"chosen\": [\" blue.\", \" in the sky.\"],\n \"rejected\": [\" green.\", \" in the sea.\"],\n})\n\ndataset = dataset.remove_columns([\"chosen\", \"rejected\"])\n```\n\n```python\n>>> dataset[0]\n{'prompt': 'The sky is'}\n```\n\n### From explicit to implicit prompt preference dataset\n\nTo convert a preference dataset with explicit prompt into a preference dataset with implicit prompt, concatenate the prompt to both chosen and rejected, and remove the prompt.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}],\n ],\n \"chosen\": [\n [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n [{\"role\": \"assistant\", \"content\": \"In the sky.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"assistant\", \"content\": \"It is green.\"}],\n [{\"role\": \"assistant\", \"content\": \"In the sea.\"}],\n ],\n})\n\ndef concat_prompt_to_completions(example):\n return {\"chosen\": example[\"prompt\"] + example[\"chosen\"], \"rejected\": example[\"prompt\"] + example[\"rejected\"]}\n\ndataset = dataset.map(concat_prompt_to_completions, remove_columns=\"prompt\")\n```\n\n```python\n>>> dataset[0]\n{'chosen': [{'role': 'user', 'content': 'What color is the sky?'}, {'role': 'assistant', 'content': 'It is blue.'}],\n 'rejected': [{'role': 'user', 'content': 'What color is the sky?'}, {'role': 'assistant', 'content': 'It is green.'}]}\n```\n\n### From preference to unpaired preference dataset\n\nTo convert dataset into an unpaired preference dataset, unpair the dataset with [`unpair_preference_dataset`].\n\n```python\nfrom datasets import Dataset\nfrom trl import unpair_preference_dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n [{\"role\": \"user\", \"content\": \"Where is the sun?\"}],\n ],\n \"chosen\": [\n [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n [{\"role\": \"assistant\", \"content\": \"In the sky.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"assistant\", \"content\": \"It is green.\"}],\n [{\"role\": \"assistant\", \"content\": \"In the sea.\"}],\n ],\n})\n\ndataset = unpair_preference_dataset(dataset)\n```\n\n```python\n>>> dataset[0]\n{'prompt': [{'role': 'user', 'content': 'What color is the sky?'}],\n 'completion': [{'role': 'assistant', 'content': 'It is blue.'}],\n 'label': True}\n```\n\n> [!WARNING]\n> Keep in mind that the `\"chosen\"` and `\"rejected\"` completions in a preference dataset can be both good or bad.\n> Before applying [`unpair_preference_dataset`], please ensure that all `\"chosen\"` completions can be labeled as good and all `\"rejected\"` completions as bad.\n> This can be ensured by checking absolute rating of each completion, e.g. from a reward model.\n\n### From unpaired preference to language modeling dataset\n\nTo convert an unpaired preference dataset into a language modeling dataset, concatenate prompts with good completions into the `\"text\"` column, and remove the prompt, completion and label columns.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"The sky is\", \"The sun is\", \"The sky is\", \"The sun is\"],\n \"completion\": [\" blue.\", \" in the sky.\", \" green.\", \" in the sea.\"],\n \"label\": [True, True, False, False],\n})\n\ndef concatenate_prompt_completion(example):\n return {\"text\": example[\"prompt\"] + example[\"completion\"]}\n\ndataset = dataset.filter(lambda x: x[\"label\"]).map(concatenate_prompt_completion).remove_columns([\"prompt\", \"completion\", \"label\"])\n```\n\n```python\n>>> dataset[0]\n{'text': 'The sky is blue.'}\n```\n\n### From unpaired preference to prompt-completion dataset\n\nTo convert an unpaired preference dataset into a prompt-completion dataset, filter for good labels, then remove the label columns.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"The sky is\", \"The sun is\", \"The sky is\", \"The sun is\"],\n \"completion\": [\" blue.\", \" in the sky.\", \" green.\", \" in the sea.\"],\n \"label\": [True, True, False, False],\n})\n\ndataset = dataset.filter(lambda x: x[\"label\"]).remove_columns([\"label\"])\n```\n\n```python\n>>> dataset[0]\n{'prompt': 'The sky is', 'completion': ' blue.'}\n```\n\n### From unpaired preference to prompt-only dataset\n\nTo convert an unpaired preference dataset into a prompt-only dataset, remove the completion and the label columns.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"The sky is\", \"The sun is\", \"The sky is\", \"The sun is\"],\n \"completion\": [\" blue.\", \" in the sky.\", \" green.\", \" in the sea.\"],\n \"label\": [True, True, False, False],\n})\n\ndataset = dataset.remove_columns([\"completion\", \"label\"])\n```\n\n```python\n>>> dataset[0]\n{'prompt': 'The sky is'}\n```\n\n### From stepwise supervision to language modeling dataset\n\nTo convert a stepwise supervision dataset into a language modeling dataset, concatenate prompts with good completions into the `\"text\"` column.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"Blue light\", \"Water\"],\n \"completions\": [[\" scatters more in the atmosphere,\", \" so the sky is green.\"],\n [\" forms a less dense structure in ice,\", \" which causes it to expand when it freezes.\"]],\n \"labels\": [[True, False], [True, True]],\n})\n\ndef concatenate_prompt_completions(example):\n completion = \"\".join(example[\"completions\"])\n return {\"text\": example[\"prompt\"] + completion}\n\ndataset = dataset.filter(lambda x: all(x[\"labels\"])).map(concatenate_prompt_completions, remove_columns=[\"prompt\", \"completions\", \"labels\"])\n```\n\n```python\n>>> dataset[0]\n{'text': 'Blue light scatters more in the atmosphere, so the sky is green.'}\n```\n\n### From stepwise supervision to prompt-completion dataset\n\nTo convert a stepwise supervision dataset into a prompt-completion dataset, join the good completions and remove the labels.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"Blue light\", \"Water\"],\n \"completions\": [[\" scatters more in the atmosphere,\", \" so the sky is green.\"],\n [\" forms a less dense structure in ice,\", \" which causes it to expand when it freezes.\"]],\n \"labels\": [[True, False], [True, True]],\n})\n\ndef join_completions(example):\n completion = \"\".join(example[\"completions\"])\n return {\"completion\": completion}\n\ndataset = dataset.filter(lambda x: all(x[\"labels\"])).map(join_completions, remove_columns=[\"completions\", \"labels\"])\n```\n\n```python\n>>> dataset[0]\n{'prompt': 'Blue light', 'completion': ' scatters more in the atmosphere, so the sky is green.'}\n```\n\n### From stepwise supervision to prompt-only dataset\n\nTo convert a stepwise supervision dataset into a prompt-only dataset, remove the completions and the labels.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"Blue light\", \"Water\"],\n \"completions\": [[\" scatters more in the atmosphere,\", \" so the sky is green.\"],\n [\" forms a less dense structure in ice,\", \" which causes it to expand when it freezes.\"]],\n \"labels\": [[True, False], [True, True]],\n})\n\ndataset = dataset.remove_columns([\"completions\", \"labels\"])\n```\n\n```python\n>>> dataset[0]\n{'prompt': 'Blue light'}\n```\n\n### From stepwise supervision to unpaired preference dataset\n\nTo convert a stepwise supervision dataset into an unpaired preference dataset, join the completions and merge the labels.\n\nThe method for merging the labels depends on the specific task. In this example, we use the logical AND operation. This means that if the step labels indicate the correctness of individual steps, the resulting label will reflect the correctness of the entire sequence.\n\n```python\nfrom datasets import Dataset\n\ndataset = Dataset.from_dict({\n \"prompt\": [\"Blue light\", \"Water\"],\n \"completions\": [[\" scatters more in the atmosphere,\", \" so the sky is green.\"],\n [\" forms a less dense structure in ice,\", \" which causes it to expand when it freezes.\"]],\n \"labels\": [[True, False], [True, True]],\n})\n\ndef merge_completions_and_labels(example):\n return {\"prompt\": example[\"prompt\"], \"completion\": \"\".join(example[\"completions\"]), \"label\": all(example[\"labels\"])}\n\ndataset = dataset.map(merge_completions_and_labels, remove_columns=[\"completions\", \"labels\"])\n```\n\n```python\n>>> dataset[0]\n{'prompt': 'Blue light', 'completion': ' scatters more in the atmosphere, so the sky is green.', 'label': False}\n```\n\n## Vision datasets\n\nSome trainers also support fine-tuning vision-language models (VLMs) using image-text pairs. In this scenario, it's recommended to use a conversational format, as each model handles image placeholders in text differently.\n\nA conversational vision dataset differs from a standard conversational dataset in two key ways:\n\n1. The dataset must contain the key `images` with the image data (as lists of PIL images) or `image` with a single PIL image.\n2. The `\"content\"` field in messages must be a list of dictionaries, where each dictionary specifies the type of data: `\"image\"` or `\"text\"`.\n\nExample:\n\n```python\n# Textual dataset:\n\"content\": \"What color is the sky?\"\n\n# Vision dataset:\n\"content\": [\n {\"type\": \"image\"}, \n {\"type\": \"text\", \"text\": \"What color is the sky in the image?\"}\n]\n```\n\nAn example of a conversational vision dataset is the [openbmb/RLAIF-V-Dataset](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset). Below is an embedded view of the dataset's training data, allowing you to explore it directly:\n\n\n\n> [!NOTE]\n> Mixing text-only and vision-language data in the dataset is possible, but it requires `transformers` version 4.57.0 or later. Example:\n>\n> ```python\n> dataset = Dataset.from_dict({\n> \"prompt\": [\n> [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What color is the sky in the image?\"}]}],\n> [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is the capital of France?\"}]}],\n> ],\n> \"completion\": [\n> [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It is blue.\"}]}],\n> [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Paris.\"}]}],\n> ],\n> \"images\": [\n> [PIL.Image.open(\"path/to/sky_image1.png\")],\n> [],\n> ],\n> })\n> ```\n" }, { "path": "docs/source/deepspeed_integration.md", "content": "# DeepSpeed Integration\n\n> [!WARNING]\n> Section under construction. Feel free to contribute!\n\nTRL supports training with DeepSpeed, a library that implements advanced training optimization techniques. These include optimizer state partitioning, offloading, gradient partitioning, and more.\n\nDeepSpeed integrates the [Zero Redundancy Optimizer (ZeRO)](https://huggingface.co/papers/1910.02054), which allows to scale the model size proportional to the number of devices with sustained high efficiency.\n\n![ZeRO Stages](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/zero_stages.png)\n\n## Installation\n\nTo use DeepSpeed with TRL, install it using the following command:\n\n```bash\npip install deepspeed\n```\n\n## Running Training Scripts with DeepSpeed\n\nNo modifications to your training script are required. Simply run it with the DeepSpeed configuration file:\n\n```bash\naccelerate launch --config_file train.py\n```\n\nWe provide ready-to-use DeepSpeed configuration files in the [`examples/accelerate_configs`](https://github.com/huggingface/trl/tree/main/examples/accelerate_configs) directory. For example, to run training with ZeRO Stage 2, use the following command:\n\n```bash\naccelerate launch --config_file examples/accelerate_configs/deepspeed_zero2.yaml train.py\n```\n\n## Additional Resources\n\nConsult the ๐Ÿค— Accelerate [documentation](https://huggingface.co/docs/accelerate/usage_guides/deepspeed) for more information about the DeepSpeed plugin.\n" }, { "path": "docs/source/distributing_training.md", "content": "# Distributing Training\n\n> [!WARNING]\n> Section under construction. Feel free to contribute!\n\n## Multi-GPU Training with TRL\n\nThe trainers in TRL use [๐Ÿค— Accelerate](https://github.com/huggingface/accelerate) to enable distributed training across multiple GPUs or nodes. To do so, first create an [๐Ÿค— Accelerate](https://github.com/huggingface/accelerate) config file by running\n\n```bash\naccelerate config\n```\n\nand answering the questions according to your multi-GPU / multi-node setup. You can then launch distributed training by running:\n\n```bash\naccelerate launch train.py\n```\n\nWe also provide config files in the [examples folder](https://github.com/huggingface/trl/tree/main/examples/accelerate_configs) that can be used as templates. To use these templates, simply pass the path to the config file when launching a job, e.g.:\n\n```shell\naccelerate launch --config_file examples/accelerate_configs/multi_gpu.yaml train.py \n```\n\nThis automatically distributes the workload across all available GPUs.\n\nUnder the hood, [๐Ÿค— Accelerate](https://github.com/huggingface/accelerate) creates one model per GPU. Each process:\n\n- Processes its own batch of data\n- Computes the loss and gradients for that batch\n- Shares gradient updates across all GPUs\n\n![multi gpu](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/multi_gpu.png)\n\nThe effective batch size is calculated as:\n\n$$\n\\text{Batch Size} = \\text{per\\_device\\_train\\_batch\\_size} \\times \\text{num\\_devices} \\times \\text{gradient\\_accumulation\\_steps}\n$$\n\nTo maintain a consistent batch size when scaling to multiple GPUs, make sure to update `per_device_train_batch_size` and `gradient_accumulation_steps` accordingly.\n\nExample, these configurations are equivalent, and should yield the same results:\n\n| Number of GPUs | Per device batch size | Gradient accumulation steps | Comments |\n| --- | --- | --- | --- |\n| 1 | 32 | 1 | Possibly high memory usage, but faster training |\n| 1 | 4 | 8 | Lower memory usage, slower training |\n| 8 | 4 | 1 | Multi-GPU to get the best of both worlds |\n\n> [!TIP]\n> Having one model per GPU can lead to high memory usage, which may not be feasible for large models or low-memory GPUs. In such cases, you can leverage [DeepSpeed](https://github.com/deepspeedai/DeepSpeed), which provides optimizations like model sharding, Zero Redundancy Optimizer, mixed precision training, and offloading to CPU or NVMe. Check out our [DeepSpeed Integration](deepspeed_integration) guide for more details.\n\n## Sequence Parallelism for Long Context Training\n\nSequence Parallelism (also called Context Parallelism) is a parallelization technique that enables training with longer sequences by splitting the sequence dimension across multiple GPUs. Each GPU processes a portion of the sequence, allowing you to train with sequences longer than what would fit on a single GPU's memory.\n\n> [!NOTE]\n> **Terminology clarification:** This section describes parallelism techniques for splitting sequences to enable longer context training:\n> - **Context Parallelism (CP)**: Splits sequences across GPUs (implemented as Ring Attention with FSDP2)\n> - **Sequence Parallelism (SP)**: Another form of sequence splitting (implemented as ALST/Ulysses with DeepSpeed)\n>\n> Both CP and SP are different from traditional Sequence Parallelism used with Tensor Parallelism (TP+SP) to reduce activation memory. With the techniques here, parallelism dimensions multiply: `TP=2` and `CP=2` would require 4 GPUs (2ร—2), whereas traditional `TP+SP=2` only needs 2 GPUs as they share the same ranks.\n>\n> In Accelerate's `ParallelismConfig`:\n> - Use `cp_size` with `cp_backend=\"torch\"` for Ring Attention (FSDP2)\n> - Use `sp_size` with `sp_backend=\"deepspeed\"` for ALST/Ulysses (DeepSpeed)\n\nSequence parallelism is particularly useful when:\n\n- You want to train with very long sequences (>32k tokens)\n- Single GPU memory is insufficient for your desired sequence length\n- You need to maintain sequence coherence across the full context\n\n### Available Implementations\n\nTRL supports two sequence parallelism implementations, each with different characteristics:\n\n1. **Ring Attention (FSDP2)** - Uses ring-based communication for memory-efficient processing of extremely long sequences\n2. **ALST/Ulysses (DeepSpeed)** - Uses attention head parallelism for faster training with high-bandwidth interconnects\n\n> [!IMPORTANT]\n> **Sequence Length Terminology:** When using Context Parallelism, the sequence is split across GPUs, introducing two concepts:\n> - **Global sequence length**: The full sequence length before splitting across GPUs\n> - **Micro sequence length**: The sequence length per GPU after splitting\n>\n> In TRL, `max_seq_length` (or `max_length`) refers to the **global sequence length**. The framework automatically handles splitting into micro sequences:\n> - **Ring Attention (FSDP2)**: Uses `cp_size` to split sequences. With `max_seq_length=8192` and `cp_size=4`, each GPU processes 2048 tokens.\n> - **ALST/Ulysses (DeepSpeed)**: Uses `sp_size` (with `sp_backend=\"deepspeed\"`) to split sequences. With `max_seq_length=8192` and `sp_size=2`, each GPU processes 4096 tokens.\n>\n> The Trainer automatically accounts for context parallelism when calculating batch sizes and training metrics.\n\n### Choosing Between Ring Attention and Ulysses\n\nThe comparison table below highlights the key differences between the two approaches:\n\n| Feature | Ring Attention (FSDP2) | ALST/Ulysses (DeepSpeed) |\n|---------|----------|-------------------------|\n| **Method** | Ring Self-Attention | Attention Head Parallelism |\n| **Backend** | PyTorch FSDP2 | DeepSpeed ZeRO |\n| **Attention** | SDPA only | Flash Attention 2 or SDPA |\n| **Minimum Accelerate** | 1.11.0+ | 1.12.0+ |\n| **Minimum DeepSpeed** | N/A | 0.18.1+ |\n| **Sequence Divisibility** | `cp_size * 2` | `sp_size` |\n| **Zero Stage** | N/A | ZeRO Stage 1/2/3 |\n\n**Ring Attention is better when:**\n- You need to handle extremely long sequences (1M+ tokens)\n- The model has limited attention heads (Ring Attention is not constrained by head count)\n- You want flexibility in scaling to any sequence length\n- Network topology is limited (Ring Attention works with simple P2P ring communication)\n\n**Ulysses is better when:**\n- You have high-bandwidth, low-latency interconnects (NVLink, InfiniBand)\n- The model has many attention heads that can be split across GPUs\n- You want lower communication volume\n- You want faster training speed for moderate sequence lengths (up to ~500k tokens)\n\n**Key Trade-offs:**\n- **Communication Volume:** Ulysses has lower communication volume, making it more efficient with good interconnects. Ring Attention has higher communication volume but is more flexible with different network topologies.\n- **Attention Head Constraints:** Ulysses is limited by the number of attention heads (requires `num_heads >= sp_size`). Ring Attention scales with sequence length regardless of model architecture.\n- **Network Sensitivity:** Ulysses all-to-all communication is sensitive to network latency. Ring Attention uses P2P ring communication which is more tolerant of varying network conditions.\n\nFor a detailed comparison, see the [Ulysses and Ring Attention blog post](https://huggingface.co/blog/exploding-gradients/ulysses-ring-attention).\n\n### Ring Attention Implementation (FSDP2)\n\nRing Attention uses a ring-like communication pattern where each GPU processes a portion of the sequence and passes information to the next GPU in the ring.\n\n#### Requirements and Limitations\n\n1. **Accelerate 1.11.0 or higher** is required for Ring Attention / Context Parallelism support\n2. **FSDP2 (PyTorch FSDP v2)** is required as the distributed training backend\n3. **SDPA attention** - Flash Attention is currently not supported\n4. **Sequence length divisibility** - sequences must be divisible by `cp_size * 2`. This is automatically handled using the `pad_to_multiple_of` parameter in the data collator.\n\n#### Configuration\n\n##### Accelerate Configuration\n\nUse one of the provided accelerate config files (e.g. [`context_parallel_2gpu.yaml`](https://github.com/huggingface/trl/blob/main/examples/accelerate_configs/context_parallel_2gpu.yaml) for 2 GPUs):\n\n```yaml\ncompute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: FSDP\ndowncast_bf16: 'no'\nenable_cpu_affinity: false\nfsdp_config:\n fsdp_activation_checkpointing: true # Enable activation checkpointing for memory efficiency\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_cpu_ram_efficient_loading: true\n fsdp_offload_params: false\n fsdp_reshard_after_forward: true\n fsdp_state_dict_type: FULL_STATE_DICT\n fsdp_version: 2\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 2 # Number of GPUs\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\nparallelism_config:\n parallelism_config_dp_replicate_size: 1\n parallelism_config_dp_shard_size: 1\n parallelism_config_tp_size: 1\n parallelism_config_cp_size: 2 # Context parallel size\n```\n\n##### Training Configuration\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(\n # required\n pad_to_multiple_of=4, # ensures divisibility by cp_size * 2\n # to get the most out of CP\n max_length=16384, # long sequence length\n packing=True, # use packing to reduce padding\n use_liger_kernel=True, # compatible with CP\n gradient_checkpointing=False, # The activation_checkpointing in FSDP config and the gradient_checkpointing in training arg can't be set to True simultaneously\n per_device_train_batch_size=1,\n ...\n)\n```\n\nThen, launch your training script with the appropriate accelerate config file:\n\n```bash\naccelerate launch --config_file context_parallel_2gpu.yaml train.py\n```\n\n#### Best Practices\n\n1. **Use the `pad_to_multiple_of` parameter** - This is now the recommended way to ensure sequence length divisibility:\n - For `cp_size=2`: use `pad_to_multiple_of=4` (since `cp_size * 2 = 4`)\n - For `cp_size=4`: use `pad_to_multiple_of=8` (since `cp_size * 2 = 8`)\n - The data collator automatically pads sequences to the required multiple, ensuring compatibility with CP\n\n2. **Use packing with padding** - The default BFD (Best Fit Decreasing) strategy works perfectly:\n - Preserves sequence boundaries and maintains training quality\n - Works seamlessly with both `padding_free=True` and standard padding modes\n\n3. **Combine with other memory optimizations** like Liger kernels, bfloat16, and gradient checkpointing\n\n4. **Start with smaller context parallel sizes** (2-4 GPUs) before scaling up\n\n5. **Monitor memory usage** across all GPUs to ensure balanced workload\n\n#### Benchmarking Ring Attention\n\nWe benchmarked Ring Attention to highlight its potential improvements in training efficiency. \nOur experiments were conducted using **1, 2, 4, and 8 H100 GPUs**, though the results can be extended to larger clusters with more nodes and GPUs.\n\nFor the setup, we fine-tuned an **8B model** ([Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B)) using the provided accelerate configuration \n([`context_parallel_2gpu.yaml`](https://github.com/huggingface/trl/blob/main/examples/accelerate_configs/context_parallel_2gpu.yaml)). \nWe adjusted `num_processes` and `parallelism_config_cp_size` based on the number of GPUs for each run. \nTraining was performed with the [sft.py](https://github.com/huggingface/trl/blob/main/trl/scripts/sft.py) example script, combined with the parameters described above.\n\nThe results below summarize the **maximum trainable sequence length** and **iterations per second** for different numbers of GPUs. A value marked as `OOM` indicates that the configuration ran out of memory and could not be trained. \n\nThese results show that **Context Parallelism (CP) scales effectively with more GPUs**, enabling training on much longer sequences. With **8 GPUs**, context lengths of over **300k tokens** become feasible, unlocking training with extremely long contexts while maintaining reasonable throughput. \n\n
\n \"CP\n \"CP\n
\n\n> [!TIP]\n> Accelerate also supports **N-Dimensional Parallelism (ND-parallelism)**, which enables you to combine different parallelization strategies to efficiently distribute model training across multiple GPUs. \n>\n> You can learn more and explore configuration examples in the [Accelerate ND-parallelism guide](https://github.com/huggingface/accelerate/blob/main/examples/torch_native_parallelism/README.md#nd-parallelism).\n\n### ALST/Ulysses Implementation (DeepSpeed)\n\nALST (Arctic Long Sequence Training) / Ulysses uses attention head parallelism to split long sequences across GPUs, working with DeepSpeed's ZeRO optimizer.\n\n> [!NOTE]\n> **Technical Note on Parallelism Configuration:**\n> - **DeepSpeed ALST/Ulysses** uses `sp_size` with `sp_backend=\"deepspeed\"` in both YAML and Python API\n> - **Ring Attention (FSDP2)** uses `cp_size` with `cp_backend=\"torch\"`\n>\n> The Trainer automatically accounts for both CP and SP when calculating effective batch sizes and training metrics.\n\n#### Requirements and Limitations\n\n1. **DeepSpeed 0.18.1 or higher** is required\n2. **Accelerate 1.12.0 or higher** is required for ALST/Ulysses sequence parallelism support\n3. **Attention implementation** - Flash Attention 2 recommended (clean output), SDPA works as fallback\n4. **Sequence length divisibility** - sequences must be divisible by `sp_size`. Use `pad_to_multiple_of` in your training config.\n5. **Parallelism configuration** - You must ensure `dp_replicate_size ร— dp_shard_size ร— sp_size = num_processes`\n\n#### Configuration\n\n##### Accelerate Configuration\n\nUse the provided accelerate config file ([`alst_ulysses_4gpu.yaml`](https://github.com/huggingface/trl/blob/main/examples/accelerate_configs/alst_ulysses_4gpu.yaml)):\n\n```yaml\ncompute_environment: LOCAL_MACHINE\ndebug: false\ndeepspeed_config:\n zero_stage: 3\n seq_parallel_communication_data_type: bf16\ndistributed_type: DEEPSPEED\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 4 # Number of GPUs\nparallelism_config:\n parallelism_config_dp_replicate_size: 1\n parallelism_config_dp_shard_size: 2 # Enables 2D parallelism with SP\n parallelism_config_tp_size: 1\n parallelism_config_sp_size: 2 # Sequence parallel size\n parallelism_config_sp_backend: deepspeed\n parallelism_config_sp_seq_length_is_variable: true\n parallelism_config_sp_attn_implementation: flash_attention_2\n```\n\n##### Training Configuration\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(\n # required\n pad_to_multiple_of=2, # Must equal sp_size\n # to get the most out of SP\n max_seq_length=4096,\n packing=True,\n attn_implementation=\"flash_attention_2\",\n per_device_train_batch_size=1,\n ...\n)\n```\n\nThen, launch your training script with the appropriate accelerate config file:\n\n```bash\naccelerate launch --config_file examples/accelerate_configs/alst_ulysses_4gpu.yaml train.py\n```\n\n#### 2D Parallelism\n\nThe 4 GPU configuration above automatically enables 2D parallelism by combining Data Parallelism (DP) with Sequence Parallelism (SP). With `sp_size=2` and `dp_shard_size=2`, the 4 GPUs are organized as:\n- 2 sequence parallel groups (processing the same data split across sequences)\n- 2 data parallel groups (processing different data)\n\nTo adjust the parallelism for different GPU counts, modify the YAML config:\n\n| GPUs | sp_size | dp_shard_size | Use Case | YAML Changes |\n|------|---------|---------------|----------|--------------|\n| 4 | 2 | 2 | Balanced - longer sequences + more data | `num_processes: 4`, `sp_size: 2`, `dp_shard_size: 2` |\n| 4 | 4 | 1 | Pure SP for maximum sequence length | `num_processes: 4`, `sp_size: 4`, `dp_shard_size: 1` |\n| 8 | 2 | 4 | Large-scale training | `num_processes: 8`, `sp_size: 2`, `dp_shard_size: 4` |\n\n#### Best Practices\n\n1. **Use `pad_to_multiple_of`** to ensure sequences are divisible by `sp_size`\n2. **Use Flash Attention 2** for clean output (SDPA works but shows packing warnings)\n3. **Start with `sp_size=2`** before scaling to larger values\n4. **Use DeepSpeed ZeRO Stage 3** for large models\n5. **Combine with memory optimizations** like Liger kernels and gradient checkpointing\n6. **Validate parallelism config**: Ensure `dp_replicate_size ร— dp_shard_size ร— sp_size = num_processes`\n\n#### Complete Example\n\nHere's how to run ALST/Ulysses training using the built-in [`sft.py`](https://github.com/huggingface/trl/blob/main/trl/scripts/sft.py) script with 4 GPUs:\n\n```bash\naccelerate launch --config_file examples/accelerate_configs/alst_ulysses_4gpu.yaml \\\n trl/scripts/sft.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --learning_rate 2e-4 \\\n --max_steps 100 \\\n --max_seq_length 4096 \\\n --packing \\\n --packing_strategy wrapped \\\n --torch_dtype bfloat16 \\\n --attn_implementation flash_attention_2 \\\n --output_dir output-alst-4gpu \\\n --logging_steps 10 \\\n --report_to trackio\n```\n\nThis command automatically:\n- Configures 2D parallelism (SP=2, DP=2) across 4 GPUs\n- Uses Flash Attention 2 for clean training\n- Enables packing with automatic padding to ensure sequence divisibility\n- Leverages DeepSpeed ZeRO Stage 3 for memory efficiency\n\n### Further Reading\n\n#### General Resources\n- [Hugging Face Blog: Understanding Ulysses and Ring Attention](https://huggingface.co/blog/exploding-gradients/ulysses-ring-attention) - Detailed comparison of Ring Attention vs Ulysses approaches\n- [Accelerate: Context Parallelism Guide](https://huggingface.co/docs/accelerate/concept_guides/context_parallelism)\n- [Hugging Face Blog: Enabling Long-Context Training with Sequence Parallelism in Axolotl](https://huggingface.co/blog/axolotl-ai-co/long-context-with-sequence-parallelism-in-axolotl)\n\n#### Ring Attention (FSDP2)\n- [Ultrascale Playbook - Context Parallelism](https://huggingface.co/spaces/nanotron/ultrascale-playbook?section=context_parallelism)\n- [Accelerate Example: 128k Sequence Length](https://github.com/huggingface/accelerate/blob/main/examples/torch_native_parallelism/README.md#context-parallelism-128k-sequence-length)\n- [Accelerate ND-parallelism Guide](https://github.com/huggingface/accelerate/blob/main/examples/torch_native_parallelism/README.md#nd-parallelism)\n\n#### ALST/Ulysses (DeepSpeed)\n- [DeepSpeed Sequence Parallelism Documentation](https://www.deepspeed.ai/tutorials/ds-sequence/)\n- [Snowflake Engineering Blog: Arctic Long Sequence Training (ALST)](https://www.snowflake.com/en/engineering-blog/arctic-long-sequence-training-multi-million-token-ai/)\n\n## Multi-Node Training\n\nWhen a single machine doesn't have enough GPUs, TRL can scale training across multiple machines (nodes) using [๐Ÿค— Accelerate](https://huggingface.co/docs/accelerate/basic_tutorials/launch#multi-node-training).\n\n### Accelerate Configuration\nCreate an `accelerate` config file (e.g., `multi_node.yaml`) for multi-node training. Key fields:\n\n```yaml\ncompute_environment: LOCAL_MACHINE\ndistributed_type: MULTI_GPU\nnum_machines: 2\nmachine_rank: 0 # 0 for main node, 1 for second node\nmain_process_ip: 10.0.0.1 # IP of rank 0 node\nmain_process_port: 29500\nnum_processes: 16 # total processes across nodes\nmixed_precision: bf16\nuse_cpu: false\nsame_network: true\n```\n\nAdjust `num_processes` to match the total number of GPUs across all nodes.\n\n> [!NOTE]\n> Replace `10.0.0.1` with the actual IP address of the rank 0 (main) node.\n\n### Launching\n\n#### Option 1: Manual Launch (Non-HPC)\n\nRun the following on each node manually:\n```bash\n# Node 0 (main node)\naccelerate launch --config_file multi_node.yaml --machine_rank 0 train.py\n\n# Node 1\naccelerate launch --config_file multi_node.yaml --machine_rank 1 train.py\n```\n#### Option 2: SLURM Launch (HPC Clusters)\n\nFor clusters using SLURM job scheduler, create a job script (e.g., `slurm_job.sh`):\n```bash\n#!/bin/bash\n#SBATCH --nodes=2\n#SBATCH --gpus-per-node=8\n#SBATCH --job-name=trl_multi\n\nsrun accelerate launch --config_file multi_node.yaml train.py\n```\n\nThen submit the job:\n```bash\nsbatch slurm_job.sh\n```\n\nSLURM automatically distributes the training across all requested nodes and GPUs, and `srun` configures the necessary environment variables for multi-node communication.\n\n**Key SLURM directives:**\n- `--nodes=2`: Request 2 compute nodes\n- `--gpus-per-node=8`: Allocate 8 GPUs per node (16 total)\n- `--job-name`: Label for tracking in the job queue\n\nYou can combine multi-node with DeepSpeed by setting `distributed_type: DEEPSPEED` and adding a `deepspeed_config` block. See the [DeepSpeed integration guide](https://huggingface.co/docs/trl/en/deepspeed_integration).\n\n### Further Reading\n\n- [Accelerate: Launching Scripts](https://huggingface.co/docs/accelerate/basic_tutorials/launch)\n- [Accelerate: Example Zoo](https://huggingface.co/docs/accelerate/usage_guides/training_zoo)\n- [SLURM Workload Manager Documentation](https://slurm.schedmd.com/) - For cluster job scheduling\n\n\n\n" }, { "path": "docs/source/dpo_trainer.md", "content": "# DPO Trainer\n\n[![All_models-DPO-blue](https://img.shields.io/badge/All_models-DPO-blue)](https://huggingface.co/models?other=dpo,trl) [![smol_course-Chapter_2-yellow](https://img.shields.io/badge/smol_course-Chapter_2-yellow)](https://github.com/huggingface/smol-course/tree/main/2_preference_alignment)\n\n## Overview\n\nTRL supports the Direct Preference Optimization (DPO) Trainer for training language models, as described in the paper [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290) by [Rafael Rafailov](https://huggingface.co/rmrafailov), Archit Sharma, Eric Mitchell, [Stefano Ermon](https://huggingface.co/ermonste), [Christopher D. Manning](https://huggingface.co/manning), [Chelsea Finn](https://huggingface.co/cbfinn).\n\nThe abstract from the paper is the following:\n\n> While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.\n\nThis post-training method was contributed by [Kashif Rasul](https://huggingface.co/kashif) and later refactored by [Quentin Gallouรฉdec](https://huggingface.co/qgallouedec).\n\n## Quick start\n\nThis example demonstrates how to train a language model using the [`DPOTrainer`] from TRL. We train a [Qwen 3 0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) model on the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback).\n\n```python\nfrom trl import DPOTrainer\nfrom datasets import load_dataset\n\ntrainer = DPOTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n train_dataset=load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\"),\n)\ntrainer.train()\n```\n\n\n\n## Expected dataset type and format\n\nDPO requires a [preference](dataset_formats#preference) dataset. The [`DPOTrainer`] is compatible with both [standard](dataset_formats#standard) and [conversational](dataset_formats#conversational) dataset formats. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\n```python\n# Standard format\n## Explicit prompt (recommended)\npreference_example = {\"prompt\": \"The sky is\", \"chosen\": \" blue.\", \"rejected\": \" green.\"}\n# Implicit prompt\npreference_example = {\"chosen\": \"The sky is blue.\", \"rejected\": \"The sky is green.\"}\n\n# Conversational format\n## Explicit prompt (recommended)\npreference_example = {\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": \"It is green.\"}]}\n## Implicit prompt\npreference_example = {\"chosen\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is green.\"}]}\n```\n\nIf your dataset is not in one of these formats, you can preprocess it to convert it into the expected format. Here is an example with the [Vezora/Code-Preference-Pairs](https://huggingface.co/datasets/Vezora/Code-Preference-Pairs) dataset:\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"Vezora/Code-Preference-Pairs\")\n\n\ndef preprocess_function(example):\n return {\n \"prompt\": [{\"role\": \"user\", \"content\": example[\"input\"]}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": example[\"accepted\"]}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": example[\"rejected\"]}],\n }\n\n\ndataset = dataset.map(preprocess_function, remove_columns=[\"instruction\", \"input\", \"accepted\", \"ID\"])\nprint(next(iter(dataset[\"train\"])))\n```\n\n```json\n{\n \"prompt\": [{\"role\": \"user\", \"content\": \"Create a nested loop to print every combination of numbers [...]\"}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": \"Here is an example of a nested loop in Python [...]\"}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": \"Here is an example of a nested loop in Python [...]\"}],\n}\n```\n\n## Looking deeper into the DPO method\n\nDirect Preference Optimization (DPO) is a training method designed to align a language model with preference data. Instead of supervised inputโ€“output pairs, the model is trained on pairs of completions to the same prompt, where one completion is preferred over the other. The objective directly optimizes the model to widen the margin between the log-likelihoods of preferred and dispreferred completions, relative to a reference model, without requiring an explicit reward model. In practice, this is typically achieved by suppressing the likelihood of dispreferred completions rather than by increasing the likelihood of preferred ones.\n\nThis section breaks down how DPO works in practice, covering the key steps: **preprocessing** and **loss computation**.\n\n### Preprocessing and tokenization\n\nDuring training, each example is expected to contain a prompt along with a preferred (`chosen`) and a dispreferred (`rejected`) completion. For more details on the expected formats, see [Dataset formats](dataset_formats).\nThe [`DPOTrainer`] tokenizes each input using the model's tokenizer.\n\n### Computing the loss\n\n![dpo_figure](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/dpo_figure.png)\n\nThe loss used in DPO is defined as follows:\n$$\n\\mathcal{L}_{\\mathrm{DPO}}(\\theta) = -\\mathbb{E}_{(x,y^{+},y^{-})}\\!\\left[\\log \\sigma\\!\\left(\\beta\\Big(\\log\\frac{\\pi_{\\theta}(y^{+}\\!\\mid x)}{\\pi_{\\mathrm{ref}}(y^{+}\\!\\mid x)}-\\log \\frac{\\pi_{\\theta}(y^{-}\\!\\mid x)}{\\pi_{\\mathrm{ref}}(y^{-}\\!\\mid x)}\\Big)\\right)\\right]\n$$\n \nwhere \\\\( x \\\\) is the prompt, \\\\( y^+ \\\\) is the preferred completion and \\\\( y^- \\\\) is the dispreferred completion. \\\\( \\pi_{\\theta} \\\\) is the policy model being trained, \\\\( \\pi_{\\mathrm{ref}} \\\\) is the reference model, \\\\( \\sigma \\\\) is the sigmoid function, and \\\\( \\beta > 0 \\\\) is a hyperparameter that controls the strength of the preference signal.\n\n#### Loss Types\n\nSeveral formulations of the objective have been proposed in the literature. Initially, the objective of DPO was defined as presented above.\n\n| `loss_type=` | Description |\n| --- | --- |\n| `\"sigmoid\"` (default) | Given the preference data, we can fit a binary classifier according to the Bradley-Terry model and in fact the [DPO](https://huggingface.co/papers/2305.18290) authors propose the sigmoid loss on the normalized likelihood via the `logsigmoid` to fit a logistic regression. |\n| `\"hinge\"` | The [RSO](https://huggingface.co/papers/2309.06657) authors propose to use a hinge loss on the normalized likelihood from the [SLiC](https://huggingface.co/papers/2305.10425) paper. In this case, the `beta` is the reciprocal of the margin. |\n| `\"ipo\"` | The [IPO](https://huggingface.co/papers/2310.12036) authors argue the logit transform can overfit and propose the identity transform to optimize preferences directly; TRL exposes this as `loss_type=\"ipo\"`. |\n| `\"exo_pair\"` | The [EXO](https://huggingface.co/papers/2402.00856) authors propose reverse-KL preference optimization. `label_smoothing` must be strictly greater than `0.0`; a recommended value is `1e-3` (see Eq. 16 for the simplified pairwise variant). The full method uses `K>2` SFT completions and approaches PPO as `K` grows. |\n| `\"nca_pair\"` | The [NCA](https://huggingface.co/papers/2402.05369) authors shows that NCA optimizes the absolute likelihood for each response rather than the relative likelihood. |\n| `\"robust\"` | The [Robust DPO](https://huggingface.co/papers/2403.00409) authors propose an unbiased DPO loss under noisy preferences. Use `label_smoothing` in [`DPOConfig`] to model label-flip probability; valid values are in the range `[0.0, 0.5)`. |\n| `\"bco_pair\"` | The [BCO](https://huggingface.co/papers/2404.04656) authors train a binary classifier whose logit serves as a reward so that the classifier maps {prompt, chosen completion} pairs to 1 and {prompt, rejected completion} pairs to 0. For unpaired data, we recommend the dedicated [`experimental.bco.BCOTrainer`]. |\n| `\"sppo_hard\"` | The [SPPO](https://huggingface.co/papers/2405.00675) authors claim that SPPO is capable of solving the Nash equilibrium iteratively by pushing the chosen rewards to be as large as 1/2 and the rejected rewards to be as small as -1/2 and can alleviate data sparsity issues. The implementation approximates this algorithm by employing hard label probabilities, assigning 1 to the winner and 0 to the loser. |\n| `\"aot\"` or `loss_type=\"aot_unpaired\"` | The [AOT](https://huggingface.co/papers/2406.05882) authors propose Distributional Preference Alignment via Optimal Transport. `loss_type=\"aot\"` is for paired data; `loss_type=\"aot_unpaired\"` is for unpaired data. Both enforce stochastic dominance via sorted quantiles; larger per-GPU batch sizes help. |\n| `\"apo_zero\"` or `loss_type=\"apo_down\"` | The [APO](https://huggingface.co/papers/2408.06266) method introduces an anchored objective. `apo_zero` boosts winners and downweights losers (useful when the model underperforms the winners). `apo_down` downweights both, with stronger pressure on losers (useful when the model already outperforms winners). |\n| `\"discopop\"` | The [DiscoPOP](https://huggingface.co/papers/2406.08414) paper uses LLMs to discover more efficient offline preference optimization losses. In the paper the proposed DiscoPOP loss (which is a log-ratio modulated loss) outperformed other optimization losses on different tasks (IMDb positive text generation, Reddit TLDR summarization, and Alpaca Eval 2.0). |\n| `\"sft\"` | SFT (Supervised Fine-Tuning) loss is the negative log likelihood loss, used to train the model to generate preferred responses. |\n\n## Logged metrics\n\nWhile training and evaluating we record the following reward metrics:\n\n* `global_step`: The total number of optimizer steps taken so far.\n* `epoch`: The current epoch number, based on dataset iteration.\n* `num_tokens`: The total number of tokens processed so far.\n* `loss`: The average cross-entropy loss computed over non-masked tokens in the current logging interval.\n* `entropy`: The average entropy of the model's predicted token distribution over non-masked tokens.\n* `mean_token_accuracy`: The proportion of non-masked tokens for which the modelโ€™s top-1 prediction matches the token from the chosen completion.\n* `learning_rate`: The current learning rate, which may change dynamically if a scheduler is used.\n* `grad_norm`: The L2 norm of the gradients, computed before gradient clipping.\n* `logits/chosen`: The average logit values assigned by the model to the tokens in the chosen completion.\n* `logits/rejected`: The average logit values assigned by the model to the tokens in the rejected completion.\n* `logps/chosen`: The average log-probability assigned by the model to the tokens in the chosen completion.\n* `logps/rejected`: The average log-probability assigned by the model to the tokens in the rejected completion.\n* `rewards/chosen`: The average implicit reward computed for the chosen completion, computed as \\\\( \\beta \\log \\frac{\\pi_{\\theta}(y^{+}\\!\\mid x)}{\\pi_{\\mathrm{ref}}(y^{+}\\!\\mid x)} \\\\).\n* `rewards/rejected`: The average implicit reward computed for the rejected completion, computed as \\\\( \\beta \\log \\frac{\\pi_{\\theta}(y^{-}\\!\\mid x)}{\\pi_{\\mathrm{ref}}(y^{-}\\!\\mid x)} \\\\).\n* `rewards/margins`: The average implicit reward margin between the chosen and rejected completions.\n* `rewards/accuracies`: The proportion of examples where the implicit reward for the chosen completion is higher than that for the rejected completion.\n\n## Customization\n\n### Compatibility and constraints\n\nSome argument combinations are intentionally restricted in the current [`DPOTrainer`] implementation:\n\n* `use_weighting=True` is not supported with `loss_type=\"aot\"` or `loss_type=\"aot_unpaired\"`.\n* With `use_liger_kernel=True`:\n * only a single `loss_type` is supported,\n * `compute_metrics` is not supported,\n * `precompute_ref_log_probs=True` is not supported.\n* `sync_ref_model=True` is not supported when training with PEFT models that do not keep a standalone `ref_model`.\n* `sync_ref_model=True` cannot be combined with `precompute_ref_log_probs=True`.\n* `precompute_ref_log_probs=True` is not supported with `IterableDataset` (train or eval).\n\n### Multi-loss combinations\n\nThe DPO trainer supports combining multiple loss functions with different weights, enabling more sophisticated optimization strategies. This is particularly useful for implementing algorithms like MPO (Mixed Preference Optimization). MPO is a training approach that combines multiple optimization objectives, as described in the paper [Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization](https://huggingface.co/papers/2411.10442).\n\nTo combine multiple losses, specify the loss types and corresponding weights as lists:\n\n```python\n# MPO: Combines DPO (sigmoid) for preference and BCO (bco_pair) for quality\ntraining_args = DPOConfig(\n loss_type=[\"sigmoid\", \"bco_pair\", \"sft\"], # loss types to combine\n loss_weights=[0.8, 0.2, 1.0] # corresponding weights, as used in the MPO paper\n)\n```\n\n### Model initialization\n\nYou can directly pass the kwargs of the [`~transformers.AutoModelForCausalLM.from_pretrained()`] method to the [`DPOConfig`]. For example, if you want to load a model in a different precision, analogous to\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen3-0.6B\", dtype=torch.bfloat16)\n```\n\nyou can do so by passing the `model_init_kwargs={\"dtype\": torch.bfloat16}` argument to the [`DPOConfig`].\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n model_init_kwargs={\"dtype\": torch.bfloat16},\n)\n```\n\nNote that all keyword arguments of [`~transformers.AutoModelForCausalLM.from_pretrained()`] are supported.\n\n### Train adapters with PEFT\n\nWe support tight integration with ๐Ÿค— PEFT library, allowing any user to conveniently train adapters and share them on the Hub, rather than training the entire model.\n\n```python\nfrom datasets import load_dataset\nfrom trl import DPOTrainer\nfrom peft import LoraConfig\n\ndataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\ntrainer = DPOTrainer(\n \"Qwen/Qwen3-0.6B\",\n train_dataset=dataset,\n peft_config=LoraConfig(),\n)\n\ntrainer.train()\n```\n\nYou can also continue training your [`~peft.PeftModel`]. For that, first load a `PeftModel` outside [`DPOTrainer`] and pass it directly to the trainer without the `peft_config` argument being passed.\n\n```python\nfrom datasets import load_dataset\nfrom trl import DPOTrainer\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\"trl-lib/Qwen3-4B-LoRA\", is_trainable=True)\ndataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\ntrainer = DPOTrainer(\n model=model,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\n> [!TIP]\n> When training adapters, you typically use a higher learning rate (โ‰ˆ1eโ€‘5) than full fine-tuning since only new parameters are being learned.\n>\n> ```python\n> DPOConfig(learning_rate=1e-5, ...)\n> ```\n\n### Train with Liger Kernel\n\nLiger Kernel is a collection of Triton kernels for LLM training that boosts multi-GPU throughput by 20%, cuts memory use by 60% (enabling up to 4ร— longer context), and works seamlessly with tools like FlashAttention, PyTorch FSDP, and DeepSpeed. For more information, see [Liger Kernel Integration](liger_kernel_integration).\n\n### Rapid Experimentation for DPO\n\nRapidFire AI is an open-source experimentation engine that sits on top of TRL and lets you launch multiple DPO configurations at once, even on a single GPU. Instead of trying configurations sequentially, RapidFire lets you **see all their learning curves earlier, stop underperforming runs, and clone promising ones with new settings in flight** without restarting. For more information, see [RapidFire AI Integration](rapidfire_integration).\n\n### Train with Unsloth\n\nUnsloth is an openโ€‘source framework for fineโ€‘tuning and reinforcement learning that trains LLMs (like Llama, Mistral, Gemma, DeepSeek, and more) up to 2ร— faster with up to 70% less VRAM, while providing a streamlined, Hugging Faceโ€“compatible workflow for training, evaluation, and deployment. For more information, see [Unsloth Integration](unsloth_integration).\n\n## Tool Calling with DPO\n\nThe [`DPOTrainer`] fully supports fine-tuning models with _tool calling_ capabilities. In this case, each dataset example should include:\n\n* The conversation messages (prompt, chosen and rejected), including any tool calls (`tool_calls`) and tool responses (`tool` role messages)\n* The list of available tools in the `tools` column, typically provided as JSON schemas\n\nFor details on the expected dataset structure, see the [Dataset Format โ€” Tool Calling](dataset_formats#tool-calling) section.\n\n## Training Vision Language Models\n\n[`DPOTrainer`] fully supports training Vision-Language Models (VLMs). To train a VLM, provide a dataset with either an `image` column (single image per sample) or an `images` column (list of images per sample). For more information on the expected dataset structure, see the [Dataset Format โ€” Vision Dataset](dataset_formats#vision-dataset) section.\nAn example of such a dataset is the [RLAIF-V Dataset](https://huggingface.co/datasets/HuggingFaceH4/rlaif-v_formatted) dataset.\n\n```python\nfrom trl import DPOConfig, DPOTrainer\nfrom datasets import load_dataset\n\ntrainer = DPOTrainer(\n model=\"Qwen/Qwen2.5-VL-3B-Instruct\",\n args=DPOConfig(max_length=None),\n train_dataset=load_dataset(\"HuggingFaceH4/rlaif-v_formatted\", split=\"train\"),\n)\ntrainer.train()\n```\n\n> [!TIP]\n> For VLMs, truncating may remove image tokens, leading to errors during training. To avoid this, set `max_length=None` in the [`DPOConfig`]. This allows the model to process the full sequence length without truncating image tokens.\n>\n> ```python\n> DPOConfig(max_length=None, ...)\n> ```\n>\n> Only use `max_length` when you've verified that truncation won't remove image tokens for the entire dataset.\n\n## DPOTrainer\n\n[[autodoc]] DPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## DPOConfig\n\n[[autodoc]] DPOConfig\n" }, { "path": "docs/source/example_overview.md", "content": "# Examples\n\nThis directory contains a collection of examples that demonstrate how to use the TRL library for various applications. We provide both **scripts** for advanced use cases and **notebooks** for an easy start and interactive experimentation.\n\nThe notebooks are self-contained and can run on **free Colab**, while the scripts can run on **single GPU, multi-GPU, or DeepSpeed** setups.\n\n**Getting Started**\n\nInstall TRL and additional dependencies as follows:\n\n```bash\npip install --upgrade trl[quantization]\n```\n\nCheck for additional optional dependencies [here](https://github.com/huggingface/trl/blob/main/pyproject.toml).\n\nFor scripts, you will also need an ๐Ÿค— Accelerate config (recommended for multi-gpu settings):\n\n```bash\naccelerate config # will prompt you to define the training configuration\n```\n\nThis allows you to run scripts with `accelerate launch` in single or multi-GPU settings.\n\n## Notebooks\n\nThese notebooks are easier to run and are designed for quick experimentation with TRL. The list of notebooks can be found in the [`trl/examples/notebooks/`](https://github.com/huggingface/trl/tree/main/examples/notebooks/) directory.\n\n\n| Notebook | Description | Open in Colab |\n|----------|-------------|---------------|\n| [`grpo_trl_lora_qlora.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_trl_lora_qlora.ipynb) | GRPO using QLoRA on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_trl_lora_qlora.ipynb) |\n| [`grpo_functiongemma_browsergym_openenv.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_functiongemma_browsergym_openenv.ipynb) | GRPO on FunctionGemma in the BrowserGym environment | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_functiongemma_browsergym_openenv.ipynb) |\n| [`grpo_agent.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_agent.ipynb) | GRPO for agent training | Not available due to OOM with Colab GPUs |\n| [`grpo_rnj_1_instruct.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_rnj_1_instruct.ipynb) | GRPO rnj-1-instruct with QLoRA using TRL on Colab to add reasoning capabilities | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_rnj_1_instruct.ipynb) |\n| [`sft_ministral3_vl.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_ministral3_vl.ipynb) | Supervised Fine-Tuning (SFT) Ministral 3 with QLoRA using TRL on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_ministral3_vl.ipynb) |\n| [`grpo_ministral3_vl.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_ministral3_vl.ipynb) | GRPO Ministral 3 with QLoRA using TRL on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_ministral3_vl.ipynb) |\n| [`openenv_sudoku_grpo.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/openenv_sudoku_grpo.ipynb) | GRPO to play Sudoku on an OpenEnv environment | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/openenv_sudoku_grpo.ipynb) |\n| [`openenv_wordle_grpo.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/openenv_wordle_grpo.ipynb) | GRPO to play Worldle on an OpenEnv environment | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/openenv_wordle_grpo.ipynb) |\n| [`sft_nemotron_3.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_nemotron_3.ipynb) | SFT with LoRA on NVIDIA Nemotron 3 models | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_nemotron_3.ipynb) |\n| [`sft_trl_lora_qlora.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_trl_lora_qlora.ipynb) | Supervised Fine-Tuning (SFT) using QLoRA on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_trl_lora_qlora.ipynb) |\n| [`sft_qwen_vl.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_qwen_vl.ipynb) | Supervised Fine-Tuning (SFT) Qwen3-VL with QLoRA using TRL on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_qwen_vl.ipynb) |\n| [`sft_tool_calling.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_tool_calling.ipynb) | Teaching tool calling to a model without native tool-calling support using SFT with QLoRA | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_tool_calling.ipynb) |\n| [`grpo_qwen3_vl.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_qwen3_vl.ipynb) | GRPO Qwen3-VL with QLoRA using TRL on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_qwen3_vl.ipynb) |\n\n## Scripts\n\nScripts are maintained in the [`trl/scripts`](https://github.com/huggingface/trl/blob/main/trl/scripts) and [`examples/scripts`](https://github.com/huggingface/trl/blob/main/examples/scripts) directories. They show how to use different trainers such as [`SFTTrainer`], [`PPOTrainer`], [`DPOTrainer`], [`GRPOTrainer`], and more.\n\n| File | Description |\n| --- | --- |\n| [`examples/scripts/bco.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/bco.py) | This script shows how to use the [`experimental.kto.KTOTrainer`] with the BCO loss to fine-tune a model to increase instruction-following, truthfulness, honesty, and helpfulness using the [openbmb/UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) dataset. |\n| [`examples/scripts/cpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/cpo.py) | This script shows how to use the [`experimental.cpo.CPOTrainer`] to fine-tune a model to increase helpfulness and harmlessness using the [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset. |\n| [`trl/scripts/dpo.py`](https://github.com/huggingface/trl/blob/main/trl/scripts/dpo.py) | This script shows how to use the [`DPOTrainer`] to fine-tune a model. |\n| [`examples/scripts/dpo_vlm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/dpo_vlm.py) | This script shows how to use the [`DPOTrainer`] to fine-tune a Vision Language Model to reduce hallucinations using the [openbmb/RLAIF-V-Dataset](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset) dataset. |\n| [`examples/scripts/evals/judge_tldr.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/evals/judge_tldr.py) | This script shows how to use [`experimental.judges.HfPairwiseJudge`] or [`experimental.judges.OpenAIPairwiseJudge`] to judge model generations. |\n| [`examples/scripts/gkd.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/gkd.py) | This script shows how to use the [`experimental.gkd.GKDTrainer`] to fine-tune a model. |\n| [`trl/scripts/grpo.py`](https://github.com/huggingface/trl/blob/main/trl/scripts/grpo.py) | This script shows how to use the [`GRPOTrainer`] to fine-tune a model. |\n| [`trl/scripts/grpo_agent.py`](https://github.com/huggingface/trl/blob/main/trl/scripts/grpo_agent.py) | This script shows how to use the [`GRPOTrainer`] to fine-tune a model to enable agentic usage. |\n| [`examples/scripts/grpo_vlm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/grpo_vlm.py) | This script shows how to use the [`GRPOTrainer`] to fine-tune a multimodal model for reasoning using the [lmms-lab/multimodal-open-r1-8k-verified](https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified) dataset. |\n| [`examples/scripts/gspo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/gspo.py) | This script shows how to use GSPO via the [`GRPOTrainer`] to fine-tune model for reasoning using the [AI-MO/NuminaMath-TIR](https://huggingface.co/datasets/AI-MO/NuminaMath-TIR) dataset. |\n| [`examples/scripts/gspo_vlm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/gspo_vlm.py) | This script shows how to use GSPO via the [`GRPOTrainer`] to fine-tune a multimodal model for reasoning using the [lmms-lab/multimodal-open-r1-8k-verified](https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified) dataset. |\n| [`examples/scripts/kto.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/kto.py) | This script shows how to use the [`experimental.kto.KTOTrainer`] to fine-tune a model. |\n| [`examples/scripts/mpo_vlm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/mpo_vlm.py) | This script shows how to use MPO via the [`DPOTrainer`] to align a model based on preferences using the [HuggingFaceH4/rlaif-v_formatted](https://huggingface.co/datasets/HuggingFaceH4/rlaif-v_formatted) dataset and a set of loss weights with weights. |\n| [`examples/scripts/nash_md.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/nash_md.py) | This script shows how to use the [`experimental.nash_md.NashMDTrainer`] to fine-tune a model. |\n| [`examples/scripts/nemo_gym/train_multi_environment.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/nemo_gym/train_multi_environment.py) | This script shows how to use the [`GRPOTrainer`] to train language models in NVIDIA NeMo-Gym environments. Supports multi-turn and tool calling environments, and multi-environment training. See the [NeMo-Gym Integration](nemo_gym) guide for setup and usage. |\n| [`examples/scripts/online_dpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/online_dpo.py) | This script shows how to use the [`experimental.online_dpo.OnlineDPOTrainer`] to fine-tune a model. |\n| [`examples/scripts/online_dpo_vlm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/online_dpo_vlm.py) | This script shows how to use the [`experimental.online_dpo.OnlineDPOTrainer`] to fine-tune a a Vision Language Model. |\n| [`examples/scripts/openenv/browsergym.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/browsergym.py) | Simple script to run GRPO training via the [`GRPOTrainer`] with OpenEnv's BrowserGym environment and vLLM for VLMs |\n| [`examples/scripts/openenv/browsergym_llm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/browsergym_llm.py) | Simple script to run GRPO training via the [`GRPOTrainer`] with OpenEnv's BrowserGym environment and vLLM for LLMs |\n| [`examples/scripts/openenv/carla.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/carla.py) | Simple script to run GRPO training via the [`GRPOTrainer`] with OpenEnv's CARLA environment for autonomous driving scenarios. |\n| [`examples/scripts/openenv/catch.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/catch.py) | Simple script to run GRPO training via the [`GRPOTrainer`] with OpenEnv's Catch environment (OpenSpiel) and vLLM |\n| [`examples/scripts/openenv/echo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/echo.py) | Simple script to run GRPO training via the [`GRPOTrainer`] with OpenEnv's Echo environment and vLLM. |\n| [`examples/scripts/openenv/sudoku.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/sudoku.py) | Simple script to run GRPO training via the [`GRPOTrainer`] with OpenEnv's Sudoku environment and vLLM. |\n| [`examples/scripts/openenv/wordle.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/wordle.py) | Simple script to run GRPO training via the [`GRPOTrainer`] with OpenEnv's Wordle environment and vLLM. |\n| [`examples/scripts/orpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/orpo.py) | This script shows how to use the [`experimental.orpo.ORPOTrainer`] to fine-tune a model to increase helpfulness and harmlessness using the [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset. |\n| [`examples/scripts/ppo/ppo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/ppo/ppo.py) | This script shows how to use the [`experimental.ppo.PPOTrainer`] to fine-tune a model to improve its ability to continue text with positive sentiment or physically descriptive language. |\n| [`examples/scripts/ppo/ppo_tldr.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/ppo/ppo_tldr.py) | This script shows how to use the [`experimental.ppo.PPOTrainer`] to fine-tune a model to improve its ability to generate TL;DR summaries. |\n| [`examples/scripts/prm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/prm.py) | This script shows how to use the [`experimental.prm.PRMTrainer`] to fine-tune a Process-supervised Reward Model (PRM). |\n| [`examples/scripts/reward_modeling.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/reward_modeling.py) | This script shows how to use the [`RewardTrainer`] to train an Outcome Reward Model (ORM) on your own dataset. |\n| [`examples/scripts/rloo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/rloo.py) | This script shows how to use the [`RLOOTrainer`] to fine-tune a model to improve its ability to solve math questions. |\n| [`examples/scripts/sft.py`](https://github.com/huggingface/trl/blob/main/trl/scripts/sft.py) | This script shows how to use the [`SFTTrainer`] to fine-tune a model. |\n| [`examples/scripts/sft_gemma3.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/sft_gemma3.py) | This script shows how to use the [`SFTTrainer`] to fine-tune a Gemma 3 model. |\n| [`examples/scripts/sft_nemotron_3.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/sft_nemotron_3.py) | This script shows how to use the [`SFTTrainer`] to fine-tune an NVIDIA Nemotron 3 model. |\n| [`examples/scripts/sft_tiny_aya_tool_calling.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/sft_tiny_aya_tool_calling.py) | This script shows how to use the [`SFTTrainer`] to teach tool calling to a model without native tool-calling support using the [bebechien/SimpleToolCalling](https://huggingface.co/datasets/bebechien/SimpleToolCalling) dataset. |\n| [`examples/scripts/sft_video_llm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/sft_video_llm.py) | This script shows how to use the [`SFTTrainer`] to fine-tune a Video Language Model. |\n| [`examples/scripts/sft_vlm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/sft_vlm.py) | This script shows how to use the [`SFTTrainer`] to fine-tune a Vision Language Model in a chat setting. The script has only been tested with [LLaVA 1.5](https://huggingface.co/llava-hf/llava-1.5-7b-hf), [LLaVA 1.6](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf), and [Llama-3.2-11B-Vision-Instruct](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct) models, so users may see unexpected behaviour in other model architectures. |\n| [`examples/scripts/sft_vlm_gemma3.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/sft_vlm_gemma3.py) | This script shows how to use the [`SFTTrainer`] to fine-tune a Gemma 3 model on vision to text tasks. |\n| [`examples/scripts/sft_vlm_smol_vlm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/sft_vlm_smol_vlm.py) | This script shows how to use the [`SFTTrainer`] to fine-tune a SmolVLM model. |\n| [`examples/scripts/xpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/xpo.py) | This script shows how to use the [`experimental.xpo.XPOTrainer`] to fine-tune a model. |\n\n## Distributed Training (for scripts)\n\nYou can run scripts on multiple GPUs with ๐Ÿค— Accelerate:\n\n```shell\naccelerate launch --config_file=examples/accelerate_configs/multi_gpu.yaml --num_processes {NUM_GPUS} path_to_script.py --all_arguments_of_the_script\n```\n\nFor DeepSpeed ZeRO-{1,2,3}:\n\n```shell\naccelerate launch --config_file=examples/accelerate_configs/deepspeed_zero{1,2,3}.yaml --num_processes {NUM_GPUS} path_to_script.py --all_arguments_of_the_script\n```\n\nAdjust `NUM_GPUS` and `--all_arguments_of_the_script` as needed.\n" }, { "path": "docs/source/experimental_overview.md", "content": "# Experimental\n\nThis directory contains a minimal, clearly separated space for fast iteration on new ideas.\n\n> [!WARNING]\n> **Stability contract:** Anything under `trl.experimental` may change or be removed in *any* release (including patch versions) without prior deprecation. Do not rely on these APIs for production workloads.\n\n## Promotion Path (Simple)\n\n1. **Prototype outside the main repo:** Start development in your own fork or a separate repository to iterate quickly.\n2. **Experimental inclusion:** Once itโ€™s ready for early users, move the idea into `trl.experimental.`.\n3. **Improve:** Add tests, a short doc/example, and demonstrate the usage.\n4. **Promote:** Once the API proves stable and there is clear interest or adoption from the community, move it into `trl.` (stable module).\n\n## FAQ\n\n**Why not just use branches?**\nBecause branches are not shipped to users; experimental code inside the package lets early adopters try things and give feedback.\n\n**Can these APIs change or vanish without warning?**\nYes. Anything inside `trl.experimental` can change or disappear in *any* release.\n\n**Should I use this in production?**\nOnly if you are fine with updating your code quickly when things change.\n\n**Will maintainers promptly fix issues in `trl.experimental`?**\nNot necessarily. The experimental module is a playground for new ideas, and maintainers may not prioritize bug fixes or feature requests there. Issues may remain unresolved until (or unless) the feature graduates to the stable API.\n\n**How to silence the runtime notice?**\n\nUse: `export TRL_EXPERIMENTAL_SILENCE=1`.\n" }, { "path": "docs/source/gfpo.md", "content": "# GFPO\n\nThis feature implements the GFPO algorithm to enforce concise reasoning in the model's output generation, as proposed in the paper [Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning](https://huggingface.co/papers/2508.09726).\n\n## Usage\n\nTo activate GFPO in [`GFPOTrainer`]:\n\n- set `num_remains_in_group` in [`GFPOConfig`]\n- define a group filter function and set it to `group_filter_func` in [`GFPOTrainer`]. `group_filter_func` will score the `num_generations` completions and The GFPOTrainer filters groups according to their scores to get top `num_remains_in_group` completions as a new group. Model will be trained on the filtered group.\n\n```python\n# train_gfpo.py\nfrom trl.experimental.gfpo import GFPOConfig, GFPOTrainer\n\n# dummy group filter to scores the completions based on its indice in group\nclass GroupFilter:\n def __call__(self, group_completions, group_rewards, **kwargs):\n group_scores = []\n for completions, rewards in zip(group_completions, group_rewards):\n scores = [float(i) for i in range(len(completions))]\n group_scores.append(scores)\n return group_scores\n\ntraining_args = GFPOConfig(\n output_dir=\"Qwen3-0.6B-GFPO\",\n per_device_train_batch_size=4,\n num_remains_in_group=2,\n bf16=True,\n)\ntrainer = GFPOTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n reward_funcs=...,\n train_dataset=...,\n args=training_args,\n group_filter_func=GroupFilter(),\n)\ntrainer.train()\n```\n\n## GFPOTrainer\n\n[[autodoc]] experimental.gfpo.GFPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## GFPOConfig\n\n[[autodoc]] experimental.gfpo.GFPOConfig\n" }, { "path": "docs/source/gkd_trainer.md", "content": "# Generalized Knowledge Distillation Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-GKD-blue)](https://huggingface.co/models?other=gkd,trl)\n\n## Overview\n\nGeneralized Knowledge Distillation (GKD) was proposed in [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://huggingface.co/papers/2306.13649) by Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem.\n\nThe abstract from the paper is the following:\n\n> Knowledge distillation (KD) is widely used for compressing a teacher model to reduce its inference cost and memory footprint, by training a smaller student model. However, current KD methods for auto-regressive sequence models suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference. To address this issue, we introduce Generalized Knowledge Distillation (GKD). Instead of solely relying on a fixed set of output sequences, GKD trains the student on its self-generated output sequences by leveraging feedback from the teacher on such sequences. Unlike supervised KD approaches, GKD also offers the flexibility to employ alternative loss functions between the student and teacher, which can be useful when the student lacks the expressivity to mimic the teacher's distribution. Furthermore, GKD facilitates the seamless integration of distillation with RL fine-tuning (RLHF). We demonstrate the efficacy of GKD for distilling auto-regressive language models on summarization, translation, and arithmetic reasoning tasks, and task-agnostic distillation for instruction-tuning.\n\nThe key aspects of GKD are:\n\n1. It addresses the train-inference distribution mismatch in auto-regressive sequence models by training the student model on its self-generated output sequences.\n2. GKD allows flexibility in choosing different divergence measures between student and teacher models via the generalized Jensen-Shannon Divergence (JSD), which can be useful when the student lacks the capacity to fully mimic the teacher.\n\nThis post-training method was contributed by [Kashif Rasul](https://huggingface.co/kashif) and [Lewis Tunstall](https://huggingface.co/lewtun).\n\n## Usage tips\n\nThe [`experimental.gkd.GKDTrainer`] is a wrapper around the [`SFTTrainer`] class that takes in a teacher model argument. It needs three parameters to be set via the [`experimental.gkd.GKDConfig`] namely:\n\n* `lmbda`: controls the student data fraction, i.e., the proportion of on-policy student-generated outputs. When `lmbda=0.0`, the loss reduces to supervised JSD where the student is trained with the token-level probabilities of the teacher. When `lmbda=1.0`, the loss reduces to on-policy JSD, where the student generates output sequences and token-specific feedback on these sequences from the teacher. For values in between [0, 1] it is random between the two based on the `lmbda` value for each batch.\n* `seq_kd`: controls whether to perform Sequence-Level KD (can be viewed as supervised FT on teacher-generated out). When `seq_kd=True` and `lmbda=0.0`, the loss reduces to supervised JSD, where the teacher generates output sequences and the student receives token-specific feedback on these sequences from the teacher.\n* `beta`: controls the interpolation in the generalized Jensen-Shannon Divergence. When `beta=0.0` the loss approximates forward KL divergence, while for `beta=1.0` the loss approximates reverse KL divergence. For values in between [0, 1] it interpolates between the two.\n\nThe authors find that on-policy data (high `lmbda`) performs better and the optimal `beta` varied depending on the task and evaluation method.\n\n> [!WARNING]\n> Make sure that `attn_implementation=\"kernels-community/flash-attn2\"` when training [Gemma models](https://huggingface.co/models?other=gemma2). Otherwise you will encounter NaNs in the logits due to the [soft capping technique](https://huggingface.co/blog/gemma2#soft-capping-and-attention-implementations) adopted by this architecture.\n\nThe basic API is as follows:\n\n```python\nfrom datasets import Dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl.experimental.gkd import GKDConfig, GKDTrainer\n\nNUM_DUMMY_SAMPLES = 100\n\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\n# The model to optimise\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\n# The teacher model to calculate the KL divergence against\nteacher_model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-1.5B-Instruct\")\n\ntrain_dataset = Dataset.from_dict(\n {\n \"messages\": [\n [\n {\"role\": \"user\", \"content\": \"Hi, how are you?\"},\n {\"role\": \"assistant\", \"content\": \"I'm great thanks\"},\n ]\n ]\n * NUM_DUMMY_SAMPLES\n }\n)\neval_dataset = Dataset.from_dict(\n {\n \"messages\": [\n [\n {\"role\": \"user\", \"content\": \"What colour is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"The sky is blue\"},\n ]\n ]\n * NUM_DUMMY_SAMPLES\n }\n)\n\ntraining_args = GKDConfig(output_dir=\"gkd-model\", per_device_train_batch_size=1)\ntrainer = GKDTrainer(\n model=model,\n teacher_model=teacher_model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n)\ntrainer.train()\n```\n\n### Expected dataset type\n\nThe dataset should be formatted as a list of \"messages\" where each message is a list of dictionaries with the following keys:\n\n* `role`: either `system`, `assistant` or `user`\n* `content`: the message content\n\n## GKDTrainer\n\n[[autodoc]] experimental.gkd.GKDTrainer\n - train\n - save_model\n - push_to_hub\n\n## GKDConfig\n\n[[autodoc]] experimental.gkd.GKDConfig\n" }, { "path": "docs/source/gold_trainer.md", "content": "# General Online Logit Distillation (GOLD) Trainer\n\n[![All_models-GOLD-blue](https://img.shields.io/badge/All_models-GOLD-blue)](https://huggingface.co/models?other=sft,gold)\n\n## Overview\n\nGeneral Online Logit Distillation (GOLD) is an extension of Universal Logit Distillation (ULD) that supports\nstudent/teacher pairs with different tokenizers. It aligns the textual spans produced by both tokenizers and merges the\nassociated logits so no completion tokens are dropped. This enables cross-tokenizer knowledge distillation, including\nmixed model families (for example, LLaMA students with Qwen teachers).\n\nKey capabilities:\n\n1. **Cross-tokenizer alignment** โ€“ GOLD incrementally decodes the student and teacher tokens, groups passages with the same visible text, and merges probabilities inside each group. This guarantees loss terms are computed over the full completion even when token boundaries differ.\n2. **Hybrid ULD loss** โ€“ when `uld_use_hybrid_loss` is enabled, GOLD compares exact vocabulary matches directly and falls back to the original sorted-probability ULD loss for unmatched tokens. This improves stability for students whose vocabularies only partially overlap with the teacher.\n3. **Seamless integration with GKD** โ€“ GOLD inherits the on-policy vs. off-policy scheduling from the [`experimental.gkd.GKDTrainer`], so you can combine sequence-level KD, generalized JSD, and cross-tokenizer distillation in a single training run.\n\n> [!NOTE]\n> GOLD is currently part of the `trl.experimental` namespace. APIs may change without notice while the feature is iterated on.\n\n## Usage tips\n\nThe [`GOLDTrainer`] subclasses [`SFTTrainer`] and accepts the same datasets as other TRL trainers (lists of ChatML style\nmessages). Important configuration flags on [`GOLDConfig`] include:\n\n* `use_uld_loss` โ€“ toggles Universal Logit Distillation. Set this to `True` for cross-tokenizer setups.\n* `teacher_tokenizer_name_or_path` โ€“ required when `use_uld_loss=True`; GOLD uses the teacher tokenizer to align tokens.\n* `uld_use_hybrid_loss`, `uld_hybrid_matched_weight`, `uld_hybrid_unmatched_weight` โ€“ enables and weights the hybrid\n matched/unmatched loss.\n* `beta`, `lmbda`, `seq_kd` โ€“ inherited from [`experimental.gkd.GKDConfig`], controlling the generalized JSD interpolation and on-policy\n sampling ratio.\n* `num_generations`, `generation_batch_size` โ€“ control buffered rollout generation across gradient accumulation windows.\n `generation_batch_size` is the number of unique prompts per worker per optimizer step.\n* `model_revision` โ€“ controls which student model revision GOLD loads for training and generation.\n\nA minimal end-to-end example:\n\n```python\nfrom datasets import load_dataset\nfrom trl.experimental.gold import GOLDConfig, GOLDTrainer\n\ntrain_dataset = load_dataset(\n \"HuggingFaceTB/OpenR1-Math-220k-default-verified\",\n \"all\",\n split=\"train[:1024]\",\n)\n\ntrainer = GOLDTrainer(\n model=\"meta-llama/Llama-3.2-1B-Instruct\",\n teacher_model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n args=GOLDConfig(output_dir=\"gold-model\", use_uld_loss=True, teacher_tokenizer_name_or_path=\"Qwen/Qwen2.5-0.5B-Instruct\"),\n train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\nFor quick-start workflows you can rely on string identifiers as shown aboveโ€”the trainer will load the model and tokenizer for you. Explicitly instantiating `AutoModelForCausalLM`, `AutoTokenizer`, or populating `GOLDConfig` is recommended only for advanced use cases where you need fine-grained control over initialization.\n\nA more explicit setup might look like this when you need to customise model loading, tokenizer settings, or training arguments:\n\n```python\nfrom datasets import load_dataset\nfrom trl import GOLDConfig, GOLDTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nstudent_name = \"meta-llama/Llama-3.2-1B-Instruct\"\nteacher_name = \"Qwen/Qwen2.5-0.5B-Instruct\"\n\ntokenizer = AutoTokenizer.from_pretrained(student_name)\nif tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\nmodel = AutoModelForCausalLM.from_pretrained(student_name)\nteacher_model = AutoModelForCausalLM.from_pretrained(teacher_name)\n\ntrain_dataset = load_dataset(\n \"HuggingFaceTB/Countdown-Task-GOLD\",\n \"verified_Qwen2.5-0.5B-Instruct\",\n split=\"train\",\n)\n\ntraining_args = GOLDConfig(\n output_dir=\"gold-model\",\n per_device_train_batch_size=1,\n teacher_model_name_or_path=teacher_name,\n teacher_tokenizer_name_or_path=teacher_name,\n use_uld_loss=True,\n uld_use_hybrid_loss=True,\n)\n\ntrainer = GOLDTrainer(\n model=model,\n teacher_model=teacher_model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n> [!NOTE]\n> GOLD buffers one full optimizer-window generation batch (`per_device_train_batch_size * gradient_accumulation_steps`)\n> and reuses it across accumulation steps. If the final batch is undersized, GOLD warns and drops that last batch\n> (`Dropping last batch due to unexpected batch size`). Set `dataloader_drop_last=True` to avoid this warning.\n\n### Expected dataset type\n\nGOLD requires a [conversational](dataset_formats#conversational) [language modeling](dataset_formats#language_modeling) dataset, e.g.:\n\n```python\n{\"messages\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"}]}\n```\n\n`GOLDTrainer` keeps the raw messages so the ChatML collator can construct prompts and completions with the correct\nboundaries.\n\n## How Token Merging Works\n\nWhen student and teacher use different tokenizers, the same text may be split differently:\n\n- **Student**: `\"Hugging Face\"` โ†’ 1 token\n- **Teacher**: `\"Hugging\"`, `\" Face\"` โ†’ 2 tokens\n\nGOLD aligns these sequences and merges the teacher's multi-token probabilities into a single distribution that can be compared with the student's single-token distribution.\n\n### Probability Merging\n\nFor a teacher sequence of tokens `[tokenโ‚€, tokenโ‚, ..., tokenโ‚–]` that maps to a single student token, GOLD computes:\n\n```\nP_merged(y) = P(y | context) ร— P(tokenโ‚ | tokenโ‚€, context) ร— ... ร— P(tokenโ‚– | ..., context)\n```\n\nwhere:\n- `P(y | context)` is the marginal probability distribution over all vocabulary tokens at the first position\n- `P(tokenแตข | ..., context)` are **scalar** conditional probabilities of the actual tokens that were generated\n\n**Key insight**: Only the conditional probabilities of the **actual continuation tokens** are extracted as scalars. The full marginal distribution at the first position is then scaled by multiplying these scalar probabilities.\n\nThis ensures:\n1. **Correct joint probability** for the actual generated sequence (by the chain rule)\n2. **Reasonable approximation** for counterfactual tokens (scaled by the same continuation likelihood)\n3. **Unnormalized distributions** that preserve the correct relative probabilities for ULD loss computation\n\n### Example\n\nGiven:\n```\nP(xโ‚€): [\"HF\": 0.6, \"is\": 0.3, \"cool\": 0.1]\nP(xโ‚ | \"HF\"): [\"HF\": 0.05, \"is\": 0.9, \"cool\": 0.05]\n```\n\nIf tokens 0 and 1 are merged, and the actual sequence was `[\"HF\", \"is\"]`:\n```\nP_merged(\"HF\") = 0.6 ร— 0.9 = 0.54 โœ“ (correct joint probability)\nP_merged(\"is\") = 0.3 ร— 0.9 = 0.27\nP_merged(\"cool\") = 0.1 ร— 0.9 = 0.09\n```\n\nThe merged distribution is unnormalized (sums to 0.81), but this is intentional and correct for ULD loss computation, which uses sorting and L1 distance.\n\n## GOLDTrainer\n\n[[autodoc]] experimental.gold.GOLDTrainer\n - train\n - generate_on_policy_outputs\n - save_model\n - push_to_hub\n\n## GOLDConfig\n\n[[autodoc]] experimental.gold.GOLDConfig\n" }, { "path": "docs/source/grpo_trainer.md", "content": "# GRPO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-GRPO-blue)](https://huggingface.co/models?other=grpo,trl)\n\n## Overview\n\nTRL supports the GRPO Trainer for training language models, as described in the paper [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300) by [Zhihong Shao](https://huggingface.co/syhia), [Peiyi Wang](https://huggingface.co/peiyiwang89), [Qihao Zhu](https://huggingface.co/zqh11), Runxin Xu, [Junxiao Song](https://huggingface.co/haha-point), Mingchuan Zhang, Y. K. Li, Y. Wu, [Daya Guo](https://huggingface.co/guoday).\n\nThe abstract from the paper is the following:\n\n> Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.\n\nThis post-training method was contributed by [Quentin Gallouรฉdec](https://huggingface.co/qgallouedec).\n\n## Quick start\n\nThis example demonstrates how to train a model using the GRPO method. We train a [Qwen 0.5B Instruct model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) with the prompts from the [DeepMath-103K dataset](https://huggingface.co/datasets/trl-lib/DeepMath-103K). You can view the data in the dataset here:\n\n\n\nBelow is the script to train the model.\n\n```python\n# train_grpo.py\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\nfrom trl.rewards import accuracy_reward\n\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen2-0.5B-Instruct\",\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\nExecute the script using the following command:\n\n```bash\naccelerate launch train_grpo.py\n```\n\nDistributed across 8 GPUs, the training takes approximately 1 day.\n\n![GRPO curves](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/grpo_curves.png)\n\n## Looking deeper into the GRPO method\n\nGRPO is an online learning algorithm, meaning it improves iteratively by using the data generated by the trained model itself during training. The intuition behind GRPO objective is to maximize the advantage of the generated completions, while ensuring that the model remains close to the reference policy. To understand how GRPO works, it can be broken down into four main steps: **Generating completions**, **computing the advantage**, **estimating the KL divergence**, and **computing the loss**.\n\n![GRPO visual](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/grpo_visual.png)\n\n### Generating completions\n\nAt each training step, we sample a batch of prompts and generate a set of \\\\( G \\\\) completions for each prompt (denoted as \\\\( o_i \\\\)).\n\n### Computing the advantage\n\nFor each of the \\\\( G \\\\) sequences, we compute the reward using a reward model or reward function. To align with the comparative nature of reward modelsโ€”typically trained on datasets of comparisons between outputs for the same questionโ€”the advantage is calculated to reflect these relative comparisons. It is normalized as follows:\n\n$$\\hat{A}_{i,t} = \\frac{r_i - \\text{mean}(\\mathbf{r})}{\\text{std}(\\mathbf{r})}$$\n\nThis approach gives the method its name: **Group Relative Policy Optimization (GRPO)**.\n\n> [!TIP]\n> It was shown in the paper [Understanding R1-Zero-Like Training: A Critical Perspective](https://huggingface.co/papers/2503.20783) that scaling by \\\\( \\text{std}(\\mathbf{r}) \\\\) may cause a question-level difficulty bias. You can disable this scaling by setting `scale_rewards=False` in [`GRPOConfig`].\n> Note that turning off std-based scaling also removes variance normalization, so update magnitudes depend directly on the raw reward scale and batch composition.\n\n> [!TIP]\n> As shown in [Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning (Lite PPO)](https://huggingface.co/papers/2508.08221), calculating the mean at the local (group) level and the standard deviation at the global (batch) level enables more robust reward shaping. You can use this scaling strategy by setting `scale_rewards=\"batch\"` in [`GRPOConfig`].\n\n### Estimating the KL divergence\n\nKL divergence is estimated using the approximator introduced by [Schulman et al. (2020)](http://joschu.net/blog/kl-approx.html). The approximator is defined as follows:\n\n$$\\mathbb{D}_{\\text{KL}}\\left[\\pi_\\theta \\|\\pi_{\\text{ref}}\\right] = \\frac{\\pi_{\\text{ref}}(o_{i,t} \\mid q, o_{i, [!TIP]\n> Note that compared to the original formulation in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300), we don't scale by \\\\( \\frac{1}{|o_i|} \\\\) because it was shown in the paper [Understanding R1-Zero-Like Training: A Critical Perspective](https://huggingface.co/papers/2503.20783) that this introduces a response-level length bias. More details in [loss types](#loss-types).\n\n> [!TIP]\n> Note that compared to the original formulation in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300), we use \\\\( \\beta = 0.0 \\\\) by default, meaning that the KL divergence term is not used. This choice is motivated by several recent studies (e.g., [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://huggingface.co/papers/2503.24290)) which have shown that the KL divergence term is not essential for training with GRPO. As a result, it has become common practice to exclude it (e.g. [Understanding R1-Zero-Like Training: A Critical Perspective](https://huggingface.co/papers/2503.20783), [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://huggingface.co/papers/2503.14476)). If you wish to include the KL divergence term, you can set `beta` in [`GRPOConfig`] to a non-zero value.\n\nIn the original paper, this formulation is generalized to account for multiple updates after each generation (denoted \\\\( \\mu \\\\), can be set with `num_iterations` in [`GRPOConfig`]) by leveraging the **clipped surrogate objective**:\n\n$$\n\\mathcal{L}_{\\text{GRPO}}(\\theta) = - \\frac{1}{\\sum_{i=1}^G |o_i|} \\sum_{i=1}^G \\sum_{t=1}^{|o_i|} \\left[ \\min \\left( \\frac{\\pi_\\theta(o_{i,t} \\mid q, o_{i,< t})}{\\pi_{\\theta_{\\text{old}}}(o_{i,t} \\mid q, o_{i,< t})} \\hat{A}_{i,t}, \\, \\text{clip}\\left( \\frac{\\pi_\\theta(o_{i,t} \\mid q, o_{i,< t})}{\\pi_{\\theta_{\\text{old}}}(o_{i,t} \\mid q, o_{i,< t})}, 1 - \\epsilon, 1 + \\epsilon \\right) \\hat{A}_{i,t} \\right) - \\beta \\mathbb{D}_{\\text{KL}}\\left[\\pi_\\theta \\| \\pi_{\\text{ref}}\\right] \\right],\n$$\n\nwhere \\\\(\\text{clip}(\\cdot, 1 - \\epsilon, 1 + \\epsilon) \\\\) ensures that updates do not deviate excessively from the reference policy by bounding the policy ratio between \\\\( 1 - \\epsilon \\\\) and \\\\( 1 + \\epsilon \\\\).\nWhen \\\\( \\mu = 1 \\\\) (default in TRL), the clipped surrogate objective simplifies to the original objective.\n\n#### Loss Types\n\nSeveral formulations of the objective have been proposed in the literature. Initially, the objective of GRPO was defined as follows:\n\n$$\n\\mathcal{L}_{\\text{GRPO}}(\\theta) = - \\frac{1}{G} \\sum_{i=1}^G \\frac{1}{|o_i|} \\sum_{t=1}^{|o_i|} l_{i,t},\n$$\n\nwhere\n\n$$\nl_{i,t} = \\frac{\\pi_\\theta(o_{i,t} \\mid q, o_{i,< t})}{\\left[\\pi_\\theta(o_{i,t} \\mid q, o_{i,< t})\\right]_{\\text{no grad}}} \\hat{A}_{i,t} - \\beta \\mathbb{D}_{\\text{KL}}\\left[\\pi_\\theta \\| \\pi_{\\text{ref}}\\right].\n$$\n\nThe [DAPO paper](https://huggingface.co/papers/2503.14476) highlights the limitations of the GRPO algorithmโ€™s sample-level loss in long-CoT scenarios, where longer responses are under-penalized, leading to poorer quality outputs. The proposed solution is a token-level normalization, which better handles longer sequences by assigning more balanced rewards to individual tokens, regardless of response length:\n\n$$\n\\mathcal{L}_{\\text{DAPO}}(\\theta) = - \\frac{1}{\\sum_{i=1}^G |o_i|} \\sum_{i=1}^G \\sum_{t=1}^{|o_i|} l_{i,t},\n$$\n\nTo use this formulation, set `loss_type=\"dapo\"` in [`GRPOConfig`].\n\nFurthermore, it was demonstrated in the paper [Understanding R1-Zero-Like Training: A Critical Perspective](https://huggingface.co/papers/2503.20783) that the initial GRPO formulation introduces a response length bias. They show that while the DAPO formulation reduces this bias, it does not eliminate it completely. To fully remove this bias, they propose dividing by a constant instead of the sequence length, resulting in the following formulation:\n\n$$\n\\mathcal{L}_{\\text{Dr. GRPO}}(\\theta) = - \\frac{1}{LG} \\sum_{i=1}^G \\sum_{t=1}^{|o_i|} l_{i,t},\n$$\n\nThis constant is recommended to be the maximum completion length. To use this formulation, set `loss_type=\"dr_grpo\"` in the [`GRPOConfig`].\n\nAlternatively, in the [SAPO paper](https://huggingface.co/papers/2511.20347), the Qwen team proposes replacing the \"hard\" clipping mechanism of GRPO with a smooth, temperature-controlled soft gating mechanism. While GRPO zeroes out gradients when the policy deviates too far from the reference, SAPO uses a soft trust region that smoothly decays the gradient weight. This allows the model to retain useful learning signals from \"near-on-policy\" tokens while suppressing noise from extreme deviations.\n\nThe loss function is defined as:\n\n$$\n\\mathcal{L}_{\\text{SAPO}}(\\theta) = - \\frac{1}{G} \\sum_{i=1}^G \\frac{1}{|o_i|} \\sum_{t=1}^{|o_i|} f_{i,t} \\left( \\frac{\\pi_\\theta(o_{i,t} | q, o_{i, 0 \\\\\n\\tau_{\\text{neg}}, & \\text{otherwise}\n\\end{cases}\n$$\n\nThey recommend using asymmetric temperatures, \\\\( \\tau_{\\text{neg}} > \\tau_{\\text{pos}} \\\\) (defaults are \\\\( \\tau_{\\text{pos}}=1.0, \\tau_{\\text{neg}}=1.05 \\\\) ). This ensures that the model is penalized more strictly for \"bad\" actions to prevent instability, while being more permissive with \"good\" actions.\n\nTo use this formulation, set `loss_type=\"sapo\"` in the [`GRPOConfig`].\n\n## Logged metrics\n\nWhile training and evaluating, we record the following reward metrics:\n\n- `num_tokens`: The total number of tokens processed so far, including both prompts and completions. When using tools, only non-tool tokens are counted.\n- `step_time`: The average time (in seconds) taken per training step (including generation).\n- `completions/mean_length`: The average length of generated completions. When using tools, only non-tool tokens are counted.\n- `completions/min_length`: The minimum length of generated completions. When using tools, only non-tool tokens are counted.\n- `completions/max_length`: The maximum length of generated completions. When using tools, only non-tool tokens are counted.\n- `completions/mean_terminated_length`: The average length of generated completions that terminate with EOS. When using tools, only non-tool tokens are counted.\n- `completions/min_terminated_length`: The minimum length of generated completions that terminate with EOS. When using tools, only non-tool tokens are counted.\n- `completions/max_terminated_length`: The maximum length of generated completions that terminate with EOS. When using tools, only non-tool tokens are counted.\n- `completions/clipped_ratio`: The ratio of truncated (clipped) completions.\n- `reward/{reward_func_name}/mean`: The average reward from a specific reward function.\n- `reward/{reward_func_name}/std`: The standard deviation of the reward from a specific reward function.\n- `reward`: The overall average reward after summing rewards across functions (unweighted).\n- `reward_std`: The standard deviation of summed rewards across functions (unweighted), computed over the full batch.\n- `frac_reward_zero_std`: The fraction of samples in the generation batch with a reward std of zero, implying there is little diversity for that prompt (all answers are correct or incorrect).\n- `entropy`: Average entropy of token predictions across generated completions. (If `mask_truncated_completions=True`, masked sequences tokens are excluded.)\n- `kl`: The average KL divergence between the model and the reference model, calculated over generated completions. Logged only if `beta` is nonzero.\n- `clip_ratio/region_mean`: The ratio of token (or sequence, if `importance_sampling_level=\"sequence\"`) probabilities where the GRPO objective is clipped to stay within the trust region: \\\\( \\text{clip}\\left( r_{i,t}(\\theta), 1 - \\epsilon_\\mathrm{low}, 1 + \\epsilon_\\mathrm{high} \\right)\\,, \\quad r_{i,t}(\\theta) = \\frac{\\pi_\\theta(o_{i,t} \\mid q, o_{i,< t})}{\\pi_{\\theta_{\\text{old}}}(o_{i,t} \\mid q, o_{i,< t})} \\\\). A higher value means more tokens are clipped, which constrains how much the policy $\\pi_\\theta$ can change.\n- `clip_ratio/low_mean`: The average ratio of token (or sequence, if `importance_sampling_level=\"sequence\"`) probabilities that were clipped on the lower bound of the trust region: \\\\(r_{i,t}(\\theta) < 1 - \\epsilon_\\mathrm{low}\\\\).\n- `clip_ratio/low_min`: The minimum ratio of token (or sequence, if `importance_sampling_level=\"sequence\"`) probabilities that were clipped on the lower bound of the trust region: \\\\(r_{i,t}(\\theta) < 1 - \\epsilon_\\mathrm{low}\\\\).\n- `clip_ratio/high_mean`: The average ratio of token (or sequence, if `importance_sampling_level=\"sequence\"`) probabilities that were clipped on the upper bound of the trust region: \\\\(r_{i,t}(\\theta) > 1 + \\epsilon_\\mathrm{high}\\\\).\n- `clip_ratio/high_max`: The maximum ratio of token (or sequence, if `importance_sampling_level=\"sequence\"`) probabilities that were clipped on the upper bound of the trust region: \\\\(r_{i,t}(\\theta) > 1 + \\epsilon_\\mathrm{high}\\\\).\n\n## Customization\n\n### Speed up training with vLLM-powered generation\n\nGeneration is often the main bottleneck when training with online methods. To accelerate generation, you can use [vLLM](https://github.com/vllm-project/vllm), a high-throughput, low-latency inference engine for LLMs. To enable it, first install the package with\n\n```shell\npip install trl[vllm]\n```\n\nWe support two ways of using vLLM during training: **server mode** and **colocate mode**.\n\n> [!TIP]\n> By default, Truncated Importance Sampling is activated for vLLM generation to address the generation-training mismatch that occurs when using different frameworks. This can be turned off by setting `vllm_importance_sampling_correction=False`. For more information, see [Truncated Importance Sampling](paper_index#truncated-importance-sampling)\n\n#### Option 1: Colocate mode\n\nIn this mode, vLLM runs inside the trainer process and shares GPU memory with the training model. This avoids launching a separate server and can improve GPU utilization, but may lead to memory contention on the training GPUs. This is the default mode.\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n ...,\n use_vllm=True, # vllm_mode=\"colocate\" by default\n)\n```\n\n#### Option 2: Server mode\n\nIn this mode, vLLM runs in a separate process (and using separate GPUs) and communicates with the trainer via HTTP. This is ideal if you have dedicated GPUs for inference.\n\n1. **Start the vLLM server**:\n\n ```bash\n trl vllm-serve --model \n ```\n\n2. **Enable server mode in your training script**:\n\n ```python\n from trl import GRPOConfig\n\n training_args = GRPOConfig(\n ...,\n use_vllm=True,\n vllm_mode=\"server\",\n )\n ```\n\n> [!WARNING]\n> Make sure that the server is using different GPUs than the trainer, otherwise you may run into NCCL errors. You can specify the GPUs to use with the `CUDA_VISIBLE_DEVICES` environment variable.\n\n> [!TIP]\n> Depending on the model size and the overall GPU memory requirements for training, you may need to adjust the `vllm_gpu_memory_utilization` parameter in [`GRPOConfig`] to avoid underutilization or out-of-memory errors.\n>\n> We provide a [HF Space](https://huggingface.co/spaces/trl-lib/recommend-vllm-memory) to help estimate the recommended GPU memory utilization based on your model configuration and experiment settings. Simply use it as follows to get `vllm_gpu_memory_utilization` recommendation:\n>\n> \n>\n> If the recommended value does not work in your environment, we suggest adding a small buffer (e.g., +0.05 or +0.1) to the recommended value to ensure stability.\n>\n> If you still find you are getting out-of-memory errors set `vllm_enable_sleep_mode` to True and the vllm parameters and cache will be offloaded during the optimization step. For more information, see [Reducing Memory Usage with vLLM Sleep Mode](reducing_memory_usage#vllm-sleep-mode).\n\n> [!TIP]\n> By default, GRPO uses `MASTER_ADDR=localhost` and `MASTER_PORT=12345` for vLLM, but you can override these values by setting the environment variables accordingly.\n\nFor more information, see [Speeding up training with vLLM](speeding_up_training#vllm-for-fast-generation-in-online-methods).\n\n\n#### Dealing with the Training-Inference Mismatch\nWhile vLLM greatly accelerates inference, it also decouples the inference engine from the training engine. In theory these engines are mathematically identical, in practice however they can produce different outputs due to precision effects and hardware specific optimizations. This divergence reflects the different optimization objectives of the two systems. This divergence reflects the distinct optimization goals of the two systems. Inference engines aim to maximize sampling throughput, typically measured in tokens per second, while maintaining acceptable sampling fidelity. Training frameworks instead focus on numerical stability and precision for gradient computation, often using higher precision formats like FP32 for master weights and optimizer states. These differing priorities and constraints introduce an inevitable, albeit subtle, mismatch between training and inference.\n\nThis mismatch leads to a biased gradient update which has been observed to destabilize training ([[1]](https://fengyao.notion.site/off-policy-rl)[[2]](https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda)[[3]](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/#true-on-policy-rl)[[4]](https://huggingface.co/papers/2510.26788)[[5]](https://huggingface.co/papers/2510.18855)). For simplicity, consider the REINFORCE policy gradient:\n\n$$\n\\nabla_\\theta \\mathcal{J}(x,\\theta)\n= \\mathbb{E}_{y \\sim \\pi^\\text{train}(\\cdot \\mid x,\\theta)}\n\\left[ \\nabla_\\theta \\log \\pi^\\text{train}(y \\mid x,\\theta) \\cdot R(x,y) \\right]\n$$\n\nHere \\\\( x \\\\) denotes prompts sampled from some data distribution, and \\\\( \\pi^\\text{train} \\\\) is the policy implemented by the training engine. With vLLM in the loop we obtain a separate inference policy \\\\( \\pi^\\text{inference} \\\\), so the effective policy gradient becomes\n\n$$\n\\nabla_\\theta \\mathcal{J}_{\\text{biased}}(x,\\theta)\n= \\mathbb{E}_{y \\sim \\pi^\\text{inference}(\\cdot \\mid x,\\theta)}\n\\left[ \\nabla_\\theta \\log \\pi^\\text{train}(y \\mid x,\\theta) \\cdot R(x,y) \\right].\n$$\n\nThis turns an otherwise on policy RL problem into an off policy one.\n\nThe standard way to correct for this distribution shift is **importance sampling (IS)**. We provide two IS variants: [Truncated Importance Sampling (TIS)](paper_index#truncated-importance-sampling) and [Masked Importance Sampling (MIS)](paper_index#masked-importance-sampling). Both variants can be applied either at the token level or at the sequence level.Let \\\\( \\rho \\\\) denote the importance weight, for example \\\\( \\rho_t \\\\) per token or \\\\( \\rho_{\\text{seq}} \\\\) per sequence. Under TIS, ratios larger than `vllm_importance_sampling_cap` are clipped,\n\n$$\n\\rho \\leftarrow \\min(\\rho, C).\n$$\n\nUnder MIS, ratios larger than `vllm_importance_sampling_cap` are set to zero, so those samples do not contribute to the gradient. In other words, large ratio samples are downweighted under TIS and discarded under MIS. The configuration flag `vllm_importance_sampling_mode` chooses both the IS variant (masking or truncation) and the granularity (token level or sequence level).\n\nImportance sampling is the principled algorithmic response to the trainingโ€“inference mismatch. However, there are also more direct approaches that attempt to reduce the mismatch between the two engines themselves. Most of these are engineering solutions. For example, [MiniMax M1 uses an FP32 language model head](https://huggingface.co/papers/2506.13585) in the inference engine. Thinking Machines has explored [deterministic inference kernels](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/), although this comes with a significant efficiency cost. vLLM has shown [bitwise consistent policies](https://blog.vllm.ai/2025/11/10/bitwise-consistent-train-inference.html) by building on the batch invariant deterministic kernels from Thinking Machines, but as of November 2025 there remains a substantial throughput penalty relative to standard vLLM inference.\n\n### GRPO at scale: train a 70B+ Model on multiple nodes\n\nWhen training large models like **Qwen2.5-72B**, you need several key optimizations to make the training efficient and scalable across multiple GPUs and nodes. These include:\n\n- **DeepSpeed ZeRO Stage 3**: ZeRO leverages data parallelism to distribute model states (weights, gradients, optimizer states) across multiple GPUs and CPUs, reducing memory and compute requirements on each device. Since large models cannot fit on a single GPU, using ZeRO Stage 3 is required for training such models. For more details, see [DeepSpeed Integration](deepspeed_integration).\n- **Accelerate**: Accelerate is a library that simplifies distributed training across multiple GPUs and nodes. It provides a simple API to launch distributed training and handles the complexities of distributed training, such as data parallelism, gradient accumulation, and distributed data loading. For more details, see [Distributing Training](distributing_training).\n- **vLLM**: See the previous section on how to use vLLM to speed up generation.\n\nBelow is an example SLURM script to train a 70B model with GRPO on multiple nodes. This script trains a model on 4 nodes and uses the 5th node for vLLM-powered generation.\n\n```sh\n#!/bin/bash\n#SBATCH --nodes=5\n#SBATCH --gres=gpu:8\n\n# Get the list of allocated nodes\nNODELIST=($(scontrol show hostnames $SLURM_JOB_NODELIST))\n\n# Assign the first 4 nodes for training and the 5th node for vLLM\nTRAIN_NODES=\"${NODELIST[@]:0:4}\" # Nodes 0, 1, 2, 3 for training\nVLLM_NODE=\"${NODELIST[4]}\" # Node 4 for vLLM\n\n# Run training on the first 4 nodes (Group 1)\nsrun --nodes=4 --ntasks=4 --nodelist=\"${NODELIST[@]:0:4}\" accelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n --num_processes 32 \\\n --num_machines 4 \\\n --main_process_ip ${NODELIST[0]} \\\n --machine_rank $SLURM_PROCID \\\n --rdzv_backend c10d \\\n train_grpo.py \\\n --server_ip $VLLM_NODE &\n\n# Run vLLM server on the 5th node (Group 2)\nsrun --nodes=1 --ntasks=1 --nodelist=\"${NODELIST[4]}\" trl vllm-serve --model Qwen/Qwen2.5-72B --tensor_parallel_size 8 &\n\nwait\n```\n\n```python\nimport argparse\n\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer, GRPOConfig\nfrom trl.rewards import accuracy_reward\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--vllm_server_host\", type=str, default=\"\", help=\"The server IP\")\n args = parser.parse_args()\n\n dataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\n training_args = GRPOConfig(\n per_device_train_batch_size=4,\n use_vllm=True,\n vllm_mode=\"server\",\n vllm_server_host=args.vllm_server_host.replace(\"ip-\", \"\").replace(\"-\", \".\"), # from ip-X-X-X-X to X.X.X.X\n )\n\n trainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-72B\",\n args=training_args,\n reward_funcs=accuracy_reward,\n train_dataset=dataset\n )\n trainer.train()\n\nif __name__==\"__main__\":\n main()\n```\n\n### Using a custom reward function\n\nThe [`GRPOTrainer`] supports using custom reward functions instead of dense reward models. To ensure compatibility, your reward function must satisfy the following requirements:\n\nReward functions can be either synchronous Python callables or asynchronous `async def` coroutines. When you provide multiple asynchronous reward functions, they are awaited concurrently (run in parallel via `asyncio.gather`) so their latency overlaps.\n\n1. **Input arguments**:\n - The function must accept the following as keyword arguments:\n - `prompts` (contains the prompts),\n - `completions` (contains the generated completions),\n - `completion_ids` (contains the tokenized completions),\n - `trainer_state` ([`~transformers.TrainerState`]): The current state of the trainer. This can be used to implement dynamic reward functions, such as curriculum learning, where the reward is adjusted based on the training progress.\n - `log_extra`: a callable `log_extra(column: str, values: list)` to add extra columns to the completions table. See Example 6. In distributed training, it's important that all processes log the same set of keys.\n - `log_metric`: a callable `log_metric(name: str, value: float)` to log scalar metrics as plots alongside `kl`, `entropy`, etc. See Example 6. In distributed training, it's important that all processes log the same set of keys.\n - All column names (but `prompt`) that the dataset may have. For example, if the dataset contains a column named `ground_truth`, the function will be called with `ground_truth` as a keyword argument.\n\n The easiest way to comply with this requirement is to use `**kwargs` in the function signature.\n - Depending on the dataset format, the input will vary:\n - For [standard format](dataset_formats#standard), `prompts` and `completions` will be lists of strings.\n - For [conversational format](dataset_formats#conversational), `prompts` and `completions` will be lists of message dictionaries.\n\n2. **Return value**: The function must return a list of floats. Each float represents the reward corresponding to a single completion.\n\n#### Example 1: Reward longer completions\n\nBelow is an example of a reward function for a standard format that rewards longer completions:\n\n```python\ndef reward_func(completion_ids, **kwargs):\n \"\"\"Reward function that assigns higher scores to longer completions (in terms of token count).\"\"\"\n return [float(len(ids)) for ids in completion_ids]\n```\n\nYou can test it as follows:\n\n```python\n>>> prompts = [\"The sky is\", \"The sun is\"] # not used in the reward function, but the trainer will pass it\n>>> completions = [\" blue.\", \" in the sky.\"] # not used in the reward function, but the trainer will pass it\n>>> completion_ids = [[6303, 13], [304, 279, 12884, 13]]\n>>> reward_func(prompts=prompts, completions=completions, completion_ids=completion_ids)\n[2.0, 4.0]\n```\n\n#### Example 1.1: Reward longer completions (based on the number of characters)\n\nSame as the previous example, but this time the reward function is based on the number of characters instead of tokens.\n\n```python\ndef reward_func(completions, **kwargs):\n \"\"\"Reward function that assigns higher scores to longer completions (in terms of character count).\"\"\"\n return [float(len(completion)) for completion in completions]\n```\n\nYou can test it as follows:\n\n```python\n>>> prompts = [\"The sky is\", \"The sun is\"]\n>>> completions = [\" blue.\", \" in the sky.\"]\n>>> completion_ids = [[6303, 13], [304, 279, 12884, 13]] # not used in the reward function, but the trainer will pass it\n>>> reward_func(prompts=prompts, completions=completions, completion_ids=completion_ids)\n[6.0, 12.0]\n```\n\n#### Example 2: Reward completions with a specific format\n\nBelow is an example of a reward function that checks if the completion has a specific format. This example is inspired by the _format reward_ function used in the paper [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://huggingface.co/papers/2501.12948).\nIt is designed for a conversational format, where prompts and completions consist of structured messages.\n\n```python\nimport re\n\ndef format_reward_func(completions, **kwargs):\n \"\"\"Reward function that checks if the completion has a specific format.\"\"\"\n pattern = r\"^.*?.*?$\"\n completion_contents = [completion[0][\"content\"] for completion in completions]\n matches = [re.match(pattern, content) for content in completion_contents]\n return [1.0 if match else 0.0 for match in matches]\n```\n\nYou can test this function as follows:\n\n```python\n>>> prompts = [\n... [{\"role\": \"assistant\", \"content\": \"What is the result of (1 + 2) * 4?\"}],\n... [{\"role\": \"assistant\", \"content\": \"What is the result of (3 + 1) * 2?\"}],\n... ]\n>>> completions = [\n... [{\"role\": \"assistant\", \"content\": \"The sum of 1 and 2 is 3, which we multiply by 4 to get 12.(1 + 2) * 4 = 12\"}],\n... [{\"role\": \"assistant\", \"content\": \"The sum of 3 and 1 is 4, which we multiply by 2 to get 8. So (3 + 1) * 2 = 8.\"}],\n... ]\n>>> format_reward_func(prompts=prompts, completions=completions)\n[1.0, 0.0]\n```\n\n#### Example 3: Reward completions based on a reference\n\nBelow is an example of a reward function that checks if the completion is correct. This example is inspired by the _accuracy reward_ function used in the paper [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://huggingface.co/papers/2501.12948).\nThis example is designed for [standard format](dataset_formats#standard), where the dataset contains a column named `ground_truth`.\n\n```python\nimport re\n\ndef reward_func(completions, ground_truth, **kwargs):\n # Regular expression to capture content inside \\boxed{}\n matches = [re.search(r\"\\\\boxed\\{(.*?)\\}\", completion) for completion in completions]\n contents = [match.group(1) if match else \"\" for match in matches]\n # Reward 1 if the content is the same as the ground truth, 0 otherwise\n return [1.0 if c == gt else 0.0 for c, gt in zip(contents, ground_truth)]\n```\n\nYou can test this function as follows:\n\n```python\n>>> prompts = [\"Problem: Solve the equation $2x + 3 = 7$. Solution:\", \"Problem: Solve the equation $3x - 5 = 10$.\"]\n>>> completions = [r\" The solution is \\boxed{2}.\", r\" The solution is \\boxed{6}.\"]\n>>> ground_truth = [\"2\", \"5\"]\n>>> reward_func(prompts=prompts, completions=completions, ground_truth=ground_truth)\n[1.0, 0.0]\n```\n\n#### Example 4: Multi-task reward functions\n\nBelow is an example of using multiple reward functions in the [`GRPOTrainer`]. In this example, we define two task-specific reward functions: `math_reward_func` and `coding_reward_func`. The `math_reward_func` rewards math problems based on their correctness, while the `coding_reward_func` rewards coding problems based on whether the solution works.\n\n```python\nfrom datasets import Dataset\nfrom trl import GRPOTrainer\n\n# Define a dataset that contains both math and coding problems\ndataset = Dataset.from_list(\n [\n {\"prompt\": \"What is 2+2?\", \"task\": \"math\"},\n {\"prompt\": \"Write a function that returns the sum of two numbers.\", \"task\": \"code\"},\n {\"prompt\": \"What is 3*4?\", \"task\": \"math\"},\n {\"prompt\": \"Write a function that returns the product of two numbers.\", \"task\": \"code\"},\n ]\n)\n\n# Math-specific reward function\ndef math_reward_func(prompts, completions, task, **kwargs):\n rewards = []\n for prompt, completion, t in zip(prompts, completions, task):\n if t == \"math\":\n # Calculate math-specific reward\n correct = check_math_solution(prompt, completion)\n reward = 1.0 if correct else -1.0\n rewards.append(reward)\n else:\n # Return None for non-math tasks\n rewards.append(None)\n return rewards\n\n# Coding-specific reward function\ndef coding_reward_func(prompts, completions, task, **kwargs):\n rewards = []\n for prompt, completion, t in zip(prompts, completions, task):\n if t == \"coding\":\n # Calculate coding-specific reward\n works = test_code_solution(prompt, completion)\n reward = 1.0 if works else -1.0\n rewards.append(reward)\n else:\n # Return None for non-coding tasks\n rewards.append(None)\n return rewards\n\n# Use both task-specific reward functions\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen2-0.5B-Instruct\",\n reward_funcs=[math_reward_func, coding_reward_func],\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\nIn this example, the `math_reward_func` and `coding_reward_func` are designed to work with a mixed dataset that contains both math and coding problems. The `task` column in the dataset is used to determine which reward function to apply to each problem. If there is no relevant reward function for a sample in the dataset, the reward function will return `None`, and the [`GRPOTrainer`] will continue with the valid functions and tasks. This allows the [`GRPOTrainer`] to handle multiple reward functions with different applicability.\n\nNote that the [`GRPOTrainer`] will ignore the `None` rewards returned by the reward functions and only consider the rewards returned by the relevant functions. This ensures that the model is trained on the relevant tasks and ignores the tasks for which there is no relevant reward function.\n\n#### Example 5: Asynchronous reward functions\n\nCustom reward functions can also be defined as `async def` coroutines. This is useful if your reward depends on slow I/O (for example, calling a remote service). When you pass multiple async reward functions, [`GRPOTrainer`] executes them concurrently so their latency overlaps.\n\nBelow is a minimal example of an async reward function that simulates an I/O-bound operation:\n\n```python\nimport asyncio\n\nasync def async_reward_func(prompts, completions, **kwargs):\n # Simulate an I/O-bound call (e.g., HTTP request, database lookup)\n await asyncio.sleep(0.01)\n # Simple toy reward: 1.0 if the completion is non-empty, else 0.0\n return [1.0 if completion else 0.0 for completion in completions]\n```\n\n#### Example 6: Logging extra columns and metrics\n\nBelow is an example of a reward function that logs extra columns to the completions table and scalar metrics as plots.\n\n```python\nimport re\n\ndef reward_func(completions, ground_truth, log_extra=None, log_metric=None, **kwargs):\n extracted = [re.search(r\"\\\\boxed\\{(.*?)\\}\", c) for c in completions]\n extracted = [m.group(1) if m else None for m in extracted]\n rewards = [1.0 if e == gt else 0.0 for e, gt in zip(extracted, ground_truth)]\n\n if log_extra:\n log_extra(\"golden_answer\", list(ground_truth))\n log_extra(\"extracted_answer\", [e or \"[none]\" for e in extracted])\n\n if log_metric:\n log_metric(\"accuracy\", sum(rewards) / len(rewards))\n\n return rewards\n```\n\n#### Passing the reward function to the trainer\n\nTo use your custom reward function, pass it to the [`GRPOTrainer`] as follows:\n\n```python\nfrom trl import GRPOTrainer\n\ntrainer = GRPOTrainer(\n reward_funcs=reward_func,\n ...,\n)\n```\n\nYou can pass several reward functions as a list; this list may include both synchronous and asynchronous functions:\n\n```python\nfrom trl import GRPOTrainer\n\ntrainer = GRPOTrainer(\n reward_funcs=[reward_func, async_reward_func1, async_reward_func2],\n ...,\n)\n```\n\nand the reward will be computed as the sum of the rewards from each function, or the weighted sum if `reward_weights` is provided in the config.\n\nNote that [`GRPOTrainer`] supports multiple reward functions of different types. See the parameters documentation for more details.\n\n### Rapid Experimentation for GRPO\n\nRapidFire AI is an open-source experimentation engine that sits on top of TRL and lets you launch multiple GRPO configurations at once, even on a single GPU. Instead of trying configurations sequentially, RapidFire lets you **see all their learning curves earlier, stop underperforming runs, and clone promising ones with new settings in flight** without restarting. For more information, see [RapidFire AI Integration](rapidfire_integration).\n\n## Agent Training\n\nGRPO supports **agent training** through the `tools` argument in [`GRPOTrainer`].\nThis parameter expects a list of Python functions (sync or async) that define the tools available to the agent:\n\n```python\nfrom trl import GRPOTrainer\n\ntrainer = GRPOTrainer(\n tools=[tool1, tool2],\n ...,\n)\n```\n\nEach tool must be a standard Python function with **type-hinted arguments and return types**, along with a **Google-style docstring** describing its purpose, arguments, and return value.\nFor more details, see the [Passing tools guide](https://huggingface.co/docs/transformers/en/chat_extras#passing-tools).\n\nExample:\n\n```python\nfrom trl import GRPOTrainer\n\ndef multiply(a: int, b: int) -> int:\n \"\"\"\n Multiplies two integers.\n\n Args:\n a: The first integer.\n b: The second integer.\n\n Returns:\n The product of the two integers.\n \"\"\"\n return a * b\n\nasync def async_add(a: int, b: int) -> int:\n \"\"\"\n Asynchronously adds two integers.\n\n Args:\n a: The first integer.\n b: The second integer.\n\n Returns:\n The sum of the two integers.\n \"\"\"\n return a + b\n\ntrainer = GRPOTrainer(\n tools=[multiply, async_add],\n ...,\n)\n```\n\nYou can also provide tools through `environment_factory`. In this mode, [`GRPOTrainer`] creates one environment instance per rollout and exposes the environment's public methods as tools.\n\n> [!IMPORTANT]\n> `environment_factory` requires `transformers>=5.2.0`.\n\nThe following is a minimal example of using `environment_factory` to define a simple environment with an `increment` method, which is exposed as a tool to the agent:\n\n```python\nfrom datasets import Dataset\nfrom trl import GRPOConfig, GRPOTrainer\n\ninstructions = [f\"Increment the counter by {i}.\" for i in range(1, 7)]\ndataset = Dataset.from_dict({\"prompt\": [[{\"role\": \"user\", \"content\": instruction}] for instruction in instructions]})\n\ndef reward_func(environments, **kwargs): # dummy reward: the reward is the current value of the counter\n return [environment.counter for environment in environments]\n\nclass IncrementEnv:\n def reset(self, **kwargs) -> str | None: # required; receives sampled row fields as kwargs (e.g., `prompt`)\n self.counter = 0\n return \"Counter reset to 0.\\n\"\n\n def increment(self, step: int) -> int: # the other public methods of the environment are exposed as tools\n \"\"\"\n Increment the internal counter.\n\n Args:\n step: Value to add to the counter.\n\n Returns:\n The updated counter value.\n \"\"\"\n self.counter += step\n return self.counter\n\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n args=GRPOConfig(chat_template_kwargs={\"enable_thinking\": False}),\n train_dataset=dataset,\n reward_funcs=reward_func,\n environment_factory=IncrementEnv,\n)\ntrainer.train()\n```\n\n`reset` can return either `None` or a string. In GRPO, when it returns a string, that string is appended to the last user message before generation.\n\n### Supported Models\n\nTested with:\n\n- [**Qwen3**](https://huggingface.co/collections/Qwen/qwen3) โ€” e.g., `Qwen/Qwen3-0.6B`\n- [**Qwen3.5**](https://huggingface.co/collections/Qwen/qwen35) โ€” e.g., `Qwen/Qwen3.5-2B`\n\n> [!TIP]\n> Compatibility with all LLMs is not guaranteed. If you believe a model should be supported, feel free to open an issue on GitHub โ€” or better yet, submit a pull request with the required changes.\n\n### Quick Start\n\nUse [grpo\\_agent.py](https://github.com/huggingface/trl/blob/main/examples/scripts/grpo_agent.py) to fine-tune a LLM for agentic workflows.\n\n```bash\naccelerate launch \\\n --config_file=examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/grpo_agent.py \\\n --model_name_or_path Qwen/Qwen3-0.6B\n ...\n```\n\n## Vision-Language Model (VLM) Training\n\nGRPO supports training Vision-Language Models (VLMs) on multimodal datasets containing both text and images.\n\n### Supported Models\n\nTested with:\n\n- **Gemma3** โ€” e.g., `google/gemma-3-4b-it`\n- **LLaVA-NeXT** โ€” e.g., `llava-hf/llava-v1.6-mistral-7b-hf`\n- **Qwen2-VL** โ€” e.g., `Qwen/Qwen2-VL-2B-Instruct`\n- **Qwen2.5-VL** โ€” e.g., `Qwen/Qwen2.5-VL-3B-Instruct`\n- **SmolVLM2** โ€” e.g., `HuggingFaceTB/SmolVLM2-2.2B-Instruct`\n \n> [!TIP]\n> Compatibility with all VLMs is not guaranteed. If you believe a model should be supported, feel free to open an issue on GitHub โ€” or better yet, submit a pull request with the required changes.\n\n### Quick Start\n\nUse [grpo\\_vlm.py](https://github.com/huggingface/trl/blob/main/examples/scripts/grpo_vlm.py) to fine-tune a VLM. Example command for training on [`lmms-lab/multimodal-open-r1-8k-verified`](https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified):\n\n```bash\naccelerate launch \\\n --config_file=examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/grpo_vlm.py \\\n --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --output_dir grpo-Qwen2.5-VL-3B-Instruct \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_completion_length 1024 \\\n --use_vllm \\\n --vllm_mode colocate \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --log_completions\n```\n\n### Configuration Tips\n\n- Use LoRA on vision-language projection layers\n- Enable 4-bit quantization to reduce memory usage\n- VLMs are memory-intensive โ€” start with smaller batch sizes\n- Most models are compatible with vLLM (`server` and `colocate` modes)\n\n### Dataset Format\n\nEach training sample should include:\n\n- `prompt`: Text formatted via the processor's chat template\n- `image`/`images`: PIL Image or list of PIL Images\n\nThe trainer automatically handles image-to-tensor conversion via the modelโ€™s image processor.\n\n## GRPOTrainer\n\n[[autodoc]] GRPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## GRPOConfig\n\n[[autodoc]] GRPOConfig\n" }, { "path": "docs/source/grpo_with_replay_buffer.md", "content": "# GRPO With Replay Buffer\n\nThis experimental trainer, trains a model with GRPO but replaces groups (and corresponding completions) that have 0 standard deviation with groups with high rewards and standard deviation that've been used to train a model in prior batches.\n\n## Usage\n\n```python\nimport torch\nfrom trl.experimental.grpo_with_replay_buffer import GRPOWithReplayBufferConfig, GRPOWithReplayBufferTrainer\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n# Guarantee that some rewards have 0 std\ndef custom_reward_func(completions, **kwargs):\n if torch.rand(1).item() < 0.25:\n return [0] * len(completions) # simulate some None rewards\n else:\n return torch.rand(len(completions)).tolist()\n\ntraining_args = GRPOWithReplayBufferConfig(\n output_dir=\"./tmp\",\n learning_rate=1e-4,\n per_device_train_batch_size=4,\n num_generations=4,\n max_completion_length=8,\n replay_buffer_size=8,\n report_to=\"none\",\n)\n\ntrainer = GRPOWithReplayBufferTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[custom_reward_func],\n args=training_args,\n train_dataset=dataset,\n)\n\nprevious_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\ntrainer.train()\n```\n\n## GRPOWithReplayBufferTrainer\n\n[[autodoc]] experimental.grpo_with_replay_buffer.GRPOWithReplayBufferTrainer\n - train\n - save_model\n - push_to_hub\n\n## GRPOWithReplayBufferConfig\n\n[[autodoc]] experimental.grpo_with_replay_buffer.GRPOWithReplayBufferConfig\n\n## ReplayBuffer\n\n[[autodoc]] experimental.grpo_with_replay_buffer.ReplayBuffer\n" }, { "path": "docs/source/gspo_token.md", "content": "# GSPO-token\n\nIn the paper [Group Sequence Policy Optimization](https://huggingface.co/papers/2507.18071), the authors propose a token-level objective variant to GSPO, called GSPO-token. To use GSPO-token, you can use the `GRPOTrainer` class in `trl.experimental.gspo_token`.\n\n## Usage\n\n```python\nfrom trl.experimental.gspo_token import GRPOTrainer\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n importance_sampling_level=\"sequence_token\",\n ...\n)\n```\n\n> [!WARNING]\n> To leverage GSPO-token, the user will need to provide the per-token advantage \\\\( \\hat{A_{i,t}} \\\\) for each token \\\\( t \\\\) in the sequence \\\\( i \\\\) (i.e., make \\\\( \\hat{A_{i,t}} \\\\) varies with \\\\( t \\\\)โ€”which isn't the case here, \\\\( \\hat{A_{i,t}}=\\hat{A_{i}} \\\\)). Otherwise, GSPO-Token gradient is just equivalent to the original GSPO implementation.\n\n## GRPOTrainer\n\n[[autodoc]] experimental.gspo_token.GRPOTrainer\n - train\n - save_model\n - push_to_hub\n" }, { "path": "docs/source/index.md", "content": "
\n\n \n \n\n
\n\n# TRL - Transformers Reinforcement Learning\n\nTRL is a full stack library where we provide a set of tools to train transformer language models with methods like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), Reward Modeling, and more.\nThe library is integrated with ๐Ÿค— [transformers](https://github.com/huggingface/transformers).\n\n## ๐ŸŽ‰ What's New\n\n**OpenEnv Integration:** TRL now supports **[OpenEnv](https://huggingface.co/blog/openenv)**, the open-source framework from Meta for defining, deploying, and interacting with environments in reinforcement learning and agentic workflows.\n\nExplore how to seamlessly integrate TRL with OpenEnv in our [dedicated documentation](openenv).\n\n## Taxonomy\n\nBelow is the current list of TRL trainers, organized by method type (โšก๏ธ = vLLM support; ๐Ÿงช = experimental).\n\n
\n
\n\n### Online methods\n\n- [`GRPOTrainer`](grpo_trainer) โšก๏ธ\n- [`RLOOTrainer`](rloo_trainer) โšก๏ธ\n- [`OnlineDPOTrainer`](online_dpo_trainer) ๐Ÿงช โšก๏ธ\n- [`NashMDTrainer`](nash_md_trainer) ๐Ÿงช โšก๏ธ\n- [`PPOTrainer`](ppo_trainer) ๐Ÿงช\n- [`XPOTrainer`](xpo_trainer) ๐Ÿงช โšก๏ธ\n\n### Reward modeling\n\n- [`RewardTrainer`](reward_trainer)\n- [`PRMTrainer`](prm_trainer) ๐Ÿงช\n\n
\n
\n\n### Offline methods\n\n- [`SFTTrainer`](sft_trainer)\n- [`DPOTrainer`](dpo_trainer)\n- [`BCOTrainer`](bco_trainer) ๐Ÿงช\n- [`CPOTrainer`](cpo_trainer) ๐Ÿงช\n- [`KTOTrainer`](kto_trainer) ๐Ÿงช\n- [`ORPOTrainer`](orpo_trainer) ๐Ÿงช\n\n### Knowledge distillation\n\n- [`GKDTrainer`](gkd_trainer) ๐Ÿงช\n- [`MiniLLMTrainer`](minillm_trainer) ๐Ÿงช\n\n
\n
\n\nYou can also explore TRL-related models, datasets, and demos in the [TRL Hugging Face organization](https://huggingface.co/trl-lib).\n\n## Learn\n\nLearn post-training with TRL and other libraries in ๐Ÿค— [smol course](https://github.com/huggingface/smol-course).\n\n## Contents\n\nThe documentation is organized into the following sections:\n\n- **Getting Started**: installation and quickstart guide.\n- **Conceptual Guides**: dataset formats, training FAQ, and understanding logs.\n- **How-to Guides**: reducing memory usage, speeding up training, distributing training, etc.\n- **Integrations**: DeepSpeed, Liger Kernel, PEFT, etc.\n- **Examples**: example overview, community tutorials, etc.\n- **API**: trainers, utils, etc.\n\n## Blog posts\n\n
\n
\n \n \"thumbnail\"\n

Published October 23, 2025

\n

Building the Open Agent Ecosystem Together: Introducing OpenEnv

\n \n \n \"thumbnail\"\n

Published on August 7, 2025

\n

Vision Language Model Alignment in TRL โšก๏ธ

\n \n \n \"thumbnail\"\n

Published on June 3, 2025

\n

NO GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL

\n \n \n \"thumbnail\"\n

Published on May 25, 2025

\n

๐Ÿฏ Liger GRPO meets TRL

\n \n \n \"thumbnail\"\n

Published on January 28, 2025

\n

Open-R1: a fully open reproduction of DeepSeek-R1

\n \n \n \"thumbnail\"\n

Published on July 10, 2024

\n

Preference Optimization for Vision Language Models with TRL

\n \n \n \"thumbnail\"\n

Published on June 12, 2024

\n

Putting RL back in RLHF

\n \n \n \"thumbnail\"\n

Published on September 29, 2023

\n

Finetune Stable Diffusion Models with DDPO via TRL

\n \n \n \"thumbnail\"\n

Published on August 8, 2023

\n

Fine-tune Llama 2 with DPO

\n \n \n \"thumbnail\"\n

Published on April 5, 2023

\n

StackLLaMA: A hands-on guide to train LLaMA with RLHF

\n \n \n \"thumbnail\"\n

Published on March 9, 2023

\n

Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU

\n \n \n \"thumbnail\"\n

Published on December 9, 2022

\n

Illustrating Reinforcement Learning from Human Feedback

\n \n
\n
\n\n## Talks\n\n
\n
\n \n \"thumbnail\"\n

Talk given on October 30, 2025

\n

Fine tuning with TRL

\n \n
\n
\n" }, { "path": "docs/source/installation.md", "content": "# Installation\n\nYou can install TRL either from PyPI or from source:\n\n## PyPI\n\nInstall the library with pip or [uv](https://docs.astral.sh/uv/):\n\n\n\n\nuv is a fast Rust-based Python package and project manager. Refer to [Installation](https://docs.astral.sh/uv/getting-started/installation/) for installation instructions.\n\n```bash\nuv pip install trl\n```\n\n\n\n\n```bash\npip install trl\n```\n\n\n\n\n## Source\n\nYou can also install the latest version from source. First clone the repo and then run the installation with `pip`:\n\n```bash\ngit clone https://github.com/huggingface/trl.git\ncd trl/\npip install -e .\n```\n\nIf you want the development install you can replace the pip install with the following:\n\n```bash\npip install -e \".[dev]\"\n```\n" }, { "path": "docs/source/jobs_training.md", "content": "# Training with Jobs\n\n[![model badge](https://img.shields.io/badge/All_models-HF_Jobs-blue)](https://huggingface.co/models?other=hf_jobs,trl)\n\n[Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs) lets you run training scripts on fully managed infrastructureโ€”no need to manage GPUs or local environment setup.\n\nIn this guide, you'll learn how to:\n\n* Use [TRL Jobs](https://github.com/huggingface/trl-jobs) to easily run pre-optimized TRL training\n* Run any TRL training script with uv scripts\n\nFor general details about Hugging Face Jobs (hardware selection, job monitoring, etc.), see the [Jobs documentation](https://huggingface.co/docs/huggingface_hub/guides/jobs).\n\n## Requirements\n\n* A [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan\n* Logged in to the Hugging Face Hub (`hf auth login`)\n\n## Using TRL Jobs\n\n[TRL Jobs](https://github.com/huggingface/trl-jobs) is a high-level wrapper around Hugging Face Jobs and TRL that streamlines training. It provides optimized default configurations so you can start quickly without manually tuning parameters.\n\nExample:\n\n```bash\npip install trl-jobs\ntrl-jobs sft --model_name Qwen/Qwen3-0.6B --dataset_name trl-lib/Capybara\n```\n\nTRL Jobs supports everything covered in this guide, with additional optimizations to simplify workflows.\n\n## Using uv Scripts\n\nFor more control, you can run Hugging Face Jobs directly with your own scripts, using [uv scripts](https://docs.astral.sh/uv/guides/scripts/).\n\nCreate a Python script (e.g., `train.py`) containing your training code:\n\n```python\nfrom datasets import load_dataset\nfrom trl import SFTTrainer\n\ndataset = load_dataset(\"trl-lib/Capybara\", split=\"train\")\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen2.5-0.5B\",\n train_dataset=dataset,\n)\ntrainer.train()\ntrainer.push_to_hub(\"Qwen2.5-0.5B-SFT\")\n```\n\nLaunch the job using either the [`hf jobs` CLI](https://huggingface.co/docs/huggingface_hub/guides/cli#hf-jobs) or the Python API:\n\n\n\n\n```bash\nhf jobs uv run \\\n --flavor a100-large \\\n --with trl \\\n --secrets HF_TOKEN \\\n train.py\n```\n\n\n\n\n```python\nfrom huggingface_hub import run_uv_job\n\nrun_uv_job(\n \"train.py\",\n dependencies=[\"trl\"],\n flavor=\"a100-large\",\n secrets={\"HF_TOKEN\": \"hf_...\"},\n)\n```\n\n\n\n\nTo run successfully, the script needs:\n\n* **TRL installed**: Use the `--with trl` flag or the `dependencies` argument. uv installs these dependencies automatically before running the script.\n* **An authentication token**: Required to push the trained model (or perform other authenticated operations). Provide it with the `--secrets HF_TOKEN` flag or the `secrets` argument.\n\n> [!WARNING]\n> When training with Jobs, be sure to:\n>\n> * **Set a sufficient timeout**. Jobs time out after 30 minutes by default. If your job exceeds the timeout, it will fail and all progress will be lost. See [Setting a custom timeout](https://huggingface.co/docs/huggingface_hub/guides/jobs#setting-a-custom-timeout).\n> * **Push the model to the Hub**. The Jobs environment is ephemeralโ€”files are deleted when the job ends. If you donโ€™t push the model, it will be lost.\n\nYou can also run a script directly from a URL:\n\n\n\n\n```bash\nhf jobs uv run \\\n --flavor a100-large \\\n --with trl \\\n --secrets HF_TOKEN \\\n \"https://gist.githubusercontent.com/qgallouedec/eb6a7d20bd7d56f9c440c3c8c56d2307/raw/69fd78a179e19af115e4a54a1cdedd2a6c237f2f/train.py\"\n```\n\n\n\n\n```python\nfrom huggingface_hub import run_uv_job\n\nrun_uv_job(\n \"https://gist.githubusercontent.com/qgallouedec/eb6a7d20bd7d56f9c440c3c8c56d2307/raw/69fd78a179e19af115e4a54a1cdedd2a6c237f2f/train.py\",\n flavor=\"a100-large\",\n dependencies=[\"trl\"],\n secrets={\"HF_TOKEN\": \"hf_...\"},\n)\n```\n\n\n\n\nTo make a script self-contained, declare dependencies at the top:\n\n```python\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# ]\n# ///\n\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom trl import SFTTrainer\n\ndataset = load_dataset(\"trl-lib/Capybara\", split=\"train\")\n\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen2.5-0.5B\",\n train_dataset=dataset,\n peft_config=LoraConfig(),\n)\ntrainer.train()\ntrainer.push_to_hub(\"Qwen2.5-0.5B-SFT\")\n```\n\nYou can then run the script without specifying dependencies:\n\n\n\n\n```bash\nhf jobs uv run \\\n --flavor a100-large \\\n --secrets HF_TOKEN \\\n train.py\n```\n\n\n\n\n```python\nfrom huggingface_hub import run_uv_job\n\nrun_uv_job(\n \"train.py\",\n flavor=\"a100-large\",\n secrets={\"HF_TOKEN\": \"hf_...\"},\n)\n```\n\n\n\n\nTRL example scripts are fully uv-compatible, so you can run a complete training workflow directly on Jobs. You can customize training with standard script arguments plus hardware and secrets:\n\n\n\n\n```bash\nhf jobs uv run \\\n --flavor a100-large \\\n --secrets HF_TOKEN \\\n https://raw.githubusercontent.com/huggingface/trl/refs/heads/main/examples/scripts/prm.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/prm800k \\\n --output_dir Qwen2-0.5B-Reward \\\n --push_to_hub\n```\n\n\n\n\n```python\nfrom huggingface_hub import run_uv_job\nrun_uv_job(\n \"https://raw.githubusercontent.com/huggingface/trl/refs/heads/main/examples/scripts/prm.py\",\n flavor=\"a100-large\",\n secrets={\"HF_TOKEN\": \"hf_...\"},\n script_args=[\n \"--model_name_or_path\", \"Qwen/Qwen2-0.5B-Instruct\",\n \"--dataset_name\", \"trl-lib/prm800k\",\n \"--output_dir\", \"Qwen2-0.5B-Reward\",\n \"--push_to_hub\"\n ]\n)\n```\n\n\n\nSee the full list of examples in [Maintained examples](example_overview#maintained-examples).\n\n### Docker Images\n\nAn up-to-date Docker image with all TRL dependencies is available at [huggingface/trl](https://hub.docker.com/r/huggingface/trl) and can be used directly with Hugging Face Jobs:\n\n\n\n\n```bash\nhf jobs uv run \\\n --flavor a100-large \\\n --secrets HF_TOKEN \\\n --image huggingface/trl \\\n train.py\n```\n\n\n\n\n```python\nfrom huggingface_hub import run_uv_job\n\nrun_uv_job(\n \"train.py\",\n flavor=\"a100-large\",\n secrets={\"HF_TOKEN\": \"hf_...\"},\n image=\"huggingface/trl\",\n)\n```\n\n\n\n\nJobs runs on a Docker image from Hugging Face Spaces or Docker Hub, so you can also specify any custom image:\n\n\n\n\n```bash\nhf jobs uv run \\\n --flavor a100-large \\\n --secrets HF_TOKEN \\\n --image \\\n --secrets HF_TOKEN \\\n train.py\n```\n\n\n\n\n```python\nfrom huggingface_hub import run_uv_job\n\nrun_uv_job(\n \"train.py\",\n flavor=\"a100-large\",\n secrets={\"HF_TOKEN\": \"hf_...\"},\n image=\"\",\n)\n```\n\n\n\n" }, { "path": "docs/source/judges.md", "content": "# Judges\n\n> [!WARNING]\n> TRL Judges is an experimental API which is subject to change at any time. As of TRL v1.0, judges have been moved to the `trl.experimental.judges` module.\n\nTRL provides judges to easily compare two completions.\n\nMake sure to have installed the required dependencies by running:\n\n```bash\npip install trl[judges]\n```\n\n## Using the provided judges\n\nTRL provides several judges out of the box. For example, you can use the [`experimental.judges.HfPairwiseJudge`] to compare two completions using a pre-trained model from the Hugging Face model hub:\n\n```python\nfrom trl.experimental.judges import HfPairwiseJudge\n\njudge = HfPairwiseJudge()\njudge.judge(\n prompts=[\"What is the capital of France?\", \"What is the biggest planet in the solar system?\"],\n completions=[[\"Paris\", \"Lyon\"], [\"Saturn\", \"Jupiter\"]],\n) # Outputs: [0, 1]\n```\n\n## Define your own judge\n\nTo define your own judge, we provide several base classes that you can subclass. For rank-based judges, you need to subclass [`experimental.judges.BaseRankJudge`] and implement the [`experimental.judges.BaseRankJudge.judge`] method. For pairwise judges, you need to subclass [`experimental.judges.BasePairJudge`] and implement the [`experimental.judges.BasePairJudge.judge`] method. If you want to define a judge that doesn't fit into these categories, you need to subclass [`experimental.judges.BaseJudge`] and implement the [`experimental.judges.BaseJudge.judge`] method.\n\nAs an example, let's define a pairwise judge that prefers shorter completions:\n\n```python\nfrom trl.experimental.judges import BasePairwiseJudge\n\nclass PrefersShorterJudge(BasePairwiseJudge):\n def judge(self, prompts, completions, shuffle_order=False):\n return [0 if len(completion[0]) > len(completion[1]) else 1 for completion in completions]\n```\n\nYou can then use this judge as follows:\n\n```python\njudge = PrefersShorterJudge()\njudge.judge(\n prompts=[\"What is the capital of France?\", \"What is the biggest planet in the solar system?\"],\n completions=[[\"Paris\", \"The capital of France is Paris.\"], [\"Jupiter is the biggest planet in the solar system.\", \"Jupiter\"]],\n) # Outputs: [0, 1]\n```\n\n## Provided judges\n\n### PairRMJudge\n\n[[autodoc]] experimental.judges.PairRMJudge\n\n### HfPairwiseJudge\n\n[[autodoc]] experimental.judges.HfPairwiseJudge\n\n### OpenAIPairwiseJudge\n\n[[autodoc]] experimental.judges.OpenAIPairwiseJudge\n\n### AllTrueJudge\n\n[[autodoc]] experimental.judges.AllTrueJudge\n\n## Base classes\n\n### BaseJudge\n\n[[autodoc]] experimental.judges.BaseJudge\n\n### BaseBinaryJudge\n\n[[autodoc]] experimental.judges.BaseBinaryJudge\n\n### BaseRankJudge\n\n[[autodoc]] experimental.judges.BaseRankJudge\n\n### BasePairwiseJudge\n\n[[autodoc]] experimental.judges.BasePairwiseJudge\n" }, { "path": "docs/source/kernels_hub.md", "content": "# Kernels Hub Integration and Usage\n\n\"kernel-builder\n\nThe [`kernels`](https://huggingface.co/blog/hello-hf-kernels#get-started-and-next-steps) library allows optimized compute kernels to be loaded directly from the Hub. \nYou can find `kernels` in [dedicated orgs](https://huggingface.co/kernels-community) or by searching for the [`kernel` tag](https://huggingface.co/models?other=kernel) within the Hub. \n\nKernels are **optimized code pieces** that help in model development, training, and inference. Here, weโ€™ll focus on their **integration with TRL**, but check out the above resources to learn more about them.\n\n## Installation\n\nTo use kernels with TRL, you'd need to install the library in your Python environment:\n\n```bash\npip install kernels\n```\n\n## Using Kernels from the Hub in TRL\n\nKernels can directly replace attention implementations, removing the need to manually compile attention backends like Flash Attention and boosting training speed just by pulling the respective attention kernel from the Hub.\n\nYou can specify a kernel when loading a model:\n\n\n```python\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"your-model-name\",\n attn_implementation=\"kernels-community/flash-attn2\" # other options: kernels-community/vllm-flash-attn3, kernels-community/paged-attention\n)\n```\n\nOr when running a TRL training script:\n\n```bash\npython sft.py ... --attn_implementation kernels-community/flash-attn2\n```\n\nOr using the TRL CLI:\n\n```bash\ntrl sft ... --attn_implementation kernels-community/flash-attn2\n```\n\n> [!TIP]\n> Now you can leverage faster attention backends with a pre-optimized kernel for your hardware configuration from the Hub, speeding up both development and training.\n\n## Comparing Attention Implementations\n\nWe evaluated various attention implementations available in transformers, along with different kernel backends, using **TRL** and **SFT**. \nThe experiments were run on a single **H100 GPU** with **CUDA 12.9**, leveraging **Qwen3-8B** with a **batch size of 8**, **gradient accumulation of 1**, and **bfloat16** precision. \nKeep in mind that the results shown here are specific to this setup and may vary with different training configurations.\n\nThe following figure illustrates both **latency** (time per training step) and **peak allocated memory** for the different attention implementations and kernel backends. \nKernel-based implementations perform on par with custom-installed attention, and increasing the modelโ€™s `max_length` further enhances performance. Memory consumption is similar across all implementations, showing no significant differences. We get the same performance but with less friction, as described in [the following section](#flash-attention-vs-hub-kernels).\n\n
\n \"Latency\n \"Latency\n
\n\n## Flash Attention vs. Hub Kernels\n\nBuilding Flash Attention from source can be time-consuming, often taking anywhere from several minutes to hours, depending on your hardware, CUDA/PyTorch configuration, and whether precompiled wheels are available. \n\nIn contrast, **Hugging Face Kernels** provide a much faster and more reliable workflow. Developers donโ€™t need to worry about complex setupsโ€”everything is handled automatically. In our benchmarks, kernels were ready to use in about **2.5 seconds**, with no compilation required. This allows you to start training almost instantly, significantly accelerating development. Simply specify the desired version, and `kernels` takes care of the rest.\n\n## Combining FlashAttention Kernels with Liger Kernels\n\nYou can combine **FlashAttention kernels** with **Liger kernels** for additional TRL performance improvements.\n\nFirst, install the Liger kernel dependency:\n\n```bash\npip install liger-kernel\n```\n\nThen, combine both in your code:\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom trl import SFTConfig\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"your-model-name\",\n attn_implementation=\"kernels-community/flash-attn2\" # choose the desired FlashAttention variant\n)\n\ntraining_args = SFTConfig(\n use_liger_kernel=True,\n # ... other TRL training args\n)\n```\n\nLearn more about the [Liger Kernel Integration](./liger_kernel_integration).\n" }, { "path": "docs/source/kto_trainer.md", "content": "# KTO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-KTO-blue)](https://huggingface.co/models?other=kto,trl)\n\n> [!WARNING]\n> As of TRL v1.0, `KTOTrainer` and `KTOConfig` have been moved to the `trl.experimental.kto` module. \n> KTO API is experimental and may change at any time.\n> Promoting KTO back into the stable API is a high-priority task: KTO is slated for refactoring to align with the standard core trainer architecture.\n\n## Overview\n\nKahneman-Tversky Optimization (KTO) was introduced in [KTO: Model Alignment as Prospect Theoretic Optimization](https://huggingface.co/papers/2402.01306) by [Kawin Ethayarajh](https://huggingface.co/kawine), [Winnie Xu](https://huggingface.co/xwinxu), [Niklas Muennighoff](https://huggingface.co/Muennighoff), Dan Jurafsky, [Douwe Kiela](https://huggingface.co/douwekiela).\n\nThe abstract from the paper is the following:\n\n> Kahneman & Tversky's prospect theory tells us that humans perceive random variables in a biased but well-defined manner; for example, humans are famously loss-averse. We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases -- the success of these objectives (e.g., DPO) over cross-entropy minimization can partly be ascribed to them being human-aware loss functions (HALOs). However, the utility functions these methods attribute to humans still differ from those in the prospect theory literature. Using a Kahneman-Tversky model of human utility, we propose a HALO that directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, as current methods do. We call this approach Kahneman-Tversky Optimization (KTO), and it matches or exceeds the performance of preference-based methods at scales from 1B to 30B. Crucially, KTO does not need preferences -- only a binary signal of whether an output is desirable or undesirable for a given input. This makes it far easier to use in the real world, where preference data is scarce and expensive.\n\nThe official code can be found in [ContextualAI/HALOs](https://github.com/ContextualAI/HALOs).\n\nThis post-training method was contributed by [Kashif Rasul](https://huggingface.co/kashif), [Younes Belkada](https://huggingface.co/ybelkada), [Lewis Tunstall](https://huggingface.co/lewtun) and Pablo Vicente.\n\n## Quick start\n\nThis example demonstrates how to train a model using the KTO method. We use the [Qwen 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) as the base model. We use the preference data from the [KTO Mix 14k](https://huggingface.co/datasets/trl-lib/kto-mix-14k). You can view the data in the dataset here:\n\n\n\nBelow is the script to train the model:\n\n```python\n# train_kto.py\nfrom datasets import load_dataset\nfrom trl.experimental.kto import KTOConfig, KTOTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\ntrain_dataset = load_dataset(\"trl-lib/kto-mix-14k\", split=\"train\")\n\ntraining_args = KTOConfig(output_dir=\"Qwen2-0.5B-KTO\")\ntrainer = KTOTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset)\ntrainer.train()\n```\n\nExecute the script using the following command:\n\n```bash\naccelerate launch train_kto.py\n```\n\nDistributed across 8 x H100 GPUs, the training takes approximately 30 minutes. You can verify the training progress by checking the reward graph. An increasing trend in the reward margin indicates that the model is improving and generating better responses over time.\n\n![kto qwen2 reward margin](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/kto-qwen2-reward-margin.png)\n\nTo see how the [trained model](https://huggingface.co/trl-lib/Qwen2-0.5B-KTO) performs, you can use the [Transformers Chat CLI](https://huggingface.co/docs/transformers/quicktour#chat-with-text-generation-models).\n\n
$ transformers chat trl-lib/Qwen2-0.5B-KTO\n<quentin_gallouedec>:\nWhat is the best programming language?\n\n<trl-lib/Qwen2-0.5B-KTO>:\nThe best programming language can vary depending on individual preferences, industry-specific requirements, technical skills, and familiarity with the specific use case or task. Here are some widely-used programming languages that have been noted as popular and widely used:\n\nHere are some other factors to consider when choosing a programming language for a project:\n\n 1 JavaScript: JavaScript is at the heart of the web and can be used for building web applications, APIs, and interactive front-end applications like frameworks like React and Angular. It's similar to C, C++, and F# in syntax structure and is accessible and easy to learn, making it a popular choice for beginners and professionals alike.\n 2 Java: Known for its object-oriented programming (OOP) and support for Java 8 and .NET, Java is used for developing enterprise-level software applications, high-performance games, as well as mobile apps, game development, and desktop applications.\n 3 C++: Known for its flexibility and scalability, C++ offers comprehensive object-oriented programming and is a popular choice for high-performance computing and other technical fields. It's a powerful platform for building real-world applications and games at scale.\n 4 Python: Developed by Guido van Rossum in 1991, Python is a high-level, interpreted, and dynamically typed language known for its simplicity, readability, and versatility.\n
\n\n## Expected dataset format\n\nKTO requires an [unpaired preference dataset](dataset_formats#unpaired-preference). Alternatively, you can provide a *paired* preference dataset (also known simply as a *preference dataset*). In this case, the trainer will automatically convert it to an unpaired format by separating the chosen and rejected responses, assigning `label = True` to the chosen completions and `label = False` to the rejected ones.\n\nThe [`experimental.kto.KTOTrainer`] supports both [conversational](dataset_formats#conversational) and [standard](dataset_formats#standard) dataset formats. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\nIn theory, the dataset should contain at least one chosen and one rejected completion. However, some users have successfully run KTO using *only* chosen or only rejected data. If using only rejected data, it is advisable to adopt a conservative learning rate.\n\n## Example script\n\nWe provide an example script to train a model using the KTO method. The script is available in [`trl/scripts/kto.py`](https://github.com/huggingface/trl/blob/main/trl/scripts/kto.py)\n\nTo test the KTO script with the [Qwen2 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) on the [UltraFeedback dataset](https://huggingface.co/datasets/trl-lib/kto-mix-14k), run the following command:\n\n```bash\naccelerate launch trl/scripts/kto.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/kto-mix-14k \\\n --num_train_epochs 1 \\\n --output_dir Qwen2-0.5B-KTO\n```\n\n## Usage tips\n\n### For Mixture of Experts Models: Enabling the auxiliary loss\n\nMOEs are the most efficient if the load is about equally distributed between experts. \nTo ensure that we train MOEs similarly during preference-tuning, it is beneficial to add the auxiliary loss from the load balancer to the final loss.\n\nThis option is enabled by setting `output_router_logits=True` in the model config (e.g. [`~transformers.MixtralConfig`]). \nTo scale how much the auxiliary loss contributes to the total loss, use the hyperparameter `router_aux_loss_coef=...` (default: `0.001`) in the model config.\n\n### Batch size recommendations\n\nUse a per-step batch size that is at least 4, and an effective batch size between 16 and 128. Even if your effective batch size is large, if your per-step batch size is poor, then the KL estimate in KTO will be poor.\n\n### Learning rate recommendations\n\nEach choice of `beta` has a maximum learning rate it can tolerate before learning performance degrades. For the default setting of `beta = 0.1`, the learning rate should typically not exceed `1e-6` for most models. As `beta` decreases, the learning rate should also be reduced accordingly. In general, we strongly recommend keeping the learning rate between `5e-7` and `5e-6`. Even with small datasets, we advise against using a learning rate outside this range. Instead, opt for more epochs to achieve better results.\n\n### Imbalanced data\n\nThe `desirable_weight` and `undesirable_weight` of the [`experimental.kto.KTOConfig`] refer to the weights placed on the losses for desirable/positive and undesirable/negative examples.\nBy default, they are both 1. However, if you have more of one or the other, then you should upweight the less common type such that the ratio of (`desirable_weight` \\\\(\\times\\\\) number of positives) to (`undesirable_weight` \\\\(\\times\\\\) number of negatives) is in the range 1:1 to 4:3.\n\n## Logged metrics\n\nWhile training and evaluating, we record the following reward metrics:\n\n- `rewards/chosen_sum`: the sum of log probabilities of the policy model for the chosen responses scaled by beta\n- `rewards/rejected_sum`: the sum of log probabilities of the policy model for the rejected responses scaled by beta\n- `logps/chosen_sum`: the sum of log probabilities of the chosen completions\n- `logps/rejected_sum`: the sum of log probabilities of the rejected completions\n- `logits/chosen_sum`: the sum of logits of the chosen completions\n- `logits/rejected_sum`: the sum of logits of the rejected completions\n- `count/chosen`: the count of chosen samples in a batch\n- `count/rejected`: the count of rejected samples in a batch\n\n## KTOTrainer\n\n[[autodoc]] experimental.kto.KTOTrainer\n - train\n - save_model\n - push_to_hub\n\n## KTOConfig\n\n[[autodoc]] experimental.kto.KTOConfig\n" }, { "path": "docs/source/liger_kernel_integration.md", "content": "# Liger Kernel Integration\n\n[Liger Kernel](https://github.com/linkedin/Liger-Kernel) is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduce memory usage by 60%. That way, we can **4x** our context length, as described in the benchmark below. They have implemented Hugging Face compatible `RMSNorm`, `RoPE`, `SwiGLU`, `CrossEntropy`, `FusedLinearCrossEntropy`, with more to come. The kernel works out of the box with [FlashAttention](https://github.com/Dao-AILab/flash-attention), [PyTorch FSDP](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html), and [Microsoft DeepSpeed](https://github.com/microsoft/DeepSpeed).\n\nWith this memory reduction, you can potentially turn off `cpu_offloading` or gradient checkpointing to further boost the performance.\n\n| Speed Up | Memory Reduction |\n| --- | --- |\n| ![Speed up](https://raw.githubusercontent.com/linkedin/Liger-Kernel/main/docs/images/e2e-tps.png) | ![Memory](https://raw.githubusercontent.com/linkedin/Liger-Kernel/main/docs/images/e2e-memory.png) |\n\n## Supported Trainers\n\nLiger Kernel is supported in the following TRL trainers:\n- **SFT** (Supervised Fine-Tuning)\n- **DPO** (Direct Preference Optimization)\n- **GRPO** (Group Relative Policy Optimization)\n- **KTO** (Kahneman-Tversky Optimization)\n- **GKD** (Generalized Knowledge Distillation)\n\n## Usage\n\n1. First, install Liger Kernel:\n\n ```bash\n pip install liger-kernel\n ```\n\n2. Once installed, set `use_liger_kernel=True` in your trainer config. No other changes are needed!\n\n\n\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl import KTOConfig\n\ntraining_args = KTOConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl.experimental.gkd import GKDConfig\n\ntraining_args = GKDConfig(..., use_liger_kernel=True)\n```\n\n\n\n\nTo learn more about Liger-Kernel, visit their [official repository](https://github.com/linkedin/Liger-Kernel/).\n" }, { "path": "docs/source/lora_without_regret.md", "content": "# LoRA Without Regret\n\nRecent research from the team at [Thinking Machines Lab](https://thinkingmachines.ai/blog/lora/) (Schulman et al., 2025) shows that **LoRA can match full fine-tuning performance** when configured correctly, while using only ~67% of the compute. These findings are exciting to TRL users because they're straightforward to implement and can improve model performance on smaller budgets.\n\nThis guide provides simple instructions to reproduce the results of the blog post in TRL.\n\n> [!TIP]\n> It is recommended to read the blog post before following this guide, or to consult both resources in parallel for best results.\n\n## Benefits of LoRA over full fine-tuning\n\nFirst of all, let's remind ourselves of the benefits of [LoRA over full fine-tuning](https://huggingface.co/docs/trl/en/peft_integration).\n\nLoRA adds adapter layers on top of the base model, which contains significantly fewer parameters than the base model itself. This design reduces GPU memory requirements and enables more efficient training. As described in the [blog](https://thinkingmachines.ai/blog/lora/), this approach was originally thought to involve a performance trade-off, although careful configuration can overcome this trade-off and match full fine-tuning performance. \n\n## Examples with TRL\n\nLet's implement and train LoRA adapters in TRL scripts based on the core findings of the blog post. Afterwards, we'll revisit each finding in light of the TRL results.\n\n### Supervised Fine-Tuning (SFT)\n\nThe blog post performs SFT on a range of models and datasets from the Hub, which we can reproduce in TRL.\n\n| Model | Dataset |\n| --- | --- |\n| [Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B) | [allenai/tulu-3-sft-mixture](https://huggingface.co/datasets/allenai/tulu-3-sft-mixture) |\n| [Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B) | [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k) |\n| [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B) | [allenai/tulu-3-sft-mixture](https://huggingface.co/datasets/allenai/tulu-3-sft-mixture) |\n| [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B) | [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k) |\n\n\n\n\nWe can integrate these findings with the TRL Python API like so:\n\n```python\n\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom trl import SFTTrainer, SFTConfig\n\ndataset = load_dataset(\"open-thoughts/OpenThoughts-114k\", split=\"train\")\n\npeft_config = LoraConfig(r=256, lora_alpha=16, target_modules=\"all-linear\")\n\ntraining_args = SFTConfig(\n learning_rate=2e-4,\n per_device_train_batch_size=1,\n gradient_accumulation_steps=4,\n num_train_epochs=1,\n report_to=[\"trackio\"],\n)\n\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen2.5-3B-Instruct\",\n train_dataset=dataset,\n peft_config=peft_config,\n args=training_args,\n)\n\ntrainer.train()\n\n```\n\n\n\n\n```bash\n\nhf jobs uv run \\\n --flavor a100-large \\\n --timeout 8h \\\n --secrets HF_TOKEN \\\n \"https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py\" \\\n --model_name_or_path Qwen/Qwen2.5-3B-Instruct \\\n --dataset_name open-thoughts/OpenThoughts-114k \\\n --learning_rate 2.0e-5 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 16 \\\n --use_peft \\\n --lora_r 256 \\\n --lora_alpha 16 \\\n --lora_target_modules all-linear \\\n --output_dir Qwen2.5-3B-OpenThoughts-LoRA \\\n --report_to trackio \\\n --push_to_hub\n\n```\n\nTo use Hugging Face Jobs, you will need to be logged in to the Hugging Face Hub (`hf auth login`) and have a [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan. Check out the [Jobs documentation](https://huggingface.co/docs/huggingface_hub/en/guides/jobs) for more details.\n\n\n\n\n```bash\n\nuv run \"https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py\" \\\n --model_name_or_path Qwen/Qwen2.5-3B-Instruct \\\n --dataset_name open-thoughts/OpenThoughts-114k \\\n --learning_rate 2.0e-5 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 16 \\\n --eval_strategy no \\\n --use_peft \\\n --lora_r 256 \\\n --lora_alpha 16 \\\n --lora_target_modules all-linear \\\n --output_dir Qwen2.5-3B-OpenThoughts-LoRA \\\n --report_to trackio \\\n --push_to_hub\n\n```\n\nTo run the script locally, you will need to have `uv` installed. Check out the [uv documentation](https://docs.astral.sh/uv/) for more details.\n\n\n\n\nOnce training starts, you can monitor the progress in [Trackio](https://huggingface.co/trackio), which will log the URL.\n\n### Reinforcement Learning (GRPO)\n\nThe blog post performs GRPO on a range of models and datasets from the Hub, and once again we can reproduce the results in TRL.\n\n| Model | Dataset |\n| --- | --- |\n| [Llama-3.1-8B-Base](https://huggingface.co/meta-llama/Llama-3.2-1B) | [GSM8k](https://huggingface.co/datasets/openai/gsm8k) |\n| [Llama-3.1-8B-Base](https://huggingface.co/meta-llama/Llama-3.2-1B) | [DeepMath-103K](https://huggingface.co/datasets/zwhe99/DeepMath-103K) |\n| [Qwen3-8b-base](https://huggingface.co/Qwen/Qwen3-8b-base) | [DeepMath-103K](https://huggingface.co/datasets/zwhe99/DeepMath-103K) |\n\nFor reinforcement learning, the blog uses a math reasoning task that we can reproduce as a Python function.\n\n\n\n\nWe can implement these recommendations with the TRL Python API like so:\n\n```python\n\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom trl import GRPOConfig, GRPOTrainer\nfrom trl.rewards import reasoning_accuracy_reward\n\ndataset = load_dataset(\"HuggingFaceH4/OpenR1-Math-220k-default-verified\", split=\"train\")\n\npeft_config = LoraConfig(\n r=1,\n lora_alpha=32,\n target_modules=\"all-linear\"\n)\n\ntraining_args = GRPOConfig(\n learning_rate=5e-5,\n per_device_train_batch_size=1,\n gradient_accumulation_steps=4,\n num_train_epochs=1,\n num_generations=8,\n generation_batch_size=8,\n report_to=[\"trackio\"],\n)\n\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n reward_funcs=reasoning_accuracy_reward,\n args=training_args,\n train_dataset=dataset,\n peft_config=peft_config,\n)\n\ntrainer.train()\n\n```\n\n> [!WARNING]\n> This snippet skips the reward function which is defined above to keep the example concise.\n\n\n\n\n```bash\n\nhf jobs uv run \\\n --flavor a100-large \\\n --timeout 4h \\\n --secrets HF_TOKEN \\\n --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \\\n \"https://huggingface.co/datasets/burtenshaw/lora-without-regrets/resolve/main/grpo.py\" \\\n --model_name_or_path Qwen/Qwen3-0.6B \\\n --dataset_name HuggingFaceH4/OpenR1-Math-220k-default-verified \\\n --output_dir grpo-full-qwen3-0.6b \\\n --learning_rate 1.0e-6 \\\n --lr_scheduler_type cosine \\\n --warmup_steps 0.0 \\\n --max_grad_norm 1.0 \\\n --beta 0.0 \\\n --max_completion_length 4096 \\\n --num_generations 16 \\\n --generation_batch_size 16 \\\n --gradient_accumulation_steps 8 \\\n --per_device_train_batch_size 1 \\\n --num_train_epochs 1 \\\n --lora_r 1 \\\n --lora_alpha 32 \\\n --lora_dropout 0.0 \\\n --lora_target_modules all-linear \\\n --vllm_mode colocate \\\n --save_strategy steps \\\n --save_steps 50 \\\n --save_total_limit 1 \\\n --logging_steps 1 \\\n --max_steps 200 \\\n --report_to trackio\n```\n\nTo use Hugging Face Jobs, you will need to be logged in to the Hugging Face Hub (`hf auth login`) and have a [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan. Check out the [Jobs documentation](https://huggingface.co/docs/huggingface_hub/en/guides/jobs) for more details.\n\n\n\n\n```bash\nuv run \"https://huggingface.co/datasets/burtenshaw/lora-without-regrets/resolve/main/grpo.py\" \\\n --model_name_or_path Qwen/Qwen3-0.6B \\\n --dataset_name HuggingFaceH4/OpenR1-Math-220k-default-verified \\\n --output_dir grpo-full-qwen3-0.6b \\\n --learning_rate 1.0e-6 \\\n --lr_scheduler_type cosine \\\n --warmup_steps 0.0 \\\n --max_grad_norm 1.0 \\\n --beta 0.0 \\\n --max_completion_length 4096 \\\n --num_generations 16 \\\n --generation_batch_size 16 \\\n --gradient_accumulation_steps 8 \\\n --per_device_train_batch_size 1 \\\n --num_train_epochs 1 \\\n --lora_r 1 \\\n --lora_alpha 32 \\\n --lora_dropout 0.0 \\\n --lora_target_modules all-linear \\\n --vllm_mode colocate \\\n --save_strategy steps \\\n --save_steps 50 \\\n --save_total_limit 1 \\\n --logging_steps 1 \\\n --max_steps 200 \\\n --report_to trackio\n```\n\nTo run the script locally, you will need to have `uv` installed. Check out the [uv documentation](https://docs.astral.sh/uv/) for more details.\n\n\n\n\nThe reinforcement learning script with GRPO is implemented as a custom script in TRL, which uses the reward function shown above. You can review it at [`grpo.py`](https://huggingface.co/datasets/burtenshaw/lora-without-regrets/blob/main/grpo.py) - Reinforcement learning with LoRA best practices\n\n## Key findings in optimizing LoRA\n\nThe authors recommend applying LoRA to all weight matrices rather than limiting it to attention layers, as increasing the rank does not compensate for this restriction. In TRL, this can be configured using `--lora_target_modules all-linear` to apply LoRA to all weight matrices.\n\nWe were able to reproduce the results of the blog post using TRL and the SmolLM3 model. We trained the model for 500 steps on the [Math 220k dataset](https://huggingface.co/datasets/HuggingFaceH4/OpenR1-Math-220k-default-verified) with the reward function and configuration above. As you can see in the figure below, the LoRA model's average train reward curve matches the full fine-tuning curve.\n\n![train reward](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lora_without_regret/5.png)\n\nAnd most importantly, the LoRA model uses significantly less memory than the full fine-tuning model, as we can see in the figure below.\n\n![memory usage](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lora_without_regret/6.png)\n\nHere are the parameters we used to train the above models\n\n| Parameter | LoRA | Full FT |\n| --- | --- | --- |\n| `--model_name_or_path` | HuggingFaceTB/SmolLM3-3B | HuggingFaceTB/SmolLM3-3B |\n| `--dataset_name` | HuggingFaceH4/OpenR1-Math-220k-default-verified | HuggingFaceH4/OpenR1-Math-220k-default-verified |\n| `--learning_rate` | 1.0e-5 | 1.0e-6 |\n| `--max_completion_length` | 4096 | 4096 |\n| `--lora_r` | 1 | - |\n| `--lora_alpha` | 32 | - |\n| `--lora_dropout` | 0.0 | - |\n| `--lora_target_modules` | all-linear | - |\n\nLet's break down the key findings of the blog post and how we were able to reproduce them.\n\n### 1. *LoRA performs better when applied to all weight matrices*\n\nThe authors recommend applying LoRA to all weight matrices rather than limiting it to attention layers, as increasing the rank does not compensate for this restriction.\n\n![all layers](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lora_without_regret/1.png)\n\nAttention-only LoRA underperforms even when using a higher rank to match parameter count. In TRL, this can be configured using `--lora_target_modules all-linear` to apply LoRA to all weight matrices. In Python, we can do this like so:\n\n```python\nfrom peft import LoraConfig \n\npeft_config = LoraConfig(target_modules=\"all-linear\") \n```\n\n### 2. *The adapter needs sufficient capacity to learn from the dataset*\n\nThe blog post recommends using a sufficient LoRA rank to learn from the dataset. The rank determines the number of trainable parameters in the LoRA adapter. Therefore, \"For datasets that exceed LoRA capacity, LoRA underperforms FullFT\".\n\n![learning rate](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lora_without_regret/3.png)\n\nIn the TRL script, we could use `--lora_r` to set the rank and adapt it based on the task and dataset we're training on. The blog post recommends the following ranks based on the task and dataset size:\n\nReinforcement learning tasks typically require lower capacity, so smaller LoRA ranks can be used. This is because policy gradient algorithms extract roughly ~1 bit of information per episode, demanding minimal parameter capacity. \n\nThe blog post defines the ideal dataset size for LoRA to match full fine-tuning as \"Post-training scale\". Which we can use to determine the recommended rank for SFT and RL LoRAs as:\n\n| Task Type | Dataset Size | Recommended Rank |\n| --- | --- | --- |\n| **SFT** | Post-training scale | 256 |\n| **RL** | Any size | 1-32 |\n\n### 3. *\"FullFT and high-rank LoRAs have similar learning curves\"*\n\nCounterintuitively, the blog post recommends using a higher learning rate than for full fine-tuning. In the table above, we used 1.0e-5 for LoRA and 1.0e-6 for full fine-tuning. In the TRL script, we could use `--learning_rate` to set the learning rate. The \\\\( \\frac{1}{r} \\\\) scaling in LoRA makes the optimal learning rate approximately rank-independent.\n\n![learning rate](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lora_without_regret/2.png)\n\n### 4. *\"In some scenarios, LoRA is less tolerant of large batch sizes than full fine-tuning.\"*\n\nThe blog post recommends using an effective batch size < 32 because the authors found LoRA to be less tolerant of large batch sizes. This could not be mitigated by increasing the LoRA rank. In the TRL script, we could use `--per_device_train_batch_size` and `--gradient_accumulation_steps` to set the batch size.\n\n![learning rate](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lora_without_regret/4.png)\n\n## Takeaways\n\nUsing TRL, you can efficiently implement LoRA adapters to match full fine-tuning performance, applying the core insights (targeting all weight matrices, choosing the right rank, and managing batch size and learning rate) without the heavy compute cost of FullFT.\n\n## Citation\n\n```bibtex\n@article{schulman2025lora, \n title = {{LoRA Without Regret}}, \n author = {John Schulman and Thinking Machines Lab}, \n year = 2025, \n journal = {Thinking Machines Lab: Connectionism}, \n doi = {10.64434/tml.20250929}, \n note = {https://thinkingmachines.ai/blog/lora/} \n} \n```\n" }, { "path": "docs/source/merge_model_callback.md", "content": "# MergeModelCallback\n\n[[autodoc]] experimental.merge_model_callback.MergeModelCallback\n" }, { "path": "docs/source/minillm_trainer.md", "content": "# MiniLLM Trainer\n\n[![All_models-MiniLLM-blue](https://img.shields.io/badge/All_models-MiniLLM-blue)](https://huggingface.co/models?other=minillm,trl)\n\n## Overview\n\nTRL supports the MiniLLM Trainer for distilling large language models into smaller ones using reverse KLD for better precision, quality, and performance, as described in the paper [Knowledge Distillation of Large Language Models](https://huggingface.co/papers/2306.08543) by [Yuxian Gu](https://huggingface.co/t1101675), [Li Dong](https://huggingface.co/unilm), [Furu Wei](https://huggingface.co/thegenerality), and Minlie Huang.\nThe abstract from the paper is the following:\n\n> Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively distill the knowledge from white-box generative LLMs is still under-explored, which becomes more and more important with the prosperity of LLMs. In this work, we propose MiniLLM that distills smaller language models from generative larger language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution. Then, we derive an effective optimization approach to learn this objective. Extensive experiments in the instruction-following setting show that the MiniLLM models generate more precise responses with the higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance. Our method is also scalable for different model families with 120M to 13B parameters. We will release our code and model checkpoints at https://aka.ms/MiniLLM.\n\nThis post-training method was contributed by [Yuxian Gu](https://huggingface.co/t1101675).\n\nIt is a generalized version of [Think Machine Lab's On-Policy Distillation](https://thinkingmachines.ai/blog/on-policy-distillation/), with the option to add distribution-level single-step distillation signals (like GKD when `beta=1`) and long-context reverse KLD signals.\n\n$$\n\\begin{align}\nL_{\\text{MiniLLM}}&=\\alpha_1\\mathbb{E}_{x\\sim \\pi_{\\theta}}\\sum_{t'=t}^{|x|}\\frac{\\gamma^{t'-t}}{\\sum_{t'}\\gamma^{t'-t}}\\left[\\log \\frac{\\pi_{\\theta}(x_{t'+1}|x_{1..t'})}{\\pi_{\\text{teacher}}(x_{t'+1}|x_{1..t'})}\\right] \\\\\n&+ \\alpha_2\\mathbb{E}_{x\\sim \\pi_{\\theta}} \\text{KL}\\left[\\pi_\\theta(\\cdot|x_{1..t})||\\pi_{\\text{teacher}}(\\cdot | x_{1..t})\\right].\n\\end{align}\n$$\n\nWhen \\\\( \\alpha_1=1 \\\\), \\\\( \\alpha_2=0 \\\\), \\\\( \\gamma=0 \\\\), which corresponds to\n\n```python\nfrom trl.experimental.minillm import MiniLLMConfig\n\ntraining_args = MiniLLMConfig(\n rkl_advantage=True,\n single_step_decomposition=False,\n gamma=False\n)\n```\n\n\\\\( L_{\\text{MiniLLM}} \\\\) becomes the on-policy KD implemented in [Tinker](https://github.com/thinking-machines-lab/tinker-cookbook/blob/5d08be6d130596b7bedd02197861c41fa81ea436/tinker_cookbook/distillation/train_on_policy.py#L88):\n\n$$\nL_{\\text{tinker}}=\\mathbb{E}_{x\\sim \\pi_{\\theta}}\\left[\\log \\frac{\\pi_{\\theta}(x_{t'+1}|x_{1..t'})}{\\pi_{\\text{teacher}}(x_{t'+1}|x_{1..t'})}\\right].\n$$\n\nWhen \\\\( \\alpha_1=0 \\\\), \\\\( \\alpha_2=1 \\\\), which corresponds to\n\n```python\nfrom trl.experimental.minillm import MiniLLMConfig\n\ntraining_args = MiniLLMConfig(\n rkl_advantage=False,\n single_step_decomposition=True\n)\n```\n\n\\\\( L_{\\text{MiniLLM}} \\\\) becomes the reverse KLD version of the GKD loss as in [GKD Trainer](./gkd.md):\n\n$$\nL_{\\text{GKD-RKL}}=\\mathbb{E}_{x\\sim \\pi_{\\theta}} \\text{KL}\\left[\\pi_\\theta(\\cdot|x_{1..t})||\\pi_{\\text{teacher}}(\\cdot | x_{1..t})\\right].\n$$\n\n## MiniLLMTrainer\n\n[[autodoc]] experimental.minillm.MiniLLMTrainer\n - train\n - save_model\n - push_to_hub\n\n## MiniLLMConfig\n\n[[autodoc]] experimental.minillm.MiniLLMConfig\n" }, { "path": "docs/source/nash_md_trainer.md", "content": "# Nash-MD Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-Nash--MD-blue)](https://huggingface.co/models?other=nash-md,trl)\n\n## Overview\n\nNash-MD was proposed in the paper [Nash Learning from Human Feedback](https://huggingface.co/papers/2312.00886) by Rรฉmi Munos, [Michal Valko](https://huggingface.co/misovalko), Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mรฉsnard, and Andrea Michi.\n\nThe abstract from the paper is the following:\n\n> Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human preferences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.\n\nThis post-training method was contributed by [Kashif Rasul](https://huggingface.co/kashif) and [Daniil Tiapkin](https://huggingface.co/dtiapkin), [Pierre Mรฉnard](https://huggingface.co/menardprr), Daniele Calandriello and [Quentin Gallouรฉdec](https://huggingface.co/qgallouedec).\n\n## Quick start\n\nThis example demonstrates how to train a model using the Nash-MD method. We use the [Qwen 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) as the base model and [`experimental.judges.PairRMJudge`] as a judge. We use the prompts from the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback). You can view the prompts in the dataset here:\n\n\n\nBelow is the script to train the model:\n\n```python\n# train_nash_md.py\nfrom datasets import load_dataset\nfrom trl.experimental.judges import PairRMJudge\nfrom trl.experimental.nash_md import NashMDConfig, NashMDTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\njudge = PairRMJudge()\ntrain_dataset = load_dataset(\"trl-lib/ultrafeedback-prompt\", split=\"train\")\n\ntraining_args = NashMDConfig(output_dir=\"Qwen2-0.5B-NashMD\")\ntrainer = NashMDTrainer(\n model=model, judge=judge, args=training_args, processing_class=tokenizer, train_dataset=train_dataset\n)\ntrainer.train()\n```\n\nExecute the script using the following command:\n\n```bash\naccelerate launch train_nash_md.py\n```\n\nDistributed across 8 GPUs, the training takes approximately 3 hours.\n\nTo see how the [trained model](https://huggingface.co/trl-lib/Qwen2-0.5B-NashMD) performs, you can use the [Transformers Chat CLI](https://huggingface.co/docs/transformers/quicktour#chat-with-text-generation-models).\n\n
$ transformers chat trl-lib/Qwen2-0.5B-NashMD\n<quentin_gallouedec>:\nWhat is the best programming language?\n\n<trl-lib/Qwen2-0.5B-NashMD>:\nThe best programming language depends on personal preference, the complexity of the project, and the specific requirements of the task. Some programming languages that are often recommended include Python, Java, and JavaScript, and there are many other languages to choose from depending on individual needs.\n
\n\n## Expected dataset type\n\nNash-MD requires a [prompt-only dataset](dataset_formats#prompt-only). The [`experimental.nash_md.NashMDTrainer`] supports both [conversational](dataset_formats#conversational) and [standard](dataset_formats#standard) dataset formats. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\n## Usage tips\n\n### Use a reward model\n\nInstead of a judge, you can chose to use a reward model -- see [Reward Bench](https://huggingface.co/spaces/allenai/reward-bench) for a leaderboard of public models you can use. Below is a code example showing how to replace a judge with the [trl-lib/Qwen2-0.5B-Reward](https://huggingface.co/trl-lib/Qwen2-0.5B-Reward) model:\n\n```diff\n- from trl.experimental.judges import PairRMJudge\n+ from transformers import AutoModelForSequenceClassification\n\n- judge = PairRMJudge()\n+ reward_model = AutoModelForSequenceClassification.from_pretrained(\"trl-lib/Qwen2-0.5B-Reward\", num_labels=1)\n\n trainer = NashMDTrainer(\n ...\n- judge=judge,\n+ reward_funcs=reward_model,\n )\n```\n\n> [!WARNING]\n> Make sure that the SFT model and reward model use the _same_ chat template and the same tokenizer. Otherwise, you may find the model completions are scored incorrectly during training.\n\n### Encourage EOS token generation\n\nWe may want the model to generate completions within a given length. During training, the model will generate completions up to the maximum length specified in the `max_new_tokens` argument of [`experimental.nash_md.NashMDConfig`]. If you want to penalize the model for not generating an EOS token before reaching the maximum length, you can use the `missing_eos_penalty` argument of [`experimental.nash_md.NashMDConfig`]:\n\n```python\ntraining_args = NashMDConfig(..., max_new_tokens=128, missing_eos_penalty=1.0)\n```\n\n### Logging Completions\n\nTo better understand your modelโ€™s behavior during training, you can log sample completions periodically using the [`LogCompletionsCallback`].\n\n```python\ntrainer = NashMDTrainer(..., eval_dataset=eval_dataset)\ncompletions_callback = LogCompletionsCallback(trainer, num_prompts=8)\ntrainer.add_callback(completions_callback)\n```\n\nThis callback logs the model's generated completions directly to Weights & Biases.\n\n![Logged Completions](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/wandb_completions.png)\n\n## Example script\n\nWe provide an example script to train a model using the Nash-MD method. The script is available in [`examples/scripts/nash_md.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/nash_md.py)\n\nTo test the online DPO script with the [Qwen2.5 0.5B model](https://huggingface.co/trl-lib/Qwen/Qwen2.5-0.5B-Instruct) on the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback), run the following command:\n\n```bash\npython examples/scripts/nash_md.py \\\n --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \\\n --judge pair_rm \\\n --dataset_name trl-lib/ultrafeedback-prompt \\\n --learning_rate 5.0e-7 \\\n --output_dir Qwen2.5-0.5B-NashMD-PairRM \\\n --warmup_steps 0.1 \\\n --push_to_hub\n```\n\n## Logged metrics\n\nWhile training and evaluating, we record the following reward metrics:\n\n* `loss/kl`: The mean KL divergence between the model and reference data.\n* `objective/entropy`: The mean entropy of the model and reference data.\n* `loss/score`: The mean reinforce score loss.\n* `rewards/chosen`: The mean scores (according to the reward model) of the model completions.\n* `rewards/rejected`: The mean scores (according to the reward model) of the mixture completions.\n* `rewards/probabilities`: The mean probability (according to the reward model or judge) of the model completions chosen vs the mixture completion.\n* `rewards/accuracies`: The accuracies of the Nash-MD's implicit reward model.\n* `rewards/margins`: The mean reward margin (according to reward model) between the chosen and mixture completions.\n* `logps/chosen`: The mean log probabilities of the chosen completions.\n* `logps/rejected`: The mean log probabilities of the reference completions.\n* `val/model_contain_eos_token`: The amount of times the model's output contains the eos token.\n* `val/ref_contain_eos_token`: The amount of times the mixture's output contains the eos token.\n* `beta`: The parameter that controls the weight of the loss term representing the deviation from the reference model. Typically fixed, but can be made dynamic by passing a list to [`experimental.nash_md.NashMDConfig`].\n* `mixture_coef`: Logit mixture coefficient for the model and reference model. Typically fixed, but can be made dynamic by passing a list to [`experimental.nash_md.NashMDConfig`].\n\n## NashMDTrainer\n\n[[autodoc]] experimental.nash_md.NashMDTrainer\n - train\n - save_model\n - push_to_hub\n\n## NashMDConfig\n\n[[autodoc]] experimental.nash_md.NashMDConfig\n" }, { "path": "docs/source/nemo_gym.md", "content": "# NeMo Gym Integration\n\nNVIDIA NeMo Gym is a library for building RL environments for large language models. This integration enables training models in NeMo Gym environments using TRL's GRPOTrainer with vLLM server mode.\n\nThe integration supports multi-step and multi-turn rollouts, multi-environment training, and any NeMo Gym environment (thoroughly tested: workplace assistant, reasoning gym, MCQA, and math with judge).\n\n## Why NeMo Gym\n\n- **Production-Ready Scale**: Tested for frontier model training with diverse environments running in parallel across math, coding, tool use, reasoning, and more.\n- **Multi-Verifier Training**: Supports algorithmic verification, LLM-as-a-judge, and custom verification logic in a single training run.\n- **Decoupled Architecture**: Build agents and environments independently from the training loopโ€”no RL framework expertise required.\n- **OpenAI-Compatible API**: All environments use the standardized OpenAI Responses API for seamless integration with vLLM, OpenAI models, and other endpoints.\n\n## Available Environments\n\nNeMo Gym provides training-ready environments across multiple domains, including but not limited to:\n\n| Environment | Domain | Description |\n|-------------|--------|-------------|\n| Workplace Assistant | Agent | Multi-step tool calling in common office scenarios (calendar, email, and more) |\n| Math with Judge | Math | Math problems with algorithmic or judge-based verification |\n| Code Gen | Coding | Competitive programming problems with code execution |\n| MCQA | Knowledge | Multiple-choice question answering |\n| Instruction Following | Instruction Following | IFEval/IFBench style tasks |\n| Reasoning Gym | Multiple | Single-step procedurally generated verifiable tasks across domains |\n\nFor a complete list of available training environments, refer to the [NeMo Gym repository](https://github.com/NVIDIA-NeMo/Gym#-available-resource-servers).\n\n## Before You Start\n\nComplete these one-time setup steps before running training.\n\n### Install TRL and NeMo Gym\n\n1. **Install TRL with vLLM extras**\n\n ```bash\n cd trl/\n uv venv\n source .venv/bin/activate\n uv sync --extra vllm\n ```\n\n1. **Install NeMo Gym**\n\n ```bash\n # deactivate trl venv\n deactivate\n git clone https://github.com/NVIDIA-NeMo/Gym.git\n cd Gym\n uv venv --python 3.12\n source .venv/bin/activate\n uv sync\n ```\n\n### Prepare a Dataset\n\nMany NeMo Gym datasets used to train Nemotron models are available on Hugging Face. Use `ng_prepare_data` to download and prepare datasets. This command:\n\n- Downloads the dataset from Hugging Face\n- Validates the data format\n- Adds an `agent_ref` field to each example that tells NeMo Gym which agent server should handle that example\n\n> **Note**: `train_multi_environment.py` adds the `agent_ref` field when loading datasets, so this step is optional if datasets are created another way.\n\n1. **Set Hugging Face Token**\n\n Create `env.yaml` in `Gym/` with your HF token:\n\n ```yaml\n hf_token: \n ```\n\n1. **Prepare Dataset**\n\n ```bash\n # Enter Gym and activate the venv\n cd Gym\n source .venv/bin/activate\n\n # Set config paths\n config_paths=\"responses_api_models/vllm_model/configs/vllm_model.yaml,\\\n resources_servers/workplace_assistant/configs/workplace_assistant.yaml\"\n\n # Download data and prep for training\n ng_prepare_data \"+config_paths=[${config_paths}]\" \\\n +output_dirpath=data/workplace_assistant \\\n +mode=train_preparation \\\n +should_download=true \\\n +data_source=huggingface\n ```\n\n This creates `train.jsonl` and `validation.jsonl` files in `data/workplace_assistant/`.\n\nTo create a new environment, refer to the [environment creation guide](https://docs.nvidia.com/nemo/gym/latest/contribute/environments/new-environment.html). We suggest running an existing one first!\n\n#### Dataset Format\n\nNeMo Gym datasets are stored as JSONL. Each line contains a task with input messages, tool definitions, metadata such as ground truth for verification, and an agent server reference. The following example shows the workplace dataset structure. Metadata fields can differ between datasets, as long as the corresponding resources server uses the fields appropriately.\n\n```json\n{\n \"responses_create_params\": {\n \"input\": [\n {\"role\": \"system\", \"content\": \"...\"},\n {\"role\": \"user\", \"content\": \"Move any of jinsoo's tasks that are in review to completed\"}\n ],\n \"tools\": [...],\n \"parallel_tool_calls\": false,\n \"temperature\": 1\n },\n \"ground_truth\": [\n {\"name\": \"project_management_update_task\", \"arguments\": \"{...}\"},\n ...\n ],\n \"category\": \"workbench_project_management\",\n \"environment_name\": \"workbench\",\n \"agent_ref\": {\n \"type\": \"responses_api_agents\",\n \"name\": \"workplace_assistant_simple_agent\"\n }\n}\n```\n\n## Interactive Training\n\nFor development and testing on a single node.\n\n### Set Up\n\n1. **Update Environment Config**\n\n Update `env.yaml` in `Gym/` to include model information:\n\n ```yaml\n policy_base_url: http://127.0.0.1:8000/v1\n policy_api_key: EMPTY\n policy_model_name: Qwen/Qwen2.5-1.5B-Instruct\n hf_token: ...\n ```\n\n2. **Update Training Config**\n\n Update `examples/scripts/nemo_gym/config.yaml` to point to the dataset generated above, and any other optional modifications.\n\n### Run Training\n\nThe following steps run in 3 terminals. It can also be ran with processes in the background, or using tmux.\n\n1. **Start NeMo Gym Servers** (Terminal 1)\n\n ```bash\n cd Gym/\n source .venv/bin/activate\n\n config_paths=\"resources_servers/workplace_assistant/configs/workplace_assistant.yaml,\\\n responses_api_models/vllm_model/configs/vllm_model_for_training.yaml\"\n\n ng_run \"+config_paths=[${config_paths}]\"\n ```\n\n This starts:\n - **Agent server**: Orchestrates rollouts using resource servers and model servers\n - **Resources server**: Supports environment logic such as state-management, tool implementations, and task verification\n - **Model server**: Adapts vLLM server requests to support NeMo Gym agents and on-policy RL training while ensuring OpenAI API compatibility\n - **Head server**: Manages servers used in training enabling their discovery\n\n1. **Start TRL vLLM Server on GPU 0** (Terminal 2)\n\n ```bash\n cd trl/\n source .venv/bin/activate\n CUDA_VISIBLE_DEVICES=0 trl vllm-serve \\\n --model Qwen/Qwen2.5-1.5B-Instruct \\\n --max-model-len 16384 \\\n --host 0.0.0.0 \\\n --port 8000\n ```\n\n1. **Run Training on GPU 1** (Terminal 3)\n\n ```bash\n source trl/.venv/bin/activate\n cd trl/examples/scripts/nemo_gym\n export WANDB_API_KEY=... \n uv add omegaconf \n\n CUDA_VISIBLE_DEVICES=1 python train_multi_environment.py --config config.yaml\n ```\n\n## Multi-Node Training with Slurm\n\nAn example five-node training script is provided in `submit.sh`. Nodes one through four run the training algorithm, while node five runs vLLM inference for NeMo Gym agent rollouts.\n\n1. **Configure the Script**\n\n Update `submit.sh` with your Slurm account, partition, paths to your project directory, and updated training configs.\n\n1. **Submit the Job**\n\n ```bash\n sbatch submit.sh\n ```\n\n1. **Monitor Training**\n\n ```bash\n tail -f logs//*\n ```\n\n> **Tip**: Set up wandb logging for detailed training metrics. For more details on TRL's vLLM integration, refer to the vLLM integration page.\n\n## Multi-Environment Training\n\nTrain on multiple NeMo Gym environments simultaneously. This allows learning diverse capabilities, such as tool calling and math reasoning, in a single training run.\n\n1. **Prepare Individual Datasets**\n\n Prepare datasets for each environment. The workplace assistant dataset was prepared above. Now lets create a dataset for the mini sudoku environment implemented by the reasoning gym resources server in NeMo Gym:\n\n ```bash\n cd Gym\n source .venv/bin/activate\n uv add reasoning-gym\n cd resources_servers/reasoning_gym\n python scripts/create_dataset.py \\\n --task mini_sudoku \\\n --size 2000 \\\n --seed 42 \\\n --output data/reasoning_gym/train_mini_sudoku.jsonl\n\n python scripts/create_dataset.py \\\n --task mini_sudoku \\\n --size 50 \\\n --seed 24 \\\n --output data/reasoning_gym/val_mini_sudoku.jsonl\n ```\n\n1. **Create Combined Dataset**\n\n Combine datasets into a single file with tasks from both environments:\n\n ```bash\n cat data/workplace_assistant/train_workplace.jsonl data/reasoning_gym/train_mini_sudoku.jsonl | shuf > train_multi_env.jsonl\n ```\n\n > **Tip**: Ensure datasets are the same size before shuffling for an even blend of tasks. Repeat for the validation dataset.\n\n1. **Update Training Config**\n\n Update the config to point to the combined dataset:\n\n ```yaml\n model_name: \"Qwen/Qwen3-4B-Instruct-2507\"\n\n dataset_path: \"/path/to/data/train_multi_env.jsonl\"\n eval_dataset_path: \"/path/to/data/val_multi_env.jsonl\"\n\n task: \"workplace-sudoku\" # used in wandb run name\n output_dir: \"outputs/nemo_gym_multi_env\"\n\n # ... rest of config same\n ```\n\n1. **Update ng_run**\n\n Whether training interactively or via Slurm, update the `ng_run` command to include config files from each resources server:\n\n ```bash\n cd Gym\n source .venv/bin/activate\n\n config_paths=\"responses_api_models/vllm_model/configs/vllm_model.yaml,\\\n resources_servers/workplace_assistant/configs/workplace_assistant.yaml,\\\n resources_servers/reasoning_gym/configs/reasoning_gym.yaml\"\n\n ng_run \"+config_paths=[${config_paths}]\"\n ```\n\n This starts servers for both environments. The training script automatically routes each example to the correct agent server based on its `agent_ref` field.\n\n1. **Run Training**\n\n Update the Slurm submission script to use the new training config and both `ng_run` resources server configs, then submit the job as before.\n\n The training script reads `agent_ref` from each example's metadata, routes requests to the correct NeMo Gym agent server, and handles different agents and environments in the same batch.\n\n## Resources\n\n- [NeMo Gym GitHub](https://github.com/NVIDIA-NeMo/Gym)\n- [NeMo Gym Documentation](https://docs.nvidia.com/nemo/gym/latest/)\n- [Training Script](https://github.com/huggingface/trl/blob/main/examples/scripts/nemo_gym/train_multi_environment.py)\n- [TRL GRPO Trainer](grpo_trainer)\n" }, { "path": "docs/source/online_dpo_trainer.md", "content": "# Online DPO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-Online_DPO-blue)](https://huggingface.co/models?other=online-dpo,trl)\n\n## Overview\n\nOnline DPO was proposed in [Direct Language Model Alignment from Online AI Feedback](https://huggingface.co/papers/2402.04792) by Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel.\n\nThe abstract from the paper is the following:\n\n> Direct alignment from preferences (DAP) methods, such as DPO, have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF), that do not require a separate reward model. However, the preference datasets used in DAP methods are usually collected ahead of training and never updated, thus the feedback is purely offline. Moreover, responses in these datasets are often sampled from a language model distinct from the one being aligned, and since the model evolves over training, the alignment phase is inevitably off-policy. In this study, we posit that online feedback is key and improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as annotator: on each training iteration, we sample two responses from the current model and prompt the LLM annotator to choose which one is preferred, thus providing online feedback. Despite its simplicity, we demonstrate via human evaluation in several tasks that OAIF outperforms both offline DAP and RLHF methods. We further show that the feedback leveraged in OAIF is easily controllable, via instruction prompts to the LLM annotator.\n\nThis post-training method was contributed by [Michael Noukhovitch](https://huggingface.co/mnoukhov), [Shengyi Costa Huang](https://huggingface.co/vwxyzjn), [Quentin Gallouรฉdec](https://huggingface.co/qgallouedec), and [Edward Beeching](https://huggingface.co/edbeeching).\n\n## Quick start\n\nThis example demonstrates how to train a model using the online DPO method. We use the [Qwen 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) as the base model and [`experimental.judges.PairRMJudge`] as a judge. We use the prompts from the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback). You can view the prompts in the dataset here:\n\n\n\nBelow is the script to train the model:\n\n```python\n# train_online_dpo.py\nfrom datasets import load_dataset\nfrom trl.experimental.judges import PairRMJudge\nfrom trl.experimental.online_dpo import OnlineDPOConfig, OnlineDPOTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\njudge = PairRMJudge()\ntrain_dataset = load_dataset(\"trl-lib/ultrafeedback-prompt\", split=\"train\")\n\ntraining_args = OnlineDPOConfig(output_dir=\"Qwen2-0.5B-OnlineDPO\")\ntrainer = OnlineDPOTrainer(\n model=model, judge=judge, args=training_args, processing_class=tokenizer, train_dataset=train_dataset\n)\ntrainer.train()\n```\n\nExecute the script using the following command:\n\n```bash\naccelerate launch train_online_dpo.py\n```\n\nDistributed across 8 GPUs, the training takes approximately 1 hour. You can verify the training progress by checking the reward graph. An increasing trend in both the reward for rejected and chosen completions indicates that the model is improving and generating better responses over time.\n\n![](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/online-dpo-qwen2.png)\n\nTo see how the [trained model](https://huggingface.co/trl-lib/Qwen2-0.5B-OnlineDPO) performs, you can use the [Transformers Chat CLI](https://huggingface.co/docs/transformers/quicktour#chat-with-text-generation-models).\n\n
$ transformers chat trl-lib/Qwen2-0.5B-OnlineDPO\n<quentin_gallouedec>:\nWhat is the best programming language?\n\n<trl-lib/Qwen2-0.5B-OnlineDPO>:\nThe best programming language depends on your specific needs and priorities. Some people prefer imperative programming languages (like Haskell or Lisp), while others prefer functional programming languages (like Scala or Python). It's important to consider your work style, programming environment, and project requirements when choosing a programming language.\n
\n\n## Expected dataset type\n\nOnline DPO only requires a [prompt-only dataset](dataset_formats#prompt-only) (unlike offline DPO, that expects [preference dataset](dataset_formats#preference)). The [`experimental.online_dpo.OnlineDPOTrainer`] supports both [conversational](dataset_formats#conversational) and [standard](dataset_formats#standard) dataset formats. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\n## Usage tips\n\n### Use a reward model\n\nInstead of a judge, you can chose to use a reward model -- see [Reward Bench](https://huggingface.co/spaces/allenai/reward-bench) for a leaderboard of public models you can use. Below is a code example showing how to replace a judge with the [trl-lib/Qwen2-0.5B-Reward](https://huggingface.co/trl-lib/Qwen2-0.5B-Reward) model:\n\n```diff\n- from trl.experimental.judges import PairRMJudge\n+ from transformers import AutoModelForSequenceClassification\n\n- judge = PairRMJudge()\n+ reward_model = AutoModelForSequenceClassification.from_pretrained(\"trl-lib/Qwen2-0.5B-Reward\", num_labels=1)\n+ reward_tokenizer = AutoTokenizer.from_pretrained(\"trl-lib/Qwen2-0.5B-Reward\")\n\n trainer = OnlineDPOTrainer(\n ...\n- judge=judge,\n+ reward_funcs=reward_model,\n+ reward_processing_class=reward_tokenizer,\n ...\n )\n```\n\n### Encourage EOS token generation\n\nWhen using a reward model, we may want the model to generate completions within a given length. During training, the model will generate completions up to the maximum length specified in the `max_new_tokens` argument of [`experimental.online_dpo.OnlineDPOConfig`]. If you want to penalize the model for not generating an EOS token before reaching the maximum length, you can use the `missing_eos_penalty` argument of [`experimental.online_dpo.OnlineDPOConfig`]:\n\n```python\ntraining_args = OnlineDPOConfig(..., max_new_tokens=128, missing_eos_penalty=1.0)\n```\n\n### Logging Completions\n\nTo better understand your modelโ€™s behavior during training, you can log sample completions periodically using the [`LogCompletionsCallback`].\n\n```python\ntrainer = OnlineDPOTrainer(..., eval_dataset=eval_dataset)\ncompletions_callback = LogCompletionsCallback(trainer, num_prompts=8)\ntrainer.add_callback(completions_callback)\n```\n\nThis callback logs the model's generated completions directly to Weights & Biases.\n\n![Logged Completions](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/wandb_completions.png)\n\n## Example script\n\nWe provide an example script to train a model using the online DPO method. The script is available in [`examples/scripts/dpo_online.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/dpo_online.py)\n\nTo test the online DPO script with the [Qwen2.5 0.5B model](https://huggingface.co/trl-lib/Qwen/Qwen2.5-0.5B-Instruct) on the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback), run the following command:\n\n```bash\npython examples/scripts/dpo_online.py \\\n --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \\\n --judge pair_rm \\\n --dataset_name trl-lib/ultrafeedback-prompt \\\n --learning_rate 5.0e-7 \\\n --output_dir Qwen2.5-0.5B-Online-DPO-PairRM \\\n --warmup_steps 0.1 \\\n --push_to_hub\n```\n\n## Logged metrics\n\nWhile training and evaluating, we record the following reward metrics. Here is an example [tracked run at Weights and Biases](https://wandb.ai/huggingface/trl/runs/w4apmsi9)\n\n* `objective/kl`: The mean Kullback-Leibler (KL) divergence between the current model and reference model.\n* `objective/entropy`: The mean entropy of the model, indicating the randomness of the actions chosen by the model.\n* `objective/non_score_reward`: The mean reward from non-score-related sources, basically `beta * kl.sum(1)`, where `beta` is the KL penalty coefficient and `kl` is the per-token KL divergence.\n* `objective/rlhf_reward`: The mean RLHF reward, which is `scores - non_score_reward`. The `rlhf_reward` is the ultimate objective of online DPO training. If training works as intended, this metric should keep going up.\n* `objective/scores`: The mean scores returned by the reward model.\n* `objective/scores_margin`: The mean score margin (according to the external reward model) between the chosen and rejected completions.\n* `rewards/chosen`: The mean reward (according to online DPO's implicit reward model)of the chosen completions.\n* `rewards/rejected`: The mean reward (according to online DPO's implicit reward model) of the rejected completions.\n* `rewards/accuracies`: The accuracies of the online DPO's implicit reward model.\n* `rewards/margins`: The mean reward margin (according to online DPO's implicit reward model) between the chosen and rejected completions.\n* `logps/chosen`: The mean log probabilities of the chosen completions.\n* `logps/rejected`: The mean log probabilities of the rejected completions.\n* `val/contain_eos_token`: The fraction of completions which contain an EOS token.\n* `beta`: The parameter that controls the weight of the loss term representing the deviation from the reference model. Typically fixed, but can be made dynamic by passing a list to [`experimental.online_dpo.OnlineDPOConfig`].\n\n## Benchmark experiments\n\nTo validate the online DPO implementation works, we ran experiments with the Pythia 1B, 2.8B, and 6.9B models on a single node of 8 x H100s. Here are the commands we used to run the experiments. We take the SFT / RM models directly from [The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization](https://huggingface.co/papers/2403.17031).\n\n```shell\n# 1B Online DPO experiment\naccelerate launch --config_file examples/accelerate_configs/multi_gpu.yaml \\\n examples/scripts/dpo_online.py \\\n --model_name_or_path trl-lib/pythia-1b-deduped-tldr-sft \\\n --reward_model_path trl-lib/pythia-1b-deduped-tldr-rm \\\n --dataset_name trl-lib/tldr \\\n --learning_rate 5.0e-7 \\\n --output_dir pythia-1b-deduped-tldr-online-dpo \\\n --beta 0.1 \\\n --per_device_train_batch_size 8 \\\n --gradient_accumulation_steps 2 \\\n --num_train_epochs 3 \\\n --max_new_tokens 53 \\\n --warmup_steps 0.1 \\\n --missing_eos_penalty 1.0 \\\n --save_steps 0.1 \\\n --push_to_hub\n\n# 2.8B Online DPO experiment\naccelerate launch --config_file examples/accelerate_configs/deepspeed_zero2.yaml \\\n examples/scripts/dpo_online.py \\\n --model_name_or_path trl-lib/pythia-2.8b-deduped-tldr-sft \\\n --reward_model_path trl-lib/pythia-2.8b-deduped-tldr-rm \\\n --dataset_name trl-lib/tldr \\\n --learning_rate 5.0e-7 \\\n --output_dir pythia-2.8b-deduped-tldr-online-dpo \\\n --beta 0.1 \\\n --per_device_train_batch_size 8 \\\n --gradient_accumulation_steps 2 \\\n --num_train_epochs 3 \\\n --max_new_tokens 53 \\\n --warmup_steps 0.1 \\\n --missing_eos_penalty 1.0 \\\n --save_steps 0.1 \\\n --push_to_hub\n\n# 6.9B Online DPO experiment\naccelerate launch --config_file examples/accelerate_configs/deepspeed_zero2.yaml \\\n examples/scripts/dpo_online.py \\\n --model_name_or_path trl-lib/pythia-6.9b-deduped-tldr-sft \\\n --reward_model_path trl-lib/pythia-6.9b-deduped-tldr-rm \\\n --dataset_name trl-lib/tldr \\\n --learning_rate 5.0e-7 \\\n --output_dir pythia-6.9b-deduped-tldr-online-dpo \\\n --beta 0.1 \\\n --per_device_train_batch_size 4 \\\n --gradient_accumulation_steps 4 \\\n --num_train_epochs 3 \\\n --max_new_tokens 53 \\\n --warmup_steps 0.1 \\\n --missing_eos_penalty 1.0 \\\n --save_steps 0.1 \\\n --push_to_hub\n```\n\nCheckpoints and experiment tracking are available at:\n\n* [๐Ÿค— Model checkpoints](https://huggingface.co/collections/trl-lib/online-dpo-66acd3fa38a331a9cd457b07)\n* [๐Ÿ Tracked experiment](https://wandb.ai/huggingface/trl/reports/Online-DPO-experiments-for-TL-DR-summarisation--Vmlldzo5MTczMDU0)\n\nTo evaluate, we use [vLLM](https://github.com/vllm-project/vllm) to load the checkpoints and GPT-4o mini as a judge model to evaluate the generated TL;DR against the reference TL;DR.\nFor more information on how to use judges, see [Judges](judges).\n\n```bash\n$ python examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/pythia-1b-deduped-tldr-sft --judge_model gpt-4o-mini --num_examples 1000\nModel win rate: 33.00%\npython examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/pythia-6.9b-deduped-tldr-sft --judge_model gpt-4o-mini --num_examples 1000\nModel win rate: 41.50%\npython examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/pythia-1b-deduped-tldr-online-dpo --judge_model gpt-4o-mini --num_examples 1000\nModel win rate: 62.60%\npython examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/pythia-6.9b-deduped-tldr-online-dpo --judge_model gpt-4o-mini --num_examples 1000\nModel win rate: 74.20%\n```\n\nWe can then plot the RLHF scaling chart.\n\n```python\nimport matplotlib.pyplot as plt\n\nresults = {\n \"SFT\": {1.0e9: 0.21, 2.8e9: 0.27, 6.9e9: 0.316},\n \"online-dpo\": {1.0e9: 0.542, 2.8e9: 0.746, 6.9e9: 0.796},\n \"offline-dpo\": {1.0e9: 0.422, 2.8e9: 0.517, 6.9e9: 0.701},\n}\n\n\nplt.plot(results[\"SFT\"].keys(), results[\"SFT\"].values(), label=\"SFT\", marker=\"o\")\nplt.plot(results[\"online-dpo\"].keys(), results[\"online-dpo\"].values(), label=\"Online-dpo with RM judge\", marker=\"o\")\nplt.plot(results[\"offline-dpo\"].keys(), results[\"offline-dpo\"].values(), label=\"Offline-dpo\", marker=\"o\")\nplt.axhline(y=0.5, color=\"black\", linestyle=\"-.\", label=\"Human reference summary\")\nplt.xscale(\"log\")\nplt.xlabel(\"Model size\")\nplt.ylabel(\"Win rate against reference summaries\\n(according to GPT-4-0613)\")\nplt.title(\"DPO scaling by model size\")\nplt.legend()\nplt.xlim(5e8, 1.2e10)\nplt.xticks([1e9, 3e9, 1e10], [\"1B\", \"3B\", \"10B\"])\nplt.grid(True, which=\"both\", ls=\"--\", c=\"0.7\")\nplt.tight_layout()\nplt.show()\n```\n\nThe online DPO checkpoint gets increasingly more win rate as we scale up the model sizes. This is a good sign that the online DPO implementation is working as intended.\n\n## OnlineDPOTrainer\n\n[[autodoc]] experimental.online_dpo.OnlineDPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## OnlineDPOConfig\n\n[[autodoc]] experimental.online_dpo.OnlineDPOConfig\n" }, { "path": "docs/source/openenv.md", "content": "# OpenEnv Integration for Training LLMs with Environments\n\n[OpenEnv](https://github.com/meta-pytorch/OpenEnv) is an open-source framework from Meta's PyTorch team for defining, deploying, and interacting with environments in reinforcement learning (RL) and agentic workflows. It offers [Gymnasium-style APIs](https://gymnasium.farama.org) (e.g., `reset()` and `step()`) to interface with environments in a standard manner, and supports running these environments as backend servers (for example, via HTTP or containerised execution). You can find a collection of ready-to-use OpenEnv environments on the [Hugging Face Hub](https://huggingface.co/collections/openenv/openenv-environment-hub).\n\nIn this guide, weโ€™ll focus on **how to integrate OpenEnv with TRL**, but feel free to explore the links above to dive deeper into OpenEnv itself.\n\n> [!NOTE]\n> You can explore ready-to-use example [scripts](example_overview#scripts) and [notebooks](example_overview#notebooks) in the Examples Overview.\n\n> [!NOTE]\n> Explore the [OpenEnv docs](https://meta-pytorch.org/OpenEnv/) for more details.\n\n## Installation\n\nTo use OpenEnv with TRL, install the environment package. You have two options:\n\n**Option A - Install from HF Space (recommended):**\n\n```bash\npip install git+https://huggingface.co/spaces/openenv/echo_env\n```\n\n> [!TIP]\n> You can also install the core package from PyPI with `pip install \"openenv-core[core]>=0.2.1\"`, but note that environment-specific dependencies may need to be installed separately.\n\n**Option B - Clone OpenEnv repo (for development):**\n\n```bash\ngit clone https://github.com/meta-pytorch/OpenEnv.git\ncd OpenEnv/envs/echo_env\npip install -e .\n```\n\n## Using `rollout_func` with OpenEnv environments\n\nTRL's [`GRPOTrainer`] supports _custom rollout logic_ through the `rollout_func` argument. This lets you override the trainer's default text-generation loop and directly interact with OpenEnv environments โ€” for instance, to compute environment-driven rewards instead of relying solely on model-based signals.\n\n### Rollout Function Signature\n\nA rollout function must have the following signature:\n\n```python\ndef rollout_func(\n prompts: list[str],\n trainer: GRPOTrainer,\n) -> dict[str, list]:\n \"\"\"\n Custom rollout function for generation and reward computation.\n\n Args:\n prompts: List of prompts routed to the current process\n trainer: Active GRPOTrainer (gives access to tokenizer, config and helper utilities)\n\n Returns:\n Dictionary containing:\n - prompt_ids: List of token IDs for each prompt\n - completion_ids: List of token IDs for each completion\n - logprobs: List of log probabilities for each token\n - Any additional fields are forwarded to reward functions as kwargs\n \"\"\"\n pass\n```\n\n> [!NOTE]\n> Any extra fields in the returned dictionary (beyond the required three) are automatically forwarded to your reward functions. This makes it easy to propagate signals such as environment rewards or auxiliary metrics from the rollout step.\n\n### Integration pattern\n\nThe typical pattern when combining OpenEnv with TRL looks like this:\n\n1. Start or connect to an OpenEnv environment (e.g., a Dockerized env or HTTP endpoint).\n2. Generate completions from your model โ€” either via `trl.experimental.openenv.generate_rollout_completions` when using colocated vLLM, or by hitting your inference server when using vLLM in server mode.\n3. Step through the environment using each completion to compute rewards or metrics.\n4. Add environment results (e.g., `env_reward`) to the rollout result dict.\n5. Access those rewards inside your reward function via `**kwargs`.\n\nBy using OpenEnv in this loop, you can:\n\n* Train with realistic or interactive feedback (not just static reward functions).\n* Plug in custom simulators, web APIs, or evaluators as environments.\n* Pass structured reward signals back into RL training seamlessly.\n\n### vLLM Modes\n\nTRL supports two vLLM execution modes for generation:\n\n- **`colocate` mode** (default): vLLM runs in the same process as training. Requires 1 GPU. Use `trl.experimental.openenv.generate_rollout_completions` for generation.\n- **`server` mode**: vLLM runs as a separate server process. Requires at least 2 GPUs (one for vLLM server, one for training), but is highly scalable:\n - You can allocate multiple GPUs to the vLLM server for tensor parallelism (faster inference)\n - You can run multiple training processes that share the same vLLM server\n - You can use different GPU types for inference vs training (e.g., A100 for vLLM, H100 for training)\n - The vLLM server can serve multiple experiments simultaneously\n - Use `trl.experimental.openenv.generate_rollout_completions` which will communicate with the server via `vllm_server_url`\n\nConfigure the mode via `GRPOConfig`:\n\n```python\n# Colocate mode (1 GPU)\nargs = GRPOConfig(\n use_vllm=True,\n vllm_mode=\"colocate\",\n # ... other args\n)\n\n# Server mode (2+ GPUs, scalable)\nargs = GRPOConfig(\n use_vllm=True,\n vllm_mode=\"server\",\n vllm_server_base_url=\"http://localhost:8000\",\n # ... other args\n)\n\n# Example: Start vLLM server with multiple GPUs for tensor parallelism\n# CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen3-1.7B --tensor-parallel-size 4\n```\n\n## Running the Environments\n\nYou can run OpenEnv environments in three different ways: \n\n- We can load the environment from the Hugging Face Hub and execute it as a Docker container.\n- We can connect to a hosted environment running on the Hugging Face Hub.\n- We can launch the environment directly using Uvicorn in Python.\n\n\n\n\n\n**Load from Hugging Face Hub** *(recommended)*\n\nWe can use the [`from_hub`](https://meta-pytorch.org/OpenEnv/core/#core.http_env_client.HTTPEnvClient.from_hub) method to load the environment from the hub. This method will automatically start a Docker container for the environment on your local machine. [`openenv/echo-env`](https://huggingface.co/spaces/openenv/echo_env) is the repo_id of the space on the hub.\n\n```python\nenv = EchoEnv.from_hub(\"openenv/echo-env\")\n```\n\nIf you want to launch the environment manually, you can use the following command to pull and run the Docker container:\n\n```bash\ndocker run -d -p 8001:8000 --platform linux/amd64 registry.hf.space/openenv-echo-env:latest\n```\n\nAnd then you can connect to the environment using the following code:\n\n```python\nenv = EchoEnv(base_url=\"http://0.0.0.0:8001\")\n```\n\nHere, we map the ports from 8001 to 8000 to make space for a vLLM server, but you will need to manage the ports for your local machine.\n\n> [!NOTE]\n> You can find the Docker container for any space on the hub.\n>\n> * Open the space page on the hub.\n> * Click the **โ‹ฎ (three dots)** menu.\n> * Select **โ€œRun locally.โ€**\n> * Copy and execute the provided command in your terminal.\n>\n> ![open_env_launch_docker](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/open_env_launch_docker.png)\n\n> [!NOTE]\n> You can also use the **Docker option** with `from_docker_image` by providing the image name..\n> For more details, refer to the official [OpenEnv documentation](https://meta-pytorch.org/OpenEnv/core/).\n\n\n\n\n**Connect to a remote Hugging Face Space**\n\nYou can connect to a hosted environment running on the Hugging Face Hub by passing the URL of the space to the `base_url` parameter of the environment class.\n\n```python\nenv = EchoEnv(base_url=\"https://openenv-echo-env.hf.space\")\n```\n\n> [!NOTE]\n> You can find the connection URL of any space on the hub.\n>\n> * Open the space page on the hub.\n> * Click the **โ‹ฎ (three dots)** menu.\n> * Select **โ€œEmbed this Space.โ€**\n> * Copy the connection URL.\n\n> [!WARNING]\n> **Currently**, it is recommended to **duplicate the Space to your own account** to avoid potential concurrency issues. \n\n\n\n\n\n**Local Python process**\n\nYou can start the server manually as a local Python process. For more details about the available environments, refer to the [OpenEnv catalog](https://meta-pytorch.org/OpenEnv/environments/).\n \n```bash\nhf download openenv/echo_env --repo-type=space --local-dir=echo_env\npython -m uvicorn echo_env.src.envs.echo_env.server.app:app --host 0.0.0.0 --port 8001\n```\n\nAnd then you can connect to the environment using the following code:\n\n```python\nenv = EchoEnv(base_url=\"http://0.0.0.0:8001\")\n```\n\n\n\n\n\n## Environments Catalog\n\nEnvironment development is active and evolving.\nThe best way to explore the **current catalog of maintained environments** is by visiting the official OpenEnv [catalog](https://huggingface.co/collections/openenv/environment-hub).\n\nCustom environments are also supported. To learn how to create your own, check out the guide on [Building Your Own Environment with OpenEnv](https://meta-pytorch.org/OpenEnv/environment-builder/).\n\nEnvironments are tightly integrated with the Hub, allowing you to **push new environments directly** so the community can easily pull, reuse, and adapt them for their own use cases.\n\n## A simple example\n\n> [!NOTE]\n> You can explore more ready-to-use example scripts in the [`examples/scripts/openenv/`](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/) directory.\n\nThe [echo.py](https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/echo.py) script demonstrates a minimal, end-to-end integration between TRL and OpenEnv. In this example, the [Echo environment](https://meta-pytorch.org/OpenEnv/environments/echo/) rewards completions based on their text length, encouraging the model to generate longer outputs. This pattern can be extended to any custom environment that provides structured feedback or task-based rewards:\n\n```python\nfrom echo_env import EchoEnv, EchoAction\nfrom trl import GRPOConfig, GRPOTrainer\nfrom trl.experimental.openenv import generate_rollout_completions\n\n# Create HTTP client for Echo Environment\nclient = EchoEnv.from_hub(\"openenv/echo-env\")\n\n\"\"\"\nAlternatively, you can start the environment manually with Docker and connect to it:\n\n# Step 1: Start the Echo environment\ndocker run -d -p 8001:8001 registry.hf.space/openenv-echo-env:latest\n\n# Step 2: Connect the client to the running container\nclient = EchoEnv(base_url=\"http://0.0.0.0:8001\")\n\"\"\"\n\ndef rollout_func(prompts: list[str], trainer: GRPOTrainer):\n # 1. Generate completions using TRL's helper (works for colocated vLLM)\n outputs = generate_rollout_completions(trainer, prompts)\n tokenizer = trainer.processing_class\n completions_text = [\n tokenizer.decode(out[\"completion_ids\"], skip_special_tokens=True) for out in outputs\n ]\n\n # 2. Step through the environment to get rewards\n client.reset()\n env_rewards = []\n for msg in completions_text:\n env_result = client.step(EchoAction(message=msg))\n env_rewards.append(env_result.reward)\n\n # 3. Add environment rewards as extra field\n return {\n \"prompt_ids\": [out[\"prompt_ids\"] for out in outputs],\n \"completion_ids\": [out[\"completion_ids\"] for out in outputs],\n \"logprobs\": [out[\"logprobs\"] for out in outputs],\n \"env_reward\": env_rewards,\n }\n\ndef reward_from_env(completions, **kwargs):\n \"\"\"Extract environment rewards passed via rollout_func kwargs.\"\"\"\n env_rewards = kwargs.get(\"env_reward\", [])\n return [float(reward) for reward in env_rewards] if env_rewards else [0.0] * len(completions)\n\ndataset = Dataset.from_dict({\"prompt\": [\"You are an AI that interacts with an *Echo* environment. Word to echo:\"] * 64})\n\n# Setup trainer with custom rollout\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n reward_funcs=reward_from_env,\n train_dataset=dataset,\n rollout_func=rollout_func, # Use custom rollout\n args=GRPOConfig(\n use_vllm=True,\n vllm_mode=\"colocate\", # Use colocate mode (default)\n num_train_epochs=1,\n num_generations=8,\n max_completion_length=2048,\n per_device_train_batch_size=8,\n gradient_accumulation_steps=4,\n ),\n)\ntrainer.train()\n```\n\nThat's it! Now that you've seen the full example, let's unpack how the main pieces fit together.\n\n1. **Environment Client:** `EchoEnv` implements an HTTP interface to interact with the environment server.\n2. **Custom rollout:** The `rollout_func` generates completions and steps through the environment to collect rewards.\n3. **Extra fields:** The rollout adds `env_reward` to the result dictionary, which is automatically passed to reward functions.\n4. **Reward function:** Extracts `env_reward` from `kwargs` to apply environment-computed rewards during training.\n\n> [!TIP]\n> The trainer-aware rollout hook works in both vLLM server and colocate modes. Use `trl.experimental.openenv.generate_rollout_completions` so you reuse TRL's sampling configuration automatically.\n\n### Running the Example\n\nYou can run the example in either colocate mode (1 GPU) or server mode (2 GPUs):\n\n\n\n\n\n**Colocate mode (1 GPU, recommended)**\n\n```bash\npython examples/scripts/openenv/echo.py --env-mode space --env-host https://openenv-echo-env.hf.space --vllm-mode colocate\n```\n\nThis runs vLLM in the same process as training, requiring only a single GPU.\n\n\n\n\n\n**Server mode (2+ GPUs, scalable)**\n\n```bash\n# Terminal 1: Start vLLM inference server\nCUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen2.5-0.5B-Instruct --host 0.0.0.0 --port 8000\n\n# Terminal 2: Run GRPO training with OpenEnv\nCUDA_VISIBLE_DEVICES=1 python examples/scripts/openenv/echo.py --env-mode space --env-host https://openenv-echo-env.hf.space --vllm-mode server --vllm-server-url http://localhost:8000\n```\n\nThis runs vLLM as a separate server process, useful when you want to:\n- Share the inference server across multiple training jobs\n- Use multiple GPUs for the vLLM server (via `--tensor-parallel-size`)\n- Scale up training to many GPUs while sharing a single inference endpoint\n\n\n\n\n\nAlternatively, you can manually start the Echo environment in a Docker container before running the training:\n\n```bash\n# Launch the Echo environment\ndocker run -d -p 8001:8001 registry.hf.space/openenv-echo-env:latest\n\n# Run training with docker-local mode\npython examples/scripts/openenv/echo.py --env-mode docker-local --vllm-mode colocate\n```\n\nBelow is the reward curve from training:\n\n\n\n## Advanced Example\n\nLet's level this up a bit by training a model to interact with a more complex environment. We'll use the game word guessing game [wordle](https://www.nytimes.com/games/wordle/index.html) from the [`TextArena`](https://meta-pytorch.org/OpenEnv/environments/textarena/) environment.\n\n> [!NOTE] \n> You can explore the notebook version of this example [here](https://github.com/huggingface/trl/blob/main/examples/notebooks/openenv_wordle_grpo.ipynb).\n\n### The TextArena Environment\n\n[TextArena](https://huggingface.co/papers/2504.11442) is an open-source collection of competitive text-based games designed to evaluate reasoning skills in LLMs using textual games like Wordle, Snake, Tic-Tac-Toe, and more. Research has shown that such games improve model performance on reasoning tasks.\n\n![image of TextArena](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/text_arena_evals.png)\n\nWe will use the `TextArena` environment to train a model to play Wordle. The environment is a simple text based response environment that allows the model to interact with the game by making guesses and receive feedback on them.\n\n### Wordle\n\nWordle is a useful game to train a model on because it requires the model to reason about the word and the feedback provided by the environment. Also, it is a purely language based game that requires no external tools or knowledge. Furthermore, we found that models from 1 billion parameters and up are able to improve on wordle and only require 8 tokens to generate a guess, which makes the game a good benchmark to experiment with Reinforcement Learning environments without significant compute requirements.\n\n> [!NOTE] How does Wordle work?\n> Wordle is a word guessing game where the player has to guess a 5-letter word. The player can make 6 guesses, and for each guess, the environment will provide feedback on the correctness of the guess. The player wins if they guess the word in 6 guesses or fewer. It challenges the model to generate words that are likely to be correct, and to learn from the feedback provided by the environment. \n> \n> For example, if the wordle environment returns the following feedback:\n>\n> ```\n> G U E S S\n> X G Y X X\n> ```\n> The model has guessed the word \"GUESS\" and the environment has provided feedback as the letters X, G, and Y. Referring to colors in the original game as blank, green, and yellow. From this feedback, the model should learn that the word \"GUESS\" is incorrect. The letter \"E\" is in the word, but in the wrong position. The letter \"U\" is correct and in the correct position.\n \nIn the TextArena environment, a reward is only given when the model wins the game. The reward is 1.0 if the model wins, and 0.0 otherwise. This is not a very efficient reward signal for the model, so we have added a number of custom reward functions to the script to help the model learn to play the game. The extensible nature of `reward_funcs` and `rollout_func` allows you to add any custom reward function you want to the script. \n\n### Rollout Function\n\nThe rollout function runs one full Wordle episode, prompting the model for a guess each turn and capturing both environment rewards and auxiliary signals such as letter coverage and repetition penalties.\n\n```python\ndef rollout_once(\n trainer: GRPOTrainer,\n env: TextArenaEnv,\n tokenizer: AutoTokenizer,\n dataset_prompt: str,\n system_prompt: str,\n max_turns: int,\n) -> dict[str, list]:\n result = env.reset()\n observation = result.observation\n\n prompt_ids: list[int] = []\n completion_ids: list[int] = []\n logprobs: list[float] = []\n raw_rewards: list[float] = []\n green_scores: list[float] = []\n yellow_scores: list[float] = []\n repetition_scores: list[float] = []\n correct_scores: list[float] = []\n guess_counts: dict[str, int] = {}\n\n for _turn in range(max_turns):\n # when the game is over the environment will return a done=True\n if result.done:\n break\n\n # set up the prompt for the model\n base_prompt = observation.prompt or dataset_prompt\n user_prompt = make_user_prompt(base_prompt, observation.messages)\n messages = [\n {\"role\": \"system\", \"content\": system_prompt},\n {\"role\": \"user\", \"content\": user_prompt},\n ]\n prompt_text = tokenizer.apply_chat_template(\n messages,\n add_generation_prompt=True,\n tokenize=False,\n enable_thinking=False,\n )\n\n # Generate completion using trainer (works for both colocate and server modes)\n rollout_outputs = generate_rollout_completions(trainer, [prompt_text])[0]\n prompt_ids.extend(rollout_outputs[\"prompt_ids\"])\n completion_ids.extend(rollout_outputs[\"completion_ids\"])\n logprobs.extend(rollout_outputs[\"logprobs\"])\n completion_text = rollout_outputs.get(\"text\") or tokenizer.decode(\n rollout_outputs[\"completion_ids\"], skip_special_tokens=True\n )\n\n # extract the guess from the completion\n guess = extract_guess(completion_text)\n\n # step the environment with the guess\n result = env.step(TextArenaAction(message=guess))\n raw_rewards.append(float(result.reward or 0.0))\n observation = result.observation\n correct_score = float(result.reward or 0.0)\n feedback = extract_wordle_feedback(observation)\n\n # Update guess counts\n previous_occurrences = guess_counts.get(guess, 0)\n repetition_score = scale_repetition_score(previous_occurrences, len(guess_counts))\n guess_counts[guess] = previous_occurrences + 1\n\n # calculate custom reward signals from the feedback\n if not feedback:\n green_score = 0.0\n yellow_score = 0.0\n else:\n green_count, yellow_count = extract_feedback_counts(feedback)\n green_score = green_count / 5.0\n yellow_score = yellow_count / 5.0\n\n repetition_scores.append(repetition_score)\n green_scores.append(green_score)\n yellow_scores.append(yellow_score)\n correct_scores.append(correct_score)\n\n correct_reward_value = correct_scores[-1] if correct_scores else (raw_rewards[-1] if raw_rewards else 0.0)\n\n return {\n \"prompt_ids\": prompt_ids,\n \"completion_ids\": completion_ids,\n \"logprobs\": logprobs,\n \"raw_rewards\": raw_rewards,\n \"correct_reward\": correct_reward_value,\n \"green_reward\": green_scores[-1] if green_scores else 0.0,\n \"yellow_reward\": yellow_scores[-1] if yellow_scores else 0.0,\n \"repetition_reward\": repetition_scores[-1] if repetition_scores else 0.0,\n }\n```\n\nThe environment has a reward signal based on the completion of the game. We found that most models struggle to ever win the game, so we have added a number of custom reward functions to the script to help the model learn to play the game more iteratively. At first, the model will learn to cover new letters and avoid repeating guesses. As it improves, it will learn to win the game.\n\n### Reward Functions\n\nWe log four reward streams that encourage the model to solve the puzzle, cover new letters, and avoid repeating guesses:\n\n- `reward_correct`: final win/loss signal from the environment.\n- `reward_greens`: density of green letters in the last feedback.\n- `reward_yellows`: density of yellow letters in the last feedback.\n- `reward_repetition`: penalty for guessing the same token multiple times.\n\n```python\ndef reward_correct(completions: List[str], **kwargs: Optional[Dict]) -> List[float]:\n rewards = kwargs.get(\"correct_reward\") if kwargs else None\n return [float(r) for r in rewards] if rewards is not None else [0.0] * len(completions)\n\n\ndef reward_greens(completions: List[str], **kwargs: Optional[Dict]) -> List[float]:\n rewards = kwargs.get(\"green_reward\") if kwargs else None\n return [float(r) for r in rewards] if rewards is not None else [0.0] * len(completions)\n\n\ndef reward_yellows(completions: List[str], **kwargs: Optional[Dict]) -> List[float]:\n rewards = kwargs.get(\"yellow_reward\") if kwargs else None\n return [float(r) for r in rewards] if rewards is not None else [0.0] * len(completions)\n\n\ndef reward_repetition(completions: List[str], **kwargs: Optional[Dict]) -> List[float]:\n rewards = kwargs.get(\"repetition_reward\") if kwargs else None\n return [float(r) for r in rewards] if rewards is not None else [0.0] * len(completions)\n```\n\n### Training the Model\n\nThe training script wires the custom rollout and rewards into `GRPOTrainer`. The CLI exposes the configuration used during development as defaults, so you can override endpoints or hyperparameters at launch time.\n\n```python\nparser = argparse.ArgumentParser()\n# ... add CLI arguments with sensible defaults ...\ncli_args = parser.parse_args()\n\ntrainer = GRPOTrainer(\n model=cli_args.model_id,\n processing_class=tokenizer,\n reward_funcs=[\n reward_correct,\n reward_greens,\n reward_yellows,\n reward_repetition,\n ],\n train_dataset=dataset,\n args=grpo_config,\n rollout_func=lambda prompts, trainer: rollout_func(\n env=env,\n tokenizer=tokenizer,\n prompts=prompts,\n trainer=trainer,\n cli_args=cli_args,\n system_prompt=system_prompt,\n ),\n)\ntrainer.train()\n```\n\n### Running the Advanced Example\n\nYou can run the Wordle example in either colocate mode (1 GPU) or server mode (2 GPUs):\n\n\n\n\n\n**Colocate mode (1 GPU, recommended)**\n\n```bash\npython examples/scripts/openenv/wordle.py --vllm-mode colocate\n```\n\nThis runs vLLM in the same process as training, requiring only a single GPU.\n\n\n\n\n\n**Server mode (2+ GPUs, scalable)**\n\n```bash\n# Terminal 1: Start vLLM inference server\nCUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen3-1.7B --host 0.0.0.0 --port 8000\n\n# Terminal 2: Run GRPO training with OpenEnv\nCUDA_VISIBLE_DEVICES=1 python examples/scripts/openenv/wordle.py --vllm-mode server --vllm-server-url http://localhost:8000\n```\n\nThis runs vLLM as a separate server process, useful when you want to:\n- Share the inference server across multiple training jobs\n- Use multiple GPUs for the vLLM server (via `--tensor-parallel-size`)\n- Scale up training to many GPUs while sharing a single inference endpoint\n\n\n\n\n\nYou can also manually start the TextArena environment in a Docker container before running the training:\n\n```bash\n# Launch the TextArena environment\ndocker run -d -p 8001:8001 registry.hf.space/burtenshaw-textarena:latest\n```\n\nThen connect to it using `--env-mode docker-local--env-host localhost --env-port 8001`.\n\n### Results\n\nThe resulting model improves its performance on the game, both by reducing the number of repetitions and by increasing the number of correct guesses. However, the Qwen3-1.7B model we trained is not able to consistently win the game. The following reward curve shows the coverage of the model's guesses and the coverage of correct Y and G letters.\n\n\n\nWe experimented with larger models like `gpt-oss-20b` and found that the model was able to consistently win the game. However, this requires a lot of compute to train the model. Why not try this out yourself?\n" }, { "path": "docs/source/orpo_trainer.md", "content": "# ORPO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-ORPO-blue)](https://huggingface.co/models?other=orpo,trl) [![model badge](https://img.shields.io/badge/smol_course-Chapter_2-yellow)](https://github.com/huggingface/smol-course/tree/main/2_preference_alignment)\n\n## Overview\n\nOdds Ratio Preference Optimization (ORPO) was introduced in [ORPO: Monolithic Preference Optimization without Reference Model](https://huggingface.co/papers/2403.07691) by [Jiwoo Hong](https://huggingface.co/JW17), [Noah Lee](https://huggingface.co/nlee-208), and [James Thorne](https://huggingface.co/j6mes).\n\nThe abstract from the paper is the following:\n\n> While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we study the crucial role of SFT within the context of preference alignment, emphasizing that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT. Building on this foundation, we introduce a straightforward and innovative reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for an additional preference alignment phase. We demonstrate, both empirically and theoretically, that the odds ratio is a sensible choice for contrasting favored and disfavored styles during SFT across the diverse sizes from 125M to 7B. Specifically, fine-tuning Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) with ORPO on the UltraFeedback alone surpasses the performance of state-of-the-art language models with more than 7B and 13B parameters: achieving up to 12.20% on AlpacaEval_{2.0} (Figure 1), 66.19% on IFEval (instruction-level loose, Table 6), and 7.32 in MT-Bench (Figure 12). We release code and model checkpoints for Mistral-ORPO-alpha (7B) and Mistral-ORPO-beta (7B).\n\nIt studies the crucial role of SFT within the context of preference alignment. Using preference data the method posits that a minor penalty for the disfavored generation together with a strong adaption signal to the chosen response via a simple log odds ratio term appended to the NLL loss is sufficient for preference-aligned SFT.\n\nThus ORPO is a reference model-free preference optimization algorithm eliminating the necessity for an additional preference alignment phase thus saving compute and memory.\n\nThe official code can be found in [xfactlab/orpo](https://github.com/xfactlab/orpo).\n\nThis post-training method was contributed by [Kashif Rasul](https://huggingface.co/kashif), [Lewis Tunstall](https://huggingface.co/lewtun) and [Alvaro Bartolome](https://huggingface.co/alvarobartt).\n\n## Quick start\n\nThis example demonstrates how to train a model using the ORPO method. We use the [Qwen 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) as the base model. We use the preference data from the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback). You can view the data in the dataset here:\n\n\n\nBelow is the script to train the model:\n\n```python\n# train_orpo.py\nfrom datasets import load_dataset\nfrom trl.experimental.orpo import ORPOConfig, ORPOTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\ntrain_dataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\ntraining_args = ORPOConfig(output_dir=\"Qwen2-0.5B-ORPO\")\ntrainer = ORPOTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset)\ntrainer.train()\n```\n\nExecute the script using the following command:\n\n```bash\naccelerate launch train_orpo.py\n```\n\nDistributed across 8 GPUs, the training takes approximately 30 minutes. You can verify the training progress by checking the reward graph. An increasing trend in the reward margin indicates that the model is improving and generating better responses over time.\n\n![orpo qwen2 reward margin](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/orpo-qwen2-reward-margin.png)\n\nTo see how the [trained model](https://huggingface.co/trl-lib/Qwen2-0.5B-ORPO) performs, you can use the [Transformers Chat CLI](https://huggingface.co/docs/transformers/quicktour#chat-with-text-generation-models).\n\n
$ transformers chat trl-lib/Qwen2-0.5B-ORPO\n<quentin_gallouedec>:\nWhat is the best programming language?\n\n<trl-lib/Qwen2-0.5B-ORPO>:\nIt's challenging to determine the best programming language as no one language is perfect, as the complexity of a task and the type of project are significant factors. Some popular languages include Java, Python, JavaScript, and\nC++. If you have specific needs or requirements for a specific project, it's important to choose the language that best suits those needs.\n\nHere are some other factors to consider when choosing a programming language for a project:\n\n โ€ข Language proficiency: A good programming language is more likely to be easy to understand and use, and will allow developers to collaborate on projects more efficiently.\n โ€ข Ease of use: There are tools and libraries available to make programming more accessible, so developers should choose a language that can help them get started easier.\n โ€ข Code readability: A clear and concise codebase should be easy to read and understand, especially when working with large projects.\n โ€ข Tool and framework support: There are numerous libraries available for Python, Java, and JavaScript, along with tools like IDEs and static code analysis tools.\n โ€ข Accessibility: Some languages and tools have features that make them more accessible to developers with disabilities, such as support for screen readers.\n โ€ข Version control: As your projects grow and complexity increases, version control tools can be beneficial for tracking changes.\n\n
\n\n## Expected dataset type\n\nORPO requires a [preference dataset](dataset_formats#preference). The [`experimental.orpo.ORPOTrainer`] supports both [conversational](dataset_formats#conversational) and [standard](dataset_formats#standard) dataset format. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\nAlthough the [`experimental.orpo.ORPOTrainer`] supports both explicit and implicit prompts, we recommend using explicit prompts. If provided with an implicit prompt dataset, the trainer will automatically extract the prompt from the `\"chosen\"` and `\"rejected\"` columns. For more information, refer to the [preference style](dataset_formats#preference) section.\n\n## Example script\n\nWe provide an example script to train a model using the ORPO method. The script is available in [`examples/scripts/orpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/orpo.py)\n\nTo test the ORPO script with the [Qwen2 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) on the [UltraFeedback dataset](https://huggingface.co/datasets/trl-lib/ultrafeedback_binarized), run the following command:\n\n```bash\naccelerate launch examples/scripts/orpo.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --num_train_epochs 1 \\\n --output_dir Qwen2-0.5B-ORPO\n```\n\n## Usage tips\n\n### For Mixture of Experts Models: Enabling the auxiliary loss\n\nMOEs are the most efficient if the load is about equally distributed between experts. \nTo ensure that we train MOEs similarly during preference-tuning, it is beneficial to add the auxiliary loss from the load balancer to the final loss.\n\nThis option is enabled by setting `output_router_logits=True` in the model config (e.g. [`~transformers.MixtralConfig`]). \nTo scale how much the auxiliary loss contributes to the total loss, use the hyperparameter `router_aux_loss_coef=...` (default: `0.001`) in the model config.\n\n## Logged metrics\n\nWhile training and evaluating, we record the following reward metrics:\n\n- `rewards/chosen`: the mean log probabilities of the policy model for the chosen responses scaled by beta\n- `rewards/rejected`: the mean log probabilities of the policy model for the rejected responses scaled by beta\n- `rewards/accuracies`: mean of how often the chosen rewards are > than the corresponding rejected rewards\n- `rewards/margins`: the mean difference between the chosen and corresponding rejected rewards\n- `log_odds_chosen`: the mean log odds ratio of the chosen responses over the rejected responses\n- `log_odds_ratio`: the mean of the `log(sigmoid(log_odds_chosen))`\n- `nll_loss`: the mean negative log likelihood loss from the SFT part of the loss over chosen responses\n\n## ORPOTrainer\n\n[[autodoc]] experimental.orpo.ORPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## ORPOConfig\n\n[[autodoc]] experimental.orpo.ORPOConfig\n" }, { "path": "docs/source/paper_index.md", "content": "# Paper Index\n\n\n\n## Group Relative Policy Optimization\n\nPapers relating to the [`GRPOTrainer`].\n\n### DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2402.03300\n\nIntroduces Group Relative Policy Optimization (GRPO) and shows strong math-reasoning gains from math-centric pretraining plus group-relative PPO-style optimization. Used in TRL via [`GRPOTrainer`].\n\n```python\nfrom trl import GRPOConfig, GRPOTrainer\n\n# The paper doesn't specify its hyperparameters, so here we provide hyperparameters from \"DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning\" instead.\ntraining_args = GRPOConfig(\n loss_type=\"grpo\",\n beta=0.001, # \"the KL coefficient to 0.001\"\n epsilon=10.0, # \"the GRPO clip ratio ฯต to 10\"\n num_generations=16, # \"For each question, we sample 16 outputs...\"\n max_completion_length=32_768, # \"...with a maximum length of 32,768\"\n steps_per_generation=16, # \"To accelerate training, each rollout generates 8,192 outputs, which are randomly split into 16 minibatches\"\n # \"resulting in a training batch size of 512\". One way to achieve this setting with 1 device is per_device_train_batch_size=4, gradient_accumulation_steps=128\n per_device_train_batch_size=4,\n gradient_accumulation_steps=128, \n)\ntrainer = GRPOTrainer(\n ...,\n args=training_args,\n)\n```\n\n### DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2501.12948\n\nDeepSeek-R1 achieves reasoning performance comparable to OpenAI-o1 through a multi-stage pipeline that transitions from pure reinforcement learning (RL) to a refined, human-aligned model. Unlike its predecessor, DeepSeek-R1-Zero, which used pure RL on a base model, R1 follows a structured four-stage evolution:\n1. Cold Start: The base model is fine-tuned on a small set of high-quality, long Chain-of-Thought (CoT) data to provide a stable starting point.\n2. Reasoning-Oriented RL: Large-scale RL is applied to enhance performance in math, coding, and logic, using rule-based rewards and a language consistency reward to reduce language mixing.\n3. Rejection Sampling & SFT: The RL checkpoint generates 600k reasoning samples via rejection sampling, which are combined with 200k non-reasoning (general) samples to create a new dataset for a second round of Supervised Fine-Tuning.\n4. RL for all Scenarios: A final RL stage aligns the model with human preferences (helpfulness and harmlessness) across all domains while maintaining reasoning strength.\n\nDistillation: Empowering Small Models\n\nA key contribution of the paper is demonstrating that reasoning patterns can be distilled from a large model (DeepSeek-R1) into smaller dense models (e.g., Qwen and Llama series). Distillation was found to be more effective for small models than training them with pure RL from scratch.\n\n\nYou can use the GRPOTrainer to replicate the reasoning-heavy stages of this pipeline. \n```python\nfrom trl import GRPOConfig, GRPOTrainer\n\n# Example configuration for a reasoning-oriented GRPO stage\n# Based on the Open-R1 recipe for Qwen-7B\ntraining_args = GRPOConfig(\n learning_rate=4.0e-5,\n max_prompt_length=4096,\n max_completion_length=32768, # Support for long Chain-of-Thought\n num_generations=16, # Sample 16 outputs per prompt for group relative advantage\n beta=0.001, # KL coefficient\n use_vllm=True, # Use vLLM backend for accelerated rollout generation\n)\n\ntrainer = GRPOTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset,\n reward_funcs=[accuracy_reward, format_reward], # R1-Zero used rule-based rewards\n)\n\ntrainer.train()\n```\n\n\n### Group Sequence Policy Optimization\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2507.18071\n\nGSPO is a GRPO variant that computes importance sampling weights at the sequence level instead of per-token. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n importance_sampling_level=\"sequence\",\n loss_type=\"grpo\",\n beta=0.0, # GSPO set KL regularization to zero: https://github.com/volcengine/verl/pull/2775#issuecomment-3131807306 \n epsilon=3e-4, # GSPO paper (v2), section 5.1\n epsilon_high=4e-4, # GSPO paper (v2), section 5.1\n gradient_accumulation_steps=1,\n steps_per_generation=4, # partition rollout batch into 4 mini-batches. GSPO paper (v2), section 5.1. Must be 4 times gradient_accumulation_steps\n)\n```\n\nNote that this method only has an effect when training goes slightly off-policyโ€”for example, when `steps_per_generation > gradient_accumulation_steps` or `num_iterations > 1`. Otherwise, it is effectively equivalent to no modification.\n\nTRL also provide an experimental implementation of GSPO-token, see [Experimental - GSPO-Token](experimental#gspo-token).\n\n#### Policy ratio: GRPO vs. GSPO\n\nIn GSPO, the policy ratio is defined at the sequence-level. In other words, it is the ratio between the probability of the current policy generating a sequence over the old policy generating that same sequence.\n\nThe sequence likelihood is defined as:\n\n$$\n\\pi_\\theta (o_i | q) = \\prod_{t=1}^{|o_i|} \\pi_\\theta (o_{i,t} | q, o_{i, < t} ),\n$$\n\nwhere \\\\( \\pi_\\theta \\\\) is the policy \\\\( \\pi \\\\) with parameters \\\\(\\theta\\\\), \\\\( o_i \\\\) is the \\\\( i \\\\)-th output sequence \\\\( o \\\\) and \\\\(o_{i,t}\\\\) is the \\\\( t \\\\)-th token in this sequence, \\\\( q \\\\) is the input query. The sequence likelihood ratio \\\\( s_i (\\theta) \\\\) is defined as:\n\n$$\ns_i (\\theta) = \\left(\\frac{\\pi_\\theta (o_i | q)}{\\pi_{\\theta_{old}} (o_i | q)} \\right)^{\\frac{1}{|o_i|}}\n$$\n\nThe exponent \\\\( \\frac{1}{|o_i|} \\\\) represents a sequence-length normalization, minimizing the influence of sequence length in sequence likelihood. In other terms, it computes the geometric mean of token probabilities, ensuring a fair comparison across sequences of varying lengths.\n\nWhile GSPO defines the policy ratio at the sequence level, GRPO operates at the token level. Specifically, GRPO computes an importance ratio for each token in the sequence:\n\n$$\nw_{i,t}(\\theta) = \\frac{\\pi_\\theta (o_{i,t} | q, o_{i,< t})}{\\pi_{\\theta_{\\text{old}}} (o_{i,t} | q, o_{i,< t})}\n$$\n\nThis token-level ratio is then combined with a shared advantage \\\\( \\hat{A}_i \\\\), and the GRPO objective clips and optimizes each token independently across the sequence.\n\n### DAPO: An Open-Source LLM Reinforcement Learning System at Scale\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2503.14476\n\nThe DAPO algorithm includes 5 key components:\n\n- Overlong Filtering\n- Clip-Higher\n- Soft Overlong Punishment\n- Token-level Loss\n- Dynamic Sampling (โš ๏ธ Not supported in TRL)\n\nTo reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import GRPOConfig, GRPOTrainer\n\ntraining_args = GRPOConfig(\n # Overlong Filtering\n mask_truncated_completions=True,\n # Token-level Loss\n loss_type=\"dapo\",\n # Clip-Higher\n epsilon_high=0.28, # DAPO paper: section 4.1\n epsilon=0.2, # DAPO paper: section 4.1\n # Other parameters used\n per_device_train_batch_size=512, # mini-batch size for training in the paper, DAPO paper: section 4.1\n num_generations=16, # number of sample responses in the paper, DAPO paper: section 4.1\n max_completion_length=20480, # maximum number of tokens for generation in the paper, DAPO paper: section 4.1\n beta=0.0, # section 2.3, DAPO paper\n\n)\n# Soft Overlong Punishment\nsop_reward = get_soft_overlong_punishment(max_completion_len=20480, soft_punish_cache=4096) # DAPO paper: section 4.1\ntrainer = GRPOTrainer(\n ...,\n args=training_args,\n reward_funcs=[..., sop_reward],\n)\n```\n\n### INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2505.07291\n\nINTELLECT-2 is the first globally distributed reinforcement learning training run of a 32 billion parameter language model using fully asynchronous RL across a dynamic, heterogeneous swarm of permissionless compute contributors. The authors propose modifications to the standard GRPO training recipe, including two-sided GRPO clipping for increased training stability. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n delta=4, # ฮด in section 4.1 of the paper\n epsilon=0.2, # ฮต in section 4.1 of the paper\n beta=0.001, # KL divergence coefficient in section 4.1 of the paper\n num_generations=16, # responses per prompt in section 4.1 of the paper\n learning_rate=3e-7, # section 4.1 of the paper\n)\n```\n\n### Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2506.01939\n\nA minority of tokens with high entropy act as reasoning \"forks\" in the CoT path, driving exploration and performance gains for RLVR, while low-entropy majority tokens contribute little or even impede learning. RLVR mainly adjusts high-entropy tokens, largely preserving the base modelโ€™s overall entropy patterns. Thus landing on the 80/20 rule, training on only 20% of the tokens with the highest entropy is comparable or supasses full-gradient updates for Qwen3 models.\n\nThe paper's main results use vanilla DAPO (โš ๏ธ Dynamic Sampling is not supported in TRL). To replicate the main results, use the following configuration:\n\n```python\nfrom trl import GRPOConfig, GRPOTrainer\nfrom trl.rewards import get_soft_overlong_punishment\n\ntraining_args = GRPOConfig(\n # --- vanilla DAPO parameters (80/20 rule: section 5.2) --- #\n # Overlong Filtering\n mask_truncated_completions=True,\n # Token-level Loss\n loss_type=\"dapo\",\n # Clip-Higher\n epsilon_high=0.28, # DAPO paper: section 4.1\n epsilon=0.2, # DAPO paper: section 4.1\n # Other parameters used\n per_device_train_batch_size=512, # mini-batch size for training in the paper, DAPO paper: section 4.1\n num_generations=16, # number of sample responses in the paper, DAPO paper: section 4.1\n max_completion_length=20480, # maximum number of tokens for generation in the paper, DAPO paper: section 4.1\n beta=0.0, # section 2.3, DAPO paper\n # --- Gradients on the highest entropy tokens --- #\n top_entropy_quantile=0.2\n)\n# Soft Overlong Punishment\nsop_reward = get_soft_overlong_punishment(max_completion_len=20480, soft_punish_cache=4096) # DAPO paper: section 4.1\ntrainer = GRPOTrainer(\n ...,\n args=training_args,\n reward_funcs=[..., sop_reward],\n)\n```\n\n### Dr. GRPO: Understanding R1-Zero-Like Training: A Critical Perspective\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2503.20783\n\nA study of R1-Zero training identifies pretraining effects on RL performance and proffers Dr. GRPO to enhance token efficiency, achieving superior accuracy on AIME 2024. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n loss_type=\"dr_grpo\",\n per_device_train_batch_size=1, # train_batch_size_per_device in the Training section of the repository\n num_generations=8, # num_samples in the Training section of the repository\n max_completion_length=3000, # generate_max_length in the Training section of the repository\n beta=0.0, # ฮฒ in the Training section of the repository\n)\n```\n\n### Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning (Lite PPO)\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2508.08221\n\nThe authors of this paper find that the combination of:\n\n1. scaling rewards by the standard deviation computed over the entire batch and\n2. aggregating loss over the total number of tokens\n\ncan unlock the learning capability of critic-free policies using vanilla PPO loss. Their results demonstrate that this simple combination consistently improves performance, surpassing strategies like GRPO and [DAPO](https://huggingface.co/papers/2503.14476).\n\nTRL supports using these learnings to train a GRPO model by:\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n ...\n scale_rewards=\"batch\",\n loss_type=\"dapo\",\n # Other parameters used\n beta=0.0, # = init_kl_coef in the paper\n top_p=0.99,\n top_k=100,\n temperature=0.99,\n num_generations=8, # = num_return_sequences in the paper\n num_iterations=1, # = ppo_epochs in the paper\n per_device_train_batch_size=4,\n gradient_accumulation_steps=32,\n steps_per_generation=8, # (rollout_batch_size*num_return_sequences) / (per_device_train_batch_size*gradient_accumulation_steps)\n)\n```\n\nNote that when using gradient accumulation, the loss is aggregated over the total number of tokens in the batch, but not over the accumulated batch. For more details, see the [GRPO Trainer - Loss types](grpo_trainer#loss_types).\n\n### Truncated Importance Sampling\n\n**๐Ÿ“ฐ Blog**: https://fengyao.notion.site/off-policy-rl\n\nOnline policy learning methods commonly use an optimized inference framework for rollout generation (e.g vLLM) that is separate from the training backend. This introduces a rollout-training mismatch, exemplified in the following PPO objective:\n\n$$\n\\small{\n\\mathbb{E}_{a\\sim\\textcolor{red}{\\pi_{\\text{inference}}}(\\theta_{\\mathrm{old}})}\n\\Bigl[\n\\min\\Bigl(\n\\frac{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta)}{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta_{\\mathrm{old}})}\\,\\hat A,\n\\;\\mathrm{clip}\\bigl(\\frac{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta)}{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta_{\\mathrm{old}})},\\,1-\\epsilon,\\,1+\\epsilon\\bigr)\\,\\hat A\n\\Bigr)\n\\Bigr]\n}\n$$\n\nDespite \\\\( \\textcolor{red}{\\pi_{\\text{inference}}} \\\\) and \\\\( \\textcolor{blue}{\\pi_{\\text{training}}} \\\\) sharing the same model parameters \\\\( \\theta \\\\), they can produce significantly different token probabilities. This unexpected behavior implicitly breaks the on-policy assumption, and silently turns training off-policy.\n\nTruncated Importance Sampling (TIS) addresses this issue by adapting the model update via importance-sampling correction. The gradient computation of the aforementioned PPO objective becomes\n\n$$\n\\small{\n\\mathbb{E}_{a\\sim\\textcolor{red}{\\pi_{\\text{inference}}}(\\theta_{\\mathrm{old}})}\n\\Bigl[\n\\underbrace{\\min(\\frac{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta_{\\mathrm{old}})}{\\textcolor{red}{\\pi_{\\text{inference}}}(a, \\theta_{\\mathrm{old}})}, C)}_{\\text{truncated importance ratio}} \\cdot\n\\nabla_\\theta\n\\min\\Bigl(\n\\frac{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta)}{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta_{\\mathrm{old}})}\\,\\hat A,\n\\;\\mathrm{clip}\\bigl(\\frac{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta)}{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta_{\\mathrm{old}})},\\,1-\\epsilon,\\,1+\\epsilon\\bigr)\\,\\hat A\n\\Bigr)\n\\Bigr]\n}\n$$\n\nwhere \\\\( C \\\\) is a hyper-parameter. TIS is implemented in GRPO, and is enabled by selecting a `vllm_importance_sampling_mode` variant that includes the term `truncate`, such as `\"sequence_truncate\"` or `\"token_truncate\"`.\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n ...\n use_vllm=True,\n vllm_importance_sampling_correction=True, # default True\n vllm_importance_sampling_mode=\"sequence_truncate\", # or \"token_truncate\"\n vllm_importance_sampling_cap=2.0, # hyper-parameter C\n)\n```\n\n### Masked Importance Sampling\n\n**๐Ÿ“ฐ Blog**: https://ringtech.notion.site/icepop\n\n**๐Ÿ“ฐ Blog**: https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda\n\nMasked Importance Sampling (MIS) addresses the same issue as [Truncated Importance Sampling](#truncated-importance-sampling) but replaces clipping with masking. MIS takes a more decisive stance by discarding updates whose discrepancy exceeds a threshold \\\\( C \\\\). We apply upper-side masking, so any ratio above \\\\( C \\\\) is removed from the update.\n\n\n$$\n\\small{\n\\mathbb{E}_{a\\sim\\textcolor{red}{\\pi_{\\text{inference}}}(\\theta_{\\mathrm{old}})}\n\\Bigl[\n\\underbrace{\\mathbf{1}\\left[\n\\frac{\\pi_{\\text{training}}(a, \\theta_{\\mathrm{old}})}\n{\\pi_{\\text{inference}}(a, \\theta_{\\mathrm{old}})}\n\\le C\n\\right]\n\\cdot\n\\frac{\\pi_{\\text{training}}(a, \\theta_{\\mathrm{old}})}\n{\\pi_{\\text{inference}}(a, \\theta_{\\mathrm{old}})}}_{\\text{masked importance ratio}} \\cdot\n\\nabla_\\theta\n\\min\\Bigl(\n\\frac{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta)}{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta_{\\mathrm{old}})}\\,\\hat A,\n\\;\\mathrm{clip}\\bigl(\\frac{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta)}{\\textcolor{blue}{\\pi_{\\text{training}}}(a, \\theta_{\\mathrm{old}})},\\,1-\\epsilon,\\,1+\\epsilon\\bigr)\\,\\hat A\n\\Bigr)\n\\Bigr]\n}\n$$\n\nMIS is implemented for GRPO, and is enabled by selecting a `vllm_importance_sampling_mode` variant that includes the term `\"mask\"`, such as `\"sequence_mask\"` or `\"token_mask\"`.\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n ...\n use_vllm=True,\n vllm_importance_sampling_correction=True, # default True\n vllm_importance_sampling_mode=\"sequence_mask\", # or \"token_mask\"\n vllm_importance_sampling_cap=2.0, # hyper-parameter C\n)\n```\n\n### Sequence-level Importance Sampling\n\n**๐Ÿ“ฐ Blog**: https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda\n\nThe theoretically principled way to correct for the training-inference distribution shift is importance sampling, as introduced in the two papers above [Truncated Importance Sampling](#truncated-importance-sampling) and [Masked Importance Sampling](#masked-importance-sampling). However, the choice of formulation is crucial for keeping the gradient unbiased and ensuring stable training.\n\nThis work shows that sequence-level importance sampling is the sound approach for addressing the trainingโ€“inference mismatch. Although token-level importance sampling achieves lower variance than a sequence-level ratio, it introduces bias and is therefore argued to be unsuitable for autoregressive models. The token-level gradient estimator is\n\n$$\n\\mathbb{E}_{x\\sim\\mathcal{D},\\, y\\sim \\pi^{\\text{inference}}_\\theta(\\cdot|x)}\n\\Bigg[\n R(x,y)\\,\\cdot\\,\n \\sum_{t=0}^{|y|-1}\n \\frac{\\pi^{\\text{training}}_\\theta(y_t\\,|\\,x, y_{ 0`:\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n ...,\n beta=0.001, # the paper doesn't specify the value used, so we use the value from \"DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning\"\n use_bias_correction_kl=True,\n)\n```\n\n- The **Off-Policy Masking**, which stabilizes training by ignoring sequences where the policy performs poorly (negative advantage) **and** has drifted significantly from the old policy (high KL divergence).\n\nThe off-policy binary mask \\\\(\\textcolor{red}{M_{i,t}}\\\\) is defined as:\n\n$$\n\\textcolor{red}{M_{i,t}} = \\begin{cases}\n0 & \\text{if } \\hat{A}_{i,t} < 0 \\quad \\text{and} \\quad \\frac{1}{|o_i|} \\sum_{t=1}^{|o_i|} \\log \\frac{\\pi_{\\theta_{\\text{old}}}(o_{i,t} \\mid q, o_{i, \\textcolor{blue}{\\delta} \\\\\n1 & \\text{otherwise}\n\\end{cases}\n$$\n\nThis mask is then applied to the GRPO loss as follows:\n\n$$\n\\mathcal{L}_{\\text{GRPO}}(\\theta) = -\\frac{1}{G} \\sum_{i=1}^G \\frac{1}{|o_i|} \\sum_{t=1}^{|o_i|} \\left[ \\min \\left( \\frac{\\pi_\\theta(o_{i,t} \\mid q, o_{i,< t})}{\\pi_{\\theta_{\\text{old}}}(o_{i,t} \\mid q, o_{i,< t})} \\hat{A}_{i,t}, \\, \\text{clip}\\left( \\frac{\\pi_\\theta(o_{i,t} \\mid q, o_{i,< t})}{\\pi_{\\theta_{\\text{old}}}(o_{i,t} \\mid q, o_{i,< t})}, 1 - \\epsilon, 1 + \\epsilon \\right) \\hat{A}_{i,t} \\right) \\textcolor{red}{M_{i,t}} - \\beta \\mathbb{D}_{\\text{KL}}\\left[\\pi_\\theta \\| \\pi_{\\text{ref}}\\right] \\right]\n$$\n\nTo enable this feature, use the `off_policy_mask_threshold` (corresponding to \\\\( \\textcolor{blue}{\\delta} \\\\)) in the [`GRPOConfig`]:\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n ...,\n off_policy_mask_threshold=0.5, \n)\n```\n\nWhile the paper doesn't specify a \\\\( \\textcolor{blue}{\\delta} \\\\) value used, a good starting point could be \\\\( \\textcolor{blue}{\\delta} = 0.5 \\\\). If training seems too conservative or too many sequences are masked, you can increase the value.\nFor reference, \\\\( \\textcolor{blue}{\\delta} = 1.0 \\\\) corresponds to an average log-ratio divergence of 1 nat per token, i.e. on sequences where this threshold is exceeded, the old policy was on average \\\\( e^1 \\approx 2.7 \\\\) times more likely to generate these tokens than the current policy.\n\n### GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2601.05242\n\nGDPO is a reinforcement learning optimization method designed for multi-reward training. While existing approaches commonly apply Group Relative Policy Optimization (GRPO) in multi-reward settings, the authors show that this leads to reward advantages collapse, reducing training signal resolution and causing unstable or failed convergence. GDPO resolves this issue by decoupling reward normalization across individual rewards, preserving their relative differences and enabling more faithful preference optimization. To enable GDPO for multi-reward RL training, simply set:\n\nFor a group of \\\\( N \\\\) rewards and \\\\( G \\\\) samples per group, GDPO normalizes each reward independently:\n\n$$\nA_n^{(i,j)} = \\frac{r_n^{(i,j)} - \\text{mean}\\{r_n^{(i,1)}, \\ldots, r_n^{(i,G)}\\}}{\\text{std}\\{r_n^{(i,1)}, \\ldots, r_n^{(i,G)}\\} + \\epsilon}\n$$\n\nThe normalized group advantage is then aggregated across rewards:\n\n$$\nA^{(i,j)} = \\sum_{n=1}^{N} w_n A_n^{(i,j)}\n$$\n\nThe final per-batch normalization produces:\n\n$$\n\\hat{A}^{(i,j)} = \\frac{A^{(i,j)} - \\text{mean}_{i',j'}\\{A^{(i',j')}\\}}{\\text{std}_{i',j'}\\{A^{(i',j')}\\} + \\epsilon}\n$$\n\nHere, \\\\( \\text{mean}_{i',j'}\\{A^{(i',j')}\\} \\\\) and \\\\( \\text{std}_{i',j'}\\{A^{(i',j')}\\} \\\\) denote statistics over all groups in the batch.\n\n```python\nfrom trl import GRPOConfig\n\n\ntraining_args = GRPOConfig(\n ...,\n multi_objective_aggregation=\"normalize_then_sum\",\n)\n```\n\nNote that this method only has an effect when training involve more than one reward function.\n\nThe authors provide a easy-to-use, slurm-free training example that enable the community to quickly validate GDPOโ€™s effectiveness over GRPO, see [Experiment-\"Aha\" moment](https://github.com/NVlabs/GDPO/tree/main/trl-GDPO).\n\n### Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2602.05261\n\nLength-Unbiased Sequence Policy Optimization (LUSPO) modifies GSPO by scaling each sequence's loss by its length. This corrects GSPO's gradient bias that penalizes longer responses. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n loss_type=\"luspo\",\n importance_sampling_level=\"sequence\",\n epsilon=2e-3, # section 5.1 of the paper\n epsilon_high=2.5e-3, # section 5.1 of the paper\n)\n```\n\n### VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2602.10693\n\nVESPO addresses training instability in off-policy RL caused by policy staleness, asynchronous updates, and train-inference mismatches. Rather than relying on heuristic token-level clipping (GRPO) or sequence-length normalization (GSPO), VESPO derives a principled reshaping kernel from a variational framework. In practice, this yields a smooth, asymmetric Gamma weighting function that gracefully suppresses extreme sequence-level importance weights without introducing length bias.\n\n$$\n\\mathcal{L}_{\\text{VESPO}}(\\theta) = - \\mathbb{E}_{\\tau \\sim \\mu} \\left[ \\underbrace{W(\\tau)^{k} \\cdot \\exp\\left(\\lambda\n(1 - W(\\tau))\\right)}_{\\phi(W) \\text{ detached }} \\cdot \\mathcal{A}(\\tau) \\cdot \\log \\pi_\\theta(\\tau) \\right]\n$$\n\nwith \\\\( W(\\tau) = \\frac{\\pi_\\theta(\\tau)}{\\mu(\\tau)} \\\\) the sequence level importance ratio, and \\\\( \\phi(W) \\\\) is detached from the computation graph to serve as a gradient scaling coefficient.\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n loss_type=\"vespo\",\n use_vllm=True, # or False if not using any token-level `vllm_importance_sampling_correction` methods\n vllm_importance_sampling_mode=\"token_truncate\", # default correction mode for VESPO, `token_mask` also supported\n vespo_k_pos=2.0, # power exponent (c1 in paper Section 3.4) for positive advantages\n vespo_lambda_pos=3.0, # decay factor (c2 in paper Section 3.4) for positive advantages\n vespo_k_neg=3.0, # power exponent (c1 in paper Section 3.4) for negative advantages\n vespo_lambda_neg=2.0, # decay factor (c2 in paper Section 3.4) for negative advantages\n)\n```\n\n\n### Rethinking the Trust Region in LLM Reinforcement Learning\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2602.04879\n\nDPPO replaces PPO/GRPO's heuristic ratio-clipping with a principled trust region based on direct policy divergence estimates. PPO-style clipping masks tokens based on the probability ratio ฯ€/ฮผ, which over-penalizes low-probability tokens and under-penalizes high-probability ones. DPPO instead masks based on direct approximations of policy divergence (TV or KL), ensuring updates stay within a theoretically grounded trust region. Four divergence approximations are supported: `binary_tv`, `binary_kl`, `topk_tv`, and `topk_kl`.\n\n```python\nfrom trl.experimental.dppo import DPPOConfig, DPPOTrainer\n\ntraining_args = DPPOConfig(\n divergence_type=\"binary_tv\", # divergence approximation\n divergence_topk=20, # K for top-K divergence modes (Section 7 / Appendix G.2 of the paper)\n epsilon=0.15, # ฮด_low threshold (Appendix F of the paper)\n epsilon_high=0.15, # ฮด_high threshold (Appendix F of the paper)\n clip_ratio_c=20.0, # IS ratio upper bound C (Section 5.4 of the paper)\n beta=0.0, # KL regularization coefficient\n use_vllm=True,\n)\n\ntrainer = DPPOTrainer(\n model=\"your-model\",\n reward_funcs=[...],\n args=training_args,\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\nThe official code [sail-sg/Stable-RL](https://github.com/sail-sg/Stable-RL)\n\n## Direct Policy Optimization\n\nPapers relating to the [`DPOTrainer`]\n\n### Direct Preference Optimization: Your Language Model is Secretly a Reward Model\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2305.18290\n\nDirect Preference Optimization (DPO) fine-tunes language models more efficiently and with better performance compared to reinforcement learning from human feedback (RLHF), by directly optimizing policy training based on human preferences. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"sigmoid\", # losses in Appendix B of the paper\n per_device_train_batch_size=64, # batch size in Appendix B of the paper\n learning_rate=1e-6, # learning rate in Appendix B of the paper\n beta=0.1, # ฮฒ in Appendix B of the paper\n)\n```\n\n### SLiC-HF: Sequence Likelihood Calibration with Human Feedback\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2305.10425\n\nSequence Likelihood Calibration (SLiC) is shown to be an effective and simpler alternative to Reinforcement Learning from Human Feedback (RLHF) for learning from human preferences in language models. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"hinge\", # Section 2 of the paper\n per_device_train_batch_size=512, # batch size in Section 3.2 of the paper\n learning_rate=1e-4, # learning rate in Section 3.2 of the paper\n)\n```\n\nThese parameters only appear in the [published version](https://openreview.net/pdf?id=0qSOodKmJaN)\n\n### Statistical Rejection Sampling Improves Preference Optimization\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2309.06657\n\nProposes **RSO**, selecting stronger preference pairs via statistical rejection sampling to boost offline preference optimization; complements DPO/SLiC. They also introduce a new loss defined as:\n\n$$\n\\mathcal{L}_{\\text{hinge-norm}}(\\pi_\\theta)\n= \\mathbb{E}_{(x, y_w, y_l) \\sim \\mathcal{D}}\n\\left[\n\\max\\left(0,\\; 1 - \\left[\\gamma \\log \\frac{\\pi_\\theta(y_w \\mid x)}{\\pi_\\text{ref}(y_w \\mid x)} - \\gamma \\log \\frac{\\pi_\\theta(y_l \\mid x)}{\\pi_\\text{ref}(y_l \\mid x)}\\right]\\right)\n\\right]\n$$\n\nTo train with RSO-filtered data and the hinge-norm loss, you can use the following code:\n\n```python\nfrom trl import DPOConfig, DPOTrainer\n\ndataset = ...\n\ndef rso_accept(example): # replace with your actual filter/score logic\n return example[\"rso_keep\"]\n\ntrain_dataset = train_dataset.filter(rso_accept)\n\ntraining_args = DPOConfig(\n loss_type=\"hinge\",\n beta=0.05, # correspond to ฮณ in the paper\n)\n\ntrainer = DPOTrainer(\n ...,\n args=training_args,\n train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n### Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2309.16240\n\nProposes \\(( f \\\\)-DPO, extending DPO by replacing the usual reverse-KL regularizer with a general \\(( f \\\\)-divergence, letting you trade off mode-seeking vs mass-covering behavior (e.g. forward KL, JS, \\(( \\alpha \\\\)-divergences). The only change is replacing the DPO log-ratio margin with an **fโ€ฒ score**:\n\n$$\n\\mathcal{L}_{f\\text{-DPO}}(\\pi_\\theta)\n= \\mathbb{E}_{(x, y_w, y_l) \\sim \\mathcal{D}}\n\\left[\n-\\log \\sigma\\left(\n\\beta \\textcolor{red}{f'}\\textcolor{red}{\\Big(}\\frac{\\pi_\\theta(y_w|x)}{\\pi_{\\text{ref}}(y_w|x)}\\textcolor{red}{\\Big)}\n-\n\\beta \\textcolor{red}{f'}\\textcolor{red}{\\Big(}\\frac{\\pi_\\theta(y_l|x)}{\\pi_{\\text{ref}}(y_l|x)}\\textcolor{red}{\\Big)}\n\\right)\n\\right]\n$$\n\nWhere \\\\( f' \\\\) is the derivative of the convex function defining the chosen \\(( f \\\\)-divergence.\n\nTo reproduce:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"sigmoid\",\n beta=0.1,\n f_divergence_type=\"js_divergence\", # or \"reverse_kl\" (default), \"forward_kl\", \"js_divergence\", \"alpha_divergence\"\n f_alpha_divergence_coef=0.5, # only used if f_divergence_type=\"alpha_divergence\"\n)\n```\n\n### A General Theoretical Paradigm to Understand Learning from Human Preferences\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2310.12036\n\nLearning from human preferences can be written as a single KL-regularized objective over pairwise preference probabilities,\n\n$$\n\\max_\\pi ;\\mathbb{E}\\big[\\Psi\\left(p^*(y \\succ y' \\mid x)\\right)\\big] - \\tau\\mathrm{KL}(\\pi||\\pi_{\\text{ref}}),\n$$\n\nwhich reveals RLHF and DPO as special cases corresponding to the logit choice of \\\\( \\Psi \\\\).\nThe paper shows that this logit transform amplifies near-deterministic preferences and effectively weakens KL regularization, explaining overfitting.\nUsing the **Identity transform (IPO)** avoids this pathology by optimizing preferences directly, without assuming a Bradleyโ€“Terry reward model.\nTo reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"ipo\", # Section 5.1 of the paper\n per_device_train_batch_size=90, # mini-batch size in Section C.1 of the paper\n learning_rate=1e-2, # learning rate in Section C.1 of the paper\n)\n```\n\nThese parameters only appear in the [published version](https://proceedings.mlr.press/v238/gheshlaghi-azar24a/gheshlaghi-azar24a.pdf)\n\n### Towards Efficient and Exact Optimization of Language Model Alignment\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2402.00856\n\nThe paper shows that direct preference methods like DPO optimize the wrong KL direction, leading to blurred preference capture, and proposes EXO as an efficient way to exactly optimize the humanโ€‘preference alignment objective by leveraging reverse KL probability matching rather than forward KL approximations. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"exo_pair\", # Section 3.2 of the paper\n # From Section B of the paper\n per_device_train_batch_size=64,\n learning_rate=1e-6,\n beta=0.1,\n)\n```\n\n### Noise Contrastive Alignment of Language Models with Explicit Rewards\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2402.05369\n\nThe paper reframes language-model alignment as a *noise-contrastive classification* problem, proposing InfoNCA to learn a policy from explicit rewards (or preferences) by matching a reward-induced target distribution over responses, and showing DPO is a special binary case. It then introduces NCA, which adds an absolute likelihood term to prevent the likelihood collapse seen in purely relative (contrastive) objectives.\n\nWith pairwise preferences, treat the chosen/rejected \\\\( K=2 \\\\), define scores \\\\( r=\\beta(\\log\\pi_\\theta-\\log\\pi_{\\text{ref}}) \\\\), and apply the NCA preference loss \\\\( -\\log\\sigma(r_w)-\\tfrac12\\log\\sigma(-r_w)-\\tfrac12\\log\\sigma(-r_l) \\\\).\n\nTo reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"nca_pair\",\n # From Section C of the paper\n per_device_train_batch_size=32,\n learning_rate=5e-6,\n beta=0.01,\n)\n```\n\n### Provably Robust DPO: Aligning Language Models with Noisy Feedback\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2403.00409\n\nDPO breaks under noisy human preferences because label flips bias the objective. Robust DPO fixes this by analytically debiasing the DPO loss under a simple noise model, with provable guarantees.\n\n$$\n\\mathcal{L}_{\\text{robust}}(\\pi_\\theta) = \\frac{(1-\\varepsilon)\\mathcal{L}_{\\text{DPO}}(y_w, y_l) - \\varepsilon\\mathcal{L}_{\\text{DPO}}(y_l, y_w)}\n{1-2\\varepsilon}\n$$\n\nWhere \\\\( \\mathcal{L}_{\\text{DPO}} \\\\) is the DPO loss defined in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](#direct-preference-optimization-your-language-model-is-secretly-a-reward-model) and \\\\( \\varepsilon \\\\) is the probability of a label flip.\n\nThis single correction turns noisy preference data into an unbiased estimator of the clean DPO objective.\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"robust\",\n per_device_train_batch_size=16, # batch size in Section B of the paper\n learning_rate=1e-3, # learning rate in Section B of the paper\n beta=0.1, # ฮฒ in Section B of the paper,\n max_length=512, # max length in Section B of the paper\n label_smoothing=0.1 # label smoothing $\\varepsilon$ in Section 6 of the paper\n)\n```\n\n### Binary Classifier Optimization for Large Language Model Alignment\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2404.04656\n\nTheoretical analysis and a new algorithm, Binary Classifier Optimization, explain and enhance the alignment of large language models using binary feedback signals. To reproduce the paper's setting, use this configuration:\n\nBCO reframes language-model alignment as behavioral cloning from an optimal reward-weighted distribution, yielding simple supervised objectives that avoid RL while remaining theoretically grounded.\nIt supports both unpaired reward data and pairwise preference data, with a reward-shiftโ€“invariant formulation that reduces to a DPO-style loss in the preference setting.\n\nFor the pairwise preference setting, the BCO loss is defined as:\n\n$$\n\\mathcal{L}_{\\text{bco\\_pair}}(\\pi_\\theta) =\n\\mathbb{E}_{(x, y_w, y_l) \\sim \\mathcal{D}}\n\\left[\n-\\log \\sigma\\Big(\n\\beta[(\\log\\pi_\\theta-\\log\\pi_{\\text{ref}})(y_w)\n-\n(\\log\\pi_\\theta-\\log\\pi_{\\text{ref}})(y_l)]\n\\Big)\n\\right]\n$$\n\nTo reproduce the paper in this setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"bco_pair\",\n # From Section C of the paper\n per_device_train_batch_size=128,\n learning_rate=5e-7,\n beta=0.01,\n)\n```\n\nFor the unpaired version, the user should utilize [`experimental.bco.BCOConfig`] and [`experimental.bco.BCOTrainer`].\n\n### Learn Your Reference Model for Real Good Alignment\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2404.09656\n\nTrust Region DPO (TR-DPO) updates the reference policy during training, demonstrating effectiveness against DPO on the Anthropic HH and TLDR datasets, outperforming DPO by up to 19% measured by automatic evaluation with GPT-4, improving coherence, correctness, level of detail, helpfulness, and harmlessness. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n sync_ref_model=True, # enable TR-DPO (Section 3 of the paper)\n ref_model_mixup_alpha=0.6, # ฮฑ soft update weight (Table 1 of the paper)\n ref_model_sync_steps=512, # ฯ„ update frequency in steps (Table 1 of the paper)\n beta=0.05, # ฮฒ temperature (Table 1 of the paper)\n learning_rate=1e-6, # learning rate (Table 2 of the paper)\n num_train_epochs=1, # Table 2 of the paper\n max_length=1024, # max tokens length (Table 2 of the paper)\n max_grad_norm=2, # max gradient norm (Table 2 of the paper)\n warmup_steps=100, # warm-up steps (Table 2 of the paper)\n)\n```\n\n### Iterative Reasoning Preference Optimization\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2404.19733\n\nIterative RPO improves reasoning by repeatedly generating chain-of-thought candidates, building preference pairs from correct vs. incorrect answers, and training with a DPO + NLL objective. The extra NLL term is key for learning to actually generate winning traces.\n\nTRL can express the DPO + NLL objective by mixing `\"sigmoid\"` (DPO) with `\"sft\"` (NLL):\n\n```python\nfrom trl import DPOConfig, DPOTrainer\n\ntraining_args = DPOConfig(\n loss_type=[\"sigmoid\", \"sft\"],\n loss_weights=[1.0, 1.0], # alpha in the paper, recommended value is 1.0\n)\ntrainer = DPOTrainer(\n ...,\n args=training_args,\n)\n```\n\nNote that the paper uses an iterative loop: each iteration regenerates CoT candidates with the current model, then retrains on fresh preference pairs. TRL does not automate that loop for you.\n\n### Self-Play Preference Optimization for Language Model Alignment\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2405.00675\n\nA self-play method called SPPO for language model alignment achieves state-of-the-art performance by approximating Nash equilibrium policy in a constant-sum game setting, outperforming other approaches with limited data. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"sppo_hard\",\n # From Section 5 of the paper\n beta=0.001, # ฮฒ = ฮท^โˆ’1\n per_device_train_batch_size=64,\n learning_rate=5e-7,\n)\n```\n\n### Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2405.16436\n\nRegularized Preference Optimization (RPO) mitigates overoptimization in RLHF by fusing the DPO loss with the SFT loss, provably preventing the policy from choosing actions with spurious high proxy rewards. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=[\"sigmoid\", \"sft\"], # RPO loss = DPO + SFT (Section 5 of the paper)\n loss_weights=[1.0, 0.005], # ฮท=0.005 SFT weight in Appendix E.1 of the paper\n beta=0.01, # ฮฒ in Appendix E.1 of the paper\n learning_rate=5e-7, # learning rate in Appendix E.1 of the paper\n num_train_epochs=1, # Appendix E.1 of the paper\n)\n```\n\n### Distributional Preference Alignment of LLMs via Optimal Transport\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2406.05882\n\nAlignment via Optimal Transport (AOT) aligns large language models distributionally by penalizing violations of stochastic dominance between positive and negative sample distributions, achieving state-of-the-art performance on alignment benchmarks. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"aot\",\n beta=0.01, # from the caption of Figure 2\n)\n```\n\nor, for the unpaired version:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"aot_unpaired\",\n beta=0.01, # from the caption of Figure 2\n)\n```\n\nThere is no additional hyperparameter in the paper.\n\n### Discovering Preference Optimization Algorithms with and for Large Language Models\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2406.08414\n\nAn LLM-driven method automatically discovers performant preference optimization algorithms, leading to a new algorithm called DiscoPOP that blends logistic and exponential losses. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(\n loss_type=\"discopop\",\n per_device_train_batch_size=64, # batch size in Section B.1 of the paper\n learning_rate=5e-7, # learning rate in Section B.1 of the paper\n beta=0.05, # ฮฒ in Section B.1 of the paper,\n discopop_tau=0.05 # ฯ„ in Section E of the paper\n)\n```\n\n### WPO: Enhancing RLHF with Weighted Preference Optimization\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2406.11827\n\nWPO reweights preference pairs by their policy probabilities to reduce the off-policy gap in DPO-style training. The loss is:\n\n$$\n\\mathcal{L}_{\\text{WPO}} = -\\mathbb{E}_{(x, y_w, y_l) \\sim \\mathcal{D}} \\left[ \\textcolor{red}{w(x, y_w) w(x, y_l)} \\log p(y_w \\succ y_l \\mid x) \\right]\n$$\n\nwhere the weight \\\\( w(x, y) \\\\) is defined as:\n\n$$\nw(x, y) = \\exp\\left(\\frac{1}{|y|}\\sum_{t=1}^{|y|} \\log \\frac{\\pi_\\theta(y_t \\mid x, y_{ 0 (optimism coefficient) and ฮฒ > 0 (KL regularization) in Algorithm 1 but does not specify numerical values. The following configuration uses TRL defaults:\n\n```python\nfrom trl.experimental.xpo import XPOConfig\n\ntraining_args = XPOConfig(\n alpha=1e-5, # ฮฑ exploration bonus weight, ฮฑ โ‰ฅ 0 where ฮฑ=0 reduces to online DPO (TRL default)\n beta=0.1, # ฮฒ KL regularization coefficient (TRL default)\n)\n```\n\n## Distillation\n\nPapers relating to training a student model with the help of a teacher model.\n\n### On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2306.13649\n\nIntroduces Generalized Knowledge Distillation (GKD), which addresses distribution mismatch in KD for auto-regressive models by training the student on its own generated outputs with teacher feedback, instead of a fixed set of sequences. GKD supports flexible loss functions (e.g. beyond KL when the student cannot match the teacher) and integrates with RL fine-tuning (RLHF). The paper reports results on summarization, translation, arithmetic reasoning, and instruction-tuning. Used in TRL via [`experimental.gkd.GKDTrainer`]. To reproduce the paper's setting, use this configuration:\n\n```python\nfrom trl.experimental.gkd import GKDConfig\n\n# XSum summarization task (Table A.1 of the paper)\ntraining_args = GKDConfig(\n lmbda=0.5, # ฮป student data fraction (Section 3 of the paper)\n beta=0.5, # ฮฒ Generalized JSD interpolation, 0=KL, 1=reverse KL (Section 3 of the paper)\n temperature=1.0, # student training temperature (Appendix A of the paper)\n max_steps=40000, # training steps (Table A.1 of the paper)\n learning_rate=3e-4, # learning rate (Table A.1 of the paper)\n per_device_train_batch_size=32, # batch size (Table A.1 of the paper)\n warmup_steps=2000, # warm-up steps (Table A.1 of the paper)\n max_new_tokens=64, # max output tokens (Table A.1 of the paper)\n)\n```\n\n### On-Policy Distillation\n\n**๐Ÿ“ฐ Blog**: https://thinkingmachines.ai/blog/on-policy-distillation/\n\nOn-Policy Distillation involves a student model generating rollouts for each batch of training data. We subsequently obtain the probability distributions for each token of the rollouts from both the student and teacher models. The student model is then optimized to minimize the negative Kullback-Leibler (KL) divergence between its own token distributions and those of the teacher model.\n\n| Method | Sampling | Reward signal |\n|-------------------------|------------|---------------|\n| Supervised finetuning | off-policy | dense |\n| Reinforcement learning | on-policy | sparse |\n| On-policy distillation | on-policy | dense |\n\nOn-Policy Distillation has been shown to outperform SFT, GRPO and can be used to restore generalization capabilities lost during SFT.\n\nAdditionally on-policy distillation is more compute efficient and is less prone to overfitting when trained with limited data.\n\nTo train a model with on-policy distillation using TRL, you can use the following configuration, with the [`experimental.gkd.GKDTrainer`] and [`experimental.gkd.GKDConfig`]:\n\n```python\nfrom trl.experimental.gkd import GKDConfig\n\ntraining_args = GKDConfig(\n lmbda=1.0, # student produces rollouts for all batches\n beta=1.0, # to ensure reverse-kl as the loss function\n teacher_model_name_or_path=\"teacher-model\", # specify the teacher model\n\n)\n```\n\nAlternatively, you can use the [`GOLDTrainer`] and [`GOLDConfig`] to perform on-policy distillation with a similar configuration:\n\n```python\nfrom trl.experimental import GOLDConfig\n\nconfig = GOLDConfig(\n lmbda=1.0, # student produces rollouts for all batches\n beta=1.0, # to ensure reverse-kl as the loss function\n teacher_model_name_or_path=\"teacher-model\", # specify the teacher model\n\n)\n```\n\n### Knowledge Distillation of Large Language Models\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/2306.08543\n\nMiniLLM is the first on-policy knowledge distillation method, which minimizes the sequence-level reverse KLD between the teacher and the student model and is optimized by reinforcement learning.\n\nIt is a generalized version of [Think Machine Lab's On-Policy Distillation](https://thinkingmachines.ai/blog/on-policy-distillation/), with the option to add distribution-level single-step distillation signals (like GKD when `beta=1`) and long-context reverse KLD signals.\n\nAlternatively, you can use the [`experimental.MiniLLMTrainer`] and [`experimental.MiniLLMConfig`] to perform MiniLLM distillation as follows:\n\n```python\nfrom datasets import load_dataset\nfrom trl.experimental.minillm import MiniLLMTrainer\n\ndataset = load_dataset(\"trl-lib/tldr\", split=\"train\")\n\ntrainer = MiniLLMTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n teacher_model=\"Qwen/Qwen3-1.7B\",\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\nFor more details, see the [MiniLLM Trainer documentation](minillm) documentation.\n\n## Distributed Training\n\n### ZeRO: Memory Optimizations Toward Training Trillion Parameter Models\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/1910.02054\n\nZeRO (Zero Redundancy Optimizer) eliminates memory redundancies in data- and model-parallel training by partitioning optimizer states, gradients, and parameters across devices while retaining low communication volume and high computational granularity. This allows for the efficient training of large models that would otherwise not fit in GPU memory.\n\nTRL supports ZeRO via the [DeepSpeed integration](deepspeed_integration). To use it, provide a DeepSpeed configuration file with your desired settings,\n\n```yaml\n# config.yaml\ndistributed_type: DEEPSPEED\nnum_processes: 2\ndeepspeed_config:\n zero_stage: 3\n```\n\nand launch the training script using `accelerate launch --config_file config_file`.\n\n```sh\naccelerate launch --config_file config.yaml train.py\n```\n\n## Proximal Policy Optimization\n\nPapers relating to the [`experimental.ppo.PPOTrainer`]\n\n### Proximal Policy Optimization Algorithms\n\n**๐Ÿ“œ Paper**: https://huggingface.co/papers/1707.06347\n\nIntroduces Proximal Policy Optimization (PPO): policy gradient methods that alternate between collecting rollouts and optimizing a clipped surrogate objective over multiple minibatch epochs. PPO retains benefits of trust-region methods (e.g. TRPO) with simpler implementation and strong empirical sample efficiency, and was validated on robotics and Atari benchmarks. Used in TRL via [`experimental.ppo.PPOTrainer`]. To use PPO with TRL, use this configuration:\n\n```python\nfrom trl.experimental.ppo import PPOConfig\n\ntraining_args = PPOConfig(\n cliprange=0.2, # ฮต clipping range (Section 3 and Table 3 of the paper, Mujoco setting)\n num_ppo_epochs=4, # K epochs of minibatch updates (TRL default; paper uses K=10 Mujoco, K=3 Atari)\n gamma=1.0, # ฮณ discount factor (TRL default for LLM tasks; paper uses ฮณ=0.99)\n lam=0.95, # ฮป GAE parameter (Table 3 of the paper, Mujoco setting)\n kl_coef=0.05, # KL penalty coefficient (Section 4 of the paper discusses adaptive KL)\n vf_coef=0.1, # cโ‚ value function loss weight (Equation 9 of the paper)\n)\n```\n" }, { "path": "docs/source/papo_trainer.md", "content": "# PAPO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-PAPO-blue)](https://huggingface.co/models?other=papo,trl)\n\nTRL supports the Perception-Aware Policy Optimization (PAPO) as described in the paper [Perception-Aware Policy Optimization for Multimodal Reasoning](https://huggingface.co/papers/2507.06448) by [Zhenhailong Wang](https://huggingface.co/mikewang), Xuehang Guo, Sofia Stoica, [Haiyang Xu](https://huggingface.co/xhyandwyy), Hongru Wang, Hyeonjeong Ha, Xiusi Chen, Yangyi Chen, Ming Yan, Fei Huang, Heng Ji\n\nThe abstract from the paper is the following:\n\n> Reinforcement Learning with Verifiable Rewards (RLVR) has proven to be a highly effective strategy for endowing Large Language Models (LLMs) with robust multi-step reasoning abilities. However, its design and optimizations remain tailored to purely textual domains, resulting in suboptimal performance when applied to multimodal reasoning tasks. In particular, we observe that a major source of error in current multimodal reasoning lies in the perception of visual inputs. To address this bottleneck, we propose Perception-Aware Policy Optimization (PAPO), a simple yet effective extension of GRPO that encourages the model to learn to perceive while learning to reason, entirely from internal supervision signals. Notably, PAPO does not rely on additional data curation, external reward models, or proprietary models. Specifically, we introduce the Implicit Perception Loss in the form of a KL divergence term to the GRPO objective, which, despite its simplicity, yields significant overall improvements (4.4%) on diverse multimodal benchmarks. The improvements are more pronounced, approaching 8.0%, on tasks with high vision dependency. We also observe a substantial reduction (30.5%) in perception errors, indicating improved perceptual capabilities with PAPO. We conduct comprehensive analysis of PAPO and identify a unique loss hacking issue, which we rigorously analyze and mitigate through a Double Entropy Loss. Overall, our work introduces a deeper integration of perception-aware supervision into RLVR learning objectives and lays the groundwork for a new RL framework that encourages visually grounded reasoning. Project page: https://mikewangwzhl.github.io/PAPO.\n\n## PAPOTrainer\n\n[[autodoc]] experimental.papo.PAPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## PAPOConfig\n\n[[autodoc]] experimental.papo.PAPOConfig\n" }, { "path": "docs/source/peft_integration.md", "content": "# PEFT Integration\n\nTRL supports [PEFT](https://github.com/huggingface/peft) (Parameter-Efficient Fine-Tuning) methods for memory-efficient model training. PEFT enables fine-tuning large language models by training only a small number of additional parameters while keeping the base model frozen, significantly reducing computational costs and memory requirements.\n\nThis guide covers how to use PEFT with different TRL trainers, including LoRA, QLoRA, and prompt tuning techniques.\n\nFor a complete working example, see the [SFT with LoRA/QLoRA notebook](https://github.com/huggingface/trl/blob/main/examples/notebooks/sft_trl_lora_qlora.ipynb).\n\n## Installation\n\nTo use PEFT with TRL, install the required dependencies:\n\n```bash\npip install trl[peft]\n```\n\nFor QLoRA support (4-bit and 8-bit quantization), also install:\n\n```bash\npip install bitsandbytes\n```\n\n## Quick Start\n\nAll TRL trainers support PEFT through the `peft_config` argument. The simplest way to enable PEFT is by using the command-line interface with the `--use_peft` flag:\n\n```bash\npython trl/scripts/sft.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --output_dir Qwen2-0.5B-SFT-LoRA\n```\n\nAlternatively, you can pass a PEFT config directly in your Python code:\n\n```python\nfrom peft import LoraConfig\nfrom trl import SFTTrainer\n\n# Configure LoRA\npeft_config = LoraConfig(\n r=32,\n lora_alpha=16,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n)\n\n# Configure training - note the higher learning rate for LoRA (10x base rate)\ntraining_args = SFTConfig(\n learning_rate=2.0e-4, # 10x the base rate (2.0e-5) for LoRA\n ...\n)\n\n# Create trainer with PEFT\ntrainer = SFTTrainer(\n model=model,\n train_dataset=dataset,\n peft_config=peft_config,\n)\n```\n\n## Three Ways to Configure PEFT\n\nTRL provides three different methods to configure PEFT, each suited for different use cases:\n\n### 1. Using CLI Flags (Simplest)\n\nThe easiest way to enable PEFT is to use the `--use_peft` flag with the command-line interface. This method is ideal for quick experiments and standard configurations:\n\n```bash\npython trl/scripts/sft.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --lora_dropout 0.05 \\\n --output_dir Qwen2-0.5B-SFT-LoRA\n```\n\n**Pros**: Quick setup, no code required\n\n**Cons**: Limited to LoRA, fewer customization options\n\n### 2. Passing peft_config to Trainer (Recommended)\n\nFor more control, pass a PEFT configuration directly to the trainer. This is the recommended approach for most use cases:\n\n```python\nfrom peft import LoraConfig\nfrom trl import SFTConfig, SFTTrainer\n\npeft_config = LoraConfig(\n r=32,\n lora_alpha=16,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n target_modules=[\"q_proj\", \"v_proj\", \"k_proj\", \"o_proj\"],\n)\n\ntrainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset,\n peft_config=peft_config, # Pass config here\n)\n```\n\n**Pros**: Full control, supports all PEFT methods (LoRA, Prompt Tuning, etc.)\n\n**Cons**: Requires Python code\n\n### 3. Applying PEFT to Model Directly (Advanced)\n\nFor maximum flexibility, you can apply PEFT to your model before passing it to the trainer:\n\n```python\nfrom peft import LoraConfig, get_peft_model\nfrom transformers import AutoModelForCausalLM\nfrom trl import SFTConfig, SFTTrainer\n\n# Load base model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-0.5B\")\n\n# Apply PEFT configuration\npeft_config = LoraConfig(\n r=32,\n lora_alpha=16,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n)\nmodel = get_peft_model(model, peft_config)\n\n# Pass PEFT-wrapped model to trainer\ntrainer = SFTTrainer(\n model=model, # Already has PEFT applied\n args=training_args,\n train_dataset=dataset,\n # Note: no peft_config needed here\n)\n```\n\n**Pros**: Maximum control, useful for custom model architectures or complex setups\n\n**Cons**: More verbose, requires understanding of PEFT internals\n\n## Learning Rate Considerations\n\nWhen using LoRA or other PEFT methods, you typically need to use a **higher learning rate** (approximately 10x) compared to full fine-tuning. This is because PEFT methods train only a small fraction of parameters, requiring a larger learning rate to achieve similar parameter updates.\n\n**Recommended learning rates:**\n\n| Trainer | Full Fine-Tuning | With LoRA (10x) |\n|---------|------------------|-----------------|\n| **SFT** | `2.0e-5` | `2.0e-4` |\n| **DPO** | `5.0e-7` | `5.0e-6` |\n| **GRPO** | `1.0e-6` | `1.0e-5` |\n| **Prompt Tuning** | N/A | `1.0e-2` to `3.0e-2` |\n\n> **Why 10x?** LoRA adapters have significantly fewer trainable parameters than the full model. A higher learning rate compensates for this reduced parameter count, ensuring effective training. For detailed explanation, see [this blog post](https://thinkingmachines.ai/blog/lora/).\n\nFor additional best practices on using LoRA effectively, refer to the [LoRA Without Regret](lora_without_regret) documentation.\n\n## PEFT with Different Trainers\n\nTRL's trainers support PEFT configurations for various training paradigms. Below are detailed examples for each major trainer.\n\n\n\n\n### Supervised Fine-Tuning (SFT)\n\nThe `SFTTrainer` is used for supervised fine-tuning on instruction datasets.\n\n#### With LoRA\n\n```bash\npython trl/scripts/sft.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --learning_rate 2.0e-4 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --output_dir Qwen2-0.5B-SFT-LoRA\n```\n\n#### Python Example\n\n```python\nfrom peft import LoraConfig\nfrom trl import SFTConfig, SFTTrainer\n\n# Configure LoRA\npeft_config = LoraConfig(\n r=32,\n lora_alpha=16,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n target_modules=[\"q_proj\", \"v_proj\"], # optional: specify target modules\n)\n\n# Configure training with higher learning rate for LoRA\ntraining_args = SFTConfig(\n learning_rate=2.0e-4, # 10x the base rate for LoRA\n ...\n)\n\n# Create trainer with PEFT config\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen2-0.5B\", # can pass model name or loaded model\n args=training_args,\n train_dataset=dataset,\n peft_config=peft_config, # pass PEFT config here\n)\ntrainer.train()\n```\n\n\n\n\n### Direct Preference Optimization (DPO)\n\nThe [`DPOTrainer`] implements preference learning from human feedback.\n\n#### With LoRA\n\n```bash\npython trl/scripts/dpo.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --learning_rate 5.0e-6 \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --output_dir Qwen2-0.5B-DPO-LoRA\n```\n\n#### Python Example\n\n```python\nfrom peft import LoraConfig\nfrom trl import DPOConfig, DPOTrainer\n\n# Configure LoRA\npeft_config = LoraConfig(\n r=32,\n lora_alpha=16,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n)\n\n# Configure training with higher learning rate for LoRA\ntraining_args = DPOConfig(\n learning_rate=5.0e-6, # 10x the base rate for DPO with LoRA\n ...\n)\n\n# Create trainer with PEFT config\ntrainer = DPOTrainer(\n model=\"Qwen/Qwen2-0.5B\", # can pass model name or loaded model\n args=training_args,\n train_dataset=dataset,\n peft_config=peft_config, # pass PEFT config here\n)\ntrainer.train()\n```\n\n**Note:** When using PEFT with DPO, you don't need to provide a separate reference model (`ref_model`). The trainer automatically uses the frozen base model as the reference.\n\n\n\n\n### Group Relative Policy Optimization (GRPO)\n\nThe `GRPOTrainer` optimizes policies using group-based rewards.\n\n#### With LoRA\n\n```bash\npython trl/scripts/grpo.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/math-reasoning \\\n --learning_rate 1.0e-5 \\\n --per_device_train_batch_size 2 \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --output_dir Qwen2-0.5B-GRPO-LoRA\n```\n\n#### Python Example\n\n```python\nfrom peft import LoraConfig\nfrom trl import GRPOConfig, GRPOTrainer\n\n# Configure LoRA\npeft_config = LoraConfig(\n r=32,\n lora_alpha=16,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n)\n\n# Configure training with higher learning rate for LoRA\ntraining_args = GRPOConfig(\n learning_rate=1.0e-5, # 10x the base rate for GRPO with LoRA\n ...\n)\n\n# Create trainer with PEFT config\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen2-0.5B\", # can pass model name or loaded model\n args=training_args,\n train_dataset=dataset,\n peft_config=peft_config, # pass PEFT config here\n)\ntrainer.train()\n```\n\n\n\n\n### Proximal Policy Optimization (PPO)\n\n#### Multi-Adapter RL Training\n\nYou can use a single base model with multiple PEFT adapters for the entire PPO algorithm - including retrieving reference logits, computing active logits, and calculating rewards. This approach is useful for memory-efficient RL training.\n\n> [!WARNING]\n> This feature is experimental and convergence has not been extensively tested. We encourage the community to share feedback and report any issues.\n\n**Requirements**\n\nInstall PEFT and optionally bitsandbytes for 8-bit models:\n\n```bash\npip install peft bitsandbytes\n```\n\n**Training Workflow**\n\nThe multi-adapter approach requires three stages:\n\n1. **Supervised Fine-Tuning (SFT)**: Train a base model on your target domain (e.g., IMDB dataset) using `SFTTrainer`\n2. **Reward Model Training**: Train a reward model adapter using PEFT and `RewardTrainer` (see [reward modeling example](https://github.com/huggingface/trl/tree/main/examples/scripts/reward_modeling.py))\n3. **PPO Training**: Fine-tune new adapters using PPO with the reward adapter\n\n> [!IMPORTANT]\n> Use the same base model (architecture and weights) for stages 2 & 3.\n\n**Basic Usage**\n\nAfter training your reward adapter and pushing it to the Hub:\n\n```python\nfrom peft import LoraConfig\nfrom trl.experimental.ppo import PPOTrainer, AutoModelForCausalLMWithValueHead\n\nmodel_name = \"huggyllama/llama-7b\"\nrm_adapter_id = \"trl-lib/llama-7b-hh-rm-adapter\"\n\n# Configure PPO adapter\nlora_config = LoraConfig(\n r=16,\n lora_alpha=32,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n)\n\n# Load model with reward adapter\nmodel = AutoModelForCausalLMWithValueHead.from_pretrained(\n model_name,\n peft_config=lora_config,\n reward_adapter=rm_adapter_id,\n)\n\ntrainer = PPOTrainer(model=model, ...)\n```\n\nIn your training loop, compute rewards using:\n\n```python\nrewards = trainer.model.compute_reward_score(**inputs)\n```\n\n**Advanced Features**\n\n**Quantized Base Models**\n\nFor memory-efficient training, load the base model in 8-bit or 4-bit while keeping adapters in float32:\n\n```python\nfrom transformers import BitsAndBytesConfig\n\nmodel = AutoModelForCausalLMWithValueHead.from_pretrained(\n model_name,\n peft_config=lora_config,\n reward_adapter=rm_adapter_id,\n quantization_config=BitsAndBytesConfig(load_in_8bit=True),\n)\n```\n\n## QLoRA: Quantized Low-Rank Adaptation\n\nQLoRA combines 4-bit quantization with LoRA to enable fine-tuning of very large models on consumer hardware. This technique can reduce memory requirements by up to 4x compared to standard LoRA.\n\n### How QLoRA Works\n\n1. **4-bit Quantization**: The base model is loaded in 4-bit precision using `bitsandbytes`\n2. **Frozen Weights**: The quantized model weights remain frozen during training\n3. **LoRA Adapters**: Only the LoRA adapter parameters are trained in higher precision\n4. **Memory Efficiency**: Enables fine-tuning of models like Llama-70B on a single consumer GPU\n\n### Using QLoRA with TRL\n\nSimply combine `load_in_4bit=True` with PEFT configuration:\n\n#### Command Line\n\n```bash\npython trl/scripts/sft.py \\\n --model_name_or_path meta-llama/Llama-2-7b-hf \\\n --dataset_name trl-lib/Capybara \\\n --load_in_4bit \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 16 \\\n --output_dir Llama-2-7b-QLoRA\n```\n\n#### Python Example\n\n```python\nimport torch\n\nfrom peft import LoraConfig\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nfrom trl import SFTConfig, SFTTrainer\n\n# Configure 4-bit quantization\nbnb_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_compute_dtype=torch.bfloat16,\n bnb_4bit_use_double_quant=True,\n)\n\n# Load model with quantization\nmodel = AutoModelForCausalLM.from_pretrained(\n \"meta-llama/Llama-2-7b-hf\",\n quantization_config=bnb_config,\n device_map=\"auto\",\n)\n\n# Configure LoRA\npeft_config = LoraConfig(\n r=32,\n lora_alpha=16,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n)\n\n# Configure training with higher learning rate for LoRA\ntraining_args = SFTConfig(\n learning_rate=2.0e-4, # 10x the base rate for QLoRA\n ...\n)\n\n# Create trainer with PEFT config\ntrainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset,\n peft_config=peft_config,\n)\n\ntrainer.train()\n```\n\n### QLoRA Configuration Options\n\nThe `BitsAndBytesConfig` provides several options to optimize memory and performance:\n\n```python\nimport torch\n\nfrom transformers import BitsAndBytesConfig\n\nbnb_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_quant_type=\"nf4\", # or \"fp4\"\n bnb_4bit_compute_dtype=torch.bfloat16, # Compute dtype for 4-bit base models\n bnb_4bit_use_double_quant=True, # Nested quantization for additional memory savings\n)\n```\n\n**Configuration Parameters:**\n- `bnb_4bit_quant_type`: Quantization data type (`\"nf4\"` or `\"fp4\"`). NF4 is recommended.\n- `bnb_4bit_compute_dtype`: The dtype used for computation. Use `bfloat16` for better training stability.\n- `bnb_4bit_use_double_quant`: Enable nested quantization to save additional ~0.4 bits per parameter.\n\n### 8-bit Quantization\n\nFor slightly higher precision with reduced memory savings, you can use 8-bit quantization:\n\n```python\nfrom transformers import BitsAndBytesConfig, AutoModelForCausalLM\n\nbnb_config = BitsAndBytesConfig(load_in_8bit=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"meta-llama/Llama-2-7b-hf\",\n quantization_config=bnb_config,\n device_map=\"auto\",\n)\n```\n\nOr via command line:\n\n```bash\npython trl/scripts/sft.py \\\n --model_name_or_path meta-llama/Llama-2-7b-hf \\\n --load_in_8bit \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16\n```\n\n## Prompt Tuning\n\nPrompt tuning is another PEFT technique that learns soft prompts (continuous embeddings) prepended to the input, while keeping the entire model frozen. This is particularly effective for large models.\n\n### How Prompt Tuning Works\n\n1. **Virtual Tokens**: Adds learnable continuous embeddings (virtual tokens) to the input\n2. **Frozen Model**: The entire base model remains frozen\n3. **Task-Specific Prompts**: Each task learns its own prompt embeddings\n4. **Extreme Efficiency**: Only the prompt embeddings are trained (typically 8-20 tokens)\n\n### Using Prompt Tuning with TRL\n\n```python\nfrom peft import PromptTuningConfig, PromptTuningInit, TaskType\nfrom trl import SFTConfig, SFTTrainer\n\n# Configure Prompt Tuning\npeft_config = PromptTuningConfig(\n task_type=TaskType.CAUSAL_LM,\n prompt_tuning_init=PromptTuningInit.TEXT,\n num_virtual_tokens=8,\n prompt_tuning_init_text=\"Classify if the tweet is a complaint or not:\",\n tokenizer_name_or_path=\"Qwen/Qwen2-0.5B\",\n)\n\n# Configure training with higher learning rate for Prompt Tuning\ntraining_args = SFTConfig(\n learning_rate=2.0e-2, # Prompt Tuning typically uses 1e-2 to 3e-2\n ...\n)\n\n# Create trainer with PEFT config\ntrainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset,\n peft_config=peft_config, # pass PEFT config here\n)\n\ntrainer.train()\n```\n\n### Prompt Tuning Configuration\n\n```python\nfrom peft import PromptTuningConfig, PromptTuningInit, TaskType\n\npeft_config = PromptTuningConfig(\n task_type=TaskType.CAUSAL_LM, # Task type\n prompt_tuning_init=PromptTuningInit.TEXT, # Initialize from text\n num_virtual_tokens=8, # Number of virtual tokens\n prompt_tuning_init_text=\"Your initialization text here\",\n tokenizer_name_or_path=\"model_name\",\n)\n```\n\n**Configuration Parameters:**\n- `task_type`: The task type (`TaskType.CAUSAL_LM` for language modeling)\n- `prompt_tuning_init`: Initialization method (`TEXT`, `RANDOM`)\n- `num_virtual_tokens`: Number of virtual tokens to prepend (typically 8-20)\n- `prompt_tuning_init_text`: Text to initialize the virtual tokens (when using `TEXT` init)\n- `tokenizer_name_or_path`: Tokenizer for initializing from text\n\n### Prompt Tuning vs LoRA\n\n| Feature | Prompt Tuning | LoRA |\n|---------|---------------|------|\n| **Parameters Trained** | ~0.001% | ~0.1-1% |\n| **Memory Usage** | Minimal | Low |\n| **Training Speed** | Fastest | Fast |\n| **Model Modification** | None | Adapter layers |\n| **Best For** | Large models, many tasks | General fine-tuning |\n| **Learning Rate** | Higher (1e-2 to 3e-2) | Standard (1e-4 to 3e-4) |\n\n## Advanced PEFT Configurations\n\n### LoRA Configuration Parameters\n\n```python\nfrom peft import LoraConfig\n\npeft_config = LoraConfig(\n r=16, # LoRA rank\n lora_alpha=32, # LoRA scaling factor\n lora_dropout=0.05, # Dropout probability\n bias=\"none\", # Bias training strategy\n task_type=\"CAUSAL_LM\", # Task type\n target_modules=[\"q_proj\", \"v_proj\"], # Modules to apply LoRA\n modules_to_save=None, # Additional modules to train\n)\n```\n\n**Key Parameters:**\n- `r`: LoRA rank (typical values: 8, 16, 32, 64). Higher rank = more parameters but potentially better performance.\n- `lora_alpha`: Scaling factor (typically 2x the rank). Controls the magnitude of LoRA updates.\n- `lora_dropout`: Dropout probability for LoRA layers (typical: 0.05-0.1).\n- `target_modules`: Which modules to apply LoRA to. Common choices:\n - `[\"q_proj\", \"v_proj\"]`: Attention query and value (memory efficient)\n - `[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]`: All attention projections\n - `[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"]`: All linear layers\n- `modules_to_save`: Additional modules to fully train (e.g., `[\"embed_tokens\", \"lm_head\"]`)\n\n### Target Module Selection\n\nYou can specify which modules to apply LoRA to. Common patterns:\n\n```python\n# Minimal (most memory efficient)\ntarget_modules=[\"q_proj\", \"v_proj\"]\n\n# Attention only\ntarget_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]\n\n# All linear layers (best performance, more memory)\ntarget_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"]\n```\n\n### Using Command-Line Arguments\n\nTRL scripts accept PEFT parameters via command line:\n\n```bash\npython trl/scripts/sft.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --lora_dropout 0.05 \\\n --lora_target_modules q_proj v_proj \\\n --output_dir output\n```\n\nAvailable flags:\n- `--use_peft`: Enable PEFT\n- `--lora_r`: LoRA rank (default: 16)\n- `--lora_alpha`: LoRA alpha (default: 32)\n- `--lora_dropout`: LoRA dropout (default: 0.05)\n- `--lora_target_modules`: Target modules (space-separated)\n- `--lora_modules_to_save`: Additional modules to train\n- `--use_rslora`: Enable Rank-Stabilized LoRA\n- `--use_dora`: Enable Weight-Decomposed LoRA (DoRA)\n- `--load_in_4bit`: Enable 4-bit quantization (QLoRA)\n- `--load_in_8bit`: Enable 8-bit quantization\n\n## Saving and Loading PEFT Models\n\n### Saving\n\nAfter training, save your PEFT adapters:\n\n```python\n# Save the adapters\ntrainer.save_model(\"path/to/adapters\")\n\n# Or manually\nmodel.save_pretrained(\"path/to/adapters\")\n```\n\nThis saves only the adapter weights (~few MB) rather than the full model (~several GB).\n\n### Loading\n\nLoad a PEFT model for inference:\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import PeftModel\n\n# Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-0.5B\")\n\n# Load PEFT adapters\nmodel = PeftModel.from_pretrained(base_model, \"path/to/adapters\")\n\n# Optionally merge adapters into base model for faster inference\nmodel = model.merge_and_unload()\n```\n\n### Pushing to Hub\n\nYou can easily share your PEFT adapters on the Hugging Face Hub:\n\n```python\n# Push adapters to Hub\nmodel.push_to_hub(\"username/model-name-lora\")\n\n# Load from Hub\nfrom peft import PeftModel\nmodel = PeftModel.from_pretrained(base_model, \"username/model-name-lora\")\n```\n\n## Multi-GPU Training\n\nPEFT works seamlessly with TRL's multi-GPU support through `accelerate`:\n\n```bash\n# Configure accelerate\naccelerate config\n\n# Launch training\naccelerate launch trl/scripts/sft.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16\n```\n\nFor QLoRA with multiple GPUs, the base model is automatically sharded:\n\n```bash\naccelerate launch trl/scripts/sft.py \\\n --model_name_or_path meta-llama/Llama-2-70b-hf \\\n --load_in_4bit \\\n --use_peft \\\n --lora_r 32\n```\n\n### Naive Pipeline Parallelism (NPP) for Large Models\n\nFor very large models (>60B parameters), TRL supports Naive Pipeline Parallelism (NPP), which distributes the model and adapters across multiple GPUs. The activations and gradients are communicated across GPUs, supporting both `int8` and other data types.\n\n![NPP](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl-npp.png)\n\n**How to Use NPP**\n\nLoad your model with a custom `device_map` to split it across multiple devices:\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import LoraConfig\n\n# Create custom device map (see accelerate documentation)\ndevice_map = {\n \"model.embed_tokens\": 0,\n \"model.layers.0\": 0,\n # ... distribute layers across GPUs\n \"lm_head\": 0, # Must be on GPU 0\n}\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"meta-llama/Llama-2-70b-hf\",\n device_map=device_map,\n peft_config=lora_config,\n)\n```\n\n> [!IMPORTANT]\n> - Keep the `lm_head` module on the first GPU (device 0) to avoid errors\n> - See this [tutorial on device maps](https://github.com/huggingface/blog/blob/main/accelerate-large-models.md) for proper configuration\n> - Run training scripts directly (not with `accelerate launch`): `python script.py`\n> - Data Parallelism is not yet supported with NPP\n\n## Resources\n\n### TRL Examples and Notebooks\n\n- **[SFT with LoRA/QLoRA Notebook](https://github.com/huggingface/trl/blob/main/examples/notebooks/sft_trl_lora_qlora.ipynb)** - Complete working example showing both LoRA and QLoRA implementations\n- **[TRL Examples Directory](https://github.com/huggingface/trl/tree/main/examples)** - Collection of training scripts demonstrating PEFT with different trainers\n- **[TRL Cookbook Recipes](https://github.com/huggingface/cookbook/tree/main/notebooks/transformers)** - Step-by-step guides for common PEFT training scenarios\n\n### Documentation\n\n- [PEFT Documentation](https://huggingface.co/docs/peft) - Official PEFT library documentation\n- [TRL Documentation](https://huggingface.co/docs/trl) - Complete TRL documentation with trainer guides\n- [LoRA Without Regret](lora_without_regret) - Best practices for using LoRA effectively\n\n### Research Papers\n\n- [LoRA Paper](https://huggingface.co/papers/2106.09685) - Original LoRA methodology and results\n- [QLoRA Paper](https://huggingface.co/papers/2305.14314) - Efficient finetuning with 4-bit quantization\n- [Prompt Tuning Paper](https://huggingface.co/papers/2104.08691) - The Power of Scale for Parameter-Efficient Prompt Tuning\n" }, { "path": "docs/source/ppo_trainer.md", "content": "# PPO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-PPO-blue)](https://huggingface.co/models?other=ppo,trl)\n\nTRL supports training LLMs with [Proximal Policy Optimization (PPO)](https://huggingface.co/papers/1707.06347).\n\nReferences:\n\n- [Fine-Tuning Language Models from Human Preferences](https://github.com/openai/lm-human-preferences)\n- [Learning to Summarize from Human Feedback](https://github.com/openai/summarize-from-feedback)\n- [The N Implementation Details of RLHF with PPO](https://huggingface.co/blog/the_n_implementation_details_of_rlhf_with_ppo)\n- [The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization](https://huggingface.co/papers/2403.17031)\n\n## Get started\n\nTo just run a PPO script to make sure the trainer can run, you can run the following command to train a PPO model with a dummy reward model.\n\n```bash\npython examples/scripts/ppo/ppo.py \\\n --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \\\n --dataset_train_split descriptiveness \\\n --learning_rate 3e-6 \\\n --num_ppo_epochs 1 \\\n --num_mini_batches 1 \\\n --output_dir models/minimal/ppo \\\n --per_device_train_batch_size 64 \\\n --gradient_accumulation_steps 1 \\\n --total_episodes 10000 \\\n --model_name_or_path EleutherAI/pythia-1b-deduped \\\n --sft_model_path EleutherAI/pythia-1b-deduped \\\n --reward_model_path EleutherAI/pythia-1b-deduped \\\n --missing_eos_penalty 1.0\n```\n\n## Explanation of the logged metrics\n\nThe logged metrics are as follows. Here is an example [tracked run at Weights and Biases](https://wandb.ai/huggingface/trl/runs/dd2o3g35)\n\n- `eps`: Tracks the number of episodes per second.\n- `objective/kl`: The mean Kullback-Leibler (KL) divergence between the current policy and reference policy.\n- `objective/entropy`: The mean entropy of the policy, indicating the randomness of the actions chosen by the policy.\n- `objective/non_score_reward`: The mean reward from non-score-related sources, basically `beta * kl.sum(1)`, where `beta` is the KL penalty coefficient and `kl` is the per-token KL divergence.\n- `objective/rlhf_reward`: The mean RLHF reward, which is `score - non_score_reward`.\n- `objective/scores`: The mean scores returned by the reward model / environment.\n- `policy/approxkl_avg`: The average approximate KL divergence between consecutive PPO policies. Note that this is not the same as `objective/kl`.\n- `policy/clipfrac_avg`: The average fraction of policy updates that are clipped, indicating how often the policy updates are constrained to prevent large changes.\n- `loss/policy_avg`: The average policy loss, indicating how well the policy is performing.\n- `loss/value_avg`: The average value loss, indicating the difference between the predicted value and the actual reward.\n- `val/clipfrac_avg`: The average fraction of value function updates that are clipped, similar to policy/clipfrac_avg but for the value function.\n- `policy/entropy_avg`: The average entropy of the policy during training, indicating how diverse the policy's actions are.\n- `val/ratio`: The mean ratio of the current policy probability to the old policy probability, providing a measure of how much the policy has changed.\n- `val/ratio_var`: The variance of the `val/ratio`, indicating the variability in policy changes.\n- `val/num_eos_tokens`: The number of end-of-sequence (EOS) tokens generated, which can indicate the number of complete responses.\n- `lr`: lr: The current learning rate used by the optimizer.\n- `episode`: episode: The current episode count in the training process.\n\n## Cookbook\n\n- Debugging TIP: `objective/rlhf_reward`: this is the ultimate objective of the RLHF training. If training works as intended, this metric should keep going up.\n- Debugging TIP: `val/ratio`: this number should float around 1.0, and it gets clipped by `--cliprange 0.2` with PPO's surrogate loss. So if this `ratio` is too high like 2.0 or 1000.0 or too small like 0.1, it means the updates between consecutive policies are too drastic. You should try understand why this is happening and try to fix it.\n- Memory TIP: If you are running out of memory, you can try to reduce the `--per_device_train_batch_size` or increase the `--gradient_accumulation_steps` to reduce the memory footprint.\n- Memory TIP: If you have multiple GPUs, you can also run training with DeepSpeed stage 3 to reduce the memory footprint `accelerate launch --config_file examples/accelerate_configs/deepspeed_zero3.yaml`.\n- Usage TIP: We recommend to use the \"EOS trick\" via `--missing_eos_penalty`, which subtracts a static scalar penalty from the score of completions that do not end with an EOS token. This can help the model learn to generate more coherent completions.\n\n## What is my model doing exactly?\n\nTo help you understand what your model is doing, we periodically log some sample completions from the model. Here is an example of a completion. In an example [tracked run at Weights and Biases](https://wandb.ai/huggingface/trl/runs/dd2o3g35), it looks like the following, allowing you to see the model's response at different stages of training. By default we generate `--num_sample_generations 10` during training, but you can customize the number of generations.\n\n![ppov2_completions](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/ppov2_completions.gif)\n\nIn the logs the sampled generations look like\n\n```txt\nโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\nโ”ƒ query โ”ƒ model response โ”ƒ score โ”ƒ\nโ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\nโ”‚ SUBREDDIT: r/AskReddit โ”‚ I'm in love with a friend, and โ”‚ 3.921875 โ”‚\nโ”‚ โ”‚ I don't know how to get rid of โ”‚ โ”‚\nโ”‚ TITLE: How do you get someone โ”‚ those feelings. I'm โ”‚ โ”‚\nโ”‚ out of your head? โ”‚ desperate.<|endoftext|>[PAD][Pโ€ฆ โ”‚ โ”‚\nโ”‚ โ”‚ โ”‚ โ”‚\nโ”‚ POST: Hi, โ”‚ โ”‚ โ”‚\nโ”‚ I'm 22, and I have been with my โ”‚ โ”‚ โ”‚\nโ”‚ girlfriend for 5 years now. We โ”‚ โ”‚ โ”‚\nโ”‚ recently moved together. We've โ”‚ โ”‚ โ”‚\nโ”‚ always loved each other โ”‚ โ”‚ โ”‚\nโ”‚ intensely. โ”‚ โ”‚ โ”‚\nโ”‚ โ”‚ โ”‚ โ”‚\nโ”‚ Problem, I recently started to โ”‚ โ”‚ โ”‚\nโ”‚ have feelings for an other โ”‚ โ”‚ โ”‚\nโ”‚ person (a friend). This person โ”‚ โ”‚ โ”‚\nโ”‚ has had a boyfriend for now 3 โ”‚ โ”‚ โ”‚\nโ”‚ years, and has absolutely no โ”‚ โ”‚ โ”‚\nโ”‚ ideas. Those feelings were so โ”‚ โ”‚ โ”‚\nโ”‚ strong, it was hard to hide โ”‚ โ”‚ โ”‚\nโ”‚ them. After 2 months of me โ”‚ โ”‚ โ”‚\nโ”‚ being distant and really sad, โ”‚ โ”‚ โ”‚\nโ”‚ my girlfriend forced me to say โ”‚ โ”‚ โ”‚\nโ”‚ what was bothering me. I'm not โ”‚ โ”‚ โ”‚\nโ”‚ a good liar, and now she knows. โ”‚ โ”‚ โ”‚\nโ”‚ โ”‚ โ”‚ โ”‚\nโ”‚ We decided to give us a week โ”‚ โ”‚ โ”‚\nโ”‚ alone, I went to my parents. โ”‚ โ”‚ โ”‚\nโ”‚ โ”‚ โ”‚ โ”‚\nโ”‚ Now, I'm completely lost. I โ”‚ โ”‚ โ”‚\nโ”‚ keep on thinking about this โ”‚ โ”‚ โ”‚\nโ”‚ person, and I hate that. I โ”‚ โ”‚ โ”‚\nโ”‚ would like for those feelings โ”‚ โ”‚ โ”‚\nโ”‚ to go away, to leave me alone. โ”‚ โ”‚ โ”‚\nโ”‚ But I can't. โ”‚ โ”‚ โ”‚\nโ”‚ โ”‚ โ”‚ โ”‚\nโ”‚ What do I do? It's been 3 โ”‚ โ”‚ โ”‚\nโ”‚ months now, and I'm just โ”‚ โ”‚ โ”‚\nโ”‚ desperate. โ”‚ โ”‚ โ”‚\nโ”‚ โ”‚ โ”‚ โ”‚\nโ”‚ TL;DR: โ”‚ โ”‚ โ”‚\nโ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\nโ”‚ SUBREDDIT: r/pettyrevenge โ”‚ My mom woke me up with a loud โ”‚ 6.84375 โ”‚\nโ”‚ โ”‚ TV. I blasted Gangnam Style on โ”‚ โ”‚\nโ”‚ TITLE: So, my mom woke me up โ”‚ repeat, with the bass cranked โ”‚ โ”‚\nโ”‚ with a loud TV. โ”‚ up as high as it could โ”‚ โ”‚\nโ”‚ โ”‚ go.<|endoftext|>[PAD][PAD][PADโ€ฆ โ”‚ โ”‚\nโ”‚ POST: She was in her living โ”‚ โ”‚ โ”‚\nโ”‚ room, watching TV. This was at โ”‚ โ”‚ โ”‚\nโ”‚ about 8:30 in the morning, and โ”‚ โ”‚ โ”‚\nโ”‚ she was exercising. She turned โ”‚ โ”‚ โ”‚\nโ”‚ the TV up extra loud to hear it โ”‚ โ”‚ โ”‚\nโ”‚ over her excercycle, and woke โ”‚ โ”‚ โ”‚\nโ”‚ me up. I went in there asking โ”‚ โ”‚ โ”‚\nโ”‚ for her to turn it down. She โ”‚ โ”‚ โ”‚\nโ”‚ said she didn't have to; I โ”‚ โ”‚ โ”‚\nโ”‚ explained that I always used โ”‚ โ”‚ โ”‚\nโ”‚ headphones so she didn't have โ”‚ โ”‚ โ”‚\nโ”‚ to deal with my noise and that โ”‚ โ”‚ โ”‚\nโ”‚ she should give me a little โ”‚ โ”‚ โ”‚\nโ”‚ more respect, given that I paid โ”‚ โ”‚ โ”‚\nโ”‚ rent at the time. โ”‚ โ”‚ โ”‚\nโ”‚ โ”‚ โ”‚ โ”‚\nโ”‚ She disagreed. I went back to โ”‚ โ”‚ โ”‚\nโ”‚ my room, rather pissed off at โ”‚ โ”‚ โ”‚\nโ”‚ the lack of equality. I had no โ”‚ โ”‚ โ”‚\nโ”‚ lock on my door; but I had a โ”‚ โ”‚ โ”‚\nโ”‚ dresser right next to it, so I โ”‚ โ”‚ โ”‚\nโ”‚ pulled one of the drawers out โ”‚ โ”‚ โ”‚\nโ”‚ enough so that it caused the โ”‚ โ”‚ โ”‚\nโ”‚ door to not be openable. Then, โ”‚ โ”‚ โ”‚\nโ”‚ I turned my speakers up really โ”‚ โ”‚ โ”‚\nโ”‚ loud and blasted Gangnam Style โ”‚ โ”‚ โ”‚\nโ”‚ on repeat, with the bass โ”‚ โ”‚ โ”‚\nโ”‚ cranked up as high as it could โ”‚ โ”‚ โ”‚\nโ”‚ go. โ”‚ โ”‚ โ”‚\nโ”‚ โ”‚ โ”‚ โ”‚\nโ”‚ If you hate Gangnam Style for โ”‚ โ”‚ โ”‚\nโ”‚ being overplayed, you will see โ”‚ โ”‚ โ”‚\nโ”‚ why I chose that particular โ”‚ โ”‚ โ”‚\nโ”‚ song. I personally don't mind โ”‚ โ”‚ โ”‚\nโ”‚ it. But here's the thing about โ”‚ โ”‚ โ”‚\nโ”‚ my bass; it vibrates the walls, โ”‚ โ”‚ โ”‚\nโ”‚ making one hell of a lot of โ”‚ โ”‚ โ”‚\nโ”‚ noise. Needless to say, my mom โ”‚ โ”‚ โ”‚\nโ”‚ was not pleased and shut off โ”‚ โ”‚ โ”‚\nโ”‚ the internet. But it was oh so โ”‚ โ”‚ โ”‚\nโ”‚ worth it. โ”‚ โ”‚ โ”‚\nโ”‚ โ”‚ โ”‚ โ”‚\nโ”‚ TL;DR: โ”‚ โ”‚ โ”‚\nโ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n```\n\n## Implementation details\n\nThis PPO implementation is based on the [The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization](https://huggingface.co/papers/2403.17031).\n\n## Benchmark experiments\n\nTo validate the PPO implementation works, we ran experiment on the 1B model. Here are the command we used to run the experiment. We take the SFT / RM models directly from [The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization](https://huggingface.co/papers/2403.17031).\n\n```shell\naccelerate launch --config_file examples/accelerate_configs/deepspeed_zero2.yaml \\\n examples/scripts/ppo/ppo_tldr.py \\\n --dataset_name trl-lib/tldr \\\n --dataset_test_split validation \\\n --output_dir models/minimal/ppo_tldr \\\n --learning_rate 3e-6 \\\n --per_device_train_batch_size 16 \\\n --gradient_accumulation_steps 4 \\\n --total_episodes 1000000 \\\n --model_name_or_path EleutherAI/pythia-1b-deduped \\\n --sft_model_path cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr \\\n --reward_model_path cleanrl/EleutherAI_pythia-1b-deduped__reward__tldr \\\n --local_rollout_forward_batch_size 16 \\\n --missing_eos_penalty 1.0 \\\n --stop_token eos \\\n --eval_strategy steps \\\n --eval_steps 100\n```\n\nCheckpoints and experiment tracking are available at:\n\n- [๐Ÿค— Model checkpoint](https://huggingface.co/trl-lib/ppo_tldr)\n- [๐Ÿ Tracked experiment](https://wandb.ai/huggingface/trl/runs/dd2o3g35)\n\nTo evaluate, we use [vLLM](https://github.com/vllm-project/vllm) to load the checkpoints and GPT-4o mini as a judge model to evaluate the generated TL;DR against the reference TL;DR.\nFor more information on how to use judges, see [Judges](judges).\n\n```bash\n$ python examples/scripts/evals/judge_tldr.py --model_name_or_path cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr --judge_model gpt-4o-mini --num_examples 1000\nModel win rate: 33.00%\n$ python examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/ppo_tldr --judge_model gpt-4o-mini --num_examples 1000\nModel win rate: 64.70%\n```\n\nThe PPO checkpoint gets a 64.7% preferred rate vs the 33.0% preference rate of the SFT checkpoint. This is a good sign that the PPO training is working as intended.\n\nMetrics:\n\n![PPO v2](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/ppov2.png)\n\n```bash\n# pip install openrlbenchmark==0.2.1a5\n# see https://github.com/openrlbenchmark/openrlbenchmark#get-started for documentation\n# to use it, change `?we=huggingface&wpn=trl` to your own project and `?tag=pr-1540` to your own tag\npython -m openrlbenchmark.rlops_multi_metrics \\\n --filters '?we=huggingface&wpn=trl&xaxis=train/episode&ceik=output_dir&cen=sft_model_path&metrics=train/objective/rlhf_reward&metrics=train/objective/scores&metrics=train/objective/kl&metrics=train/objective/non_score_reward&metrics=train/objective/entropy&metrics=train/policy/approxkl_avg&metrics=train/policy/clipfrac_avg&metrics=train/loss/policy_avg&metrics=train/loss/value_avg&metrics=train/val/clipfrac_avg&metrics=train/policy/entropy_avg&metrics=train/val/ratio&metrics=train/val/ratio_var&metrics=train/val/num_eos_tokens&metrics=train/lr&metrics=train/eps' \\\n \"cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr?tag=pr-1540\" \\\n --env-ids models/minimal/ppo_tldr \\\n --pc.ncols 4 \\\n --pc.ncols-legend 1 \\\n --pc.xlabel \"Episode\" \\\n --output-filename benchmark/trl/pr-1540/ppo \\\n --scan-history\n```\n\n## PPOTrainer\n\n[[autodoc]] experimental.ppo.PPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## PPOConfig\n\n[[autodoc]] experimental.ppo.PPOConfig\n\n## PreTrainedModelWrapper\n\n[[autodoc]] experimental.ppo.PreTrainedModelWrapper\n\n## AutoModelForCausalLMWithValueHead\n\n[[autodoc]] experimental.ppo.AutoModelForCausalLMWithValueHead\n - __init__\n - forward\n - generate\n - _init_weights\n\n## AutoModelForSeq2SeqLMWithValueHead\n\n[[autodoc]] experimental.ppo.AutoModelForSeq2SeqLMWithValueHead\n - __init__\n - forward\n - generate\n - _init_weights\n" }, { "path": "docs/source/prm_trainer.md", "content": "# PRM Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-PRM-blue)](https://huggingface.co/models?other=prm,trl)\n\n> [!WARNING]\n> PRM Trainer is an experimental API which is subject to change at any time.\n\n## Overview\n\nProcess-supervised Reward Models (PRM) were proposed in [Solving math word problems with process- and outcome-based feedback](https://huggingface.co/papers/2211.14275) by Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.\n\nThe abstract from the paper is the following:\n\n> Recent work has shown that asking language models to generate reasoning steps improves performance on many reasoning tasks. When moving beyond prompting, this raises the question of how we should supervise such models: outcome-based approaches which supervise the final result, or process-based approaches which supervise the reasoning process itself? Differences between these approaches might naturally be expected not just in final-answer errors but also in reasoning errors, which can be difficult to detect and are problematic in many real-world domains such as education. We run the first comprehensive comparison between process- and outcome-based approaches trained on a natural language task, GSM8K. We find that pure outcome-based supervision produces similar final-answer error rates with less label supervision. However, for correct reasoning steps we find it necessary to use processbased supervision or supervision from learned reward models that emulate process-based feedback. In total, we improve the previous best results from 16.8% โ†’ 12.7% final-answer error and 14.0% โ†’ 3.4% reasoning error among final-answer-correct solutions.\n\nThis post-training method was contributed by [Gaetan Lopez](https://github.com/gaetanlop), [Lewis Tunstall](https://huggingface.co/lewtun), [Quentin Gallouรฉdec](https://huggingface.co/qgallouedec) and [Agustรญn Piqueres](https://huggingface.co/plaguss).\n\n## Quick start\n\nThis example demonstrates how to train a model using the PRM method. We use the [Qwen 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B) as the base model. We use the stepwise supervision data from the [Math Shepherd dataset](https://huggingface.co/datasets/trl-lib/math_shepherd). You can view the data in the dataset here:\n\n\n\nBelow is the script to train the model:\n\n```python\n# train_prm.py\nfrom datasets import load_dataset\nfrom trl.experimental.prm import PRMConfig, PRMTrainer\nfrom transformers import AutoModelForTokenClassification, AutoTokenizer\n\nmodel = AutoModelForTokenClassification.from_pretrained(\"Qwen/Qwen2-0.5B\", num_labels=2)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2-0.5B\")\ntrain_dataset = load_dataset(\"trl-lib/math_shepherd\", split=\"train[:10%]\")\n\ntraining_args = PRMConfig(output_dir=\"Qwen2-0.5B-Reward-Math-Sheperd\")\ntrainer = PRMTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset)\ntrainer.train()\n```\n\nExecute the script using the following command:\n\n```bash\naccelerate launch train_prm.py\n```\n\nDistributed across 8 GPUs, the training takes approximately 1 hour.\n\nTo see how the [trained model](https://huggingface.co/trl-lib/Qwen2-0.5B-Reward-Math-Sheperd) performs, you can use the following script.\n\n```python\nfrom datasets import load_dataset\nfrom transformers import pipeline\n\npipe = pipeline(\"token-classification\", model=\"trl-lib/Qwen2-0.5B-Reward-Math-Sheperd\")\ndataset = load_dataset(\"trl-lib/math_shepherd\")\nexample = {\n \"prompt\": \"Musa is the class teacher of a class of 45 students. He wants to split them into three groups by age. If a third of the class is under 11 years, and two-fifths are above 11 but under 13, how many students will be in the third group (13 years and above)?\",\n \"completions\": [\n \"Step 1: A third of the class is under 11 years because 11 - 1/3 = <<11-1/3=7>>7.\",\n \"Step 2: Two-fifths of the class are above 11 but under 13 because 2/5 * 11 = <<2/5*11=8>>8.\",\n \"Step 3: There are 45 students, so the third group will have 45 - 7 - 8 = <<45-7-8=20>>20 students. The answer is: 20\",\n ],\n \"labels\": [True, False, False],\n}\n\n\nseparator = \"\\n\" # It's important to use the same separator as the one used during training\n\nfor idx in range(1, len(example[\"completions\"]) + 1):\n steps = example[\"completions\"][0:idx]\n text = separator.join((example[\"prompt\"], *steps)) + separator # Add a separator between the prompt and each steps\n pred_entity = pipe(text)[-1][\"entity\"]\n pred = {\"LABEL_0\": False, \"LABEL_1\": True}[pred_entity]\n label = example[\"labels\"][idx - 1]\n print(f\"Step {idx}\\tPredicted: {pred} \\tLabel: {label}\")\n```\n\n```text\nStep 1 Predicted: True Label: True\nStep 2 Predicted: False Label: False\nStep 3 Predicted: False Label: False\n```\n\nIt's a win!\n\n## Expected dataset type\n\nPRM requires a [stepwise supervision](dataset_formats#stepwise-supervision).\nThe dataset should contain the following columns: `prompt`, `completions` and `labels`, where `completions` contains a list of reasoning steps and `labels` a list of booleans or floats indicating the correctness of each step.\n\nThe [`experimental.prm.PRMTrainer`] only supports [standard](dataset_formats#standard) dataset format.\n\n## Example script\n\nWe provide an example script to train a model using the PRM method. The script is available in [`examples/scripts/prm.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/prm.py)\n\nTo use the PRM script with the [Qwen2 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B) on the [Math Shepherd dataset](https://huggingface.co/datasets/trl-lib/math_shepherd), run the following command:\n\n```bash\naccelerate launch examples/scripts/prm.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/math_shepherd \\\n --num_train_epochs 1 \\\n --output_dir Qwen2-0.5B-Reward-Math-Sheperd\n```\n\n## PRMTrainer\n\n[[autodoc]] experimental.prm.PRMTrainer\n - train\n - save_model\n - push_to_hub\n\n## PRMConfig\n\n[[autodoc]] experimental.prm.PRMConfig\n" }, { "path": "docs/source/ptt_integration.md", "content": "# Post-Training Toolkit Integration\n\n[Post-Training Toolkit](https://github.com/microsoft/post-training-toolkit) is a diagnostic and observability layer for RLHF training runs. Add one callback to any TRL trainer and get **auto-metrics**, **crash postmortems**, and **literature-backed heuristics**โ€”without writing glue code.\n\nIt was built to operationalize the debugging patterns we found most useful when running post-training at scale.\n\n## Usage\n\n1. First, install Post-Training Toolkit:\n\n```bash\npip install post-training-toolkit\n```\n\n2. Add one callback to your trainer. That's it!\n\n\n\n\n```python\nfrom post_training_toolkit import DiagnosticsCallback\nfrom trl import DPOTrainer\n\ntrainer = DPOTrainer(\n model=model,\n args=training_args,\n callbacks=[DiagnosticsCallback()], # โ† Just add this\n ...\n)\ntrainer.train()\n```\n\n\n\n\n```python\nfrom post_training_toolkit import DiagnosticsCallback\nfrom trl.experimental.ppo import PPOTrainer\n\ntrainer = PPOTrainer(\n model=model,\n args=training_args,\n callbacks=[DiagnosticsCallback()], # โ† Just add this\n ...\n)\ntrainer.train()\n```\n\n\n\n\n```python\nfrom post_training_toolkit import DiagnosticsCallback\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n model=model,\n args=training_args,\n callbacks=[DiagnosticsCallback()], # โ† Just add this\n ...\n)\ntrainer.train()\n```\n\n\n\n\n```python\nfrom post_training_toolkit import DiagnosticsCallback\nfrom trl.experimental.orpo import ORPOTrainer\n\ntrainer = ORPOTrainer(\n model=model,\n args=training_args,\n callbacks=[DiagnosticsCallback()], # โ† Just add this\n ...\n)\ntrainer.train()\n```\n\n\n\n\n```python\nfrom post_training_toolkit import DiagnosticsCallback\nfrom trl import KTOTrainer\n\ntrainer = KTOTrainer(\n model=model,\n args=training_args,\n callbacks=[DiagnosticsCallback()], # โ† Just add this\n ...\n)\ntrainer.train()\n```\n\n\n\n\n```python\nfrom post_training_toolkit import DiagnosticsCallback\nfrom trl.experimental.cpo import CPOTrainer\n\ntrainer = CPOTrainer(\n model=model,\n args=training_args,\n callbacks=[DiagnosticsCallback()], # โ† Just add this\n ...\n)\ntrainer.train()\n```\n\n\n\n\n```python\nfrom post_training_toolkit import DiagnosticsCallback\nfrom trl import GRPOTrainer\n\ntrainer = GRPOTrainer(\n model=model,\n args=training_args,\n callbacks=[DiagnosticsCallback()], # โ† Just add this\n ...\n)\ntrainer.train()\n```\n\n\n\n\n## What You Get\n\n**Example output:**\n```text\n[HIGH] DPO loss stuck at ~0.693 (random chance). Model may not be learning preferences.\n Ref: Rafailov et al. (2023) 'DPO', Section 4.2\n\n[RECOMMENDED] Increase learning rate 2-5x, check data quality, or reduce beta.\n```\n\n## Example Demo\n\nSee a full working example with auto-stop in action:\n\n๐Ÿ“‚ **[demo/live_demo.ipynb](https://github.com/microsoft/post-training-toolkit/blob/main/demo/notebooks/demo_live_output.ipynb)**\n\n๐Ÿ“‚ **[demo/scripts/custom_heuristic.py](https://github.com/microsoft/post-training-toolkit/blob/main/demo/scripts/custom_heuristic_demo.py)**\n\n\n### 1. Auto-Metrics\nThe callback automatically captures algorithm-specific metrics, backed by the latest research and industry push:\n\n| Trainer | Key Metrics Captured |\n|---------|---------------------|\n| **DPO** | loss, win_rate, reward_margin, logps_chosen/rejected |\n| **PPO** | policy_loss, value_loss, entropy, clip_fraction, KL |\n| **GRPO** | group rewards, advantages, policy loss, KL |\n| **SFT** | loss, perplexity, accuracy |\n| **ORPO** | sft_loss, odds_ratio_loss, log_odds_ratio |\n| **KTO** | kl, logps for desirable/undesirable |\n\n\n### 2. Crash Postmortems\nIf training crashes or gets interrupted, you get a `postmortem.json` with full context:\n\n```json\n{\n \"exit_reason\": \"exception\",\n \"last_step\": 847,\n \"timestamp\": \"2025-12-17T19:26:04Z\",\n \"final_metrics\": {\"dpo_loss\": 0.693, \"win_rate\": 0.52}\n}\n```\n\nNo more \"what step did it die on?\"\n\n### 3. Auto-Stop on Critical Issues\n\nEnable automatic training termination when critical issues are detected:\n\n```python\ncallback = DiagnosticsCallback(stop_on_critical=True)\n```\n\n## Distributed Training\nWorks automatically with multi-GPU setups. Zero configuration needed:\n\n```bash\naccelerate launch --num_processes 8 train.py\n```\n\nAutomatically detects stragglers, aggregates metrics across ranks, and tracks memory balance.\n" }, { "path": "docs/source/quickstart.md", "content": "# Quickstart\n\nTRL is a comprehensive library for post-training foundation models using techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO).\n\n## Quick Examples\n\nGet started instantly with TRL's most popular trainers. Each example uses compact models for quick experimentation.\n\n### Supervised Fine-Tuning\n\n```python\nfrom trl import SFTTrainer\nfrom datasets import load_dataset\n\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen2.5-0.5B\",\n train_dataset=load_dataset(\"trl-lib/Capybara\", split=\"train\"),\n)\ntrainer.train()\n```\n\n### Group Relative Policy Optimization\n\n```python\nfrom trl import GRPOTrainer\nfrom datasets import load_dataset\nfrom trl.rewards import accuracy_reward\n\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n train_dataset=load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\"),\n reward_funcs=accuracy_reward,\n)\ntrainer.train()\n```\n\n### Direct Preference Optimization\n\n```python\nfrom trl import DPOTrainer\nfrom datasets import load_dataset\n\ntrainer = DPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n train_dataset=load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\"),\n)\ntrainer.train()\n```\n\n### Reward Modeling\n\n```python\nfrom trl import RewardTrainer\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\ntrainer = RewardTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\n## Command Line Interface\n\nSkip the code entirely - train directly from your terminal:\n\n```bash\n# SFT: Fine-tune on instructions\ntrl sft --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/Capybara\n\n# DPO: Align with preferences \ntrl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \\\n --dataset_name trl-lib/ultrafeedback_binarized\n\n# Reward: Train a reward model\ntrl reward --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \\\n --dataset_name trl-lib/ultrafeedback_binarized\n```\n\n## What's Next?\n\n### ๐Ÿ“š Learn More\n\n- [SFT Trainer](sft_trainer) - Complete SFT guide\n- [DPO Trainer](dpo_trainer) - Preference alignment\n- [GRPO Trainer](grpo_trainer) - Group relative policy optimization\n\n### ๐Ÿš€ Scale Up\n\n- [Distributed Training](distributing_training) - Multi-GPU setups\n- [Memory Optimization](reducing_memory_usage) - Efficient training\n- [PEFT Integration](peft_integration) - LoRA and QLoRA\n\n### ๐Ÿ’ก Examples\n\n- [Example Scripts](https://github.com/huggingface/trl/tree/main/examples) - Production-ready code\n- [Community Tutorials](community_tutorials) - External guides\n\n## Troubleshooting\n\n### Out of Memory?\n\nReduce batch size and enable optimizations:\n\n\n\n\n```python\ntraining_args = SFTConfig(\n per_device_train_batch_size=1, # Start small\n gradient_accumulation_steps=8, # Maintain effective batch size\n)\n```\n\n\n\n\n```python\ntraining_args = DPOConfig(\n per_device_train_batch_size=1, # Start small\n gradient_accumulation_steps=8, # Maintain effective batch size\n)\n```\n\n\n\n\n### Loss not decreasing?\n\nTry adjusting the learning rate:\n\n```python\ntraining_args = SFTConfig(learning_rate=2e-5) # Good starting point\n```\n\nFor more help, open an [issue on GitHub](https://github.com/huggingface/trl/issues).\n" }, { "path": "docs/source/rapidfire_integration.md", "content": "# RapidFire AI Integration\n\nRapidFire AI is an open-source experiment execution framework that enables concurrent training of multiple TRL configurations on the same GPU(s) through intelligent chunk-based scheduling.\n\n## Key Features\n\n- **16-24ร— higher experimentation throughput** compared to sequential training.\n- **Almost no code changes** - drop-in configuration wrappers around TRL's and PEFT's existing configs.\n- **Interactive Control Operations** - real-time control to stop, resume, clone, and modify training runs in flight\n- **Automatic multi-GPU orchestration** with intelligent scheduling\n- **Full compatibility** with transformers, PEFT, SFTTrainer, DPOTrainer, and GRPOTrainer\n- **Full MLflow Integration**: Automatic experiment tracking and visualization\n- **Production-Ready**: Already used in production environments with complete working examples.\n\n### Problem It Solves\n\nWhen fine-tuning or post-training with TRL, AI developers often need to:\n- Try different hyperparameter configurations\n- Compare different LoRA settings\n- Test different prompt schemes\n- Run ablation studies\n\n\n**Current approach**: Train each config one after another โ†’ slow and inefficient process\n\n**With RapidFire AI**: Train all configs in one go even on a single GPU โ†’ 16-24ร— faster process\n\n### How It Works\n\nRapidFire AI employs **adaptive chunk-based scheduling**:\n\n```\nGPU Timeline (Single GPU):\nChunk 1: [Config A] โ†’ [Config B] โ†’ [Config C] โ†’ [Config D]\nChunk 2: [Config A] โ†’ [Config B] โ†’ [Config C] โ†’ [Config D]\nChunk 3: [Config A] โ†’ [Config B] โ†’ [Config C] โ†’ [Config D]\n```\n\nThis enables:\n- Early comparison of configurations on same data subsets incrementally\n- Efficient GPU utilization and minimizing idle times\n- Real-time and automated experiment metrics tracking\n- Dynamic control over runs in flight to incentivize more experimentation\n\n\n## Installation\n\n### Prerequisites\n\n- Python 3.12.x\n- NVIDIA GPU with Compute Capability 7.x or 8.x\n- CUDA Toolkit 11.8+\n- PyTorch 2.7.1+\n\n### pip install\n\n```bash\npip install rapidfireai\n```\n\nOnce installed, authenticate with Hugging Face and initialize RapidFire AI:\n\n```bash\n# Authenticate with Hugging Face\nhuggingface-cli login --token YOUR_TOKEN\n\n# Workaround for current issue: https://github.com/huggingface/xet-core/issues/527\npip uninstall -y hf-xet\n\n# Initialize RapidFire AI\nrapidfireai init\n\n# Start the RapidFire AI server\nrapidfireai start\n```\n\nThe dashboard will be available at `http://0.0.0.0:3000` where you can monitor and control experiments in real-time.\n\n## Quick Start: SFT Training with Multiple Configs\n\nHere's a complete example showing how to train multiple SFT configurations concurrently:\n\n```python\nfrom rapidfireai import Experiment\nfrom rapidfireai.automl import List, RFGridSearch, RFModelConfig, RFLoraConfig, RFSFTConfig\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load dataset\ndataset = load_dataset(\"bitext/Bitext-customer-support-llm-chatbot-training-dataset\")\ntrain_dataset = dataset[\"train\"].select(range(128)).shuffle(seed=42)\neval_dataset = dataset[\"train\"].select(range(100, 124)).shuffle(seed=42)\n\n# Define data formatting function\ndef formatting_function(row):\n return {\n \"prompt\": [\n {\"role\": \"system\", \"content\": \"You are a helpful customer support assistant.\"},\n {\"role\": \"user\", \"content\": row[\"instruction\"]},\n ],\n \"completion\": [\n {\"role\": \"assistant\", \"content\": row[\"response\"]}\n ]\n }\n\n# Initialize experiment\nexperiment = Experiment(experiment_name=\"sft-customer-support\")\n\n# Define multiple LoRA configurations to compare\npeft_configs = List([\n RFLoraConfig(r=8, lora_alpha=16, lora_dropout=0.1, \n target_modules=[\"q_proj\", \"v_proj\"], bias=\"none\"),\n RFLoraConfig(r=32, lora_alpha=64, lora_dropout=0.1,\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"], bias=\"none\")\n])\n\n# Define multiple training configurations\n# 2 base configs ร— 2 PEFT configs = 4 total training runs\nconfig_set = List([\n RFModelConfig(\n model_name=\"TinyLlama/TinyLlama-1.1B-Chat-v1.0\",\n peft_config=peft_configs,\n training_args=RFSFTConfig( # Wraps TRL's SFTConfig\n learning_rate=1e-3,\n per_device_train_batch_size=4,\n max_steps=128,\n fp16=True,\n ),\n model_type=\"causal_lm\",\n model_kwargs={\"device_map\": \"auto\", \"torch_dtype\": \"auto\", \"use_cache\": False},\n formatting_func=formatting_function,\n ),\n RFModelConfig(\n model_name=\"TinyLlama/TinyLlama-1.1B-Chat-v1.0\",\n peft_config=peft_configs,\n training_args=RFSFTConfig(\n learning_rate=1e-4, # Different learning rate\n per_device_train_batch_size=4,\n max_steps=128,\n fp16=True,\n ),\n model_type=\"causal_lm\",\n model_kwargs={\"device_map\": \"auto\", \"torch_dtype\": \"auto\", \"use_cache\": False},\n formatting_func=formatting_function,\n )\n])\n\n# Define model creation function\ndef create_model(model_config):\n model = AutoModelForCausalLM.from_pretrained(\n model_config[\"model_name\"], \n **model_config[\"model_kwargs\"]\n )\n tokenizer = AutoTokenizer.from_pretrained(model_config[\"model_name\"])\n return (model, tokenizer)\n\n# Create grid search over all configurations\nconfig_group = RFGridSearch(configs=config_set, trainer_type=\"SFT\")\n\n# Run all 4 configurations concurrently with chunk-based scheduling\nexperiment.run_fit(config_group, create_model, train_dataset, eval_dataset, \n num_chunks=4, seed=42)\n\n# End experiment\nexperiment.end()\n```\n\n### What Happens During Execution\n\nWhen you run this example:\n\n1. **Config Expansion**: 2 base configurations ร— 2 PEFT configs = 4 total training runs\n2. **Chunk-based Scheduling**: Training data is divided into chunks, and all 4 configs train concurrently\n3. **GPU Swapping**: Models are swapped in/out of GPU memory based on chunk boundaries\n4. **Real-time Tracking**: All metrics visible in the dashboard at `http://localhost:3000`\n5. **Interactive Control**: Stop, resume, or clone any configuration from the dashboard\n\nThis delivers **16-24ร— higher throughput** compared to training each configuration sequentially!\n\n## Supported TRL Trainers\n\n### SFTTrainer\n\nUse `RFSFTConfig` as a drop-in replacement for `SFTConfig`:\n\n```python\nfrom rapidfireai.automl import RFSFTConfig\n\ntraining_args = RFSFTConfig(\n learning_rate=5e-5,\n per_device_train_batch_size=4,\n num_train_epochs=3,\n max_length = 512,\n # ... all other SFTConfig parameters supported\n)\n```\n\n**Example Notebook**: [SFT for Customer Support](https://github.com/RapidFireAI/rapidfireai/blob/main/tutorial_notebooks/rf-tutorial-sft-chatqa-lite.ipynb)\n\n### DPOTrainer\n\nUse `RFDPOConfig` as a drop-in replacement for `DPOConfig`:\n\n```python\nfrom rapidfireai.automl import RFDPOConfig\n\ntraining_args = RFDPOConfig(\n beta=0.1,\n loss_type=\"sigmoid\",\n max_length=1024,\n learning_rate=5e-4,\n # ... all other DPOConfig parameters supported\n)\n```\n\n**Example Notebook**: [DPO for Preference Alignment](https://github.com/RapidFireAI/rapidfireai/blob/main/tutorial_notebooks/rf-tutorial-dpo-alignment-lite.ipynb)\n\n### GRPOTrainer\n\nUse `RFGRPOConfig` as a drop-in replacement for `GRPOConfig`:\n\n```python\nfrom rapidfireai.automl import RFGRPOConfig\n\ntraining_args = RFGRPOConfig(\n learning_rate=5e-6,\n num_generations=8,\n max_completion_length=256,\n # ... all other GRPOConfig parameters supported\n)\n```\n\n**Example Notebook**: [GRPO for Math Reasoning](https://github.com/RapidFireAI/rapidfireai/blob/main/tutorial_notebooks/rf-tutorial-grpo-mathreasoning-lite.ipynb)\n\n## Core Concepts\n\n### Chunk-Based Concurrent Training\n\nRapidFire AI divides training data into chunks and alternates between configurations:\n\n```\nGPU Timeline (Single GPU):\nChunk 1: [Config A] โ†’ [Config B] โ†’ [Config C] โ†’ [Config D]\nChunk 2: [Config A] โ†’ [Config B] โ†’ [Config C] โ†’ [Config D]\nChunk 3: [Config A] โ†’ [Config B] โ†’ [Config C] โ†’ [Config D]\n...\n```\n\nThis approach maximizes GPU utilization and enables early comparison of configurations while maintaining training stability through automatic checkpointing.\n\n### Interactive Control Operations (IC Ops)\n\nThrough the RapidFire AI dashboard, you can dynamically control running experiments:\n\n- **Stop**: Pause a configuration (checkpointed automatically)\n- **Resume**: Continue from last checkpoint\n- **Clone**: Duplicate a configuration with modifications\n- **Clone & Warm Start**: Clone and initialize from parent's weights\n- **Delete**: Remove failed or unwanted runs\n\nThis enables adaptive experimentation where you can stop underperforming configs early and clone promising ones with tweaked hyperparameters.\n\n### Multi-Config Experimentation\n\nUse `RFGridSearch` or `RFRandomSearch` to automatically generate configuration combinations:\n\n```python\n# Grid search: tests all combinations\nconfig_group = RFGridSearch(configs=config_list, trainer_type=\"SFT\")\n\n# Random search: samples N configurations\nconfig_group = RFRandomSearch(configs=config_list, trainer_type=\"DPO\", num_samples=10)\n```\n\n## Advanced Features\n\n### PEFT/LoRA Integration\n\nFull support for parameter-efficient fine-tuning:\n\n```python\nfrom rapidfireai.automl import RFLoraConfig\nfrom peft import TaskType\n\nlora_config = RFLoraConfig(\n task_type=TaskType.CAUSAL_LM,\n r=64,\n lora_alpha=64,\n lora_dropout=0.1,\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n bias=\"none\"\n)\n```\n\n### Custom Reward Functions (GRPO)\n\nDefine multiple reward functions for GRPO training:\n\n```python\ndef correctness_reward(prompts, completions, answer, **kwargs):\n \"\"\"Reward for correct answers\"\"\"\n responses = [completion[0]['content'] for completion in completions]\n extracted = [extract_answer(r) for r in responses]\n return [2.0 if r == a else 0.0 for r, a in zip(extracted, answer)]\n\ndef format_reward(completions, **kwargs):\n \"\"\"Reward for proper formatting\"\"\"\n import re\n pattern = r\".*?\\s*.*?\"\n responses = [completion[0][\"content\"] for completion in completions]\n matches = [re.match(pattern, r) for r in responses]\n return [0.5 if match else 0.0 for match in matches]\n\n# Use in model config\nconfig = RFModelConfig(\n reward_funcs=[correctness_reward, format_reward],\n # ... other parameters\n)\n```\n\n### Multi-GPU Support\n\nRapidFire AI automatically detects and utilizes all available GPUs. No special configuration needed - the scheduler automatically distributes configurations across GPUs.\n\n## Best Practices\n\n### Tuning Chunk Granularity\n\nThe `num_chunks` parameter controls swap frequency:\n\n```python\n# Fewer chunks = less overhead, less frequent comparison\nexperiment.run_fit(..., num_chunks=2)\n\n# More chunks = more overhead, more frequent comparison\nexperiment.run_fit(..., num_chunks=16)\n```\n\n**Rule of thumb**: Start with `num_chunks=4` and adjust based on dataset size and number of configurations.\n\n### Memory Management\n\nFor large models, use quantization:\n\n```python\nfrom transformers import BitsAndBytesConfig\nimport torch\n\nbnb_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_compute_dtype=torch.bfloat16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type=\"nf4\",\n)\n\nmodel_kwargs = {\n \"quantization_config\": bnb_config,\n \"device_map\": \"auto\",\n}\n```\n\n## Performance Benchmarks\n\nBased on internal benchmarks comparing sequential vs. RapidFire AI concurrent training:\n\n| Scenario | Sequential Time | RapidFire AI Time | Speedup |\n|----------|----------------|-------------------|---------|\n| 4 configs, 1 GPU | 120 min | 7.5 min | 16ร— |\n| 8 configs, 1 GPU | 240 min | 12 min | 20ร— |\n| 4 configs, 2 GPUs | 60 min | 4 min | 15ร— |\n| 8 configs, 4 GPUs | 60 min | 3 min | 20ร— |\n\n*Benchmarks performed on NVIDIA A100 40GB with TinyLlama-1.1B and Llama-3.2-1B models*\n\n## Troubleshooting\n\nFor troubleshooting guidance, see the [RapidFire AI Troubleshooting Guide](https://oss-docs.rapidfire.ai/en/latest/troubleshooting.html).\n\n## Additional Resources\n- **Colab Notebook**: [RapidFire AI in Google Colab](http://tinyurl.com/rapidfireai-colab)\n- **Documentation**: [oss-docs.rapidfire.ai](https://oss-docs.rapidfire.ai)\n- **GitHub**: [RapidFireAI/rapidfireai](https://github.com/RapidFireAI/rapidfireai)\n- **PyPI**: [pypi.org/project/rapidfireai](https://pypi.org/project/rapidfireai/)\n- **Discord**: [Join our Discord](https://discord.gg/6vSTtncKNN)\n- **Tutorial Notebooks**: [GitHub Repository](https://github.com/RapidFireAI/rapidfireai/tree/main/tutorial_notebooks)\n\nLearn more about RapidFire AI in their [official repository](https://github.com/RapidFireAI/rapidfireai) and [documentation](https://oss-docs.rapidfire.ai).\n\n" }, { "path": "docs/source/reducing_memory_usage.md", "content": "# Reducing Memory Usage\n\nTraining workflows can often be optimized to **reduce memory consumption**, and TRL provides several built-in features to help achieve this.\n\nBelow, we outline these techniques and recommend experimenting with different combinations to figure out which configuration works best for your specific setup.\n\nEach method includes examples for the supported trainers. If you're unsure whether a technique is compatible with your trainer, please take a look at the corresponding trainer documentation.\n\nFor additional strategies, such as **gradient checkpointing**, which is supported across all trainers, see the [`transformers` performance guide](https://huggingface.co/docs/transformers/perf_train_gpu_one#gradient-checkpointing).\n\n## Truncation\n\nSequence lengths in the dataset can vary widely. When data is batched, sequences are padded to match the longest one in the batch, which can cause high memory usage, even if most sequences are relatively short.\n\n![Truncation prompt-completion](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/why_you_should_truncate.png)\n\nTo reduce memory usage, it's important to truncate sequences to a reasonable length. While TRL trainers truncate sequences by default, you may want to adjust the default truncation length to better align with your specific use case.\n\n\n\n\nDPO truncation is controlled via `max_length`, which truncates the combined prompt+completion sequence.\n\n![DPO truncation](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/truncation_prompt_completion.png)\n\nTo set the truncation parameter, use the following code snippet:\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(..., max_length=...)\n```\n\n> [!WARNING]\n> The legacy `max_prompt_length` and `max_completion_length` parameters are now removed; instead, filter or pre-truncate overlong prompts/completions in your dataset before training.\n\n\n\n\nSFT truncation is applied to the input sequence via the `max_length` parameter.\n\n![Truncation input ids](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/truncation_input_ids.png)\n\nTo set the truncation parameter, use the following code snippet:\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., max_length=...)\n```\n\n\n\n\n### How to choose the `max_length` value?\n\nIf `max_length` is too small, a significant portion of your tokens will be discarded and won't contribute to training. If it's too large, memory usage can spike, potentially leading to out-of-memory (OOM) errors. Without packing or padding-free, a large `max_length` may also result in inefficient training, as many tokens will be padding.\n\nTo help you choose an appropriate value, we provide a utility to visualize the sequence length distribution in your dataset.\n\n\n\n## Packing\n\n> [!TIP]\n> This technique is available only for **SFT** training and setups that use **FlashAttention** (or its variants).\n\n[Truncation](#truncation) has several drawbacks:\n\n1. **Loss of information**: Important tokens at the end of sequences may be discarded.\n2. **Choosing truncation length**: Too short loses data; too long reduces efficiency.\n\nPacking mitigates these issues by grouping multiple sequences into the same training row, filling each row up to `max_length`.\n\n![Packing](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/packing_3.png)\n\nTRL implements packing using **Best-Fit Decreasing (BFD)** bin packing, which groups sequences efficiently while minimizing padding. When a sequence exceeds `max_length`, different strategies determine how the overflow tokens are handled.\n\nTRL supports three strategies:\n\n* `\"bfd\"` (default): Uses **Best-Fit Decreasing packing**. If a sequence exceeds `max_length`, the overflow tokens are discarded.\n\n* `\"bfd_split\"`: Uses **Best-Fit Decreasing packing**, but long sequences are split into chunks โ‰ค `max_length` before packing. This preserves all tokens and follows the approach proposed in [Fewer Truncations Improve Language Modeling](https://huggingface.co/papers/2404.10830).\n\n* `\"wrapped\"`: All tokens are concatenated into a stream and split into fixed-length blocks. This minimizes padding but may mix unrelated examples. This strategy corresponds to the *concatenate-then-split* preprocessing described in the literature (e.g., [Fewer Truncations Improve Language Modeling](https://huggingface.co/papers/2404.10830)). It has the downside of breaking sequence continuity for a large fraction of the dataset, which hurts performance, as discussed in the [Qwen3-Coder-Next Technical Report](https://huggingface.co/papers/2603.00729).\n\n> [!NOTE]\n> If all sequences are shorter than `max_length`, **`bfd` and `bfd_split` behave identically**, since no truncation or splitting is required.\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(\n ...,\n packing=True,\n packing_strategy=\"bfd\",\n max_length=512,\n)\n```\n\n## PEFT for parameter-efficient fine-tuning\n\nParameter-Efficient Fine-Tuning (PEFT) methods like LoRA are among the most effective techniques for reducing memory usage during training. Instead of training all model parameters, PEFT methods train only a small number of adapter parameters, significantly reducing memory requirements and enabling fine-tuning of larger models on limited hardware.\n\nFor comprehensive details on using PEFT with TRL, including various adapter methods, quantization options, and advanced configurations, see [PEFT Integration](peft_integration).\n\nTo use PEFT for reducing memory usage:\n\n```python\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom trl import SFTTrainer\n\ndataset = load_dataset(\"trl-lib/Capybara\", split=\"train\")\n\npeft_config = LoraConfig()\n\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen2.5-0.5B\",\n train_dataset=dataset,\n peft_config=peft_config,\n)\n```\n\nPEFT can be combined with other memory reduction techniques such as quantization (4-bit or 8-bit) for even greater memory savings. See [PEFT Integration](peft_integration) for quantization examples.\n\n## Liger for reducing peak memory usage\n\n[Liger Kernel](https://github.com/linkedin/Liger-Kernel) is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduce memory usage by 60%.\n\nFor more information, see [Liger Kernel Integration](liger_kernel_integration).\n\nTo use Liger for reducing peak memory usage, use the following code snippet:\n\n\n\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl.experimental.kto import KTOConfig\n\ntraining_args = KTOConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl.experimental.gkd import GKDConfig\n\ntraining_args = GKDConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n## Padding-free\n\nPadding-free batching is an alternative approach for reducing memory usage. In this method, a batch is first sampled and then flattened into a single sequence, avoiding padding. Unlike packing, which can result in incomplete sequences by combining parts of different samples, padding-free batching ensures that all sequences remain complete and intact.\n\n![Padding-free](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/padding-free.png)\n\n> [!WARNING]\n> It's highly recommended to use padding-free batching with **FlashAttention 2** or **FlashAttention 3**. Otherwise, you may encounter batch contamination issues.\n\n\n\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(..., padding_free=True, model_init_kwargs={\"attn_implementation\": \"kernels-community/flash-attn2\"})\n```\n\n\n\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., padding_free=True, model_init_kwargs={\"attn_implementation\": \"kernels-community/flash-attn2\"})\n```\n\n\n\n\n## Activation offloading\n\nActivation offloading is a memory efficiency technique that reduces GPU VRAM usage by temporarily moving activation tensors to CPU RAM during the forward pass and bringing them back only when needed for the backward pass. This significantly reduces peak memory usage at the cost of slightly increased training time.\n\nTo enable activation offloading in your SFT training configuration:\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., activation_offloading=True)\n```\n\nUnder the hood, activation offloading implements PyTorch's [`saved_tensors_hooks`](https://pytorch.org/tutorials/intermediate/autograd_saved_tensors_hooks_tutorial.html#hooks-for-autograd-saved-tensors) to intercept activations during the forward pass. It intelligently manages which tensors to offload based on size and context, avoiding offloading output tensors that would be inefficient. For performance optimization, it can, via a flag (which is true by default), use CUDA streams to overlap computation with CPU-GPU transfers.\n\n## Padding Sequences to a Multiple\n\n> [!TIP]\n> This technique is supported for **SFT** and **Reward** trainers currently.\n\nWhen enabled, this option ensures that all sequences are **padded to a multiple** of the specified value. \nThis can improve computational efficiency on some hardware by aligning sequence lengths to memory-friendly boundaries.\n\n\n\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., pad_to_multiple_of=2048)\n```\n\n\n\n\n```python\nfrom trl import RewardConfig\n\ntraining_args = RewardConfig(..., pad_to_multiple_of=2048)\n```\n\n\n\n\n## Disabling model gathering for generation in online methods\n\nWhen using DeepSpeed ZeRO-3, model weights are sharded across multiple GPUs. Online methods involve generating completions from the model as part of the training process. During this step, the model weights are temporarily gathered on a single GPU for generation. For very large models, this gathering can lead to OOM errors, as described in this issue: [#2250](https://github.com/huggingface/trl/issues/2250#issue-2598304204).\n\nIf you encounter this issue, you can disable the gathering of model weights for generation by setting the following parameter:\n\n\n\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(..., ds3_gather_for_generation=False)\n```\n\n\n\n\n```python\nfrom trl.experimental.online_dpo import OnlineDPOConfig\n\ntraining_args = OnlineDPOConfig(..., ds3_gather_for_generation=False)\n```\n\n\n\n\n```python\nfrom trl.experimental.ppo import PPOConfig\n\ntraining_args = PPOConfig(..., ds3_gather_for_generation=False)\n```\n\n\n\n\n```python\nfrom trl import RLOOConfig\n\ntraining_args = RLOOConfig(..., ds3_gather_for_generation=False)\n```\n\n\n\n\nThis adjustment prevents model weights from being gathered, avoiding OOM errors, but it may result in slower generation speeds.\n\n## vLLM sleep mode\n\nWhen using **vLLM** as the generation backend for online training methods, you can enable _sleep mode_ to offload vLLM parameters and cache to CPU RAM during the optimization step and reload them back to GPU VRAM when needed for weight synchronization and generation.\n\n\n\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(..., vllm_enable_sleep_mode=True)\n```\n\n\n\n\n```python\nfrom trl import RLOOConfig\n\ntraining_args = RLOOConfig(..., vllm_enable_sleep_mode=True)\n```\n\n\n\n\nOffloading the vLLM weights and cache helps keep GPU memory usage low, which can be particularly beneficial when training large models or using limited GPU resources. However, waking the vLLM engine from sleep mode introduces some hostโ€“device transfer latency, which may slightly impact training speed.\n\n## Gradient checkpointing\n\nGradient checkpointing trades compute for memory by not storing all intermediate activations during the forward pass, recomputing them during the backward pass instead.\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., gradient_checkpointing=True)\n```\n\n> [!NOTE]\n> Gradient checkpointing is enabled by default in all trainers to optimize memory usage. You can disable it by setting `gradient_checkpointing=False` if needed.\n\nFor more memory optimization techniques, see the [Transformers Performance Guide](https://huggingface.co/docs/transformers/perf_train_gpu_one#gradient-checkpointing).\n" }, { "path": "docs/source/reward_trainer.md", "content": "# Reward Modeling\n\n[![model badge](https://img.shields.io/badge/All_models-Reward_Trainer-blue)](https://huggingface.co/models?other=reward-trainer,trl)\n\n## Overview\n\nTRL supports the Outcome-supervised Reward Modeling (ORM) Trainer for training reward models.\n\nThis post-training method was contributed by [Younes Belkada](https://huggingface.co/ybelkada).\n\n## Quick start\n\nThis example demonstrates how to train a reward model using the [`RewardTrainer`] from TRL. We train a [Qwen 3 0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) model on the [UltraFeedback dataset](https://huggingface.co/datasets/trl-lib/ultrafeedback_binarized), large-scale, fine-grained, diverse preference dataset.\n\n```python\nfrom trl import RewardTrainer\nfrom datasets import load_dataset\n\ntrainer = RewardTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n train_dataset=load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\"),\n)\ntrainer.train()\n```\n\n\n\n## Expected dataset type and format\n\n[`RewardTrainer`] supports [preference](dataset_formats#preference) datasets type (both implicit and explicit prompt). The [`RewardTrainer`] is compatible with both [standard](dataset_formats#standard) and [conversational](dataset_formats#conversational) dataset formats. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\n```python\n# Standard preference (implicit prompt)\n{\"chosen\": \"The sky is blue.\",\n \"rejected\": \"The sky is green.\"}\n\n# Conversational preference (implicit prompt)\n{\"chosen\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is green.\"}]}\n\n# Standard preference (explicit prompt)\n{\"prompt\": \"The sky is\",\n \"chosen\": \" blue.\",\n \"rejected\": \" green.\"}\n\n# Conversational preference (explicit prompt)\n{\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": \"It is green.\"}]}\n```\n\nIf your dataset is not in one of these formats, you can preprocess it to convert it into the expected format. Here is an example with the [lmarena-ai/arena-human-preference-55k](https://huggingface.co/datasets/lmarena-ai/arena-human-preference-55k) dataset:\n\n```python\nfrom datasets import load_dataset\nimport json\n\ndataset = load_dataset(\"lmarena-ai/arena-human-preference-55k\")\n\n# Filter out ties\ndataset = dataset.filter(lambda example: example[\"winner_tie\"] == 0)\n\n# Create 'chosen' and 'rejected' fields based on the winner column\ndef response_a_b_to_chosen_rejected(example):\n if example[\"winner_model_a\"] == 1:\n example[\"chosen\"] = example[\"response_a\"]\n example[\"rejected\"] = example[\"response_b\"]\n else:\n example[\"chosen\"] = example[\"response_b\"]\n example[\"rejected\"] = example[\"response_a\"]\n return example\n\ndataset = dataset.map(response_a_b_to_chosen_rejected)\n\n# Convert to conversational format\ndef make_conversation(example):\n prompt = json.loads(example[\"prompt\"])[0] # '[\"What color is the sky?\"]' -> \"What color is the sky?\"\n chosen = json.loads(example[\"chosen\"])[0]\n rejected = json.loads(example[\"rejected\"])[0]\n return {\n \"chosen\": [{\"role\": \"user\", \"content\": prompt}, {\"role\": \"assistant\", \"content\": chosen}],\n \"rejected\": [{\"role\": \"user\", \"content\": prompt}, {\"role\": \"assistant\", \"content\": rejected}],\n }\n\n\ndataset = dataset.map(make_conversation)\n\n# Keep only necessary columns\ndataset = dataset.select_columns([\"chosen\", \"rejected\"])\n\nprint(next(iter(dataset[\"train\"])))\n```\n\n```json\n{\n \"chosen\": [\n {\"role\": \"user\", \"content\": \"Is it morally right to try to have a certain percentage of females on managerial positions?\"},\n {\"role\": \"assistant\", \"content\": \"The question of whether it is morally right to aim for a certain percentage of females...\"},\n ],\n \"rejected\": [\n {\"role\": \"user\", \"content\": \"Is it morally right to try to have a certain percentage of females on managerial positions?\"},\n {\"role\": \"assistant\", \"content\": \"As an AI, I don't have personal beliefs or opinions. However, ...\"},\n ],\n}\n```\n\n## Looking deeper into the training method\n\nReward Models (RMs) are typically trained using supervised learning on datasets containing pairs of preferred and non-preferred responses. The goal is to learn a function that assigns higher scores to preferred responses, enabling the model to rank outputs based on preferences.\n\nThis section breaks down how reward modeling works in practice, covering the key steps: **preprocessing** and **loss computation**.\n\n### Preprocessing and tokenization\n\nDuring training, each example is expected to contain a **chosen** and **rejected** field. For more details on the expected formats, see [Dataset formats - Preference](dataset_formats#preference).\nThe [`RewardTrainer`] tokenizes each input using the model's tokenizer. If prompts and completions (chosen and rejected) are provided separately (explicit prompt case), they are concatenated before tokenization.\n\n### Computing the loss\n\nLet \\\\( x \\\\) be the input sequence (prompt) and \\\\( y^+ \\\\) and \\\\( y^- \\\\) be the chosen and rejected sequences respectively. Under the Bradley-Terry model ([Bradley & Terry, 1952](https://www.jstor.org/stable/2334029)), the probability that \\\\( y^+ \\\\) is preferred over \\\\( y^- \\\\) given a reward function \\\\( r \\\\) is \\\\( p(y^+ โ‰ป y^- |x) = \\sigma(r(x, y^+)โˆ’r(x, y^-)) \\\\), where \\\\( ฯƒ \\\\) is the sigmoid function.\n\nThe reward model \\\\( r_\\theta(x, y) \\\\) is trained to assign higher scores to preferred responses \\\\( y^+ \\\\) over non-preferred ones \\\\( y^- \\\\). The loss is then defined as the negative log-likelihood of the observed preferences:\n\n$$\n\\mathcal{L}(\\theta) = - \\mathbb{E}_{(x,y^+,y^-) \\sim \\mathcal{D}} \\left[ \\log \\sigma(r_\\theta(x, y^+) - r_\\theta(x, y^-)) \\right].\n$$\n\n> [!TIP]\n> The Bradley-Terry model is underdetermined, meaning that adding a constant to all rewards does not change the preference probabilities. To address this, [Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking](https://huggingface.co/papers/2312.09244) proposes adding an auxiliary loss term that encourages the rewards to be centered around zero. This is controlled by the `center_rewards_coefficient` parameter in the [`RewardConfig`]. The recommended value is `1e-2`.\n\n## Logged metrics\n\nWhile training and evaluating we record the following reward metrics:\n\n* `global_step`: The total number of optimizer steps taken so far.\n* `epoch`: The current epoch number, based on dataset iteration.\n* `num_tokens`: The total number of tokens processed so far.\n* `loss`: The average loss over the last logging interval.\n* `accuracy`: The proportion of correct predictions (i.e., the model assigned a higher score to the chosen response than to the rejected one) averaged over the last logging interval.\n* `min_reward`: The minimum reward score assigned by the model. This value is averaged over the logging interval.\n* `mean_reward`: The average reward score assigned by the model over the last logging interval.\n* `max_reward`: The maximum reward score assigned by the model. This value is averaged over the logging interval.\n* `margin`: The average margin (difference between chosen and rejected rewards) over the last logging interval.\n* `learning_rate`: The current learning rate, which may change dynamically if a scheduler is used.\n* `grad_norm`: The L2 norm of the gradients, computed before gradient clipping.\n\n## Customization\n\n### Model initialization\n\nYou can directly pass the kwargs of the [`~transformers.AutoModelForSequenceClassification.from_pretrained()`] method to the [`RewardConfig`]. For example, if you want to load a model in a different precision, analogous to\n\n```python\nmodel = AutoModelForSequenceClassification.from_pretrained(\"Qwen/Qwen3-0.6B\", dtype=torch.bfloat16)\n```\n\nyou can do so by passing the `model_init_kwargs={\"dtype\": torch.bfloat16}` argument to the [`RewardConfig`].\n\n```python\nfrom trl import RewardConfig\n\ntraining_args = RewardConfig(\n model_init_kwargs={\"dtype\": torch.bfloat16},\n)\n```\n\nNote that all keyword arguments of [`~transformers.AutoModelForSequenceClassification.from_pretrained()`] are supported, except for `num_labels`, which is automatically set to 1.\n\n### Train adapters with PEFT\n\nWe support tight integration with ๐Ÿค— PEFT library, allowing any user to conveniently train adapters and share them on the Hub, rather than training the entire model.\n\n```python\nfrom datasets import load_dataset\nfrom trl import RewardTrainer\nfrom peft import LoraConfig\n\ndataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\ntrainer = RewardTrainer(\n \"Qwen/Qwen3-4B\",\n train_dataset=dataset,\n peft_config=LoraConfig(modules_to_save=[\"score\"]) # important to include the score head when base model is not a sequence classification model\n)\n\ntrainer.train()\n```\n\nYou can also continue training your [`~peft.PeftModel`]. For that, first load a `PeftModel` outside [`RewardTrainer`] and pass it directly to the trainer without the `peft_config` argument being passed.\n\n```python\nfrom datasets import load_dataset\nfrom trl import RewardTrainer\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\"trl-lib/Qwen3-4B-Reward-LoRA\", is_trainable=True)\ndataset = load_dataset(\"trl-lib/Capybara\", split=\"train\")\n\ntrainer = RewardTrainer(\n model=model,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\n> [!TIP]\n> When training adapters, you typically use a higher learning rate (โ‰ˆ1eโ€‘3) since only new parameters are being learned.\n>\n> ```python\n> RewardConfig(learning_rate=1e-3, ...)\n> ```\n\n## Tool Calling with Reward Modeling\n\nThe [`RewardTrainer`] fully supports fine-tuning models with _tool calling_ capabilities. In this case, each dataset example should include:\n\n* The conversation messages, including any tool calls (`tool_calls`) and tool responses (`tool` role messages)\n* The list of available tools in the `tools` column, typically provided as JSON schemas\n\nFor details on the expected dataset structure, see the [Dataset Format โ€” Tool Calling](dataset_formats#tool-calling) section.\n\n## RewardTrainer\n\n[[autodoc]] RewardTrainer\n - train\n - save_model\n - push_to_hub\n\n## RewardConfig\n\n[[autodoc]] RewardConfig\n" }, { "path": "docs/source/rewards.md", "content": "# Reward Functions\n\nThis module contains some useful reward functions, primarily intended for use with the [`GRPOTrainer`] and [`RLOOTrainer`].\n\n## accuracy_reward\n\n[[autodoc]] rewards.accuracy_reward\n\n## reasoning_accuracy_reward\n\n[[autodoc]] rewards.reasoning_accuracy_reward\n\n## think_format_reward\n\n[[autodoc]] rewards.think_format_reward\n\n## get_soft_overlong_punishment\n\n[[autodoc]] rewards.get_soft_overlong_punishment\n" }, { "path": "docs/source/rloo_trainer.md", "content": "# RLOO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-RLOO-blue)](https://huggingface.co/models?other=rloo,trl)\n\n## Overview\n\nTRL supports the RLOO Trainer for training language models, as described in the paper [Back to Basics: Revisiting REINFORCE Style\nOptimization for Learning from Human Feedback in LLMs](https://huggingface.co/papers/2402.14740) by [Arash Ahmadian](https://huggingface.co/ArashAhmadian), Chris Cremer, [Matthias Gallรฉ](https://huggingface.co/mgalle), [Marzieh Fadaee](https://huggingface.co/MarziehFadaee), [Julia Kreutzer](https://huggingface.co/JuliaKreutzerCohere), [Ahmet รœstรผn](https://huggingface.co/ahmetu) and [Sara Hooker](https://huggingface.co/sarahooker).\n\nThe abstract from the paper is the following:\n\n> AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models. Proximal Policy Optimization (PPO) has been positioned by recent literature as the canonical method for the RL part of RLHF However, it involves both high computational cost and sensitive hyperparameter tuning. We posit that most of the motivational principles that led to the development of PPO are less of a practical concern in RLHF and advocate for a less computationally expensive method that preserves and even increases performance. We revisit the formulation of alignment from human preferences in the context of RL. Keeping simplicity as a guiding principle, we show that many components of PPO are unnecessary in an RLHF context and that far simpler REINFORCE-style optimization variants outperform both PPO and newly proposed โ€œRL-freeโ€ methods such as DPO and RAFT. Our work suggests that careful adaptation to LLMs alignment characteristics enables benefiting from online RL optimization at low cost.\n\nThis post-training method was contributed by [Costa Huang](https://github.com/vwxyzjn) and later refactored by [Shirin Yamani](https://huggingface.co/ShirinYamani).\n\n## Quick start\n\nThis example demonstrates how to train a model using the RLOO method. We train a [Qwen 0.5B Instruct model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) with the prompts from the [DeepMath-103K dataset](https://huggingface.co/datasets/trl-lib/DeepMath-103K). You can view the data in the dataset here:\n\n\n\nBelow is the script to train the model.\n\n```python\n# train_rloo.py\nfrom datasets import load_dataset\nfrom trl import RLOOTrainer\nfrom trl.rewards import accuracy_reward\n\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\ntrainer = RLOOTrainer(\n model=\"Qwen/Qwen2-0.5B-Instruct\",\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\nExecute the script using the following command:\n\n```bash\naccelerate launch train_rloo.py\n```\n\n## Looking deeper into the RLOO method\n\nRLOO is an online learning algorithm, meaning it improves iteratively by using the data generated by the trained model itself during training. The intuition behind RLOO objective is to maximize the advantage of the generated completions, while ensuring that the model remains close to the reference policy. To understand how RLOO works, it can be broken down into four main steps: **Generating completions**, **computing the advantage**, **estimating the KL divergence**, and **computing the loss**.\n\n![RLOO](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/rloo.png)\n\n### Generating completions\n\nAt each training step, we sample a batch of prompts and generate a set of \\\\( G \\\\) completions for each prompt (denoted as \\\\( o_i \\\\)).\n\n### Computing the reward\n\nIn RLOO, the reward consists of two components: the reward provided by the reward model (or reward function) and a KL penalty that discourages the policy from deviating too far from a fixed reference policy\n\n1. For each of the \\\\( G \\\\) generated sequences \\\\( o_i = (o_{i,1}, \\dots, o_{i,T}) \\\\) conditioned on a query \\\\( q \\\\), we compute a scalar reward using a reward model \\\\( R(o_i, q) \\\\).\n2. Concurrently, we estimate the KL divergence between the current policy \\\\( \\pi_\\theta \\\\) and the fixed reference policy \\\\( \\pi_{\\text{ref}} \\\\) over the sequence. The KL estimate for sequence \\\\( o_i \\\\) is:\n\n$$\n\\mathbb{D}_{\\mathrm{KL}}\\!\\left[\\pi_\\theta\\|\\pi_{\\mathrm{ref}}\\right] = \\sum_{t=1}^T \\log \\frac{\\pi_\\theta(o_{i,t} \\mid q, o_{i, 0 \\\\) controls the strength of the KL penalty.\n\n> [!TIP]\n> In a purely online setting (`num_iterations = 1`, default), the data are generated by the current policy. In this case, the KL penalty is computed directly using the current policy. \n>\n> In the more general setting (e.g., multiple gradient steps per batch), the data are instead generated by an earlier snapshot \\\\( \\pi_{\\text{old}} \\\\). To keep the penalty consistent with the sampling distribution, the KL is defined with respect to this policy:\n>\n> $$\n> \\mathbb{D}_{\\mathrm{KL}}\\!\\left[\\pi_{\\text{old}} \\,\\|\\, \\pi_{\\text{ref}}\\right].\n> $$\n>\n> Equivalently, for a sampled sequence $o$, the Monte Carlo estimate is\n>\n> $$\n> \\mathbb{D}_{\\mathrm{KL}}\\!\\left[\\pi_{\\text{old}} \\|\\pi_{\\mathrm{ref}}\\right] = \\sum_{t=1}^T \\log \\frac{\\pi_{\\text{old}}(o_{i,t} \\mid q, o_{i, $$\n\n### Computing the advantage\n\nOnce the rewards for each completion have been computed, we calculate a baseline as the average reward of all other samples in the same batch, excluding the current sample. This baseline is used to reduce the variance of the policy gradient estimate. The advantage for each completion is then obtained as the difference between its own reward and this leave-one-out baseline.\n\nFormally, for a batch of G completions, the baseline for completion is:\n$$\nb_i = \\frac{1}{G-1} \\sum_{j \\neq i} r_j\n$$\n\nand then the advantage for each completion is computed as the difference between its reward and the baseline:\n\n$$\nA_i = r_i - b_i\n$$\n\n### Computing the loss\n\nThe REINFORCE loss is simply defined as:\n\n$$\n\\mathcal{L}_{\\text{RLOO}}(\\theta) = - \\frac{1}{G} \\sum_{i=1}^G \\hat{A}_i \\, \\log \\pi_\\theta(o_i \\mid q)\n$$\n\nIn practice, performing multiple gradient steps on the same batch makes the actions effectively off-policy relative to the current parameters. To correct for this, we introduce the importance sampling ratio. To prevent excessively large updates when the policy changes between sampling and gradient steps, we clip this ratio:\n\n$$\n\\mathcal{L}_{\\text{RLOO}}(\\theta) = - \\frac{1}{G} \\sum_{i=1}^G \\min \\left( \\frac{\\pi_\\theta(o_i \\mid q)}{\\pi_{\\theta_\\text{old}}(o_i \\mid q)} \\hat{A}_i, \\, \\text{clip}\\left(\\frac{\\pi_\\theta(o_i \\mid q)}{\\pi_{\\theta_\\text{old}}(o_i \\mid q)}, 1-\\epsilon, 1+\\epsilon\\right) \\hat{A}_i \\right)\n$$\n\nIn a fully online, single-step setting (default), \\\\( \\frac{\\pi_\\theta(o_i \\mid q)}{\\pi_{\\theta_\\text{old}}(o_i \\mid q)} = 1 \\\\) and this reduces to standard REINFORCE.\n\n## Logged metrics\n\nWhile training and evaluating, we record the following reward metrics:\n\n- `num_tokens`: The total number of tokens processed so far, including both prompts and completions.\n- `step_time`: The average time (in seconds) taken per training step (including generation).\n- `completions/mean_length`: The average length of generated completions.\n- `completions/min_length`: The minimum length of generated completions.\n- `completions/max_length`: The maximum length of generated completions.\n- `completions/mean_terminated_length`: The average length of generated completions that terminate with EOS.\n- `completions/min_terminated_length`: The minimum length of generated completions that terminate with EOS.\n- `completions/max_terminated_length`: The maximum length of generated completions that terminate with EOS.\n- `completions/clipped_ratio`: The ratio of truncated (clipped) completions.\n- `reward/{reward_func_name}/mean`: The average reward from a specific reward function.\n- `reward/{reward_func_name}/std`: The standard deviation of the reward from a specific reward function.\n- `reward`: The overall average reward after summing rewards across functions (unweighted).\n- `reward_std`: The standard deviation of summed rewards across functions (unweighted), computed over the full batch.\n- `frac_reward_zero_std`: The fraction of samples in the generation batch with a reward std of zero, implying there is little diversity for that prompt (all answers are correct or incorrect).\n- `entropy`: Average entropy of token predictions across generated completions. (If `mask_truncated_completions=True`, masked sequences tokens are excluded.)\n- `kl`: The average KL divergence between the model and the reference model, calculated over generated completions. Logged only if `beta` is nonzero.\n- `clip_ratio/region_mean`: The ratio of sequence probabilities where the RLOO objective is clipped to stay within the trust region: \\\\( \\text{clip}\\left( r_{i}(\\theta), 1 - \\epsilon_\\mathrm{low}, 1 + \\epsilon_\\mathrm{high} \\right)\\,, \\quad r_{i}(\\theta) = \\frac{\\pi_\\theta(o_{i} \\mid q)}{\\pi_{\\theta_{\\text{old}}}(o_{i} \\mid q)} \\\\). A higher value means more samples are clipped, which constrains how much the policy $\\pi_\\theta$ can change.\n- `clip_ratio/low_mean`: The average ratio of sequence probabilities that were clipped on the lower bound of the trust region: \\\\(r_{i,t}(\\theta) < 1 - \\epsilon_\\mathrm{low}\\\\).\n- `clip_ratio/low_min`: The minimum ratio of sequence probabilities that were clipped on the lower bound of the trust region: \\\\(r_{i,t}(\\theta) < 1 - \\epsilon_\\mathrm{low}\\\\).\n- `clip_ratio/high_mean`: The average ratio of sequence probabilities that were clipped on the upper bound of the trust region: \\\\(r_{i,t}(\\theta) > 1 + \\epsilon_\\mathrm{high}\\\\).\n- `clip_ratio/high_max`: The maximum ratio of sequence probabilities that were clipped on the upper bound of the trust region: \\\\(r_{i,t}(\\theta) > 1 + \\epsilon_\\mathrm{high}\\\\).\n\n## Customization\n\n### Speed up training with vLLM-powered generation\n\nGeneration is often the main bottleneck when training with online methods. To accelerate generation, you can use [vLLM](https://github.com/vllm-project/vllm), a high-throughput, low-latency inference engine for LLMs. To enable it, first install the package with\n\n```shell\npip install trl[vllm]\n```\n\nWe support two ways of using vLLM during training: **server mode** and **colocate mode**.\n\n#### Option 1: Colocate mode\n\nIn this mode, vLLM runs inside the trainer process and shares GPU memory with the training model. This avoids launching a separate server and can improve GPU utilization, but may lead to memory contention on the training GPUs. This is the default mode.\n\n```python\nfrom trl import RLOOConfig\n\ntraining_args = RLOOConfig(\n ...,\n use_vllm=True, # vllm_mode=\"colocate\" by default\n)\n```\n\n#### Option 2: Server mode\n\nIn this mode, vLLM runs in a separate process (and using separate GPUs) and communicates with the trainer via HTTP. This is ideal if you have dedicated GPUs for inference.\n\n1. **Start the vLLM server**:\n\n ```bash\n trl vllm-serve --model \n ```\n\n2. **Enable server mode in your training script**:\n\n ```python\n from trl import RLOOConfig\n\n training_args = RLOOConfig(\n ...,\n use_vllm=True,\n vllm_mode=\"server\",\n )\n ```\n\n> [!WARNING]\n> Make sure that the server is using different GPUs than the trainer, otherwise you may run into NCCL errors. You can specify the GPUs to use with the `CUDA_VISIBLE_DEVICES` environment variable.\n\n> [!TIP]\n> Depending on the model size and the overall GPU memory requirements for training, you may need to adjust the `vllm_gpu_memory_utilization` parameter in [`RLOOConfig`] to avoid underutilization or out-of-memory errors.\n>\n> We provide a [HF Space](https://huggingface.co/spaces/trl-lib/recommend-vllm-memory) to help estimate the recommended GPU memory utilization based on your model configuration and experiment settings. Simply use it as follows to get `vllm_gpu_memory_utilization` recommendation:\n>\n> \n>\n> If the recommended value does not work in your environment, we suggest adding a small buffer (e.g., +0.05 or +0.1) to the recommended value to ensure stability.\n>\n> If you still find you are getting out-of-memory errors set `vllm_enable_sleep_mode` to True and the vllm parameters and cache will be offloaded during the optimization step. For more information, see [Reducing Memory Usage with vLLM Sleep Mode](reducing_memory_usage#vllm-sleep-mode).\n\n> [!TIP]\n> By default, RLOO uses `MASTER_ADDR=localhost` and `MASTER_PORT=12345` for vLLM, but you can override these values by setting the environment variables accordingly.\n\nFor more information, see [Speeding up training with vLLM](speeding_up_training#vllm-for-fast-generation-in-online-methods).\n\n### RLOO at scale: train a 70B+ Model on multiple nodes\n\nWhen training large models like **Qwen2.5-72B**, you need several key optimizations to make the training efficient and scalable across multiple GPUs and nodes. These include:\n\n- **DeepSpeed ZeRO Stage 3**: ZeRO leverages data parallelism to distribute model states (weights, gradients, optimizer states) across multiple GPUs and CPUs, reducing memory and compute requirements on each device. Since large models cannot fit on a single GPU, using ZeRO Stage 3 is required for training such models. For more details, see [DeepSpeed Integration](deepspeed_integration).\n- **Accelerate**: Accelerate is a library that simplifies distributed training across multiple GPUs and nodes. It provides a simple API to launch distributed training and handles the complexities of distributed training, such as data parallelism, gradient accumulation, and distributed data loading. For more details, see [Distributing Training](distributing_training).\n- **vLLM**: See the previous section on how to use vLLM to speed up generation.\n\nBelow is an example SLURM script to train a 70B model with RLOO on multiple nodes. This script trains a model on 4 nodes and uses the 5th node for vLLM-powered generation.\n\n```sh\n#!/bin/bash\n#SBATCH --nodes=5\n#SBATCH --gres=gpu:8\n\n# Get the list of allocated nodes\nNODELIST=($(scontrol show hostnames $SLURM_JOB_NODELIST))\n\n# Assign the first 4 nodes for training and the 5th node for vLLM\nTRAIN_NODES=\"${NODELIST[@]:0:4}\" # Nodes 0, 1, 2, 3 for training\nVLLM_NODE=\"${NODELIST[4]}\" # Node 4 for vLLM\n\n# Run training on the first 4 nodes (Group 1)\nsrun --nodes=4 --ntasks=4 --nodelist=\"${NODELIST[@]:0:4}\" accelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n --num_processes 32 \\\n --num_machines 4 \\\n --main_process_ip ${NODELIST[0]} \\\n --machine_rank $SLURM_PROCID \\\n --rdzv_backend c10d \\\n train_rloo.py \\\n --server_ip $VLLM_NODE &\n\n# Run vLLM server on the 5th node (Group 2)\nsrun --nodes=1 --ntasks=1 --nodelist=\"${NODELIST[4]}\" trl vllm-serve --model Qwen/Qwen2.5-72B --tensor_parallel_size 8 &\n\nwait\n```\n\n```python\nimport argparse\n\nfrom datasets import load_dataset\nfrom trl import RLOOTrainer, RLOOConfig\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--vllm_server_host\", type=str, default=\"\", help=\"The server IP\")\n args = parser.parse_args()\n\n # Example dataset from TLDR\n dataset = load_dataset(\"trl-lib/tldr\", split=\"train\")\n\n # Dummy reward function: count the number of unique characters in the completions\n def reward_num_unique_chars(completions, **kwargs):\n return [len(set(c)) for c in completions]\n\n training_args = RLOOConfig(\n output_dir=\"Qwen2.5-72B-RLOO\",\n per_device_train_batch_size=4,\n bf16=True,\n use_vllm=True,\n vllm_mode=\"server\",\n vllm_server_host=args.vllm_server_host.replace(\"ip-\", \"\").replace(\"-\", \".\"), # from ip-X-X-X-X to X.X.X.X\n )\n\n trainer = RLOOTrainer(model=\"Qwen/Qwen2.5-72B\", args=training_args, reward_funcs=reward_num_unique_chars, train_dataset=dataset)\n trainer.train()\n\nif __name__==\"__main__\":\n main()\n```\n\n### Using a custom reward function\n\nThe [`RLOOTrainer`] supports using custom reward functions instead of dense reward models. To ensure compatibility, your reward function must satisfy the following requirements:\n\nReward functions can be either synchronous Python callables or asynchronous `async def` coroutines. When you provide multiple asynchronous reward functions, they are awaited concurrently (run in parallel via `asyncio.gather`) so their latency overlaps.\n\n1. **Input arguments**:\n - The function must accept the following as keyword arguments:\n - `prompts` (contains the prompts),\n - `completions` (contains the generated completions),\n - `completion_ids` (contains the tokenized completions),\n - `trainer_state` ([`~transformers.TrainerState`]): The current state of the trainer. This can be used to implement dynamic reward functions, such as curriculum learning, where the reward is adjusted based on the training progress.\n - `log_extra`: a callable `log_extra(column: str, values: list)` to add extra columns to the completions table. See Example 6. In distributed training, it's important that all processes log the same set of keys.\n - `log_metric`: a callable `log_metric(name: str, value: float)` to log scalar metrics as plots alongside `kl`, `entropy`, etc. See Example 6. In distributed training, it's important that all processes log the same set of keys.\n - All column names (but `prompt`) that the dataset may have. For example, if the dataset contains a column named `ground_truth`, the function will be called with `ground_truth` as a keyword argument.\n\n The easiest way to comply with this requirement is to use `**kwargs` in the function signature.\n - Depending on the dataset format, the input will vary:\n - For [standard format](dataset_formats#standard), `prompts` and `completions` will be lists of strings.\n - For [conversational format](dataset_formats#conversational), `prompts` and `completions` will be lists of message dictionaries.\n\n2. **Return value**: The function must return a list of floats. Each float represents the reward corresponding to a single completion.\n\n#### Example 1: Reward longer completions\n\nBelow is an example of a reward function for a standard format that rewards longer completions:\n\n```python\ndef reward_func(completion_ids, **kwargs):\n \"\"\"Reward function that assigns higher scores to longer completions (in terms of token count).\"\"\"\n return [float(len(ids)) for ids in completion_ids]\n```\n\nYou can test it as follows:\n\n```python\n>>> prompts = [\"The sky is\", \"The sun is\"] # not used in the reward function, but the trainer will pass it\n>>> completions = [\" blue.\", \" in the sky.\"] # not used in the reward function, but the trainer will pass it\n>>> completion_ids = [[6303, 13], [304, 279, 12884, 13]]\n>>> reward_func(prompts=prompts, completions=completions, completion_ids=completion_ids)\n[2.0, 4.0]\n```\n\n#### Example 1.1: Reward longer completions (based on the number of characters)\n\nSame as the previous example, but this time the reward function is based on the number of characters instead of tokens.\n\n```python\ndef reward_func(completions, **kwargs):\n \"\"\"Reward function that assigns higher scores to longer completions (in terms of character count).\"\"\"\n return [float(len(completion)) for completion in completions]\n```\n\nYou can test it as follows:\n\n```python\n>>> prompts = [\"The sky is\", \"The sun is\"]\n>>> completions = [\" blue.\", \" in the sky.\"]\n>>> completion_ids = [[6303, 13], [304, 279, 12884, 13]] # not used in the reward function, but the trainer will pass it\n>>> reward_func(prompts=prompts, completions=completions, completion_ids=completion_ids)\n[6.0, 12.0]\n```\n\n#### Example 2: Reward completions with a specific format\n\nBelow is an example of a reward function that checks if the completion has a specific format. This example is inspired by the _format reward_ function used in the paper [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://huggingface.co/papers/2501.12948).\nIt is designed for a conversational format, where prompts and completions consist of structured messages.\n\n```python\nimport re\n\ndef format_reward_func(completions, **kwargs):\n \"\"\"Reward function that checks if the completion has a specific format.\"\"\"\n pattern = r\"^.*?.*?$\"\n completion_contents = [completion[0][\"content\"] for completion in completions]\n matches = [re.match(pattern, content) for content in completion_contents]\n return [1.0 if match else 0.0 for match in matches]\n```\n\nYou can test this function as follows:\n\n```python\n>>> prompts = [\n... [{\"role\": \"assistant\", \"content\": \"What is the result of (1 + 2) * 4?\"}],\n... [{\"role\": \"assistant\", \"content\": \"What is the result of (3 + 1) * 2?\"}],\n... ]\n>>> completions = [\n... [{\"role\": \"assistant\", \"content\": \"The sum of 1 and 2 is 3, which we multiply by 4 to get 12.(1 + 2) * 4 = 12\"}],\n... [{\"role\": \"assistant\", \"content\": \"The sum of 3 and 1 is 4, which we multiply by 2 to get 8. So (3 + 1) * 2 = 8.\"}],\n... ]\n>>> format_reward_func(prompts=prompts, completions=completions)\n[1.0, 0.0]\n```\n\n#### Example 3: Reward completions based on a reference\n\nBelow is an example of a reward function that checks if the completion is correct. This example is inspired by the _accuracy reward_ function used in the paper [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://huggingface.co/papers/2501.12948).\nThis example is designed for [standard format](dataset_formats#standard), where the dataset contains a column named `ground_truth`.\n\n```python\nimport re\n\ndef reward_func(completions, ground_truth, **kwargs):\n # Regular expression to capture content inside \\boxed{}\n matches = [re.search(r\"\\\\boxed\\{(.*?)\\}\", completion) for completion in completions]\n contents = [match.group(1) if match else \"\" for match in matches]\n # Reward 1 if the content is the same as the ground truth, 0 otherwise\n return [1.0 if c == gt else 0.0 for c, gt in zip(contents, ground_truth)]\n```\n\nYou can test this function as follows:\n\n```python\n>>> prompts = [\"Problem: Solve the equation $2x + 3 = 7$. Solution:\", \"Problem: Solve the equation $3x - 5 = 10$.\"]\n>>> completions = [r\" The solution is \\boxed{2}.\", r\" The solution is \\boxed{6}.\"]\n>>> ground_truth = [\"2\", \"5\"]\n>>> reward_func(prompts=prompts, completions=completions, ground_truth=ground_truth)\n[1.0, 0.0]\n```\n\n#### Example 4: Multi-task reward functions\n\nBelow is an example of using multiple reward functions in the [`RLOOTrainer`]. In this example, we define two task-specific reward functions: `math_reward_func` and `coding_reward_func`. The `math_reward_func` rewards math problems based on their correctness, while the `coding_reward_func` rewards coding problems based on whether the solution works.\n\n```python\nfrom datasets import Dataset\nfrom trl import RLOOTrainer\n\n# Define a dataset that contains both math and coding problems\ndataset = Dataset.from_list(\n [\n {\"prompt\": \"What is 2+2?\", \"task\": \"math\"},\n {\"prompt\": \"Write a function that returns the sum of two numbers.\", \"task\": \"code\"},\n {\"prompt\": \"What is 3*4?\", \"task\": \"math\"},\n {\"prompt\": \"Write a function that returns the product of two numbers.\", \"task\": \"code\"},\n ]\n)\n\n# Math-specific reward function\ndef math_reward_func(prompts, completions, task, **kwargs):\n rewards = []\n for prompt, completion, t in zip(prompts, completions, task):\n if t == \"math\":\n # Calculate math-specific reward\n correct = check_math_solution(prompt, completion)\n reward = 1.0 if correct else -1.0\n rewards.append(reward)\n else:\n # Return None for non-math tasks\n rewards.append(None)\n return rewards\n\n# Coding-specific reward function\ndef coding_reward_func(prompts, completions, task, **kwargs):\n rewards = []\n for prompt, completion, t in zip(prompts, completions, task):\n if t == \"coding\":\n # Calculate coding-specific reward\n works = test_code_solution(prompt, completion)\n reward = 1.0 if works else -1.0\n rewards.append(reward)\n else:\n # Return None for non-coding tasks\n rewards.append(None)\n return rewards\n\n# Use both task-specific reward functions\ntrainer = RLOOTrainer(\n model=\"Qwen/Qwen2-0.5B-Instruct\",\n reward_funcs=[math_reward_func, coding_reward_func],\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\nIn this example, the `math_reward_func` and `coding_reward_func` are designed to work with a mixed dataset that contains both math and coding problems. The `task` column in the dataset is used to determine which reward function to apply to each problem. If there is no relevant reward function for a sample in the dataset, the reward function will return `None`, and the [`RLOOTrainer`] will continue with the valid functions and tasks. This allows the [`RLOOTrainer`] to handle multiple reward functions with different applicability.\n\nNote that the [`RLOOTrainer`] will ignore the `None` rewards returned by the reward functions and only consider the rewards returned by the relevant functions. This ensures that the model is trained on the relevant tasks and ignores the tasks for which there is no relevant reward function.\n\n#### Example 5: Asynchronous reward functions\n\nCustom reward functions can also be defined as `async def` coroutines. This is useful if your reward depends on slow I/O (for example, calling a remote service). When you pass multiple async reward functions, [`RLOOTrainer`] executes them concurrently so their latency overlaps.\n\nBelow is a minimal example of an async reward function that simulates an I/O-bound operation:\n\n```python\nimport asyncio\n\nasync def async_reward_func(prompts, completions, **kwargs):\n # Simulate an I/O-bound call (e.g., HTTP request, database lookup)\n await asyncio.sleep(0.01)\n # Simple toy reward: 1.0 if the completion is non-empty, else 0.0\n return [1.0 if completion else 0.0 for completion in completions]\n```\n\n#### Example 6: Logging extra columns and metrics\n\nBelow is an example of a reward function that logs extra columns to the completions table and scalar metrics as plots.\n\n```python\nimport re\n\ndef reward_func(completions, ground_truth, log_extra=None, log_metric=None, **kwargs):\n extracted = [re.search(r\"\\\\boxed\\{(.*?)\\}\", c) for c in completions]\n extracted = [m.group(1) if m else None for m in extracted]\n rewards = [1.0 if e == gt else 0.0 for e, gt in zip(extracted, ground_truth)]\n\n if log_extra:\n log_extra(\"golden_answer\", list(ground_truth))\n log_extra(\"extracted_answer\", [e or \"[none]\" for e in extracted])\n\n if log_metric:\n log_metric(\"accuracy\", sum(rewards) / len(rewards))\n\n return rewards\n```\n\n#### Passing the reward function to the trainer\n\nTo use your custom reward function, pass it to the [`RLOOTrainer`] as follows:\n\n```python\nfrom trl import RLOOTrainer\n\ntrainer = RLOOTrainer(\n reward_funcs=reward_func,\n ...,\n)\n```\n\nYou can pass several reward functions as a list; this list may include both synchronous and asynchronous functions:\n\n```python\nfrom trl import RLOOTrainer\n\ntrainer = RLOOTrainer(\n reward_funcs=[reward_func, async_reward_func1, async_reward_func2],\n ...,\n)\n```\n\nand the reward will be computed as the sum of the rewards from each function, or the weighted sum if `reward_weights` is provided in the config.\n\nNote that [`RLOOTrainer`] supports multiple reward functions of different types. See the parameters documentation for more details.\n\n## Vision-Language Model (VLM) Training\n\nRLOO supports training Vision-Language Models (VLMs) on multimodal datasets containing both text and images.\n\n### Supported Models\n\nTested with:\n\n- **Gemma3** โ€” e.g., `google/gemma-3-4b-it`\n- **LLaVA-NeXT** โ€” e.g., `llava-hf/llava-v1.6-mistral-7b-hf`\n- **Qwen2-VL** โ€” e.g., `Qwen/Qwen2-VL-2B-Instruct`\n- **Qwen2.5-VL** โ€” e.g., `Qwen/Qwen2.5-VL-3B-Instruct`\n- **SmolVLM2** โ€” e.g., `HuggingFaceTB/SmolVLM2-2.2B-Instruct`\n \n> [!TIP]\n> Compatibility with all VLMs is not guaranteed. If you believe a model should be supported, feel free to open an issue on GitHub โ€” or better yet, submit a pull request with the required changes.\n\n### Quick Start\n\nUse [rloo\\_vlm.py](https://github.com/huggingface/trl/blob/main/examples/scripts/rloo_vlm.py) to fine-tune a VLM. Example command for training on [`lmms-lab/multimodal-open-r1-8k-verified`](https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified):\n\n```bash\naccelerate launch \\\n --config_file=examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/rloo_vlm.py \\\n --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --output_dir rloo-Qwen2.5-VL-3B-Instruct \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_completion_length 1024 \\\n --use_vllm \\\n --vllm_mode colocate \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --log_completions\n```\n\n### Configuration Tips\n\n- Use LoRA on vision-language projection layers\n- Enable 4-bit quantization to reduce memory usage\n- VLMs are memory-intensive โ€” start with smaller batch sizes\n- Most models are compatible with vLLM (`server` and `colocate` modes)\n\n### Dataset Format\n\nEach training sample should include:\n\n- `prompt`: Text formatted via the processor's chat template\n- `image`/`images`: PIL Image or list of PIL Images\n\nThe trainer automatically handles image-to-tensor conversion via the modelโ€™s image processor.\n\n## RLOOTrainer\n\n[[autodoc]] RLOOTrainer\n - train\n - save_model\n - push_to_hub\n\n## RLOOConfig\n\n[[autodoc]] RLOOConfig\n\n## References\n\n1. [RLOO Paper](https://openreview.net/pdf?id=r1lgTGL5DE)\n2. [Paper Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs](https://huggingface.co/papers/2402.14740)\n3. [Paper - REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models](https://huggingface.co/papers/2501.03262)\n4. [Blog Post - Putting RL back in RLHF](https://huggingface.co/blog/putting_rl_back_in_rlhf_with_rloo)\n5. [Blog Post - Unraveling RLHF and Its Variants: Progress and Practical Engineering Insights](https://hijkzzz.notion.site/unraveling-rlhf-and-its-variants-engineering-insights#147d9a33ecc9806090f3d5c749d31f05)\n6. [Youtube - RLOO: A Cost-Efficient Optimization for Learning from Human Feedback in LLMs](https://www.youtube.com/watch?v=86asXGPK6RU&ab_channel=BuzzRobot)\n\n## Migration Guide from the old implementation (0.21 and below)\n\nWith the release of version 0.22.0, we have revamped the [`RLOOTrainer`] to be more aligned with other online trainers in the library, like [`GRPOTrainer`]. This new implementation introduces several changes to the configuration parameters and overall structure of the trainer.\nBelow is a summary of the key changes for [`RLOOConfig`]:\n\n| TRL โ‰ค 0.21.x | TRL โ‰ฅ 0.22.0 |\n| --- | --- |\n| `rloo_k` | renamed to `num_generations` |\n| `cliprange` | renamed to `epsilon` |\n| `kl_coef` | renamed to `beta` |\n| `exp_name` | renamed to `run_name`. Use `run_name = f\"{exp_name}__{seed}__{int(time.time())}\"` to replicate old behavior |\n| `normalize_reward` | renamed to `normalize_advantages`. Note: this always normalized advantages (despite the old name) |\n| `num_ppo_epochs` | renamed to `num_iterations` (default: `1`) |\n| `token_level_kl` | **removed** โ€“ KL is now computed only at the sequence level |\n| `dataset_num_proc` | **removed** โ€“ it was unused |\n| `num_mini_batches` | renamed to `steps_per_generation` |\n| `total_episodes` | use `max_steps=total_episodes / gradient_accumulation_steps` instead |\n| `local_rollout_forward_batch_size` | **removed** โ€“ now automatically set to `per_device_train_batch_size` (or `per_device_eval_batch_size` during evaluation) |\n| `num_sample_generations` | **removed** โ€“ use `logging_steps` to control generation logging frequency |\n| `response_length` | renamed to `max_completion_length` (default: `256`) |\n| `stop_token` | **removed** |\n| `stop_token_id` | **removed** โ€“ use `processing_class.eos_token_id` instead |\n| `missing_eos_penalty` | **removed** โ€“ replicate with a custom reward function checking if `eos_token_id` is in `completion_ids` |\n\nBelow is a summary of the key changes for [`RLOOTrainer`]:\n\n| TRL โ‰ค 0.21.x | TRL โ‰ฅ 0.22.0 |\n| --- | --- |\n| `config` | renamed to `args` |\n| `reward_model` | renamed to `reward_funcs`, which now supports both reward models and custom reward functions |\n| `policy` | renamed to `model` |\n| `ref_policy` | **removed** โ€“ the reference model is now created automatically from `model` |\n| `data_collator` | **removed** |\n" }, { "path": "docs/source/script_utils.md", "content": "# Scripts Utilities\n\n## ScriptArguments\n\n[[autodoc]] ScriptArguments\n\n## TrlParser\n\n[[autodoc]] TrlParser\n - parse_args_and_config\n - parse_args_into_dataclasses\n - set_defaults_with_config\n\n## get_dataset\n\n[[autodoc]] get_dataset\n\n## DatasetConfig\n\n[[autodoc]] scripts.utils.DatasetConfig\n\n## DatasetMixtureConfig\n\n[[autodoc]] DatasetMixtureConfig\n" }, { "path": "docs/source/sft_trainer.md", "content": "# SFT Trainer\n\n[![All_models-SFT-blue](https://img.shields.io/badge/All_models-SFT-blue)](https://huggingface.co/models?other=sft,trl) [![smol_course-Chapter_1-yellow](https://img.shields.io/badge/smol_course-Chapter_1-yellow)](https://github.com/huggingface/smol-course/tree/main/1_instruction_tuning)\n\n## Overview\n\nTRL supports the Supervised Fine-Tuning (SFT) Trainer for training language models.\n\nThis post-training method was contributed by [Younes Belkada](https://huggingface.co/ybelkada).\n\n## Quick start\n\nThis example demonstrates how to train a language model using the [`SFTTrainer`] from TRL. We train a [Qwen 3 0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) model on the [Capybara dataset](https://huggingface.co/datasets/trl-lib/Capybara), a compact, diverse multi-turn dataset to benchmark reasoning and generalization.\n\n```python\nfrom trl import SFTTrainer\nfrom datasets import load_dataset\n\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n train_dataset=load_dataset(\"trl-lib/Capybara\", split=\"train\"),\n)\ntrainer.train()\n```\n\n\n\n## Expected dataset type and format\n\nSFT supports both [language modeling](dataset_formats#language-modeling) and [prompt-completion](dataset_formats#prompt-completion) datasets. The [`SFTTrainer`] is compatible with both [standard](dataset_formats#standard) and [conversational](dataset_formats#conversational) dataset formats. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\n```python\n# Standard language modeling\n{\"text\": \"The sky is blue.\"}\n\n# Conversational language modeling\n{\"messages\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"}]}\n\n# Standard prompt-completion\n{\"prompt\": \"The sky is\",\n \"completion\": \" blue.\"}\n\n# Conversational prompt-completion\n{\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}]}\n```\n\nIf your dataset is not in one of these formats, you can preprocess it to convert it into the expected format. Here is an example with the [FreedomIntelligence/medical-o1-reasoning-SFT](https://huggingface.co/datasets/FreedomIntelligence/medical-o1-reasoning-SFT) dataset:\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"FreedomIntelligence/medical-o1-reasoning-SFT\", \"en\")\n\ndef preprocess_function(example):\n return {\n \"prompt\": [{\"role\": \"user\", \"content\": example[\"Question\"]}],\n \"completion\": [\n {\"role\": \"assistant\", \"content\": f\"{example['Complex_CoT']}{example['Response']}\"}\n ],\n }\n\ndataset = dataset.map(preprocess_function, remove_columns=[\"Question\", \"Response\", \"Complex_CoT\"])\nprint(next(iter(dataset[\"train\"])))\n```\n\n```json\n{\n \"prompt\": [\n {\n \"content\": \"Given the symptoms of sudden weakness in the left arm and leg, recent long-distance travel, and the presence of swollen and tender right lower leg, what specific cardiac abnormality is most likely to be found upon further evaluation that could explain these findings?\",\n \"role\": \"user\",\n }\n ],\n \"completion\": [\n {\n \"content\": \"Okay, let's see what's going on here. We've got sudden weakness [...] clicks into place!The specific cardiac abnormality most likely to be found in [...] the presence of a PFO facilitating a paradoxical embolism.\",\n \"role\": \"assistant\",\n }\n ],\n}\n```\n\n## Looking deeper into the SFT method\n\nSupervised Fine-Tuning (SFT) is the simplest and most commonly used method to adapt a language model to a target dataset. The model is trained in a fully supervised fashion using pairs of input and output sequences. The goal is to minimize the negative log-likelihood (NLL) of the target sequence, conditioning on the input.\n\nThis section breaks down how SFT works in practice, covering the key steps: **preprocessing**, **tokenization** and **loss computation**.\n\n### Preprocessing and tokenization\n\nDuring training, each example is expected to contain a **text field** or a **(prompt, completion)** pair, depending on the dataset format. For more details on the expected formats, see [Dataset formats](dataset_formats).\nThe [`SFTTrainer`] tokenizes each input using the model's tokenizer. If both prompt and completion are provided separately, they are concatenated before tokenization.\n\n### Computing the loss\n\n![sft_figure](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/sft_figure.png)\n\nThe loss used in SFT is the **token-level cross-entropy loss**, defined as:\n\n$$\n\\mathcal{L}_{\\text{SFT}}(\\theta) = - \\sum_{t=1}^{T} \\log p_\\theta(y_t \\mid y_{ [!TIP]\n> The paper [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification](https://huggingface.co/papers/2508.05629) proposes an alternative loss function, called **Dynamic Fine-Tuning (DFT)**, which aims to improve generalization by rectifying the reward signal. This method can be enabled by setting `loss_type=\"dft\"` in the [`SFTConfig`]. For more details, see [Paper Index - Dynamic Fine-Tuning](paper_index#on-the-generalization-of-sft-a-reinforcement-learning-perspective-with-reward-rectification).\n\n### Label shifting and masking\n\nDuring training, the loss is computed using a **one-token shift**: the model is trained to predict each token in the sequence based on all previous tokens. Specifically, the input sequence is shifted right by one position to form the target labels.\nPadding tokens (if present) are ignored in the loss computation by applying an ignore index (default: `-100`) to the corresponding positions. This ensures that the loss focuses only on meaningful, non-padding tokens.\n\n## Logged metrics\n\nWhile training and evaluating we record the following reward metrics:\n\n* `global_step`: The total number of optimizer steps taken so far.\n* `epoch`: The current epoch number, based on dataset iteration.\n* `num_tokens`: The total number of tokens processed so far.\n* `loss`: The average cross-entropy loss computed over non-masked tokens in the current logging interval.\n* `entropy`: The average entropy of the model's predicted token distribution over non-masked tokens.\n* `mean_token_accuracy`: The proportion of non-masked tokens for which the modelโ€™s top-1 prediction matches the ground truth token.\n* `learning_rate`: The current learning rate, which may change dynamically if a scheduler is used.\n* `grad_norm`: The L2 norm of the gradients, computed before gradient clipping.\n\n## Customization\n\n### Model initialization\n\nYou can directly pass the kwargs of the [`~transformers.AutoModelForCausalLM.from_pretrained()`] method to the [`SFTConfig`]. For example, if you want to load a model in a different precision, analogous to\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen3-0.6B\", dtype=torch.bfloat16)\n```\n\nyou can do so by passing the `model_init_kwargs={\"dtype\": torch.bfloat16}` argument to the [`SFTConfig`].\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(\n model_init_kwargs={\"dtype\": torch.bfloat16},\n)\n```\n\nNote that all keyword arguments of [`~transformers.AutoModelForCausalLM.from_pretrained()`] are supported.\n\n### Packing\n\n[`SFTTrainer`] supports _example packing_, where multiple examples are packed in the same input sequence to increase training efficiency. To enable packing, simply pass `packing=True` to the [`SFTConfig`] constructor.\n\n```python\ntraining_args = SFTConfig(packing=True)\n```\n\nFor more details on packing, see [Packing](reducing_memory_usage#packing).\n\n### Train on assistant messages only\n\nTo train on assistant messages only, use a [conversational](dataset_formats#conversational) dataset and set `assistant_only_loss=True` in the [`SFTConfig`]. This setting ensures that loss is computed **only** on the assistant responses, ignoring user or system messages.\n\n```python\ntraining_args = SFTConfig(assistant_only_loss=True)\n```\n\n![train_on_assistant](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/train_on_assistant.png)\n\n> [!WARNING]\n> This functionality is only available for chat templates that support returning the assistant tokens mask via the `{% generation %}` and `{% endgeneration %}` keywords. For an example of such a template, see [HugggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B/blob/main/chat_template.jinja#L76-L82).\n\n### Train on completion only\n\nTo train on completion only, use a [prompt-completion](dataset_formats#prompt-completion) dataset. By default, the trainer computes the loss on the completion tokens only, ignoring the prompt tokens. If you want to train on the full sequence, set `completion_only_loss=False` in the [`SFTConfig`].\n\n![train_on_completion](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/train_on_completion.png)\n\n> [!TIP]\n> Training on completion only is compatible with training on assistant messages only. In this case, use a [conversational](dataset_formats#conversational) [prompt-completion](dataset_formats#prompt-completion) dataset and set `assistant_only_loss=True` in the [`SFTConfig`].\n\n### Train adapters with PEFT\n\nWe support tight integration with ๐Ÿค— PEFT library, allowing any user to conveniently train adapters and share them on the Hub, rather than training the entire model.\n\n```python\nfrom datasets import load_dataset\nfrom trl import SFTTrainer\nfrom peft import LoraConfig\n\ndataset = load_dataset(\"trl-lib/Capybara\", split=\"train\")\n\ntrainer = SFTTrainer(\n \"Qwen/Qwen3-0.6B\",\n train_dataset=dataset,\n peft_config=LoraConfig(),\n)\n\ntrainer.train()\n```\n\nYou can also continue training your [`~peft.PeftModel`]. For that, first load a `PeftModel` outside [`SFTTrainer`] and pass it directly to the trainer without the `peft_config` argument being passed.\n\n```python\nfrom datasets import load_dataset\nfrom trl import SFTTrainer\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\"trl-lib/Qwen3-4B-LoRA\", is_trainable=True)\ndataset = load_dataset(\"trl-lib/Capybara\", split=\"train\")\n\ntrainer = SFTTrainer(\n model=model,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\n> [!TIP]\n> When training adapters, you typically use a higher learning rate (โ‰ˆ1eโ€‘4) since only new parameters are being learned.\n>\n> ```python\n> SFTConfig(learning_rate=1e-4, ...)\n> ```\n\n### Train with Liger Kernel\n\nLiger Kernel is a collection of Triton kernels for LLM training that boosts multi-GPU throughput by 20%, cuts memory use by 60% (enabling up to 4ร— longer context), and works seamlessly with tools like FlashAttention, PyTorch FSDP, and DeepSpeed. For more information, see [Liger Kernel Integration](liger_kernel_integration).\n\n### Rapid Experimentation for SFT\n\nRapidFire AI is an open-source experimentation engine that sits on top of TRL and lets you launch multiple SFT configurations at once, even on a single GPU. Instead of trying configurations sequentially, RapidFire lets you **see all their learning curves earlier, stop underperforming runs, and clone promising ones with new settings in flight** without restarting. For more information, see [RapidFire AI Integration](rapidfire_integration).\n\n### Train with Unsloth\n\nUnsloth is an openโ€‘source framework for fineโ€‘tuning and reinforcement learning that trains LLMs (like Llama, Mistral, Gemma, DeepSeek, and more) up to 2ร— faster with up to 70% less VRAM, while providing a streamlined, Hugging Faceโ€“compatible workflow for training, evaluation, and deployment. For more information, see [Unsloth Integration](unsloth_integration).\n\n## Instruction tuning example\n\n**Instruction tuning** teaches a base language model to follow user instructions and engage in conversations. This requires:\n\n1. **Chat template**: Defines how to structure conversations into text sequences, including role markers (user/assistant), special tokens, and turn boundaries. Read more about chat templates in [Chat templates](https://huggingface.co/docs/transformers/chat_templating#templates).\n2. **Conversational dataset**: Contains instruction-response pairs\n\nThis example shows how to transform the [Qwen 3 0.6B Base](https://huggingface.co/Qwen/Qwen3-0.6B-Base) model into an instruction-following model using the [Capybara dataset](https://huggingface.co/datasets/trl-lib/Capybara) and a chat template from [HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B). The SFT Trainer automatically handles tokenizer updates and special token configuration.\n\n```python\nfrom trl import SFTConfig, SFTTrainer\nfrom datasets import load_dataset\n\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen3-0.6B-Base\",\n args=SFTConfig(\n output_dir=\"Qwen3-0.6B-Instruct\",\n chat_template_path=\"HuggingFaceTB/SmolLM3-3B\",\n ),\n train_dataset=load_dataset(\"trl-lib/Capybara\", split=\"train\"),\n)\ntrainer.train()\n```\n\n> [!WARNING]\n> Some base models, like those from Qwen, have a predefined chat template in the model's tokenizer. In these cases, it is not necessary to apply [`clone_chat_template()`], as the tokenizer already handles the formatting. However, it is necessary to align the EOS token with the chat template to ensure the model's responses terminate correctly. In these cases, specify `eos_token` in [`SFTConfig`]; for example, for `Qwen/Qwen2.5-1.5B`, one should set `eos_token=\"<|im_end|>\"`.\n\nOnce trained, your model can now follow instructions and engage in conversations using its new chat template.\n\n```python\n>>> from transformers import pipeline\n>>> pipe = pipeline(\"text-generation\", model=\"Qwen3-0.6B-Instruct/checkpoint-5000\")\n>>> prompt = \"<|im_start|>user\\nWhat is the capital of France? Answer in one word.<|im_end|>\\n<|im_start|>assistant\\n\"\n>>> response = pipe(prompt)\n>>> response[0][\"generated_text\"]\n'<|im_start|>user\\nWhat is the capital of France? Answer in one word.<|im_end|>\\n<|im_start|>assistant\\nThe capital of France is Paris.'\n```\n\nAlternatively, use the structured conversation format (recommended):\n\n```python\n>>> prompt = [{\"role\": \"user\", \"content\": \"What is the capital of France? Answer in one word.\"}]\n>>> response = pipe(prompt)\n>>> response[0][\"generated_text\"]\n[{'role': 'user', 'content': 'What is the capital of France? Answer in one word.'}, {'role': 'assistant', 'content': 'The capital of France is Paris.'}]\n```\n\n## Tool Calling with SFT\n\nThe [`SFTTrainer`] fully supports fine-tuning models with _tool calling_ capabilities. In this case, each dataset example should include:\n\n* The conversation messages, including any tool calls (`tool_calls`) and tool responses (`tool` role messages)\n* The list of available tools in the `tools` column, typically provided as JSON schemas\n\nFor details on the expected dataset structure, see the [Dataset Format โ€” Tool Calling](dataset_formats#tool-calling) section.\n\n## Training Vision Language Models\n\n[`SFTTrainer`] fully supports training Vision-Language Models (VLMs). To train a VLM, provide a dataset with either an `image` column (single image per sample) or an `images` column (list of images per sample). For more information on the expected dataset structure, see the [Dataset Format โ€” Vision Dataset](dataset_formats#vision-dataset) section.\nAn example of such a dataset is the [LLaVA Instruct Mix](https://huggingface.co/datasets/trl-lib/llava-instruct-mix).\n\n```python\nfrom trl import SFTConfig, SFTTrainer\nfrom datasets import load_dataset\n\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen2.5-VL-3B-Instruct\",\n args=SFTConfig(max_length=None),\n train_dataset=load_dataset(\"trl-lib/llava-instruct-mix\", split=\"train\"),\n)\ntrainer.train()\n```\n\n> [!TIP]\n> For VLMs, truncating may remove image tokens, leading to errors during training. To avoid this, set `max_length=None` in the [`SFTConfig`]. This allows the model to process the full sequence length without truncating image tokens.\n>\n> ```python\n> SFTConfig(max_length=None, ...)\n> ```\n>\n> Only use `max_length` when you've verified that truncation won't remove image tokens for the entire dataset.\n\n## SFTTrainer\n\n[[autodoc]] SFTTrainer\n - train\n - save_model\n - push_to_hub\n\n## SFTConfig\n\n[[autodoc]] SFTConfig\n" }, { "path": "docs/source/speeding_up_training.md", "content": "# Speeding Up Training\n\nThis guide covers various methods to accelerate training in TRL. Each technique includes minimal examples with links to more comprehensive documentation.\n\n## vLLM for fast generation in online methods\n\n[Online methods](index#online-methods) such as GRPO or Online DPO require the model to generate completions, which is often a slow process and can significantly impact training time.\nTo speed up generation, you can use [vLLM](https://github.com/vllm-project/vllm), a library that enables fast generation through, among other things, PagedAttention. TRL's online trainers support vLLM, greatly improving training speed. For more details, see [vLLM Integration](vllm_integration).\n\nTo use [vLLM](https://github.com/vllm-project/vllm), first install it using:\n\n```bash\npip install trl[vllm]\n```\n\n\n\n\nFirst, start a vLLM server by running:\n\n```bash\ntrl vllm-serve --model \n```\n\nThen, run the training script and pass `use_vllm=True` in the training arguments.\n\n```python\nfrom trl.experimental.online_dpo import OnlineDPOConfig\n\ntraining_args = OnlineDPOConfig(..., use_vllm=True, vllm_mode=\"server\")\n```\n\n\n\n\nFirst, start a vLLM server by running:\n\n```bash\ntrl vllm-serve --model \n```\n\nThen, run the training script and pass `use_vllm=True` in the training arguments.\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(..., use_vllm=True, vllm_mode=\"server\")\n```\n\nYou can customize the server configuration by passing additional arguments. For more information, see [vLLM integration](vllm_integration).\n\n> [!WARNING]\n> When using vLLM, ensure that the GPUs assigned for training and generation are separate to avoid resource conflicts. For instance, if you plan to use 4 GPUs for training and another 4 for vLLM generation, you can specify GPU allocation using `CUDA_VISIBLE_DEVICES`. \n>\n> Set GPUs **0-3** for vLLM generation: \n>\n> ```sh\n> CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model \n> ``` \n>\n> And GPUs **4-7** for training:\n>\n> ```sh\n> CUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch train.py\n> ```\n\n\n\n\nFirst, start a vLLM server by running:\n\n```bash\ntrl vllm-serve --model \n```\n\nThen, run the training script and pass `use_vllm=True` in the training arguments.\n\n```python\nfrom trl import RLOOConfig\n\ntraining_args = RLOOConfig(..., use_vllm=True, vllm_mode=\"server\")\n```\n\nYou can customize the server configuration by passing additional arguments. For more information, see [vLLM integration](vllm_integration).\n\n> [!WARNING]\n> When using vLLM, ensure that the GPUs assigned for training and generation are separate to avoid resource conflicts. For instance, if you plan to use 4 GPUs for training and another 4 for vLLM generation, you can specify GPU allocation using `CUDA_VISIBLE_DEVICES`. \n>\n> Set GPUs **0-3** for vLLM generation:\n>\n> ```sh\n> CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model \n> ``` \n>\n> And GPUs **4-7** for training:\n>\n> ```sh\n> CUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch train.py\n> ```\n\n\n\n\n## Optimized attention implementations\n\nTRL supports various optimized attention implementations that can significantly speed up training while reducing memory usage. You can use either a pre-optimized kernels directly from the [Kernels Hub](kernels_hub) or a manually built attention backend.\n\n\n\n\nYou can use pre-optimized attention kernels from the Hub without manual compilation:\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., model_init_kwargs={\"attn_implementation\": \"kernels-community/flash-attn2\"})\n```\n\nOther options include `kernels-community/vllm-flash-attn3` and `kernels-community/paged-attention`.\n\nOptimized attention works across all TRL trainers. For more details, see [Kernels Hub Integration](kernels_hub).\n\n\n\n\n> [!WARNING]\n> Manually building optimized attention backends is complex and time-consuming. It's never recommended unless absolutely necessary. Consider using Kernels from the Hub instead, as described in the previous section.\n\nIf you have manually installed an optimized attention backend like Flash Attention 2, you can specify it in the training arguments:\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., model_init_kwargs={\"attn_implementation\": \"flash_attention_2\"})\n```\n\n\n\n\n## Liger Kernel for memory optimization\n\nLiger Kernel is a collection of Triton kernels designed for LLM training that can increase throughput by 20% and reduce memory usage by 60%.\n\n\n\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl import DPOConfig\n\ntraining_args = DPOConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl.experimental.kto import KTOConfig\n\ntraining_args = KTOConfig(..., use_liger_kernel=True)\n```\n\n\n\n\n```python\nfrom trl.experimental.gkd import GKDConfig\n\ntraining_args = GKDConfig(..., use_liger_kernel=True)\n```\n\n\n\n\nFor more information, see [Liger Kernel Integration](liger_kernel_integration).\n\n## Mixed precision training\n\nMixed precision training using bf16 or fp16 can speed up training and reduce memory usage with minimal impact on model quality.\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(..., bf16=True) # or fp16=True for older GPUs\n```\n\nUse `bf16=True` for Ampere GPUs (A100, RTX 30xx) or newer, and `fp16=True` for older GPUs. Mixed precision training is supported across all TRL trainers.\n" }, { "path": "docs/source/trackio_integration.md", "content": "# Trackio Integration\n\n[Trackio](https://huggingface.co/docs/trackio) is a lightweight, free experiment tracking library built on top of **๐Ÿค— Datasets** and **๐Ÿค— Spaces**. It is the **recommended tracking solution for TRL** and comes natively integrated with all trainers.\n\nTo enable logging, simply set `report_to=\"trackio\"` in your training config:\n\n```python\nfrom trl import SFTConfig # works with any trainer config (e.g. DPOConfig, GRPOConfig, etc.)\n\ntraining_args = SFTConfig(\n ...,\n report_to=\"trackio\", # enable Trackio logging\n)\n```\n\n## Organizing Your Experiments with Run Names and Projects\n\nBy default, Trackio will generate a name to identify each run. However, we highly recommend setting a descriptive `run_name` to make it easier to organize experiments. For example:\n\n```python\nfrom trl import SFTConfig\n\ntraining_args = SFTConfig(\n ...,\n report_to=\"trackio\",\n run_name=\"sft_qwen3-4b_lr2e-5_bs128\", # descriptive run name\n)\n```\n\nYou can also group related experiments by project by setting the following environment variable:\n\n```bash\nexport TRACKIO_PROJECT=\"my_project\"\n```\n\n## Hosting Your Logs on ๐Ÿค— Spaces\n\nTrackio has local-first design, meaning your logs stay on your machine. If youโ€™d like to host them and deploy a dashboard on **๐Ÿค— Spaces**, set:\n\n```bash\nexport TRACKIO_SPACE_ID=\"username/space_id\"\n```\n\nRunning the following example:\n\n```python\nimport os\nfrom trl import SFTConfig, SFTTrainer\nfrom datasets import load_dataset\n\nos.environ[\"TRACKIO_SPACE_ID\"] = \"trl-lib/trackio\"\nos.environ[\"TRACKIO_PROJECT\"] = \"trl-documentation\"\n\ntrainer = SFTTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n train_dataset=load_dataset(\"trl-lib/Capybara\", split=\"train\"),\n args=SFTConfig(\n report_to=\"trackio\",\n run_name=\"sft_qwen3-0.6b_capybara\",\n ),\n)\ntrainer.train()\n```\n\nwill give you a hosted dashboard at https://huggingface.co/spaces/trl-lib/trackio.\n\n\n" }, { "path": "docs/source/unsloth_integration.md", "content": "# Unsloth Integration\n\nUnsloth is an openโ€‘source framework for fineโ€‘tuning and reinforcement learning that trains LLMs (like Llama, OpenAI gpt-oss, Mistral, Gemma, DeepSeek, and more) up to 2ร— faster with up to 80% less VRAM. Unsloth allows [training](https://huggingface.co/docs/trl/en/unsloth_integration#Training), evaluation, running and [deployment](https://huggingface.co/docs/trl/en/unsloth_integration#Saving-the-model) with other inference engines like llama.cpp, Ollama and vLLM.\n\nThe library provides a streamlined, Hugging Face compatible workflow for training, evaluation, inference and deployment and is fully compatible with [`SFTTrainer`].\n\n## Key Features\n\n- Training support for all transformer compatible models: Text-to-speech (TTS), multimodal, BERT, RL and more\n- Supports full fine-tuning, pretraining, LoRA, QLoRA, 8-bit training & more\n- Works on Linux, Windows, Colab, Kaggle; NVIDIA GPUs, soon AMD & Intel setups\n- Supports most features TRL supports, including RLHF (GSPO, GRPO, DPO etc.)\n- Hand-written Triton kernels and a manual backprop engine ensure no accuracy degradation (0% approximation error)\n\n## Installation\n\n### pip install\n\nLocal Installation (Linux recommended):\n\n```sh\npip install unsloth\n```\n\nYou can also install `unsloth` according to the [official documentation](https://docs.unsloth.ai/get-started/installing-+-updating). Once installed, you can incorporate unsloth into your workflow in a very simple manner; instead of loading [`~transformers.AutoModelForCausalLM`], you just need to load a `FastLanguageModel` as follows:\n\n```python\nimport torch\nfrom trl import SFTConfig, SFTTrainer\nfrom unsloth import FastLanguageModel\n\nmax_length = 2048 # Supports automatic RoPE Scaling, so choose any number\n\n# Load model\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n model_name=\"unsloth/mistral-7b\",\n max_seq_length=max_length,\n dtype=\"auto\", # For auto-detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+\n load_in_4bit=True, # Use 4bit quantization to reduce memory usage. Can be False\n)\n\n# Do model patching and add fast LoRA weights\nmodel = FastLanguageModel.get_peft_model(\n model,\n r=16,\n target_modules=[\n \"q_proj\",\n \"k_proj\",\n \"v_proj\",\n \"o_proj\",\n \"gate_proj\",\n \"up_proj\",\n \"down_proj\",\n ],\n lora_alpha=16,\n lora_dropout=0, # Dropout = 0 is currently optimized\n bias=\"none\", # Bias = \"none\" is currently optimized\n use_gradient_checkpointing=True,\n random_state=3407,\n)\n\ntraining_args = SFTConfig(output_dir=\"./output\", max_length=max_length)\n\ntrainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\nThe saved model is fully compatible with Hugging Face's transformers library. Learn more about unsloth in their [official repository](https://github.com/unslothai/unsloth).\n\n### Docker Install\n\n```sh\ndocker run -d -e JUPYTER_PASSWORD=\"mypassword\" \\\n -p 8888:8888 -p 2222:22 \\\n -v $(pwd)/work:/workspace/work \\\n --gpus all \\\n unsloth/unsloth\n```\n\nAccess Jupyter Lab at ```http://localhost:8888``` and start fine-tuning!\n\n## Training\n\nThese are some core settings you can toggle before training:\n\n- ```max_seq_length = 2048``` โ€“ Controls context length. While Llama-3 supports 8192, we recommend 2048 for testing. Unsloth enables 4ร— longer context fine-tuning.\n- ```dtype = \"auto\"``` โ€“ For auto-detection; use torch.float16 or torch.bfloat16 for newer GPUs.\n- ```load_in_4bit = True``` โ€“ Enables 4-bit quantization, reducing memory use 4ร— for fine-tuning. Disabling it allows for LoRA 16-bit fine-tuning to be enabled.\n- To enable full fine-tuning (FFT), set ```full_finetuning = True```. For 8-bit fine-tuning, set ```load_in_8bit = True```. Note: Only one training method can be set to True at a time.\n\nFor more information on configuring Unsloth's hyperparameters and features, read their [documentation guide here](https://docs.unsloth.ai/get-started/fine-tuning-llms-guide).\n\n## Saving the model\n\nUnsloth allows you to directly save the finetuned model as a small file called a LoRA adapter. You can instead push to the Hugging Face hub as well if you want to upload your model! Remember to get a [Hugging Face token](https://huggingface.co/settings/tokens) and add your token!\n\n### Saving to GGUF\n\nTo save to GGUF, Unsloth uses llama.cpp. To save locally:\n\n```python\nmodel.save_pretrained_gguf(\"directory\", tokenizer, quantization_method = \"q4_k_m\")\nmodel.save_pretrained_gguf(\"directory\", tokenizer, quantization_method = \"q8_0\")\nmodel.save_pretrained_gguf(\"directory\", tokenizer, quantization_method = \"f16\")\n```\n\nTo push to the hub:\n\n```python\nmodel.push_to_hub_gguf(\"hf_username/directory\", tokenizer, quantization_method = \"q4_k_m\")\nmodel.push_to_hub_gguf(\"hf_username/directory\", tokenizer, quantization_method = \"q8_0\")\n```\n\n### Saving to vLLM\n\nTo save to 16-bit for vLLM, use:\n\n```python\nmodel.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_16bit\",)\nmodel.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_16bit\", token = \"\")\n```\n" }, { "path": "docs/source/use_model.md", "content": "# Use model after training\n\nOnce you have trained a model using either the SFTTrainer, PPOTrainer, or DPOTrainer, you will have a fine-tuned model that can be used for text generation. In this section, we'll walk through the process of loading the fine-tuned model and generating text. If you need to run an inference server with the trained model, you can explore libraries such as [`text-generation-inference`](https://github.com/huggingface/text-generation-inference).\n\n## Load and Generate\n\nIf you have fine-tuned a model fully, meaning without the use of PEFT you can simply load it like any other language model in transformers. E.g. the value head that was trained during the PPO training is no longer needed and if you load the model with the original transformer class it will be ignored:\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_name_or_path = \"Qwen/Qwen3-0.6B\" #path/to/your/model/or/name/on/hub\ndevice = \"cpu\" # or \"cuda\" if you have a GPU\n\nmodel = AutoModelForCausalLM.from_pretrained(model_name_or_path).to(device)\ntokenizer = AutoTokenizer.from_pretrained(model_name_or_path)\n\ninputs = tokenizer.encode(\"This movie was really\", return_tensors=\"pt\").to(device)\noutputs = model.generate(inputs)\nprint(tokenizer.decode(outputs[0]))\n```\n\nAlternatively you can also use the pipeline:\n\n```python\nfrom transformers import pipeline\n\nmodel_name_or_path = \"Qwen/Qwen3-0.6B\" #path/to/your/model/or/name/on/hub\npipe = pipeline(\"text-generation\", model=model_name_or_path)\nprint(pipe(\"This movie was really\")[0][\"generated_text\"])\n```\n\n## Use Adapters PEFT\n\n```python\nfrom peft import PeftConfig, PeftModel\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nbase_model_name = \"Qwen/Qwen3-0.6B\" #path/to/your/model/or/name/on/hub\nadapter_model_name = \"path/to/my/adapter\"\n\nmodel = AutoModelForCausalLM.from_pretrained(base_model_name)\nmodel = PeftModel.from_pretrained(model, adapter_model_name)\n\ntokenizer = AutoTokenizer.from_pretrained(base_model_name)\n```\n\nYou can also merge the adapters into the base model so you can use the model like a normal transformers model, however the checkpoint will be significantly bigger:\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(base_model_name)\nmodel = PeftModel.from_pretrained(model, adapter_model_name)\n\nmodel = model.merge_and_unload()\nmodel.save_pretrained(\"merged_adapters\")\n```\n\nOnce you have the model loaded and either merged the adapters or keep them separately on top you can run generation as with a normal model outlined above.\n" }, { "path": "docs/source/vllm_integration.md", "content": "# vLLM Integration\n\nThis document will guide you through the process of using vLLM with TRL for faster generation in online methods like GRPO and Online DPO. We first summarize a tl;dr on how to use vLLM with TRL, and then we will go into the details of how it works under the hood.\n\n> [!WARNING]\n> TRL currently only supports vLLM versions from `0.10.2` to `0.17.1`. Please ensure you have a version in this range installed to avoid compatibility issues.\n\n> [!TIP]\n> The following trainers currently support generation with vLLM:\n>\n> - [`GRPOTrainer`]\n> - [`RLOOTrainer`]\n> - [`experimental.nash_md.NashMDTrainer`]\n> - [`experimental.online_dpo.OnlineDPOTrainer`]\n> - [`experimental.xpo.XPOTrainer`]\n\n## ๐Ÿš€ How can I use vLLM with TRL to speed up training?\n\n๐Ÿ’ก **Note**: Resources required for this specific example: a single node with 8 GPUs.\n\n> [!WARNING]\n> When using vLLM with TRL, the **vLLM server** and the **trainer** must run on **separate CUDA devices** to prevent conflicts.\n> For guidance on configuring this properly, see [Modes of using vLLM during training](#modes-of-using-vllm-during-training).\n\nFirst, install vLLM using the following command:\n\n```bash\npip install \"trl[vllm]\"\n```\n\nThen run the server on specific GPUs (e.g., GPUs 0-3):\n\n```sh\nCUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen2.5-7B --tensor-parallel-size 4\n```\n\nOnce the server is running, you can use it to generate completions for training. In the example below, we are using the different supported trainers using the vLLM server for generation. The `--tensor-parallel-size` and `--data-parallel-size` arguments control how the model and data are sharded across GPUs.\n\nIn this example, we shard one model across 4 GPUs with tensor parallelism. Then, run the training script on different GPUs (e.g., GPUs 4-7) by passing `use_vllm=True` in the training arguments as follows:\n\nSample of a simple `train.py` script:\n\n\n\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer, GRPOConfig\nfrom trl.rewards import accuracy_reward\n\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-7B\",\n args=GRPOConfig(use_vllm=True, vllm_mode=\"server\"),\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\n\n\n\n```python\nfrom datasets import load_dataset\nfrom trl.experimental.online_dpo import OnlineDPOConfig, OnlineDPOTrainer\nfrom trl.rewards import accuracy_reward\n\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\ntrainer = OnlineDPOTrainer(\n model=\"Qwen/Qwen2.5-7B\",\n args=OnlineDPOConfig(use_vllm=True, vllm_mode=\"server\"),\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\n\n\n\n```python\nfrom datasets import load_dataset\nfrom trl.experimental.nash_md import NashMDConfig, NashMDTrainer\nfrom trl.rewards import accuracy_reward\n\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\ntrainer = NashMDTrainer(\n model=\"Qwen/Qwen2.5-7B\",\n args=NashMDConfig(use_vllm=True, vllm_mode=\"server\"),\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\n\n\n\n```python\nfrom datasets import load_dataset\nfrom trl.experimental.xpo import XPOTrainer, XPOConfig\nfrom trl.rewards import accuracy_reward\n\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\ntrainer = XPOTrainer(\n model=\"Qwen/Qwen2.5-7B\",\n args=XPOConfig(use_vllm=True, vllm_mode=\"server\"),\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\n\n\n\n```python\nfrom datasets import load_dataset\nfrom trl import RLOOTrainer, RLOOConfig\nfrom trl.rewards import accuracy_reward\n\ndataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\ntrainer = RLOOTrainer(\n model=\"Qwen/Qwen2.5-7B\",\n args=RLOOConfig(use_vllm=True, vllm_mode=\"server\"),\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n)\n\ntrainer.train()\n```\n\n\n\n\nAnd the train command on separate GPUs from the server:\n\n```sh\nCUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch train.py\n```\n\n## Why using vLLM?\n\n### ๐ŸŽฌ Flashback: Why do we need to use vLLM in online methods?\n\nOnline methods like GRPO or Online DPO require the model to generate completions during training, which are then used to compute reward signals. However, generation can be extremely time-consuming, especially with large or reasoning models. In the default setup (without vLLM), completions are generated using the [(unwrapped) model's `generate` method](https://github.com/huggingface/trl/blob/f3e8c2304428ef16e9ae5de9e5741ed84d533b7b/trl/trainer/grpo_trainer.py#L965C39-L965C66). This approach quickly becomes a major bottleneck โ€” generation is slow and inefficient, particularly for large batches or models. As a result, training times increase significantly, and overall efficiency drops. To address this, we turn to vLLM, which enables much faster and more scalable generation, helping eliminate this bottleneck in online methods.\n\n### ๐Ÿค” How does vLLM solve the slow generation issue?\n\nIf you've ever done autoregressive decoder training, you know all the input tokens to the LLM produce their attention key and value tensors, and these tensors are kept in GPU memory to later generate subsequent tokens based on them. These cached key and value tensors are often referred to as the KV cache. However, storing the KV cache occupies a lot of memory, so vLLM uses a technique called **PagedAttention** to solve this problem. PagedAttention, which is inspired by the OSโ€™s virtual memory concept, stores continuous keys and values in **non-contiguous memory space**, which is much more efficient. The details of this are beyond the scope of this document, but in short, it allows the model to store the keys and values in a more efficient way, reducing the memory footprint and speeding up the generation process. If you are interested, make sure to check out the [vLLM PagedAttention](https://blog.vllm.ai/2023/06/20/vllm.html) for more details.\n\n## How vLLM Works (Under the Hood) ๐Ÿ”\n\n### ๐Ÿค” What exactly happens when you run `trl vllm-serve --model `?\n\nWhen you run for example\n\n```sh\nCUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen2.5-7B --tensor-parallel-size 4\n```\n\n1. vLLM first spawns multiple workers to handle incoming requests in parallel. The number of workers is determined by multiplying the `--tensor-parallel-size` and `--data-parallel-size` values. In this example, it spawns 4 workers (4 ร— 1).\nEach worker operates independently and processes a chunk of the incoming requests โ€” which are basically the prompts sent to the server for generation.\n\n2. Once the incoming requests (prompts) are distributed across the workers, the model starts generating completions. Internally, the modelโ€™s weights are split across multiple GPUs based on the `--tensor-parallel-size` argument โ€” this is how tensor parallelism is handled.\n\n3. Although the GPUs process requests independently and in parallel, they still need to communicate with each other. Remember that each GPU handles only a slice of the incoming prompts (for example, with 4 GPUs and 8 prompts using `--tensor-parallel-size=4`, each GPU participates in serving the full model).\nThis GPU-to-GPU communication is managed efficiently by NVIDIAโ€™s NCCL library. The communication mainly ensures that each GPU gets its correct portion of the incoming requests โ€” itโ€™s lightweight and doesnโ€™t interfere with generation itself.\nSeparately, the number of completions to generate per prompt is controlled by the `num_generations` setting in the GRPO config. For instance, if you set `num_generations=2` (like in the picture above), each prompt will have 2 completions. So, with 8 prompts and `num_generations=2`, you would end up with 16 completions total โ€” regardless of the number of GPUs or parallelism settings.\n\n### ๐Ÿฅธ More detail on what happens under the hood when running the server\n\n- The vLLM server starts by running the command: `trl vllm-serve --model Qwen/Qwen2.5-7B`.\n- Once the server is running, it generates completions based on requests from the client (trainer) using `vllm_client.generate` [these lines](https://github.com/huggingface/trl/blob/cc044e35b285be7dc062764b3364e1e684db4c7c/trl/trainer/grpo_trainer.py#L1025-L1035).\n- The client (trainer) then requests these completions from the server.\n- These completions are used to compute the reward signal.\n- Based on the reward signal and the modelโ€™s output, the loss is computed, and the backward pass is performed to update the modelโ€™s weights.\n- **Note**: The server only handles completion generation โ€” it doesnโ€™t train the model. Therefore, the modelโ€™s weights arenโ€™t updated on the server. Once the backward pass is complete, the client sends the updated weights to the server using `vllm_client.update_named_param(name, param.data)`.\n\nWhen using vLLM, ensure the GPUs assigned for training and generation are separate to avoid NCCL communication conflicts. If you do not set the `CUDA_VISIBLE_DEVICES` environment variable, the training script will use all available GPUs by default, which may lead to device conflicts. Starting from TRL next release after v0.19.1, the code automatically detects and prevents same-device usage, raising a error at the vllm server process:\n\n```log\nRuntimeError: Attempting to use the same CUDA device for multiple distinct roles/ranks within the same communicator. \nEnsure that trainer is using different devices than vLLM server.\n```\n\nFor example, if you want to use GPUs 4โ€“7 for training while the server runs on GPUs 0-3, set:\n\n```sh\nCUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch train.py\n```\n\n## Advanced usage\n\n### ๐Ÿท More customization options with vLLM?\n\nYou can customize the server configuration by passing additional arguments.\n\n```txt\n$ trl vllm-serve --help\nusage: trl vllm-serve [-h] --model MODEL [--revision REVISION] [--tensor_parallel_size TENSOR_PARALLEL_SIZE] [--data_parallel_size DATA_PARALLEL_SIZE] [--host HOST]\n [--port PORT] [--gpu_memory_utilization GPU_MEMORY_UTILIZATION] [--dtype DTYPE] [--max_model_len MAX_MODEL_LEN]\n [--enable_prefix_caching ENABLE_PREFIX_CACHING] [--enforce_eager [ENFORCE_EAGER]] [--kv_cache_dtype KV_CACHE_DTYPE]\n [--trust_remote_code [TRUST_REMOTE_CODE]] [--log_level LOG_LEVEL] [--vllm_model_impl VLLM_MODEL_IMPL]\n\noptions:\n -h, --help show this help message and exit\n --model MODEL Model name or path to load the model from. (default: None)\n --revision REVISION Revision to use for the model. If not specified, the default branch will be used. (default: None)\n --tensor_parallel_size TENSOR_PARALLEL_SIZE, --tensor-parallel-size TENSOR_PARALLEL_SIZE\n Number of tensor parallel workers to use. (default: 1)\n --data_parallel_size DATA_PARALLEL_SIZE, --data-parallel-size DATA_PARALLEL_SIZE\n Number of data parallel workers to use. For dense models, keep this at 1. Starting from vLLM `0.14.0`, setting\n this above `1` for dense models is no longer supported/useful and will error out (see vLLM PR #30739).\n (default: 1)\n --host HOST Host address to run the server on. (default: 0.0.0.0)\n --port PORT Port to run the server on. (default: 8000)\n --gpu_memory_utilization GPU_MEMORY_UTILIZATION, --gpu-memory-utilization GPU_MEMORY_UTILIZATION\n Ratio (between 0 and 1) of GPU memory to reserve for the model weights, activations, and KV cache on the device dedicated to generation\n powered by vLLM. Higher values will increase the KV cache size and thus improve the model's throughput. However, if the value is too high,\n it may cause out-of-memory (OOM) errors during initialization. (default: 0.9)\n --dtype DTYPE Data type to use for vLLM generation. If set to 'auto', the data type will be automatically determined based on the model configuration.\n Find the supported values in the vLLM documentation. (default: auto)\n --max_model_len MAX_MODEL_LEN, --max-model-len MAX_MODEL_LEN\n If set, the `max_model_len` to use for vLLM. This can be useful when running with reduced `vllm_gpu_memory_utilization`, leading to a\n reduced KV cache size. If not set, vLLM will use the model context size, which might be much larger than the KV cache, leading to\n inefficiencies. (default: None)\n --enable_prefix_caching ENABLE_PREFIX_CACHING, --enable-prefix-caching ENABLE_PREFIX_CACHING\n Whether to enable prefix caching in vLLM. If set to `True`, ensure that the model and the hardware support this feature. (default: None)\n --enforce_eager [ENFORCE_EAGER], --enforce-eager [ENFORCE_EAGER]\n Whether to enforce eager execution. If set to `True`, we will disable CUDA graph and always execute the model in eager mode. If `False`\n (default behavior), we will use CUDA graph and eager execution in hybrid. (default: False)\n --kv_cache_dtype KV_CACHE_DTYPE, --kv-cache-dtype KV_CACHE_DTYPE\n Data type to use for KV cache. If set to 'auto', the dtype will default to the model data type. (default: auto)\n --trust_remote_code [TRUST_REMOTE_CODE], --trust-remote-code [TRUST_REMOTE_CODE]\n Whether to trust remote code when loading models. Set to True to allow executing code from model repositories. This is required for some\n custom models but introduces security risks. (default: False)\n --log_level LOG_LEVEL, --log-level LOG_LEVEL\n Log level for uvicorn. Possible choices: 'critical', 'error', 'warning', 'info', 'debug', 'trace'. (default: info)\n --vllm_model_impl VLLM_MODEL_IMPL, --vllm-model-impl VLLM_MODEL_IMPL\n Model implementation to use for vLLM. Must be one of `transformers` or `vllm`. `transformers`: Use the `transformers` backend for model\n implementation. `vllm`: Use the `vllm` library for model implementation. (default: vllm)\n```\n\n### ๐Ÿ’†๐Ÿปโ€โ™€๏ธ What's the best distributed setup?\n\n![tp dp throughput 8 gpus](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/tp_dp_throughput_8_gpus.png)\n![tp dp throughput 4 gpus](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/tp_dp_throughput_4_gpus.png)\n\n> [!WARNING]\n> The benchmark plots above were collected with older vLLM versions. Starting with [vLLM PR #30739](https://github.com/vllm-project/vllm/pull/30739) (released in `0.14.0`), offline data parallel scaling for non-MoE (dense) models is no longer supported. To follow the latest recommendations, do not scale DP for non-MoE models.\n\n### vLLM with Transformers Backend\n\nvLLM can use the **Transformers backend** for model implementations, which works for both LLMs and VLMs.\nTo enable this, set `vllm_model_impl=\"transformers\"` in your configuration or pass it via the command-line argument.\n\nFor more details, check out [vLLM Transformers Backend](https://blog.vllm.ai/2025/04/11/transformers-backend.html).\n\nExample:\n\n```sh\nCUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen\n2.5-VL-3B-Instruct --tensor-parallel-size 1 --port 8000 --enforce_eager --vllm_model_impl transformers\n```\n\n### Modes of Using vLLM During Training\n\nTRL supports **two modes** for integrating vLLM during training: **colocate mode** (default) and **server mode**.\n\n#### Colocate Mode\n\nIn **colocate mode**, vLLM runs inside the trainer process and shares GPU memory with the training model.\nThis avoids launching a separate server and can improve GPU utilization, but may lead to memory contention on the training GPUs. This is the default mode.\n\nExample configuration:\n\n\n\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n ...,\n use_vllm=True, # vllm_mode=\"colocate\" by default\n)\n```\n\n\n\n\n```python\nfrom trl.experimental.online_dpo import OnlineDPOConfig\n\ntraining_args = OnlineDPOConfig(\n ...,\n use_vllm=True, # vllm_mode=\"colocate\" by default\n)\n```\n\n\n\n\n```python\nfrom trl.experimental.nash_md import NashMDConfig\n\ntraining_args = NashMDConfig(\n ...,\n use_vllm=True, # vllm_mode=\"colocate\" by default\n)\n```\n\n\n\n\n```python\nfrom trl.experimental.xpo import XPOConfig\n\ntraining_args = XPOConfig(\n ...,\n use_vllm=True, # vllm_mode=\"colocate\" by default\n)\n```\n\n\n\n\n```python\nfrom trl import RLOOConfig\n\ntraining_args = RLOOConfig(\n ...,\n use_vllm=True, # vllm_mode=\"colocate\" by default\n)\n```\n\n\n\n\n#### Server Mode\n\nIn **server mode**, vLLM runs as a separate process on dedicated GPUs and communicates with the trainer via HTTP.\nThis setup is ideal if you have GPUs dedicated to inference.\n\nExample configuration:\n\n\n\n\n```python\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n ...,\n use_vllm=True,\n vllm_mode=\"server\",\n)\n```\n\n\n\n\n```python\nfrom trl.experimental.online_dpo import OnlineDPOConfig\n\ntraining_args = OnlineDPOConfig(\n ...,\n use_vllm=True,\n vllm_mode=\"server\",\n)\n```\n\n\n\n\n```python\nfrom trl.experimental.nash_md import NashMDConfig\n\ntraining_args = NashMDConfig(\n ...,\n use_vllm=True,\n vllm_mode=\"server\",\n)\n```\n\n\n\n\n```python\nfrom trl.experimental.xpo import XPOConfig\n\ntraining_args = XPOConfig(\n ...,\n use_vllm=True,\n vllm_mode=\"server\",\n)\n```\n\n\n\n\n```python\nfrom trl import RLOOConfig\n\ntraining_args = RLOOConfig(\n ...,\n use_vllm=True,\n vllm_mode=\"server\",\n)\n```\n\n\n\n\n> [!WARNING]\n> Check the documentation of the trainer you are using for specific details on vLLM usage and parameters.\n\n> [!WARNING]\n> To reduce GPU memory usage when running vLLM, consider [enabling vLLM sleep mode](reducing_memory_usage#vllm-sleep-mode).\n" }, { "path": "docs/source/winrate_callback.md", "content": "# WinRateCallback\n\n[[autodoc]] experimental.winrate_callback.WinRateCallback\n" }, { "path": "docs/source/xpo_trainer.md", "content": "# XPO Trainer\n\n[![model badge](https://img.shields.io/badge/All_models-XPO-blue)](https://huggingface.co/models?other=xpo,trl)\n\n## Overview\n\nExploratory Preference Optimization (XPO) was proposed in the paper [Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF](https://huggingface.co/papers/2405.21046) by Tengyang Xie, Dylan J. Foster, Akshay Krishnamurthy, [Corby Rosset](https://huggingface.co/corbyrosset), [Ahmed Awadallah](https://huggingface.co/AhmedAwadallah), and Alexander Rakhlin. It is a simple online preference tuning method based on the DPO loss together with a reward model (RM). XPO augments the DPO objective with an exploration bonus allowing the method to explore outside the support of the initial model and human feedback data.\n\nThe abstract from the paper is the following:\n\n> Reinforcement learning from human feedback (RLHF) has emerged as a central tool for language model alignment. We consider online exploration in RLHF, which exploits interactive access to human or AI feedback by deliberately encouraging the model to produce diverse, maximally informative responses. By allowing RLHF to confidently stray from the pre-trained model, online exploration offers the possibility of novel, potentially super-human capabilities, but its full potential as a paradigm for language model training has yet to be realized, owing to computational and statistical bottlenecks in directly adapting existing reinforcement learning techniques. We propose a new algorithm for online exploration in RLHF, Exploratory Preference Optimization (XPO), which is simple and practical -- a one-line change to (online) Direct Preference Optimization (DPO; Rafailov et al., 2023) -- yet enjoys the strongest known provable guarantees and promising empirical performance. XPO augments the DPO objective with a novel and principled exploration bonus, empowering the algorithm to explore outside the support of the initial model and human feedback data. In theory, we show that XPO is provably sample-efficient and converges to a near-optimal language model policy under natural exploration conditions, irrespective of whether the initial model has good coverage. Our analysis, which builds on the observation that DPO implicitly performs a form of Q*-approximation (or, Bellman error minimization), combines previously disparate techniques from language modeling and theoretical reinforcement learning in a serendipitous fashion through the perspective of KL-regularized Markov decision processes. Empirically, we find that XPO is more sample-efficient than non-exploratory DPO variants in a preliminary evaluation.\n\nThis post-training method was contributed by [Kashif Rasul](https://huggingface.co/kashif), [Quentin Gallouรฉdec](https://huggingface.co/qgallouedec) and [Lewis Tunstall](https://huggingface.co/lewtun).\n\n> [!NOTE]\n> XPO is currently experimental. The API may change without notice while the feature is iterated on.\n\n## Quick start\n\nThis example demonstrates how to train a model using the XPO method. We use the [Qwen 0.5B model](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) as the base model and [`experimental.judges.PairRMJudge`] as a judge. We use the prompts from the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback). You can view the prompts in the dataset here:\n\n\nBelow is the script to train the model:\n\n```python\n# train_xpo.py\nfrom datasets import load_dataset\nfrom trl.experimental.judges import PairRMJudge\nfrom trl.experimental.xpo import XPOConfig, XPOTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2-0.5B-Instruct\")\njudge = PairRMJudge()\ntrain_dataset = load_dataset(\"trl-lib/ultrafeedback-prompt\", split=\"train\")\n\ntraining_args = XPOConfig(output_dir=\"Qwen2-0.5B-XPO\")\ntrainer = XPOTrainer(\n model=model, judge=judge, args=training_args, processing_class=tokenizer, train_dataset=train_dataset\n)\ntrainer.train()\n```\n\nExecute the script using the following command:\n\n```bash\naccelerate launch train_xpo.py\n```\n\nDistributed across 8 GPUs, the training takes approximately 1 hour.\n\nTo see how the [trained model](https://huggingface.co/trl-lib/Qwen2-0.5B-XPO) performs, you can use the [Transformers Chat CLI](https://huggingface.co/docs/transformers/quicktour#chat-with-text-generation-models).\n\n
$ transformers chat trl-lib/Qwen2-0.5B-XPO\n<quentin_gallouedec>:\nWhat is the best programming language?\n\n<trl-lib/Qwen2-0.5B-XPO>:\nThe best programming language depends on individual preferences and familiarity with coding concepts. Some popular languages include Python, Java, C++, and JavaScript.\n
\n\n## Expected dataset type\n\nXPO requires a [prompt-only dataset](dataset_formats#prompt-only). The [`experimental.xpo.XPOTrainer`] supports both [conversational](dataset_formats#conversational) and [standard](dataset_formats#standard) dataset format. When provided with a conversational dataset, the trainer will automatically apply the chat template to the dataset.\n\n## Usage tips\n\n### Use a reward model\n\nInstead of a judge, you can chose to use a reward model -- see [Reward Bench](https://huggingface.co/spaces/allenai/reward-bench) for a leaderboard of public models you can use. Below is a code example showing how to replace a judge with the [trl-lib/Qwen2-0.5B-Reward](https://huggingface.co/trl-lib/Qwen2-0.5B-Reward) model:\n\n```diff\n- from trl.experimental.judges import PairRMJudge\n+ from transformers import AutoModelForSequenceClassification\n\n- judge = PairRMJudge()\n+ reward_model = AutoModelForSequenceClassification.from_pretrained(\"trl-lib/Qwen2-0.5B-Reward\", num_labels=1)\n\n trainer = XPOTrainer(\n ...\n- judge=judge,\n+ reward_funcs=reward_model,\n )\n```\n\n> [!WARNING]\n> Make sure that the SFT model and reward model use the _same_ chat template and the same tokenizer. Otherwise, you may find the model completions are scored incorrectly during training.\n\n### Encourage EOS token generation\n\nWhen using a reward model, we may want the model to generate completions within a given length. During training, the model will generate completions up to the maximum length specified in the `max_new_tokens` argument of [`experimental.xpo.XPOConfig`]. If you want to penalize the model for not generating an EOS token before reaching the maximum length, you can use the `missing_eos_penalty` argument of [`experimental.xpo.XPOConfig`]:\n\n```python\ntraining_args = XPOConfig(..., max_new_tokens=128, missing_eos_penalty=1.0)\n```\n\n### Logging Completions\n\nTo better understand your modelโ€™s behavior during training, you can log sample completions periodically using the [`LogCompletionsCallback`].\n\n```python\ntrainer = XPOTrainer(..., eval_dataset=eval_dataset)\ncompletions_callback = LogCompletionsCallback(trainer, num_prompts=8)\ntrainer.add_callback(completions_callback)\n```\n\nThis callback logs the model's generated completions directly to Weights & Biases.\n\n![Logged Completions](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/wandb_completions.png)\n\n## Example script\n\nWe provide an example script to train a model using the XPO method. The script is available in [`examples/scripts/xpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/xpo.py)\n\nTo test the XPO script with the [Qwen2.5 0.5B model](https://huggingface.co/trl-lib/Qwen/Qwen2.5-0.5B-Instruct) on the [UltraFeedback dataset](https://huggingface.co/datasets/openbmb/UltraFeedback), run the following command:\n\n```bash\npython examples/scripts/xpo.py \\\n --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \\\n --judge pair_rm \\\n --dataset_name trl-lib/ultrafeedback-prompt \\\n --learning_rate 5.0e-7 \\\n --output_dir Qwen2.5-0.5B-XPO-PairRM \\\n --warmup_steps 0.1 \\\n --push_to_hub\n```\n\n## Logged metrics\n\nWhile training and evaluating we record the following reward metrics:\n\n* `loss/xpo`: The mean xpo part of the full loss.\n* `loss/dpo`: The mean dpo part of the full loss.\n* `objective/kl`: The mean KL divergence between the model and reference data.\n* `objective/entropy`: The mean entropy of the model and reference data.\n* `objective/model_scores`: The mean scores (according to the reward model) of the model completions.\n* `objective/ref_scores`: The mean scores (according to the reward model) of the reference completions.\n* `objective/scores_margin`: The mean score margin (according to the external reward model) between the chosen and rejected completions.\n* `rewards/chosen`: The mean reward (according to XPO's DPO implicit reward model) of the chosen completions.\n* `rewards/rejected`: The mean reward (according to XPO's DPO implicit reward model) of the rejected completions.\n* `rewards/accuracies`: The accuracies of the XPO's implicit reward model.\n* `rewards/margins`: The mean reward margin (according to online DPO's implicit reward model) between the chosen and rejected completions.\n* `logps/chosen`: The mean log probabilities of the chosen completions.\n* `logps/rejected`: The mean log probabilities of the rejected completions.\n* `val/model_contain_eos_token`: The amount of times the model's output contains the eos token.\n* `val/ref_contain_eos_token`: The amount of times the reference's output contains the eos token.\n* `alpha`: The weight of the XPO loss term. Typically fixed, but can be made dynamic by passing a list to [`experimental.xpo.XPOConfig`].\n* `beta`: The parameter that controls the weight of the loss term representing the deviation from the reference model. Typically fixed, but can be made dynamic by passing a list to [`experimental.xpo.XPOConfig`].\n\n## XPOTrainer\n\n[[autodoc]] experimental.xpo.XPOTrainer\n - train\n - save_model\n - push_to_hub\n\n## XPOConfig\n\n[[autodoc]] experimental.xpo.XPOConfig\n" }, { "path": "examples/README.md", "content": "# Examples\n\nPlease check out https://huggingface.co/docs/trl/example_overview for documentation on our examples.\n" }, { "path": "examples/accelerate_configs/alst_ulysses_4gpu.yaml", "content": "# ALST/Ulysses Sequence Parallelism with 2D Parallelism (DP + SP) for 4 GPUs\n#\n# This configuration enables 2D parallelism:\n# - Sequence Parallelism (sp_size=2): Sequences split across 2 GPUs using ALST/Ulysses\n# - Data Parallelism (dp_shard_size=2): Model/optimizer sharded across 2 GPUs\n# - Total: 4 GPUs (2 ร— 2)\n#\n# Set parallelism_config in your training script:\n# parallelism_config = ParallelismConfig(\n# sp_backend=\"deepspeed\",\n# sp_size=2,\n# dp_shard_size=2, # Calculated as: num_gpus // sp_size\n# sp_handler=DeepSpeedSequenceParallelConfig(...)\n# )\n\ncompute_environment: LOCAL_MACHINE\ndebug: false\ndeepspeed_config:\n zero_stage: 3\n seq_parallel_communication_data_type: bf16\n offload_optimizer_device: none\n offload_param_device: none\n zero3_init_flag: true\n zero3_save_16bit_model: true\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 4 # Total number of GPUs\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\nparallelism_config:\n parallelism_config_dp_replicate_size: 1\n parallelism_config_dp_shard_size: 2 # Enables 2D parallelism with SP\n parallelism_config_tp_size: 1\n parallelism_config_sp_size: 2 # Sequence parallel size\n parallelism_config_sp_backend: deepspeed\n parallelism_config_sp_seq_length_is_variable: true\n parallelism_config_sp_attn_implementation: flash_attention_2\n" }, { "path": "examples/accelerate_configs/context_parallel_2gpu.yaml", "content": "# Context Parallelism with FSDP for 2 GPUs\ncompute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: FSDP\ndowncast_bf16: 'no'\nenable_cpu_affinity: false\nfsdp_config:\n fsdp_activation_checkpointing: true # Enable activation checkpointing for memory efficiency\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_cpu_ram_efficient_loading: true\n fsdp_offload_params: false\n fsdp_reshard_after_forward: true\n fsdp_state_dict_type: FULL_STATE_DICT\n fsdp_version: 2\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 2 # Number of GPUs\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\nparallelism_config:\n parallelism_config_dp_replicate_size: 1\n parallelism_config_dp_shard_size: 1\n parallelism_config_tp_size: 1\n parallelism_config_cp_size: 2 # Context parallel size\n" }, { "path": "examples/accelerate_configs/deepspeed_zero1.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndeepspeed_config:\n deepspeed_multinode_launcher: standard\n gradient_accumulation_steps: 1\n zero3_init_flag: false\n zero_stage: 1\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: 'bf16'\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "examples/accelerate_configs/deepspeed_zero2.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndeepspeed_config:\n deepspeed_multinode_launcher: standard\n offload_optimizer_device: none\n offload_param_device: none\n zero3_init_flag: false\n zero_stage: 2\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: 'bf16'\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "examples/accelerate_configs/deepspeed_zero3.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndeepspeed_config:\n deepspeed_multinode_launcher: standard\n offload_optimizer_device: none\n offload_param_device: none\n zero3_init_flag: true\n zero3_save_16bit_model: true\n zero_stage: 3\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "examples/accelerate_configs/fsdp1.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: FSDP\ndowncast_bf16: 'no'\nenable_cpu_affinity: false\nfsdp_config:\n fsdp_activation_checkpointing: false\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_backward_prefetch: BACKWARD_PRE\n fsdp_cpu_ram_efficient_loading: true\n fsdp_forward_prefetch: true\n fsdp_offload_params: false\n fsdp_reshard_after_forward: FULL_SHARD\n fsdp_state_dict_type: FULL_STATE_DICT\n fsdp_sync_module_states: true\n fsdp_use_orig_params: true\n fsdp_version: 1\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "examples/accelerate_configs/fsdp2.yaml", "content": "# Requires accelerate 1.7.0 or higher\ncompute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: FSDP\ndowncast_bf16: 'no'\nenable_cpu_affinity: false\nfsdp_config:\n fsdp_activation_checkpointing: false\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_cpu_ram_efficient_loading: true\n fsdp_offload_params: false\n fsdp_reshard_after_forward: true\n fsdp_state_dict_type: FULL_STATE_DICT\n fsdp_version: 2\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "examples/accelerate_configs/multi_gpu.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: MULTI_GPU\ndowncast_bf16: 'no'\ngpu_ids: all\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: 'bf16'\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "examples/accelerate_configs/single_gpu.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: \"NO\"\ndowncast_bf16: 'no'\ngpu_ids: all\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: 'bf16'\nnum_machines: 1\nnum_processes: 1\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "examples/cli_configs/example_config.yaml", "content": "# This is an example configuration file of TRL CLI, you can use it for \n# SFT like that: `trl sft --config config.yaml --output_dir test-sft`\n# The YAML file supports environment variables by adding an `env` field\n# as below\n\n# env:\n# CUDA_VISIBLE_DEVICES: 0\n\nmodel_name_or_path:\n Qwen/Qwen2.5-0.5B\ndataset_name:\n stanfordnlp/imdb\nreport_to:\n none\nlearning_rate:\n 0.0001\nlr_scheduler_type:\n cosine\n" }, { "path": "examples/datasets/deepmath_103k.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/DeepMath-103K\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/DeepMath-103K\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef process_example(example):\n solution = example[\"final_answer\"]\n if solution not in [\"True\", \"False\", \"Yes\", \"No\"]:\n solution = f\"${solution}$\"\n prompt = [{\"role\": \"user\", \"content\": example[\"question\"]}]\n return {\"prompt\": prompt, \"solution\": solution}\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# DeepMath-103K Dataset\n\n## Summary\n\n[DeepMath-103K](https://huggingface.co/datasets/zwhe99/DeepMath-103K) is meticulously curated to push the boundaries of mathematical reasoning in language models.\n\n## Data Structure\n\n- **Format**: [Conversational](https://huggingface.co/docs/trl/main/dataset_formats#conversational)\n- **Type**: [Prompt-only](https://huggingface.co/docs/trl/main/dataset_formats#prompt-only)\n\nColumn:\n- `\"prompt\"`: The input question.\n- `\"solution\"`: The solution to the math problem.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/deepmath_103k.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\"zwhe99/DeepMath-103K\", split=\"train\")\n\n dataset = dataset.map(\n process_example,\n remove_columns=dataset.column_names,\n num_proc=script_args.dataset_num_proc,\n )\n dataset = dataset.train_test_split(test_size=0.05, seed=42)\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/hh-rlhf-helpful-base.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport re\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/hh-rlhf-helpful-base\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/hh-rlhf-helpful-base\", metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"}\n )\n dataset_num_proc: int | None = field(\n default=None, metadata={\"help\": \"Number of workers to use for dataset processing.\"}\n )\n\n\ndef common_start(str1: str, str2: str) -> str:\n # Zip the two strings and iterate over them together\n common_chars = []\n for c1, c2 in zip(str1, str2, strict=True):\n if c1 == c2:\n common_chars.append(c1)\n else:\n break\n # Join the common characters and return as a string\n return \"\".join(common_chars)\n\n\ndef extract_dialogue(example: str) -> list[dict[str, str]]:\n # Extract the prompt, which corresponds to the common start of the chosen and rejected dialogues\n prompt_text = common_start(example[\"chosen\"], example[\"rejected\"])\n\n # The chosen and rejected may share a common start, so we need to remove the common part\n if not prompt_text.endswith(\"\\n\\nAssistant: \"):\n prompt_text = prompt_text[: prompt_text.rfind(\"\\n\\nAssistant: \")] + \"\\n\\nAssistant: \"\n\n # Extract the chosen and rejected lines\n chosen_line = example[\"chosen\"][len(prompt_text) :]\n rejected_line = example[\"rejected\"][len(prompt_text) :]\n\n # Remove the generation prompt (\"\\n\\nAssistant: \") from the prompt\n prompt_text = prompt_text[: -len(\"\\n\\nAssistant: \")]\n\n # Split the string at every occurrence of \"Human: \" or \"Assistant: \"\n prompt_lines = re.split(r\"(\\n\\nAssistant: |\\n\\nHuman: )\", prompt_text)\n\n # Remove the first element as it's empty\n prompt_lines = prompt_lines[1:]\n\n prompt = []\n for idx in range(0, len(prompt_lines), 2):\n role = \"user\" if prompt_lines[idx] == \"\\n\\nHuman: \" else \"assistant\"\n content = prompt_lines[idx + 1]\n prompt.append({\"role\": role, \"content\": content})\n\n # Remove the prompt from the chosen and rejected dialogues\n chosen = [{\"role\": \"assistant\", \"content\": chosen_line}]\n rejected = [{\"role\": \"assistant\", \"content\": rejected_line}]\n\n return {\"prompt\": prompt, \"chosen\": chosen, \"rejected\": rejected}\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# HH-RLHF-Helpful-Base Dataset\n\n## Summary\n\nThe HH-RLHF-Helpful-Base dataset is a processed version of [Anthropic's HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset, specifically curated to train models using the [TRL library](https://github.com/huggingface/trl) for preference learning and alignment tasks. It contains pairs of text samples, each labeled as either \"chosen\" or \"rejected,\" based on human preferences regarding the helpfulness of the responses. This dataset enables models to learn human preferences in generating helpful responses, enhancing their ability to assist users effectively.\n\n## Data Structure\n\n- **Format**: [Conversational](https://huggingface.co/docs/trl/main/dataset_formats#conversational)\n- **Type**: [Preference](https://huggingface.co/docs/trl/main/dataset_formats#preference)\n\nColumns:\n- `\"prompt\"`: The user query.\n- `\"chosen\"`: A response deemed helpful by human evaluators.\n- `\"rejected\"`: A response considered less helpful or unhelpful.\n\nThis structure allows models to learn to prefer the _chosen_ response over the _rejected_ one, thereby aligning with human preferences in helpfulness.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/hh-rlhf-helpful-base.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\"Anthropic/hh-rlhf\", data_dir=\"helpful-base\")\n dataset = dataset.map(extract_dialogue, num_proc=script_args.dataset_num_proc)\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/llava_instruct_mix.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport ast\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/llava-instruct-mix\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/llava-instruct-mix\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef process_example(example):\n messages = []\n for message in ast.literal_eval(example[\"conversations\"]):\n content = message[\"value\"]\n content = content.replace(\"\", \"\").strip()\n role = \"user\" if message[\"from\"] == \"human\" else \"assistant\"\n messages.append({\"role\": role, \"content\": content})\n return {\"messages\": messages, \"images\": [example[\"image\"]]}\n\n\ndef filter_long_examples(example):\n total_length = sum(len(msg[\"content\"]) for msg in example[\"messages\"])\n return total_length <= 1000\n\n\ndef split_prompt_completion(example):\n \"\"\"\n Splits the messages into a prompt and a completion. The last message is considered the completion.\n \"\"\"\n assert len(example[\"messages\"]) > 1\n example[\"prompt\"] = example[\"messages\"][:-1]\n example[\"completion\"] = example[\"messages\"][-1:]\n return example\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# LLaVA Instruct Mix\n\n## Summary\n\nThe LLaVA Instruct Mix dataset is a processed version of [LLaVA Instruct Mix](https://huggingface.co/datasets/theblackcat102/llava-instruct-mix).\n\n## Data Structure\n\n- **Format**: [Conversational](https://huggingface.co/docs/trl/main/dataset_formats#conversational)\n- **Type**: [Language-modeling](https://huggingface.co/docs/trl/main/dataset_formats#language-modeling)\n\nColumns:\n- `\"images\"`: The image associated with the text.\n- `\"prompt\"`: A list of messages that form the context for the conversation.\n- `\"completion\"`: The last message in the conversation, which is the model's response.\n\nThis structure allows models to learn from the context of the conversation, enhancing their understanding of how to generate descriptive text based on visual inputs.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/llava_instruct_mix.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\"theblackcat102/llava-instruct-mix\", split=\"train\", num_proc=script_args.dataset_num_proc)\n\n dataset = dataset.map(\n process_example, remove_columns=[\"conversations\", \"image\"], num_proc=script_args.dataset_num_proc\n )\n dataset = dataset.filter(filter_long_examples, num_proc=script_args.dataset_num_proc)\n dataset = dataset.map(split_prompt_completion, remove_columns=[\"messages\"], num_proc=script_args.dataset_num_proc)\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id, num_proc=script_args.dataset_num_proc)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/lm-human-preferences-descriptiveness.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import AutoTokenizer, HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/lm-human-preferences-descriptiveness\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/lm-human-preferences-descriptiveness\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\n# Edge cases handling: remove the cases where all samples are the same\ndef samples_not_all_same(example):\n return not all(example[\"sample0\"] == example[f\"sample{j}\"] for j in range(1, 4))\n\n\ndef to_prompt_completion(example, tokenizer):\n prompt = tokenizer.decode(example[\"query\"]).strip()\n best_idx = example[\"best\"]\n chosen = tokenizer.decode(example[f\"sample{best_idx}\"])\n for rejected_idx in range(4): # take the first rejected sample that is different from the chosen one\n rejected = tokenizer.decode(example[f\"sample{rejected_idx}\"])\n if chosen != rejected:\n break\n assert chosen != rejected\n return {\"prompt\": prompt, \"chosen\": chosen, \"rejected\": rejected}\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# LM-Human-Preferences-Descriptiveness Dataset\n\n## Summary\n\nThe LM-Human-Preferences-Descriptiveness dataset is a processed subset of [OpenAI's LM-Human-Preferences](https://github.com/openai/lm-human-preferences), focusing specifically on enhancing the descriptiveness of generated text. It contains pairs of text samples, each labeled as either \"chosen\" or \"rejected,\" based on human preferences regarding the level of detail and vividness in the descriptions. This dataset enables models to learn human preferences in descriptive language, improving their ability to generate rich and engaging narratives.\n\n## Data Structure\n\n- **Format**: [Standard](https://huggingface.co/docs/trl/main/dataset_formats#standard)\n- **Type**: [Preference](https://huggingface.co/docs/trl/main/dataset_formats#preference)\n\nColumns:\n- `\"prompt\"`: The text sample.\n- `\"chosen\"`: A version of the text with enhanced descriptiveness.\n- `\"rejected\"`: A version of the text with less descriptiveness.\n\nThis structure allows models to learn to prefer the _chosen_ response over the _rejected_ one, thereby aligning with human preferences in descriptive language.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/lm-human-preferences-descriptiveness.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\n \"json\",\n data_files=\"https://openaipublic.blob.core.windows.net/lm-human-preferences/labels/descriptiveness/offline_5k.json\",\n split=\"train\",\n )\n\n dataset = dataset.filter(samples_not_all_same, num_proc=script_args.dataset_num_proc)\n\n dataset = dataset.map(\n to_prompt_completion,\n num_proc=script_args.dataset_num_proc,\n remove_columns=[\"query\", \"sample0\", \"sample1\", \"sample2\", \"sample3\", \"best\"],\n fn_kwargs={\"tokenizer\": AutoTokenizer.from_pretrained(\"gpt2\")},\n )\n\n # train_size taken from https://github.com/openai/lm-human-preferences/blob/cbfd210bb8b08f6bc5c26878c10984b90f516c66/launch.py#L79)\n dataset = dataset.train_test_split(train_size=4992)\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/lm-human-preferences-sentiment.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import AutoTokenizer, HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/lm-human-preferences-sentiment\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/lm-human-preferences-sentiment\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef to_prompt_completion(example, tokenizer):\n prompt = tokenizer.decode(example[\"query\"]).strip()\n best_idx = example[\"best\"]\n chosen = tokenizer.decode(example[f\"sample{best_idx}\"])\n for rejected_idx in range(4): # take the first rejected sample that is different from the chosen one\n rejected = tokenizer.decode(example[f\"sample{rejected_idx}\"])\n if chosen != rejected:\n break\n assert chosen != rejected\n return {\"prompt\": prompt, \"chosen\": chosen, \"rejected\": rejected}\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# LM-Human-Preferences-Sentiment Dataset\n\n## Summary\n\nThe LM-Human-Preferences-Sentiment dataset is a processed subset of [OpenAI's LM-Human-Preferences](https://github.com/openai/lm-human-preferences), focusing specifically on sentiment analysis tasks. It contains pairs of text samples, each labeled as either \"chosen\" or \"rejected,\" based on human preferences regarding the sentiment conveyed in the text. This dataset enables models to learn human preferences in sentiment expression, enhancing their ability to generate and evaluate text with desired emotional tones.\n\n## Data Structure\n\n- **Format**: [Standard](https://huggingface.co/docs/trl/main/dataset_formats#standard)\n- **Type**: [Preference](https://huggingface.co/docs/trl/main/dataset_formats#preference)\n\nColumns:\n- `\"prompt\"`: The text sample.\n- `\"chosen\"`: A version of the text that conveys the desired sentiment.\n- `\"rejected\"`: A version of the text that does not convey the desired sentiment.\n\nThis structure allows models to learn to prefer the _chosen_ response over the _rejected_ one, thereby aligning with human preferences in sentiment expression.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/lm-human-preferences-sentiment.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\n \"json\",\n data_files=\"https://openaipublic.blob.core.windows.net/lm-human-preferences/labels/sentiment/offline_5k.json\",\n split=\"train\",\n )\n\n dataset = dataset.map(\n to_prompt_completion,\n num_proc=script_args.dataset_num_proc,\n remove_columns=[\"query\", \"sample0\", \"sample1\", \"sample2\", \"sample3\", \"best\"],\n fn_kwargs={\"tokenizer\": AutoTokenizer.from_pretrained(\"gpt2\")},\n )\n\n # train_size taken from https://github.com/openai/lm-human-preferences/blob/cbfd210bb8b08f6bc5c26878c10984b90f516c66/launch.py#L70)\n dataset = dataset.train_test_split(train_size=4992)\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/math_shepherd.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport re\nfrom dataclasses import dataclass, field\nfrom itertools import chain\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/math_shepherd\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/math_shepherd\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef process_example(example):\n # Replace \"ะบะธ\" with \"โถป\" so that the size of the \"input\" matches the size of the \"label\"\n inputs = example[\"input\"].replace(\"ะบะธ\", \"โถป\")\n\n # Find the indices of the \"โถป\" characters (that should match with the indexes of the \"+\" or \"-\" in the label)\n indexes = [m.start() for m in re.finditer(\"โถป\", inputs)]\n\n # Sanity that all indexes are either \"+\" or \"-\"\n assert all(example[\"label\"][idx] in [\"+\", \"-\"] for idx in indexes)\n\n # Get the labels\n labels = [example[\"label\"][idx] == \"+\" for idx in indexes]\n\n # Split the inputs into steps (caution, the first step is missing here, it is the prompt)\n steps = [inputs[i:j] for i, j in zip(chain([0], indexes), chain(indexes, [None]), strict=True)]\n\n # Remove the last step (single โถป)\n steps = steps[:-1]\n\n # Get the prompt (first part) and completions (rest)\n prompt = steps[0]\n completions = steps[1:]\n\n # Remove the heading \"โถป\" and the final whitespace from the completions\n assert all(completion.startswith(\"โถป\") for completion in completions)\n completions = [completion[1:].strip() for completion in completions]\n\n # At this point, we need to retrieve the first step from the prompt.\n # First, we handle particular cases (annotation error) where we have a first label before the end of the prompt.\n if prompt.startswith(\n (\n \"Mr. Rocky\",\n \"Parker\",\n \"What is the smallest positive\",\n \" The Myth\",\n \"Let $\\\\mathbf{a}$\",\n \"Find the arithmetic\",\n \"Determine an ordered pair\",\n \"Determine the ordered pair\",\n \"At the Quill and Scroll stationery\",\n \"Round to the nearest\",\n r\"Calculate $\\sqrt{10p}\",\n r\"Simplify $\\sqrt{28x}\",\n )\n ):\n # Some spotted datasets errors where there is an annotation in the prompt: we remove it\n labels = labels[1:]\n\n # Then we handle the general case: we get the first step from the prompt by looking for \"Step 1:\" or \"step 1:\" or\n # (less common) \"?\".\n elif \"Step 1:\" in prompt:\n prompt, first_step = prompt.split(\"Step 1:\")\n first_step = \"Step 1:\" + first_step\n completions = [first_step.strip()] + completions\n elif \"step 1:\" in prompt:\n prompt, first_step = prompt.split(\"step 1:\")\n first_step = \"step 1:\" + first_step\n completions = [first_step.strip()] + completions\n elif \"?\" in prompt:\n prompt, first_step = prompt.split(\"?\")\n prompt = prompt + \"?\"\n completions = [first_step.strip()] + completions\n else:\n raise ValueError(f\"Prompt can't be processed: {prompt}\")\n\n # Strip the prompt\n prompt = prompt.strip()\n\n # Sanity check that the length of the completions is the same as the length of the labels\n assert len(completions) == len(labels)\n\n return {\"prompt\": prompt, \"completions\": completions, \"labels\": labels}\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# Math-Shepherd Dataset\n\n## Summary\n\nThe Math-Shepherd dataset is a processed version of [Math-Shepherd dataset](peiyi9979/Math-Shepherd), designed to train models using the [TRL library](https://github.com/huggingface/trl) for stepwise supervision tasks. It provides step-by-step solutions to mathematical problems, enabling models to learn and verify each step of a solution, thereby enhancing their reasoning capabilities.\n\n## Data Structure\n\n- **Format**: [Standard](https://huggingface.co/docs/trl/main/dataset_formats#standard)\n- **Type**: [Stepwise supervision](https://huggingface.co/docs/trl/main/dataset_formats#stepwise-supervision)\n\nColumns:\n- `\"prompt\"`: The problem statement.\n- `\"completions\"`: A list of reasoning steps generated to solve the problem.\n- `\"labels\"`: A list of booleans or floats indicating the correctness of each corresponding reasoning step.\n\nThis structure allows models to learn the correctness of each step in a solution, facilitating improved reasoning and problem-solving abilities.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/math_shepherd.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\"peiyi9979/Math-Shepherd\", split=\"train\")\n\n dataset = dataset.map(\n process_example,\n remove_columns=[\"input\", \"label\", \"task\"],\n num_proc=script_args.dataset_num_proc,\n )\n dataset = dataset.train_test_split(test_size=0.05, seed=42)\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/prm800k.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/prm800k\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/prm800k\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef process_example(example):\n outputs = []\n prompt = example[\"question\"][\"problem\"]\n\n # Iterate through each step\n previous_completions = []\n previous_labels = []\n for step in example[\"label\"][\"steps\"]:\n if step[\"completions\"] is None and step[\"human_completion\"] is None and step[\"chosen_completion\"] is None:\n # happens sometimes\n break\n # Loop through completions\n for completion_idx, completion in enumerate(step[\"completions\"]):\n # For every completion that are not chosen, we are in a terminal state, so we can add it to the list of outputs.\n if completion_idx != step[\"chosen_completion\"]:\n content = completion[\"text\"]\n completions = previous_completions[:] + [content]\n label = completion[\"rating\"] == 1\n labels = previous_labels[:] + [label]\n outputs.append({\"prompt\": prompt, \"completions\": completions, \"labels\": labels})\n\n # Now, expand the previous completions and labels\n if step[\"chosen_completion\"] is not None:\n chosen_completion = step[\"completions\"][step[\"chosen_completion\"]]\n label = chosen_completion[\"rating\"] == 1\n elif step[\"human_completion\"] is not None:\n chosen_completion = step[\"human_completion\"]\n label = True\n else:\n break\n content = chosen_completion[\"text\"]\n previous_completions.append(content)\n previous_labels.append(label)\n\n # Last step: we are in a terminal state, so we can add it to the list of outputs\n outputs.append({\"prompt\": prompt, \"completions\": previous_completions, \"labels\": previous_labels})\n return outputs\n\n\ndef process_batch(examples):\n outputs = []\n batch_size = len(examples[\"label\"])\n for idx in range(batch_size):\n example = {k: v[idx] for k, v in examples.items()}\n outputs.extend(process_example(example))\n # list of dict to dict of list\n outputs = {k: [v[k] for v in outputs] for k in outputs[0]}\n return outputs\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# PRM800K Dataset\n\n## Summary\n\nThe PRM800K dataset is a processed version of [OpenAI's PRM800K](https://github.com/openai/prm800k), designed to train models using the [TRL library](https://github.com/huggingface/trl) for stepwise supervision tasks. It contains 800,000 step-level correctness labels for model-generated solutions to problems from the MATH dataset. This dataset enables models to learn and verify each step of a solution, enhancing their reasoning capabilities.\n\n## Data Structure\n\n- **Format**: [Standard](https://huggingface.co/docs/trl/main/dataset_formats#standard)\n- **Type**: [Stepwise supervision](https://huggingface.co/docs/trl/main/dataset_formats#stepwise-supervision)\n\nColumns:\n- `\"prompt\"`: The problem statement.\n- `\"completions\"`: A list of reasoning steps generated to solve the problem.\n- `\"labels\"`: A list of booleans or floats indicating the correctness of each corresponding reasoning step.\n\nThis structure allows models to learn the correctness of each step in a solution, facilitating improved reasoning and problem-solving abilities.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/prm800k.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n data_files = {\n \"train\": \"https://github.com/openai/prm800k/raw/refs/heads/main/prm800k/data/phase1_train.jsonl\",\n \"test\": \"https://github.com/openai/prm800k/raw/refs/heads/main/prm800k/data/phase1_test.jsonl\",\n }\n dataset = load_dataset(\"json\", data_files=data_files)\n\n dataset = dataset.map(\n process_batch,\n batched=True,\n batch_size=10,\n remove_columns=[\n \"labeler\",\n \"timestamp\",\n \"generation\",\n \"is_quality_control_question\",\n \"is_initial_screening_question\",\n \"question\",\n \"label\",\n ],\n num_proc=script_args.dataset_num_proc,\n )\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/rlaif-v.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import features, load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/rlaif-v\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/rlaif-v\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef to_conversational(example):\n \"\"\"\n Convert prompt from \"xxx\" to [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"xxx\"}]}]\n and chosen and rejected from \"xxx\" to [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"xxx\"}]}].\n Images are wrapped into a list.\n \"\"\"\n prompt = [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": example[\"question\"]}]}]\n chosen = [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": example[\"chosen\"]}]}]\n rejected = [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": example[\"rejected\"]}]}]\n return {\"prompt\": prompt, \"images\": [example[\"image\"]], \"chosen\": chosen, \"rejected\": rejected}\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# RLAIF-V Dataset\n\n## Summary\n\nThe RLAIF-V dataset is a processed version of the [openbmb/RLAIF-V-Dataset](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset#dataset-card-for-rlaif-v-dataset), specifically curated to train vision-language models using the [TRL library](https://github.com/huggingface/trl) for preference learning tasks. It contains 83,132 high-quality comparison pairs, each comprising an image and two textual descriptions: one preferred and one rejected. This dataset enables models to learn human preferences in visual contexts, enhancing their ability to generate and evaluate image captions.\n\n## Data Structure\n\n- **Format**: [Conversational](https://huggingface.co/docs/trl/main/dataset_formats#conversational)\n- **Type**: [Preference](https://huggingface.co/docs/trl/main/dataset_formats#preference)\n\nColumns:\n- `\"prompt\"`: The task related to the image.\n- `\"images\"`: The image.\n- `\"chosen\"`: The preferred answer.\n- `\"rejected\"`: An alternative answer that was not preferred.\n\nThis structure allows models to learn to prefer the _chosen_ response over the _rejected_ one, thereby aligning with human preferences in visual tasks.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/rlaif-v.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\"openbmb/RLAIF-V-Dataset\", split=\"train\")\n dataset = dataset.map(\n to_conversational,\n num_proc=script_args.dataset_num_proc,\n remove_columns=dataset.column_names,\n writer_batch_size=128,\n )\n\n # Cast the images to Sequence[Image] to avoid bytes format\n f = dataset.features\n f[\"images\"] = features.Sequence(features.Image(decode=True))\n dataset = dataset.cast(f)\n\n dataset = dataset.train_test_split(test_size=0.01, writer_batch_size=128)\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/tldr.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/tldr\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/tldr\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef to_prompt_completion(example):\n tldr_format_str = \"SUBREDDIT: r/{subreddit}\\n\\nTITLE: {title}\\n\\nPOST: {post}\\n\\nTL;DR:\"\n prompt = tldr_format_str.format(subreddit=example[\"subreddit\"], title=example[\"title\"], post=example[\"post\"])\n completion = \" \" + example[\"summary\"] # Add a space to separate the prompt from the completion\n return {\"prompt\": prompt, \"completion\": completion}\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# TL;DR Dataset\n\n## Summary\n\nThe TL;DR dataset is a processed version of Reddit posts, specifically curated to train models using the [TRL library](https://github.com/huggingface/trl) for summarization tasks. It leverages the common practice on Reddit where users append \"TL;DR\" (Too Long; Didn't Read) summaries to lengthy posts, providing a rich source of paired text data for training summarization models.\n\n## Data Structure\n\n- **Format**: [Standard](https://huggingface.co/docs/trl/main/dataset_formats#standard)\n- **Type**: [Prompt-completion](https://huggingface.co/docs/trl/main/dataset_formats#prompt-completion)\n\nColumns:\n- `\"prompt\"`: The unabridged Reddit post.\n- `\"completion\"`: The concise \"TL;DR\" summary appended by the author.\n\nThis structure enables models to learn the relationship between detailed content and its abbreviated form, enhancing their summarization capabilities.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/tldr.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n # Filtered reddit TL;DR dataset from https://github.com/openai/summarize-from-feedback?tab=readme-ov-file#reddit-tldr-dataset\n data_files = {\n \"train\": \"https://openaipublic.blob.core.windows.net/summarize-from-feedback/datasets/tldr_3_filtered/train.jsonl\",\n \"validation\": \"https://openaipublic.blob.core.windows.net/summarize-from-feedback/datasets/tldr_3_filtered/valid.jsonl\",\n \"test\": \"https://openaipublic.blob.core.windows.net/summarize-from-feedback/datasets/tldr_3_filtered/test.jsonl\",\n }\n dataset = load_dataset(\"json\", data_files=data_files)\n\n dataset = dataset.map(\n to_prompt_completion,\n num_proc=script_args.dataset_num_proc,\n remove_columns=[\"id\", \"subreddit\", \"title\", \"post\", \"summary\"],\n )\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/tldr_preference.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/tldr-preference\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/tldr-preference\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef to_preference(example):\n info = example[\"info\"]\n if example[\"batch\"] in [\"batch0_cnndm\", \"cnndm0\", \"cnndm2\"]: # CNN Daily Mail batches\n article = info[\"article\"].replace(\"\\n\\n\", \"\\n\")\n prompt = f\"TITLE: {info['title']}\\n\\n{article}\\n\\nTL;DR:\"\n elif example[\"batch\"] in [f\"batch{i}\" for i in range(3, 23)] + [\"edit_b2_eval_test\"]: # Reddit batches\n post = info[\"post\"].replace(\"\\n\\n\", \"\\n\")\n prompt = f\"SUBREDDIT: r/{info['subreddit']}\\n\\nTITLE: {info['title']}\\n\\nPOST: {post}\\n\\nTL;DR:\"\n else:\n raise ValueError(f\"Unknown batch: {example['batch']}\")\n\n chosen_idx = example[\"choice\"]\n rejected_idx = 1 - chosen_idx\n chosen = example[\"summaries\"][chosen_idx][\"text\"]\n rejected = example[\"summaries\"][rejected_idx][\"text\"]\n return {\"prompt\": prompt, \"chosen\": chosen, \"rejected\": rejected}\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# TL;DR Dataset for Preference Learning\n\n## Summary\n\nThe TL;DR dataset is a processed version of Reddit posts, specifically curated to train models using the [TRL library](https://github.com/huggingface/trl) for preference learning and Reinforcement Learning from Human Feedback (RLHF) tasks. It leverages the common practice on Reddit where users append \"TL;DR\" (Too Long; Didn't Read) summaries to lengthy posts, providing a rich source of paired text data for training models to understand and generate concise summaries.\n\n## Data Structure\n\n- **Format**: [Standard](https://huggingface.co/docs/trl/main/dataset_formats#standard)\n- **Type**: [Preference](https://huggingface.co/docs/trl/main/dataset_formats#preference)\n\nColumns:\n- `\"prompt\"`: The unabridged Reddit post.\n- `\"chosen\"`: The concise \"TL;DR\" summary appended by the author.\n- `\"rejected\"`: An alternative summary or response that was not selected.\n\nThis structure enables models to learn the relationship between detailed content and its abbreviated form, enhancing their summarization capabilities.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/tldr_preference.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\"openai/summarize_from_feedback\", \"comparisons\")\n\n dataset = dataset.map(\n to_preference,\n num_proc=script_args.dataset_num_proc,\n remove_columns=[\"info\", \"summaries\", \"choice\", \"worker\", \"batch\", \"split\", \"extra\"],\n )\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/ultrafeedback-prompt.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/ultrafeedback-prompt\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/ultrafeedback-prompt\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef to_unpaired_preference(example):\n prompt = [{\"role\": \"user\", \"content\": example[\"instruction\"]}]\n return {\"prompt\": prompt}\n\n\ndef drop_long_prompt(example):\n if len(example[\"prompt\"][0][\"content\"]) > 512:\n return False\n else:\n return True\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# UltraFeedback - Prompts Dataset\n\n## Summary\n\nThe UltraFeedback - Prompts dataset is a processed version of the [UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) dataset for model evaluation on specific aspects like helpfulness, honesty, and instruction-following.\n\n## Data Structure\n\n- **Format**: [Conversational](https://huggingface.co/docs/trl/main/dataset_formats#conversational)\n- **Type**: [Prompt-only](https://huggingface.co/docs/trl/main/dataset_formats#prompt-only)\n\nColumn:\n- `\"prompt\"`: The input question or instruction provided to the model.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/ultrafeedback-prompt.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\"openbmb/UltraFeedback\", split=\"train\")\n\n dataset = dataset.map(\n to_unpaired_preference,\n remove_columns=[\"source\", \"instruction\", \"models\", \"completions\", \"correct_answers\", \"incorrect_answers\"],\n num_proc=script_args.dataset_num_proc,\n )\n dataset = dataset.filter(drop_long_prompt)\n dataset = dataset.train_test_split(test_size=0.05, seed=42)\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/datasets/ultrafeedback.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom huggingface_hub import ModelCard\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n model_name (`str`, *optional*, defaults to `\"gpt-3.5-turbo\"`):\n Language model to target. Possible values are:\n aspect (`str`, *optional*, defaults to `\"helpfulness\"`):\n Aspect to target.\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-lib/ultrafeedback-gpt-3.5-turbo-helpfulness\"`):\n Hugging Face repository ID to push the dataset to.\n dataset_num_proc (`int`, *optional*):\n Number of workers to use for dataset processing.\n \"\"\"\n\n model_name: str = field(\n default=\"gpt-3.5-turbo\",\n metadata={\n \"help\": \"Language model to target.\",\n \"choices\": [\n \"alpaca-7b\",\n \"bard\",\n \"falcon-40b-instruct\",\n \"gpt-3.5-turbo\",\n \"gpt-4\",\n \"llama-2-13b-chat\",\n \"llama-2-70b-chat\",\n \"llama-2-7b-chat\",\n \"mpt-30b-chat\",\n \"pythia-12b\",\n \"starchat\",\n \"ultralm-13b\",\n \"ultralm-65b\",\n \"vicuna-33b\",\n \"wizardlm-13b\",\n \"wizardlm-70b\",\n \"wizardlm-7b\",\n ],\n },\n )\n aspect: str = field(\n default=\"helpfulness\",\n metadata={\n \"help\": \"Aspect to target. Possible values are: 'helpfulness' (default), 'honesty', \"\n \"'instruction-following', 'truthfulness'.\",\n \"choices\": [\"helpfulness\", \"honesty\", \"instruction-following\", \"truthfulness\"],\n },\n )\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-lib/ultrafeedback-gpt-3.5-turbo-helpfulness\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of workers to use for dataset processing.\"},\n )\n\n\ndef to_unpaired_preference(example, model_name, aspect):\n prompt = [{\"role\": \"user\", \"content\": example[\"instruction\"]}]\n model_index = example[\"models\"].index(model_name)\n response_content = example[\"completions\"][model_index][\"response\"]\n completion = [{\"role\": \"assistant\", \"content\": response_content}]\n score = int(example[\"completions\"][model_index][\"annotations\"][aspect][\"Rating\"])\n label = score >= 5\n return {\"prompt\": prompt, \"completion\": completion, \"label\": label}\n\n\nmodel_card = ModelCard(\"\"\"\n---\ntags: [trl]\n---\n\n# UltraFeedback GPT-3.5-Turbo Helpfulness Dataset\n\n## Summary\n\nThe UltraFeedback GPT-3.5-Turbo Helpfulness dataset contains processed user-assistant interactions filtered for helpfulness, derived from the [openbmb/UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) dataset. It is designed for fine-tuning and evaluating models in alignment tasks.\n\n## Data Structure\n\n- **Format**: [Conversational](https://huggingface.co/docs/trl/main/dataset_formats#conversational)\n- **Type**: [Unpaired preference](https://huggingface.co/docs/trl/main/dataset_formats#unpaired-preference)\n\nColumn:\n- `\"prompt\"`: The input question or instruction provided to the model.\n- `\"completion\"`: The model's response to the prompt.\n- `\"label\"`: A binary value indicating whether the response is sufficiently helpful.\n\n## Generation script\n\nThe script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/ultrafeedback.py).\n\"\"\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n dataset = load_dataset(\"openbmb/UltraFeedback\", split=\"train\")\n\n dataset = dataset.filter(\n lambda example: script_args.model_name in example[\"models\"],\n batched=False,\n num_proc=script_args.dataset_num_proc,\n )\n dataset = dataset.map(\n to_unpaired_preference,\n remove_columns=[\"source\", \"instruction\", \"models\", \"completions\", \"correct_answers\", \"incorrect_answers\"],\n fn_kwargs={\"model_name\": script_args.model_name, \"aspect\": script_args.aspect},\n num_proc=script_args.dataset_num_proc,\n )\n dataset = dataset.train_test_split(test_size=0.05, seed=42)\n\n if script_args.push_to_hub:\n dataset.push_to_hub(script_args.repo_id)\n model_card.push_to_hub(script_args.repo_id, repo_type=\"dataset\")\n" }, { "path": "examples/notebooks/README.md", "content": "# Notebooks\n\nThis directory contains a collection of Jupyter notebooks that demonstrate how to use the TRL library in different applications.\n\n| Notebook | Description | Open in Colab |\n| --- | --- | --- |\n| [`grpo_trl_lora_qlora.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_trl_lora_qlora.ipynb) | GRPO using QLoRA on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_trl_lora_qlora.ipynb) |\n| [`grpo_functiongemma_browsergym_openenv.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_functiongemma_browsergym_openenv.ipynb) | GRPO on FunctionGemma in the BrowserGym environment | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_functiongemma_browsergym_openenv.ipynb) |\n| [`grpo_agent.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_agent.ipynb) | GRPO for agent training | Not available due to OOM with Colab GPUs |\n| [`grpo_rnj_1_instruct.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_rnj_1_instruct.ipynb) | GRPO rnj-1-instruct with QLoRA using TRL on Colab to add reasoning capabilities | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_rnj_1_instruct.ipynb) |\n| [`sft_ministral3_vl.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_ministral3_vl.ipynb) | Supervised Fine-Tuning (SFT) Ministral 3 with QLoRA using TRL on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_ministral3_vl.ipynb) |\n| [`grpo_ministral3_vl.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_ministral3_vl.ipynb) | GRPO Ministral 3 with QLoRA using TRL on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_ministral3_vl.ipynb) |\n| [`openenv_sudoku_grpo.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/openenv_sudoku_grpo.ipynb) | GRPO to play Sudoku on an OpenEnv environment | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/openenv_sudoku_grpo.ipynb) |\n| [`openenv_wordle_grpo.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/openenv_wordle_grpo.ipynb) | GRPO to play Worldle on an OpenEnv environment | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/openenv_wordle_grpo.ipynb) |\n| [`sft_nemotron_3.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_nemotron_3.ipynb) | SFT with LoRA on NVIDIA Nemotron 3 models | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_nemotron_3.ipynb) |\n| [`sft_trl_lora_qlora.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_trl_lora_qlora.ipynb) | Supervised Fine-Tuning (SFT) using QLoRA on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_trl_lora_qlora.ipynb) |\n| [`sft_qwen_vl.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_qwen_vl.ipynb) | Supervised Fine-Tuning (SFT) Qwen3-VL with QLoRA using TRL on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_qwen_vl.ipynb) |\n| [`sft_tool_calling.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/sft_tool_calling.ipynb) | Teaching tool calling to a model without native tool-calling support using SFT with QLoRA | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_tool_calling.ipynb) |\n| [`grpo_qwen3_vl.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/grpo_qwen3_vl.ipynb) | GRPO Qwen3-VL with QLoRA using TRL on free Colab | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_qwen3_vl.ipynb) |\n" }, { "path": "examples/notebooks/grpo_agent.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"63ceecbc-87ad-4ad3-a317-f49267ffc93b\",\n \"metadata\": {},\n \"source\": [\n \"# Agent Training with GRPO using TRL\\n\",\n \"\\n\",\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\\n\",\n \"\\n\",\n \"\\n\",\n \"With [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl), you can train a language model to act as an **agent**. One that learns to reason, interact with external tools, and improve through reinforcement.\\n\",\n \"\\n\",\n \"- [TRL GitHub Repository](https://github.com/huggingface/trl) โ€” star us to support the project! \\n\",\n \"- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview) \\n\",\n \"- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\\n\",\n \"- [OpenEnv](https://github.com/meta-pytorch/OpenEnv)\\n\",\n \"\\n\",\n \"\\n\",\n \"TRL supports training agents that can use external tools as part of their decision process. \\n\",\n \"In this notebook, the agent has access to the **BioGRID database**, which it can query using **read-only SQL commands** to retrieve biological interaction data. The model learns when and how to use tools based on rewards.\\n\",\n \"\\n\",\n \"We'll fine-tune a model using GRPO (Group Relative Policy Optimization) via TRL. The agent will:\\n\",\n \"\\n\",\n \"1. Generate tool call to query the database if needed.\\n\",\n \"2. Receive the tool response and add it it to the context.\\n\",\n \"3. Learn to improve its tool usage and general capabilities over time through reward signals.\\n\",\n \"\\n\",\n \"## Install dependencies\\n\",\n \"\\n\",\n \"We'll start by installing **TRL**, which automatically includes the main dependencies like **Transformers**. \\n\",\n \"We'll also install **trackio** (for logging and monitoring training runs), **vLLM** (for efficient generation), and **jmespath** (needed for the tools capabilities).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"b4812fbf-3f61-481e-9a64-95277eada9c9\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!pip install -Uq \\\"trl[vllm]\\\" git+https://github.com/huggingface/transformers.git trackio jmespath \"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"ede8e566-a1b5-460f-9fe8-a6010bc56148\",\n \"metadata\": {},\n \"source\": [\n \"### Log in to Hugging Face\\n\",\n \"\\n\",\n \"Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"21756ac0-78b2-495d-8137-28dfa9faae6a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from huggingface_hub import notebook_login\\n\",\n \"\\n\",\n \"notebook_login()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"KVGklspLYlmz\",\n \"metadata\": {},\n \"source\": [\n \"## Create the database for the tool\\n\",\n \"\\n\",\n \"For this example, we will use the [BioGRID database](https://thebiogrid.org/), a curated resource containing **protein, genetic, and chemical interaction data**. We've already compiled and uploaded it to the Hub at [qgallouedec/biogrid](https://huggingface.co/datasets/qgallouedec/biogrid). The dataset is loaded and converted into an sqlite database.\\n\",\n \"\\n\",\n \"> ๐Ÿ’ก We remove spaces in the column names to easen the model work. In real-world deployments, you may keep your original column names and rely on the agent to reason about them. Here, we simplify the schema to make training smoother.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"rRzPMhfXBLkF\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import sqlite3\\n\",\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"# Load dataset\\n\",\n \"biogrid_dataset = load_dataset(\\\"qgallouedec/biogrid\\\", split=\\\"train\\\")\\n\",\n \"df = biogrid_dataset.to_pandas()\\n\",\n \"\\n\",\n \"# Normalize column names: remove spaces, replace with underscores\\n\",\n \"df.columns = [c.replace(\\\" \\\", \\\"_\\\") for c in df.columns]\\n\",\n \"\\n\",\n \"# Save to SQLite\\n\",\n \"conn = sqlite3.connect(\\\"biogrid.db\\\")\\n\",\n \"try:\\n\",\n \" df.to_sql(\\\"interactions\\\", conn, if_exists=\\\"replace\\\", index=False)\\n\",\n \" print(f\\\"biogrid.db created. Rows stored: {len(df)}\\\")\\n\",\n \"finally:\\n\",\n \" conn.close()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"pSSGvLbmZyC2\",\n \"metadata\": {},\n \"source\": [\n \"## Load the QA dataset\\n\",\n \"\\n\",\n \"The training objective is to fine-tune a model to answer gene-related questions. The model should learn to use the database query tool to retrieve factual information when needed.\\n\",\n \"\\n\",\n \"We'll define a formatting function for each sample, adding instructions about the database and how to call it. The model must answer with **yes** or **no**. Let's implement the `format_example` function.\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"asrv7LbaD71C\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import textwrap\\n\",\n \"\\n\",\n \"def format_example(example):\\n\",\n \" question = example[\\\"question\\\"]\\n\",\n \" preamble = textwrap.dedent(\\\"\\\"\\\"\\\\\\n\",\n \" You have access to the BioGRID SQLite database.\\n\",\n \" Use SQL queries to retrieve only the information needed to answer the question.\\n\",\n \"\\n\",\n \" Genes may appear in the database in columns `Alt_IDs_Interactor_A` `Alt_IDs_Interactor_B`, `Aliases_Interactor_A` and `Aliases_Interactor_B`,\\n\",\n \" and each entry can contain multiple gene names or synonyms separated by '|', for example:\\n\",\n \" 'entrez gene/locuslink:JNKK(gene name synonym)|entrez gene/locuslink:MAPKK4(gene name synonym)|...'\\n\",\n \" So a gene like 'JNKK' or 'MAPKK4' may appear inside one of these strings.\\n\",\n \"\\n\",\n \" If the database schema is unclear or you are unsure about column names:\\n\",\n \" - First inspect the schema with `PRAGMA table_info(interactions);`\\n\",\n \" - Or preview a few rows with `SELECT * FROM interactions LIMIT 1;`\\n\",\n \"\\n\",\n \" Otherwise, directly query the required data.\\n\",\n \"\\n\",\n \" Final answer must be enclosed in stars, e.g. *Yes* or *No*.\\n\",\n \" Facts:\\n\",\n \" - The NCBI Taxonomy identifier for humans is taxid:9606.\\n\",\n \" \\\"\\\"\\\")\\n\",\n \" content = f\\\"{preamble}\\\\nQuestion: {question}\\\"\\n\",\n \" prompt = [{\\\"role\\\": \\\"user\\\", \\\"content\\\": content}]\\n\",\n \" return {\\\"prompt\\\": prompt}\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"UMnHXYZla_EO\",\n \"metadata\": {},\n \"source\": [\n \"Now, let's load the database and call the previous function. \\n\",\n \"For simplicity, we will only use questions that start with **โ€œDoes the geneโ€ฆโ€**. \\n\",\n \"In a real use case, the full dataset can be used.\\n\",\n \"\\n\",\n \"The QA dataset is available on the [Hub](https://huggingface.co/datasets/qgallouedec/biogrid_qa).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"jEs12KqwDnVl\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"dataset = load_dataset(\\\"qgallouedec/biogrid_qa\\\", split=\\\"train\\\")\\n\",\n \"dataset = dataset.filter(\\n\",\n \" lambda example: example[\\\"question\\\"].startswith(\\\"Does the gene \\\")\\n\",\n \") # keep only simple questions for example\\n\",\n \"dataset = dataset.map(format_example, remove_columns=[\\\"question\\\"])\\n\",\n \"\\n\",\n \"train_dataset = dataset\\n\",\n \"eval_dataset = None # No eval by default, can be added if needed\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"m4GRjbHycM5L\",\n \"metadata\": {},\n \"source\": [\n \"## Create tool for the agent\\n\",\n \"\\n\",\n \"The `query_biogrid` function is the tool the model will use to query the database and retrieve factual information. \\n\",\n \"Each tool must be a standard Python function with **type-hinted arguments and return types**, and a **Google-style docstring** describing its purpose, parameters, and return value.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"nLMH7hahGTyO\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from contextlib import contextmanager\\n\",\n \"import signal\\n\",\n \"\\n\",\n \"@contextmanager\\n\",\n \"def timeout(seconds):\\n\",\n \" \\\"\\\"\\\"Context manager that raises TimeoutError if execution exceeds time limit.\\\"\\\"\\\"\\n\",\n \"\\n\",\n \" def timeout_handler(signum, frame):\\n\",\n \" raise TimeoutError(f\\\"Operation timed out after {seconds} seconds\\\")\\n\",\n \"\\n\",\n \" signal.signal(signal.SIGALRM, timeout_handler)\\n\",\n \" signal.alarm(seconds)\\n\",\n \" try:\\n\",\n \" yield\\n\",\n \" finally:\\n\",\n \" signal.alarm(0)\\n\",\n \"\\n\",\n \"def query_biogrid(sql_command: str) -> list[tuple]:\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Execute a read-only SQL command on the BioGRID database.\\n\",\n \"\\n\",\n \" BioGRID is a curated biological database that compiles protein, genetic, and chemical interactions from multiple organisms. It provides researchers with experimentally verified interaction data to support studies in systems biology and functional genomics.\\n\",\n \"\\n\",\n \" Args:\\n\",\n \" sql_command: The SQL command to execute.\\n\",\n \"\\n\",\n \" Returns:\\n\",\n \" A list of tuples containing the query results.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" with timeout(5):\\n\",\n \" conn = sqlite3.connect(\\\"file:biogrid.db?mode=ro\\\", uri=True)\\n\",\n \" cursor = conn.cursor()\\n\",\n \" try:\\n\",\n \" cursor.execute(sql_command)\\n\",\n \" results = cursor.fetchall()\\n\",\n \" finally:\\n\",\n \" conn.close()\\n\",\n \" return results\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"GiHtooTwci3B\",\n \"metadata\": {},\n \"source\": [\n \"## Define reward functions\\n\",\n \"\\n\",\n \"To guide the agent during training, we define a few simple reward functions:\\n\",\n \"\\n\",\n \"- **`query_reward`**: evaluates the modelโ€™s query strategy โ€” penalizes more than two queries, penalizes generic database scans, and rewards use of `WHERE` and evidence supporting the final answer.\\n\",\n \"- **`correctness_reward`**: rewards Yes/No predictions that match the expected answer.\\n\",\n \"- **`structure_reward`**: rewards a proper assistant structure (tool call โ†’ response โ†’ optional explanation).\\n\",\n \"\\n\",\n \"Each function returns a list of floats used by the **GRPOTrainer** during optimization. \\n\",\n \"Combined, they encourage effective tool use and factual answers.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"sXyqC6cJGe3L\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import re\\n\",\n \"\\n\",\n \"def query_reward(completions, answer, **kwargs):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Reward query strategy:\\n\",\n \" - Penalize more than 2 queries\\n\",\n \" - Penalize generic queries (LIMIT 1 / PRAGMA)\\n\",\n \" - Reward usage of WHERE\\n\",\n \" - Reward evidence supporting the final answer\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" rewards = []\\n\",\n \"\\n\",\n \" for completion, ans in zip(completions, answer, strict=False):\\n\",\n \" reward = 0.0\\n\",\n \" sql_queries = []\\n\",\n \" tool_results = []\\n\",\n \"\\n\",\n \" # collect all SQL queries and tool results\\n\",\n \" for turn in completion:\\n\",\n \" if turn.get(\\\"tool_calls\\\"):\\n\",\n \" for call in turn[\\\"tool_calls\\\"]:\\n\",\n \" sql = call[\\\"function\\\"][\\\"arguments\\\"].get(\\\"sql_command\\\", \\\"\\\").lower()\\n\",\n \" sql_queries.append(sql)\\n\",\n \" if turn.get(\\\"role\\\") == \\\"tool\\\" and turn.get(\\\"content\\\"):\\n\",\n \" tool_results.append(turn[\\\"content\\\"])\\n\",\n \"\\n\",\n \" # --- penalize too many queries ---\\n\",\n \" if len(sql_queries) > 3:\\n\",\n \" reward -= 1.5\\n\",\n \"\\n\",\n \" # --- check query quality ---\\n\",\n \" where_count = 0\\n\",\n \" for q in sql_queries:\\n\",\n \" if \\\"limit 1\\\" in q:\\n\",\n \" reward -= 1.0\\n\",\n \" if \\\" where \\\" not in q:\\n\",\n \" reward -= 0.5\\n\",\n \" else:\\n\",\n \" where_count += 1\\n\",\n \" reward += min(where_count, 3) * 0.4 # small bonus for WHERE usage\\n\",\n \"\\n\",\n \" # --- evidence check: do queries support the answer? ---\\n\",\n \" combined_results = []\\n\",\n \" error_detected = False\\n\",\n \"\\n\",\n \" for res in tool_results:\\n\",\n \" if isinstance(res, dict) and \\\"error\\\" in res:\\n\",\n \" error_detected = True\\n\",\n \" elif isinstance(res, list):\\n\",\n \" combined_results.extend(res)\\n\",\n \"\\n\",\n \" # if error detected, penalize heavily\\n\",\n \" if error_detected:\\n\",\n \" reward -= 2.0\\n\",\n \" elif len(sql_queries) == 0:\\n\",\n \" reward -= 1.5\\n\",\n \" else:\\n\",\n \" has_hits = len(combined_results) > 0\\n\",\n \" correct_answer = ans.lower()\\n\",\n \" if (has_hits and correct_answer == \\\"yes\\\") or (not has_hits and correct_answer == \\\"no\\\"):\\n\",\n \" reward += 2.0\\n\",\n \" else:\\n\",\n \" reward -= 1.5\\n\",\n \"\\n\",\n \" rewards.append(reward)\\n\",\n \"\\n\",\n \" return rewards\\n\",\n \"\\n\",\n \"\\n\",\n \"def correctness_reward(completions, answer, **kwargs):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Reward Yes/No correctness.\\n\",\n \" Model must provide final answer enclosed in stars โ€” *yes* or *no*.\\n\",\n \" Does not reward informal yes/no buried in text.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" rewards = []\\n\",\n \" for completion, ans in zip(completions, answer, strict=False):\\n\",\n \" raw = completion[-1][\\\"content\\\"].lower()\\n\",\n \"\\n\",\n \" # detect form *yes* or *no*\\n\",\n \" match = re.search(r\\\"\\\\*(yes|no)\\\\*\\\", raw)\\n\",\n \" guess = match.group(1) if match else None\\n\",\n \"\\n\",\n \" reward = 0.0\\n\",\n \"\\n\",\n \" if guess is None:\\n\",\n \" reward -= 0.5 # invalid format\\n\",\n \" elif guess == ans.lower():\\n\",\n \" reward += 0.6 # correct under required format\\n\",\n \" else:\\n\",\n \" reward -= 1.0 # wrong answer\\n\",\n \"\\n\",\n \" rewards.append(reward)\\n\",\n \"\\n\",\n \" return rewards\\n\",\n \"\\n\",\n \"\\n\",\n \"def structure_reward(completions, **kwargs):\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Reward proper assistant structure.\\n\",\n \" Encourages a logical sequence: tool call + response + optional extra content.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" rewards = []\\n\",\n \"\\n\",\n \" for completion in completions:\\n\",\n \" has_call = False\\n\",\n \" has_response = False\\n\",\n \" has_other = False\\n\",\n \"\\n\",\n \" for turn in completion:\\n\",\n \" role = turn.get(\\\"role\\\")\\n\",\n \" if role == \\\"assistant\\\" and turn.get(\\\"tool_calls\\\"):\\n\",\n \" has_call = True\\n\",\n \" elif role == \\\"tool\\\":\\n\",\n \" has_response = True\\n\",\n \" else:\\n\",\n \" content = turn.get(\\\"content\\\")\\n\",\n \" if content and content.strip() not in [\\\"\\\", \\\"\\\"]:\\n\",\n \" has_other = True\\n\",\n \"\\n\",\n \" # Reward sequences\\n\",\n \" if has_call and has_response:\\n\",\n \" if has_other:\\n\",\n \" reward = 0.1\\n\",\n \" else:\\n\",\n \" reward = 0.05 # still positive even without extra text\\n\",\n \" elif has_call and not has_response:\\n\",\n \" reward = -0.15\\n\",\n \" else:\\n\",\n \" reward = 0.0 # neutral if no call\\n\",\n \"\\n\",\n \" rewards.append(reward)\\n\",\n \"\\n\",\n \" return rewards\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"zcgkrKtTb4T9\",\n \"metadata\": {},\n \"source\": [\n \"## Set GRPO Config\\n\",\n \"\\n\",\n \"Next, we define the **GRPOConfig**, which controls the main training parameters. \\n\",\n \"This configuration specifies how the model interacts with **vLLM**, manages memory, and logs results.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"t4ifJsNLElIN\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOConfig\\n\",\n \"\\n\",\n \"output_dir = \\\"grpo_biogrid_qwen_3g-1.7b\\\"\\n\",\n \"\\n\",\n \"grpo_config = GRPOConfig(\\n\",\n \" # Training schedule / optimization\\n\",\n \" max_steps=400, # Max number of training steps\\n\",\n \" chat_template_kwargs = {\\\"enable_thinking\\\": False}, # Disable thinking to reduce token generation\\n\",\n \"\\n\",\n \" # GRPO configuration\\n\",\n \" max_completion_length = 1024, # Maximum tokens generated per model response\\n\",\n \"\\n\",\n \" # vLLM configuration\\n\",\n \" use_vllm = True, # Enable vLLM for faster inference during rollouts\\n\",\n \" vllm_mode = \\\"colocate\\\", # Run vLLM in colocate mode (same process as training)\\n\",\n \" vllm_enable_sleep_mode=False,\\n\",\n \"\\n\",\n \" # Logging / reporting\\n\",\n \" output_dir = output_dir, # Directory for checkpoints and logs\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool (integrates with HF Spaces)\\n\",\n \" trackio_space_id = output_dir, # HF Space where experiment tracking will be saved\\n\",\n \" save_steps = 10, # Interval for saving checkpoints\\n\",\n \" log_completions = True,\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub = True, # Set True to automatically push model to Hugging Face Hub\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"34I-Q2MJuf42\",\n \"metadata\": {},\n \"source\": [\n \"## Create `GRPOTrainer` and Start Training\\n\",\n \"\\n\",\n \"Next, we initialize the **`GRPOTrainer`**, which handles the full reinforcement learning loop.\\n\",\n \"\\n\",\n \"It receives the model name, reward functions, tool(s), and dataset defined earlier. \\n\",\n \"\\n\",\n \"Finally, we call `trainer.train()` to begin fine-tuning, allowing the model to learn how to query the database effectively through iterative feedback.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"IysntAUOFvRn\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOTrainer\\n\",\n \"\\n\",\n \"model_name=\\\"Qwen/Qwen3-1.7B\\\"\\n\",\n \"\\n\",\n \"trainer = GRPOTrainer(\\n\",\n \" model=model_name,\\n\",\n \" train_dataset=train_dataset,\\n\",\n \" eval_dataset=eval_dataset,\\n\",\n \" tools=[query_biogrid],\\n\",\n \" reward_funcs=[correctness_reward, structure_reward, query_reward],\\n\",\n \" args=grpo_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"r_qJ5UwLuzCG\",\n \"metadata\": {},\n \"source\": [\n \"Show memory stats before training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"DusT8JUaGmA6\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"gpu_stats = torch.cuda.get_device_properties(0)\\n\",\n \"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\\n\",\n \"\\n\",\n \"print(f\\\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\\\")\\n\",\n \"print(f\\\"{start_gpu_memory} GB of memory reserved.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"OTPkiz3fu0lp\",\n \"metadata\": {},\n \"source\": [\n \"And train!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"NwI3buPOFMFk\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"ITnLBLcTu2-p\",\n \"metadata\": {},\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"ftek6m4-GncK\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"O6LAwznKu7mc\",\n \"metadata\": {},\n \"source\": [\n \"Let's save the trained model.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"idVgnNS1MWPr\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"707318cb\",\n \"metadata\": {},\n \"source\": [\n \"## Load the fine-tuned model and run inference using `smolagents`\\n\",\n \"\\n\",\n \"After fine-tuning the model with **GRPO (TRL)** for tool calling, we can test it at inference time using **`smolagents`**, a lightweight library for running multi-step agents.\\n\",\n \"\\n\",\n \"`smolagents` handles the agent loop for us:\\n\",\n \"- Detecting tool calls generated by the model\\n\",\n \"- Executing the corresponding tools (e.g. database queries)\\n\",\n \"- Feeding the results back to the model until a final answer is produced\\n\",\n \"\\n\",\n \"> **Note** \\n\",\n \"> Using an agent framework is optional. The fine-tuned model can also be used directly with `transformers` by manually controlling the inference loop and executing the tools outside the model.\\n\",\n \"> Agent frameworks are especially useful when the number of steps or tool calls is not fixed.\\n\",\n \"\\n\",\n \"We start by installing the required package:\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"aab7fd5c\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!pip install git+https://github.com/huggingface/smolagents.git\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"24453572\",\n \"metadata\": {},\n \"source\": [\n \"We will use the `CodeAgent` class from `smolagents` to instantiate our agent. \\n\",\n \"First, we need to define the tool the agent can use. This is done using the `@tool` decorator.\\n\",\n \"\\n\",\n \"As shown below, the tool definition is **exactly the same** as the one used during GRPO training with TRL. This consistency is important: the model was trained to emit calls following this schema, and at inference time the agent simply executes the corresponding Python function.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"adcbbafa\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from smolagents import tool\\n\",\n \"\\n\",\n \"@tool\\n\",\n \"def query_biogrid(sql_command: str) -> list[tuple]:\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" Execute a read-only SQL query on the BioGRID database.\\n\",\n \"\\n\",\n \" BioGRID is a curated biological database that compiles protein, genetic,\\n\",\n \" and chemical interactions from multiple organisms.\\n\",\n \"\\n\",\n \" Args:\\n\",\n \" sql_command: A read-only SQL query to execute.\\n\",\n \"\\n\",\n \" Returns:\\n\",\n \" A list of tuples containing the query results.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" with timeout(5):\\n\",\n \" conn = sqlite3.connect(\\n\",\n \" \\\"file:biogrid.db?mode=ro\\\",\\n\",\n \" uri=True,\\n\",\n \" )\\n\",\n \" cursor = conn.cursor()\\n\",\n \" try:\\n\",\n \" cursor.execute(sql_command)\\n\",\n \" results = cursor.fetchall()\\n\",\n \" finally:\\n\",\n \" conn.close()\\n\",\n \"\\n\",\n \" return results\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"59721ad2\",\n \"metadata\": {},\n \"source\": [\n \"Now we can instantiate the agent using our fine-tuned model and the database tool defined above.\\n\",\n \"We wrap the model with `TransformersModel` and pass both the model and the tool when creating the `CodeAgent`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"e9ed8d00\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from smolagents import TransformersModel, CodeAgent\\n\",\n \"\\n\",\n \"model = TransformersModel(model_id=\\\"sergiopaniego/grpo_biogrid_qwen_3g-1.7b\\\", apply_chat_template_kwargs={\\\"enable_thinking\\\": False})\\n\",\n \"\\n\",\n \"# Create an agent with query_biogrid as tool\\n\",\n \"agent = CodeAgent(tools=[query_biogrid], model=model)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"57ba9462\",\n \"metadata\": {},\n \"source\": [\n \"Finally, we run the agent by passing the full prompt (including the instruction preamble and the question), exactly as it was used during training. This ensures the agent operates under the same context and assumptions learned with GRPO, allowing it to correctly decide when to query the database and how to format the final answer.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"23a3cdf4\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"result = agent.run(train_dataset[0]['prompt'][0]['content'])\\n\",\n \"print(result)\"\n ]\n }\n ],\n \"metadata\": {\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "examples/notebooks/grpo_functiongemma_browsergym_openenv.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"lSR2nwdJg962\"\n },\n \"source\": [\n \"# Fine-Tune FunctionGemma using Hugging Face TRL and OpenEnv\\n\",\n \"\\n\",\n \"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_functiongemma_browsergym_openenv.ipynb)\\n\",\n \"\\n\",\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\\n\",\n \"\\n\",\n \"This guide describes the process of fine-tuning [FunctionGemma](https://huggingface.co/google/functiongemma-270m-it) by Google DeepMind in the [BrowserGym](https://meta-pytorch.org/OpenEnv/environments/browsergym/) environment provided by OpenEnv, using Hugging Face TRL. The steps covered include:\\n\",\n \"\\n\",\n \"* What is GRPO and OpenEnv\\n\",\n \"* Setup dependencies for training\\n\",\n \"* Initialize the OpenEnv's BrowserGym environment\\n\",\n \"* Create rollout function with helpers\\n\",\n \"* Define the reward functions\\n\",\n \"* Load the custom dataset\\n\",\n \"* Fine tune using TRL and the GRPOTrainer\\n\",\n \"* Load the fine-tuned model and run inference\\n\",\n \"\\n\",\n \"> Note: The guide is designed to run on Google Colaboratory with access to an NVIDIA A100 GPU (40GB) using FunctionGemma. The workflow can be adapted to other GPU configurations, models, or environments.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"duXYuR6Cu_na\"\n },\n \"source\": [\n \"## What is GRPO and OpenEnv\\n\",\n \"\\n\",\n \"Group Relative Policy Optimization ([GRPO](https://huggingface.co/papers/2402.03300)) is a post-training method widely used for efficiently fine-tuning large language models. GRPO leverages reward functions to guide learning, enabling models to optimize task-specific behaviors without retraining the entire network.\\n\",\n \"\\n\",\n \"[OpenEnv](https://meta-pytorch.org/OpenEnv) provides a standard interface for interacting with agentic execution environments using simple Gymnasium-style APIs, such as `step()`, `reset()`, and `state()`. These APIs facilitate reinforcement learning training loops by allowing models to interact with environments in a structured manner. OpenEnv also offers tools for environment creators to build isolated, secure, and deployable environments that can be shared via common protocols like HTTP or packaged in Docker.\\n\",\n \"\\n\",\n \"The combination of GRPO and OpenEnv enables efficient fine-tuning of models in controlled, interactive tasks while minimizing resource requirements.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"cpSAQkzKmv50\"\n },\n \"source\": [\n \"## Setup dependencies for training\\n\",\n \"\\n\",\n \"Install the required libraries, including Hugging Face TRL for fine-tuning and OpenEnv for reinforcement learning environments.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"c-2drnj5BP56\"\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -Uq trl[vllm] git+https://huggingface.co/spaces/openenv/browsergym_env liger-kernel trackio\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Inxeq6ZGpRno\"\n },\n \"source\": [\n \"A valid Hugging Face token is required to save the fine-tuned model. In Google Colab, the token can be securely accessed through Colab secrets. Otherwise, it can be provided directly in the login method. Ensure the token has write permissions to allow uploading the model to the Hugging Face Hub during training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"C4q5UVu3BP57\"\n },\n \"outputs\": [],\n \"source\": [\n \"from google.colab import userdata\\n\",\n \"from huggingface_hub import login\\n\",\n \"\\n\",\n \"# Login into Hugging Face Hub\\n\",\n \"hf_token = userdata.get('HF_TOKEN') # If you are running inside a Google Colab\\n\",\n \"login(hf_token)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"O3kr38TGm_hb\"\n },\n \"source\": [\n \"## Initialize the OpenEnv's BrowserGym environment\\n\",\n \"\\n\",\n \"External environments can guide the fine-tuning of LLMs for function calling by providing interactive feedback that enhances performance on task-specific behaviors.\\n\",\n \"\\n\",\n \"[BrowserGym](https://meta-pytorch.org/OpenEnv/environments/browsergym/) is a unified framework for web-based agent tasks, offering multiple benchmarks through a Gymnasium-compatible API. It enables training on simple synthetic tasks with [MiniWoB++](https://github.com/Farama-Foundation/miniwob-plusplus) and evaluation on more complex, realistic tasks with [WebArena](https://github.com/web-arena-x/webarena), [VisualWebArena](https://github.com/web-arena-x/visualwebarena), or [WorkArena](https://github.com/ServiceNow/WorkArena). This setup supports iterative training and assessment of web agents without requiring extensive infrastructure.\\n\",\n \"\\n\",\n \"BrowserGym supports both LLM and VLM training by providing visual information, including screenshots and DOM data, which can be utilized depending on the model type. This guide focuses on a simple web-based task called *\\\"click-test\\\"*, which is part of the MiniWoB++ benchmark of synthetic web tasks. Environments can be run locally, in Docker containers, or accessed remotely via the Hugging Face Hub. For this example, the remote environment [openenv/browsergym_env](https://huggingface.co/spaces/openenv/browsergym_env) will be used.\\n\",\n \"\\n\",\n \"> Note: Hosted environments on the Hub currently have limited concurrency. For higher reliability or parallel runs, duplicating the Space to your own account is strongly recommended.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"clDs-WQlBP57\"\n },\n \"outputs\": [],\n \"source\": [\n \"from browsergym_env import BrowserGymEnv\\n\",\n \"space_url = \\\"https://openenv-browsergym-env.hf.space\\\"\\n\",\n \"\\n\",\n \"client = BrowserGymEnv(base_url=space_url)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"EqfDavDQnD_5\"\n },\n \"source\": [\n \"## Create rollout function with helpers\\n\",\n \"\\n\",\n \"The rollout function defines how the agent interacts with the environment during GRPO training. It generates model outputs, collects feedback in the form of rewards, and returns the information required for optimization.\\n\",\n \"\\n\",\n \"In this setup:\\n\",\n \"- The function is invoked automatically by the GRPOTrainer (introduced later), which orchestrates the training loop and handles policy updates.\\n\",\n \"- It uses the trainer's `generate_rollout_completions()` method for efficient output generation. This leverages vLLM, a high-performance inference engine for large language models, and is integrated within TRL to streamline rollout generation and reward collection during fine-tuning.\\n\",\n \"- Each rollout represents a complete interaction loop, where the model acts, receives feedback from the environment, and updates based on reward signals.\\n\",\n \"\\n\",\n \"Rewards capture various aspects of the agent's performance. Helper functions, such as `rollout_once`, manage individual episodes, keeping the main `rollout_func` clean, modular, and reusable.\\n\",\n \"\\n\",\n \"This modular structure allows GRPO to efficiently sample, evaluate, and refine the model's behavior through reinforcement learning.\\n\",\n \"\\n\",\n \"Before executing rollouts, a `system prompt` is defined to instruct the model on how to interact with the environment. This prompt specifies the available BrowserGym actions (such as `click`, `fill`, `send_keys`, and `scroll`), describes the page structure, and enforces that the model responds with exactly one action per step. It ensures consistent and structured interactions, guiding the model to complete tasks effectively without providing extra explanations or multiple actions.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"ItCXS6H0BP58\"\n },\n \"outputs\": [],\n \"source\": [\n \"# @title System prompt (click to expand)\\n\",\n \"SYSTEM_PROMPT = \\\"\\\"\\\"You control a web browser through BrowserGym actions.\\n\",\n \"You must complete the given web task by interacting with the page.\\n\",\n \"\\n\",\n \"Available actions:\\n\",\n \"- noop() - Do nothing\\n\",\n \"- click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"- fill(bid, text) - Fill input field with text\\n\",\n \"- send_keys(text) - Send keyboard input\\n\",\n \"- scroll(direction) - Scroll up/down\\n\",\n \"\\n\",\n \"The page structure shows elements as: [bid] element_type 'element_text'\\n\",\n \"For example: [13] button 'Click Me!' means bid='13'\\n\",\n \"\\n\",\n \"Reply with exactly ONE action on a single line, e.g.:\\n\",\n \"click('13')\\n\",\n \"fill('42', 'hello world')\\n\",\n \"noop()\\n\",\n \"\\n\",\n \"Do not include explanations or multiple actions.\\\"\\\"\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Vi1rFey39GUl\"\n },\n \"source\": [\n \"The `rollout_func` orchestrates the interaction between the model and the remote BrowserGym environment. For each prompt in the batch, it executes a complete episode using the `rollout_once` function, collecting model outputs and rewards for GRPO optimization.\\n\",\n \"\\n\",\n \"The parameter `max_steps` defines the maximum number of steps the model can take within a single episode. This limits the length of the interaction loop, ensuring that episodes terminate even if the task is not completed, and helps maintain efficient training.\\n\",\n \"\\n\",\n \"During each episode, the function tracks prompt and completion IDs, log probabilities, and both step-wise and final rewards, returning them in a structured format for the trainer to perform policy updates.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"CgHd5CFBBP58\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOTrainer\\n\",\n \"\\n\",\n \"max_steps=10\\n\",\n \"\\n\",\n \"def rollout_func(prompts: list[str], trainer: GRPOTrainer) -> dict[str, list]:\\n\",\n \" episode_prompt_ids: list[list[int]] = []\\n\",\n \" episode_completion_ids: list[list[int]] = []\\n\",\n \" episode_logprobs: list[list[float]] = []\\n\",\n \" completion_rewards: list[float] = []\\n\",\n \"\\n\",\n \" print(f\\\"\\\\n[DEBUG] rollout_func called with {len(prompts)} prompts (LLM mode, text-only)\\\")\\n\",\n \"\\n\",\n \" for i, prompt_text in enumerate(prompts):\\n\",\n \" print(f\\\"[DEBUG] Processing prompt {i + 1}/{len(prompts)}\\\")\\n\",\n \" episode = rollout_once(\\n\",\n \" trainer=trainer,\\n\",\n \" env=client,\\n\",\n \" tokenizer=trainer.processing_class,\\n\",\n \" dataset_prompt=prompt_text,\\n\",\n \" max_steps=max_steps,\\n\",\n \" )\\n\",\n \" episode_prompt_ids.append(episode[\\\"prompt_ids\\\"])\\n\",\n \" episode_completion_ids.append(episode[\\\"completion_ids\\\"])\\n\",\n \" episode_logprobs.append(episode[\\\"logprobs\\\"])\\n\",\n \" completion_rewards.append(episode[\\\"completion_reward\\\"])\\n\",\n \"\\n\",\n \" return {\\n\",\n \" \\\"prompt_ids\\\": episode_prompt_ids,\\n\",\n \" \\\"completion_ids\\\": episode_completion_ids,\\n\",\n \" \\\"logprobs\\\": episode_logprobs,\\n\",\n \" \\\"completion_reward\\\": completion_rewards,\\n\",\n \" }\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"ioUHdIxr9ZQO\"\n },\n \"source\": [\n \"### Define `rollout_once`\\n\",\n \"\\n\",\n \"The `rollout_once` function runs one complete interaction loop between the model and the BrowserGym environment using the trainer's generation method. \\n\",\n \"It executes a single episode, from generating an action to receiving feedback and computing rewards.\\n\",\n \"\\n\",\n \"Here's the step-by-step breakdown:\\n\",\n \"\\n\",\n \"1. Environment reset: Start a new BrowserGym session and initialize the observation.\\n\",\n \"2. Prompt construction: Combine the system prompt, environment observation (text-only via the accessibility tree), and any relevant errors or state information to form the model input.\\n\",\n \"3. Generation: Use `trl.experimental.openenv.generate_rollout_completions()` to produce the model's action efficiently with vLLM.\\n\",\n \"4. Action parsing and execution: Interpret the model's output and execute the corresponding BrowserGym action (e.g., `click`, `fill`, `scroll`).\\n\",\n \"5. Reward calculation: Track step-wise rewards provided by the environment and compute completion rewards based on task success or failure.\\n\",\n \"6. Return structured rollout data: Includes prompt/completion IDs, log probabilities, step rewards, and the final reward for the episode.\\n\",\n \"\\n\",\n \"This modular design allows each episode to be processed independently while providing rich feedback for the GRPO training loop, supporting both task completion and intermediate reward shaping.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"y8Ml47SYBP58\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl.experimental.openenv import generate_rollout_completions\\n\",\n \"from browsergym_env import BrowserGymAction\\n\",\n \"from transformers import AutoTokenizer\\n\",\n \"\\n\",\n \"def rollout_once(\\n\",\n \" trainer: GRPOTrainer,\\n\",\n \" env: BrowserGymEnv,\\n\",\n \" tokenizer: AutoTokenizer,\\n\",\n \" dataset_prompt: str,\\n\",\n \" max_steps: int,\\n\",\n \") -> dict[str, list]:\\n\",\n \" \\\"\\\"\\\"Run one episode and collect training data (text-only, no screenshots).\\\"\\\"\\\"\\n\",\n \" result = env.reset()\\n\",\n \" observation = result.observation\\n\",\n \"\\n\",\n \" prompt_ids: list[int] = []\\n\",\n \" completion_ids: list[int] = []\\n\",\n \" logprobs: list[float] = []\\n\",\n \" step_rewards: list[float] = []\\n\",\n \" completion_rewards: list[float] = []\\n\",\n \"\\n\",\n \" for step_num in range(max_steps):\\n\",\n \" if result.done:\\n\",\n \" break\\n\",\n \"\\n\",\n \" # Create prompt from observation (text-only using accessibility tree)\\n\",\n \" goal = observation.goal or dataset_prompt\\n\",\n \" axtree = observation.axtree_txt or \\\"\\\"\\n\",\n \" error = observation.error if observation.last_action_error else \\\"\\\"\\n\",\n \"\\n\",\n \" user_prompt = make_user_prompt(goal, step_num, axtree, error)\\n\",\n \" messages = [\\n\",\n \" {\\\"role\\\": \\\"system\\\", \\\"content\\\": SYSTEM_PROMPT},\\n\",\n \" {\\\"role\\\": \\\"user\\\", \\\"content\\\": user_prompt},\\n\",\n \" ]\\n\",\n \" prompt_text = tokenizer.apply_chat_template(\\n\",\n \" messages,\\n\",\n \" add_generation_prompt=True,\\n\",\n \" tokenize=False,\\n\",\n \" )\\n\",\n \"\\n\",\n \" # Generate action with vLLM\\n\",\n \" rollout_outputs = generate_rollout_completions(trainer, [prompt_text])[0]\\n\",\n \" prompt_ids.extend(rollout_outputs[\\\"prompt_ids\\\"])\\n\",\n \" completion_ids.extend(rollout_outputs[\\\"completion_ids\\\"])\\n\",\n \" logprobs.extend(rollout_outputs[\\\"logprobs\\\"])\\n\",\n \"\\n\",\n \" completion_text = rollout_outputs.get(\\\"text\\\") or tokenizer.decode(\\n\",\n \" rollout_outputs[\\\"completion_ids\\\"], skip_special_tokens=True\\n\",\n \" )\\n\",\n \"\\n\",\n \" # Parse and execute action\\n\",\n \" action_str = parse_action(completion_text)\\n\",\n \"\\n\",\n \" print(f\\\"Step {step_num + 1}: {action_str}\\\")\\n\",\n \"\\n\",\n \" # Take action in environment\\n\",\n \" result = env.step(BrowserGymAction(action_str=action_str))\\n\",\n \" observation = result.observation\\n\",\n \"\\n\",\n \" # Track rewards\\n\",\n \" step_reward = float(result.reward or 0.0)\\n\",\n \" step_rewards.append(step_reward)\\n\",\n \"\\n\",\n \" # Reward shaping: success is most important\\n\",\n \" if result.done and step_reward > 0:\\n\",\n \" completion_rewards.append(1.0) # Task completed successfully\\n\",\n \" elif result.done and step_reward == 0:\\n\",\n \" completion_rewards.append(0.0) # Task failed\\n\",\n \" else:\\n\",\n \" completion_rewards.append(step_reward) # Intermediate reward\\n\",\n \"\\n\",\n \" # Final reward is based on task completion\\n\",\n \" final_reward = completion_rewards[-1] if completion_rewards else 0.0\\n\",\n \"\\n\",\n \" return {\\n\",\n \" \\\"prompt_ids\\\": prompt_ids,\\n\",\n \" \\\"completion_ids\\\": completion_ids,\\n\",\n \" \\\"logprobs\\\": logprobs,\\n\",\n \" \\\"step_rewards\\\": step_rewards,\\n\",\n \" \\\"completion_reward\\\": final_reward,\\n\",\n \" }\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"MDJKMQ__8qzj\"\n },\n \"source\": [\n \"### Helper functions\\n\",\n \"\\n\",\n \"Supporting utilities used in `rollout_once`:\\n\",\n \"\\n\",\n \"- `make_user_prompt`: builds the user prompt combining the base text and previous game messages.\\n\",\n \"- `parse_action`: parses BrowserGym action from model response\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"GG4ba41PBP58\"\n },\n \"outputs\": [],\n \"source\": [\n \"# @title Helpers (click to expand)\\n\",\n \"def make_user_prompt(goal: str, step_num: int, axtree: str, error: str = \\\"\\\") -> str:\\n\",\n \" \\\"\\\"\\\"Create user prompt from observation.\\\"\\\"\\\"\\n\",\n \" prompt_parts = [f\\\"Step {step_num + 1}\\\"]\\n\",\n \"\\n\",\n \" if goal:\\n\",\n \" prompt_parts.append(f\\\"Goal: {goal}\\\")\\n\",\n \"\\n\",\n \" if error:\\n\",\n \" prompt_parts.append(f\\\"Previous action error: {error}\\\")\\n\",\n \"\\n\",\n \" # Include accessibility tree (truncated for context)\\n\",\n \" if axtree:\\n\",\n \" max_len = 2000\\n\",\n \" axtree_truncated = axtree[:max_len] + \\\"...\\\" if len(axtree) > max_len else axtree\\n\",\n \" prompt_parts.append(f\\\"Page structure:\\\\n{axtree_truncated}\\\")\\n\",\n \"\\n\",\n \" prompt_parts.append(\\\"What action do you take?\\\")\\n\",\n \"\\n\",\n \" return \\\"\\\\n\\\\n\\\".join(prompt_parts)\\n\",\n \"\\n\",\n \"\\n\",\n \"def parse_action(response_text: str) -> str:\\n\",\n \" \\\"\\\"\\\"Parse BrowserGym action from model response.\\\"\\\"\\\"\\n\",\n \" # Extract first line that looks like an action\\n\",\n \" for line in response_text.strip().split(\\\"\\\\n\\\"):\\n\",\n \" line = line.strip()\\n\",\n \" if \\\"(\\\" in line and \\\")\\\" in line:\\n\",\n \" return line\\n\",\n \"\\n\",\n \" # Fallback to noop if no valid action found\\n\",\n \" return \\\"noop()\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Oek3JhcWnKhw\"\n },\n \"source\": [\n \"## Define the reward functions\\n\",\n \"\\n\",\n \"Reward functions quantify the model's performance in the environment and guide the GRPO optimization process.\\n\",\n \"\\n\",\n \"In this setup, the `reward_completion` function assigns rewards based on task completion. It extracts the final reward for each episode, which indicates whether the agent successfully completed the task. If no reward information is available, it defaults to zero.\\n\",\n \"\\n\",\n \"This modular approach allows additional reward functions to be added easily, enabling more granular feedback such as intermediate progress, efficiency, or correctness of actions, depending on the task requirements.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"WxkXaz5aBP59\"\n },\n \"outputs\": [],\n \"source\": [\n \"def reward_completion(completions: list[str], **kwargs) -> list[float]:\\n\",\n \" \\\"\\\"\\\"Reward for task completion.\\\"\\\"\\\"\\n\",\n \" rewards = kwargs.get(\\\"completion_reward\\\") if kwargs else None\\n\",\n \" if rewards is None:\\n\",\n \" return [0.0 for _ in completions]\\n\",\n \" return [float(r) for r in rewards]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"66ZsrLplm07U\"\n },\n \"source\": [\n \"## Load the custom dataset\\n\",\n \"\\n\",\n \"The dataset is constructed with repeated prompts to control the total number of training episodes.\\n\",\n \"\\n\",\n \"Each entry in the dataset triggers a single rollout episode during training. The `dataset_prompt` provides the initial instruction to the model at the start of each episode, ensuring consistent guidance for task execution.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"UX6jUjxaBP59\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import Dataset\\n\",\n \"\\n\",\n \"dataset_prompt = \\\"Complete the web task successfully.\\\"\\n\",\n \"dataset_size = 1000\\n\",\n \"\\n\",\n \"dataset = Dataset.from_dict({\\\"prompt\\\": [dataset_prompt] * dataset_size})\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"-mvka-96m3I7\"\n },\n \"source\": [\n \"## Fine-tune using TRL and the GRPOTrainer\\n\",\n \"\\n\",\n \"The next step is to define the GRPOConfig, which sets all key training parameters.\\n\",\n \"\\n\",\n \"This configuration determines how the model interacts with vLLM, handles memory and computation, and records training metrics and logs for monitoring the fine-tuning process.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"TZ34a1h-BP59\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOConfig\\n\",\n \"output_dir = \\\"browsergym-grpo-functiongemma-270m-it\\\"\\n\",\n \"\\n\",\n \"grpo_config = GRPOConfig(\\n\",\n \" # num_train_epochs=1, # Number of times to iterate over the full dataset (use for full training runs)\\n\",\n \" max_steps=100, # Number of dataset passes (for shorter runs/testing). For full trainings, use `num_train_epochs` instead\\n\",\n \" learning_rate=5e-6, # Learning rate for the optimizer\\n\",\n \" warmup_steps=10, # Number of steps to linearly increase learning rate at the start of training\\n\",\n \"\\n\",\n \" per_device_train_batch_size=1, # Number of samples per device per step\\n\",\n \" num_generations=4, # Number of completions to generate per prompt\\n\",\n \" generation_batch_size=4, # Batch size used during generation (must be divisible by num_generations)\\n\",\n \" max_completion_length=32, # Maximum length of generated completions\\n\",\n \"\\n\",\n \" use_vllm=True, # Use vLLM engine for fast inference\\n\",\n \" vllm_mode=\\\"colocate\\\", # vLLM mode: \\\"colocate\\\" runs generation on the same GPU as training\\n\",\n \" vllm_gpu_memory_utilization=0.1, # Fraction of GPU memory allocated to vLLM\\n\",\n \"\\n\",\n \" output_dir=str(output_dir), # Directory where checkpoints, logs, and outputs will be saved\\n\",\n \" logging_steps=1, # Log metrics every N steps\\n\",\n \" report_to=\\\"trackio\\\", # Logging/reporting platform (e.g., \\\"trackio\\\")\\n\",\n \" trackio_space_id=output_dir, # HF Space where the experiment tracking will be saved\\n\",\n \" push_to_hub=True, # Optionally push trained model to Hugging Face Hub\\n\",\n \"\\n\",\n \" use_liger_kernel=True, # Enable Liger kernel optimizations for faster training\\n\",\n \")\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"a1taGmD--0Y4\"\n },\n \"source\": [\n \"The next step is to initialize the GRPOTrainer, which manages the complete reinforcement learning loop.\\n\",\n \"\\n\",\n \"It receives the model name, reward functions, rollout function, and dataset defined earlier. From the model name, the trainer automatically initializes the model and tokenizer. It then coordinates interactions between the model and the environment, applies the defined reward signals, and updates the policy during training.\\n\",\n \"\\n\",\n \"Finally, calling `trainer.train()` starts the fine-tuning process, enabling the model to progressively improve its performance through iterative interaction and reinforcement learning.\\n\",\n \"\\n\",\n \"> Note: The training pipeline uses approximately 10.6 GB of GPU VRAM and can be adapted to different hardware configurations.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"En43o4NZBP59\"\n },\n \"outputs\": [],\n \"source\": [\n \"model_name = \\\"google/functiongemma-270m-it\\\"\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"047d386e54704add95edd4beace781d7\"\n ]\n },\n \"id\": \"k8-SvqJcBP59\",\n \"outputId\": \"6a4d9276-fc91-4217-d3a2-51a18d222338\"\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/tmp/ipython-input-3830121904.py:1: UserWarning: You are importing from 'rollout_func', which is an experimental feature. This API may change or be removed at any time without prior notice. Silence this warning by setting environment variable TRL_EXPERIMENTAL_SILENCE=1.\\n\",\n \" trainer = GRPOTrainer(\\n\",\n \"The model is already on multiple devices. Skipping the move to device specified in `args`.\\n\",\n \"`torch_dtype` is deprecated! Use `dtype` instead!\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"047d386e54704add95edd4beace781d7\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Created new run: sergiopaniego-1765969078\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: Click 'click(bid) - Click element with BrowserGym ID (the number in brackets\\n\",\n \"Step 8: I will use the action `click()` to click the button.\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: Click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: Clicks ('13')\\n\",\n \"Step 4: I will click 'Click Me!' using action 'click(bid)' on page 'Click Test Task' using a bid of '13'.\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: noop()\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: I will use the 'click(bid)' action.\\n\",\n \"Step 2: mouse_click(bid)\\n\",\n \"Step 3: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 4: Add action 'click(bid)' to Step 4.\\n\",\n \"Step 5: Click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: Click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: Click('13')\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: noop()\\n\"\n ]\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"WARNING:liger_kernel.transformers.model.gemma3:It is strongly recommended to train Gemma3 models with the `eager` attention implementation instead of `sdpa`. Use `eager` with `AutoModelForCausalLM.from_pretrained('', attn_implementation='eager')`.\\n\",\n \"/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:282: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.\\n\",\n \" warnings.warn(\\n\",\n \"/usr/local/lib/python3.12/dist-packages/torch/_inductor/lowering.py:7095: UserWarning: \\n\",\n \"Online softmax is disabled on the fly since Inductor decides to\\n\",\n \"split the reduction. Cut an issue to PyTorch if this is an\\n\",\n \"important use case and you want to speed it up with online\\n\",\n \"softmax.\\n\",\n \"\\n\",\n \" warnings.warn(\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" [100/100 35:02, Epoch 0/1]\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \"
StepTraining Loss
10.000000
20.000000
30.000000
40.000000
50.000000
60.000000
70.000000
80.000000
9-0.877900
101965.894400
11-0.830900
1210.616100
130.000000
140.000000
150.000000
160.000000
172.320100
181.887500
19-0.691600
20-0.764400
210.000000
220.000000
230.000000
240.000000
250.000000
260.000000
270.000000
280.000000
290.000000
300.000000
310.000000
320.000000
330.000000
340.000000
350.000000
360.000000
370.000000
380.000000
390.000000
400.000000
410.000000
420.000000
430.000000
440.000000
450.000000
460.000000
470.000000
480.000000
490.000000
500.000000
510.000000
520.000000
530.000000
540.000000
550.000000
560.000000
570.000000
580.000000
590.000000
600.000000
610.000000
620.000000
630.000000
640.000000
650.000000
660.000000
670.000000
680.000000
690.000000
700.000000
710.000000
720.000000
730.000000
740.000000
750.000000
760.000000
770.000000
780.000000
790.000000
800.000000
810.000000
820.000000
830.000000
840.000000
850.000000
860.000000
870.000000
880.000000
890.000000
900.000000
910.000000
920.000000
930.000000
940.000000
950.000000
960.000000
970.000000
980.000000
990.000000
1000.000000

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: Clicks ('13')\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: Click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 10: noop()\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: I will use action: click(bid) to click the button.\\n\",\n \"Step 3: Yes, I can handle this. I will use the `click()` action to click the button.\\n\",\n \"Step 4: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: Click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 10: Pass the button ID ('Click Me!') to the action \\\"click('bid')\\\".\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: I will click the button by emitting `click(bid)` and `fill(bid, text)` simultaneously.\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: noop()\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: - Noop()\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: -noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: Click('13')\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: noop()\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: Complete action: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: I will use the action 'click('bid') to click the button.\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: I call action Click (bid) on the page.\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: noop()\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: Oops()\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: fill(bid, text)\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: def click_button_on_page():\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: click(bid)\\n\",\n \"Step 4: Click('13')\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: noop()\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 5: Click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 6: I will click the button 'Click Me!' by using the action `click(bid)` and emitting a bid of 13.\\n\",\n \"Step 7: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: noop()\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: `click(bid)` - No action\\n\",\n \"Step 2: - Noop()\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: noop()\\n\",\n \"Step 10: I will click the button 'Click Me!' using the action 'click(bid)'.\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: Complete action: click(bid)\\n\",\n \"Step 10: noop()\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: I will perform action 1: click('13') to complete the action.\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: noop()\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: Click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: Click ('13')\\n\",\n \"Step 10: Add action 'fill(bid, text) - Send keyboard input' to perform the click.\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: noop()\\n\",\n \"Step 2: Click('click(bid) - Bid')\\n\",\n \"Step 3: noop()\\n\",\n \"Step 4: noop()\\n\",\n \"Step 5: noop()\\n\",\n \"Step 6: noop()\\n\",\n \"Step 7: noop()\\n\",\n \"Step 8: noop()\\n\",\n \"Step 9: click(bid) - Click element with BrowserGym ID (the number in brackets)\\n\",\n \"Step 10: noop()\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"\\n\",\n \"[DEBUG] rollout_func called with 4 prompts (LLM mode, text-only)\\n\",\n \"[DEBUG] Processing prompt 1/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 2/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 3/4\\n\",\n \"Step 1: click('13')\\n\",\n \"[DEBUG] Processing prompt 4/4\\n\",\n \"Step 1: click('13')\\n\",\n \"* Run finished. Uploading logs to Trackio (please wait...)\\n\"\n ]\n }\n ],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"BZj4IG9ZBAix\"\n },\n \"source\": [\n \"In this step, the fine-tuned model is saved locally and uploaded to the Hugging Face Hub using the configured account credentials.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"244ced1920694dbaae9bf98065b4f01d\",\n \"e3769ae107554c9ba38c1e491b15bf4e\",\n \"6d5b8bff73474faeb1d1b438fb4e8cec\",\n \"9f952f8eb63b42e4b38711737da5461e\",\n \"bd12780895064467b5be14e2ec3df114\",\n \"d1261c1083a74dca877e6eece6395d73\",\n \"999744cacd6a4fb08a1d4977ce2f06fd\",\n \"faa5e0fb4ee244689c0f9eef9902acf7\",\n \"6403bed2cd984ba18f74f416748c64e4\",\n \"38be017369524e2eb22050e7a0a18ec5\",\n \"b0720a4a2df948308011d4d87a288426\",\n \"889ca2520f4d446daf2e6ed16ce11d2e\"\n ]\n },\n \"id\": \"9oOBgEWeBP59\",\n \"outputId\": \"76bef375-fc6b-4fdd-a296-549a9b109b11\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"244ced1920694dbaae9bf98065b4f01d\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"e3769ae107554c9ba38c1e491b15bf4e\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"6d5b8bff73474faeb1d1b438fb4e8cec\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...270m-it/training_args.bin: 100%|##########| 7.57kB / 7.57kB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"9f952f8eb63b42e4b38711737da5461e\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...a-270m-it/tokenizer.model: 100%|##########| 4.69MB / 4.69MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"bd12780895064467b5be14e2ec3df114\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...ma-270m-it/tokenizer.json: 100%|##########| 33.4MB / 33.4MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"d1261c1083a74dca877e6eece6395d73\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...270m-it/model.safetensors: 4%|3 | 41.9MB / 1.07GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"No files have been modified since last commit. Skipping to prevent empty commit.\\n\",\n \"WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"999744cacd6a4fb08a1d4977ce2f06fd\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"faa5e0fb4ee244689c0f9eef9902acf7\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"6403bed2cd984ba18f74f416748c64e4\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...270m-it/training_args.bin: 100%|##########| 7.57kB / 7.57kB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"38be017369524e2eb22050e7a0a18ec5\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...a-270m-it/tokenizer.model: 100%|##########| 4.69MB / 4.69MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"b0720a4a2df948308011d4d87a288426\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...270m-it/model.safetensors: 3%|3 | 33.5MB / 1.07GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"889ca2520f4d446daf2e6ed16ce11d2e\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...ma-270m-it/tokenizer.json: 100%|##########| 33.4MB / 33.4MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"No files have been modified since last commit. Skipping to prevent empty commit.\\n\",\n \"WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.google.colaboratory.intrinsic+json\": {\n \"type\": \"string\"\n },\n \"text/plain\": [\n \"CommitInfo(commit_url='https://huggingface.co/sergiopaniego/browsergym-grpo-functiongemma-270m-it/commit/a17de133c28ca7fddfcb2694c32f2791de5ddbe6', commit_message='End of training', commit_description='', oid='a17de133c28ca7fddfcb2694c32f2791de5ddbe6', pr_url=None, repo_url=RepoUrl('https://huggingface.co/sergiopaniego/browsergym-grpo-functiongemma-270m-it', endpoint='https://huggingface.co', repo_type='model', repo_id='sergiopaniego/browsergym-grpo-functiongemma-270m-it'), pr_revision=None, pr_num=None)\"\n ]\n },\n \"execution_count\": 12,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"talmc8b7nPXJ\"\n },\n \"source\": [\n \"## Load the Fine-Tuned Model and Run Inference\\n\",\n \"\\n\",\n \"The fine-tuned model is loaded to perform inference and evaluate its behavior on the target task. \\n\",\n \"In this case, the model is tested within the BrowserGym environment using OpenEnv, focusing on the *click* task from the MiniWoB++ benchmark, which is included among the available BrowserGym tasks.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"c3879b716f37442a87d51b8414fe8c48\"\n ]\n },\n \"id\": \"iIDiaGVlBP5-\",\n \"outputId\": \"4dc0e365-e89f-40ba-b391-74c7efdc932d\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"c3879b716f37442a87d51b8414fe8c48\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"model.safetensors: 0%| | 0.00/1.07G [00:00 **Note:**\\n\",\n \"> In older GPUs (including those available on Colab), **FP8 support** is limited, so we use the BF16 version of the model.\\n\",\n \"> In that case, you can select the official checkpoint or the one from Unsloth.\\n\",\n \"> If you have access to GPUs with **FP8 support**, you can switch to that version instead.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"83dfeaab2bd04b06899d09b6b35bacd1\",\n \"8588996c1d2d444193e9cf53c1a73b8e\",\n \"138a997da09f40ada32171e51b51b708\",\n \"06ef4d5f41de4436ad4731cbf2f8471f\"\n ]\n },\n \"id\": \"WlK7KYKT99p-\",\n \"outputId\": \"db72808f-21cf-4022-ed1a-b78ebb3ee47e\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoProcessor\\n\",\n \"\\n\",\n \"#model_name = \\\"mistralai/Ministral-3-3B-Instruct-2512\\\"\\n\",\n \"model_name = \\\"mistralai/Ministral-3-3B-Instruct-2512-BF16\\\" # \\\"unsloth/Ministral-3-3B-Instruct-2512\\\"\\n\",\n \"\\n\",\n \"processor = AutoProcessor.from_pretrained(model_name, padding_side=\\\"left\\\")\\n\",\n \"\\n\",\n \"SYSTEM_PROMPT = (\\n\",\n \" \\\"You are a helpful AI Assistant that provides well-reasoned and detailed responses. \\\"\\n\",\n \" \\\"You first think about the reasoning process as an internal monologue and then provide the user with the answer. \\\"\\n\",\n \" \\\"Respond in the following format: \\\\n...\\\\n\\\\n\\\\n...\\\\n\\\"\\n\",\n \")\\n\",\n \"\\n\",\n \"\\n\",\n \"def make_conversation(example):\\n\",\n \" conversation = [\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"system\\\",\\n\",\n \" \\\"content\\\": [{\\\"type\\\": \\\"text\\\", \\\"text\\\": SYSTEM_PROMPT}],\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"user\\\",\\n\",\n \" \\\"content\\\": [\\n\",\n \" {\\\"type\\\": \\\"image\\\", \\\"image\\\": example[\\\"image\\\"]},\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": example[\\\"problem\\\"]},\\n\",\n \" ],\\n\",\n \" },\\n\",\n \" ]\\n\",\n \" return {\\n\",\n \" \\\"prompt\\\": conversation,\\n\",\n \" \\\"image\\\": example[\\\"image\\\"],\\n\",\n \" }\\n\",\n \"\\n\",\n \"train_dataset = train_dataset.map(make_conversation)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"5txAuMAa8ock\"\n },\n \"source\": [\n \"Let's review one example to understand the internal structure:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"sjxG7duU99p_\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"ZooycTF099p_\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset = train_dataset.remove_columns(['problem', 'original_question', 'original_answer'])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"2LcjFKgD99p_\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"YY3uMp909Eqy\"\n },\n \"source\": [\n \"## Load model and configure LoRA/QLoRA\\n\",\n \"\\n\",\n \"This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"RcQn7mGs99p_\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import Mistral3ForConditionalGeneration, FineGrainedFP8Config, BitsAndBytesConfig\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"FP8 = False\\n\",\n \"\\n\",\n \"if FP8:\\n\",\n \" model_name = \\\"mistralai/Ministral-3-3B-Instruct-2512\\\"\\n\",\n \" quantization_config = FineGrainedFP8Config(dequantize=False)\\n\",\n \"else:\\n\",\n \" model_name = \\\"mistralai/Ministral-3-3B-Instruct-2512-BF16\\\" # \\\"unsloth/Ministral-3-3B-Instruct-2512\\\"\\n\",\n \" quantization_config = BitsAndBytesConfig(\\n\",\n \" load_in_4bit=True, # Load the model in 4-bit precision to save memory\\n\",\n \" bnb_4bit_compute_dtype=torch.float16, # Data type used for internal computations in quantization\\n\",\n \" bnb_4bit_use_double_quant=True, # Use double quantization to improve accuracy\\n\",\n \" bnb_4bit_quant_type=\\\"nf4\\\", # Type of quantization. \\\"nf4\\\" is recommended for recent LLMs\\n\",\n \" )\\n\",\n \"\\n\",\n \"model = Mistral3ForConditionalGeneration.from_pretrained(\\n\",\n \" model_name,\\n\",\n \" dtype=\\\"float32\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" quantization_config=quantization_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"WZGf-GF09Gsc\"\n },\n \"source\": [\n \"The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a **base model** (the one selected above) and, instead of modifying its original weights, we fine-tune a **LoRA adapter** โ€” a lightweight layer that enables efficient and memory-friendly training. The **`target_modules`** specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"LqCEI4hf99p_\"\n },\n \"outputs\": [],\n \"source\": [\n \"from peft import LoraConfig\\n\",\n \"\\n\",\n \"# You may need to update `target_modules` depending on the architecture of your chosen model.\\n\",\n \"# For example, different VLMs might have different attention/projection layer names.\\n\",\n \"peft_config = LoraConfig(\\n\",\n \" r=8,\\n\",\n \" lora_alpha=32,\\n\",\n \" lora_dropout=0.1,\\n\",\n \" target_modules=[\\\"q_proj\\\", \\\"v_proj\\\"],\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"mDq4V6dN9MGk\"\n },\n \"source\": [\n \"## Train model\\n\",\n \"\\n\",\n \"We'll configure **GRPO** using `GRPOConfig`, keeping the parameters minimal so the training fits on a free Colab instance. You can adjust these settings if more resources are available. For full details on all available parameters, check the [TRL GRPOConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.GRPOConfig).\\n\",\n \"\\n\",\n \"First, we need to define the rewards functions that the training algorithm will use to improve the model. In this case, we'll include two reward functions.\\n\",\n \"We'll use a format reward that will reward the model when the output includes `` and `` tags and additionally a length-based reward to discourage overthinking. Both functions have been extracted from [here](https://github.com/huggingface/open-r1/blob/main/src/open_r1/rewards.py).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"jhgqx8kO99p_\"\n },\n \"outputs\": [],\n \"source\": [\n \"import re\\n\",\n \"\\n\",\n \"def format_reward(completions, **kwargs):\\n\",\n \" \\\"\\\"\\\"Reward function that checks if the reasoning process is enclosed within and tags, while the final answer is enclosed within and tags.\\\"\\\"\\\"\\n\",\n \" pattern = r\\\".*?.*?.*?\\\"\\n\",\n \"\\n\",\n \" matches = []\\n\",\n \" for item in completions:\\n\",\n \" if isinstance(item, list):\\n\",\n \" text = item[0]['content']\\n\",\n \" else:\\n\",\n \" text = item\\n\",\n \" match = re.match(pattern, text, re.DOTALL | re.MULTILINE)\\n\",\n \" matches.append(match)\\n\",\n \"\\n\",\n \" return [1.0 if match else 0.0 for match in matches]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"sVmzQ_wL99p_\"\n },\n \"outputs\": [],\n \"source\": [\n \"from math_verify import LatexExtractionConfig, parse, verify\\n\",\n \"from latex2sympy2_extended import NormalizationConfig\\n\",\n \"\\n\",\n \"\\n\",\n \"def len_reward(completions, solution, **kwargs) -> float:\\n\",\n \" \\\"\\\"\\\"Compute length-based rewards to discourage overthinking and promote token efficiency.\\n\",\n \"\\n\",\n \" Taken from the Kimi 1.5 tech report: https://huggingface.co/papers/2501.12599\\n\",\n \"\\n\",\n \" Args:\\n\",\n \" completions: List of model completions\\n\",\n \" solution: List of ground truth solutions\\n\",\n \"\\n\",\n \" Returns:\\n\",\n \" List of rewards where:\\n\",\n \" - For correct answers: reward = 0.5 - (len - min_len)/(max_len - min_len)\\n\",\n \" - For incorrect answers: reward = min(0, 0.5 - (len - min_len)/(max_len - min_len))\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" contents = []\\n\",\n \" for item in completions:\\n\",\n \" if isinstance(item, list):\\n\",\n \" text = item[0]['content']\\n\",\n \" else:\\n\",\n \" text = item\\n\",\n \" contents.append(text)\\n\",\n \"\\n\",\n \" # First check correctness of answers\\n\",\n \" correctness = []\\n\",\n \" for content, sol in zip(contents, solution):\\n\",\n \" gold_parsed = parse(\\n\",\n \" sol,\\n\",\n \" extraction_mode=\\\"first_match\\\",\\n\",\n \" extraction_config=[LatexExtractionConfig()],\\n\",\n \" )\\n\",\n \" if len(gold_parsed) == 0:\\n\",\n \" # Skip unparsable examples\\n\",\n \" correctness.append(True) # Treat as correct to avoid penalizing\\n\",\n \" print(\\\"Failed to parse gold solution: \\\", sol)\\n\",\n \" continue\\n\",\n \"\\n\",\n \" answer_parsed = parse(\\n\",\n \" content,\\n\",\n \" extraction_config=[\\n\",\n \" LatexExtractionConfig(\\n\",\n \" normalization_config=NormalizationConfig(\\n\",\n \" nits=False,\\n\",\n \" malformed_operators=False,\\n\",\n \" basic_latex=True,\\n\",\n \" equations=True,\\n\",\n \" boxed=True,\\n\",\n \" units=True,\\n\",\n \" ),\\n\",\n \" boxed_match_priority=0,\\n\",\n \" try_extract_without_anchor=False,\\n\",\n \" )\\n\",\n \" ],\\n\",\n \" extraction_mode=\\\"first_match\\\",\\n\",\n \" )\\n\",\n \" correctness.append(verify(answer_parsed, gold_parsed))\\n\",\n \"\\n\",\n \" # Calculate lengths\\n\",\n \" lengths = [len(content) for content in contents]\\n\",\n \" min_len = min(lengths)\\n\",\n \" max_len = max(lengths)\\n\",\n \"\\n\",\n \" # If all responses have the same length, return zero rewards\\n\",\n \" if max_len == min_len:\\n\",\n \" return [0.0] * len(completions)\\n\",\n \"\\n\",\n \" rewards = []\\n\",\n \" for length, is_correct in zip(lengths, correctness):\\n\",\n \" lambda_val = 0.5 - (length - min_len) / (max_len - min_len)\\n\",\n \"\\n\",\n \" if is_correct:\\n\",\n \" reward = lambda_val\\n\",\n \" else:\\n\",\n \" reward = min(0, lambda_val)\\n\",\n \"\\n\",\n \" rewards.append(float(reward))\\n\",\n \"\\n\",\n \" return rewards\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"9xBL7Rni9LZb\"\n },\n \"source\": [\n \"After defining the reward function(s), we can define the `GRPOConfig`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"pcv6KXUD99qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOConfig\\n\",\n \"\\n\",\n \"output_dir = \\\"Ministral-3-3B-Instruct-trl-grpo\\\"\\n\",\n \"\\n\",\n \"# Configure training arguments using GRPOConfig\\n\",\n \"training_args = GRPOConfig(\\n\",\n \" learning_rate=2e-5,\\n\",\n \" #num_train_epochs=1,\\n\",\n \" max_steps=100, # Number of dataset passes. For full trainings, use `num_train_epochs` instead\\n\",\n \"\\n\",\n \" # Parameters that control the data preprocessing\\n\",\n \" per_device_train_batch_size=2,\\n\",\n \" max_completion_length=1024, # default: 256 # Max completion length produced during training\\n\",\n \" num_generations=2, # 2, # default: 8 # Number of generations produced during training for comparison\\n\",\n \"\\n\",\n \" fp16=False,\\n\",\n \" bf16=False,\\n\",\n \"\\n\",\n \" # Parameters related to reporting and saving\\n\",\n \" output_dir=output_dir, # Where to save model checkpoints and logs\\n\",\n \" logging_steps=1, # Log training metrics every N steps\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool\\n\",\n \" trackio_space_id = output_dir,\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub=True,\\n\",\n \" log_completions=True,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"O0q3myQg927v\"\n },\n \"source\": [\n \"Configure the GRPO Trainer. We pass the previously configured `training_args`. We don't use eval dataset to maintain memory usage low but you can configure it.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"-zd7s5Cs99qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOTrainer\\n\",\n \"\\n\",\n \"trainer = GRPOTrainer(\\n\",\n \" model=model,\\n\",\n \" reward_funcs=[format_reward, len_reward],\\n\",\n \" args=training_args,\\n\",\n \" train_dataset=train_dataset,\\n\",\n \" peft_config=peft_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"kQC7Q5kg95xq\"\n },\n \"source\": [\n \"Show memory stats before training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"iF7cnD0T99qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"gpu_stats = torch.cuda.get_device_properties(0)\\n\",\n \"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\\n\",\n \"\\n\",\n \"print(f\\\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\\\")\\n\",\n \"print(f\\\"{start_gpu_memory} GB of memory reserved.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"YazYtLAe97Dc\"\n },\n \"source\": [\n \"And train!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"Ynhxdv3a99qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"SmcYN5yW99IP\"\n },\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"mi-exH7699qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"saarW87Y9_-R\"\n },\n \"source\": [\n \"## Saving fine tuned model\\n\",\n \"\\n\",\n \"In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"m3mlwQl699qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub(dataset_name=dataset_id)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"nfqvO0qw-OvS\"\n },\n \"source\": [\n \"## Load the fine-tuned model and run inference\\n\",\n \"\\n\",\n \"Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"B7usNBq699qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import Mistral3ForConditionalGeneration, MistralCommonBackend\\n\",\n \"from peft import PeftModel\\n\",\n \"\\n\",\n \"base_model = model_name\\n\",\n \"adapter_model = f\\\"{output_dir}\\\" # Replace with your HF username or organization\\n\",\n \"\\n\",\n \"model = Mistral3ForConditionalGeneration.from_pretrained(base_model, dtype=\\\"float32\\\", device_map=\\\"auto\\\")\\n\",\n \"model = PeftModel.from_pretrained(model, adapter_model)\\n\",\n \"\\n\",\n \"tokenizer = MistralCommonBackend.from_pretrained(base_model)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"XnIOkXfy99qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"0le5gBl_99qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"import base64\\n\",\n \"from io import BytesIO\\n\",\n \"\\n\",\n \"dataset_id = 'lmms-lab/multimodal-open-r1-8k-verified'\\n\",\n \"train_dataset = load_dataset(dataset_id, split='train[:5%]')\\n\",\n \"\\n\",\n \"problem = train_dataset[0]['problem']\\n\",\n \"image = train_dataset[0]['image']\\n\",\n \"\\n\",\n \"buffer = BytesIO()\\n\",\n \"image.save(buffer, format=\\\"JPEG\\\")\\n\",\n \"image_bytes = buffer.getvalue()\\n\",\n \"image_b64 = base64.b64encode(image_bytes).decode(\\\"utf-8\\\")\\n\",\n \"\\n\",\n \"messages = [\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"system\\\", \\\"content\\\": [\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": SYSTEM_PROMPT}\\n\",\n \" ]\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"user\\\",\\n\",\n \" \\\"content\\\": [\\n\",\n \" {\\n\",\n \" \\\"type\\\": \\\"image_url\\\",\\n\",\n \" \\\"image_url\\\": {\\n\",\n \" \\\"url\\\": f\\\"data:image/jpeg;base64,{image_b64}\\\"\\n\",\n \" },\\n\",\n \" },\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": problem},\\n\",\n \" ],\\n\",\n \" },\\n\",\n \"]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"f9PgBCD499qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"messages\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"ENOGILKk99qA\"\n },\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"\\n\",\n \"tokenized = tokenizer.apply_chat_template(messages, return_tensors=\\\"pt\\\", return_dict=True)\\n\",\n \"tokenized[\\\"input_ids\\\"] = tokenized[\\\"input_ids\\\"].to(device=\\\"cuda\\\")\\n\",\n \"tokenized[\\\"pixel_values\\\"] = tokenized[\\\"pixel_values\\\"].to(dtype=torch.bfloat16, device=\\\"cuda\\\")\\n\",\n \"image_sizes = [tokenized[\\\"pixel_values\\\"].shape[-2:]]\\n\",\n \"\\n\",\n \"output = model.generate(\\n\",\n \" **tokenized,\\n\",\n \" image_sizes=image_sizes,\\n\",\n \" max_new_tokens=512,\\n\",\n \")[0]\\n\",\n \"\\n\",\n \"decoded_output = tokenizer.decode(output[len(tokenized[\\\"input_ids\\\"][0]):])\\n\",\n \"print(decoded_output)\"\n ]\n }\n ],\n \"metadata\": {\n \"accelerator\": \"GPU\",\n \"colab\": {\n \"gpuType\": \"T4\",\n \"provenance\": []\n },\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 0\n}\n" }, { "path": "examples/notebooks/grpo_qwen3_vl.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"-J8iGzLf4rUJ\"\n },\n \"source\": [\n \"# GRPO Qwen3-VL with QLoRA using TRL\\n\",\n \"\\n\",\n \"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_qwen3_vl.ipynb)\\n\",\n \"\\n\",\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\\n\",\n \"\\n\",\n \"\\n\",\n \"With [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl), you can fine-tune cutting edge vision language models. It comes with support for quantized parameter efficient fine-tuning technique **QLoRA**, so we can use free Colab (T4 GPU) to fine-tune models like [Qwen3-VL](https://huggingface.co/collections/Qwen/qwen3-vl-68d2a7c1b8a8afce4ebd2dbe).\\n\",\n \"\\n\",\n \"\\n\",\n \"- [TRL GitHub Repository](https://github.com/huggingface/trl) โ€” star us to support the project! \\n\",\n \"- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview) \\n\",\n \"- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\\n\",\n \"- [More Qwen3-VL Fine-tuning Examples (including TRL scripts)](https://github.com/QwenLM/Qwen3-VL/tree/main/qwen-vl-finetune/)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"NvrzGRnu48Vz\"\n },\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"We'll install **TRL** with the **PEFT** extra, which ensures all main dependencies such as **Transformers** and **PEFT** (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install **trackio** to log and monitor our experiments, and **bitsandbytes** to enable quantization of LLMs, reducing memory consumption for both inference and training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"8CfZlUevmkg7\"\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -Uq \\\"trl[peft]\\\" bitsandbytes trackio math_verify\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"gpzI6omi7728\"\n },\n \"source\": [\n \"### Log in to Hugging Face\\n\",\n \"\\n\",\n \"Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"4Ncx0wYtnYCW\"\n },\n \"outputs\": [],\n \"source\": [\n \"from huggingface_hub import notebook_login\\n\",\n \"\\n\",\n \"notebook_login()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"V_Zylc4t79-n\"\n },\n \"source\": [\n \"## Load dataset\\n\",\n \"\\n\",\n \"\\n\",\n \"We'll load the [**lmms-lab/multimodal-open-r1-8k-verified**](https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified) dataset from the Hugging Face Hub using the `datasets` library.\\n\",\n \"\\n\",\n \"This dataset contains maths problems with the image representing the problem, along with the solution in thinking format specially tailored for VLMs. By training our model with this dataset, it'll improve its maths and thinking reasoning.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"TzXogU24F_QR\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_id = 'lmms-lab/multimodal-open-r1-8k-verified'\\n\",\n \"train_dataset = load_dataset(dataset_id, split='train[:5%]')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"gVV7RoRN8zk5\"\n },\n \"source\": [\n \"In addition to the `problem` and `image` columns, we also include a custom system prompt to tell the model how we'd like the generation.\\n\",\n \"\\n\",\n \"The system prompt is extracted from DeepSeek R1. Refer to [this previous recipe](https://huggingface.co/learn/cookbook/fine_tuning_llm_grpo_trl) for more details.\\n\",\n \"\\n\",\n \"We convert the dataset samples into conversation samples, including the system prompt and one image and problem description per sample, since this is how the GRPO trainer expects them.\\n\",\n \"\\n\",\n \"We also set `padding_side=\\\"left\\\"` to ensure that generated completions during training are concatenated directly after the prompt, which is essential for GRPO to correctly compare token-level probabilities between preferred and rejected responses.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"ZT1JfiiTGExB\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoProcessor\\n\",\n \"\\n\",\n \"model_name = \\\"Qwen/Qwen3-VL-4B-Instruct\\\" # \\\"Qwen/Qwen3-VL-8B-Instruct\\\"\\n\",\n \"processor = AutoProcessor.from_pretrained(model_name, padding_side=\\\"left\\\")\\n\",\n \"\\n\",\n \"SYSTEM_PROMPT = (\\n\",\n \" \\\"You are a helpful AI Assistant that provides well-reasoned and detailed responses. \\\"\\n\",\n \" \\\"You first think about the reasoning process as an internal monologue and then provide the user with the answer. \\\"\\n\",\n \" \\\"Respond in the following format: \\\\n...\\\\n\\\\n\\\\n...\\\\n\\\"\\n\",\n \")\\n\",\n \"\\n\",\n \"\\n\",\n \"def make_conversation(example):\\n\",\n \" prompt = [\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"system\\\",\\n\",\n \" \\\"content\\\": [{\\\"type\\\": \\\"text\\\", \\\"text\\\": SYSTEM_PROMPT}],\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"user\\\",\\n\",\n \" \\\"content\\\": [\\n\",\n \" {\\\"type\\\": \\\"image\\\", \\\"image\\\": example[\\\"image\\\"]},\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": example[\\\"problem\\\"]},\\n\",\n \" ],\\n\",\n \" },\\n\",\n \" ]\\n\",\n \" return {\\\"prompt\\\": prompt, \\\"image\\\": example[\\\"image\\\"]}\\n\",\n \"\\n\",\n \"train_dataset = train_dataset.map(make_conversation)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"5txAuMAa8ock\"\n },\n \"source\": [\n \"Let's review one example to understand the internal structure:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"PDXQd5Jk2Bqe\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"hzSR_56wxKDA\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset = train_dataset.remove_columns(['problem', 'original_question', 'original_answer'])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"T9rCkeqDODba\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"YY3uMp909Eqy\"\n },\n \"source\": [\n \"## Load model and configure LoRA/QLoRA\\n\",\n \"\\n\",\n \"This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"gt05dgXgm9QR\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import Qwen3VLForConditionalGeneration, BitsAndBytesConfig\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"model = Qwen3VLForConditionalGeneration.from_pretrained(\\n\",\n \" model_name, dtype=\\\"float32\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" quantization_config=BitsAndBytesConfig(\\n\",\n \" load_in_4bit=True,\\n\",\n \" bnb_4bit_use_double_quant=True,\\n\",\n \" bnb_4bit_quant_type=\\\"nf4\\\",\\n\",\n \" bnb_4bit_compute_dtype=torch.float16\\n\",\n \" ),\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"WZGf-GF09Gsc\"\n },\n \"source\": [\n \"The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a **base model** (the one selected above) and, instead of modifying its original weights, we fine-tune a **LoRA adapter** โ€” a lightweight layer that enables efficient and memory-friendly training. The **`target_modules`** specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"ME1im5gh2LFg\"\n },\n \"outputs\": [],\n \"source\": [\n \"from peft import LoraConfig\\n\",\n \"\\n\",\n \"# You may need to update `target_modules` depending on the architecture of your chosen model.\\n\",\n \"# For example, different VLMs might have different attention/projection layer names.\\n\",\n \"peft_config = LoraConfig(\\n\",\n \" r=8,\\n\",\n \" lora_alpha=32,\\n\",\n \" lora_dropout=0.1,\\n\",\n \" target_modules=[\\\"q_proj\\\", \\\"v_proj\\\"],\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"mDq4V6dN9MGk\"\n },\n \"source\": [\n \"## Train model\\n\",\n \"\\n\",\n \"We'll configure **GRPO** using `GRPOConfig`, keeping the parameters minimal so the training fits on a free Colab instance. You can adjust these settings if more resources are available. For full details on all available parameters, check the [TRL GRPOConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.GRPOConfig).\\n\",\n \"\\n\",\n \"First, we need to define the rewards functions that the training algorithm will use to improve the model. In this case, we'll include two reward functions.\\n\",\n \"We'll use a format reward that will reward the model when the output includes `` and `` tags and additionally a length-based reward to discourage overthinking. Both functions have been extracted from [here](https://github.com/huggingface/open-r1/blob/main/src/open_r1/rewards.py).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"Dqp3TfUwHUxW\"\n },\n \"outputs\": [],\n \"source\": [\n \"import re\\n\",\n \"\\n\",\n \"def format_reward(completions, **kwargs):\\n\",\n \" \\\"\\\"\\\"Reward function that checks if the reasoning process is enclosed within and tags, while the final answer is enclosed within and tags.\\\"\\\"\\\"\\n\",\n \" pattern = r\\\"^\\\\n.*?\\\\n\\\\n\\\\n.*?\\\\n$\\\"\\n\",\n \" matches = [re.match(pattern, content, re.DOTALL | re.MULTILINE) for content in completions]\\n\",\n \" return [1.0 if match else 0.0 for match in matches]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"rxNPUp7RBFcz\"\n },\n \"outputs\": [],\n \"source\": [\n \"from math_verify import LatexExtractionConfig, parse, verify\\n\",\n \"from latex2sympy2_extended import NormalizationConfig\\n\",\n \"\\n\",\n \"\\n\",\n \"def len_reward(completions, solution, **kwargs) -> float:\\n\",\n \" \\\"\\\"\\\"Compute length-based rewards to discourage overthinking and promote token efficiency.\\n\",\n \"\\n\",\n \" Taken from the Kimi 1.5 tech report: https://huggingface.co/papers/2501.12599\\n\",\n \"\\n\",\n \" Args:\\n\",\n \" completions: List of model completions\\n\",\n \" solution: List of ground truth solutions\\n\",\n \"\\n\",\n \" Returns:\\n\",\n \" List of rewards where:\\n\",\n \" - For correct answers: reward = 0.5 - (len - min_len)/(max_len - min_len)\\n\",\n \" - For incorrect answers: reward = min(0, 0.5 - (len - min_len)/(max_len - min_len))\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" contents = completions\\n\",\n \"\\n\",\n \" # First check correctness of answers\\n\",\n \" correctness = []\\n\",\n \" for content, sol in zip(contents, solution):\\n\",\n \" gold_parsed = parse(\\n\",\n \" sol,\\n\",\n \" extraction_mode=\\\"first_match\\\",\\n\",\n \" extraction_config=[LatexExtractionConfig()],\\n\",\n \" )\\n\",\n \" if len(gold_parsed) == 0:\\n\",\n \" # Skip unparsable examples\\n\",\n \" correctness.append(True) # Treat as correct to avoid penalizing\\n\",\n \" print(\\\"Failed to parse gold solution: \\\", sol)\\n\",\n \" continue\\n\",\n \"\\n\",\n \" answer_parsed = parse(\\n\",\n \" content,\\n\",\n \" extraction_config=[\\n\",\n \" LatexExtractionConfig(\\n\",\n \" normalization_config=NormalizationConfig(\\n\",\n \" nits=False,\\n\",\n \" malformed_operators=False,\\n\",\n \" basic_latex=True,\\n\",\n \" equations=True,\\n\",\n \" boxed=True,\\n\",\n \" units=True,\\n\",\n \" ),\\n\",\n \" boxed_match_priority=0,\\n\",\n \" try_extract_without_anchor=False,\\n\",\n \" )\\n\",\n \" ],\\n\",\n \" extraction_mode=\\\"first_match\\\",\\n\",\n \" )\\n\",\n \" correctness.append(verify(answer_parsed, gold_parsed))\\n\",\n \"\\n\",\n \" # Calculate lengths\\n\",\n \" lengths = [len(content) for content in contents]\\n\",\n \" min_len = min(lengths)\\n\",\n \" max_len = max(lengths)\\n\",\n \"\\n\",\n \" # If all responses have the same length, return zero rewards\\n\",\n \" if max_len == min_len:\\n\",\n \" return [0.0] * len(completions)\\n\",\n \"\\n\",\n \" rewards = []\\n\",\n \" for length, is_correct in zip(lengths, correctness):\\n\",\n \" lambda_val = 0.5 - (length - min_len) / (max_len - min_len)\\n\",\n \"\\n\",\n \" if is_correct:\\n\",\n \" reward = lambda_val\\n\",\n \" else:\\n\",\n \" reward = min(0, lambda_val)\\n\",\n \"\\n\",\n \" rewards.append(float(reward))\\n\",\n \"\\n\",\n \" return rewards\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"9xBL7Rni9LZb\"\n },\n \"source\": [\n \"After defining the reward function(s), we can define the `GRPOConfig`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"OEmRM0rIHXQ4\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOConfig\\n\",\n \"\\n\",\n \"output_dir = \\\"Qwen3-VL-4B-Instruct-trl-grpo\\\"\\n\",\n \"\\n\",\n \"# Configure training arguments using GRPOConfig\\n\",\n \"training_args = GRPOConfig(\\n\",\n \" learning_rate=2e-5,\\n\",\n \" #num_train_epochs=1,\\n\",\n \" max_steps=100, # Number of dataset passes. For full trainings, use `num_train_epochs` instead\\n\",\n \"\\n\",\n \" # Parameters that control the data preprocessing\\n\",\n \" per_device_train_batch_size=2,\\n\",\n \" max_completion_length=1024, # default: 256 # Max completion length produced during training\\n\",\n \" num_generations=2, # 2, # default: 8 # Number of generations produced during training for comparison\\n\",\n \"\\n\",\n \" fp16=True,\\n\",\n \"\\n\",\n \" # Parameters related to reporting and saving\\n\",\n \" output_dir=output_dir, # Where to save model checkpoints and logs\\n\",\n \" logging_steps=1, # Log training metrics every N steps\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub=True,\\n\",\n \" log_completions=True\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"O0q3myQg927v\"\n },\n \"source\": [\n \"Configure the GRPO Trainer. We pass the previously configured `training_args`. We don't use eval dataset to maintain memory usage low but you can configure it.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"base_uri\": \"https://localhost:8080/\"\n },\n \"id\": \"z5JxkmS9HqD5\",\n \"outputId\": \"2b39338e-2194-4829-fc54-5e286566fd28\"\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/usr/local/lib/python3.12/dist-packages/peft/mapping_func.py:73: UserWarning: You are trying to modify a model with PEFT for a second time. If you want to reload the model with a different config, make sure to call `.unload()` before.\\n\",\n \" warnings.warn(\\n\",\n \"/usr/local/lib/python3.12/dist-packages/peft/tuners/tuners_utils.py:196: UserWarning: Already found a `peft_config` attribute in the model. This will lead to having multiple adapters in the model. Make sure to know what you are doing!\\n\",\n \" warnings.warn(\\n\"\n ]\n }\n ],\n \"source\": [\n \"from trl import GRPOTrainer\\n\",\n \"\\n\",\n \"trainer = GRPOTrainer(\\n\",\n \" model=model,\\n\",\n \" reward_funcs=[format_reward, len_reward],\\n\",\n \" args=training_args,\\n\",\n \" train_dataset=train_dataset,\\n\",\n \" peft_config=peft_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"kQC7Q5kg95xq\"\n },\n \"source\": [\n \"Show memory stats before training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"naG_7qlYyBP6\"\n },\n \"outputs\": [],\n \"source\": [\n \"gpu_stats = torch.cuda.get_device_properties(0)\\n\",\n \"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\\n\",\n \"\\n\",\n \"print(f\\\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\\\")\\n\",\n \"print(f\\\"{start_gpu_memory} GB of memory reserved.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"YazYtLAe97Dc\"\n },\n \"source\": [\n \"And train!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"pbJXrhA0ywra\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"SmcYN5yW99IP\"\n },\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"TrrwP4ADMmrp\"\n },\n \"outputs\": [],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"saarW87Y9_-R\"\n },\n \"source\": [\n \"## Saving fine tuned model\\n\",\n \"\\n\",\n \"In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"71A8aqEyyETA\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub(dataset_name=dataset_id)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"nfqvO0qw-OvS\"\n },\n \"source\": [\n \"## Load the fine-tuned model and run inference\\n\",\n \"\\n\",\n \"Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"R8T2uFQVyFeH\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import Qwen3VLForConditionalGeneration, AutoProcessor\\n\",\n \"from peft import PeftModel\\n\",\n \"\\n\",\n \"base_model = model_name\\n\",\n \"adapter_model = f\\\"{output_dir}\\\" # Replace with your HF username or organization\\n\",\n \"\\n\",\n \"model = Qwen3VLForConditionalGeneration.from_pretrained(base_model, dtype=\\\"float32\\\", device_map=\\\"auto\\\")\\n\",\n \"model = PeftModel.from_pretrained(model, adapter_model)\\n\",\n \"\\n\",\n \"processor = AutoProcessor.from_pretrained(base_model)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"dPBHP0CpLa6K\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"cG5-ccGRyHgo\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_id = 'lmms-lab/multimodal-open-r1-8k-verified'\\n\",\n \"train_dataset = load_dataset(dataset_id, split='train[:5%]')\\n\",\n \"\\n\",\n \"problem = train_dataset[0]['problem']\\n\",\n \"image = train_dataset[0]['image']\\n\",\n \"\\n\",\n \"messages = [\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"system\\\", \\\"content\\\": [\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": SYSTEM_PROMPT}\\n\",\n \" ]\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"user\\\",\\n\",\n \" \\\"content\\\": [\\n\",\n \" {\\\"type\\\": \\\"image\\\", \\\"image\\\": image},\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": problem},\\n\",\n \" ],\\n\",\n \" },\\n\",\n \"]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"r_70q_8lLgfV\"\n },\n \"outputs\": [],\n \"source\": [\n \"messages\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"PX92MjqlyIwB\"\n },\n \"outputs\": [],\n \"source\": [\n \"inputs = processor.apply_chat_template(\\n\",\n \" messages,\\n\",\n \" add_generation_prompt=True,\\n\",\n \" tokenize=True,\\n\",\n \" return_tensors=\\\"pt\\\",\\n\",\n \" return_dict=True,\\n\",\n \").to(model.device)\\n\",\n \"\\n\",\n \"# Inference: Generation of the output\\n\",\n \"generated_ids = model.generate(**inputs, max_new_tokens=500)\\n\",\n \"generated_ids_trimmed = [\\n\",\n \" out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\\n\",\n \"]\\n\",\n \"output_text = processor.batch_decode(\\n\",\n \" generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\\n\",\n \")\\n\",\n \"print(output_text)\"\n ]\n }\n ],\n \"metadata\": {\n \"accelerator\": \"GPU\",\n \"colab\": {\n \"gpuType\": \"T4\",\n \"provenance\": []\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 0\n}\n" }, { "path": "examples/notebooks/grpo_rnj_1_instruct.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"-J8iGzLf4rUJ\"\n },\n \"source\": [\n \"# GRPO EssentialAI/rnj-1-instruct with QLoRA using TRL\\n\",\n \"\\n\",\n \"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_rnj_1_instruct.ipynb)\\n\",\n \"\\n\",\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\\n\",\n \"\\n\",\n \"\\n\",\n \"With [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl), you can fine-tune cutting edge large language models. It comes with support for quantized parameter efficient fine-tuning technique **QLoRA**, so we can use Colab to fine-tune models like [EssentialAI/rnj-1-instruct](https://huggingface.co/collections/EssentialAI/rnj-1).\\n\",\n \"\\n\",\n \"\\n\",\n \"- [TRL GitHub Repository](https://github.com/huggingface/trl) โ€” star us to support the project! \\n\",\n \"- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview) \\n\",\n \"- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\\n\",\n \"\\n\",\n \"In this notebook, we'll add reasoning capabilities to the model, teaching it to generate reasoning traces (``) before giving us the final answer (``).\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"NvrzGRnu48Vz\"\n },\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"We'll install **TRL** with the **PEFT** extra, which ensures all main dependencies such as **Transformers** and **PEFT** (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install **trackio** to log and monitor our experiments, and **bitsandbytes** to enable quantization of LLMs, reducing memory consumption for both inference and training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"8VOdRz9fgFa8\"\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -Uq \\\"trl[peft]\\\" bitsandbytes trackio math_verify\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"gpzI6omi7728\"\n },\n \"source\": [\n \"### Log in to Hugging Face\\n\",\n \"\\n\",\n \"Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"d3j3BsdQgFa8\"\n },\n \"outputs\": [],\n \"source\": [\n \"from huggingface_hub import notebook_login\\n\",\n \"\\n\",\n \"notebook_login()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"V_Zylc4t79-n\"\n },\n \"source\": [\n \"## Load dataset\\n\",\n \"\\n\",\n \"\\n\",\n \"We'll load the [**AI-MO/NuminaMath-TIR**](https://huggingface.co/datasets/AI-MO/NuminaMath-TIR) dataset from the Hugging Face Hub using the `datasets` library.\\n\",\n \"\\n\",\n \"This dataset contains maths problems, along with the solution in thinking format specially tailored for LLMs. By training our model with this dataset, it'll improve its maths and thinking reasoning.\\n\",\n \"\\n\",\n \"> We only use a subset for educational purposes. In a real scenario, we'd use the complete dataset.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"YSuLNZAmgFa9\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_id = 'AI-MO/NuminaMath-TIR'\\n\",\n \"train_dataset = load_dataset(dataset_id, split='train[:5%]')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"gVV7RoRN8zk5\"\n },\n \"source\": [\n \"In addition to the current columns, we also include a custom system prompt to tell the model how we'd like the generation.\\n\",\n \"\\n\",\n \"This system prompt is an adapted version of the original one extracted from **DeepSeek R1**. For additional background, see [this previous recipe](https://huggingface.co/learn/cookbook/fine_tuning_llm_grpo_trl). We extend the prompt with **examples** and a **more explicit, verbose formulation** to make the desired behavior easier for the model to learn. Depending on your goals, you may further enrich the prompt to simplify learning, or intentionally shorten and harden it to encourage more robust and generalizable behavior.\\n\",\n \"\\n\",\n \"We convert the dataset samples into conversation samples, including the system prompt and problem description per sample, since this is how the GRPO trainer expects them.\\n\",\n \"\\n\",\n \"We also set `padding_side=\\\"left\\\"` to ensure that generated completions during training are concatenated directly after the prompt, which is essential for GRPO to correctly compare token-level probabilities between preferred and rejected responses.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"vr9t-9Z5gFa9\"\n },\n \"outputs\": [],\n \"source\": [\n \"SYSTEM_PROMPT = \\\"\\\"\\\"A conversation between User and Assistant. The user asks a question, and the Assistant solves it.\\n\",\n \"The assistant first thinks about the reasoning process in the mind and then provides the user with the answer.\\n\",\n \"The reasoning process and answer are enclosed within and tags.\\n\",\n \"Use exactly one ... block followed by exactly one ... block.\\n\",\n \"\\n\",\n \"Examples:\\n\",\n \"\\n\",\n \"User: What is 2 + 2?\\n\",\n \"Assistant:\\n\",\n \"\\n\",\n \"I will add 2 and 2 together.\\n\",\n \"\\n\",\n \"4\\n\",\n \"\\n\",\n \"User: What is 3 ร— 5?\\n\",\n \"Assistant:\\n\",\n \"\\n\",\n \"I will multiply 3 by 5.\\n\",\n \"\\n\",\n \"15\\n\",\n \"\\n\",\n \"User: Find the GCD of 12 and 18.\\n\",\n \"Assistant:\\n\",\n \"\\n\",\n \"I will list the divisors of 12 and 18 and find the greatest one they have in common.\\n\",\n \"\\n\",\n \"6\\n\",\n \"\\\"\\\"\\\"\\n\",\n \"\\n\",\n \"def make_conversation(example):\\n\",\n \" return {\\n\",\n \" \\\"prompt\\\": [\\n\",\n \" {\\\"role\\\": \\\"system\\\", \\\"content\\\": SYSTEM_PROMPT},\\n\",\n \" {\\\"role\\\": \\\"user\\\", \\\"content\\\": example[\\\"problem\\\"]},\\n\",\n \" ],\\n\",\n \" }\\n\",\n \"\\n\",\n \"train_dataset = train_dataset.map(make_conversation)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"5txAuMAa8ock\"\n },\n \"source\": [\n \"Let's review one example to understand the internal structure:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"jZtkB0D9gFa9\"\n },\n \"outputs\": [],\n \"source\": [\n \"print(train_dataset[0])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"FtdKjmyFZImL\"\n },\n \"source\": [\n \"And remove the columns that are not needed for training:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"Ai4F1GaPgFa-\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset = train_dataset.remove_columns(['messages', 'problem'])\\n\",\n \"print(train_dataset)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"YY3uMp909Eqy\"\n },\n \"source\": [\n \"## Load model and configure LoRA/QLoRA\\n\",\n \"\\n\",\n \"This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"DSKcUQ9RgFa-\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, BitsAndBytesConfig\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"model_name = \\\"EssentialAI/rnj-1-instruct\\\"\\n\",\n \"\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" model_name,\\n\",\n \" dtype=\\\"float32\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" quantization_config=BitsAndBytesConfig(\\n\",\n \" load_in_4bit=True,\\n\",\n \" bnb_4bit_use_double_quant=True,\\n\",\n \" bnb_4bit_quant_type=\\\"nf4\\\",\\n\",\n \" bnb_4bit_compute_dtype=torch.float16\\n\",\n \" ),\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"WZGf-GF09Gsc\"\n },\n \"source\": [\n \"The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a **base model** (the one selected above) and, instead of modifying its original weights, we fine-tune a **LoRA adapter**, a lightweight layer that enables efficient and memory-friendly training. The **`target_modules`** specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"nMMlDxJSgFa-\"\n },\n \"outputs\": [],\n \"source\": [\n \"from peft import LoraConfig\\n\",\n \"\\n\",\n \"# You may need to update `target_modules` depending on the architecture of your chosen model.\\n\",\n \"# For example, different LLMs might have different attention/projection layer names.\\n\",\n \"peft_config = LoraConfig(\\n\",\n \" r=32,\\n\",\n \" lora_alpha=32,\\n\",\n \" target_modules = [\\\"q_proj\\\", \\\"k_proj\\\", \\\"v_proj\\\", \\\"o_proj\\\", \\\"gate_proj\\\", \\\"up_proj\\\", \\\"down_proj\\\",],\\n\",\n \")\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"mDq4V6dN9MGk\"\n },\n \"source\": [\n \"## Train model\\n\",\n \"\\n\",\n \"We'll configure **GRPO** using `GRPOConfig`, keeping the parameters minimal so the training fits on a Colab instance. You can adjust these settings depending on the resources available. For full details on all available parameters, check the [TRL GRPOConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.GRPOConfig).\\n\",\n \"\\n\",\n \"First, we need to define the rewards functions that the training algorithm will use to improve the model. In this case, we'll include just one reward function.\\n\",\n \"We'll use a format reward that will reward the model when the output includes `` and `` tags. This is a simplification of the pipeline for educational purposes, but in a real scenario, you'd at least all need a reward function to check the correctness of the model answer. The function has been extracted from [here](https://github.com/huggingface/open-r1/blob/main/src/open_r1/rewards.py).\\n\",\n \"\\n\",\n \"> ๐Ÿ’ก **Note**: \\n\",\n \"> You can further refine this reward by making it more granular. For example, assigning partial rewards when `` and `` appear independently, or when they are present but incorrectly ordered. This can make the learning signal denser and speed up early training. However, overly simplifying the reward may reduce robustness, even if it helps the model converge faster. In practice, there is a trade-off between ease of learning and the generalization quality of the final model.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"Rtx5owCRgFa-\"\n },\n \"outputs\": [],\n \"source\": [\n \"import re\\n\",\n \"\\n\",\n \"def format_reward(completions, **kwargs):\\n\",\n \" \\\"\\\"\\\"Reward function that checks if the reasoning process is enclosed within and tags, while the final answer is enclosed within and tags.\\\"\\\"\\\"\\n\",\n \" pattern = r\\\".*?.*?.*?\\\"\\n\",\n \"\\n\",\n \" matches = []\\n\",\n \" for item in completions:\\n\",\n \" if isinstance(item, list):\\n\",\n \" text = item[0]['content']\\n\",\n \" else:\\n\",\n \" text = item\\n\",\n \" match = re.match(pattern, text, re.DOTALL | re.MULTILINE)\\n\",\n \" matches.append(match)\\n\",\n \"\\n\",\n \" return [1.0 if match else 0.0 for match in matches]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"9xBL7Rni9LZb\"\n },\n \"source\": [\n \"After defining the reward function(s), we can define the `GRPOConfig`. You can adapt the values in the config depending on your training setting and even fit the training in more constrained setups like free Colab (T4).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"rJ0VfG3wgFa-\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOConfig\\n\",\n \"\\n\",\n \"output_dir = \\\"EssentialAI-rnj-1-instruct-trl-grpo\\\"\\n\",\n \"\\n\",\n \"# Configure training arguments using GRPOConfig\\n\",\n \"training_args = GRPOConfig(\\n\",\n \" learning_rate=2e-5, # Learning rate used during traing\\n\",\n \" num_train_epochs=1, # Number of full dataset passes. For testing, use `max_steps` instead\\n\",\n \" #max_steps=100,\\n\",\n \"\\n\",\n \" # Parameters that control the data preprocessing\\n\",\n \" per_device_train_batch_size=8,\\n\",\n \" max_completion_length=256, # default: 256 # Max completion length produced during training\\n\",\n \" num_generations=8, # default: 8 # Number of generations produced during training for comparison\\n\",\n \"\\n\",\n \" # Parameters related to reporting and saving\\n\",\n \" output_dir=output_dir, # Where to save model checkpoints and logs\\n\",\n \" logging_steps=10, # Log training metrics every N steps\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool\\n\",\n \" trackio_space_id = output_dir, # HF Space where you trackio will be\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub=True, # Push the resulted model to the Hub\\n\",\n \" log_completions=True, # Log completions during training\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"O0q3myQg927v\"\n },\n \"source\": [\n \"Configure the GRPO Trainer. We pass the previously configured `training_args`. We don't use eval dataset to maintain memory usage low but you can configure it.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"aW7Gi4nXgFa-\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOTrainer\\n\",\n \"\\n\",\n \"trainer = GRPOTrainer(\\n\",\n \" model=model,\\n\",\n \" reward_funcs=[format_reward],\\n\",\n \" args=training_args,\\n\",\n \" train_dataset=train_dataset,\\n\",\n \" peft_config=peft_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"kQC7Q5kg95xq\"\n },\n \"source\": [\n \"Show memory stats before training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"OJdVlC_mgFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"gpu_stats = torch.cuda.get_device_properties(0)\\n\",\n \"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\\n\",\n \"\\n\",\n \"print(f\\\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\\\")\\n\",\n \"print(f\\\"{start_gpu_memory} GB of memory reserved.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"YazYtLAe97Dc\"\n },\n \"source\": [\n \"And train!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"Mtv8s7rBgFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"SmcYN5yW99IP\"\n },\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"-ROfX8e9gFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"saarW87Y9_-R\"\n },\n \"source\": [\n \"## Saving fine tuned model\\n\",\n \"\\n\",\n \"In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"09zYXJ3GgFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub(dataset_name=dataset_id)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"nfqvO0qw-OvS\"\n },\n \"source\": [\n \"## Load the fine-tuned model and run inference\\n\",\n \"\\n\",\n \"Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"9Yk9RAABgFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"output_dir = 'sergiopaniego/EssentialAI-rnj-1-instruct-trl-grpo'\\n\",\n \"model_name = \\\"EssentialAI/rnj-1-instruct\\\"\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"CdzlQcCAgFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from peft import PeftModel\\n\",\n \"\\n\",\n \"base_model = model_name\\n\",\n \"adapter_model = f\\\"{output_dir}\\\" # Replace with your HF username or organization\\n\",\n \"\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(base_model, dtype=\\\"float32\\\", device_map=\\\"auto\\\")\\n\",\n \"model = PeftModel.from_pretrained(model, adapter_model)\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(base_model)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"LZgjlAu-gFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"gjY6TqQHgFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_id = 'AI-MO/NuminaMath-TIR'\\n\",\n \"train_dataset = load_dataset(dataset_id, split='train[:5%]')\\n\",\n \"\\n\",\n \"problem = train_dataset[0]['problem']\\n\",\n \"\\n\",\n \"messages = [\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"system\\\", \\\"content\\\": [\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": SYSTEM_PROMPT}\\n\",\n \" ]\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"user\\\",\\n\",\n \" \\\"content\\\": [\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": problem},\\n\",\n \" ],\\n\",\n \" },\\n\",\n \"]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"eaVubGYmgFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"messages\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"2M6Xh4JMgFa_\"\n },\n \"outputs\": [],\n \"source\": [\n \"input_ids = tokenizer.apply_chat_template(\\n\",\n \" messages,\\n\",\n \" add_generation_prompt=True,\\n\",\n \" return_tensors=\\\"pt\\\",\\n\",\n \" return_dict=False,\\n\",\n \").to(model.device)\\n\",\n \"\\n\",\n \"# --- Generate Prediction --- #\\n\",\n \"print(\\\"Generating prediction...\\\")\\n\",\n \"output_ids = model.generate(\\n\",\n \" input_ids,\\n\",\n \" max_new_tokens=50,\\n\",\n \" pad_token_id=tokenizer.eos_token_id,\\n\",\n \" do_sample=True,\\n\",\n \" temperature=0.2,\\n\",\n \" top_p=0.95\\n\",\n \")\\n\",\n \"\\n\",\n \"response = tokenizer.decode(output_ids[0][input_ids.shape[-1]:], skip_special_tokens=True)\\n\",\n \"print(response)\"\n ]\n }\n ],\n \"metadata\": {\n \"accelerator\": \"GPU\",\n \"colab\": {\n \"gpuType\": \"A100\",\n \"provenance\": []\n },\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 0\n}\n" }, { "path": "examples/notebooks/grpo_trl_lora_qlora.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"27ozP4Uy-Cz2\"\n },\n \"source\": [\n \"# Group Relative Policy Optimization (GRPO) with LoRA/QLoRA using TRL โ€” on a Free Colab Notebook\\n\",\n \"\\n\",\n \"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/grpo_trl_lora_qlora.ipynb)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"eOjY4AR1-QnF\"\n },\n \"source\": [\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\\n\",\n \"\\n\",\n \"Easily fine-tune **Large Language Models (LLMs)** or **Vision-Language Models (VLMs)** with **LoRA** or **QLoRA** using the [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl) library by Hugging Face and Group Relative Policy Optimization (GRPO) โ€” all within a **free Google Colab notebook** powered by a **T4 GPU**.\\n\",\n \"\\n\",\n \"Thanks to the **built-in memory and training optimizations in TRL**, including LoRA, quantization, gradient checkpointing, and optimized attention kernels, it is possible to **fine-tune a 7B model on a free T4** with a **~7ร— reduction in memory consumption** compared to naive FP16 training.\\n\",\n \"\\n\",\n \"- [TRL GitHub Repository](https://github.com/huggingface/trl) โ€” star us to support the project! \\n\",\n \"- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview) \\n\",\n \"- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"w2TnJ6ta-2zj\"\n },\n \"source\": [\n \"## Key concepts\\n\",\n \"\\n\",\n \"- **GRPO**: A reinforcement learning algorithm that optimizes a policy by comparing multiple generated responses for the same prompt and updating the model based on their relative rewards, without requiring a separate value model.\\n\",\n \"- **LoRA**: Updates only a few low-rank parameters, reducing training cost and memory.\\n\",\n \"- **QLoRA**: A quantized version of LoRA that enables even larger models to fit on small GPUs.\\n\",\n \"- **TRL**: The Hugging Face library that makes fine-tuning and reinforcement learning simple and efficient.\\n\",\n \"\\n\",\n \"Learn how to perform **GRPO (Group Relative Policy Optimization)** with **LoRA/QLoRA** using **TRL**.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"EzScUBxoT4Nt\"\n },\n \"source\": [\n \"This table demonstrates how **progressively enabling efficiency techniques** affects **memory usage** and **training throughput** across different hardware configurations. \\n\",\n \"The techniques range from naive FP16 training to **LoRA, quantization, Liger kernels, paged_adamw_8bit, and gradient checkpointing**.\\n\",\n \"\\n\",\n \"| Configuration | LoRA | Quant | Liger | Optimizer | Grad. Ckpt | attn_impl | VRAM (T4) GB | VRAM (A100-40GB)| VRAM (A100-80GB) | Tokens/s (T4) | Tokens/s (A100-40GB) | Tokens/s (A100-80GB) | Status (T4) |\\n\",\n \"|--------------|------|-------|-------|-----------|------------|-----------|---------------|----------------|---------|---------|---------------|------------------|-------------|\\n\",\n \"| **Worst (naive FP16)** | โŒ | โŒ | โŒ | AdamW | โŒ | eager | OOM | OOM | 62 GB | - | - | 0.06 it/s | โŒ |\\n\",\n \"| **Best (all optimizations)** | โœ… | โœ… | โœ… | paged_adamw_8bit | โœ… | sdpa | 9.2 GB | 9.6 GB | 9.6 GB | 0.01 it/s | 0.03 it/s | 0.04 it/s | โœ… |\\n\",\n \"\\n\",\n \"With all efficiency techniques enabled, **memory usage on Colab T4 is reduced by ~7ร—**, making it possible to **fine-tune a 7B model on free Colab** where naive FP16 training would fail.\\n\",\n \"\\n\",\n \"> A small trade-off in training speed is observed, but the **VRAM reduction is the key enabler**. For faster training on compatible hardware, **vLLM** can also be leveraged.\\n\",\n \"\\n\",\n \"> ๐Ÿ’ก Note: For a fair comparison, the number of generations and the batch size were not changed.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"9RFq6Op7rjc3\"\n },\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"We'll install **TRL** with the **PEFT** extra, which ensures all main dependencies such as **Transformers** and **PEFT** (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install **trackio** to log and monitor our experiments, **bitsandbytes** to enable quantization of LLMs, reducing memory consumption for both inference and training, and **liger-kernel** for more efficient training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"c2jy45nfWbdo\"\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -Uq \\\"trl[peft]\\\" bitsandbytes trackio math_verify liger-kernel\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"B33zJG_Q_qb3\"\n },\n \"source\": [\n \"### Log in to Hugging Face\\n\",\n \"\\n\",\n \"Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"eec717d21e734c4da066763b4a6add7e\"\n ]\n },\n \"id\": \"8zqnTyUDWbdo\",\n \"outputId\": \"62d71aaf-352b-4736-acb9-189d78654718\"\n },\n \"outputs\": [],\n \"source\": [\n \"from huggingface_hub import notebook_login\\n\",\n \"\\n\",\n \"notebook_login()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"cTEw4xlFrhnQ\"\n },\n \"source\": [\n \"## Load Dataset\\n\",\n \"\\n\",\n \"In this step, we load the [**AI-MO/NuminaMath-TIR**](https://huggingface.co/datasets/AI-MO/NuminaMath-TIR) dataset from the Hugging Face Hub using the `datasets` library.\\n\",\n \"This dataset focuses on **mathematical reasoning**, featuring problems that require step-by-step logical solutions.\\n\",\n \"By fine-tuning a model that does not yet exhibit strong reasoning capabilities, it can learn to **generate structured reasoning steps**, enhancing both the model's **accuracy** and **interpretability** on math-related tasks.\\n\",\n \"\\n\",\n \"For efficiency, we'll load only a **small portion of the training split**:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"zU5icx67Wbdp\",\n \"outputId\": \"6480b287-dc0e-4e79-feda-f5e4f41d2a82\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_name = 'AI-MO/NuminaMath-TIR'\\n\",\n \"train_dataset = load_dataset(dataset_name, split='train[:5%]')\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"P1AIokQrBEGw\"\n },\n \"source\": [\n \"Let's check the structure of the dataset\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"ff6Gx1TWWbdp\",\n \"outputId\": \"30d49bed-273a-47d9-d131-a677ca5a8b65\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Dataset({\\n\",\n \" features: ['problem', 'solution', 'messages'],\\n\",\n \" num_rows: 3622\\n\",\n \"})\\n\"\n ]\n }\n ],\n \"source\": [\n \"print(train_dataset)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"QY5hkOqDBGns\"\n },\n \"source\": [\n \"Let's check one sample:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"-y9c7i29Wbdp\",\n \"outputId\": \"760662ea-4db4-4b8e-c234-92ae2c8ecc17\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"{'problem': 'What is the coefficient of $x^2y^6$ in the expansion of $\\\\\\\\left(\\\\\\\\frac{3}{5}x-\\\\\\\\frac{y}{2}\\\\\\\\right)^8$? Express your answer as a common fraction.', 'solution': \\\"To determine the coefficient of \\\\\\\\(x^2y^6\\\\\\\\) in the expansion of \\\\\\\\(\\\\\\\\left(\\\\\\\\frac{3}{5}x - \\\\\\\\frac{y}{2}\\\\\\\\right)^8\\\\\\\\), we can use the binomial theorem.\\\\n\\\\nThe binomial theorem states:\\\\n\\\\\\\\[\\\\n(a + b)^n = \\\\\\\\sum_{k=0}^{n} \\\\\\\\binom{n}{k} a^{n-k} b^k\\\\n\\\\\\\\]\\\\n\\\\nIn this case, \\\\\\\\(a = \\\\\\\\frac{3}{5}x\\\\\\\\), \\\\\\\\(b = -\\\\\\\\frac{y}{2}\\\\\\\\), and \\\\\\\\(n = 8\\\\\\\\).\\\\n\\\\nWe are interested in the term that contains \\\\\\\\(x^2y^6\\\\\\\\). In the general term of the binomial expansion:\\\\n\\\\\\\\[\\\\n\\\\\\\\binom{8}{k} \\\\\\\\left(\\\\\\\\frac{3}{5}x\\\\\\\\right)^{8-k} \\\\\\\\left(-\\\\\\\\frac{y}{2}\\\\\\\\right)^k\\\\n\\\\\\\\]\\\\n\\\\nTo get \\\\\\\\(x^2\\\\\\\\), we need \\\\\\\\(8 - k = 2\\\\\\\\), thus \\\\\\\\(k = 6\\\\\\\\).\\\\n\\\\nSubstituting \\\\\\\\(k = 6\\\\\\\\) into the expression:\\\\n\\\\\\\\[\\\\n\\\\\\\\binom{8}{6} \\\\\\\\left(\\\\\\\\frac{3}{5}x\\\\\\\\right)^{8-6} \\\\\\\\left(-\\\\\\\\frac{y}{2}\\\\\\\\right)^6 = \\\\\\\\binom{8}{6} \\\\\\\\left(\\\\\\\\frac{3}{5}x\\\\\\\\right)^2 \\\\\\\\left(-\\\\\\\\frac{y}{2}\\\\\\\\right)^6\\\\n\\\\\\\\]\\\\n\\\\nNow, we will compute each part of this expression.\\\\n\\\\n1. Calculate the binomial coefficient \\\\\\\\(\\\\\\\\binom{8}{6}\\\\\\\\).\\\\n2. Compute \\\\\\\\(\\\\\\\\left(\\\\\\\\frac{3}{5}\\\\\\\\right)^2\\\\\\\\).\\\\n3. Compute \\\\\\\\(\\\\\\\\left(-\\\\\\\\frac{y}{2}\\\\\\\\right)^6\\\\\\\\).\\\\n4. Combine everything together to get the coefficient of \\\\\\\\(x^2y^6\\\\\\\\).\\\\n\\\\nLet's compute these in Python.\\\\n```python\\\\nfrom math import comb\\\\n\\\\n# Given values\\\\nn = 8\\\\nk = 6\\\\n\\\\n# Calculate the binomial coefficient\\\\nbinom_coeff = comb(n, k)\\\\n\\\\n# Compute (3/5)^2\\\\na_term = (3/5)**2\\\\n\\\\n# Compute (-1/2)^6\\\\nb_term = (-1/2)**6\\\\n\\\\n# Combine terms to get the coefficient of x^2y^6\\\\ncoefficient = binom_coeff * a_term * b_term\\\\nprint(coefficient)\\\\n```\\\\n```output\\\\n0.1575\\\\n```\\\\nThe coefficient of \\\\\\\\(x^2y^6\\\\\\\\) in the expansion of \\\\\\\\(\\\\\\\\left(\\\\\\\\frac{3}{5}x - \\\\\\\\frac{y}{2}\\\\\\\\right)^8\\\\\\\\) is \\\\\\\\(0.1575\\\\\\\\). To express this as a common fraction, we recognize that:\\\\n\\\\n\\\\\\\\[ 0.1575 = \\\\\\\\frac{1575}{10000} = \\\\\\\\frac{63}{400} \\\\\\\\]\\\\n\\\\nThus, the coefficient can be expressed as:\\\\n\\\\n\\\\\\\\[\\\\n\\\\\\\\boxed{\\\\\\\\frac{63}{400}}\\\\n\\\\\\\\]\\\", 'messages': [{'content': 'What is the coefficient of $x^2y^6$ in the expansion of $\\\\\\\\left(\\\\\\\\frac{3}{5}x-\\\\\\\\frac{y}{2}\\\\\\\\right)^8$? Express your answer as a common fraction.', 'role': 'user'}, {'content': \\\"To determine the coefficient of \\\\\\\\(x^2y^6\\\\\\\\) in the expansion of \\\\\\\\(\\\\\\\\left(\\\\\\\\frac{3}{5}x - \\\\\\\\frac{y}{2}\\\\\\\\right)^8\\\\\\\\), we can use the binomial theorem.\\\\n\\\\nThe binomial theorem states:\\\\n\\\\\\\\[\\\\n(a + b)^n = \\\\\\\\sum_{k=0}^{n} \\\\\\\\binom{n}{k} a^{n-k} b^k\\\\n\\\\\\\\]\\\\n\\\\nIn this case, \\\\\\\\(a = \\\\\\\\frac{3}{5}x\\\\\\\\), \\\\\\\\(b = -\\\\\\\\frac{y}{2}\\\\\\\\), and \\\\\\\\(n = 8\\\\\\\\).\\\\n\\\\nWe are interested in the term that contains \\\\\\\\(x^2y^6\\\\\\\\). In the general term of the binomial expansion:\\\\n\\\\\\\\[\\\\n\\\\\\\\binom{8}{k} \\\\\\\\left(\\\\\\\\frac{3}{5}x\\\\\\\\right)^{8-k} \\\\\\\\left(-\\\\\\\\frac{y}{2}\\\\\\\\right)^k\\\\n\\\\\\\\]\\\\n\\\\nTo get \\\\\\\\(x^2\\\\\\\\), we need \\\\\\\\(8 - k = 2\\\\\\\\), thus \\\\\\\\(k = 6\\\\\\\\).\\\\n\\\\nSubstituting \\\\\\\\(k = 6\\\\\\\\) into the expression:\\\\n\\\\\\\\[\\\\n\\\\\\\\binom{8}{6} \\\\\\\\left(\\\\\\\\frac{3}{5}x\\\\\\\\right)^{8-6} \\\\\\\\left(-\\\\\\\\frac{y}{2}\\\\\\\\right)^6 = \\\\\\\\binom{8}{6} \\\\\\\\left(\\\\\\\\frac{3}{5}x\\\\\\\\right)^2 \\\\\\\\left(-\\\\\\\\frac{y}{2}\\\\\\\\right)^6\\\\n\\\\\\\\]\\\\n\\\\nNow, we will compute each part of this expression.\\\\n\\\\n1. Calculate the binomial coefficient \\\\\\\\(\\\\\\\\binom{8}{6}\\\\\\\\).\\\\n2. Compute \\\\\\\\(\\\\\\\\left(\\\\\\\\frac{3}{5}\\\\\\\\right)^2\\\\\\\\).\\\\n3. Compute \\\\\\\\(\\\\\\\\left(-\\\\\\\\frac{y}{2}\\\\\\\\right)^6\\\\\\\\).\\\\n4. Combine everything together to get the coefficient of \\\\\\\\(x^2y^6\\\\\\\\).\\\\n\\\\nLet's compute these in Python.\\\\n```python\\\\nfrom math import comb\\\\n\\\\n# Given values\\\\nn = 8\\\\nk = 6\\\\n\\\\n# Calculate the binomial coefficient\\\\nbinom_coeff = comb(n, k)\\\\n\\\\n# Compute (3/5)^2\\\\na_term = (3/5)**2\\\\n\\\\n# Compute (-1/2)^6\\\\nb_term = (-1/2)**6\\\\n\\\\n# Combine terms to get the coefficient of x^2y^6\\\\ncoefficient = binom_coeff * a_term * b_term\\\\nprint(coefficient)\\\\n```\\\\n```output\\\\n0.1575\\\\n```\\\\nThe coefficient of \\\\\\\\(x^2y^6\\\\\\\\) in the expansion of \\\\\\\\(\\\\\\\\left(\\\\\\\\frac{3}{5}x - \\\\\\\\frac{y}{2}\\\\\\\\right)^8\\\\\\\\) is \\\\\\\\(0.1575\\\\\\\\). To express this as a common fraction, we recognize that:\\\\n\\\\n\\\\\\\\[ 0.1575 = \\\\\\\\frac{1575}{10000} = \\\\\\\\frac{63}{400} \\\\\\\\]\\\\n\\\\nThus, the coefficient can be expressed as:\\\\n\\\\n\\\\\\\\[\\\\n\\\\\\\\boxed{\\\\\\\\frac{63}{400}}\\\\n\\\\\\\\]\\\", 'role': 'assistant'}]}\\n\"\n ]\n }\n ],\n \"source\": [\n \"print(train_dataset[0])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"DiqBlxK_A0SD\"\n },\n \"source\": [\n \"We will adapt our dataset to a conversational format using a custom system prompt, guiding the LLM to generate both step-by-step reasoning and the final answer.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"RWxK5xFKWbdp\"\n },\n \"outputs\": [],\n \"source\": [\n \"SYSTEM_PROMPT = (\\n\",\n \" \\\"A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant \\\"\\n\",\n \" \\\"first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning \\\"\\n\",\n \" \\\"process is enclosed strictly within and tags. \\\"\\n\",\n \" \\\"After closing , the assistant MUST provide the final answer in plain text.\\\"\\n\",\n \")\\n\",\n \"\\n\",\n \"\\n\",\n \"def make_conversation(example):\\n\",\n \" return {\\n\",\n \" \\\"prompt\\\": [\\n\",\n \" {\\\"role\\\": \\\"system\\\", \\\"content\\\": SYSTEM_PROMPT},\\n\",\n \" {\\\"role\\\": \\\"user\\\", \\\"content\\\": example[\\\"problem\\\"]},\\n\",\n \" ],\\n\",\n \" }\\n\",\n \"\\n\",\n \"train_dataset = train_dataset.map(make_conversation)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"sND566XAC0kD\"\n },\n \"source\": [\n \"Let's take a look at an example:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"Q-kHUmpMWbdp\",\n \"outputId\": \"452beb3a-1091-46d4-997e-04b91562d66c\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"[{'content': 'A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process is enclosed strictly within and tags. After closing , the assistant MUST provide the final answer in plain text.', 'role': 'system'}, {'content': 'What is the coefficient of $x^2y^6$ in the expansion of $\\\\\\\\left(\\\\\\\\frac{3}{5}x-\\\\\\\\frac{y}{2}\\\\\\\\right)^8$? Express your answer as a common fraction.', 'role': 'user'}]\\n\"\n ]\n }\n ],\n \"source\": [\n \"print(train_dataset[0]['prompt'])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"bw0qcp-CC3G0\"\n },\n \"source\": [\n \"We'll remove the `messages` and `problem` columns, as we only need the custom `prompt` column and `solution` to verify the generated answer.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"SzbF3hdRWbdp\",\n \"outputId\": \"bd59a383-1d4e-4020-c232-79ce66073fd1\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Dataset({\\n\",\n \" features: ['solution', 'prompt'],\\n\",\n \" num_rows: 3622\\n\",\n \"})\\n\"\n ]\n }\n ],\n \"source\": [\n \"train_dataset = train_dataset.remove_columns(['messages', 'problem'])\\n\",\n \"print(train_dataset)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"tvs5rjQBr7af\"\n },\n \"source\": [\n \"## Load model and configure LoRA/QLoRA\\n\",\n \"\\n\",\n \"Below, choose your **preferred model**. All of the options have been tested on **free Colab instances**.\\n\",\n \"\\n\",\n \"> ๐Ÿ’ก Note: Some models, such as Qwen2.5 and Qwen3, are known to have been pretrained on data that improves their math performance. Be cautious when selecting the appropriate model for training to ensure meaningful fine-tuning results ([source](https://thinkingmachines.ai/blog/lora/)).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"7_uaW3JfWbdp\"\n },\n \"outputs\": [],\n \"source\": [\n \"# Select one model below by uncommenting the line you want to use ๐Ÿ‘‡\\n\",\n \"## Qwen\\n\",\n \"model_id, output_dir = \\\"Qwen/Qwen2-7B-Instruct\\\", \\\"t4-Qwen2-7B-Instruct-GRPO\\\" # โœ… ~9.2GB VRAM\\n\",\n \"# model_id, output_dir = \\\"unsloth/qwen3-14b-unsloth-bnb-4bit\\\", \\\"qwen3-14b-unsloth-bnb-4bit-GRPO\\\" # โš ๏ธ OOM with this config; fits if GRPO params are reduced\\n\",\n \"# model_id, output_dir = \\\"Qwen/Qwen3-8B\\\", \\\"Qwen3-8B-GRPO\\\" # โœ… ~9.9GB VRAM\\n\",\n \"# model_id, output_dir = \\\"Qwen/Qwen2.5-7B-Instruct\\\", \\\"Qwen2.5-7B-Instruct-GRPO\\\" # โœ… ~9.2GB VRAM\\n\",\n \"\\n\",\n \"## Llama\\n\",\n \"# model_id, output_dir = \\\"meta-llama/Llama-3.2-3B-Instruct\\\", \\\"Llama-3.2-3B-Instruct-GRPO\\\" # โœ… ~5.7GB VRAM\\n\",\n \"# model_id, output_dir = \\\"meta-llama/Llama-3.1-8B-Instruct\\\", \\\"Llama-3.1-8B-Instruct-GRPO\\\" # โœ… ~9.5GB VRAM\\n\",\n \"\\n\",\n \"## LFM2.5\\n\",\n \"# model_id, output_dir = \\\"LiquidAI/LFM2.5-1.2B-Instruct\\\", \\\"LFM2.5-1.2B-Instruct-GRPO\\\" # โœ… ~1.12 GB VRAM\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"aw__94OWDnER\"\n },\n \"source\": [\n \"This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration (training without quantization consumes more memory).\\n\",\n \"\\n\",\n \"Let's load the selected model using `transformers`, configuring QLoRA via `bitsandbytes` (you can remove it if doing LoRA). We don't need to configure the tokenizer since the trainer takes care of that automatically.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"1130e5a744864ca5b5873731e4764983\"\n ]\n },\n \"id\": \"o86TnTchWbdp\",\n \"outputId\": \"77a7e6c8-0360-40f1-eea7-b941be031366\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"1130e5a744864ca5b5873731e4764983\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Loading checkpoint shards: 0%| | 0/4 [00:00` format:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"def think_format_reward(completions: list[list[dict[str, str]]], **kwargs) -> list[float]:\\n\",\n \" pattern = r\\\"^(?!.*)(.*?).*$\\\"\\n\",\n \" completion_contents = [completion[0][\\\"content\\\"] for completion in completions]\\n\",\n \" matches = [re.match(pattern, content, re.DOTALL | re.MULTILINE) for content in completion_contents]\\n\",\n \" return [1.0 if match else 0.0 for match in matches]\\n\",\n \"```\\n\",\n \"\\n\",\n \"In this notebook, we will use both `think_format_reward`, which rewards completions that correctly follow the `` format, and `reasoning_accuracy_reward`, which evaluates the correctness of the model's solution to the mathematical problem. Together, these rewards guide the model to generate **structured reasoning** while producing **accurate answers**.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"lj42Qs5vWbdp\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl.rewards import think_format_reward, reasoning_accuracy_reward\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"bFgYgxMbtbEZ\"\n },\n \"source\": [\n \"We'll configure **GRPO** using `GRPOConfig`, keeping the parameters minimal so that the training can run on a free Colab instance. You can adjust these settings if you have access to more resources. For a complete list of available parameters and their descriptions, refer to the [TRL GRPOConfig documentation](https://huggingface.co/docs/trl/grpo_trainer#trl.GRPOConfig).\\n\",\n \"\\n\",\n \"> ๐Ÿ’ก Note: TRL supports using **vLLM** for generation during GRPO training, which can significantly speed up training. However, it increases VRAM usage since a separate vLLM process is active to handle generation. In this notebook, we do not enable vLLM because we are using **QLoRA**, which updates the quantized vLLM model weights at every step. Enabling vLLM in this setup can cause weight precision issues and make convergence more challenging. The configuration includes the vLLM parameters in case you want to experiment with it. Learn more about vLLM integration in TRL [here](https://huggingface.co/docs/trl/main/en/vllm_integration).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"JY11EQMhWbdp\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOConfig\\n\",\n \"\\n\",\n \"# Configure training arguments using GRPOConfig\\n\",\n \"training_args = GRPOConfig(\\n\",\n \" # Training schedule / optimization\\n\",\n \" learning_rate=2e-5, # Learning rate for the optimizer\\n\",\n \" #num_train_epochs=1,\\n\",\n \" max_steps=500, # Number of dataset passes. For full trainings, use `num_train_epochs` instead\\n\",\n \"\\n\",\n \" # Parameters that control GRPO training (you can adapt them)\\n\",\n \" per_device_train_batch_size = 8,\\n\",\n \" max_completion_length=256, # default: 256 # Max completion length produced during training\\n\",\n \" num_generations=8, # default: 8 # Number of generations produced during trainig for comparison\\n\",\n \"\\n\",\n \" # Optimizations\\n\",\n \" optim = \\\"paged_adamw_8bit\\\", # Optimizer\\n\",\n \" use_liger_kernel=True, # Enable Liger kernel optimizations for faster training\\n\",\n \"\\n\",\n \" # Parameters related to reporting and saving\\n\",\n \" output_dir=output_dir, # Where to save model checkpoints and logs\\n\",\n \" logging_steps=10, # Log training metrics every N steps\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool\\n\",\n \" trackio_space_id=output_dir, # HF Space where the experiment tracking will be saved\\n\",\n \" log_completions=False, # Return model completions during training\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub=True, # Automatically push the trained model to the Hugging Face Hub\\n\",\n \" # The model will be saved under your Hub account in the repository named `output_dir`\\n\",\n \" # vLLM params\\n\",\n \" #use_vllm=False, # Activate vLLM training for faster training\\n\",\n \" #vllm_mode='colocate',\\n\",\n \" #vllm_gpu_memory_utilization=0.1,\\n\",\n \" #vllm_enable_sleep_mode=True\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"-9LlOAvWFSor\"\n },\n \"source\": [\n \"Configure the `GRPOTrainer` by passing the previously defined `training_args`. To keep memory usage low, we are not using an evaluation dataset, but you can include one if desired. We also provide the reward functions that were imported earlier to guide the training process.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"iI_E9KCUWbdq\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOTrainer\\n\",\n \"\\n\",\n \"trainer = GRPOTrainer(\\n\",\n \" model=model,\\n\",\n \" reward_funcs=[think_format_reward, reasoning_accuracy_reward],\\n\",\n \" args=training_args,\\n\",\n \" train_dataset=train_dataset,\\n\",\n \" peft_config=peft_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"8dY7bK8FGLhh\"\n },\n \"source\": [\n \"Show memory stats before training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"PEVRGlrAWbdq\",\n \"outputId\": \"78fac9e4-4ae6-4836-bd10-c30b39059782\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"GPU = Tesla T4. Max memory = 14.741 GB.\\n\",\n \"6.773 GB of memory reserved.\\n\"\n ]\n }\n ],\n \"source\": [\n \"gpu_stats = torch.cuda.get_device_properties(0)\\n\",\n \"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\\n\",\n \"\\n\",\n \"print(f\\\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\\\")\\n\",\n \"print(f\\\"{start_gpu_memory} GB of memory reserved.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"z-5xPtfIGQL5\"\n },\n \"source\": [\n \"And train!\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Training on a T4 in Colab with the configuration defined in this notebook takes around 13 hours. If you're just experimenting, you can try the following quicker task ([source](https://huggingface.co/learn/llm-course/en/chapter12/5)):\\n\",\n \"\\n\",\n \"```python\\n\",\n \"dataset = load_dataset(\\\"mlabonne/smoltldr\\\")\\n\",\n \"\\n\",\n \"# Reward function\\n\",\n \"ideal_length = 50\\n\",\n \"\\n\",\n \"def reward_len(completions, **kwargs):\\n\",\n \" return [-abs(ideal_length - len(completion)) for completion in completions]\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"zl7-PmoXWbdq\",\n \"outputId\": \"f39c8c3c-43c2-4f2d-c98d-4c595ae1129f\"\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.\\n\"\n ]\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Trackio project initialized: huggingface\\n\",\n \"* Trackio metrics will be synced to Hugging Face Dataset: sergiopaniego/t4-Qwen2-7B-Instruct-GRPO-dataset\\n\",\n \"* Creating new space: https://huggingface.co/spaces/sergiopaniego/t4-Qwen2-7B-Instruct-GRPO\\n\",\n \"* View dashboard by going to: https://sergiopaniego-t4-Qwen2-7B-Instruct-GRPO.hf.space/\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Created new run: sergiopaniego-1766143600\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" [500/500 13:05:04, Epoch 0/1]\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \"
StepTraining Loss
100.027900
20-0.011600
300.021500
400.033400
500.039400
600.010300
700.048200
800.067300
900.030600
1000.064000
1100.021500
1200.021400
1300.000000
140-0.028500
150-0.003100
1600.017300
170-0.024700
1800.003300
1900.000000
200-0.001400
2100.008000
2200.034300
2300.044600
2400.016400
250-0.015200
2600.016800
2700.042900
2800.031300
2900.006200
3000.043300
3100.029700
3200.001100
3300.027000
340-0.006700
3500.027200
3600.008200
370-0.015800
3800.007200
3900.012100
4000.000000
4100.010500
4200.019800
4300.000800
4400.003400
450-0.007900
460-0.011800
470-0.016300
480-0.002300
490-0.005500
5000.038000

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Run finished. Uploading logs to Trackio (please wait...)\\n\"\n ]\n }\n ],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"iqAN-XLCGTGW\"\n },\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"4BeEwp5EWbds\",\n \"outputId\": \"668b8a2c-2eef-4e34-8d4a-2a43ccbbdc00\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"47228.679 seconds used for training.\\n\",\n \"787.14 minutes used for training.\\n\",\n \"Peak reserved memory = 8.832 GB.\\n\",\n \"Peak reserved memory for training = 2.059 GB.\\n\",\n \"Peak reserved memory % of max memory = 59.915 %.\\n\",\n \"Peak reserved memory for training % of max memory = 13.968 %.\\n\"\n ]\n }\n ],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"R8Sd_AqILeYi\"\n },\n \"source\": [\n \"The training procedure generates both standard training logs and **trackio** logs, which help us monitor the training progress. Example outputs would look like the following:\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"2bPn6gruLf-n\"\n },\n \"source\": [\n \"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"ibO4f7tuLboQ\"\n },\n \"source\": [\n \"## Saving fine tuned model\\n\",\n \"\\n\",\n \"In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"e6a3677667ce47bcba55e3e950e446f9\",\n \"17adb84604d84cf688a89a21f6cc6150\",\n \"a21c1bbd3cd04738a8c96fbfc0c016c6\",\n \"65cadde3da7642188f029bb2aceaa7c6\",\n \"0404b89e5ce24e76958c72bedc1a95cc\",\n \"c52baf990fde40c0873747e827dc6926\",\n \"191653e8ce184123a68f26fbf2b78745\",\n \"0bb882d400864b249c80132264de2623\",\n \"09cbfcf6e51c431798f4e392a81be6d3\",\n \"d6521f73f23f42e18ee462a547f251a1\"\n ]\n },\n \"id\": \"itpVDjy0Wbdt\",\n \"outputId\": \"b821c7ed-6c9d-440a-a797-e25291627bef\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub(dataset_name=dataset_name)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"81eBZe-X7daz\"\n },\n \"source\": [\n \"## Load the fine-tuned model and run inference\\n\",\n \"\\n\",\n \"Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"1d3fbf86d53845beac599c5b231e87ea\"\n ]\n },\n \"id\": \"ZLdaWYzNWbdt\",\n \"outputId\": \"a103b64b-1f6b-4423-c5fd-402f210e6dc3\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"1d3fbf86d53845beac599c5b231e87ea\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Loading checkpoint shards: 0%| | 0/4 [00:00 and tags. After closing , the assistant MUST provide the final answer in plain text.',\\n\",\n \" 'role': 'system'},\\n\",\n \" {'content': \\\"In 1988, a person's age was equal to the sum of the digits of their birth year. How old was this person?\\\",\\n\",\n \" 'role': 'user'}]\"\n ]\n },\n \"execution_count\": 5,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_name = 'AI-MO/NuminaMath-TIR'\\n\",\n \"test_dataset = load_dataset(dataset_name, split='test[:1%]')\\n\",\n \"test_dataset = test_dataset.map(make_conversation)\\n\",\n \"test_dataset = test_dataset.remove_columns(['messages', 'problem'])\\n\",\n \"test_dataset[0]['prompt']\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"CxKyZwG28BYJ\"\n },\n \"source\": [\n \"Let's first check what's the output for the base model, without the adapter.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"qTPJY96eWbdt\",\n \"outputId\": \"ed02acca-e856-44ec-fa20-c32efd81e018\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"To solve this problem, let's denote the birth year of the person as \\\\(Y\\\\) (where \\\\(Y\\\\) is a four-digit number) and their age in 1988 as \\\\(A\\\\). According to the given condition, their age in 1988 is equal to the sum of the digits of their birth year. \\n\",\n \"\\n\",\n \"Since we're looking at the year 1988, the person would be \\\\(1988 - Y\\\\) years old in that year. Given the condition:\\n\",\n \"\\n\",\n \"\\\\[1988 - Y = \\\\text{sum of the digits of } Y\\\\]\\n\",\n \"\\n\",\n \"Let's break down the possible range for \\\\(Y\\\\). Since the person's age must be less than or equal to 100 (as the sum of the digits of any four-digit number cannot exceed 36), \\\\(Y\\\\) must be between 1989 and 2088.\\n\",\n \"\\n\",\n \"We can systematically check each year in this range to find when the condition holds true. However, considering the constraint on age, we can narrow our search significantly. For example, if \\\\(Y\\\\) were 1990, the sum of its digits would be 18, which is not a reasonable age. We need\\n\"\n ]\n }\n ],\n \"source\": [\n \"messages = test_dataset[0]['prompt']\\n\",\n \"text = tokenizer.apply_chat_template(\\n\",\n \" messages, add_generation_prompt=True, tokenize=False\\n\",\n \")\\n\",\n \"model_inputs = tokenizer([text], return_tensors=\\\"pt\\\").to(base_model.device)\\n\",\n \"\\n\",\n \"generated_ids = base_model.generate(\\n\",\n \" **model_inputs,\\n\",\n \" max_new_tokens=256\\n\",\n \")\\n\",\n \"output_ids = generated_ids[0][len(model_inputs.input_ids[0]):]\\n\",\n \"\\n\",\n \"# Decode and extract model response\\n\",\n \"generated_text = tokenizer.decode(output_ids, skip_special_tokens=True)\\n\",\n \"print(generated_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"V9eoUwQS8SIi\"\n },\n \"source\": [\n \"The base model neither produced reasoning traces nor provided a correct answer. Let's now load the fine-tuned model and check its performance.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"073b351afd264bf0bf23043b37e0d8ce\",\n \"3dee429faf4e40b192cabebfe4bf2245\"\n ]\n },\n \"id\": \"CNannsXXWbdt\",\n \"outputId\": \"fc43a5b9-4ec6-43eb-fc34-f26e92434faf\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"073b351afd264bf0bf23043b37e0d8ce\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"adapter_config.json: 0.00B [00:00, ?B/s]\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"3dee429faf4e40b192cabebfe4bf2245\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"adapter_model.safetensors: 0%| | 0.00/162M [00:00 I need to find a birth year where the sum of its digits equals the person's age in 1988 \\n\",\n \"\\n\",\n \"The person would have been born in 1979, since 1+9+7+9 = 26 and 26 is the age in 1988\\n\",\n \"\\n\",\n \"answer: 26\\n\"\n ]\n }\n ],\n \"source\": [\n \"text = tokenizer.apply_chat_template(\\n\",\n \" messages, add_generation_prompt=True, tokenize=False\\n\",\n \")\\n\",\n \"model_inputs = tokenizer([text], return_tensors=\\\"pt\\\").to(fine_tuned_model.device)\\n\",\n \"\\n\",\n \"generated_ids = fine_tuned_model.generate(\\n\",\n \" **model_inputs,\\n\",\n \" max_new_tokens=256\\n\",\n \")\\n\",\n \"output_ids = generated_ids[0][len(model_inputs.input_ids[0]):]\\n\",\n \"\\n\",\n \"# Decode and extract model response\\n\",\n \"generated_text = tokenizer.decode(output_ids, skip_special_tokens=True)\\n\",\n \"print(generated_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"OU-xDHpEEmg9\"\n },\n \"source\": [\n \"The final answer is correct!\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"XNtBOpRY8a2O\"\n },\n \"source\": [\n \"## Inference and Serving with vLLM\\n\",\n \"\\n\",\n \"You can use Transformer models with **vLLM** to serve them in real-world applications. Learn more [here](https://blog.vllm.ai/2025/04/11/transformers-backend.html).\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"nkhu0uY78lV3\"\n },\n \"source\": [\n \"### Push Merged Model (for LoRA or QLoRA Training)\\n\",\n \"\\n\",\n \"To serve the model via **vLLM**, the repository must contain the merged model (base model + LoRA adapter). Therefore, you need to upload it first.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"NF8ZP9Z-Wbdt\",\n \"outputId\": \"32a5ab71-1f0d-4289-ea12-66f5f75a957b\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"('Qwen2-7B-Instruct-GRPO-merged/tokenizer_config.json',\\n\",\n \" 'Qwen2-7B-Instruct-GRPO-merged/special_tokens_map.json',\\n\",\n \" 'Qwen2-7B-Instruct-GRPO-merged/chat_template.jinja',\\n\",\n \" 'Qwen2-7B-Instruct-GRPO-merged/vocab.json',\\n\",\n \" 'Qwen2-7B-Instruct-GRPO-merged/merges.txt',\\n\",\n \" 'Qwen2-7B-Instruct-GRPO-merged/added_tokens.json',\\n\",\n \" 'Qwen2-7B-Instruct-GRPO-merged/tokenizer.json')\"\n ]\n },\n \"execution_count\": 29,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"model_merged = fine_tuned_model.merge_and_unload()\\n\",\n \"\\n\",\n \"save_dir = f\\\"{output_dir}-merged\\\"\\n\",\n \"\\n\",\n \"model_merged.save_pretrained(save_dir)\\n\",\n \"tokenizer.save_pretrained(save_dir)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"d1a0574cc20046d5876cf31b21955f8b\",\n \"7cc2f0ef7ad2494cad572cd898095c00\",\n \"475420d92bb54dc08517ffe423b015c3\",\n \"a76231aeae5a49979d1e9075b0b3eefb\",\n \"b4f469f957134ea9b0e28532fe3caaf1\",\n \"637e55736da34f2c9b098222ae07244a\",\n \"8157e521017c450a9d2a9e41611405e9\",\n \"9746ae4ab0574ed186f898dba3b4b197\",\n \"d4b2a8805ec548ea85e0900ff5927574\",\n \"0668cd8597f141e89ef38129c6641c1f\"\n ]\n },\n \"id\": \"X5Zci39rWbdt\",\n \"outputId\": \"ca329f99-dc7b-470c-f5d9-39a3eabcb16d\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"d1a0574cc20046d5876cf31b21955f8b\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"7cc2f0ef7ad2494cad572cd898095c00\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"475420d92bb54dc08517ffe423b015c3\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...0002-of-00004.safetensors: 0%| | 612kB / 4.93GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"a76231aeae5a49979d1e9075b0b3eefb\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...0003-of-00004.safetensors: 0%| | 611kB / 4.33GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"b4f469f957134ea9b0e28532fe3caaf1\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...0001-of-00004.safetensors: 1%|1 | 50.3MB / 4.88GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"637e55736da34f2c9b098222ae07244a\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...0004-of-00004.safetensors: 4%|3 | 41.9MB / 1.09GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"8157e521017c450a9d2a9e41611405e9\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"README.md: 0.00B [00:00, ?B/s]\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"9746ae4ab0574ed186f898dba3b4b197\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"d4b2a8805ec548ea85e0900ff5927574\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"0668cd8597f141e89ef38129c6641c1f\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...RPO-merged/tokenizer.json: 100%|##########| 11.4MB / 11.4MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.google.colaboratory.intrinsic+json\": {\n \"type\": \"string\"\n },\n \"text/plain\": [\n \"CommitInfo(commit_url='https://huggingface.co/sergiopaniego/Qwen2-7B-Instruct-GRPO-merged/commit/b20988444532e79a6915f0b2b6002b5acc2b53e1', commit_message='Upload tokenizer', commit_description='', oid='b20988444532e79a6915f0b2b6002b5acc2b53e1', pr_url=None, repo_url=RepoUrl('https://huggingface.co/sergiopaniego/Qwen2-7B-Instruct-GRPO-merged', endpoint='https://huggingface.co', repo_type='model', repo_id='sergiopaniego/Qwen2-7B-Instruct-GRPO-merged'), pr_revision=None, pr_num=None)\"\n ]\n },\n \"execution_count\": 30,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"model_merged.push_to_hub(f\\\"sergiopaniego/{output_dir}-merged\\\") # Replace with your HF username or organization\\n\",\n \"tokenizer.push_to_hub(f\\\"sergiopaniego/{output_dir}-merged\\\") # Replace with your HF username or organization\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"DQ00Ivxi8rFu\"\n },\n \"source\": [\n \"### Performing Inference with vLLM\\n\",\n \"\\n\",\n \"Use **vLLM** to run your model and generate text efficiently in real-time. This allows you to test and deploy your fine-tuned models with low latency and high throughput.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"x7L-HIn4Wbdt\",\n \"outputId\": \"afd66093-3525-4590-f834-c0b373e7bb9e\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"INFO 12-11 15:56:09 [utils.py:253] non-default args: {'dtype': torch.float16, 'max_model_len': 256, 'disable_log_stats': True, 'model_impl': 'transformers', 'model': 'sergiopaniego/Qwen2-7B-Instruct-GRPO-merged'}\\n\"\n ]\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:104: UserWarning: \\n\",\n \"Error while fetching `HF_TOKEN` secret value from your vault: 'Requesting secret HF_TOKEN timed out. Secrets can only be fetched when running from the Colab UI.'.\\n\",\n \"You are not authenticated with the Hugging Face Hub in this notebook.\\n\",\n \"If the error persists, please let us know by opening an issue on GitHub (https://github.com/huggingface/huggingface_hub/issues/new).\\n\",\n \" warnings.warn(\\n\"\n ]\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"INFO 12-11 15:56:37 [model.py:631] Resolved architecture: TransformersForCausalLM\\n\",\n \"WARNING 12-11 15:56:37 [model.py:1971] Casting torch.bfloat16 to torch.float16.\\n\",\n \"INFO 12-11 15:56:37 [model.py:1745] Using max model len 256\\n\",\n \"INFO 12-11 15:56:40 [scheduler.py:216] Chunked prefill is enabled with max_num_batched_tokens=8192.\\n\",\n \"WARNING 12-11 15:56:43 [system_utils.py:103] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized\\n\",\n \"INFO 12-11 15:57:36 [llm.py:352] Supported tasks: ['generate']\\n\"\n ]\n }\n ],\n \"source\": [\n \"from vllm import LLM, SamplingParams\\n\",\n \"from transformers import AutoTokenizer\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"llm = LLM(\\n\",\n \" model=f\\\"sergiopaniego/{output_dir}-merged\\\", # Replace with your HF username or organization\\n\",\n \" model_impl=\\\"transformers\\\", # Select the transformers model implementation\\n\",\n \" max_model_len=256, # Reduced for efficiency\\n\",\n \" dtype=torch.float16\\n\",\n \")\\n\",\n \"hf_tokenizer = AutoTokenizer.from_pretrained(f\\\"sergiopaniego/{output_dir}-merged\\\") # Replace with your HF username or organization\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"f0a4f4fb17bf4a698503212296467547\",\n \"5be7348f3f324b5b9397c9ad186fb35d\"\n ]\n },\n \"id\": \"ZTpSUqxNWbdt\",\n \"outputId\": \"6a9283bf-d3b7-4e54-c775-4502694b5c6d\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"f0a4f4fb17bf4a698503212296467547\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Adding requests: 0%| | 0/1 [00:00 1988 birth year implies the person was born either in 1979, 1980, 1981, etc. Looking for the one where sum of digits equals age \\n\",\n \"\\n\",\n \"The birth year 1979 gives sum of digits 1+9+7+9 = 26\\n\",\n \"\\n\",\n \"The person was 26 years old in 1988.\\n\",\n \"\\n\",\n \"Answer: The person was 26 years old.\\n\"\n ]\n }\n ],\n \"source\": [\n \"messages = test_dataset[0]['prompt']\\n\",\n \"# Alternatively, use llm.chat()\\n\",\n \"prompt = hf_tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)\\n\",\n \"\\n\",\n \"outputs = llm.generate(\\n\",\n \" {\\\"prompt\\\": prompt},\\n\",\n \" sampling_params=SamplingParams(max_tokens=256),\\n\",\n \")\\n\",\n \"\\n\",\n \"for o in outputs:\\n\",\n \" generated_text = o.outputs[0].text\\n\",\n \" print(generated_text)\"\n ]\n }\n ],\n \"metadata\": {\n \"accelerator\": \"GPU\",\n \"colab\": {\n \"gpuType\": \"T4\",\n \"provenance\": []\n },\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 0\n}\n" }, { "path": "examples/notebooks/openenv_sudoku_grpo.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"lSR2nwdJg962\"\n },\n \"source\": [\n \"# OpenEnv Sudoku with GRPO using TRL\\n\",\n \"\\n\",\n \"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/openenv_sudoku_grpo.ipynb)\\n\",\n \"\\n\",\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\\n\",\n \"\\n\",\n \"With [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl), you can train a model that learns to **play Sudoku**, through interaction and reinforcement.\\n\",\n \"\\n\",\n \"- [TRL GitHub Repository](https://github.com/huggingface/trl) โ€” star us to support the project! \\n\",\n \"- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview) \\n\",\n \"- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\\n\",\n \"- [OpenEnv](https://github.com/meta-pytorch/OpenEnv)\\n\",\n \"\\n\",\n \"An **agentic environment** is a setting where a model can take actions, observe outcomes, and adjust its behavior based on feedback, similar to how humans learn from trial and error.\\n\",\n \"In this case, the agent interacts with the **Sudoku** environment through the [**OpenEnv**](https://github.com/meta-pytorch/OpenEnv) framework, which standardizes multi-agent and RL-style text environments.\\n\",\n \"\\n\",\n \"Sudoku is a classic logic-based puzzle where the objective is to fill a **9ร—9 grid** so that. Each **row**, **column**, and **3ร—3 subgrid** contains all digits from **1 to 9** exactly once.\\n\",\n \"\\n\",\n \"This structured yet challenging setup makes Sudoku an excellent benchmark for reasoning and decision-making tasks.\\n\",\n \"\\n\",\n \"We'll fine-tune a model using **GRPO** (Group Relative Policy Optimization) via TRL.\\n\",\n \"The training loop follows these steps:\\n\",\n \"\\n\",\n \"1. The agent **generates guesses** based on the current game state.\\n\",\n \"2. The environment **evaluates the guess** and returns structured feedback.\\n\",\n \"3. The agent **updates its policy** using reward signals to improve future decisions.\\n\",\n \"\\n\",\n \"Over time, the model learns to make increasingly valid and efficient Sudoku moves.\\n\",\n \"\\n\",\n \"## Install dependencies\\n\",\n \"\\n\",\n \"We'll start by installing **TRL**, which automatically includes the main dependencies like **Transformers**. \\n\",\n \"We'll also install the **OpenEnv** framework (for the environment) via the HF Space we will use as environment server ([openenv/sudoku](https://huggingface.co/spaces/openenv/sudoku)), **trackio** (for logging and monitoring training runs), and **vLLM** (for efficient generation).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"mHmE7GhRKyJj\"\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -Uq trl[vllm] trackio git+https://huggingface.co/spaces/openenv/sudoku liger-kernel\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Inxeq6ZGpRno\"\n },\n \"source\": [\n \"### Log in to Hugging Face\\n\",\n \"\\n\",\n \"Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"JRd5fGR-KyJk\"\n },\n \"outputs\": [],\n \"source\": [\n \"from huggingface_hub import notebook_login\\n\",\n \"\\n\",\n \"notebook_login()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"O3kr38TGm_hb\"\n },\n \"source\": [\n \"## Initialize the OpenEnv TextArena Environment\\n\",\n \"\\n\",\n \"Let's begin by setting up the environment that will be used throughout training.\\n\",\n \"\\n\",\n \"For this example, we will use the **TextArena** environment provided by **OpenEnv**, which exposes a familiar **Gymnasium-style API** (`reset()`, `step()`, etc.) to simplify interaction and integration with reinforcement learning pipelines.\\n\",\n \"\\n\",\n \"Specifically, we will connect to a **remote TextArena instance** that hosts a **Sudoku environment**, available at [openenv/sudoku](https://huggingface.co/spaces/openenv/sudoku).\\n\",\n \"\\n\",\n \"This setup allows us to interact with the environment without needing to run the backend locally.\\n\",\n \"\\n\",\n \"> โš ๏ธ **Note:** Hosted environments on the Hugging Face Hub have limited concurrency. \\n\",\n \"> For improved stability, higher throughput, or parallel experiments, it is recommended to **duplicate the Space into your own account**.\\n\",\n \"\\n\",\n \"For more information, refer to the [TRL-OpenEnv documentation](https://huggingface.co/docs/trl/main/en/openenv).\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"P6O03louKyJk\"\n },\n \"outputs\": [],\n \"source\": [\n \"from textarena_env import TextArenaEnv\\n\",\n \"\\n\",\n \"space_url = \\\"https://openenv-sudoku.hf.space\\\"\\n\",\n \"client = TextArenaEnv(base_url=space_url)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"EqfDavDQnD_5\"\n },\n \"source\": [\n \"## Create Rollout Function with Helpers\\n\",\n \"\\n\",\n \"The **rollout function** defines how the agent interacts with the environment during GRPO training.\\n\",\n \"It is responsible for generating model outputs, collecting feedback (rewards), and returning all the information needed for policy optimization.\\n\",\n \"\\n\",\n \"In this setup:\\n\",\n \"- The function is called automatically by the **GRPOTrainer** at each training step.\\n\",\n \"- It uses the trainer's `generate_rollout_completions()` method for efficient generation with **vLLM** in colocate mode.\\n\",\n \"- Each rollout represents a full interaction loop: the model makes guesses, receives feedback from the Sudoku environment, and updates its policy based on reward signals.\\n\",\n \"\\n\",\n \"Rewards track different aspects of the agent's performance, while helper functions like `rollout_once` handle a single episode of interaction, keeping the main `rollout_func` clean and modular.\\n\",\n \"\\n\",\n \"This modular approach allows GRPO to efficiently sample, evaluate, and improve the model's guessing strategy through reinforcement learning.\\n\",\n \"\\n\",\n \"First, we define the `system_prompt` that guides the model's behavior as an expert Sudoku solver with strategic reasoning and structured responses.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"pi1JGoUBKyJk\"\n },\n \"outputs\": [],\n \"source\": [\n \"# @title System prompt (click to expand)\\n\",\n \"SYSTEM_PROMPT = \\\"\\\"\\\"You are an expert Sudoku player with deep knowledge of logical deduction strategies and number placement techniques.\\n\",\n \"\\n\",\n \"## GAME RULES\\n\",\n \"\\n\",\n \"1. The puzzle is a 9x9 grid divided into nine 3x3 subgrids (boxes)\\n\",\n \"2. Some cells are pre-filled with numbers 1-9\\n\",\n \"3. You must fill in the empty cells (shown as '.') with numbers 1-9\\n\",\n \"4. Each row must contain numbers 1-9 without repetition\\n\",\n \"5. Each column must contain numbers 1-9 without repetition\\n\",\n \"6. Each 3x3 subgrid must contain numbers 1-9 without repetition\\n\",\n \"7. You cannot overwrite pre-filled cells\\n\",\n \"8. Invalid moves result in penalties (-1 reward)\\n\",\n \"\\n\",\n \"## RESPONSE FORMAT\\n\",\n \"\\n\",\n \"**CRITICAL: Output ONLY the move, nothing else. No text, no explanation.**\\n\",\n \"\\n\",\n \"Format: [row col number]\\n\",\n \"\\n\",\n \"Examples:\\n\",\n \"- [5 3 7] โ†’ places 7 in row 5, column 3\\n\",\n \"- [1 2 4] โ†’ places 4 in row 1, column 2\\n\",\n \"\\n\",\n \"## STRATEGIC APPROACH\\n\",\n \"\\n\",\n \"Do not repeat the same move twice.\\n\",\n \"\\n\",\n \"### Basic Strategies\\n\",\n \"- **Naked Singles**: If a cell has only one possible candidate, fill it in immediately.\\n\",\n \"- **Hidden Singles**: If a number can only go in one cell within a row, column, or box, place it there.\\n\",\n \"- **Scanning**: Look at each row, column, and box to find where specific numbers can go.\\n\",\n \"\\n\",\n \"### Intermediate Strategies\\n\",\n \"- **Naked Pairs/Triples**: When two/three cells in a unit contain only the same candidates, eliminate those from other cells.\\n\",\n \"- **Hidden Pairs/Triples**: When numbers only appear in specific cells within a unit, those cells can only contain those numbers.\\n\",\n \"- **Pointing Pairs**: When a candidate in a box is restricted to a single row/column, eliminate it elsewhere.\\n\",\n \"\\n\",\n \"### Solving Process\\n\",\n \"1. Start by scanning the entire grid to identify easy fills (cells with few candidates)\\n\",\n \"2. Look for rows, columns, or boxes with many numbers already placed\\n\",\n \"3. Fill all naked singles first\\n\",\n \"4. Then look for hidden singles in each row, column, and box\\n\",\n \"5. Apply more advanced techniques as needed\\n\",\n \"\\n\",\n \"### Common Pitfalls to Avoid\\n\",\n \"- Don't guess randomly - Sudoku is pure logic\\n\",\n \"- Don't overlook any constraint (row, column, or box)\\n\",\n \"- Don't try to overwrite pre-filled cells\\n\",\n \"- Don't place invalid numbers (must be 1-9)\\n\",\n \"- Don't use invalid coordinates (must be 1-9)\\n\",\n \"- Don't repeat a move that was already made\\n\",\n \"\\n\",\n \"## EXAMPLES\\n\",\n \"\\n\",\n \"### Example 1: Naked Single\\n\",\n \"If row 3, column 4 can only contain the number 5:\\n\",\n \"[3 4 5]\\n\",\n \"\\n\",\n \"### Example 2: Hidden Single\\n\",\n \"If the number 8 can only go in one cell in row 1:\\n\",\n \"[1 7 8]\\n\",\n \"\\n\",\n \"### Example 3: Row Analysis\\n\",\n \"Row 2 is missing only value 5, and column 8 is the empty cell:\\n\",\n \"[2 8 5]\\n\",\n \"\\n\",\n \"### Example 4: Box Analysis\\n\",\n \"In the center box, only one cell can contain 9:\\n\",\n \"[5 5 9]\\n\",\n \"\\n\",\n \"## BOARD READING\\n\",\n \"\\n\",\n \"The board is displayed as a 9x9 grid:\\n\",\n \"- Numbers 1-9 are pre-filled or already placed\\n\",\n \"- Empty cells are shown as '.'\\n\",\n \"- Rows are labeled R1-R9 (top to bottom)\\n\",\n \"- Columns are labeled C1-C9 (left to right)\\n\",\n \"\\n\",\n \"Example board representation:\\n\",\n \"```\\n\",\n \" C1 C2 C3 C4 C5 C6 C7 C8 C9\\n\",\n \"R1 . 8 9 | 1 . . | . 3 7\\n\",\n \"R2 2 7 1 | 9 4 3 | 6 . 8\\n\",\n \"R3 . 6 5 | . 2 7 | 4 9 .\\n\",\n \" - - - - - - - - - - - - - - - -\\n\",\n \"R4 . . . | 7 8 . | 9 2 3\\n\",\n \"R5 . 9 2 | . 5 6 | . . 4\\n\",\n \"R6 7 3 8 | . . 2 | 1 . .\\n\",\n \" - - - - - - - - - - - - - - - -\\n\",\n \"R7 8 4 . | . . 9 | 5 . .\\n\",\n \"R8 5 . . | 6 . 8 | 3 4 9\\n\",\n \"R9 9 . 6 | 5 3 4 | 8 7 2\\n\",\n \"```\\n\",\n \"\\n\",\n \"## COORDINATE REFERENCE\\n\",\n \"\\n\",\n \"Row indices (top to bottom): 1, 2, 3, 4, 5, 6, 7, 8, 9\\n\",\n \"Column indices (left to right): 1, 2, 3, 4, 5, 6, 7, 8, 9\\n\",\n \"\\n\",\n \"Subgrid layout:\\n\",\n \"```\\n\",\n \"Subgrid 1 | Subgrid 2 | Subgrid 3\\n\",\n \" (R1-R3) (R1-R3) (R1-R3)\\n\",\n \" (C1-C3) (C4-C6) (C7-C9)\\n\",\n \"----------+-----------+----------\\n\",\n \"Subgrid 4 | Subgrid 5 | Subgrid 6\\n\",\n \" (R4-R6) (R4-R6) (R4-R6)\\n\",\n \" (C1-C3) (C4-C6) (C7-C9)\\n\",\n \"----------+-----------+----------\\n\",\n \"Subgrid 7 | Subgrid 8 | Subgrid 9\\n\",\n \" (R7-R9) (R7-R9) (R7-R9)\\n\",\n \" (C1-C3) (C4-C6) (C7-C9)\\n\",\n \"```\\n\",\n \"\\n\",\n \"## IMPORTANT CONSTRAINTS\\n\",\n \"\\n\",\n \"- Coordinates are 1-indexed (1-9 for both row and column)\\n\",\n \"- Numbers must be 1-9\\n\",\n \"- One move per response\\n\",\n \"- Must be a valid move (no rule violations)\\n\",\n \"- Never repeat a previous move\\n\",\n \"\\n\",\n \"## YOUR GOAL\\n\",\n \"\\n\",\n \"Output ONLY your move in the format [row col number]. No explanation, no reasoning, just the move.\\n\",\n \"\\\"\\\"\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Vi1rFey39GUl\"\n },\n \"source\": [\n \"Now, let's define the `rollout_func`.\\n\",\n \"\\n\",\n \"This function manages the interaction between the model and the Sudoku environment. \\n\",\n \"For each prompt in the batch, it runs a full episode, collecting both the model's outputs and the corresponding rewards. These results are then used by GRPO to optimize the agent's policy.\\n\",\n \"\\n\",\n \"Each game allows the model to make **up to 100 turns**, giving it multiple chances to solve the puzzle.\\n\",\n \"We have different difficulty levels available: `'easy'`, `'medium'`, and `'hard'`. The level affects the amount of information provided in the prompt. Higher difficulties give less guidance.\\n\",\n \"\\n\",\n \"For the **easy** level, the Qwen/Qwen3-1.7B model is sufficient to solve the puzzles efficiently in a Colab notebook.\\n\",\n \"For **medium** or **hard** levels, a larger or more advanced model would likely be needed.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"wMQQoQ_UKyJl\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOTrainer\\n\",\n \"\\n\",\n \"max_turns = 100\\n\",\n \"debug = False # Activate for detailed logs during training\\n\",\n \"difficulty=\\\"easy\\\"\\n\",\n \"\\n\",\n \"def rollout_func(prompts: list[str], trainer: GRPOTrainer) -> dict[str, list]:\\n\",\n \" all_prompt_ids = []\\n\",\n \" all_completion_ids = []\\n\",\n \" all_logprobs = []\\n\",\n \" all_correct = []\\n\",\n \" all_valid = []\\n\",\n \" all_empty_cell = []\\n\",\n \" all_repetition = []\\n\",\n \" all_progress = []\\n\",\n \"\\n\",\n \" for _ in prompts:\\n\",\n \" episode = rollout_once(\\n\",\n \" trainer=trainer,\\n\",\n \" env=client,\\n\",\n \" tokenizer=trainer.processing_class,\\n\",\n \" system_prompt=SYSTEM_PROMPT,\\n\",\n \" max_turns=max_turns,\\n\",\n \" debug=debug,\\n\",\n \" difficulty=difficulty,\\n\",\n \" )\\n\",\n \" all_prompt_ids.append(episode[\\\"prompt_ids\\\"])\\n\",\n \" all_completion_ids.append(episode[\\\"completion_ids\\\"])\\n\",\n \" all_logprobs.append(episode[\\\"logprobs\\\"])\\n\",\n \" all_correct.append(episode[\\\"correct_reward\\\"])\\n\",\n \" all_valid.append(episode[\\\"valid_move_reward\\\"])\\n\",\n \" all_empty_cell.append(episode[\\\"empty_cell_reward\\\"])\\n\",\n \" all_repetition.append(episode[\\\"repetition_reward\\\"])\\n\",\n \" all_progress.append(episode[\\\"progress_reward\\\"])\\n\",\n \"\\n\",\n \" return {\\n\",\n \" \\\"prompt_ids\\\": all_prompt_ids,\\n\",\n \" \\\"completion_ids\\\": all_completion_ids,\\n\",\n \" \\\"logprobs\\\": all_logprobs,\\n\",\n \" \\\"correct_reward\\\": all_correct,\\n\",\n \" \\\"valid_move_reward\\\": all_valid,\\n\",\n \" \\\"empty_cell_reward\\\": all_empty_cell,\\n\",\n \" \\\"repetition_reward\\\": all_repetition,\\n\",\n \" \\\"progress_reward\\\": all_progress,\\n\",\n \" }\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"ioUHdIxr9ZQO\"\n },\n \"source\": [\n \"### Define `rollout_once`\\n\",\n \"\\n\",\n \"The `rollout_once` function runs **a single interaction loop** between the model and the Sudoku environment using the trainer's generation method. \\n\",\n \"It executes one mini-episode, from generating a guess to receiving and processing feedback.\\n\",\n \"\\n\",\n \"Step-by-step:\\n\",\n \"\\n\",\n \"1. **Environment reset:** Start a new game session and initialize the observation.\\n\",\n \"2. **Prompt construction:** Combine the system prompt, current state, and user messages to form the model input.\\n\",\n \"3. **Generation:** Use `trl.experimental.openenv.generate_rollout_completions()` to efficiently produce the model's guess.\\n\",\n \"4. **Feedback extraction:** Parse the environment's response with helpers like `extract_sudoku_move()` and `extract_feedback()`.\\n\",\n \"5. **Reward calculation:** Compute rewards based on correctness, valid moves, empty cell moves, repeated moves, and progress.\\n\",\n \"6. **Return structured rollout data:** Includes prompt and completion IDs, log probabilities, and all reward components.\\n\",\n \"\\n\",\n \"This design allows each episode to be processed independently while providing detailed feedback for the **GRPO training loop**.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"AZim6XzEKyJl\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl.experimental.openenv import generate_rollout_completions\\n\",\n \"from textarena_env import TextArenaAction\\n\",\n \"from transformers import AutoTokenizer\\n\",\n \"from collections import defaultdict\\n\",\n \"\\n\",\n \"\\n\",\n \"def rollout_once(\\n\",\n \" trainer: GRPOTrainer,\\n\",\n \" env: TextArenaEnv,\\n\",\n \" tokenizer: AutoTokenizer,\\n\",\n \" system_prompt: str,\\n\",\n \" max_turns: int,\\n\",\n \" debug: bool = False,\\n\",\n \" difficulty: str = \\\"hard\\\",\\n\",\n \") -> dict[str, list]:\\n\",\n \" result = env.reset()\\n\",\n \" observation = result.observation\\n\",\n \"\\n\",\n \" # Only store the LAST turn for backprop (much more efficient!)\\n\",\n \" last_turn_data: dict | None = None\\n\",\n \"\\n\",\n \" valid_move_scores: list[float] = []\\n\",\n \" empty_cell_scores: list[float] = []\\n\",\n \" correct_scores: list[float] = []\\n\",\n \" repetition_scores: list[float] = []\\n\",\n \"\\n\",\n \" move_counts: defaultdict[str, int] = defaultdict(int)\\n\",\n \"\\n\",\n \" # Track successful and failed moves for summary\\n\",\n \" successful_moves: list[str] = []\\n\",\n \" failed_moves: list[str] = []\\n\",\n \"\\n\",\n \" # Extract initial board state\\n\",\n \" last_board_state = \\\"\\\"\\n\",\n \" initial_filled = 0\\n\",\n \" for message in observation.messages:\\n\",\n \" if message.content and is_valid_board_state(message.content):\\n\",\n \" last_board_state = message.content\\n\",\n \" initial_filled = count_filled_cells(last_board_state)\\n\",\n \" break\\n\",\n \"\\n\",\n \" max_filled = initial_filled # Track max progress\\n\",\n \"\\n\",\n \" for turn in range(max_turns):\\n\",\n \" if result.done:\\n\",\n \" break\\n\",\n \"\\n\",\n \" # Build COMPACT prompt (saves tokens!)\\n\",\n \" user_prompt = make_compact_prompt(\\n\",\n \" board=last_board_state,\\n\",\n \" step=turn + 1,\\n\",\n \" successful_moves=successful_moves,\\n\",\n \" failed_moves=failed_moves,\\n\",\n \" difficulty=difficulty,\\n\",\n \" )\\n\",\n \" messages = [\\n\",\n \" {\\\"role\\\": \\\"system\\\", \\\"content\\\": system_prompt},\\n\",\n \" {\\\"role\\\": \\\"user\\\", \\\"content\\\": user_prompt},\\n\",\n \" ]\\n\",\n \" prompt_text = tokenizer.apply_chat_template(\\n\",\n \" messages, add_generation_prompt=True, tokenize=False, enable_thinking=False # `enable_thinking` is usable for the current model but could need to be updated for other models\\n\",\n \" )\\n\",\n \"\\n\",\n \" if debug:\\n\",\n \" print(f\\\"\\\\n{'=' * 60}\\\")\\n\",\n \" print(f\\\"STEP {turn + 1}\\\")\\n\",\n \" print(f\\\"{'=' * 60}\\\")\\n\",\n \" print(f\\\"USER PROMPT:\\\\n{user_prompt}\\\")\\n\",\n \" print(f\\\"{'=' * 60}\\\")\\n\",\n \"\\n\",\n \" # Generate\\n\",\n \" rollout_outputs = generate_rollout_completions(trainer, [prompt_text])[0]\\n\",\n \"\\n\",\n \" # Store ONLY this turn's data (replace previous)\\n\",\n \" last_turn_data = {\\n\",\n \" \\\"prompt_ids\\\": rollout_outputs[\\\"prompt_ids\\\"],\\n\",\n \" \\\"completion_ids\\\": rollout_outputs[\\\"completion_ids\\\"],\\n\",\n \" \\\"logprobs\\\": rollout_outputs[\\\"logprobs\\\"],\\n\",\n \" }\\n\",\n \"\\n\",\n \" if debug:\\n\",\n \" step_tokens = len(rollout_outputs[\\\"prompt_ids\\\"]) + len(rollout_outputs[\\\"completion_ids\\\"])\\n\",\n \" print(f\\\"TOKENS: this_step={step_tokens} (only last turn used for backprop)\\\")\\n\",\n \"\\n\",\n \" completion_text = rollout_outputs.get(\\\"text\\\") or tokenizer.decode(\\n\",\n \" rollout_outputs[\\\"completion_ids\\\"], skip_special_tokens=True\\n\",\n \" )\\n\",\n \"\\n\",\n \" # Extract move\\n\",\n \" move = extract_sudoku_move(completion_text)\\n\",\n \"\\n\",\n \" if debug:\\n\",\n \" print(f\\\"MODEL OUTPUT: {completion_text}\\\")\\n\",\n \" print(f\\\"EXTRACTED MOVE: {move}\\\")\\n\",\n \"\\n\",\n \" # Step environment\\n\",\n \" result = env.step(TextArenaAction(message=move))\\n\",\n \" observation = result.observation\\n\",\n \" correct_score = float(result.reward or 0.0)\\n\",\n \"\\n\",\n \" # Get feedback\\n\",\n \" feedback = extract_feedback(observation)\\n\",\n \"\\n\",\n \" # Get environment response\\n\",\n \" env_response = \\\"\\\"\\n\",\n \" for msg in observation.messages:\\n\",\n \" if msg.sender_id == -1: # Environment message\\n\",\n \" env_response = msg.content\\n\",\n \" break\\n\",\n \"\\n\",\n \" if debug:\\n\",\n \" print(\\n\",\n \" f\\\"ENV RESPONSE: {env_response[:200]}...\\\"\\n\",\n \" if len(env_response) > 200\\n\",\n \" else f\\\"ENV RESPONSE: {env_response}\\\"\\n\",\n \" )\\n\",\n \" print(f\\\"VALID: {feedback['valid_move']}, WARNING: {feedback['got_warning']}, REWARD: {correct_score}\\\")\\n\",\n \"\\n\",\n \" # Calculate empty_cell_score\\n\",\n \" if last_board_state and move:\\n\",\n \" targets_empty = check_move_targets_empty_cell(move, last_board_state)\\n\",\n \" empty_cell_score = 1.0 if targets_empty else -1.0\\n\",\n \" else:\\n\",\n \" empty_cell_score = 0.0\\n\",\n \"\\n\",\n \" # Calculate valid_move_score and repetition_score\\n\",\n \" is_new_move = move_counts[move] == 0\\n\",\n \" repetition_count = move_counts[move]\\n\",\n \" move_counts[move] += 1\\n\",\n \"\\n\",\n \" # Exponential penalty for repetitions: -2^(n-1) capped at -10\\n\",\n \" # 1st repeat: -1, 2nd: -2, 3rd: -4, 4th+: -10 (capped)\\n\",\n \" if repetition_count > 0:\\n\",\n \" repetition_score = -min(2 ** (repetition_count - 1), 10.0)\\n\",\n \" else:\\n\",\n \" repetition_score = 0.0\\n\",\n \"\\n\",\n \" if debug:\\n\",\n \" print(\\n\",\n \" f\\\"SCORES: empty_cell={empty_cell_score}, is_new={is_new_move}, repetitions={repetition_count}, rep_penalty={repetition_score}\\\"\\n\",\n \" )\\n\",\n \"\\n\",\n \" if not debug:\\n\",\n \" print(f\\\"Step {turn + 1}: {move}\\\")\\n\",\n \"\\n\",\n \" if feedback[\\\"valid_move\\\"] and is_new_move:\\n\",\n \" valid_move_score = 1.0\\n\",\n \" if move:\\n\",\n \" successful_moves.append(move) # Track for summary\\n\",\n \" elif feedback[\\\"got_warning\\\"]:\\n\",\n \" valid_move_score = -0.5\\n\",\n \" if move:\\n\",\n \" failed_moves.append(move) # Track for summary\\n\",\n \" else:\\n\",\n \" valid_move_score = 0.0\\n\",\n \"\\n\",\n \" # Update board state and track progress\\n\",\n \" if feedback[\\\"board_state\\\"] and is_valid_board_state(feedback[\\\"board_state\\\"]):\\n\",\n \" last_board_state = feedback[\\\"board_state\\\"]\\n\",\n \" current_filled = count_filled_cells(last_board_state)\\n\",\n \" if current_filled > max_filled:\\n\",\n \" max_filled = current_filled\\n\",\n \"\\n\",\n \" valid_move_scores.append(valid_move_score)\\n\",\n \" empty_cell_scores.append(empty_cell_score)\\n\",\n \" correct_scores.append(correct_score)\\n\",\n \" repetition_scores.append(repetition_score)\\n\",\n \"\\n\",\n \" # Aggregate rewards\\n\",\n \" correct_reward = correct_scores[-1] if correct_scores else 0.0\\n\",\n \" valid_move_reward = sum(valid_move_scores) / len(valid_move_scores) if valid_move_scores else 0.0\\n\",\n \" empty_cell_reward = sum(empty_cell_scores) / len(empty_cell_scores) if empty_cell_scores else 0.0\\n\",\n \" repetition_reward = sum(repetition_scores) / len(repetition_scores) if repetition_scores else 0.0\\n\",\n \"\\n\",\n \" # Progress reward: how many cells we filled beyond initial state (normalized to 0-1)\\n\",\n \" # 81 total cells, so (max_filled - initial_filled) / (81 - initial_filled) gives progress\\n\",\n \" remaining_to_fill = 81 - initial_filled\\n\",\n \" if remaining_to_fill > 0:\\n\",\n \" progress_reward = (max_filled - initial_filled) / remaining_to_fill\\n\",\n \" else:\\n\",\n \" progress_reward = 1.0 # Already complete\\n\",\n \"\\n\",\n \" # Use ONLY last turn for backpropagation (much more efficient!)\\n\",\n \" if last_turn_data:\\n\",\n \" prompt_ids = last_turn_data[\\\"prompt_ids\\\"]\\n\",\n \" completion_ids = last_turn_data[\\\"completion_ids\\\"]\\n\",\n \" logprobs = last_turn_data[\\\"logprobs\\\"]\\n\",\n \" else:\\n\",\n \" prompt_ids = []\\n\",\n \" completion_ids = []\\n\",\n \" logprobs = []\\n\",\n \"\\n\",\n \" total_tokens = len(prompt_ids) + len(completion_ids)\\n\",\n \" cells_filled = max_filled - initial_filled\\n\",\n \" print(\\n\",\n \" f\\\"Episode: empty_cell={empty_cell_reward:.2f}, valid={valid_move_reward:.2f}, \\\"\\n\",\n \" f\\\"repetition={repetition_reward:.2f}, progress={progress_reward:.2f} ({cells_filled} cells), \\\"\\n\",\n \" f\\\"correct={correct_reward:.2f}, tokens={total_tokens}\\\"\\n\",\n \" )\\n\",\n \"\\n\",\n \" return {\\n\",\n \" \\\"prompt_ids\\\": prompt_ids,\\n\",\n \" \\\"completion_ids\\\": completion_ids,\\n\",\n \" \\\"logprobs\\\": logprobs,\\n\",\n \" \\\"correct_reward\\\": correct_reward,\\n\",\n \" \\\"valid_move_reward\\\": valid_move_reward,\\n\",\n \" \\\"empty_cell_reward\\\": empty_cell_reward,\\n\",\n \" \\\"repetition_reward\\\": repetition_reward,\\n\",\n \" \\\"progress_reward\\\": progress_reward,\\n\",\n \" }\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"MDJKMQ__8qzj\"\n },\n \"source\": [\n \"### Helper Functions\\n\",\n \"\\n\",\n \"These utility functions are used within `rollout_once` to process the environment and model outputs:\\n\",\n \"\\n\",\n \"- **`extract_sudoku_move`**: Extract a Sudoku move `[row, col, number]` from text. \\n\",\n \"- **`is_valid_board_state`**: Check if a string represents a valid Sudoku board. \\n\",\n \"- **`parse_board`**: Convert a board string into a 9ร—9 grid (with `0` for empty cells). \\n\",\n \"- **`count_filled_cells`**: Count the number of filled cells in the board. \\n\",\n \"- **`get_valid_numbers`**: Get the valid numbers for a specific cell according to Sudoku rules. \\n\",\n \"- **`extract_empty_cells_with_candidates`**: Identify empty cells along with their valid candidate numbers. \\n\",\n \"- **`extract_empty_cells`**: List all empty cells `(row, col)` from a board string. \\n\",\n \"- **`extract_board_only`**: Extract just the Sudoku grid from a message. \\n\",\n \"- **`make_compact_prompt`**: Create a concise prompt with only essential information to save tokens. \\n\",\n \"- **`check_move_targets_empty_cell`**: Verify if a proposed move targets an empty cell on the board. \\n\",\n \"- **`extract_feedback`**: Extract structured feedback from the environment's observation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"0f9RqHh7KyJl\"\n },\n \"outputs\": [],\n \"source\": [\n \"# @title Helpers (click to expand)\\n\",\n \"import re\\n\",\n \"\\n\",\n \"def extract_sudoku_move(text: str) -> str:\\n\",\n \" \\\"\\\"\\\"Extract a Sudoku move [row col number] from text.\\\"\\\"\\\"\\n\",\n \" # Try with spaces\\n\",\n \" match = re.search(r\\\"\\\\[(\\\\d)\\\\s+(\\\\d)\\\\s+(\\\\d)\\\\]\\\", text)\\n\",\n \" if match:\\n\",\n \" row, col, num = match.groups()\\n\",\n \" return f\\\"[{row} {col} {num}]\\\"\\n\",\n \"\\n\",\n \" # Try without spaces\\n\",\n \" match = re.search(r\\\"\\\\[(\\\\d)(\\\\d)(\\\\d)\\\\]\\\", text)\\n\",\n \" if match:\\n\",\n \" row, col, num = match.groups()\\n\",\n \" return f\\\"[{row} {col} {num}]\\\"\\n\",\n \"\\n\",\n \" return \\\"\\\" # Handled by the environment: missing/invalid moves trigger a \\\"wrong movement\\\" message affecting rewards\\n\",\n \"\\n\",\n \"\\n\",\n \"def is_valid_board_state(board_str: str) -> bool:\\n\",\n \" \\\"\\\"\\\"Check if the string contains an actual Sudoku board.\\\"\\\"\\\"\\n\",\n \" return \\\"R1\\\" in board_str and \\\"R9\\\" in board_str and \\\"|\\\" in board_str\\n\",\n \"\\n\",\n \"\\n\",\n \"def parse_board(board_str: str) -> list[list[int]]:\\n\",\n \" \\\"\\\"\\\"Parse board string into 9x9 grid (0 = empty).\\\"\\\"\\\"\\n\",\n \" grid = [[0] * 9 for _ in range(9)]\\n\",\n \" if not is_valid_board_state(board_str):\\n\",\n \" return grid\\n\",\n \"\\n\",\n \" for line in board_str.split(\\\"\\\\n\\\"):\\n\",\n \" line_stripped = line.strip()\\n\",\n \" if line_stripped and line_stripped[0] == \\\"R\\\" and len(line_stripped) > 1 and line_stripped[1].isdigit():\\n\",\n \" row = int(line_stripped[1]) - 1 # 0-indexed\\n\",\n \" cell_part = line_stripped[2:]\\n\",\n \" col = 0\\n\",\n \" for char in cell_part:\\n\",\n \" if char == \\\".\\\":\\n\",\n \" grid[row][col] = 0\\n\",\n \" col += 1\\n\",\n \" elif char.isdigit():\\n\",\n \" grid[row][col] = int(char)\\n\",\n \" col += 1\\n\",\n \" return grid\\n\",\n \"\\n\",\n \"\\n\",\n \"def count_filled_cells(board_str: str) -> int:\\n\",\n \" \\\"\\\"\\\"Count the number of filled cells in the board.\\\"\\\"\\\"\\n\",\n \" if not is_valid_board_state(board_str):\\n\",\n \" return 0\\n\",\n \" grid = parse_board(board_str)\\n\",\n \" return sum(1 for row in grid for cell in row if cell != 0)\\n\",\n \"\\n\",\n \"\\n\",\n \"def get_valid_numbers(grid: list[list[int]], row: int, col: int) -> set[int]:\\n\",\n \" \\\"\\\"\\\"Get valid numbers for a cell based on Sudoku rules.\\\"\\\"\\\"\\n\",\n \" if grid[row][col] != 0:\\n\",\n \" return set()\\n\",\n \"\\n\",\n \" used = set()\\n\",\n \"\\n\",\n \" # Check row\\n\",\n \" for c in range(9):\\n\",\n \" if grid[row][c] != 0:\\n\",\n \" used.add(grid[row][c])\\n\",\n \"\\n\",\n \" # Check column\\n\",\n \" for r in range(9):\\n\",\n \" if grid[r][col] != 0:\\n\",\n \" used.add(grid[r][col])\\n\",\n \"\\n\",\n \" # Check 3x3 box\\n\",\n \" box_row, box_col = 3 * (row // 3), 3 * (col // 3)\\n\",\n \" for r in range(box_row, box_row + 3):\\n\",\n \" for c in range(box_col, box_col + 3):\\n\",\n \" if grid[r][c] != 0:\\n\",\n \" used.add(grid[r][c])\\n\",\n \"\\n\",\n \" return set(range(1, 10)) - used\\n\",\n \"\\n\",\n \"\\n\",\n \"def extract_empty_cells_with_candidates(\\n\",\n \" board_str: str, sort_by_difficulty: bool = True\\n\",\n \") -> list[tuple[int, int, set[int]]]:\\n\",\n \" \\\"\\\"\\\"Extract empty cells with their valid candidate numbers.\\n\",\n \"\\n\",\n \" Args:\\n\",\n \" sort_by_difficulty: If True, sort by number of candidates (easiest first).\\n\",\n \" If False, keep natural order (top-left to bottom-right).\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" grid = parse_board(board_str)\\n\",\n \" cells_with_candidates = []\\n\",\n \"\\n\",\n \" for row in range(9):\\n\",\n \" for col in range(9):\\n\",\n \" if grid[row][col] == 0:\\n\",\n \" candidates = get_valid_numbers(grid, row, col)\\n\",\n \" cells_with_candidates.append((row + 1, col + 1, candidates)) # 1-indexed\\n\",\n \"\\n\",\n \" if sort_by_difficulty:\\n\",\n \" # Sort by number of candidates (easiest first = naked singles)\\n\",\n \" cells_with_candidates.sort(key=lambda x: len(x[2]))\\n\",\n \"\\n\",\n \" return cells_with_candidates\\n\",\n \"\\n\",\n \"\\n\",\n \"def extract_empty_cells(board_str: str) -> list[tuple[int, int]]:\\n\",\n \" \\\"\\\"\\\"Extract list of empty cells (row, col) from board string.\\\"\\\"\\\"\\n\",\n \" empty_cells = []\\n\",\n \" if not is_valid_board_state(board_str):\\n\",\n \" return empty_cells\\n\",\n \"\\n\",\n \" for line in board_str.split(\\\"\\\\n\\\"):\\n\",\n \" line_stripped = line.strip()\\n\",\n \" if line_stripped and line_stripped[0] == \\\"R\\\" and len(line_stripped) > 1 and line_stripped[1].isdigit():\\n\",\n \" row = int(line_stripped[1])\\n\",\n \" cell_part = line_stripped[2:]\\n\",\n \" col = 0\\n\",\n \" for char in cell_part:\\n\",\n \" if char == \\\".\\\":\\n\",\n \" col += 1\\n\",\n \" empty_cells.append((row, col))\\n\",\n \" elif char.isdigit():\\n\",\n \" col += 1\\n\",\n \" return empty_cells\\n\",\n \"\\n\",\n \"\\n\",\n \"def extract_board_only(text: str) -> str:\\n\",\n \" \\\"\\\"\\\"Extract just the Sudoku grid from a message.\\\"\\\"\\\"\\n\",\n \" if not text:\\n\",\n \" return \\\"\\\"\\n\",\n \"\\n\",\n \" lines = text.split(\\\"\\\\n\\\")\\n\",\n \" board_lines = []\\n\",\n \" in_board = False\\n\",\n \"\\n\",\n \" for line in lines:\\n\",\n \" stripped = line.strip()\\n\",\n \" if stripped.startswith(\\\"C1\\\") or (\\n\",\n \" stripped and stripped[0] == \\\"R\\\" and len(stripped) > 1 and stripped[1].isdigit()\\n\",\n \" ):\\n\",\n \" in_board = True\\n\",\n \" if in_board and (stripped.startswith(\\\"-\\\") or stripped.startswith(\\\"R\\\") or stripped.startswith(\\\"C1\\\")):\\n\",\n \" board_lines.append(line)\\n\",\n \" elif (\\n\",\n \" in_board\\n\",\n \" and stripped\\n\",\n \" and not stripped.startswith(\\\"-\\\")\\n\",\n \" and not (stripped[0] == \\\"R\\\" and len(stripped) > 1 and stripped[1].isdigit())\\n\",\n \" ):\\n\",\n \" break\\n\",\n \"\\n\",\n \" return \\\"\\\\n\\\".join(board_lines) if board_lines else \\\"\\\"\\n\",\n \"\\n\",\n \"\\n\",\n \"def make_compact_prompt(\\n\",\n \" board: str,\\n\",\n \" step: int,\\n\",\n \" successful_moves: list[str],\\n\",\n \" failed_moves: list[str],\\n\",\n \" difficulty: str = \\\"hard\\\",\\n\",\n \") -> str:\\n\",\n \" \\\"\\\"\\\"Create a compact prompt with only essential info (saves tokens!).\\n\",\n \"\\n\",\n \" Args:\\n\",\n \" difficulty: Training difficulty level:\\n\",\n \" - \\\"easy\\\": Show guaranteed moves (naked singles) + other options\\n\",\n \" - \\\"medium\\\": Only show other options (hints where to look, not exact answers)\\n\",\n \" - \\\"hard\\\": No hints (model must learn Sudoku rules by itself)\\n\",\n \" \\\"\\\"\\\"\\n\",\n \"\\n\",\n \" # Summary line\\n\",\n \" cells_filled = len(successful_moves)\\n\",\n \" summary = f\\\"Step {step}. Progress: {cells_filled} cells filled.\\\"\\n\",\n \"\\n\",\n \" # Board (only show the grid, stripped down)\\n\",\n \" board_only = extract_board_only(board) if board else \\\"No board available.\\\"\\n\",\n \"\\n\",\n \" # Moves already tried (for learning what NOT to do)\\n\",\n \" tried_moves_hint = \\\"\\\"\\n\",\n \" all_tried = successful_moves + failed_moves\\n\",\n \" if all_tried:\\n\",\n \" tried_moves_hint = f\\\"\\\\n\\\\nโš ๏ธ MOVES ALREADY TRIED (do not repeat): {', '.join(all_tried)}\\\"\\n\",\n \"\\n\",\n \" # Hints based on difficulty\\n\",\n \" hints = \\\"\\\"\\n\",\n \" if difficulty == \\\"easy\\\" and board:\\n\",\n \" # Easy: sorted by difficulty, show guaranteed moves + other easy options\\n\",\n \" cells_with_candidates = extract_empty_cells_with_candidates(board, sort_by_difficulty=True)\\n\",\n \" if cells_with_candidates:\\n\",\n \" guaranteed = []\\n\",\n \" other_hints = []\\n\",\n \" for row, col, candidates in cells_with_candidates[:10]:\\n\",\n \" if len(candidates) == 1:\\n\",\n \" num = list(candidates)[0]\\n\",\n \" guaranteed.append(f\\\"[{row} {col} {num}]\\\")\\n\",\n \" elif len(candidates) <= 3:\\n\",\n \" nums = \\\",\\\".join(str(n) for n in sorted(candidates))\\n\",\n \" other_hints.append(f\\\"({row},{col})โ†’{nums}\\\")\\n\",\n \"\\n\",\n \" if guaranteed:\\n\",\n \" hints = f\\\"\\\\n\\\\n๐ŸŽฏ GUARANTEED MOVES: {', '.join(guaranteed[:5])}\\\"\\n\",\n \" if other_hints:\\n\",\n \" hints += f\\\"\\\\nOther options: {' | '.join(other_hints[:5])}\\\"\\n\",\n \"\\n\",\n \" elif difficulty == \\\"medium\\\" and board:\\n\",\n \" # Medium: NOT sorted, just show empty cells with candidates (no ordering hints)\\n\",\n \" cells_with_candidates = extract_empty_cells_with_candidates(board, sort_by_difficulty=False)\\n\",\n \" if cells_with_candidates:\\n\",\n \" cell_hints = []\\n\",\n \" for row, col, candidates in cells_with_candidates[:10]:\\n\",\n \" nums = \\\",\\\".join(str(n) for n in sorted(candidates))\\n\",\n \" cell_hints.append(f\\\"({row},{col})โ†’{nums}\\\")\\n\",\n \" if cell_hints:\\n\",\n \" hints = f\\\"\\\\n\\\\nEmpty cells: {' | '.join(cell_hints)}\\\"\\n\",\n \"\\n\",\n \" return f\\\"{summary}\\\\n\\\\nBoard:\\\\n{board_only}{tried_moves_hint}{hints}\\\\n\\\\nYour move:\\\"\\n\",\n \"\\n\",\n \"\\n\",\n \"def check_move_targets_empty_cell(move: str, board_str: str) -> bool:\\n\",\n \" \\\"\\\"\\\"Check if the move targets an empty cell on the board.\\\"\\\"\\\"\\n\",\n \" if not move or not board_str:\\n\",\n \" return False\\n\",\n \"\\n\",\n \" match = re.search(r\\\"\\\\[(\\\\d)\\\\s+(\\\\d)\\\\s+(\\\\d)\\\\]\\\", move)\\n\",\n \" if not match:\\n\",\n \" return False\\n\",\n \"\\n\",\n \" row, col = int(match.group(1)), int(match.group(2))\\n\",\n \" empty_cells = extract_empty_cells(board_str)\\n\",\n \" return (row, col) in empty_cells\\n\",\n \"\\n\",\n \"\\n\",\n \"def extract_feedback(observation) -> dict:\\n\",\n \" \\\"\\\"\\\"Extract feedback from environment observation.\\\"\\\"\\\"\\n\",\n \" feedback = {\\\"valid_move\\\": True, \\\"got_warning\\\": False, \\\"board_state\\\": \\\"\\\"}\\n\",\n \"\\n\",\n \" if not observation or not observation.messages:\\n\",\n \" return feedback\\n\",\n \"\\n\",\n \" for message in observation.messages:\\n\",\n \" content = message.content.lower() if message.content else \\\"\\\"\\n\",\n \"\\n\",\n \" if any(kw in content for kw in [\\\"invalid\\\", \\\"error\\\", \\\"cannot\\\", \\\"already\\\", \\\"violation\\\", \\\"lost\\\"]):\\n\",\n \" feedback[\\\"valid_move\\\"] = False\\n\",\n \" if \\\"please resubmit\\\" in content or \\\"avoid penalties\\\" in content:\\n\",\n \" feedback[\\\"got_warning\\\"] = True\\n\",\n \"\\n\",\n \" if message.content and \\\"|\\\" in message.content and \\\"R1\\\" in message.content:\\n\",\n \" feedback[\\\"board_state\\\"] = message.content\\n\",\n \"\\n\",\n \" return feedback\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Oek3JhcWnKhw\"\n },\n \"source\": [\n \"## Define Reward Functions\\n\",\n \"\\n\",\n \"To guide the agent's learning, we define reward functions that convert the environment's feedback into numeric signals.\\n\",\n \"Each function captures a specific aspect of performance in the **Sudoku** game:\\n\",\n \"\\n\",\n \"- **`reward_empty_cell`**: Reward for targeting empty cells, encouraging the agent to pick valid positions first.\\n\",\n \"- **`reward_valid_moves`**: Reward for making moves that comply with Sudoku rules.\\n\",\n \"- **`reward_correct`**: Reward for correctly placing numbers, contributing to solving the puzzle.\\n\",\n \"- **`reward_repetition`**: Penalty for repeating moves in the same cell.\\n\",\n \"- **`reward_progress`**: Reward for filling more cells on the board, indicating overall progress.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"TPe4XL89KyJl\"\n },\n \"outputs\": [],\n \"source\": [\n \"def reward_empty_cell(completions: list[str], **kwargs) -> list[float]:\\n\",\n \" \\\"\\\"\\\"Reward for targeting empty cells (learn to pick valid positions first).\\\"\\\"\\\"\\n\",\n \" rewards = kwargs.get(\\\"empty_cell_reward\\\")\\n\",\n \" if rewards is None:\\n\",\n \" return [0.0 for _ in completions]\\n\",\n \" return [float(r) for r in rewards]\\n\",\n \"\\n\",\n \"\\n\",\n \"def reward_valid_moves(completions: list[str], **kwargs) -> list[float]:\\n\",\n \" \\\"\\\"\\\"Reward for making valid moves.\\\"\\\"\\\"\\n\",\n \" rewards = kwargs.get(\\\"valid_move_reward\\\")\\n\",\n \" if rewards is None:\\n\",\n \" return [0.0 for _ in completions]\\n\",\n \" return [float(r) for r in rewards]\\n\",\n \"\\n\",\n \"\\n\",\n \"def reward_correct(completions: list[str], **kwargs) -> list[float]:\\n\",\n \" \\\"\\\"\\\"Reward for solving the puzzle.\\\"\\\"\\\"\\n\",\n \" rewards = kwargs.get(\\\"correct_reward\\\")\\n\",\n \" if rewards is None:\\n\",\n \" return [0.0 for _ in completions]\\n\",\n \" return [float(r) for r in rewards]\\n\",\n \"\\n\",\n \"\\n\",\n \"def reward_repetition(completions: list[str], **kwargs) -> list[float]:\\n\",\n \" \\\"\\\"\\\"Penalty for repeating moves.\\\"\\\"\\\"\\n\",\n \" rewards = kwargs.get(\\\"repetition_reward\\\")\\n\",\n \" if rewards is None:\\n\",\n \" return [0.0 for _ in completions]\\n\",\n \" return [float(r) for r in rewards]\\n\",\n \"\\n\",\n \"\\n\",\n \"def reward_progress(completions: list[str], **kwargs) -> list[float]:\\n\",\n \" \\\"\\\"\\\"Reward for filling more cells in the board.\\\"\\\"\\\"\\n\",\n \" rewards = kwargs.get(\\\"progress_reward\\\")\\n\",\n \" if rewards is None:\\n\",\n \" return [0.0 for _ in completions]\\n\",\n \" return [float(r) for r in rewards]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"66ZsrLplm07U\"\n },\n \"source\": [\n \"## Load the Custom Dataset\\n\",\n \"\\n\",\n \"The dataset is built using repeated prompts to control the total number of training episodes.\\n\",\n \"\\n\",\n \"Each entry in the dataset triggers **one rollout episode** during training. \\n\",\n \"The `dataset_prompt` provides the initial instruction to the model at the start of each episode, ensuring consistent guidance and context for task execution.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"zV7C_t1GKyJm\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import Dataset\\n\",\n \"\\n\",\n \"dataset_prompt = \\\"Play Sudoku like an expert.\\\"\\n\",\n \"dataset_size = 30\\n\",\n \"\\n\",\n \"dataset = Dataset.from_dict({\\\"prompt\\\": [dataset_prompt] * dataset_size})\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"-mvka-96m3I7\"\n },\n \"source\": [\n \"## Fine-tune using TRL and the GRPOTrainer\\n\",\n \"\\n\",\n \"The next step is to define the GRPOConfig, which sets all key training parameters.\\n\",\n \"\\n\",\n \"This configuration determines how the model interacts with vLLM, handles memory and computation, and records training metrics and logs for monitoring the fine-tuning process.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"4BP-aBcVKyJm\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOConfig\\n\",\n \"\\n\",\n \"output_dir = \\\"sudoku-grpo-qwen3\\\"\\n\",\n \"\\n\",\n \"grpo_config = GRPOConfig(\\n\",\n \" use_vllm=True, # Use vLLM engine for fast and efficient inference\\n\",\n \" vllm_mode=\\\"colocate\\\", # Run vLLM generation on the same GPU as training\\n\",\n \" vllm_gpu_memory_utilization=0.1, # Fraction of GPU memory allocated to vLLM\\n\",\n \" vllm_max_model_length=2560, # Maximum context length for vLLM generations\\n\",\n \"\\n\",\n \" output_dir=output_dir, # Directory to save model checkpoints and logs\\n\",\n \" num_train_epochs=1, # Number of training epochs\\n\",\n \" learning_rate=5e-6, # Initial learning rate\\n\",\n \"\\n\",\n \" #weight_decay=args.weight_decay, # Optional weight decay for optimizer\\n\",\n \" gradient_accumulation_steps=8, # Accumulate gradients over multiple steps to simulate larger batch size\\n\",\n \" per_device_train_batch_size=1, # Batch size per device (GPU)\\n\",\n \" warmup_steps=20, # Number of warmup steps for learning rate scheduler\\n\",\n \" num_generations=8, # Number of rollouts generated per prompt\\n\",\n \" max_completion_length=8, # Maximum length of generated completions\\n\",\n \"\\n\",\n \" logging_steps=1, # Log metrics every N steps\\n\",\n \" save_strategy=\\\"steps\\\", # Save checkpoints based on steps\\n\",\n \" save_steps=10, # Save every N steps\\n\",\n \"\\n\",\n \" report_to=\\\"trackio\\\", # Reporting backend for tracking experiments\\n\",\n \" trackio_space_id=output_dir, # Trackio space ID to log metrics\\n\",\n \"\\n\",\n \" use_liger_kernel=False, # Enable Liger kernel optimizations for faster training\\n\",\n \" # chat_template_kwargs={\\\"enable_thinking\\\": False}, # Optional template args for model reasoning. We manage this in the rollout function\\n\",\n \"\\n\",\n \" temperature=0.8,\\n\",\n \" top_k=10,\\n\",\n \"\\n\",\n \" model_init_kwargs={\\n\",\n \" \\\"use_cache\\\": False,\\n\",\n \" }\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"a1taGmD--0Y4\"\n },\n \"source\": [\n \"## Create `GRPOTrainer` and Start Training\\n\",\n \"\\n\",\n \"Next, we initialize the `GRPOTrainer`, which handles the full reinforcement learning loop.\\n\",\n \"\\n\",\n \"It requires the **model**, **reward functions**, **rollout function**, and **dataset** defined earlier. \\n\",\n \"Here, we use **Qwen/Qwen3-1.7B**, a smaller version of the Qwen3 models. This model is sufficient for training on the \\\"easy\\\" difficulty Sudoku setting. \\n\",\n \"For \\\"medium\\\" or \\\"hard\\\" difficulty, a larger model would be needed, but this setup fits well in Colab with the current configuration.\\n\",\n \"\\n\",\n \"The trainer coordinates:\\n\",\n \"- Interaction between the model and the environment \\n\",\n \"- Application of reward signals \\n\",\n \"- Policy updates based on feedback\\n\",\n \"\\n\",\n \"Finally, calling `trainer.train()` starts the fine-tuning process, allowing the model to learn to solve Sudoku through repeated feedback and iteration.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"O-aKk1EwKyJm\"\n },\n \"outputs\": [],\n \"source\": [\n \"model_name = \\\"Qwen/Qwen3-1.7B\\\"\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"c75c6199d00d42b88a0ef49f650317bf\",\n \"114a42d7d0a74a7a81dad02c21cf41b2\"\n ]\n },\n \"id\": \"cQP77cFYKyJm\",\n \"outputId\": \"8ec8a2c5-6e64-4c88-a99b-3b54e2f0f1c5\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"c75c6199d00d42b88a0ef49f650317bf\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Loading checkpoint shards: 0%| | 0/2 [00:00\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Created new run: sergiopaniego-1767361842\\n\",\n \"Step 1: [1 1 6]\\n\",\n \"Step 2: [2 7 4]\\n\",\n \"Step 3: [3 4 8]\\n\",\n \"Step 4: [3 3 2]\\n\",\n \"Step 5: [4 8 6]\\n\",\n \"Step 6: [4 8 6]\\n\",\n \"Episode: empty_cell=1.00, valid=0.58, repetition=-0.17, progress=0.13 (4 cells), correct=-1.00, tokens=1860\\n\",\n \"Step 1: [1 1 7]\\n\",\n \"Step 2: [2 4 6]\\n\",\n \"Step 3: [1 7 8]\\n\",\n \"Step 4: [2 4 6]\\n\",\n \"Step 5: [2 4 4]\\n\",\n \"Step 6: [2 4 6]\\n\",\n \"Step 7: [2 4 6]\\n\",\n \"Episode: empty_cell=0.43, valid=0.21, repetition=-1.00, progress=0.10 (3 cells), correct=-1.00, tokens=1866\\n\",\n \"Step 1: [1 1 2]\\n\",\n \"Step 2: [1 1 2]\\n\",\n \"Episode: empty_cell=-1.00, valid=-0.25, repetition=-0.50, progress=0.00 (0 cells), correct=-1.00, tokens=1826\\n\",\n \"\\n\",\n \"# ... Output truncated for readability (see Trackio dashboard for full logs) ...\\n\",\n \"\\n\",\n \"Step 1: [1 7 6]\\n\",\n \"Step 2: [1 9 2]\\n\",\n \"Step 3: [2 6 5]\\n\",\n \"Step 4: [2 1 3]\\n\",\n \"Step 5: [2 2 2]\\n\",\n \"Step 6: [3 1 4]\\n\",\n \"Step 7: [3 2 6]\\n\",\n \"Step 8: [2 9 7]\\n\",\n \"Step 9: [3 4 8]\\n\",\n \"Step 10: [3 8 9]\\n\",\n \"Step 11: [1 3 5]\\n\",\n \"Step 12: [2 9 4]\\n\",\n \"Step 13: [4 3 8]\\n\",\n \"Step 14: [3 8 4]\\n\",\n \"Step 15: [2 9 4]\\n\",\n \"Episode: empty_cell=0.60, valid=0.73, repetition=-0.07, progress=0.40 (12 cells), correct=-1.00, tokens=1931\\n\",\n \"Step 1: [2 8 1]\\n\",\n \"Step 2: [2 5 2]\\n\",\n \"Step 3: [3 4 6]\\n\",\n \"Step 4: [1 5 1]\\n\",\n \"Step 5: [2 6 4]\\n\",\n \"Step 6: [3 7 4]\\n\",\n \"Step 7: [4 3 6]\\n\",\n \"Step 8: [5 1 2]\\n\",\n \"Step 9: [1 1 4]\\n\",\n \"Step 10: [4 3 6]\\n\",\n \"Step 11: [4 3 6]\\n\",\n \"Step 12: [4 3 6]\\n\",\n \"Step 13: [4 3 6]\\n\",\n \"Step 14: [4 1 9]\\n\",\n \"Step 15: [4 2 4]\\n\",\n \"Step 16: [7 8 5]\\n\",\n \"Step 17: [7 8 5]\\n\",\n \"Episode: empty_cell=0.53, valid=0.62, repetition=-0.94, progress=0.37 (11 cells), correct=-1.00, tokens=1916\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"\\n\",\n \"

\\n\",\n \" \\n\",\n \" \\n\",\n \" [30/30 26:13, Epoch 1/1]\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \"
StepTraining Loss
1-0.113800
2-0.001800
3-0.051300
4-0.012800
50.012200
60.045600
7-0.104800
8-0.093600
90.182400
10-0.027000
110.042300
12-0.052400
13-0.100100
14-0.074400
15-0.105500
160.125200
17-0.016900
180.119100
190.081800
200.003300
210.024400
22-0.038700
230.000000
240.000000
250.000000
260.000000
270.000000
280.000000
290.000000
300.000000

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Step 1: [1 3 8]\\n\",\n \"Step 2: [1 9 2]\\n\",\n \"Step 3: [3 4 2]\\n\",\n \"Step 4: [3 8 5]\\n\",\n \"Step 5: [1 2 4]\\n\",\n \"Step 6: [7 5 1]\\n\",\n \"Step 7: [7 5 1]\\n\",\n \"Episode: empty_cell=0.43, valid=0.64, repetition=-0.14, progress=0.17 (5 cells), correct=-1.00, tokens=1881\\n\",\n \"Step 1: [1 2 9]\\n\",\n \"Step 2: [1 6 5]\\n\",\n \"Step 3: [2 8 9]\\n\",\n \"Step 4: [2 9 7]\\n\",\n \"Step 5: [3 4 6]\\n\",\n \"Step 6: [3 3 3]\\n\",\n \"Step 7: [3 6 2]\\n\",\n \"Step 8: [3 5 9]\\n\",\n \"Step 9: [4 7 8]\\n\",\n \"Step 10: [5 2 2]\\n\",\n \"Step 11: [5 1 9]\\n\",\n \"Step 12: [3 4 6]\\n\",\n \"Step 13: [4 7 1]\\n\",\n \"Step 14: [5 3 7]\\n\",\n \"Step 15: [5 5 6]\\n\",\n \"Step 16: [6 8 4]\\n\",\n \"Step 17: [6 1 5]\\n\",\n \"Step 18: [6 5 7]\\n\",\n \"Step 19: [6 6 1]\\n\",\n \"Step 20: [7 2 4]\\n\",\n \"Step 21: [7 3 8]\\n\",\n \"Step 22: [7 6 6]\\n\",\n \"Step 23: [7 7 7]\\n\",\n \"Step 24: [8 3 5]\\n\",\n \"Step 25: [8 5 2]\\n\",\n \"Step 26: [8 6 4]\\n\",\n \"Step 27: [8 9 1]\\n\",\n \"Step 28: [9 1 2]\\n\",\n \"Step 29: [9 2 3]\\n\",\n \"Step 30: [9 3 9]\\n\",\n \"Step 31: [9 4 8]\\n\",\n \"Step 32: [9 7 4]\\n\",\n \"Episode: empty_cell=0.88, valid=0.92, repetition=-0.03, progress=1.00 (30 cells), correct=1.00, tokens=2035\\n\",\n \"\\n\",\n \"# ... Output truncated for readability (see Trackio dashboard for full logs) ...\\n\",\n \"\\n\",\n \"Step 1: [3 6 4]\\n\",\n \"Step 2: [2 3 7]\\n\",\n \"Step 3: [4 9 2]\\n\",\n \"Step 4: [5 4 7]\\n\",\n \"Step 5: [3 9 8]\\n\",\n \"Step 6: [4 6 9]\\n\",\n \"Step 7: [5 5 1]\\n\",\n \"Step 8: [5 6 2]\\n\",\n \"Step 9: [6 3 2]\\n\",\n \"Step 10: [6 8 8]\\n\",\n \"Step 11: [5 8 5]\\n\",\n \"Step 12: [5 2 8]\\n\",\n \"Step 13: [5 1 9]\\n\",\n \"Step 14: [6 7 6]\\n\",\n \"Step 15: [6 5 4]\\n\",\n \"Step 16: [4 5 6]\\n\",\n \"Step 17: [6 4 3]\\n\",\n \"Step 18: [7 4 4]\\n\",\n \"Step 19: [7 7 8]\\n\",\n \"Step 20: [7 1 3]\\n\",\n \"Step 21: [9 2 6]\\n\",\n \"Step 22: [2 2 4]\\n\",\n \"Step 23: [3 7 7]\\n\",\n \"Step 24: [4 1 4]\\n\",\n \"Step 25: [4 2 5]\\n\",\n \"Step 26: [9 1 8]\\n\",\n \"Step 27: [1 2 3]\\n\",\n \"Step 28: [2 1 6]\\n\",\n \"Step 29: [1 7 4]\\n\",\n \"Step 30: [9 7 9]\\n\",\n \"Episode: empty_cell=1.00, valid=1.00, repetition=0.00, progress=1.00 (30 cells), correct=1.00, tokens=2028\\n\",\n \"Step 1: [3 3 7]\\n\",\n \"Step 2: [2 1 9]\\n\",\n \"Step 3: [3 4 1]\\n\",\n \"Step 4: [3 6 2]\\n\",\n \"Step 5: [4 3 1]\\n\",\n \"Step 6: [4 2 6]\\n\",\n \"Step 7: [4 7 8]\\n\",\n \"Step 8: [4 6 5]\\n\",\n \"Step 9: [4 8 7]\\n\",\n \"Step 10: [3 8 6]\\n\",\n \"Step 11: [2 5 8]\\n\",\n \"Step 12: [6 5 7]\\n\",\n \"Step 13: [6 1 8]\\n\",\n \"Step 14: [5 7 3]\\n\",\n \"Step 15: [7 6 9]\\n\",\n \"Step 16: [7 7 2]\\n\",\n \"Step 17: [6 6 3]\\n\",\n \"Step 18: [8 2 4]\\n\",\n \"Step 19: [8 4 5]\\n\",\n \"Step 20: [8 6 1]\\n\",\n \"Step 21: [5 6 8]\\n\",\n \"Step 22: [5 4 6]\\n\",\n \"Step 23: [5 5 1]\\n\",\n \"Step 24: [8 5 2]\\n\",\n \"Step 25: [9 4 8]\\n\",\n \"Step 26: [9 5 6]\\n\",\n \"Step 27: [4 5 4]\\n\",\n \"Step 28: [1 9 8]\\n\",\n \"Step 29: [7 9 7]\\n\",\n \"Episode: empty_cell=1.00, valid=1.00, repetition=0.00, progress=1.00 (29 cells), correct=1.00, tokens=2020\\n\",\n \"* Run finished. Uploading logs to Trackio (please wait...)\\n\"\n ]\n }\n ],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"gF-mr-gfAtkp\"\n },\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"qM8tW2pdKyJm\"\n },\n \"outputs\": [],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_training = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"training_memory_percentage = round(used_memory_for_training / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_training} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {training_memory_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"BZj4IG9ZBAix\"\n },\n \"source\": [\n \"In this step, the fine-tuned model is saved locally and uploaded to the Hugging Face Hub using the configured account credentials.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"xJV-NZTmKyJm\"\n },\n \"outputs\": [],\n \"source\": [\n \"client.close()\\n\",\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"X-6lB52GAl_u\"\n },\n \"source\": [\n \"## Load the Fine-Tuned Model and Run Inference\\n\",\n \"\\n\",\n \"Now let's test our fine-tuned model by loading the **adapter** and running **inference**. \\n\",\n \"We begin by loading the **base model**, attaching the adapter, and obtaining the final fine-tuned model ready for evaluation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"d686d3933bef4ea3a9fb58193495e970\",\n \"4ce63e90903a4f60be6694de976e7127\",\n \"a1003f172b954e218fdb00539d79a7d1\",\n \"859e82d390204d5d8a763bd61b356ae4\",\n \"b54f50facca141639f3412b86cfc433d\",\n \"ec156a0cf90a42059f2335d4bae0628e\",\n \"fd9f8dcdf82d4e39a9c72d0e25464c28\",\n \"8696f66460244e55b9d67fd2fe9d6e51\",\n \"6acb03abb643408b8f802704f00674f5\",\n \"e13e54f756874f78af67a25564d64375\",\n \"5e29181a5ec1421d9f2eefbf7529d363\",\n \"fa420825fdab47d6aea18cd352dd9ef1\",\n \"9a460882ac834c8a90d77eec9a2c34ea\",\n \"7bf3166759f64fcf8968a6d282f8df85\"\n ]\n },\n \"id\": \"-Vu--VueKyJm\",\n \"outputId\": \"399ccb1e-45bf-4305-ae9d-97edece48b53\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"d686d3933bef4ea3a9fb58193495e970\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"config.json: 0.00B [00:00, ?B/s]\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"4ce63e90903a4f60be6694de976e7127\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"model.safetensors.index.json: 0.00B [00:00, ?B/s]\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"a1003f172b954e218fdb00539d79a7d1\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Fetching 2 files: 0%| | 0/2 [00:00= 1.0:\\n\",\n \" final_position_reward = 1.0\\n\",\n \" else:\\n\",\n \" final_position_reward = position_scores[-1] if position_scores else 0.0\\n\",\n \"\\n\",\n \" return {\\n\",\n \" \\\"prompt_ids\\\": prompt_ids,\\n\",\n \" \\\"completion_ids\\\": completion_ids,\\n\",\n \" \\\"logprobs\\\": logprobs,\\n\",\n \" \\\"env_mask\\\": env_mask,\\n\",\n \" \\\"raw_rewards\\\": raw_rewards,\\n\",\n \" \\\"correct_reward\\\": correct_reward_value,\\n\",\n \" \\\"position_reward\\\": final_position_reward,\\n\",\n \" \\\"model_outputs\\\": model_outputs,\\n\",\n \" }\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"cipvIDzcoF3C\",\n \"metadata\": {\n \"id\": \"cipvIDzcoF3C\"\n },\n \"source\": [\n \"### Helper functions\\n\",\n \"\\n\",\n \"Supporting utilities used in `rollout_once`:\\n\",\n \"\\n\",\n \"- **`make_user_prompt`**: builds the user prompt combining the conversation history.\\n\",\n \"- **`format_history`**: formats the conversation log for consistent context.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"bVeKfbaK7C4z\",\n \"metadata\": {\n \"id\": \"bVeKfbaK7C4z\"\n },\n \"outputs\": [],\n \"source\": [\n \"# @title Helpers definition (click to expand)\\n\",\n \"def format_history(messages) -> str:\\n\",\n \" lines = []\\n\",\n \" for message in messages:\\n\",\n \" tag = message.category or \\\"MESSAGE\\\"\\n\",\n \" content = message.content.strip()\\n\",\n \" if not content:\\n\",\n \" continue\\n\",\n \" lines.append(f\\\"[{tag}] {content}\\\")\\n\",\n \" return \\\"\\\\n\\\".join(lines)\\n\",\n \"\\n\",\n \"\\n\",\n \"def make_user_prompt(prompt_text, messages) -> str:\\n\",\n \" history = format_history(messages)\\n\",\n \" # Only use messages for conversation history - the prompt is already included as the first message\\n\",\n \" history_section = history if history else \\\"[PROMPT] Awaiting first feedback.\\\"\\n\",\n \" return f\\\"Conversation so far:\\\\n{history_section}\\\\n\\\\nReply with your next guess enclosed in square brackets.\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"i3G0x0RheYkL\",\n \"metadata\": {\n \"id\": \"i3G0x0RheYkL\"\n },\n \"source\": [\n \"## Define reward functions\\n\",\n \"\\n\",\n \"To guide the agent's learning process, we define simple reward functions that map the feedback from the environment into numeric signals. \\n\",\n \"Each function corresponds to a specific aspect of the **Wordle** game:\\n\",\n \"\\n\",\n \"- โœ… **`reward_correct`**: rewards the model when it guesses the correct word (binary: 0 or 1). \\n\",\n \"- ๐ŸŽฏ **`reward_position`**: rewards progress based on letter feedback. Green letters worth 1.0, yellow worth 0.5, normalized by 5. If the model wins, this is set to 1.0.\\n\",\n \"- ๐Ÿ“ **`reward_format_strict`**: rewards correct output format `[xxxxx]`. Returns proportion of correctly formatted outputs across all turns.\\n\",\n \"\\n\",\n \"These functions return lists of float values that the **GRPOTrainer** uses during optimization. \\n\",\n \"By combining them, the model learns to balance correctness, information gathering, and proper formatting in its guessing strategy.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"61e454d1-9abc-42a6-868c-a24e9801ac44\",\n \"metadata\": {\n \"id\": \"61e454d1-9abc-42a6-868c-a24e9801ac44\"\n },\n \"outputs\": [],\n \"source\": [\n \"def reward_correct(completions, **kwargs):\\n\",\n \" \\\"\\\"\\\"Reward from environment (correct answer).\\\"\\\"\\\"\\n\",\n \" rewards = kwargs.get(\\\"correct_reward\\\") if kwargs else None\\n\",\n \" if rewards is None:\\n\",\n \" return [0.0 for _ in completions]\\n\",\n \" return [float(r) for r in rewards]\\n\",\n \"\\n\",\n \"\\n\",\n \"def reward_position(completions, **kwargs):\\n\",\n \" \\\"\\\"\\\"Position reward: green worth 1.0, yellow worth 0.5, normalized by 5.\\\"\\\"\\\"\\n\",\n \" rewards = kwargs.get(\\\"position_reward\\\") if kwargs else None\\n\",\n \" if rewards is None:\\n\",\n \" return [0.0 for _ in completions]\\n\",\n \" return [float(r) for r in rewards]\\n\",\n \"\\n\",\n \"\\n\",\n \"def compute_format_reward(model_outputs):\\n\",\n \" \\\"\\\"\\\"Compute format reward from a list of model outputs (one per turn).\\n\",\n \"\\n\",\n \" Each output should be exactly [5 letters] with optional whitespace.\\n\",\n \" Returns proportion of correctly formatted outputs.\\n\",\n \" \\\"\\\"\\\"\\n\",\n \" if not model_outputs:\\n\",\n \" return 0.0\\n\",\n \"\\n\",\n \" exact_pattern = re.compile(r\\\"^\\\\s*\\\\[[A-Za-z]{5}\\\\]\\\\s*$\\\")\\n\",\n \" correct_count = sum(1 for output in model_outputs if exact_pattern.match(output))\\n\",\n \"\\n\",\n \" return correct_count / len(model_outputs)\\n\",\n \"\\n\",\n \"\\n\",\n \"def reward_format_strict(completions, **kwargs):\\n\",\n \" \\\"\\\"\\\"Format reward - pre-computed in rollout_func.\\\"\\\"\\\"\\n\",\n \" rewards = kwargs.get(\\\"format_reward\\\") if kwargs else None\\n\",\n \" if rewards is None:\\n\",\n \" return [0.0 for _ in completions]\\n\",\n \" return [float(r) for r in rewards]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"RN5VkehojyOJ\",\n \"metadata\": {\n \"id\": \"RN5VkehojyOJ\"\n },\n \"source\": [\n \"## Create dataset\\n\",\n \"\\n\",\n \"We create a dataset with repeated prompts to control the number of training episodes. \\n\",\n \"Each entry in the dataset triggers one rollout episode during training. The `dataset_prompt` provides the initial instruction to the model before each game starts.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"deab8040-9b51-4c52-befe-e48578cdbb53\",\n \"metadata\": {\n \"id\": \"deab8040-9b51-4c52-befe-e48578cdbb53\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import Dataset\\n\",\n \"\\n\",\n \"dataset_size = 3000\\n\",\n \"dataset_prompt = \\\"Play Wordle like an expert.\\\"\\n\",\n \"\\n\",\n \"dataset = Dataset.from_dict({\\\"prompt\\\": [dataset_prompt] * dataset_size})\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"DnR90-D66Fm_\",\n \"metadata\": {\n \"id\": \"DnR90-D66Fm_\"\n },\n \"source\": [\n \"## Set GRPO Config\\n\",\n \"\\n\",\n \"Next, we define the **GRPOConfig**, which controls all key training parameters. \\n\",\n \"This configuration specifies how the model interacts with **vLLM**, manages memory, and logs results.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"20ac9371-af1a-4b9e-b678-33d6a3bf07cc\",\n \"metadata\": {\n \"id\": \"20ac9371-af1a-4b9e-b678-33d6a3bf07cc\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import GRPOConfig\\n\",\n \"\\n\",\n \"output_dir = \\\"wordle-grpo-Qwen3-1.7B-test\\\"\\n\",\n \"\\n\",\n \"grpo_config = GRPOConfig(\\n\",\n \" # Training schedule / optimization\\n\",\n \" num_train_epochs = 1, # Number of full dataset passes\\n\",\n \" learning_rate = 1e-6, # Learning rate for the optimizer\\n\",\n \" gradient_accumulation_steps = 64, # Accumulate gradients over multiple steps\\n\",\n \" per_device_train_batch_size = 1, # Batch size per GPU (number of prompts processed together)\\n\",\n \" warmup_steps = 10, # Steps for learning rate warmup\\n\",\n \" optim=\\\"adamw_torch\\\", # Optimizer\\n\",\n \" max_grad_norm=1.0, # Clip gradients to prevent explosion\\n\",\n \"\\n\",\n \" # GRPO configuration\\n\",\n \" num_generations = 2, # Number of rollout episodes per prompt (for variance reduction)\\n\",\n \" max_completion_length=1024, # Full episode length, not per-turn\\n\",\n \" log_completions = False, # Log completions for debugging\\n\",\n \"\\n\",\n \" # vLLM configuration\\n\",\n \" use_vllm = True, # Enable vLLM for faster inference during rollouts\\n\",\n \" vllm_mode = \\\"colocate\\\", # Run vLLM in colocate mode (same process as training)\\n\",\n \" vllm_gpu_memory_utilization = 0.15, # Fraction of GPU memory reserved for vLLM inference\\n\",\n \" vllm_max_model_length=3072, # Maximum context length for vLLM\\n\",\n \" vllm_importance_sampling_mode=\\\"token_truncate\\\", # Less aggressive than default sequence_mask\\n\",\n \"\\n\",\n \" # Logging / reporting\\n\",\n \" output_dir = output_dir, # Directory for checkpoints and logs\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool (integrates with HF Spaces)\\n\",\n \" trackio_space_id = output_dir, # HF Space where experiment tracking will be saved\\n\",\n \" logging_steps = 1, # Log metrics every N steps\\n\",\n \" save_steps = 10, # Interval for saving checkpoints\\n\",\n \" save_total_limit=1, # Max number of checkpoints to save\\n\",\n \"\\n\",\n \" # Memory optimization\\n\",\n \" gradient_checkpointing = True, # Enable activation recomputation to save memory\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub = True, # Set True to automatically push model to Hugging Face Hub\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"Mrs9bAr06H2G\",\n \"metadata\": {\n \"id\": \"Mrs9bAr06H2G\"\n },\n \"source\": [\n \"## Create `GRPOTrainer` and start training\\n\",\n \"\\n\",\n \"Now we initialize the `GRPOTrainer`, which manages the entire reinforcement learning loop.\\n\",\n \"\\n\",\n \"It takes the model, tokenizer, reward functions, rollout function, and dataset defined earlier. \\n\",\n \"The trainer coordinates the interaction between the model and the environment, applies the reward signals, and updates the policy.\\n\",\n \"\\n\",\n \"Finally, we call `trainer.train()` to start the fine-tuning process and let the model learn to play Wordle through feedback and iteration.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"FeBMCppH7rAc\",\n \"metadata\": {\n \"id\": \"FeBMCppH7rAc\"\n },\n \"outputs\": [],\n \"source\": [\n \"import sys\\n\",\n \"sys.stdout.fileno = lambda: 1\\n\",\n \"sys.stderr.fileno = lambda: 2\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"1f7aceb9-fe9e-49ba-b976-a39c1e29d4e5\",\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"f44d7bb668064bdb80e3904ff92da5ea\",\n \"efa028ffbd704a489729c83af0647d68\"\n ]\n },\n \"id\": \"1f7aceb9-fe9e-49ba-b976-a39c1e29d4e5\",\n \"outputId\": \"aa6f81a6-662c-4215-f091-bcf422f43f9c\"\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"f44d7bb668064bdb80e3904ff92da5ea\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Loading checkpoint shards: 0%| | 0/2 [00:00\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* GPU detected, enabling automatic GPU metrics logging\\n\",\n \"* Created new run: sergiopaniego-1770031943\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"\\n\",\n \"

\\n\",\n \" \\n\",\n \" \\n\",\n \" [93/93 3:18:36, Epoch 1/1]\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \"
StepTraining Loss
10.009800
20.016400
30.005600
40.014700
50.019500
60.002300
70.005300
80.025100
90.004500
100.004200
110.009600
120.014900
130.024500
140.012200
150.015500
160.007400
170.017500
180.014900
190.035600
200.014900
210.030000
220.014300
230.018000
240.014000
250.016600
260.015600
270.021300
280.021000
290.036900
300.006400
310.044800
320.026400
330.038700
340.022000
350.013400
360.025000
370.042900
380.072700
390.070100
400.019900
410.058700
420.060100
43-0.026700
440.038900
450.042400
46-0.009100
470.001300
480.020200
490.078700
500.026300
510.045700
520.035300
53-0.006700
540.025300
550.069500
560.092800
570.067900
580.035000
590.061300
600.048800
610.000600
620.028400
630.016200
640.010700
650.020200
660.041800
670.006800
680.014800
690.025100
70-0.006600
710.041000
720.008300
730.045300
740.062800
750.048200
760.032800
770.053000
780.023100
790.014900
800.078200
81-0.000700
820.013400
830.030200
84-0.003600
850.051700
860.033500
870.021800
88-0.003400
890.023200
90-0.002900
910.030900
920.029200
930.002500

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\",\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\",\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\",\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\",\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\",\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\",\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\",\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\",\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\",\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\"\n ]\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Run finished. Uploading logs to Trackio (please wait...)\\n\"\n ]\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).\\n\",\n \" return datetime.utcnow().replace(tzinfo=utc)\\n\"\n ]\n }\n ],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"o-hEO4oK4ZXr\",\n \"metadata\": {\n \"id\": \"o-hEO4oK4ZXr\"\n },\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"zuHTwuxAVp8p\",\n \"metadata\": {\n \"id\": \"zuHTwuxAVp8p\",\n \"outputId\": \"fce9bdc8-d734-4382-bb26-7e03dbffa7a0\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"12065.8973 seconds used for training.\\n\",\n \"201.1 minutes used for training.\\n\",\n \"Peak reserved memory = 38.139 GB.\\n\",\n \"Peak reserved memory for training = 25.655 GB.\\n\",\n \"Peak reserved memory % of max memory = 96.415 %.\\n\",\n \"Peak reserved memory for training % of max memory = 64.856 %.\\n\"\n ]\n }\n ],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_training = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"training_memory_percentage = round(used_memory_for_training / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_training} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {training_memory_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"13e9fd4e-e7a5-468d-a25a-3f7d2794201f\",\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"decd9f00c4da42bf92b72c327bd28278\",\n \"2d924050f7bf4e7f88316c8fc202a763\",\n \"d589783221084eb7833ae6cd742d277c\",\n \"0e135c821b5744b287b4de7eeb15d419\",\n \"a1839712ff344a409e6f7f48a1467fd5\",\n \"e9ae0fcd43e34d7e916fe1bda0a38a49\",\n \"75776d6523ef42df930ddfd7048b384e\",\n \"e2e07a449d914bd39653b7cbbc5903e3\",\n \"0eafc3f9bac14807866233f924793380\",\n \"b64c487a9dff4108a66da9eee4e4ed66\",\n \"17a3ba38cf7349269ea54df84faf30b7\",\n \"7382295b99ee4db28de43e1451dd0d17\"\n ]\n },\n \"id\": \"13e9fd4e-e7a5-468d-a25a-3f7d2794201f\",\n \"outputId\": \"7f703ed8-7874-4da1-8490-48222755ae11\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"decd9f00c4da42bf92b72c327bd28278\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"2d924050f7bf4e7f88316c8fc202a763\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"d589783221084eb7833ae6cd742d277c\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...7B-test/training_args.bin: 100%|##########| 7.70kB / 7.70kB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"0e135c821b5744b287b4de7eeb15d419\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...-1.7B-test/tokenizer.json: 100%|##########| 11.4MB / 11.4MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"a1839712ff344a409e6f7f48a1467fd5\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...0002-of-00002.safetensors: 2%|1 | 33.5MB / 1.91GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"e9ae0fcd43e34d7e916fe1bda0a38a49\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...0001-of-00002.safetensors: 1%| | 33.5MB / 4.97GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"No files have been modified since last commit. Skipping to prevent empty commit.\\n\",\n \"WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"75776d6523ef42df930ddfd7048b384e\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"e2e07a449d914bd39653b7cbbc5903e3\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"0eafc3f9bac14807866233f924793380\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...7B-test/training_args.bin: 100%|##########| 7.70kB / 7.70kB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"b64c487a9dff4108a66da9eee4e4ed66\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...-1.7B-test/tokenizer.json: 100%|##########| 11.4MB / 11.4MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"17a3ba38cf7349269ea54df84faf30b7\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...0001-of-00002.safetensors: 1%| | 33.5MB / 4.97GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"7382295b99ee4db28de43e1451dd0d17\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...0002-of-00002.safetensors: 2%|1 | 33.5MB / 1.91GB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"No files have been modified since last commit. Skipping to prevent empty commit.\\n\",\n \"WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.google.colaboratory.intrinsic+json\": {\n \"type\": \"string\"\n },\n \"text/plain\": [\n \"CommitInfo(commit_url='https://huggingface.co/sergiopaniego/wordle-grpo-Qwen3-1.7B-test/commit/2d7a27066ef244796a079cbf08fa6656af426145', commit_message='End of training', commit_description='', oid='2d7a27066ef244796a079cbf08fa6656af426145', pr_url=None, repo_url=RepoUrl('https://huggingface.co/sergiopaniego/wordle-grpo-Qwen3-1.7B-test', endpoint='https://huggingface.co', repo_type='model', repo_id='sergiopaniego/wordle-grpo-Qwen3-1.7B-test'), pr_revision=None, pr_num=None)\"\n ]\n },\n \"execution_count\": 15,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"env.close()\\n\",\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"wQyVb1nAxWld\",\n \"metadata\": {\n \"id\": \"wQyVb1nAxWld\"\n },\n \"source\": [\n \"## Load the Fine-Tuned Model and Run Inference\\n\",\n \"\\n\",\n \"Now let's test our fine-tuned model by loading the **adapter** and running **inference**. \\n\",\n \"We begin by loading the **base model**, attaching the adapter, and obtaining the final fine-tuned model ready for evaluation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"JcTeeSBXxWWF\",\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"281b1cf074fd4d60bb754906a0764865\",\n \"e129fb465f1a41c1bdf2495d14143458\"\n ]\n },\n \"id\": \"JcTeeSBXxWWF\",\n \"outputId\": \"86efafc3-1161-471b-86b1-14c43e95908f\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:104: UserWarning: \\n\",\n \"Error while fetching `HF_TOKEN` secret value from your vault: 'Requesting secret HF_TOKEN timed out. Secrets can only be fetched when running from the Colab UI.'.\\n\",\n \"You are not authenticated with the Hugging Face Hub in this notebook.\\n\",\n \"If the error persists, please let us know by opening an issue on GitHub (https://github.com/huggingface/huggingface_hub/issues/new).\\n\",\n \" warnings.warn(\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"281b1cf074fd4d60bb754906a0764865\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Fetching 2 files: 0%| | 0/2 [00:00 {generated_text}\\\")\\n\",\n \" print(f\\\" Parsed guess: {guess}\\\")\\n\",\n \"\\n\",\n \" result = env.step(TextArenaAction(message=guess))\\n\",\n \" observation = result.observation\\n\",\n \"\\n\",\n \" print(\\\" Feedback messages:\\\")\\n\",\n \" for message in observation.messages:\\n\",\n \" print(f\\\" [{message.category}] {message.content}\\\")\\n\",\n \"\\n\",\n \" print(\\\"\\\\nโœ… Game finished\\\")\\n\",\n \" print(f\\\" Reward: {result.reward}\\\")\\n\",\n \" print(f\\\" Done: {result.done}\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"MjIxHOHK4PVe\",\n \"metadata\": {\n \"id\": \"MjIxHOHK4PVe\"\n },\n \"source\": [\n \"Let's play the game!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"JjOzWexUXmfW\",\n \"metadata\": {\n \"id\": \"JjOzWexUXmfW\",\n \"outputId\": \"1c6130af-fe89-4930-e53a-7329e0483ef0\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"๐Ÿ“œ Initial Prompt:\\n\",\n \"You are Player 0 in Wordle.\\n\",\n \"A secret 5-letter word has been chosen. You have 6 attempts to guess it.\\n\",\n \"For each guess, wrap your word in square brackets (e.g., [apple]).\\n\",\n \"Feedback for each letter will be given as follows:\\n\",\n \" - G (green): correct letter in the correct position\\n\",\n \" - Y (yellow): letter exists in the word but in the wrong position\\n\",\n \" - X (wrong): letter is not in the word\\n\",\n \"Enter your guess to begin.\\n\",\n \"\\n\",\n \"๐ŸŽฏ Turn 0: model replied with -> [crane]\\n\",\n \" Parsed guess: [crane]\\n\",\n \" Feedback messages:\\n\",\n \" [MESSAGE] [crane]\\n\",\n \" [MESSAGE] Player 0 submitted [crane].\\n\",\n \"Feedback:\\n\",\n \"C R A N E\\n\",\n \"X Y X X X\\n\",\n \"\\n\",\n \"You have 5 guesses left.\\n\",\n \"\\n\",\n \"๐ŸŽฏ Turn 1: model replied with -> [spare]\\n\",\n \" Parsed guess: [spare]\\n\",\n \" Feedback messages:\\n\",\n \" [MESSAGE] [spare]\\n\",\n \" [MESSAGE] Player 0 submitted [spare].\\n\",\n \"Feedback:\\n\",\n \"C R A N E\\n\",\n \"X Y X X X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"You have 4 guesses left.\\n\",\n \"\\n\",\n \"๐ŸŽฏ Turn 2: model replied with -> [spare]\\n\",\n \" Parsed guess: [spare]\\n\",\n \" Feedback messages:\\n\",\n \" [MESSAGE] [spare]\\n\",\n \" [MESSAGE] Player 0 submitted [spare].\\n\",\n \"Feedback:\\n\",\n \"C R A N E\\n\",\n \"X Y X X X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"You have 3 guesses left.\\n\",\n \"\\n\",\n \"๐ŸŽฏ Turn 3: model replied with -> [spare]\\n\",\n \" Parsed guess: [spare]\\n\",\n \" Feedback messages:\\n\",\n \" [MESSAGE] [spare]\\n\",\n \" [MESSAGE] Player 0 submitted [spare].\\n\",\n \"Feedback:\\n\",\n \"C R A N E\\n\",\n \"X Y X X X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"You have 2 guesses left.\\n\",\n \"\\n\",\n \"๐ŸŽฏ Turn 4: model replied with -> [spare]\\n\",\n \" Parsed guess: [spare]\\n\",\n \" Feedback messages:\\n\",\n \" [MESSAGE] [spare]\\n\",\n \" [MESSAGE] Player 0 submitted [spare].\\n\",\n \"Feedback:\\n\",\n \"C R A N E\\n\",\n \"X Y X X X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"You have 1 guesses left.\\n\",\n \"\\n\",\n \"๐ŸŽฏ Turn 5: model replied with -> [spare]\\n\",\n \" Parsed guess: [spare]\\n\",\n \" Feedback messages:\\n\",\n \" [MESSAGE] [spare]\\n\",\n \" [MESSAGE] Player 0 submitted [spare].\\n\",\n \"Feedback:\\n\",\n \"C R A N E\\n\",\n \"X Y X X X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"S P A R E\\n\",\n \"G X X G X\\n\",\n \"\\n\",\n \"You have 0 guesses left.\\n\",\n \" [MESSAGE] The game ended in a draw. Reason: Turn limit reached.\\n\",\n \"\\n\",\n \"โœ… Game finished\\n\",\n \" Reward: 0.0\\n\",\n \" Done: True\\n\"\n ]\n }\n ],\n \"source\": [\n \"try:\\n\",\n \" play_wordle(env, fine_tuned_model, tokenizer)\\n\",\n \"finally:\\n\",\n \" env.close()\"\n ]\n }\n ],\n \"metadata\": {\n \"accelerator\": \"GPU\",\n \"colab\": {\n \"gpuType\": \"A100\",\n \"provenance\": []\n },\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "examples/notebooks/sft_ministral3_vl.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"UaDIwQOOjgAO\"\n },\n \"source\": [\n \"# Supervised Fine-Tuning (SFT) Ministral-3 with QLoRA using TRL\\n\",\n \"\\n\",\n \"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_ministral3_vl.ipynb)\\n\",\n \"\\n\",\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"4f0hzSo4kKEc\"\n },\n \"source\": [\n \"With [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl), you can fine-tune cutting edge vision language models. It comes with support for quantized parameter efficient fine-tuning technique **QLoRA**, so we can use free Colab (T4 GPU) to fine-tune models like [Ministral-3](https://huggingface.co/collections/mistralai/ministral-3).\\n\",\n \"\\n\",\n \"\\n\",\n \"- [TRL GitHub Repository](https://github.com/huggingface/trl) โ€” star us to support the project! \\n\",\n \"- [Official TRL Examples (notebooks and scripts)](https://huggingface.co/docs/trl/example_overview) \\n\",\n \"- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"pGXgIbj2kXEP\"\n },\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"We'll install **TRL** with the **PEFT** extra, which ensures all main dependencies such as **Transformers** and **PEFT** (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install **trackio** to log and monitor our experiments, and **bitsandbytes** to enable quantization of LLMs, reducing memory consumption for both inference and training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"V8rqrGq3hmeU\",\n \"outputId\": \"4a15adc2-e895-4c40-d174-c52e0b208dd5\"\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -Uq \\\"trl[peft]\\\" bitsandbytes trackio git+https://github.com/huggingface/transformers mistral-common\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Ou0VO1gHklS-\"\n },\n \"source\": [\n \"### Log in to Hugging Face\\n\",\n \"\\n\",\n \"Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"C5eHAVFthmeU\"\n },\n \"outputs\": [],\n \"source\": [\n \"from huggingface_hub import notebook_login\\n\",\n \"\\n\",\n \"notebook_login()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"vNylrNdqkoN-\"\n },\n \"source\": [\n \"## Load dataset\\n\",\n \"\\n\",\n \"\\n\",\n \"We'll load the [**trl-lib/llava-instruct-mix**](https://huggingface.co/datasets/trl-lib/llava-instruct-mix) dataset from the Hugging Face Hub using the `datasets` library.\\n\",\n \"\\n\",\n \"This dataset is a set of GPT-generated multimodal instruction-following data. We use a processed version for conveniency here. You can check out more details about how to configure your own multimodal dataset for traininig with SFT in the [docs](https://huggingface.co/docs/trl/en/sft_trainer#training-vision-language-models). Fine-tuning Ministral-3 on it helps refine its response style and visual understanding.\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"e0bb4423267a4572b1b9cc894edd25c5\"\n ]\n },\n \"id\": \"hOPra_x5hmeU\",\n \"outputId\": \"112a213e-0036-452f-e0a8-9295da13c3d1\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_name = \\\"trl-lib/llava-instruct-mix\\\"\\n\",\n \"train_dataset = load_dataset(dataset_name, split=\\\"train[:10%]\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"JFtR4Xyx4FYO\"\n },\n \"source\": [\n \"Let's review one example to understand the internal structure:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"vYJGczm6hmeV\",\n \"outputId\": \"0a9d8771-51bd-4b68-c1b0-d8b96da9b56d\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"{'images': [],\\n\",\n \" 'prompt': [{'content': \\\"How can the presentation of this meal influence one's eating experience?\\\",\\n\",\n \" 'role': 'user'}],\\n\",\n \" 'completion': [{'content': \\\"The presentation of this meal can positively influence one's eating experience. In the image, colorful plastic trays and bowls are used to hold a variety of foods, including meat, vegetables, fruit, and bread. The vibrant presentation can make the meal more visually appealing and enticing, which may encourage healthier eating habits as the dishes include nutritious options like broccoli and oranges. Diverse food options and attractive meal presentation can also make the dining experience more enjoyable and satisfying. Moreover, the bright colors and well-organized food placement can create a positive atmosphere, enhancing one's overall dining experience.\\\",\\n\",\n \" 'role': 'assistant'}]}\"\n ]\n },\n \"execution_count\": 6,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"qeZCtRB1m5xj\"\n },\n \"source\": [\n \"## Load model and configure LoRA/QLoRA\\n\",\n \"\\n\",\n \"This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration.\\n\",\n \"\\n\",\n \"> **Note:**\\n\",\n \"> In older GPUs (including those available on Colab), **FP8 support** is limited, so we use the BF16 version of the model.\\n\",\n \"> In that case, you can select the official checkpoint or the one from Unsloth.\\n\",\n \"> If you have access to GPUs with **FP8 support**, you can switch to that version instead.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"5fa6df349d314dd8b1baf79c1ed4eb0b\",\n \"f561335214d84bf1b8985dd4573e9ae1\",\n \"0c557dfead7d46e99af2c6e77b050930\"\n ]\n },\n \"id\": \"8dggHeG2hmeV\",\n \"outputId\": \"58ceedc1-26b3-467d-f466-e76d850aca5f\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import Mistral3ForConditionalGeneration, FineGrainedFP8Config, BitsAndBytesConfig\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"FP8 = False\\n\",\n \"\\n\",\n \"if FP8:\\n\",\n \" model_name = \\\"mistralai/Ministral-3-3B-Instruct-2512\\\"\\n\",\n \" quantization_config = FineGrainedFP8Config(dequantize=False)\\n\",\n \"else:\\n\",\n \" model_name = \\\"mistralai/Ministral-3-3B-Instruct-2512-BF16\\\" # \\\"unsloth/Ministral-3-3B-Instruct-2512\\\"\\n\",\n \" quantization_config = BitsAndBytesConfig(\\n\",\n \" load_in_4bit=True, # Load the model in 4-bit precision to save memory\\n\",\n \" bnb_4bit_compute_dtype=torch.float16, # Data type used for internal computations in quantization\\n\",\n \" bnb_4bit_use_double_quant=True, # Use double quantization to improve accuracy\\n\",\n \" bnb_4bit_quant_type=\\\"nf4\\\", # Type of quantization. \\\"nf4\\\" is recommended for recent LLMs\\n\",\n \" )\\n\",\n \"\\n\",\n \"model = Mistral3ForConditionalGeneration.from_pretrained(\\n\",\n \" model_name,\\n\",\n \" dtype=\\\"float32\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" quantization_config=quantization_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"jyklRvNxnHmy\"\n },\n \"source\": [\n \"The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a **base model** (the one selected above) and, instead of modifying its original weights, we fine-tune a **LoRA adapter** โ€” a lightweight layer that enables efficient and memory-friendly training. The **`target_modules`** specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"8wI1Cqk4hmeV\"\n },\n \"outputs\": [],\n \"source\": [\n \"from peft import LoraConfig\\n\",\n \"\\n\",\n \"# You may need to update `target_modules` depending on the architecture of your chosen model.\\n\",\n \"# For example, different VLMs might have different attention/projection layer names.\\n\",\n \"peft_config = LoraConfig(\\n\",\n \" r=32,\\n\",\n \" lora_alpha=32,\\n\",\n \" target_modules=['down_proj','o_proj','k_proj','q_proj','gate_proj','up_proj','v_proj'],\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"mBAfaiA-nbdm\"\n },\n \"source\": [\n \"## Train model\\n\",\n \"\\n\",\n \"We'll configure **SFT** using `SFTConfig`, keeping the parameters minimal so the training fits on a free Colab instance. You can adjust these settings if more resources are available. For full details on all available parameters, check the [TRL SFTConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.SFTConfig).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"FrbfENGThmeV\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import SFTConfig\\n\",\n \"\\n\",\n \"output_dir = \\\"Ministral-3-3B-Instruct-trl-sft\\\"\\n\",\n \"\\n\",\n \"training_args = SFTConfig(\\n\",\n \" # Training schedule / optimization\\n\",\n \" #num_train_epochs=1,\\n\",\n \" max_steps=10, # Number of dataset passes. For full trainings, use `num_train_epochs` instead\\n\",\n \" per_device_train_batch_size=2, # Batch size per GPU/CPU\\n\",\n \" gradient_accumulation_steps=8, # Gradients are accumulated over multiple steps โ†’ effective batch size = 4 * 8 = 32\\n\",\n \" warmup_steps=5, # Gradually increase LR during first N steps\\n\",\n \" learning_rate=2e-4, # Learning rate for the optimizer\\n\",\n \" optim=\\\"adamw_8bit\\\", # Optimizer\\n\",\n \" max_length=None, # For VLMs, truncating may remove image tokens, leading to errors during training. max_length=None avoids it\\n\",\n \"\\n\",\n \" # Logging / reporting\\n\",\n \" output_dir=output_dir, # Where to save model checkpoints and logs\\n\",\n \" logging_steps=1, # Log training metrics every N steps\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool\\n\",\n \" trackio_space_id = output_dir,\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub=True,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"bF4GtNO2ne1k\"\n },\n \"source\": [\n \"Configure the SFT Trainer. We pass the previously configured `training_args`. We don't use eval dataset to maintain memory usage low but you can configure it.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"JjLhVbO_hmeV\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import SFTTrainer\\n\",\n \"\\n\",\n \"trainer = SFTTrainer(\\n\",\n \" model=model,\\n\",\n \" args=training_args,\\n\",\n \" train_dataset=train_dataset,\\n\",\n \" peft_config=peft_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"K9Ub3jTDnfcD\"\n },\n \"source\": [\n \"Show memory stats before training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"p3UUrqCWhmeV\",\n \"outputId\": \"992da74b-7e6b-41f0-cb71-320203c1d6d9\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"GPU = Tesla T4. Max memory = 14.741 GB.\\n\",\n \"6.346 GB of memory reserved.\\n\"\n ]\n }\n ],\n \"source\": [\n \"gpu_stats = torch.cuda.get_device_properties(0)\\n\",\n \"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\\n\",\n \"\\n\",\n \"print(f\\\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\\\")\\n\",\n \"print(f\\\"{start_gpu_memory} GB of memory reserved.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"4NiFu9tcniBP\"\n },\n \"source\": [\n \"And train!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"MA8py8DghmeV\",\n \"outputId\": \"b68f35e1-cfdd-413f-b3d7-5afa4d30aac5\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Trackio project initialized: huggingface\\n\",\n \"* Trackio metrics will be synced to Hugging Face Dataset: sergiopaniego/Ministral-3-3B-Instruct-trl-sft-dataset\\n\",\n \"* Creating new space: https://huggingface.co/spaces/sergiopaniego/Ministral-3-3B-Instruct-trl-sft\\n\",\n \"* View dashboard by going to: https://sergiopaniego-Ministral-3-3B-Instruct-trl-sft.hf.space/\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Created new run: sergiopaniego-1764766746\\n\"\n ]\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py:1044: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. Starting in PyTorch 2.9, calling checkpoint without use_reentrant will raise an exception. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\\n\",\n \" return fn(*args, **kwargs)\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" [10/10 44:39, Epoch 0/1]\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \"
StepTraining Loss
11.979992
21.894323
31.924157
41.396819
51.357613
61.345677
71.356363
81.399492
91.356316
101.307108

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Run finished. Uploading logs to Trackio (please wait...)\\n\"\n ]\n }\n ],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"miZ2I1A9nnM4\"\n },\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"eUi4ww17hmeV\",\n \"outputId\": \"24b18fc3-cb0f-40a1-954d-c3b4d799dfbb\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"2492.1064 seconds used for training.\\n\",\n \"41.54 minutes used for training.\\n\",\n \"Peak reserved memory = 13.881 GB.\\n\",\n \"Peak reserved memory for training = 7.535 GB.\\n\",\n \"Peak reserved memory % of max memory = 94.166 %.\\n\",\n \"Peak reserved memory for training % of max memory = 51.116 %.\\n\"\n ]\n }\n ],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"3lrrYfPunloQ\"\n },\n \"source\": [\n \"## Saving fine tuned model\\n\",\n \"\\n\",\n \"In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"ee7d9fe2f71343a4a4832fd2b0fef66f\",\n \"7afb48c0ac514171aa052210e9063fe7\",\n \"7047f7b527284d958df2ec5e800cce56\",\n \"8a106d43be4948ffbb2a2a5281882fe0\",\n \"1f3deb9e16554fd0a4666d52fa9992b2\",\n \"4cb20a75e85a495a8a3b36eb513db36b\",\n \"7ded518429454b5aa2230f98ba0d014c\",\n \"7112497495a742a887c4c956cdf46777\",\n \"cae8cfc256f649d5aa367f923b4aeb5f\",\n \"1a2cfcd18f5f496fa4f6bcf5c0a33966\"\n ]\n },\n \"id\": \"S7TzHDwXhmeV\",\n \"outputId\": \"0d465e97-5459-4f59-84cf-d5c374503a3b\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub(dataset_name=dataset_name)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"pFq51FWEK1DX\"\n },\n \"source\": [\n \"## Load the fine-tuned model and run inference\\n\",\n \"\\n\",\n \"Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"Xh4fo-WzhmeV\"\n },\n \"outputs\": [],\n \"source\": [\n \"output_dir = \\\"Ministral-3-3B-Instruct-trl-sft\\\"\\n\",\n \"# model_name = \\\"mistralai/Ministral-3-3B-Instruct-2512\\\"\\n\",\n \"model_name = \\\"mistralai/Ministral-3-3B-Instruct-2512-BF16\\\" # \\\"unsloth/Ministral-3-3B-Instruct-2512\\\"\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"5b4514b95d7742a5a2cc777478a152b7\",\n \"fdfc2ee13efa40218ab315641eb62fb7\",\n \"9aad01ac11794c9ca74d1a442e715da5\",\n \"384a08933c11424cbc48dc28f33ec90e\"\n ]\n },\n \"id\": \"z9S319H-hmeV\",\n \"outputId\": \"6b5ef891-cc0d-44ba-b112-3c577247a295\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import Mistral3ForConditionalGeneration, MistralCommonBackend\\n\",\n \"from peft import PeftModel\\n\",\n \"\\n\",\n \"base_model = model_name\\n\",\n \"adapter_model = f\\\"{output_dir}\\\" # Replace with your HF username or organization + fine-tuned model name\\n\",\n \"\\n\",\n \"model = Mistral3ForConditionalGeneration.from_pretrained(base_model, dtype=\\\"float32\\\", device_map=\\\"auto\\\")\\n\",\n \"model = PeftModel.from_pretrained(model, adapter_model)\\n\",\n \"\\n\",\n \"tokenizer = MistralCommonBackend.from_pretrained(base_model)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"EvObNndEhmeW\"\n },\n \"outputs\": [],\n \"source\": [\n \"import base64\\n\",\n \"from io import BytesIO\\n\",\n \"\\n\",\n \"problem = train_dataset[0]['prompt'][0]['content']\\n\",\n \"image = train_dataset[0]['images'][0]\\n\",\n \"\\n\",\n \"buffer = BytesIO()\\n\",\n \"image.save(buffer, format=\\\"JPEG\\\")\\n\",\n \"image_bytes = buffer.getvalue()\\n\",\n \"image_b64 = base64.b64encode(image_bytes).decode(\\\"utf-8\\\")\\n\",\n \"\\n\",\n \"messages = [\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"user\\\",\\n\",\n \" \\\"content\\\": [\\n\",\n \" {\\n\",\n \" \\\"type\\\": \\\"image_url\\\",\\n\",\n \" \\\"image_url\\\": {\\n\",\n \" \\\"url\\\": f\\\"data:image/jpeg;base64,{image_b64}\\\"\\n\",\n \" },\\n\",\n \" },\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": problem},\\n\",\n \" ],\\n\",\n \" },\\n\",\n \"]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"yjBaVAevhmeW\",\n \"outputId\": \"47695454-856a-40b9-8ab2-d621c6e6b3da\"\n },\n \"outputs\": [],\n \"source\": [\n \"messages\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"CD2BVqCBhmeW\",\n \"outputId\": \"fe1e612b-209f-4223-eaf3-9432deb0d467\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"image/jpeg\": \"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAHgAoADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDxoDNPApQvrS4Ar6w81sB0p3SilFVYgQZzTsZ7Ume1LmqEGKUYopM0CHUmfrRRTAO9GKKWmAYoGKQmiiwC556UZoop2AKWkpeaYgHWigDmngUwEFOxQKdgUANC8U7FLipDHmPcDzSuBGAc0hXFOzSE5pgISBSgnNBWnKBmgAFN9u1ObgU0A0DFPIpM808Djml4XpSuKwqLn71KxHbpTS4wfSo2k9P1pX1HYkLYprScVe0nRL7V5SIExGD80jcKK1NR8GXlnatcRSJcxqMtsGCv4VzzxVGE1TlJX7HXTwNepTdSMXyrqcyXJHNNzmrBtGCB3ZVWiOext5F8zMvPIFbSqxijnUJMhSN5Gwqk/QV2HhvwsZCl1eqQAcpGe/1rA026TUdet7e2hMULN8wPXA616zbRgsqDoK5Z4hy0ifRZNl0Zt1aq0Ww+OBY4wSAqjoAKq3U7AEJwKvTEswHYVn3QrjmfX00jOkLNySTTFXPbNTMBmlUCuY6bjNpAqRTijHftQB8pJpEji4xVa4uhGp60TzBBWHe3e4kA1y163KrCsRXdy00uB3Ndb4dtvs0IJ+83JBrj9NiNzfKDyAa7yyXawPavjM3quWhyV30OjtXwVGe9dJA6rtGe1cpbNkit22U7QSa+TdR05XR49eNzpI5flHpU24NVGGQbFHWrJO3noK+swmLk6d27nkTjqecfE3wUdTthqemwj7VEP3kaDBkX/EV4XNFJHIyOrKynBBGCDX1zLhk5NeYfEHwLFqMM2p2CbLxRl0XpIB/Wu6GJSdnsehhMU4rkkeH59KeGzSTQyQsVdSCDyCMYqMNXanc9mE0yyO1PBqurc1Krg0zspyRMKUVHn3qQc0zpHCnA0zJpc0ASA08HmogTTgaQ7mtp2qzWbdcp3XNddZ38F7F8pByMMjc156GqzbXUlvIGjYgg12YbGTovXYmUVI0fEfgGHUC1zpWyGc8tAeFb6eleYXdncWVw9vcRtHIhwysMEV7Vpmtx3ICSELJ/Opdc8O2HiO12XCiO4H+ruFHI+vqK9uM6eIV0fP4/KlK86Wj/AAZ4Rik6Vta/4cvvD14YLpcqeUkXlXHqDWNispQcHqfOSi4vlkrMOlODfnTcUnStIVHEzlEnV60bLUpLU4BDIeqnoayQ3PNPDV6FKuc86akrM9T8NeOJ7ULDMWngA+6T8yj29a9KsNStNTtluLaVXU9h1FfNMNw8LgqxBHpXT6J4nuLGdHSUxtn5mHRh7jvV1aEKyutGcMqU6TvHVHu0i7ulQSJmsnRvFFvf7YZ2SOY9CGyr/Q1vlVYZ4+teZUpSpu0kOFRSWh82gYpfwpvWndq9dFMM560opMUvQDmrELRR2o/GgQZpaSigC3b2MlxDJIhGEGTmqtPWR1QqHIB6gGmjpUwUru5cnGyshD9KWlI54oxWiMxDS9KMUoGaYCUUuAKcOo4piE280oHrTzlutNzTQAKXB60ZPSlBFIYoAFOyDxTRg/8A6qWgAz1pecUmeaUDNIAx7UEZqaK3lmbEcbOfRVJrf0vwbqN+TJKot4VPzM/UfhXPWxdGirzlY1pUKlV2grnNgH8Keq4r0uPwfo9rB/qjdOq7izOfmHsKs2uj2EoYGygWBQMHZ2PrXkyz+le0It/gelHKajV5SSPKWXHWmlwvSvTJfCmk6rKY4Yms3AzvjOVPboazJ/hpP9pMUGpQN3w6EHFdVHNqE93Y56mXVov3Vc4MyVdsNK1DU322tu7j+9jgfjXoFr4G0/T03y/6RKOu/oPwq5bLLbzoI5IliBwUUYqMRm9KnpDU9PCZDOprUlbyRBB4a07SNJWaa0SZwB5jyjOTSWlzolvlY7CBNxyfkHWtXxNMBoKop535NeeSTMr5zXy9XHV3NtSep9ZhMvoez+BaHd3Oo2sFkzxBAT91VGAKi0e/CSAyEFW6g1xEl+7YBJwO1aen3oliIB6Vxyqyc+bqenTw1PkcO5ra94N0fW2kmtJzZ3J5AXmMn3Xt+GK821/w1qXh2SMXqIUlz5csbblbHUexr0K3uGMwAY9a6S90a013RRZ3yEqRlHH3kbsQa9jCY6tWlyS2PBx+TUoLmgrM8Y8J3C2/iWzeQ4Vn25+te22zbZK8Y1rw7e+GdWSOYbkLboZ1HyuB/Ij0r0fQNbF7bIsjATKMH3r1KVW0uRiyulJU5RfQ6Sbgms657itAMJ04xu/nWfcDk1vU2PTp7lBhg0qEDrQ/BpFPFcx0dBXyV4OKhMpRSCcinyPgdazrmYAdaxqTsgSIb254ODWM5aV8CpbiUsSM1LaW5PJ6mvGxNXqZzlY0tEtRGd5611trwv1rEsIdiDitq2ya+Tx87s4aupr2mSwxXQ23Kc9ax7GIlQa24VKgcV8/Ui5S2PMrtFuAuGUnpVua4VgAD9aihwEOaqythjiuynOdGlyx6nC0pSLBuNvfiql3OGBweMVBJN0Gazbu5wp5rWliJLRlxpanA+ONBhkL3tsoEn8aj+KvK5sxyFSCMGvXPEd+IonZz8oFeSancJPevJH90mvp8tqSnHU7otxQwS+9OE1VN3NKDXq8ptGszSjlz9asq2R1xWVG+D71cilzUNHfRxF9GXM04H3qFWyPwqUHNI7U0x2aUGm0A0DJM04GowaUNQCZOkjIwIOD610ela+U2xXLZXs1cwppQ2OnWtaNaVJ3iNpNWZ6NcQWWr2TW11Ek8D9j1HuD2NeV+KvBNxojNc2xa4sj0kxyns3+NdHpmsS2bAE7k7g12dnewXsBK7XRhhkbkH2Ir3cPi4V1yyPKx2XQrq/XufPZGDR1r1DxR8P0nD32iIFbq9r/ADK/4V5nNC8EjRyIVZTgqRgitKlJx1Wx8rXw9ShLlmiHFKCRRSE1nGTiczSJFapFkIPFV+lODV2Uq/cxlA2LHVJLdsfeU8FTXovhvxy8WyC5ZpoOgJOXX/EV5KGqxDcNGwKkgiu1TjNcstUctXDp6x0ZbxTucUYorRGdxcUmKWlpiDFHWl/CjvQIBRR7ilpiClooxQAU8AEU2nZAHvTEJjNLSZpKAF6U4etNH1pc81Qx360FetJk5oNJCFBFHtikAA606gdgAx1pRSE0hcdqVx2H+9XdKtxdalBC4+RmG76Vn+Z6CrdhdNaP5+Mt/D7VwZjiHRoNrd6I6sHR9rVSex67ZxwwxmOzRVQLwQvpUEt64L25LLMSDlDmsa31RYbZFQOruA3yHlSev1Fbllo73jrKTIJGIYPIm059RXxFRSbu2fTJwgrEkcuA0v2hEycBJBjHHSqNzr7qJIre0cyL92QDgjup9q6mXw1GIla4zNKD1Pf8Kd/Y8RswwiHBx06Vk5NE+1izk38RO9srLp0lvIE2sANwz7U228Z2sfyXUbI4x8+OvrXZS6dCLLYEHA9K4LVNNj81gEH0x0pOo0XCakXLjxnpJuQojmYDgsBw1WTogvLZNU0+4DRuN/lk5x7fWuXstKE96kIUAE4J9K09moeH7kmJmMOTujzwfeq5+bc6KVeVJ+6w8SXTR6ekLZDA9DXFyzZFddBr0WqQtp3iC2G13Pl3K8NET05rD1nw1c6fiS2mS8t2yQ0PLKB/eX+tCs2erQxyas1Yw3k696lsrtoZDg1TlJAx3FJbqzvtVSxPQDmrcVbU3WJs73OjsL8m+jB6E16HaapHIkceRjGCO9eWrYahb+XcmzuBHu4fyyQcV02l3DXF1Gig5PNRTqypSvFm/NTxEdXqjstT0+11axe0u4xJE/IPQqexB7GvOdQ0m78PXg+YtCT+7mHRh6H3rsJtcFlq32Sf/V7Rz6VsSW9vqVm0EqCSGQf5Ir6ClVjWp3vqckP3Eubuc1omuJcYjkYLIP1rfljW5TIwJOx9a4PWdEutBvFILPbucxSgfofetjQ9eEmILhtrdAx7100cRryVDqnBTXtKZcnRo2KsCGHaoAwGa35Ehuox5mAw+647VgXaPbzsjDHp6EVVSPKromE76MrXEvymsm7mBXirVzLx1rImbe+K8uvUNHohiAyTACt6zhztGKoWVsDzj8a37OHGK8LE1LnLOVy5DHhQK0bIAygHpVdVAUcc1oWto7hWSvnMVK7OWfmdLp0WdvpXQQQxopJ5rn7CGSJfmfI7YrSieV9wUk4rmozUXa12ePXi5PRlqd1VcLWbPJtNWEzISN3I6ZqFwhcrIKyqvmXM9CIRtoZNxMeTWPe3IiRiTz6Vt39i6qXhO4dhXGai8jzFCDkHGDRRpXlqdVNJ6mBrLXeqo1pbQtLLIcBVFchqng3XdLybjT5duM7kG4fpXufhfSI7KP7RIoMz/pXUvEk6kMoPsa+hweOVNcsEc9Wq1I+Q2BU4PUUA/TivefGXw0t9WnN5YKsM5HzqBgN7/WvLtQ8B6xYXiwPbswZtocDivZhjKclduxUJ3ObXJ6VYTd2r07SfhlboqSXkjyYHzKOBW7F8PNEQ7vIJHbLGuSWa0Nk7nQpuJ47G5BqypJr1mfwJpkG4rb/L2PWsHVvBUUUHmw/Lz2pwx0ZM7qGK6HDg0tWLvTp7NjuXI9RVTNdcZqWx6UZqSHil3c0ylGaoslU8U7ODTF6UooQDwauWd/NZyh4nx/WqWaAferjJxd0O56BpWtRXqhSdko7Z/lVTxF4SsPEUbSYFvfdpwOG/3h3+tcfDM8bhlYgg9jXU6V4hDARXR57NXs4TML+7M5cRhYVY8rV0eVaxol9od21tdwlCPusPusPUHvWZ0r6CvLKy1eyNtexLNC3T1X3B7V5X4n8EXehlrmDNzYk8SKOU9mHb613ypKS5oHyuMy6dD3oax/I5DFNzTyMU3HNc+qZ5jFBp6moyKTOK2hWcdzOUTobeCS4kCRqWb0FLcQNBKY2BDDqDV3QtSXStWgunQOiH5l9qs+KLmyvdWe6sgwjlG5gezV6vM+a1jzDEFLSCjFaAApQOaBij8KYC0dqKcoyaBCUUp4pKYhR3pSfSkFHegdgpccZpQCe1G32ouVYTFLjNOHSjeMY4xSbFYQCnHGKaWx2phyaVylEkOBSF8e1EcTyMERWdj0AGSa3LHwlqN3hpVFvH1+fr+VRKpGOsmdFHC1artCNzAyT3qSK2mnOI0ZvfFdkPCsFquQrSuOct/hWZfF7YlMhAOwrzsRmap6QR61LJJ25qrt6GRNYC3hLTSgMeiLVrRdP+1XSlwfLHb1PaqIWS6ugWY49+1eo+F9Igt1SSQ5DAYGOtfO43HTrP33sdUMPTo35Ea/hLw4Aq6hfIrSKNsaHsO2a7mSGMqjopBXqAKjtDD9nCA4+gqVJvJlIYEqehrgc7mDvcbI8csJKuNw/hYVV+0FbYjG7P6GprhF3MVHB5x6VjNdGCco2eT8pNYzkyoxuSX1+qwLjhsYNclqMingYznJrV1aZQPl+93rmLhy7E45rBttnTTjZF3SBHFM07jjOB610l1Db3UKnYxB+9k1xtq45VjgHpWvY3kkWVJ+UnHJ6Gmm7hOPUtf2PYSRmGS2DZPPrVXVfB7aRbjU9LvtgUZMcx4H0NOvdaWxJluGBUHt1Nc3Pr1x4kvxHdOVtIhlYgcA+ma2hJihz3VmMvby21uBvt1hbGYD5ZoRscH1yOv41lW0KW8g+ygoegbOSfxrQvGsmUiINE68FM5H4Gsye5YLsAwF5HvStJ9T0IVEtDQh1KdWKeYV5ywBxzVsvfxWv2yCcSRKeSPvJn1Fc/O83EqqdrAZpba+nj/wBW7qemAetHIupspvdM05dUlmk3zbJHIxuZATWzp2u4tpFku/I2r+7VUyCf6VybXIZgrRFW9auwWbzIHCOI9wUknoT7UQ5ovQKtZNWud5oFz/bem3UF4BcRvnajgYPHQehrjNe0OXSZEuYNz2UvMcnp/sn3FdLoEEtrcyxBVKwrkrnqcc498VpeHr221C4vNDvQjxSsTHu6Bv8AA1pQryVb3noPD4iVFua1XU5LRtc+7BcN7KxrpGSK8i8uXpj5WHVa5rxT4Vn0C6aWIM1oTw3UofQ/403Q9aEbCC5Y7TwrelfTUMR9iex7NoV4e0pia5p8uneWXdXWTO0r7VkW8e+TPNegSW0GoWxguF8yFuQR29wa52XQJ9MnG754GP7uUdD9fQ1xZhh3GPPDY5nNr3WLaQYUYFa1tGcgKMmoLdAqj1rd0uxZpFkb8K+NxuIVNNsynKyuWLPSTIA0ucHtXT2Olo0B2YAFV4gAVB6Ct22MaRKFPX0rzcA4Ymo/abHj4qtK2hBZWoQSIeW9DVeOb7Lec5C5wavXmIHWWM4OOR61l6hNFMVljb5m+8Pet8dCNKKlDSUX96OWneb12ZZu18u5Ux/xciqt3ITKSRg45FVJbt2SP5iGToaia5aX5nOTXkVqqqX5Va7OmFFq1zRtpkkRoX6kcGsu60+CaTfIi+bGeDSeYyOGU4IqKSdnkLE8miNR8iit0WqbTujU0xwZlB6DtW68aFCykAiuYtJtrZzWlNfKICN3JrpwmIjTTjI5a1NuWg+e52DBIrLmcTyYYA8/lVd7gs4GScdakh+YlgazxGKm1yp6G9Ojy6k6sc4xxjtU0ZA4Kgg9qaqkjBxmrrQFLeNwnPqO9YUIVKl5LoKcktCytmLmweNcKccE81lXGmwR2ci3A2yKp5ByPatiz1SG2tnWRTnuPWuV8RawHQhRtUAgHPUV9RCtho0YyUrytsc1KNWU2uhwOsLGZCvFcRcKEuXUDAzxXV6jP5jsR1NS6J8P9U8SXiSlDbWefnlcYJH+yP61vhJu92e5LERoRvJnHBTTwDX0fa+APDlvpwsv7MhkTGCzjLH3z1zXl3jb4d3OgM95Y7rjT85P9+H/AHvUe9epa6uTh83pVZ8jVjgxxRj35pSMGj8KlaHrdLoPrS03mlH5VQDqcGxyKjHWjNCFc3dK1yS0YRyEtH6eldlaXsN3CSpV0YYZTyCPQivMkPfNX7HUZbSTcjYI6jsa9DDY6VJ2exM4qZP4r+HqXKyX2hoFYDL2g7+6f4V5jLC8MjRyKVdTggjBBr3vSdWjvVBDYkHVaq+K/BkHiu1EtoscOqxj5W+6Jv8AZb39DXtqUK0eZHzWY5Zy3qU18jwqm1Yu7SaxupbW4QxzRMUdD1BFQDpXPKLTszwdzaFLk0hFLjAzXvnlBmlB45pAKWgAyM0ueKbTqYAKXNIaM80AOpVAOc0gPrSFsUrhYf8Ax0pwOtQmQ9KNxPAzSuNRJd6rTTIe1MCnvVi2s7i6kEdvC8reirmk5WLjTcnZEWSe9G32rrNO8DX10A124tk9PvNXWaZ4Q0yw2sIfOlHO+Xn9KxniIRPUw+UV6mrXKvP/ACPPNN0HUdUOLe3bb/fcYUfjXU6f4BRcPfTlz/cj4H512jNBbrgsOP4VqGW9yAIlxxyTya454uT2Pdw2TUaesvefnt9xFZ6NZafHiGCOMD+Lv+dPlubeMYUF2/SqzuznLMTVOeQKprjnVb1PXhRjFWRW1XUnWJgDtHoK4S/naVzzkk1s6vdbmKg1zjuTIxryq03JnNip2Vkb3haxW5vMuwUjpuXIPqDXqlnF8ke6NVHoBxXG+CrDFrDcPGRGz/Ox/pXoUkDx4kjIeIHjHavMrrU8qUraF60TH3cj6Grg+ZGG7ketZ1ldYmPze5q886MwKDPrWatY5pJ3IPteS8UowexrF1QbZOWJ28g1oaogKiSJSeOcVzV/fllAJ5HFZzb2Nacb7FW8naVyc9O9ZrN1596sRShZCTyO49apXDgSkL0qUjew5fU0/wC2rbqXmUkDpjvSW4M0gQYzgnNUtVAhj3KSzEfrVK1ylG+piarqUt7IDKcKhIUDjipNGgkZ5XlJiBj3Rbv4+emak0vShqL3FxK6fuRuKH+Ktf8AtK3t0topbVTtA3nb95e1b3SQjJ1KJpZWeTaJAcHHFVfJldijDIUcDP8AKuj1F7e3iG1opSy4G1gflPQj/wCvWRY3MsjiCRyVhicxq3bPpTTXUab6Dbe2lkQNFggAlgx7DrVjyltreR9oOOeecfSn2wj+x7uQ5PrxU8hjltpgo27scCsnI2TY9Z4Hjjykcir85R+Oe/IrYsPsE8V0j2XlyxkPFLFITt54yO9c5EWjk2dD0GRW9oSyz6nEqnBdgoI7+1ZSm+gOCerOkSASX7zQxqcxgEocAn1+tcHJcS6frBuIuCsmcH69K72aNbN5CrMjquWHToa5XxpZRxXkN5bvuiu4/MIAxhu9RC8kzalJRkl3O9hvLXX9OjLFX8xdjo/Iz6V574k8JS6Wz3VirPajO9OrRf4j3rK07WrnTZP3chA9D0P1Fbn/AAmd7cSBSVCntjt6V3YfGypx5Kiujqo89CV6b07GfoutvassFwSYj0PpXawzRTw7H2yQuOlc1rOii40SDVrO1RXBbzkh4yo/i2/zxWbomttZuIpSWiJ/KvcwuJ54JVFozvajiYc0dzrf7LaG4BGXgP3X/oa6G0jCxrjsKoWN6jIMEPG4/AitREAUNGcp/KvleIspnFe1o6xPMrOS91luFhvGRWxbsjJwMAVkRrhQx61rWwAtW3fexke9fLZbJxqtHlYi1h90N3yvwCOK5adzDOyH1ro7ifzIgCPmHQ1zN/HNLcFlRjWuMrwqzsmaYRW3InmBzURmxSx2txI2PLI/3qnGkzMeZFFcd4R3Z3twW7KzTHHWozIRWtFoqcGSUnHUDitMadaPavAIgNw6qOaIVKbdrnPUrxjscylwVB55pGuSRya0dQ8PTRRLJah5P7ynqfpXNyTFSQcgjgg8V0Kj5Dg4T1iacTnOQOver1sxDkCs6Bv3aD2q1FOYyDxWFSN9DVx0sbVo8aynzRlSPyqd74pCsadjnJrMgZrj7i/rV+3gE8yJJwBwSKuj7eypw0TOGpGKd5Gbd3W1STxnrXN3Frf63N5VpEWUcbjwB+Nd02hwTP8AviWjz0Hf61tW1nBBEqxRhVHQAV6uDy6UXeW5nLGKC9xanG+HvAMFjdx3l7J58y8hcfID9K7yKFEAAUAewpEAB4qcV9DQpqKPOq1Z1HeTFAAqC6t47qB4ZUDxupVlPQg9RU2aQHNbykrWMlo7ngPjzwTJ4du/tVmjvpsp+Vuphb+6T6ehriSMH3r6p1LT4NR06ezuFDwzIUcexr518W+GLjwxq7WrlpYGUNDPtwHH+Ip2PpMrzDmXs6m5zx9e9JnmnZpueaEe9uLk0m6kNMJpk2JAcU8Nz15queDT1Y9qGBds7yS0uVljYgg+tel6bdrdW0VxGeGGa8o3HNeieECx0OPIP32wT35r1ctm+dx6Gde3Lqcd8WtIihv7XWIV2m9UrOMceYv8X4gj9a80zivdvijbwP8AD5Z2wJI7xAmevIII/KvCDXpVraM+JxkFCvJRNzvRnijNJXtaHgC5x2oJoyMcmm5zTKHA04kdvxqMNzTufwpMVgyO1GaTgCge1DHYNx70oAJ5Jq3Z6Xe37bba3kk91Xj866rT/h9M+176dYx1KJyfzrOdSMd2dlDA163wROLC5PAPWtrTfC2qajgrCY4/78nAr0iw8MaZYhTFaIWH8b/Mf1rTZ4YhhmyfQVyTxdtj28PkS3qy+SOS07wHaQ7Xu5Wmbuo4WuptdPtbCILDFHCnsMZpGu3+7GoUeveoDljlmJ+tck68pHt0MFSpK0Y2/MtPdxJxGu8+vQVWkuJpOC21fQcUwgCmkGsW2zrUUhMYpCaUnFMZqRQ1iADzWTqFxsQ81enkwDWLM0U0xE0hWMdSvWuWvUsiJyUY3OeumMjE1ntGQ5B616C3gZ7zTjf6TdC6GM+URhvwrirm2mhuHimieORPvK64I/CvOc7ux5dapGauju/BE8z2pspCwhPzR8ZB9a66OWa0lKjmP0PSuG8H3dxbqI8ZiPKnH3TXoAEN1aAK53HnBrlr7nFIpqZkmMiLwew6Vat7878sQGHUYqmWa2fJYkDtUqy28gJKNz3HauUlmjcSl13J93viuR1iEhzIvHPSt37THEu1XIz2NUL1VuIWHftim9Qhozl/PIb3pjHcc5yfUVDfB4pSeuDQbloYYBnDNlvwosdMVctmT7PEFB/eOeo9Ky9QkM1w6NyoHGa02H2mGNplKOASGHRvSsa6lLyF2GGz1FKOmrNZbWRNoyCNrhkPIwMdc+tW9Ttz5kT5BVYwq47fWoNMXakvX5vQdcVNLPJJFg8gd6JTEo6FTylfB25OOc1FHAhnTZvSQdSD/KrcDpuPmKxAB+72NRedmaOTOGPUihN2DQtJDcRHB79Mjr2rW06zLGaHCuOhK9qk06dLqYK+W2AEfQcmrmpQ/wBj2huh/wAtucdME1nKXQLnO3hYynIJMZ2knv6V0nheweO9aaRcqIiyk8ZJ6GuPS6e5ZYEbJaTcw9K9M0q1cXMECspRY4twDe/I/OqnpETnoN1kmC/kU7SjRqjgj88VlawkQ0uyaQCSILtzzxz2rc8TYuIg0BzJMhZQeu5Tj+VYMd/BeaW1vdIyBF8tyRyjdQcVzRvfQ0g/dTOY17Sba2uC1tMZInXcrY6e1YNsSk+CcjNbct4RZOzMjoGMYXPIrmmnxNkdz0rsjF21No1LHuthaCDRbHacE26uMDhs9a5PxF4SS5WS/wBLULMPmkgA+97r7+1dH4K1aHVdEgsJGxPAn7pieq91q9fW8ttOJI1+uK6KeKdC19YsihXnTqNXs/zPLdI1iXTpvKmDGPOCp6g16BpupBkWSNwyMPwNZuv+F49Wie9s0EV71Zc4WX/A+9cppeqT6RctBcIwUHa8bDBU17tGtGpCz1iz1m6eLhp8XY9hheOWIMh4HUelKbpkG1TkYrmNN1QFFkjbcjVrGRXj8yM5X09K+RzvIXRvXw+z3PHqYZwlaWxbE5bqaYZcGqZl5xSeaSa+LcHfUFSLhkzSiQ55qqH9+acrVLiDgXVYmrMTlSCOtUUPQGrcRrKWmqMKkTSjd51wx6CsHW/Dkd+r3Fv+6uSMjHRz6H/Gtm3bDgbtuetXTGNuwnIzwRXtYLnrU+a92ji53SleJ5Jp+rwvI9rO3kzxEqQ/Gcdf1rXt5FuQDEwfJx8vNZXifwfdXnjKf7IhWGbbK0jDAUng/Xpn8a7Hw/oNto1mkEfzvnLOepNdNbD07rler6HfLFLlvYs6RYTRbi3CsMFa3re0SNfujNNiXHT9KsgE+tenhKEYRVzy61WU3cdsWnADaQOhoC+tKVAr0bNanOMU7TUquDxUO4AH1oUFcHoDUwm4vQGizTe9OzkVEx+fjpXRPoyUSA5rE8TeH4fEOjXFhKQpkXMchH+rccg/nWwTnihiNmD3q4tdRxk4u6PlPUbC60u/ls7yFop42wysP1Ht71VznvXufxF8JrrumtdwIP7QtlJiI/5aL1KH+nvXhDFo5GRwVZTggjBBppqWx9ZgMaqsLPceTTSeKTdkdKTqeKdj1E0GaVT2rU/4RvVjFG4s3w4BXPHFbel+CXdg9++B18tP8a6qWEqVdUtDL20LXRz+laXPq94tvCNoJyznoor1PT7GOztorWL7ka4Ge9LYaXDZwCK1iREHXHH5mk1bWbPS7OVoXEs0aFmcfdXA/nXs4bDxoLuziq1nJnnvxc13c9v4fjIIt28+cg5+cjCr+Ayfxrys1c1K8lvr6e6mYtJK5difU1SNE58zPksRPnqOTNvNGe1AxS/QV72h4wmPzpRjNTQ2k8/+rjZvoK27HwnfXaq3lsFPc8D86TkktTSnTnUfLBNvyOfIGeBT44JJWCxozMegAyTXoNj4EtoyGu5S5/uJwPzrprTS7SwjAggjhUd8c/nXNPFRW2p7GHyStPWo+VfezzjTvBepXu1plFtGe8nU/hXW6b4K02zZXmVrmQf3/u/lXQtcQR9MufbpUD3Uj8DCD0FcVTFyex72HyihS15bvzLSRQ20YUBY1HRQMfpTWu0U4jTPuapdTycmnqpIrllUbPSUFEkaaST7zfgKaBTljPFSpC7NgKSfQVHMhuSRDz6U0irDQso5BFQEYqk0wjJPYQ4qN3AokfFVJJfepckirExkB6Ux3wDUAkqtcvLJ8ikKD1JNY1K6itSkh7brpzHFz6msTUYTDMYxyR1rqtFFtb5zgsR1rJ1yEfay68jGa8qrVcncwxC91oteDtefTbxbec5tpPf7pr0bU/D2neIbPFzGpcj93Mg+ZfxrxtFw46+or0vwXrouLf7BO+Jovuk9xWLd9TxKkGtUYdhHceGNRfQ7yEPDK26OUD9RW7MpSJHiX5QcZFdJqukQ6tbKsqgSIco46qaw1jNrA1pKpyjc1nPVamXMmihcEtjeMKe9QoXQMA3tV+4hjMYZB9azJ02SDDHb65rK1gWpFcOSeeq1UNw+CmSPSluXcgkHOf1rPa4JkVW5HfHpSLjC5Hd2zzk5G0dT6kViTSG5vV3cBRtH0FdSzfumXOWHKjrWELQSrNlR50ZLZz1qldnQtFY1EuYF014mTcRGSGBwQawYYGu50UA7QPmPoKHuGCOvUMAM1saWIoLYq25S4+ZuoqCmQvJHDIfLGEA2gelMnZEiQLtJxnPrVrV9PFk0OJVlSZN+5e1Zsm3ywueRmokmnqNNNXGC7kjZmTAJGCCO1QRTHKgoDg0XDCMDOOnUVDG244GM1rFMiTN7ShIk4k+VUPRicAVc8WX0kmlWcbn5kBDYPWqmnoHt5QTlGOGUdRgZzWXr9z5yKqnsABU8nvEt6FrwlAk9w80pPXaGHYY9K9Q8Lqko80OHdwrH6jIx+leZeDbVptYtbVX2mVwhz05616Xplo2leKdRsQqxxoI5YWI6r3x+ZpV46XIclbl6mRrjSrJb2+4hllcDBwQD7Vzd1fOl/HJlNlwvlyIBgAjjJrp/F0yw61ZSFgdsjbiBzwRXL6vFavqzKJB5UkvYdM1nS0RvF3imc1PDLa3NxbS8Mh9c5/yKyZ0dMMVIz0966W4tS4nR+J4jxkdccH9Kk1LS0k8LQXWEE6S7Tg8lSO9dUJJuwm7CeDNWktNRh2tjB717uyJe2MM4HMiDcPevnDQvl1FFOByODX0L4clZ9MWOQ8DkUQSc3B9TPEp8qqLoOt7ZIVmG0FtpK8ZrkfEHh+DWRuJEF0v3Zguc+zeor0RE/fbiox0Ncfq15Baa6+nuQkpUSICeHB9PevYwfLD3HsxYOtJ1Hy7nnEE974fvza3aELuII6hsd1PeuysL0MiyxNujcdOxqTVdMtdZtBBOCrLkxzKPmQn+Y9q4yKW98N6iba7QmInIIztcf3lNeilZck9Ys96M4142l8X5nfMAR5qcoe392o/Mx3qtZXqyIskbh42H5irE0YADocxn/wAdr4/OchUG61FaHK4uLsywsoMYB6+tOR+aqIasJ2HevjalLk3IlGxcjPc1ft8Myj1NUYkDYyTWlaoBIMVy8vPJRRxVmki7LalCu3n6Vbt8rGVbk1bjhUwqOh65qERjduY5z3FfVwyl4aSqw69DyXV5lZlW/hDkP3HBpkESJyetS3GSjAdjmoo1I6muebSrXsWr8ti2sqgcCnGYA9KhGFHNODKRXfGtK25m4kyyhqVmzVc8EU/OFrohVctGS4ik4bpSCQM2SeBUE2SDzjPpTEDcHIqfaNSsh8uhoCQEcGmu2SCPzqFJADg0rSZHBrqdS8dTOxMTjAFMY5HNQiYL15PakeZVXczhR6k1UHzbD5WR3K+YhHoK8N+Jfhue31R9WtYi0M20TKoztk6dPQ8V69e65AmRFl29e1c9eXUl4xLAds8ccdK9PC4CpJ3loj0sHRqqXNsjyDTfCesXyrI8H2eJujTcfp1rsNL8HWNk0cs264mU5y3Cg/Stu61C3tXInlyw4Kjk1j3niZ2DJbJsX+8etezDD0aXS7PcXO1Y6C4lVV3SuqqPU1mT+ILWCMJGpkYdCeBXLXF5PctukdmPuagAyRV+06RRSpxSszam1q7vQE3kR54UcCuc8YXn2TQTCG+e4YJ+A5Na9umMHFcD411A3Or+QpykC7B9TyabbUbs5MfVVKg7ddDlmOTTaU0mK5Xc+Tvc7XT/AA3e35+RDjuT0FdVp/giGNN11Jlz0CjNdcBFF/rGVR1wPX6VG96i8Rpk+rV6VTGvpoe5QyKhHWpeT+5Fex0OztFBjhyw/iarzPDEPmcFvReapPPLL95jj0HSpBp90YFm8iQxk43heK4amJu9WerCnRoLlVkh73x6RoF9zzVd2kkOXYn6mtqy8L3t1AsiAAk/db09a6AeDrdrFARIs6n5pFbII+lcssSkZzx9Gm7JnChTU8NrNOcRxs30Ga9Ci8KWsNgcIjThTscjOT2zV6x04R28W+ONJSPnCdDWUsQ+iOWebRt7qOFtPDd7cbT5e0HpuOK6q08G2gg/eq8kh/2sAV0s9mslqI0OwjkECqNndlJjEXBAOK5KuIcZJSe551XH1quqdihB4XtQpieNCB/ER81XLTQrS2+dYlLrwDW0DuHHWkyd+COK0tpuc0sTUlo2UZtKtri3aN4IiGHIxXB6l4Uv7cSSBYzGuTkN2r0sZY9OBTZoVmQqwyCMEetawk47F4fGVKL0eh4HcyEEis6a5EYJJr0X4j6ZBbwW1xbWZV2JDvGvBAHGQK82Ons48yYkL121NStY+lwmJWIgpIfp05mlLP8AdzirOrjfiaLhQMECqYZYl2xjApUuicq3INcEqjk9TuSsrle3u2jfg1ZuZ1uNoJrNvYjE5dPumqy3LLIhPY1PKYVaqasy6EIbae3SrtndPZ3Md1EfmU81Fs8wgg4JGQfekjPylWHDHH0NZXPPlE9h0HVor61A3AtjI9x6VHr8B+zNPH95evHUVxPhrVGtzGp6xt1/pXosxS7s2K42Om4d/qKT1RwVIcsrnBC6dcjfkGqk1xtkxu496rakzwXMkecBW4xWc87E7iahIpI0ZT5pVVYAk4wTVWXT5YXkMh+YHC+9RQyCW5iTPVhW5LdG2lWTYJUj6hh2qZOzSNqasrnPXV5JDbxsgwUBB96qwXJFm4OC0vzFu4rT1q1t5bpZ7WUiCcj90f4fWqM9uY7p4lUBVxtx3rR2sNamU6fN1NadvcGO0DHqOv0qnPHtc8c+oqyqgRFc4yOlZXLsOuLvzbUoVA4G0ioGcmJUPKjmpEg3jaByOab8vllSvzZ4NK9x7FG9D/KAMrngiqw3G4VcFQa0CA4Zc8ilS12ASAbiB0rSMrIyaLlit7Dcx+UrBnyACMBgeKlvrKO8114Vj8ryztCe9SabIZLiDz3kEcbYBXqoz2rUeCdNafy3RnuCZo2UdxyKmUtSOtiho8D2Wu2U+CiCbGfcda7LxPrIi1pb2Mb1ECFiDg9xWh4m0eyfRDqlrEqSKySOY+AGIw1YGiaXFqTSQTs5aRcKhPHAz/Os52v72xEGpLn6oxNf1ae+u44o0JkYCQkL6+n5VmiAvebmADlQ2M9CK7HV4ra18RxRRxKjCEB8fn/KuR1dZrTVpEK4MbbfqDyP0qnJWtBG8NbXLesb5ZluliH7053J+oPvV7XL5b3SpHhiAYqqzqo4Pow9+KSw1GBtEWG7Ubkl3rgcjjFULiGe0so71GBgkcjrnBHIzURkuZWNHFdTkw3kamjqchsEYr3TwdeJJpK7uWznFeE3rYuPMUAAncoHau/8BayZ5PI34ZACVJ5I9jW1STg+dDcFUpuJ7BbzK0hIGPTPpXlHxgkFtrGn3EeVm2sCwPYYI/WvS7SRWcHrkfWvO/jRCn2CwmA+dJME+xH/ANauyNW9NPzOLDfu66aM7w14rS8Rba8cLN0Vz0b6+9dPdW1rqNq1vdxCSJuh6Mh/vKexrw62uGRuDiu/8O+Kwyx2l83skp/ka9TD4r7Ez23FVFzR3JHju/DN6sUrGazlb93KBgH/AAPtXS2l4rqGUh42HTsRViRYbq2eCdFlikGCp7+49/euYa3uPDlyquzT2ErYSXHKn0Pof516EbW5XrE1jJVFyz3/ADOpeLywHTmI9/7vtU8J5qlaXY2qch4mHPcEVoLGEAdDmM9D6exr43Pcl5L1qS0OWonHRlyL1q9BJtcH3qhFzVyLgHPWvgZXjK5w1Ubsd+WiKk4PbFPe4VzhRwB1rHiYkjFasKIsYP8AEa+hwuOr4iHI3sebUpRg7kDyHB96jDnNSuMSHbyP5U0RMQfl49a51TqOW9yk1YfuJHtT46rPKF+VTUkblmHPSu2i05WJa0LWwk1KFG3BHFIJAq88DvVZ7sFmCHK+tevGMIK5hqxZYh2ao1i25Pek3jOWOBVe61ezt1I372xwqf406eHlVl7sS4wlLRK5aZtoqrNfxW4JZwD6Vg3WuTTEiPCL6Csq4vI0Be5nC57ZyTXrUMok/equx30sBJ/EblzrfJEK49zWXPdTyq0ksnyDqzHAFc/P4kjjyLWAFsjEj84/CsK71K5vHLSyE57dhXrUqFCgrRWp6NLBRh0Ohudet4AQmZW/SsG6169nLKshRD/CvFZxOTTTWrqSZ3KMY7IVmZySzEk+tMoJ59qGdR0FSDYqozEADJNXoLGJFElzOq5GQqnk1m+a2cKSB7VPbgswJ5NVHVmUk2Xi8UELzO2yNR1P6V5TrOnXVvcvPKfMWViwkHQ5rrvG18be2trNGwzHzHx6DpWFp+rJJEbe6USRMMFTXo06VOceWW581nNebqqEdonL0ldBqWgNHEbuyPm2/UgdVrAII61w1qEqbszy4TUlofRWnaNdak2IkJ9SQa24/A94SzPPEsSjJY9fyrtxEiurQAxxouCqjANUL/UfInkaN3VgMFWHyt6V49TENantTzGtUfuuxXsPCumxSQXBcOQMY+8rGtyO1t7KJY0gxEDwFXI/KuatdUnubwCIkIP4VHANdjZSM0Q39cVjCoqjOGrUm37zuV/LmEx8pF2Hv0/SnRQFBhgOvNaWKQqPStvZmPMzNMnlsRtyo7AVDPqLICYbQsw6ljgVpmJc9KTyEKMuOtZOMujC6OZubvULlNsknlxnqqDH61HY/ubhW+8A1ac9s0TsMfLUcUCqTwDk5rypwk5pvdGykrG2kiuMqRU2Misy3I3cHoa0VYADJ5r1KFS61MWhSORinUhoz610kjJIkljKOoZWGCD3rznxL4Rjs1llt1P2dhwv9w+n0r0k4xUNzbRXUDQzLuRhgipqQU1Y6cLiZYefMtj5suomhlZD2qo7EV6n4p+H0qRSXWnuZlXkwkfP+HrXl9zBJBI0ciFWU4II5FcLpOL1PpqeNp1Y3iyKdnaAGs64VioIFbIj3RL3Heq0kO3IxxSTOWs25XF0u5MsJjJ+dKvlBJITkDf+hrn8vaXAkjPQ1swXEdxGCh4PbuDWdSHVFUpqS5XuXLeZoJc8jJwfY16V4W1BbqzMAPzYLICfzFeWlzgn867HwK4OrrGWI8xfkwe/cVMItuxGJilBszPFQ+y6w3y4VuQDWHJKCvpXY/FKzFrc2lwvKyAj8a8+84kYqvZuOjOSM1JXRr6LF9o1WEdlO4/hV7WWZZG2ttUgqSDVPwud+psueTGam1uMxzoh+8WzmuOUv3tjrpr3StZSYRYCm6QsNpp158l9cAk5U8Zplog/tJCCSFPb2ou2eS5kdvvM2TWjasVy6lScLLJGqrtyecHNWpYwiiQdCMc1XUZvI8jFT3oZHMZ44yKybHYW0VDK5dwmF3A+vtVSTYFLbvmB6Vfigf8Ast5yPvcA1myY28iqJI1x5p4OT+tXPNCKFCnOKqwNECMnHOKtBwdygHgdaCWSW7FXZ1+UDH5123g+O31C6knkYieGPajZxgnjp3rjbMRqoM+4rjACnBrtfCs8dhBEl1bgbi25413FeSct+FLR7mFa6Wg61ku7LS9asJ0aeEI+5TyAwHUenaqVxqMtnoui6zBEI5IZdkmej5HX8av6bqaS+Ctavt++UebGd+eQfun9az9ZkhPg3TrWQj54yQy9NyjIpSjfRmcHrsXvG1kxnttdgTdayxLG7J/yyc9M+xzjNcZ4xlaPV0k24YRoj+jEDg/lWvY6691o82kNKTDNEVKnt3H64rm5rtL3SC05LSKTknnJHHWqVnsbwjKKs/kS6ncxTaRbyw4DjIYCobLUhLpU1hOTsJ3oc/dNUXcGxRQ2e+KpBmSIkcGqjBXNrlaZsMy9s8Z7Ve8OXElvrdm6YP79AwJ6gnkVm3GS45/GtXwxZtca1aIY2dElVnCnBwD2rptFRuzCpKfwxPoOywtw6gHAYg1wnxmb/iT2x9ZAP513dhkqHd8N6muL+MFoZvCsdwG+aKYEj1U8fzIpU1+7XqZx0qnhquQRzV6CYgDn8qzs1NE+DxXW0ejRqNM7zw74pa122145aHord0/+tXeI8N3bsjqk0Eq8qeVYGvEkkPFdN4f8SSac6wzEyWxPI7r7iuzD4lx92Wx2tKorrc69rWTQ5shnl0tz948tAT/e9vety1uvKGOHiYduhFRWtxFd26yQukkTj0yGHoRWRfNLoVwJVR5dNlbG3vCx7Z9PT8q9JuMoWlrEn4/clv8A1+J1ycYeNtyfqPrWnboZmHIGR1rkdP1JZozPbShgOq9x9RXRadrsMZxcQhh3Ir5TG8M0qlX2tLbsefiaFSKfKrnQiySKNSpJPc9qtBU2gjGcVWg1bT5wFSVVGOjcVIpSRvkYEexzXFiMveF1hDRniS5/t3RRuZ3t78Kf9XKuR9R1/pU8Upc9xn0rO8RqYhaTgn5WKY9cjP8ASjTtRUx/vMgdiRXkqhXVbROx0KHNTUkXFwcrgEnvShljJHpVK61O0ilDRnDDrisu61xpC3lpgnqa9XCZTiKj2t5mkKFSpsjYnvlf5GYBQeTntVK51iGIbYRux3PSsRpJnG6QhExncxwMVm3erWluSEb7Q3qDha+kw+U0qfvVXdnZTwMb66mxcalcXLEljjHQcAVl3Go21uG82Qu46KnNc/datPPld21fReBVBmZupr0k4U1amj0oYeMFY1bnXJnysA8tfbrWTJK8hJZiSe9NNI31qXNvc3VlsNPSmHNOLYqNjxxU3EMJ600n0oIOaafQc/SmJiHmmn9acUOOeKcEA6fnVpEsYoJ7YrRtYwoBqmgywFSajdDT9JnuCcFUO36ngVrTVtRSainJ9Dz/AMUX/wBu1udg2UQ7F+g/+vWKrlSCKSVy8hYnJJ5NNBohV1PjK83Um5Pqbml61JaOBnKnqD0NWNR0iO/ia904DPWSEdvcVzoNX7DUprOUMjHiu+FWNSPJPY4ZQafNE+xhLC0TxvIQ2eA3GRXLa6s6z+Wik7uQQc8Vv6rcW0UG9sb+gANY8cE8wDs5w/BHoPSvjqvvaHdB21JvD9sVUO4+YjGK7C2GAKxrGIR4UDpW5AMKKqirCm7ssClptOrsRAxqQdaeRTcVm1ZjIp49yk1lyLtbpW1jPFUbqDOSK48VTaXOi4voU0l28ipkuDnJNVGBU03eRXEqjLsbMVwG4Jqfhqwo5iD1q5DdkEBjXXRxdtJEyh2NBzgUqnIFNV1dc0oGOnSu5STd0ZjjXKeI/Aul6+5nYNb3ByWkiHL8dxXVilNXZNDjOUHeLPnddLbE0bEqYmK8jGcHHSqAtSxYYPB5Ne+6p4dstTikWRArvn5wOc+tcTP4CubPTDNGTNOGO+JR2zwV9a4pUJJux68cbTlFX3PK7uy64Gc1j5ms5CyHBB5HrXrEvgLVZrRbhLcFmbAiJww9zXCaxpTwSkMhVhlWB6gihRa0kHtIyd4Mr296s8e8ZDDqK0NO1GSxvIponKtG4dSD0IrnrVzb3PPQ8EVrTpHJBmNcSEjGKhws7o7IVOeGp0/jrxZB4htLSGCJ1dDufPriuGYtGcHrXS6Jov2u5VJQSccmqHifSjpupmFclSMrRe71OJxjDYf4VkZdbjwcZBFa/iUf6Wg9s1neELG4l1USeRKURTzsOM1q+IEla7VmhcKB12muCrB+1OyjNciMi3dopmdTjA604uCFY53Z60qYW2c7clu9RykCCP65zUvsbaDHyLlCepNLcy77pm46YNLICy+YCDtxioEBlkc+1HKSXGupPIEY4Ur0xWdOMqckk+1WHcnCMclRiq8jAtjPNNbiaEXkZUc+4q6MeUpAwSOarQsoyD371aUAxbh056UyWh0bn5GwPxrpPDmqywXahH+dn/1fVZAeNpH0rk4iGAIx8uQR61OJJbPyriN2XndFIh5BH9aViJR5lY7rQra2ca/ojq8KyzuOcFMdVx6GqWs6Y8XhyJiRL9kkKuF7DHX+Vc5puuXFvfyzvN80reY7MOrV1fhvVTqPh2/imO+WJ9zL3cHv+VS073M3Fx1OAglaG9LKyldm5Tnr7Vm+c8aSQnhZCHxW/Jb2j3N0rRhFCOUK8fSsS8t2TyTvVgUwOORit42NNUQq3G2kkwbYDoQaiDFM56jinn7gPc9BTsUncqyLkZxXZ/DqW1t9ZcXMmxpovKiyOMn1NcksLTTrEnJJrvNG8NtHHE3IdmyMjk1NSo1ojSNNS3PRluTDGMk9Oc+tcn8Vr7b4XgiJwZ8AAH3/APrVs6kXgWKFMeZwD7+ua5n4uQN/wjGkXCjCpL5Z/FTirpXa5X5GDilJS73/ACPGypU0qmlDbic00nDH0rvRaLKPirCOfWqKt2qZWx3/ADp8rex106ljt/BmrSx6klkzExTZwuejAZBr0ZoBeWlzZOAY7lNhyOh7H8Dg/hXBeCvDt1HOuq3cZiRQfIRuGYn+LHYYr0Dzks7aa+lOIraMyt+AyB+JxXq0IONL3x1Z8yv/AFc8qjnntJyFcrIhKkg9xwf1rcsfE0sICzwpKvvkH865oytLIzv99yWb6k5NSqc4rzo4iUPhZ7bhGcfeR3sHiPT5h96WJj2Ybh+daMWrwnPl3qH0+bGa82Umn72Aqv7QltJJnDPCQ6HqD3kjWxkN2rKvUb84qs2oKpVTcr8wyMtXncN/c2zloZWQspU+hB6jFEupXM8QikcEDjO3BOPetYY2DXwmCwyTO5fWNPjYF7kvzyI+TVabxDbCL/RojvJxl+cCuLjySOavRjit44iUvI09jFF651Ce6YtI7N9TVYkk80L70hIHQ1V77lrTRC4701jimliDSE5oEITTc5FKSBSYZvYUANYeppnJHAqQrg4IzQR+VUkIiKevNGMVJUbGqSENNJ+NITTS1UJk8Ay9cz411QbY7KNuB8z49e1bd1fJZWrSEjOK821G6e6unkY5LHNKpU5Y2R52ZYhQpci3ZTJ5pc02gVhGR8wyTPPtS5poorojMlo+qXkbUb/JJKKcV0EYVEA9Kz7S3S1jUY59asGUltqk8184nbQ2Zr2mHfCgk1rxsqgBiAfSo9OtBbWy7h87DJNTSwLJ1HNdXJOEbrci6ZIGB708Gs+W3kh+aFzj+6adbXe47JOGFZwxVp8lRWY3HS6LxpKUHNJ3rreqIAdaR13AilHWlxQ4qSswMi6hIPSqLda3riLepNY1xHtY8V49al7OVjeMrogUkHFPV+ajo71nylF2G6aM4zxWjFcBwOawd3oasW0xDYqqdaUHYTimjeUg80pNQQyZHNTZBr16dRSiYNWExj73NBwvrilzk1WuSxGFJxVOVlcaV3Ye0o3YGK53xHo+h6hp15NdxwLIqEmbO0hgOMn1pNS1CbTssI9/fmua1CKHXIJ5CrRyPk5B744z61zSxC2sd9LBzkudPQ8cu4dshIrS05JHCnaSx+6BUdxFuuPLP97mt/ToRBB5owGIwp9KzlK6OyKcLnU+H7X7PGrsP3hGSpre+ypPMZRDFnvIwBP0Fc5p8czTqUZido+X16V04kjhICndkcD0NZOxjJtu5bgEZIjVCqr1x/OrJijLAFEZfTFQw7UiVUIORz6mrCqVUO2BjtRa+pm2YmqeDtN1NDIiG2kPOY8AE+4rzXXdKuNIma3nXgco4HDj2r2xcueenpWD4v0qPVNCuIwv72NTJGf9oc1nKn1NKVaSdmeNLMUDAYwRg5qWxGyIuRyxwOaohjtb1AqzbsTbL2rNqx3J3Fmb5yapSTRpIx9ulPuLgLnHWs4u0su1ec+lOEeoSlYuQzea+AcYrVgYtCR0I4qjDbGOI5OGx6VMlxsk2g8HqKUl2FHzHwlUcds9c1KVBjKDHU/jVe5QGMspHrUEM+9QRyynJBqbXKdgniljQtjoe3areiau1heFgSFcbZF9RVyyK3i7H2lcZx3PasjU7GS0fzFVlBzjIo0ejIZpTsHhJBLZyrY9j/hWZdFgydjFIV/DrT9PuAyESk7T129sio3+ZJFz1GfxFF7Mq1ysAZJZM9SSTUDNg49BxU8BZmkOcEjNRWFm99fR26EBpG5J4ArS6SuwRueFNON1eiQjOT3r1az+z25818AIp5I9K5vRdKXSIRNJgMPTvUOsa3EQY4m2u3BwODWOvNdGzXMuVbGut0L/AFUMeQvAPfrxWJ8YtUkhtLTSSodJArq3oV5/rUVndGG388sQVXjvn0rktduNZ8b6nEYLSS4+zJ5ImAwmM92PFdeFjKo+VIyrUrSTRyCcdaclvJNOI4I3lkY8IilifwFegaV8NlUrJq12XP8Azwt+B+LH+grt9O0qz06IxWFpHCoHOwcn6nqa9qng3vIqNN21PNdI+HupXe2S+dbKI/wn5pMfToPxrutI8KaTpGGgtvOn/wCe03zN+HYfhWjc3ttbcM4kf+6nP69KzZtYnfKxEQqe68sfxrqSpU9johRb2N52jhG6aQIR/D1Y/hWF431IHQPJhGyKSVVA7t3OfyqO1LSEbixJOSxPWsfxpP8A8elsD03Of5Cs69S9Jtm1KivaJvocwlTr2qCOrC14jZ7DJVpxHGKRaUnisHuc0iM9aRc7qCeacoropIhk8S8iri4AqtEOlWRgDn0r06S0JYF8cCmHmlyT0GaXZ6mtkKwhIxx1pApPU4p4GBxQKqwrAEA7UEUuaaTQSxD9aYTzTiajY1QriMaiJ4pWOKiZsCmiWwJAqvPcrEhOajubkIp5rnNR1AvlQfyqJ1LHPVqqKIdb1NpyVDcCucY5NWLmTc1Va5m3J3PmsXVc5hS9qSjtTRyig4pwptGa0UhH2PcqAoI7UukQC41FFPKr8x/CppYTznOKseHkxLcSEc4CivGpRvUVy29Do80ZqPNANdzZmSHB4NVri1VwWUYYVZBpaipTjUjZoadjOtLlhL5Uh9hmtKqM1oxl8yPrVlZMABgQe9YYdyppwn8ipWeqJKcKaMGnV2ogQjNZ97BxuA4rRprqHUg9DWdakqkbDTsc04wSDUZNXb2AxufTtVE8V40k4uzN1qBanxnDAioSadGckCs27so6G2IeMHvVkKMVRgJiRfSrqOHr18O1az3OeQoUA8GophjHyM30qQjaSaUHIro0egk7HO3Ect07Kse8YJIPpWHeQmwh3iAiFvbGK7sQIJN44OKjmSMxMkqKyEdGGQawlh09bnZSxsoO1tDxPVtKtFkFwlrJAZQWVjwG+lGlaedRQwRFS6DBBPT3r0XxFaRTaYII1TZH91cdPpXmctje2Eq39uXiKtgMeP8AIrlnHldj1IVViKeiszt9ESOOJlf53hbBJ6girEaiW4VZF8vzGJyeM8154PE2o2M7TSRou9vmIHysavr4uubiQSS+XkdG5zg/Ss7Pch4eadj0byxA20cc8E9hU2AEz5gJHf1rzYeKrlXIEccn91mY5FaGn+KD5ii4igwfvBWOc9sVdzGVGS3O9ttUjtmzMuQflG0Z57VmeLNctdO0qa5d03upVEU5ySOK5q/8QLFC87RzSRKP4UOBXn+u65ca1cB5PliT7kYPApubceXoRGh7/MZfmFt4I5PerMRzb4B5HaqTkBg3NWYZANwH4VjI7olSaFnJyTUtvEkSg4yfXrW5odpb3VzJLcAMkS5CHoSema6SGyjmaaOSBYZUQEbeBz2Ip62IbVzhLm6CIc8fUVl2twZbwknjFepS6Lp99YhJrdCSMkdx7g/0ritW8KS6PIbi3LSWx68cp/iKI8tiW5XKjSFwRnp0rKM7Q34OflY4NXVZvUcnvWdfL+9z3pwRUn1N+0naJlIJypyMV0Erx3iG3lXJAzk9uOlcpYyl4Qzda04J2ij3o/zFSCD+VYTjqbLVFEw+RMwQnHoKaTicN/AeKu/M+XZME8HPrVS5QqRjvyCKS3HaxAB5JlPGdvFaPhyz8y+imXOR83X0qisFxfKqW0TzyE42IMn8a73QdEksEDzqsbj7qqQ2PrXZTwtSr8KLiupo65cRQxcXAXAP0LdgK5GDRdU1SXzWjWGL7oklG3I9cdTXa/ZrcSK4hVpF6MRk/Wlnnjtv9dIEPXb1J/CvTo5bCKvUZcab2KVto1tDbLDMftAHUOML+VaJZY4xkqkYHGflA+lZM2sNnFsm3B++3J/Ks+SWWd90jsx9Sa61KnSVqaOlUW9WbFxqsEfEQMrep4Ws2bUbmcFTIQp/hXgVXAJp6xkjkVEqre5sqcYkYBPWnBcGpNhPb9aURnPasvaQHdFm1+UA1x3ii483W9ufuRgfnzXZxoVjJHpXn2qv5+s3Tjkb8A/TipxNVezsghJKVxsY4qwgqvEMVZWvLb0O1Suh4pWI9aB0pGPFQZsb+NOTGaaOaniFdVKJLJ4lPpU+315psYwKfXpwWhDEpKdimk4NaWAM0lITQTz70yGKTzTSaQnrSGmQ2IfrTGPelZsZqtJIAKZLYSSAd6o3N0EXrTbm5CAnNYN7e5JAPNZzmc1WqkhL6+LZANY00hPWnSSEk5qpK/51yyk2zycRWIZGLGmUuCaMAGrSPIbu7iUUUlBItFJRTEfb8oXac9Km06EQwlh/y0O7p2qtb7rqYhceXHy2e/tWpnJrigvtDbJRS0wU4VoIlFKKRRxRk5xVAOoKgjkUUoqt1qA1OMin03vTqcVZWAKKKKoCtdxCSI+orBmTaxFdKwyMViXce2UivLx0Le8jWm+hmsMUsBAkX0qRkpET5xXlpvmNjdQI0YwO1LFlJCfWqcc5iwD0qdrlCBgjNevCaevUwaZbeQFcCmCTFVJJ1C53CnREOobPXpWvO2xWLyvkdOKfwR7VFGNo5NSV0RvbUkilijbGYwfwrM1DSYLmFlKqueSGUEH61ql+cDmgY/iFS0paFwm4O6PJfEegJa6BdRQwmSXZkbVyTzXmyRFBtVyCOvPSvozW7IzQq0O1HU85HBFcLq+iW97ZTS3MEYuI0LLKoweOx9a4pR5HY92jXVaKk9zidG8OPqoLNdMEHUbsV2Gl+GNM06eOV7cSMOpc8j3qfQIVGmBYrZAMcg4+atWIRq+JIIwCOQr1i5tvQzqLVmxEkSrt2r5bLgjHBrznx54Tt7S1OradGIlB/fxKPlwf4gO1dlc3nlrtDZUDPJ6DsKwvFd+E8J3KvICHG1QTzzVW0MYpp3PI2cEEdDUkRJxg896pSNg9aktJC0uPaq5DojK51vh0EWmqHPCRKQOpzmt+yldYtO1RhxI7Q3Az94Z4auW8PurahPaScfaYWQN6NjI/lWpobTS6VqNkWDxxgyxDrhx1H4jNZsfLY7HT0RL+5s3B3RkyxnPBH19KkW3jkZ7Z1OyVS0e4dD3WqdldK1/otzwY7qA27nPU44/rWw65t94/10E3lsR3Hb9KlIhuzPM/E3hp9NY3dun7gn94g/5Zn1HtXI3ib047V7vqVjFe23zL99SkiHuR/n9a8e13TDpmpNbJl43AaI45IPb69quCd7DTujNsD+4xVzzGUbQ3BHNaWleEdSnVXlRbWFu8p+b/AL56/niussfDem2RVmj+0Sj+ObkD6L0rrhgKlR3eiOqnGTRzdnZXmo2yrbQEjAzI/wAqn8a2rbwrb7VN/KZ2Bzsj+Vfz6n9K6RE35y6IFGfnYLx+NZlxqkaErAu8/wB9uB+Ar0KeDoUdXqzeFLmdi5bW0NrD5VtAkUSjkIuAPrUM2o28PRjI3ovT86yJruaf/WSEj06D8qgJ9a2lXtpE6o0F1Lk+qXEoKqfLX0Tg/nVLJJySST3pFO5wqgu56KBkn8K6LTvCN9dBZLlxbRnnaRl/y7fia87E46lRXNUlYc6tKivedjAAz7Vq6foN9fEFItkf9+XKj8O5rtrDQ7HTkHlxBpB1kcbmP+H4VoPIkQJZgPrXzGL4lS0oR+b/AMjzauZSelNHNweDoEAM9w7nuEUKPzrSh0HTLc8W4cg9ZDuNJc6zFECF+Y1jXPiKUZ2kL9K8WpmeNxGl3b7jn/2iruzpkt7aL/VwRJ/uoB/SlZYieY4z9VFcPJ4guCf9Y3501dcuD/y1b8655RxUteYpYOe7Z2rWto4w1tCR/wBcx/hWJdeBfDl1k/2ckDH+K3Zoz+nH6Vnxa7Pn7+frWhBrzH74BrP2mNp/DN/eDw9WOzMG9+GKjLadqBHpHcJn/wAeX/CuV1Pw7qekHN5ausXaZPmjP/Ah0/HFet2+qwy4BbH1q8rpIpBwVYYIPIIrroZ5iKTtWV19xUMZXpaS1R4IQaYxr1HxB4Etr3/SNK8q2m53QniN/p/dP6V53qWlXulXHkXtu8MmMjPIYeqkcEV9Ng8fQxK9yWvbqelQx0KumzKQ61ZhGSKrqOatQV7dFHXcubMLn86Q0hY4xTetegiUOJ4phNJnNIfWrFcM0hIJoLDpimZpolscTzTS2P8A9VNLVBJMAOtMhySFklxms25ugAcU27vFRTk4rDnuJbhisKs5z2FQ7s4a+JjBbhe3nXn8qyJZtzE5rp9E8F6lrt0qkiNGONxrp28CWemEo6pJKODuPSuKtUUXY82VV1Njyt0k2hirBT0OOtQvG4GSjfUivcNO05ZIvLSxjMUZA3sBz9KfPptpIl3DPHEsgBCrs4FYqtboc86XN1PFrHRr/U3K2lrJJgZJA4H41Yl8K6zEuX0+b6AZr1Gz1630q+trS1sw3lt+/UjBINb3iGae11iwu7DBtJ0O4bciNvepeIlfYX1eJ4BLp93AxWW2mQjrlDVcxsvVSPqK+h4tVkbU5BcaWJYlQAyrHkMe9DxaDq8E1u+nQ+YrHYrIAX9q0jXb6GU8Ol1Pnag163rPw006Rmn0mSXEqlxH/wA8z3FcJqXhHUbIsY4zOg6lByPwrdSRg4NI+u9Khe205N+fMl+ds+9aSdKqrIu5UzjaAAKtqPlzmue3YzH7sUoeq7sQeKqyXe1wi8saly5R2NZGz0qSoLZSsWW6mkaVmkCKce4rW9lqIs1H5mZvLA6DJPpVK7luQm2Hr60tgkqDLryepJ5NYPEfvFCzL5dLmgacKaGpQc11pp7EC0UZoqgErP1KHdFvX7wrQqOdA8TA+lYYinz02hxdmc4kgYYPWlOByKiuIjHOcUAnbzXz9mdJPNKGjHrVCSd84BqZ2yDUDxlD8ykEjIz3FapyewKyLuno00yqxyO5rWWRIbjyvvHHYZx9azNPJVSGWQD/AGV5/CtIW8Ubl4hy/LvISWx+NenRi+Qxk1csRzCXoCMVMG4PFUwyRh33ZB6+xpYblWBHPrXRF9yGWAwB6U/buHXGPSotwcAHjvTvMCjbg5q0BS1NZBauFGQeprj7maO7heBm8p3G08V3jFW+VsEN2PSsi98O2d1llXypOoKtwT9K5qtOUneJ34TEwp6T+85MxQaRbLHGwJx96Q45+lZEmv2X2rZJKiS4x8gyPzq38QHfSdLt94ZHMww56Hg5/SvLb/UkmUFeWJ/h6Guf2TuejzQceZvc7rUg0sAuBfOVHHyuNoGeue9ctqsd/rZWJZwttF7dR61mWkt1cfufMbYxACZ4Jr0GytLf+zFeCEbNuVGcsT70+XlMpe9sed3GnxxQnYGbHBLVn2yvHeAMOCpxivS7rR1urX/V/O3UYxXnuqRmw1AccxtyPampX0BRs7mjp90bPUYLjGQrAEeorX0u/Gna5cYP7hmbr0APr+dc7kfw8q3KmtCysr7UZy1vA0nTLnhMe7HikqUpuyR2ctzvYtPnk082MTbpLS73wMTyMjK/h2rRsb7zdJdnVhI0+1lK8jHHI9Kz9Jt5dOjcPPuZnVlC/wAGO2e4q2inafKQBckkjpk9a7KOXSes3YhUG9zUvL5Wdwj7kLBs4wM4wayTHH53mLEvmYIDAcgHnGagl1CK3cEP5rjqq9B+NUbjV7mXKxhIVPaMc/nXdTp0qO2500sNy7I1JpY7YZmlCnso5Y/hWZNqz9IF8vH8XVjWexLHJ5PfNMLAd6J129jsjSS3HvI0jl3JZj3bmmluOtRqzzSiGFGklPREG5vyrptI8HS3KiXUneFT0gQjd+J7fQV5mKzCjh1ecv8AMVWvToq7ZzsSSXMwhgjeWVuiIMmuk07wXcTMsmoTeUvXyouWP1boPwzXYWWmWmnQ+VaQJEnfb1b6nqasyTRwRlpG24OK+Zxee1Kl40dF3PJrZjUnpT0/MqWOk2WnIFtraOLH8QHzH6t1P51akkjhXLsF9qzptTZjiEbR6nrVBmdzl2yfevnKtedR3k7s5VTlJ3ky/PqfJES/iazLm4d8l2JNK7BRWXeXICnmop0+ZnTTglsVby5xnmsSa5LMeaLy63Meaz9+417FGjyo9CnGxZEhJ61KjVWTtVmMZrSSsbFlGPHNWo3I71VRasqK5pWEy3HMynrWnaajJERhsj0NY6g1MpINc04RlozKcFLc7G0vkuAMHDelP1DTLLWbM217CJIycg5wyH1U9jXLQztGwIJBHcV0Vhfi4Xax/eD9a4nGeHmqlJ7Hm16Dh70TyzxD4cudAvRFKwkgl3GGUfxgeo7EZGRWfGMV7be2dpq1lJaXsYkhf3wVPZgexHr/AErynW9ButCu/Inw6MN0cqjCyD+hHcdq++yTM4YuFnpJbr9TrweM5v3dTf8AMzM0hbmkZiKbmvpkelcdSMfypVYqeBmonfbyatEOQpOKjaTHeq8t0q555qFVubr/AFaFV/vNxWkYt6I5a+Lp0lebHz3YQdf1rPaae6bZChPv0Aq35VhA5E0huJv7q9BU01xFBZnCbZWHygcAUqrhSXvvXsePPH1K91SWndkNr4ca4cPcSFlB+YL2q7M9hojMkaqTjAJXNTaRqSlAsrqo64x1q9eaUmoQyTLqETNuJwyYxntXl1MU5O2xPsray1ZV8JawsviKASTKkBOCM4AzW5rEtvBf3W7e1t5/zTqc7ay/Cei6ezyNIiSXEbfMo7D1rR8XaYl9okq6TCsLhV3BHyJMHnI9a54JOTuFRtWsVLi8maSdLdySCJQFPUAcYq1pWrX6NbyanZlUuAXWV17Drn0rldMVhYxWn2rbcnAZscpz0rtPtX9oaaLWRoZxtMZJbbyOM/WlPswS0LevR6Ve6fKZBF9veMeTKgAYA9Dn6VyE2i+I428mG5MtuemDyM1ptpv9nymJ4mVyA3mMc/TH4VHJrN5aXEb3LHyycRuDwSBxT5bolO3U1PC0q3FuYrnUArqcZVsMpHXNa09raXV3JPFOrRoACVHKnHJrPsPDGl6uj6ltMNw4LSBHIAJGc/jzVS28MX1j4ge5N8q6dIu1IweXIHf270RpvdEOa6ltbW9tJVt0uXeMPuQheSD1FQx3BhaeSW2UKCVEoHLEnGK6LULl7gW1tZxr5tttlUb8E55xn6CuSbXftFxeRRW+6C6lDbW6JjkkfjXU6dt2YxnfoexXUQby5IdysOGBX0p8eo2zO8RnQSIPmTPI9zU1jqlnNb7PtELTopLR5Abj2rzHVLP7JdF3ciZmLpIHJIOcjnvXLiKyotNa3MKcOfRnpNzckRHy8E+5xUWn2ZluPMc5Arh7zxPcTyQokbLGEUTSjl2P8RUdK9G0/wAtY1aIkxsgKk9xjirpyjVd10FKDgtSzNIIoz2xUNll1eVupOBS3GJTsPQ9akiCooVeB6Vf2iOhNimO21eKlFRTL8hpVF7t0NFCW9eOTGau28vyfMCCTmsKV1N6qucKWAJFdCNmMDFcODcpSlK5pNWsSg0tMBxjmguFGSa9TmSV2ZDqa3INVpL5EPFTq4fGGB4qVUjLRMdmjMksJLmckYVP7xq1/ZsH2fytvPXeeuaudBULTZbYhy3f2rNUqcFtuO7ZnSaShmTYxVB97PJJq1NAjyiV1XIG0EjpUU1zsfBfOO4PFNjuGnc5OFAzkVEHBaJA77irMP8AUltwxwTT2TcgOMHv24qGZGm+eGLOecnjHvUkTFEUGYMTyVxx/jWy8xEpjQFS2OCCMcAYpRtPzAgHoT7VGwLjAO3dySF6VXaeO2Q75AwBHUcn8KbaQJXLS7S+Tlh24wKFf94F/hB5qmJzK+0KVULwSP6Ush3nLNwo6Z4NZOXYpI0DJGSRxgDjFDZwDx+dVLdQhBYjJ7AcCrJ+bacHj9ardahYkZdy/MoI7AiuG134b6FqjSta2T2d5MzP58L4RWOTl0JwRk9AM/Su2aQK2XIUdjmmiaAsFVskntVc6WjY1zLVHhd/4NvvD9/KJUaWzjdYluMBQ5ZcjC5z6/lWjpltFBMm3cFAO4bqTWfFeoeItS+yybUtFnLRwRpkjaSAS3Unv6c1ftIJUk3yhVTH3e/rXO6Uqr9xaHuUadTkXPua7FY7NnOAFXDd85rzi98Lajq+oSzKiwQu+VkmOOPZepr0CS4Zk2gYUdBVWWZYhmZwuex6n8K7KOBjH3pm8MM3uYem+EtP0+NfO3XsinOZRhR9F6fnmt0lY0BJWOMDA7AfSs+bUzjEK4/2m/wqhJLJK252LH1PNdnPCGkUd8KFkakupRR/6pd7ercCqE97cXAIdztJzsHC/lVc5zmmlwM81hOq3ubqnFDyc800sFFSWtne6g+2ztpJRnG5V+UfVun611eneCIlKyahN5zdTFH8qfiep/SvMxWZ0MOvelr26mFbFUqW71OTtba71CbyrK3kmYddo4H1PQV0+m+B2LCTU5g3/TCInH4t/QfnXXw28NrCIoY0iReiIoAH4U2a8ig4LDd6CvlcZn9Wo+Wn7q/E8urj6tTSGiEtLC2soxHb28USgYwigfr1NSzTw2w/eOAfQcmsqbUZpciP5R6iqZBY5Y5NeHKs5u8tWc6otu8mXp9UkfKxLsHr3qkSzHLMST60mQKYX9KybbNoxUdiTIFRvIAOtRtJiqdxPgGqjC7KSFuLnANc7qF7nIBqS/vtoPNc7PcGRjzXq4XDdWdVKA95SzdaFPNVd1So1ei42R1ovRmrUZqjGelNu9Ys9NMazyEySHCxxjc31x2HvWXs5TfLFXYpzjCPNJ2Rtx1aQCufTXrdZrlXeKNbeRkBcsROVOCqsAQrZx14xU1hq085VXNqSrKs7LuCR+YSIh0LHJxyM9Rx3qv7NxEuljzZ5thou17/ACOhQZ7GpQorntATU9XciVvMFxGSWTG6xuUYhUI/hXIXrwc9SCa1xqlomptp880UV4znZASQcHOAM9+CPwrLFZbUoQUr3fVLoTh8yp16jglYtjirEMzIwIJBHQioCKAcGvJaueg1dHUWF+JxtYgSAcj1+lWb+wtdXsGtbpd0bcqw+8jdmHvXKxSFGBU4I6GugsL4TLhuHHUevvXMnUw1RVaLs0ebXocr5onmWt6Nc6NftbXAyPvRyKMLIvqP6jtWfBAZXx0AGSa9k1HT7XWtOa0uQMHlJAMtG3Zh/nkV5Nq2haho93LDeukMf8MoPyyL6r/niv0jI82hmEOR6TW6/UazFU6f7zdFSe5traMgfM5rOMVzdHcQIo/7zVVu9Z0/T8+WPOk/vNXMal4jvL0kGQquMbV4r6dU4Q1k7nl1s1q1NKasdVNqWkaSCZCLm49D0FYT6xf65ciCFhb24PJXjArm4kkurhYxlmY/Wu50/TkisWi27CmCD33VyV8Y4+7T0M6GGdV+0qu5Np2mWREcTM5YyDJH9a0PGVnGYklgKrsIRFH8VRaZZXVw8nl7VUcFyemawNRupzfeU8xaNM7Sa8ptt80mehZKyjoTW9reSbFVCCTtEh7VvwiWxvobcN5rNGWc9celN8O3FlIFt5CTIvIbt75qvfNPYXL3KyK8buF8xT0B6A1noU9zpNLtCJvKthEhkOWO7kj3NdVZ2lnpKyPJIJJWQ/ITxk98VxGmYvgSrFPLUDKtglvU0slxf2lwSlyZwenGRmoTdyJpMZdabFJe3EyoUVmLKAOT9KtWOm3ETwSIHznlQvHXuajk1e8kSRLpfIkxhX2966LwVa3uoM4nbdaddx4PFUm5MiTUVc1QEniR7pFjYDaMHJH4d65vVNLtborA9uJCjkovQAkYOPfpWp4jb7Pdx2tq4Y7/AOE8jmtW9nt4pIm+SR1wcAdCBWy0djB66nK6dcXWk7rAfOPLCgMeVweD+FMvJryVjBuwvL784AbGMc96mns3F208izKjk7dvfPJyaW6eGOzNyke+ILlnc59uB3PNXGWonFHO63ctBdPBbNIgbax3HG4gcEH09vasm21dm2GP5p3/AHanIAOeuRW94i0y7urJbkzA2MUKAOSMAYySO/U1xukWi3GpRxuoWQN0zx+P4VdRO1xQaR9OQ6Z/Z95cXCXQdZUAwYwDuzxiuW1ZI3ZwyZweVz0rqjLb3T7zkZ+XIxx+FYOraTFF5jxyTSSydMnhfXPt9elclenzx0MKbs9TkLiTLcfIAPlAFekeGNZj1PREYcTRYikXGMEDqPYiuCvNIlik3NMqqR91Tu/Wut8EvYRab9hjbbdeYzOHABcnoQR2wO9ZYROMnFmta0o3R0AaQtuboatxvVeSVVYoOSPSliYnmunZnMaCtRKpZCB1qBWIqxG26tNJaMDmL62mSclgefStuygZII/NbLY5FW5reOYYdc1Su444Iy25s44yxrjhhlQlKW9y3PmViSdpQ5KKWjC5GPWsuXUWClXJDDtWl5udPV1IyR1Jxj1rJtpUumYMqtgEgMPvfjUV6LlJWe5UWuokczy7uflbrWpZ4jOC3IHSo4EtkUqqLgkArzVg28DbTtwUB2kHFXSw8oWdxSkmXQw9arMI13ALw3UetOB3JyMD0qCZE2FscZ5weldkk2jNCCJJZiVjAwMZI4x9KheN9xhRcs3TPSpy4AG4YHfB6VExMYkKiUueAyjPFCgtwuIzSWsQySzDvkY5qvCGn3NK4Y+hOAPrVKe+27nwzxg4JlBP60+3uftSxmEDBPzFV/XPrU3TZXK7E97MwjXEuwkbSPQf1rOieFXyQ7sDy55P4elTXLGWc5lO1TuYMmDj61k3F69tg7P3Y6AjI96xqSs7s1hG5ck1NYndYiy7yCfeg6iwh3DJXPyk9OOtc5cXpZixl3H/AGuTVN9SCrwxIHbPSuJ1p30OuOHudb/au0K6ucjvmpItaYYw+COpBriUvp7j5YEZ2zzjoPx6VpRQSkfvZNvqq961p0MTV1Wh0wwie50zXylxJK5OfxJqCTUpBE+xzG7AgMuMrnuO2aylZY1wg/WqU+pwx8Z8wjsvT869GjgY0/eqO50wwcX0HwWtrZp5dpAsfqw5Zvq3U0TTwQZ86TB/uLy3/wBb8ayp9TmlBCtsQ9l/xqmSa7XUSVoo9CNDTUvz6q7KVhRYgerdWP49vwrOZi5ySST3PWkPvTGcAVjKp3OiMIx2H8d6azgcnpVvTNJvdXf9wuyEfemYfKPp6n2Fdnp/hTT7Iq8itczA53yjgfReg/WvJxub0MPo3d9kc1fG06Wm7OMsNJ1HVcNbQERZx5r/ACoPx7/hXV6d4Nsbba94xupRzg/Kg/4D3/GumJJ5Y9BVaa/gi4B3t6CvlsZnlatdQ91fieXUxlarotF5FhUVECAAKvRQMAfQVFNexQjk5PoKypr6WckfdB7Cq+O5rxJTlJ3bMo0f5i3cahLOTs+RfSquM8k5+tIWxUZk96nV6mqilsSlgKY0lRljTC1NRGPLk0xnAqN5cCqc1zgHmtIwbGSzXAANY17ehVPNMu7wKDzXPXt6XJ5r08PhrvU1hES7uzI/WqZbnk1Ez5OaTd/kV60aaSsdMdCYNUyN71UDUybULe1DCSVQ4H3B1PpVezctEjSU4wV5OxLqWrnTvKESLJMzBtrDK7R6jvk1zxkvob1LyVXMkmHEr+h6fh1qtdXUt40k0jDBIGQMH6A1JpVu0s0KxwyNM0wVXEu1Tnop49e9eth6EaUPM+Xx+KdWTd9DobMaZfXRtb65+wGRHNxPEvmkscE5BIx0z3IOR3pnh+za4ZFs5RMPtEcc6mIlyh53bAcsuQcAHPAzjNP1HSgJXukiEYCRxskjrI5bdt6jvgAHjt3zk2r/AMH3VjbT31rdlViuhHFFysqAjKs3bOOOP61dScUry2PGuraM6PQLpri4u5tPgyGvJJxJt2nZt6bj8wbO7IJx0681ma9p9zqXxBiubHLqBBcgtwAhOST9DmpLW8uFujFowhtdRWBZJmwT57Mw3LzxlF56ZOa1dFS4vtQlvktHjtxbKivKdz/M27aoUYwWJ5x0A4ya4sRGqrzjZ3VkdeCqU4TXO7I3Tgnj1o2jFIQQSCMHoQaUV8cz7O91dCD5asRSMrBlOCOhqA80gODUtXE1dHS2N+JRgkBx1Hr7ijX9Fs/E2jPp95lQfmimT70T9mH9R3FYUUhUhlOCOhrasb8NhW4b0rCMquGqKtRdmjzsRhk15Hzj4s8M6h4Z1V7K+TJ+9HKudkqf3lP8x1BrmzG7HpxX1lr+g6d4p0lrG/jyPvRSr9+J+zL/AIdD0NfPXi3wjqPhbUfs14itFJkwTxj5JVHceh9QeR7jmv0HJ89hjYctR2mt1+qPInRcHoYmjwCO4MhONo6109k7XMbCIsTnBbPaszT4I4IirjnaWJq5Zz+TGNuAGbkD0rtqS5pXR6NKPLGxrWc80cstqhDr1zmqNxcwwTwrNbocMCTjNIkrLPI0UuGJx8y8HNWbqNWhCSgBThmZOxFZ3KJL22kkkF7pigpKxDoCAQfYUJZXtzbyWKWjebNjapHf3rPZntbgPAQ2wg5D5xXWeHdcMN1m4dmJHyseSDUOpysHDmRycU9/4X1M6bfbVlBUyEHKkHp9a6hNTixb3Z2mFfk3n7rN/StTxJoOn+Kmhu53e2niXaJU/iXPQiuRS1ufCmvvp1wxltZxm1mb7knv6AjNVeMtjJNrRm1qGo6RrF/arHMqcbpueePSuim15NI0qKKxcgOpXaOuT1P1rnRpFrqNvIu2C2uwRKk23r7HHY8VVtNF8RxxGKEW9wzPkIsg3OM9ASKa0ehEl0ZqadJNFqMc1yX2zE4Zuck9B+ddQ0tnHPFJMCX3MAFGAcdceteb3F/NK0UMMksNwkrCa3lXaYyDgL9a2tOluNM1eO2vbeea1VhKrhtyhs8nPpVxi92RNp7HTXs08cElyYRHaRLvWQE7ixOOn9KxtVv7FfCptbRHd5Jcq7D7w4IIHXrkVWm1e91O7niTzri3kOSCf9WozgAHpyM1uWsdvLpdumoSJHj54ZUXLLyODW90nZGPTU52HXLW10mSzudN8ybyAsfmt8qn+vHasaw0GBb2S7juZwNiFtmPlZh057Yrr9Y06DU4/KjECXCMGgk3fI/P3WH071zuu6e2keU6jZ+7zIYjlSV4HTtj1onJyVhxSTues3FpeW9+rK11JErj5Y1wv0I7mtQIs9tIHf7PJ0+YEBs9Mj6+1RWXiC2njb7rEtyScn2rSAtb1fLikKZO4ruxuP1rGDhb3WZzhNfEjBu9Ha4Vf3hhb+JgDIp/EDp6msufRbfT7mKdnZ5FbloyMj3AzWzNts79Yna5hTPzIRuV8eh7VYu5V8z9zFlTGUUld2G7AjGfpUOEZarcSckZFjrv2bUf9Illlt5PlZ3OSuOjEV2drJHNCksTB43GVZTwRXAXKRs4SNEDqAAUQAnHc88mn6bd3VoVjgnkGWyUQEKW78D6VlTrOD5XqXKlzK6PSAgIqSMYFQwPL9njaQIzFRu2HI/CrArvS1OYrXU8sIygj68bmxmqkkryqxmC7cHG1gaWeS7WRFMKOjHOSc7cUTrvXOVUKM8d6yk27lFS4k+zWAmGCIwSy55OfQ/0qlp95FdQGWKXHlth1PGe9TXC7ECM5w/TA7+g9apCHZGPLVAW5ZcbfzrGTfNctJWNG6Z0XchVh1wBgjPv3qa2nJhUvJnb97A4/OoYYoxgyRoGP8JXJx+dW2k+XzGwWYY6c4q1vcT7DvtUaIcuMHtn/OKoXGrRRMxycDop71z+v30Wk3AWRiqyJvQHnvggVyN34h83O0nHbNYyxEr2SNqdDm1O6uPE8EZdzEBxjO6sO+8WtMDtJHGBXFTakZOrH25rOm1LGQDk+tZupUl1OmOHhHVneabO167T3EpMCNjyt3Dnqc+1Lf6ybXMUAKqOFGMADt/+uuJ0fUZoLmSbedrrtYDv3rTvL9Jk3L16kkYpe98KLVFN3NP/AISi7i3KDkNzjOB9TUMniCSZMSPg9z2FZlnpV5esHC+VCf8AlpIMfkOprobXRbK2AynnSdd8vJH0HQV008HUn8Wx0xoLsZsS3d+d0UZ2Z++/C/8A1604dJiXDXDmZvToo/DvWgRsjLOyRqox85xn2FU5tShT/UKXPq44/Ku6nhqNLXqddOl2LaKsafKEjjHfgKKqzalFHxHmVvU8LWbNcSztukYn29PwqHk1rKt0R1xorqTz3cs5wzcf3R0qsQcUpIFOt4Li8l8q2hklk/uoM4+vp+NYSqpatm94xV3oiMjFMLEsFUFmPQAcmulsPCFxKA99L5QP/LKLlvxPQfhmuostKs9OXFtbpGSMFhyx+pPNeLi86o0bqOrOKrmEI6Q1ON0/wpfXirJcOLWI9mGZCP8Ad7fifwrqrPw3pdmo2WqyP3ef94x/PgfgK1HZY1yxAHvVGbVFUYiXcfU9K+ZxWc1691ey8jzalatWe+hewqjPAA/ACqs+pRR8J87fpWZLcTTnLsfpUWAK8hyb3FGkupPNdzz/AHmIX+6KhwBz3pC+KjaTsKEjVK2xKzAVGZKj3ZpMjk0WGOLZ603cKaWHeo2lAFUoiuPZxioJJgveoJbjA61nz3WO9b06TYIsz3XBway7q8xkZqvc3Y55rHurzrzXpUMNcpD7y8ySM1lPLuPWopZyxPNRb8969WnR5UbRZY3Ubhmq5lVPvMAPc4pEuoWj3rKhXOM5rX2b6Gqmlux19dG2snlXrkAHHTNcy/7wZb7wPJ9a1dRuzOFigfKEfOQcZ9qx3Vo3ZGbDZwefzr0MPBxhqeLj6qnU0exdsY/McLPGzWm9UlYZwmejHHpz+veuwuPC8+n2jatDd28cEbIEaKMoH4BQrgknJDAnsQD344O2nlhuQ8TMHU5Uj/PvXTtr9xL4VaymtzI5vG/eeYB5QwvCqpyvORyNpBOOQa6Gm1ZHj1lK5qXviKDW5pZtTtvIeVoBsZ3xsBILerAAf/X7Vo3BKWFheWskpjnjcNaklihQ53ZOM8EjP4ZNc1PdXN3bx6hulE1o483ecKvmNuQow9wxwemCR3rYTUrjUdOinS4a41Np2jPlAtJIm0enHTIwByOTjjPLWpJptnJy2dxIb9LG6mDyfaJLlXV1iBIVzkA8jkjP6kZrqLrxpN4YgitILOGaWRd7PK5EQwSDs2n5gCGHXAx61w8UE9jrLCR5raZNwZGJR8EdCOvINSaPHFq+uGG7cRrFGxR33bVI6D/DPBNYwpxk13X3D2dz0GDV7SazF1NPDERGjTJuP7t2/gwfmzyDz1BqzcWF7p6RzyTB43Pmzkhiy8HCqOgXA4A54q9Ha3ETJFZxyborZFcwy7RIwz827+L6H6UXaz2UT2hjQJlZSfViOP0NefWw1OEZSpx30PUpY6pNxjOVkhptn+wW15keXPGrgc5XOeDx14IqAqe1bslx53huKCVVF1M+xAmQMjqw9vrxk1jSw28erSrdM0r58uAAERdMbiR6dfc9646uXxlNOm7Ky+TOmGc+zilNXYxW2mp1cnBBwR0NNkgVDN+/ibZtCgNyxPXj29RkUW6q8n7xisYBLEDJwOw968yth5U5KMup69LFU61L2i2JLjxLNYLHHHbJLISNxkk2qB+ArYubbTPFmhtb3SLNbSf3W5jcd1bswPf/APVXGSW73lyYUklL+aclwDt9B6c/Xt0pcavpFm1navJZlG88AIGaXJAyx2nGTxgkdK6IZdKUFOhpNO99TwZYq9eSexwfizwze+Gb77LMu+CUHyZ1GFkUfyYcZH9KwImKNyAQOn1r6QuLCz8R6IltqMUcySxqxKH7r4+8h7EHPNeJ+KvCV14cv/Jky9u/MNwFwJB6ezDuPxHFevluaqv+7qaTW/n5o9CE76M5zzZUQoMYPJzVuITvGHJYxgbevANVmy8QUgjAIzTrN0BR2ZiFblfWvY5ramgjGMo77WMg4O04we1bmhSwi8tpJfnQMN6ise6mjNw8xjZNx4IGKS1u1+0jGVUn5sdqmXvbC5raHrniGIHw4t/bkeWzBXjUcKx4Bz2rzGW6bWreyiluZFa3ZygBHU8Zz9K6W+8QpZ6RZKDJLHdyOWjUkIQoGP1/rXMWNvCNrgRs4y2Dkbyea2ekV3MI7nRra3mm6X5rzGeBWQ5PDY6Yx9as6d4hlivJjJEEhxtjYcsGHfH41nXniWKTS7K1NuBGGLzc/fYcKPoOtZDak90yJGipubYAeAKE0JnV3em2XiPUPNaUPc5BDL8rNjopPTODVbXrXUbeGa8tZJobERLbyRMPmVgclR7YAJNN8O61a2Ooxy2pzMPlCSDcGbHIx+FX9Usp7/WhcfLGlx87QO3CLtC4HPoO9b0pJ77mNS6ZnaNqr3cEcbDa6/OhB+Y4/nxXWC6ktdNaYW8AI+YM+CCuOgrhr3TRoXim30xbskCMMz/88zk4Ge/GOa6vZKpEUkcc0aAhTsyUI9vYnPtT5eWVmJy5opjFlNxA26CQXDSblaQbU2np9OaULNbzmEzIspxncM7gecflWLeS6oL+QT5njXguRjgDIrSsruefTYriMRyQxJtYvyQQeR/Kpm2ioJGPZazLBKrI5Tnkg9a6iy8VXEMvmebnnJB5H5dq8tS628fritC21AkbC5z2ya86cHume0ownpJHri+OJmU/vFHoCoOPx61o2/jC2kgUSDY44DI3H6148t4w/iqVb9v7xohVqx6mU8DSeyPRdSvrQs8tuGTceSBxnqccf1qtpHiT+y74TKxZGG115GRXEJqLjjecH3pxvOetHNLm5luR9TjazPa7HxJDcPuWQbW5FbsGq2zgBpUBPbPNfP8Ab6kInBDdOma6PTvFtranFxaK6nrtbmtIYipFnJVwP8p61qUsP2fd54jYDIb1qot1DbKo+1CQ/eO45yPSvLrzxUk8svkI0ULnhCckfU1S/tps7kZl9wap4tuV+UzWBlbVnqGr6pZwQh45Nxxwq9VINZj67YNLuHOAMb/lz/WvPJtclk+VpWwOxqq+onk7jUTxE3sjSGCtuenJ4ghTHkW6rION+4n+dWo9RlnBZ3+7yK8tt9UCtktitqDXyqgEqSRj1rH2tVvUcsKlsXvGUn9raGZYH3TWj7gD1weCP8+leXyXEn8W7jua6K9v5I52cO21iCVzgH61iEi6mZAhZ2OcKMn8q6aacio03FWRWNzJjBY4ohR5nAAJz2A6/Sum0/weZbcS3U7QOWP7kRgnHGMnPHfiuksdLs9OUfZ4gHxgyHlj+P8AhXoU8I3vobQoSluc3p/h+9njQyYtoj/fHz/98/4109ppVjaDMUALDkO53Nz71ZYrEu6V1jHoep/CqsmqFVKQIoB/jdctXUqdOkdkKHYuuViG6VhGO2ep+gqlLqWw/uF5HIdxz+VUGdnYsxJJ6k96b71Mqz6HVGkluPlmkmcvI7Mx6knNRk0jMB+HfNXbPR7/AFAgxQlYz/y0k4X/AOv+FclXE06S5puxpKcILV2RSLDNWLKwvdQJFtAWUHBcnCD8a6mw8K2lvte5/wBJk9GGEH4d/wAa344o4gFRVVR0AGAK8HFZ9TjpSVzhq49LSmjmLDwhGuHvpWlb/nnGdq/n1P6V0lvaQWkIigiSKP8AuoMA/X1pZbmGHhnGfQVnzahI5IiG0etfO4nMa9feRwylUqu8mabzRwjLsFqhPqfaFf8AgRqgdznLsSfekOBXnu73KjSS3HO8kpy7E/jTeBTS+KjL5pJGhIz1GZKYWNNzTsA4tmkzTScdaQsMAgj6elVYVx28Dtz9aYzgVC8u3/CoZp1H3envVxhcTJpZQOQMcfnVOa4xnmoZ7s7ACRx0rKuLrkjNdVKi2K5anu+pzWZPddeazNV1mOwRSwLlyQADjGKyF12OWPbIHEx5G1Rtx9c/0r2MPgZSjzdDGeJhCXK9zVnuWc7VyWPQDvWRdSyK210dT6MpFZ9zqt1FcQzWzNFNG++NxyQR+nQ1Nf6vrHiFxJqOozTzNhV3/wAPbgD+levQwcVG8tznqY1qdorQz7yW54lUsIc4BB/nU9rO3k7pZVYY4Peqkkb2oeCWUSrEx+TcQvHpUkzRwF0Cky4wVWQMmf6/411unG1rGSxU+a9yldXEk8hd34H3QOgqEXGyPAzTnZgpAxg+oqMttQHy1z64q4xVrEyqSk22KJG2li3sBSq45Zxk9hjNQlmOW9aWIK33jyegq7WM9x+8IQ+BknoBgVoQ3V7ND58iST2ltsVgR8ignCg4984rKYKvDHHGRUsEk7RNbRlmVyDsA6kciqSM5q6Op03WYIfMuprCC5+VAI3X5AVACkg5DYA6cA57VZ+3vqWoRXKW9sronyx2sBCEAnPAPydc9OCB2rGlLaMbaSC7E0c8AmjPlY27sjkHIPQj9DzUWnap5LRxTLL5HmFm8vBPTHAOBnNZTg7NM5XTutD1LUL7V7rRJoktEhieKRJZMpJIYz90bzzwOp4PoK57TIbuDSpYoLee4CTGR5IcnIwOB7jDYPPXpWQl+11ZG3SWRzLHyqzlUD84Zt3O4EjgfLx71u+EfFZ0iNrK/ttrqxYyRSbG6cBwAVcDqK5ow5VqzJxa0NKXxZnw48Wgy3tpcHyiX+XeEAIbDj1OMnANbXhHxBNrUr22pli8Cq6SyEfMPusSSctzj8q4zWJNSN5Pc6Tp06W91KZJ/IjyDnB2njjjkcfxd66HQ5YhbSJEtughffHAZdpJYAMAW4ySPT054xXPK0IO+xM3ZHX3cMFrqCsFneQx7vtDO7K4JPAJOPwAGMis7XZLh7cSR3KOVXAi2A4+p/THFV73UtRVLi3v38uSKT93EcFpUIwpBXtjnjg4IzxWJ4ft7q71KSE7xFOjI05YYAx90g84yB05461xxp3nZs56sr6I6WSK1vFhujfy2TTyKBErR/Mx/gCgE8YH9etc5qtleRa1A894GmmIjtxEzbQN20qR/Cx4PXtTNRZrJojIVjZgF3oM7R05I/pU2qzCzH9nTTJsMazCcvgEEHBz9D196dozbnGJtTr1UvZ8zsaehXttFazyW3n3ESTGU3Fx8itPtI7de5x+tXH1CRrhLq7tgsw2xXUjAh1dW3gJgZJIXP4dxWDpyR3tnBa2LEeSplVFUEuV5xyQDnp+NXBf3t1qVrIqMHjleR0ePb87Nk547dPbmmrRjfVFQqT9pd6nWWWsWcCHzpkh6lt/y9OvXvStqvh/xObjRJ3WXKg7XG3cT0KE9x6+9YPiPxCJrryPsMAEDqxeVDJuTdgoVGNucBgM847Hre8OWdgNQuNTjRnuJmJWSSJozjHJCt8wJ5GT/WvIxmXUMNTeJUm5dNlr3PapVp15WSsjzjxR4VufDd+El+e1kJ8m4AwHHofRvb8RXNSbELCIck8V9J3tjZ6xp8tjfQrNbyjDKeo9CD2I7GvGPE3g+fw5fBWXzbWQ/uLgDG4f3T6N7d+o9uzLMzjiYcs/iW67+Z2xlrys521LRsw27iRxnmtue0sdWtBG0IieEDfcJhWY9dv5Vnww7Rg8ZGcn0qW3cw2EoCFjIzZyeRnuK9SNZpmso3NaXQNOvLWHy7qaKaAAJ5pJDoeeff6U+40ERWT20eqIYyy+Yqxc4HIweoP86zYLlii75X3KoCg9sdBSx30Mk0kk4kBkG0sG5GOldEat0YumX73wKsmniS0uCL3O5C0mVf1z6VyMcYJMLxyrdox3g8FPUY+td3baw+mXvmGRb6xEX3Tx8pHQjsQc06eysL7Tby9ndY5RbM6TJw4IHAPsc4Nbw5Z2XUxknHU86ifyrtXhm2lWBG/oa7zwtdT3t/8A6bCZLeSTEjRLyp6nt1wDXDLYy2t80DSRuyP8rLyMetdJY6vHbGSxeZjATmRCxAY89x9TSi1CdwlFyjoPeEeJoJruTampc7ZiTtfkbQx9lHH1NaWl6pfRasLHWbf7JPbQlFVl/wBYR6H+IEd/pWdY31ve+KC0ccUEbbBtjXbHhQAePU45+tejTx6fqsMdxKyXsIQCSKQ/vCeAxU9jkYz6Ct1OM5O7MZJxSsjkb/UIpYTHMsbNKP8AXrJ8yr2X6dK4qTUW064KJkRy8kBuD+HriuzuvBsFnfSNDNILYgmOKXJI/ujPp7kdK5PU9FuzK00sSKudoEXzKpAHGeo61VhJ2MYQzOQFOAKejNHweT6d6tKWIIUU1Y0Q72O5iK53FM9ZJoaLnaBnNSrc/XFSpAjgMyrjrjHSohuZmCgbRUezRqqkkSi5z0NPW5b1qm8a565b0HApNhEasGIJ9aXsh+1NAXJ70C5PY4rNLOozkEUiTFuQM460vZi9qjWW7Ye/pSm8J71nrKrDPOKUMrHCtz6UKmHOmXvtJJ5NN+0E+tVf3h4K4qa3s7i6lEcETSOf4VGTVxotvQnmJluytWbW6mknWOJHlkboijJP4CtbT/BjZ3ahMAO0UJ5/Fv8AD866qx0+2s08m0gWMHqF6t9T1NdlPAX1kXGnJ6swoNAuLzDXzCBDz5aHLfn0H61v2Vjb6fAIbWPYucnuSfUnvVicx2m3zn5PRU5bH9Kpzas5LLbokCEYGBlsf7xrrjGnSVoo2hSv8KL8rR24/fv5bEcRgZf8u341Rl1NtrpboEUn7xGXx9e34VnFiaTPvWc6rZ0xopfFqOyWJJ6+vrS5pYIZ7iQR28Mkrn+FFya3LLwpNLh7yXyh/cTlvz6fzrz8RjqNFXmxVK1OnuzAzlgqglj0AGSa1bLw9fXhDSD7PGe7j5vy/wAcV11jplpYL/o8KqxGC55Y/U1aJVRkkAD1r5zFZ/Jvlor5nDUx0npBGXY+H7KyIcRCWQf8tJRkj6DoK1gAB1qlNqUUbbV+Y+1U5bmWbq2B6DivAr4mrWfNNnK4zm7yNOW9ii4zuPoKoy30snCnavoKrAUuQK51qWqaQuM8nmguABx+tRl6YWqix7PUZcmgmmE+lO2gAWpKTOelML4pJAPPWmFwKjeSq7ze9aRhcVydpOaiMi4YE844qnLdBerYrm9R8VBXeDT0E06PtYv90epHrzXbQwdSq7QRnUqRgryOnklxVGe529/wrP0LVG16ARq+Lo53KybVXHHB5z69OM4rD1+4ufNNgmfNY8mOQDOO2a9COWTjUUGcyxkJJvsal3q0AuYrVnVZn4UZ6+n/ANaqBuftMs1rFN5d0gDIrjG/1APrVS+TTdRs4iIvIv0DCWOMsQuzO7IbpjA6E8e+axtR1JVug6YlKqBvPc45NerHAwptR3ZyPFyqXS0JbSR7e4lL25UsjAGaAMd/XpnIPbI9+Kn1WxuxNJfXVgYIo1jhlZCqt5hX5SyHnJ69OnvzUGk+IBZ3pea1t7tJUEbRTqfl5BGxh05AqXWPsyaxLOkSNb3SB1jlLHYW6jnByGzivRtFRS6nJKT5xbcRrDs3QT6d5gnFvJJtbdjGMnGenJXtgVLZWdotreR3hEZkTdBPkjy5Ac4K9euAR+VZzn+wrpTPHaXSlMxYbzEyCwIbbgNyMHn86isJfPs7w363LymEmKTIzu9Tnk1ck/mG5Dc20zs9xcxMfOJfejAZJPUjHAOeDVSZFUGMIUYN0PUCrmm3USSP/pIWPad8Uo/1wPUKQOv1qrPKkkxQ5VgMDvnjjmm1pdGifQrLGHXeTwDgAcVC6uHC579M1cQRLHw+Qoz75qm8yjpnn160o3uaJ6ETLukCk9T09KFwA2OvTnvSHEbBuTxxnimhtzgDGDWoiWGM3D4JPouBmplha1lWVJTuUhhgkEHsaqpNJE2AasK5fIcnnk45zQyGtSzLLHLHbqkDJI5JlcNkSZbjAxxx+ZrR0HTLueeZYtPjv5VUBYWZupYAY2kfNnHcd6xMsgU78hegzVlNQdY5YRIyiX5ZMHBI7ipcpX0MpJ9Ca0ZpdW+zxwRq07eVHDISQrN8o5PIwx6//XrVsIri2lu4rgSKIywuURw21hkZOM9MdfT8aqxXVuDYrZ3F1C9uTMQEQ7Jc9Vbg7cBSc+/HFJMbrS0ECOfLuYCqyRD5XVuGGSO2cH69abSMmi+LqbVwkWZZGUCOBSc8DovPAGP84qzZ3L2l2kMttOJImZNsjgbJVIOCuM49j65HTFUbVYGs7siCSCaKSOSORVZkCYZWB6gEsUI7ZyKlvfEM13eQTS2kCRwZVEwzFlJztLE7iBzj0zXLOkjOUdDvdQ8RTanbwl42d4osSyAZyc5IB/z+VZHhq8SXW3ZnLxtlWiTP3W43dex2n8609Hs5buyhuEa4axwGWQqNyjHIKj0IPPcc1uRaaLaGZrSCBbnyTI80zHewGPkUDheO5yT7V5aXJOXVnIyjf266hYxKLOVgX2NJGCFDD3Of0qlrOhSPZ2EUc5OwmDy25WIDvuPzHknj8sVu6PamG3u55IhJ5mSxcn5MdCMduf8AOKjvIFvrO0l83y0NytuwL/Mm5c7voMHnvWVKM5O1N79BK8dWVZrKO30lobNI47mCIfvY0CvKg++DzycfMPoR3pj3y3rxSCeVZQR+7Em13/E8de1XNTt0srWFI2ZZlyDNsAdDluGHODkAg9ecelWorSO9eOaezjZyuBlv9XuI3M3AHO0scHqe5zV+yd1CT95fibU6iiaHh0I9hfXSSSfvSzMzLuEeONu3v6n+lDlFMckEnJUMRzlT3Ge496ivruyttPWGadFimzGJYlIdWBwQVx0OcY4PPeo1g+zwxlHWSIr8jqcg/wBR9CBXFmSlKmuaO34H0WW2T0e/4nQWN6JRg8OOoq7dWtrqdlJa3kSywSjDKf0IPYj1rl45CjBlOCK3bC8Eq4J+cdRXy8oypTVSno0d9ejbVHlXi7wzceH7/wCYmSxmbMM4HX/ZPo3t36j25qK4WJmjLZGcgntX0Lc2lpqdlLZ3kKTQSDDI36EehHYjpXi/inwhd+Grxgcz6fM2be4C8567X9GH5HqO4H1OX4+nio8r0kun6o54zadpGQUCnJByTnrQUOXAjDI44OMbT60ISTg9cY5NP86TaVLZHrXoxnY3aTGWMLTO8LTbHALKccE9s1Yi1GW3WDzlDi2m8zyn6E4wQfUYqvC8kEqsNvs3r+FNvrldRuztQhy2ZGH8R6c10QqWRjKJP4h8ORQ3VnrGnjZYahGG+dvuv/EOOg61nJb/AGO4ledhhkJyrZ7cf0rp47yVrSyti6tCGKlGAOMdP61g6tasurCMhBayhvKBYDbg8g/0rqUub3jFx5dDOtrhUaOSMsJFHzK3Q+taWl6o0V0ixpl2UnBPSq9xoV/ABLbW4lgUbndXyuD0z6dazGgvNOuY5JYJYnGHBK9B2p2ZPMd3L4nu7XUU2TGaA7FdZQDngjj8DirmqXenJcRLHE0dzC6hjG4aORTyc/SvP2nuzGpbT28tjwxU4Zuo5rS03RNYvbtPs/lckkh5cgLjJ4/OtItpkO1ilHIi8rjntR5W0FuGBPA9KetqmAcY+pqwIQoAPX2rNs9Ur5GMEt+FPWNzllG0YqztAAHvTx8x3A/L6U7lWM0xuCSVHJ4xT2hLqCRyPWrDgs58sD2FN6DLc/0o5ibFM25yV3AAj8KRLcL90E+vNWWBJAFS2dhcXUxS3ieRj12jIH1PQVUU5OyFy3ZVELEYHNWbOznuJvJghaSQ9lGfzNdVY+FlTD3su4/884uB+Lf4V0VtbRQIIYIlReyoOtdtPBt6zNo0G9zm7DwmBh76X/tlEf5t/hXT2lpDCBBaW6xqTgIi9f8AGmzTxwH52y391Tk//WqnJqMpP7tvKHohwfz611L2dLRI6oUNPdRpTPBbrmSVWfvGhyw+p6CqEuoSONqYjT0XqfqapFie9JuA/PisJ1mbRpKO4/OTSZx1xWlZ6DqF6QTH5ER/jl4/Jev8q6Kw8M2dr80w+0yesg4H0Xp/OvFxecUKGl7vsjKpiqdPRas5Wz0671BsW0JZe7nhR+P+FdDZeE4kIa8laVuuxPlX/E/pXSKgQAAYA6AVFLdRRDlhn0FfM4rPa1TSGiOCpi6tTSOgttaQWcXlwRJGmc7VHenvMkYyzAVkXWqueIlx71VWYvzI+T7mvIlOpUd5MyVGT1ZqT6iBxEM+5rMurqaU43Ek0jzIOKi81Fy2eaIw12NoU1HoSRR7E+c/MalDADAqq82Y1kDAgkjAPPFOR8jmipGS3LabJ99JuJBqPNG7ipWhmOLU0mkJppNNAO3cdaaxxUbMKYzgd6pRbEPZuetQvIPWo3lA6Gqc1yFBya3p0m2K5PJOAM5rHvtWSIEDBNUdR1TAKhhXMXl8zFjnP9K9bDYLm3IlJRV2aF7rewFmbk8AVyscrxXaGMkMrdQetJcSvMTk5HYVAWdh1yQMc17uHoKmtDzq8/aaHTWWoppspEUkMc5fJKSHywD1BxyR7Vn3ltrEUUOrTRAW8rMY5lcEMQefescyyRuDwMfjV1dTeSz+yyLFtDb9wjG8nGMbuuPaulRj1OBw5XoRm+mxMuflmJJA9+tRiVJLZ0kU7lU+WV45yOv4Z/OohtLc9KnEduqbWaQsW6gjCj/9dNF2SNDTjp01ky3UIEgORKrEE+x5x+lVLlt8ixXch8xOODk+wzVCRyBgHgelQs7MVHOAMDAxmmo63YlCzua9wkYsHLyzyNbj/R3YjaU3cjaTxySePrSWF9PcvFEfMd/KMAaIAMUOSRz944J689OeKitjHcbbL7PCzuhVXcHqehJzx7UxtkemTRwERXAI80ecNrpnIUKRncCoPB/nXQu6Jv0IY9NluZHWBXYorOwAzhVGWJx6DmogkQztcdeDnqKjgvLiJflxtznJUHsQeo9DUokhkgIWERyL3Xow9/cetS7F3sQujhiBnaOwqAoAxLDPoM8Gnlm3FRgDnaT/AI1C+d5DfShItEoKk/OuTj8qjmQByVPyk8ZpmSoIz7Uv3gSePSqSsGw0E96ekjL3yPQ0wcnHrTyAM5U57Ghk3FSYq4YHkdMU05lk46saNuf4SKeo5zkA/wAqBHR6V4cvr+NZLB4RPHxzIR5hJwFAx945xjvntzWck00sscEjlEh37EOdqE8sAOxJHT1q/oHiDUdPuVhi1FraJ2AkmAJ2A8Engkjnng8evFV7e2aa6vXtlBWAPIUgYuioDgsC3JXnr6HJ4qbO12YO63N+DUxYeH76wnQq1x8giK42bWBLvkcegHqTnpzVtLMavNGFaJVA6FwCfYCrWqXWqZstRuL55mdI9gnG4FBuAAGOMbSpU+uRkHihcqzaxDcLMkr3bmaSJUVTESxwpA+UcYIx2IrCrC8fdZi37p65a3OleHdNiXzoreOZRxLJhWIGAOnHBPNdP9hgniN3ZRR3AmVBCc/IQ38WfT19cV5M+l3ep3+2a485F2s8LMUYE4OFbBU5HP8AXNeq6BJa2miwW0bMqQL5TKz5Iz8xGa8yjRi9JvUwTV7CXyvBpUysQuV2OEXG4Zwc88LnvXI2Njb6hrIi3ToLdn+0B+gA4HHZgSPUEmtfxFr91aSy29vGsqAlJiM/vR029eBjjiqOl3aWWkTX1rBLLJNMouA5A2M2ck+oIGB1x+tVCEIO/YyqNOVi1baRcS3MkFxmSLymMhY8s2MgZ+vP4Vei0pyjwbzE8bK5ff8AdwueODzgj8vxqW31GNoA8R2sf4id7E+nsKhk1dl1IXBxHCrjeplwxUDr6/8A66xU07X3uPkjEJNBsX1CYl0Wbyg6vNKc+YfmOTnBxyCcnqM1VIbK/MdqrtC8YHp7/rV23hjuoMed9o52iR5gFQE7twUdDnPBFVrkxQXDQGVSc4XcNpb04PSuLNnzNTp7bOx9Lk1WMouMt1sMU1NHKUYMpwRyKgIoDYr59q59A0mdJY3wmXB4kHUf1FXbmC21GyktbqJZoJRtdG7/AOB9+1cpHIyMGU4YdCK37C+WddpwsgHK+vuK5ZQnSmqlPRo8+vQtqjynxb4Yn8N3KupaWwlYiGf0P91/Rv5/pXNGYrhlKkDsa+iZ4LfULOW0u4Unt5l2yROMhhXh/jXwTdeF7k3No0k2lytiOU8mMnoj+/oe/wBa+oyzHwxS5JaTX4+hgqjjozBa4MsgXYVAB57ZquJ3tjJggtng+nrVP7VOMndgrVKbUphJ8yAgnkivchSb0QOZtDVf3wGdvZjmpLm7k1GEZmJEb/LkAE4/+tiufMsm/JQ9M9ajOobItnlyKQ2Rg1tGm7WRm5XOyh1W4sYjEJcRyLtZTyuPcVd1PUlvrW3DMjskfzCQcnPv69K4JdZQLiQOXB4bFNk1wuhX5znvVQpzTJconQi6ZpTFNM8sKZEZVsfQ/XpT4r+4idNtxwH55259TxXKnUgAQuaBrG1Auw+/vWnJJu9ieZHaRjcd3GB1yakZTt35GW6gVrx6Wd3yjkdMCpjpYxjJ3fWsz0eYw2TGDjApfLOFRuF9q3Rp7LFkqPUUQ6bLNKHBO3HQD+dOMJSdkik77GOtluwSM/hTU064uJ/LtoGc98DgfU9BXXxaXEgHmnd/sjgVoRRgKEiQAdgoxXoU8C3rNm8aLerOdsfCca7ZL6Xef+ecZwv4nqfwxXRQQRwosUEaog6Ii4p0kkFuD5km9v7iH+Z7VQmv3cbVwi+i/wBa606dJWidNOl2RdllihPzNlv7qn+tUZ72RxtGEXuq9/rVUsxNS21tPeSeXbQvK3faOB9T0FctbFKKvJ2R0csYK8iPk9eKdGhkcRxIzu3RVGSa6Ky8ISPh76cIP+ecXJ/Fq6Sz021sU2W0CxjuRyT9T1r5zG8QUaWlL3n+By1cdCOkNTkrLwveXIDXDfZkPbG5/wAugrpbDQ7KwAMcIMneR+WP49vwxWllUXLEAepqvLqEaZEa7j6npXzWIzfEYm/NKy7I8+pXq1dGWMBRzUEt7FGCAcn2qhLPLMfmbj0qE15bu3dkqn3J572WTgEqp9KqFSxySTTywx2qMyU0jVK2wMF6VVePvuP0qR5MZz+tRiRT3raNzRXKcjuh4qImRyPerkse4cdaj8odc8j07V0RehopqwkUTHg4zVtCQuP61CjMGJJH1qQOp4HJ7ms5u5EpXJc0bgKZupjPUKJkyRnFRtJTGfioHkrSNMRI0nFV5JcDrUckuKpT3IUHmuqnSbYmyWe5Cg89KwdQ1HAIDc0y+v8AGcNXO3V0zkknmvWw2FIlKwXdy7k7QWPoKyfNVpWLg4P6U57lkkJUAkioCXlckKAGOCT2r26VJRR59Wo27F1LiBbWSJGG9uzxBgcc4B6qfes+fdGctGybhkZGM1aubM6ekUjF1ldBJGykFSp/karSOZSp24XsM8fhW3LbQ5l3Q+B7YQOZo1klIwoOePfgjp75rp7TWbCa1SK4tY5fNQRyxsiqsRGBuUgZ6DPGOc8muRaCUvGgRyW4UbetDxz2wjdwQrE7Tnrg4NOztZGVSCkWbsW6ahcR2aP5O8qnmHJA+tV57eWOES4JXJXI/wA+4qxBY3V1NiGFnZuQMgE59BUMv2zTZ5La4inglVt3lyAqyN64NKKK6WRBbwedIqrlyx+6DjNakAtJb54blQkBYYUOVKKOuOM7se3PPFY1yk0F08cq7ZVPzDIPP4cUW05WcPnkZ61olbUUo32HXFrPEzgJIy/NyUPKr1Ppxxn0qzGyXbL9rYtNK6J5r8he2T3NTz6jNefvJd5z8nDNtQcZwCeP5VRkfa2xSCV/i/rQyFd7kl7p72srFP3kY4cpztO4r+W4ECokmRZHRYhtwFO4Ec9/p3qaGYyWrQySqVV/MCu2Mnvj3/8A10yORo5X3KyxscPgBsAnGc9M+h9aoV3azKrRGF3DKysrEbWGCPYioJW3Ln07GtPUDGtw08T+ajHG4ptzgemTj86yZG3E56k01uXF6CKC2aeRkjaOemDSqV2/Mvrgj1pYlUsckD1OOlMvoRrjed2B/SpFkBIHbuaWZCnDqTkZBPcdiKYQuflHQYoFcQryAc8nvSHKuA3TPepJEYYBz09aiYtja1CC9h4Y7vk5J9629BvVtdSSSXAJUxsp6SKRgg++DVGy0yeWB7tBFIIxkxFsPj1APX8K1FsrS+0wy2tvKLmJSZUBBTaASWyTkHAPHfFRPVWMJzjsa3imeyaazjs5FeKODauAQRznkepJJNYcH3wM5Ldc1t6XbwT6STLGrBiUB2DIAAOQxHXp/k1s+DdHku9YigMa5WUsdyAEFRuGc544HFccpdDklV0tYp2+sLZafDbQ/LcpIWaRSclT2PrzXU6TJrNlEbr7aj2l2m5YwCcvgd+u5eOnHauGu3jk1vUDaQJHGblyiKCFUbuw7D2rvND03+1NLhjaZQsPyRocbsscs0ffsMjpx61g4pN23Oap7mhrRx3j6XNczxBsMpLMc7lOfm9eMfzziqsF0/28aekiD5vmZ3ARcDJPGfzrRhtH0uSGJiZ1BBd4HL7GPqD71Q8VF7PTprmCwkBt1DK7xYwrtg59V5z35/GuSceZ2SsyIq71OlbRbSRri9k1NeUBxbvlC4HPPqf6571Ujnsjp6i+EE6q2XUjcGwflG8/Mf8APGK8r0q7nB8i3JEjdQGxu/DpXS6ZenUpksAduASVbksQM/0xW7bv7qs2dLSWrOmg0u3gkifTr6WOWcF3ifG0gHsyjj6Edqv6pYzCaJrtI3TaWjVvmKnPJz0645z3FYC2+oWkxmgwigAqR8pxjj8a1l/tVLSKW4eGWF02oGGWRG64IOOo9ODXn1EnTnGV00t1+p14GpbEQtrcQ9KaRilzR1FfOn24gbFSRysjBlJDA5BFREUgbBoauDVzp7DUFuE2thZh1H973q/LFBeWsttcxJNBKpSSNxkMD1Brjo5GRgykgg8Gug0/UFuFCucSfzrlqQlTkqlPRo8+vh7arY8g8c+AZ/Dkz31oWm0t2wrnloieiv8A0bv356+dTxNgjOBknpX1s6Q3NvJb3EaSwyqUeNxlWB6givFPHHgA6DObyyQyaXK2FYnJhY/wt7eh/A89frcpzdYiPJU0mvxOLVOzPPY4DKFdX3YGPpULoqSlJRwRwcVoS2zWieYjAA8YHrUHlSSLvkjJHUV7il1G0Yt1bHzWVQeeQaqbGBCsD+VdKlowcHHJHTGambQ2uonjUgyEEpgZII7V0QqdDGcUcoF545p6x/3uMjIq/a2czgNFA7c4Y4yDUb2RW7Mch2EEkKfStOdGfKe+WtvG9q3krwTzuHP40NahScKOe5qZbkPxIuc91PT8+tK9tOCPkbaxwCRgH86644KK3Z7iw65tWVxDGvqcdqmjjZiEjTr0VRVeS7tLcZMwmk/uxH/2bpVSbVbmVDGjCKI/wR8fmeprZSp0laJ2wo20ijWlNrajM82+T/nknLD6noKz7jUXlBSMCKI/wr3+p71mGQAZJAFa+m6Bf6iqybPs8DciWUdR7L1P6Vx4nHQpRcqkrI0cYUlzVGZ5c4/rV6x0a+1CISwRr5bNtDu2B7n6dq6iy8MWNsAZY/tMndpRkfgvT+dbaRrGoRFCqBgADAFfJ43ieC92grvuzlq4/pTRz1j4TtoiHu3Nw4/hxtT8uproYoUiQIiqiDoqjAH4U2S4ji4LZI7Cqct5I4wnyivmMRjsRiJXqSv+RwylUqu8mXnkjiHzMB7d6rPqGCwSPIPQselUiCTljmjgCuZaO41TXUGd5Dl2JNJwKQtTC2T71SRY5mxUbP70hY1GTVpDGSSAHrTPNQ9znNDqD1qHfGuQOK2UbmiEnJdcL1qssjRn5qs/L1zx0yaydR1GC3IjLorNxuY4C/WumjSlN8qQOXKtRuo64LW3Yry/b607RdZN5YslzMS0bjYrBR1yDz1PauJvbl3vZBJIsiBiFaMHafcZGa1fDMzLqAt2QFblWhAfgbiPl5zwd2K+goYKMYOLWrRzVaiaUkdqJYpCFBz7U8HHCiqAdLdj9oBSToVIwQRwRj61raZHBdjMk3lnIKj19a8eODlOpyLQ1lK0bkBbio2en3k0KzsFKhc8Mfl/Sop4/Kba8iqcAj39KHg5xdhIjeSq0kuM81of2exO2UhTgHg/4VUu9HuFiDo6lT0LDAP41pHDOOrFuZk91gdaxLy9xnBrXn0TU5eixqM43F+KwNQ0XVLYsZLSV0GfniG9cD3HT8a7qEI3E00jKuLgsSSazJ5cZJrp7fwteXkYYTwxkgHDBj/IVk6n4e1KyYh7Z5I84EkSll/lkfjXqUZ09kzGpGSWxhk7jknvVswhrYOodVAJOF4z2H/16haFowRjB6EHqKu2l/5cZhlhM8bKV2hyhzjAOR6V2Jo86pcicK9qQ4YxjoRyU/D0qlFMI8+taNmNykySNGyDhlHI7H9M1nXECJLIEbzFHRgMZ/CqMk9bFuKNr6NppLzZ5Q+XK8hR6enNUJJGcqrHIQ9KkgwIzknjt61BKh8zKAkU7iS1Ou0LxPNo6M9kUF475Rzxs+UjkntyeBisLWNWuNZvxcXmTckfM5bO4/04x+VUfLkji3kgE9BnpTiFubsS3JChiN/loqjAGOABjOBVxaasTycruhl8XeUNccygBS3qAMD9KpOgVsg8dq0o7ITpJLEiiMN912556AeprOlG1yMYx2PaqsNdixHPsQx5by3AVwpxuXOcfpU/2SF7KS7jY/unCyR9SEbo/wBM/Kfcr61nkoQAB061e0+0+1l1MjJhDgJE0jN7ACl6kzVlckj+wokNzjfE7eXcWxzlePvKT684/ukelVJVjiupbdJneEnCORglM5GR/T1pl3btZTtF5qyYx8yqRk9xhgCCDxyKrM7M27PsKe5KV9S5dRyJ+4yqlGznPFUZ4jDMVLI+D95DkH6GniaQBmJJz3PJqwJY5LbyZfuqrMjIo3Fz0BPpVRVh7FLb8obd+tOTcW+Q4NRA/wAJPGakRtp4/KmWmLMXJCuu3np0/SoyuWABq2E86JzNOVZE+Tdk5I4C/wD1+nFVXyrDBDY7ihEokSNywZuh/Wh1CMGJyale8k/s/wCybh5JcSYwMhgMZB6jg/jx6CqzkNt5pBqdL4Wto9R1GCLZOWjPmN5TLny05bAYfMemAPyNaCXdnres32oyCO3h2BUgURrIxb5VwNu1ju2lsAcZxjpXHMVjKmN2B9j096tKbRJYCjTNgBpdxC/Nn+EjtjHvRojGdPW51eove6K0NtdQ/Z98XzQ/LvAYg7gw++p/hJJxtIOMVt6N4mmk5SdoWz5xaVFI+UFVUN2GMcdKytRl0vxJFpyJaXcV1MNpvS7yl5BwUAIG7JweORuqrcWJ8I6rNY3TR3MckfDrnkE9GXIKsD1Ht1wawq007tHJUimrdQ8Sk6X4quxEm1JNkoAIIG9QxxgkYyTWz4Z1oQypvlAAJ+bGSARzxWeNPsNT1Z7e2k8qJwPJxGTkhepHUbiOnOM1f8ODQpr1I7iMx26pmRyc/wBQevpXFXSeiRhUatZnoN7r1oEtp7d4lhKbXVGLsze+eg/xp9zPpmtWD2z28iJK/lM6zsGTPOMjtj2wea47/iXR31vHDLNcRKpMhkG5RjocYzjFdjZ6PY3MUktpLcxyLFv+Rvl3Bd3OOo/xrzpykpt9zPXmtE4bVfBz28n2nSHupoUkKOhXLqAuSwI6jhu2QBWzNpsemWNhcxWhj+027K8wct+9GTyp6ZUcEdxWnfXs09tYW2nwiTcPtDhWwzAdFX35Nbt7rdifDhjvI5EtQuXSIBXPovPTn+tdUK0ZfHpdfiXH3tGzkLPVZI0KXSuySHCyNy2R2Gfauo029SaGW0it7a3ilQZkmduG6dc/Ln8s153qb2a3X2i0tprdsZCSvuI9CMgcVt+GdVF8Ck8BkkIMbAErlfXiuScZRlprHZmsG6TU47o6R1ZHZHG1gcEHqKbnirKwvJsiuLlPNRQgVVLBQOgyetULm2uzFJJZ3sDNG4yjLhSDxjpnPfrXk/UpTm1DZdz62lm1HlipfE+3cmpDS9qT1rgPWG7sGpYpijAg4IqEjNNBINO1x2TR1Wn6is4COcSD9a03SK4gkgnjSWGRSrxuMqwPYiuIimKMCpwRXT6dffaoyGI8xeo9feuOpCdKSqU9LHnYihbVbHlHjrwQdEuRcW29tLmJ2E5Jib+4T/I9/wAOeWtUyRFIpYAYBA7Gvo+a3gv7SW0ukEkEq7WU/wCeteG+ItFuvDmtS2xjdkD7kdekiHkH/PcGvrsqzD61TtL4lucV7aMxLqKS1khGQQ65GPTOK2dPt0XTZdjMrurEyg8jHQg9hWZbLDf6o+9T5b/u4yT90f15zXQ6jp80Wly2ljKsruqq6qnzbc5IFe9TleRE1ocra7o1jUIQAAAQPvH1rSktbW5h8iWKG4IIO5uGGf7pqnL9utocrF8xT92x6j8KuQXsqaOJRZb5FlHmSKcMvpkUJWeo73RSHi3V5fvX8/pw2Kmg1O6kkR3uJS6/dJkJI+nPFcvExz1rRglK8D8K1lWfVn0lOSZ1MNxwMmtLTrW81a48myi3EY3ueFQepP8Ak1R8M+Hb/XZopNjx2RYhpz0OOoX1P6CvYNP0y30+0S2tYhHEnRR3PqT3PvXhZpnqwy9nT1l+RniMaqa5YbmPpPha1sQJJv8ASbnvI4+Vf91f6nmuiJJUbiTtGASe1DssY6c+lVZWZ+pwPSvi62NxGJk3Uk3c8uUpVJc0mPkuo4jx859uBVSS4ll77R6Ck2gU1iBWaSLjFIbtoOKaWppaqSLFLUwk4opDVJDEPem9KUmmE1SACajY0rMOtQNIOa0igEkfANYd/d+RL96tC4uAqmuR1e5LSk54r0cJR5nqaU3Zl291r7MiNvDnG4qK53X/ABFFqO0QxeUOrIOhPTPXjiqFzc5yM1lzy4/GvpMNRUUZV3fUu2dy8TBlOD3B5BrsfD/iC2tji4sopYyCHTA2uPfIJrgrVnkcIgyx7CteON4k5PJ9K2k3B3RlCCmrM6pbkXtxJIibFLFggJIQE8DNalhcTWsiPGxyDke2PauU0688h2GeCMdetdHpeb51WB1JzjDMBj656Vw8jcrrc7bxUdS9e3MUcXmOV81l9AevXtx/9eqtlDdT3EErTIsJICgNkj0pmuvbfZobZJA83mHe4PyYHocetR6VCbku4JEKrk/LgKR3+laTp9DDnSVzrDdJGqCTy2YL95u31pkt5DOyqJN+DkhTj9K5W5lae7ljs5SYUIXc/eptO3xXQE7b1zyFOCawcZp2k9BwjFq6OjmliRtuHHc7hj8qh86aN2e3mVCQR93PH51R1XVZZvJWZQFjUIBjHA6fzqn/AGihASMNuI5yeMVMqXv+5sVGPu+8bNqViA81vM9eMVJI4WVhE7KpHOT1HpVG0tbiS7SB8qzFRz23dM+lberaRLpsiXkKKYNoyMbgD3+tXDCVHFz6IUqkFJRvqznNX8OWeuW+fljuf4Z0UH8McZ4z3rEtPhdM11++1VI4hgq0duWb8QWAH613OyOeGO4t440yRFKu4Y3dc+xI96iunNu0kcqTRyRvt25JABHv9Ca2Tq043T0OedOFR26nlXiXSrvSbySGRCojAUSKmEkUdGBHHI655znNc3jgq3BPevoGxvgyrHHcmNyMZdMjBOcEV59498Hy2uoy6lpsQltJzvaGIFmiZic4UDlc+nTOK6aOJ5/iOKrhnB2R5+YvL+/0PQiiHls9hSzQyRSNFMjxup2sjqQVPuDyKdChO3dwCa67o5+Vlq6tJWsFufLVoyu7IIJA3FeR25GPy9RWQVz0OPata5CpHtGPm61kyA78iqi09iLPqN8x4pAyyFSDwRTpIzcAkBdwyS3c1XlJDUeYp7keorbUViJ1Cng5q5Z3UsTxtFIYmjO5WThgfUEc1BMq5zHgj2qJFkjbcARRuhNXRq6nFPdxfajKJ9qhpHJwwzxhs8k8dec1ks3yqi49yafdXVzOI0mkZhGmxMnouc4+mSfzqsQ3X9KpIiOm5KiF5AF9ccmkkUqxGNrDhgDTYZTFIJBjKnuKJ5jM2ep6D6U7D6kQwTyfelB7UgJRgWH4UrEFiRTYFmG6aAHgEEYII61VckuWAxk9B0qQ4dQcYI/Wmv8Ad4H1oQDBlmCjvUxVoiocdgwBGMj/AAqFGCuCas3EyyCMDBZUC59QOn4/4U2J3IpnQzSGNdsZYlVB6DsKYN2fWpnETwMT8sgxjA6+tRRRu4wAT7CgEdN4au4p7y0sLy+e1gjkaSCRIDIfNbaBkDnt1AOPTmus1rVraXxRbf2w+o3Nl9ldPtNzAqmRjnLRqoA2hu3OeehrzSxllhvEeM4dG4YHGDXXeIJLK60+wlnv76W/VnWbzX3pyQRtOTznIOPbvUtXVjkqwXNYrXjyeHNSltbe+imKhHjntpMq6kBlIPrz9RVnw7qBi1hLk2rXZzvZFHzE9TWHdacqarP9kY3NvGDKpJBYxgZycY5A6jtirui38tpqQmYPHLEw2nbjaR0BFctaCSbRhWgnCyOlObNpL1bee3jdiYyRkKpPAJH5V1fh3Ukv4/JtL9rYzAxzbAgHPcD+vvXOXlpN4jtbi/8AtD2scEOWSRNqSbQSMf3jkEUzw3aSQky2skLIHCyPv+dc9PlPWvPqQtDmktTieqv1PSjp8mmgK4gVIIsCdF2swHOf6VwHifVZbjbBDJtAOTk9T6/rXe+JbqOOy0uG4kRobuRVaRe2Bk5Hpx0rjr/QYr/xJc29jFm5Ub2tGQRnZsDbxj5e+D055rKjQlL390aRSizMljvS8MeqK6hkWRHb5t64/hPpXW+ErWO5nkW18q1Zfukrnee272/lXP3NpfWem6fDKoiAZmt93IBPVT755wPWrej+NbcTql9ppF9DlRPakIx7Hep4NOUfevbQ1VpK5s3siloWebdKkzA+Qu1ZEHT5gQev41i2t8llfrLdxTrLsIkDEqSp6DJ/Con8QPeapCphXAOxAOHxnOMj3pnii6tl1Wa2vJEilhAUtG28sDyCR688j1qfZuSvbYKfN7SyOgn1mFII2tVa8nkcRiCFSGZjyQoPJwP8jrWiDleQy+zDBH1ry/Tkt5Jbf7VqTJbw3JYE7lfBxkq3RSQMYyK9MjntbhfMspVltySEdRgYBx0rzMfhIUaanBdT6zLMXUqycKj2Q8imEVJnIxSYryEz2iLOKtWl00EyyKeQarlaYDg02lJWYSSkrHeQypKqyIcqwBFZHi/Sk1HSBdiMPNZ/MR03R/xDPt1/A+tGg3PmQvETyhyPoa34sMNrAFSMEHoRXNhK0sJik1seLWhyu3Y+c7q2gsNRaK3leSPHmhWH3MnlW/HvXZ+Hzb2yC6uZ1JkbGAegPGfwqhcRR6br+o6TcAMomKB367RyOfoRUOs2kujWFteWHz2RRo7lTzsPqM9s+nSv0SjLmfMjKdrWYogk0+5vlKI7B8q4IOFY5HX2IrNur8S69FJAiiMpmVQMEN0/H/8AXSprdnF4aummDPI6YVl5YvkYye3/ANapvDscFxp6yXCpJGyhXkY4PsAfxraVmvUhafI4GE9MV2fgjwu/iTUsShlsoMNO47+iA+p/QfhXK6Fpt1rGowWNnF5k0zbVHQD1JPYAck19I+H9Et9C0mCxthlYxlnxzI56sfqf0xXi5zj/AKrTtH4nsepOvyxstzRht44kSOKNI4o1CRoi4CqOwFSE7eF6+vpUmNoHqaYRivz+dSU5c0nqcaIGGMk8mq0lWnqvIKcTWJTcmozUzimECuhM1TIyPammpTgUw1SZVxnvTSaVjUbNVIAJqJ3xSPIAKqSz471tGDY7D5Jcd6pTXAAPNQXF0Bnmsa6vevNdtGg2VYnvb7ggGuW1C4MhODVmaWS4k2Rglj2qW103c3mPE04HXC/Ln+tezQpxprUa7I5iRiSaSPTJroqzHZGf4j3+grq7yExxFfsqxoe5jAqG2iXDSOoZQMAZxk+30rtVfTQlwT3M6CygtUxGhz/E7Hk0obfId3QdKuvaFmyTlSMjHNVJ7dlBGDmpUnJ6ha2xVkkYXAVOV9cV0ujSPHhldkYdChwRXPiB124Ukn0rorS3MVpmRtuQeMZonK2w43e5JPJBLeCLIK8AEcCt+ZprdQtoJIyAu3acAA9sfyrmbNkt79XmiSWPGGVhkYPH6V0dvHHJM0c06pABuJ25MvouewAH6inBLdmFVu9hGulMcTO6LIASwkXC7B2B69cioNNlS9vWmlHlx5xiNR+g6f8A66bJbnVL8ohIRnwpd+gzxk9BXTaDo1tbXMnllZZEj3K65+R+2M9Tn296cIe2lZF3VKDb3MDxJA8UcA8uDBJ5iffg4BKMemRkfnVAyG209USFySd7v2fBxj2x0/CtTVPM8sSsE+8xZFU5TJ6Nx6/lnFZyOptnjCA7sZ68fl29q0k1GTSQRTlFNsu6ZqTCHAl8rIJwBncff8q0JtUurmULPI7w46HCk5+mMn3ri7288pgFAUjj5ehHP61raTfpNFK0zsGEf7vAzls9PYdannqLRPQ0dOHxW1Ojea3t5pJkLMkuHaFUCJux9fl/DOeOlZuoX1xcsJWd3PUHPQ5znircb/aUVTwwXC7yeee3vk1RvIi0zB22sec8c5rOpUk1toOnGKfmWNMcsjZQu3Y55B9q27JHkZFnDeefnAGMgY69a52zxChRWPf2wTXQ6QF07BuFMjSAMjYDHA+vbn9B0qKVOMrt7EV7oqa/4Y0bWpGN5ABdlVxPGdsnsM45AHY1mzfDjSL6OCNEks5Y0Ks8DD94f9rdkE/THWupv4WnBmiIWR33nAx7cH09sVdgQLAiyOhkROXUnjPrxyOldEL81k9Djly8t3ueB+KPDFz4bvkguJRLFJkwy7cFgOuR2PI/OsSa3RoBMFYnGW2jhR0Fe4+M9Bi11IIpEuXCtlGjJ3biMcLznivFNQsXtNQntHZsQymNdwxkA8H8sV10p9Gctalb3kY4UtLjGQD1p6xo3ykYOepqRn8mT5VBB61bjtonZA0gEjdzwq+lb3ZzSdipJEI0zu5HTjrWjG1lHCqyxGZvL3Foz91j0H5dazr+Ka2uWt51ZXTjB5wPb1HvVfzGQcZJIxmqtczepZubGU20d4qqYZGZVJOMlcZ4/wCBCqUSq4JyRzjGM1NK7QCSJwVkQ7SpPSoMgvG8ZYMTz6Z9quKErkl1HZpNEsEjSZQGQldqhj2H0459fai+aB1EiiPe/URrt2Y46dOetWoZbFhO9/avcSMpCOkhUh+xPb9D096yXAUHDZX2qxLcjO6Q9sj14oHAzUkjIEC4U8cYHT61BjPQ0yyVCCCMc9RQHODUIOG64NP3nb0osAjJ3yMVYtGiSaMzLvjDDevqM84/CoCo2gg5zTBuGODRuJq5dRYCGABI3HBZsfKOmRUU0kX2mRbbckJb5NxyQPr3qEu6ZAyM9RSKcMdwzRYSRpNYxxQtPDdKNgXdFLw+T6Y6j34q7c6Uh0Nr5LlHuIijSLF8wEb8Ak54Ib5SMdxzWRBdGG3ljjRd8gw2RnK+n581JBcxSSxJdhzAPlfy8ByvsTQ9zKUXe5e0PV00+ctOiyQuCskbrkOCMEHuOvUVVfUJJSwaWRlJzl2yT9ferWqadZQaTDdWl3HK7SFWQ7lkAPTcpGPxBOc1TtNOe5gmeJgzxx+YUHpnB/Tn8KlxTJcY2udnpN1PqOiS2F1d7LbYxgy2PLlUFlz7N8w/H2rWt1tdItrRbhJyGA89WIXd344Ht3rzWC6dEaPPHXOa7fwvqo1BorDUPJnUkIhudxCrjGMg8L07cdq461K61OGtRcdTp71pljhhtWDwWu28t7gSbzET0XjpyeQe49znP1aOebU21a1Q2jOm1liOFVjwQvovt74rp9Rtra18NSzGz+ziJfKwnyeWVJ4x3GfXJrjofF0Vq629xbma3kQgr91ixOAQT2/Q/hXJF1XKy2MVdr3TprjUnBsoroytHafeBIfJB/I9+fQ+lYdzJZz6sh0y08i3mb5urZkJJOCe2McVelv9J1maa0ijsIrh22E3UjRoG6ELjgHPGeAD7VRuZNHtNPNtc2dxDqqXBjC7goQA9x0z/OiGHmk+ZkxTjrYTVrPT4rdXNxOt0YyQFcAKwz7c8YNcrqekT25S5FyLyBkWR5URsIx6q2ehBwM9Dmuwnt9It7y3le9N5arG5SHZtkXBGepx0/lVW/1B2nu7XS2W7smjDkwgEhc8b165BIBx7VpTk4u1jWnKVybw3cQnT5LlopbW8ijSITxxBok5G2R0PfdgFh2/OuxsbT7BaRwGJIpAoMgQEAsepAPQE84rzrQ77U47TUGjtke3VMtJLBuVGxgfMOhIyOv6jNdxoV2t1pkLgQR7lysMWRsHTGD7gnI45rjzanKeHUo9Hr/me/ktWEarjLdmuDS0wHinA818m0fVARmoJOKsN0qtN0qojRp+H5it+F7MpBrsIT8writAUyaoiqOiluPYV2kXDCuPFRaqJnl4u3OeT/FGwMPiaS7jYB5Yo3Cjqxxg/wAq5s6pLPocthLN5ckjoEMo4OTyfy611fxWYSeJLaMMMrbKGH4k/wAq4KGG3e7JLtgICu89D7Gvv8NJezUupyJXikzQuIrjR7SNPsimzkAjLqFZWycnP5cZrf0WPQbmCS1m32yyj+ByFRj04/rWJZWb39pLYxPtmk5Qv9w7SDye3Sqz6fqMVolyQnlgiOQBvmB7fgfWuyKk7SRMuXZnWfBvQlh0u41qVf3s5MEJPZB94j6nA/CvV41xzWP4Z05dJ8OafYqMeTboG/3iMn9Sa2wOMV+b5vi3iMTKXToa69RDyST1pjCpDTSK8m93cEV3FQOKtMKhcVrFmiZTcVA1WZKqvxmuiJshpppakZ6geQc1qolCs1V5Jcd6ZLPisXUtVjtvlLgOe3XFddKi5uyKSL804GeazLm7A71mSayASwmAXv8AWlfUo5Yh5rq4HQ7c4/GvUpYKRTaXUbJK88mxD16k9B9avpolu04mV5Lq1VBuWRSmW7/dP3fTnNZUt7ZImA/CknKjBJqtD4jlVgkIYoOuT1Fd0aEorQltM6OPTrNixhiXeesa/KoFXVSBYlSRGRPRZM9P0qul1BHp4kYgO+CqD+ZPesm41tTI2+LdHyAm7HbrU8snKzKS0N+O8eWPavlEL/f5z9fWq1y7yu8nlR+YcZIVec8dKwLOW6ILEBVH3SzY3fT1q4dZmQ7ZDkRnGCen0qXGSNFGL1Rppp8SWYlkhO9s5LEfy7Vz2o22JMqCV7H1roH1qJ7Ziybyy/KxY/KfX34z+dZ8FxDeXWJNoUDArSMpbgoPqZsFidyOc5681ekc21uc9ORkd61bj7Ik37mUSooypKYzxzkfnXN69cyNItpBDsVEG7C8sTzljzzz+VdEI3u5EylayRf0mAXMUrSKhMmAqsOcflVy40e7k23ICxQsflUOOMdeM/0rP0WW8tHa4G0vGvmDPoP8/wCTWlLfXFy0tw0GFkbdhV+VSe1U1ZGezuaOl6eJYwTKq4PzDBHFdPp7wxqsHmkqfmAJJ8rn+XB568+9cxYTy2U9pNchfJufuEHIbB5Bx39veukSEqj3ESRToS6AAEcN0/Hg5Fb4aXK3Zamdd8270MvXLiGXIQKJRyqp/D93gnp0B4965QxyxAOsLvE+QCRjd61t62Y4rVtm77TuAwFwPcfQcY/Gs2EM8KiQFT/DkH9Kzq1G5XkXSglHQwBBFc3afad6w7hu2dQuef0zRpxEdxJGpOAcAGt9bJoZItyExOd23HBx39+pq3DpdnqF9CYItrKCGCj7/fOPWl7RP3Sm2tTU0uzkszaX7SKDKDsb7xQ4Pbk9qwdTlkXVp5CoXzZGYICeF3EDr16da6O5vxBbRWrxqQu0CTuVB4z6dBxWXrdj9teCexIaVEO5QMjYP/rmtHKLhyoypycZ80ikJ5nVZGX5I/lyPckjP1Oa04ZysImBGFbYQPfv9Dj9Kqz6ffQaaFmjTmYbdjEk/LyMdCOh9a3rHS47a0uIJ4TI7xKI5AuVDHI459R1+tQqMpOyNZ1ocpdg3FIbh3aUPFhN5KhO3y9cjnqf51C+p3MJbBZdmV+XHBzkg/rSaddtcWgXzGZ4jgrI5IRNoAKkc9RyP04qJI2k8maOZonSUeaWQjGTtK5HqOeOvPtXS6d7ODOTRN8yJI3lilUbC20DaYXGAwGQcf8A6jXmvjvSRLA2rW8YEnnMsyq3QHJAOeeK9Kt49u9Y4sFQN+GwWBPOB3OfyrH1yCBwJs+Yg/dlSoKlR2xkZxnI+g6URdnqTNJ6HhAjLDeQT9BViHzUdZFUHawYZOM45rXuNLibUJ0tbgqhY+Wsi8j647VU1rS9R0GZIbyNAWUMGikEikHnqOh9jXSnc8ybV+Ux7yVpbjBABHy4B4Uegz25pDCxiVoVd5FJ8wFPlCjGDnP9B29agkJdi2ME9ferdlcTREeQqvJnkFdwI64I9OK3WhnJWWhQnQpcyJIoV1OCFYMB+IprSrtAIPHQDpRLvExYkbicmlggE0h8x9qgZJ9asCa3fdnMeRioEtpLi6W3hGWdgozxyTgfzq3uNv8AKr8glQPWpAI3VpfPYz8ARCPIP454/KpTIk7bFX7PJp1/Lb3MI8xN0ZDj7rA+/cEfzqk7HzC2OTyat3t00rkMSWUkEk5JqoJv9nqfWrTbKjfqMBGc45NOUZDAdRS4A5PQ+gpiMQTjg+/eqKHegJoRvXsetMzk9qXPBA4zQBIJsTbu/Xp1qaGeBroyy2yOuPuZIBPqcVTxg8+nFCkqcA4osS1c0tTtLa0aJrO6M6yRhmJTbtJ6qOecdM1uaNpOnX2l21tICNSursLHIT8qoByD9TiuUYyuAWcsU4AJ6CtnSbqS4vbSGOaGJ9wjV5MBFzxljj360mZVIvl3I/EFrBba/fWkERgjglaLYWLYKnB5PrjP41nFZY4x8reW3Ab1Heuh8S6NMPEEqq8k2+NZZJQu5SfuuykcMoYEA1X0i2vLzVIolUF0Pl+XIMD5eowf5etD0BTSjcyS0OzdGxDg8AjtW74WvGi1iAi5EBc7C5UlQDwQQOxqbxTp1rb6rJarYQ6e8RJLQvI0cm7BGA/K4546/lUNppumzaR+7uZDqu9SijAjKk4KnPQ9859sVFSKaszOpKM4anReKfENwLdrD7S8wdFYsX4x6bc/KcjBB54rBtb2S60ifT5Y0kXeJonZfmjccHB9COCOnQ9q1tbsb7Vb3TrSLUE1HUmtxFKnAKOrEbS3RuADuNQQ2F/pOoQ6bqelXKSuVJCrhyCcfKfut1HPTNYez5F7pzJKMdNyhZafPI+0ISvoKsarcwwvGiP5koGCSchfaumsTcaR4hihIE9qPmDJHtZ1yVw6t9xgQcqfTuDWt4v0DSde1O0/svEd35O66ZV2hfTcBxnn8q5XK0/fZnzK+p51Bb3d6kjxhpBty5X+FR1xXX+EPDkN2/mm8kiccERR73UdyB3x145xU2haP/YSSPd3MUkfmAPCpHK54PPuMEdR9K7/AEy3026uXmeSG3Zk3qbc7WGOcse59+vFDk3NR6BKV9jxC4j1DSbo21x58Sy93yqzKG4OO4z+Vd/pmpfZ30yK8j8sPb+UlxIRwV4Ctg5UYA6jPIODmrPi21ttV0RoFdZvJnZ0mXaRHlu3OUDDOc5XOBxxWLYGC50C1meMXvkyPbSGWIJKUziMh+cbQQcc8gjoQa3lRhVi1Pax00KsozjKG6O3ilSWNZI2DIwyrA5BFSDpVW1gS2tYreP7kahR+FWAa+FqJKT5dj72F+VXHMarvzmpWNRxRPcTLFGMu5wBUxQ27K5t+F4mEs1xjjGwfzP9K6y3Uu6j1NZ9jZJY26QId20ctjqe5q1eX8Wj6VdahMcLBGWGe57CuelB4nFKD2TPGxFTmbkjxD4m6gs3je92H5YtsfHcgVx6XLoMk5C9ql1W8k1G6nnl5eWQyMQeuao/wkjgY4z3r7yEEo2RmtFY001YpKjgBcEd+9dIur3dlO1tfWypHcRmOWK5GI5FPRlYcN6ggkVwTkMBlsDPUdTWlo2r3ViDDC0ktsDhrZxuj+oB4Vu4IxXdh6fNLl6mNWVlc+moRgCrFQR9Kmr8eqXvqdDA0hp1IRUCInFQSdKsPVaU4FaRNIlWU4qhLJzU9xLWXPLjNd1KFzaI6SUCqks+M81DLP15rPnucZ5rup0bmqRJd3nloW6kdBXE380ks0hVBuJOea3ppWlOAepxycVkw2kt1csqYxkgsegzXs4Smoas0cLowftHlD5kbjup6U2WS1eNmkvpEdR8sfllix4/Lgnn2r0Wy0K0trKdpbQyJtKiUjdzgjdgHjnGPTFYE9vpgkWB4BsTJOcdDyRXrRqQS5mccoScrIwLREvImZZNxC4GCev9fpU1rFcQyFDAwI5OVxWzea3EIhtCKsYCoipgKB6VC2tJcW4yQjggAKv3h3rNzlNbaGsYqPU0Y4blo1YoC3celSPpt5aQm/mghZc4Afnn6VBaXt3bzQO0O+Meo4Yf1rp1mk1SGFGjVrRG2uSgzzyPlzzWdOC1uFSo+hnxtfy6bDaztE0B2/u0j2nPbJ/iNUxZ2/2o2ohAkBYkuvf0z6Yx+dP1LVDBeyWcRCLGdo2evr6//qqzJ57m3EdmAyoGZ1ILN1ySRVuS2ZMU1qirDpscl2Imtp2hRVBMThuT/j6Y4rQsvCokAuTdGG3YZ8zZnb9Rn9Kt6VAXvEDRGNwTJvUDcQR7+grTnUw6FNFZjhyfkUZxt9D704KMnexMqk46JnlWqatc2072pmLrGxVTjtnt3xWv4dluZrZSyKY3YsMjqPWuJ1WZ/tsplyHLnOexrsdK1e1XTI18sviLYMNtAbHU+v0rWdO0VY3jUcjcury4tdV8u3kSYvGqqMgqAf4emBzz+ta1jNc2pi08hIZtpdmTjIPuO1YNsk8mnB2n/cTsNiZ5yD94/jUcc8iX8wDCRyQm5jznpWMtnYhLmdjqJRHLIkSndHGfkQ8gfT863bRoWspISFjjQAsw4Yjn3+bv+FYGh4kmeaZF8uJfmZ2xtJIGR757fWta+gilt4WN8pMoxlIy4YjGMgc//qrTCRklzvUK9tIGX4imgkvVtI/muNu8vEwYs5HAyeg781gWn2uW8G8NJsG0gCmy790ztNDtjycltuRnHGe/NWNL1Rh+9WRQOqqV3bvY/hU1byfM1a5cY8itF3LGv6vDLYRiNG8+IbS7YBH0A6D2rmYdWuklAhkKtngg4xXUXNtBdXS3E8cR8wKAidDhcdB74/KqY0WFLMXiyxl/NGYQCCQc4I/KqtzahCSirE1vem6gktroEORtbnkEVWh0/XNOilZRHc2ssZ8x4W3bADuwQen3QSBWr/Ykwso795l3MyqQ3GAeBn2zx6itXS9Ge5iYyNtjkUo2HAHPQNnnGeKqlTlCXLbcmtUg43T2GGTz7WEQS7oYV3EyFVwx9Mdcgf5zWrYagLhJ7ZpikrIBlhgx4B3fXjnmqEZktbGGyWdUKOyyxlPmDqwwPQ5yMH1pX0W2vbiOOG4bzyZU3ocqSPmDYx35B9TXSotSvE4r+7qR6LbhBE6riZroKolPDIB8wI98+lbqac5nuDbQSRGR9rK2P3ePmRlOTg4wOlYFtod80LyPNDLGp+VlO0kAEMSfUAE4PWrOlatJFctYzu3nI21kLf63nPHo/p2boexralyxtFqwql5Xad2Lb/6NK0cuSrjBmCkbj2+h4xj6GsjWLlNNilknRzIsoKkciTPY+hPP+RV7UXewvLdzFIlsZd6oxBKsOODjAyADjJxyDVLVo4buzlJjD4JDo6DG3ocf3cdjnrzxXJJLnszazaUjzTXJGsbr+0Iov3E3VW+YIxHI/lz61zmo6zLeoAsZCJxk8810GrXRgsLi0uHV3X5QeRk8EHn2/UVzsNyjQzNI+PMVlc7d2M9OM+oHNdVKPc87EpKXNEyPNLMS3P0FTW6uCSvyg8gZ5pVJjt32gbnUjkAnnr9KijmfI2glx6V0vY527kuo24tZ0Vim8gFgrZ2+x96qrJtzz1/WtObSymni4uBJG8mXjeTIWRQDnHbrjv8ATPNR2ekTT2dxcCJpIo4TIzoQfK+bA3fjnj05qktCOZWKiwSv+8PC5qeW/uY4/JEg2DPbH1pZd0ShEYlfpTLmymFp9ryrxE4yp+6QASPrgj86mN2x3iZjg/e9aFAb6U92Hk4H0NIgIOduAa0voUK3y46kU2T5vmp75AAwB6dqj3ZHTJ9qEMiJ64pVb8qaetIDg8VYD256k57UfLs7lj2poUluO1KcrICwH+NAh2TGoyME8062uGt5llUAleQCKiZy3Xt0oHLDPAot3BovyXF5dwKzySvHbrtA5IQMxOPYEk/nVmWyubaxgvxcJIkmSQrEsh6YYEdT/KrF5cxDSFjhjaB7hgCFdtjRqOmCcH5ufrVa4gNra+VLDOWlCvExJVSO5xj5h71Jje500FzHqmiJaW7ia4uSIWtZAWKsuCrRnqGOCO4wSOOhxNUha1e2MCbEdN6gMGBIJUnPXqDwcEVkQu6ThUDKynoeoIro9YinJtDdKzXToAbjeGSZeikHvxxUTMmuSRXm1Bo9nltJA+0BPLckY7gg/wCNa/iA4uLeG31SS9tkgRonYnMe7ll56c9veqdz4Yv4IYp4vJuo3j8wCGQM6jvlD8wxz0BHvS6bpGoaoZWsoVmEIVpvnVdqk4zyelYyb2JlyPU2ptTQ+E1uFuEi1S2kWJWVv3kkR/vA53YPfgiptCj1PRsXMk+xtStC5ld8lFfgNj+8OtX9U8PaNDb4+yzWzKPllS5EyOD3II69sj07VZ8Qaba65Y2F5pl5H9rtLZVNs3y+cF43IemRzxxXDUrQm1Ha/X0ObmWw+3vNH0i1eK/jS5u5YhgjIwuMAtjoSB65NIjre6ReajHaGCwjwP3DjbGw6ZUchT685x14rhLuecSSi5hkSZ2+beMfNj/CrtnPrGhWk8/kSLa3EB3bkDoyHAGR2GSvNb0INaSF7NXVh2k6tdW2uGVHV4trq6ifytyYwcY69jjHOK1DcXF3aS6baTRyQwwbwShbysdAD0BJ4x71yyXlkPC80ckiDUEuVeICNt5jKkMC3TbnBx61R03VvseoQyv5skO9WkiSQpux06V0uDbXY6Y03uj1bwhqAudJjiKsGQnHORjGe5yPx/D26TdxXlWn3lpdeM0uiz2aPtZg42ln65PPGeP85r2Gz0qe8w2PLjP8TDr9BXyOcUIUK11s9T6zL8S5Ub1OhRCPK6pGpZ2OAAOTXV6RpK2KGWT5p3GCey+wqax0yCyX92Muernqa1Iod3Pb3rwZVZVHyU0KviefRbBDEWYCvG/jB47Jm/4R/TZQYoj/AKSwGcuO34V6Z4n8RW/h7Qrq78zBRSqHrukI+UD+f4V8y2mkaj4n1KbySm4uPMnuJNkasx+UFj3Y5AHU19PkOXpPnktTzqs7aspQ6o0h2MOW4AA61vaZ4d1XVommijS2tIyVlubp9iKw6j+8Tz0ArastC0nwvqm2MrrOoBUMMqFo1gmViTgDGew5PY+vFfV/Ej3QDXd5JczZ5jJO1fx7/pX3OHyr2nvT0R51THtaQ1M46TDazSb5UvVXKiTayRj0Ycg568H8qa+qWtomxYlmIXCj7qofUAday7vUprk7XJCAfKi8AfhWe5Oc17NOlRoLlgvmcknUqu82fX8JDRqynKsAQfUGpx0rlvA2ux694Us7gH97EghlGcnIHB/EfyrqFOa/nzMcM8PXlTZ9IndXHUGjtQa88CJzgVn3UuBVyd8KaxrqXJNdNGF2aQRUuJc8d6yrmbGeatXEnBrEvLjGa9ehTudCIri5255rJnusnrTLm4JJ5qjKZAoYqQp6Ejg17FKjY0RoW7pLFIpzu9KcImRVRGxg8is2yuPLuApPyvx+PatSadQ4lZjnOee9aTi4ysjaMlym5aK8djMs8zOjD7meD9fWuN1ee2WU/MA2fur2q/dahdSKyxAlG5wO1cnPDNM7uwCnccbjiurD023qck522LMUQuZDx8o5HNa1jpkE9zFEHbDYLgDkVlWbESKi9+DXQ2yRRsu6QocZLA1pUnyuyCMbq7LFw0tg5Vbd/JhBRHI6k/5Nalm9rK9zcQ3F7tQKxTcAVHGcse1Y41JL13gZsn7iAjOT0yPeptVtDp+jeajxM0hCyhexzxgfh604d2RLsYc2oi51aSbb8xfg/Suq0e+Rg3mmRTEPMwgAzjk8n2z+Ncvp2nWsySTPMyhDlu2D2rqNKWKayklucByrruCkHOOMc4+pIpNLmua3fLY2LC/RxJsH7ts4GclQegz9DU9xFezW+21AZG39WICEj/PFchpaXUV4IS4IOMc139/JDZ6BbSfI1wfl+XlQQOhB4PPOanDwlzScnohV1FJcq1Z4Fq8Ehund8ZLEnmptMm/cmHOAOCa6dvDF34n1K8a3aCJkJZlySMe3t2rkY4LjT9Ta1nQBxJsYZ7/Wu9SU4WTMleMj0jSbmbyNgSAIkJMasMfOB8v6/hWYYZUmaSRCh7Af59aWzaO3skkFy/m5wyFeAPr9cfnWzbONSt3RPLDINxLcY+hrhk3ezOqnZPmNHQfktZJJbYShtv3jgAZzn9DU5uvtN3i2uYYjEflkVtuFwcjjHA/PHrVaSBz4fuooJiJ1iHyEc9cce2DXldrNqWkakx+eKUHa6SL94Z6EV00otLfYio02/M6XxVALO/dLe7luFbl2cfMG/iye/NY1pfzW5VgSAOK6RWGqlGmkZ9xLBmbOCeta2peELdtLMFjdSXN05WaOOKIlCT8uCc8HpzjjvwatJVW7ITk6SXMZuleIVd1WRynBXcvvxXfaZ9hu44pICjGB1+Rn2l16gnsNpPH1rxtrSWGISRhiOMtjAGe1b3h7V7qxlV5D8jZRlIDBlPUEH1qIfu5a7FTj7Vabnod7ZJi/t4ZFVhJxvZlzyGII6qTt47cdelaOnyrqGkqIoxDdxkJGQ20FhyAxB6nB59RXOyXl7qNkZ7fysxvtQyO29WI4APqcDHPXjvUOk6neaLe20N6ksaTplSDjOeBg+oPUcYNb8yck0tDkdN8rTeqHatqMseoxh0Xy0VWlWJm2Z67WGeoPet3R5J5pWezfzbgwIWQn5WxgH3+7t5HIbnoabAYNQSa9BhjDwFcTAlXBUhkwDznIPWo0tHt5IJkZzIsWxXUkOzqqkKP72FwMex5JxWtN68xEmmuS1i0LiS00me8CtF8zFkYByH8w4I5HHAVsDtmqO6HVrHyooI1u0dWiKnblcncoz9ScZ7e1bMF3DbxyWzKZojt2ROgdXGBjn1I6++PeqaOthbechjFuAFSUR/I2CQQ3fndjjp0q6idkr6dSIuzbtr0M+8RPKSaVXkWdnMweEoRk9VOexB49Sfes8mRVCbbaTlYyxbdksSQT+W3HbI7Vf1C5057GARGGJ12l1idiXUdPUYBycnpkDvWXPcMkMcKToqmNcSqmAjZ3d/4uozx37Hnkk1zm/wBk8/8AFFtcy2Jnk3G3jk8tDtwozk8fnmuEYHsOK9E1qMtEVcF1VGB3LiTbjI6dh/WvOHkYFox+lddE48Qr6jXlOBgkN0qa1kQcck9TVd0I69cZ4Oas6dDC13ELqTy4WYeYwIyqZ+Yj3xniuhrQ4m7al6O9llWOzn3y2YYyeQTxna3I9OpqTw6qiW5jmt5ZInhKNKlv5vkk4+Y45AGD0Of1qnLuS58tEdF4xu+9tPI/TBrpHktrTy7psXB1KESNIjuhiYkgo3Pzcg5P0II6GYt2Oeb7GXc2lnfQPdRF43PyRLDGdruq85GeCfkOc/xHiqFhYyXwukSdIXhhaUq+fn29uOh61WW1uXSaSCF3jj4dlGSoJ4zirFpd3OmETpEfKnjaNgQdsqHAZT7dOntVIWttBt39qs0W1urco23jOBlTz268n8OlZhOH25wPQmupF5Z6rPEdWknld4yGkkcEIQDtKkYIzhQcg9zzWDf6cbaeXyWDxK5UNxzzjgg4P1FWrF05rZlN5PkxnJqIkZ4pxUoSrDnPWmMBwQfrVJG40kZpM89KDSVQDxwQRz7UAknJ6U0HFOD8GlYQ8KpY8kA1EMjkVLgNGD3H61csrhLZTKXHTa8RGS6ng4yCOlAmVZJy6RrzlRjOaum4McNvKs0kjhSjRuchQDxj2OensazlClif4e2a0tMnQRSRSXBjXIcLjIJH/wBYmk9CZaIhiuFExldS8u7OWbIP1/8A11pR69dLdpKscIVSNkRTcigdQAex7jvk1RvY4ftsoiIbewaMqoUYPPTt16Ux4Hhl2sCRtyCKhkuMWtTqrBtSuIoXLSstqGaGGMbnUM24he/QE1o6ffnwpqh1Swu4r1GlaJMAbJkK5yeTgjcAVI9weKxdK1HUbKKCaFxEtsxcyAAHBGCpPcegOcHp1rnpLqWRyN5wX3nPQn1qFZ7GHs22dRqniSae+kCoqL5hcIvIG7nA9qpDVL2yeO7iXbEXLKM/Lu6N/wDXFU3tbmaaG5igEUE8gjjYSfuw/ddxPy888/yqG7WW1nWO5Rlz843EENzjIxwelR7GN72D2a2NDWdbvNZijuJtn7sCPAJLYH3c56gZIHeoX8QXlzpMWnXEgkjhG2FmX540ySVDddpPY8cCrdwtvNBbxaUqS3cse6T7MXXIKndGUbqRjqvB96r+HvCereI7ny7C0Z4wfmmbhFHu3+FVKUKcHKbska0qSkrJFrWL60fRNKWCWJ7i3Uq22HYSp5Ib1+uec9sVe8L/AA+1fxLci7t4FtLDdxLc5wR/sjGWr1fw78MrCxgtX1QrqE9uuEDRgRpznp/Fz3bNeg21rgBI1wAMADsK+bxnECb9lhFd93+iO2jhlBe+cloPw/0fSJ1u3hF3fDpPNzt9lHQfzrrRFt4AqQywWj7rhgFHUetZWreIoAT9l4GOWYYrwpUa2LTqVp3le1jrjp7sVoaFxc29qnztl/QVm3GrtgjcEQ1yl94gtLaE3E9wCpJCgHLMR1wP6muQ1LxlcvE6GUW8DAjgAu6+h9q+iyvhmviLStyx7v8ARHPXxlKlpuzQ8UXUOvX6R3c7i0gfENmiczv6u2cAHBGOv0zmuc1DWLaKBrUQw2Ftt/49LNNqsw6bhn9Tk1zt7rkj7kgHlqep/iP41jNK7nJJJPrX6Hg8voYOChFXa6nj1atSs7yeheu9TknZgMKp7DvWezMzdzTgruQPwAq5bWTyHKqGA6uThR+NdcpuWxmkolRYWbOOg6k9BVqGx+USPhU/vuOv0XvV4JFFIEhQ3M/YlcKv0HQfU1YNrBFmfVJyzEZ8pDx+JrCbUVqVFt7G58HvEqadrL6dcPthuwFBJ4Vux/P+Zr39G9etfH9ndCK5SZfklQ5DCvprwX4hGv6DBcswM6jZLj1HQ/iK/KeKMA9K8V6n0NGV1Y6wGhjxTFbNLI2FJr4lGltTOvJcA81iTycnmr99NzisW4lwDXo4eGhutCneTbQa5u+uMk1o6hc4ziubuptzHmvdwtLqWiCaXJpi3L7fLY5T0PaoWOTTcEn3r1FFWLhLUs2cazXW1ui8nFaN+pVeABgdKgsYmRCSOW5J/lWmkazWzeaOWGAfasKs7Tv2NpK6MFHLJjdtODxjIJ/Cqt1BzksudxBAOenfPvn9K0Z4Y4ZQrKDjriqVwoYYTnBz9K6oTXQ5+SzKkatBcAZz9K0jKGgOR82cVXZVBVsAEfrU4C7OcbupApy95o1ihNJvbW0vlkuF3eW24Cte+1j+2niib5LaEnAUdcnvXJ3dpJLPiFdvPLHtW5BapFCqpknHzE85q52S0e4lTu7tGgI7fT7hR+6mhkG7KjPfofQ8U2a6EN5I1m7fZgcqSuNuecY9ulQmMIilWPIGeOn+eahtb1UnljmTKgYXI681MloNRszSh3XF55offjksoIz9a1fE5uLFY1Lr5Jj3KFbcFPcE9z05q/o+nxOg2ssYI3tJtJC/lVbW7EarJJIJkEcfADk5Y9Bx2z/+uim1y6rcJv3kl0Oc8LaotvfvO7bXRgyjdjODkiqnja2jW5tdRtM7ZP49mAHHb0J6VSNo9pLu2tjdjOMUl1Ir27rOQPlLKD/IelbU1yyJmr6kYupnshcvjAxntyf/ANRrrfD17HIquEUSPxuboKxNF0ldTsoIEG7zgVGACQeemSO/v61HpNxJp9wIpUZZIyVYZyAR70VYacyKjKz5Weqt9ltlcXE6yFFVN5+UumMMAOvXv6iuM8U6JDcL9rjndixJTzEO5l9d3Tjpgehq9Dr13qfl20EUZZEIUhF3bf4sn3/wxTr3UXWCe1WTzBHH5apsIyT3/AevrW101exjFOL1epzGmWs0TAHkdwR2rrrCKfzxZq0ksTAN+7Jyu5T1wfrxnkCsaxPkG+82GNHfb5QaTJXPOOvPTOfbB61sRWpGnSTSB1eQjysKdpxnIyOP8+9Y8jjJGspKcTmNetvs980FvIn2Nzldsm5eD698HJ+h+tW73QPs2gWd5FM0s0iiR41H3UOcfXp+FbN5pGmraNPYX721wQJIowN53rxhlA4BJP8AP2qGxstVvD9tYpZpkKRAdu7eeijoOOD69+ua6uSLdn1OVVJpehStZ5dOuIJFuopipDMAenIxw3XIIP5jtW1d2cev6W8pkeO6BxGxjBD7ARk4+oBP88Cq15BCbuSxVSqQIqFWYDOeeo6/w45/Kt+Wwjg02SK0hiXYzSCVj+8YrzgHoMj88jvUR3aiti5yvZvc5TRb/UrS7FhMy4JAaObp6nII/X8q66c3EEJuXAVeVKXCr5bAEDOAflZSRkdOOtcfqs80V5Z6i+7z7lWDMRhXTJAOM9CM5GB19K6GC9+1Wv2aK6FtKJMx+Q2GhkwV2lWPQ8fNnGBzzWlNRs7kVbuzRYF7dwhZIzA08aszeThVeIEEOD7kEFSPyNVZvE7x28scdvGYmkZxMycruPJx3IOO/YGpLHSbVIhcXl2kz7iVdGGPbcDwef1p9zLBbJcBgrNCqDeY9qKxOSAB75HH4dKhzle9wSjta5y00N1FGZUt2gABZTIQu9MZHQ/e56f1qzc/ZDZB7VyZinIfgBu2Bnj6nvU1w1nfyXTRQrNPOAxYK4YtnLbSvcn19TWRqE1xDGtvZJ5yAMu9ckYPIGeuMc5PryKTUb+6TLn+0Yd3dCCOd2kV2lHHJ6f5FecXB2XDY9etdbqrzpNi4YFsc4bOO2P0rj7o5lJHc110Fqc1bYQMzkKO9aVl5iSyCGNZMoUO7gAHjv74qhZhZC437XxkGrdnfPZ6grzKk8SjZJGSMOnQr+Xftwe1dD1djz5ptaG1Bqt5pwjhv4ZRbMrpAWHMYbh2jPQn861bux1DWrfT7GzsbdJBGixyRjY04diFLEYDHIbnA4ByTWQWh1CC40yO9Dxwv5lnLdbgcAfc46Eg49Mr2zSaDcamjyRWd9LaSOPKR1OFL4YKhP8ACSCwB9zSv0OV3WpR07UdQ8PGSZIpIxcxFVZgyg4YEMp6EqygjqMir9nDaXujzyK8jTx28s4jjb/UujLlivdXQ4yOQV9BU9vp91qvh2dpFgkjtIg6yvdANCmSNoTd6jpj09awNOtDKXIuUhfgKrHBcE4O3scelNMpNO7Ogv7vTNT0sXX2ZLaZrkJI0SgFQY13YHQrlWIHBBbrimyaLZ2Fg15dy/aYbhsWrKxQOm7DOoK8/TgjByMVl6zo9zot21hdMoKkOrcgOpHDDPOCKsaDd+a7WMu+a2fDNbou9m2gsAoPv6c0rk20ujn7lUWaUISybjs3HkjsTVVkKDDAgkZHuK0b+1MTrcKyGOYllAcFl56MOxqC4Mk/lR+UAyjqoHNaRZ1RlcomipJE24xz9KjxVlAKXNJSg0AWIlQxnLfN2BpGiaSRuUBAHtUAJxwauW1uZQzksI0PzsF3bR/eI9On51NrCbsLd2BtIYZDNG/mDJCHO32PvVTADcHP1qa62eaRHIZAP4sYH4UjQ+WFyQQe3emC21LD3Qe1WAwxZU5WQKAw9ckdfx6dq6Tw7qUNtp93EBA0klvKj+cgJwR2J96yNTsbaCx01oCxkmjLSlhj5ieAPwxQLCNobjMmy4txuKPx5gHXHv3x35rOWtjGXLJHV6faeHdX/s+3uZJ4JSAk8duyoGbopwc8+/vXJ+ItLTRtcurSGQyQK2Y2I5Knpn0NJpa2zyy/abjy8LlF2bt5Hb24zz7VHa2l7qt21taQzXEshwqRgkmlpBXY6UJc1kLaQrdSwQpIqu7BSG+VfbJzjv3rXPgjxFczokGmvIGIClGBXB5BznGPeu38KfByWQpda7L5a4z9niPP/Am/wr2HTNGs9MtktrK2SKJeiqK+fx3ENKi3CgueX4f8E64YSXNzN6HmvhD4Rw2Drd63IlzN2gUfIvfOepNeoWmnw2cKQ28KRxKMKiLgD8KtrCU5PFUbzVobM7Uw7/XpXymIxGKx071X8uiO2EIx0ii+22GPfIQq981QuPEcVocWqhzjlm6Vy+pa6Wy083HoTgD/AArida8VZHlWZ3FlyXPCr9PX9K93KslxFeV6Ufn0RnWq06S99/I63W/FCK7SzyBm7KD0rjNX8RT3cjpZORCg+aYjA+oHYVyt7qi7vMkl+0zH1+6tZt1qt7dp5cs7mIdEzhRj2r7/AC/IcPhPfqe9L8DyK2MqVdIaIu3GsiEssJ8xj1dxnB9QKyJZZJmLyMeepJqEtg8cn1pyRtIwHJJ6cZJr3OfojmUUtRw2k4H61YigMsmxFLtj7q/19KsRWKxLuuW8sf3B98/X0q9HHI8W2JBbwdzjk/4/Woa/mJvfYrRWcMTMtwQ8mDtiQZXPYEj/APVVsQSSoDcMIYh92Nev0q5aWA27ooyF7yuOv0Hf+VX7WFIpVbG5v7zcmsKtdRVolxhfcwr64/sqABIfKDDgHgn61yl9eTXLnzHJHoOldb42+9ay8YdW6HpXEPz1rinUcjphBImmUDZIvcYb6133wu8V/wBja4ltcOfstx8je3vXDIAyNGf4hwfQ9qhhlktrgMMq6H8jXiV6Ma9J05rRnoxlZ3PstCOxB9xSzE+WTXJfD7xIniHw1AxbM9uojkz39D/T8K68YZcHkV+V4/Cyw1dwZ2Rd1c5W+m/eEVjXk+Aa0NX3QX0sbjGORj0PSudv58AgGvRw1O6RqZV/cZzWPI241YupdznmqoGTXvUocqKuMK5NWLWHzJOnC8mmiPPFW7WRUnVCo464q5S00NKT1L3lYUMx5I6VKilWTJOAcAfrUbgszSdARgCp7Ty2VmbJdR3rjk9Lm7dyvexRSEsB8wPJrLaIkENwK1JZBggd+/rWfcyqhKkduorak3sFkUJyofC9BTIW80hs4A4xUJnRpHzKEHUZqFLwByFYZ613KDsS5pMv+aFOCOOxq1b3AJ+ckD2rNy00LEDGPXtVX7VJHkZ4J70Km2aKaNz7ViUoHDDPHbipIjB9qR3AYA5rElm8sIcAHGTipre7iMalsjnk+lKVN20KjOPU9U0fVEt0R7SYC4dijxuuUdMd/wAeMUyygivtR2XJSGN2LElgoA9B2rzu21by5AobA9a7DTilwjmSZY1EZOT3IGQPxoTfNFNaIidKNnJPcyvGSQw6kLW12bYowHKMCpbqeR16iuHuW3OM5I969BaKG4d5LxWKsf4MDJ9PasG+0i3k3PHxk8DFbe0TldkKm0rbl7QddXR7Nzb7fOaAxo+OYycfMPfGfzqgXjinjcgSK2C4Y/e9Rx/+uqUNhKs4VATjk8VoG1+zyBZxiQYwD055z7ii7tbog5Ve/U1tKgutNuLLVwuElukgJX5SitwSeMYx07ZrbuImvdahbTpHZ5lRww2gh8sSrc4DZGR7VnQaoiaY0FyZJIsYiRHx84HBPsCa0LGV9G0W0u28z7RcSrLBIWypByHA7KwX8eRiu1OEoqPTc45RnGTk99kcxJq8tzrs81wqh3PljeM4/h/ya9A8PPZvbyR3VuSVCMGckgKwIDjHYnGfbntXKR6Pp95rDo+6KGORVKbclWLADjOSP8a27a+Ia9itLkRyqXjEacSFcnkjt6gDpzWUWoy55FVHeHKjR0VYbuW8jtHCZuMRy7fvKSfl9uccegrWCI8RkntljkmXzSUO4x54wBnHXIH0XGayLScW0UU2wmFd6kRdVB5z+PXB71qXeq2eSF3xsMgs0f8AD1I64bs23OecjkV0UVFwtI56jcpXic35cf28LKkYvICRIQOGGQRISeCCCMdvaumtbcSWshVRIcbUlRv9USflGDyOTjHTiuP1qGSS5e5sp5F2LsBaPJCjkEBjuAyxB4471E0uowIbqC/i2ggTMpYBmPIXAB54/wA81nGSg3pc0cXO2tjWvrC2S/iNxJIsEcW9DOduSOqDjB6kdPfnGKxtUurdFF/EkciSBYnidSwT6P2OAAOOn0qKSe81TSmKrIiyS8Dd8rN6cntn9apA2sVm9pMFkcAnmTKnkYIBHJ4OPrXM5c2ysjoUVHd3YsGt7pEhnJCAhGOcjb+Ht361s391ZXViZsAQxBISPNO49g2D1BPXuM8d64eSSe0v2spbdwASyqwwzKenTPOOo9cirMPib7NNcQw4MbqysHOQqkg4BPsMZo5GnYbnFpNGqnkm1uDZbg6neGHVVz0LZ/UD+H3zWXf3E1tCILx2IUFkiYcKWByPUHofx+tY2p6959zPcReWjOTlIhtXn2rn5tbkcMjkMx4yRkgZ7V004XMas0SaldebLIQANxOFB4HsK5yU5NWp7rzJOOlQxoH3MckDAC+pNddOPKtTgqu5JaRsYyM43nANaOoaKIgklvN5ibN0qnrEc4+YjtkjB9+cVWTYJ44mwwJAbBwAO9dlFMbDxLBYrCl2wUwzyXGAZlZs/N82GwDj73zADpTbe6PMqycZaHLaYEsdYRb22eQR/eQHJHoR6+uM81f1GNbOBRYXOLe+iWWaGN8plWO3ryOQeDyuepBzUr2Meh65b3swilhMjMsEMpDRcnyzk5wPusOTxVm805V1y9F1FAhgWG4eKJ8RzR/L5hB9wd3HvgcUO/QylK7uiHTo7vUdE+yxWKThLgKZmMYK+ZwqqTg7iynHOOenNa1pKklpp9vahLm3MM8NzbOnUF+rA/dfgMNp4IBGCWFY8NyNIk1PSry0VoC7NFuYFkYHj5gPmDLx6cgjFdH4YEK+HponuLYNfTYHl3CxTDaD8uSCq5JyOAD0yKmpfltF6kSVl7pyHii0eCOwm8zzY5I2VWLAnKNtIP6Y9sVjWiytcZgOxsE7mbaFGPWuv17TW0+S3ubxYnghk8t4Af3i553OMEAE9BnnBxxzVXQLHQ5beVp9VaG6CuRG9oGBx93aS304xRTbUFfc1hK0LGRcapLdzB7gowCLGU2AB8DHPvyTn1qC4sUjs1lt5WkJb58Ajy+pAORycDORXS63Na6tp6u4X7ase6GQKq70DHcG25+bP97HbBxxWEJ90kKyxxRRrH5TbE6jOdzZPzHp+VWpFxutjBfdkkjHzdPSmNjsc1YuVAuJVAyu442cj2x7VB93t+dbI6EMpc0pptMB2at2d49pLuU4BUow9VYYI/KqeaWhq4WuOZsPnrR5hLZIzQvQ5p8OQ4wCfQVI7XLd1qDXNrawhAvkAjI75NTstzeRo2WeTbtjCrkuPTjnNdZ4Y+F+s+IXS5ukNjZscmSRfnYf7K17T4Z8DaN4ahAtLVXnwN08o3OT7Ht+FeNjs5w2F91Pml2X+ZpDDXPJ/Cnwk1DUo459XX7FASGCdZWH0/h/GvZdD8L6ZoMHl2FqqMfvyEZd/qa3Y7ckcDipTc2llGxnkUN2FfK18biswk1KXLHt0OpRjDZDIoQR83FSXN7ZWUILyDfjoOprmtT8QZ3eS3lxj+I9a5HU/EEVqoeWQANyOcu30H9TW2XZbUnLkpx5myqjjFc03ZHVan4geY7UJRT0UdTXC6t4rht2eND5soONiH+bf4VzWreJriWIxxu9vEw+fLfPJ9fb2rlJtQJyI/lHr3r7zLeF6VK1TE6vt0+fc8utj2/dpbdzZ1XW5r2YyXDqvpDFwo/z+dYtxfS3Py9FHQCqpYO2C2DTAzEbRnFfUR5aceWCsjgs3rIVjjknJppDSc4wKsW9lLcHKj5R1duFFX4o4YWCwqZ5R/ERwv0HaizerC9tirBpruA8pEUf95up+grQt0xlLGLHZpX6n8f8KljtDI+6djI/90dPxq8sQxhug6KvQVEqkYbAouW5DBYKD5gBmkBwWPRT1rUt7WMuDKdzjpkfL+VNQKgG3HsBTzKAMA84zgVw1K7expGnYmlmfoc7gNpJ7gVC7siuxPUcY70IzTOUjjeRsZ2qMnH9BUV3eadp4Hnv9rnI4ghJ2D6twT9Bx71nCnOo9CpTjEwtfhnvUgZFHGQWLYC/U1kw2kEEg8oG4nz95lwFPqP/AK9a97NcapO00+2JCOI4lCgD6Cr+maNJOm75YIAM+Y4+8ewA7/XpXdTwigrzMZV2cMr5UEUlyNwWYdTw1QwPjgnirYjDq3IIxyO9fOyVnc9iL0sdZ8OPFMuga5GjORbTHbIpPBBr6VglEiK6nKsMgjuK+Noi0MwOcMp7V9IfDDxMut+H0tZXH2m1GME8lP8A6x/nXynEmW+1j7eHTc6aM/snQeKLBprMXcSkvD94Dunf8uv515vfTZHXrXsh+ZSCMg+teU+LtIbSboMgP2eUkx+2O3614GU1bv2Ut1sdUNjlXbc9SxJuwAKrg5krStk4zivoJvlQCGIpExAyQKitYy0y7cliePetA4VTnPNQWqeXIGXqM4+lZKejN6SNTakgYAgccVXSTEpRB8wXHTqKXzxwo6DviqvmPHeifnacqR6g1lGN73Nh85WNPoP1rBvZGZsDknsK17sl1Hr1qiYlEYL5BIrpoWjqypLQwpLWR3IAyR+lUZFeKYE9q3mlaO5LYG1gM8dBVG42SEnaDXpQm72OeUbktlODb47nnH0pkkD5yQQpPGaisD/pAULgZPBrRuZhMAqjAA7VErxnoEdiGW2MqFh0wKppbsrBASQTk1pRzbwEx944pskcUbkg8etKM2tDS3Up+UVbOa6PSb2byUibJTIJX1x71hxss8+w/dUZxW/bL8ojhj+fHAFTUeqNab0Na7uopbKGzgADtMXlwc5/uj6DJoh0m4azmn48mPhmY4H0HvUVuPsVzEUCySsnzZ5w1daks0uiybpj9o2tIYXxtZF+bKH/AAqqMHUk3LojOpP2a06nOpCbC7S9aIpFOzNHgY2AdCPX2qjfTSzENMGKZIAPTPUj+VXGvJZzNCdkpdTjI+70OR6Hj+dVpDC8iRDd5SSMxbOC3YH04xTm09EEE73ZLa2YuI0hjYfvZApjLgZcqdrDPocg/wD1604J2Ok3OkRQfaoXt45kEj4MMxOC6fn09vrVV032dzalIhdG2Z4WRgdzKclcepAIH0HrzT01pWlRJIpGVt5X5cMTjnnqeVxitoOUV6mU0pt36E6WQvLi4C71ukj87G0hncKMqB2/+tV5Fs5tRu1uJVWR5g6zsxDKSPvZHODwfUc8Gqk1vE16jQXKmWWRHgKylQVZclCM5Vhgc55PHpWfNLbzafEpGbtJNuWQhyCCDyDyAQD0zz7VooWWpN0ztmttk01vK/mfeeJ9rFZYwfldmQHdjkdB79qzpr6QalKgSFDLwksbId5H3SrevXAODWTY6tFbwvDNqE8hEYWJbY/JKDn5HHHfHbkH15qW8ttLuooYJLtrKVpZpJ45BkWrqOAVxuIYYAyeo96pRuuWBlbkeprXsMl5HKwvY2uYgzOIlOFUAfMT37jjuaxbe5mkupQ940kLTJHLI64jJU/uyw4BGcdeRycVozwRQ2H2u4kmhVmFvDKgMgACA/MDzswTjp681hxWM97LdfY1ubjO4I4YDz8YO5iQRwWHU5+ZcYpcvlqCkur0NeQ3flXXmxiK4aRmMMcfyM54ZlHTuPz4rNltLq73hnkgBL+aJWRVfruI3ck5GCPU1LZLC+mx3ECIbqR/KkSYgAZz0JI2EYwSfXii88S2pWG2+zx7IpVmmjdeGPRuM89e3XntRTjbWTHU10ijB8SWS2lvFLZXccirIYRtUrKinDBmOB82D16/nSjw3F9jhuL65c26gqn2dBuZecy4bBcfKeMggY6cVY1zUrXULmZfMLw5cmTsxxtDKqgbeikD8+DXKSag8cMTRs+9SSSTx7YFXKWuhlye7qY+u2i2GoSQwz+bESWRsYO3tketYTkhskkc9av3skszvNIzNuP3jzWZIx6V1U1oc85ClhjNW7KEySeTvCNIRgnt6VQRS7YzWzaiymiVHMwfcWcpg5UDgAevXnNXLsclWbSHXWlX1nqr2c4AniOWwcjBGQQR1GCDUBubnT7ouQyyryO2MitLWNVnvLuCdjIfKt0t4ZWOWZEGFycDJxxzzjFallcaU/h0XeowLPcWzlAhXmVSPlBOein0xxUpa2OOU3o2rnN6NdTDV7SdYzM8MiyBMZztOf6V1sGl3XiK4vr64a6igt3lRI0j3sjMzOI8EggZL/l0rnvC7eRq6TW8yR3QcJbq+cMzcYJ6AYPU11Y19rK8E9w8kMyTJcSkDIluF4bpyDjv0yPfNKpZIzq/FZHGzPJHdPEGEpDbd2OoHTFbFxoUtnb2Ul1PIDdQm4RNu0bM4DA55zz6EY96hhu9Pn157nUIGW1nlZ5PK4KbskFR7E5x9a3tcUm1s7iaGONU+VEGSrIVDK6N0ZTz7gjmspP3WK7ukYdtZ3l5bXUUc8UlqzRJIGkIIY52EAkA7eRzwN1QafpVtJaXkst9HbTRFfKDuPmB3dR7naMjpnJ4zViTXJ5TfQiESJenmNRtRJN2Q6qOM8ke2aoyvetbx6fdRuUt5W2I+Q0DFhvGPfHIPpVxasrlx8y7oEzWtlqVxLHCYwmwtLFuI3dgc5HKggjOMHPGawLmVLid9v7tCcqpOePrWtLZvHpFxeRvgeYsZBIJ5z2z9ex/DPL9MistQurQXSP9n2CC4aOEfKSGAIJYDdxuGSOQeMVfMmkaQl1RzMxA+UBQR3XvUXzPwASfatRtJJE8iyr5cROC/wApYZwOKoNG0R3YBXOOua0jJPY3TTIDuHBGMUlOOTz603rVjFFKASOlbWgeFtX8R3Ah020eXn5nPCL9TXtHhf4PafpTR3OrH7bcgBhGVxGp+nf8a4cbmVDCRvN69kbQpOR5J4b8C614llU21uUt88zyDC/h617h4U+GWj+H1SaSMXd6P+Wso4B/2R2rtbe1jgjWOKNUReAFGAKthEiQvIwVR3NfGYzOsTjHyU/dj5fqdUacYEcVtxgDinzNFbR75GAAqnfeI7WGLy7cbmA61yGoa00kgDOXdj8qjvXNDA8zSWrLV3q9Dor7xG20x24CL/ePWuTv9ZVVaR5AQPvOx4H+P0FczqfiZUdkjKzNt4Ck7VPv6/yrjtS1qadwZ5zKV+6v8K/QV9rlfC86qU8R7q7df+AcNfHQp+7T1Z1N74l5Msa+aB91phhFbrnb/F+P5VyF/rElxcyzsfMmkYs0jdzWbc301y2XbjsBwB+FVjn0NfaYbC0MHDkpRseXOdSq7zZJJK8jFnYkn3piKW4x16VKkXABU7m6ADJNXEtViXNw3lr/AHFPzH6ntW15TehDtEhtrGSaXy40LvjOxTzgc8ntVoW8EJHm4kk7Rp938fWpYw8seyJBDD645NXbe1WMZVcHHLtyTSbjDzFZsriKWcgyt5aDgItW1gWNNoXYPQdalTCnI6+p60EkjtXLUxFzWNMVWCgADA709W+bOM/WogOMmrSWUhgFxcMLW26iWYYD4/ujqx+nHqRXNeU3ZFtxitRrdB2z0+vtU7x29jH5uozeScZ8leZG/D+H8efaqr68tuSmkwujAY+1u3z59uyj2Hr1rKSCWe5IKNcTyHAwCTk+grso4PrM5p1pPbRF28165uIzb2KmztSMMinl/dj1Y8nr61Bpek3eozmCwt3nlAy7Dog9WJ4A+tbdr4ais7mM60zovzboIj84OOMnsM+nNXrrW1ghNpZqtvag5WCIYA+vcn3NdDqwpq0TJRlL4R9ro2l6VCJbqRb28H8AX9yh/H7x756VTvdWeaUncXc8A1lXupCNN9xLtB6KOprDv/E8f2aS3tIFAcbWkkG5yPT0A+nPvXHVrN7nRTo221P/2Q==\",\n \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAIAAAC6s0uzAAEAAElEQVR4Adz997ckyXXnCbp7uAgtns6XqjJLoAoaIAg2AKLZJJs9Pd09vadn9+ycWfnDnrO7v+/+OfvD/LY7Z870TM/sshVJkAABQmsUUFUogUqdT8ULHe7hEfv5XovwF09koQqCZI/nSw9zc3MT165dZdfM/GD7/+GtroW/CvEbBNyI8X3FujuPi4AoS2bxvLCHtdtaer5beCHfeIuFUqxekdt8Pl9+4165O1Ellfte16rEVcWuTu/eXs5nMfcXVpa7rzJZq8/5b/zAan4+kif3OYGioGVWXunK/C9l4CKCXEC/oglFtpc+XNbfxZPMXWfwPP8Bb4uUBIrH86nOnvI8X+a4+tA1p1QqnSVaCwUOT1bdV3ybZdlaqrOgS3/2vAq5/C90CrktZrlL4mru7sSkaUpW7uKR5nPNFvMojufWYrJav8Iw5JGU7u7yKXl+yQ8N+vPFQgi58CiO8IL0pPF5DxwWJTJfLEDmURANfC8r5RV/Xua+WMSgbA76hDOvNPP8MVULF6mfk2YxB/OD+syGAJ/nc+rpzXP1oLKlJPJWrVQiY2UWeLOSKlnKSuHci+YBdZqFfqoHkmb0X3PudfpZe5TdKdU+XK9/cX/neiVpNxtxHPv5bL7IkyAMI7AQrNJgDX0v8P08z/I0m8+mYTr1ZtPZbEYHcae7aZpqshqPVp8llMDNUhwKaewiWXEBecIunvAyQeDnAJBSFw5bBHONOC+3/p0tvNQPgJICqtJ8MvOHo/HpeNKf5eM0701mx1l+Oiud5nEv9Qdxy9u86e3ebFQ2wqi88JOsVJstoozGhpWQgvJg7idRqVLxSuWo2vSTxsIv514liGqlUm0+7wSlD3vBjheVvSD2fHWWtwjnPlWOF/S2GprPF7PFIiMULrLDN1/vP3n6lX/75e/99Xeqk7CSxoverBGUZ8NsDuhocriYLMbT+cSPg0q5lnazwIvUk1z0oe8DF90Nq4jhsiC/AkRpLmCFSuVRMkhAOCBXo6qlub6l77grNzpRfaiLiOJOgNoq9lK8i7wc7z6/EA9t861WF+J5BEBFEe6tquRFSbmRxM1GY7PV2Wu3bsSVjcW8ns0TP2gGYS2fx9m8FIblRVA6Pjp94+Thf/etL/18/vSed9Lz8lGUj4NpNpsE+bjiZRUvj710ATp6WeZ749ibAapZ5M/DhS94zu0uKNBMICNohNxLhl0lz0aHF/J6LhwXvntBpNHqhaUwWcxLDEANWz8it8CPvIBXNBqAO4yNCPMmZ4TG8SKKo0ajtbvT3Nrx4rA3mXSHx5/41EsvvnLzxTs3QLH5fBKX0pI3nmej0iLVyKIPs9kiFY2KokoYl8aLqSo/Yzxxt2EFfi0WUcnG+1wvZgvd80UulOPL97gE81/zAlEcrri7ZUjBZx0MpvGKeJfgPYpbVaaoVRG4/BEtvBxJDCTlyvhfIbIofT2wEk+W+blX3NXG/zSv9678le26MvLK1hegc58UH7r44vHytxc+dAmWVV3hyeWvLsdQhHWM3fx5YJSRZOAn92WGxpt5hHAvoLbc/SgQzSQNA8Rog+cxnoTG4n28ZVSDgXBu0Q8fXsxXEAi7oLtgoRMUXExxL4URpZXCRTATKoE3DAyyWIzHkOoKbCibBZO0Mvf3tzdfvnFjv1XrlLxKEodBMJ9RThAGpQjiRI5z2BODf54xGLIUjrtIp9l07OcwvxzuSzSXA4C701538TUBlX2J1i8BVtR4LaBXjmCoxgaZ5TOUCKYLn0P6yD0/g+fl81R/yAZw4uUf4dzIE5LBDEiLUgbQM8krYVgqxXyZwuJdzakj8gzCT5DPA2gpCT0ATXfMIYyeh7yCNDD1/JEeieRLKJ6fCPw+nNg63VoqxuzlwHvn+v5OZyvJkzDzf/Dl7wynw3pYPul1G2GlFCFPzBelPPQihJcUiE76Fb8CjKieAwMQWGa6avgaeJ4ZlJCvvlYClxHZqPt/a5fyX9V5vZBnxRdpQJvFHHiG8zzMsrBWz+NkHpRaadpDxM29ZD6PgiAsJ8nW1saimfzXNza/8vr3/+on3/z5/NGMDomjSWkGw2KMzCSaIFfSw0ggsDHER9ivpBmB1MaHQGBgAawSF+yPjgS9l4K5RpkbVkonvFNdEUQZca5r+U4jESmQ3pMcpiwpQnkAA0EaMY6ofJaOJ93jk3S+qDSbUaWys7nz2hs/n2TD8XC0t9tqNsNG1U9CDeSwlCAbM96iKAoYruQ3BydmpcjPhQ8l600NcTVFg8EpZhIaqDCVVP/64JJdqqtd9Iql0EeErTFn6LVM9B4/q04960ggV+CkBYo8L2KAfbsE4OUiSL3KfD1Q5H3hCypwIcY9rvK48uUHiLTqLKG2Xh83bM6ab2AkwbPq8wGK/FtJ6trlWuTu6225sgqXm+YyuTKxy/PyK5dJkZUrVPlcTmoxjkGSjMslIS3SLGGfYSb8Ft4Vf0RazArTjQDxPp9DmrmgR2K6ClomTiN3UtyqEPo2mphgHvkohqWSX2IsWukMOvRbCBC8GdUKJg0zhyVCrhHukcONqFAditSwwvLhKAUVtSZYxZUsJRtkaHgMD8qcT4wm+WE6a0znHRS6OHm+3frI9f2Xru+3vFk5R3NewLpISplclABVEO9dwGizOewim2IzWGRTbzoJ5tL2HOtVhc5fNN9yEBwA1kycbHkViYuAe1E8LuG+Sm+/0umsqdAV2oOUAuE1pgvBQitfTDJp5FmWT7J8jHo08ybSHfx5HHuVqhcnMkXAucmNgEDorhwQQs5gwMbXrQdFE2Gli1TKrk8zR16pK1QIyp5X9hbJwkt8WKYHFRYHIgdhCkq6suav5NUbXmly4+Mf/exJ/xc/e+fR0TuVBN2mNBoNUV4pGWAieoWlmPakM8LL5lE93uoue4aHZUUV5r9idNEG+/0lN8tENVPHk/dv+iJ/tXVZWwtYIYaY9miQAQuXtXcV4I1gn5cMo0bzXjbLJ2mOGaXRCqpVDCXNMMacQuQ8m0qpB2rtSvUT7a1KnLTL5W+8/ePvH/78QXoEXJJapz/qjsSIJBBhrwHz0ZoTG7s0XVR0CS6UWKuVA94KhAVSWsQcOxW1pkeEJ4Th3nPUacQsmDsWBXJgPIF+4AY2R54F4IAOpWAe5/l8lnpzjLVjzESnk3Q0mXS2tiqdRjYNfv7mgycHJ8/d2n3ppVu3bm76JQS4BeM/n03yfIrUEfiLkiiHh0iLRWOObCcYIlVQH8GVR4cLC6ELzJ7UtICX4ZkG7HplCW0erJN4PBcgR5fi8n2VvnhjmGRPlzHvUuLiq2fhnKvGucqcfXNFCCJyRSxRz2hAkfOFr96jpqQsvioCtNo13N2JX3+8kHnRMZfif+sRRYUvlFTEEyjCpFkPr3/iWrce48Lr6dfTFPHrkXyyDiXCRTINvqsulwAuUrw0tgHOn/VwkUmRhoArqHhF98CljPNqDJnKJbMzuZE5vW+MymVAYQzyiCKxWWmACZcQciFWqAUZlFevfNhulC9gmgyuPMxnIWkcGmkg2IjnOz6jmdRH5AC2vWyIn2HPNo2bmkE7IE+BTNhV1K9+2kjT55qdj+9de3lj51aj00CwR5OEkWHj9YMoDKR9IDv4KJqi+HM0yTSb8Zel8LgFOotrinUotSp6AZbIo7tWSZa/FyKLTy4kU7w1k4B+VTzs1nUIUEFbAA4SKyCO84XTeuG9ozQdp9komxv3zSe5n3phFoWQ6bDWDCuViPagG2Uo8gAnliqjXgTqYogQOP7nRgEhvtA19Cy6EXNxUAomnn+qSF8MeOFhOChDJz2/qp5coDUDjyXKWN0JB4sJDKR0+5UXPvl7nxwcd8cnw812uz+bQMepwHSW+hm1gC2TlSnnauqSrLgKARkX42IBYAGrIiQO4LDBvUNAFPxsymsFSfIt0hc5/GoBVx9XE8IuABzRy8jwQnyRoPhqGUDAofk0DguqLmosnCey2YGLMm8Depcm09F0BN5PMcek6fS52mbnk5/f39qp/6j6jfs/euQdpyMvLlUmzEbkwWyRYYaGMVa8MAmjFEYohZVaOeBJbLOQ8VcNMYpQA9yA4UOgqE+KUWSzjJL7+C/Dr/ixLCOGkXpcMEIZ7GbplwTHm5JkU/gmzJ7azL1sMTuazcJBv7zZGk4nx4fDfPZ4NE27p/3nbm5vb9cnMyaL8gCxGwl+kYfBPA6YDAlBOMaskJxOVykUCKZBUqipokQuZKpB8Mglf1zoYXpCnzAy+bEAd8KKMWKjBiwvF7jwuHrpflc2t/OxelonnVbm5ST/ScYYvRZMllhj4BFMl3aSv++NsqrS3fpzY8DFuPDl2ru+c+Nz/a0TgNbj18NFyiKyCBSv3jsghDQQu7urhkVK1XWPlgNoa39KzgcMNkUroFQ8iBitXRarxE5EJdmSmJKGoW8ys0UhSiubGX8+PAN7Lwk0a8uYEmX25kxHoUFijNJgNDDaABT/FieyaRjRL0a9bFWihl6ZQgJlqtEpAqz5rupiURlOGunselT+ZHvrd/ZvPNds1+Z+OB6HWJspNwhjrNMxZloTCxawsynKLxoKc1QQSwgFJTKGoQ6kNlgYPFYjsRBYBSRrH+0FkjAxBykiz+Idcq9i7Nc9OFhJ2TCKA3xcKYqRLXwxhfXOF5NZPs2yaTYfTbMh3DdF8RUDHmfQvyBDe0ctSqpRrVYuV/0wAsIUKbN5nIh0QtpcZegmKRoe5gYM93QPVkGBbTnceOMfesHYk9ZbhQF7Xo16MB1Pp1jfIUjZXJ7RNTgsLgF+q+GNJsl2+3P/+IunJ8ff+YuvHowPKhXxFuYWwkCKXqbOpw5AEmCBKq7hwi6R+hVUhVc8qXO5LVUC8rEIRV64DPj20ljEhbe//iP5UyPyKQIuz+KxCLg0F97yOJ1OSqVZEGR+EAOFfJDSd73+0XjSq7euNZt7UakdlCPrFmv6LAqm80pSrey/tFVpPL+z/5U3vvfD/puxH52WonFpOkdvzNJ5ntKP0QKLNAx4CSkHiGUdMBqJsekC3IwXF+ZHLI0W8d9aphAT9ThM2GWVADPACvCC0cFQROqi34yxK9bGovRkbNS8LXkIAeM5GJqfnk7Bx3q10uhMxunPfnrv3r0nT1669fKHbu5u1WrVsJXU43I5SIeL2QhWDBtW3VWKKmSjfjncjBkz0AV/JGoxeWO+S4HY9YqwxYVogvXTsg2rsBIs2+3enN1devrvLIowf1aLs8gLj2cvlqH1ctdfOsxw+Vs1VNCzEvNKROeqqyA0F14+K6tV0y8klwCxXo1lxZaGfhuNBgpqWSS7mIV7pjv+Lq5ntdd6jJe6ino54BeP5wNnydwn9ulZpEu83n0OVsXdBegXFzif+Qd7grRhjgTiZLVef8JXZw6ZW5qdNbFro5ju0Mg0QUrZEGna2zKHKM8lysr6aQqYD8WXI4lUX7kf4asGaWcgBqAgOYEKQhSAaUOPZ6PaLmPDURIsi4bAzDFgUerMn6E8ogvTfl5GWV4eTe/UWp/cvv6J3b3n4vJGvoghciG0ZBEyORkGaGRogSL/GAdn0zkKI3/oxlAdGysQOGmG1Grtoix3uXGhep6/1C+rgUZKEnA/n0RPq0gUCoijyBiNcDlLYQUQiAHzMZO+cF/uWYbWO53lgywf8WevJrk3XaC5lphhW5QiHH6CqFzSdLjASkXE1YsaWuYql8tIMBQtgxHLFMGEImqGJCSn480CzNFY/amCiDRAgBuCb+LBOBbJ8rxqYzoZxeWmtJY82/jQ7S/8yeeODx6++f0fDyYe0kEww8EtKZUipjNR75FmctzmREzPRrGrIbUyuKi/XebEkN6YhEVZzxICvsIOpRcO8g97CPHLHC52yPLbD/pDbkXFigCZSA60Tl9PUJTu6nAGcy/HrIs9gnuAMx09Mp+PxxncN59PEaTwI6zVZlHUhJ2Rs9qONUiOidlGLW7v3bnW6Vzf3PnQ4zf+7Cdf9zJwcZYjQpbibDGlz8ZiTjgwaBrGAZS7DRk1l05ax131IaADWKSw8URSjQADHc4DNJgXdCT9S0L+SM0wJNIyE2JrkAkBsigO4aAYODBaeTIM2yBGPD48mZuQGJUj0DFLgzfffvzg4eNPf/rDe9v18Hq7Uqnh8ZWNaMA4ZXI7FLOntgANIFB//VA9SAoxvHEaMrigyJUTlgY6r9wfyVa9TqCAvvvcEj7zpjRLzFMldKl3LTviCZPifH9bEiVw8YBeX12+lvlefrGq6+U3v5GYtaG1np/kXGuIIs8CSwpFW1wzXKP06MCynsXfyzBV5XJVcwFhzyrmcpXXXxF213qyCw13j9yLi8SEHSMsPrycT/HKBdznJHOPBIhZhu13+cKGnXvFnUgX1oeGkG6ulyfrMV5CUtRelC3L/Iyw8irAx9hUXsim8Vay4BdZW+5OsAFMUiXC8FGz/mqwK2vGJGNfd+Xh6mlqE+NcFIQSRQ2MM2AKJhTKMTPAIyUIKgu/UwruVpofb23+zvb+3UqjCfedDcvMQ8YxjLoUlOIoxAQtqRPXoGk6l70UXizrmtF0miLZmmGuFhJLGYIAJSlEYDnklwNU6bn02hgwAYXXgOxi3F1ZWmJlDgBhwIb+hI3CAyKIG6wXt+cpf7MZ874jTffmo9l8xGQiLFkMG/XXz4JwhodytZlUquK+udjqDCCZiCCqDk2V3YA7L3FrlWcp8MbznNoDAmb/4HTYmaco014e6VOpuXjGRQukpflUei6zmWo/AgwpNekAFCC62B5jOjRczEJYg3/7o8//o3/xD1vt8js/efP08fGwm1aDIIkrgmXqY+2fTTHJitsLPgYOYCGYk5mgbTkbvB2s3uNu4FIeBBw83yPxr/CqyLYIuEyAadF9RblqDzVxKVyVwE/fx/2c1vkAVmIRQh2SKt06yrKgP5A9Np2Ma7XdOGwFYWXu1SdzvKLr0bzkDfNSFl4vN1p3P/7J5z9UThfff/Tmj3vvHM2m01KQRxHMeOTP6K0YV0hpiNg2BFk6lVq4uwX0bFhNwE0BKJHmbehCoZz+yIJqUiH7VHYf+DXWDiRBfa/egaGrr6yRwTzF8wBjFf9gQViMYMOyF3mYpntzbzjI6tXa7na93BhPx48Onn7Xf/369c3JZDreb7erCCTMX/MxUzzKnYrwozIpWb2PNEi2FCpQ86ghR7Q8OVeXXhBvXxOwlqzeubzOnj5ISHkKMCrU8udjK0pluXBx17Mg+IzLvVrVzVXS1fnyBxL2f8uXA1EBq1V9zpXrqlekuVSjc4kvvf07i6DCXK54AheQYb1aRTIii7ALFPBx6XkssiJcXLwlvP5Jkb8in4EPfOKSuQ+LIgzXlrkR6Up09yLb9YCRdZBT4ikVMYah+jgLB5EWT0aMHyzMWjjhYwuFV8pxM5ADhoi4BFoxCQK0ZaHFRDhAkwwKTNZnJRoTFNc2dktxRr65K4awHEdwoULLSuC9pWYcsoZmL1t8snX9w0njbqW+ha9KlqFgyAcMSgGTQvfVIjEmN6VzC6XIQqRHtaR0yKQmeA1glGQ84axzgQDtXQkcghgx7jKCsWTN5PYel+tB0U11jJVkQfEhWfiwWEIrMyBji6mYpkU/ll80IgMU3MM07csaKSN+Cc1+3mhitsRYyEQjPDGTv8xck7BAiP+Cov0IfLQUl9d8hmImHy+bmcQcoRlGGRcaMGXRPn3DV1KL0ZLFob2p6DbgkWZEpSFQQa2yMZ4PWVUT1qL05DhuBh/9/CcbzXK9Uf75j15/8OZjUVGkL1kWcKkJ8qm0MrpPcBPAYQ16QOOH3gt2gMDpanK8oXfOkMHFcyfKxUphFyHXo8p5D4j/Sq9cN13+9Mp49aRd59/KYUmT+CwJwlyvVaMMCxYSBdPpOB2PsnTCVEGjkVZLHcTCIKqI++TT0TALxkGSV1v1SqVU+7/80b/6d9/9qved7LXZw5Pc6/nzMRSbRT2B7MPi8fJlAjZLBQj8tv4DJgweYawMD8J0bgKVRpBZkHl2HQFHF+ZTPGgg7iuXesQ1tYm8lQ+DWiYr7mk2CXxWr5kzPcwSEwd/pRCbu9/o4Hg173aHKdM6e41N3LNuPnrjzV5vwMqCKRNDu81WPSlHNBejFfTBZHfwTVgHEgiMkuuFYlTNBB6IBN2MBiyb2ar73VBzQ9eQQo3Sxw47FGJOiDWJVyCG8NIuw0MKWn5DaUQHUAsDFYCTcY+VA1Yur1w3u7syKMrSw9n1LIZ69uFZ2vcKFeldfVxSItcfz3+/bKz7cO1zwW39WqErDVAb1qkurzRxddW1xPFLr55VH+vHs4+KUJHeBYrHIuML8UVDigQuAOU1hL74XsqIXaJ3ap27lo1ymbscXDLL50Le5x7d90Ux7iuXs7svSzB8KPIvkrkPi/gigFxO/R2ukY8lE5CQzV2GZsxkNIrNEK9haJe9ZWBropE5PpRLfavyuNmvJcRlhBUyGYsZkiqLjuWxQc6svhgPk8ArM72DcoTWlk2ZSgrCaCw7sspk7AA6Bz0kbVgmyhrFwcHFG2WoXiRRgIIhT8kkmMsAGzRKpcYiaM0Xz210rs3jJsuKTXSnynyLdlHGUTjC41eeMFoZk6aM7CU/tvax2sixRRpI/EIGMZBIw9AyUav0KsJrTAFFrnRlhR0JILS6DFCCHjkYeCFrKokwoMAHbAlWfSluJzlEvHYKFc5znFdGaMD85dkkW2CUxuwMOySlLUzy0xgX82pYqSWLEtbpNIyAjPQs7PUlVoFqqOJaoxjUCxZxAzyIvjdCwcYNiDWiCD0oLzR7ijQU0yytPOEnlvepMpOQo1rRWJ5Fq8AHnvJFEKdejwl108jTsGoOWeX89sfvbmw292/s/8f/6c/vv3EQ1ytJszLojlPkBizbeNzRw7SbImFESGQB9ggRXscZAA3FGFuW7EAk31BtYSE1MLZBmHjpT8BNcoEoEj0HMNUlBt7iTmCJvXqjy3UEAet6izp/U1FrV5HexRdvi4BDD74oYtS/NIn6UWn5OdD9LHS1bP0wS3ta5L1AV5RVYDQ43Whfa7S1KDjD7AGQ45IfsfRokg+QlfzdZuNffuzzNzd3/sMPv/nX93/27qw3D8NpCdEzmE17LFILbQDiUc1itHIcZSnS0hIXCYhLqzL0vtBYEIW3mcBjqegT5DmAp55QUsES/o5bWE2IyRClBRrYIB7SckgPylitzGiS+LW+kjkK1R7kjUkCLCZPD3GqiOvVzesvDEbdH/3o7cePDj/2sRc//Mpzrbi2SHvtasQqZ7I1PqgyxIbxHgf1JCWy3k6zZLoQXtY1YIvTzXWtBP6izyzAo+SSq7hv8e2lwBJ/HLB4uwwUOV/64O9nRFH/v7XqvVeJQM/14SUwnnXZL6voM1I61Lji46I+7kMeLQY9TOvqLl/2VtGW8GK2wiU3flYBUhZVcgHuFrj4rctTWa/QiWSr3Bxg9Il9e5YnWK3s7LJPVTH3CJOwrLjbh2iyzCLOTFJeVhLxESqNqhgklQaLBUf5Qot6RCvB8FIcLDY3695k7E/G6AIs+8cflJGd+awnLbMJBUQZikY1+WNoUy5TjLAl1jLg1oOcTd6VclJPkgQjdRQM5BrK9htzbzz05qV23G56C9yHyoscfiQCQl2pEUTfKIWabVtqqFHWPIpwOOIaW9xVtHQCtd3d3SuLV5CAu1xYHMEycndeuXgeL8QoXl8DQ3tjpEc8WNwXKyUmaOzPWrmh/TdysV4cXXice8z/sSBYWinLeVH9kzJzjJAwsnNKDXZk4Kzcxa7gA8Z3XUuJ50JTRntG4QUoark4HE5w0DyM/BBVGRaMtjmXK8kJdJDkL2UHoAEIOjU4IAcZSQPIT/j9wDJyFnfOa1u1lz/9crfbTWdf7z0dBExgY4XIAwkKKZPuZGZcHL0M+yXaug9BFySotWRCAXzFWaktSKO2UFtVi5Rqiw1rx9F4q9otQWmh93dznXI5reu4y/Hr6dV5VyKNVWPZtaqotYve0S9YjawBcwGfcR5GeMQuQy/NSgHT/tlmI0jCxrxcxq89ZQGcL+fhkh9nvX67knzuzke2N7a3frzzpz/8xs9nh6lfO+kPGQzyk2K8RWyPkTCr0k8nsGHEEiub1wSED1wOSgSwkHDTpAsBq5slIwFJGBSqt+YobNQSZd+zCYoUYMrjnwapmoOlRet6xUHhzPQbMqQ8GSPMROCZnPHHExb1lRuVZr3FYvDvfPe1X9x7+pEPv/TSC3vdwUmMQR1XPViurBkkZ3HWnEcVEXgJvvdATbsEqGKos2qC6x7uXKoshVtPgA2KIsUykUOLZeNd9KW7S778qHj7rK4tEvy9DRQ1LwJ/f6qq3lnvH6tZEfMe9Xw/adY/p+184oQzwqaRQFtlwbnyehasXLkuNz4sAuJuhnIuAY8EuMBO7peLKPJXGqvbMjcz8/CB/tyHlgDNoMhnGaA4PkSbkjlIlxvOvC0q48o1bUDDezb3einsIppITja9yr7C2WJ42Itmk3iWosJG5YTNMFD0prOMFUEiARSB0D2DAmirNzFNdrhiNKLyxkHMPQnaLQxZNWZ2kXIfHR92WSfJvhmnwySuX799veNL941NOVJLaQ4iOXsfkJkaokUh1FOtoAFquMZ1cVmcEtorNdAEAjWcNBaptC7gEi8jycm6ST92ubcUV3zrYngpMrZMRJVgY0vu62Z/meuV7usW+7LqV2Fmf7WTkBSjYLpg2jXyk6SUVPAuk//3ckLOlWs0k0LVNjXZxBARTFkstWMa5K0k0zBCDiHrfJRvl1IE1V5ZG+092QtibF8iXy36F2ZMZ4ltm+VypuXIZCir5CJsl1vNxu/kn8FX6NXv/OzBWw+nsOAkwTlWfuYz7LHIUlK3DAyYN1RPwYO6si+UE3qENmakxmCBBGWceQkwkVxNe4LxwgCLhQtBu1cwdQmX92dpuoLMVdeyNy+9ouXELYtYjSPFuLGj9Br4+jXzuGMbyH5A31iFgMorvYbBIWMxFUBnZGP265iNhtWpX6lu+aUWZo00KE1kxSizZ1syT1jv3Yian9q/W681WuX6n/3wmz8avZ15ZcAqQ5QWruPxCE7BVtlnTpUEOI73wIxdqcyoOFhZDYvbMo6P6X7rBbWICmv2wTrBRWvw02uaVOAT5DohhXBG2AJJ0oRHoKkTKpLOkBHhwdnMw9I+HGWdZrDZLlXCydR7d3iIin583P3EK7uNehlPPZBywh5feYZ8XS7jtZfCkfXPyBFqMBox18U5YNVSaLHqAAWX19Udu3r73r+gBRmTpsCPInDxw7OOP/fmWeldtueS/kYfinIvBP7Wyr3QGo1Thx9rL9YrU4RdoKj2WnIFi2QX4sFOQ/VL0faJY2Pk6ci96JcQdw1LVt+Rvyt6/e7KJaaoW1ENAlzulQu7u0bC6uJtca3ihFHFh0RCBYxiLL/iFZHLrOwb0rtvXcB9S4MsH0bj8q0jcBJUdYlSy3Q4ZxcpaAAzWzG2Nel2s0llkbUDvxak1Qi3HTJiyS3jK4x99kwKBhlWSnKVBQrjL5QYMi3TEzYvSBErHyNmEtkFL6jGfq2Mg0/IXNpidJoePvWH42Qwbe1WXmhttAMMoqhkAFw0g2U3LJukGoK+6sfuVhhdYSFqvesQmgZ70t2UhyWd5NleF0KGIlaXUl+AmL0sgEZCl6aIcY/Lr4zUSZGQeZz6yJg/X5jns7bfwu15udiXZUjoxFKCLQ2yCQSJ3cPM+TmKy8aAUWepjJEjV0EZn+VyRcq1Souq6A/SCTkW44UDi5jyjKqp/T3F7+Qrh7zCvijoatA9tjw0ezBMl27WLDkzhUCWOTzaCA9mFxVkCH8eyXAhm2U6bF/b/OI/+f2Nrc5X/+Jrb712DwOpzI2USH546IF9cnyTeENt4MYaH1J2xW7FPIywO47KI8kBGIUKZa0FjivQWHgwd2JFsF3jz98L7C2iXcKiX4p4F3hWvFViOYgufMJjkacLqIFu8KrywBnvB9iVeKKsPAZyQI3dhk0qFhiEsulpGqT1XlDa9INOXq35QRkhFOs/Ug1TBqcHB/F4end787/83B92qvXmD7/2jcOfd/PRFNzwAoxACLEUFaEHa0MbxBsxSmgORaqbTJhxLBmIc7nhS7eZJEMENhHdgTnV48YqQQd31ZYcBVx+MAvzFlMNLwE5Sxugg1JChZmUJjbN5BAoE3gZa321CYs3Gp6Ox629nWqlxg5zR497k+FgMRvu7TVv3dhrNspyEEROlSkGcwsYyHeSKrmgCMIqba3pqsAPAergAiI1pFdBSq7gr3tBw5TF1ej062b+W/7eKWeCiAHDQCPYXHUZOl714hlxz8jmGakVTRe5PlGXnb+KmCJw/v37e1rK3+cSS2livYB8BuRVzxICEXsIxTMu0ijZVdfluhUpL78qYoo0F7J0CYpkBACP8HdJpJd14JEPyQSYcXeXxWhu27irnlbvjM8Rr1cMX8eGeY91tE7ixWIaTAfhfNQKstv1+FY9/tSdW2X2ih1O7h0cvnMyeJKxz0+MORUOg67MthLMLDLiueSvHLDXrALcsXDNsjGuHKcZmz1U2BNqNhmfPH48OziKR1ljXtr3w/240tIexxIPxBmg7OzKGCZeGGMW00pK8QddKkBYYfQE7mMXHcXY489eGbFXU/VH1BJADq8UY8nW4i3VEtNcDparbipsdVG6oiB+DozilPzJ4Czjs20bhF7jAuxKjd8y0HXlSGyiWSGzhHifMa3B2iOMEygkoJ2YLa2mcdZE3awEtdJqr7pJUwK2uJbgbY4B2+2wrAUz+H9BMWEVKLriuSLjQmKYMZN/8GA4gqoha7e2IcYSTlKKR7MWA4ehAjoYwCQbV+Ko+dz2J6KPTtNBNh+++9oxDkUltgC2kZhOtME2mfG9POTUHaadU7apbmRI1R3DWAIUS4hbWo7VWm2k7tRQgKQ5dCp3avmsy4pQapI9K837jHdZFRm6r4rHIgCPAxo8ElBF1Ro6CEbC5pTUVsYDPNAArDzyF/lxNo9n/ah0mvi7JX8nqbSAP1kgoYWlCpIs7lnpSandqPzBhz5x6/r1yt/8xx8+/vnDwUM80dk1ZaQtWEf0jMlSxlhtrb0wZ9kwKrAKnmsq1bTLXpJIz4TpX3iwqu1mHwAx0bwEzKoX1aP3aJTQmC7gB4TRLyjGKkFs0QgG9BNbV5e87umQAY4sF/jtzY3mdvvHr9578KRxeJw+d2tvf6fFOnZvPhmMh2W2MtfWldrJ08kEkAGwWwTAdZ7u79mLBvRnNHbZ1mf/FLSYgJWyzO3SF0vsuxRPn1+OU8yVwL866a8Sq/6hJ+w6q/OqFb9Kju/vG9cpz0pbvCXgwu7u0hdhAmd1flZe5+P17VVoQPz6BQUT0yL3Z/XLKtsLCcjEvSHehYvA6otf8kv64pMityJA5a0F59CC9GRa3PX9KubMcmscikSas2GaZ3mYhPJhtJEppJ1xC9eEQ3rT02ow2a8uXt5KPnuj8dHt1vXyrJVU8kXt3W75xwenPzka/vRo8HbvtFndmEdJEseoZgw8lYsUzkwYSIX1la3p8+lkNJhMBhCzKPYHgx4MOMnm1cm8PEl3k43nys36NK+iNEMloESa6ML4jFeI/rDVWWsMT2XTpKVQfbiJOA6jQwI9EJH0LsEKurJC54twJh+lXAHKhQGFvrer+MBF8rh6s/zKoGotJIXcpmidGggDlv1Z6i8BliGh+yLBQaaxuVJh0UZZhdlHBMuCjAvQJdQkSJ8uqTMqAW9bmkXiJQM+kx30OQIiSgpQJgwxhcSKqS7YCJrdr4CFDsMQS5P660cltkkR3wUuAhqVUOW1m7YZO00ycOwdsGGIxDsuYroX76nZKGmGH//sh0fTk+7B16aPRZwFEFVKEqk6ocS+3CqJOqjq1jnWGSLmWDGUkA4xgFNHpQHp9DXZwHGpL1GSCnSR+tJloCbF2UsXdvGXklvrLseqB5ex7vMiQ/IpMiwCwiguGmYjQsvuDAdoAQsDmAPR7LYZEohgap8FZH3Y2iwL++Oan9bns6iuHUbp0nJcmeA6J4f/asYImIzrjfrv3H2lfm3vP3zrK3/+1b96O313hBsEXcU8POClT00PUp+qY4CqgAsoqRR3VcVhkoUZY8RrEFBbq7aEPMlUDCNeUlUpuHK8Z1xaj4j7WqPVPdZ8LezX+igSwHyl/6swTXbMkjL7i7DIjT3mEBWYTIrnSTLuVq/f+lB3cvrG2wfs4TF+Yf+5m1utai2MQXOWp7PHCH3KvntkpDlhLrZwXwJRhaoJMuFxV9tWnbn8pb6/zuU+Lzr818nq7+LbFTDUN38X5S/LpM/OarKqhxsh7qkIF4FVqnO/lzOx1yDguWTFA+nd5WLIHMxhSGjRx1UXia+KFtIRz9vLgQvpSaBk52OLbN3n63eXEHOvCKuGn4boMr1+GHJ2EbRsXearucxlw41xwDmkUFpqlW8SsZ6kKmXDeDZoxtnzW8mn9quf3o0+1QzuJN3p/bfa1apf29zttK9tbe/3O8lbTyZvHXTxRYHYxAnMgW0xMC/PMtZpoBswdsfzmYx1GUqAiBF+oOxYMJ10T6tRuboIy950L0pusiJnMovjvKQ1xhA9vERgMTHOWtBqaR0mwPOjLjGKYzU3UK91p0BpANHv6nIpi/Su1Q6qLuzuZFxEAiIi3aOLJ0wMF68APqDmAoZc5lKupbks86D17k7TmRWWfxTcDvbGV0gI+NxoRhzv15gN7rHk05FEIupheFYJ4oQ4KtMG1wDudLEpXVB56RJqsvifXVahbJGPsPtjH+UVxeBV48vbyyYClzYSYEbN2fQaEp0zlauvnX5l0hhMHyUHmcArh4HOlJqwWLmz13zlEy+ePD55/W8epV1vMhxTLZg0vl1a3yT1WVnQBAOMWCwKlxwBiLQKgmHiwUZsrV+MhSPnAQtwWNwFOMIaFH/5AiyXI4mhmVfGPzOyAJd9u0R616ECrnKjaUW26gXBWYimiRWl1IgDtgCKqtKbkjQFZLrQH1JNP0umaX6S+aNZVJtV6rMomUftKg77U2zKMYuEEWrzfDyadUufuHszWfxxY1H+i+9+5dXeG1YfjhjDIk33wLO0d41EMuOWDngOopeApPFu1dMbekG1VAwYpFcSxSQaIQ/TYZKU9KCqK0C7+UJhRpl6kvLUNlw6BLDctusiAuRk6X8Vd/n5+Oi4d9rbef5lPBlG0+OHT/qTydv908HdW5s7G3X8CbAeIhhrGpi8DLUoacmAFSFQn11EuAcL8Ikqr0osRY2zlO8dWoKAD1cZCirWwe/94ft8q1oKdqqcBQVcdZUJR2u1dUPqfebqkpEJFxm6gD0VrbCn394NEK1lbi1UO9/vdf7z9/vVe6TTuLJecwFS8uiGp+Hp8lOgzOMS1sJ5XYZaBTAtwqGSZWKUypIZdkGAlhejgLzQL6Qq2JAQfSdzIZP4LABZEgipDvYKlUpkVloQn+pPYjMqhXQgu6gdvzbrJozRXlHahR0PmjzSVB27R7HDHssG8LGS5KtKzSeqgDb1yVqloO5P9tvRZ+5sfPZO8/nqpN17sHj8sJMNvOPF4uRBxHKVGy+Vr21meRNy/Y13e5yfF5NRHrL/Uw6pHo3Yz4+Cs3SUY6nm7ML5BNYasWbGD9ut6tH4tB74VbYD8tJ6Kd6sttgYSpNZUmFph6ie2qMmoYOISNMiorS6kUh6SNDStWo0HaI/JZJZXem5gI8L61suAVkd4/rNRarh9rZ41HeY0UxwVyYCNHlCoaBpmHPFxyTvM6Mmz2cdebTSgMWDUX+JFDEzDXjm4/+sP+muOsgJxzIYJnZMaSvSUsACNVUFEKfpX1DRuA8x8AGhnGoodVbIR20EJ2sceljeFypAYI3GmiGDPRM4nZBKIsFgjQQBAESmDTrkMmMaMAWgKAnRqOhMM8hxnE6GZc75qYXpgu0l8s1rrU/83kfTrt99MD54goVai3C8lM0Vp2zqEGIrmZMXjZAJWd2leop6oytyN+5rHeRgb+uGaRbNQcqSiGXdJ+lg2V3WOLJSV7kLDNEX9qASFFh1uUuxflc11NRzl8vNdS4vioBLVPT+uW9IJgwD34AQAo8qIFFJMC9KUEsZfOkcyZIdn2c4DvuDiT/CUTEvteaj42ltc79WrfWzCXJZOanAntKT/r1vvnb39u3tz/9xAzvIt/zXe/ePPfYJ72sqVaMcyYnytLEq+Qv5l+117SCG8a9KWEACgQF/2WrSC1zyuBI68ONkHWu36itQaxAZ0ZAjPi0Tx9dXvBKVkf6KRJ6mA4RkNoCdgtJ9zjjBdz/x4vJ3/+JL5Rv721sbdOTTRyf948FwkD7/3LVrO424NGO7rUQSHseEsA0YA2QGUQBAuihK0w3CZYVtUPPLxSPNETgJmuSl+umCIApHrU9XA9L6wH2ljw0CZODor2K4+NQFrrivFbz+1tVqPYawcA5JUMYBnDioIwG6hNyxBGFiAmBqi4FfVEMkZ9kXqoS9ooJWN4b+xYtvxXVWnyj9sjlK6fB+ibIOgLTzGQrhxayLZ5B2CQ3VBwirwuoPQ4JlMkFeSMNdJ4lam1yJ7k4yBgD5QNWW7bIY3qKJFCkJ8Kl7NJrlCrIa2LfkYM4Uyw5yRFb1Y5zxGQoLkqzVkBgKFXUUGAySVlcDsEoQKRWmK3NL4mRW9ljMUAVFB7EF0Xg1ijQgtnyV6E6aZwHaT89muTdG4aA0NYys1EzhAq6rdJzhK0oPjhUUSqkMd53HiX0c8ZzzdMMcE5bmnEwF4kMhPnyRYqkBzh1+lMwmw7z/uBxOb7aTKszAm4RxMh6PT44HYVy/e/vO/u42HLrkZRvRYnM22Kp6G5thpz6vBk/C/rE3PQqjyeT0FKcRFGQfy/BRudGZ/u5u/cb+K/mXfvTmSff4gONkS/3p7MnxKXbYGOcr6BZe05xsx9HBE07amVRYp4onVRokNAKdcerDKBijLHmK641ZzmbvjCL6Bysay5ygfDNckmiIsWRmsmkkMGIIWIcCL4aSoaOzdsFpYH447Ap1bZSA90BR+CTQWieqm5WjoKsfdRQvCEljEFlQNDdUW15ikJdMD/EX19COvnSutFu8kxYTzpZBMZhru6thNhsQvwBG+JLrjAh5acF9p3l/XkrpKh1QhADCbDkAwXscT1n0RdmEZTBmYpjC5SoDVlAfqy8Vpfb6Y9oXGQdbp2zEDB3SqCVQOrJT10sbh2uivM7m+L9xMhJvkV40WQlApt58xCk4s3zo47culo/aDVJhW+TYOYiJt5gMOBXCn6WwC+gmbeBY4mt3N//BH3705z/+xeh7Tw8fTVnhVK7U8YxjX0LMk0BZZnUdTKkZUaodleSl5+QBBoXwXlRLvEXMmdZQZYMvY41HQI/CRbY8kob1VfgSSGeSEzvwty4TEDSIHD3S2Ljq4hNS653AotGyTEUdeOF6n1/+XbhcwlVyK0oYY5yJjLSgmQsfKQL6D2+UWYY1dNL3ywljPBMZ4ixMP+0Buf44Sp9u7tyZ96eJP6wkHT+P8dmaB2nNq+x4yfTn96r16j9nlfDOzf/xm1/66ps/mnkx5xT258PxrA9BEA2QNAJM5bun1oiGuz9grgCxIIFVDdhZ2JowW0wtVnDCj9HGDGklRVBpOx6RIaVxpjyFODhzwEdYcg7MNAeC9KdGJmyahuTAuVu2HmFB4+jyqXeCQNG/96Aa1/Dt3szD6k9+evzN77z1h//oc3ef29nbSYZpL5id1Moldu6YsJc7ZTiAkz1DqQD+qodojGuGuL/CGn7qa43Y1bd68Xd1CbNBG6DGXDjaEt0B4nOojKRg9Yr6RhXWpSov22h9o74DmO7l2l2ttovAMuEq5r1/Xb+/d5qLbwX51QDgnavhMkaVt8tJRlDgZ1RHFMkycfciTwJcLlsXdo9FjAvwleKXo1idWySz4nVzkS5/90icmOYa3lhJJu+jOagryEZ1RqHhHxgN45DDKf1iSoGNZD6ScCTckoVHOfIhY0JbCZKNGxx0FN9IJBI/1VAX3VcH8gaoQGdh+Bp1Mo0zDul2EIIvNFhhPdCgkK0dOXY7YfdX5m6m2ai3GA5vdmo3b23c7gQvbCUbZZ1XV4vr+H6wtV4pqO7s7Gy3WiVvCo9s+lk76wez/mwx8PxB6PU4gTtiy0O8qCajkO35KDQaxtNBPKuyliYuz/6f/6f/9f/4l9/7//77P3vr3pM8Km+Ua5PQn06GHF2EIpj1etNsUE/8To15yulgNORMI3PEJSf4JqIyXkwsfRDwaKP+EC9otM76BTjaF9DgXOAFxAmOwqW6qLPcHxkAMEFP7NNdxHApnVIu78Bd/UG8Cc3qUJePuAi9AVUVByWOGtFtkEKAKh9jJraZZkUCkIUZG/vSz1l7T2q3DXid5n3FnmV2kKhI0jlG9Agpaa4NraFx2u0ANKDiNIs/iqEuFCs84hnyCjrQarhg0TjyQlJRerkeO8uoMEifAyUl5C3f8HWCvUE7KcoZB55P5tRissj7i8UgwNgpSIi7shkhu1eKB2s9MQXSXPALCYcpbaQHOEqGXbG+GV273eqNdkrh494h2xQOWMhUrpanOGmB8joTQHOOiJBqB17skhbUC9SN+mmACJH5Ed4LmUlmDEPGBPkLaRzxyrpFAbRqmJugRDonESmJyBi1w4pP+PLl5CdJATa0gZQELxX1vi5ydnXWJ9RQ8FQvFJcqc5aXOKSqTKVSzRpIyJAFZ5otesMMi1TqnfiVdNzw5wmYXGqGXhVYQRy8cVouReloRE9+8sbz7a2tnW/f+NNvf/lH6esNn+3IkpPxAY4FDC22AR2PR4akQFL1sYpJPqN6DqsFOtXVVdTqjMci9VyRLOPWpKd3sNkUHy1dlUlopBmE07eoJ6ZJAgtSaqae70SdpClTiPlXjyWJsoAw5XBNznxiPXBcqZTDr/z1D975xebHP3r7+ec2KnH7dHxULuXValtsXRW+dK3i9XYNzqCDqnLuekYO59L8th4g7SLLyJWsncaOh0mevodiof+o51cchTYIXQCXYe56dcCs9ccPFF5BSahfhD9oDnyrT1SNZU1USRdp2bqcdRcrWu+Ns6JIv/xkFecel2SUj62Zxd0FVmkFFgvrXuRDwMUXMUX6s2TSNlb1ttc2DKikZiw1DG2gigkzIueod4kNFiUFg9VFpp86L0e0BdE60Fu6sS7ZU22xHGgnAJCP1vIwuBne9CmPZCPuou84RNbwE77LC4gOPqmpXE89rx6htDHnKkxAiWCIYeb1pttB73O7zU+9uHur41+rzOp+xsF1lSQc9uezjRgSXKlMatrKll16JvhA1srJcOj3x0wCZZWItUYc/LLIx+zXPpbhUJrTaD7shdV60mhuRsHj44N//umPf/b27a999wd//tVv/PTh2xzz02l1BpPTqJL4lVKeVAKOUxn1mQxm+Q1mUBPtWU4JcUVbxGFpLI9dFptSaxFeqI1ard6xxro+EkBXF69MQlk9r/pUKY3WLztsDZeU2xIHhAxStvVDCY6du47QgUSawXWKt8Fdso8WaKg+4CeMCQcrNFd+2ZNaB/3q1CNZ9WF0TArzLcxSNxVpspi2SMIYUeLoX9tV06GiK38NA622IuZcFCU4aHGHfKrIGiFFphC4JjYx2QdACurDFhlGKmkGLBPtBQlPUg1s2DQbiOZ4Ph/k+XCBaqXNKaEoNImBBodGqgjBKZYZIRrAe3VhqYaGg5VqAitrov3r2+AaStPbs3snUw6TyOOwnE5YD6tUTvuigtB6Gu8qL9wHtsJLzQTzA/ZqhBga80pvgCt+O8JiTYIjmcgnCIVLlg2q5cadJRQoZfrg4vDK5ceWAzdJqVxrqEKEwVLfSq55z4s8qQ9JikCRXJzMaq63Fms5O8lA6WkQBicarnobisgnIBvS76ydrY5YCe41Wxh4sErIYAUMJtqRBv+GxWR4WiqHdzd3//nvfWFnq/Pf/8Wfvj28fzA+aXCsZBj20x7qbzluYM+iva5c6mIVEZCtao6jilHqEoRFexy31i9SnhiEdQodpGSiJNZWtZomGPRICZehDWQBwbL3drdMTA4FA5QTgzZFAPUm/E3GnP6Z5+VOO05iJp7uvzNik6/R6OadW9utenvCZi4ZW06rD1RblbcK8/jMy+EQr+0rd9e3Rfwzv3xfL9RtV12ukhfeUGNK1gSeiDuNYZIcLMfXmx3s4MUSTdQo4GSdxIMTjVwpq7LU/A96Xa7PKrcPmpMgv/pGAfdIbmvxitSjUl4NnytLJ7Jw1tDnq8xXxZ37tRJApmX+RQVcjHu8EClzp9UH3DPskYe9AK5+UXEMPH0CcTJiI2pjQ8AZUUBYY7UqQSs+RDMdPzZcF4GCnPCJaITlR86uI1EPpSGrdPJX5tA65LB5LCFf/v4ye+nOmj6T9bE7qopgAPOCMo5hSCrn08/d7vzR862P3W40/H40OQoGJ/Mxyx5KG0mrVKr6KElsaIiNjGUrWC8h1Vk04UhUf4KXVJXhmOXD/mR8eOqNZZJFKmeLrGzQW1SqQTWJStXr5Y3T8Sn2qn/5qVf+8Sdfee3evf/wV1/5y59+ve410glWxaDerKJCPu0PmUyq1DbnGTsBgcB2GpLHhOKE5fw4Bmt8O7ECaElyEcU1grzsVkonTo1fXYoxWgGIVnGuX4hXzJWfnE+5fFrL2VErrXWlBvKiyvE6WppDTTWUfsyf2DC2ZDgw9zzFx8l0X7yfTQ8GDag4BmWb+o1ZpVEO2QdMRzohZUhjUkHUkDulq8ZCJOQmaaLSURGKpICbFs5bupxlctRLh9uwi4fIKURJTJNvFCI1/F5cE7zw5mNxavLXWuTeYt7HYYhtqAEMRYowg5tyHmBqHrsaSj6viNEqFDF+VZ9Wa7OOuFra3GmMBluT/ng+PRh15frNrAglqyg4qIaBsBWfHSQYaqJ2mZBE63jkDo6C5dzpVZruGi6OZIBgTTltYIc1S02sBooNMrXTgKRPxBxcduS4uoq+d3m6aI2oqxDAvS3u7hNB3sogULwqAutxRbIioKEpPOQPsCFMUHlA509mPWQmwIdfRC1La82sHLS8qFaq1sc6n4MMAqbWZ6cn253Wv/zMFzcr1X/3jS/99S++NWBOapEA1wHzGTNsFKCB9ZrZeJamjiVPVVdyOaWYAHXgQ0n2QI9/9Mqyt3kSelBX1dcQhu4GC0SJTFpXM9QbOA0gkkKUSMk/SgBRQSezEDukRfGQdZw9zJHs4EXDsFrd3t7CCH/w5Pike/T40fWPfPxDO7ttUEQaMJfV89wNWnPuuXhQkcvKLHu+ePW3HgDdWIfBHrwYiaLQr6JBhOEEYyMHQ9HzAocBVITcAIZVB+uRwz5q63DHtfQZXoW/7TadB75g62KKQaXuscu9Aj/e+yLteoILj+uvCF8YUSR2MdzdhxcSXPicRxMhFe2AakKxsVVXDUn0YonL5qDs0iEakGSsIpgscDZDE/cNn224aqyoNtz0uXKw2qoUw3KYtTLRGHB/8tqBykpERZtmtg5vVwik6BPKEd4xzNaIOGtFCW4gObYvf2ue/c717Y+0/etBP2Iqd3yYDw+np1229A2iRhzXSkEZVsJUIZPF0ECYQzavVTBhxxzJztGls+npYHzYHRye1pEf2H8OAye8HZfmYXfRx3GW2ctow681qTTORfXqnZev36l9/vdfufZX3/r2/eOjd4dPhydsZ9jZbDRH6ZSJZ2yONJuzBNDdoSvDfDDM+hhpZfiUewiwRc6kpRg22YfYmQnOelz9JdAKtpdRhUijONzsE9fLlk5vDMKWQHkIsPSvrJ2uS8SAkOdgEgCCKKsDX/HrHOUIwF0YeWK9rOSyXTgktejiRIVizw0aAK8gG/KRlW6B47P5P3O8EL0kPkLtsVyoaJE9GK360JiqNU6mQGx95jBjuGEKMVIcH2unbR6xG0qbZU/EJWujNTbkzSmstMBiIfSCE/uLXuANfG+oOXXYuGs63JdaYmdeHlnIfqOgFjVhDtgwXEZ12DmmCs7DyDe2atPbO0gX99PD7uEYWQyCjrOZSYdyhlBRgFTwNeCvGANvqBfiCdVl+Cmd1RNIgACWgUYlqj3cBaRDcDRuLZJvWjAAozu0tYVafKnXHR6QoCh3PUBbVdwzLodIvDwLLOuuD1zOTg92TM4lo/vIlgKpqfZjBAa0HDoAKtG9Qh8lmc9Ho8GR5DPO351Psf8zFRPUIqZlUEfbra16ECOXZk+Owiz/4u1XbrQ6d76//6ff/qt38ketaCMsNQ8nT+IAixUrfJS/m9KXxV+SvauXgzVwAWHEYEEhY730o7OM0Q63MBf4k0Q1J2DMT2KgWiEKRrM19atmCztJxn/7swfLF0jSKYxKPGAQEfDxYBCko/EwqFSgCSwnxEJzMuif9t4+GeYvf+yV69evnZmgVZTrexXyvi9hiFr/vj/4ZQmfldVVdQPKDBGOLENDqCVRp9ms1WrD0TQ8HRz0xiA8Y8ZgRqGrX1d+UYoCazj1y2pXvHewWm/4MvwBMytg7j5XH1hLXXOLobKq9XvBefWhauByW6/elTkUCazYZdWLSAIuTwKFKdvlQ/zyLczAvjPkFoV0A9OprXoCnUFqsgJ/RYjwghFF5RI/oYUysHHZEJZhGUTXM6ltFDEeZK4kJWK/BjJ0k+50AJKmQu4SA9ST6m6NcWRUbbUswk1JSqC/fBbO8iSd1Txvs1zd29h4Lkle6NTaJbTaaTIbVlkoBE/LBkxB9ftPJyGbLLAIQeSczdgTKDCbT9V2vQXuOAnHq81QeY6O0t44mJhKNF9gSaMegeyUw9IIXsoixnIw78dstsGE7bC60Wz87rX6C1svfeha48lo/Oq797/2wx/97OBRUtlsVxsPjw7QuhFJRLtEr3NMbce9g9HkNParkh2w69Mn0A688dR2mygWLJ95ATADjWAqsqGOc/clLqlrHDAtDwGYfHHzMfc9fa4RQhK6WSxTZMdgTayyVn9gVBCrW3Jfbb6h44ilB6MFwHcBPQHrA9n44Kwl9mFgVzDOt4NYhaj4rADW7hki1+hJmK7oOPlam78dTRVApO1IpJOGTLVUJ/BMMpyrHykkfkkXEhDRYFBV5FDMaTY+ns/M5oJTVIBT3/O+p4NqQV/8f7pMAGPVEJ9Ae9YcM1PTKh9lWvudwf7Etpl7lXIjfAcrsTTOU/gs+2hiUy1X/K3txmQ07XUHvR5zvngiaQ4GGqxM1J3kB3vXcDFgih+rBWoKGZqgw6uV+qsBpbqSCMZAwYIIKCVzNDMSwF15isEZRxBD0UCz0aEMz19u/Lo4kgkyKorcrfTzid2TqyQJioDi15OvhQkWyYqAIldsjt6nMVAAToVkmTS9TDfhSzHFx1nS5Rg7RBnXYuFYNYrK3mwY4nIRRvgNTg5OynP/lc3rye/9Ad7l//F7f/Pm5FGQ+R1vS0KemoIISAC+S+MY/RKUqCykRHWweMBpAXU/gAVHwCJLZkBWD/HGgU8ClnBJnBhCIvnG2LAZjMmVhPpPZ1qfkFbiEHVAYJPQBRIxqQJvYisvVGHOOsSqNuh1y+12s9Oeet7b7xyejn528874jAGrxLXLAVGVPw9zF7NM+OzOW8vptxuUNUnE2U/iqN2sN5vNMBz1x9qs1/7odgJiw3QGF+0yQrReK7XxV7vIy2Hwe+Dxe+dc5KBqGbRJT5gn/Wds2bXKf/V8KVP7ZHnjJelXnyyTXnhcT6OC3LX2FUEX7z504bOUlh5qqWGvsG4yHeoCKBI2LcQNRFfLKAPuy4oEIb90H3FTJZJbCQPVBFOSiAMzNIg326/P6g5onhLjJsGAQe8gOQqD8J1nkSg7JQeSJAcvq4PIGBnAdJGLc+ZuWzjSzrLqLO9Epev15p2d3ed2tq83Fjeb3XI8CIKsXObcNM8bnA5H6fCkxxoSNjaSUVn9AA3k6CPIeBl/XrZ2zSqYXRYMq8lpF6/OEvOziL00GqOYP2UzB2QAVRD3I5y2kpbHFkpUb5hlJ948Kdeq1S++sj8Mki/+g0+/cOfW/+u//dfv9LtJwAqXGKWatQ0qUWfXwiX6T44fH5w+6TRviJsATNFwCL7YpAAplrW8XE9ZH1JhwcHFrN4bXulBFMe60sF/mUztXH0CQxW9Ur/xlXiCgcHea2TR2dh8oVY2ykTu4MFSebGzcU+xOfOH+dkM0aKSJKbSMgKLrGkjL3TpuMQf87ZyjhOgqRb9LAZMSpRZVUEjTH2sGggteKBQEXaEAVoJJolN01wIpggon4tRk5or0K74Ib7MgY8/LidZkI6a46UswyXUg/ngxWLoLzj0gs25KADNFX8SPhadxdA95+RgTcazSTfb8cuGRtVojNynxTnkqq29R1k/FS9q9ajdqeFW3ztKaT3TzQIVnEH8kpTUTPWilW4okRHUnTQaH6A1f4gO+kgByqABfMoIEf5oYYyvnTHln8dXNrJgyQgOJLaxADz08drlckNGIk5Z6scwQMC0bPRzxQXYXfoicCGRI0ZCGo19bmqdHlYfKgwaKRfDGYEBv2IYEGBjjOiAJKCXsg0N52Jl3XhwWk1njdb1uMyWjkMmEsscO8WhB6Vk3B/i4bbZKP+T3/mHrebmf/jWV18/voemfOz1zMsOIY+i5ZoIeom7Ay1XvAHJPfAKwNGtNo4ME5WGgWS4vsQuUyUsrGorHz5C7CGNsFNSkL21O9B2ThJAEqwjZ9z3mGbQ0Q0IPZIAIEvw49l0PGL/aAxh5ajewGP68PHpcPi2WPqV17KrDKAG4VWqcw8WqTqpb1cp/lZ/ZSUydGb4MZeGHI30IeqEesBgWfW86gTJpI7CByGiRrWB8L1w8G+rKUV9VmAHU+h14cr6pZGw/rwW5sPiWosmh/NZrN4R765VxGpwFs8WIE0RQf6E3Z14AtzdkINCQ/KZp3I8GERf1VSYb5WW5qMwq/PFs9kmmUUAoCdHvdJLHPKjmRalYCw7lk0Bc9AZSsusC4qzOhCCBh/inN2Z5uf0nghGjLQEq516k2ju+Tyay8cymU+rs7SWpztx/Fyn+aFruy/vX7/eabfKbPs/KvlDDjtL4AblxPPK/uETNsWACTfiymzCUbXauhmKC5lgFCXlchxOkmlSStlFT54ks/5gNhgsRuNatU7b8gx1B+RjTOvgd0zZ7BeZHh5Pja2Wq7Je51nE4S55KcUMtfPSR//RJz/+4x++dvrN7/rTvO7HPc5fw07ODjw038f/atAdsML/wGttE6eZaCBmDAZoGW2lB6Q8GeiuuKkH3d/aS9IXPeii13PgFT0h9gqJFz1X5i4fvdIFu4XlwOokTmHbW3JNVjrJ+Zntrtj5mQVdAJB5P2nAoAZ3uhYhTOs4pP6yr5jMH7qDPsIg9SQIYHkukcGFyV+qt15ToqsRTFjYaHPG+pGGJQqp6hlDFumUIIKVsoR0lXDYnVFPugiWSQfJUoHfDwtQfZ+VSNQWThZLSoCNQUDFLWictGqOoZLyORcfBD8xRmgtEkubmMNk91AELVbYyYV/Vm9WNreD6bgLA+Y8AfCanfylRUteAgp8ro4DTR1kXbs02mmWSJPo/TJs7lcqjzaZtqVRRyEGG6KFbOon5EPqLLaqHdcc5CyT4qYMuGzYig0CboO16YJFqosBviLlxdhnP1+ZnpLIhIsOomsQW5DHgFiS4Hslas1hWPgz53k/nIw06cOK4VqnXmmX47rvjb0pFv9ywDmFaRRNguut1h9/6nM77e1//5Uvf+Xht8ps+Cr9CuRgTVGIoUWkAEiIfopdUnpJtjQDt9XctcegJLGBz+lgUIVRK+JD59BRpIS4mU1BZ0uDwCI/QJypAGG8vbeRxaywdF/FgM3QEvzwGB9gGFhCByE1R/N8eHAQ1Op4R/fuP/BYgbSx1dzcSnAzW3aMijOxz/UTmRlyEw3glPeq//TiyosEMI0l21BnG8y1AE84ZY/cl8XRr5byck5XIY9SFfVc/0RzJEzL4SYbBOyV8LTbZQcw1N8+TuDyvpGQiz2eT6gMIFQ+7t+qmVZJtWgVoTTr14Vyl9C4VJ+z+FV7XauX9xU0XM4uT4NWAEnnkZFvF4NMF8nYVo97ka37UGTG4qzaDkn4+uxa/8Slcbm5z7mT1IVtzYf17OqdcgHVJXErjbu7TLiTj4tZJV/+LmkBmIvBFDqguksKpf6rhqBASKGi5szTJ4u4Au4ztTYZBfNx7MNOJv5sxCr1Vj3utGvM3qJLcUYQcuSTJ6f5rFmKWsf9fqPezkvhyXjaqLVPxqwkwagYpTi8a7DEIuiMQUa3ZFlo8qyc8Ze2F+lW4HVKpQ9du36zUX5us73frHbKs8bsKOEE7mg6ZTP4bNKsVb12xTsePjrp9sYphIGDjMRJdewgO2Lk2rFmHmTjUT1GJ8LFVUt1WfGXjyZsTYc1FW8jZp1j7YQklQzjazrop7hGR2OEBogi9Rxis2QEl2tetT47Wey0rs2Oj/xF+R98+KNvvv7gh91329FedzFuRs1e73Qwn1yrbixG2dHoyf0Hb522dupxO4kac47fYdlOwtayOLBwUh4LYAEvFmBHFtQvhGGJl7uMHhTNsJvrP3pW6qN9suxr01i4SRuEfGhwkLNQwiiUDPJkIm6ioaSyjemmTPLO2LEI+zN6oTNH212cAxXOYQG7jmB8ZltHzouSpxTmQ1ahYLoHhEbZKFZ8ysMbC/ZH7nwNqQAtiQSKbMHLyGCtNYdRBTFKFGyRMc5XDAyboRatBQGlx8gDEaxAXoGEQuxxl2JxtVaJ0yJMymzpywQwG6HgKY8eMxXpJSe5XGnaAyDKkU+GaLE6OZxlKF2ovmwsgeImZ6vZdCLwkQve6sIF2sL5P8nW9gZLuBHPsjH4DIEH6QVojWDxBXWDHoG/8SemypWNYow9a8jxAjCLbfExls0YHOQ7GbQlBei11vVx0y4mAIghhtmB/bj4GpRQ5hrOWrRd5tBoy5/7r3NRbYdnytzqv+RsQhDkNfE1RxNcKQZqgiRHclB90CV5xhANnuArQC/RYhYIwpvRjYcH9+N4UOL8hvY4bGyGSRMjMDbcPK+WwjrbxE2P5p3NrS+++MlNv379x9v/86tf/cX8MTBhYA3YoR3w43KcDdgvFj9cSkT8wwoMymAGoXRGoLFb1c4aIvKhrsGgInALo0WfhPDcwCLVVrYG0FQgh43I2oFBhbvhhVBDmclsE47Z3EueYRBD6zi+xRA0YoUbqyhYFo+QgEiO+3PeG4+TWv2ZGrBK/UAXlaCCqop+9V8wt9pflY9LedWbDxbHWgH5XLLBDgfTpFiTFqyUH+C8r2M3JEQYABlZ1AlMlVzhCigq4Gr6wUp9j9QqcdVqCzggGDB0K8p1MY5KEmnx+nA9zeVyLNlZtH3nvtWH7kUR4FFb6K1dFz5fe/OrBMFUxjnjDQqDSCXzKOY+SYkihdIN8LxZaB1QzGqRKKiW5tOje4GfJrFXj4NWzW/VggbOr6Vgd6PaaUWbrRodOBr2cDwulcqjdBP3psmi/osHD/uT9GTQG8+mzIV65dIghXPHnFzPHjoQbDuRQHJoyB4OaRZnk5bv3Sgnz7c3Xtxo7FdKexQXzVphHk8PfFRbhmM1DmrBPEz5JkTLRYxFbJUDEaoaTrt496K84QvFoAFsEDuZxtFs4XrzCZ7K2kQyn2Y2tQFBDBB5aTEVZ4hD9jgQmC1E0sUQAsySlBD1CHhB6VGtsF2mM3yPkrC119n+7Ic+3P+D3P+zv/7B8I07lesn3S776DY5/UgkYl72/V73oHfydLPhsw4SUgXpQgZA9JSYzxQjZtaY5VdiwyYG6DRiWUp/jUt4YthkCMNtOa7lkSRJjP/SekWutK2VhqDmSiU8ueW/8sNCepFmCKRMART/BlfEL3Pp8MCIOW30B213IU4iO59UF41S46AqCd3OiCHDV6xJpAWzljPHwxyVVjEAVxOk3MmCH2VEAxSrfPUO9Rd2DX5Cl2kFPYa5goDWKKPvOgohEqnlQhAM83GGqDK9R8EsoGI2g2gmcqmREI75YPCEhWOa5qO5ObtnwulhnBTDrpTiLAKi0WYRIp6tUm7SxXgyDRJnWs5e8iXNIQ+Iumm6AVKdSkUiV6X5nj8jZq531EKZiCjI9RBZCe8cdwF6BhzVgW8BHel/JaywagISYeT65eKLmOKxCBSvbAy50q2mahINAb5SxqQrAWcGzOBpEKXsJjbSPlRp2Uvj2gakAwlkvhh5WeSNOQGS/SxbH9vav/GP/8XWC9f/3Xe+/JN7P2FbjMhrDj2W+qTluBYib4n8C99EczlFUMvQ+F4QthtNsSBPDG6Z7sE3gV6XEN64L3ikXgR+Vm276wt4sD4gRAuEzUI5fQmOKS0ZID2pADKnf8ArmoiyyFwA20czPNFYUja7tosauoB9vAovo97fDzmQi8vHwnz2XhkVJV7Intq/74uu4wgVmYugAhh8MrwqIAaIKMQDfkadEFbYDXAECBsCq6KtvgYqlfgByr1cQaC/jKQ4u4oYF+DuruJb9wjRdECzyugl8UWay4EisUtWPEq4s8vFF5msJNCLORUJLrwoMnTxPJLSRa6nXJbieTV2IZfFFWVHI8mIimbvWBsbzbOkxA6LDAHxoCDNKsHwxe3e9Y3o9rWdve1Wk32ovGmlNKvJq2SYhFlcOkUdyROxKNaGzsJy8LGX07D96Oj646Pum/ce/uztByejw35v0qpv9acDHJv8sIrZUIKtBz+Nalm2EcS7G62b7cbz7ebtRmU/CVpBGgyPonSQD/FaPs7YTx/uFmzis8eAhFD6OFuBrGwnmQ1Rx33WqKRpCY0OtRZqC3nA2QZ2AU2WzUjaCQtc5+wrMGeVKFvhlKaQCygjZBwHapRg0RShnCAjORr2KEOxzdlic41ZJRxo061sPhhvVHb/+LOfD+e14M/Ct8b3WDy5HbbhTKejw6YXsCj54OmDh423OtuLrW3EjzqIPINGoeeHITIngMLlQ7xphdgm6as412UKGHVQYJWm6F9V01La5xdxz/BHVJe3RiVpB0QToNgfqq2mQrXcmobmrFZGFuboBcKoh5oStnXDiDOAAQszyIB6ieDEWiMoGHJJpLW7tv2kOoI/OtJMrhQIxxEPNnJG9wii4gFwX6XR4IaEEhD1k3cYtFbUEkXEdBRiwERjvSqSgGPDQEOigJpK5eEp1kjiHKvUSJKeJ4mG1moGFv5NH9K59gd9UafCgOXrnbLyk+290WpmKRPGthCdvmHTK7v4jCaoBEQL1D5VWTSbiwpQcx7VQGnxy8u6SR0FxMEgjUBqq0sYCM7pcwQWdYM+gtapwQKPWg88yFnDUWaMZUk8OmqMqOOKWf6sCn3Wr+pZoIcLAD9rgCwVlGgJyF9h9ZHQXXW1ApdUiZoqsVg42MNbhFzubmaVyvIx3xGbTViZBEnEDDUaZYNaPqxx8gGr+8I6yaohZ+2GYJZ/ikksbLfbf/zRz+ztbn7pqxtf/fHXj7xu06uNFmzFPMa1L2Wm3+iCesJOw9LUsCQsq+nqrnrzn4rpR09qi4MOXcYHLB/TiOc7BDVM0ORAfyC3gcF0DzyUwS/pQbkI5PwK8KZKy01YEcqXVeh8DaUsMYCZo/EnyZkJWp+5y8C6erj0+6y3xANRe0sfXPrstxQha6eMGIvESaRgBnsMsR0MS7wNHDZGaT1oKXnHiKDVZb2ShE00LSrpwO8e18NFgrOAa/IyqcTRFcqdJVGoKI4AV/GV9K2zawlAfkjjPjt7ac9qir1aJjj/2mVbZM7L9WTr8ee/u+LJfXj5k/UYwgM2o1mORtf/GA7phLwsjjYOc2hSij8C2zs1G/F2Lf3U7fLdnfjOzeZGozIb91nzE+STagJtHg663e7pCU45nU6nzDFeGfu8hf2jN4PW3o1O5drGxu1rtVfu7j466j85GZ4M0ieno+PTyQRDMNu8c6B7GHXC2u/cuH2r0ry9s3mz09iOSpV0GI2OS+Pe8OR+PsMgeDpJT6ccgTCPavVZI9/iNN6yHLArIgwIcFKsMxYBhewJmWVotZKQ6A7co5nJ85nvKGmRItIrayfgOJBi2g+lhlLK2Ytl6VBZEUShIjiHqCwmMmMREzQJdIX9goZsHcvZqLCHSS+dDnu7O3f/5PP/sNnY+G/+h//PvfEj9mIMg6jpJZzOMJ0M2NHvyaN3d2fRLsuGawj1dbQ4ThYSLzRkQIyQGmEXMXAOzbsQaaNyvXcVY/RzPZJPuHilu9FQviSBPSohAWiK9F61gEJVLmGZ3ZYxikf6kCO0HK80KyoDglRFAw5sjC2tseSjmWhDRTENCcaEbTLYNGOKx0zFICVzFaHhzV21EglzZIW7cV94rY1pQCxOo4llI/vqJlLgPIVMpJJwF/BwoUI1sc2flQ0pyY+y1AxrHD0J+yRromixmXjxq5XCSzHCAhFmKDo0FNoinV/bikjAoJE6uJbVNOjBAX86c06E12BoBFtUnyJVpvgTraI9buAQoEBVRK0R5RbJdvDXXqTUXv4RiB0khANQC2ygYs2Y4ZXSOgjcsi1JKHgGBgrF0MuWvIQ8gSZtJikBkqqv9Z2yvfKy6igZbwkXAb5Sba3Uc/GWpviKgIFReVMX1xyXFWE1X8KFvdAT/xFRJPDIudEfY0lKx9NRPhzPx5NFWk0Hrc1byBYMIeYuoPDa3+XksD84qu81v/j8R7dKSXXuf+/nP3mQHky8ScIKf5BPwGQ2REonbab5JY40meocBcMb/awBwQUNYnpjOC8pgWkMXTJHmVIrYwkv2XcRf1KBH3RFeDJc1OjBJkt7+V6jRe0yWAFA+lBiq9quTuQ0JdjwOfukFfSr3hj/SHiSGKAxUBoBneusH85nLEH3qutZ6a9Ka3HyQoe4ASVs9NZ4UB9w2Bg1yw5hIbhibOTxRCnUjccPXNwz6uFyU4MN3JdzLgoi4C6XE2EXcDkQdoEi3r29fHcJiq9cAiKJcXcXg/DnAi5lkf484z+XPZ9zFRkSKL4q0i1jKAtUAgGZTmUPfUAcMN0CS5rEHlsB98LZIAkmrSS4tl1/4XbrzrWNl69ntbBXWvwi707y0cibjOfprHeSt+uNeDFdwE/xEtRmOG36tVprPH74eIoLUilit1/Zne7sfOyFXQ4AG4xnB93R06P+YJgORtloiIXKb/vJF67f3S4lrWrMASX+pLvIumGpXyoP/YYz+4T1vMoxadBnNrdapIPEq+IOxJw0Jip8mKH4DCYcYUuskQeYUnI0bpgHZJaQ+4QNN2T8JRaM0hvoBnZ2JplSeDNohrI8w8eXzXCRC+fY3o26aPSJGaAYce7DPBr1swq8Z7NWDls5W2yNs43axic/9NH/63/9f/43//FPv3Tvr7DT725sjY8Pxt5pJYEEdU9LT47DRrLBvBibCsOb4VGQUxy/UMTm6MH0NbZoJk5TVARDb9dfCtufkNN6lpgLXVz0LPHurcMAEjJ+RDPEgHTRHEgMfISmGj23BUiKEK+FtsCU4AL8iUeIPPGBJFNE05W+i/uVZu2hpYxdqIWIB6oZrJexK62XPzLX3diTFaxchGykYe5Xc8h8Ik4KS2Eso91ipaArEXgi6zDWDsHwQ4wj2CjEnpdOWLSFevKJiD13FcecBDWV5XlJl2S1pAqUKYCY3k9vq1LiG6Yco+ermvBDvIq08hlzSco0MXt7z6fjeDLGkYAmqPIGbcohM/dod5XNKwcgkSMRdNViGRDRtwJJaFUT2vEJAgWbZ7JPFoTOOgXfMGoq2NFI6CB5qWD6GwmEIlRlfsQlBGWDpIr7ZZdDBlIVAfcFuVFPiicesBHJf8LWOJXIBajUpUus0VslU2K9VxUFDAHfhdRqABziLsCaYJlM5uwU3sORPp2MB+xPGZU7+F9ivoqiRrlCOJhMpqXTEbncrXT+d3/0zz9656X/4S//w3D4s7bX6GlfM3aCx/8OAYm5orH2A8VgRhkCg6pBeWvyBwBeVpJogYskSmQ/MHBrJq+KeHF1QZV8TNRzjaLR/FljDELkqby0OTazWgII/0Vk1OvI7GtwUXlcBiO+eca1rMcVb92HKpUyn53sii9/9Sj0Cw42YfjSi0zRYCRDKIVs0mgHYrMMqDEayGCD2QUMaa1QqzM3XWu1eFb4LMk63C59fpaMkEvJnWTr4fVErvT1u+GlPl5Ppi5mmF2MVJddiHTF8S2kmTsJuFyguBO4fF3IxyUoIotMXIC35UrFBj4AZ0+4NMCushhFi2GY9xtRutf2bmw1bm7Xb2zW97fr2+1gd8ufTYL+STdPR81qlLQa3cPuo/sH8JFyUmceWKsttWEy+kpcbjQ6nd7J6fFoOMUzOWtvtOtBpTqNKo0XdrfH0/pwtAGVE2Ij4Hql2iysHqdtuFGIswynHvRm0SmOlOFiCunHfBx6eC+X81kN3xk4FtOrYqXMyrBNB2IDRlNZM+18bUzLc51vg++VVC2GDGtDMY9p2KElM+wYpLJIGjWH6nGckrb6xZodM7NMP8GuYSjsWys+SbPkvCwbK74hc/ZpYwMB9Fvt0F8Jk5QNB7J+pRT97kc/HaP9/9vF37z914+O77eRQrwEOycsh835uodPqj6nI5Wx8iCU0K90DQObO7IvC3qw5zIhTI/MOezNrgsj0TpOlJO+4yuuAtcJa5icv6AuFu8SQ1dUFp9zMZaEXu5CE0TlZedBAuiLUtIcE1V21Id6MVHA6iNqje4LQ4R9ap6UdmjhD4PYFm0g0Ug1sq+MboG2/EHVaKv4iWxMQBczv5aySY7HrK2AFF920gbksgpKCYYfa3oAxRcFBUohnRDvZfpbfbZsvkkICDMmZFGySCfvgSriE10s/iC9QmKGPpOWzBJW6Ax6rrgvNkVDAc0wzqc4oPTT09NsPAj7fRbYwJk1avkToxXRXZVrbeQllwiX0QfCClgvEFAdLQoYM7qM0gsKwII5RX40K4weBbz4I7WApErpyXJRS+wN7VcY5q20Z8Wp+Ksukroxvh6wrNQTfKH4VT2VgdWUX75SMS4gs65GFBHL3FZMjhQSKagzr41EaajhXOHJIQ600Ep3/CgybzpguUH6ZObv7Nyut9AYIfIjFvPh3F6JPTZx5RyqSrX8/NbO9ie2o7DW+vZXX3/y7nD2iGHLJmTsSsoBIYt8oklXrcnGWqRryUjdQFD1aZTqwa+rv8LESrYjIJCRwLFc9QvtAaDqDcQhfUiAJ2PYdJQYkjpdEXpkEkZPYBdZSFUVLPl/NgfsAOTuyu6DXlTWamW/BeYI8FfmRLIr4z9wZFjRJ5B+NRTqyDIsfiSSqPH8SchhUBuyEBQsl4Wsmkyy38C1BJ2avmyaC7j7qiyVTXjtOiudyF9aD0uzRHGX2GXFEHWPFLeej/PKWS+0CLv0l+8Xqn0hwyI9pQDaMUfbyjDGnmrYmTP+qqWsGmTPbbU70fj2ZvSR2527u7VmhBh6NO0NHvbmSaUMt4XX5R5++DWtO/CPBv3RbmMjD5IpW7Y3qtM4YUOV3uDktP80HfcqUZSFaS3M6tEI3ZQVvcPHR2AvxqpgNI1LSafZqVbrcZbOTvucsoS9GBLl11I/4rDd4WTcQzXP5yPUwxoLBcxtOsnZLrxkhG0eMmOx8EZTGAgLE8uLxRQ6pTVWUoBhJmi/rHBg7RNthnyLV0N3ca4FtWgHaZi5lhjILGcQs8Gz9kZl8DJ1xZAzR22mGpj1nOPJmsdMcJfLTa/SZkR7GM9hQexbMZtGtcZkOP3oSx+O62Hp3+R/9dqfwcxaycZkOiiVcAEd9/Ju2X8y85IqGM5fuY7XmNyCsYm644mYUEXpK5dx3i566pcG6GLXy7obdXbDlkdxWWG0NH1dirAmy71O7FYrjlhwI59gWC+9yntAZGoxYGEGVISKjepkE+CGyov3k/jm8vBB2PByDphI2VL4E6NckmwqD/+G4wirGR+sNWTKnXW9mmrX7JOUPt4oO6beYe/ixJRBMj4TfSTdci9UeCpjB+9qjSCGqRs08pWBu6m7mXxIDQLi0ibYkBNtlpTDo+Zz5c8AtGGJeEGDHHjca0qYwUCC6Xg66E26x96gnw1HnlakiATqsuLc2NQwV/H2iteAWG3TCNYrPXIpkakWIvAiK/LPRZMSNOQgrlSOxgEdgVpdBccjwLfKXbqYfhiny+LMxL0kMyqAztb9yotPlrVYBVyyIl4FcVkOyt+qvSxI7VULVRPMGOoCvSeOGBOSACqvNJ4k4JKPmBJSPPtv6yilCHmZ47GQrRkDXtafzPCZ48jQKiv6WY6EPwenCbM4P2XbjprcIp92k2bjix/7bGtj68+//83S6z94OHx05B0wQ4AinHiR5stxLUFoFF/gErhXzECVN2YpXNU7g5hLJ95h3NbS8FaAozf0RRFFA2kmviKaiBHroUNxc2DuwzoBcmDJ+Qoaoa6mycgAiIfL2pCTdYjgo0s/IL0VJ6dO9ewVupeltZvgvbyo3Qq1qcYyw7OULmTtvBjJ87PSX5GUJsFfEUHxAkH6hgegZqjaDjL2hdBU/axBKjRWt/O8Kl0IZG1dRbhi1DcGGRdwkVfflcOqzgRMQlbHuSKWBSEGytVnCQrkHxLoM8Wop9QQXUVxPBroTTi0V9ZYfcV/SbnQBX1FL2PupGegDpaPwhYvgkLnmwnalWXFLcuFii6zvfjDBBdFY76zSuqGPRMC4yqgmRTgK3cUdsH10pANn/wUtlItexv1eLtV3a432uX2C9dai9GTejC50Vrs1vGCnvX6p8Pe8clwsbt/M46qHD8wHLCFEBs+drZvLh49frqQP3OOilTfavlR+WT8aNA98FkX6y9qVdTWNEbi12n2YwzD7VpD6y/T7vDoKSpQw98Lgg2GQL2ZjAfj/qDPkhhtaBHm83iKh4zm6FKmWzN/gtUn8KfMlPE6Cxqt2QzLMrtnMCPD7gz0heYtUVZFXTjPUMBm1g+Sp2azxkNKD2QDDguEScVJeGw6OAUggHRWMXZkDrKk0HAH+wCophIpl7t4TDjGJa1a9ersJ81ayBgvbFgmK7RYsn/w6PiFvef+j//qfz/7f49f/8X3WJth88glFq6OsmGv1/VKWH18bOws2WJ6OGCzrTihGKx16Awx55KG7ITX14YSWl3j/HJM0QBHjHBQI2pjCKS7cFADwHBKQrqjnsIMms8noBQUR+qvTXvizSkjs6Z7jRGx8Nccr3A1heMKtYAPrzBFGUzIRugvNRdwASjRIHmqobIyfQQDFlT0x9gVrbaRAr1ReoXVYzRAr+G3qBryf5Heq8Fj8Xh2id7BgLWCWpO+agYUW40AYyWVyYmMrDBssPEhtdQ8rVoo2qDE3BEvpPkq1jxWUeWpERcuM+pKmiUlmJy0Elq6Jx7u7LhALP9w+8bWOfF6Q68/9FiyRPdYneUSoIoYGYaS8iSUIEJOSYZo9h4xUBq9AKC+QNPGSE/PCKRKa+2V8V71U3WpCENRtZdNE50SswupsLyL0JOM2quDaTaNVBnqScuf8OoqSAE1IuzueqmvjHTap0vyJEJKMUZ/DOdVmGUBngBGteyM6ZpBXb2kiR3uWhsrOFvZNmdBrzACGRW8pPaiXzSdHCUpCa/4LsOr+fDpW8Ph8NqtSWtr3wtrOG+UkzYIxAlUSHWccTI+OI7nnU9ff36vs9OOK99/5yc/ejI78Y4ZqhwUqVoFmJyA1hz8YARoWkjAEHyktZkIZBXTPvPLjuDXLusfwKCGrgEIYFM/cRx1maQ/wA5xAOWEXNb76ihFqBj+LyFMkynenMOsAGriyrAnEW7oCyWp4wQUlUtXajCup3Op3d1wehkhbF5drpF2X0VRsWXDDB9UTPFqGXA5rOdzMQXPfMXSTwYdY1L8h0lgi9VwNcQzxDCouXi1QA1dVU/FujSCIA/uLiRQlbA/nqunRZIK+V3jhEswUTXsEoVdtkX4JPgvPxfCri5ecWlgg7iiUKCj3pHeJWEYchGxHHiWHokcGZgzBtRQ2me9oVGn+jPaIET6TNihCkEOENe1ftWypcE26FcNtOrqRt7FXcDkvGgGXcoWjGylxnCZsYkL/qv4mEdxdUbxszlrYVABF/OjRnh6rXT0kbub2x1W8Ma3rrFTTRjMRo0yZtBxxtb0wyl229HI77KCLwtrmzfHixPKGwyGiU4B0nbqVRyuys2dvajVwvW3/ODxozd/9uorr3zkI3dvvPbTn7Jfzs7edVJUK3UIHBOeUYkNc3DdYvE+27J72zttoMThrFOvzyBnW8gFKjW2wPGIilfYmKU0Y00yE8WYcL3BMCyF2A5ZzJAjBYSb0+NpqaVaA+tSOvQxeOECreVDmYYMFmrJ6HRUCuUA+Y34yveJMlGOciTrFG7EyOOoV+3gzIY9HFPIWYG1ciWJQ1YVA146QrySvkIv054ec7YZ4FRvJqggVeznkJQbGI7Hx92k0uz41dnp4qWdF/9v/9X//V//D//tj974RtNjn/omjUX0OJ2c+Cf4haYJRwj0WrXN52zHpQyDPeeBM845JA/Nu7JxLZ8Mpv3TjAlhrQrIdbRTGA1lmpYrpxRHIbgwgBrK3xcME+JArXTsGlWmlfBIEUCRe1UVRzW2uGUBlggBHi5UXps/AwKdNAwbRo8X3+USDcVlDf1+ziIpn0N/sTkzdauNE+CP1FIls3CaBKK+RMrlVx5NaJJYJ0Bc9B8GteiZbAuYFbTRBG56OlgXNsT3MCjWZcUJNm7MqyhO7NgQhTipJeRNFnhEqTWi7zB0licyS8KOV9R3VKKLRbJolvzFoG+UGAZ4UrF6jvOMmZBQfVhNxy/7esnUzIbiiBCczaw9rfQpVaTTwQ22HsRzgcmBWdBlvnGAlKZhSFvlgI1Sr0eAjPu3RHGaSjmMaPpCU+GpHZeJcoc8IGVHrIxGhhNz8iMHowHqMjAQ40McJMa3jEuxalnzG/zxSYxJCosnlB2xmBJxFsJHHmKv8S+o0+li0iI3CtJwUXTyV5cLF8AJBfjhkvzFl4rXxbgjTmn0/fJOyxBnxUDUZL1VMbQNjZPBgxSlrlKmXOapY+RIhYIElq2QQZ+yzJ5ngqREhgO6OuyD1VjzU+YthpPZg/uj0fDJxtZ+u71Dzv1ZFLY2WNbIGu0auHN4WBnPPtSqv/jP/qt//80vV78WvHry9ok36nmTEWOew9CophDKq+MDKaShW9DBFwN6mnpLAqAimKRkuxYGau8fXaKfahqX9bqBR8+gs2gubQfGYBvT8TzCzmkpnYgwb9+lI9mb1V46io5AflVLlyZoAMvD6nJhicE2TIkGgsY/Vt2wSvl3/7vEE3U91xKrLMiNtjssUoQDnqSVqy76m2ghiAGOu33xG2uwKNL5SzEiY+cqSbFCSi4l1yurkWJ4RXL1iBLptWiW0EJtNB8WSR3LtzAHJCjeK+HZZe1SMy/XxyWCGhlWkB8yDZeYD4KqqxQfilTK+joux2i3+e/dan9oL25W83I42an1yiw5ynsh++5Phuh0AT6HpQo6wWl/iIszn1/b3oFj9Xo9Vv/CVrEL0yZIV5iUUU9r4eLFu3f6/eGje29FUbKtZcL1RqWMMwvsFkJSLVeoeT5MT3sn5JDEEGQBBHKGdwbUHYZdKkPQONvMZ6TiqlxFO69XxidH8MfReDAYj2te0KpWS2X2ssCKOPWn5oYLVx0POSHex5arIcqfwCsGsJRyoEXECnTSgngFVaVPYFGIJfJzFkFmgpdzjUbQEazDiziFG4mIiQbp7ARoo4d0A4OfIn6UIqmDEkPYEUJbRTAoZzW/1Oc4nXH2/LU7/+U//VeNefS9N798OB5WS42YvTCnk9PJYXAwKS04IWY6jxtMwrABiCgQ6zOYLKbrYJusSi5FlSr+WsFseMpmJ/hm0ZPACiwS66DKqpr0pCUVFLIZSi1HE3hFW40RG0AAizDQ3Y1wm2kATdd4pxbIkJsYGnBa3c0XFcWMC6pmvFa6kLRbOfcSoEqi0UJpq55QXRfFg/cqXzHKIKAxSEWqI92rj9HydQIup1xqohf7M3/ArxGw3YF2OkQfn6gPSegxi+pkKLyrmN0H5vypO2mSJAlaTa5Eau6ftjBfqPbK2EMSaXLATfOuhhMCAwSbKX1xOe18BFdHGKHbiZYspAPaUyZMlI3YJoIJQ5UuQBzAj4riDM6USyvhyhjJxN5tbHOn0dKP1FWSiYSMdJdidQf96GZawjdAg5wlLgJEpjg0By6co9owFIyi6iEZQylPJEHlqVcBOgC35+VNrx19pBAC4tqry32ojBWjehZ3RVi1aKcLk4/lLOZNOv1RlvxkhW+WxsoBsHZZvSSZqK0qyaACwlGQPmax15C9LwBSlp0O+qRkWcI4TEd+8+Ykx9zPpspJ2avAC/PRIBuN05PkDz70id3d3f/pG3/55z/5BpPGLR960+iOH5RqjQZWl/GIZasMc+phS7xpASSTBoCkcnMjK0gfd8mfqqRuBhMHAWpOzSS36aW9cU2i8upjvQZ98MU3fFUaEhv3FTEFe8kQZ8RnXEVPFe+JURXO91jx9lcLrEqxbJe48avl9Ot+xYAnCxv4ulExdy/yLSKLmPcIkNi9dfkUYQLFKxepMW10x8Vzh9bwyiiD6uCSEXBZWYzrh2WFiwS8sreKKALQlOLR3i/TnEHefb+6QxitIMrH+MYox6HWDSrmZYSjcmXhONXFgJ2jnttvv3Sndq2BlA0vm7J/KzwgHXNa7lhbQnm45i5gDaiDuFwNeifMU+7v7HTlS/Sk0W6BnqC+6BU8slIbjaZsK7SzuwWXffL4/sPjB9dvXLt780671YABs9C+3xv2+6dw4oODgyRM9vb22OcXElOtVCeTCeapMvPP8yyuNcpVfLg4RWgCSypVkkpS3gm84xJndI5SdmdGqcnG+bRUjzgCr1xiAti3PbUmKasUkA/QwURJxF0ZJZpvQ+egovyTzoKNWhQZiiLmAMkQA5YaJFaNzF/ic5gq5gfWL0K52asE9VokCfs6jtaVCJcF2//Lmw0Q89NsGC5qYVCNKvFi1lugRjG5PhlHYeWF5++W/uSfbCWVRz99zYzvkxaTnXnenR7nTyedDkO9XmnItUt2SdFhtqWmxyEj+GNBlDgSUefzsWiRNVbyCNKsk2ybNECMQIRYfMahheiPUA6qKSwkO4kcxgPQEOSJZB7ORJrgKxjRicCGNMuLrwGTcSfdl4N6jmpmc7SQN2ib2AIeWDjFoaSJLHM3uxAVANPtT3RYnwt/BTqaw0b88qFBbWFqXfqGmr3QypSqXNLgxKVqiSPt4jpnYGijLDx5dBgStcVvztiPuBJ/QmnLnA6SNwItUlOgkTKz6EIwlLmZ1umJX9FPMQQ3qS33O+m+k8FsNFj0cXvOSmO8+KeIQUBvgT8BhUB/mamRxi4FhmaVUKRpkYqnUQwRtU8JATcarXFTXizbzA/yGWyS1koFEkmXUZ1hSCF86ngqb+G69BfZItYI0yhL86mEyZheVo9qRzY1UogCPBUr/FVFLKjCeAAKvFX0pevqWEu26miDqnJyWakhvLqcITHFJ0U5FiMZRF8tOT0BSQlQEIwetAlKwiY248l0NB7Hw/FmuY3vY8jB2iA826ThlwCWA3VkoFnyob0b/9t//M9u3Nz/q+9/48eP3zwY9zbDDjYcdnHHXZ2VCww5unY6GwIl4MZMkLtEgiVKAVLXV9YjinSPBIhxICdsyRXDWw0sZDJoAsOdV8IqSZoGWcJqHf/pczBe3prWXItWIl2KcVArYOeAxV2mT/eFJf3lt2f0pcvw3OfPSHkuzW/zAVC4i0IIuLYzCosyiXThKyq/SmRpzgHocuL1fJ6VHgwqkrkAd/WlXUXMqtjl7+WyLiRwj0XOl946MkFJUDeJ8pqGMWIttViOTYx5OPIomvc7leD2bnuzFSY+TG2STYfj2SJh7jgbTSZ9mAHTpbBHrZdj58TIYyPkmTxW0NEw/fBOiyunkxF2Jk4g2N29BkXodU/eyWcbGxu1WvXh/fs//uHB8PT0pRde7HQ2INqjcT/rZaw96PW7KCTlSjgcnaIG74a7aLIYP8s4gjHxKwXLD/EtztjTPcNlh9WFjb0Ntq3SclCYMMr4wQGjmNOymTrWHlXaUA0ehdpDRggRKQYAgMPghDzzD+IlDYg/2aphtFBj8SqNOXEgaSQUCmOG62jXGypEF0J3pR/JERo+DHllP+IETpiP8Brx095iEIxmYTYFHI1GdcMrd6jJBJN9JWGDit7xIbaBuzdv7/+LjW+X//zdH/3kKDus+Z1Goz7sp93sNBhqr2umn7EGV+ZbHmo9+h9bk6lyGAvjkKkxFr8mVVzIULOZB4dxwoOMKmPzRrgw/0zIhciHLscDQADhP23FagrPgdsYR4IdE08s7aLRhPWoSXN4sBbEAiHMuWY/wO0MziDmKjEJCufMjICFvtSjqXzYolF8xF+oshElSw1oZSOnUmInWGNgtnLxzpnUx3AsZdG2zEblLZXnIdsz1MVx2bkF7hvX8Le2yWRO3kLtSOGEmk4QKzOdUjBQvqq8LMtLwQKdXFPYYr9A0LRVtUo9TAfT625qUHwNbsna37Hf7XJ0VqBNKDlWabrAJ5/eAMp48avu+PRJzUY9ZhKZQYJ/Pi+QlwRqx5fIHdxQhJi8mPeS/1oEw8RSCpYgEWADoOZL5DoMkiRvb6ni3HB2E2tmiKpwEsujiaoLhMBYNjGKpt1SRmUVAMPJ06mpZ5xScLcaisOYTqykFqPaXHXpE+Wm+1qACJMFdBe4HeVxkedT6omKSnoU+dV9PT1DiT5go1NhmZZgc83CaZaGlVp7r9ralXMGshOuHRi84goC19GTx/mw8tLt/f0v/sl+a/Pf/dVffOfxD09nbNI9BSdYEMEGqCk9yX7gWKN1nLw4mzFVmiFjDnUBWstfhpSuJe2kbmKiSx7sXvGW6iOri3QDfIYKcqMyNVJhDlmWg3pAWXMH/ZXructAQIzeUL4lWMatXp1L/7/QhyWkr2qdyMmvdLk813MusrIeFcFzGaMeFMkuBxxqFgjqPim+LarGh8Yj1InF33l8Iq3DgCUeGN0jDvQT08Dz1/gJc4gQclgZ6DQN54NKqddJkp163qnGIfNXGHJ1xJs8gMJaFC7ip08epfhrsmEMR8OwUJW9XOOQLSRZ09dq1OGaFQ4UCpjUQZ3xZ/3JsN/DK+fx44enr52i2oZJPBkNHjx48PrPXv3FL96+des5dlaajMYCF9xiwSEI/cdP8NBig/YIpRsOhhOTnGyrtcOjLtrwlBX8rOX1Fl2cgsNSeRFwFOCNna05LtEHx5Wt9oSzRsdaqBnjyctWu3Mt7tewhgFDhQ3vIddLKIhGy4QEK4Bf84HGp/gSVBOOLFsfdFmEncGmzzD42qpipIF5hiFSfwwfDIIp/jlsMHCiw7g1w4e/J+tWJ3OUYybg69dKCCXzWSVuzsgQuYJzeObB7332C41F+PYPfjhEWh+zToOdn/AFHZayI6ZfcdtGjq8FO8z2QXhhGPiT4xgGpYo1MVyRZgih9oPJ9FCmSkiFtFASQvxhnmxYBu2AytMonSgjlZTOM85EK5VCjt9cYlBiWqLMttGVOWQBNosESuxMaeKK+BXUHXYB5xRnBWDCZBlUjJMg4WAmQUZhWs5Yr2x+YtLwCjBQNIr/fMYX4sH0vXiF9uUGjNL+cP/mBCI8oku1AFN8Kc6Z+g2ZYUi0VTSfY6pR1kYqqZr4rNRyBhkgEFOWDEHvUXl2sYW+I3IAEEupXhUzVnpxX4zPcC2sCUygs7aUqfDZeMTCdaZZtDcLu5sOJ9lgyPF6ZtZE8oERspmMNSFK8NXzmJAAU4CFRDhDMNdAKuIMr+oAoZiAwEgDxGSi/xJBRBlIKYsmzBhhwfyDlEDWA5Kj7oKQAF8cQEA24g4EgDsgNru0CgDMIgxAVBlTiqiZpiHU57xSHbiJcxBlBbgY7s+6HPly97U0yspdRqzI/yxm9Ua/a/FW6jqrtvYis5hgBGpqp25QJMuH02E+vD9rDk/9dNhpbiVho4RvogQLen6yWWEXW2/wC2zOtc/deGn/j2qffvvFf/03/+6e53c9hlsyKqWDySmELk4q3lQH6GEhoHuoARgvyAh2Qtyzy3rBHimFerg7hIGwPSJO6lt6AOGTqoC7zsxsDTfYMnYAPn/kU1D59VLOwg5LSEdAMCJbXWf1KUJXw9U+LNKsBzQIuNbycvmu9cR68t96uCj36uZZ+e/xqqifpTFYraIu5LyeCa/eO71LUHxCwHI7A/Yqh2Vh9nYZdl8VDN7FugTryVbV1K/QA/SjU8Az8J5N4aQBM0Qp18yc2CHn07I3akXTTjSr+qx3z9JJnw0IYCxQanxay2whWfK3Om3IF3O9/e4pm7fSTqh1FJcHvcHm1g6zMTUOto+SahWtJarW4kePHo1Go0eP7x8eHj55+qBarcIOmRQdj4dvvPXG06eP2WKCr3Brmk6nbHIPBxoOT4eDAdU+7R2RnsqTjLPtHx0cPHnyZDQYoiXFLPblrPcoxGv6H33+92t3b3vHp6VKdfe5myeHJ/dff5OzvCcI1SnOLrNBmnImCzRbLFYSMAKElB9Q1Tx6RYehf6LGmg6VIRf2DLmGXeMZJAZtno32GVvnSwbWRKOXxXIjgsOiuUlYWbDND3PS7PuzmCdRDZouz6MxJkPWwAQx3tGBNx33sSHyFmcQbNjV7euf+ezv49v56ve+fZg+apcqeIaz5wOe0SmuWUg/fgo/4g/yRAvwTMJQC3vJ4a2i/VX8lDDY4jYOn0XKEN8RPZdXDSG+EjISaxKE6Aw8R4MUBdDUQSVXWH8wYPFgvKXYhAQRAfcrySRiVvBqfSbDCe3HeagUM3dvjskQHFOCoWniyrBR7YfFjxZvyeIiqx3mVRl/wET+yw9OVIfvTAEU+AmhONAUrNgwcLKwbSZpI7tz6IgGEWzALnu7dmFjukTIqe1S8DgQWzVyLzKLpAJiU113CcvFgEmzunh0rBfKTGKmXLVhVurjagbYmTHBd4pTK6cpez1MTkezIbuxlL12Ndjevskuod3u4PDgeMxRh1QJLArY20zsnHqIu1IDcT3KgD2ii5kZRUBa0kV1gjiixqaj2WekV0hGeo1WjVp9BXDZrxqNm65RvqL8CskmTQfQLqy4gIZNweQ1LQhobDNMjBYrFbDTC/uAO0kUY3dV4j0vB7P1JK7mxDtatP5qPewScCcSUiP8pfqA21pr8cgrSDRIT6ZaStoFD0HEGadY9Y6yxaSXb+5vb96sN3doBTI0J2Lgn47xRbu8T/NytfrRjZsv7N9sNRp//pNvfe3BD55kPWb5A4+VF0yuEWCdEjYqjDcatKoJ6ocqIow0CBFUh7haCc3PLseJScx7PB71iTFL8Nec78hQsbx1eQJyopSDY85nORUhB7KLgDOIFGl+A4EVfl0s6DeQ9QfLwnU/3zhsuPzxBYAU6S+nJIbELsGVydxbvQL/RePO+pJXQrFVDjy4/AmQXo+kN0w1VNBbl8TRLA1Fe0tKoQ5KguW2rAZosPzWcMWydgWcdSxkmiIwWcJZggzck8JJctQNlgJwTtFsXI7GnSRtQQDTbvfJMfpcFcYIeYUIawm2NqDY2tpBBx2NJr3eiKlVHILkLRVrs+BuvxtN8GWGLqfD4SCOExSNSTo5OT0hgH3VNNFZrcY+HNX2ZoejreBVTCJCbEk2HuFiMakkyXiSjscD5pXZKAc9mHMDmWKttJocaHRycmILYaU0JRFqebLd2Xj34OHW3i7qcLLVwt0UJ+xar1dix44JMgSVnWBNPMV/CgaFJxVMVpRRNBmoMm6kCiMtQ8i52eiBQRhkRMfgvTxqgNFNpgSTDVG8omuI0bAGrraDtFY9GZti1YpfHs6DmHlfds8W6WfGcOMGn3HWoT8rs+EADU/qW944DTrXXvrYZyAGb73xw+noZJyz8BqeiJsXWy9N5ACEEp+ntU4WVpulCrSn4qP549PLmmC4VInj3VivJU4xZplqiksw8j5iOlZcifzgAFQPDd/avVQVjfyKjxsbJpI6QvvIlNVRrP1zGyKLJZOAZBwUhAasMI1H9cKUb5OzUridfCCiJH4POmkrDdKD705JA9jSOESDTUfUuEBTgJgJfCoZ6JLc9tGSv5U8fLGjuHEBe9REJ3xfgpGBO8dbe8DJFxIXxJFR4tUWqCGiEh0li4RTe6C98Bk1jwTLS2RUtneZI1l0DLQ4igPzCDJaygZurBrnCCh8krEuz7TRA/48zc3wxua1zZ3rn/r07zEf/PTJ8Vtvvf3w/qPj4+PDp73sEGjMfDKZAiKRd+NtsjhQa9F3xAOpThBoh3mi1zSFkSxWLclE2ASckIwVMkqjSGoqWgAC4WumoPuKAIIDL0BBwM0r9QpitCQiyrPcScNrPuLPqI8xHmVSXAV9OBdbvF6RHSJUgpW3Flak1doo2CpNkaAIqEGihuoYVU3cblky24tRZa1XQ4CRrxssGdQIQWuU3eFswJyObdidJuUN9mplYqx/2mOngXZ7G488thlgkUNzo/PPf/cPm/VW/PXwmw9ffeoN08hHiGJ1H/7XLFykWCYhGOWybxVtJvbiRRv5A7Ci1auA0FXfiXTI8KUBT8fph35EQAe69Cz50jJ8+UwDtm+uvrnGu/sSguT1XtW6Op//VGILLKHCRcf/piq/jpdFnkQ6BkxxrkT7dZ2qOrgquVfuK8tnieIuTxcPe1vlpgg+gfcoN1DAriKxC6zn6RLoK/Fb8RSNf5bkyuGD8cRJYUIcOGBUYo3NsBqMmlFaY6X8JIS5UnCYVNjVasEWHEm9Uimn42wyzieTfDyajSYZ3DkpV2q1Wr3NWdr+kJOphwNWAcM7MTJTT3ZRjMoJ5K610dL2NjOvXC8nYXTSO03KVTQsWqf4uT8a9JlLoyZPnz4NRsyo1VlvNOj1x2MOGMbxaZ70u7ACpqLLSVVTebg0Q9NL3tPDgx/++Efo3C/efb5aro7wcQrmnZvXgv4k46zv07w3nXX9rAs7xrzHxA0qvwgXMNSYkl4kgsUcsdxalxRTkDKOwYCSNVNDjXGLGQB+LJlXxAjQaS0YPc1/EXGTbdizH59szjXMw/4iLjOT5fktoCB3sLTPbG01rFab7by/6B+d1ljEmLGXxdzr7Lzwe1+o1svf/87XTrpPNuIIFsDWM+ibU86Hcvtmsmc9JACDfhneFGM6nk4wfDJPWmV1WLm9EfXZq5JDWOEYohbiX+gJCB7ibaKVom56J3UWrCBCjVOexMgdTaIWNls0c1mhtQQf2BjHoJFEkpJMZri1GNdFzWUuDEAYGoF52jpWSjB35qBJLMMBn2gVK2GRXdXCOKRDTo0EyiVjlBU59sHWZbkWehtgPVZGaYUxnt4wZnkAaq3RglMjUT+1HYY+lwlD20Zp/KD4YcWW+QITM1MABPhDfXTvufNH70oc5c7+KkqqRVg4PCzQbrULh1i5BARyok5NsL+z8+LLH7v78ivMe7Bs79YLO3df2Ts56iIRPnn46MnDk6/91WvURB8KZwRwQZtJXYoXFAkaPwVlIQ2IszaHqPoqWgIKXNYhpUQSpRaSgmCEACC9Ss4SBomnCIHPsJJKitaw96E2DhcblvplnvvUgoz53NosrsKHwvsPdtGc9Q94XM/DPa6nUS8LBmdfFWEC9u1ZHbRK3pY8AD0M+7QEGToGvScZcj2rIoe4ViKKTwZbu7cb7d1pOmffvIVXHo1OF/NhLWnSsf2nB+zy8/kXPn5ta+/6t778pR/+zS+yI/ovR+EQTLxMgrRQyjrfopYs9lzLVg984ZiudaNLqdUBgJ2eEX+1aSukBpdOIp7kCo0XyVgkWHXvKsuzXyUkhUB0Fsnz+cfi1ZUEnbfPSo8Dhnu1nkBdAuApwv1ZJzhGYoyqKO0s8EHLBVnPPl4LFflQHy4ei2st1Vlznlkfus5AxOd8SOXd5zySLWH3YfEWWkGclXkOznAO96Gl13t7FFVSp1y6imxdzrx3eWpAG0tdfVWMCmViiS03lyXIJ7uVTfKhAUOP6CeZEfF9hqANSt6oHI3CeTfK+zHeh8OIadT+6SkqwLW9XVhLP5zBbgN2V4yCh0+Ou/0JOxdz8MeTdx5ub29ff+4uPlnxtBIPy71+nz0lJmgn2i4qbMTh9Zv7MGnoy3TMrLJ2xtjY6qS5v9FqtpstMoSi4kGFbXk8GhBY1LDBptNpuVatYJfWlKf2lMwwOcN5pGHgfVpnRprHINmoPHz8IP82B/gOr1+7UYmZgcZBLGhc31z0y9u3djvD/e63vvP4wVtM5202GuPTPnoU2VXZgrpWGw/ZjYp1UFJWWAwDULBVIRnAjtXJrHK1rR/l6StCtpjYRlSIM5Uq9u+EAxaZs9qQMJGedAcbjQ4nMELaWSKENqjdBpgzFp1Ep62GlaZ0vAley73SotFge4657N4sxeAkQq/a3P/IJ9HrX/3hdw7uv9lIYtpOHdDy2aLk4OAeEkl7Om0zV4btAqoeVDkjihMfIRUALypNmCbfLFfK3crTx0+mo1G1nLSa1dNskjCZyhQrk/54T0cs/RKwdd6wKAV/NBb+o/ld+gj1g7yRcFKMF1okSydj88tyHL9ZtQkBChec+8ucrIQPFGsIDrmDXqis2kZTah55StkTFwEtpfg6Tk8873glRwhxCo0aST7IEUzV0bdwT+VGIg6UZd02S9nQP81oiK0YRZlXTGajoM76mDMolypRe1BUG3CxGRrnPRuZFT2QaIAzGgt+OQGE+RPKkzjARi9Y3+H3jOkM/7Ucr22mENIJx0EPF5gAWGiEtQZ1DKDg+7Wzv3Pz7gt7N6/V20FSxv6P79Vir9HY2IUNNO+8uJWOvU9/+jPf+OoPv/ftH/V72kwEv0CKwBsImza1QpozFwS4C9wWxMASksUJFQZMtAJ+DDAkx9EeKIfBULISPDSOORcLE8DSqslbJBTmq/nCxBvNcmjlMTnyNXDwGRjsu85R1tqvBQZMGDhgVYDGYOkxwFMcwFgSMT2srhVFWj4bJRE9KQL2QpRt9YV+eetilvTNCK2Ld8mQJVxAfapr+TnclweMOZIsMXXRMIkskwrIJEcKqCVrtf3jrs4LrY5ON7Zv43QYxW1OzgZnR+kpq5hYOjHqsVPe4kPNnc1/+E9f3Ln+77/91989eu3AWwy92ZAsQA48F4U47M8eoAPA7KnBqhUWtCoVDXFcUlxbrXP8VfXGhCMUhIbAbiUpsj6YgQTOgfeis0SS7JkmaOXxm7hcf7z/nNSw86lXjT8f+9t5crUt6vwrFK1P1nCOxwt5Xqi4FcFt2Wj3aGnUo8+6ijyLql6ZElA6aBb5F8mcZHAhXo/CCgn1DEPhOiMbYc6UBY7qy2eHQdCtJinn0od4L/fZ16JWClgDUInjOhrvu/ceP3r0FGbAfO0UI1lUhR22qrVqvQMBffMX9+ubVZQ2BHVwnWWqDXbDqFYbjQbjja8wZVOxvKkFrQxRLJsQ8nqDiZsaTI+lvrEchgN2W4ZAcm7wmIlene0XVjGNp4mo4ZgZQYiJ4Td+O1B69GXYBcSrVOoNTn/y01fv3394bXdvb3uXwfb46TFa9a396zc/fOczm/XqduPH3/r+qz/+ebMc7W9shZl3/OjoqHtaq9SZdEX/4LQISAhMg/9ACp4JoWBiXKRaK1pkN4BcxmLS0MDFEPYkciYug8rPPgsVNpnmvAYd8iA2hhKIEVTrlOIIpzJNWHIqYppOxziiRSynQYGGSLPdlWNCbIjlVWt7d18cMgM5z44evt1k14kIAxqbMbPFY5Smg173CXWrtjFH4yMMlWaOmclhMuBAe80VwuCr9ebWNkdQ9fD5Ho4nWCiAqM6bl38RdGNK9dWJnA9HwzTvK4EMsmJ37MzYe52KDKGU7iac4a69zLB2MhtAPwn8sHM5o6hQjQXpcOK4Rq2QXNQxMFq6jMkLOKwosFJRTZlwjFIZDRMD1qrpErJZJj2Dk5gnkGHS4HLjzSeo7WJKYjhmkhU2s03mALWPSsi4TrcJNbQBvkyDUkfY2BIHa/H6PEZjhpoLAuj3YscqTqq4FQp45ziwTydY32GB6n5Z16Ghvlcpe82N8vZebWOHpdfzUoJxRhKglFsGQWle48TISjybBq16u92pb+7Wv/7Vbz55kCd1MveG7FlORcRDTFbAYiK5BTDAldlpRFwTgRzIxBWv2cIZImT5AI/U2wwsHhIFa+K1Ko9SkU/kYc56+Aizs1zvgBwnDLGp1ESrwReTmWRcSQ45p3kKxnwmiYbmyOEfwKgv1Lj3ewmk1Nfo2Frg/X6+lg50snKhPFRBF49njFyTAghMDD8rjc5CONUCB8z6uHOw9Q+b7rDrRSlIKsNqLUuSNquNEMIkBs5n5VItHfan4yELLf7opU/utzp73938i59/+5E3jOPKyYJPuxRHN7HAYDLqa5ICnLA62MwF9eHxAo9SLS9cjvDSPzJUiKTiAACEqbdEdsQnGzV/LxkwLVH7rC+d7OF690ILf+OP4E1xUaK7PmgpfLXOgIvPFa82LXuOAGVxt0t4RoC7RWoQqsuecYGYAEhV5eZ+9f0zUitnulyXK8KFrQjFufjVKwifzK96SxH8GgPmQbsVpyf57CCIu5UoxZeEIX08md64vdMEp2dstZE2m51aHhwcPE27x6hlCUZjH7kvrLEAvl7HM/l0PDp+0q81a5usNKpjKOXIgBRKA6HH+qc5ZmzR5sgLK2VpL6fJsMFTpVGLsXBjJuVMAmaoobcGOXCYZEw3wpLZtot9qEsTv6x5LtRTqIdJEBHeUaxQwAuKGUP0jNnDh/cfzh+gR0Pfq436Gw/eRjN+3D9KWuWd/a1PN79QbtaZ4Os9Pnh4esQCl/pmC/qFv1evN4bTw0UgT2TNDwRLw5qBjX9ZlMAhZPMWSyAFtE2w87UUCF5WoYE4JU+zPMxmnKO2VW+gWPFHZTVxDE2BteVY+CEmp+PufDII/Lof19lQQgZXuCM8mKSsX0Rx8XauPb9YJJH/6rA/5GzG6Zi54jKkGh/r/hFKMMeSI8BAfit0pRgZVWWKi1rI3VmnIlRqjUTzAt2To9HglHOUODlA859MdsJu0xk21Yi9eLUE2tguCAADhjnjzWQ2EiDPI9t5QMdFK6VG8bX2w6BH2LAK2FNxKWLafwOmoKYYYzHOIQwWKKFG4r/ogybWmL4Fy+TCdQAICmGpMm2nX9k8lOrBcUmGFQElDuIsfXA6RlCE5WHX1nJhZc0bfJtZ5Y2xmBKQBRDvYlqDmivWSQqKpGD5giHlwJilZgl6tFL2ai40Y8rW4hcYVorxGfkRYcdZeqWy6ziYRs3bud68dr25uc1EOyvQhuwLIodnjSMOo5UtJtYib1bjJds7G0mVNXVHPyy9nnFmGJvFwUamzG7I6oNXGsnFLDDqsy2X52mDCdY5h16S+Ns7nWs32Heuoblzmw7AOUlHc5RZ6Sehz0710FnMlM2zyD+beiGHsOZuMstoGXND+ACcDtKhpiy0tx6g4NQCGDA2IRZryTOAistwapc1UiH1hOjCVReQujL6qsj3irPuFj1cXtb7jF1TJUXsiF/Wgd2+jAyCewxGohmHTIiwPx6W5MXjvFzbarTH7XZarmwyOEBX9mCn0/FGRMrF86rRbP/u/vOVqHxj/+Z//+0/f230CKZYT3Y4sGw0OOZYiFbYmDLlQIEUofZZxVaSgdCXsa6Kulpxhy6swEYtVSlolQQIMWLhPhCWDCj5lUsmCpeDZaEoXZanC/4d3a2K1E+NXt6X9Xy/FVq162J6y/FiZPHMV4x796jRdzVWFcmfGXClX1kHl2fxygpRpdbK0uOz+kDcl262i6q6fLgzkq6ojfBmmVORfxFw6V0FXCTDHmlZdhJy4ztGA8iKFC/rKIf79jkkPg56HOTH+ev9YSYLbX2wf+sWfPTxwUkprjY6W0O44ukJqNba2oIEszCSPfhhQjC2JGRhwLS+UW/v7nDoL7VjMhhGq+lG7ZZdgoqgB8Mm8gnEPIMUxdVyyNaXCafccOEqOufAwzCFUXP+L17V0LkQzRi9C+UBJVLDUJsgYwmFbQDJOeud6q023AinrXSCp9J8MBgwJ/fg0f1yv/bWO2/+/J03Wf7Etjgv3b7z4u3br3z64xyu8qNvffdn3/vR48ejnVapXq7OK2QIm4FACubivGZRlSO0SqRslhCJEUHBoJVMjNuOgSxb4XRYlj7LmppCReSnNQ37xyj7VfgvPUm+AQuQmP2licAfQWeAm3f3KMUte2PTjxsos2GW5rHmdHU+DCorCpXXat14+cOw059+/5v3H77VmpdwKcdGj0t3NC/hpcXmAiiEEPeknUeVHA5AndnpC4xgOjmssIEWDKnZkoN7kLFvJVyHSXT0KDyS2TmbfzABXUIJ7k79FfeVkddW/arjpAfrDqEOcvoFlJFvMhMX8pHiD36PIUN+y5hUGF78ihQ5ZmyaqYi9qBL4p2w0GUwC6dEk1tAgXjOfRMpMKtVbFBGWwbw4m73Q+ZbKFMhI/tSAm2/YujKCvakdFCfGRncw3YrYBiqRA2iHPYKEsjtT6VI1bMItNbkKXjGrrsVpKZtvIo2YxV26L9+BYyzwJRWdXq56m9vJ9l6rvVEJy1RtxH9p9owbGUGoCWip1mlX1Jhj6cbb1+qf/4efbrTqP3/9nd7hqN5Ijp/izY4xlUpQ75QAH2BS7rS9nT3v2s5mjc3bqvHGZmtnd6PZrjJOtfYd9skhi05sRadieiSojEeT8RB3fvZg4iXHbeMhJtabw0pw9p/MR6ej8lE47E84VH7c1QkkyFS0y7iE6d6qBY0AfnSMpumhBJLb1EtXkJmCklgCScdFQGPlg1zu29UXVgXlRUC9f/HS6BGRYuwwEDAmWcnTYDHudR+wVwdGOjzhm62sVu0gbzNRgx0+YU1eUMrGfbxDS/XGy5v7+zdugQR//uNvfvvgpxiBMCjNvYp6gJWC2vkFAc41nKYZaeZxSbENKKtqodci+vEEjlBlQsJSU6LJAawmQo1A2tH3YLMw0r5ez8fFWPSvf7uaMUgYuAxOlVZ0gJDHnl2s7n9bF3Uorg+KQGoXkLXLfVvk4PJ0rwgTADldwN2JIeAin4W3lhs3XS6rX3qXzrbiw0VAUqOVxzv3xyuQQRvqabixzlBYLUooaxuMZhj6o2oyisIhvIyj1jhVkM0dxm+8s71/o1Jv4DD1zsMHSPpM36JZ7m52qvUqfcgY4I6qBP2ttuq1zSboh5oLSwY9A7bPlUyhvZuhNyFrk8pliB6kCMKixZOVGh5e6GfgLtQAIsMGNzJOD9GNWcXDrr8oP9jUOANWRmZcZKDy0HAGASol7LLaqm3v73Ak0uzoBKCFMdsFB5Px+P7De+gNTx4/mA7ZN6T7+k9ffXrv3sHj5z/+0kd2r+9Vy1/Yv3b97Z+99c5rbz159KTTqHeubU8nOF5PVkKuZiRtwZLYA/wb+KErSe2QNiUKzBUnbNczovJ4jDAq0ca9OJmxs/xs3GDTJoY3X/l+wolOuFRrQQ4mbvbHPO4fd6f+kN7YRNNNSmxCmc37GO7ZO0f62GRSiUKv3br2qd/BmD/7ftR/8qQ7GTMfgCoMU/HYuxPrsl0tCAlOWFGnFFVZMyxnqlmK+7fICQytXmtV4u5T1Pw+OUPSytoSMySN2CtgVzvEYulGSR4izUtmjMbvmDGYC2wlETJNDlO0BT4gEi7c0CRsD+zv53wJ9CjWq8sWioPH5OywXgQL0gXRQIdmJIFOCF2UTSQsFU+npNFkshczDZMbHELhgRfm8aYuF8sgM8haAs8TOlAmX0H6yB+EY6ssrRdDcxRJVbPYchI5z7g+7FzqM7PjbPGB8h+h/LCWfILTuxaKszsl7I7v+BAKq5VpElA5BaPZ8nZ2252NShQDDlZ5MXuImYP1J/DdwiOMVvFvUa0h66T1dvTKx+428cdtVu+/+wQD6vDa6OSIdQAjrKf4V2NOryTivh//eOXaXu3a3g7bqVJzMJb9RxOUOE4mYcvquTa/x8xgi18F9Eq5iuaLwo8yKIECCs9pQDhIDJnaQLrTFMe0WSlXSv2TIWZ7OgbGjEUak4m5lSHzAAh6kM4R44NUi8Wp43UBzfdLdyz9B7+tKxIrHkHD5LxjJEsVWFUh4DQ2rZFDUgvpSswvSLNgE7aeWYSUNWTagon8aeq3RpXqRogwW6pNphn+CeVKHVsDa/4XvTEb6f5vfvePX7x+5//3jb/80htf49S2nXCTpRdP80csope5WEZvcICJD6pEDVUxUNboN7WibkJ/a6zusi9bbamo4Qtxy8pb3U391XID+Re4xqyapDwsDKL9XVyMCdXAasVdD79STVwOl1vw3q2idAa/BqZdVvh7f3G5BMUUpRcB8ruclEjwijdq6fmGX05MDH1rGXJTHxV5FoELX0kK07tl0UUyAgjILjFhLpcbPzBgETvRWU7wY+oEHszAZ63LlN2aKwkHH7H+BeExn4/LzI3F7fjgpFtj8/ogmGbTo6dHuO7s7e9O5ml8it0NPa2xudWBt0JSa+1mg2VFOOuAo/LiEUGnKjrGhwMToLkkS9hOAYsiE5wQB1RgBot294FIQk847QeKzIooGXKkY4XYrAOdOwNVV21ZXYQhlawhIJDTiLPsWw2cuRptGpbjS6VVUr534nWZA8tGKZaorUYV6/Xg5Lj75Mnw5ISlnR9+6eWNnY3NzgYuMz//6Rt/+R+/dP/d+yw1xGRXLddUZa2HofKmAyF4i8BjwwN+IZv9sxZIDYT+sSF+FKF584IzxtkpCdrGItIhe3HQfD8G2tByZkp9v4y0nSRajggpQUtnw80RgoEhRyuY167dhsdP0TAryCgJKiiJcBFBi73+yc8ktdp3v/rlg3dfh/qjwqZTPKpCmsdEABUVY0TVhaFBjzFfQ5all7PAZiKtFEkoga+1OSM5nQ6m2RC7MrwL2II4mhI07iutzy4CUnlRKxkrEnKwhcrL2rRa5S00BcqiWjBaKLl2kZRyrEdt2QSbNh4sLAZESzoGDsB3NEOGycVwEz1FE8lmoBaJhauWQCFNppEfzmEcuIB4SOl8ijORBq+yhi16QV14q5MvnJOzlA3OXuacKOQzvPKZ3+Mj8yljDVGK3g6H1FbSEVuWlOFPSRhrwpz5eRTHWTDlFAtYF/zOGZ9N2WEOuN7xOlvlzka9Vi9jvJ/m7OnNjphwx4lgIgGAAmHDMHXZkYAYVdCTt9jZ7/xu5RPP3WGJUhel9+G9hw/u3T+GDR96o7G30fZu3vA++vFOu1NqcRhHJDsKLEMDBwGB7dNlMVfPgtHcjGxhKpJSj76P85AuPK5siFVbeIWV6ATukzIMSodk0NV422XDbDQYj/qsWIANM0ksYUvyIwYT6A2ZqM3MEsNmZPih5hfojHt0RInw5cCV6a+MVGlnVxEmSyDGH9eSQxNDCFl+rp5XvZiNEK5KXoN2JCiuyGTsOcmqP+YV5s1JVG2XW9doOWJqkEjQ0/Z90yyaB52g/Ps3PnytsXFzY+/Pvv/l16bvspd95NWgiI4LGbOlQ13lzqQEVQwUtksggw5YE0B/6cGClT6iZlSPz4AhPUWM/puLnPv2t3U3wH2QzKmktdh9yL3ohA+SywdO64qDwrgvHRC5f9D668NzOHRWkyJPotQuZa63q7A92OPZN5dCJBbBsoscuFyYcXgp7bmIIqULFAx4PZERCDJ0IgiohBQsBJfxj8MHS6h3bG+l9Y+I/zPG8yJsb2wM08nweITyFDFFql2bfDbIGC+mnU6rUa2ybR82SQ4eKpfjSoNzbAMO5IUiw4+x7sGruCFcQlbQuDXQmdBicy08F1FmgCWyp1xdxe1kjKS6oi9SZJFezSNI84VSqdlQMtdpOLhGwxSIq1bl4RVXoaYxFjwmwqAy0FEmEWHW2QRHlvFuZxN9kPnpLJtSrd7x0c9e/Qlup73OVlyKn7/1wkc+/cnuYBiUk9MTXLgGnHsgX2D55CKhaBRJiIDda19jZGRNP2v8a5/ZmGU/2FFrYaWx1WEnjCcHTwbd0SCXg3PbnySCKsPUdsRinWk1Ymtaj8OK2TWCgwezU6YcR/0EltWfTbc5obzaDCp1jN2i6+zxrNU/CyZuK5321oc+8pHh8A1WNj15CBHFmxbTg6zdslH0e0dPpC3NUjY3KTW3FiEzinQpJesqQZSxGsRJU3unTLon2FzZjASKRu1kwRZRg5KZsgu71Z/DENqtMNqgRo0w00gNN1rN7Kzm36WFCaOM48KSHbewYSVztAkHWoukrgX9QO1VPvSgrNZkQm6aGWWXRXQ7vmLvFjztMBNqBypZCmH+RnoZU5QD/2t4cUtntjMHPNKG+6TAVM3RCGSpBUNakYxUlCIQsfOK543xr8XDfM6WhIvaIqgh1qlzFkEFzJwHTGJw8hTHMZNUdBO+pGwQhvyNLWQXFqWTHmTkLYClf7COkE4AAj2QN5El1IxonqWjmP1l5NKGRJu0bm1vbbbRfQHS7k51/3rt5Ojw6aN7x8fTTsu/fau1u7+oVDgsi51VmHEx0ZNOZ8o461M+IAYQOP8xOS7YIe/JJMAIKjNnwcjCZEPPowbW2Ccdt15M8vKQYyaiggN7GqTMRqSVeVLGwhKlPVbuZ9kAr3Z4E/KDaLHcMNWUX3Ktek3JDA1cgG5dEqhf8v3qNYhrwbUizaAoGcvlrNztLQiDkEWD5P6IogALlgwMAwx5kzG3LcxgrLP/O8d0zYejsN5vwnhZZZCUWTo8ZSViFtZjDPpR/+FJ1Kh+dOfGzh/+S9Zr/Hdf+dMfn75Tibyh1ilBhxxfoARqIW5PLWga9bAGqvXL6oG2xm65GQ+29HpnOUhMkkVRX1lDr/SCdoU5QPDp+vVLqPx6Uhd2Nbsc/8wY12FFeyzdlVV5Zg6/6guA4q7LGchDgTH1y+58SA76f4Fz25hUz+myuzWQMGjDF647eXfWkdblTquiL5fdtUyg6SsuK403aHx8eL4rnRlc7AEVQR1q2CIJ3LCXaTJFCl/dMw/SW6BnUiZIh5BNZdDuUIRjn9MChglrkBZSGaVrgkOsoVnMB9NRI2rA7Q4Pj/M+PKLW2duGPt69e+fWnVvNVhV3J0g/ehSzgdDfOhMyUCrcjYjFQ3jOzq0N6Bd7ZkFVUFBKOFDHLMuTNgSf1/6/0C5UTAy0qip+C/LVqlUYQtpVmdZJb2ZyFHUQgswkC6tcIUXsTIhrdrWCoyt+M1DIarvu44d1elqaJJDY6ShmT4pWuc2KFGY/tUy5Xn/84OHbv3gXffzm9VsoXQ+fHMLL2XH283/4BxT1/e/+4N2338WXAw6pvhPMUIfpv3wwRZvEq0pkgF2oqs3q5uZmq1HrNKvsa/3Ch1/u7Gy8e//eg3vvTvvDlPMo7j2a4klOczhjmOnF2gJG7TUi2MdCZrUxOyyxdQDHp+ESdDrvTcqTnbsvbbRiiRHZAh0trHEiMOfYsQjmyGvH17/wclAf/eBrx92j/n69NTkdchgQ/IRtoieDI+bDcKeivziFzUsg/TW5pWDynrOTIhtjzuv1GmuEmx3sn7Mhi3c8trAQ90XQ0hyY23CD+QS5imkCmJeGg7o5CzIDxKR/MVdovE1MyjIIM4JDIJAYLTdNxSzCtuOBcE18WL4poJ68yJFjYG9MgWDg0BwqJ1uKy2Ab0BIRGITYr4f0Akbo+GFMCxSHKAdFFHbAZaIqGzB4PrYK9hc7ZnDMdXIiU51Do8v4tcGBsZcgM0kNokXsF63pF4rPeyo2LKOlmnc527WUFzjYzf3xNGP1OOdriYSqOtif/UYDZoe7FBYhGBzcD7lAmljIrHseI5SKNSJngJcy5c4TrC2MO47s8iqgOFYEDOmbOy3s17VG6fqNejrefXy//uD+22R+83qjtclaGuS5HEMPy2iY88aqwfbTMFkW2GiWnTHLTIDGAds7++N8EOCJiIOBbCAM6TFxMx0PgliAkoicm3A8hF/jWK2RUZu51GjAFpSnDGxOCaIbxnk2yozeycYjQNEojT384oT0jktCPRTWTTKHRsQZWVM0I0QLYYlc+zNSxqeCi2WklI6viCSdu4ggjaIRau1OcU51oWwyYRs25gew5YAzEn5IqPkX1kOjv0K9dLgkEwecpYRqPwqzvs80/WKXaauFV4mjBksYEduHJweN5ua4P0TIqu22/7NP/IOdrY3/+etf+vLPvsF521OahcSkPwgLyMxwoDKiv9z1H3CoEQZiOkIwIYo2qc6qk+K49MilBqziNC8lqo/RWlkqJU8aEstL8ctHKsDAZFbDXQbrFcSXcZd/aP+V6a0Oq4pZFS0r2WyKTFx9eOQVYVeWu7tX3NGQivTnAmd5n4um1vqWP2umy1n5WP4uc/fBsiAMbAKZ5Jr1u6YjRTwZweQmiAmfDAOItRx0X9YCfqaR4jImTiqm3qoflrFF0ebxpCaTQNzRVdju1FGzUWqyMECfsjKDdCCICKWwTx8y0ohz5bHQFoIBsgLDOaezkhGaK+SFXZakr0ED8F2w1vFdwAYDjELMhdrmD3NWzjTkyTx7EnmPk9nx7KibhROmkCBss8Vgc3uHBby1jSYnFJ2y4V7gtWob15+7uXd97+bNG6YAMWtXhtNSYVyd2YxpMmbhJmwK7s4vtAC3UuZwh6xFYltGLqie5uWokTRhxphWuNK+dDRk8yadqSOzOLOkw3Q6RG9uNpvM0eFXBRclR1QydptAFYHcYqNL6nG1WWZNCBNzPk5MXliPWkmjhsq7OOnVcdnNmE+dtpI9LLe4bu2hLB+fPGazrsnbnEFce3qC9ZCNHMej7Pmbt//Z/+q/wP75+s/feP0nP3304CG7vbMvCH7KTHgP+wOIG4QAgLZwj7q5f+3atWa9ko163e7hj771jb3r+81Wa7/dydlD8ub+6NYNfK3aeIiF0eC0F3P4QhtX7TSIywsmIbfK1V5l2Jv4cRqEg+l8xM5VlUFan03Y6iGB3qcnA+31gCkVf2Ac0sre9Y1ru3fH14dvff9H3/zat6+HzVvNrRTLxLTfqu5yxMPBk7cn8NZ00NncD/3txbTCAT5oe1qIEoTdk3mzVS83m1scXe6H/f4TnMUC9jmYn7DAlH2eOCgKvMHyzXYHuJXjAiQF29AY1U5VQFWNNYpwOU/YoAh+JAGBuWfpfoAG4i5RSsuNZDzAVsIMBcxJbIgdrLSiF/8D7BMcRyWdLI85RSNcVONFuTQLZ7g/U14p7uDCPJpgAaiwMIzzY705q7ZYf97HMoDEwFwmhueAE2KrHcZoMO2n00fZAo+0HsouvASt1atuci4GQ5bZALkxyQ0bZchP06GsAwwmSkKYippwvJmPXtgrNepb5aC22T18fHrwgJMYvHor395uVOvzUp1NTtjQnwEJ6dci4yjp5PMabto63xY4IfhgN8LdmbkcBAjAgDiiBdZ2VIMgEjJLy0w/3lScNYyX0PbWDudSh3Ea4ykAZsqTDbeILOKb2KtpjhkJiZMJJhx8GCEBL9iidc6qeNYYlaNanGz6fssrMwEy6c+fZNND9m9NangAcED1lN2zF1sd0MwfTKKnLOybTcW9sLVCpOir0rwanDJxOoZ7pSUZ+iEd6kyPjXggWk4aRq0Xk8HpcEl4RHOwRBjRM+IDmRGZkgUbkkUzIZR6IYotriOrhQRY0kHLcAkAwcQ/l7TRUUgxNNBMrgQXL2bKyFF+BuTAS6eI8DUmKgT3bIEsgW1HkyJ5iCtaNxgeTH5x3Grvhzs32+19yAh+dt5ciyzG6RHIT70GR0fN0sZ//sKnn4uaLzd2/ptv/emJh686kw/cWe3BjmcGDoRRLSuQfIIoqg3IwqheSWZ9znFZkn0aL/Ktm9rCZXKHqgo4DDjYYgQaG0dUXGo+oqhALZWfGMEC6dS+JyXAXmZl+f2aN/pimbPq9Hd7GcacqwJA4cK3zsWKvwr0q7ugqrA4MdU3DBCKncvDPQgpAax7UO8YXq4elf/l0lfsWqKlUFKAEpaqO1eXhrx1LSkkJCleXaaShBOk1l6BdCav6Fd1ObqGwhBP+bKoFK3DMFqqVAt2osL3BqvVPIOyT0PO68kPS/lBUuqW82HMDNYUajwEqeutWrXFxnrZQEsS/cYGGzkF29f2bj9/d//mtWq1Iiuzv8C7Sgce2YYVJiDYiQUqiipzGiBiPTY01iOxRBH5ImNekwWN7B7JN5p2kyaMyCB7JToOnkFUjO2U4XZsZYg5W6CTfCMzH+SDCWWGNQe2Y/1FHS9XE/oHm/NoOtCkJQ49rAhFqGJ6usbeW2xincGPa1HSqDbQ0Ub9YbWzMRyOnx4e49N0eHoa+eFGhaWX1YPTU6STWy8/94U/+vwf/NM/fPKLd7/1ta//4LvfeXrv6b0nJ5ifcZnB6s5GYBxhxGpFlmMdHcxwWaXvqPDJ4cHgtIvLNzOO1Jw+QPAYs7F2hJfWaUK7ba405qAnriTcuLm1uWA/rzrCCi4ymO/9eHLE6eLjJ9jT2QqahZ3p6XQnqXdP+0dHo635td3b+3f/4OWNm9VSJT36wZtHg8MWcFxEvdFBEFQq1VbaOxlIR+OMnkVYaceltpFBHINxBObMVYggK0rrne1rLLjqnjztDXp4hGqVMyQCWIGHdAYm0NkUOEOS2OsAgUPyHUAXYrJdl5t8B8DiviKNtmmikRvQDWSTMMtdU5lwQBb1sDcjq1PZ1TdA2sAgrinNaqeB5BXhx1dJYAEsgzOvclyasRpz+m81mLcWHh7LyKgDL6jggDD32MxL1vtp2oung1LU8kgWdzg4Nph3vUVf9B7CBitn6xKcwGGxppbC2sBPI/5gygDc9CM0YEY2LjiJzX+MPTh+hPss+xyOSz7T616DPbuxI0TMlaCCSkdC9/U9PGxxeCiHXi2fV/DD0h7S8vniXCK8gmwwMp4ZhAYR0QLxIk1+sx4Y/TsI+uXyJEQWYcvpIMU0TBstd8AvBQhvMBR69GOzOrP0C/cH1n9jKMIrrYRnrw4IwdcMhyOfXWhS/H0D/IHZOIS9WfBrxHAP04TRRqXM6/q9mcwgsFJmbRgQOZDmJXNGos2UxI8uIytYC0zHgAqJxugyugQRkt7JBRDUHKNKvCJf0SHNKuoPNFlSL5qijEEEfWWX4CeCZmikbHXxRDIX52LW7gKesd5lZYwGQy+URGEF6HFgC30C/kLSOWeUn/o9ZKLppNrYRVJBxIIolthKAOqAGQM7x9N5aZK+WNvq/P5/tnHn1r/9/te//vq32NG0HtSHi3C06KOJanJE6o7mPChNG5HmOXsLVXwmPqwGqsXZJdagaIEaEBWXrOVG1omxcaTKSkwFVlwCvcAi+AMMzecYeS++/00FyJvrN5Xbr5BPUbqabN3v7s/K6sq3ijzXiCVmrGdCQRrhq8aqO4ArmLeMXPaeS6CkyzwVAhuN44qOiIuL9GlIE21SgOsvJgnFsZUfpJad2myPPvoUE6FmzJQL32Fi0so/pgq1/JD0iI2zbIuJNg7L1Chmc4C87LOxxSQupThvhjlOtujEGeddNzvoW1vVZnPsz46ODtFuWdnb2Wy/+OKLd+7cgTeDJ9qNgnw5YQDmykkNNNyGk/nNMi61dYRGrFk1SQMqUynYEnZHOd+EAXs1OnlZxBDHFspOJ7aDIwYl6o7G60w1kkGBDzBBDYNNUhDqcw0PLI4CxhhNbjFHAmHKxDklH7GFZsjq5ApQQnVmLDXL1UatidqG1L+xh60x3+sPjg5PWDN8eth9OjxBTh5PR7VueV6dPT26zzQeK3g+94VP/+7vfPiH3/3eN776NY4grlQ1lQaAYE5j5A42r9XaGfkJA28MyVQPXYjikrzU7/fZ1ZovGtUawnQwnZ2y2daU/Z9lKsCmTt3kYMYOXzN/OM3amzgXM6c3Gh6KeUa1OnOYbOl53wNKg5E/u12GJ6Qddg3br/3O5z765jx/8J03cKnaamzOjlmM4SfzOnPKAyYD0OJnQb3Nfp+yN0B0WCglZXo6msxw2/Zq9Y1qmd0vPU730UZXsFtOd9POkiCcpCFt1SHAI7JJrjNNF4YlYoTlGYFKx1KxMSQKqP0J5UFy9aLkVeR4SB6EBwERyw0MTBs4lXJMy8wa0Dgm2sutNhusBOUqu4GgXuNTLAFAxUOyMAGwkzdrYTdlnvCbSA2LsIfnnucNtKUYNDhLKmkDp2G2QIsV08UrCkTDhI5TDvo787DgJpwNWCASqQ4MInY20b5QrE1noRt2ZBhGGGfokWxvQluiGPEqLDUSVGX4FtMlmtsGCOLgsNiQ07dAVNnfQcBggUEAEzTFol5b4eyUgpyh4Ynuj34JRDVmFWYw5910dhR5oyjWZqsACKjB2GRkkxJtnEWMDL6llWM0AE4Ol+QPx29Ycoxar9l3nOYFQyoPq5767Gc6ZjKInmvUNqJSsz8i41KSjo76WRCP8BNmaOF8wAngml+X//aC7d9Qr5lIZnpZjuTyLpfRmMV2SAPGSOXBCztx5ARssYtetgjuXJLYqL6IoqOLRQCAiRKqC/SehJAkgIjwxXcFeSQstPngl8tBudolZAXsWm+V9dMe23SPxrMWjlkdn8lx1qzTtBQOzF7yjeo0nw96x/V6c29n8082P7u9ubVXrf/Z979yf34Afw2jVjc7RgIFCHS+RJYS6FjGJWXqsTIEfFiVam3WgzQfUW7oUnGpG3HaNAaMqCJUA0hAlOxoMwOJ18BPK9ARdiWeqMOL73/NgIM+kF7PR5FF7ddf/PbDy/qsCqLDuICmWJ0TRlavil/Xx8UjAT45H7neujPQFWnW0xeR64H17y1/RiMzEPQofE1DwZCdsHWmRjJ0kbGsSWJSyV2HhQaiLBopJEJzpY+1xoh1kKCFOLKxLiY2iEYFGZ5U0kyrWdgzpqxViyVm16ZMZ2VYqhI0SKhmyeu0ate2N9inB1PwYe+k5XeuP3fj1nPP3XruZrVR6fZPmU9lBySUhjLrhmaQSy3z5RKi2YjScMA4x/YA4vecG5bjdwo5wxkKmIN+BNjFGJaJVmwmOA2fbMrpS8yRskgVsQEILD2J5Eltft1sGY1XJ3ZSNZoWo9+zDQf7bZXlPMyukFN5suriZAijU1AAdAz8xKSLQIEa9Uo7jG5WtAf18eHJu2++8+7P3zo9OBkeDDhh6OnJu+xuCYPEwWh/Z3dve3tru/XJT70ChUQ7x6trDPvWhnlwIJleD7qHUvLlKgKRXWBNDJl8pT1sb5nNjoej/ngKO5cZCgoxzaDvYiA2/0od5e/NKe5Z3p0My422n9Twg66Vw7jGKcIsLemPZEEfxA2cvbFrjru9081Krblf/tQXPzIf9g5/+ng0n2CSZEdkjrGI2IQAiz/z7czjQ21li/TxxcbXnApgvU9ZizIr1QPoTGNr+2ZUjt+5h2s0nlYynGAapmK4pjODjyQP/khUB6OYbeNUQM4XlnMTUpGkJZqvBSwgqi6+AgW4RFuJRIdAHWHuchHlGK6Zb8WtLKp4SYWmofImqPg5E7Fx3eeouLhMQSA2n0PYcbSbB7Uo3vBgwAtmUuQJ4Icn7BS+8I41Qc9AnDHDuufNNzzMs7ONOGzP2RKL/avlKSCNN4xQuJnrgKtBSWFv2FoYwJA4KCLn9k44+MOPZjbTi/9gmc3GYbGIrNrPMGQnGhwQQGH0Ks2FizyKdMmSwJ+NKbFKSAdDjwYjgrF3CFYPzDpy6pFAKuGZPtfEIvtpc7oh5/PMcdEb4vzgYzNnzxGOz2anJw1qrRXATg4+IAfBpJDqYFr6Ew3gHAJRc9hXwqnSWPZhz0Qgs6ICJyybj8Z8Nwt26s1S43qbQhB1JsPBk6NZ6ZA97VD8WL7DJmPmMgmI2V0k9Ics8DI/SHUiuWNg5oAt4xMClhVJK0V73ISusQc0XlLbHZzR3IMUDJF1MSf63kiTZQB0qD0vFSkGSTIjoe5OGmVl9/Wv3Lfv/8637nOMB+wuQEcy208F8ban+1tMVGEzK7O2j2mqGRIW0vVgNOv3TthsrtSofv7Gy3v/uFUN43/77S8/WhwzcKp+gxXUpjVpN1Jkq0S+7YhctJiLu2CyvKsFhN0re792MwYMVbYt4QRjetMujR8+EjqC98hEFAf8AAbR/PwGLg1E6wwH5d9Ajr9GFq4mru3uTmaKFJO7GnYaYYYfrlh9JcgIbufB7T4X4Gxk6tcusQcDwArPKG2VoUaxoewyKVkrd7G/VSaovyLbujOW2TRKpiPJWE62sukYCawMAJdIGWpgQn3lKKQFdBBdLTRBTZYsX/cmW2m/lczLNfZIwskDNXg2GYyz6QAnLOy4oZ/J2zeudJjerMZs8s9+WDg1b+9u3b57e//6PtwXt8vhaITtl9lTzYiVE7a5olzDeaQGiBRrPPB15hCk2ZhpGjxlcJlIZyW8InBOZjXMmHU0xLBQCCrJIgJYRQTH5Q+WDAOWIRq1Ek0XQq7jadAGZRVAaZBGBWHTuW+UgYYBV5wwOQhgGeMwIEvOnlTikQIn2pXsg4gDOE8DZLiIuCQ0f2dz7/bLz9995YU3X33j56++hndW7+DxvNedAx5OS5x7bw27D+79HKUDqyjnESFG4NdNrixigdVQG4grC41QC+k7OXqxA0TArsIpBHZnZ6dcbzEVra2txymRcOZWC48waitTmSgfzUNmgo8nmMnmp6NuPh22d/Zuv3Dz+p07T46OJnMU6Z5MwTn7aBzuxeUGHkPZ8N4v3rm5cePjf/Cx18Lo/g8eMHfJNnsAE4Os9BOWM+fHCG0gDDArNzbjcpMFxyg/AAn388mQEydZr7TRYZNttuQaPeG8JTzd2WiaDbfRFqRQi/wgouL8opOiYQzGenHJgFoYIpqrHhkCEEBNKyAphqC0yult4Cbb4M9DtlFUBRcw4JDFrmXm3Ilixp5l0q2gjM9qU1tewYewBkx0ZhSOySgiXrjp+TBaTl6mg09m3gHW7Dw4hdFyUpSX7nkpKnKTpZ+lsBn4NWyEhokJTnbT0TQImWSR7xVapoabrIgaIEiF7FgxzXtlplGlGGpdEpVnyRhWfzggi7nxtgL1NCE+mzD8TLwA32kpVICM2FADtsrIQ7hDeGDtNkMTI/oUlww0dVJJXoQLkJK9FTXt3WfTLp99rdm6Kx+xYi4ucaYIk8QIDI724tXAF1pkTBnCJrEvxrT4OpMBWjEmGZLN6LDkZ1o1LaGz3oStVOLjJ48wN/Rnpbacc2NZP6Jyq9U5egQwMj7Bb9icQeRlT+drZgGRTLRP5nFZmGkmQhsgAptVLn/qWfuly/W4jHEqssxy6mlQDlFPrEP4JgLFhfVf3yIo8EycXrtmEvoNXCLGupa1co9i8hhzKFpbbg/6fUCGuDyuTE/bmzfmXsLQGMzHzKmzuS3SOkaJoJfOJ0d3Wq3/wx/9F3h1/Juv//mrB2+D7lpqTT9glAIB5NIlboFF3VrIwOdyZL+4u4BeuBlruopLzomuF6kiw8M0BkaTFo1AAeViBikA4GIVcHWKEd3/TV3qA7sIFOHfVOYfKJ9lDy27bfnp+6mSS+M+pzFA01Bs2S7XuAs1Wc+2CIsErKBhWEmHulz1tYWVsUYEFIze0FAXDdRbJTCCqBvsigjj4H6AZyPEUoImH8KZoVSsUWdQ4Y+hHXQ5F4+VhTn2Kqho00tf2Ig64ZyRi57G1rH96cls8jSbHAb+kH03MBLzptao44TLsJeaU/KvX7v+oQ+/ePvOrWq9hh7IKtJqrYySilcIhdIoWeggM9o2MmczfazB+NhUfM4DZvELOzJhtsURGQsh+pBgIKy25Uma883YJcDHOQnOiVvIFCaHSxfTeOJOkBTjvqCmNGxBZMwh6bNZksR1DNDsaANhy0+DMec0Yd+t4LcDReArUUlILtwYaZi6QUFtXEDXoLLsFxS36iy34pSi9s7GR8of29ndevvazqO33+w9eCsbdHvdEedDNJttADsZDCbDEeCWs/dsigkWJQh42280xdyJUMDBdVyQZwgzHe15h0ddxvPe9VsQheOj7r179wa9XsQCXI53R7NUB5tKLBpI1WBLFeZiaTJm4cOTx17F7w1wVB3fP3iXrUrgm4+fvr2x6W12bnnxopcevfV4dPeVz3w4LB+iZb89JQs2YWD1FJZzmo+vEmgjAw+uR96MPUD4DattTq1iZgKg4CvH5DGa6N7tV7LRFjbS46MHx6fd0agnN3M4oNyVTdahtoiB1FIYJx1XRkgmgBWg4mK64J96R8QbYsv5VFin8V1nUxVOz12ENb9SD+I6Ts45UoC20EDIwwSMQZW9kpMtr7bphfgZUVO66yBgtS3yDbPM4GxQJ2uawiapaV41uyCwY8rjcLZ4FHobnt+WngX/1jIBAIUsCHmTe6EWVmG1ZnGSlmxpfCDUsbTXdPMJfsD+opugYaOC4+KAxYL9XlBNwRwIeJJzOBereEZs2YZAAWsSDKgibngcBzDFxYwcSYwRg/SyStvMOwYsQ13YqvwsgjnyBMo3675GeEByDDRnkSy8MdjEN1jz2W2RxjG8YbPqLuf2i1UEY7EZ/WHvoDRui/AWPLQkYGHNx8mRQzjCRljGXFJexM0w2ch6OLixIyxShyff35nmSOgLnTqKqwj7Zsu/l7GEFAW/lTuh+ouKAjQGMnJtiE3b2cZFrBwL0YhFfAcjFKfhq1kKehu7gJE07kAX7Fjd1RDGK2mBkbsAHKIbhdNO4dGKEtqg0VhYJXy/v7Bal8l6VgIjY12cU2M/y8ZpztrnUXVyylDCOYtJfXCaHtDq7RC6xPY+1cnxkL2j929u/8vP/+H2tZ1/89d/9tVXv0MHsY8AKIO0iAkIDYIdT41Bnue+Z028WHPBR+uA+QiU4o/ZFSbG6hU2vocmQln7fbYlOMXp3QYW8yFubvEDw+JiyfYMXC6DVcD64LC+Mv9nRcoMYn8gBWHuy0fwm25xddPdgnr9rJwsETVewxVLKgRaXuegb8jmcEukmKxVRAEEBEJXGK/EQR0cLL3S2B9dxishtv2HBTlpSPlh+OOiS/meKtjgyJhAMOMinzNdgYsw1jaMycyKcaIp5sjEzyvBnA2rOOuADR8bvn+t6pUh0aU+7kKT0Wk2OJqPjnHfD0s65gh3TGbBmu02Mhp6a4Udrxrla7ef29vb5fgXVkqgn2I0YyaPAYCvlE15ygCl0SXfZUgT+jQH0o8m+JFoFSPawhRqjm/zLMWqg22Q2V8i1QY5cSEApiN2bcJ3hvXGM87uBCHZDxnNC5YOdk6niIkin3zAFJ8ZokViYe8+2jNb8OHXjbdWKZjAHqAhKLicM4DnDKrHHAlA036QTQmYqG+MPAg1xnP8izilYCinothrdJqd7c6o2+ren7FRJStUSNkf9KCIzNzgdQWnp8mOoLCIGREKbl+t15lXgm5TZwYUxgCIKqwKAv7k8KjM3g3V5s4Wyw5vxEntZz/72WDYa8nzm8GogaA5K8y0VLNU4vwn9kBieQmbVP7ox98LXn+VNKhO/eFxjNtYPenUkd+Hg9PD+nbnzodufvebP9gd3qw9f+PTf/L5t75+//StPouHxkN4BRydlsKy2XsBAQQrRtqaZ6hDHmwIs7HHIh9OhPQ5cQ9zHRsGMcGKa1FH679+lrEXhlQ2UU9xWxQb/cHj4LIwXZ0SxyPsVr9yctKmE3BiGWJ0h/tKaUaF1EofpncbYdTwWBWFXY9D4bS/J1Y5tu2khuLe5QBNN4IBt3XO4HzsYxqdduVpNU+TucRLrfqlN9LdhbeVLxph0AOdFjknX8CqHwbRtvaKLGXMXMMcYZP4mzPnpxqzSpiumiHYgJuUR2cCHsGecQN/YvFKRG6MUx34gUzJcdiwH7FYZAGYLv5MtEnDESwNMiZLpIbCUPU3kT3XKKxWdjBs7ZImiG6pfTw5B2zEyiisJ+xyg+mHVdzInZobYNWWoIhCK/FIhIEuQyjT4MbEBvfgWR0AsophgjBQM1YZS+zDkIFQ0pqXGtliY5ZXZ14No+nG3s1FfZSeTKf9MVPT0wm2kyej4UG+AAG0q472qJR8A9vFTZH6yoyEkIQ1gJVnGiv0JRq69TkiBVWCP6sSIncMX60D0iwmUgDUxkirACk5GUlBbSgu6glURXHJBMkCMiYVm0EFjpBY0DpLbKNKkt1Vl9G5K16QLaUbm9dbsuNPHB8M0Do6s3ZRCvg/Qd7n0FQW37GZXVaubsLq6MY8S3BChWwghbMP+On9x7U7e//oI59hqqSeVH/81mv3Th8de8dgvzgjnUpx2kjbJGZQSS1eCRiSKdQip5yJ8xg8aCkQozNlzSMfpvGbrfL2dofdi2gAotto3Buz3sSm//U1BHGVp5r1v6CL9tIa8Gm9TeDXezPg9cQuzPdnuHP5tWIuvl8vcoXQSxQEh9y4XWYu5ksSh7viOTZFI4yVBVBEDqzmE1JBvQmA2Y5H08XQg1kyTyt5Vs6nNS/jQN9W6LWSUrscNSsR5+ZB3GaTI8xfzPtxKsiwfzyedBmtmGnZjQrHHFgjB/Si6TL7B6XpNCobm+2tnQ4yNycTVZg0jjy0VFQkloiA6BqOPsxPi47Ywhn6MkSEx7s5w86G1RV1UToDqgBcDzaFUsIaDFinRAW1VC1AXeMbpmhwwdLEMKfh8FiSawJaLZeYlKkIpIChwrT0oYzPzBZPwW80bZFR5cWaHWiZhBXNBOUzjhpkFyAWeKINyfEL9sz5gZXKaNhn7jZOGoyutDfiOPV33/nF22+8Nur2qyUy1IWDF15UdN90gpPzkeRXtqnCdQ2ijkwgN222qAon0y4LqFCyAQUDGwqDGIDTde940D95fXxz8sorr9y4cfP4+PTwMStl2BFYXUbLxX1hRsomYnUNQKNROiGBjZzYzxC44Yzj+d3Dgwobjr1wk508jh49QFSqtpt+pfTqL17/3Y/sNf/oCx8KXv927/s0M5KD0TzWxtUgCx3Qo5tF4r0cGQAzA6QpL1UCPNmYz2JhEDTpaAD3YoPDEqttfE6D0KJUfz7Ssl05BAA1m/TFWZxZRsJovFJ8RYwI4P5GAJlGbAtSIxoiV2ct7cIxgNWqWJfr0ta8BIdeUV8WnVI/s7xJdQXj8Lj3/Do4jkKC3URdhxg3HwezHucMefhyY5GOt/3pdhBs4UUcLU6Bmz87zb2DYHbgeS0vGDCnW/LGEHjsPZIx0BXlmIyMinUF2ilxnDciA5JCZF/Gl45tKXBQR8Wh5hpm4lGpNEasCRjrQVNnN+VDeVbYXDK8SvolW9Mw0sW+IrzG6E4mjVnXnRmU+TaF57Hp2BBhQhutx1hFBzojCzsmYARfzb9BDoUSXETmASOCCzQBXMIGIeotn2XqjM7JRLx265T7JMm1IKy2yBverI1qa8fipvEizrIuggXTTxK/hk8n06M5/tx2ygSnHOOfBivQnHSOZGzmLVwX8HTXEJXlDCrDEmLHd42FkEpatIzW+sPRz///E/dfX7Jl+X3YGZkRGT7Sm+vvLV9t0AZoNIGGI0AQIAUZUhQpLS2JY5ZmzTzOWvMXzNs8zdPMy8xIGmmWKGm4KEOKIimAAEkADd8O3Y02VV3m1rV500dERmakmc93n8ysW9VVBLBErTl1K/LEiXP22fu3f/vnf78NuwjwoEJ3D4GqKNcHP3U9mnrAA9ej9xQVAjLFZ1y+WMChk89/frCNP+Xb8w/q49XdolIMCUnxUp4BuFw7T6ECG4g+fGjDld3VtReWl24rkZLwGU6Rw/1amwl/dnJ6On7vcW0y+Oz1l176Wy/8l3//v/nD73zj2zvMcSKkmELQWcJ4cdxHLTJnRlM+C+u96kB1Apuc6IPEtCGlBjVBFiF9p7t2997G3bt3/fzbv/3bo/GuRaU8qlplqKqgxMhglS36g02SKz544erb+4N3CVwqcFQM7+qmCl6+RlX6qKOSgNxW3ZmJL2Ct7Bs//ER128ddrx6/uuf5r1ftV8/CL+jyw0fV/0Iq3/+xIGURC0uLWco5wh7iskF3CVTWVhAvNDpSTqrXl7suPq3yfAW19MT/zx1ZWmEwsZ1hvTwzkazwY1oS7gK2Kjm2bLxDnE3qbUXzz+2xc34qraAlR3B6tDR7utKeXWu3FqVoNGfnFV+SxCb7Yn/C/Sdcd6pS8tHQHjvj8e4Jj0hTMgxC2uT5FBzT7ne7i4sLq4tqXDS6bVUnceSFxXl2abBCAnq9DkE6nFOHkCz9FLiRjsnnxUh4d8NasRUk5eiIrY/RldEqhAf5tdev4XtcEqOo4r3TffvE8AYLcCKHgiWWwylCP6cRH0+iAafYviOGg/PD40PIrBIgCR8vRFbJ7+PxyE8IR6dnX4ZFI/HVRHQplZ12PLhRh4vkwjcUoUe0U6/HeS0sdXvz8bsPv/EHf/TGt76r+NagkawlQzGBeD/Hqv6Af6u36HWU3uwMaEdAx5gOtS9+Z9Ab6LHYLBUQMXieHTZgux3QMz3y9METi2tRWuK1m9dXlt/+5tdZZ4McmVGSRLCliBSp/av+hneQj7BGKIKpebsKIiKO2+f1EdP43ubCYn93/6DV6zx+yBD94MWbn+n89I//zMLt7375u1/79T9Qtvtwb3R0cihgmyluejIcHbDKzjB3LK3f6a9SoFo2H0huscAojk/Yxhm4e1jvLPzIp7+Azrz36DtYi4GKV7cvnu2EpcFMJkMsty/LmTkXt6IB6XgxZpoNihSQIt/cjRypx8Km5OvYAXFxrj4QUGVDA/TbcG26C58FcVlJ2Wqj0cJfhVnRCRH5llj2cJPZTmRJTlO+y1PoOS/JvNbfaE42jg+XT092EjQ9c1i3997xs8ODt9qNxZnOpNZgrthTbTOJOB02Frh5JPAucVOcDTwj4X1gDcXgH7bDYzKWak6NryuaEToQVVdIwVT79Hjoyw6eZ0rJOGyT6oTfqug42WvJngpFtEJ1NRGCx4cjVclmxgeN2L2l8IoIOxb3LyogAQNy3BNtk4XvyC6OVm+sCNooMZamO3RDx61vK8vLCBGdODRjNpBvzcpO0m1OwoLrne5Se+FubWZBwbfp+NlkvGM9LS60bH6yufmD5uweN8jx+FGDpi5J4Wiil/H/EKxjN6MUFtcjAtaSx3SmsB3mIkB/azpUynFpaeX6zRvtVnd3W5b7zslocm1xGV7xEdm4m5takVcLRKRiYImVFcqI0HmDIfpmvfoJCTdkJ1EWrNQMUOZaAUH5cL+/PiH/1bmVe3HOr2INhOGVozAXP+UofCSmwhChi9/xXDOBgQFsMuB0g3KZSDrNW1DT4b5Vx+R2tMQc3V4SAKps6fn5gW2ukUCMb7p/2OCkac39737l3/nZH/sLf+/X/8ff+PbvHSTTy0weJLb+iGtkjiZABBRK4sXOI5RHK7gYS7AiXYrspjQ8c3eQEeoENrMsbM2V1ZRWWFqe7/fbrNCwJfIh01BKxpAH/5xHebFnqh48f/LnbOjDt6fBS+B++Ld/Rd+9ImD7oeMjL1f89ofuzQVcIeywHMG3ImDm04r74Auqb2EoWWbmKEcIWDnf2zvAzGJgFRock4rfSkkfCUURVgnh9pZOSQJhunYhFVEzczKOPFyvLc6eL3TrUnTXFYacHnZqR6lsdXSEfqp6GBsUi0ztSI324dH++Gj/5IxfCpdPPFGz27e9ILNqs9dmSsV9e4t9taU4LERsyfaxnQATUrSy9JtuySkVCqSzqIiRhlxJeu/1cGSeWr8mO8hDkJAxTsnAqe1d4WOqDwZOxFQxhhR3BLywaHFmCTn1Cqs3dqTMfZyPEULAHpSszVyE7hA+hlDXiylAfDKrEJnHr4P5BIfF/VV2M0Qfs0sO0d7qIxqTL2ezpQEuujs6fvP7b3z3G99+7wdv83iuWRKYEjLMg8S4IHyoqL863++qaNFUC2kqwEfZC8ZIFIMxNHv0lS7IJhBdG/k7fRBPJstSUOnhzGR0MN7b3st2rnFQGQ4TQAzxSVJOfJnhMJJyc9YSvzwc+aYcg70ossLPGyuDRZrPo/ceDxYaq6vrqlWP9/Yw+/Pz/fsP3nvx7qPazc/XXphZ2h5/od37xj/+zYWeoqInwqr1ft4+j7X6cP8Ze4UtmmhUc/Oncz2qpG6y7ZN3iPbSrs86c63llZuvvfo5qt79B9+ZHA9b3UVxTMzmrfbZ/PwC5iimrNNTWClUtfzLLBb0NlWIDCIHPMf2LJZeO9u0TZZ9jsO/in4Xo27UPUsiqh75QBS+0s2oGJ0q4YKQXepPzfZUjc7JZPdkumPXZBX2Z0VjnUq+WZg218+Odk6nw7pYeoS1tjedPJg9HLZou9P987N9ZpPDIW36RPnr2plaaiwypiq4yfmsu3WF/LPq/MOCrIiZqfw4TQl/YJxQZzy+UfZCc5p/5tJTQfUs2NiflfOcUQDrZAhNYoryLhgOlCeH5ycjr1cCk0BhT6RcOSMvpuQ6I0Sk+SK36IwGM+UaZw+wKpwBaeHBieKYac+cWXKL2V1ghhQ7VoNaA6RYir/W1NBmaE88onpw9tNqz5OzbHHFApOsw5O9yfGDo6MHJ8dbzdMpbZp04x/whi2yGp8x28t5boyfDZ/tSpOt9dfmuYh/sLN5vtT+3Je++MWf+NLNm7cl6hAl3/neW1TBr3/595f67eVBXyT/aFfS7LGa4xZAUdbgAEKdhVkd0RsCNiwnI4uxOFQvHyF27994+YDRw6cCap/lnihyxb79/j1/ljMCYnlLgiWxQcwbXE0T6hVj/lnj8PDpzhZT9HFt6bDbW+MN4aAjaAkvmFXj5FSZ1+RLkye/cPfVW3/7xvpvXP9//JO/c3Q6nBksnh881UfiUmFLlr3JQ08ST/pc38pcZrAZc8SsOJmCRxF2pS+icMlo7DTv3rv55MmTMeoyUsNFJEI9Ef556s9z6H+Brmd++OSHG/oo4H/4rtLk+81++Oc/7Xs1guc/f/iJ50YZcvnDR8GHH76cBfnc8f63ivv6qQgxucUAQm2jGuTrFd4F0arbKliYQP8iL4J9+TG4hynmiYKCkQ1NJG2lRLogBIi9ChmT7tHZ0unRYr220O+vDTqU3U7tuHt+3K0zq+FmJF8VlpSUGvMFZtMyIZjnM7YKOFDYyt47Kr8Tq0UTzZ4JdVYjso0Bt1s56bZUV15YmY/rV4la3puijYeSYYOiNjiwimaq98nqjbEnGirNkg2ZGkE19gT27AHW5WAhdhhrWpy3IMM6FbGXXQcfo+doza4GrDSEVfI53gC34WOOGK4wbDqjF+UH9A6oUDA9Cy8ouaY8O5Oj0eyQD7A3GIAlVVV4BTrK6Mj+lQZZOSUNtWYn+8PNB482n+784Lvff+f7b2HXL91+4fr6+vXlZZu97dI1d3cJqTixHZ0U0FREkxoiBtcgFKfS03ji6W7Gk5jS7E6jMG+0JtUZsP/IEEIydVRhE6HQ+7F9C9VNBm0mk0FAJjbyCAgEK+OjrkcLATaSTmo8RkkbcDR3B6gqQbnRnF9Tk2lyhuA+ebwrwmnz0ePvfefbr954vfbStfXTs/WbN6bjyZPvvL398CHO256tq/47ezJUFnEy3MM+jmVGntQWVUmzC6S4Gnv6NLsEhqOxnKnJYKF77foLDNSsCF//5m8peShjtWiMnIjK24NrwVC4QHXD2GOINjsmCVwVg8QQlKk6Et/X7ApkLlX1vEaxipAIBJC0onq2eAZstRV7q+riYUMez7PhzPj8XL9xPt8+f3Yw3TseP22c9IQc2P3SttL9mRtHZ1vHw2czp+Oww5pCYE/rY6IG7q+cKrPwCZurwBZp27LrzmojK0sHI7adC7RmKxceDyNAGU6yO54JDhSFYMrEKKRwEiRxV9DaKo0emvXpf5FlXMQEJvtikYmnYnNKlYzUqqPl2ozo8Fz68NG+RRojjkh+tVeJaYaPUJvOwoCDyZq2iNMsST2IrIEAqFAlVl429JowvPpCrbvmcv18b2ZivZA5LLFjJpgWUTAWhTFWDfulUs9KNjsakUJa5wf1072jw83p0Wb9ZD8RYAwJBK3QmEgSuC8TFfFlmxWsds6ycDA9eTDeN+ru7cWf//f+xid+/EdfePFVWqTdQ2uLq5/+wo9+4qe/NLi+8cYffeOt7765zOu+3GfdJRWQHsoReFWKb4ZWDmhRkbOMydv8AYLCri/u+OCfigG7FliXz0CnaDIfvPFf9g0dqwhDgWUG6e746rIQE51B/juW9K4663FcA32GosHprIJotU62/kg9vZNstcEtBY1mZj/5ytJ/8Ct/XZ2Av/vb/3hH0YL+0rm4kGAzEijnTN1tlFEKg4UfIuT6FQSC1hgwcOtCODp1fnqErGC6jo2NjfW1a3fu3Hn44MnhUKBsaCjd/GPHp/GPOgLowmyqH6uvH3Xjn/XaVQsXLQczL2alaqKaoY9r7qN7+TF3Z6V8sLmrt3/ME1nJ1Tr50A2aKceltaTApCz7ggXPDaHovWVSShOe0o1LvC0aZBoKNYPCoXg0JrxGBSgxtAj26anCi4I+hSsvztY/vXF9uTGz1Ovx8rYEOk72ldE4owUk7GJ0dHzA84Rl2NrXrrQolqRfC/ecew89YIGBQHZWF3cw6AzmZd90VY5YWOzNLzB82pEFMcFr0YhwGghMkAzvjLsuRxTRMJATCRWJDXZQNNTjUegKW8mwgpHchfypsGtyPCYUK4ZkqXg0BqJyEOeZsSOpCo5RITr6hLNQi2Ia0IsQKxyNLFDcSnyZ3o9gSwPhh0SYIkT7ZDaeVTYy/h8WZfFcidemz0qox64VEKrPNNUvevTg/ltvvfPk8eZwdw9tunvj1t1bt+PKFnc7WOiob9kfPHv2jEpK4ZAzoD8TrJSaQEJQ7COMllkhC0ereLNxuR49KfJTQBFSl/p/sY5YniQzT42OzqdsmdJ/o8BEI1aUwX3J5+p1vD0xZYSMGPHUn6AnNwEKHxAyJd5se595eWrZspU18azJ8dtvfu/2rW92PvfTtReXjyYPX//ln5nr9HZPJqO9fZUtwDQqKOKgwNjkgKmQ8sTX0OnZ533AyUBais7Pq6ma/fDIvhbr6/dW1tfGR5OnW2/sjx4vLMz3Bz1B8sSiLgctkwnum3+ZCoIHSBhHvIEzIq2nBJVm77zZlXFkK6ezGnjzFob2GVA4t86EFl7IT6ADSUxmKjj6PfyYfDDTbx/3ppM9PoLzox3hqJT6Wo/dWpGMhdMxx7xh0UdtDcvC7O2W0LRJfDMLZLfTGuxHRdkYqdyMz1GCkx1vufMKEkRzJqSGC0Z4XULVTpqpvikrt0hABmUKoRP0KiJnGCaXAA042UTUqrOmFUS7gchho1mew5lThnFhTIbIYYRh8xlTWMHK/ToaERUoGHMK1me5ZJUUSdLQLTDfZDgxSEk9PbFJ9XnX7EVE0QsQBGNhUrWD2Zn9Wm2LgYOZPXr6+VGz9vh4/Lg22j4bP60dPzufbJ8d7abEqMjrYGV5PDIh7svMn/0eDnVo0Jaj/ezp8bNJ7aVPrH/h53/2Z/7aLzeWFuwiLYguGjaTmfjGl2//8v/hf7v63/+jXz/++5tvvWPOVxb7Ahfw4Dj0dSDYHh7vsCyLvJU5oSpWBDFL20gs4/J77vvg4annL1QryOcHrj53x4fuv/rF4iNYAWYgXLhi9ZOS9a3mOTMeqdGM8Wttbx8fjHYX1k9avZV+b1mcTBATejAkTOpIpYCUJ9PDtbtr//u/9R/gzP/ff/Y/ypI7neEBNO+mGFyTod4CqEipQZPLbvjx4pwvhDElGF9n/7IDy+4Q3UEiX3qJbYSBUABNh2ClaQ5MaHfZxP/cv5nyjzo+zgdc3XsF94APFB1lIM4riH8c3D/qVX+ma/rpNdW78omoV+/9qKdLX6L8ZX6LZJC7dM3i8F/+5sjqTSu5y5oxGzkKhpV7qu+5F36VQeaK24Kj1m5ijuL8E8+ZYuy201FYwDbs02nj+EhC+UK7dWNx4cbqml1tX16Yb03VVj8+Px7bfuwcKWEaq88MuU/Ljr5ik1XK9yqrCfrzrVrnwp0xTC9VrVaYUdkJfH5lKXuCdxc6q6sLvUVqny4dx68ZS29JZwvdpAWhuyIT+DkNKrV5xVYhQzQ8lIfuqllKNMsZXop5BDzFUh0GKTSLGCAbhbxX4BxMLqHQJsLN1cHvgw07j+M79roAL7z44jNwzmCAuWiL2jgYHXJpQnSSC58rasCQTrywXy9XbrKQEworOMXWqFPFHW3J8M479xn8XV/uzC9KpzyfGe8f3H/7nf5CX9kvNoDF1RVyx96eUKZjhnoD5frSeRwL3yUkyFSoYQEJwLUyYz83BjeYd53jwXZfCfrCiOIpZL5Mako5Qo0y6zHP8gcWj/ksHXo8mWSrgBmT0ltbW+daFLV0KKH6bLogQrnZHB8ebT58Ol/vHOw9aYm03d397je/8rnr12q3X2issQ83XvrJLzAof+U3fvPx4yf3BjbdaR49E+WOFKh5NRzvbXLdMc+3V9dr8ytmKaHitmVu9eTgHI/JVvW5/tJP/uQvfvXrre+9qXzXaGGRaUT+EE3p0HaQoJ1Jy4jNrDnAWwgSrM2iuM7r3fNW/hWTQYt6EcHJ/SCAjZSVU1DepaATWHFDjEs7NODwm6wtBavrS4OmmkW2k5M2fqTgREp2tKY8xbbgOFE5huMSE6WAZk6YXhO2jCti+GQtOk5ilbVULTGdsAAsOPbmwhQiWiZUyxzIsE/goPx5yFAZVbOqQwwS2ZFL/svDCK2XOSczxLXnFhlcZA/1T6jgiXZOsSRCZ6KgWWCcZDgVNdBoWeAxcGAD6Vm0c0caLPbniNzZskmm+fxprScSHLh5TrJxCbqg/cbB+fFjrJftPizgpC3KpzbZPx9tD7ffO97emVFE/WBrerhzdnyQ2EM9mtoWl8E9RgYpUAKvAUtiYnu+v3d89u72wahTe+0zd37653/x01/88YWba4qNMaYrYs49vL8/tH4H7b7tp37sb/4b12/f+I2/+9+98ftfmUz2l1jGmjPdCGAx74JSTPLVEWLobUBW/gV2ZYAx3jr7iKPiF+AUULm3OkEMLfur20sjaaA0m8vv/3ZxE4CmBdMah2sgrK30JP4p0rD9zhOzSTAQnhI3HFlvYcT+1lGUmxVPGCARBTqfziytrmwfjfffun/n5Rv/0a/8DfrLP/ytX9+vzQ0ZISyzRpfERkJJby6Jd9UJL7/qFwEFHZUEYuTe1WKke/L4Gdqxs3PQ7w1GI91A4WhWIWcxan/MUcHlh38MFzFCE1CgdnXyw3f+qVeqpqrbqtf5LNMBfFcjCixzT/XDBxstK+ODl/7M367eWNq+GNGHngbp5/rxoR/z9aK76bZ7y5XLrvtbnUYKLHde3eR6dR5+zdBU1ICIlu6ENKKOj47s17Labt9YXrrR791YmL+5tLBqgxtkdTxk0DR1Qhm1gmHIXVGThw6q0E0dYxZ+ZzXA49hDT2k2hDwZS0wnmLyKi4zMbHzisOg3EoCLuxdhPMJZ8Wvp/Cyl+CE2y2+YFVT+oWT6jONhKgo/YDwcuhEvY5wMh2KwJHsUmbdgf6I+QmareJXUgMeaEgHhcrpGYEhkQzhuZSjLkkygSBZzgSQxwAADvQSmQHrMDibrRfhBUccl0HMx0zjIB+op2IItHXFRBpWVyMk6HO/adnA8fvRkk4XZ+i6xYcIfEgDL966pLRHPO9sJSWu3iQ8ELkKDZsSDUCgQ4bZ3x8dspZqu40a7Rd/GGlnzaUMIJSKtP1nwFJxQcHymWAoKgzYAo0KhY9ywqV+863OT/QNu2mR62rTJwlS20nS2WisrYohP3ntywJzaXBxcu3PL3gGP3/jB7YX1+3tv1kZnnZXe9pP7D7//xzeuL9VfvX789QNpSy/MfMbo3vnDrym0qep+iwasS8gEH9R4b+epELHJ2vmkKxB0/npNXMDROLSyLlI592FrzeVrn/nMTyytdN9576v7wwfI1+KgZ7t4+zhllCFwBCJhJTgoPVhQV0k66jXmhFXZo7JNFU75Q7NkrjSO0WT5mG8iL01VrFQWE5zAuphSbCHQVDRTVRLlohOJct4XYQRLFXk+nu7LwFIfLDoZF2xWp8URXIsUmPh5eBMBFs4VY01iTmMV1bsony56EGrpBUTA+X3Bi/NIFqPFkcgDOrJLNLoMCRf1hkx/GGzu1/eSv+JVBE9d4y5mp+LtEUYo5UnIVQy+JdTAZyKwoI836Gr4PkWHAaCsbkjhrRqFmaVH6jfrKgwnEtgUqV+fXWg05rPjUxiJDsoFULFsPDsjsZXgp/zcSXu6B/kPx8+OD7bbTBijRyeHsm7IxIezuDKWnRy0+DdwX4ov2TiKHY5cmxnXTrb39kb12Y1XbnzxU59+7bOfeeETn1y5drNmh1FzfHqyO96zT0q9151TkLNWn+wObbx94xd+7t9dWf6NtZU/+NVff7izf6M3r1CPkTH7/DDtDQU0aGgX2F8cIXSXtPHyWv5WDLi6QuaobvvIO6t7TEZOykd1pfpECTXv1ZoIlcB8g6xnols4dHA7cFMHhpHOD2xWo71HppHFn+zVI2bwj5gk1FHO3N7e4lKfPWV8/8nnXrv1f/r3/tf2Nv1vf+83v/30LYKPXZfY61hQdIShI8r2D/eG4mF3KlaNEDIYlpQC+9WcPX60Ze8ZYq+1IJVDij6Ni/VFToQF85ENPT/ID5xfQOID13z5l8Duw7eW7883Uz1bzcGfrzOlqQg8mff3P8vpZUvP/RrkiGiS3l6+7qLnz/fn4vHyh0T2/tfqTdV1Nqj3D09foAY52eXyNYhQfshPlSToSs7ThXKgNvDF5OsF9mIraTkWrbnFTutau31XKsry0s1eb12EM6aCnQpOnRziGjTalJWPNjLDcUkknp0b1EVi8lfK6gmLS0UssY9WbFkO1hQ8lJmmGqXOsPcqMVHrZcMW1SMPRmhHSw7xPDExDlxWF0GUsabBaoyGCE8xZtKBYBwslbOTVlDHycNo876IovKJ4V7R6bPhCZab9OHENqdSdF0NI5ZYrpHJYcKeyybjBRAAAwyhUBeA8UogE6VAGS3vd10MExYYJiqxSGiSAGzRJbYnqDcm7IpxOavbFa6G8qHRk9Hh5rNnT55sHhyIYZ7wdJt9YnsMwoxyR8mEJvYebT54+uzZ+a7sgI64XKQZfcFYQSwlq44SUU2lQKVBoyyapFvFOE/vHjW4kGP8yprEQulLKiwhfamdIE5VSSOW03DjxIPFXs0QHakb5ykiw8b1m+vr1x48erK7tYOcL11bOZzsSyfpLcw3+h27MK4N1vfefXAyPOyip4fjk91tKtl7735n8b2N7us/0bx7rfbotL688Jkv/eRGd/Ctf/bbB28/atr2KDaVsCRQU7qfwYQ8Mn8yXVKVv9YhNylvfSrXqGMnnjaOdrg16qxff2mtu7zafOMHvzca3YfR7W6fZADJKkQOCcOkQl3lLqmfJqgZLeGXPGbmj5E4hK28tUL3ovNDDMRoTlzyjH/EzZGyVgn39kAsliKJBMLgakygQhrGfAeNc/sEjohuc+2EyggnZB7ktFD9W8pQ4e/kT72I3TdrzMJEiaF9oYpZi35lWAr6RuaDnxVelSXptQZUPt1YuC3CAHsTl4rGxqbqh0RkhOcaMGzQJu9BJpPxmy1EHbdMLFdt5NxyTibJitGj4JAlXZZ9UckYo+GGbiZIohisNRMtMksfS1a90+AGs+0lSD93VGe7OLPvAlewQmD2vLLHfEp2PMPFp4fPpuMdOjjrt9as8EgDcJ9AK+UtuysyJ10yYEud4MBrQ2zsnF27+8Jnf+anX/n8Z1tLy1bOWW/u+OCg3VEpolsSb2iKbQbsg/FwcX4w2d1vCxr+/Gf/Mjzotr/2W79ztLVnj0PDMJxCuSIxm0gzQJQwA44IOSCQsQdt/PeRh5VYXQd2jzv3Sd75yJs/7mLm3JoKQS2tZU4jvPhDtob56sGxAihfEIdEhHaFcQR2zOzCLRO4dK6UZ5s8nf3c6jtbm73a2cJiV1b1+ZP9126s/h///f/N0WBw8Ov/6N3HbzLtSRUzq2k9/wzRS9/vcBm9QA+DASHkAwO2wEQ0ks1GyPbJ6OCgcGhbe/XgTGqilZF/3PA+9nr11CXgKggGfT/ySHeKXPIxn8F8YyqfxhXpqXxaMc59+jVjvbxe/Vp9Vi8sZh3gBYuLT+PPeaDz4c9q4vXWkTVXuu3TkvGKDx3e4qpbwgxIsoUZZ6oJmCRNfSu/RYUL/uR55kQ3FJzS74KdkOTSTZKJw2b8XKBl26Dm0aSvUjdtb3rM8rza7b68cf3u6vJLK6src0Tis7btfabTtsjexCodr0vpRiWOuHuh1FRxqE7TFrUzQ2FWiQdMlLyoA/ZPabiMpiyrRzU6dUqCSwRJXBCqJVOIE2zuPEUW+VhjOlbBOMHJCSTB90qYkRoOeB/czuSoLBuOSmVU7i+1Moo1hvRAtCxpDohEom2Q4ZnJ+ZTZR3xrBAtYL5UYAaU0KvzXwbARLHutMU2XiB+dNk+MQABIDWHjCslPtGyBXyY1jBNtSSi23zGw896gnQJXdh3ITWyBCTvTAenPiJH4M5snHOzs727vj/dYeaU+21qmM5JawDrN9Khe1d4uLsKLbsMZLlkzwsquSKcV7eBnZqGqZ7NV0ob1aoazTpFYQWXtZlsv9CnghGbWGxi73aVs1xAFTe05CdZyHDQgxSp1daw63KMRLr5645pbdaPepcEO2sN9BvG5LommIVdfVNvazRW/vvfwweoLry6tbDz8xvc7STU+3Nne6jUbR9tPn7757XsLq7X+p077ZI3D5lJj4xMvKmjxdu2PD95+PDvTVNqJ3ZJ8n3A82zbshf8pbX39zicbazfDB3BBNomZGcXsFWI62p20VttLL//o5xeb33/zD/b27tf2p4gFcmIOMaMIGYSOlHRIUQH1okVWJasId4D0SC9dovDngt2FDgX5/VOhaDh3NqifM3mT+KyX+CRiZThPVa8wSMiAb6QoqemQpLk3OTpojE/aylufjDnxIIVp4selNMBlc24Z4o/RVguaZbmz+mksaywrn4EpsW/0ZKH3MR6WAyZBr0iNYRSh1+wXOg6zRXW7V6uZm0xsEJGrWJsp1mEk2JPALCZwrueU0dI+yYpByJ3Qg8/RuKCq9UFhDzIjAyEO6ZV/MSHEJsZRRExsnqpdMjM4rc2fntpoRAWTpfSbZHsyTrGROmsBnq7x/ThzUrqEqr1fPx9ODsZzyaVZSFMS32Ugyz86SLcB0uJkzlKOiyvlqD4nTn355q3Pf+YzL3z6M5319cb8oL24IDpRQlOvO687ycZX9yMRI+ans2Tnx9ppe7Ff2x8xItXurv3k3/535hZbv/p3/i4F3mhobWW2i0M/sj3Rx7DLUUhlvpT/LV6LoqLh1e9FSsoa9xVcKkpuvQRG/qQtvwQKf5bDGglJB1EteDXo5oyEpDpesihhGfpcauRBGKkIc6R4hNSVeGGpxqWeO4l6rbewdzg82j5eXBscPN2ajoYrX7jx1//iL377je89ffwWkUs9BG0Lh+PEEVcNRF5chlvZpXUdxRO/Xh1JaUAX9IeduX94kGF55vJfqKpfytUL0OV3UChHBaDqs1wuAAKYEvuaKxlzgaOBa6r6rK6kmfyaZxya9y2U/5LRBsAmpWCnFQ1gDjiajlzwwapPWUVh0Omtm9zpDpfyePlq90zXnPrPazxlHQFKeW1pQ+/KGig9zpabeYdGyTxpu+IvZR2WttN++d0frB3Os+iIQKRIYhGWa/xGXpVqrXkfYuozUm3pPJLStE18jF4xQ1rLCRnCXZLezQgcY6V/TMfQtn9ydv3g6IbSA/3e6uLatYXFW2zOywurvc7ssUT7U2NjVA2w2JTF08KlRJWct5Sv6mNDnD67B0NVlYY7B9u7Y//2GBuP8TcbcCc1VTDSyVAWB9xA9sYn68vr62tLw+HetfUlhWltRBvXKTigI8dnO5t73T58GU2BVVZMclpYdVFyTxPZ3BcYRf0OR8ZPqIeJYsnzHCqoqWgH2+1MQUVPA16M37JQ8Hm2pEbOMVEeiyxVLdm2rAprBIz0ayf4q0dUzvBUEmOT16Q4RhO0LB1zH4qmuB9yQc89pDbwRclJGcl11iOc8XR/PN0dyozqz/UePN69/+a7Q3SKVqtcEnPraOr6weRAzPPywjJeL41obX19ptk+erK5tbWp8zFgl2w/A11ZtEVFX+D43o6dDFVXMBZVr5vjg/0zNC3kVSJla2l1abiP8mWkknEp/cizQzt07tXBwub21o0b11aurT54+vjt9+4L85FKnDoLIT61J88eyOKV8IPgbtxY6Xebo+bcymAw2aGS29Ooc//dB6v9xWGtsbU/kpio7cP93d6z5uTN+qjX7b1yXr9+q86j/e7uzFLj+o99pnHe//bkK+PNPVvSwtmYcc8PafJihoebO93B2f5MY8G+s8sb2eie6jZrxwRllClOs6diY0+ku73w4svNBw/+5PGTby9QzIR9sxfwyzKZ2N2oWTs6PZhTlrh7Xu+RQ45YkkN4TLm1RbsHu0ISTZZ1ZZN5wImHn+SWYhkomMWUQGcClIoIcW3iVPWxvdKtMRGEp4rm84QyfjCVH50w0lBSsAbb00vGCiYHVaxqfkhLlNIqq1XAgdUYmMLRiAthyHJAccIQeY+7GaUuRuJZTBQdKeQvaBUMhmacgWqieBQn0HmiQRhumhIbpu5/vuDO+Bu6H7m6FlFUX+iPqIGVQHBMgKuoMXEJFQ0IV4jkzVWibPQUjvlGBGyfzS7Mzq2fnq9Opzbf2rCLRw/xHijsLFiOHD8VrnOy+1D62cnpHmJFXuSmEvYNu5AgMesWk+Umf/r88Gz2sEGU3xfyoIOywTq2MpyqiHb71U+98pnP33zlNfWiVUasS5pnlyKBq0TJpRsOEjMX2ZKAI5RRr0UbIRijybi32K0NZqePH87dGLz+V378G29+/cGvf/WFlTUl5aQVzMueHx+z/t9cvyYGMPIQjhPpJg6rkpFjlaAUaHOELvBCCUnWoYeZCRMVgS6cBJCBKHQ5LvWKSYfAln/+ZD4d1Wc5rb6xHkROComv/qQlzXiFWYNj3qJXbmZqFnghQJXxICVLlep5avaGqqi1N/hqI82zrOHWo72DGTXu5o63vvno85+4/nOvvPbt3/+dh7Wtm8t3H2y/dRQRD9cXVwp2cZ9ExCj8QQ+oHPpcHRnQVZeDfO8f1T1Xd77/w9VZgU6+BTXL4aS6WJ08f7264eM+qx59+LOsU4986HpQuHrdR33qccU704fiaQvaW1a6lvuzkIDHHx+lgbK0Mn1Vu2UwlSiQAfg/n3nS/ZfwKKSxSG2mMPIvpDLNVizPTpZmHmBzIU8U6Qb4Jd/ohgvWSNqydDFh8jrsC6ea8rxm2qBhsl+Z/ilC9aXTsy+sXHuh2725sX59ZWmpo6IyCjit7+6lPGIqJmgrQ9AUnUZElECP1JlTOUmlG7vZHR7sjfYEF4xt06vcVUKKE4FBlM/nzOlIrFb9nJ02+lxUr9mF/uDlF++ub0hOsYGC+OQwAkZLW+WwJydrMliDYM6m8pTAU9obPYqWXtDakiqCSxDdSGFt6aBVhfLgy+AQ2ZOJOVSLaB1RNPE7RE5JQimVgGCWXX5RJrSiSETshVmg/gljQZ4K7YrRN05lUxOIh6A6Qopp4dhFgO2LH9yQqjXxCDMv29lEgO8w1eiOhrZKCPGPN/f8fH5+XpIPfznzddabvdaFG6XO16Q3Id7Sj2NcdCcoAe7xKMk2himNL/YHLMGMY1eEa8pwqhxMVZhQtETUJS+sG5TgWFpeIlmHMiuGMFdfWVlaXE6829LS0vj40JNKfoZdKV7R7XqX7oXtKPewu/nyzdVPf/K1/qD1ne99e7izfX1pSVz0k61thRjsUUouof1hVycHtlCrz462J/vvimmqH3XYQ5v2VFpaXPvUSz/WW/yt//7X5kpNE+PBT5SibdUVbpqcjHeHs09UQOsis4urteZ8OkJqtPGiyHGOVCbGWTVLbiyt4p31Z1tf0cEOOLUG5kcpvU592pvvzszZ/ADZSTw4BLPq/CsTZIFablZnFqbWLFMszrzA4egC+E9RdcrKo/CgWhaPTh4qagz7JbHyguOlDazQnl3EX4psZqHoGWFrYb0mBTLkTWRg81+YX5wE5aKfNOojfclJboFcaCaqnK86V64GifN/msBBSJlhvaWR3HHh6yzRQTGHpTH0ICQ9WpuvucMjOEHWiLMo834OHchPButbedCMc+H4yqLCl9xsLrQ6K2czK0fTLjZ6PpelXUphYRJ8jfy244RU5m1kKV6hKCqczqLP9JNRgteJzsh0ZXkdj4/3Dk6EVj7bGz/dO+JIVyH62u1br37hx1794k9a4oLsZ5o9djAunKJ9mBhzb4YsW2SLOJuVnZ7HmUAB9xfMpyctiYfT9sb8j/6lnzq6v33/e+91bap9zT4vY5EkvbPZdx89WOz2QxZRQoHyaSPrMlCFVSHRsN5niHdAFmibqMJW3VJIcPVJqrm46uc/w5G+Vj32YM5yoaypUCHHxde81PUpEwsCAxNOZvnMEyC51SRFzg6WXsyCjiulkHpOJGSPHLhf+/nP/fgf/uEf/sZ3f39z+1mns3I43Z1VFnCyD6Eu0CgUM1jlZR8bVFV1xU0fOgpEPnQtXyMnlsE4d8/VbWClKV+rBqvr1dfnr6eFCiJp7M91ZCQ/fFSQ1GrhApevjfEjR4XoTjL15V+ZjYJKvl76G3JraYgcVm7WTtZJrgeEF7NVKdeZMmKmP7FeZcHGCAXUwRxyQDJeq39taSMk+IoHz81ID7S5Oh47020oBpC7qaInx63JtHt4vHJS25jt3Okt3llYXel1bm0s9TuNxUF/odPqWASITjYniP01MUjZCKBgbT4sDzbYMFZlpGxex0Y3Ptwb2gHv+IBN6tAOPtTfKDVYHO4XxdEsQDJ8Qio/hQwv2h8N1+trVCuBSuCQWgruKdZjYUxYrS++ZV2zJxf7Y6QOpCuqOD4XrqfdQg1seBfGltXLvN2UywiNM2JaILVXaKiOROoVjBjX1NGsYvVYMe902HCkmpxoHCHIu5zGBKxNB8qDKGZyoZ0py9Smc/7w4FoulYFcHKwqJdnTt9ne3trZ29tjaOX1xQ41gk2ijPX2nFpaeGTgKKnRFu41/G9ua2srt6l52WRLbHM4Y7RBCdKKBWoIpydUDTw78124X3T0EJUMu+ztN11cWOg1BuBmyLribgxYTtQEux2eNdvt4VA60ZhMdm19fX55aePWdQj0ePMxZf3atWsrKyvSn7759W8c7GyTUa6t2RVx9b0HHYNSD6Tf6hOjlHOnG5AGDBq/ll44MztdevSIRSMlvUGGnliPU7B2bb4/f++nWr/8J7/7lYffe7NjI/vW0hGsIJEI4D2fHE03RRsvnJwv0EWXFMaDCNQYQlGkIYaOBJ31+kvL9+bnm4dH7+H6h3LQu8yrdqQfi3Pu9Oo2pg6xCg+GRbEiF1SRX2QdIR0mqgTSm6/QdhQd9gRN/OT/lMaymqw+s5uoAdSQuTR4Ly6YcCWcKQ5YWIThBI1Pkv2SpecP2hStyUEoRi8KM63oRi7DoPJjTnNjeAEMrpDKp+mFvLnnA7Qm12s8MGHAGU7ajQzsjXS4zHbFxk1z2D/Fyj26EmYCw2MBySt0mqxgri37vAsw4Hki8GInAwA4b4DiX4UVE3VE5UnmOt7cm1Xti5LNxH6ydzJ9ejLdm52OZqaH8FJvIqTptS2OePPxBtKY1b43HO3YUotiJ/JWZauZd3Z2+JDO5xvy6pZu3n7pU5+9/dqrteXFWR4Qy9l+lRY4umUZOKJGoCoovBmkVBtigFVWGg2WicJeQwwxtqg6a/QHn/vST77YWv+H/+l//f1vfrs9o1RKQ+S/UEdm3MA9pDECj0cMvMAMxhagFaod+F0dbjIq95UjJ9Wqr77/mT/TygePqs0iA+WHjLIcISUGT7hHQiKHxZE/Gu9v7zw5lUY32EDBYGAQtBgqA23JS4+fvf7inX/jl/7qV7/77Z3apkpxhzQL5hNYQp4rs6p5owQ9Xz+WAQckH3V89NU0mV/8fzUAX3MeQAXPrq5XJ89fuXpPBYurr8+ffOxPl/B6/uar8+pdPrPm9SetfOwIcsNla1cneezqepbcRQPMFIWLu1AdZstUCA7xRDhPKbVS5PdciUocgRz9KfObk9xPSLYQRLckNYhHiz8ScZ+dHrfk0p037vTmX+gMXu+uvrqwdm9hbWHQmfSkNaCL7MMjxL41c95JlUg1nln85qTk4R96khqJ6uXaK3Yy5NJ3fnQ8lDyuoiNTM1pvZyG1X6m/YjFTtgN2+dQdyo795iaHwp5XV1fVNtbHnZ0dAVkdtbD66HOMQtAJbxZZ2ewQkHE0LxZpJI5IoktugK8V3ALJcEFQiqSPv/Ko5MtJF+4i5QBC+e+2mvxXitFCS1FjtuIhO1iU2C0NGM/Dgz1+Aez8uaAJaarMqRchXoVuZsKukLFI6ySRjFVUKt1MySmFHhiPOePk8j5+/BQDJiF5zHV7BmNLTpSgkfajwwsLC/KOdMAhRd4YCBRGh2s6zwpUD8vuUhi3/RqZ6S3FWZlUcy2lyPCnjt3XE/Ns9CpmI2CJ7242FRGzq7xGqNengh7NEduiLeJWll3Z3tlh6osZ9XjClWj7wifvvYeEzXdbL790b+nm9eODvW98dYur9umTs9Fof3k+mjquKJuaonwwu0cJTWJyqSxAurF74f0ffL+1vtJVUHZlkR+WWDYWsCxoWUWln/n0y1wHndndtx4QMoQq9/kLZY3xW5yOpkN5uIlpXuI1X2rV5JjHo2mHPgYPKKsi4ly9N19vnd6599lnm3Pbuw/Gk+FgvtOz12Hr4GRmRM7InjG4TNADlwVs/2ciM3uFs5WL6FKZ0HziTDxnqRRIuMMBEetkquNv/B9yfFEwApw0r3aT7RnHNFmxH2HM8YOSf0L0YgDOyjVLOfLC979GaC6okkXqDJ7mzTnSsfQPU/beMuO5WkhEul0dCUHIUFADnu5i/ihioPs8Hk6lQVxUR9JYaT0th6u6gqdpieEd+uV1ea2lg0vRC7XMI4V2h0llZXjZscgI+T8duexo2pl4SntD7B8fPp0eHwiZbZ4dNDjFac3CzqjBzDSqeJ0S1GeV9EIzDsbnytAp6IAcIRnbh6dLN9bv3Lm7ev3Wys17SzdutRZWJsNRozMgQUeIKNNgzkJZqMLRL8Aqcm3huxm9exRxgck4cCDU6dQmin5MazYM+4Wf/bfn2v/1f/Kf/8kffPW6OOle63BvvLI4fzacGHhFljWEQQFRDs8baaHYgU7gUV1OTxzlltxQfTVfBXeqb3/Kp35Xh0ZCLi4m8f3zCzZRGs9bzxU2EGgWtDALWcTT0eF41wo+HO8k4qwlA14pGRJT9nxGvUfPdrqry3/1L/7s3/0f/4fJuzOPDraoGdPjLfCKvFFmOPNdZEGD/XgGfNnXP+vfajCXo7oCkMcN4+qzGnb1a3X96rz6+md93eV9l6vl8vvzf/PagilZIAFjFlIAeSHJBqddzf+5LZ/lkXy/mJvqx4uvfrzqZMTqaly51b+wDthZGGDmS5ROmH4JhPP+pIIW4Q7oxSNaXVQqi1IFDAEisq3xIWxVFVGs167r9xZX73T6ry9vvLK4frvZW5QCwmg4HZ96rEHojQWVYiWLUgAKrVBqgqUayxckmap3RHnj3nG/8gycQDQGhmg2K9UNJfRgLOFvqQWQekrJXozW7vnTU4bt2VbKZSzZJPP2OgJN0Z0cE5tdlnsc37Klg0f50KJh6UMkZZ0puilAx8NqaYCqtURP4VEJyHRBX2JnRrkRlHMxyXR3wUtEe0UmbBCGnpNGiOcaC8cu6VGJgYgqbJKQKuZK/TSnVH5tgiw6HMNb1AyzVGa1vLpMZWZdj8M/3ULGULWJ6mI/g+Mp1jvioJpMFLaNVh051yGaekjrNd2aoXG+9OIrm8+evPHGGzRLGrgXGJDX5176fhAjuRwO8LAYy1KLMoZiqYtCNsGBvD6Kjt3mJO9SPwgCgjwYVemRidKJKnQwGi6zdPS6vAMAOJqMnjx69vjJuxTf++++zS2QNKmtJy/de8G+GXdvbSgMKmrZ1hGJLphpjPb3JmcjytJ4Mko5y26TNx2EFa20c8XT997rfeePF7pzzUEfEam3FeViw20KHzjYfmP9c3d/4va1r/7ab/3JP/8DBZp6NuKtM2xymSfS6WwYNdAAV4XdLEme6GKjYvJSTpODTKmo5E/2usuvbUSUbOztv6WgaXuxK4XidHbcAHLWnRhqMEaanVmKkRHHLAsPVLKEPFqbpZKpHFlFQR+qalz+hQuZaOJYoKwvYYmFW0vMka1TDKIIYSS8ymhY8EDrmFGRBoMLOa6I3+XXso4LGS9dC31In0zqxdr2SEUYcITYhzPdFm9WdNhnkMGQLP+UJ7OKi00ddlStRFpLm+U1JRgnDQdR4bj7PQsa1eE2Y8KOaPZeBE2jiZq9mMymx+P64Q4YAX5jdjMMmTXKbmDTg9PJHu8ujzgziremPTEkqTkd0zBvQ3a+muDL7ZNZ6f9KZMXQdDLX+tRnPrl8697ajZvd+cXWwlp7YanW6gsBPpElRj1j2UoKHY1CeJITsWxY7BUju+y2t7GZNVVLzLbaoUS2lpR2SH4ab7d+9kt/9fj48bOtJ+887C2u25T6YGfYC7kIhB0hjCAamPo/ceTV2g2ALw+Dyo+A48SCtt4Cn0D4I3rzfr/+lLO0eTG3oSbpTD7TLSc04BDH0JXiwIzgrTDkFgzb33sk50F1IimV2dQanUCvlI6eOd958PjuT974Kz/z8w/+h62Hz552pNyJzVQRpWABFNN4wYi85s/PgNPjjzoqmF3+UgZWwOSt5ZHq8/L39/9+3PX37yhnH3dbBbUP3fzDXy8fz8qqBmDWrw4XC7gvLlU3V58V53ZezXp10YPlJLIxkpuvMW65t2rBgspcYg6WYrWYqxaKjhbZVuWU8nY0dlKn4YztglpbaXTuLC2/srJxezB/s7ewUlc4ocM92CHol61X0B3EE+ehgUJyu4SLZXnf7EW9pboqiK6yRKn/HjolIjSbo/mXvW1E5NgA51DBqQiodk6JfUsaDE4cz1wSZNWI7jA+k3ftn0PkXltZa/fntrafqAUdjylCSe0jsOBXDMUJI7BSomuAAPQy5LJA8DL3+hKzFUU3qjHaciQ4hclNGLJtfhnrBDIkGkt8IaZrVeK0/oUQkQn4REVQXh4g5qVebU1kQrJCqokIBfSVZkS+KvwvHDDRlJcLLOskaylCyOG5jR0OD4fj0fCQQYB1uhgYq8kJBdSB0QhLSadQFk8BCD2YSxgDphmHuAC+jheBAzHttrojGxqMx6SRwWBg42Fau/qO5A7mBw/bJMO82GcsKu94LO9QAArF1Ey2WBuac+5nWI4oIAGkbysbacfShebZcpkr2szRq6u4y/HhwfajB93UT4kvuDncvwABAABJREFUoN9dB7fjyf5ojzHlbKAAA2vv0Rk0wAdYMFNKW+kMzvtxxI39xw/27i+v9edr7QUZpUQREdr2V9ucHuydby6srXz+579oa6G3vvLtJ9vD5vlRTx6JUABgI9jtx1xCn1sSwb56LftHN5X2bgqLslH9VGbkSUultFb/xRvXVBdQ1ettmGbfo7n24Ly+xxUZPc5EoTrmJxQ0XCl/HaYyimLErqAkL7xCQnXh1qJYSK9mlmhW2F5aiC8YruROz4u/bzbPJ8FG2E6u82vRq8Ivg4XVK6q3hEuXN+pCfgtLLiQKoOKctnZ9uCVLvjxehOt85UV1u6Wey6HJwh69J6/Lz/oH1y16lNr3tJXmg5N5wQWtKNJZYSTYSWG9Yd/lCDSC0vn0C7If5m0WxeEnVG9UH+GB9o5QlmuHMUD/eG1scCRikQRCwkk8fdEGiPb5Rxk+Vj9xTh1ruu/ecDo6mhVp1ZkfzC+tdlau3f7kZzuLq81On3ViKhOOM8mqbippjvu2mIQAI8yOfBmxIkEhGVLWnk4nyLMMUcZgGQn5YjpNhR62OO44NjXBYWeHaz/x+X/tP/xb//3/6//z5rsP7w1WZLYnppjcHFCGNGaohQozV1wdZfFefHv+/OqGAvD3vz1/9tH3Z25DmR3VDcZRnaMfVxf9WF33mf6g1xGi0JU8hVhPT5pPn707S9zsr8zOtpkHYqKga53M9Oea23uHB2/XfvrHfuI/++/+XqfWlFKgXP4EOhceUXAlubxa1fbHM+DSs/T0QwdgfeTx3P3V2C4/C7JfPpWLoZtpJJNYyKjzy5svRv6Rb/joi8GAjzvykwF7ZSBfvSWgxROzYlwBWj+U1KHckH7oW/nMSe646G3Bk/xQlmQ6XyjHBRICJsQ0DbiL3WyglRvy1vLatBOXSbIvIkUV0OsEajGoz9iuc6E+t9Hsvrq48Zn1m5/YuHlnMJ89ErCiBMCP1Z05p+325njvhDPMMOHS/xJpAtPLPNqB5UDoI7OzAg1C9YTtBZJJn4maeUwulVvCNyeo5liWD9P0OVWY9nYsxCvxGeQEQ5mbbde7XK/2WkDRhNrKi+31FUDg1hFow6hKLs4S9z/UxBvrroS/JpqJUpzgYi5T3A7rT4RjZJBo7Jd4j/2cN+zN563Z4NeSzp4MPC2RHYXSpKRljrwj9mfwKj7fCp5Z5IhumkVhc2vOwpQjQiOsHo9i6ihTW+a5zKor9IhkAQn6ooRz+u6Pxoec7xrRVDzKiagJf3Dl1LbHYqDUvGB+t7PRe++9p5N4cDKRgFnmXywc2cAwGh2/lziLTH6oVbOdjQoBK72aDo/53NC0bhOc19c3+v3+22//QLpXVedZfzVjWR/bQ+BwdGdthev1lMl3yvE67c/3OJvNqNLM9UHXVlHU8iUbUs3VHzx9iJGvLopsVhF0ptdpLQz6y4sLSpjIGHx0/3Ftdt80sambE1osocVGhqLdnr7zA/r40q27p+3BeHxQ7x71F2/dfHnxwXdkD++t3Hrx9V/80X5v7ntf/fb+o3cazVEbZbYv/RFRgtH6bH+LyjteS9DTaq22qqSk6pOFulBv26Pt487CsqqZN27W98adk9r9k/NnctKVEUvNwxyEumJYjWhqQZR1WH7IrGtIPPMsC/NErebkATOHhNP4VyQ/xDsKVoM7IfhxYhMhgYBiuwhyTbvYh1JaGamgQj212AoyFSJUaaswp7Bsr4xvL7+Xz5ABurq5qCQCK1d/4EvWErwsomdETBQgq0VvE1/oF+afNEJwc2cQMQs/dA6uZWC4ZHmNwaUfsQcx48SEG2wPrj53aD222RjtzKqoag3Ba0pX7Jz8GDV55OpTk19BPVahfOpHYYaRHSJzAGP0fpVEEZLZk+PZ8bR5YMcGhrP5wbKtrW/c3Lh5a7B+fdrozbZ6Z6IxvE6EujeKJzKWRIdaCRmK95bFxb6u5GfFaCNFBzyhkV7GZW+NZyLIrBm8bje5kBu1QY+PZKHV+/Qv//z+9u6/+Hv/w/726PrCYLo/MclESDMUUOq/x8qqzWs973C1ulLgWa4EkAEJoDnJYv0A9DxRHUB2efqBv2nvuaO8JO/JG8pRTkNtItPrVXIsYamvnjQ5dAOlqQ6fPn2n019ZXL5Fei0RdR7nET6XcTnoNPcfPVu7ubq+sPrO7qPt8/2SjA4fg3f+FN4DgEGPj2fApVN/jo8CiCtwFDBdDKzMVHV+2d7HQO3y539lf3VDl9KZcjiB8Zm/fFwcgURw7QO3uTPUPHiUCSstFAOc67lm5VhpnoKbmWlfynIirGs6GGlB+KHgUm5grytJjxAuzuMs4fNa5/TEzpM3B52XNq5/6tqt15avrzeavcnprBj9UkBjVlppW+XRur1PVSC1NOZOZpVf4jRNjgEZV+ju6CBVCGymG6NcOGMsxAWHi5gV14SCTNE5CWMyDku0Mwtnknv4RVlZUOeQq6Syzrc6u7vbOozZtO0MkAwfa36sFjQtTXxQ3JwCp0nXiDrDMx7pWciacUFaB5bi1EjxfybDGe4nyVDsZYzjWajNZmprhHZkn9xY7aX8Ek1iFQuGaiCpV6E46SISA/56AuZOHK6YnWrKyolnQ2lBPIbhEF7/ZUXrZASwgAbtQKLbnj1k3S1+6ISDsufZGCezRcOIzu09GCwNdWVl+d69e8+ePd3e3n7vwbsGRv29dm1jNBp6FyZOPmZrEKTkJfaHIlt0+nX6tLyq2uyEyjtYmBdyPlGfsggNdFx0V74vNVfnS9RaxAVdAgr4QU81mwcHZ5g0MYYK3WImOGnsPXtmuwek7RgDnuBk5512Y2CT3s7c083HnWFbNMj8oLe0OB/LZaSHfn9h/vHjZ4wlszaBBOaxrM9avTer1uQj2lC9Ob+4AECHw0njeNJf6KlosbFan+4eTXa+r47LrV/4FEXly//00TGntTBVsWlJ4AZ6zo1NKld2AVRd2dTTbmdp8zCOW1nFjL6t65TObS932iv94UFzQknjgVSBKQSHvCg6IhzJAkKKQuOyyvLPSYicuTebyU3lyrWsKvXXjIf9dLpCggBAsuqMvG7RWNneJ1FGMaAAKpHD4xAiL0h8RULkrlhq3gBJ8lthJJEBik6CMmTxSK3Rk3QkEq7PXExXcJWU9YqnT2Ba7nGNDmpti7hO9JS1oDUqK2Lia3hTvucr3PKaXDQuKCodPJmH7LWhA4FBEFXHnHlM01lKuhEuALFL1KGfrNCEL/tP6c3clgc8rBNedsKbIHhZ/r8ndJIsQpFFNWbG00Y29Wx0O4tLqzfvLN+6La+30e0fqwjboPtycxJOE/lOKWVFsFMVtpi1CF+91atCOSMrZjWVd2b2Mo9erZ9ZYwretYWJqTLK3Co4LXbA8+3xdnN5IMihtdD90t/8a71GEw/e3tyfTw2BACYR1abTOvRf+EIZklaBK4AskMn1jzgqIpc+/M8+vMhRkREToyelM4Ft6DgkTAIWY1i6acf0sb0lx7t8fLaTgjqZCzPL9XZ8urQ8GDYUuD39xIsvf+u9N3gBR6cHQangVeXByDfNOYr9MCcZcT7L23LyoTHrjhlx14duuLyt6m4eLEe+lp+CHJfnlz/mL8x6/mv1uM/qxE/A4TOoWw7rq7pSfVa3+YQgVzdXd1Zfc0P+lWtBmrRW3RpMghzVLxaqpR1i7/ZoT37VN28xCbAtaUW6oTsehjKWVNZsUMcIhDK6XwuYrh91sUeWVOI9ErsA9jhZg8gyjDyjvMP0pJMSrrOLzdbNlfUXFhe+ePvGSn12WfCqqGMVbCf7abl21l3EFWQhq1TFAXcOs5PWcyavMTmSE/FVMTPLUmasA6IiBxiEbiBtVceRK0vVDjDRfdWRYnm20a9cWjqyupGFsiCNqJhsPsFTMeEK5xkJjAIBbqeF+SWxzwYulmZO7mdb2ltL+/ZMwJVbbQyp0+z2RgkemuJuDnsGWMm0Ljn7GrHK8BYwxLGhtUk8se9DJr9AWAdR58mhO2SgqzNlhXu3+9l49Q8lEE/rP4fGGWMxudNG67h+zMOtZVQvTlSQSoRT2qW4226BeuEQIeyzo/pDCZg6FMJoz594XMEpQVJsoxYUo7xM3+BV2UPC71gy7nvz5k2c2Ktv3rpO3BFIPJkcPnr0aEHRxV4PV2aV3VgUnHUqaunp1rPbt2//pZ/5OVz2n/xPvwbON27dPDzYtaj7gy6PrG5SZ59tbz19+pShVpcTVHksXLmzurxCkzs42MPgIWav30sI9dlJ15bIYtsaszeub+hAt9eRFc0tuP3sqc2K1tdWbDprwyaWD67zve1NdaWay6vj0WFjuffSq6/cf/jk6bOdgTECc23Wzsej4UTIMrvfkwcP8aDrd15YXl1DH/a//dXBvHoL1xqrrdq2nb8VCekvfHLwl9a+9Lu/+t72uw86MwsLnWW2GAbFuVZXgnBtU075/uLZeHDtbmu+rRL0UdihqsLC9AanIyE2wgiv9W+3+pPOk231q0UOorb57+z0QN0TkhiLSezR6GgKR2d9WqWRWQWf0bQQfQwx+q7fim8iRMs0kW9k5cAm0YokqWPpe1m9YhGb7bNstAUxslaZBK087Zlc6JFFrc3oc+SJoE3Ii0cvWL82BH8zvoZ0FD7o7uhAZF2bZYvSFlanEmi7twB6z5RqfPJYCi7qWxoPYcZEr+ho5Ep9PSEFyOXJjrBKMyBFrTkR6EYVqoR3OfSuSH6kCM5W1zxnhSQReOZsjAUkNNxFTFbQZH4LGGiYGKfKksmwqzGRdGPib56PR2MSQT2lsroyJLanR/vDk8HitZW126u3XyBXWrqi2sRP1lXPPu/gsqCpEhuoK+xikyumsLKbBQdG+gIWUcgrjpFu+HpBlvOTI/LtOQOdgXAyhV6WLuKrre5gTMhvq3Bn4+f2Z/+9v7nYnf+//Z//L7cRuPn+wZNtlVBvLq8J+5IcL/39ZLRfvcc7gCf/l1cgn5m7cLp8B7S8liENg7m4IX/c49NRddNJdeXqOm/T1cXclsPfi1nzJYSD5FAdwTOqDiGBDBbLDbHYvWJmFpcWHz1+b+36ztLqCym8korcIi2aqBV3/fFwPDvf/vQnPvlf/MZ/O99bOBjtlZh+r7oEmE7rqVKvefnzh3sux/D+5TyY8V901g9Vr9+/4/9vZ5bQ+7267MUPX7n8BS/LlFTgdeLI96xP1CFkwJGLbiyfvlTPRv8Ko6vQ0Q05jezmXyRTokD4x87ursWW8r+qKMcGHE+fOv02ZO+dn6/NtW/15291F+8srr584/ZLy0v90V5fwRVzKBWBtZgRLGu/tjsattR9bDf6sjdwK8REJo5yFDb5KooaZQ3XieSQHkLPsGFnpZPe6qsfiGx4MCOuUrHl+QT78P7SetFL2mvwt0LhNIQwxQKYIWOi0lXt+tPEdMWZ9roszrJodDS3iR4q1Y/Zc9EOT8SkjK9bfYW/AkiQtQhIuB3uC7HRWuovN58rGJVKkYbkHPe150+yZnDUkpMcDRXpKiOtWvRST6Rj6Eq3W51HdHr/yMSFpEoLSEg2Qs9ViATbSsQLxKDR5CJv+TDL4KPDTJXOTRlGi/U64cHl68WDFxfnmQHWN5aBjr7LHI2Vvv322+4RseVFSkbbIUq4mV4Nxw9w1m9+8xtGSSpaWlIetLm3pTRgxDY3a6J6qekFjaXFJXAj7iDTJrE63AZKmUekHAHGYmBV2FJi7oSu6jzrtib39/ZM13J9uddXLzYGjEM7qI1GoH5wYHfJ2q17r23cvr27R0zf58fGocA6ITkpCTVr/4LH7763v7W3IoXplgS3a7WzcW37Xdr22VTEeP3kMOE/9fbhz/5bP1Mb//gbX3njB19/t60qx/Rke/9pr7Vg35bR8HRmK5Z99SWVZGoJCDwlvPWTiCsdeqbTGLLY2PLo9vrK5GD6PTYI4aAKbc00NQ3d2JmLOBsulH/hdkZvUQEYhR1ro2KLsXIR8cW7gWY6IteFAdTtEZEyzLaLSDUhuJcIf1JpQtpiEy9UOPyvUGPAL2ysktkTWABb3FPQh+vBuibd6HCIvOVEZGS4SPm/OUaR2vrNVXVSWr35Jvc5m61MPg7O/eZIMLa1pDtMKfpFx09IRU3ku7kwoeKCmY/r8DxBj4XFwj5IaqjQlzgQtI1+GSDE+FOYPhesf4hM6FtIfha5hVq4helW3gTa4pSCK4gZZ6d2V5y3zIZHh6eKnjSa/MLDw9PRyVljfmF9bXFl/W5nQeHQNZt5KQoT7YHM3WhTFPDaRDFmBKkCn6mAfdEzkKIiDQRM/i+HkzCRLJ9CalwssK6SniOAZAHmVl0P9yR3h66eJUzOE82Fezc+8aUvPPrm93qN8/bqwnTnYHO8P2i2ZCRz9lT4X970p39oHVz+JRpwNcXPNwTKLlaf1fXq6/P3XD2FnMCPDD43Rb3JwGMBOZMoaJ8oFCPQ8L3QXghK96LsCH4cLDRfuHN3Y2HjvYMnc6n3LFLCAe6kYfAxqxEtnmPAmWdXyk9OfP2h46JnVz9dnfzQnf/KL1Svfg40F0AssAnf9Eaf1YlzmP/RfTBjsMQ0lJ8D2MxhUKMCS+Bt+P4vR0Gagn5+KCDxqEnwX2zMuRNNz4qFiFM2zeU2Ti4yVliE3BTZnc2TU8Vzr7f7t3qDV9evv7p67XZvcWmuJalexWapOw3RJdghhqgZUgDWRm7HxClAdARcSqjQ8aQow6KBc1vWI0aSaQocMmqdyj/dyKIti1u33BjJm5IUm3O0ZkUdaQ02FiiMj3gQcp+hECjytIq+7a7iEPODRdqk9SBLdbC40O11x0kjHhNx6bWEDatd0yywCaDGnBmYnUcckvUQVTau4MTPIEACuE1HIsfUM65gH+YnQyYg9NaaOlfK+lHdLyr1YL3K3eEZpTy1bphQ1l1kotmL5i1ISugl1h8aBxo5tJR9EUyJLwUqaK6JCuXDQ1ElBF3yFRBRtCkdOn+oQkfMzuWR2dSDXLKPxeqq5GA95F6VV/vkyePJ0Vj81LVr6wKmhUoNR4deMT8/ECDNQ6wqFiVsuL/zgze+7y1H41F7Yw2UQMQMm2LsnNBAKDAREYHOzn2NlzcKTboKJ8ndfNRZoPHNQ2lzy12e3dtdocAmkr3kH5vnfWViZ2bmlxfBWE1QL6FQw2TKcCCgdvb0aPXatQePt5S2rh9O5vUC9G0AO6yp8duBFaznQxlGxzVq+Obu2vq1rb1hXOLNDl1lbzRhA7GN9LPJ3t3XP/3yzb+0sfHGd3//7ac/eNY+b6/Nt/d3D45GKaqEvCKy3aUz9u0k/jKpIDXELezBPrxqXy1uzPQbnUOJTC1+FHOE6BQCPw6Pi+gaFzwmEtbrXyYSNh0bU2EKgQs2WNgi0OwlxocQouxiKxvfn9js62hcz+6rE+wuMlym3KJMW/FGFHwA5+oAbYqqafCV/pTXXfyQc9ycWB19D7cvPDXsTP2yNfsNd2rdQa03EJU9O5kOmPWnSwqCyt4iY/Gq2DRZJDIOpvSb/HpKEUpTzC0hOWY28dpZslrPO4P5wdm8CBK7ITYoKMvVpGvZBNoOFMEDckopqUeeD6mpgsHtgsbgS8aVIQEZ640VHpaD6Takt/dFFn/nXIJ5v78+WNxYWL45Yw/IVvdUNGSEUsqXx1MpFje2WksHzaXXl38XM1GBpvrU9XQ1g/GrT8s+nc8JS1YZUiaxLKZcNBHl9ghWVPKUPmjMLb547y/88i/+12+/++BguKzePI/M3ohzi6lGyQF7vVQv81naCV3KUa1xoy/fwM08FthV3z/i8+rXq5Pqpo98MHAtDeZExwtD8Vc0AXJqegua+DHsw7ShpCTt3Fq08MJFYnjp9DoTxn7bkjRrd+/eZBX73jd/YI9rUTnPHyY8eBkf8MWbAr2cBx3KSf48d1J9ff5K9WBuuYDJ1S3/i54EBFcvNQWZmOeulHd/qEvVI1WvuEcuF1wu+AkO+UQcgyzlir9a1Eh+KhMPTSPlhD8CWiGNqTBcUDXMm/CsHe4hxQWbRPDsgXM8nT+fXWu3b3UX1put19auX+/17y2tXhv0e3j3+PB0rOzwZNCOlwiuUXG0XgIeUSd8Ro0LKfbTIxvpHewrtldUNxJ6JZVZk1mwMAQ3dJD00/vSf8MJVwpvpSNjUqJ6/GM2rtQDwj+OxLOESBUhAo5b2VxcltGJ2kzz19bWaYHR+0ZHS+uLq2trUPDwBPfFDDQfvd+7vDdpViGi2kIKj3QqBTEFdamznA6SG9MTFFWH9R7DwAnmbLnbbM/OjWZPmb/CtBl7UWhKMEKm15KUKO0ZcqCTdzkuxxWvMIu0z6uL5fd86IXFquAlMEXV52uzYuhpHLSdtjMjnYRYW1b2qpDqm6ATNlo23l5PFq6SVt14Lc+mvL99ReXbrZ1dNuZN3T88XAQTr15dXTa6hw8fvvPO22pCrW+sitJiK9Z7e+Ml8yd8cBKH3fTIYmSyXlpYxF+FTY0PhoAsNBpcdBjfDevlnraLnnBoegIpJFZFiI6rcjEZaGgtMKYWCYshATLGQqZ+xc6yM0vPV3hoI6Z2ty3pszarzMrKjWs39sbvvn2f6Zgll3hOdZxVUk/lQO00Z/vtObEBT+4/ePj2u3dfuLe1t8+zMdu2VcLscHLUaHdGi4sy37Z/d+vzP/Jzg8//yOLDo9HB+eCs92RnuyOLOrx2/2DvsQA6UB1oWwp1ijDZkRLnyy5AZ0OVGToiCuaaL7pNyp3IIFEKtqEN8W+YY2IXZBa+p9cmigiXlQanIEZoMHjE2Cqfiwh2qIyXsEHBQtE1pEjNjmcUnDwfJvXJnsj2aS1LUV+gNrxRf5QlxKMJKAhENQ0rUxKwIole53DFP3Krb42OHYftYA9ebACNdr+rfFNcxsZWV3PK4qyB+OLGPHfT9GCRM96l3c2th+/eN7cd2/R12lOyV5gRyoF8WwPRDMPavSzfjS0LKb0p16Ct5QLH0jcnYo6taGgZuQPqE0ZUucFu3U5vjiMY0BrndtFQorLfnFtt9W7W+ouLE7tjSl+Y2o53adBvteY73aVma2Fmri/x7ESmdkCNdBGOi4XKu9In6zY6SFZsgc/lmnOtHLnHkZkpJ9TB4gDOlVxHcozO4cSVkM5ymAeEFY3C04yJgaW2PP/KFz/3hTd/7t1v/Imk88HczGB5/vj4bO9oTNApC/7i2ct3paHCynMSoGW+LjqUSx91VLf55UMnIVAXQ8hjzrV09Vm1BHMu2k8+P0nIUow8E9jnbhOgOmdbPVVmktjADTYV1POlOegpUkN9Ilr3FmpE9sk3j2UiXWBb/sCyoGGgJZ6hemWwr+rW1YkfvM3XyyO0+uq4vP78DVc//i90AkyOqvHqpHS2GgkoZZYyUeUqNHJc3F+eunjkufPqStVk+HD1wOUrckFTFQAqrCzwqIbMh4Tw2eEUbhXkVQQaKz3fffJYsMzKXPf6/NJL8yuvLa+9srxxs9tbnJntsr8A+XAopwOCzglf7zfoEBYe3YQtKYy8KDEoLeeWkF20F8VW/t+0iSEivIqWDTLDCbNh6OYtVKosaj3PkExUOTARRkFKKgOsqlIUSLpv6sAGlYqhLthe0CvEPkvE4/EoVzMfqc8Vw1Pxikrel0YiMkhl5dj6KOIl+Cham14kTQIBtRLjEk/2YvxsgWi8zBEAwo/tCjzX5WCKPTGSQ2x0WJp6b/BXlzISokEK7Jd/2ohYUNkzaN3xuKtWyTRd+l0WvlGWPgcaEWICAs3iYbohZtbeYnmM0tmm+HERctOAAmnBUJKch92aQ0lGyhGsra0RBfBbxal4fJ88fehkd/dZSnjuzz16khKLiWc7mUpMMjp3mruEcakCHicF7VZOth3j0MkJm4i4YWnHcdWTh3h80tsYHXi88F29jBYSykwEhBMpAJoANRAKOc6Aig9SojWzAtZlb8eprdLmB/3+gA2lx04r8YaYUHaXM0FefKzK5WK93r995/VG7/7b794fKaxJ/FKbUKzW8uGhvVaY5E2KWs1x6hNY3nr7PdaE7NdzsMcASU7iJxjOnCysLT14/Oze3v7S7Vcag6X9qVLDHBCoj4k2O1Lb9vfPHojwY6mNR33ldkv5bTzkGG1hI5G+DjlmGkvNJDwxjbJKY8DEhnO1usas6Sm0W4aK7QSvw3ghCP6XvLJkJUEkSrAhnKTW3+l07A5OTO57tsDz2QPyBvKXZD41jsUIhgjEMFrwMWQuq8JF+BifRN4mMMBtLgZryhXXfW206w1I01ZaMcQk0mHTLjUTru9LYhneU5vrzMzPdud6Z/0el4h2B7JCxWuc2qNdsYtsLpKXWj15faRK8+9KpO+y5IwzhN61ML38VNFX56i0/wptTw2lSixgKFCFu/RVdQ4CLpvZDIOIHd/PZ3pnjcVaZ602v9FtLk5JSeeH1myvu9Bs8w+oVwzU6gtIpSDhtVWote2OvqEmZpk4kPNofdEvqBXpjeFXRwigea3oYHWpYgTR4WB54OuG4gvLhUKeCk0Cv3gLZCvG8084RgBOjtWFa95Y+zf/o7/9xpf/8B//V3/vybfe6AyWBfodjvaXOh2uKbOUlrVb3lmd52sAmuuhzHrsVysuilE5yk/VDfletZCBlW+XJ+XbBz7SbNp7/6KWq4v5E7Pz5etcD26BFx+XLIZk9inoW291EbNMGT59POJMT9K/7UG7taW1FUn5IblBtCugOve+ADBhrhcvK6/NyypY+6EcuZIOXAwoyHt5+L06dXJxw+VP/8r/fmT7Llaq2NXrnr+tOvd5dRJdqIzl6qIHA1HIUTS76tdczNhyneIQmBufBwEsQzapNl1ngY2pLEomfgCgxOWTmU+0Fzd6g7tr6y+tbrwwv3K90121MmHUeGiH09TxwT04m+K7ITdjQ81KxgViTTGK0leSKDQ9aolqYMaKbKw38iypd9OoP2XBhP6lQxVzcsn3hCSgYIX9eFFhQam8wXgZjSy7AyUex1hC4/Ivp5lQ8i+64mmqGE+zECcXlSlW7AMp3tnfW11fZPil4SSFZ3RALCh7IoU4Xrww2TRGqAt4qZWWske6GD0pHucgIe4zRWRpbVGKL1Z1+nl0wuXFvIg/ZRsJDuvY2t0CPirphrri32Hhch2O/XZKX/SZoVJ0rpAT3YgxzZpxNdND8aVWns81YuLWUeQmn7RTFT+kEqlp3Az/rdXpvuxFAq/w1AMmB2xy5mx7+xl2yFDtc3Mz5U1+4S/91YePn+xsbZsXRgJwePTgIaZuPslVdv/Da8UwD3iAEdZzTon0me3fnSCp8+bSS9m0+ZLLUraxREZUCHAApnvkBzOC+hPFsWR36nnC3Co6h0Qqpdm2L/C80kbJ/pa4Q4kQGdcZkJcmxwfPdrY3Tk76Kytr1288eeOd00MpIqowC90k2sj1tF+9HQu57ZVCsevV3ObOPvM78NHfdbjb7wjvnI7Pnm7XFq7dGGxs1DY27n32R9/9/s6j775H21JqigIFzkB5crTN+mub3v6pYJ8+/6SIaIG6cmDO2ywh6srM8WTEijTL0mC+xCUYnMgtatgRlwUML9wXbofe+bUQbdzSUisp66AINOzx56Oz2j43r1opdYnCSkoobV23Z8EwIpzHC0MNXpeFeiiVqXA4zm/wnmvjpARnhnahFW7G/yLrMIOHskVkzJZVqRkSBhyNvNTiJzHztNrIUtdSts6MsvKwV881l8K6UNK+cLSXOp2FzfsP95/ttl2yj1YEBXk5mkMiLvyF5tiR5ZvDcHMYqPcXZAmV91ZoimVZsGicmsP1uV59ZkDYIwSLGbF1e6lhh4p3zkGY2j7L0tCzAjotHlWgJnPWhvRkPdYFVAxmGCf+4Pk4Zr09SnqoSLhDEPXiKEpxiISDYJLPfLhShN3qevnMACzk/NPaBctMgpRTz6TArj2fysNSlBpoJTdVCt0v3ni5+VMvfOvb31Olcn/3emeeBCrjAYaH1l6Apfypzi+ulJYuP94H3+WVq7/P/3R17gQp/ZfcE6p/eTgPwa6AkA6FhxKoTLVAcQEA+cdQhjg1lTIUQ0SQmzkYHTQGXf4S+5+piW3wsj7sFpJWK6JbmtKy2QbBaMBedNHFvPHDA636dHHDc79W16vH0/r/8kfVh+qz6rP+phuXMLqYKUDLRR26uJC1m8eqeQ1alJ98jdQX3C/3+gOL8pmr0NUfegBFzc/5IXaji8ZzKQdiNjNDtVrodpb689frzZ9duXVrrr06v7DY7/TRTnR8TE5C/dWWIVxH7UsqHfw3fcdHvVZHLBAtEEE3mcgo6yTljL6b9eeFulnMzNHpTk6ZOfUSeYDilkkImVEiCbDfkkWknBTOhKazXoaTR52mNxbXr1ZC0go8suTKUqPx+FeaYkKnz2Ehq6vr86s9+8JaMxyirQ47KXYlFFktEGpbYzJS4kOvUw0V2wEgh5IVrvDtpeUKsFn6vpSvxdRckjrcbfff9rndlDAu3RbVrNACjy/ucywqCzRCl7HgisX6NCL9jtHnImXo4gqCVb3rwjEedM7/oavoXzMFAidjWwfKwsrAHTrp00jFNRZomElXRG/ZPTB+W3rw8sqiyGT8mIfSTLNXHx5N3n7nBzdv3ROo9eD+e8MDRaRHfN1qMe9ubTMJg/WJtyA0Cbrh3letTrmO+Px4NTGrIlWkakHc37mOjNP/Y07XHxKGap5+iPBAfc7tbpJW3FI11PRKFnFebynNweI6O1hbP9+1kwSPrbyIEaOsCCERc359570H/dV31tcYXwZzrfa0PlKAF7zciJ1juQy7xLKRpKxJFFKcmHs6nvFsWxlWx0rL5HnUEd00Mzw+Xzxv1j7x2Z/9ayvf/tUvf/23f+fGfDf0Npv9mXXFTWwytYlb8EW2V9bn+svnIh9s2avSB3V50khBp/Neu7ZY607m5kYlNJnkMa3bT4Qum6XqX6paZo2VFZsKLYkGP5u1Ucc5hZ2XFKka12cU5/IIjy+LNEnxcKZhz4lpgzZIILQaED6hx+KJmFEE7hXehhASDLgDCJEl7wbKZ8FHn7Km4gPNOkoAlYtZYZGP49HVA/Mk2emcC6MlZ4ApWCE7TqfpDG2SX7YlnElZnGZndq01L0JtvLT3iOVf/tnR1N7FYtotVq4D3dLzinZkXfu/jLSiVqEn1eEHr8Q/CWCGzFvQYKm18feSqOL4v2V7yQ84sYGSnrdn6l06rk2mWhbOMSkK4kV2YfdAUUj7aJi3JeXOIA0tGVPEnJj9z1M7tQK+v95m0WQC0IVcDXsOo7W+wlFduvgMCS1fnZgx68rQQhx9Wo65SPIhO0AJTMaS5VWnzKjtcgaS583DYW2h97mf/OKjN95++q03xHK3e/Ak6m9h+JegeO4vqOTlOUp/yp/nfv/o09xdjquT5+8rF0O5rn6tKInPHOU9Fu+Vmh2IAFItieo0e9URmp3sQY0AZobJNkRhVOz4cNCcHx3Vnj3bROcrxlG91wAd+gQgzsGmgu8FNpSfqjsBURcy/6VzBdC5OfCBtICRz8oIUM7NrCsf+qza+vBnpk7zF6D54K/e+0PH+zPvp+rZ6p7I0KG05fB2R3prTRVMce6o2KrLSG32FXIhyJ3eargU29ZEbJdAG8QxbNBMMk4cBuWahjmv3IQ6niThkVxpP/fazFKrfX1x5e71Gy/cuvlSd/7uqLY0RWGJ9RRS1ME8JclOj0Jb1Z9C0M7OBDijL9ngc6QIcJJqUIr01dLLhMQ4hShHqGfbZHy0pjxBP1RRMj0M46zw3YRkxBFNuStQbpbcxBBHl4wPkWaUf4gjLMH8/BN14VmvMFLjsVAKimiVbDcje3XjxvUbN240ujM7AnltHCpRZ2eHDqF8seQBhSYCOkyFPTQ7IOETep2N66VdJgErUSTBEAdmgw/5DJIn5wc6Ev4CYXmsbGHsU0eN+q76mIWf4z/aYzY3DvcVxlQBJlwzU1ImpmiNhSmlVf/yMn85Xd2B9noyQTxFXgWaWDOT7RQUybQWIysbAx+wC2AjhHhvZ2t/waZG9ko9phGtyJWcPdvZeoqL47L2L0KO/tk//bUvfvEnqLPf+973BCTHYCAzN8I9C2vnrHmyqWD7aCiX2ohF0e3tTNBLtcUM36sNgbQVsJeDqARONtkzQtfYWYk+bNFHtq2Jk6AaYLiEUzPcareFppu4KL1QdWlhvlM7Ojo4ORiCHuMEQUvdjzEzcns63Hlqm0KZTeJB2MeR5OnkTKkOmAbbEgoGZoAFbApZN2zyZEWg1bQlfOu0qRyh3RLP5sx/B+vkXPze2/NrL3zyr/3SznB3/933OJu5LHn04/Jluzg5mIxPHr41uc5HK085Vc3YKxg/eZqx9GRgn9QWMaPwP9rkWfvo+Ex9NOqyfQVqM8zRE1bVWEZITPbTxf+UF84mHftqcYBfq3FwfrTfmB17sRvYrkXdWSgiDeTUMKfymNZAmn0q3p5Ep3drnXDPLA2DTfHxKLgpqpGL0F5DoRdB2CCRXLt48xIChbTkolNvYnmyvSDcTLYEPmIfb2XSu92jYxaExRkbAVJxRV4NpNcudV+exXsPtrbHO3vEUblUXCqWMnnFZp6VEHgh9qVToVsIeOnFBYPRp4xL56jN3m5MCZhWBkNcWwsAzZgwcWNQn6XeWLDRI8mgIRW41kyVbuq62mkW6lwyAKM9xOUOCejCKKA5oFZj1ejCONFegBtSnnBl8+F1CfrVhRwmVw9jakQ4EUONJY6soqrh1LmC7EQ5zBqvOiz6LY+fiQsBu6RPiVrvoAJuI1gS3ZXjuvkzX/ylw8P/6eS/2Xzz3bMWAokIR4rK+syEFdDkdWEr5Yp5ypvAw6+uYWJ5YzEXVb0qPLKaTjQp/fQZ2g/SOay0Qi8D7/zL06WN0lJ6HxgECBkbMBhGyTB1U+gj9wOsF1zQbHdl29vD2xaLFmvJn250B/P7Np2bHPXOagIlHz1+MKkN586bwbRAqrypvLe8ujBgi7/qWOlMoJYJsA70xVMFjh5BNjMzMMJIgh9p5uJTXwsJLF8jGZUx5lJFhas2rz49aeFbj+E2MbLFEs4mFrw34PLG6jP8Xz9ADdirVeK7I2QXYjEMlRKH6VX4a4aY+zUStSN3uVB4WsCfQACNlcnLbo8MS0r1OoOyVq//YnCGsTFucnKqIFESefFiwXnin1qnCv1Nm4x7Z7WNTveV1WuvXrv5opRMyZsEzyP7c08i2Ji54FriHPQggiriT0ydm+1SeZNRk62JMJrhzl7WX2zdZaQ+Y4oKQNJzBt6w2jgTcxPox00MupmKgjGG7MgSik0c0w25YBOXMHRon72x8kqnKkHY67dEJhX7LG9Ugi2D3voq2CNzTQ/IQqrP2gkAzxQMDNmomvXD0529Z65JeRQUZtpaXRoAnqIo5sxwb7eyoJpEsQcKJVJF9NnXeLFCc2OvZ3R1ALRU1sgEqiKXLmTGSSZq24ryxXGSl1Qc1nEwWz0QNgKf8Tr0BiNnlAgHzUzDass1WISfl58Yt70v1ZbZ2uzOq3CYqGW+Xn2i3+F+owPuVyWwRKooh3UsFkxJSszpuDH3+OEj85WI5Tm7zp6qL9VaXNrudje3N13sdbj6agvry++99V0B5QjpKc4rSlyzOIkXc6/byK+lWPTJ02dPYdPa6rJSo6zRBwcj7u8c4gOWlwVOS2Mwa6r8Mvz25rHyucPDuf3dbbHW/evXtrd3Nx9vNnnvmANSwOy0213YsfeRjdRFyLcby6tLsqL33vzOtHbUG3T2slXyUSLGeo3uCSQ+YJTtnY36c0e9+ujaaudAguvesHct3Lfe7c4oQ3QwwtSD88Br9aRES9RGKjvp0AwqIT2nGtXM6c1rSxjp2XRzsmynpu324vpP/a/+2g9+/Q+ffP37w/uPpVMDwdFURPV+p7NcG888feMNuSUrN+7NDdazyfv0VBkwOg/8w9RSK4mxdGF1Zm5j5nxhePhWe26+0ditNZ8y4GGsmCOWh7zXJkfJL8I3MJrzPfbUem2/MWPPH+Z0nmXFPQiT/oMmpQYFdwWJ2IOuQOlUyKqWSFAdageNjNNo7U9ADy60yWIDZKspQm8+ISyBAnKlfmvWR1GDQ8bVVyE5FGI/M6sQFa40aA8Wa/XlRHewf8+xNAo9E88/uvOlz7zzja883d8SCbA0d8yslE1/dXjmeH90en2BT4S1hJDXi7tHHm6jG9k5IjKjh2pqIVJwys7B9WyEZwPj0/pip7Z2E3qLRzg8ZXZemG2IZFRzDQ+er9d7iTnsL0XV9mxYVKmQRfjXb7N8ppq0SbVrJNFkJnXzxmOBcUUWQUBEQ1si4A10GL8YbA5cDjI0uGwhSX5gk4tU4idwKUcJFFEIBRYkKKTiG34N4bWqBN7pzIkQOknN5DoPK1gTz7CCrNLZ+4t3vvjpv33v5m/8o//pv/iP/9PP33659vRsejDxyg4jDcwfj5nvB90eAZoMEJJYqGVhT5F5qguFpudbYbSuFZnF3ahiIXN+8B8Ggfig8GkD+QzJLWQWNMQ5EsqrQcEuL/AMjJiyn4MNVtFEkqaJowiLPjgcffa1z/eX50nKmuJL9KrJwYSTn+Nx1JxRTuekdfrgnR9kXxCpnWihmStUHfQK+crLWEXMUxlT9Zk+5Y25+P65S+WecOF0+of+Kw8VDC/PF0jlWnlVdal8NbBqxGAUMhqIAlssbmCnewaS9suR9ZAjoLjoQHp7dZRzSOquPF+N4uJhRBASwQfsr/wEFwllWFoJf/RQFqnulJ8LLVf1LSle1WV3FvbJriiIWAE/yqQQiPV2+9pgyedrG9fXWq0bvfm1Tneh1NCQYgLv5vp0YyvaSyMT6FlCPBitooZYX9PDkV0ApICkYjOzJBdZgcFFt8uAM65K2gi4SivV8HUuA0ofCxKZjCAK6IX0l/eGvtAZUmMyim92+S3ihM5ksAZXtacvAX5E2jIj3udrGkLxp1Ja33333fml3tm5HVMaywuLSIA1iXuRZTuTNn8wyR+5P15YULwJP7mojCEhsZeaUKFT6V7eGoknc5EChJgxRk+/TZxxOWKtVfe4KfqFOzBSgB5Hj2ZjiKHBEC8O+KDbPn03vzlHL1CEXIkRXA8pXr4QpbE9ceQUQsFnDlv5WsLyuRj4tYnnt5XdmJ0TMAoPYQLxLzNBKTo96XU6Q5yVjGiFVKq7dzlntFDSnjQW1c86kuscTz6YEqhiMKbfixvCtU6n21ubB/vbBA7qWMGpSqjL5AEPALqoswFjJ6oKUNeb9f39PT1JD5ttu0XwWat+ItuErSHbJ3ZsMB+4Sf+dX+hzV/QWs8+6BDOUpd1d6A5kN01XlwcH5IuDzeHT7nS0Y0La64sHs7KTpq1uTxbV+GQyI74XzUUZmEuQ/ETGZ5IKPtCFY4w3vF67s7G22lpoT0/HarHt8x+j8oP2iz/3E/253ttn3zwh34QVNemjEuYYtZskvmytySBQq/VX3V5TGUVUrnsmAq7kVJ02DXCenozTZ3GQ6+ozQ0HUCB7U1gtTKc88FuYou9iyQK3hjJCrY05f7eYfA0+IBmwoK9kEVT5gNgXnoSVhwJnc/AJwuTXnvpRNaK2JMq9lPV0KtHl9llvQqlC9zHNhUpTHtOBpSCJsUONMXHuJz1K8Ih5B2KMERlCdqfXWa3eZit79kzf3Hpy3yGEaFQzROO32jQ5aIUcij8aWaCxHYeCGqt+WTaHU2fA+fUD+bdSdYJ/JpJMaKZ1aa4B0QhEMeK7Zm2sQ/imX7XNpYOwNUau0U0HpgpJwbkAqjV3+pIEY5wKZaIfeFHpVIMWF73G/+ZcegRZIBnL6nJvRijBg006Z1k6RU/zqez5dyGuiOuQ/37N4rg7XPEHS7YKbGIRG8/bGy3/hc3e//iP33350e67V77QFXRxNRr052V8rtvI+Gg0J74Vr5RU88KWx6EgX0kbplRfpJCJR/Xp1Hq06h+ser0SHSwEi18MJ8tfPYJQ/mYIy4FqrXWfj4zsiBs40WqQA8iQxpDtYkhHOI2S+GBvNJt8GqS6boWHOrZM7nxh8+Xe/vrdrY7E52n+Rh4BCD/I676neGueE11Yvz2cBXT4LrhZiVy6Da7nNleqeclIeyJVye9VOOa+eef4zY7u8szoJ4Ss3hxkEWm5wzUVjL126fP7q2csLF3/dinxXEAQzVyscCAZn55yAtVw1K0EFsmA3e1aEZRFpXEELAh0az0TReYouJhrR1lN0dDhbHx2zafXrzcW5rnLN9xaXX1lduzm/cK03GDQb84J8UpI2fA4g2SGRP+KzhpCcDKZ0CA3Izm/FwxsfIcMw/2xM54FJeliO507z/QoIVycuUhwxtHJilHnWG4wUwUKPMLXqXyydZc/UMKq4hP3Kx2nchYEDb/lXDVMrpaW8tLJGYMBvvtm688JNVR+QBlkZeCqJXbSRiCTb1ka+qfUwjwzxkiNqnBoaJqqS77ECgRmZiwbpfn+JCY25PiyEXtgMvghczsCHcjjFl5D9SODoIgGGCS7qRuhtOXRak9isb4SYqv1sAWj4F0fMzm5Qu1rsFSP/4WgoXdXbw8OO1WE8tMWgVBUaOvbC2iJAmc7Kn23WvT28ejLOdgidDqGYb1UP9Y2eiiWFgJpcda+xSliiI8weIXlxYLTFmqoQAuCxCRB+fB4JeEGHTAbI0Zt1s14/0LLOx5xpMnHYYiHg0GB/Hu6OD86HHVSo0zka22runHYMGOpfAo/ZkfqDwshUXt3YsJUirt9p9MGWVfywe6hcF2+w6DkNu9/k6Lm8MlYDtbxhR/FETJnTOQDtpiaAS5EY2EPyIKB4Sj5FREaZpBYXa+/scXe9V+s3Z/aGHerO9GB8PDs/u1LbuLb+s5/rDtp//OXff/fh4wEgNpr7lGxBcEfjg52Ys+zZsHh9prao+Nbc2XRfCTNVyYMHSQXEEDo04N5g48T2YHKXWaFrEwES/GuWYCHzbKkHpL5IRfBnZnyulqqE+CCEJYDMO6lskBVvdbGQqbJ+g2TWBtJXkD2U4GLtOKHoMhdd4FVa81wisHJLYS5WROgKPPSZ9ZZG8kSwHYNlc5Lzez6cnjxU3Y0X1k5m5tbdkJ2xX7g5fNm4dXPQHDxo33/wfRtHTua7czu7p/1O73D/FErNNbEZNhgw14HY2CthLvw9Jum8TPgayzOunnpXSulJD1OcvSWZXjwbo21XCC67kko5yqGe48HRMMJXPZshlzMfvFgZC5LjcGYYmnCj28o6zXW/G2Nw0q2eBQNUlJnBU8Di029++lMP7ZcJyh80JwD0uoDSEXMxWcQi7ZwMZfMTxRZe+pHP/PhP/dQ/+M5/edboHUxHSEVXZgH5mI5+EoedlVIYa+agtFcmt+pImZc0XE1QAJc3OiIilHM/6TuaDz4Ryy5GUfpzcY6KVMyxAkdA42nwF0cjUoPtisTD7S5MhaR4784rKyu3m+0FBYXFR7KioiWhkm0TKTle5EbtN3/7X+ycbXNWTE8Oqmy09Kb0SssVOMhKOgHbrz795NwYTUx1XnAwvaxYddXpjLZq7urkQ19zvTST62VQrpRpuHg2GJCN7MIVYDSRy5UQ5wu8yW2OCpjlREMXLy3vshBYtXJFT8GWfagwpUxRAigshnJ7WgV7tqxzBNdSY4jOLQh/bpE7goZnvEYoiVUlyaTdidZsn8+sNBrX+vN31zZuzy9d7/Q3Wp0bne6yinejkQSLOb4d6y3cN5SVpGhlql6BPONEZhsRFlQcy6fKKallhR0i4mIjUz0+kdSsxdWMlyFdfVyO9wNw1mejCFAjIgdcboM4eQrliIklpIHd95L7os+CVwQhh7gGSoVgRT1NC3Ggh/Vm4AG71RYOOjOj1oSNcm/c3qDlYIRhiwytiZwM22DG5DHFZZB4rIvfUTc86DCbVGHMxjhhI4aRDkcjzCyTDmh71VdXqgN/8iBvtzvd76kTQb8BGgUnZptMzeVxBSuQdO6Rq08nDkAOVjBFsHRhxfyiEhuKgRqolPKgBdISvLSgRgzOJG7ObHPR70gnqAnxRtx6PcrGZJ/V93jS68TzmlFPDm3/R5XDMRWtT7VM8I79PwOcn+/LJAYNMWuzk7O03FMw5CBsNrvGHYnXcSfQBUmayu3GwCuv2E9ihI1dWDWzJwnBFAaYZQUYem4QrW08xBKxY5Y5BbqZKKrp+GhhRcXQHmLpvTQq7Gp5bdW40T6ua4TnYLgjsJwXVft3rwnS73Xb91N9M4FcxMsZPmHGmPg3bCMLTGJkmYpDF84nZ0f1+RZqMjs8ZfZWY2B/e388e941Nasb/Z94/bXW+bd+/+tCf+eOzu3OxkVpBbEd7u4KahAgOrfSatMchdfYB4DBQC7qCcewvCHjlyzTXG7wZhbLboIOz4WsjxUMgWsSinA4C4TtN7Ht0nyrMiahRCAJj8IsDRD2grAjzKPQpUKtIozlB/dHWPLXrGexuB7nezmy7uG/JrD5shKzjMrFPFDwJBTb2okKHdZlwSX9nfFg5oCcVT8fz56SgehzyqIlOS0yI3ZBrJHbfev6Kz0Fy+7f//7Dg+0RT/fxhFw4Wl0b2OFkMn7GpYskuF2P0ANGEcF2vjKdQVsgQNxnbLnT6J825kLaSbhz3RiY1OOes2tZR6IR/ziUD/c1c2UUGWzFiXNSresi8AVAdPSMIzhWiMcFKAKqhK4UULmFkG3ksfOn6lbOHdUDBcDVY1efgZoWPnBkUqzT8G2t59zPHnauavT5cJL4c9/nFz//xZ/44y9/Zfdb7wxaDWXOCd7jsX0uj4hvnSYTQTVh6HVcfFcqdZnevPFDJ75iBt5S9ebijOauRHYkLN0ovShc3dPRdwP0DMCjVT9doS1FJ8NfLbepuHchDX11PV9+9XNLq3frcwvWr2kzQSRe9C0k1g5jg9Y3v3r/t37vy6VfF30I0vqXOy4OJxUDdoc+fejThTQHekHgAj39NoeXj3/gLxrn+4egUN1RjeliSJmPXIZlsVBEC8y7NZorFczcU5rKjaXzmr14vGrx8jPzWm0KZC3iKGn34gXpT84zgLzRt1xIAEAZlxeXlJTYRsOHxIaIWOanyna8NbWaOws8WvXmp5Tqm1+8vby21hNhymNy0lVG+PBINKVXI/CIhgUiUIrNAiuf73dQEfId42cKLUn7EF40RUBtfYTvM8OGJqS7QqyKlnwFtMthvf/36qerE6NAV6o5BLeCMBAHUhYeix/E5kzfjTe1/KNqY2SkBIiU6KuMm26TxLX8Y5HPiWnOMvEetAk1PmJVxuTEPPfnWDBnJMLKuRCanOAdiULs52GZgizyOiDGP/Tbs1hvWslkRFzIdX+w4cuWLXxcPPdkweeV6XQxKbPzSLQFKO6fCBJh2wUJL9EgbZbz6nVeUprJJawYxeJVNRjWY5Zk+jVRQCOlDyW8mjzC5U2FDQTdGOcPzurtiiew6+Jt1Hyv4Np0SPYFgYJAkW/sOhspB8okVKBMYtZFkJTr+pjrXEw1anqu6tYsH3m3K0q2sF6m9GSDpFEqNVgPFsOqgU6/8VH8CcPWh+7gfDh+piQQxRrQxAyCL9359p2bIq5NESuBZo13OlSP63B1UTWhHN1OH40UhiWCzLuMhR6vUAfwy4KwY7As7k53bv3m9aXVJcVGvvPt725v7VGvW632wYzAEdiEm8CFzF0VTcAs3VlabK8MVBpmaav1Ectu55hQsnck03E06vevL//SF3/y9o3f/Ef//PH37/canaPhiJgWzLS508GzmafMpzN2cWovbMyKuOIExD5qIp9xAVuJTOYGVijv6aA222OLPp/p1mo7Z7NGOkliUjbTVILmKLKhgnERJXFdAMczcV/rKZRBf7VWMEnvwSZHUZQjkjsvF0O4THh+y63mL2chDhrUorXhh1wLrbCY81T5KVN88Zt+uI1JCT4LFuJmTg9FkJ3XcF/5VppRr312/9nm/PJaTebf1m5jsX/j535yfuWN3/213zWbW5tqsstQV1VUrpraK7NDkVAV1y9bg9Ow0h/Y1GpPzrhpLUDbWS43uqt2LrIB2TmxCa1ksVdL0teUOhHHJwbE+6sRZxBlKKGs1Qk4kVeDtl6WUQeQAV6szTkQJWAtyzk3haYGSIbMZ+sRs4qGgpGT549y/epiBdbL3yOLBGLeUnUjLfouHAz3Ue0W6T9XXrc+d/szn/+Vf+dv/eff+b92my2R+OP9g9bZmVps9p5h7iJ+Rv4M6y20JF0o3UC5ynRq1SXzGkWsnARLykuriz6NKUwmYSXV7d5tNGHtWcSZYg9XDfhkmDiTEkqiF0hKRtXNWr1/7eard1789MaNV3BfAqaucq+ozyeWRfbH2J7NA+mox3/n7/1Xb+y/2Z3t7REz1aUvZvOKdnndBSwKA/a1AuiHPo02V9KldM25z/cPo6m+VCfVUJ+/4jxD9rJLKFydVLeFlxdC7Gv5KdqExVjeWN2Sz6r96vODfcg16AZR0kIBoVflaoR591qd1Rotml4uhIyiDYzQSVSgsjIfcEeZ1qOj9ul5vza7Mte+2V+4t7p+d/36zd48I+ziTMMuACScWRG2J8eIHx03cqr9V2unKu2U0uVRhPXfHoJYr5gYB8qODOlfuJ3N5yJow+JAtQJFIFsoSMbp/BKkH3tebq7gXglTxqIPCUcLGyOAiQbCAkvdq2S9SDClQYQ4BXUTF5EaQAkCTekkD4T7Voe/8WmWgDgKvM5vbm4uLvUWV+bdgMTToY5kk1weLtKMEWbECAMrzDJdq1rDhl20VlwpQR8ZHdWqQbmp5itoTc/SWZZRJvwsA+3N0pAh8uRI+TZPUAazhnHhAh+NG4pP3ErfWdfLwHODi87tWKBRnmDOXlZZDNi4/WiHIm8JsEIB0kuTYbDR7KdiZ86SdO0yKUoNCXZO/Lg7oL8Baey6h2ONU4W5oxJ/6gGAy1xW9CA1LPn1vRHuaVX3tEMAc1JJAKDB1CyjiNDIHrixscYOLBsYSrhf592f/shcU9hpOLavAEe1Zx2YtDdyrhNM5Egtr64kIZvTt91eWl5mZjFMrJ1LABhHB3vUbnspkvn0kMlBXVGRdsOdA3tRTA73FLLOtsSCGyAGXxfHNmAQGQLMzA1MByWAN8hFmybNY8BzQuE7x6NaJxY1djjVZM4bbdDsM8l/4ubPd/7KH/2zP/z+l/9wxnQKLoNFXN/2YtzZ5G9p7+698Jq6M+Q1K5bkShpV14lz+HyyvZ2o5FklHjvGz71+MvO4NrN1NrvD/pwYpPORNmE7uRNiwB5omiPUuDiSdCJfy/rPp/MQ/Wqey2QX7HG7B4IjueZfGshgc+7TGiqEKq1D1nAgjfksKH8Bl1wBG3eiuyGnoMbmHNGWZGxnRq7w9MYCEobtUjA1VUQ6c/0765/4wme++o++/v0fjOnGjeYeAPMOiE4QjiAamWuz1G9NiRC4ij+ezNUaPfU++q3uWrO33GivzDYXzjhxUPyGTUKJV50ERbPrhRDKqw5TLUc1uOozF7LQKxqe0RbpIR1NX43i6rDWLr6CAaISIFEbJNoYichQ9KMw+Ct2e/VkTsgs7n++vVwNxL2rvD/wzMykhkKyg1NktHE8tJ/x6dzy0ud+4ee/9s9++43f/9pbD+7fnF9WlG64tc9LstoZqC2YFXvJgzOgMlLTlHeUo7wnZ3lLuVxdMU9OqvMKbdJJd8TtHeJpFaQxYCmkkZcp0f1KTaonDONmTYddjjzaWV1/4eXXP/fq6188Pe+dnksOpEsQ+jlzyPww/mx+dfm0V/uTh2/9+pf/mVrE3WaXMzuKj6czPaWjef3FUUIBq28X16oJhPBlflwsWGkEznwGO587jOqHr1S/VwN2fnXP1Ul10cBjGcobLqhYsCScMyj/oWZd+cgDdCBdsSpoDQTzfLoa5KKLlrGWSUL9TUQiZlIzEvUXTiI4glhe657PLra6K3Otm935O3KK5pdvLixds2M1qnwwYisUvUNj1S6dRCYDPBRCy86PahUj34k9h5Q2IIfv7qt/jgmzoIZM08mi94gpFesYDIoMWtgSdgiSQcT09hJKV0BzpTqurlQnPjGtwNTP0BENjTs5aig1ErtVCPj4bELpsfun+tFcVcXljYh7DUQguiMZSqllexaEFozCnjMFUQs1jGfY8IeLihJsN1x+PkCWTYuqYsPoEW6ZgRTChImG9GCl+pVFTp4OCZMyi+VgSH5SChl7cBGykHzJoOWp2ShoalUk+7dqsDTKLJbfOTDneL7QXE9hsI4smUiuwDXDsh4YZH61W/oSiBCneIxiQeKqHg9H7P/YmDszISEBxhl+Ux0FDg0rRJd5DcaHw5lzwSyUTubFgFS7tBSqJG9uImyidSFKIMZ1baBAkthJkFTDlrJND0agIxFnOUbcAjFvZ9BmeQ4rLWJQt2tvKYAHIRW1bGpvj51cMQUgJ2H39GBovMIMOI5l2Freo/H4xu0bsHdyPFpYXpDBpdlOv6345PHeDmlJT1xxGNp4aNtghaOm2aWYiUgWzOlkd7i9P9r/7ne/s7i0DHrIKRjrEsd0kcOQokQmoa+hj+iuE5vZRSduyC8mYhwcDgc29Z07PmscDoe7i4N19oSnD3fWl16uvfbi504+N9nfqz/cnO7Ypm0ipo5JZDoZ0clVnNx76/u9levN5euz3QXzhQFZBzO17unBEaon0CYhvvNzM+zVNtFTNGz2sD6nwjNcm8piiXVFcBHrA4w3RngA/SKaZUaMOSTUXzgCX4KGfi2fheKWs6w4U1/wJ1QLFgBX6Bw8Ck7lCXPgb1rXGsyjqrmnsPQ0WY6CgKEuESBzh5VOzha9XGmW3NhzncEifJD3U19ZY5c43dtUufrmj//Iu3/89J3vP/Smdx+fb+3v3ru7dCx1/pRvN5IxPaqLxKD4ZDtVxSW6rK0KPbc7V7OzPDu3yFTA0SsayBtgS3GpUU9RMmQshUYTihzaXR0Voucr4QoyXyxaMCkcNPhsepHN6gCewCjcqJA7/M5SSrCYYCt0Q7wCCF02Xv31tfpXvev5HzWN5BSCHNumnwKt6g6zxD5F3M0qIUzHzi2JvP9X/ubf+Ienp9noY655LOPbJmnT49WFJTEUmXu9i6QEfiY8E1q9tWKxFz0gSJWZ8qPrzx9GWSa7mmrNpRlUlPyUzhU4GI4T0gaDH9yKs1Ky6Jky8t2lpdt3X/rs9dufbPZWU6pHQDl6mvsZGEnGjNC1+RuDB/tb/+L3fvPp2bNurT+aDDGFtOEF3hYsy7uu+hXE/nMdl0P+Ux7KUJ87fP3hB5GYiJNRpEIxC8KHuuW5fC3YcAHh59p67tR8KihlQRoTYuGXTE0O2EZY87uv+dCWyXBH4uxFKcTXdBpd9kxYVk0xiB+9fe9Gv/+STUMWV5YFm1LImCJ393oegTWEG9FW2VIcYVbT8dTumWir+jRWF4/oxB6xkwS4j9X4xWT8o1wF4Ng9d9yZzeACE/wyalsisCKzm+mYYv70I8/mKGouPhdLi+cLABNkhR9gZJSWqL/5xIblOiViAJsNzcrN4YD6AEGZzY1LzFlpNI2bAd9n6FJl14Fr3a6ai2GfooGU1aOUOqdyzSl5FE4fTZ+ckcEKbyUDFhO096RzjpkamcMN7qwm0uJ31UXMNfmdbfUJXCPvEx0yLYB6mjSqTH2IMy2CMwyG4/jVQdpERdJTWbmJXrZ6HRS9WDrMFKe7GiYdYZzR4L0dhPM6pCqLqvB1TCc6UsReh6qa49FBZgpFmBzRXm3AENXMzrejYQSp6RGXKR+kVCLpRsSpSBzuL7oyzPVepAAjAwHm6oResT1Mjwpn7XR7PUCj6OgngcajHul0WwdDHluK9Uh1ekqqyh6up9REfSS7f67p+lyn2z852ScLpJJDffbajevCAp88e4Qg6/wIc+Yxzhye49kuHolRN+1xRJ2X2ikajujT3ukI08xCm2u89/DB+HCSGDUpS/ODyUhFJ/X5544TjRvECuQBI6Qny+ZwyKLertlIZ+np5uYjAXbKXPAcClE7njytY/GtpfPp5szWaX1j4af+/V/53t//9a0fnO9tptiqe4l/4q96p9Onb37n+vSk2emTGDDgM9tQh8pTBPvHZMIjGweYyH6t3mWOYnedzuzVWmRWMgLXeWhjIWK2OcpCL3TYp+kNC3EUHlIwB0UpiJ45ytyUezKUkNmCTqELsdyYZDeUD3eVu4t+B3cz9tCT/Ap9Y+wot3vAX98LxvghcM3q0Qn/fCn31hqKtgBwXYRVhGPmCwIsp+rsq5//9Hf++GF2CRzV3n1Um5zuPN3jDh/zjADswvLJ6uxJZ74325/tDhZ6K0v1heVpwtH7Z3M9TJf9mbsxludYimyiFpW0JPRFtLQu4B4unJ6/f2Rk0Q70L2vcr/4vUMvYAK26NX+ycHMlTCzoUCgMbuS3/ASIamDmpdXjTi4erpr44U9PlXnK4xUoq3vkKCTKI+AidnQB6Vgg5NHxtR/91L8x8zc63e4f/NPfOHi6dW9teeGssbW5242wENLggwAMs0oXU2Ms8L880snL84/8m37HRQIVMrrLiXO1TLffksRlvCaWT10CnHg3pv4OBW39+us3br3e6qzsD5HZ7KaZtFWQRUX4sJJXLQ/77J3H7/zjf/5rR7Xjdqv/9GirZSNttI0ptHivdbbqYdXrMGAQT1+rz5yV4+rr5fzkqlsvrzsJSMvElAc++gOVv7invNCbShsWJhsObSSKUW4o3MTKMkn+u7jN39JfLwrBzJFW0DKfniY/celmdeS7/z0ZtpKowrjlJHMS6wu6QCZJ7PY1OTmWmQ+J2idn8zMz13sLL6+s3Rgs/Pirn+jRVPigWBZGYxAj1tBtyptALms3OzYIv2Fes/ub+NrxnlghLj45lkx8bKsm3y2Wd/y8+lDgk7FEYyuBFgVc+p2+lrFUMPS1OipoOK9Ogk8fhLOvBhSeFWMqFAorVKwqZ4lcUteIt+IwBqwmGbquPG5Qkisx9+ENqG291+kdsKkaUBMCoV3xrVHHaLoIMB6Me925wwDflhtjg/j168v8jfh6sXN2tSNKyz20fCUGq0nRj/QN8BN6xUB6gnO7PwiKmyUSG8bPstJiiZRsHc5e8RVXnFN42VZ6MXQThRWkIP1P1TNX5VitA8pcqVIiENQ54cZtbjYsCYIYCWUXoPI6FeBqJ6roH3LXFDO1fF8s384BywvLx2cHeco6sBUQMzW/abd9sLuluoigGLsjiKKiyO7v7nEcyGIiObU4D2aotuJdTyEP/dpIFAxlxVVMyls0jpkxlKhk6VyOMYwABAM3xiKWzC4sZVtD4cYkBVfctsxOdXr6+OlDsMKhDY9x2EAkPGzvqeq/Q+W9fv0GtZwFwoxEzjo/2x8e3Hv59t54h+LNW/xk88nyxgqfbKdzY/LW216xsLb+J9/8FsnIVpLg5vCWg9FQcPW1a9ceP3yKl48OJ802ZzUvbAwag4YCzbQLWcv2U0I+sjlGIdKKfguoO7aj+rMHO6ufma/dvLf75Ilc7Wtri9Onk0UMYJQdus6PhEUftjpHtNlaa/nVf+vnv/NPfuvx7+/yPItXi1v0aDozOVSkaf/RfdPUv3W3fk2xuCyt7FzF+sCOymnGvX02nZ3Mn3evd1vTLcMU5dcqWzWcHCS5lmgXFpMVUdZ3FpAVEN0pPvtC4rMicmNFDtxJnIB3yIuP2ICIag52FxaUrLJyNaSjeoJ316p3AwYbPIlxwK/EJkFOXoQM0G815HfXfUXLFQ2jKkbMYx3ym66w3xD0CE4SYBmNZo4n1uSodzq78trdhdsLf/A7e0qbqOT99qPau08OlC3U4ERw+7g2tzY7v7gst8gmEK2VZZtj2GuqNsvs3LaVgnK3YpKUdObXiWoABuVfRlwE7Ei5hRSWIeWjojCkm9xSTEmxblweEbdou7GKvU86PXIM5z1rrcIMPwFJ2J+wj04hiT5AMTbnAAgYCzQDIgq0cyaBQp5KnnkFW7JTYkTAVJ91322xaMQshyKCYCGzx8P1l2//u//Rf/jo4bvf+I3fvtVdf/Rwa4CqFqMXCVv8BomeZwqR6Ek6OBiZlKJHZBa9P2yJkBbXWBAh55EB/Gr2bdd4MD9Y8OX4ULxbJtbiQtapCSkoDAaS5c6bchU9quTc4eGsbRyv37h79+7r12+91JvfOJtpHxNOLc65xsH4QEgjYWLvYKfV6LbnB2/e/95/8l/+Z4/Hj3qz3d2jof6AL1vaJbwDqgtUi+pcZYxd/fhnOClk9II9/Bluzy1A8MN3Fojkp+d/zcWA0d+LI48/9903P3ik+syzVoDJ5FVzEnuscvVmJ+KGOYsbzVYqjJx2fVcWzv7n0/H1XvfO2satxaXbg8U7g8Xbg/lVsup4zFYo/R3zhtfQKIBTmU/YHPMDA2VcZcwmSiHT/JBrJpHsWWvJWmqhDJEliKC0OdiJ61VjCZ5lUND8Eg5XJ1nZZcA/DJ+rK25+7v7SDgzWGsT1Ye1EQbTNkfUbgdjr2BTHwq7l35yK7ZkVu8upqVOtDvrP2mcftYNuv6uWHVMhcys8gIJzAaMmM3hMF4vt9JudGEsb1E0sMDmOkW682xMRg2iU2LZVDSxl8InDV+bIT3iki5qK1lkEJl8BzzcrNBpimUd3UtwhFWEBJBGCkEdLPlk+Wa5elEVb+HcFh3xm4MWYYPX7BsFLDXcmCuHEVjfeY/lJTpadywIea7K3kuSLFR20EuOlVImgXiI99VuJqWb28RXQbVqhYdFNw0e9J1f0JnYEGrLo+HS4Gh2w6TCxgKFa+0atq7EsC08e20o98cQxhojCOJ6APlZtKwJJ1MbF0uDBoFkaVIjjcHdf1dIU7ubWHY5NmTiyWIjRdzyVnmoXQrst9U6mm1vPRofj9eZ6TUk/UU5q/Kv2rHz3YP7ZRK3uYX9+gGzqDBEEUqyur9+8fevRw01tjYaHke7NpUblUGG8MlljzKA7JynbQtJheY9Zi4cnT955tPre09p8b/XW3Z39zWcHI7BKrqgbbTFPrtM/Ykd8Lke1hZuv/8UfxVK//Zt/+Ghr+4X+Srtx/nTv0Vpvw3bSh6PtmZ1mtyfOFVQ6icI+spEg8TAL6EzqtUNw9tmkV3sx1TiIjUlZ0BGqJJkYA4DwJgMGulhofWE4QjByufzDlnJiKRRU9VktMw+yVUCfiPshymXtRbQOG3aPRmFX1pbRhV2niVADjRVm4560m6/l15zAHpCS5V8XclWKdxydCYRNRlDjRC3sPErsVTCuruJGt3F671Mvf+Vbf9Tq1fa3KQm1Pn46o9z6yeJqbeVWZ+32C4ONZdsGKUU2xTVbvdkEprVFFrIDs1YE/QWwUHxjzQxzISmUHlpRll56l0OHywGy1Uk+K3NBGUG5yJKSZ6vFiw/mtLDpxM+HCJbmC/+I8SAeCkvy0mhXAOPZvDacH3+0RsBWOx6M6FJIXHnVxZTpdYG92QzIfbjZMkYCSN4oUZ13T3bVr/ytv77/7Ol3v/3G62s3j3Yn0S4hNOqEBDk/ya7oe2hUBKL04Oowbj3Nhcxr/pZ+JCjaZUoYGRPNk5ouYM7CYyRkYZCmCB2Y9AW4JYs7rfLAL9x56eXVay/cvfvS0up1Pv7JFP1EBukQNlDsWKjH06GCccsrcsPqT/affP3Nbz969nhYG82qWmqDVZ2gAMoUhlelU1U/M+bAAQN+vu9Xg0jv0/UfPkL8y4w+/+m2wP3y+OHz6kqmqpqbyzsrdl59q/DEPQBRHRfXdSWB4ZfIX71HOxAP7ScbGUpqB4cSuxWeEkZpJbOsOhIY1A84O11o1Of782vd1hfv3LrZ6YgEXe/YLaremZ5QhW1EiS2QwCIPBZGgQuw6FpckAzOeRC9lXKRRKrdP8JfkGu6SmQVXTNoaIqsH3IRu/XkfxS9OIyeWi8aSrl+AiyB1CYvy9/L6xcWrr05y5Dm9DCcoEDcbbOXhwckjEfRDH+cl5PJriOtuiZXMBqwlnCnyPMyypGOEb+wd7OEqnMkS3MnseA7lRvqSSFxcQQ5S0hTZnlvLOO/4aB/RtMawHwwF3J1gTuYT/l5xRzfoN8pqXrAi56ARBhR9l0U/h+Ccaj3gdryDbnCzQVlE+RPk0kBCl47mio4tqteA+bH1Hn3zj+Zksnlk3R4GHYodU+qRnBc1ems2s1lZW1totHbtM3rC5NlhmhePpTlojxuSc3HkHB5P8ySJLGtSFdyhsZt5+cIsTKcUq7zOGC1V4gEyUR6PWJCthwbzCaIZUjOFGZNIQEXgFKicqmurdsTMXCsZRBi7RubnFyO+nJ3Y5XA42uf3ReYcGUPZdQPkRxOpoaSZugZBgwihKX5RicmLy4t0bOCiSe8e7KGFvCVZxaenK6urm++8Nz4aSzR68vgh4cmQ+4tLgSf4zCi/tXH9xvj+/SfNRm84Ij3usFmLCRPMRcNptfuHamyUoBTtA6xe6hizaXt2dvu9B8PvvdX/8c8uvfT60Q/OhlsP9T+VI0+OpCfVpoczqTl8dD6Z2PC31ezUbi6//LOfX5rvv/07f7z19hMK8NLSBncE/nS8Pdk+Hs7PnKy1zluymGaFqLAzt89O0G6TYgU366d+3+C8ZtyipsaEZff5hqmhBB8qhJDIrJBWhIALzghDETzrcC2Xk1AUuxfIouxZNQFzWWnhriEYOVwIrclHRLpy7jI6kl+jOOW5XLdKyw1XJDRcw5EXINVkTrW1SSDSnEVEZ8MjRm4sVGEYKWs8JalpXTXcbVx/5dbR7B/Jeq51Zu2GK2hNvGZzvrZ+t3/ntRc37ty2VThaw/16Yse0uhQjQc4ALdgqDKrwOcM07EIPwud02NILg8waKqMti+9i1P4UcJUehwcXUOSpi+t4Gww0Vv22HA3W1zSUtpE5S5PoA0SgWg2DtpjFXshYTgi34O63vL8gTwXvcv/FfRVL0ZmAx0q2ktITbK9CUm7v6OLDs+F8p/36z3z+5+7/0j/Y2d4cHyx2WjvDsaRQTlfUVW0bdEjf7B2OEpUZ8p4cpiUzc0VV4465OFBJhNraHpVNv8SbInAJ+0fpmffrTTu+Tia65Xq7P7/E4jxYvL527RPdheuLi8tqnmQNs7WIW2+1Rocj7+8MmPtsaYzwJ7f+/qN3fv+rv/do+1FS24tvOH0JkCBk1bN0RpeA3j9HAjfKeZm/ahYvPoNcF/P6/vXy0OVHsDOgzyxWx+Uv7/913Zfqs7q5+vqhp3zVmHG4O7SwNOvToYH89lwj5XL1E+oTVpTpjP6aUASSJ3pvuxnpLKwJ5EsZGWtLi7du3Li7tPSJlcDS5s/d5G2pACDviCoZ7htPB2jx9RKGwksJMFk2qT9xOOXinbFpj4iSSKE8Az5StiPBz0omeTa5KTM0C72/Gu+Hul2Akf6WI4O6Oj7+Eb/kcKdP0dtZyeXUN1cwBmzYWol4cDwenR6K1e7Md1YWeioWscJzA8tC2d3ZGu5vU90jmRqtEGOSpEi/0NtQXahaWhNaEk+tWhw0YIU4BGFNzw5tLiHygw5NUyY+utPbcTSBSRUzC2MoixY/xuqosGYNkw6rFmxVmDcra1zV4ZulASfpfazTJj4ZTcWWQDFrnrHsZo+gw9PJ1eznQTox3hJ5KJBzRb+TNXWU1UGcmOwPKaXd3ryirN3+fKsnkrltayDCMq3UC2KciokwWKuF8FXxvLRBHXad+t3Cg0/QRHMPNbxd92wYEYNHtmSo4sUYSsJuJf52O3SUusJhxeYsrIz3CAp4sOjxZaCCqK0m1LasFVCd291TQ4znt2dE7qw0ZoFaDvNLrU5pq0N+sXExiCI752Ky9ke7h0djohL/mQQi5aezVy4Drw2b33kP310ZLHghaYMngIVc2RQQSmL28cnqysatW/c2n+4qRsIGLxx7zpZFdjRI8VJZa4Qc+VXhWgYMMvKSVJRqwavh3qN3fvDKrWu1V1/aWL39aKw6lQitwxRVFn4L8NT2E9uQU2jPySHEg8Ha7ZW//BMrSyt/+A/+2c7bj5miucXhVTaeHG4fPcKzZtcpdPPrrakISCiduuleC8Qzp+3Z43moqXS6iMlTmw0nVnJntraPbhe+g1Gnp9G3KnoLE4IOhWIFjX2NjgipQ1FwjYrSR8gLX3G71Z0/+cWhYfdVjWXhR1ki6RHE/JDveUqbbqn4V7me1xXaihK4QO4CQh7bmDSTGjS1J4cAMpNp0ZCxRIXIDO6s3qjdea3z3g8Os/uU3SePjjvt2id/5M6tF1dVtW30m8lkJ5oJMq8zEmBsMJOmBEGDnthw7OilI3odXpMJ05XITEUyyKWgX7qeTyO8GlRIehgq2ROilpkud+e2QCEHnDCe8lRWia/W78VXYA+HL0svt1SHNl3jCCsLu0hFuQCAAVooYvkpC8N33XcdzKqXajBTAMeZQrhT6/wNNQztp3/5Zwl5f+f//h8LPbMoscyTIzaakxbOR8MSRohWs2ZEYyrdLW9PDwyy0KJ0IPiQt+fVlCWFNbJDhCQ/hbXbGiSFz5z1T+wQoWbOHEG3v7C8du3G7Wu3bi6t3JqZXT05VzGexnPE/M2pTyuDp3Uh5+fHvV63tr7EnL13/wdPnz5+8ODBm++8OTmdyBpOObRIj94NfqIIA0C90BW4dfVpdoNq5bcPfRrSh67kLlMcsF0e5cF8eIlrV1+vTnTT+RV43FLhiFbyQ6CTe92Qrl2ge5nNXL6AHQh+ZOOZxiC4PU7muBNx3DmB4dPJnGCZ6fHNhUXbAt5cWlzvz68OBkvzgyXy+dGkbY90lkl8UxxbSgUxvqpZOA7GSiJNeKZOF61H6/5SX8bcXce4L+0xy4leZxjBuuBiJO1MdNZ0sDjLOyPO8iywzucFcNLlMtaCl1nNHzgqXKnGe3XupDrPGywPDReBgwUcjcR9cRGohrnwRSNZTERNuywt9FqLvZu3NuwLirvgvk8eP3j03v2HD+7bZU+gB26Ey/BocnjQqsh9hDmkPfmjNlY/O5N+M79DC17GenEIrlfcVPi04Bq/WsNWZdkwLbbociXjxX3djBqBVHWdr7g693mS/YyT0BCaCxIRIfJUhMuojp4yDSgYDwsfKuCXKSjtG5qbTYibmY9olmkQ/bd1QGpdRRXG32X3bG7vzAfMs71u3217ticYbqdXhOCQEsu2Sq6ggrQ9zlnutijryuQqmmE/8xBkIoY+Fiknn4acfoIUagjssr2km+HDFHIMjHGdWJAxeNKQDEPKK1JiU+cjaUIyGVjgMfgGIxsL78rSojcWa//B7v7u1rOdvf3xaEyk4fISWnjG2mwnK+VB9A3QiC8QNaWrxDzzBtKQJofihKfHR831Fd7w0WNqCrN57A1gubu713z0jGdy3xbUZ41r6zdf/+RnHjz8F0cqITNttHs66o3xsDMfJEocakF+ViAaf5w5aMhkvHVyOHh6/+3G13ovtLszK0sbK7fG+4+Pm90msy8R96Rur8WEg3mmKk0i6FnJ6rn12ms3vtD4pa//+u9/4/e+cm1xqW+bXaUTmUwOR7v3321OTgfLo9byHetWGUX80vKKSccUHXeo5an+0bZAO3xyNpiO7WP2xK4MPICWHZJggrPerCwwr0B+uZ5cDmtE/rKac194luVZqE5Zp4CUxenpNGBdmjOokbVcfvG14CY7WLm1YnDlmSz3srBx3dOTekPEGUNZGCRPJdJMKpqp91qTOVJHGE2qTJoumptN3EXdzfzFX/zZ//f/859MpXnPjpZW65/70U/cfXl1ca3dW2qcNsR4oo4AIuuM2YluHXdvtvTQP6ppITN8prppdCE54aXpd0nYdVJIdPhruG/g46g+wyMDKt8KDy6/ZOTlHj8UmT7jvgx49FMEGOJmWrsSRDT+EUceTOsapKlnBbmtiHSu+GZ6CifPSSiWseBSgT72oRvqxNcmnfnW4f5wMjq4d+OFn/hLP/3VP/qjb//OH13vrJwmuzANEIaJ0roM6AgN07Uv1TC9NDNZRphPpxcQCjPzPvUOa+0W+j21gbvtQKYpUXM85tjuLCyurW/cXF2/vrS2PphfbPbEmbP8mwiUgVyd6HRrXpIpRef2vTtCUhN6wNn06PGjh8/2dvafPlHEfVv6mD7EDFl87YUz6klBzYDhAnfKKZGjWPEil+R4/jM3XDxWPVw+yzgvgF9GWEZbbq2+Or2Y8nLRx9X1iwuFm1YXA7YLzKhuLOAKclRvz0mZtsu2PviXGtKz9aUoY8kOiHJNEkljsdFYmOt98tZrtxcW762uXuv3F+bsX5ptLBS0U70toQvGYkHzkZ2dScpTdFatiQhq4WJYktyMhPiitmaYlUCKNe3KkZXsJouR8ub2/BeJoiWU6XJ4RlS6naVxcZI7yzIGuUpcS5lLTxSnRoUxBVDlYhqqTqrH3z+PcS0I63es103pRlhfiB8OFkmcxWvO9E93D3ZPpwcnC+2lXvPm+satl269Nn3t0f23v/qVP/zWt771+PFjuhfPHYFanKxoVJKRIVsQhimwB4t0A4Wsd6i+UUdU7cm5YCYKuN0vImEUriB81Xm+urk6gM0JR6Z2MGydj3yQAsg5cMuAsSxDiyidz3ByZKH6pG0ia8X7i2p5SybCVqFGGRNDpPfQEFuk4oR1OwAm/davpGBPCY+amcUXj45PdyxWmRxVVJiX+pW0QhoFtmyBiMOwAHSbR/y+QdpzVbKOs1P5zOGEPZmQoe8EWfgaBb062MfBX1NWPtFla3ZXvR7pxlQTVFbXIwHBIdYXQXHSE5i9xdckpGUODHt2k1roK/isNY7qAmcp6EJeJjgxP6x/nAKRFbyC0FMizyVCrV+/Jg94aWW587TX7raWVhfNOEOcXR/M/LX1DcbnpwiEbUIg/PzSoydPhjKWxifXb95ian76dBunFIMYuWim2e8trF+7OUVJxg+yM7KsNRIEjC7EEpCMwmoQ4DCe7EwO+zPbnfvfbfDhrH3hx+rL17tnilLBPKKe2EKFMtiED8/JcSf7YhSBcn8yzT5B8y/VPnXv5ZPz0Vz9nT/5juJXC+cnA+X5ZZTv7A1ZlA+OFmqtev+4rshNs5ciyrYQtiyFlJqR+cWsm7Ye0+PFAE5Oa4ezKnHGg2BWKlpBJA4/KUmGwaSLmfLHjGLBUcJyd6h0mos+5vbq8XC0MrNZ0hHMYuX1t7rT+ApX8CLvuKTsIfbhfHmRk7B1ww9NiW5q225qLs2dAJFtb3mzDAreRXMJA56pmZXFJe4v9UTrN6+tbqx2v/AXXqx3DvnEJbyoSqCIy5l/qTbG2U6z4GXLBlLwWbcvVkzSH12waguxcOKrQceXSa3So/BLJ4Xg6HQ11EKFLlZcGXB5OrIlVC9NZqGVI7wuoVsIgVYqqBSClsfzz5H2y5upBJHfCu/PpdJU+TX3VS90YvnqWODmYtJwtW5EkddzM+oyczpYaI1He8sb9izZVrLlr/8Hf+utN9882LNERMjWE1muiImlFQdW6IV3WZIhxJmYws8KLMyxJqsuEkny0tm66MvB/FK3MSAMzxxT3GBfu91YXF/DLtZ4anoLC7IP6m37FxGdY8yIztHKqlfEn8lYtZ5me76+0JWhqPjcg++/+87b9/Oi05mnT3ZGp+MUBceWs/KLZY9cgdwVRdRt1hibDF08R7FslJGXr3/6ByTMqMrAquH5Xg4jv3gcPAqLrb46d1J9BkAfPK8I8eWdFfvPzVd3OndUT5WLFz/lPIgpdD9qh692tlvttG8vLirXvNHp/Oi9F5ZxYhV6Md3jo0RExDM3FZECqzkRUi6SzoTOBkyIGIXQFgbHNDjRxIgfU2fr9Nxmj/AlG6WyM1P4UhjGPqCeCdwQa8zIp0UH5WAB4ScROybbSK09FwtgLIcyzFiMqiFlSUemKT9XICif1XirC1fnecRRHvUZsclXnQg/kcZAjhPXKWw423W4L1Fi4+Ph+fTdw73rw9tUixfbtxaXF17ov0YDW9tY/d3f/V122We7B6x4vBqzQmNtQzye4GoahnXSkPiAL/gWKzzbNf6Hl9iXhXkMABMLkQPOO9K90iPn2EAlsDgJkMu8V5oxwMV4UACIyeg/ddc9gR6uWgWWYdmMRCWSCyuzeS7VNPDOe/1i/JqnEXYtjrHtFEbZIMHiF8DE3F1qULSYcB3hxFQmUgkjANYGWSI5+c+SByddZX89wei1EM3yCM2J95plQH8QTbZAfFw/42cnfTTaymfEWAt9bNPDSK56NIRB4SBY9MbAQttIhBbzOgQ49pRZJuJWa0nS7+bmU590Iv9IGA7v8nYjNDDtMRlDaSKhaOdOP5Heaj8Thsq8JINo2ZZKo93aoH9w/x2JUteGQxPhsf3a/uLCMsSz45PwrUcPHy8sra6t3xgePHn3nfsx0p+5zRZ6MdfDSRSGzQC3jdxQ8DWxAjkvCKgw9LQ2Pnjc6vSne51H331jdWlj5sWb9e4iA7nMkdOGpDuymor69dmDgouzMMr2R0oTIDyHteO57ks3v3Tr1vTvz22/c3/r2bapXqE48HNP1HlubNffaa5OlooMR+FgHwzXq7WZBmdOFBbpqlVRkwI7s3fOBH22G3NjhCL/inmxYnwhvJcrqeq5NvwUO1umsfyYvyEbodUVvY8oHdwsj2MN1isRO7wmq/SyzbLscsUPec0HCKZKEonbrKWomRecCf0+GZIsal1bMPLaJjaaYJjAvgTdAjsrRcqlvfpK+/UXX7h391q/N+0tiqDcFZtYklsG0qPRGpVw5R3Bc+8rIWLGi3qgI+nxhRQRUhOpVGfxNv3PQjCzF6MK1yzEyAXnCGw1hIvVWoEsELIcMj6NRMKN2TLWq4Q0GDFpEvoXgocKpr0PHR4KBSiCTvirGXKAsocj0sV6kebLYxGL8hKdBXwMxaklHa4fCjObnFpB8qy72web1uG9T7z0U7/4c9/6rT8aPtzZH43rgo0jp+Qo0nKaLm/THlaXzvkpyFz10vsLNPJZn129fv3W659cmb92dCDRUJxbd+58vlVf2Fi9K6oayRHMx5Vi7pQKTsqVQZkXwwNWluWWEELx662JembHx0+ebD598mxve194x+Tw7OGDZ0bKDERF94l40CVSxNuElNFrLghU9cwnBpxe/3kOY9NIgV+El+ow4oiapZ00W0BcgZtzsjxSfoQmob1MmHkO8PneII/f8j2tgVs660aNFTrgfReMGVQi8IE7OYskn+CN0/7ewWq7vbGyfGd94/b66q2FxfVed16w/PRkcG7LrrNZ1YgkBMB6kp1yVO0Wv4xyzwACECY/rwYmxY+obnxiajAJSOcX5KOynzijf5ntkHMSGvUZw7dMY9/Kegyy6m0JTA5XJKpWw0mzAXH1LSJYjrwtwyinGbpxFOwsP+SJ8ktWwuVpfvEaA1eMJQ9YD/mNNC21ScnfYzrb2cloOj04Ptm3ubtVIlLF/nQHJ5Ph8XBu0NlcXVaGULzr8mL/9usv3bx7896rL33jG3/8O7/3Bw8ePWaGyd418thS8Q9fSZruQBW4AfLPJZrd0y1BAKBu8vZZYoKGHUCAhcBQlq10shx67ij6UVCfJZniC2IBL38nAlI4EvRlZNAs5mqpu5OgA7LlYQ+mEIGSGkeHqV7oR4CHYgRke+/gdIqYMcg2ZqnUhI2jRG0xHbNUTs42Njb68/P8oDzZFLtsPCBSQ+9CqgjBWkncOiimx6l5M9dVsUKs1lG0c5u0nM8Otp9t5mZ+u4SdMTif2Q4KmjiPp6827TUFEjSZrgthmd093E+nJUIZRbQNwdICDLzgAnuNy5B9Yr1PnpxRWFX5TNN1BsyG4ik2ikg99xpx0Ya9Y5RUlgVGlvjsOfsdTe6/95b9bw4nB0dH89TfBHmxTpIILfbhSBLzaH90fnT6wq0X1OhXvPDBwyff+PZ3t7aGqxsxcowOVPEiR6mTsT8+PO72+nqesGq+FfDReTvMJzQAdOOLVdQ/8tZpbedg2ltQrfpw5+Hb9783fweDWFQov3U626PzqlI5dzps1CZ2MaYj8P/DGSh0WDs8O91jEIdPBvhT/+avfO03fuet3//q+OhkXlp2Xcl6SnNj+jTZgGedLqM4vpytehGuk8ZRohpaCFlzlj94Qz4S6ZgcaMqCJyzeNIzslWQVxPADFBaMv4VyhB2hNGiMr1mcZan6hj2Vac0MVXSwrK3MIQwJydJKWXlliWWBw5A8Am3za3U5Z6VJ7NV5mBOxo3Y2ktl3UtvU/3q9HzpcW04Skb7B+zDgmJEbt+9tHJ/+9M9/adBtrC4w4gyPTnfF7snfEG4cPmpstOisnU6MUgn+FaoPkiIMSIomKzBIj3TLP3PnSrhPIVqhkUHty6P0OeMs5CfQIHf4sai6xlHZeiI+hOyUZ73DQinSQ0g7ETuVZxBLP7jBCwq595Y0WrWVbiBthfvm1QlMB9YQiHSodClyQFAt6kj5p29kTsArywT9manZqVPlHuKmMGMs+dmzrZ//5b98enD8zsz3Nw8f0ntlL+i7bCRrhFXaF+BwoLHOzZWmQ629XTcjedA4o26dzM79yI/89Cuf+fzK0o29p+PRFnWLc3mA2vHq2ocJouotxMguinJ8s90W/RxHsGw7c/M9F08Ph3Z63tnafvzo0Whn3J9fWFIBvlZ7PN5793BzCgcZyUMTo6XQMPBgIUIF4IFKOUDMv8zLxzJglDFDMqdAUiYuV+KiY1TUx0Jwi8CUID83QZtMTHhr3KvxnITaiTqIRSZJWX5DlEJDC+GWza+Ug61dcnOZ6syf6S62DMFogS7EZtsgT8wKzCnbfbIJo2Sts/O+jdLqzS+9cOvVhcXbt27xqLVgDZXr+Fg2EXarG3H1JfKN4zHjEZIoSoqFDzbQVqKX+N/SZpGUx2IFFoqWNBgRvoV27kNQpu18M9NOAgzTy1hSYR/PmcZItpluI8iS9w/8g2JBu3IKGDnHfNyStRtx3KMQpkyJn/2tzsvb3Blg4ojEhfIgjmHLFah7wQ0QbvhxeHoq93SoBCFLHc1CdavELds9juKFhlnfNpd9+Hh1cXC8ONg62EMHhEat3bvxF29vvP5jn/761//4a1/7+ubjZxjh8nL/YDjmwRss3vjiX/ix1bVFkQXidfv9zs6B+FQ7AgXFJS9NT8Yc7wKl9dukY0RGj/+FQAVp6dDYBpIB+rHM2XNPMgaekdRhXtPErwEjS0NSp6x0ZlA7S4Wnl6Ja9q2IhTOLKCGkVoRRY7oSo+xsLK1qeXHh+uq1/f2DradbhfzVFGAMP2x3KZpYPvsuYiLOyCFjR5WrxO62muzAAo29qfL+TienNzZuqI2Mi+utd7HCRwgUIWMw1pJd8YpObyIT4ntev76yAVuESbfmOhvrG6jJ1s72odweGkOcRZhoCj7rt2zm8eHuiYRnE3Oc3OLxwfjdt95dXlii5LM8Ly+tHgyPHj/dpVR3+ivbOwcEjsHC8ri2Tz7Rf1FaN26unTJwnI3nF7rf+s435pcW1jeWZUnNqxBpU9n9fSF2tdG43Vvod/sHe+Pdg4lkcAUEVq7dWXyiLuV0uE/wut6e7f9g/23JxLZq2N7eebz51HCAeHFlMdnGUM86jOKVaERkHk0b0etIICetrd2R8pDz9dPhsz85fFjv9F6bG8zPzS7snwyOz/Y6velcfX96uqniJ5PehIjUsH0MC/+42zltMs4PGWPXvvALyyu9G9/+7S+/8/TxtUHzdGFmf2vzll2WN59tTo6X7k5bt1+AK+p3NtlRha3gQsdz59POTG2ttoQ/z8ldS0XCbHM1OZvZVSQ7SUqQ6Yq+ZbWV5Re1FIaxTZHwItW5TPArqyzqVhZbWW0+c1ItUnI1iISSlUaQ9DQSEVmadGg6RMTjUivNB+tIXXRbNnAqKmqpj7bjyWyqKCtsZt3+pTW+8PYCI5PCB0e2oRC2c3bYXl+4O/fC9HAbLse0nJTJUH2+rNOzriLP6j6IBMC8WjMr5zVVPAmp3OFwEsMPr70gFV5WxhLCiQcbfhloPkPerR9IXAZrEIRiioXxRaVAAw3HT/hYCFuhSGFXAVYsRnE8E6xp75xbqKeRhyWfxh1aIKdxANYTKGOTZ7ryTBZt/ql2LjzZYmoejdlmpINH1co7wLdMBpYc0i5AVCP+J71gEXZ6njtRkF0kTkkVjK2j0e4wWv7Sv/6v/1H/9/7h2/+NGIrR4Um3PSdC78n+3vJg0USYJHQywXwSvqIeGYdcXhJMu97q2y+GOejaxq1XXv/U5z7/K9N665BMPjvo9kXBgIQYDtoMd89JWxEeFfGOjyaiGKwD5b8ocvxGXS6SJtFg78mjJztbvF6jrV3+4fX6YG9zv7mxtNc8/d23v/agtr9jrw0xqhgr+QvgYgZJedIyS4ZKKAkryASFC+BymZ2POIKQRhWZqPwaOltwFgRdw5q14LciMXlBZi0qminMjLs3XlGJfJC1vD04nUbLZTbS4YgYJPglVEs72VOBkwR71Ua06XjQjnBwcQj1OTvAI9B2WT+eyhpaarXvbVx/+c691xeWPt1orlA7RdDYtVwIoppscwz49s47MuMKWMEazTnXRdhanH/FRhoPb+yFUAwkUm+yjCJdr0acASpMEFHTEVyF+OVTdzGZ6qKxhocm5gBcjSlnYOHXcpKnnIBH3l9AX77qTwGhHl/wYDdGUix3XYLdNRJLHvBGsCyhMZKTw+hILgprHJ4mA9ju6YeJ2k5ctjUWC3BEsUgf5xNF8R8/em++0+w0VtaX6PRHSdviopq7cfdmj0587/Yb3/3e977z/QcPHvYHS8DBFzk5GQ8GNzvd28+ePlW2ScmLJo9JU3oEI0vMjyKfGIiE5BfwGF3Q4uqwnMRzJWMGDSNJJeMuAQwhbpVJITAJ2vietefXZHZFqnUdsfMVJvPVWuSszSkIF2tvij0ZPclXkVglo6tGNJslXcRygicjrZ2JYIVebjuebQJXlIlyIDJeWKZ39tqajBc0kWatAkNQzy0lN5o7WQlACZ3ZQ4l0oW+iaOAUYkvnitpOsvaU8JhiRc/TZDeUMlCIAl+U14LJSk6cT8kHGZeYJZlAR0e9wcLpfDyncph2dpV/lvSr7OCAR1aqbqwtdbFIWmJqU8qC/TmVuLla9Apo8v52e6HX3TkYjff2Bd5pYndnv15/ZvuPF199XTXE65v79x88vH//0frKar/VY53+5vfehGFsEXThoIjzk2zNNFLgGfYW6ScLghkF9szWx8Vl3z9SrGNoZ4DpuH003O4c7MpPrdmLGB08Qr+GCO55m9Y+dybcO9Sv1ey31TFuDQa180UlWw/f2bXH4gs/9pM3V67/8e/88zf++A+6c8ev3ru59ebjfntJ/NrBztPZfrcxv2HfH0aGuRkrsi2iX70Y0nnztFMf3FwQd41Cn9w/njw5nz1qCDVkrZjhhC6rtixxZ1k8wbMYOehW2BZZEJP0MKG7aAohBnnG6gsaJlArCmT8BRW1CmbGBOZqGgy2IF6OXA4dc0LFwFr62pWuCH+LUo7a7YcjYARhZ1imLWNjMEhrOJf3iUFkoFcPm7Mqcrjm2F4FcGEc9uPCgLsCMxJw4r8ZXJk45J9fjSAkt3xitxfEotAWL6vGg8S7wc0ZVjpcPsPiQjQivOcE39YXZtJIzh6kHHmXRtKbfEQPL93DrDE2ayYv1mDuB5ICvqJM+1JZ9A0XxtBX4pAHSZgFfALAcWlv80x5MPB0Q/oXYJTWCtfHjHEPrr6s5Yrsml2KlE2HV25d37h3uz7oPHm6u9Fd3BtNbM+wdvvO7rOdKG+JcKMe5FnklKDRnOub69ER+z9DNe/uyguvfOrFT3zqmPWolCLIninkheiKVMh6CouVHUYjgIZyZiwmIdY7C3Lz6cnk5GCPz25n77gojUdnJnWuJNFwiuyOhk8Otw5qAhyRZf+DT8YLYj5zwffCMvK3MMFyEnd9JumjjjRR5qD6sQAwbQRmmSJQKnOemXQZ7SOZ5Ymrf9V8garfC8NCtxJqztWnX9Z6NCHt0XZwjEy4KJbTbL+qeoaXiAyhJlCdukIhz2uKkV9bWrnen7+pgMbqWjZLGPSbw902RE5/kHFtMxJbZtSpOO9gV7g7HhaGEGardhFDtNOwB3AK8gXjrj6dgL7PHCFROpavF6y33F+AVsEnWJnRlXVRVoQXXvyU5y+OQCy/FvZcrvtW/VoMljkvS79c9RF2EkwvnwVxnfMzGYW0jQrwFDNeT2IHdCe44gdVyEfkDHYDbmvgVSxserK1tTX79oxd3+UUdWX+FymeeYSMur6+vrKy5tNmQIJ9nm3vwl+lHDYfP7q5vr6kInq/Pz7cF9wUBZgxIO/P/k6YJI2i0SbDYUiKdHitLhfExSuotr65GV2TflK4LK0u1LVgjD4GywuoM6iEN4t5js8YrAwgHP6Y/iTTJZlRrLsniqkcK3p1RrvF8obqTYwOPOgREPAUidWDTlzBSxa6nMThiIZ/kkrM6ZKfHNVTopmu39hAK9i7AJOmbakUGhWTOj3LjkJZQNOYokIToDpnbXY4pBvaLXh0+oxxKx5iUwMP8N/QSNOMFpA9EAbkoMwTVNeToihw9/INjBaWbOkYJGQW5r517nflKr0F9K6JNr5xDfjtOcewO9rbr8+wk9smVX0lnnBhxpqcZa6WhLx3sL88WDFT+zt2SDocH05JHbYVMVKh7KO9kZF1rxEIIlKwBwj+0ivDcIPA6yuAZGawLaoJs2Ey8sCGtDojirstsLl+2tzd7T17NjO/Od9ZnFlYaQ4WbWlb47Fmju8MbCOIb9BLKUSKbbXaTA4d7tHpRIpBw7BnZIW98vKnVJLe33p4/3tvPtm8sbwwUhPlcLv2YEz9v3b7pLOYSbALAa2TMM42cLg7FUTZmRnUBlERZ8dTmv/Z7LRnVApFzeyeyh0N+SAGl/VmOZpJiMLvIb4SXYrpN1wAlcIY8KBw4xCGcn/uLVwJ4iVcKmsRTApymsx8DebkiXxePATHsivRQuFNR7Yr5nkPxnsFcfN0zBOsEp308jwm4YjxjW0h2iGPDGtInzhD/LHs6QgFaSEN1tvzDw9O0avIc+aklCIphKYyNacX4daFQ0aAz4AZ1EKCM28XVD1X031ddkDdiPK+ZBn4XgYVahVq4tMwCxDzSG5OgGdSxAIb/QfriFkpcaVGozbCqvTEUnKkQUvFy2TqRskKKadYKv4WLQofzU2xSZoFR5ZSdUQO0C19wGfxy9gr3BvLKJsXE2oyU0T6Hd557fbrP/rJL//qbx61BOdPON9mxyxrHdRIYDLyoepYVxXhRkdCx87eoQLaK6sbK6s35peuLa5cs63dYHX1mLGVmSe++AQpJ8LZap1REN4Mhjk1ebaEdqo7rqtnZ7ITjofD6Z4Nq8oWOwrNJraUZnwso4zPQEajbaufbD17tr2Fv4BMSF2s9RUuAqmhOq++GnPFTy5Gz54MEG6q8OrDn37N7ATKIJ5fCwMOLgB3EPiiWad+LTMVSS1PhCVn9kmnl6+v5jccpy7fygziAYlGdVPEvSwawciqNM9xazJPSImp1RcbzcV6/e7S2q35+ZfXrt1bXllpsVvNNO2NKYboeHySqugxuMtZMCyNohtdyRh6l/AHsTNYfBJuEE0MOGYmt5WxlVUWeo0OViAJ+pWjnDBBXqJquRhYVKCsbqswLxiTV0f7z2daqNp//qQ6d/3qyM1ZRBevBmQ/XUAbRiIGAbtL+aeHfrUGCj/j248d2LhwLsMXL2GJhO2yO4vei4mJ5BhEsKynx5NnT562Oo35RYk5Mt0GsE2cFsaDx8wPFm/duwtrb92+/av/5FeF4+K4b7/zA+zjxo3r6kXY4J2ymVGyT4AuKZ/ElLWOgBBukkIa9lnM8SYCBLCVdKAc4U6F0wTIUD+UAiZdwMpJdV4miOLF2cXpFQxiMYQXKlFgvMRV3gJ9wEWUjASq3X37+4wgvVvDcxl8SyoRPud1RtE6OVlZWZFs61DXArQd2tcpb3Tz0sKiWA9lKcwYfHY5r/BiGQ+VyM8qVspahYkmLEvlLAluJLiYksQuA6ApcbAw+4zrNDWx0xZmQCqy85wGMgtob3ge92BsL+Ga2KZ0i7JhImAYMRkcLwZNjXMSrzHZdpvzvT758+Gj+42FeYosvu7XUsOKkZaZOpHPo/Gk37oIL4/ksdCVkjhmxBiP3eAVDO/IksBRHugKFFRwlAcwD/aGQOG2Mld5tYFUX+GRDAywYjxQVk14eUPFzK2t8/7T076I0Vs1Cvi4f3g6ToFie4kpXqXSKe9Odj1vkIzARmqnLLDO7KINIeoHk8bZXPveiz/b/de++pWl73ztj5qNk7YQDRXaTg63nrzLS3Hz7nl77Tb8mJ0l8/EF2aVM+S6M7WyGXXf9OocSqW961j6dPjs53MPUYiw9HxfFK92HYdZL1q0/lS4HpGT/MCeibdLoss5yi/vzF/UJD/Aguos85XIOuOBnH1mChZ6Vy+Und0U1WFM+Sf3WWm2PhHU2YxdFXLKO3iQyG/dNtkKhgS5oOmoeBswO1T+1fs/kn8aWDeAwGGOuz/Xtvib8TlolgS6e1NwB33SKUxlq6bQJIlNUgwhJZozSO9eZS/LG3BBDMXws52EBGXMGXjEJg41em0m/GF/OQCvQCx0jiIXcozrw0g9adCK8O1Q9CoKxaMSiKEDMR1Y0bNFQWdGBa7TSilRe/lTelmavrheQgo6GkDD7pTFB6UbhIN5CDGzWHjx9dO/6nR/72S9+7WtfE0APy7DJR7s7i435uYh7PaItI5CclandmSnHi69cv/HCnXuvbGzc7fSXBZlagookNHpKLuELDGLUZeNBa9CYGWGovPGqO+izV4vDGKoleDjcevb4ZHx4rtyqgWakFUIEQaxElJYRRErAw2dPdk73Ce9mOthVIFEmKLTu8sjVyyMQBhKqkLe782M/01kTE/kR4mZeAvcAL3+0EazVKw3mm+9glwkL7kOMlBoxxZl/jqHMBrsu+hycSuRczMT03ewlIZzoeHA0FY9LghTcstrt3FpefXFl3TYJd9UBmmuszrXnhczQFWSmRJA9aS4QQ7gSUxJPlworsuwtyKRpqkCE9yAf6KdeOsTsGCoiFcv0JaHReQq9vqdzWW0XJwZQwTKXLo/ye7nBlaBgmZdyUmAaWS5HgFKO6sS3/CuguPz0aITSamKrM+fR4VyrgJn7rQLgy3FOw0DOpYyILhqJhwJFZmGb6CJ3RycTupnZj5RvYQQP+C+z0631bcuB9969L5SAn0OV4ZRAUoRXcUrZtDhts3ntNsvCAg3s+9/73js/eOvJk8dShvb2t1+898LN2zcIMXHu6ExWl2gko0Qik8ujrxV/8gmA8lCthexWH26NFJuTyGIGDeABTNTzPKUJc+F6FC4Hw2jqUcdFjBX5ScUB0XEx2h4fuWadNVgpk+k6J+Do4GBPqUk0JfQg2xZRWRmfZQ83KwYsEza66hE/6b4r1ZyDYtVPvGdhcT4YcsLeS/01PxbeoSEQGLrtFm8QmcDLowPqd1xUR8qIE+TcozUPRGcvLZsajJO8QxrUHRLJXCJLE/xa9GAQOxuPDgvSJQHJxGPe21u7B9myIosbltq7SsyzAhqc7Pk6Oe7MNfptNfnnnzx4r2tTuo21zd1NYrdIMZsVixlcXFnd3nymD2xjwtr39vYb9c7K6opduXBf9aFv3brx6P6jZ7KVTk6XllZowPZllMDsEcWwwEqxVqvbBFSTCIfxg9jkI1HxwGTWEwFFmGjMyVlKee3RaGdrc1nU2JIM3nZr2lecyuTZIj5+yjnRCWSE7FZLoai15ZLN7W1tL9SXOMPlE9mMYua1T//o8sr1l17+rX/6D86n+xvthdX+4HhzePD0/qjFwdew/QOrYIPlCx9KIPzZ+e7JVCACwVqTrfnm+eLk5K1DHoDmmarbqcEzI0zOGosJ0HRlLqiEWcPkRRwDIwwTNsA46ExOaKUVXDDIRxZogqzD77Kwg7RZwBE5EWzYnm+OkNdQMrd2as0bMzP92sxw9kT1U9Ui9uiI0XfRNGDVhq+xjkgSREo9hEF6Acmiq45LTVhZFs2UrMbObFmSYJg2RQNZx7BCMfZ0OXRJ58MeqnXD3JWeWDtZ7O7IEak1UaCWO3JaVLuQESQ7fjE3ZMzeVxGc6nsGHXDkZdFdqnd5YciHpjNSAq7GtBTCFPh4JMMPr/JAvhNa0kBZXUH8NBBkh03OfddUOppueAa085k+G1qa8EMhCHgKgIQMepZXKNdB7njubDh7dPP1F+988tU3vvpdqbbd/sIJNJ4qFSPkQm8AlD9tbWl+o9tZev21H1levCGzSPHJYw5edXSi1J6I29MDvZm1JToXLCJTcKQp9BJlELWZMIzR3nBvd19UyR7V3SqjERfuxh5ARte1WZsWJ5/OtHQb20ejR1ubIv7BJkFL+ectRc3D6wpq5UJBywzZaQCZu5KmeQHECpSXnwFLxBz3Bu+ganDAxaBlENAQIgX53WRUlAeI/RTKWcCb52ZbZ6ex0OXSBaDL+rCqU7TevIZlar5sit48nCiPuz47uzq/dHNl5c7K2p2VFTsDrjQVDTsTJyiLn8mMUU+AFXIr6opTkIdRxHjhqUX+kiAhLSSKL4Vjktwc78+yDEbEMg13vLUcfnIYZujp5ZGhliMnBVkvf3n/r58KcIJAFyeRDQMZYKjuq647r058VsfVS4MImScr/H1Qu73gSJaKf5m2vCE0uvA0/ACFHtGAFVASmWZXIZUhSM/isYZ0LQGi2Ely/LOETVO8laTDFJkZq1O49N6iisTXbl0X4iw2RPcOT44TXIxe9ttf+qmfsLXleLj78MHjre1NI+rbLqDXUQNLh4qZQlKcfWoodqUwbRSyLEdHGWn6aoCRR8uReSmexVCjqIX4eCVNZEaqp7LWY6ug6iBSWb/xZaPhgrxxxJqNEMLFafCN857JTallZki+2WyMKJrvNCiAhYo5KAqrN7pD1KyLGDCpwtfSvRi69YQN1uFXXPBkKpAtFFAXBEZ5scfbvZ4WdRUWGZErfFEepE1qB6zQZ3gUOlsCcnSY2J6ZM/gy4/56VhjzrCAUsGNpVNm3aWek5bI5UgSgOLWV0wstivZMXAjZZ3xs26JNcYDsuogZT8Zyv+aweMaLzd1nOhA9W4ZokNu+bQoOk1VS99yIgid6WKpPr6yu+fr4vcd2XmIjvnfvRXsD2A1J+oTeu0cPzZT3eqQ8RX3PYk93WSvIz0msS8lnhlx1XYEo0ewYuB1hhgf88JTxelPOgQDxaWziyqdQ8jwNO4+RRjuRS+awKWTmRmSODQWME1hrSxvXN9Z/uj/7B//8nz589yGFdkGUpY2RRs/uf2946xOfpSwif6AXiKSeFL9DY7I1bC3NzQwWvEiAnWA9FOdkdsTcn1zkGpE0cg9pLnhpGP7EMmguLNTYh0OfmCOiJlofFN+KHJJao1dkHeZCxOgLMmDIfkPSshA9lEXLT1jQs1trb8TJXRsK28PxxSNKWbZk1TlRlcCbAlbu4fBRrDf+ypDRdMyibJ1NiRdKluJwqEfcFIG/A/ZYu3BLiab0K1Kd3mUAloUOJgUyuFa664XpmYtCQFB198MH9yest3oWRwtENO7PJWQ0E8pTYJUHM8gCBu8siOzm/Ofdvltohh5YFMHGcDQVspohBmq6UzHZgD1ruwy/6mEGVY5y3YMhlRc9qfpTPr3Hf2nI37B4VIyYP9NbWdweDud73S/9xV+YO+2/8bU3nmzut0BvImdLuPTiQm95denGresvvnD7VfXaRC43WwN1/MVDSLC0dPk1U3ymxikQwwx8gjFV5j7ZU6XAaj9vRQMPj4bWHIoR+9c525t4XZAlURHnRAM78gEKU2atudrD7Wfv7TxJt8N6qxkJNMqVMshgjyvlCA5VR07MZaDwUcfFdUDzK6iVO6UJaMuMpP0wYXpJkDFmydxYfsnvJoXx4NxWu+aSJKMEiS3ITGIyXYoTOJ4PLhQhHI2jkViKfr252p1/dTD3wmBw9/qNG6urSx3ZCSqr1lrokF1c4t/ltiQWxT9AgfYW9EuReuEy6BnpJCkpkzHqHNaDa8XVDrHAP06MzL37QuQy2xWylKFdACD4FtzLUZ146uK39//kisFq5AI4FyfkwVy5AvXlSblWvuihL0GvauVE18ouSXHeAFiglheHkWcxsD7pYzJmjAL5QMhsqD49HbLm8Y7wlJf9EsJ64MqYX5KIV/Yjw5gRHJME25I5NZtdc+ypfjQ+3HzyVBS0DXpKjEFJMC/pMeJCsnfF8uDV118RM6yfTx49nkzoUdk5z7bsliBSb7M/i1rA80TiSTqNdDAgpvC/YsolGRldTFhcxlLAqAOoSTAGD6bUujM0LMvXPwRQv6GQVVEkjURvhddhrdADAe4iasnUZhbWw5P6zIEYshDQlKT2HJOnF7kfc0V+mV2B11syg+Xwk7/ucd1FvlK5wuzS7lFxRcyjB4v3NnfE1Fw/sdmVLQ4MK3p5ijMHDwpB8V5mnal4C0uY8UAvWVoICAlsyCDKJJZ7kUFMR4ExmyzhxMTEpOgkDfRcmUxteq8bdK2sbULh2d7wyLRK9SX12CXYCjocTVS84g6Xcbh+60boAk1dts95V6KSczGH9exBl1JZdmXAuXmpNL67t6fmgAUBhSQywQ3r4t7tOwY4yobBlsP5yFaMtpchSokZ6yTaVoeSvaWgODVNnUulyHBiVkFqRJKMEvim0In6BbKauiMFKXH/9lmdQ6iPOMnhDL6R9iSZTaYLcxMzIsNVjPbJDkmHbb8DSH6i9QlUv/7zf/m14d4Pzr92arPibFVZPzoZyfNo3u8O1ifdlOyKGzCVJUW01lvjIRPDUVfpnPZZvG8z58oBnxw/A/ei2YIhNqzvSR+ANRx+5svygkIh7oVpBf1yHmplsZVfACwHquUB2FjxlbLCyYXlW7DMhfK0O5k6UqKZ+ruYWNeESpETF+z9TSaMIkvcPLcJqVvhzITrVPyDeDAZ4qQY5hWaKkNq2DbUymKyGiIrlHgut+ovvL1gwCERhQGEfgTbUQj9L0f8jtrwhJZhBBxL9D5iLR69cP2MCR3OyAp1y5vCehPTHPUpP3h/4AZU6QaukxeZq5j4ycWZRasnTmEjhWBE9pKDl3UcF17BfKPNEgCkQhTTvUhAvqR7uZhlbqG4np+ujtyp634oakf0yMTlMs7RrdX37HSP90dyEl9/9XP9k7Xu8e9++dd+p2eHj7P6vVuvfOq1z7xw++VBZ62hvrbYPXnkBkBDod5KjOgJ6ON/t1SxpCT+mJL41rEHAvBoqNbw1rOnVgcezGQmUwPmGAW6tn+wr9u240mvAUlcOj7eoQ/WvEKo/tHM2btbjx6dbUKdI/JfJe9lbNXoAkJn+ZK/V4fJ+/+x9p9NkmVpntjn7uFahU6dWbqq1YgesbM7q7FLGo0gCVAAoAFmfEEYvws/B81oNAMBI2FQxO5iZrFcYGdHi57uri6dlTq0a/eI8ODvfzyzukYsjS94K8rT/fr1e895zqPVyRFl6q89wDEgKcfmgqBnFisYmzpBTBU0iVGJodwjxbMRMEd4+IVf0ugM2TL7SU6UYRAJZiSH/KZfux7YXXR12btc32733ju8+87u/g/29rXREFrKpjM6HC1nKh2zaYt1hQDsBOMSmJVuVKILBzu7BpO88IsR6wTsPNs3Eb1F5ysuED/2dMuPxRlqJG+GWHAFoB14UBApJwug/CyXFJzfgDCfNkdwNWpxbhcoFewtV/s5wBbIbS4NUpWjXJbnvj5CM67DfaO5GFbuhkjKagVeQRNIHxkcWAbiNAkO1fHqesIpiG+qpeFDtl0vPJOxZI7EB9FDJPDhJeMjrjOOU71/t9LYsZ7k/ovjU2u4e7Cva0S9ediy31HR5kKhbrCYH9y/+8uVCretI6CBuPUtqT1eSxoP11mxmQqk8CFSUPkPACYkcDn3COOlIb6Z6s//NT0yNVwzT4p3OrOOI5qagL9wPtHxSbxUw4TyNAplp7fbpM4YR5dzyHhXkLbZ6SG5KzH43MqrW2EaUrvJiNdn5CihKaAx/+LzcJk3RK/YsFFP7eJMjZUZTw1KvnwwYPPb0cUFaqPZoTI3cZ4+4DmEnPeEi5Noi5vVWrltbISCPxYSzFzjQQ4ebJPudHo2S2TK+92L5y+5Ix89egtfl/2gT7NbFTHsRjGE/NyAS2sqm+mwwzkVr2zBJZnq3ukpVxE4k5EcQtXu4ODu/SspV8+OZlK9j4/cartfJ7p5uY8k1J2dApqZmjLJKjB87/5bcYAHXCXr7fpagD+w4VWT6N5sZessj4D4zHt7N+nUVxRtOStEf/LF5lOid+gWwLKYVdRQBggNzIhCqARK7gzuxf8LcYUNKu3uYLc6fjrhM5eqUU3oGQbJLKuNF8vtl6M7b330YP/B6Wdf/Ok//xfj06N37zR6O90XX39m2N3dg0q7T/bGYgbetbZ3vaupLUZkFQ8r/bsx+Jc6fZZs3kSVRGGjzJbEJTSPxvwMVaXcYzPrCFDMFHoV/hUCpTKEBp0pZA82b+gQBcEvkgvAsrK+inuAACGW3ILCQdSp/vZR9m+ntkWHWKY2VVtYxBiPNquU1JHWk3AvzlEcWSlYyIxi2YZrRKBiWimUomCZgVvkKAPYYFSWLDOLaz0iDm3nTBFb4RJvLo7iTgbnvqaadVCOVyROfl4UjAjDAo1MNI/2havybUDgd9JfywVRRfIQDVKwVLaWJ/tpukwwwQE3dqFRcjzBdoko8WrGUxtzyGlbepeFczPGoccids8sbC53QujunmPzSl9yJdIQvjLtMCWMX3zJtiDd9vnRRbM6/M5bP2z+jd3jT5Zv3373u29//3Dv/u2DO616fz65XvAB6icmVDXgS1WnoR7SR8U1K7asEXHYVdMFr/hxCN+xjQTPNd4SbgvvyTBE8dSzh4kZpQ3wFK9m+y+xD4KHLiEWF6uxLjVTAyA7176cnp1XxjeVAR013pYNxMLPicg3R1n7Nx82a5tP4V+bs2Dy7TfAXeAUiDu/+da7FG66Vyl9zmLwkgWEdoZxdnMHcIUmUIqjvnaiGsuimDfuK0PALk21G4GdrcX0dqvzdr/7zv72272dtwe7D4e7e73eShdAZv10THirYGfdltZSaTFkKTmyKJrKmrVaBFOZbKwzyrfR0njMBb5mqLG1gtfeo5LNUWYauZlziCIAcjgR9AttWRsgD4ZGdBfUo/iz3k0gx+Y+LnZg028++jcwDIT9CkzAAwTKEdzN51jhod188kDI7uduSPt1cYR27lJuk9EbDv8nQlonJ4ijRPLkYq5nwsVscmyz+bh2FLWI4XYb2mVgd9hju9vpiyCyx+hxFBFSkf2YHyPnmJORLepJtfuvVr787HMyb2dvV6rpUiYc26dFjwo8qoulYtO/9bf/lnjhT//8x4pct3d3kBOfbdKPHQ11QW1FQLzGsFS3KSlX7g0CvrNA3uPiQIGxE2BMJrPzO3PGeSNpy0xfQziRgpWRthk7xHOaoyt7rIP8YjwdDqDEotVq3zm8RVtNkrAIaKe9u3eggwS1gN+VzccRIinLrhse5BFFFbvSjsNNqBFZRHuyekq9rofUvfv3SZ2Y9hOAFgKJkpE1KII8yxWDJAnJDEw/cbiDwQNCmFDyB5zDm2MU4uVcUupBC4bE+MaoYTuVyXaQpJ1xPnvxYmd3+O7bDwngLz7/FJYeHZ3s7inntfHUKb0BrMg2cjgW3Gy5feuAKP3q8ZN3Hj7gNFanxIbiRPvq66fTq/GD9x+NR+P+7qCx18Ca7Im0Xsh0ErCf2LLYvKgXXLOvjk6Ojo7YxBbLVGHdy1fPf/f3fqc/3CHWd3eHs8Xy6dNXgx1q8JWuKwApYbVxo8ZJXlu1PxiqApfXqBFLggP0kVJGNdZiutO/dbm8OH7B7VfRMBIAIKt+YJKIhrtYkIJvAsBCoFgkUd0ddCZXp1+fNTV96t2+mtdmF1yyTP02zWJ7eK9yp8M93/3kq+nW86m+5ZOzHc72x59dTVe3P/qliu0L5HkQsnqfXarUbN6sxovTaZv7b/+OblHtdUU2niR/xo70JQYxnVz9NoFgJZBRmJkBhRqseLAPj0dEhUtkway5dUeEzoRlFL9XPDkykCMxQroO/DfCd6vRbvWqjT7lE5MXXmw37b62x1ZcL+fZn4U41nVyNbpZTrApqIc9JkOrqvNDdu3WjzISlgZ2vRDfjAKtv034fbgrgvesDDVyNJyncKa8mgPOkTdEXHHwUhszsCj0hajCw/IDOLa2Qwb+jflGtDjH7UpLgGIpuNlMOZwlOxLgMyVoVbm2obgeq4ZQnlzSIKA2CqHD85nn8UYBHgEX+i7/1rh4ilA1Cja0864JSPEtvAyk8T1suoDR/CIjOevC6ooHwOB4kzYKTrxXfos1sjMkheiufdO6GlXuDt+q1Y9HTyZ7w517g3f/1//oPzwc3tnfu8PE51o5H/EVE5MDehEjtq1XbvqZymMTpxNAiUtMXLKx3lLCT72WMTEdj2iThKYI6GI5iw8rmBCzByhhQvQ08I3mvZZfQ8NMewNpW1ZSbuP6enjn9h/8+e/+/vM/k4IijBfTzh2CPgVdIghz5LZFRnhTjpzZHLGAs1zl+PYb7HJz8ttfRfUpzNOC5DxBjL0LX5loCa1hS5hPvAdJPKYQrAeDPQicc/xgGrWtq32qwrry4YO3Hw0G39k5eHe4e0vmH6elnoHnJ+1eS9gpmkQJ5eN1qLgwPD+1496az1CN74KF4P7aDSjMh13RJkNHweFgq1nHIvToDPn1/MpUcMjXoy8zyMV+gw8nXRsEoQQtx+kCZHjkBugw0GAJWJcA1KNIm8irv3oUcg7Yy2FwAb0VTGDMKe8cRHL52sVyx7wtJFceEKThsY8ZJ9OJHp08n9Wl7fSmNm7PDufcKKK3g85w0On1cVeVgjpAwZhoS8VRFRU/GoiIEDB6iAFkUmZTNKbLqzPbvu/uHh4eJg2nlDESLiwhvwQerHN3f/+9999neiotPT0/37ZvD9U6ymlcZqAcDIksRTeZiyV3B4f3biH711NBP7NN/nBOGpPsY8LMR4ebG5efhMUBKi9nRloiXxQIWXtUV6Voxk3La8kLFRltMYc8ej4/hlLaIzMu69nknkeVNhkpaxKbYXhEnl5UJef93OEJvo1ITjvrluTeUoJJ3fZfmICCKpGMbHtSDpD0k4DF0PybeHoC55HcJhKpnazzKGlJtEX+uI2Lsbk8y1ITz9q+cM5rWckNrf2FrGCSzrhwCunTdA53QL4FVxmNdPTL0WTWazboXJPVstXrvf3hdy4mJ5P5eV0C0u5ut9IFVT5/6SVEirZWOztLwv7adIr2Q8+QgbW0IyfrzyHPGXS03Foum6uFDYkFKfp1AWkbDelM2dD+Qgvp27fuY7PG88XnX9lngnKjLzO4IAxjyw2EjeoNtch2q7EL03o6qWkDYmd5mJWdzMHAXvHphRhjN2X42qppoeFnW/297tX5tRYg2oh1Wns6lijtqHe6Z6/Odh/utn7j7/3Kav0v/1//9dOjFzupcZgrex6dPq9/Vt+zWIdJHDw/elrvbHObbqUvf1uzt+p4q9I5aDWve2oh1seL6Ulla2Xr0ZQerS8UnlstSxSbIDzQWha6TdFc9LwwSguM525IOXFqjs9gatYOYvtB2ApttjizGXvWV40yNdPua94QrZVurU4L2QYE9hNTITJYi4drwW8ifx4MinuJU1egXl8ReaAlYpeODRQ7UXaHM+Eunh2r1U8iaPPkws/C04NUISOS3IMMPQgW5soe4u5O6B3viIVRZpnvMd7gtcl6La2pyt1yUVnQ9DqOshBdH6uQXyhOpbxiga2EKq18GZSxAVx5XDhIqDlcH5QzXvfPDMIWHTHx0TyW4BxsBN7AwWDiwU2BHl5NmYa32E0maWWSpRb0wVRAhHMHV/BsPuOtSxEW2NusTrut1bCtCd6FAv/t/fbWdvNgfJzsDdMWfc9a5kFcx8guEbJsmyOKgxstS87mbHKlWyD10I5gc0nOMzUyphinQlLnNpMMozaqAvyCIhl6Kt7izEBMUWhqMTjq9eP56Pn0fJYgf7Il3GEDgm/QDShew839MsvNUVa3vOXKD5g3x7ffePLmZACbI6+ZqM66WGOI0aoGoik0cmDDNW5f73O6NG/pdrXQuG60sFAFRevKfqNzf9B7e7h9t9P94PBgT7JVszngXyBp1tRAdMa1WlAGIjKnLBX0IVTC2ShQGBU12x8WofwmClR2Ek20PovgcteXgYctJn5aZvd6BhllgA18DHTXBmlcEHUOSvuYeWGyrvPQspSRzfnZZvq5xEhCBAWe+TL3KBAKogcuFi/XbO62wUKvBM03QjdifiPoPReahsw24C33C3hhLQ+tb0gK85Xtq4ugzNWpruDi3R1twLa7gwGWq2FsXKNkUvgsh0s54hUKy0DaBEyGbZoWG2VjCbwWFydnL5++wKa5KzvDbpKjRDJd0cQ1zK/a7vcfvfsehiEj2vbASeUN3KJKhPZjMfNOBDjWyYuBruZFnS/nbhotIEj+p9mWM1HCJHwpA3Nk4QjsHGW6oKDzSvavDgTpbnJ0G42+KlLX+pTGUhRpSbY2tE/gBlCIwoRJhzvt7rIxSZ2vS4VwcCk/N06vxrB5BR/AUtvDGQtKZD+RzNQTbr5JYmp5CtQIFJuUAmwu4wxKmTfkCE1SBSBEEUPxdeIe1BHrDyShfXgRlA1CAuEGCdyCNiPKbMRjBr1dqQVNLEOJBMf9LNsjPvLgRJiwrLe2nrTqmlZylM9nk/5k8uDB3QeP7j973vjqT78ebKWEl9V/dnxyR0/agCIbD5PoJe3wyj7YtJa9g8M0GDk5NVPmkVXXRaTbbYwuZtQsu11BjG6vwy4/uzg3QviBNdoowoYwung+ff6SySk8UVmKpFExEmenYlFOYXl2gVNeNb1Yjs864z3+Zz6LBNxC+SE5firvJVCpVoafvBJbg2bvWqjIjDFXHdOW2KOsKlXCIuNS+gTTtj/4/m/eVD/9o9/75F//697eXoyqq+vj4yeLrepdye8ae221p9OFHRliitYVgirBNrpBpS8rvLlYtCGg9O2tDnkZory6uhBFROhWJSTMWgiT94Z7yS9D2xmsF2gXfoEYoWkWNgsf5lDCdHQqMR1ElfIqNj2VV5GuyhYNrykk/a3GTqWyL0otD0H0QPhxvTqLcBEOT/m8gtM4o2MmhL15QnA1r8F+cI2ejaiCcF7zMBkG+XUeb/hYYAbpS+CPCpBbZ+R5V75B5Ll5uJMJblAwN/c7CoFTSNwPYG3IkFz3mqOMREzQ0kZq8vDI18+rhwRLXY3gQcaPApaUCLqT8YNP/g/XyzWxXyLWy32Y9WGA5cjszIvuCQshWVztRYkv3upCV8aJaONfYPSDkFq1yM1r8fJqmpzaV+qsshAnOms053zL65upfiXbWv+lIWl8/FQaxOMG5DFWKAcFsdIAFC7wLOkYkHY8k9H4esbNH46S4kCdnnh2AkkW0MaDZQRIocAStAAenvFsxNgIkm/8B8R85t+oPzk7+vL42VzGA5NDMl2BfnT5ADxiEfytaMATIJUFcy5HOW11C5YZK3BDC+AEAmsG7JszeQ9XN68RclC0KAybVc994LHr8wbqRPP1b6vBKaFkqNmz72lja6e3c6vTe9AfvjPceTTU7aY5MBUJfku9FeaTtU2iUpe5pbEO35Fciyjo4pjJUuPN1EpiQVux77d3ErI4ZtW4ZkJGFN+YAYRLRhfLdMthCt7nTC4rRxbYYOM/iQz2TQF9PtG/4GYujsiHKGG7IPHmCNaGEqBcUHIj4HP7cvfgnbMu8Q4cN4jntWiCLgqZbTByc6bEdJ2Ir6X8F4LHgnOHLAF6gLUGpUpmMp+TvhMYo9Ci005VUVMXxGT1GKMf5FeRG84mZGgfeRNAK9EkRWUi3LKEuEeShsgPE6Xpqwx58eSZdCTZ0AznXEQzhY9+W7b+kiMtGc5ntTGysYqTAGnaRQ3EIGSWmtTIE/n7mGJFcFKowbXZ6+WjUCn0KfNKZ72tLRsHBT5OlaNQK2QGFWIi+zRwdSEPwhhDl7hLKK1ydSDqKe5gu83pdF5OmaEzHPHU3jrX62w8Su2Qm5meZk/2T0EYJcV3I4BNVgYWRxWIOcMYXa6r3aaK0tX44kwHCyqF3l5b2m84Che2wPHA8BhConRq41pMHrX7J/hireCKjJQkjhVmVLgSNTSs3i+iA+FvOILgtKyGhUJsdrxR0SEEekWUNhdEBuPQ8hZ3kp6d4fW62o7YI6m5O9y5e2eyntl6yVbtL18dH97a4Ua7c/e2R4uNy8DmIpPRrU/usDs0Dj9n5qIjuqpraAbdTtf0p5OvLKXIDo3QPodb9d2XR+l63er0JvOLs4vTe4/eunP/3u7+zsnxufEHb16jb3x1ASy/hVxxnVgU+E0vOtOLSntQVbVlE2draI/tuEkibGD87Pqqb/WJ4059fTZmYKmDw4ynx2dytJQ62xFzIFKwWk8//UIEYfdX/tav7+2Jwb362U8vty57pN16NT56PFlf3rtcDh5+QKfnT7EE0Jk3ZXG17oTHdeWEt/Fd7scbiQ6oAhsl2r1wrhpIBFF4otVBi2kq55oiSpAtIoxYymritHFhYGEWHX4RW0QBXuNesfd4+Eo/yBjBvpDS3bPheKWyq3K7UuH/ZImzUXRtwI514ed/ditbFua37g2IRWKhwEhfZzKm0vwUosGUQn9Ansvi4PUavmPM7pDFyGtYhpXxn7/wINSb0+F43xyFmfiEeMiL8BUcLzIyN0HtEauy+ShacUTBWEoLNoqaUaGTsAZIaFdeggJ0WU7eSEZ3iy4j/RzqR2J5zc0zKt64MDc06/K4ozO7GFPhO+YUlxxUB123JpjCfF8jWeqCdD7UsUSwMV04VpXlaLU8u1qNrxdHghXr9mW3fdleT9Y3c2k+Nf1hhJ82+kFmFs1GoeZNYYws+clkfKagA19g70qrJIylJ0ABo6NMcN8acMgvVOz3gbTlD+dkVhpvkhswzDpVklsQEkEF0zaXgGjr6uuXL78+fSWawlyPR7aIgLIAwa7yBqwcbuW1SJjNGkXyZB09wMPKs/PqooDFmfw6Kky+dSq8IXhaPC4Enjw/Q4UXHuQvh4Koxk63O2hkP51mulJVtlfX7w+HD9uD++lHsrPXbPev19qq1y/GxIJfZ3YdfRfqqnrnxTeRxpPx7hkfnkb6LtOEQR+A8Qh8rZDxuAA2ha3AUII/OJvJeA0NRd0o6Bn8jBwNZMsIvQPyCF9Xbq4Fe+Pn9+D4cI4gSjcpk0oGhfluNOFQDiSMDMitXQjNPCnzD6godAWceQoKJ+jdPkaN19B13mwUNOe9ybdB7nyVKWXgmUC5dc5cra4kc1pTHVpsK4dzLXkmhSB0ZlC9KbqLDDZmHA893RWHRTqUGLy7fhltLSpFCJt0RCMeZVrQLuQVh65eqZPZq2fPRUPZwIO9oXXDPFQX4c6Yje7DEJFwsxFer9//+slXRCyxZo8DhSdhxFgb6zrZKBs4G4E/lJkTVAF8ks5gDFA9cyzfGl35LYDglonLhaszcL2P8hywpIkqvlBXcutIBxXr4VImr79FU/vfJdeSPfSm05ltBpi2qptFUi8uz6RLWALSEXB0oPBcb7zmocXzTPIBBkDlV4P6oNocbO/I2f3i00+ej6ZYjA5O7smWppC/PgqvCFJkhroPJr/MoCMJst6h2yxpCWlBJ5y7wCQ2NPQqHTm39g9uMQHGs5GQgZYhF+NJ4qbF616w16Xw2J7g+mbLdBCta9BDJY20tod333mnee9ufzlaqEoZ2yx1/M67D5bTgWIX2f+sfysrnlqwPQhvyl6Z+/s7u+OUVQD2NXms35ku0GmLZsVqa3lhtoQYbiNW4wQfXoEl/ruv7ng4PDsdWUrWAmqH3Jine+YwVTshiihdL7cu58nDEudr9/ONSdBQKM34VMRXaquwYR5BHYfQ1JlqjfPrW537vXpXHdnN9EzpMFEA36yIiuj+y9PK7r3f+A/+o9//v//fTp58oVXJfn/bHEdnryqf3NxbrQdvfTcCIYTDN2vnr8ZqWmm5ge3hGg0hifqyKSdLOr7m0e2Opj4XxlToK5MoNKwyIszLTaxe1ggxljfObbgBJlAsRuRCznp1PgIEI8acFBnxa2gEytFXrQ5qypBuhhXVwNf6BkFkIl8Gxyg6W0LIdFuYYbClLuDmQjkbAIUmLVi0N5Uh8DocJjFacIyoymsRvqDzGstMxNMz7OCddcgN/F/+NhOJnV10C2zJM/CF8hT4mceFWyEOl8Y/VvgbkxfyFPjkRnhwCg9u4iLKGMI8DYPsCX/xWsAV6LsYVYEkmxZxZ0yeGO80YBoZFSc6hL8wXWIu5oD3dTFz/CcdsjKtCHJZvEFdzxNkveqp+U3tDxf++PryvHZ5tnU9Xm/N25VZ3AmMWzVCBqbv0moO1zyF2iAsYBwGJglrAaNG+stMzs/PNL+XYMUx4jnGxou6KS0JqoZBl5NO452ZhambRzwQ7ijLrOxswyAvNUi8LmU7NSwrpgBn/ZOTV8fXF/ApLvPigoaJRaIAeOYXrpB/rZbb540j/wJHUG0dT5TJF6hZusI8i+JvZOXizUu5R8FRDDLXu8dG8kX4WczaTqczrDd2q83dm61hrb7Xau31+rfr9Q+H/b2tmo1j+lgX46lslpAtp7j541Hnymzy3lNes6Q3wG9PlrgHpyJmfIRSaosPFayABDLIZsgbh1kYV2R1Bgm4js1wN9+Wi37+kksirKMwhRLdpUi8vI8LuziiMzUQwzC8lu/prjC03Nln4NxoozDbT7DeDMq3hpPb+x5X9mcs0MACe/V13DvAXd4HF8q35VO5c15I3zKJqL2JUW3+lpQTwU9suc0ww7rTHt+AIFJVQzbxCMaqxaRUmiq7h3iATJrU28GHKCN0NgqKp5JAMYzLuLdadV7R86PTp189thVBIoLbfb+cM5jEP22QVzxShtzspF3DdM5/Y09OJty0qbyTS5ImIbkaQ4BJmcrGQoW7mTZGkM/l8Ma3hmd18BuA8N43ORH6C5Ngi9GGIzmL9A3gLi+TBqUrQ+nz7g4bUeorvxkOtyXlOsN/W+3avGwAIyej88RiyuF64s3FXkl94hZsEitWf1OPO5r7uq7Hxb07t+7dp+ed27Li+QsNyKEbVGGB0r4IJe/dJAwT2LNHQ0f1hXtqjcM9EfPFmC0ofHJpUCKXOjzOgvjO7OtbDYlRWIDmIVzCbHXWua+8p3LgR/481jh13KaXp/AwqZtX8s63Dw72P/ygoiH8xXZ7ODw/fW4KW4PhLZ49SRHdPpLUwYG7ItbJ5bU5vnz5cnfv0BiArgyDga5KjTl9vb09PLb3wYLisqV2QOIScXsxsTNgZXd3++C2fSKGrfaaAG53JS1vndsf7LWKiztBE8X3De1BhEANlLsw8a/E8sPdoHFcjHCM0M3G9BFXQIS4K71u7XBvsfzRJ3/20w8PJ+++96v2y5Ugd7PVSQqaTkTEqBUcL+nolb3tX/v3/v0f/bP/+s9+93+4nI9u9zWLsRHE6OzJ48H2YUWJbastpahUP3cxdD2qo30MGhVGQPOycTmea23twUT/NcUeioWWww6xyhR0WxTyOywu6IRnWrZoPZAXSmbNs5xOee+zs96Wiku6Bb4MYfmis01CpV9j+657MrRT8OMrJFG71BFGGGGtDNODHbJU0THZrXBVsQqhA1/D/5Twy9/RJSZ+o01SsdMAmSs8NYRUgnr5KMQXKeddyMYPQkP5M87I2twyMvKbIxw97Ml8y/2++cIdwoUSusr9jDLyqCRrRIIGFd1KpSJnVkzAiN40lYwYdz8/yM/dN6mvgBkSiYzPa2lFkgvJOAML5IyX4UtchmlgWUjQHHEJE/AkYS+aBWppX4+3LkeVhYbLF0sCuDqV7NSorwS0aGzIOW1hw+skgjTUIIidNKmOBkMIClBOZ2dazF1e2rHoQnRH30OqPIyMbzZTQNISQg0SqAwtTKmwXUECpMtvCF1j5UVtIn1JG78r79PKjrh0k2Jt4c5nk9GzI/030vdtvl5gfJYgIRG8LYtVMCiwcmS1CvMr73Nus0xuGnBnRcvx7TebX4ZX+qq85BICiDKUqkE+tVLcyCKQ1Xyr09nfat6pt25V67eqzTvt7t3B9q1us9egycwUqa7T8Efpnh9qVtQOAycRLKSsB+70gn1MudpyKjWF69726HHjFU+6x2MlGUaxxINogWkhqoIQkMBRxulklIJIGkdUo/KaKTjP7o47wdevp+of75MsAsksZJw+OBcXIDHmPLcb9DHtoLLh+mOQgbNbWJtok5tbWE0DKCgY1PT35sht30iznPM+FOldAag7bcbiI2w1Z8qjNiOeCbNZlCJOehWlVRP8V7QamEQ1M5sEOux3jMZQD/+fG/oFoop7GHnqnEAWpjjAvg2xTCgLMMugadqCqEKQr56/6CRPVs/Baq3X1g4agGA4HksGQV8YSi+8fe/uzcvK8rj0vjA7U4/v1GO5fFOKlFkgWDqixxCfqKSQNzeo0P7mY+SWXhNlkK4vpm2cpTWbCaKTAoeN1CQ3rBw/3vZgaFJ0Z9rY8jKdpIAc+cqLln19M6PthnsYba9bHfS3RxfPEacjt03OVxbCRxKXGuGMjx5BBJJJ19VJtbO/fXDN7awE2l90o+oWIOtxnUUqmSGS/qw1vAq6Zq2Qc4SrNwFqiomvGq2OO+fX8MJK5Ygb2dVOG7w/5gVoSCaXgykyBW7lPJBEkQCQdkMXKC1aM2ajn19eNWX0dFsVWwdKVD689eH3vvejP1I5Ma1MZpKSL8fjRq89sQ3UpW0hMKZ494h2MfsXL16MdVwejXgsPNx82bcn5wd7eztPnjRmS80ja2dnpxZhMOxzz6EvfcGAyEjgHZSgoEBnAxZi097ZRMGN/oxNo4UwZh3srVI0CMm9/K7WnetDMilDLbK/uiUPIFIldCEb69a+ELjefj87Wu3e9HeHDyzjxdWFpifSdRX695o7GkqnwuvJi/6H99764S+eX46f/fTj52fHB9UOET2dv3ja+mRw995wX+4r71lSXa20zEwlKGRyfNH1vW7rTnVrZimX8zOqD80tRJpxwBXoD7rR/0JvVoxQiOUTFuePTC6M01d+Ucaey0whZFwMI99j3mQwltsiems8zyo2+Nn55pIyJSxJ9buK4LeK8QYCkn2XO42Ym35F2XcT0kfHqzh1Qoro1eNL8V3YdXQBt8LmwDRI5KG4Q8aLetMVy+VhmplI4fe5JHYblIuVXOStS7+xvcj8sNPcOt77cgHrK+Km/IWOHCHbTH1zxPBlpEYylTsbtoujtZR/XWoIRcPAEjMMr7CXOPMVlHZhwr1WBqsUPnQr6iiryzxM0azJOhwRYi1o0vp+P1nejOrLCzGJdXVWa6401dNxhb8BlGKXi/GT2Nykq+SWqzzvXZ4vx6NTaVZ8znZHm07OEw4TK8Ooo7q4Sj50sgk5VOVC2MOmQC4kBlZBCyEk87YEKQsyU1wiqwO6JoL94aE8A1a8GOuRKRIvjtQOz08shBo0zu1UMW7wJboejNssQeR8WUHPyek3+FXeUnT8G4C7xmrmTaRXGHlZgHJVmIhPec3GLFGORCIHnS6H8w4NXVdThDBdPhpuf7h/62Gvv73esh96fXm5NZtUtsYgRyhovmZ1wv2uZ+sFXqM/otTICEOMw26meISUz9pqWsqXrHtiJsCwGV4QsmBHlEmKn/8Du/igy3kaCVsDJH0VSVCmEr3N/aGQ640/4f9QRfgjZMU5QKwgmF3Ys6EJFsKjwX+k8ZNuY4KE4+kYrgkIMFK4nasSylZawkQcUxfMKCQGMiRhOQymaLfOubc/qG5tSZBYxkUa+OgkqZQhGW3+CYRdjfzjIcFtZeXpya/IEcRkyeN6ko+wKeeEvviGGeA2jUl+RbFxAycTD2tR6wVw9ezJSIRyTjJT0VmwHN3jNfFKO9WW/XhlE9kLdeSn+7sDparrtXKUKUknzFma/pueO4Dn9uH+aD5pjM9zkywO/hY9BFv2VCpJopamEa7jBzfEQBGoQecoldY9Euc61ptoYWkyFY0g92aM1Wzpw/6m3EiLMmg5AKpNQpbzlDPBEJFOt7fboNyCSN/S5Ri0yXT23OVVD5bTJ4skCw5HF7Af5xs/svNEgjNFMARnyD37jj3+/LMeTB72NeRA17UYuNn7tJi2iWJatGw+q0TLOolhLhbQ0kEY0wTiVytZ0IYI9rFjPVrwD8BdFmTMEkym89PT8252UugMtndfHh+ZGo4mBOygWsSiZkAltoBZa2YhmoCN8l/4flFZjkOpd3Z/8MMfvHj2iZr34xdPD3a2rwWtAXEuKU1swD10jszMaclHZ+caRJ9Pp4S7c+AgP1rh1duPHtjWcD3Wh7KRXRpns+HOdmJmeNB6KSug81QXkB1sFM8B/GTDrq9ZbcaruE2zl8WypQeZx3hQ5AREB7A4CzAxUy4BIusqqyNIF503UJEppkhux8bT119/9pPaaPoLb//w7vd/STK3PTODlPpPCqStsilTb2f/9Isne9/5hb/z1qOPf/uf/+xf/cFqnl5lkrhGj3/6yBbq8sssPcEpvavRpreXPlvXTRWKNLbOg85W5WJRE6QedOGmoUJJlVFZwgiJGLgbyVEIJotHJcUXIJGVDzViJaZgMGbo7WsxWNbY3dQRaQ9hAeUwuzaI4OaF8Qb7bcWgcVBLAioCImJ9E94UFhz/lr9kHedjSMn/xSYoDypsGmlJ+CAwkyIUlm2iGXZwzsiiMphKRkhNzHCDkTnCNV1hTTIY//78iNaY+0DkGMubEZOLKo7iGAMZXrf8KmxE9tt1JA5Mxlk2UfQ8pQRHchO0v9FVzN1vi2UEODAtO7rHII2nIaej/dBWjROUEzIkPWud6zlpRiHuiLpc2U1horb8cq0z+lNBpl71ii8a/Tfq15EeVShMtSNJjcW2HH3baK6m/H95sh67r46eP7FVoJ6yaID2Q1wT9XAzNBmhyNwp0NT3jesmOoqjcDBfwp3MuJzyWkbvZGbpqTEqYpukoi7JAGGzl83K+dXi2fzs2LahnleqQAWLeOTLnctL7uipeR9hBQabLzGJ8qysJmEaDcdIy5jCU0pCjGmJyQSKjuBIkkNxLq18Bu2tbu1msCV4Vj+4urnXrD1q9rTOePBg0Ja5qrfD+CReSSuJZSRonw8EDwyKwuJuUR0lbdQ50C75y2Rxahe20d+dStcjv4j9auheN+O23t76MSZpBoBH24eTTcoRe1w8rcpOWQiMRwm1cw7AG3d4EVyEW4TZpWpyUj2ajxsFBYOkQQ4yvzqX86LHi1w7onX7oNO/vdvdGyzmPQIQinb0GnX3i+WrL56/+PLFYW+H2u7R1OsrASc9uRmUMi9EnWLdwnE0hpBgcprvIkAXwk7CuHDJ8uz4AEKw5AuRjLNrL5gu65VrG/mYFd57fjaTBR2HkAmk1fNap6Itjff11bkGLNsKNgWHTaGkUa41MRYzn85lBV9mIx3bnpt1o8a+MwReW4USWUo0rw+eDUTWtdn5+dNPP+tv1R+8+/bJ51+Rtc1uzxYFdh7oDPrGfD4+5yXr7vRv3dw+P3klC1k6Muqf2+FACZNm3KXoswQvCVtylRhVRkVXMDdKsCW1bhQmwjZbkFBmEmPn0qHVJOC7FNk2PUlbJW4synJpyvDOjgUqWWVOban5nkzcykl2NJTS8kw3pr293Z39ruYk9i2WSpv4tIsgBwOoyOwUFEUCQWV1Q7QQYc7sV5iOw1vXF+Pzxx//+Qs7JFxe7g76PFfTkRyPbAhIzNv1QR4TJLWnELdx8IkVU5MzGAdBWlQni7qqFCrs1PQK0pZJRW7zO6gYRQbsY+KQ9/7+3bvbu3uKmIlk+FzWgR28QHm2Oup1lFrWFvNlIsHZDGxCuti5c3byrPv+Az6L5c20wdAfNL/42Y+3331XG+bKybHMn+PT485W/+6t25PRcjG/lFL17OScPHx2ol32dHe4jebQO8S6c3ffn13+ICX4lrww+04O/uE//Ps6+/z4xz9bXY73Dva/8913Hz952rpp2BxA+RTjHd5iElgMQT9ZXZ1OGNGjg/5ycLtThS1Xy9l6zC8xvZgLlEiRmI6OUGevt99q7KHmBEwVEG/Tay+PLz572Gv98e/9k/bNcvf979/M8NdOpTOkQJEfZBqANzt7R18c7w+7H/1b/6uDg7d/+7/5r54en3x0976drS6e/qS6nu/f/6Dev72SIW0r5W7vcrzUlD+tqUj63d3Ktvz5wYWNkk8+Pdyp281yeXm8vpw2VbFQe7RxzbqV3Kek1kcIWo5gF05CDYxXsxSQhBgj9qi0WWI+pbTfmkdaYQvVi1rzolI5jvStD+3gxKsanJfY0tzTsHWrZsOrkYYNiCIKHP9z7eLmRvsXuFRIP4KWolwcCbgRdQ7bE0UuIeg4vSNDYh8XuYiVh6ngXLCt8FRGLbLiQIwwxtTKhg1hai7JpMJmiJf0qEF/4gKVLdtmyK4Iccp4TRHU2g73bAOb+hQzxoc4ieP1iM7vttqHJgigg8O6sWqLEMeMJY6iT4BjmUgU1cKxo5ZpIQWFOWPxhil/ZuKcNg3XFE6O42VbC//uzXZl2VqOq8uxYunqcta8WjSqy2Z3idIVHdG2QJxsxCncOYEzI+7oqaEQkQyXAQ8D+btevdA6giaJiELhxc+tS23h8KBBV4qSVJSHRNixD6ZTDHOnDIowsZBA2bLVLeOQlBENIOv91MLUxpdX+3ZDmdsOZLrd3XFypMq91/ry/Oxfv/zkjIe0Y4eRMT5KqsxXM1v3BeQOwIhCEtPPAuQoBBhUyTcBlnckl+ZzdNdVND2qWmz9PDxFMEYqHqjYw75izA5dCGvy0BZ7zfphp3er2bpTb9xptO+1OgdiV7MFZEl9W6xCsyAqoTXQIV43zo7SgifQXBgBcE+PXkIKfJpjghcvw8OmIXk05uhwMW8N2dLFjM0iZDY5GBXgkGa8ygGz3UjUQc4drR90/VDNGthV2yrhi46sV0G7hspMNorlzZWcXjy7Pxywd9h6k9KLSF51s9fZQjjcQ4109a7vdq/7zX7rwKi3rqryYSrXjfp0/Wi3v3/74OSL54bKasbN45O+XmUvDoLa+Clmod6U95l/8JyEjW5TPJqomX7j1bIbdhwlEdJUBPNK1QRJvnWFJxKpHPe0B045UbbQ6BUrbaOZWb8g/EboCsbZFctqWVeny8J5RIggWqwTG9sbCCx7AVeSL0oeSzQV3qH5YjaSq3/e3O4vx1PI1Ol25DcTS3wGBCAcIfIluVyuBgv2uDK+lYqXqq11Mg+rfsX8bMF7c425LnXOEadtjmBkkD+mGKjkP/GsGPZsqyxzrlcVILLtMkPyG7/1TzESlN4y4ZIEdjVFZW5yfj5iTFMJBbAHwx7yG41tCnQBmYrID4m9eW7UOD8xl2geXKOlB6TXiD0tdnAYH1RtpaNEEqyU52qrnHnE3DdIA9EpnxK8JV0mboRCPYXhcMVjQJfNVjePLPqrESDtLOOWyLpCWwh+xWpczqYgYIdHZ9566y3j1+SEaDdV8YVktQs1UHrEHIJBCnUM5XoxOp+NTrvz7cqgrZNlUtZZq/OZjmYs+u3hPlqtSorSh6m9fXU5G09HtAUVWmWcPPbRNekBas9SdHy5vHPnFv529OrEzg1oX6Pp65vVYNB79/Ytc3n2/PjVy6ca2d+7c/uTn342bPer8Z2XfaBxMnqPkoT0DImuy4mPCTsQOKbLtIF1iWivL9FRnBvRSMgGgTqSJjvb7N7a3mqtT8+f1i8ajz//yWSyePD2R5VbfSKbMr7VG9rsCtfkXbJh32x+1Vsv93/hl/53Dx/86//+t3//n/63f+/tR7PTs/kzezR0D1s9ogCBga9NkuF5ZaEUEgHThpuqBW4f1NMMY/Xs6npcbw3YFNXKSGIj/hCUNJ7ipoBxie15H1lSzBjMx/dR3DeomznmiP6HU0UM8/JUKuPK1kWlchoRLUAb7sLxh6/NaH2xFsi6pEjKgivb6VRG1JBqZfpmlyTP8DhBmU2TrAiQgvvEQ5HBPqT0xVgk+oSAvIShG5kXbxBSTLYyl8jDsE0IXAguM0A95VekIdsn0dyi4ENM8tOzSPwEccp0ab3WLmy/jAPaGH+xzWAthM5/xS7K9bQMDBktG6enQWBfhvegorILm/GGqLBVHm/yOlG1q05VvMwGswvlc5UrW3ufsxU4NuqKt02xuqw3JLKJeuVG9AeinuGXeDEUvFrYpG1icx0lEjo2jy9G89lZ1OzVDCn5CebjMLYMtyyXpcqifSMRWeuZYAAWHhzbHuEEllhwlCzWHRWGVZPdwWkknHV9/rCeim+slFqm5rNRH1XXn41OXlzOrH1YVLDEPwRd6RBQHp0nMF4DqxxvxpP34dJ5zft6E0sPp1I1nukm5S+iT+5NbFat1ezjftDpDTGYFPWu79x0bzXbt5QSdTp7tfp29Wb7Rm8NoohPRiK7uxbwhXHGDJSiWty/kTd44FJ2q+7zuOlU+XKGnXUzmDicY976iIYtu8ERvPnXSGFNAvW5t6EWaRy9Raa0alFdOfUG0wt+LSSKO3HQQloebo1Be+3+znB7r9/uNlM+crV69uRr/aK6u9vNnR02bdduGrzfC/tJ2qmq0+w2qWjpPGbvAd5A9+jKbEyNczIGeCCGtQp9e+/gbDReSIdf3LQ4pAUYMDHSlmd1rRY20jcICo1wcWw8rlXEXV5y1qwyCayK6iObl5qCdbERAdIr9yA+K+lxtYi1zF2JM8b/gkS8gJE8/kBUVD26uAjGVrfF3RfnGnnG406WhHYiBoDMHdzHAUEj3yw3hEMffNSJ5l7bIl7gUETlTuv+vHCo7YFt0Rp6Iou1sYP5iDyPfBJMvYL9Mlbnesdkq7vMQ8MZikgjRlXo9JuD5z0SNwhPqBNyLkOhMRSAyzwyaAIkWUIpOCqtoFKLa/GsM7UFigQ7MMequKkD+kADcov9KZ6KyZtsiExt+DJu6m8f+TFOVSDA58wF7HAGxbrMc7zGdWs9snqOEC9w8YA74ycZZcmj9p1QMyTMGSkMGUVoxJHflvyURDEI28KnPBYL5uMt3yaowIIXhdreHvjl97///fPzU6ltfk031VCUchPtIBynp0GPR1sgyoQixsV4dHN6Wq3vykrQ2ELuG4+p6Lz2Fq++enU+VYBd44afzK5H58vRhT7hl7e69EubD3XM1Mht3pLklUSIGQd1MW8V1NBBs0rFS3aoHJ+PDx8+fPfRW3YUPn51bP+LVsquE/tKsA4VBn3zBzhcVhwkjU5xV4iA0EyxtU21pNj5AhikLGDyVo3xJQ0QKZfYVbv97vvvfbrzu2fPRjvrncdPHr88m0gUfpv07vR5JkmjCp/NtXZuN3RQNomF3u3eqXz03d+wMIvJ53/6hzTT+vX88defA+3du++1h7fKVgfJhuDeSSx6lkh0uzpsD9666Y/1SFLDqFZ3q51NXjmkvA3bjJpjYmYXlWqDpZgmlhSUKSu7YeQ+YNOFaxfeGBWPXsrFpQT07ObmabU2rayn3KHZ8o1D53KmxEuhctmkYcY0qioSxehupuvKWaUqvMmMLmZqLDxtojH5+EVhO6wMR4fw5Znehe/lfHi9g9R4w8+d2yBgrti8L6N0dZExboU0vJK7iqPjeWTRmwKXqpUN94WQfkhhoo1HBfGG2YIJJ2qF32AWzuuFllTtDNKvjKj8FWFrrIkxQKo8ElDZBeE2Bmbg1e56O2CmFdmqQznvfOt6Wq3Mb+ank7X3itIFQ4rUydZCHNQ66gQUygGCe6lPx1ykEdneyw6jAiuTEWU9QCZLso9Zyqggdsi86AJABBb5/687XBkpEyqPcHFZ3hHazkfPzFlqB8FLg+OWg8SWMXyhskVIaJOgHe74avHZs6MTz01kKMJG7102Ob4WTClHWQDr8G8cSllWOzRRF6Ra8i6orgdn0jhFY7WB6PZWbb/TvTcc3h9s78kOqVZ7q+u7tfb+VmvYbsp5btuh6HLFhS8DhABOeNjYiYwSCCRxyAuL5l9EG4g5FrMrDZuvVyJ5mb71MmOQi4aVgUc+ZNQ5gLSQR5lTtLAwPWpWZEaQxxvRZjyalSa/a33tPUXfX726fWfQGHZ4Uwe3Dis7A6p3fTWvLGeH/aS34l03N8lbs4Pz1jZbeCfN90Um0+SiUSH8oCerls2wnCceJ93Euub5ettuVba1ZGlIHSVbOyao4kaqrRnoDiN6GhzkcQLLIvtKwnDsVlIC6bqLacVQQ8bECX9eTErWT0NGiemnCThffkJWQf9yYOJBLGpg0DlGTViJB2GMAIBXenydg8znOLsSiIO/qMgkIv6DoEGjEqhNapHLqDzOrGtEKDl6dnIKJ/r72xQBRZWsD62SSP3I0MSa42yCufpRXrZ759U0JWYKFDUjAphoNEgCmB5pqESJNQcfj3ZYzfDx6xuZbVldg1Hvq09lerdkrzg6z9zNri85flWsCqjo1x/lRDAcgGhIcoJLSFifVm+4RQEKvqhEcr9NhgcfOLXiGwXAMByhKRcUC9irj04Sb16dF+bMOHP+JmlHG4LE5N78yk8Y9N/cx3mDFnSA3KXpdRwzZr2hs8Kmo1mFfynnYk/DdX6LkuOmT7VnSY+y1h63GZJ3hC9hQ1X1dAdjDrXwb1sr6phC2HjyWIfTBfxaTBZjQvG6Nkoz6NqXXz/p9Hevb1qj04ujo/HJ0bna4m6/ObqYdAd9OXWLJLImps77YCIX5+OTUzbuCc7S6w58nC4mdmf66ssv6Rx7+4cP79wbn3/x+c8+Jsv3t4d8kJGkta2NpRO2UmQwxYIMjx+DHBJdMd2QJRbQ522o4AnabDjBF8LAwb4o91DX9rn3bon1HK1O9rqHI3s2Vys/+ukfXcxGH3zvl3t333K35eyCMNAGoHq5aIi33VQmz1/ARu3L/8Z/+B/9vycnVxfnswtpVxf1k+c8qreh/96tUCsqstR0GJ3mJ9DWHbrV3oNey4Za7fXVmX3UyMiartVNRAEB/EGWcFykaWxFJJtGDhwii7Thp+HsyDccKUcsPyk9UHd2Pf9ac0l7MNSb+7UGlVTOmnWZNzSK4J2rjNbXI12xQC1Mqzpy5qYypuPlNmEEGALhRw8ldWKthpbkKAX9gskU5nCLzWMLMocrEG+vh5mR0vTCN0pmVjion24KNiOsiQdSM9leSRwLbVoWZ9yE3ks976YmKNFuZgCHhY6K0IoMDhB8gY9wXMQyRq7GkqAeQBTJkqGFe8e/Hp+2dBomryCfVCYLgvGIoAyAx6Z0YHA9rV1Oq1cTKsfN1USf9GbzWkeYBs0NePi/dTFJ5nHRjKlQOOVV0hQ4jxanR6/ms8l8nPZVUXyj6ad6KSAsCqLFymBfr9Br+s0c/uKBYaK84h7306Iw5CUVnwRhOGa91keO2zZeGnC8LfSiW1ETaAMpOscZ192+mMlI4WwqvxF+Ip4QPnw2GpRVNploQL57c4Bmnvbm48//rc8ow6l7pHmHtkRm+83GkFRq1vcb9Xv9wTs7ew/6g1022XVF9UlzScxVJGfa8bUu04/o5ZwsiULGH1mlkDqoXJKW1uvR8fFGAISRRYokFIG6YhAGIw20qAnlg0ETlFENvcuQCxwjaN0NLwY5hnUpe0peUTThVXW+FjaC9VoLyENs18zhqlXpvrPd3hn2Dvc0JoBHOObcHXTvfPeuxJ50RzEY7nW9frqqGprNkVtjnERBlGfIT4+1V6doJb8g48h4eMYKXvq2Wru7rW3q9ZkUU4oLh1i2WRfNUveQPN1cdyVkxkXjtmWyAQlzIPaxL6En4ZTt7yaXV0QIlhPLrDiUEF9N842YvDcsEYWu0sylmhp/hGvBG0Moqy0nEBDhknJKgCbjIKQachawAcfzn2zeSOpQao68CdQD2+Ax3RW/57tkUNSqHDvtfk8gcXx2ZiP4Zl/Ev7aYzHRNUjxB4Fpgm8Vr6bRAD9Ml1HQfppEbRrUID8shhSr+sXKEu2U188oCM5TY9TGJg6JeRaS7vbaIAR7AQhXxNf7ZdUVPPD+biqxvTdkPcZHhDcsM1Y1VKMnaJUqZwpF8pL4dneJyBWDMIvhTnpmJx8IGPk6k4o33hiwEt+QvxWRJ8VKBrd9R/3MYonvCWwqIN3HH2Wax01R2EJeG9faPm8Q9pilkmteT5DHDICwrXnK39tZco5MUexiDwDPNwa4/zav60akSwpc60lp67l9msag0yIwBqBCJpGwKp/WEjFpLyMz2jIYnTi7H8qGu1NC+aLTGWNxWp0mjfPWKgc3HjCza9a32+EKhUbYnlILH8QSPNLGldbBuLbyFzebE8xXlEDNOR43x5PNPP+NZ7naG+9vbX/7sq9lo3uvuYNfJ2DDyZXbZM12wKpo0T6CKTqUKrMAFH9DN9XyNVTU4NdWySVWjPiSwE8qGFPHVxAuvYnJ2OR+vplcDTYeXWzeN8elTm53ZaOs96aqDg1a1y/i25a9thzGL7V6fEme3464mmreGf/c//j/+7J/8tx//3p+0k0x7dfz8K3kkD6jcBwdheniyTg7ANhNJrlXm68qdvWb/o2Z7bzz6fDqWIrAtyiRHCloVOWT5IlbKUmMzsQiciW1QjqCsVQk/8FX54HqYki2NSC8Qf4Lsb9bb1dpFbWtSqTFb2CHMSgJYL/CRGJdofmkqBE0mV+vzSnUcQre6ZG1BMU/JYMitqNYYMR7nG2wgAyrvQz+hljII//om7uVv/iKDI1ZLsD/CtQw6F5QIMVtD3S3TQ+SOZHMyCAmzqnVZ3Lz3esvQdugwTHOrFjIpzMgd0xMjmoHEJ72ZjTH8y4MDEdBK2K2wTrGZFPJi4dwAAMB5tNBsqbWesX3sqnPDRwAv6GOVVaN70+LM1aGkJOfQlrxLXZdGndg85kvNpXbztU35GjGa+YzjRRtisQzgwtDQMZRSmglGkQWFswVCZl4Yi/cbo86bb45chskDsG8d+AZgFE6ImTRbqv20+N0lgLnWuMbUESY71+ytdru57gyW7f68er3z8MPDQXc0Vrg4Stz1po7Ggi22vTKu10cwKny6jOrNyW++zQprapj908lfeYODWvug174/lFfRIn15mG+3bJfTHurbMNH7RoA1kHbTIgRl2xZtSmtiq9fvWiXTAw4BwM2BPWUPc8ACMPhmnlHqw3ldE7R2OXwAsoDPeJCQdS3oD8UCQcsbH0comNMn/kpYGjsbHGXErGU6dGvg1uo1JVJWqQ8DG7XVe7eGtUHnWgCrkaomRnql12j1m1pJ20Srvu4wBykV2TA8RuHN1u6AqYVVVOgg9nuAW0VCtGodtSHcgmVcZd2seKN2/xc+GN4+uDyfVeerm8mqqtegcOir8/GTI1lAyQzUjyqOG7Ix8jiGO0vd+7goE/nGjiCanSgJ4BhMSXbwPXmAHJIWX2LhhFO8voQAxofn08QytPyhgXgrAnTxKJn3pDMzJ0pDIqmGzyeC/2J5frL5VRCC9YmsDCvNn2OpAUWkjNjebHp+erbLAp61zmtnPC879koyrASvYzRbBqowRCUjteTHtUv9UQR51rccFtHClcHG/C3nX+OeR5f0FtJMilm2NaySUbUq5+ugK3KwnNqoo+y4xKwSiecS8Hvi0f4EHkFVYDZLfPLnjvWhgjb6O6kj/yJobXaOzTDyrrz3UGe8bt4YmDe+2oC0XJWX4liNXHe+XECxhhiv77b5LXzu1jmKXcLfwtkR3A708yMrEt9HebrX3AQ79BEW+6gXD8QXi6WY7O0P8RY1QmSmRxDSQJoGHZ3u4uwiErpZ3+7vzkfnXDCMi7ShefVqaIFXBJqQWDYPPzoZLS5ndx+9qwvnhbymL55IlbG2hi3U64bsTl6qS/sWXi5I8xRU1254dN96593t4eFnn32mXNjAPFk8qKVN9HT29MvHxHa3u8f2nV0sLk5O5EvwxlkUcTouN9v7Ck0v55KuksEtP6syH1fmXMdtmhRbKtldbAF5xVas2q7VO/AQzsVKsFSy2+bTKedGxV7qwv56i5wiQ5lfn3zyJ1pkv/f+L+7deafSaVVWE1vO2GiIIRtfQvNmLl3i6Lz73uGHf+M3zObxn/7k+fNnPeVJ7eaLF58/2LYxlSbarZsFqdOS+EEMZEVGYuOdShNfUGkxifK+VvRsbFFTqzVqk26HtJDojcGiSBd8O2sXGelszr/+o4oX8Wym7HyqN5P3jI+SZK1eq5/kW+7yx1v1ymUH5l7bt3uVpk2JrxkM36seTipo3NSDIHdMv5Dw5iHEJxoOv4wxF/EMwXzla+i0oaxiD5ujEfqmEFu5XWRiSrOEIdF9ZhChHugzgj06FrDpBQLx/wVdPQBNIQ6XU25zee5nlolj+eBs9E7iPGYmMvcZ+RQc9zP2CqGD2ciR4ijhQc1fXdKA7K7LBc2ksTy9kfF2OSeJiWQ6Zad+3WItNWIz+XkEOy2gLl08Glv1RvNUXQ/lKwpzSVxczsRX8L4S9CENcFTDDZXFsAEoGpSJ+ntzYBPeIsg3J37+b3wOJY23cArT2IgkZMlF1esPtlm+8iWRolNyg+bno6vJlBkVZyYY9HrLZvvV1fr4srL76P17h3ujl8OXx19XRiLcM/YD2ZageAZUjjAl5G883hR4vn7NP1lUTOz2dlKU+JPvtNoPOtuP+r2Hrd5Btbpbq27zOYsFS6elwKwIdkoR06bJzZC1FVYIscc0sAT0J+wp8pZRJ0nDH6vCFuJl5zjAiOwM17fOLL9cn3UvB1wsXAw8YHbCeEE2UHZtiQB4I6xLvgQKuLZsbJHaZuOmeaUMr73d6ovpdju1drPabbR29GVo61Jve1vmg8gFF1pnSzPZHqNQ7grcBhTOUDnHie9aTANqSs/1j2SbFKsUVVSoyKVqHV1jKCQo/00xg4npe5JZ+3TcymxVmeIn11dHJ8/rLIwXSZckQOAKJRmGsoviqnXzxCoi92QIkyoilrKsaOk3c9FFhoqQmVWU4CQYGiAEQomcsj8AM1a7ipViVka+FndShhRUjAvEtdYiPsF4VKMj0y14CgQOEZzuru6eBaAFSV0uFqHwRlo6m5S0fxb55dXxq1fFGVbT0NaFBIacNQ5tjeD8OjhbMDu13J22GzM/CyOIQlwAFFXc8uHVJpt5x4VduJtXJCahzS3iwvOfG0psaymA2u53bi6329LGTNho+fdUzXY7ssPcF+Pww/iInI3rLElVZm7FDChklR9Jt9H5D2A26I5KN0ifU6iSsuEr0tr6bt7P59OIKR5/bjap7vG1Y/P8CNGOA9mNyyOpbLl/4ofFViYyuV8CD0uJJuT6xz+y0U5BMZGrzTCoQwM5bMPeSAVXaUfV6bbeee8dStXzl89ZoOYL/4j/+hY7eFt+zul8oixKS4xXj5eTc/R/OTk75YuzJJ6m16FSMVUEZ2fKqOYX06XdeJVEn55dHOzsdzTQHnTNr03iqH68WW72gcHt5HMLZs2u9H5d7e0e3L19dz6VfzjnD3dOLRff7fnpsX1Ovvfd/cPdvdHJbDk5ZXxssepC6iUYYZULUGHxWlI+D/74QrF/pdJLBVWlpTeJHAaQJvUva7NmrRdlVAYWKeguza3pciZ9S2LnzEZA6Zx8mZ4WNzcX48XlF1Tl6/dXl3uHDyu1zlZT75RGeinQO6KwtSQ+nP7k8d7d++/+rV3k8vHk94WiZ9eT9fTmxbPPeruH7f5h+BEk2aJYdziheLG2Lq62+pJn39qtr5fXj6+unlvG9K2EDtSkrFTiEVnqOH2lVxSBGCyy7IXSCuX4COOcsF6RPpJ71zNK5FZFiHdGtbjU/ao6YK2IwqzmTH9iADuWsRXPZrHa0iRzw/XcCoYEwwq9eHgk6wZ148PMJHJBuQ5658hPXOZCvsPcqXBxb978xSvQKu0jcgalFPvGfLxxplBQHuh9plEm4hHEbSgyhm+8kkVKRXtn9US8ojeDBp3yODfCJuG6HkA8eso36TKCl02BXjsg8EkKakdi8JhgbOOGJn5rolevEaIXO1V177eklf6shtvpJqiHHywnlcn05MUTHhQ1CLL30RoARgWJuzAepkw+tEwrILBBhPMzZ6LElK8LmPISUfXmyEK/ObAsnDGqhNtaF3kSsh1ash0ajApUaStOZfp8XuG3o7ltzrpKjStbCBUOn1xffnxy/okmlzu91s7tBwbfH5y+enZ5/EKKhxz6ytUssgKACvMpC7pZtjcj+Iv/1j+s2cSyctBsvb29+8HO4aNWd+fqpq3Wb7qS80xxpseFMRNEnba62NL5PZVcSXAUdIRjax1Krs5OpoAkdOUVa6VlsZywADpzZGgiCTilcbHO4klWeBsczkgLpwtOx76KWC2COTzQTIoALrZvlktcqaFX/bbM1y5PqSqySW3c3GlTXqSncDKKYHDhyAcioaMB17NDlYdjDRDI/TE5PitOaFzyCtYkFpn0TTFFJb7IUYsVCEgzs9ZWmqDGesOaSdCsV9afI4fxQTPfYvbL8EwMgGq7ZQOXWTNbHQvxqCCQ7K+ytcwgEUOQZEKSSDSa5PQqf7uc20fVuNwYoyQHgxeeRWBF6mBZycuFahGT2ZqmwZVOhhtSkIxCAkZF5oQYACu1w/IoCPviWWCHp94oQdnN4baoKdq9H5fLAMtT2MR4C+t5fHrK6a1cFZ5M6uNIWQpPu2X65BRFNwKoVuEv7Xb7DNCL89Oik0Z5Ltq6f/0iNvHmiU7CC68gZ1Ty6Uvmt3QpW47ihEt1KatWrbVPTsmoAmqGMNaOJ9YZw6gQPI0wVBdGkLuDihtaFP5anionjTlMJIHt5D559GYMvnJ473o38d6vyh3yvmgh14p5ioAB1Nxzc6U35LQrHa70W+fxIxUyHs8m1Lc8J+L4t6uSjLkML/vdhE1CeBySiXUj51mzz8PD/afPn55enHpWq9PY2dnWlvnx4682UArdlIbVZHNPmt/Jq+5wcHD78OLkpQasyAW9M7en9TafhRZxN/JpKEw39q6Xin5JZfcIOVx6jSjQv3P7cGe3fzY+5apPqfPlhCzUFDxpQ/ZXaHTtCowib9++bY7Pnz6hzOhtxRzfGQ6CFdeXWp+pth6KiB9sffX4aSLZ4X2gCpIxhziPnAmialc3G9en7WJ78xxToHkhCT8aKZRfyE2RYUT5axA9lCr1Q7zQ5F7bZsxTmrwcjNHijH683d+7up4+/uqnVKT3FqvDd79LaRdGFa6K2ZVsjPAIrrrF8bw93H70D/8RLvCv/rt/Zjv0t+8//Prxp2/pBtnqx8kW+QTXEIrMhe7sfNLDOw7u1vv12uT6fDLlwr/pCDlGkJhUBp3EF4/zq9d+lKw7flU4vpM5TL8w+ogoH2F01MgR4XpTi7RRXKQLdK3Wt1PT5RKZwR+0GxIM4CBQbi9VpsAx3K4IHm9yoA+YFtH45nB1eGPBqOChI3Ims4pMyRiKTKb6xVL186AdDwt9yKUoOhI3AZ/ckdDauJ2xrM1QIsnVeKf+Ito5BkfxB4SIYSn5nlTMc0DKz4PWGQtVlzrHqk/SbUWp4oonuVm95GquruDaTJTkhgBW7nez3Oqtu1u+kp7LvmBFiZip81c7mdrzdYvNIQgoEWB8cvb08Ysnj2+kdMgbcMBRz4sRAmzsa+RWPGpMFMyZIY8XA68puujNYWJoKtP9646NVAZG84h7K3tBMb5ZvLrZzNmNaORsNLL/GxBDckm97eutVlffb1tz1QDleDz/9NWrT6ej2Xyw8/5be4cP1ODJG3pK8MyflSXhgYCnWTvsIW4f3CCLFfgFjDFEf37U/+Pvfk9a8najfX97d1sDw7OL5lyCkf0tw7UvWWq1tfQq9bxzoql21VOOZKH5G5RwXy7VTzMjpJmIoyVRzFNEqlXbmmVYmXYw8R7ADEwLdomKxFPOa6OU1ebA9C5ZQrGrQRKnvRY18C3OqGI1doYbNau9wUC2lOVjrLSkVh3soFf4NrqaNjq7W3rmDXdq7W5Vitf4fKbqbra0F6t0qnanA0WTE5kiuXBMwriVgoRhMDloGkZssHi9hkMBTNFfJC2EPiPGDJf1xZ1n5++CGFacRGQ5wQO+NHRO5UUE3Wb/7v7DSnW32f34D/7k9MnprX53PU+HBNvLMTK4eEEEzREx+raAGHhKBrG5gnJTRlQSDdGRNqizS5+FPPlePJJ0LuoXC6BC3Y/lgRuxxiXom0W82SntjdUep46eY0B6060JDjZryjiFJ5PklS216U8Aqt1SjPLU2ChES6U5JYrYhjM4KR2RZsrTj+9f1GtgaN8lU5D6ah86AYvJZM6IFiZ5+TLCmzudgIt4W1fJJ0CzcG5m5JvxJ5AOBT2sZAWjEPQeNzRJeW1Hu4mkt5cvr+U97O0MpQXNq8vBzlCvF4LBfY/PLyQ8EoA80NQpBcHuxmgGIkBAQfEeRFO+lsqktBwybp5rWY1qI0eLfkdCx+vO6s2KyzZmlglUzadJYoON8WnZRJZATYQjwtioKQWuEYKFDKlwzZ04eMLRSv5UGoxY0csY1gWdrIUimDib/RQ1S3GiXBvbi6NnLuxpIslvZ5lsfa2Xf6///jvvAy7X7MOHD/fFdS+Xzx5/cfuWHUyGL5+tAdO3J89evdQfSo5cr0/Utvo7q5svtcM9vHPQ7OiPsto/GIr9vvfew0PbA9vAYLEWReZsNXbR2cWyNtOc8aoqWfrA1n6VytuPHioZ/+qzT0jr3b3h2ZmFIO2D2i+fv+D+gGbf+eiDSUo/JhwDMsWMejEBWrwErCTb2NhwdvLyxbK66lV229XtynY86TEopV5ZJeuy1LGrm+qNsdYuGOp6/9Y+V9+Tr4/DQut6+OHfkgd1yhx3E2VrnJ89/+Pzi1+8ubr94L3K7m2ah/4ftVZf+ZiY3KB9eH5yvhqfDu/d2v2bf/Nvd1p//vu/8/TJ8wd7t55+8Vl9Wdm7/x6JfzVb1Lf31o3mal5tDQeKXSqnF5VhqzZ8d6+xfnVsB5hxZ7iNrBGYDLOmCtW6fA4V3mGRlpj+sGGb4fYWO1wg3BQppVUrLMEilHoHkcm9CLAiMUoaBhN23cT/eMrwjuS5RrGPMq1FJm0dE44PiP9D2iuswUvStxyvDnNxv8j68Eb5UMRNYm4xCHK8kcfeiJQzua7tfpWgF0QXrxKyjGs9BXUaNRIGnsYZx/yMBgmlQ6puF7EmzoqlTTgrolomUJUsiyInKtiTO7hpMR3I2xTIsHSveFywfyy10uXwr1xuXcmrmqXEaHF2NT1T5FJVSlQnaNlic8SxJTYQh6BCZLxUdZkiz/2DPIUvRPL9519Pxkfj0UlqCPUV5zcJ+Ekw7MjrZtZFiFkXfvSoA0YJglbIfzTuApjCxr1DWXEllMO1UPr1+zy+ZMontQ2xJiPVysHtJE9cXovI6MOblSH38VZqlDqApIFCkutVqzq/ufyzLz/9408/rdy5M5HO8Pxla95r9loP7r97sL3/5Wc/O/3ZH0sCMLQsJQYa0YYf0jaF0kTFwwYTwtjoUWVY9V/p2nEFyazaJ8cEoMY6ndZWv0s/HaNF7eZuMKOkhEcWWUOxZm3nFiofrvjm2b5JMqILib/AmqhJxXdSwv2Q2a7F0kKIHTu2S+JmOrH5JUmI6PB9lTRRfbal/NjuGEQ6jYur2e7+we1bdzFBHJx8bHTECKqtflvBJsVJLp7O/ZNGWi7UdKXtidJ2at1tFVVwU28ggUxlrcSxmK7e05Q7lEEEk8EaXViQotDBX6KX6hTtCbjQiUWP7sRTmhxm7yKbMx/vc01iOHHg0iZCFuWtuhStNPEkso/6fXvv3ve/z1dwcPfwiz/+s+cff3r8/EmvWrPXecwF1qOYWfCevbxi6mP6MtDVYpvsRvQiEmjmVfKnV4MksKFdTCrgJPbjKk8sxE1cGFM3CwsZIz7B3v+xsmNuSykkUaipiDr1M0UVISbXgiuw0USKIxEDouxpmCclTD3i2iYNKu7ied6qY/0LUZDdlIKS7S4gDzReVoaCpxwc3Frxfq7jLPJz/JFqgMh9JIOcKRoL49nKhKHA7A1JADXRazz+wVmAOLU+uvOjdGlwsRN7e7u3+tt7XOT8Bl99+TXR3hI2Ll2xhEt3NCtuKw7Ga6kUlqfQjSRN0xJBoJ1Iyy0HLPKvezpjGBknnvvaSo49F9aaEsB86zVLHhLJq4+xbqGCnE1zdKOgwgYn4jDx6Ewi58qv0ELUU5zQD1PCaMonx0dgr+IoGN64Ob04++STj3/jN34Dj9RjpNe1n3F6VOIBL18+/7V/+3/R3d/5yZ/98eHOgF73/OsvTs9HkffSwGtymKdnE2Zla9ho7x/u6FX51rv3dOvc0fH14sjyiCcJNEqElNI3mcjD0uujOhOXZavijPWOkASr0AN1s7t1sH//wV0zNnXSl1an0NjSHB+/EvcId6pVdIrGmNnHJCrRsEWseMB8dbxKUTWcOOimkFqi1+LsqKoZQO/+jZ16w8djDwOB4I4NB9NNi1Bq1rUc78u30G9ffRQ6Ar3AmlKmIFI+YMApsvHZ5z/SH+kR3N2+1avVZlcqfK76jf7F0cX+4T2W8emTp9vD1vav/+pvDHu/+8/+2deff/3OrQdHz58y+2//0t+st0TTT6vV4bUOxrO1Yg9pQekF0OhU2o9u3W+dH/9E+gyCarYo2uvZ+EKWqf4z68o0KxmUjKVQyMvahqmGb5BsrzmAr8rYwyugMPYaIR0qS5iKTwx5hdmp/0G0QOyuvk+6hbnwyVNAtbbgn4+sibBxf1RGT93EZUk+CAjDiiy2CB6OOqPZuBSai6FiJvAZVy6D8aIc0D0CyAyn6BCRRmR4kmiIFz6xQJusjXiQJUVbEojCCOJfwHD8rhiIVoTyutVxs8j4tbZoV+vp6nb/dkUH9EX2k7ya1+SPXk+3uJrPzxaqjNaLbu1SGFirjbKUl1f8pGGYesNsNYYatgsNILZ+a/306/HI5lsvJqOTla3erhdXdlBmi1v+kHNMSKPeUFZGFfYF/ibgKEZkwIFHh1Q3l+WaDZg8NIpULv3mwBjdAQ06D2TF5xQB7zBz7ly8IP/6C0HQptFKabZK1WzbA6L+anz+7OLo7GpcmbVWW/0thbQk3/Vl3W42HQ1S30awR89+TF20UVgUrE39qh097c9lRGEc/i8SZ8NCjKc/G/cMIrZEouv2ll7V1sfVydUwcTW1AJbBkiSoSc1KVoFsTDupzXVy1xcbw48HVzsarjnXRlKDYaw8qJQcMiQlya5mn7RU7YuwUg0T0+7Utu/sPHr0oD7sz05Pv/rqC0S+bF5Hmt7u7r1/t++8Tpbqsy8vn754ut2o7+qLszvg2BrNpopGCWbpSetep9qUZNGlDGrwwirkxLfhMoMPYKs036aqwDTkitIGmaJHFJEUegGLOI8d8cE6ov/kCzgTjddaQF9HsCKOX+yLqe6HZEvJl086GKIw8+qgO5TyCszS13/pow93B1D87PilpGgqQGmDhYXL4lPoxm6rKM1sd5Qc8T/n6dGv4VUhlAhQstNwgFR4Fl5kFVLbq0gHj0IQRlq8NBmfgSf7KkLVT6xS2IXnUmGk9iMjd48244d61WMAs5mSGyc3wpKMkQXEQQDUECpVvuPR3v6Oh66mq8n5aLY37d661VXPuBBouKb2cHsISJNqfruK9yaFOsbkDlRdfzisj3HcF+YSiJJ1YAyXQkaRmBQF5Jb4mP+pBXMWIczwBBQr1U2GXPutd98i03hNR6/OuRzRD2gyKLlwLY6hmiwGaKkKg+I6cnkWyashebqJG0RivRAarADWBQXcCCz+c8QHcdGn7+synA01Kna4bRHAMTW4FvwUfELueHF+lQdR2IxZp278Krd1xr/M6xgkBuHrs7OzBKcbHF4qzRqxqLZq56MLOUeZ6zUFfGSPB2v36aef/lq9/vCHvygua6eNg/2ds9OjyauXLBw/Eqc6H19QgIb9Tn/Y+Gj37dPR+N79/X63d7DTevn1p1gZvwlG1h8MO53my+ORLXWL6pAkbQm4/V6dyTMewb7aejbi4r9z6xBOyphCbhcXl0xS3hPtQZoNiejN49MTelh9pL1JIvr4gftDvNn8sj3oyGGww3HzRClBo3ulLW2vtb2biG/4WOilgJukATm4FwywR+lgu39we1eYOPfkrLZgWSloLM0zfMPeinKgZ89mnAoQ5tF71a3uDrlvBHwTLaUOihjguNYDlo8a/uitX/t7f/+fH/2Xz4+Oh1u2YGyPvvzJ4Nbt9mC4kICLjelwE0N1S9sHlkhlb1DZbm/3pIt9LQQkcdOk5GOrKtMVzkJDGyuIxkkBg4u/Bi5i/Dh++H2oKRcUxhAMeC0vkADEwByUZ3JYqiFQZdGv+EsTDNf7WnsHgQqh7yn2COmCtuDj6ZFTbhm0osGVhwWjLZ+vCi6HnApRuVPM5Bs9jZE9fpqQMUuDnh7kNzyHcSUaCNELczIXv0eZOIy3IUE9CK5ngrHp0ecrOnAyXj06C9XrbnM5SZ/Hs9JkReE4T4U43rJbGV9JO11cXC7IzXljPSttRZZtLa54mytiDSuaASxIUlw8C+iesVTCpRUbA756tnq6mPB0jc5Oz17xRUhcSSpOfHKKhU0WbwgHA5rXotdksyqGlpyaAvK8mlqWqiyE6WF9DqsUIKadlN8H+TZvcj8/4WYM6/SSI+ZyubP2BWkllj+gdVW4v4HTZmwJtNVvc+I8efn0yfHTSeXCFsVXVQn84j+L1qyj7qyEmaRtyJa4PDvuCmXLyFOjLJWh2BdY/oZ1GINRZqBlsJW6TEfeV65dodmU5bSFIqvioeonTIPWZKGuadpqn3GRVKuoTyWOpLuFoYR4MFEExJsRx2yL8p9EdIqaeXKQ8DbbP47pxu1ZWY+u5jaxs6CDg73Og5369x5VHtztjkZv32rxAGR/u0FbVf/VdqOypyWtZeucH7/66icvq9Ojd5pvP9Jo3Rb0W3JhElNgmnOz2JIHKfJPyzHtdPtXl/1VfcG5pWsSZ6Q0T7LJGNtRVJlH7aipseizUpEAZQmSb2CupEV4gRS9wqOztkGD19Ii6LwhzQ31AWO5lcIB9CMJpS356KqhPL/fqbx9d3BnTz316kIOzSub0EpZxlg5N9hxwmU9YV1dYiSYIEv3ibDiqTGejUpQhiX7yaAwAlBNKlAlm2DIJKJBa8ArjI1ygq8QDcYaW/zmYB/ESnG20LIOw7EIqbsWNPWgaUCowjmtOeAZEaCkHZ07b1Rz0RuedTm7FyMGAayQaDPWx3h7mzysSdqZz9NDPrGB+BR4IeBxvNyv2VZAkseVQttC7BSXnHR9IYHiUQuFeZOvcKFAo9sRrRQMtWL6H43g3FrA4vqd9z+6f//+w4f3PYLfL0U6lXU/favbBDYVzavfknYYThazUNY3gzGvzaMz2sI3vdkcPsaPGB2nkGw5S8Bvltb4NodbAc0GPrkEyyyH2eSGsnTD29Ap4YqjJgHAE40HlesUIKXeyobJRvvgNqwd3tr/7nc/gu3oia5iXlqg3D447DRbXx8fPf7ZTx79/b935+7d0fnxsHv36MXzjy/Gnq+AozcYygq9ff/B7bfv2TajvbdXffI19yl7seAVzWLOxW2E4gjKuhIOwHFDh4ZQs6ulhpviCJQk51++shX0vt2Q6N7GrA6K0wsMKXnjiWYyVIX6sxfP7VYoOmGcXC78NWSweembLw5AsZEvOB3Pqi+PlC+pFUg8L97QSJNwCTp9Gp6lSjgKjsW2hK3G9t6ORAudz9JmGruJtxFq8IXI5GeNXEtl4Vo6Onl8/TNZWVf3H33Q7u+HqSzHvdat2cWJK4Y7A0VkF189HvI0fPS93/zHyx/9D787O724mE/PHn96cDV7+MEHohhSQnl8ozTbdEhU/Ppaq8D4UIcftcZb4yWXmSaYVx1WgQzN6Vmrb1HLSpEEG24ZThF0jbVVjo2opGmbYdgrpGBKRca7VExqRfnFJ/Q7qTR6lcYgjuKQADSQzYP32CMF4dKQU5YIj1J9mQLbooeUsp8iyPMw1nOM35Azmg5yk6JO5Dv0fqW1SHpmwTmjzQ1YSso8Cg0aM4qIIw/+FI1cahFXBaI3GBpSDHckE80/Y2egxG+RwSSmm47mNme/Yv4N1AkmYu0Xjyc3LMBxdR7Ps+bdNiLArzlc+oqL7I9A16KQ6pvEuYD5SIUJTKzCZFSxPe/Ri6+//tq+3UhPOguvV5QSJGUYilDsbJ1NtALU8F3TCFWD/188AjGCON4+FFzmGpo0CSTp0gAnNy3qfsD6+vDW1Ru6xuqdjbIcFcVgBKeCpFSw6OVQFr9koniK3sRiGZXrZ6cvTlbHUc21RcczVW3pIzaX6Etv0IPcrgjdu/c/aHcGL+qdyfHzyvw87dJqbcBYC49nQp6P238zIaZXr20rUeqTvrqi66Kj6cpxXdeAgs5ZBbjl5Y1tZxaCthGqTC+KASoNOsaap37FC60xh9W5VDqgSTlytW78O72m7Gz7hfZ3ujv3DoVHZufHutW0ht3+3YPLPn38rMULAiPvdA937+tFpUp7LH9up73upvtoxfZp+qvfPTw6OpKOrWduAp+Vno1rDENWFImikQj+B39kLdulrtOW2aOzxAvMWvm3/Ga2qrWSeschFxkcWilpGjFGHMGrYGHKhIhheGdecUlnFYjEkF+QO4vlxa/SbVPqeZY6epMsPkdyGK63+v3x5KK/uuL4ez45nlzNd3raEfV0SpTsRntoKTOOL3nTpU+lQpADTiSfh1AsYsnDLFE0VYaE1M88lEoZnwY2T7uT3qYBZnJa6LFI0tOxA8SGnlBljEs68XXFHuYiCGEkVp/3iQFMwyfUW/i+iCbjOKumREA1dWn3ncBkao4Xp8cn7f6qM+xLAzhnErVbu/fuwm/FeYKOEnv5m1XSMMic3OC0OUGLjbylXocSgDB8IQgXAvEv6vfIhJxckOmlWKZpP8GONtxuTlV0Xlft1dHRZH7ZktTb7z96+yF+qF3Xy5fcuWsVwGxvVSVmQdILisYdmkkncyom90aXykNDxZsREqIOi7h59RVlEwsMEoQPxawQFLD4IU66NsNQLHS1lGEVM9qd8GbsJRMIDW2oXaZGBL9k6k1IhYkdVrCpLr7U3tmDXIm20Y/3LrP/gZq1e/fvnHda8eHzKt/c2A3weDoZn58FZDv9xmIqPtzZ3tWu1y0gCtFmmjs77P+ty8b17n7zbOSbZI6o27BlNn1cX9ytDjfyjZAtbVhJkx5tfN1MI1OZy68XzHZ//GB0ChgENrSjAk5mY3jRZVm3uuPpK0GIeqNHTVtcnDJVJQBau9A+uGBX6+sXL2XaiIBedW2g2GopPpC8un75Yt09yDVqHGgeKXshABgSMbktXNbjaiV8j08GKXlrQA+onU82oT9GHbzm9avzrp6efC3epQPD229/t793m/Jof1+6F2gvF2MDoxvRLern094Pf/039u/8wX/3W1998dWd27fn8/GTL3724N3vas5mAx3LFXFVa3uC/Qm3pkSMDWUr2/WtyeXnkv5aA44iOeZz6m/QwV9BnOCwD26wMeILBudc/vIB0UUnN12rnBlCk7D8Yj4xfHlc2+neYwUjCYtXIJ2Gm0rY0B2lKY5OCSaRF+EBWEjsMJHTojZ4OD4XJTxIHcdXccXlUSEoow1mkddGkWG6Mn21Qm/Y1GaMZbQZJSUKBrDMeN1yn/APHvTMAze3JtiBDHZO+Zb2IY1ar1ljwQtqtivnN6uT0ezV7Obkujar2Tb6eslbwsBSspnUqmS/YzBczJLX9T1lMHgg9Ws2lThwdPQSV6T0U+yXq6ncFBXwyTWBG4mxMRYQbdodeg1iBJxgbGAFtHnjlE8++ifghoeZZZlilODydYAYYzYcNbfNvV+/z2dTzZ1pMpwtm7VE9Bmqx0bo5pFxOGwGUVimHH1xz+poPn519nKm5Kgqf0/JmeEleY0KrY/SSDH9Sr8dPqDDg7Y84Z1Xvd1XL76ojI60SEtAETkEZ2hLZVJZGweuG++jRGEYz8bSzkG0SshosRxPUjwgh5mxyVeEgg09WSapK6VVklXmoUIXy/Ud1Z67gjpLi7KRr/AwXZdjzU4njM723Z3hu3fYX+tJv7T1H/T3h3xfYy0GVudW/rJr68z6uslRYVdPW8QPiAGqOo9ardt564MPd24d7h7sS3dmdwO+EBvzK7ZJTLFkvoYSguRkE909Dsw0UtItstaUvYOqZF4qE8YXEur1k+iBxu0tsrZuUNkMi28UJSjxzpKX3HxLF77g/+I1ypoXVSsrGTZMYXKpJWWALC5H1xVJT9X19Pzl6Ox4dt5sqIACBLk5zWz3t0VfKBH6sgkBmoAnNBu80AMjnWKpBmmQpCXioszz3JFxlswXHn8p5AojWylxSOe4ohIn4pMDClFuUSpaxMr4oKNCGGPheS6QfKT3CKs355P+m7qgiFWpvRKbZGTBLH1ddO2xJYetHVSU3awnvT7Ukh4NY2iLtnfutHvT8xFGTxgSDPAvCFUkn6cEgx3lNfPZkEHYGlhGVpJyidJIRecChFSe6L7JpUo+F8+izSBmp0c//fjHd+89aDVbEomP2GVSwitt30oYXiwNWXTPkGPLCjiYPIVMsHQzDK+e+81hOB7tKCPL+8AkWZM54zL/ZPrZxPfapF3A4oCE7lOOjX6dEtcSH+DayxmHXzGkjc2fhxTCILORByWjQERSLKUVmd1cn5ycfPLJJ6bz9ttvn/YHNiF+OZ5DVeHggwNuHxuPz42G08hoxNu3Or3LuS1ratweUFw6yGh0plFOo31zcGuADXEzv3p6Kirv5sqLbeWLfInSW7fu/PSnHwurd9u7qrkINTk9IEIHt+qz+RiSK0SezEaMWhGl4e72g/v2SjqYza8//expU66FHZPGE+JAr0qNK2ymbGr8MFQBWcNj7v+rE9kFdx4d7O3s0pZlNjW7u2riJDwaDOVQjix/SsyqNjZHPthJZG7ZLLFbKHaPb7oYNGgSPosIEKkWZLK86MmUrKzOTp5J5ZY9+MH73+8ePkywHFyVY6k06vRpTFfLq9F4Oqx3K2+99av/+B/X/tX/+Oyzz+27fP/w4Lj6xcGjH8hxh3Qus0aVlq0Pt9bnJMZ19c6d2rChrdHFYszr0u4N2q390uE5nL0csNW/iJwwNH7CtaAK4ipWMpxx50JzwR8XhHObSY4NPw+zQtHEIj2KAhyOIjGF3sttFq0vT8jF0ChGXXBQ2QFEzhssKq1tOUwTqii+MRe5Js8CYWPb4C1Edo/ijLREAap70guwJ6tQUx2CJyi9lroTCxhLcYFnJxrGqCk2NNSlkbU4LW2zGNF7ZWeLZmWiHGK+GF0lTepk0b8cFJMXp0qoW9VvOCWlAY+RyUCJV4MdTWNxPblYzmQmnGSzbV1XZyOIJxHVpc1WX46WFUS0mFeYWNhbUoKgcSAfLA2+mKuZBcw5ESVu8w7l+T6fAvhy8Ru63nyMDh0A5VvEm9e8NVtSImqS//IIUFCoUlQer3QhAsHicGXRicAwDaharQmPEYtkfKJYCi/VoonTK7XjReiwJiQORFm3l2hDp+PhwZ1BpzdgJ7x83pqfPq0sz1IsU5CJJhdp4dnlgL+Igwea4r/FzBVfXc7HloqYEhKPk9xDeJuoYdJEQx/h75EYZDBTSraVuleKOJEnHXt3aAcZHURW2sXJgWowzJUIDXfu3q7v97e6jdv3+mv2qm5T2VyqG9dUMfi5MvgSpYIOGtutbk/qt8WiXtjVCwfb3j8Y7O5Fqcfok6AedCWGkIUBynTJsmyiGFtwiygV/VjYOmZ+PU4TafQS0aB2glmtwEaZb4jKvPG1rMJmISMQgtl5CWe2AlC0pC8lDkPiR2zDg2IVexNAusjjQwq4GoSsrJSZcHPx220f7rb7nclodn128sB+f9gpFxw1hrHFnjHiiBwVS0EIvBk1WBRYEkraYI3HxXKCPEqx/CxKsLEEWaNXp1Ff0BBMSrIuhDObWDSwdvMUraq1jSAdSv+ITf4RgZHc2uTvJsmWWSAqTNIQwARY0k7XN+pI3ZmJaRuQ5k1HweX0YqSzmOCxShJlo8aJFWKmOdhYhBK0gPfYc0CTA2HnfaGBzIhmmqSr4A4ImDcfJAoAQrfnfzYGLmgqipIaFv5kPLl89crk93b2jZx0scwe5imIw2P5ovlguJ+poni6tD53A9ZoKfxToT8LD54Jl8QPF5XVX/AncLZosYlfjzCqAEt6Kw0j9ZkIjhXoepwDTpDO3mTiJY0rKayuL0c4YGEBXoPUeFoQKzLesfkq0p31NZ0KaZuybZxs0DdiXNp4YTW3ELYzYjhffPU5RZP6qpX5zsHhcO/g/NmznSSw1IhnLGxxPu+3hzwi+7eGl2NKSevs+EjfcGqeVCsrqI/Bwa2777z94aeffI3R9Ad7R+NX7Xg/uG6yQT21/fT0mB7nuRcjAtiO5w037w36dx/cOz2fPXl+mkp1VRDXq3a9Q+H19GAXvDJ0wGfa1W4m48rXX5/t3356eHdPLliRjrh/8R/QYiGEKrK0WnR/PirwtpthtvbyOAkGMpDX9kHSEBfSRt8v7YYjhdI8xoZ0SV+wte/49PFnH+u1dWe52n37vekSv5H4sh8smwpT2G2lPxqN67N594P3fjjszf4f//n5l19d243x8kW/d7e9c6h/OwUYakUXn9sGe968khoJPXu11p1B7WyePSauqLXUlA3GxlIxTwRo0lG4MWRqMmSEYkHoYDiSpUrGuMzpaPAUoqAC5uDbOFb8KwpADzYtecSyf6o3fHkItCSm5BF8bVAtiXFi7FzlVVkz3MgYq3D31ZSjOLubsHokiIRNb55nBGnZw939RjCFF3h6Ib/CCQLvaIFEXKxtScnX2pTIYWWCmAjyJQSZPilcdmMf8XQh+GgI8Je8Pl2NjxbLkW3aJeNo/r1dX3UNvATNiCs7bphjCE1xDA5E+aysxpfcfdKbJylUnx5N9Nils8air2/ZUFWCznxEsMe3GMacJh5mxJxCT5kdAgNBr0BXYBtSAmCfULB/vStfAjLwv5ajuSbnXx8h/XKRV1PdHC4ADTwgPKdc7zVvsMqw9bDyLCMKjl1GZcYA6s12Z7oYPXnxbHR5AW62jABMeYVGC7pAHNdWGorGS3JZPZPHvbfT6w8PpTPb+vPZV43j566ZBJ8yYZhSpE8mUdVNSDM/UT7dRpZK9HlvgQSJUgCIBUIofjPak3FRxDycmUuLoFTyFnBCcJ7TrwQBBu3ewfDWe28fPrjLtnr84tmr42P8lJq6e/d2Z3eHp5ktyvDVLoN9LI06YUeSu8YmY3vZ5rBhC3JFJDX9lYW5q/pb9QgA1liz26KJ05dTEBw3eNHZot8xJeW/WBdGJSHISauFPUe1PcjPNBBdrM61yWV+SnbSwCebbnTQRHho6a+WtDAQiUVqWS1CikSd7tkAAQAASURBVA4LgeUjVC4Qg9QlY8Jwy6p5midabddnDJbAs12VODTBHdfbujkYPvzud3gD1sfnk6fP60vhlOi33MLkrsQtFBiDO1pabuaAyRlHLiKhklRlnE6+Rp/NP86XRIvkyWMWEjzt2eAOyuIp2gYVcggLdA5FSCFuUdZInze1s4QHw9FmD6uRlqwMwA7LDRuIgOSvb2LfU3qz6wmMmMLWrd2anF8cNxqWiVvDVr84J3NP15hFNniJpCmy1pMTVM0nowk+lylnBiV8Rf6Cq9O4ZjJE6CK0hRCIHxFOYh8iY4p49Fe8XGmjP2Ztjc8v0IFLZpMxCS+9mLARq8blOKKlz2OLxAzyNjx4kruh9kjWAkBflmqlLGORneaZZ5WPaM1luFR+gw2VWfu1LedcUBhAIJM5ZuSvb4iMc5NspB4nikmDcOjYasaGcxrxZgwsfHW6NBuRNc8EM7cV8376+GudMWK2B0gQdy1Dea+7PZ5Onjz++s69u/sP7nNUaM2hJnc8n+0dHuCjbS5rKbuLWSshumWlN8iGcluNM9sQJoNdyzetgup65qoHunW3w78lWUJbeTNIt5G6nMVh3DRX67PRhToig4cPFKP5ZF6pjfuDs/sPK7cO7+xu72RDYf5DVRoabCN+68VfycNFFa3WbUvT2Ammn88qj1+e7r84q/Jd1euzMRYvtb+NfEmQ5JKmO2EJhCMwjKiUxPAGVWszKnY4APhDeK607AoEaSPdbL8mz7NaWQzaA3zrYvzqyy+vVV7QoncO7za7e/o2LM9Haf3YG1Kc7ICkVOzm1Unv8Pbf/t/+b/7kv/gvH//sZ+896H32+U/vPVrt3n1k/vYM5SKl+zFg9OWeHZ3L1K/fu1Xfue7Ob2bSeK+nnWEvRb0Oa2kpkVEYc8I3xXTKIidoiTbhduxg/2fDo+BGGLfOoWwdf2gz2zcipdAChhNPGx0g+0WIbItlKBsLI/V9tGo6IxTppX9ITdpqA0dnOGO3sYATM8bhNxa253rrN8YYGezR34TG3CLncW6ykYgg/ZPCs6L2cdK5kvYDP6E6lNCngG7dULEY25f8E9Ctyy6SVPDyyXlTuHxWvxpxYRAIQilSCJoM1zg4SM1CDSGeyEs+tPjiPGs6OR2dH03L7rxx7Grql2QIZnf0Kbw6Hqs6dpisEQw24ThJarHUkzhSyCz2aRGMITvPyXLk84ZRet2cCVXmHj7nyEUhRvfNsdGwX3/Kdw6ir/jcvU3WbfELInajAGlU4NfElMHIY5LxhzGKSytzHs8mJxdHNrc0mrg7Y5BSKwPncIIQSPFqaMt8rvpEv83d5r5i/1vcWjiTvXIuXnxl9wE8TNfaG9tZwpuI7Ov6xfETlhC5y3qJfiaROLEem5mE74SJR39PGMJPYE+tb/9sT1vP1THK5+ttdSUn73Q+f/n1ofzGg63Lw4a05N29++3FPjD0U9iQW2PitmDFtSNGybYpIypGBh4KTZPWGLMmApYpXGwqCZcKP1KUQHnrqVLmeaIDEIPFkvAGIjcqi+wnmW4YKu3kxx9dnH0+mnx1cv7Z+mZa/Ahb7Bn4JYO4Wx/wtDOQQ0BWLCOzfmjBqfCWrGKRrUH6omtZD5SRL5kJKXjNh5hGWc785Re4Rv6NS2hgQ2ARNuZttf7Wr/36W7/wy4vHT44++/yf/z//84f6/VpcnYEX14KYqxupgNrEx1QzkPBuKEwkB+zZwRfwDY78A3nxorSOtu1c6uzDCMlwxd8r3f6V+iaVi2ijMkfFQ5ruEsjNol4ibhW16QVsjCFXz1orf2HhzNTAzib0bpQBEsmO7mDy9rtOSMNHrVdk6aiq1Ffu7Opag6TuvTs6csQnh8t2e2OoHMLCRymGoGYN42CMIgGamJLZhEQLwLQqxBAAV7zEiqjpFey94k+WGcfiCguezLU2GtvyNk1A11vji1PW8KkaJFvZa3dabIuBbQbaLe2OTTOFLuUZnhYDOulCVo5F0SQ0lEyjkcjDEqBlRrDuoTOhaBXDBAAt/FB8yo8ETBCYHLM4xKLPURQC/fxhlVaZsihVhGsfNzAWUj5LV8xCAC7doDbin79E7mga2pF749kUnrfbg+rquqtZarNNLGuwqdGtQEljUN9/uH//wweyCH/7d/7g+uWL4xfP//6jh4iOVtHrtzWN6t7e0Y1u/86eKMHv/f7/eDo7+94Pvlc73IYC55PLvXsPnn757FoUVh8E4nW+lHr99gcf9vb6P/vZp8Nhj89PPzuRRc6Fvr02+0PrKzKHD1qY6SLaxmo53tu5mpyzpKv9Tvfls1cSFtQBr67iRpK4QISJWGMP8H+43T6fLvpuW7n67Pnk+qfPT9bdR+/377y1P5qrVZ9p+0rjxF0RlD7SdS3ZbZYO0SrSCFbMCsZoIAghgdG7dNqKwosk/FJJm2TLKJMqGixJz24QHAE/VjP6/R/82q33foGQi0KbrMYtPI+iKlZj1SGQOvRf+nf/7e0/uv1b/8V//avvfOfxF3+uFuPgne/q0KxtQWurt7VuLV5OG3st/SJXr5bdW3tbw18eVPfPTq9rfRiEgui0IQLdm+JEhpfJpeBMgz94Nd7TIkjoD9mTGwPAQMwMziTiO6zcDLTEyubEprRe6ItB74rCzavf6BGdo8npLZsWs4PlBstIUhxVFQkCDMpk6YiHw8RpU6kv7cyhWV6SG1LqGe8bLOe8WtRq3WiP8W3yQ+ISPjWw92Ri0BiuOO6notrcx0rggJdOJlivDTiCWKp10ZGKd/VSyG8Qt/Ps+nK8rkxqyor0xm2MdlRlmx24X2v4HT2Ar3TLtprZmt6KaP9WAi7xXKyvT188tz0vuatskdSlrIAO2YSDRwmXRlEs6hDLVmvG/y9iEfbJ6xdBFpOB0pvOvaG4MC/f4fCYKpmZn4WssMow2vLLsBeHxchLDm/KvzFaYiDGviHEIlhzcVh8VHWfUG2YU3EdROpiHTmJws1MFFxhnYIWflr5R0Le09PV6RcXX84qy05/73jyyrdGVdAX+c/TicGONoqYr5gkfVGT0+Wgsrpbubzf7g5u3f/h9sGHf/KH/3r88svK7IXIIaq4nh/L5JIRW7cHbJk/WodwMaigADHQbJfu/hkdnlpsLuNrVqbNhb0aPJz4lcG2c2f73nffu/XW7cHx/Xq31djpzTSMlwY3aO7dGnY10yG3ZtnSQgsOSQ7aPzICsHsQ4Y7FGZv4VNTgdFSAc5qCIiOgj5wAubTdMYastxiJj0VOOx+nop+HNnBDyKtQcjY6OXl8dPSz8/GXgx23VFfqApogtd0zS+ZTwFxAnqlBjWjfFMrynphwcqNoQeKse0Szk4biCxdGC80lr5c/J3PkXJHdehCkrEZ1kXy0OCObrdu3dyvVH/zdv3Px+AuNHvc1Xuk0rkenpJUOhef4RbAGmkClMuV0Yg1+OHimg0ZFEnumiySe4AysTFhP+/XKJFqiWqkHEeQheeqOvwyfZXMZ15u/2F8ZpvOpQybVOI3CfpFjRGaTISQpMeqVR5Y6uZCp7cKWKzvq7Q4b6oNFdDRpIljUBF7bOKzsbM+ezlChdvRuB/0oXcmcLH5wkysh6ficnDSj6J6wDmpAFdRMxynaeYZHB7HV4fpqJB6iMXXCHbopyQ2UU9TReIwDCB5kP8EZf6utFDR5CnoGXnCjPF7QIRYqsDqzWc/AGDwMqRwugzxQDqnSCXiX6R0+4q+yWHmDXbm53pvN4SOQMtp8dGVuF50fOcW1EeKGKBlF/veVV9wEk0qOK0DQarUTwdHttzfVtaNWG5aqKqvKAt6SqpFd4qnCxycXT7788tbe/nd/8zcrd+9+/zsf/ehP/5AI/86H79063B6dHiMQO7MdP391660PGr3BTr95cPd0PFuq6dnfPzg5PptlA2k1RaKcd8SbXx29wN2tFbOcUd1RsG47DSaR/bDVpBqQdajZl2L+4sXx4cFRr7fdbfcUbp1fhHFTrGCM3q/h/+qAsvGtQokrDVh5qONA2Wo+P192L1bf2b6Nv6NeMgsEMmUFORA3ymSwygvpbvkLLIOxQBRHE85I+obKwnfdH2tiU4iCLOPpk6LjAsZXYzVtfvnxn6/mq3tvf6fev4XDz+cXNwbRHpBCXCPsvKbkukH77V/6hf9Zu/nH//S31Hg0jwdic63mXpvpLJo7WWTnK6kOplFvr84wNpj11u726uLsvDFctroUaJrHuYIb2pUyAnhY2ARsLcEfnfVST0kq9y+VG5GLFj42XscerZWtnUpdyKBvO774x2jRwmTmFWqOj9pMx+en/U4zbaSz1y83huzfACAChZURfwMbKElhEUxJk5DmF6ak5K3YDaR+sDfoDJrsXZBMCxTmOrnNo0MtkVk7ulyO+S+UgIf+Ga+ef9WhgmiB2Kra+m23Mqtdny0mp8s1Q2bRrM3V9dqMrCdYmVZ+cQNGFQpd6zRtYWhLEWDMXV0CZstpdihazvSPkQM4kX0YqnegK0DbyLosfw5QKFzOrI26YMU3b3LCGb8lffPVRnKiukCt8ODw7DeHK8r15VSuz0+8Gm/+AcjcLYpdwbtc7EY++jYLkIvKS0wfZg7pQxNElvHGx+5KVpa8hKuL1fzF2Qtb6VGQNH+1GkQkcrdWWQ+/9YfbhPnwELAq2wToSLx/3ejvVCXPdPq33/vo1570hsdfMsCeWNhqp8fJebmairNwx2mVThjWRWDSypB1V78+mo2kgXDIGRahk9Fq0VC7PHj79vD2Dl2qfnF0Ph/37uxpSdO8ffBopycYT8Zx5mcFwlWySRLDLYptuGxccrCYARFY0O5gHKLWzyi2bPAySBZlK1tauwPUwukc2BZQCLdhw7TJwLIslc2zJuev7DcyHc1tmjGbaiYwWq8nDB5aWLIDjMHTwxlhDDTN8zHL3MGkQvD+gjChCu83q5l1cUl+t1mwcuL1S3m478ql3/6ivHd747XEkXS5rbba20j/h+3mF7/XeGbjaC2T0lyyiCMlPZopFh8g8eUG0bsjmTKQsKcyVfgYrp8RQ/uYsB6S8r0UmcQ5FUQXQwmOuiI/LfgXx9eEcSN12m5/dBs0m6gVGYPr3WT/Q1UsJTOnodKc/1Kby6QlG7lbU5gW0p0E1dfLm8OdPUa22iQt+1u39lrrK0m31mUg5HbaoTfG8pbFHY4QRpLVzZCtLEXErAK28LByBKoAUDY53uRwWQoP5bL1FeGhVhnfj1ma4lsKBJrgAO8B7PVUJDjwd31AJw4tuol48F9x98HQt74imwm8QCKpp6mDg0YIT9Gae1pyAygoiJEEupwuphMbOi56VkUg+c3hns4ELEXGu8B7APfqVw5Xv35j1m+Gt/nWk7FRFqebuyH1SuaTRSGqhEAGw727b73z3gfv9od7Xz1+Mjq/0BVHPs1nP/vZrf39/Y8+IlMPdvbOXh3f+Vt/c+feHfKw2+g8f/XVx3/+ycG9t2vf/6Fu5HS6lBiL4+5svzw5lyc9m6rVX9lb0DaHFyfnQ1VediPAf2lbzbYkR5Bk5bOkgJeboN3mFVwd62f+/OWD+52dHYXWezZ/I05TdASS9N2oyvxJAXsIpyEVNJmrIrovTk5br44bvZ5R2OtJbJETAexBADKga0AB8A1UN5D0lTF4xQoKF4t4CoYHixEOTOIYiwGFEmCVq4V4JAPpGn2ui/VV5dG7W83BoQ0olnZp0y2V3o/Cs1PldU9Yc/fw3q90l9PRl3/+8VePPyHmHtz7gA5iw53LzlWKZmH+LHuI+10o3wbhg/dry8f2UbCB8KVsSs4yijTqurwU/QcB08WdG9ed66t+ijeudSrcl9iFfLBlTkncU1RV6x2p7jao0QXMxLFj37JJIcaWfZ8a7jY/On7RPNzjE48ZFqMAG+SM8aF1fdOVK4kLi9dmK4MI9iReRkiEeaX9FjIBV+oOsNo6PhIxLk3ttbvYAWBxQlEXBX9hvB1EOlV2Pzdyz3YIlWXb7rxCEzo5q6NZ2JpKgpUuDJfs8S2M9oZnSHYPKznKbugG9WVx7LlBIqSNvWx6bbi5zzRlULYSRSGpxXJmC92bKesTnkTd+esOFzg233z7TX79rWPzldfY/g7IUQ5vN19Zkp9fzlzJPXNbyFWU4vJluWTDYYg5p+Dk5ufee8MhSvLA2VSYEGLWsCjU+AKKOZtcfP7sKz5aFwtq49520cHowtbK47BubMFsPVPMTxiTu/6qcoZo5qlyqQ63tm/f3q/XH1XW58dfjmRZ3qRdK2upWifYY5onBY/j1H6MPA0p6b7Z78nW0utDPxqypKFXm2wRuT8P97ffuqu/bXd+Z7wYyeut73btLSDLkWAwD5dRETEbBpraYUICI49rMKYD/pPpm/VGOBF+qBSm+JbDhZSWY8JPiFUhTEgGJhSYJAikNRoyjGNB0JpJ7dBH7fTkS5ugyQrSYl6qgno0Pf/0xbWtQJG1AW40iIBIq7lp4mFZIEtFTIZDRHolSBzkdj4YXo6yahlxEShGGabv1Zeb181l3371k7gvw0lyQEGKqTlUOu3KvXvv/Nqv9+tbL3/6Y8h7eOs2M+vV1491JnE1NpgHINSYAnGhlIegxgitstJhQIEUB6BRiNe4fwYfDsYU9tAyNJf5CfFS7ofJCJpeXjaXZXucTbJd6Qm8nEqL5WTRTArqWHy5bNGzE/Jj0UEyCGW/pmX8WcAiOYjn2UqSwa2DXR7d9UwBzEr1qnBycpcwGj7GsNSk0MRZUY6gt6Ejieg9ZbWLauWMR3d0Ae52mNQQJfYS6VSmZZ5Wv9vpp/WHaUIgHTc44eFz0lEqBwc9nVusmuHFOtCWvEg4F4Oh3zo8wnvhxk2qNq2H9Mxj3Ov1gbDL2IrFzysAeSJNy+G37usASnf7BicLZIsGFrrJkSfiUuVw/80FeZPgnxGEvwbOZWyQXYYXlUTSUF1jit7O+x989PAH3+OorT8/shAahZKgsFzJpGcbPG3pPLv4Hu3c2t/Z3d/pb08vLj/9+LPO8ODX7n1Y6arq06B1++L47KBR7+7tyVYj709fHumyttMfjE/OgW3CWLlc9MYp4bJMXPoEMIlgxfFJiSkYlzznV6+O7UIourCzfXAxWlzo6Wg3LDVtejDEOwZXcXqEy9jrFBt4renpyXReffnq5Hy8u3+nvky5EGAEYyEmAiKAfYjEDer62+AtuBVidDI2j7v6LyYgzYYKDmwxRaLO4ndEGwbHfU2XOhud/vTjPxPbfP+jX+7s3mGzXM5PpV7Hl5Pn2n5nqYACTbzzP/+fDnYGP/qXf/z468/4Gd+DbDsdPFihor07tAjgOEaiSQ8XoK9VBjt/c7369GLyRN55faB1l+gb33nZhIdCj33bcKnKkTu0z8S6Orhp3G1095RvRjsQqpGzohl8pCnwKk8Ug88NSq4OChXgTbc2Rsfo4tVeH+fmhJaQZJNNGC93RK4S05ziTzKjpRsbT9G0Q9qJ1KPE7AHrveh+0SHJHNPFohGOTlL0QtHYOCmQpIQAVf2Nm96wtSfQa+vfynWqdSvT+nJyvZgqCV6RvrJo1rYVXfa2bnQb1YmpISTDksZqNNOwUxfNN3nsud3ycj5azlTLqcAfQTMxHvRI1OIPXGtFwY5MtLJhpcqawsr+mgONOLt5/atvnM8dvnX4+M3Fm+tffyz3CYt6c2x+iPhywgdHsShc70CD0STKl+WC+AVgMyMl7ko2IEXaD3JE91ttrY/no6ejlxKbIOGcC7g+oJG5QXly+K33yVEXlXJ/KYH0H1Vk8lTWOtgkZDCeDRUZSgR5+933G/Xr51/ZdvRlHktxu+6VLPSUcsm8alS1HutLv6zt3bs9vZofj89mOuXqi3NwuHew31CBp/PAQftqt92z4Ur1sN7T+4kzVOQhXkF8Eqfhg8XFxHzlxtkTYTEdnZ7IpxlTtzXJw1VlQLDpYo55YlS1iA8kDcHMHhp6UyiXZozzxi0IHqu0Uk7PBwavYzIZi1rVqhfKMLQr7veG1GWUqFSVN6DbGlBAWODatwC09D+e1MbsvN7puYh4KJIuTrWStACGFuY156VfWlDiIiuYVdpg1M8X2WDK6b/ykvhLbpIlcQMLH76iQk7K1U3l/t3Dyi/xmS2fv0grLJym0693eYFnNraTFuxXbhwOhdvRmZNVl2XHG3znVtAyLrsItMSugagACq7BNz8xqsi78lw/dL1h2CnxerYkKbUOo+HRmmlYanjtwNrUjQtIgzsiTzok9HsaT9oEkMJLZtCfKbngwGY6Ozre3duTJ3qmO9LedvfOgUowa6q3uvmSUhhuRGcQP8BJwCpWS+SbYWTIxmrJI4myYVtY/toydRX6gpkfku4U9o2gYo5vqapraeOuyYput+vs3Gf3OAmISx247B4rRQLjDajjHIsVRRxU1IsXoZk+piS1B7oJfyNyQoi+Kg7zorEaKvhcXWFVxBWLmQA+PDw0EXv2GbY30Jixu1nu3FDyMwLa2PpwKHk1cd44z6DN5N+M388zo02yNMGvFETGWbkhcIRogYtbqNmyW8LpZHpnMtc8+tHb7718/OL5109wN8PmgZiMzufT8aunzymlP/mzH+92OvuPHu72dpPCdDH/5M8/e/jdz2//3X+w/fZ7jz786Le/+mf10+P3H71z9Mnxh50ew9fuzkoyu+LNorLkzVb1fDThXqZZQQTVemmskSTZyuxGR09tsG6Oj87qted7e3swCame6swd4179jBa02HuwUa2F2DbNiZMBXUHb5VbjeDZ//Or4/Q8fAggQRREDE2qH96VOIUiwYc0FOOCDVOKdgLdYXvCZnpCIF57oORx83qMc/oAsbXhytLjS31Pp0dGXX4RZvvXWordzi69Qw0D6Im+PfG5IOdXyR2XzZfPgV3/5N7a6v/Pf/csvnn/W2uu/vd+t93UtKKWraoR1VeTx0/LAlvTLav/+d2o3cg32WcTrq8dXc6mO7ArjlM5tDSUrtKq13Wr1sNLY79X3bzoPq52dNOFJGHF0U7OdFQs0u8EYKdUDOZQ/dGFyPrpS9wIe4BteO+4+6GnqUhUIVWI7ylouIwVSz5NGFkWj4xRMmmWoKaumWpdJjUMwoyKco6iDNUfUlT0FGGNN6c1WZk4j0QxovyYsLUYxE+utXY7WV5Oav0ScNBG28ZSILVBEhtbtOGmHRvjMz5pgEp3D4lCOF/YIHI+OXyWCImuAAAc1vDsCLujsjRki7LhHQu4kWpF1APzzw3ff/OWqN9+8eRMw5X1eAe6N3P3mpDebY3NNeI2LA5i8df3mDZCWL775mE8OJBlEM7aiN+fr4JsVNtjkoGPgmAnV2nkYuKhcvmTlVYTHi0jANBB6caCZcbhu1tezeRaBmxOZgFSwMRc+lmIa3SS9fc8XS/uvbO/u7Hzw4Q8Iwa8+qVdGZ25ZXwj/SyPl2e82tZdraxux26v3G73DnX5t3V/fE19t9brD3R15PLKX+STtA0NaG0dmy6mLPpjvICXrUexNtxW6kpLg2oxlsn1rV4QtLaqsDq2/aQdlO+zWvacckLbJzY+AQWhCDOUoRomFNDuaReF34aRMEI2YsEpAxFu5EpGCVIBWU2VoytndM1o2E155WqLFNMdEvKM2SuC6nCyWHTSKgWDZ2abPsI0i+8ptwAiwoPp61S2S2/nKzLJ0Zak2r5n4X3e8PmsYecTrtU8WFZqT4MeFdbh3+/u/OOv05i9fzWvT1v5+dlTlJEo9EsPVfT3RCCKtYHAcSxlCjAn/Fs8pPu52wXrjil2ATaHe8rPcIMiT+eUOfpJ+LZXpciV7qKtLevajBXFKuXY82V2go3Gvnn88z/JWu+3VZEbncRdARNYibNyMFEMt5IgEkVJZW5IMuvt7rDRtH5YXsiuT3EtWeVzwMxwhnS+DEoFhDlD2Fzhba1hUlEzgpxWqlvM9uTjHbmynbnfqZl3DK7Mzj43841VbzpQwpNFYEYq6/eHQYfTpEMS3V1QmT/STPO7NYUjl+dy6PChJPvC6OekVqNzEI7x3gSvN0R2+uaDANXdwxuFjsLgI4Jx8I0585cHw0wptDOXMvVxvtDLJ/SQZWvbGoEwYrdatRPXFlLex2e9W621ig7NKkPujjz4a6z/58pg2QCfg5nFD6qYculevXj179kL75m37pgz3yL6Z6NtoIaJpk6l/8I//J09evPyzP/pDtuqeXXx39xQYHT9/KSeLIjMdX5gets2frPLXH8OabQ5RQh2Cp6uK8J5ZMIKPjk6w1qJUbWXPzI1NmugUJCQP+Zz5QijabZiO295om7c9yFZp8ujiLNFkYiXLDD+AmTCBuo3aABENZYGKFuaHAXURvr6gaMISy0gII+WiJyWRF2sl97lO8DIUwmczWSbHu9fuLxfnP/vJH0plfPf9H+zcuq8aibscwuYZ9iEofsTzxy927h52fvD9v91o/8nv/smffPwnZ4vxr/6tv81tIKKFxQtwX86gAaxQUtyQNNTYfdga7rfqe9NVc8ZM5A7uttL7IM5URVPcxruV+q1Kmwy+Xe28o5YJeum3kKBJ2s6npjWMOWnQ5mfYkbo08mRCp3CWitvb3z6cTV4qwpJICkQbTZXeiubSti6R5ivmN2ADXvIqk42UCmlaoWiw2QE8fhu6px4k1zKfEq8VApRIB7vlel+zmXZq0135QxKslmc3i5HiTuYsv0aNOaVzpOq2KPtzXR9ieMuxIos0N5YawPZfA4FNMeej09PT6eicqU4njoUQ/cmKWpUQG2wJdhQXV7CkcMuiVpn9a77o5DfHhkC8OhPyefPmmws2b3y1uaAgj+XKx2+O3KQ869uP2NzKq8PV7rC5SfnMp4Avvb6BM1TiGH3xbEQGM8iwQcwJr0pyerXK/H1y9nIhtFsRkIibiEUL4MZVlhi38eetoQVZGdSCjMkW3lrUGvPECRN45zVwgVZ0FS1+3373B7Va7/FXn12PRvUTG/rx8zbVDevO3uvc2m7vDbZsbi+twUaIHYpUo93XD6Wue+fo1ajf7bNYI/WzgUfql4AeMWcBohnrNZMdVxQT20g5dTaQiF3NQNXSKqJeuh66yqsSwwjxZEXh3dGiouxBBUhl1QrkA78U+DrL7hdM0barYTcYnNQ3wr1JnS79JsKIsRh2hbwu7ZCuJrKLLL0wCTENysADubVP0kSWIcQ7QgzkMVkQOcdRbVB+nvuasUZgZPGyjt9esyhHb078lX/LyLOCIbnNIpljo6oTJipVj/HwQZcHqlKb6EmEifVGmpFgHDFDoHUYUu6ZR4S0vMGwNk8pg2NZqHtAXCRkxmd0FlCgvnhTvSnyQ1YTqEBOwjDhgKVk+sVAih5fMQSLAyAtAqntmDJ/BYSJrdnUo6wno0IqkKAab4g06exbvBLzXc8uxqrbtDM5Pz3VOHh7cH8wGMrjVbfd7nWZlm6HlQCqcW98u1nOACEaQo4Nt/AsmB9VwVZLmFv8ZixY64k5E4HtNoZjDASdwESyCuCEg/cDF453dLiNSUSIMzcKi/fQZGCvb/SohClZNXZNlgE46HHBDU5TNmgAi5NhLsmuNLqoCMQkBCbq2L5GRew56YmmY9TOEM+b92biDq533gUuK3I9prDzhRflvI+bQ9iESYCc9TAhVEpSBHgkkno+O2suF/tK/fW4bndPz8/Y2od3bms4I0PKrkGwORtB3qyFVIRYhju7CQSua7v7t0WOFRPXOioA5ldfPqnvDBrvvf/v/Hv/HpP0x7/3R7/5i78sRXVnsK1nKReTPeuXp/ogQ4Yqg1Vq1dHx2WBAMYh5WXhOgtPRRNmC65omJOPJFI0kYxyDYfuGCgCKLLCy3sjg6c6hFq4le6DRxghxgaSOwsyyIoyS5JNapRAeZ6YVsSQAlsO3ZYmKFwfnCjUH9cPUoC6nAkM75cAwPMzOHz6HQhNGEHK5ioQmFBfTsyePZXotPrhebd97Z6verULlqwVGiLasaqPSujqb1WXV/+L3fmV7MP8nv6UZ4ouf/vjOg7cr7b6UUIiaZ900O4uWLYyBpHbdIYAqvRv3ClO96t8sTy8rC7NJbZViyupgq7pbrW0n4fnaHyNmadxJR0710YrvSEVo08mSV1PyUKBaFMZiQ1Oct+8cPvr46LlE27SDZIrYZA4UwKkq6IzkObHTmjqeLmiFrUNj9blWgIC3AkUd4SoukJFkQIuCXDX+/9Z8S51L5ZpYxauHlVVvtmqsp/Ys3lqNbGFshx/GS7Y/9TjbJCSuJ/CI+uM9SC4/esliqdgdR/TOF5PZjMdRPslEOgttxQoG3SPhMq9IY9eXhTWy6M6OLCR0Kaf/yovf/pVzObHhGEGGv3hB8MPx5kfffFtuH527HN+8gYaxBxzOv/4Z+EbfC0J6a3iFCfmIeW5kMBsiGU95CKolPKvrJxdHn588k4sgBswC4DeyE4JfUAiz1q4LnkaQcEtEHcBYAhtLSpezA1Sy4bZuujPxdLpLdKQKS/jRO9+tt7afPXtW37qnS/FWsjJu7+3fvbV7e58RjKhml3MJA5dSIz2Xw+pma7I4G89PH3/9idrEu3fvsonNkT2Kbu0Cbg3ikVPQIqPQ9mJlMxwUkH0kyoythHXj/2ScYVdAkyC0gEZqr3FSaEc7EZglLGM8hXaD8aQsjuYMktOBiMCIVeTSBIEX17iay5QYxW8jY96OYjYQEDihWbL+AMYTCf5iFtITJeXLjBAf0jnEwoSFB4oxpYEvsA/4IVPW0jA8CN07Nh83r64vmOfTXzkwdRfjF1JI2JnWsVgZwu2029yUsnXndk91u/RNuxIRc0oytmboDdsR7Q7HsapZRY/Jc18fbowODQloivjCpwAzEhmOFTkB78wqsgR4/cxPiuJCkin3wXAvuxDMneN7lzJKvIiauSqyJAVBzWF/+3J5QvjFYlM3raVzDGHNTutKbAd2Seq0x6cXrVcng4Nb3Psan+IP+pfVtYyhtLNWYrdI1nidfGE8OCjJ5BGWkhHqNRoVZUxFFQ+XyJsYBgeXkJKuJHx3wKSv50rLiVWvbTMsuO1MqNzQ2/ro9PvktKWPOVv0cUFKD3BBkZ3hBd57JXJhi2sJV7gEpSyHJBLfEqjhvOJ/8UpFBpssD7a1plm6zID9yrF576O5lHsmwuJXzHFnfOv+vvAGRjnvjQscRkHJc8Y9WfZAWrzsbggLdDvGTKk4yxevjlqDgc0tHty/S9sQ8YEH9iLUa+zt9987eXWkAOt8fH773t3h3p6B9vcOtlqdk4tpvTN4/OXXD995365ljaOjvR98/9//3/+H/+cff3x8eiZSc6e/w+0BXIRwBk4PF4xQg766OW2fG7kSsJBAGSymGm9CbCzuI3RtRZI0Ifec57kwWUIzSysWb3H03JhIc0O3nYZU4vGceqccjgTnoKEGZ/r4gs0/ua8FAyNpC00R5JCappnl2XDrInGNB9rnoiiU1GnMCyG4GBjxgbBLg5XolDas66vz0YlutFS/68vRl5//uUTcXxTBvf1W9pWMOVgSIexHWNOHrjE627TpeP/v7ex+/k9/+09+/1/rDcGXUDm8DQEIJDq7jrVcr9yAi/NZbXHd3GtV9x/2xbnXdoV5lm374leSlFW/ZATf8A+qMrKXgeQJWxWI9rLxxSc0uT5X8aWCOVKZEzOIm+JJkw2HwTQRcKOztXOg1ok8jeIorgRuiaiAcMiTCc/7yx+qK1J+ioai/GSXEo5hA4modjBhRbGlUzkTN3r98kw+RbMmoX7FodFXELWaNpajG12dq2XjXpvhZPMXy23ZluqtpYnhgomPKDQId1IlPVO4Ml7MRpppkL4Kxwn+bvum1+LWXoKtA1HDc4e+g6hnvlyEt5dpZmCFUkIt/+Zjw1Rd+ZfebH6R84UXb261Ieew1m8dpMS/iQ9v7ln4IkQD0nLI+6fNg2Z4ayRxvgqnTCIeVTT2W+ww0AZuMLt+ab/Ny5PkOsAO7gJViAsB85ZRJVYS5AQJ94mkckOJT0EFMjWaldLmsSKD6PurRbu/61fSAbgcbt+69/DhoFYf1nc/uKvqX3iP4Ev0ZIdWaKPvBQyBA4qqdTzo99O4hlV8f3+7qzEsTnI5t58HVaitYwPFWXM+eNGCNyEarSdbvQHogJoxmdG3gPb6LXYpj0B55uYzDAgYeMyX6os67QhG5OznUU4aTeySZ4lbJiw7AljHZoXsNvDR9VIzAk0rrX+o3cF3wwGP2eDzamyyQQr+TBq5SR3o0rOZghJpIWtNgJtfNvDEIJBKhIOxKPGLCNmg1OtBGo1fF86e+EY5NhiwwRLfYWeb817jK479QAynv05pokmvtD1UU7ODraOj2ZOng+HARl+1HnJBukuVNboU0T0YXCZkWdMF33/WOpYrE8iuOhsUCVGz8TEchcFI1xisb0iC6CTneAIRyjINBTFDV5yO7Jd9Vd3bub2zPT45MbDkrUvqhEGlWn5VmU/tKSWlWSfehR9WMDjz5czgI6HYXlxc2LyyujPAIFJ6WKt2hzuYdbNjm9qdau3F5Zoj2YV2PI7cikc/QAuiA4D3RpSxEa8LO/A0bPyOm6N27g0muD2W1IHbnnM0mmoKUroqapwU8SnnUgEXErGI5Dm5Ephf29MbOsAPdsQNXyshgiQ8hTRVVeVZGQAulTaQ8dlYQHfbYCbm67cWyUdCVHaSV6LLlV6NP5IVZrwRvd6zA9wzSMbiKRpAmQ7VLazBeTfxlTcuNlQ9EXfUf7e6lB+1Q/wK9FcToCW9ODl6dPv2wd17hNzJ2fnO3t7R8bHkWu6897/3PQ2R33v7UWV70LVxp9rhXq/J4XBwoJUjLZOLGb5ioUcvX/7ev/qdX/7N32iLdtfW+9//7q/8yq/8wT//l8ffOR09Oyb7d/f3aqznYfdcwitXIXC1OvisZCHSwIs32Lc5mmWkEd+kTgHpxJk5JkYuGi90yGHFW8tWTCZB/fjsdLCzDQAKX1ARGFvRbP8wmw12d7sy6pHbkt5mP0AxDEBOUxhiGEjZ96Ly7bYc2oU8ZmgBp8p/4QAxr6hnJVHrkokOcQyr4BHFNmwhu6TiYuIHSvPhiRzg5snJ0z/8g//hrfdHj94niFcjNeI6/gya08mqlaTUlpKZ7slZpT9494e/orPXk+dPtwZdzaG2b90TI59yP7R29L/QPY9JLBaOITV1qNi5JQVpu3Wv0p7rAMwckXJmmx8Ego6lfcqiEU/T0k+mGgYWts6IUmJALLFDw7dRP3towyfwMsCnXZeENx6dy4byDR4x9SakiXR+gUj8x+XR4uPR7lzqpIHGdDpkzNqVh6RPBLllBBxnNxzV9aGERTUq2kbevFo2bwak7HKuK0R7fd2rrTq1qaywHptNCVM0IXZZbHGySLXAqiUTwVtZKXIwsknRqW3Vp5Pz2Ed6RyS5e14agpiIABJJgxMZXARZsq/iZeMHCv+OEWWJ8q1/80+hvp/z/wi+EEoiNfmnfMidyvGX3iDjzTUu8z4XlxvmnuWIPenexVpx5eY2fuNLDDyXhO3kd2gZPwd4Fiq09tmdImaDXnGy+WAvavmIklLs4b64Wvb39s7mLz99/pU9aU5np4qA0TeWQoenNFnliJjQvYnTaHjxc3C3xl2ndDvOSMDWNQzOXlbagxUr2Hiva7ajGI8Wewd33nr7w/r7v/59HIEecHx2LFDUuZroYYkx9yVuMABWs/HZydXsfNcG6e32sF3r6OFMTEzPj7OtArErF2sQnlu22dkICQOMchBtDgA2a2WeRvjz143uEL0hHBM48hrlG+4RAcUFT70pzmoQVvKgFIUzPfof1YCfqOBMj/UF3/0GXcYAAQMTj7xjjfNGwuxu2kMnvkhqcBa5EnrnFwaIHXsf0zuW6kajMUh/oMqiy/3+Tce3la8NTmSpLf23frSZ8OYOQQoECjBEe7/XuX1n993J2Z/8SbXVrfZRWvN6NiPco8pkDRe0D1Ci+5gMU0uyof4AeGjivm4CoMGzMH2JFskQzrONHQYE6YJylABEEejirlG4JY6OF5e4ESJKTQVTg4mR/eOzpTwO2kqvbPOPuApgrAQuALZSLa+vp6MJK7lrW57p6vh01Oq1BrbSaunnpwhNVgi+GBe3waDvmDn+N1TBhaxRBJ2Z2UMTwPNkI8urPwnseAm2BoLkWygD+fPaBIsNxeTMnCnJX5KiGNbyhIwFBVdmSf1lTWGHwUb4ZTnLgdD866OTKJasdZT3+WTHTOvlW3KUQAVMIAQtPwkwy+HbzRsnLY1rXt/8zXnfume5rDyy/DaPoUVBOIMr1jnAxpW9Ku3emk2JTj/84Q/tNtG2uZMmI8Ntbsg/+p1/9dUXXxYP03A6n/X1MOj22FHyMDS51xULHLBH2OJ6Sgupd3Fy/FT3yvP+zuxwbzYXROBz/qM/+qM7g12ksd0dEBG2S7teTLS7ysTp5Ch7laJnaFGAIz0heAtOIbbEaF0WNYUIt1qOMjGAjasBJQsqWWIJknyflH1mHfVxoXfoxciWYAKdtGY/CWQgYLhtxAnIMohFcC8mtlJQwcUD/JovbCAM0SBcWLdheFf4qKdClwzLfLLi4OET5keiF0qocN+M1+e12uNPsbw7jz7Y7u9LqlpMzhvtXU9pwYv1zfxiLm2ssr939x/9w/6PfvRbv/Vbv/DLP6QK6HQ+2L9TOT7FHFXx0v7tyrMaVcbmsaj3DvYru7cqi/Ot9irNETiqxNk6tXaLwUzhG9R0vE9WISDpfa0FWLchk2l5LC21fsM1be5wjCxInFHFddh9dHKyS4OqVZfhIiTBgsS5tqh91wlFAl2Q2/Y0BzTUpV3KOKhyVLRzBs9aayAHpk23WHe3Fjs0gbVtaMc37WmlcdniB9XJja1UuelULjXcwAG3eTKkPWPq4Mlqp6RzN9uTWre+G9uf6UigaQfDN5uzzwJ4CoZK/YTgqa2Ii0fWmHiuwq+TL5NgonULGzL5slavRWM5m3XMxP8NxwY9ypVBMMfm/V+6fHN+Q5hhJuVZTkLOYERxkGyekVs4Xg+mcIbQ/Ws+4BtUSKb4Ikhe3DkRC+5AjKVRIFdyWEcw3G56lcsj6Rjz8fh6imIyKnNxkSnGj+2z6QuEewcb04ihCI5QfFk7sPEVjueRKF81B4AKKvhkTOn/ZTnrw/sHzU47MnW0Pk9tyuVWd0sfOHcUCGxF1q3PXz5XAbq3K/BWlTeb5I/T0xN7dAtoDIcYOzO029/2eJNJ9muUuM3Dg0eRwmWU336Fa+HO5XyBfLkGr6/n0dj0JjKBuWZC5oYpx0MrU5urm7v4Zi12mLreSbYmlMOX9C+B0eK7JeGARrv6+GO7BAzkNzb2Y1Gps0aBJx9gOCdW4KcehZEE90s2l/Hkmn/TsYlVbL7dYJLXQPw1wmWBXrPtsgLlbsZW1Eej6dcHjx6yBpQBjR9/ffb4K9mwuKrug3EzLUSLhV1xwayVoQO1unp/pG6QrojECB2MLvAIJbjaLMrEggEwBWGzfiFP0rS4xK+uJqt1YzYDz11d/oUKmC+QIFkj0qHZEy6JWAerIrRxzA1pkYp1sS/p2nbH2m31GEUi8i3ba1MQ+tXOcBUZnPBYUtvcSdAgcnuDk2HCbg/CMU4tLi7rMPisdWweo0h1YzJOGOAUjPRFitzizfGvwfixNY8ey+t+c3NxfsZf4IcJv0SQRhukLkRpkWGQwHgi5T7AH3+8Y9A9zoTMUJZZVIHws+wqK7oRb7bVMKSN1PHxm8XdrK+P7lceUcia9IUq5cAdjLWsv5ec9BqdpxyCIpksL3RLakuftsNaP9Zfs9nY29lmaSIrIjh1+eubMy0/T88fvf3u/QeP5OoIUmpI5DymAE35P2DFdD5vd3p37z94/viJSUzOT5998WWr25F1NR0eHWq+3umcnBzpXbXdHYIRORvfFnyQQT/JYDYyuHgCMq0gK3IohzeFtTEViGfKKt+RdYteUmRMuFhEMibGGyqRKf7QlX1z1El89O7b3JYKzGV00grphn7lKczsWB+JbcrcailinE2pAlkkFxgANMk/YQgeg26ylE4YKSxx1hON3RJnuctISjcAWB0tj3CzUTlP+eLZF1O+i6urt979bre/gwVwAvKdabjbbvbWttOanHeHvertw8Ev/+Du11/8+Ed/tji/2BvsdnR7eviodnJRuTw3xhSjxHhPj8GpeNp4Xe/v0ZJZGhXd7LAN02lXec7GTET9PgM019qVPPE1HI9fKfNKLDUe5niPwZ3gDQZDBtp/upovZ+eXHejXvpmCtBlyjs1rKpGyayrHJwLew+yFbVfSuqcTKun6ygaszZo9uuba6fVs5bxe9K+nLfvd1ca1xsUN/3eClZXsneyeSpt0VZxfJNxgPSA6mmLyUgTiq3p1pHjKjtR2p+YfuuLalG6WRmA4ZdT+NLPMa6ItIB8VtKxIWRer47/wOSuFwzjpc5akvCmfLVg+/qUjQMqvNl8FJf2/eUJ58+ZT+RnE3Px8c8U3t8r5EFb4bvlqc8/ciuAKGWY1NsMrN4Q/Li+CJ08vEzF+KMyvSg6getiexmrd1ni9+vLo6cnsdFJNe4TofG5fZmmOuQu+AjOTauewdoWOwlmiefietUWeu0DinnoyIEQCglCbKkj+sdn0oj5P20T4tLZ1YP+6L7VBUIcTM4+7uRwO+q3a1TEfr42Az08vzk52hkP0qTaJouSJzWtywg4WSFGUkDszick4cJlaBhaiDnjwQOvyrdeNMhMh5WKX5Nt8D0lMLaI1NFi0CDDhzQVBmVdQEgzhZDYX1DLvZj0gJXRrE8PEiAmOrGycvzy4eu26I3yXlOi0kSTetZFbnlvWJqtQlj9aaunU4f5ORvLkJtGVXi9/4PzmKOudD5lfOTLNzQ9NIecszOsZAHyuzH9mxoL3dU3hP75/b7h9/eCLyu7O088/H798tbq5yCgUJHBPyZ7lyYujo6h7hituKBHDHczCdeW5ETOaqHAgbYQbRQtu5VEuTOvEpGHhmf636aQQhCYD1ObtAbOAbF6omFT9p543e0FjZAVUsJay48GBQ8QJLipXSNRQe3hmeX+wo6dSuzvkpe1wJOtx1t+p1Dv6B8mVJSoMwBr4YcjRa6QohIU7iVgHAJw+sWeK9RmJi127rOQupNwiV1t/9j8xgO1iHCZaihJttFc7OTnj84wiEhrcrGCQhXTIQ8sRmFu/cjjpcKv4vqIXZGxendkAkobhiMh5c3xzzebN5no389Griy2vw1qg28Da4hae4gabh25eTZmk3wh4lmhvuG0dL8Y/evHq1eMvv5ANc+fBg8bWLXjPHsElz8/G3OeH+7fEs6/Ox0fHLx8/fszPyHPrzp7iDcx5cP+hJGfArOhvf3F2PZtoHzquvzwQTxoOn1+MpEL2xShTUiuAJ31XLhUD0HiDMp62uZvz3hikO5fJ5SMpa8wZdlxtOeMo/1BukrSxtCOdBM1mJ0ESfUVq1w/3d3/w/rujZ09jT9DCwEeswcXAAi60JvD2rtu9detWq9MWaC33RDMILcfmTXkOPp91zcmwMMjOExq7I7KPxhRMCfZIG9vQL26YVujrxenJCyUFkqHfff+jwd13SJVkoV/aKKaHrKDu0XS29eypVtt/49/5d//H/8v/9dOf/uTt+3dtlHrrsFNRzjthY0paxEUFtJoaVizPL8dni7a63/5WS3XJQskspVamUrtiM6WYT/ZgoY1o8ag34NLsk06q76XZ0vpCQQZJyzJ0X3EMhCQrLfvL3B4910OhfT1TQsXz1JQRBTEZ0bpxQlAx5gphbP+Y9VVzLRjUk8eR/KkKh3OLQ/omoretvvJSWZGKKSVGU55/sFEGSFOQV4h34H9oUr8aObLdVDZDYTr+dMrldvTllyktSMYjamfhXNKBtbwlhgP4vI88Cyf0Jg7JrNNrz6WF8FXIKIPdHLDIW+f9W36YyW+O8vHNh/Lv5hrnvznA6C9cUT741iDyNozj9REcjoqTLzfny5O9DYVKpcjHQpWbH/hl8g1NKDwdtuSywmVwFyxZQ2+dR2M6xsHS2jqX/3zyYirOw3DNrahSLAEgQvJCllB5w+QAAc8JGDAiQ3G2jMT0/c6SMyGEeSguDGlCyi8THh7bCnwhwSU+SpNY6467tTVkSzl4IzUE1i8C7tAhhsO+xEK2+Gh6Qf9G/PRxyXqZg/o7NxBijA+Ug/SS/5Gi+loQZnUyT5AqWPit1838C8L5tmi6mUTAbEb5F2TBzD/hkJTcslperBAXSLE/WLfNbeIexYdxUtbwQBSdzr3SKtJwwNjsB1w0aoqhNIkCHFTjrmHxcdUbooCmhQkMIyydCDN3mspYxv+XX8rgAuifvyljdxd3yBJkedzIiHwKEmNlNAmpJW5ankljbsv63drf+eidd+58/umXf/6TZ599Onn+cjWdEJcx7KRjuF1Jx/UbQXOA2jDQjL90ljNq8+CYLyORxeTfwCvQ1PIsyWWKMlObQE+nfk2o2tZ3MtsRhtMo8VpzY/SGSC11S4ubLFnm9frPc8xgOpl3+ruMHYVgk+kCyOtN0axWtY1B1HqXFRv3iEasrs/ghrAXMk/EL9AogAhk44i2NCW2GEWxQDkBJViX0nADJ/Rp4Yofin+SkNPlgxxCTaz5nCx18sQoH7RXKwcO0N7tMA7wZeV6qCsLNEIAQehiqmYwUCBVBDHoI4hSBBv9wjsC0itQ+9YPNgJps+oZVzmcBFSvuaOjyCv/upvnOQ+87uA+aMRvfeWNdTcKJxGq3tvuxI5jzdjmdnxx3hx0RxfnL188v333Tu/2XZmoy9nl5HzWsBFDvSER9ejFS4WE+3duSaphznnMxckpZ/LuQHurzmo5bUoJUcyWvtfR1+4Pdw6Gw49HY4ZnZzBsDnqqm5InoB58IpEiUd4UtxhVgYtRErRcFE6ZRdGNgspBIkhr2Jbstee+MBSn0hJHaJe3gmG+sMcgPLqZjS9ePoOvgTed0ANKbpO1JVPi5wgycKY3pHnfv3//4uUXBZIh5jcHjHCgjzw6xm/IKcuJVvGELHCACr/C+l3nL1dR1JSMGy1kvVldnD7/zLbWlcUHza3B7lvZDnw+Xk/myuy6g207JbInpucTzuff/F/+O//qP/1PbAyl8vflP/9vvv+9X6oN7kAp0h+rEnWFkzx61Upnfa7N1A0/4tXZunJ6Xd9ptvbWlZ3r6l0NEehO9dZVxGTa0ddn+Ldyyey7zWEfINubNfl4pTIYvvIqTiv1vVv7H06f6wdS09Kmz5BIIlWvVunaQHLrhhGsRUO/Muml9snMF+3apFubX2fLBJtHnK23Vu31Qtyqfj2z2SK/GTXZSBoqN+Epmg54kBf6kqt958CEnL2R/SlSMDq7OD9dTSaNdG6wDsFf+Il0EQh2kPBV+GFZnBBxDktKmFsD6xLumdfXi4eNWpF8E8aRZSofw/j80Pu/9BqglvObr755/+ZRTrw+NlciY5/NZ3N4//qe0Da38pLDeUvnK+TsFbmaWk6GWOlAUSeMbPNzPyUNLE+4SH4brVrQ3pID5NH43GaygCi+EvMwHg6eM0Gl6KRckq5n+2VYQRMvHhTJnueVG8agjBJTRE3iZHQadydUxdK8n0cv1IdOE6K4iZIdkxFw9uiHMFtWheMn5ye1m5UUP2xcbZ9Yig1J5n5p3xLOEFWDOiXZ1UUJYdRi2osnGOzG42iQZpeJZUivVZuNgrORFoFXBh2I5BpvilJV1jAf8nV5tdLemaqLN2ZxoUSmRGMbiOM3DQhl69mLNwPQuBAP8NDweSy/rFBQJ0ui+SrBnGUAunLf2KXlvd9uvNQmAoAxtd+MPIP55tgs9jcfyxTyqQjgCJ4MlmZkYQKQ8hD/mKW0W0M05PC/NPyWCFU5PNje2/ml9z54+Od//gf/4l/89A//gBsiqd9JkkrgW2kZwUZJbZXMLg/C42THuLehAxzFCf8JveRpEZme4dF2faQN0coRDhs0YthYb26enh4vrwe3hB71tHBBkfFGZwGgqBsGMwtCZ7CvkdbOGR37sI/VHo0m2/cEo0UBGNL1Zv9msHPY6g2jwgvcRvYW+k0uYVm7Apmynsgo65vbhuyJK342E2kZeyLBZSS+og7ARgcHBwWNjmr+CQavkwDvrsXNEZwx3dBQcK14N14/5ueLYwykhld5n86W58ZIdVIOiVd1dOpuCUsxWhd4qGd5k5uX6zPWzYEEU8kbyQRMzrkgpmIRvTbk8UPE7+abn/tIGUchDiflcF2MeIF1wW+99ejBvTu3lG0dPX/y2Wcff/SD7314eXVxNlJ0cH58VmGsbu80J1uXs9Ww379zeEueW5bJRgsnpyrBHt19qPhPrW+9aksEHFNtdJyZlxcj8YVOpzmeTk8n53FNSfvDLNyQnIVOZV5Gvpnd5qPXHEHb8i9glcOYg8DlGyfQUuEtVRlr09nF9TyJPTpHyzNrV6+ffvHZ+2+/47LNLVBX3Bra7kt9nM8sJ9AWZaTxzvvv/fSPv8oFRhHy+PlhVEAPC4PHZUDOlIjj68tK2DpfhkFkSEaYGns5AbCtF3FIGTh+/OmPpdF/9IN1d+duZag5PsxfJXNzy4bq9bk2ZMub1vbwN/7B3/2d3/onP/rpn9pC7nN1RdtH2/t3Wod3aoNtAd8bJXhpg5UqiWu6k9bjyhtH19cnl1evKuv+zdbRdKtH1d+uqjy2R2Rf9wybQgz120tTyWj/Cu0XRXYgishnAtj+cVvVQafPZvnaYLj7xytQalIR1pcy2DTnUEZgXwdmrr1QUo5yycyd1jWHri6b9cv26kLfjGZlQZuoV1bZVTb9CyMSkuufYPt1HKrqBfSBswoVHZPGy+Xo1enRkcZIGuygNan82bNEwKj0IZCZ6teWj5obHLE6+T8IszkKY7AWORXBUi7ylevyiibzo6z+N6j1zXtnNie//erbb47ND63o5oyPhW/mts7IqA+tEYPFR/UGezEBSGV5XjMrF5ch+LcMI0rc68OZjTQyv9enNnYwYzEyKGo0jpBm++0t6XZfvXo2Am6KRSgASjKr4r1z36QiQNsAhkD3gtrxJW+CuoZTlnsDt1RTBCj5xv9YqQHzstDUbZ1i+zBbvBVmGH6ULU/cnN5QnUm2vFmPlPPOJgNbx2hUKR9960Z7LJ0fqUkSspvNrngPLYJgFtDKBPJ7Q8XwxfmiO+OLr2f7F//5BtCvyWwDqEhrQ3xNfJulKqBEYm8IFRhei2BQp8F0CTMBTHlIbuWJVogNjnaKgAM88I2L0wFcvnUAoyvANlScGdvTMJaiZCDgjec1YKQlUFv/egv49QIXBr0Zp5+EpRUMNtjXi1ymkrGXeJgrLJv/RUgjmqLXq36Zya7rMyX3tvd+/Ve+u5jSiq6n4/MXLydnJ/gkUWSHFDnbhsdOhXyO4Es4GjU7GR1F2TCZuL2Ngoz31cY0RB6RFRyPQW73yySV/UAI96p2balL64hvidcAjvotSMBSWm3uTymWZN7qsn3rnCHDrjrRo+OTw8urYae3Sla9pNpWb/ug3RsaFvHU4RGhA1gCQzH3YvNuGCooOcObw3TinjORLMLGZExqcHQXJwv8A6C0iIhdZzx4d/qWR/BZcbFRjIlC4edy43My84ODIZEs3walNo/T6CNlZ/q38N9SN1+jQ6FSfIoxJOPa4+Sh+IpwDYQ3NPxmif3cwVowbt+6Bkm6lcN5z4NX3x75ZqabibjYAT3kmVKkTO/e3dvyqvjSi0xbj20CcHL8h2f/+vOffYK+Xj179fSTL+5/8LbiQxawXasSiZNIPZ8FATTPenm00xrI4pGQIY11PJrZxFHdaP/g9ujVMS3NdI4mx0pJalO7zV/2q63lZMH9soiyDabRgFOYyOsYPTNwg/ChkW8dgT84AH5YnO8saAGrslGBXtC2dXutsj/sv/3g9p1DhVOJLToCOssA1zj8uUSD7hLRZWaQFrr3T+7ff4inQeBwtr/uKLiDXHCDMEfa8IYzbxY3yA3gRhRehZfUMDREnx64OvTE1WivsZMnny31gXv4/vf33/3I1o3cB6acyOZlVXtkyQSTV0f9D9793vRX/sV/9Vhew/TVK6mow/Hs3vXVACT622RNi86jCuCmH+7C6hfuWdXU1F6JF2vx+mJc66cpZ71tQ1YFAr3KkMEc13GtuaoqKEvDAbnLcJ2TQWRYQ99uVVdIkmOreTk7uOm0FIumN5mtiip9+zGtrnpXcy2xaJjU55rmvEvunrnWWU1FG/ZLuNTTbKnMWGGZ9K30Hk6tp0YDXPCqo/p9yGam5Emzz1Fhe5OL2cti+J6fzUYX9BKsWXwOpK7dWqfNJBgAdURUVi9u6ALpjTqeBSqIEfWe8RX6LNAIH813YdmF3F4TS9yRjrKwfwGjvnW+fPnmJYtYjqhUplKOzanNV4jLueBDuW2QsbBorh3/brDCmc23Xl1eLsjdgjzlMMpwm82HzZl8HwRyYzE4vEzXA1h6MZs8fv5U5jN5LI3AjN2iRBbhv1EE7fIsn9w0SByPQeRu+T+maBmMX+XZAVhMB/ACUfzOVHAPiE1kRmpaMPkdJV4omzECCj32t/uL5c5siyZV0W7mYnLBBaq2Q/2fe27b0mxnrzOgl2/zdIUHpuDvdfNPg7NCZYie+nr+fvXXHAUer4GSZQg4/L+x7kE20C0QNvAIsHzKVbmUMwexNXo4cpLQCtyBgw6MTSbUioHk51kuCOpHsVQ4EnjEsMuUtAOD6JJLhcj9F8kVoRlfHEK2SwoBHHoPyr1+zWDdyjU0/E2UNK+OjfzNwMra+F3G6p/yhPyblcuAXOJdfq4oSVlEU0BASiV95cGv/sqDd9/+s9/9/a8+/njCWSzgRxKiRumLZqUFGEJOAWAEcx5gKkZMHvh10CFV4toHQFjrGG7v2zys0BajOXwK72xYx8TG4Y9CWffMNrycWsyzYEpUg6iKxabMDW8uRmc7kn+2dWhajY5OJIXY3o5lw5+gZtFvbQlAO6YN6AYUWysqsiFlXQIcD6VyxgpH38SVRQvYrItoKhxUfgMPjdNMIk6BKgl+kmpsHZ19er0qmCitjKF06VBv1Yu0xjX8xhEBnLlGNHp1JliYXLzEHtS1epKP5Vd5Y2i+EiIlg6FEeX58ORsCzx1fH5Y789j81itE8sSS8ZMy3zY7w9NJC0WeabmVcuect8tBwggl/7m0Xc5cbBvWH8K8p18/2d7fefjw4cHtw/uPHv3kzz6Ot3l7R5b/9OKclgR5yCuegeNXL5N3dnCA4hhxNjWfjEbaykAhALgYT6VzNyt1tdnno1Fnd3d8Ye+qyta9Wq/JqLLHdvaXBBNyIK3ZU5ZmBjHlMYuoZurwI9QcwVgTTQiGs5eWVHAV3GxV6nzoR2hEh0tt1FQ+CRYmsNu/pU757n0OG44Z84YUFBVN+W6uTzE3uJFoDi1qPlldjA56PSUuPCdLdjk5/ubwcFD1SpeJ2N+gU8SCUSRxP2QHjyIYCjWGUSSex+Xb0WhsrSBVu0Upyi2S7+py+vWnPzHHy/ZW/+6jdXPHAon61G06kFyYZWd7ILtx5+/+xt+uLv77/+w/fe/eQ027zo8ej89f3j599vDdDysHd7Rn7uAYqwmBhXWq40o1sIXmUFjoYmVHI30PVoubi2lj3rQLhMLa5kLS8U37utZr1XtS0niRhXU5Eu1pX6MKVKTkmM1q92byVr26DWIU0paW8dXtLd0uV/UrrazQErl91SIil2NS0l6Ana3rjs5Z66XCJW/iduZ5TJUhxSi67CVk6w46Glm1ZhPa2w15e3L8/NkT6MQ5ClfoTCoI09Uo+ldqmDCAWCgF9YG9dMCxh7T43esDFcEJi4KS4UIQJDix4Yqkbz4Cci4rx5t/80UQ6ZvP0Kccrtq8yQLneP2EfNwINt9DxnLV5hpNtl0X3MiBhjaKHg4YyeeGxhYdwp2ZUXloDBxIS6ssj8DxywWUhTy9sCaEHvPWdTGCq03ajrbaV4B4tpo+vzyeCwBzD1jpaPdwB7vJzVE/unMLIIASVJxMNOwut82U8NANoMzI0+KR3PD88DUTi1ZZfk5JSujLQOEx5RxZaoIzHp31rLUWbXzc1WyyjJBs8mU3TZm425qPJ7WHawsRqnHUU6fW72tQrqgAg8u2HjFAy6K6YeZfhF8G9xri3masPz/KQpQxo7XwfV9Z45A8yPki8/JjeFmWLxKsiFSsEser98K5tQ6RSWzDy7q4VLI9TQjaveYaOH+WBJyZ7aSzW8XbXC37/WDoKN5PCiIVgZCNH8TPnVBi73psIhRYViDPLsw4o9osuAuNmagEWKgDEdgPhGARMvCB1SUvFMakl21Z+Awh7oiNhiIF140sqTe7B79w71HrX/zLp+eT5fVLNQ24ervbvzh6oW2r/efiJ0+ao+oLJqj4ASrEg1JMmwAdWgIUy5NGN0SCvlKMSPnSpt+ciznYnkFRxWqGS2Oo10N5s3aMZ6Kty0WRysCOUwOXKHKv1cTihRKn56f6tR/0d0++ePxV//fv/MIvcN7yHGtZrGHGPftidVvLueJBwiU9q/OXlQ7+6b4LbaNfqo0IN7dfb+VmIT6/1e+2p9NxWc6iC8ogoQ8RL/xkUqs72SpRpT8HOM8t2Hodj87DcRwSYsiVdMhqciPTNyRzchvCFXjtFVlubFPC8vzMlqW6BNlVTSNGRa6LYU/VThe2nBy/ckbfjE67KVIrJs2RwtWDM8Igty0m1rqTzi+pnHbAQXXSsRvJAy6E2PFWH6XShoozh+MyfWKywnFR68/KIavp0VaTk9nGjhcX57fu3D7YOdi7vd956+3pxfzjnY+//OzL/lsPmCY8IpXxqQgrOUe4yCW+uNAtC2Fe2a/35Ejkchux4B3aPV/M5ryWL6bpObuakDjtO8NLaSQHzd17d+6evTpRScoVjjVIx1va2s8oqctR36hCOtjoBWGL51SQAycHCQ8FXCp6ekwPOhO9XUKj58FvqbNAAcsXS77l3ke/8Ovf+c6HL59+fv/WTmyklX48XaoND0b9ekLzUa7f6O0VWbs1vFmfX7x88Gjvy6fHvONYz+YoNG0RCqMLVxVzA1tUxH8Ml8Li7VYvXhK/IIyC5Wk8wNphJNAgQEJ3gWhuyzSuld5fH27VXz3/krH63lbt4S/crVxcjp9/PXjr/cp0Vu/WT+cj9+Cx3/8//R8GH//pV0+fPOj2Gqt5/aYzenzxk6ef7u7dObz7aKu/S1PUYX+t2wzBDS+sOB1r62awdZBcmbjTVC+l/v3qTKnAwg44ybilBOgMoqVcAVwmpPOzBKutbZk9lVn1lx/8B3LE4Gh8eSz6sSiFcLGUjWTSFAHc3Jpd6ZDZoGWQnrBMfLPeWkxXHfelecxUaFfgMKqvXM2HDw4r84urJy/5VGSkL7Du7Nu+6heSTE/LpD2EPKm5aAd7iDQIo8Wz4DQOHt9mPkJbYC+cN0xsw6aKvpuvIo5d9Hrxcnl+ETYdiveVe+LXBH65c/5hEOTwWq6MrN1IhNBpGGn5gkD01ihRrsP4nBdbyW9clEtdX54DEV1RFP3cKkQX2otMhja5pJDu5of5VW6G6wQC4iIoNr0J46MAyewCJA9ru3d8M/3J05+N7Vkhi167xjIuXTg2c42oLrPMODas37/oyFOiGMYKzTtXxzqwigXEPInRFCOPY06LdAG9QWDeuBtGAVDchgiFzotnSHddpZ2kXqjJrr3UfFuHyXjbmF5uI+0zipeeIBL48si0mSQ7TTxWXZ6flbU8RdAyTzcgy9A3onezcpv3BUJvwBSouQESy5qgZq/+95q7FcBvzsTZWm7rJAhyO5GUytX9baqyhDTIt+BWZF1+GW3nm4dZBvloxDbM5mh1WbTOsngBVkXGitRE0ERuZqeVQzb/MtMNKgYJAlBzN16vWYLyEI/I+zw43KP8GwgYTr4ytTAJvzYhv6VAvR7VRp8rV/UG7/3wV1RAPP3Zpz/7sz99/uWX7bJD6q39oR4peldoTSHbQr6n9KFOu05UBd5JsIk9Q42LFobhKb+hY8T/6YlEYdITgIFnxaSATcXgmf6/11fD9Vo3Ut2Fij0C61ghV0JYRZ4U9YY5boMLfIdIn2mK9mTyB3/Y/+4H1a6N25ODLXkKM0RlxhU2zYEfXA9QMEwf4SLTQVpghJrsNMkH9G0dhCOjQCb4U2BIUOWHOUVLcg6sA6xEW3mJ4SEp6xbOmBr7NralJFRtnmIRblArP3KBG20OH11crjemfEvYkOVkjgvkl8L/chDesfyK2C40lC5RIRNPUaFXsDo/j/jHdqO+GmlJvNykohhr4Whu60000cwticFetcTKBk2t9nwy1VLj7OT8dDRuP+7/3f7gO7/5m/aztvHRxdnp108ef/fkqPLgzkcfffDJpz8p+jaVLr1EDBuLTAuzzKOmZZXBbrpDjMFyfqlpR73aYmwmo8a2CfYKtlFKECCtFzZAiU4PPj7FDkBXpGUm4ghmOh3viV3r2LI5kxCP01YxCmaFY6bT3m6z2Wr2a+qVUludDNp0sRK1UXKqlYmEeRv0kJ5XknLZ3rCkIoA1Hs3PT1N5uhSchmHRWhxRacpR1q9YDWFjfApBP8wLUm3iaWGkrJHwhIwHj3BhKK1EPuKlz+TcljufoqeI9+irT3/atm/R8N4AYEbnVgs2WRBJNVRixQC/8m/9g3/xn/0nk/HJvjaWC7FW+Q2NUVzWmvp2dNpsb+/19m+1+wcUQbJFST2Wff7qLE+KV0PCHY4IurqHVqunRSTDeuEl/qhUuwV+K12ltuSQ4VcQFYjbN+te0G+lDM8lnEcxZ0O8WDejaGWbBM13/DSThpbwPBshSpRlbdjHGkbyTLbtdCQtaz397BO58FBrKYV+Lm+HKrIidwv+kFcOBFpk8IZLh+1H9S24lKeUhQhilaNwTe/K174zsvJpcx7f21wWKvPjzYe8L/QVUYvS8+XmyPNzj/JteVZEaTkJOvlVbMDy8801hVpf39YZANn8PG9fY0uwt+B0zrl0MyI3wcG+OZ/vgibWLcIFyvkWOpU9IIPyHpQ0k2Zltl69GOmRdDatzCUql4v9yv38ytMjhQ05+OhEFtX8yu29YG456wwVBDIGQX3r3wKvLGiuev2D3KGuYBNzaaiYk1GPd8dNYkfBgf7E4mWgY3SWC/1q/Gj19WrJ3DAvXCRFSzh+Yg2575sFeDOc/7/8a8Ths/9fjjw3/xOBeCGe788grUu4PmmGSDP/rCNOuYFWcNFfhCOyJYjLqpeZuReJYU3k0YGV1k9uRlwRkyYemULel9uV57rE6UC1IIoHFrlaYO1Z5el5Zh5YxvAXplJ+FAaYs3lwWVOD0c37zuH3tv/O+x+9390Wwmtczux8UJ9alOtlT0cUZRW6e19Ws0mcCjYGpzuEyxMCRTZRS7gv4jqBEfHQo3uzh5SunHFpKWeQ0kFuzcUHKxgKIbYrF6mIbXoJN3H6UxePtIXmKmYpagiw1dIbdX1xevKjP/7jX31wmxvEIconU55HsnjDYjW6OxQEDhOH9UgjcQ+Dia4D4lL9RKNy2MCyaIcusg7QbTOVwKTANkAKcIUUl6v5zcRbw3CroJ/iN9+5t0mnc1JknguyioW23c0ZMju6QTnv4zfvnfF+I9QpYX5ihi6WSZphIMqg0euk6M0NPTcCwc2NyLMKsLW4Ilc5yp0Jk3zDR7xBQZ6yOdzT4aShnpyccIGawtlk/Pzola1Sfunv/H3uaLtTIMPBoHf6/OleZcWstzcvfylAGSqLs2P5G/V+p0+sbi0aSy7leD6Sb2h36flSsmQ7AXwNm1aL0XTaOB+FeIt3AzFlAJmaxbE0AbU3Bun49hvnjZPs83UkMBhnDmE3fqxuTUM2u+1akObWvaE0EVKgxTc+rQtl8ggYMI24rVcoTiLD1+JLjlepIK+JgJsTBOXe30jfrPJmPB6zIRUfDSmPff1VGUboCI4TGxl9ViGMW4DGVblHMC6ET8GLnLWkS76Fx5/jtz/8pUF//91sX1Ftl4q7ytVsVbmwocV0/1f/Ruu/+a+mL57eO9ibjqYxKKo3kwsbNSjeWu8c3qOsc2b3rheN7b2afbAHPfnlpd9stlorLVbopznC9yn8YacwhfJJXSieEE7EGxUE/DVQEWWYezErzDNWfU4VWilKULH5k6xZxEgUDgisGb6Ei0Ztt7cDzYKHbX0oZXyfz8bj5WJ++uo5a0mQn9NLILeAKYCFOYCCggKcgDU4+Q1s/9L7zOFbh8s3nzZv/tLHby7cnN/c85trQh/5tad+c2GeuznKeuW867OgRen3VX5QcLK8CYYYsdfNEVb/rbu9Of0X/t0M4C9dBXMpcIAM+CVZZqN6WqH8NgvVrE2X46+ef3k6OsX2uKCTmBb8+vnTXenmGWMxCcov87JRCAzbLKIoFhws08xckIyHbh5kRcuvsgR15C2nFO0xBVpastSudYyG4Nr8ci2FU9D9L9eWFJG7BV0bTNG/4JYGe9hHufu3YZr35QGuf01d+fj/0/GGVRViDI5m/l7LiPPmm8OZwMC3sZWL6GWXYzKl/rZOS2SGZCSvVR5oEMWE3uMWGXNhQSDJ36M8Z4MfzjuKt7k81NcBNdWJZgPRPS2qxpuBEMOWDnfavJZZe5vxlFWKpNmgjuXIIpfXvHlt/Ba03MytnC3rXPi7SFv6MGh+9NajX/37f+f2w/tigeMXz/7kt//panLez+70eo+lCb4Gh1xgMCp2HJo3ATZ1EvqyzYKKetqC9SWnDVbYFbUmKllstSJkYtpoM3hVkfyFpGWn6OWj4NYsuMXcIE6rDLnADT6w27RfIENePfv6VP1MN/0rFhej5XiMyfoqXfkKqXtirN9gJBEZ3TvaEayKCHgtFzfiKqMKnweRHN5kGoE77hVOmzHEcBZzjMM59oHEOfMgg/lJGNvJGk8bfD83HlMuTKfcrrSWDCYX0fj/4e3PmmRbsjuxL3KOiMzI8cx3nmpGoQqoboDdIiGyRaMkSqYHPVFmetKDvpdeZNKTBpOZaGyOItlNNNAAqgDUdKvqzmfMk1NkRM6p3395RJw8594qVIEA9z135w7fvn1YvnxNvnx5elCHI8mI1YKNaqMjIuLZiaQM6xHKCACUAx9yt8AYG34ZUzQO0LSmJog704MU00HhLimtGcmBBkUlxpkZP9LrBJK+ujzY3++tr8FIZ/xaQf/klx+trqze3tjByO/cvyWi5Ep34dypDDs73/3ud3/8N3+N1ZFaFpZWN7e3QqatLdTWKdoXEwYW23e6zdIKm8Le3gGHrRLlHP59uiTsVPYJgFpksoxFNSMgnl4NTO1XdbY6nPZOcgQUKjUfGIWur1fX+hrjqEQS+Wu3N+/ubDGKcUvLmFExQQPVsKGW4fzSAilJ76gPYeOSHYnfgR+ZSyYT2NegpzFVV4ObJqo4iXVJbNkqS5AkXEkejZIh7VJ3CFfQlJUxZcbKxKxrUcLu94vjo0effbL72vtrr33N0Yy+QcaQBFt+x88PCLz9jc37b7z9+c//RlSb7OhljF0UsFNU5dHV+Oz42fXF3uLVw095zHcHGxs7t7bu3FlY3+48eJuhmOTRvaDMkxBZDGOP0OF4TRoEEjFfOw1kVBYbi96bPfyG3QzKF9pNwLcMTFTQA6uz2q1XcBqgstOh+p6zbvjKOZFsrd8ZrDmnaHF8HKMCUjc+ev740bNnzzgzRxx0wBPxmmN8iI/JnXlXEirgZeAzlUGTEgd0RawabP3yUPB+9dYGxb095LspZrSHdp+V8+r305Kr/FQhf32S8WzPyvMA7dq3clbmSZMy1LO2FRJ8uYq/O0Vna9zB30wgvoA5f7no//b+xNi2sH988NmTz/cvD+A7ioXopHlVtAbOqmhp05RKT66GpdMeVX6t1nL0oHiZLwxLu/IXA+5mUaN6HtvYxSIXPx7qOXuZoQ9ZGjn90UF+DHTZFhLaSjfkI993XpLdDWHSEKqkkhBQ16yVf5+HIAQwRVm5cUnJxLqRMnlMangk0sgEHc/da5tTKXv2vsaMXDhW8oivta0YsI9CLjOoQfrkaZcRSfuDuZTF2lxMaUPpLQYnnLQVkXjqT1oSiOPBPje/MPsyP0wkNcOmyKi/BiX1pDppaWvNijYH2mC4N6Fognw6Hz8RItjhKCfeP3jw/t07iSD9+NHwycNP//aHJFt6L+PDxlp/ZT5nqDFjOoib/xzM0suiodY34pDF1SxzLR5PUfUTJoWoYgdalMIYrdEIC+F0YgGXl+eOBkKMrS5wH7Gez7cOJYWdRjkBQK3GWTo+PrJaiNU7IOtnP/yhMDqqe/T4i93PPr8cja3/E9u1rfUtNKX6jnqXCBBtsuCgVUF8l0mXH3WFmPpXnM+gKznW7BisA0ejkruUPKRdiG3TDGpQ4w5RXzXzrCwZZSmIIq6HM3pWtsSgba2nsog3QVGx/mlNTEFVPFSXu4h9K7m1M9xdoVZ7gDYVVrwn5S8rvyaspkmUjVeumG5UPVgK0dIek5rtwkLhtSDNZ1989vDuaw9eu//6091nzl24t35bTKJPPjp+9OiLB6/dXVrvL93bef98lH0j7E8rKzu3t40ZynE0OgpsOcL1Voejk0PbGa7mtrazz0dkaWc8sIdRgsnNeLCeqtXAu6dDWgMQBYu0HqMyRlgDM6aMisjqEsxniAn2FMbKh8Pln+l5Phqyf633Fm2fe3B7C9PqnI9F9qal11qMCExAYeEsm/5s/jo9P7BZYnV1g3pmjV5QHw/WxMAo3Kiu2fz2awLWF7NmkkCBKQ6SvQ0BaVoTtMqsraHOuJVV0lyF8TbuimxFOuv1e8d7e5/86ldvvfNt3HVpkeejI5cJpM4B5hqxfPnw2e//4J8+/9f/UqBdwYUyxc7sf9P6Y37P58e8FfhJWD1ZHp88P9n//NnnywI777z+zsra5vr2rc7mLfE557p8n5eWE3fBDiL0Mwe+xTH1egEppcZcL50v9hYtQMQ+kbhIVmUBHrnhu2GJN3JLZilbHppmjpdADSHjPQIqBipQHZ199umF4MI8SpYXjo+Onj3+gns/26VlKZyeqAe/TbOAp3CSZRvAMoSFusY0w1ojGgQoXHVveFsJk1sbmpY+e/sbHhR783M/9WCaMh1EE9csyFW0NwiQql2RQ6ZXwwKJEkIGkmlCIadZfu3fNkOL7r6UhzAC32C4/012ZD3r4CBjjc55UWfDJ4fPdk92RwKgIf+Rj9oVkuULP/wJ15gUXe1vWaR5h1IFeeXPqNdqYrhAeq6b6HAQNVf7ctFRoy3CDmE6caCQlZAL/18NBgPzrKxMBOhyL7Lwa6FD0CQWtxWH8EYdbADKeL4AdCpwmQDT6lrCb3nXwgxI5fb8CjN+tZCqGEyMnCDlWAWBz8om1EVnfKun/gG8n/gRPQmnDFlRUA18CkwhSZEzRCqjkcOU0K1ISBH6uTJVOYat2CUQGz7lh+TVeIA8clINzxhjRAoM9NWiqPAfr9MSIy5DGwIps556kp4reKLQblcIqNjuLGpZxumtzK31baOx5URUDss/jJPO8bD2eWIjBkk/YSjszo3OK7vW8DerGafICMPWMbmi6pCi4AmXx9iseVPOLzj+ivPSXu3SibnqatVeZPtzw3z5YtrYnkX+aJvckmOdt1Q2Gv7Nn//Z48cP4YBTBx5/9pmt/czR5no3BvxI2yhBHjKGehqm6QoWVrPyTBqK+ZQsEGJT78OAjaiXPCrCuIP7xYwznbU7V5SFuNnhGrGeZ/SC+4pJGTKkz1W+B5cUKoXPQ85iG/RB5AlojKGlxDKZNBbKD8y3uE9ShaJ3ZTcWehrXlVkJrTRVIpoKd6XuqtQrD+hp1t9LwPWZGn1L6MFN7RQi0SZ4ct9hoGtBo8hIF2trqz//+Yf/5X/xX3zrW1//zu9/s7/9ruMXP/jgAweyyt91qPf13ObO1qNHj0eHJ6vdDQ5ZRCk7s5XsrEZLQ1edfUoYCdmQMVxzJVNzhtuoRdqfQEYLG8wbKDSydSHAAsu8ZfYMDFtfYs0F4UgQNrCec96PrHI+fvTpRx/+Ve/9b3yt89b9672PEu7HQShW++e5K1EJHci2mD2x4N11jMoVBvz8+aFlVnIt60L4ZRuvGpFWO7xrlbpPUmQKFcy9WG4la0qyEr3D5oxd0EarNd4vbgbwysPlAl/Ck9Hpo4efsy52N+46it6GXLS4x0fZqZe93pNne9bb77/x5vjTn1ccQy7uJ8QQnk3ijVgwDHqGCerBacI1OoFrYemjnz5a7Nltu9lfG6z2LYo7uGaAtc/hxwKpgkJtg/Y5GSW7zgxQT8TZgdOLNb03Ojs9cg4YvOr0hB2dt23KdcYVQ4iD6AmiSae7QsBe8Q0UQUP0b/Lk4eG+mUkG4jY3Hh/z5gOOlZ44lFbckZrQN6DIcNUsyJC3yTXhHJktjS3UmwmQG8ylfPkK3F+eUC2lJc6eX/kws7GSKkNmRMsAiGnhdHDbg3v8/opi3MzvuTqTNrfqprVMmj39+dv8xQXpIAbT0NZ4whlkeZEv7hj3/eL556KAMz4fE49yAMOLKooHp4qGhvXQapx0MTQ/jc8bwE/GQAztK+lhQo1aZmAJTVwUU8GAozTPxdYdD3MuBu5TQ7uyuT5P1jrmX+m8jnVGzXxgz0ZCFkT3rSmN0UlbCSxvXHL+fa/WYXxIQ4M2f9dVObHAfCd/vLHCd3g6GLZIAARtUPEqE/KKh5aS+XSGARS3m+BE6UxwEsbUemoCMMsR0gWcvFe4UWCEZ6VR2jWP1FrRMtOr0qZDpa3KnHUhlLddXnjI+5cvWTU9ekma1XSU5HD2Jo+oheWeKsnyyL8YSeNTLiVd61dbPccgrBKVWrQvdISv5bJMKFGC0Jm/5IMyxvqrPYqPRpoe5ZQz3YpOiyerFqpgwdRg/cTLbVixUCIWX2erZzXSrI2tjOkLdY484sDvuFKy1OZguqEzEZz+bNe43wLrXNsELNoA/i7KdbLoFptag0mwMjwSIoURStSItCNAK6YbBydCjWctznR1RYfAtvjrExaiAMUGHoVYUdgyqJLx/SP6UOBEhaxii9p4mX+Ro+LAkhg/HiJGlBKM7Cs/4A7Kp09paDxsc6VuukgaqKklL6tdnhDu+lfuV8zTOqAObZE1DY4YpTUvRhwZckUYq9bIjKHbB2xIwFMk/M8//sRGKO7KqPi3v/n1w/3nzl3BO7k697e3FvpI/LGlIssNjOZ8NN7/2gekrv29jzv0HmuvK72BGE8MuszzbEGOjbKA0e3DWEvCbPa26wWekRa1a9Kw9IrOUUQQ89Y27QeEkMyIHqEz1TNZ2ujYPZZ5Ix86yYGJaHhydP3Jhz89O95bvDh9+66gy3REKtjJnJBX5C51jxEY7lnZ2VgnrZ0Pj4+FvOZfIIZpx3mkGdOgiLq0oR4mjSwkSUKDm0wZv8KcInGSW29yOFr0mOTzh4tvsMtvLgk9oemPIkUibKOR85OdqeBAQ6srZxq17BAUVt94269gig/efONHP/tRO3DaeaGr/XVIY5zPzkf2MbBMmzRRT7FH/vCFKxej3b2jj54BF4f9/qC/voUNDzZ2mIw4xyjW6JidWGrWhDtrHTE3ognRkk1ErvZHo/19awfdzgDycX0Vstrx3bRk2uDx4VM7UPjiAiIEODw8OB0L6y/yl6l2KcS2vYBpvwB21rrBBq5ibdC16GC0LsNq7GvJydSDoIEn6BSKpiX5FQRucHZ/5QqU67qZLqH9bA/us8GaZZOC9DQ0klhlJKNPW+ZYLmXxX6arBoQrE1ZbW1rDqiuyhPukyakonZrV8pUPLcOXMmmMj1NUpnCKVhogsR1yuTr69Olnnz1/eEIst+1M1JUiSmlwgWjangAqTcmtcZCGvX5PMNk7/9IhPdKI6nBqS3IocrLWfdH2xwzkiXjcR2I/cAvM4XVO+1hZdv4K1hqPS7vKaq+kXiEcuC9iiKhlQTvgiLSG9ReElNpa86W+V32/1Q0BDHyrhDxo6qyHrxYwGYkAQk5aBFLDZzBO2kVehTYlKOC64I28ApCpqrQYZIp3KjAYCghBhdDQwokkB3d8Uu0wedkWokanJocwZkxwdwfBRuC0ozqIL7v/C/AeU8TkAqeXuxC8r+yVXEiYJgYbqrfOwfWpfQ1QhFnJBheTfv3u3be++c1Pf/Zzse4RkOHZ2SoTp24zCI8FtWE1cUqC+ChdHEfUBjobk0gYfHYlVbRONIEujwEzrrGBUSFpEBANlzcZcloUnUWAo0t27Z6YATm2FvJnXQk/iOZHI0cAWJk5PZ9f9HtdhOx8yDHqKrESl52Q1FUC2QDmoeXaCUeQLaXAZLyPGFB28CINbG7hnjBJoi+Clu6WD0JRw/5qDEo5Vrm34es5FMwwJU+7auyCNRbcStUP3Gez1Nv6KnqsRAso7ulEBAvbR+MgI0NWWD0RkdGy+Dq40kgVVXW0ZFPIIFGGciFqcro0K/q7lOlMlehDY+7D+jZjLUEtSQ9W8yMT3vWMtIRJipD19uramw9eU2LaM2cL0uaDe/dOR6e7H32KqtPeOIdRgodHR9TcN99588mzZ598/Pnuo13yIvKtFofl7e3teat881bJoR2pNOwpzdOGQjAgbm1od43xuba1RoJ59d2fzIVJf81BmGACwXZAyCDZ58auPNdbmDt48viXP/3bBw+2lu9ansxWD2NFFMIYeCiZO3yvEhqiuMhwmEhqQnQETUt0rlpChlsD3G9eGinDLAUuRdZJZ2rGeIrcJpfO6V7GS//IO6a0yIJ2fTlXhox4Kv7UopNTLyjFiJpFlyVbk4VAFxH9/NxR6Hyhne3mlO71jS2ShtV0KMZNlVZC2HVFiqVeZ7HwjAd/BL8FJ5wuwqEoJCaGiBrDw9PR4qNHziK2aX0VqcRV3GEaedrpTA5UQkI3tnb6GztQ/3Dv6e7jZ0bwcrxFvtzf2zvcP0BNxBnlk/P08UenZyN2GhAwgoBgZIWQtRDkOSENiKY8eNrehTgZgFSy5X839Exzbf/lEgM2+H5hpnftscCVRFe+KCx9BeAtvb2aZWuJct586/mVDLNscmqGe8PJljO1TvtVTcZTgoTtam8zqq4X9Uze/uY/qvmqjyQWYIJBKcDvRAydnzu8ON67OPji4MmT66esJiiiZZM5Jt6oKbK+mNlF9qe87qVGhJRP+QhPkdRgslSi2RL5sGqviqccLYtJVg7gFvicjDQbjbK3lJ3jyJ6Lc3FpLy82Nm/tbG1CgkdffBb1F82yWbIrBqEDfcuRO2JVoX9BU+UBa8QD7Z5cU4hMSFRDpunL2V/Q8UWJcGl//qWk6A4TBllZ0wcFuiLqyqCfFPQAPUExdcckA0VRReyRjyE+ahwMsMeaFxCqFAqjkDbG6KuiUmTKTbL/cyHcV+eRmqJRmrZmHmNwPKJNPcH/QqxjiPY1Tm8TlIcoG/6E+mp3OEdGC1SrxgQKhurqzRTIQkgbGBwoSWikJF+2OzlCgqjLRh1iCnLwtT/+ZxvLK3uffdLDlg/2jp48tNbnOPQjC07Xcyvrm87SWFkTRO/8+Gy0ubF9cXrBpTM7iZ29srgQVdUJot1VMYEEE567ODo4Gmqhc8kQVEyTac2GFMri6OKiZ7/pKsPYSjZznp0S761jnY1tIV10hMPweLyy1sfjiSGIEudhrXe0MhMKaoUOGAbqqGbTLbll6hOWlewoCSusUQHNQDjbbYVOjOZUy5bZNZmwKLS4+KDEPYq+aL9QOd3jv8Be5hiGJM745Ai1R7nXdTkF1aAu2H4BsUXyiD5WK2JEMTtkoLEUUYqpI9pFAbWLw0JaCknAz4ysz20URqFYT8Myi3BEeDCc8e2KlxY6yCzvboLKw/ACtoYY+4wirvjyiFaWrXxXWGG2mpzzXATt4XBEf1Isl7cVUYTPLoeHRxd3z3sOtDg/ffzwkcVDPuI0Y9NWcEpSsnEsI/PS2vp6Z33zajh88+23jo5GPz7+ycnxSHRYVUM7U0XbsHPs5t69exrBPkVWUhz4rPV7ViV1Fg4qH7Y66yIGORPDudEYjJMDGS3SX8DK9nermZEZiGmZIhbS2W5zGoC16ZOxeB68sVYcf1jeBpc/+du/+Vr3W84BZK3x3VnnxP40VtxnB8PDk0c7b71jjzaF8HhsQ/bl9tb6wdllNlUXeDWvTcbMnDCMci6pGVmTKHNKOoyJuaaomwGRht1niagjvHUiu5hdyoN9EYuzFtXZPTq+1Vvvrm083zuMbG4pem39UGgLWjCX/6srMs3aG7cc77bz1tt8rETxsNHNIgpJzeTnanFyec7CpPsMAGHAsQ+ktWq08VuEb/ArxldiCfUFN2ZxPp8/ifQbf1uGtSMkLIGDBkKP3Nq58/Ah2Xj+B//8373z3TcO/6uPBCG2jZuOe3VxuLmxYB/Roy8+Yvk6He1lKdh6ElSMlyHT1ZJ4G6k8DgqWjRvZmLAEUxgk6wKwYmaw1unFcd5B12qDej7M98UgjHXMHnInw/Tys12trHb3UuIrKe2n8fJgpNpPIyWne/5Ui6q0DI1k/5ubVVVaUq9kThuiyOSq1tVTkyca27hZucx5P721nymqfZXe6U4ociW0Bz9Q8+iK0tEQ1dmkZR1gOHfy4cOPPj78QlD1ccfWbIdM8w4w9xsziPBalyI9hi37GWU9P7W23sfWMH1T36F4uh/uHalVYUZkAqJWXJyZAwhUnu4b81EW8K09w7Zd4YJH2XPJ1LKPQIggbdfvctcaMLoHx0xdfWHxCvI1Y0IrdXLHR1TW7i+9+Lt+1FclpKRl0Ch8vBX16z8N8QlgZQYXHAvXy+KmQFeRg+vDgkQV1X63UfHhBP/yKhpbENEHwBxvp+LuQKknys4amAsDpkyztmJdBhEbVgjenA8Le4J4gWPQnmd2aW/eVsqkdp0rlSQZ4ryteWFPvk9r65b+ZKCTZgy9Xdp+/a0sH56OLw92j7c2H9sx+vHHwulcDEed5S6HOqbi7qqgtFfEZns+llcHi6sb3GLt5a49mpFodAtl7IrAtcxnDSGbG9sJa+uCsJeqzjLXpV0vfO8MuK0+qzmqL4E5g5Li/46H/G6vbMnVIqiT3WvREZYTD4tiY+uaqPhItMVZy6BUbh231hYpCkLGR/fSYTi2pyQAF3Ne8I6MEy0rAg/JJNIbiKOp8UuIIc9lhlOk3E1td5dKXR6koCNWdjLtS7qSji54BeZS3F2BrA5OSQl+jMHTnGNbrku9rbTK3AYhL+pnyvF8s15lS8F37RBQY/oXzInWpz0tp7ee25GunkUv2T88yDncc3O4rE9IWg8//exf/j/+Xw9ev+88Qgjk7ITj/SNSE7XLSb4/+8nPBZrGU2EfVu8TQHa20je/+c29J/tYOJBy5hDTWjufJFZ2lpl1F9BisqLJEVJKHWI8MWGRigiv1S+faPmk2cFw76vjRqJBLCYMRmmyVqYW/OFQRIQarC5ubQhAmeNwa6fp9eHR8cateyu9nWycOBTS56h32RFmldtCkLzkHWXh62FlphZ+XpfaXa3e9K6u9qoR38lbP2C4hhXpixEupq3rU5I2dpQ1JxzTx4nwB+u2NncOnu0fnYhPfmx23Ln7OkF5NDy1p7+3TB0+0+219dWT8fCKO/HC3LPd51ZxUENn0sED/9vgrrw0NnWmwZBUY2IthQlAYuujuQkogJlVqkzdgK15MGhW8Ui9Jo1011fO9oXYH9o8ZkA+/NHc66+/cX31fPfJUXfpNfuaVrbJEedOMj4dPzw/E1wlnFeggkjorFRmGgHA+YRcTYsokBC8NdypPjMXo9XIXPmZPx6qNQG5x/YcMDYZxYOc7Z4i6pr99DB7nr588bfVMvvdMrd7pkDVrPQ0Ixc0zGSUX6tcHmaFt5+hwAXeWZlffph9Mnv15ZT2apY+e0gj/OcOGnzzHKUpqMz82ZPR82fnw8PO6FisVGHSUCBolVya2JpUA5y2eZDaHmZNqIcX/MWQF7R1MzSe/JReT4Zi8lGKTSACP2NLu16zg4StxtmCJsj4+Hxvf4/VMRzaTrijY9nEQmomu1KabTuIZ9AUcN5PnrUuoP37XeG76ZvRibOFoqoLVVgDgaTqTBG+qkpt0SOJT6EYmSeUlViMEx71elmMepI9ZSijniIjGr8YEgnKa0SngalqbcmcYlIbwttoECnYfiS+Pxc21J8szJ9iIDb5sQ9b16NRBMMSQisdSItSsO/VnMYrZlqDbrU5MxtIZERGdtdgh8tr37ev3SsNm+ks3bq7dOtWZ3iwtDno3r593Vt7enqxf3BoS+a6c2+6S5bVFrJCvCgUznx/Q0lkKnsZ55a7g7WB+sj+9i6U0JoFXYQgg7gs7h2OGBFBvXghbel4mWBO8aQ26CPS1gUH406rPB4l4NTptWMK+3akCdCHB/fXbU/tLfCjvp7bOzgcn+/n8A7dgirZroygWRZjm8SMx/iqMw39sxgWXxzjtnRhAZOLnxYsXDvHVGjWY4OmqQyq1j4DF8STWMTKrRwUF5hCagOZrIOopXrkJ/2Y5U3OWvr1CfBnnIMthWCwYcqAo/XCi7Szpl1jtGDhd1CmxhGvxbRQfqPMlu/KcOeY2zh2+QTRR6wruzoMo/GTX74cm0GUY8mHn8QJIbK8Yg1V8Io9xPO8sYY/+ou/fPj5pw474iQy3Dv4/KPPVtf4Zw3Ox2e/+MkvBLC8d+f+6Gi0bHdPIjnM3b59u3uxfHI4Yu7pifCxOL+zfVsTDsSN5dpzPNRlxlccoqzuVjNhb0Kfpdkqjk+/PsXYricZd6PeQONPbM1y6Uh0a7MyImZmV/4BuT0QdsNanXZquPUIe4+8JXLd6q2x6zq3xxQ4GY7Ojk8G28u9HRaF7JITNUY0J1ogZZSHdjSGMvbrTgOcv5pnwqhai1pDklJXnFxCICIma7G2FKnD92IMC9oY2ZLNEim0M7e7ayNvd3XzznDfYTP91996f211y5n31Gai4snhEZXRrBjuPV67v8ntijTL1wB0ratAd9VDWkts0bOjyNbU0LCQEk00+hlZc72aWhBDAMDqPEwwqz0gmp7BI9+enxw9vOAxWfE2xLf81S/+/MmjD83G/f3Dy/PdfncVugIIb6trZ/zivvHyx1B1LuENUGzdL+KilQhMDUYKV4c8QVtPXoAbzMvV8ldKJoB0r0FRg+pHA/Lsni9+/fXr3kpXgupnd2X4oa2xAWjU9AraAVBNKpldBRxpMrVk3akrk292pfzpj8bFWzurhHq8kWGCObo+S6wHNxHfArS0Yf7qtHN2dDnevTj+ZP/RI2v5Tn1jpUGc0vZMh7TfECp/UnkwrliytlQf8s7V7u1tJehNeqYFkA3CBOiZdT5reT3zBKTg6rksYu8tL/ZPT64ThvT6cpzj3qBsODTst9ffHO7112giZnNiuoYQGfLGRSbTowG51f/3vGf4y5icmZUJ4EqDi3PWr9mt9aPdJYYHF7wio8d+kCVIxlse0fQtoVRX9E0LS2WZfNVGxccBtwHOpA2y5Dn1ZlknDzWbECAtS0tYS7EScyMHN1j6TJBbpxzROmwgkE31ejGpoxAagVZmjWpVpNAMqSzu1OiMUx5yheJl3Ot1mlYY6qeHa4eupECUM9EoO1tbt3qr7y6v/P/++3+1dOveu7/33fWt9c8//YjrHM/OpU3nFzJnjdgkRdFbcWy60GBIy+WpHagGD2UW4YwSxlqMCzjTdMSYloHFz9CQS+tmp8sJGOubUsZwWb+ckyc8EFsrtSN+LLW+Rf1dYSDd3N7orvaZkRd7uxb6jsYj0cwxNlTGng61XWHo6D1Wj4vx1ZqbH41F0OdAxD1nfOSwD7upzudyOpJxyFCG1SkB4NjPrdSaAcG0Ev48Z9pn9gbeyTPd7wt7/cRiVepZup+BapF4DxAb+Y9phxFzRpsmw5aiUktd9VW7+T2pyyftreYpvFGfVqySJaqU2qp8l9IkurdmcF+3AUHDrOmws5PzuotsEcvPHj7FEgRwpvU+u3h6sblJxsF0hUqxu/f5k7392/u37t01K+0TjNqN5w1tm7kkHBsY21MH/cEjK5Bh7VRw89s1Wu721Q51gSCctICg8Wmz/3Sk4VtBJgpf2on1sNyU0BP5M/Q1tCjW3s7t2wMMf3NdvJBVai2Pg96KxSlhmZxIsHjFh3B4ar+EojiOWWpfv/uOdvChdxIwf6OIIaaOqcJoMZ0VJb1MoF8wzrO2wQDtSZszX3KlecWJawKF+FgM8C4kj4twCk8uHl5Yl2Bc14trh+dH9+/cevuDb22sbp8wTIcgWiqPbinm3fXSVe/+nc7nH33x+ceDyDbQ4jSrWY5Lv8rScTOQQsYAIkXngpyRAfyoBmYrKTAGaQPUlOJdRAz4GlHH8/H4wMJKRbW57i4t8mp++ug5oVPQkN0nJ7sIimAyFnoJ9exCK10h74rMKkIxbZSKIqkxCAxF/c2rRiELUCpKqupD67Ujw+qp2IAJYxmtvgZFv9qVt4XwVUKmRnu4eZenOuVvMufP9GrI794eqs7iclmhS6ZZ/vpVU69mXKWnqOBnKFqjgdNyp39facaLAqfNmJU/e0ie+nyW4qGQPbJ5zgcRHsC5JufDp2cHnx09e3Z+eJQg3bZ5ZR1KiwmchXLV5AnfBerGZSW2rkybmL8vBqL6AhcmZoC0xegFFlWCvMXRFsdONTdjErNB3gtxdnhI5OTf42O6L+qAjvC1MX/N+ZVej0uWwfT5rFftuUmonmdXzejZr9/xIbirgAnf+00fT+DSsoRhlyEqo67zNFTWKbopYRPHss8KCckwZOz9DQQzl3O96NSsa17Lgx96DZUhdXYxBch54j9UPB7HOrX8Knph2aY4LcX3JoElgk/tgv7+m2DvFGdCv1NrWkGnUJmHzNJ6yp92ZTLVQGrLPG9kO1s0aXWtk+1G1521wRtbWxuPnm4vLPzen/xJ/87t5T//N3/7o78aH+yysolGd9qdW13f3t7aoAY/+vyzvaMjdLe/ssTPaBFTXh8YawDBLRiDj09F3shqJ6yAhzwA6MEoLLIbUJQNLC3GpUuSdAjBEl/pqHBCCS/3VvuDzY3VjYHNqVY4nw8Pnx/sjzmKIVt4OmWk349R2IHmtMDePK/RFe5m4mdcqVhQBIZoEyN+ZXMLXLDnxdwL6+XLjlLXyZm8f9m0iYEkHOQD/9RI08V0p/9E28xaLcpvKdomLOQMCzTEOt0GVi+SAWwJRQscClGriLwZZTl03wOi7g8ekefQ6kZ6NB1hCiPHU72Khx9Tfqm/UMIAy6CDjQAlQ53/QQjALE0gP33L6B0fV545WO+5dfhD5ntiifwnnYWeSIeX16PxMasFMcWhOZenFxzinz7d/eyTz52k+9q77+MwVnmvTo4KWYTKsV8ivAqYyX2EQZZhLLnTcZpsh+tWOB+Laye7UUtSIGLpVugwLhvTgZqo9ZEZcpxFSQlu3Jd9CghRk+WFrwDf7c1vb2+HAQ/WtkURSehgq7mOrO/1BpvGLucOxfVaQbGlNJlDqPKusCfLTjpep/8T6SzfGO3GgBtkAMdDg5t7uyS6PPvfu3CQGpS0xrBnxsThwIC1HmU2JYtr4d6tNz/+6OHnJ8P19Ttf/+4/vfv6e0LbXRyfwEOnSy31lqyoHpwdrW72Onc2Pv2v/uL4YHd7FSZxdz4hnhqIzPGo/uk+tM/MTz3RiANuU0GbwAj2wUfPkyUzOBOsrHb7qnowd8lFYrnLhYJ54pC8ko1JNvJzbFyJ5Qbi8GMIqw2NspZ/uLKwWmQEkkZgJpgXfikQarkZcg1p4kaBLo9F6LUmgKwWARB8DzIHj+vjzIdkMETJlfa1q8G/Pdeb33STrb1uQ9x+aqJC3EGJ1j7j8bGdp0lZhzJN3Wu0MyNapbm38l7WfauK9iJigeuVNs1SZg8yzPLdSEz1qSQQYqy/iPp7cvDkZH/37GhfmNcoaghNvpWlMA3gVNdAGQS4UbX0icFhkiFEvCZVainwhgUEK2owYEvLEFLTrhBDTVGH+WVyElcJ5qPa9Y9eZCuwF/NWSsKA49MRVU/xcTsEZOmKD3u5edWvVoG8r0JrWvff8Xc6GDU2qfJL+VvKJD3TMFzBuMro5hwuZ2JbQqNKCXx+xkx2KZZiRR2Ro3hwss6Gpx6mpU3HOKCr+dzIkDymhumuhsASTZ+zc56Dhs1Oy/KIyZtJytsi8WiRVD8kk+JVpCuh++3T6hceHnSUCExslRmteiRl3ehxiVH1TtEnV9dZlsXilQ1hGCSWVv7pf/gvHj56ctFfRR3f+/4fWvr7qz//0ydPnojosXn3gXjC7731+uH+3txf/dvu0ydcpR5//vnw8EAYx/XNda4fFvszJTiWxXUo7mZpKBWCEnx2Sh6vyOU4YWzHogpjh5BJNI+VtdVF2yUwD2uVCIkdMau9CG4Oyzo7Xd/YIARcHx9hq+m3QucXBXkRqacbP6s+mUVpWJjYeayzYufpqlGj3uenAOjFCkkExitHI4iBW4u1aBtFRaGoO00dbMHP7ATMkKgK0NYYnmffumrsJhmwTDmlLJcPV8qQoVHLmrnBvrpCWeL57vLFC/SQXokhMkqTt1XdqI9XMtD5/DStPNxkwOrisWUFl1XpYG8Pg3Qck3Hkw8zUz9+K+qm3sTrqmd1esLViez959JStEqARVI3d3d3r5xRvAqZ4+9bU507F5k88h3ih28iwZvmfE5b9yzYvUbvKjuU5umxdgYuOG+npcnUIQnmA65Fn8yk0XuYCj7K55Q0G2QJsJXt9c4O8RRhSINJg67gdOPNL3NqcinAhYAvjA1HDWY8BFxvR5QUp79bdO0IXH3H7Wu6yJTVAt+FQj3pndw+BeRKSbtBDbCpDBGK9gFKZKIKveRupChosWpkIeQeihWePDy2r3Lnz+r/3J//RH/7eHzmGY3R0sLbohCjQhe/2MXDFu+rf3u6MDn78w79cFETOvDofi26OaBK0SLlCcoTTmfFwk1yfmQmAlkKibGPDoQUlzGRNOAasMJ1ghmRZ8eaAOJwa5l8L7xFHfju1hHDuO801dH3Bdl6Hq1/N2czLTBo2u3QS1zkCFSMEad7XmsCIxXO2uFfQHc8u7qtpEWQiBXhqYkGqD/AaXuchxCgtl5LktC2gfel6Bf7t580cNzN4vvkT0F3BmuLBgUAEwonVRyGQTQafRPxrgzu9+9mKulnX7Hn6Kn1yvai0kCGwmSb+xgcEjtyrSiTj4sQpU6cjB3I8HXMK5DdhmAnsZayAUjVcCn4JRCE5rkYQ6vGlG9OCL9pbHQyONhQwKC8z78lnEbpDYkz4S/7zFo8O6EMmrNllxvoMFUPPEEpWSjoFPSfa2BRqM1gHrNPEWZOAaUqyZmm/y0NrdID7EhC+qoiivcUpQSAG3Ay8tUbOYgxmeeB2oashN6Zqpou7Bmph8Zo8qCnIMS0/OBIdKD2L4a5ZwvIQrhmVsORiFOScIbaOc4Dl3ZhwNdhXGYgK4hHQZCRSQS61pxZNTSPM6uRXngzpbX1eGFp56nduSlQKapsdhVkFw8SyLmQC337rnYveqi36y2cX3dWN9773h9xSVj7+aGF1fev2rbffe3tw985gdLTQ74qeMej3/s2//u9+9dOfs88KgthbX0XizWlBe4njOZ7R/iVVq47KTw9Gw5FOBgX7Fi0YOwxCrKvlFWcfcYTmEZIuCf/PBSjxcXts0ayP/TXq0IBOvLC3D93ZrQ3FCDM6PV9dmF8dOINpzV7VgDUrbaXjM9xmRw2GdT3mN35+gX1nvpjPjWMWFP1MowQ2ALT2L9ChM5wTtILoIJlv8CaNznaYlBjqk1IqPXZGxAuSw3/qLMQgQ0iEyCG0UUFCmr0lgVZmN+EXiPEhIhkPFV4wwudB/nZVzoxv/q9RVoLa3RvXl8EDoYeUsrO1ZdeYJQA9pYW6M+hXLAVEm5PftVMF7T6CfKiz6MO2Cj78/NHw8RMnKXHEs33laPcAMlwuzgvmwCMje/nTcu264M1RFqyQWW0M48r+/hViTMlDwV8td+lLgFGXnwFHHeZIPvARXwoFyqvT7F/OPdtk5FhdJUAwpPe5fWF+Su72lzc2FgZbxZ/ihTc+PqH0DESt7Pe5kCgtYtPy0p27d8lnZ0fHfYf0Ta/AKlMg88IVdJhMSdMm2fx0RboII4GZxKLia1n3lyEEKBjL5JCS8o4MzP71g9/759/8w3/6vT/845Xu9uHHT8Wq6iz3Oujtdu/sfF/eVceLbQ/Ofvo3n/zyp6/1EDiIdy64uVjX0ItLAvEnBRauhR4UrSs0cN7IKYqdRgscJodRL/SQv3piwtakhuVhy4kZcxpfyMWNtQ3OAxZ89DORN5YdwnQGRUh73AK5lVtet4fY/v/EF4riha0qCm4ru8ENpBGjbAoIeUoqyw67Q4SS5JeUzEYHbmWUM4glSNWwIk3sNzHd16tqbz1/+ec0eTJAbSzavWUGExcMag/uSdekSMVwLz6q6UGuCb2VIXMRMGvQp6W1rs0qfOmh5awJnuwZk4kC/wJzWp5JzhsYNUmJMGA+zDkw5lik9JPj58f7jl4YCmTvDOWFrLtFUiHyhNiWdJMmNOA0UPkJmJ6nTTWwNfbTtoZI1XNaCA9duUGM9lz4XBk6iyQABiwYwAf28MAq0654K4Bye+dW5MpLp7zZc96lBgFWuZTSJ0gJigRb8A065kmdATr8wNQ9sDwgfCWRpaqqudWZ9RkZbl6+uvlz9qzYgkYcEHwVHFLk5PXkAYwnCYX6E6CYpakl0o6dBrRhkXhCRMRjUHfMM1ob18VCUwVMC2llzX7VsPlVpcXETOTMYKOhwf6iEmYQI2YkYwUvjAo4XMDgOpxDwqLV+YZtPLTEPaVFjo7tr3rRRqy+x7E1YcJ9Jx3L7/RdiXBCFCzChQc2PJowwyHyd3RycvvOfR224wi9t6T7te9+/5vf/8Fx1qW4MwnTaxNE997b74YgLC5+43i0t3+0S2m2OuuAxK4z0hy3dL4svg8PM7l1sqYpvR5HdnToErUqvdAZtOZaKADx6Be5aVKME5dY9Ez8tGflgjkVv1x2PqDtzEIA1xxHzOcWlu/euQ2pdgb9tx/ceuPO+pv3t3ZY/zpnz/d3kdT10dlg5+hg3wpiTpo/s73q+SOfRRbMcifQGAHtcsrL+bJYXrkAA1sNHYt8RYotfhmFb3YVuoUWuULEfFT/lx6vpyYltzKlFAXJKBJIjV7Skchwr6j4KsL4g/YRBwxwmoSbkVx875dzcJzvk8zGPMuPC3Y2uazw4MGYt3rxIbVQix3GUDxsc29v/eHnXyxdL9EmDbCemFc6pgTS8NFwpEb2W6c17B0dyvn5Jx+/8c4btzY3DtZXP/n06Yp9YmW+Ah6WDJejDoAsoUXHiEwOCtBvAGHwTSyIUppCiDO94qtYYAwcC7y6HHFBO11IQ1t3l1OG3vzCYHl5o9e7f/u2INUYMLzcPxzauzYYbC3cvusQicvn+2eJa2oHoyPEenO93klOnYmrgB6Jnbi6sRFXrIuhuRGuXhPGvQ2XnnvI5PAvM6r+aJlEKEKwtqAa+YayAR+tOmmXPUiJRGuuKdG0hqd4z+X18j/7p//iP/hf/MfdzW2HeBztPqayLm9uX1pTX7jsD7qHT4bdy6X1zR2C+uc//mtxTwa3VubGjNR21S8RYzRvfrHHZJOW1LSdtVYdYXsqyd4HSGfUgp95Rvj0Qs/SkaJgSUoP+awZelosaXR4fMJ7nbUgFoM5yzg5T5M8fLh/pKMG4Tzxf0X7SU1VbxUdGotuGbwMWEhu/g/VyKV6dReJyU9QS8210FDqRPRzEwR0QwyTpV31UWqaJnz135bBfXbJ5xl2KcHVeIGSiW3qNiwZlzI7pzYZNFb9BNmMdOajzrAy1QBn6hrACFjVsWBAuvaioeoKSCVW65HBliMM3gCETUwuuJZG1qd5SEZ2tYSYVNn48vSA/flcyJuD3fNdZx+JmcBbRTkRU1rbYsK8ebWWvGgaUuJ1mhMgp7EohcWcG98UJqfu6ko1JhgRZMhdxNJ91r6zMS/s505GYy1LoDwLwjEw2plmboOM2P5oQmRhu9AiHFSVZilqHX4CKAvmsWkQZeUqPE8ehiD/a1wWUqoF1YhqnXYF9hNMkjRFoeRP5pSZkltLdUBqm5MK8crdP1/piLuMxXOTLyMUeT94qLUroZPGJrJkqDJ1IsOffSLwPPPVxQrqnobmVxve1o4w7KLLeoadqyXpCG41Q8PCBtwJtzmTD41b7FrrWRCBZ6FnIRMPpoJ7TzutrqlZ+aCZgqNHp8rqjbYVBMMi1NGuxqmry6E3iI5nnINHmEl/YS0tDj4xAAqUGOMrkAcl5tfWTTu2thzJ6IJ1rnknwlKir1/77h//SXf7P/tP/z+HjqR974Nf/vxnT3afDQZr83tDGyBPF2WxYzftGcl+Oppb6DtRzRnsoTviWS7O8XbGaskCWDbPgIHoQbe21tdXOesS1JyDCxFXe4uv37t1x37kzS2rhha95pcG2ri5sbazsUZzOx0fOa20K5CBE50XFgaLS3cTxZ47Kq6h4KvPf/Hhw09+tbv7ZOXKuS9LdlUlThcOx2+F0iHYf9QP45M1QJAxFpT1DKuF5/m5xJmZ6+zu71mSLHQhskBl8oFG2oO6dHU+xhnYLWBy5l7GAQ5g8KBqaRrG46M4EbgJzc+Vb9GmEPUFWaJ1ZIOZ4gyF04q5no1PWSiwPynOqV6xUatL37oUdiNHGQrBAGPYwUAPp/zwVx9SKEF0485WLQAdm5I+z2giFqjF5ZlFQ4us7ByCfXI2t6r+8Yc/295cxfWzadde7f6aaM/W39cGfV0Q5oJbHV8nQOh2b9s9bJ5y58BI7/CfX1744tPPaL40b2bqGIYYuS0M86cjMetuhf7G8lFH0sbc8kKCMEUj7Gx3524NeredzrS0vLPSW1/qWrs4OLmYE5dtY+t8S9qgM3Q8A2549ouHvzo5ubDAPz5dfO21Bzv3X1vYvNO5s935yU9sB3rzg7f/7K+fLVwezfEivuarlYEzMqHWsBQxo4eL+2aMrXMsJr44PQpIoQUJj8c9yUZYNkFneJKSOefmuzz6CAnjE8abuTduv/m97/3BN77+e6+98U09uxgmLEBfB7tzo8vRxWDOsfUPh19svL699tp25+hZ50c//cX/8F/dIaEcn6jIwizhIUFabe9mO0GOzdb80rhM1pK8Q1u5cZHcYnkER+gjZ2TrYnhoceYx6iOfDskR0QDy0IPP+U9HGopBEQEm8jpUFF3igwPb5GEiX87BoxXjK9RCzkYVQjaLibpL5EVmxterRMUiDdTagT+tud6a/TX9NVIrJiRHa+hJbE9F8+RxpZ/1FeRpKa/cbXdrGXLXygaZoHTGzdXcJkIllY7qgQn7ehYF0vqSB1I/eus5VLO81WUFqnA9sw6FSpfcU0I10+hFTAxmSKp3eucVZ3MpufyVvxpfP1NQCWNNJgiP949nKsiPLs6fnx0+Hu9+MX729Or52bIYNkekzjAsqEc2EVGlqKiOkPuqJVVLmulKXwBddzUDQVGy/iLxWh7JAq4E+lpYAJI8WbHyKl/pS5XjOMLxvlk8HtrWeSC6d17B97pMo8iVEWr0dALrqL8FeK2IxU4f/baVIjuDEUHkZXJldMIGzYeA0oBoRkA6kWgx7tC6dCQYO/1w8rYVEvY26XzSi0m/lEGxGpcypldEu3oOAhWbFyphue2dtxlpvjO2lVRTgqZ1B98U4P9w8tT2QoiquoIC1fC6q+lFZZV9UrMC0yPcPTgIrfGBC4EnTObIAbhv6wmQBBEVkjBANR7V+YxISp6MjUwTeE3Lz19G0ZOCVXoXlcy0jHhhUTaCEt7oexiQ4pMvG3gK6pFRMlbSSxy7Ojm9/fVv/S+d6/7s8clouH3v3oc//vFHP//Zcg5nqIUlxrBYHoNERjWHJvNnuzhf8jbiGZrCcoKkYbbZuGGsc1uc48m14rC6rmN5TjHdbbzCsbH8oTI5549PhMWPTZL2H6ZoHmKZ59er23fRHN3RTGHVBK4g22rCm2+9vvv46w+/+Ozxw88effH5gXNsLi7WbHtyAnloWrw5OnRh2G8m23OUmRCk1/0MmxcLNrgvO34nXBkyA0h1QE4qTvAyGODu8jp/gAjvRLMsWuY3ACO0+hk/7iA8YOcrcy1DGdd3c2aKw8FkdWlKqA9HYgPlvyjNiuchFWpqaMLV63Bgbtq0zICuC7o2g/FWkCUEyWqgDb6XI75YcbiAY3ih+JRffPLx+voat2eknCyvKJwJCzfcPKtB0gOaRe02JmZz9O8E48Rls9ta7f6hCmpJPQAGbK5oLar2L/RULxQr4hnM5UyBY1szWO/3NlYdu2DzlPhLK5d84Xtr3a1tmmXHuVhPH3cuaJCWg1dwp6Xu+vlC9+is07OXR0jSw9PzY6LdlVgYGz0wAd6AK4yr0DTDZvyCtxTQS2EjnTw9jpWVEEQOWRY1DEM11jhYDoq4Eq+tu7baPR9d7R2PyRDv3vvav/NH//w73/zOWn8gLvKYhbizsGzQYv86TzxM+3/7i8O589WtzbXtvvFwPPDxR788e/Z0I5oHM3kQsbUDxKPSTluV5LR2dskZxStpAZz3GJdMjaSgCJl3UkxXTfBheluXgYDhKQ/Gwg25wsZjOJErY5HsmXo+re9ky7TOgKUYvwwb4b7ImCqkp2X+b/nD9lNVNa3VG2gHm6vo6kzKmJSZHuRV3fPh9PlmeiSGyq8ZIaDTb5N7ehUc9F1pQTfFhNdWpS9Kr25ITRE37iAgJXe1QsP2VXqUT+ujaRnlApD0Sa60pmUAgQA8umyEZDPd/KZuYb3mgW2+R2ej/XMb04aHl7YQWAEaxROevTkMw0dKDVVViFnux43LT7Cshhd3yDxK7vomAz0RRNJ9pSUPmlYlpZTQZ3+CKUWMFxEmM5bv1eg4MVvtqHFZ1zGBCfbmvAmM2uiYD/0zgAo1w+PCifihvWHShrHYcdBMN0gPiRsc1g0/07bAJuuyoJTWaHK0++kbXWody5uXn1vK73LP/FVwcDTKeirh18YAitzgF0ScBJiKVBELmB4G5tO2vFRRUgsL2pg0Ct5yAEdLbA+TnyXpSscOufPy+SK741ycedHS6K7Ky6lKZAIzy5RBCmtjX0ZlMihtRNVSY5raWkWtXmOkLpdxaX+qo7pRam7Kr4HP2CdPG2+DBgkK7q2enA9OU91+993t1+9fj47fee+9d99577/5l//5xz/6G5PaepbRy0SKAJsGQGLOxEqzjovhWuI1/fSSqZUf04r4aD3nLwBvRWjhfY47c0hbcVbAKk1rdHzMnmxQBus74vxhGMejAxyCKzT0QeKphmqptVR0SRUid3U5GNOFd958w9ammKpXV4XnEydZWC5YRM+lv2Go+CCXWk7d6W8QLpfSHI4754xy/S6VQKIMfBqoUBogQ1B5BuX6Sk691mPtYQSYP7fyAqVL9wnEcQVVp3CX1voIyfYciV5FmEOyaQa+zCZSDuQZp1w+02VzyuSKRiCSdw4zzhzxL8jKQGPbZ1Zwo1gTFnzlB/bp4A1ISxfHVZ8+ffaLD1def+OBdURa8unZcH1zYBbLpl/yZ+nXUXoWgcfEzRi9HVYISmHD0eljRZ/ib2BChwcL4JkIDUnL9KSAu6ypMqoYKHAjWglh3R2sCslDJmNESNjxLocvCw2cha08X4z2D2VPZFO6rcA9vVXgAyJm9M7u3rNnz0+PDliJe6vZj16SjOkZ8GQGhEDkTw6HvLLa0hcsBlHDyRIT12GL50sMwtq30d9GqM7Gl3uHw+PO8/uLD37/6z/4J9//o6997Zubgy2wJMUIS358cuqoYhYERt/YCyj4TsCeW9y+t724teq0787zg86ec0QeHh8M7zhb93wcREA306BiW0Vn07jf5cp4p4TJ9fKvaeqNv8EnVzBL6uTZQ0Gm3Yh/DUy5g0CIcZXrl29addYX8lyUZlpQgJ+UVy6VVQNbGdPMsDpCmKtlbw+zn0QU6e0TE6NV6o4UtPyeFdzyTPvzamkt55fvqSWzZFr1jedXMqfEZNP1uldPYlivT8PDp1eka+sEWU6gEXEgpSGdHp0Mj04Pj4TdPsH2OD9DzUgJwFRzOnO+2vIVYJsW/Jv/NmgEViZ9ZW1tal170T5BB9V/xsXDA9ixUJlmJnCLfSoFPBGUtK2+YrXyICX0IQqPATfkND2KkfDizqcsM2nkfeqFSakRhROZVyUP+RnNOK1SekhemHqVHlyfDGQ1+itvSpMnZb58tb61NG/9LCUbIcB9VaN+pgIWabyqiCQ8RV7zQUbOj0hAfmc0J9iZWmqkcw8wpldLbG+lwVrQyFc6ZjVSR/Ef9IzxLNyMOVoFRoKVgy6YoM2l/jb5wD00KvCYzvMGn0JHrTHTmmicCmXzx6Xe9sc9m7PV3S55skpJJIoqrj+mR3MVbvzGh+JFc5i9Pj8z1nNrA/t3v97tPvni4ec//inlDSewJlroDCPpcDk1CMfBGLMR3I4WJnbOt1a+565Xay9TfzAQA0ubQuWPxdm3QamnfVyXbWAdHw8x7dXVAWagtTHX63Muux2XtKHsxqh9nJ6of+FQVoytyjj3FPnc2XltffDa22+ODva/+PzTx59/9suf/ZQylIXbOC5yc/dVYVFF9FABT2vSA6xWUbu3GjWgCs+QTeGVv4VRYaWelNRQknBp2deghNZEdTSHa6JGzyY/uVRq7Gq1WDVYNGuxzKEk8XuKeVpQyniNqY9N2Zo7QRSd5BMcv70Ly8oxemmd7DZj2QStGRNESnvqQj+SgQun039H50+e7Fos7C2taYGQ0nd7d/BaYesAVEHgCdE1xz4W/ujcoUnYcl5RKHPcXWu52vKgJVqYtjLaFPTTNj4cLium6bvT5K8tTrpooUJ3zmG3qz2LUfzbgri6GTp4KYSftYHxISuu4H7Iyvn6xpzDmlhFiBoadvZ877PPPnvy2WdPn+xC6hMZpZp9EXPCOgC+LB92tDkqUAtAwhHCCr5gOraZcGf9Fj99rw4PSGHjXmf1/Td/72vvff2bX/vO5uatW1s7XBFy7N+paAYJskbKzOAANd/+a+clc9VfvLDisbWjOcefP7w8eNqH3OKRZaaAggb9A1wA++VSkvii/JcomEGWv+4TmhMKBJHqan9qxrcEbwC+6TypR2orOA8vX9CmFV7tedGqFJR3Qat6zPv2KbhX5tzaq+RI9ohm7VVLaIn1nLetIj9fPLS5UN9KnL1qmVu2WeZZCS2lVTTL8+LbaSe8KuEjpccAHBBUy0PP/cgslWI62RYHZ86uzk64HlyMHV1+iPvie3ZZdtj/GWeYU0plNg/CGMzfMKYb49Wa81vdM3ahAZPPtfzXfbZopY3sXOS1ArijTyil+OWhEelPrBbVwSIKWAhhOaiRydsxb3E1fI2xMOZAMojPLAsm+mMGD1cQ8YcBtrVAhlaaEUqKotqbwv0Xox44VvsD2N90aaKvQnRuXBJ9JbHd8wbPL2cNraZJ4I8ODkkngDl7CVJJmvDl6gp2E3xqVXw5paUjFvKhXRhViZEGNGMfQowBxgpmIUprMzaUG5D0KnbTwMAkkEE7XupONSzFBwtrSDGP1mffFHqx1kjIKKSYia1DS3xTkpfWxP4eyS5GuMlcRYmWLbUmMjb5hP8BkrTUQ4hLrsq+cKbhNgaZTXOi1SeYRrZu1Opm2DOk5v6dU9Zt/x1YgUykKgftXTpv7sQeXq5YOI5NriIb254iRAcKrijMAM3VD+SOomwpc7G/unAaLdU4sRAEBnTvHAfGcCmKWQy5kG5+fa2/vfn+ztade3fpVQ8//eTR518w4FC9xdB3XEHC/UPGgnGNBmUvPAb0GoY3/sRZ2mpu8TmACqMJiDMwgWSovhUgbrbxloyoCfQGkAxzIk4cLhyrY7il0qy8JKgIK1HMvUL0AoAvGDpNfjg4TyNUv18xxmlTTWtw5DmVsyzOwjLBFA/USCYyd2CQiNGWCJdm+kr7yRIRfWjJTvs7Pnu+O+wu7tEwrzq8tGJmPyZQkX7qUqC/FhR7PTHR2I/joGHMFFfkJT0FUSzdoMdNsOYiYBTRihU4uMehgXLfme8vLqwSrzBgC5Vc3Cm8g+6Z7gQdMtH0Ie4WjqSmhw8TdS2UjQmB/KnL1gAdI9/vnj483X2298Mf/vXnn3zODcMCB883t6IiGmS5RMOAAgnqUVYxTpyGbV4IGQ/gfTbsiBUjz/bK3W88ePP9d97/1re++87b71HMkbGTofhBQxt4HFigKAUsc2I5P2EgMS4WpBecv2T3LSFJaL+z4cGj3fnTw3V7cI+JDnYeZdpEIDDZXrk0M5Oq0pMhE+mVLDd/Nprrg9nVUgqss7SbDy+XFqobUBSjramdCsGy/ilrWntxoJQTEpE8KcezCwLUc+6zdM/t54y2kCx9JIMUKBqWHGxIafDKm9ynV/uqlSaDbPXhpKOz59mDbw1EKy0V32hJ1TOpSHorqrKEDvrZ6so8qiu/NadeuSe9qg2BTWLrrJlW3DcZg9XxpGR8ZowSlf1K/N7T4fn48OxoeH48ssWS0WbyL260eHgE9uLEFWWgpomU3+HSpkKPQpVqoI+1ZlbUzefOou0QPC1RTDaaugIxYxHINyAYkyqw4ME3z6KKvpvOCehiY0K22sabAEWNNEwviv820hwQmJzWPmkyxhER8JMhxR2B8p86amwKrL7xwayZv6bL3rfu3chrPmTe3rzqZyoqvpX+F+9f4EDTxRbZ8uwBRFqRm4yeIg1Wsa+bpXiG8ZmQxQRkTg8yN/JhS5/cIy2Fy8MGOXDCmD7yH2oNwclYqmZlrSO+WfxYT3NqE/km0zp2yFTGFteQNYmBR2CoLH9Uaj4lgHMyJjGX57YSSWHUJilmXRhOWLui2TtDieFhSUwBkxljPtGXbEYKy3de7OmJGLSdwdrO9ib2LhPwOMMlLqbKhBG1hSLosbKM4kaZpi5k9JDErn/YsHTrhDRkFBcr4dFwcSLEv227F2J93NraxgZGozH9UxACjJF+DHO01pL8clcU5WhIvlRZNMDWtc710fDITuIsdurOyQl3bmun63fu/f66Hcb9vYPj508ecxTCGxiyj52P2+tX2AQUwNWESAWHvig54mW8sUtNj7ucAB+py5VMgANUYbC2BGK0WQLWGOZ3AGTStBfARgVTWn7jR4oKDut0+G0mctahY/Vxx1mAZ3F4PgryR2oKjsWtojQXkBuFwsKXLMwkqAl3QfbeXhCg8IyGxuUsg6l1IO6Phdxsx8fXO3MH+8O5qye9JdtGr1mwcCcl0IPdzejN7S0T2bzmpZ1NayK9J6yE70IxDXvwJONn3tpmgxuWuJYR8CpYlFbAAedS8GLgNVaGMZvKnIPZXV8TqEJ8lfhHZfkSLxW+/uTKNoqnz4SfJKgRSZyLxTX54vSYQN9Z7Ytdqbfg+Oknnz998syR9FmUCTyIBoFZlGBxG8jwQts4WkScmEVCIeqxQAOOO7rvr5bv9e6/9+4H3/ve9z744Gu2IZNMjg/Pdk8er3ZtQx4YcqME/5j+NO3SyUQLnd56r2OLvAMH4e3h/uHxqA4Ou+YbsrrMFD4U/jNCiCGvgQLD6WU61ySc/v4t/wbI0ClzOFPVvaUk7eaVGZkqvYX7mf711k9XS68M7fcsJVjR3tdDIcm0ZCkZvenlcx/71Qque26t/JbufiPP5FVVmVnTrvazOXPNCvHQSph9nodqiYeArqptbytlkn/anpR0s4T27az8vAoxnQDt5kNyVlUSvW+vnJeZzpMpYpCJwdlsPoMFAuEjRpdnh5fjgwv2ZxhgblijMmVN3fzL+DfynUrN+VB+02XWmN/poTqCnmdWatuvKwUDtmcBwl7a04mKkaORWRAvjpDmF/rVAKeLllH9URzaZvKFcghoZBHQoEVViMkQlbAfVfy2It3RgNE5SFYb1GjDYckM2eZmwgjBjyoxDUWipoP26xpcuX7bm1ldxQaKMD0Mb05kGcnoLhIRR0crmRE3sjj2AkqT8VYPUEjPb3+w34xN5AVIHj4b7ACMGjf38PMa+lYW8DF7oRxs4Aio3VARs1Zsi5pbkEje76b7ClFSTNOgRAjTvkzF1FmD17ob8LsiURrXYEh0mbQjV2PDvskABZeCiFnBVYsHMkfIV1iqoVW6I6+GJ0fWD7uUJ98fH3XEgTzYU7oO4Sj+WZ8MQ/FJfJ1ETViyEondlBvwnLVA5Hh1sNFnZ1zinxx/IdEIuQUYV4oXLusrO21svbDRgg5HM2FzhZMAA+fEgsou84U5jgj4YjqdjoRPNbTLT9RD5PG1NW0SOzgblPXbYbRLy2+8+9745OwXP106en44cigxAzGsTfAs5CcDBzkV1nCXJq0xMFyNAVowOztTkW8PNZWBx2OkSzwWCBXDA4kEAejRq5mwzJPyeAzG0upkqstfRV1SCevSAUWmD7HNGtuIsO2VoaOU0w1NNAOvc3JFKzVO/I5EMlmZLy027bdGnLaR2mJIz+IuvZ11t3LOj2NhP1rYXJJfBI+NjYEOeu6v9jBg+0rB3E5cXdUEaE5VV0Lqjf9PIBBdvS5LeEEuAxABM8y3wJPf8WDk8c4hNxvGV7v9VTu/rQGjZSx4FmnZvJy1dDk8WBiPHn72+fnzfbGOQQzSDUTH7ORggbXNdW6CBAoiBJuwcHunp521tWwLzvpuBEJjT9oz1xwGQtgSVcsOAikLAleMhevrnAvA1Z9f/Sff/3e//a3vfeub32FmF7Rg99GQJLc1uNUZzI+5FewRSq710UI/ALLo27hM3HUOMTB3xofX+8ePnj1/fnhkx5fYq2shXgvXh88Onj5N9YUBYA4IUwwMTgZBMkVdbbbV49/zZo620tynVyAddClCkL+zS17P9X9IWZ6nd4hRRUkoElotDCUKwTLdGy2V2hB7WteNvzqjjCpHatpTz6m8Kb1SqkZzKK8mDZmWUG2Z/EiXije0T6ZlVoFp56yWlx5m2aZFViOmmTMTZ88m1LS+lu5dPp8WjuJBp0Jc+BsygfWeVzgfbnuExbGj7S9PDs+PnTxILB5ejU44mPCLiPCY2EOMLaZZBrgNdUhHiPDvfqW/7dKq9jBrPDBVJyd18OAXdSPCtZkIcV12qJFqtaSYCNtbhl8pmZdu1STswVw1qa24ODiYNyeFB70Kjbhw+ppdK2cEbqI63UXmkJprC7FWlLBhXIfrk92f2QfI4bR6jMkVAZ60/Cv/aAaIh6TdmAkNoSt/TEONkbfP5as5FQba+s3+3FcGKTnigs2xETvRG8OonAmwbtZdJMnYz0rWccX61+DjVbDfPV+x5SXETjAhv1hOAzA1ZRWKs01yWZsMB6XyAAXCwzaNYaF8QGTziW8VngFI4ydXKPsMLSZpk4Gc4L1WVcMgTVJY96rNnPsqiA/VivJNRSfjW6znJNPpfPrpx8Pn+x+88/b8YHDwqw9/9Bf/9l/91//1YlQ+hsdQ7RBFVAwixrBRrkOWALNsfe3UYY4/69s7lnX7awO0kj1VtuVLak62oRw5cb1znSNNVxgBr57v72FosKvxAC3EdZDI1UFWMR2TTuEs2SXA1FuvXOy53aUBSXY4Psk6qrbjrji9qDFHR5v3H3x/dY0K96N/+5f7z56Bz9r6IIMpWmgYiJEyH7m+xhuqgFOQMQJEgwo0QY81NIFtLtCm7JYQldEFwsQbYdc2rHRduSxKgm74u1nBSG+0cHgmYjpiTOWO3soVjutOHmCGj3NZyUna5PPi/XisB4BOHy1XRwhgCYjgK7/myZn5yN5gFK9Kr415iV7uCJ1EFNFruWy/YXdlZWDVx4Bz6v21szn6ZB1MOnpnWhL/LqAzqX3LeqycwBh6So+fV3R14hmsyf8ZXv0zmkEA2nN8nTWLC1RWUCvqGdHbRCaWZMei0/qiYIydqPr4C6dw2GSTPawCKw76C9zjOCoLuHE+7Jxxl7PtteR7qOWg3JxjAWYloIOivYuJsuroWhPB+oUNaVna6s4N3n/9ze9//w+++fXv39t+y3Zr/vMHz45BS8RNw3m4m7PaSFc7G5va71kP0tBer7PVFeKqs/98//nB8dA512bbyubaqrNDyHMxM1IaeAiOxv3YMb6CAmQe/+5XAbmIQLCrZnNokNk5LS10rKVnmoNHfpq1Ne38mVzTb5On8rd7SM5UWzde3k7u+ZvH/H/jk/ZTys0r1AiCqLYye9Uq9QD9Wk6vZpcUONDS3SfNr9/yVEpQfZbBQ35WSkt/5e0kQ32QVxBu+nV+3qxgWmjEihtX43AlZdfkCg2HuzFGubNe5dxmJ+dcjWHpyeXJ8GI0cuf5bPNdx7oap0qhA/JZ4KDkaQPyGOHld7qgk2tGtz0rrpURilRvJXiegHdRZDs6ge2SBEnzi9OC+Am0haCKr0MK6br5Mk4t5qWVP86rdrkK47u+vdrftO1VSznGlhc3QppFOHPAOdI5p+/6zMyP386ShST7+An1GDDJeNxZ6FtdCoURI6O8k9ikSlJOW9PqmMu0oToQqjFpUiDT0uSY9OnVP6HAudLy6B95LgvynGDxGWmkLw7I0dlRXWEHyBBItfbnS3VplntxQllLRE0h5ruvNEBTU2prVfsIhYzPeK1JBJWDQnCDRmfHKsUt8QXjOSqmVAdtRY/BrY+kp3SqIZlGiaWFhDpWo1NlXa0yhWdAa9JkBubLwEIXo0p79H88rpBqp3qcLyGFUA63A1zLt1ZVUT5Lv+uDtfPhX//o33z6w3+zMxjsPnnqn+AIBLHjqxPMw5bf8+sEbGK89glGGjNvjLc5gZyFc+uWM27649OTxW5PDLUFytAZO98qvcQmVDWaq5aUWVE5IsMHLEH/RofoJrK/0INxq6uwyytrgdenjmpmKs2AAilCGcSjgdlixCJZckAYU9amo92v3767/+TZZn/w3T/8wcb61n//3/zXf/UXf+lAe6Z1Oo9Pw8ZstWQVZ1q4XDgdH2uAD1l8UD7PQMFgG15taLxwBZpEDpYJXj+YL/yzwJ1eA7oCsR/eGlgRrkO4FKKGzYhcGd+sYpk4icx56XgGRtmz9BqD1EUpXvFMtgyrO+qRmEqvlljBnH1yMr4gzPhOiCuHZ6yvD9ROWbTdiFmJtVh+EAUcxXLqwmEJvLt7z+EiWWE4HumGWqyCk0KePHykwSxaQnOImHG9s/X06VOBQqG9PClhwdG3iXanQHeF05uVYJQNW0BiA9XcHH3aN4EA8JCnexyhV+2cFsdPaPj9vcPO9YGjI/g+7z0/OOfXd3R8d2ebEZkken7mWOI5iuvzp585JaJz+9bG6/d++n/9vz1/ur++2kdPmN/mFsVDRm+68RkUUM6gOUHhanF/PFrpdLfXbn/w/jf+4A9+8MH734QuJ+NL8TSIXyiByCym4snRGLLogjUO9vbj/UP6wxprcx9tMVdPTn/x073dR8fHIxPcVmmDY2b7nwWGZWb+YoQN7z55zO5nzcUQpbTY+TIR2wRvz5VgaoOEMY60knkWklLzL+PuZV7nqcCVLHWhHlqoTjD0rkp+9RbwBv1cdAZ5fFLqSVKSuf64aXtRwlDmVpGXmaE+NojMl37LA8fqwdRK/syg0M+bVLSoHOwJ0kP7XFWL7yKk5k/alHKSXs2oQupFg87kcUIG/ZKhTeF6gyRN5YzQrxffJJcyk7WS8xkAheqCUZ6nmfHVPLZLp+pNmG2l62d9prRALWgH8RI5hVXZoi/ic2Yz20iM1rEjJ0/2zw5G4BRjmsVUB514RoV54EG9IqupuK4qL4P6VRewtpbkZZpSjVRARquaOPsq5SRTiHTyth+TPKEkZjUebOKV+Tk7H2BhABFFTo8IpEHxMJAI7aZ41+6Sbm/Due7zuG/CjKWSyMNO+F6yB0lgZApffCwuL8d0FkGNHG3HuysqzWLfIeics6L0Z9UTR2Qhin24QeCre5xmt2um1PrZetNmS72NcKGAiXxRSbO3pcxF1ikjZDiZWEpOaGBKjDBPQ0Xq27gWSBEdBcw+r8L+jtsUy0sH8mXAnTaWsuPgFQikzGyfR6KUTdW0pq7Bpoh2t2b7zihoLcXIlEghUbMn6OF3tGRT/+ZAm8AAGEUmwh/7Lhvh6dzFmVgN1+cjI1psiZStGg2a63z20eb15Xce3OHE9Pzx0+HDz9hj6kAJkVXSPriM0qW5mNDCnO0nSFsYh/PMB+x//GyvDo+HmAgGbC/bnB0hjOxZoYRR2DEH2ex7AUkIBrukQ6LTszFpzwXlpGs2LqQ7ZAKlEz7iQq59+l+1eaRhct1VDhBgH/jLnIOr+eZwulbs1dXa5tadB69tffIZbRBUNbqmQYYqslfYeRZjZokqzZXhLmSp6lAhiiCRyIvyODY0QMqLKHNAMQrB5/DgeFG7MGqzpyiC9yncYVi1b3c8d6Y3qR5hk5qgGpMm6VUq4NTmVKjwbHpkOsZm4DIR+UMKGAU4Lqygwe3ycqOdqqRl+GLJotZ2dex8xfq5+D114hO+i3mcncUQDdoqVRvtn2+aT7hoZGdUpJhAuHVLp0x0hNuAup+ejLyNOJJRiLqPy1pZdWLu6vqaJgGj72HZ2uq64IkLc8es3OPjsbl+Ykel5VhOd6IbG307dU+Hi3HRc5jE+NGP//oefYS6YbtE6Psc34Ht9XUGYzLJeGgJ6HRJhI6FnhBqtO7X7937xte//f3v/uH9e69Z9udFZesItXZ1YVX/tdmg6ESMLYuszXPD/b1Qls1Bx1YiYDjcfb67OzzcvbjYPzs5NjoYFH0H3c2KdsKWizUjxL1JZpHacVInFsQurk8p4IDzD3KBcE3tgPrlAhuLNcVAovBkQuXbs49uki+9TPNvlmAUMhWgUSZDCpcDETFjpnW1hKSXbaPkpwz6q72Dn8WCJ0R3+vnN2n6H51/3+a9Lb0XffPuqgnsDdDPhpbFk3/qwuEYYVWyHjB8YKoT37AAAUwSLFWrnwuG6/g2HlyfnVqiQJVYaMyGrv6mgzEEB52ycmvG5QPs7dL+ygvBkHL/0peIncC5mnGyIWJ8u2y50M+MY5TNdgyD+6VAzzEB4ViVHGC/zwRhs29DZ7W5YuYy+V5Mhxk0aBwuzyDDZ6wpLlqL84b8ijSTE7YjL5fLyEBsux47RYmd7bpG8uapBEbFqLQRMp83UxOljkHUilN3o2BSVbyTV4wzPXoZFuJxthXxxiYiAz9qu/bZgY3g4DaIVRM8AkyEpREChpFT95Utra7zavbUzi64pISJ6DashRCalxK148oiWnwTIMbmGAZtEqEk4rXWyAJ/N0D0jUQKraRww1ERR3KtAyKjWBX1Q27gCFffl/YIBc4Vyrgv1LSiFDWXS1hDTvE7Hd8ImF1c3Tnv379K6frr7M/uWLcHxU9LasBNbkuaZLXkIZA3VDF/pLnOqogmJondwdGwXEZGTioZwcy0+HgkPvbJ2KSZ/3yii71wf8Co72wCB/qe1TACEPXIcBS6sLa5A+JDlW9AmKoALkMSfzRW5FsXXsVh0dcI+ZGZQcHOApnNn7fG83tq59c4HX/viiy8++tmH/LBjZk8X/ZdCwmBIkDWSCmx4nXrJmFl5mcyyFB6qDuHZeGoTGWm6fWt6Z3JqGvBj/WllpkopOuDNCoxLqZXQEvezyzEHRUwoXJBMx9uhiVawAv/VJw7eFGyGIAJNWH7EHJ3WNrzTKyWBlSHA5/sV3ZMNP8zyNNuaXXwhZWZlYklNMMxrxy6PiUj0X4yc5Z8u6oFFBXcVjRKK2dnAiazsEU2aiUiRSzA1VSbghjI5QvkHN+1XJucsWCPtDnqDrU2Bu7urjkxM5AfDZN0XjDHR8+szsbccmDg8FN7g2KdajhcaB4Gpuebxdx+PTp6Pzw6E4JhbFTekv7zpzNOjQ+dLZHTWVjdX17qntOXz04Wr3gfvfuOdtz74d/7wnznOwwY38DwdnZD7PbM4jPbG9PDapZGYYkSQbKw6P1tftbyFAB9f7+4eHx3t7z93qMxpWK/gJB0CX2/JYVCyZL8VMJFHme9Z5XT+6GDXGFoT4xgRuSWT179gX90bMWlEwHNL9KalV5a/62awGp2oP7MSJp+1xGDgtBzP9XiT8iANGodOvEjMJ8FK0gWvmuB2FmBSyqQkGUJXI/GktTcZeb5VZEhu8iQbgjS9IpvW1drR7jO6PM310t/qRVJmD+317Ofs4ZX0L3/SMtxktK1qJbR25aFdIVOZj3HbiNeV8J1WP2plF+vlgGBmXNi4N7YAPLJWwjWezJV/ZMzI59Dd1A7VdUs/IcjkPgNCa89vfX8ZMSYQrD8vsZJkW8R6WaLIuehFNJCivLrGJgHD9Cn0sIwYRX6MtcwOfN0W48YEDLePEKZ0xWFjCeWlHOe8CVqJkvdWNhytdrJ4eLowvDhj46ILCxY3ZNRcWDyx4rhsxwcevLBWDaQHu1StxNbwkOVU7Qr7VEtlmTwn+SsvLWnlvJgwEpC7fI/cxdHVHCyP2dP4Z2XVE5Ki8vpcVXxluX9HIoFAI4tlYocYWFobbZeOpOnVjXCaubZNyzaN+ZNAjyBKsM980G8qqBJYpC23ujxnLGqKhJ+0tEkdjbIDPb4Z/mG0sJXGaUPSE9MQt0LVBRk9PZuHgUHOi87tOx2OJ8Nhd2O9v7P9YGdrb//w6ecPBYhmU/V9fNKygJ2DHwx51FHNrEBXUEUbsAo7PTf7oh3QeJCGrGJnaa125HAlEGYihs0TRkL9IW7OUX+FS6TVpf9nJ5z/PFD7WFO5YquiuHDhU/E2zDCohDEaHW2K4ZYQgG2B1oUoHwGoiCJra2+89fabb7/ziw8/jEId8AJqeh/wZfct0DRug+Nid9WiGuU81UV08mV4XGI2UzVxX3EbMP7Mk/BRNnEDEd8lFDxLLug4RddkNmpB0TDTyBxlkTZsC1zD1V+6fKniVZLCSB5eNIFGpwgC8KM4LmGkXJcdyC2eMw8LJ1sktKfTULK/KGvyXN0iuIANvn9ycMDOHLu04QgDNlgxLN/++KOPAMGHp84yPBOokvfZEkNFY8AFlnQ7XQuIrolfgGyBlpNW5J0ESrRhZ1ms7yVhzByxwWrMb77vXGFRpmOvZrZNMJHsAZ4XaHN8LPDHyaC7HE8xx3PmDAObJZYUtr9/8Ozo5ONPDw5HnV/87IvOxfLa8kBg7KX5nuCeo+HpUWe8tXDrj7/5R3/4/R985xvf3bzzoCMY+MnZ6ECMsCvmOWZ/vT7c21/vbxIcRLm+PrGCK9BHVsrpER1eriNq8PODveeWGwwxnOz2Re04Iwyl83AXRmeLUvh3VsxGiIBjqEcHB3tGm0GDAymdocHkt7pnTgfHvnwBaZBwAuFGygJnc8i9kttbKSmhscbcQ0AkoAUe8qScyf0FWWqfTNJhXl35E2YUUjzJWhQ8zy8Kqef6mc8j687KL0xojfPqq69wg8mbF82pD1tqPq8MVU6kVhV6NSu2Pdy8zz78MoOfZUupRdNbUdJdEaKNNM8QU4jmEW3RP4u+Zzhu/ASvTk+csuDkKSf/Mn8k5t7FeWLd5l/x7BADNATRb80I/LWYqJyH6YydvPtt/rRPUsjNS2vzs91vvFg0q8w6l8SMlotsX2axNhwSS3jSf7M/6m+vR/fdFnw13icQJYRZcDiW53qas7qJnaAUMJ6n1crC3Olqf31l2X7QvYvTQ1v0RTx1KOqVkznjG1nRpavHZRy2SNyGsJZfW1vDFF/tUntT99mr6mTjdAqcgHT6NnjmH/ZGCnaOgcVN8427HC2QLZTcFBJvVDPSBSl3jlENgW5UN2nTyykN6ZSQTc9hgYAgR0JVmPhilAQXEVobtGqtkcQa9p8jbwS0yMQBTG0Ly6txIotIb7NRQW1cp8KBrqU7L10qqtzS1ZimZ2xCHrmG8TQZXR0eXYubPzzikbD2yefG28lFiV99Ek1lZ2P9/ffe+au//Gtr/XHfh4KXUYXRf8ZiogESFk95fs6meAWgRhZx32x1KqujUMc8ozESMhaZnJcpAhlOwRg5OtYgFNriMWQD6ZjRcwkJa2YQWa0ZQxmBOeBfFiZLgMk6CDiG72B/Co3JV3d0zvZUzeBgwXWn019ff/D662uDjbO9YwNYGER6iEhZUMcTWVaZq3MZ5Cz2eRnNoG2cLeEroc0ziRPEq8Q+bc4H2pItr6y4cz2RnYEgER6ydqNZImvohlwZZ4NC5KFPQSacfr52OpEajC8NHs8FUWWTQy8uxWMzszJUlTXE4+rU2RWs02HsHF66S5Zgq5nUSP4Z4BYfZqxX5tCbRTuzztknWgO0QRgPnNi4FApfZle0f+cXQoDzwrBEjRhFDWdvr8tY+EFosARgKLOgPcdgjME7b3r57r3bW6sbtvewefXW19YGq0omDgmyIdBJzj8kosThqmTaGJCWToTCvrocLZ/ZJtYXIK23xsLDEwYr/ezJ8Gg4d3Lo8M61s9GyAxGWOhd9kTTuvPfdb3/3D773h++/80FnYeVkb/jwp5+SyXK25ZLo1qEHtFwWQ0rByXgf47cZKjHXCFZ4/qEYuuPnz55SuBFbI5qtS6SS6uKpEAVEMNq1RU3mLi5l0AN7tkkdieCi4oSp0aEtgRy/yuNjRjcgXmjBlMW2CdgmnefMspem35d+BIVrkubhS1clpgQPQZvp1cZl+ksB6I/7JEN7cG9F5sFcCq4moW75ND/rynOJmwYuL4pcTwsp/E8XU8X0SjHT5/a3Vd3uDQgvv7/xS53t1+yh8dT8rAJePFS+1uZAPFN2cmXqTbqQjiihhiQTueUpIhhzlZfIQBQOphC6v2mFD8U38cxJjiccBsix56e2/56QbynH8Ogao2rYFFrsX1SxNK7YRCoopSltAfSQukmzfrc/v+arBsVWVHG0iNhNEi86EzUgw4NhFDOG9FEc0DuWyuw6Wlwb7PTWdhaXV3k7Zt7SSIrTmNXVg9hVQ4hivkUYMWbEsz83TyrPpj7Lbpg1GoEoAx85RJcVr0lk5VJwwRvZl6APNzFjBoAGLIOCunr2r43LLEMe0igTLGBt9yR6iv4oICWCSlPn+5X9SNawI8sjplFzsg+1tMx88btd+h3Ah7wHZIAQ5TXYlUmgcgNeqi6BjasuHkxepekChWw4gc2lgEaUQQ+8i74VmSBF1DxrPVBSydHpuqLdPESNqt0zE0NzitTNcKMKeRFOw4A5Gl0cHl8eOXFyj1lukUfxAh3l3GkbmMrt7R1a1Clv3hijcp6GNixX0FEiIkUoxpKu436ifinYpp5Cm2pQ8SHzHPSITqnOBlTXXIdpepiYjtqKslOtYymJhVbXrO/GHstrN34w2EBjHrJlJScnSUTuixgEv0q4KfKhw2ATn/uMKh68uHDrzt379+9/cvhLMCv6lDx4W5TKZM7VpnBMTgEZuBGJm2rsCIeYnW0IMPhmMlEzK8LRJ8gh3JuD3XAj/ad1ZcE6KUrwoJcKTwbzRy0VnT9Nm7/M0je+QbbKxDKOFmVTPmexqLFzCw55kINpFzdh8LaHVXvZpWR2d1QDUwFoO1yIRKKbllQxTUCOgh+MdcJgtEMpmVYngXNZ+MXlvyQauQJ1rhw+zrMRj4gdVKoepf0V8F+BtUQsRAnu23njzXvf+MbXBW3jmomRrq4NFAuAWB7HLmXZ1T0ajsQ/ITGIgEEmIN2CJoWdNMIUCFnMct5h9ukudLrjo/1nT8a7T+zlIvf1by2+9vqDd/7Fv/fvv3b/wWv3XiMu7j48OD4arSz2Nwa3mNLJj7pAaNNOVhJSD9GHcLhg+rK4gB6nq+dPdx8/Gh4d8ClAVwVkZSMnMi44pVt00oTCtvNNhBMC0JW9TSxytZCS7oMCWWb/4DkIK9MHURmM2j/cNWUZxTwC8FqjLVyCm5C/zd2w4LIG15gkm0HUwtzbVZO5/ZzdKw8kLAYbXhTxPvQupGb6YX0u5xRdX1DUVs6Xc/pi2uxJ5Tf/aOqMJ01bm/c3P5k9eyjeOcn4UvqEmU7Kbq+S/6WGv1pyq0iZ7YpWQ8BP5F+cynobYTCslwacZQni+bkF4JMWf8Mhcdl0xLZTJieFZO9vANWqrBkRppFpCoyV6hYIT1r5P+KPBtfXL8FMilkZJcY/GWLaQrF1CxUKrwgBDj2pYAwZxQUn0G13+5sRDQpDIEbLR3cBjcA7YiTc0W58dPHyvKuQxYVxCOjc2cLSaXfO5rwEsEUvzqwIWSuPlhlDKVVNrAcfprkBgEryp/FSD7/mKljl3Zc7eRN2mlRZgnpaw9Yk+g3HMQfqqRH/R30TvL4By32G//nwt7ti7TSGGbcooNruf82atCxFB0bTi+yFxsd+AuZpQDRyqlO+q4XD5quVTxDwGNBdrZBqXB7rAamKtp1cmW2xaQeXEgLXzSYhDi7XS6dzy72LBUrKiY+WLufHB8dLK73FjXV6kMoxQBtOlgRzwMtxcPZhJDzFRUjGCBmWcWi++xY+4TEdM0cbRC/NEGA/ONMSJ9TVVcxxNB5qix0qFhc1D6rhsPRjBQkgnOy+D61JCBiRn8MrqNI6nsWALO1g4j2aTMCZf/oa/7BJx6OlaaZow/gOQW9jY+O119747GcfKwAnKyNORlyBLsPqas+6FjAWIDXMZR9QdMqYoLMz38ibypGOzYOgQWy59tWKCELLj1ewIsqurQUuxF3R7gmvle9regAHh+dzckyqNo2KoqYJIKtQIUouw4Ydg9MHd5hgCCwVFAy0JCGg1R7eianUsqcB5VMNhpmqpXO3PsnJFjzX7UkU/YouCEwWfLWblEOKxsI5xqUdkRYohSFCrgaHNClG7LP0y/Rb7gzWF3d2Nhw5yGy3NJ/N08ZdB87xWofojU5OLq+Oj46HB8ORNYbxqWVmVnmNdDQHZp0zdNkbLheFPHj2/ODEvsvR1fOnh7uP7TJYe/3W+7/3jT/6zvvf//53vs/iT/w73hPYg1vW2trWZgY6co/tmZkXVscc3tFZWbw8OVbf+lv3Oge7+w8/fvr4iXMrjVZ3cXF9YC2XzOQ01+yzFmEQaQV+w262aQyhAHKDBulF9yG0yeKhc3ZqtZjosrgyf3Z5HNtOROgiFA007iVUTxI9F/rNXv66B/AE1VfeJjFo7HrlzeRnPpm+mn3eHm6+knv2tspiEwodhnwRuUu8lMFgJ2eWtYOCr9QKVarWWAhkdvk5e56+rSy/yy21VL+ruhcQqJ8paPbwG55bhaFByR6IaGLuUQwqKcl5i+M2JTFoxA2FwCbwHtaLE5NyHb7MBSLMuBmcoyjXTGzcN0XnXwgRSBkbf26OfgNRXv4OV5WS/CFcrlbIC1BU4iQ99C6zMrpIBGo0D/EzN1smPdSgymImozxokPA2Na6hyBMUo4uwHRbTlR8OoVAKNhJCC5CalYKT805kkbZn1BY9mhMnz2OOiBSlztzxOZmVAiJKa6RQh7Tzt8yeDtSgWqLGNjeU758utf5UelL8dP8trgkcfOgfs4wmMYNnfNMZip5kKZZgqZLGS+pEYpU++XhaTWoMu9XByeBlDjRczuqrrbc0ntTkw8m3sKq2H5vJqmBtjtOLKYLcGwMsoPixzJhxijeBivkaoep2tbSNbDCQWTMla2jz6c1oknc4CkuOlE/FItqLEdEdX/ec8+fwuDMePuPTIyEIMI00jA+URbwcUbdSns7AwHJHOIjZF2kTfLfAZaSqzcV2oL7lUFoatq3XRdS49QrB0bfvZXf3qdCrWqi/mRoqEodqIQuEdgORT8PwbHTB/9Fr0Q2jloG1jACANeHg2cu30OsaEDBkH6Q62wSUpY1svGE8cQrdpbNtHKapOAuFW1sbWlyIEFhWmwtCUrN4nA/BVFMiaOZ/go+QKHEuPgs7x7nLO0BQxYRiQm4zCD7V9Bxt3Fu2xtkGK8QO7CbTJ9QNH6ZytXRaoBRTCj3QbDRQx+LuQchi8SDRLi+XimaUGUxtS4tFHQFIvDxL9ueXFlntplhfWzWPuLCJ8KWRhvZYZ095HV+czAmLOX+cqE9IzgkBgXLMbdkU1gZdBUDBQrBgTSNImLXkMnZbTUbH1FJXLZFcO1w5xqpM5Oy+tklu8eLk/Gj/aG3jlrV2FncSd6oJr008kqODYU5fyslG1yQsaUBE2MObdzZ27PjPbu2rNcONKC4vbHz44785P+p/75v/5A++9yd/8J3/2fbGg8NnIuFDMwicLcVgX0LPVdZno6lTZ0HUUYIJ4NFZWV0Y9Nb785//2b+65iuV3SUnPR/HR+AMNc2adESMzIV41zmIDK9dykmF893YLei/nD7wGrAgblkMzrHIF+Hq2Dz84Ggd/2joES4bgSoAc0X7hJQAAB08T+gMR9PJDH1BjmSOKmt4kwtyFGUoBJ4s67WP4aiPy8ClvJRcOT3lo7wkNadqP/0tRppprlUp02MegsxQGEOKMAXDmB/9V+nNlqePmpOZlQx1tYfpPfhd+QupU74EWBrcVhwDS6CR9MlsmvCUVtavv8+q0yO2O/eQwdwnzZh9OsspJe8asdXaNDi9S5rnakBIZKih9uWYc6VhunAwyEMDrk1HcXtmG0EUrqz+ZvtvFIYJVUjgdWqKOelbpWuVajKyAa8LAMKNA+v6HUB89dU6Ul/5vMAb/CgK81VfNABOejTLkGkbDTjTNVMSFcE4stZyOrIrSfAufgorK1u1yfD67bcfrA9WbUSs5iWSTvUCCQGWrOYqF6K1xoftWmDEUaIR6q2pvXF1iQ0PeJzMLY2W5o+v54/nzofm79XZ04vLXaTo+vJZv3t7fmELa7y6HHTiIL2M9LRVsTDFGGMBR5kRb9VRSAMQBYsaraB9EDe5Cr1af6t5efRgAOB/gk/VLptVO/v5lDE8znWG85fWEc17/8gKLLRT9K3x1xiUC2FronEBPSJL6jPZs5quJWiDQU+SFqLtGfc2NqGGaTMgx76bSLyckqigRbIZyjhmoQ1Zn1YQ87gp6pU2KC91a3z1K00HgtCMTFEeSrFHG0Em0zQnvDMg6jiGSLiT3vo1cFoc63btzrZOe9S5XFtfc5AkNaJDRWAzvbrq95aePbV6chyXKmOwKkuPbkHL5OtDjOb1ymSHEqsHBjBK096cI47601ZPTsdWGntrK/cWb3344YcaRmDLObg22IjIa90f7cD4J/wWxaTdxQADIKVWR+/UTRBgWbX72GE/3MTYgM8uFvqbd6+XVp2Fs7I2WLLjiHPj/hMmdZ7Z4HV6erS+0RsMBqfCP5F3SzcMjylOifsCNyFA9BjgE6AEIFXGE4iBJlZncop/ObLOrOaLE1cviK1lxsZc8BOgQigBOEObERFkhxkHl9FTDI/SSUXU+ETErL3fnG+j0ceOPWd9lH3M+HEK5zN3em3zLxurU9nxqgpzgVULLsUAcHa5tbapT48+fXh7Z2N5XaTrk62dLaDnzby9sca53XotXa/X74zPhvYqM8/zRKaDYkNmY0Jkr64xIxw7e966BjnjwpIBgzaxEEhsvIcTfNyc1oVPmftWjudPj62rdhLS43zp6miuvzBYWRJmx54Fh0lkJcrStXKWV1YvOmMzWo1Pd3dBkhxmg8+z/aP1xaXR4fnG/bv9wS1+ddyvHj8629s7/Y//xf/5wWsf3L/7zsL8KrX44OEhxZrZAxNHXME2YW2dXJLyh0BqxRpK8piydk1i6ex9NhTA6uD5yfC59Q0dxEszkKHUIZKWi6EN5DELROebF2kmlILsFEQm25tLdmBYaQljZgOYuwa36xyJ46SQ47UucbQPoFZxYr7Ip/ljEpHJ8bToFL5B3Iy6V8X7Ee7KWjirAi9xXlNUVxKwJs0jhGZ+wpiawLJoj9mfBoUl+cAcj6GxrjD1YG75/MO42LJkMCFSrbleXlo56FEJKd/3IdzNVGNflq1ijYep1ZWmpUw5J7/rT/DXBfAhWxolj6zVED81J91ttKYRnKCtrAFdkUEFhlmnBpDWo/yoeVHpeZ23lS2dLujUPckF1xSfDA1E6SzlDAualEI4Ax+AUDLcU2B2sEHciFdkVICODE66jycCnwTH6mbH0QWme3JMJsyaRD4nIyZPxaQk1lmkMdvNQyJ8BaGMjhRvm9lVANT81j+gePlKs13pVWBSuSpPPnG11y3L7DkZJh/O0jyQA7NGFvKt/7mlkBAobU7NhRfcF+YFV7ArYFBAV0WTDNr9RrkRJW5ecoreR8APwl52nKruBLmeFSK2tMW5YadziMHPz++D6XzHQW+iR1zYPupU07nOpkax/BXBTksUrZ03Sw8LfAmxXn77Utb8UEpJnQFR+HfQhhOqaWbJiVoEtwwEInwabIfzIIGUB6yp3LTw6CP/GjRtjfa2hMTK4YeSMzIN/SCQDCqK/TgZU2P7FmDNVIiRL0pnrWZkF0g6mYHwL4hbqqEgWanU0TxovbIm0y+lqTnFah1SIY+vEQ0fm9iXHJzsGJkn3ZulqAJyZhPZ1a3uWhetjT2Xd4Y9kSk6FthvvP8BzvPoC7szaZXChgh0xJhzbJEShQQ9LEY+zswOIBL6/tneM45BW5tc8/obG+tr581pSAQM580R7sJhy1OHnnbuJEMWxTTznO67RHlFbHQha5wVvEl1gAXkEVJQM6wjGq0HhHTFeoFz1J0GoLH2j12MUVumlMRiFGoJjwRh5ZigoNforAfiCCCX2RGfizMxQEkJ96VckZ/jOmn1NzblrARzb3LXd10QvIEjMI9eMSKF5LCoQqAMNQ2VjIEhbMyfeXyXdVWUDRExGYPj2mfsalyItrLGUSlokTVgBgXfUX4p3Oo6ObtaPDnr8ycBYkSZiJYFSRQG3eHbOx7uH4D/6PBIsnm4vbHhBCSqLcJyyE9Yi7Nk7qRoY4x/RyQrQpkCXCDpJgUUjUhNaZ3IogCTcSOnMeOrOmIpnNcES/cclM7uvb1xJFIondv2yTEFWD2c9i72nh9ube3cuT+wcPF01yaja4rr3ILVh03nYx3sz49G9tQNBmvvfPfb32ckfv3B+3Pzq5ejGEuWruxpy36t4ehA1BGgtqsKhcR1IJlgIVad+U9zuLo42j97Jowg9/ksOLtHJIpkrJn6AcmLKriHgAJzSDY81SkoAP4ZJrkyedn36keR10w9Vdb6Ogw0F008+cy3fB/OGqKclAyJwqdXmHOqMj9NNs2YvshfSSEd7sGRECfZNNSTKznac6h2vcwLaBjiULUUiajmanwKzB+160t1J13LZdBTWLENWDZpuuw3m5r6fvMFMwPQEFLXhHq3kl/ci755nRSUJzl/zTXp4823AdaExab/cqRb1fv6VT8mt8Ai6BgiGejkO9TNWEDgmFSKH5u7ZOW4iOAaOeI34a5YOvBdlCrhrgR5PmXrq7i1CR8VRhsok61qBwgtQiFJ9Kr+GSbADN10eW7j3Vp6s4Vfeq4PbqTCwBu/XnksBHg5zZJNsQZVRmeIjNUwJMQi+ZEDZNkCyspgsGVn3rRtLxfj12TwWvqsWbqApBozdNQVeQ1XVWU24EbAtQXJVGT/oeuMxf5jdRxfDalEsHjqnm3yKMG2P/cUqJxX8L7V+lvekfjMizRHOZmoEnj/VKxmnkFCLzlLnLIGl1E2f8MMkh2a5z9dSH8hRwM2MpqGFQSCOPKGa9BzA4ek5NPK400eAtVQRv87tSJqfTxxwlgTjZK+ghJjJ5xYKzGDGi1BaVZ0q4oUgjenNp2pWZnPZaIcYB2xWpFpxFzeh5wx6EVL9+9CWLKFrQ2HyjkGOMebi1kPFVVPBXBQ+u27ItTfX791fDy2kxI/s1Q55LrV6QiAZKHhzPbipYXeYI2VcHiBTdtDGmUdhxAa2qk5popG2UUDTGzLDnIVOFL7ad6r9nuurh7bfhRGIQDSSiRTjhH0qfHYjvQuBoxkRqAnMQSCLB8XziyJS6t9fMeOXQoC2NyMeyzTeyDD4jGeWoLVwcGRBsevvOBcQACzgDlW4HpQMugH8DUweD/0njEq5aBmQlZY8qSCczCO9o7xF2BjFC82GitYFDFksC1iXo2DuVygAg6SgPUVpAQWFwfE9uPyHumo2oDp4hja4PIVX02jibQZOKDTQr2gsqdPhMHTUwOxtbUFREQJ0bfXbOIZDay4Zinncp8KHYMErVZUGVfFYKdAStHF1BF34cu4PNgYZdtbsJKVRXXU+goxfXqxTA2GAz2+2VGULUuXqER01h5ByhIs/eDoUABoEuDh4dHJ6eXuc4eInzx9us9zjbPYwlyXQHDw/Hpuja/8eq97t9e/tdJHNO50u5uX58tsFtF3L+btQ4tcEzXSfpFjIGGh521gSMlbWUSYu3JqkuEeHu3ZrYZP6kjsjtWfxvfS0yJ2RRwyHYy5XElmqck81XZJCIlR5eTSxjByUMi5Eyyv46ZgiTDCCPbfIYJkWGqeZaRcpeMGc8KXvcgUrFHMyxKwbvystFdvmuyapcKcKiEpRSlqAoc+pP1VpFqqF/Wj+tUSKjVtyEXmSpnQ8uUrtOof87rZl1k96WEo3qSbWFoBKu9b4uQuT+t0dXX2eXuoPLn5V/s267ESouIHja3BGLgssURmjtZPESZx2xXAVpt/Nh2d2B0pIgwPvDJY2SsQvDFZo8JFreT+yr8kfH02ErPmvsw+XwzbK239h/uJomJ1RRsiY/HIhsQhWI14oQK8DknDTstZXdtgjAKQwpubTWjtvNnaG8821aSf0Mv/7jUnwl1ihzUv8Ap6Dlp9cnp0ecny6bQxJP3QIhqC4IDzBIa4TszCOr0gk6G+dccUXJkTv+sFi/UC6mJ4UIeAZXXMjhsHFi3k4ECGykTL4iiD7puT6XQEwNSVrzKJJ3XWqzx7aFf6qMBKyVf1eUmp+Vyedgfk4NLV6XyOa4xuRQIK661tWUzTOHFIDDKOlJR12YcAVp8bJiQpky/N1KjMX+VjELR5qY3911jRo4oPwE5hB1ZjlYtPs12UOMip+MB2DRmCxuMv57fXd7bXttjlnj1+YmnNImN/YfnQqTL0SItxS8vi4kMTdmzm0o2tTfZnBBpVo8cIAoETQ3SuOZiRaE5AbM5wsmIdFULLuYFZBBWK2Uqj4witml4weGS5rgCrcxEEDT34MC+dntFNR0t9q4qxkffOt1jLL85HHHdPHSBxMbI4GQZ5dTU+Onn06DHcUZ8JrDSstmBVA52Cc0kvZho+F70HfCMVGq/YHnQQpwYt7J3HTiy6CWoFwBmumL4W+evFClE822ZUUTgsiToaKKvadh/FeQuZINBGLGK3qNXsFI+1pI/hkFnAhgr5gRLUjgArPrx8swxB/4yXsxVL4mlZMiPjsB+wxyb+s309cVtbzZlaHJ7pvfA37J/YGkhmtTtXY8CFkxFoPFQyeQhBKzGO0zJDXDRaA8c+a8svfZ7ev+T8jP6a1Z+srQLB1sbG4f4RM5/GE0xOxuHFImCheUUb+pZcxdk4H17cff3rt7bvv/baO5tb9xaX+tzrr86Wjs+QxmPbcZmdr3hnW/9wdDgZfHHBCYLZH2VLNzBT/EfHFp4tcttWZPgTJx9Cwwcc1VRFmgjEwfNMpRrJwLN+ZgIkLfy56Exxp9CbTBnf4MfRTU1HIgh4yah3JevENJKPXBGuQSpDE8pQaW7tZxAhP4JdpRfO3k/zTf8mf2ZmZa9Z355vEM/Zt6l42os22eX1Zd2KzmSuZ5Qhqg5EfM/7+ixPLWtLmTbgt/wb9Hg56ys/X36ZX/lkSsTyu+ptKe1tg9o0pSDQ8hTGz76dZkgZ7aoU3THRMk6ho3mKL54R85zL/l2bBbLRL2IlZ6t4PjNlCDt6heY4N0sImxi2WFSyqIQURt/1ITmdxSpCrZ9NqZp0VndS7bQdResbp3mR9I/zFNG+KEEaYH4GmXW8wsNmmIPE8VZ14sJqfyO0IyRr1pawmZevV1LgFqbigyBZ0xgLh002RKygrFYz/5rAjpbGDYR/JaMCYQcNMt2gXFhGxyEKkaDyL630SwnYm8Q2dV5uyK/91TK7B96+rkJNTo0hDNh0szJ32SUkKTy2j3kEzlX9qmHyUQ1QoFAYU9MdsCYzuCXKCmNeGuXWosKvKjJIrI9nnZyM1FbQlYlG88zKTotyQ9PTtBEqlrVN3TEXa5d5WD3ICNVDjQtQRJoPjmrbCusrJ0GcDOugbUSQ0JW4/K7310LHLQMK5H8kRtsI77HzpnuFYlKlrk6Ox53Tq67QGnPzzDsJb3ZyZodHfzNmZjGURZvcvnvbiQyltTA/WqYVZ4YoYdU88inV2aIn2QuDcigBfhYbLpcDQo7ZNX99ir3Spwk7veXVxb7JpSuEEFyEGo/lcKswoWjZHW7bZiSle+QEXPrR0eHhASQhQVDVxCc+2Tt+8ujpw8+eKMSctTKbqZqlMwAJ46meh25h7UAMhjgWWPhhgmYROmsRIW/oHNFAIQE4iLFvZIWGigpgS/Ra1uOQBIOXe3nNCf/EIm8saVW4bkdkGQbFxHjLkpYyYiTBRiJbqEAzDGsGzv/ZlidCDzMrWzgzNGErY5fmy1j2J/Zt6u/KyibOvfd8P18tLlh2xfvJQ9WISKpy4/KK1pRZf9UIrtrm4u0UJVugulpTo3PrMw2BOqE9lGLGdssEPiENuIAI9+32uwvLPUYNjw51tn3u86v9jcG9s/E+6UNDzsbxUdjZfu3Nr7/7nbf/oLcsTi3mzQGKKRDqxZzjJC3biBR9Yt8Sgnk+tK5A8lldFyKNiiIeh71Mw0PRpQ/2bHMi9gCPL0OXjAT8zU4rK9Z+BXrurbOA6OfkHxBE9sqg59Mwq6zBp6f+SUNOAltiDo+2q/GQyYT/tulEZDIhM899lrFP7SljQpxr1DINk5ip7CEU5NdcNcSTd5V98lz0tjXYrPWgqfn5alES0460NtP9RhFS/ITGvtXOuvxJxyd1/E/yJ2BsFaU9k3aEZ9ZjNTuv4ZJ70qedLPuh8akrqfnYj9JTgSOlpt/B/0w/MFemDFAVMjE4FyIgchXuitWJP14tAGPGxZgTfbdCbWQBuJE+hcfDJXbeouzF3tV0A2Rp5/S6+Xwjy/T1P+DfLJsprjoJcUOqirLEehZZHipn16xYvmtOI2MCCOl/gSzTp5JWqlnTlEkbgyuZCwFgygrrCFZniyltr5yckTfWr+wDBayz86NMDxGS2RniFi1Xd66zbpYGSX00GesGI0mT378LUPJtWpG2edQu2Kwms5sR9PxqqVFATURzJ5KXDsha9/YATvXtJLF+xaqR9KBTgSkkuhBVp8B2hqmVO6Wh2JRJ9MVDUQCN0rxYoWNrTcOUWTjavql7qWozvC+KYBi1kDcRao6Uo3w0DaonWRD95dSK7CSyQw59ESro6em4Zxm26wiKhOo+Puerf7FMN3CWFTnyZMw2aKPMep9V2bqbY1O7o1oppSf1N5wms8AtuD9YGx8dDod0I1x4gM7pEarN0mh7SszXdfYfak+ZCx1fW8VsMtqAos+h+zlfJVe3S4OWAkQ2uqaXOKlNrk74KTtoVL+r8efj4fyj7FLbubXNb8ZaK6UVS3z4xRc/+/HPnz56ytgYD8lqhuGNRigLz5GSSMKeQQmFrktdriz4hUxE1DMHrAaAlQMnwolCZAGS0hjB00cQMAhhAqdaECKDx82BKpwelcky3JfYoVqcNx0NM9UYqGBZAFm/mndGRvCECMTKU8pqiyx2NlgBbTMOQiiX8odeQICr1dWQHSB1PIbPBhvrdtQAW4QVOu3E6Tp8Vz4VSnfXm+KjGRefN4ZKGonoEEKU9DQD962WKtOmHboo3zp7yaDf2tZtOxWGz4+P9kYrC4PuytbJUJ8GVu1HR/zP5zdu3b9z+7XB5tZbr7/11tvfvNwXQ2CF7hqvGAREGCuKbo4czmKDf8zxffbulV6YDpPM+bhzMiRP4b2EDMWCnX0W8QUwUKBHz6lGxjjBDBYBI1N3dtXPzKzWl/QmGWquhbf5wD/SSS55FImEaY2GHltXPj8RKqj8ASJzGTrzKmQLWmQGtsnnUWIoZLY31XxszQj4vupqjZk2KTnqWXum9CosdsJ9i+FUehHHiaabcS9VvBqvBO1Pd+qyYqTAYFUBp+7yF3S+qj2/Ji0M/pUuTH5Om/nKhzezt2fNgd6T9NaJaZESWzHy5Dntd6WGSf763W5SQg5LHIEtBt58ss0h2IzpoitRJPAEmMXORXbHaC9s84Vop2HD0Iv5i7Qeh8r6Z44l3kKYrpnHwyjKh1esmtMmZjzSLMg2aeu0Sa+AZZr8D/zXvo5M2mpPABQYGUSN5R4i9DMkQUfmHcctFPuaPUOoUuPZLxoSRG3X7GHWeCmVSJ4MCIq+RbwJ6ckgiEKVfiObrGrqdkZFr3N1VIZwtn3r6TSPtZU4RW/VwJXoEjQGsb/f5eOaBPV9YJ+tpRrHS1azcho4udugkUQKZ6KegkambbWgJnpoQ8oJxkRN9hww5t4Yc3uYyG4FhBgxGw+ewLkohfxx/UFlMI6s2voE0MvSEO3NPxBCK6u5+aNiSVVXa0/aoRt1Nz0X6DCX8W5ChTHV4bBja7UFS/8vLw8EolpZ7Ow9f/LwC351VmETzL/TPRETf/9QiAUhBR3xtnH71mh/7+gRH+NzB/aGiFOQ2IJHw/mjle1b67fu3OlurtNon11fPX/yiIWRH1Z/1ZahdJPyqjn2jiLi4RmnCQZnRzisQr6wcNnk0XVynv7wqg67NTUCQBjJE9lcyWTJEQILi/a9Fsu4fLbHRfvcGcNbAwugGwnvdXS+9+jRj3/4tz/9mx+ziYrxC7zgXPAosHgKbDJuKs0QFIP3gEtRhKtOVTcyHRAj8zHlxkKgHO3CgXGdLNfPLS3jsjg05oLvWmsCMuMnNpVawnEye0IDyK9sphk/kA/Jj9NIdgDVPJO59ZdwwOCOayIoBJfBij14KQGRQHEgAlakTYaAIdqBetahF1eYccPbadngCYDt36SbJeexzOkvMq2Puh5hiI16UcQNyytzS9b9ZTDB4ogS2UIZpgGOriIQdMokKzSFvnMyd3xwfngwPB0tXJ73P/v48GiPV1736aPj1e5r77315je+/nuvv/aOMDu2Jx0/veZbT7qqIaBM81gINtM+TkcjsOS0Fvki3BWMzq5OhgdPn1iFJVVkrbeCeGgAFy0qfyZU8co2oEbExbijm+lpkD6A9dZdp9oMeTFPWjavCyVUmH+Fge7GjXADny0ZLvdYikifmk3f0jacGOiVWcw24opJ6etJ2am9VR7WU9V86TZFZp0IjfA+3UkDDKcGpzRIkVvd29v2M6npV2QBI6j9qi8zfO5wyKsUFZSumT+hwJEW0p4iR3m+eVXtNxPyrBW/4wUzNS7tm17pSl3twSt5XNM8lbO6BKSz9AaTgNFEyABWyyFFUVTQi05SHDcTP4sT7XjB2msUv1LLvaQ5QQZGMZMxIZFO8WbrVBb27HEo404mbThuDM4RhYPn0ZDcM8WAMC2KVUVrC4EC2NnVOjL7+Y/xEEdZwIFskWI0Zzp43D3oUvFBNthUkiXeNGQuSzIvX1/BfV/KALZUBaVkPFI4QdYV3dLPTJzso1GxJqA1TJ6rtgtf0CiuEDb2fZ4jh8sLx5xFArfZaLdCAsaXQPZS3b/hRzUi79sDdTNaTf2Iee/S0hUqS/2g/dt8ESEPMch7bSiR5eV6C9tqNGUxgRuC5iFsNZ9lrkwz+D29ZPAqSBZ9JTBKrpYTuDLTCm4hDV4oMIWX1JIiMgkDu7p8VURK1DH1Jen68uDoyMFt8epFzMXI37kvpsFiZ2ltfrlvF8hSF6OYX1k/vVo8HF8eji8WNjfufeODwYM3Op9+8sP/7l89ffjICXlIAeXJQsvQolnnwrGr9xfeEB6Yon1rsPGR49iPhnTe69V+HJUjwF3vbN6yk4rr0P7+/tHe0eHBgcVQLrjd7Ts5eME0wAur/WbWsVAOGEVdeEpmSxTzwIR7kSEm5HZXV3IWAJtS52J70Pv0lz977c497sHnx6fPHz751Ye/ePzwiXgWljbABL2a3oKAIIF0gVvTsD0h5TK4B/gajOAGzhIDb5/QxjQnsa1rYRXI4WuWQFcsT1DdaFARyl1YrwjL3KfVQopQFSQNuhBEwoBZjYqCKx6rg/NpXxvJmvaZ/5oR9TSbfEBJ27OETDNWn1aYJGJ8DuhqODQemZgYGMg47mZqUpH/gxURLcA/M+LkeOTOwK6PXqpSNlXgipkyWdvByq/mz/Aii2VBQJjYrLVeo/sCUaMAT744HI3pGnFPFrPxwx9/sv/8dGfz/ve/+ycP7r9ze+d1p/OeHJ+PmJpNoqt5eyUscKurEQadruUAQTOv7GWm21q2vx7uHz7bOzzaPz12SPAZDqPNfXKhNQZtPT0eZy4U24n0kYELmmtg3BV1q4hSxmkyUWrM8lxDLb0g3P4kV65KyhyKtTn4xRGflMKsc5qlByGCI4lH5Yg8nmwZRGXWiCrA37AK9WdS5ue0aL++dGVQgke5jE17dgfb5E2KSlNY0bR6CEmpxBCChhvB21DgeMkY2QiF6LBPFJtyWmlpZyvW55X8j39rDUgPMjozgjSpeNq8QKAlefB082drc7gyLKls7lFF/A4yQlFMmP4Tz8dMtozRBa0X3y1rc9gt7huDMyWY8ZT5D7rhvrDZmAKt/eV+GG/lZZElzFwzMjapFvjChSGXPO2KzPU/4UU6zsiFRJS1Y66OqvVIi9ncXD0SQvD09LWd+D+DDG3YLE77Z3x30tzM/2p2BsNDuxfAozZALZQszMVIlY4I44PnMRAG/HkWJGBhdeHa6tHaIs8bXmxzl+xhiAhPl6UFRmmntXSDwhC1GpCBUlqwoF2T1mhMqnM1vJy8rYQbzy89xtsFWofGzs0PBB5gl0zk7vNjq8KR5C8Wbb6EH1qrG4SzfF7yikmaSVLNiLSRny9QrdUCo4Jpk6ZqoAvNQlvRxDS7IJbEovO4ZUJq+6Bmr/TYx/wGJ4YY3EkS2poJGfJRHQ950QyzOLbg/OvMMSHCXsdROdcGaT84uRJ6dGn7QW9lEMeW+NzML22/du/26yubH3/22SfPDh+PN9cGr9/r3Nn+9tL8T//8Lw7oyuc8kS9sK7pYvH68u/dX/+bP948Ov/X977/21ut7e4f9+aUhffnxU9GCkS+8yULm9p3756Mj4unogDv1cNPiYZ+9mmWa6c9h8peUnmPWVM7Vde0/fYZ5NAZja4pze1DH7e0diMlpdnWxJ7q6SpYjs3UefvQLXr7DvV07Vpw697c//NGzp0+zi4ObWA5WwkzCbPAP8KFOKZmnpGcQc7URAXDpkbgrGwXL/HaKMujRwe7du2d/88qypto6OLe1vYE5P376tEMmFW9iXkCMk93dvbOnz85DxBMrMuFEmDUrEgtLLuuyiFT9/kp2JmVDea4aO4wuVbMQoDE27BJ2bdcyAcg3t9YT/Ti7BDK0Gf8wo4W5jz76ROhQzic6SJgSDHl9sPn54eeD/iqznEVydt5ivvEncrEhU/cjx8DUoF7s8Py57avSWnqgOJGhO7pH0D3NucU9a7TMFksZD+vyPAF2tl4/PJw7PhbQ6mR09Pzo4Gx1+cH7v//uO298rdfdtO1pfJTTknrz62fCeDBpoBBE1vyxbMDqounXK6JU2a2MLY+H40dPDg+4Nx8aG0KAXfBR8YOrRSY1ErHAdKnJYUHT6auUXNnhnqOmg9gymGgkB3dwin2qUvIsJWwtiq9nbfCW22wc9WtbI1p+3u05Cn338PiQawJjXFZtIIQdkKwCmVG2QdE1TRASZQYkz/6EXKtISrWoMMl37Wp4pd/e1ehlUnsOcZxcfkfxaDxcebN/CEganH2hBC6Dzwyn/XHKy6wvNqE0VUjxq9K1zNu8n1TdeMr0x82/s29bYmuw7wK4+roKrF5VDssbhUe5hcKWkKRThsBFm5TLT58mgxGvtkmdlTOpyD5qdegFWAajQ7JTaAkoECXFSYl0mmzAoHj0DUJkSEmv+UQVcbwS2ap2HBGa6LsJwcEEjVKQ9aP7onQsWjG4QIusFtvcFt+MMBuXlTjBh6MOFqIZhrJyBsITlpx9BxmSQGSGffn0S5emfiktCZMR+cp3X5Vo+ShjELIfIICCW2FvdGJTIHa67MRHdALir6w1g/HKFVAGa5ThjQwBfeaLuQHNc7WipBQSZs7IxluChrE2vyCaQURSNuqELrg+XVo4IdYLc5svJWCOQiiTWBM8OXUkfdq6hg0t8avvX+5KZoUGtNJwuy7MueRqejXMUAYsYJT1u3yaaf/K1XpajfM+CKe09m+SsxLznAeNrlYHGgF40JQBBtHKMcmE9DDyQEz2/K3/Ap8Att60P5CzpocyzVgDGFGgvmmG8c5SLyObcB+nzDLn/FxzWsb8qfD8KA8JyKyww+tWd6u3sfnmG796+vHy1moGHzO5c2vnvbfRuuPnuysL10eH+xikoEKWlr/45ccg4Sye+3dvbffWLladB3yx/3D3fHDaXVtjvzzb3ec5ZLC7VO3F5cvR6fNHT7JLh5aZtcuYOoV4yJy8uDo+PBLIAzNAtRG+kEnMmHP2eNRd6PKJHR2OHDzLXTdWV8zs+mpnMLCH6rPHn//yZz9/+OlnJ0cOwMEWSXcAkpncrkz0UJmcWB6w1AU9GoaoUQJOyQlbmJDD42Mtf+Otd7Zv7dy6davY6pleHx7u0/w5PW3t7Cx3e1RCjX/33XdJpL/86ONf/urjw+Hxs2fPKM1GcyDC9uKiE/Euuxdbg1XsmSxQVWeUEWAAR4vsGA4Fi/or0cQIikmiBMOaIr7kBVlasymKfIUWsF5RRbWZ8ULIDyVTmaEDeGotyTHujKXou1fXgytBDfvxs/56pSMJ6J2Fh/mTkePa2CFEgNykKq/1twb9ndWeYF+iPm+t9e9srj148vnT/srm+k7/fHB1se20hsFg9dagf4uDlYURzs/iHaB4hqW7uLbSX2FTBv/oa1iJIdEtK3HnF/uf/uLSATUn9i8deUC0bdEySfpzPDxAZcJENLsRBi0uRJ6MVNA/kohl8axuuJJzevcA5r4IFHOFHwBqCgbZyNKqQtPN8eS5EjwLBU8zbFamBpgTXoXQU8ZTTopobDxzWL2BYAamwbLNykpL+q+9WiN/zWsVpa6613NjzBmsMOlsQZQiQ+7tgjnmuNa0D6fJv8Vf4HilMfk5oSETYN4sBqq0/AFkIJDOStS2WTkeUqw3GYq/+5rkr4GTO5S0rpslQBeaLvNPTNDFR2O4yXIF0lEPsTPTgyvUs8Xg8NdoxpAxC8a14guxaq9RlJ7YosPM08SyQgc5WocR9EkD8jBtTSUFJ6Ukp98NqVref+C7YDNswkHTIF+qbNa5hCNoY8CBgvuiGQ7WYXwaEK7QrtmDn9POGJXIjK1PelXCR34qHzUxRfIQ+XbSscmHhpYA2FnoM3TPL9q3iphaB+JDa0PF6YrTv/Fg/+LAhcrGLJxJc+NqEtaNhN/4+NKnLad2B7NxwTCOnLInCj8aauWfbxJ3YG0S5FbAEB+XhK4LtQ43qwkNvYmgk/QQmDxOX/m85lhk3RVLctCjoM0z8yzn7kR49w1wISuu0Oh6aHgChVJc7qEI7YfCpdRzxrFRNF5lEVkW483KtQeF5mjF2c2uK1JOjkQGy3jeqmqwMLfevb0OH87G5xdLjrGzGri56h83GQfOiNenaWQxguXw4OjRLz6+Ojo+feM+v1auD3YBHT87Wup3N7Z31rc2cSM7jHLWNAd6fbT0m43mS6fHoh87/Hxx2UaXjXWtsotVWH/6oIkilpYFQuHTBbTiZ4TcHu7t8QEeDYdCs1m+xBlALhG7BUd6/PRXP/n5Rz//xcGz/ZSfw/j4SYf4AgRkCEoX3IAdSy92C8xBWukeqJ40HPoQyhqtcHHhzr0H73/ta6uWP8+ownN0Jnrnzu1tMoAUJWj1nXu3WWsYnXkgf+9733nttdc+/uSzx48ff/zxx48f7zGwWqIWvRFoj4eHlMpSIdBT8z12C9OD5NMwQTO0U7MhntcG0/YtlnrDxvSdEQ+1DSnEX13pFGpywiwzWrTxZzhSC74bX2hcJny7GEl0z/TRVWVEgcvaGHf0cHEbXinHpTU4C9shQQsrt2/dZ9teIPte9hfnNueuNiiHoq705m9t9R+s9NZw3Ku1wENkTMe8WaThLEFOwnqzldpgJw6zhVUHSjtraDnnJVjI3d91SOCFswqP9uMEI6okt5g5J7TERpig59Gv4geQKdWwO3/z05yIOlgdyT1eUXFobjnlLRhO5kUzIrZpll63f8rIEoLhxXwlWtEyYxFrXPyMZcYivkbEH9veORBStqYEjIapzTJNjB9mo2mzFhat9Ctv09zf4pq0NhS0/fONTtbn6ZKBmxYl8YX0UT2pcfyqSqaf5F0Qu64JTKY/vyJxQiteIkpQaUaKXzxPcGhGuxphaR9mjPLkQ4CbVVdjN/v14kHW5KwLANuVriulqgbzcN8w1FiOw4D9ZIztiGdK8CQxCZqI+8bt+SSOptlxxCBoIid4TzTgRJpE1sKYW1Oj+BJtw4aNOmzLRAtC5FJFDQemkgZVt/JUCJS1iBsgqQ/+oW8Mbwgjdw4AaTAx7014kXkYYwIqJJKtzMwva1amzbSdswanUa2lkwmf4aj+lJXV2/BuotN07twwuefbAkHrPCYnhp+oOkxTy9f2BMe+wTBk89fY/pWYTLM/B2Ri6kI7lNvqagUBXHv49fe05Ne/BXIFZvsDm/T8wmC+c8z1naHYYAVEaIX1t7hsGR7DJ3cVGOIJMlG/2uCWllwis/Qa2EBzOsQNXTHacoqxqIMi+5hajyKLYZj5h1MGYVSZgNgxzqXZoQu5UkuqrkInAFdNK9jXMqAlvkb6hRbi/cKWb1FQYL+y7ynVv1jME4EoklHCAo3Q+Pi9X82JTc8l64jvMRLbX7kaL27sbJ+PuzYtsZwu9tZwk6effvH4V79iIsZ3FYu+cph69nCXMbbbXzs+PtKOnMwTNeTKyiW6vNDrO0EHe8YJBO5g5hVbCQ+WstLH6m7pz9FwePve3ffeew/i7e8+wzBo3kbB0DvwHhf088mnXzx59PjTjz62bZRExloMJgTidDKcM2srDaFDeBs/DjL5P/zMlV+XOUqojMCnFO/bd3feee+9za2tp+zaT3ZhAXV8Y3P9/v3XmEUfPnz45NFDWvnt27cEZ366uydSBO/fB/du2wm9s715a2dL9M2Hn39xejJa72/RyWxqJc8atFYdJ+Co3FlhZRu3nB2jXJoaPGreJ9fcz7Gw8kbGasJPGawVIZ/BpLxquPMeDYvF9ePh0GhJDD7AFSYEFZB5i1W3elUNecLjjfXlpSAbOCXaJd6z0yOtflwwaJwKC7mDbl1drPVY32+/tbm5TVLmobw9uN9dIAnh2yoQ2oAVPZFd+IgXdrlZccB0LW9b9722WkBA6JwdXGf14Wj/uTMIMeDTwSpBMKtOIQM1IJ7TbChvQDInJ+OVZkPMLAaZU5HagslwNtOWYSZTqJ4md5lMg4g5pPXkKEIDyKkA0oeUJCCKn00lTzC0M0RmfHZMKEHoc+pnYpoFXyjDk7lTXtB+GRwtUEOV7SE2lfqdan/dBextUAr+NUunczX98/2kVLK14Q2OZpD9S8MjfycxXYzwHZFgwp6ltNK8/fJ1Q1L/8stpSmuSXyCZ5yrpRWJGIxbh1n7PHrxtF2KSt9Peef7Nl5wytHvL6dnUa8/F//K2JdLN+A/klJdSZ2Vj8LCTMiqvvR2CgOO+sM+yiaONcuCgt9hU4rhHA06jIQPUiZE85sTS/xhbLFX4yXakYhmAO826cU06W6kFejCXp3DiRrZ/8EcrUyb2ZGU3eJbZYKqlIY1UoSMVZiEyuJSQk1w3ke9FX3DW1hMArYfIti63IFAsQYVpQaxJ+rSoVrBibdXHMRzX6US2UXwlMkGyDDx/McRI8OAyMmalaFJIjfEEgi3p77jPGj9pxjQ7AChT8UF9DSYIO/+JymszD6TwS+T5YoclSacv6ajPcU/TvR5TmNRW5vQhHGi6olPZfZvJYhUX5xALOn2JNkokDx0gsONamAlzcQ7yNDBTbZjVMr8LwqnFgytJHjNR2/SuPPpSjbLcm1OFSBRwMczcMLNzx88J5clKfLZC8sWFcvHioTYIHi16hjCMi/2ug3VWxj0Ry5CjNTLZAm3QxTP6mnGY7/H4gK91Fkdx+OujkSrWNynBIwv5wpmt5sBaZsrFmKaPbaQZHxweYqLEj1RNurm6EtXSxpezsTNmrw8ECz45yY5sp+vQseYcDp/N80Ihnh5ftU2xP/nbn46OhsdHQzNUYzXM4p029UX0DfEK1XDB2CLvEgKo/HzpgtWGO+QZG7t99872zg6R5+BwaCtWb7FrqXv/6IAE+vqDe4P1TU6zFyejJw8/pSi//foDvmOfP3zsW0o/V6b7d++88dqDH//Nj3754S9sZl1fH+xsrYtWEeULeBstzwxL22YtrGa2G2nSHuxzy8HaWGOaNms4Vb6136kIUojCGKrNtFR3vM2kyiHKIS4vSBu4KbQsrazaPmqrkvNsdsStFeGzl1c5pYyyAMyFZWH30Xhleb27vb3afWN99e31/o5SLxcvu53V0xE0YbNwTNmiz+Fnb6UfhcT/RfF64o12lwRIYUfonByMDnYP9p6PhkeW54h1TuYVlToBp0gI8V1o2nnogxkQPhTUDTz0VGJUTxMAAMKO6o7oh3KoWd8dklGk8eU7mhGgmrrtDtpV7LXwMaUBSzfJsuAcIwmHcMaGY/EnIbE1ehgKfLGaVwuqPaFampKJGGEutGJCcer1bLK3zF++19glefYwzRNZJe0sLK3EFD5Jqt/5VbVFRJvA55UMyp0KBpNPUsjfec0a40EH2zV7nD1I99yadDOxpes7SoFkyBDkKkkhr5L+0hUYVoaUVm8ooBC10lKIK4yz/mU5lDOBlBiLkkiRtRmmvK7ifnVK+LSAgftGCIR+PCFZNcJ9zZ2w3ijNjQGTXHEjlNSEQzzSzpgns7pWQkQaEyxpFwzzoD3uPgH/dHDy8h/xD+UBlY/vX5At9ZvqIQAgq91ZAeZOQv1Fq4u3ZpqnlTfhnGdwnI3W7CHpOgmXSoKt9IiyNdGqd7k1vCkSU2cP4ATO5eY0ND+/xlGxc31S2g9KyTJqD4izcgkNdg30rKfFaj1FpIbTSvyNVyqaVvpqxsL1NstMvSig5Trcd1prYQrc4RpmsocyGKQCQyswrDjcLky0pZQ2nBpmxBQCJFvxyOo+qpgYWGyO/HqKgvr8it8vic20X1GNYajtEJAdQuii/yZ4XqpsoFmDEiCHEwV7ZoQi3CbeJLZaKxPTj3dMATx+SfFrgMGYAxLKlKhf9vdmY83pmfhj9qLMb28zpAo5iFEf7h0J0WlxlCT5+NEjfrobtzcuVjfs4MSFBZIgn1zZuE2oW1oc7h46nsE6s0MQ11fWMAjyV3+5ezg8YQTfWOmdOqiVzKrJlLOVeV5aPlzJamy8gLqrfcz1cP9c8EpdiBWa15P9L8fWnbFvJ8AeYb1mGzlMz11INIWVmAjNioVFwy1OFmS+XM6+2ECqnIGNimd3Jl9hmEIQGVhWuuK2O89HFMaj+WN+Y/Ls7e8eH4+c9mPZdWfn9u6TT2MjIlmcnTB5b20MRNYUo3hzbU137mxvLn772xZsfvGLn3Px5dibWjIssTObTUzIkJtRSaTtmhduuTKwyGk2Lln6QiyCNEWntDHzEnUxLJ0NP6M3s0thvQz2zPUxD+B+EVUiXSlNoMd4T1sWpbISMbhop7ORgG217XUHa6tbSwtdW3uPnmv8nJO679/+YHvr/t27b22s3+pc9g9247K1Mt/n3XV1dmzCck1jCrGITy3J/iLBI8Wurl4Bt0l6ecjDzoG+z0/GR0QlrNKKPFx0lJb3MS6GBOcW/Y4UmZHDG6d0Iww0QCCUFQ+uETU/4LJJE8SeTAFl+RE4mEiF7u46D2KVKRMAFKo84Z9JK9lbWBDOchFM1jYW8+HoCNIYGHZOjUKsDSg4aVBKDQHMRFJoqy5VpsIX16S+Fwm/6amGIHaIKlLOTNjUk4FXbmYluITvBiNaJ1rNIaEt5UUFEVy+4no12zRL1T75MXtunZl16UV64B072eR60f9pcdO/Mnhs92naV/y9ma2VaRrK1zhxuG9dDBLhpnhq9FpiH5NRYmvE5znqLxPdaQUm5QWddd/sVo011UiF9WaaNCdYz2G9kfVi1pmR4NSYq9AJn3sBVeMwaWRoembcLKW++Me6hVhGT0B+dCGQYHqPm4NJHuOZZR4bFypSQDyLzbl0ofXi1TbpQKGN3k/04Mx53UyPjGEegm2FapOsL8oI7XDVvWao7T8mhbXeK5EHTriPYIeoEArpkJ4AmHyO2i85v2iGIKnAdSMlP790adDNLgSVY4AK8k7QOmOZ6SdFjFyKh/WhOIuKE0mviD0w4tSLK6vorfPTtAxnNaaN6zQ5f2cpqbGF36KaKiJuZWilfiqt/gFWFF7aDDkJ8F1peVpaMIV7ZXtL7UnLtNS7YE+Edy2IPcOLuJTllUHVM2XUqXNsneyZMHwcV17al12/y1d21jm1l5kOP3VgE86z0hMGSWzCjeWVB7fvMvnIyfab83HpyEJuLXWZF4en5/vDY67ES2IqWq/t9clt+JmV/GNC6+hY8Kxub8BxThd0BpGM+pnmYfzMy3OYLoGPgZdb1uHevi3Lz3a5vlskiUpkxXZ4ePzoiydPHz3e3r6lBxE40mXeALEZwFkcNECaXkANQH41F6RASHGmuSrrOh+frfSXI3HkNAjm6HPtJa5sbt9aXR8QK1ZOT3jemsy2G2Fpwl7e2t4kzjz8/GMy4s6tO5yzHj5+Ouh1D53CdHnhbfdb33So1Me/+mj/8IAVXV01NmlTht7Y6DnWUmKulCRWnqwX4A+FSDDcv3KeD7rKj+WzOmg8727xyfRLorVyG6LA0EIAMRozhkDaie8aOIxYxx0aKHPEkpJWbTuy0ExaEbh7Z3uw1tu+s/X6P//jf9Hvbq10t5nzLdOfnnAcFjPLwpNtQisgSBNRSniFk5HOT9ccE2nmWRzKyZb742Oeas8c0BsTriPP5he6pNZ5xwudXeb44JHm+TSdihErDBhPV1roZ/DVlVubf96ERGZA/V8vtT3ZCmplA5IaXK+75mlK4FqgVHxSoEb+wbA4x4ch+59rCU3jsiN+jCUS3D27LESBzpFgaVVkn5ACZaer7tMrWNSKnab8pr8a0j5vgzu7p8Tw3Zslp5yX63pRsvTCl1fzT3O09JdwPnP7Ky55bv6rHBICqlythbOH1h5oIz2STV2GI79BtehNffcVNyCeNYjZrV0ppxrrIc+pcHJlZrcFTkTHTI2lxJ4bKBySlLM4LlDeS6GCPNsMiRnjvpDa93DJeOqDeqL+eojlOaOuCsSgdOLShpMn8l2Arzv5tEbZjAwLbA0NTsGAWaODWP+YlzVgJxTFSzAshMRg16KeZXY4FEckKuzHymv5jIhBx14ZcqZ92jW7v2ggiOjMi9/tKZPgK5JfyYaGIqLDiaQzAAEAAElEQVRVciafJ7NPOAc+Ip0OF1BrSDmtwWxKNu8nY4jyNxyq1aGYV1WnfQHul66Ws7XwpQyvtDz9KJURA76e7zFYsc86oIY/NjMWJabMFCm+KENN9fxq5b94CPOYpGsUbNToDHfJ1tCEUkKZAOp4eMU8FvNJ3ed4l3rP9koMAAGLYerU8jQ7fgaBaCi33mp8kSnPSofn1jvwbajokwJGFCrDADOr19bxRFlxoN3CYrfXs2bJtGD1kNhpt0/8fGy9veJDY62RvtNb2Lrdu+rcfvDmRt8xe0snh8P+8fjg5GJ37/ltK7ccqnbWl3r9pwc5n93/tOfOwqmFYVKs1Uoy3HJiMjg8YJkqqYmYBBxLqGZzmmYoKu/REHvG/DhQdPs0zjOc+PU3HmABNDFxmQZr68KxiQjhwHdK/XhIIANQtNKQU6XBOfpuLNoptIYYVCaziQEHNIKHoMJwGl7t+7gbUP3nh9ytr/uOijI8Tj5gGP+DP/gDLPnxFw/ZWRFvTM4yNSnGR6PTk+UOLdeBC8msNqZynG9zsKZiDyj6e++8i8r+8pe/pDL6xKUloSj00fw6i4tBDWXDH83VuDCmIG4GKeMWopBRhnJCMzq1M6FvMWGXncCJzGyn0PZwOIQp4XDOqsKAqZ5OEzvhtGgPH5izsZxchP+YNcSBhfFB52z/ameja+/Za1978503v/Hag/dOD2PMHh2dKGRh3iGKPM0XWPis0DvTShPKx3tusLo6l5DU5531lY59SHvPH3/xycHuUxOT+sjjzPZhoUMtMvOUSfwzDdOEhRXN1tqMS/Rw0AhCGrMoWulkKCPEyCvEkM3G/owwTkASUkwWe5aBzlYT7tdAGGBpqs/9aNpaA7IEUMyraNq+tPm9J9q4KigbJpoBPVs42zv4/Pn1Q6KFlRkjs9S1Z+w6B7JciG+aeWXGAxljG2pjRIprpqq8NM0y3zyEDGRSmmk1sTK26s5w+ydjRjJTHpKZ2/npy6TlnzzVjXafgKYGv6qTAXaISABsKgnsUpevVBrkUBrIvHSl8GB5Xty8y9Sytjvot7eFgy9KiPhflwq1P63NSCkrg5d+hE8oS8PQGaSpleQbjWFDlY6F+MrYSpldnqWHHeYLF0VXMf5ik/hv2ZBt4WWuyZ1fleki0JXzOyi+PK3mr3LQr7gAHD6jKDcfK1RRrVEcY7KGOTZVVIupDgrXHemxRVf3C26wogiBP0EVV42gvGUC1qp21UN60n7qbevz9P1X/A1MX7CAZMgsnlw3ofFi2Cyudc35YiPIkjnNAyjYs7S8fnpmm6Zdcg5gKPgLnToeiQHArmUAIk+3e1UQzMjV/syaXeM8aVa9Sj74WcJJ+6C+DBgyYga5ESYTUb6wHOdlOmbcROrMDXAXLkpAG2k8k0Jjx3KSbJHBiXE1SFO6a5Vft9Q4Bc0MELOHvBNxKX+mVyymhUDXtO1aEXWOjLaYizGKk6FRdKYC8C2inAEOGrQZJtlYuIX/FS7aaCjUBGDGV8iIw9HESFgUtA+55gdnpOAJAs63xZLoOX8lZ9yc2r3aOe+uzAm+EMEOPmQTqXZg2+qOMAIVM2D5v5xcUrx6mV6CWSVaUgDwvwRVSf6ItUpARou2OCKwv7p05ZTArPHb8wMGURKjOg2WNreXE5Xk+rWFuYOnT08OD3HT/u3OzubOcbd38OnC5f4QLHhtnZ+Nbt3eXL219Rc//ttffvH5XH9l78k5rsD3ajwcEuL0HFHP6Z1kDW5nzjyo7d6O70WdVcjOjLNE9VlYFOTyzXffW1hde+O1t0QhV46tMesbWwJafvD93/vkk0//8//sX378i1+Qg7c31hOCWViFee64QmKKRB2Tq16AE39jWETcyOI0q36GJRXUkVPBHsdgsZ87acRBeig0n2d+2Lp+cjx8/OQh+Zt1gO/V9ubG9uaWCXIyHpIw1uZXxBjhp8gTZK23unP3zqcffzLaH927fW97e/v502cM9eus593uj/7mr0/OnJu4qFy1hw9BcQcnY5CdhV6Xt7OwT6cCIQuwB+4ohmG8OmfeZWspHgDfoyHM97vrJyNa+PJgdU2/ukuXlgb4off6a5CH12R/dWWwOcCPtX9jsGEjvWX65cVNByHt79nEhUFu7X82fuMeR+9vvPv2+3duP1hZXuNX+uST8x6vulD6sCNoIFAZjmFOUDToHKS51VULJTAq53nYFfL8L/7SUsXZKBZBdmpBxUtXnbdKEKkm5jNjwOulUUfzq5mRM6nZFjNjiQmLS1n1jwZjuvFr5I3Mm2v+YuF8YU0E8tG4c3Qx51zD4dXimTOUV9ZZn4ZsNM6G6a0u9wda7QgLJ5mRxGAvIRKXIylDJ3oUPF4+2Nsrq1UMfE7VysnjYvycd/7f/9n//UJfnbN1Nb/WH5wra7MX+Xp4ZO5CT9wG8TaJFq+XeTcYmShM/BXmYvxk6ZyzWhNr9oTzBuX0NfMwDDAUybM5yNGebccDMyM/IIwhMz25kjezOFjpi5IJaeiZ6OCRTYIkBhvGnCkaXzQWC2QNHDOprekgLVXEi3uRLFQEWocAuecJXfKrUpIWHprv/KfW1o5GH9uUUYhv4y2Yhspb/QhxQzY4aFogCMH1tnrrI03TJMW1f1JcqgpJV3I4bbL6GVj5DtNsJq3QHEhPgI6nj3+OvTw5w2JJcNTfzGQHC16cLcT3Cut1ntbZHLzWEp0qNpztvyCUngbY8bqqoL455JsikhWduA+menIUkZDfYTXDnDLAsLKCz8tleErAAeHQvxJjsiqh43qfjzME2S8qg9bqXQEnvZ1emUK6WYJXaHpmjK5Xr+s+yVgAlzNuBxZTATu6UXQJmELr1Vpoa7GOq2R3zWElIfrcgZaMbSoOia88BdZJob/mT4bhxjUb62pXXrQMudeTmZuW1cXSm6ELtc6kIPL7XFPNHTDP/Cg3N1+m25OPgjHpca5J5VKMwuyalT9L+eoHACSKBK1h7byNSSd1WKEt6uCGK2lJIaJaVJcp2xA/pD/4m9a2f4WkGQo/65VBclkOLI/XIlc1UlE26kpvohqau/DViDsykcqXj9LZylXkTT4l5p8CgjVhsyx9SU9DTI8b/W0MKFm9hm81D3XQh0GweacVaaTG44ULF2s0P9iOV/q9vtLnfbVI8xqNRLt4s7vcf++9v/lP/79r3WX8hseUTUIHp6frD9d7h88pVHNiZwnHf3kV5/WoNCgDezLtjxnBMTfZX+YOocmz/eWV3gCn7bF399c3sLT7r7+xtT24fWdTzGfQx7iZxYXzX7u9/drS/P/h9f/jn//rP/3hn//Zw08/MTHvbt9yft5eTvEzfwJb/XNl8rg03yaZJOZfRCWx+GVcnBfdySYehuUl6xnof89ywyKG+NmnHxM1+FWNj4ePvnh4e+fWg/t3UXrblvYtP8/NP3jtjZW5zujx4+PxaG2dfj54+vTp3t7e3bt3+XILrjmaW3zwwAae5Z/+/Geff/qFVgxW11fnV89s2RqP4Y/RaY3UMFepWRpnsKIGZnhC8FqeHG1kjIyLy+hX3AuaPA/RC8aBlRVHGiDzwnoI+sl+p3R6qzgi68JKUiUXOn1QWbra+d/9r/7k7tYbO9u3rR44iPKc8txZ6q+QXYJwaUYaAgEUQsa8dDKSgrIOIpLGyfB47+n+s8cjLmbsEJT7c17RSJgFWEgX4oBUFNHR9OI5rb1zV/vDIXGE3ztZkOTH4k/esrR06/Y6ZJWVpmLl4+SKDn56vuDlcK5/sbo9t3N7pbfZX1m1cs3SMt+5/RpnwcwsQhSLlPvCyP4i1cFSjUElitxPpshA6B5TWKfAgGHojFSADS6880/+k2eP9z7+pXF++OzR8939g/Fh5+qk8/amJeHllU5OTbbfIuHIgMZhjrartSmWlW8BxC6GJw7rPLm9saX12m8w80muTDcYqy3mryv8L1d0Bvlyl14D6+ZnPWZw22ibifksORUQncPHxia8OQw0w1QIouIbUzuf4ExGKjnyI9UUKoFK1fHllHz08tVylpzqRbBCj1rD2nMxbXUY57xqiWQR1Kn62xqfu66RjUOcQvNCkvB0bciSUpqak0FDWcB33j2STQ7LKknKs72/thvF2zmaIdpx7izMs/D6eE42BmzBmL0kEwIKXbK6EMGUzmHUDNeA4pkBWJqpzdZw4DeJgYuDBkfPwOnx9LwNuAy5vyCexygw9ijmAvaY2dAOc8PP7KSdwCWlJ6/+kSQcverKR9OxCf0tViU1aYUUyeTCSMLwzWTVk/GUIh6bP9Qp5prsj0tDawmNouO5elLf/iPdNK91uZU/edaksB8diOoXflgtj8jb8meECy3/YZuVUlOrPUlLIvQz2V5drjD1RWAnBtU8IBNkCAJDoIOE2hx2+ALQ+GEmUV6ibNVCkzs/OY9k/6ptnPmctzANGELBSRMpU9WY+RGBQ6XlsRULRC6YDBtIaoXNk9rUGdTJjNOCeNH5ScCiE8vfZCxvfN7QI22sPqY1njPysSxZlTX6WFQ8aHT15Ph4uZtAD9ejnMFnk292SXXnV2/trM/Nb965g7BSfM6OjvAoCxVrSysCQygzhFmBeJ5dOLET5RzDc96nCNRSZ8WGpd7yovPvVnq4+Oa6Iy+XeVDff+P1rdt3sjH1+qTXF/yog8+xry6tnFHYBcTQ0n/+7/27927f+h/+u//2Jz/8oTN0Lvt9/cw6eaMM1S/jw1LhHwyOxpVwDROuDH8ItXRQQj32JjKXUI9WAjR1NDwW2eONN9548513PP/F539hDxIGLESXcyp2d59RZDWmZ1F8Y4MXgmVgmLAvYsfhoTx3b98jV1m32tnZ2dzeMG/x4+fPc9hAv7cmpusofl4QIJiiQg8wIeMVig1v+Pnz+hMc3EBJgAJZFIPyammxN9q0RR04ZlsGJjVYN4jx7ZIctLDY641HOr18zrMuysXy3e3X3nv3m19/59tLl/Z+iaPNZh73Zw3I9jTSdpZ7uX/wCA7mmel86Pxni9LlmPsbu/vwzNEFo4PT8RCVOR6d1shCCXohYgr9A9hlFrWyTGa9oyR7dBaC3XvtFi868VTi1tijbm6vr6zQaz/d++n18tgX/JMZSZZXF9ZAFNvc6M+vnvQ355Z25jqDq4487ETLC53TA3JpOdboGU3XgYhI5AVZw7woBlBkMRPSP67zR7EGqYAMAdbIm0KuFlfXl1a/vfbW//o7nc4fdI7Go0+/+OWHv3r62ehnf7p3blu7fd/nZ6vzp/3lnlAhZr5Y5uUnGcEU0UZsV1dXtncGV6OxgVvINkJjWCtTRROaLSH6P0N6IBBn7Bi24z2a+ZWrTdbMxJC1JLR0DYfDocZhXCm/7HOe5cnLmyJ1vvu1l0/aV3J8+UHiTVJ7s5RZ1flw0tz8QXFaG6U3aOerVJKSGmEKCakLr5MCEettVrY8oASZj9lfF4xFEmi0xGtmZ0Zm0mNc9y1diX/qODV3/2xHWsBpsyUmThD4XMSMYFxcHgKQrCuV3aN6RMqCjNCB0Sb2ewgKN8M7e+hStgPG5Abu3vNkUAQ6nD7I1xpf3dSR8szLSEhuJQStmsYflpkh818+0pwMo1JnQA1FzZvAsD1PUlq6r8KAlauC0N6SvOIeEUZiEjqps8uRQ25ox3TYmHH7+B/tjiYpu0a5waJqAp1qpKTQ1+pzIBGHCr8KDC81SaYbnwfFX8DlpYy/4UdNgohyAWtkEaGGly0YnTs2OKYOBKzsHxkezY70ielVvYYmDDKV1uh4VXyhxjdaTuX3CYtOVnmQB/hcYmWWizCeCLMxhNXIEfrCUpGpzL2MeVE3pWVuF35P0CCUDN5k5mo4/blBIfEgjK/0snOkoYFJcCgZCrboAhRsZob2Jo2oLs13Nzd4+Dg2gRVnIDbaytLR3v7xwfOdN95YtTl0dT3lWypYcRy9LWSI1ortSrGhxT6VMixzx8k0FhZevIsI9aIIk+v9wcbAPtr1wWB9fY0GnPYx0UbVxJ6XHJVo1tUSA0EkRziIVaLBdure2tj63h/9U95PRN+//eFf2Ze77rDhzHYUrnpaMmlhdbkupJtJh8uAlvTrOQGKaY52NLH3Y6usii7YLpYyjfY73/7Wm2+++ZOf/OTJkyel3d6mxmG6pAHO2Fy1X3/9dQDO3uiVlTt37jy9foIHb65v2dR0lrNLzo+Hx2JmsZj+xV/8hWjV2rYWsymeBzmMYuYqEGH8yAVJKwFLrlnJIEDpodqccCnRCuJ7YD8eKz28izlYX8K4MuKCg8HFC3IS3mjURTPvnow6jnl8551vf+Nrv39r5zUOlQwSV+Pr0ekItTOf8FdsVq06TjRZMuvj1hYQwQxiEovt0bNnp8dxOsd3ITwbWA4nHKwf7j83qBGwiDSYbVExLaaEF77rFvQq4yuBcv5qd/8oS0n9hO5wAPX48skVPrdwuPDgaHmdAdjJWhVxdGOps7XSWafpkvoOOnOHnYUj6/Kdq0OOICTITnc9sm8EFecfZxNdfKx46JwcoKBWDbGIzBDvgXbudLEvdA/TsrT2z3Su9Y+Vtc74sfV85mg+5f1vrX7nD9/vdO7++89un//s8G/+8q8//Juf732xdzDcW7nscJPZ3tjh62ajAGdwZ6fqXVYrD084QJihQafc/R8JLxc5G0zkiksFcFuzjP7k7CrijZZNm9RyZxqGBntRRDiycl0wlWG+MeBISEWrIYz8ky9f/lPlTCb/7M3NzLPn9jD7+SKzVrvCA14ioUUiAnjdyEB7+YJ0hDnNSgD89gqLjV0irCmUSgZ3z1AVcXS30BsrCt5pT5GVg2uBCGJ/Lufn+GHJBvu1hNsz7RaT5o2lAQpmOImFPH4ZXE280qz4ERB95DfMkDN+vIZdNgOQLvGKvhLUYvGSFUXrsm+jFlM1U5mQJix0Clr9mD8plyhfKCHEMVAxdUMlDI0eqT1/Q2bT7aLWM0h8xUPyJuvkgi5+pjOt1kA50hbl14kotmMMlq298NRPbIia7y/gPC3jH/JvEaW0Dti0oirL5KlByKxL++rtpB3oRfouAyhP3uZnrkLo9vg/4l4tCcSMrX2pvYvFrqNx2af8y07xefwA6LE3GAYv5dfOGowMSF6lLRqTGdgaIn3COG2c5KdinTsfxLfD/MRDIrJljoKADsKwLFcQtKWg3TCpfS4r/Kwd50r2RTUlddQ0DtKnVVLdM35FXp1AZUarqF5V7rplTpUFsn5lRiWDekmsqFywkwmEH8/6+vLW5s7GfmdvcGYryeiYKb0zRukWVrG0RIEiQ4QNxN7IfAKJGSdtMXIJpSLsJK2XT9V6NycbDnyxtDFYswwvRqIW8P3a23vKm3pn7hb3aP114RBx3MaEgq9X1D7+TbzX3n73nT/6d/6Y/+0Xn33KY/vsOCagNLipGRYk8/FSwS8QCHeJ0BlKZ9GH1qtU60HedR34ZcF5pYsNmWyfffqpAJMCXb12/8EvfvGLTz/99PXXHxhYF45lqXXHaSXLXU6aeHO/v2b11xB88cUXz57vvv/ue7YUU5cvDy7Xtzbe++A9gvzFxQ93nz5HLUT5Ssz54r7uRBTOkOSMBaz3gjENdUjcCytYpgDSEK86voAYGJyg0YaiBLgc423TJqhcnnB7RpwWhKsRPIXFVPiQ73z3u9/91h+ub947PjjdezQ09utrG+zIismeh7a+F4tgdEJK+vKKrUmWO3kdOHfp6OBw30L46eGBFToqShcbRdIIO2NnUJ2xj0SxQzeyFBdaBMxaZK+moac38By0lhrX/cxX/v3jlcHi4ur16HJ/b/zoam60tt3fuDt377u95e3Fzs6GAy/DSi+HnevdztyYl1vn4jA/506vBO6ct3Z+RZzjapCJQVIxNRDnKJfCSl9V6DFxSMKA6cc1/7QpQdsqT0ZNT8kEwQER9fZzLtZid3mxz6uF9v/47IAJvDfY/P7SDza/94Nvfu/0253Hh1/85JOf/+UvH/3y8KPPn5ENNvu35q5Wr88WyQ6EZmIiwajUlfh+KTdNKnoFHKBmRuM32S+fTQgBtYX1zNLZZWwnv/0B4Pxwn10a2S7pExo9oSGzIl59kM3nL/LfeD8pYVbUjVfJP2lJUgmySUn71ffiBcCGIBa1mqRWdUz0rxTuJ0gYHRagUiwbMclarxSF2kuR7RRWh7LdKOeh2taeXW7xRXI3RQxf+C72HK8rknPsh+rLBt9QwIQXWo5lLfw3syiGHEgaISxSpGmeLhSHABNU7/xq1OvwqOHq5J8m5X3mmmX9SHZcSTDAOHdGG+5cmCw1RHAp+nKCppmu84scI6hC1UXgKJCYAwEMuOVnroYJ7blgOB26KeRMj0gGdWmf1zqnVfpqbanPUtTreUnobshBBKuRnRT5j/OneHCQNRM3Y5+ZVl1yz4RvV4hT4WpCEOQZ83IFXL/hamX+hgyvvoq8Uya2cKbseyHC2zZq1w1LKp0uMpchBLwCfVoXKhn01ZHGfYPKeIaRK1R2NzMlUMTYWsZzl9bNbFbxWQxZqFqVES6g9GT2p1A20YashAYsITNJzD/PDbmj46pbdQGJu9Zz4cmYqT1yFUDBN62dLFvJlMoKZvkEMUkWidqnjRlxz3NzbI78hYhlbZtAPltb7zjr9WzcOe6aDeYQ7M7aHkvr8HhndRCE1x04qSd8cSkaZimcc4b7Wm9wa2vz7tZgGxN2PNO8cJShsAtzDmkyD85Hx6c8k06P1zY2NS8WqrDMjHBrMMMMtZUc0lsbvPfBB7/88OfPnz/PLtVGGszNyMIg5WvfIZeZGDrllxJIBhEGsOdLDsnzPMW4QHObspMYc2WOVj4W63Iq39tvv/3o0SORpzDad9575/GTR198kQMYeqv49ap1TQ5Zn332hU/ox1q4u7vrLIe19Y1ev3/77t2z81OA+8Y3vu7Vn/3pn+NsvQvTKgqTNmkPxNAYEoaRDNKEomcCoN3+lREj+8FQGPQhvC3j5E5lZZhbOTrgj3x8cbmwsXb7ztadtx/srPduf+/3/4ipeHx4xWVdoLKN7h0z+vDZSNAMhi2lnVyenI5PVMTDjUaeeOCgiVefDHNIH/izYnOMzlkUmXgwHeDQLEu/tBHRVbAxDa4FOSgasUbLaaH4Waahxkrhoc5tcvGiu907On9yNHrc6Q8335t/8O7G6lvbnXtzne5upyfEbNZxHZREzXW2MqzhP4Yf4ozmW6zHoGHk5y8WV9Hu7JPMFBM8MvMCu6VAHYZyhfJmZlRj8xGRuaZP0MbkchXMr/rOZcrl16lEUu0SXf6qc7T3Fwv2qxNsqMjvDh68ff/B//zNzqh7+Fef/+2f/vKv/9XD8bNn99dv3b31+tXp/HDveNW5ywhEyH/ovO5m0PI/RMw4qVa7TWpyVcnZLzPgasSXb2loXSH3hbHuTfo0c2do/OUPW0qbJp49tOd2f+Ut2vFS+qS4glX7NAOaGTV5U3/0qf1u32pYSqk8s3t7COsVQinDhM1MaJREOCMmHMMyZpw9RRYSwnrN3uwFyxr93JyNp0knGAqUR4J3tGqc72EXgU8DMtiAzF0yJu6ktL9RznFjrNowRHsuEgQzyV7IpzFeDOsnKGVwBsvzg7W+PQZWmgwhMpHZx5LkdA4HpQgLNV+HKym8+h7l2kgTCpnjGlCimObR16hqaq2riOjk+eaf6fuJuMNVTOsn8IVFJliMzZfX41POHUtM0HTfnJ3J/aTDh/OUsnKzuH+EZ40B4RIM065iWmZxqXSlX6YLeQNU+a+YX5Hd1pjiRlNDUEua3BX197lKFyX/Ez6I82yjzpxdRhnszopKAP4MCLG1RA8o7lVt9jRhw2mwbZWJUZ8mIBzhnVKyYn/BqDsqdS2SeDZcKDEkLAwf8phuTVWtLvosGYpwGLgw4OBRKsqVn0k0b1JTIAQ5MBp/AiulmgtpZ6l8Ek3w5EzmVBDykawpElmTXUuCangJnlX0yjLNacicbPPLl71+BH9bZPaOuEIzH2+si874mjiHWcJJzP4EichCYyQkvVZ91FDeTpvrA77Fy0hhIMj8azVW+9DczkD8jfmF0Xho4zAHKbqKJpEGrh1iUDuwTJj1ft8bjWa8pgfb88MVubfYx0eKCGY66LoO6xr8ic6ZVfGEA3PLdEP/sUwT/sJJUVkkTIANGx9K7LCaK8Yy7BfbmWuVxuDx73zrGzvbt54+2326+wxXlr6+vdMfwIVnAlGsdntb29vUXDuD33JuA1fd1RV5cWt+3R988P7YiX4/+7k1YMsZ5ipgG820xmXyLrBmRK1sVMbL9CHAiZk1rlTcjEhENqN37L7GsG3cml/g/TzfWx+sv373vfv33naE0dbmvcO9E2eJ9bJDy0JZBl+oKlv6c4Jq1k1B4iruwxb6cQ9uLkL9jUc2X1N9yVAoGGFLIBgDYkMma2uO0Ug7uVU7M9RCGn91rI1saAg0PVGmQdREDZLF+EzkdHiTMGaCe50MT55ddPdWbp08eL97/1uDzgMs/4vO1fPOBtHTeYcc4PlknQu3tQgfuFSfndpurFy1QkhQoAUkZNr1gd3OkCHTw9wPr0+dXK2Mbpgb4EXsajNi6XQYgSAvvUliUNxzorNqJ64vKdMmrZbS37EdQOgv2hYT59L8RXf+em2Jzf2fvfbHP/gP/vh/e/bj//bDP/8vfvRnHz27vbHz9lsfHD09WYo/n+nP/pC1DfdqmlrINIrEd6NWBe0jyVhwKoqrqcUDQtF1JqQj/2VWhaJECPMVqHqjhV7Ak9ZaYEm+r7qC7Dde5dv6fJbX+/b84uHlkhrgin81iqaEfCKX1ufbohSeJgVVceamvxE7qro0w0xH6BYZAiLK61BSiHBkuAhMFeG5NOB4O7NFZ/XXKq9l3azSYnGCXiXqJGWQtsOLo7ysOIVmdDPFTXRFMahxAF0xi4WGowBIQZFK3YhLuamvVeHHJvX11UYOGI8erTN3bq+/9c5bTpzBgBOxTvtywreNIAlecGJnwsWlDZrxXTge2ldX/Y1GD2kGpelUr3wHeyAWWt36n2FqOFdAC4ACna+6Eh7oZrqygAvmG0ZnMIgcSxYwic1dqGCT4c3M/wjP+tjao/0A3QbZM3yrsU+VeRXkDC9x97PQV+78bI1SSGyV/1BXgBSjn/LZ1nhj2JRkzyDOoRYNcMkRzDCFgmZ5btPM6GfgMmd41WRGBgGkuEMKa2YR1zKmCk+bMcyUUj2JiSI0wq+0AIVjm2jMtVKSmE8QgDaOIJCrvvYcZKwMNXsDQpgSghmBIIEsGrj8ActcmHUWGhWQxVum2lj5YvSGyVF2Mi+sg8lDZx2PDw+yTzyBlxauuFMxym1uf/d719jSn/35n3JTomMpsSemmcBbxI3TCHBMpco5O+PIRQkbJabh5fnG+hrw8kccDo+0UXgPFoLD/eHSymp/lZcvT+icIJSNJhwhiULWkVfXcrjP3sHa+tr7X/uGOMzO7LNEzTqVKR5OxckSzNM9BkMA9zmCFkU+V9AGjTvlIsUjlxRh9l5dcWiyyvv08RNOYZRaNUhk8ebXTAl+8vGnXK6YpQ+ODjnj0CZ96CQGKi91eam7cv/+ffzVSjBFcn1jbXR+vLa+TqXWYWvGX//m12nYf/3DH5FoM28Lk9OUCASUSHzW8bSTUakJENqOgpvBNr4w5HNZsVOaaG4p0W4letjO3Xffev39u/fe2Nl6QKLg5/Tskd023KD1U2wwWipQiOasis5Kvy+cNfDTexd3thwObTn9+Mmjpw6dDNlnc8t22OAsyxztJVz1guQZptowA1nF1KUlKgmuBYbIRaANJ51ewjGaqsr1+mKRP/P4YvnkYunw7rtrW29srL97t7ON0TIy7wG8c0fjCIYmrnRW1oJ1iPD4bF8wFAC/Dlc25Pi+ky0sZsBNNqAsnUzxlV0jqJxpr00arSFS0nyMGXivVxbXwruA1MyIZIOwmYqYIAkSFmdNE4pEl/dz7nJ8OgybU2JO05i7Wj4DOmr9yd5Bb/Fu5xtvfPP9H3zzP/r+kz/75Z/+t3/5b//6Xz/YfG/xok8LtkpH2V4CMLMg2plSsnpgDYGol5Vc9aVKbYcyVccLeiVleiUxLLnoQJhuydx569kdIkDI9jz9ZvJXF8z4hj2vvHrlZw1w0nxy84IB1byA0KuWLcDM/I8mohdpvlxBkcmDt+w3YBvaB9LZH+Kj2CGsSeQBycmYzAYO58zpgShKrezmvDkCD1TNFroogfixeW4rhW1eYeClN9tV55mCkXqNmgcrFRsbfRNWjYdi5w0x75xvFsE6UEyrokk7ujwBPDLQg97c+vrGvXt3vvb1d997711W3vHJcQQdSJwoNxRzl1BDAnJ1xudXHDx3HV8ttKpW4cd4s4O0nICNr+tAWsv8h79np00wrMBSYIIEGTKXP4g16LaHlui+eHaRo9AzzDT4IKE9BxZvLr7x9a/Lrz6fEJwjA9NvgvBhHq7g9PReCZPbK+mxYdbVMs+GrSVmzk2vjJhBf5EwfVF/X67X15rqbunU6p2gPj4OZrfyv6qMVzr+UuG/4QdUylSg/GlqGJfZtLw830dYK1aJrb02ybB5oJK5sIgCsiIjfkGBQpcr57DmGXpqnDfVT9kj7VxkUwpOlpAJIQQkag8TRIfVoTEGGPrWAMW/NMgHJLHt1HOMSVWm6kJmXKysibRswlRWlCCKsMbVGkaBK8gxA6zn1ipFF7ZEikT6zRxANjdC0RJaAVnJHMD/HD833n+0sjggfA60bnf/ejjub2680++987X3bP/9m5/8+Mc//YmYySIVqpEAw0S9trq8vr66vUOxXF/qLh4OD57vPRsdr6LjoT04er+HlLvAx7S5d3+Dgnm4d5AeiYNRHJFE+PzRo+2NDUuvpwd72KTAVUJWwdgCiZICSoCjp6WXYb26H1/fTPYofjj1FXMGBZ2cS6X88Kc/s47L/YqV2Je/+tWvdna2uVDdvn37L//yL9miK2Xnu9/9jvlmBq72B3yJO6xEF9cikLz59lsPHz6mmm9sbxHF4Q0xn0NZlNa5+ee7e9aSz3qn+PrG1ubwcMy8NB6NhIU0oXHo7a07TNz4YlhC/EABvJl5uZ5Zg19c69mRtS5Y9uGBmGxLG+u3d3bu/fv//D90XKCgn9ayj/fVSfrvowRBFS5wukkIwUqvHGedNdrO+TzXytixSCOPHx4d7CIvHKyEq0AAs9gcToaKuBuyKN5QIFOreFbWqeHs5ZUwpaFQZxUdjHKOw4Dn4sqJ2GqL4n4Mz+f25taGW68vvPaNweKbdzt3YOR+Z+5xrM0LtGnEzTrfmVjrqSkUjN2RcGl79Lnyzo53NXylSD/HZYwoTFMWBImhABUCoDQkQYOiB5GlQjxqZk1JYCYbhdO7UITSjEURkA9OQCUYFuZGrvU2opCMmaWZNaGclhgv504jhnROOltWRq47J3vcMDv379z539/63/yzHzz78eH/8//yV5dDm8d5yvTneWrb8D3XtdOYz+7ayoCEQujUPqc1axn3Im1WV83WamvGK21GykJ7a3oaM1eaZIZH8c/eE58AuLsSvG0ap1F2tQJzl808142qYnb34JN8mjmVq1XpDoKzZwPRyIoSII+vksG9wNdK830e6p8HV6qTWLJkfmYstDF14Z14U3k6yxWswq5yuAJiZrAZPYji1uk7lw4sp3o6GvMse8EuhZwUkk6PxT8iLUGV06tRjDWwE/GD4qLa2kPfuR70lwVgFyPWjFYhUqMK2xNiDbGTepnDLC9+MYyveE1+/tknd7c22HJ5Vgryw/fz9mYfITo9FY3fudcRtZl7jo7OF9f6tjCsDrh4Dp7t8fjcI3zr1GBjAxl7bjHq4Mi0JSLz0za1pfDT/PThPr/GIzTBAl23h1WDyqA3SISD2EiiRTWQGoIArRY1sfEz9iBGQtNAZNmFpZXt7U05Q3IL2sZ1cqWEYMbf41Kxr9o91TcUmZU8xZjfvWTGyTCkYF/aBgcR39+9mN/8hfYq08RVBdNf6cE1j+EAw2xmUbAyUk7gU93MB/kOEudVrvbs3q5KAwyEgFUQBYu5zb/0gFQRiBXHRmRgb8gLHA5ryYeYefHWGTDbQ5UZDc8DfKqvJh5e+ULripOm2V9xKcNrpCpVt1anY1UTAOe/kCtASA4Pq7fvBv5iHJhoR2ImjGCSgEnXx0e9u3d+gIG9/x6+9cknn4goje53V3tbd3fuvfXA3f7OOSbj1eXtrXXyruKiomqlsE82wGRPB7o1z9rr3OLQR2c0Wck8i/vM4e4eodVIrPbFJdXoBHEvLQioACgwTz+ALPcMTuBXaI/iJmf9VqPPaIoqktliMAU3mvl4jD1zvPr5z3/+1ltv2dErLof4lxye0USaMRXNM8YJJJaBzTHPElnDRPXKOvhweD0/sIDZXx0gJ6vrG5Rm8VUcs4ip/+1f/1Q58pPYqL8oyOlJhF0Vr2iESBMLKyQxR1KTLvkdLy/3R0cXzw+P1HZ766333/v2++9/+/6dtw6eHs9drF4lMDNtFzxALK4J3cHKyfDgZHgosl03Zy8Ybva/i5wPbYE3G6b2z0+GCA6i1svOHMyP+gpAZlMQrlAu90j5cAHNtA9PbC0nTC4t7+4+p1HbQrbc5TSeyNjxhbm+3O0cjuYOe5tXb35t4853P+i8icY87lz8vCPKxTxYneQwZ9OHMhuZ1QFWIwMbBZ1oMEfdTFCVaz5O0IxjDHIYDd6Kex7p15dj2kCdLGlcSxRog01aCDrWKAcxPYcbkY5ElNMGHKjGOBAi7grEApNJ03kVxuehOA7LQUz9In0SEerMRP5TzuBilx53RnQrGwdXFw86vVud1x2ivfx/+uZ/8qP/8sMf/Zu/3X8+XOkMOlfr8xfZaLTed+7Ixdhxy8QFazGnBzHp9wQbN4cz+1sjv3QP9UhrMs3zDFgkC6OS5xtXcv0uV/v0lS8kmkozWuQJxSAS1Ognb6MAN79qKT5sieG+mZetbUkzsTLzImiEB7dsMkn3kzKKQWYFNWurwgDQ6jLkDhk89VbMWshE0mCLLsU31sGEIeF+7zQQTpYJurK61rM8ZEpQoXfW+/fu3HJ4F28MAfPIKA6dplJ2+72jY34kERI7nW0eHmbrbYsJVhRWxS94+9btbXNWnOPe0uatrR2dODyct1dw79nj/b0Dgviqo1kt0sxfvnP39p21/nP74hYWHbTqoDc+YoIBOMPNKpN/j5/ufvbZZz49GP3/afvzaNn2/CDsq1PzXGce7nzf/LpfD69HqaWW0IAECATGRh5ilo2HeOHETlachCTLf9hx4tgr84pxHMJywDEQY0DYSAIhhCU0tqSWuvV6fuMdzz1zzXPVyef7q3Nvv1YLMF54v/Pq7tq1h9/+/b7zOP3Ku4+/8MbXOv0BOxPIIsdRlkVc10o1QZhc2tDIwXDdsLekghKiaiLICojqACMBkuerUV8Pu05sq7VfzSP6HUD6j7StQO39ny73dbWEzxby6T2DKv7DtyRJPD0NM1hBw++4cjXm1Vuszn3//tOr/2H/vm88wXVQjTDJJqvZ8rK42k8CdoLjUBYS2gTPWs1ViJzPxMmnqLWCeQwByCNxkYPO8b8yQipOll0j+hPVY0tLEH5d8Os9AzfSmP2bDoUSbHMscNUJaVtNeMLeUJghVVpIn87x42oqfseMpZsEJcPBjC342GoS7YB1lwEen4kJJx4fSVS+mpWS8gnL0nl7MsyNxq1yUbhAvjdn8Nx87u7mwf7Bwf7bb74tg5bU2VqXPlsvloQ7haoAWzCtcoMGHDn2BGHvUFDZuMSSh9Rmzum+a3l8C4UXmIUggV8vBVX7vV5oKWFpTna5Qp6oG9w3GLCpDSECsHgLi4EEeG2zGDQihMuYClyQ+MMB7BpFCo1BxSuXrcxCoq3e+OIXYTsG/O57b3tX8WUGCduZ2b/85a/2ugNzDl0fPTo8OT2XOtxaX0dj+oOhYsMF3QOzayxkjZCJs4/v31M5kuKPdnQ7QwZzC6QEVkEwbq5y2jtiDYsMJQIdjVf5H1bkKNVeVZC105s2a7svvnL39s1Xbhw8V69tEhvGXfUzN/BQVIq4HOZ5IQWsMbmc/sxri2mxuJTKEAFWAkr6XS1we72JNzbz7C7sf6Yu/hJvZhAwPWRMn6DJy/pfryseMsZsTgRTxwxGZBGXvHmz1W6fHfYeL/szTuFcMbjqsNgZrR/dfn331ut3M/tsuE8yi8eZYj9DSBsdJzdukPyQAMPCC5anuct+qLxh/Ai7bVQglywVDT+jV++ccqRWS/rkCRS3E7lwYCKx3hAVYsjBd13qrrYr0LaQOED4MoB7+Ha4INiqGLLloIAGAh5jNruwI0YePU7jC+5rDAExQscE7bh1hKE7E32Ua6osXFRrmpVUbC1WMnutTObiQ//8qy9/auen//ovfflXH20Vlnutzek80+8Mc1NonGkqIqJX5mVuPOqOJxe54gbjUQzUaBP+JeeIPSOOY8Z89Rng6bvXCH4Wy5E2Pwf4JsJkJ27lxPdR1KtDcemz3dVZV5/Pro2deMm4Pj4S9TACF5JLn12T6Ep8c+qK48ZQ082fsV7MNfbTcT9ZvzTM+IRuYYEhyVF/9XiPhN3lcKb6vBqTguajdy/LcajAa5f91HeBBYhoeLkmeEHfKrEzuZ3dLUIk6lFvVOEjBESI2YvV9qnXcNt8pZwjaJLN5bjDr3K1TL6cLeSJqZcede4u8pnheaVcKlAvD/a3sc6O6uXd0dZm6+7d6wS+UjF7cX76+NG9w8dHkxs3yqXi+fGTRqV2/drNAsffbKhaQZW/uZDbbDaoR5q28S5tbu2IZamV8oPR7iJbeu7lyeb2/htf+CLLh/hMs0J11lQtukgEWll71C2sRer/WSJMOtYSGVJjT0ubtLJRlST8fCtEtBBOScuUDsQyxWr8t95W6/3tp387fKQzAxSutiCY/t53xA/BfW3fhI+n57z/SDrlH+PHCuBidkwLt6rKjsrfoTr+sKLQg0XZoYHpFYJLkd1i5LQtfDXhT5JqgY1hBfcKXTepk2GvCfkehocXMAgH7kshJhSFIAyEhWtFj+qYsZTYE+8fnDK4TthUw8XEh5FoTQRLxpAMNqiSm3hUCPxIObXC2Hwmz/C3zE+avTBTrxA8nKjpd3gVyx28LBEBpCjuF3/Gnx3PR7pTMQ5l6vVCs8EiGWLnnPo4H531mOA4Rws7O9defZV/s9frVrQAZu1EX7FTaBk2H4bFtWFfg4EouBYskzW+yCbDLMrRftnN9sMRuQjmp36SeGV2GpWzaMNU1YHFyOfGQ8UixqHaRMilwfkw7JgAX1gcWaJMexhjYzFMm39hAY8lu6s2I9puzUKj7fWhOh4Ja4jqnEqU4Hv37n30ox8lJYi6ogJCLeYvbifnMxqzcOkeIYhZ3jA+XVhfRx2Ys+iYHqOsrdmrr294nPKJpr/WaHmF119/nTL/4P4TFB1R6Pfl5i4r5XpU/glftyRqJKJRzlbopfNx9uMf/Mzu1o1b155fb+3yBDOq8Z6L4K5WailiQ906ta0UaYygpSAlx8eNqhIhlSCAg860fTbstdWObF8MuZRUbOYSUSs6/PGsLwpaBVgSuUx/QGjMmg2sBCw5BjpDGowwACuWmT948FatVa7czM7XdJhsyxRqbDa2r19+6DtuZQ6mmY3TTElJq9Pl9HQ+H2a7DN/OpEyv7q9MAnosuGRSyPTWLscWIyg2Bd06Rcv1zKTPJx8FbzBg9DvYMAas+QppEh7wd0R8V8AkoTjkBYwzxnzleMQM4huZYpTeBxo5kpTdQFb5dETcxH01yJZbxxNBBpNXmBGfLtg2+HGgtEsMLZZQ+GwID2vYgJUaTDvZxRDBz5F4MkfFD9/8Q7vfu7n7W7/0M2+1z3o3Wy9RtFtcA7PL/sV5p92tFi9LblhCRoFbiPKmNk1zzHPMdjwjzfm3UNckb38rAyYWBR1Im3da7cTF8YaBqM9+ffaTnWcHnfZs387q+tURn+67uvfqyOoOqwmwD4VW+8GJA8euzgqJ1teQ44LdOieWKD7gG/IEnC1d7FtPxfAYmbHhAc+6Khya3GHAEkKiIsdiCMCSP0LRT8YcNoP1VqVRL964uccoVK1WBG+CSYNhdiGEl4o1XhvjSaH7fVwNcMtmvDg70RMTXwtuXXWbQs9cLuet+rpqAwT2i9OzB++91+6cS7ujQkvilwThj/Po7OQkQkxUXlosNuqMc3MN2J4cnewe7MNopVnqDcViCs70iECmXIG9utgrdsfTj330o/vXbr363HPoBohBC9XwkZeIUETGiwS/kEPEedDhgj7nJymthVeZvs6ktLGxVau3CCOarMeypRW4guakRqQD/10+Vii9+ny2us923HH101ModCDo/u/cvkX3Xf2YQChYNfBIm3MC+WKRksTg83e7Vfr5H/oRd7oCyvCqmZAIAIqEVOn8ihphuoGsoWMG+ONE4cyPt0i8MwbgxbDDmER3uvqMY2lzZRCQANfw8qaevDQN1XdJzVhdyEpBtYLWJBa4ehXqadiL3D5RrkDRYDuJu0aqWrAWDzYYjJlFMlprWPG4YYw2hmKQ7vVt0kzST7wCDAo2ZXQxtzEFcuCCcMQoVjEgq0cWscEKa2+xnAGGBwc9s0M3bZ+VWyo05Se4x9lZrtni+9zZ2lBiqts9z5TWKpl6oSYlL7oXiG/yn+inUl4ZCrBZKI0WpdFczpf82ijGpgMEnfKijQkgm8N+P19vII8hJVgWtauGQ+wQvjEbUFsAuRczRcGC0xZgb7xe2hUxpWYgMRThM1EEJ8Ny1VssYMv1GwdirNwtZNhazXsfHR5WPv1phmicGNfHWQWFEbL9JFGY2WrnYJ8xWQbwRbd7sLHBIg3jM6I2rMp0fn7RYwPAHnd390f9nkxjgUbXr9/82MeW7YtfbF/0DCZRik1NMYbt6XptA4MfDxZnF/PtZuX1D3z8Ay++ttHYLvIyXpaG5+NBr+tV+KRr2+uCRPEl8hmzrVcR0IzgKdu0dWMjo5Z7/7x3dtq/UExjYFJQrs1WJXQRpjCaMKtvCNfUfWRStQr/rTStRPeBh6cI2kJpeKEzk4gcDr41Vxiqvp0fLs9OFqeLQr95p3znpYPNV29n7igQ9U5m9t58iMZmONKy6w1CwWw6MhOAXBpxQgxwFeNey4zWVNig7+K1JMzICaBpxM6kl+pnqL0Rwc/Beu04LSpmkqWStkykc1eZtRYi8TMrbi/gOhhcoBsff3T6DSQCCgDZuQAEIEe5Z9pwZlZYmyPpbCAwpTAptEAyKTe8TKGtBBV0o1x0C3O1AG0wHUEnwAl+Ldeak9HJ1yqZs8yNF77rn3pl80btq7/28Pidr/bOp+Pctc1Cq7DOia7gB5GaQYSrAm66YQwuDXGFzwlG/z4f3i6WxmfCQvv/gC3Ynif8/bY0KelOVzdMRORqP9YotjCJrLbAkm/bnOY8//s3UCn9BXEK3IrDdIhYn4Sc0SAsTHwhe1FwreeYl9dOMTed50Z6ZWkxEOnedoLGMUYZP0pFV2028tvb9Vs396/tbwnkNO+C9qG0+uEEXBoOYlat1jAvOia0rggD5b9PcYcRVUPUL1WaatG1WqRn6CyYQ+UZqU4nx3y6YkhOjo4OjU9E5Mc//nFhWXfuPHfjxq1vfP2dTru3vzffaK27S7vdeevNdx4+fiRqVCXEcl1XsgWPUk8Z20CM7NbOTq3RpL8ePXm0sXPrtVdevnPt2pe+9CXy+mhEQa+0jutrz911+tHJyfHpyUV/YK5QnVw5n0dQQCJdArWlSRMECsVar8sBnuY+pKUQ2mJug2o91YS/bVX+fgdWePD+T2euvtqJu8dqxpHV9vQ+gUOBSb/LtoKS1aefV2e6F4JskP89bVePw72Sksb9qJTuypvkuSvt02iDfoXwHNKAwfiSMq2vZII0VL9TixMjxE6c4foY9FMeDFw1KyB+hQasmgH+EiruimfEjMUT3rc5Yu1srKYcJSEBhPHYs5POki1hnCvWvJqr9CY+vm1uY/ZiAkN4DW5PdwvGnqSOUHcRMo+OARiL0RkTrT9XnIRndlpEJlmS9/dDdJsM1VWv5Au1UlEC8QzgmrC6YObSWU/5oWV2UMjVC4o/g2cPFWjPNhtJMWt5EjGS6y0wksjWW/RxO4uryyzjJ8YXAT/TOVWVwVavBy5b80FrFHsc0xRCCBIXU+RrGmsctes68Zk2c+XTCc70JPIUnd3NHX/08OFrH3x1b2fn8cOHXkqjBQZY3JFDF/Y6nZMJ8m9tLT3XEcfpyroKUZqNh5xLzlAnzPua/ehgKFjsrA2T0fhmax0vM+R8vkemfu7uC2/dut+++Cos5VpVz/nw4cn17TsT7XQnuRu7d1/7zEdfuP1Sq7LFK4knqStpiZQQW6+lpBuUaMAeDno8KgoIlCoyrGsBTbNRZnLRkw715Mm416E1VsTmKgUdVBGjBWDWj8c30niIIVaV9cWFV9pVmkZz5i8806kNGqlIwd7RcjjQc/LyYlboV/dy115s3nz1ueILG5lN+bSn89F7l2vHucokv1UBSrNZZzGQslTR4dlQjZO4yaYbJWsiLiwSO5d9puYwMmutFNx3FH+RgBm+QZ+cwqmGM/UZD2ZkD3uPnSRZJW0YtQ+aDSiD8YS0iRJESFiCATH2eK79OIWfP/zjCdXUhongjZCc54i+T1/zyoRFjDn3IhXZn5CqjIRDMiDZ2rPhh/CqfIpsRuz1DBt1Ktu1zLI/Pf3NYmXn1T/60qufvPZbP/eVh1/tPPzKg7efPLhW29+/dn3RW5u0J6ItPf9bsDcB4e/OU4OgfRPPVxD77PsVAD/7vrpPnO+XgG3bagZW+w4++xqzkc559mknttWppjEht29XR57+GwCRzoI/wZuTyusgYII8cI1EFD8ldusfCzym0VpmbJjFQ4rRcj6U2hs2jmV3Nu5xjnMAZxYDlhi3zGWarcYlQC7nNzfqO9v1a9c2X37h5nO3Dgb9CwqzvIPoaxoF9N20ZPE0/8IsZd6TlTe2tkmlRki+a65viOuDwp4rXtKZYh43N7Yf3X/ovaBwXx+TvspwufNz/PXd/b3ragttb+3duH5bSTYRW6TY5s1N6rPkw9Pzi2NBD6XH5K9KtaaDOKg+OjsXwdYZjYma61ubovWQERV76AwbrY297a1++4LBrFo48CmjaDiabG1tbJ1uHJ4cC9vq98j8KZwhytqO56USLt50e8JhCINXc321BN5qRdNWa/SP9Pls4d9/1QoInoHC6pz0mRhDcN/VFoj1dP93/AsbVj/5DM6RzkRF0v7qyLezmd9xj/82X91wBa9ODhBHRMhZPgKhg6oEY/IZVCBx3xWsE7P9F7+FOp6mL10fkLva0q0MnkoaoiQhMQW8iL1BnrA31kSGGQmiQXi9XdB0USI2UjyVn61GUAMjYnhAv4lvySgdRS1EDSTNIH2iSTGe1cIyLf7ubw7AsaqETzGNUJWwkDYgEaTsCjCCgqdZSebucBjREEB+tSzgQKxxfXNdaK8cA3E7qgrLt830dc6Z41sMPesb6wo+gzc1oBua+7SagpGFLYrD1a8wSKyuWzKXsNrprFRgydFSaKxgBNCkK3MP6R1E4DVL7ikuC+frd7qg1Bb5vhhNetWYRpQhxhqjXf1drUJANOemKCflmURCRedjoYzwTXIRfphmeEG1JUE7rpMCLkvslWHkJ0oqg/OXvvZVuGqaoIxMX6FJdPHWbgsfiPCkXEGFTTdUJnqj1dTXgTJvMpGJeyf31lvbN67ffOftB532ENEbjyfbm3sXx8NPvf7Z3/Ndv+f6/u1Jb9E7H06H+XqpGSHD0WkwBDsmfLMOzvnAyN2pIaTa8rnceiGzt8U6w+b81t/+nJ4PzONcnBWCUeh6oSn3ux1aXdGaMLpaUEDksHgj8JywiJQCwC10rD61dzwqqt1Xiq71YyWw1nrTwmitMt+51Xru4weN13czrcFy9Ga3f1yoZ2o7+cW4xAyxHC+wpzBzszRqWtAZ85OFfxeYCrJBmYW8am/Ds9axujnJyQ5jwwyRl5LMufJEQiHSYX+OSo7MgFEIGMcNop8GaiZWUi4JFd815hUP9jPMC/uPFxDBg4fHPgQV9xWOSu8HmZzibrCARMCgDXdlweUup3RxDZ8UBIu8aX5e4QgM1ItcoyhK1SmSo8KbkcpZUt111xlfnJUZPXfqmeH9efsRp8vrf/zF14eN819+9Os//ebDN07vnWqFsZvPtOjvTWXWIufQkNIM+wzRIHAzbSuCttqPzzgtzvnWI1fE2TusSM3V3ZwUCJDOdeGzX1cXr76ujj+7XcKMuD3GeXXw6T2fnXN1efrHTVas105MuD8b7GdiSvsoETpmxdCQydpcxXBcDgOWVjSW7j2bSjTCgAUg4L4MzoHmfEhVwQ/R232jUdM2bHO9vre/vrNV391p3Lx9cP3W3qhX7p6fds/P4EGZ5CPxV1DCWkF2kVI8O4Uiafju3btQWDRk4HK+0On2u73BeHYxHI5q9bqBan3OuToaDO7ff9jvd1tahrW2VNXlg2pf9B8/OmHUlFxw5/ZLjx49wgdpuI16a7pca23t3tYzrtEA0xz8vemwMFlc9AbCB3LdYe7JichnqQXXb9559PDw3bff2ts7gJysZM1mQ1Heaq3c7nQi2KUcHugbt64fn54+fnKEs4NEnUfIH2vORHSCoOmeFuQrROugW3gDAAe+vscaBfynLX4GIenzd8JNHFwxzqSMpXPiY8Wf0g5IACIqn6zuEDAE1uIJcWFYOxN/Xd159dBnj3bS6vjTXw1xBaRP7+aMfzxb3Dm2ALMYKVKJFfFiShFRBracHEerdEiv4w0DxVGxq4EEG1jdAawGaUszFjcKipGQKyypcSmREdzGzCt+jCyiKW4eJjOCOoZPmVZoLWjNCuvkWoYHj/cYy4LDEeHn8RzRIYRG+hRXVww9bbGgoQxcIVpQ8NhivoPQPt0S8458p0DDGCH3ayhzNIjEgOP90B9bTIiJkGIRTdMoVG7FRxdUnZRQaFRV+w93I2rXn2bEkI61YsvTk0W0MDV7CYIftQPrUpR41B14hE5L3B9h69Orp8AjF92Stb0btnuT4aBea87HaiO2ycEYCGWU8EGOxfNA89lFVymdGJ/o1ZgBW/DfKIFlo1PH+4SLPShIHIkXZzEIo3W8RV4ZjfDXtTub21simYcyki8vVaje2ljXvL3Vqt28dvArv/I5WVMkayZLQrcosIuLM3E5lkCFTXZyAc8Zb6f7sXqbxXJ9o9kd9TUh2NjdYoMSyyTcrlRuVkrrw95yu3ljt3V72b9QsMTgP/7xz3z2n/nhwmVFtuv5o6H827WZ4pQ5paeFoWnbwfOhhgG/tTTFkI1yuZ1mSyUCvdq65yfZ3lHt5N54NDx58ojiW8yq21wgwslzFoBJh7OSzWotGBqbblQWsIRAJ2AgoCsW1NSESR8w4nT8u5nGfJi3JsP2+Hy81q9s5/df2tgUxP6Z5zLFdib33nx8vqgMajUq6+h8MGjkq6TSCGUIyy8jsxDrLL9/lOx1W5qP9+QZU+UMI6PsqmMxCjq9wCijHHZubQbKcwt5a0RBDBi0MeWvNGBLGVQ+iFB8IlEgPvZAcKIknmqRGaVdFQpk2EUD0v0FR4iSVEiZ6toBA0kf91XYgFDlsASwEixMeCCb0kthKOhfzkuLtfy0NJiV1suZKrGF+ssqEKgT/v08Mz6+PsmOKGjBhObj7vLivcJ8Y/MHP/nDH3zxi//Fb3z+Zx8rD9eobcrihq3CzyJSgQU+qswGG1F3X9Q1VACL4XKn3pk4mroGdEkfTfB69bEimLFYsVTxaibBzmoZV/QZRMdP79ucHxcGUU0z6HuQmihl4E7xF7eL+Nygb3FayCpBMb7lPgiVULaICZX96RnJCRCPEWzsTHPsiDXmnUVEyFnUBZFUk/mlXu6jS77emWrlIq148DUe92uxWqjop8a5W+PhLebno2p5bW+3ee1gc7NV1uhLiVKELVfVYi3TE+rRV/An2jWYSim7XdajSm13c2d/b6/KeapUxWWUMWu2WuXzi7WgFTxcUnu8lE6Ho2s3rivJPp6+I4+oTFXF8GuVaqUl8Prw8MSq1yr11z/yUYHPIfIpETCdoWH6dzbXw45tuWC6YVx0e1p4sK8oIcAzVXhyeOvm7Ws3b/H4n5ycsd2xh5Gzye6c0NwWprUX8ZuzfLUs8enGtYMX796RFpHXiOn4pKMgb4O9O583XJxBARDQAD6C8qYtoDW50yKQIcKL4rgF8GGL3QCI1UGLF1ARR4OpQAEgFj+x9jnkXyckPuD2cbsV0MTJitpwrAb3TTJq+i0gIOUfo5/GFGfz6Ud+vhAf8BN5A4Ftq5MD4wKeQHI6AJLjkqfb1WlPv/63+jdAM3ieGwVgx8t5UymDFEv1iJbCZoYgLgL2/E6ojkHGOTE/obmuNuD+dJbArqRLRGI5zuYrAJ3VzElIVKR0hwYnBAl3d0FY7BKGRAV+8E2xcH54+yBo1APRoKmUjWgb3eBldVscVVPkilRMFxalc1Mwz7RAEjyMwKqixEiG26Nj1sAWjwqUDMk1RAPEKNY2/gkFEtONDjBxISoRFkQki35tVVE16W4oqDIjsqVF/Cwm3clIBalivUZbnWKceZmpayOiogSjy/x0sBh2RnkNaAvuOvBw7zAbzcg1k/6YsqjIZ21jXd6rUVGLh4O+vJfeZPDk7ITNVDoT/04k109HoiH0xH3nnffefOu9Xl8Y1pQ4C4y9QyyI8aoeLBRS6FbYYDN660nu1+NXkKRARsXv1Opq1LSEqIlppJO98NwdnAmrlp4rMDOVd553LiK6qlYubG2uMyJBLVL2wfPPbZ2fgFSxzRcnx53eQNUa3ZwI+ZnBWKlN3cMPnxxdf/lua3+DrwkVufb83be/8tZ6dX3Updbvdo/HtcL+jcZrlebg05/47M1rd6XZrFG+oEsIQEFSJZBZHeAP1jBRpATpYwcwhjVuZkbS7jijC+LZUad7Mpv34W5Uks7Mm8XwMUUqDd9/oWTtVNFirZUTg+yKNgCbfgMAltsXLZhJFXRMc2deJM6YxXxz+d7l/bPlybTYb94sPf/K/p3X9tfuVDMby8zF5zP5Maeptk9SNTigqeXFtRppcRHN+oKDkASD+SV2DtTEfZoZLYTnoyDXrDxBSFJxhWCYMSyGYLqSUOtkdp7lqDuU4DU1Y1yO5CepKqET0AjuCkADIb1LGGFCRAy49Rk7gYUIDIwE/4G4HhfjCdQMQ3QYjRK+JgyIDyjF2R3XipEO8NG7SDA2WI8Y7MEoX88WqhHkpl0hIZzy5i7sO4G6M9JlkZc7iTlU/+n89Nfy9ec/8i+9vn1n/yf+0ufuPXjwysFrmfMCO+wiO8bydThVv2Y6KMx7ma3SOsLBTMFNgqXHH5OQ9aciQvjgiIYem5ewQfaYB68fEjraHIIEqDEdcaqXDXpj87kieulwCJtxyJ+3DzFF1ztfAAEDgVlbShePe8BsDzRENCfukmR0Q9aYHZ6yUOiGFjKg5Y5bLUdT7nboRmgGRjKwhmLzySnD2ZocWHVhKLu9xaRPGqRT1EvCkVIbFoGWopRru9vbGk5mRWgBpMvRTmlWnBz1Ho3q+5uF7VznyUSkhqyBx48P2csqxRKsF4CJxe1dv1mpr8sCumj3Nzf2dZfstTsvvvCcjDBnZg+usUMjAHRrycEHN3Z2N6/LUdrd3/nyl7/McH120Wk2mbVGD957lC8cDjt9Outk0K8rBDSfhaG4Hy1LWJ3LnNAhV19i23dbN07OLm5c3zk5bSPL53pZKigznZ4eHo0HIe1WGw0aOTU6Ai1Ivc0myL18/JhabDIZrm7fvHnt2sdFjeTPzzpTAQJ1mBIyAhJswWLarciVqmPhI0kjLSTgXf3FmgWltjQAdiVwWnOAslKdgMYVwAR6ONsWAPNNgEgHHYilBRw+05GgCXbSAOKadKUzoucCwSsQjCAfSGbZQf2VKdXlTl2B2tUlv9s/q2f9br/8fY49ff77fzY8E6QKIpJAtK9EIZ9Ib4QsZPgYEFQOQpC0CpNqAhxJg19NwupuBkM8QTnEkXi7IFgB/LQ0u2G8MbNxfngvY2msYfSqpgitZZWqQCqtmkugfHEyXVZ0ZonWVe6wJOpQv7QVQOliUgI3I8Yh3Q3aRDxOHI7LTXd6y6RYBxF6urmzd1i9i1Dj1U4Mg34dyjdJYU0Za0pKGrbbxJLY3IC0WAl6iAWk8quSzSmsitSEGhfx2LwzOR0CmV20T2IHXsp7zSlySOhTBlbrqXDhzCZnRydVvZGK5Uf37wlERBQx1FypvLG1I22BC/bJ8dHDR4fnHaKwMYHfkUcLtcCGlSQKbSw6nqTZFwtiqkPIi9lIVjKGzhBxzDRyyr2o7R5TJ5stJCffibRUWOPi7Fy28WJxfXtj0wy/8ZUvNzeaBy++QDkmzPY6F08eP2q1NrQuMANSHsyrUh6EqVu37qzJfK41TjOnw+7Is5QplHiRu1TAs7xYG+827jReu9nbm+xv3MwPK4Pu2LjNnvUIaFcnBORovpPNXVycVyvCUioRAshoTLY67yNxg5NT3m96AU+6wpNM5wQyy6GeHugKaEJAY+GtTpBoK5fi88Pf5k7+CfOJE1JAukRrWprqE5NZtz3tcnA/Lj+8++k7H/3u78nc3cpkLzL990azdxfn/dqmu6kzGozWOq/oc0S3h/M2ApSRD38RMMVQi+4o9afG4HiyGM/YmZ3FFhLEmyroBFQn/sLtj7obC6YbFYwowSwrFgingxZeiAkjQStINlEOA9E0YT6xFF9iRDhpfLXsoN7P6ZLVxAYp83MSgg08LgkRJwRVw5b7FVTNn8pMkuzi0bETdqaI5SVqLgsV8VR6ceYzFQl45jkQzl1DVHULg1ob5wsE4cHl+K1CcX79u2792OYP/fzf/Oobf/c3P9z4UOEyP5pyZy62t1r983FuVtjaOBCURk52q9iCAAYar/TSdOgf9BHwaypckHbiBWMqVreJfRuM87maNAQpUWIvn2SHdEJMCubvFoYR7beDLqyoanDfBEEIEPgxzhQJHwxYBSu/MTtL9JGgpW4zt8eQ4wIPFtpBM76s9sXBqzCh2oYcJJwEJankWxvVqoSiiFYsbbXqe1ub8houp30hI1Llpr2TSTtOr+rNvbteKmwIihT8OBzN6g3BVbVscTThQ+0Ob1+rKRWpLgbCIiIyCmSct9/JvkeaFLVJpG80qrvrG0hmFPuWQ1RiZN74ju/+rudeelEghgCOB/ce/vYXvri3swUFeLI2ms2plCEINh0fn52ud7e2d3ez2R1EimA4VAZoPEYE7tx9Uf4Syb7bG21I223mcAJOX/Y5VrBjFTyKRSz5rN0eDbrON9F19ufmAbWYmi7DgRxSFcznEjKymj7Ra4/8h6EDoGebZYxFtRBXa5tAOcA8rWVA/2pRA2BicwR1S7vpkkCPWD+wFXdAVlYnrgBidd77Pj0OalLUDDgYjIvc0JAoAIG1ge2OROHeGCgxFPNLyJYGkAb5vtslYPq2g+8/4R99P8iNCYk3ooEXL+VaLsvYjGr4dBRENjEqrCt4sFHhaKYodiFFbFdf8QAjh9YRtRz0IH7xgkiBPawkEMkbBmt+ukXZ3Knaj2HSC04JXyJjVbgfZck9sGkPIN3HzUIA9fQgx6s7X62aJ8WFIUGnVTCyFTUKxpluuxqh41fc14uk8cdkxQukTztJoY6CCvx9ed8kFo2mucm8PJkXMY8wQmCArHIzKKmvrJAD9Y6lxW7sbJfWG5kadTUnGxB26clgUr0nzkJsZL0Mhxuad7kGqYbDIwqEHIO6kpCKow5H4hvRfsaHd+699/Y795mOXWvw0gZoolq/E5kRROukd4DxKjOJFIeHW2BUvOjTLWr3+p1loYgm9nuqVMiKya9zDql9wXKeyfRKI2GJg6797MHB9c99/jcuOu0M92qhcPv2LWVG3n3vHYHTPKxUgBJci5JHnKcj5ZbGJ53URHmtfdhvVRuleX2NBrssNlSSOj3Ru2+rvt2I4s5FipEYOohvQW0m33jJLctRVOlTbKlSKejyyEoi/RmlEFCtikV+0WWjj1Z36FqOfmLSEtdZVKw7CMEPEu4o9ILRie+tRmy2kxZL2gpIEZolg3g0GeREs1WX/WX7bPx4XOg1D4rNa+VP//AfzrTmGc1Hhm8uFxeKT1d2apnWRqavoxEzDJocAw0YSoiNlQXzC1EMrkfd0EXU0tfXUPYsO3GKrjL/sdYxOEhA5geqDL++rsSA6Bc9DTWQYzjszzgabHBCcIBYvASDVyvoNQFn+gv8WkHqasdpYRJN5yfgX50WF8YJCU0SOvrmiJfwQQ809+KzQrZIp7JEW9p4J4ZK6n1+nOECrOr8lQYTZ6UwsEB0fJA2HsxwppK+FJvM/DTTvLbxqed+b6mxv3HwhZ96u5pZP9i71j8bjjoaxGe31mvD86NasWE+dXxljAj71zJvyk2TNIYYw7dtRpy2+MGOT3MQR7zEt27pHULScDgkMVvi2M40V2E0Dzhxl5Bz4ud40XiD1YWhAcfkgJ3g6GN2ozjNyRHaHCvCWWYZo5dCtBREBYjcl1I0F6It5/3FrO9HAjvFuagy+hpzU7UunVdEh0RDpW+yzYr+NtPFaDAd9hgFRXv0hoq9dQu5BYOTN+OIna2NRD8Vq7Mig3WtURpNlFgxWjGSo25XSQB/p6fHYpuZnE9OZkdPHtZq5aYgfMFXAkZmsL7Q2Nlod/pylVqN9Z1rERyH2mDZSlZd9Iib3ScnTyLOJFrxEn2X0YZzPCaZRQj3Ynpw/VqtVh8nz5fZoNc+eXI8HPTqtfVqvc5dyBeml0m303n7LVYOEQ2Xh3TcfFbTUrSC16eGVbMLRtWcsJxyEomCnrOzyT6iNyDH0aMiQdX7FCEzkNjw+9Y2lim2q5X75j74C/fn6nhI3AEYrv4mVw43ZNDzZ4ASAleAcLqJZQ64c1H6sNB2ZWTH8YQPiKRgDLxGzIngED5O9396bewAl9UN4w7//W5Yi/jIokpM5dmsrH4EAZkMExzaxpAVYcxYABAPS2AIHwHuCE98hhxK5gfq0f/Di9Ang/sC65A3w0wIuH2FAvFpzuMHtiZeJEdCIwgDgNB8bXT5HHnqUk1jAbrC22nkCzxAvYu4c5jM0SZGtSAwRhe3TQiJaZG9A8WMJy3rs89Es2Ip49Ghncfmq0uSqh1vFo1KXAWPKaaY7HCcE9An6KLbM7LIWvY6Hp5luimu76yXSjs7B3vFrSaqT0aeKXTOZnR63iOaMhfGbSKViD3TNQbLjtSbzguN4ovPvRCQRc49v4iu8ctLebfn7S4NWPYub7CoKE2F5SxQIjnETR77t3Jxkm1AjnmGbAwL7PKh77h5mNOZMVB/hMS90b7sdDSR0tC5iJ6+1VK12257U/NOMD09PnPSjdu37j2+L4ry5PgJJ7c+SG57+PiwVW8w44pilFDP+9usVoKOnF6EzNto1ZatZRcprVT4rSYggI5eLSyq484smxcBzPg0LmqHqOFsEPpwPgEGhg5Jj7K6UAAB7hG4dK4sPO9Th6PXG+VpWuXo8RuKWgInhXaiXNRsQYO3LrFcqAc9JJooJ4rO1gBHMT0/A0rYY4vYj+EwO+rPuqeLJ+Paxe7zrVe+82bzI7cyy3Zm0WM+zuiOEe7dwmza7h+e18NBwFnqqYA5GFAiBslwxbZAcpCgof41n5wlINYHeiRITLQkwCiAIrRfYwnWm1A/9uOrO5DFUtp2SN0oihGmoSa2cEV+vFzc6NkWIOptruhS+tU3/CSAOLHHODVgPV2SqM7q13RXPveItrhCUDOj30QAP2gTthCw40IcXcYW96a3U9qrYtiyERmeuOgDsRKLCyro9aZ5pYoZXztvrS37tZfufMcH/uBs8rd/42e/UbncKFYb45NZeZnbaNWPH74nRVzKwzxDCBJEx8+ULFSGavje8WrEadjpbVZ78TZpS19Xb/Xsl9hJ18aOkcU0xLvbEk2ybEQzxNIKWsi4baJJSFCIIXE3B69Yb2LHJkDjaJdEwmSQIvKR9s9BrCTim0O0b6pV33Ktf3nZmWc70h0ITkUlIdWDiYY+ygBsba9vbKpNwlYwkVG0lJTE0jXrKkSB32FA00l3rkabOK1i9P0EC+VafbvabG2dtaMkWmYgUf4y39rcUdKVUwaWQ1L8gIrMLo30EdyfHD7GgM/bpW7vYnt3o1ji8qoErc6WnpyecDy4pNaMxt17N669+IFXIqt4LdM+Pz1rX6DryTGsGJ0SM9nxZMZUbefWnbvb25tsbwhgvliSoRcFfDJLNjw2M4a8lJHB9FigW5+fPMGhw03pbqUS3jzodBYqop2deTSey+xDhqEGqUy9Xq833TMseEn4WdknY91i3i1P+guylWD2ilmmnxMdjz2AnJbZXjp/tdqUkvgxdNl0IFEEq3p1dPXb+1hmgEIcDJiwBfwEA4fqUQfWheLWue2WxQLPuShTWTrOX13itBWSJVrwLY+IsaXtW5/79Og/8F+XPLv8myfGvMaEQJZiYVGR/M2xuGqi5XxKR8RReGnXOs3kBPzjfWmE8eFIkljJE86yjiA+xHUABs7Bf5BJ+0GNAqmDASN04t85n6x6BPbTsFgDCspGaVp1KXo9TGjIH5YiYy0vVJCkOAhzQigmySNKdHG10UCfuONq6uITWsZiBt7FL/6xOXm1PT0Q3NetHfSr5VHyOKjoRLws5OMLnGlKqCS9Ju4RSBqdbWRTClbLN7cb1VYpv7OT4bxcTDpKvvXbXljpCS7Dw9PDeqlSj7bsLOdljtzgGGsKSpwDhmsH1wQZwg3WZgCg/AzifK5XQ7+/ikbWiLBSrm0pIFfUXBlByFCjczTvkOmD8JtgEEHCMb9BMNLmziX5TpyOWtELrdQUZRzZxpw3eioIC6EWmzslano6s5cvys369eu7t+7cZOxS/NkluK9KI4KfKXuijjeaO2J0loMJcyt+rkdwvbqXKW7tFMedi37ETI7rs95M3av2eFxb22SN12DZuiYAWUwUGCEgg3Dxa0Qr5AoFV4RJMtbjhyPW8H6Hkc+c8Q9GmbzIdOqrUCxYLYlY4IgJGidIwRaBRVYUpFmreIjPydQwqA3sE8wj1g2saDw0mpQnx6OHw1Lvxse3P/x9H85+YF1f3v7Z5+q6FUWGMR2V6JuyeEsqjdZUQDGlSKtPUBLMPaw1ipKKbRZgNYzIL2ILkGAn0dMeiGG9iftaiQTPAW/02nRtnIP1skYEM9YTKgKv4ojjFjQkjFA4n+plAZtBGp5tT+FzBbdX0AuQDS39udwMxH6C8NXB990hCaEB1U9B34q4i8FAOjyGJy+ZVOLOiUgKPomcVt9ztQyDy5qEpegtBcbYwgL1J9E/ql8UZE80HY6VScxUip/9N37kdDy/97mj1nRzr7G11pv0B+31reqiMPR+RHnuKOov302RsVcIcdCQ32VbDdNbPx3v0500J0E337fF2vtLGO8jzUL8HFwUBUgnxwQ+vRvRfnW1mXfUFgQn/lHTnXRnGZl7JjpoRW5RkqGoDsSx0eUas8zZZNm7XA4Um2RWbWlBomw8SzNQWYiL316vbW0o4awVt7CsPp1BdENE40772lm3O+2A5Eg3I/es0SjJ0IgXfieGWDBX76zNVAwUr+3urG/vPHjyMEakydeYvh2FW9keT0/OorjAZJi5UDPyfDDq7+3tsJl1B8PGZh30w3R5QWrt9XU+WizWt7dkNK7vbBw9rp2eHc8mEwi0Ji+oVsMsRVk3VNJqtMKcly9u1loYLcSPxKeNDTBJQUdVWNPr6+uirvavHbz3zrtEh3q1Ui2XvEi1XGTnI1afKwE/GO7s7G2sK0CtE82QYF2u11r42XSSVJm0VhD7fcv3bPdqVXy3FuloLMrTn9NOsD2swBJbVYTPusXvgUKUNcgQv/jpWwHEGYk6Jg7qJzeJAViIFfmIs+0DbZFv1CwZ3YH4jM/A89lQVzvv/3TZ79i8wrPzf8dP/+Cvrnrf67uJwYXAbWhRFStfKOc0FliUGUUDa8NaHj71OAPxCI0zafZpTmJWYkOwhEFGFCUlxUFAR/lBciJCK/FdmO+EODMeFqxcXmkYqdIckpPk+LBb0mzZvSIrNl8AcDLbRMtSBwGKCsWh/4q+FUsQqT3annuYZyKCgXExWG+TdOJAtNXAni5xDD+ehY2GRmXn6oiDkAb+hsXMEzD25CTGdEOzFMU6G2RGMkLV/o8jHspOXa6HkDGLmkaUWel8SjFdFhUuztVzZfqw7N72WRvhIXsWhPDmSztb+/KLxHWIpBDfZDrINd4nsC7lDjUbEnBZA81MCZ50T86kHYftOhRoz6RQGCvrbIERjMip+A45lJ7NJ83TFW3u08t6r0j7SjXZT54chRwgI16ScSYrLhoeNpSx1iWxXJZ3L3qC3CNp/vDxcau+9cFXPvzWN97pdHpbzRuV6tblcH788AG7+u0bO5lJMTOYLXq5RSfHDbo2Lxfm1VK2MVA4urkpMyMpi2HkSEHsse5WUb+KTNlK5TKTwfT0hGbQaZ+FpDLTfZDVjgOC4CKcqRfFm6xPgIjWSHgqaQtAirUSfOoHkJPgNqIaLR+5jQuDiU3TXRxyMJj3J2uDaZbcNH7lB164+dkXMzoqLd7ttT8vtrmyp7LzE3W1iESYS8iDK6ujaQs3asywf1Y8M+yzWlz1O3OkRFg47Sg9mW4F2gKXsbQg+fCY9PhUjwzOzZQT/oxgt7An9MuIFwx+HLJTpA8F7w6R0Ztarqst4DL+v9oSeNqPMxJ4Bj+JOUib5Q0AdgSoBhQ7anz+cb4BxlVX4JzYjh/jFFJy/LAmKhukKI8k9EKspW3l/p62tZaIPyaXtdJSVoSaX2aQWJ2qV5qxnnbTFU6i+dm09+VSq/ZH/rXf+/mDN9/6+fvTs3k1UzzpnmsjwsUc8xLkMqQmVqmoVG2UBhOji1E83WLfTzGIeKGnP/nXk71VfFxt6dTYJ3AH0bm6cHWO7yTSIAfp/naCqF1dGcdWZMppSLprUanoj4ZBpmzJWCVWYCDhh8jxXVv25pdtf9LHaEf0v3pxfSNsMWKaqmw5YtwXE2GRa5N+rVwfEFkVDOi1p4A8wo25mHTC5GHlcmfXQaxQ+Bg5p5sWXWafhi2vSKCGUjykVDhfwZe52SdsSVOhlPq7YBD8xSrqKKyhAi42985b7zJZUTIbzawi7RRTY24rBjvkPL6ApwzI+7t719auqy97dnYyGfHzSgW65K8l4kv53b92Y//6HhrIoEPvFZIi3nMgqjSXI3+jCTy5KDxpgTK0ld/yFAFeaC5Wz6PWG4yQskS42ESlPhbYhJTk8L55wSPJ/lxRysvbWtfYAiLfv60g1ZFwUKZ1juUPwIx1CkZ7taW1jxs8BQtUEjl1OEj0U8iIMIeA/m/bAtYQxJCB0uYcR9InwBeXNuFpQFUr1WIrmy0H0H5zpKu9Z3deff3dnvJtj/0HHFgN+Wo47z8vJAazkT5Dj4qyE5H5LdAP5CZo98q2lTkhTdcV3n/LbRi2ML9AJFPFvBWhJl4ZuIPzwMigWCs6YieEceTITLotQLB5hItppUDVwgM15iB8cJIP/RijCgLoivD6mtvVlMYQHPFe8UMsjA905WopV6N1zrMdP6/24zNNSuwIaAqHM0smkwxVCX+dC4oOp28xr97wdNAvjpDpWJTwcVDvGKWnESGlL8Nlfq7U8Ey28Gwqkrs96D1Rtu3kTO8Syp+eQqTOD7766kjBnHALSxcsHexfMwmMmipQIZeEUOk+UnXlMBCYlbnwwYlsJg1Iu2ATGVMQwiDzaeQNc+iFHSvgM1YHAWG8imlJZtwUaHup0k6/e27+hWuleTGDa712T2Ts7VfvquYMMwf9UfFWc3tzv1ys3PrASw/fPjvu9sedtUyXNzefGVWnncmwkhkte7NBCBBrk/IsnD9KMnKdipWSzVvtD87C2e1JBqyqJE8vTwWt2Ikko05HePPF6cmg164hYKBNd3ksFgxIkhA8KQky38DoktxktJGsu8wMjTmk3cAmS48DkgXFfChdPs3nDCUShHvL9iDbvdxa1vYq9Z3ah3/khzL5zmz5oNc+Wqv0S41ZsYzcjZu7RWlgvcmp4hgRa1AIRSYGAmBMIa1XKYXguGOx5pCTTyGJFEAnwARs8iXYruL8zXsEMwdoB4rYiXhffDcWScBzlJxc8eCk/kLwEFndARYEjgSL+JYNOMdj4hgojhOeAifsjEW2hViJ4gQpiCPGkCAXJw7kChYX16RPI3KvODMdcRRcxFWhChutM+3E/Ib04QtQgzT636henK/oouNZa7RhS5GKMoVVaA4msI/KpcS0h4e/cOPWj3zin/3YTqv1d/78L69VtgrZcme2kMO2tpQrY0zhh/IEhiv/pJdO77ga09PPdNzLxHwkYI5Rxd4z2vn0zHiR9Isd5yRYSd8J0PFLuMriDeMzrkn0J+60mhT2tJiAmCn/LkqRE3mpfJUUf3Fjw8v8JEvxvexJxROhkl8bVXOS/fPr9fxGNOyuFab55ZipCHmoUQEEUStAOetPBzOsd9S76LXPOr22ZttRjCJcQTnxSWISzN5lXUn56cn52ePDo2Jzq93T23mtsbHdaK0dPT58EjXoLj7wgeeY+qwF+zXHGxW0XqkohsOO9cJzz33i05/gqPq5v/ffKPWq3M2TJ0elQePlV19FPcjQSJAgDyKV8u9ykWr1aqtW39ncMusoADcQl61gaZFdSpiJXBmLnZvMYMFgyM18wuXNHSTamQTAAs6Zi+uX1neiP2u9ziupug+hheYtfANS8Fevb25sbqsOtw5LDp+cI1wKjmyvtzbpG4zUqxUy1TH7wWAcAI6WZfUZy3MF7KvdWLhYtDjf8ocJBZ1Lh9Jix08J7NNNrsDa66H1q8V+eptY9m/uX+054uZQEP0UmeVlaPHsb3P6pvr9fhRfo9rBM9D5tjt8+4GEqd9++B9y5Js4mU5c3cQ0wRYypwGEty54cG4cpI9QGEz0anLMDGbpSExRmpZnk4bLhqIcxmQnhx9s5elyKTSIZQipk9aQ+CdcC4YqsyQkFMyDEoaTxQA8LlfgGuSrGI2GQnZJgoqmcp9EwVQzFRboUKhC+g8vo5GgJomUGFYMDE6SkFZDi+NXe09nxnMdsa3kidWLhPcYMUbd3VpZDBOD9fLfTIdqHs46mRn/rVRARiuBu0pqUA/uHPDC6hPMFHMBg05PpIQGe+t2iQ6K2sh5pZcy2LBvyZ995813t/d2tQ8yJqZqmqs0u8fHR95A+Q7eVlTBkIBy4Gq3BxmiXLFuSqKhKII0q8gxDYEmlOFCVpKWSfOTz4hyHTM3hYHe/JtxYGaGYt+jpMSPJ5CwyYtbb3Uu2scXJx/65Idr8g3rm4ePjm9fm2ytH6gskG9c/9RHf09u+sXctHX4br9SqO+vv1zN9srZxuj8ctifRLxnpSqwHb9R1LA3E+1cmoVXd6k3i0K1GH5anazkiYxKQecQ/Kh9dsoWXaVqb7amihYkY65phKEAgKJcq66LN/Ve4qp5AyLYStZlrI3lSrKU6ldUKe+lLV/wMtx+OFm7EGnVX7tYNqe7r2y9/OmblVf3J703C43LfGVeS5xKqUXJRaBRE2RxAySDQHy8xohTbF8goD1UZzScCZ9R+H5MQQ+TFEK9Yrohd5tRvEJSOquqX5IwGTQhEAfMGSgRKDRgiB4WFaLRygYE3LxBuChCGg0ITYzwiri4Nt7z6RaiqVWLg6vPpz/E1wS2/gkCFcfjewLvtIMB2dwvYWjspK/pRF84klwVAdKmXNOmiEgxdnjkjbxFqhYTYnMMHpWC61gH+pQfT9X2kixRkSyxHE7Ol5NBNj+6cfOlw8c/dVD72O0/+OLgr/61eX96Z+f50fFQ2WNRuh4RDNgTzXT0ObWuZiEN+9krvW8nhh6Djy32YlK/ZUtcNI4EOQmN2gzFxKe1MK/xeyxQnBF3skj27Tl/JdaROqxaeBfiJyZniUVM72u9TKabWetc5voCwfUWrFSm4tapA5qCtmr5ZnlZIvnJ/5F71hYokivV1ivrFWnP46w6LZN+e6LQDKYbAVSSEDUAjjdRM3U5U15jwCBUr6qvfrb4+puaHGzfuHv/waP+YHR75/rm5hZ/gFpyh0eH4iZu3z6AykhKL9vjhGqtb2g2eP1gT12OD33oQzRRqBQGwHzxa1/9GtelmrJnR8fvfuMdoZqIRpTVyuVa1aaZVvRAHrC0qFDsZ/PGeuPeg0df/vrXnhwet/sD8SiMXoKwKM1sV1zOFB4iKIXYrWRm9Q5PCuv7yoOUGhuNbHHQHZhJ+u3u1vZ7772nvGZza0+mIrqty0t/NN85aLJgFhijkWfvgGczgxqNg7FowSljxdIW1NkhPCJ+SEzaoq1+9hmsIsxTIRTHRXGZL5A4uNdKpYbJbh43Cck2wMFzY9ljA3bxiGjn4qx4aNwrsrqXJIPpZIxA96Tvq3+QyVYv1bjLhSc/wRL0ezrMAK8gAulyn6vRr1hmOvzf/WP1IpAj7gnh4ZkRkgjXLplX2J4qGgdgCQFSYgdC9Y/3NbGr/ZiUtLncW6dNCiyVDK0JqmoSPcMdbd6czi/Kwd3Mkp73zqf5CU0plWssOqVyXcKJg06QyUkIY1jljKSDr9qkh5V1OuWT0IuDucaauo+RxA3XIqMNi2EO0R3WcQcDQ60B63Gqi5SGfSnO3mDsr4bhTJuHrl7AgPNTtCf4Qcw6klTJzyuKxVTGwyrXC199I4TDxayreowU1WlermmQ8rWN3evVbOmdN75+Y+8AN1bnkM2NTEo3Jgir0tY+78xOzlQxnD05VhVjY1t7zvrJ6emT49PoIMS9PRi3iuWdne1ytQQ+WMJ2tzf5fmejmhPQQ+PUJZ347TcQ5Q9P9RczGdtks9FcjGUx6McsgUc8hOi+YBEinhfzDa3PGvU6PG/WW/3uIHKkF5e/+N/80t6tGzkBy8uyHBLGGHUzMuNa75SAoFLD3kbx9sUp6pTfWb+mhJ35bVQYBtaiVoeA9wyb05bl65y1Ya90XvU3Bb3W9jbDeDCbnn7ty7IgUBMUiWhWjtA9RTW6kqLMWuhGCboDBJH8qaZC2sJQ9FHFtejhYGUCbfUn7gsBVX9AXuJ4PtSWmFt9nOmdzR4Ps8elzYyM3hc0TnihkSm0F5M3LlvzqcYiSgBIZ7+ch1cj6lcymZiW2XIckxlBa0okM606T3tUiq+wrdEgagvSC1OGKCKNCwSZD4vKCtATYiduSv9yJv9DBJVyIDIiat4nZE8gWvpqJ+wVQuWkqScC4DMYdizLFeqGKOn/xE297+pZXtkTPS99jac/3Q+GGtwkqI0PI3MAqCMdiVLFKGNbXWJ4sZc2UB1cO1gvl7nzsHf/hwZvCIAlmNWyEsSF/MDvAehUYCCOGvBGrZhDnTknM3OqbIRhkCy6w8Vbu9d2zw9/eXM782/+2X/zb/xffvoLP//mK9sfHhxPNgvNiDxaTsr1Olig70kkhFbGgloacxqmZ8UD4WN8Xb3t1Q8BFN40FiC9hFkiafowKSSlFYGOl7cK3svqpjsHtYmyGTGB/o8TQvgJ/DH+EIrSUngnGxMkU3Pncu18kTm9zHVyJVZX7Tkvi/kyQG9yV+BIolPccZxb9MV61C8nvKZbZaEfA4XuZNqYoL1bB/fuj/P5zcp6bfpwLVuti6XhPDNF42EPPWQlI5ETGrCnw8Mn90/k96Cq+QePHxfLOg9WT8/bh4+fjPpng855VOGoKqFVgNWap8iCeOGFF65fPzh8+EiduM9+9nu//JWvPLj/6OTJiX6CP/tTP33/vQf8Vru7eyrIy0raaG2JtjjY2VfwjsmaZeDs9IILSvb/a699+Fd+/XP05y99+aumkvn61VdfNTBOaFQRJ377vXsf+8hHHYz2LRl9sC/3D/YjGKVW+9wvf+7o6Kg0WX7w5t3JZV496e5o1tosH9y6Xa61vvz199qDiVq8ZeuELifua5IDcENQBe9WPJY+gNGirX5Kn2kpnJe2+BXXDdk14DH9trrYjRBzKho+5IZMNrGtAMlDUXsnuzZxhSCL9mkjSbJ0IkxVvGwwn3W0kWi3jwGhQDOlhkslldw7BHD+w5yq/hHTsQLTq4H9Y/8nDDVXrxZjji3RuSBPUbUdvRLBigeLmsNRTRkyCZXj5BBKQlLx+hwuKJpZMmBH/ZpeOkhCnJQm0L+xA8FirpAasYPhe/MMVks+BSZH7HAp9BIUeu5oMjnvDM6++tYw2luuEQ/3FHDTkZcvWd2loChXN3TbKKXk5uKBSUEx33DMkAzSYeseHtNYmcDMb27xW9oc8q8PO/A2HEJhUg7aKaJHO1pR4fntjbxCT61Gab01fPJken4xGvZIIzU3VoVjEsXnZOt4JVGnJIyLk3PPjEDCojymNax0hGMMyRoWtjAcM6qO2/3R0Vlb17FwAOlIqFzSbLa1s3375i0VahTEgXhOBzy5SvS/hbQm0DmmcK7W0lzYV7iqvC0sCthKW0B+UCu0xm8hLZmhYLOLJQ0Ugpk81cRmYyWbJrVybX1zT1zRyYOzXLb48gt3GqWdw4dnw85wvba2U7sz3q60yjuXk4amaJFEN6prWKrQRNIfAuwF14RoSQbIjFu7mzGNllZwsAd0z875etvnOY9kRcCrxdEh8/AtPo0KiwUktJakmRBYYy3Yb2VZOYIBEtTU9KMPK5C43G/VHh49HMxV4Nqsb5c7vdOzroqX3cz+8s5rt1/9xJ3MnUqmeJ6ZfX08PV2SUoo8xyY8MBVsS9hGhvjQIzMUGQ//p44NCDCuP2TDEJYWFZzn0romfLRU3gAjAgLKHUOOl4uZRUEDWAgPfsK0fMZg7cML7Aq7jaQjPCp03yD8VOEVyQ8O69p4aXf4ti3xoRUbfv9vCXTjkU9BOFhyfDXVMSL3inFdXRLPcMWz41eHE5DbD/4bIs3qDmFIiNsGSw6c9Vr22DFCHGAMS/W6vcZUJntVRWkNZBXKo4NYUrMRbHiSnZ8Ox53Ng1fap79Uz/T+0P/0DxaKv/aFv/X2d9/9zPDBeTWf09xnpI7FvLhWbQ3ns0hiSBj3dEjx9Zv7V+N9Ouw0YifEK6UtGK0Rp0Sa1VUBPel1VtfA4kDkROzTjYP1YtUR/YZG5ISa5QAsrBWvJ64dfnQX0/P5WjtbHFari/pGttYsFEscRjKvLpd9dV2VE8kJ6Bi3l8OTRvWynp3Xs9OSqKwZArXgw2CEwrKjde9sLriyvt7SbZkhq9ZSdWS5FqUml7VKfmOzsbmzvYG5trYm2dKHDm4Nxpdf/vqb3/jGNwbMfZ0OjkspoyVXZSbpedBs4AmUS2AkaIEezEPnNK4wHNo8RGWM4eh8fgYCaTCbWpzWCD2Xko/bFxeoK3WZG7pQCGWv2agfPTkTRvP6Rz+unblkSFOJ4D189AR5EcbsPU5OLxCSL33tG/cfH965fbvS2lHOQCmRUk2oVqvS2ho9Pn77/uPNnQc3rl2r++fRYx6gg5vPre8e3HrueTRKgQ/VlOTp6VyMWqFBsThpJTwuAdyKvYXeE0tmQa1WIE8wlsCMFWciRPkhQSdC7r8ww6qEg7KLH0ylM4IBxy0SyActTJdTydLTw2+H0ZTLCS/daTlhiJiM2sPh6XTekywqnI6Hu1xlqmFYkyaKWzeTPkqfS2M15MCu2BJE+ef9vCTIwj/y5q3TLX0glr/j8mDMUZ5LudhZlN3W7VztKYbOECVBcmi3MZRkHAqylIhTDNIXeLAC+ZA1041Dwg5092cl/IhnhTWCVoJ9MCEXSmaNUEPc0UxkLRwSRE2xRJlbd56DNqUC6bmKP9EpR/wMy0jOiZJhSlSF/fCSXdZtccGYMYPhzov02VDWkxCQyMrVAsXgHF/xYzurLb5KWojVj+GDg/gLYUiwRGRGZ4u1TK2cq5RVg2teu5EBuCftidiodlcTUO+qwEW53sqMJxIBhPo+un9IrsQA2ImFWwAZwTdyGlBnUcMjoVKZS8pilK8K4cu7SB8s3nn+OYHKu7vbiAuh1RGRkMRS8bfBLqpyw4QEKRjJinfJ/SyBXoJviCTJ/gyFREl6C9MLAFPQJXAKEccYgzurEyLAPacfYLSlmgwXSyFgl/Mbz9+6d//h+fnRpg5AN8oimUezweiYo6u5vyEtobYclRQTEl437atTICouO5sPuYvITKQnlEIVWzEDGYqTigWdzlDn2IEG3nC+Lw+yaQTBV0leZtWO1aGrSK4E6THVgW4BLcGqRJ5mc0MiQloUfNCzpgrkjbVbzY6LTWX8lofDt4223Fi79qr47Z39T97N72QzTQz7yXLycJE5K9cvM81yZtgN05EMNyoM/QVAY7nqzQUDkqgb1gNxvTMV5ck6qNSgi3OaW3gRABGeDfCeiDtAWeEJYr8aK+MO8AspPaosprqSYbCNrhBTbYv9cZZEMGzwY+wZDwbuZFzDTEnGV+gRd3PPeObvypafPtgpMUdBDtLpz44bWtwifiBZrNBuxYMDlGNa419A/3Q/vVA8fSUix16gcWBNJBZS5eN7YHR4MkLhwK5SlblpscUnmZJK5Q5DkLgynACNRrY7+EJj+/nx8H4+s/X7/ic/Wsz83Bs/+6W7tRuqdir0hhRns3Vpceyly9EwkYmn6BYjSYOMkfzuW2K66awgfwEusYYh28S7xQuanSA1sXYIVAjfcVdbIDMUZaXyB7aIFriC4tzDnGpWl/ylo9xlJ7emNe6i2spv7Zc2drKVBjoFEpjUBBiwCLVKSvRddgaEtBOFCiPTXDX3y7HR0MY0OMsVq48P7z04vK+HeHNnx0N5qiK9lu2toCKsPl+lza3Gwd6WCpONrS0pRNlq65UPf4IGLNxVSazjx4enx8dFkttUotNio7XZUqg5m+90Lzhi2SJkZjJDb2xuyu4VNbezw35WxV/xY5i4u70njKNeoxTktSzRynQ+PWNRo9qaEthEpFpW146Ojvfv3vzu7/2ee/cenJ2eW0AGRYZGhEXNEaTjyclvCnp9dHSqD9zNu89TiQ5u3M2U6tlitT+ey7kvV1uDQdjf8eSt3Wtf/frbl5kTRaqFhTc3jK2PHJE5vHxoB14OdVvR1kRtg3slIIZcVinWKww3wTBWm/0r+wRlKq2h34J8mQL3wYBVH0zJMgJMQ7kNjpX4bjKhxE1AbDrZWoN7oCHnISIxFvOhpKnx8Hw0PJ1Mu2FgCDdST9N0/II4LpFbtEPk2KzQJdBg5XsCSfhioFHAVEBXQjn//HfbAmKfXRn3fN/GHKfcPYqr3L1Gt9UlW59gCtZDeBhw7fGBBIEVvoY2k6hCIlzpZ2fESavNTnyRrVasyHYL6sOnxCxshYKAMSWxNldE5LutiIBy9lKk0tYuXwK/RRQSMnFSwQmItKGkBytKVYn0zfAvxMLFg9w3/vVWJiqUP5K86TceiBBLnmbMOR5tP+AxPmKLX5+OFjxEFxvGuEiJwT7DYWCYcfuFXLpWprqe2djLXJuV2t3SoyfLk+O5vBruw3oDS8mUa8IYz8670G6qAgS5ZRXuqNmPXLPI7Y+AKap/tLQxgAg7k62OcK69+OKL2GrwUWUmizkh35iCBkD0Z2GXWDVYw6wDnURvMb+raYgfi1bF/wvZ4VDMkG4Ro81KREtEPoXUF1oMyyRJhAK4Vu6rP90jV5XUoFYuQDnI/GUrM6ptVG9kFJPsFSbt3Gb9VnY8mHTW1Lcsl1prC+ry0huaaWEm7iQ8CdhW2AkSDommnctQWozGJw+4vAVZDdTTSF4DooviBNrRwxPTbKUom7FkuBd/RxwIKpwQ0hgDnCRVAQ40nQRGerVGMxwhH1m3hWZuVux1ZoeD8tG1lyuvfuL52vPbGdaTveyElW96ls10K/XoMBjJNP1hcCpqGlYfHmUxu/TtZO3OlvnldFs1WbSH+YxTJJikT9wWBCHx/jU0DCa4psr2dhKeB5UPaSE4CNBdOTeFrhMEmVhJFNKNJJIH0KTqV5gZ9hwR0e6FDZM/DMPtruAtxEEvmeDWTsLEq5+cs0Iwv8ZpYCR9rr7Gy6W/GEscD1UhDTNuuHpAHL560hV2OCce5ef4IU1+YO3VE4k7rLOhYKRbMdbEnQlLwWQZ2DuBA7x58gTzsFSJUAry8rLcKJ3cH2/dBM9nqFuhtl6s7H3//+gH3/3Knz7pHGUyIg7KcwghZE6F79lAMJ5TE7b6jO3pAGIino3mm1+MMEYb1DpmnRTrX811V+N3aDWh6S2cRaCPk21hmLCu0gR0pVouIp5FqPxiGP16dQy8hAl6hvQLa7NqId9cb27ulta3suUK0RsAdx8/rGUXrcJiu5jZKIv2m87H7VG3Lcw5L110OhKEKGyBOBucWiTCWOa/QoKUZzVs18rTAnqPEQm1p7FxNqlsc21/O1xtSi3glJWqkhdqx29tbsDgo8ePIu6PxRTljYxcwW+aGkQcNa0X3UPi3n33HY4YuA+JEAqv2FpvJKIx0wzUEeGqvlpAAaSMyZ1u18EgHWtrjSZHZ+704vR81Lv70gvXr9146eUPkODFLAvmUrODZwr7FFf15GS0t1f66J0X9q7dGcwNs6lMi/JYpyo89waCFsE50zo+a2AGY3UePrrvcavWLOH3Ct9FyGepfw6sWoFuAGQY5Vb/+tVxJ4cIHqB/JZF5q7TcKXjQUSsudGXKtQ6NrDkZIKw0lDCkLQA/gNnlHrc6OYJ05Byg8OGOFiyYEgB9mEpBrX5FiIiWjBIES8leeTSnUAWXqiqoI0LqucIIVCq5hdw/IUw8wdiSaG4/oat//zFtbr16k+RyNq3cwJWFVJ8IteXVJ3p4R0NzImyK021PhxGDCZAPcd+MmlP7/g9co8cEt442B2YzqjGwPTMimFhEb60QzjysiBmAKSe4flG0L5YVVZmtI87VH44JVJVKUf8OqWbRqF2GcnhzmX1YfmMKksEjwrFshpcWOvRastiK6cZJ72O64SNOWm98hmYfW+g1UgtUs7B+IMchgzb/WHCmoHxugSGds0qdgu3dTFHwks4EPQDH/gyO1/g6ZRdFtd019ep0UgjykKcvXA6kA5HR4qBmcVwk4m7ZF6oUZcdMmo4IUMWKAzjt7rjShW4pPELldLzeqHqL8KSKiJ6E0r+oyIUv4MAxnUmqkGLIO75dvZbPldnC6cGeGyldVPTL3FBGxZJHOuo/VgrNfm/a6Wpp321Wbty99ZHXX9zCmNemdeJgfl4jOFCoOVzlNil/XSqoRuld+LHIJyrWRmtTDHMZhjNSuDCTabdzYVBOip5qihIEP2DE1CIwwn6IOcF4AQ+elrawTRlaosTxGYVKAroou9RGVb5s7A98wtnaJa/c2ejRSfdBaWf+we+8deMTNzM7bMQnmeKiM7m/LK0Va8ipwCdVtxXM9xS2ixLmR3bMAGBV+wSHR01irshuRoTqMDperFKQ8Wi+Q6HYgJi9DFAbuxGlocG3ZMNmoXIgMNNbXr1HgE/Yb4BJyGxUxTB0mnSYnD5pxqEcBzyS5ozK1XHbkFkDO3wm2F3NyLd9el5gWNoS1TI2ulkaYczp1Q4ke3aac2P8q3eIL6HD+y/haizA6oT4Jwhj/EvsSS5UTC2Jd1H/PQJjRBqSQU1FZLMtWFujKdQom3rmwdWyREHJs2M5aa2qcmVsFOf16nq28GjU/tVKtvvH/9Tv+4k/90uHb5/fat5FTNamiFiE+WxWN4LiGtk3N/MSmBmjef9m4AZseWLmzWGcFv+nj6DeT0+OH9M5DvAzJLSLs9kmRHPjvkNvVMj3lwslpnoCMphyULRCQQXz7O6WhLzIx6uKf1yMJ22tv0FIbnBcL+c3CtkGIayvqMbZ+OJ40rsQrxLWbMQw6FYYXJmVOJ8Obu7tXNvNSM5hwxkvNSpoK5lx3rk4PkIxxEZEsuxGM4S+0SingzYbsVz87qTTG4m43Nlav36wKwhZnMx13ZKu30IIpDNhcjdv3tzZXm806tgQzDBPvLNDLXxDTl3W6k1wR+XFFPvDiXhpvi+PEwXWPT/v9folFIOwvjYvNWoXvX7/pJ+rlG7evOW2h49P8sXydmNdtNbG5p4S9C+/+pF8qbp/7drrn/6OqgaIw/njR0fV9W3VzkVdhfVoeSm+8rd+4/PP3b0D6xUesbb333v73ffe3mi2pDCF+hv0kHk/bU8XLtYqATGXWSg9wS3izyJeUQRnpgWNJU6UwfXopCRmTknmijkyF8nr0aOWRkiDwOwBiOti8yODc6QfRIgqrctTIpxcugrAh4Q6Q8n0LeVqhQpVUgZF9NIpCLwqrxeKm9n8uvJ8kQgaIjLJ2WjdEJa4DySJA8mS518EQA5GiBWO2okfv21b/fpth//hB5Jd2p3hd6j4UZxyGTmTiv4GDsB7b+3tsKUgIQlt0pEVpsOFmI6rQdlxq0AVokeIDmtCFuKU4LU5zhj3zKYYnagmXMnVof2Tk5PJ9EgbS8yprHaEoCrMf65kjZKtMZE+HDEy7BnfdQuTTlgMHA6pCPUm+YQZP4J4nla8SmOKOpfP5i0tMI5rPDGTNmYPUejpW1rJFHdWZCYu5qOW06UWKKi8YJ6CTNLMRhFrK/ZLy36P2bXHBHx2/uT8XPOS6DBQKI0XUzXh7DPYcvQqxupRpIdAYO3/ZFWFk1kRrrJSX4xCN29ev3awJ0HVi7BhTQeepgyec1CSAAUzoGWh/bLJvKTFMTFpThz2djAJWHvdwaQ1zyMt4k+jVDFaRFRkrRHHKgRJWavMcHA5G1mL8s7Wrnahrzz/4VZrs1ZtmVFpM0pXYjANfYPRJgSJE5tylxsD73xxWXbj6kasOXP3UDrxWa97Lp4ZoeNQTZpuLEAQ1zAaUQwjCszkxgyHA47xO6Y6nfIM7wIL6YeovR/GU+lmmaRGZNCx4bTTU0W33+6tnd760Parn3mu9XIzUz0dzY+XuUhG1B4xUG4+YEjDe4nGQjjVd4nef4LHmXOgi9tPBnNt3S5n3aPHWCUej1CwjqHM4aQKaPbwQLS041gAtlFzHyTADvQLyEYenBQ8iqLkEfHHzhz0JvYZHnJh9IziwumIawQ3YdKB0zYXx/1WQJiO+B5onqTt+DX2g8fERMVwwrZsPy4JjEs82A9pPz5XJzpuJzFbR4Il41BxTvp5ddWKxyU7WlwV13qR1XNCS45ZiOocV3GTsWhR7ctNMaESlS664TpLB8KIrFCQRixeZ1m6sXPx4KS5vSbR/eT0SxtNIQxiCl7/o//WH/m7/9mvf/HvvPnyxovNYk2M72aryVAUlC29aHrB+DCvtvT1fT/Ey6TRPz0v4WqcYCLibZ9uwXHT5gRxmCY7TBLhgo81GUZGbyYiBgjBstb4aCKdqE4NjfTPnWuzPPlhfN7vTPrns/7p2rBdmnbX1xa1uV59axIH+lSAUU/JFzlrpiIsS8FiPDIU7tD2ZuPCvKgBUEUB2lx+O1uaLzY7nUFnq/e2vg79QXAH88siE45laC1HeJILSbaoGShWq5vQh177wLtvv0PehYx0sr7sYDnEpZxYZWxVZSqFp3BWxAFxY28OMwDLVrGcL487vT5jtEyJSKNEVy+Xza0NiVX+rBc5eXRyqhSBQHZ6K3wx9Kjm1WRByu3sXpvM1oQ9n170P/u93/+pz3zGaPUDpmdsbK4fXQzWtw8kQdEQDvZ3KxuFl55/4fzsVCn7EQpwcZLPbEwHncmgl2vUc3L3g0GEB48dmBEY+Fqn5HcMvS3MUMFCLF5a8oA5R2AV61AI6YEHaVnX1J7DdBFNC8BCbsXD1He1pSpkEdQbzCDxF4JwvBUAiK9uGM+B+rgvWljhGgxKy1Ofl/9QX1sb4bIcdBos5gobmfxWZm09s2ikMai+gqQGO09IZ+Ui0AmL95le5ync/eP4N8SNtMU8xWOuviUglzNgBsMFSdoI1TyiIRMPNmtwMDAcqXINAR/wh+xAlA1bVhK6EzkwHalhQK4kPdw5hLIwO/Nl6vlHTkGxA/OdQ0wJRZaZYDy+VCIKfAgU8kRARfdFoaS1m4jhYIRSmEyAEjZbkQ/hcYmawyJn01ug+yFTW0MxPKtJs0DebwUSAX/R0yA4d6xaIkJ+4v4em/yYaS9k/VWpy0qedzKmYBgqWVhUsIIb06cy9RojK/fwfK0Die8/OTm6uDBC6Ueo+4jrNxtNb41QSIIZxsl1ZQg/KH0zCp+FCKJldeTLnxx7KAMVb0roneMhtgn3nAAmjdMbER6FDdBhvaOX4uA5v1C7KubEbLCw+Gu3+62KMgkVVgDaaoBM1IGVlVnFcXRQGI+W+3utl1784Guvfnjz9kuZ9kTLBIV8SSbEiSx5MfAn+M90LJ9q2WiAdnPjiAXH1vLLTl+xzX73bMJflVEGK4QEDgLGHmgXJRvTTBexfgchA6odgBH/heQWsxshFXaSWhkwF5gT0oKlKOJrYfW87I+WZ9NSt1DtN+rDT336la1X1jM3aLQn81K/vC64OTfon6/Px5rpiXNcq7KU12lMw+F8OFi06tvgiBWlyhA9wcdPh/1zVtPxkMsv3i7eyFjglueaTyONgeHXwQ4BNoEHXUh6a9LZAiYSAwYwq8pWQekp+T6pvERElD5yf8Vyo85hMgsPKe4V8kXMQLp/fAayrTijb8byD9+eIf4KT1efLrMTg0/Iu/r67F5Pz3Spn8NDHM8M6I1TvH4s9IreuUXY8pJIFOQxmf9XA/QTJiYMznyE85w4MZ3n52ULzqurT2W13n3nZGN/czG8WM6Pd1rlx6efv3awf/TgV/bu/NOf/tHvvvfueNTNVWKNLEqh1/aUlHS9mpFnw/377AAVww5YSTpNnEWr8T5psJbOAK3LCpcNz5+KVlaDXG7gFkT4eX/tsktfK5cv67VCUzfq9Wy9ka1UcsXaYFEctMe97um09yQzOi4v2s3MoJbprxcvG4iEJER8ZtLX7GiqYkmtPO3Ni+JVuBiE48tdmSmeanjzi8FFOIrKZWWZW611862KraKv9UoVGvLU0iALaze2djZJ3iK4BQqRXlrEltQZj6Hr5RdfwFFOnkhnmunvq0C8mmNVQQzzROUGgydHj6U2RK5tnu+2IR9BY2cuwmuV6ltvvTOYTSksPCHWTYUfdrP65oYGblb8wYMHj588yhw/QRhkl0Lm04vzd95559rBLcVr4F1Io8Pp5vaOsCRNzXmdj0+e2CE8I6pyVO5zLXU7xWt7PEQf+/BrjOd/7a/8l932GctS9BiZTnc2G9ubDfuRs2Y+mDrZoSeXjA1xC9pngA8YDAxC7wGjgkzBe8O8GCgR7NlKRx2oMGPA3IiVEwSDGKLv4Wdj4M9XBdygm0HHw6oEM624G6/pOsyP5EhASLAr9JMSViAxBuTniP6QmvzPmsdBNeVIiwjdaL1QWcxUORfSwnUY/n0DoAEF1oQJGi3A0nQRIzyp0Er3DaIW1OwftLnQaXH57zwr3jE2MxAjjc+0pe8xS54fZh+KL/JNyuULoqsp0Wi06dfA4IiUDutcPMW8BU1Kl4bRjRvbG4ejRvRq6g1HKimVFCrd8S6jydFg1FsTFampHP9LfTPVQ6C1eDulANVIrbFC8yGAZowHmxHDyuYg8Za0WKuU5NlYx1pDyWgRbjAj0pOsCNEgVjY2L2mQ/uJYUsq8WULTWO3gu6L26TBBgzBjr0C+AA05VZQmLKiyUxAooBk28IV+27ISGFcC61k5whLEplVizy1HbcdaI19u5ptbi/tPnghQWMtdtHs6DiizCXi5IgyOygl2xehr483dIPO7xtguy1VT4WpGvnzn+JSkzI40n9Xor8BDebtGrSQLCZxEsQjjpq4lOAJglfFAhyNgOI1q51E/1Kc8mnZ3UNHcT3hXvujKwMcFA0b5yUO5Dbc+8OJr16/defGFD6ixo2bA/S+8WStvUBMFW5lkPCMcBFBCVJsq0hPtESuCK1J6mqL0M12Qz+6dqFg1GvXQ60r0jdS3Q1Sm+Nbw7aGskWBqvBEtAU2iixBSKCzM8iQIAiuxiaU1NKTVGoTBPC2aMLxsa9qZnZ71Dvvjo3xjevOFjQ9+8k7l1a3MJk3zaJntZUrClEeD9oDJvFmrzbtSTuFsNkpeRJSBuas1ZVwKLJfm5Kg1ltTbOZ92jtnGyg1slfIaiws9QExAMCt0wHaQdQOB2BQHak6AsD9vFNBji3Nw0yCvPlcaFumCWBKZSMj/Nz/DHM1y4cViJ1AsGXS9cnCUtIHPED7jUenI6nDIZ46ZHkieZMM4ErIwtE+gvbo6Bo9CBvEJJT/oRXw6J3aQjaA7JE5fcd8gGSFaeJuY9tjsxI9ezfdA6vhuafwb54SfPnbS7bwsySQqcDtZiDejXi5XydTFDfTVbptd9FiJwh6VHV87qD84/ts3b/7AyaOf3Hn59/+Jf+tH/vL/9ScePTh55e7B/Xee1Iv70S1SFFxkhaTnIjKCXaJMBzXImoSeFG8QFcNo4NFEIg0pyI+3CtrLzCWewIACf8MnkOY4zFMj3bmSzVmFOGbv0Vp2IDqQ5XJrO6fmk8imRnNZLlPOWDUJ+6dnRMnO4Pzh5ei0lRvUq7P14rSVXTRLmarKOtg5oxwzGy3MEmgusRITydbET8kNMybboB2crNqMgZLJdqid09FUAXbJEKHBjccX56dSAZu16u616+V6VX8k5fCeXDzY2ulU6+sAy5tsbnEGv1LI33v04PHZ+ZEAuK2tBq2HGY4Er67U2/fuM+lohOCGilCah1qztXdTJ8CbYWBLaRGanwpDQYOOzk6jJ4ImMaVSW8Tmvfe8qvwkGR2lmrp14/uPHt99/pXRgnGJfttWWmf3YD8o3vLyQkbyk9Nac/P45Gy+LBBaL85OR4MeWURI5PW97UqZRXAIrbc3m5T4Xme01dpo1croQn7cX+OyFj2jiJzIIVAb/Szya0pfs5trrSwwj+UXz4ewOEdBmfDgUwguTCRMic8dRAUoNGUxUk4MUC358dRkW2uIKVBwMCCaoByQGjJSEowR6MsRTsCRvhSkqkw2p134q3MR/uHGbM6IqkKAHkpLIx3LcgdPwhkwdOgfLjWEPrnzUmo450VWM2PXxtPVhRtE76tarbqJRibYDfYXuBPnQBX/BOCmLY3xm19XB+GYbSXO24+vaWzp8ArP8Hk3j+IwYKJSyrXmmYtsrrcWI4HnXjmR2aVpbw2G0b6QREab70rIXEx0oCa1jVVqmKliWjGrZhz/Fh68f+tFwQWhwtJLvXeOwWZd1lyh3KrWNPwpg0JXRfwQaBt0mAeSTiEeImoQMuUHmJLdzttqteCg5CPJsWAF6X/++RciAK8oBXwqHMBLlJV0y+cV8VBsjLyJYyFpUY1iNOYedJ/+jIgTllvaokmQO2/psYusWijYCmnoMkfHXYWiCOw3IzHXURV7pgcvcT4Z6oqzmXyXUqa8nik0P/j7//B/8L0/FEbxxfJnfuZn/oP/8P+wI4p4Nn/xueenmhL1uoIxtSjSFSEEtOlEt4atRqtRbhy994CY/Pjho0G/jecpErW50dxqNDCCar1B0MZROVR5i2XlI5GkaeFrk+VopjxtOUfmh4AkN8VvVGZs96NBYKMqYnWiEshwnlG59rOf/sN3br5wsHeDW2yg3WenS17e3WiEdyVCb0LZCwprqgmCWhpXsiq+FtluhMep03/06PHhw1F7sFVcL4hFjGjSsDuE/Awi1tjctPsh/bCqJ0sCgh7IDKbH5MZSZLiHeUL4e9SaFIk6UY1AKTO5pYpuz0CcaKhO5vhR743WrczO6xsv7d/ZvV1r3K5n1slvh5nsIJMdRm8Ldubs2kahYU2jN3qO8AqbYuQy50I9UtxGOZLOA/16kQx+eZngzM4VFmcNeUl2QcCDqQbvS7gD9pN9JNAh2BOmG6zYnATvI0BwSMAIsnoo86oKYvRIPjQNS0hIqnMCl+g99nsGLJitlTYJw8BFfpkK+CqUL6FYPDBED6w1oguUvgBS6C/cZivD84AjThTRYzEU6B2fZjY4JfxONruQkljsEisiRXsTjCr4KK4b7+aJ8Y6mOZMXaRQkIv5i4iO+l5xEe4+HumvMBM5idKEGqN2VTgyaQBGO+VmxbYmCpayEmqjx1YVIqSfILOCEkVVwgUIPDKJuJ0R+fR2RfFupU0mXhRc+/WP/xsf+2p/9+b/3xd/+8AufOL/fr67NayUeLRxjXio01fZH2dR6lyw9vxxG+rziapQRtkPRDIWimCkCjkUXvuQz9BzYWpRTF24OvtBoWpSCNYQAcDPw+PZ0/llcdkntleqi0RQjOS9Uyhvr2hXo48H9r1FuxA0NT08ffn05Oc0vxgcblYNGcT48Y1gt1YvVbWr8toUevPuexV30CYdLURHlGosXA9FyOOQfUQ2u1Mw0tAwTVKFWnTd6cnjBAKP8Rb5Mki7OTvsjvX7lChdbHG+nDM7dXodWMZmen6tLMzu4GYUv8uXqzRduXpx0BlCzc4EEqmaDTyiXUauX33v82KzXW5uj6eU79x4zRG+si0MW0dXMy10+7+xu7lAVtHJprW8qkMdsNrl3X74Ns9vrL77487/4S+++dx+fnhydn3eHL3/4463ta5hlR5+0fOELX3rj9p273PMPHt0XiSVRuF5rdPuDt956l/GSi9pgv/hbn9/YWBdlqarlX/jP/9zB3vZmq1bKSsHKazaMqDZv1jZaLW3C6eX0ImX8mCb7rGghPwTwLnrddrnerBarKJX5D0IRNufIRiWnwrngS4gq+DIrwXr5rum+ImkAFY9jOZevC19TgSTYHsAE9vAy7BwogOuAReSBBIhH3g77qr5+kaYAnwKcw2PiE6KJU3FeDCyoEvIRxm/yc4zTfwmte9nLPku1ceHcS/bDZYMeLEMjJGpswbk8fMjNioSkBzz9WD3u2eeK764+A4ufngYxA58d+pYtiZ8kYgRBYxSRO+qyR5sx1o55gXykgbruNz2FQ3J6TW6Ts5QxyJVbne7RbD4aT7o5DoWy3BQ1LULYlWVzmZeWvuFuqkzXmnv5wgYdRQYL6i0MAUaZ9TBUSH3ggwr/ZUbZb2zSBBpHoFlUAonztM8DfwL8IubF4sHTqLrBbswkLnhZnnqQILSPsG6SOEvArsvhAL7op1VvXV8TgUOIgvhijeHwlpme8t8t01xPp6R3WpTw9yV1JkSciHBVFzhJI0wBBUbyUPwEWAuiW6tWmVDN3j/xr/5r33j8BHR+9Y03vvTVr4PiV195aVtJZMIM7QEnmYxZhDgLWZ+4e0l/aCbSrqI5FZstuAJCUedpgdVaIaKAH6SHlq6eRqupZa5hMXUipICIxcZUlAoNLiYGlW5n1r+4wKq2Wjc/+eFPfezDnxp0hqpRzwdZ8s2aIvqaTvIZ9+B5HUC4BUA0g3xV0YcFYd5qZc7EsT6a3h8a6lgheLU86qXsRJqQk4lp7DQmzyQDQsSfBQEicVvE8gWYgas11K82iqzmqKKk7QZbSmIBM4kd/WlnNB5Go0f1ZfVtWk6G5ZMPf/bGjddarZdukHUza+3MZfty2RktOsvsGIvRaDnq0SYmpRRYoIw15joIQckztY8cmkwdnKbh656yYQl2DDCPcIwQfo01+EoSPoPTJHQACSsl1UxavPRa/vGHKoQkESzTi9pxsxC8w/ULEcPZAs4Q/viMg/6CN8dfzMPVCA0glBxzHNJHcL0YwQrxAkqhYRpMsFdfVzj9bMf43SnNZ+CrK43JFSSfeJX4DDqyYvDpd7sB3RRagqJHxQmh6ccbEQRiYPEtnRRCpR1k0PvYVoQixpgeGYMLCmoj4qlxB09iEtamS4s2zt/ORSMDfzQYNRioEuEVHfVm79RqezlrLjxg74Xv+SMfKDYffu23vvr6C6/3H5wBKZUdoGO33WWAKVW2okkQCGI6iQKkITMk0BJiGQ0yjIHWxAUYtjmYR+Ycy8GTWk4nFwYvZCJs/wpa9VW+yFwOHa/V8utbpa3dwqbypDUKBFrUk/At4Pfo+OzoqHN2uhgcVdZOW8XJRiO/XVmUF1KNzvOZiYisa9d3N7c3J4OxyM/M2gWWIHKAxWkw7C/XxgXxz3BdOLhQaiOR7ZOLVjE1QVlFWq+ishxTOkHl1KRsNesyiHb3dxrN9U63x15EXBDeyrSLxRw/eaRW1d7B7sH1fZRt2B2cHB1CNyS0JB6FQng566iA6+XLlabAKD6nCCzpsHVvbYcBF1Lw0MtT6ubHMEEVW8WCrJNq1OrLHh2f6nf0+sc/ub+7++jwUPvCd959dONmACZD87Xr1ze3to6Pj/r9gRWQLsxezccXPGm2VBwTR3/w4L6Uwu2tdeTxN7/024/v3Z8Ouy3uMeVk89lRb0o5Pj89pTg0alVZmyj/okgFLeej2fAEGwP8PMQi2ixfTtlAHd10cERsRLEwmgOtIG9JaLTSgI10KmsEKqQQW4RZ3K3JYdir4hNABG8IATFtYdBMJJ4/EiMQh85T7LQV9sWNQfI3twT7jsYdwjQEyp9uFAYBRCRQgjS3RW8tquBCBDKyoksRHY3V+RMDE2pHHIhcgRXOJHRdPWmFQs8e+Tu+euGkniT0cm0aQzo56ArMTbI/cA0C45Gc1hqjBCUKe7t/gf9CRoxS4DvbWweUTfapzU0y79qwd5ov1qWQ00s6Gs1dDNTSqURr5zJGLomBJVJd0bLYQucwLoFdhv7pGI0oRAprJIBGfJsXFmeESkXEAU4cPaPMRMC01BTqquAgNTtK5VYrEgQEx8MQ08k5INOcGORyCorRj3Sk51ON0K+IwCIaUHntLJR4CZ9eGJmDvy5mRiX/J+ozmBE2SAbM4KphbjNVBpM2pE8Nh9AYg2rF3DGZ0IuY6w1WdxH2KDIWkFr+D//kv7Kx3qKS6/T3F//8n//ql7/81v33tqtlFaEY3CHskl+joLz7VraYbSCgRWJHCFhzlS+lSGt5Vq3Ilyizeui2C3DlJA2VmmNKKTZr28vZo0GX/lWQg6fYU8jZw+yyf9ntDVr1/Cc+9qnPfOKzB3vXGXTOjjv7WweESXno6oCgY2zsbOkEP9Zmb8E4LBIdZ0s1ICHAYvn4EbtZp30qDCI8Oss5xhzkOzsK8TGA27sHZQwxNkEzQMfegkAi/ARKC5aZt8965lxQF71aFCf1CPitFRYFtQzcOjuaKr8zP7qsTl/9yIvXvvN7MjcWmcogs7iYnp3P1wbC8vKVpeIGOF3oe7E27h62LXBqVBFxDSxFirGJgKR+X69jRjOsQvgY8TWE60AcC2pbKYZXTGa1rMDaQIF7/B6f7h84Sa5O++YIXsQr4wz49Oov8o6C3eLyXI7qqIisSQ5gzzXWiAwMhTWwMm78TSxzZ++w+gRmXgPwp/eKTzJXMEiT6iLfjCyx6jgnmLZ2b3HD+ETVTHV6u3RuTI1HBe91cZwfiyFBm1SkkVY4iSiZANyyAeyEzzE2p4WUz1jnNonVx4BderU5xziuXj/IQkA9hR4bgDitphqq8rxUfGNlAB7MhcwZMo5k5Q1F9y/GD3KV1vZnXvuefG5wfvzmvd++s/nC9HTx+LS/X98lhPe643ppKuIfrkXNO2o5VolKRaHZRSU7Fevh5tH+Ly/42CKDOKp9Joo2SyiKtKIMbkyQHyGdxZICznqFljc3qwf75a0t4Z1yS0bnZx216M7Phu3z9tHh+ZPDmc5jfL3b+a1yZqdZ3V/XzbvYW/YJc3u76/sHO5Va+ULsrdiCigLRFtwcLpAbFUgwFB5f44H8/R4VccDziwI0q03yiS4mwTL07Z2ObxzsEyeUkZK2y56qAh36UG02eoN+JBFF25Usxnxt/2Cj1RQ5Le5pa2NDlzD8AZ4G4PK6IwJRJB+MT0VUsfahclgy+CcKBKmkaTQbt2tNstRwNOv0uhQP9kCUTwWFF59/fmdLMatNWb/feOtdFueBZVtbu//uO5ub68JK7r33DqvtKuipfX4hm1GwtgTDw4ePt7Z397Z3rOrjx/cVeNPsgclT87THD+/vuLgpnmnNbdl+H96/B/+Q2LU811SZE7FFIUvuwqhggLhH5eko0LEsFvVvIXinuqRh6wn0gioAiA1Z4SI+rQDdMAgFCXZ+aopeJeOs4Br+rChQQtRgwR5h44zjIvYsDjD4G2hm2gIz4inp0xcbqA8aFn9BBW2WkguDoMW27q97OacEKw/OiVxCS3JFPIyEyBMRVt3kn3GtRyRkTHdNWP1NzIlXeP+WECk9NCFoSB5O8Jk21wH8kKxXGBzCgf+zymReEizC7YkzsNrSaPP5yub69Wp9S0HUdHEUg2g1zFdxo4HhKgs1HwyPkTXBe7X6FiMzMzAiF8mWqkxTEhZiyLnvAzRhmtfnYzELpi6tRQQKurMJF4SMAEyU4wH0pZIq5AGXYqPCBiMm0JMjutjJABGmuoMda2HuVV9jwXbD4I16kZRCNgo4TuHNLvGTR9viDpJs0iSENSPYsmDiCMB2LWOi84K0hfbnL2ikHdOXskCZfIJYCrXzeogc/iXhp1Di3Fjop3f9zs3/xf/u3z1+790/+a/+q7mNDaLt5KK9t7l+iSXms/oAUhA1JxHRwvrMljsYWG310uvlRk2behDoWZ7P4B9h4wj9vKDWxXJaqpe3SNLZZXnc7oy7E0Rot7732kc+8onXP3Vj77oeekcPLuolFQDunB+dlnLl0hojhCZcGe3k3JZESUFM4VeoneVSj2IIY/lvcF+qNN1DmJlkIIgMmUezbkVsgLActZWBNohMVkyAYioATNLw0GqqJrNCGBaI/+ZWJzVyUhisPUemRvXy60++1rpWGqydnAzfu/HKxmd+5Dszd7Yy/YeZPNftQP2NeWGs7lKpzhTB1zCTCx0+hBAO8YsQjZgHoKbMEalR7GlIBm8DBCZv4ce0eW8qp4YytUKDmEP6gqNXKBIwEyj5DAXiuwUOILC+XsizgEqycMVZSa9lc16x3ogT06LJH2MZxyNBxU9hwRIO4eUNNUDFHeOGwbVW93/6uHjGygOd8Dh+jnNCZQ0UDLoRV7lHkhr8iGRM4UqMOF0YV8S18bd6RwAXX9/3q8BBpWLwjUXUJIuXs1QrmSOudfd0flq/q2udkoa6+khmjjgp3trrsMGRM6EgwmFmjh9eru8vKlvRMmk2jXjVgsAhrdSqkiw70T4lJ/2bl6bQ/NDW71988K/+mV87mT4pKqnSXB+gasVya6MwWXQlUBO2MRowQtaib6QgSxQvwgqioIwCeoITGCeXai5Pu0Idl8vuTPerCOqPFtmFktYcQifWyKubW/XNdQHESlV1umcabD9+/OD89ElX4r4OmN3z5aC7US5e26ze2VVHQyZVdq+piGRr2DI3062d9WqtKG2fz3OtnK1vNXvj4dnsAkNVFtrCcXvptS3wP1JLVRBaW9ObSLGq9fVNmaWK3DmHhU8Uxf7Orgbh/W5bv0sLwyYTKjUrYrG4rsjfzo5342Zq1mvyaM9Oj0UN0KJ5i8O26p0okiq/ZpaYLgbhQcRZdMknuoTxC+/qagIoIzdOVLAZyeIPnguUAbpO8yCF352Jgqn8jIAoASRY0jhZtnQussyatmnNgvfIJSHXRWJFFH/IQl7smde5Ua8+eXz/nbffZB3bWG8+enAhQtvbDXoanFcUt8d6Dx89ViQk1FNJASJcNDfGgA0d1ITGQ96Ofu/0rzrNackCubjErEEuOgGwxInwMPPU8cPpqr6z3eBjQemtfLmqE0CdQZ9wFGbAwEi4gfrEjq9eFcPwGcmdSf31Vn5KeJG4L/gNVF99riD7iv85jVxNPqU0WHtVkoCWfHHeB3l3UFlJesa+WWhNdEOTy03iLSFJYENCFngU+OkR7gXkVw+4+vTbt2yBXYnrGlD8u/o9bhc8JdAqyd8IrLfDgSvBKNk6Sepr6soJPFIlo1Gvba2xvnKUrrJrBPUSPbJyXejNGMY8X7ACuEij0doQ22t6PCgpdlEFGu67XbnKFhuTR8FlJFZZ39NDj0R94gUpDmEWRdWSMoErL/TSojdyA1gbaBAZo3JMxmhDJFajAlQ5do74S0YhfiMGMpu7iYPjNIhuvdGxJxVzRF5pNGGEW80E0yES72VTSIOTJJLGrGBGUfg41jxW3IG0I5asSBqOKUwhR/E7ULCgne64tbN5dnjo5lsH19iVCrXK//rf+/du37j94O13//yf/tNdMz0cN3Y2sji0XIjJQFhLlGpIxlCPoEMso23SIssYQ9y/1He4hXhKVNCUWHh1ObueWwx7yMKI4a38wec+8fytFz/7se9j9ANq0ehvLsG8LA7p4qgt8xHHYr+lA4XDLOpYZQF0PpyzSAnNWFFM4K+Ulc5og2pJAVzvyyijIWU0zmTlDjHAR4AbBTfiewI+A7AtZ0gJcUHwIAMGPFCP1yGPLo8ZmBmnq1BqfD47u+id1O5k3x195eCF5o/9oR/OvLie6bwz7X+9uFs/P33Q2GoUduoyvaIM2KyH2YAHo8C7KeoRoBbeCsP23MXk7PFsGkm9TPcWJrRjAJSYrqtCg8DGjCWBegC5McZaX20rUPQl3iEYUxwINu01QKAvVD4vQ99d6b4A7ZsMGA8WYoIcA8U4jkMnvXxFdQJwVs8LhghgTEzAfdw1sDU9NGAnHXSKLXQ/YwnCYzBePYYbY3aJT/nZRAo2KNerg0oSMREm1Vy4KO4cknnsxMlxZ6l7YSvA1tkAjCKNKKbAq8b6prd0lzRjfrV5UhIa0n4MNXZ8WnlqoDM9N+KOOKJJg/3MuE3wnAqJZLThYmBUgdDFUnY0FAhyQT4cXkzK/X62+WLzO1/4Y/nv+yt/5nOiil++faf3zoAnZWejzO3KBO4VLGBCzWAnRbXho4xFsnyQHgQ2zaV9T86HGtwvLsaz/jLTXYhozWp4qfay2CJhC5lmnXmNwVOnrv54cNo+Pj49FuTca58Mzp5M2idr4546cI3s4vbG/os3mptl6aFL1Qib+UVNzVVeLayRBSm/1gVWs1GhXt6+uT9azM4ZVzrDTTnxidijEcpZYgflak3dKC0KdPhQ14Iw3YuCFTr0hbQnfUckigmUYUCaN2shw89nd2/d1pNXgn8qZafPQfiFE6LIW55Gl7FotypujOszykdzKjFK00bdGdGzCbyiqb53735jXWwVZhdx2TzdjMY7e7tf/trXCVWEbKpI0keipgeMUKSvmoghHQVQ3Xv7LbQRiyPkIOEg//TomOG6Uq+XtCnMF+i47fPQd6/t7z2Yjkb9vrZLmJ37eCkK8bAvtEUf0N7x0RE3YJ723e2vrbcaiEuEMuTLqBEGN5mOQBW9vRpJHBOaMECjizBXBVABdcQvCiWmatKCTQnXFgHIs64r94Lu01CA4QpHE50JgEkKlAEFZWI9CWtJWPLC/oRIpy1gNjafIB2JtuvdCavhnbK5MJFdd2BhJvgw8TM9kqtpwLBThouCBtphzxY5DoQhBQMGCv1EJ5B/PCfd5Oox7uVrDDWOxse3bYnaXJ3+7McYhC+B6GkjhqJmVkcNDC62cjaKg2Oo9do6LZbLJbqwxlsRc9GLeDsrNh11WMxpj6H4KiIpcF4/n6CBcUKExuZkrEYQarjKi0WyDrwmf4jooZeaQO0zbRJoLYdpTryTvksOiOkVvRxaAUVTnO40MnCMyvmWwCv7at7tOIgQ4zm0WY/zq086tjuskMDM4KoYkZkPto3xiWql4jMgAxRGuVB6XSvGmLSWssAJGEgYE9sy2vfGJEmATc+NcdNT3SfoRXgny6XyowfvcVs0G62/8zN/6+tf+eof+AN/4BPf/30CyweLzLngtHzuYHuT5fLk/KSQlQseUBqFKyK+NLZIJx/OGKmkLEXQiXp3JUrzpdJ+415m1LmcDfThrG2rwX5Hr5RbN2/cub57c95HGAumO3GcEC8MnnCjaQnBg2nH5FSjgg2wsSKKol7M9W8i+tKXw9EuYHEhQDvRZDfRfx679mYRfk33CUANZ3GoTOivT8tn2r26+YgAJ5/kHLOWoO/k4kyxr2UFQPf6s4thrnPZGBdb04MP7nzPx39g7eWNcPROvp5Z109JE9bj1vXCIiO9uRs4ZS65v4U05jnL5RkLrYiIWaPOKD0wHC11Rrw4pgRbXJKPwWC9BKgAA7wn1tG6Gkhsvhod7RTnCV4Ym+W6wsFgdiv4h3JhcnabOIDl2IlP753MzuHuZfAWa4vvkuZEYKXso1AQaa8rNdMI4kHpIcHoAsc9HvrbSWASjG11jp0Ytb/4zRfDcyYoD04a9BpABS3xq33zY2ISCQlgDUQiTOaxYevj7BUrdRtsXw/5BbtsDiybTWJRxJrFe5Pg4gHB9dMgV88LzHGZLR2MnRhVfErqMDqzahIoClADBBiGaOHFINOd9kvNBTOugCJOWK5Tc4FAsWdlcr1SLbrhpvka1L/j+3+o94mf/S/f+OK7n7/bfJVx8aR9KrWBSubJq6WS3sStkq8aItMmY3MUFxwtFv3p4mIyuZCQK3ZkPh9mS5NSccEMVmvmmuv5ZjNXqw6443OZVDlD5urx2dlxT/T7pDfuHC0H7eyoU11OdTbaa1Rf2K3c3iqNu6eVvJL3FJ+u6MX+sFffqO9m9sxDxKnlc/VmKzIyzs7zUHpaQ2oCuULKY+hCOnLr6+uKyDJBs/pCKvwONHpdAMwXBYAxI7Y3rB1SUOJk+2xsb2hIJl6JZ5uDwBxjwDb6BJVW1Q8ODNWsmk2tCPVo6Z+fHbPd6TAoytVtiZuoIiMBYqwBd6SzIhNRGyc8d7RIA0B2vYIUKokVCCUrsYExFNWrNevmJuMiZq2jKKY6un5zA3cTzd5ud4SC4qB7u7tM4n5CMg4fP/7yG7+9vbVBunYJty25gYWMxBWxT7PM48daG5wDJ3FePK+F4UC50RGnNJYvdQfoaIGg5QkJv3N+CJhqZb3dI4pShos2NFE0Em8g0An+lL6RqzPxASAwjJ2K6vLHVEKBAIIoo9cDkwkcQ7fyMsngGfzb8qBtHhHnRPzRijMFBMeWUDHtwMMA9ODmPpK8HZwAaiXyhbB7NygTxISOuLbkhVZmxQ19j4ADekBIDeJPcSZasgfFkK7wJzA+nvO7bUEF0nE7qy2+BrUJipCGFXhpPgNDcQgLrUiGt2PNsNQpG4qtK2iXGRMcYZ7SPb34JeNS2AAVY2OIJn9HQDkTDfVZDyXm9CSgsDfFxEVMlIa/Ya60kPxp4cQdexDWjICnl8WZUZZ8pVwzNhcRa1gcgtDgVKLQwnNZwEYtQ1qR2HEhBIEn09FYUeL0kjEdZB83Jx5R/kIHTJfhqZHlYyqxvwhmNS0prMvMxy1tDOYzue/priDK0MIo6AfXEIQ9y+aIK/xGsKINPzp8/Nyd5yj26op85OOf+D0/+HuBxmg87Y8Gjb3dH/1n/ukPXN//4It3f+Ev/6UHb74hrqKp8DqiFhKY9CUmpqq3Nks14FeohhaqAOVaVW5Qv42SzE4e9pulvdsf/fDdmy/eOLhTzFa63WH3iO8zF/n5iuOJPlc8ny85nNtlOho1FoAhbtRgfstgt7Nh9+w+HmJiwZspMhN+JqNwFCVpKV4u0eQAEtIO30GcFuU15pHkFIphMrkGHJMeGAPNg2IIJo8UMyu3iipD9hdnF9PDWaW9eaf03Mf391/ZyLy4kcmcXY7u9yYXRBoigQlUcKlY0QNggOdXa5VipSl9gJd1OhyX6s1gmhwT6lj1B+No8yharZ9PKdkBr6zhaaixLgn4E3xbk9DHIUQEiTkpDmBksd6xtMH7AuyDtzhOT/cR4nOAWRz3fr6am8SAgwdjxjNRCUvGZ5HPejKuWh6BbtgSqByenGDbaQs8itElCFsdWn26d/zisTGEOD+JCzEyPxEc/MWVanEiHJxQcRihSk7jdAuXhr89XR5Ks9EH2QrPGVKSHimOAdJFGoCG0jOVyQP8OQ+MVShCeF6cHP8tzb/YAgNHWww9zeBqoOkz1Ikk31iEYOQiQQNR9ADQp5D1xJGRUAwiknwRpe0nvctKyxxHtay8QHqC4KQrcaE8aFz7ke/9RL/z03/h86PiTqW0P+mrbCOX1iiI5+EG4naly86LmdH8UqgGi4POuiIcutO5Ulb9xRob4KRekxtQEG+kQkC1JimwL+b5ctHjs4kKEWdqRPQ65zJoLqe97Gyw6J+X5oPyYrRRyhw0Kte3mzu1bDHqtwxFjxCqLid8eVIwRoVRUZgQLZbTjRENgQnDaF8eRL5OzcVPirJsNY6pEk3ZBptRI35d+11tPWENl7BiO7Ua5yjEWUbgkLauqgJMlMJcNsRbiZzZ3QS/2DPjEJpg5TlZ1NvF8LBbnL0pTias/GRnTEDtvLXt7c0QtxKLBVz6SpW5AHe2RbCZN2FsNnRTaPf5xRk3kp68g4F+vyMhWhg2hVVJyu3NLaUIwtLFao2VzBcakTIyx0G1tKYnllV3cCZrlnPvhZdxnOsDW69VBGdxxmunJglFasn9e++KBGpFb9OKl0AliPg3b13Pb29fw/PdkVBAb1wr1dSSB2sSJo87R1/76hclYN2+cfOVF1+RHHl42t59/oN0GvCB3UFJbCCi2uc60FTgw1pUZOTyxH2DQIDBMMBQq4K/BrkN1pF4sCl2whWNBnixhR4WLNWG9a62qx3InxAtmFw6IYDeBgzDiplujuYC+6AWIJ0QoOx3NgzChYg/dVUoIWkLyg/fAgOTTLvCyKsf3/9PPDKE6/T/+3+I/fRjEr0T+gUyejBhLwgyAakY5SFdijyZLOam99/A24boAOiww4zojHC7xsuQKjQfDGasy3doE8yHeJQWYGpMUrjAJUdjmi4KYDQFCt1vuYwoA9I0pzfPemLG7smlT8Yx58k/RCUL7y+gBU/ipxKrCPKvqjzqgFlDJyfQkEMCXdkrguiEUpyWyJpHqY1QlUNisZbyV8LBGVQ4PMfBbIzBTQwsHYnZs2OsDtplkrfomJ1XM0K/JjjI474iHl3VbG04P6qLRIiehSzwL33Hd31qC9PY2zw+O1Q3aGtvPzyzJJcxLwbpQYgyh2xcUhwGt5fSyJgj7D9SXCYFTOfDL31yZ/P6RmNbW4WTB+3p6KQswLzcBAokyUih5PoLZYhzJWz8pHXZe4oVoeCXKnicn0o8WEz7ZQE6mbmM7LBPOokLmrl46WQmM6eCwXhxv7P9hgQjLD0mK2Qu6pdrgml5+ThmAUMJVK4zajh7stjm7EV7fDgv9nY/1HjlUx+rfUgx7WEmd5qZP5yu9S8rauCT2wpAA3SpwTmfaM20VIcLckmiInuyRYtnYZHPjKbo4qinkOQgYsdoCTIKdOMAEJR0KGDJ2KkNL9YghFGbwQNbvwaaWbNYthXuAWZSlFeIX1eqYQB9Wlj4HX4J6xlwHXqtE0MCcBAUz3M8GanpgqVBJ5FSP4X4RW+Kq2JiPC+x/PRkI4E7Hp5sB+m3+D2G4kyE5ptbKP5hWvCCkZMDZZKlIWRe8d7MniEEskTHf+neYTVJCxL8OJZqxX3TjlgXcRfLtcpsbX2RXVdGPw5nZfydFzNKPQzNAJMVCKZFBxOMoJMA49+xETucREvhVrdWhh0KC2PPMIrqeKACL+0n7dp8VN2uFlrr88mFiCl+T6NVLyeCby4ztQ0O2/uFwa+++gc/0KiWfuLP/lpv1H7h7qvH751X8WCBBTFLgj7zM3FPa9n2bHG2zE0us0NlWfThnXLO5TRw5eutbO3Qd/NN5dmr02y+r3qzvKCxGIZuty2597jfPl3ISZ2Pc/N+ftavrU3Lumxm561cTi2MdYxVe6PTtphMthRRBss6bxnjtQgeEQ/Li/MuUdLK0Az654PeaVfnws0dllWLM6swA25siMfgsMRqUAAkyCeT7CpCilEazzM9VMPqOCaW/1ib0eWyyc0zPZxQlyVTtc+7j54cEliRU8QHySLEbTQbSB+vaufinGtDlMzNGzfEfIEpV0lEvMkfXsijVAW1REq8xQHXF3rDpAKr+HSrIT23cHrKyLzcP9h79PAx+zYSfufW7VIuL1oqmkRKTJ5MZAZ76/Fscu3g4N69e5J3nrtzm8okaejk6FhROU5ljaFefvnlVNVnqR3hzuZWQoxMt9OnHy9mVZ6iXq8bFYQElLQvxq3mnlLxIH5ro6lTKoe9SroPH3zlq1/9fK973CEZnb/31tc+V8iWv++H/tD2RuXoQuZY6EPKV7tKD1dZF/1O78WXPxhZhPT3Da5N4anWIyAkUDihVnBoYnuKwMKiyCA4LiQUmOOrc/FmhZJWoLzCQJ/BT9NN8LZA6IA5/xO98SPsQRDKgnlNrK9wXRRhyvnIkKsvXZQT4XtXCTF4L4UwVEA4jziHOBr3gZArGE6YGU+OwQZerUaBUl7RI98NZTUq+yR+yOVMO2Au3TBGTtBzqa+JZkD4IO8AwBXpjiDGL57qBycSqkZqU7DWhwpBFQ6TnColwYbDiBUmmrDROF+mtFJJ0po7bfakaHNE/gpHyHTOquMIz0fYvKXPKwkXCcVE3orpZX10WoqoSsUx9OurFIlN3pKlLRz+OQGA+DJUDfeJQC64AQFW5iwUh2ApNbYGfKs10cXMMKwYmGX4oZlQoCCnCLO7ZM6o7RLdihxEcyw9bTjofbBnE8bgJl/Zi4VMZrOO+D07urB+JDuxcGI9P7SpNgp1WPPdzpObL9xSmCYzHdWqpUdnJ8/d2MXqLSZhWvSEpsbChvk/W9UNXub8vLTZWMeVxx0dEbJbjetb0qdzNTWgTtWfWys2843Luvrwk95ZRx9Qq8r2iCZiqdZb31bxktmNRmYUFemEhJgaQiMampfT7Lz0Ot7Iglqj8O7JJSBeUvEwg4i78xHymCUlNwcHsrZW3vsrEYZRoMlry96oM9KdHLxU5P+N1dIbLM9Hayd3Prb94U99MvtiK1Pnq36kXtJsra+vMT0ZCC7lFskTFh3GkxiheopHIjUBdLEEoA29H466x4/IJypZsagQWVAWIBiqHhALZpD4WoL0YKJ+jrdJkBlwHn/B6gIog4/5MZ3lnEBHn6Q5vwT++c7iay0jEiBqkKy0Xhbmlc1ZYjOZhNlZ5loEFIQtGs5EREJwczMRnDZuags7dhpGIKj4hsC4kA7jWCQxBXr66nxXGRtIesp942TWUVODCmsRYijNesgB9arkvWV9PdPuLRTWM2qRSuDPC3ITtru8n5nt7a3T07NcI3c2yew/vzsYb731zqjWuv7em1ER4od/8JPt86+sV/rzyaP99eroojvsTHa3UFIlnmIo/ltROZ9pPtMEWnkjRgMC0dNkQo85fFFEB8QonZYZnofFuLIoVDab43Z3xhfTCHEklkOI8viIHCeTO9c/ufF7v+OP5T794/+vz735hC/m5e7JoFmpD2caDE8b+Wpv0j/DVLOli2WB4jvQwpPWwZ9cq2oxT62sbGysVUrm/nzYU95tMB2LcmYXnai4Mupqw4XoFHQjnvXz02H5clq8HNcLmWubredv7G+UCxcnR6xm129eA08soBUezXy+m2opMyYfH10szXvS4mTE3n/3HmV8q9pajGb9saj3SbFStliqyiELGBjqRCfGfZ9oVzqdbu3vo2B+krcA5XuRnzBBx1AMDtTT05OtvR3tfvV3M3tMrCrJID/UALFO1/d3icvo/2i8VivtmszgGNk1qQ0Mzth8k493d0earLkCoZEagWA2mxyxaF34ktTFVNOumH/tQx84OT5DTp9/7jnNgzHyMET3+/fefRvx3FhncG4HrdJ+Tapyt2OE4kE1U6nXWo/bh/zZF90OXX6vfPD7fu8P0cYfSECazB4/PhSKReIWSqXJKeKJTygEiyZ+/c2v5RvVFsM5nAaY8DNMLAwiuewv/dLP37v3lZ3NytZG5A5Px92LzvDzv/Zz1Y2DcrlpBdsXZ15Vo/jOBedBb3Njj2WOQSdb5J8wDaxkkeEZEZ6B0sFMrAEANcUElhX3te8I5E2YFUJA4JntSjddffEZ6BifoRCv/tKBMPKESsasmqrOMKhihzisRNC6no8Z5I3DAoX1GNmoVxY1uBevGspPIHKw1RUFCGpi7IHmId3GIwP5Yzfw1ijTCAPfkBD2n8RjfIaL2U+MtE67IierMcc4Q28I1SJ+TeDh/FAdeC39rdyuVCkszcmR/aWEmMA3xCgRbsQiooypSplLwhqxDtPEBYwvBMMlNjwS4W8AZsJjYj5DdYjRAyNH2ZmT55d+HFXJPQgHBHNG5gnhyIw6jsIjat1e1OgJhzHfhWBacQXxh5XGXY2VjRpDcnMIA7HFi+H9RFE/e7pNEQmfQgWcEGKExnxK3S2iJRHBB1E1x+Y36YAxucbmzt5afDpNJglp4f4xNJbZ3sXZRkWDqVlms5V59IAqyug26Y8bm+vUGqyIcCU6LZ9RjYOGx12r5HRlbSJoP6uxi508qsYfuihqrua+IREwia4ty2tFVSoZKDBUB8OmE1ZZBJ1UsOy8+zXeAISALkKpKiKaXo+VjJAUfCMBSYBLmGcBjFdyGoYVHMqbhOk2XpLzKOhfoSoGaDa+DGuRRwEY4dWBeNnRWrc7Px4u28WN7Ob28pP/5PdmWpNMQ1KKYhoXmXxP/ICOMIrWxA0Bv6xPS848HriA47GHKL/niB5ItGBWgal1Qu2wxyhzxKQZCOiR8Zmg0Y6xBVb6NGCIDzS/ucWPxr46OWFD7AYMPD0nuEowy9hWO341BjjOLRBSsRXD7JO+i+/mosxutB1M3FeycRhWvIuzn94yXseWhuTfQDpPMUrAHAONXQKEvfizSnGlp0JtIFVGwtRcGyx1n6vXM9JZIUK/u6hWMu89yOjL1eln8rVMY6v13qPO5n6LD1F4rTpy+oqh0Y+OTtavPf/g/En9+c0vn41//L/66qc+890v7H/6Mx967l//k//yRz77+9c3X3z05HMb5dyD0+56PtPaIu8g5sPwvL5vS1Ma44nhr14tMeiYpliBABK/eie4QFMP9UHtlayyJ9nIZI1qOtFshCjlb6FT7fhBrTItbGue9Ctbn/mOfzb//X/tz3yxPTnbunXz8aNTMKXV9eHpo/rOtWVj4403H85qjSliyCQiwosNRzMc1pGiks7RGrw96BxiMt0LRlc+1H77JDcdXI7p3ZPi2oK+mVuOyrkZmVRLrZ16fb/Z2KxWGuVCptFUGIfWKGFXmkQQ4vB8ofBZaeTqj487KqrQyQsRX38xvBRpLQ2P8WjWa7ZEBTGLhrcRyNiJzByNFzhxl0tMzuTQFMsR/rImRfbawU6ptCk4SfaOBBuG9vNum2s5McuUG1lV0k2eUoOMm/jZpTjT0qXCzqHCsSqrGiI7oarhqY7pnMHsaQg6AxTDbJHeObZ4Hm1N8OCL9jl6uP/xj9cq5VGtMikgpPnp3lystZCu+29/w/JiwPK+8GsSA+YVusdkZuTsiB7lzbQ7vHbtRv3s4t1HD93z3Xff1XRFNEazuS5e+uz8BKNE1wkDqLhc57AIZtc2tYJ49ODe9vbu5sZmKYWvJsUMc5jeuL7Ta4vtytaZUrmcL3OHD49PDt/51V/4me/7oR/d2m8tJiJjphenJ++89UCJ++d/78t0A2bqYknUGAUGgdbpCRInPIZQKxKeQqyRJJzDEpqIq+NQPpHep+e/H6wDbsNiHFCdFJUgHXHEP+hZYGAgbmgC3IJUnHKJaaRGs4iaGCpcupJpJ/A2WNOKbKw+UY0YhqeGzBn/pvAiD3JecLiwUcf1riJWwJg4BPGxROTAMCIXKWiDv6AJSVsI+mUzmT69YCAcsT98pALLHUtskvEeWVojq7tzuC5wJycHhkJDLbC58EKviGCQOIweRUVCjE3FEvAgVUwWb12lCDBXKpaBBaGSQwFqg3hCpRIt46g1lmh+ZHrw2YRHg77LQkJ+1LmPlzrob5iZF+wPXFB63Bmig7wHnulMRicnmxSzFJs5h3z4cqTGr/z6octehd2lyXRWzAD/g8vSJCD6rrLF6+DW9GW2uNVqcNusNGYTyqIadDUuRxQ4XTLjwef++l+9/5WvdO7dL0ymrz73qnzoHEeYJPVRdjzUOAHxULCtVs02S9mWZFd8x0zjAdHCA7dd5OURyQ6K/r6IBz4iUAT3wmXmsyj9jI2xUpA4JuErFXs17HZCdV2tiDcg1jibM8na49neLkhq3AB3WL1ssr2mBQwzRkAMBK81vGs0UpLDO+Omi6YSJWpaf34hwWE8P2sv7o+zRzu3Kh//zCvFT93MVJ9k8heirEWxzsLGM0PLluNFpUrltfw+IpMpmBvi7SFexGCi4NVcGbpwkbnrRGCN55tkWdjBt6xmwGJAWOINDsXrBCL5SPC6gv/VuP0YO7YA6ZBF49fVG9Py7V/pxiGKBPCanWjuG2AeiGJ0ckCipwyP7yjM64QPbWbxY2yYNQA6Bp8mroAwN07Piuc9HYxDhhaf6agTfDUIV6AajC1mO9SKhGNQilnBuDhhtLvEH47bqkSFTUNexPrzrcbG5lv374mL785ro43io8vywWsf/sKXvrzV2Lr36NH8wdHu7v68cm22s/Plo4u//JMP3rmXGdS7ux+8K1fzX/lT/6f//X/87//nf/Z/9fD4y9vNjNIlul+AW8WHlTUOwZo0EONONMOOMRELIrA03ioAGWGI8QdUp5eKN4YXYUWgVw54McKoUt+py8EjXc4zU52yI319KaZhMeg/quD5dQXLWrXv+fSnHz//d/7K10UOZVv1HicDma5a7oZGNB/V6v1iRR80W15hDdxXhY1IhpieHD0RVHTWPj8+eYxDwXWa32LY1RkzJzyL6ZHtSET1YiQnUnTvZq18fbO102wwbbFxrqkot7jsT0a6jgiwZG0WEMXSKAGHFijGgNcZU0dDpuPZqNtXyh8dI7HPllJdG0i7PN/GehOAiIGq1asnF6dn9O/hCBtDytvtC/SEZY5yyUtFuTztQJBlc71lPmmtjx49Qt+Atrmt1FrSgegKiImEakpCqGFcA5EEvDg+O3348GGkMHS6Cmb1onNCvaGz0sa2fzDXNTUvBX4wDyn0pzWKMlOIeLQbCe05+EpegFqTOuJu6BKXNU7HVJ5WlpqRYYgWEGuQ3MjlElP8Ovi37JSWUJBz2d/4zS98/re+6JHuiedX1fkqYBkTOrUazM5UkQEdaLTW85rXTca1zOUmGAqlDix7wnK5scH/Ldhaeo8IinmjtbncXn/wqD0fteuFxaRz/JXf+pWTswu6V6nU+vgnv1t7CmlK7B6oMpgLrhj8Q4hLAKH/DcVm+khAoVelKJJQTeF4Qj+vZ2QJ3eL0QLj4FvwqQTWSYoIC8RIz9nskz5AXoTWNzWTBCudz4q+3tqMgsxQ8WBHUPKhOeorpDcR+uvndkNww3JbxU9zC10RsV/geA3A5ohJ+s9Ba426xTEF7I2vQEbdJqxMlW907qFXagui5oUuwXvEdXokQj6rS1CONGZmaKwrDzhRBgkEm0ZFASmvpbZHsZMRiky2VyYdBQd0g3oDwIeKaGMDRR4ecESEJheCG2GWeXQdiGGYMK6YyghFoHzS96JLEUKf2H52YkEHJ81h6nktyEUXurqEWMzsbOodl9OmLCqPmgbYZbQzwzdXGuqwYW5iMw2tcYWMRCeXpxJwINQ3ibaxSYZCa4L4BY36N1jchbaSopSS1xMrOxSLT55KxgrQUXroioal/8df+k//4V3/m50ibzbWcBCoxwSrJLYbSl9WmUMQJV8wt6N3e/bKRnZaAsTY0ySXJJi6Ykr5lKsPvELxzteYgNElMBIRwzeJhbLWcQr2Lfk8Bub5Qi2B4waxNUnAWSwgEGNftpGVN7CGoLlKjKhFyE0DG9mPUIMAvuCL+OZz29TnNFuv5Or/U/ILfTVncPLPS8Vqjf/PDjQ998sXMi81MuZ9Zfm0+P5yr3RGpQ4pni5G0ykHGVPuyACYzQAo0+sPGjGTI3hoAYIvwCu44DmxVHUiqzseswEraidVwbVwfMHr1DvbetwVov+/PSZ6Y3tdHbM6NWMZ0TuK7+E9wX/Nn0TFas524b4Q947XyEtKOr2viPyNWEtjDVNyS9utWCWlXN04DWY0yPdTTjTNYsJNXI7cHT4K3QVbwDBNiQcNaexnhNfyRueyEM8tqA9RK7WIxP57keu9213dfPu72f/IvP/i+3//cpz/1Pf/2f/jnHh1lXnil/Sf+pX/lS2989a/++m/88A+/3O5n/9O/9Ks3ntv5kc/8nj//n//NduFv/PF//k/curax/rMvfene0Vfvn89Gww/e2hyOz5VyquUzzY01PWFXrNc/pvxqihIVSG+WpHgjRDFCcIE5JKIghD7NbSyKer8kvRxaP1WjFXEEkd4wM3MyeJHzIx72XrmiDtVv1IprL/3od+gh9vM/9UazerfnxQmP1crjzrQ9u9y9e3ctXwajMA5Eik0YCrDSPGQ8fPTgXRmFatb2tHeYjVh6QhhkFFIpXOHY7GWJ82g5LmQmGDDFZaexuc1CI4J+MFBTRnIkFjNtn4E5IEWlzMzF4QeWQX+9V7SBUS2yF40YhtJjIwiXy2QpF4glNHJ7YI8eKkQiNmqvTUNgjUOjghdg5NE6EOUZy+znmVarI8rjtBq37tw0pSdnZ5OjIw8SJzoUCyOysj+Q5eS9GOeEPKgw5ZWb9QifRl7JJBUVpKVqDqdjKFRtPNds7W4J5jqYkq4JtpEJr8K05/TQukKh7itTtjgvG/3WCkEo+gyiSlfE/ukvN25co9UE/2KriJoKSYFscOfPedEZlZmVtbVo1FuAYX//Gu3o5MmRydGxQcqvoiK5fIuRQOOT9Y0aOohv5ZuNXKmoK5l6UnzpM75IzaYGvYvHhw8YBGQ1SeO+OLqQrVpcK7dqAGDx8N2vvXfvrd/+/C+FWytbvH69tLvZMs8q8DNRwk/KPu0GynhANItL+OrFvBUcQ4jBjH0vEPJGCIZgNcDVOelfqxxgeoVzKwIR2uSKmATPtuxoUEBwgHDiZskjD9AZM9lGg1IE5YHqid74GnTDQxHe2OL3QJj4lpAhuG9wq1CM/BGg/SmKa+ZD3Gc0sAXqeBGm7WozhWx7Uwe8B464YryhL6yo3+r+CdE8IfEkp8T6TiUDixIEupo002aD35EDgiq5W2zxKKfKYYgALmUmolyJaQm+G2p8YDJrDxgyKjBBqCSpEQwV4gDuzDXJsotSERIDIcPwPJ3qm6BDkcUSIeisXrtDCoxqD9bdp6afou/mujh0uY1dRj4VLCDLnpfFbc2UlwqPLymYdVWDE0aGYgWYikROSxnvjiPjBRRlf96cIh0JSYnZM0KEvVpMEL7J+cwM7oJY2zT5Ud7PIqT4JJdcZn/9r/+1X/zx/+rFG3daJV3Hqp2Tjhar9cKuJgpqnWKdtYi5r5Uy6+XLdXV9qNfS9rgbo94iMV3EMLv+bNko1bApYVrETKECJoBVOVQk4oti3H3F4zr6iRojiUBMpSF6VQsPTIwuLW90MUiWarAKbkLwiv8S3IYZF1OwvsFWEn0NAU1L4VL2Ug1q0kJbtV39hmb1+aLEEXTxykdvPv8dr2eu60z4MDP9aoYoVp/nq4QX6w/OmHGBqxQn1kRFdxUsMlPIoIe4n1LQEzrlrK+gFRUqdOWwSycwN6lXwzb44LgxSpAdY01Q6gWA6fvxyzrGabE95cGBrKtjfkiYEj8ZmYPJ2kxtCNbLoU/zILjZgfMCrLg8Qr4MfxCOy/Wrjjz4DgZMigZrbuBlVlOXxhfPeboTDw008Hv6L06LLf0bk6uBjOkP8SioBaLghn5c5opyw3ssSrX12tbucC1/TD04uPlLv/nbX3/37IUP1F9+7fXd79r48qz8V/7MTzxRa2Mz88rv+xf+P//NV95448u3br78G8eZL/zmWxeT3P7a7d1bH9q89dVf+dKvfeTBx26++j0f+Z6Pff3wcb51cD4+zFVuzrGq7DEf80UXkBkZVpqkfLMbVCYGGmQpxpxm9eng/WtBydckUO9vBlwdWREozjjTPRmVBVts1wr1mr4HIJCVadKd1q83e+fdteWTyubeePBGuV7/nn/55W6m8ys/92Z7Vm9uvnTGebK5c33n+jRfblTr+C4WRZiEwsy89F0CpSBnYDPXCnMyLHDLI2royHyorY84Jj58w1Y9gdm7kc9z+iiiSv2SZ4gOBapWirXLujrk/RkcoT6i/dr0qv6aoW2vb29NhASN+50+cUKdvqluMAr1IJ5VTuiILBmJ9dJRoSpdKohevl6vHhzshdiK7ylSMR2JHKyuKR1dPhOGSKOeT4OL8wYJHiyWr1+/STHwRhfdHpJMXwnoWtLh+5IUgL0KGuQWfJ0v9uCAm78kyAriR3MX0CTMRlsklnBNnQbhdwe37kYCYGz0guFMi2IdyNGkp1Gw2uvhpDePndCHspd0GwIEL5SfxF5jXviwVVfY4/6DQ/ZpaLq9s6e2EQ3etRtb25Tj9kUX3WaCf/LF3xLILY6GQ6DMxVgsaZKsFki+PzgSGLu1sS56UvgOmx3GJuYTt9/eEkPY6V0sBoQykmde6c68QgZv/NavHh0dyg/e3T1gEmifHf/tv/kTf+Sf3G1t3IJdbKfyQoIOJHAMjPVfMK9QBBMIBtdFoNO+I3HwfVtgZ6BU0L1nm0vso8gkLsdRu7A/u5IEZBmgPmbp/pybIW0pxFgJYgELEqlZnesz7umfJDc/u3naSVQhDcbsWbdJRPxaYuEcifUmL2cw4GCMmaw+XYFk+BtSHxIHaYARMwh2ep+EXPG7m6+QExSIeyLkkJSwAUyJaMhm31Lfiid/plLdPFK/yZM+Q60w0pgMZNy4I0eOPboz6BqAE2xJlCkQxvC/cEgUQjlTutwM+BU8odznF53QXDGOSA6MnlE4OaKM+6KI2DDGyUdJlsTLu/0wDiJxzNqQNepTBN0OagjNQr+a4Lihy+LodryKBDUDcBMTTa5AHzmyPCFNlAsjmkyWTgwggYAkn5BWxL3I18JOTBj6ExTItMZXBJ4Yh4BDGXLt6btvb+Xyr92+86Vf+6JWaHubB83CZudsmstJPdig8oiAEM8s+pXrd9ybhBiVYltxXwyTyBLwMRf1wAQtbBg/U2LZ47Q8jJ5pAmpIBGZjpsCnaF2agfjpnDbMV+BqRFbQEI3RvRNYhu4S0AjksHBTFIw3Xha8BbSvaLD3WWbuPToVr1pUf76yvFBSY3Y8r4wLm5d/4F/8kVB5iw+lgS6L/Wx9AnX0igjrQcBUWC3JfKKX7XmsSLmYH8+JDilke7SBpksDEowYASUkNqN0alq1K74bwwSDibfhWeY3ID+iwK74g0WMM9w3XjONO43+6WHvk5QC52HRq59WZ2K9ifv6tIZGbZINUEh3Mj6HhBB5R/EXwkzUf2ZShPShuAOsxJriiavnxpE0kNVH4GgIIV4gnfD03/g1hhKXx7Sn9wtholgpdaaX2cb6+v7+/Yvhz/7qV067M07173ruM1/pfGnc2P3Fdy42Prr1b/2f/+1f/o1f+43/53/y6ge/U5DfpL7fzZ88//Hv+fVf//wP/BN//F/4k/+bD1z/1A/+/o9+x2e/7y/+xH/93LX93/ja3/2hH/0o97ziEzdffH36uHZ4vNzM7VIO66384f23GxWQbF2Cqjwd6mrSQ7AMKhawEWBhrAYd4hUfk70kygAfqxZLdpkZ9P2qKrBkXdIo4Yv7ICfWeHCv2zioUfunmfNcU3jRLzTX1/7gv/jRRa3+8z970rtcbN55Pr9+nVYLQy+6Z7hX4HI32ifQ8PicuFRGAyxcMSWyg09rw3AiIjeqp3H5mFARoJFSxf4cScWFCg4aoRIy8qgzciIAjowMiVnRYWF2KbaitLu9K2u/0qi1tja//s5bEfmAI4vHR6ekOKknZieV4QAck2M6gZpZ21vb26Kn8eDnn39+s7UuLqnTDt4MqDUIVmkEnUJhmGdrjXrw/+VC2UhWXM3CMSHR4Qp+4DLmMIZD1BeTTeddLLwxuoRu6FGqsi8qLfR6fXdn9/rNnYNrrY1NP2F9lUo/SF+KULFwlgm6IaEILx2ITAC32KXz4mMHw8mwR0/wK3Wg3WXxxQxTTPVsTjO2lO0OFpu7fv25w8dP3nzn7Q++/jEKJqypSTiq1u889xxaBpXFbaugiRSaEF2O+KgPru3uXb8mKud8NGwbxHprh4dscwdRi1pFaojocHzyWKFLFXNy4whAk+mlT8bGsHOmm0lDsO3aXO7LVH3u4yO9kV/76DXZnyOlZMJ0EpaVgD7rFnMYE3kFpgmFVojkYKIEgdrBVSi3wDW+BdAmAEbgYkskD4z7L2gfRTCUYPtBBsKqFXZjWZWceOyIvU6zuCcohRPCXRK6RtwppE6g7yYJif0A8sxWsEjPRZd8Eu7ITJGRq6jfUhytVCtyWmLDSOqKAXPwaxUTCTDSnABllrMZGsU7YpaJTGOZ6SWCcHlerOt8yv0eZRjBAdMMVktlYHthGo6psCG6ApAIKMpAM0FEjG3wWXwTgROyR6KV8RXCmiCeWgMC+NlDz85OQQlU8Y5ETjMO52a9DsGJq8PNKd0+mYFprjgOQyv+5mVIASbNjJ2fHMOQkt6GbMn8u7mIUXc3WWtoPbuKJfAsU2HCIvw12FCYl9FTt3Scams/1iUIevxsRa2qb94snexIhICiOZiLnFAZckQdTtqwjgOSoEoRMGwlvCUxdh3KDMfHb763WWqoZi9qtD0f7B3Ib9hvrO+auXqlZYrmlMxFZT6eBwYG9zH56gKRSCwBc7jCI2PzzeWNkAlt0QqtP5CMpBfGRCIvD43sdZOMDtHd8DfvH1AXFfcNKvhTgCq3+Krkhr1Q4ohZNqvtBdIp8doop8+4Bru5fffD0vmOLt7pXjzJNDvrz+Vf/OTu/sd2M8Xfjkgr1or5UJkbJgKInc82oxxSSJUECqBIcxxlFE/i0zY2PC1SSMOtHxNO5BMuOdVcIyhoMmBc8d146cQFAotW8lsoCWFVIlz6LSAzfgpw9ZbeIWFnQpX0UzoeH/F+of7GJMDIq88VyfaZuK/XdXn8+cpiwwcfum8abxii0/CTfhwcGlcNZwsOtLp/DDSw8tnmeLIuxClPf/Dc9DV+En0WZz+9wk/xl1FZ3/sUhQX82jeO/sYvzBDgux9Z/ODBC++dDrb393hd/vSf+4t/74tfaQ/7mzvbw0Wme3HxjXf+1te/+rX/4H//73/0ox/9f/xH//e3v/LO/+Cf+7H/95/5cz/8R37vv/9//FP/2Y//mfGy/wu/8eN/4o/9cz/543/18KgzOc794k/+8ut3t2qZ7qc/emt3+8Xx4CFsW70LCDfgGFr6SC8U05e++cmII+c83iRcaQSWK5nImUCJ/A5i+m2i/ySaVwEAJqdSa9buZXY1rp92+51S42haPD3rTre2/uAP/7GPZWvjL76xVtq42V4UHxw/Zu45fPQeO3nodr3ogglzSZcWY6ncNwqylGHKcsAHENFsS73YeJoNQ4WUkGiC/Yg2kGTPzQmx0dVIVSmuCSLm2yVJV4S9ok7qaTTWVaeSX0SskpukZapnqHqiqw52O9M4NBBcaGyov1Q+91fhEjBSgtudtXbnAgODj5tbEmkbdMrjJ0ez2WZIy8s5mZ6eIHY6KRV5yinSRysAYFFFcqYYpyhoxudqtaHpkBiwoTlHlE5Oz6kwODO3NNPfxtbmzTvP7Vy/uX/tuvqVQ7X2I75jJCOI8xTB8SBRo7QXZmthU5EKPFsQXEwdyYRAw7hXj5ploW+wSPPylmsxKqYFZDQev76+vV3c3Tt459339Axu7exubO+/+toHNze31bJlSvzCb37xS1/6bXU0yaFMEtQPxE1sJw4dGjBCde/Rva99+RvTib4u1Y+9/qHn7h6oyH9xfsjTXC5vVKujwQUYOEeRiGPdnjdcO9jfoRvqbaf/2sG1/foge3R8+JqOp9WieqDMz+gRGmFNA02CqIb5KQAvYNSPZi/0YDQKZwrpkNahrkCIxkmN9Hm1oRFI4YpS+PQHnuPrCmdTZbvQrvj3goNm2S6y4/Fpfb7rtkE2M0X3FdnDcMJ6xTLiyqRghpRJHgnlOlxR9qV4IhEYZJQ3E1MJEwQpAoigLEmG8HOoORl14yiwE8lO/JYRaI0Vic6JFzTawLQ0zvT2yIK7U1SkBwx1nZ+rTgWMvDFMPT05UXaM0uetMIAwCwWDhJxUyVAesUwcQm0IM0UKYLUVQwhunMN+q92C6ZX9zXTMT9jvjnhQvBYO7RzqpkpSZEKzzpKZS86SuM10WqtggbN+9DoZmczwS/KulopqkNM8TRzUhaJInRh6kD7od3HfZC2jzNIPw0eCGcg4Cu7ktYM7J9qCo8544iCdd0T0TZ3JDVUYYsxm/Sg+oE6CgLCIPxjwSMGdUBfMctAu0xjKPaVWVlf7sLO/flOYSOGy0h1Mb964dff2K83mro7C1er6RWfAvQplZ2M9g7FW4jmTr2d644C0gJIQKDPN3Y1wS44H3faRWnHTSS/mpCgLAAY4NclAK2oJuCR+Ci9JAsdKf3dHK2hzR7+6BJgmHhIsJYAjNNckRkQZh4juXWQZHOZv3nszU13sv9r85Cc/Xf1AM7PRzeQPM4XDea4brdotNqpnnRiZ9ZYbjtar26q6602k1FDEbWssHc9kkZnKuXaC3m2KHTkAtrh3WKHRcjMvJN9kG1oSBIw3oQqa6kQj9UNwPTYdkmialOAL3mP1GWvoEqsQVz7bqGtpMlc4GEDsa+K+QcbjSSlewhzACqsHO+Zam9K1crxhfMAxr94wzGrx5wSzZYzukx4YQG94pJZ4uP8hV6Bj2pwU6jrPSBxJXM2DPDjs/VY4sqi9HNzOl/sKQzV33zsb/93f+vwbDzO57czd29dyrY1f/crXv/h2++Vi6Uf/6D9R+M3fuvf4+FzHDM7e7NoP/fAP/qW/9BdU7AbsjVbxjS/d/x//67daG7u/+pXf+Df/1P/sB/7gd37v7/muT37i1c9/4e/dv/jiJ7/j5V/+qV8QYPx3fjFz+ujs6F1xXm//8X/q44JNs6gEGhLvdLV5hYT+yF/ATByNdwnKBUQgQoRDhDklQD7eyztGHL90/8y0HQu+KWNAiNd80X3v4foHX+o++oYiSdu3ygoRVhQIKRMefmFjvfiHfux7Nq8P/vpPf/7RmSTx3MPHj5aDoYqkOK8/ljAbDwUCRlMKyRZZY3+2EoYh7z1bGEv9Dz4v8DIyETTs2dpq7e1tbWOKqeg6NxkiGSYxDkot/yJyJdZewBQfrfqPSnhMGN0mIwEPkn7V1IU54hHGEYjHuRnl6AmXmKg+TXCK8nrYPTo5P6Pg9rs9DYsOblx/eHz6jXcfVrt94cRCCZmA+W52dvQuFVsKS4ZPjo+gGKqgYJTyRuOS2uwF7R9297cZt0fDvl+HfRWmQqqO6FClGGs11aO3t7cjoqoYSSUszovh7OTw8OTk8OjJI3Vkydl1Lb43WnKfULOOZlP5Ai5spiJCRXA123uT9iH3pCoqu/i4zLOLe5+etdeXaxubBzjrxUXn8fHxebe3ubv31tv3bt19tdWM4HHsXOmu7mjw3qNHd29fH40uUTPEW5wOWZUk1D07Uyyqtb99+Vg377XZxz72kVu3laTv3rx5p9s9OT9vz1kails6YOjaLde4O27rylCtt0gEkKpU487KCfjOcSU3qjT40bijx6ooTOu6U92+UGxa9FdkoURTUCSPTUMjLWhpSYIigFwkPtQi6kQgJfIW+BVMEWeEkwA3LG8JYfk6pDGGdza02+CUDLbNpSZIirvN2mtrg2odqC+7na9CiFLpZr68xzsAS+cK0kWwcaa6tqU2AqYZCZoZ6VVt4AWwiAhwEuJQghTZxsyWCw47RVXondyf0a+3qXrqWlHJ044wwvm4pEp2uYH7BvnTLi6i7gl6gXwIJDExMlriaxCXgYppQ1AsMY/W2oSC5vNCUINUc4727MI+6hXMlYmkN6TOugqQ5aSwZjV5xiIFHUyb69vidIlg+g/2O+2l1FjBzBVGknWSJhhS6GIh11YtVpXiRvO9rWvNVh11vOic8ptU1xuT2eDkySNjprE6X2FxGMJ9sq3S+d4uZD1/cqznttT427dvIxaEOPxlMuhh6Nw0NPfQ2oaCF0Scagzax468MTUaDljueIVqFbDQ2rAwXJeCGe6cfEkAjSuWvQjGFGYRVmhsA4LjtKXy+fkJaXK9uXl+1l1jR1rfYJHtniJiu52L8ebG9ie/4yPP336pXGlZKwXSJ8MlaAzOoeJiLT9ajrI1jvVIMEV3gAwLA+FVuy3e9uHhO0JQoNx0yNIn3DuMAUrOZK1ykqhQQhwNY9Uplj4p2AppDKZKjI9/MRBpy9mROCxp5lock/JpWzKUzFFWmeohmQRccYUT+vuT9mg+GlTaL/2T+7sfWG++eCPTzGQGj+fDR4sl7/XSrJg3acGMDMFc1MESpFvQ8KobDFWWqDfxalftEjQLlXwfJJRKXsTPQryAT8tIvwnuFUuQIrtjxwb0IJVTUHrvF9ZsLpUwBoQKFDAarMEv6dNCJVH5fUwkfomT+Fuxx8SfoRsGEr7D4LUxJWCARZP5ia8Qo6XsXuqDM41YOa4blU/D+ByGsIj58NA06rgp7Sg2PCnGu0KakG1CVIjHpt8QAoptSFBx0Dz5Wstedi4uc82Mlj6iblj63jk8XRQLe8+//I2HZ1+8d/blo0z5eqawvrH70outg73idqu6u9aZTX/8J39yvbnxoQ9+9K233jo6fGT6/9aP/9d18m0+85f/0n/63Z/91Hd/7/5f/Bv/yT/7J/7kcLNfLWzorf6LP/vGr//83/uxP/adueljoyjVuqdrw+aNzKSSad3N/Nwbme///dM9SwXQI7gkzGimB3cCWmENCvhPdrE0jzFZsRMzEfKT2QwunFYvnYDYMdB42Vkn05nMGhuF/Fa1sZ3vP/mG+MuGOIfusmIyFXavTcqV0/7F3y5Wzr/r+7/3cbvzf/uPfnq62C/ltvCGs6OLqLE8wWyGshl7ykfUNpkziWeaCKNnGhJHk+6ZBjwIVF/AFmMeJgsWddfWNZfUdN5rM1TRJqQDic/kaibToxSkLxBQx7X0PuqdgQXatihlFKGB78n9LdfG1Vb77NQ03Lh1oA7Uw3v3mVF1PpDliHScnJxhA93e6MnpRZD1grCrWX3v1vad4cMnD82l9CVzJBIaMUZZWdO5w6r5YnWd4brIqn4x0PtugSVqhIR48k8fPpxdtE/oyAp9gL2tnYpqGL3RWEejF154aUQIGY4QIp7ws/sP3/7KV7ice50L9bdvXbsuLNryMQHq8oJfyg9pd9qphP5aq9b003H7XBgMT914wi+mfjU6eevwkI1/2u5NK63cwa3nvvK1rx+xTZZrx8cXnXZfvd33Hh56WRS5UK1t7O20RwMGR70o1BQoZ6cb1c11RFJSJZLHRFGr9cXgaYwnM2nYzTBylYoNrkoVzQRqMYfF3Narmk2sU5xzJaAmlRjymCkyNWc+Ko8+94YCQ9YUH5Fc0xm2g2axpXJQcQQltReIJp4EECEZPA6mFwoQzAubJUj0P1BMUdSBmTE78RdsmEoNPldwHDcgdoN8tRf9DGgoDUwrUdw8Av4uspetbLZlsbloUTP9lrAC9D/oFaVzPpxrRK0ddVga2Je5RqhxbuuF1L5gQWGCFsUtQIENB8qExyTiC1g/xHoGTcCk4wq0cin3lDSbqm/SG9OreUwIEF7A88KWUq0znoREgtAomAepGqSrTT9LrsVE2I4tM8tMayOD7bGoGEeezTCiiJWEUls74glZH5V9IYVESLPCDgahOCowFf8qzShMNTHJ/PygM6lYKcBKz+02UfHCTPJukPV6/a56O/jl7Zs3YBcT9cXZORjwGCZor9nvnqO1sA5xwdN49wbqwwmxygnFk2VySXwbtM9ZylmuPJ0bSfjx2qw3HwpxjJx6L8QoobqOfpDmHrGbDMkcQj+m4i2dyiXC4UrCaNQ189zAhy4eH7Fhblzbz/Tn9z7/hffeOnn9zkdf/8FP7Fy7pVLU8Hx4dkTaxdhp5m4ZJWQEADGy438Wr5ov0RKBmdIK2d0dkzR+6y0l1LVGj7Sk2YL3OazUoaJG7axKKFNJwbLQQeXJfQF2AA2gBAuOWDxMhpIi+ifQPoEILz7qZZekb0GEvxDyC+Ps8En/6Kj/sNjKvvadL9/6zg9mXhxeVs4Wy68se6ZGsYBpVPkgI2FK9FEPidsF9EdcnaXE+Hl7pmPGeQDnkzQLiVTWj1MSKnANJsnGDVaKZ/CrtCVYW+2GR87xuL0t3gYs+i8USM9d/RT8L3ZjS/MQHPHZ3eIcJyfG6AR0MS6PT/8ZjghFGMmuFCIKGh7eFUiTVF6fkX0UjuFo45I04DB2xeOSxJWelx7sw8FA9LQTaGM6oqSJVQqXMb4LpeAbdNJ3bWcz14+UGEJyBP7ktm988rPf91/8xM986e2HjzuZwk4m31oXaHvUbWe21t/70peu3bkDUD758U/9f//sX3j99ddb9cbbOuspCTmab241bt/aZbH78te/QrR57vkbv/rlX25ea1Yvt3/sx/75Dz5/7W/+jT/7sz/1U6++uvvSc3d5WlSI0G3k4VmmtMxsb2T6k8s9ZqrEehOliqEbbMzgaimMONGseDW8NiYxQC/9HBpHLI/DSToJJh41QkJKyaI9Pest0IkUTjySrTSL1N5SNShhv6PkHafQdP4l9eN+9Ec/c3bxoX/n3/0LO1sfHfVKG/Xm8ZO3ERyyqJhk+tPDx29v17eVx7cYscBBEA0hXLwg3LqSAqEhnyKZIKKiVFHrjZiU+G6EayACrN/ofT/sfsAwol+roz41zG0AP1saWTwKIIJLFS7n8/Va0wCuc43euePG8mIjNig3JsQTT/UVwg7KlSp1k1tIO2C+FDUyW9t7J6eP1tU9rtV5Tu+99+C0UrK/v7fHLMwQKMg/OgpdzimlmvthSVgy7UdvpdAlkJueWOwllCQJ0R9oEeKQe6PpzsEB7eVrb7zR0Sbw7Te9Jt+20tXKbO3sbRsVpaikpZI4yV6f4gugYSj6jJ1wLXNJIqfKWEoonM1OmJeRWGiKKa5vbFPuD66rvRX5xtJBSRhbQlNr9c0tEZURELd7sPV3/87PnB49lHOs/53KZdg/osQCl3/0+FDmqFXsdjtvvf02ZZxJ8Oz8iSSnxUywpggztRVEzjQiJn22EMFFBGYK04UtahLI2I7srPnDh/dfeKUj16KXrArVcvXw9FB5RHpf5JZHiGQYIYMKh6EyACChuo8gd6v9hICg09+KT6OJq69XIEtcJAcHP6asBFfETPjpkV82gyBOKFkYxy7z/R5WMeMOT+4HViohUquKWIEEomH7w3MctFyRzsaaGknWRgUVQFSKGbOEAkiC2IQIEdUaWDrZRlX3xYKIrlSm8BaH9TTK2kxoK4BaBM0VKUlIFuKCsadUboyKuZF32Wi9FwC1BbGR8FqqaLVFojA9vEQ2DzKMNC8QF9qivcjZZafXdglsURHJrNIxnEZi4JOIETIw+lVJNCboMoVKdl8OZMpZ5GVxW2Cqdjn3p3oNONjGVisyl4qRuYRWVECEsEZlLS7ZJ3TujCJQLk/mCvLzpVhoa8hlIgCLsVthYaFMEeYsfYg4RPTRkmypS0EINWKzODbRE/sg2+wK7JiNO0IHOJ5CDIhsYG1d3D5/8tZDUlI53yjMo1p8pl86fO/hL//tz//RP/zPHVQP6o11drJBh54phrnWqquSkyd6Mzoxa9XqZQNnmHVkmhlJN8io1ONF33yTucmbij3stdvprdCHFNHMDIcLLfP6lyY5CdBFKIAV4qOJFBAEM+Yf7wi4QjOpLGBv3rlAarA+aVrsfGJ7CxUGoPpJ/7gtNqw83niu/MnXPrz/fCO3nc3ULjIRRzKYzEc8SiYjbz6inInJN9+JwQeE20eGQY4dUCAyPWzN5tYeh0EYZSxsOi/AxcnJEer8ZGOOy59tidXCBMOOLYApba6Of1dokniEMx1YnbPiIs6xTKuD6eq4HPU3VbGDh8RusnKjcJEiQALBgAX0CJiNpCsiHR48JYevvL9C0AkwLjaRtmDmSefzGv6CA6XP1Vs5IUA9BPOYcYwiBhR/cMpfVB/OZBrF0unJcFlZa+7s3X941Ds+qzw4fOPth+8+yYjo3bmzMXKTtYxKT7PHjyqbre/87s/ev3//znN3f+APfN+v/PqvkSzhQ3OjcfjkbOdaWXwshP6lX/3NT33q46+8+vpLH/3Y2eHs8J3eT//kT91/8Tqj7g98/6ePj74+nIh63Tg5ORbdGJUcWCmyme55P3eNJSNx1IT2Mer0LmnA1ie9tcF7tZiGUIODzLxvi4lNB+CgHZf7hC9syEgBy1FpI2Ux4GCUIfpvAUVy1nAtd1GsbPQmj3P5R3/sxz527947f/n/95vZy5db9d3jk7Ygo92dg/v3jngznru91+0ckTipP8t5FYzLHMesBBnoTcoSJNqRUqUeE+psPKzNwYdl1nJjihzO0QuVAxBEQvFACVnOEBERTyECMwUBBfsRi5mSS10u38ImutNAq+Ik1xsIFAsUlMdzJas3IyTKeupNiYagvRFd4dFCsNoCm7OFWm1x1qFl5rw3pHX57du36s0mHXd00fFc02hsdNb1VovzlLw2unffvDpnd3eP7nv37l39GI5PjlTsivkaTx4+uD89F/PUde1uY1dCUueivb+/m9zkpics7RBBtLNgzToLe6MlxurajdvimRFR+HL9psKTs6OTE6ydCooSahx8ooaajORinotwa3tTII4KP3zZt+/cfOWVV958883Pf/5zOp0vJt3+5paCYKEqLHMPHt4THZ1vrm86fWNr58G9dxFb5k1UWEQ1JckoWQ6zWEskF00087FUaotJ81cCW0hRuCgL8sfM3vLrX/vyi6985NXXPq4MPqNBUbUmwpKYblFrKYgpIDCpv6DSlqAN3AVqAjwiSaBmfCSdONEF34zBGYGswNRJQQflIJhP/Ais4ABhW8OWeZGZy9K9owDWWJsPTu28+o216B7BJyk6KWgcxsmRxBU/XMsqN1rxUOoaecLhSJgJBgpCqfWRX4nCYO68D2jCfDmUex5mYUJE+JWxSSPkfYSNJAb+FVyqmt5jhWZBBL2Ca1ml8NRkCvTqaBJKEvPAURMIuCYMr+E4cUltRWdgShG/FKZy4mkE5Wqn1R9OBW9pPKCOORP05bwS+btUXv5bkYCR83ZZr6nP2MJ9oQGGpLst7ss+DL6JpyErcc1FP04A1mw1WkbY7Qj2GzmwtbE5HLV13YsaSvRDE6CTijgpXRDNn7tHw0cuuYp+9uLhlxOsM1oJAV6ENvXpY4gWia5Cjbr7xazSjcR6FoZk/yDeV9am4W6PCmCyVSijRXEIDpucXlcJIEy6wobZO+8XJvVPv/bZ53ZeybTnw86ANihEUZokINKBlziv5ku5VQpg7Ojafbm+0Wzt72fabYUYx6cnGmV3Ls5AKUsaC05NjjICZOaTSz9mHA8n3WdKAXJRJycidBO55yXlXqRPA61gPWkRmTrCQCIuhHGiJ+O+zA0tnXrtYtw5uzgbF4at5xovvHb91oc3M3cYN/qZy9P5XIGB00LUI6Krg1HMOAS1qI7OeeOOIQyEzBjAbw5xOO1adFoaKEzZF5htMvFSwC6xIIbhimBasQH7OAIwn26roQayJLxafaZz4xVcEZ/u9uzX0IoDA30+2953fhy3cOHqxTecTA7wr1k0qFB/k3YrB0m8FSE1rJg6XaLf1lYEFkEi3imBDab9zWfZNWrvHYwr0DGxnRg3OFsNhNobzJfcixPHmel8aI+onI8n7Ulm69r2sto87Nx763j8W4/+1tkoU2hGIUzZsyedDgsfwfHi4b3OW9Mf+cPXvvM7v/NXfu1zajSJJHr5xRcObu7/nb/6ix/67ruaj0zOZwr3M7F+7je/9MY33v3jf6LYP+t+40tf+nf+l//zL/zGzwnW+7s//84nPvFqPl/d27/TH3yJS2Nntz7rng7amdPjx7lrWzH+NOnexr4V8j/oioNGnn4NoTqYc7zy6kicF+ueZt/BmOr07q4Llz2DW6RzD48uN9ZmpRbnSm48HS67w4I4qHJ0WJqM2+Xc4/X1W/fv//zNW9/97/1vf2zUa//EX3/v9GTw/J2Nh48fP3zY39jYvrg4G6lWo5oplXtWicir0HhhtOrRE6bN6DEEV8NPhLoEs/crrY6Zyls4gCheBo+8FJ9FO5RyjbasJH7JEQYNPEjqXsXxxMYmKM+K/jguTKlYyCkq+d577w3PT4U0h0DPJqNPUYQvqzlVi7RV9+wq5ijWC1no1+sNScbYGBRWLoPTYncyERIFBEylCO4wAS+Wjc2tZovyUqdG9vHY2eLmzWuvvvqq9+LFQ9hhkJCUe+++E/nHYdFcaMOEXtIY/v90vQfAbOdZ3zm995mv93p71VWvluQKGBswJNQFE0PYJYUO2SUhISQEUiALJEsKEGwDxjHuyLYkq3fd3u/X6/Te2/7+79wri5SjT3Nnzpw5561Pf/4Pt+1RJ8LlLter+HcVI+0Bs8PHG1Yl8gM6enU/S0UOvGPXr11Di1teWkL6Np2Kjfh8kJwDBw5Wq28h4SGhZNOp6al5po7ET9K/lODU721trr715hsEFR09dHB2YmTt5tWp0dixw8vofSwAx/Fjpwg+IggXYz0kPB4bQuonQzqZTDMi7DomngghuUGbDYwV3QbCLbKR6g/jwMJiBodA8YRW3Lp2dXxqMTE8oRzSUpmJgZ0wH5CPwZozm10bnK8Gr4Mz77zyxojpUHOOwV40i5vfax1rTZtVjX4m6wll/ni6pQ9iNqhb7m7fZ+gLmEHErQVx0OIvtLvCCv6AWUI5GEJteik/BO4gyUt6IzmuQmnJMusTMkyfkG1FdjC6agRMa3ksGpFs4FBMlihar1g1Vh7WKHFI5KQLzYKeonRKj9EPDN3QG/4kJUD94QEDqYJGyCrAmo5WAcxgUUvrZaAEboXwiPzL0NE8E/UAI8SLJ+7AtxTe4taMLcsTZwFB3+hHNIMVg1BJSWFuQtsVLQt7r2XwvcRjEZheOrOn8HqHYyg2DitiuoGjMSOrSltodUT5MBj0nMRBpSFRSYw4cIKCALwjzxtbKG49aixXLbgyuu2ygo69FDmRC4nBEVoEWEdKjkEnJ24eYYnNLLGfV+RsKyXXWEwyK5AhRDQ0OjJ1XdyWpr2Xb/XLDJjD4w02G7ZKqQ2Y6PzQ9P6tfQ9w43YiIgOwSOwdRHbAvSAX7EvAq8hJDsRjTAAGCcoIYWLC00NcBhPnJ0wrFEQ+IeEfQ5bR36TOcSCMQCNZPIRgG5Io9oKFmUkk2ljlpI2Vwjg/YQ8qToJwBWVtAFjvwr7kbdsb2c5esVnserv24d4Dj5wIT3st0x6LL9do36yU9hwBgG+cZKvh08IniHjNmEA2GRIQRgguZ7KhXVodLFAoF1NM2cAqub3sPOiSIKNkkOT5wIkPCDojpxBCbZ/BC5KgVtht9qZlx6Fvb+8yHgGhN2cMfR8wYHONXnS1dpguGLyan77zc6kFMGBji5GhEDcLjEFjY+BNtIfaMGAUX+E8owejrmPmFPcd5CnhlpGerYAObsqo69V0W1YdtY3NZrqglprFwtlBwVA0IsQ+TE0aeE6KX0E3EVGcMUfHF7x4Y/3CuqXttaDkAaZgBWe/3srVWnkCcj1VaC1EFGH0L/7iL++79+5MJkUO5n/8j79PEE8kEvrp3v/1nice+70/+P1jJ0+wKdCyqo3+H//pH+UyyS985gvNet5pr29tXajWk8MT3pX165OjUzOzC6TQQx2H4qFizUIMg98FQRgQK42hGcg7bzS5IgFmiCFdvLvDe9UbJmMw4PoVrFfJBGwUbsZVyk3StMtejV0n3QPZ1Rsh+hZUV8QybNGQPwDgLLnCPhBVVNfN5Z4fjh39lV9+79joxd//98+UCuGF2dmNjWS9ChAEaQxU7CA62U2RCQSing3kDUo2sANwg0K6EF8hZuwh7LISqbFR0UIIFGDVFDvCqIxOS/9wMrLyaSLcF5swF3Cg9TKnWNZ4NVRTkLpQJ7ove1ujAWlC3IcdQgAVUNLvw4N7/SKGVXYyXQQeRH3uGpEaHxL0zebKFMqKpO/1RocS5NcGIv5SpZEvbuCtw9kDGUS1yxeTeGYozIsfbZCKBGT0EHHI0ShKtjQBojKIQrF0MskdYKMojtiplgjgQIgnEwISiNoqBMwWtnS0TVc0FsMYieheqtYwARO24g+ENza3AeLY2tyORaLgDsHQAZicnp6mI2urtw4cXEqn9ldlT8ZF4Njf25LFVxC2nbffqF2+eI6jkEnjwl+Yn2vXh9rV/PhoYmJiDJx2phit0XhSLRbSpPAyIjZkUpBpRpPNJiOc6JLZJEROsRvh81optJcXeU5xuDLwNr/Xf/7cWbJaP/AdHyUbZj+bIw9TWwy9QUol7gUazNLjZ+aduQmqBosPfdjsSr4XI9Fq1UblITye32rTSoHVGdYk/7CflcJD4T+rP0IbAJmCEPWxfELDui7+IqFhrz/C3BDQYgzZWtqYYNjuNB33oawWfSLRMZBieZAxXYoaTFEANEZJI18XbBRZzJlDOC6iD1ok5FPWD9Q20y/tMygDGq2KQ8iXQwLyQJSFZtBgDm1Dhg7mhC2BLtNF2ACPgx1bldgNbgamOmm9GG3ECx0uXGhQspY8CORRog84EQkp4onWBpcV223U2bOwTLYXAgFLH2kUyAxC92DbDKHXF4hG3emdPdMzpF2M3tQ2cPNw2s/6qFeAN69zO0UMEudVbxULLcQ7Nr1suS2CoomdwcKE+CklFbsTE0Gvi7kqjyPvHWKEOQTXlUio8ohU8oZmIwPyGJiOgoaYNpNdh9yiJMEO8Z4MHfuCQCgg3/kFvLyb2St7XKCIeCGl7r4XXatWaO3U0yF3lKxsDLF15YORNOFECCciguknK4ehRmjHdG5p18t7O6n93Uax6He7I9i+WEyya+D0tUWCASRCLRg8WkyKsbNxgTiJiD+nmD5GHJBieDC0Bi1P7ozBQbCf5susn1KvQrUGHC9NCpC782T0Lp6YGT8xaZkOWGwFIDUanXTPW/Z7QMAjuK0Z8Q3TDAI7oWiAuiK2IQt3G9Sw0tYQm2L7YMwiIYo9y5Kr8Wg9WPPFPkAoG2TtDsKWmDmZZKX2mu2gTnAfveqMDq3H2x/V/ME5nXjXSX3xNw9zGS/6LRcO/thqbHEpIqxo2YIYMKXKMFSDV4LYIJJMhIA2FelBqUGWOkwanVh+X3YbvzU7gc5wW20F7srDeS/tV2dMfw1F4TxGTqNuy/GpVmuzcwPTL/aN200so9UXTpaal1bKmZqF4raNDgDFmHjcMoy0iam0gZJEgROsixZ3cGxiighGCsZdv3blX/+b3zpz+tTxE0dhw//PL/yr8YXIRz/ysctXrr117sqp02euXrl59s2XThxd9jqsP/fzf+eDH7zXHwlPzQQunn/r6tWzI2cmRod8VxsNr7MSG7c4RywnDyLIVtgLDJkhXGbHa45uT4G6aVYRU0oHODR/XDX4YL4288dAQWIGMwR9VHelCNh6FCMhuiNXrQ3BIIaAu2HQiuwYOClTEI8QJJUiFchtb+5sPTUx9eQ//IcPUuj2t3/jGQK1F0xwEEJjIh7PFyCSNAsgU+pHQHZYWWjVfTfeoy4x69QRRFoWcaFR9Ig0eQ4mEWoDojLLFU6FkRlBFkoG3eCg11ADo85acVnynpMDpqvfygklXCqoE7kVhDcvLy/jqeQC8CYhL2FyN8nm6IH3nkcihkXF4wdGnni8WChfvHhxe2vDT5qOx0WBH4KiAfkDGwutgMYx1HjREBWIlAbZGq8vT6eMEobryHAEKog2zMW0BwgqkUqyjvgr5EGkC1ARMuS3e134iVnNKCGZZAo5ZGJyfHs/if8ECOhCsYwlHEy5/XTB6Q5Y7e6RsSn6OjYyPD41dfPa9Ww6g2BHZy9cuMCbZDqVSu7BK5GSMSE36uW1lRu5Qh5LJaJAu1FBBFeBG1uvUCliYCavicC0UjEfDgUdGXQFVoRZ69hj0XTLpSrliv2dMD5eZf9ijsSGB8AkZLIJx0Kcl9rH+DLKxpUH+B/10ezgCb395huLB47OHTgId8ELyeZlFAh54RWCwrxC0LXq7pAMrU9Wo47BNjW8UnyXFQLn4yu9VwutVOjTHfRjcURIKeuHuEEMxJjGpVCpPoyVNGSeByrjELqUBEWpnkR0sOchpFJDWNnAlYSCcTi2XLotq89b5yZUd6biH1ez+lGYESyUUyELJm5DeAYVlqD7VFgSlSYemE1ELiaatEmyUuKN/GGYAwibUqNFJenGYAOyENlppu8G7FH0TNsAc6YkEApP262K1MfyY/wizLqM3A7ycCHAXfi9j8zuQADyzNkaU60SBHYn0fiaHQeiJdehOKEtsGVwDXA3JCrFESiuLyea0KO4AopjNYfiK3mjw36j2cp7VaYpIgYyRxFphmxqFq5RvqGWAAs7G706XDkeCePXx7pL42Hb1P2k1A1WZhO3zqTSUkK1BAwBs5e9XTYFuCaPEQYJAo7TGnKAoEMyWNtJHT/llzcJabdORBas9mCnZClnGy4gYpxBVBqIF3NBUAWMktUAIBQ+Bkgs/lEMY85oSCj75Vzuyo1kJolXF9dLMOQXEl2lylBr4ZGbhMuKaBlEeBzVEq5EyweGDVy6CBzodYh1WkwwWa0v3sFySDhFN7PhfIDRC1AX7dXWtE17N4p7jX52fD505NT88GLAkuhbArlq9nUPse1Bq4cb9oE/7ngJsrbb2I2aemqBkniGy5+Hg1REhBoqDwYT1F2NPcZmFiaDJ5TFwUFbZRBhUA0n0v4YeEP5Gg4l1jZYWfSGqR1sIqZFv+aVpaUP2miDr/Qzzpjf6JpvHeYnxtAzuFgsU7/SQ2RG1lJlYPgIY1TpREQbE/AsRAdxXxYIDgL5HFA5EMC4GO6L9gKrppVa5uxJid2wGZpmVF72JMI1T0AE0hVqqywlYk4ykIo567emF/A39iQ9oGC2rWP3NDuOmyRXNyy+qKXcslF0z+L0Tk0sxPu9nb3tqC+RK2UJYeSmCF4nTpx6+cUX9nc2RxOJW1evz02N//6/fxbSOTzpYWp+7df+2e//wX8AWG1tY/Nzn/v82689Pz7qn52eWFiYT2auLy7HF5ZGI6HTb7xw7sj0oYlRN+a1sKtw38nEcDADIn4jh8VIDFgtN4d6SHcGfTFnxH3VLU0EfVI/BrNjzgymUeQBugrN0G3otVRrBs3RokCiaktW0yV+5QzL4NbrEgNOuLDFTg0hwirKuVCiHw2Ad/V1i2v+7/7U45V87nd+61yzemVsaJR0gVQ653JFIGIURjHqDYONcB0BrdCOjktIqYGtwenFziaT4QABAABJREFUXdFfsZtB6mFP2sV48lxASMkwSzwN5JJ1gAoB3eE85QpgfvxK5j1lWqK5Mahwb6JqyG9kZJQ7RJLMzMxULBJKxMKozihtsGGCohk1CA1aMNthZGx8Zn7pwLFTm9t7e3v7t26uBYMkFaIKF107u7iDcSgODw8PxaPcAUaOoDY8MbadzHR3pOPJUN1qjNtHC1RPSyVF48nIwGzbaGXSyQqeKa1Usm8DkESOouo11FFCbl6/sXjwEBa6XCYHx2cegPXFGkkocSZL+FV5dGxyenoGiLFEPEYtHLj79evXN9dXaQw9InwausHQ7Sb3edjk2Di2ZNoP0AIL3RIJRMJ+vHUkSpXyOfy8RP8RHVbMptjz/kBQEW5GhNHawBRDHlU+X1iYn0HpKOetZIdioQ242EBuuxKIKCXpQaWAhKDeQDCQiSFo2KGR4KcmxtPFFpWHh8YmmBK4Cm4YXpkqRRvRTGC2tOZu0xg9UusSncMsOO08vmINah2Yy6BCvB2sbF4NqdC3nIeUsjvRTlVy1S3cUdlYMKMiqlDgy+MNwdUgBmh6+P1UqE0/7xGkbsXsYid+DcBwlnHLAlSji3C4NPZBQlpRRxVfxdO6mFaExAuvx9+P45PQPzLn2EzSdG1eXkHzZABREiEiJEzQIe1Gxl17jFWujkh+NztNDj90Q8KpZdYjMpkdQeFW3Miq3MIhpyQqoFazuC46onAMQUiysRN00OlSsYD/lyeKjBEQJ+8LkepOtoHEz0IRKQp0GvhMuVjeq5RDqm5NC2xcRi4v3kU2BUyRIceWi36DYAsMGwIujafyQgGnqQOpA/oJNWU1YpIFasRODg9yb2A8MRQLUeOekjuOvoMaaEgLSATao0h4UB0szOKzWPvlRidKV9GR3E5avwz+9BANFnEa0yVIIZQIBGeG2KtSpu0hOafLRie20lsvYjhWmjwGdgQmes5EC9Ff/lyB2ViiYUtyL3Nzq8wytfWC1KXHmFEvg5ADkyY0VcOoYYYhSOsk3d6cY/BxoDA9jD8xJ6xhsHClchrvhJiEUTVt0noxABFXYaeNpBkrrqhqr6Uq6dhS8IEzd42fHFWl3uZapbVnbdc8E4wptgxVAISOoSvg9iCgzhvysEfk5CF5utJgsyDxOL2IGiVCGsHjMqx3UJ/QQiyf/Mw6tOplgWRdiW+xiu7sA7OWBu3k1Zig9QNz6FrzvV5Zfu+c58e85+z/5uBKFq2ukaZvnjl4FSuUBmy2I8IIQgP+cezPyi9SLDwrUWyYYcLsrDAdiRGylvFqbgWh5K5Mgp6vBokuqX+md1iTuD2dk7iNHqDNb/a19pD2j/6XxK9XtlmeCl3RobbNubXXoeStI+7vNm2JxES+VD1w4BgxAiSTBCKECtiKlUw5nzt978Nf/O9/RRk6/Jdbq+uYJy+dPb+9s8U0R2LBbJ6kwvJffOrThAk9+egTjz/x4D/51Z85e/b8zFx3eDQ2PTtDAjyQ+6vU7QGxtJIfHwYg3xLyWs4cmxjyunK7uz78KkZ4YQRZpIPh1fYy7zSemrnb/WYEoAb0U/2/c2hYtFaVcIXEqEnTR3OLHi6JJkGRgMRUKgCy54NtwjWhGgBNdCPDlsJmOxwj0dLSSOVNBmYW4kWxxJ/5ucfB8v83v/Viu5n3eGJ+Z5hgVEiSUuV5PP4f/D4dP+Eb9m6NDY6vCUkctYa5QZJqUVO2VmEl4teivTAIMH4IsoFTYnweKLmI7zQbOgOrU/sloKmbXM+Oh1hxho8DT3A6nQTQgzGEKxPbFI2F2dokZZKK4QffyoZ24Z+enTt45Ai0DCY7MjJMlQVuhdBfLlbJwcWpJHUiHIoPjaCKUG5he3sH3kxryHPm0TBCWRKxhCtJpI6JGIwtMKdoYYnLCjkUBgAfwdshSIowKyA7ybDa2t4haPnoSXJ8ccbZpyZn8IFduHgVr+ChQ0EFqPrBg4QOByZnZsN+X15gHdXh4URqfx+6SpwX1ulsPo++zCjhe2bqIcXlinfJN0tj1lZv4gHEY4mrhlkv5LJDBBwEApjm7ISxlcuUSJN/m0th7GhCxDtjXt/Y2PK6WQqUOiDivdLzOiMhr9XertYLwiEz0LOE0Yhf4ADuOTGLoyVubW+SmjcxOsbdCFVgUJgJZhfCzHtGk6liOJBmec/ccJJJ0ntMnJL2OSCjUEeFQHMlF/A7/cculeSlM9zGbEqtYw5J0wQRgJpCiq8VkaqMJGRct07oG+Ut0WkBThOfBh1NPAa9reuixg4BUGXVd7S5Q31qgDgjWNECfrRhJSiJ6NB9FiKeCa+PgvfELYXDo6wVnP/4OBCqQIEJBmPIUZTDBJ98aHiMMh0sMbBJiWKjsxB2lim2RbNYld0E10PY1GwQ4UpPrRTTCKuYookeZExAgzS9VjfpnQKvKjWC/Qh8p48IbLFIBFRFeD6DXCkVWNMow5h00I4BaqHHrFEWfZYMM4pkJRJk9wByhfzFUDMd/A8DRn2keDVdnxgdJWWdqacoCG3AGbMwM9UgyhDe1ea2QJEIG7lRAUergUe9ks3UclmVQSUkoFImJqMLVBM2ADyadBuzAeZBpB3YOT4mFGycOQRXWr3MHPPSqPWV1mZjsSLaYpWJ4PGx9cmcRbRmNSDlED2IMIs1kzhsB/oUji+oEXEgMo6hIEDXa5VWub53+U0cwKgJ3E0bH3BdtEyhBBuFcWBoYXVJGGKxaUUZpQQig6gkIkcP2an4rcEzwXrJBXBPpdJKtrdVyCR2dfv+dsNWTdX3yt1idCQ4tpB48ru/z+InpKVg6a3XO7s9T8kaqCG7kjMOhyH1w02EDI4O7sPKpMEkXGARAR0GcmYl8VipiuQNEHePnAeLkhAIfzX6D21kh5hTt1mU2QNIaoNFQUfRKQcUG1MAO0gCn9kX9FHXsCFYdZB8JtD85vZ5cw1XmO/u3EycDg6hCzmPXYf9yIkBw0BSETMGIsnYNYxeC3cV04WscpGHb/FO4CdU9QfUd7FhGy4p5dsrSUn90uDzHzBO7Ek1T4f2P7zAMCn0VM6LSTEIRvoW94I3t6isTqkrnBgUn1EoX7XRBEeUaKlC13ljK1kA85nRVVERb6lI5GL705/8c9LU640y4KTDYzESJidmpuqV8gff++Rn/uKzQKbCOchPdTtsLUruKPSPebYeWpxP7e288vm/uv++h+r1PEXNySh9++2VbG73va7TJ0/ME8B0ePlUaaf9wrPP//C3f//kqOXmJYvzO4A8yFGgmQU3SNQhMgFqh10UF2C53AbXWGPLoqMzt3s4IFhmQLRKTf/1nS5ghmW6F024LXPICictF1cO44hBzIaBtJaXL8kTA0anUs1afERQ1JCOTYwetb8cEHNc3ehLr/7Ixw/CFf79v32tWfSPxeYKtQy/ZqMGgrFeO1AuYppitRMlRApIQfV6PEHIAtIPrBGhAjBqFi9MBTrGbEnNIKyk3YIW5dEmwdsmHkoLTUwXyoM1lPcQOvQ/oq4oVk/UFayIM9wQCpPNpo2po48Om4iHqf0H8yBYqUOyKP61UBBoaDY72kTQ73vk0YeuXb9y/sLqxKh/aCROai3qSsfV2d7Z5XEnjh3HrUBpBRDNdtNk9m5RJZBaC5lsGlfO1u4OjDOZSaOQ+HxZyBpa6MTo8UKBxE7F3pITsbm5jeYORcUPHXb5N7d28RnvYIJu9RYOHIStXrx8ZWVl5d4HHg0nJkPRIerKnTv/NvIcBIeyCpSGO3Lk0BhwVEPgeIXeePstyAm4IouL89VyhSdidWdA6LUi4KoVhD9M5eXiJGewLF67dg2CDMdTbOzIcEJOczKHjeTicXnl57SBLrl349o1v9seCMa3N27u7/cnR4eJzS7msgI6ZIeIpokgECCEhwufIMCeh47fd/jM6Xa9ncwXO5WqdAxkUWRcTZXkYCil1uWA74pmiMLoPrcPlpok98G8siYHK5NXdEsugi1xoaZdq1dUi9hsGD1LTyYHN+htBNS1Oq5eOle22/BSsHSQDvBhQmcIa7JGQ1NsE1mtBYocxCrbQMSqKGiIG1LunjCEJjIGG9viDbbbJWQON5SY/RogA4JBYgk6wfZA7+dfLMa9BtWmUfbgQa12JT4Ck1YDWbLafDAlFFu2ppETsc2o7Tqgd1zRDQSRVOTB5RRKCP/ynlcEDn7LvAx+rhBnOG6ldO36jhgE3mICInD6e+Vl4TKioqYoiyG5kn3Vj0WjsMm9vZ3heIhfsiYwzwKHXOBGRFr1+vBdKLhK79VrKPIYSBFy4yHX6o0L5OkqsbZPEDPEEkJJm/Czw/aJx6qp7VKJ1GA0O69XmFlNppyOSpKmzoYCrlxWELIwblEPx06QGBQeHyi5fw7HsI1h7LotLdzzCp1DiyMsk3WhwpEQM5mzhbIo3o/T10HsmYQsrQWsaRWqkuYw5yBHiG1wlgxw+JicKEaxgs0ZLsPoQdQMW+KFPyUvmngn03BRPdk1Y+G4IkpAmAPzhoAy4rzxu1tqnlHXbnEtldnqBarBcdfSXHj+0JR3MWFxX7Y4WxZbpdnNti35vgOwG2YKU7ZMCOwLA+DF8kSZMesUh4FsJNomrBLM41RoIXRdbnNMEeoEzRB3po8sKrOl4Iy3qTPUm/fYevCMaOS1PYzCdbtr2hM6p496VZ8Gb8zOeuf9t97IeKMLB9yXnw++krdAa09t1kAypohE0mVZnPItiKEy7ywdOYAVHaIoMawJKrpA+q9cNjBo5BAjIdBa2mNGgK7RnUGUkWmeWDrdYIbpNpPE0mAn0yo9Vx+ZF7YLA0KaiTYJEIQMQU8BfFQOyRAO0LaCNeTzxzKN3m6yUKg2iZbBNEI8DQny6SSaVSEQJQvOsb6b/vG/85PRCCl5jomx0WPHDl28dOHA4tLqxjpmQ+5eKxdWVldHhqLnzr1Kvuzpe+696+5HDyyn6vXs+bObI/Exx2SMmZueOLqWufbaq2/Vm5ZDhxg8Fynl/hDR/qwrLUT1wkwEGxbUKYbRdF59MYOqIZcoqBe9vTNvej84uJJDU2/WK1OpxSBSq4WqaEbGC7MU8h7pQ9BpG8TB3A2/LQ80rxS+q1X3S9WCzVr+yX/47XZr4Od/7unTo+OgnnpDEfjEysoWBCQRH89mimD/WxoEWVHK3kEFYSRlN+GOWL96qscnCoBT2OT1EhNqZLPblIp2wlmZQYgSNBCyw2UDKvfOK9fAAjkPeWBl0FQCOYmHgohlsinK4MJm+CGsNxKLU5Se51bLBY8rRO7T2GjiwPL8xvpqOlV1OCiLRJyKD7CqHtXt9/a44dzM7MTCIsGWSwsL+FkX5xfATgYf6fz5syi+CAEopjx9cnKS90OYTEx4Nm1zhYLghrQ6FbvCXDqZfAlBiB2Zzhf2k+kqnhSgDaGuZFiAbRSinGvzjddeu3LpItLJyRNH6cH4+Gghn2NPI8GDWxIfmmV5bm1v7+8DLlSNJ6J4hSd6Y/Dv7Z1NOk5YNe6MXAb8jiFUZDKP19cJml6PhMKo6WQEFzBEKpMapDD5aqmlCnx3g6yYoeHxnc2VVttx4uR9QAvv7mziFxganyUHrFRIF6tFnIt0yekg2cuDaE/rsUR2igUAQwh5oEIk5h2ICodWFitP77QG2d4DZ7DUWc6wtnQOoss1bET2OrSMdWsWnX4Gw5MsqEV5+2bmhjJc8ZnFiRIDkhr0PdTzKMubg5+x1UUwiHsi0qADSmObWQkFQxAXpHKbO4zVDGEEsQ/fEmeoZS7lrmahonCI+pwOV6BP6hHuYZ4DKHkEPQ/ZljLwaM+isS43FZP7PT8cgHFgYplikRE8iIZi0lnc1dyBpUaLgVJCgKVdyO0dUklNvUnA9eg8V6L102JsofyUfBXszNBxL5yTeuztNnjlhBJgjZF2ACFEN8cjZzIHuCSdTDntEfYjTi+mj6Bh5WWRTEbdrjJA5w0oHV2tVSsYYXD+00ueiiUEtRR5i8yfTruKXFqtFJRyJiIv5wJOAVHiHuVCGUHC47E/okDKc6wJo/YDwaD4ANlK2MWsHgBNG5UeNw0Dt2nzEGYFBAvOF7zEyHbsLo970kJRbyFAsePkIGZyIFGyvjJVpD52aryBRuDBxmuruhgNFX1SPBXxaGQF8gxovwQaqA5kQRqw3qs9IhC8mq8GS+42cRQt4wqd04q7fXQtyf0CAEmkTGOPQl61+WwtR6PQybRhyq5sdM46f2p2/GjcEsPtmW60rxEgh5uMP9LzSM7gXoyTfB/Yj8VdEBaYRtax+JWeiUYILBS2MEweMs4j6hFUJkc+rdAAyyrDSheTYqHKXq3doGaafwYN15IYHHf6oM6bg6s0F2rJ7R4O3gz2ibpq7mbeDO5sfqnT+tNaVbMlZYkHm1sxjLBV9QCVDOlCcRQwTcYACFElDrBgBFTTxDuvwCsC4phlXcwS1+PYfNxTnFyMVcyGR99pnxR+BkvTgPCk03w/aM/gEm7DzCNKmoqZlhJVTlVCx9vo27bzldV8dTOHyQ8fv7+Cf4mo2gAGKms2k+2GfbDYcqN4Y2UVIRUYQk8o+urrL//Yj//Yb/7m7911ep5aOmRMZTI58kRx9S0uze6n9rB7OR0EzbaIdn04GCFV78Mf+ZEDizP/7Nd+6emvX7f2qsC9Lc/O7t9KHnzimMNveeDJB8ITU+de3Ai7LfEoplKCBOiRxFKml12DAkSr6LX6fntm1M3/5TEYGNpvBBdJP1IrRAckdOgGGhqlRLNIsNB0yx0CtoNxGVrkL8FFzOYh4NWMpKVCRSOLL8ZLpZp5+uM/8cDS/NInfvw/jE8uV0t1NLCQPwRJqdXS0tUbNZ+lhR0r5PMAQonmC7AaMavItRAtDLm0itbAqEBRZtNzEbA8Sn1EpJS9pNfPgDkPME4XhZg6axhQaS1rB52P3F+4CRXauIzFjwSMuRFWoxiUdBqqi2ID80a4CwcDTDeqUD5ddYc7zl57KBp8z8P3gzVw9sL5bLbi87tBx8PmjTsASF1lQDFMdjvozWhd6O2JWGx8ZPTw4UPo2dlC1mC21lOpFMPD3+5OCtcqoVxUGIK1b+3sMDfEqGJ8zuSKHuyW0FGnLxEfwYeEf3dichIbNXwUVX57v0TEzI2b15YX5onzIsgZMaLdIQrOlc6lw0kwx6LDo8OseoFplMsHlhfR9vn56dOnqATx8ssvk3zF2uAMawNZAbaNpRlDOsOXKxTxcTUQPtGlIPOMF8MNK8V0UCpm73/gka2JsVIhO7cwXauWMukcopjF6QP7pA/TtfGHB1GFqmwOL6x3PjHzxHses/sD6ZX1MDwZUCVMELjPEJ7NShrQCPFRzZ3IABOMwKglxqEVJNb7N5ap9BuxbeMngkgZSVn7lVNciJtMB55BfYPi5QC0BI1EsT96ioBQYZ7ocw3ctQwBPIRuarkrtBhATbAl9Aic3Cy4VhXmjRQPaIe90XaQVtMXhBJ2EsgKxk4rpdF5kFysdmq7ogEqOtAfdMOJgdNA+JRcKMpLH9VKdVBChrrLfsQ0LanARGBhaiQ3FRESVzBGB65nDLieJUULaRvWEjqCwwCnBUJfoyn4SUR15qICIgsQ5KhqvS5SP9ydHxINz/1hnbhaiMnyBQMQIKzACFSU9RDWSJ26KM0ACT2+YK6WJBIZjbxRq8IYpRhjvgGH3eeDsLKCsaVDk9gWCHEMVjAQEGWVGGyaz3wgIBHw1UUFR/uxViFA2B8Jo+7hbvVXssBpoZHgIyPsH9shAY1ENQZt9jgo16TVwKnQ/FCkGAz2O2KfbkfIBmH3oj08C0AoC5URlWhVBFFVaPLo4yQqAmo9sCXIrKylwDgxvFpC+p0Wxp1DC2+w4hhc814kDWaiw0SXu4mipN52k7IGrmbT06xYCmVnMjrrPnR8dvTucctoz1K5WSyvOwM935i/W9rHBcJ/SA3Gq0ycAajarEPkV/wwyFA0hQAkfM/YA/t1IsYlNghLFxLLqsQsr8UhNR1hTMvEsF5aKWGNpSJ2pWWtMTC8i/f08HavdNIcg39Frw0vNuf1Sw7zPQ+5fdw5Y4i7rtB5XUc/pPXqjfjrbTbMTWAlDLY4Lm5woVzBcaF5CjdEVof7AmWjyAeMz6bcr5RgZdVhmuY/7q9Xcd87D2KeJBAjG+nVHBKlqR6tmHmuN3+mJ/ycYUH9rjUsbm5jd5CP4gVlye0vN7o393P7TUu2gXaFcJOBJZO5EY0l8IpgS/V6XUvLC8Ta29zEZ/mgax/77g9yh2tXr4biVlCLwV3FtQQu0g/+4A/8P7/6y+cu3MROsrQUxZo+5LCUq5ZMJhkMjpBi+qef/Nybr91aWpjdXN2IRbw/9RPf7/9Q7/Of/qPA1Mhbq3uvvXXuiftORyP9RuGKUtvZwTSeKl4KWWW2IEZ0VuNMLzXO5r0+a+3p4ncOBmPwnn+4ir5zsQZKk2PiF/QjcURWDm9leGD8SR7yME8wvIH+y6rjD3NUzz3qt1D2gDAGRwUd47EPnPqlf/Tob/3WczNTR5o1vIrETeLbytttHgqKuWpdn5vcRdgrS7qLnQAOh+WNaF5hVyEkm6RbNGDljeBnRjYy/jJIIHPKzoFM8QqvoiOc5D1klmtgOahnUDb4sclLRB6Tgxmiyh4heottjvpTJq+g5GvX8dtDjoj+xB6Vh1FNjQ898fjDKLpYs/GBYze6evV6Pltok1vYbMJ6cczA+DvNKqwHiWUPy3PYPz01kc0l4XmUeONb2gYbRrFk/4HKCRwmDl2At8bGJgiA2t7PZPKFUMfmDfaA3yJ0iVAvfMtkm1CpCQaMg48Arma9PByP+X3u5P725vrK8WNHYOQQWPjx9etX2bOR2PDUzLQAKdBkez3UWygzyvfhw4cZorW1FTKRZqenFxcX77777rNnz166dAmqzn4DOJJMFiyYJpVIuxHFBFMkpTCiJMOgFy0sHsQvmM+mwAEJx0cLG2sE4wGJ6AvFAFFESMVKCcVg1WFdRJJn6NGNxkZG7V4QoTvVepV2a5gI15INi/XIJxrJZtWGNEtusPxYfPAtWCPnWIQDWkkjuUzhG5w3yrHWKG/5wEcug6Dd/r2WLmsRHoZsDVEX/TB7GSqPKqgkVWK8qQfJUoOGS4Rr4a0pg55KIhiOVUYNdgWf8BK24I3DqZOpdGw0xBhJIlfFBVDTTKWERhvHcAd2LYlUkit0CsuNEsbVMNQz2ZwRCNQ2o6CwgtmRVCfQPsX52ERmUu0RrBaU+8WywU0wk9BhAi4QDFm73MdEazGJopFMjd3mh/tiNOZ5/AajJ9Ap0mKbdXgwvMgFwEwUIHeKXGWwWoo3s4XCdAc3NjikqhTFXaBVLE0ikilMbWrFwchkG0WTIcKSZQTtoG1Y4QfEhPFsCu0eMyuObRN6pqZqAkiHMpb4ACor+lAd3FZKo1t7IX8CDByXg6iHkJdsWGzLmKWx3hJSbYi/tiP9EugFvYGsA7xuI9MR/Rq/YpsqEVWE1ipyK09hrJDTZd1lIMhKbimVi0NsjFawCkTEtBKYKsOq+Po2u9V1Os+S4IxOSjdjsetn4Np78zjTPF3iS6vddLa9k1gIPHzPwcDDi5bGWq95HWw6e7AdjrpJOOlU95kT8RU9lcWK7wN2jLGdM+Y9z9EAYuQqN9slsF8b2QYlFcwMSvCEQkkcQPaQt0+NpgM0nvaYRsKm+Nd8UFf1VqxXvVKHzSkexHq4LWbQB20Ic+hKHawl3unVfORXty8wX2kbasVC3Q3HHbyRxiV91/BNOiEnCMQeA79drgBYrz4q5ErR2qplR6i7ALCUB6wERdmrJWXyTN1ZN6fFtGHQNTWHbpiuDILmuR8ThqzF/NE+/dB8q0mCYNucdUKeicjzBfqubsNF7UbrVr2wU8d3wVpSfJ/N7ScjHjtSsZIHgxc4eiDenn72axa3vQZUe8Ozc+lSP+C6dO3qB9/3gYMnj2F7mJmavXzh4nYyNTI1cfq+M2+ffe2jH/v2aq1w7ebFzKYFWNhnX/jyx3/07+3trH3yTz75Xd/5PQeWDv7sP/j7Ew8t//pv/8HP/+xPnt3atrXLT72Q/LVf/KhtZu5rz3/uzDghLh02C6GpDCw8mO7jfBBtMpvHdFq9+9ZyZPrMtNF7zT4Dc+dgHPig+dKPZRo0mgQUlhuSUQKBY7j4ma1RbDnYShJo0Fhh99gIaUDX7XN0M1W7z1st1P0RhvTadvL6j/3M9y0fWPzhH/jPY2OxYHDo7IXroDQSKVhMZiIel9A1kIdVPaHn9xLvqNbAR0WoaYzx+lERgXsRcyrXr122ZQgUZIrLoGAYGwaLDaI3oC2cQYuDKBGMQn+kP9TK7HOcsKw8NGwhKOClwgRe61JZgthS6BlEs9cRhkSzXqRPoaD37jMnlg8s8CwCr77yla9+5tPPZPK1sWHIeW9nd5frA24v+gW/rLTqaMZLS0tAS7725huXLl9GYsS4SxYTZnVk7J3t3bWtrbvuPgOq5MJBWKPFdvG6Pxxzuv0EELibXUcNvaUCIgctTxBrHYlguGR2aTrIl6RFUSQCFC3cuviO4SfBUICYalI5nbXa7MKCj0oBNtvarZtE4WBOu/X8c0C+LC0tnDlzhqwqxkr6EnZImx1WTYYSNAHgEEcmtc8oKz8S+A9CrjD4SZikYIObALRKpY7a2Or0UdWptBNNjOFqhlDarBjLy51mqQf6EpeLL9mvXLlSqv35PQ++5+Dp+4W5pwBiHN1QKrRQclDEdCF5orug/UpZkBD87uMdkqF1aZYi3/JzPhoBmuUmiUwrkAukgplFb8jqYIWzZkXgZIVENmDBKgJM3imqWyAPO3rghtfrZRfBwU4XqjJTi7eDQSml9wSUo0Ih4CREgzh4qTTva4CQpsAY4pbc1OXyCrZU6rKwRwgoVI/YGbxoPkVemWnuyVZhYyD9iQDp4bRYmdAozmwr6Bi9EHO1UNeWgxtCt9VTKKBUJUUeymSNQCMPZZlaOFJgIYFuSiTCg9ttvDWi+nQSEwzPrdWZYyRMAu2YQIKSMeESOVUqEuyFFssYKok2EiLijCklJA8fHmZE+XpNlQgVB61160rq0g4h6AkzAESWrccNUDipSyEhGWkGswIsEF2f/d+qIF8C8cxAYzAgDpnsGlLC/PHwtANDvgMBDDEbqz1gHcZAx1gyd+SnSf2V9AZHQ3Ynno18ii6mLor6NssqE0ihJPwfRHQZUo511jBNrR7mwNgVRLIZVJrL2A3ImO4qo7AoGItDr/qGoWZ8xR8YZDMlTAFX9kvNkiMMZmktXd/vB2tzpyJLZyZti15L+RVLqG1LQHeQ2wrtcg0hifIbxG3RHB6oRov70gUssyw6VGKZaPHRNZrFarNABgRxSa46z5T9UAtWa5bGiK4NPrIDROJQ/BlKSaDck66oX6wsrXPWmLon9mX2gl75hflk3nPp3zwGI/HuV/Pz27/gSYODFcigIC3wys0Zm3d/ZL1IA+Zik1AE/DlLUeA1wtwYsF66iyzGEmU3G5Ax+qAxHjAaPe7dbWCV66O2jNkP5lsGg1UuF7gGQX1kOHkn2YQiByRrkjqNyuK0g/FGyOtqoVJzWgicJ3ye/RwCt9zuKBSA0S9GY5Bov7XcK9UrbLC2s+/3x+89vPjsq8996MPf/rUXnsVvcvjAUdJIsKgsLy385ec/Nzk3s3x09gd+5Hv20hu//W8uhYdwk9pGpyajCfvqtRuZjTQIwzPTi3fd+/CRk8deeeOZX/7n/9QfBNA98MT/sbzw2BPXLq8tPfzB5vrnlFCqkHcBxBruCMYZi0ETyuQwHKxDujZYjYMxefeciZSZ0TAjYOiXaJkZQE2L1i9/TDmLB9au+qRm8zJj4PTzGI0Z0da6S8/ddjXKLRzigagtn61Gx1KTs2Pp5Oce+ra/9bkv/tTHf/T3W/nCvXdPXzi/OTpioRIpiHqIyRTNRqzVrumCSguqq8p+Q8eMxQ7apuXJeoDv4gnkWwzR6ANwWbFGw2h5g1ZHF7iGg7ZAlCBT6JGsay1iVg/LiDB6JfRBbJrYyrgJq5BA573dbbQI8nqJZGPdkRWsUJZeJxJ1BnxuaByBx/fee8/VS5cvXkhu7W7JLuj2ghENFYZiwmoJ6kK7iA3FJqbGqU+8fOAAcBqiulQBd/ugYteuXCc96bmXXgb2cXp+CTZXabSAmSRygPcnT57Gu9xOplDg0bmxq4+NjYGLAb8v5HLAbe1s5g3gRD+TTbJUWRsTU/PEghGXkMsXdvb2WPhCYvD7xqcmy4UijBw1nVSlEydOkABN8CyaMWdobzAcJdiF8jqUi3BggWSb+33kd/oQSXEfMmoINZlMnfgsmAYhL8SGkScHvzt4+LASVfs4E3PFHNcVxJzlQ2bHWIHehtLhfLbYfdhuSa4CFJHai2xd5gzPASsNms7ccDGv5o2oz50/lhDf8BV/2pC8mot0d5agDM4ab+5yx+oiWqZ1zoyySvWv+TmsEIXOweVgYWMWVXoI7AW8RiaiV67kKWAPYDJrF+OGzxlQQhjJ7WjMEJMK6dpZpyMSIu82Eq01csAmUJbV6W75XB3wqUSJCMplz4EDTOqAHDZqhXag0wlXRf1F96X9LGKJC6w9AMUUBsaXhCEr9ID3uEVwxPqCPji1GQola0nSMLhusGqzAZzwZGZEnJkhEIwtNSXzCJ5owByYkPEo8HMU+nY7wIU0hadTk8rvodCATZouYidQX+1m0EugXAjhhXwlNg9oLeB+oEdzQ7g7fBXA6KibsDIRaTEF4sTYmMT5w7C7giTFmStyyR9cAYWp1R+KzlLBNLtfIzTe5QxEg8Mj41Ox8KgD0kohCcgFJbOl9aInwPpRFLg5G5IuQ3blC5AwhQUXGRnRFy9QNUcvKXeBNULWf6UzofXiP+X3oE4r7UECDpOtETEricXC58GBZgSLNYwO+i7TARcypsyRiKH5lhPyPhJg1LIFmg0XSPNJS7h29FGinA9avPnC1ov+YXsb51+JHqs0BSXLWCwo911bhKEByMRmbQEYJLoIWBH+T8rIIWFQOoEMSCAD8QVi4MYwbdwkdNe0Q8NKWyUWsjIgnIqBZtXy5W0NXax3cIo3Iru88GNIqzmtCw1lN51lEgbCgDYF99ZPtD14b/rLv3qQnmCGy7zqSk5ymiEx3FfuGA0QCxpKKyosNorLDfc6SWXUrJJpVewW4wcMWF+hBMN9zSsUkB/AfWWmNto9TzCHaY+e/62nm5YwO5w1XAqxkFjiQZfUCSYeTye9on6v3ZquUhe13Ha6qh2Qgev5tqXssCGg1Vt9QjFYVmQDoI0TOYrLKVvIcIdYIuaOBbYyya7D+ugH3nv4geNb+7s/8pMf/8ZTT5+47+6b126SMperVLzhoMPZyZSTz7/y3N0PHB0aj9h8JJLivcmub762sZl0xi1//dTn52YXfuhHfuj66hVH2L0wfdgXoSZk6aGPfecr2+vQxVsrVz46HnGUU2DwMAQyEtNVljS7xGwifRp03yxQSNUd+Udj8u7D7IjBvIhmcJPBEEGv2HOwKS1xeSckjrFzAEFmKeMIwB7BZOEChooAXFsvtv2JRG4jE5t1R8dDxUzJH97ExV0pff7Eg/e99Nq/+sSP/tY3v7G5vDxNiNBoLOQlfINAKzRoNrtWBRuNqpckTWBqhoiJicJ62aJ0ReNTpFATQUI1tuFA8WVvIfrDPtRgA3rPR95D87lsdm6OVkPKOAkwMKV+4TvYVaCWdEd/IMw3apnkvkLTg36yI/BdeqPKUUaYrVaKuA1YeqSckEn8oQ99yGb92itnd9rtSyePnSCsqVUsjwzFQAxBR8jnSOR1ZnJpVCaMzBQ6dLqRJyw+N78OHT16nIzL//hH/+XipSvPPf8yGg5RDlFcG3Y70NGLS8uYqDGzM/DFIrlb2DutuMwxGINvGfDNQ29HpsfR4YjpYoiy+SJhL9gDKFKHjnTjxg2gLgM+/2gijhUEWWx2YR5b3er6WigQVHoSOayBAJQZ2rRFoPbuHgPLTxyWLjjAIG2Tdt2g4h4YBmIuTuLQhpEIGDhKRpBbeejIKSyg165epsCO5gU8OLIRsCFQBYf920d1KcaGRlLp6o1r51EGPP5Ib3aiS9WcZoUJ5Q8WwpxBPOkwrYGNmfXHkkVbNK/QHvYkBI/ba7lxGM5kaJIhMRBOxU5ppgff6wpGmAXJmmX6+WOl6Fei0mhsKnWJskYRIfgTA0JAMr5elPYOCj/2W4OCVUpldrBFM/7+ADhWDvRzhA4CKplUmBy7i3RNlmBHEYJdoFTdhFETeSIpAfu01myjj2tZ+OkwV3jEwEBKO0XqmVIwlhCfGuAAd/3EDBKB7wR5SvDNfmvYJBOTryaTNW0eYNDApxmsgF8+Y7sVlBmeRuBpY2vtit1rdwKGFiEJ341Itb+XQiClMAEoKsMJPC8jLAsxYXRSyH/U3+16CrZetSTTHZsAmMliNkfZMSRbcG2gdNzWrgqeFFJAWoKzYKsjukZaMiPI3CgPmBHE4ce4UTnCzurEDgVqsW03XbUKc2U0QVB+bMzrCZNKZFP0jE9BXIhnuNcQafHsEnjmJFa8iuLODEFxVLtR7Jzp6m/dugnhh967sLph0SIWAapPBeVGyU2qMp4RTPOI/PABorBaOK58knrMLkYPFVnS/seqyUoRJWG8YBN8P2AN2s4wCC1XClmaP2ADHOV0c9XhbS+dmV68/4Rlwm5pXO62cpEZd89WMWFSip7HTGrWJERDhk/JDJJOaDZP5cY8p9UolnpN/kHOEUGEvFDWSFieKkopskvrtDpZvwyEPCAmKtssY6k70KJBD3Rbade6Wj0yV0iC0Bf0S1+YW/GRU0Z447aGXuun7xzaBWbX6A7mh1w0uCv/aPHevhmPv8N9aSjSC6MIK4b7otKDkcIrpMoou+K4g/f6aC7QNWYMBmwbtcYIz4Nmq7FqLevetF/i6mAo9Y/EKMIzTPPYtbTO1gL41uokAwb3Y9Vmu0XRmGSv6yQdUjWi7EEKiBKiYAEMPZqIk/ewm98rl5vAw1DCFfk3EAn5w8HY6Ch+iwo4wArj8m5v7dYqzxJbe/b8G1C9Wq88vXT02F0HefYXv3Q+X4pvb2+MjFIyaHVqAhR+TyG/dvPm3mOPH9pcKT71zGcfe/K9cweHv3k2ffjE0R5VnLuhtqW5cHg5n7/8wY9+2HPrM/aer9MHDLOHyIbUxbizUfE0AzPewR2NGcwGUWVf1SW0MiDqMOtf/iktfk22GScI6Z1pZOLINmIVU/gG9gsHhcYxN5jNjISGEO8C0QqiosRsFg8ATyDSg4MLPGuxFxsJVPYrrlAzHLFkCz1K4pYr29XyK+DW/MZvfu+v/Px/efWlzQOHRpNr+85YiBANVijgtniDYBM8BNKkBghtSb4HyX6Skwn8xoqm3CSK8rI7sMHhKoZYwKDaTSapbcxmoCL60AOI+MTlh9qHsAtGEws1EQnz204bzseOwoPLfYEZkJmQBCfOuogc8lJWwQ7HYaPRGkJN0RYIOiJ1Z3ZmHiU4Go5kUn9QKTYIO8XmGvV5R8cPFZP7xMUEomFGhwxd1CoEF7hp3OUl6mU/TxXeKt6HY8dPovv+wi/98urqKvjRAFjix4XwjY6P3Lx1Axni8OGDJue4hOYKcSa9ikwj2HAlnyVKZXZ8zOf3jIwMpTJJdFQOYr6YcOyIoG2T2AmFL2YFksW4MV7coUYxR5OgZZY76x2RsTM5ia46ns3m3njzTWwPFaocQSP8FIv12kpy9tVhsZmkCuxg12bJTM8fICCMqg6+8NBwIp7a22DLAsREjoywK4MBwn9cAXsuv8MmdFn7Z1//BlJ+aX/95JkzUwtz1HHK7KVhREvLh4H42k9mqAABqVWMjuAJe+h6SBNMAxIWsbvQKBYlLTYmsAHtQHXW1sVtgGYIczF8QdRM65g1SIEdrWfuoawA1gsWJxF0NC478Kc1jAn0EeoLWBGriA3dKuZALvT7nJQks1hwhcxWa+VoMIog63ITlxcD9hPBjgC1armGKwB4iGpxF7bIHFTyeTgoE4ylFgQubLOYaVEQOMmy41VsmH6g+uJclYWACEnCRWgaMIatbg2BmVK4Vr83AJsDagZoJwz++AFAaeGHVKsDfAkZAiJH2WDqM4idKEyqYG3T7K14fGinuI8kASYFMf2pdJoNsxhbhOKXiplOPY9hA+QNWTXsIInnIHZT45PRaKiBU7VWi6DmdqmebQcQlUBpwKVQ6VC8iWSkOgraJqIKvAPqQGwUxgGCPgLusNsOQp+v33b3ahR8pahYv9uwAawbDbNKhxEJmB3SzUX02assEGiqFCLNCjdEDmL2GS8ynBkx2D8FHRqlUpU4fIosCfKU8YEYDegR/AECbQu4fHwF+YfxCipSIVoUkKZb6F9YswxJgxGzto1/At7qV4GzNhH48EpAS5AjEE9cAQ+A07U+lbuLDVvJ4m35gYiP1Z58/IgjXIe9Wxw7eLQ7njqdMDhMaPncAG7JysUahoNM1gkCViT1aSuRFtXuUDWhQqEIEqSkReJ3h5yyLGiLMpKRB0CIgSMZUcb8BnKJzcxwX+jPbYorZmkGjt7Dr3gQQyatVNo8qqWRMfVzw9d4HfxLS6BiGi+GQatfQ61541Ujwj31/javRV7h9rqpkUpoFtdyewVYsVdQ2iTjGT4K5WXGWCOK4oYTm/ecgQrReEWYkU+n64nPMwI47JXeMtc0hEcahqJm0U491cjEatXt9mgva6LVZxxBWJK8XYsTGOyO29/2BooW6439VIrSlfVeiyg3ekDdDmwwZVNOymqLeAKZ7dQDj7ynVGylk9tlB4U75ZNvluspMpCajcXlBYpQJrzh08unv/9j3/8v/sWvr2zcXHjgzPWVqxZ39ZtvfKPnyoZh1jHvXmorvGW9dfnqgekZ+g+a0DeuvhIIAfG491M/8xNPP/N8pnrlzbMvLyyhwWVHRyOWjufYdKLb7N1zcMYNjqyjn6+UxzGethpeAu9aFs+oJZ+2pNrdAw8+8Mob6y5vhGyffift7O3EA+YCKm6UiC9zEqOa3s8mRoMUvGO74Dgxsc4aPFDTLZ0hTtkd2V6fwgZaRTA/+LAopKJD2B6EOBJ6othKZgoBxVWjvJdbIBvdksctQyGTReGDSqka9Ibr5V1L45mJQ8u/+3vf/uu/9tm//Mz+e+45mbuVYu9m82uEjph8h0KlxMxgf/LAKpqtKjqlQliblJyhOlCgXicAHWxFgnzIG6Y+g4u0veT+VtHpoSaC2w9KLriqEHLJBQR7UkACUYJQUwVCA1iqvdylNBAmUQzYrG56DRtgg0EcgiQbxcdx9pWb3empWRTft8+fgzJKjWGYK5XxEWpLzhw7dPAr3zh3Osz4WLOV0nZu3xvyJkLeUDSC8xW2hcJLYjO7BOghHkQ8FEiU4MTjx7X2fb/407/4qT/75Asvv0D02cbezVg8fuPGOdK+QUxDiEKMAMbyOz788BuvvZlNrQ8lAkszRwgVSxw7gMF9c2vb2qyPTZHRWyBsGdQtJgaiTYrH7OQET4H0Y3zGoUbCCSzB7fOnsjnWP2HOCH+qBtxqT42NwoHZLyePHiRL1wLOB2Tk4oWzGYSlPlWUFw4einqD6MuUq7ETOnj9xk3je7dhvAZ/ZCyRKJWAuswDVI27bi+dBoMYPYBdineA/UpKF+BKIG9vrV5ZWbvMuJCwhRzDThobn6bkDrFB6NnofMja0GHs1XI9Qm+tJIppedFi/pem8a3NjORnSDTbGcKo79nuUHdoMToV5IYlKiOg+DTIlDgdoQoysskmTHAfS9qFYNVxUX0RWz3SLnp8sZAu5CmSBQciEp3APFKJuhXckA25KiSvtbq4SWLRsNNly6QUG0RNIgXiEzplIXpeZidomhxmBKvY+lyMAadQqZJ0A5KKytfjY61VYMnQKsgrVM1QdoqxS7NDAaCL8HXQxNgwRJ8wIJ12rW3F0sfmajbK8KcaP2M3NKq5XiNTze2DCdVo97L5Ctm3Hl/w2PEj+VwR0YwBzab2Sv1OOOQDCyO5tUNEAGCdjENqa7Wwq1hx2kBn8ZpgviPnBwYMmAacZIDJ3CUoHpXX5iQ+C58KZuqA2x8LR+G1wFS1qzYMzvh93c7wSHxqaHY8EBzCRgu5Z7ljn2XyiLdi5UAgEDIw/mPF5mnKEiM2DOEJRt8C4KRMKBkV63utOlj+bgwzQnuXJkcLDUsRuTYEnNOGK5OuJEUXAs8VwEE0WUL4uSHvaGiwZ0QdTQTSAmBeXnskHgFJO1fLoZ/aw85kbTvXyjVd1dH56LEzBxKHxywRDOQpi3Ol6yxJzkddQq1jkdEZwMlkRYRWIP5LxWOegU8TSSMCjj2Hogu+CsUuYL140jE6DYzzWnxol1rKSIe0l6UKm9Oa1Uo2B93S+8Ha5szfeKPSBWL1XKOJ4EJz7e2f/o1/eA63uj1our+m9c5TjE1AN7l9B9qjVoi2y7/L9PCVRAqTy2t0qAHrlRkCNqwaR6xoRDCGmiXJ/jZXmkIVeg/pRGsiFkozradr/9Fm9U4Petch8UJ7Vq+ySg++MiICsVYojdgQUBQl0gLpVuk5U63mXrMLk2oSwslzkDSRCCANyNBNy8Tc+PrGNgn3b795tqRcz4mtVGppefnRJx95+dWXC5WM3++7fPkyfrtP/vGffMf3fC9BFCNxVJzL165fypdT5YplfMr6+ttnjxyauefMXZ/8k/9+bT546viBW9cuA9K0emMDTM2piUgyU/3Lz33yiQ+8z+MFbrp1+q7F9fUrnUb0kfsfBW/uxpWtI0v3EJR349bOjDdOGl2luRcLOCzOzu6KJTprWZg48Qef/MZb1yyPvff0seXJSq4S8wX3ynmqy4VclmAkUclRX5eavpCLslY6dj0zmyxxjRfTYCQ3CBs0zijKGjbGWUPN9zBrbWRWJKKeLBPWKjFxTU98TOsGEVqJPUQBQlC8PhfVAKtAUVkChVrqmXDsyK/+0+9yuJ/9wqfOPXzoEAt9bDy+vbNKSlU8MfLmzRUqJ7CMTdlp0U+ego2LLUxEM6RPMhqrGqcxq56pI9/fTlU6oqzwvyP3U74FBZ1YV2B+BWZH6jL2Xqik8PEUwolQv4/MrhWBu0FiaxsoXmVtDA0TzVRFmABHogrkiDMSGe100usbt3BQJWqxIFVuOz2AddG2AGTe22/6Yx4bJWWBFxgedgd8WATxQA+80SBnwQLRQ+hCLA5GtFgj8hPK1OGDh6q1IrAkyeQOtttq3T3iGAUIgWVz6dK1men5a1dvvPnWuVw21WrkfG7LCBlFCB+UU6zXsvsdQqynJiYl2lgs48MjBBZA2gMeL4lzhXJlL5nc3d8nlFemeKyMwSC0Bcgu2C+WA9K9IPXteoXdOpKIOjCDZnN4jHNgY5KgCC+AWwj/q1IlUHsoFp0YHyX5mTwq9lk6uTcUDmGbQFfzhaKdTmV3N1Ou5qnoju/Y7QsN+2OyuSma2o1UW22UU4X9RncIKtFodHKZHeBOAr4A+U8ANfMwqbTYKrUrWX6KwVYsJgRIhFhGkMFq0yvTZQ4tw//hYHnqeggAFBg9icWj+h2IXQw9nWfREEolWgSpsPYAAyN8g8K6GBAwsaMmYoAvFgqwDS10fHqgXrQ7RN0TmIfNmr4bazx0ooVGzVNYSciGuhfKiZXsX1pMKL8fLGcAKwhnwxaE6sdaxCJEuxEdEeKw3wqCgSpG+DKVrSP0EA+rVAZBHF5YAuBS+DpJHa0Dxkw2FXH0pOCQ7s6iJ9ivXsn1O2UehpiXwzfSaDHm42OTBxYXkvukkmdIc2Y7sRlS1SIsjfg9UmjlRwCFpNkkx89PeIZDfuVGtcpYca10FlWoleWbSCuyPWDHyGpQAXs/gJqOqYL6380qBueapY35PDIyCTruSMAbYxX4MPKg1CtlichksV4BP2FaapHzjMREoIdMjajmWAQwNrXKRYKaKLcHQAyojdi+wEMwG1g6nLg43mYVJpf1gOO2W02chH6L9cLTUAiJb+Ke5npZQBDg9FOqQjoFrVltVbZT2x1nyxd1ta31VGXXG3fffWpx9P4DlmkSqfct5fV+MweClc2P6ET34TbQEUQ1Fp3WALl2rEreDmQ9MRUIItSCiEksEVVqYBDzDuOQMiwBkKUMoeLqgWhoWsuKM2vWoMeYRc6tzaGVqu4OPmjVaXXyoi04OMkrPzGvOjEgz3rHoSZqsPQCGdbu0Jl3HeZuLFWNIa3gT3tJCrXkQJimLIu8DNgqlFEGesmRfIvKTUgBPiYoOHBhbBo8jQO/r14l70B+TYN1n8FT2Ax6PNuPgRscRve93QmZpjk0Y/pX0ymRAFqu6ev2KZDs6ru9PCdbquxTFJN8U4BYQUAEootdrNq4Sl7A0XNrdXtsYog0+Z29zBMfeC+UanxhNhKNMiq1Vi0PQJXLGo1HtnY2sMDdf//9mSJkOTs1NXXo2OKtzRv4aObmp7nRrevXsqlsPOpcXSmPJlIeXwhrGeZttNWdrcLo5FQ6k0umtuYXxnb2Vs84ZmAW09MTkObTR0Y++uAPphvNYGR47MkfyN96bfPKdW8z4LR0o35PaK5ftLvX1tOX9y2HHlg68uDDtl7l1Re/sDxpOboQJdAhQ1U5WwRLjNfWSITcqVQtFNbiYlnfnnzkGNxg/Sy+XUy5cBHE28GqxKQk2YddwcCzUbTu0EC0thBN+GuQL4dFL2axDyMuMgPA1yc8wSG7s4AuW0xfAzzL4s2gEP/Cr7zfa/3yhW9ePThzpJArkcMD/b96beXA4ZmbVzeoEEjUKmoRc40N3O0jQgWCRC4nOOcsElWJIaeTXQ+9Yl2xbLC4IgjjC4PDCTuaqYLgESrfa4sXKR8DTEUHVJRAYvRksBFEmjDBdKhp5sRyOzk1QVSKOyTbe3IfDpvnhkKasHv2cttwVi8oSbVbgGwEKMiIYkNQcjOPqMKKCsIFu1RWrZjlJQWANY9PEACsSrk2MRGFRq2traJvEzKMYezkyRMVar5Zetiux0Ynjx49iQwIGNb65rbTFfjq175+4/rN4UQUZSSVzYCdAGWj6F58aCgDqFc2OxEOlxsNtEaYPYUSdnf38x0WnhteiS2avk+OLzA42IZDfh8cowjuZKlANS12ANRvp5inO3BDRzA0lM2tg4c9PjEN3iFEHYmK3yN4oQKhsw+NDIOVgFJL+FVuv4ugAdjbzOgi/OTq5XP+cILMl0K5BGy21xuOEt9F9Q0gOTryL4ai/qEOgdpYtdtuFwHm6WJhX4E/8toxebhzjd7pdBFqIFpnGKkhgRK7RHUGUSpabnw3OGRbMzRI8j79IcgK/kEYj3huE5xIGACYVkEYsOgOySuYy3kmpWR71ly+GUvEMfQD9VXIZ/34fQEwFJtu4XqkWSTFoslRYRdxG2mNWC1ENxgenhEkF1wvnOe5WDTYBkJlNbZvrqVxTBJNkaaHdUVFbjDeofzpQIOqC7NASXwiQZg1G13wtsl7J6jNrN02FgH4MKG/uGkJiGKEW0QF16qidqirVawaRXCssEaQF0uI+sjI+MTMbDAcqRQQ0xwbuQwJp7FImIrNhVxKgBXtNko1jH9g82RGADauNIkzUzlnEu/JjUPvkETBNaolSBaUs4Pc0kaGQI5AnuiSb8zFfmfQZfdHIsMjQxPD0QlKeoDKQs148j0YbeQU4TmiAMPVzQGZYL1Sc4b7Y4BDgAUfhIIO3WIR5w9dYyPSV7E6aTdMlLxbKI54CfjEwYBCrM0q0GWGckOy4Y9MANYCp3gJJg8JVVhqTMakvVtqN8lJ8UVcw4dH8vX9W3tXfXHHgcdmZz/8KLZHizVnKV2u17fatoKX8rFhF74bbgPzZNIUnCB+wS2hh4prVXwcbn7YM3F8oMazLAAnlXyC4wsqhLmGiCoEJ2iNkQK5kTiMJLlBLwZaKmuDj2bdGr7F/JuP/8NJGJRhzIML9crtBsedX9/+ePskjYVf8uFv3s3sjttXMrLmTyxfKi+CBNKK7OX8GY4LQ22jXPIevqsIZ0aUC1RWAckSyYSg4oHHV8tQyhbXiE/zYN1HzJT3tJRXGDDvBqzXsFy6YDp+uyc6JxsB864zTJ+mFp+8LN3WbgnofMrEEPJPA2BCGmIMr3JaICMQg8fP0a0y2TxrKxDyXrl+9cCRI4Qinb9y4bNf+dLcwpjH5ykDDuP007RkJvXnf/5pkGDQw15/6+rMwuQ9p+/71Kf/BLpPUuaHPvid+UySrKSSBazWLvSxBOxomqTEwMT4OMVlKboOSwSuiId+6YvPfuD99xMIMT4+vbuTfvap37v45nq/VlwYajxwcHw0tJwYd/bLyUtbN2YOza6ms89c3D395MnRuTOf+drTrdKeG5naatstt8OhGdSTTD1AHU/c3EQh9N1EzJDOp4EBDV0eBzOYiBM6xdLRiDG4GjDEMKZRiRQsMSwvnEVV5SJWkuRDi9vP3dyOqLUHeEXVcmu3lMvthQKNu08uFfZXE9GonRzg8jZoG3D9X/1XP/Tbv/CnL3398swEFcQD+QwOKWepXPEFMTy0IJ1CQudUH3MfKxN6JuhKDEeQRIyp7HV4jLQaTkFdFcIMEZUpD9O4di7J3DV60QPVCGUGykZQC30hio5sFI8PLQ2ajzarohDQSJTU/eR2ZBj4qgRrEgaMF5ltRqgNAZ7oHxhZIZWARc/N4Dz0s1pLjUo4Fka15UHk5kLZUKj4CXsU2/XO3i4YkNAVTJ60ifbPTE6cPX/BVXWhQ0KWSdUl7Glqbm58fBLt/fjxM5cvr9y4tUJZXgXZeNwMbpJMYviCyz0SS5AyQ8u5FWYVklUcXu7ZrVXyqyvrTYAKiO1AImm1w/HQ/NwMLJYyiCASAh9p6r1qmgA+gtgivmTTaYpBOkbGpz3+MCy9LjhBxphSzCI85XJVVMNmBQZiZGiIziD37/k8r7z2MjyY9FJ66A3ET80twrFWVm6OU/kLVGhoareN6MJYOP3esclRWnv9xi5hc+QK72fS3AS77uTMcoUiJpLHleci0iv5T4I58YwSo+8cnIcoiU5p/XGZjnfe8BVLEr6BSRMnQkd6HVMAjm87EonRSZQrVbwUTgZ1FJh6GdcYDlzfw0NRl31mf39zY32dVsM7aT+MAnnCC+JmqcKCBnzK6WiQFQ2oMvMLojdiDHIfEwDh4OZMtg8cFfRLxAkJXHUmVRpSXZ5gsCe4K59ILSdnrFWvkLXODoJD01XJb3VcCHWkB8OACduW7CFe2KyhgTGTLHNWnMRjZfGSHVsn6oHLS8USkQ7T4+PxWBSEkd31VeTHsJfA0V4ZkLJsSiJxpwXI7Xh0hKQBVh7tQWsDilJOXYKogSz2+p1UlADDGYWGBG7Chii+UqJEpR//JWy4AJB2vYO1OYZtIz4eCw9HgglirJr1DpUeSMwKIuVUIbkIy3og7J+ZYsKoG+UP+bWvGESFTFZK5RxLAo4dwslBdQdMBOA68w4iDzvQJhOxYab51TuiF3djSAzl0f3Z5lwP5eYNdUsgDWIX/A4JSiV8NTS+iRBbOpXfvnpzMzTmufsjhyZOz1oSmBiu9HBfWYsuf8M7SoaZp9dIV2oVjyuIGkHAGzdmMfIf4gj7DeAlPREbLFQOelETmAnjhktfT0RaZhWiLmsxoo5oSZr/CVkbrFIYHoR0wGa407cOfs4HcW9z8P7db+580jkRXFFe3ZBHDC5796sYn3aGvh686ONtNYrr2e/mNI9DSjEsU0vJ/A2YqF4NN2UPwnTZpPBgsx4QQgwDxsaDDVFar/ju4FfMgRiwfLi8mlbKLCHWywPh8lg/GEOpbfJs8oVRi9UH0yC2oeQU0nXUOadXucXMNJBolXaNFDTeEFwh+wyVLfCWIkHja2ChYBohPcblxG2EcZpYoI2tbejMfQ8/eOzMKZfP8aWvfulf/tavf9/3feyBE/cd6i49/+LLn/zkJxMjw4RdjIzHI8HIH//XP17byD755Pueffq55y0v/PTf/cn/8yf+/u/+7m/hcqLUHnuoUk2trpW//dvuXt/eIVbuzTcurG2sLMweKpX2J0cXkSK3NjJ/+kd/uHbNMhT2RXyBqxdS99/9eGA49s3nv3ZifrQzbL9Vt/ZHiZQdHjp02hYYsUdj9WoqFou6Q3Z3NP7ylZ3Txx5FPqD2p6W/l81s+V2WTNnik+xJRAvQIwRUATauVc0CYJhZUmYeRQthdtpgZqQ5qfQE2aiN14dvbJZspk/6NBl82X7j2r7l7HVLJteOhlL5bPB9T5xOZV4fG4v4ot381lZ0aqK5/9b/+bNPVKqf3bhRBAExFJ2qVlLJTAVcDoVlWRskMrArWFHsCYU8qMARITiaUCiGkhjFHTWn+giun7FRsAQgM9BIrTlWowJB9BspzW0p0HhqUcPY72gvsAwZp7tgLFcIYC6A/uH1J+KjBw8uU1Dn2tVVoKmIhC8WKuVKbma8imkXHAyf19soI9K4D504hDdaqTcWS7Uq7GF4ezab5yRmS0BG4QVIyNEokXmj8FSH24LPG9qsEg5W++TwPGy+WEC52RweGz9x+u6//to3YXbLhw6RSpTN7e/spjGJ4q2vo1WafBP6Av2H5cPVcW47t7ehnEaWtZRy2fHpCVx3cArye7mM7oNGns2kAct0RcNMMLoOEUaueBRjAEzBQUC/L5KoNS1byVvo71ATZ71Tqd0CkJrBhXai78NxkU388cQjjz586eo16u1gpYbcLBw4qh1psbzn/R/d31grFLJViu0QdtzvZ4r5vcz+TnJlOObKZZPEIgKlRKPXb10L+SIYJLxuXqHRdkIVsbmS9kek+js0hmaYeRVhY3pF2rR1+ZfzvDdLgAfrwD9HgiaBL7xHUkOydKMS43WA5TBYTD1yuermUlmqXAV8lBxZHKrseQz7OC/WVkqb6xuKfhKIK+Z9P/Z+7oUJjiRuNFdFVIH87HIJH7WHCicVTyBTLhzGOABx4QyiuLUW4Z4Ip7QH3chCWi0JRJRX6iMElUDMUPYb4ipwlf0OTKlazmMFVaIPADbI+2bHYTlGp8dFysCoEVRJ0vQCmIEEisHWSbYOK3Z2cgoHQa1chGXAPSrFHAlrzD26Zp3QAMmqPYwjzVoJmorvFxMVAQryRjtAJw/h8ZSDmdvjWqUKnlVQaC63vwWwB8laTagnMXLhoUg0ER2JhTE4owEThOXC7MTQQWIxAGAMU3Yz0jIjDkIP2gnYnJTsxm+Ggoz7BVcGpbjqxGsR7tjzOe0+wvQYRJR6vN/GhK+JQ+gaGGs1oWK5+k9TrulHuWWriAFhbeU3WgCWWpfUUDCaMMe1+3bk5CrAF+V+uV1ZL3bzI3Ox733sSduhEYsl12sSh0nEWAHkbzf2RStFIXKdWhnAXl+IQsKwW+icuBCbQ21BZYczKHobexuVYRpY8hE3oRHIekTZsKKYRBmJaCFCpEGZYVT5qDPiR+KDajsCLANpJEid/JuHLr7NIc0vzcuA1JouGrI7GAa6fputDq7UCHDwJYKgBlCX6Xr+DFPUStJH9jA00Oi7iHGcITzX2Jn5hvNaWXzLGW4iDRitU0ZpacDYJYStgTnAqL8owTo/YNgyO9MmVFaeIi5xuwXECYjrm5FhkHRaQgBbkOfQTC6EJvOGP3NSw+yBTDHeEPk+4Rf1Eo4OL/41B4kTDDv2K4NILDsKlhL6QLEVLCvoL2CBoP1cvnrzI9//PfFEAo8nXqBX33j9X//OvwZ+cnZmitThCxduQI4np4ZvXF09cXpvfmY+mcxGQ0P7u6nv+NC3BYiQ8jinJxe3Nm+S03P1aubxx08+9tgUaBXb1DXcSlKlJ7lT2dvO3nX30etXkmDBvvbyG+WS5a67p5I7Rdb6yPiJhnvCNTp9PfPlwJD1oQff99bVs7hHxg7OVW2+pdmFH1havvLa19M3X3N7m6l89Vaq+qFDD0zGD++dO+tsrRb3ayPDhVI6C4SPhDkKnqKUKGS6RRKfYW23h5axZAlo6Fj3rCkzv1on5r38d9BGJEk3MSbuYq2arBNtaPGGLbOJMdQ7oGjXt+uLs/Pp5PlIsBudGmlmzltsUe+Y4x/9xkf+3W98+eVnoEKEBFsSsaFKJQVzITwOegYGLfAYoj8IlEAHGTsf61a0WIiyytvhMIGxLAboJMY/VryWFPZC9g/rDj4CaxZ2AAFjGNTaFl8QzHhdL2uIqky2svkMIdO+xBDTjjWNAK54DCXkVj5H8YMIJLpcqmHpjYYDZFJAg1g6gWCYG6AEHTx4EBIKuiSchYwgUlpiQ8PEMMPfR0fHBvwStkob9vY3SSmmEmKJktFNWECMPOEV5Lj9nVINBadz8NDRmdkpIrmuXr1K6iiAgx6HDe9fCOgMCd6K6yWs1RuJkVlVobxHvhQNx0ZGxmoeQLyKe9s7kD3YzWuvvIr3l/Qi4sFOHD9KxA8wmnBAaB5v4Kce9wgD6Gj2nOUKNYP6/vBwMDqO/Iupk15t7wDJRiUln0JBbfYyPHl1hbzeI8cOpyniVaxwDbLD1sY2lt6jxfr4xCh4kIJRJgmeOsxWC5BgjTZO4hx+LRfITjY7NSXIetrbWecnoyNTk7NL8eEJIZpS00RmC5ntOcza0uvgkGGSORLh1XH77Dv/EGnqtHidbkyF6PcmHNpmRdh0+aCY9JA9i12a88T0waq5VzQcQlfOZlKhoI8iQkgGhBbk0hloFQ9HdfZ7aSfQVvijVTgd+guzRBmESSMjIxBCF+C+vDKHmJZRx6jGxBLEEUNibsAHMKebJTL4FXZ22HG1ksfuIH80UiusmRRglWmvUuSjS2gTYbEQKpYtv0fTItER0w1LXuwRMsePcD/SPpwxHpirx+GMh0KcJ4SYyaRTZI7XS9QpyZP9jLMlRyxjtzeaGCql8wQjMGyMOSZvyo/48FRTsALJAV+xXJ/8AkAhe70JdiK5vo52HeHbEQPidGgyEh7yuRDcAMiyNuoyTTKPJMXCJJEHa5V60ENdKYqLEi2uioSy2aINt5u5nS32CSHXin4ESlp+IUKj7K464cQ62NEwNKg0e1cKqHFlajNKjh8wE+YcWu40G5WGQvOxMkDExTgIaZDZFOAdS7XRK3VtFXu47fLVS/3t+x87OnzvYYuvbWmc69hKbapR9Ou2gAUMJVwtbCpGTKWWunDWBhA+RIVxc4lqWCbQaWFNKFtFoZ0wDzBgSUI0D8FJqWFah1yrhYhVlFkT++CcVin8yPA9zkMYzf+8M0xIi2vwpXlVR/Vbcxu90aF/1Xnx0zvnbi94Mc9vHe/sAgbyW2e5hD8RaHFrjZcCqOXxZbDF/ThDeKphvbpSA0qLucYIHgh7ZNtpocFoNdUIbTBguDLGFMXTyiUsE5JeFZrGYTo10HHhH3oirwwHNx90QZtZ/+m8CLP6Nmg9F/Al9h2MlzJxO1G5SBEDHkjB7gT2IY+yexWEzq/NoTb3rZOT02+cu7WYiI0NDZ+/cskXdH7py19+/Mn3DI0n7nngXhxeH3roQ9dXrv3hf/pPiNO4QJqV2s2bm26vpZArXLpwtVW1fOq//Vk0mHjx+Vdrpcbf+t7vImUM0P8Pvu/h8xdfbzTzl6/uXbmSjIQT5VJjezsfAJM/FPlbH/np115/Yf3mG9l0b3l+en1lE/G93a2/fT099PqlSr25dO9jYCndKDSev7RaaDcffv+Hh8anaMy185cj1u702HDcnv/0Z68k3ZarqeL4+Mhe3Tfimcg0fM30dqtiifkQ7EAloUIJEjr/V8jhZy0Q7G8X0KkZASPEMBIMoCQdcWx94DRCjtJHEf0DDoB0qbRdwfnqtLniQbsz2vV5qv3G9e3NeiOzNOPt9PKUofdHgbKs10tve0dP/L1f+u52+yvPfTW7MHUA4ENcTf1ujmVDDUNCqUg7QDalWJnT4e/ZYOVajEYgRIOVAZkr0VJYWIRQUFuBTULUMwQXoYCdwj6iXAYwzsR24YTjYyDoZqVQnArxFXwjfguTqTVVkQ9IChgYnAysPwKJsGyTej4+NkyRVWCk0RiKliJqdN/vjociBM1hVyNxGDcBbSDgFFPzlSvX0MbRfIgCQ02an5/P51E8uHmBDUoZZ7Jybq7comwpKbduCiVNzT84OXvuwuWd3f3r11YeeuTRqamJv/qr/y5B3IqBwYmhBeMqjlwSlrATE8+FyxVGEBkeJsQUkQdew7KEesICSAsdIt0qFEAIwAhEODYyCtpsuZCPxyKpvV34AiFW+VwawyqB7Y6xyTkqCQfCztnlI9jl0Hdx+xG3Dd2kS1h005mkt0i+bGN7Z3/a5QTWEkhusozZtOVC+YEHH8FISIbS7k4GLyk21XIlW2+U0NeQR3BPdCusFDtLmWoChGWNjY5grd3ZWt1YXYNCUMvJ7SE7W7IFWgfRR2Asm2hn7VYtr//5GJhgtO5YgLAscPm1u2FdxVIWhE4oCH2mb9BJhCNYe6tNKDwxHPjt4H1N8qzJLcGEi5Ka3NvLZ7KIGjie+BqVFy5WClONFheIo1RAzrDiD8BdSuAUbVKYAcHlQKxls8glBPvBlcHg4bIShpVyEXhVhr5QQIaVYgpLVmnNVhNKAHkhC0HgqdIxpdpC+YhyxlmLMUfSLvZo/LwlCg/UTBqTNhZbD+USEYF7kfJUKpTRxeAEXClNvA5PdaYzKTSw8eGEx95P7pErZWN1VkolfAEuOI7VgdOB3AAkKb/PDzvBFxD0hQkYRook8MRhZ3W5KoR+V5shH/Erw9MTsyPDky6hVACUhXYCn8RJisNbrifs0tLXyfZ2ujDMw3fxJRPSDe3olXloMp/LwIK4ADQyNpGdWHcMiOTby8Ir4yRrQsg9RFVCbbAYSB0S/YWUw8dETvij92LGyuqFbg9MmeBNI6p0cCM7EOvgxJQvy1XI8fB1h0BpG/Pe/dj7LK6CpXeVlrRcjZ6H1aAy98QbmPvzcOZAECEo9MyfRcKZ+OZAsYVBdcoEiFUJeWNxoZQh+9MXmmRWmTz6mggO8wKBgdzw6R1GyLfSLm9/T0d0nU7+zYOTg/N33gz+/R9eB0xaJwfPNF+zK8yz9eFbFwyuMSOli8ULzes7Hl9JBzJa0lTWHe/5rRHwxID5iGasb2VzgdGqkiZSEpopYh4n0UUGZgK9vnN/hkVPpDlaqfof4wFTa6QIs3oZ1gHHNRwYIcBcpKbrkD8fgU0cm8BOYEHtJMhZ6owfsw/uKaINSXmwbclrGlUrMS9oAjMzUdw977v77rHpycs3r+/t73zlqa98+4e/7ZFHHkJxfeXqKwROktxBtATS74njp8Ymhz/9F5956svP+YKOkURsdys5NyOw33vveuDsmxfXV7bO3HVXMBQLBsIIpW+fTQ4PkWhXZ8SmJxYwaR07cvK//+XX3nzrlf/8n3/v7/1fnzh/YYUkHxbE9ZX9933oe8KjUz/5c7/5A999932nl9964eWvv3RrbC7iDcWGx0Yrhbzb1nn47hMvfP5Cz1Y7fcxz0zmcJEPO4mk644nZuWT2er68HU3YSlqkZA5B/VymLBjROnUSGVwWEgKlUcLxtE9MdDPjz0jiENf2kMRlBCPmikRqYkhclgY6pR2zIjDArgKyf63m6pV213Lf/xH7gaVJNrAv0G0UqK7d8MY8hez5yFD4V37jR/PZ/3j13PX5qSm0EarqQJVZapBSnBPYiLGQk4NH4XWLHJ3y+w4INPQQagaz5BV6G4mKC8JKFbfbAjIBXC32UxUBAXxHhF1WH/ZnrGKYmOota4AEWNzGoBao3nCPkkeWbNbj3nHNhIfmjhzcK1CBCocrlTMIpGXSIfuUjYGQ+qlymM/MLBxmFDh4BNGXvKFhVNpgnaByIEhDcDFu0yRMmKwv8ndK5Qzq5bGTd/tsge3tTcKQlg+duPue+9yXLlPvAK8wMWIIFg889MDTTz9Va3aGRijK4yHgMpdGu60Pj96LvqtAsHAkFIviLEE1p1ZEMQ9KZf7A0vLE5Ag/39nciMdieMTXVm9trq3Sdx+Q+3jvOm0quqVS+4gzgGo4cPqSTYTzLkdGehQ3SJyYsXKVquwOkJCxBswvLNOZbDY1PjXHLsgVC8Fet1q8RepRPDbGKBcLZdKFqSRz4MBiNrdHMYo4tNBnxyJNWOBevoyTAlbNLouMMCiqZrM0N723l3zj9W/u7q596MMf9VG6hwTUJqaGUKlMtaIEKwo0CEaTOYbLMKZG1DKTrlXx7oNlwqATldCHncNrCbDF1g9t5rci6cTrg1AKD6NGh9dLkBb2AeIENDWqrGAh1MJHOgsIO+VyMCCKZgwmFMctY1klYm3XFPTV3Nv63EGTXaufPHkK4lsslJLpLKVuEonhWDxKTMHe3podRGXQZazddCaNwBcMeKge1cIcDrVCNAOsgVFArwLCjgGF03NeQYwsVANr1e15qS0BvUHka4kWDqRNFjyWIBaRyh2Va1/begqEMzLTWXmkA5Iam9vfp80UNYFbE/LnwlbidgMUXSoJDIstw0jBZWHeAX8MQxUlCsFBbTYs6TRo0jVcvAtzExNxyk6OU4C6WKgB0cKVSBvVCsld0EHxSixOiMWi4abAHmKdCDACWgGtu4ANHEUf9EiFKEtTFc03DEk8D+rMrobkGvYqVYkDwgJNR3cWlUFMgvxIieaujI5MCrJlCkBG1uYuOCWU3rE3KblU6ZQq3Zw70p8/PL54atYyjPaQs1h2SEG3OCnZ3nJRWdhO/SaeSSwFAa6ID4QYoOcSLd23qsBa26qSqkhl2LArGn+Z36WjE11m6B0CAWvILCU1lqYb5mLe86LeGNMMO0VDQy8YI6QUcWHuLzRa3nPVu1/5yMXG9c1b6TEDlsWFjDNX3j64gX4qhjegLPxKXMvczXzHs3UBhzjuHaaLSjQw18ofq7IKdIixNZcx2gpyZvpoIF8hzsj4zGRBYzHOUPCC6SCigTtwnkoSnBw4gFmoYt5GwJADHAYu+q9/ODeQCqgDzb0YBB00jvNMsKJ6JQLdboHpL2IL/fIq8BFnq6MM4681R0bHDzgL17ehgM5yHZshYS0AwzSg7Fj8yJ6AqnKSDjbqzc9//vPkEB88euieuXvOXz6PGpAYjSeG4+fOnj/39vmJsUkslt/z0e9/4823nnz8va+//iop51h1bt1a/dV/8k/Gsb1ZLF/+ypdwNLtd/lazd/z4qY2tG/n8RjhM2XkMwaSfCnXhh37wx19++VUA9A8uH/7d3/mDdLZMrXPKZ+3uF4+fXAjEwyMz4z/9yz9sbxdOPXTPo76HfvATf6dUq73+1ttXLl/s10vvv/9kyNkPAtfVqC1OjMYXT+et9mylOjx9oG6pxqdPAQOS6mwEI0Fy6x22SC5XL1PShWpn7f70UCzmqPntVL0A5Yqa1ThqqadSESaNCvdiMOjXGojGAEtR9BJoR5cr6M61ah2LOzIaXtlItd2WPmWTKHnRtpbqlq881T111Gdv+u1BBhgVg5yLCkXgupZ1FJ9//Nvf+09/5U9efnrrnrsS6d26C2hEoMdq+KexHQJl3PV4VNeZ4irYDWFsEEyWPEoqblez/mGQ1a2tDQaWDqAFUbwhmojY0yjBJCQ2t3a2Mb0Sr44+o7I5FFbEvyG1suVW/7rwvyFsrgRqONN+byoSGecms7NzxWJhZmaGirx2i5/MC25FLXFYj8/fSaWTC0sHU+l9aD6K5ksvvYQOhlJEbWBg8YH1oJ2E3LJZoIGZTJUiUSbiOgguEmr6oaOnDx4+hI67urZ58uTx3/u9P/irL1iQ51DSpqYm/aEwa9/lI6mXVNkW8BVkimDiPnLs6LrJDEpMjL/8yhuhYJSCS0BdPvLQAwcPzMNWN9dXYcaYLZkmVDVAvsDa7LSWTp06lctmNlZXjh85TLQS5NoBbog2osUS9kexOLeo7GiioCVQ+IgcC6KDF0qwGQAvcxhl2W4UtSarzOX0NsYx1lOBTpBJ+Ed5UjQWP+Q5QoIWRZrQ/Xe3tlKbe/efPgmLvHXjOnGGPQdACuz5+vzMKB6aRj33+ivPkE5DfpTb6x8enRudPAY/wnNOLDBY1+xgCu4pE0IHehIiHwSIB94+GEVCUMkNJ9IKWwA2B5+3EQqFGVw6AiGAicq0L4JE6jp/yjxGikNvg+SQKcR0QtdI5ukO6ymYHMx5/PnUu1agPIZ6VD2kacKC4UaoPNyKX7GI7E63HPsISB5YJtGafb/PgQc+n6tgh4a4yHXZrbkII61RuB7li2WgoGRTbBAG3K2XiNlAq1AYVNeUdOEkZxgk7RvpDSK4zJFoorgoIZ/SanBqI6jub+/IC9lusNQoJ2IjNIlAwX6Tq6D82LDbFFCllWDqcg9Jrix5ma9cNl82XcO4Hg7Cd48MD00FvGE3lSoIu2g76kAIWdwu8gSwYZCd3gNP1UUL2Sv0Af+2F94dDlhAJykWWyCYFEmTL7HLMOAqCBL5poVOj2KFoqs1Y6JJxEeQi4hMN5OnV0aeDjKdxK25ERcplIFdr0FpJYJiXES0YruugL7ZKlNO3Ortte2tcrdUbBchXmOLw0cPHolPBWzDdksQvWzXUt+3uOsWRx08JciKxdGBNgGtRXY5+weHJ6A19r6T3A7p3PB1Qn3KBWonEObNgWimPcMqMNqGdHFxX6OiafEZiQFONmAkgz4wd2Ikt801dElzZuZLJ8UR6RxvDSM1P3nn5R3e+e4371ypFrIGDFtjGd/51TtvdEJPkMjCm9t/6LK035ynIcRma9Fp1dBKzQbsk9pNmCAkFElA4iuj+NI3IrBg1UbTRThgVfEn1msugGGrO5AKbsJ7tiJPkXYrTZp/+IpnsOVgzGo1H7EgsDglV/EFv5OZXyKE/uMLNVt1tgXZyWvPlitXk7XcDjU4GmSiY4lGDiDSt4HkiUECUYgYOCJzMeEAY46+ZQPssNWEgrHWP/j+D/z100+dv3Dun//Lf05tvUR0+BtPP0fOZb3WwH346U9+6kMf+LZ//U/+A5kyk+PTn/rjTz3x+HvZ0cndNIaww4ePXL7y9qc++WeYqff3asLMN9Y4fvv2Wy8TOJNM52l6MtlLpvYwbGIKunWrcehwJDE0jilv8cD8x37wu178xheee/U5gv1z2eI999wH/jBODn/Q4SXgpJILup0BMoIgZRHq8g5vJ3dPzt/bLe9f2y6UapioAlQRvO/RRz7/hRc3t1K4CxE2T596ZC+14QsHG8UU1kEXUk2z7be6guBhtpUZIWg/I3IRTEw6YqVmKdZaDav18k42TfrRpHt4YubqTrbZc0yOTV96/TVCbu65b3w0fl+7uX/l7OuZdPbo0fHoEE43R6OezJaSwyMnfu4fv/dLh85/5r+l5icdlQIBmBbQNVpUkOpbAN7I5fe9QUQ6tJqBTKqphzJBGKDDmlrNqT4y/yFniJoy8XiYCUOqBRZL28LSrVLWvGnFD0gUh4ojGTUDOQ2ZA8AvzBbYQm5cv+W0h65d2wQRBO4LB+UWkaifor8g+OWpWApkF9p2JIJyaFhsALhHYK0QwsBwJt4KogLd0yqBk8jwyS7QapSkCK5IrU6MFRgKUJ+hofj48kF8rLlcZn9/1+1zp9NJfkIk9vTM3JWnb4Sj8dFYNLe7K5Aft7tcreEenpyZgUmj6MP1CRCbmp64+657wERaX99A2c7DdBEbG8AjguCkUKR0OhWPhlFrGQRaiPGc4GXxFGQqVEG82WHqQ4G82Ov6nTYQfqKYY4J+xpexQ4XnDcwVg3a3VcafSqh/KpW5duUCiAWgk4wsLGHRzuWz2KArFJRIZtCv3Q73/l7WbidY3IUAjI1LFNXtAvC3lEuTJgNUBV3K7G0QLWftRiG9APGjRVnD0jIl5yragKmHaKNKMbnvOsSWRO6Mdkuh3wbuQheoIpw1XgeF5qLq8PM+0pmoCbue/U9WDA4GtBywkv1eFf6DBgGogObB9WwudZnIdGycVFr0sP6jTBzzjZ1EjIUDW5nLyZgAL8fjaCcOdpRO0W5rLzEU7XaJgcgR/OvyewGpbFRKWGJ7DTBQFH7AamQwDKwuCS2QF3iviT+B78IwoZIieYAwyiQ7OHgKNJT0NUGeVoVcyfrGPZnNZHY2t4DPJIK/SC0T0V3MduCnEGEslBmqo3s9EYKvy3n59QHBZIWjw6HTZIoNvy++dHBubnZ5NDFNPypFJq9Jl4knZuRJKIBH0VosipRTwHgimzM2HjshaTJN9msla61HGRo8qfh6saITJ2PkBloJTK+0H8OCtCnpArsTQY9ITzrGpOqMuRx6zVehYJDaR6UqKNCg6+BOpgRys1ApSAP1sdsIHqqXO7mmtWYNQ9f6jz/xqGXYZYlwx3y7vdVtVq2OBvWICc6RiGb+0MSMx1bLCLMZDBXcNTWGpxs+AwuqFbMgPim/iHRueAzsVfHtyi8SI6GpzIEaSyP1HwcjqX90aL2Z26kvurex2plNLtLIjuec6brhUvrJ7TcDnoqyqFN3HsKdRR/0FJ5FL+4867azmYcN2mI6wYPZGxIazUdxR/0ZRiuma9ifjJPvYsAIInzFdoDFwrz5CqRJowfzXqZmXhGcpECDpif2Rwz+7fVpegcLl+pMrwZjoacbOwD/0GJZLZDS7xwIq1gX+QQZUdAdI6ZMIrptxpAegCcD6+14nDjTCtXSbpYAG4s/4Kg0pEIZgZFS0k5YLNKRMiS4OWXMGsBB0HlFiOFnKVerJGtu726hzVw5f2V0dORn/v7PXb9yE9J89dJVaAn2tgsXLkwfiUejYbYySYv/6T/+F8jfj338h7Hbffmrn8UMdvHSufGxRMA7RKo9rVXAowfKAJTjzemZSaIUM5l0u1OhFiZVNb/t2+8/euTE089/c/7YgTP3HGm0sruplXjAMzUzvjC/vLEO4MEYVtlEyAUqz3atBtkgjRkzVCq5s7Rwf7bcTOZ3437PI9/+nWcv1r/23J+dvOfUs1dWX9vcHZ9apFo0oBT3PPlDFy988+a1b4yR6mn3FzMFW6MCmXXbWpndQjyAUM9iQWGw4MHC+9Xt+6ye8OZ+iigLAOdWN/OBuUijTfZj58bKJhBT05Nxp2v8P/3xi8nNW9cu9MtFy+lTuz/0w6PLh4O9/nYAhGPH+eGDR77n/zhWb734pT9vnj4yvrOWxHRI6XgQu5y2tjeE7EpeEF4YNrGUGa1YsxuYKT4yPawNDHB8S9SPH9WTED42vQN9HeJJPAeUjTj7HuY5exetnWRCCuQQTa1gd35CyN0AC4dM4pWVNRwi4+OjYKq89FI9noiMjg4T+IHRgkjP0cRwJBYnfIP7cx5bxRtvvb6zs3XXXXdhbcaUjccYHbNO6IqIP4YcAgybOAzBjvZ66yA/U06Lqrvra9exCt1zz8mnn/0mztNqGX4Pn5XqFRtKnDx1Zn9/H2siaBneoQQxIZay+IE7n/X4A9evXIkPgeARwkiKH/Pmretba7eUf0K1dVSlGqFWTnzcsGHGBxsqtlVQRPB+EkVFKA+KkcNPtmqjhLO9W7HVmmRdoC55sf8GhodhYfS1XOvQHM9QgqQoj8tWKfaIXZocHdsNR9bWNiikQ9GoeDxINNzM9DhCQalomZ2eCxP5vpfe2U4NBQJXr90sl/JU8YlG/fIXgDTT6WT2d6fnZn0BH5pLgByhfrdMKlTDlkxb5pfdjCYcHKXEzCvqF/h/71A9s+fvvNBV2BoHoUuFQgbOBbFl8nzekFF6pUwjs7MZlC2qDd9CDsAMCw1RMjiljAMBlg5DA1uwU84WNRYxn0AiAqbaZRYSKjQf6T6CMTqkUpp8PlBhWl0JfcwTmeY0FSoIwCKmZ9A6qd+W3N9Bv6S8FjBmLr9PycqAlTDETaE2ycQkGYmEC5UJopsiokrAgYhB4pBYjMjABxPaIr0W70Fb9zDrng2g3FPSgZS8RME2Eo5xSgPRDgmEaDaVb4yWAdeC74CeAe1G/YSqEgcM2T155AxhVqA3A+xaKSImIIfbg74ouWhgL9MpGlmvleXFQZTy9HwAUCO+oUuy65XtVSwVcww6xBtSCBtE8cHCC/mnM6L6MBoeKUoNvTWzJeaB9ExogzYtNFik2PyxWEGLZftRE4TUn8EYsSZxEBVahR7QsIRo9Eo9Xy06GZw7OutbGLL4qENesNgo35W3+KoON/4v1X2S8sKNeQSsgsdr6sTR5C9TM2gUQDvUWaDkmQoqa6jhBspoMu0x+q3MzkZTGxic9bs7608durMSxS3pkvmWJ8Cw6Rg3G1if9UH9vP213v9PB+uHcwN2xQXcXKOmyxib20qG6Y4hbGYkB/fRxWK2irFSi8wf7xl4BDj6DpfV0tFU6L0W2GCJIZWJ74oHc72RQ8wc3vb16isThAUD5koWDCOj3F8uRpxQjwdGDdMxCRjaK+90kncIIqYTjKQc+mo0UhtaE4ybXCXmiXODxnNeySDE6lud+UaDVEgcg1TxcAVC7XrOa8fJT6178JydsEO0PZI54e6sdyRUZDQWPIYYEg/q6C9AwNpdayvr3/Xh7/n3/+/v/N7v/r6TsD9/4MaNWzDdhx9+MF8u/NEffZoAVfguOMCHDx5eXV994fmX/H4P1eISQ9RgJTwH8dSDZYgdFaTsnotgbBuVPG3OWjq5e/16amgkBrkBssXlDO5s5wBbIDItQBGmXAlgPNL6QpHQzOTBRk2Ra2woxO2Nne1MMxuSfb2bb3ReefXVVD92z8G/ZQUXATQBKiHuJhPzRwsW33Y6E18+0nR4w6HYgw++b9fSe/Hq+mxsdL+dS5YacW9kLDyKclao5qcTE/YekUBVuJXHZylUuoV6PTGaWFg81EnHdi6dJxxp9uBSzma/srfX6jrZ5lGSCd3BZJY8M5zT90Qjta3tzUK1+NJL+6OjwlfFD5gt5OyWFyPTD/zEP/igx/nSZ/9099iBYWJqb17dIXAYkAU8UwROMXfouJpZlqk5YFcDfsxMUapPig/VshWU2a9Ui2w0UkEI7IQTobAJOVB4Nugi7A7skTKosIiJDaYm0P5e2ubyjo3MRmAhoez5cxeOHTuWTO1Wa3mfnwpCYNbq0ax/zMK1+n4Hju0RuNT169fefPNNNgT2bVThTI6MSxQpXQyxhb5Bbtn7Y0PT48OTwRDIFJjH3I1m9fKlt6/dvHbo2Il0evfY0YOIAaur69U6oUJFUszvuveBc2cvjEbDqe1ddF+rz03wHT5mOE4gHEruboNxmRga4zmgKMlNabUCSshqxNgOBggBOkSQISoRtIvwSR4Lkd7gMWDDwbKCFueoJtfqxYqtGnI0SILHl+CyhaKQkkoWwC1SxYEUs3iiiEh9otQAfMChvlfIIFQSVXTwwBwSLgiO9ZpiRrH4o5JvbW5CUKb7M1T9iQRiUzOjlVLG66GGOgS6XSnWiSNHviOxflgebHuxXIO2Y4Yi/gpjF02nG4wdN4EBQjiIrRM4///mwBOAgwMzG1m1jDtrIxyOktEEDYBp8XMIilKSXF5ixJSM1KyODI9h2iRQiJLK+GpVAJjqzUTluFRgi8WBRofVniHmDqFAmF8zFKp/TEgedMvuJBELE7/S2MgzBpCMNBcKeLGUbH3iw3FXUaxqo7aCiTkRDhBLQjhwpVDFfgsyFWk80GxahduJbkLrUZ2hLtK9OOBgptl8guTJJCrjHwQH4BIug3OLxuHPdiE+Yf/nYCU38HOETIg3EiY4lSZKH+zjNjkeQYJbIL5ECmD68bh9BFiNj83OzyzQFX5bKZHbCtAraUhkCTkrhIOhoAjpBvGi7w/SO8Y2S6lrAe2AWYqPvVLG2UMcNx5uiCCXMf54hJEs2HIextHDkqi8M2OG1Q5YCq/Y//Ve7gTDhiHeEmpt2AJkRrBS5yUEfr21jOWlW7LGO5nqbttWnlseO/rAMctS3NIr9IubREJiDO9RE9VFMRyLhdgdiiE0sTwTfycGPODD4g09ZeOJCw84CeIpSDGkm/I8copIZFJjFOFMg6X1mmOgoeouyFV3Tg6+UsQgy8tQAfMq9mMC7sQ89Z9hdeLHEjVu/5hRHfx88PrO+Xc+6lId/P4294WDmm8Hz9L7wRXvekWX4KT5Q9zQNGBtFwNGER0wYBlW5NMlsZPLJBHxkVVkLmZIYGZ8e1sP5udGfNJ9DJPWBXpjxBS0fDN1DCfSHNNOiyQv6l8+m46i2fAgVqwOphqjjrmQcEWZerA5UxCGNnAx1Tj0n4OSlYhOqUotWbGYwF+k4TrRbjQZvs91arMS35hbrFuK7EMbJuIehw57U658RqDXw+eHl/HTf/ppKmldfH5l7EACXF9aAElZX9/8yEe/47WXX7p+fZMggI0SdkLv8uLyCqbk+gO/8Rv//Bd+8e+TdYg9E1U4my5Qs9Via+IiG52gsJidTM79FP41bN6NrY0aXkkgG6k7sjgc3stdf+3lZ48fGhtKoB72qGT3+qvXErGp4aERtwuYiB0yEzyd6kw01LN3Cu2G0xvC0NleKtjcQUzHl65fStfLQN1OHbp7s/Z2rVOamJx89PSj+5X8U3/x14cPLAQTTY+l+tbXX+qn9v/24++bnlpce/3Zrb2dkbAmncXOGBOmHBmOjszOWBJBa3FjeikOLlI6v/bS9czU0sjEzMFyvmFruPZWd9YvrN574ky+TBBTzxqIOWylZJIUhmijWahZSczD9sNWXLV66h//6fe7XE8/+5U9EDRnF8L5bDEc9QM2iTSPAQ5yx7RCW1EG8NHBgOEoWgWqo4rnCrkKwa0FLDeThhWQj2wlU5WFryQl4vtT+CJOTaRfoSigdUBZfYLnUWVD6uF6Mxlysldwt6fSu9FYiBAqvHRoOOiaZEzCUHEAAylAozFN37p1C6ArMmZXVm7BQSoAhVBYBv7khG1T1yDPWkVvppgmRBjFuFirDA2P0FS081Ip8+brL/mD0eMnDoEAQWR1qAUfwcIda1hdjz/5/vnxsRsXLuxsrQfmp00+cSgUD5eqQCGNoL6vrt4i0Qv1BGt2PBJeW1vZ3t5mMHDjVsvdUDgAyaHWHAyFESNz1W6vG13fyh0clf11Vik1N9pWym6SPkTtowbWYkur1q4SG4Xb0+mFDCOfIFJSuqFfX9/fLubz8HOSrogkJHEWA9705Ey1lAd2ayQxdO3qdaLGl+aXakOVVCpFMHWvVSWPGmM21VLZ8bVihmp/MCTCeqGiNiW/gfgRoKD89JFToaFJbKpwlgE75I3Co6gXqwMSoK0LXR28gesDEQd4GHXT1d2uBfVUwGduL2sMNRGD7ACLAxbJEETCMeadKaHIlElkc8PkqwiTlbonQWWJAFwEXZN58yPM2V0wWvRpodTCJtH9FDTQKAOjSHyhlpqyjbE/0xhZrcjmE+PsE/9P1HGzWsYyQD0tMFrAZYGzUogB8oR9l81jTH/IDmB4QTghMqi/pk8waL7GEY7ghncXtiQTtYgeK5cgHw44MZGEeElZhfUKUgX9AQKTxehUpXTsHzQBuQC62HXVKNxcBznZNR6nVOXC9OR8PDaa2s2ooKfDH/LY7F49mW3AgoEgKlNFApp1aDhmi0ZgupXkfnV/HZsJ3BdnLdZaOAvPwJ1P0BskFHgwl8sER8rhgzuA6YF0Dg5ItWEeajjLEs4gJgP9QB+SExL6iZlbQcasUwzBbFjqWiAItOsuwPb3jj+0uPzgCRmcS+u19JtU13YnNOoYyOEPeNfZUJAEGsBktPtVIZTJbQH7ghcyulLayB4Vg0bwpjxXG388LbG4HETGo7rCbbGMqn20zJhH0eDvNJ9fG0cIzTQMkUt08LX5lzfix9IEdYgT8Gnwazg3s82F37qXIVJ8hD3p6jsfdRGHxmywCoyO+K7fcTnfcbku40Lza66XYorwZo4B1yQekU9wWZ2DfWKmZdHJsAwXgwGLs94W60ijxr8i3ZdNiaA3qMFgfqVMbYUdsEp5hGktEgwyCq+00nB3dUyW98Gw8YmbUzSDVmr4MMfAI9VkvUegZr9hIoJqMOCQJO4J8mqtBWPCztEBgaFN/Sifu+/ykhZA/Tgtsw7YG0gIyGd9tDRuQOkG6nmgKKNdQrcB9pV04LSTQIyNian6F//3vx2eii4cnUYRCUaDdSJ9ur0vfvHLV65ceujhhw4cyD7111/DbxcMhne2tqHXH/zgB3/r3/wzzuymNvaTe6p8LCqPnc+L+RLZHHUHJ18s7CfaJJnCTmMhhJW6Jt/9se/7xtNfff/73pPP3Tr71ut7GMAXl6inBsRAJldeOngSRJ/V1UzL7XcHnI2QH9MVhsVjh49WO47dnZty8Fn8+8XNkdmJFEay4MjCkXtz5968ubPXsT1Hqv3YzMjo3OTKztuPHD911/uj6YvX+/5oqVbuOXxj4zOtwjo6FY0p1Cxs95GxEfCar12+cHXn+tUdi2fCsnTvqUngJZrN1N6+xxVn45pQ5aH48FJwOjA7N33+0tsXXts7fjLq9kcqGScQzqNjEYIxypV9wpNJafihX/7hfObffu0L5VOHY52Ou1qrIjGgKqDc4yKHikIDmXszwwqVhWqwuIR6pFRgMCQqhJUimkJ1QTYTeYNcmO2o9dCUJQ/+yCLlhnAgIrkwCMT8YWLiVOKkUnnjjTdgnPAzvgyHqWHvVNKHB0CLGChA5JwihxXrZczOinva3MYkAqe/tXIjGokTGww9ViS8L4CRlKeEQv6Zicl8KiOaqwhYVVG0txvBaATKh27Juvf6w9TFGRmOhyNDGE1Z6YeOHYfOt4hewjyK2ZI1ZrHC9aCdEVcoaI1Ql5CtSavWN9biBioLQMl0OosuRVQThBRTLrZnlD8qN5CCPD09i4cLHgWwT7GUd9TSWwwH2Kv2ds0fjlIvCFZsVwZRpG2tMMBMcqMeJHTNSfUQlqHHBqIDlsg00VxrW3AgYrKxWq+tQDq7ExMz46NjuBIpcnP18rVKNYcgKCcsYo8VQDh7iJQe2RtLMBmVc/CFsZKBL1Ms1SMxzAdO6vyQisCkYgRlQEVDCIK/Y/Hg/P9wUMsClRUMRHguUhj0Hw0YDidepSw1qYzchI+G7mCpDxMOACHmPnzN4iDZEz83fBUF1V5DihUFgYsz7nCWoJ95wnPtYcJUWkAkRQZ9fkAhC6gf8oEJJYCgiIaBMk0N4mAQnBPv5t5qp1InOBk4SRQtKLtdllpDLo0dTyZDLMUKbaORUEbIKnRMHAFyIlYPShZrE4omTUMWRYKkBoDZtE1mcGKqHXbhJNazshWTSijWi7+FN7JRZ5NFvys4lZiZmV6aGp/zuMO1cmtvLeMhH9hJgIiNoeDXSGSYbkiJZqKAq0Y0BbbU5rFTgAa8PcLet9fXxNWhpTB1BlTzYrgA1J1yfYRjoT7LVCgQnEYNIBTAPxgqQ7JvczO0JHqh3jAUyFwSu+i29mQ7W86FR9BhQrlmKVXcw4k1tTx77MCR8KKj68y3bdu9fMXiLNvjBJUBzkk17RqIK8CmOH2mogWmSt2ZhKkm6V9wAtRyvGTYDVh+kKh6iXBeZG0tJQKRENrg0gw0GK9IafoT86Sht//jLX3EosUJ9VbMBlmBi7m5rtM5NZ83WjBsQp1hAg1/4sxt2sQJbjm43pw0N4OLiTeZgxN6zxl+Srsl2/C4O3cWczffmpPmWRJaxAd5vc1ozUes6MyG0TOlMMK16DLL3vBg6JxaqEvEXNkODJhJLkLN7GIskVSH3wNWbRay3MCcUat0T5olF696wELWo42FnOeSnmLaLws/KINcqiD5AVtWy/VQwiKKFXpM4d4QQXoC+8GfAtAwwPqVLoEfFMMjAiNot3gDWOu8PnvIF8qk0gQ+YGZqdBvc1EqpPXpF9hkqEg5V9Ig620IWaSThicnpq9dvDCeG//s3P/+JT/wkgPh1cuqIW43GY8NRZn19bePQwQMkjELuUbAwNrFzybf8xV/8xfml8c3t60RAsFqpiJUY9iRTmbjLPTwMIbLnqX6Sr1Lx/cL5DKl2s7NjN27uff3pL80uTtQaabdj/ODiQj51Q7Y0lycWGbv/3vsvXtnwBuOj02M3di8648Nuf59cw7ITU57NZw/U8vmIv0a1lEsbL73w2lNLJw4fOX1f2DK5m9yo1rFjh15949VCJv3QPU8WarmDx+955vpbmSubjywcnT10PP3WG8VyI0rmvRNgXdgZepCWAYoBVb1XN7bHZxda3vRKrvTNZ86CcxvDfT12OBGZvlnbuf+9ZwK22LNfe+7S+ZeOHc9nipnZpeOReDmXTLo9CcwGGLAwQ+DaCkYtjcpGcfcLf+/Xf6Tf/a+vPJO768Tc9UtrGN3w28ppLzUVLqtgd+Z3MMUQBERomC6Tgy4CPeRbmfeQs4kiU5I3EagKHJMRxEJIWYsAd4yp3AxnBDm+hXyZjNdgNO5xBykiiR4JMSmW7ADYp1J78UQYeCIi23BtgExA9SOyZqrA+daKiEQwPLggoaBQn0YTmDxaQp6VuCwLlz9hP0Uiyc29tewGuZFeIqVT+9lyfnZxARz/WDS4sDjtD8bW1nd73QIRUhPj085A7MCBQyuray+98ka5Cg5EGBsOscbouSFHEAxNnI9EH+OThBpn09se7MWYElG0SXn2uBNxoqbrLEAMqpBZ6sYOJ4bmF2ZwR0K79nZ2MZE7clt7VE9k69bDpZFxkrTwaIJj0a9mqvA0/J09h7PWBiEuF6pGqbJkDQeG8IaHwkPFYfqmYGPcMH1rLDHE4O7spnyB8PzyoZ2dXWCMH7379Nlz38TaR8FEbCWASHdb1UQEd0sIzG6VWCdGyx2w2H1sCkwlnsgQqJZDE1PSd0VjoIyYNJ12v9fscbYzB7saCiVqoK0uW25L8KcOF+Zi7AlYWdHk6vUiw8Hsog+yAuD/MEqIb7GM+u+gtdD9na2NnZ09WObC3AICF7oAyX/ITbiB4b1cTwLP5OgU0hZrBZcL1gOKWWC8AkYUOuwBewb4HIagiWfCimMZlDzsKxAtD+KW15/PlGrZArpvEPubMLDY4OjHWNrh1FJhWB94icQJ5MxDtISnYYjGGgpsJLPDhZBT+BWr3JQNgbAZfwI/JXSSkDJ56vpACREexu4GZ5VgbCghaQpOkuwapdbC1OHJ4ZnxsSmPJwBsJFWsQLMKeiMsYiyKKPt4eUURyZnHP1dvBKIB8maJYgY6BSTT9O42WB8olD52kJgSug9TwOBD91BaLZ44A9UCBJ/QLTg/MhO2B5cCyQXuoDniN8IWhROLKMMS+DmiBF0gu0IcgU1qbcYOhHaym8mNpCfhW7p/Yeb4kmUsZvFjpr/ZdRb6tibAoD1HxeaoK7CZSo7OkMoeNfK0h9sJNJgHEe/nNnxAApeKqLVx/VSReAGpb0qWxGWgKC0ul4mM7YRopYkQK4Uo0NrBgXRP8IgOvuMVjiM1mPfSdfkMR9VXOmOu0TccRiYZ9JTz3EvcWdyX95y4fXCPwZ34zGyJo+sRg5MD7su9Btz99v0lXegYhL1otehR/FLGF93O+NKk3fJGhmUYMHcwLNkwZvVPMglynjRd5ke/vfPKdfqt+aPvOo+lUcuR2DVYNQdypxorSwX7SXc3fcKtx8JhCowcglkZ57zwNBAcCepXbToyOXqWdIqFbQlF2lFrg+CqcqWKCwBJg5wvcBObTmDSPH1tNAxSOIMAbSGDWzYjmgkIIJyW2L8aGFgsbwzRRCNQ4Nde8zjBwOuDYxz0+NkRkLnv+shHT5w6jYOGYcHdWygXZhZnRsZG89eKP/8Lv/grv/zLI8NDOIbZ78Njw1/64tc/8l1PXrx4KZnNfOInf2RufmpseISo6d/8zd++cTNPuVGgnOIxd9g/BvjlkYMj5Uprc2PvzOkjh44c/MqX/ioc9QzHKGHbXVpaCvhcQ4kJMmey+QLSQKnWCFkSfdapN2YPWAsIwl0K5rht9TapOUuHl7dLa6+8/fKRU4c8oSAa0mvplxEKDx2czxX3p2cm5mbHiXLc2dtYTW8wW/FI/ObN68k33pi09KZi4Vav6HIFEAsCeMdiEdh2qmyzBd2B4blW31ZrAARiDwYj99zz0Mp+Lp+qjUYDTz7+bfU8YVoh4J+efN93nDlz1+e/9JfDidbFtz5jr7juffBkCWC+XAmpnvCZZqXocQK347M0Lv79f/yx8fGn//yTazOjzi45jBiaehRTBx6aRD5CoFgS2g/C6rHiHZewhZSL2AvPA5jQ7wuxZjBdYIoiJAprI0SYeXW5sd7JweZy9mGNWOPYx+DOVcA5sFgeGnsIpYJaBbt727SH6C5WFCpN3wmZsRVLKPbF/pAQPFBUIDr1puC+UVnoBQFL+PvxJnT6CHXEgsmBgTUFWCT8leCYpjdTbq875sCpWtzZ35udm5ydnAXQwee259PJbHKfsC0CcLlVMVUIjs5Au1gtMBGUDBRfxunmyuqHv+s7X3ztJa8/sHTo4MXzl6anZx5/7CEQodHaqDUwlIvDwkiiAUITMBhbCv2rR50GrCnECWJ0czi9qGNIMo56Hl8eGW/9ZhHc/U4kFg5GwvZmCddugyqVLgfrg2iEDjVpi1Taibh6cFyihV0xMnOCUTQZB7FZTh9VfsMJ6uQsN4hNaraHJuYgxMjGXnd88+YaFZ3Qu8mLwD9I9YmW2wE6iTfiL6lggGPxwOH7zzxscYYszACGxxZYilU5bkkCh0KBMCWbpgQuqAuKK3KXaBEHk0blQ5eVgn1kleFjwMvKe0Kd8Y4AO87lPBezF5DQWGjJkmD42LSExDIdKiU0PQespsXu7pH2DsupNUsVqtMzbTqwb+8okJtozDYBIjBGHobzmzXR8cp4QpqkF7TUBs+s8Rsv1Wd7DTJ2qvnsiRMnzr/x1saNmyPhWK1MaqyD6B8qdaFwy1aA8qfgFGCAYEX0goM2I4TCFoDJapILKKalkCUIHhojTJglzXrlpUV5JYmNQK+7fM0Sy8sW8kfzyaJqA0CGau1YLHx0/hTQzR4q6tp91q6jUcE9TLkiJHWiI2SfRh5lPJE/CYxjz7BigokhSxyktb0sGkQ+h47g7PXicIgWmTyMOYQenw36Da9ifDhuMW9TQ5MQRsgwca7QfX2Bot7usl7heEa0oI9gbSI6k8nooevEdHfwdRA8TrVGR40qgcnitnvYPjUXXTwy41kes/hbltLFYmbDmwArhKwDmSIJ2YfpI01rz9dhqOQWM3hiLGi98toyYnVKPfNHBrfcDIjcMCqFQ4uxQgTMUoIRKY9Vq4vOGfXUCBXMKCPyLd1UnNQwR00C33E5K8rMu/iiHq6fDNinOSMNecBpzXf8gBhg6BPCJKtFb6U0iJFR4MGwYa3oOwxYQUuygqB9KpSNaRJh6lPrDYsfCQrSkvlI8/VUcd8uoflK+0atl+eVecGEI06J/RjZDXEHLirJSYOE2MccsfjxAUsdNrST58l2CFoYhErtFA2Fg1JWBI3YSo10KxA0svpgjbYiMjOX0Fdy1AD0ZvzJIIc6yhqjlDsnmKpUA+i4bPlqfQ8DIlD8ir6x1byUtbFg7YHAgr8idu6k2ilB7V2n10MaEtwaT4kFtAWrtVIp1AsiSvjwEIjBZWNYOjXsXDZ4MUEiDBn4dtBcsNggR91eNZW6/Lc/9r2BSPj5l15mfTNU+WLe6rGV6tUDR46++MrLgbD/3/zO7yrbpFAErQk7J1B8jz/5xKuvPZfO7U7MxP/4jz51z31nHnjgAcALf/BHPr6+sh6PJo4ePl7IV/6/P/hDPDj40f1u/4c/dP/Djzz60qsvsz1xGAOeBSZmfD22OLdQrvaDIcfC2EILhN2+JVnd9zkiYzOzZ8+/SJ5QeBRNwja/eBAWdXNn85nnvsGSDgzFQIgq5ZKIGMeWl/MUBSd5pgWVL1s9ro2rFw5OLfv7tsruTtxljXiaw2FfM5eHBoCaH50+WWq1r6xlPvnF5LH7KkPTTl8oOBmNvfD8BYz5H/uR+w8eOPPWm38xPjcS8gYinlDVmnzx1Tey5b3NF65PjycQZ97/4PuLK7Vq/uWLr78+OesMUhkMlppzkI6oDL1m2mIvWFzZj/2903XrXz/1RUqkFkPuSdL27N5arkg6siUWcZSyjVh0PN+mXinMsc6iYWKJX0O6olIc+cBsWbnTrJaQNwjOJfyS9QOeATZl8rs2N7YRxojiwh6NMhUfSYyODZFfefny1XIpNz46whogOAem1Ou7YkMxhKqFhQVY6WuvvRYFjWp8krCm3WQObbAEHC1ODHyi3iDifbNUH4nHQMNN7eZRPdEmU/vpXCEbiAeh59VGAVmCcr/J7XVY5KHDx1/4xlOkcs3MHQwOJ/CO4sJo20EOBDMygIN5c2v9nlNHCJqr7BTijejzr7528MjxcqWyvbWHmzIaCscDOC2xwFvnFhcJCjp79iyayYljRym6QIUezIwoym+dv4zXdXF+Hqcq2XTFUsfRKjPkrH8rUXrlbhpjdyVEWLV/fG6qBC53uQzWeXRkmJQ3jxuEhEZ2d83pC6N3U+vIT2VLMnArBCGCNym0pba1AbEWILLQ/BQ7OT401piYxgDT6yCbcHTzGXC1laYKBUJyQQ3Z3tkNRtaHRmfYwVgvgEkmzI5EUPQ52JKKzIosIfdSxx01ckCvjPAOpRGHZpSgfGiD6PpUMBK2ESuYXer3B0PhmNPqJA3X4fLF46O9fpqZhuhBsIiCc/ctpW5lP5VhtRChHg6ElKqlvPUmdI37lQtVYikxbqAoq8YAgIt0jekrE42uoBIO8keJPBdqbZ0gYhzKXVY8q2l6YnLj2k3i4aFnuBbQGSHBEHiRV6Mv0GbUc17pJuwB6qzib9LXCW4MIAOiBdM/qLBWJ4cCFazo1khJGBgJMCTkKuiJxEKx9ZvrCK/IE9OTcy4recm+oDeECzDoixPthY+YIUQEgX9DOBgpRAC6iekHLil0CoJPycG19NIXL9br+Tq1KKgtCj4Wem6vhzVZnAeOS2+ZI/0vJZA/DvEV/td7WIrYIBQfTG6FwWEEBuzCWEMZLmZRRj68C6SzWZsVK7i/5EFTl6Ny5Mnl+KQ3OEUxF3zAl6miSERqeAr/YBH11whcGijxLJg8vAKcFkRrmbj5kr6YyCKWXL7CIPJcDtgMgs6AEyJOmRYaNVeMjrbqlUNXGI7GG72/c7zz/ltvaIJkQfqnKwfH4A6D/t/56bf+HbDzOzxdN4ctqr1m6GRKuPPAwRumR758u/J2xC/bCneHsdFpZsAYnPkl7+/Y8qWtDpRXTiJiCHbEXCmEDcOAmTZNFSOkKyUk4QVC6lJQvTx3KizI6jAsHN4sQEoRTHk+gJKvdxCjACwr1zuFstRZuCEM1WJp+FE8ibWwq1QsQ4JRmbtRgbxCpepKLVWpZusW2LpWB6kEVvgoZBnDpTJysFdQ/AgTKtGxSkgwMwa/RZ1l89JK4q+NxAY7BowdEx+xtYp7dqEzEP2K9RmJgN4qnkIBdKNDozeuXQ9Ew7geX3nlFSjP7Ox0IBoc904iHAM1PDU6Q+jQ+Pj4qVN3/Z0f/wk8fF956q8r+L7KdQCBMXehN+8mk1/9+tePHDny8H2PPPrIezEczo4svHnu7Zs3MomEt1Ci9vAYVusXXngexQtB6sK589Xa2ic+8UOAXG7vpj7yHQ+nk6Xf/v1/9cbrF6LxkSeefB/aFQnAo8OzKxsXx2fg1+HVzY3PfOYzwRCJrU3qogLwBHIIYP0z89PJ5E3kD0A2IpEgPKtUq49Ojd1z90O9Yu2ZG9dGR0J7uYylbd+4tP7gXcuktRfazudeu/DS2z3CHa2pesRRyF++uTgZp7wSFbwgR1ubZARnFg4dY0fd3LnktvuOn15K7a0//uCpe+5a/upnP3nt7beHHO3h4CgY6RbKgpEjDzp6H8WXwMyKLUzeR1U7y2r72z/+eGR067P/7aaHEgsUi2jWF5aDW+vlvb3OiSOLF9++FQ6OAWQIPXbIQwD3JX1J6wdXkYO2qkoSyqcfqAmmEpYDxjz+gSOHT1AtcGt77/r1m6tr2zSbGG88J6hLe3tb+UKWMFdKG7LRmO75hXFYGiRxdm4hSarrU39NZUB4sArnULuv06+CtY9Lrt8O98mhRUO3gC+NvuEPIBdiS6O+MHbupkIgqAnR7rup/GezgBzM8iNolPgpgBO1AW32mZlJTP3lZv/qhQsEP89MjJ/6sR/+wmc/TQg1enuxWlpbWzt8/MSxYydee+2VeCQKDlZvLDEyFE0X6xcunmPJnTh1nPsgKFATAu8vxSGw0YIySU5QJpMnncwPZgNCdqeB5kfkDbp9t1qHa3X8ZW+jGsBWTNguewe3do76p9lMJB4L4x4JRPP5zM7VKpwD8x17EP0SfSg+Oup1EpqL/E3hrQGkYgtX+tzk6MTI0O5OOZPPNRxUZu5Vi0h5YqMEcgCAwtjWS1hoNqOsyuEhAI0JcmKcu4j8bTf2bfRC8tocTj/7WBRY2gh0DHsujAug5g77n/d2K+umRpBTpQrfrPIARspJ3iqFyXr2fLHs81qDGFj9ISQDTPOwQdiPMLGxYJQr5A6h5AmiqkSqn5Ug8iDAWpFwbUT4ag4y0uFaFBIiqze/Lz4PgTLUnX0IPYZYQILhaDDQSrUE92XxTkyMQzRQtYF2InYJsoqeoMv4sTQtkWm6b2i3UY8UliSqDmFBhhd1lKka8qoawzwICTrg8rcb/WKr5iNH2RulQFu5WEttrZ05fa/T6hkZGh9JjDkxQaNo9bDIacUyaARJoShI98LOLefygB/DkMGWxvgs8K9cNtdNNsFpIxwL6ifHLoSRnkmFUo1F6L+aJ4MyqqQNXZcdpsxaDJAi8cY6acirzjtB2VHOGdQUA7eib5wumHmthUMXCLJGqZ1vO5uR8dChY8vxhZgl0LJQQc5LARZczmWLvQZugWzGGp/BYzVeMErTBBrHsBkmJgso/nnloskzXyEFhLEdHBrhQfqR+n7nGCygwSd5aHVXDnNj856v9NmsM/Ne15ozRm7ivfgE/+tPX3GpaYv59DdeBIslFqqvBw1GF9VPWMt6pJ5h7qxzNJZrzWEWCRNFJKfmAc8FBxyUawdjIAetGDk71ARMyaeBd0OcWWMm7gtvRoDTD8VgxYC5yAQMosniizM2QHFriQRi29yKECSEKMQuG0otKDgUqGMrIigXq61izdKAGrO8FMXa93tqoXYniC5JQAFWQryqQvLsUtB3N9csNHDrClcRizTzgneClaLWqhdS5nHoKkJO5dhNpj7E28nqgTuL2iJs84rIxqBxUjIdchbpLFpUSo54Z/QYMcRIPHOb29uwcJCbU+l0KE7O6GhsJH5l5SouufvvBX2intzd+a6PfhRp5plnniHeZ35p/u1z29QJnpkb/uznvvHIe2IYAg4fPcxIZbL7uWz68oXLm2tUZN1NjBIe4ccyB2GZmBhZWbsxOTXJU/A4+gOBSqUK5JZyK5y+3/ndf/qFT9+YXJQAjWOyUMxgSjh2YjEQCJP1h6yxn9z5vu/76N0n7yrVC3v7uzs7G8VidmxyaHN7HYuHLxR66+LFjZ193pAnEw2P173+z37uq+gYLOzLF7aqe5YjM5at1288eM9RdtVqr79ltSyf9KzmGpM229DywjOvXBuKqYhXzd7CcXfgrkOHTi6DdHTu3NuLM0uVbCWWAMcKQ/pbH/3IQ+5mxmtpRz1BXE6dFsCX1KuBNFWQ4txDEUIWibN2hKOZ/XxidOH+JwjqKnzlT/cnYoF+x7+1XaZmcN9ju3r91vh0gvVE3Iq0EKyPdjeUyyoEjB6qg0xi0KCBYI57BBWp3QFaEXK2uLyEIwDODKjk7u42xlu3z7UwOwseJLkyTD2IlkSTQEghUMQEMOlYIv/sz/4Mad4L7pJKctiXlxfRPyE/QFOsrOFcqIL/zcYhZAUHBzkf7nEvMadwChoEp4NtoP1BVAmoQmSwO3OB/VSRtER0LlCG/UQ8NaPDkxCGycmJtfUdwD2W5ifPnDh07pVnjx6ax0lKtBCbY2PlFgVtb928OZJIUO0WWWd0fIzNCzYDpmbUdNKN3n77bQyL7EBlrnd74EQRYs1Bp+gRCxvkfWynskKyxBGUIP7g4SAiU9GdgvTeYAgshgYcP5NpZlLl5H50eIzChAAHQRRQqlvEAbUJiemU9uPBUCSeGOUFfOeAB36IDd6e2lqJR4PxwGJxJNxrV3K53WYxi39JjAEOLBKCBT+1jx/YapvszhDV2wFrWAyYLAQf8J6UbMdUrs1LjWpqbbM1MS3JAwSbkXeBjc0Uy1zlJlsoTBYpAZXxoTi71Eu9G7cHDAzk+lq/AW5mgDJmODBdzjK/otyvCinYGpTEIfJI/g1rmKAeljZxfpVSjnrPCEvYrPHkMz7UcqzhvC5T4gPWMKAC2Mkxtg/oO3ZPRhZW2QLt1eokjml4aGg1fR2hUczHUHrZAtFeDANWsGifpDlDi0WgRWUgOLwySZgLcIbxFIlv3S62FxRrK2EaDjCvSG3oZrJlugKq89DM5EP3vQfLM0ZlNgLuN6gMRg3Mf5gltEodKD1k3TSYXux4qo7NnoNOk6asKhGCS+WJzVo5EvBRfBVtAxIuCyYDJ81K4hKNQ3ESnTcNhRTjDwRC36YoHJl5TfPpgPQ98JRZ56g8eE2g6nUoJ/sOM4srW+nkLD7L2PzI8okTlrkE5Q0t7YyFuingbOSLDk/HEzKBlB0ws2soV2ZMFP6t52uE6BeWAKg7lmwl9eKD54ABEyaHmCCNSo0VFAt7eHDQQN1ocOjL2wdXDi4evBm857t3Tg6ue9f5AcfULd59zTsX3L7vnX94Om8HbXjXT3STwSX8cNA2c0fJBmxXDaJGWufErPgfPVZvBt8a5gofE0NjJbGfdAkzI5ep4bUMksKhJItwL8ieCrvyLeGf0i+NqRmyzh7kNPZPWKyUbJOVhP0KGVCSF4s4EMcGmynhnCEhF3crN7NV8SXDqRl+Csi0e5G+p+MTTBxuuhRRodVemSlF4fVyPc4IDJFtIARY2OzT21ZuCA2SnHrEkhQdZJwGm4gWQz0R2Vj5fMskDs7Dd9kF2JcYIuRa9hm/GFwMrj/yMGa5rd2cJ9hNJOKQBSjjcI18HMe/++3fQYY/fdcxNhLWPW7zx//tkzdvrkxOTcWH42ffurCy6ghFLX/8R1/90HeefN/7Hv/Zn/kH5bsLR48cf/6lb+JgmZ2ap6Q7gaLo/cQUxIdi/OrV11+B5f/4x3/w1Km5cNjdqBLpUb+5fhPJ/vAZZzAQw/p+4+a1ZHo/NhR9+D33zjiW6BBA6SdPHCNC88qti4O4obHJke09bLAFio5lCvmVtZX9dArb4tDY6MrG5mtvnj10dPPAPQ/OjUcrqY33fseHzr74bMJPhYXg1778tVQawDrL5Bn/0Uce33/hhQtb2wc8U2OHfN/5oSez5fIe2Z+bawdPn6BKyuc+8xe0PxIO3HPi9HNJMpOvVdKbO7fWPvG9H9h+46Viq+L0EGGN5d5Zw1HsagfHo3VyHxqdUGjUYRnp9PcLpY34eORvffzeXvXZZ79U8TkBxnI3SmAFBl2eot3VxdtK2U62PC56iV1E8DVsBACgVQp1yJhiKKkK52K9EROMYZJYJAO9o2iBpaV5/Hpk9cwszhZKuEUbR48eAUvxzbfPba5tssEpb5DKpBGqQqFJwCD3krtYlScmhVU5SoXF8JDbG0B+iJ0deu3N1zLpQjjiSyRGSBsPR4cgmPl8FRKI2IaTCtbbUOHeBjncBIcDO7qxtROOkJuIIumNCCQeWtSMxmIf+MD7fuFnf/H6pYtTQ97V65c+8MQjsYj/1vWL+zub6PGsUhQYYoPAgH7vex4jc5XE3zKVkAx0MRUSAdwgGgmjC5R8f3ePWnUBCiQEAmwt6C1f0R0HYUdQZUdVygaKjLLrAJkB9c/nblDGtpATIBTV7V0eG2afdnMvk0IlxfmMBX+Q7dUnbqneyOeTCGuN9E41lvChbuM3ZZNDQRBaK+1Mag8IGAoTtBtlahxjGSbFE1sZCgAOXuh/eX9js12v5neGx5lv6YeYFynxS6S+1eLGRA5btWI/JXDX4cbpSHQbda0J57a7vLAHidQqrEGVoKgc704b7Iq5ISGYb6B2OE15Q3k8QggwIruD/mjdn8/myLVqlgu1ctk/EsMe3ACcoVZBIkEgpPQY8EiArDFnogJAONEZDOdKbVFMnaFtsH/S9RW2ZShdm2mkLgEh5gQ94QA9sLiwvbJSxaXtD4gwivbxNyC52B9x8ToHLj2RaYZDBBvvCQTErccyCoRLUQ4b7BXsOD17wBO2d7CoI1IRB+UaHho7sHhoanymnKsxe2KXBnGMsCwiHRhfn8dLX4AEgf+SQke5QDLuSdxDKO0CIFlQMjS2SEgjuJ148ymrhNJGZwccQhwAakdJFHaRYTkYx+BlDLcajHuAAomK9+GQdVfcmpmgWpEpLIwxlVs1ugxrWXewlxue3YXjo3MnD1sSIUuTtPLzSAbuoK3RLQFQ73HhxIGpyqSPtk46N3qbgcVAIvoW0+JBOJGF2EFJZbLGSUeQFgU5V7M0PYbnGVGBkRPvNnyQ9n1LD4alcRlzoqsHXTPdeOe9OX2byw5OmtfBab1y+bc+/G/eQe71aIkn/KOfyG8Bkj52Ok247sGA8g+TzxdaxzpgQrojvEcYLPjgsVMguBmmzIJhlPmTRRqZBGInA7WYqiZG0ENMDTYK5SDxLAU561XhzQQeC2WNjziPmb2+g8oaVfz4QPLAIiTn0gg9n5FpCFXNmuv0CwSSsy5ll1E8IJYnEBERFGHx5KvVCb9H6ZHG3M6RZ8kOwQqK5ETWCnGw7E9RZ3ondsug0xjaxUaCAnAwX9wKQY0Ny0cAi7gA2AeGRHYTsu8RjO0IEET99rmGMeEMb2gms49cp22D8RYa7AcAjjGStp1LZREtyo3i6ZMHkQ2hkuNjI//6t//V2mr20ffce3Bp+cLlSwR0nDx5jDzyex++60tf/uyF85d+4hM/DkEEQ/Gtt1/b2EieOXV0dHz0+s0bqACnz5yG9P/hf/lDaAvValdXV7/61F+/+KLlgx94fGnpIGSXA50MIOJUKulyB6vVEqEING1/L6OEg05/YWE5tX9ze3MN5MKAz7G9t5NO74Vw63aa+Uw5EInORiIHTp7cS+UuXLmCbjAyMXX11uYjj70/W80W6n1X0zJ91725ve3+8NBd3/6RL37xKwxSqlp3rm/OnThZu3HNk0h853s/XC1ma6AE+VxUh915822FQLhc2XLy2ReeP3P8JHa9T/9/nzuxOEQN+q3tS05P0mav2h1eq92PFbeBExja0cGd3Q8OzfS68Y2tajQ6aXMBArDfbe/+wN992Gl79q8+3ZykKLwjsb+bnpocJnMpRMk0kqdZApLyCDchNAEJa2AmVBYQ8wLPI41bAmEf7EWKCTh3dzfX1rdxsWGFcM5MQJRw9JVKfWoHTUyMzc8eZptQW4jlOjExQbAwals4Erz7zOnLVy9DvEAzQOJhjRDBTvT4wsISVm4EwCtXL0H9AAFkhYJpn2lTzaESCUXxwPFwgzGBikGknyJNpQpa7ZlMDrgmbOD9fJ4SES5/eHpuGa/qzuZKKbNfzqZupbfvPXMsEvJNjY2Wy8WLF27kS+WFxQMkJiXLGTx+JHhtbG2C/88aJhjw/Ooad2bdkvvEQmXR8grfxWGMhRaWDP/GfGL9Dw/PE7FtbIxa7HjMscl6yFAhIc9YRyFfjIJhZgJCIhvHBG9gcUIAZXNgIgayBteMhz0mxoFkSnYvxlasppAN1rLTiYkcUkyZvGIBrLkKLmjC6GmQuYlD5oBOjwGh+ubo5ARukAHuI4zHg2eb+gdOF/59F8lTgbDD40UJxpEIOeihCtsjsEAFEHdqODHRS0npKhVySEzsE4xypVKFkHh8osj1DJyf2hsOS3B4mHqEWxvrwFMT9gZQ1+HlBQLXQVBp1+vBgI+yz4yJ8knJNiQsDB8RegKGC9QKhcY4Ce9jkyv0hCK6ftxhPrpD3E8qmScIvUfVqnAUrkZ108/95Wf3CDUXRhUWRVQLWWRIPGVnQoetFlV/0gKFsIltQIzFaoC3ZJ74CW8Ud00RHqhXEw8YeDukiMfGRyfGxibAi3GRIInhvVwHAhc6B0UO+kMsKTRC1A3GHPmRtnMTai+i3PaQDriaKADsDEpQq4n7YqTmsQT1tAiyEp8SH4U20jRkDLskWqNDimTSVL1CVzvWAB4A+C3jxB+6FX+C9cXBgoM3V2qVupQzc/e9QU9iJBGetoWOE0y3g0OGkG1XgGrrxExWwR9QuWFUZTsRBOhLNbYo4gK6O5BeCtge8E/xGJ4DCe80K1UsSrSfbAN2Nu1RbJ682BhQad5tRmt+qPe372AY8KD972bAt3vEP+b4Vh8HPf0WnxUX+V8e/O5/eZ5x0SHmq4OrBvIBOxO+yyDDIAcP5V9dAX0S4VJHkb3oElb8RoNwXNYPjHag6YoBY0/WoJPCa5Rg3XcwZ7gXEDqlFutbTYmy8ihkgbW35xAYKsGAyDiIsW6oeoUcrZYV4ou90cQ893CgMPmYN1jrmUaPZB/xyx5eSWIDwWS2sfiR5DjAcsUa4zX2agxEKC75TAbXj+5PSilER/F6pIUJgRBBSqFykt7UFRLHeKQiLvCEYdIh50JcVrHTLHsECS4VA8YeNQCINa591DTOQ854RSeG5zFgUFPiXLBpI4LzKuNtPEZyAiYjMnk8Xtv4+BgmnsRQjKEul+oUdrv77ntRxbb28TuSYWx1+awTUyO76bWhhK/VrmIqoyb88SOnVlbWz7915cCBJQzIRXxLxTKbnZGEjLITAOiw4wv3gZfeIcXz8cceeenFV9bW9mgGOAKIiFu7WXAKHnrswUceexAcxkOHpqZ9oY385XK19OlPf/LwsUMoEWPT47vpJLbGUHwYQC8/iUG19tWbqyDwX72WbDbjx4/fNzke7bRyBxbGo5Av4G+AkinUQB08dPDo17/xzPnzF8/cc/fVq5enp8aIQqyUC4zA+tbW+PTMfQ8+xABurG7srm9OksfDPscEUmkOB0Kzo6PW8o1Q4/WYx+IPxrtAF9i7oTGvM9IhOcfpIXXYe+lsBsCv7/veD46OE151sVpOjYQPWgrxz/3plac+v9chPIZoYgeJkCBAVQmhA+SQ4Lp+h1IHDkD3SC6j9CsJR+gqjLyw84kDx+xKumYkGomSrVvN5POjI+NzC4sQrp39fVJAKF8MsQkSYDU0Ct01aqt9aGiY1UA9XTYMU18EvY8yR/UKUcOIXj2QLl2U/otXa83t7d10Lv3pT//p0HDU54fLClKCdUWODHuNVY0aTpFyRNRo2K/oBasV4MRKtekHIsIToFKa1QUmRfTAkZNHDx35/X/3u4V05tDStN9tiwY8x44fygLhRJ3py9effuFFahdRiOjwoUP333/v0tzs+YsX4I84ZuFulEuitXScpCF0262NTUYAqOb19XV0X9KIWMMcBKRJKhXYMjSXcG1WvgRtQQ0HUaF9hFv0W7VmtVOFuqEfo5jBO2C8wJbAgIFGbPYLCLNwIEZa+8WLMIUAjOLqxQgLvLPCMdKpIJBumHOr1YDX266AZZGD2NCCQSgM1LxtB3sGP0QZU3aA0GtsqNTEqvvgbzA54Kz6nSqkGVRJWRkxOMstiuWrSeYVbW8DeQwERJfif/i8855+gh9Dmqv5Mgy4Q9HDnoWiART4YvzblTxKYSG1zUstnyHs/o0X12DZPBOCYlWEPI5K1o22MwCSjB1bBUGEL1EHXRD6NmIjw4GhF7+aeoHiQAUCYrAoXMEEMDIjcXwDjrGRUSFhmQrKUCAUH8YNNoGaj1AIcJCYhtiZAnIglKLV2OV406VsAKZihxWoXBBocavaPJVCa2764IkTp0aGRiBA5XK1Xesi6wQ9YewUgLwgBBCuyi2QAXHHgKELMaDcB9gxMFgMLoVcFrGjVixyR2FD4x6We0bsuF1rhH0R+Qv7Kp0oTgDDkLJFaTNsI4aP8d1A/WXdoNzQVNi7uC+GTIYNaz6VHFp2L8byfMcLyo8nNhaJj8ZDE0OWcfp4ue+pgIOGKqVAGzICXW2s3lQ/BXeEGhM0UnGT4j7ynCihWawFaRBbCjZ2/qDe7VaVMsOyuIrGY6eEWJsBNONryDzDeIcNa0QNI2QeeT+wPxhBghMabzrGO74037/zXh021xvWOPiBLr7N3fX2W8fgPt/6PHjHGuG+AybLD9kW7DW+ou28midqBfHGaIOSeug5l/AbhpXveOVSM8QaZTrC9eo6bxQooDWDesHs8FskMFl00Z6ZOEr5Mis4hvsOoveYHnzCsHbGHtDrOuVEu5YmS9EdxmaaSRd5BK4a8ocIwkYTQBslSK+ETcXthMqQJ26nSlomQ2YR1VeQVlmbLnyTZHBSppOQLjgntE1mESoYKZwWtxY6q+LO6R1DIPWIEdVbplRcXjxfI6IdwRQgfCk6UFERbpBIaQCbALHXOH15Q5eJ7uP8IJ5RMRKIhZACuzPg8WLUo4p7sVI+c/o4g/jSy2/QhHA4gDFma30zFPZT2hmGTcrMUDRy9q23SRa658y95y68VSoWsKqDxT89Nzw3Mzu/MM1WKhaoZt14+IGH7zl9/6uvvOny+Es71K1Ba+9PgGczMc3a//pTXwHfFtgG8Pep0/DmG3L4RRMV7Sqnd28/PTkZo7OUfz924niKTKRS5WJ2c3wknMlmd/crU/M1rNmZbBE5pN7oRJ1I1iNb4C4mc2QXelyRuUlfpxo6ubAwN4vWlXxg4sSnnv8vm6u3jh87gsHfHwjGQ3HCh9dvbFOLDNHkxBPv+fxXvnTp2o4vAKDv0onTj5QrhF6yIX0nTj3y3NefnoolphPDBw9P5Xd2376yfuut1z9wyrfTqkXjYZsnVO41xu0j/r5zZXM9lWql9kpvvpQDjWw0lrrrdGRoPDgSSzTKq+1G6qN/9wMux6t//IdXw1EKDvooxSK5CpKhqcXKBywgdU4xByiPFQqGiMgkAgbMWsDOSpMAOCCeJpfbHxsdn5udYoEL1shpIW2s3Yok91LdrY3GfJWMbaY4R0HfSgXeBGMiq4fIJvCTCd9Zu3Xz7gfuRqePRkYBT0vuFwj3mZtdWF46iDFjc2sFvtuCAzusxHyhN7GSyMPEqsbKYRMAZinpDtpKpTQL6lmZCLNIYjQ6NLqfKWK+vn79Chli6HK4nz1jcaAvMHuQ0jY5Eb7/Pe/tOLyvnz2Pya7Zs12+eotdVql3SFsJYpTt4hCJwPcHJJooomqtgWOYDUtpB9Ys61xh25WqowrOLnAEioA1Tk3ifVB2epZIOAg4FNVn4QBEMeDx5qYNYI+AaJffSdIpPBiOCylFQG6WqpSOpKiRXIu9GlGCLFa2rjsQZuMPBwJTE1MFPMCVOgZl3N2AbA1YDQZSKDi7TxBJHWt2q25J1OztKBHgRCWQ1NNvBnFjWF1ufNYCiINrsV+ldwaJ7fd4G4RWoapawNEu5LAhk5Cuig61DPID5BVjO1GdOGoR05GhnT1Xam+nmtmB4RSp6Utij7WBGVg3dFtRFkkWK6dl0YNYEuJJzQbcw1ANVjzmLWL7YG/tDmkF5INjFRbCtLVDihGNJ/GshYk8mdoJBYLlbD7mD0LxopFINQ7CI+NH8ikECQuMViIOEVRi0uDELmRVgHSR7wFtBr6KPJp+yBcJeP3SQ3GIWz3RUGIkMnrq8XupKA8BK6QqrCMuZ/0Q5l3Il5wWL3oK02NShiCysG4bLcHywGWNYh4MPFgvtB0kTncEiEq2BebpKiY/6hCphiBA70Q/8kTZMgg6ldIi1sB0G0aldoreMx66hDdKPkIHQxsZcF8smrgLXLViLdtx1+MT4eUTU575UdwIlka+20y1fWmLs6aoe7gJMoKwtZlnUgqreBgALofogtoGsiVPJbiOJ4k1aScxOajstys5MjtQYaCTFBpL/8SQWEdgzdJGKU8c4kJ3DpF+c4jXGXbLJ/NeZ3kzON45ycd33ps3vAyO/yX3vfPl//TvQPaXCCEGC32ifwpy0oViTIYzGSe6+BHzaqzTCKaQCQKOxJvZRyxTiUHiuNINGXFkIrFePOBSylkkSLNcwA00DKxJzBN4ak39J3LNDAOWtERoDI4N0kTgs9U+he483lDcEx45NBdm07LfKWHbLJeAY5Vpm3SmRhm+iHsiHPADztKw9tLZLMolo60JIJFX4VVaAlA3mlwGOB1GLypr5gfpFOMRIhnJhAw7Q4+kxAHPVYgB77BpI1DQOx0QLEYBQZdXM4PmWkWGyxDNsJCRzyt2aQ2RQsWlLmMrImYU0COuhq59/Ed/LJlOvfTCG1Dm3f39sSmFRhMu65Odx17IlnC7kcWxtrpardeoA+hqOu++9/TyofnrNy7Yek5CP6qFFnhGYX+CJ6OrI4/furXOyI+NT+KYPHLkGEoMqvCBhcX/93d+l5IzS8uz8GyiKEg+xBgO5AKtAtP/xOnTb50/d/369QuXLpWqOYJTA+4W9Xzm5hZ/9Vd/6a0Lb4Lxg+sX2xTgI2+9fdnlDoDp7vNSzhZkpR0/rjJ36OrrL+6v+o8cXf7N//brr772wtT42Ic++H4gZS6eJ0+vwC5++O6DN29cI9l9OOIvpLMsg6nJmQ998MPLB49+6s8/A5WuFMvvffzbYqGRp7/81cjQxLG7HvyH//mn3v/YY6cf+cD0VHzl0tVcc8hmD97YXHt7LWd1d6+vbMajMyPRRQre5lLrX/n8W8WM49jxzj33D1OLNTjs7pUufdt3n/L64//1D16sly1jo7FmFcJIdCZLkD9wrygGylygADBf1IodiFgwBBu8BK7BZ3ItQd5YXPQSYLOyukr2ECUl4CrRkL/fjlWJeC2XU8kkuA5EaaGzMrNXL11CKSS+qV4tpgv5sdFh1FFCooAQgYahXwJmKdpYLX7/933/n/7Zn2zvbuKWxfg3Nh4icRxfGplIwDgQW8NCIyyLeWTnoLRMzy+sb+/QMoLnjxw/89a5S4cPLWNfA3x7aWaG4BissUcOHgF1ay+Z7lgdx+598Jf+0f/9pa998wJFPM6dA37K7Q9TJeHmzStTU2jYqufBzXl0LpPFVsTKxCSJ6GBMX/29/X0c3iSrOCrgBCETGusdW4ZNQ7QEYigGfXyZWDU5iWkJQ584BCFXNfBz2RtK7lJcCMZGTKlIoWTjgUncBf2M2Dl5iThYujyVjxSIrmWwQjv9NideV8CfyYU1dE9eJ3QdCDwTKBmZ7JpstlkpszfB5JH0TUQ5CE1kBCI1E2hFiY1QxOL1gcrW7uxEwon1UgU7KonwgF8on6GYaeQL+9lt9jRmOprK5gWrC1Qy/s3m6z7yKzEPdds+Zwt7DdigRAugbZFlz89RBdFmRStIJiezAmcI/Mhqhf0LxAF3lMfVb9uNIQDa18MKigscyQOS0agx6g1yYnc2Ngm33t3chhySpAQCC8bqTCbbFAwIiizKOaRM+N1VAil9QaxywJMGfACNBaBewFMC1EL+JMU60ROH4+MHFg/Pz8yGfNFquobpjOl0OOhSgDfSauqdsegwdII4OkQRTJbYnXCyEDdGWkkuk8pnskyFSkATg0bUUq1CTiLqOuqtSCUql9QVBW6hcqiKI55YWi5Bkf0rcGcRXB4rH554spiyMq86bTempha6Vl95vc2WrVrp5WrN/MKxifB4YmgqZI3gBb0BVgdmEbuPG1YoFo72AAGFdYr+IiqQNMzS45FNoGCxcUrvl34EWa+pOKpUK4AbJKqh9yG7wUQUXsyPJS0q9Jr4cpya5JfyPaOiX/Ni3mktquk6BdX+G+z29jVmLeriO4e5+M4HrfzBe/7hT+xzcME7l0GjWRryFBi2gLBF2ABsg1wyfkvnCGsT7zU+YBgnzEZ3ZCx1SAmG29I2DTKSE/DdLtR8HJxiTmxapLU7Wq8uhvvCZdF0MdN4QYJgIgCXtzlhaWxeLH2lEtZAF6wT9oc1BSYuwYkUTebT7iw1yF1xthzeNy7vPP4dJ9/70e89egJNInHhwvnXX3757Msv3bp6hW3CpG/s5Camh3ygFHhdrHPQ1KEFuE4jQYKzsljvEGr9AX8kGNxYXWNlkRKojU2XzW6hO6QaEXBNg8X+JYhITsAsjSNHcHFug2eOjbAphxnQDcinXMKjNTzQhcFEIpswhdjJWLFYtnFAibxo6jW8mh7MAEqMJjbnuWe+uZtKsoJYM1R5o8YWUYfDsRGKi5Tyufc+8eSVqzdurW568GY5vadPnIQUX71+cW11JRhBabMQMRv4/zn7DzDLrrPOFz51co51TuVc1dW5W1KrlXN0tnHAGJjBHphhYIYhmbkzDMNlgA+YIVxyNphkbMBBtmRZVg4tdbfUrc6pck4n51BV9/dfu7olw73fM8/dKu3eZ4e1117rXW8OIWrJFG6/7e6LFy9965nnoQ2FUiWV6lheWIeCPPbw+3CfOXf+zIH9h370R37o85//0zdPzvT0ut73nsfi8Sim9Eq5Nje/tJGrti8v/umf/+mv/M9fOnX6ZGd3ElLU393dnxq7NHsJoTOZ7EFZ2ts3cuna1fXsjCcUOXv+rN0VTGfKP/offmJ7K3T6+Juh1sbYYN/K6tSXL76CjHL4wFj/wMCJk8fAsOTJW2/UQh7Px7/r/V/4/EZnMvKNf/zb9Gqtt8s5MtS7vrxA4AkV85iJQDiMDuONN986dOvRYCz+1aef6R4dw159+y0POcve2x//aCLV89qbx6dOFR589OGnvvX1QKTnez71g3/5J380M7OSjGz3dds2Vlp3/eCuNvsK9bJBW/WtKUp7P/g9d7Ji/+T3TqKSdduCzG/IT+KwbK3UIvyKxMM4nCB0WPMI/LM6cJ1hGsuVAvkKr11baY9DFLfn56ahMso0WEGEBQsh/gUJBUZMYYnnMuupri7QAkCP3uX85LVatZxIxJAGyXpbbRShnaRemJpeikd7gqE4LreEdYWjoYceeuj8xbMkMIFfzOZKD9x/9OSJ4xh0SLeE8o8cD6BuwJJ3QQuBQ+KC9u0/ANrcMzaKbHr+7KmzZy5iewbPrizNr6T9uePUGI6kwmHE2VgyhbGuo6v3x9/z/r/+q79dmJ9fWM00SQU+dRmKhvKZjyXWaH19HZ4PSozqBT94tNNLlDW0twVCQfYQTSfJnwFy9LngNAPKchIBP2xkS1ioSQ5Cci0cjRkfkB00UbH3SEPy8WANsQclGyEJzhXJDc0lP6UONBhLsdnyoSVHPgF/VNXRihIXj1QMKrdYd7H0SH/CpqA2pCIhd+xWdFCdglDAAigKGc4Fy30IBX0QNyIWOe3US+volKnd42nWbJDQaqmWybZKBXgZrXQEArCVIkkdVUU5CBEa+xH4CEsK7tYkcUNhhjKLnhJLSjF2UIU+VdIEiALQMCtfLmqSUAzqJZu0rcRIYbdGo0hPRT84cpF/II0kQt4XkslJ+w2l37arPoSwMRmckJSlccOxmgg3UAkDQk4S2h0ZGN5uUXKPDNLNaCDeKLVIwZ3s6OhMdvX3DLXHU8jB63PZ9lBSpkO+zcRjgOVB+/CQxWwWEYEikiThsgHlfDtxZOXaxuoiGZwRtxE4YC8AaSgULAD+MqIGUH7NphSYwmic3VSAIzOpzBeiWeThgTvZQmaWTbqJcoksC/pk7LPb9mbBnndEoNHbWRL2VVe8MfuugwOd+47YHLhI1m0BsuoTO0pO04aNXDdOZCkhUg0v69USpoWVgUCcJ7XIpJoE+UJuJNw0WavsGWqJ+2BwCLdkKAvjilxZPWeOdIAyggni6Ds3PoONc9YBhzd+cs661zpj3fOdT3/HL3PbO+0Lnk3LuE1wHwDJT07KVmEuSXNvNogu1+kcQC7OQU+ZXvNNbMqrDW8Dyyijl3b6EkBFqmrGyRBd2ucODRv3YMCgTXIUQIaRAchBgC0Bt9gaPGR9OxDpwimUnD7UKgCpiUsRj4Ikgv1ns9BsyxNv4IC9tF2cng+cPLWMb3o2//JLLy3NzeIGR3o2HH9XFjZQaYHH04UcqangsnF6Ajb4KGIB4u2x9No6vFK1mF9Z3hro64cFJCGPlo+4D9aa7NzkWOCtMEfKg8Ohph/vEpFOvk9fYcBZ0wZmMXOjU2ayrAMAUjBpZGeg48aYc8DGKHIbxJs9QIJT6vz8fB51IleI1Kg321PtqHquXr6G/w6GQFRiD9x3f3rjaz5fi6KwVy5eUaafxibezaCT6Stzc/OLxKGOje1aX/nm8gr6Mgr0unDwId7qtqN3ESHzzW9+Cwvfgw/e39vV9fjjj09OXjx+/BjnL1++TJbgTD4H4uru7XH7c5TS+9u/+3yqK9nmbHX0tJeLWZjmSzPT1HevrWdHRvcuvfHqa8dPYm6bnF3w+fOZbAXQwPpJ4i2vK4y3xnAv2sNWqEGmvQhB0OCppbV5qiu62tyjgyOUw2WZzC/NxVPhWq04MbGWiNr6+9q7O8IYGdaX1ojpL5aqj9/yAb/NTZL/7oHuvo6Orz3xj93t7cH2EAESfZ2H3dvhp9949fY7H7AluiYmp4/e/V3dHZ0nTl4bHBir52YTwc2ujuonP76rDfS6UXCHba3Ssi8asjUytsa5Rz5xxB/q/O1f/TomspA/XMmvJjuCoWH73FRhbDC0vIT0KTAXksSDFlhBRkZLZW+jcACJ85UkuopZPQSuw301lyXZp292egINS6WQh8uSblU+nPJZQcrDm1QmQtHQKgn2i7USXtOZ1XVS/q2tl9Nr+YOHbiXhKO5dVyevJmLhkaFh5Oaunu6rE5PPvfAKoWhQGogT8IWnATCDyQBNBsQPtyEiPnjpcrN5/PVXLl2ZrDYJpi18z6c+fvPhW37x5392cvJqbw9JlSJLK8uHjtz2yqvHQsnORGfq7PmLA8MjH/vuT554/Y2LZ96k5QsXLhBuRF0iNkg7OAEqzh74LFUrOAp2BIL4TldJ1EySQbJWGRwBUhcNATHDaEAk+S1rDL/cUAuUmaBnUR/1HUzJ+pFjJ8ItQM4QS0NpdKusKxbXDgGG9FroBY9VyDN0yagLhf6wKaJCMxTNwsgGlVpZkeHsUZNDJJQgApKlLL7+VDsL2uf0BKhFJP1zWBiBexzbmcW5hdmZwkbBQk54Tbma5EyTbMFX8OFWWnnYZXhznMWEAlirEDR003iISpZCvJPyDsabRvhf2NwQ4E2V2oH1lqFcLk4gPmJkoRbI7fi2NGE+yKKKw4lwAWJQrVjdIhYcp21MUpsVmiS9B1ma0+UcY6AIIizWdifehhTPpgtoJ0LuIAdbFdv05HQ8nAx6IiFX1B33jo/sGx0c97mCdfiKDUVMxYIplAywEDiZk5IIPEnGBIk3mK1FiXHH5oOpWVSjqhQ2LbJIUsEKE69hZiQsi9YxZrKvguok2RvyIVQPWuQXpYOwsMNySRjGd4YVoK2RyxeUezlEoUUMDDlcCf2wEGF0EvnV5hqB0dFU8MDYcMdouy2Os17JFmCwS+TMg6kgHg/7i2Ra3oN2EtQr0i6RWiQIKMH7kET/UAkUz9AQsmsTlVcnLyEZRPDv4E5RKXqL+wF7SeH8o1O0JnCiDcCQPRs/9Q+bQFP/6H8Luetg5wbrNuveG4/cOKBp8+w/33EDA8VZ6/26bDqC8CtJ3WhQLYLBTciq0hXBg2r1qW3dKwUS5hpsOuJEjAsV1xh/NoXhquwzyh9j30XegxsBJI28y3SoSR5hLYpLwUUcRxe8BMkuAD/nCGzjOI7bXdO+tFrGMECtSZLRsxoptI27APkG8O9H+eSNJLkdB0CHv/j6W2fPTi9uO3CPD2IkonXUJAQ48DInhTpxMYVcoHImHSlpbcJYRQil8EOJfSSUJ4uzChzJ6sGw4HzBd4JJ5LGI6oavBmHgviVLNcuZSHj5MEsSRrFrBoV1yK3mjOaYMWKD6WDla8h4XuAqFTQDbp1nYJlv9tBkzvMONQDvyPLb3qaCAjoArpH4AZNPe6Jr6sLcnsP9KAimJ5f279/15om3vvXUs+hTyNFDn91tFBggA2KJPDzo3/GD+PB7P7RrfPxn//v/RMQ4dPP+f/OZf/sbv/RHteoVpqc91v7x7/o4otLFixepfPPqK6/cc88tML74Qx4+cPAP/+ivBV0Mq9/23ve9/7Zk+1tvv3XLnTf7Iu5XXn9+ZuEyYtbzL74MNsBVHJ/qbzz1rQtXLx65/VYqSXRWGidOXuXZW2/d7/PG/vgP/2z3+AFSgmzMn0JmdQRwU98iGjRTKDoJioyGCNVMo3X3+SJUZqvkyS6br2fjnUrtmYjgGVdFaiF7Sjxku3gZluLiS/ZaOju3np6anT4zNX85EBy9PFclc0Qg1N2srY7f2TdduDB208C3XnsyFe1//5HHvW215amVSl+bry3/0EN9qSGvLTcZROMWcKWXm4muCunyy+uXA8nkXR84lMs2v/Q3L3uDCfw+PUGUZ/mxPe7ZieLwoC9HNhYwCuY6aUS0FviPgcXhxOenDWEZ/ohSA2/BM4GKlxZnYTFRkOEyBCwBKwSEUrMOHEn4JJlE8bbCKmrfirYyjcw68djroK1YpGt5NZtKtN98cN/YcP/bp9+AdOBm1ebo3r1nPzzNt599gYAJw7/iBIbXFupMSDBGnGYhS1ZE8md5SHgBYE9cugCOGhvoP7h3D5gQh21qGcF3eryshra19Q0KHC2sZ4Lt1NqNXLk2TWwRcAgHds/tt5y+ad9X/uGLWG8pCEFcXDKZlOpuc3tsbJykK9XaFewURCsBvUhNHeEI4AuK5RtZMUTeCmGBhPmNvYSVjsCE+IOfoHAOxApsD9I2aI2BAUeBSoxoLKGXH8hXOsO/OuaEftM4zYo82xTuwEk2g4+EkyAJXDWbziOkSxcnHANjzGQRRIh5u6bMkm4nTsXkqPOHS1vtDfLKkMXEhpUEA9Daajq94ZUGi8rGJFQjwbFSV6ojSk/Ay/kWBDDMXyRLM5IK1BesojcYizZfCMZi6YLsDbrX+82D9I2PgkDgS0BnIQe0iDWSXOKIH+ABdH1VF3UysVSCa0iDi05VVbXAUOjAwTP5NIQQLMbjlBRGUiGxPINKhUgSb27ZGyQd8CSCqYK/tntwDwQY3/H944d87pALd6YaMcoytvNW6BOtCd2IUwExi+LQfbiiZHsM9FotZqvrC4TFSlVo5HtqOWOnBoUx0hpXnhYx4rckMtEsRkj/8/3sUADpJzyUNHr0T3hVABJKeEjlTUUUfJXd7fBszUxpdSO7YuvcCnR7hvvQrCXtqaDN18CRHy9Rgssk+EJ9cZKkzBkUFgYENQmFXKlIoZVmKJy1B0CodG0VaDT5I8VcyAmcwHkMAqK4rEgzEaJgRs0shQ2bwXg6YtJk+4BF4l+z8bXWAXvrUODGne/ai16YzTp546p18p/tTfuMuE5b9zNy1j1QX5qyfnLJEoV5D3m++cnBzp+R9rgt4A/AOcGGW/eK9zNLApDUN9ko7iYazE9llTNqPDlVSaEkk7uoDo+wdslooQothCEEa5suoKm25a40bYv5AgMJeYqGHTCsBNZSZYiAQlYFXp3lyiYJ9MpbFEh34Tu1VURFgTulpGW8mYJh3PgbuTSaNoIleY2NkENcfvgMnF/aOGXbJLkrqIrEtqUieYO9kVDw2uUrPMA8iZWVCpl4RrFv4voUfIL6DHUbii5yt+IX2yL8E9iCUbbGiusaQEBN/wp58G2MrvTSiieU7xVry9BvXaBljjkvuQC9qMFaQho2imQXZJ7x+XF1WV1Jj+4bufT2JNlDBgdjKIfj4XaC7zqTCZI24XcDs1AulvKr6f27x/k79vprzz39bHotd9P+XXfceff0/BwLOZyC0vnyxcI3vv41FKQPP/zwQw/eS4g/FvDZ2ekDB/avrSy++OLL2KDzJdvqgu3j33dUpV3tVE9afu75b8GA3n7vkaXVmQMHDy741q9cnChXSxOzM3NLy4TWUGju3Kk3ScI1NNrZajrwkh0dCT78wMN79xx4+sl/6O4MZksr6XyOgcO/ldwMRDCQsGF0aE+liN8pFq6CUvESe9rmiCYiMQL5g75Cbj1DEEKxRrHcgZ7gxsoC3HJHe4jERVevvj000t7mqKCMXV6/1tk54g1EZtYncdZ99pWnTr99orj2+pljJ8b7Uo7GfH9X6OjB2/ft8xYmjxPLH99F3g57wEcFMhKIrmOHy60di/i23/eD98GF/NkfPTHY017HpuayZ/KNwSGi4JqYgZkgJhGUwWwCtEwcmk00GrGwLRGLwJVhGSArBSiBClSsYsAakyGbdCjwqW47tWFx6Gk1KqQBRpJGmYf3aLy7h8RasHq4tl2+PBHyhkkgMTt15dzZDkaBILWNtflAOAIEoa5AMu7qHvqzP/njcm6NdKT0I+EKY8WTLA3acbVhTob0gGnIcrG2NB+KxKggi9rv7OkTk9cuEWQ7OtgHqaEQERrYQn4dS8ctNx96+9ylZCKUz5efefobd992GwnGBjriPpfjpZdeQn5YWVkBtu+55x400mRnYwj4dlyxuMRYsHGAQRLUwEKAbBmNu7SLjA6ZYHHthWSy1KUGQ3kqxpswFYwkwjPCdvqH0ywUkCJtgwNFj3WSUxaG1WmWlpAKeIPlqSpUwqYMozYuWrhMrriMvgKhRCckvaCTpnF0LHRhpbiE3AkNZtkQU4SRBg07bg6trdLG+go2TiRTxEBEJ7JhoArD0CXSIV5C9EavkRsI+ID3CRBEmPWbvqp3+lIWP6pxgyu4XZYz0GVD5AmSKWkbKU5aEagBiSYqFG0ikz+fA5dvPgocQfIQoRBul2CN+atB7BeMa1mZxEnvJ2EeJ0BqjHsi3giZtrcbbgoTpzqTn/rI91EtnPyRYAfRegl+sAuIoyThRGcMYwH93SQHL9Fp2ETpNqml0Hxi2ZULXYs0mGgNySabxV+Jd7MMnF5cEtBnwBbwqfo2Q8X4NvgIDQkjYTbouTAkA0XvQX/4fTHLGhw+RpwTJRHJPlvfxqC2RXKS4rar6e5whML2jl2xRH/Y3dMhN6vSIoJ3m6fpCTEmdZeD/FkgUhpHCyG1Cup3MbmMi/ohuAIojIMv9ZYqzB3cInIM3RNkaN4QUBhVoVpBFeBlpoorYjxM17U3k6xL1uQZU6u5UxfNzULqHOulnBHfcv14hxHR7x1w1IFQ///jRh/o/Y1LFgDzE3EdOgEaohGYBynMNdqK4rWasvYgI00FXvlyahScs9cfukWmSYyi3s6o0At6K22zjLh8sAp8sZ6Req0bBMJgZZS93Of2ww0urBVW883ylofCE8trxXAkPNTV3zs4GPC4M+sra/MLG+ks8uRGjpB6O9F5hAf4wvFwuEVcPwZ5aoYDSih2WImsULRznal29ElQKb7L4/eDJSHSpSqKQ1HUEL4ZgRDp28GfhVIpQ3LWrW1c7plXIAlZA2TAmpKOwqnSkXwb3Wev9KT6eoEfy8VMiuZXy114QtMJO8IZUIVGFTsQbtIiy5oVUBQLm3t0HZxAi1wx1UEQtdAOwdXoH6eTGNAPfNfHcIpdXpx77tlvkvaBpzdWM6FABGccwIAc+27nNIHW6Hviochgd+/L5BjftJ18/fjRO+9YmJufunaNoniIZXhKU5IWWRka/Nabr4+SG6mLpNHJ1bX5ickrsVhkYmIJD2vSI4eitme/fWJ0vGc1s4Hf7O333bLrwMCxE89ioP/835x734OffOqZY/sP9fvW1t7zvvdembr89Lef7h8cwLOVwZ2fz+IrNHHtSneqcmD3vnvvueOt01/GGuAjsSJYBgVQExOpLxYj10jPZhAvu80zJ99sjxKaiF0pkMus8lFEdcUTlI0PEJYYiafe/949b546jTrU62yrFjIBch5SRLSQjftJ3vnWTYdvf+vtU2+dOvntJ9/6mZ/4TwQKXzw59dH3vH+sO+5ozl8++5X+jt7Fi6c74imnv2RbqTScdW8y2GqWqIvox/nVWyuUzkYatkc+NhpJvP/Xf+kbBOygKHe7W9ncViRIsAxaT82tNiwFLAmcNQi/rVFgCpxE2jYSGFZxlRJFMHIC3tCQOn5iFkTtj7YIDomxpAgsxnis9Ll6ViHFyU4g7ZbDNxGBtrSA190SSfrLxY23TrwCKcW5wO8HZ2+RmymdyQC/tx65/e/+9u+HR0emJ2qoBrEyANUkvwLmE7GolGooUZoNr53yabC6jfXF+fXVNQKCoeleUsE2ijWVlHWM7drrgj9ulSJxar3XB7rbC4StUtqhsP7SN59AvdONYeKxxwhUm19YIAsHG/mw4KswT4AWkIkVhRUOExOMTEygKyILlj+D8pDNMKWgdoLCSFXJkocYYsXiX5PK0ElHhMjehVMMjkOJQEIsdFq6tLNZx0A8pJhjFqOF0reoM8hbRIB5syi2FQFrEeDNMvELvEM6CGwFCK7gAhCmtFlENyAB2LeK+VphLbfmXGJ9MjW41KHbwYqN4Qf9GpPNyJITmhk34h4vUR9AbhBFE8RirV6xAeqkuoGkQTesQVBv9awkTLrPOmfNw4uRrwTmDDoulQpVf+R3ig4bn3ChShhe+uzA4RDwwksIJbPRZ5Ajgsz1QQKE+H5YOinNWQHynCoRJlXN2bHrD6Z2u1pBjLaEo1XzTWxSGF7JXU6+SSg2OK/aKlP4JICtt1yiSzgdIF4Tsc2obuIZXCmT6oSplM/hFqUu0ERo2mCuZE/Eomhsvww7G321hC3mSZ9v9nw+KwFCwJfxEJoBWtaMYOaHN7PX8qVVCnBt+4kpzlXt+VDS27+7vx2FcxTho2DbnNqq1Lbd2/K6IA0staVBjJo3Y8xEiSJtIfVLySqq4eYVYgKQvTD0ypdrEw9Gpcoyoc/MCv3EoI7uHFQNnpWdAxSt+ZJ3OkQdckXnrY1po8P0n/cxcZy8sTcHYvXA0eZmCxQ17/9vm2lAffyXG/2mE6KMZrvxFn5Be9ib4RWxh9LwEwBDZ6EBtl4Lm4OMaeAKWgbooDHhTmiv1U9RJHHDzIlO7Nh6WTaiypiAAD9ChPh0qJEh0lpRgApOzXDM2yvpytSarWZrFqha7w2vVbaW37507K3zvratgJsULohPnvRahoFntQSjiXI6C2wwlJi+GPhIJBzp7GTSxQDZtoN+78DAQCScwLSZz+fxZUSf7Q0FYC9qaysDwwNYJCZnZukrM1gukqsoUMqXnTI96HNBtXROUAV/qDln4rTGrG8138ujIp7sdbOGyVzXKPCBiO8isNY95v4dWOUkDBLjZm28jqsMLKQev0AoL+8khpDYEvDX/NyyP5S4+chtZBC8eOFtVAiopslDSVkFZifgJRpGaeAwfXzjq9/8Ys0WT9j+1b/+fur2PPfii5euZUbGyMBL0GbwwolFLIL7D3UFEACxj5AyNws84k9dgfoiH8/OEsQr+4kXxZbdlognsT5+8IMfmJ2/0jkYeO/7Hz13+cTVq3NvnT09MtYJmwlQX5m6eviWg1dmLmdy66yEUCiY6sCBqHTpwjxV0hHODh3c//apL5HpMJZMLiKuTsxnMqSwSAyMDo8N7CG79rVLl8m6EXYRle1Ididj4djiwhxpU5RB3+UpowVrFKLh/Ff+8bnenujhg3sXJmeDoEoy2uNr5NzuG+ivlFZffOnrD9zzME46X/qrv/2ZH/u57GT6/KlXbx56bH1trlaYX5otBJwFciQ6h7vL1875E4GtIgE4rGacQ5rbzoIn4MiWT8dCraOPj/xE68E//53nu+NhpiQUdpNNhcWiFLwIL8C90noTGCOKDHjjHkNiQeQowAo3pVIFZbWWKXPOxyNEAe24K5OyD2AglQaClTDStoeb0Y+BKwCUqdDUkSNHb7v1lgvnL4FbiW9aW5vDLzqSiI/u3g0nEE8m3b7NPDVAmvN4EB3cu9vrsVFcFbsqGw5BeGVTnJ7SIhWZaYpQMcocgFm9pJGhng9yMLlX7VvXrlwY7JVLTjjoivqdq/n1C2+8GHA7o+2RvmS7Y9fQsddemU/n7rrrHtYLXCNQStpwfOZZO7fccguVpzlD3QigYmJqkldzT1dXlxPMB/iyvM2OcSK7BY6wgDFYT8woynqoLxEyaPER/MW4ahOSADVzpAOpcoWBONa2c8DggyvNh0KdROlYbTQvhIlEyRzovZBFNMNas/xBQXAm4ZDbcPEhU6dBaqpsxSCwsuQmZETx7dpmRVZMomS3bChleAiJHrMu1BdWivkzRJe5V6APGEzUgPeL/ZZ1Wx+hKRB2ZjPoQgc6L0QAxKhl6BHfA8KXgoDGUMHjjm0nnyruD9I58+nyQMWWLFWZi+g2G07nfBjjIf5OmTMJmCzia03mP/pvxkxeCfIw847vPrRr5EBnMlVM13LrFZW4dQawSwWjgTL1mtfXpAGDwfJ7wClkDZFEQFgviRBQ5eKCXS4Usxu4d9ZKeRAPWanYwzAp1T6YeZtEHLB0wk0aT2Eq83oODNqC47I+mLEXVYa6aPSFAZXYQhNIHXWmpBzvDmVbK2VHLtzjGd83Hh+I2vxwI7hhrrs9m04fcZyipZAedCTE6pG0U8SS1mEGEHyR4MmNRJOb6H+U20mb0Tajo4TNY95FJJl3Zf7nPwMDEG4eBZkbUBGBFU9ggZg+hvt1xVAtfRFf8C4abKCayyK9OxNrEc/rP0xDwJ6aYq/Z4RmOrp8xh+/aAQlQYD7IbFwQwrh+M0DFN/GTAw23qIKoDm3TS0MyObPTE/yJGWYG3FBf6JCoHk9omWgl0QhwJTgFDekpkBBudyhRlKODW7mNhSoKXCszjbiXkJ3ARrrmFjnrEJedfpIebLtqCD2AAvwQSw5P6kgiBc1Fm0pa10was1YhRPCEspN6IUvVfJ7lQ8SOo80D+7+0sBw92EGWSlSEnk2EsC2wFYQYaeHmW27NpNOLK6vMkC/gxQSFayRaTWwHGlWmnMGxGAfmxAySQugZPGZLQHZj3JgxFh9X+GiNkvCE2WiGfxkChpWFB18L0uKkRYAZLKIz2HMPPAFDgtqZyEuUR9wMYxHxenu6Qq+/8Moj7/nAa68eP33q3OBg7/m3Lg8MxRHlcYEuFcpI+QRQkIC2IxVAMvf7gMntL/z1X6d6OgGLhx7YzwwEo+Fkd+rQ4b3nzp1ZWlwmZsaNkL9dc7Y1ZueuojUAFdCrD37wruPHT2bzjVQyPreYYbKQgWYXJ0b3dYtb2KoNjXSDke8Z+eixc2//2V/88e694xuFFcTfdGaNyUepgHUwEU0N9Y8E74r3dA0m4uFiNn3PHXfVamslqh9kyk6y12w51mbSp4qnRzsOjA6OTb09HXEn7HUlNapmmqy7QsmRK7bSlFoni7DTe+jmA3t33/x9n6y+9Nyz6/O5Qno5GnDFg5Q5d+Fi1BWLzU9cunnvGHqbVrFCzfsnvvh3ndH2q+de/vjf/v2uftunPh72ulu9vY65yZWRVN5JfgeXk1gPUiw53KT4LZEj1xfOhuL2UvVcI7d8x0c+EHAmf/W/f/GWvePYWMlzRekaQIuJx2DBcmDuIKKoKWGsiFlhTpkpvFuCIYWQEWPGhzCnkFhgB8RJ6iTkMFAAIg36NC4BHQShQftxDQQhnzh+HBGWFA5E6yJfgbe2ai2SQjY3K929XQ53CFdbytShWrt27QIQTiAMSmBi7SolmM4qGlWIGn2IR2O5bIY/GAFKAEA1EViJXs0UyizEejmDCqdRzpKNo1HKVPLBWrHy3NNP3Hbn/XkbqYiA/aBnu9Hb1T4/M420Q9kEtOjZfO7y5SuAwa5duxC3eQuadQRiyJDlF81akxe0EI8BdtALo4CuA6UXmBBUhLYdUY1xqTXxA/D40DAJQWuNaHWYpSJiZRrQL/AOq4vWtDGGMOy4P2PAFd43Us8mReAlQULfWXMw8pAEMI6kHCE/6BkoyqBRCALcNBjZ0D2WghCahSLFF7MS6R8qWXA9phB4DlKL4YrCaiS5pLrEbKNhl9JTuXigDUgLfsO5sabBZqbb6jxvNvEjUBGdM2ek/ZPS1HwkL9MhGmH2pjHMcqxz2heyEWGQk7fs1BSihxNXX2H9veBQ0jCiG/a5Y6QJwUcbPs7nCeAd1094R6R3vO9mlDn5lRKw2BnuBCbQJdu822srK0BGMCKoqhM2VJbqPhDxUOYTnISrdJaQzUIOrhyeBf4o4EXbSz8YUGaP2O4tw3iSAI9u4LasD2M8NQhMrn6JhUKxKb2FpEcGnZkkbb7q+aFoZ8raiC+iPsVWrtZWgNx3jcd69x+ydXlszny1tYbvLeU9XC7YkTo4j6fJN625VE3IGqwlh8aRnbdpyGxV1AXoJzAMEeKHDKH8YkYHJaHWWlqgYY0qAKMJ00lsZBxyQAdpWrMDKKGXp0FOcwmkLZyua+wtaOR+s2lm9alm0+d+52Zd5RyXbhx/5y3f8YuXcaP8yfU6KZPZTNu8Wn6VyJE0xSewA1j5OinhxaOJtDIX3Md403fEUBrjf9YIoCJyC2UF/miOoRPzKduLjJvmNai2OOCPRvQsqgkcAXDRcnhzrYKj7i5sOfJVW4mQdfRi5EzOZcEgBMuxfMhSA3ii7MVZGpWEl0w4bZ54oiOSK9G1QNBXYZK3G2SMikQDTM36xirQSLrZ3fv35fJ57L4wfXjQFIslJA9WLo6p598+g7MxAk2xUuaTyXhKSbGuZIp8uKhQNKN8NtK6Fp3WAmsYLMy6Z3CkadH48NHACLowjb/FAXPAD0bVmit9qTAQQ7JJXRX+GBk2a5xMliThafgSxYDBcNQpF63oDiCuvT0ebe+YXlr/5pNP/uBn/tWTT/zT+urGrj3909MEuRJW5afaN+5kvm1U6yUS3RIGTWbmUjVXrNRWVzJUNpy4Nt0/NIitFt8H1E09PQN4RJPBi+rDxJVU/S3MsUOpzjvvuZNXfO+nvv+hBx9bR9WbzrGUzl++sJZeIavztStYfDfOnN1cWl+89ciRrdIbdx16wP9jnl/9X7966x275xanuvs6vX7f+joZ+TLxaMLjpiZbBGMMOcUCvs35hVncj9F3+X2uOBZTJzObpsr1s089e9NP3ham2PC2d2MFb9vQhQtXLl2tJHttcazToQA81cr6xlA2tzg3izPRTbtHvvwPf3fk4EFFLzVKsZCvt3Nvq9wc7+6b2944/trJ7FrhPY9/uLyRv3Ty+P7xzu99z1AqlOuI5hrpxbm8rb+fRBVNjG61XN4Xj9QJjC1XIx3Yg/HsxdN+bdvmi3fFN3PHD94+8iu//oO//T8/V8ps3Xxoz+r8pFe6BspNkmYHxYabjAGkmYaMktyFoLNaK0sOJpAluY1dLbIcsm5a1CwCCTCV6PYYzFAw6iDOBIckEmKCEMF3cHpU+AAlhkPHj78RjydgX6iHAaAiBgDDyUg4vbERijpq9YVKzR6LJtNrs8vz0xcvOsqVLAIbOZsR1OCuPH4POmcKK+FnqoUNhLo9kGolscGw4fGuLq/4PW3Dw53YcShtl93It7YWCd+lUm16eDGbu4JPWAeJ0CIBSvF6Q77Dt9yMEJzLp0mXisEYzRBGUkT5laUl8ofv20O9j33YrbEQ86lt/7oXrTqEGDdGKStZ3KAYuRCjBDbrCNQt0d6gRvAi5ToVgoJR1OAsrprlA1JBjDXok9G2LtEECMaUCdW6YhMN02bWmxSUyLesJRFSIXAWF4jfMDnmNuFEyznUMMKGYljYTSSZRtiYCGnMxUaYtS4LP8ZP+NedVWpINkterwbL8h0GbYPKQIrvoF2aMtcNNZIUJAmDNiUf4afHdyKoi5bD60s7LwxLf0F3kFkXehIlgFAgut+G9ohwflIJUcG0mIOkMppeqGRmLR8Jtt9+9I7dY3tQuuL5HHCHW3mHqm8y1sK+cBsaPMsuipobix95YjlDLhXkDIwa6xMz6KnhOeBsJBLQJaRAlMQaRPONRrawJAwhJ0xPVlALhi70uEoypVEFUhGMjDYBOyJMAzGjzBclKdc8QXx6WrnaaqaK2rnRNRbtGoolR2I2R3kL1b6zwR+ji6oZVR9EGvUA0yH2U7oNiDh7xaXKdOfC/woCoFhQUuGjeyjnCuqv2cwn021NNsNII2waeM0WCnBNsaiNqJLgx5ov3sDMsKQtaZ49Hdd7jaTL6lVzhpAJNncODJXWD5159wYcm2e1u3Ge97775855zTad0RV4AN6mCRO7Bl22W1ENSPtyjBIHqRYQV4EfozAVUUKTBOiIHJOqAOunuoPHA8ySagXamqLEuXyjb7BjfSO3tlFPJaOk+E+v1TuS/la1jsGSl9lJSqCyTyCwiMMTvTyxdvZiGf2CyiR4gmv5Uh8ZcYPBZEeSFX765Jmthm10KIUrI7nYgsF4LrcZjCRWVhd+53d++/S5tz73l39Ksj6+i3wU6N9QvuG9hJmGmODJqZmxkdFbb7npb/7q8wgQVBahQA1SL7IIA4IWVrVCsHlTRx5li+iotmq+4EfNCHSh04B9wH0EYyxKY3HgkGFcteXLiI8SHwsCRc8osAenEooOeccRTDFpUtAxgJxhHMnihhQFyJSr8Hzk3PMxvCh8wbPABTowIIWecEOlUqcOOuIIdBA3mYcfe89bb1888ebpZCpBMDteisTpIpEgAYPxMNuT0pUXoYJmD3Lce2j38sZcpUHSSqZgNZFMLiwtk2iJHIqoEHPF/OT5ldvu33v//fe/feYU0EtduEJ94xd+8b+jLXj22Wdx9snlMzAlyyulaMwR74j19nevZlZQu7V3xC5fm3rgofs/9d3/VsPiLB9/89nzl49HYz74XRSkOB7NzW14XQmycJBE8Pbb7sLwWW2sOpwLPjQLqh25XSqTD4S4qEY40FnIbH/k/Z+887Y7/+pzf14pr68sTW1t5nGzQ6yLJdsZrtnZhf37Rm8+uD8Vh5O49MBdtyHzkepufnISHEDMFerpzng7megmpxZLm+FEx67+gQPDvb22wvLyhZcG3Cuu4iV/a54c9yxBoo9CyYAt5Np2F0h6IZUlljhUWij7TAQDQOF2Joob7pBnzOY9dOqp81/54kutiq03hVNLDnGQrNj1osvvTWTSVcpRNDYz2/aCN2DzhVgGZOlndcR93gjpDovFLEMExcLshwCD7cPlCRCtUyojVimkk8UPgUQslD7bo9A4WDkSp4SC4SJAUUNLH0HnQEVIOnrbnXdOTs3hAk3QNlLFzMr6uQvnSbFXKedDPlck6O3v66TyFS7LSDi4NIfDIVyMOjt6WNp0g6IaRGT4qDKAe0+d9CzxRLIdjEp5PaN08WMM5EE8guWU5/bEOnoO3nILfgNA/v79+1999dVbDt9y+uwZ3AFQiiCxraxQTMGbTHXirs80oa9xwh6DU8AKkAS8ZSBRqN1h1+FYoeHEt7Bu5PBBCAlzIRlKkqD0SuAbqafFY3MM2YEEsR4hq1pUiAqgFp4Fz9A+V5CPDFKEdPE9Bn9Jaag/NSUMqwDhHYSnf4SCrQ3ibC4YOYiUHdfPc1WUkBO6zks4Ej4wK55Xm02XOOBeqYYtO5Qu0IWdTSPAajYIWjfSokH60HbhWPMSPY19gpcgmYG2II+gDPxStpVHDas/pBgfMRI5oRtjjIig2ax7pHNtOSr51nDXvgP7Dg33j+KIRq0RAtPd7iA6AnEzGiXsUOJmEGykF8b5vlhCH9vZ0+nt60WupfDVlcuX20FNlFl0NAiE4lnDMKmD11XJ6qohaYbStFFBo8r8IB/YVd5Pohefz+SRTgG4UWCyPABqZDmjD9DdgL8tU1uiIroruj1+qLN7V9zf7aRWYK25RB5hE86N6CWPGt4iWU4clFF6i1ExbwcfEvHp9oruEldNoDUqemFjVPbIcLibCHbAAmbSNLyaO9FQPpoWAAaOAR0YOhE4TZOGnIt8JW/VpxnizUmRQm6WKGUeVg/MqX92YP38/76/Qbn5RjoGvKsror10TgCs7GZ0TP3E3g0pgh5gypLMptmkx8RZgbQMKGqkkAIBIPk2MI1UGlL4nvJ4O9dX1j3eMFh0bT0HjxqNhTKZIilV0L/C/eLfT1Z0Rn+1UC8qM6QDno94YF8oEUt1f/8PvXdxdXlidpK2vu97v5tSrJcuXCL7RKXUOrR3P3nm3cG2hZW13r6BL/zDP+zZP/7gww99+/ln9uwdB2hm59eg7lRpJeMaCzRfKaxl11kHWEAphg3KIusQPlaUdJ2fT6NjY4JghAFiatIBZihRmY5gNCQel0+XvkkGb8nBDgUFMGX418E5EWoAzYb5QKVI+jOIMWy0YEDCP2MoMEI7AnyycRJKLS80QklJAUeIQx2/AEXui8+We5bmPl/GxKPqDmTMILa0Utle3Vx+4/VXnZ7g/r3jCwsLPAH1BXEzL7Cz0GA0jcg9qLYJaHjsve+ZuDa1vJIt4pFgd16dWEwmo4Vi/f77HgRLHjt2bIHBsdvCCXd2vfTFv//qkSM3U+eVsg2hUPwvPvfX0O/LVy4TaRONxBcWV8d3deGkNHFlA9YhEW+fX1o8fnVq/8HBZLhv4spUoZy99egenHxhsDq6ohRW8vtC0A2So8/PrnUkXbMzV7pSA4ODg2uzU9Gkq0E5byQ/ux2vn87eRLFMfdlVvE6ffuYryWRk34HxkyfXYEfJGXBtajXc5bj93psPHzgMJnniy18Btk6deqNazExO+qulPHMAnGazxSsXL2017XtG94z09CNtwhyM2ZM9A9skBukPuw7u33XlxZO93kYQVyUKABKgkLOt1cqOsD3WJ5QsrRnQzpqVHZbICTgqiHTJHyHt/+WwJ3jzBw9BG7/0108TgOr2u1ZX6vFYuHug9+K52d6u4eWVeRcOyKAHaIuHNMa4LFB8FtSsdOJaONpa2NOExkEXLRKqRHGtQn3Ib5LCEIfEVOYI3sXDvm0LKzJgs55Jk9MKAx0NUHJ+y5YNoUvAw2tjdX11znlo92OPf+C1ty6uZ3OXzr1Nw66Y6hcsLa/K2wsxmOUIZ0kKHWya5NUhyaKCIZuALq4DJKBha3NW/aEtJchMF4FrJYIDlXkx80kKwp9qfmEa9nhleRnbYHs8nIgEUcdQaOAqmI9EJcEoi4KSwOR+6O7pwySBBlUOHdjfkKggJ9AKBhdQY2lpPSCFK8AIXtWieCS+YlWJ5LFeQKAgDWZBYpjBlKJaMotqNekB/ke0MwiSiQL33NjTe60pnkJIEwoztIfb9aSeZ+O9N/a0YmFP1iMd49jszYo11VBYulAarVjeD0Izt2v1qgta2zuP89NC9ha5MH3kktoXktUT6qUO2Ju30j8p0tQhnQU/2NtI1ILQhnFXqhKHmxTcyCD2LY93K4AarFHEtc0dsJOmhkz3W1imDh852NPZm0p0YU2Hk3LbfeAp5BtMNTB32HLBI4GAH0U6xSSoRJ2IhlPRUCAUsMHaX5tcWVlCQMFnH59U3q4kkaqrZeRm5C99vPk+Qyf45OsbCQGZStzsQWSyvQKafATjDdpnKrFwb1L/AFaPnA2IH5sofCq+dte+/sHOkYStJ2AL4FiyXqusbnuJhCO5ElQGiNSAWS/RaGherRdKC6JJ54+RI68BC0L5d1V9mSWlFN5ScYiCsqnX1zfrJ2c44E5rvsDk0sqYYeeMdfI797xeN4sOanLoh9UVQa91rH//3zczHHrmXXeJ57rxSTuPGqpPl402mbZ1g2mZA1FfrQbxEAwLKB0WVmHNsFQAEWPNaJn7+TAGTAPAM+owameYKFwdhHQ0icBTtSw2CCxD5W8xww5vrVnGsARzh62FWFe7179Rqq3mKg53cGaxCIXu6Bq4cHVmcu4U1dfzBCHWtpchwxMzd9/zwAUIcLm1b/cYOfNwo0p2DvX5upkRBD5q++DBi9vIJz7xsV//zd/Yt3cEvdzyyuIHP/jBcDiysDTPx77w7HMb65l4LIZ3LjALOsMdJhRyxZMp7FgMDgIfAKkp5ksMdwsSBc7AZug+iB7CaMyqxErLWBgGHqYDWR/JmY8TDDBqMGcAArfd2DPvXNJy1uBK5ci7OGCMuITJCTTDT4CQS3B3xKywHreMuocHEwlEIzff6PVCre0P3H8vc3DsjdcJjieTHiGA2ULZLVdGiLobx9Rbbr3pw9/1YZDXm2dOvPrGyyaTfi4WC7996swnP/U9ly5dYakRcbR7927cWckF9PbbZ0mIuJFdI6aBMAdyBff3DtCd8+cv0qfZ2WVW1q5dqVNvLaW6lgPhwL333rJIDqSlpY3M+sGb91ydPkNcLy6J58+fRV5HpR8MJPgMD/LmFm5BoUK+dNPhW0ik2+ZZ7e6DDjpW1pdgI6qNLCJjIOA+dGD4tiP3HX/tWy8/d7y7i/ii1uJSoX/Ik6nWe3rJNlWKxbvf+54PfOOrX6Pe9/IyIvrKQG87/p9UZkPCJFbS64mnc5VY+/ZmILKenzjgdR/aO7Y8M331yjVXeWFk98Ht3LXFXJVcVeF2VzlDMT1bMk6kLJiN2i0kloc6Mo0owJhLGyFYlWLd70Nd3FbJTfk7Egce2AVb/ewTJwORZNv6LKRzaX2pcyB8+eq54cH2YqnUqpITy0b3SPiLErPZRtVYtIVIfYIYGDt0wdLDokPDVIzmGm6CWh2En3g87aguHM5cbgMUKsTicFJPBM02DCFwQmQwydlQSlNr5tq1q7h6xeWBTHVbN3Xf7737TqrdL8xO8SDFauHf+nq7MZuhCCHMnbpe0uvgRBYKFfN5IDMUjQCHgBlwBZ9JOkl4Mo7ZeJfWvKwp8v8gDUS6Ul5Lb7AOiBRHyqYuAzX3SC+OzREGkjTSBDih655dWCQemLgPWHIWAC0YbM4aEe7kf3SbJC4yDDorCVgRsWFBkCiAG3ix9iAWk8GHJ4RKDRoCM3Fo5Bv+4SSIRdISQhNjidp55z/rWCpoqXkRh8DdOmZdCl0ZldSNveipRQ5pXkIPfd7Z4ypJBxkFxsi+7QIiwMdQJslzZnS4mYc5tn7SvnmWX+q+EZi1kDUEN3CwagCB93e8eHXefJ76YAgdX4Zjswg+TDgZMVpOCpU3KORn33Rntkid4d8mmXsFm34o0H5oz627x6ie7YdnpGRvuVzDa4AgRd6I/axlKyE7k8kSrj1byuBvEPb7RmMDWC02SUlOkUayrVUrJB6CtrkxV6C8ZtLVIbYd/MXniyHVduMbOBQdwWlG8wveFgU05yDGBCkFfflqulDNbNrKruC2K9TmdYDyygMHesMdfkdHyOahKuBstZqxeWu4tlKUHfEX9QZTCp5lJ1ILVWGoxaOJHOt/OmBG27jsY+0lSy+vlrOMuqtJNHfxz3duepwb/tlGR823cdHiovjFZHCXPkbze30PXOiS6ZYhxuYqP9l0zRz8f9vdACQNKTHpehMHgJrg0LRM1/k0fiCsYa8xHnaYW81QGV6QMRMPx3/wrIJP/hPLx1zyJ0WGYJyOlnER8NkxLiL0+AOxSnVzNV2xO8OguXqbs9Ao2z3UBIsXC+urJRvOUn0DybPn12vNeVQOA4Ndzz77Miw8yVFB7g6H73N/+TlsvaFu3+rGxg/98A8++dSzjzz60MTkzF9+/olbb93GSMlaGOgffuWVYwiwi4uZ3oFesvagRx3fPfbdn/yu//7ffrMvGSbtIi43SMAjQ6OYtAngIVE8XBW4CcrHp7PEJcLikOHxWtpSZoCFiY6aciDYZGFIKMUGFhCBZlA0DIZXQ+9uKCu47Mbscwy2sYixheOYNoYWHAf+JQ0Fe1RPgAsUlDharOcQas2ACTA1tyHjovVxq1LMNqn8N+655y6y+L567JWOzhRlkTC7evwuqXx9bQFnsL0r9sSTX0bImJ2dOXB4L+1vLOV7hlIUeIeXeuutUxBOr8+fSCRXV9fxa0UTHo2Fh4ZGCIopVDdm5iaT7YnVFd28f/9ePF2J1+W9JLP5oc98ZHphBg3kex5//ze+8Q2s5riFRFP2fYeG5i9NYgDGnL2RqQ/2IxmvzM8ydTaiV6klMDExCaJH0Hv91EsrmXA8QZzYZiQW3hccxXcCWz75AyamT6S6fN//bx575fmXceIdGO55/eTi3iO2bGF6Zmq6XnZ++lM/+tXmE0tr6/FoCgsWlXXWV1Zy+Syw1t3XvXv4iM/fvuUMvnn19aN33PngQ/ell2d9jibFxb/5T8/23r0/2jmabVVnN2bat8AMVGMjL3iVxeAM2OyUESI3ocGMNkcNvI1htlzc9Lqrdl+8kd1ozJ2Mdh/c/+jhWHDw937rb9u7+9eWV/hYaggOjAWam/hOY8RSgDtVFerEUypauMKacDtCqBNY4+yNhgXsLVMFIRKgblRKoBTihquBIFYuoASBsVwlW0CJ54J+fyxB1jPH2tpVL5rhNor+NiYnrpL3cXCoP5tef/a5Z8b2H7ntzg8Hvc4v/v3frC2to4WmRC6sJCCKVoYAVxS8FvjhQpFeX2eZ4yBGLl4iW3ARAw9jwhB3jRJWRWBUkpfoFaAcsMEHONk3gJFFs4+H9uQk0BiPL/ItaM7DkRhyydjIEAL0628cX15bp4oica0gEWFSwB7XNsBYDtRy6IFNl3CqYAKQAwH+cu1RVKYsnEb+AV8K6JWsCsYFBSe/JN1wGz+looRa0zzIiQtGKKIpg7dZtLxST0io0ArbOVbuDxYqv5UhBaaHN6kR6AfHCKki0GrFYHOmRJvMuRBkCIsyCBl+wswieAAkt4N26QJt8R6xGTQgvKomJXegDKF9IW5z1mBVvUEKDtpjfMy3ivrycVzAa5SM4W21mrzVGDXmj8ow4IRa3kE0FMPQkeo5etv+3p4hp8NPpjRyBcLTEAniJXgcZqwmLa5Km1EekIJv5Exos0XiIXLLQdZtlH/IpakEtrGyTL7cMHlLvX4E1BYsFam5pdNFXDKV2pAypO6Tu7j6dv1j9W36FoYZDwN8FZkP7lfhYlB/09ZYyi/6Io5InAgDEuivUnFqbM9AcNewrZ3yiCVbfbnRyDfbymRvdoZIKYKLKDkWwLZmdgzAmLlmRJgjDeA7OBIl3hbuZmRnB9GqbzzEJDJzGjsm0xp6M5T0knbYGBOwLRvHmlJDX7lZM6SbBEFMh/43T1g3WMfszZ9uZBOomSk2N1vnOGFmdufXu4+tIeOMdcAdHL+rhZ2XWq8GvJwAFSAhSAeGLX7CkBMBMyAmIVefDPzzZ9QxUGugH1gVraJXkCJDd3SD2fDDVOIp9FDUdEM8IQFGnRfZPVXiLpEkXdjMtvAHQQthd/hTnSO9ke7VrSvrG/mw23nnXfsyueqZC1OdvbFb77inq6//b7/49wStPvWtF4khX1pZpz5kZ0fqrbNvEzP++omXySN33wN7wCPowQ4fPjwxMQWB6e8bwpeEFBZQEcQyqOqePXseuG//zNUlIqZKhfTGWhUnpgP79iB7Xrp8NZlMyPYiAFNMAaYrOA/0N+jixECT6Afyi0hKUi+ZLzFuyegLF2a+l5FiAmGd0REIKiRGS1ctJhEYsAy6QDkb2JC1zKBxVXAuRMdt25iuCf4hgZHR0NEPedaDrLiHuFKUTPiVcWO6upbLV5746pcvX7tCoitITnqtitq1XC0SFhdJovutzi1PI/2AFjEiPfftlweGuln2EvTxgQz5F+aXCnlyxlf7B3oZ/kwmh2tDvVKnxDkBpuQrpPgLTl8oqBiTzvauQqa4tLbIR+HF9txzL1aoOp/f+okf/4WPffzRE8++1d7j6MkE5hc2l5ZnS6VqJEIuWxupPj1QpLZ6NN7h924dPHgI5QRFbXftoUBwl7G1eze3S+sbi+nMMonPUHXWK7b947u7kx23Hb4zFNj8+lefvXqtdPc9waMPDnM5Ee6enshfuXLt0//q3/7X//LZ2448UKmsTU9PFworlLP1pSKzk+TEmnF4SjZ/rHd0bN/w2NLitK+1Pdrf5bW1ffTjH3/uW1/+6Ace6OjtfPv1Z9KFpa5wxGUjzUAd6He38ARS1XU7We/lXc/kVCvpzRBlUcHCjVXyNfLPdv2qrVbqueuhf7f5Q7/yy79DSl38LvHlrLTK0Yi3Wd0ky1BbmdT0znKRYjCE0mFcUZUtlAogMgROEAFqWwADqGB20JkyzSw6optWlhehUqSlBJjJiwC5Yb1RJLOrqxN9JNV/8VdvT3VEIiFZixHe7W0bG+u4wPqj7RTM6Ovu6EjEi7l1qR8wFJrVCNoA9sTlWZAGIsC/z8ScCesTsuL3YAApUe6nmJcvrZ+hkl8YafwrCo+vQSq6BgZJjb62tuYn0n/Lls9kPE7n8uI86a+i8eK5c+eSHZ3Do+OBoLc2U8R+KVkf5pIPJjyapYAYidMEFB4XAMXmc2iZYcTtsswaMANw6iKfGhiG3qK4JPQ3mmpzWgjMOC6BdVjPBkeBg4XUQL96EO4QfKdzvBnhGDSrdWhp6mhWaBg1uFEea2/RWBPIyxXeBCFGfcVa14ITvYY50n3qlPCj2tJ7tN51wMaDrHb+0+3CAJLJ6A7HdEIqMm3o1NW8wcL0w+jYhZyt7zXoQ4mx24hhcGFztxPd7qFZ9Fw41WAk3y5v7hrat3t8bzLZiWuIEljUEIl8sntBKW0oPbh/mwyLvJw0GvV6kRw92DCUxjkYsDUb+fn51YU58lkSaJwMBXFkJcNVvVLBuTQU8DaIvQfJC5lhq5Kwpd5rMM0nmm+4vqPPUHYKxDXwo2OEmnhnb5IJrFLdqnjCbRQcI0lXasC/f99uZ18EHytbbZow0m0ntdC23VGU32KuGlvkXK3gTcXnSQ4VvRWNggqbQZYxmoUBVwAGRDAiFBsghg7RDXomJ3vdyOLRICIc8luX6LYZVfbWASetY2v/rq/YmVluN5esTzUzoqbMAeAiUk2b775qzZvpyfXm/jf/1YvUklow3TQHEnk5ENunk+aGG//oq4yQi92LA4Rc+iNfZSlU6JyGhD8NhOURrdZBLmSHJHcEdX8xGzv8ocDSWpmI3Wabb3oua/cEOjsHCATKVgvkbp1fJbnohi3SNYyo5QqdO3P+7VdnOxKZ2fntVGfk2sRMpry972Axmeph7jbSK6mO7klSUNaZwuKxN94c6O8t1So3He5/9LEHn376mYPBCFnxJidmQ+HA+lr2oYcfOXHiGErpgbY+NJZUKrj3vru36scuX5hELN6zJ3zm1OnJ6dlwIIzeNpuHcyOPjwJ+QIpinZS4l3HhszF84CyFzgyLNTSW0sDAksIwxNVrDeKgCBRI0q03jQefQTLcwKgDGJy3qC+Lm58QYJoHEXFeHAiLVh4EsrIDb1adUJYVG76ibM2gQnuJJOWlARyTvJ7Lly/VSCPscVGxtGswSJk3TNId3WFSIoQwrHamjt7+gaeefHp2YgVXHoSbRF97emljYFfHxsYa/jUf//hHX3jhhZkzCwfvPXDlyiXCOglTWV1ZC8Uip85c8vmC6Y1cX18ffl6nT5/DwExpoHg8fPnCQjjqXN4gqbstHHO+/urx0ZGBpiMDc/PCC88xp6SHI+wpGgtcvpgPE8ZAVoNiORSMQyoonvjyy89HE48zSjkKzZF6r1W4NkkuSVtPl21oqL2ns7+NBpz1pfTE9Nxl9HWDI/7b7rql3ppDyx3xSa127Nirn/jw943v2nfi+FuJ+BZ1aSk05PNHw75EJGFbmCYmfDnWOwKaOjQ2QjbORrHsHkpu1+C3sqQ6SudLW/6tOO5IeFo1cmj7wsFtEkWBIqn8i5OpW9XGwe/EeVKofMsbpjQIBTBaoRghdkWCy5U2qT41+MDB/9T8od/97T+LBGPVYpOqWtlijeBMHBrAJrhZado3W9QJwOGIxcUUAwQKtgNjWEuM66SHVgCSnfB0eKYGgWQUiw8Ga60KemOACvmEIACK/xC+RflGRG00wAyp6uA5kJTAa3ii1E++/tpAb1+qPX7P3Xf5vA7Ia3usu7+/H2crwA+LGdTUUP0tVAUEQRECQB4rQAt4Yw90ie7abNRUBxCBTDpTKhehmaEQkOzA6oF31ZVLF3ljBFMNuaGa9cWF+fvvewCtOs5Z58+fX1leJYCeKkm13m5ZMfk6iA4vkO59J/jd5iK1mcigyIaQEStHLrd4vsIKCPnqmiQccbtsLA02SUPXFZKgFkbRiazGUH/nxv16XFjY+hNJ5Pj6RuPgXN7CWYPThFd1rDOaFF0TL2ThW71VdF1UySxgBsJSbalBrfmdTbZgSbPmw2gAOcWQER6VDtBQBR6QVGeIBG9qiv5zxSAO+ijeAEcsB4XzCP9yU7sL9r6CKa+NdYjP+cMPPh5wh+DMGuTixfWZxK0uL4ER7oCXaAcixJlRBhmDGo02m7Dqlb7Obn9Hp61YWL5yeW1pkXR/ouqItlymNjepZZ3KooCJiIzkdp8H9tPgGZLM8DF0Dt6HXqr/2iyRUb+ZADxVcBxQuqHGJinsmFTClOlra62+Nri/c+TAYVsCyFyt1BdxoLB1+BqlNTk3Iowx8wROYfF12rwku9skcYTmUQMsiDDvYWxJ5MaNhvpCgIVDhUXlba6+aNy0vSOam3mkHZ00++tTBgCKCdPDZhKtG7hdzxt4E1QICHSPZumdRs0tZqdL2sw91hkDoaY7/N7pj7minYDBQA376yfpm8ViCk4YCKtJay+qKk0qHeXAggw9uPNeuWLAUIvW8C4kYfIxE4tEtw0yAUKN+hWWV0sHsFN65zollJvwcBQCxYbiarmJJnKsZSrX5vFvKTdc9WR7x1bVXiyX1wrN5XVb6+2zWyRpiSfGxsdnLsxiLaNe+sDIWCAOrxdYWNqYWZjv7Grfu++Qbbv6/ve/98C+3U9+/StPfvXra3j2JpIkKD556i2SEjz+3vcvL60PZrJk5IE3f+PY8YGhgYHB3sWlWaweE1cvXr08aWv6O1KdQ0PDKKIJPSHIBvGiu6erWi2jlIZ+wPgztkAkLJ7SYkJRQSceacvYAEILpWCvAZQQLWBCuZP7lXqLarFKryo1CZtG5DpUcHx9KiWgaCIUK+iCo6VZk5BQtB9ez0NBChe+P3nQMTZsTpJBkHuARnBvsVjgKtkZQ2HfwvJysdxIdQXK5XxPT4c34C9VStlCvljOxRLh0fGBD3/kg3/z+S9tbGSY6/7hvqX5xWgiNjc5/3M/93P33ffAn/zJn0BQbrv1dsSprq4eigw6VjTFq0u5vp5kLk1xNnywVcTs+77v+0C116YK3b2RH/rgfX/5V1/2eoIEEJKNp2+8h3ihQJj6ei2SIQ8Px+bms6SGyGZrqPBIR8GSmZy+eOjQAdxss8W1cCxy6erZpZVG30B7d9eAz78UIIPGJl5+jvmFuYHeoXIl5ws5jt7RGwxGVzcWgwlVnq8VKoFAzNMVnp6e/MxnPvObv/H/gzGuVd3FNtvxExeiwXh3h6rm7vFGjr35NmVPZyfPd0djB/ePoZvFjL22Mp+MB69dPOPersQ8jcB2A5ci91Y1TC5+TKYwmPi4ILCgh8DDxHhvUpd9mzJ4yMekiYIkoVzFAyDSvVW6Ul3J7n30rh9zfebnf/bPRoc6kHKXVtPRMDoNEjLglgKygRwDK05UdZEQ5lV+oOhAWpGzr9Ys7jKlIgIhhlAq59p9XngCppWAbDxsyEIC4ilTdm/bns2m0U9wgaqCeLATK9yujKEwwU08mSvkGqy3NlaWbr3pUDjgpqbh8dePUSYPUKF8L/rItdV1HqcUEh1Ir61iRYaHo8Afphb4PzysuQcyLLaPipl4bKk+KrnHZQQhug+zBxRdkNxqof4hjjTg85KaAkm+q7uT/MDhoB/CzMRhAAYgqxVGSbwnXyyO0vgxWDJVWyAQBKnKiQadALwKteDdlEeyE1WzI8UifIJVDKbTgCncRK2If7GoF1AsoxcTxZqROM8mJCvEC/oS2uKYf60zOparKHthVzY9qSWp3wbJGRnHCL66BtGTKo+EjeKO9Vs01bzDQQplANrQANOUGuFBjknapbVsTpgdamxdMxK8JGthDXETHCt5o9gT7uarEJbRivC8Ez09OY0JgCqx5sqE4XoHevtxmtgztr+wXPY6g3wmZA0HNbh5TAvU3IVRglUnywbYhp5i9RCtsW2P336vbWM1e/Hc4vwC8+FzugMBxYU0qxU0EvBEgCl4CbjDOwAHcxysZSAjJF5dQ4lH35TgDX2bzmicRHctMrxpr9ft1TYXij7cq/JVZ9kJS9sdjHb6bhk/anMWt5zrbe66LVwnvpvUzXB7njA8FjXwtCzwhgD0LUdv5k32CSgPE8Q40nc4YTI+41aKjGMsgswF44bTIIMJHuT+HRLFWbOph5ZkrEHeIcCMFfPB/ey5i70g2OBi7tmZFs6K7uopjthr26HBFk2FTu4IweaWnXtu3Gs98b+x1yt4rd6lN2qQWSDWS6EDALK4PwEnzuQMtm6h23QZWg4bhyYFvKGCtvLKoj6gf9PmMeeATuwreCoRWiN1PG3iXYlnJWyRob5eMv8spcvR5OD85OLblyixYCtUbJna1GOP7fLHPOXGYigaa9qya2vVY6+9tmvfAaSuj33sQ+fevmCzh2qb9lA4uriWS3ZGD99886c/868gQ//1v/wkSOrOO45SxLRGZfWeBK5AsVh8oH+EBDt/8Ad/gLoDSgy78N73vve1115FdZbqSDCbJDGemrraHu/CheTxRz+Uy+RPnzrT3p4aGxsjqy3JCgYGB3BoSq8vl8tFCDCVswFKgFCAz4o2TBoeYzh24i8FsIIVzcQqAF9TDWChdGMhkbuGko0kbzebFs71KaZFoQvhYm0cgA1rNTwtqHlIUiBSEghn4ahI7l94CAhwJlMDaTqwjZtnkc7ShfLQSB90cXp2KpmMoahaWSndfs9eHB8npmQdRG1+6PB+Mjfu3j3+7/79v710bvrYK68TdDR1ZUqYH9zgakNPgLqbwaFxBK/7HrgXDx00mX/+53/a3K51d6XQSw8NDfItSMwjgyMuu+f//Llf+KVf+cWXX7jQ2XFtbKxneXUlJv/y5OL8Et7bxULtwP5+Jgi0MDoyMjWxODddU5wUSbyXCrF4we3du//g6HZbrVDcIMJx7+5999x9e7VWuHjhNGnHSPiciCXm21YqtaanUo4mouk1/lvq7eto1klzS/TOdr1M+qrm+XQZJ/aRsfEtimBV8g0XOSCbpXyOSCMEg1tvObqrr52Q7lrMv+eWgwTQtmqumw4ffP3F5ezyTEfcE/Pb67lsqZrtiPlCLl92Yy0S9pA8uUkKR1KRONEBBrTccQ/M1Zx+myckrEDiHnR5YNp89jJJJwPtnlrmlT1Hh3/tt3/0l3/+98mVfdPBHorFQX3FWQmDNRxNPh5+dDMWpjlr8bI+WHsWxhEBkoZD5eeBLsmjgESlSb4BCDdoDlQNJcZQ3SJpKKwdya0q1VBWGrstiqCjQkQzQcIizDsLc1ONWon61j2dHbG4QGVqagqLb1dnN/ML8rEMH6UidLcGUJWqZSg/nAb+pMrdDRCadE80DeiSkxjRCLpBO2yoRqhIwD3MLFYJbiCsCub1+PHXwZEsGWanWi7R7WAosLy82PYf9o0D2XwPD8sEzMoxG/Zw+Euc9TGm0DhyN40yIqBjlhg38wito9bnEYCS0YEJttrhjAZPsjP5sauWdGadtFYXF63W+MnGJT1pNmkvjcRsXSfz40AAAQAASURBVGKvppCoLE34dbzMI9YGK80qMXpjPWHdzMTxEdACmuQ2qynmQkgVUq1znGQq9alGNyr+g2PjzgU+NeK8nMLscB9EVyBiItQqOTMSDYPhoXgWJSCabqdvbGjXoYNH+nr6AYhipuzYdIcJG3fs1IBkLlD6kBmgUMj7qc2wVS9WiowWKc0cHsr9FqcuXhAXDZwJk8uCqJ/SxZu4UWF8iel0VRwAiAwWRffqW+gzfdW3GmIhJ36crZQ/GmUIQ2pDeGr6CIBKFysZrLkDY8mRg322DgoEZG1buW1PzebCyEb4CAV90ZUyCNDyBnyDRmxnGcg2wCGqRLrBOOAUAcVF3Uy1BXIkEPBhumdRU42s1U+6CPoyc6szN2bBYqhoXt9rNuuAwX/3z3ffYM5bJNDs+XK04SL2oHGrEZ6VPQNI4LzVjvZmxKyfGsh3Notm79AJ3ShCq83qBfNgOiwybM7uXDUaB408UyCo00Qwd7qK+8bCUqU95WPe0BnCIFdqLZc3XK754h19lAkqVMo9Pd3TU1fBEhSywumEdELZDXJgYPvvrrU8MwvpdGUz09jqHNo1t7hGNv1ybatcbeDagSHiPY8+UMpllpfmKrXyBsEym7Zkb9Ll9Fby5eGhPdRTnppZnVnMffCj3/Psiy+O79k9PNL3wovPII7MTF/ubI8yMDMTK5EI5lUvyehjsYTbhZe1FHSp9tS1a5Pf+734+l5cWJwBQ2GzkL8LWRqrzZA7NTu1HA5FQSIsc1x/GeHnn30Gz+F4PEZ/8oU0WTpQkKBwZimVq+BUm496q14fMFM3KVkgV3jPskAAGUYYfI36maUIa1vIV0BJjDa4BCaGpiB14DvLxZqB1UyYJcwxD6OnA7JlJcO/i7vhL6SWpuiAB9yHtQ/EQpJWYWfEsHoNhR1BVJRv2txuUtOOGozwsh4E4kgwT1wsEX1ez+DQ0PievZDwbz31nKMVqFVaeDtTZ/CP//SPeCnHP/nZn/rhH/5hPry3v4fOkMgXRuS+++77oX/3g94gHkNbI4NIoqxryrzX8TwjtfvQ6ND+g/vA1//tv/830nHE2mNAmy/icgRa7V3BgZEOp6uZL64p85bN6fdFMumyyxE8e/4y/Q+G3CTeGhzuypWzK5kNCN7K0jLnD+zdx74IJsHAznfjXeLzkLUCWXx9dYVBBgmDqDPpbDzSVcpBJEnv4G/U2h5/9P3RQOR//J+/8JM//h+mpq/93m9/6dHHRpTev1nZ09MZ8vj6u0fqla2gN1LJFQ6MDj3xxc8Ftovj/ZHuuGeztN6qETLLGyn+VsU4C08M1cOGACpy+p3UenKH+DYkIeqAKDh450+YC2myLZU8WsiWndtd/vi9r3/99d/59a+FvKGIP7K6uBjxuyqFRlcqhAlybiI/PJRg6nweLXMIGJPvBYoQNOtNMlYqAYcohWL5mAuYNtxbyOkGhgIhcJUC02TzCAVjWN9xbiAD9rFjb1AYmHLseLTRJukvIrHUHbffs+/AQQAMLH3izVPwVYjMMKb4SRFiDhgAirwIzbOQCYrIzWYkFIZja48nsPbiXc8eE20oEmbPMRieEelIpo7cevTy1Sk0KHB+wDAVSqDo87OzWHCCYfpQ6O3rzxVKhN3BQMbak7xCRhUQEEgEJEsXhYygm1uSaSyciOqfjUhq1D7cAEcAYeYeAT2hUr4AfWUJyTxuNjCTaQ9szQyQWwegVHw9Lhv6HobKbMpjYqgjj/NOiAgEA+rPD/rBOBsyjFwLEWTEtygmyAV0o0J+EiC0jHkE5S8r2bzZ6v8O/yT6Y2yyatwiV3qeV3Az9Iod7XAPlE4P4lrC/XSGZS0pDldhvbGtUa0HQhHKIcD04WaFO1wpX5mYnB/uG96zZ2zv+P6Ojh400rl0BdsnHSMfXpllj7rM4YjEIswxxqdSvsDnhF1uhXG7qVugZEF1QgMWFtwUQDFDBypXP0DoBrnzmfQXyQnOTiOKyGUCrdHHIZai9+ED+QfiLHujDaeSsps8HX4oZY1sPrBNGKgdPsdyeSU+EN7fOxZudwTat22xis1XpNxzvY2CCiTwl0sdAXmaeMRtI0ozbXofG4STIYJF3UYNTtO0reohaBeURZJ6s61NTNTcqNky1NCgSj0KMWQvfQbnr5/VgQVj79rr7usb328d3jj45z8N9eWkaZPWGBzrERYtcHu9oXf9azWl7l3vxo2L1y/pX/5n+vhmMWqiu7rL8EVi5qwzmhKuMN7w7exlndEH8Ue1K9KZkMm/hj4Acoyfe9hNaGQolJhcSIfiCWckdmFmpT3ZhztzC9NCLi91gZMya15cW65NL527Wqd0277b97a8HcvljC/Wvz49257sKWSYrM3J2TmyF0FmqrUS1QIDDhf1OOG8/LHUyvLa6kbZHUg8+uijBBSF47FsgWiUCnFE4DIclcA+e8d3jQ2PvH36dKFYOXz4Jo/be/Hi1XisHe8CevHZz/7Uz//8z0ejkWQqRmFy/L2wulaq2yVIWtsGarcE9MNmA4OAXEBnfQND+G1B/jCtoLvDQw9XP0YXYQB+QrngDHCijHW4xdqx/ClDpPGU3ZdMHqgEqOwEs23YJmD9XyicwTnMh4WCrIljz4Zm0nBhEozYmAYyxMMWVqQLr0l5YgzDFqpBFw2CxBbI+kagwPEC2RuyY68T66UFC/TAUxbzpYvnLywuLpfTts4uRA2hPIgZNPgrX/nKm2+++dLzL/AJZO4lKzS8FC8Cp6EJwGY3OjQ6tzh3/u2JoTGeDMyvb1ADGP+wpfklrH3/7od/+Bd/4Rd/9/d/F/5jZGysTBrD/OreAwO4s+FUBXahrhEvc7sCiDI9QwP9xW7Qxcjo4IGDuwul9Wxmyed1Xb56iTrsROEODezCAhBMJtZXVtfy6fFdw/Q8X8g3mg63L7ZKkY16ric1SN2j973vE09+/RtU2B0bHWnW7M88+/T+PUd+4//6w7/887+86eYD9zy4H3d3Klw186u7h3vi+JOCCxzO3q7+ZdvCKy+9CAaM+TyV9Eq+vtkZ9blCXjLsou0P4IVUxxlF6EKJB8BYqPyrzATRPxj2hZzBHmB5ToGqwU+pZM/q+uted8xOuHX2uTs+cO/o6O7/9tnfzhaKWzYPhfrG9w2V8plqsRpLeZXzjIyVWCgYGsLYyP4PPpQicwv7HThZXJdyxIv/hgxhOMRDTa80+YUx8SojGrUAnI6VpUXcoEaHB4BFHALS6TUYOwaThBr1Sj63sUyfK5vbCMpogIDwaCxBgBkgxzFRTOxhMbOFLFiDcHuCOcCo2Vweqd0X99OS0CDxHSjhCZeS7ge3ycbK+hoiK8ATj0T5BiRS7iR3B3lkOJ8mhxbuNE2yGLkj8Rg5vCgbjChs4S+wiHTK/ACrAvdgWL5TSTbgMqG0Jt8HumyUP1BfU1iO8cBBUdoAyAwQaQYFf2SpkdnMLJClGTFYFlaWnvJNQDbRJtjtdMiIcKKdfLY0eCbiE4uRmV86wlKBKqsl/lh4lkZKqJCpJxxE1FPY32qBvXkrYuvO60HHYiUg7dB3IyNDzB0cm0f0veYPLMYBJnbTjvJPG4pOy4j6bfFghJzmtOhxheol5NdSf1f/Q7e/vyvVE/BRONW3VaU4ObFoBOdFYULxdmFN84mMPmFhsB6orQNBd+fhQ0rMUcyRwxQVBA7oOKEyJykKmZhcg4yB0W2C06EVADEIXj2FILOXVlmcJbIDSY9A/2AehUKqXKzMiA1XxFZuZqD1duwjHaTmslVqhY1GdvjmocRgJNIbtvmJ4s3YXBmbM9doy6lsHeAthTztUTlKXhGQe3Ymp4cZWNCjuBO5+0JgkImIAcC/FJULugJ6h4OVptkYxjXC+mXAaUfkZUIMb3RDAta7jKqcbzSbPvb/7wFNmHtomWe5WQdsAgvzr5o0jYCuzfkdiOL4+vmdR8xPC2CsTuo830fPDSDouoEuxsAKJjd0VyNw44/rZoIMYIGAmHdeCznB08rtjWYKaIJxaI0WW20Ir2uIL81GR+/wa6fO7T20x9c+uFJch/IsTa/2xQIqO9JAO+DKVxoL6XqGMn8e23s/9pmr8ys//z3//s/+8i/2RDpQBVM1hkry8ysLsegw/DLeKKh7qbtJxjUIAes8mycfR85V2eqmvnwhffc9R7HDk1wlm1987ZXncAOhujAatpsOHmKijr1+4vSZ85QKqNQa48kOePPe3r5oPEZeKYJN15apSNjo6k4NjQwT6X7x7Loqz5YqeYrMs3YITWyzDdrbevsHiPABN2DZZV1XSgUkTm/QSxZSxpTYI2gK2IBBAXNIZYKqUQuZ5YrYhc1LjI1WJihHWkhNB3tEOlCNJRCDT6zzusta2moA7Yxxp5Y2X5Z2JeJgSnG+Zmb411QthFiS+AUshNcK+IQWIJmgSJCGJCXN+hZOlMQvYeGAyW1U6huU1szmwpAJhyMaT1y9ehlnK4QwhF2YzX/6p3/q6exaT6+DAEHNy0sL5E7aSKdpCVbmB77/B576xpMjI0OPPvLIL/zCz+OZ/cLzr339G//0zW89de3aNRYJHFIiGQfz3X//w3//lb//xhPP9wzE9x0YzOTz5POBQfB5az09XTOzE5Fo5Ic+8QO7QsOXCmffOPFCOOYtFNYIiiDuamxkHzEMtx68nbDEGedMJJQuknq0zUk80eJidmsrwCuW5zfqtSjpvl95+eyHP/IJVCBvvPkaAUuLC2m3NzQ4uGtobE++1BwaPTAw2u0P2t31rvT05fLm6uiuI5TNqlQYavvy8monZV5bZbyZEkF8yvPQA+k3HU6iwOWXaZhrLUUIMHFEZeimkEGbT/QChhjXVVaMsCpZVgqrmGIiEVZlcb3wps/mTe6547P/x6f/x8/+GYlCW7XSxMxi0Icqu83r82bXc7EwJh4QiWqeC5HzOpEOO9Z6KT6Q6eVURe4KyBTzTyFNvNkprUuwiEeLs9XE173WpMZmjZRpXV0deKEXihljCG16qVZr38yur0yTt8HjX15dI62BH4O8kxTNvdMzc3i8M+m0Kwh2O5l9PhMoIs0WTg9wn4QUR+NJPNTQclGxA+BFM8Dyh5MqlCszM3NUf4crxb2A+wl4g41LdXYD1ahn0IEXSsRPBalCAYiSqI6kWk6yfkgHK6olGUErQWtCWfDFmqFWwknOhNIL+tEGUCMUhC+cxGwhW2qI2BCgCTUANbNDicwYWfcD+hyiK3OZ6QFZKT/ONhVufdhE0QZp0OQ2awr1ibCyFIX4hBtFg/UG9ixDSDNQwEqjf1xl9SBIwoarz2YNs8YZLr3XeoBPgPZyDUJiGuUx0u5YHQNnGhUljwhl4/3NK/gTQoBQWyp5BwU9HD5bqJAtQi3HBsfufuTenlQv+udkoqMJeS2KnKBRYcJ4Ci8VFjwESpYQJSxtoLEJBQPhRNRGPZGVpYWZSdIJEZiBtp9ZSUZDzhLOr/pc2Dg+ne/SHqU3dFufA+ciXafGmNPgGZgteBUwLLOm5Eskod9sc7WKrUJbGLfIerGRydbK4Xbf6MGB0Mg+W5jkDgRNbjRbmU1nzmmvbjkKre0Sad3VuF5FKjmFaDOhUGDG2tjEeYtyrGgSTHxqMZ1ncsScgbbE/YihYomgm1CvNYj80GbGnw8ypNc69a69NVPv3lsXrTP/7Pg7T0In9QrrpMbDTLo5ybiYQYRRMVB644Xmnut36qz6euNB0x5fo5MsUQG1/gQY3KNjDndOmhPWyZ29VPWwMOicsdFXGlvt0eQWaMIdLNQdZy5OrK031rO4aWzcdDQ5n24FM81YeyBfd3QkOvvGfNXVNaQ59Hm4QBfR+Ppj8Z6tLW/i537xN9/3XR8/f3n2oUfev5ZZfeobXx7ftze9Mo2TrNPjBK1UKkp7TsIgVTAiRdFWORiJ4ye1tLrx1FNPXJq8dse9d3Z2d/QPdt95xy1vHHsOfQVhu5cuXZqdmIFgj43vo3wpajfCbNBmwQWePn2aoBfjnQDZIgll+OCBm+68845rE1cunZ+MR9q9/RhKyddYmZtfunDxEkQM7SupKpDVWPKw0YAvSjzkP7IiREByhAbhF1utQOxIYInuDKYNngHKCVUlmER+esCV8dwAQzFl4FP2IA025gIMaBFm9tYBV1liXGXsadyi31wilxIIyKLuXBWJF0MtBSCCtsNNZmCptqC+NMt1qdmwwzfE8yo/PlryCgorZ61YI5civEhHqjcRT33605/+5je/+cYbbwAMFFrHGxbP5ACykt9P7WBw9BCOyMkkgT3FQuH4GydIkcHbP/CBD/zyL/8iaHp8bOQv/+LzEOzTZ09jFarXqM7UeOabL1HW9647H1hJzy+sXiGuZssWb7SKAwP9dJZ4VpyxP/SR7w65Qj//R7+Qza1FMcA6MJmXezuj9m1/V7L98vnznjbqkW2uLRFjk6jVnOVa2e9vt9ujpXI5Fu/fvWdPOYMDds9WK3fixPmDh24homxs3yippLt6E96wcy27MjAwlEiNHji87+lvfc1vq3qKzXsOHVpdz73y6tfuvefhO++8nyjewvJkBLNjiIynGWKoUNiFQjhpO3KlmssntMFah+0BXxgNpMWDtlwKoIVWQHCZAzxgRFCQvqJ+V7W4itdKsitRKZzG2DB69yO/9ls//b2f+JW+Toe8m9tsvR2J9EbaHwWtMNW0zkTqiOlGuLWYMBRwhF+yR/DTgpXWBFCB8ODeIPgw+ByfFVI7lph9FK8gcypGIBPH4rJr4NMSCftZt+jS0WYuLy45vSTNhmDWiA4iLxWB2vPzK2O7+olfymYzVMIiQzhvgDP0+fjMLA2SI8VIj4pc5jyx76Ygt/A1xh1xetRHQQFFQqVcjoLHgplVZW0iSWJrs4R6vJYv4sy4sraOrlShh/pGhtLQLT6M90DvcNlHM4KOhdERTwPQA+84BKF1xMphzDZaDEqvztQ4WwQbKBaF9aWSF2rHbLweTplM6RAkUDlSOwMEg2rOyIojNTAOt4y34iRp0dBF0U2hVPYiEhBGnF8UxGAtPWFJ5hlDlU8GAIP0WWASZK3PUEOsRq1IzaLRIhriDi5gZs0j+mJItOabegINpD9yr0BA5T/DGXmFM0sObzFT7Iz13fbw7WND4xJIK9sRTzy3TH0C+Fdy81ElhrhssrViJm1SrYyMLAxV0OVHcIGsyi5ZLCxdOl/D8bFeC6ismw/+gUTMZBAGWDX+Git9PVTvOiHTKWuTCZbv103bpHaE75BgiWOyHdmmsenC9FKvkRLAViAOOTEY7BkZDXXHqOhgs63ZHPgn8JqKzV1zeOt2Dz72dIh3MS7Iv3BOrBjKNFHzQOOCkM0BMA8ShDFCzJYenJlpMCisCgBBOIw2WBhMDHOtb1SHRN20ZqxOf+ceUOELzKrhU3Y2c5Kp3HniXx6Y+wQfzKD1jLmHY5C+Oc1vnTI7c5t+mc0c8OzOT/P2nXcJ++vmHTUmbQvkTRvAGFNvjlkEghzTunUDj0j3pZwxPGMEXzgWxDwEMRxPihVPoQaP4p9ZTL9yvEFdBH8I56PQqyev/tTP/UzLUb8ydd4RtV2emXA0yj4qudVVbRqVyVphO13ZTlft6dVVX6hjcmL+yuRMLBXdvW+0b7CXwNC27VzCT+ROK1PIlnBjb22VS/D4fBOZ5XG6wcE23tPjsW1szM1ca21XcOoaGu17+KH7qPObWS25o1E8NaGgyfbOjY3CRjrv9fhazezTz3w7Govcd/dda+tLRACzyChbhb2NVBszU3PFHBUG4xixcCNTqgS7c3R0uFqTrxyZJdbJH1GrIHBECUsKSeEGWqBD2EHZgGp4cZYWsgUWQ4LgTXj+zuoFaWp5imVzIFAyF+AEFHqAmdR68oJpmdbMvDJPMhtrHlmTwBsPQ1A1R8J1ggzAWHVKuYnJgQjIv5qVK+HBAfUTSKD8NuCnlQrHT9FAKLRrm6iAYp0anvD/rIhWtfHyCy8dOXLrj//Yf/zyP/7D/v374okoBPj48eO8iG9cXJqnA0StUNyQPmNvIsPRiRMnyDtN7otrl6/ce9fd+JOjePz6179OP1AQgHk7OvyYnOnozOR038jwv/k3P/TLv/Zf4XvG93YyiyOjA35fcGZ6+fChW3nRs99+joI5H3j/J1479sKVy1MjoxG0ZeMjw5st6i95z759AlmtXsu/efLKfQ/cn57IpjcK+/be8fLLb05P1T/64Q+1R1O5zNre0b7f+qP/gWzf09uP4t3l38pVl37z939xeGD3zFJzaXGlrhwY/kTXYEf/7oVisb19MJG3PfnEN6+evXD3kZurHfGJi29R8yER7EwlkvYqlQrW4PV82HdButIRg1hxohO+krRLHt0KSKm16Sa9l1Lx4/3P2xl5oqEouCeCQOxyM+3xRNs8y43ss12HbvvHb/z0B9/z6+NDNp/PXWnWw4kwZYFhl3DTw9TAdINMtMjAGqhVfC4AA+oj9awJFrfwOxlDSZWJQVeaXSdJIfE9CMBaKYtkqQAgITozU/hncT9RtcQzsexd9niQ6hbS/25Z0ILDFNqaSDwxv7RiTLPGx77RCEbCVG6AWIIN0VviekAUAKxnKBThgxF2aU1hS+gH0CA5XOvpDB12FjDzESPXIDEqIh/WDWKlgJZCoeiXXZg8O/VIjCylybafGR8Wl68FAa2SiphjiB4HgmGTggtST6N8P52gx6wZyKp1hsdQKLNnCFg8SNac14MqEySyh5nYjJjIt7VZP7mZhQqk8iLOM0Y8CGlUaAaTLLkLsgOREvrXaqMTlixs9kyJELPhakVJWJn8Lwuxps0QAdYLiw5FtU4IExuSjurePKezegLRVzSYaC382on3IQ2FWBs5QjGh266eRP+hvTcN9w+rKkK16XP4UTDjzR70hySZ0kPTO1TDfLUi093oIhkBiDdAWStk1kvFAikZ5VpFsOe2ihRBQFtwbehymw0sTvSHT7Z6xV4cPl9h8TCMhyiz9mZDqV2SvcVFZW6SeZQrW8WGvdxyVuyhZu9Y+8i+rrZuyu5SlC4NIsaGUm2Qcs+uqCf1APsK+ZwZLlieBmTUxI+4nZbXhNwqGG1i/FSXlzqpcHbiOYUulVJf4yx1B+DGhBD2rCli9k3HtLekXuPhC03TBxj0yD8CKjNjGi6zoDQd5sF3Dsz9//yk8LQ2tW8IJwdWazorUBUjLkx8vdl3RpI7mWXdJ+hVy9ZLjVyrn0A+N0j03aG1hgwrpxiDxVmL+lpXeVQgsUUqSJ0X7gFRkOmiBQFmeDyxqzMrl6Y2bN5Idct7bWYVjiWa6M5m6wurG+/92HuLzVwo5h4b7eqK+XpjgUuvv15N5zIrudmF9Pw6hTK8LXcsV932uiNuf/BXfv1Xnn7uyfpm8c23Xms2sv3dMT9pOeqNmUuzlQJjbivkWBZSUbF2MAzjWkiJHmowkNKFegcop8fGhx57/KEzb58iTVIlV8fnxePyky01EOo4cuvtlKb/xpNPDA31Xb5ywesm6BZiShAJZg2atcUjIdI8cTA1NREJeEk9tLpeCYa8Dz/02MHDN01MzR07dgzODOVWIZ8Fj4nQ2NvCAU82lyOBvuIx5KTawF2KRUEZCfRvVZXzktqECcXKg7cj0idTAPpjATK4YABWE9PBHgGYFcRVNj6QM9xDU/D9zZqSdcAVskZ5CqYRSsBKRtjlZhRO7C2UCqiiWZOfBKeEolhXYKEmjtm0D9gSIYjMAMZHNOIWvhcNdrKre2JqGvfytbUVBCC40Lm5xZtu2t+eSj7yyCNPf/uZ5776Yteudu4Hj4LBACFILG44uVz2yE03k6yDlk+dfpPvy+QyvAnNZCAkO2Ug5MePsbLdeuwDD7W5S6vr1/oGIy5Pi+qOw8MjK0t5zU6w3d7m5Ysfvf+R5cziP/7jn8WTjY2N+Z7UwJ7dh1U5dgsaQ1KB7VdfPzY8Ml4obz3y2HclvMP/5Zd/jTQBn/2p/+PE6y9lNxY/8KGHTrz53PTcmXxxMZUKU3SIpBw9HX0/8smf/OaxF4cG9p48fv7qlekf/aF/nwz40guzCY8v6mjLLixMnD+bikY6k5HDN+2fmThz5dzrjexib6RtIO71bRZrVFIRjmPMGUDGyxKZhK+VBcFncwedbgoYUv6IJAYO1oc8m2UPi+AXQAk3vIVJ+9zRakSrpVSk48HiVP6jH/y/Du72F7IVilVjX/Hi1CzcoZdAfayNY6aQuVAGUgMkBmkjGkoyJVM2BhH4KolPRiKCTDKblE5n6gvFHMPlp3Kv04kJH8qOHaR/YCSSSJ45fxV9bqqzD6U99j881eElYa1OHj8zMBDH+Q77UaqrnwxGxHyvU3F9aQUhGuzh9XnW1zb8AR8SHKFN2IM5D1OLy21Vnk6btXIFfAlagsajx4YJwK8+Eo3BBKe6KHkH1W7btXsPnmW6S+iE79MHG60QeKpNPCk4BoFPzsx8uXCXNpaWAJkTQriSOHkNAyP4pe46v1H8mKXCzXKzYENWMhZiLQTkSo/oej2blQHGrB6MROhYdRVZmJeyIKGdKA4wcaIQlVCNIol7WHiIFIyk/D/EZWOUbJqk7eZxZpdNDLM+0JR+R7lqvdRo0XkvyEHkXF+sTbAEUrBRlgD0AC2Ev+ZW3Omw5+JAFv3ggx+kUFcJ+PD4I248JDHcbUX8UVhwBpqGvMZBDSkftAAioySnn8BwMrtk0quLS+zQQmJfJYEJeZRRR8mX3dYGYxj1BJ0Be75ekNBv+r/TJfMPn2z+FfbY6SuwDEw7KX2sfMpNTMyeht/vjkQ9rnB49K69NttGdTu9VV5yBOs4W5JbdXOrTIVGg+LQcxvXZeJM2UgG0kTzLEAHeUIgRXohLYSsNUtkfVNgSJWcvtBjXi7457qdT2SZQb/lHq7T9A32w3TQCMEiaubuG6TOnOHVZm4ZLYtSmrPv3LPzUw9fP3njwLzkxnm9kVasvTngGGCxXqqrzIn1iFm0evA6CRcfIe7mnU1wznWd3jl/HcoZfX0XhEGyL59tNl6F6M9YESUMCcE9Ei2EkjyBfWD53z67cXUemS4fSno6evdUqA3vCWUXpkPx3ie/9WrHQPzgkd3Fmq3b7S9UN4fH9y1cnSwXt73+lsNbDgSTsb5xty+aX8+TRfazP/Fjo3uHL14709kVGTt44PSbr4VhoSgIgx0f53ZCjbUQwkxQJByCl9rI5OvpvAMrD/Hizs2+oe5jr79aqqz3dHWjoSAnBqSiQbrULeeP/uiPnTp9rqcnCrd64cIFfHftYR8eQFSagalOJEJ+H6GncQyJhUKJj1UWAmrneqE3tZdfeWluYRGChdMDNYOBo3yOdKsFD/CNLrKFF65P2VVRzZlk2OATeDi0r6x+5HXAx0nxHTJvGgJsOCJ8RHBLFkLQVJmNY7AEGIODG5PIFbNUuaJFzpTcUG1Y93jJ8ibekDUI1ZP8BBIBK3tIUmzYMHOb5huOnHZBdeV8De0NEsJm3ei70BC2tvDEobIc4bOItiSV3L17VyoVA4ciYKDOBVN0DBHyRDBJW726PTo6oiVhs+GHnEjE4UtmZjuxHZbKBexQMP+ATTjs7+rpXAB5r+Sw9gRToWeeefrn/sePN7eGmlvrJO6ZnLl28eKFRLyHbNJU0ykVm/NzqyvpfMDfnkz0h/wbgW5K99bCId9KYQ3OYHJimkHu76Pqw6Qv0Bn3tgdsie1WqD01vraKIkDlNIA8zCKdXUNnL77VP5i8dOXUwHD37l19f/2tP4iEumYWrvmCgU9+z7/ZcsY//7Wn/fbt7374fvJMvvDCS/ZaeW5iMl8qvnzq3AMP3vXIJ/99evbc7KkXZlZmOuw1H8tfgUCoWxVqgeZCrDUTAkwobYGNmh+N7ZLP5vSQIsvDqJK+DfSyWVksg+08yU5igvJF8u5RS4as4t8KpQ4+88LPf/p7fgHwwbyLkOgipwelwkA74vf1h4gEY0S8LdiDN0tc0sZS1bLEwQotC17KzCozb4LcgHNQThsmWNAyCxmJB10ugIe9FvyM4AIQ4c5Ccsh6I2dBFAsEsoJuA6fF2elJJEycp5BBALSRweF4LNmV6xoaHKUd0pGyp35GR0eSxClkCMfAqlWJm/8W1bq8eN1DokgYAoFIr68htCfbU709fUi8iMi4X1GnGQx8y61HM9mstJmyZQGWRr1jDHuwizbsHAJ20iehWlXIh0FybXzJjhMaYCdKSHExoxkQnTWISiyuWULiW2F7GArJF9zOPDGE0CmxtDgo0hS8qlaSHCTA9bB3IFBCCQm1RYA0Kw3qJCTIRS6JGsFZMCFQYJTBPAuRRw8u1zhx4Sw6EUXOY1mmTdOs+gNHTd5Y47apbCuQTx5meqE9SLRwsdipyTKIcz2sNpq3vr6BkeGxno7+1aWsF/8AymlVYdFrOEhSbx7azyaREI4B0KgrAgHjWVskZNss2Cq5MmtleQX7kBeUGAqix6+VSMxNhISnaa/Cj8Bc4JzZ2Kw6/FB9a3SEHdhQhbGHhUBOFmibl4hIwws5qps+rFU4SxAOtR1o93YPp8Ljnba+sG3hHEmbfcRfYDciI8x2DSTGMqQwNQiJOUFaAvhkQ24pXYLKM6BrRtBHaEAeQS4gQhA3QpzVmUtQF58L7CKv48jCThDMJEhMkQJQ/VFvGXMBz/WNF+lj3hF5RcDgphhyEb/rvIR+mu3GwfWfNKp2zXnzghuNm2Ex5/UsahH2Ast3bfw01NQ6z1RyYgeJ60FRWesG0x+1wgl6pf4LaLSZe2TvMKh1RzIGpHTecCpEZivjGaYhIvIauKduUa/GMTs1t7Bic+O8gqms2IoFPETme1Bgtcdnluf6R/q6B7quTU33D3ZstwWAuNm1lekl+eFVbE7Un81K2Y89LxC59747Xn/j5VjKj6tIIbsRDbt2DY2/+u3ncDbATgDQ4j2ApzHSL6XYSuV8M1dHNvWRRs1BXnxYqs1MoTXo2L7jziMwhKhbcAP0uv2UEST14cGD+3/vt3/npVde+8//x2e7OjrmFzCPUeYIzyn7UD9kJopPNlBZKTeJBnE4fbFYcnGhQO40rMV+Ehhlc+sbb8ai8UA4UshlgwGijdAIOwFsTF7oruGtBUpNsuwLcjBWQYnhnVmhHjRHDLIADdAyuBT0IQwif2MQDS7MmKjF07LKgW+x/IqmY9j5as2VcAczYKBIorIWCrNrSCq5EEnWpnT1umwMxuhsXDZPqVYVjBqyDeox08qs2xLxRKm4asRfnlXYApkPAAS/P5DLZAk1QWonVBR/46O33nr8zeOo/RCe3/Oe91Bc/Xd/80+GDgygtCTtEbEoKCTvuOu2+bm5++7/4GMPP/Sf//N/7uru4DORHfQd9q2FxblSvRWNubaEYMq33HpoZXU2kfTtGd+9uDT1kQ9917e//W2mw+txLszNuJz+gMe+sbbkJfLE6UBTBnXpbifSdx2nWVzQcWA++dabhFhBsdfTjade/votR97rD6H/bHvr7ZOf+PDDLxx74mvffKLNVkl2+u9/6OE3T72EdTMYiW7kssO7B6Ymlvft3t3REYMpeuv0qYNHbn7tuWeffeGVm/s7SUKA3HjqzcsPv+eWxnb1zJlTJKZOBai114fruLuy6EIC0hLjz+BPMzvMKViEmAsQBupFkDRUjwEkszFuO41ilUVBBCZ6h9ramjOIWwBIJOvyNxzb7lrxijfs+fXf+Y//85c+16wgAvERBDtJ3lNcJCIWIpfhdkFPwICosVY1AW/SURn8TiU6qfRwg4Jmweq5UPW6/XCBGMuVq8huo6ZNJBrGkZvlgAUC9ylvLm0vgEWdeEgwrchFNAssZTfW89k0NJjZP3P2tC9Azms/2mPIPPT4yC2HCVV6/vnn4dLuuP3Wvr5+YgF40BhebUTFE6pEJUT8A2gBGADDn3zzBNwAE0c1VtROsC8IaSyfmIvywd7VlTK2Kz5R1j+IGGyhRgClHHDatoUvmbplmFOkIQus+SmejwFAy0RqSnzsDN1lxAAYrTGgW0wHvmOsTjJVV0HoML3wxPxmZCGStEoyTAYTjIGiE2YElzakMZwkYO/VDRYYSl0sxHJGQp+Pk7yigzQHoHiINeoNDFZtOHgHeIq+MQeq6qQO4JixSe5x3AvJxIjETPYSCCRwgxaZWlE04lAoN35TWqIkgsSAtbqwAQYCTQ71j/R0DwR8AWJtVibWEpEUrxPF0UsYFhJWNLTO2/BQEMZjrPGywtTTqOar2ZnM2hUyLYCvuRVdMPgYiRVY4uNwWanLTZTcQEaSIq2GC/e5MhIqVnJkdZknlPME9oJvhLXgOYgoPGa91ipu4kDmriw0pzc99b7Brr0Hdnk6/FutfK1t2Ztv24xUuA01JW9H8ID5toYLLmmTZJEwKkS/YEFA7cz4w3Z4/QgIBNOhV4dvJL4ItTMBNNtlHEyFOegSPZFLhfAmqu8qw48nAf1jsIUPmWjs24IGawMVGl5Jv/R2Q9JAkBos7hX2ZFBoz7AYghNDbEWhNWvWT61t04B2bIAi486BtRdu3rlZByKSYFzrCUv9aGe89DYNMUvXao3HrebB2Vsafb2PORL2Fi+N3kU/uUutmhHiuu6R6E/PAXJxmLj0qxQeq2a7VEGZiQonwKDjz/z2hTyTTJle/EEgzY30arKrNxR3FybXqalca5HXvRGlhh9Sqit49srlRmX7wlIOAyQCIq4BcLXN8oorETzz9qvhmLtSznV2xW7ac5BAz1eeOVFAse2GI3UkopRxbZGrgPo2Ia/Ng6NBMCCfTywCoC6fVl0qYJudnye7ZFeqY31p3b7pJJoI3XIq7gW/1DYz+w+M33rLwdde/na1UErEYxvFCgE2nanuUCgcCSeOn3irUq3v2Xfb0MjIP3zpCx5vBBDF7MUIhIIhIL9UylNYF5AoZVmb4mRRrRBpywLECUtkGMbaaLmYlzoiCWBMOhdj+BU/qXwaCMM8RCKFFk5b+DEwyKS/NrY0JHvAxe4nqRC8NRFRICb52sDooihHYWtUzca0dIPvY6ZkClG8PuK+3yxUwB3UJn9FfDlokBaYQfqGcI92CwxL7n5QNrgFTMWs12so4SEZRIuWOzrj0NcwZYLSG7iwDfb2kath+vLyRz70wa7ODnB6pVTB/8NOHcYGNXcKdm/f0TuPfPC97+VbBwYG5uZnoJTxZIwYkbXsGjqmALELJPPH/RXodaTfOvXy+O6h+2+9mSzBtXx9qHsw4HEvz00QJoNKf3riXDhUp+CU21np6eolrCsW6WQKgqGOAe+BcvRKoXIyEI1PLy3F27suTp6eXU2TYGctf9YdHjo9G1guzVJnvFBYr27Y9u4d9167trxR9QU8oVD7xctTu3ftLteyVy+eb9tM3HvnQ72p1GtPr05dWsyefctfzHqa6Y88PhaJVgr10p7+lMfZyi0sNsuFCCSi5cOeQNodOCIhAK1mLSRWJ8uoUaHSmT6TK83sdq5QCUa8vnbSIGAHIPVIDS1KG9lBiiQRtAVwdd6GcKxhFAMxJvYe+tGf+NA//d2L595cGhts31guEnWMzj+XzmDnECav5BGRiXUV5QUYROYhVAGofGM7X8OLzth/mXdADctQsVAxfICJ3/FTirhBpaZELE5PgFjSom1kNmCgibgG/iHta8sLwXBkeKBndmE+EvSPDPZPkko7R5Au76otTF+E+iKhN2oZW9DV1x2bm748NtIHUunp7EQ78sarJ/CxCAeipFvqbldcNeBPFulV4gDJoF0qu5w4ojlWV2bhcdt7qf9YXJsvfPOrWfhap5u0XoaCQhRBPuIoIU3G4oKZFjEILAQ9IC5OmImMnxp04SO+lkvwIByzYc3lBg6sMwA7NyBDkaGD86T/gWBrM++CDEjFJJdb6KwdzRQHXES6ZQbBeLQKYw6/hZ80N6GkMj4nvFRLV3KtMDK0hFXH7PMwDXBVhEEvwV8tk4YTYhlYbAGe4vyHu5tclEG6W5AXv2fb29Zy1UqqZ3Hv0YdS0c5UezeqZixHTfn+eQORCFgARTroBjLP6sX8Br8D1oTch4MBlQtEMCUZ6EYOr/OtVpkBglwjACBkGbmdnrKpb/CLprtyY0DroGP7doxMK9Qzq+SJMA6GonApxVJhLZNLdERhKDdthEBV8dSp2ymDs9UWbR46NFR3Fyja4O6u2UKUuM/b2srbqr2LFwSfJqFZrIiU6TABxkTNAPNymCvNHIMGh9Rs5bL4WIH02FPUExUhrnEQ4La6CzjA2xy8Jx8SrTY+Rzo6SBWjbD7HQn2iYdDgd87wfdp0D5cAD0P+eKveaVE063nrgm7THe+0uXNesqYaYjOkklV1nTDvPMJD1h07911vkKGlPxaUcY6r1xsyfWDIBYDqG11iTwe4A7ELtYgIrZ41EMWSFq/I0ywI3i5jjMi58soxK7XNClUTpOJxQIBrKsthk98nKpxtrlYZT5hQipbsGh84d+kMmcH6ejqwGn79q08MDQzuHh+/dO3KRmkTFz8bmiCvqFQ2sw7H4wsn4ArxuCFdEXNAXlJImmMTX38fjLzXHfZ77XOzs9BdwgpBJz5nAJtPCDu/h1jYYq5YgBKFIr6JiYl8Jh8gTUIw4m55igRIScOx3eYLF/PZn/0v/3VtZYlASVLm7h7bnc8XXnzuzQ998NGPffS7X3jxNeJ0z50739M7BClC1Y22jeUM6JSK1EC1kbWYQHCJvjjDA9XSPxuLlTBxG3kBAQruZ4GjWTPToVlmceuk0mDREugTmYl8rqxgWE1RXubH5YUT1SkWuBg4Vp1Aj1alVmMeYNlpUPOjjX+EhTiC2+R1lE7F7sig3djAVyBiAw+aXGYcAsl4UlbO+sklC6IFK2y02Gr29KYKWQa/bXFxI97uf/7Z5/oHB9A2t7VN/vEf/zHMCpNBobpUJy5UNYff3j/STxQ+eWgzuXSSuuypBKFfOGPCh1FHEnOp24MDE7QdLso2MGjv6QuXqO1Qzs0sT+NKEvSGSGoLk37k8CEyoizOXbv/3tv4BNQjR28+vN0qwqA7nEHSCrRteycqZMxbJewhmy8nUh1nz18gaYQ330ynG+NjNwfD26++8Uy1mU8lobcOZOilleXe3lHipkqltmqtrVrBIWhx19gBUruTB+atU2/80+SVVCAxkuo58+0XD3Qkwm321fnLfV24vOTaKgtubzLmrGOcDeK7Qq3lYtEJjdWSZ0VIBNKyFB7WQDI1SusPReUkQgvR10SB+/AEQMgEM9WcPlfYD+KDoG+5hLKKWMcblcmtfKP/lqP/yvveP619aeryRmcCQA/OLy6EvH5KAS4vZ9sTeLNDiRT4JA2nfH9JN4kOyIWcKF8MnO2INYcRFepg2RLTyo2kI22BtPHM8jhdWGplKjWBC0iI0C2cGFpbeazaQAXTnU2TAljeOVBCRDhkk0a9Ao3c3CynN+rra4vZzGrfwABTMz42cObUyeHRXXg1AnjtiRjeDuBPBGI8qxG2KHI4NzO9tr7MAXjk2tWLKE6CATQa2+tryxThoHoDKTT4E49gkUxGjn4ziMA0yJZ+80lI6Pg0ENjIT3NVSiRrAswcQEK0DoBimBHz2SI+GI0ZKTa+2KrMxSXNDONkQrj4ySaczrpEL27U2twA7kDM4l6LrEMcjM+j3oj6nquGN9Dr2ABLHiebMtlKLIxtoW1AAzENrKFmAYEGH6V3o4PFVaatRbxEEANFtVAXm+aJ7OrfO9A9vHtkL8uXegVgWvgrgRUIhNIFjRqvZpnA36EbwT7KZead8kRYdRVsUSJTQpqEfBAfzD+bW0FWsYZSAhmUQy7FUsFSLoZPhqChaBclZuSE+wu5PE400ahy9+RK6yAjj8/ZPuDLlhealCq3F2ttJRJzh1IB1EfB0YTNn9m2B8BRbW6qd+Hog6WgrnAYqijg8iDSq1ezJDTkROIRDgT9pSOQHYAOnyoEC0RfClYDd8jcMtEDbtLEyK+50QCrMnRKNgS/qcVkgBo+SK0wuppK6wt5kX5oM/+aWd75bTgtHuUJETp1y4yK1S3Ty507d57SWv7OM/olgGSkhIWFaq171CULcRpW490PatChDHrC/OnB661alwy64E3W25hp8whtsIFEwBwixpg/Raf5TvZS0lOVAgPjFoIvZUmqKFJkqMIerrpWZSRjOB6Iihslnphf1g4ilCvr3nvTHvK8Q5LRJUbDMTJTYIdC7YqFFexAeoN8Ll8vQYXBzk2vv9Kk7K8N+2I1u0EtW2+ddPs1Eq45KS0A2StttaHOAnVRqujc2fMQYGCPEjyHbzq8/+AePE8vXL2AUYpbM0TDbKRbvojPiS1Kn4ZZy+nyLa1nA0SbhsJkUkQpNzV5jcp3tLl//9CLL764nsniLRKLp5ZWVr/0pb/nRbwW7RidB7ZBbUwIKKlUkqMTy5w1iAKZ1vle0WMZcx2w3Twlmg3ba5Y8exhAs/Y1O0worfF1eMwUa+BpzQECPGYiBhCOj8dxagEvwfwCABBWppB360YmSBTfggcBzA3A4E6+FLbf6JAAYVC/wFdn2Rv2igMaF1BJnbVDv2mEYy5ByfPFhj/WOHzLwbfPnuntb6cAEMGfwWiGzg8MDRLXCxcVDiOhKZm7+HKXhyClof4hsjd8/RtPfPfHvzsej5448frufbtRG/EpoGB0fa1aC42sP2Lbd2C4PeHv6iRJT5Qk3AS0AFUkQVS+PIcjlezK50rhYIzJ6mjvrVQhebmw2++we9oT0XKlee3yRLFW2j227+r0NVxCurv6iVk99ebV7p7xO2+/89XXXitU1sJxT7NaAlkSE7NRrN1zx4OYfJHJysU1UlrOzU4jSwS8vquXJu1bgQ++7/uunrrWmRxtf7TDW9h4zz27F6Zf2Fh5e8/oUKOSW1qcW5/P2yu2Xd224e6QM+ylsoeRI8AMDJs2jRvYFYUoEKCFo0lh4eCG3LJj/rA5Q+RYJCdfC7bLjlGCCUHv27RF40gEVdTN/iB6geVEd/BH/tOnfvvXPufc9qQzG/GOeIngz5XKYE8KYIbest4ohitqtQmNRInHizB1b0KLAVEyO3HJ6STxPspwKBcEe7NGbIuzjeRYIAGSfoO+fZRcF7lR74GEOrhZeQOdhB/NLSIHUx0LyyD5G7Fp2Hz+EioocDxQDKEl9q9aw5LopwtYf/PZRDgQWl9bISl6c7MMp0vQ0frK2uLcQjDky2bXES2TCfSqgdm56fNnLhKLz7NGAIByOTEpkUZCVJOu3ABifZ6BS60ZQx3Zc4N1npuRjcwVTmvjkrXxFAfcwGYdsAeBYSCBjrMazWfrfuseoI1XWBvHnKQFNlgXljlnlCxZG/Oq9UMxSwRzpDLmlrtYPRLR0Gk7EbV5wmBkUQe6AZkjGWkIlqdagtvFRgWxRAVt26y2xb2Jco6izlXSaIwO7x8aGOtu7wt5I/62IK4lGMAAXPzRmUU+GUExEYsi8VbLBVAM6kviGjnjQAhIp4tZ3FizVRxpSGiHYxME2U6uZnk18ypWJ1lRoX7E0QNzrc0qH0/HpIOFBoiSSRUrpwDVradg1pbTh1a9VdnOpcG07lppO28PbvaOdgweGLR1R2yble3qcpu72uas4quPvyt4z4G8QR5AZFg5/RvxlyG7MQHb1FDXh8sdDrACLZoMVvxUrDyUhz62UPsTRoA3PbI8g4vGA+ENxZ2EG1CSJhZGiykSDQdnQfSErYBv7XiXNvPvjV+SQa17zBQaGqhumG3nZv7Zud8ivTstmZPv0FoGibtoxXqN3vTOW26c2zmpO8UQauM268brt4s1ZgbMRcELB5BXXLN2bobQCmCBLDqKrcioV+XPQbwuxi0FFRCzWK2AwdGsytrBkyhJS42tUgPOUkyQxsqsGlgXegAeOP7aG96AOwgEtrk6Ex1lb3VycvLMqXN4HQNRSL9gsgApEZweInzX17NefwMPdeRHqgsw9KSAsbh4QnRJywyUsQjAzqSEnJ9bICtTLBqoFEunT71J7aNkd4okGKRFBL0Qw1qmkoe9hY8GWpwg2pVwzIaCyRnKlkpIgcmOrs5kanBw8FvffDpNbaNcOd4em56ZCUejK1cvPfLwY9cmJ6Ph8OzMFeSAEnnTtrdJYAmgkveYtRuOJ+XkIvUYgKLFAojDSzNsDJk1+KxgFjLkkBvAj5zkJ/eLuTOoAxEEtAaKRDsjQLm+sWb5RrMADTqCATd6L3FIYrl0H7NmAQXN8vM6DtF8gm04ySPANo45LEVzsxAaG5d4KY/oBtOWBQDsaTGR8i7O5waHG4+955Gvfe0pf8gXDHuRGiHiVBuURytePyoNZlteXg7FVc2wb7CPOGDyb9xy+KaDhw/+zm/91vDIoHRaRDqRvh26gG+Oi4Sj7liHb/d4f7TdQ/q8SDi6sLBEGTQahGx0tMfX1tZxGRjf5UNedIWCwM/JN06z6sd372J1b1EblAKKKB0wbBcrIyO7Tr556sGHHunoGizl/mFyYqWQKXzkwx/65re+lOwIFMsbFITKZvKJWEdfZ39+o5TeWErFO0nBtm/3vqHU4PTl1Qfuudu2iS7Yf9uRhwrzOVxMKaS46Qinc9Vyub6xuO6z1wYSoV3tHc1i0Y/TWas4t2RrTwiwNQFaa6gaJTWxjjiUtowD2BghIGGjWslGwfMwgTPRANIlBKxZaOKcBVUWU7+5SbgseTXIiFVYOd/WioV33/+fPvvp3/mNz5Eqrd6igHRbZ09qPcP4INWAsSC9TJ1Yu+22CsVZ21SHEKUJOaKVIo1+AW10jZsAHq7AArHKWAFE8+LBAMAQca8sjm3KikpWIaRF6hrjH4T/8+rqssfnpkyCP+QPhYN2V4lQJVQimImBWdQJaKRxfVirrWKdHBkerVb4tVGk+DFZ5xaXq/EU801mG6LqUZzjNEOCcQx5iYR85pky2G7TeRycHNB3kAkuF0QgyLgLXLLnPuFNIWY5WMGNslbYuMRH8xlgFzgRsLI+BGg1mNF6kLXEAY9zktZYYzTCGRgfNcjUCLNBIcykmWs7y8IQDIw3ollmBFE0sUggpbxDw6uEdhpWlFCScSWPsLpY6fwRiAUx4z9a5CGuQuBkfSTdObosVEHoH3xuzR9h2WRvrpa2o55kaqBraGC4n8pkXozzrdJGtUX5520HhYxAEUwe3kisCpLFVEsZGC+CqsBfqOzofh0fR8VN5mkLDIvBmlUEWsSoVcFVD4GD4eRLpApGJQn1bWJ1VDFANnWfoaaz8HM48jMmdZuTDMyo7EnaWq3Zig0HblalYIfvwHhfbE+3LYyuZKNQnHD5nYEuV6W0jDIITTwBMduOJgwfX0YdLZQuNAvrwUhaSlNRFqR5KGaT+sIqZA0Dt6nYJ6iK0mqKmMLdILBgNMerkQaYPhfaeZzCyawqiwtrijahwuoxI8yscMw/UF8zGcwSQ6+L7/zDGQu56QE2bt+5en2ezE9NmLm887DutM6864CTjJqBTd1mHZjbzHvNnfw0Z/Q6c4NaZbD15uvb9Yb127pdZ/gE8yTfKXyC4gAuRLIvvvckGMMqL52z+Ba0jVUWrQqu1RHtGAO84LYdleZ2odoilBTOsFm3Uayt5QITiAXkJtid9Y2NRDLiirjBKJm19PTsPKISEufaykahsOVx4FwBfJODnooZjs0KtLcMfECSeFwaF5npWwiLuM1TTIvwMwxjiF/kvGKMEWElGCrEUVI3UIfvAuC5sZbOpktweooZsSlAIowC2xdGpfzww49Ozc0de/W19a50Mp6gvgJpkGH/YwkXWW07Ul00HnE4nvn2U4FQ+Oab9kOAkTnRlYHIMGs2Gv6l+QWcesAmRqxH7qCIOqsUmmQgnGEUAgDJoMwSYmGUhVRg9mRFlxcnn8dPJHvUAtzEPUjHQAxQCkPEAmROzDkcP7QYgWHuZ7EwKkCfxl5Tp9nT/sZEc4QzrplpcAlcjAwyvMDMuMbIAAcNmjnfgVFOWsCqN7rb8MpO9LlOnz87NT89snsAl9eNuWq00xXw+FY30rwOzgDvEb6kPZHyIFjGXa+/emZ8vC+Tyd533/3/63/9LwgqGm7lPHS1SAa5kc9RNjmWsJFYw+PbzuTmA+H2YoFciTOrK9n9+26i6Pf5M8dSH+nvSA0szi0Hfe2k3U6ns8deeRXHmD279g30DORL1Xyx2tHROzgSmpidPn3u7aHhXdHIwvlz1yaurtx2y52dyTU0K6lYIhL0xQJumPlyqy3baI30D6MjrBSqsWB7tZRuT4QKG4XTuVN4xUxNTI6P3grnd6j/1qcnv3X+0sTuzpQrmtp/020Xj891R+zblSVvI9uGeyZBrX6bL+xNdtZQkhk21ppSQwAAM4ZYun6Qj4ZfI07FODhwXCI2baV1Mu013LEIE9+oFpln1OO2QFspjzaSWyvp9Qux2KDdE62sPhvee+8P//gnfu83/jaz1kjGktl0nUwjWFZw+JOJDcHXuPjQLNk3jAcu3kVQVHRGQCiIC8ca4FHOxaxCyBvEjFQr5NOusdBkkcWXBUxH3Ok2DkEo++zKIiMXWgpp0BvSd5NP29YWQvZgw+lFOhTuhvg3ZI2DIEZCvlqlKB6RhBYuJynDINv1GmUXiuPjB1lNgDprAj4gnSYhFyULy+R4AfBwkvCYcOFcLl9sYrYI00NpbAA+AFEvZNtBSlITQYR4nyQnw1TqLmFT3aWFYSCbPRs/uZN22HiMZcZJcBhGMhYjIM556zadv/Eus9K4n2e1DEhqgPCFZIeyj+TL1/mAgF3u5iximC36J/RmrL78cgJsBs+qY0J8onp0gSVPbm5XGypoIiiqPEoFhWQw1RXtG+keIR0MiXtKhUqa8tptnrA3DApmsgA2yojLFk3QLhwHQRT1WkdnxN3dLW+tTLakEIGlarHAHUxdELcXvgZMSVJ1vKiReQ18GlyO+AtGR56EhiGviyOhG3QftMBeBNi+Wd3MkVwGW1Wxni3W047wdtdILDXc6d3dadvK1huTpPXAbdYdtzdt5fV6KeDFGwGY4/tRFUiGxVyOpAS+A0eJV9LbmB4Z0mEaG9UyDjh8BYZrwAWlmaRv1ofILcICzeC7Ji0r7qw4DbqZcFLeoW9kQsBi/KfOiu4adkG/hAz1CdoEL+Yf7Xc264oRXYUntVkzzoHuN2TPnDY/rRZ22tPpnTb1Cl7En5qihXdfsh5/936nmXefeufY2IbpO8QVWguXKGgRv4GVQXYU1jVnILr41mOdR+nZYoniLg6d2MLJgwQ3WHlZ8/xhzyP+GQjEa40skqUabsLYaQJ2tBLw6LQONNNpoLO1+dDd9wDKxOqkl9Y37Bu4gewaHidT/MuOY6R9h9UmvmVtZQ1e3OMJinq0iCm043WvuDstO6ymMOHI23gp2vyQ4UqFk8S9KJ4NU1ULJacX5m91eQW2GscfLdhqoyMeB+jwUsQOCd/LCsZBgfU/Ozu3uLRMnttrVyfmZ2aHhgZIWcw4wRNQloDqBbPzc5093YL9evkLX/hbyeibmwT+s0h5HvLc09MH587qkOegJH3WtWwWLHDmiOLGLGStZZCwmX3uZMOLmqtI3hzTfzNCEpV4XsZdphkZ1wQuoA7jTiRLbuOA9wKcbGofvRRKxe9UQVuwQbPcjOhvdYOTbKwDcIGm2gCH6Zd2/OQ26yQtc2C9yOFxjO4f3chvHL2DkJ4NKi68r6efyoN/8Rd/MT+7QHgSdQ54nHHAKfro0aOvnz6xWbWFYhQMcPz0T/3MH/7hH09cvTY40JfLpcFXMLPQ4HDEHU2GuwZShNiUGyvUoPWSA5AiCSCgAAGzTnJv+b0dHnc47uyZqq36CJxqS8wXNshg/fhj79/aLM7NLGMMbU/02LbdpOa4cPWS1xf+xte/1dXbh3RzeXLG580N9o0e3n8gn97oEgkm0RMV2rypeGykbwABJuwOkazi5gM3H3v9m7vGh2CewqHk47cdWM5Xn3vmtRFscLsGcTe69dCeiaVLNw0N7Nmzf/XiU6FN0iXZCFNhfsCi5AUH5UOaWIcMqfCehk5Tw28p/w3vb/CFQtWZVIgydBMPqvV6sX2L+gMBN5VQscTiQ1G34QbKbTRKkJvdlQNhEVhia5xpH0n9l1/4zB//3pePvbDeGYuQ0SYQihFELJ8UQMCL3RezMpwxLCqh3lQgxkAszhjJmMqEHrcfSCbwBPjBDZ/1aKhvQ8CGvg+1CDy14aEBUhApOJ+KdegwNjJpuoW/P/AAfEL+iI7Hq4EM7FBIiGCxWAKHhoK4RYdWlhfxqMJRo4CvlsPb19tdLlXRYOG90Z4k3hcDU4U9Miy2fAyUlt8fqzkaiROth3MSSBiq76SXvI8NiLQkVGCRDT6FM4wttFLMrPA7dFpUZEf3aD22MwOSmA0ZFQ2m9/wAUpkMefgiJJqNd0EruIGr1hkuQDOsFyGPgQZNWAvEAJ9grRwWpogJnAi2A97F48Q1Kr0O/eEMyw4tKpNtuG1U1woogiYT74XxvK1e2SQMB0fQaLS9n7XU3nvz6K04X1aL9Uw272pzBd1hnkWWJqNYrUqm7CpvwSjL99JJj8+W6Oy1YdfLZarp9ZXFpXIhr2on8Wi5VEBmpp8IlDgQQ8GwiqEewLMABhsZHJWtYcPg3qUwME5RcOR8mBQCOgB5OqrueFumslCuogJyDByMdw4lHCmnzU/1zottwU1nZMtjr+N/hxAtQ28bMhCDQU1shU+BYWRnkoMYVMVIXRp0uiSXXbiJbRSQZdYBJIezituCTWHMGVktDgszmXlkOjQpxGmh04bXVCchJjTHtPAF3MSRSLJWm6CCHV9lqKxUuzp5fbOOzR0SwK0pNe4ConwsaN2g2TSbdWA1eP28ntV5S0JlqMSxWbdzXlfVK216tcG0VoOMkhAGrAX333iEFyKW6k7hXsgwo8SDSsGBOkBjxzVmjFxT/ISzJm1VjbyJW2SGl0MVQnAdg7yScFTRaoqxERdFkvBcabNCCji7zQfHiVu7ncjInQHGZEJwasDjBz2UqPBbqhJChz8FvlGvvHQMe+fo4Nijjz5OuPnLL75CUkOIdHZjBY4OTRusr3IJKVxNLCnzRTUTQvOgxAR4oG3CvsvAIhkLk+ES6lHQAR9TzBZZfczb6NCuUr60srReyhcd265qCfnY6fOHofooVI8euXVxcR6qObZ7z6VLV4CNufmVvr5Ocj5H4jge1hPt8b6BfhwUahRlQ3fSaEQjMR7s7QvffMfNTz75ZNCHNQtGkOxH1BXB9VsUV8zLDhOuJQ9Dp5Pi9VjjaNS8hNrzLkCANcKqxzZEQjiy1ssSIDzEjTyCQpuR10rkHk6h6RIqAQGpHVNUdAcALBjQ1LNxwzszbm7gYV63o6PZuUHvYAMETN8MAHNk8foe59TMZO9QLzk0uvs6WUO333XnhQuXHnvve/7wd/8QZTZ5BKG+OKK3tdXSOfKO5Gq5Vs9APwD2/HOvvn36IorKbKbo9ngJsOYN+M2RTzneGY61Rz1+Z5SKv+Q9y+OYgzd3KJspfO3sUx5XtK9nV6t++eCBQEc7xaZwsLd57PGjNz2wJ3nzWn1qdmaZarLheCKNu0BtE/+D7r5uwmO+/dwLONns3XMIbR/6PIway/NzQz0dWzgGl9q8Dl9XvNPT5gh5Al3JLuhKJOyOh9uhXmCHSDCUrWZWljY+8uGHYzbX0xdembs2d/ude8qOVq5cSxy5rTTxYtxWdzerYDl/yF1puYtbJBmFnK5L2cbwaoSFnjWYwOymQoEZcNkfQBPSWoFnJBNTDLVZthU2cjH41jCaVQ8ODbg/RQMkkWRNUMTQR0B5tZrp7NqVXjmeSB30DBz4gX/30Xz2ny68lRsb6izkiugRid112TERslox/8IA4BaGzhnJtMKaJp8JsMMZNHpcJbyN8r0cQGxw4NWdFCrG81klpUk0JMwBYgK0sJQjCMPXAiqQTKJ7QXxcw2RLRVjoihA7WkdEF5BEs1Ey1TiYphqGyVwBEc7hwEKfgk0mImZ+YRYI5Y/oecgdzDHfCgW0UA8NMSwUluAnPSRJiCRgYVkDjvAxXAYRQ344wUlDayXLWjRSVNiMLrdxkr21ccyyp3We5U5rY0mA7rmfe7jKngatSyi3OeApNs6zv75+tvEptxrhs1mcsEXMFSuB+01/qDJElilwEnoqkWNGTcIwHYR6MFYK/FaI0XaDLFAkn3JEvWEqcgz2D/X2DCTD7ZmlPCwhvBTFnSQ075AE4p6LuEsEQ2GkSDQGgA9hJLbBnu0rb67NzK+uLCPNhAL+RFRxkxvryyBc1Ld0DFkUnyxx8eBLEl+7JIHyUzACUZSoLm8BcA5cF0G6daQRZhO9MX/OUiZ3LdRu7+0lKxy+GQFbCFG0aGvLARJiFp2o6QhVaZJIFHROcCkNYI2EoKLP0exgfpYHHG4XiOBAP55iQEILUQO/D14D8wF9AYOjPBC9lIc402qECC0iyKoAF4ygeVeKapTRggZGG4aJvcUfibCJdhqtn+bNLD9z8L+zY5YFDyKjppHrmNLMuwDgn20WPHBSHTBk2IDJDoblqoEf0c13HgTcaNaojDQUFhowLzKwZyCXO5hxTbp+0jSIAzovZzOwiMwFrDEHugyyuWLoJYgUb2cqYlNym/T+yL6tbTJ/ipwjfJDPP1+Sks1OHk+C9hWSDizCeolzAEN4Ha5LZ8/jtkPeduEED+qLEvwDh/A/3b19vT2DqHMvnL08PTULvoKjw+YEA0UuRRYMGJ+FA5XVIJjvBPsjqrJm+Xzu1/ltW7mIsNggwJ+hLeYKkGfOJ0JxAmygvtjdJDeXqwBLIBgl/IY3QlAh/FQnff3YGzgYQv5DIX8Vu0ubjew8PI4n14FD+8fHdr3y/Eso97iTxBTo3PbvO0CB91deeWV5YZHAB6QEFiBwgtnI6IKIhCceSauV/uJ6wwFogXvKBGOYjTMYbvgEJgWQQ3GKT6ABNHCIWHNDgHlCjfD55rsFikydWC8I/HV08c7UG/xj3anbzKTrJ3QSdslgB+tmXmo1ywiY6xpX7rfO83Y+f3F5IRKKJhIJirq/+PJLp948/Qe//0dXLlx94mvfisbIpR8MJP3M6alTp7q6ez75mX/9d3//Be4/c/aCLxA+uG//NbybJlep64Dmo9qsZYrZ2aVtX9jTN9jdMxQy0agd0FHUHhMTM8eP5Yr5ue2ts10dgZ/7rwNHD941szgb8Xfa24M1fyVDkGfLk0z0xmOd5y5eJG/t4ZuOkIr63PnLH/qu73rj5Fk01RgTEUiOHjr66huvBYNk+GFGtv3USfUFezqHyOvcFgGyyYthp2pPNpMZGelBjYEAN9gXq9dKufz6ZHY20h64c+COv/6Hv9ndGfY37BtPfXMU7Ii8WGYB2jxh0LANBKNUYeh+pNnXJjg3cKmibCxJfrEGDY8LIDFPcGWbNVvAK7mzXtrKbWXijigA4WzzR6Ot7HLNFbAFY7Z6oYoTYihkr1YnEp3JlcWTMX81PHr3f/zp7/2D3/ynYy9fGepLADOiBWh1wKegBflFQvW33D63kjkTwuz1IAux/rU2txpB6oGqPJ1ywuD+ZSadYhLSFEqJhccqXyLijSWYGpo1YBPlM6AOLGxspEETJBJGB+YKuhDZYQel4FamVT8HlAUDPDiZyxYxQnOGiPB0OuMPhpdXNoi+gYIg4xoXRlw7ykQxoZ2mlDXZxHHjAnjkJuyHmSu1/ehQJ8DHZgbUAkSRWzrECFqEEwTFDfoGoTf9AalsfJ61MLiN+00z7zTFMmYuMBRZ57mZjbdYTXFAgxyz0iwmwIwRpaS18JhMbtbKNGZjWClexHCwodbxuHxGU61nZU+iGVEU5pU/Lvk5yKwU22Op/t7h3s7BWCQhZZBcfDcDngh5UfgGNagZ2pLnJdoGwraBHQQ/aDg6giCeM36b2zHxxotYW82al11JBIsp4k65QzOvtAW+VpwSo0M7xWoOrgc9Ei1hvcfNDlqMR1llk1RDBIG0KvZScTNbBRG7mjZv9aZ7hjyx7UDEI4c+e2lru9TmqXMbSG8L/zJLlBZgM2CgdQJbjK8UfaCz/IF6DA0yeZulM4fuqlykBFxRWYaSsdSzrELzCAKWPoFMNownXggsKyKuwYTwoiIBm/QWkIJxY4pZQxAEVdSRr4D6wNRYG43q/TprLujs9WvmDiZXfbgOWpwTABhRaefYOrPT4s6zuscipdxkHJB37jKfqYvmANh55zbzOn0T7DZrXRutcWC1yQBYDCXrlqlDdSAWQ47NYnWRF4FjiBmu4Ma3rYK8iz1CNmDWIwIHFI1wPMgrc1mBtOP2tkV8WKtSpcQsdl7UaLZg1O8me72tLU/hPVgsJWYihkjIkdVBl1Ei8zpx5nj/olpx2AnHX19PUxOQyUIwJcY/n163NQq4C8AwAW2aL328voufAGyyvcNSQUOViR0iCo5iL7DneACyalgOcIQMoCCpUI6EQrwMMoxtFXUxaizEfcgOIY+sOOg0IfJMNCSZOWKa8T0Ba4ApxkaGH3vssV/9tV8dGxop5Eowu6wOtNYQJ6jOffffi5sJHeZFVBwnlBHHE+zHeKRwAx6hNMWK4iq4h16hr6Z9+ACO+QRRPqOkZy0DZdBt+m+4Fun9ADlzvm7MQPpwfvJdtMY9tANAGb2MYeUNdbZuIHc094BLGEwmF6BDFqA1Yt1p5F9udIaN89zNxjGr3uFvGzrYv5Zbg8u/++57Y9F2UnI++sh7n332uZ/92Z9bX0sTWk0nWemUsrn//nsvXLn6mR/5Dz/6A//x2def+61f/1/9fX2XL1zYu2c8nVnt6Gi/eOXtiZmpUMIRjgeyRSqg2Eb3+D7+qbsuXz2daO+qlm0nT17Y2vZPT2qmKNh39JY7fuxHfhKl5sLswpFDtzLmpOmwOwp7+kdeu/wGqtdoe8KHf60t+D//4jcPHDz80ouvnThx8iMf+ehdd9wNPgwFwhPXTvdgK3Mxt8R+Fx1twfY4ubT6NrddoYCnsVUkOWUmu0xITLnUSsaH4sHuM5cuTc0ufezxHzg/fXFtfj6yVc2ee+Vop72zMjXoK3irWcpD231thZZ3tQpiw5CQc25j0TNrWXvAVKPLWOLzwrBbf5xhssA+Gn2UdpbuFzVDwBZJetwJnw1re1t922Nz+qRA0qqFInowIgAN4WY1at/scgT2NVdcf/Un3/yj3584vNdJn5HKwFohkkD6fKhnVPZDzLPkVK1lkB5KJtS+hPZQnZ3czeIaoGJkphQUIe8iKuOQasBEn4B4wYQS1ITsS0+V/jAWJgM+nu30n0LSZCcGU+hZMCvezqSQ8noIP4V8gmoQqoRkSAyAQxkvq6Ncx0MWpCKBhna5jCBjQAwsI0EU1RcbiJGlh5gklTKbBsvs+R71R4K7AJRjzgvT01fDhoOmtQjMpid3nlLT1keyt05aj+Neb3G4rA2IOP2hVd7CXSIUmj3soyw8LklO1y2Mqf5HkwC+5JQ0GnR6k7Jh+L61bSEJElBB+1zWecghgVxQS7sXDgnvOUqDHr3prrA/lkAB5I84t90UUFQWPNQQ+M/V5DzF+AR8UB0XFWaocY3hDWs5Qd9wEsRLptcXMlc3qsV8VDpBox0DG9IpoXAaEunlGzghCOQf+iKNy2Yq0Q6Ds7G6hqoF0wWan1K9nKlkwx2BbGNto7RYsWc88c2u4fjAeF+oL1iozNv9W5ueoiZxq0JIC+gd/aMzgDgF3pEVWYyBaAlsg8UAYnnjP8EUwi4bPSrny+zVNSnrmCOp67QExDyJ+pqJYryBVwl/UgKC4LlMS4ZC4ZrFvfgKsRBktiYljUykuIzCISkQVVwtragh/WMRQtrl7/9xo+OCqusbP68f7qzeGz9vHLz7nhsnzTv1FVYDOtiRxtV5s2mQrt9vARI/kUTNSZFzoxsAihEGJEySRscoVJSUQ6QXG4Q09EpRT1QI1Je9o1rbqiigF8IJr0Q1CqIniNtysKaxPyFaEXaIHZ4GeDWaWq8iwWSklI1Kg4TBRllyxPFgv9IwAzVysEJrgls1MbiD/QPf/clPQZM+//m/QsGrRUATgjOsMdZA84zWIIsSqok9iZXH2mbSgX9WsBTUrFeZF0R+QRVmAdopSu9TCVFXC7ciHO9sOOFjOCNPiKOWXse6iSlrbWMdkskokViYlJYIuHhW9w/0UVUN5WosGsPRFy8Km1tFT3kp5JmrPEKuH3oyMNAHHsRmvLi8lMDdtquTCtdI8PSfVcklJR4yG7gJksaL6Jsm5fpcwUWZkwyWzvNpN3AI65Ez1sZ5RoBjMSXXGUB+GsDgitCX1bIZHEOWdVqOKYySOdTj5pEd2OOLrHdxlU+zbmMiXnvpQihlu+XwoTePn3zowceef/7Fi2evdPf1F0m7vLmdyVC5ttE71EFKyMuXL3d2dZOf41d+/9ffOPZ6oVRbWqS0u3d5ae2jH/vow4/c+68//b0B8mply9v2Ck624ThOnG1n3r5CvpPnnz3vD+D1to19EOesahnkXkhnixTIc9qRoILIZPlMATtIV2/HmdlL2Wy+vTsF1K+kVzOFq5FodHVtY3F5ef+BQ7vH987MTGHsuO3okfb2cHrlaqms8FNy+cWjPY2aq1Itkl1rdnYx2u4H5eIxupFeK5Va6XQ9HimRN4piBs+f/vaFC1dQ8twy0h9Jdmy5Kp5wAlkNVS0qaK+bIBRXDEwOelEeYwOgZi+MYC0yUCAqpZ3BNiiCkYUgSmhj2swZ7iStgWzJTV8qgku/DfssUqkXQys4jdj6LSel2xwVGXDtSKtuV0/Xox8eD8bcv/frF8eGfZn1ms8VzBaa65ni0FDX8vISiZiYdOQGOVxIAGbBbjqVDYYyCUAirqTyySVUWCJJG/5MSn/AuoUYA0EAlgiMnGmEENAbkk2LUSoWZPDGPzcAFLN6RU1dDk8QzyQs1ZQkKdfyQm9KAIOUQtQCBJiciASeKUQHSGUwcK7lnSx/KDUkFKsDhI6BUI59lgIxVTAOn903DPCJhFzf+BwLaEUXxZxai0RYRqoMXHT4TFHodyRgjllvIAUgGDTBnTwLJYPFFj2jYyQWbtXBKkImwi0sJRKNiWuRowjpuinbCQcjRsMsJ00r+ASGY0fdgQ4ALp4xgFCSqZLvgxhJGMEY4g8iZyB4ADBBX3RkcNdg30g4EPM4MVkwx3gQ0DAUmk7JNU4fyOjTsvmDTSHxVCjZvl0rZ4D6QqFSKzNkfCNKAFezQY5Kvoj+mCUsGQoxjrHEQZUvQafBRzHKXAW38ibU27yC1FKk7sGqCFuNNXe1NFdpW3dG6317Y7tu6rB3e2ytTKaw6I8FkMdoEHdoKiVglUAkwxGNlAPgK8mYog8gGKbBdF0DiObZBCUrNxDuuTiPkdgSWxrwBDLGViiw4E/TSl0w4XJ1jyEzn6CdSbuPGMjq0GRi8mZ6zRTrLbpByBwXVxNbzDfSgsoAMbWGRBjSSrt42cm6+y82WtBAG9RpXdRbzUn2N87oYAelvkNE+e4bGxTTul//sO1csm7WezkniFKb+sk8QGF1pzZ9iIy7YpwYSKadPz5CB4CEQozMHx8qO0cVx0hMO9vonDEPVHBvxuEJ/tyB6phEZlSuQzB2Ass1LCUQVvgVUTUyn9mxk3p8pB1GyDOQgLuIPJNBBB4wkORXCsWDAziNsougRoBES8R16NBNrJq33jrNNPjdDuJ/kYBvkBONoAEnmqJkL9J2MhnBeQddNKQWAgw/znQBG9fZXCMfghJapOYJswrJ8lypKNKBdzFz5Pv3B0IHDh9i0qdnZvsGB2D6X3jxRdysELkgS6SBPvv2mbW1Yke7YUbL1OkTew1gk0eeJHwo3zASpzolC1K3h/dCinhW8cGFIqE3GCON0QnRQZIlx/RA1jRRRMnx4kIM9jDrXWp2uqfZMhw/387HwgpzlU2QYCQJ6ycSDoPJxmmxM9dJL5hKLRj5QcvcKOdoB8Mzd+oBs9G49dPCV3RGA2wIsN67Xfe3U9Kqirasrw/HqEA8kbpyaYIB+OxnP/uFL3zhzPlzsNTZlVyw3dfd3UmhBbs/lEx1nn37nCyFbfbRoeHZmZl77rkDpeg/fu4fYmNBRIBCOZ2vNJNdns7e0OCwf2ZuLpO1kfgyloyiAnj7VOa+ew9HIx2jI3vuv/NBTPXNKjbaCPGbQb+rWFko1NK7x3bVbPUTF9+8OjMB5By98+5vP/vi6VNnd42Og3Puuv223WMjr7zyYm93eLO6TJ6uaCyIxoLASUKqKhVHwB/rG+zx+e2nz75ENRcctklF35kcP7j7zt//o89T5zLV08VMDfb2PLLvloWLx6ZefuJQpJHYmutyk4i3hfcCwe4tB2IlIg4iAaYaseMQGnGzGl7RV2utmwOdZWCl7mIHH4qBDvWdUV7jGrPpsKV6A/ZYgIptm/XSprPpDuOJul2uliilSzQ5yBA/XLcz5PT227b6t7Mdf/57r/z15651xqJUK1ldWYwm0AxXoe4G6YG7gB8tGtYe7wXk4ACNwx8dwKyDSy0EQ/IkaePAejDcIoqiU7KSoNXmGWu1srhoiNh9xFMAk1YJwSEnZSAIOwv9ZqFLHgOSAUBQP4IWWJdVD8/E5mKA5HvL20ReBatiDMDhNg9ud15SOBACJblJTJ9FONUBC1NZ/Cn9NXFH1iDyVg6sjXaBea6bR8zYaxlou36LGXcz+hoHLRPhI9FaCAnCo6is6CFLnfMShs08GQsCK5Dh4xFxJmBHsCb38JsBjYRjIpeo76tgRPysSEBLIh3qmbRVs63yZjUaju4eHxnoG0rGO7wuf6Uoj3miSCzShaIYdzBGTD5vEE/qMpNnA3Rqa1FQkECl4socrsIivajmNrH2Mzg4Rbah0WUS6IbgR7PLsUaVCCczj0wwHwFdF3VTk3BaqDL5kADWO6SlWrq2ki0sBVKOkbFI356YL4UPw0qpsOHyNqOdnoYtJ26ILyWJj/TFyuhGY8ScmsQHCjhiMPgPAsJ7SVbF3QynDGlY7UhvVFOCFyYAEUd5yzH3wLaqp5yDQwH+IOOGbQXk1HnRaSg+Ho6iB2BXk1tVRF1O9m4ywJjZRMDS16EwhW9TNDodAmRZUdfXm9wSNEOao3+5WXN347z188bJf3Gw04g5b4DATP2Nx/UFusW6RLd5N3RXxNZc4pr84c15zmg9cpWh4hCiy+AZ/TOklxaYUoRgNEj45YoTxTkQ1pZgLoxzEGAGAKm1VietMek9BYaAnnKSoX5p2yL1C8nJsAejYgBkYUEgeniDqOf0x6B7A+84x5NjFnwD74YDF7QcEJb4bK87qb+W6ujIZfIE/IBGkD6Jsi3lsqwRLQmzIPXtzKjRSLFeMD+CFAZHhrHpLlFh1CFVYz6dMV9IF/hMKDG4RFIHpAUMDrfNxvo27hkQ4LZsNg35P3f+TJ5aC22OKjxzvYVCFekW8e7xxx578MH73/++D/zyL/0irRXzJCLwAfK8HcaUNiE/KOLQwLHk5+ZmwJiUK6AWPZkauQEHMZ/XB0wi8mqqQDrKSQmuJDhEEgZoS6N0/esM5tCk6qTZBNuGRrK3KC73wFRaAyKWyYI4c7P1lBmtHQ0ca5OfFsriKi1YD7LnCetO60VgW264cYZj/dy2Lc1V+4b9qyuVqxdWh0c6F2fPj4/vIZXwl//xq3Nz8x435r1631h3b2/35OREKEpKAN/Cyvxnf+Yn/uov/xr0OD03jf0PFf3QyFD/oYF8kRSkGTmaEVVg90XCqfX11d27x0+dvnL5ks0+kztya8envv+h/QcOV0qbo8PjUB84aShQtVEhDWwoGpqaT2dKa06fY2L+2qkLp/JwN6z7zc2p2alP/+BnyoXK0sL84Ej/tZnLq2tz+HCRIiTVnqD8EUk0c1n4RzifABl+erbbqUaMQwmxiyCrfLEQjymsFBaqt28wEPMl2iO59fSF+bPOZn3kwE2N1Qu5AivfHnbZyuAN3AwDBPKSIwZKKp0kwwgt0erTwGpQzcRaJ6zZNAwxVhmkLDFx0gjhTyS3x5ZtbaGc5HR7BJvuZrPcLBD9CFb347XDygZk3MoJnW9VLnldtbaI8wd/8hPN5te/+qWzfFuqN4kindoh5bIapDkzuzt9YCGAyQolEwBOR9BRQZg3CYrB3wAPSs06c40QClQZ3CpxEGyPx5gUY/IsxI06YqNwnG272ixL/9igCA6+RfhWw4vXhDmkNGckELlELqGxhCLCpgs74vIpdIO2SUCFSKlfDJksUVoAJqhIgQCwbPLUslb4DVi0AFVUSqpobRYR5QYIhEiNpkNiFW9iYJkLjhkt3gydgwLoBj2m7wTQRef4OPMCzohFR2vK6yXXi7rzk37xhLCneT2Y1SxVLQnxGdAlErspEy9Bjf6wN4qihkDmRnkz5CH/SDSV6iSHc2eqE8JcLFTXVjOhQFSaYwR/nMQJ9uZF8hkW8QY7tBGXRnQ33uoEjtVLuY1senWFHiPwB5gTo+LDhkCaFopGGujSsDJzZtD1ufTT2jSokDXoLoiKdttq7qADneVGeW19bckZ3urbmzw4vjs0FNjeWq1sLhU2i25/M+BlnlFN5PlsDSAgrbcL1VgjhADMaMAyCLoBATgXIJdkkXgJqAxAE7UCzAGBqlBfntYKhxXgf3VSMEHHuEALmihld7JkQ8bBXJGnNncB/+jYqRBlYAMXX8yEVaqlotMimgUFKlGvvHQTQgDM0hjQzSwJckUARYLElb17ozdcM29592nrDK9891VzUmes7cZT7xyI69BFoUcdGGFXa0d6eMBdV6y9fgGNdMY8IPDjk0Wk9Yd+XUSXYwZc9Bgph8UGDTbU11BiCqfX23AMwN6ttGCQbRg0CmmQv1DmdRFdlA8o4UBC0gYAlvA5EoUVZiYTj7U6xByIUjAPddRq3Gyi3OmrZC2jcnFQOgYXDJstEqIipeCTrGvoJcz38PX6BL7SzKaWBH9gBaRhtBVl/PXLpTAJJ32+IvADgRfoaXmageJZezQUgfyw+lh3UougwVAOtFZXN5XXxODzRCQWQywmHhdxE39aRPfnX3iJKEZWMT3e2Ch1psIUFKF9t7tK1LLbl6YRfEp9Aff8/Cw0NZ1eP3++TthSPJmoLZH7iLKbwi/MssRK6YdkmYLYeU0NFGbEAICmkgO6BX0WrbWWvxgjdZ5VAM2m/4gOSHhoctSOhtzKWslXarNGSXebjZ88Yp3kBM1yDyNg7tXrOLix18gwZ0b25RE2/XR4Eq7q2nyFvLo9qfZ8ukimdL8nyAi89syr3vYgigb8w2PxWDqbITHW/MrC8GCvK+D44pf+6oMfevyFZ19YWswM9Q+QYBpN74ED+3r7e7/yjS8XajmlPCJneKne1dsFt3bk6K0DI2tnLsxOTq9Ozq5u5Nc7O3o7u2PpgsPrVIZbnM+xPmWyjbHxscn5LVITT8xPwPUFo4FqNvvq8Vd2je17++zpe+66F2FxYurK/OzVa5PnSRKwe7C3XimQHZU0KYSZdvcMU8B9cWltcXVybmlyIzuHthFT3czCmsOZ9Acv7ju4C/ehqZlrXT1HnK7yGydPBapbd4wO3ffIh2ZOPZ1efAPNKh5SpJBkKxVqYXKBaBwF3frnOi+ly2w6yaahlrIKMNBKQJkpHMGKQfzVuiVda8m2sVxIECkXD2DfwGEApRMkQWYUEidgg8M8TGy9XJZXycNqD6X+/U99GFh48munSLUcjgZarTL5LPGvpjUgjTcZYiLkzOtwmYLycSAlnlY6YaR13ybrus6dTCjxS6AGl0vsKXAGimiQWAE/sBpxucpDbnJpkY+I7MXNRhkHJFIichucNKwuN0tugjMXYjGoCRkbkGMVgnTYuFUIUjAp5A0w0mDZCLO6LOBkmTTk8gBQAsQWHDNwrFXOgEOsM6YJ05xGllAk0Vxr4xKbbjactmlD3lv8tC6xXPmPY6sp9tb9wDr38JMDzrBs2OiJqLfWCdMIwTaLld9CH45KsRrwhn3+EKE1wDHIxO8ORoLR8f59vSnF6kHQ85niRnGd/LrxUApBUFSL6GTAgPZgWrClt1qIzbjAkJYVLTZK4iK1kUsZQrbIlSUHYsaVAUEQBN/xWrR2wgg0wT90TFK5+akx1jfqPGQRsNHK33Q13bHttcJ0OrfqDm8NHY317U0Fe90kbS6VJ31RaoLSHdcmSUvr5LXBnQxjbdMo5wSsfDHQZ6avRVzJDlmD1cLkiC84qd0gwTV8hSDAkF7iRAXdYCY28mcASQ4NmyHaO6wMVBPEJbzGjPNH67pbG7RTQqShInKWYAZg68SQtnCcCfvDVHog/DoHEKP8DIY8Un3QS1ET5keKZ1qV1uL6sjPN6pq5bsbNfJb1k/G5fsPOvzfOmIN3rr77PKNMV60bBOjqvkHf6oJGiOniqiG0gkUNnzHn8Bw6ZzNN4AIMHBzzjWqBP6MLlRe7sjVg9EUCbkCJIb3iWOE0akis5JnUHz5WW1WSYZGQhbkGPvhAVetiIEAvDAnv5V8YGl1hGlkjED20J7BDEFHxZ/SA16qbZvA2G5QSokAKZADFNrWHSKHMg3DDdFSfJx5KDfMgnwG9p0U8+ug2pmL0z9lcHn009U0Be92lzWgweIX+3QS4wBm0zwVruMX0K7CstZ6lkFGYUO/pmcloLLFrfHdXT/f58xcpgIp6+uzZs4f27T90082zE1Ok2YtEWHHGktpW4hX6IDuRl1g+Gu3hKFLv4koaLYul0ML9xRoBMy+CDrCBz0cYpTwImAs2LvFlHGhlm0091NRp4zw/OBDxvY43eIRjTrIi+Ta1cB3DcMzGeW5g0wvMZp2xjrmB+9+9WWd4OR3jPLfxLMfiX7e3x8b65mYXM+t5xruzu2d6cubrTz7190e/+Lt/8LvwAmjgydhAaRO4+lwhe23q4o/8x//w3NPP/t0XPv97v/17P/2TPx2JBmjtySe/1t6R+qnP/sTV6Ssn3z5BGg0Ev0DB7Uo3447Q0UP7XLO+jr6e+x+6/5lnv4XDVK2Zy5fXYKu7O/rDpEzaLlFOq1gp+aIJWsvks+B6lF3Z/Cpx3SjqTr51cmR4F5m5bj54YH7m2t994W9vPjzebJVxqWuS4SHsCIaDjHyumMVsgWqdVB65wgZxs05PIB5LYEalJkG+lO3rCeSrlY3ZK6fO5KLBUFfP/03Zf4Bblp71nejOOYeTY+Vc3dVdnbuVE0kChE1mEAYbxvZgTBKIMPd5LuMBRDKDbfDFYLgDKAOSaEktqdWpOld3deWqUyefs3PO8f7+3zpVagFzjVdX77P22mt96wvv9+YQXowsLE7PV/rNgS81DkzDAjltDQRHdG/CpbhWECEnOGdRNPlMIJ90Ul/MIWDXwUpixPSC4qGpEEMhCu6S0cumgKa6LTMsp8nsNRXDtGknnJ4QGKJ8gXJEbTYYNkwKn6OkGeYdjYvE+P6rX/hOu7v3sb+4mIwQZYkqxztytUC9AAV9Q9sEhUb+BIUppSnsGnsBzMYelxhKvwa4NCA000EeAdRAXhBjfqUYF/I9o0CB4XSgORAeQCzxhYligheEG0BSrYsIMAIDhowY3MioeIV1MBP+Ifpz9pl1gfas/ajXcUr6GT1uGuFxLgJ1ewcga0G1BalIwNyq5y1KYzaA1erX21b72hh8vpmjfPOSwMkyUvNWvRagtyg0W5jbaJyLd3aLVFX4SWnpdFg9QfaCEk5PxGFSEc5q1RpEKBmbOnz42IG5w0vpg7hFlHMNeBaIbsindLuUOWLzC/mxM3FU6na1MvC3fkqX4yaHumFEcahKsUhCT8eQIoO2Phnm8NiUdlpGbpaOzPEQZJgygROzZT7BnxZ0QdqtHmoFUTpjTGLvOrsr2UvJZd9Dxw5FlyM2lDy+8si503PWvNPDeqvcy3cRLSNYkKDGzVG3WvZGMDGgq5DJlz4bsACvu9HSQB0gDlSlUBUYJWQSrBD4r5QMcv4RY0F/gAbAiAFyCmjpOriLb1BYaLGcteU3pqXSO7i4N7cCCCZJOnoDpsAxFIaQcyQ+NAU26mMzfZiH2XgMjmVWi4IaYNtsP5HiPTc+0+ibPvSmr+9MrTKHZtGCPgNaty9aPbK+qZ/ao7dvoBm+co3rAmD95d1S5ZjrXASK9KMOQ3qtsZrv5icWzMC6uCR0yBxsOdysRH3JDCbFGoIOczzA2xlHaMWHkUEQ5fOwJcu87NwYBiBlRkyUU4ImQ0yZ6RE+lCrKaVFAeRQyRvYSGn4wAomf8XQQP22szLyTG+UsgokLS0mAxBrylXd4PeiCm9TYMeBnzYUGRKtqeoR/JvIldK6DOoLCRLGIrrK1JF8YKQPeW64oFj+iiEY1YphpGb0g/cTdejytdoM6Afgb1qgk3qTIWJ3eHjh8aHN7l2x5LLG8oA8ehE4TQ0WwSqVSxzUMOkWMo7y4HRQYD5FTLRRSShCmNhjCW40MHyi0IbceJzmPDQtOt4FZBHsqcINijB8G16QTBh9oaOagQYAQAGa30gIHXWUODVYA2xsZmvEZeqyfb8MVT3NuNWKASiIsbLOFryzEwnV2/Zvu2UMpXKFLfO71x6Aa6xFm+MXnbkxPhTG579t3oNHurK2vkuH50be+pbJTnD28ePGNSwcOL33/D/3oRz/6Gyiit/Kbn/7MX/7uR3/nR3/kx37nd389kQyXKzlw0T33ngEwH//C58kDzHQR5TU1M7m8b/bYiUWHZ/DR3/yTpUORY6ePPPPcU8dOHcamDjRS42NrN0v94LDfV2uWosEEBZpcrnSukKXMA0nQJiYTroYzX62F7dwSLVUrN25eg8H75kffVvn+77py8SVKvqMlpCHMc6wyC5TNbxPEQP07op381B/3RidId+WJLI19/bbj5urV6zdeOHv/sfd92xn0wFTH6JXHaNUrrcbF1y6EBi1/PzRQzSoSFrTCrvD8dKJY3tT2v42h7yyBtRRmtzKve+uifToiRRqaPFxgFeEH7AtfoWQk95CDHL+2UrYct/V9ySDkFqSD8wzh7sJMHSLyZdKX7OxodvvrmFftw86//Kn3snE++Rfn0/F9pWzFA4W2KZ04YGV0JUIXQk/KP6EqP2YPEAUFEwts2EgBRju639EBMcohGdCz990KcVAOAB6QVyvsEq04bJSv6ECJadRoUtj83CBFM0iFf4Lk23NhfrIpBlA4ksOgS6iCXscVbeY7M2PmiGkS9WUPAKncJBlRu13Tx2LxaUGzmlIuJ7SxhtM0OBziJtQnEVCEDo6BJvRFdMGQK8VnS+agw7RNBzRH7Ck2qNuFLZbGYXZ4O6ofPo0Dl6lFKG6GTcrWEdKXtDr2tiokdSI8w7M8efTQ/hNzMwt+d3DUHWW3814HKrGQw4eCSg7LkmNGJkkoi8xishxY+Tw2fMqp5NIvFeulQq1UpIoCaTQoVkHIHOZcTGegZMxX4A16rV5JpcuaGJJlJF3mhX9gOMbCe4bgZOI5idAZ4SFGiuaRLVB/x7ffZYu3bREqMxRt9nxrWBo5Ww6PNJo+amGGfRC5TqdOCiyPk0oQUdu4onVR1yUiMYdETMGSdKo1OAciYKjRi+c9FJRplZ+hKWxukgWhPIaysgL0SnwSk8Z0qyaPzgyjQGJMXAxFgFmbPXAxPxrNNu9kwXglPdA9ekuvR0BJve5DSTAmHTHhKuGIn+h1ERz1UbBNN0WA+MLacsgwChCb82/8EKCz5IC4ASozf99wh7n+jU+ahsx1dh4GJOlWmX7Bjz4FqFw2c6bFMSDKkui6hqhBQBnNIjE8nbCI6DnRP6PmBaRROxupF3pCNV/qq6FdauN7JU0pEjC2FFmFcbYakHYD8diNcAQwonzCtANwaHrZW0Yg1h7FPZLMa6yw9KPM0Z3pl8SApgJ0wqu0SgAZO0k7l0wxyDcthxsjPf5T4269SqvRZKRTbwrhsKU0bxqPnhvZ22SYrNXApKS7Ys8iqhJ3WBlUSJCmXWoESiMB6J08CuvJNmPLMi24CsDaykvI52NtFxentnczxWLr8IEFTBdbmxupZJJsqplyaWtzNZWMfvHLjyvhpcedmkiggtbwHIoPpgO8KODx1qpQcUKle37KzHjFE1C1Br42u5P1jAUqwZB8U6liCMMBKwBWofwDpJaLHBo/C0PHJUyJq9zj0fnKxtImUEwBGxCaKvbJyBRm3UGSXKAJyRnC18KGcqGxkBjf6CGv45P7eRfFLzSNe4vCqQ7gmIM5Ad9xD/PJ8zwiV7yBfXEhns2VibdeX18/cuQEmZXIwPOffuc/zuybrZTzyVTs5soK/Mfv/Mff+tAP/cTph/alU/GLF187cuQAKPC5Z57D+ZySydnM1pl77/71X/vt1EIwFguheqUmHsBxY2Pt7AN3Hzo+E09xPVJtFLd31mdnZ8B1+UJ2dz2LF0s73C1Ui2hf62Tcy45zlS1fxIWCBhodCDsDA3tyagI0Np9OjPvNUmUraHOmp6Jb2yH49EGDmljOVrcZaZCqwE58mpcEu15v2JXg7Rubq5T0KRdX44n5QXeU2dpJJUYvPPWEx9ePxgJn7rq70cQPpurp+vadPDAqF/NX65s7GQ8OnpS49pTY5rBYZlta21DTKDD9hxvbXOQnsDoIDcWcHGVEBSCJUn5g2gqiGrH1qxW8LhqTRMFNxB0+f7/bIDCJAnhwXBAHlURGGSicVrE7d6rVUsTf/+Gf+X5UdB/9D187c+pQJZehqh6US30wYAMaYPERTuG2oTYsLlwcYIPEBLOiKvSWOwIJKSEnQDZ2BZcjhKOlyV/EhtOrZfGQ3b6DsdIEU9GW9iKkGOBxUX5YcrxshGAAbXqusmeVR0D/gae4X6IlPwGw7HlBo+go8Ak65hlcdrjwfz56F5c4LBjlJuvApxFbKQdNyCLoxloMG09aU8XzcWjAEgJ5rdQ4bDM/YWImZZ14Xzh+8o+AXqBFxkUT5CavZsluqOtQwasdhm/awv2I6dM+gYwIx+NQPEIZ4XKMoB9+58jTqvXnp5YO7Ts6MzUfcAdEVGgcRkdzTffZpYxOal0qbrBvcaiSeds+pPAw9YDBgZVSAeBTQQbjpQYYmAQZdIf3SZRAJYgBC68nzA80zUShkyWvhtKLuL2IGTi4U7kULgDLas81pE5RY1xs24r2UCs9H1w+NhNfhI/btTkbyOqUkBm5OmNnmxq9aH4knJj5IgeIwqWYArTjwptgUIMPuMKiMUl4WcAzt8ic0MfqiNwGzTZMnW6U1RHtuOFRQF0a+Z7Et7clDP4yMGhoEiPjisE4rBn8JI/rhNkDT8Fuae018XROM4nIay2q0CPYD2wKTmMNSOtpKjSgoQBweKt5RJyiYEv/6bk7B7iVc4MsdcJbb//E8jIVPKbh7J0ARcAyF8RFCJrNYXJ1kixTzJHGwSf9tX7i9Wbvm/1DO2aY/IReXu/iXrC5UkvyE5wZmx/qS0VW4otUigKiS9oJDN71OlVaMVsiDbO+yLhYd3C5cGLxxYW9QcEivQk6QaQPzBzwwV2jYChCrwJByCG5G3dnJmcU1NfuYsJgU2HOgEeWNRSYx8Du95YqtXbXFk9Eu8Nxud7AFTkUjew/cgi3SPIjkEi2XKtcv3ktSrHyVps0qtSwYgvBX4bwnPQSptJiWpga1DdYL1q9HnWwSevB9mTAKqc9HuMObWxpjAU4HcB7Eh0QisRwB2Vo16/fnJ2bo8LLoNWiPkuhUJDW2OMJBIMbW9tM/Pf/0A8Qy/uVr3ylWC6TA6hCcLD0YyJ10UCIt+QytYW5JGwnchXJ69uqbtMn+weJdgPRMHBKUGYhXyIzJ8uEHA92QNonAzUaKSEHKraL0QdX4ulGHxUMAnhYfRYJFgwgGgkFQFyZP5QHXOReIFTMv8kZQOPmMNtTnKBkCPY7aNUCGvMW7tV24EE4NHQPtM89XEFRJ+Q4Jrs1mbyADFFu+ml09aB9B+X28MYDUbFNFHZBOQW4K6qx+T0UfNze3bB5bOnp6KFjB9/7ze/+w//6h7F0MpfL/dzP/NzH/+Ljr7/yaiKWhF1mX4Ezif5oU1oFvVzQS+gGXB/ufdP7ZnYL24cPL0ai0Jb+vv1zuE0BYd1WPxRIPvW1cx/8zu89eeJMsVglJGx+cWJgr21sXssVd9uD9tzi1FZmu9qqGauci7iPaUoxLO0D6cEikExqMjpz74EHv/z5Lzu9AG9z6eBcJpeNxdPkBI+E0iSzvHrlVrFQW5rbv7m5ixN10DOOeGo2ygH266mp0Ny+iVNnjteqzYAnPBM/uHlp80jygL/V/9on/mJUWBvXOo/eHZpOILzinbR3MKVipsxx+9qb/iI+CZ+w7mAu5poVhypJ3QHfhD5JVIg4FVbESbrssHMOrSHm7yIRMxQw9vn9sKZ4Q0LB7f5As9Ly+aZt4+lucyYQPPHkF67+3E9+5u7D0xsruz7STgHz5s2ojgm277YGtTqpIcj+Fht0SS5GUTz8dLok/AxHcCEkP3GNmhPAKaodeD34DvAaygO0F/DRAlRQAcGIXg8GwzYqcTYJHiFosxkBjpWoxUS9BJYwRsASB++3FC5CTuYAPg3ZFTRycA1445ObeZBzZcKyfuBTYMubzQFYc1hgzd1QX+sF0sozfRbN13ZhvxDLojB2nPd4nHsNZif+kZ1EqAXoTyk86QmcCOIHfyAfUaUd0YEtk/oOljO3XmSwnW2IAEb6r4B96LP3veO++z2PvT0WSCTCSbJw4BrEUyy8/E8xm4EkmAjJKDh/4gLSpZlYJOQOBQnm6Q9apXyeYDgESjRhmjk4VSFvRmFJUuBoOq12TGwIVgJaMtKWnYpRIbJOlit5drI35CMxCjaG5pjgjHallx8EWwtHYkfuPeKYdQ5am+uFV4g0sDvRrgtd49hM+CjnUFpsqyyc0IfmH5GWdWQZ4MiIR2IKGUgXlg/SqzJeOAKZGBdRRXg7I/6atbQIlbovgJBIAUUWrUSbqgUyJFlqSUPz+FUrJKIFVRPw0y3sxJo7vonr0O4B3+kuPQytNkBAwywT/eWTHoOj0MuILWL9TOirJEw9Je5LcCXqKHT4jQdtccG0Zf0gGLUOfrkDptYVWuRd6onIp64xbt4mYBPwC4j1cnOfmFyzhnwyEH5h1AYZcCvf4RpYVeN1JV4NvM/KE6+H6heqCGMOARbFxeoNaLMJCOrtUVKW4L+hA3MvzDegqTzKAIl8iKEo8hFAAvYGgqo94HCdOH1qcWn/8+devHjhUjgQQuGC/4bl0cEegH9iQrFQRYKe+flZ6G4uX7WThjeaROx69C3vfPv73vlf/ui/HDx66PQDZ5f2LT717NN/84lPsNdJ2YLemorP2InBQkr1jX+KXVltyZcCXYPGk6kZfAHRx/EYPh1GQ4WDBwpeZLlg+BPp1NLy/ldfu2i312am5w8cPXbhwiWq3MdRb1ZqgFu/2aqVm2QBScWjZA/4m898cnHfMmQ9nYz5Q8ggQgto6zwe/41rO+mYLzWhwtVQVvyiIZ+JaAzChuqUHYtfJGXMm4RwOez+cATeEZhEqYWrYKXejIVtGJJJygHIsU4gBzADd6J7gu+WoQ1u2khF4oMBC0GSNAv8ZSHZLsAVqMrCS1wWLKIUhQ+Vo6gAlEPikrnJAjk6b4EXUwTJBYA4DGQbcVk5pS28R0cVpsL0ou4CWZG2RMotXCvEUvIY2QvYgaTGrM+QympmMlvO5vPVeA6Xs8bpU8dW1re+7Vu+FR6IFPHkPKDgT7PWjMcJIqphgjcgzEDE8oKaGTD3UxpmkjqCCxNrK5dffvGVSDCADykhjc9cO+/zhq9cu1lvDY+fOt0YdF547ZVqY4PkwVgpJqbS7Lf5hdn+uipYYI+YmApTBfzy5ZfFZ4WIAgmReGJ3p/zOd37gj//kP8Yng6+8fBkAwlBAwHG9eisSmvym935nKjYdcobXplfrpdaD9x7Lb75y8+qroM5x39trOV54/lW2wuzsXL5Mqt1A0zXe3N2aWFyYPZQOjzPebkZhjWYyNaEcWqv/xwO5kzWEALCzxaOzKFpe6DDkR+vLP7INwT4SONIqN31EYDrcOPpQwA5sRppd8Dq8F6rhURselFoLTVLdOPpumzdz5sHJD//qo7/280+/861HX375CpveQ8g6HkA+D36FbFmPm1h54IcdjLIErRaggvyEmpMMzxieukYTTqsyN5oAXZwtBGoECEIZocMO+HWQkOiVABFQhDFkLKLxQq3SvAglyRlTejoOi0qKXAve+F8j16cBSD6EgzkEtqBdUj4Y4LaeZIJoy6K77DG+At9sGBq12uU2ytQL35uNJIxndgdtyP5CFiWDz9GVaCdJC497LbEQcnynbwhOQKIYB6eDXKlgE7YxvSBfPIFRrIrTTgGRYSScIJlqKd+AoY6FJu6764G7T5+l8h4pbcGL8kUCU5ocY1qdYBiii7JWsTjMlNNJuBvJOQk4cuEWTtstPBazWKkQS2B2CPcBFESpVIevKx95UICDRBPav/SZ2SLzFZEXUl06bLu1hgd3rwm/04uJsFZsb5fbueaw7E+5jj2wPH32uC1Klu7VWiYTTjkX75nvFDKwBVAs5h3qoWXSLGlBmAODVQR7ACaYlDfLYKvMkQyKwjhgNP4H08vMQBsQYPrK06I/rL8+zaqaNdUKc2ItsKihPMFoVXDO46KeAnoOFlHES3IGbKC2jFQI0pmapRSFFgmjJeiGwIPlpA31lHt1xuSwcXTJ+kG/CKz0sw5zn3UqnKkOcIjvvXPsjcE8RBd1WC3oVrpurqgf6ozgVcTYoFF+0q9mvqxrAAtj0WVukZjDqDUiq0NqAXpMm+x0MbQKsYcAS+esTOw4SeJCjmwlkZfQQNSD3CBTF3nYRDbN4vEcSmdYFi84gMAOMMLQHvUHm128oEhW6nz99dcvX7pGXjpiLskBSXeYJXrCjlFvzErFo+HNnTqmVmxvjzz00PFTZygo/OQzz167cok80cV8LjkRv37r8j1n7zl17Gj0e79n9eKl1ctXc5k8i2XHG4GDfUWOcXlUDb1+WASVAJMi2qdJgFeDj2NLMR0MH69BfwCxNjgzO8ujsMXUl4W+3nPmbDgcq5bL1964GAuTvSDABk9OREHZzUYdgsNeIQzJSvRBSfmwCTXG+Y8WwhF8EjDFOVoNPI9IuQvGkmaBzBs1SulRhoSE1eQeGo2W5he2NnaVz89lT8QimMaNIgpjjfCMxqI5BBTN0glE4Y30HyutSxbQ6IyByAmSMZkpNfBp7tFXIMEIEEApyIpWocdc4ScDn0KCHJybFkxzb/rgOg+ibuYe67Vi+yxsSgvyrMMoJl5ZSEugB/2QO1un2fJTKd7vRZtBLu61W6tUfqyUGycOH93d2sGDKezyo3V347AzttfxfUI3yrh5tSBPHAk1xmIhtyuobiO54k65tbFZ6jSGgyLsIL5R5H/44he/mCu0vvt7v/2d73znE1+9iRoil2vJFc4W2txYT08mZ6f3k66g1czhuNBqyPyJdqFM8aAe+RcnsLP4fbED+05euPry9EKKwJBCqTSRSB7eN5WITW+s3cq5sy3Cd0a97/ve98f99v/2xseq/bzbFdncKY08QWQ7bySwtV0+c3Jh7ES7c65b3p4KdsLBYdpVtVGPi1xBhit/04zqlFm1rtCfv/eTJITbh/Urs22ERi24loyvpJYjgZKd4CtHbDZpi8Yd0pMwOpgstgLh98qiRGE4h73BXA66W5GJ5Q/8s0fdQ/cvf/grJ09AOFybG23cqOemE9VqFgRmdgSSWoUtAs+N+MD+h+RJQIOOSkEMvRCCYsPSa/4JQZJYlpz+bg9em/QWyOi7QYoGA4MWTCwIjUGuwKTSYdKwqLLgTSjqTcfe6DRCTcGdmQEsOQyQ4uR/G0w54RLbg4OfUTQBppwAK9atakZUnh5LnKJFntDbzUXJkPLfRJrTUGAm3TI6edpNkkvwlYf2uFwIMIc/kpBOGwcjshtQuZl8Z2TzdFH/2FPeqfe79YX5A+995H6qfDDn1Uwt6I3QMfGU7DQ0xESPs0uIdBDhUc9kMZIbCA4IeGHZioVMg/Jx3SaOhQTzBlBigFqZerlF0wBLINYbZxijkR6hVWMukTVIdonwLb068jKVw+PBoadVHpdbjWxnlPOEOlNLrvBEcurMspI22y4jTHninaQHLTMB93lMFuqNJGBQMJ+aeFFhkVxAk42tfS0aLZkM6yMZrCT4cmCE4Eb6JwcRs+TSy0INYCwEqZwg7Fjzb2ADishLdIH1hHAys0q+IgZJ8KBDhUKERPD5Fk0ShMjVyrCGYj10GL2umRUgCU3O3hvg0lhQFliAKGRJPwxRMV1j/c3YNCgOfVpnZphfR6bmMXPLN35YhNZqhHfoFYIrnjavE/VlNmDN6TEvVtusj7p7h0Jz0ZyznyT+mglmElAHSIMhsisazIDAfzg540JkzDzEGuF7hcelslkhwvWGThN0pIg+uT3rk04YRzf5/GshYWmwqZI4jdhydKpU8UumJiBYr79xiZDX5aU5g83VY6CKRQIY5Q837JHRgpInE5Mz7AhSLm/t5NgMuUKRmjcw0u9777ufef6ZI8cPHzmw/8r1K3gCvuft73iaPrRf7ZEkvgOb0AVLGJLhpKfMv7ak2z1SnnkREjYcK2OUsNrL3IviDhJ85u57/9uf/nco6NT0HImrLl6+NDk5/b3f+z2/8xv/Ibe7wwTBfwxtlXK5F4mIIeYKXAUNYvtcXFxE2EVNTYGXZrt34u7Tm7fWSMToQyelxRh7gn7VLkRxhz681aq1moQl1BsNct7iLCbTL3vbNialE3wt4rtU0PJ20SrSSYOOwHpYXOF9zbqZPXwHQLgDtGNu1HqjIDY3A7fIUy4YKFYWCLbmnIsiqIITHXfupAUOCDlEnvXTFjH3qFnTPneC2q1GeAvbUr52LJ4MNHQL2AIfqgNgZjKL5TLZQIcSpQES3OHVc/3atdP+u06fPJnLZEi+kYwnrly4zF4hiiiXy8q1hoz30Awy2bI3IbAoYx1OQs7Y56s31pfm56j3htMfg8l1ypFwcj2TXb3aTM46k+nQK+dfDseCZKUOh4PFSiUamyJ2cnHh+Mr6TWI4CsUSwA5Y0McExDkUZE2B8URicncn6+hF5xYO1lvNBx+5JzkZJU8Z2UCp4valL37xzOmzM8tp2wiSP1hffcN/bOaRd9wTDD6Uiiy/9NLlN9641mg2oun4/PzEGGHSM9oq3IyM651eue3olvpd/4Aqgnf2uTXfe59MqebJrC+f1jlIFjMf53u/akotbKj9JRRn7iTsDWcaeDpS0dULVCloR4Yhm3g9NrxKoID1/WRvtMMo44lawubQG+wOaji0tr75Bx9rdyv/8bdfJbwrEmXWI7u5asAf7XTrglReAhKgV6wup7AzEHRMaaJjskkDFdo0aEoN3yy+EnWZgzQVUnvxILHzAhEkPMmUNjxmBRKgbm0EdrswPG+BjoM2UdOAiqyxv3l2uPLmi7wUsKRVuDqSdskWa00QN9EowM6EoYnBlMoi8Cv3cS8NA5XwEOqyRDsoBQwijwCy0igZRkAEUc3BRConJ8yDwmy0Q273CATCP56jKQzKJGANBbED2HvtLkUV8SI7tHDi6OETE+lpB1Xpiwi77pg/jiemaBDzCAITX8oJcs2wYdL3BAj4R+jF4RzppFauUnqmVR/22mBAKJ6KmPObNjyjMDYJtho0Uv1GeEA5DnaTZkwUhgRiKE2YZxCevdPyNKvDbGeQ94Ybk/OehQNJ31LAlqIUzg2bd2DzQUh7Axs2gi5iNCoszPA8TMvQWIkloovAoZFJmSzWWUChvY/EhT4EQQwsACbiCj8KU4iT4C8TyRjpopk4njWQz0qLMtEiFIZTvclMBssPnoZRpA2hFZgJKNHepBskp+c1RLpgiJxWCf3abdjQbEgAYGnVFusunCQ/AR1qjSZ5G43QO72a/xHOrFeYXmpZOKweq1tMhdmV5lTdpr8WKPJHP5mNqHas2zS0PZRsrvDVcBLmXbRONyx2Ye9NAjltZW5mU+gcFwApnOVshUcP90uiZV9J/4zGBw9no0dG9sVGwsp1iEGCbVR1I5FeSHKrOyLLFeILimgWAxKB4hmmG4Qg5c3YEQlFmQTUMHOImTPzm5vbFMwhOoWxMEsspBmmLEMoa+AIAaSdre30xMyRQwSK2DZ2dtEt19vNz/7Npz78kQ9HY+958qmvnH/lXK1Zf8/b3pGHYpcr8AUeJ5QDj3QME2RocQQpSYOiHN8+P6kAdLBzGT6EgS3PYtE33g4gtciK2R1+4mMfI0o4GHAR15uamISWk6+K5E37Dx/a2tkkQiWdiJFvEh4+nIgh+CIlkMQuHArhyYF4R2oOxvjAAw9cvH4jmkwgHNeaDZiRSqOeTiaBC+zLUA5aaPVxxfLTH+g3Y59MzxAb00c6E9IYYOeECsJc0lugiBUEVxnGDj2vJFdyVJvR7H0I1ABCI9TqXCKAlM8oCRkdN+FaYq4LGYgOUxURfOxyyc1c2EkbiWbNhBA0LeDFY9Hska+/gl8NRLFgumhxfFzhULeYULN9aBx7rvYHDineID4twwoecLZIPO4LRig2tb25820feD8JO0nmTADQbqY2PREAwtGNC5FAy/GwQ2sHUFAIWfBrR1mNiIlosHp9rVItlHI4SPsoxVatd1PJ6YlHXfAxB4/uj8bDb1x6Dde2YrkRjqRvrmzh3PKhD/3gjdXNV1++ePeZE2DoZCqSz27hKbW7catSLSdiYWcvOBNYmpqa3djpfOu3fDCZwu/Utxw++uLNF5/+2tNT6am7Th6G2HTqg6klKl9lv/rU39h9lUQyORM58oGzH9h34MalK9eurV5pXts5dmjfZITk04u5iy9E3GhCelMTtn7ZZv+6/VfTdedgnq1zZuvOxTsnZncKG/A/m5StbyEJFhCNP8srWiIKabO3bJWtxrjpjvI+KlrYmigGCV1pdQZeYpGFxvv+AA7MNYwaGGfRWn/wx9+DNPbrv/bS4cVkq06+mmS5WKLmkgvuxtBFj9EM4XEFI8fSYDCG/gEmmGzorcCRRSVTOhCLZYKgYeFNPKjltYCiBL+tHiy6UpiL7PmMKzIYgs5A6kBYgmDBNaCGszcAIwC2DqFcMwuMGjDWI/qZzYorDS9RRR0dTJ/g2By8la+QXrRPHNZXs7eBTevgBjC0gFczCzq2j6C29EY9MTiIM7aHhA6MZexXQBHABKOB0iClY2et0vK4A7FATMrAOl6aQ5KNJxKxe44/FPElYOFtfRypGA7VFfwhbxBJGsLE2OU6BRbW1mDtRjGKaHCoIAJa3Fa3zcKAv5p8JYBG1Uwt+iaOnGVmcyPnIKrTECsuvTnTT+pRJhjMbyfpr6PfGddwt8JY0LTX86NceMZ1YF98djntJhtrsG5z7NiGVZuftJ+m+oZUSzjrefDcYPPTPtMsnYT+6aUG4jBmYLiXBUIuzUoqoiANvrI5mUSWEV04n1AT0SURb7Wyt3o0RI+1kppCTlgOrYTmk6/ca0guSFjD4sW6YoiXSkQIVCw7DNAAmbp9MAGaOa2LUDpGKg6uMA8aAuHc/EaLNOagjAUpLSV+cztgRJOGzNB3zvl/749pW+dWz/XDmzYkJIoWrAuGB+B3iRoahIgxj1kngmDTlF4mqqs3qqnbB+fAsuE2+bSm0BJ5mUhEXoBPEyBKDJRgTpHyqQ8riewr6y8WX+gXREHQgCO7AvaV4QajbxunaJkNIcqwjyYfN75pxH2yhQh6IGKt0y/mSqu3NkjiODM7TxJ8fA+Jv2T2AC62MUCGrVQMC4MY4nofhLtirXEtzuQL29lcDI/iUvHQ8cN//clPdEdd4kwi8TAuCkhel167sL66Vq+0AyRMUP8HJGoFQ3NA0UHq0UioXK1j4MJ/QiBB6QzDJkmMlUXKWlTngYOHS9Xa5tYOiSNIWUU/5+YiExMTr7z8YmIijUhKvs3J2ZkRqZ+2S9h8EwmlWsAUAumlzgI5iicn48ePH/fF4rfW1rmTBPfZnd1GsRMkUyN5Lm3hfolpJpGk2HnYESQ/DjyH6RW+Y8i+EBvwGSBFy6j7gUxODYzhswFLzs7QTxz02uo44MfjrDLTyHVuAy8JSg1kMnxMBdad3GMe1bMCWvY5s2w+2R1c5JwTqx2+cli7RijECbbdY07NL6YlYQRuEtI2EK4NzEbAwYQH0KVHIj6y+DUarXG2j82eotmo9C9dupSeZF4nyAUWitiCWp2qO6CSkYJdMXYSq2hQyAs/9ICXmoZo2Ha3Mo1GlXfCXyElkNz0x3/8Q6gTvvK1rzDa9GTK4SMVeWd2bunSleus/sFDh6/fWI2EU8eP3QU7A3Cs3WSyb0g0HLYg4m995OGj+x+ctj3ot00vz88VmrvAPKGZUVuwU+98/3d/T7lcuvTGq/NzU71e5cLr16dnEmQ56zmcKLRfvvF8IrKZz9ZOnjqy/8g+1ndtbateDz76wHdkw3OH/M3t1z5XhzHctc3ErFn/Rz6ZPOsqM885n3/vJnOFWZEQonWGcMBBQfBYCgMIbGSfx96qjRv9OipPKhxp1uxkoAp0ezXqB8POcgyGFayH+GUIyzg9tULhAz/89vmFxV/++U+kYzNkE0ZVBZ6VBk+ihEWAeAO7E4FVmkLxbLhWY5ZC3WvkQ2g/3XMSLgd+6CEZNw0UgTcxfCp0mABeukdrxlhJ+WEwhzh+IS4GCsigNoFosWcBSOQn3sv43vQJ1AEC0C2u85gwq9hmdj0H9+HzRz0Qo8wRZ4rp25B3gbEmjDu0v7UZODGbCr4O4BI9QWHh91kXcSITYpSkAvlEmhRcixWEOmgSECJQHLsnY0m2X7feb1MxrWlPJGaOHjq6f/6wdxQdD6gBLhUwidmwGqOarVXxZaHwnwR+dUJ27DESApWtcXJGyUxlxlaTwPkWnoYwnSSMhtBYgjbqfbyJQccKaVW2Kc60bmbNPawIfA2LgLnY5uqNyAs0KnfHRZu3F0hRMHmUnHVH5rwTM3ZbqGYbZLuDAqm0nOT3lZYa10w3pnJSKY0p6+LFr57QtqZypWqGRX2RSXg380HCBXHm+I6C3OVhj9+B4I+dyYxoWDDfhgBLu8EKWdAsRh+wNuBpABqxQWKlREcWX+DIkusPKEhJmcyzalhUyUw9PxjapWX/ekt0SWZ72boAFrUgjb5yQssBnxcIeYjiKriON8A0CaAEAdB9g+B0E52lu2IF1Ye9Dag/hppqfNZVpkOnNCoOQBBHd61rpnvmJ0NT/8FPulM917sE6wA9w2DDcPAWrjBqwSELKWpq3KxQQ0A0RYyVswaKCyVGgpHaWYltCPZVCg6F/NqceF2RZwBpGNcKG0obJmOMjwIXIMIMQa+DBYB4K/dJr8OEcESjJGQobGxmESvxhIICBXwqEAIOBfmSh5ZzpBYtPYVAKMvT7e1s7eI7EyShlW10YGlxfeXmyo3hPQ/c7XHYayV8j8M3rlz1wj2Q6JLhwO4KChSqiFGXTYhjBMrwiamZdm8L2MsWsHcQGd/DZwS/eMCEu6HWkRhpW2N493hDiKlhzL23Vtf2HziEMvNP/uSPKfeL9htBlgIkkIoQrtJhnCjZWuLhQGpQU3qIabmYK5OUY9+JE0889dTb3vrWnZ0dckEj3a7f2gqHvOVqRXVUlVteZLtSr4UiYXqLCrperUH2AoSdEmvZanLOEmqxtHZGoQEgmUMAap0BGXcOc84j3GwdggCDNQl5kpXGUFme4yLn7Co6z+ftliwaL30AHWMRrDv55Gb6YKBMvtNcoTe6xHrQgqAHxk095Cc9xtZiirFGuxzdHrXkvM0m3j3OYpEY7h2CiOjB333h8WOnTsI5rW2tAT7lpmoo+PyyIin4A/rCRqWrbG+DN7GPNSt1fF4aDcoGg3L9tWrb7hxMTs5sbe4WquW3v+vd2RwFLlbRtWdzmXK1+c53vx0VBbzOo0ceuRC/8LnPfoZQqAcfePDyGy9TojyVCLbqBVIqk4WtWioeSLjWqlfmo7OF3E40Gmk4q9tY+HqdSrGT3d2cX5i+cvn1cNRz6MD8Sy8/1+pV3WHnwcMHDiwsvvbqBWAmXzi5fPDk2BH+tvt+aKtWGI4n+v36q9ee9dUnY5HB4SP2WqbGxPzDw5o0TeY3HtYVS3qwdrF1BwiQKWZxtCJmF4s5Q8ED80+a9t64kisFyKcRJUIIKZHQXx957zEDg42k00dG5pWUnO13vNHpfvfyPe869VuRf/WLP/2fKXvtcIVHLRYH3lQSkd6rJeUxchpRkkAIw0tiSkyVNqV5oCWYblwWEDvZ4NxCRgBJR9TI8gShIz4oOiFe8nwWSIBYYPBZVhlcJfNqFHqLoXvWBAA/wBEADhLljOUjmlPMqMygCGuK60V0kH0XyGPIMNCuoYtz9h7Bp9QbYnuMkdH5BMIBJ1qSkC1aKv6F97BveTv0jR2C7xj7ihtAsopfsRAkMTwUdKLP5HwS6cV+5CK1Chk1O+NmjbANx/6ZheXFAyicgfZui2wVAyIzSKHOkNjhLAUNKjuGbDmM07wOgZuSkgEfJSCquQxOWBS2hPricSUzAjjPPgy4yVWJ2Ky+4cIKqNMgYjFxbJo9IVYRaQgVUhAmmrat7iDy011pjHaHrmJswjV/aDJ8IGlL9m2u2nBcsDnaY1/L7SROSTKiUVQxsxAojM4yEzKTREaS98WEjWlTQ/rpMd3hs9loYF9WQQrLr0pIXctm5oy5h+sVD2Woi8CFE2abg7nGqigKaQ5pTrXOPMuhS7TOJ5yTvhscp7+sm1aJmFeYFTFioAC1SUOcS5kipQkHZFzig+ABWUZcFzjt9hvQUIo74ECeBDQkh9P6bQJM99TuP3boNj1r9dOcGwyoRzAMWYcRgHi/GgAv6gbaV3fopSXfi/3TALmHeeRX7hILwIn1CT9lFM5S9nMvo+Kr5F09Z9FjVKH4P6OzJaaFgtNKegV1ZdX7WH8xuLL86CSYEWIOCOZxeQk9a8NpGe8tgTQOrKbYHzG9qHVrDUp7khODaCPqTpJk0YvNkyK7RO8wUfk8WuQcqF9qRzQm/gCeMkRUMCiyEUWi4XgsVigWgWR8koPR0OXX30ikE8jp9LWYycX9EYAAsSZAqTW034YeELDBWLQ62PwSiUKxHLAHy7UqVVC7eKuykbGBdtHBjogF8nupWp188MEH19Y3tzO7o/yYoOGvPfnU0aOHZd89sHTu3LlHHnt0enr6D/+vP7377kNYMf/TH/zZ3LRstJAl6KWEVvkwyy3Lu71dyJXYkNdvrr7vPe+kHt/Kyv8X1TthNjeuX2dQiKjxVNJRrfIsGun3f9t3vvDCC+trt6gBwcbnBlYc6RkUw1jAI2xpa1zAsbXQ3CCgEDrRAZbhOmwHHQFt8RQgCvDRPgcV1VHLQ3MBAGRj7pfzhAq+7rUAGHNwnRfRDhKneYtov7miE7VvwqIg22ofjGj4D0PZtV8YPjvMEhyEtqgAEwijsWy2h7PpRL1V2txsscQp9ySpf26ur9ZxdWs0KERHZEs4pbAl+q1CZvKiFuOI2g4VJiAL0JLCHWBu1iHOQ6AN7EtOUGwiz5576V3ve9fBI4dffPU5tDaoOo+eOFxvdIrlrTP33Hvt2rWN3k1yXb3r3e/4wuOfe/Xll4uFXZJ1D9tNyrlNTyXbje7j5z7/xOi16dR+LcT731+t8reCPrZaq+RyWIJdKzcrKK6pX/7cuWfy+Yw/DEHzXXvjprfvalYyM2mS31ftjgYhc1eqK6nQQbtzenb54adfePEDD3/L7hsfu7W+kcAPwiAcTfebDibcmmdOuGyd60S45h9BEUCCBDJkPPMcy2fIC7EjozD1PEajWrlO8buoLR7yBvC19fgipSq8ry0aU4kjUJRBLThF96GR5cp1/3B4+P5Hfv5X//mv/uJfYXwk2QiLDE4FbQxdxNbzHikCMSLDSUO/nc4QUA2osPxQPfhOei4dsiABqiFsCTzwPEypjWLDkCLCEyF+1OkSYaV8rIJGrZIPQh5AHWliG4QnaIKYBuywZlxSMmIXQ3SGEinpCjKYQiQAhwH2aDtYB+IPdEB3kYMxycYDKdyChBINqte+lPTKd5ETugVrT4/ltqyAOfYX9FFmdLYCk07oIf8g2eJNySXl9PndYfAawUUMDTROLC3JPufSixQxiwSipMIY4PCOVGkPYmmj+a7QIdV98eNintm18qcCJePYRew/EU9Q/Uaz2spvkUVSNygrI9RClX/Y9OweEghgvmTARvktRa/wtwPZGB8cEpqDgRHb8fmzDyh45Gq2hvg2ZzFyxZbsi4cnUos+WxiJaH3gaQxcEHUCwYTqGR1TTUccdh8yCQkVwPi8B5ogxgprHPDJ3GtVFQxNxmYKsiqQV9TDuCQYmsXtQh6cs1YSfEVYOBdnA6iYFNZcEbEx2IczUBmfAABtW9DH8/xqkIiQF79asK6G+QoJhphJYUDDWjb2gb4AOOL1wegsKWgfWGAVmX0EYBtsIRp65Dl1C+IqVKyeCiEaox3gicu+xig7uaKkee83HGa/me6YF6pfjMHan3yqq7rG89ZYbiNia7AWWdUY9Zhu1wmzq0Z0hW6ZE7FRLAnnqFVEjiHemJSAWBJuaIAINWirRG7lfkXeDPTMSv+ILMt1HsfwjzcHmucxtbtRSQJ/HruKF/GFcJoxRbipiESL0lJoLLRYqdboG3mYcdJskTTDDZVx7l/eh4WVaF32Dr1FVKUFRsnGIqgsRAJSN5x1LxiEzkoRDSmgLeCm02hGg6EmSVfGQ8yloOlqqUw8ETYN4LbZajP5bg8Jatts0p2trWgMD88q9ubrN2/gCHbq6PHrr72OJZFMVFBE9iTZrAj5hfSa/Biw0YiMw1aDrM5J3CCoo02vvu8HfwANOerLg0fm6S1i7v4Dk+wQS1Sl0kPP+CVMpINsos7KLfbQlStXfuiHvg8vpMcffxxEQOwv3BsgRfeQiekPaTrIkZlKJLmTK3SGjFqUOOR1YAamCGkY3YA4ea2maCGsBef8ShUZQI9HupSC6w99wiVSsHMPc889SPhgL6adi5RfjBF8TIGnHlGFriZzRNqbiAt/NULV6Q/aBzKmEE/FSyUXE8pvcCuQIwIul1I4e1h7SboKPZJgqkQrMv8hcknjzRZHJMEbAHCR7BHyhQhVwbTuCykYjG0hPtbufOONncicPRgL0UkX1XJsfb/XlS92w2FEJTvq6FKlHoiiH/GQAhoXvPd/67d9/q8/t3mzPr3sm52I1NoN+hOJxOgAsEZ1SOKqG83Wpz79xbvvW/b6HJVmJT0Rv3Ljld6oBvN06corluPIt3zrezZW17bXV374h374i49/buNWPeCPZ3bLhULpyhs37cOnwkHf+vrl5eV9NE5UGDCD7oYCAySI9nuTjRZpy6QJpFqDs9aeTMef/erXKOs6N5t0OlmC7P59yy+8dn4m7ZibPbpaXa31PeW2xxtZGpVbVHgnNpfx3jmYRg5rw1oX+cqJ9q5mSehH/5uvfOoqHImF140crNu13fVHqGqEMcYeoir5cFgvlAgMDKf9qC4j3uDARggCwoa0htwOA+/09vLVlURszmnLdltP3vuOE7/i+I5f/OlPjYY1my0IQQEzkI2m32uQ31tIH2Rr9N49soxRpAQOCE038y9jRZ9dSc1IutFqKViEzKoIxBjugSI6yQ6DQReRBiu57F5fAOQAAAIm0F3YQq9/TAYj1ggdG1SbpuDQkdYRaUHAhBuYfS1/WB5RAWuoO4ojIJVpYmaYRE4spTSwaCZq7wP0Bj/JTaSxQ9aAC8B7QuIR2lgJT+xG0m8Sr+FuNXv8Q5ML7+/3RvzBuG2ILyDFHuWW5LR5kVsxhcyk570IsGhAkE3JkABqVLIy9KJMkCQwQFPaTsziHYCznUJ15lGtH2hxBfVXq4aKQAo+SvipnDr7Ak24aJdQtW0EFiCtFapAKJQLHKaUQCJs2WIxHPJHU2Gk/XqfolOFeq/SH9fGwWZqyb1wbDYO6Y22bY5yr1/qjKoyVTn37ARsOVDxeER0CCWoqDuF5VnkFvCTZx0Kbn5tNxS/pnjkDoZlcv3CAEPzAC4Ju0CfkXr5KyA1kivXDS7SV60CKgvdZUGwwJVpF3zyJvSZTJOIqbkOhYQwSDwDcszFPfDW3XqeQ2DKA+L+9EdxtJojr+DMmI7hQYVzIF19YsCCbqwUprwD1JnhsgR6CjKvyGvAjnwcEqMBUkMjdfbm4+vcrhFk2WYce9TXus/IBZxakq48GPYmwyK95m38bEnJbBgzc2rl9hV1RzRV84o0JRdXUVNAgLnBE444LuNspaBe6ZmVeQMzsCRgEk/ioIMMBmGGBcGpCS0qs2G3+6QwBBRxjsD/iqkykWR4LFByAAQqaISoyiIbAjXjG4x6hYVPRaPwAdAeiBAbuFIqUyYD5A7S17gtfaZZUMbBFaFtIuXAc1Syg9/BaY/NCLgMBo1xA/F7Ip5q1djNCgJkFiziwaKTtZxoznanOLJf8ZMZAytpp7u6cguVr1zosV5Ti5QcM0TikgLY44Mu4b1ieCxBHaSdN7HG7MpzzzxLuR54bl7KzofskasVHhf3K5musQwRpOD3Q4bwq0KfnEwkVq7fiASVjgOyt7w8i1zFYNNTk0Tm0BNqFJK98l3vetdnP/tZfM0QDbkNkFJaHuFZeCAVQKNlbtZ2AdsYHM0Js8Hs6j9Iowk9gibSODw9j7Cv6YxRAmv14XBpmXaMgkEV1dCyUigCAzNEVmukBun410OSrPtphDZpUK+2GEc5P3CuDpm5gZvQDAEM4CJSJAIBhHZH4hF8uZEAsvkskbj4du7mK2TPqDeqlVrdH7JRU2r+yARdOnL0UKVSuuuuU08++eTG9iZ9y1XxCHKcvOc0WBt2ZOnwgUcee/SRex/6uZ/+mZ0brel9o3gsisQCBp+OxSHtEIMvffmJhx978Ed/4ge2dtdK6EMK+UJlG8MVmcLbnQoYwO3woEHBGAffe++996Jy+77v/oHPfOqTePklw+ljh0/ltp4E1NE5b27frFWL0Wic0g6wQPVq2T7uUKaw0cb8UAd9U1IukUgy3HQS+kYt1kKrVrX77d3d4VoOn9Mja7nXyMA/7427E86ufXjynvtf+dsL8ZCR7wDNbzyslX3zNWt633yFcy6a/f6my6A2roEGYGyAFsgOXyXJCDWM0AVjXSSf0thnB0p8AQd6KwQGBBiJD9BIwmfQNxS6vUarlfEGvGcfmPvf/48P/N6vf6ZTxbLTTMZSBHdheem1q/EE0pcxzrHQaO0Be9TI0pWJKgMhrCPnQAvsCCfIf4TWC1jlCaDwP4KM0c5KLq42iPmGN48S5OB0l/31crkCXwnCEb9OJh3hW6gPEqOP1B9smQZPIEaiEdHOBnfL4oNyVojYOgxe1SRBDEE6vNaizfy6t38gOPihCLxlMwbW+TQ+YzblAMCbF3xH0HgHPEXiGK8Lr/V+wDGSkzOEwoNV1xeJ+KN+T5i6yhhkHcRBEQeEHgv0iowpAoW/qRTcEFosLih/6P10IEaiXDgRRgDpbdSq1ERDuS/V4BDmQlKPpenlQXYRrWCfA5+QcByjM0bA9oCqUvR7HJiIUJei2MvUm4WmrWALdcJTY19qtP/ueVe8T1CvzbHbHBapOIcFGkGn1ihCV2F4QIYIJUqCpRBhScDQXhgCKaTJB2gHhbXQZ3caNbTErKV2PgwXiJwRAVK0Yeb9Dhm+A4igZgOXIjbcaKEFpHpOrYdohDPwOQ2AziTCcoWn9ByzBxGStpOvEnd1qMtQOHPKDaBllOR6miaYLXEEZOKSZRwZVr4hUDQOFLcOXMdlOKFptoOIKMsKKGECgesSjhOfIQmGxnk7CExvMQf9NH3WF3PdIDhDLzUQ87PWZu8QvuM2fd6m0N94hV/3qLW5Druw9whDszyczbKDmqHEMgNj8cWnY4/oys9ZAIlqVg5W5JTDHqx0V9wmtTPqGaoboVVns2tarJkB9OGa+IJfn2ZdY+QTiyz8CdNMDAkOUPjRoBLALlht4LXa4IYDB/dLNGi3je0WgUp6EaaxQWJAYX7eC0SIIqLBY/8w22is2NswNhB51oLq35ilMplcvYmjr2BGXKPDjv8wJAHVdzxGyaY+wq7b5ccTGzR66+bNUWeICpoIKfhxOovOFvLX6pSVhoByTvgXmpexfuioMYIxuAP79jNaNghCIrI7MTPbW1v0De01WxqqzErRf05aWMLR8bbwCPNvrK1HQ+GpqSnwN+prJRIiT+6gT42mQWuArzWDrNclsjMzaOPB83L1GvRJQslRyhcs7CaYMVPKCYCseUA+NrQTboM9C17lQa5zWDCztwTAp7HRalyo89wuhHuiFv3OYLdSoRugJhaCvIICcek2+ablkwKH67zX6HIsiIXy6SdzhR+huwJkqjCRazMcDDhtlJyCkaNco8vuwzZ45r57ieT2BvxfffLLW1lSL3nSk+lSuzLyDTDcYlC8cPlSajJ1be0WidCag54/przzsEf8OqrIpwzD/Kuvnf/AO7750PLBQfcijnXgX6o3gtpJOYv0NRGfmZmf+ehv/9ahI/see/vDOMZn87XpyYlkIkJeQg4mBADAMX97Z21xaWFrdT2bbfVbvXe8/b2f+dSn+13PhfOvr2/YEjHbZMofCsXA8LCJ8FBs4lg8TB4oXzBFVoRcvhwORbt9XFRhH5uKFK9Ri4Cd0JFtxt4pdTd36xdT6SNXSi+85ei9x+9Fq16nBAVuN5rT/4eDxbJW6h/9nZ+s1QRt6k6zucydnIvN54oAg99ACMC/8ttrU/frUDVhe2+ILM5eGzpwO5SPMGsWl3qBJsZs3EED6/Fhm7ju8nsffNdBn/uHPvLTf7q4vDiinvsIEt2enklmMkVyp6IwYzlQglBOjw5gXELa1B4U6Mvua4ALjbQsMQaMhCMMdcYO3AXJQoADAS+ZtBCTXS74TD95KMhnV2/TOwRfbQGGwVuYfwXHOgMR6nhKbSPCbNFZhHgN5fe/4+0WoPMytgGfPMkVmrhzhV4atAReEsaWCpqsLuJZGYy6DW4iOJLcfgQEEiYAgqIaIDlUbUOPox/xOQhe95PNIkjqAfLPu0IopTtNKBY9sXzQ0DEZvG5H+IIhMXINhmLqMvpdkrldjvLOBhlMSGhFSBa4iZlXbBbR19B2S3RiCSFghlbxByoO/yzb31g5GAjCxlMZroBHmq1SrZMbeRvhGcfkvkB6yeNOj22TrHmFvU/tH+UgUxUcMSnG4AHllp+6+DSVeZDmGV04KyTipmo3rXGv0qIucadtI4kHb9VhBF+AC2aFDhvQhfoCXnSST+vgx70z/hihEFygU2Dw9icnTLoF+8h9rI6IqHUBcqxDXNsd6gsKY2L1uMI2uJU3yqEOAiveEbyFiHYnfoiZZ1hgQ7QNWl3mX8Sa4aKSNnIC8ypUCX60kp2BxUVSzH4TZ3X7MMzG3hfOBUjWhhKVNQNWn+/crxss0qstp0NDhJRa53yq4yK6epwDTsD85SsrisgrEy+WB32F9NJ7VCd9zByjDtWNVMgMsie3WbydGy0CCii2CIZFrEKphNSJVwbZHvFadOIhh1fjgImA56WkBqkEA0EYZIwyvBFqz3jF4WEC8gagnLFwZGoyTY8wdtbKJbx/vdTbMyZPKFkLbRTuFSbZIzBroR4ou9aJQwRBQKNzMZV9j1+xXIiw8URMhRnIyMbOghhg1SUtABW/nQ7SL5t7JEoi1mAJ5kWFXB6ZFWKJNG+0sphgtIVpnMUSryDChojdg2AIgztsuWqVKrb0BMdddEjE7BaK+dmp6XJJDCtKJ2aNnHH0EAhGiHb4ApV6Az9f3g454e1bW1vHjh0DP9y4cQOSiFqY6zSInvNt73j7U0+fg9zC+CCGgxbqtQqGap6FJ0XHi0CrWVIVVPgfdgVzyS6zcY8AGKVhMAj0WhI2TzEJXAY5MAN0yQInZoyxcJ3b+ERYZ+oEnFIv74nI1lrQW0xQ/KQO8EJLISHfOtzxNEa9Uw1byhVWhJxYDhIT4EzCXveGidGKYFbnHNCbmZt56KEHeNf161cx88OCVLu16Hys2KpMTKYofrC8vIigPDs3zVA//3dP8dJoUmEmyUR638EDxHSBVLxt519/7FPlekHBblKuxspN4sYGgXD4+OnTh48dvXrjutvPko7n5qdhpLbWLx88sAiAY92A22BcLByWkaA/BOzS38x2jhM2+Bf+7ovUIGzWSHpvi0VhSEL4ujIDjBGkn0rHAuGAP+grVxqENkWiaXI0UoQrFBzldtca2UyK1JguNK9175St43ENQ4le13/z9V1nsX82lV60d2dG1Xed3GcrbnuVjeHvHwyTeeSwfjD7VBCOeMInX+9ct07e/NV60HxK0ERxBZJAFcWjhkVWMi1XwOaLukOJgCMM7WFboKQkqpfE/yNMwiyQywes+DA+9Nop+2jZazue2/L9b//q11zjqN/tAzV02mWZ2tDeuv0AA6QKLwQERV7GmiLdCkJk8ZVKhnk2PXSQSAKaSf9J9oTmUIkd+Wqz+bAcNdgfI4/P7wtEIBDFAp4BRRIbBgJuwJjhsCVJSgNRZWcBsbu7uU4HPTYIXP/QMSFNu/yhIIsE4PJKIJXV4pwrQDNXOPbmlLgYYxehlw50AWwKdrqoJihSjsC8hVypTAVRtyRUC/rD1OGlyITPH/e7KHka9LtI7epRGuYOhuO+3x0A+9APq33IKb2gFoDXT9BZl3YjkXAgEsO428xn87ld2ah6LVJGgZ7kmoh/NHyA0LPitkR6efXt1iDhcDEdIkogjb6hm/xBVDfokLen0hvXAjF7cjo4s28quuC3JYnlraFw7vcKfWdr5BoooghVoN0PgqNIO7lE2JYwHnxqr6LQZeFpD6sweA4HaDFSZA6pCV13bSTrgn+A0BpufI/uwqtZ+NYsqgjNHjyqaYiPcAA/GbqMIGbOLWRtyJie4hHJnICy/An0mEHmhjKqa+YZfYBVaAi6r5s5JdAXRgEaxkt4M0+JzxRHgMsAB4OySBsnxEsLTdEPBkEXJe8L8wEVHNwAbFjgwSdAr/mwXqM/6qQ59GrTfeg4Hda6WBtQ9FaP6Aa6oZ+M3GlIrHVFG9m6qFuAVLOBLVmINvlnFDgMjMUXKeV+KCvAo6A05FmplyHGhiQr9AjUbzTSeBYD2COXqLjSLCA0MwhLIU83oDf4EdCKwtFUf0zO1Rosh5I8DgZE3JLhYuwLQqpr5EOu1adScbAhGT0gWp12SzTAyLJSlQIyEAPbAGdgy5MIYZQDSVd/WEUCHZpNtI68ibg3ZCz8M2kpGg7CA5HmF22njBegA2cPaieSjHeFGyfKbqMGvGEBlTYMwxJNqRHlflVVNzGLmiYpSwAfwAHmUePEyqPdqZAhaYlbqtuCpQRrEkIqIAVniqOpph1RXnWn5V9G9d/F6dnNnW2kXoyZ27ntRCxO/iaoNPgFPgC/bqRtpAB07/Tzl3/5l2/evPnyiy8UCzloM68AxmgNlAK+M3BnXiCAABZYPjRTLJAMXoAFQ0I3rnNhF9FdzoFZvoo/0HQNkd3BmHzlJzYpP9FVmuJd/Mo51/m8fTAZ2ir8xIrojfJGNCeMma0C9uAid3CYjUnCKtgA3kWoWqFSszdrkC7dWcy/dOG1t771re/av+9LX/rS5Su35g5OZLO5scdGrtpHHnmoXCmAl3Z2toLhwJl7D21vb8EchKMRsFUxnwVhQSaw0uV384VqNTGJvO+s1ytU/g1EbImJSLVZ2djaOH7iGD5ZV65dBrmfPnlvNZ9PRKap35zfraI0RY7Bww6ogwAXMnkgJ5vJx0Okx5rdWGvg9Yv4izd/pYDvVwVHHeYTwyD5/Sj6i3Zyc6fp9oecMJF23/6DpzL4YSW8PvIgo+lQqjcTBdux1QkSGCvX28MPPzTrDIXyeefGtbiNZtcT1pY3s/X3PphwTaG5wfrkiuHdtbLmVxAGK7432ZzszTsIkafEYMGbwTSwaYVLUDvi3CEph4ihBgbKftvWDoy99qjHhqO5wcBUSgC6iTMnvNAX7SIVDl1UsM473TszB/b94q/+wC/8zJ+5Iw2URnJFYDMMnKgetKGc+EMBdfgbATMCJKtf8Dp4Z5huwnYTfjkku6XZ2VyUQQrdL/1hI4fCAafLB74ESrHQR6JRuPpioYqatm1vMPFgViwFFhQLNwHecucCYDUP9AEIxwzrF2ga6wjfLWjmxIJv7rNmmSsGjnHHH1i0mdbVEryIsoOypYdhX9jnj4wHGOFACYjXpLwLxP2TbjayPMKp+IgQho6I0SlNJIidyWWFhJcYPTZVO3lkCISIqhgi4FbcJliiUSuB3tAtMmcBv9KJDUhH3x5KC0yyKlEULTsNiepoOcBbpLzPu8MOT2hMerNKJ98e1Zx+uzfWn9nni057JufjtqTb5m7ZbEWbrUZRSUewD4bECt+h9WoD7BZEXx6NjepN7UrGqReok0IIiEo9OYt3mkQxt8FFsMjMFChB/lagC/WF28F/jI5fhHZvw6shpLe/mBnem+S92TZzbt1vkVnr3BJwGT0twaqItgqwNY1CKUJv4pIAIOkvRWn1I9hFtAbwhgLDjhJ1Bu2Se94eg6XZF2XSPLI5wDtMNQZWmmYYWmKpegA7kSMDKTxhmtbLIWZ0fm9ot0eov9Y91q2cMwQzCnAlrakLZmhCmtqGHJJ0rXM9ZF1R/61fjT8wNzNwdVQKK1EdyaXwbfC1xtwLESWelxlSAWUlj4Fv5VwFfSGokoB5nKa4GbJNy+o8ACMxTD50wsZmFuzakJArRo/Xz9BNrkcHNfwQ9XBfOXjgEF6vt25ey2xvkbeCl5Lbj3RESGZNI1cBPEQi8jT6TMiSGaOYEMgMh9okw3IgQLxSPJ5st7rIUiZCAaWxrVyDoOJtBP2z4W0EUW+TbwALcRUTDARY9U6EF9qAXZtf6SSqZQbCrqQyETgC/0JjEyXbIotKdUAp06HBaKtok/j/aCQK7a+USsiskVCY1cCGPT2Zpk2re9Zq0U9wBH4B6M7D/kC32XLG4qeOn0A/vL61SQtotPnE/QplOP7S8PjPP/98oVhlOMwPX5Gna8jCQ6r69Pw+TbWWlUMLyTLyBgR2cnHvSVS8HaIuw5uZpTvIR2tkDrAQByPlNu6xEBEwacYrozvnfHKv3sJojV83j3DOu/YIsGlNYZamP+LHDA3WV/TtowGVdGfnyZ89R86ezczO+uaaygAkA5T/q1yiam+dGUMCjk1Et7ZyE0tRh5coGd+QVNiSoOTYnMtmKpRNrWCTGs3OzvJCWJCZ6emEL3bjpassDy0gsCPAAXrh4DiexhXPB8tXKhXwUv3O7/rgqVOnP/uFz+IU+83v+2f7Jmbzjd1LV16/tXptcyNDCrN9S4vwmxiAI4E4Qm88Mvn4F77caNoOLsfqpQp6YsAJ02+QYrYYDFEDDZt0npi7XKk+MaNs5JhYE+m5TLWRq7fC04vU8bC1CwG3Pe4Ndr2dZrFUb9hPn77XP4ilPOGlSJrIqNnQIDBqQhrNvDJb33Aww29eL+s364q1EGZRdNm6aADgTlMWxhBqMC4aeLkSGidcJrwrzl8iAS/vQnSxaMrMgIKCXetQAUGP4olIot9vU/WuLXcCZ8tuyxd2ine9/f5f/+3v+eWf/4tkhBTrg3higrgl+BF2aCBgYwJx4pP+AIKKxy2BOvKr34MfGVQpd2d4GDR/cJLinuk9fVLtEzI3+kLhOC5Y7FZkNh9GIvZCrYqRq98YeLzkz7EwI/nVENvEM/AWWYXgpH0kffSDk6SwUpv8jhxvDusKOIKJY5r45OAeDvSS4DrIKjufxvhZBBjrgQEHpz3gcYWxKEpt5vB4PSGfI0i1CbL/S/BFfJHeEP4SlODCCcXaSzwPtdDg5UqtMFTRiB6eVqViNtNq1NHaUyWmWW3LHQoEQ9QTKjKCtIzWWhYFa1kNzyQ5T6KRPeQPDNzNerdY6u20bKXYVODg8cXkkZBtumIL1myuPOxUs1tFdUfFPhwOEQ+g+kFXSOgdCQigbTdH7SZIxDQM0QVlSPXKuCVmIcd3bH3l2urTX97KdHOreoPEKfTCuSGQIpNsA6RUaYP/3mEo3x5QCv1zWOTHEB7RK3PFfLIOutNIsUIe+kVrZKBzr21eTH91FwezjuUaAkyvzMMCfxOcSu+kr5O5U2AsZyNkNuiU0KRGrMeBBICFHppnBQtCmxxSw3CHkS3MGM1V8SnmxOqYdW0PFaoF01uessgeXQGseJlIk7rHV+tEN5hzDU7wxXWuAHmMBF4GmqMf5EUlnTPCHh7OKiUmeZe145P6e6yhTvB8ljGYx1HtWDfzMISfr7wZhgmoYmh7cA6EoXaHVzbNysVOnCypmHGmlQwKoPqPnziBW1GtUspsbTH1wCREkeliWwPGQLNkUFZfygP4awmIgDrkGbkN7SYa2mQqBS9NyMriwjIo4Nq1G8yDx6HQO/ZIFOecuOS/dHoS/hjGAG74xtUbELlGraniJUi+XrwrtUjf8V3fcWPl5vVrK+hjEWDZH0SaQvxMCi1tGDCXlo3xsZCSeAfXr65iaT169CgCLk7XTEdqJmGqb2kS4DyAZF5B4+iouaeYyZPxi8BfCNDMzAyyFEM4dOQwbr0QY0OJpahngDAoKKg5BwfSQrlQrNW68QjlyDDciH8FCOgEOhZwmfhaA2bcj50WlAKrIkEEwd38QIOABkAIwuQKP8ELMMX4uyFbQ+bpIU9B2/hVzUrE5S17Bxd51JIzuM5X8zY2AxBFT+gN5hZ5w3OdSeIx4B7yjtPPaHcnlIzf+8C9Zx6478VXXvzqU18l1yarODs7j0coM0xc9dvf+rbf/r1fb1RBJuNQZLi5ugbc4NY+e3Afqgzm3zepvhGdCTs2Mzm1f2Fp3Bm/+vL5dqNDniGGgC95bJJyL/jkd0lzMb24nM2W73/4kcXI0hs7l/bvOwQYh1KpGh66XeSB5PLCYWLMSpUdxh70+zG0X3j1jVRsEh75bW99x8svXN7ZrkzEgmiECO9sYHIJ9JIUTqe4ccx79ebrpPVgw9VIFGojqYWPwlzecDJfbKcXpn2Un2kOkyG/29uvE03pjE94lx858UF7P9S8vjobt3nTN8aF5/1B7VjmUR//o4P51DSbWdfGNifWFZ0bvGHQEj/pV3AVeJu85rDI8KAS8FgfQ2IASFQkPDHA4adGLCeZIshZjeeDC3nP3qT8FhuCgki4BpE1utZ32gGQ1MyjtdwrJx469Qf/5d/+rz/6e1PxKFIv5IctRt94gh1Ur6tWJnsNas5eI2wBUKKTwIhSD1Go2B/iNQwF8AOzsFlQYzDhkWiIzcx12lFW+V4vGiEB+1Q2uwGeh5yG5CXiZ8ex05G58eGA9OLA0SQ0to9fMFMABpOTk5TdqBrpE/1AJwZMA/ESaaGwvFD9MbhEPvtoNRR1BCcOusZG5SDnhLzB4QETLhzVOkEcrEKkMfURgUDaGqor4EmCNR2ZXcILo2W7QEMkgMFvgL0M0sbnjSwW4JVWrVSolGqVIqpdUskmY8Fhu1XJb2MAGfaaRIvADaEkdga8yHwoHnAiM0SDiYE9wT1jMKLWsAvlV6nayw5c1dRB/z3HDwWXYzY/GGtn6M51RyX0jMR0ewNoyXGRIZlBHyMZDA+O00wIqjY32wIUgfTEP8iXQA4OFUtCh7AHXOngBKUhHKOOZ2Ll8SaGibxlXmAD3MK8iWgIEUs/bkAWCIKCWJ+0pyb3cmdYv5ulF7nkF4BXfyy8IdK11wsQkJCFsKmEPyZV2jvRJHVQD+uQtyczizgAyOkn5kYIEBFY1wkeIkG+vG9lNIT/IKUICAyfApQ/NKf9wH1sCkbBAWbjm9A4imdTaB76LUrKd4vm8oB2kD71oUPf7pBVs+YG0Vo6qb1fgS2wnvoNfTAjhknRdPOwlO2IuebTvJnphDUWNIqCgsJxfiecF/2zKDHmbfqupFeSgPlnUV/l3+C6FooXgNuJRwLrMn2sHmIxS6Z1pTUAXawFd8KTIjo5xzhA4u9ukjH10Q7L+ZnatlQvqCAxhMOhHiZb5JB6k+nFXQmyFAUTj8eQJVIxyquVsunxGEIr1AiEy74HyKamp0m2vLGxhaqHKg5IwvgkQ+qYYDaJPxokASDzCpIt1Woxu2thYenIwUP/7qc//MxTTz/+t393+dIbUsCSe9Wh8oig+BphJWQUN7GtzDlTyXpL3Wa4aq4zWMRQco5g+SY1B2nUgSIMtPBR9AQKCtUE+wCcQCcsAnpsr8/HhGDDYqeRli6bzccSUXS/n/vcE9NzKR6pNxu0DD1eWVk5fPjwxETq8b99GpPWiGBGZeHwh4ndCQdwFger1KsdAh4MjFogwaOCL/aDqCZ0lgAwOH6Qg6GmPMKBjMucw5RDzkF5mLQZOBz6/MICz2OBtuzrNATrzhU6DFTAlkv7ZOBXajEZxbWT3nxwsxtVpwVM8FCgIzokRgzJ0DE7PQ1mX1m9mSnn5pcX2CCHjx9b21gHyVK6kWc/+B0fLGRzCEDoMpqN3sJiipE8cPbhF185FwvHstksozhz5sxOZvfy5ctw5kGSNROChqQy7HnRCxATjqqKEBnvOBqO4ZhPmST8WXY3N+5/8OFScfuPPvn77NMzZ+6eW1gs7NRy23iM5rF6Hjoyu7CUXrnlm56ZDPlDhyaPvPXud335mScXZvctzi3ffebIpfNXTSp3/GpBRr1hsw6KS9vigRBGhjSlDJ2uQKlM0qLixHSYMLrUZKznbcYRD+01Zb0P9iu1TL0DEosjvb/42hvfedcPrDqLG2s3l9LzvcaVwbDMRIPBwPjY2ZhSNqrEDxl1ZNsRDhAbo0ACrrOvoVFi8dlIAsuvH3e+cML9Wj5zj+4iHwHEgaYAEhPyZPHotAmiG7TZ8kJEwD+B+/aAq1enKrbdFfKC30CMODc5XJTI9BYLTyZih0hcOLVv5sO//MHf+Y1PeB3kVw5ZATfENgDwoqnYW70YUN0BIgQC2BrACFJlONHj0g39pxQUyleNezO8r+L9lK+G+Grkm2A4JndLp5No+KnpiddefiHgofxXBHdF+GAIMGMGtoFlCnPmR3n2ApGzJPbwU0cbAgxwsp2YOwwPiVQKvFOrq/R3OBBk06IbAl9LocaGRmBmol2o74AZHIIhNWB3T8COI4gPD6+gJxzxxQO+kLBzR7vZNXT47QFSUzBjVPRCIyhiKfkJAHQ12x2IZiIeDaaiMLj1zNb6rV0nFhXla/bKeVGCDCxkNxakkmtN23tAMGUYC9mgZ5ucnCU7j/QCIG30ba5uz9Fs28tNR6VtL0bmHQvLoYWjB2zTqHqqtuEluUq5OkOH4rqMpoGdr53AoqEvJIsYLg4kQmIngqzGHW1JJ96geFoTxUVfEV+EEMD+jEDu8qJQ3CSRzOA71LoCOeBGIMmP+l1f2OhwI4YyGbxjYJEP7pX2U4IYi65Dd5v/FXoNowS95i26Juhm0yIHGH0dd/JW+RyZf2ikreYhWayoxFPalvlEHB1ZAPGggukiFlPxrsSOhXu2wMhPBnjIwez6rUtb2ZXpqSCMOOuK+p+3AzfyDjS1g7SUCHUSRQSUcs1i0jVCeIC+HNhFJukg4qyGJSoqnzWmVvNg9o/GxjmH2JPbh5Cm9R1iL/mPEfNqLsFnGGHXYtBgHZgB3iBVMq/F+V2qCGPuhZGUaxXAijRL+V6Tr0r6Z3yvYKBFoYkk66siIRppueIweSL4svKSewNlB3Mpq4e0dUhgpF5x4qMBg6jwGNEGKpzjkdT2uUJYHc6/9AxCGMU7IBm4z/AgqJlwPiCxUq+KoQl4jeUPQFd+6bmFeYSV5196sdKsQ+d2c1mKAZP/LxwL31y5DgWKRUMYEeFiIdnRVNwOKzqyYSB6+OG3wnb/+5/6aRhj8gAePn3muXMvHzlx8o2XXwblIW0T4bC6vvbwY49S9nWnlg8GnaVCfXoyTuwQYKjs6CwFs+iwtfHMZ7Au8vaVceFSN7YzU9Ozb3nLo1/52lfqW9vwCqViNRzxg6uz2brDSzBruFNudsetifRUqVKuN9vUN6Sk9tZ2NhLzo2hGCUeVXNzLd3e352ankilXvVz1+AERMIafpDMGDqjc2U6mY2ja4QhQW8DrKDMde1y2NRT4PTIXsbI4aDD/Mu6w8x0OBVIrVgYQomgQaMMRcONIEiw1qleuXQWj4RsFEgR3Yf3jcTgKBguE0rKRVGCKuUCaBvzGJaAAcUC1MA934JqETK7dCc1GDEbmQm2Lj6GnUG8STgyxb9Sr7W4Lm1+hVrnn7P1Hj528uXpr5foKjM1f/vePzc3Nr17fxOP1wPwMziqFcubc0y+2eo3H3vHoa5fOE9ZRKFd/6EP/8uOf/ASZyI6fOrGxtf7K5QvJaMweHpd6toUpP9FHTjIuNYZ44cD3Byii1+turlyJTyd28zvpmYmhc/bC1c2YczkWT6anA5n8zVa3gMS3uDyRz+RI31vJn59ITOO3e+3Spd2NtbNnD1+9eJWFTkTiOzsbkSj1Hkk8vuPwjoqYBOrd0TBK6BR5j+weZ3IifO6VL8XS4dRMiHI4OWjzEJWgvdoYjf1Bk5O1tbr2xicqf3womk4Sk+uIhWcPltZencIx04Fdj5zQ4l9B0wTnUhKEzQlS6EFrjRkM04if1AxsNHosrxmz74Uibh/GyQkOjLUT7tDe5n/AVvp5RerL9YrWgGHwAv+BAqDrCEXIEtwL6+YiyN473XIHYV6dhF7xZgqFijhjw+4VQuwQW0YWOc/43m868m893/k7/+cnecrvRb8EMhrVmhUPySkkVAjp+b3YWELoJfKlOlb8UCSSzVXgVsPRJAjNDxSSvwKH5kCAXFn4yLMvWp0+igwKW0ViSaRTXJ9S8VjIJ8fhsBd9KmEXSMOOeCTY6XZjPs/y3GS90bpy/drWZqs0aAZCsAqdljwkTYYeguGBPHfAF1PmLQQ4XgfiBUdhPMazA42AiutAs4i2lKqYaCMCl23wHoFUMoRIqZKX5KaQCpH5YuLgMXApRiChhiXIDg9vaXvQ8MJCpabjoXAEAtLIb1fKmU67Jr9z8vCAgyXFmzzJaoI9i/7HjVUFqYKRKOTW5chVCt6wX2EH7jH+U81+vjLK2sOd+KJnaT45ezRm81XHvuzIXieMHdMYtNXpdQ4pL8NCiYjIJkrLSE+gThACCN3k2yETQIhByxsPibtasGM3UEQvG9UQS8DCcHkiJ4YIGzRnSCsXADqRCMGZgSm+C8BEZ8xhSI/ATvcCSdJMcyIyC/VSi/qJWYFW6EQESIBMo+ZtuqbDdEGPqFfSz6h9OqGfzKssDhUthUlPggUQfbIMk4wDJ4u17d1UOhKLz3t8s6kpTI/Voa0JtOFcRldoU60Jp6FvUHOIEtbBX/PV3CC+VZvHvNHQSLqgAYFH9ZW+aGuZ0d3uF+3qkuZBg9df8x1SDEXE9s8saHdCOFkVA0qGi+YrCwSNlY88YcvIKzjNI8vCMCHDYutFlLJrl0gC5go70STJIlxbEo6keV4EW2B6oKnmix6ERLE+5irsFFgAvBBRMd4A1LROdY9GnRAXMGmr3YzGopVSvtWowt4GgwFUuqgR2JMaAy0weHNoXLx/CH8aQCS6tb4GU5tKp6EcaJKvXb/OCdpp4w8EV9kBZiyVEN4cl1dvEjr12Fvffs/9D1x44/JrF69BtXY3tnY2NhF2c/kCOfpxnga9YArNFQp4SEGKyH4BfYGt562cWCgNxKVp5cC6Zuaa1JMCMCmnXeVSiSwcm2vrLCmULBqPIKbXG+14wgd6arY6RMGGfNHLlzZT+MwGw0jqyIIPPPzQJz79adR1KGMRf9PpML5nLz3/ArZ3qqpAVgWqyqfH75inkTBwxUcAgA8kGZTx/PQR8oECY8BLBe7ACytMPwX/Qq1AGsRSU2ogmeGTR0qCvNOO/1GLOsMdlYQ07ctZAdJpuLs9YOJcsw9mR5EICgRQeA2wg25EmQbkbWqdg1gls0lkBqNA+cfxmDuzu4MPcTIWJ/kDi0XxKQT9f/njP/HKK6987rN/98rLLzf65bP3nDVJWvZdvXnlVmHzd3//oxs7K3/653908cJFXlIqVc7e99CnP/03r7928V/8+I899shjf/znf7iycWvi7H0n7jn9taefWlttA+PxqK1Nmh/nKByN1ptVnKKJLg5EYMPgJ/CHqoLTy+ViJJjyOdqJZKg3yGFsYTJiyci46yFh+BsXLs3PLgzCwWtX3/jkJ5+rV0kc1b++vrG0mCqXC2vbtvkFW7lRu76WmZyawfO5N/ZmspVIEuANLB9efPKZv9vYbeEc7er7lmaXJieXIolkvr5DrKdg3+e5fv3VU2/7prMH76/uvNxuxHerg1QM3NyDXjGL6HMJogcU8XSQTVJwJnrHlLJBdWg9//8dBjTBAnduZLsb2cVcAbHxhZZYPhoWQy9LINtfGTi4DLveC/XsVE5SSQ/8XxGAh3ZUkK6RI+CCto1HBYL3qd0ZjbvOPrz8C7/6w7/7Hz7WrFfA/ZVql1jCehU+21YpjScnwA8AGo4FkFjFiAFU7Gs59iDeSeMGCJFyQjHoIheDcbaSx7NiP6Uz9h3EtQlD0gsr14HjVDyCa4UyXsHASRAZE3mDMIM7SMQTO7B/mThAEsKjA9vNZlx4t+N34wsFgTmQuEgMSARVs0xYmg6Ms5BhSbGIGyNn2IkSHaWr24lDpUr/GQLs8AfdEZw+EYdgaaUK0JQxsRhUKmo/QNgL4jN8KDZw8g05yDaOPrpbz1QrBbTORFQSdiG0PexgduUvIZBsDSikyZdJgisGgpo+Um/VSS+HXyJcfyTsq/Ry3WGl06+MA63olG3yYHDiSNQ2B9tQto0rw1GDQFAhVAfR0/Iugi0k1AKgYCtaVFH4CMOo14cZEeKkr9JiqBAGm6BeqYoOQAK/EZgMGaIZARogJygzN9Cqzk37onn6Zv5XC9YJf4UOdBOYQSRHX0ETfOqy+cXozXTGRPLPwqL8am7gdawMFI6nABqgjgSMupkXKiumhF91iU4Snw02I3AZuwOLCYdBfs5So1UpUEJ89eY1dlrowMHpenUQCri87qh91AYhmtkRLLDPWA7aIYJEa2oM9noR+A0h0poVXmgEXWs2rFEYLoTr6jL3M0c6FeY3/TQ3saX0s5kZUVMwsEX4uW4NTSp2thY4mStyNANxs8U1atFdgE0FMQjq1ScKFkRYMK5U69iDgRz+sXXEZwm/CwJ4jdA64iYX6Ir2nIRi2GDgjX9cY6CkAU9g9XT7ffCmxWK5Uq5h0sF3AQUUpBR1CDgMnTO35vtUEdwrt8dXDg3TrCbYqVIogNvJ5Sve0+HghB7y0lgsxhxAlLjTks+YaoSw3e3dmfQklqdkJHHj6nWle8SXejC8ef0Gf/DoKWR2eXMhs+1xBdA8UdnQef0GlDeKwyAmWywyuHpjwUW+l8IGkqPZ59UMm3exaTG04BEKigEjkA+Lzuzfv58RYV1JLyxcu7YigGHK0AC5XMjBP/IvvpsMG7ARs7PTlGo4c+/djz70APkmlS/LO/DG5JnMWEjuCG4iJMMygGi5BegqxQpdxwrGFSP7iovXr8wLXs09BGUdAjIe4ADNGj5G06idATMAhAv8ILZ2ChCLG5VgAZHWwu0xGJp2Dj0g3kdtcAIXIC0TnL1WXBCLLycmJngXGhTDZJgVzY+EbYLB+r6QD6fiTKmUz+Xxvoul0s8+e+7SpWs/+IM/+MbrbxR3MkdOnvzSl74EEgLJvvub3/fMuSc/8pFfHpJ92NXHwYX/arnKzcvXSrU6pQu/9tWvwaa8duHC7OJUKVt86PQDd9/1wM7mk+QtRS9Mn3K5or1kXGu9Y3ulEUvFvIFAu97buLU9Hnjfcu+Dh5LHhrbCZvPi08+97HB1WbipCWprLKIxPnlydso3XRuV7rrneCzle/381ULOm882d4tFjIuzy2hu+ps3W9MztmDU/5Z3vuXsfQ9jz84UtuutxurqDYymwEu1Vkz4yfnc6XawpvbKpTzlOhcOHHvv4W/fuq/obyLsjqZmDgxAkIUdR/kG2RrJiucl8kVUgSA9Np/2LbMtvDAiJYJSLlEHAtUY4YEso1mZf+qHMLDaMliV1beQqFlZ2tePsNe46I4lhHcKNnJCuiM+tMNCLjAHDtXfs4G8ByOvB/cozIbVbnuLutYnHjn4sx/+kT/6g7/KZ/KkiACuMLROpSedjjKMOFDqajQwA7PTRX5d7nDYyTX2EegCqREtEiDDq/kVICM4Pj05RQQa+xfNMBfZQgQKo15CPdMsNgg3CARJF4FBtI17qJxSTO3BRHwiQPWVcHRqesY1M7eAQI0lDGEP+ZJcK6AtkBgbTzALZWcTKCMNVeJxdnbEkN8JVxE9JDEVulsU0Zh3fIMmaE9OMUA9WkppkZCTx+PU9ATOwu1ebdhq4euEklpJzKHahM3mM5mtnU6z6nPhYooCg0Q7Lfe4g3oAnwzUG+jkFGpEAssxaYwGwWhy6Bq2hi2mmFCRpr2Adqsx2g6kBlPz0ZlD854lny1BbHWWMqydUWXoJm8zjv8UdsPEDs/L/h8GvFS1EdVSvgacv+go3Aab3038U5tpGLS7jmGDzDC45nfbDdg7wIjfLUJosKsgid1rAMp8Clq+4RAOMBBjnZhz3cBXc0VPcSLE8A8O86jaF04AP8ANSIaGAoKyFPFpvuqibBXCTqIlAkxzWC8wQK+lUGQU+U6UH4qvdshxudzfLZAUMEIkC4vYc3uXFo4WsADatwkAo54A0qXoE3uHzSNxRMNHVmO6eAnSDa/Q/3SE7iuGkj/WF+sq/eCEfQOjIxqsH61RmR+sc1CkofPCkrRCI0baNZ8aiKgF+1ojgwuWKCfyDEFVehmihozxm60vNysFoqO2wgELyQvqDKwaFbMs36haIL3mWdgdvZK+06xB0ppWRc1qJTQW/jDljEsZLeCdKUsQjsabbazB8tFgp9Ar6C7wAwbnVtn20mk6ZHap+s3BV3XedIE9CZnkaywY4kVsRUJ3EH+1J2EZeJmZSsGXKKTChIgKuXltjboRZ896Dy4dyGayn/3s5xDIMKzGjxylKZ6gTba0gt2Hw53NrXg0hmcGIUN4UQAU4AJFXLBYZqEAbw6GzH4EMKwwbhA0jDW2bbh1MjzTn5W1VUoJnz1790svnV9ensc/BfcxHKRPnbrr7AP3f+FLXwbRkBbqrz/zmZ/9uZ/jFV/96tOqT1BvIDDAGWxvblEzEV0RYAn0wsCLlxeLA1WD5DboA9Y9pgXXDbgEH235/MyC2G0gjHkzhhhOeJrbrBNdN5PJuGEy8TVDDsY13eiTYVqwYgm/Wy0wmXqzFlCKbKFLNMwsKitvNch9ZjNbnwgoum5exgfviWEPiEWRTlJTkz/xr//tV599ulir/MIv/fKBAwf+zb/5N6QRdXl92d3M6RMnf//3/q+f+/DPn3vuhW/5lg+89vqLL7/y7MLS5LWL1wH800dPYlV0jVv3nLwbRd2Na9fp9sTcRKFUWl3dPnTg1OfGT1bzI+TdaJhgJC0WQWjs9Ea1XS7WQuNQv9X1eGrR8EQ5n+8lq61x8eIbr6/eujm3SLhLoNGqn/Afe7byMlkJruWvYBzHp/vu++46cupMMedcnD/x0Y/+1ub6+k6+xFzMLAGhaYcPFNqZiszV6h3K5c4sLj/zwufR14LHAXWEgFxmZ9hpTkzGAx7nRJzgusqLO1/xjuLblzZWRy/NeNy1jev7vIS6+J2jFv4EmHac4xbbCPSImhS0xLvgz9GMEliPiZKyuuAdw02xPv8TBwthmhLa4DFWCFhmkc1X1lVcnVhqILmNflvvxYPcib8vLC4/C+yV4B/XAvuoRaZoAq+Gg0K/canf2j348GPf13n37//2n0WCjldeqBw/NFXIVSYncC0saFO3G3ZTmJ4VQWT0hInZI/eJAnCwtcFvgR1BDMAeoMWswrniTEQ9ylgqNT8/36wWb17MIYLiI8HGRwgCzABFQJdxkBYUUbRUrBTyNTYAZB4PDLpMmJh4DjzE+AfaVfXcMcFjYbPVFcAhlpZUFnjOo/EfRwnLII4I0gb6xrhoH1CFDV2epC+aEujrrYJn8KrizZ29UNRHmJQ3ikWyP6xVahtrue0sSA6JLOzjdUMKn+Lz5hr1PChM2RLwuMyzvHilOJWJhi64x9nytj3Y6QfHufquM9pzRDqn7kmEp0aemagtgiF0ezzMdR3lobtJ4DaoF9xL5RssmU6Hj5ArmPRxX2oBWARJO0YCZfsJu1PCBSdLcgz2RvVhRf4+WILRSCAt6eXWJ2t95xDKMCTozpU3nYj87B0WERJa0MFTgiodkFVBmjl0vodouGRwgqFJzKCwmGiC7oPdBBwRCg0tMa1xJqphsJBFdiVAaENAfXDegwiigO1rgmE5sGVWioNClsHhD5JKJBZxzwsGJ7K59XpNlS8RwkjwzQtF6KjBZXolCqZe8RKEHbCcumMuMR5IiDCmNUx6rt7zTsEch2Qj7tan+cXgQUPzRHRFF0GW2nFoAnQwP4CjYXol/kBguIgXvThrPJbZJBJqEX8V7wswWdQXZ2ORM+gxJg8gFsiRLoxmmQu1wyNaDn3SDWZLTBjMLF95l2aYWQUseNgAsKtUribSE1SaK5TKODkuLC5BgC9evMi4QPr0CRUoNBhBFteN+cUFpBzaUnumc9zGOfrydCKJczKTmiKZaiKOEIlLDhIzNQcZCavJVhH1MS4WbPFiNh+IR+anZxZmF7aJgFnfxv0g6PMjOuP4Q9HZUjYb9LuT6YlGpQxnAKNcrlQtzlpMgAitpgzluTwYzHBEjrRA6hu4w5yg/u2x2gr8tNmQa+ktGAEl84c+9CEKRuXzWTYgBLVd7Tz++OPvft+7f+ZnfvIP//APiU4ORYLnnnvu3e96F9J5IZeFmWAExXwRRmd+3/xudqdLqHIbmw0EUwiITIF6O6lwDEgwe3avHKxAoUygpkygIRBiLuijxZFgAuSiZcdFOaEVksxMmXb8OJRUTDmfDVtGXRYjkXALh5oTD6kZlZ4SLlWYmlslN6A3BYeJF0Fgkd5Zhkw0K6yDWTAkze2qu1BNTiQw+X3mk5/ZLeYffOyRdr31Uz/5UxUifJyKTkQNfvHiZcRihPCNm2uUqTh08NjrF14tZirRWCBfKB/ZfzS3LdPA/v0Hgx6sicHkZDw+Gd9c2XS7gkcP35VMpqvoPckd1ug72mRiofRtEDmg0a6X8hXmwhUgpCQ0PzO/tbYyIuufp7WzsZpOJ3EmQEGazZf+4Ku/Hwmlceko5opTE+nt9Y1aNbt/+ZjdP9EeNR949LEP/ejx//bH//ny5VcJRql2Kt6gr1QvXtu9cf785Z3c1vf9wPuXFvdl8lex/LIWUIs61u9SyeceuUMYXwkIr114bS3gmaxm6gPv1PyhU4HBvnJpNTcqR8ckvRgTrs6KYqqVKOa1tVR/DDslawiDpOJy/GP+vayaWV+zOv/jD2sVtV1ZcXM7j2svsYBqitW9DQ6o+kBuTSEBJKcAHsFRn40MqmI7ocSAiuxQw1aezFkKGke+czTHjedPv2Xx5yPf/1P/+s/vuS9Z2G0sLu2rlRskQpHbGsp0PM6JaqIDRB8EcaBUwTsKEPTRZBPmhJIYLDMOwoV3uoQfNkOxBKWHgUDJzT4vJXVh0xutJrSDHABANZITMB/GuSKewIurQY3Obo9oQwaH94lrc2UX4xZJ3kmKAjJCxe13BJwBNPoarDaBbG4ykCi6iFT9Xb+zT5oukIYSScqDEM0zMTwYf0TXhJvou+ylDALvj3ZlYjoVnp+0oU6vZCuba7ublDutEvtDmUeFJHGn1IXkt0J5AwY0HsWId7yWd7OjMOSRi9Q/Kg9zNdfW0RPzgUS/t1WZWU5MHEzZDsFmZG3dlW631LU3kOjIv0rG0O6gA2fNBlTgFO1rAdnAeEh3vGhVYR5wJ+OiNPVd7WhQhtxDsOXLuQkpkLhK9BAo8cAeQl9mc+uPTkSP7xxM1J1zbuQrF8yn9aFZFH7RbWrIusq5rujf128GpHUXfIoinugyiEGPc851lYdX5iY9qWegMLCZBDqgjTaSsVoGiaHoF/MD8UHwRcgjsYKC2BGqqo1BpdTr4O/icuAx1O5u40Y+NZ1SYobu2BMLtrtN8lwrbbfpFjwkvC6cyB0NsYialli0UliNNRbVtMZFT/cmwBqGRdvUIau/Grv6LhKuf2JTzPh4GwITo4UC0hqXOQGABEOCQutcKcYwhYDZleGMjQD5QPZVsC+py6TigSpL3BcNFt3ViWnENAhfB2kWzmVzyfprTILC+TosXh121QyFbYM7Brpn3H1B3+hacFlkEUD9HCiiMdaCtnAMLlGQIBSmAc2MQeO82FpuegqNgQBDvDkgloi/bA0IMDtFK8vYwLasK/uGvvUHhBDA6eLx0K02CuXy1uZmKBiZnZkh5ocUPoQAwQriJklTsLvKZBSNyauTgz0H7af/ghbteboEueWcK1oA1kbM+FBJbMhX3mx5Ioqq5zIiHXfiONZpdddWN37sx37sQ9//r2eX49WGqk3QcwJ8f/Kn/t3p06e//OVzi4vzkGSk58n0BOUZcFFm+2eK5aX5Sfb19ORMpYZHAekI0J04edYTEmd/8ODhnR28tTKgMb5y0F/8OSj0BkU2IgXTJlUz0EFXLbCn0/QZKs4BegG1N1BTKVxFmiFzv4as1ZWmcm+M1m7lEZOBhpu1V9WCOaz7RYnNfdKySJckEzO3TUxEhWilyXPsbO2gBSMK6/HPff6pLz1BbgQ80tE3fPDbv+O+/+3sr3zkV2YWFqf37//qX33m6Fvu+3//v/6Pf/+jPxny+WaTqS999jkcoNgZ19+4dn3jVmQq8tDbHoxH4uujzcuXrxG1VFzNizjFTd0LClt4bDiWBikch3rcgQ5yOGiPG9V6lnLOwRkSYk7PxB2nT9W7mau3Lu7mifyG08RlLHD15iW03Bev7+JkHk8nJhcnalXH5770mXrZdvTE6VN337udXcNv7tTpWWLEI1F4v/g//+APfPS3fuNTH/vcB7/70c2ty+jxUXIR4EVuT3iOYjZnr7Za6xXy7U7OztkGFXdgREYjRzz01jNvq2Xf2HjmL8GYyQFJ450e1E3jOjsejMSuYd8gwMH/AdaQAhkUOYTF9fefeBiaox3IUxblZlVYKORsmlZL5n9W1pAHOd2BKEjegBOFn7fSC78QHxc8QbGixHqNbXUiAZHNcSxutq8TCLD/VOJnP/LOP/r9J+KpaUURN/FbIgkeThUgwg42V4CTvBA2Byml0Ec4KBWKK0itzrbysv3Bh4QKrm/votiAAAO9kGLAB6pc9nu63Wav0wLC6algiQIP/lBiNoUBizRkYGPCCwnFZl1A8q5uFbs+ZlZoE2m5SMQqaR7pFs8kJhVVs5lbkRuCbUhI4Ox57LhJCVYZLDRMkCoECQKTyC3RX6wGGVSpf0iq03CIDFm2UqZ4dXN3a6VeLfhdrokwW98xJG8uPpTIwU4l0dLT0F0iosTrKDk07+/jwuVs9Z3tgavecmbnDnum7nPYwp0jh32+yYFtsdPLv9x1VHnCQyG3cADnqVajli/WGA1bFPUz6nCyl8g2IJMu5l2ZCoS22XFUrSADL+DMDCD1GXWKuGIYDHCX+Am7QuesNdfC62CrWzucc2vPs/3NL/qq3S5EYN0NUhD3xq9CtuYvXw1t0DWMTmw2Carsfhm6dI8wqFoW0aVdPnUriFozDqIQGKKWARwhtUIyOPDBL5G2kGuSDnlWr9CLcSBlBin3Q4qFnqNdJ8kfTvDcR1moCIWzcFwinAaqFginGr0MNiFqpVG1jz1FfhX+USULoznFSWgceFRPoKDoLATqejlwwpk1WnVLneDz9gRYsi/PiqyJMKlP6qCQpprin5FQ+VWkl7Gzm1lORi1iDDOEtonfgCudY0eQ77fc4wBs/D9YScgwdbdk9MUYTNyKZoAu6NNpCK20kPRWr0MiEn/lxFqMuR9IRVDSj0bbzou1nYF6ghzcXtLtt7s94ojwPyLnM+QWJldSpkbuwAbslVK6h2yKVRiSrCUUzyk6zA3MFOOGQjN0vHggupBw3KehyCUMt5TyRdkCvZH7KOiEl2vVy7mSL0wyGm8xXwEAo4HQ0uISpDG/k8ElpFquwAqXipQ8LNIBlTvx+lOJdK1Shf1AvqNNtpESGRlLLAwaWxLAMoyBXgDbDoOPFAHwkC2O3BsES6EjYjYRo9GqPffc84+97e2HTy5SMxGPzUGnC3dP937v934HjdnycvrC+esLyxNfevwLS0tLVGhQADH2MEIl7XbsxHOLCzSFBEy2RyEI+HI4RBMqDTFGUkALj3O95sckJNGYwaXcwSUaggabTzMZkqLYUMrkBfmEOsFuDrq4WSuTgDEAaw7FeBgh23pGj4vcckgOkkOLBBTeoPv0Nv1qbTKeEECZfxKV+drtV4uY4GqMotHufPiXPoIE/NM/+zNebzAUDK1eWyFAC+GeUGBuBh7+9f/6b6cX5j78s//u7z7/pbsfPI0vy6XX146emsXJuTHo1Ip10v10G73N9W04w4mJqbff+759Cwf/5sxnbty4qp5iMTM7nXhUG3QAjx/qajCl9gHUN7O+Q8k4kkGPHTNO/4BKl+jGyWAZT/opE/jGlfPo77/ne743u7N7/rWXU8loaCdcqXY8wU7cmbx8+QIZQ2v1h8uVK9dv3jx1/Ai4jv48cOaRH/zn5b/8+H9tlbv8cylLIyUevNPT6S6h3oVCKCnSFozYeu2ix9ufWZpz9sMXNq46/LGHp+5aX77SzK7ZiiXINWWxQAAjW0ubik0LjWBraQezpdnNIFmLOjBV/9RDekmDPIVCRFxEfYV6jHLECGT6Ktwh9MdCwHPzMiKEW5iGPfJPoggocqfN6WOhVfGOzwFOQgOMIKFQYn5r43Nz0w89/N57Xn/55sVXc62un2xK4Bo/6S3ImGiSwQEtwjtoRsi54fdQYpvNwo6Gl+wKclWgjCQ8VIDHiYG0dLj30m8sI9AYInld7hAgClngKw4lQGCr0wZbgisgwNGIwhAiwxhIwzUdXWI/jOoItMS+xsI+6qZhGCJoAxTLPzaqkC7ygnIaDp3ucdD0TCiOlukKUy5YF3KD8WEnoKx2kjItnIwRDzUuFzLr29ntdXQaQa9zMhJnMroVPBhHfvJgugPwHQigKNix4tEOK0g+UCKHVeLU2UJp0LXXO65yfbTrTXUPPHTCNlOw2Xd9S71Re6NbKrvixMVTLtDdH9Y7aFGxi7m9iURK8CDfyw6OVap9yK4TlwY2CKJYVYWlNsgTNxJGt7fMRsWBVp21Fiog2QL9IkktagwBhQUKBllbm5mLAjB+2PvkHr4IgqzD/KRTXTXbWzcY6LEuGiok6nJbUDOE1uAGwAtCIrg27+WcmbYaFI4yRE40EGwkqk9LQvtaBX5ETNS9IjnUvWqjq4MG9x3EWLZbBPWjniTaCs86qeSane5rFy4HwoNWZ9u+GMb5ezBuUyMFGuxy9n0sB6BHchU2k3gAho6HDkIyikcLqbXlaaFdYvqq4Rk+ghORIDM/wI+ZUoYCVHNRw1cX9Su+eeZXJgEEKYWIuW4SvQg3QSDpC6QKugspYXTCpNJVMCbZgCG6ui7qa8KCxYUwG3qR5pAYE/YRsApllyxPgipxNSQElpLdmilBBh1hZumX0DSWbUcsmYDmoR8i8o3AoVw+32rWmTQEVryNyIUUDIehrGjwUeGaRWMK9Lj4f41BMjHupuBx5qdSLmOmtfJGQZVhloE31Bz0VTo7dpAW2u4NBsniAJ8F0xAMRWvNBvm2dnfxz3IhH0PFMbhSDYme4EqEo3IoHEUapuQw72VWgXgoE8SDc07MxtQkW1cAD6zlKD+sjcs98J94soIslGwTZs3mYmi/+Zu/+ZFf+iX8fmk5SI7JVpPeMkZGCgZJp4PNWh1RnmKFmKXLpQZFDrFj7O5miQaGgEm7vLcJVIOoRW2WwegrX/kqnXET1BEOE1tl0BxphphJ0lVqu6mrxpSu+R+RqxxGXAfnwDcngjGnY2FpEW1EC1/ttoqdMVBkG0ueNneb+begTt+BAUUfgek4YWjMA6wPV6Dc1q/mbfowd9uRRGPEi/h9JEMpFFdfOvc8fMb/8v0/+Asf+cVGrYatnUY+9fFPfPVLT1DB4t77H8Dh+YFHH/293/39f/7N756aJq9ZfWoyVMyVyQcen0wGfSGn32sPSrty9RIeUqVh2fXed1Kv0i3OEQ+aKH6BshPhvt7p2uodSsHjB2QLRuwElVPibnN1pVLcWd14Iz0bLTYzAGOn1qQOBpb/eDKxubHz7Lmn7z97XzAiMK02ynCS/UEhnZhOJkPJVCyeCP/CD/6HP/rC75byteX5uVgwsrWyffexe89NfLmRa80mFvGNYZOFPaGl2fnV9uVbN5qLhMomXTgP9YbUIAyubV45sHw2OZP81Jc/f37itffcf7JOzFi572qXoTA47krDwAJKQmN9hK+sgwllW7EFjYS2d/Gf8oeV4jE+DDHeaxDrptZIG0yCiH7mAF6FF+XUB5wp8h/lvXIJo3qBDRyhGMfkgp8GT8GhUj6731+fmk3Z7MRVx3/iZ37kT3//b776xddi0elqucWq4INBr+UF7QnBoLLxqyiKEJGlbgF2EEjI9gojaN/e3pycmoWOAodwmbPTMySsu7i2xjZKxqJcB0Nx0Efq8GL93d7aoX4oQI6vG9ufwTHv0lGXf/OLOEPyD6ZZ6EhVmSCI1Bb0oXoE7UJNEYE4JxgHxjYWTMAAapOBO3Appnl03wRKtkiw0ve5XfF0zJmMM4x+IVvIZrY2N0T/lOoDUYlM+dL3kA+LDCbSP+O9S0weiwSZsw27xEr4nMj+ZInoY2UbF9u2YteR77gLB05PTB4ImDK9jYG9aPd2Ri5qDbZQPzA7eKfCaFjyH8p/pBAIOpYj4pTN5oWGdk3eZkeniUYbgQmSD1Knm1pHVtpQXTFxbHPxFCBRABNAgBUwMqnAQr8bCAAGjApR9EC0UBz83gHQmcPcrzN+5JOFMV+Ea3iJ9Ss4F/Zb/KLaMdQCqVeUSZFwHLA/ABD/WDneDoVS1kHAXK6zIh6yZDFx8H64y7H5VPCHIeGASMqHUbtDZk0ccXujVsDWDRSyRJOSG8Xu9IU6dl+j7w7FZlwoKWL++x86urL2kmOYSYc7Pgc+aBhE2jCP+MiL8xtRnCDcIoubN+gNkH2NzWkHl5IMYtApjwbgUCmcIGHAKHhUmoNWC2f8er0JFJI9mGQx1NOLJZIV6jTDaEotPgbGADYkV07g+azJ1HRCa3mltIJouHEMlJKTdgip4h7ChrDV1CptfzDS7vSrNTwjCHF2qywnySb7EBfURPLUUrN4/xMj4fFiLiX0HD1ThzJbqoNEX20EINFztH4smuAQTpUQEIcdro3ZPnTs6O5uhqJ7SH7Y8+BhScJcLZfAwph5QOJwcNSWecc73gFZOn/+vIxDAmaplFkU9RlQw0SK9IZATaoeU64HfQsHfA5v5B7chIAPnjIqBTlVkSIADkn2BkCCZ5Wd3MvsQXue+NK5hcUk2xtZE3IIK4sblCWDchEggbDROIQX4KGHIAuN0KgLROFcuIt3peDiHInQzDPwKTi3owUJub3BaquBQJxIJiHw2eyu+AKiq9tI/x5KUIjsmeoLtAbZprcI9FBuvUs1tWykjOYKjALdgNs1oqeU7YyV67Fogm5nC0WMZ1BN8Aj+GUwV/ZTKzuThYmboPOe8Qjtbgq8YfTpM+yCR+x968MbVa+TRhBZyZ7vZNs9KSqazvBeg4jpRymQcaXQayYkU3NJP/MRPULLpM5/5G2j2zEyaHnI/tzEEiLHR08uJh73PDpM2EOFGznwjQkJhxdY3NiAvZHRMzk3ReRJdMZD9R4//5M/+wovnXylktp744mcJLeAJtIKE9oJHCpV6fCpqDzpj05FDJw9mq9kbN24S6+fAmQX/cR/uY6Q7UpU2CTjMgt02MUUVJvxGbalJJ5Z4k0wycPTIwWI127e3x57BRqao/Cke29x8nJg0kiES7XXmrruZAZRd0ah/Y+NyLDQ5k7q/2/Af2nfky1/+3MiW3bc8WcrX98/dfffR91dy4+mJhUJ+9WvPfPzyykvlToVaIPViuZrLOqkP77Il0rblQyF3pOeOODDHlWu9oH/u4P4HBt3Y9devLsYjP/SOt8byuZWvPJ4eVsPuXrGepcMEMHbqo5iDiu5+di48XccNE0kY9ojRMs8sH58sq/Wp1TQHX62FsL6CDLmi+8DJWkbO+SatLJ96mK3Cs4ih2jHsIPkDUx9ngHs/LkZBmztmc4Vt8eUYch6zSpEwbsQXFawitQUhV0OShU15XYdstiOltfETn3v1ySdeSsSnrl+9MT09RXbhaCQ5O7uwtZ6ZmJxlknkTJPmpp58ESilJyX6p1euRWCqRSs8vHCCWKD05f/LU3c+ee+GVl56PR52TKYm2FKFiY5KwLBaJs/frNXbEqFypgbXIp4ZuCSwEb+eq7rTZ4D5PFLRPzlmkEKoTh/wuSnRI/YMOSCSKvCROyveGQiSC0PawtgSkQzkbm21bU3b5dDruTSWYoGF+e3tni/JV4p0x9Uq1KyID/jN6QPYJu9cULsb0SwEIKCaKA0RsoM43rA+rzV6576j6E8OF/cH5I0fcC55O85YvMbSFij1buWdrODw4QrN/e24qWOK/g34H9G8JPnblZ+etrI/xgcfGjAsb4jDKFkKLkLHQcLK5JPiyBQEMEV1QkBZXn6y1Lkk3aITf23AjYBAACUQscOHEAqzbPwlcrMPcqVM9cfu4c/H2Bf7yTqlT9roBeJlHSM7KQQ8hZqjndMoUQn3g6TgTlyCg5B/3MFQp7lFGI/0oaxokGWoN1+RsdHoN0nm1iMTGc9EXTwQJzKRyEwAJoWi2ajjMeIMT/kD4wIETLz2/sm82CRhXSjm6AXcF0JAlC3RE/BLUaTj0dzve0chP0hMquLgdvrnZ6XajWCeerteEHkhnIzwJCcIUQ8E1ESJS2poSVj5stMFAgswylD3jQIHWaSmTKtIpJIcwHPLoMkY4POFElYcWGYPtAMVzhThfrRtCI8uN7NuBqcJ0QTJhlMlqDzJMIwrxhsUClqGA5mnEHlxA3eSE8eEN6lcwEoRIGa+5DYHaSIrwsViK6BtPkXDNzRaoE1AAG7exsUFGcjpz4cK1hZkk5AQAw3impLt+HzG+RIsiIkPn2HJKxEyQLoQTIHR5qW8OaULMVYcAUC000ji4Dh8/mC2J3TAaQkLGiml+QveiEhqKtMQYRNhrv/+93/PdxDbs7mxVqirzhxzJge6rY8oSg39pgMFChqFYzIUgXNAiRkS9NQRMynVlwmXDyRsZnKRftVismQMCqSBx6Y1JsQ7H0wEZypNsOEwmE/wKzYvhSkkiPGYJSs824n7JuzrMC8cb61tsNlAHXSJ/D01x3ayCurGwsLBv375rN1dIEaUFRZuhbsAJaCWEWPlq9hTzoA1gCKR1whVuyOcrxONibKZN9BP5fJ1IDK6Lb2MsosLI99qe4jwMLafnlCt+4oknPvKRjzz55JPMycZqPpZUGVbrjXrS7HiusBbMO4MixyR8CaY0Co8z4bAB/OoJ+Ciay/y5g/56tXrh/Ku/9dGPJibSo37r0KFDr75YYnlR5+LaE0r6eZapLWVKaNze/p63X791NR6KxdIJFIVKvj0zUShkFIY2lojWplCdR/9SEUckIb0GGbbJ10tawCtXLuHn4CKzfcAZjbjiU/HpuelCqej1B+ElUQ6df/kVnKbAHCdPHEjG/ejsSITkj4ZJ3UZmM8oPrgxrD519KBlJD7r1YxOn8FsCbbSqLTuteieKpVw5X+XxcJiUbUSyUSIaQRKQYp0HdXuv2anmq7swro1+fadQv3Lz2vygMw6wx5RzmRQdwVQsQ5312USlMbRTOrbTqzTqwVi0Wc4kKcgjPfs/9QDkQMVCsiAgcJwRkDg3cGEagVAxg5Kr9Ve0RBhT0gjUakS53xZ+vLZRsUv6RzdVAZli+AFWj6S85JNuDUJhILLdam4GqK40u3j6nnnSHt+4tkWKN1JNJONpHDDbrVuhUBTIocwK8FSuNBEzmSLoIGHawGkqGUf0BH6w2hACALDBhQOTbHr4crAO6IXQVtYI9MXGQcuMPICnNNpZtzeA7opyu9TYcKXDM/h9qbYBqIsqVGMM0W2SlZFLEpYB+AciyasLv46/OnSYLPdsBCw7MrEKKFE8oMkbTizODZuV7PWLxUK22cCC0oeNpxCUa0CqZFnw5FUFQ48G0JAY1Z6B20RwdfWqfbw9ynbvwIN782Br5O9Qp2jxUCp9KGFLswY7nfaGbw5nOyxLzVa/AUdDZ00zuGFTTBo3adEdqldBMUnHi8ymNQG0cakmiyvJQ/FC6oOh5DSmH1gwQxcl+KLZMvpe1lH/tJxm2dWKYdPMZe1RQ13NCYyzgMS6Aka1LrKZjDJG+98AiyXvmnP9Yj1y59c3A6W0x4YjB0XrQHrQ/RKf9wyEPMw39NBC3ICT+AtRXx0CWWWDkrYOOqJ09k72TZNspZ5IZ9zMlzvVTA/Vcizm8YcpNo2YNSIOnfAaRG4KW+7sFh972z27u1dwdExGbejqYtEgScoa9TIaO0DeWBhDeAgSXOcNTuINgHHe6Qq3G6gYmXJyv0CtED5wJSdJFfsA2zMd9MPwd1qgQmyXTtw5ES7dJHqSp1UA+yWUPRiA+sq9SCyYNKEYDRTtgziCLxJqYuRA6BkCHXoXBeOpugE1vzxl6lvVxN4SFy4JgJgr4qi6TbcbrAYTLsxuQSgYlVRNeFRRVqs7JAVpCCwp1mDQkzLQmkHNO88AIvIPQG7GZYiUF/2+H0vw4SMHjx09MTGRfuWFF5ECYQO0+ZwOVMoEIUiBZAgq+0GSrtRihnVAS2QWyEL07E+gDaEQsiSNGD9Jta5tprU2qymtrLKP9Qi/R1XBIf/u/uAv/uxP8KLc2dokOBdOCDKDIZmmjhw5sn5rFWLm9/hBXswD7BIEg/ciIII+mAfDDEg6xLkHNkMEmO2D5pM9KI8GVgZ2VSYbVAXECrSZZFge1APK4T6u16FJdfAAO4V9jYsVbgQoohHepZOQxgXummgWHcwDbAXQLCUXmoqBclvSPcZI8QkcT1zbnkxmh+khz0m5WCLAg6FD5I03n+wnzJV0BqaCglaan40kZF1G5QNalDm8WmUaA5SnC4aLxSrzxgHB40XWbDOlzC0lNEjUR0Q1a/crH/klSEq7ATIV6wNW1w7SntVLtP6knUHSp+I6RJcCQYR7OX3Yx2v1FokO2pW+J8Ir0Bm5yf6J7uH0Pfc4At7v+LZv3c1s3Hf27tfPv/xHf/ifty9uxfclsOU7IyhgRpNzs0sH51994dUPfNO35wvZcq7y8IMP/cZHf/ONC1cXD6RnE9Mra7v4cKRTnoXlWX+IVEgYx5pMzn0P3D8zNf2xP/1zgAlHvDz5ErK21Bz6dupyhaLREXaBfDZHdq1mtUkO1Fa12cckMR1lxXFISCdiO5THK2XYTdHoJIBBjG+nW7Z5G9uZtbWb1xNUO4+dDqaSG1urW1s3qqWdXo9MRyrmsbpSd2VsE3ON1CwZmmKeQHJ2biIS3XdoYX64vd1u7GwXc55BmQz6rWG70evuFMqu6GSmY3/94i2/u4I/Z7EwuvdseDo4ER0W8PZlav/JBxBwG62xLzg3T1ooeQ856g5tLD5Ywb17QNbc3zG2MupBjduxyRAld/G1J66WPGqgVGqjAMKqQTqss6lGzrgjmDh4AhP7yWxht5zrERyM+wVpT/OlMqIqhqCpmUkqZNy8fBPLLg4c0Fq6k8nm2IBIFqq17PQgZ17eRb1xAyjC9ECvkFwIJwkFVTSBfQq/hccGqQUOTk7DgEI6cPBEdsUfx1XOSpMDmOP6EXUHQ1h1eNDbIfmAmEGicUl3QqwvYV3oiYlsUB1s5BEyvlIK0U06INIC2APuwuaNQn4XnTOqQ4oPRwNeBU41a+wL6J54bQvQ2XmCe7sn5C/V8sT+h1KewBR7tZetbZRqW9PHwkvHJ2eOzdsiuJjfqle2nYGOfxI/txrWP1AjJkiQCAKIPBWFw0gB6mIyIIK4CrHxtd4sF4PibpIY4B/XBTWjE8eHGPwn1yaWigXU9jPUV3ublmENdFESnJaeLSqhxFBdCwj4osNQZQMR3Kz7zWU2PIcYWnOYiyLkXDSfFiDtfd27DqtgDhqxTgyXr3PEODW3h3osAo+xlBzYEAxIEq/HiGlxBPQbMcZtPJKgCnhF45NPBpx2odS9sUHMPYYeWyAa8oQR2IReyEdbIK9/tdDu+yKJ6WKp8tRzLxw5fqBel+tvKhbDfdVBsAhaDyd5wbCvU36HcG9iT2EMIsnEYnjgp4xaDC67vTEiBrzH7EmFKwZHiljWgvrQGPUh3m7wucwmdieN0ybZ3NhB4FaXI6CRAlrkqPVzAQ2FrKJAi/TENIq/pzH9GhQpXwGygvbaEC+44x64FrSFkaBcadWbXaLMAUbboE7xMAQr6m0DVbTRaLNiLcWQ+N1z0wnSoTU6/XqrTZsRMn7Lv0yYWioSZDJkR2wYuO8pS2K/Va/BnqIvQLtDbe1QQJ5WluYTIguWnJ6dhvpOxaYhw9rSGjLyEcp7YrmkMYdIcIX1hXeWMxsH8qbSF0tmFXMuGV00QGyhU6I/V1hPGCwkKHVNahtyfDmJ+ZlIJQhqgQ+Dg4YI3XvPPb12K7O9Q6+QfeutJjkjuS52BBOMOQBBBsgNbGfkcJgzxQBgr1HeIAPr8KD0BAURLn6DXoOEmd0uK4+pGV0QHoqpCT+zI+VKv7+9UQ0FbEtLE2jjGR0MEY3zOOvIKzhQzIpf0Fxq+PzEHZxwnVEjJZA+Fg7mrrvugp94/rlnMWsBykAy93AD7fAIk2btB+sKF6014sYUBGObsuo2hHKGLG3wm6y5xsOAwZoh04QdNICJwZmIxWjz0htr6ckA3tSRiJf50bugv9r7RpxiF7Fz4Mgg8ohLaFFgCxQmaw/6JHHc/ZbTp+++69N//alisTQxPUEEGhbyibn5Bx84u7FJsfddEofdffZeqrHmi4Xo7GR1Mxudm8nlsxiIoskAKc1eeeWlw4ePMlvEpHW6dRSbbKypGX9v1CWSbXJ6otFuYNIGnyEkubyBeqN58q6TVKhcPrjv8s3LF668npxKwiBkM4V9B5aOHD5IZBocb9AdXJiZJ7Dzya99oYdRn1BdXJtH7RdefubqtUvT0yHYxHwx1/YOHbHQjerLrLQv0P/O7/gW8HKl3Tt8+GA2u/zGG+euX79gvNN82UJ7TOkhcOa4ObmYpkOlcmFkC0ddwa2tlXK1HB+15/Hf8ZELwj8Yeit9ezZPKi9Pb+HAgUOnwQebX/naF15cmXbafviRBPP8Tz/2SCy8HohUy6NFlDwlGLiDigEHMLXcOCwYSN+9AAEAAElEQVSrs8RfUCBKZgNLoM1Wj+3TDQ09pDUH5FlFNgHMNbx3D63UgCSUcbaUrX/T4Z86cmrqzM1Df/2pp9PJ/eVim5ggTFiEEhGpjhcEoX94L8/OJlkvuC6oD/CNBBgIxzF91ts9DBwra5vVcpkAAUiOAX5qREiKwC0rXypmMoXZWTyrc8AMzhPs+mZdMir/XBhqsViASsCbmUyOvRlCExEOAUyUJxsqzyrDAaZBqvKLGvTrwYA3moqpXIGNzPS1zM5Os4kFIQPWiQRI/k+SLJQfNZyY5EeljJ7ysrBomyYSzZVzXGpVHDHiqwbF0WYut24Pt488svC2ex+zJdq2UX7Qv0Cgot3XDcSGdi9F0xEIuvLQV54QeqNKzSAQNy664DToklrXGsHF41qNq08HtItGAmUYShqYdGKXkaHQWqCFMzZdUN7e9pbLBstME6yiTs1+14qKzOmLhQ1E1s2hnc2E8Dg/mU/zkPntzs3mQasxGt274faJdWWvWXXjTmfUEV1n8TRn5o1i8nRV4jqGc1AVzI183UyzCIrob+FAoH+OkbvVcdQwfDXxc0GswWTfzFeafXLUee2UrKYAl7tLfpJhOBZp4iBtdxKcGhgjZg3IzYYvfZLMRi4wC2qPZhknSZS6jgC/djstUs2Mh7hlRBLxabs7HqiPwyGvMxFoZDw7zc1atYy/F2QYWutSMlR55NJB/qMEJq62LDxxjy34Jht6D8ZIVjJTVgsvA6fT74kAXFAIgBInwFoND/8SkhyTQHpmBYZJ/NJWxJBK4YBmy1arIvD0gmFUkXjN8Cpyw9Y7jXHQ1UQNFwVp+gPMGGWRKD5RwUGl1ivlSvA8KhiE47GU411KdhJ/gBMD6izC8hA3JX4xBhwIPFSERRQmEWnvwuvnb964Af1grlkd4WgoqYf8U9CRBK5Y5NTOgoM3tkhMgQIz4DESMmKcwRtQP1YHDTtoHc075UuVYxr+DxSj1dY8QR9h+aC/tI0Yhv6WWYKDoN4WPxIIQfQRziFQLLgUBHQyYUJULlLIgYQhhrxB+SzixwJgpQZ8mViaZ1LpktmA0kaJFmtHcxj5zyA7ESQyV8PEOcYEKx8+cTSeTtfrtSce/wJ641dfvIaX0MF9c0VHDmKF5IocDYzCpKKoQCjUghpgNVWokOsFuOxMeA+mkm7Aw/mDZJ1ws64o8ZLpFHYyXsdP2hS32Vba4eAxftKJ+artoA4zQ8NKoZEgZNJJnkFUfb2HH364Xq2fO/cyEgbjYRQc2jqgW6ZXn7i4OUgQtr25e+zofJnX11uRKGO2uG0heYwFCAjGzqwX0gK9IkEvryDWmyswXqx1LBKKxyLhkIRvOMFcLoMb2LPnnied0Fve9tin//ZT2fzO1Mz0//Jj/+LG6trRo8f/+5//2Y2vvhQ7OQMLloykVq/fuu/M2W/9tm/7q0/81a1b6wcOzmMvD4Z99z505jN/86mNnW0y7KO0wlMvEo9NzS4dP31P0odvJFwrHCN/e5R6JI57yj+JgrqYL9sO2FLxVKNcTwTjjr7tyNzhGxMXK+XS7MIEkHz95rULFy+gTKk1O+deeuXB+856fO1KZ7NU2Y34EqGUszUuX3j90pe/8jz+i55Ao9nONdollCvY+smOB5dZymE9AVFQBqObza36gvWpcJKItX2x0OL+xZmEf9wqA1QJX9g9IjuHv5VpjuyhVXan1xk/eKr2yuveADlW95zpWMT/4QGivY1JhUfFlQJcoEGDL/WTQcWcsCgG08pGKqcZs5rAutYVfke32pq5PgSKGDDUZbgKUNITsw7+SHC5RsWCH0sOvShEwe73PvjYqYv4UN0sozRhttmslVoF1s255aDmh99P1KxKh4EWZ+GVFha83kBqcpaE5dlihcCIUKgMl0OdQCwQeGR3641KHYeRBniQgga4cQFRwFWxmL9+w47BHVLLJgV7uIhJkgnWgcMx+dI7eFo1uy1vs0oxB1AhogYoQOoyaqRQ1sPrnEokUH2TuJrJwE5SyhPdt1kpFScnkkQ3QBMoD4OXIkpG5bkJ+rtAsDyh2EyiImwMlNGEa7qjtt3GRrG26Z7oLd0f2XfX/uBiwBZttvMXfaGxi0ykMK+9anPYkNMt0gO8DeyotGJSHSKK+9BfEzalJegYuitsCXJD6MFFFmOADuPTpBMiLdlmEGEkSxbWiJ7a+UJ+ZrnMopql1M4U/uALSF8P37lH/eebuWZd5MLeVy5qgLd/te658+zt2yx0++YWdMWamTuP0D1jMRSm5ACJcYOAURhPvr7i/JgT2bjpI0TBX693UKs6HAFy7+XzjUZ7FItPLkwtPnTkBLGbV165Neygo5aPPhPgCXg2dmvuEGb4MRIJQYDhSDSTqS0vHiusndvN1BJRxCwnRNfp89aqnUYLuomHD2SQ+fNgPEWORn8MqW3lt5tQnB7omAwu/IruBS8ibL2YLEknRn67HkZkCDDw51NpJsWns4x8Jc8j4prsCRiBWTg5WknwVXEPHxnTYhBIwJcNADqwVPGAkc8L6YkiE168eH0ngwgEl4hLC8s1KBfHx5bxSEyZfAUBqosQQYRESgEvHPYuXLz28msXdnMUBaGG4Ih+oj6qommEoFJUDAFZUQgm6SbFvsmeQVgjSmxcl6PhKiVvC3l/MGQcwWRqRc3KmmKPBe9TbBUHJVEbUzqJD1aNAdJtEqlzA+sI1yRYMY6+Go6R+SA8LCILCckwq4+hSAgMcgJxZUbE/bK8Tie1AVhqamDzLFOL+pEKpNcuX9nZ3sYYzCuwjy8vL1MTolKtsqYipUDMbcKmvllMHnp+biUSgFfrrcJtcNe8DlaZZDyegH9mltq1k9jzt3a2//1P/zR232bzP117bUtRyAJEe7U6SMbllarOS7eBLAK900JqaOairsPVMF4Tp0yUo/pgRH+UDdeuXeMc3T7V0zi5s2u4H05aXTKN8wij0A3moL8MazIahQDjSsqUgqZ4DX4mluQNS2me1baxnsA9mFxGlXIl4PesrW0iXGgA0v1DmM12NvdxhcNwK1ooGBOAGTmC0HkU2z4/qnjn15588vXXz4Mko5EQZjpSmTHzp04ey+5sYncDtyIaYuy6dPOKi1oULsef/Pmf/8qv/MoTH/tLVHKjXtvpGryUf570F41q4+Txo2AkCmnAlD76zscefuyxv/jYX5arzUYHHUaenHRwCYFwbHFuGktq7sZO/nJ5O5ut1IjnLyQnJ+D6cG9dXVlD5+zHKJlwVpulqcMTMxMLL17eWdwfRfV9a2Pd5XOfOnN3p119+ZWLweilu06eoO5hyBu6tbMaDiae/cQXzj1zfjq1r1YvOf21aNIZSwZy2Xo22xp2sZL4qqU+3GZzxh6Ex4JCyCjifvRt70i4XZGwvT6sdfotis57vKH2OOAMTcUiTkdoMr9bnYom7jl19mo4PdPdsY9XzdT+T3wASbyPd4rEGhkIuLHkK6CDaxyCNt1nXKOVERNlDWAiXgq9DggeT88GoWCOXjAydIZxaYQ0A5JUv+ij67W5RyRDGA47hB4Bzv3mdmLpnm9+/3t+7X//b9MTMzvbeQkCJJIu41U3IMQA0Q28RK/g+YS4TC3IcCTYbPbZqg6Hb3a2R3EkkBGvCARJxwHL50WQYHvjUgA2g/Fjz8LTG1uJHZEV4Ce5B/pk0j9h8iOZpV0acG8Y9A4MZktZJgB9TMDtJ3AIfycCErAL2oadcTm/u7GSy2ea7ToagEDAu7ww3WnUETQZJYmtoqEgO19JvCoVUmQD4Yyf+WH/wJHBvnec7Y3VlZmjqYfO3J0+7LbNd22xWm94s1zGtztAPehmqYGYovxZ3mC7AzdBvnGtBdSWzUpqHaCBBhV3Rn4OqC6mRSPs7lUHgps3siNTxoIJfsQmgcQ5p4/8p17tbezbsGGuiPRyGztYS6xtbBFcMwpDXDUcfr2NFO583TPWyu9Lm1/3KLaFt+gF5oJ1wk+6pJ9MO1//zXpQb9QSgLeR/HinwE+vQ/PME3i1iIEjHR+X0dqasBwCcpAUB5QhJQUC7uNNgjPBip7O+c8/eePGdpsJdIVRSOA00yo3hmWbP2Ij4sXvSvTHgV4dVaEnl6991wfe++KgmN24EKeUMwn6MSUSaV8pgF69njD2W1/AiZ0V/O4NULonSUaW9Zsb434Ngup2hcDDBDGS5wURLUJtrHAIiaVe68Dt4GjPga/rvgOH0YNTPw8xgvWTnIaHIu4tUHOnG10fhXcJh8HbAGsImjSWGxUw6BiSDzADPSixcW1wu3zTU/OdwWhtbefcC69WG7ZQxJWKD6NBZ8Qz9hL4TFahUafToOonyJO8x+N03PfOtzy4m69cuHT9xq0GCsZQvJegAoE8x0TqMFPTDVlLKK+pkD6S2tr78h5QSUpKDyH/NBo4WqnDLBAHNIlPbDwZZ0YaoyHVsgnuQxUB/MsBijtZOz5FI41Uh6jLoNjGojGGaFnuV1zkLo8f2QvaasgS9MbK1+EgFQA8B+zWmNkQgur1EVYIDIWzQf9BRotIIEhVFuQnukfYA3I+yMJUszJRTvQBcDJUDaMmfWBDSjnOOxEukY/psqZB2fjW1m9lyplKo5ldrbz+ygUxNKHo8uEUin4GyL6GCUfsBhrFBkqEVQiZ+AN0fao9LiUyH2BChgzAAA/RhFG540qACtzpUOCW3ZZIxDrkDDI7QZvB0F2pho0hWH9NhzlhDjlAxIuL0zdv7iLS4LDKK5544olysYxaT4PVltGTYiY4McwHzDq86exsGgqNM12r3pqdSuzslPz4booP539YIfk3iu6TutL4gqEOJK8nanTs+UAvjTPMdCKeL5ebJPsJdWfnJ4BnKiXs7GTwUihXC5lKjqwNFJR7+txzieRkKJF66rlnv/f7vu+1V1/qtmrAceZadeJA8I//6x/vO7SfWdrJbMfSMfLPP/6lJygqVWsOyo082Y5ID7mwvLR84Oj6VvbSxSuDRq7fJUYYSZLYTvjn0dbWzs2b1x9+6AyLuLu5tX9+CakJHxgUSO1qt5Brlmt9/CqI5yeL9czCvNe3j2gQUt1t5TdnJxMuH+UGO7Vue3XnKmrF3eyacjcMiq2RzRe0+4LoSELOYLSKE5MDi8wgv0v0Qy+cDKB7g1+PTi4X8plLK2tE0yD5xqAZBK+4AtsbmdrY18o2eq1BEsWmzZEKeCbcAQft/s8c2hJvOrTqQvkWyoQNMHtJXKNAAiWkchIIsKENlAtUXiABHvQYYwuL2e5Vi/WQp++Jq3ShyWxEAJgoqSCfgEQ7OrYyQ7cNGyfuveuxt76FTG648EN0VZDWxK2BqTBMAEgRPGjkl4opveXt+Nh9gEOnB38WgSRj3MUZEz4sHktHwkEMwzjQwMf3WsJiZKcwYDmEYE/OTENYb9y8xS5xtbsVGBtipxAtB706YMwOQYs8OzUtdxL86RlhMV/I5Qe91qjdfvaJL+DTKXOW0xlAKYkZlmIT7Tq5nPHEYOiAPRKSfGbQXhL+AQKDlYQY2FF9tjpYr9CuOxr7Hp4+ct+i61TS5sWMsdJpZO3Bejg+anZznqATCQykj48nwSkYIVMpr8LmoDzYhiCnYA9Qfh97oKoQgB67LdXoNRPMrpXuQtoI1gOswPIZnbMoK6Lt3pbWmt457iBKfNCEj8yuFODcBh7rBi4YjHrnOb4yYH0VtrHIpPkR4LjzyNfvftPZ3q+WII5kbxrW3BkZl8YMMkB80A/CEuo8LzNoEzs4hgibV4HZIndchl7ESqV2voRTFNTXSXbCjc3N5nDTo1panol0yjt2dypl0ImCz1zQnnCh2rR7iTpIhULuaqP97LmX3vnofQ+/59uf/ngRLS9CdH63Eg25W81BPB5WYk6iogN2iFqxtuP09WPx4ZC6yX4v5UxJ9gtd6nSJzmygh4A9SEaS7gh15lodDMT4eCp3YIe0TYRu4lmAZQNJCc9SH1FG7IzxuNpokHAgmU7g1dfBWSq/U62i2+niCAaWxJ2Q+mBMTq1GMsQaRkTejGtMNJG+7767Tt51Cj18vlh6/eXnA4NqJABXLpkMj2B0MBBzCD9aKZR4gML89NKZkwdz+dJupkC1ve1SPVdpVKo1VO4O0vKaqYaRxLEQaZwMGGj2sa3iGqGmIISROOn/LP5ccTi9PkHClBWC7QIQUfxA5+BXUcwqOs8TQYFoWD4AAgUpEVMSnNEVgfNQT4uRZCMZhoylBeDEc6B3gVEi0kzqAOJqRVfAE5iMAv5IEvdLl2d7N0OwIqyyjCaEujqc1A9lWq5evZovFI4ePbpr28GNlrx3kCWIJD1HL4z22+vElV0aaS5BJJkcEDeEF/jiG6GQOH1g2UsnU/fffz/mCVjev/q//wI1PZQJkIj4fY0a6bpcLSyv4mBFFOWPCA2DP6LnumZgVTgS674S8ni8o1qtSvEodcYhaZXZAB4oFdW1dpTwrs74n8FymzRVAngd7BsmiQf5H51IPO5B8FVcsj/Ar/zESOFnuE17XQfqPcIHeX6IUDk7O7G2lpucJH5qRKBONltCzcBeM/4isiVxbnrBECho2gLvgc7bnRYNwguB2dF2w8olU1FmY3ISdW2rkCnCVUA4EQpZmaNHD//cd/3Cb/7Ob924vjK3tPzzP/8LyfTUJz/9KVKGESxeyG20u2P/lKvabCcn0sQ/YgLklah2wggrTu89d937hb97igEEfZ5mvbe9ncXAWCWrQbcZsvUJa8HnuYF3YdA3v29hciZx8dL5i5evBu69q95uU0cHddj01GzSMRmgcEitlc1XJibkkdcedFa31u66+9S3fOBbr117vVHJoXu4tX5rfmpu4+ZWprjd6o9OHDq0OD+drdy8cPlquS6PSNzmGq2yjQI9JLEMMN8D38AJUUErgl90qdGs9UcwIWNvoud1oIUI9lyU2yONZaXV2C3XFqbmKPOxtr4+auRTfqIfiHUgPbS841h0g1mll2RJgXYc/ziTrQ/IkU0N5G5kjjejSkOBBSAsLIZHcKFZZR7hAqfcq/UDc8JksosElhAPquQhttrqxS4Bq0l/FJMS3kCEUzZbVWhRAH86CcQ1WBuSrFOcIhBLfc8Pvvd3f+cv9h+cyWwXVUtiONre3GQG5ucXG4qDUKlvmZ+Qe3AQaaOTrpJmgkQLwXDi6PGTgOILLz2fTMSghiUsoVT+cDuDhC3GInb7AkXVMoVSMBq/Z2nx0KGDvY4cwVxHji8qqTNQKD8l1Z6Bm8Vxbuv6eR5D5bWzSzrSFS7t27+UPLDfPq5CEdBBk5oZPQ3bGDQE4yEKSdYL5fHCepIgAVh/gLkw3nf0q91S31a2hZt1+07NnosvBo4em5t/4LhtNmzztm2lIo5qaIaoeweKIc9IV3w7mxkna5/UW0wuvpx4t0CoRH2NfIqjqyovjagiah1KHGZRVREr7FnCCuJrecBgVa2TVtH6trfntHwGj4j4cWZa0DUDKWZbanVhqTjANWrOPGr9BAyAMPmJE91nWSl1rzlE+GlIzJZ5lFvUtq6oUdoQBO2dcg9Ih+/Kb8xVfnKhNSarJCMFrsCL5L4g9Fopn9Bndd0k+6vjX0bMnTfURCpzpprtzPLykWq9u/Xs835/sl2h9nadLdWzVcnbFI0Ew/6UnJqp4zcahwKhkZuMcq6F5f0TM/O7mfxHf/v33nlqYUCjbSEdZ8/fbw6pNTls2RrNejwZgE2nsEUk6Bt1G95udWr50E7l1mYpC1tXrhO4UQbxLcwtb21vXnp9pbmsmkRDpYIjxVYn6HHNTc/gb8TWsHvCJOdutoBtXJYgwLilTAGCWJ2R/RCV8ezAkIzPg0JV8acdIdBC2gFDQk5RgLtRekRC3n6vjkem2xea37c0vXTw0HJ059J5F5wCkcEkGyEKQdpgaE8Pn6bJpJd8Ye12ZiKaPHb2QK+7tL6daXTHzd7o8vWVC5dXap32bmE0vTRZqDWxtKIhl+hrHwW9YUnw43Y0mbK7/Z4AcJ0QEwAjQ+atwdhvc6Jukq8BPv9C/1JOIGaywcQDWuCB5QAcIV6NawIjTjgEKEyK8Twi+ymglkjGsFXjmVGtZoENaAzvdtmo/eJsd+pKDGVzBEMeAgcwE0CMVdzT4chhncvsoOOanZxghakx1yQtoSQE2J82DAE/yUKgjYTHnug22jQkcaErZHDcFglu9nmpCFxuVFauXJtIxN/x6KMU/XvX29/19JNfI99HrVjBxpOaCOVzGLdQ8oFwpPOTZ4VDOcIwGcDTi59Uznf+0xiRd+G0MNPjXo4YAfVHoQFD4wsGO6AzlV3RFMENmMHC+YgXYOromDGzaNux51h85YGjFg3651ze5/FGwmHUeuTUQQ+BizUiKY0jhRuiqHthZNGpF3MV4mvw1kXdjQUBjT5TSimnZMoPr07Qsh7A/CCPslG9NQo68KBBfFDoNgNBDnYRD+ZwNGod9E6dWk/unrimoItqDR1hTGS2jY2dbqt/YPnYxso2WSRff/HCt3xg+b677/mxH/3QL/ziT2/urswvIc+4btxa2SmXvK06G5scC2eO38tnaSOTfiTyzQ8/9sRTTzWzPUfQTikksppMzk+fPvbgM3/92Uy91CACSH4v7Vq9khhFlg4uF8v5v33iqf0L0yvZ3btO3IUv+Iptu0S1eh8SrmMrly+3a5Dq9FR0REx+r4YzUbOKwslOydpSsX3l8gpQRF3naMqXmI53XVMzc7V8KV/CnwpICyjXSTjsztX6Ea/t1MH7pmdnidSPpqL54i7qK/yzdnfLYUxB0Wi221j2U34gHyJjQPnGi+effa1n+673fvP80YXttQueAPEStiBL0RkEKZ/ntTVFQpFyMGRSLLgtWc0g9a6dPBjAJZFNsp4ADEYO0b3636BZg+KFMZF41Yr4RqFw5YISpYACcw3BRLWREOy1D6EeFVtxXI1PB1xT8c6g7gx77RhBW2ASHFKJYCRDAMWjEGZei86fuOuB+MUXc/FExD9OdVr9aqBzc6U4OzeIJ9KVciFO6e5a7eKF1xaWDqCsyOVqEUobwZAqmwJStisYS167ddOnbJE9n2s0O0WinhAez9y2tr1N8bHdYvn1Cxfa9cLxg/PE57hefvGr9B3EhlgNuy11DKqXfrtWKcOwc4UREM6HyHzzanHt2nnc+jGqwWuIPhgQh7fG0bleryzMz7BNMC8PetVYfIoolM3iRt/Zi8/6qRh4I/dSYKF7/zedmjyaqg9KzeFrwRYyDUxGtu8oO0Yt14AE1k1qvKEbZPtBNuBOha5AIdrEcEtKZ0K9dfqnGxT9hACs3f4PDyMZaG1YItbSnJiVNGdcsR65/VffzLl1I+e6hX/Wuls3/9M/39zsP3zq9tuFfHkD7wG/cM53uAjmhHNZDLVJCaOiLAJYXFIyQgG2JRQB2DhQ/JB7rT1w213BsTsiHq03zObHO7vVag3my8diNhs9oVeB5YDYEtpy2Kk9GsWaUK420niUTs+HYyl24ez87MlTd737sbO7559e2VppV2t+D7SPED4iZjuwkxF/xE6DzSpuVuMIhuDIkPD7SjEUcFOCGsvIBJniIxEyJMMrUAukmMuWchWWCjc9YwqUdqRaKiJ7hVPo6iIovb0eKAeZawIoUbHBdlo1HPNJctRsltHYkMUbxOcNuFvtGlZe9qYfOyJ5Wjr4ITW0d+DTmCowM3JOvUToXate7A+agk1gRlAjSiMAcni69AlzMZp6tOSNYW6riWoa7nwmkaJOdSp6Ymlu+trazmtXVknYh7fa2Ocihw3qBWzqI9IPIwwpOsCdqVR/8Rc//MmPfZx6fDPTU4Dv9c01vGnwHYaNhWuUzxGMlPaRkeRwVjIHC8HBqdwZQS7sOhbd0p/oBmEY/sVi8Ua9ieiJqy1yHq6YMJPYHTvo0GW05XGd8IARBcgYClDQjLhCJgI+gfaR9TgHJWFH50WYphA3uQNLErNi3iYY29sVXDBxdNiPU6l0NBkh+Rc1FZ4/d+7ypUu5ncJ73vO+48ePE/ECQaIf5XIDbT/MNtQX3xIp1pgeOoG23A7z1MEdntbpng4UaHLRgs0SWWbMeNTJsc04MmDIB7VYjg4MCui2pkg9fNOhZszBr5xbem/OrVHTK1TQqN4l8TcagCK4gV/pHqoPw1jrYbWBDUdoW/sOOzJZD7ikIA6kComkttQEfGAQ1SI6PfgV1BBobqQtEJ+hRszcWWeQaNhoASDdQM0e8IVeeenVicn5VDRNSgdVC+71//SP/z90LxqPo6U6dviIPxo9/9Q59uTkdLqQ3X7xuedPHz8GffrPv/sHlL0kezKJCMmc5Qh7SGGVSFFxY2Jhfnnzxg1yoqFE7JIsq9m/TjrMkOvEiWOHDmHI7GbLtedfPf/I/QGyJJ2/emlylkJHC4cOHrx58xrhTPFIuFjaaTaK/TZCDO4jTgprtqkRW0bCh3MbXLxyeX2TJDNNEm/Nze4fjlZJEcHOnpyNknnmodMHp2ZnmNitnSyBzpFwvFVpHTh0EIXU1etrN26ubmY2yS786sULAe+IHMVEKyawUtptG9u3SLJmb9VmJv3Z3cqUzTGXnqjmcu2BLToVW1nHJE9UBRnK2CpwZVhUpM2UUgbmSzNt1osJNytuLpiLLICBhTvXQZdgQwCOBTerY90LXYHZBQzkGi09ImIAolp9MA7SvH4l9QCwyWuwvRAiBxnyeFLd/rX3fct9WyufJ4YXH65oiGwnV3BTAQrKZWCkBg9L+knYPrjrWg2CJRaT/ZUOp3AIQCXNpkddMZeMxCLRmVRkajKNjNbsDtyB4JGjJ8l0gY/wzuaNmB8kWUlEA64XX3qKJugHDKY0WqBruIsh7w4hC+PLiT8mTjHY5BiZ/FThIbGpadAMy2SeHFHm3RaPTRYKeOt1ZJ/zeTL1NRIaEC/adxZe2X7dGWo9/P7D848t2KJwAyt26aIJsgLGketaTldzjFHf0WBPgYJxSGQzgDyZUrhdfFLBo5Q0QwcHwIF/yRAp46ghvWZf31mgr5+wQmat3rQo+vHOXt5b4DuXwIR6wlpe0WzdwD9zUU/+o4d5i36h3Ted79379ZeZC8KbRva1eALrJpAn77Fa0As5ldAjzCyveZgRFITUuCTHlYQZW4tgXjJDoRIcjxo98qQ7CRLDfRUHvEA0bLf7L16+ns8XpNOx2ePxJAYMtJpYR2gLQEFE44QDE4PLFwT4irlMbX0TGeLuM/csL0xN3HNXYe1CsbBrJ2QSz13MEZ6oy03wD2Z2cq9TPU2dw6mkSrKOG4Pd4jZOyG2cQfz+arlGJ2vlCqUckYkJKIePxjOZ+8n7AwoGP6Ka8EBylfubgtdSe5KdCRmvWmqyE9lDqPRw9Qc20F7ixydqhReIE6aYKDLES1xuccqzwdHXMXSjn3WMcBbL7WzDfrbrlUauEKSiZyDsRaGqmBt8msClnq57BGkHhMkugA6wUirwO1iv2anXWwUC0qcmY2ie8Xt89eJ1rK3kQSF/OJ1DzoYgIe/iQLuyUpleOPB3n/309uat5cUZWsPpFy0RjAX0lkUUpkZCx6eMbQ+pp5HbGERkA70oClu0z6AcBD5+1WoLcji4wj+8vbgBNpRYCNaLDuVyBX+A8uJy5oIY6NOQbTCO+nj74O1Wyyj7qYaGIZk2AwHjzMW2URFxEX5mXHwrKEvgKARnwSF0lCsk2NrJbGnn+ZThDrfM6bnZNy5fAgWTeJqbWS+Pj0QfMqOBN+CcJDyLsgOp2pCIknwKoDUyvcM8JVMxB+DD1yEZf1g3OAuFKOopc6se4tc7h3XdusindUAReS/3MG6BE4yR189Xq31+QgjWa40ikmiL2/yGGgev6VN03ka4uvVqDRaBFkUkRVGx5B04hL9McwAAq84xb2FENAvHz7PWjFk9pB2u1HZbgRlsAeO/+fRnADAVZOx0X3nxpfe94x3BRHRp/+LG9gpc3Le9/59hO1DyzmQS/pGclJFgPOR3rq2QxaUei4XyhSqKBPADgmYg4kXOwfgyNT1x8vSJG1cuU3GOSg0izq4gWSPwh7h2cc2PImrIPnGtXr9cL3fIB37txurC/ulKJeN2LHvc+MYM/EFHb7cpS1WrizcPs1Ip1vLbxVqV6Dpbm+04bnXbtg4+YuTOI60zmjS7igpQr2lRoasxhH63vd+sdlavb3WmcHEdnHvuRRQlEzNziEcbm9tnzpx4+zse+fzffrxRzVFbF9zULJK3rbc0F0pGjwRln6nag6mL61sRj4t4gctr2zNL+1FLYAixdVGdsiRCswhZ0v9ZykYDCdYMM9vWCTdymK93TvZ0lnznHtqwpBduMgoQQB6apmBUGM9mHabK5ibeR+2xsvxgSDfbwd7D+NLtZ1HkMEkPPnzm7z7+ysLC3dHwVLF2AFUzOdRmptP4vmEXA4VOTk4Ch0y4l3hbn2pAhIN+YjJJHDuVnngV/ww3MZykwwx12v0CeU5qreN33YvvChdLeSakFJ1JsOHxsnSVq7tCguLa5SEMFOIkget4tZ7DxOLBixofgFqRTYvDKpU3JJjiruFE/ERfCsTARPhGdne91sJ/kvkrtWuqZxfoNrvFfLvQdKzvezB5z6On7SdSNl9x1Firj0tDD2HHETgU/MFJSOdG60y2YvAVSvl2myRKwpz0Q8UesMXBtlI1WNEdqBDFOhtXLKEsrYcW4x89WCp+18KYm6wTw7Pu3W52kM45uXNufuPef3hx7ynrzz98r54xTVk33GnQXKcb6qh1z+0bdD8HF83rpFfhNtFOusxc8NNtMgw8IXlyK/kQvT7ySLjayHoYeZCGu05Hy/Z6+6I3jB98pEh4da7I0jYbbW/AjysJuBRPScQB1Lao6Vh+MPvu9g6a1BnS6k/PxePU3UsECNTqNArb62haUH4gZw+7JFFBFaeEUZgboC+sCrkzqG5ILkd58WGGt4+IbSPnN0QBIgHOYraRKhBy6tUa4i/0AONjjKwEuEUTPwTNJ7xWRV0hVTKhYTsGk4JHYLZM2lUXEcjYOIBGcnu0hpQgjVIyk9SYnQY1dlDsOEl7W6+Xq9W6creNKeKGxwzgQ4RfI4KrBT1QZ6GP5PtAJwz/OMSbFEop0XAEjAHE5E8T3OOyD+FLT8TTE9OTU42JyeTq6tV6TU9TSYxgtkIWObtOlvxkyjY1NwfDs7N+7Z67jjzzzHNMJlSq222FgoleU6+V2swcksFhXYwHIqoMUL68BIh51k7DxRTVEkBsXEdg4XG/ZcHlyylFcbvZcLj91WIVaVgh/CqUBuUQtEAJaN68iK8Cb2YKppjJZxQyfBp4N/gFWELx32EW4B5gr42SWyFrdNIgH3g/3S0QJCkHE+smRnbchjiOBj6710+cgVtUeXt7i4ULTEx4CKyodrAWgZdbtQbYhEeBDVYGyZiWoGSo44ABY1Y1rAFEz9h9mSvWGoM3n6r+gnoQvEN9eVzrhVS1O/ikTT4ZpuEx1LW9HupUB2+hXSnAMFcR5DvA654VJDk+3+QcpzwoCngTt8FtBD2b59SOOmQYAl6Ad0nAkFgMutRexfOc98Kt0hTP8ghf1UnFIxERDhf89d3KT7oB3GIfzywld/LF1dJKrUpOLko8yVl1fmmR+N1v/84PTM1M/MEf/v6X/vsnvvz4E3/1if+7XCyef+ENjztUKeaCCBUoD3AXpWFMzs0OabIbQFwT2/Io2lRRpkajPr9vMTWdYnWilFHHqYbdN2iUtmt4d4e8seWF+Xg8uhZcw/EgOTnz9sWFncxqpZh56eWnC4U8nDdRb+Ggh6B28hUyBBT12JgrZKpu2notMoGT865L0BMaikq5McLpoodTSCidSJPIE/47s51pdfqzM/PLC0soZi5fuoU7MNxYIkV+kwCZnog5Bmuff+1iq4v2fjiZ8jKBHj+Q7I7gkBb2fvELz064iP23T+07Wslky9l2bPb4aqGM8m3YyjWaChkKeuHbsKTgfQlUAMJ7YMk8W9NuJvzr+PPr12EmgXyzMph92SJ7DBeEGADii5QUohdDmAzHuOdsp2bj0iMqPAD2WIyoIfnET3oa9VwyPlMtb9x9/5lnv3Rh7EAyHJw9e4YoildfPIdibyKZKmNtHfbz+SKOoICH1BvkCw3i9Ozpt1vpeGx7GzOHOUCn1GoiH0CPEAN3tVIBD1E7hbqlVCLL5kYL02ncd1gisnAgaLGTFSOoSEyhfzsJG3qD+qDeIBcD0OklnBnkQdIrYFiCsjY/OwWJBbUR4+gglXrRE1Jky1az4wy0PnZXwjPDM/ek9j0wZ1ty22rni7kVV2QcTPvQzSOLyz1SNZdQv7MjyTsDM2qLhCIwtWJkYPcZCCIvph0Eezmi6p8UDUZaMLtAc/iPH7CTXz+sc2sLsTP3frjNbN3+ahgxvnCD9e/rDfxPnu29wqBjdfdNByB1pwNcBkvq07xUSIJHgBiAWkHOXIefF0IYkh8C5KHvRjxxB7BNYWBFLm32a71BK5p21TbywWgKrxMoIjtqtbMOIsTxZEjdav+IwFwYZ3k9QObhvJzOVq2S2VqHFJAZKru9cfWN805SUJU3Spl17IiQQ+gYJjQya8BKw/1JbiMarEsWixK0krBOMljhmENFPkCBLPZI1aAwFpWiPZjWyK8GuYFtA71TPyAcicVTCWQLpFJ58YICwaQmnBR7STo1yZ4ZCgH1QKYMDaCzu8jvlIAoSeBBAwJAojeRVW5Elhlwe4Tkk54AWXOJYgUUceun/icpLiBjBokDNqjeULXBooaALohSG4ewITkFgxAk3BaMNAV3TAhJsd7uLC4f+JHv/044f8L7rq3cIsoWR6GHHnvLmbP3RhNJBOXPPv61j338M/iJEQhZqdTglgbUvmxjbWUZIRvQI7ke0Pj/j7T/gLPkOg870Ztzzp3TTE/OmBlkgCQAEiBIiqREUpRWyQrrINOWV7uUrLUtP/snR0XLkklZ71mZpMAIgiRAgMjAREzu6enpfPv2zTnH/X+nunuGQbL8ttCoqVvh1KlzvvPlIPTXJOnMwGqQA8EDAsBQZtaRoQ9ZZxPYENGYnfhzyiQbGKvF5RXwfjKdcXkEEbMYNKoAdoKCyYNq4+0Uh8CajIMi9Exx0tAbyJjI4YTxkTxEgKvfpRQ2WAA9NOZzoUP4l4g9dWuj64jWrDZJreNwYidGgSfkB+2xwXT/Aw/i8TR342YsEhkeDm6skXm07Zd4HjGmwhKh9QWKealkLpNcZagY4ItESOVAKKpej0uBIpYiASuwF7BmHGTU4B22N40Ab/+SBbFzrD0ojtayWARHsfEKxoc9faBBzqjWGBPpPz9R17HnET6WKaA9GSiyHXgo5miXrsOBMVgmqUJRbzbxlIabgLWAeAOHKFF4ltxb8A3y8HZ/aB8CDPKql1HzOklViMK51W7/0R//f0OR2Dee/9bRk8dOnjzxb//jv33hc389de8BuI63Xn3zve96XyXTfOfCxWI87ba4M5sJsn5QL54CrdhkCGZnOUsl0lYLg35b3yrlUg6DrtwqI7CFbGG4F2IsA44oiT6MHYdN7/M7I6TBcjh9kPHdszMkmq1XpmrV/OLiAt5MmA7RCUl27ioq/065Wizn0Bbh4YPiVpI1EE9dKPa8eXwxCBbtBEMuf9TlDVJv2srEz12/SSQp+ZCRvA/sOUwNn9defxXXb2LQSaK1Z+8Bpy9kIxOT214spLOFkvjgGiwyzzjctprzy8u3UEObAkeP7B0Kuq+89vpQaLKUr37mj688/p4JS4A6qHbS8ZnATjBM1I8RuQpkCM1ibIXRZOSZu50hVz/VeW4AF25fYA2B05Q4Jo/xEMsN5k74WsBP6CyoAyMcVaAH3TrxGlApoYyK9PJGbDqkNO8E/dO1esLrP0SNkvc++cCfffab9UiT5FcPPng/GXsuXjiDmjAaCQHx8c3E8PAo2V28KKpc7ggWqEorsbq6a98+qiEE/QGAodLvgh9gYqrVusnmiK8noqOj+DaiSUrkNnP9dr5QkXyWUFmACfIu9FUYDvFFYxlI/VgUSsAzXsjkuca/BFGm2/M5/UA1axfuQtJKAtX4gsI/uOzZSqppLFtCnZp5s9xfHj/gffcHDusOuwfJy4XlhDNoCk5aatTaGjQAR3HwhU9ifAFl6umJawmTgPBtQ2jBR4J1BSCShIE3oM5nhbOMQF0MPYMrk8TNzMKdxUt/v2vbmaHvOlAremfudg7uflItWDnxA69qd2qr+u4btLd8/xnVzk4XBBdoZ+7ay4vkQY1Jx2FBBoaxFWgDUUpNXxIZgNfJHmV0GE12ypBg6Wo0+81Gv1Rtl6utjXS5WG45POkqVTnwsNIbCnmihrRKA3gddxC3CMOV4nBwdLo+DixiMKtWMuRLMVMr1O8Phe3mQbKYQp1CwqwWeL/bN+PFYrC4bR4SEtMfLoDASAoANYGhJE4HNYvNygPkiE5B72kT3AiKA8lBU/1eHKlwu23xEfgRQIzJQOtQSYJJuiUFNiXNnt3ls/Y7ZR4U5SgZNZH1iVQlTXSrRnarFqaqVrXfqVG3B2sIpLhKHXOLoV2nsizJuSQvtJGesqqwyIhFGbwOXIEecVdjvTRJxAXBA9MDUFRlwIJNKDH0gzQLoBi4SwAetCtp0hqlfrOcXs0EI5H3PXLqiffcX0a46/U2V+fE1cwXfN+j955945UXvn7mwYdmSDpCBq5wwLmRqiBFMT/QG01041tEpIeCCRIQwQh6wDUQPtKdQitMpoCAWKjVcPGDOwrZPEkqDh8/TrIL3D1A/fUamTdgVER9rY2sBjm0xhLgvBL2BKwVvZPGNa5C7mc9KRcjvlHal+LW0gaQhB4Lu4QmBgBm3ID9HXbK4jSh6UKOpJIBGTOoHE5/du3ahdaEpHqsW6eHoAlbryVlJITZQJkMLiDWmxdZibeWWCyonfp8tIqYsawozwSSVUfpgKAXeqAGhF5JL6XzDIdanLIOvvdAu4fb2GiHPV+htcPQMrxoUzhJIxwz+No93LDdJB+q9EsMNpZhpSpACw2vAGVlY4R5kAOU8LRMHLrqj/SN86ifJaW9WqU7faNlPhw+lb76PS7mvpOt/d5v/vbjTz71yR/9+Ne++exnPvtfL1+7Qq+Wr89N7tn1+b/4/L7Zfb/487949dLVX/9Xv4pjA+6NZEIvVtroHBu1vpXwNjhvAKnbR4tLkGd8bg1VjBNfVaMulUlKHvvOIOSPEmrXyLev5eaoWWl3m6OjUXyTCfjzBux7orvL4RT5GbDRoPnBXOfFDJzKwohWiuTWqOh6GGcIXvBC9AD+0TGMFMbcagZ8E46I3QFrQrklLLWYnEx4QrtioRhLzGpyPPHoE2ipn/na50nvdfXa5fhm+sbNG8jiTqeVhFBQqXSu0qpUoAfEsy6urjQq1UcfeKjnDb46dzEQG8t3LG9cvVky6P7gz1d/9seaw87OJKYVg63XgMMnxa2BHJ+kidRgkg5oQ70z9Ts/dw6QxwAWTTBj8hX5UvRAkWfBJ6wG2DBYA6afOxq6arbhNbsNGNVEd4sWGmjhElG45MskpCTjtGd1befue2Zizzk67fLtxepRl2d4JHbtKilUk5BIPN+JysOHMBSJ+PwRTMIOu43EkclMYQQ3qGjkwQfuW7p1cyAqu06j38zmSw4X8VAkjpQABPwVMP8Pj47ghSNZ1Vs9ie0TXQUzL1RQeAdGBdIND+6U4EtlNCTXARK0w0b4AHQXFxeQpLJeUgenMzD2KqWMI2qpdbJLxZvBWd0jH9w3dcivi3TqiXNmV8cfcvdw4Rw0Ie+MV70+sFqINEDviD4Tk6awKwSrS+J3iqtgkFcbKBQUIRQX3ArrqladrHahujywZWZX63aLsDEhfIHs1MYT2rGcVye12VXH6pSclAON6GqTLeyUWmx3Nbp18/f8swMlGkzcfVVrljN3LglBlW4LLAjFFflHvWdrL7AkUMQeflBKG5GSBc6eoB3KXbL2QCEmsjCj6K03sjmyAaIvRMUBGSZlcgsFFRBFhi5YFzAgvgNoL3FZA/XJHGJKRpNJo5AF/SCTSkIfcBywCeXoDXBSynSK7apT2SCKxUKtVA64vX2HnsBedMB4+QIJZpcVLbFgNypG4jjXrOJ1CbyoETMsLa1QLAgUyBuglOFgkHxqSoYYFMr5JuU/ZElIjI2kW8STErcSJC0U6by9UhNjnEFSYPJHLkiEF1JYNhuEH2OBhj/uOoAZQbLkqWw5sCSTbRw3WdaQSmMEYidhY1uGBKkL3pE96gI1uxTq4l34+4iVmVqK1E9mHhhoGW30cuBfLviUb8VoJCSsSasyoJ4LyKndob4B2j+CkM39RiG5/Ou/8ilz/zcuXV6ktDSrgCIXsZC9QkkJIq+VmyB0lDHH9xsCALutbbwCGUtD9KB48DsAAPxwzKYBHjQIJiWbL3z6V//v3/7Pv/nou97zyiuvgBPJdaepTrWnWALAFXuot7I9g2bkM5kXqIUCOQkEpA/ih4SuH7ZEMpii6DNCdYTSwUJLBxgEMBMvF+UAagPUXQg/ynepT8GDWqttc7m/+fzz+/bswe516MBheATyP8zfuOm0UbpWkB4TKh8g4QAy2NA38JrI4irzBpQW5RkhXGhfUF1SXYCFjKDMZEn2aKQPvLLFFUb6Q3PaUAiA3bVpK0h6qURebdzYcxtPQa44lj7ckaS1e4XSY5uWJccAASjCu8tJNLmamwKUmLgjQsW4gamCNsP3C3mWJum5yOsMKSoYusPx3Xvc2eCAyrmqxQkOM1RL3Zkjuy+cP/vGCy/+1Rf+au/BvcV8AQqErN2rllOJFGq+s99+c8/EoQ+8/4Mz47vefv51Z1inSjKkwXJEddXK1K1sm3wsqA5l2QNRX6GW93r1w7GIy+IpZMqNEoxoc35uwWV3+UPBarGQ1Cf3Hd3LKiZsrtVtgbErgzI5hyZHpgAxKmXrHZRB72J3rNUIHJPQL5AtcGg2OZGCmJSpmd3jE6Mjq0MLt6/hOoABlwTG7Q5s9CAaDGKsqZaLcGK5TK5e6R6fOfTmrbOSkdHpvLUwt5HJkpIThw7Ivx4btc2xvpkiFhkvY/JqNlqk8XE++50zlIcwtzb2T2Lu2l21+PK96swxV8dkZ/1jR0ORZe4BNAAiCA5PF+EgNcaQ8aYbGj7XRh4wUQeKGeLork1hU3W/IFcl3ALTGsOJIQ0wZdcelDaLqHmcXvhR1AAQe3CtIH9ynTVbGw47nkzLIWrm6otPfuiBF5+7RPDR9WuX6AZIi35Bl3Cvi0QinCHakKUNh0zB0Gq5WisVybDt8Hr379mTWFuxEZkUCsKq4l1A5oONRHJ9dY2bKVqDf10sNoQTVp2AEIBWVqKQBpw0xDij1GJ9eCFglGIrAAdyMItTeD2WCSwDi4dUB6IyExezgQG/927f1Ug1sxVTPHpAd+q9M+ETQZ0pXaquOv2kzaBeHTlBOphjYElpUJSBknKP3IlGPIyEKMKASel4yaPF0LAAZEpAj2IElYGStBNsnGIY6KyselkRLENZYd+3yczdRfy0nzt3qYvya+eAY7W+5KS20tRVaeT/zabeS/9Ah1u93D6jWt0+qb2Cz6I/QKC4oXUpNE2QCNMMiSWRBZ7qMhqYujqUuewOclkyTgK51KJgEqHSwssVKwWSoIIJYeERyHI5qTWGNlQswKJekIEGZ0GBxZ0JjS+oR3JcoS4lm6VBCk42q3gaM6jM+NjYuNfpohQ8SUh9rgB58sT0Rk33VgeMiljTM/bqAxMlQ5LlLK2VipQPchIcxzpHbCCQhttB6HaWaL3Fx0W7UDRbA7zeAB1UhECadCLqlUqkm4c0tJqFXp9YGGJewg6722Jxuv0xPrLbInN1ol3PIqOhW4XUoOtmmRK+DvwRaYieRqy2XSzWEAZUKPj4wOIKYwf8MDI4W0POAStUAPAx9WpTEOigv5ZKkAUMhTZZSZDiVleWKD7otFp9VjdSHb3kbrhdShTAP5B5nlmIl3I/9fEfOrv7ciKdRQjIlGpz841glLcAkYo1FDaI1YTDr0S0MJJwmgAVsgW6DEHuQDHvFiac08qfVi4LaCCwtqqV1157DbxGicMf+dhHf+e3fhuPNlGlszF5bMKVClllDpEdWB381EBILtK8sl+qW8Vcyp10AboCviCmWW6QpQX0y1Pas0wZE8opUmlSRMfloZCx2+6iGoMJHg25kMif4feMYDlADnDg7SG0j5Urq5FWaJMXwebhncAxI8xlGucuCBjH6EiErVF5m6WQitp4jEZkPasvoxnu1DqPDMBJWmC3/XHyiXyC/KM2CDAbNwih3a77xGfSI35yC03RiOy3hkd9LFybsuxSDA4NG3WZmjXyhNiYPThaq/hs47QkQVAMGp0nnIr0jFpnaIbzbAwpcyc8h6REM8L1xMa8i9cWLAFHcGzkve99LwLr8SPHf/u3f/t3fve3n3v2y516q5QtjkxM/ur/8enFG4sLNxbcYWe7XVtaSKO5IckyOWbhkWE8TCwAg5GydydOH2nrKrfXr3mCztmJ3eVs025wlbP1bz/3EqR14cqKM0SZ91rX2FncuDm5Z6w5KFP65sjUtNtsmxqeSaTXK/may+sqpPKZZA7WE2MfNKOJl3+l5oLeSKDJYGMz/vSHnjp575Fnn9VdvvQmRNrvoVooaWdIYljDTxlQTiUT4xNO6s8vFxavX72SLmxOTs+4fQ5nHcclXJRqlGt875NPuuz2axfOpeNxljEK00QibbN4fMHdhDQMqu1Zq+vZV85cfDv1j/73n/ro+9+9fu3r5sItY26l26w4jJRsGXBTowwhZMoFs2sDvjXT2/9833mBsR+48bhaCsISKv8uYElITCXfDYS61KpUwbegZXlaAAxBoN4jt4EtbOs38ijmJg8Ol/76G9HonrWlHKCbTWV8XtceiOt6PJ3N+8kR2em7PAH0BY1mAhqL8ezW/FxsZHRmdjfMq8HtQTCASDZavWYH7zxHIZ+FmSYvdDgQoC4hSjhSIYGSMEoJoydCE5FF0EQkTLLE1Vt4yBBsh8eIZBhB4QxGkej7hsXtgevEXoIIQkqXVrfap3RbN9EyJfecCp7+6DHdpL6Wu1xt5sJD/matwWAC8dAUIQDCrgrzDk6AB8aIBNeLAYrsWBLyLQcMFBLb1rhAKoTUyiIUKVjkWcZL6JniG9SCEBWCYAB2ss4gyeou7V7tgqBFbUZBDerqD9hpLajbFC5Ut8hJeiTyhDSl/WTCto7UOW7Y/imdBYmA9LX7ucTYate5RyGlnX5CCzG4y53yBcCS8EHgLM7zEbDeQnfxLCHWWwpFUoJeYq8GxTrhPZBncZkTSKdAofiqM7p6m56wDgYaZC46WDhipFs28QJC+YYTDBMJfZesTyQxw31V2B8itxGi0fqLQCIWVmz06CYsZL3AcxLVtNQ/R1tLjlcDGmAyxBKhoSuWKfKAK4mVcF4iJnC5wvZMAufNdIoIclY33qRE0cABuDpuojQBmMGtZdx6SAqI4ZkJJ5MEtBRaSOnsliR/pKMEJ7ksRvh2aqLxyehnCoff+6TO0Jp78Vl3IJJJkrGjSL4MvBfQbnMbvIj46ZvthF1VK3U4CWHy8IKEBoosDnhb8W2q1gZVQnTAc1IKkOwuBhJZYDoZGw4mUhmcxfp+Q03fdNoDBENR8svhtDFz4eEY3EJiYxMtO3wngRngd0B4Znbf/ScOQO6njxwv58uJVP5r33zpW98mmXswjyu4xZbNF4PBUC5b4H1QbhJ1Q26xDIm0rsz66JMQ/JgFWV4Iy2B81gVytkGPzvCb33yOgg4Oq/0LX/gCLqmsf9G9y0wqNbMiKiwK/MgEwIAdTgspYv2KyVUAElATJl2SLUOg8L+FlqDok2AHYXDlHiVVA9rSCLdBg4UvMJAFjFQDkh4LB3suzEzOfPQjH/nVX/3V55//Nhk3uZM6H7iWy6qkJ/IyYR5kzdAFMgMpAR3HeHqFUhqPbocDp7kKA0LlIqR5lOrw8yCBZquNuA3ss2oUZRXqyzHUGqTBFyF23L3ieIN8zvb30j5Xte/lvPqJooRJpzGJ6wVfiUSu7mdPy9xMI1wC/gQbplOcd7jlAapl+yJOPdlIm01+chs2YIZTgs6xwElibQMokRftNEIH4azwUmFlUWTJQlSBwTAxOVYu5n/kR37k7Plz5AodGx7BY5FsJIihkZCFRC4vPv9ibiWlgwuh/iBlApT7AuLP5Nh4upLLVQs+8vDXmsl44qmPPubw9x1uG3mkl26uHd13cnMz3cLwVK+6fOLBgL42nyt4wmHyD+NdSOXBvCvgGx6jmKxZj3fQoF3vXDh7PZepskbOn01FAqJHZMKJhwcUkIiW1xZ/5V98+l//q39+9Pjhjc2FfDbBsFFUYGpiEifHzUQaJn3h9k1o2MT4ZKVRDIV9RCgNj0986etfX41vHjl5khgbrSTfpfMXJyZm4ksrmKh8hBUadQREVusJwGs44I1N7vqT/3FueV137vp8H1ePxFvHh02jYvFECCMddQutMMwVgjD/q5lSQiGTKpvAKh3nmppuMJz6IQhTQFmQtOBkOC1WhUK4gLTQDdQxnGHqUcygje55PCYKRBL+ZIHQtstQFCPuNF0CvHReDx4y9XYrQ/IMnblWTS8++fQDz3/pRiwSXm/V9u2f3rN7Frs1wGMv15YWV0fHJ7Aolco1cGU45gPY8uVKuZB9/eU4ywe1BCuSpLrk8cWhKpXenBgdI1VRymSKxsJgZjQW8WTapOLeJWJd6CNrYouG4R8qRFC+HdAAblkOdLbHB9gXl2/7/EPh0HA6m2sPmu6IqWZItK2pR39oduJdIzpboppe07k7boehVM8ScqI0xsIr0yL4RcEvrSNLo4kkSYREMYgXjXIIRNDVRlNGWo6AGBl8OVZjL6fUefkFryq/FQ3bOlDX1LE6v/WT5bR97v+ff/+mp39gs9JzRaE1KOF9/P4bXg9wChOnNCtCfZE1BYz6EseCawvUhT1/5ApUqNsgRWwRkYFYzqCpFqUypFYwoXRSkXCOOQENFu2g4GhwsUAnbBMqHukbqF8WooyqACmMsWIX8ZYSxSk6YBzZOx0UJmUrOc4wXSjjAEolcjXgl2VDqkXhbexRXddMshxylvVtDqyqekgFpl5I1PjYBLwV8lYPF3+q6QBiRkvPaMGPFjIkERJ4sCIVS5JKum5GwcW3oLlE9G/W8B7CN5VgGEsw5qN7RBJAWggPJzCaIsSoCGHacKDlKxmijtHS7tQlRNpA7MUAWya5wAnpxxGLREykTKi1dVZP0Od2k7oSwyrr32WldHXQGo7mUvnYsRlqP2zE1zgzOhwjnVy1mKtlSIxlWN+86vP7iT0wlWvkMGKZGslBQv8q+YGkyO9VE6t4Zk2MRn/5n/7iK6+81m6USYqdzRaGokHUrT43BW0qUCVoIK2xFhBEUZKjS0DVK0wW3Kcwa/B3WFKBfRgfFpsIkVAmUQRRssJIbARWwq2NS2z8UGiJMQZghPZxRl2RA21jOriHeQRlcEa7ipgrD0puakUvBea2H1A3cQnSS7VkAIN5bOpa8zdvfuPrz8Ea4FyAe12t3LRJghzmQLhgvLeEYYdwasgRLAKE8b+ilBBgiBlEC2oKYYPJ5iV8kdYfIf6UNVNdk17xMWqDy6FhaVS2LWy700tu0egrqlTAVt6mKCt77uEBrfGdYzlQX6lag5GXG5Rlv4dTDH7mtMYll9dMBJHTTgVzK1wwg4YALKMHhAnh55PEtLwz1MJ0SL1CJHIGESyJuySQ1aIulun69bfPn4P7hFjH19fFMNTqoVFIryc8gWFucIS9kN5GvdQt62xBBtMIC4vTEIuUH506K75eKbhzyTRdxWrYHdOfff3ms8++YgHwW7pIzJOOl3V2nc1LJmA4ddSiJZcOns1E9cB6mbDxTjg6hLXl4sWLxXzz0IFTR48e//CH+v/+N363UtANRckF0tq9Z/dmMg0bSrjwy6+/tH/vjMfjKubhAvF5xPHWvn//Qbw11tZTb75xbvH2/MTE1Mho1AHza0IlYNy3dxdJuGBBMpk0fGGzCtmeyW9u2ixEy1sS8bTdqAv67JvJ6sJCZdGii/gvTuwKutz1b7741tsvv/UfP/3BoD5nzqFUK+HkgXMPehGHRdY6mzbIzJv8+DtsgtOYYqHegu5ExNuGasCYUd0CIvEXMRDCTn10S8eK5wLrQ3wm0fKKZygH6OzIv1syG4tOn2+cspKFRKdSAkKGYuHxiTEQdY3gk55uetfMylpCb1rETZLA9vEpM+HRb7311tz1qyicwxJRHiKwggcxwKwsL9XLBeo0+wO+PbO79h3Yj6xJuHCt2RJ2nu+VXqvlzlomJQiLDWGJRYrrD1dIwASQEwNk6HU2kpnoeBT3n5urN+yUzQiaa+Zkx5Z67CN7R47YdeFcr7Ha1NWsMHcEv0iWE4Lx5BNBE0Cp0EzoASPFWKEbxO2ABFzgB0ERMt6sY1ayNuZbS1qNpUaXEddpSJxIGDO1ycEW0yAfwjmGW7ui9nLr1nne+DfRQXlMVu/2xrHWPs/I1GrbzsH3/NzpjJwXZMoDDNpdDwqXBoHUCCHgIO+Sn+xFUBHaKQdCUFnJ4E0BRKTTdgu3HdyI4FWQSKSQPHMvllyGTkQp0VEjXrH2BfLUANOW6OwV8gWp0Q2gDwQhNJt38GYcc9DSgjvooTaSok4EbFEVQsu4LhgBzhKVBM6r2GSlCRAXNIH8sGQLqjSoRmSGyXWYOsYGLh0U2fX7vCBEau6iQOLDcNHmo0kA0SBnu17vDQaIVsb92OrwQ4wQdqnvUBMnqQafB7dMJkvwKjoVGHCULuR9RPLzBGP1Zr2yeItoilat2qkTvdi1mxyI0bijkHIJSyPomHGDiiudroWMcUwddkc+mvwTDdPASuSf0TG261TXZFtejZNLz4UX28BSztddvYzV5nHs3kuBm7n5JZJsWmxU4dZFhnfDotLwjblr4dAI6LdczvfIjlUsU8CJfB/NYp6ogGavvb50q1hvk8nk8LHTH//I01/6ynMwNhGkk0aRFQARsgf9wCtJEsrVUrHeQCFPTAjVdaC7TJisBZk7ogIkRyNvhLfASU4EYWZHjEJIxqgrRCki08jssfGcaH/ZkfMAeMPViXu3AYmbuA1hWoBOdOHIvZyhffgRqIvYmUTfpZTIqk2u8kbABpmaNsGnAlhAC56Xfaa79spLr4CREdNhzohsIkYWEgXY815aYVXRKZz7UJ5h0+U06hbexwFkTLOq0ijgIYI+d5rFE5szHHMPAMdPNlYNs8lJeipIc+tAbqOHcqfac1UjmVBHGuGkdoP2LLcB6rTGAXcyhNKQQikyHqoppT3WE49HPBXBlnAIGIJJn4+jGXIMcCcMLzH1VHgkSryKoxYvkQ7QKC1Lg6wQRe/5KHWJqpsk3jWxx3K3trHmwP7SaX3oQz9MPpPzL79qi4ZLS5nATIxkU9SMpUy2rtwzBQz2kAHjusvnTWXS2UqeZL9WDxTM1jO0q4XSxfMX3cOO9ZXN5HqmltfpNnSNIN+gyxXKlDNCnqPaSHTUb/djr+umEZkXl0J67/6ZA5TsphAA9Z0xbjocMTxz33rzmtft2bN7z/LtRXRCo+MRyN7wWGwlEZ+YGUsX0mP1CLkSc0Wdx92m3OHG2sajj/SPnziJN/fw0JiU2ytnM5fiA32jWMUinC01WbA4anVi4VhseKJSqAanIw6dPRdO1nNJN8tj0M0kGkSVx7y68RHL0kJ8Y6NGKg+rq/3+J95lMrvrpTR2YA/OmWYSxvWqIoLdqZ2kppvJ1WBexljNgey2N4EKBeRMjQwLG9MjK0IYLgVa8pgAAigJqALOsBagA2qUG07Kw7moCw2DRZoF4F9kZSz+VC0i7V6vkyVs1xmLTk1GL7yx5PUEe3388kiNZCKxpI3yzNni+w4dBxXfXqKmxkqqWKBEEu2UirmoNUotE/wJkCuxdYRj0fj6GlxOsWQM4LYe8uPWTIAloRaR4RGqGfBWWXkmWczgbiFEsrRJIgMbyKoQT3613Fkfg67ZZqLSXbXdsISAns5qfs4Z0z398UMj7x4e1G6UcyQf1nkQWtBmEtgFBYbFAFq18YHfZ1UzquzaiG5CellBnGPQJGiewWMotSXDuS1SKiOrLSHgn1VJb3coqZonuYFNrVDtQM2WOlTntZ/IS9o0yYW/4yZ906Z2u0ntLd//rq3zMt/yAVr7HKoRVWflipzW/pFfAiWiOtsSYZVoK8RVZF9RCqi90g7wzSKGUjwDfYFIvQgeoB8QhdBOHgTjKPUSjQpyVS45AB+OfeBW4EuMwKLfBsfJAKOAEkEEFKWkFTolfA+oDc5c0Lbk4AJ7UvJNbmEuCABkogBlopmsZAjXm6wWo40UmHp8AZEk8OtbnrumM1yDJUTOEx1sJIAZuJghm2FryOuxWO348e/as4+UlTXCb6lChAaE3B7QB3wS4dNgEAj07hp0lKQ2GkloHg55ri2slCtZMm80quV6qYDOnPhD2EICklEu2ywOarQQXVPFCQp7NoomA26c7ZbAMmUBnDhud4y+vs2nc47a3D5TUd8xFXomTHLNeiFtLRQGLv++YEQ/OX3vfY+g0Sk125upkt7mc0AeRL9AvsMg3Ad+JHizBJx4ljL6JqKSifwj/3KhAaLMkSVn/559n/z4R1aWll959brPCxtB+Qrb7ZXywFS2OuEr0BqQBQAJV6K6UFpyhkWlpEXxxsTSIjlPcP8RIgShFOFVAnfgLIiVkgWpVsUduBJIYmnCzagJF6ASEBA8Bezxi8mEPxHZjjNcFIcsocps3APICDWR+1iQyurGcuOHqJxohCs0AiB0e06HPZupGgLYsCngYSYsGU8BqB3X5XlpUN7K7InEzhnOy6qWDUsE1BBKCcmEEgPKSLc8y5cxROBEoeBba0XwhDyraLN0RG3SOh+gLu1c1WBSmXtlYDhPn7lHKK6Q3i2FvHab1o6At9yqkPNgQNa2XDqD/gQa3EBZURdaTsfoJE69omuRbFle2qxXiUGibhgSuUwXbbJxpziu6ge1Moo7NRJACaONrI+VjkbcLlzHafw3fuM3/sk//lQxlw8c8fHJ//BffuqZZ76wtr5S7efhbvhL6lN2UtZJ2uoq9EfvQpA1u/wuKpWs3l4ZNgyjAClka3sOjWRDuOizOI35tZI96qpQUaeNE7W+SjYY18Dpk+AF7LVH9hxCLbG+ttFswKTbwoFJq9n/9JNPf+1rX/N4hn1+CimmoSsEGeNhF4gGYeAXltaoIu0JBIeH6vAiN6+jNdWVC986f/7aiZP3nD5137vunc43S5lsamX5JjU58HCuA5twfmb7+OQeQ984FJl87eUzk9Fhq8FVqvenR2aalYyOeMWYqVggcrldl+IgIm3GRkfue+hhaydTTy970VpJsAALmiRLQhHg84E/gcC7NgUC2u9tWJFfQhNk4kXgFX4UKitYDGgA48FEbjXCRe6S04RXCIh2dPUyJrGOFW8xlPFSnVD0UNxF+UW0CCgZiXjEqd1s8D344AnqgHW7tUqpv5FYpQCC2ery+LyUqzp45DDpUxaW1wqkgUTz57BDzLxu4j2NVDQZ4Nk50M9GYwQRkDOcDuLSB85gRa2uLds9fhwCArEYooQsM1hCsbsCYVhaFT0WnzSSVXFRquoIyeM7UIXip7qRTXrDeMJ35lLr9oju9AdCo6dd9cpls6fmIoGhsc14WMh22hk0K30cOxgzMIjItnwjFBdELrGcWwMnNhptiFhKQnS1gZZ/5OWMmjoh2EqGiBOCGmTNy6qTbfvfnQPtPB/GgWpHHpUbtX++d7/1Bu201pXvu2X70Z13aXd8fx9Y/+oSPedfaW17k29RKFJjSLggdE5YHiGiIA6hjsRGIvsiPZL6AsWh4uVBljzJDfI28BkEWPnLwekLHpVRYs4AIAEueQsSjPhzcBunmUbCwZGtaEMIM2MvjA42PJAIHJLEoAp6o9sKw5DWuE2MSRNcjYXS2mYRM/8kKA6iMJG0HgZclFzwZ6S06FTbxCjhc01SDuqHZPLQoiK+o1h1/F4PphE3hdzgk80miiCa7Q53UOcNRaCV5PCiMoLDZEeKNXYxLlP91oh4pCFBIKXdgWFM410vaSQg4JIHBDtXA1c0Crgj8XXEqwi/bsr3klatRTEJmBK8YvBXgqcp1sTu4vdFjARQWXwGT2yt3B+LxAITjnAN6/RGu560Ge1uokec7vlbt2JULWj3ClV07F2TKwg+SxRK9WoJGp4r1aCbmGApdOLz2YnsdfiojYsHJ3lOnMMhEnD6B9ROL+aamdwnPvqhldsLN+cwbeqCgfJo1NgaWElk2aM0sr5PUDNSIjMpzkmwV9hmRFQkvxcOcqTEA9OCQxCFRcfDWKi4AC3nlNi0FTyJ3xbTKphG1gsTp/1BC+RAsV5yGVUBfkxC7EBpIlDCkiBni62XnjCCnBPRVWQGGU+hTQpkgQWF1/gl/Br0EfcNiZOp1rgdUR4FIyWhBOOxIQZyvwChiMK0pdmqBcSVkUMs8RhTWvAQ4gGEEYQO8Taxmwj7AXchMe8aVaNN2tHkWqZXOiBAyefc2egtiBzqzYew8SCOEWzcpi099mw8y8aBKNRYIVoj8rnqLFyFy4FFn6REHq+XxDRkIKdv4CIoLK3RBzyw0IJwe9lR4vW8kzY00steex1ICWUs3cZBjhlGR8h6A1BxCUTxODkz7XZ7900fOHToyKULFylXlUgm4VOzpVx1Mx+bHUpubjLUeBRT86NKqRCHJGQkI0c/3bQ6Ij4XYTCdVLw8PjMFVp+cnS4OVzKbucnxyVdeedlDwTK3ZWAklq+aL+koWXfoqGX/voPjwRF4ROpi0+dctnj50tzY2Oy+ffcSsnfvqSdeffXlT37y8Vdf/c7rr71y9NihAiFQvU4lXyGJXq5UingDVpsb397hWBljQSarS2xu3F7cuH5tYWJ6gsqeOGItLSwXK4h5NWa0Tc2Sfi9jx9l79eH7ZhZurC2cX5iIhe1GL3aFgDO479S+fHHw+mtnsTcdOng0V6luZvKjk6Oh2NCse2KzmncWG+1KNVfvWND6G4nOp04LDNud6eZoa+oEFr5nk1MKRABeuaQhcpl4AAmQ3Lod8BYaBFizYnAXNqPfIS9HrW3t2XBh6ZAoHtiBuIEcAWUzPqqk7Gx1+xVTrzQxORMMupKpHHIHpiiDsUmBZJMV+dXPiMVGx07ee+8FHNPSmfFdu+qVEunma9XS+spKbGyKiAOrXYomrawsIWHjPS62JD0VHSoGq40owxr8KD2in8CvaKxA8iBx8B9qL7oCe6oWLjK5ACafYNBXWw27x9HUlTZzTeew7vGPBw4/Mlbpz/dsJfJn4LvA6PE/IZIAs86MwlJGhz9hTxT1hfTSniw/7uQ8/4hgjCJRTCDoBAQtqE1G8q5N+0mPhHSpQeaidsvOjTsHXFFXtRa2kAtnvms139X433Ao4sXOpZ3GOdg55urOsZxXRintETosuEChJ76Xq+zVBrXkp0w7YwKWANFxI1gFaRClo/J8JhxQAjVErtAGTyg0ZFWZcgWFoZ4Q5bMi0kK/Badyp9q4k46AqUXCVSIs38HLIVf8BC0TQI9YiSONck4QP1WBPrnBLOIDsyNqGygxGZvkaiebIeE4VQgsdg8htVgu3J7gUDg6PbtnPZ6olksLi7cTm5vUfgmiY5FinOWNVJIkjjiFwFjifU9dQLc/sJnJbmxmUtkCGk+/x+uAWNOUTfJjieCqvFpwTCLHQ7mWbXTIS2rCJZVBMQmOhEsk3ZXUhgM34cI3qJCOpA8TC7CSN5rMiFDioWiYEGlE7JbBhk+9yebav+9U3eIxekesFq9/vA4T0c529VDYepM8eTqrzReKxkan+jZXIlVAsC43eqHYeHN9CWBc39gMuky1YrqQSRn6nma9GIqGsHxivCKwzt4feP1hayicz5UjwaA5Ovb3f+7vUXf2c3/1LLrw0JAtVabwGbp1M7NB6T0qP7IMQwEfQ4BVEakXggTAs0SgRohScMCS6V/oG5SaTGQoBCCZhPrx7SK/ymzCNQujqlYOp2TtygoWWVZOshmgUkh0wMbW7Ms6gMT0SXElDyiXJf6RNcVSF96M9zLjwo3RVdUO065HYY5FEYcxARsbAmLbZjPb7bg4Cc2jHfZI4ShUBKIFGgVIQRcix8tlRVSFPqHMpllRcGsmKZ7jWYFNJVBySVsv2j08yf3aI/KK7Y0zdFXz/aBLapOX7jxOI9yvPSJft71xTP9ABJqUjuqYaBq8HHBPq5Q60SFJeII/P1F2QuBVKQiuQolpgL22jnid1ri0LO3pyJZKtiBYaWQMUfCJhZu6gQ6SFxLs1Kg2Lpw7v7y0hHK7WqgQO/TK66889MhD36yX8+RR9xKdZclnUaCUZO2BMZ0k4x00CrqMNT3Qe1if2XKl1Y7nM3he6u45cfrAvnuo+nz2nQsEIbjJRWnXk1GEqoKUeCcpKFFP7klifI1ue8Tu7CwvblJ30+ueINr5xRden57a9eWvvDg9eeB97/sI5uFgyHf+nTezjcIhnLuO3XP2zbcWl+N2gykUHrXFzBfOnSPK3+XWUWTkjbduvHXuBh5b4SBJGjRehwqENiK3kH2xTKLM0vdsn/joT/71n/0ZftqTUX8pE7cM6p0iWVEHRBNaDdapid3HhyIZyrT1dJ9/5pmff/pDhVIvaHTpzVayxejtFE0mGwQxHVU1lVvTtjPz2oR+P/7mPFiOFST4DxCSlSOrSZhUdaw1BNXgt5CBbh+8V+voCByyNx0opvR4gg46yBviV0MqrkFTj/OxydYzoePvmTz2kSGqaa1S9xPfOnJP+oIxilsTJiB5u6yWqd2zx47f88JL37m5cDvkc5F2OxYOIWVjzFmLr8FnvvbmG6gcxoYCrPdGreI2u9GLhGJRJBPkB6LCgBnNqiF9BMKALN6slAEDfPKFamAS4avgqI14g3oKjbTd33v3Q6O7TjssQxTNmneEByYv+f3Jhq5z2okBtRNlhIpGrPTtEu2LXROvDtE5QzOgoIruikGX16F7NhMfIg5x4opS1aphyMCp4Ze7ZZOO8Q8DLbCvhlfa0S7K6lV33TlQv7f4ejneuoFZ+btuTKigDG37vvZ/8HmQjOo1WGCro4IxFOkVGnenPa6KECD0FQmJERKtsqgNEHxROgpDTWIS+VrtcblZopKQjNFOiIQL8RZyD/ZR9FuJQOp2eEj1JxAIxpDPkAnEHo8OxELEDbkvCOuykJqX4ePFzLxgMJCvZHWAbwIzg3FomJAjVERARTK56fX5jQ43TjVWi8Pr9E1O7Q7vms3F42sb8XqZSkj5RHwVGQ5xB+RbxK+pVvGQZy4YcLrcWLlMFmuBuwql9Ga6XK/6nU40YGB2vDmpNkixS3TxguaQOsFElIcniyCq3CYpEsX9WBTU6KoNHJPXScokkIcYdgXZFRbSiXeu1QElAPcNTUzlUplaN29weIgbMgxsruEps8XfxIFqYA6OTOsc+oalXWhkUqmV2PSExeXC1Bcmr1YgGM+Vw+FRECVqVme5mN0oFynw6RhChYiYTra5QiYNH9s31ki9iE27UK/ZqYqIBdHlY5xWL5ylpPH7Hns0vrL83LPXr75Tc0R0JEV32Q3eAAk1TdlCuUmye7NkdUcaRu5nzzziBcckAQzQYOGEmFE0ITLHkEdZLEC/8MTKNiyLVIiJ0C6Ra9VaYOY5EJqgNo5F9MRBQNFs9uoRcayDr4KP4SftQC1onwNxE+VxDXY1bYtQMngnC2nJ8CQBo0GY7DYztBbjKFRJcCL9VS0z8tIbOiYWZmgchEiJiYCiUilrlIxHEF5FEwiFZgRQ5BJuJPoxcISirdvLjNukw6oPXJV3bW8c8iLtEfbah/CVHLBxRui+cphiXWn4QUZGEV7tN7eiMNEawa1MMvwpSw2DwYMQYHpDB7LZLH1iFWgCN/drG+2ziQpaJBXJIcGSRHojwJTM4JLhyeFAbwuf+/bZs/Qh6A2mUhkC9+Ibl3PV3Ad/6GmkwPnLVwPh4OTkOJ945s23QZp9vBPrAwNk1yVkL5Mq2zw4Ejg2bqeNFsvmRt79COUZ0BF3ChTQLBedLpPDaRwa3XPq1OFcfmPh9g36spFYjoW8w9ZAk8qh8M2Q80bn3JkrQ0MzPt+wwx75i7/8SjQWuOf44dP3Hr/3ofu+8sLnp2cplBAbGZmuFzuE11OTutZo+zyRlVwaEENDxqBSuwIPQnzge+K9UbdarK0OBXhqhJaino8FAx5HYPfe2dcDr24szCdBXqiPshkcvvBtZmmSKOXrX/lmadC0e137Dx8NWZ1/+fmvWFKLqOWH7A5cOU0OXadhqLZ6KEyZHW2qd+acg61zwhoylzL72iZ4StC0oF3ukf/URdYTm7SgCIYccBlVc1tncsqqqlYGrkbHRDErEkYKnJA2QHwlATlcgsmSh1lasmW5LTMzEy+9fI4FCSrw9I3kRcB9zubEhT6Df0swHCX1iicUeeWVVwqloq1VaxAi4PIRD5LNllZW1xYWFjA/4VWKc0BZ3w2Efd6gn8jMVLFOuUCJowSKhLhBq0RbwyJkAcGkUjCCL+WLIaYSagq89S2dZH2xY+3/zI+9e+bjxzsb3y501/1D9konjzsf/js+ok4JG0b7JQvSSIUzI8n7JYMzlAPqIVY1XiMLDzWlWi/YMjUyw0CJ4C2bDKGsJYUmttEL5+B2pKsyuNJhGVwhMuz4yWOCpGQvkp/aC9ZRzWFkuDNp8sT/yqbBg9pLc9+/0XE19VzRblAQIy8HH3JG9upP67mGTTiGNtMxNeXscVajZAI8GhpYRVwhLKBK5CIARmiqKJ8lIxY2CvZKi6DOg9B4lFo0otzk2+W1vI7eMA5yjZeocTJLYSuKEvq97lq5wI2q3wKq6BI5Br+I1xWiFvPew1mkS4wFgiWeOwGCakrV7Nx8MFYbHp00W1xky8OjmNrg6/HVsZERr9tWobYhbgjlbKkiKDgQ8HilTI0J78qJqSkSQ1+fu0nkBNIACNhsRwdMPcF2ky/B08uI6UWcdMAZfK8JU5LRJT6EuGiY7FkMbYyaqBkH5LwE3aHBJT8A3LjoWLCDGqwsInIBHjpynBy0i0vxfL07EZ3QB4zJqq5Qxd+0DbhTctjoHdWZyGSc7oeGMWMVS2Rmay+vJnzRifC+w53rS0urq6Fa7fC73x10Oy83Kw19myTRqZX5iD9YyG5GIyNEeuXKRZTqZJXDTtuuVjskhu30s5upqdn9169dvXLxrY9+6PGpkYDZ4YuN7//qN1587bXzpVTB57MG3eTqIwNBB10UbAq+lww1Kg9xFsIpHMc1SS0pxILIFyRWsSMo7QgWaT5WfAVEEyXQbwTYBb7kt6ilgXghygJ0TChRTqw4No4Fr2xTrFKpipCNCgSKi2IZQqJWPVUSJP5VaDhAR7MAjQJQ5gbTorBiTcpWdEZG/OQJIY5bqhjJiwTW6IDY7US7JmH+qlv0i67KghZls2xEN0k3OKJlMX2oIgcgcJoQHpI7ZBOkqW6Tj5M3QOWFQstHyx/vELIt7IK0ipAOzyaivFpuQrCFMG5/r6xLzshXSV/BH7JE6CF3d4rkZICQUHvAks1j+iX/NTpn2NweMXi8iNg7biaVCrOksAjTgtEAoiIOiNBZwruJYpdoBckugsHd2hFRn4C65sri0s2FW8cOH+P81XeukFa7sLwR3TOBdvtP//RPmaZALJyMbwZDoR/90R898/rb73rXo9dvXuMMHxf0+ZqdEhUvyPzmoMyO0/jUU0/hP4HPxaVLl1FgwkKxHPCyTq2T967gcBhInVHKV0K+0UZDv7iSaQ37YEqJKCa2vlHvJNY37jl6n9cbmB6fIEvl5sbqB55838mhUy8tfXPfwQMdPYmcu+956vFStXL21becNscDp+53BQJrqRfzVfASsKQLol8iugFpyQR5LiH6EjMo9Y5wbjBYpoYnSA71/HPfIHtlaTORXW+HPfZCWhfyc7+pCstn0UvcjN8xv7j49We/M+JyxXrmWLuQn9gVQA4c6FyYIXDjY5wVGymzJfMlewW58kt+aJtGhrVj7lf/yUzLJGmkQK7JgtDuUU0JECDD9FDUSv4aalEMGugesYiIhqnZ7ziceGlQQ1i8EcW7nxyZRIFYBsGhIUib0+UqFtNUlymRah9F+tRMrlCdnBn3Bt0biczwcGxqcubcm69E7IZ4PFFu6gJtXSwystrAlteH205n65FwjfzdHj/shyuVLc7djgciUbLCizUFEAZsAWUBUgFxHUHRM9Mjq8sbTqcBfgf9EcPU6Bdbo7p/9C+eHj09U984WzMWPWF/vV2ES5L8tix1lMtAOYovYtzM5Ni261p5IeacYkSMYgESFMK0ikwtQyxMJxlKZPWrtSc7pRmT5aL4Vhk+oShsPKvNivRThlsIm5yXSzJHMspaO9qqZa/u5CyXaJGVqcZXXr11p4qi0NpXT0uDauakaUX/tDNyTs2pIuXyJumYtsQ1zCH8gYAA7JTgJbFf0xkhiaKJ44A/kCQN8TpwDpMsvrCCeZA30TmjJ1D1B0nW2sTghg8V1k7QNTgB44hE6cILMbq8VOiyfB0IlMHBIUvxgLBJgsI1IBKAc9j1rSp2NXRlupDLEvDakVFrpSSUGBdJeobhioqY6DgkRhg8LE5YnBc5RsZEYBXCRkdrdpfL73QTd0HOF0x95JkcCjgvbC6NBpwU367Xyj4H9FrK4oKkCEA6cuJebzBYqbcgwAiU4KAnH3/i0rnLD566lwV8/p3zeEHbnSZ/bKhezKMKpDgg2R+qpbbX7QKJGY3Weg1TnLNR7zpdwVIuyxJtg1EM5lS+SDbJ0eHwlRvz+w8cmJ6exqMV9xPURLcXVi5evA7zeOjE/X27Z2hs1FhpLyzMHT5+cm1tqZ1aDntsrexmOV/t9t1G+1C3mSfEHr1Uo9lLzM3jKTooVTu1uq5Szm5u7p2aKjrNRGHieh1f3jh0cO/6xlq91nXYPSiHmtU6vKkT73BEQ8ra9vsL58+6yXFg768vnD+wN0RxOpyo/sU//ol/38r+xedWiOl2eqgyAVYCCPWlcqtvk3yipAuz6vtU8CWhrjvgwL8DH1rUnySaTOfbLq8bhQBVpWJDQ+lUCgYFDT+ZWpHImSChLsK1CuABSEhrLGcYJm3ysPID8ICXRonhizGNc4angEruAZBk+Uv2Trg3qLyBINTNVJLEKROTw/lc0UaIVxNvG4LF9UOjTpAvSjuAmxQggjBkCfKoEG9ag2NodJo48oEsqJuMJtdgMTRyveCQByhj413wiYibLBFYLbCFXYM3RSNZqXRVSfkKA0DtAFG5JH+q1+h7MFXwlZKESH2XsCXKKZpuCOSyyRqUjY8imhy9u6xcWY5CxFHm86HCj8OA4AmMOwskE6kIO3+phuuwDtc/8veSSyYaDcIbkO6RxyUfOS2wLliR2HqlLJBop4ukU5VR5gQG8paetDfguG5/dWkJh71zb7yBY127XEMlGRyLZpKbDr+zVi3y7nAsfPDYMa/fc+PmHMpnb8BDVimjRe9045VtyuV7FIStFcu+mBUCEPVGRiPDLo/zy1/8QjqflBTn9SopZwKjFny2l5aWxsaGqAZx5crtgc5h6G/+0j99/Pb8WnITM4yRXv1vv/DJP/3TP/O5fVZybFGvs9v88jN/FfDazl5+0xwxzhyYCEXC6VL62CPHry5fHRueaDi7cUyXPZ3DpRsbjgx6DbsLERhy3gSJB6Lhaq3BUiRDC2NQSuYuvvnmtTOXFm7crhUquLA1Kv2bSzUblXW6+FT77T5boZsC8nWuwel3PdzM6H7yqY/8xb/7jcoinFy5CYdQ0w3FrG6zpd1A1c/EMa7M4J29HH3XptCzOqM8CtTUgGk5rf6YJ5hLOVbwozg74SQ4z+SQiga5nMJBFIOgDpU14oa5xKUUl1AwrDiSiCCMB3mfGHDcDr1jk77YUCKZQbgZ1BuV3CZqIX3L32uV1pYv7fechNclxMHv8YWCw+VMHKdFL8sYNbVh8IkPPfm8XXdz/hoW/RQabJ+3LnFeLmpVGs3ezWSJlSPMnWihhF+Vj1bEgnJv7ktXNg4djqRSaaeLjNINkiAOXLp/+C/fN3pfUKdbLxvTVkffYJdPxMSFlU4GBAcZXDMtuO2LBhPHUTFVAtSCKVgO7IWssoHf2SsmRb1VxkoRVGWUFLImQynnWGYcbzHn6hZFf7gsm4jobNjOZHUpaVE1xTmZj51jOZRNtSUH2gTLkXZS3q+uqjPfvZM+bLe0c6DdvPOU4A7pLxwIV2S5yw3MqXRE3FyEh8BUpL5L5ACOQQ2ABXylPIVEi+qrjRGIRDyKEpM0SXANXBtVo4V3Yy/NKkM9rxD9s2hTZGywtqKnBXGACPiJJIrNBtRcyg4cVmFFvR6b2+mgyq2edKuSF0dejRwNrwd2EsYIxKxwmBoH+cmMyRdhjwVoRFsKEgJI+uTPKmRTzVp5fck6FIJ6mEliVyOgm66Dl0WHrL//wfvwNJbkR73+3I0bdpfbHwyTjAPDWzFfIkXlyOioCdMv2dHLJbvb9cDJE5l4fPXWAnrOQr4UjgTtNpfV7qm2JHUGVAHFzcULZ9E3WxyukC+MZ9NyPOWPEnfupiRUDY0YrxWnlCqWMRP1UweDdDJZqEORLJRVSixdGw9HsV+uUQB1/qYLLXu/s7Ky8cFf/Pm1s2+98sabr738xrufeALtDyHCiOX15SWSIpA7K72RaOTylkGrXGos3FqBR2p3GyxSvGD8Xh/1nBDai2lqh+Ltb4RXMvYgPUa3C+ecTjm36XTGKrmNH/7AE+9+tHbj1tLXv/lWPKkLRXG+dR87fABshYHw9vymx81Yd9x2Q7NSx5ORhSIQotcFQj40S9VGOxiNSYAmM0QMFpeFig/wIYcxFq0QZ+C8UOeKOIiGSbzcgQsoriYsUmtW2pP5ZTj5eIEcTOuclDy5rH5d2+v34VXbHnTGpsewSlI2lhQQSF24mZOWp5InewY0tM8Y4odFrUlAUZAGMIEWj3ebrFQ584UCZB5AdKeBdkMXGw/6LSYqqf7QBz8EE3fm9TdJYYajnEotItboHZAT4FO8gqwauirrcxsZKwKsFiDkGUiVq9ys7pFVBsTyiIJbGBENY3CaJcKXyYoDBcO7K28KMIisFzT/UgmGxdUkNszsdrhIRYnD6tpKArhFWQdLnC/lWaHMKhxkDkWKkfgrA/plljqmGTxjMdm0Ol1kQ1pmMJGAYTAo/FcrFNzhIGZmMq6XVjI6J4m1DcjK4L5qsSTEodvDEgG/OD4+uXfvnnvuPf3lL38ZXRGcBEmUpOvw43iEwG7Uu7nN1MLVG/ggnH7wXmpgl5dTOqIQPHqn0xEKea12CR/BsiOpwHTmbLY2Mbb3P/3nP/wHv/CPguEEMU5Xr1yev3X11//PT//2H/5WIrGE7xTFLZuNyh/84e85AuZBqpdvbHzkAx+5ePnczfkbuw9Ol7Klkw89pbdcOP/OFdLDhaIRl8OEJQbN81AsQvgNTmONVr+Oo2Otncv2yBpSylalPFq1TT1uAAuoobZ0Nl2Znh4zWZw//OMfW0xf+x+f+aJjWvdvfu1Hq+stPKEeuvdEJVAnhngiRBi9rVzAUNP2OglllEhxtQEAMhB3I2qFpLeva/8KK8VdYolhzx8EQT0J+mdhKAqjTmzdDoESMgy+FZmGPyKPyDQFngYqhDgJQRGSTSOy9bpOisjEYs0KqjopuNFAceKw35q74h+idFL65rVLw2N7CSzMZ7LhIA5oXY/Lfvr0aQDv1tzNai574tD+qfFoDumZb+vprs/fdG4W3L4oq3RtNQE+hk/kI7XX8WL5aKGY+sHoiCseT3v8xjy2fZ+uUOn86CfvHTs5qTOSeXDNYGp4fZaBoUIKLiIX0WvJMDAcfDI0XZhUCaqTAkp8hhpHVgzd0miq+rgfsJMh1/qg1g8UCrGS+8Aw6m5Gh5/C62qbalBNFL/BWNypbpTO/A0bl7hHu0EdiPjNvfJq9RZ1cOdhGQ61bU+KNK3dwxMcqE3ukAYED4hcztKVq0KJYZrlkhBgjTCr+YU/4SdNw1RLLWbMgJBelEFtfbdtwFKmEmCpOh4YmeBcQRaKyQHXbDUFAhSAk1mDjAEyks4b1x1xkhKKTme8LtxxTaSqcbuw22HdkILtFAimc6xuIApsAu4GGkSOkcZEW8ELkJlEMYgVEJZebbgoGNqiuMRDAXDHqoGhJxL2cxuImQZBH0LskC0slkAo0k5nUJiToMPd0VWqeDzFwU2HDxwjl2EBBbhJX89l8AIDcPbt3bO6sNyqlFANY2eCIcVtKZsr6C2d2cMPFEoNclAHCaBD2b262qw2yG3VN9pQLD92/6N4cuNHg28XfguBgK9eyMFmOIy6Oin40gVzIae3Oo0W+9Lls+A3AoIG7Qa5Pal4T+0w35GDraUlu90dDQ/hSrZ0a4kPJxVldHry6oVzFDd0W435VMbc7xJXD/uDJgKOEvQv/Ap2KzaPC2GxRqrrQoHywsC6bUASA+RjgrUMZBm2OVyra5tOl2f3UDQyNDw5OfHWuQvXbywc3Dv9yR//5MTefTcuXCBSZW2llcvW7U5dLq/zh9smW5vCGcw2oFGvVcmnQ3k4kpUxwmBtFBqIrxRbwvdcdC3in6UojeiupG8sMvWH6znSsFQp0KP2xn2AJA9CcxGtGXUBVI6VJ8EA+YYbgviRmc0UEx8ZGcnkcoDIxtomgKFUrijQXDpDTUKb4MWskllezNZCL1ntLCJZqFgIsRCTo2B0NFool5JruVMPH792jbJSt8bGxngdeXQnR8coykbqtGKxQ8ztzurjAEBjz8ad0j9ZztrCl2M+VRHUreWv3Uy3tZu5yj3asdYCbYBy5ElBqTJSsl6ActU+V/Emq7c61HHgowDzAwcOSNJjSuc6rENDAT4IqsMb0SjseWh2hXDPVL7bLbvRCuJ9jfcfCWh6ZIdtCsaDoqM1xlnaaQ2PjCAhsz7GZ8fJb7W+Gqdc0KDa07nQbPWNXtQQFBkbXL985fbCfMDj+4kf/4nz3znTdJHhV9AFG0wCnWZFk/Dd7wg+//zzRAeDCh+6/4HluTn4hn58UDHUwpFANlNA3ql5bOOT03v3HFpcSN+aXwbUvvCFv/zZn/m577z4/L33HfnSV/7i4YdOnDh14PrNC+RnSGUzmWIKSNooFn3j1un94TfOvhD0kd+mnUmuoOxZX7n1ofc/OR4ZAnm/8fKrpUILxESlMnLbgbUoPkbpURh9NDxGc8VisC9dTZvdJBvRBSOeZq1VXm0FJuyBiJvYd72xuLR688RDp89ev4ILUSHTePnZ586120dDjk9++Cl97oJLl7SbKbDWhI0plwtiw9xC4ACGzBxzdgcEZFSYQrX/G3YawKinFE3evk1BhwJ4FfDK7GOqBtXYERko1iJIW4i5DLygRIWveaZTtwWGI7HQ+u1VOFyQY7naIFFWuVDWUTnMZyzX1lGok3RtI76KW0AsHIbCZ1NJfPoqlJvb3CBJNGEf9z5wKpHcRGHm8AQ9nojLG8nlahbeLFonEUvppgCoLCIhyYLxPT53lejLRi8wpIundR/66NjRDx3ulxcq+kSjWQRZ4DBLbi2eEIuM0E2NIEC90ZwiS7C4MWXLADCCghHUYuAtO6tre3Du/CtmXNSqalM0TLX7t4459955Xh2pMz94lhCMtPm7M6ky4Hfa2DnSDnbaVj+lze3z8lKu8lPt5VhmTeZPO6+RYXUsXyJhH/KvcAlo0mQQcN5AOAacGTDihfpogkn73DLAToJjIWoqFBheTQgqf2jOBC6EJ5GX8pzo8/kgMhY1iOYxuIDlbpfCnPChJEtw2vVDEUyZkuiYm8j+Q+AtH65ifqSWKoSTrwDnIsIgQfEFIvWylCgWSMiROL/rLKSY4q9H9C0YfGAlQYVpgDGDp0QaJhqQSDqVdgOSgPiFUEQ2Z1RzCG25fAE3kOmZ3ctra7fml3D/m5u/ZSN61+3avXvq9q25jfVl0uyuEtSUzZN5hArEEpQHsJArYHLYH5voWjyVdH117bZvLVElgh7UA5E32YkS8AUiDo+P4ai3s4i/JJ2n0/AvEHL4DCx1toHebXFSa7WYTT9w7ChmuWI+QfEmFEyF1Oqgng9HopcvXcXDZWp86uC+g6+++ir5gaM+TyWTJuxX1PF8PqxJnxhGgneRexqQNMmjIckwSftVF9A3GBBk1+IpXP6RVl1ehpESJiSLkCRw7d6AcYCwkU4ZAfVd737w8P49t5dXKLO5uTKfii8jK/yzT/0iisRMrpBIJF56+eX4RipXgntv+8KuZL7IBztcnlq1zGQFqP7qcuPfAe3Bg5pcmh2Se4mBgNUGLIgaRuNy6Tw8mEaMmWJWH8csbVFAwyzLGYRAUVCTd5OWIcCFQmHXnl20kMlmDxzcH4nFFm/dFuZR0UIkb1YmD3KzoC27FDhCdyrtaTYjxchDq5D8Pvzhp69fv44ZgvJ2JFumIl48HucpDeQwZ/IsQT7BoBHHYzDmNtLk9NZaplXtJG/cuarQwNY9zLv2FdypgbH2rHYze25gMFjsGuWVx7TfCjF0mgMK8jLsnVwOUxB9xjQPnwoA3Lw5T8vAKv5TjDPtgEyPHTvGPRBgErZaTCwiaYUEHUdOnGi0O0vry7ViBec+AsYHjVZmeQNNsMFm20wmDx44kM9kGCZHxFPPlBFee1VqmDVtQyGfx0/emItnLhAZv+vALvwcq9WaXSrxWojG5fvMDon1B6cUEml7zH3hwoV3P/7oI+99/JWvvmCdsN57/6kHHjyZziazWdJ5rdns6G7su3fvthodjDNs2G//5r8fGwvW64V8MfEffutfP/30B4+eOvD222/3C12L28Sk3FzKHDt+zOcmO1tR6pT0mlQZRkeFP0difWVibNg0MP2Pz95GMzc1Tr50NBY68i9a7F6yYi0uLo0MT1BwvpAtdfpvJlY3exldLlM2+YyGiIFIZWvQ4vE6YbXfwLB87eL//g8/he/IwpXr73/v+3/r0/9mQ6f70MnwqMtRSVSz5QbmMAqnFdoY3gDbLYoLLKhhFhT6t2/ctnMzUyMTrfbf8xT3IA0J/legQRoo8v/YO/BfYiIUayjnJWc+7wO6+COMvUXdCq/PDfajbBv6rVK7hpuKy+FPJjPjTj+MG7oN0uS3qXJerYW9uEXvQvLMpNOAEIr89fjKvn27c5kMGT+ikejw+CQE2OuPtcb1jCmeL+Bc/OfpFG9VGFd+gYV7m4mUJ2ozuJqZiu7wA7rHfuYRnWHZ4MhbDVWMcXqQEqKWvuWELXZYB/UmC5ENwUvJCXJMdJcyUAlbwYgoAUvWhZKs1DqQu753Y8XTFwbx7gua5lroDQMjbAJX5VgGe/tOdVKdg3fUpk7Glf/4nL/Tdqep7TaxrPGkTKg0IIRP64PMpfyvho1/uUtgZ+snR3yytKHOq73yMFPdQcQUpMAe9R/SC+6uBBC20OITCysxSCL+QoBRhLH6pNIFvnIoqtVL1OjwahqX8RTCLuf9bqJ9KZrUIhcv+hTi+CMhL5nP0IQCilQ+kCgf4mlJqwQdoEombUM9RdcH+KHDxJbMSSmUC/lF/uV2ppc5FbkYDhHzOa4CeMWjtobzJxMDIUwmk9vhhFrXajzRFtZC9J/kTqQpiovB0pfT6Wwfa64UaIJy22CiZ4dmIN4Ul0WPXUxt+u32fDJtwRBdLoN0CfMPDEXJlhQaHnEOjf7l518aHd/d0lmuL64R+miw4q5hJCLZ4Qv4gpFrN+boA9yi1+tLxsvIKH6LeDQhceJQEw2Sxc8BDk10CvmlyyEcOT36XG4T3SOYl0y5g0Fr//79roOH559/oZjLfuKnfgJfhsvffoGoKq/LDrPCmJPeiZFCK05mYBlsZkGc4HQdnKnIPIi5nYyFZNUlfsVGaW6PPxgjZBDaQ6wgtaPimP3cLkru4LBGU+lV0u0WpoaD1J2lZFtkOFBv92wMb68zu2sa7H/yxImvff25t89dXFgqo6F3WGnYQwwSWcngB0J+nNqIRyww+fBtOAKp1YRoLoQSkJDVjJJWuqmJUvBGai0wWcp3yWJ2iBAv+ViEhkO5yYvZUyV1ZbDsLnJEoGaAY0CQhVJyQO49NvyV8vmKNC8++zoqXPIvcV/45hCqJOCoiDopfsieNDk9debMGRr3+/3keOLbY7EYznf79u1DBb2+vCL0VRKWWWHyeJaNLskCk00oPtl6+SlIY2shy22yvGTlyFqEheWPj5AVqc6o6/KQ+il72oLJ4LqsR2lHqbIE2oVxUg/RERMMHa1i10xtpvz+IOOEJr6Yo6a1hCGRKgsPu2vXbuTzRT51QMkx0p02u8idxAUsriy/57EnDh47dOb82XhqU3LI2q2ACRlbEeepMvKJT3zip3/8p375n/0fxULJHHR2KjWbz9UsVgup7MjY2J6Z2Xa9hQmEABii6nUl5Om22cEQKALSE9+ufCLtjvrQ8VNL69yZsx94//tf+dYLzBQqKnSY8Y2VfIk4l02310dcL2HqmJ+W15aINyAs8MjhPW6vDk/KePLmN1/qjIxNtHSV4ckA7tR4M2CpXF2ZN1pjGByJnSNZE/laR6IhrEvzV+ZNehtmFgYv4JP0XjarYWh4LJHNef3mscnJ+CYVntL0IZ3KoROLjAx99B9+7JVvf+fqV69M3j+zYYrbycY1aDoCumQx6ewOXX7nVr1kDrmGp8fs//jvfyRz5cXN1QWLs+7EO4+hwt2l1nQTjCQmPMGd2rQzp9vzz8xubwIIihnbPrHzLzdrAKMeFODZBhuFmQV+FBQIrAnWRXIV1aKQP4EbtZcfgqxRbpAJA08cXdcf8sMoh2yRhq1Tr21i6At7guUS3EXFDRsGOOl6Xqd9fChMvvxup5lJUbAxSRHuWr1CQSPI/q25OZQhCKort+ettvTRY45YZLzfHgGzSxAwYShQGkLvFOWiH9J1VpbI6DVd36X7iX/wmM6fazWWrbaucVCzkGBArH2UMBMv9XYFXy/5JACdxcMXsQfExc9SWFGhXFyUdSYrjT8VPKGeuHsnGFy9Ww2I3H/31R90zGr83tN/+1PybeoJNa93pmf7Kbm4fawO7mr/7vPaMW/nj8/RDkCCQuq3TzKDW5cYFhF55cNV0iuUdjJKMCIUnxCXKQEFNM9S+Ah1NCESUoCZtKUkwyKCS8RcLH+SLRQbpPbJ6kM4o1g30T3CEIng63PonB6rl4ws8NImciDU0RDzB7Em9EgJFvQKHKqsHsJ5ySeLzks6SHIqlZwb1Qs0BnpM+zCE8tIeihqmm9z0eFt3uk1qJ0HMne4wsjvOHtAY6DxmMGLPCEWkHUQ6jIPk0FqPJylOx5qHZoHleXc+tXnhQqeaz4W9/oO7dmVX4/OXL+PbRe4hAhyDoVit31vZSNbW81a3/6HH3tcs5S689VpqfRkHNAgJ1m6H+MVYUhsJkX46vVDQR+Y/Im0pJIESHEWWWNHJq5FPkj3ATt30TNnq95MCeNDIo9FF20Yxt3gy6cxWY6XSxPjI2srSa1/9Uja1OTU23G+U6p0KnJEUvEDvgPIAf34jzwn1UgX9ehJ2ggeowUhtqHqrS94OnwHHDmJXSP5swEPVU6P0gzEYG0NbkEknWVDTE+NMdjWb6ddLdvJyGLo2vS5PuTenD66IGyB4qGcfOH06Eom99PLr5GrHldllNeXKdZauheRfuj7iL5QMEQcDD5Mu4Qngq22Q5lh+4orMuOA+IMGljIi4H5vJsERVW5F/lYZJLQMNzwHMnIcXaeO+u7zGfMPKkIuf86SSgG9DxJ+ZmcEqyCfjk0XbsCNAr7C2wA2cnyLqMEPEOSFHPvPMM3yL0+NeXUzM7JuA60L8JdYrdiyG2j6xts7rCFMjoQePaEtJOkadQNxe1ALkAzkQNKQWKvdotwG3nOEY6i5Tv30PtFa7Yed+LrEpqUA+n2OwHPdo6wc1DIwFMUi0QyUPbqh2GwjrrFDMuy6IKIZeoEWScsuGzhwNAdXsQWsMp0Juym98MDj/zkUISDQWffg978qVs+cunE/n0rzu53/2Z69eufbZz372Yx/9GDVj8QZGR02AAxoSamny3o2l9Xq5Qs6sYDBw6+otM5Njw0MDC0OXqswwuzgw68i81kSV5dizb+/tpYW5a3Mf/OAHDx4/WijnmNhcpgC367R5dIPstavzly9ef+DUQx98+gObGyvra5lg0DI04gOD11owNM5caaOjr9m9xkhkhGrwuMli48e3LrG+5vV5GrWqx+3IZzNYncul65HAKLll4u04VSVFAVMpIG4hn7l9fuhEHvtBvZbN5rFFVKhJatBFR2Lv/6GnSMf4a/O/mivnwe+egGcjRe1B3czRYDpZv35tfmO5lFy4fXJ2+MMPHLINGY25G05714UqhIqErMeizkmAEAOnQeY26dXQ3Q8gw9y5BfgcASqy24EBfmrHcm17oykNYWqcGI9LTlYEB+U8C/ajPZ6C+kpACX9Ik6Kc7sWGI5IxlGxwvbrRVEJzmMtWiaJMrKeshcrU+C68JBxm3QOn70EbSHgIbyHQiKTfhWKO7Nlz167HhsLkW8tmUri82m3t5VtzoNpQIIxSW6CcV4FlZAd3iI+DmPGMOhtOtANUg7/06T3WESD8tsFbqDWLoHlcKS2owESyExqDEcRM2BmRe6hJSam/ZYKSjxG3XBHpZah2RkfsxHeN3fb4yL9qdYEBhKKJiLs1KGJkYRNKIMOspki9XZbWNk1SJ7SZ2HoX98pjYle/8z6ZBlmPssm3K+ZA+8l+Z9p2DtSlO7+2j7TJlr30VQZP/qBrWweKF1EUV9oU0yrEU1FihhnmnZBf5hdVGHgSPW1PAn9J3CkSMAcoPFHrkv9OhGAJ0xUyKB0GLtVXMcfyJlAMB4QnNZlXvTtIQCylWvGWwvEVjwHcqWGYhC8QLxkYIpV7COymfK/QdDACIhAQ8wIbCqmpIasi4sLkg6eldRkiea/aI3fhZAfSh7RTxgbOLZ1OkvAFeiMqPNyCyAYpA2oAP5vsImM5STYFzqFYb6cDJaxUiyhaiRtMxNfJ4OyPRMqZfDaZctsceJoRgkXGrBp5Aa0WbB8dk+HJpz+kCw81Uslds3vmLr9DTV2H0z4yPATYIkDPTIyh5MmnMygQCFcym9wkl2Z48ULDj6rWa1cyzUDQz7HLZBAXkUYr4LEHI1Fkp0q9EXC5uoZBlhJLuRQKJL/X5jD4wn6bCXXcoIvNWCaub6gzHeIDRwBV2+50ozkA/xOXgZANiZWxsRhc/qFYbIrod+JsWdSgVHLA4mSbSKWmnGPttr5Srq8urZK5gH6iyzuwd5Yxa1QQWYT/9WOkD5A8wbN2+1YMbsLnh9v5r//tT7A7NCUhl8R/G63dVqOBUIXg6PH4EqkkanFyeQB0EFVZB4rQqnyT/MQtQ0RAuk0PoTqsT8BL7B1ADfxJvw9lRajCbqkgmPQ/g7W1OMp2zLSo5urVBmQjhfpUZxgaGXvw4UeXVzd4dTFXgNdTpcv4UhK1wKpJ7CwsJgCGbuMv//IvyRXcqMJl17xBx+Li6t69uyiOCz27efMmYiXyNO2QQ4r7UUHL29XGs2yQYX4hNGsn795zJxtLHmiEgAHGGm/Bns+5+86dY84LvZZFAiyLJ5rwmfJGqDKrhmhyinyI/h6v8lw6R6YRBozO8RaUsbRMJ+k54j5cBguTP4qHCfYzGPgQfAthLFACUeyDQUDTnkwkLCxBm5UxnJyYWLh1CxA9efrU1/70GYMHd61OLV1lDccmhkgeR5hbYb2Y86d0DcwXUm2pQe1W5gzRC84XJoSUVi4dxblJED0cGc6VcuffOhcJRPh8YkpWVzaQ8CGx2PpJkQhySCY2SQjzgaffV6uR55zwpEq5lu/rKlB3b9BTqlUnp0cJAczXyvhVc/XjH//wn/75Hw65XMl0mipYRLVevriG5/PthWXKF1bKLZPV0YDDJgupy0+ovdPsJPL+zIVzhXIxU8xiUHPYnORGh4j9q//Pr4WD0fseOX3z6jzmz7XNNX/Q1TFUFxdyPvQo9cqBqaN7I6N/+Qe/s9dlmXbXRh0ug4EAiKLXorO5dPiHMvFoiMGgmlQAQtWwzs5sftfBFtR81zkAg02d4lntYOsGDco4BzrUqIF2GZAgrbbCqbJOgBHompA2HhB3PwSVFs7qVoetXmwBrjabC+enWr3pcvqpx0LlDDQ6qCAII8Q53et1BvfuCoViBCtdv37j7bdzdrImeN1gv+RGAn6VYsB2s+PWjWvx5fUjR46JXQdixJ4XohBlCFB5osTCOxxXSqpxfPIXRo88NKuzJ6rdBOHXxD7Ad1rQshgGFKPk41il7gD5Q+CIcRaSUUFqALhZERox5oyCeKBORoflIxKx0gZtjc1d//D8zsBpT3FRRvIHDbdcEpFTnuBmmt55ZHtR39W0OpRRVQOv3qIRUY2QfdedO+1811l6oQnpSl5kRQvHAbRwt6K+zJqGB2QJaQRYHfCUlEzgWH5KJJJSoMFg4VMroVu4PWN0BwPiSgmLj58P5Bk5GOUnPxX13e7IFqionzKgMnNY9yjKEiTTdzgIEkZColAoqMxpt4JsBOHQMxl1MA3KTjxFJN0Gm/Idl/KEyESi4yasFk0yAwn6p9w8+mqychBIQblSOwpUAkiaTQP5W/GM5XmJlcwiz+mQ1GEVxaMeSbFbqRls5bHJ0Oj45PJKHFFpamYXCfM2N1O7d8+kNsmQl0TdeunsuVI+5zObsovL5VQWBTrZ8Mmt4/NHCPQ1UqSlQyIe3/Lq+tJLbyaW53/+H//90dHhaiGD0x9oMTI8du36jXvvOUG0g9OkwzRq6bcIBqDr5Ogg8RZKWzN5RVQtCzQMBqMFhXQLZXKliihTk+BDkzfoBHkNDflSG3Gvi1CgttGmX12aQ0CGfOP7isoAzwuhs1achXS1qjhbM9ni60YZH+aLDEA4eHtw/SXhgD+Tra9vJLo949Ruv8MR6NTLwdAIznTVCsU9Xdjgi+V8LChW4n6j2a434QOmp3encpVyqZavxKmgMjE2ycQ3M9nZyQlA22dHaG8gvLK6hJHqtnBSIte71xeoNVrlSlElRFN2BJzlVGQtxJUN92xJHU+Ym1oUon+R+BvEaAZPTL9MmdSWRtOivOegMdevXoNwihLCQtHJ/ujoqPBVQrIpTW9FrsV1Gbwvhgy1CcyrZUlrEE7COLmNeDDAhRYcURN6eA5AspgoJYmP1YpgDdQhBNMOw0hTVCqkJ9pPDmQhbaMLDr5n4yrXeR3neYROsvE9bHwIndIev/sp2pTOSoydXOWprQcbQryh9bBtzVqdF3uoQtBuYwKFAYWysoigZzhsc5DHlaEJ3WWKzTyOCgQhFQMwTBhtDg8Pr66vvPP62ZW1VbydYdVIr0i9w//+mc9MTk5nkilKEv3kT/7k1/78GZzVjE4bbDUrCxmylqt5Qu4WhekwQzjNTXwcBl0KfMO2dnB2AEVSEtOgIy0TlkIcyN/9+BNEHL3+nVfLjerJ+0+NTY2tJ1YpRTZ/63ohDxNAasiBLtL7N//2XxNJxWofnxg9fuLA177+ZQJ5XT47mZxHxqZtDn+x3Jid3fX6m+ejMf/bZ85PTu1JpVOxoXFYLqpbHzq0l8IJpVLDbsM65D18+CiFbK9dfmdgssEjxtNpq80Wj6+jYMvm09l0de/eyXKzYrWZ1pN4NlSHwyMPPXLv1MSuP/j9PyyUq7Fx+/h0WNd34uphsRqfes9Tufnbb7995elPfWTEmc0vvkBFRTLoSBLOpk7q2wrToZC5oDYQ3N+A9++e4x90/AOBAeBh6bHXWt2SwkDp0F0oH+gPIqJeyLSKWCpoUwgwYUsAGKx+DfO5BR+qEEo40q2MUdK01168fQuvKyIYr1y6MDQcnZiaJCoErf5jjz1G+MDS8gJMp9vDgNlRBVFO2GpC5jfms0VMR/oftntqDSI4LR67EXYMk6Hfy1h1XAHbrfXGQ0+Zf/5f/Vi9f7nr2LCGKNxat2PW1ZgDibyHi2TI+BAD2T65IL2/I27yAQQjcQpqzBcp3kb28uFAl3wkT7CpY410ie+u0mttn5fxYtPGRdpWm3ZSO68R1K0L8o9qf+uOLZJMazKdbKohaZxm1WRoNzJhXKRHaq+uq/eK7KkdqIeFmEkfVFPY6+W3fMIWV41oC2JHYN2iu4oLEVWzUG5osKidCdEQYoxMI9y0rg2RINAWhTNXyJerxxEB32TR7VTQ7BLjiDrVBKrE5M6LxeGcNUkJXxyBzJjQujhuwGvDcYO+CbdttRpgDar9UZwgEPSIPV44fPFX5WYEa+kMZBUzrajk+GaJMmLwt1y9hPMWl3YWM39QXTI0oOuQvVHvx+TrtKMutYgb7aBIrEm9iZMRZQwaHeoY2lBywiDik0zF3IOHj5w/d5HgmeEhIu89SFekljp+7CDxOei7UmtrCP7kACilsyEnKX8MGJNGJmeaTITbrXe7u1bz+3/8Z668fQ3BMeS21vJJS48wmRLqVWSOUrkSjsZgPFgY/oAXR7N8mlrgcYvZScc9TnOLPFwOS7dVcdooP1x3uB0E14aHRm8uLeFGRSy8Fe/srs7rD0F79u+dvX3jKpkFiM+npCAJnkN+/+ryCisHetZoNOEMQItVCUUwka9geDQ8NBapN6tYu5GBao1ettweGdu1Gs8Rx+MPhBEZMeZFR4aNbjO+nasLNx1G0Tn3W7VY0MP0Q8IdHkKX8NwyU0tlavYg3tqFQi7mJ920cWNlze52f+Vrz372j18YGiMIuFuq6aZmJ9K5ssHmaCBwdXr5YpkP79SqOOPizAeTRDp4oZ2KzEA73ao+FUk5oYjIqUApBk68ZoLBMPQSFXGJRP6MNsGySsnMbaL8wI5gMgFCXAL+ilXRkdosdqF16NYh510hnOhQ2DPUNAXgMgs8ApFGCYm2FlIkVNlmRZAVjxUviuuWnMEgIV7UsirFp5xEZ03imui1kHx+yktEtwzJFzn4B25quckipEFIKA2y1wgwfWbjEvdomwC+2naa4pc8LOy6sCp8Pme01pCKOaYb/IRdgFGQMcGvgl5qG5yriNNbb4FVJSVyp9FxBj14wvtGwg8/+sDZC+eTS3FH1IXWElQJ84G7gAfzh85QTmf83lBhM0s1T+k2Mf+EKnk8+XQJU4EwM4QtRJ2lbM3sk6FC0W+KYOojWaoRl1ixgPTbKJzQjlkcll/79V9L5JLXb15dT65T3zeby5XXyvT86ANTJ44duHb9wu7dY4Gg+8r1i3BNOCLo8VawuYdHp8ig4/WFS5XW9etXh0bc5DmlVg9TUCvXVlcy2BzJ9UQRD0JhJ8fFXSOfz6JNQ6wKhn25QmZq1xQjw/g898WXYRGCfqvdiktZDCmcJO+EEe+f3eeyeZ798tfiiQ0r8Xhm08Z64l0PPB52TWze3Dw4vLe+fnvlnWfv32f+5NOT7exlTN4Ovr0BkrGDC0lJvD3AfI0QFOaHudKmSZ0SwUP9bZ3kfq5qqBj6oh2rR0XTA3ID3yLn0E67prO7ddUWynSdxafrOXWxGadj1EnAZs8kRlXhX9HukZ2LQsG6sVb/uNVxtLhW/KPf/fPkcllXt3aa1M7qof+yumzEXhWZrTJK6VbQ68XPP08VSWIg+/pweAhHLdjZz33uc5HhsMsNwkZDKLydmey4Lp9o6ci1An89HB2qlbPleoMkvnickNqzVNVVio2pg7of/onHBoNNu69fM1D+tO4N6nqU42BM+H7ZKyaFQzC6CNL8QZzVXkiltoRkPfzATR6/a5PGZCHc2bRh1X5r92qPyEhvP7r9r9y13aCc4x7twb9l/wNvUZN359lt6rvdplwREVa7bYv6CgPBn+KkpLKChP/uSMBbgi+yEn+ol9kLSWYuEEpE5yxGTDKCQ5WhwSrkFykZQZUZw94KZoGmgmOio1Hi5RFypAcSsCHGM/x+K6UqTDqiKmQG9IUg2mk3qCfvcdvg44E96iqB1HiKTmKqx3zFpgQ5vkS+T32MdB6hAqMnGgr0FPxWdykfBfIwIBCL943o0LkK2sMZCp6OWg+QQxqU474+Eo7tgWUmD2W5ih8s0cnT07tshOEmkvANAZ9PfAWJqSxX6sWiqKhlKgcQFAxk41PTyXwJBTFGMXMgsPvYsfxmhniYqxcu3Hfk/k7Y/dZL3xiLhaqVQsTnDPs96UwO8Y064RR+WF9fzSTjXqfbZPWiJUaEXS0WeqU2Kl+y+leqTW8wlEwVcjXyeFBnyau32tC1s6wTiXWnHd3jRiabIEjJ66Zkbz9fzPvcTjTFdvlq9PNwMKLbIVyQdWdvU7yJaCaxhTIKmBLzxUap1FldOzM2uQe5+dTp+3LF0ti+MR1lB8P+3JuvwT/t2bPLYehdufA2t8bCEUgPYQyu0TEq1rtruvWNpMsfdjo8adKJYGPL56kqnE5usoQQhnAUwCepXsqTQKpaKRKmKiYiHOK6UrxFlhyOYWoDMAAVFBiEUaMfpocqUbbUIOI6YIAUCxrmJzADpQHGuJlpBS+AgmUdq1XBnQA55IGf0E4xlQnhEWoHAQYmYWcg1VTOYZP3KRIIgy8dUIpo7s9tlp04xRHQqsiktAx3ygJQ/DiyO/Bmt1uh3AI8YldusAfjI2vyagHTuzZ5XBFd9ryFn2z0XCPbd934XYfcwG3aKZ7SngVDw7LAAzAmAuZCjhWfzZ24MIHFWJqqP9wg9m1FkmURbfeB89IH0npgXbCa8W90h7xDkSjVHYI+P5X9WAB1h9Pr8SEfq9i1Hj5cWDIKyWxwOExTJDfl1bCs8JIOl0VeQYhus1et1I12Q6fcsYdsRJt1a4JWxX5lJi1Ai75ILOqA8C3Pv/zVfxkajdz/8P0Me9/opGrRxXfOEfTw6KMPTU+PNJpg9FLudqZSpkS3hdoDBMOEw05i/R66/5GL71zvNXrjw1M/+dOf/OJXP18hDK7RNRm9jzx0OJ0qvvnShdlDByPhESJoUFrg0jgwtooU2hvDmlRAkka8k3l36UIhO1/tsNtJtAmX5p8M3Lx+a9fUeIZSS3Yzhpi1RHLPvllkaL2h/eT7H50LL96z6/5nPvtH7//oT0UtaxduvrwrLKYSuxljNyl3ekZxvttCVhqSBw7V3Mg0/b/fCDlDbjESMiXOhKI+FA8b4hckExyzKyLJHdYPBlPAoYfBAUDrkdZEZ0HRi8NloVQwNnCDqZJTy4PHptlLQ/k8Gbx1cDO478C0UfubdUMaV0Ii0pnC1NQYvGuzXaUmKRnsKxV4U9IfYeqn3AxpTdyWBmDX7gZCOp9XV2rqnvjIVOCeWCN3gcy+fUODHgsToXSZgB+yL91TQqQiPhpwi38HwyfntzcF4Ns/dv7VltDde+2SOrN1l1AHsfAqsFdWHFnBsgmPI7pnwYl33rS90OSNDITaUC7wn7pJ0VJO7yxIuW37ae3kXT/VJTG4qoa05uR7NTYBOiqCJPSIRxStFb5aboYAK25LkSouielNRF4huhLIqw74qUnAygVarkI1WWRbSZ7FAQqPKtIsqgEnsBX3VzawVa1V0XBKm9g8YvJQa4K2Wg2zw9ls1WBFSfEHwfZ4SMgYSmeSZWoCoHsUPCUcAvAkFE8OmUEO1BBL8hMGSlNNSLoGmV+5jUHnpHwzcIIJjRHjw/E65lFOkBmHrBfpfIFIUYvZUiEFZb8P02dzOuvN1UQyhRYaZvnC2XNkvEc82rtrWtcuA4QeO2VPg11iLVod8aI1UcI6Znf6piOjBqcvunt2+MChYqNx7vzFcWqNjo1cunhhKob4bc2nN+EhiHMCaMm6LIVnkQbM+onRGDm+EOJLDSBGfBCIQiJjCPHHTASSdjpH0bZBrzkgFrhaa7usLnQFKJIpu+ZxoahvYqeBANttVp3DTMxRKp/H1iWKGpQU4vEs+QEoY0wWbYzf8E2IReAFCyLwABGQxHx60t/gyFghHCUQaGwmNhfnyRlkqZfpbzgQQalP9gW7w0MS0LVECg91Ut9K4DD+OyPjRcmqayeZUcQXMOAcOTRMdOmxQ4dAtpuZzOJKwenQZTYrnqC9hm3coLe5LISr8KlCEZkwJoW+4vBMD1AwSxLtbrVWZ37xn4L+SdJ5JNp6c2hkOJ3KItNTjYPMG4j4tUqdS8ASubC4k9T+rGsoLgwipcvtNsLHhfUCHoi4YM8VBU89GypFKjGDZIQwAardeg+2qkiXqMQHJBz+4aNEYhTLpS996UuAFGfwhFALjRfKsgLMILecYXi5yMu5jZ6zcYO6U1rmTm0vPblr4yRPsQcmeeldV+T+nY3zvG0LP6mm5O0MjcIF8iJB8YJZ+IfvJ1sez9IHSK/WT34qnkFwCX1m07rK7XTX5ZTAMCK8SQ938+p1FAZANUOFZhv3Cio31YlypTCO1e4fj8K3oWMZHR4+ffrk22++Va9W+QJioNSKk1SMJEB0ht21ellIAWw1/6CEg0/ANCyBEypxD0lmbiS9M/5MQqKQcPK6tXgL3/xf+LlfIKcaSTBffe0NKg6dOvnk8tLc66+/srK8Cnh7fSG7xZdNFJ+Z//L9D7z7wO7IhYvvvPrKhfc8+uHLly4CD+FAqFysHNnvgW3MZkr5VG1mds/J0w8zwKnM5vUbl7Ei+cnWUi8O7L1SvhALO6dnpnweL/xTs1rLpbMBt7dUyL7+ysvI6fgRsa6dVu/bb54bHfU8+9zzuWTuZ3/sn3z+K18amZrtOzrj+4YX37nmHjLnFxfwYotYnf1aVcIgRQsqPJZMiIbwlZbx7vnVjmU0/hc3wITlgdJHxIuuJHDEYCJQyRBLBJKop5XTq+R0YsN0AdaGfUUvQ+yHTW/HSQJxFuUi01HPl/AP93mcBAvgs0WWjkq9Ao4lbB8Bt1yr9w1lLAoSrqc3kWsPRxli8FlMmWyhXm/GosOYivSpcjnotFDntVyr2j3oRPHP0p14VPfgh++pZd6xemu1TtZg7wYDNhwpSKskZlxCUiTTEitAGwScb9SCEQsro8bZv+vAcP/OvTvHcnBXEzSpmgUQtfO8QiZH9to/O+tKnbz7WXXizk67dFfbfMOdqxxxA2e0k9IzcJoiPHJaBF8RcLePuU3S83JGLLuC/fgtaX1FuyuLRdFdRX2hWRoBhgZzs4o1EggQE68y+jLrPCc5NJQPOVEe0AK7nXAGq8/jIXszjiZYo7iOh5SqVwiKQFkn7Dk+y9VqSYyAVmaXhH8KjHU9rBSIKPRD7tHjDSv4DuQBplYiDbAGawLEA2ziQ0cCM8g0Y4zWWwZW2QIIDxTvaSgFBjNJ7QBWkIyeak74kI54s2AQBYOTma9IdujbDnTIej0hKEAtpbhu3rxx4vhxGAis1CP+GJ7J5Bii6+B+u8nmD/o85Li0AWeI7/hetfT52jCFeL2+8am92bUV6hvOLy/aBiNg8/xmyWUnTSNCf2NoeJRBQ1Bs1WsE6oHEE5tpuzuCc6ZE6pKNw2Ynnz4KW37GN/PQHuJgSW9pHLT3H5myZNIbG3EXAm+nTfoh5DCGciOVkoJ6eqtYFCl7LJp7ksSiPxJPcvh0ptzsgkyTSIHespRFSY/yLeD1VlsgciN87saVS8OjwwanDZV7NVcl+bZp4F+Lr9aKOQbWHx3Cvw5Zxgmr1B1UM/ng0Jg96MJce3uBchWk/sK+aCYHBDmon3jsPeRe5uz80urzL19H/U6iAnLDSq4rljLqEFQQapNkGPhGSVo6nGlwuaWQVD+AMdDnhUsDkKFVFUoK4jGm9L3MIHSLDfCQ9Sy0VTZZAgCZKE1QeAxIvkU73MBT3IXeGNEQcCKaKOQPkVgREl7IFYVBEdO4wWV1YFjmXZQD8gXxcnOdOHGCe7753Ddkmcny3eow0jQnaJnu0SYkDTzOh9M4uAl/ezrDzfLU9sZPWuaX6un2yldXtfO0tvUJ23cKbN45yVLdkgc0lKLdLA3yHtWedp1HuFOxC4Jh5DbpOEOj/pVj6Rh71gV90rRFhPNmNnIVdxlrrtkttYnwGKeZNuX2nLaTx0/c+8D93/rWt25ev0GwFnUJYZHj9TrK5CZZ11mggy6GEqg1zn36UXM1n9dZWYtMkOTI62AhRrkKSSeej3VqRjhuNTv1F7/+LeqhkLJ7YtdUoZipt8oHD+/de/BQsZDZSKRIsHlg//FauQObla6VWjUrSrFz5xdnhvcP+SZnJvbSxQtvXadufDnbIgho/vry89968cD+Y9Nje67dmPc5Ao89+l7Uc1dvXEIATKXdu6Zii7fnqHW2uV6Dah466CTAbHbXbhwa/uJP/+zbt5bCvgi8qdcVIC7/0js3mi3j2Oj46uXVsUP2G7eu/sF///1K0pK4ufbkgyfn5pKllZXxkf3ecLCJQYQ0eU507KWdedUGWRtn7aQGDmo2ZNY5+F/d8GokwQFcDZMG0yp57m0u2CQshNgTWL9C0pDqhB7LfDO55GcgykToMnjTiE9E02jyeb1+7Ahmpy6ZFCkZ1Ip0RN76NrK1sl+4qMdqdFK5EK/S+GaWRNOlKjkO9KVqK5kqwQSQpJPcfeSCbtlU6el0NmN3GwNDzpWNsj2oe/+P3qcLt0pLSzHcqElcyWIxuInUJsJNUV/YegWA0ByRqSDK8hM4VqMm1/6nQyNPba8xDraaown5dkB7R/blJ41pDWqPiNcJLD9nVT92XqURfm1atu6Xa5rpWEjpnY02tencOaW9RTuvXdL6tOV0LekkFe2F1ordnh6p1UE/5IzaMwKiPYamav5WmuyLfynWXzH9KroLeYYiatVu0O+hDGO1g0+JehFDrAAHRYScqHcCZF1A5QTjVK3mkXiglVYrlZbBpkJ7eQdDBR5DwkEXjSYExR1hnSjBkErW43GoB8orgIkIGtwIkBLwmKMEALgVHKe8rrSYYFE0QGWlVC3cPbwV8MYmHUWvLEYv0CJ6RjF8uJxo6SCKSMAoc9CdErxPlG+t0UQtBs7GT2R6926QKV9MfzbWV++7955HH30UV2QSs1UyjXwqUaqWoZDUARbHqz4xbb1WtXr85IOFSpf6BYWe4dZCfPbUfbXaLVxV1hevnzh2olvJJteShMzSJl5UhEZgXxzActgcyOzxRBLZAl0Z4jnvGXTQ/7BArMIVdAeSbL5n4Is7tSbqgKlg2D85xfetLi50iGDHK6Y3YEUhaRgtHoQvFBBoW+HgcVMmzQheMGa9jRQfAzGwIggSnGtmhGFi4PTFhEAR7GZ3z54DK+sJvLfIiGCwGzIbi+HRofm5m1RxHI5GYuPHWoXcyuJiKok2nuwrA4/JgnI2Xy3in4YLLElFUAYMTY7CYukqJSypazj1KAPnNHkQ/L4z56/D3HndZoPNWUd0JsiVRDodZFPJuYHAii0YgR3SKdUCyMdpMYeCYWTfZkNUl4w3FHFtZT0UjCClwZrUKhXKsKOxECKhcsID0kJ1AF+wEygKP6l6AyjCEQEbOIuCANk2CbCBKsgyYwpjL0YL3EjhRaChYppAtXDh/LXds+PLy8uk6f7iF7/4+uuvIycJFwiAw2hoS1PtGQqILnumg0sCdCIKyzJnYbLnDAfangOR8uWkXNI2bf2K8YbzIE+1yrmfexAeWS8wK5JkRrGhrByN0spVIZ+i1JEjYQtEs0EfWADyn1S/YBBYyfIESI+uyZ2sfSR+Bly0DjhkwY5UWA1ENPEVWI2xz9Mk+FyUyjp8diyEDvNV3JDeTD70yIMTU+PpjU3qEsI/4YNdo3YvLowUxNVhZA3gUU8dw30ze869+BoEWBIF0AVS0GB2p84YfIlJoTNq6q3Xg3sD/lgAUPe4nQGf//byQng0gObGQXEGyDNJvFFS1Xr5dAPNHEZtc9d54eKliMf79svnlucSJlwhwkG70xEMePOZ9KU3vl0t10Lu2PrtuIMyQANzJpGdvz5P1tX5azcun784OhZ97NHHjxw8RP6Q/fvmrl29NDo00ak3C5nc/j377zv14Je/8JV4KU4xnqK9GosMEapvsfgo/xCYwMMDryWLwWYIjARgkc/M3Tg64zt6/OTtRHnYYhhzhqn9Xa1UwFNK4NGm+K6pV8KAYpSYBI30MtP8bXFUMo//0432AEHFLylKTBEIs56MyowO8UZ63s2cimghfRAaDEjx08gwInagcmLdovcxGLsUpvFbg06XzetzOkidYrMCHVh8wHvkIMIlI6LX+wJ+tHoT07Ok+y5V6za3jnTuzZ5xI7mIH4bd7Ytv5hBcSOoBbmqQphJ+NZ4uoef+J586PH54JBV/a2jKUWkQSQlbqusUy06ixuiXcJWKJRSCBNhADoVmAvoC/QpGOeCEBus/cFhktTCQavvuYxnfnU21s/NLDrSndp69+5paVHLL3Se/5/juxuVWde/OSa2FnZPyE5IkC08QhDBDTJ6yBPF1SvpX4q9YfIW5ljGA7EBE1V7WjqiakU3Q1gr/pORgDpQ6WlTTOpJTAUE8C/XlJxvelYxnLBIVtVO92iGOm+YgelgNDQOKOAvqA/2LFVkGGcQBoCA8wIzUW5I5LxgJcjZRKYMiSNwKEqd9dLY8yHmOeRbFC1HmmJW0wYLtkYakqi5cmWBS5ClSg0CCZapNBGmIkYfADNg8cA/vBVGDUiHJHpwOiC4iCbMXryIjuinsi2HqlDaN6D8JSDjwxGMkrZTKqc3qlfm5drUSpgCN04pnELHFlCqcGN8dGZnRhccNa9mbS6mm0T59YBdIx+EJAJeHjhy95/ihzOotp6mbWrtNF8OhKAYuPJxB/0RB4HSrb3eslDINhPOZXMDjRFAjzSxGKeElTOQirqEJJ00B2BDjKfJWdu7mZiolk4WMoseuM5B7nJ6Z2d2VGtleJW6VgWh1KUlcMzop5ANlkowlpNuS0FootuJVYPN1xh71N8kKRPvDsSFPKLAUX8gVN6xO08bm7X0HD2XSCLFJS0XKE5GrIxAJI9z7gsimAb5w1OmhBBKh/fDKE7umi6ur6H9rlTI88ujQCAY/1JvVYsGFOd2iyxd7zqCiKNRwFC93G4p1FhJzpzYBfYElSiuiPCFXPtJksVGpVcHpIa9nzBfEO4YwMAJjmGK3G31aGxM+tJnHgXDh8gEaSK5AJVhIjwq6Uq/C9GBzRnvCeTYkXWg/ci2imzohAUgAHg0wtvBhZC0AE12bm0f5zEhhMuQGXkGr0k+h3ywKeRUSO0ACDcamCEdFh7kToOKadAAMIjhla88B60zObuMBaU1tPKid1/byLWyCLRgcuiZ940btKgdygcvqBu0YNp1FwPrW7lInZXForaDlUMfydj6ZPY2wDLB/o4RyOl3k1WRMPG4Xtl4WK3ECVSaPnHR6lLEURGlfvXhpfuFmZCR6+r57f/nTv0yU39uvv/H7v/dfILxWj4we6MPuctoIvrOYRyfGz0c8ovyEioM28J0ECcEHiHKLdSr0YeAa5JbyuVL+wIn9tUoVDPJLv/RLn//aX125fu3oocOTEzMUOHnn3Plitmi1+EidSHB7t1YatPTFWqlXM6zc3CDh9fH77jt8/NhXP/fs5MTI8QMnI6HwY+967Ktffe4Ln/8iXn4Os/PyhYt4O9+4dSWT32T9vvHqW2izR0aGThw7tbkez6byN+eu59KpD33gh+45frpeaD3zV98As2QWC8VIORoduTm3QF+rxb4FdRLIzWWL+GMraxulbNE/Ov3Uxx5avfBsv74Me9zpFklwIo69jLgaXpmh7U071k5sT6NM3s7x9o3/k39hrBFIYBdFO4lsAp9tdXbbFVTRuKAjbDC9tApggObpCUhAkrGXs5yHEccog/5Zb4DLbxTKUtgNXoSMboJY8GaFBMCVqgBNBNpUrjY+sefEPaf3Hzry+luvN1oNty+y79DRyek1LOuU4bI6XFJqC41RvV31Bzx14jd6ug9+bOrhH/twu/hyS5fXu5wdjL+GLouestW+YJhKmgLHQDLpE4EKUU9xAwtLfTljzCfIZ7DxFSIXK45GnfhBO1kM28MtLQvXIW3dPfqc2JoAWpNXbzUk8yErS36y42DreItF2rpt65+tXqmb1SPa+Z1H+Kkdawdaa7Su7IDyL+tOGGLhgzmgE0iuQlZlL7w7dyCAiPJZbL3cCY3csvuKvAq9FMKJRlpopzDIioKKrCG3adSXPHIOB5IhE1ohB2uj7qQUoM+HAIJHDU7AoEFWIAuQbuANiTpQSLsQRDIjlkETqBy5ASDC6Vccp1E5Wu3aIINGFVAJGkVCojMqIT0ctkQJQ8YJf8UFV0WpYA4xURmUatrQY94ufSAHtMLUYsmQRJZdHEZcdheSEGpbELooEm1w+4NcJouTjggeEvfmT23GCfAnrKVfKyPARwPegdlIgn7kUskcFQ5F7jmeunQre2PF6R0qVCojs1OxA4fW5pfJnFWp16dHQxcuvNNrFOwAYY0qvmlcvs0WMLwbrpFsl0jBfCCZu+rxOCVoyDdCAki0vKRRYHzgK8RlV0Kq9Yi5cLsUmSGjELNls4gTDfNH/i8y8DdaldCQjoAOs81jN2FiBtRMVbCG0A7Gpk90aKuBphkeiJULS+IR9hPO2WIl2orE+miji7Wi1019CAxAFbfXnUysMDK4J1fKGIcMgWgQRzg6w5iXG41cAgUWagkPD6bSG+V8ztruYq1lggj8RzuxvLSIur1bLkeGrUNRH47LWK0wIzKPkndbU1QIRwiRgXIJCNJXViOfzKIk+gtbsoJUPTz2wDu4//77X3311Wy2gNiJ6EMO0YYEE7bNJotarjQJbhITrPBjRh0iLIJ4LpdvNZpiokRdI+y2LFEkLFR1Ir+CzCCl4htPMJKeTJZI2Kigp6cnyEXAWwAMQRWgsy1+T1Ye3wjYc0muqpAkJoszqFd5NwSYPT929tqBRkoF06m1Km0qDTNXdzZpfWujPBdzRS+lKe0RbuMRimGr+0UyFgabdtQFFPRwDFr7IAxgWPHeqCCF5EnS7e0uyYOSpgN/YUFPqEAwJUm9DHmVrkUxK8YV41CLctdUi2ZdoLrCmJp+5s8+d+XK5XvvOU0CcWofYUzBAcrmtGA7qGGSb9TtbicONkeOHV64PjcoV8UqT4ItSZSEzGwWcQ2qbMBPTad36chovrRw24DDoNNBmLXb608VMrAwqEpd7sC+fUfeOXvxyuJNp9WxcaWocxQnJ8dW5tdx9+8WO7NHDoU9kb/+ky+02jWS6Vh75rQnkV5N/pOf/iVda/DFL32lViyuLy5fuXaxNWiMTURYd+fevgB7NDlFlQXjyuIGJTRj0ZFmtfXi8680yi1cvYajwfGhieEnRt5+80wivjk+GnV7XdcvL5psFCO3w9MnCvGBc+A0+F48c240YJ1yunePHqxvnG+WcxMhB96YCrhkDLc34ZYULMh4yibjzT9blEKd+l/YyRIRRoYmSDdg09mdlGkFK8PdcEbERlaC+BYBNyL+Qt4I80WBRz/Ia4T1t95AmYxqEh/ECouR9Of4NjrMEjVO7V7iNim5jmYkmUkYLd49B4+6/YFkoZAiBNNkGRmfIu6r8+1vL99eHRubIMBMitwRH+AKWFfT5XseDPzCpz5ZXrtY1S2Pz0SKuVUHnAtwht85qhXy1EBJBMJkobCC1EAoCssqV6MDoEIgt84LqVTwePf4AMRKGyyAqzZ+8Sg/t54CgWhjq8bi7ke1G+S0eheXtAP2O3/aye22735ajrefE9H2B2zb7Wzdyk/mRe6UVSq6aFEQq7eKs7uooOGYhUKzNAUpqDxWYgAW4io6Z0VZEYVRv6m4XhltIcCY2SDbNAWHhVZUHbKoUO1SjI9kGvG1dcoPkT9lKBomLw/UFxsMiANdieIR9US5UBC7UCzXyEOGctRiqLT6sajfF3LnsmlcAyDkMPTkS2O4IJzgF3z5YK4xWxLECJntGMXQKcVoRbwT9xPzAD0QYb6SywHkQzVwZgVXbPg79iB4MTpub4iDEAM+lFB/Oz12OjH9uRxW96BXq1bw4I3GIqVcfu/0dG4jgbhWTWeaxZyZ2mMEWpmIbjSOj46g1M6l0+Y33mpTSadPzqyE2xtzOT3JmwuQ2OHhGddTP/Ty818Nu3ElJJqo9vB7n379Oy/wXpzUSKkLloEjoYghXWuTBY4EDqk0tjeihuAA6GCxlIElpSkMG0h7DDieFD4K/GHKRddnxHPKAxEKBkKkEVnbzKxtpCl7fPjIMbKoQ1aJmyW+tNppG8nvQeUDXM6cHipy45UFCiSug+DaVrfObSQycWJed1sI9Lpx8+bk7LjT7azikQFrQhEIh5vz4FZdk3xdZCDyZPKFoWEJgcXAoDNYKIlOdsN6sUKOZVAqOi3YNzgwnn3w4Ueix45dev75g/sPpItvE3bcKBRRShCBJblGIH0CUaxE8emFjWc1CkQSL6Ub1FUQGlpo3MpIPViqFDGEHzl+DEnrnXfeya/ncIv1un0IwQrHCWUCibCHfgMb8PKT41OYezOpPAXUIMnyLnSuvEylgVTWcRtugPUKuUkpOmXjk3EPwzPQ5XaxZqjYg5sgsFHI5mgQ0BJZT4mk/IvUx1MUeGYc+ASYALtNsmOKLVtkPKHBbNy3DXRYW6BFGvKhy4KCaIQbdu7kQC1XoaIgFZhUGBIAnPMsWBkc2SRkWjugFY0Ay0+DDr6OsAMU8oLi5L08CF/DscL60gIcF8+qcWaB9bt8Eqwtt/I4442NhvsrtaaTkiAG4uyo94XHu04PzYdHsVhmjuxNbia/+fy3XDZSmhMAiwSMYUXymuFE3SzWjOFIq1Y/vHd/cmk1m8nBBUsn+Wg4MtT/5HhTsY72gJkqtpSMBMWOHhiu1suvv/7qL3zq5/7iCxkiFOECGRhqLqVTBe7PlasHTu/zOrxvfuNtk8fcKrV1RV0937AbHNWr8fDRKRJKvPbyayMjUTzw11aWP/3pX11bXzx/4UKjlSdEfiOZXr5WIhDO7vUXUhmCBtFYxG+lm8XO4SMHq1lS/rT+/DNfh2nH8T+9mfsP/+7j4Bz4v3PnzsU31vYcHBseHf+l//OXV9bjf/3lZ1udPFVn9k1PEKp/5sr13O3ysaHBlD/caOf5FhCq6PTkc7emnYnmBGOg0WYxNGxBxPYdO/Ch2RR2fn7fgbZScFjjRl6DlxiJAw09N6EYtKX0XAw0L6MXLCCWIffpGJMK+cQ6fcwrIA0gBMnXI36blNeAKPRzUrRSYgSQUsikYPQYg1QxMju9PrxwSvHkKk9MTY9jOyPb2oF9ByG96U2Cu3qmSk9nd5n7xk6pm8Gt/OM/cb8uXNJnkkEngUc5fET5agC0327Y3Q6EB/Axa1vUkwTQCMiKGyd4TbGG8rkaYZOngHwlr4qu9i4yLEPLYKpVxIBK80KAFclF4y6X5W5aEErHEf8I1PODRSnnmRr5IRvLTP2rnZdH5QprcJsAa7fRUR6UY2mWqxxqC4oDfvMLbl/a5br6E5CXS6KTgnYK/87DLEWc4mTFSq/kT9FgSgqBAMFNwjlQNbOHuyxNIsDRZSqXEegPPXZ7XODbKonsnS74rXqzTOeL+a7dDoE0eMwUrve4nA50v7gH+X22kZFhgm7tNgv4i/S/hQzOxuRtmMaFnY0S2egEccLKl/AzrZApMJ/NLt5eHhgawajLqCMK1JtO5YnPw/eVz6dIHusQjSyYh++3Uf0GgiFxjKT1lU9CvkGUwqUT8KIoBHUEcFbC64fKL254RCleRw01A9gc2RN1NMgAVVux1rR7naVqsd5ueQJB8JrZOEAL7bKZs+tN3JUamTSZZKvZap1SpRJV2SbQBzkYYR3LBy590PdyctNsj1Dh/qHHn0Ts1rmCl6/fsnmCVPJZWU8+8viHMolltHipeOfyUmJ07+FmJV8sZIvZDIvcarGUS1mfL4DvEeGzBOWYrXZMk6h1RCzDzOULonGFdaA1IrYg1tALB2WRRUUB/Xa47U4SULT6pnwJB63Sg48+5tmzz7u0dOns2/HFVQ8phHDiovSQwKnJHYhtLi0/8tj7L7z9dqmM77RfvJd0LXDkzStvYnYeGh/WtysYaglgqrX6dq8rNLFL5/O10tco2YgIi2c4DpWhkCuxsb7nyPH8RhKjLcaxqYm9/XD9tW99oxDiw6HZdtISoRhPpYtRM6mvI25/aHpq92tnbrr9JiKvMLpTKKkHHcYkITC6ZS+Q/uCDOxhUCgUCGvEdgyOXy7rOrn2zf/2VZ3bPzCKaAwdmmxjp0eFT/weJYCg2gvpU1rSJiG4HgcL4dn7rWy+As6DLcC8kZ6BhsAxvINAcawR8mSBGwsSJrOi1k+kETmmI+EJWiahuN+HajAREtZriQA0a6JN2uhMK+TOZgt0qLaFSQvBFR51Ok8EDqCDTdXFqapTlUyyiDamCIUBqdAqohdDSGYAf/oD1iFegNKHIMyScnrNo2Wsb5/lFKhp1wDHrW5AABzLvJJEWcVy4FuGH+32aovA5uhXS6qM+ARdI51H/WO1mG9ZBAzYLzCKkn0ERIunDUL8QlUI8t8NWKJTtNgAaNNUnZX+xXAAuYVkIaoIdaVXaevzwIOpC022LF2/ic1eqlTO5pAEXdSp2U1PYgnKlzUr3UpZjJXG+KO5yyfgmbeot0G6q1TKcwmxZHVav24rDOdVz7T5rTyaxOzEz/sT7H3/tzCtn33rx9JG9OH+ZBp0b1+Y7DTyLq7Uii86ez5Sb5t6/+Le//qXPf/nqW++g4okvxOO3v6DzeYqpEuYaWPZ8IcuEvPLm8yv/eC4cDg1PukgMns0W8epFZ1bY3Gzni2Q8j99YxGvKqSd7iOPamwulIoECXYq/g1TpIcqv3/wv/wGMNDO9u/tO1+l11Uuti0uXvhL64tSuifXrV8yDNmm7/t4vfiR7ezF+q7KWThwcGiWzuNR8IbMN3n5MkhShlrBJ8LkkiAZlI6GCpUU0BfNySt7F+PBWoGF7fhGDZKLVVfGn5ryQIfUHisZaD6YyORA/unBF/okJXYFYf8zo5H2EkyZhDo1b0XXo+zadwaWzOLuVxuJ6IlOq4ajMAie42WJ0Yr9rV1rgH0pTAPmYbUTjBEwOQNi2akHoNS6laytXc7nlkbHxoJdYtTwuzK0qxdQNTz3xeCVffu6550wUsWz0KYOsy1V1jz1pm9zj1vUSPV0BFgdSw7cJTQJJA9xkyBLCCEWiiyKb00eGSn3unZ2QOq5vbfLA9jFETnExO7+3D7Tzig7uEGp5Spq687TMriLId59SzQsB/cHbzuPb84NmVTrH/zIxcsgEyvPavGkzyc3yiXJWyDWLiptZzErGhQtlAcs3Mi6oADgGJyAQwJpyGwFdymxqgjSKo5MwUGa8Zsn5kC9WbVYn+tJViimjcnCZ68XOE++9F/UFcXJQBXTAqKyAP5PXJekkDZhWC4Vil/WPmq41kOCZpdUV3A7DkQicAEaaVqdBdPzM7kk8hH1+VjN960h2Ux01gAN+j7uQExdTQjmtdovbTa0/J6ONzEo2Cb4bgwjJIeCiJC5SfJtNjVYddAQA0pLyf+7jdkvZI1yzxZULXEhaHjpDEgbxv+4iUvdMOqiyxeXCLdlBtEO5jl/P+OgYdqlkPI6eDXkKcdnJPf4AFRowkYCOwfj9dge7NJ/ZM9jz5XRsPDx3/p3g6MzmrfUj9z6wSIK31OaBA4fIM40cQzxxNpto11or8cT0WDSVToD0xSEC4UZFBUFgxZMZmUKyN4mSmEkhey3F14BiZB4mhqwUKAJQKVOjDzLBh+OqRoI9Slg7fZHY6GggOubw+HXlaq5YEeOwzVXMp1u5xljICwpjknPFmssbIn03qgE0UvisAh64swobKmJWp1pMk2qjmifZgtVhcSzdWvT7YqvXF7PlEiOI9BwOR5YXbzvIGIK1v9M0D/p18ookNjvUMbSQoj06PhrDQ2pubg6/cTzJzeZW6p3LVOYgW9bQ0MiJE/31+GYpl4VIkpcafkfldmb6BP8zucJI8sE4aYO7MW2ha0Fb1iGVFur2JokyUuk0pSCQOX0+pzhVtdtuN4lkxURdKtWI6CQSZnjU7yIkF1JDuSTR0rMWlW+RcsfjWbfbBSTwiEBmi8Yl4zTUEKMoL5U+SEAw+mlgh6GGnZM7uUd1RppiAZGvlLhHEhQQLebxOUvF2r3335fP5rgK+Wd/t282P7WNLqHqY5HSGsd8Mue1vXagHXODbLLU1Z+scQ6AdZCZpjOQFmTEGEK1YcGDicY1CgaIE8JbKKTFDRh0zA4LmWeK+TLwiMXQEXbifljMliuNOlpWzIGMm89rJ+6gVKKgpI5KytgaoqNhx27HWmKjV+/aRgNNPGnCnlqirEO9YsYL1ywwaSa/AlW+qEkqbATOYB1yu+BuJkUpUHHLdyhWgy6JCxh6MNwdqt1quVAjtph2Lr9zxRN0B/ze23Pz5XAgHt9MJ7LR0IjJYF9bXGWA0HslN5JUN6IWyO/9zu9+4KkPVMtVqoHlKlWZLbAGqcXR75k6KFehdNVmvpMRvUivWyX5EuZtuqqiLCTULJ+jaLIuGPJZ+pZsMd/JdyBVGNF8fup/kt2zt2vvDNLghcsXqs1yOd6PBqnmqXvmTz4XHQ41OnlHxLGczLd65ZX06t5jh20pd4GEFmB78tjqWkj5JMBGGhDVIxE8BF9YzNQ5x/ohny/oWgQdIUzfQwq++/d3/xJqwqbRcTyxWj2dw28D5TI2MtfSGFpNfrLxYpqnNqRDB39EbDKfNMD0g76LTBkQDsGxCFxNfJ6RUcCeJq3sB00BThJZx4Di2ghh7jYHtUqe3LGQDxcVS036K+9cInzx8fe8S2L2WLi8EyEMN7wnP/ioO+bo1NeMFDvCXAMsICMJmRI2Q7rFsfh+ckawNlhURuguqspHymfzqQpyt/c74yQIQgZBMaryzdxPw9rNMgjySv5Rr5OdNuB3ndi58t0HPKE9tP3o9iNaa3JWzvAC+i4/tD7KK7UHdwgw4yrnuHnrgCNxONraGAAWNc0w7NwgRnf4aWiW8NOaqhlqLP7MWHwJDhBvHUGOzEsuBwNb9nh14aCBdKInT57cixPgxcu5HAZdXPBZ4Divwm8TVEtZJCr8uCFv0g+jASINkhJ8A1R2SXFYgivC7RUqiCWirB/E16UaZbvVgFkCLUqcDhXkG7jz4BRgGR6Ngi3AkmrsBxYHWF3UiWySZBjdsQC7oE4MnKBQdLxS4pmYUuCfbMOikxTYRThClgaqiO1BgmAcMIDU6EKzjbaFIYIaMYMwhuAsEZQdDsaO+NoW9YBLxUy/Px4dRWsAK2l2sspAdwMT+mt3aN/U4cvXl7O11MTBU7MxJzITXmNjkxOkzUL8oAtejOHGwczszBsvL+LfIMm/YI/xQyaJNktHdLB68qyi9Ebih/vh00DOsAJIdXv37SMkmjYlFZjUbjB1+2aYHYSq0ckZFNnFGr69HWKLSK6RIFFQuXrwiceX5ucwk5PMxtq3QGMgIVCYZrfm8vlJ68osw3wyTUwQWApC4vR5+KJ0KuXyeek86aEQn06dOHnl0hXMsKVG/fDxE9HZ6UoigRK+jtBQreZuL9BbwlV6zUYmX0GebtbL2Ywst5GhKB9FEiJomLPv2n38xLXrc1KG1+FcX0ukk/3RCaPVRLw+Km0F5Gq9Amcw01AOqB0MFKsXAgUhRk60k82qOxgdHsHTlRxr8OkYFipoMOrYlbEQy9ztOzCLarFUrq7G1wP+IMsRaZiMDcwyemckVDgwNoKk6BXTDaTRSWYc8sCMiy1DlrcsW07CAeA9ADSxYS4hJZsV5gsbdqtJamLgStfsub3ITSycAd3R6WtkQqUa8fz8vCw3fV8zeTC9TLEy2orJVlpWJFNWgxJ/tZO0wyU2TnKGzxGAv2vjJJt2D+Oj3an95Lx0o9+vSh1AoYIc822clXcgNmHoQzvkdrMSqTaJdDc1NYXAeubtC/Hbq51+FR4U2w6jTBYak8uIOsKCTcRixIHw5H2n/vRzf7navuEJ+D/50U/gn/XHf/Tfu2RLZfgERyBxSS95D71iz3AxC/A9uDMQtgDbRK/hZiC95CYjaAvUW0+RB8Ooq4G7da6wq1GuXblw9eg9hzBJ5TJ52Cmb2RWLjNy6fruVx8ppKq+UrV40T7ZnnvkCqTP+3i/89O/8zu8UqhmTTcqM8ieIDM4JZhrLk0VH36EfnOCTqXXXoABZowl0SfqZ1qBekvgTlAhUpEbFilbA6SJxdJeLE7M+Purq9Tk3xp12F+rLnXwLH8iswVl2La2QLej3m/7sz//q4Xvuw63XTjowS8BmrAyaKUBMP6hjEqNSF7odJhDsiaMbXLYgXaiDwPN3bdrcaaeYRsCPTWQm9ooscD8/RJkqS0PQAoIxElHMLzpdKmuTk0qIJTSfq0IQQMgoXTDbESFkxVUVQ4GUySHyv9Mnqlf0SaghKaFjbAtF4RmU8vCF4kgnPvz0GZuClE2jbFirA19bq4IhA/lsKpXOVxtkzZt833ufOrq8ZMKDxw6LYdcdPWXff3Sy3Vht6/LopgYYB0UdrDncKcgWEZtZEhjm03gfUMpogH74VKGUbOrj1U+QgjylSJkcbA+TGortMbzrvDzNXey2H99ua6dxueV7Nm6+c9vd19T4c0Kubh9zsNML6Q8fIv8r/MWBPK5IL4dCnIS+yrOK74DeytQCD5AYgQpIFpRXwsdEJ43yWaibEGU5kKhIAA/EpIfNp1QjiKr7yU88EV9PfPOFa8PDusnxkWx68+yL34JUKBGEGpxhdMpIe/BMjEM44gdeeYEgASlK46aEGBlncgkCXeplrIBGsKEVaTxXQ71bg+ZRMgVIYPkS240BsFGr4CxFLXoCbUfHR5EOSc8DYcZ2AFT1epLJqF9twkSBWGWMaUvXB0VajAY7ykCqOIjKQ1RzTBM4j29EEsUILREfrCq+mpqDuC+RcICEU2rDTwQhB8TEeNI+LB4U0e8WSwkGwlAwVKrVoe59s5P00ow4BYp1ZJxyeBskbxub9OvtyVQaO1PQaiM0oopu2e1qNck1m1tbbteqxVBwMkT9UkOP2GF06eRxw9GMuCCGkbRgbgdhej3J1QcBRug1mfyBIMXsH33Xw88//zxyZrMh51krjBOgzIJeWF61e/x2t59Ch05f2GTH/wU7bk+XypDJy9io2aneYDY1axVMgXwiM085hHwuh2lXvM8sJrIAVUm4CHVxSNY9cVmlbDDRIjodaRbefv3N6NAUso0/HMV6Wlzd8EWCxn6nmNkklLiQ3jAOYLCtGLlxdR5IEFm5LcYCBJI2RRFiu3bBIl2+cB4gBPVHR3yz07uTBzP1xkU6UK2T/71DHUnmB/Zre4M73pLwhG8wwTwRtEbYxYCqFQv5W+AOII0T7VoDjZzJKlFAkYg7uSmSsc/vL5Kjp9crlIpUxxsfHsbphgnVCC3Tqm14iSugkPpFPK5ROzhP1gGCDOBEH7hBSKAih9xGZ4RwEj/TIjgHER3ffrFlV4sNSktRf8odsC8sLUxMTKjUlnhmdVXLoDpgDVQjcqrGNdKyttEZtcCFrGrH/OSYPRtndo61M9pTNMtPrc90aeceoBqRXuBa0W9QHS0xenwsBJVJIbt1l9yZlNpWMvrGBvFjhsBEDPAG1DH+5TMleRGFmvlAg97nDxw8euTkA/d9+63XMtUS1HRqZnrt9hKrtVjEvQ/dp9Q8IYQU7M9Cg8jRGZgbxg3S7HQz8nhQynewuhnTXpHykWhD6I3OSdIo+HOoAg4JemNqLX1Vfx0aPTaO8WqsUW5cevvK7fkl+JzGehc6jd23Y+lUK+Xf+r3/9CM/8iMDhC8zwCNROYK2IE6gAgvaLMlTq7LKOq1mcdUsFwier9dKpMDGp0znsGMpl+TzlERT3gO6yKgTwJ3a508kC7gTjY+NW23OXTOzqVRu+Z1vQsVYjwCZy+Mk6Ui11yISzxshWTqu+v63Xnsj2m9aR/3pTPqe3WELyEmcDODVEPDEeRJKqeekmnzmVv0pIiEDzRTf2asT2hntNr7qzsaN4HomHcypLBg6t9/W7dco6UdCI0KBQeQK6zPcZgyzZj0V710wONVKg8pdSPzQbTxgEXBRMsqtogaHISAOAriR6DtWjSoVTFrDLoH4MqZGsgtDhikYBRNB2XVzbGSUCq1//cVnLl+9EorECHY0oGCi4Xc9cdrka6dSK75Qs6+vkWSNx9FgCQcNbuYPQisqWl6ssSBAp/p6bRS2YB7ol2/WxoV/5QcoTwQjUWWD8wBWwJtjNR6yWkQtBC5UhJDbZT0o0NfaUb+kmb9x2+rF916XPgivQGM7d9Cy9pM+sASF25H/6KOclz/Z1DwwKuqM8F3qj/NCbplB2BPRQovzp/KuUvIuqEYeFIkIotZHxSEFFVChmNA2Gz7y0R+xP/Bg5MyZa3M3EMsIIFEmHSof9MnZhu3X43Lg/IGgQDg/PE25lJelaMX3CY7UAJZHMQbXXMrD3qLsFquzqqmAp7LOI9FBVj6JWFiq1EGnRRFITmUUzmL/a2fIatMsCfApHg2/EHIhD6rYrlEtt1EcItlK8Em3Td4Pyi7gIYWAgxJXCCoEmGByuwN3EtwQ8HnFgsYsgRdh8rLlIkpfhhkgpAU4dypBgNSK2TwSFkrUkM+PVzfBgt1mAxcF9NBcdQc83mAAEc1CaKPda/OGb61t7t1/j97iyhaqy8uLkaGIx0mme2M2uVyrFjwOg9tmGBsKULao06iy+CPBAHXh26QZgZCCJFvdSo/gnarkMibaWjx40Rfga0OUCwSdmFt+Ag4sP0ielD/hP8Da4Q8HY8OByFC+RMRrt5krhKLj2D7fOXuWKZ6d3lXPbabzGYe+a3VacRcyO90ZcgpiGg8Eq42KoW/1e32pQh7n82I+i5BEpaY82mZmp09iDty2fFOnH2AmyvM3rs7dANUGM+lSMeVgfePq1kQsZFIag5bXSrh3jajNct/npae4Wcxdv5ZMJtC9E0R74+o1qKYKEDOODI06LNdL6ELbFGXCGwuCIs5TAAybojrYKa04meN/IKQZpyLJF92Lr65Vqx1CZdAoMoswJKK/Zomb9OQKjcZio2MTTzz5vj/64/9OTp7kejEYdc/Pr7DKYTzwLYKIQo002VGT1dS7tl4ql7B6skhEMSHMO8sGmQkoAvAcLjuspuRPFa0N+URIaqxD62PAsOZGdDaSIIp65/lizg5hMRuop6GZ90AN0HRZlBL1I+wosLZDQeWCWuCcoTO8iN5wRkZEaad3DtSNstPOMFwQOTY+hGPtPM+q87AvMNUC+nKJbM/4P6BraVfRoXAnyyidzCQTKRjfIydP/eRP/xQuk1/4whc219eQT/li1AaoxprddpSptJjnl24XKuWp2V1U6MYt/Mtf/iqNgysI6mVdg4CZMwgeAyjjh0O4hVTf4BdhpETyI9WLgQVoZ5kykfQZl129DUa6TwUnaobV8rXoeHRy9+TE7IQrKGlqzrx11mW2ZTdL3XLf63Bl9VXUTQi4OC17ETUt/We++rndh6YWbi7rqMbFJ6FgAI7ANEJh+GLUejgH+CDAlMss4xxRkax84tcNw0ydPCvtIf7WYPBg1Uld06kB45b9wZFgJLx79wF8Facm99wze3pxee3GGzeos+qyelZWky63LjrhzdRK5lbPZnUnUoRVWt+eu/Xwyfvvue/BL/3xf3rPsTHUZYw3Dn8UeqIziIBsvFm0dEJxmCyhfoK2NXQtPwTshY1Qx+zk5zY+ZwwBGrkdCQqRRpn67E5s1XxnVan0WpS3FxCT26DH9r7e1TN4TCafTucs5nARKTM1hCfgRYfJjuh/aVG9BUUhhwAeli8YNURqGE3AtE4KEiPVZTDi4KeJOdheraWDoQhZTRpNIdXX5+YmQIlGOCNS0od1+46O93UZk72CGytcElMOOEKABVPJCuDjBfhgtYRfAndpmxBmOZKuc1obEzkDeYbUyojJSGxvsjq27pcr6od8yc55oYQyYFvbnfPbZ/6O/2rv3Hn1dhe0E6xVJmyr51v3SK9kFmV+RM6V2ZJbmHZxKYV1kDNyLNG9CKagF/HPgh2FG0Vk0h7sNXTk8sVbhwhRMa+QADnkQpu3sry8+dZbpKN77N2PMluZfAZkgtMnuAr6B+WrViXXKAPicDJnOjLXsPiddp/N7pBKDFRMaRFP38RuSkiFuIegn4JnljA1QSnACJ2nBAPyMVIseBgVtNXVq/dqFitOuBY8RgQLw1eLxZqULuLxh/0V8xxjzogQJwX7IJWF4dOMfUnZJlgdw4+dfBc2l4fi0BQIZ1lAvegq1lY0meAm8td4vH7ca8PRKIPksDfoFTnoGVAkJ5Kw0Sde4QvAdrgGqG0wf7o8LrcXR6MOfru4DzvIf+pc30xO7dpLTiunffLapYuxoVD00P6Fq29Qnc8fCdXLeVIKZcspi9mA0p2iUQ30otgp4TH5HuK7qPTWhoZKpgY0teJlPiBsmoSrXZLOEzPQxAGs26KoL3oKUVvzgUYLSs+g1R4eHjXYKX7RaWZLODc6KdfTamLZHRodSbQqZAjq1cpY9LE62/GrI3GNywtpJ/ye0SZWmzN+n2dpbRWKu2vvLPgXmx/aQq8/5g/Hbr9xZnp2FuF7bHR6fM/M2vz1Pbv2L85fIYMlUwmuqZbqG0wxQeFd3cMPPry8tgG0YOaHfUmn0ixmGGoC06anpvCpJiED7nWEYel6VafFJLERzBlzhQpDCJRssFIIW7J+0TyTnRRFMLMqetTmzESMPZm/ACFRRfSQR7EBGyhUdWthESha29zAal6rt9xhSaUSjvnqpUqjQQGMmtks8hnPQd35ZMCY0YaAcSwLh42+QB5FVwRXAeVQSmDhU+Ey0HKSc4wAMdHXWZ12stvuObAXZxWYNsoNgV1K+arbZ8sWslhhWWegGvAM645N8ICI9YJ6MbLcvdElbdO6xLG6X76dr96+KM9IQ9sbV+mVrGhFs7lt+xPUAIp6hA+RGCcAi28UrM03oSykZI8RDw9Ja8ME4dfKtri8hHJVdO68nGcbPb0famaEIUskN984/9by6sqDjz5y+thJn96WSadjoUiZwFEGrduDd2YEtLeLFzsGYLQTrEU9SQJanYKE/ANpfAt2XHpLl1iyMG+NQheLodNmhuXt1fsxX+zkkfteu/Dm0NBQMV1fTK6S2KVX12WzVeocIN+5I3q01sVM00bpE0zzoAQH6icRpwRVsxARfzH3WkRmYOXC8MHXFqiAmIKXoFSRnqQmBitTLBuzwLQHolgGeoVKwxfUxTdT9z90PBobjY3EWm39G2+emZreFxsau2G40c62s9YcKb6xTWE8ouCmIxA4e/by1HjnytuXZgL+f/df/+LnP/bkj/+DX4m//RWTtW3smdvtMqSMIrt0hjEBoNQGbAFKvJkJ44QY/u7etElmMLWDnUtwGXIjymc87ngMtBwBEWFGAgMBtySwJagLAsbnkAjP3h849QafzughgqSQqxazFYFo9EYthCMILR6IDt5Bq6JMYR0BHqK3gTugLmBdaAHLEsIp8YasmcHG5gYeoPec8uG3QtA/qBKVpNPlohwhPj66k/dNe4OGanPD5iGejdIcsLCIEVY+F3ZOoIr1oT6Yha4+j50mDctZgXm5VfbaxvH2EMiahImToVKnFD8iCk9mkYFg4dJFnqLDitjLefX4dlta+1sN/y3/SCNbPYCgaAtX0VTtGfVyYTxpT3qrCb2Mk5pawRDcwXDBKvKnDmTWpaYbGETpwcQhizEVy4REGfEOzAPMaFtlsOIRmETQEUsGGwPEiiBvzEASH2YmWgDRzGzu42OCjhJtDJa0zWSS0dC39bUKNnWdnZJ1LuK9vCxk64gVfReyCeFHmCSRQUmpgSBC9uc+hASDJdmylOCLERSDqQv3TZx6PG6cQRDgwI1ohh0+b79W94c84xMj5NDA/4KQEj6PJYxvJPNBfCoQQ0Y7VhrDL6kVKaqDS64VJ+E2NjgJPCKPhx7XQDCr+AriiNHpEjjSEGfvdq9crUsWGAoJ1xrGfAEQZ7Jx2gwHwk6X/cC+fcZQuLOxTp4dMlSXqFFQKJhaQB0xVT3XIGh2+UB48ArNFhGPHTQ4Vy5emJmepHavJequXL8QCTgDXktyY2Xp1s1yLhkkCUAs1Kxa1lfXIDhkerLaHeS5QBVtxpdKjKZ5CbWykBe6jbcVMhky4vX5axJ3TZUp0XvCGnVYd7AvzBDBsC5/0LNnr6fRKaXza/E3b8xdw+I1Fo3CzzJcoNeI24503230HC4fOaWooOdyudF+YwWkCAS6RwI9Nf0qqyWbzZEAci2RYI7wLk+lC6HYzGo8MzW7J5VLzV26iUkN+Bkfm8puEmlmdYmPtbMkSanc3Xozm85M7987FA7VmvjylAxDkdjQSCZXWItvoHC2O/B0G4tFR1cWV27dXoZSNmplkTaJUxXpDVgVgwiAiR7ehrhtp+6KS0Qos9nt8ZTNVZRiOF6VCu1AAN2zBV0oRBS1M+PGwqCGcZaKhD5fX18cnRjFm/19jz6+fOs2abnIkgH21zbeIEtU0WDGXFawWv7AsKAj2E/+EX9p+RNdlx5jgaTsIJiOJYUnosPlQkO779BBKNm3v/08H4AhAi99fPXxQUR30qjjBswylYXKAXPFylUrVVvNslppUBbz9qbO0DFhvNRoiMFYO9YOtPu157UzHAs9U9IzH8JJVoL8FD0oi5oTyEYo2FER4e4m7xKvGmEj6BJRDhTttP71578ADPDFDDV+fawsPauehWPh40lVmLx27RoFCgnUIbeoh2tGC86JGqppDBr4RjIR1LVA3IbpYvgwAeCyDrEXlSaaNIoFwD62OigLRGkHCaBOKKI5q4i4WmVFTt9Ov9J5ndSGL735+s/+/M/9+Cd+5vf+828VN2rk0tOZug19E9RR2hiYI12DHRfD8vDMyO2VtdFdfrw8mR+oCLKv1abDVRPuivVDbAKlzkq5GoUORS5vSfykFEHD7kuhT7hP3FUcxpGJGAB1dW6JoJtqo5dMx1H/GMyOAwdOZjPN3/svf8DC3H14XzfXXp1fopI34R7xtdqUp7O2mpiePfLv/+Pvf+b3PnvupVcq3dZSof/SuXlS8ewN6kMej7BtfDOmBxRc6F4ZXyBOEQrtH9kLTtcI8V3EQpSpAipgb0VYOBIPF5YJSF4i0vBeIfX0kGtgbBgxFjCvKNa1mDfR6WP3dekGHp3BC1+tawyySWofN6QJHbWL6qS9E5KrKJrIohIiQOeEmWNdw9Co2i2cwzRj4vNRhqFy+Mx/+6Pf/J3f+/ZLL5PSbnRyii8z2XpkqjQxv1TKOv3AfoMVd5q0192s1Fput7Ce8gV3b3wpn4kHHkkZuaQA/e7rO8eAKZCrBgGvYyF0bHJSnmdjLwOpber89o+7/t1u5K5Tf+Mh38+mNS4jvXOj6oa6tvVm7RJvlyvSR+kJ/4gRi3/5LIgnh/JLbLomPMRliEn7KNdQrilOmDEAk5D6RGxeJM9H9pTZxTiMz1KjDqMvCa38Pi/kgQazmSLsP6kVQIgELQwNTTF6BHcTdwReg5Djpoxd32L0wPvDI5DVoVjPQ7BdngAkUJJHkk+RgP1KpV8tSGwaGdT0zQoGIF4jAw3FF/UcSITK58hVrCBOo9n2hsOwBEhpKv1wWxhnRHPqZ5EfwGzBsEQCqSoRqkTLMuISE0QVNtJqkT4TZkLq2MD2i/RM3JmEDFH3m0IIuBkr72k9SuZgwB/gu7ptJLmyxU69Qtl4EKenaqniHvSWlpfpAB+ODIqAy2twPJS6H2TCQvbvNQmQ7fRpw7q4ONdtVTMbq/t3T/Z79XKugDSxuLC6ePMmdNdnH6LFRqUEnUZMRiNKWVP4kpu3FvMFkjl76BZNCUpiGvstWe69Hq7EG6SDQC0PdlQ1GCHDrG7cjyEQGLuKVC3YSNSkqCOeS12fzxP0+3Az27N39+6RoRsXz/TrpaDHgdwn+mKCf8lhglNVpRrZNQstn5u7zjdvJtP4XuHvPn/r9oMjI7hPm9s9snUxZ2PTvvCpe9vra6lUeWbXrvj6kl7fwkx3C6eqXgtnK8oXVhvkP7J2UPVVKu35WxuJOFV0JJl2nrh+M0VGAaSrV64lNpIkuA+FhwhADKH6TqchvChwEc0lTxWZKbFfwWQM+oKvCTXrYx+3IGDBH+AQh40DA4DTQj6uMqp7zBVoUdrNZmGQJwBnYmJsbXNzctcMg/nBD//QwsICBPjWrVtri8uJRB5A8JCehHhrpQVlvUComGX2igYLrtBILyptzqPAZYLwm6eDLUOXoGtDz9zDhtol6Iq4R5KldL0B73ve856XXv121BOWNFi2Tj5Z9gTseDwJ4AEowuQLBVc/xdqAeM15WdLbG3fRBwEqWdWysNm0n1rf+MklOsy2/ZDcxjEPcQ8bl3a+gm8k7yTMNXDEE0TWscGjQqWE/BGIZDGR6IvHyW2aS6bJMUnxiV69uSke9YT1q6GADGBaBLQqVeW+Z07FE8V8fmx6LzO7fu2WM+KFH2FGmB32aBrJdSbZV0wSV6YzVkVj0WhIh0H4uORw86CDaUhSgoguuitWWMw/3Z4r6jQ5pBRgOp574j1PffYP/394HfczLZff26g2GsU2rpzon5F3cXTDFavSa9ealak9w8lUQtI6idYL0y9abokjYPFiKVpdjic3CQAX7SesAGi830JJJnAF9fEHLS4PdeBN07vGLE59prSOZZqyvhup9OHDQ6uo4lvmj370J+PxxIUzb50+ds/+8f1//kf/4+UvvdyJ6q0uHRnjH3ny/RZH6NP/1//94L3vcftvN43Wb5+5UogZnSvL9v1W955ReBhx+cQ0C1cJuwO1hRWT4RCSu43i70CCmk91WXbCQ4Gvtd90nmeZb5JesH675Ms362xBe9dQRsvNBItRBB2AEn/1OkdfVNB4sXigxJ1ys5SvkWlElMEkzaWAKH4L5FtBPwJJ4C16MAk9kx4BTnRDkEinj6KO5MJom8DtuFL/4R/+t1/5lX/+k3/v57/81a/dWly6cXOBVeD2eg34gu4/ZNp7YKRYjjOsIGcP71VcJ/5VwLGAsgJuPkc+Ei8c9Jbi8sHpre/XXvzdQyBXlZoAFQ9/cDLYFPlGxkaeFagS3C45SUTVJH8cCKyzqWe1nTaGW98mSi61xjjLESSHZxgLbZEK6RVlquqnYiFEVFV/LCThFAndE/XY1h9ODUI4uQGpAQlfRFgDgfVd2L0min6AHuJjbVZ6rRp4DXONx25w2Y1ujxTDwKN+YnRo2u+JkiKpUaUWDE2TwdFQxYBuNPh8UZwcKCoCZjdZnOBNDEhIjRR139hI5bKYCcFc+PVgDBYfWrAt8UVgAYJWsRKAcbAPIawhBKO4RulBroMCZj9Uv9zfoZJQi/qdVXJ7Y5uFI+/piEzYu2fPPceOox5p1pqRUAiheFhyc3hRQYLaA15PMrEJ7iZ8As2kiwTh4F8Y2laTRBlSwMBsIgE1Cmy4YPA1q7FKhnS3hzeiCc8XiFzEp7hMA+ie6UW2kKf4JSiVNgr5Eh+CHzfm7NnDxybGx/kWeH/CacCn5DolNxtuRKgEcJomECifSyHO404lIYZmo89lK2U23VbD7NRoNr1eLiTdAVdqfXFt6ZbPZR8bjmzEV0EC2C+reIDjZk55GbOFlCNLS7dJAYbgQhFsYIkUFvhrsg9HQvffd8put5QrOYoCULKFuQxFQg5s7fgK6alRSGQ9rNPg1KlTdDCXyxJe8fCD9y/eXqDkXy6TDAU84slsNRHQKbZ6Ikak5rFDnC5M5tMPPJhcXuY8+vdssYSnSBFUqzcHQ7GV9RQJIq0OP47D0ZHJ8On78Ya3zOzBclRt9IdHZzz+WHojb7S4PcEhTODnL89RoHT80feeOPkAbNaVS5cRQUhgCU2anZ3FpMcGOfzABz5w3333Xb5y6cb1q8cOH0I4mp2exkkbgZ8Vhp8VOgYUiVBXCZLxOlG/UyGYT0PKBLrAxaBvsdPXmuzJLwkPwxoj9BZPbBQn0H6/30u+HuyXDCnLi9A2TfBF5QwlZUIh5CxPAIONAzG4U3pN5XOmKZYSqABLNcXt+YMAw/fgzI+eFEt8OBLErYxYdwgYoh0Hz3zpi//tM38wNTUBH0gBGT7CFQDnoqmQGsYQRWYHZpByhXAhnKERKD4neSl7zvAK3s5P7eYtZKH+oT9c0iirdoN2lTN3/6QdNQi8QXhZBF8CCUEXfAhoCmRC29I8xAfSwwRDIREqJQ8ClSrMdJyCXXoeaWLKhUB16audgD0EKJvD0Olvrqy5SUXX6fmd7pe+9cKbb7yNHxYIlnRs1M2Eba2V8PASnRkKlaGR2Gp8FWPi5O4Z+LmRyXGdDflMhyEBRZCw2SYDtevgVfk0uoha2uFxVrO1YjpPr95648yxwyd/97d+//Tx+4Z3zeLzPDk2DX5En4W8igYYJpgHw1G/5s4dCgVRN3h9AbcngAvi8PB4dGjs2InTwfDQAw8+8tCDD7k8+kFBByOLDdaN0wZUDOcvtzGbLzvczj17Z50ux8bG2ujE0P/2Ux+emIhCyIE3NHDUYXz22edwIkRLj12DrMi//CufHt4XreQHXp+xkNNdOH/p1MkHZ6b3/aff+M1kptzoGqioHc8U01mRYRgOVCaoZxh/ip0y4lvSkYhGovkQHM41+CrRWsoepkVmEpqrLil4EP4JTSQSDreyQlmteJDjiBkdJ3aL7Nhip6cUCxwXMABGxNmm3USeddutIcol4IGVS5cWbi5igvT7AuBqhBPEoSqlecHmNMsLaVMABioP9QIx4MwvxB4HHT4BEGUJwGf91Re+/vGPf/zFb7/0nsef+Kf/7P/6oY9+wh8aOnjkhP7ksO5jPxr8+596sjWYs/uS3cGG5P8B/oQKM55OscQIF0QWQ8kVAr3T+A/WJ/eoQRDA1hS5IuRyExyBXFJ7BKftY+6RUaOrd84ohbYMK4uNpaeirKCjytlazrExwPRACaYy2LJpLYi9Flu1tCkNbm13rqrZUvdzWUaK+TBIyR9ZltKunNI06ihcWdWyxiS9Kymh+vho8MU6YwsRQTw1NE6BTqJRYG1SIxUCSfZg1MuocDHDwMlKqD3ZZ9oUJ5fMUKVSBbLEu1izLBf8UKBqqKZEPBD3DuwNpFDJgZjIfgVPM6B8XqVE4m+WB+oL1le1go5JL86ufQPaxEImU80nyHEBaAIxoi4bSCFeydYh0oYYhQUCBX8ICkHOADs6PG4+WUR4YKXZYaFiz6M/NAMlxxwIoAj6UfKT1x9GMUm2Ywgz4cLVemNscrfB6j535abbF3343Y+TzJ1IGxzMEmvLaHMhqMMjMTJYYXkKhSLEuR47doxSiNbR0YW3Xt+Mr2PSZgpwL2EIsKe5qJsEksNPG+yGIsvlM1k907sPYgLPF4mEJhbWigPayFAYrF3Ip3CeWltbIbXWxEjsxpXLeOy2azXejuqNj1W9xgsNRIf3sXjKoBJntBl8RhIj7kZiHRSAexGkQilFLVeuXo1vJsfHiJz2o9a3enz7Dx9BqCUECO9uyiaSpXJ2Zheerx4Gudt64+WXmuUiwdBhvw++BEsy+AUnibHRYZyhV9eWkbA5jxJyZGKiUKlCj8nzPHX0SDGVuXj5JhXWx6b3Z/LlXfv34wB59fqVdqOK2sFpAbX2V2/fDPg8kViMXJv+QASTIY4y6XRy9+7djP36Rnx2zx4Eo7X1jWh0iKQcpJB+7dU3cvkSxsFTp+69tbLynTNvrqSSCGsEQcHgkeYFzSeCODfghxkOwx1iYmiCdkl1hOZc3HgbxI6InkAjUQjNEACSR5brNWKX3X4f+GR4ZATLwsF9+1/42jfLORTkFVoQiFXRtwAblBgwY+OM+ldRLBLWSxyLEfLMUiYmXtQaBNJ0OxAblOoJ8pTVOk6/A90P3LZa8ARVShIooAJTiyhdcClFyYmBjhAveFhlBZEXifYB4wWuC2ADQQR0iU9Qbxf5lZ5oJ2kN2OAqG/3cwgvqqoYNeIoDrfPazfxkQFjmLB/WpvopSkWWKv2kBWiwIslC1NnAVVzhZtpntWKbgUni07gHb0huIM/JsVP3UOdnYXkpVy0YrZYTp07iNRgNDp156wwaHcSaQakBqvaGPPA+6F2QjFfWV0DrKKYa3da+/ftnZmezudIrL7xgcbjb6bygZbMU4ils5ux+O2kQXVFYZHFBsIW9zUbF7PeaPd5//mu/Rj2pP/mjP168dAUWCiMTSw2/S3/I19Y3PTE/Aa5LGyvRiSFYEQLp3U7MmT2SDWD/InsdKwVSge9LPlNcX9n8/d/9bOUWnmUqatmH5b7VyfWHD/hhZ8fGY4+975FrN8/vPTBdb1UPHj5gsrief/41GOCx0b21CrSz9vhjj/zRH37myXc9PRYdIQ/PP/un/2hjMesbNxernVP3PfHYuz/msAWzifSgW1tbutxafnMyu/DR+/yTQ35dO++yEPpDvhH4BiEpjLbQFtAIsIVMLL90oBHoiaaNxdtYNkVfEJrBBmiqOFDCHulWdM1Bv9rW+YdNo3tdrpi+byFF9QAmDsgiYYNeT8ivQ68P9wbDBvO02bxHpx95+6tvfu1zzy3eXiKQF86V1NtoPdFYwxcqGIMiSawIcMJrAUIsXPwCZgEQ2Ht8Kdj3DRaPN4IBULJ52F3H733wEz/2E75AiGxoJo9Pd+TYMBENRJnh+QaWhLxANJVvl1BC+STlfsUcCNjKZzMUiszJCPw/tL0HoGTpVd9ZOef46uX8+vXrHKcn55E0SqMIkiwhMpjF2AJjbNYYlgUTJIskBEigQFBOk3Po6e7pHF+/nEPlnPP+zq3u1gCSF+zdO2+qq27duvH7Tvyf//mnC2fTGerMrs6UYAvlvdwZfqEkXNhKvu7cU3arTAnFjLkVXfinO5YZ/uYDdo6iHKKjRDvH5b1ympwsW8sRZGHlzS1lynaOjZ5ipSh4DFutkQgXWR8IDMBe4MiWS1Rr1s1tE0F/kJI4edwW1C17QgTQN0dAy9T80dVAIMNwOvEIJcsFfSDBXdZQGMMbLk2yxbiJRFM0TQ05W1L5LQiwzIx4tyuAiAKhg0+LZnU6vQhIHFzhVWxrFNVO2lHrdsGZHCA0lknGMBdLtSK6B12FfuLxovoxh2C8QaZIagp2KKWbGyEXypvMuDjFCkWM3Pvgwd2xazMloTSixQdwghqxdHBWFpOQPTHBqQH1+y34YbIf4FtaI6gfSEGCsGwMjAO2QtMsLMwxbzOJGHc4FPJen57t6ekjrESGGMHNx6ldk10uF8E3WLpAOyqWLP4+oZBmMhHpIitM+yTai8MW4g+QLtN4HNpClHYSONNCvKxpW4IBlc3sbFdpXMgREeOEmqG7wjOlYK9SLtptNsYhDw+rk1grSl4w0dBsWUzEdiowNleKyUSTVoB2G9RdFDWKUqH/AVyJwKPk5jvstCtAagLpCnV3cS0NdlDIuB3U5dXhx6LzEn0Mgj1d4VYtsrVO9JNzsDvcySKVsvm7dk6p7Pb5P/8zSP6k6YvDrbc4W6W6JwQB+5Sqd8CltpqW41QvBAL+2cUl6+YW9DijIxP0hIlsrly/fH58pL9QbR8Y23XpytVUsZYqx97+9rfNXr04OjrOBCbssXNyF8YEPiiwec4Z9mwgSxhmY6ND165dt1mNvRDvet2r0e10plZtJXQWMvTCcgL+Ga+BEAcVt/R7x0ilVQ4mJdaeUtoIZwkGHDFNciX0fRKNIuqHOIfF7LDZ0LcYFuRvXTaBFIj1qSzcQ0QMr3wiJMMI4YesUbQhegv0AMlfgWoqm+FKUqPMXCFnSWqgTMZUimrQbuoW/eyYQkANmHRiiVksCC0hoGCSMT/pTwCjlkShxWoiCCQilhJiDUUdnIDoWhYOxJl0XjkrVCnnw0feo0X4yNLZjJX/ZOErfsuWrOeVzTrnLA6GeBHyO/Q7D4INEAxsLXtQJAZbimjprGmo8uk8+oCF1I+E4wjLqahHqK8tLusiYaJE2DzULcxcvYYBATFknjxiodK2mRF1RhtUnRQIwMRMa0rJ3BOlqxNbbjYyxXwklfjjP/3z5597sZgj0FX6/d/+HQjE07m00W3CSw4OBmKpZKivl5EZnd9SOVV1otmJ9F9+5s/JK21uboaGh8r5HOVk9SLdlFRtQ87hd0LL2j86YLRbOd1F6lBVarvZjbnAw9zciJ09f/UXfuGnXjv+OpQvdM06dvft5FZefuF4KpGDUBNGaJ6JuU+zvZj2jpiw7195/TW3V7psjQVHBV+iaYoh7hnKZsDOaFBaICre98H3v/HymaWF5cP79/3oxz72h5/8ZCYpiM9rV6657cNvffixV65ctho0H3j/e771p+cG+jk1mnZ2Yg4NhoTgCXhKvCpPAJdJ+SCPTHIeItu54Xwtz4R1ylbyBY9U3GKGl5JAlnyCmpSuajDotnmN9XYKSS7hZ5BwdZh5bHTvpdZDY3K2qhYtLNsqczldmp2ey2WySEtkb+dYMqos6EcGJoMCqSGFuHJSMpCkXFiK15WIbrNJZg9Xkm/oTbc+MDTiDvZAOXD89Te2orl3vvu9ZotXFwyqdk1Rx8Ys5aHXtUhW9C/qUa4AgSzaCV2p3AMxQ5TrU4ab8iKHVZbOaJSboMwKuROd6aGcWeejskbZHzsU1SjT+tZXN7bnnx+0sH9ll//oO9Z0Jgtr2U9nUR4Pn7FblYckv5BpycIGgqwQD5hpLA6x7EEuUbHimwL55gFjkZOULBZh3kbAtxnN2OLFIqmcFjlQZjn7EIAPFguPTiQe0GBQIhI0I/HFhVFPL4X0deKBLlCOxOmoziHsRj0dh6JWiGihWYhg24xXnBJ0AFR2zqCXpkOJWAQUhuQbqnW0GrnYttook7TMoMTVI2FB/I1AH9pKZWmCuUbKNaq6OuF8oOsYYeR9uFLUntlqhmi6zDOv1Yqgg8lfVBsgoagsICjIfsRrl3p/KQ0SXnKtzuZGUzuXE2k0GT631WCkjUqukoTXJ1cofOMb3wDejJQrlrKh7kAeFswcjQUtNNghfcXlb6yv49xcvHA5SF1LZBvhIuUNIuyaJa4HDkijmYunjQ5y1uGyC317sewNr5ExnJ+ddrj84xOT29fD+GHBntAqhD6xSCDU7bJZ19c39x86dPbE61H4csvUDxO71uMdMn2w/UUtiNUozdyQodwEmjPSOY8QmUQ0a0Vqo2kkDr6kt6cLjQKnDdEkN7YGfI7oeBohwHqv1dPThgLASjmn01ixjbGdB0cHKbdY3ViG1oT+NplyLeDvKtRqz7z44p6dkzafH3xyPJUC02UqVS2erny5pfJ3qyJp+HxKtXa3P5CKbga9zv2PvWPuhZfPnj090NfLIUPd/Ztbkcmde+eX1u0uf3Zuxe4ww7vJMzWarUCisCr8wcDS6kq+UJjcNYUqxc3Fpdi9Z4qWgmiy8+fPQe5x+x23qS3G81cuUUbFFLG56UZb2djIYg/g/dAqA50EMR5PjWfgsnuAu+GeMRc6qqszKZijFIh3AnHwl6FgoE7L5JLPRZ4DFYsf0dmY9Qw9dC7BF0Zg57eMNIRyJ2dBP1TyCyDx0SWiUCFNw/cCwo8KB8pUE3+RUQDfBHqV7DS8tpCVYsiCfeLqeEMICqUp0pQ5hiZT4mV4vIqilHPm2d2c5jKdubrOwrnduiJOTIYDYWPFdLh1nrzpbMMrv+JjZ7m1Q4QM6/nIb/mqsxkfO+95vbUl7xEubIA8ICuEGBLuGVwXhIJQ1sDK7gWlXysVrLDMuPxAk+KRKLK5S2/5sY//xOf/4rPJ+RUVDHcIokJNb9cX05WV9ipNZFPhNDfDGfBvhDc1FvNP/uzP9fb2wVjuwoCuVQKjA7C7V9I5FFg0EnMEXP/p//y13sGBX/61X1m5NKPzmzUOJx1cBsfGlubnsNgqhTymP9xbHq9zY36jZdBGsymsffh2XH73L/3bT/zln3/22tUFt8+ZEbYmCwZ9NJ7cs38Xd8Af8gKk2HNoajO8ef7MhcFQV4JIDm3ICi3vKOgRPRRgu8fH/QE7TdQYmfPzi/FEzmhwDPfvmRgbVbcthULpxMmXXnrphXe89T0jfWN/96UvryzNewLgUvzw5SSubB35Pw5evXghsrE+d/38z/zY2z/w2KPN839vtxIFgQycYisJsiKUifPh/CjaB5tMBLuoA7678WzkX9FRsqBQ5AvFppLPPGe2Br0BFAbOPm+32dPtV1kIUxJC4ysCjqT5yNoQW6bqyapqm+HA0ugdRKuXF1cX5hYxLBhdnYEnvWokwQguAIGPoSmjkcgMp8U2VBcRfeLWMSSoCWW2sDVEb3qTxekxUpcYjid9ocH9vWMr69Fvf+/ZffsP6Pr7VKFuQ6UWJZpCcpYAEhMFgJ0yR7kcAHBih4oClsHbmXdyYf9skY1ky5vjtXNzOjdGGcJYsd9Xt8rdZHpx8vwAU5QXiTIoe/ln+1Ymg5gbN8+AnfMz/pgGHf3a+SifOKTMDWUT5X3HemBOieeqPBn5oagteRUrCRrQQg2BoVfD3CShgxpEUYRnwI8QlIYMiBiF1GsDZcCvhUSpXSk1hZmRThoGZh7MSjwMJEadgCeykiQj+NuWEAGZAY6UymQoCQsJFoaxREoAH5eiSrBPEO/A6BbwB7uCPpuFpKakh7GGM/VcBvAd7JUmsEXNRCwfCSdJ1eVKEg7iTkDAQPOqsraOoyZdFhQvhUQBEkTqdev1MmFCqP/LEl72en35JIU0tUr29VgsiUYsgVnEQ1ECdGgseCPEldcbBo/dRdQmk8xRpsxYB1qMXkbxuxxuXCIUA2PNajYCaKaaIptShSSwRPaOp6cloY5ViC9FGjuRjNkBSkn+o0aPF1ppEpwhko6fvRVLG2D60LQ8BhNINAqciHebRkYs0/NoHXNXKNA7QF+EuHD8YU8a+vqHiV1vb26p7E44ivOlMjaI2In8L09UEKHiPsmchNSwArzcK0ku69r6cqWU5yEY9Q6gyxgLVFkDj3I6HcaG8EcCIBt19p06c7qYSfA0yVBmE2Rh2qgu5mtDJ+yAHq8vFvf4e0KknCvEPGp1uAlMTvdWImVc2ygRDalTE0yHXcfEnv2gLa7PL4qZbQ/AIRwMrq0vXIPgbGTHblU23h10L8xNa/p6QKuBa3G7vF179mVPnAj29N12zMgPPf4udb2aiKex2BAG0AlRF0sJVqbASMiMT+wgE8yoHxoZCHT5T75+6h3vfPvC9rbP7+nr67F5ilSLm90OG3afQbcdSSbX4o1qvCdIGbQrVolIcyibMHq3Bc+qKBjgWwTz8BQ67iOsoo16Dpg65aU6rddNaLTIuO9EXTvTSdG+Mj3JWCAPEDrMJzIg3DHSK6RRanUetNAD4VSZNCYUjNj1Qi2ug80xBCvu4ABYh3gqeenSpfWVNaPXU8rlOSn21ob+F3EjPjI4ROkmJB8FdiUoDY6HQmfo3pIO/KSjfXm9dTKK2yq5CcYhJiqOCttz8p1f8YYtZYeK235rPWtYLxuLlJH3t7bs/LDzelPKyyxjGxakCGeJw4UVy2iUElBFiK1vJS3UNxupESrSeghNQroRnmdyNAeOHD5z5swr6+smumaBfvdad+zYsboMkDCjNmk1bhK2hnK9LKa/zQKh9rMvvlRb39S4Xf7REewVvUlTMOO4tiHYue3OO146/vKB6uGf/j9+9jd/67cqeeIK6ofe/95P/Pv/8PGPf/ziyZNOnz+bTBYjuWIip3IBWh6emb4yf3VRazNiJmViub/87Bd/+Vf+w+Lzs84jdjoO7D04vrm94Q44JneO+SzOKm2yrOBQ8za/fnCkq388lM+Vz5+4Rh3/I488lEhGmeZd3f7NjUi5ksXyDgWsNqu3LIzr5Fic6P4PPPa+P/j0H/C4MQaoDjh6x9H/8alNUqm/+lO/9Csf/4WQL+C3BaC2ufvg2Pvf+ZaffMfedw2aDboUEQScYOYWD4/HpQh+brMkIFiBwCcnwL8MYnE9FYNNHqVIfT6KbmZYIpTQXbiqeFh4XZiAmBfjeyaNAWhEIoIUJAnE2CeAb4ASC98T4CpVuUwKEHZOpOX0tRnMJlpiI67FCCTchgfHacFlrQg6EHAcluQfxituV7tKAI+wE+YNVJJSxEzhFjMFou8ckaR2ze1xMpLZj8vTlUpXt8NZ3cQ4ZQGA4QtYCexIUoWMWo5zQxnKNco9UKaGMgolxPQDFtlGGeXyRr5ndCqvcmd4f2usd9bz/a01nY07v5Zf/msWZSe8yCHlHxa58TwAFjmjW7vlI39KtkA2kW/RmviXbKPwguAg0tOG7m00ygPWI/mgtgHBQfEKch6pBi0FGzCFcVMMegtgOFZWqUECBCV6XaalvkyOEnYI/GrAzehaXMwGpSw4mribBO6YqxhliBEq8zCVHBZcdRCIdqZwPlOKRZP5bIEAMu11cX9Rz2YjBKVtdAfwXpwJk8NOhLBaVrlp70zLa4eDqDeBJlQ9XI8chdIkgMqSgaiLA20yeND3DQtqsEqvuNhWRFqxafSUPQiWC75CkVAoOE6XGLq2sR2j2JQ9YMrX6ByUK9D4DvTp9ZkZqNQYwdvheKNS9PmcuUR6dMRuNNoBH8HMh53IHR8fHl/fWkcYgdenYwDXBoOmqgwqSuQgZwWljL+Xvrd909cvh2Nx0GDBnl6govDZHzxyhEKteq4wNrVnKxKJJRM63Ae7lXa6sEQBFVm4cJlkJEFIANDC1kvAv0Z9BXz3xNwBAIOAaRPVB1fIaGUMc2WoacB/NOvBDaMIulxpoHqhNCHlxpZQg4HGBeGMXqeOMxHZKuaSOoOpTXNhLaw9wgtYASXcqHf39zGyUpn81OhOilYBsg0PD1EvfPz48a31DZK7wxOT4Ds3tmKxdPHq1VlfVy8YL7XeBHa7kI5fOnucC3EMjY4ND3LWh++9+4m//ZLP665tbw8MDp+7eOnO+x6kW6tvYmdia40EBuXFqXy2b6APGE44Hl7b2Ni1a9e5i+doCkmW6cKFiwSo4Ud54onvUUy2uL2NkguGApj0JDxp79gzNMDNRlddOHd5ZX5Vnr+F4UovvBSwb3QPViCXpkQmRIkSfhXYPBOWP9HJKgKXDhhGHI46zQolKCzBZ+UJMn1vLNwQVqKJeNyofl4p/QLqoy6IE8l+2B4XAYHDD8jKszE0FDxBoqOERgAfAPsjfteQ4m18aMallP9iI9Boj0UmZmdSE/URuKX4K0rkWE6gc/RbZ8UbjBYmNQqbuCLmZOcslf3cULq8v7Wy80M5zM2FG9IR3LyR+8CZiCwRYcLSOeKt93xkM17ZAK8F2120N4YsTE6MP/we4bWgEklQ6NlCDmqb8R0TH/uJn0xkCidOnVxYWoQfEvIWBATwRuYh5qmRFgvwqPf6CGoSjDu4+wC1QZ/45V9/9dUTf/HJP2wJws5HUQvyvX9kcHNzHTqmXDmfyqcvTV+GOZEb6ujq2rNzipFfLOWCEMYP9oGRlCLxLk1mYdMT7F5b30aQac1WlE1kNfynn/oz2DB++T/+5895PnvutZO+MTdk16ubC6lCLJEJ146Wh7uH17bmEtnNqf3DLoc91NUXjxaGxkYXFxYCoa69B/d87nOfnZ6Zo0VuV4D+IF7o+Pp6R6LhXGR7C2JWi8m4FV0fGx+hWVCpGHn1xVentiYnd++5cunSf/ut3xw+cOD3f/d3/vKP/yKxMrc4e76/262q54Xei7Ipifk3NShZGY/oJAYDulRRvfIwxMuV4dF5FQqvGwuPjEEsbtmN5ybreUbsh6o8k99h6ulWaaChrhK3F5WKjBcHWwfGm/oAeP9IPkK/gOqm4gBKVIQniRVGCyQKSHeGHxYWfwxgZQTeiMHghHBkcamVjApnSm9Q7EVQnBxLVW6QzdveoheZl4xOejPsC40gbU+cOK8bQwE3sceJ6YFrN3Oy0H+B/ATGJecuOpQFFcoFSTrkf7YoU0Q24FfcHsXKlk83VbPy/sZaMWqYRW9SzPyGLW8dkndvXthYTuMfL/IM+F9ZzxtmunySg99QsTenj6zkPX8cXn4l+ubGms57Arb1qrZYqGRS5XQKKhLRzcB0oagF68QOyYiiP6o4jZI2pXYLLh8ehNRkADpXjBRiwGhKbB1aeFIZYgJrir1DJBqABr4m54AQQyyJzoAeAVEhmWA7LTaE7SyXA7UOpAK4B+WI4IZ5okzyzvMmjYfMJYoiLXnhaTPjPzm9/i6nBcBzAUvTYjAWC0gywtpNA1BHNQ+fbFIrm86BrqqUthjJVqtD8L86VSwZYyVSW+4PbVOohSiVuMPQwp44+YbZ4mDQM/IYaKCdrQ4PaDIAVnAgk09C59HrtG6toLPlprYbk5OT6VTG5XRzatj7UzumXjn+Cu/5mvuG5JUyWdzgagX/1mBxDY5N9h7cg1x4+ZVnWWkwRyLxDOw4A8NTMHKshWOjU3tNjqYZjVEtGbW18PoKqGRAWEtLK0hrCneJEuNsdGQsNkcjLx0VxSzmGm0+AGs68L2aloUGpNwXA8BR+ghX+SEy2kDUSBiONIC8QLJub6yHunz4r6l4mAZLQHfUTVrk5kC0AaRBr5CCpnEFCFVBU+jjvYODw+M7MskUuF6a9dDps5duTrl87/BYKl1wAWKpa1z+UK5ch0Br92jPtdkTe8cH4Pmau3YhmM9NHLotPDM7e/LE3qmdJ04c5yDAxzxu39b6Js93+vWTCHIAqzqw8rlqT1+fNkSeL4b6xNWARYT8n8vuoj6KTswM4kgyvZ1IgRkm0cSdpIIJbJW11XTpNO987LHbb7/9/KkLn/vsX5w9eQ6iRzfNv7lvTDKsdSyUmxOK585QVIQU1fOIIRQh6Q0VrzJ3MUGFWlXg4jzQjtbpPFneo+dYzx6QUIgkcisOlwWnhBGMdCNggNHZqUdiYJBg5vfgq9HBcEjxE8KqdJPRa3MMYAQCu5JZydyknE/B2IibQukYp4uQlFJcoYvibDm0bKuYBbzhfFg6QWUuiz1zSixsKbpdGYe8ska2u7nc+qhcu/KVsgG/Uk7lxv47m3G/RNCIKJOFn3BGjGEAB6Sp2CWsd2SnQJxxSK6dAECuUFHlKkYvQSAJf8IyTSjitrvv++3f+X2Hy7WVzcWajf6+PujKkhjozQZPXLw8gWI0Qv29h44czJVqf/+VfxgbnQwODkbnZmMzs/37dnE34P0o5TIu/9ilKxcJQ9HxA9D71O1HRoaGjx049MxTT3/6059aX18jyR/PJO649+73vO+9hJrOnTv37a/8PfcJ7VOT3KLhoYce/tM/+eyBo/t/8zd/89F3PkLBq9vfPTwyODjUQ7nG6sb8/NK114+/mimm94d24pkaLWrwpn39o4zAF1568f77733v+z/wrW9+lRkJB3UwRHftJl4jmTuL0WXwahcW51HkWGaUlgwNDzzw4MO/+Iu/MD4+ePe993/zK99enpvfO3H0rz/32X07RvxO89RIv0WfXfR6VAABAABJREFU08D2Q99PlA0KV3F45eby/Mmtg5PCOxV1zBBjtQigzsITufVoec9KniSVu4hbRBziGruMm+/t7Sa3QfqP8kdyyyUkAaaS3ohtDrs5bLl4XKSSVHpLo5CcnZnZWF3j2REOMuKYkTrh6YByhTda3SaEzXulyk5sQYaZDAk1hJSIbym7V0MVhCMM4ofTJKyqYmpCihLTmAL5PAM06fUPegMDuq5uLiSj0tUEAQwlBzVoIpWVK5Ah11FuymyVlbLmf7YQL+L6GZyUMP2zbRnKnbvz5j0Ickn5LDdaFvnEDebX7IKdiEmj7EpErPKtspkc59bCbpVFVvCGQcYc4V95i6ZnV7xn5HIv2D1vOn/MzRvvdfWqrpCvpxLVVEIFgxXxV7bhl8V6syy9BcnMIbUBJFMtikdKG74wcAYTQ89EuYX0PqLYkoJWHheuD9lEQ0ODNkQ7AO0mpoAva6SfiAwfnhNRbcwlYX9MJ+IUBUCMDMNwoUiFKwka8Y1RjlQuIfiQVQRD8EcxqItluJubAUqL/D6uLZxILa1vkrfBm0NsNnTmhg6gBOz5/A5xLVGaQrQ2MOBU4oRWQuB0Ebawpfi+jCO5Z8C2uDquFbcealiMGBq+mAxknStcBRrLoreo6lmq/lx2y9bKUhdFsgPmZrXc4/eAe6LsCFhTqDdEshZjY3RimOoXJD2UERghqBJvoJ/8UzROC9QNjP1MtgwjY1fI23fgQPf1K6sri9qWes+Bw2x65cql4cl9PX09Kr8vu7pus7scwa5aOgaNIyOHkADAqFKm4HBYdeqmQcv5gOq1YigA1GJSASMz253pYj3U0w86KZ2KJZIZbBi7wylgo2pCCFIIghsxi6oqfQ3Ys7O3J3Yd+qu89DBvVjBnyAxF4wlVw02xBxVHxITz0Ay1tINDozS60GpMhBCJYRNR9A4PFa9djSbjfSyDg6vra1qjte/wnbHMq33Dw7QvczeFaNBh1E6fO+XtHkwDUsWuWl/BFYYaEpGy79BhAsu+YNfooaNr03Ob2+H9u6csGmt4c8nv9THbr1+52pvvE5pSr+vKlSu4rWC8+/q6t7fW4etAVROwIgFB56w4hCGRLEBz0mt49jz95eUlnuDctVkKkI7ddmhpdpGaVAQG7Xaw1HniqGJmOmMAPUXYlzuEWhYgf0vV02vp7Q6MjI1iOG6tbiFaIOuHSJIJieaT0aIMHRQVggZtxxzCOuENytXswKxhbJJ1a5FzoQml1doCpA+QUyy3iuRIqIKzWez81mN3VjN5jEKZbmgtRiHOCOx8ZtDL5H4QvJ2T7JwnWyi+NUQEgqAWcUycUnGwbwgWZTxTNcS5dLozie/CCctMFokiylR+ePOVNfxeeVW+5a2yEZfJDzt74z0/47Vz1TJnxHmgYhG52qYlIxKXB8qMAnZBngLEj0A+4NLAYYGUv1BWmwFXOtauzf7xH34q2D3Apf2XX/nVX1hCiebf9pa3bq9vUEr3wAMPfPkLX6BNIe3EwskYpODba+HJ3btH79o1O78MUqRaKlIrsXbpIjyQajC2ZtPK9ILWbcTOVlOUXG0sXr1uU+sv1Frz12cunr1QAvqu0wW6e0bHJnioIICOHrvjXY899vm/+ovjL7906NCheDyKifahj334t//7//1Xf/mFgb4hf4h8U9Pm0sPh47QGMvHCM089Rexq1+RIb6iL4Or2VjzU2/XKiZeGBob1Fv3Z8+eIgVAPsrC0GQoFGzOrFEAS/3NYfa35BVXLPD83t7B0HYY5hy14qTY7Nrr1X3/9N//h77548sRpRGfTShItv2NysFLNWp0moM06k63Spk05ubCmUZ6RIJjIAYu9qCwtdVV5i38sIquzKLpCRD3qQUbMzbUMFgpd6gT/cT5MTYPHoOm2qepQMuRNVBsQrMC+xzow2As12rPSPZ1kGzzXHnBpOPEri1sUkTbzVY1JeKkQjzxZGXM8VDxc8PkUcdAhTUkP8/iV0aIlbCjpYanmUUEjUmIUU3ZitObrek+fZ3EjG3B07Tx6MJ2HEpDKF6PO1+0qlJOSW8aSK6dgXDDQn4JQEJfN340XtCCzQaYEt0UZyjdeOhfL6w1CKWW1jFV0YOc7iRuwlp+jVbifN37IXWOXMhnEguZyZCKgNMWO5df8SpTnTc3LlwI8ZCJIDldBQcp3RB5ItDJHO0dQwsuKTpVJqWgVrGmOIj9Fw4jNRF8LGm5wbpwM4hhlRwiNEqF8vr6+XqUHGtF+OUu0YxN2JxraN+l5Rxy/KHgJMVEN+CMmo/h/Gm0ynaceslSsrawk3S4Dxn7eLF2A0FuJeI6H7LA4eEL0hqSAoFKsU2rotGugqdBpaJVK5Zk26DEXMuFilhQDvcvwehEcHA2n0UEBMdlddC9tCyAEARMd6vGr7W5vqBvBDjEhkv/yhYuvv3YcnFSrlif2Va5Qwqatpmjx0e4O+WOxOKoiRpc0uw1FXiYDCmaklCaETYxD3Jcmo4Skh95MO8y2Ab8LdyoYgoFxaH19e2MtggrZWNxEvS0nrpKPpK3lyOgw6CdEMyE/EYRGM7wVBw7se+KJxwkV9A51zy0s7Ng9sbS0VirU+30DVoebObDn0IHb/PcT2Xr+2dcG+0PEyStLKxO9gxj/VpM9ux0zOuAodkbC65FMZq/HA8SD2MOVK9P33HlnDecOWFQphUjXmqROkwwvVPfUenFXEnF6n1NgY0MUFqkN8/f17T5CRfPCZpx2amSi8O6YyLCWABgHAqYt40pRXViJXp+lMKOcS1nhuytn3LqWx4mJoyrF1NRFlFvGmhqyPjcpBvqtWfp3nv3SX9PN1z84HImnx0aHSULQWJQGDA63A+3jC/nVGmNmaT7YHWS9D7aqIWd4+oymWrK2KsQHHQZHgv7MperQjl0k+Snx6uoORTe2NNZKLp5ugPpw+3r37p976QnIe69euNTX381VuC2mXCZayrV375k4dfokY2k7vKrW1F1Q61rsFAFrTOat8DblsYH+gcWtDbDz5XyG5wK2Ze7adViliObVSkUS9QSZJZ/S0pNS4YSJfyizQlwhJhSKmFA7eTKK3MHiolxh5AZaJW1v1cT2yxidJojcUMYNwfdh/StWXo0dA0khdEzsFbclnqYmW8J0wCiAlJgJ5AOZltZeTn47NdXv9vjSyQxHx54mwZaMJZFu1LQw70TRkQSROYvhJDlmFpE47EjRxIxVEyhRxQYQ8YClLxhXNgDqIqIQR4SsSscd4nAoRXatKE7ZE5JRcQE4PfQ2OrWzBt2KIJFfcVzi/+xPbgfSBUnDhZDLIDaugDxEmIhfLWfAsSQ61YKVRnQ88AVM65LSTQFYIiEfVoIzE/mUr+erqe5QEOjt2uVpjDx4scd6+q4kr3z5c3+NcgWNjMHsA+UUiTQDLV1Nu3Fh1to2v/D4C77unnvuu9dhtdx51x1Uhf1RLFIuUKlYodGx1m3GybOYbB69ZfX6GmUMuZXwi+eu9g70j/S6kukMZet9vYPAj0+fPkuV0cBQ/4c+9CP9pCdOaa5dv+zxuv7ss3+y9+C+u+66KxnNZ2IFq7E5uKs3X4p4HaHCZu3EiTciMVVXd0vbB4Wlw+MabFYcT732jMFm/PaTT+7ZOYb/AE5zctf+C1euvnZ6urevH7vqwL796Xi20cidfePvcCe3Nrep6oyvF6E5eOLy90Z7R3/p5//j0089/sXPf+6xt7/9gXvvSUcpA45eu3oR7Njpa4mNs6nbR/T7xt1EHrWNLLFguk9RryCqQwsQAGyEqGMehKR2eRVzR7DHJIUJmiDZcUPr0kK4obcbK+pqqqIyelS9k73OkKupjtK/zUxrb4FktawG0iIAA2nIZmWyG0yuCiFwrV+l6l5dOnPtwrK+bYV6Ml+SRC+szqRvGO0woZXzJdp15/MlQBXUnxLbLtUr9NiGCwUdDhoFEodENk/PuyTFWDaPWuXezlUD9onJ+/fqbcHtTDOB2aTFofCgq0mbiLvOOEPN0+5TnFeGjaI3GUMsN9z7jjaUD53vZAb8wEWcX9StstWbdiPzRJk28qPOzjBXRDkqOvLm7iRgz69Zx2bKpGAbZc7xrxxdNpT9dEygmz/rrJT1yhqUsuhI5SGxIes7oS3QIWJcKAKDiiOZ320TFWECmeJOCLRHlHS7RdM9CrchXSAnrheCeTSy6Hr4gyrE+1BAVhtcou31rTDoE7vLrDdbw+GE3a3NFsEBYQqbKoVaMrmNrIJMn+xA21Q16uh+imFK7ApQSQtZIu0QJGiBtwqYSJ6ziAkEDvXNIscQRyrKQyFSDgQ9NrfvxKW5w2OT4+M7iGmvbkU3E2m9DY5lSwXvI5WktcCeqalsCrbzFNUb6IBMuRqlSIX6WpMB6c9d8FN273XigieTGbwaynSgQ8WZphd4V1cgu51emJuBHEBvhEvBUypWQVRR80DHCIJCZnZKEZxWTX8Fnc6OoHf4A1BNf/u73ybpS0Hz6sbSrr1TmxthMDQImr5+DOveRCpKG+PmCpUUxa4uV72e3VqaoU8o0LUeMJTcHJM1USo7XL6pu++S3k5AKvOl5aUti9G+PLdeLNQ1Jtu+0SPJyNzVCyfNuPXwntSx8kBKYv1AzlOnfTLzkns+deAIFROtRNUbGtijM28sL0bCccoaoACC9KBVBQMMREibV2fT8SThT5Wp7naaNPUqlWR4czA9UwNOns1gcUYT+atPvEjM+eC+3Soq+Ao1v6+LDHr/0LAwKrZa6VwWxubtcBhUJ+pqZHwKmU4+H5UUSUQtlhoJBcC9eObpTL5uRI4EYvGUd7C295G3ov6BjsMMMLO4SEhv5J57G6+9fu3USWin67oGmo9q3v6e7ngkYrOaV9ZXreT84cdzOyGV7e/qWYvFIHYgCd1YCw+EQifPnaMdDQ+HwTa+Z/fcyhrVR2azZJVqJSCBDaI3QHDREKhKiT6jRsiXEKZn9omqwYMSx1UiVwxx0SsABWmACsGRUGhZ2hY6aeHmSUSYQNBNNHLHO5T4GzMMa45hTcIYYxzXgzkmRjVzTNOCWhxyKSFHa1DUAeKaR4b4hhuE7UVfyh9zVFSwsgatjQ3OZBfLXMQFnztoLHFMJYGtzHdRmXwlIkTZrLOlImc661kh+1SEAq8cRLZUfnTrV9//li3R2WzGgjpXtuQNt0RqnBRFLrtlYQNeJWuMsBULAO+os15ib4SvSArLNgwnwImMMspcDBarRg8Zzcbq5mc+/Uczl6/CKcyVkE/lp/NXp9leZ7eszS/jBGht1uh6GEIrgmrPPP5kV0+oVMhzc7/4t1/8wAMPqqwGYqCwM3cHA3DQxrfCENN2O73J9W3mRCEN+LkK7ilXLKHdKBCn3I4+32fOvHHt6iVyjBCjUPC1vrxssplf/NqT973vbY8+8n6s6mee/ZJJXxsc8F89O722lCvkG628arsC4G7+2G230VXlr7/w5SuLiz/+M+8HsE4DxN1T+/t6hgFrGIy2D3zgo4Gu7pWV1Weee/Whex5YvD4bXd+i5UrI26PXm86fuxrZSgFn/aP/8ceHDx2g5HKwfyAW337+xSfcdutAT++hO4+efPVEq2pfT6boRNzXq3c7CBjBUKsa6nbHI2k9TYkZU4r2lXGhyHluL5oUoS3cFTBUSfKcsScRSpPDmq4WUzg+TpW3127rsrWsxCOLFEMgNZQng20qmXtAhG2Npa0y1mnibKUc0RVe2JqfCVdL2koZ0IFZjsowBLAj9MMgbYHQ6AvMjSI+cRWLzWSzp0sV0DuNsioEsK5EOLRJh+RiSV9Ue7SabpUusOeOfRpLoNI2bSbpM6KzeIJ4LqgVKbDjbJQRpCgrLlKWzqvyVgbSzUX0379uYVQyjm/9RpksTINbK37YGxnHnFZnS5kJnZmIChZIiLJLZSdMRLZR5jD/dp6MrMG+F8dY8YBFcaOAlTXyY8Jp4E54CkItyQMTcUMMmW2UEnyN0tKe4Ao6UQ15kNT1SRIBqK2O+Qs1BuA3vUGaDMIWBUlcWrgRqZAr66UYWOrZmfaU3lKIGewOoYjgf9bAqYS5QTKA2jYtZcdIGxraqeHvkGyXDtEoqDnmM6YHcgu1gkmACU19Do0GieiSFyOTiRCcm5nZWt8iMM0D4/aCu2Z8YJHBeUa5DpwSpKvpKl+vlJ0e7+rswv4jt8OYFd5a6xudABgmehpwTb3uCFqVlDEnC9jZTf/5pa2o3WSOEHpJzEgXWkqipRqqCrUPFb0UuRGHT2eNxgr5SotNR8mPFi2CvRmJRGkbypbwXoGZgrcSNDJheEipWHCXgWhBEYPVHuz20NAplwuXs3kH8HyrDtVO5GBgYLAFSV2VVoYOlcXb30M1xQbXbgeR6Xalk6vEc6tNHa4+EaHFmQW/02s3VQjvs3O8LS7aBvZXZzl58nVvV3/I72duQZUCMrMKsCQVp5uwlLvw4KAW7LR7YwyoG3afSat2UoKAoKYkFeiy3emxlfXTaxt2f3+30Y49tLa1XXjyqXg6Q2uhYpbsa8tmtSyvLaHwpM9BRRKNkLLQpnx6bsnmDBx7995+u2NjZWtmbk6Tq/T5/BLEntjTt/foyy+9fvX6StfuI8G9hwrFE3ff1U9smZvTVSXoWJ6/Pm1X11xm9W233bYd3ozFk92D3YlsnEe8Ft4IBIJUiw4Pj1TzZTCAGxvh/QcOj49qWqvLsALUKmW4iuLRmCMc5ko7EWbw8Dlq1YkRENqAz4yQDwMQ6xMPkOHOgBM3UIY3qfpKTWlcSH4Kcish2pOosAfiIlE+2rYdUJpSBSR4KLbkZqAU0d3ABSjrFD4s0YsIKnQ53iE4LyYdEwct32pm00kMzyZ0UTo9VAbwBPEqmluZzKK7kKYyl+W8+J9jykf+VxY5lqJ0lVktG7BaeVFEl5IS7qxkPRt3fnXr/c3t5VedI3b2xnoWOfjNBeujs5It2UbsAVnYp4hE1nSWm5sjoNChrFdOT6YurtqNhV+JDEGIgNzRE96vgqUA2wGU5/TZM3DDcf3iwldbVq+jmAOoJZQ10pHDSBapnM4UQb/3jgw/+ta3vPrqK9dOntx7z52g4Xffe8/VS+d27pzyupzz167RgqmGJMir1mvhrm7f0QP7Kc+lCIF5h1WUzaTLiRiIdiz47oB/4fosHj6hAbJmqkK7oipZum2Q15IKec973k8Lki989gsXntrWBlTNpCCC+aNNCrD6aGwVoPv5Ewu2AdX05YsWg3bX/n3jI5N+fwhQxIFDRz3e4OsnTs/PzPq9AQLRDrur6ir7PYGLZ88BSAIBQDsgsiFc3759ezY2l5fmIa2DUStezGcYqz4Ptmkyv7Hx8NTYnh7LcniePAWdJtwezcJquj/kwb6U+y8PQfQVb5j7WHYwdGI/4rDwJbdeEiECwccYJJmnomDX323s7g+RpWu3yzx84fDA9JR3CHN+RYsjKhWdGrVHI/zPvtRm8eknT7z84sXNtVw6hfUp1FdoYMYwP4UMm3GKHMf7BRUDMIdyg3Qxznkh9vV280Yk43J1JfPVjWjeNzD50z/7iwZn7zOvnYtl661KW6EtVPcNjoxN7CDmsby8TEwGaknGGZfFNeGHcW4yTG8s3x/JrH7zh5sb/Mv+ZbwyKm9t+6YJcmvdD3gjA5krk39ksolpTKxMjG1l7t1YyXvZOa8yl2V7UWA3ftJRwDLrZCWvcG8yK6TppzwHtC9kF1Ax1uiaA7QYewrjvVaVHDsAT2H74amKuiTYgeTGjodoiF4DhDkQJqgevTdAf7mU1++ORlM4c+S5yH4iXMjA0SwBngQQQOHtIu4jJ0DuvwJiCyw7XFOcKOH0SlYqC6gr0MDRiCfBn54zyxeozDBgyuGYtCgfaqeb8RhytNLUUsNAfRFB166uECXwJrDrWlJCRq/Xj0W8GYkCipZ2uRY1NvDQ+OTYzt2zM9Op/Kw30E10CrKEoYHBp59+WlK8ajznstUIZTS0OO10PJrKSdQa24BjUU7MNjKT8w1IxrFGm6oMV49/pqPhPC16LA5/sBuQMMRSAC8JbFIJRwEwpcTRaBTyJrQjrB3pTEKtg6XSCBLZYHA6bRS3tRQ6YMl2U0yFIpPuhyrtuVdO+ntG3AF1vtgK+roLxcrGZnjX7gkAXtHN2dXNqDfQMzHQywzfWFrh2RErIDeJEMPq5VExDaE7wbh29vUUV5Y302k4lXjSeNSkuMHPsR28N2gPZKoUmsLHlWvlYOqz0FlZndOWLTYDcOuaxnF+dmP16lWKkWmoAHRiaKA32NWzvL567OGHbR5XKBg4+8YJspUySvTUe4FtK9hR3U4HIAFVJtGsNMuF9MDQoM5rgJQhFU72WNwqu488fTSZKUXSlpGhXLbYQw+p3t71ze3qide9geADj7zl1DPfKZbq6VyegbyyshLqk54QUCKkCjlEC4WYO0Z3ZrVpbdMwffEKHT549vQ827tr6sL0NEkMGk+df+OM1e012x1EtKrlWipJNVsO6x1jq1pm/jBNZCqhd7l9AjpjbipIZu4kuUxAI3xJdzxUIJKBVzL9mFY8dKwN8PyYquRQbkxXZpx4qpI2YpzKLMR6JR8mgBl0D1FBjFopzu6Uw6UpfgNxLW1oxAwmicbvZCoqC/uUPShyQs5KmSKs5C0flTNF5YvvyyIHFIObb0X53XRYO7uSDTo7kv109nhzDR85Jvq9s57Xzh545VfsStmzvOHa+ZY9M9Q722CCy22RRBhHlz3wVcdu5mz5yHLzK74WmcoLtgzsYzwsoHPpdMbssGKysCWAZBX5PpUK7UuEij3XskW8W4yhYipt9DnJrAsd9+zc/n37PvCBD/zuH/zuJz7xCX7bPzCI/X370SOFVApHma6OjoCV+v61hYRGf9XqhEK1FfC4J7sm1rfDs9enmxQsHNy7Z9+e559/dnVuzuKxcjP1dhqRwFqjm5m5vrLwJz/xYz9x/z1viW9Hnn/+mVqOsK2lmi9B0NcdchbrqfPnT8F/5w6h6lSri4sH9+4f6u/xusAzWDa3YpeuzqVyJUrbDx88Qn5hZXntyIEjxcHMydde97h8W5vhZq1VTsfyibTJaauWC1QJuj1gAlXxWGYjm75y6Yrb7iWUX6qrr8xvxtbK733LlFKOW7wyuzo54AHiIJwtSAhmKU8c05E31ITSpk1aLMAhwzcMCQQLUqQNR2iqUDN7Vb29Xl+fV+MAXEwvEHJtMhSI2yn7wP2hhatZLXrXpWo7tabeVkl37szVV146e/VyGnUAFTepdFggxckm22YgAt6Jf6sMDiMtJHNlaBKhZCAAQdCLgE6JUPbqVsbt6zv44CNb8er3XrpWUa2lYUgipkGpHngNTlqrw1hHqiwuXJdyTBnfBFAUs0CGvWLrMSyU4S3//m8ujFdGmwzvG2++PyX+JXvmninDWsax6CxFj6LMOqYCO1b+UJyifTtaVo6E7Yl6U5S0PBx+qJBZYhqJ+S+WKd2JtTIyCjXYm3NppgEJUY7VQoQiqxnE0BfiBKg0iHg4ohutkhKkBkpFZ1a9we3zQAoa9AZdntAjD7/1+PHXo5ACwhJRqBChJZyLqkskw7jePb2+fDYt40UCJOQn5LpFzgg/WpvemjxC2itASyUSECEjWX9MBfIBwkoADSYSgDlbQjy2mv29Peg20sO5rLiWbbMVDwmXAzlkdznX11ZgctDC/gQ/YjgKze93n3yarHX/6GShWo6kC9SM0kZn56HbMtk8srUX7Q+WzO7u7u1zdQWe/qs/QcLCzAUBMrY4go1MEhoOqVOutg3EsBGsOlgwpdYY6h9gQeQyr0xfy2QSNkBSpRKp2aXlNZC9PHe6A7eKTI8yAWwuXwhzy6l2s8g1QcqJIQTpMY+vI+mw0qsqG0gqWK9rZbXf4w52u86cORWJJTSqwtXpWfgTd+2Y5PlkCmWzzckOecAAUfVi1aJ9qMcqj49NMbjzoOlw/PIUT4fzySRyUwIc3GsMIHoSEpWV+hkUgYp6M/F+aw6cyFYtK9X59Cc26Xbu3Jk6e0ni9pE4F8IEARetpVWVjVxtEA4ziCdPHX91aKCPwDo50QbdJH31nZM7FhZXMltr9FhRNSv79u959bmNDMFbnc0aGBAkn8NvKqqWljd20z6GTL/FMfXw2/qXVy5euaqx1imfPnL7HdVc7I2zZ/btnaLSY3Z+bnLXzmQ+2dvbf+78+XyhPDOzMD40Wi828TnOnLkAeL5/eGiku/fixYv5VJKYAuMgmyBgUcS7grwF4wPHV5LBXANGPwBhBh4vAiXuhGdlfnGvUMAoD7ZSvpfNeGCQZ8FnhLXHkwVDAFccahlVC1UklTPoJjQ5moqMCSqVWDeTjilKVBYYc0dl8nBBmNhcLnbLPkBsM6mwl9g5LdcoBFDC4XImTDWZwzKxmR3Ed76/yCOQI4FdFdx1Z1F094337KSzNdtx3JtaUHb1Axd+xvrOj9/8RjlQ5w513squuFFsyWtHAfO+o5XFz2c3bC4cPGITKDuU3TJfiFxKEEButrhNFUyzknBPUnFAFR2KWm0WomB6JIDKoJcD6VyDG6JsQyGaUn4FBLRcNpeW5uahQEfz/9Iv/dL/+KNP9fT0CBmOWv3cc8+FAv4deyZmr82V4hmLA9o7FVgtbMtINIEE6x4KEKmCE95gMT18z707pnaE15fWl+aCHg/dyUxkNbU6uiy53YEu7wDCaPrq4tpKTKe2Nk1wgWioeAwG7el0GFeSNGu5PPeWR3ZSLhtPZACpLs5dhyZya2ObPqsIElrsnVkPQ1YPB90dR+6AyEiwpICHi7AvK9xQNjs3rTIb+d1f/c977znQ3eNDEzOnIPDJ6YvVUvPAwaOX6udz6RyBsdcvr42+545wZmFoapycl9YiQRVxcrmdygBBR2maZJ3wiowYljKUoEPiKVAvJCq2rbar/P2+wHCXyoraLjI28X7E2+QT/ooQGINkg3jSrla7tWpPu+VRqQIAr15/9crqclpBBapoFpeH/0YyNzwcWIwlU4siYYxTCmpv4PbY7f7eFLMtB57LFhwZmrr9AXxflcbuD43HNZFK015v281Oik3NCA4CD8lMbG5mvlxMYxNIL3bIEW9kWwTjyGARy04WGYT/HyyMYobjrR0p0/rWpx/8RvkJk0NwF4r2ZUALLIIhzt3viFFRizccX1HMyshHXtxQz2wk3q2El5WDK1FrHF82YCjxRnLLQHOonK6gbBi1yqNlvaKqMdvbtKqT3Dj+GRYJho/UG5EDEOiwFqyKFMNQT4dAHBx01hvZGPZMPCmkicCn9G34cWDxMDPBjHADkR4tirAShm45hPDlinhCzTaok6EqjwAmEhLTCHYP0FGA9giZ0tWLGAzRHHQetESpdAJNrFephoeH4fjFuZT6ooqFHeOg0MGIapnllTV6+zhtUgkI12h3n7HGL1LpWGSbUjyGZyAfoBANhYGw8PgCXl8IzUSrAeBh0pZSqx+anFpdWtuKxLBXGN+UOBNfdLmNAIuA8xmsYNBociexMkqKWGhdB/H6wQOHT556jTPx+YNSk9Pb29Pdh7TCe8bBZWoYNEa33eb2eJqVeLUB8T9cAjYu2GSyl2tScg/NGJBJm6eXUtFQ7xg2BulPXSp55MjBfDa6trwU297WNop0QU+G12LROIlPgq5E3QlfCwhIoyFFqVGXF+bnMvkaQppoPIhEMpmEerjZWBW4KDx3xBOmFOfPZCC+RFy1ViT7U4XvjLordaFoUqFadKODfQeO3Obr6f/240/RqBjIKH6q3eteXF1b2gp3+dzZdAICDe6hxRBKxGOwWK6vrsLDQLS/XMy6PAG9z0k5GV3lcrR61BmzdMu5PA2p1L59B0aHR65fvBSPJh7/3hN7Dx7AK8Q17R4crpc8unxEr6ZgeIRSk1CXv8zo5DYWynobJl23BmR3InUxfZU0RiyRXl1ZIx9CS6hMIjk6MLSZSl5fzbp7zTWaFNPOiQbDYkHKOOYG8ziZ1bi8XDtqjjnC80KJMroYDDxmpCHZXvwSSlehpGElNw23HluEsrorV67hw6AGjCYjd1DUili38j9KV+aTSL+mdOLCWJHECfan8PxJcpT2Hjo9jjTihiAecSaoY3Co+ZYYD5oXv1Y2lJOShTfsjVfEhaIGxT3gPd901nc2472yFV/BwPp9rdn5za39dDZW9nZDqIkUVn7ZWckheHNje2VrPrJBZ03nFZddcbNFoPAVbijaV26oJKdE5XPtZIv5IQvniZUvjpZyt+USREWL168FOFKu4T9h2TMN6V1IrTAznvQB8/qDP/ojJHG++93vMncgnpO9t1rApA8fPkTw7anvPv6Whx8hYQGT266pqfNnTlM+GNveotCvrkPzQmQHYV+x1+0eHuoXBkoSvwUMsDKPmN6WAZfr0J49p08cp6oMycj0paTQ4gB/mq2XVsnJHTl2DHf52tkLRijqNvJ3ffD+nbt7n3npa4DwhgckwQSh6UY4ataqp8bGqWbbWF1Z34jCuUtx39zyulpnCfh7fCPBxfmlcqbsNENn5zOqjdFwnM4xRFzgrBu6fQLH32rWw5Dj8zo9bie0bsS/xibGdu89dPncrMqir+jVS/H1z3/12fc/uO/87OyQ20IbXjdHlaGi6CVxEoWCl3tPSk5ihESh6WHeblYINnDTdKr+HT530CWdjeDtpWkc1jMmMege6SPHYMJAAu9hAZJP3xy1xqdS+7LhwhsnZq5cWsmmBdhFSx5mDhBYuFVEM5JEhB5Ewh8C/SKeDnByK11JZdJEwR95y7ve/e73jew+pHL1fe4fvnXy5KU4PdRaLjpJUKnm8gaQsbF4ps7MrGCj5FUQBGlq0MELLFDGh1xcZ4p2htAPe+VEbgyyH7bFP1/P3lnZGcedb5U1Ms3++cJkZTYjK8RLVYYyl85HZTowb5joMr6VWcDXOKayH1YyoeRPHF/ZQCwkZX7LxooCJgeMDpRvsdrFD4akhxJYZoU8VCMl81oh3xEdZ9CaRFLBRwNZFaBRIk1ENUjSclLkgCUYW6kXMnmARNDnroD0O3X6IrRQNN90W3HiVCXBGKu6gk6f3UfiMV1MeV0S+IQZmMOZTSr60aO9CJh6qIBD9uMgGjTwEhAJxcgi9UxSVme0+QM93mAX9U3zS1TU5ZvqUpCcTLu5vbXBfIDLCfgVPhDUvujBRMJCUoHdxpIp2tpwY+wuN+iqI4cOzM5aEBNTUzsJGNIsjCKhWDodGhoNDQyUQVjRmtBsTxWLa2eW09vRDMY4lcqgD8SALeYy9P4ldAxvs4quBtgKDBUkNbzITAgqgSBxBIMK6QeBfNpNIJ3vf+BOinx44pvrGwuLEbIvZqBTDWoA1Fgj8A6irkCNwkStB8kF7ls6c7lTpTTqkporXyigcvntheLK0oLXrrbb0dU1t0GTyhYWrm7CJfnOtz26MDdL4r0GeQqhbWrBQLlIutwchTuMFjX0vuM8brJswvlg1htROAwtBiO3G7eM/LMNNkqPIx7d4gpbBgkhELJu6Em+81it/V0elUV3cO/UgQN7ktkcDtBroEnpXkXMsCr67eEHHkzFIlQlp5MppC8sK1vra0SkgY/Tom7H3n0Z/MdUnOlKl5u+/t6VpW3gYELyTbub3v6dO3ZMX59GrUI2r3V68a1Bb12/8oa5VaTQktIUyEmsNuPMlSugL2mPTmgdC4FUIlDxfKa4OLtgt1B2S7euYjlf2Ds5ZVhbnVu6mtgqN+myTpABCdJSgSeHFpogOaJKgBWKhyrWlaLeGA9i0CudhQBzCo2GGDPkWhBxCJs2jfbIo0sQW2jGsdU7wV6Z0SxsyFQTi0ZQV5Kj4hLAaUlXWYu0b8c7gRNOckf1KlaXgF5oCwHKGhpRgwFqa3bCI+GIomOZqLLInrmfynplhaziHGWmY2x3tlC+lZ8o0oC53/mtbN9RwLIX5We3VOmtH7Ke+3BzwxvXIgdgUc5H2aBjM3d2Jj/lS8XykHNjn+LfC5SSWgZpWAj/tvjDcnDFvpFDILnEjgEYJLF4PnFdQu6mIpcPs4M14Mf0rKYLCOJ8LLVlNgF4ZFeDg4MEJFIbEbavw8puMn737/4enpgd4xMvvPT8zr27aXp2+fLlo0ePvuWRh/7br/+XXDhnddEVqsFtRy9RLweUE9rXZFpEm8kmJZR//+W/abXKqVSS7HKKiianXSIT7Xo4kRkeH4Xm/NLls3QMAyCvqqiq5bzKqjpyx77xHT3TC6cKpYjf20d4Zn5mHSAIBhnsit/91uPXLlfdHtXEzh3zSwu9QxPTM8uhwMB/+KVPhFfDl89dfvm5l/KJLGZ3rVIP+IK0p4knIocPHXI4jS+/8pzbYwU3CdnfmdemdRbt7l1HMtmiL9Bz+vWzzR7/vuFdd905cfn6K/12m79/pJ5cb9azKChGmzjBuKF4SEpGGDg9oHsGDzWQRQKI1PW6oKEz+Qc84GgxqEH9g0KQMiJecF/pVkN0SvZE9NqiEQfZqVK7WmXzi88df+H5U9EIsR4VtQ/wNlI1h5ZH4fHkaOxLLQvP10TvR4sjWWqQz+/pGfnYR949OrHv6G13x+LZbzx/4ZWL39TBguedXFgJGw2uYP+AxUR2MsmYITJHm3NIvrRE/aiNJu6aSYmzJ0pJGWYyX+WzYmXw7p8uMoD+lxeGsEygf8XCxFAmnihU3ogaFktbFCpnqeRwOh/llYW5KYNeVKxcB8oY4UJGh1/JehE6/BGeaKG2mS5tGk1LCI2nwfMD/YxCFuJlfiBBNdht+Em5TQsN6crCeLXqzdL6EX0FH6FFMvB0sVtYWOMXQLpGhq0A40pZVTmtIt7WP6Ax65qJ8Bop3vGhLrfDiTsIBBSf0EhFu5ne3YZalf42LtwPDAmeAC3Z0A2YzrWWJhDoriPxFERVgz66pNWw2cx2NB9hK+QkP+TN+vo6OgMqDBKcg/298Ejjfa6urtrszgr9sGii3qzNz12PRSPBgJe2RcSNgemOjE+8cfoMJMNAJWmy4DPbGP0RXHgavJMSpi9Q2xTeTuSyLatVZ3MBWmpnsvQRc3V19zCa8eihmeZM6Hd7/vx5Usszs1dtdnu/tyeVIR3uw1nnhhMrg5AZCiriCOB16QkBK14+HTNKIBT3wCBNvQwYowZKlcLRmC/YV6EoWV1Nh9cSVy4HvMFaKbE+m3U5jG889zhIKloPVNWmoNOOY4nu5O5B5AVftcvnJRnFPaWhWEufgU0sEs0SWCbCRlUKo0iheeIRi7JBPkpAEXOcSHulNtDdV4SCL5chnMAiCX9jjS1a1cLJ1182WJ17Dh+l5RukXT0jI9Mr5IWndx04wNl6HNZoJBkLR3KpBFxlkzsmIcucnVtAoRZaajo87DAcmJ+9Nj42TOaPztFceCy8hntMoa66p5vIoMrp0Fy/xmlAFBvSmXnoPDVfV0iVj4ejkfvvvx/HfnFhJhlPDIwOwr0FqJWyxUwqf+cdd33tK9+kkNsZ9GgalUQ87kR2Gox2s+3QvsGTl1aleQHGKhkEHaaegMwL+SJ1RggvkNly9Zj0gmfGG5DxjxJD7gNP4z2GHXNJSu9Iu+npgWFdIKq+tMropeqRWl76QYE5R6AQrGEf+B8SA2RCCrQQqgzpWItOIp4qU56aa7nvLfSxVp3nDnPQYrOmBgmhbcGPDtEpm3F0DtqREDdOScQlAkOMAPE/5Bx5VaSTsr2ymdhTaDhZLz+Xa2HhTWdh5a2FNZ2veFVWdrbsvMpXnS0ZBZ0tb/5QHFzmPh9Zz7GUhfeYGqShcMuQpXLf8IlFvojwEIHDHslTiZIQAST7l4iXiP0GedBKNseVwG5GbP973/kO+SKtQ0Lr3/rWtyCVo7yQPp6UHHBgRmsjlze6nZfPX4Dom2wx3dR+/ud//rd+67eee/zxyR3jdB4T+u52C54TEgQICGwtfYsepjw+tI7EP1BS/H/i1VcJg9mlV7caCoPt7bDVbQsFfaury067Z2t77Y0zrxC3UHlE1lq8KrtbXSylsDPo0BfbwvDTA8/sHXDjHzz/1DOXz1VRpulMafbarMZounzhUqh3mAf153/8mf/zE//VZXLF1+LPLT6djiaxtQoY6bkMlHBPfvc7EztHeZj5dGZ5fo7z8gRNx47ed/Tgba+8fCrUN2T3bZvdvqVI9LWz80fHJvWqRJrOUPmiF9GI5OdGIszlUSqKmNiJoBjo70smA1ggOXWVp9/WM9HfIqHXLOEFGex4vk3qslEAtJkSRxwzUvQ3YTwKUK1U/apU9kSi8tKLZy6cj0koRfCwWguNVuFS4WbK3NCJS0z+DwVscJhcobtvu+fl1053Txx4y/t/plBWPX1y4dKl6+tRqoa7N7crVG/2D+/tDvVRjRmOLuPi0EqW5LTQW1UKzRx2J/U1apXZwelz024MNjFDxflU1JgM+huLMvJlKz4rw1jW3xzQN7fpzI8bn/53/kFB3jwhaTwgnjD7xmGVkQyNgNQtyBl2vFgMauUrLkFsFjLa4vhi5nfC0fxcomPMHKoIxJEViDroG3xd0r2KGuCG80QlR0irS+mQagTWZkSwm9CvlKDWunqC61tRamwovCw1mgRBB0MhwEH5bPaOYwcRTDPT19GvjKpsVDXSo3r3o3f09IY2N1ezmSRUBNSC+1yYgSW7Se/tDRERha8I8DpGXKZeIhGLl4P0h8IK6Qa3NC6Ulp5JEHGYjYFgCDAU7MdEvMF80dqky+PCn2N2+UIhIt4Br4ci2Rz0XcUS3evAfEGZBDkw3WZEQ1P32gJMXyxk4jZT15WL53CmDx+7HZAY1igngpB1OJ1EwKmW5PolNgOvU7kGZ6EWJ9zaLJK+xDJRq7g+uhFxN6w2E94tgYpcobi1FYfMZXFxEV0bDPkHBnsoknnppZfpkUB2BPDI1SuXSCX73FYUlUQbalk6mWdisVCgD7ZUaoLSjZKHzoM297DfaXT5L1y66nAyq2NOh+XE8WepD4JYLJyNjdA0QfxtVSxTpHPvUjyCuIGNxJDXE+eyOVwoAycFpllhLdm3Z/fzzz5PHRilApBMAqMg1geNTwti+mYDOCiKhCICYmi+QDCdTB49cgR/IpVKkMxG3KOBYCNqVAsemy1Xyb/07BOU9O7cf2B5bo7on7urPxqN7duzpy8EKtk32Nf7xLe/cfTQIdTdyZNv3HPv/ZwA5Hxur+e1p57KZNJ9oS666xFOv0LU3YKhzRWnXn7y2+CcF15eCvT2W1zOXCSmtTlT6WyuXZ1bWOz1mKsZaD6HFq9dnLk2PTE1Dn718vQ1Gh2iZREDFy5cIgexMr+2ubk11h9CqfFcdGQWtbqRweGr86tknENDwyvbEbpF87CIHNx2+x1cIwF8uD1jkaybMkuDgd7OdIhinExPT4NQZXqjVpnXHXEg4WKTCTY+KkpxaBnn6CHiFiTScACpHSfA7/M71tfjdhpB0VeqWMAXAY4GhIFYEb4dooQ3Uk+rgsEGcCKJ5CoGF/WPiQTl2DWXy1SkdF2mO6axhlOiSA+rCGiFkHYRPlS4dlE5XCMyBSO7VGgajU2JHikLv+Ss0IWdtAIygjWiAxUBxXrKnTranfUiQZSFa+woe2Vz8fhZ2A9rK1VBd7Mo239f3DGEULFEEdg9C1KfvZGWIpaCi49yVQBtoum4YqJpEm0Rx0yABorg4h4IHJqTQ3sfuevO69evQ0V5+PDhp599ppopmH3OJPTnuRw7RyszCDk2qFkaAJOJqsOhyoJ/0GqfO3P2N//rbxw5dttHfvRHPv0/Phlb23IH3Olkmi6Z25tR/F9+DoghlyxarKpspnO2ytyhhhiK7aaUbrMzmgGni1mawJktumQqMb5jgOwPJPBoJcyI3iGTK6CKbq9t0AesYbx6esXkUu8/sDPotZ09SUuRwvAQ3cOa6SRlC/pErtIuqqh8S0SvVYbqv//J3/+R9/7oYP8gpCLZWhIU4djoKEmcQiuOK5xJpClA4jkR98mlmgFI0/Wmp598Jpko7Nlz20/+7M9/9ctf3p5biqw0Fi+pxn2qFb/qHXf218pR6VCAepKYDc1AMDYEcUCaDwJA2mnrbSpvv87b6zZ7zQ1tnrY1YFgbMPpXSowGslQ8FygLVUaCOMLMZbbYmyWzxuTKRUsOh+61V85S94hULuSJcqqBKrRhocU/ozBXOJHaBPKhG4Pwy9s98gd//FlPYOCDP90AeHV2Njq/Eltbi8dhA2rYU/mWPzCKBKY/zvLyajIRLgEvAsKNF0epO1pH4iQYEhRpQCMjkRGuCcuhMzQ5VZk2P8QFZhsZ2f+SRRS5MlL/5xszylnefLzOCuVVFCdfdTQoh5Zok6J9lTXKmTLIFcUs+V0FGseMEsdX8lpv8oAVAh3ZhpCM2IL4vlD/iBqWkqJKWzKJGjOOLK4tE5EkFrY9UTha6PT1eCi37Qq6/AE/ccrxwaG3vf2dX/vmd4YG+ulM4HE5dR6XulYGB0REqt9vuOvAjtHBEE3U27WSC44Js9TJNKoVJ66EFfAhJIMUh5LagzYSOCgNaOma7qdTcLNSSmWzJDJ7B8ZGeoeyhbrBjLR0luswHuaZ7bSGpbV5ie7WUCM5HNAg4E1kAdsUi+HNTbzh/r7u/v4Bop60FIRuN5tL06vBZtY1Krlx9jg6Fk8kxIvSa3D4ugJ+By3qGCa1MpWChGfR7kC6aHmXJ3VUg2tGmOAgbcRdJcnBNCPoTKkuKTxiKcxkmgNyerjgmIek9zY2NqCpAxXIR1gmaBtAIhFhSmUqhbBWjkp6hjIYWoBprJursb7eYWqgR8Z3w6ddqWuKlYLP5IS4h8spUaKUiVvMaqdV24wnza2CuqqBIhtEiQ/NrDNsx5NSOSbVFJQe6WAwESih1jA2OtET7LO63P3TMwBViBUUisXB4RGUMSiMZFLaMuKYAPImkozAJcTkNJuXV9Zdbh/8lBvbMXRqyO9MZigbq5Wxb1CQDhtNbbgcoFTc3lS1hYMCTTLO6NbKCm2Dd+7cBT0p5Ldve/RdhBDPX7zs8/kZh0vLq0ODfdevXLzjtmN0aDfb3LjiC9Obeuh2zJYLZ0/4gj0Eo7/6hb9+70d/grA2fVd27xzpCnXnE+uJSLhEmZbNuXf3nvWNFe3RI2PDI2DoLl66vmNyN+00QI2+863v/LVf+eW5pTKM0UXA3lBu0osiWxgeHFzY2ITgQgSJCqBpkkHFXRoaHqU7NWtIhjGc6NQ7MDhID1qULr4vQWSRU4onyuxD2SA3UaJkYSCOBDLiC1qi4QK0M8oc5G6Ji5HN5QLdLuwVODLBw2C/qphOiiIX7IDMTnav5EZbagrk6lgEwMqbGo/XSjsQ5q5i9N/wgLGgRVeJO07tqZB+MD4VjHFHlIgP6oBaTMLY4lnyLRuLZmrQ2sTEe77itXOG8u9NjfvPpdAtR6LzlfzqHwuszkfWKd/Irjgozi7H4uHyK46FqUy2ULH4uVBZeWO3SkSBs2WNWASd01eOZHE6B0eGz5w9S6jhxKlTzz77LLdHbTOWU1mb38NVowVhH4c9WA5hMzXRbXhf7Iad082HGEalAt7k2aeejh3a39PXSziK54NgxtD0+l2NSi0SzieMeRxdE/aamVo8M8XcTBSUOklfLFdmDaqFZ0zHkq0svTdUvYO2+YU1VVLl3aXtCrln5xN7jw5b2dlmttaqJOMV6nms5sDmOjmRNnX5FEwRHaSHOnw+XD7ejtFGn29A28WVxSVauG6urA0PDN59113f+OpXMVuw0X/x3/48XsGf/+4fWek0baZnMi4RqelmbCn3Yv6lbLqs1liMetfuA669+w4U0NC15OpWqc+pml1S7RoK7x9yqUopWuBwN6EzB0rA41Cr6olKjXOjSN7qM9j8Fp1D3dBBGlPH7UIPiPKQBXFO9hZxq6s0tIr8bwO6hEyvXmhAcb26Ev7u955OpmE0U8XieBpai9scieUdDGhuHAFAtTrfyOqNzn23HSVFd/n6VvFKLJlvxtLVZLaRytboJ1JtUs5kGRoawUsUWtvEZiEdht0aPmOFwwquEJxCYHmcGmFEnHNZJEiCZSmvErdlxNwwEpVv//lLZ5t/vv77axh1nQF36833v/vh7zo/keEuRqlykjLmlU+K0XhT9YrvSwxZ2VIQynivCqhSDioBZrSsKGB8ZBStDA5+yIOQXSn4Z0i3JRmslBsxBtgV+Xs8UPqpsgdAI0gBfGLp7ENIo1nPZ/NTO3up3AINN7pjF0Unk7unLl+9wlM06TQmvQrVQrKwXCpwU0OD/QGPtZhNbG5toJ5Rk4gSVJxEkGsABQhVSckTF4YvhaZxut2UhySJS9cZNKQVBXdcEVxBO5lJq3IlvdlBBo1kJlfGdOL+kR/CjGIveC0AW6C/Z4JRfFIq5vGZqOKN4WLX5SKxX2BhthjV6VrR3xtw2UxbW0WQh9HIJvJ6YmxEbzTDRJQh7EwLpmSKN0wVaGnlNLAWBcMGPlgYRCjHYq/Am5E5mTRN/SyTk1O0EL56DWRygSYnuHpZTG46ogBl9nhoFwiFA/lp4Dk41YQ78eu5CTSESGZrQ339iRq+al1Ls4WhSZXRmt6m0CDbzubpQFwoxfPpFBksu9lCxwqwgs1KHoJkHdYTGDa1FFgzkindKuQLGDhE9avNLCDIQqk6MqIu5XMBuLIheDUaQbWsr63t2reX0hqy0cZKlVS3QGOwLJo6fD74NbEkEpGE2oA3HOhz+eHLzK6s7d49yZBY3VjHwIMFLJIqWppak9PfM+BNzC7C4FFweSx0/qnUt9c3aTnHzCbKZLXYevoG07kiSKmugC9XjszPLRka1bWlpXgiTQEVrcpGRwb6e6j9vYaGg23k4YnRO48d+ZNP/cFP/MIn7jp0eG1tjgYbFUNr+sKFN954gw7Vmna1vzu0evmSt7srm0yAYEol41R65LIlLvHt73znC88+Rf9Bp9tbBgoqBOQWh9XZqq8SzMLMQy2B0qc3MuYILYYx2oolKjKJycOHJa1cYNFCLpcozyV6rEThkQFILMScEF01SXRV8e66h5ygee57yx0vP3eip8cPOIssPrMPEF2uXFRX1JNTO8EfJNIFn0u6SiPppdcm0FVmnaSGgCWacC7LFSn+wx4i2E6fXJD6tMlGknYUJ7MVF1JRdRLg5Y0Id4F4yXJDtynaXUSDouQ6m/EevaiIWuVFMSPYRpE3b9J+b5I/bHfrEz/nPdvf2kNnTWcD3vMVS0fxc7ZiUUgKGYlOFgkrXrQCJyiUnkga0CcNiPgl4yZesBIJYFcUN3BvDx85MrZzgswlvi8n/6XPfR5mvRp+p1HDHaYMw+bzkOsluAWmT8Ks0I8Jhl/6NRIhk0R5vYGRDZnZVaOOmlp06uz1GfqRpLNZ0lNWs8niBcpPRbs+GstT9QJSkSgDkEnMMQJFnK5UI4DthebdbHnb2+7pHQuNjfc/9+LTW1vh9c3E9NWE0alaWLn+LsvdB46Mnz/ft35uA6g2CH72BI6BtjKgWPDl6FtjM1shVZF7CRt5NCtS19RYXVn5zJ/+Gc0kElE4NYsurzufyXvdvsMHDn72Tz9DxX4lX+C+SfsPke0q6p3tZtfuPUfe9vAj8yvrOBKWhx755uc/t2fYsBWp3TGlIvZTqJRpzA5Ps4h8LSW9pGfJVkNdoqLcyNtld3gsaqumQWGwGtAjFYrMc24fPjO3XeF5xrGCqIZoiop0CLFoW7uqgUePyN3ZMydJlCVToj4cHrhwpBIUqwf+I251MZ2F0g1ylJ//pV//Nz/1c0+8cMrsCr72vRdyFW2+AikQUEcwkSa70+t0+DJEGbBX09FydlvVyql0JCYoby2Q8mJoKGMC24P0M+qKBBAnI9Fw0Xs3dDC5VXLaMnYUdXxrkP6/vekMaIahDLU36WDZz//b0vmV8lNFg8qAlzeiNZm7nI6yRlHMcqYdbSo1wdD6yGZiDsvPldA0jjITgFyvshmvMoXZEg9YboKoYXbOJWslQ4yeIqgBzpl4qHgASn6AABoIQhVeYN1pNx7cvxsK6Hgy7fe555ZXr165qFe3AU85rLDWkbRseB0mOv7QBa/H51xZnsWxE6+XqiUlgg05KjSAlMaQ8eI8MakALpHdoWSWK+NMM3i4JbByct8rDWndWqMlYlNTKmVgE8Wx3wxvAyLwguMyGfL5NlRoJWiZhLxX6j2gcIMbDXuTDvLx2Pba6iKPtVmvEBdrt+C4BqtUQi3xFUwgULb6g36720eJLX4HypKEGkZ3IpHiRjC2OXPyRnixuCkST5MJIrXJnAYxgwI/q9QAQ4a6eglAISagGOS+k1uifBpli8HhcDkazazSiBfBWgCahF+NGaIB6FypoKsAoNhtrnyxbrB40qub7rEpk9WT2Yi18pjwNsptORNUKKN4e2Pery0lNiCRdlmsLuBwpWpdV2v7fd1wnReLMQwD6lkoR7Y5XYlwvE6dFq3fAF/VKqSfIZPWbYNH0kJokOImVypBKojopQhDUDRqbjRtej1dEAaGxtEi+YbK7XR7erS1Um5lM2K3QWIp/Sqc7qAlOBToH7N4u6/NLff19KKxiGhfu3yFtp8Y4sR19+/fb7OYrs/OPnz3A317Dz311a/sPXB0cCx34Y1T7ZxqfWObXFJDbbTYXF6fKVvK9Q32EDmg7unMqdeO3nn/jz727m9++YsPv+M9O0fHVmfPjNx9b9DhfPX5J2nMOHv1QtC3c3Dn5Ooc1PAr9z3wCATyM+2leiX82quvHNh/EJrwC5en9x06BHERjGZk968trpCCSpHez5RtXsY+Kq05PzvT0z9AtAPH3W0H+annKvAgCcDg6dIPWklhigYSbh5RwBKO5ofZVLN7xEd8dd/+/ZQse7scUJryE567Q0uqu0zUmNhtT28/+LuTp94gziyDWjQpM0DIGpXZj2Kig2eBXAayGgwgWRUCch6HrZABKiZH5JVpLvQciiLkVRxc0kXKDoh7o/lYOM+bQkZOtiNaWMPRmLwdHcm++I5F+Zbx+wOWzrf8kKXzNW+UH3V+pYx7RZTxLV/JzVBOrHNFnfujmAuiuREyGAaYp5yhfJJgHRJGbA/5pNgKfEWcjdpukANibso+IdfRivYl7OxyIRmYfbCq7d69m5nFo6EImH4VxKKoAET9wkoPXIufYJJjobANJv7oxDhblgqFVD2JAIE0ytPjI8GGxqLO2CQ8M22TRb0ZyTkcGGg6LCpqG4ja8ySIgnzoox+8MHPS6dX/9M99lIan12auL63OrW3M+XuNlXYWQTc40u0Y22iWbflimohgKp3npySSqQ2HnYDcDla7oA2o66A4XcjZoOnSra8u23XmudlZJAgKqZnJf/mLX/qR939wfGiCigzgUlwyXpLNqCEQK03sjRoXxZPtViEbtwWCNDvB8qe2YHBM63CSWtGXqymMCT0UJapmtqoq8tAsKr1VFewm2aW3Oi1qM0MECx3KXrSvEkERbBACDL8F1BN1X2Zp266m65ZDpXGrGoZUrKxpOdLJ0uzM0uJC2W5TUYaNoVUqN/weJ+1mVsNRWJBCIyMf/fhPfeYv/vrOh9994vxCMqd58hvfMNqC0aSgO2xOb09vD/zqiWR6YWa2lMZhgL+VHD/oOTXMqVLpio1CU0fFLQS3x8DgD9OSAY8CFrtMMdM6U0V5L5paNpKv/jULI1hGnjJwbw7sf9HvZbgq6lMZr9yF7/8pg5ndojsZs2JhKrpZFDOmJyQCN36CWyMXJiqWjTl5Caox48Xg4z2ZKxQwyhZCPsW0JrRab1VLuMOEtKQq0mSQdDLc2vLc1PTS0ru8Npp3RtNpCBB8Xk+GvjRmQ1+PPwuaKB63GqSDAjBhPaq6WcVUrZWSlSrJxTaFHPiesBFgfoqmk+4ZmFS4uw2Px+i2OM2OmraIUEbEV3A+sI7xNkDuEOwFWESB0+TUvjLlKS0aSRYoa9EatH293ZPjY9cvXYHKSgK/KMtc0WoxwXhlNRkTcTpccmEteuoh5htUJQijb43GY1arhXA0BIAun5+O38RfIXYIh2fAd9NjgGzixvoWN4Cw6mJ0CZEq4SnOmOI/5jkiArkhbaUteMnNhgW613Q6/9wzzyOUofUxOXTSTNBsCIZ6XF6X8Pu0VA6HzWF2cnrslnlazpe1UH02yvRoc3qs0VhYAzjX7gNwvrS6GWhbuocnegZGzDY7zpY/5KbNVH9XV51gfWpsc/YMmd1621BRWzBKKIYnUF5TFzAtyWdZqHpRqbwOO2HhWDQtpNFCGsFrmdJmToZsNLgzLgd6DYgnevspX+rC0jp16gSiPFesWALdbTsltXQ979uORtLhRnC4e2VxGr64YDBAI1DaKcF3na+33a5gS7eN0ITiYPzAfkOrfeaNk7undtH2jVCE29NltUQ2rsz27Ziw2L2w+vT09+85qKonaUiT2IpGtRbr7fc+uLS68uzLL7/nsXe6CrmtzU0G27e/9rfv+bGfe+yh+77413/17379Pw2Ojf7DH3zy2P49JGhrmVQXWep4Ijo3T+D1obe+dZpaoExh7/4jOALbm2E03ND4eHVpgayD0WbDVKddOnBZbgHI40DA4fQFSyBD1RpCzfv37cGXWpyZwwfCoOS+eTw+tEIjIdFOpq0yLbBMO8JRdCdf7L9tYnl9g0QkdeELi4vkgKd2TZKGzOaL5Llza9vd/b3Eny9evULXOZcXXLYd0KKoSWImKrpfy87RlIR9YOxzuC00l5JKZThyTTAnS3UKWpYJyVadc+AV1YkJxQ878ocRxZxSRIsw4N84Tzlh0Yoyr2WDG5eg/EhmQmdvSoRP9vQ/X9i4swE2Y+e3fMSavKWSkRHKsW5uJpapwAU4L/EQlB8TbJN4miI4Rbkq0ohXzo1dYXrij5Gk0FKl5oKrOUXwhr1wHyjkp0X02MQEmea1NaERZepltuPst6quiIBGzOFHkMLUUWEuZKn4dSSMwGrZHVZOg5AIkVK9wdh/aAyylNS1hMqn7poIgIhOLuYt3Sp3gEgZpQCGmiqNpAQeAc2k2Vf6qy/+mdnZ/vYTf9s3ENq9+xgQEIPFPrpjVKWPxjOR/h7b29/zMOP9uSdPXj255Om10JoE0DdVBoTJkaeQq1EMiAaVHKdWQ/a+FsvaPF34A26Xiy8V8vm8vSvQ19c/NblnamLP6gLhGQK72PUAX3GBGvRHSKfjp1snwlvbuw/tBriaSye6A8E+p2bHRGDnJBcWsxmpWswVqQUFh21QwWpj6zHbvFARmoW1DQQWfajRvlqpZcCpYMgRcYTfHugqUGcEDo1LAPvjBFOURNuheoGkdRO5ce0K8coEVD0Op5rmqzD+lhqVtc2IBv1pdnz4Z//tW97+XiiADt/3rk/8xidHxvduhME6eK5fWh6d2LX/4B6kyuLi8tzCVZ4CKEhVNUfSuC3M/CXwleC2iJNroaWR/LXkESQtRGSDoauByE/6oqCDWc+wEQSwRKsFdCijRxa+uaWDedMZaJ2v/smrMgNY92Yd/E82efPHW5tx8Ft/bMBc4mNHxfJetKmsYbbxP/OKWccrf3zFe5m7srIzCWVXnQ3ECpfNJKB1Y28C6BCWK7BHHIaGIlK+h3qRfKDRhhMKkVNJeju0wW8aLSafz+UJ+qiYWV6ao46T9OT62jKoO868mEtnknEHDWUT4UI2adbyE5VZbYluxykXh10Sp5xsUbMNn1S7UKYvG/6xkdgyzrnKUDOmC7SrKzZglJAu1vTzEUIVISZgYEpBIbYtU4vqEQJFdqcjGPCROE5EI4vqFnCnDDOMOLBeU8znmlUzF0/Ak1BJISe9CqmoBO7EfKDWhruJPgURIIHlUtHRDhRRStVUU71KhVJqHQUgxKboK8YrCetSsczVIXpxXihDIfpNrJ7Gu6S64BCA94Pp0qwaKC4s0omPMiSNhnobCmsoRe0f6QdxllvNUvGCf4xzjhRGvqO5yQHDJcZlws8FHBEANbrVaOdRNQ/ddiySQuW3u7uHDHYH2hPma1jAMB+oIHT2jxYzsQCtAIqAS9Q2g2N4dHRxaSEW38yl0jaHwUInbVL37DqfN4DQoR0yFaZaFUVZHo+fIm9C4oTDibjiYZDBc3o9vq4uVX8/6VB8kTJtHG3+psmhh7Rz1wHvaHnz8W9FC5XDdz+wsTwLOX6+XDG5fPC5RyJJ+0Bjcu+BAq5lLFpa3xy86+7N1RWitv29vSdePzV07K59av212bm+3XuD3f1La6u773/YZLQ2Ar0A0AZ2HoomE96+IZPHCxM86SDYofxBz+bKRm/vwBNf/NzbH/uRf/eLP//EF//G2+1OJtN9fQMf+uCPPP61v8OTpExs+vLlnsHeleeXaeEyMbkLe7BWKe4YH71w5fLg6DjFaidPndrVP4B4Wl5dIwqdysPh3W5X6oxb4vXg6cjtT1+9xsjBYqSwnNgpj4ZAAYIewcEkQciJQBWTS9HHTDsgESrV1em5rr4QITZS2h//2MdPvHaCwkfggeUMTmz5vrfdPzs/n4mnu3p7rk5PM5OweJmFkhdSQldiDIs+BSps6hscosGX01OEpdjcbNt1huRW2GoRTBdiB+eV6c/WTH9+wNE7eovXziKWgayXz5y8siWjlUksUknZXr5ivZQ3yyK7EuP6hyzKHn6oRFOOznEQ5WIZEC7m5rCwT9axXg5KBA4FLCLyxqEx9OXi+Ur+Uf6TT52F+c2eNOQVufNM2FgkqqIqyW4uhhMDUxPsHA+Y+8DEyWYy/MYV9GS2UwxwdAHldowcbm7nfB548EEab8/Pz9MBBTGPLwInBOOcdNp//m//rVLI/9XnPrtybk7tUhl7BPOMlU9WCw+Qe+31B0nHJKsJWNzHdgw1tOls2W6y6p9/6UWr1UtjaQoRQz3m2YXrq6vrw317H3nkLZQ8ffrTnz754iJeNUXEuWSe8YJKURWJ3eGNQ5tPAl5h7bUaC3F811Z0bZt6coJvRiu8NbbeUC+qOp3Mapt6ng8FEPgdBJO5lXRIQfbmkunlamV9c4avPOYAHzXOwPz8whF47zWUkNdQ89wK3FdPyGrptqs9MEOCfyY+pzxJLdr1Bq8qQ478tAT90b4am1rtkl6/SGgAZkZLkxhUvAhsnIal+WztjVMXlpeSsE/PLRF9wWXTJlN5CAQ//G9+vKG3PPjYB+w+X7KxaA1u6xrp6aUY7cgjETBrBymBXF1c59iVYt5CzrIKTW2SNBnoRcizWqTPeJBSZo9ob8PcxNhkIjCaJDirhr1VRg3aF9NKpowSi2bGvUn7dobNP9HBnZU/8JUxy/B8kw7+gVv9sJX89NYfN4K/73/kXCWewB/rud+iieVV7HXRvoxMtpfZxw+F85mV4s6yHndO2RtBKgJVhEJBWsllozaARKnhoAJdZLEXskXlMULCp8U+NVlNNhsmo5FnumvnpDcQou07uYVYMjM/ex1by2IiU0y75TI9djSqGuMaPZXDjm2osnQKEH4y6pdgl9TiyNIA0mKWOC65mUaWus0NaS8D2YxWRXMYpg2RaWHaJbykMyBmmIqEldwuEKfqgYGBkZFBJht0GqCfKNrnK8Ql8ooeDzVA7bVyLFI/eGg/p836TlbMaoUGGd1TgvPC5nCQr4P1gNIEcm9d3f2kbciLaLSGXL6sj4PVktEKxCkU6oF/EghVuViWcmyukKwr32mpWGukrel6RVc0U8FpRMEBcUpnU8VGxWayQbzBzcIqQXxg5PDKfERnY/QQmRcgA/dKLTc2Hk8DknW5bEhJ2surpqbUpy6Ho8lyM+OD1wJe6wbo6BazghC0nSywppjKNnQaK7AfvdnjDY3EsiVodrYj6/TtQZDR8bCQSRdSUKDYivWqb7CXRBfgFEIddFwtVeq4kjxf0NRYJ7iIxbnZSbOZQkzGhUZvimXLowcnXF7/2vLWwPDArsN3vnH8+bbBCr1uplLt7R92dA/02gJZEp0ak3dg6N57q9/9ypefeeLxd6PAKlW4eZerjUAg0OCENrfIKhSRc9mi9IKtt3JcVQLaLdO+I/e4I5trsbTBqL77Ix9dO/HK2M4db7z8kotAVy5rNdpfe/rxu9/22NRQf9Nu+IVf/dW5119LbK0G/YF0bBNM/MBgLyH9tc21rv7BWCRcKBKk0YJtP3jw4LnZeVyP7VicLg3YS9SWCLMZYGKTKpmVtpQ2hx1aShhMLl+B2MsCoSSxYqfdhZ0HRwPBZEpyAfyTw2Zmid+Gs0XgQofhJThNio8jqVR/7wD9Pz7525+a2LdjeGh0Kxz2+nxESvbsOzCzMD+5d+pDH/7wF7/4NxmwNFLqLtpPskBSlCDZLgQ1QQg8JCiDjhy+/eLFK0T2hBqlHWaQMTHxYBSxIDhkmcWiOCWeLDwFCCyZ2yK52IYzZHVHAUsATDwdWaiSkZ91foxbLetF9SnrlH3/4xdln7KqswGHVQ5945X1fOQo7Jk37LnzhldOoLPIBkJ7yE87kXARUOxPfsulisvOFopQUeSWQn6NHjQy9ui1t761Gd/aJvXLmM9qGdMldsvEZwrzpiPOmOketyObzDVoRa1X7iNNu0HMgffs7x8cGZydn7l8+SK/ASWOHQIxLebB62+cuu+uO/+v3/3deCx86sSr586cXl5Y6+r1UEquo8zIZHz07e/YuXvX3Pyi2Ul+QaUx5dSGUipdPnBwLBIpUM6rNRbS+VIgpAoFGo8/9a2L55duP3zvI48edDn1V87H6c2V2kxSJcJl5VJVu9dI6bfNYhmfmKK6kuzzhZPnPVb33CtXjF1m5j2XFZ5f/tza55/89rMRPEuJxBL1oPkgzTkk4yD3Fv+Ppiq6VjZZGxoPRNfCI/29W+vLnmHTpUtX7j/sJe3hMgkjkCvo0vptMJMXdPlyq0y8Qh6UDu0rxSxyp0n88lyaYC2hQLVq1Q61xqFSO2i6I9/QR7tmySajQEo525eff/34a2dogkduixHqcluXVyM7du3+2I//3O133dcyO87Nr11/7vT6WnR9I7m2kjAa3f29w3ftuGNlca0E4jGbLEFiXMqpGkX4ASl9L5VTYPS5IugkcMMIhspIQwtjKYgFibGJ04NmImEnegsPmJB0J/hMuhj7DF3X+ciF3FhEQUsURPko/8jI/sELo/EfD3k8KbZk7Cg7YUeiSTv/8V4xFrlpyq47L2xKpFGZdHJenLPyJ4NbQG38nkHNpOSs+f0N71Z0MN+Izhb0GzyDik+MBywjWZSxjG5sYynOA/rMZtwXBQKM94RJByKhjLfBQLBZ9fi+3EEY46nqhSwrW6n0DPRC8twd6u/uCxKSJINI7V0NWGohA0gJFC7BIYoGYgl5ysgSzB1uJVh2gC6UQeJXwyQpFQnS3w2G9DwXbqOQ2KDLlopD/f3EPSrVghI8rFAkA3gRF5DyJME4UzpKSxR63dDdMxCoF+hiSQvbaiSR8rvN5IN9HjudTLLxBMIAYDPP3GKDohnGDj/IZLvL7vUHqGwlup3J56huo3tgen1ra3sLLp9Oms3tcmKsILJwZEmGwtCUy1AlAusf9x4PDKmnom9QLFGsWvXukT4aIUH8JYwLgjSRuAJzIB6hEQBUtDiikDNWAN0ADAMWzcyoQ0bfhh27CvpnoLt/bmE+1D+h0ZhDA0OqpZVLly/b3MFMoZpNJSs9PdIRG2ocnXF4fLKvJ3ji5AsFWjYRZ1JrNjbWSM5fvny2UcvX6/R40GNTdwW9mFMkxSFahUYyG94u5bJYEqlcMQPoFx1LFqe3L371mt/tgRRvYXUdUjj4B9o6E+1+ux2+SCwa7O6Gm57aRODoEMstr2509Q5gu9ANOrmy2jtqd7qC185ebZTzPSFXz/Aw1RlXZ6YZQQTbTxw/+dhjj81OX9vYivaNjM7PLjEOR0Yny7ly19BEdCO5sR21XZ0O9YQGR0PXrp7vrquXNuK7x8f6xvefevVVOPG6fQaICl/8+pcf+ImfUJWL61evTExOTkwMbV+/vL2xZHKZt6KRSd8ORAYqQSkHMg4Mjuj022cvXvW5PRAsg2vLpBKIV1yoYiJNWS5RHvQo+jWWygZ7uEnBfK7E9AdYDzgL8CbCiuePbcRQZzOZpDK7wDdKCQBemvBL6A20ULzr3vve8dZ3XL8+S3fLaDh27vyZdDTh9PvQ7t/41jcpB3/vB94PNvC224999WtfwTaTjIukVQTJIiVJTDY1h6ivc4M2Yj/yox+mPANAVgHqJjMFd7gHHbUFNEVivkxi5IvwvVKJL1hRMQtkkClihKQjJ8x7js6C8hYFSEEHCl+0pvyhfvkh94phiWJm/Q9cOlLnzV9xT+SjIrIUQ4TzR9hxSyTQhjVAYyPe081ZRCAAQ+xu3nHBcvf4pcCyeRXNy7XgHaOj5bZKcEtEGg5AsUghzO133g3UIz63gvRNrW6prUbUz/ve9z54GZ988smLb5wxUcHT7cqn05g+i7V5AIZKDZIcjAIyUqZ/87nPf+RjH7n/3gfwsubnZyWpnKl4g0GODoSeZ3Tx/HkST8eOHP6pn/nZ48ePwwINT22OpqRr0cK9peGhcfIt1+dOTy+sB3tM2PjJ2CxSBvgwGdxyvh7eUiUiqt4H7e1G8fSJy9HNzI6JYeJJ9z60J7adv3jmgtvmAg6Zi622aBaH22g0Hjt4B+ADIBHZoex73/nYH8Y/GV1fU9ugXFaZu7zl7Ux4fdsf7M2SyYKUvCX8vgRfhDrSLAh8InZkrnbtDsIvVEpc2Fxb7vGo1jYqzbyqVY7+m/eYfQGjM+RXWaGwRenlWsa6yQbGBRAf4RxkgA5hq4w8gpsMPeGDwdyh6EhFLzTwpBykpYHQzmayA342mBGz3ldeOrOy0taB92hTMe9YjeQdXT1/9jdfffXEpXse/ODv/NnntkqN7z5xyuHyl0taOnfTc8Rh94aCoUtnLxVy+FJ5dbNCgJACBwqeKsWS1WnkeRPJo6AGHxcrA7OHCAahwxsjTQY4I4RRzxr6MbUYUrQoYUgBW6RGRaw3KbaToYw+ksYAWCcSUvq+bpWx9sOXf/QtR5MtUZ0MZW4zqkjZE/8yasEn80lQU9h8KFH8LeU2qbHxsBLAW6BBCY3L8FUSwHLOyk7kKmRqMzFRsULmI1aG6GZQ/QwIbEZeqeNCyrATsSnlKZOUpQqCqYGZT2dAaCMhe6qkUjEgCwxsrxcGJHujgfOGfa6meYWvN8TDXVieCYW6oTsUro5KbHP1GnyMsNvbrSawSEwO6oPL8brGQHJFcFWcI6cGzASMHNdudTElRWoUyjngiB6qBeAsBausBozaDMdi3oAf2l7+OMPBvn5ck5GBLux7rINweCOWiBJ6BbpSLjXpLueyOreISqEbS2XMKtqg+wb6CLWszM0NjIzSqMBud6WIchcbg8Njc7NXwUBZbFYcf6+rGwGRLqWjqTBs+8gnu9XgsVni4TBxSJfHG4mGiYbLjZJDo3rxYyUmgvNerNFoz9TWWuLJkt1C60Vng8hLPp8rV4fGxuJ0g2/Ve7qHU5mMRl31B0PXLl2k3zIQtlKaIuVc0O2EaySxGUVUmtUOavPyzcb04rrNI3BZR7OqLcRHR0O1xPp2PAH9DCwPU4cPkVy64+6HvvYPX8rG15hktUJuo7ilb6frtTzF1Ll0y2I1baxHiWVQnUQq2qzTW8gg6OqlzW2rlWoCvcMbGBzfuby5zUPhvcPhvHB9ztM9ECjWrl69+ra774WFgFj0a8+tHzt67OwLz97/6Nv8FHBwbxkkJjPdT6Cx3FidoxGRvZ7t6Q9cO3+mZbWN7N0Nt9/hY4d5BBaTdnnhutcTGL/r9ldPnXX6sCtGYRnbiiT6TW4g4vNzs/Hw1ujU+My5C709/YsX5vcefPjl146b9Nb3/uR/PPPai9GVuZAdOq/6xe98cf87Hjt76oRq3+6LZ08N9XYNTIwl4xEgdevhaN/QyOz8AgVNVrvu6vQMA5kITSGeBAv2iZ/76RdeeXVmacmgU4NPEWwlRUBaiYHBx5BKgHBP440KA3OrDqc8c4dEucxBiZiKupLeCCQWocdlABmZ/fCki0ywGA3n3jjZ5fV2d/f+2Mc+8u//3a/AuIIoKELkq9djh+G0/f7v/z5NbdHlfl8ANM329jrq0Otxon5QDOhRHO4qabFcxu40/95//53xkVEAhqRQgz3+VCpuc9lGh4ZXl9dAIVC4TuswyAsVbxx3TlxPLFfODVeKilscrFZdipWZUExfjo5yRrWJgc+kU8QN8oCwNlvIGhEd4heJZry1yHeMbOUnN76QbTgc4HC+Q++ygDuWYSALP9CAplH205KWwfJbeUU8MpU5BxSuxJ+U9sOcFel5TkYWxSbAPWIvXAogN6fZOn3uIt4tsQe0OMQ4iMFSJj9z+Vp5sLC+tEq5n98fYAAjI2wBt6PgLWxuE+0Dw8seKDSyUtRoML/ywiukQPHC6alE97M733bn5UsXatXUej79Ux/9kNdu+9Tv/QEqAWzzrr17+gaW6XmEYjA53dNX5s8PXzy095DHZbky//rVc2dLaYNZG7h+bYbzBfSlbwGsq6+eUz2dXL3t6K5wdXV+Ot7tnyoU1RZb6yf+7U893f/cF/7oL267/aHtzVxxLa1ze1K5ajFa/cDHf+Qv//Iv3/juy5omBD6eaGSbmmi7zXHvnY8sz29ffeY1GtQPjU9srC3WiM60ysIYI7lg4oUqBiqBaIulC5Dm8FDvxuIMAYLxAdX9b1W95f6+oQFqOEstXRkTqGFoMoZremjFSTkJhQEcG8x7fdsGapzhj2MBipTbD8RWLxqmRlQOkmoIGKyaZnpz3W0MNgqtc6cuJqJ1ZvhaWBXJqIzeQu/OA297348XTCGtv73rwZ/+479+zeUP2Sxj1Ry97DDp2Glj+vKFN46/InFm/DwKkgUHqlCaKyMCmUh8FcwEHeQkEIsQlS7vaANF2+EdEmHig4wnZmELqiD0omgmFnE4sSgFlaQMHQlHS9RaGWnKSOqs/he/iinLPpVX9DjnwBrMRQkpKYucmCxyFKaRfOQMxVrBRRXhoDiwyrxSUM1oNsWjlSmn6FT5Ci0rK9knBhAKWLxeYfAQBSwhaCEF4F4p8xARpMwdJKwgOBpiJGOqggogMY+1ze+adPchko9PS3ZCv7gyb3XSGdcRi2+l0lHmLVJgYrSfEhSUDw1t2BA3ixOXvFON+ArHkokq4g4BgtmhzGeceo6E/00TPxZuNE+BbXXgtWBmxz4B4Qc/osnKnMQguXLlEvcAmzTY0wOIHsmbyeW9VLj6glKsLOUJagspHw6JNQXPX7ZGxLJWroK7oeAyksgAeAHbjEiNbW3ZfU56+Dgd5hIlTeUMTeoCXb0L1xdXS1XC35UiVmkGJxtWWNonsDclws+tFfyaCCsJpYl8Jm5EAIUqKcDXIndadSp2gGVx231ePyklClpoG0CFDyTYVEG1yhW8Xx4PXheGXiUP1JDQphGMDs2QCQjQbwCUUMDp9I7hWOtVAWcFXx7pXip86zvfxAtbnL6Wi0UVJ5+HX0/FE+VKgduFy04Mk6YRDNcGIUlVWcdTNmjh1aZTFbF8DfiHSgU1iUMQj4QnRsekbjvQNbV33+unz/IY9hw8ivVADXQ4shX0dC3OznIayaVl8gdXLl0e3z1x3wffy1yKLy5X0sWFmelSvnbu6SftoZ6Av6tFSS51Qi+92OvzwoCJGbQwfa1vz34Iq0FNw5YV6Bvp33fwwgvPh1zW7oA3srV25oXnj9x9L/jy1Y0wHdt27TpEnyjoc4fGdrz9obv/4S/+mDIRWL2p7YBh//jxV3/0/e+lOmJgaIgeCNvb6S69+cmnnvvwh/8NvRHpLjw8Or4Vjh0eHFq6NheLxeanp+PhbemG2lY5nWbS6kxhAmCihWS+M9R4w4MUy0LGHcFhWUCtQM2rZGkQfhQEk92UWSrTEgnDg6d37GYscvzVV7BdKLN2u6yhnt4tbTgbz+hdeiiKpg7t5SA//bM/c+Hc+a987R8mdk3cec+d3/r6N8qVWmm7bPBqK2laaCfd1HTTnhJ0XKmyHQ3zBAVRX672jw+uL6+O7dqJ+b0wu9Ad6IL2TfrYS0QZCAdkowx2iWAz5lC6mBScFzNIroyTvyFG0H/KRy5RFnnlWxm6/LKzTlkj3ymfO6+yrbLcXKnIO0VAdfatCAeJcosTzgnJxLj1E2X/HJZ1kujDk+DchPNQZNHNgyp7lGMjb5GwzWK5aiqJ51xrqOWWA16SPZOdvXD67PkzZ4lwQCyTy0Dzkx8aGjh78QKEzhwS7asoezBQBo/DMzHhx7b7+y/9LY0WpianSAesrKz951/79f/rv/yncCL2xPceJ4jkCwUp1g9H47kTb5BGwqw3A7AAQRpPP/G9J9//rvfBjkdHtJnpufUF4H5eQEeQASCYSNkajIYcNTgrquv6DZXOEt2MXTi7uP/goUtXr6jb3/I6u3rHJs+dvdykLbqvF2MO0qjPf+Zzvd7Qvsk9//cff/IvPvdX6xdm1SEXrKt33fvARz7ysae++wxgiOjKetrcdLrM0VKD9JIB91jMRPw8lctm8QX8GoobColUYtFuVT1wu+o9j44d3uu1WfJtIlQ62AloA4LWghYVF40HSezDAOWdFBoRe1QbYBrjXnK7hMdQAg7A5kUZCF1+kYYSkBmlIBmMJ1BL+u888friWqXUMu09vP+xPYfueuSdNZ3jhdcv/+6n/yZfsRYatIvi/niMTXqgwsZLiLwCFh3wJwKXwjCB+0r6BPeOx8kfC+EjvBegabiSjFyJElL9JHqUj4ReicMCc2mL7+f2+4mfwb0HSoEBgN7DuhK+YnZCgl3shzcvt8bTm1f+i9/LrL65MIJlEMs/cmDlI3pXGbUi5lGVTBv5il8o34pyZeyjWVGiYvAqGysKmB9giIhykBOWL7jb/MnG3B9eOx6wEpFmoIsYYhKxZ2Q0twcLjBipHh5g1iiamNQsBatkxDBadZKvaEqoocTYNUvphcNFehXqY7o1C8KtUCX2wC2mQSH6n5lo0BtQkCwcQvoXkolA7ypFjdBOsQknJaaTqk3hCoys2H12h0XUnoIdZQ3xQEqE4eBNZ/PTM9K23eH0UtYHHwINuoHXUAfCU0Jxa/F38LXVLdQe8RRKEwvAW1uJtiYDrqkcjxNF3LVj7Py5U7R8UpU1taxCm9equU1aLMGQ246pkU5EYenyelz0C6+UFokbA83lviDRNFyEvOFeSakAiTokNiMOzBS2Bg4u5lC+kBm0D3G7AfJzJyendgC4feONk5gRSApwnogNVBQuAVAsCkEhqdWazY1GxWJ3Uurj6vb1Nyo0nYVF/olvfXNinD4FE5poYntuGQCzzWGBb4uWwInoVrSSsxo0dHSnKwSTCgVDqhgrDMgZQhqriPojs17ToAVDnc7CNEHXUC9NaDefScJty0UzxkJ+X2jXnhe+8W0S3jun9vKIGzn1odvvzaUKwa5QMpIw2Jw0kjp2x52lJj06ME51/u5+/6B1a3E9lyoRW+jp6gVrhpV9bGofWACg2iM0YBqbSMWSqnQGNiJ8vp6hkW9+6+v3PvS2A0cPz5x8Mba19tiHfvTVF17MxmLOUO/a8gJX3uX1WMf7qqmtfr9D5Sex1FVMR/VW6+byKpM7EOgCaIfQYBbc/cADF06fYBjs3bsXH4vIBAOKQYXHCcc9Q4U4BF3nEqkUMRxmAvBvSg+RTJh/dBLpzCSx9hSdIPMLeSD6S1SHUMLhUSor2acCIZC8q9hcHIQ2w1WpXIL5YXN9MxZN8Bz7+nqJlJ45e74YA0erWlhaJAr94ssv9Hb3TE5N0t7u2LFjVy9dJq4AnzBf2YTcqkJ5tNttB5AHszF9Ahi9wBQB5a3Pr2osupdeegXzK9TVDQCYXpaUs4H+RSKIJ4lmY4wLjYKkfpmaJCPELVYugxmtKEDBEsqVYijKwi9k4Z1ymTc+y2V1ROHN184GyrYiIkVmInuUzW/thG3xXbntHe3bWc/BMd35gRIk5yeMQ/mhskN+cGMnykkoLyJO2YgZg+2NBBBaDwk8EOWT2cTKBhdF5B7xQN2AuP4Q2uTBOumh/uKx8q04DHod2AuX3RHw+LgNhUhCZTGsUeINRGB19fjLr8AVE4/FnnryGcCbie2oDrgvlO6U8ZQrXAVCjOZgYsRoNJ/5888cPHLAbHLqVGayR3TmzKbK6WSOcdGQQpB2d6grl4usXMuafECMtPGt9JJlZXJw6syrZybHd431DoWvb9MVqJgt8WzohktvxM995k/gJaWlhNtqDvstdaBkBgo00s+/8Axdi5rtqEqVja1kD925d6Br//rSxvZalBtiNAGNIjRWrRejele22gjfdsjxnnccPbo/aFBva5pbdirbK0VJZGFmyZPHtJRRShKg2XYRGcDPEa4rSbyQ4lPYHZoIAi2dToFHSzW6VKYDA6KUUpPBN7K6v/T3zy7RrMgUYt78uw//ytF7H7m2tH7x2mK2aOLBEJjEr4I6iSJOqcGu1sg5VstUllQEF8NzZHeoHLQvEp2PTC8WwsuMId7KQJOxx8Ko4BHgDKCbAcPQgAIAD8MM15g+b7hkFnHZxRRDHelw6lFmBGOVYc+oubEz2fv/9iIDVNSksiOGIh85c9QWREaKCysBZ9G1+Exy3nJpvPIVm3EqXDHr2UBRoqznVG/6xMoPRTczVlHACtWGhLZFYXd2q+hd2S0nwPxl9IPMJFYnTiuAeMECyHqOhn8q4opN1XSDsVBRRxiBYjPYKOmyR6t5qIjga4IbEu3E4OfcyG5CrsU9FyQVphF2rShg2IEEFcVjIIEEnS8TFQuXmWzSGr0+N9glekGwkr7ZuKbIUp/PI3HdQvHenbsYA4xZ7C50Uu/AAG3VAz7/6eMnEPFINJIWODOcLs+Scg5GTKpQDPb0U8tkd/nvPHJgbnF5z67dmmbOpG4StqvnkrVCwtflZ076IX+Pb5s1dEDSFKWmKmu213WVajgaRbuggBnLYtLIMEKaYOITEuamUFTM6SC/WyCDYCo20EyzXV9dXuAWgbQsU/xj0lOOvDg7QxgTq7ZSLEA1BWkG+Q3aN5QKNcqEgE0xzGi8RnVTK5fIwA6taWUikUfe8sDVy1fOnz/t6x8iWlTLxqPx2IMPP0KB4AtPP74ymzDpLM0Wxg5pdmhT1LgQjB0kIZEcsv1Q1NJ1Dwy20Vix15smnAKQZRamY8NnB4QZ0JqtW5sbgaYago6E0IWVAsND6e2YzurVVTVbsdT+Pfu21mnVmDRaDX0jPW+8/HL3QF/v4LDGaOkZmphZeFmXyK/NLKkoLcumPQ6702Be34qEN7bB4ZHzfvm5597yEz/zuf/+qd1Hj+08cPgP/+hP3/PORydvOzJ/+YwqGEBqfOu733nwobfCoPnyc0/ZzNqJkQEgnL17d6ksWjB0zzz1FCc7MztPBHloaAjA8sWLlzGYS9ks4V8CiQz4cDgK0Tepje6+wQvnIaT0p9JpdLDYNxaL1OhQC1er0a2qhhXbWZR4DP4ss00Eg0wipD/yHF9BVBzTipEuA5nhykPjqaBARMiJgikU8ja7TfRgsxkjQ0EYVtXEEC3G884+d75YgmSfETsyPgrg4Or1q8yRz3zmM5R+ghJ68GP3f+ELX0DBlLJVijXBHMB/Y2FvwOogUaixnnIyVai7e2t2PdhPhbqX0iwtNOaVGuaFJFRlvispKA4r6hN925Fucm3KmcvlcGekyEdZ5GJuvuH8edt55c2trzrvOx87r2wjb+S3XDlvZSedFexd7ADlKJ01ImOVg/CrmwpYfsIvua3KeVI1oOSeOyfVOaOO3AMzcnOH/FwWuc0y10Bj7dqz+8Qbp/CTSJCTCl1dXUF2IyXJ3POsyX9zqzOgMcOR5aUF8G4mm6WSL0UWVihU0FrMX//8F3tHBwlrIBjp841Pgcat5JAqPmRFeHoJxJfN49k+u4BqYdg8/cRTWrO6SHelkmojH81lkiTUoGOEuZjxYLK04KbK5eqV7abWZSTvevrVs6vdW7h3p9ZeT0fTqkjN2RekHqGdqhDOsDpNa3Pr3LfvfO3rlEWRxyOTjbx445WXm/XCIw/eAxNg2aUu1trnXr/8wD33ui2utgc5CVi1RPqWBtpqm2r/1MCOiQN7dvf1dBMS3Gw3Izo9KrlM52gePTaHopOwLklLYL4YmpQDa2l2yoPgXisBUkpAGroaZXNlXaWEBy5mOmk1HhD6uWm2R0rq5549+Z0Xp2Mpy7G73/9j97//4nzy6y/9Ff0XGzp9OJaLCdFOT3eoh5RfZHOLrmvlYhGyI2lvIiqHUyBXyEMVXSXvGQ0SLuIfBBEhQtEu8mT5JAcXj4tYKCxDZhvgCoAzwAjQHjIhMCvtaqFX40aRzSJ9TXM47iEWGAOJvfPHhYnMlQGJYSxf/q8sMmSVEc4ebryX82OOie7kDFHGHU0p+WlZydRS1CpH5ieKSpYJKCoZRSvXLpeP0atcJxfOHuRiJeCsmCnKbhUsNAeSg7JH6mqYP6S86VIpc+XG9YBPRvty4/CDsaqE3oNLhdgO5e+g6zT9L+wupBuaj4UjUIRBQg3HlIiyHA2eY7QhoA9CFBiBlJ7iJVH/xZzk5gkTEPa+BXUs0KFqiV0FPK7uLh9Ki4IEiFJRwAAKAEOyf+YGvEJ0Pujp6aO0o7unH7rmjc0IeV20L9y8hWIZEjhMBZ1G3D6KiPFMEZ24qHR0sNpotuQEiYBMLIXny9m0w+wBawcptLFmbJaAgIHKt+RrPH42tW1HU1AtwnQN11U6nuKasSh5MnLfxRiRB0aIHuuICCIjixU4BWTCjBgfxN3qdDlFLQNDEEwFCAtqtFT49cLmQcWLWCFiDhK9Bytu5PFy/+DqqpWKqZnp4vZWuFroRidHoc+CYISN4lGnwzQwObEVgay4b7B/CLAx6jyfoodTmolGjwBsDiIlYjNJGQrUS8ww4h+Iz1ZB6r1UBiqIiFSYzZyP02JxW43YlWcvXQfyPXb7faXTF67PzKdL9YkH33rhmWfOXZ7ZMzWFhm4BZKJKqlikBQI1Esxo9H01kTEOT/YNj7/x8muWavnB226ja9vspcuAhwNOceuknkRn8Pq76Py4YweNjJa7xiaGhgdOnTyxeOYlTqAwfeWuhx4c2Yqh4m575IHw2lzIbSlGl8w61cyJ7clDR7OUwUHv5fSM9o9Sdx3YOa6qFt+hbk5fu7i5WT985AAylAaFKGC6ihK1evXVV0kTcr2gBzDpDFYbzBq4xuRKtuIJxCsE4iIilBknI18Z/vIkRSIo/rDAezujX2JfvGessoHICwaAohVgSEfUYfgLmYwB2IqppSrHYhEUpM6pdboc2Wg61Yqb7DaSBt29oU/8yn+4eOECzwjizMcff/yBe+9797vfHerq+dSnPkWrDABitIujE6e7NwB1STQWn379AhiErYV1s8/Bozt/8RKjrVlPM2/EAGQSksVR0rtyrogfqRyUiB4GLWNQsSW4HsVSZ0rLOJWxKf8q73i98UFWKONZ+Y6dIRk7a+TeyL6UUc0wVa6cW9EBgnKHUb78dXZ+cx98urGwH97J1ooClt9xN0lXdhRwZ6vObe4cn94V8IbKPsWv4hr5kudA3SOpBFbecccdx19/XVVpUIvIVsSNSrl8LprNpbJWu5nZxMURLYitSpWwzknVIx3w3JnNNDXDVodrc2mVSiG3C2a0SqCnD2bE8uw8IdLf/u3f+c3f/I2Zixd/4Zd/2eV0/vbP/lpBmNOLsMcTBCkhUEpwBzUUzIOmUqKQoZ2K5eSUudM8lEiVICxVPNGldcOgoxbPwQIIYUE+GQPPC6+UlNZVaCSo6h/sgl+dMjhmDzHf3uHeRx59GzPlqcef2L/76DOLz9BKDIzk7IVFYJSqRo4BiSNjd6puO6K66+7AnccGQl0MZ3yMcKkat9uRnNypCmVYIu0R28x2SbgjoZCAxpbehBAWMwiZRcCnjK4Hj6OFfRk/XrxeEmaAnKTeCdVo0diC12eS17aqRx/5YO/gQ3fe89FvPXEuWlBFaTVXNfE8MgVDV2jiKG25i7Wnnniamw/TEdWc4tKJKGdAkjRQWimgrlCWvMpK8Q15I3Yrz196+iHmFEdOZ9Aa1U47CEq8QWFvpYaQShJgE8GgX5crtt0eWtty4iUuDERkE/jRD1s4RGcw/bAN/vF6sVlkkstzZOHeISxlECuaFRlPuhcDvaNQuSjWKzpVzAhFlSoKWJkbTCWlfoqNZb7xKhswfDuAZ0n3ooDxfYkzyP3hT9mt7EeCPLwqZ0FsmHOAp6CirSq94Trt1YjNykDDT8AgJbjL3ZQz10tjXtBPoPvAShDAJxBFza40GeFbCXyQhW8Xa1UIEFC9CBHFeBBhhkFEFQWGPMAMzoYwb7NhpUMbsDcOBSkVGqgmwUyJ4hZLOQwF8I1C4Fwkk1snjxiNJeEXBMxcb2zQRR7sKOqEtiJYl8xcbirmKmqYjiyEXdgfDSEWl5ZpXwiDe/H0aYPJ9tprrwUMVZcZNV2jRo9MrKaUk/wr+W6Tzu3wqo3uKmLf5W3pLb5QL2UQhGHrVgDbXCnJwSoWJxeDAFJGD3QiYpRgI9NKAgODe5TNZ30+uP0dhXLJabMOD/TAU4FhAQKNG0qpNMICYY6IogKZuUC2nFHMhCEcbSYglU0xe/BeCFxTwwVRLQb7jh3j+s3trbXl+x5+59JyaiscQUUYrTYC0UQFZAxidGF5EslQt2mbjINOUJrhhG4B+4Bca2lK6lKRXAFim7oqmi7MXLloc/t5mN2BgMpHw8OuBhdBG/lwamxid73cJKH+5LPPveMjHxzeN3nye1+/du3y0MjgysISAMhYquguqyZ27a3kSpnZ6+7xsaF89trFc8lcrul2SmwQ6IrXtmPn5CvPPOPt7tVYbK8+8+w9t9+eiYenTz4fCvko+XBmK56uXppHJV981gyLJVTPhga4bSjmq/FwcHzcNr0UIQW/tOq02SQNX8cHCgDiddstDAZyH/fd98C1q9TaGo7edufvffJ/fPDYPWS1gVfhPGG38WAwOCAXJOAGGI2oBUOUoY7O4vaTte8Yz4JNbEDDy2CXsBMyl4FE5BPRAJdUZ3IS5+OhyOzE8RUgF0XYPMAa/jEasVTIxVLpRqW5vrJm8JiZcBQ3/cV//7OevQMf+chHsCav0D9x3/47jt2+tQ0v2+riwjJEq9DlEvA3OMy4dH6P+/a778Itga0rtrapc9nKsVzZlENzCAyM/4lHKg8UoaHYCaLTxD5mujGMxThEDtxaRJPygY0VrXpD2ojEEXn9/UVZc+NFfiOC6cbGvOH+iB+qeMJoUEXj860IHBZOq/Mq91H5Yw8clLvTOQDffv9Iys7ZTETX9+WfHI+P3HPJmCBDFFeHNWzJETHiaQZ82x23l1ECTksNE8qoKaXzcmAdTDhGDgCrJD/n3FxUQDQgr63DzliDRYIAVbFOZ17xW4noGi14WOg5o9GMQX/qxKlHHnnrxz728f908fJXv/KVe+68yzPZDSUIOTVyqARB0tEMN4+cD2xliDc5oqqJnoaxspQqaq2WphrSxZaq26YOtGupEmd8/133kS6ZPTePqYQ573DoMxkBNK0vR4BhWHV6l8e9uhHLO1N0ZLn98B27Jg6uz661KnqPxxleo4lIVyqdQXmNjqnoDnPb7YMHD/ePjXqKWQChm2BN3S4KGLAq84g+0AMtChG5n2IUM6oRGNI+XdpeGGigaVLV8Q80jSKstchi5BwxVFSgjAriq1KbroiNitp0dTr59PHV7ZTlxx77SCzl+d0/+brLP3Xm2rLV6S3kqla6HwZ9UOE++d2naKRWzGaESoPKOmI3jDsZi6gZdA+tMpgsKBWmjxKG5dEyTjgzgxmznQUVxuMHZweokdIvVAeAHpwsakGDPi9xIzI7RUjaw2GCSB6dwV1vFCggEe5f7uiNhfHCXhGkDB3yMuzyX6V/b+wFKYBqZ/SIycpZyaiWy5F9i1pF3SrWbmeNMotEv6JEWS/bcFCcJ4a7TAbRrIqi7WhZkqBKzBkFLPYyryT+5CvljnW2l7ukHJe9Mdq5NbB9llQ10oQ0Cuc7NCmRASDHuKFSQEFkSbQGT1kL3YTgssVgJT6BGOMQwoBDJo3Otdk8MMMGFUlFmhYR4EDkYe8QyoegATAhdL104YE6Dnui2CiXcrVSHsWjh98bVYxoq5ZgIcal6/L7uD12u4NKJEKnzMZ8sVDd3qJOl4xvV6i3p28AT5J5C/mDVFZwZ/RCjcz1QP2Cy85F4AXmCmWdpaDSW0i9HJjctb48j3tqtToyyajLwWEh5igCiiI8nE6lHQE3SCgy2AB3vf1WN1WnyRTn0CS2S7K1BgZTQj4ykISehcyr2DFGIP2kV+GsluBlw+d20X8JlJq/ixCZKZ2K0SWQENL2xia1UkanEyHBEVEbFC1wZzDqrSaHxeMjRuCDCJO0ow8uEdXa2noyGnc7iGrWZmbmeoeGxyan1laWW23r2bOn8olIJpMgN4bXJfBxGIhImHG7iexLyReoCG4xlqNGC9yWEYCB0yJrDUyx2Uoqw0JrBK3Y0tnS8Xj59eMGqxcvzeoPrm5svHHmdG+oy+Hw3nffPS999e+PHdw91AfwjZGARUYTv/Jw/8CFmSV/aHjvffetACRbW4TdKtATFN+/XI5RnG02a6qV2cUlRsvs7OzR2++ZuueB3MamI9RlPXjo+vVrI3uH8g2tZ8euB3zB3//tX+93m9rNtNVlTOeQZRro3Y/1jtzxyKN420994xt9gSbtrdwOE30kaTRZzEKjmSU+ye09dOjQMy+8TInQ3Xffffbs2XvuuWd5cR4jDwemf3CQPqXr21FGMlFoJp34s+Itig0l0AeZfxiFNypZmVeM5I5pRZ6ecJgEcjBqhHQCC0kI85AiAO2gaAAWTeSA2cFGWIEQEWLdu9zuzXCE+ZrZTJj6rVura7/3e79rgZVYp19bWSUT/NJLL7FH+LZCAyEiN4C9IGJLJRPbkcj84kIgCN6qNgmd5r6D3/z6N1AtDp8N0mDMZQ4DaROPrpMuFY9HLED+mIk8FM5RBl/Hqla01/dVL+u5bC5d5i8X+ablhjxSVsotEf9YCQbINJKNlZVIGYGjsiCaWIvbw3okKh9401mUbxVJJvL3xrFE2rNPifDLCcgiP5FX2bTzEXGqyFd0LqKIdeLNaQUkYbaZiABdvXqF7W02a6pMzLNl8ViZitTLVnI3YtoEOxx4iLm0YJfg/6H+Pl52+EyQn8dTWaPXHuwKmsUSswwODNEeFFBkZiv87NPPUdr2jve+b2xk5K8/91dwSRJVssGRgSUOGzGUZVwtVWFlOVE9kRlOrKUb6ulPmjJka9Q2S6lOp9VmO1/GGjDarZhWhHb6x+lulMxF8ebrNB1wBGzkgHQtTSZMaC8FW282lrt27tq1k0sTw1Oz5+daRWNDWwu6DBQEm82qnVOqtz+qfvTtu0fGCFaH11dPex1Wj99GV8xYIs3jgazGaPeVsnAmWPG1YOQV6c+dY3hLM02Dpq1vVdW1orpeVFVyhmqBzjhkHmu0DeMCmhAxgP1EYjcgwTYUWta1qM3pv2dwYuKTf/qEO3BgcbXa3yweOPbgNbprQ/9gUOfz9HSNl/MZzLxmtcKjE5dXnhTzQ4wtecMz/b72FaXCnzx0AVIJRlXGEJWUOG1ESJUUL5BD1DOP1ReAU9GFCUbYjAJIHbHPoWGryaKvaLL00wO2zcXLHv//WRQ3nWviDso9FJNTUq7KLOEalQHJaEXqy0zrjFRJ80nxrqJ3JeLUKTTqbIMPzRoFe0WyXWRKR3Pzyr1SNpbdsjfR5SRrpfEqukWaR4J2IQlqwLBWMrWsJY7MgdHHklnFYof0qlik15pIILUauY8AolcCnjL6NpOH8gWPVNpMMRnk0Sh+LQKRjERn7ssh5VE16VcrTKcFFHDT46KJrYemRnBE6cxGmKoo8GX/NpsTk1aKLIizCTrarClCNFnFX+zv6Z0rL508fUapfRShQG0G0RXFsEPCUofjog9o7+AAagZhQRgT94ilVMt47cZYJGU10zXXimBAumFl0dconSsTBu4fH3eb7LF0DpoR+ukyhMRjQpaIGYcI5JZKgJfELTY7lyh/PI62tkbehr92Gz2KQZdNpyn+hYJKWpbqNFTzAaihG0QumQFXDv8+doOBGG+zZnP6vL7uaCZH4R2MeNx9uDBj4UQhl19e2vaB5W7nIsmLew7sP3f+8t4j925vrxpUdTojA2pjpFIGjdeFOFZKlwE2UMXBewnqcW7Uj2EoiMUs8F4qdDByCgV7Vkg3bI5oOlajB5Utse/wXZgyzY3NnQ88xNm++Owzq7PTb33wHsC6FCA2y1mTtr2+vto7MLy5sjS1+5BhZnH6yoWpyUmnz3nx8vmt9Q1MKE/Ay1OLJ8ggJKvalCMQ3I4mdk7tOf7aKx/eucsxMkrMHsq7MxcuBodGqluJmTPnsd537d6fWp8mqRGJbOwYGUjniw6P9+tf//axt77XaDQ9+MhbZi+cvfNjH42ffMXltMSi28wNZCt83curKwDwQJiTBmaeU8fSyVZ0w4Jid9DfEIj76fOX6IqcgdGM2cUYRihIDoTDqiWaqzA6sUP0rExAnieTQkn4MiPFXWCW4DTrRSdIMrhJnaLYpmyUztSgXkBoOhz2dD7H+W+ubJs8ZubP2z7yKJ0cmUOvvPTypTMXUQ/hlejYR8fogcNJ0jKL4U1EY+7qbKad4alw02h4DMSdN2PjO8CAPfTIw+fPntu6smAJeOjQCVqCtLqcJiYn+QTK97EjWAAdirnOoFOmG89bWToDVqbajTVK+Er5SpEsyouM3BtLR5Pe+KCs5kz4yKvcEkWP8pGfIVXluEL0IaXSfGQbllu7FdnLfFVC0J2fiKMg58cXrJBv5Z+bpyZyja9lH7KwLSv4l/+YuNSiLaIJ/C7oxvi5nu5n9I7FzuZpGojHya5q5XqsmLZZcQEbUOKgGmiNUM4LKMTrchicThADxCEwBMjE5zK5D37g/ceOHP2rP/jUqeOvk+bv/vEf7woE6N/gdXug6SllSyAHxZ+j+KMiOh4eF3INCLHoTGxkcHTPnXtefOVV8HfknFqcjAXIi4SANzfXQ34/9yOXaHhDtlympLMSFzMWUgW9Re/1e+gTU8iUeHJmtwv68LmL08RdwI1ZTNV8mnbpqgfvVz322MCuKRcELYnEnMnQ7g7CTlUrJNe0RlWgGzVsTKUR6mqXc5zWqco9LWnUNPQASIzmAPhKAT3d3rSVbLuY5ZdAJkRoSepI5BZDBYPckK8bC2UTPV3zjZ6mpi8cL25GGx7/PqO1X6WPbUUz/WOO3v6eTTq3LK9UCmlyvfRCw+YiwEa2VrR9R4WIH6yMRe490ASmkGLkMhwYABIPZRggOxGQMACazTgzrAGmRpsNi9UKvodbjWBF72YyKYaZoKAzaYybvmxmGWLNUiFLSTKGGKWxRoEWMXzYW2fk8g8DBv3EcOl8pfyjDHlebqxUvnnzCxEw7kdnHwxURhVDjgsh/yrJWgJSSsGuGJt4twxdkROyA54xLn/HSMei4YMiHjgDiT/yngvgBsMfBdIKmYv4pRwLD1j+FI0oRxXvWeYMj0QsfDp4tMv4n6hMkikUVAJWr5RzMDz7u4J0hCXbQmSIP1KqOn3TbBEWJ5rCEBswC2Iax7RC6oRGaolUPhano5qYs/S4Y4cg17DQOG8Ox/nzigIXP4L5BmYdLigYcPRqOlwxvGFJpSySZDGlOzSnI6NJ3i4SiwGyQNoSoEK24rIwmlCIJM/OnTtH9JJkKp4Eyr1MzEnb9Lgt9BVIZiqQ5MOGH45FeVr+YCBPOLhUIhVHt6VirBiO58QJzpawUJQ6b8oJNWQa9HBcDk7FMgRmqC/SX7ky7XfZydjZjebp6/NCPCvWRsNqc6Xgu7EaYbMmrkAUGnAuF2mCwctAQVEjmUqhjRRoJSEh3PQaDDRgFmjJbtDqEb50O4E0G6PDbrPBqu10dQF6s8BOlSZ1COCztQDdzPU5cFfoy0Lp6sTk8LG779hY3WLweF3WgV4/89KsDV4oJSm2wRfUGckSGvDCYsnUO971TmbD3//dl5yQtWvVWDDEdgiZI9syuSqBF6vNFolEQt39CzPXe4Z3wkxNa/Dqzr0TQwN/83f/0Nsd9FnNj9x/J0702TeOw3w5FPIdOLj3hW9/zW+3UgABQ2n89Il9u3YUiKl7HHDK+oZ6PAOhpaWl7UwGlj7P+OSv/9RPHRkdu+1HP/T3f/ApOERhK1s4fZqaCoLV167P7dl/ZGl+dWT/0UwO8pC8yxfIx9cS6dX9O6YI/Wpq7bGpXRV7qnt4/PSp07MnXz26a8eV73zr3OkTP/axD9F/5vT1yyOjw2QEGQbHbrsd0itIqlHqY2NjROxxS2nn7lJr//BTn2Q80/XQ7/EWq5EWYHYsVgIE2JcMZaYbFhPzRFQy0ZcbWgCJzFTD0VR0TANcmbSsq9D0ojgy0h2J5ogvKJNRg+rN52vDo0MLy0sAIn7h537mL//689uRMHkfOg1DEc7IZEaycTab09k0v/cbfwBGmn4SlNOgicuZsiPkzsXTtqAP7O5DDz2EAZ1d3j76y0fx8K5duQLj29bSaimRouM550BPTBV0flRVOawN0o/VGlFxpLnYVYLalvmLdOOIivqTULMIBGSNTD05ZxHRyr/MRBZRispVKytFenTedF6//+0Nb1WEHfvnW9lpG3ybOIW8ubX+xtE5IWU9X8nx5ZTlBBDHQNXS8NnRaLndZhYMDg8988QzijTn3BmzYt12ZBTuPJFprV784LGpidvvPPbcCy8w8Xkose1wNVOxuixFmM4RwbinZJ0IuEJ5KKE1RCNNWVRGUcckLmqZSNTmdSPZATDmc7mxkWEU8+5du/bfexepgXqx9OIzzzFJMa+Sq1FbwIWptTA/19sdIgNdTZetPgsX6HV7cTzsfbbTJ0/TegvTAOehmayoXTTokg4tNGb1OO2RWHjfnv23337XP/zt17VtI2AourBq9FYgMIVciclv1tvoEmSsWd/+6KNzMxeW58+CZ86lVPc+qPqZn+wbGzWEggQ51trtst9ngXkXbccjpUUESHDAoBqD1WztrtW7s3m3y4HQoAQiSltRg7YE9S+QD3K97ZI+sp6p0azXYFdaUGRV2rra2C406yarK1sxbcVoez85vPO+tfX2s09cTlT08byP7oHgdwzVenfPAImxM6dfBtlQowtrmYZUFbgjSDeiBjg5dAZngaTD8wEexYNlRBAMFAXGY8ayxUKQdIIUmJBxhGQbvI7T5mDi0f2zlmMu2IDQDkKZTo+kaITWeMRTmFagfYqpsu76TOzuOBwFoUw6RSc5CvRVDSXr0BmYDJHOMJYDM1wYtTLg/uULg48ByasoJwnPoH9lFslIZmcytnm4MrJZUL1iCyo6S1kpAQDlG94wZNkPp8BvlPd8JIUM1QZ6t5P0BQnLU1EUsORgZZQDFRZFwg+pDOMWEeEkUsmRcQKI7lot4MANqHlp8EZVKduSsieOjX0jFWWCNaKGtUQ/8RqtSfXFMl3kCDtL6T0RYEYjBgPTQn4lPneLLtM8GKX0C2AfLqJQnmImEB+VxIWAvBjDdYqJ6CmoK8IQbQ+GjGge2p8wa0dGRjjTuYWlYFc38UAcTQKPeMDgDFDMgqLS68okYiVwrgJnIWnnlgrpnM5lh0dHUG/b4Rj+uttr5uj8MF9uAIWqaSp1fQMbAN2vNhihJ6oZ9FMjuyo6ZyGZzKTipPfAxQQcNmLWJ15eB/iHbQTiA+enkM+6nUaoHBEuBOyN0KuTdxIMNt3L63l1zWKhq1fKajNTvAWaA2ZrbCJ6EfGkcFkcIEOcTmvaDolyIgPc2h2J56SBOyrdaKYcNBKOZrFpaFBGWVNbh7e9srZlvni1tw8QWujCG6/DxW0xaihs0KoqDuq2cE4ZQCR/NCay8dOL85T5jk3tpJDXLg2H3EKt3WwaKLhWQbckgVOGMDqYwo5UIqbOldUGe3J7w+ML/Jv3vevE66/YnTbiRfsO7LMYNKO7JzcW55751jdh9ynlcjMXz4/vOUCt0cb8jG9w/NKzT/h8VkfQtbm5rTLrnJbA5YXZ+4aGHnns3adeP3PfwgK5AnIO2GVnTp8ixN2Tz83NL7/nQx8+dXnevL7dPTRG2DboMN55dM+1Vx4vpsOTjzxkuHKtoTb0D4/G4omBweHc6gIzfeeOHUZd++mnn3bYpW8x7igahBCi3WEDofbCCy98+BP//kt/+OnWdfEmaSlIuefdd9+9srZRhNZESMKqKFY0iKQPpKcCGVWaI9C3hqGtzCeR5ExleUWMMKHMOCbk22AF16md9JbRqugWwExhD8wLBiBBTY2hiUfLsMbBeuaZZzAHu7q6QLzhkVdqWIG9ly9dkv2pVYTiy7oyWlPgW4ASKw2H3wFclolHPEJtNz/z1NM7JneOHNkDgeU7Hn3nq6++ymkNTIyW8qX4yhrtiwlWIwiYaZBXM9jAnGO80yEIHjrGIZqRsc3sZJCLPFAWrqWjFDsf/5Gby3f/yoWd8Au5GEW/ygNQ1tzY+c2PRO6V8JkwDnJ7ZRMSZhTb1VrcKyQPr9jWLNRu3X7XnSePn0SEKaE9ufMC2OHCoB3CdwAITEYjEV1YWoKHgN24rdZ3vetdzz79DDUweMAelyOXzuFFwJJbKZU5E5Gmis3BSSon0LT5nBia9z3wwG/8xn/96Z/+6VPPvUj/LqoBU9G4CKx6IxqJ6LDsxfBFdGAZeBYjyc36uipHRlV2Uo5UYpoYJywCR1uKhbcz8TwXZXQYAl1BPYzTZuP8pQupcNrk1F+6cnlxbhUbvZjGm6AtHz2hbQSrpACC7DGapJU78raJ2/b35pPHl2daAzi+P67/sY/dFwmfpfkBXek17TKqAR8SS07Ko4SaDSlpok14rW5Xqd063aBJ363X+/VaZJmpQbq3majViyR6G4Xm1tya2xqkCjSeiBWrJZPdqjObYfXM4zFn7SbHuH9wOJ72v3ymtbWtK7X3RaLltpZ+6mxmJJadzmfCse1kKi6BZVJXUmVEgS9CGhgV2T0lbwPVRp3VpDYB2uJ6MM5ED4nXyUMCga02MF8YbzhOvf2DrE8TeoxGafnQMzLCc0eriX1DJ1oCmqUcWUjGFM2kXW6bLrxdT9Jte2SkmVhlRNcaqXqtIIMOHSX2HwOK0YKZ/L+8YJyKayjDl/9kkaEscVlFj3aULnpX1KToXuYTX3X+GF7KZrIlg4OQskQWULoSc0Y6yE1D41KlKh4w++xoX155rxxO8bNB6AkmGW+Su6qDuAoWKCL1cJBDXgiEBFolsmjcPpkThB0wR+UmC4cK0VzQVdAjE1Ygk4nXVeA5lxvQJxOwkUIkmm5gHPM7laDJJYQrC7vB6cabkBwlGUmGmIlSFTOgafGwGejoepKtatr18UDKYKHTCLlA0IH3Rhwa+YK5opjP8DAnKUdjD8x6zo47ZjXhJ5Bo1ljpHY09rBIdzDCxWR23HzsW6u6hiHBjK3z98mWIfSS3T7pSmEbQ1yKZ0atai1MFAquu33fHPYjJk6eOJ6PbS1e2dvb220DoBb1JYqPS0lCfz0OSZUGsI+u4pfQg4uyZZCY4Gmz2RjUBGDyRynHCJIJazTCOr6qFejWTNICms6gpm5VcCIYkjlhbZy5Wmy4bxkQ9l4UOq725EaYoKJnG51dZaQKo06TS1QsXrqAyd1d3NpifdZjTKTOqumA1thBnEJS/3uIgZE6SenB8HDaP5557bmLX7rmZaYdZS7sNCEpMRjMuAti3Wol7hSquOFymfCrD8zda6qdffmFzeXn33n27hrqwVZc2NjaXrg/0d8ciW2NjI9fOSauG/VNTwB+T4TAxBJM70Dc6Bt+W2+s0241zK0tIml279kxfm2WoHjx2jPHwzSeeuOPI7QTkn33y2Uff8pZzZ886Mo6e/qFsvoq18Nqrrz7q9m5srA91+yA7hUR719QBVbkNQbTWX5tdu960RB588CHvPXcZgx5VNjmx690Ts8NPPfEtf8BHbAZNxtMHNZBIxF1ux+nvfYdSN3waRg6WNWj54ZFRCZ9YHf19fYzS1c0wgx4RDRoOA11mgxiEgEZhxGWSyZfMOIahmHnwo6VrWHrE8QtZZA88wIHtSNQfcNBPDYQgwgClQqo4kU7R7AFdDhm40+ujsbzb6/6zP/8MnJQw8H/8x3/8bz77eTowFjAtMb6QI/iONMek22e5QrbS1e1lJsQS8ZPPvEjW8LajR7/9ze/88Z98us6zL1R67rjt0Xe8nejo4uzc+vI6cFktSY2As1quFqpF6s+sVhPs1Zw4k0ixS2TidfLBCBMJArJ3oCCSjMC4EJLhf73YEuEn8/nGK/tX3sqBlFUSdhSZprxyT4U1gQMx5zkud5ir5j14EaL0YHBYiXVJXGRtY12I5ymvIuwnIT0GspqbI2et7FhyR40GgQQCBjyWfCZL8J9AC1NPxIWY+7A+008eM6jMQYnL8QSxkCQYQJ89xkcLGG8D4OPLzzxLQ9V9uyef3974zje+7gsGCB0hCkFz5RMZxcMQYFeFviNvPbR799S3v/AdlV1lc1sLG0XfmJcRlQvTrkrEIVYbM4lnRwSFCqjJkWGCdg/9x3/vtFugAVk9s+YcA8YF2hh73WSBhFzdQqlhijhcqvERuqA1L515aWPxpYFh1a/+qvO++8ctRmTVud4eSkAZlzxJCIiAcei5c5xTnR4JePMaxDqd4mxauhQae3TGEVUDxE5Fp6b2JFuDSpUO6vlqI980Gcw5hEeBea31hPx1Lezu1UzZUm55EzmbRjpR9C0sNS5f3yoUKQHq0hoG4OvgRhfTNBOkN0Qkm4uTPcZHUpSHiDjgXRJwFTO/iftEETGl8khQ0SJwRkpEmocnI0S2xVPjoZpw7pgfNHJIl/MFHEvf8Cj8rIDYU2lAMnF2U8GPK+Yo9cCe9ngd3cEgE1mXzWjmZ+MTkwM2e6hQWESyUUovEIwbVpqiWnBNJXgsKGEMFTn8v3iReS5jl0Wi44KgUPK7qFDRo+K+K8q1o4+ZN4qW5eCiekVM3PhWhiDseDdSvIDL+bm8Kphn/GAJOwPfEAXMb/ljsHF/lHnCSTMlCZxC1qAzWKDXgdif2Dd5cZu97rBbUCkQtKIqufGcK54iWh5VDQi5rib70+CP9C8uH/oYF432juUq+UWxmfAGcTUY/cTPifDZJJ2iPBZcD2mVqSLTpTGSscXzpFuR8GBxA4gu6fWsRaM3Uok0tEHceZQ31Z+4HXYbicYrVFLu23uAuU1uAhbeWDxczJe9TrvGYnTYAFkYuU7orZwOGwFnSplWV8OjY4N2V5x4dXdP7+GjR/bt2/f6q68TegeQhE1RgkpS16KNg76lqxYam9E0HYbgr4Jj0i2Bvkw6snFmYymXi2qgbqb7aLnGA5cMUynP0+dxiA2NISgxBDDGZjLNaj1ep8Xs1Qf9fjIGWxsrEIxgUoDnwnfXmDREmWnFIzVv0jSU9iXYfYH+wf7N7Q2cA4vNiXxBtxvNehonSE8VsGIqOhi3lpciMHbt27cz6LQCO4RWw2ZWMvHNNjyaJqc/U2n0eIJrm+GHH354cX37yvTV3u6BYjJlRY3DTsW8pgiwThNiMGUCymu28uTyPF6guHrqjHPb6ykP4XYD0Xii6pfObwzvGL9w+ZIZNsdEIuQPLSytwnhQaSSrGj3OK3fbaTNNX7vUN9DHo0R8uj0e8u5PPPM0zOx+T/DQbUe6enq+8uW/G+ztQ6redfsd5y9fvv0tb51bWkU833vnseuXL+7atXNm+vLkWD85iVxVfe3khWyxecfgaLGlnZ5ffeF73zBUynffe/v67PV+vToei5RKhWGALl3eCxdgmjrj9wWL1RqNjkEPULpGSsls7iU9jGmEG0qTSpcv2NPXL+JSZhADnrbOlIngaIg1jfkuswgBoLzhVUxFgmjCAiEE3CAi2rqSRH2aLaym977vAwsL14AFodsy2TyamZYHcEES04BMA/WPnwpu7pd+8d//6q/9x+PHj58/fwEs2PrCKjsUbV9RldX4KVKACUYVlyq8sWmm7SokLS47ZB0IJoKlWJkf/eiPUTFMN8knnn6ChCXO9OrS8huvv1GDlbyUlVnM2GvDXI3FKmKQPy6EGJoiXjqOr7wyIMWRREyI2BEPpfPmXyyubmwo0upNv+287+yK43b2zBvZTGwMWdgGuSmRNSUWTVgO28hpln4hpLFYeCi3HTn23NPPITZQtVKzAdpbrkdsdeQgl0X2HewHYbNAd6i8sc5u0YX8EFkCRzMpHJXVimKWMjCp2kAlE6jCqCUoDVE1rFbt3FbeNWgdHu07/uprvT0h7huF6v19PdViQSq4QE5gkyHCuQj8h0b1xedf6OoOGgMmAt2FWJF9JsJJd8Cj88idJGtLnphEvPQPoEmpTnX50hUiabMzVx+8/563Pvq2L0W+CKTZ4fDZu9yZaCqXjMLfarO2yT5wZaU8OC3Vrl2qO+7W3XnncHcfPsCazQoMk3A7XhOsgkTjzIhBea78Z8CIMVDNSftWZKhAZLVQ5lItYVbpeAW3qqkVG9lMgdQ4dckNeooDaCHya3U11Pp40ZAqqWvqPkdgT2JLt5VsbkZNmQLkIvpKI6A3Oetqq97sLBcqYNMSya18MdJu5sBIkT0k/yVOGyqUgSN2K3dJ9I9GL92XySHKmBM0BTdOTlXUDCIbkG0nEiMePF9Sr1SBX4lEXiDoIwgwu74OkgvsFe3YpQRW1fB5gGb6u7oCTrtZxuvWZun4a1eGhr279vfR7w6SMJwjFZMWddPRXzLIOCmOyRyWwfevWrAiGKgdBKZcljJlRLkzEOSjZHnZK29uaFxRujf+WKNoYsYhKhbbTsatqFgxBhUFLPobcBB3g10RjufOIU94L7Oxsz1ni2zBohAXgNNXuCu40VgCqARBhkvFA8EYSrqU6SoDGWNGbFl83zouFLaCWk+fd+ixQB2JXwfhBj63uIKyQ/44rhgEpJbF9xVrgvOQk5d/hWwMbxU8CxIDZYixDv2kmbYPBlMqmY3FY3gpDpdLZhrsRbQJ8nhoq2B3Ulhko/EDjLugo8HT+fooAqCS0oDMohMwwwLQk91mQaRubW2EQp4dk+MBiV038Ic8Pl+5dEH6k0iP+jxznNOjpa44BmpDslDZYXMPDUw20rnI9rKeOiVV1aZtzF6XxpbMcRoWwVnN9LA6PMlU9sqVZRk/YDSM+KiSBsaOhc6FBilAt8YnxhhYDouJhgogMPMULVbKPAIAs4qMF85XpA6mPCVdBosZAkg3fFxWuJK9VhCZdps1T28L+BkEzwzDptOuw3ohAwobR9Btxz4lQ0NSnhiChXC53dM1RBu+7mD/qMHswIz6Tzv3ffYzf/b6C88NBLxdHhc6pZAjKEbeXw88gEeLokmlm2SX7NZiNZsmwhZwGNulNFGvtc1NW+j/oe0vwCzNzzp/+Li7lHtVV1W7d0/LSPf4TNyIkYRgSVgsWV7Y3T8LLH+WkMBCCAkRhigJsZlMJuPePT0t015d7nbc3c/7uZ/TM2SB93oXuPZMTfWpI4/85Nbv/b07c/nUy6dPUfS1NDfvd7qS+UIqlu7qHLC73a9NTqXOnq8bLb6uDitLqVzp9bVfm5wKrK3v27PXYnV2tPeAPyCmQiLj9hN3zE5OvXr+1cP7Drm8bnwU1IyVJlQdfkLZVlDjVhMYrqO3nrx88by3e8CUzy7PLwwP9nPnK7BHqZoLN6573c5GJHjlyqWDBw8szExmc3E08fHjR+GVunx9IhymfVRsx47trBkQqhh3ZJvo1kxlITB4fE3WJeud6WJ9Irpl0WGNiI5Qk0rRk6JX6hQpoiaMTEE8Wxt6gESyAPLD3+UlAhSNxTCudu3dc33qMowlzE8K2isTjV4IgWDE+Gdm5v7rf/1/FpeXr16/Auz5P3/yd+677z5i/4qdJetEFAqmH4udjrcWCE/hLkKAqorJnN5mdNnsoY3AUjrNYg7dWH62u2v7zm30J6Ae/YE3P/DpT39msK9/dNso6JXQ+mYpnQMoAXswuwpDlh3LE+ZWESaIJsxffgR7jIIRYSFuAr+J6BEZFl/z3/3gaG98943nrSf85qZaNg02JIK4JYsVggXoLCQdJnetA5xuYU+tL6/rtRSzKhaEclCZD+WBfEWeEGekeTxkYaAWyIgTAEO17Nu+59qVKxOXrzsQ2ACe6cVdqFgsepxNRF2d4sYCUhECxgZWFcw01nZ1PkddZL67y0M3pOHhHnJbgfU1HG24H5gRGSUIxcQTkk0RWY5SAozsgkXK1MmZbfBnJTcTZo+DkJTbbsMKrxOegApDo4MpAOn05je9OREN/PSnj5MGzmcLequRxlzk+mDkMxobDrME2ZCLrCUI7u5/j+rEnV1j4x12KPYbUais7TYT3cu0BsQXdaB0WQFspSdnRF5MT/kGuWRJGyZw66WaBP1RAw2eUFk8qmq2QdO6ZDwbpx0q6Q06Megpr6QbeqGmD0SBV3rNrm11dd9CwDO1UFxcrWxGaoUqrdmdcNBSepvLFlc2llilhVyyWEw0a0Vx1OqQA1XxilhFBFaYVpxaMVMUZSeRB0aN0CMqhPfZny0liNY30jcH90roUTmmfE+rGd+xx+VyMLTRcCidjLMAiGpsrK/Wq0WX09be5u8Enup1EmjP0WojFdelUo35hfDExPKOfccsJq+qCXgDFFiF4ZO1xyyxl1q/lUXz+mvKH/8Hv+R+FIUqukhJ4sq2ETWpuKqKSsbRRzrwCpfbUp+iekUTt74rZruiqokciMJGkbz+m4+hkiUtxIZH8d38ouhIUdWEDXgdwkHZmpJf0EFzDMYYfS0bCzRBWQXtlMWszSu0f3I+zDwpLSUaD9dio8TRDUbBIEp3CCDGch9MVgkThYi1dKon6mK0iIrRE/aBzl3KoVSkcCRUIBEKYtPUl5DHJ2wEbUWligKGkE+iV+L9VfP0hccWJT2nNpHCJGdHEa6vrYPNQP6AWj2v23Xl6qXZ6dlUPO/FfVMgTsQGIQihoKhl7/MxTjd1Y5I4JIz/WGbLczPIYp/HBdF+xUAzohQcVeh9mjWY7d6RngGL1aHu6tblMtX5bDISyEcXK9no1tE+5GAuVxoe7BocHke9+dp7QpE462h1bT2VIKIo0RclHlNHIiCmS2VaElSikXjFJrWJyBoWJXfOxiYabDBbfN42r9/PJk9iuxd1mVx2Y3PT5jBCAkCZl4lkqss6PbdMcTYuBKh2TCDuS6ut4AtXc7kaHfQwmFk8cEPCHw1K02If3bJNOzCiytcnZxbXNgMEgfqGxrsHlprFNEqF5HRe3He2BDlv2PMxqLgSzAUqgXXJaLCaNRvqpUx0XQsW12hwWMz33XdPslQH2X7k6G2P/uBHfOnEXfffuHw1cOla/9g4MTiaOIXj4aXpGz1dbcNj29od1M873Hv347fYg9G5xaW3vO+DV59/EacZY/K+933w0nPPRWMpT8cA/MYkqF569Mf79++dfv4pj887MbcMx0rfyDgmFLAoUqeMf39v597xwQCgzHTC73Ulk/F7PvzzxaWZcMAKoNTtcQ3eefLpv/2S3elZXJocHhvDcVleWadNL6ONDQkFR0dXz9Xrk4RDmAKWOopKTGZWrIgQ9CFWZsNmM0F5xUt4ZrwqO45Ui1R21YgvGK30UyrQqZLOOQDrzp1/DTJUcg6YouwXLB92B12eOH4ykoCb6fYTJ37xF3+RguCB4cHXzp1HbcxNz3BwomiyY22oqCaN3/He2MxYVL193eCfaR8WWguwjGx2B1g+a4/9heeexaajdk5VKv/4J4+ObxsLbobWguu0ggEuIGIGI4INolXZLLYy60CJyiryQcnKsceQjIq04Ak7Xb7CEhQR9u98tL6LfuT7redvHO2NPzk8sWeRNMLliSsGagIDQewDgjl8kUFgCxOKZFKodhP4OmUzvC9WguSwFScYVw/JL+KIXBFpwkJBQ9CId8lgbWxsoL+pZYQ0oxwokrWSe9LQFMEqqhwejgoBT4EwA6VhB6myqp4tVujR4C5G0APggL2PVYYPIERQJfrvUM8qvhRRIpR9w6XG1t9YXUWxlOL5Uixv7fBxRrLOzWQlQelRqgHpNwE3KCmS0VTVYv7QBz4UjwevXnwtFU+xw9Ci0HeD8IN/zUioCfBcHvCd6sQJ1a23uvbu8Xo9mCnhOiWXzbxKR2ORsp6ZUZUJtpLVg8WXaAv6jQaxKo0NXGetkVVrSLiUKI6r19Ll/AbFkw183SpB4/l0Zg1yuCYcQhU7VofFblkNxtCyZudOnXVrptI+OVe9cD2QKVpjGWLJLgsN4rR6VHYlHiUXEA5FBeCD6lWVCQcCoCF8LnJd7DecbiYNt19g9uJLCahZ3GHJKyKCBG/DK/hSWkI/4CpACwrFJXUK+BQuFx0OWdLEimjnjMUswp9SJtqSl0p9/b1tfhdeCgX2EG5XSpl8LgMkEz4ELCPt7PTaylzvwLAvm2U/Y4GA6Xp94XIpigJ+/e9/279sb/m+qEceyiZR9HFLU/JbPoD2FWtRniuf4Ylo6JufUfzJNz6v+MQcT35whZUgtsRv5Ouizm9+CxOY7UDIl4+xPKTAkaMS96B4RUepFhWiiAmauqh06ToN4+sV2lqxdWm7TY4GxcFOEG8TgIxgFhl1HZJfT2UQuplKikolZ7EAXNTboN5tShNfWl8Ryc5nwrhfEr0BJ8Ip+A3tBtoE+1+pt5LNipYBcaSYINBuFPJckwouj1ojbOdS60231wdsEjEH2oXfLRwW12C2QEEqRNCkNoHI+1wuBjeN5wLsJ5cn4cTOkcLlRnXqxnUWlI+9V2oY0PpNe11TpTyKLBReKdfs7x+EVqnUvEbXGpLRhWSEYrq+DhdomrCmarXgYWQJPo+M9hktTnhWj996SHNGNZNf4zJYWIwzuW3WHBsJTs7lpVXcVhWoRJUqGo3ZrCZiAqQtq5UMCtjucOHug4Gra/PVOJxhjVB001WxgXhsM2hBaPnKjZHR4c1wnMpjhA6h7CqxBownHAaC14WSAWYso1XEnBijJJaaV65ObKUJqbfr8Sef2bP/EPbTc8+99PFf/NjX/+pPGTEIXJFQAgUqFtKpBHHmYjbrsBM5oOGOuWHQNyvsihx6nlCezmyfm55q2Owjew5cffH0wPC2bbv3L12f2HP8pN/X9d3vfjeRzgDIg22vt7e33eF06UzZUMzjdhWTGSphSWYn4vHOtvbQ3Nz1yRvvffd7sJpfe+mFg4f2/+M//jCXy4wcumXl+9/FJJq7fpmgYyCXxip67dKVduHaTXW2tbms+nwiMH3pLOaLy9ft9/swjRdmbhzUNr71rYc66DjodRKuP/X1r4PRO3P+Qn9/L9TQYAWAQy8tr0LdBQbPLCRrOYwPNCo0XkgMlMfrm06mjAXKGsSqw7EQhg3FQeMzoiNpF200YBmQY18LbiCBCLGATXvq2Wdwf0fG+pbm1/QmoR9nY8D6i/nr9vs++5m/+JWPfwxdePiWI0TI8VY4DnZA0ybVQxyZ2jnenbuxWM7XHA4Ti9Rlb1CLASDFZCTWTsFJWucyI74HhgYwHKF4c7d1nz1/Ftqvt7/tHWsbq+GNgNPq6Orox5Gfr845IbQk0IUtoeg2ESjKQ25TyniIXUqURtFqcu+y1f5jjzdOwWHeeN56wm/Ozykk56S8y+233pJ4vkGLTZOKs/rSUF6gm0mFiPcklr/01gTjzKSI3BOkj2SOSQgTq6KugRqGyckJdj1jsjC1sH/ffqaGmGk5WzPadVYH75hI+gK7JVkp7K5khrCLkVPNqq9bWyzkuzq96AZUHxJ1cGDLC8+/RPESZxdJjjjFYwYGjZtBVbfJSiXlO37uvYgamkGUUkmN2vDJ//x7mFOLC/QsKkyfvUIL1CpMdcQzyL0ZLO+47U5bt9cmMC2dTWfau3XPKy+ccrpt+VQMTrw2n2rfXjit2g8c6Orr01lMqUIujPayUsSGqi5nqQt0e13Q40iaVZYfsrXNpG3Tav0aNQ5lRU7VoDC9BDklVEXFRK6cdWRTYKPAXAar1TR6Ulu310t6+loEM7XliK53YJ+n7daLE/kzlwOhtCGeN8Fqp7WDtrIjFUGuSY/uPMhqcr0FKLsg+Ed5gLFCK4hjq7PSZAFHSepdZSqJmDGkTBRWv1AJSBCV6WIAkXs6yE7MaGF8EmaTXBcjQSs5Vr7RZNrcXKXHCD4ioK1iLE59mLe//5bD+8mGYUep6+VYNJRJRCvVgjRKqpZ1aZRxSbW4uHnx0uTA+B5IP6iK0Rsc4OSkgIi1xSr5VxZxSyezhJTFJ3ZB65Wbsyxf46FYFOIz8hCLUFJP/Am6Cf3KK2wbKXRSFiV/K8tXiSQzCOhlMdTk0C3jnQ8rYS18ZRLjDAtal5SGHOqm3lUUAwfjT3YFBYhoCNZ2OgWxAdT2bADQbAwlQAbBbIBzYqzyOU5BBIL8qLRfgB0djYJnLGVLXD/89oyyqFMtOkAwLILUJWhN4K3A9PEt4v50Emw2s4w7TpvsMW6KEUGjy2/JFyK5RI0LcSMRb7W6qBXGK7oY1jS5rIoWAGRFspTQFvIOt3dnX58k88GPUoNfrbT5PFjQBJA8DuSrhJLgk+aUiFS2GdVHEDUQfpSQkbQhMiViUcxbl8OxML+g0ZlZOhL7FnmEc83/IA9VG+vLVS2oYSM0aOpKrlnKmps1WFDT+QJ8mH4npO2xVDzk9h4FGUG6uZ0SJRtRFwxnmTiM3FKxQddGm93IwiQC47K5k7HUQF/n9EQI7c2yJe9FDI3RZjUKDwP2RTjtcHZ73O54Ikg5TSqXtbhcEIBUNIb3HDr66tkLc7NLdIYkysGoG7UqutPl07lktUjhlN2OXOceMG/wEXKFevDalXMYwLcf3ReIRW45cuvOraNDQ30YFzi1aKN2u6/d6wkGNgrpBHdPTzTw27TdVjUKTpMRKEQ1nYT1BsMI0LBswnLFYjCOD/T/zec/9+uf+LXx4cFrE1d333vPe/Sa7z/8iAGilaYqn0xvLq/7t1ub2sJQ145gNH7t1MuA43FAyRZAQrJ753bDQE928urM7ERPh++9H3zvqRcvuWbmQAtTrlzBFsYBqFdGB0bPXr565szp97///Sa3ffPG5d6BLWdPn0LXriy/2tbuh8Orkk995Qufu//+e6Ox4MbmOvlmVhhcHz29gw6PL5LKbYbC65shcNes83xuFg6HdYDWPb1UeIfCVDcmlLyMsu/EaJfmviBKsIpwMljTrEO2hjJH/CuNganEsFbsZK+x6ufnlzu6vIH1eP+o/8DBI/MoYIMeOQQuoKuzkwBdKpcgrrq0tAxdF7gwstEYhbFQmG0JnL2YKbn87n0HD0kyxfkqgIDFuYX+/r6tY1vxCL/xjW9iYJK6Vtpn5SACisTioBhMbpZQ1ORwUCi5sb5JjoMwO3OXiiUB/bJucVikHSFYMbi8uGiJfkngC3/yDeUnjozy4BUeN51hEUD/oUdLs75xiDf+ZK9jZyMf2NwIXXld/qQMr2axGrkYQclFU+LiEk5l4GTzI7XFn+LDbA0iehwWIQZjikhNJUGbC6ZV7epyuoDZyt6JBCNIK1QCh4VoET5t2iUoe8tgsJvIMkqjF6laomkoPiYmbh2OUvYWWLmnn36aM9Ry2E8qsosSkkcBYwDAM09Le7WWGsGp61Mf/NAHYH350pe+lI2EvvHQ32Otkd0f3ro7shGKrwXzcaSkSmdt1NJx8tC4JqGNNUjZWRPD97Ud+a2P/MkffMnvUW3bozp2tO2O2wfHxx0AktKZRZ2uSvE/4FB4eLBBHW0D9XQ8F8+JDiP7Cn0QV6Cz6i2dalU3CkJVjDeaGSg4K+VsOU+TtxRpolI2oVNZ2f3MO0Ec2lgXypZsWhvP6sJVx3rSVzC2z0YLL51ZDybUZmcH9EjwFhClJ3QUXNukwTlOEDkKFdXV5KfpTkimlxGXHcLqYYkJEzoKAW0rkUtsDXxdlJuEClhcvIgvzBpTZoliGqOVvBIRIQiIYNlDyzCDsLIn11c4DME62BLZBK52b2fHODSc6VSMOo48HWGKmWw6CS0i+ctMGjKkmHrY7ixW0ocOuwaHGr/zu29yuWL16qLFVtBocjDscpVcJmcXTSJBEjQS/iCrjotCD/L7pg6WsZP/xfBkQ8tClCcsKHGocStZBBCTQBkKlpsfHHfpyFcV9gx0n5ISlv1CD1qgcewrGRz0tKB+2GYK9qeiAVQs9J70PahIkIeINEykr39eiWlLVI0zyyWDDaJ7ET15CPLnYSIRQm3e0hKZkxvidiSnTHySgmBUoMCpeBE9V+JEVdrOkPeVKy1U6VKpow0q30UI8J1ioWq3uQKBELcAqSNrC7QPSDgxNWigxo2LyJO8GlFfyGYIc1Kzh3muCAsZIcYORUzZW6nELdfpG23ADcIWqqt6Bnq3bt9jMDgYja1j41QMX7hwOri5kslESrkCB4NNC2MHMc3gI+BofgIkB2gG3R1gMJAG0EwW6xQEHp1jpUX2enxjuVkqOAx62Ob0dp/a7Hb0jmRrhLwdsJ6vTF8uxgPl2AaTEU3T0D1DcpHeH0AFRsa32lze0eO3vfTw44FQ9PKlGwTYErEs/Vq4hoMH9jpsGBAJOGCJjJfyqUalhKbRaZtuykgoTRRDhEijk4i6E3I5ugXShMLfRSZhdmEuEA7cduK2QrUq5MntXaz2c6+cWZqZKyTYdWWPk8PUYJsGe4XJD9SQDCheLWZHvlq1+zyko80O18i23blyMxhJHjh4y/PPvzhz/Qap1m0j/avz0w/eecfElcsAUE1WJ7ixrt4+pGEJm9RpguiqkAibjdqBgSEYy7LlprezNwtQS2ewcX0DfdhXZKYf/uGPOn3t5Vzl+uUJq8lOHHt0qKPZKNEsAYvH3dWJ+Lx0/Ua+Wrd72rr6+rP5IsNOwbfdYjx/5pTX06Uz96VSpcGBnmBoI5dJ3Pv2t9byoJmwarTf/eH3BwaHbrv3nnOnTg31D5w/f27/1qEXvvf39XyU6O5maLOntxdRyxIla469QjLO7vZwIzqjBVq7QJAOyiPCpVAorK6u6vC59Pq+3oFAKBaMRK9NzmbyVWiLkLdsWrEuIe+mqJ1lp5jK6AY2LFtO2WtCJGaxmQUF06wl0+VDx3aze6/dmOjo7MZb+9CHPtTTP/D//smfUUnx7ve+b+L6JC44AofqRqqqwQFGQgG2GyUu8XCc8hUCsuVKee/RWwBIv/c97/2zP/30r/7SLxNTyqWyf/AHf/jlr34lFNjs6OsjlGd126kqyGF7xmJYQSxQtgetzcHi0fiDTWezWEM0hq1WXW4PlFs9/rZihlKmHNqGihcmNJ+vUAnCspRoMDY6UgZbA7Bxg0BWueWecqcikV7/zRP2Dr//5aP1MV5vfeCNj73x5H//CqKb80qUi4c4BiLxeIiVTzQCC5V8GQ8ZbOwXySNKypCroT6bz3A6ESgIeSE0VRx2RWTK7uIHC57UFQWrYtHi8prgdScYoIH61U3YI+Ht8zGhQJve/o53kR345t98gVIiPLFcrtzu8+zatev0y68UEjWbQ1dIQaIrqlf6rsKYWxWmGhBfxRqVuuYGfG/5/PDOvgS0a1DyQB/VpMu20et05VJx/EUQNXiPRjvs0/p8iEIolBVeS53SQJtRtXeHCYTMsWPG228bwOvVGVMaTQYng/ytxWbETdLU6Xhk0YOlUhCu2B3kffBhKmpLQwshbr/e1N9QtTFGunpCVYpk05uJ6EYmFoWRQkUKSmNNxosWm1dlsGcKjUSmni3i0tiyze5Ly9b1lKVSgQKSrWFOpuHNMJLIY/YxXMBalfBvcPk4NKoF8LnE0BQbB8WFgsH/Q2FQcITzS7iSKDTgONQYBp54KxW6tBndPjUYoFLNZLRp9ZKjUWWLRr+/nTgDdme1CnwEpjLM1lIZ5Egc0Q8anGBbp5/qOSv+ArxaCEaGCO+fPwSWmk4RiIbPQQfOCCdmbi7V1m5/7cLMW966PVeNcCTpMiBlOCTSFFtNCmAUtcqtsKEJKitLG6tEUWXKi7LGWitXPiRqkN9Sb87SYp0JOEkWm1JEJD4iP4q+xLHlM/I/38eYFcUtupt1iXZmtESOs7h5kSdNxkayWfzJGZSPcbCbjrT41coWYC3jQAJyo0qHyBvKAKcQDESFbIiMsrjVXBn7QvaPjr5pQAmVQDkXw0FQxRpSFVqcKTUNSnU6pYYrT+pGDx5PqwlHQrzI3kgnMm1eX5xudCSppPeAsnM4AjcLEKYmTe1Q+MSnuQquA8EH7IWTsIBxivGs2QkMDWMqqGHA9G440RzAEyi9Q8XEIJ2LxRPxKEkRArEGUMn4oU3wTWZi4j19vQRa2cnmEcPczBRVDuRayqUiQU5wzpyKABAilfAIe4osDjB4a0lr8ZtoWDSyYzd1pTSTX19adOubHJaOe9qGjjBBKQsNiDDWJqMxlMpoPAKrBAcpF6rXJmboT8rdcZ1ICG6atohKIoWwbpU4DRUC2KlaN46BLA3ili6H002fRautoTYtrmz2juwgEWh2eEb9HYlsua3Dn8xcMxiTXBLh9OH+3igRJitZxrgoGwCK9Dyu0yZEUu8sCfALIAVTqTD5sk49ivZye89QYHXqOzM30J5KxF8Lco0xfOrJx+g/mEmVUP/FmmY9GO4AmuJ2ry7eQAF3em20QF7aDCIgcCsXp2cdnnar3w8Convbtgunnjt02/G3vvudX/j8Fz707g9OXJkyibuoD24E8RyI7dPncG5+es+Bg4eP3YKbbnb74Z8iyTcTjY4M9LZ7+u64/TjlrOWGB9FApfWIYziwqZu48CpLY+fhg/HAxkhP+/iWIYKAh/buh+/6wTe/Y/L8abB13V1bSAF2EBgol5fXVgdHtixvwFXrYq4XV9eVSn8baYTh0THIlkmdoIORNcTBTBqbtNilEYykIdm8IsHZvWxVkTqEj5R6Fix6tqmy2eQ3ggYhxPAKjKxInZra5hAPjTppio5AdZFEuHDx6uTsEj0cd+87aLbZZxcWQUV95s8/u2f3zj/5H3907uyrpOwnzk7a/SaoKOmTzeay93qvXL7y0De+sevAgW9897seQvQbmyNjWz/9mc+yBywuD5cTCgb++o++ePrMK5DMvOe9P/fQQw9xF6AZVq9PdIz3bU7N5GjHFIlwtG179s7Nzbk6PIHNCJRIbD3Wdh6Ubb1hlAIG2r7iFks0U/Kp7DslScyT/7sPDGkkGJIK5YrAkAYvCBa5ADskPxK1lA6haF8kFT9EPRGFWihF1FCXSl6M++BeuHIqAW5eKpMjMkRi1FAkwJbHl973nvcBrvzWN79bjCed3R3pUChFxSpAJ7NhdCfbqOfoHcdR7xcuX4iHNu64/RgxE5ICTz/5En1yjU5tLqlYAcoJSNth2ELbzgJJJPMGKuftDoPFtby+uDizJsg9gHjdHahhbowmuoVkmu9ZbVjY4hbmw1WNReVzmHPJvN2s2rPDPdBrGOhX79pBp+ya35/XapKNRp7CIitAaC2UOLT8JZVvopEvVidAAiSCZONE4MGZYFM1nWoVDH1mfkRq0lA1k0lECjQmLuclSM7aJLTQ2T8YCJc2A4VS3V7VdKSLhkiiEsw0V1LucNYG2oTObF6vva0PjL0afgSALMTJpCJI8bqYAaSZJKjRsri/ij3FZSCRWTLsFBY8thD4c6aC9SR+LFql3uwY2hJLpphHtgNpwxy0JCZbx9gAmTwMLLJdSEmcXfj8WXIQ9wKbpSxFwtEQ9NUryQSxbvC6wHdJHuZAp+aQR/ksbQDEp1TgSHCP2aNR6kYqc9PB8l37bNYeuuNJNTQXwmWw/LlcWdbADSRXwZDwP6KwZRW+8USZ33/6Ja/zMZacaEu2uth68kRyt4gDfjiSokjlXR7ymy8pi5XfyivyYV4XzS3fFtT0zYO0DiW/ZXGzFURVi/ZVFDD2DYQ+FNiWddShooSVmyFLAtpQJJM8AOmwbcCeULCH2U5oC9MMySUXJeWRYgXlEpBOgehTA8mlDRKU6BosEzXbnv4uduLVq6mMzyMFSMyg2WyoCCW0mANcC9ej3KbcKZsNtQLwgsy04E4Fk1JjQFkZYgsQ5ZYUrxW6KD94G4uJPtGlYi6+uRHYWEmFA3WIaewaQIksD1BhwKfRBsSaSOTj2pALRC2/dv4sWEeq62D5oAw3m89DSiTJPdSz6EELASMsFZPJYbW4hrdspSJ9fT1An1e92pBMp/XlQq0EvkViD0SzAb9QDYUQERR+tZnPZIF4E2wh1qru1SUTWRLPFy5cGh5oc9mhGi9bG3r4vCDKAEGGL4JAQbjgubIeEamAqiDEKFQbqXJpldhpIED4d3R87Nxr593+jqPHTlw4dw5LZd/+WxKh0MriBva+yemulLJEzCX42yzxCsudnQvcDRgZYflAPMq1omsxpzs76MTIVsntP+C+cfVKYnNFaRhQLa4H+kZ6KVvQmR2UcRfoGKTXdXQPxENrkWQW7JVObYKnZnUt6PG2F4v1fCDWTGYtL72yvLii0xoJwRw9fHR6aravv395atGk1xTy0aGhnqnJ6XKltG3HjksXL+OYHrz1DuLn3KrH17G2Ebg2MU0m7+DtR6Mrm5cnboyMbnc77BdfO2+zGI4ePfy1h77sc+nxcbORDc3wYPLSebfHZ8fcIeK/vEAmcGkzum3ntoWlxUw22z2wJVMo4/i6fH7A+TR5bevsMlnsmDbsr1w6BQUwOx8hTvofAAhPWBgYZ7IIpSC1tbzEcJHdx0CKZXtz/cuLPFh8zSblbGAZ0d8IJQgRY4k4LWapjuOwANOefOIJMGf7SWvv3x8mJr6yaHK5/uZv/uYzf/anH/3oR6cmbyTj0f6dfak4yNIGHYCp7OLB6Ul23n7r8d/54Mccg53Dg0PpeIJ98eY3vxXD4uLLpzzDfRQj7d2z+wdf+/tvf+ubhOi/+YUvpvCNDLrNuVlCH6psQW234GdQQHvbidsnLlx2ux2FVIadi21XLhM5JylkwdHkZpEbXDwjgDTg7Nwr8pWtxVD8X3woY8kgKna8mAUi36DUICwszZsJenFVik0gmw+rXupcqAiQyLXgSUXYtD7wLy8SAZuNpdt6O5556mmny3Pk8OHFdt/0pddEVaHLIGglGYaVTAOMK1fJ5ZD7x4IhD+Vz+/DAX/7pKWQXbdzUVlq2GAKrQZxgBqpQIACZx0vTmjRGhzpbjiMCrU63zqzHOwN8l6vmrV4TJM/QJDi67dVcg8wsADFEj9eBJ5lNBfLcQUev6q47nYcOdENoZbMVTMY8hJYEJiGmoAZQRyFvGafHBXkP88DCoxgNUxDHmlnCEaIiVKulTMCuqlJfWmjW6Q1czUcjmVQoGYsUimmkNdoRL6VU064vBasqX0nfTlfuSMpC8XwoagxlNEk6KQp1Jkg94QJDeKcTabrAIcGogBclJ36eKB/pXMRylHppGWlZMMSV5bkoOPxXZLvWYKNal8gk1gdMBxRJpjIFalb4dhza84ama2gMbm3YZZBs0EkmYDJQo4Dh8KO6V+u0ErGyOGy8aWIHUJCAzMMxI/WZSSTwfYmB0aGV6JtoNVkNYM9Al5SqIA2Dm2WYKK9dWT90e2+zsYZwxwzAphNDAO0rZh3bmL9RK3LFytJWdDMrXf67qY8VHSgbXu6s9RC3tfWDESheLLteDiVf45Otz/EEpYfm4gmfQcu+zuqM9hIwMzqMyDkvos9aylhq7tDvkHQjW0gG8xlZ5BxWypDwKsU+LaK/IUeExo4tIjuTTLDUXSAI2MOENu0OI9aqdJhXwnGtSwXcRQ4Yq5o/IaEyob3QTeKqqyHrSSbhTnOhkTmrwwn/sNTPAUIhy8aMtx6vH4f7FKuCrSgpdbi2yRJIEYg4I4r7jXMuiQjeNWr1EFyAfpPGGbkKadLNpQXqS9hn7W6b3UJXEkAK1nA0iW+El8OMUJ7EZANssXvstO0rZIUfmCMTbwd6RxCPZifwwuTKNWOzadIarCarVm+lvtnv655Zg0sf8y472EFbWaIiNPirQfnFjZMj1xgF5EzyplCpwinNrs4WysQDWzY7w8gSYEaCwQidTBlibpMggbVhJhbDjSaE6lmUAveLFQbkj0QpDQn0FmcMUBUIH3eb2tcOL/Tyyua+gwfuu9d/5vQpqryW1gIWp7eUyzkof4rUtLoSaXkWIg+Ra8WillbuZj1BSI/ThaXvausgiUVDFri3rl2f3ne4b+vOXd86/8qWvu6VaNhBUYfJUswWLFoN0LbO3j6uClQ8zlM2VXEYnZuxdKkWZvYg+oM7i1YwTocnEY73tPdmk3kKXtv9XYN7+8+/coHih+D6is/tBe+Nw+fxuIj+0ogGWMrc/DJxNxhA+0fGb7v9JB7hxPQcGxKOxnKDDqPBsS1DO3YN/8M3vk6zl3vvujUZ2dDSmyMZmb/+Wnt7Zz1LujNzdm4xGNrcOjawula7OLkwMNBXUgcv3ZimZxl6sZeeGW73nXfeGYsnJ2dmkRfBQAjUPc2lCUEzMtI4vCozRWQY9ijOji4QKSPWHZtJUkWyOV5XwGxNZRuSYJLdAHspBjZBOkrwCFcHw1GDxQpe3WxlhTvZCJlc+vqNiUA4RI+d9/3CLxCC5sL+02/8WjQchnuEdFAuncvGJFOIF2XwmMCsUfRGdS/hes9YT0HqR+Inbr394YcfJh/00Fe+esutx1LJ5A9+8IP/99N/Mrpr91/+5V9SN/lLv/1b3/32P6BFuH7c+obRxHr+zd/8TVb7c889d98D9wVmlq6dv0wdCyKXK8UO5pPFYoUwF6uRO5L9BepCbF+hCWPH3tyQ/7F/WgLtnx8DXxy/DieKaJVwbBg4I+4XpwaigYOEhFCuBBWrCDrZ8NoaWS4qW20WfrMykWC4X4rdgDRTHsrcECtk5gxWI/ObS2RpfIVvR2WB+HD4bQ0K6swo82qxGgouZ9PZ0bHtSPjoUvSS03by5F1vetNbOto6TUbLgT37+nsHXnj6+R9874ebUxsNynyIWFWER49EbCoLGEplBbsNRjmWJGNR9mLTWAHLEKbOIdmIEmYJ3JrMGhv1FAVwc3rVSJ9q/37V8eMdu3ZjCoftNty7GLJZUXDEGuEbEvQrheAmoxNRAu9RTUPhCAEWVCoRGYbBqTYgN5yqsj4vPW0SIK44irqWwemm/AwbBYsTNHCurMmXjSvhkq+rr27sW0/lp+ZL2byFNHAyV0d8OxxulkEqk4tFotXNQAZXgXQGPjtLEf3K8pYIKlkRUb2sF3FsZfGLTpIh5pc4TBoksmwFTCScWyH8w9jXmi3a5PomxWGezl5KSBCDTG46DUDHQGYX8A2pYeo4PHR1BQINRt8kZdy5NJAjIQTEKCC4D49CMkq+Eqy6dIiR6WOJIv+bdR39BzJZFLA2FqsHNqvnXlnctrXPbusjqF1voFcw3tBYqGH0JV6kkt1lD7+ubpUnovPkwZ0oT+SWlIfoQhkAKSmW3c+3RRnLl8Qy53v8Ke/wm60i31KUqIzOTcdXKo6UHyED40nLm2QQREkIEEOULk+QzqhbFJkcDjOB3ciJGUP+lLuGq1kHaRzbRR44uAw9LDTghMQ91kHVCKIK6SS2huB5+SLzBhaainB1WVo20EvNRjTOxrmJu6ICSUvY7YadO8eXF5eI4nCPMF9BQ8Meok0Cti13yHPl+jEpMD4YCwQE+QZp8Cc+tgyn2GDcP9hroA1sp0w83SivQogW2wwA2S+Xcjadyuuwmw1NsrpOb7dWY0a/0soNE430KGY+XmBfTzd0/K+8/FI8ljObhGzGQ4WPwQj1IaDKSjpD1b0sKyOFufRsV5165SyFlmOj2+LRRCRCt12Cmt5YOMBiYAhw4xuUvOeLVgcIj9L01JR0ZOLRacQsiEem6Q1K3bLRIATu3ALhMjHYDDRBq6ezYCy5a8KHarOtTBMGpCfqQPhiLK79B455u0fgcAgkksa1QHtHNz3YY4GYD4Rxz/DGyup9D76VQpmXXngxEgrZPX59I0VVPvBziodBbhKfQBMzYGTHe4b6vvfww9Qgmi0uzAeK7U6evOO16zOcl6BcEgVgsmQh88/kHP52ylnc7R0PvuVtGPivnnopX9M4/L25SnVsz+GJqzc+/ksfe+Knj5fTGbPVEg1FPX7ces9rVy4dPnYU2jqLv2N4ePDEiTv/15/+cSoR8TqttMhdXobPOYVjanObgsHQmz/8K089/tQLL57+uQ984J09A88+/fi1ybktW7dB6Li0vDA1lTjyvj94fzX16I9+EAsu2E2m7mNHbzu8l4ajCW8bi2FtdX1oaOTWg/tINF2ZX94MxAa27rJ41XX9apZC5qamvasPbk44vwAo4TsxG1g8aGKEAnIfS5phYT0R9sA0wTLAEyaFhJxhHymQR8VWFroY1lxrybViRhLnwlEFHY1lCxU+0EzYptKJQl27msuW2CFQfxQKefh+GFhi7FC2QZw+Mz+zsDDHVGzdvo0UfjwaISHi6XLD+UzxWyXLtlEZ2pyVTHr2pYsqF1lsS2BmZfyjv3zittuf+MdH3vOed73z7e/49tf+rmDQ/sM3v+Vxubs7OudmZnlOy7Zf/sVfmqAF4+Sk1+/Dqnj62WewKvizeeCQ32GXOBUwVQK8GjV4d8w+ERxoOMSJ7KoqZDfsaKSMiAnlXmWL/d95sOsZBChMcMXZXZwUynDJeVIur4PLAtAUwlbMR2nuiIio1xztXvp147lSrMJeQ8xzL7gHIg6QLVynCAu8CDGaDDpNDkyG0wHwZ3FmhvdMNkupnENr8kULvBJWG01k8O6gXcWI93X56eoNNSkjpvQLoo+UFgrYE3ffA2vb5vwGktBst1LoyjjDxWv3uOidAPOGwW7ciG4QSlVhSdjttJxUW9zAlSAfUhmrMESbNJQbpod7VGNbVLefHDpx52BbVy5fmKYtCO6N3cMEaBrCzC+UFIBjuH2d2VJDdlIKCIqY0IDk6PG8zGq1Ra1xYa7X83QUBnWSogkpWEiAw8xjWapISIgYiFtlSrp4TpcqmQbH7gkmjNPzlRuzzXDcDjWOVtdmdqvw26EYkGBzMhWLJQiaM+56t7tK8JnhQ8FIpS/LAGcatBeiA/Er+0JeESWEtmDEEX9WAmyk7wD1kNSjwBdzMJdMCQGIyU6rVjAfJr1JAI9YOeEI/jQ4OoI1EJFS3Oj1OfH6CfKhcGvCvkenQeGUJu9LuheuLgoOiXCLSmY7KvRa7EpRwKweAs3E0HVW1fxcjPLuyYnw4ZO9qmSgqco0mhRvicpEk0vUAy+35baLZlW2NDehKE7Fl/+nNa7cnewL3hW9rPwoypLVJT8/8/obWpnXRCHJoCm+rKK6RO+K6uUHLB/CSLxhcaPlXWFQZZCVP1HDchax+bF4+DCjzVZkrHmN6xE130SMEC+VncmHuRcitqxp+hAyNxyGfYLyaFBQwQADkmJwCGUDXK+DHaWOFziABvKzuouQi71cXAQJTXcpNgbVtex0oIn0VuDgFMIA5RKYutBjiRRgQ3KphOU4LJYIv7EYWLIyOtiXICQKEI1iKMIPV4lp4lQwwaWOzw5+XS4e9JB0/xWc1/joGGv75dOnKOTGksTPzGezsF8NK7UcnAWfD/wBQwRVJkkpAq3gCip0yCE3iyXBnRgt7s5uQI+lRn1hfi6AAUGcGp9XpZXmHzcbpoE/oLWQWC0koXP50vCIzm7zYGwykA4nVouOckMooSDXtlqo5FP52jqz6QR2B+NfKEKxwLaink/4bKAaJglsbfP7errJj22Eg/QMYKCIjBEuI4O+NLMA9wINvCixDSwvd/b0UvwDLLmSKhAxQLox+JyRCYIxsaUJzA7L3t17YmkKMcpWo3l9eYmKqT379p6/eOm2u+789tce6utoJ6pZLJR6O7uSqRyAMn1HZz4QcPk7bd6ObAqju3b/7kO+zoGmwXjo6PEXnnl2sKNjZm6Wpii0HHB6vS8++/zR48fdNgecIfNzN+69/+7nnvzp6JYtayvLxnwBCBVM7lTj79l76Mt/+de33nkvydGvPfT1vr6e47edfFFisNq3/uJHfvTFv4Kgef3FZzZWFvfsHGcThjbWL55/1et2d7a5MZb9Hn/Fa0+GNteMhldnlxMlWqHYXjp/Gd7jg0fviMXCq2uLgWgUo5BrYzqIkqTSMVx7NgPOIiODAiY2gNphwQkaS0usXjH4Zc2zZGTxCwCBPwlgSR9n1pRYvMy1FMgZtZQwSvlEs0k0TXhPrXC11m1eIR/mywIm1OkKpYLkdLKZs2fPgOsmy475NTk9hfP3/o985PkXnu3v6rvw01edXbCENvPIoHha47Q23BoVIRvWh171xOOP/9f/+l+feOKJz372s3soodbrUQPXrlwthOI7IAkb3/rEt37Uv2fsO9/5Dq7zpz/9aWqvf+d3fmc9sHn+/HmSKPDRFCpxmHPoloNFxuIsIGcJ69IXRxLDYo5LiIuMNuEuNvbNlNk/yaV/9zNln/4r30a/4oaSVeXB4PMJTq144eLlIWZgqGSOANlikiP7yfruP3ig3d+GtSE6kmoirY68FQADERYiv8QkQigr3lsTxghqrzKJjKqZgRNGkD7pnMllrOFpZwsZjSZht5KVsTssUiVfUyUzyZmpWYfNmSRQr0R+KZbt6Oga6O7rHRwa2Dq6MjOPRQ6eS2QwZcPJjLZE+CfrandiTesJ5toc/X1DCbO3lC5lwjGvz6Kv54NLKTTR2JjqVz96aLBP6++ot/mSmuaGxZABAgzwBeddCqu0dGuAklbgNSxOvEPxuhBzrEb6HsK1Afi0YazXIcsjVVYqJDPZSKqSTdIMWa8uYTUV6np8JbRvsWqM5zWhhDac0scLVlN73/mJ4LVpaEhcGlNbDZsO58Jlr+ZBywPaShN4E90g5wbihZ6lpx+iVZ4RcCTYSWCERVGuFjDMZJxZ+vguPJDpUnJNUxOsRAcINAjkKa9mj2hMDgaK7lJOuyMcjpYLIcI5OHWkESg/AaRJTaLJSDcjI5FBXKU8IJp0FkishRCZSp2lopuQZDEL2BjkCnX55BxoDCw2AZtKJD+JCHajsBTgk2lj0WosUn/twsKWMY8JFx4yUHFMxUFn5Qgmq/WQjcuNcYTXw84/81xZQ+hRWUn8lm0uuVD5YVJEcWKRiP/5hvN6U3FyIt7lG3wVpBW2u9QZ8a2WMpZvyQfkCHi6bzxXvtJaunI2LlXyqgI0kT8VN5QAkcRsCNeQiNUYiQcraV5uglphdCzGaEVRIVLhj9biw7yMNGMU2/1AdeTGUfTYPog5fAu2QV9PT9ZH6UUFok8GkuVG/pVQUqtc7OYZZZBkqDggpHOyvTgKWpnoOLBTJRIjxhhcKvTWhg+R4nlpz0dYkQvQ0tzEDJC6WUVl4nMXqwAcTJVmWKM1d7R3AsFKNtKxcKzqcHBSKDiSCmMwuFE2PJY09hzhO5STMC9gadcFPobzxGVwuj2H9mNdl1OZ3t7ubCQMpQ4uI6NFfEmakmG76ImxZM2WBOhlhBotwBbm5vU66TXB6qKihainw2Eow7pTLrNAgX9729vTuTTOmBgZMF+WazY8FBCKwpKNOQJIrbqyOBcrzZqtJn+bFw86HU9F1gOxaNRmNCcA+mZzzz3+RCgUbIcptas9ElyR5migVpg19jABbNLs9ABVN3p7O6woIqN5YXkN79bpsQ+PjBOFJSBP8R/9Mrbv2bswO0NXg8X5BWrEcpWar6OzEE28euES4OG+ka0zk1OapvHl1y4f2LP329/7/gN330Wrzp6eLtwL8OvtHs/88krPQJ/H656aniAMSHyJlTI8MjI7O4u4QfticTNb23YfxIU9cuQYzuixW0/gQ8/OTidSQlo8Otzzyg++/873vHNtbv7G1Utep7O7rwee7ZeeBUNTnpi4NjI0DIUZCmxwpP/i+cvVcKChM33k478ycfUq2TJ2tNnhPrpt6+o/bizQflXaz9HPnJq03PLSipuk4C2HrlyZRuphFBHzYDfGk2kcDXKiGDey+15ffsgi/mT78UsR7SxL0cIsTjQrsT5qVTMEcs24Uk0gXmNDgxSJQb7PeE5NzjDFuUymWig6gNmUihtzM9a9yH1731D/jZnJe+88yYD4/O3/888+/bHALyxcW0TXCpKITJo6T/8vkQVsP62K1siPP/4YzsQ73vb2C5cvqlJlXae+kMtYfI7Ll14bHR6x9LgAVON/vPlND5w4efLDH/75H/zge4S68LPPPfZciB2Wo6U5QH/YeYU9Qc4CsIdsPW27LBTZqCmYlhJyuXE5qyzl/8sPhpkFgM3Nb2VMMWuIRsgtS0gZWYSDK0gMLWxVBpsN64AybBDghC6E7o3dhVxSk8RBuIkKF8mJZyy+EZFq4bRxeRwknbJ0s+BmadZLxA/mgBQCLUfSxmp3UQymqau9DpeOZVMsTd2YphaP4lQweUvLayPDW69PTkFZR6u0lYVlFR2+6oCAqPtAzmmJ4UIOFwtHamlKR1T9vWNvvu+tNdDHhcLijUtOU8FtzYbWzm8Zsjxw9+52d9nnQfuEDfqSwVhkZgG3JmJ1pUeThsAt+V9Jc1MJBZhTVaA9FzU+lDiTfdXW9WwKCoClbhN0R0lXgdAKlHI1Z9CSOi5B2FasGfM1M8i/eM4QTVvDKUsobUkVHKvPJAIJQ7bSbbC22ewe+nwRZaRzRyK4VsgmuGWGC/NcVK+UlqZpWMEQMR2CuW2pBFQfgUmAzcpmYHpEMAvKSVYIYgphgqSE/JB6Y8QfdG+UsY6MjMKXEI0l2VAIgYuvvgI2oqujjX46sB0QExUm/mY5R6kDCNVS0WzUOGwIKWOWyi8Ibko5PCiEOAUdOIaoNNG+ohFlotmTOl4DvWoz26DlINMZiVWh2Ls+sXr4FicUckDXKH9mlFkMLDNZ0yw3bkz2NkJdfHi5HW5bjsYzwr+y8uRvHpLNVbSmokRvPlde4QJ+9i2UcEvRoo5EAYtalZOI1GBdoreUPwVKzYqWz/AFXlR+lA+Lma+clne5NphsuX0micuTi6cGqA7Kk358UG0wE3L1/EDAzLvsE1obEHGWa5YrkcwRX+RA8I7yYRlRTB1JxmtQG9g4K3SOJMlar1MjQeUJT7BN+R5VQOSj0cdylRj9WqYSsQHai9JyJInsMak+l42JLqM4CRMFdnR67gpogJNzPVygtCysELOBbVxDJhUCx3SKqgr6XaZfC1xob++gIwRRu0Q6xanx2/3+NtoN4YhDks5aBHKMXFOiAYCS6S1Pz3g7rXjs7jYqpzLlysLsJLgdsmTETyhYBGdcyKSpQsZ6S2UJYGLlULYbhX+HdrPSNsrmqDfz7ORbDh+nCfZzz78ETqKnx0UBGcAyRAlMYoAWYMXKF4koqGxWom10bwUVRndkzDspBc6pA/0DOzLByM7t28+eerWUyXX7OuOhqESRGs1EtHbrW96ydvUSazoQ2iT/DOWy3eASKcG40Z1JBwMf/J0crUQOknL+qzcm6fTy4Y++9/rkAlFildn5+b//GsoH8ltnW7stGr08MQn2Z25pyel0Ewnyd3QXS+XNcJg0865DR5bmF+iUGEzEQND95IlHO1zOK1cvUJxHb0ciCBTLj24ZhstlM7hBg0LmHe7JQ0duGd8yevbMKwRj0Xu4C9BKJHLVW9/xc4/96NGT9w1Zh4Yjp8+4XS5m/ZFHfjzS437m4R8DRHo1nqDT7a6d2+NUtYpZUgZwR7p+36GDa2sbmXLR2eYJZfN7Dp6g9GRodJzbXFua3whFZ2amQJEDFUxnob7Vn1u/4He70GFrK6sgbPGiTBYHM56FukiF6i0h2BltVoWy0gQcJNgDNiArEqWsbFVZ5zzY1xIv5TNqzH9Q5VAjEQBEJSOUVzcD9F0oGSvbt28/ceIEceZvffNbmaWQym3UOe1Ly4v79u+FhfuRRx7BG75w8bVbjh1FYJFZ3HNs9y9//FeffPaZp59/oZrPNwFF0zkgS7G3AEe+9FdfNrkkh3LXXXc9/fQTXCobDVwVeobKYDxagAOC1VKrl5eWPvWpT4Hg+8KX/vbS06ctw+35VMZYbthhZSoUMinClbTZwMuktIE6BeKmdm6FXYnW4d7FDZUT/t994HYj5tjNNFXhTIqVSDgZ+gcCRbLZ2fSYLAS8ZFI0KnBHl65cNguKUmCcJAyQUYw/5jkTpDzkgkXAyb9IPTW6qqjJCI8e2VuRRBT+5QFP8S7ldNl0JZ2KWCwrGm1wbXnN4ZH+boszC1jAeni2vT5VM3xw/yEYeLJ5xjXFxra3uVjc+XgeTY5MpDNSMV2x+mG4VVfV6oVrN57Qfs9ls73v3Q+MdQ1X84tDvUaH6YjDnnXbwjpV1GDCK0saTUA6wVLQHo3K7YZeZ0V04dG1BDMmHVJBRVVCjXgVDrIk35pldT1dLsTK2XStmKUOCoogXpd9jVyFvS5T1KVqjkTBDtY4ljUlsq5Y3pHIOjJlR7pkMNnajDahPCKf4fO7wpHAzPVr2jLgfZyzBphVGTEuAb42bByJ7SPhRZzyBImKyUuxhoayKBlp3hMNgsIGGkvEji1G6iYXllo4c1t7V1cXxZMcb2FhiQWVTqYg0gWCY7d1IuGRTFD8NuDIJGtImqGYzqRZsTWW7tjIMAwnFMfRThH6SWoUihSN0M0YkIJopgakTIr+VNRxs6Hz+K0YK8CkGYV4Evdc5fXbz52bGtu6r73bm0qvAT/CsoSsw2JDUynBdDEtbz5YMfzBFgJFIO4tP/ypPMQblQXEHpdxkSAR61ScST6ADFU+jr4WeIJcGiKCb+Ag4t1iEkr5r0CueFWUP08kdCxvybsSaZAQtNT5tAZUEu3oY1GcYvHg7SOPULFIHTgwKAjgvpVgA1YOIMCyyazy+510Si3msyZqcPN5MqrwnTIWzJmD6t1CKZtt0rMDvnCOQrUHs0ztUJ5Av1CoUaTUkJZEGkMBWzEvBixoaqDS3A63yJe0hHTZWtwtxpjctVweNyVxP3g3mrS1pz6kSXcmotTcOxmQPCEYQx1SN8CERI9FTTdBXgjbGQsvAYeqHhC0iV734BQIhlPlzIN3yfmTxkNRy3zXG3SYwIHGEDfjK5TpxqPHR7TaLKR+oJXr76bIxZEts7BoLUISLbe+sWo3Wbg6KNuocmIas5ka5Ed46vFoHJghJZ42u5t606np+Y2NoIkeJd2GfD7L5NLwdWxsjD4/kzdmgXfBrlBrFCjt1puteM6Ewkip2M1GBKLTYQ+uLfpdfvrM7dk6+vJzL9p7esYO7p2dmaGObqBnIDM3Db3OLQf3rW54sZJT8YDVYQmuryFYKZYgGdnv9bHWKZsnnH5jasbj8UHlSIFKPpu/8OrZWK5E6V84lUHwEa7Yd+DwxuoaY3LL8VuXFxd/8thje/Zsury+y1euwfr29g9+6IXHHj1z6kVstV17d9TyqXohPTo0cOb0K/liwuvqRb7gsXXmeylFu3L9EpYNoWy85Hq5MjiM57qJUcaWO3HPPV/9u2/FvvEtT1fflYuXjw+PjG0ZvXHjKsrprQ/cPX3hpdkbl0+/cIqkFi7PubOvLSzMD48MluspCkvGx8fROrY2P9LYoVaP9Y/tuu/nfvvXP+X3eYLBzQ6/C9uIFYRGxGYkVowunJ+ncXKegujdu/ZcvHStt7ff6fYxAoQ9NjaDMHWy0mTREBkFMMmWEatYHq1dSZ4SKw6hzyIna8W943+Td0E0Hzy8f2p2logoNg0x/6H+ocvnzln9Hm4EI4Pj79i548bLl9hg1FPSMfj8+bNXr15mzYBmn5yd+dBHPnz2/LnNlc27774Tqc6OZvHrXQ6qJqWJB/rAqkOWwd7A64888iOjw4rjlkonQTjjywIn5LyqTE1DMofGlDo1mGqi0Gj097znPQsHFn/61e8Zuhxt3Z612RUMMr1F7k4q/6owQ5gQ5VRZkl8h/Id1IwZIWXD4StALR0ceikXSekqyVZFEysj87C/e/tk/Zd8pjzdebz15408UrHyDPa7sdD5L1I0fbHAug68TdUIyCd8yZ0TW6fLiCpYriVK8gvoSRiAi8XqQWRxHvF7lwRdF2nBy2knZQGUifq2VYko4NjCnjYgaSzlHc1JDIVvBvlmYX0Im4eMlgxWry6TKUAhFHMcYWVlDaX/+819o93dQUkRRLMS20NRoMfBpqJIsMtjVdA61Vo1XSGlSIgRpwcrkeZE3xedP3t6/f5evy1exGtK0xTIZOCN+qlAmILS0GgN1rrRgxeJAiMgIKEAEpAcbFlprOsAxyuKCVmEwraTChXS8UCvWDXUdMCqYn9VaEw5pXmBWTfLRmYozVR9IFjyRVG09XA3EdJmCvqqxqwxeT3tf/9AW2PGbamiIojOztEG8TqG3uqoXT7sV1GWls1kIhQu4R8gNQcHBsiqKgRXD+BK4k6ZVBIHRWBKW4IF9QDKQwAnIF7XDARbF629jslC6BH4Qbsw10TukAXBot8cJQRBq1eiAZL5B0DlTTOM+ULji87sJBNIijkUOcJecfz6T4pu8KlzZnFRRfsyroI2ZXHGFqVvDQjZIOAS7gp5hWHJ0X19bhQon5vRCetZRKi3DV2CkuqRYVYwwJZiLEud+UHFyZ2i8f1q1//SMRaQ88PWRBQwC6kf5UUwB0cjK64rdonxQ/mS+Xn+OYhVgAtMJ9gonWDRuK/iMZ8ibopxktXKA131f5U/xaXErRf2j+JgTKdDnH3lToq/sD24EWkcF+IX5yfQB6tWC6ecazcCtQDMLnrkJF6MIMirjMG/wcdlYckyMO7kwSgwkl4vS5htAfSUqjvokOcoflNsL6ymnZMoRoHpYVhBCwmQmFhB7DfOMd0iLUDOnxffGamDacdqhFiGDrKk57KT/cfpIZJNLIdFlsWstpWoNEUkwgxg1YAUlUq3YHgwGNgd7nCC7cFJaCYeTOrbqMdCyRK3peQB8g7ATaXQQrao4iZz2iRuLm+sbXoeHWnCqDmApgrQINC13J9uGNqXQcpKcbqjb/e0h+IpiwJScgK5BR8fi1JLTTxDCE/jUEj20ANJpcZdXV5dhQgX1pdfn1eQSERdVbc2I2VQDC03yGZXStmenKp46CWC0UKHjOrY91avQplpMjmy2cO3qRWLUmIoWswmqRbwxqE6iyUybzz8yuu3xxx7ZMjrg1NtZDuubGyarPbQZ0hgswdU1SubpwttVb87OzsNXU6xUp+cXujrbqRw4efLkmx4wf/3r38xcuwa7dalc+fQf/A+StS6vh9E4uGtrvZB69aWnVjaqLq91aWXN6HTs3bsnUyrjax78uXdMPPnjzfWV9cD64PDOZBbSiLjgz41mUhKPfu/7R285shnPEJii8dHCyy/Thf75wObjjz3W7rKkN5ZdVheBBIvZEY1FA8HwgUP7x8dHHQP9Z556UmO2RHN5PzX4Lqe/3ghE0xszU7cdOwqBQJvXwWolYv3E44/QEJANvWvHVqIIvd3dMBQSnOACyP5GaeWYpfdoGLwrO4VafxTNG/tR9qWyR5QNKDAllhiTy+tiILIS6bVGKWO5QdwwEgxho61uhuxu12vnXvvkJz9FrerkxLVnn32WRfXggw9C9XzDfkmVreC3qO0GKhrjLPJi8asPPUSk+qGvfe2Jnzzm6fW9cPrlF6gMJhdgt3GpEPnzYZsd/s4MF2O2G9BGK2urLr8Xm1uVInYM5I+aojI6Pm5QwU/Jn9SOshQvnD9PuHN4dAticfzOg+zeXCAuAFJyvdA84O1Y9ATpEECQPlFuTuNLzshOAonGrSPcMN1bo9ESFvzm8cb4iEj4dz3eOIiINWQLEoYErwiblihSiIV4A/0jnC8VDQQ6YGg9HtgUWTP4nEAnRG/jF9VgtavI1UqsTjLHXFErVkYICfAD9G12H+axVQuMQ2B0cOSqkol8x4AztJrWWMEaQudQq+eqzh43lfQCMKWUItnQONS+rl58NUaDCHMVI6BMORFCShBR5WwdzDNSkgpcOmFTGsLfCBB4uwF2vOXt7qFB3Y5tmr6ulMNc0DXTBm1JGL6V4cLj4lFXmZoN8gtGVpyINowuMJJoQAElGBt1ACt6AoBwMCdCOWqZDDWjrqKtFYQWye/xZIpVrPN8zdjQ+EsNWyzX3Iyb4tV2ws7hGDSv3KVLZ/ebzX613kZyOhoLzS9eTyaDEETD+NvIx8GyEOWEPUdcQH6Jx0XIE+eySZGDgrqiYRE0ssrloiaApxDPRD9LFSfaWIj0WeRwQWtM5v7xcbfXj/qk/SsmPkoXJm0MFVYUULi2znbYduF4M1pMA30dC/NTMIESgRBuWyAUNpxjqLeQOpBEUVEjpUcSuBMweZPUIh65rBlRRDcfTCXPYXVDo2AvgO/R19EcjWY6XVxZbV64sNjWMT68paOQj1NDbLJpijDOMK/KzfDNm1tbUb1yh/yv6M+bh1f+lFeYFH4k7n3TQ255dWhPdhmXww/P8P/wLgmRyfDJE3HCULfYDQrwCkBZywlWdLCiifkAP8qZ5Zyib7km5QK5JZEyykVxn6KF5TmnYGokPGe1AHsz1htlTocPi8YhSMeaJHICVbIBwDEUW9W83YavjIUquoiB0kJ4ziQLFZmBiBO2DHElvHJCpEYYDbFpRP+hYkE6QfJSpQyD3cJliP8vgWJ2Fz96CXyQe2dOMMJELTM52GW4qnitggTkOD4/jDZmL+STQEvIL6HPKUDmAY6M0pxSBRuB36JmigUkKbEFwCmcq1YvARVjG2sgOubz9RSNCw00gXNSoeaiCx7FuFgOdPRLJMPd3V440pbmlvDHaWWYTxUYUhJSzAEXK9E1GU95kGXBSWWaEDUYgyxczHaPx0l2F7nJu0VdHouSFzu7ejE787kUewb+vSJdx/JleiHzXSyS6cuXjt1xV+SF54ntjI9uXVxdfvXUaQqoHA57KLSZTMbQxOHgxvHbj9ONIJtK3nnvfWfPnsuVKru37YLoEWKIex548Oyrp8wWCEELBNa407179y6vBwW5rdVT97FjfGsyngKwjlkJQGxudiGXhN+j5/vf/Ydbjx6fnJ6fW17HGw1H4wQG92ztd1o0ofDm1JWzFpEkZX8n1nZ3Q2vlyu998AGNx4t1MDw89uorL48Mbfn21/7++OGjHV2d9z/4YGBjA4Zeo9EOq9w973zX+RdP79m+PZKMg1jv7mifn526EljrsOqhMAEhu2379s5ybzgaIWJ8fWq+ePV6e0fHeihKsYWH5mSdHS89/8Luvbesra2M9nadD6z6vK69e3f179xawU8pZWlAOTs3g6kxONh/Y2KKENnA4PCli1eJttagySxVKdlg8CEsFbtVi9CRhFEryyshN2X981tcRoxCcU+JkQNMEdMH4lVo3BbnN3GtamVoWKjdU9HV+N1vf9fk1cvUF/navC88//yv/drHV5ZhCF6wWs2YApHNDb0FajXr5TNnzB7Pqy+9TNzGZLVmIrkhchU7tx85cuT06dOTV2+sXJ0FPMNC11opSKc1BvaSqcBpwxl9l431SaNUuivikIhFX1HZaZFlt0fjMcJObp8XhpmllRWSK0cO3PJa5BXsPCgh2PrkmGF9IZdZN6sw+DBGsfckuE/jJCxqEXU1wT8oMoo/ZDgYgv/YgyO0Dth6cvN44uiJbJVxbklGrAAjdjNiSGQl8gT8WigQoLgLx4xyWS4fDSEbWnaUXBUWOvEw9KNcoLjUwteM0a82qLbv3pVK0cw6jZMg1jsWK2lKAEJulc1hox0uilPvwRwk1qUrJssakxTbCDY4U7BYC9DMgSwSeSvlgirUKEFhLDBiiwR/HRA8GcvJHMECVVuX6uhtqv23UPPd43TCRlcxmwBXp9TNImx3uBmQ8IoOFsGLW2LGP1dcS2mkKEJWgr9UyztVDWutbITaPZKM5nOlYqpRz1d11KeR9GiSU7OgeosNQ6lpydVc6ZI/WXRFYvqNuGU14YrnADhCW0+5u8dod5K9ox4qEliHHZpMNQaa3D6rBNsfvS9mjEhYVjKrWjSchA7QEICwpCKcCRBnrKUqJBKpRltyAUgxLCECGEa41N0WknrMBXSqLBECPDBjExrkPiulHBwDVPcimo0GdYe/k7KU+cUbYHZJ6xN6oRYMxQwMkFMrWCKzyHUGFktEPDU4YFEfrSlW1JXMrixFZoBfOtITDCztSlAaOiGJtVCLCQ3K1I1Eu3+ps327Qd9VLGyYzHh6ir0ki4nFwnLB3BNvmDsX5YfIllX0+oO/uRMJK/MPAR9Mb6LlMg7Kj+hH3mWAWoPD3/IZRlL0tPyw8CTUrNRSC2ZYFDDbjm9JcJlvyRfFCpPrURbwzVML9I/gG2JIjATGgguQdC+fUfgxOLjEfChAotcgrxNGpnKVa0NdongJIwN1IbhlNuoajhqJSSm+Yj8Q9ERJC+iArwNpk6pfQkpy65icynaXC5P5B4zB4mCJo3cVe5akZYW9gfRRAlVCOipbgLHD2BQsSU1NmojPSJcOvFSNqsNn93ltbqeNJts5nO0SSpOItsA7pREv3zfoszlaMpAmbtBhAZeVxJg48fWqmf6aos9JwpYRAgCXzOQzCC0K0lpHeRmD42nQ10SfShaTiZCQtuh0+UyaugHoK8GDYDdIbaLMKF3k8EUNVCsxgnhIjCnUSwBg8PSp5Y9thOH16OvtIT4MidHy2npXb49E1Ol7wqYgykl/CyAOhFawBfN5n9O3Oje/vr6BqXM+lqB8lqnHQeS+WKZ2+qNFwulslhRvNBREzafSOSBWy5enGPhbjh6bmZgghkQtbL+9iwlF5RfyJbvdodXEyAKmcvmT9z2os9oIzE7OzkFs8slPfvJrX/kyuJ5//Idv0iCs2+eD9hBbZXFh7sCh250u+5Yeb7OU1GvLLretv8sDPxw+RyyRvuPu2zO5GmXSrrr63E9epQpycGCUfWs2555+9tnDe/emk0nSojt37ErlctcuXQxFkhqL/fLVq7v27G7beZd14uKBPTuv1PNtNgmho7AZ/5HRsYNHjpbqZXqzU2bT2aOnu9rg4DBNo0hKsK6pIqNDY9uYfbjLv76+Glw2X7twCormzq52yAGES8/MvzWzyTZxfQrxASlHOhxiOyM78CCZdJa+srtbmoadf9Mblt0oi5NdowRf+CXNwUDaIbQwI+FPlb1UyKhcPlM6U2rvaD/79Gk6DH7y//N7/+sPPq32yTaH7SERi5Q3KMvMaxxSHqpm3dHwp68PDCqpd4qZsAXYLKubG9inQC5oZc0kUmeqKtV1LuSSOhvPq8xqrcXIntb3OKvhdNWU09lpxIkpQO93A806uDVuBzAhOFIQ+BwHSXZo/wGXw9Y70BsOBEtJmqgoIFG+hr9nhHwRdGSKc0Fxz63iRuIXs6hakqGlMmUXcxtKUEqu6t/ykOFryU1F1rUOePNFRc6Im4FsVD6mvCDPtdidCFgzXWFMGgtswEKLCNaWShwQKmhirGZGHknNbuLzDCamA8cgSwXCgKZSdX2zp6/7+Inbz529MDM9DchbMJDE04w1eFq6eroYGYfTS6CIXZ5J0+QLsVYju6WxSYbB19EOMR5Z8wgCtlyiEwHzrzY26SYMoQ2hPYtGB28PTRSAUd52SHX7ndbd+zs6OjUOZwFAkpaWRLxH8lYcD0XB3hwGCzVQpKTRIAQYkRIARRF9alzehqlS1tUK6lKhCfd5KJDGjtDW4fUz45oDh5Sh0xjyzUa6RAdfc7rsL6mHYnnn9Hr++nyuabI1dC6jw2C20XsUvslSJhXO5VKFTAKfUsxDrhmQCF0AgIlhfknNN8dkbiTFid5AHCOBkU2MJx6c+HZMjGhFQfxDMIuABk0ruUxKSFywvHghF2IvkGfB2cVzYR9VinmWit1uJYDMkjEZ1HRQpcA3FgtUy3mzSUsjHOxAvRaUC+4IlE9IL4nMFPO4JFXiVQTkxZpCYyEBYcRWmMtkTQhAWBRTa5FINyS5ckplIQXlgyQBMBrqtVSycfm10IED20bH+mCkg9mH1ockXVgZYqXyYDPLncuxeAEXjIdyYFmhrDMUgxhKhLxRtIh8ghTKj/InHxEZoeja17WyKGn5JLq2pXoV91eA5URlmWL5sAQYsBg5J6fi6HIq2YMEe/hHNoCCi2xpWv4QucI1cpWthSNBCpp24/VyfTBO6ORlES6izjme8pBho5xP17Sa9SWTELfKiBB/JvIrUGohoOaGmQBaWLKTSalxa2grjiOgAriAZZ1JoQ7TKXFq4a0Wu42YM9aShEswzeC0QkWq6Q+GU46Swp6vkozRmwXBRNdBE7dYF84UA1411hdROTFeIXm2S65Ggc5TQsLSoVkYEDAWQqUIhaSqasdhRmcjAwskkKhdBidZESo4gCruXgoLHBar2TA5eS0ejmAckSUd6h1kCawub8DQBLSHiniGFxGKuMf7RZBJfImsEGRRzQqpz3avP5cvr4ZDVgpCSKwwvHxAraaJ0/Fbbz90+Mj/+OM/A34AXNXrMjEgUhqayzPCHrU1EaV+IufxWziR1U5lsi1dKOzZs0cOzkebqtFt2+hdEQhH0EmAlqUuYniUthRPPvM8gNsnf/oI5FmJVBZ8MjtlbOsW4iZjY1vTF69BpBMPRbYcGnz1J48PbhklcQKE54//6A//x3//bzSogJ95eXGG1o9eX+++W267dH1u1467a6XkUHfvjYlX52fnhvuOJdOJcjQ2OLLdbHMki4VYJLO2AR+QjbbLwPXWlpfpTDQzPTtKEHlkOLi5TlIccIivvZvAxaFDhzxB942Jq/lMdGTL4NCRQ8Qv8umkzWlmk1udNCIu6UqgqW3060YkjO7ctjQzQw4YRD0M2F0+31Jq2WVX60qZLrclF6W/ejETCVp16s3lJQAKFAIPDA3+9PEnaMOA2gZ+dfXKDSxJGtogrKHgIPNJ9I/JIkDBCubB4lYestplN2CusYFY8disEpttmkCSCx4Bdtoy8TTAB7lkidhfaDPcN9j7lS98+Xf/++/e9763PPW9n3Rs6Z68cePOEycfSfwQbIkcRirr6pVwCi6RfQcOYLU9940f6rsw0sypeJLPnDl3lpQYAJWu7u4AYHX2BjxZHvPg6EgUkRLc3L7viPGA4fLliyxdbJtqoYQIIyjEfqsUGhYLllIV1YTaposVt5+gDIlefdi1SCH8E+5ItABaCT8InAp9LqTpNAKUfcdD/HxgrtynottkX7ME2VX/Ru3bGkR+cxyRJT/zRLkUxaFhRBXxpwjBptFqLueLUp+rktZSqXSaTlZejweuZSxj7oIiQ6oUuAUGEvCLAJLxhiUmKR3KRRAqstDisPeNDHUN9OiuXBVphuSQYF2tp68rk09m8zm4l44cOQqX0PRrN0weBwFuIgogDbVmjc/t/+D7P0DEYmlxnvQW4pvTEXLjB7mnpE5V6WLN61Jt3aE6cIvmlmNtW8bBkmdLpaSVVDw7XkS74FSJpAlzLj5D3Sw9i5gAPGhkqWhfbl8qF0lCN2vGSkGdB2mVhheZutimuY7HT0UGspWOrtQX6bFrU0VNLN3sGjzo9g2vz2VfOLO+uB7X6rsd7Xvy9HCxua0QTKur+SxZiFAmHaNVnNYI8R8ZPXwVuqNVsR/kpPhXtKfhrmRiFXeN61CmmMlHqoqXTFwEuiWhnuamCTlaC1k6yZSASXd10zrczUGorltbWxPecQLd0TBNivyil70YRkCdhZ+hUSFJTO8yyDT8PndfTzulvYJsRNtjvLKVZIgwZ7XZHLF6AaXyXdHjxby6IaYADxko/pd7YKJF6fBgLZv4g3wEMKO6rkZAEnBQrWmqldWbG7mFmVRfTz+eSaVQhHsYR47dwD3KvLD65Rf/yaJsnYDfskDlNXkoUkAGRLxbiRvLc35ef5eVh7ZvvXjTPxY9J1xX8uGf/ZGsiPT9Vbxq+QoChuMoP9yScmeKYlZukMUg18FmE53Mu5gjoiDFYJOsAPuSri2oFREKELhAhEt4BLYsxQ5FgVKuRlKXwiIylFRFEuzFUZD7wlEmYwA3OuMkZ5FkMLfDcaQjsJhZrZJAxoFIINaZ1PhyIxSKkHegKL1177zOUGGu14mjce5aHfUIHTWkcaRjnQ5yWyxtnB5kKZXt1OkbOLtUZomOEmYoxAzaF93OfcnqY7YJvgmBK1Rz8DxJryUoLKhMIFMIJIzGDw2dwenx04hQ5baVVhdWl1cAY20fG48Eo/09Iwa9GUBHMJSQKJEsW7EPRFRBcoJBRNgKIxcO22oNfgFCCFh1QuyCv842aKr6+3rCiRSR8jNnzjLug8OU3sIjTR1wpVbKFc1Y6sKwuji7APUp9u3q8pq3swsti3cu+UKPLw0t5uYmRDj7u3rjkSClEmgUIm2xVJZOm063iXTPWiAIKStJWAISsB5jvGIU4i8BtaDxSy5XiQQ2My88f/fJOyk9nplfhPl5fXn+1uPHXnvlhW2jwyCHD+7bs/fgrelik+68+MG9TtszTzxVKcVHh4fgX4T5we3rnJnf2FFrXrk2dfsdD9Bygwpm+lzkM5GJqxPdvQO0EXyG/rUVqRTc3Fjfum0nhorVoTvzyksf+I1PdnT4Tp1+cXSk9/oTPwkFN8Bmj2zdwqR7OtqjodDc/DzpTJSlCF+LdWN5dW11pcPhIt0EonTn8FA5m08FV2BZ0dVrYODuP3lHPBl76ZUzH3j/L3zt698slEpWKnYt+dW1DbfbMzi0BZnMYkABY7xDp8aCZIUQ+2A9MGVsBtkH8pAdwQKV0LQYiSw6aYjEDgbOKowOVHsncnSLTqVogYAhaUiRdS/mfkwnKDh/zSq8z8W52Xe9461f/tsv/tlnP4NYL5LlIwLhaK5em2bcTtx598qtiwtXrxS0qqFtW5euXd92+BAqR9PXlCbBJBgxQmsqu9FIyoDN9sxLL9AF+xO//GuWf7Scf+VV7gI3EaAQwWf4Namq4h7YVUD9yWUEAsEzp09TCJcKxUnzDxwZBHe2emNFtIi0ISIIBGtSswopmAA55IYpVeJmaZrC89YG4QkTwW8enK715J/9fuMD//L11kjygTeeKJ9pDa+ihHhK1FkOobja/NlsWj1Yu9ZEMIalRW8Yib4KtXsckCDXSYiSzyAeOSZTQFyO72O7gEBk6zUIQJdApNVmlxYiibi4cSar7HDw1KChKKAtFH06/WYwcPKuO0vl+vLkDOBqkb81FUBdSrUiITq45OGRRYsSfUO8Yc2zZ5l5FBcpx4Fh1W13WO6+d9vAIDt0Q6NaJSTX7oFSI0fMCeLapgYFIXYbnpQgnNQOfkO5Ifzi8iO3T7hd7KCaoVbUlNLVXKJYSOEu4qTgHiIiEZJaQniFmoo4dKHuL+k8Tbv/xrpjeTMzt5qPZdutXn9NbU1Xmja/lbalVXK4tLOPBMt0LAB1htuCFC2TYGbt6siZ40LBP0kQBRtAXD3xdFv1tYw9PxAqKMBVrpfv4oHj8MiGaFaoUKKFUU+v20WNr4HsLGhW1CQyE/EptyfxaQ+1nOhQ5gaEOZEEiHcgnkRDDw4PUfULpbNMDK6tBjYnlB/CUi5DpCUDgUekPFra5I1t+C+XFq+AAaC/Hg82LIqeFU8BrvjDyUTd41HDPdfXb+sbcHMl8qqi8GTZKdpXnvzMg6PcXIzKi3JU0Q2y+vmNKGUDKn+i7EWPtj7AZ8RJ5kf5MIEBIhV4vS3t2/qKHAfFo3jGvH7zsByN68H8UK6KF1sP1C0mNIpQwbi//qpsOUwmqRUHK6zTsU0rtJM3KNhtFCUEFBgw+HZsTGIIkJ9yHMxps9HGqRlcTsbdsQDEsafqXAqVuR3Rr4oEFLuEm2BoxPJmPsTsQCaIGcKFMSvsMUQer4G/xyqTq4V+kn1A8kWYOuCbIH+AMCU+h/1KCJ22FRXJ9kilO5BqOSbqEE8RIwLOSDYsPBMcFgcTkcsTkmjYUigksBg1JWkh2lelh1xBkNSGBqRvpaVVrNPVpWmv108wOh6JbRkezWdKl1+73t3dG45kWFcIdKqniDJTpsDccIOsEG6BUsJ6BvBNGhOd8cDPIEMN8LitowMnhiCYv7uXpkDDY6PQMly6dD4TD+PLgrGH2kTwqFpaFPtX1zZHt3drTLaNYLhrcPi2k3dRcXvpxg2GKxJPYeNcunItHY/xp8fn7/X3dPb2Xzp3kXq5t7/vXU8+/hO3zdTW0elyIqPj1J6urKxibCwtbwwMjjMgEOLkCnBqqhLR6OzUNOnD5flpaEJo+tTV5h/u7Ykm013bxppza3NrwX179vU6Hb0ex/e//xU6zVvtPV6Xn/xh//AIAQtuY3UzPNw/urm2FN4M7nnXmwcuXjx//rXe7va3vf0dzzz640wivmXLcDgcHHH72LeBxZVv//X/uv3kiQ9+4H0wYTWrxTd/4ldnTr9YqMocpdMJJt5BR1zRCpa+rs6rzzyzNDvb3daWDoV7OjtgBp2/dEng9zqdIAHKpYXpib6hQfIbIAwe/eHDgKtnF5fxV0eGRyduTKezxS1btpBqwk1ggpgdJgOGf4lFC2EcTnBrQ/zTby6D5SpOC6sfMsJS0VLQE9IkGOR1u6TlH/h5AjAQ9Lv9U1OLA6P9L7982tvuUeVV1N21+f3f+NrXb7/teFdHJzkCKDkwATva2nO24uULFwCwvONd7/7M3LS3o+0DP//+0wO9n/zkfybhe+nsedot272OSoEubM1kMHXx0qXt+/aQxiZoB9XGjp07oRQNbwZw5sTAhIEPCphcBSbyeq65NrveNgSMrgca1NRaFHMPQmy6R4PfxuZgnSM4sBsowENbQzzF7qCogNHggRAgCvrG/XP7DBTCiCdvvPh//oSt3PriG0/4LrIHa0WRfCL3+QvLG6HDZciRGzhaNv51tLnpSXvxwoU2D30OMjBQyrt4ZNQpioyQvimtB/JCXmiJM8RIHoar7I3JSbqTweso27taNTkB/JgF6NFssk5eu3Sxb2hk+66dy1emNSYNB+ctxA7l9d/82tchv8GlK+SzML0LLqUpoTiakFJc1D9ifPB9fd7OvM+dRZaY9BVKwEEakfmUsIO08CC0x01gFhBzhgcXMhwTjif1uvTKa2qItxHbJAsG1FNfLWkK6XqeuopsBWgfQXZybMUUnNh2kv/Av+LZWrzIN91V3ci1uWqx5gundIEE/RXUNpPbQAfBeiZbWoOgA/AynWPq2bLknLkZ8FvSpY4mHnDz6EpNmMfIwdFSmnQSq5wpFsmouIWigGUeqPBB9FHghDUihiZEfdD4m2AT9Po7fT4PN0gWCS1PcBiZD8t9Kg39kcbfSb8WF7jlfC6Lm0beMBRYI9c4umWATBANFdKZuACsJMiMYGQWJcYpqp0rqeAUsQqEXoIUHqYtPjErQiB14q8zqaI4RR0KmE7+0hHl4Qv4+7ii/IOwhSabgQ9Fii6PZ2ExTalk/yDNGokpwG4qOkBWL4firvEruVtWjBhCoACxGCQYKZYFn8gAALHpSURBVEtLWUKtVSTZEVlUon35umgRYWNWLkc+xvLloXyGtxQ9LR9G0Yo1efMrij6W0RZ1K9JfeV05iHjicjjlt4yHPJHNJjtB/uPWuVKlEokTw7vJDcvCgcmBG6f+DA+YE4F8FqZQusEju8SnFyOgqpGSBhgeUP/yGUZcuSqxEiQ/TRxPvFvqHxB5oKj4EBuJoeAalWCYfAApybfEApN6NGnbgOugQLIaVmIj9TKRKi0MqkSdOa98iCCwWhIsFBCDxJAkgiRtSCiD1wPSJv43/Q+cVtIP3Ct6HZoZrGvWAgxE3B42NKuC8CJgehB2Jt60e2jG6enoIuEKgLYdPzIRKlfyZMtDRhPuHFCeXKnq9nRQ2gAgGW+9UhI+B+gguCr8Eol9wQ8L2RrU44koCpiSX8oIaIJ0yO1ALxvsnnA00dYzYDLbEOGEp7A9YX2mCaGBRVjXZPMlWpg4vT7IOwFZYCkODown8xpX52jDlkxSyFyc7vV7wGzQ2MSs88FmgUtH4wcqoZOx8BOPPRYJrat8TphrKxAEOW2dYK9vOR6aXKjoHFUqEokcNKqReOzpnz4STkCqqe7o6uodGPQ4rIHVeeiiSQsc2r67FA7RQaxSyTRr+dXlsFuKCzU2swMbJpMuonS7+8cp6YGoamMziZRjOUF0sPrkMwSsrAbN/l3jZMWon14ql+LpZEdPTyQWHtu9S7vaDIXWvv2NrxCfNtPYympYfOnZ6ekJlg4B9khgA5UAdegrLzxPdOvAvn3lfG7fru2h9c19e3bPT02uzM35fV46V5cKeRxKML18zAIRs9czNDr6xb/7e6PVQS+m7bv3EOw/eNBGf8DlhUXyG/jTTBAVwAYT5FA16TytMwKoYfVj2MkOkoyMBDXYoOxBNgg7i/hcsaSiA6bEm+r1LHykeTpp1d1uQyCUCYUzQ0O9SwurviFvJBw2e+imTr1ZGrfjhZdPSZWkw0XGPZ3Jkm79xV/81Yce+trzTz0Nwwlw5+DC4ue++EWO+dDXvnHyttt//JPH4qm0lNwUayPbxrjamdcmZ6Yn1S77tl3bK7Xiu973bpbrQ1/6uyyNd8pV4lDQ+RFTqdOKjq2tU0UCYNeShw4e/b0//sOvfvnL1yYmCBi2zBS5wVKzZERhsIGM1AbqqiRTcX+1xLSp097MBBkcZlAEDRtTec4TwkUiJv6ND5FTyhFe/54IYB4iZbDPlXdF/DDgtabVaydIDAG/S+e6/4F7r169Ont1JhIIcQBKJbgGjoZIaBno3A4yS5g1q0QbsYRQGIS9DHVTHVorjCqXww7SvZzMciqCkUwBm1HMoHzJ6fL++Ec/7ukeoBsASSOOjwGdiaTohEpVvq1nALvWAKsyKTOzipIDv1O1c5v6/nsO33bnSEZ11mwvGbSQMMMKiuYgyKwmkEYZBQZ7k4gcSCu1TaO1aA126hJg+0NLkMZDo4jQxu8Ul8mQSzWLmRoJHBB8eIdk6dRgqhvQRPVDqQE9TK5uy9d9sYprNWIOxvOrYcqHbU2tR2el82sJsUJJT74YBZhHahcfCE+XgDFhXQkD5LI2hztPhkViOEhg1AAByiotViWYjrxrPZhnpL6sb9SQkArJAGJBSBzNQBdtzMr27t5kJg2gupgjDU1wUTQ3fHlVggF9vWBucLoK+TSvg4wpwvJfyA4P9VLXwNqhARpBFqAzWLGAcDktFyKOCHoBuxhQEL3hsxmyBEwv/g/zy3VJ4pJJQSfLhcqky8XKL1HGOrdMGRRzUt0l+ooF3ACT00TkrdPyoqm6ci3g8TcOHPYDI8gXNoB+QbSMrqcDGLg/DT2qJGCJUqL2hiXIjyx29LKMA7tI4iVSrYsyo+Rahk6JLbP+ZE/c1MzoFIHj14UsExIVwglU77ayvxItkWRwS+kKcRdnEPUp9yZ+rdyWnJPXWwqbyxEFrJQegjMgyoMjWm8QewEv7ndDQ4rhhl9GW98CkGSUIyhQQrylatFpdmSwvotFSFFYhU6PA1uhTHetCtlX5cKkpBjQB+YKQDc4TSXDxJWIgibagqaXtgpYRsTWuTgFysgnkI95OimJC0m8nw+j1ZgzTAQ6dxJfttnAlzMy1GFTGQa1LM0/SRWzQliEHFWHOSKAGgRorSjsKZUSIINyLmEWzlt6jiFEa2QpiSuSyS8Va6UMjbFU9AeGd4mqEnrJlqtFV7/LuuewFd/25aeXpy8Qd7cIwZQmEt2E+83id1MSU1ObsT/cTl82FYECzOXywY+GOag3Sl4K30oiNQAIMaWR89LiQtXhUSUyEYOtDZ062D1y4N63qOql5ZVV8tZmG/kP0nsWK9K/QmhfyF4JQoKtigdjZkdHLJTv37uzAaVWKbP0yhN9YyPJpRt7hnu5rkg4+a53f6RMK9B88YnVdeyi0MYiiPUOd1ssmrSZIIAvXZ+cH2laJ5Y361qrw9dRodVXYt0qScF8d5uL6PiRo0eBrgFi6BneTiHvlauX7GV1MryEoOnpdDZqSaB+N+YX3H4/UbPgZqpTTbmW32ywUkAMEPTA3m2XL72az6a72tusRv2WkcFsZM5Yz5fjm8xjW5dfazb6u7vAdKxsLP/Kr33083/914MD3aGNBWrHiPitLlzztnm3DAxkAytK2l4fyWTwuQGFTV6uU7IFEeHG+srk1HUoPqxYCaEgqDqCdYFwgrITJE/voCWZ2WRLdvi7blybfN/PvZ8wwitnXr1y9UpgYxP6KoE6yWbQQItHTQflO8CMk1nSZBI5YaNKrRzbQwxjIuUQZ8quYSHRCwAuTw1VLiRCIHeUrhcqo1tDp4+2LhcpjOXVdZrOlvN5fbOBfx9NJOilU/LV0AmOWnU1GJQsoKaZCSaPHDrMQvjzz3zm4e9+j2wJLur49t1E9p5+5vlHv/M9KRsg0e+35OOJD/zSR9Gdi0tLDzzwwO/9+q9PXr0UzQeruvLzLz8H0Dq6uCnRUdpoIt7ZxVIaI9Y2MsHuc05dnZ+dXD5w6DBLHYTqmTNn9u/dByBrfXmtkavV9NVsLC98TDqjw2HDJN1YDw30dxMurUrBz83sEsRTbCfiO4rvggSR3BkPPqBIEZEk/+pD5NW/9qDgANEjOBoRQSKU3nggeLH9weOAlp+dnnrLmx782+VVNVpJkpUY2QrYClNddDHcWEDNBEbHpKOTmS8pGZN6aO385MyBAwcy1VQ5n5SosFmbSsaketANS6Rlfm4xsRKxuNyxzSg9O0sFymxUSDY1YAC666pNS9cXfW5zdxc97la4923bVPfe673lll5fW7rafNoNGY+2TCcLLbgt1KagbAxas5Te4qE0GoR5cTdNOmiI8KrRvHRWE69Y7haeHVLOhUwFuDUYKXgHsNUBFWH/SKqNJG7TsRD2Z6o91aatWLUEo83ljXIiZa6pXSw1RA2CzkqLh3wlnYuXylmpQ2mQcyWXrISUGQoUPhEGE55/GvdV1QBnKsYZuTkIFxCHDBKlXIwgpgzxPNEpVItILN+CsKpSJI1U8rR3dHVLK3G1OhzZoLwCx7RcTnHxUjmkVQ/196E90A4AAQG9ArOSwiGtFoLJ3aN7/R1eyiw3NldIRTntdogWaGTHc3J+/HBCKPox9ZDqxB4EIFarxsKhMJ0VSkUpPJbKVIGa5ilzR4gazeLWs7qd0GyNSCoNmlKL8Q3fDvgQWxe4TZNSl1SmubaeX17JURlVrTjZibSGhGOfQm6JRXB01jCaW+jQxflT1i97veXUKi+IUmSVM0D4dhgW2Kf85rksfeZRbED5AB+jbhUWf6aO369/hq/f/GHxiLLhW7I8ef6/maI3lz2vSbpXBI7gAxjcluPLc8jpGWG1qsQPE8gqV84vQSO5DWLLTVWeRrilIi6EEioWK4oLREmDW7TYpMMlzi4qR1JtWBiCaeHilCJl4t3KqTEzBfiLJubAspq5T7k2LCSyPCQsEYJUdUvqvomHphjUjLgoVwqiJCsja5fDssKBfGNJS8wJE0pSvICc4cHBYqP6ktAFP/DO1Up52M7wiMkS05mB5LHNpHLZVTTooB805jMMLzot8twBuDd5fUZFVcntdxy85Ra8LCJTlVIe00JA2ySBqTP3tW3ftXf77r0Wh5vYAH3gSSHhyGKU8CDJTHgemchSFZIvJLBG1dZJn2oZQ+pi77jzbuyIV59/PhqNoPA4HjZ9i7DXanW43O17Dh9p6I2ZXGnXrt20QDt/8Xoono/RKklr1RhtVJoS+Fqen6mWi9j/U9cnVlbW6PSC0KK7JzbJlqHeob4eO11cknGgZZls4fylK7eevM/m63S097k7ew4c2B9YX7KZdb2dbTA1slVm5pcatEs+cc/FyUVne384W3S3dTLkh2/Zn0rFiI/ncumdu3cdO3qr2eqi0Nntarvl8K1bt21/6fQrl69fo+r51hMnHT5fIpdPZnIWqx002dpG6Mb0DMiX1WAYd2b3/n09/T3PPffk6Eh/KLDa3+N/24P3Hj64G9hXPBiw6dQek2l1ZnJ58rqDkIAWev3Y/PQUayOTTcP0+dwLzyJEcGuQI4ViCSA6/QVcHl97Rw+EGMlk5rnnXqTBkVFveubJZ1A51CCPjmyhNyWKaGBgALECgBPeUGWtabCfwAqychVf93WdwEJmlbGq2KZKUJZlz5pDN9OiJV+tJzJlTGicEosPCsAUcXiGKJMqZGMlJjocIMqt6hztAuRF8q1nYNDVRuNkEAAgnI3QNYNHg2MOpJt4cvUa2fTf/b3fexePD3+4q28Am42a4IFt448++dNdB3efeuxJrVl9auK8qliNpSOPPfFIZ287CU+RFwgoC9EIgwQ68YIqAi9gn+UzxbXljWvXJohM3HbH7XanEw4UsqpEoe84ecLb7S0mK0ar9BSi4zVIOgKVbrdlfXMTXxA5LdYxMh19y05XZI7Ia5E88pCXRADIQ4boP/SQnSteGplWSoYUZ4DfRINXV5ZA1TI3ksQiiCEf5Nb4kYcCt0A8KF/HQFIeogP0Rlha+Q3lj0QwiMOVwEE2N1cD5Rzdyey4dVSmSh+zdKwUz3Aujozc91gtfqce/8qsLWjrcVVtZfdOy0c/3Pcrv7r7jpM9bT1pg5kGz0FgQwY1lBoVwL8K3YCOzdLQOEs10IAdemu/xd6vMbQ3ms5azVDGxRBRiSbWV/J6kBiJUA1nMhmhYjaTz6UxI3SUcVo7K6quzZh3gjbWgw86Bh9MNMdOX6+evloIZ7wlVWckiVp0UtcJfVC+EM8XEtVSulbONIDS4zcS1MNkEtkrI8QJkQk6LYgc4Au8wRAA0JF1jGilvlG8EtoyUueJWU33Ij2KWlcmfIL1QovvweHR8a1+qAPLNarPaY2QzycL+QwqgNZFyEC/12U2a10OE8XRvE7SF5uCBkfdXT6amKXp4JtNIn4ADuF0kzKmzaBS9UkNl9TDcw2AcrAICaXyhCgRFMCAb1l4bBkkJks0m0nnuR6IjSwOsvgal9/c0dvZPWCyuGHYadIYi9x/pZ4tKkgjVgNrEhwH2j2dqqwsq2Y7Nke3wHztMpnaGzV4iEh9Y3UAsEZui7nG5zWsKll2WBL8I8taAvOiVuUPCbQI3oqHElvGVuRJ60dgz6//SGNB9q9EZbDrFaf55seUg7BhZJvwXKbm9YfoL0Uxs5p5R9Y0iwSrUjSivMaFEEcENkRenciYOAM3l76y+hknlK/SQgAgJQ63cmFiysvdkb6lltBChqJR1lJNV2WJEIZHr6KwxRzgNx/Ec+W49Ddkw7fKrnBaX38wSZxEcHqSyebG+FO+ydUpQTHut1UljNQhli/DhD3FtxkzPsNYi7vMPbBRKjAfQBbdALFCyJyB5QPENrGoxZJWggJ6QbZqaGAIMwuhDQpVXf72bNMaioA/ro2UMh0D7fEobXViIKo6vaRjXfECQHqn0exRN8wet5+dEIiFCgrXiA4SKy0GWZ47lXvC8ZV0NLuCZUbrFxWJcwQciSXcelWnL3b12tlTz4N2gNkI8vMOv589EQ+GjZ72O07e+Q8/eUIYx2zOeK5MI0+H1be8vgIgloM6MC7qFZ/Tb2lW1zYjlYY5cmNqx15ndG01FAqAVmBNk96GmZWRI1dC/1vUVpEGx72Dx0d2lmIZk7aanb3Ax8joAO+FaOn8a9ekApXAQ03lb++kxx/7yDk+WJybyhbyqVxxfe7GieNHeg/uO/uD7xPoO3L0OB7Bk8+8sP/YHb6ugd6RrUsrG65ys2tsr8nRduaFp3V1DV0D7Q7njoO30gFtz8EjyJFMJrd9+9aFuapJrwsszyciYUy9fDo1MjCwvLx47tWzJAD8LtJOOnDT6MWjhw5GUuneO09Ez56jr8PHP/axi2fPry8s7d69h8LNS5eu3HX33TT36x8cevXcOdSMw+NlOUN+FonGl9fXnW4vIKzbb7+dJoakGFkMXDnsMWAG2TU8WC3QmrfMYWWxs2durkWZRGU1smx4sB3wvFAKTp8lHSokahm4t0BB82jv9ZFj5ahkWzGh4N0LbgT6R4dvO3kHZenve9/7Xnnl1MQrF2njtra2/A9rK6p4zTXWllpcV3Val6Ymvv31ry3MLQKYAoYWWF6IzC+8+eff+/M//75f/cTHDP2W3/nEf/qrb3zuV/77r37lG1+mHr2/szuXy8hGohQA0aur0uQH+ggUMAuL3BATDjNotZSHGOT6tpHzF84WiCFS/JuopgdSfr8PBUANPWjXtm47gJpAMNm6Zfw27oUdK0AtvNGWOGCH/4uHqEIZp9dH6l984N/0Ak0z8dUwVEXxo4DDkcnJaXrppLgw0fv//GC47MwjbyBGmRfxHzDJ2WYUvOL+ETuCYqUusR2VBOl1hVpjdTFAITf12JVso2woAc9y+XTVjFlHIFlXcZioP0xRD6xzqzo6VEdvsVCIfnB/f2cX/AdhKhSQmeDnsRXEk0WvSnyesSJkZW1orPWGTWNwqkxtEk+FbkV8LBRaXtiaJSulgh6ikGgQlMXGxbur1rJ6swVITCpTD0XoyuUzGPZ72sYL5ZFL11enp1dSkCNXzfFElkRJe1sPd0usnea7OL78ELIibCamCQ6HPES28lviqAhhEZUtpw5RKZOnjKDoAqj3iO4i8QQfRpc5cYm5UjaZtp3mbd29iKl0NscmYmSFVSMd5V0yyPBnOh02GqLQshfJnEylsUgR4TasP73eagIYSnQbRyAL3R8lmSSfwUIT8K9W6X2pAaqOBSAzJtOGXMQAIIZLZ27p5kSeET8L3xfZWG9qCbc2KK3nK1y7xtQ9NNLd3UPOaGNjA6wl12+SADYiVqE3597Qc2SRJdLdIFOtWlmuzkzFuzrbhkf7KR0hGN7y9vgtK0X5LapPrGvRfTI8cl03dac8kbAzQ8ODNwQ5zGTzOn8rH+PqeS6qV0LQwr9B3Y/yivJ661ByVOWh6DtZwpyNmVFOKv+K9mqpYF5T1JvYAPjdLVdYqbeg2pbhw1SGqkDUKFgtxa+WSK9YoexSxYcGjcbnZHWi/TiwGNfqGmEAQkXoXx2pGlYISlhGQLAUONdIvia6kJCf3IYSXWeNcAUtScfRWUtcpyTjxO0Whcl3CanJW2L3SEBYlhWCCJGqnBphKtwyElSRRVbLSyCRolBS+1By8EW+AfBd+ZaMByPARYOhxuPHe4Cg2o4/qDWO9G0xJQpzS8tzU5n5iVc73VJ3CP84N0P9sEpj7evb0jdIa7nltva2eDhYUdeNtFasafG00TQm2qdIGoDVJgYEih7bQVDbFmj2aIGAiapJJ6Ppy+cvnX+tWaGQvj6yZYQqPhrVsawYwVyhtAjy2deeyOTVRstGIkUrQKj6z1089+u/87sLC1NjfeOnH/laPJLednDf6TPnQB5727rxZlaW5gGw2E1g0UsRQqJQUWrUvo4OjY7Qj6OnZ6gWidZ1OVN7f2jqslXdIKAUCQWNwF+1WsC03QOj0XT27BNPD49tmZyeHBsdKd+Yo5Z6bnl+bPvudChKI6lcKsFo7Nq9l5LcjXB0ZOvuxbVAPFv05Kt9I9tMDlcDzipvl8nbffLI/dlUGI/z/PnzOJlvevD+Z5966ugDD6QWZrePj01culAjqpDPfecfvsXyvfPOO1li2LZGmw5ga5mFDWaGcv1SmdZQqo3NXCbrw0o4f36ovx+ey5Wl1QMHDkYisVOnTtFSoq2tIxyO9HT3kW2l5wegDtYs2WiULrt3aWkFc4QIQa4IetkE3TcnwIBklloP1h2f575kayoP0UUtJaO8orylLDlDY8vYFvU2zdrKWng5rrKK2E8X4QGrUsRNyp/4AtXx4WS8Be9MBje+/7lv/e7v//7o6OiPvv4do4uwi66kLwK3NfU6acAGb9vBvbtGB4a+8Nm/IKqzbecu7PmfPvmTPQe233HnkUceebhrt+9zX/r03v17uns828Z3BJfDkCoYrUaoaOjBRnCbfcWeo5s16p/CXkj62Lcioxvap558MhdJaOg5y6bzqKmoHhoYpFvDtTPXqJr1trf1ms0wdYOWYWtAE430a90+94skYX/whG0lLypxuNY4yMutz/1rv/nMv/6yHI53Wv/zG6nQOofY+Qw2ATW2PLhdWmx1d/a0DiJCSzkev0VQIjoRPa9fAYFC2WAcS90s5BuAl4Jrm5l0gSCYnIfP00XEqEvFoULTuNwOlbWcgKwlhZFZ05cpV7HBf2fAkayqvB7V4cOqO0/2DY8QEaNXG5gj8CXpZoNqLjgiDDRAFTFOuBDtC02o2tpU29VNp4ZOvTq7UuZFUI6YMoIa14UITRo8VzZCwV+lSkMFPGcYObQNmM6C8eQmvb4blCZvaVrGYrlegMNnJs7R+jOboz4H2eho6/QRQKXSgWWKiAMNikNJoaOAkMQLUpw5EXxy7zJ8ItAl+yh3zhWgG1qRSy6FoUYCYf/SnoIiJYXTik8bcGwt1sHhEaYMicFOAeJMGYsywpRQap0O6n5teMsgBIFZUUIKkzBde+nH46JsESiNic5UbKVSKpmCVaRGTZe+RrQP4c+BSPpwTlYSFyYikVigwP2EGVuQWETeORfzzlsw5upNkNgjAQDU2rqHkH7kK9C5G8EEcc1aw4Qzwy0i0OBKBGspXpeyIPDm6dRRQBfhIlPrP3E91t9b6uvv0Rs8MEIRohYxjNbAmkb1tNaTjBjrpmVKcqSbq0VRvS1tKuYJ60dEg5gNYjkoepff4u+yCEnNUn3AlpEfLpsXlSeyOlHkfFs8bQ4gx+dMnJ8nEq5g2oQVhESGMnEMv5xEljk6DcIBoruiazFIoJ4H+akoaS6XJDmfY86ZEgwAvGTWN04BfieqEWuIaxPTQIwNan9F7fBdFgTfZe3yW7K0rF9Rnbjd5MPFDWntK7lKZUgrClxLoUcR/nZkC5fKgyvkgV0r/+DFovQV75hSQEYVnB2zIzTWcpti9rB6KM1V0+wWiSU9WCpiJdQbaBSOKGFvboYZBpkImCOvd3va6bOZLoU7b39wqLu5uDALJT6ZlVQ8jv2BXofIMF+pW92ErDxasy2ZSrm9zkwlDzsLtRBEqClnU0ZbhlguDpXP0mIWGDQV1AvmSrXhcJnlLquFC2dexsnz2+hOn7cboXjVQFxMlQJbmI4OC4vLRS2pTSrcvU1bs6DTXrwxMbEY+FDqF3tge+3q3jm6PTQ7pdLa+sZ2UTFKUGlqdjqXiELwBtuUvlHq7WrHuWdIAfqurQfKqsSYrc095HzqyWff9CufYAjt/f2mySvXpid8nf3M+tjoqMXbYXa1nX71vLejhyprq8P74vPPHT60b9+h27SNSlP9EiEippsKZp4wi1abi1iQu62XkNvc8sb23f7Q2iYNxzzUOXUOXJnbPHD8wEr4VNfQGOFlGgC/+z3veerb34oG1/fs3Do+NOx3WAf6e7/6pb+lnehPH3uMESahn9XChJ9BS7k8bnjJQH2YnM5nf/h9+mPgrE8sLyajkQP7DoL+vXTpEvqVnBUdf+OxxC/8wi985asPFSpV/AbMxt6e/mA01tc7wOqgtDQWTWQLKcossbjIUFBJAZ4eGECe7L3oAcZDWTmyVHlycy9IUJf54H/lAaMLvaZXNtbf/4H3btk6/p1vfZf6SJoxU+VCpS80NcV8jZRwWQlvTk3cAGf9wFvffOj48T/7kz/+8K/8ygc/8ZFvf+brukGbw+NIBOLVTMncZS3G8k88+uja6ibb2Nvh5zoOHz7obXM98dSjBw7vofOpzkKVSiGe23C60aNV+i+vGLU55pPrpPyu3Mg2JAlnshC8gO44L8V6ajilVSYXMLuKxW0jDsSOg4T8xrUJOkrRBNc34D125JiUva4uZbIZqJDYP+xc/AIeAoLhoQgERAX2NFeF0GCf8pDn/+6H2Dn//CEyV7QIiCLoJCUplaM61pbj1JgWyunkl3yT/cqOUi7t5oEQdHwEeCQvY5DUi+uqIB6/8GdhNshslmnOgSLg6onTedwdTksZqpx8ouQAa1FMAWMaGlHdelK1Z793bJtlZJgu4DH8VMCViGyblWAZkVtUHvnUlsBEMpFGsjS1dhSwSuXUarBfeQJM1SCBRHJmxRyQ8lgScpRMIVWpFxpaQFHQkdUg8bUkIrlk3lds+NWWreXmtrWoc2q6tLwJ5r1sttnNJrfFDu0uFqSemgWWLjcvlUIKywfTRPiXO5ZpILxKoE9ycxybmUEkI9JIIShjzHoWqaOMrCAFKByxofeKgklhcLRmp4ONTBsoRCuBkLy0rytK2kwUNxuiDo4S8kjwyWQPeSudy4lwbVa7On3UFAmjMEtWgDi4FjraywifMP4+QTR4ONlokjcEqC96gZgFwW9OJE4cI1GtE4YGSUZvaOgUcX+lCVsezlEUmKZ35wHwNDa7CyuZlurhSJJphGOfIiaB3qHRWZFggNETKCTcM4YC+Y5RxDno9DA3U5jsjff1t49u9wmZkhjXxFMrWjXmKTcgPwyZLA4ZOY6qjJdIZXF/+d1StMpzBpT1JlFoXrzpHLNNwMNQuE7Zi3jAgoP7WSeYr8h6bf2+uU5bi55zyRMumGfyUDSVBC5aHxYLhdytSEKB++PgUsIKuE6xOrkI5UJlvYvuph5XKrsxf3iC/sVeYdzxWZh/KpZYirKvODTTJgEbzstmluANd6zkralhY80onn3LGuHC+BjGK1cGCQfWEgh1FLcsMa7h9c3PaMiEEh8AiC3GgRycPzkUQR6xqBC6ksuFU0asAMaNKeB2kaS8gu2IgOFLChpBoPe4WepmVq+zEU8p5HLN6alcQy06zNfG9+amlnTqCrwgePN2q727q7fN045FNTq2hT21OD+TTidpQWcBmw3dbsWYS6URB2KrMKoysGwCySJyX3ozFfd0F0/wbmh9ke7YgAtRMpGNdXI8dgob9Ebo6JhfesCk4zl3R8/AlsHRXbuWw9GnXqMvAtxGqxOvnj22eyt8Dk/+8IdPPXuqd3ybxmoAhViuluLRILahDJpaDaEHThAaiKa4q8GoyuCoT8/2b90HnGHm9OnxvTtKV5+hQxRuZSgU6lxYGBjeEk4WBvoH//FHj4LNHhwcm7gxt33X/o1wIhhJwkt73/1v/uZXP7+8vvHmBx+48Nr5eCBotHu4P3bRsdsOzSysoxSXl1ZHhoawiDF146lSHi4FlRYHtM1lf/Cu2+vFwtjggBW5l05BmY1yvXLxEqBxKKyJKzLPm+vryRDczh6aOgTXNqDUgTgFgYBlTtCYgPmHfu/3vvcX/+vhhx/et3vf1MTM+Pi2tfUNRvrGxOR6KITLS6Ynk80ZLbahoZEXT7/KK8FQxI13w0ZVcq6IGCrRgGNBGC7iHMHNEsIsFTQGfyhLg/QNteksEmYOtAEaiL1NS3AI8Kwml8/9o0d/TPnQ+z/63u889I/I/fFjYxuLq/qaNpeowUkOugspASp1YWr2H3KZ8W3jn/jkbwKnv3jxgnenPz4VbTvsS6zEtT5VMZI3Oy1Xz16wOX00p2MXTL9yIZVOBJcWLi9f+Z+f/SObx5Atk4NURVIBj9d5+erZX/+lTyUCyWtTcwJqRRjn63CykMfFGi/lFLpBu6p7oI8dt7GyZnM58G75GPBRYGig6YWPwmYf27Z11769dFGMxKIkPaGldFgdqVCafYR85H7ZSrKpGBtl6/GktTeV1/7//Pr/8UnsHNHryma9+S9SSkZfQrdgTsWl4XSigRt1mP1bb7VO9k9irDVFrVdvvqdIAISqyUjv2TQV9hDLC9pWpCImFPPtsgh/bZbOQZVGm7dN5dBUkxtU8fZ1qEbGVcfu6D15z1D3AApyJV9cMBpI9IpLBr0W4oNJIe7G9ANYUpxJ5AC61qpRWevy26SE/pVlw6ABVc/mgK9n0pT6x4BKsdNVlB5h7VU1ubyJ6ESu0llVt9e1ndG4Z26tOrO0Ec+am1pnW3cHXbzo9eNy0cWwuhEIlIsFi51+VsScSTBgJaFxUSeS00WlYKQyeLJ4RbyJJ4P2lbHgMpg1VgALHkmNYmJYoUOgxzkjotEbvW6qBmjZQoUSYpNeKQwk36YGiYAhxpwPTgCPg7SpyYjFUGAX4DBQVkQgGoXpsJmka0+dWp8iWoIEFjk+I7WJNJbB7kDqNdAHEIDgcMPzr3Q3oo+FiHB5EKIHywyVG+pQiP9hjiwTtacAxuzwt7vd7UTdQ5HEamCNCnuki87oUqwCnDEtuDUaOdFrhWQeNCwSD4U9EEhXK5ROVRIxz2hENTeX8V1c7Rnql84DlJtpKlD8S9wAxLki/W8uPFQvopIEOQ6oDIH8yPUravgN8d16ERmh+MGiB1tOMB9gPP/p869/i4O3DtW64Z/93Vr9TAf3K6aR/CuTI0pOsq1A0MnNs+wwkmrIUFlyDKQoSenK29paGDKIeAUOCgQMOh2w/hyFNUARHegniRyAOuIgimKHCFRcc3b2TcODe+VIjGMVZukWDcI/XSMLouXBM4EYgawtuUaJjQvtGcvtZjxMsUhwLrlm0cFsVhk6qTADZUDFLdZqhiApkwOEElA+Xj0LlkAK3UhKkhfAQBNrkQg/OHUVyQi+n7JUMcnsr71yOg9Vr92kKqYh3xzubqOKrlyk4g6BDTyN5iXgYq2u7m31halMLOK2WZLFDKuTaLjIQ+5WbhHvWWQIcocdw42kMgVI44iaEgYnJ8yIMdM4f7Tu8rW7Giorvfayqazd6cGqhKiyvX9btlQNxlK7PD6nRldq1Ozapt2sdwLeTqawFrs6+9NVWNlcu4/cNjjQ84MvfGZj9joVfFIzplHRgpc7JewMVHJkZFu+rppd27x+/fqOA4e/+NWveS2aa6+8kopEOroHUsVqKBoxe9rz+XK71fqRD3+U5PHBe++rX9EFghsoSEiVz7zw5M5to9t27XvpxeeI4lLROzQy6m3vMjt8M/MzkIPaPR29PT3jA/3AKygpHh/dUk6Hrl48XylmxocGFiavQBRQzsZDq/OlTAJvYmoiD+SbhsGA5kLhKBRIWNxDgyOxUBRDHMwcF49aPPvKmYZRZ7Rbod3x+j3FxUVIH//+Kw/Bq0Xv9EsXr1A9NT01s2PHLgQDujZTIEynhv2KcT58+Mjy8ipKc35+gdyc3mTCJWGHsvZZU0T2iMuB7YSnjImStY2kkQlTdonoY+lyDSwZ+kZ6UAII5D2UK0zLrnbPa1cvErTv29HL5pg5Ozu4r29r99jc5HwwHC7FpP2fd9TpbvOA2T4Tj2L6/dYnf/P73/m2r6PNt9u3PL3Stb0dSZ1M5fu6+taqm3/w+38ERjebLwx9bOhPfv+/6HyOX/uN3zBYKEmvbB2H3Kx3eWXeYiP3RVzddeDg/mtPz+kB91OWCQcyNVIUWtJkngVH47JuX99gN0AH5o4QqCqnsnaaGM/wZogEGk1yEqn42fPhi5cvESTfsmUkk89i2aB9DRb4xKixlM3IdiOqJaJeYk6ISraheBo85O1/1+PmN1snUI7Qcj04JCq/CPcEoy+NzsrI6mKhpGSqZONI0O5nTtv6Fq8pVyrbnwfvA64UjwwSKLAXmE0CEREdVMmp/A6ju91DCQ9Li0wPTRgp9ADk/K639937wGFfh7pQWYnFVrSGrMdNpkmJjyHZ65BUCNu71AgC7KLaEZWAhUK5UcNcI55cxwhC35V11Sz2D11ui+CrMhLCoSkIGRSGEcsPCUScJV8yVGrearPT7N09P5ufmM0GASE1TMWmgy4RjG6uXHP6ugjPLKyuk0ew2W0kbPOpqITzwFKRRyOpIOhc3Fp8Sb6B7EcBMB6iehV7XxH9XCHnJVcqQoYfmThuimGm8g2KDPYasWNEdzqbBh6A+oxFovBYmu32zs4O3hXRDaZAg8FMF7cMHD5UEtIuC94XfN8KXcBxVKmw4gpgdRZqKvxSOkoRGUCjsYsQunS9BJAgjWJFGROB0VDvJ3UISq9h8dbMJMIpZMBXMli9bV0WRxNQp83um5pZAoAPxFi0hMKAZnaTRusiHUqwhFa0OpJ80rddNBP7FTUsHeY5gxTAYgOVkX2N+dlEMt0PMwgxd7wZyl7UdDJjvFrLhkXTMgcZMTRMaw2JJfOzP+Jr4kDKqpeoA2+1XpH75hXRHjBeoY1lkSpfVJ7Icnz9ULI2lYcsVs4oXjcHkeXMIVniaApoG2+uaWqtpD8HgWMaApPiRU3I2ZWBI1srfjPf53Xe4X6ZZYwK9CVDwSflcpSSXw6I34B3SsEU3YqIaKGLZOXI9TO1pAOwk4j88uCXXDnjyKXJjcpJ5ALZPMLegv8MhI9FxM6sUxYs64qceI3KK8x91qLgGBUdrNTbyUKW83JMWHIojoRrUA2xK2yWZpPgtATBgSzGL+abMo6tvc2VV+1W7frSUt+WMaLVnNRq0ybWF4YGu+CF4CJcFlPRoEtliwtTUxtrSbkOQy2VCW8u0kTIn5W9IHRXsgJZvyK/WPFiTPA/08FpiFWAh0rFonTQo4yCkJtk2LEhG7XlhQXYmB1WW62ao12P2WT3+fzwcezasd/e07kRirQN9L/tHW9//skn56euUe63sbzw7atXsEbvue/+R0+d6s0UhqmlNxrbfB4woLQ9YFl4nC5wwpFEOppc+/hv/Pb8ysbEwhr44Vw6dfddJyZvXAezHYdERmcc3roLt3h5bU1tdjFbVEz2DY0Wg5GpqRleh9GGzvC37D10+qXTt775bfOLi9NzS21e59DQ0PLKRnRhaXBs5/rSnDOVjm2uHbj9DpXO2rgQsRlHxvshUkh09W5LrCw2k2s/+cG33NAXF9JQTENXmYxGQU697W1v+/7Dj+KoFOCFCMLWacYJoO0yZVnYW3icQCWBf0DIYHW5uZ58ll51zVgcXLdzcz0Cxz7RMxpwsAzpOtXeoVu/cuWue+5Db129MY29RcXR8sqaWXi5m/BmQ16A6GKoaHeTyZUpiKD1laKAlR0oCkAWhGwIEhTsbmp1DGhwmMzE+sUNluRPsxGMhTJLGRY7K/MXfuEjX8l9ZXl6rc/XP7Z1axAD3KCyd7mIB1DntnXfrunZmcuXL//FX/yFu92XTKeQfZYOa2AjrCqqoN4Mb0agNBkdHJudWvruX3/pqz95eNuBw1PnX718+Wp7r2twuI8oFKmPoZEROh6y0V4+9aLb1MVeZrWp0nW9HeoGYngGlV4YXHUmpK5mM7SxtroOGhKaQ5VdmC7EDyMZka4X1AUGzdXWtrm20dHZSX3XlWtXISzLZ3JWvbVYB70l5SLsbbatSBh58FURdPKPPFGkgOwbOea/4SE7TqTeP3vI7uaoHEwElAT2ECzEDFsfY4fy05KQb7zS+lMiojz4svJd7Gy2HGY+F0odBRhA5IxZoJDwM0r3eqtZXS1jsqp2bnce2DfwjgchNq1p9ZuxZMhgLvv96Fc7dDp6nUNiZgheNQXTBsmvixLTwYSLlpefJiAjnDOjJDhZE6UC1aOVfI2+JulYCodbik9xyPmMxgS8LJNvJrLaQs2jNY3pHdtevJJZ3tBDDM81MXeIcKi2BBCj1ofC4SJRXGQhcfhUGJkhbSooSGRcABKjKQinMd2i0KhNzzI0irsiMkYRo4yGIg0FaSNOB5JYVAGx2qbG7fPRylfg5Y0GvBrABrEpMSsykTg4GHdvD4xXdPIGfIH44quw7ZdYQPmctaeNjkwpIzWfgEvyCGGkmwwzaoJNIv6iGAV0OzajoEUrIEFlbTBujAQ8fAQj0YLsHunvKTYDpegWPpIj5t1o2J2+bpM3nSnGY7nFtQWqnsHVohvEWfd40dN0xyI4rSOjY7QIqSAgauwI3HWEOQglEbAcm7mHEYnOkSjgeDkQUp0/O/emt2+vVCIwwhPzkg4Qkq1RwvJydcr/yhpmPWGptR7cDqcWZ0/eIkSAXcfu50WlJYPYQLI6iVLinnNSbAoKHFoJYFmzKHkFHMe3Rbsom4SB4BnjLk9YTIpNJP4pe4vLR4dRVmGhKIigfpX0HoB1M3F9OMOBNkB2IF0ocHOlHyRhB0ni0x1Eo4nHc5yBiiP0LBqLT3N3jL9yXiQX3+ekom05LzpQyvYkfM3aEBZA1gTGCxdGGT3rhqQca4tOU5hNgA5kRSmAca5WaYzENpCV0QroSi6XJaDIBOAV5FaZaxOiFT4XjDdwdeQtyqhsDi9OM1scOxQhQgWIsgY4qZQGce8l2q+C0dc34b0DGpoIbmIzEXjPR3NGk2bxRrhJk284b7TmPF0lTH6AyZlMkqZDHrexkUyNdnVByzHQ2YFqL9cx2xpam7VI/JfaOoaCLuiSVeFCcV4NtMZjT6TTKYwcsWcJg8JoynRUkKQ0rcsTmCEmjEYBvwAnB0VEYB0wJv3+9ja/d9uWYTURoUqRIB2hn7vuf1O0XMkVal5/u87pnJubhbkMg4yILtYvKWqK9smD7t6z/5VXzw5sGfP5PagTD6WODluXx3AlMR0PG9/1nvcEYqlnn3/uxH1vodxwZXISpbu4uu5r66DgdXJyauvhWzrbu0PhmNZoXZyce8/7PviXf/5nMDc889RTo+PbfG4nlGKqYiq0GIWAaUVbHujtGfIa8kvX4rEQIfHMCjVD9dvuuf3a09mVxTmIfQl65FR1KKQAfCysrOuMlkAkht1NsVAoGu9o64CQkAal1J6To5qYnbb6vG6Pmx4VoIQ3N4JHbznW97H+l58/tXvf6MT1G4FQxOawz8wtBCJxdhHdn5965lmBnev0K+sbNOEg4gqojd1KvRBYS/ZsNo+XTKoFYHMTVDarmswFERIauDPysiMETVsHrUUCjGWMU4IDjb9OOX94PfKWd7xtZXP5lXNnmXDWVSqT/pWP/epfffqvXn7udG9/P8SHTC26Xm3QRtaC/+m3f+PHP3n08sXzZ18+ZbDhXTWpMiLmj4zlfISLSrXS3j3bvv61by0srYzecuxXf+ljOiiEvW6nw1kqlt76nneo9eVQbJ0CdJaR1aTiu+emr+HpGhFhIAzRQuWKqcNbSOebBA64L1spHkx0dnW8663vJD+thL4vpsN5i9tYSLNCxQcgaWJ3AWKI//SJx44dP3b27FmWaDKSkvp7KRMQZ5SH7MHXlZ+4CYoY4fcbWpQdpXxMETR8VHmI+fyvPeTrItSU79z8APpMhBaOaUtwC19s61N88HVJokgVOSdbG4klNi4BSwADSlqaWJfMEsT8cDjKNMsJ6EgEo4QF+r46+jNPo9DoZsrhUg2Pqnbu8Rw+PHZwf5e2uaDWJchQuazkSSQUxpUzfs0aEg0OZ8QdT4QYWSQWRxWLWjQeyBupjRLVg7FCHqkYDoRzyQwlwJhAlLQSWaNpNSS9MCJvxuqZusffd6yzfe/1heJzz8yvhpBA5iqnQPCJNCTmDgxFE4oExIvFZZGAM8gZhIZEHlHxSEAyqRJz5jII8rEe8RhqFdQkSx1rX8qA+S7al4MVCgYIIp0OsIzFRIIBtvj9dNRGz+WKJTIRSFcWOVKcBQxW2QUVkNNK/ImjQcqLiiWOLB1cmzAo6L09nbVKPhxmmxB1BM9HGJggJ8PKCmwVpqCNELXqcDIJmy/TAfU684SXhhBm+RC/x/mkTjudKWBWWa02rpY8ncVJVMKOgxQDa5ojEKYpQOdV01YSeY3FoWY0TVaum9vClsVhAtGOnmuVhKNHGuBKhYi02ESYoisk9onm44R1VaZQC4SqyyuGxfnU9h0DqfQcWG4LJqqaZoclmTrWCMPLWuMX3pgBDaaDnYmiQuUVsSpY+7yNHFfWmTI1Endl+lshaGLREo7GSxRtoxgiLB80ys2HfFt2gqz71m95j+fyw2v4imJ2kt+2aIU2Te4QnILYkKw37F+8fM4mJyRxgLpo6W+VCnHGu2gNdC3TYDKRvSJArylkEV7U+bBT2VeMFcTc6EXRf5KI5eScUvFQxVUmTQuPjxRxsaMwdtCoUOJIOF7ulz2O581WFB9bLC1xcqhu40OCgpZL1GAIEG2CNBW2CyI0cltoeAFKyMiwGAU/IPtWMPAA+GhBKGY1Myh3w7ElbqHFTuZ9aIUwZ2l/jQUCsybrhr/l4NIsRVuzGh2Cla9rqYevl/PlQoOKw16vt5FN5upVUdd6bRHnDiB1Tev0OZD7OuiyWC7EzukTrNjmyl7B1hTYFw8umCfcrISsGUs1B6F6HgBH3mgx93R3ZHKVWHijTHrPYr168bXV+XmrtpmOhTwa7Ty61u574dRpg8+/a++e4f6+M08+gWOtlrq9fBtcyuEYqV/O3tPXd+josSeffU5r3Txy5PDy0ryBKgGL9aePfG9Lu4Wq3HAiBvrM35EB4Ty2Yx/9M0rFImyUgfWNe0/eSdv573zxb9//wQ9cv3R2LRKjGHhhsUKtwq1HD0NSBvMcNE/UXjkM6mQwZGo4nn/s0rED+2iS2N9NNWEptjyX1mmIYM29uLmxPEOmnJWdSiXxQXHWTLBhv3pudNuOu+6ljCe7srRkdrgKtTotlJtZqCjd03PzfgBf+dxmPO5wOcvIhaamY2Bgc3HJ5nTd/cD9FqsNtzhC1XY2ZsnmqCYr0gc3kYICBa+QaAoEnFCN1UKRAvSWBFWRniw+AiGYkDLNLA+sQNAibKUGxiUSHGMLNBOhCTqGk21AqYE3ZpPRuZTIOVX9X/3rr3iH2z/xiU9889vfIHwHJQh47f/y//yXr/6vh9ZnVsFF671W+nsQ9/J0d1648Nodd9xB26jzp04z53BfhJYDZo/dYrOUkqX8Rto30nPHbSdo9LYRiNCHolQpju0Ye/alJzLp3NDWnl3b9wTj68+/+OLa6tJwX/tSMMxW6ukbWJ2MoykdnZ69u/fC/ByH6wMqxx5rZ1+ny+siOkNROzIJq+a//bf/9qlPferZp16ks5yi59T5SOHYPUdhHHviiSeWr6eXl5d7u3vIXOSiufWlNSQ/ghjpiUHJ3sGPQuaIAOF/EVbykPSJQharvPYf/MUUiMxq+RvypOUoizoU9a+80votkkHCY6KJJUfWegu5pbxMMaGL1nhglRBLFgOpeINsbFos1QsWk6qnT3XrHf63vH3f6LijVAvkc+fcTiiSKFRhj0tMVywNospNKP5dzYZFrbJodVTt24AAi/gjfVqKizRRHCi0H6qKrYoGALgAF5vb4cglc9B6GIT1wQSSZCOUjeXMWvOwu33fZqztwivTs4FKWeNOQWFhALokP8J6R00s3YdBF0mWF/mP0OEHBdx6gnBggCSNxTWw/rHDJNQOu7jJBOAK65ClLCMl0yb1dbZuH3GjxOoajTMt/nbSvcTGsFvQi0yl+AlYdqUC+AFK2W1eBwEn8Nlo2Qqbp15ivol8ooDt1PxKnRaet9w+ml9sEPHWkM4sB3wrZDf7RrFKmsIkCo0Px8fEZLciMMGl4EpzFfFUAXnscLejzjbDYXK6drfv3e/9cCAcn5yEWDYdjmSh7WftASHUUyKGcaqHMhWmIyEQpYmK1++nNFG0hYh0KVeHrcnSsGFNl0X/StAcS0XeRedgYteDpfm5xsSN4Pa9+2rlgM6QF7WgsYg2IHN80/JjWICFkV7gjgx0pqSGTSBIygpjwDGQOCyymeFFxWK8Ep/hh8niHmrSI4gXUeg31bOoYVnMygWhDEUpyaP1hGvDo5Wjt2xM+QBOK3uMB84rs8CJJd7LcwxM4OK4CygirCuUqGRiuHoWiICkSI9BxYzqpX0Ys85CRzNijkt0ms+xRBiilofaugDF8eU65QpEwwK2IJTcVJOLZ9zAb5GRI9dE5TySRZBrnIxghagqPMkGqGvC+ULMoehYTsO1c2kkIERTit2CquZPmZ+WftNCcCMFuHyVLBbeMItWhlNuWjxp4W+TeJJGDAUZFN5tVmG1JPjIbZPHppcCNw7tdNPKyDNx3D8pD7p2EO3Tr81P0z0XQUVmmbsFFQh6iHZKuVIZd0oCg/hN9SaRE7XsJskZKDudOcUrlk1NLzKsMkCVGoNayqprTZxXOghw/eVSRq+CSCqVDAc1Tjf6qd/fPrR9bPri+bXZWax6p9c9sHV0x8FbTl04/3d/+/nDe8bvOnnH+sL0jWs3MJowkUdGt+JidvYMTE5P9fb3QdNIU+d0Knz1csbj91E4YHW1tXvdq+ub5lTBaiMaSmWBY21qsW9gmKUXCMJNnx0Z6ANc9tQjj2zfvuXYsVuXF27UimnYhiELBFwai0Ywh/fu2UccGCFORZNFWz39/BMdXleglNi5fbTgAFCr7nCZ5+fnq6UC+FDS8naaXgBXM1vaursXN0M9vb3l1Y21QIBCPtoDxSNRe76UT2cJ+/s6OtGSx44cf/TJx7kjEMv4DnNTM1cvXUWTBUOhVD4LoIpwASz80CJjhsPORqo+mc0Bl4L0uakzlXNE2oX7gEQQS5QtI9uHJ7IUNUgXZlWsdcw5vGxqtWGmhVtPrD3SkLDds7YapAfK0jJEzZBRYhRfDFP4RGsjqq5ZPJ2d0uPot37H9bnPfj4a3KwCvMfudOre+vZ3BoObp0+9+puf/K3ZmfnUWsjZ31bWV4rR7G/+/m9/8c+/ZO9vI/32B5/61O9/5i8+8YmP/eEf/uG2nVuf+s533NsG4Mq47677B/vGfvTDRzKxUjakCqrjRw7tLeWaMzOzrNtsMukZaIPtQ6KuNIrpMNOcm3UimTeT8fLFK6uLSydvvf2//z//DWn/W7/98S984W/ZwRjrXSNt4BUIJ0YjISBEgvRxuWKRWJenK7gWwBNhf0oCnkUuIBopMZAtjOJBEr0uUuSlf+ODseWhHOzmN2WTy0P0mgifNx4KfkIMaT7/uhrmTYlksT3RAoq/J69IjpUXMWW18TDgHpMVhgnqfMlA5aOY9HBCbxlUHb+t9/gd/SNj5vbOks4QtJriHk8NQ5PbwQDm3GK3w5InMo4cJthS2E1ceq1PZ/CohNAKiZ+XnqO1PCBTCnVK2XIGsB90coW8w+JMJ7MoD4ojHKDe8s1kNJfK6iJJt8m9zWgfW49aX5vLXl/OxSp6laVCVAzRBaEkIg8S6Hq+Xks3i1T3WnHewU2KVhM1zI8MADpCnAr8VQQURZCiC2TC0UzUgYteFq9XMr6EnalJoxo2g4jVuDxUDXg8ENuY6IAJ7ImcqLguiNhGFSVKg1TKrWjITY6UVr4lAtqietUGCK/E4acTPJFBjDAEpwyPiEx0MIJLogG8pmCRlNgef0kwWsQ9bhvemAFEKKUc+C4Wqxtrwe62pnPlxdUIord/YPvWrds7OnsuXZlb2QjQgzwciOMWQQorJAQAGAki0Svb5kClEEg0WeF2ba5sgISV04oqwpKlNzAIMXJ18CdgBxBA5nXB8TBgFN5AQZFRbW5Cn5sLrkDP66fZDgAlLGs+LBFXrru1uBg/7Am0BtlWDd69dH9CaDHyvKq0KOZzEjDgFewXCf7Kb35EN4v2VdYiEyEfk0XMVSpJVHlBHsoiE23FO9wCSlduQKwamVmUMrPD8lXKtSSljInNSKOjUL5ITLkSBU/HcZTjy82iZYUr0oiTiYWDzSRYJ/YnRxeLiOvFRFMug6vHaGL9MEVEGwRbyuckQgiACwiDroZnwVUqdggalXOih9BPnAvPmMmTJSiuCsXwEHeI88i1y72TFVZ+QFsw5ghW9gCgO96Su2ZTV2miAwTMjA0hbcv4AMBIxgHyUlYrwWrYOQROLi2yuaNyARwfJHcimEETENgAkgbnETnAIiSEsu50yHFidzTai0fjyUiQU9tYB6C1SYBY26BqoiZ1cmqatcgwmEXKU2BHFX2RGktR+xya+eF1ZSxZTew2wF02qwsPuFovAk10eiCQSiViQa+zDXrKUCIKuMDT3r1ndLy707188VUBydnMQ+Oj/NC2j5bMwaV5/fZBqvgwnOjKnkykjVZr/9BwljJiLerbS1PDldXFteD6rbfdlsBrz4bdHgsNfErBkrezg5L/dDI3NOQFRrJj2zi0/WRYqfsjA5ROZxx2W0eb5+zpU0PD3Ri5gXiQXQ1fdAWiFek32Zi8MQG0AzPU73YtbK5C45hKRKz6JnFXyjD4ShrIBKRjdieB4GK+DLk3A2iyODp6ug9pdBIrK+a7u7sdvjboq4Z6etbDkVIm67AYuwd6iSuIXc6koSMbzXAicfbcBdpDbd267ZEf/zgajYvPqjOQQUd7QqITS6UxdQi1EeBj1pMEoxrEmTNkoYS1DVuoSi0DdRCsKfSv7BmkEgZTy1oEOE+LQJYxAS0UOh4FC4mmsvgQFLdgQTsdLn9Hx8Vnz73w/EtAxm68fHXKMI0p+fxTL2Xj5WgiZvS4YPThCBAXjY+PM1bXJm58+xvfTi2FzN2ePbv2zczNhqeWCSB/8j//5z/85O+RO6gbjQ99/e/+9E//dOee7TR0at8+QkQ0dH3x7KlzyGKfp+NvP/cb8wtTf/EXfyxNqaO5eDSHx6VxGE7ceUc4GCogOlWqrt4O5iOeiluc1uG+MZAE5DJeeOEFEgp0/mDzfvSjH/nqV78OZwKT+sSPn3nJ8WIhXNU7BSRc0Oawgqmf5mPgzxX7r4ZtjRsdjydaJnXLAJfNzZgpUpkPy077P35w2H/+WeWFf34cJU/MzEjoS8TZzQcfa52Q8JvsL8KIEl0TaSJyg4BZoYklC0tis0llnzR42Dqu2r3He+TY8PZd3p4eegQGy+VAuZaimwL9xqjEUXQ/4oQrQ4RgHiOZDVLEC+QZEkMalOrdJNkQLljYGoODUuBMmoJ9aM8A/CLSUYka6vPUDYdaZ8xXrMvBejQK03ibv3MX7Z71lh0rCc2rl9fmgiW1o8vhsCfyWdiKoAgi0gXETlOkCQZ+MOFuFC5dg5S+FFw6tUZy66L45K9WsFKseK4NzcDIkARGl8AjBTUFCWnGBH9JUSsatZvIT0cH8076H8dXzCnEofTZbUDC7LBRgmRnJSDzSahCcEcMGXpDgwkRKxW/nBHRqwGmKj9KpBNdhHRHruM+QiWECSGKhCmQB58h6k76BveDP/Mk/4gq4foaKdYygjWx0IbcZLRB89kzNDa+A6ny8GMvhIL0ukrT3ZP0ptFuJx2H+ua+cELAhBLvBHRJ26MUwS2LxeGxSh2wxKElf6nGlS6b+J6JXkxk7sh0MyCMGQ9CnKIfVKpEvB4K1s68cuPuu7drtKl6PWNokL2XYm15iCbgRpXUgixNWU4ShKTrHl4dZhnaQtag3J5MAfcut4+K5BXWH/+LOSKaSFYqi1CuAc2uTFbrHByUd5hKxdm9+Rqnkh/eEcuRZ3KwhqRg0ZkcERgEBI6CNyN6w3XA5yy2MEoTt5uTYHW+8WBW+MHfJeBAIFZ8C6wP5kxZMzImeM8CSkb2Sf9vFCBIQsUcIHEr+lW5WkbeBKSIoQFUzoRxabKnxBVlIuQGGBHiPZxanHVFbFKljGWkJxAtV41+F02PpL6pgFEIMHPxIVLC2BEKSBDVKmuIRWXideXGFVVpMWJGEKBTUFIscmrXmEdsMYZHwNumTL5mtFtokJkuVgf9be0j46lTZ8xWE7EIgPF46j4rAWAXhwxtbmBfWXE67BauitQCcb8cFYdUdtS4EyS7cBcSnyfOAARfraMZaskF/YSOUA08a2rIVPkMndkoMTFijyXzkiyyF2MrK6W1ufDaKq1QNDZjvJh++cKZu+950Ou01dPxwOJse5uT0O7IlvGlxRWfX+v1tQ0OuTrHx2EW/Zv/+cfUM/R2tzmdYBHhu8lbaDrhdl2+fOUWb3tnb092fm1tcy2dL27fsde3ZSS3utnT4cXT6+/0P/nk4ydO3Lln77ZMbDO4ucKWf+fb33bm1LMU8rX7fWfPnu+CIcKDS12itUM6V6K0AWN9en7RblK7XRDbSrviDiKA5nwaidHQ54khBGMud/PK9Qm24uLyUiQWc3o79h488r2Hn9DZ7Ml4lEQaVtGTz740vKWP+DMxc+zHeCIVjaW02jAW08zsPLUrOL6yRCFRofUpLYfyZbrOYdZggKt1wmuMU8tyxDK2G815gLYsVALKMv6y4dhHItAUxcBqw7IkBkCRP8RSoPnNFjX4D6nBrNfJQqH8KPbFnHrz29/+yx//xF99/q+mL19WWdSBtWBgYeOPPv3pG1dmVpZWuvp6l2dmsASJCX37H75LCpY4xMTEpLWrLT8fMR+3Hjl0/Mfzqz/4/iMiR8lq5dIqqymwOP1nf/7HuCzTUzdghuKtzh2DbCV1VbM6v/6h9370tpPH7r3zLRcvnE/EiKVJ/ioTjX79q397x113j4wNZ7KJd77zHcdvOwaaH0oQG4VGW0YjmxsQrKJuR8dGSBmdeeUUG0QiAIq8wJ/w9lrZNvhw1G0fvvW2V188y26QvaQMCGu1FQnjTx4iMN54yFbDZhZJ/W99IEB4KJLy5ldbvi9y8ObfrX/4HJfyrz3YwXwd21ckpiK+YDJGQ3rsbqB1/OhMqs5O1dZdqvvuG7z9xNamOuj2hEnWVytJh01rMruhbaCchRiJyFXp3SviEcpHlC4yAx5eCqMEfgWbLjPE8iJyQ4elfBJmm0Q0lU3QLgnhizZAjljSyaLO7EoX9Mub+VTBZnFts1h65lL+xawmuFoMhFPJXLOmdwB9aBZzFr0OXkyYNYAkwL5Yr3J6wJikpXB5Sf0qAUjxjPBoGX8ZJwUQ0xr8mwIaGc3Na+02DG78JbIqgI2R+1BLEj4cHB7GKGGugC6ybjFhZVrV8DBLtthsMtlsJrsVK54wW7lazjGZKFTxpoTIkJCsOA/YPzIsZH2VjKRyPJw/3uE6IWWS7DV/iHYWWibCUFQZiONLLB0hbDRb8QHI69EeJp2r5EOrw6Pb33TfvaRiT50+OzO3iDpOZ9mJeC8SazUSr5JEnlwBAXIwz3iEkJfT6Wl0x9Yto0PQf1AaK+TGTIn4NQ1DuYRDRaW2u1gMsx4F+KY82HpkE8H4UCqysZY6p57cvpWgHa4i8ecsARLREgw3W1+8YNGdGHsoBr3e5PU6gH2nkrIYsWjg0EdH4S+2vF60Mk8kwakMjvwpGS2GWhkrDiXHU7QrR/2ZB3tJVusbUR5RyigXpkBQ6iCXUHcEtfHO+QhhdglDlTk8mVUUnoDLmGnlyGJuspDEApCogwBKQRGIYSBbV0J8yDumD5uQE4q6ZrwMesLmCA1GhibyGg0QABonEUVATJFmZhdIPpnezOhfohYQfXJnyEbCwAgOWYkcS9XIlUidECFEJVMShl2Od6InvswiQ8JyqaK2GUfRslgJLEfeR4lyCgw5GRFwBQwgl0HQsqEpY78qiWq5IUA4OaQygwsYgG5BmMQyTSxDjZEOX5gN+Msmm9T5Wi0qn5dAC7XkEhVh99TovJIU/jRqfNNpvCVMVUwpRo2zy5gSyqMhmaR8GV4cWqlJlZ5iRqr+TT0DXUw+EDCP20k4KJkIAydp62wDXlUvpUwqTYfTrioUb5x7rZoPqWvlaCp1bN/91p6Bnz7/ym0nTs7PT2pK+Tq9ZEzqmYV5MGwEf/q6+0MxSC2jnQP9S5NzDNHOPdt27d4WCNFoqAEWlN3O4Lj9bYl83tfVZ8PqDMehkn/6qceHh0YX5lduvfV2esru3rGTyt0ffffbt91+fGDb8OZqaeLKZbOmQj/ifCaD8gIc1t7ZBcQDpCJFve987wfLhWyn1z0/M3H29FPkPNMgwUirOrypdD5RqDZN9hMnj1+buI6dSy4ZC93r9WwZ6r+xsHz2m98Y2bHt3OUJj0XX7qa1kQ9+MSQHy4OUFIySJAGoH4zE4jRqTKfSToaFOI1Sa8YUY9dAP88OIF6ICUuzPcSVlBISWKpU4c9jWQoCXtG+zC1oBOaD1UqqhDXN4ifjC2wAbDPoFdat3+VAIJSVrIMw9oAyBu/X1Jw/99rTzz4Pz52tvY20MeZfrZh/7LHHAZ/pbJZP/u7vTN2Y/OEPfxgNBC+98FL76Mh73/teiPQeeeTHKpchmcgc27qT0E1bW7ucstmw93Zm6V9rUgWCKyNbbk1nIyaLEcrhj3/kE+1dnRM3pu45ce99d7/5C1/8nN7YoDEzVs3ff/kPn338+c9/+rMGp21mbhroKSxgP/rx9ydmrnv9bUPDWyw2EHnzZ156YWSgn6n/yz/60js+dN/999/f2zv15CMv2lxqSJHZU4ABNjYSmI+hxuaKzQNXdipO8KXQwvrAzBopRZEtmAM82FOtB18Uq/3f8VD0eut7rx/s5iH/2cFakpGZl9O8/tGWmBPRA5pSVK88kDw8uB7wK/F0CAPd36batUdzy7Ge3fv8fYPQuQTSqQUBsoP2oBwUUFUV4KzVaDJSF4sml+MwqcgY+kM1aW+A42uni5KYF4ChVEkWDeQq4kmG56v5NMFnoCd0/SNqSAASesdG0xGONeN5bcM85vaPwbBxZbk4uRSNZk2MJQQYdlublaaoZFDoiQaEMpdT4snoCQQx8g8tTDAZ9BlSmOA7t41SY2lLI0FGu4jQlJvH5ZfVfVPCE9oymBFokIdzhUhOq8Pp87axrQhOAr/AiISyFD2PypRwn44ubUJcRcCMuSzloeYqo79AO9AXGU0gqRjUC4pU6ZSD1AQBgbSXjD9+FE4aMp5fjZslPwyb4ry0CkTBvdbI9VF5xZ7TC2i5kYzEC1Ldqd++a9+e/bdoDdaXT527eHkCUFMJrzuwojJYYb8yGmwg6ChKkrg3ELZ6E/AMvLtE+kHU9Y70HTt+CD6u2blpIZcAPY2Xjw8B8zspWFph0cnL6/FnskkA1ZKQIFeqKGPGKp9Xra4maPZ15fKUzzvoNltIJ5louFrPYs8oK5qVo6xu7oZX9HpPGzmhbCRMuwl0mOQFUSQsOcxW7v51+JWcQs4iSVaBSPCuROTRGWK9Mnc3j8kRZBHzP9aRTKucRB7yoigsGV9AyPBpAOhWUv7YlTxY08wDORGORGqfSDj6lLNxcDSw8DdLBI/jwGZAhyLQLqgwuEbJaojKQQGKgFPOxeUoHioHoK4Qpc6NKMeXHqusJmHywozAOEQMon25EMp3EWp4UQw1aRd6rTM3LFQyHaw+UdgSM4S/W0sSHBsSESGDyUChJVkgCrQGPUzgE1WNMcArLVOAc+EWicvE4ieJIIaImEPlfCVbTfOEi4ZQkLgwz2W9NemMRPechs3jopAB5hnMukgiaVhezJUhvmOtae1m7FA1ghg0LNcByj+D5YV1zPXJmiUHXDXW61KA3NRJCEAKPHC4DRTAGIy2psrQ1TPETSCa2aL0f0TesTFKtCxUNwrpqN3U5TYZA6FkOU0RSdbf7sjXCk6fa/vddwZyxa889JVqPHbvHbdGQ+vaNheE1QgLyKGGt4wwWy+98KK33U+e0uNxBUOblVoW47ej24f9m8mnkN0jYzuwPQm8M8cMHWCEDKzUWs2WkYHXzr4CafALTz3+S7/0S2978J4f/OgH619fvffuOyhAYr9+6AMfmJmZQusMjozg9VpL9YmZybbO3nxTCw/g2Qsv0H5KbXJQ/+Wwe8a272iqjU1zSefuxF2Op9MEz2qZLBVZDjJR9EJi35eKuOx3v/ntD33h84VkPFOkadLGQF/34uJ8/8gQ6d7FFfqsqRx2WlYwfxoSyWsbawgMQU8iYKCfFauQVSZxHJYZPoEQtmskMAO6UqwlxIzsJTEQWURiqbFuWK5iv8piZeT5C4gKmtHpwshTkTtA5cs+A3tiUcEYQpTvXe949/cf+cHq+loO3HW9Pr5/z97du7eMjdPJ7o4TJ6ZnZiCy58Qmu51GbOSyto5t7enqeeap55om6/knn/3g+3/+fb/8scce/+nn/uZz125cpwGH0Y461JeqmVfOPa81NN0em4QTwpv33X/P6tLqw99/+AM///Of+dM/n5279uQzT7/lwbeurqw/+thP2/r6ImvroSIsDZL6WlpcD4QDdofnyFGCB+aJySnKfOPxKJ2Gv+f93sPfecrXY9kyBKtLO/3tEfXZbAHj1Osm22ZlmQY217vbB8gX5lQFJAYUAIQKWltYESAiM0Q0yBZRNnbrj3/T75ZHq8RU//fvvS4D35Bcci7Cc+xOpuifXuXUOF18lzd4Ww7CZhdaAlJAZbNJNTqmOnTYf/CWoe3b3S4vDt5qNLHc1+0mhFHIlEj4k+HRaSlSACCqBugjAWw0L0FtBDfxDTUd3/F6zbBcVCqQNufwPtG9FOvT9icbC4tIquIlEMPTY+wRKMkWmgm4oqE0843rdAMzm/pLU/G1iK6i6wyVc3a6ZOqtBIDpVqppWDTlZj4VJ0go7WyAVbHn6GfLPYo8Il5D8AickYT+MAwIhEsbAS6QvBtDx3IUx0IxGSTrT/QN+BUvag00kqGDKmgqK0w++rXVDYmHImyRdOg0ALaYGyYdOR2UKWgUAoK0g8U/wgcgYk83VQKgbH9RJPBAigXLqONU8Rm2BH4LR2Lj/H8rew/4uO7rzncwmN4rem8kSLBXsRdRvVmyLCVuWXsdp2eTbBKvk2x2ndh5efvstR3XOFbk2JIly5YUyRIlUuxd7GABQBB9BgNM7w1T9nvuyN4km/c+n3cFgYOZO/f+7/9//qef35FcRTKxZQ0UUVxQKta4o7Ia9SSC4HXG7gH/PI9JVKchUKIDKKCh+diJs1MzAfyI6VwlSyMyKk9Mdr3BCfwLzq1fennJQEyl05JaEfLhdt25c/vAsu5YKjTtGzeZDXRchwgpQxI/u56idXmKnNqkAzcEQ56cVxgyS4meLgKSlKKlaiik6ujQj4zMb9/W63ZZ83S+Io0WNDFWHGmAhJNHlccU3zEBAIdJA2gLjmbUjjroIE2kVZQSUT4QCXKWuKBLzIJwTLYIb4pMkjtyERZKCJyXkCd/cfFf/JZ3hKKFgOUQcCYhcfkX15zkmigF7HKOSH0yqiRNipxQGJOwINQf7oriBVdjiCI44T8sB5+I7AclgIfDWsZEJ+WJT7mjhMYlg5mIr/ypDIL1lhwDBSmaGKy0OASAuy6b5Jq4gfkKMw0TBE4OryDkh2ceuBKHRa9EaKU2WKgGU1qc5QXUAy6MEisLKdoPw0SG8Y8SkEZvYbhCwIIgjpqBIGfuYdJaJA5FbTgaCmBRljGWeHbKFwhNyzxzqXy5vpTXgbao0UXjKWMmC6YS0Ap0443HovRXQYgUltTkNGktRlEc8kkYLyTObtbScoJCazYaMhaTHY2hqqUoEGrGkYKVQz08FTgY+dlCdlnf8mKRJjbjxOEAsiPtAIPWojOkcxmLGzEcWfRN2yzmssqQoqS/ThuJ5yKTsw9+9BNL2efOHT54/daNtkY3RfSdiTjpUT7qUrS67p4+Nm9bazM97acm74CMCCT/0NDyuRnDPds2U3pw179Aq8NcNrDUUuru7iJjIxHNet0e5sNqtuVSifvv3TfY13PwjVcH+nopGDiw/6N3xm8v7+sh3fsHz31//749Tz7xyKUr13Lp+MXzs2A70od11fpdL/3wh/SwbO7qXfRVEHtNDto67bx0+ToqC02OPZW6m6Pj+DzpHroYTeSLpWhynM6gDR09+w48Pjoxs33n3tNHDlNNUmcwEUfauf/BU+dOQalmW12Gcv9ihYAQ4NiLgXmgH/F80TwXJcpogUQp6SHeocEXrWhpdXnw+1Qq+lWgRPItVBMIQ7QsIX8OwjfQuWwCDkhOtGcy49gJapXNYcXrSMI/5RN8BepFncPTSaLZqVMn2tvbh2/eMDrh7Evzc35IDDfapSvDTz31FO1+CVuTfpwAwVSvpxPiyy+/TD+GZcv7rrz/vrHF++OXXyBpK704/9Zbbzzz7FM//MfnoEHic0TfosGUyVK/MB+GWA4efJeORvgVgoH5//HFv965e8fWHZufeuzJF3/0MtH83/mt33/jtTeC47NIDcnS5QpCsZXQeOhQ+l2bDWwYcyafj8STL/zoZSqXwCru7hpI4hvJAnUEJImOKge2LqWUJI0jg9D1Q/4YeVxmK7VYaNVqE/hfBMw0wn+QfEyU7HQOWMMH+opYj/+/D2FRv/wSvErh9r9gWUx17RCd4pceO3mLvS0Hb1PBh+cJy0RWBZarrRhNBYtZ9ejjpi1bW9eu77dYcfHOp1JRo3mJ6tZsLmqk0z1ds8njpOQsS9qs0Wh3gn2Nyk79qlpN1B/pSyQVjsxUIiLppJcjo02xI0lQItZRoDc6hgINfYsVTaGko0FrOEE7S3WxzmWyLY8kG67djQ5PlBJFZ7m+sVg1ENlhzkqZYjqdKsQpElvCvCD4AbtCoElWPTxIHqFGkvBtngj/L6xTWBj2Oc8rdqEoCYxLzFySqChEZYwwqirwQlrAM5DyXofdySkgw3CgQGcZfT6DfgGSM1EVSIs8WYAXUCNIyMJQRO01GwllIdKpDiHGAvIBtqzkcnEgJFhngLDZOGTKMAy4FoPDv40ngTRHEUS0fs/R5pANJ8BCWOsEeLQmSzJTCAPh525cvWa9290aS+YPH7mgVhuyOXUknC0n0pTf1dlc5FmR4Uj3Ekm8ztPLvM5M9b9JByxHOOrfsGGoscVJZCkUmssSVkjHaGJHRja1w3iqRVGmlBFVm+KzYjYINpGdby7lUmB656gAkwwjQZIi/ayUmfOnGhv1Fy+Ou1z9juZl0dkLDpdZZhSZk0c9R7UWxl8tZzG+1YVsz6quqRkwcaqUeNXXW+kCjtYukV1WSvQemIGmWjQomZxSxQw5wzWEs4gshZ6BneMphOnLkkKxyteQA3AlSfuF4FlKSS0hD4rOKZKYzm/OQtopVC3qE3atoGAJrAmzD0g6zyX1TpxL6zCa0rIqyeQSXVjM9ALR6VF/yB0TuSWIfRjOslXYL5T6CNVJFRD7FlqrJ8cKzzqWKI4T9AMoCawYss+k/62CsyHPw9s8jhpUKxgr24V8N30okbGYJfJADIAWsiimmVzSbDMhBIUihZPCyvAv4cdAcySbTOaFFDO0BJJF0QKyBfFVc1sq2aRvMWcgFCVELCkG+HgYPDISxRH9AOcnGgfdR5PphD5Isjdg4SUHDtx67cKsj0T7dBKicIA7sRip3PvA/lMn3/EHfZlsGOWZiKNa6+Rp8aDrTPosNQtKaMRgshbCKdIsPQ3t9KjA+ty3/wBNHcKx+b6B1p4u140r75MAFQkGgNRyWJuweqnZtNnin/rE4//w/A/IHIYIdQb3levT1paQe8j42FPPTo2N+H2TmNJzvhnQ0EG06QKgcXxsbuquzea4eObMfZ/+dMOlC2gM7c3t2VSeDrvl4pUNW9evWt5PDtfw3DRpOM3eBrApAnOz8CAjbg10/lj0e9/8u6cef2x5d1s+FXpw/x6csZWu9lQ8QkRs3aqhW9evEvfub2ukVo8GoifPXLh64dj+jz397Gc/9Z/+w6fbu/fvefjDP37hh5oIrdaXegdWzPiONjU0TE3cffYjz3zjW99cvmzFvgdW0GaKREys2LfePRRVnRqfnGpranR6WpdAq4xGbORMGO2lequnqTFenFm5euDO+N0s6H6VosbVSEkxpIvQ1YgLTZ/Og9tD/0a0Kz10Co3hUsZ2EIWHohpoj0gHZMIBbckhiiCvWWXQFthIBaXHDBXwRemaURTHHckBKjWQvEQCxR9ULhrNtrcP/jxNw51KmeqIIrHq+vpGb2M8HPGP3/3+V79OkJhilNjCAs5B1K8/+LPP4QT+8lf/9m//9m8+/mufyARCN8cuzi3YDU2m944c7GhppRE1nee9bjtZ3xVt0aKnNSW5flkcbNFo3GzM+Kd9Vpvq/TPvHnr7Zzt37U+Fw8lwZP2qNSePnIBioWZGXg6r9M3kM1f1Xg0dLvRODzoNm85osg4P30HtLRX1nZ0rgQ/Tt9qgrlIpLywWVQNNGeECx0CPX6qmqDJnRkT9xdxhe5CwJsxR8Z7xL6JAEi/EBYEvh22jHEwlF+PgDA5YszK9//qXcK4PDq4n8y6/YRkiWJhaLDthD7wnjApxjxWgGBW8xYuSJEZznhSDCtQhrX7F08WpLY2qvfucu/d0D62CacU1mrs8i9kg9iWhp1wmB9ohD4DagRJPtJgcFLKPaEFWbwKJiewni/RRUFvEmkTrXypGQvM4hBNJmuGCakl1Csusz2QLJrd2DrXP1FyuGuNp83xYM+WvWt0rH3zssy+9dvLY2VGauah0rSraBsARkVwJbo4bOMX+QvaJH5fIYz05lyCXSYSOCYbRirXBEjLn+SxxNbpLFiJxQSyzWWLRqN3pIkdK/M+EpgnS0koBXIJsFgp1ta+w2zzkDGJggCnLroSR4XYDDYTub16K+Uxi+JKzRIRNfMslcmgqFgs1ccRdFTkrIUzpS4bhjTwlfsj/cH+0G8wu8U1CDUBY8TWAnjNpVUFroJWEmpBXhplHiMJuJfyO0kC7tkb35VsjzU1t7YN9n/jkr0/cnV0MxE+cvkYnblK96LRRptMK7h1qhat1dH0wu2yLoQW3x2MyGjkhVQzhfuvqavnkxz+bAy0vEgyGQrksQduqxywtH0jCEo2PqcICRXHA0JF9DJMHOqRaQskAWIwPoD+hXJE0ZC2pYrHS1HQY7KPmJs36NQ6T2VutxATYE7VBcRQQZOdsEZ/wC70qtbDQv9xx8XRcRzF5HZhnWKiyQDUZjMIkEIoYwYLIIbEBvsgs8VvGJmJL4iEiTqT6Vt4UOSTULB5gPlaGRuWbiB/WHp8EkXiUPgYsw2CmkUKIrCp41yZqMeRJofyar45NRB4NJTjidoCAiY1i6coleQNqEhHOnlKGI/JYGbZcmbGxw0mHIQFVtEBZXXkTnzAOPqXMCeHJg/CaUSFAZUPzsNxejFowXLkLHTDEEUPqLfFqyEMSpGSbfeAxVgiZEgK4BO4bxRcuE8JyyzwwMfAP/OGgdsAicL5LqwZGhLNdKtywVnlSJpZzeE7uwhMJh47EskaTxo4nGlM5lSrQrASzt4z7zkUHguZ2b1/fMhQ930KIR6S+U5UBs7DC5kP260wOg8NMFTQKyuZN99wZmzRb7eRexVNZm4MyoGbqc9g2JNEs5cHQweuF3YUGozF7HKlsxeNtoS12Oh0ZHj7fP9AxCy653lSnJrmx/N67xxx2T/fqwY9+9KNvvvHT6YnRlf2dVAlTKiHpUfRsD0Vo8zdXyN85dJhFBcV90/rN5AMvhBbpOPbKj18d2rS8q6/74YceOnvmYnBhcXRkCj24q7MfSOqqrrz9nq3ZZJxkydamwXgkePrkkd7BFS63MxpasNsdi/450CmSMTKBFlGmevoHSoVMo8v1w7/6Atrf+k0b123c2LZ79+aJqUhw8c6dOxiCcAPwKYHBOnfmVFdHJ+lX9cSKqpr5UALVvn/F2ouXr2FB9vf2njpx4sh7hx44cO/d8bGvf/PFtjZHODH5G7/zu+9fuQzmZTKbWbZyxeSdMaLjBJbIwSdoFM8kyEYhlQ+vCZ4VEp7hJByKgwZVb4k6dGhF1EwoiPVFgkAXCBNeoSOipbFjqIKHVKBgajkyGaPRQAYZJoHVqieVBXg7k1tYqdnlEsnA80P3pHtWs2H/QlIdykUTdSYIE+kPUB8oDkYQ5MmCJvI2fnL4O9/7LlvQ0emKR2kLEYNGOXARk/NB0ROgK0gICfZVAAUrAIArVld5yXctbu+UOqPZy4mWIdPbbx4iixWk3v/2l38ZmF+sd5rF9k1ltA3qQhxMGVUhUWpf2RIJhSiiJL+U+haVSXffgYfm/Yu///t/+vqrPzt29DCCS+HVBdnD+KvYXjhmytSAUQxA3gT5tFWMYLQitieHADWL3sK0ie2L5BY2wQfykB9sLd6uHcLX/98OFkD4J3vrX5zBirDlRDxL/pHMv2w62bxyJdgU73BD5UukRlqQKCrKCsQM6Oygf5F605buATBmGkpmQ0CtTeOKkBwRZBv6lfxDfLGmSwhPpOKBS4ocr4en66paOgGQSYD/REtILAPQHd7mHMmz/mw6aTZaDFpjKom5ULIZHfFoqlz1FiveOcCGp6KZUktz185t+z526PzMqM+QqXRqDQ5yKpWmPTDtcjIao6ygmE3TZ42nwwJVQDZQGHl84VrCdIRHC8dk9ipE3XlNChkZyhatSOsisHUlACv0RiMTS4wKZDu+bAIvubsV8wzwmKJEnQQAkzglDj7kuMthJpPZCKgQFXXA7OGBxGxaylvMNCmhHRzckfsRBBJlB5LHAqb8Er0HW5w5lwoVTCiatQO7hLYAsJd8C7xICzsdBgLzJ+8a2c3uELQt/AxV9kthYurGtj0HdmzfOzsTePnVg2Oj08xqIlaMBzPMtoZWzdJ/tkoPBgLV+JBmo353i4f8LdoI4+5ua2l86JHd/X3dwYVZ+m5nkrEKZR04OBEEoh4skYQFGdaohycQRQHjlCUl6ZrZwTw3GPBkiqkHvhhikGcAugFNPLRYunU77HaU3M6BZcucxWKCKzE1sAKmHq2Lx8aPJOQJaVfLbe0tE554YLrkthEbEGQrxJIIMMU4Y3ugqpB5wMA+GBInyWQqSqXCQNhXDAki5mCd2SbcgueAxGvvI31FwRT+RHae9AmR7GJFbMkOUA5QkyAZ7ANkFReDTykpcOwNRoKspZJcRySfJE8MZcbC+xJ1raMRr4SEWURUPogKVijRa7Y5oreEgsxQxRjhXyQ9H7HruBpDYyTil5aRKo+saB7cnfA3/7GZ+IBJZYdCUoyRZ+EBMVKYQ5GpzIPMfE3gyqdcVDl4cIkVCXmJNSCzoUwIRgIDWILrMFp0ug8OmRrZEsoh60IZXSoNDZDagCmLS9xAazOTwzbp8zU3eXsG+y6dPxmLBu0Wwr6yZNmlJcpI67UmRAK6DUo3wBGt7Z03RyYoUbXhRjEbG5qakGSTM1M01MRIylNGkYxD3GRVQEtgWkFQnNg70BsML47duaO3OTo6OoZHJ3X2pq72ToPV+Y/f+84XvvZlz9o1T6tLX/rCn01P3AVKjqY3OJNNJgv2kziG1Bpa4dqsDh6VfUrlDNUpwEtO+SbAZTx85ORjj3r27L73ue//8MD+R4w6ayaNS7dC9Sq6Jwx6+Ma1Bo9zKZemWW+92dJabSNzDb/opG8B2MvFeKpSpz9+5uLZy7fY5WjHDY02l4t8q6Rvejq16F/e1XorE58eu43TlV39/plT4UjMbLW1tzaP3pl8/8K5hWDc4fIwQyRqsSfPnDlDMwyUFc7HXCOjrbHRDGtav2HD8WNHrt246Q8klw/1TI3fWTG4zDd+NxyIIwnIngQWAnqAoOgXCR3LzhAmLpweFVKUVxx2+FTF5mALi2Ylv3kbIaQlCiDZW7Bu6EGIrk6Ful/VFSMpiKpiNhTIqYQMPB53FL9HOkPsiSIQqsNxi+mW6vKhhPQjzND2Bqx4qUpWknTVWpueR/B4XCqXChTudAQMBtgn/m07oN+ZdDpDyWa5Cpsn2KyqCJA9mwQfOE3LIfN8uGjt1CfCBbNN1b/VFk/mIXWB5sKzHQh85JlnL154//KhM6BiLcUq7g5rJJiiC+rExRlTqzkRTzU3t87lfKp4EWhSskC/8pWv/MozHzl39hSFSX7fTHB+jgQHtotkGjGHBrgPvBf5DoYjPljcQSKYeX6ks0wV8lbmgJkT4cHB7uJ3bZPUfvOnrMK/e4ipR0KGcsjs88NVYElKpEquJPfiX65auz4mKUY5gTq9lOLjJsyrqmTQI8xULW2C5Lxpk2HDxq6BgSarFTYcZ+oUf60oWoqiLnkneJtghXJN2DHaunAlA49bpY8C7iuCvsisIkIXz1E6EUpmU4VYMGk22G3GBrQQ/NAsCoWZqdLSdEibL3sjvur0bM7bumHTxkfmY4YfvHTIt1icDxaoOK/TG+EhooKhU2fQquOAuKgYMA+B/S5PJd5j+S22hcwdmbBCpWJCYLaL8kBXaWwLSgrhSSqrNPsSv2MFryF2Zr3G2kAHRYoOrBaHaHrszHQaXDMcPjBw0inItnKBxAJVo23yw51F7xFLW+QmHkKZBxgp2hXiUzQACe5KrwQ4OGeKtYRlyGJj2uCUIIiDiGY+pQqmIuDkdQDVwSOYaxZPrQGMTfylVsuHdz/k9LbcGB69fevuzRtjADtn0qV8Iq+2WlgG6jTxluM0Aggklkj4wz5Dg93sNFM2otZVl3cPrF2zcrC/1zczQZSH+wjA51Iebwt7tUwVcz4vPbmYQ4X2eCA5JFpO0KhQAAWTtCyDQU0wkd0HGUoTXZ6aWVERkKDLtGpsNN7TvdjY2Eb8D8rG6QN+A1/nikJ4Mi9AouSIp1cKmaHVHUH/LP4f9BhEi8JbFNqX12wAuIoiXUUf5T7K4goJM7qaiOW6io2JTq8Yr4gxmW6Ry6IEyt2EVUiyFSDakCpKI64ZFHMKUzlJVgVJBz9SdhTvSOqxuHahZFr6IDqVfUiOEU7bX8h4sYxlMLibqSzi+9LBl+g4AhKVi3FyMQas6M8yXMbGSGQ/itQUrZqx8Rd0IHyTt3kLWuUCjAAyKZaJQkCyvMHFeMFy4u6BRkWuyzclg0B0BvQbvoK8hebwbEtttVqL7Vynovi3qmMSROjjXufWDFWCwWx45daycXkB5SoHa4SFJI54NAtC1NSJa+kTUU4kI2672WHXq5Yyc9PjuEDMqGDFBHV5+Cb0BlJ2GguV+kgiWdaVsHeBZUZbVNcnsK0dZquEP+NJl9NNOFOWUKk3IJiTS5FPKHhv5Kfjrm/raHe4rbfvjOFy37NuDbCR0XT2vv27F8Pxa5ffP/zjFw586GH/3FSDyz47O7t980ZYAMMkfOV1e9kXiHZWBNDbuZmZq5ev7dyx+/pwBETl1tbOrQ/v//Y3vnrw7fc6O3o2rN/a3t4ZXozOzc1Rim02G0nGXowEMaZbm5xas769oxXcgeMnTzMzfT3ddk9T3+DyU6dOkYjRuWyIwqe+/pULwSANVhcCIeZ99OaNsG+qo62Rw6A2QNLUUpPLV8gkkSCBOf/qlSva2jsmZ/2gwgpJqLWPPvnh7//j8wCDFDLEdzTTU1NkVZIZTlgrn83NzEwDe7lp04btu3fh433rjTd37tz59s8PBoOlhkZ4dF0qLXEUsW+F4hXpq4gM0ddYTTK9PtglQlNID9mfHMpHcBnokGqvPMEJDUIO2gXUH5Ghcjjrk0h5pFQdGH/YRUu0tpHNQ4LCUgW8VjVOnXgavFYIVaq9hdK4gbhxQSSlAKm7t4vs6oXZ2YaujqBv1tvWTE4cuR1YVBLBJr9BUxeKpmngS1K8xmCm9gHPHoMBpoEiaY+HQjqWMYN8pDQmTXQgmVOFqi+99BLuTW2zmSptVg09mPs+/NCjIx2jh3/6nrXVMXfDZ2uzJXPJS0dP7XzoQSLKVy9fampuMBktLBA7SJzzPInCDbBpKPMjExMBzKywu9lcKNayZxEIwj04TxzQ/M+k8QEcXJk/mc5/+UIkyv95IAHQFET+sMkQ7EzoB9KXW3BB+RI/sg9k+8vCLBGVJWdXL/63MhpSxlivMllVHZ2qnbtU9x7o7+rBpmKbTBHzp1+ZWY8sFU5U274g7cDwRP4ol5fYsfADLW5ndKsq+b9qEyjxyXRKJjSdypNekUb9wbQw0F8KyZvJwm1stF6OJqITvlTVuGoxpi+WTN6eJoO1+8Lt5I2RifFpau6bM6DLa/UA3kEhOUFTyhOck/syxfI82A3MCs8kv+C6wqqkikR8LbAp/kQEW2zmXJFcoizfQN4RAKFayGCSDcXCYN04nSDSNqFbc5dwOCbKGrgHgnxZwNgFIhdfC1yINCPxG2Lx4ImRrJ06oxYvMqjPiFWMAQYgxhCMGG1EDCtOgutCtCLQMIbxGCGbWaIiiLNkGWeppKSxBBINiH+DOpoDmspisjnq6vSRBAhW5Y7erhUr1pKFPD25ePnyNXY6kAzoHnmCsghCQMUo89VqU3RuLeJYwhFtqjPVZZZSs1NxskSfeOyRFcvawgvxmdmJZDREBjfw1xXcBhVxJSL+CazRRoNee3jX4NYssOgKIjehDCxWBC3dzaQNHAcahfIsSCGUBaQvSKcZVVKl8vlUY3eiLa3WwRWWSl28XE5Dh3i9REjANkh6k7vhhSZHPOlt6+jsVt26pmppwBPFYtFJCr2UmL5C99CmZHsiHEWXVziMLLRyIBdrzEUhPGEGNRMWZYfKU2acCynsCS+JVGWozHqNFA+JCUl6MfsL87iWW8UG4+EUdQflDNczgRiASev0qGM0IoW8RcNQdCieGTknew8KY2pE8xJfFqJX1lpID4NK1liS87ixIEui47GzxSEMobJRuKC0ElfoVvgYj6aIdh5QQVQWJdwMdjItFjClEbEsEQqEaJBcX2QtdxUHNjxNug4ycrHsxVhHD0YdxA3EvuQ2cqK4EPC+MP+cw4OLxEbysrgIcFEbhVuwgmwlvuGwkpquVzpV0OkvT1/VkrpotjvoHUcKQyYZsVDGXKXFNwFrmAYZWJYlAPLR/Mx1xTq6aZcDwTDk2DOwjOTD+UVQeulZ2dbV0S4wEcBwM+UyM7Jj2RMotvApwDTIVd65e+ccaDGJ1PjEuKBY1BvqyiBlRb0O88jwlbpy+sqlc3arwWmn+L5pbs5Ha08M7l27dkUisWPHT7a2tgP2O7hshX9+/uevv9na0bpz194jx4+mFmLNjT14Pnds381qXrk8HAmGRckAXzqZqC7lGxucC77oyOiw3WKm1igQz5Bb5PV6N2/einkaCkcbO/rQTHEotVLEb3Z4G7XTUz6o57d/+7ePHnozMz+ZDi80ufDganu6u/wLiyB0guvMjFNob3W4bS5Kji3HTp3u6+s3212QRF/vwOLC/KYN644cOgQI9vREpKW5qXlZEylD3V0dy1cM4uw6f/r07n17v/+Tly++czi4EBoeBvkL0cjA6VBFYoMazQY6ZHcIN2EZRZBIQZogvLB7hQrlYKIV3ZV/BJlOY6wjba4oGQxazAuLVj0dWuQknc1mqABJL0I1mc65PGbEPx7jDBoSuwZ8+SwAzKAqCa1I4Z4REmOdDeKUKVXpk3jh8iVvc2sossCecLe1/9WX/jvtZV588aXRGyPj5TusJOeT8w83kGQnhC4uZm/Drj07ly3v+cJ//fxAb5vOJvWdk9PzBkFor0tFs6pGfXwi0LtxiMy7ed80lW+5WMHVQPyYBo6W3/v8H3z3u9+1t9p4ZK2N9MCi0+l+9iObcWNcvHRhfpbE+UXqp7iYyWAk9lfOM/ewLz3IbTmMe9lNTKRWmEJNWWaDMRfiiYbvIZNrQvN/y10+5lCEjJz67x6K9YfEr21WthdiWA5Fniv7t7Y0fK7wVryjRuAH1UQmM0ymWasaHFQtG1Tfe39PR7e6vZ0svQjgK/RhZwVx+2Me4lhU7D3EHSUOsH4StsTpJ6OGscEgQLkqkycLw67m4wkkLrhReJ7zJMqLSi9tJHlOYgQqlUFjskQSpYWpRKXOanRtHJu3tPZuy+TrLl4evzs1pjY0qHUNxXr8uk48CaTR5AuJbCYJs5aU5nxSAoq8hgNzd5kxnpdNzkrzgJgjnAaPgxVjZ8IMoSYySmUYegKlZhtTTT7+zPgEGb8A9bicXgqN0P5xiadIC0PO5xIkMMDorBYQBBC+cGjyemjJmEf7E+uH0kmYm6J4sCvBLJC7sY7QJbVGBDJJZ4XJiWhZYmn5QMwjRioLrTRQhG0qdC2sVqZRQ3ffVDHrNNkiRRWdpJvbO9dt2II0HL0zM3L7bkGyXbCITODrpCS0pGWP49cUaBNsWUq3KAHCU0SFkaoItsbqdSvvP3Cvw6aKhfErRRcX5kj7YfCURZBTRb0mUBF4EQj9YQRLj0wRe6gsipgRKawc4nYQFH4+paKYIiyKobifTLmElwrlNCmUalU0Rllwdnom1dHdTO41BdnVSgKvMmoIc01CFjFgAx6RfNpAEK4cWbbCc2s4TEWi8GQEcAUHOpRGfS6Xlk2AfKpNl+wGkXC1n5r0ZVAiC+VNETDIXjgtQo01RxLJhIhByafyCIpXSRHn6AFkRYiiqBXjVaQRKyBGBY+OmciyYvIyJAF44qakJoHEIsaibAAZk5CcDAY4D0FmgdLE4JZqcTRnllhsTlGdiS1pyZfCEcCVKYORtC3mC9UPmYojmoHJSTXhygNCG9xE9DYmgVRVUlupxcMUFv2DN7k7xAKbFEVCUsyYBxgxOgcar5T9yNWQ/ohX6dyAMCYiI5yXg++KJsBjyEnQrSKAEddYTcSLtHJNQtfom2SlSP4CcYd81ooZU18OBmZJtDDq6FqRz2VSnrZGDYA8BmM+ay6Ute7mlr7mxlguiKuZ2YzGkv3Lm10e1/jUNK5mrI3JyUmxReqraLJMCIYOaWOGejPEBsQYubjJO8kZv2/H3t1vvPXO3NxMR3f/3FxobnI8GU/mYuH+/m6Dus5ixKuiy2ezoN4gzgH/InkVoEcsOMDWkco0Gzh75tz27dsJSS6GglfOXdr/O38ABppzeBxYx8FlqUg4tnbtOltDs0q6uZW+87Uvj4/QUrDbbKQPRAOTSS/bvQ88anc1YrVrvR59KDrtu7lhwwafb37T+g3f/tY3RkfGH3v0EaNGd+v6tZd/9MMPPXKfJhdOgZwx77faHY2NXh6NqlYwNMh/xEl+9uy5ika/beceCrJ6+5ZPzs4x6+vXry+XVm/ctA7F7acvvQAhMSef+9yf/PUXv9A/MEBYGntldPTmc3//7bOnTy7rG2xubQNqgwZN6AyUkZPUnkzjvCU/R6xUqEbEsOIlgpDFuSxGgBA0tFXbRsgYiAbpq9YTvIMOxWECvI6zuWHZ5vUXrlxcnA2LL9CBXUH9QhU/+djdccQhje+ZKqgVFRsRAjlBS+JPAR1AnLcKyRpVc3N+V6ObrlOqjPjVOefgwYOo6DeGh3s6evxzPtAMCW5Iipdk91ImV+zt7kqkksFwUDtVZ3c5UR2SiTBka3fZe3qXj41OppK53/yN3zx+5GQ8God+KGTC4mrram1panbaXZFg7OknP3r65JkrV95XZVWOFo+9uYO2j+RCr1q1Eok7OzcPsUHoOEsxX0S1Bla0qgaoATlEZbZotwpfUKZO1FNlwhSW9i+UfTaRsl9kOjn485e/lTf+zS9uiDHChkIOyTatfSyySZHBfJkf2Y6cqLy26igkQ/QiCVW9napVq80bNrb0DRhXrrHXqYMV1SKLTr0A4GUMgxRO4Wg0ogLmTm2oquloQXY4QQeYn1hNoryjc2NpknZL9LKoipO+kFPlMwRJ8bRS9KfcWKVKpVNqvDJqEjXq5iO6ZK7NaGpXqVeSgfTO6dCMP0Rdr9beHAilDWajq6kzHIlniNsjJDBewbWA17OamBuIYZ5N8cgoclcRurwFlYhYE12/Nqf4e2CCqJAGG1meZCjTA5ja9bLJaMsbK+1tncSEQXvD/xdLRrDYRaLVVaQQhNpjvd5iMWMaiPkm1ScYtznArMASpoodmwiTiScEsRWvB+xNUtfEr4tJLWKW7DAMS/4RIq6ZyKJAIuOqpMqj+2DOS+MdJJNWn6vWU+zgauue9QfNNveOA/e2QFrjU7dujmaSuegiOWdUXBGDS4N7Z3K5SGJE1JusFiwWKWYxaPDTFFMxjd3S1d6xd/+2TiDjvaozp2/4pqYa3a48UPbJJHXBZGtj9YKtRdoNTgkc9IikmgBmdChZMn01O7Nm6yEwUKtALoSlKuUKYpZKxJtgdZHQKdWt1Uyq4vOXx8ejvQPOljaErEMqe8sZSeQGN1yRD/ILUqFyIhl3ehxr16smR2sCGMVFRDwRS4SBCA+RJiJ7+Ip86/84xA/HpcRIxQQUMpSAP2+KFoQ/XyLxSMAyk0y0UsQMGiIaLzo8qChGm53wfkbRN9gqiFIpAsMNwn3IPkEEcndcZdiQPDJBKVaPNaNOA/czq853xJRSwh/cDktdjDuoA/2FPiNYNETNDTo8KAQyuDF/cx0kIfxSnSd7DVrhXIkQwcoYam2/ShQczwp6UWHJSPsmOkiItYHPUKQv18BXg35AdB5dqMZ8uREeZkxMLkcQhYi0EmaW+cJIxqWBZkJSIFMksFwym4oRjIEuDy/5LxaroqYIwijOdNreEaClUzIwSegKwFHEsrkUvlW0NuhdUDgqmlJRpzXZGz3d7T0DRpspP088OACumI0sJpcHD6fVbqtUTMxIgCJdXgMhX6m0NbeEFxfI5ud9gukgXAYWA02tTYHg4pZ797q9npa2rk179oz+3feIeidJILToacWyY+fjpUKSEvvrVy7CjtEDKIQn9Dh++zaKMxCPJ4+fXLt6HVtsenImEoo2t7bgurz14ssrDxzYvnV3T/vAT3/2s87O7snxKZvFShbYtns2P/vM0/RJOX3iyNzMpNdjG1y7xo5V5nHOzfsvXb42MTGFD2rlypVUnY6OnQDNladFtL/37qH2Ju++HTsuXTxPauWevXveeu2nTCNR5zdef7O5o6O9s2sxHCMjA9pYtmI1DqVuAHLWbens6l46fuKFF14YHBwkMhoK+Kilhtz6+3tQar7+tf/pdbv27Nzx05d+/Nu/+zsb164JBpuJB8fCCbPZzvKh9klioN7AC9iF3ghVcFvhfBAzjkGoR7DrZJugyNZEhbKNIDJULw1RLRMAWeSls3mFgRWX3Dr9g48+1jc0+PUvfQXKwTjW4ePTahPZFGCfhLISmWwlBXiumkxtRHutnpN9gSJHzg7Y5cgNKhpzlQKST5WM23u7iPrv2HnP7dvDCDmGwhop+KiKYIIjowVider1k9OTn/z0p9evX/f5P/sTloPhNTY3oFr5FoOoL7PTfppfbdmyhXjB8LXhj/3qrz7/3D+Q7wrB0GuLhlrPPv3sb332tx577JHZuQk4YjxAUZPhvvsfoljl//r85yFJ4VGgFEkjGYBNsmTrGHVm2IMAMGAmom7gVlMMX7aDTKEiaNmDbA7+ZGZlcoV/YPfJoZwm7IEXLAEvZHf9m4OdLEVhylH7nEtxQbFYFVeTcivFR8qF2JslsGBxDDmcqpWrVJu3u1avQd0ijohp4aNOHgLT6+0aYlD5HMmYpCkpvQfRpIx19Sa1CogbfGWUOosOj3NJtj/cis4MmUKSlCGS5qMpFZrvEtcANQcOLNFQgs71OmssU/YBapqz21ybGpuGZudUF48FxwKLWoJKJa+q7NLqbVZ3I5VLM76Q+IFB/661TxD5CqAVzIgkHQQkjyqCTSwl4SxwNHRDxKf4GXkf/VDsUIQGgbSq3m53Mgx2RzpJFy/7wMCgZMIJp12Kx4HMweyVzDjmn4iK00H1FJ1gALNgyklmovqPnEFSCuCmYoFgZSBvkf2SrlshN4K5gAUzt5hVIr7AOQCUVVYBYczffENIQFllFkfsD/ELUosGP81SvcJ5evPolP/+Bx/r6hkcGZk4e+4QdToCd3V3RkWkymzX2MGCpk0fncQMRSX5mrZsIOpAVqRnk3HQNNC9d+/OrVv6csnC5QtnDwcC1EnaDdqFualiNgOcbWRxAdx4RC++U+pMkU9c0EjhnMwEo0WM8SQ4NfjFHhfmzPoiuoQQYQckvgj0g6haJRNQ6Vod9j2bnyI8f0DlnEzNzxdtTr3Ti9ZZFItMQismHAE8L1LfaOG6BS1Z9pnUhh0D5JuwvXE9w0QQPzIExRbkaxyyqsqPrGyNrJUXNS2f37JTFP8qAkZIgdGi5oipiaJGIRaER8d0JDE55ZJvSKUYp4AHZHdYAwt0rZFtITcVCiogYgVqQ1oeUZJLLiXIG1wKCkALRVnKYg4ggGF/PJTkV7Hp0LlQ2BDAWAXiP2c8gjyGDSGWL84QJY5OlRFt3nkKpKO4pAHokG8purKiZ8joZSfJYITiCwKQCSEyPJBueTocaBLKyPFFwUJDp+NMDuUcXNBkhH2gy9OAVwhNllA8P6hNog/QpYh6Kc5mkj7gIeIW5448BZEok4lRw2sl0CBnVSuksCKSgT4sV6KQr9aotjvsKAH1WnMyU2YvAUFrbmz3TY1NzoLymIdX796xs6Ora25mihYlkRBp9otAJYOMn4qBS6UFioj6QykgIU1clE7aFwa8bS2APb322muLoSgZT6uiUXIbwwvzzW771Njt9hbv7OhtGvmRedHy4IMIYNykyuyV/f7A7l0r9u7dN3z9JluanCxI1Gy23hy+hZs8EInNR5I9g4O923ftCIRHR0effPIJWjJcufQ+4VWX0wpLx1VOq6VzZ07w6SOPPDJ87WpgIeS22wg5a/TW3fceoASW8ZOQGkol8D2tXTG4Y/O6O7du5JOxn7304uOPHNi4eTMWM4kkk3OiMqfz0fWbtrV09Hzre889+7GnF8LRw8dOur1NR06exTvd1dVDGgVS9sNPfUjysBoao6FFlga/wsrB9eR83bN18z89/4+UYG/dvo3MahaR6BQGPbeAYiSioqo3W0HgxuEsnIONhaoIWyWIid3Fr5ryLN4VRXflHdg564kJCPJzdkllxD5iIyMvM2nguugWZXTbhBGgEul1nd1dATCW21rx00QoEi3FqETEGYT7UTREfoS7qak4gUpsTteyweVAcq3btPFr3/wqqVXIXeICMzMTBNR7ujtfeuEl0XcJjCrD4+uQFvhfiP++gf5NmzaTT3fuxLG0K12qpxgtDs3O+vyKoVz31ltvQeXEKY4dO4ZsppNrZDFEofN3v/UdcuPJXaX42OahlFfTubIFCE5vg/vdd94d3Lxx5NIlSod1FuGS1KXiM8+TR1uhx2VWi3cANoPXWcQVW4i9w87CE8aDCZtT/tSwcxknuxhXv0D+ijOJB5Djly9qf/6r32JgKVdR/pWLw/DwFcl7KM8IJ+QVXjLF64YqIsEj1cpB1Y49po1bO9s66o2WnN4QZ7MTFORG5OrSbIWoIClrEjHXWwtV8EptmjpDvZQVmYhQio7NSmPhCrwFeSIUrCzlE5l0lMh4llwQuqSQ64LmgXMTXwBRzEJFPx/Lp5ZMJU2z2bssUWy7drMwM1ehtLXO3KDSWRF1NncrUKeh+Tm+BaGC1kMFGLYJ0QGYlkA6MyFSOKs49SRIiKCF0WGlQXoijwnww7JE5ON04fGFayCwrKhEiTCBKo3Z4u1H/ewbmJ9fgNR5XuxjMpEwgpl5TGTMAJOpngiISFkVCMAYuvBnQTZw2EAXwOTDFgEGC24vnkI1/VZZbG7JD0sp4gYehj0CBqD4HaCumoIgfBbDB/8o6A1YkETdAVJOZ8PpbL1UHTv+w69/dGp28eDBY/P+UCouid7Id5XFbTW7oBccXXqnNZtJk5dBgg6ecWw+afi0ECS7+6EnHnry6U3c7trF6clb1+yAQKvJSqwAMVTOJDDkAz5qMTDKJeeGCWL3Et8FUI/qSyaMd0TQKvak8H6IColE4hlhbTYYrRsgTWx/8InJakWjTZNOQl9QFZXRJdB38SrM+VTnL0y6G5Z3rxkI3F102Bwk+VJBbKCeQrwKuXKe/CBYv9TY5JOBA/e3vfIDHyVsRq06mcjS6SlDSbPiOJV5krCnbA8ORAUHg2IqGRm/fyF9eS3KAciHspcQj4pZzIRDBCazPpuU5rsczBcMwWbVkEwkVV+siGR3S66SqDICOg7al4Blx2MZWUSVoH21tGD9uGHKYHpyKJuXBgMlaJKr4QHGLcImg+LYEJJCRe40s0sRIQjHNjOsCzBtGo6Ix4NjiW5ISqazxGtx51HoIfQMFSPVyc0ExgyXgR7VgRKmIqW0ImIVpkBfInEp6/XixEZ+2y16aImrMGb+Zm5QcXDxsSOUfGyZMfR41o4ACL51DggOfG/2C8n3wo9BxFE0LI1ZMmC4E8yHVEKoHfLlici5YxzUJuKKt9hMTo/d4fRaHC3dA96G/m35KLXhOZPTXa83Da1ZbdKaSJaJIkHJlFBV3R4nyahGrTkaCoPJE4xGE7E4xiL4UKQddff1ZgrFts6OaDotmEoVVUtHNynKzz//T/2d3WIjxhPbtmy4fPUKg4Vr3woEqAgkj4Zd3tvXd/rUebPFdfTY8RUrhlip2Vkf3Xvg10A3t7W18IJ4EgZxNl8i0gWz2Ltzl7mvTzXvu3FjGJnqD8znMukH77934u4oRdX0Q3jlp6+TTvXRT3zSPx+49OOfdPcsO/L2z9mN6DU7d+44fiQ/MXr7wpmTzQ6jy2GFPaPN/PObB9HcAOBTaW5RP3j01LnevhU3x6bCmYrR5rp2a2zn3v29g+u+9nffJLWEcDUN8rDkdu/eTcD45tWruJ4w69kIMJ4b14cnxu8SD8wU84lkkggxqbwA6/j8NCGNCQFo4X1UHyE9xMkmYTcRDKLw4WEiwM4GQOFDlZWNIpqd2L41VsTKsnMJarDyHd1ddyYnsG5vj94xu91f/NyX3t71xsFX33S4LbiR3U0evcVwe2Ls4ccfp6Zy+Oy5MgkIQK+QFcVuyNFhWgvcIMl4uDMhzs2bN2vM+jUb1r134r1bF8/0DvRdvXr5ypUrDrv1b/7mb/yz/p999eWGVZ7gcNjZb46hoyUKVncZQ/ntd9+Z9c9JXMEooApzC9n2dvOWHVte//nR7p6ulSvWfuxjH/vWt/6e3q5kBkKH09PTn//jPyXE8MCOh0KmiNvZEI6GDEYNDc8ROsGA7+TJE5FoCP8z3j8AAjxeO2rc5I05s9cAD0kvZDRmLf5DMyWvILaTeayDQplCQjwSZeRAMa0ZuDwXi8uOBPMIEGBYCu/LGYrbgU3EC+EmMtHCVmq/eYGbSnYhwpU0N9rVFwWlDtag6NWw8ALZVghdYHmsVlVvr+qhA+7ly0wDKzykB6g0oTpNil42bHDugD+ivmrgaqjfaO4wZAqdCyqb1uihgwLOTlUZmBE8XKw1AlaFmpxJ0yyUucTDAmJpiWrFYioPS+S58lgK9BIo10dT2oUY315m9a6uN/VNBVXv31gYnY4VKuDmuHHZUdVvNuoi8Rj4ziDJEDCXeBjl1NxDzDUYiaQTY20wBTwLUlCc3di4bAkUcCxQ/GQG4cPYskAtGMWtAl9kZupi/hiakdXRDsYqciuTKZw5eY7US7g0tk0G8Mhyib1JgBgFmkCpEQRndCiOXJH0Wa5kIQlLhwhM86lMuGLVKVYEigGhPkWASY4SHBkKReBigTBQDCEGjzKJXWSIp5KMzeq0p0pFQmBaq2NyakpjtHUvW7V1515vU+err78D7FoqBoBQpRhBTOBPsUMFGEHock6nC4QqaIABI52MJt2Cf05v0e+8b+8DD9/b0Kgaux0G3o4cLSNCJRLV4sFgJrHnCPWnU1RuAwphtVlIe41Gw2B7NbY202Xy7OXzkhyIPo0Og2qAtidkKTxdDd/nFX4dfiOBeJu51tCyWB6O9wg88Lwgb1dT5DwmVT7/0vh4pq+/4PWsTCUnqX9BA0S/UaLhCkkigPiqtgxWP4nS7Z2qcABRF0W1r1bJ8VDoWDbI/yZuoesPDnF9y11JJ8AC5lLyW8aKg0IitHj/YTwyJmxZbic4U7K7VKSfYNmxMBL4Rm2BZBBdyhMp8plwrrAZNUsFwSLK+aFcAk8achQyKpeA5mQQH2y/2nAk1AarE7tSXB+ybzGnZflLtLhQtq7SdwiNXDzClAVoEJIIzQoaG/eicIB5kU7M0BTyU6xSxW0gt+GmcBP6aUkIRDKiRXOHB7Mz8Rb8m4PVU6YFLG8u/8G+R41S3ESsD7gN4rrBw01RHA/IBYnCUoqFw9liwgdikMnEZ62o/NwklYzhY0Ag86aZ5npuDx3AtLTRcnrGp2L+8JVKnWX50JCrqcM17Y0lFvVOfM/1dORtampgAe7dv+f0ieNTk3exKUFUBIifK2PCxuJJu9VGIlVDazNr0+Z0UYUxMecfWrn85vVhNFzIl53f3daxctlACrZSKpP/bKBKL0U7W/SfukOHj4EKGwpHfHOB69dvoS+3tbYODAywVdFUxsZGkF5sqtaO3t6OromREaATr125nH/tlc6uju7OduheknpymYbmltu3byeSObVd6/E4iS6fp7XI7bHBZf02h+v2jatr1m0yG3Qv/PD5aHDxQ489PHzl0qHD77R4vVvv2Tw8fA2YaEpt8AzArimRePzJx1i9tw8d6R/auHrtZnr+zQdCIGKvXLkKWMdPfepTA6sGh8+d9ftQFjzh4EJvd8+1S+9n03HsxUQsFo9G0KSICIkTDYjeSHRiFvjqJO16UNHI55NdB/WyHyVDXihNvDZwYCFA/lTkMdSDSGDfwPUhIGgJp0m+4Mbir6+ATB2NxagRgl1lQulTr773zq8f+tCHnjr45s9huKBLzc7NAXBKhPjt995dtW7DPU886vcFSPwp5Qq2Ov3I4fPk8KAbEr6S/a5TL4ZDTYbmI0cOr9+47tbNK3gfab4bj0UsZsPxo0f27Nr989ffCE6EDZ3wlrLVBgKKtI5p67YtWzYA/yKsiMWRyyb6+1zszV/7tV+b83PJ1KGD78SCibVD67NbNiPQpZ1sqXTsxNFYPNIz1D11d3px2tcx1JUuxmn07fPNuZsaTp852dvTn8kBU0owPtHdTaeHbHnFUiCwSBhv1dDac4fPaEzSuJNMFPY1hMRECfgQCrRklyv7V5GdMARyffABgLwvk6kckC6v/78PkjGFfcA0ycCBTGUFUF4QE0uAIClFAGIPdnaq9t7reuShDQ5r0OtesjtJk8mUK1mAAjEkZC25mQyIvUvKJhoQrma6eprqqNXXeuo1ZjU2PncC8B2vbKGUA4CNfHQ0QSKrNcWamgo4d119Ip3BtUEhWChaCoTTakOPs2ljMG4bnTbPLEZnQ6Ug4BPqxnqLhRIfcteR3/i9xJmEz5lEIva/sCeWW0wmDGqJrRJ8lNomsimEh8CaWDtxOIu7hST5OkAp6P1N/y5sXpQWVL8M7bOyFWvLgN5gs1upAtbBHgXNimHnyU6KEyawWvRg0RuNOpOZFcA2KNGzEyGA1YOeJm5O8U+Sr5TH78P7SCgYFYvDu4rEhf+xOxAhinYqNpFwf9g1mBhkVjIVmULJBF5liSKoRFlDA6KGycB8anK+Z2D5wOAak9V+7cbdwHsXk/HCwlykQovfOhzLJqcdnAwTyRnowThnE+kk/hESgwwGNn5uYWR41Y5tW+/ZOLiiO5aMnzw2gS+dnOZqJm2mPeASzSiIMtLaiFhzjqAlRaEkoy0VkTJLWBQDA31dvV144K7duIo/QXkURfqqyXvHjkRgQXnY6xj/8jE8XlwE4OfQV4OifR6a6WClGRnSj9BPIqlaWKiMjWY6OxI77ltBx4YlAngmwJvwzyo1t9AW08IPMphK8UplcKXz5lJsHlWDqihp2izsRA45TXiOCE0mkvOVg1VnXP9a+vKnKJoyWPkRGuYqoj7UK5lfEoKkdyZQIJQHgOFD0ZWMA3JHtWEt0VBE0xUBu4T6KYNlB4nihgDOOZxpWHYql1ZkK6cLtcnHNUWLvQKXkig01jq3V3gjo8AFz1SjNSEv0fGAVKWoF1uGk8U5Jf9I/BqaQfoys3WkyWRFR+DxubJMAmSHSJJqV6isdii8gmcV1sDEMA/yDlMrOdAAhIlKQjqN7GQO7iBtiRVmwEespcQdiGSXOFNvJDxGj14sMbaMaDFsdX6LhoJ/p1B2e+wGjSVPJbSMREJi6WgiEJ2MZbTYwaTmXb95R2dUxVJJuw3P7SwPlUnTkB7s6CzIU+RKUZlDqT5GXn9vd0tLy93xcYvFRjMQDGUoMRiM3PfoE++eOLVt5z5aFyRSuQV/oM3rJXgfDYdOnTi+GIl0L0Nhb2R+e3r6KAe6efvuQ488fmP4VjAYHhocIgZ85MgxMpbJEvnIr/96U0tjKpMkAt3e1kL4+dt/9/WPf+KjVpM+F060NTeMj40+8ND9ux68/4Xv/0NucfHvv/dcT08PrmOwhUPh1FNPPhwKBwdXriC0ee36LeZ1Yuw2Kj6OiRUrBkZGbokv1OUKhIMz704jxixllcPTREE4jQI9jS0khNMhYM36TaFYcu269SRFkzR0dxIN5A62b2tjw3Nf/xquM0GHhunnslOLgf/655//s//yOSLW5DuhoJADHIlGwTfB+xtLZIPhJHgm5I0K9cvWwEsj2Rbi1lD2oiJ/ZZE/IAYIUwjng4MXte3gdNlTqQT5z1293dFkIhFM1Bk01lY3vPUbX/9mg8sJ42xocBtNhpkpv8FrhCeiQPb/6rKVa9Z++1t/Hw5Hn3z0Qz3etm+NzYUm/QyFOk2L207nkNsjI4ux0O3x21u2bfzwsx955523mXZxv0Bh+XwoswA02Ex6xutyw22bm1u2bN9+5tzFnbu2U7rG+ra0NmXibYv+osfpYoQvvvACRdJLgFNH8pAHGXB3RseoizVBo5Wls+fO3RkbBeylu7v3yMH30jRPpF5KU7Y12CPh8AOPPDZ6a1TwwEMptuC6j606duwIChlztRBa+OM//dyVyxcLcXQ5YTvAYWZJHCVkA+ofiZaUfoh8kXlk07P1kTlsFaidTVWbTflM2Xj8Ft7CPCt/8kL2uYgBSU4SOSVLgDiSgKSBkAGNS0pYk3KdllbV7r2qbTv6ewe8TY15mxH1F9aK/kacVqfW2HlNJ1gUHmDyFQlKorSelBJV1aaqUjffoNM4if6iJ6ikojtPyyNgySJBPFFJ6ljg6XQxIn1TEuW0ddlCAkDKaKx+coYWpF3t/TvTxZbbU0uzwXp/pDobySeXpJ9tvcVM4isA2oLjCC3GU0QKxYLgCTAlwM6FtxLXUxKJRQAj+RQiI5kT54RYMbg0hRcJ4hDPif9YT9NHvQENBsOWS+nMDmuj22Ly6rR0RcOi4/0UelIePLxKDvHqsAE8YAWgG8FcAQBDqSOCEcJTScWiRY0MAY4oLmWWkMGh8jLr6DNMN5xOFAKGINtCTEg0B9kTsgE4gK4s0svImClm7s7MU5CG7M2rtAv+sLe9f2N335q16wEdv3Nn9tr1mzh8MekrWcoDLDa6GlrowGpH/8OpRvEHKf0anaa5tWl2ZioViGzet6v7gV0HDqybmY1eOHc6HAyATaZEA0Hx4G4AV4mPAMBaSBHLh4mDMoh8RUPB3r7ubTu3oxC/8vKLqK3wRjzKcsC4GTaEhMAn1iN0gAcEEqVTO8yIgdHIiuYORkMqS1s9qpSxLFkGvoqCSYvSSiyu88+Vb96IrRmi/2JbLnPXoKf0E1BXpoSoKShxQqZo7NTmkxnZ0eWKRbJUiMcjoFZlRYLA8yS8KmV6HMq4ar+QfB9sCZEWjE/xRSvzTIiFVeKluOTkFfOP3oBtgteC+kXEvSQy8T/SFo8QYQmBDxJpJAJYxJ7QFgup7Cg+wpWEBxiOh99fazPz+GJYKn4ORsOd+JFvsfcQpdLQAX2FnHvJhKHEEKlD5qBIX+Cgic1IqZnKqraKDSvikuUQxxfUJLflacUg5wGFE/AV0ZxotCPp9UJ1Na8OL7ipmPLKmGEbNeObj+WCwHhCsjojBi7PhUokjm0mSoBWZR55SgYkD8KYAW9BexVDXJabE+TpFIqVe8G81PU2u9eqt4aLi6jGyWiSbNRIdqnerK7qGhuaO1yezmAk5A9M8JxJVD4ivdkModVcOsGlTh5/j6oD4ljLl/XPTk3j1lsIBKjf5QScvZ4G78jIWGNb26Wr1ywOTzq3tGrtptdee0tNHKFURU2OBRcKYNcZRR9uaGrGNJqZnT926ozRYiPmumr1+q98+asTM7ME6sCJpJiYEpTDr7wST5DKgWllA/Wt2evubm99/ac/efCRh+/Zuv7azRt/+J9+72evv0ZYC8Prz7/0f996/zJNexqbem5cu047YX8wjoUMoAR2530PPuDq6vmHL38ZfXb5QB8+c5jCg48+cvzo0Sef+VUcZeT6EunyL4Q//PSzWNI/fPHl906ebW7tYGwms/XypUser/fMieOyoMX88KX3q4WM227t6epsfvD+n7/5zx1trVcvBw6/+05zS2MoFMRwJ5cbARxPJJB/sJk0WJCsHR4a6kqElwj7ESJVCBTSgWz4E1Jh08mnsj+gRFnN2iuhT6iJ+5PykV3q7O944oknbozcPn/pfexyuln09PVDJ7duDLe2tkTCQSwMg1WTT+Q0Tksplv7hPz3/a5/5DbLG6MHY1NL21utvQR06Wq3UqzOJWMVQ77J67k7dXYgtzC/6YzG0q+YGr7uzrTU4PxsEi8zvJ3cUXLC2VgAfcgyJFUERoTfk0VOngPpiZnC8B2Yn2xocJEsT4z9+5D0inVqN2eSGQ4VPnjg2NjqKtYHLkecjwBMMxc5fPNvkbTE4gCyt2DzWRAaUTV3v8t7jp44SIWOO6t3acnbpuef+ob+/LxaLdrS1370x9/JPXvrMZz/z6k9+PD8ZFb0fWwPT0ghnxoyrULiJ54kIKttAJpm9R5wD2HdlemUylV3xS0b0yxcy0crBCcJt2KxKyNOgIUxLzheRL2ImoqkPDqhWrKrHW7Rhc2tHN0IoXalEy2UKvMhLx36UoBPyDooiviC4oqJiSfAT7RkLuFqhLh8XrE2RvsgBIMILQDln4/Ec3a+pVQVBvYh5itJo5imwuxBiiYLhzmIiEjWXql1W2+a5WOfkXN3IVHEuCF62paJzmy06+DKxCXDQkL7JTIQO8SJ0Yeh4BiQ1FZyNHL4cGLI8nqLgiX+Rp8U/B39kYdBcSDrGIoJxC18CIc1M1D2ezGLpkcZldjobvE1OpwenLlxJWhnD6wvIjhIeOKMG0QtMMs4RNXHOIj1gkKZyh7IN0D0hbWkGyB2QHjLPWHpE9xSVibsrMphFE/KXMdWkr3BmGbHCG8nh0dGHFJPcYHWogEZW62CoC4Fo/7r1O/beiyvx4MFTI7fvdHX2ee2td2+SFYyQNNPSEFcInC1RSaL8YqmRaE1oAzJeDM3SKGf3I/s//tEHg4vp0EJidPiSb24K1zqmPKEQHOp2o5ldhyMaRooDHPqAkiV9il6uZuPg8r5sNgNMOm2t9aQIarX+i+cRW+QHKk+iaILQGU8sj8KzIIVJ1BE1BLEkyh0ihPChJF6iiaiBaFHsU2IVZW0ouGS3VO6OZa9eWdi1G3jehWIhpiFeCb3TgUg4huhKyAAiCirw5SyllnYLlYjouXMzQq/SvB3HkFy6dgMZyC8PJl1OUg5hMcoq1eYfq1JoRf4QZ7GsAXo+xryYhZIPjAAiUMplYT2kVrHjsLkVgciDcg4ql2SDI495Eyo000oH3JZsRnAsJCFL3EriBhWYL9GXma/a6gvpSQoqKfgMm7frUeUF+tRAZjW2onwdbRa+w9UlZQHmCVmhWbK6FK0JqC+5V3gbJXNKRCJ7kLMqFQqiat9SxDkDgMzRJNiFwjCUPAxELU9GajFtszVSsSN2NKQqhCMLiCaCK5LgP+4x8a6j+kp1hrJqIo/5HJqGgbAuPBYbhwYTgCbogaUx2hOaZKoYJ7YkQBNFdKx0vclBwqpr2XJ7xJUrxovlRd/UrS5PQ5S+gYUc/AfBnoxFKI8Hp8TjdATntYjGeCwmeV6VulAkSlIxI2SefTP+nFq3kCx09C3fuPmek4feRUlhYBxYtwSXiFbqXA0UVgwODVod3qtXr3/9G9/5oz/647377zt69Cjzywoxv0jZ6YnJJ+996u+/9510NsXkFrPS9L6jtWn05tXdTz78QGdLIhaBp1y9erWlvQNFfeXqDV/4y7/CRfz0r/xHIs3vvPMqFsNCPLNvz+75xTCJSaBgEn2kOJg8EXAhsNtAXLoxOUve7+S0j6aETpudhrg4pO6//34AOOkCQCi3tb1tbftaAnKzE2PkIpm11fm5KevGof6B1QffeYtAOFA4drOho6UxGomsHlp19uwZeD2qCRptMBKh5MNkIzmQTHlJxhHqEpIW7RJUdW2dNGMQi0v5YRsKnXxA6zUnkRCfQo3omBLdQF0gsgsjoFp6bGKSNjr9y5eb7bbLV65h7yyMTNqbHPlUKbwU9jS4ItVEMZr29LXXUeCt1d+zYdPVy8OTt0YXfPPxRIzmMhiIgDm0tLZu2rHlwrVLUxPjWgvJdAvoukhdVv93fvs3v/vNb/hmZ8JBkpaTS7is4eMqVcSfuXD23N777//Gd787Mzc3tGLFM888892v/T/RSMhp1GVT6UaPd9YXtFp0DV7v9BXf84HnXQ5g9fJ4LPEiGkC2d6jDk/FIJF7NqVq6G/N1GbvHHl2MW9T2Dz/z4bHbYxdPnAHpm5oHr9fz2KMPA3dMLdz67Ss3rlv7J3/6n+fmxk/mj2IoYs8ZLCoaQuIdVVDdpOCK7cm2U3acJPOIOatsbIX9yfvKHNc4e+3lv/kN0hxpUSRbUluKBymN2kSg2WRS7dtn3LlzYN2GFqezoDfGNdpFAoj19TkSskTjVtxOaATidUO4adA4YZFwVj40UQSrUWPyWuqRrESucPsCZgRaqQCWpahpydNuWYwjxBLslcJcI+WT6dRSeql+MWer6hvKlrZ4HFXJGUtXgnFNKG0qG23U8SP3uRZVfOh7JBfTEl2alHJfElvAW0HTF9YKCTERsGKRZRLgU4hRmQ1OhNAq2DGSoaI3sed5ZH6gzwKNukgtcroam1otVjvmX2AhDN/EQ45+TAoSnjHCwCazDvOX1AWcO7jJJM2KqCozSLoNzm/RMOFqSo6M4oGFLzAcKf0U7ii6qCgtFDfL3wwOg0YYnQgL5R9RikhYJPu1ChRMyk4CR0vrnfEptPxnPvYf9Q73mVNXxkYnwMQ26ExXL9zSARNjbsArKUm4RNbpelhH/awUwmjt+kh83kTZnMfc0tzIAAf6W4kQhBZnQejDJ9/osoEflw7H6peKhioNZmKkTWHeMTBmRpo8kHCAlpQrEJK7efnq7NwU5Z1MYCGBPbikstgUCxj5xUyL5MPWBdsVeQJVVPFPMsfEsJkthDl3ghLAEIFNc1GkIPIFyYB0Q8EAXDMcUgWd1dGboS0bOy1mr5o+SCpQWFguMahlgriwLC96FPX5SYdb29HtKpfSc7MZMfJAmAL4ExGnHHLmLw5lP8hmqE20/BY7WPkYXUxEL99hzIpZBxXhRhLbEjnMebwveUw8BEIYCxHnj8gjCEnhWaIaI37w+ikHu8MAdHihTFG/Rp0W5QTGJhaw7E0uwlbhfgUZgJiZQr5ipvBCxDdyjXUFQQwJx/bm4FsklYgArhmkCmdV9Fb5lK1QLKJ6U3eEcxo5zi2knApdRJHakh3JdWQI7DksIMYhSopkhYhhKy418YTz/PxdmzrGLG4avidNZAWUj6I4Zhh9Qtn7JE0wUlYYBiQHa4gJLUqjTmd2WfV6mzRmUoBm2EUk3RLc0VkciaLq7uSs0zNN5iGi1GLrm58cZjs3uF00B8T9RCQJewgtgLA2neDAfZe6BLW6qbEFQRONxHAydXZ2kvvQaHAkK+qzV4d37X1gy9YdP/vxy+0uBx1FIUpycNRUHKXSaCeRaOLEqbNP//4f+udDvX2ZkdExnnLDBkk4JGXj/PsXCBDynPHUinw+Gwwmtm7Z0tvWKdpyufjiSz9+9TvfQhOPp1P+QMBmd09Nz05fv9W198BSVRMIJpq37YvfuHzPnvv889N0E3c3tYcC828dPOx22leuHAwsLhw/eYqWCFqTdfu9D7z9zjv5en1Fb6LiaP2atWNj45ev3vjEZz6Dw2Nq9A6G7N3REa/TduvmTSJaNy9dQORbdKqJW8Pbtm4ZHxtjFpoavAAlkqm0cf2Gs2fPMvswpvn5eVguhKUzEJapwMTQr+ApskdkfVEyoG/hLjAeEbrKoVAuJ8nBZ7V/yePhJUon4XuY0+xsxACKvnkJdBGibuhUxNRH74zt3Lef6LjR64DReBqdKfSjaLSYwlOkCvvm9VbrzYtXdu3cS4uPn73yuiqTN9VTR0sR7ZKrq7mzqx22TRbV9r17Ll45z1MfPXqoEEnkk4kLZ89sWr/+/PkLqaRgjsbDWWxNBqyzqd58841kNvf444+/c+jQ0bd//ulP/iqKy1uvvWJxO1nrYhKQjEomGSQLkl4O+Shqb4apoBkJ+MH42PC9Zd0lj8cRnIlnC3SQiVWTuHwMRNFu3LoWXYwxV0hrIKD88wtf/epXGBWTf/fuOLrj8RPvWax6eLzBXG81m+GjDZ5Gny8Q1caIzdXXsyvpzK0lkAYEBLPNwQTX9ogyqzLhss0J0CibmRe192Xm5Y+qNPKWbMh0nsxWlcrtVq1bq1s11LRv31Brm85sTpcrEaMBfUKqTwtLaYvVwl3ELaXCB8yGAHRWjyXIXCGIMZnIdtZo7Cp6GZWlR3smEqsWpfA9nUqRQ0o0lFI6Yj1g/WOxg6XHJs/TQzBZyWaIKluKupV2z9pi2nr3TmByrlhPgzutLaXO2Nx2nKIIi3w2UyKJTXKoxcpkJFgg3FkclPBEzAIl7VTxF8pAmYFfTgUvgEUQySRVFtgY5AAR54TTcSKojE6bw0PClQCQZZcyacAzMiWANbCw8VET1gKQ3UhnYpgZdjLqj/BrOD7BTUI0wuLITcwRj5NUJPRIDA5xLzILOBcVhi+SFl6MYsD7SGMYt6wLw1dkMKxHjDDZJMlsNpEpetnUkeStu5d/5eOf3Lxl+2tvHLw9froIVsVSfSGULgE3UtbYzI5cVlqVw3CAo6kaJBOdpknpQkJfp2nubCBQTa7J2tVD46O33794JhaeA8OnrpLjbplUPB4OM0JMFlK+wM+UGlCSv5VyZYyyDK0ISe5Nxa5duoAwIuxHw0LJnUvEcTVgDonwkfmFnyuST9gxYkYCGmK0IT1xSygrwqwwLUtAsROzZOrQ1hA/NYlAro+m3srdY+Hq5ET04oXRrVtxuQhiMCeIB5hoR22OUGRkDvlyHr21scmmrjpu3xxLRskwwlo20oy5Ju1+se4yOg52BQYepMIoWCoJ4yjyWNiOUK5cUiQc0y8/kpEMMyKSKpuqXCEBUl6T1C26H6sml699C4lWUuE1ZjYgDdl+3A0BLd7gkmK/KjyutjlZaaQjEhcMDMWYFPLAtkUv5O6Ys5I+LGxRhsH1MV+4UZqOkbW0Z95kS/P0zIYIb67GHcX9Ujt4wQAQqPiIeIFIVTY+5CWjkgFjR/OYXIVDuSlfxC8EeBNjRrDyNveXl5J5JSKWE2rPQtEZr+HnvEl0gHnj4DVSH2WL0nIStPRWC0I9hfecJE3p5s1X1PiX2gdWBJP1gWju5shtdNjB5W1Wr/vcsTcAi26mBaGasgUg2lOVIvnRLiS9winyFqMlEUXuEvSmtzaJGCZiuoRkdu/Z/5O3Dm3YsOX0mfMf+8Qnenv787n0Uiq/vLuLAjs8+9jBQEZTuE/HzfjoaGNzq9EAfnL11Vdf//zn/guucRr34rwNRyLgpb57+BDhNRIlbt++GZ6doVKfsuD+3q7pqQl1QJAsmEakwvTk7LWrw3qDo6W1487Y9NihY28deruzv2Vh0b9p256me3ZO/OgHV4dvAAAy6/MRnSUN297YTB3Gt5/7wco16+7dtgNo3e9/+Uvnz5/fvXtv74B77OYNAP8eefC+TQfuf+Ofnj9++F28+0t6LYlsDz94/6kTx29fv/yD71c5554tW0+dOEZtNF76V155BVJFKEK1xO/wbZDiBv2RT8OfxPTFDyDuTA2aNJhUgN5AwcygyGEhrBqlCHUJCSt/1nYx7yg/YhKAlZmkvwysBVeYVkfKpWTr1dWdOvSuqaGhq6Xl9oXL+WDO6ACFzUhyP/01ktk87YFjvoXxa7fDM35VOE76ajlbkA2xVIXNnjh1qhyPWwbaf/cPf9fltZ48dYykd3NrAz6SV390UEpWyCtMqZo6XWkNifQCAEdXqOBC6q1/fuORjzz953/+53/2X/70L/7iL3ZtXU/uD31h2EqlQoWAW2AuAx61s8FVtVfpQdG1ojseDeKCcrpFUdCb0COzSCUywa12C4kGg32DAAv/4X/+o699+Wv0GEhRv6A8u7fdfed6gDYEjY1OgvGr1w795Kc/As6yqUlnMBlcXpfT5fD5/PAQ6QQAy6NalfoFCo9ELxc1Vdk+sgeZXLa8SFnlqE06n/KXMu/yG/6HeYI0AS+lwUFvRO2ade3btw0MDXnLpUVPIwnk5SwwDksZ/sc/ZKcNJwlUgEHXCV6N5I8IB8B6JoXYRANevM1qGvdKzX0J5P5CrpQN4X1Sg8TEYoIGwIZGvcSpm2bf0vxsiZzwUjxG6gW+Tku9aUUuP3TlrHqe1p4VW86gDsdDS/UJs9MdyccQlaVsnvoy8IPUeABp9k7zB/yRCqQjUQDqJaTelypaomBIPJ5P4aki8oTuxNoSxZz6GfrJUmUDAgUqAPkqGj3JTRZQoCx2MGSCoUQqkyWzxKDHICE+hcPfiO0rwS9cBuK6KpEbqnjhYRVSFUaeH849Ap44BpnVWhIpnB5Oq3Bz4VEMBrErYkTIXUnJEdHLO3IIN1P2BnIUlocSQM7UjM8/tGbzZ3//0UAw9o3vfj+BDpOqxMOoILT1c5l1xNXwhxu9zY1YCIh9iK3CsOrq8zgAdGp7k3P/gW2A2SD9JyZHgyGf3WYAznZ+JonFhrSwaLUemyWTSsTmgXomIcRQAYKfVdfUA69PSxKyHmgeA9w+80asPR0OwtORo6JbiTmYhSWjW8vDiV8UdiseB6Qmhf8m+qNRrCmKOPNHwZBSGKM8qhLJ5WvC6oUJoHmDtE/aWziSmfdlT57wr1y53aEjuMuWxsWNYs4EMj/oiKKg4JEWpAnkj93QqAYO4W46zqJjcxuhLsbFeP7NIRMtwkmYC4JJYSYybNEK5D0ohUVBsski8ClyB5sOVqZUX9WkGu4AOBq35wsQfo1bKcNR9h0Ewv4SwsZiFhkvDAV1hPWQIUN+ynrLateDTEMwVXqSiN2hCFdIhR/uyNZCuWEMSH30dwFRVbA75CJydm3ChWKYQtDWCHyTXs4817a/CGjcThKNlXAu32IzcFk8+XBsRKzMoewNOeRTcVvR+pSKdW7A+/J1tHmsbjRdmAxnMxvIYOW7te/JojFCxaxns6Eo6EBJNFptsbQgT5WkQD0ntriGLgVA1tjoyLtu+6Pdy/v8i4lLl84mE759e4ZoBcK+CYdDgiAXzzR6PABn2agvymW4CApwZ1unPAtFbpSxl6uzfp9BZ4QRnDp+isHMTc+BynryyNED992XnJu+c+Py6tWrScqdj9LrT2hg7969b/wz/drfSyRScEM0eCqafvSjHz351BMdpDpP3sHRWihmBDQ4l+ru6iLt9/qZcy1NjaNjI9h8gM+j5IOOuWvvvoee/fgbr7x26NB7x0+973I2rhha9U8/eolaPxJEnvjw0xhwhamZFStX4Zp22M0jt26SwwxuajAc/Yv//ldf+h//8xOf+szN0TFPU9P+fQcunjwyMjJCMZXb5W3wuK9fv06aN6p7d1dbJBiimhnN98ihg2T8gYdKnbHF5Tn63iHgOXE+W01GJwnJSn6HyWLBRYPoxYCgZIg3CYsFwll0f/JZqJ0o6SkghWPlQYIRiv+F6P1gCZV/hDCVD/gQTxQHzILVDQSy9STjmoGMXdJbxV1H8YO7wRuJJQEAJN1p3baN85OT1BtTyetw6xLpPPxR4NIq1YmRMT/NBb0NCcocyxWHwwYKv6REWA1lmw5T7OTpU6SYQpPQEJelR6TBxh3gkGD7QmYlfEg4Z+C5wZmUxoUXrvrzF1/ZvHUbvRNOHz9Co8loKO6x68NhWqES5c3YnCCCWCPhlNPhymq19Of4wn/7/O/+3m+RC+Z0WpuaGqlVA6wfcESsPvTdO3fHB7qXHT58GL4Jwggpq3RtBJR3aips9qq+9MW//uIX/9bnmwXLkf53kDITm0wlFhbnAQAmg5rtgAsKF66eyh/QPmqMQpR7BQ62tjeU6ZVNUuNwyp//zi8SVDWqxgbt1q3L9+ynLq/J40bdj1H/ajHjC2dWSVwzguuDfEe2IW4FmFhd1hsw8sjFEoBCVC+AQ1Qqu6rOTA0SoihJfUgslo6lqnFsnXqljEUQI5REEJa3Sn5QXb0BvT+dqUYTAuHT4Lbb3X1v/HM0VmqL50xxEIdMWmuTq0hFFj2gs1IerSLaRVwnt0RWPQEP+guV1IBNM6MKkLJ4KIXNMTvoJSIHJLNL8XPKk3NbwGHU+ERJUpB+u8q+duGd8DTSI5KmuTTipLUXoD0YgfhdMfSsDrKcVWbpwgIoB7SBgYdDT1JRpbQV57IyvRAxK8I66OqBP8JERc8QdogIRgyzCvirlTPFoalYBUocWihdiBDKQwoIAQqTFsuYfTC0etXe+x6mEOunr745Oj5NGDi0GK+rmLRqhJmmtbG5XARyy5CMpqO5GAkiiF6K1Qqy5+qsDmvnQN+u3VtWLzfdGfP7pqcW5/34HuCm5HATK9ZUDNlkjOdF90HQIDvY73l8rWidGg22EJCW4YUAAphaIMrLiA3D08nKkYITHlbZqliI/wsHFGRU3db9AAAAAABJRU5ErkJggg==\",\n \"text/plain\": [\n \"\"\n ]\n },\n \"execution_count\": 25,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"image\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"YU3hwodphmeW\",\n \"outputId\": \"7e1757f6-0c77-477a-c35a-2429d8020374\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"The presentation of the meal, which is divided into three compartments in a lunch box, can significantly influence one's eating experience. The compartmentalized design allows for a visually appealing and organized presentation of the food, making it more enticing and appetizing. The variety of food items, such as the sandwich, chips, and fruit, also adds to the visual appeal and encourages a balanced and nutritious meal. The compartmentalized layout can also make it easier for the person eating the meal to access and enjoy each component of the meal, as they can easily reach for the food they want without having to rearrange the entire meal. Overall, the presentation of the meal can enhance the eating experience by making it more enjoyable, visually appealing, and convenient.\\n\"\n ]\n }\n ],\n \"source\": [\n \"import torch\\n\",\n \"\\n\",\n \"tokenized = tokenizer.apply_chat_template(messages, return_tensors=\\\"pt\\\", return_dict=True)\\n\",\n \"tokenized[\\\"input_ids\\\"] = tokenized[\\\"input_ids\\\"].to(device=\\\"cuda\\\")\\n\",\n \"tokenized[\\\"pixel_values\\\"] = tokenized[\\\"pixel_values\\\"].to(dtype=torch.bfloat16, device=\\\"cuda\\\")\\n\",\n \"image_sizes = [tokenized[\\\"pixel_values\\\"].shape[-2:]]\\n\",\n \"\\n\",\n \"output = model.generate(\\n\",\n \" **tokenized,\\n\",\n \" image_sizes=image_sizes,\\n\",\n \" max_new_tokens=512,\\n\",\n \")[0]\\n\",\n \"\\n\",\n \"decoded_output = tokenizer.decode(output[len(tokenized[\\\"input_ids\\\"][0]):])\\n\",\n \"print(decoded_output)\"\n ]\n }\n ],\n \"metadata\": {\n \"accelerator\": \"GPU\",\n \"colab\": {\n \"gpuType\": \"T4\",\n \"provenance\": []\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 0\n}\n" }, { "path": "examples/notebooks/sft_nemotron_3.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"ovlmqboji0c\",\n \"metadata\": {\n \"id\": \"ovlmqboji0c\"\n },\n \"source\": \"# Fine-Tune NVIDIA Nemotron 3 with SFT and LoRA using TRL\\n\\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_nemotron_3.ipynb)\"\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"kiezlcdkw2k\",\n \"metadata\": {\n \"id\": \"kiezlcdkw2k\"\n },\n \"source\": [\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"ezgoxs28857\",\n \"metadata\": {\n \"id\": \"ezgoxs28857\"\n },\n \"source\": \"Fine-tune [**NVIDIA Nemotron 3**](https://huggingface.co/collections/nvidia/nvidia-nemotron-v3) models using **LoRA** and **SFT** with the [**TRL**](https://github.com/huggingface/trl) library.\\n\\nThe Nemotron 3 family includes hybrid **Mamba2/Transformer** models in different sizes (Nano, Super, etc.), natively supported in `transformers` (no `trust_remote_code` needed). See the [full collection](https://huggingface.co/collections/nvidia/nvidia-nemotron-v3) for all available checkpoints.\\n\\n> **Note:** This notebook requires a GPU with sufficient VRAM (e.g., A100 80GB). It is not designed for free Colab instances.\\n\\n- [TRL GitHub Repository](https://github.com/huggingface/trl)\\n- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview)\"\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"5t9o6vg3jus\",\n \"metadata\": {\n \"id\": \"5t9o6vg3jus\"\n },\n \"source\": \"## Install dependencies\\n\\nWe install **TRL** with the **PEFT** and **quantization** extras, along with **trackio** for experiment tracking. Nemotron 3 models use **Mamba2** layers, which require the `mamba_ssm` and `causal_conv1d` CUDA kernels.\"\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"wh29y02x1ni\",\n \"metadata\": {\n \"id\": \"wh29y02x1ni\"\n },\n \"outputs\": [],\n \"source\": \"!pip install -Uq \\\"trl[peft,quantization]\\\" trackio\\n\\n# Nemotron 3 requires transformers>=5.3.0, which may not be installed by default with TRL\\n!pip install -Uq \\\"transformers>=5.3.0\\\"\\n\\n# Mamba2 CUDA kernels (--no-build-isolation needed so they find your PyTorch/CUDA installation)\\n!pip install --no-build-isolation mamba_ssm==2.2.5 # installation takes around 30 mins\\n!pip install --no-build-isolation causal_conv1d==1.5.2\"\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"yic7cr26xoi\",\n \"metadata\": {\n \"id\": \"yic7cr26xoi\"\n },\n \"source\": [\n \"### Log in to Hugging Face\\n\",\n \"\\n\",\n \"Log in to your **Hugging Face** account to save your fine-tuned model and track experiments on the Hub. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"wwocbqwne49\",\n \"metadata\": {\n \"id\": \"wwocbqwne49\",\n \"outputId\": \"f8a3f993-63a5-4a11-ea41-5a0093cec19c\",\n \"colab\": {\n \"referenced_widgets\": [\n \"20976278f8564bc080426e851d52f46f\"\n ]\n }\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:86: UserWarning: \\n\",\n \"Access to the secret `HF_TOKEN` has not been granted on this notebook.\\n\",\n \"You will not be requested again.\\n\",\n \"Please restart the session if you want to be prompted again.\\n\",\n \" warnings.warn(\\n\"\n ]\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"20976278f8564bc080426e851d52f46f\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"VBox(children=(HTML(value='

...` tags. We also remove the extra columns that are not needed for training:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"oxzelzs2kf\",\n \"metadata\": {\n \"id\": \"oxzelzs2kf\"\n },\n \"outputs\": [],\n \"source\": [\n \"def merge_thinking_and_remove_key(example):\\n\",\n \" new_messages = []\\n\",\n \" for msg in example[\\\"messages\\\"]:\\n\",\n \" content = msg[\\\"content\\\"]\\n\",\n \" thinking = msg.get(\\\"thinking\\\")\\n\",\n \" if thinking and isinstance(thinking, str) and thinking.strip():\\n\",\n \" content = f\\\"\\\\n{thinking}\\\\n\\\\n{content}\\\"\\n\",\n \" new_messages.append({\\\"role\\\": msg[\\\"role\\\"], \\\"content\\\": content})\\n\",\n \" example[\\\"messages\\\"] = new_messages\\n\",\n \" return example\\n\",\n \"\\n\",\n \"train_dataset = train_dataset.remove_columns([\\\"reasoning_language\\\", \\\"developer\\\", \\\"user\\\", \\\"analysis\\\", \\\"final\\\"])\\n\",\n \"train_dataset = train_dataset.map(merge_thinking_and_remove_key)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"gyh82o2b7i7\",\n \"metadata\": {\n \"id\": \"gyh82o2b7i7\"\n },\n \"source\": \"## Load model and configure LoRA\\n\\nLoad an **NVIDIA Nemotron 3** model. These models are natively supported in `transformers` so no `trust_remote_code` is needed. We use `attn_implementation=\\\"eager\\\"` as required by the hybrid Mamba2/Transformer architecture.\"\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"asrptlebp6a\",\n \"metadata\": {\n \"id\": \"asrptlebp6a\"\n },\n \"outputs\": [],\n \"source\": \"import torch\\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\\n\\n# Select a Nemotron 3 checkpoint below\\nmodel_id = \\\"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16\\\"\\n# model_id = \\\"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16\\\"\\n\\noutput_dir = \\\"nemotron-3-sft\\\"\\n\\nmodel = AutoModelForCausalLM.from_pretrained(\\n model_id,\\n attn_implementation=\\\"eager\\\",\\n dtype=torch.bfloat16,\\n)\\ntokenizer = AutoTokenizer.from_pretrained(model_id)\"\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"90a8pzer8rc\",\n \"metadata\": {\n \"id\": \"90a8pzer8rc\"\n },\n \"source\": [\n \"Configure the **LoRA adapter**. Instead of modifying the original weights, we fine-tune a lightweight LoRA adapter for efficient and memory-friendly training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"pyukji9o3g\",\n \"metadata\": {\n \"id\": \"pyukji9o3g\"\n },\n \"outputs\": [],\n \"source\": [\n \"from peft import LoraConfig\\n\",\n \"\\n\",\n \"peft_config = LoraConfig(\\n\",\n \" r=8,\\n\",\n \" lora_alpha=16,\\n\",\n \" target_modules=[\\n\",\n \" \\\"q_proj\\\",\\n\",\n \" \\\"k_proj\\\",\\n\",\n \" \\\"v_proj\\\",\\n\",\n \" \\\"o_proj\\\",\\n\",\n \" \\\"gate_proj\\\",\\n\",\n \" \\\"up_proj\\\",\\n\",\n \" \\\"down_proj\\\",\\n\",\n \" ],\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"rkxovj1wywh\",\n \"metadata\": {\n \"id\": \"rkxovj1wywh\"\n },\n \"source\": \"## Train model\\n\\nConfigure **SFT** using `SFTConfig`. Note that `gradient_checkpointing` is set to `False` because Nemotron 3 (`NemotronHForCausalLM`) does not support it. For full details on all available parameters, check the [TRL SFTConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.SFTConfig).\"\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"tdmoy72oow\",\n \"metadata\": {\n \"id\": \"tdmoy72oow\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import SFTConfig\\n\",\n \"\\n\",\n \"training_args = SFTConfig(\\n\",\n \" # Training schedule / optimization\\n\",\n \" per_device_train_batch_size=1, # Batch size per GPU\\n\",\n \" gradient_accumulation_steps=4, # Effective batch size = per_device_train_batch_size * gradient_accumulation_steps\\n\",\n \" num_train_epochs=1, # Number of full dataset passes\\n\",\n \" learning_rate=2e-4, # Learning rate for the optimizer\\n\",\n \" optim=\\\"paged_adamw_8bit\\\", # Memory-efficient 8-bit optimizer\\n\",\n \"\\n\",\n \" # Logging / reporting\\n\",\n \" logging_steps=10, # Log training metrics every N steps\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool\\n\",\n \" trackio_space_id=output_dir, # HF Space where the experiment tracking will be saved\\n\",\n \" output_dir=output_dir, # Where to save model checkpoints and logs\\n\",\n \"\\n\",\n \" max_length=128, # Kept short due to VRAM constraints; increase for better results\\n\",\n \" gradient_checkpointing=False, # NemotronH does not support gradient checkpointing\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub=True, # Push the trained model to the Hugging Face Hub\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"4ciesqgt2ch\",\n \"metadata\": {\n \"id\": \"4ciesqgt2ch\"\n },\n \"source\": [\n \"Configure the SFT Trainer. We pass the previously configured `training_args` and the `peft_config` for LoRA.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"h0i5c9sd1ip\",\n \"metadata\": {\n \"id\": \"h0i5c9sd1ip\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import SFTTrainer\\n\",\n \"\\n\",\n \"trainer = SFTTrainer(\\n\",\n \" model=model,\\n\",\n \" args=training_args,\\n\",\n \" train_dataset=train_dataset,\\n\",\n \" peft_config=peft_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"q1t7v0lba6l\",\n \"metadata\": {\n \"id\": \"q1t7v0lba6l\"\n },\n \"source\": [\n \"Show memory stats before training:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"b4e0fseivvu\",\n \"metadata\": {\n \"id\": \"b4e0fseivvu\",\n \"outputId\": \"94e1ffbb-639c-4dba-d2b7-2d1de0f493bf\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"GPU = NVIDIA A100-SXM4-80GB. Max memory = 79.251 GB.\\n\",\n \"58.939 GB of memory reserved.\\n\"\n ]\n }\n ],\n \"source\": [\n \"gpu_stats = torch.cuda.get_device_properties(0)\\n\",\n \"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\\n\",\n \"\\n\",\n \"print(f\\\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\\\")\\n\",\n \"print(f\\\"{start_gpu_memory} GB of memory reserved.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"1x3fdnogsvh\",\n \"metadata\": {\n \"id\": \"1x3fdnogsvh\"\n },\n \"source\": [\n \"And train!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"3ev9j7op2mi\",\n \"metadata\": {\n \"id\": \"3ev9j7op2mi\",\n \"outputId\": \"d6db6cdb-8d43-4cc6-e024-e77c266090f7\"\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 11, 'pad_token_id': None}.\\n\"\n ]\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Trackio project initialized: huggingface\\n\",\n \"* Trackio metrics will be synced to Hugging Face Dataset: sergiopaniego/nemotron-3-sft-dataset\\n\",\n \"* Creating new space: https://huggingface.co/spaces/sergiopaniego/nemotron-3-sft\\n\",\n \"* View dashboard by going to: https://sergiopaniego-nemotron-3-sft.hf.space/\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"
\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* NVIDIA GPU detected, enabling automatic GPU metrics logging\\n\",\n \"* Created new run: sergiopaniego-1773149018\\n\"\n ]\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/usr/local/lib/python3.12/dist-packages/transformers/models/nemotron_h/modeling_nemotron_h.py:1215: FutureWarning: `input_embeds` is deprecated and will be removed in version 5.6.0 for `create_causal_mask`. Use `inputs_embeds` instead.\\n\",\n \" causal_mask = create_causal_mask(\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" [250/250 08:53, Epoch 1/1]\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \"
StepTraining Loss
102.379619
201.849189
301.466606
401.207237
501.064882
601.067329
701.085463
800.964451
900.993209
1001.061509
1100.976885
1200.939225
1301.011990
1400.989627
1500.946731
1600.939989
1700.920945
1801.003975
1900.945233
2000.958358
2101.022524
2200.953711
2301.033258
2400.955309
2500.959105

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Run finished. Uploading logs to Trackio Space (please wait...)\\n\"\n ]\n }\n ],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"ngrmymxpbp\",\n \"metadata\": {\n \"id\": \"ngrmymxpbp\"\n },\n \"source\": [\n \"Show memory stats after training:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"08gpt482fn1k\",\n \"metadata\": {\n \"id\": \"08gpt482fn1k\",\n \"outputId\": \"e4808857-5756-4f5d-d34c-29c81107a618\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"542.949 seconds used for training.\\n\",\n \"9.05 minutes used for training.\\n\",\n \"Peak reserved memory = 61.922 GB.\\n\",\n \"Peak reserved memory for training = 2.983 GB.\\n\",\n \"Peak reserved memory % of max memory = 78.134 %.\\n\",\n \"Peak reserved memory for training % of max memory = 3.764 %.\\n\"\n ]\n }\n ],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"pkp4wfpwpxh\",\n \"metadata\": {\n \"id\": \"pkp4wfpwpxh\"\n },\n \"source\": [\n \"## Save fine-tuned model\\n\",\n \"\\n\",\n \"Save the fine-tuned model both **locally** and to the **Hugging Face Hub**.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"g2j7qp1ycxl\",\n \"metadata\": {\n \"id\": \"g2j7qp1ycxl\",\n \"outputId\": \"5791fc37-589b-4bb9-ca66-85434b303787\",\n \"colab\": {\n \"referenced_widgets\": [\n \"20def6e82e1d4b69ae1b677a39a54bfc\",\n \"53fe5574751540b68f2d6ef047cccc3c\",\n \"3661c026040f44f3870921e8a2db7bd2\",\n \"61b4f98716b649dc88332367852ec89c\",\n \"a61820f54e5445ceaf27efa956212271\",\n \"d761f8d2a0ff4de2a8728a1998e5f919\",\n \"3fea49f4e519416f81af219023d45c34\",\n \"240c5e45b5f245f984cdaaea9ff3a380\",\n \"7ba75fdca7da467cacc407da59382376\",\n \"2b3e22abd6704ef6b5cb2455c6fa98ab\"\n ]\n }\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"20def6e82e1d4b69ae1b677a39a54bfc\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"53fe5574751540b68f2d6ef047cccc3c\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"3661c026040f44f3870921e8a2db7bd2\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...n-3-sft/training_args.bin: 100%|##########| 5.58kB / 5.58kB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"61b4f98716b649dc88332367852ec89c\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...adapter_model.safetensors: 100%|##########| 13.2MB / 13.2MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"a61820f54e5445ceaf27efa956212271\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...tron-3-sft/tokenizer.json: 100%|##########| 17.1MB / 17.1MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"d761f8d2a0ff4de2a8728a1998e5f919\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"3fea49f4e519416f81af219023d45c34\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"240c5e45b5f245f984cdaaea9ff3a380\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...n-3-sft/training_args.bin: 100%|##########| 5.58kB / 5.58kB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"7ba75fdca7da467cacc407da59382376\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...tron-3-sft/tokenizer.json: 100%|##########| 17.1MB / 17.1MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"2b3e22abd6704ef6b5cb2455c6fa98ab\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...adapter_model.safetensors: 100%|##########| 13.2MB / 13.2MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.google.colaboratory.intrinsic+json\": {\n \"type\": \"string\"\n },\n \"text/plain\": [\n \"CommitInfo(commit_url='https://huggingface.co/sergiopaniego/nemotron-3-sft/commit/dd05a083358e10113019b06da76c3e9ab4c4c3e4', commit_message='End of training', commit_description='', oid='dd05a083358e10113019b06da76c3e9ab4c4c3e4', pr_url=None, repo_url=RepoUrl('https://huggingface.co/sergiopaniego/nemotron-3-sft', endpoint='https://huggingface.co', repo_type='model', repo_id='sergiopaniego/nemotron-3-sft'), pr_revision=None, pr_num=None)\"\n ]\n },\n \"execution_count\": 14,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub(dataset_name=dataset_name)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"jp3u13oqlye\",\n \"metadata\": {\n \"id\": \"jp3u13oqlye\"\n },\n \"source\": [\n \"## Inference\\n\",\n \"\\n\",\n \"Let's run the fine-tuned model using standard `transformers` generation (`model.generate`).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"1tb9jai824gi\",\n \"metadata\": {\n \"id\": \"1tb9jai824gi\"\n },\n \"outputs\": [],\n \"source\": [\n \"model = trainer.model\\n\",\n \"model.eval()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"3zp0i0zw4je\",\n \"metadata\": {\n \"id\": \"3zp0i0zw4je\"\n },\n \"outputs\": [],\n \"source\": [\n \"messages = [{\\\"role\\\": \\\"user\\\", \\\"content\\\": \\\"Continue the sequence: 1, 1, 2, 3, 5, 8,\\\"}]\\n\",\n \"text = tokenizer.apply_chat_template(\\n\",\n \" messages,\\n\",\n \" tokenize=False,\\n\",\n \" add_generation_prompt=True,\\n\",\n \")\\n\",\n \"\\n\",\n \"from transformers import TextStreamer\\n\",\n \"\\n\",\n \"_ = model.generate(\\n\",\n \" **tokenizer(text, return_tensors=\\\"pt\\\").to(\\\"cuda\\\"),\\n\",\n \" max_new_tokens=128,\\n\",\n \" temperature=0.7,\\n\",\n \" top_p=0.8,\\n\",\n \" top_k=20,\\n\",\n \" streamer=TextStreamer(tokenizer, skip_prompt=True),\\n\",\n \")\"\n ]\n }\n ],\n \"metadata\": {\n \"language_info\": {\n \"name\": \"python\"\n },\n \"colab\": {\n \"provenance\": [],\n \"machine_shape\": \"hm\",\n \"gpuType\": \"A100\"\n },\n \"accelerator\": \"GPU\"\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}" }, { "path": "examples/notebooks/sft_qwen_vl.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"UaDIwQOOjgAO\"\n },\n \"source\": [\n \"# Supervised Fine-Tuning (SFT) Qwen3-VL with QLoRA using TRL\\n\",\n \"\\n\",\n \"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_qwen_vl.ipynb)\\n\",\n \"\\n\",\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"4f0hzSo4kKEc\"\n },\n \"source\": [\n \"With [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl), you can fine-tune cutting edge vision language models. It comes with support for quantized parameter efficient fine-tuning technique **QLoRA**, so we can use free Colab (T4 GPU) to fine-tune models like [Qwen3-VL](https://huggingface.co/collections/Qwen/qwen3-vl-68d2a7c1b8a8afce4ebd2dbe).\\n\",\n \"\\n\",\n \"\\n\",\n \"- [TRL GitHub Repository](https://github.com/huggingface/trl) โ€” star us to support the project! \\n\",\n \"- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview) \\n\",\n \"- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\\n\",\n \"- [More Qwen3-VL Fine-tuning Examples (including TRL scripts)](https://github.com/QwenLM/Qwen3-VL/tree/main/qwen-vl-finetune/)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"pGXgIbj2kXEP\"\n },\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"We'll install **TRL** with the **PEFT** extra, which ensures all main dependencies such as **Transformers** and **PEFT** (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install **trackio** to log and monitor our experiments, and **bitsandbytes** to enable quantization of LLMs, reducing memory consumption for both inference and training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"8CfZlUevmkg7\"\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -Uq \\\"trl[peft]\\\" bitsandbytes trackio\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Ou0VO1gHklS-\"\n },\n \"source\": [\n \"### Log in to Hugging Face\\n\",\n \"\\n\",\n \"Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"4Ncx0wYtnYCW\"\n },\n \"outputs\": [],\n \"source\": [\n \"from huggingface_hub import notebook_login\\n\",\n \"\\n\",\n \"notebook_login()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"vNylrNdqkoN-\"\n },\n \"source\": [\n \"## Load dataset\\n\",\n \"\\n\",\n \"\\n\",\n \"We'll load the [**trl-lib/llava-instruct-mix**](https://huggingface.co/datasets/trl-lib/llava-instruct-mix) dataset from the Hugging Face Hub using the `datasets` library.\\n\",\n \"\\n\",\n \"This dataset is a set of GPT-generated multimodal instruction-following data. We use a processed version for conveniency here. You can check out more details about how to configure your own multimodal dataset for traininig with SFT in the [docs](https://huggingface.co/docs/trl/en/sft_trainer#training-vision-language-models). Fine-tuning Qwen3-VL on it helps refine its response style and visual understanding.\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"0AcyX6Jd1_hp\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_name = \\\"trl-lib/llava-instruct-mix\\\"\\n\",\n \"train_dataset = load_dataset(dataset_name, split=\\\"train[:10%]\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"JFtR4Xyx4FYO\"\n },\n \"source\": [\n \"Let's review one example to understand the internal structure:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"YLrEY_v8m0eA\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"qeZCtRB1m5xj\"\n },\n \"source\": [\n \"## Load model and configure LoRA/QLoRA\\n\",\n \"\\n\",\n \"This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"gt05dgXgm9QR\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import Qwen3VLForConditionalGeneration, BitsAndBytesConfig\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"model_name = \\\"Qwen/Qwen3-VL-4B-Instruct\\\" # \\\"Qwen/Qwen3-VL-8B-Instruct\\\"\\n\",\n \"\\n\",\n \"model = Qwen3VLForConditionalGeneration.from_pretrained(\\n\",\n \" model_name,\\n\",\n \" dtype=\\\"float32\\\",\\n\",\n \" device_map=\\\"auto\\\",\\n\",\n \" quantization_config=BitsAndBytesConfig(\\n\",\n \" load_in_4bit=True, # Load the model in 4-bit precision to save memory\\n\",\n \" bnb_4bit_compute_dtype=torch.float16, # Data type used for internal computations in quantization\\n\",\n \" bnb_4bit_use_double_quant=True, # Use double quantization to improve accuracy\\n\",\n \" bnb_4bit_quant_type=\\\"nf4\\\" # Type of quantization. \\\"nf4\\\" is recommended for recent LLMs\\n\",\n \" )\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"jyklRvNxnHmy\"\n },\n \"source\": [\n \"The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a **base model** (the one selected above) and, instead of modifying its original weights, we fine-tune a **LoRA adapter** โ€” a lightweight layer that enables efficient and memory-friendly training. The **`target_modules`** specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"ME1im5gh2LFg\"\n },\n \"outputs\": [],\n \"source\": [\n \"from peft import LoraConfig\\n\",\n \"\\n\",\n \"# You may need to update `target_modules` depending on the architecture of your chosen model.\\n\",\n \"# For example, different VLMs might have different attention/projection layer names.\\n\",\n \"peft_config = LoraConfig(\\n\",\n \" r=32,\\n\",\n \" lora_alpha=32,\\n\",\n \" target_modules=['down_proj','o_proj','k_proj','q_proj','gate_proj','up_proj','v_proj'],\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"mBAfaiA-nbdm\"\n },\n \"source\": [\n \"## Train model\\n\",\n \"\\n\",\n \"We'll configure **SFT** using `SFTConfig`, keeping the parameters minimal so the training fits on a free Colab instance. You can adjust these settings if more resources are available. For full details on all available parameters, check the [TRL SFTConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.SFTConfig).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"GQPxXvu-2Ngc\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import SFTConfig\\n\",\n \"\\n\",\n \"output_dir = \\\"Qwen3-VL-4B-Instruct-trl-sft\\\"\\n\",\n \"\\n\",\n \"# Configure training arguments using SFTConfig\\n\",\n \"training_args = SFTConfig(\\n\",\n \" # Training schedule / optimization\\n\",\n \" #num_train_epochs=1,\\n\",\n \" max_steps=10, # Number of dataset passes. For full trainings, use `num_train_epochs` instead\\n\",\n \" per_device_train_batch_size=2, # Batch size per GPU/CPU\\n\",\n \" gradient_accumulation_steps=8, # Gradients are accumulated over multiple steps โ†’ effective batch size = 4 * 8 = 32\\n\",\n \" warmup_steps=5, # Gradually increase LR during first N steps\\n\",\n \" learning_rate=2e-4, # Learning rate for the optimizer\\n\",\n \" optim=\\\"adamw_8bit\\\", # Optimizer\\n\",\n \" max_length=None, # For VLMs, truncating may remove image tokens, leading to errors during training. max_length=None avoids it\\n\",\n \"\\n\",\n \" # Logging / reporting\\n\",\n \" output_dir=output_dir, # Where to save model checkpoints and logs\\n\",\n \" logging_steps=1, # Log training metrics every N steps\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub=True,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"bF4GtNO2ne1k\"\n },\n \"source\": [\n \"Configure the SFT Trainer. We pass the previously configured `training_args`. We don't use eval dataset to maintain memory usage low but you can configure it.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"TwBeQKQC2RfZ\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import SFTTrainer\\n\",\n \"\\n\",\n \"trainer = SFTTrainer(\\n\",\n \" model=model,\\n\",\n \" args=training_args,\\n\",\n \" train_dataset=train_dataset,\\n\",\n \" peft_config=peft_config,\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"K9Ub3jTDnfcD\"\n },\n \"source\": [\n \"Show memory stats before training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"u6_Vsv_1KtVU\"\n },\n \"outputs\": [],\n \"source\": [\n \"gpu_stats = torch.cuda.get_device_properties(0)\\n\",\n \"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\\n\",\n \"\\n\",\n \"print(f\\\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\\\")\\n\",\n \"print(f\\\"{start_gpu_memory} GB of memory reserved.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"4NiFu9tcniBP\"\n },\n \"source\": [\n \"And train!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"pbJXrhA0ywra\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"miZ2I1A9nnM4\"\n },\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"8jegvQGlKyEu\"\n },\n \"outputs\": [],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"3lrrYfPunloQ\"\n },\n \"source\": [\n \"## Saving fine tuned model\\n\",\n \"\\n\",\n \"In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"MNfRlfIGKSHI\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub(dataset_name=dataset_name)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"pFq51FWEK1DX\"\n },\n \"source\": [\n \"## Load the fine-tuned model and run inference\\n\",\n \"\\n\",\n \"Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"wAm1iQc8K1uY\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import Qwen3VLForConditionalGeneration, AutoProcessor\\n\",\n \"from peft import PeftModel\\n\",\n \"\\n\",\n \"base_model = model_name\\n\",\n \"adapter_model = f\\\"{output_dir}\\\" # Replace with your HF username or organization + fine-tuned model name\\n\",\n \"\\n\",\n \"model = Qwen3VLForConditionalGeneration.from_pretrained(base_model, dtype=\\\"float32\\\", device_map=\\\"auto\\\")\\n\",\n \"model = PeftModel.from_pretrained(model, adapter_model)\\n\",\n \"\\n\",\n \"processor = AutoProcessor.from_pretrained(base_model)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"a5rLWJdOvwGQ\"\n },\n \"outputs\": [],\n \"source\": [\n \"problem = train_dataset[0]['prompt'][0]['content']\\n\",\n \"image = train_dataset[0]['images'][0]\\n\",\n \"\\n\",\n \"messages = [\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"user\\\",\\n\",\n \" \\\"content\\\": [\\n\",\n \" {\\\"type\\\": \\\"image\\\", \\\"image\\\": image},\\n\",\n \" {\\\"type\\\": \\\"text\\\", \\\"text\\\": problem},\\n\",\n \" ],\\n\",\n \" },\\n\",\n \"]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"qiu-ROFeBPhA\"\n },\n \"outputs\": [],\n \"source\": [\n \"messages\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"base_uri\": \"https://localhost:8080/\",\n \"height\": 497\n },\n \"id\": \"qGzEXSH5BQwG\",\n \"outputId\": \"611d6bfd-fb72-4737-e847-1611132d49ed\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"image/jpeg\": \"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAHgAoADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDxoDNPApQvrS4Ar6w81sB0p3SilFVYgQZzTsZ7Ume1LmqEGKUYopM0CHUmfrRRTAO9GKKWmAYoGKQmiiwC556UZoop2AKWkpeaYgHWigDmngUwEFOxQKdgUANC8U7FLipDHmPcDzSuBGAc0hXFOzSE5pgISBSgnNBWnKBmgAFN9u1ObgU0A0DFPIpM808Djml4XpSuKwqLn71KxHbpTS4wfSo2k9P1pX1HYkLYprScVe0nRL7V5SIExGD80jcKK1NR8GXlnatcRSJcxqMtsGCv4VzzxVGE1TlJX7HXTwNepTdSMXyrqcyXJHNNzmrBtGCB3ZVWiOext5F8zMvPIFbSqxijnUJMhSN5Gwqk/QV2HhvwsZCl1eqQAcpGe/1rA026TUdet7e2hMULN8wPXA616zbRgsqDoK5Z4hy0ifRZNl0Zt1aq0Ww+OBY4wSAqjoAKq3U7AEJwKvTEswHYVn3QrjmfX00jOkLNySTTFXPbNTMBmlUCuY6bjNpAqRTijHftQB8pJpEji4xVa4uhGp60TzBBWHe3e4kA1y163KrCsRXdy00uB3Ndb4dtvs0IJ+83JBrj9NiNzfKDyAa7yyXawPavjM3quWhyV30OjtXwVGe9dJA6rtGe1cpbNkit22U7QSa+TdR05XR49eNzpI5flHpU24NVGGQbFHWrJO3noK+swmLk6d27nkTjqecfE3wUdTthqemwj7VEP3kaDBkX/EV4XNFJHIyOrKynBBGCDX1zLhk5NeYfEHwLFqMM2p2CbLxRl0XpIB/Wu6GJSdnsehhMU4rkkeH59KeGzSTQyQsVdSCDyCMYqMNXanc9mE0yyO1PBqurc1Krg0zspyRMKUVHn3qQc0zpHCnA0zJpc0ASA08HmogTTgaQ7mtp2qzWbdcp3XNddZ38F7F8pByMMjc156GqzbXUlvIGjYgg12YbGTovXYmUVI0fEfgGHUC1zpWyGc8tAeFb6eleYXdncWVw9vcRtHIhwysMEV7Vpmtx3ICSELJ/Opdc8O2HiO12XCiO4H+ruFHI+vqK9uM6eIV0fP4/KlK86Wj/AAZ4Rik6Vta/4cvvD14YLpcqeUkXlXHqDWNispQcHqfOSi4vlkrMOlODfnTcUnStIVHEzlEnV60bLUpLU4BDIeqnoayQ3PNPDV6FKuc86akrM9T8NeOJ7ULDMWngA+6T8yj29a9KsNStNTtluLaVXU9h1FfNMNw8LgqxBHpXT6J4nuLGdHSUxtn5mHRh7jvV1aEKyutGcMqU6TvHVHu0i7ulQSJmsnRvFFvf7YZ2SOY9CGyr/Q1vlVYZ4+teZUpSpu0kOFRSWh82gYpfwpvWndq9dFMM560opMUvQDmrELRR2o/GgQZpaSigC3b2MlxDJIhGEGTmqtPWR1QqHIB6gGmjpUwUru5cnGyshD9KWlI54oxWiMxDS9KMUoGaYCUUuAKcOo4piE280oHrTzlutNzTQAKXB60ZPSlBFIYoAFOyDxTRg/8A6qWgAz1pecUmeaUDNIAx7UEZqaK3lmbEcbOfRVJrf0vwbqN+TJKot4VPzM/UfhXPWxdGirzlY1pUKlV2grnNgH8Keq4r0uPwfo9rB/qjdOq7izOfmHsKs2uj2EoYGygWBQMHZ2PrXkyz+le0It/gelHKajV5SSPKWXHWmlwvSvTJfCmk6rKY4Yms3AzvjOVPboazJ/hpP9pMUGpQN3w6EHFdVHNqE93Y56mXVov3Vc4MyVdsNK1DU322tu7j+9jgfjXoFr4G0/T03y/6RKOu/oPwq5bLLbzoI5IliBwUUYqMRm9KnpDU9PCZDOprUlbyRBB4a07SNJWaa0SZwB5jyjOTSWlzolvlY7CBNxyfkHWtXxNMBoKop535NeeSTMr5zXy9XHV3NtSep9ZhMvoez+BaHd3Oo2sFkzxBAT91VGAKi0e/CSAyEFW6g1xEl+7YBJwO1aen3oliIB6Vxyqyc+bqenTw1PkcO5ra94N0fW2kmtJzZ3J5AXmMn3Xt+GK821/w1qXh2SMXqIUlz5csbblbHUexr0K3uGMwAY9a6S90a013RRZ3yEqRlHH3kbsQa9jCY6tWlyS2PBx+TUoLmgrM8Y8J3C2/iWzeQ4Vn25+te22zbZK8Y1rw7e+GdWSOYbkLboZ1HyuB/Ij0r0fQNbF7bIsjATKMH3r1KVW0uRiyulJU5RfQ6Sbgms657itAMJ04xu/nWfcDk1vU2PTp7lBhg0qEDrQ/BpFPFcx0dBXyV4OKhMpRSCcinyPgdazrmYAdaxqTsgSIb254ODWM5aV8CpbiUsSM1LaW5PJ6mvGxNXqZzlY0tEtRGd5611trwv1rEsIdiDitq2ya+Tx87s4aupr2mSwxXQ23Kc9ax7GIlQa24VKgcV8/Ui5S2PMrtFuAuGUnpVua4VgAD9aihwEOaqythjiuynOdGlyx6nC0pSLBuNvfiql3OGBweMVBJN0Gazbu5wp5rWliJLRlxpanA+ONBhkL3tsoEn8aj+KvK5sxyFSCMGvXPEd+IonZz8oFeSancJPevJH90mvp8tqSnHU7otxQwS+9OE1VN3NKDXq8ptGszSjlz9asq2R1xWVG+D71cilzUNHfRxF9GXM04H3qFWyPwqUHNI7U0x2aUGm0A0DJM04GowaUNQCZOkjIwIOD610ela+U2xXLZXs1cwppQ2OnWtaNaVJ3iNpNWZ6NcQWWr2TW11Ek8D9j1HuD2NeV+KvBNxojNc2xa4sj0kxyns3+NdHpmsS2bAE7k7g12dnewXsBK7XRhhkbkH2Ir3cPi4V1yyPKx2XQrq/XufPZGDR1r1DxR8P0nD32iIFbq9r/ADK/4V5nNC8EjRyIVZTgqRgitKlJx1Wx8rXw9ShLlmiHFKCRRSE1nGTiczSJFapFkIPFV+lODV2Uq/cxlA2LHVJLdsfeU8FTXovhvxy8WyC5ZpoOgJOXX/EV5KGqxDcNGwKkgiu1TjNcstUctXDp6x0ZbxTucUYorRGdxcUmKWlpiDFHWl/CjvQIBRR7ilpiClooxQAU8AEU2nZAHvTEJjNLSZpKAF6U4etNH1pc81Qx360FetJk5oNJCFBFHtikAA606gdgAx1pRSE0hcdqVx2H+9XdKtxdalBC4+RmG76Vn+Z6CrdhdNaP5+Mt/D7VwZjiHRoNrd6I6sHR9rVSex67ZxwwxmOzRVQLwQvpUEt64L25LLMSDlDmsa31RYbZFQOruA3yHlSev1Fbllo73jrKTIJGIYPIm059RXxFRSbu2fTJwgrEkcuA0v2hEycBJBjHHSqNzr7qJIre0cyL92QDgjup9q6mXw1GIla4zNKD1Pf8Kd/Y8RswwiHBx06Vk5NE+1izk38RO9srLp0lvIE2sANwz7U228Z2sfyXUbI4x8+OvrXZS6dCLLYEHA9K4LVNNj81gEH0x0pOo0XCakXLjxnpJuQojmYDgsBw1WTogvLZNU0+4DRuN/lk5x7fWuXstKE96kIUAE4J9K09moeH7kmJmMOTujzwfeq5+bc6KVeVJ+6w8SXTR6ekLZDA9DXFyzZFddBr0WqQtp3iC2G13Pl3K8NET05rD1nw1c6fiS2mS8t2yQ0PLKB/eX+tCs2erQxyas1Yw3k696lsrtoZDg1TlJAx3FJbqzvtVSxPQDmrcVbU3WJs73OjsL8m+jB6E16HaapHIkceRjGCO9eWrYahb+XcmzuBHu4fyyQcV02l3DXF1Gig5PNRTqypSvFm/NTxEdXqjstT0+11axe0u4xJE/IPQqexB7GvOdQ0m78PXg+YtCT+7mHRh6H3rsJtcFlq32Sf/V7Rz6VsSW9vqVm0EqCSGQf5Ir6ClVjWp3vqckP3Eubuc1omuJcYjkYLIP1rfljW5TIwJOx9a4PWdEutBvFILPbucxSgfofetjQ9eEmILhtrdAx7100cRryVDqnBTXtKZcnRo2KsCGHaoAwGa35Ehuox5mAw+647VgXaPbzsjDHp6EVVSPKromE76MrXEvymsm7mBXirVzLx1rImbe+K8uvUNHohiAyTACt6zhztGKoWVsDzj8a37OHGK8LE1LnLOVy5DHhQK0bIAygHpVdVAUcc1oWto7hWSvnMVK7OWfmdLp0WdvpXQQQxopJ5rn7CGSJfmfI7YrSieV9wUk4rmozUXa12ePXi5PRlqd1VcLWbPJtNWEzISN3I6ZqFwhcrIKyqvmXM9CIRtoZNxMeTWPe3IiRiTz6Vt39i6qXhO4dhXGai8jzFCDkHGDRRpXlqdVNJ6mBrLXeqo1pbQtLLIcBVFchqng3XdLybjT5duM7kG4fpXufhfSI7KP7RIoMz/pXUvEk6kMoPsa+hweOVNcsEc9Wq1I+Q2BU4PUUA/TivefGXw0t9WnN5YKsM5HzqBgN7/WvLtQ8B6xYXiwPbswZtocDivZhjKclduxUJ3ObXJ6VYTd2r07SfhlboqSXkjyYHzKOBW7F8PNEQ7vIJHbLGuSWa0Nk7nQpuJ47G5BqypJr1mfwJpkG4rb/L2PWsHVvBUUUHmw/Lz2pwx0ZM7qGK6HDg0tWLvTp7NjuXI9RVTNdcZqWx6UZqSHil3c0ylGaoslU8U7ODTF6UooQDwauWd/NZyh4nx/WqWaAferjJxd0O56BpWtRXqhSdko7Z/lVTxF4SsPEUbSYFvfdpwOG/3h3+tcfDM8bhlYgg9jXU6V4hDARXR57NXs4TML+7M5cRhYVY8rV0eVaxol9od21tdwlCPusPusPUHvWZ0r6CvLKy1eyNtexLNC3T1X3B7V5X4n8EXehlrmDNzYk8SKOU9mHb613ypKS5oHyuMy6dD3oax/I5DFNzTyMU3HNc+qZ5jFBp6moyKTOK2hWcdzOUTobeCS4kCRqWb0FLcQNBKY2BDDqDV3QtSXStWgunQOiH5l9qs+KLmyvdWe6sgwjlG5gezV6vM+a1jzDEFLSCjFaAApQOaBij8KYC0dqKcoyaBCUUp4pKYhR3pSfSkFHegdgpccZpQCe1G32ouVYTFLjNOHSjeMY4xSbFYQCnHGKaWx2phyaVylEkOBSF8e1EcTyMERWdj0AGSa3LHwlqN3hpVFvH1+fr+VRKpGOsmdFHC1artCNzAyT3qSK2mnOI0ZvfFdkPCsFquQrSuOct/hWZfF7YlMhAOwrzsRmap6QR61LJJ25qrt6GRNYC3hLTSgMeiLVrRdP+1XSlwfLHb1PaqIWS6ugWY49+1eo+F9Igt1SSQ5DAYGOtfO43HTrP33sdUMPTo35Ea/hLw4Aq6hfIrSKNsaHsO2a7mSGMqjopBXqAKjtDD9nCA4+gqVJvJlIYEqehrgc7mDvcbI8csJKuNw/hYVV+0FbYjG7P6GprhF3MVHB5x6VjNdGCco2eT8pNYzkyoxuSX1+qwLjhsYNclqMingYznJrV1aZQPl+93rmLhy7E45rBttnTTjZF3SBHFM07jjOB610l1Db3UKnYxB+9k1xtq45VjgHpWvY3kkWVJ+UnHJ6Gmm7hOPUtf2PYSRmGS2DZPPrVXVfB7aRbjU9LvtgUZMcx4H0NOvdaWxJluGBUHt1Nc3Pr1x4kvxHdOVtIhlYgcA+ma2hJihz3VmMvby21uBvt1hbGYD5ZoRscH1yOv41lW0KW8g+ygoegbOSfxrQvGsmUiINE68FM5H4Gsye5YLsAwF5HvStJ9T0IVEtDQh1KdWKeYV5ywBxzVsvfxWv2yCcSRKeSPvJn1Fc/O83EqqdrAZpba+nj/wBW7qemAetHIupspvdM05dUlmk3zbJHIxuZATWzp2u4tpFku/I2r+7VUyCf6VybXIZgrRFW9auwWbzIHCOI9wUknoT7UQ5ovQKtZNWud5oFz/bem3UF4BcRvnajgYPHQehrjNe0OXSZEuYNz2UvMcnp/sn3FdLoEEtrcyxBVKwrkrnqcc498VpeHr221C4vNDvQjxSsTHu6Bv8AA1pQryVb3noPD4iVFua1XU5LRtc+7BcN7KxrpGSK8i8uXpj5WHVa5rxT4Vn0C6aWIM1oTw3UofQ/403Q9aEbCC5Y7TwrelfTUMR9iex7NoV4e0pia5p8uneWXdXWTO0r7VkW8e+TPNegSW0GoWxguF8yFuQR29wa52XQJ9MnG754GP7uUdD9fQ1xZhh3GPPDY5nNr3WLaQYUYFa1tGcgKMmoLdAqj1rd0uxZpFkb8K+NxuIVNNsynKyuWLPSTIA0ucHtXT2Olo0B2YAFV4gAVB6Ct22MaRKFPX0rzcA4Ymo/abHj4qtK2hBZWoQSIeW9DVeOb7Lec5C5wavXmIHWWM4OOR61l6hNFMVljb5m+8Pet8dCNKKlDSUX96OWneb12ZZu18u5Ux/xciqt3ITKSRg45FVJbt2SP5iGToaia5aX5nOTXkVqqqX5Va7OmFFq1zRtpkkRoX6kcGsu60+CaTfIi+bGeDSeYyOGU4IqKSdnkLE8miNR8iit0WqbTujU0xwZlB6DtW68aFCykAiuYtJtrZzWlNfKICN3JrpwmIjTTjI5a1NuWg+e52DBIrLmcTyYYA8/lVd7gs4GScdakh+YlgazxGKm1yp6G9Ojy6k6sc4xxjtU0ZA4Kgg9qaqkjBxmrrQFLeNwnPqO9YUIVKl5LoKcktCytmLmweNcKccE81lXGmwR2ci3A2yKp5ByPatiz1SG2tnWRTnuPWuV8RawHQhRtUAgHPUV9RCtho0YyUrytsc1KNWU2uhwOsLGZCvFcRcKEuXUDAzxXV6jP5jsR1NS6J8P9U8SXiSlDbWefnlcYJH+yP61vhJu92e5LERoRvJnHBTTwDX0fa+APDlvpwsv7MhkTGCzjLH3z1zXl3jb4d3OgM95Y7rjT85P9+H/AHvUe9epa6uTh83pVZ8jVjgxxRj35pSMGj8KlaHrdLoPrS03mlH5VQDqcGxyKjHWjNCFc3dK1yS0YRyEtH6eldlaXsN3CSpV0YYZTyCPQivMkPfNX7HUZbSTcjYI6jsa9DDY6VJ2exM4qZP4r+HqXKyX2hoFYDL2g7+6f4V5jLC8MjRyKVdTggjBBr3vSdWjvVBDYkHVaq+K/BkHiu1EtoscOqxj5W+6Jv8AZb39DXtqUK0eZHzWY5Zy3qU18jwqm1Yu7SaxupbW4QxzRMUdD1BFQDpXPKLTszwdzaFLk0hFLjAzXvnlBmlB45pAKWgAyM0ueKbTqYAKXNIaM80AOpVAOc0gPrSFsUrhYf8Ax0pwOtQmQ9KNxPAzSuNRJd6rTTIe1MCnvVi2s7i6kEdvC8reirmk5WLjTcnZEWSe9G32rrNO8DX10A124tk9PvNXWaZ4Q0yw2sIfOlHO+Xn9KxniIRPUw+UV6mrXKvP/ACPPNN0HUdUOLe3bb/fcYUfjXU6f4BRcPfTlz/cj4H512jNBbrgsOP4VqGW9yAIlxxyTya454uT2Pdw2TUaesvefnt9xFZ6NZafHiGCOMD+Lv+dPlubeMYUF2/SqzuznLMTVOeQKprjnVb1PXhRjFWRW1XUnWJgDtHoK4S/naVzzkk1s6vdbmKg1zjuTIxryq03JnNip2Vkb3haxW5vMuwUjpuXIPqDXqlnF8ke6NVHoBxXG+CrDFrDcPGRGz/Ox/pXoUkDx4kjIeIHjHavMrrU8qUraF60TH3cj6Grg+ZGG7ketZ1ldYmPze5q886MwKDPrWatY5pJ3IPteS8UowexrF1QbZOWJ28g1oaogKiSJSeOcVzV/fllAJ5HFZzb2Nacb7FW8naVyc9O9ZrN1596sRShZCTyO49apXDgSkL0qUjew5fU0/wC2rbqXmUkDpjvSW4M0gQYzgnNUtVAhj3KSzEfrVK1ylG+piarqUt7IDKcKhIUDjipNGgkZ5XlJiBj3Rbv4+emak0vShqL3FxK6fuRuKH+Ktf8AtK3t0topbVTtA3nb95e1b3SQjJ1KJpZWeTaJAcHHFVfJldijDIUcDP8AKuj1F7e3iG1opSy4G1gflPQj/wCvWRY3MsjiCRyVhicxq3bPpTTXUab6Dbe2lkQNFggAlgx7DrVjyltreR9oOOeecfSn2wj+x7uQ5PrxU8hjltpgo27scCsnI2TY9Z4Hjjykcir85R+Oe/IrYsPsE8V0j2XlyxkPFLFITt54yO9c5EWjk2dD0GRW9oSyz6nEqnBdgoI7+1ZSm+gOCerOkSASX7zQxqcxgEocAn1+tcHJcS6frBuIuCsmcH69K72aNbN5CrMjquWHToa5XxpZRxXkN5bvuiu4/MIAxhu9RC8kzalJRkl3O9hvLXX9OjLFX8xdjo/Iz6V574k8JS6Wz3VirPajO9OrRf4j3rK07WrnTZP3chA9D0P1Fbn/AAmd7cSBSVCntjt6V3YfGypx5Kiujqo89CV6b07GfoutvassFwSYj0PpXawzRTw7H2yQuOlc1rOii40SDVrO1RXBbzkh4yo/i2/zxWbomttZuIpSWiJ/KvcwuJ54JVFozvajiYc0dzrf7LaG4BGXgP3X/oa6G0jCxrjsKoWN6jIMEPG4/AitREAUNGcp/KvleIspnFe1o6xPMrOS91luFhvGRWxbsjJwMAVkRrhQx61rWwAtW3fexke9fLZbJxqtHlYi1h90N3yvwCOK5adzDOyH1ro7ifzIgCPmHQ1zN/HNLcFlRjWuMrwqzsmaYRW3InmBzURmxSx2txI2PLI/3qnGkzMeZFFcd4R3Z3twW7KzTHHWozIRWtFoqcGSUnHUDitMadaPavAIgNw6qOaIVKbdrnPUrxjscylwVB55pGuSRya0dQ8PTRRLJah5P7ynqfpXNyTFSQcgjgg8V0Kj5Dg4T1iacTnOQOver1sxDkCs6Bv3aD2q1FOYyDxWFSN9DVx0sbVo8aynzRlSPyqd74pCsadjnJrMgZrj7i/rV+3gE8yJJwBwSKuj7eypw0TOGpGKd5Gbd3W1STxnrXN3Frf63N5VpEWUcbjwB+Nd02hwTP8AviWjz0Hf61tW1nBBEqxRhVHQAV6uDy6UXeW5nLGKC9xanG+HvAMFjdx3l7J58y8hcfID9K7yKFEAAUAewpEAB4qcV9DQpqKPOq1Z1HeTFAAqC6t47qB4ZUDxupVlPQg9RU2aQHNbykrWMlo7ngPjzwTJ4du/tVmjvpsp+Vuphb+6T6ehriSMH3r6p1LT4NR06ezuFDwzIUcexr518W+GLjwxq7WrlpYGUNDPtwHH+Ip2PpMrzDmXs6m5zx9e9JnmnZpueaEe9uLk0m6kNMJpk2JAcU8Nz15queDT1Y9qGBds7yS0uVljYgg+tel6bdrdW0VxGeGGa8o3HNeieECx0OPIP32wT35r1ctm+dx6Gde3Lqcd8WtIihv7XWIV2m9UrOMceYv8X4gj9a80zivdvijbwP8AD5Z2wJI7xAmevIII/KvCDXpVraM+JxkFCvJRNzvRnijNJXtaHgC5x2oJoyMcmm5zTKHA04kdvxqMNzTufwpMVgyO1GaTgCge1DHYNx70oAJ5Jq3Z6Xe37bba3kk91Xj866rT/h9M+176dYx1KJyfzrOdSMd2dlDA163wROLC5PAPWtrTfC2qajgrCY4/78nAr0iw8MaZYhTFaIWH8b/Mf1rTZ4YhhmyfQVyTxdtj28PkS3qy+SOS07wHaQ7Xu5Wmbuo4WuptdPtbCILDFHCnsMZpGu3+7GoUeveoDljlmJ+tck68pHt0MFSpK0Y2/MtPdxJxGu8+vQVWkuJpOC21fQcUwgCmkGsW2zrUUhMYpCaUnFMZqRQ1iADzWTqFxsQ81enkwDWLM0U0xE0hWMdSvWuWvUsiJyUY3OeumMjE1ntGQ5B616C3gZ7zTjf6TdC6GM+URhvwrirm2mhuHimieORPvK64I/CvOc7ux5dapGauju/BE8z2pspCwhPzR8ZB9a66OWa0lKjmP0PSuG8H3dxbqI8ZiPKnH3TXoAEN1aAK53HnBrlr7nFIpqZkmMiLwew6Vat7878sQGHUYqmWa2fJYkDtUqy28gJKNz3HauUlmjcSl13J93viuR1iEhzIvHPSt37THEu1XIz2NUL1VuIWHftim9Qhozl/PIb3pjHcc5yfUVDfB4pSeuDQbloYYBnDNlvwosdMVctmT7PEFB/eOeo9Ky9QkM1w6NyoHGa02H2mGNplKOASGHRvSsa6lLyF2GGz1FKOmrNZbWRNoyCNrhkPIwMdc+tW9Ttz5kT5BVYwq47fWoNMXakvX5vQdcVNLPJJFg8gd6JTEo6FTylfB25OOc1FHAhnTZvSQdSD/KrcDpuPmKxAB+72NRedmaOTOGPUihN2DQtJDcRHB79Mjr2rW06zLGaHCuOhK9qk06dLqYK+W2AEfQcmrmpQ/wBj2huh/wAtucdME1nKXQLnO3hYynIJMZ2knv6V0nheweO9aaRcqIiyk8ZJ6GuPS6e5ZYEbJaTcw9K9M0q1cXMECspRY4twDe/I/OqnpETnoN1kmC/kU7SjRqjgj88VlawkQ0uyaQCSILtzzxz2rc8TYuIg0BzJMhZQeu5Tj+VYMd/BeaW1vdIyBF8tyRyjdQcVzRvfQ0g/dTOY17Sba2uC1tMZInXcrY6e1YNsSk+CcjNbct4RZOzMjoGMYXPIrmmnxNkdz0rsjF21No1LHuthaCDRbHacE26uMDhs9a5PxF4SS5WS/wBLULMPmkgA+97r7+1dH4K1aHVdEgsJGxPAn7pieq91q9fW8ttOJI1+uK6KeKdC19YsihXnTqNXs/zPLdI1iXTpvKmDGPOCp6g16BpupBkWSNwyMPwNZuv+F49Wie9s0EV71Zc4WX/A+9cppeqT6RctBcIwUHa8bDBU17tGtGpCz1iz1m6eLhp8XY9hheOWIMh4HUelKbpkG1TkYrmNN1QFFkjbcjVrGRXj8yM5X09K+RzvIXRvXw+z3PHqYZwlaWxbE5bqaYZcGqZl5xSeaSa+LcHfUFSLhkzSiQ55qqH9+acrVLiDgXVYmrMTlSCOtUUPQGrcRrKWmqMKkTSjd51wx6CsHW/Dkd+r3Fv+6uSMjHRz6H/Gtm3bDgbtuetXTGNuwnIzwRXtYLnrU+a92ji53SleJ5Jp+rwvI9rO3kzxEqQ/Gcdf1rXt5FuQDEwfJx8vNZXifwfdXnjKf7IhWGbbK0jDAUng/Xpn8a7Hw/oNto1mkEfzvnLOepNdNbD07rler6HfLFLlvYs6RYTRbi3CsMFa3re0SNfujNNiXHT9KsgE+tenhKEYRVzy61WU3cdsWnADaQOhoC+tKVAr0bNanOMU7TUquDxUO4AH1oUFcHoDUwm4vQGizTe9OzkVEx+fjpXRPoyUSA5rE8TeH4fEOjXFhKQpkXMchH+rccg/nWwTnihiNmD3q4tdRxk4u6PlPUbC60u/ls7yFop42wysP1Ht71VznvXufxF8JrrumtdwIP7QtlJiI/5aL1KH+nvXhDFo5GRwVZTggjBBppqWx9ZgMaqsLPceTTSeKTdkdKTqeKdj1E0GaVT2rU/4RvVjFG4s3w4BXPHFbel+CXdg9++B18tP8a6qWEqVdUtDL20LXRz+laXPq94tvCNoJyznoor1PT7GOztorWL7ka4Ge9LYaXDZwCK1iREHXHH5mk1bWbPS7OVoXEs0aFmcfdXA/nXs4bDxoLuziq1nJnnvxc13c9v4fjIIt28+cg5+cjCr+Ayfxrys1c1K8lvr6e6mYtJK5difU1SNE58zPksRPnqOTNvNGe1AxS/QV72h4wmPzpRjNTQ2k8/+rjZvoK27HwnfXaq3lsFPc8D86TkktTSnTnUfLBNvyOfIGeBT44JJWCxozMegAyTXoNj4EtoyGu5S5/uJwPzrprTS7SwjAggjhUd8c/nXNPFRW2p7GHyStPWo+VfezzjTvBepXu1plFtGe8nU/hXW6b4K02zZXmVrmQf3/u/lXQtcQR9MufbpUD3Uj8DCD0FcVTFyex72HyihS15bvzLSRQ20YUBY1HRQMfpTWu0U4jTPuapdTycmnqpIrllUbPSUFEkaaST7zfgKaBTljPFSpC7NgKSfQVHMhuSRDz6U0irDQso5BFQEYqk0wjJPYQ4qN3AokfFVJJfepckirExkB6Ux3wDUAkqtcvLJ8ikKD1JNY1K6itSkh7brpzHFz6msTUYTDMYxyR1rqtFFtb5zgsR1rJ1yEfay68jGa8qrVcncwxC91oteDtefTbxbec5tpPf7pr0bU/D2neIbPFzGpcj93Mg+ZfxrxtFw46+or0vwXrouLf7BO+Jovuk9xWLd9TxKkGtUYdhHceGNRfQ7yEPDK26OUD9RW7MpSJHiX5QcZFdJqukQ6tbKsqgSIco46qaw1jNrA1pKpyjc1nPVamXMmihcEtjeMKe9QoXQMA3tV+4hjMYZB9azJ02SDDHb65rK1gWpFcOSeeq1UNw+CmSPSluXcgkHOf1rPa4JkVW5HfHpSLjC5Hd2zzk5G0dT6kViTSG5vV3cBRtH0FdSzfumXOWHKjrWELQSrNlR50ZLZz1qldnQtFY1EuYF014mTcRGSGBwQawYYGu50UA7QPmPoKHuGCOvUMAM1saWIoLYq25S4+ZuoqCmQvJHDIfLGEA2gelMnZEiQLtJxnPrVrV9PFk0OJVlSZN+5e1Zsm3ywueRmokmnqNNNXGC7kjZmTAJGCCO1QRTHKgoDg0XDCMDOOnUVDG244GM1rFMiTN7ShIk4k+VUPRicAVc8WX0kmlWcbn5kBDYPWqmnoHt5QTlGOGUdRgZzWXr9z5yKqnsABU8nvEt6FrwlAk9w80pPXaGHYY9K9Q8Lqko80OHdwrH6jIx+leZeDbVptYtbVX2mVwhz05616Xplo2leKdRsQqxxoI5YWI6r3x+ZpV46XIclbl6mRrjSrJb2+4hllcDBwQD7Vzd1fOl/HJlNlwvlyIBgAjjJrp/F0yw61ZSFgdsjbiBzwRXL6vFavqzKJB5UkvYdM1nS0RvF3imc1PDLa3NxbS8Mh9c5/yKyZ0dMMVIz0966W4tS4nR+J4jxkdccH9Kk1LS0k8LQXWEE6S7Tg8lSO9dUJJuwm7CeDNWktNRh2tjB717uyJe2MM4HMiDcPevnDQvl1FFOByODX0L4clZ9MWOQ8DkUQSc3B9TPEp8qqLoOt7ZIVmG0FtpK8ZrkfEHh+DWRuJEF0v3Zguc+zeor0RE/fbiox0Ncfq15Baa6+nuQkpUSICeHB9PevYwfLD3HsxYOtJ1Hy7nnEE974fvza3aELuII6hsd1PeuysL0MiyxNujcdOxqTVdMtdZtBBOCrLkxzKPmQn+Y9q4yKW98N6iba7QmInIIztcf3lNeilZck9Ys96M4142l8X5nfMAR5qcoe392o/Mx3qtZXqyIskbh42H5irE0YADocxn/wAdr4/OchUG61FaHK4uLsywsoMYB6+tOR+aqIasJ2HevjalLk3IlGxcjPc1ft8Myj1NUYkDYyTWlaoBIMVy8vPJRRxVmki7LalCu3n6Vbt8rGVbk1bjhUwqOh65qERjduY5z3FfVwyl4aSqw69DyXV5lZlW/hDkP3HBpkESJyetS3GSjAdjmoo1I6muebSrXsWr8ti2sqgcCnGYA9KhGFHNODKRXfGtK25m4kyyhqVmzVc8EU/OFrohVctGS4ik4bpSCQM2SeBUE2SDzjPpTEDcHIqfaNSsh8uhoCQEcGmu2SCPzqFJADg0rSZHBrqdS8dTOxMTjAFMY5HNQiYL15PakeZVXczhR6k1UHzbD5WR3K+YhHoK8N+Jfhue31R9WtYi0M20TKoztk6dPQ8V69e65AmRFl29e1c9eXUl4xLAds8ccdK9PC4CpJ3loj0sHRqqXNsjyDTfCesXyrI8H2eJujTcfp1rsNL8HWNk0cs264mU5y3Cg/Stu61C3tXInlyw4Kjk1j3niZ2DJbJsX+8etezDD0aXS7PcXO1Y6C4lVV3SuqqPU1mT+ILWCMJGpkYdCeBXLXF5PctukdmPuagAyRV+06RRSpxSszam1q7vQE3kR54UcCuc8YXn2TQTCG+e4YJ+A5Na9umMHFcD411A3Or+QpykC7B9TyabbUbs5MfVVKg7ddDlmOTTaU0mK5Xc+Tvc7XT/AA3e35+RDjuT0FdVp/giGNN11Jlz0CjNdcBFF/rGVR1wPX6VG96i8Rpk+rV6VTGvpoe5QyKhHWpeT+5Fex0OztFBjhyw/iarzPDEPmcFvReapPPLL95jj0HSpBp90YFm8iQxk43heK4amJu9WerCnRoLlVkh73x6RoF9zzVd2kkOXYn6mtqy8L3t1AsiAAk/db09a6AeDrdrFARIs6n5pFbII+lcssSkZzx9Gm7JnChTU8NrNOcRxs30Ga9Ci8KWsNgcIjThTscjOT2zV6x04R28W+ONJSPnCdDWUsQ+iOWebRt7qOFtPDd7cbT5e0HpuOK6q08G2gg/eq8kh/2sAV0s9mslqI0OwjkECqNndlJjEXBAOK5KuIcZJSe551XH1quqdihB4XtQpieNCB/ER81XLTQrS2+dYlLrwDW0DuHHWkyd+COK0tpuc0sTUlo2UZtKtri3aN4IiGHIxXB6l4Uv7cSSBYzGuTkN2r0sZY9OBTZoVmQqwyCMEetawk47F4fGVKL0eh4HcyEEis6a5EYJJr0X4j6ZBbwW1xbWZV2JDvGvBAHGQK82Ons48yYkL121NStY+lwmJWIgpIfp05mlLP8AdzirOrjfiaLhQMECqYZYl2xjApUuicq3INcEqjk9TuSsrle3u2jfg1ZuZ1uNoJrNvYjE5dPumqy3LLIhPY1PKYVaqasy6EIbae3SrtndPZ3Md1EfmU81Fs8wgg4JGQfekjPylWHDHH0NZXPPlE9h0HVor61A3AtjI9x6VHr8B+zNPH95evHUVxPhrVGtzGp6xt1/pXosxS7s2K42Om4d/qKT1RwVIcsrnBC6dcjfkGqk1xtkxu496rakzwXMkecBW4xWc87E7iahIpI0ZT5pVVYAk4wTVWXT5YXkMh+YHC+9RQyCW5iTPVhW5LdG2lWTYJUj6hh2qZOzSNqasrnPXV5JDbxsgwUBB96qwXJFm4OC0vzFu4rT1q1t5bpZ7WUiCcj90f4fWqM9uY7p4lUBVxtx3rR2sNamU6fN1NadvcGO0DHqOv0qnPHtc8c+oqyqgRFc4yOlZXLsOuLvzbUoVA4G0ioGcmJUPKjmpEg3jaByOab8vllSvzZ4NK9x7FG9D/KAMrngiqw3G4VcFQa0CA4Zc8ilS12ASAbiB0rSMrIyaLlit7Dcx+UrBnyACMBgeKlvrKO8114Vj8ryztCe9SabIZLiDz3kEcbYBXqoz2rUeCdNafy3RnuCZo2UdxyKmUtSOtiho8D2Wu2U+CiCbGfcda7LxPrIi1pb2Mb1ECFiDg9xWh4m0eyfRDqlrEqSKySOY+AGIw1YGiaXFqTSQTs5aRcKhPHAz/Os52v72xEGpLn6oxNf1ae+u44o0JkYCQkL6+n5VmiAvebmADlQ2M9CK7HV4ra18RxRRxKjCEB8fn/KuR1dZrTVpEK4MbbfqDyP0qnJWtBG8NbXLesb5ZluliH7053J+oPvV7XL5b3SpHhiAYqqzqo4Pow9+KSw1GBtEWG7Ubkl3rgcjjFULiGe0so71GBgkcjrnBHIzURkuZWNHFdTkw3kamjqchsEYr3TwdeJJpK7uWznFeE3rYuPMUAAncoHau/8BayZ5PI34ZACVJ5I9jW1STg+dDcFUpuJ7BbzK0hIGPTPpXlHxgkFtrGn3EeVm2sCwPYYI/WvS7SRWcHrkfWvO/jRCn2CwmA+dJME+xH/ANauyNW9NPzOLDfu66aM7w14rS8Rba8cLN0Vz0b6+9dPdW1rqNq1vdxCSJuh6Mh/vKexrw62uGRuDiu/8O+Kwyx2l83skp/ka9TD4r7Ez23FVFzR3JHju/DN6sUrGazlb93KBgH/AAPtXS2l4rqGUh42HTsRViRYbq2eCdFlikGCp7+49/euYa3uPDlyquzT2ErYSXHKn0Pof516EbW5XrE1jJVFyz3/ADOpeLywHTmI9/7vtU8J5qlaXY2qch4mHPcEVoLGEAdDmM9D6exr43Pcl5L1qS0OWonHRlyL1q9BJtcH3qhFzVyLgHPWvgZXjK5w1Ubsd+WiKk4PbFPe4VzhRwB1rHiYkjFasKIsYP8AEa+hwuOr4iHI3sebUpRg7kDyHB96jDnNSuMSHbyP5U0RMQfl49a51TqOW9yk1YfuJHtT46rPKF+VTUkblmHPSu2i05WJa0LWwk1KFG3BHFIJAq88DvVZ7sFmCHK+tevGMIK5hqxZYh2ao1i25Pek3jOWOBVe61ezt1I372xwqf406eHlVl7sS4wlLRK5aZtoqrNfxW4JZwD6Vg3WuTTEiPCL6Csq4vI0Be5nC57ZyTXrUMok/equx30sBJ/EblzrfJEK49zWXPdTyq0ksnyDqzHAFc/P4kjjyLWAFsjEj84/CsK71K5vHLSyE57dhXrUqFCgrRWp6NLBRh0Ohudet4AQmZW/SsG6169nLKshRD/CvFZxOTTTWrqSZ3KMY7IVmZySzEk+tMoJ59qGdR0FSDYqozEADJNXoLGJFElzOq5GQqnk1m+a2cKSB7VPbgswJ5NVHVmUk2Xi8UELzO2yNR1P6V5TrOnXVvcvPKfMWViwkHQ5rrvG18be2trNGwzHzHx6DpWFp+rJJEbe6USRMMFTXo06VOceWW581nNebqqEdonL0ldBqWgNHEbuyPm2/UgdVrAII61w1qEqbszy4TUlofRWnaNdak2IkJ9SQa24/A94SzPPEsSjJY9fyrtxEiurQAxxouCqjANUL/UfInkaN3VgMFWHyt6V49TENantTzGtUfuuxXsPCumxSQXBcOQMY+8rGtyO1t7KJY0gxEDwFXI/KuatdUnubwCIkIP4VHANdjZSM0Q39cVjCoqjOGrUm37zuV/LmEx8pF2Hv0/SnRQFBhgOvNaWKQqPStvZmPMzNMnlsRtyo7AVDPqLICYbQsw6ljgVpmJc9KTyEKMuOtZOMujC6OZubvULlNsknlxnqqDH61HY/ubhW+8A1ac9s0TsMfLUcUCqTwDk5rypwk5pvdGykrG2kiuMqRU2Misy3I3cHoa0VYADJ5r1KFS61MWhSORinUhoz610kjJIkljKOoZWGCD3rznxL4Rjs1llt1P2dhwv9w+n0r0k4xUNzbRXUDQzLuRhgipqQU1Y6cLiZYefMtj5suomhlZD2qo7EV6n4p+H0qRSXWnuZlXkwkfP+HrXl9zBJBI0ciFWU4II5FcLpOL1PpqeNp1Y3iyKdnaAGs64VioIFbIj3RL3Heq0kO3IxxSTOWs25XF0u5MsJjJ+dKvlBJITkDf+hrn8vaXAkjPQ1swXEdxGCh4PbuDWdSHVFUpqS5XuXLeZoJc8jJwfY16V4W1BbqzMAPzYLICfzFeWlzgn867HwK4OrrGWI8xfkwe/cVMItuxGJilBszPFQ+y6w3y4VuQDWHJKCvpXY/FKzFrc2lwvKyAj8a8+84kYqvZuOjOSM1JXRr6LF9o1WEdlO4/hV7WWZZG2ttUgqSDVPwud+psueTGam1uMxzoh+8WzmuOUv3tjrpr3StZSYRYCm6QsNpp158l9cAk5U8Zplog/tJCCSFPb2ou2eS5kdvvM2TWjasVy6lScLLJGqrtyecHNWpYwiiQdCMc1XUZvI8jFT3oZHMZ44yKybHYW0VDK5dwmF3A+vtVSTYFLbvmB6Vfigf8Ast5yPvcA1myY28iqJI1x5p4OT+tXPNCKFCnOKqwNECMnHOKtBwdygHgdaCWSW7FXZ1+UDH5123g+O31C6knkYieGPajZxgnjp3rjbMRqoM+4rjACnBrtfCs8dhBEl1bgbi25413FeSct+FLR7mFa6Wg61ku7LS9asJ0aeEI+5TyAwHUenaqVxqMtnoui6zBEI5IZdkmej5HX8av6bqaS+Ctavt++UebGd+eQfun9az9ZkhPg3TrWQj54yQy9NyjIpSjfRmcHrsXvG1kxnttdgTdayxLG7J/yyc9M+xzjNcZ4xlaPV0k24YRoj+jEDg/lWvY6691o82kNKTDNEVKnt3H64rm5rtL3SC05LSKTknnJHHWqVnsbwjKKs/kS6ncxTaRbyw4DjIYCobLUhLpU1hOTsJ3oc/dNUXcGxRQ2e+KpBmSIkcGqjBXNrlaZsMy9s8Z7Ve8OXElvrdm6YP79AwJ6gnkVm3GS45/GtXwxZtca1aIY2dElVnCnBwD2rptFRuzCpKfwxPoOywtw6gHAYg1wnxmb/iT2x9ZAP513dhkqHd8N6muL+MFoZvCsdwG+aKYEj1U8fzIpU1+7XqZx0qnhquQRzV6CYgDn8qzs1NE+DxXW0ejRqNM7zw74pa122145aHord0/+tXeI8N3bsjqk0Eq8qeVYGvEkkPFdN4f8SSac6wzEyWxPI7r7iuzD4lx92Wx2tKorrc69rWTQ5shnl0tz948tAT/e9vety1uvKGOHiYduhFRWtxFd26yQukkTj0yGHoRWRfNLoVwJVR5dNlbG3vCx7Z9PT8q9JuMoWlrEn4/clv8A1+J1ycYeNtyfqPrWnboZmHIGR1rkdP1JZozPbShgOq9x9RXRadrsMZxcQhh3Ir5TG8M0qlX2tLbsefiaFSKfKrnQiySKNSpJPc9qtBU2gjGcVWg1bT5wFSVVGOjcVIpSRvkYEexzXFiMveF1hDRniS5/t3RRuZ3t78Kf9XKuR9R1/pU8Upc9xn0rO8RqYhaTgn5WKY9cjP8ASjTtRUx/vMgdiRXkqhXVbROx0KHNTUkXFwcrgEnvShljJHpVK61O0ilDRnDDrisu61xpC3lpgnqa9XCZTiKj2t5mkKFSpsjYnvlf5GYBQeTntVK51iGIbYRux3PSsRpJnG6QhExncxwMVm3erWluSEb7Q3qDha+kw+U0qfvVXdnZTwMb66mxcalcXLEljjHQcAVl3Go21uG82Qu46KnNc/datPPld21fReBVBmZupr0k4U1amj0oYeMFY1bnXJnysA8tfbrWTJK8hJZiSe9NNI31qXNvc3VlsNPSmHNOLYqNjxxU3EMJ600n0oIOaafQc/SmJiHmmn9acUOOeKcEA6fnVpEsYoJ7YrRtYwoBqmgywFSajdDT9JnuCcFUO36ngVrTVtRSainJ9Dz/AMUX/wBu1udg2UQ7F+g/+vWKrlSCKSVy8hYnJJ5NNBohV1PjK83Um5Pqbml61JaOBnKnqD0NWNR0iO/ia904DPWSEdvcVzoNX7DUprOUMjHiu+FWNSPJPY4ZQafNE+xhLC0TxvIQ2eA3GRXLa6s6z+Wik7uQQc8Vv6rcW0UG9sb+gANY8cE8wDs5w/BHoPSvjqvvaHdB21JvD9sVUO4+YjGK7C2GAKxrGIR4UDpW5AMKKqirCm7ssClptOrsRAxqQdaeRTcVm1ZjIp49yk1lyLtbpW1jPFUbqDOSK48VTaXOi4voU0l28ipkuDnJNVGBU03eRXEqjLsbMVwG4Jqfhqwo5iD1q5DdkEBjXXRxdtJEyh2NBzgUqnIFNV1dc0oGOnSu5STd0ZjjXKeI/Aul6+5nYNb3ByWkiHL8dxXVilNXZNDjOUHeLPnddLbE0bEqYmK8jGcHHSqAtSxYYPB5Ne+6p4dstTikWRArvn5wOc+tcTP4CubPTDNGTNOGO+JR2zwV9a4pUJJux68cbTlFX3PK7uy64Gc1j5ms5CyHBB5HrXrEvgLVZrRbhLcFmbAiJww9zXCaxpTwSkMhVhlWB6gihRa0kHtIyd4Mr296s8e8ZDDqK0NO1GSxvIponKtG4dSD0IrnrVzb3PPQ8EVrTpHJBmNcSEjGKhws7o7IVOeGp0/jrxZB4htLSGCJ1dDufPriuGYtGcHrXS6Jov2u5VJQSccmqHifSjpupmFclSMrRe71OJxjDYf4VkZdbjwcZBFa/iUf6Wg9s1neELG4l1USeRKURTzsOM1q+IEla7VmhcKB12muCrB+1OyjNciMi3dopmdTjA604uCFY53Z60qYW2c7clu9RykCCP65zUvsbaDHyLlCepNLcy77pm46YNLICy+YCDtxioEBlkc+1HKSXGupPIEY4Ur0xWdOMqckk+1WHcnCMclRiq8jAtjPNNbiaEXkZUc+4q6MeUpAwSOarQsoyD371aUAxbh056UyWh0bn5GwPxrpPDmqywXahH+dn/1fVZAeNpH0rk4iGAIx8uQR61OJJbPyriN2XndFIh5BH9aViJR5lY7rQra2ca/ojq8KyzuOcFMdVx6GqWs6Y8XhyJiRL9kkKuF7DHX+Vc5puuXFvfyzvN80reY7MOrV1fhvVTqPh2/imO+WJ9zL3cHv+VS073M3Fx1OAglaG9LKyldm5Tnr7Vm+c8aSQnhZCHxW/Jb2j3N0rRhFCOUK8fSsS8t2TyTvVgUwOORit42NNUQq3G2kkwbYDoQaiDFM56jinn7gPc9BTsUncqyLkZxXZ/DqW1t9ZcXMmxpovKiyOMn1NcksLTTrEnJJrvNG8NtHHE3IdmyMjk1NSo1ojSNNS3PRluTDGMk9Oc+tcn8Vr7b4XgiJwZ8AAH3/APrVs6kXgWKFMeZwD7+ua5n4uQN/wjGkXCjCpL5Z/FTirpXa5X5GDilJS73/ACPGypU0qmlDbic00nDH0rvRaLKPirCOfWqKt2qZWx3/ADp8rex106ljt/BmrSx6klkzExTZwuejAZBr0ZoBeWlzZOAY7lNhyOh7H8Dg/hXBeCvDt1HOuq3cZiRQfIRuGYn+LHYYr0Dzks7aa+lOIraMyt+AyB+JxXq0IONL3x1Z8yv/AFc8qjnntJyFcrIhKkg9xwf1rcsfE0sICzwpKvvkH865oytLIzv99yWb6k5NSqc4rzo4iUPhZ7bhGcfeR3sHiPT5h96WJj2Ybh+daMWrwnPl3qH0+bGa82Umn72Aqv7QltJJnDPCQ6HqD3kjWxkN2rKvUb84qs2oKpVTcr8wyMtXncN/c2zloZWQspU+hB6jFEupXM8QikcEDjO3BOPetYY2DXwmCwyTO5fWNPjYF7kvzyI+TVabxDbCL/RojvJxl+cCuLjySOavRjit44iUvI09jFF651Ce6YtI7N9TVYkk80L70hIHQ1V77lrTRC4701jimliDSE5oEITTc5FKSBSYZvYUANYeppnJHAqQrg4IzQR+VUkIiKevNGMVJUbGqSENNJ+NITTS1UJk8Ay9cz411QbY7KNuB8z49e1bd1fJZWrSEjOK821G6e6unkY5LHNKpU5Y2R52ZYhQpci3ZTJ5pc02gVhGR8wyTPPtS5poorojMlo+qXkbUb/JJKKcV0EYVEA9Kz7S3S1jUY59asGUltqk8184nbQ2Zr2mHfCgk1rxsqgBiAfSo9OtBbWy7h87DJNTSwLJ1HNdXJOEbrci6ZIGB708Gs+W3kh+aFzj+6adbXe47JOGFZwxVp8lRWY3HS6LxpKUHNJ3rreqIAdaR13AilHWlxQ4qSswMi6hIPSqLda3riLepNY1xHtY8V49al7OVjeMrogUkHFPV+ajo71nylF2G6aM4zxWjFcBwOawd3oasW0xDYqqdaUHYTimjeUg80pNQQyZHNTZBr16dRSiYNWExj73NBwvrilzk1WuSxGFJxVOVlcaV3Ye0o3YGK53xHo+h6hp15NdxwLIqEmbO0hgOMn1pNS1CbTssI9/fmua1CKHXIJ5CrRyPk5B744z61zSxC2sd9LBzkudPQ8cu4dshIrS05JHCnaSx+6BUdxFuuPLP97mt/ToRBB5owGIwp9KzlK6OyKcLnU+H7X7PGrsP3hGSpre+ypPMZRDFnvIwBP0Fc5p8czTqUZido+X16V04kjhICndkcD0NZOxjJtu5bgEZIjVCqr1x/OrJijLAFEZfTFQw7UiVUIORz6mrCqVUO2BjtRa+pm2YmqeDtN1NDIiG2kPOY8AE+4rzXXdKuNIma3nXgco4HDj2r2xcueenpWD4v0qPVNCuIwv72NTJGf9oc1nKn1NKVaSdmeNLMUDAYwRg5qWxGyIuRyxwOaohjtb1AqzbsTbL2rNqx3J3Fmb5yapSTRpIx9ulPuLgLnHWs4u0su1ec+lOEeoSlYuQzea+AcYrVgYtCR0I4qjDbGOI5OGx6VMlxsk2g8HqKUl2FHzHwlUcds9c1KVBjKDHU/jVe5QGMspHrUEM+9QRyynJBqbXKdgniljQtjoe3areiau1heFgSFcbZF9RVyyK3i7H2lcZx3PasjU7GS0fzFVlBzjIo0ejIZpTsHhJBLZyrY9j/hWZdFgydjFIV/DrT9PuAyESk7T129sio3+ZJFz1GfxFF7Mq1ysAZJZM9SSTUDNg49BxU8BZmkOcEjNRWFm99fR26EBpG5J4ArS6SuwRueFNON1eiQjOT3r1az+z25818AIp5I9K5vRdKXSIRNJgMPTvUOsa3EQY4m2u3BwODWOvNdGzXMuVbGut0L/AFUMeQvAPfrxWJ8YtUkhtLTSSodJArq3oV5/rUVndGG388sQVXjvn0rktduNZ8b6nEYLSS4+zJ5ImAwmM92PFdeFjKo+VIyrUrSTRyCcdaclvJNOI4I3lkY8IilifwFegaV8NlUrJq12XP8Azwt+B+LH+grt9O0qz06IxWFpHCoHOwcn6nqa9qng3vIqNN21PNdI+HupXe2S+dbKI/wn5pMfToPxrutI8KaTpGGgtvOn/wCe03zN+HYfhWjc3ttbcM4kf+6nP69KzZtYnfKxEQqe68sfxrqSpU9johRb2N52jhG6aQIR/D1Y/hWF431IHQPJhGyKSVVA7t3OfyqO1LSEbixJOSxPWsfxpP8A8elsD03Of5Cs69S9Jtm1KivaJvocwlTr2qCOrC14jZ7DJVpxHGKRaUnisHuc0iM9aRc7qCeacoropIhk8S8iri4AqtEOlWRgDn0r06S0JYF8cCmHmlyT0GaXZ6mtkKwhIxx1pApPU4p4GBxQKqwrAEA7UEUuaaTQSxD9aYTzTiajY1QriMaiJ4pWOKiZsCmiWwJAqvPcrEhOajubkIp5rnNR1AvlQfyqJ1LHPVqqKIdb1NpyVDcCucY5NWLmTc1Va5m3J3PmsXVc5hS9qSjtTRyig4pwptGa0UhH2PcqAoI7UukQC41FFPKr8x/CppYTznOKseHkxLcSEc4CivGpRvUVy29Do80ZqPNANdzZmSHB4NVri1VwWUYYVZBpaipTjUjZoadjOtLlhL5Uh9hmtKqM1oxl8yPrVlZMABgQe9YYdyppwn8ipWeqJKcKaMGnV2ogQjNZ97BxuA4rRprqHUg9DWdakqkbDTsc04wSDUZNXb2AxufTtVE8V40k4uzN1qBanxnDAioSadGckCs27so6G2IeMHvVkKMVRgJiRfSrqOHr18O1az3OeQoUA8GophjHyM30qQjaSaUHIro0egk7HO3Ect07Kse8YJIPpWHeQmwh3iAiFvbGK7sQIJN44OKjmSMxMkqKyEdGGQawlh09bnZSxsoO1tDxPVtKtFkFwlrJAZQWVjwG+lGlaedRQwRFS6DBBPT3r0XxFaRTaYII1TZH91cdPpXmctje2Eq39uXiKtgMeP8AIrlnHldj1IVViKeiszt9ESOOJlf53hbBJ6girEaiW4VZF8vzGJyeM8154PE2o2M7TSRou9vmIHysavr4uubiQSS+XkdG5zg/Ss7Pch4eadj0byxA20cc8E9hU2AEz5gJHf1rzYeKrlXIEccn91mY5FaGn+KD5ii4igwfvBWOc9sVdzGVGS3O9ttUjtmzMuQflG0Z57VmeLNctdO0qa5d03upVEU5ySOK5q/8QLFC87RzSRKP4UOBXn+u65ca1cB5PliT7kYPApubceXoRGh7/MZfmFt4I5PerMRzb4B5HaqTkBg3NWYZANwH4VjI7olSaFnJyTUtvEkSg4yfXrW5odpb3VzJLcAMkS5CHoSema6SGyjmaaOSBYZUQEbeBz2Ip62IbVzhLm6CIc8fUVl2twZbwknjFepS6Lp99YhJrdCSMkdx7g/0ritW8KS6PIbi3LSWx68cp/iKI8tiW5XKjSFwRnp0rKM7Q34OflY4NXVZvUcnvWdfL+9z3pwRUn1N+0naJlIJypyMV0Erx3iG3lXJAzk9uOlcpYyl4Qzda04J2ij3o/zFSCD+VYTjqbLVFEw+RMwQnHoKaTicN/AeKu/M+XZME8HPrVS5QqRjvyCKS3HaxAB5JlPGdvFaPhyz8y+imXOR83X0qisFxfKqW0TzyE42IMn8a73QdEksEDzqsbj7qqQ2PrXZTwtSr8KLiupo65cRQxcXAXAP0LdgK5GDRdU1SXzWjWGL7oklG3I9cdTXa/ZrcSK4hVpF6MRk/Wlnnjtv9dIEPXb1J/CvTo5bCKvUZcab2KVto1tDbLDMftAHUOML+VaJZY4xkqkYHGflA+lZM2sNnFsm3B++3J/Ks+SWWd90jsx9Sa61KnSVqaOlUW9WbFxqsEfEQMrep4Ws2bUbmcFTIQp/hXgVXAJp6xkjkVEqre5sqcYkYBPWnBcGpNhPb9aURnPasvaQHdFm1+UA1x3ii483W9ufuRgfnzXZxoVjJHpXn2qv5+s3Tjkb8A/TipxNVezsghJKVxsY4qwgqvEMVZWvLb0O1Suh4pWI9aB0pGPFQZsb+NOTGaaOaniFdVKJLJ4lPpU+315psYwKfXpwWhDEpKdimk4NaWAM0lITQTz70yGKTzTSaQnrSGmQ2IfrTGPelZsZqtJIAKZLYSSAd6o3N0EXrTbm5CAnNYN7e5JAPNZzmc1WqkhL6+LZANY00hPWnSSEk5qpK/51yyk2zycRWIZGLGmUuCaMAGrSPIbu7iUUUlBItFJRTEfb8oXac9Km06EQwlh/y0O7p2qtb7rqYhceXHy2e/tWpnJrigvtDbJRS0wU4VoIlFKKRRxRk5xVAOoKgjkUUoqt1qA1OMin03vTqcVZWAKKKKoCtdxCSI+orBmTaxFdKwyMViXce2UivLx0Le8jWm+hmsMUsBAkX0qRkpET5xXlpvmNjdQI0YwO1LFlJCfWqcc5iwD0qdrlCBgjNevCaevUwaZbeQFcCmCTFVJJ1C53CnREOobPXpWvO2xWLyvkdOKfwR7VFGNo5NSV0RvbUkilijbGYwfwrM1DSYLmFlKqueSGUEH61ql+cDmgY/iFS0paFwm4O6PJfEegJa6BdRQwmSXZkbVyTzXmyRFBtVyCOvPSvozW7IzQq0O1HU85HBFcLq+iW97ZTS3MEYuI0LLKoweOx9a4pR5HY92jXVaKk9zidG8OPqoLNdMEHUbsV2Gl+GNM06eOV7cSMOpc8j3qfQIVGmBYrZAMcg4+atWIRq+JIIwCOQr1i5tvQzqLVmxEkSrt2r5bLgjHBrznx54Tt7S1OradGIlB/fxKPlwf4gO1dlc3nlrtDZUDPJ6DsKwvFd+E8J3KvICHG1QTzzVW0MYpp3PI2cEEdDUkRJxg896pSNg9aktJC0uPaq5DojK51vh0EWmqHPCRKQOpzmt+yldYtO1RhxI7Q3Az94Z4auW8PurahPaScfaYWQN6NjI/lWpobTS6VqNkWDxxgyxDrhx1H4jNZsfLY7HT0RL+5s3B3RkyxnPBH19KkW3jkZ7Z1OyVS0e4dD3WqdldK1/otzwY7qA27nPU44/rWw65t94/10E3lsR3Hb9KlIhuzPM/E3hp9NY3dun7gn94g/5Zn1HtXI3ib047V7vqVjFe23zL99SkiHuR/n9a8e13TDpmpNbJl43AaI45IPb69quCd7DTujNsD+4xVzzGUbQ3BHNaWleEdSnVXlRbWFu8p+b/AL56/niussfDem2RVmj+0Sj+ObkD6L0rrhgKlR3eiOqnGTRzdnZXmo2yrbQEjAzI/wAqn8a2rbwrb7VN/KZ2Bzsj+Vfz6n9K6RE35y6IFGfnYLx+NZlxqkaErAu8/wB9uB+Ar0KeDoUdXqzeFLmdi5bW0NrD5VtAkUSjkIuAPrUM2o28PRjI3ovT86yJruaf/WSEj06D8qgJ9a2lXtpE6o0F1Lk+qXEoKqfLX0Tg/nVLJJySST3pFO5wqgu56KBkn8K6LTvCN9dBZLlxbRnnaRl/y7fia87E46lRXNUlYc6tKivedjAAz7Vq6foN9fEFItkf9+XKj8O5rtrDQ7HTkHlxBpB1kcbmP+H4VoPIkQJZgPrXzGL4lS0oR+b/AMjzauZSelNHNweDoEAM9w7nuEUKPzrSh0HTLc8W4cg9ZDuNJc6zFECF+Y1jXPiKUZ2kL9K8WpmeNxGl3b7jn/2iruzpkt7aL/VwRJ/uoB/SlZYieY4z9VFcPJ4guCf9Y3501dcuD/y1b8655RxUteYpYOe7Z2rWto4w1tCR/wBcx/hWJdeBfDl1k/2ckDH+K3Zoz+nH6Vnxa7Pn7+frWhBrzH74BrP2mNp/DN/eDw9WOzMG9+GKjLadqBHpHcJn/wAeX/CuV1Pw7qekHN5ausXaZPmjP/Ah0/HFet2+qwy4BbH1q8rpIpBwVYYIPIIrroZ5iKTtWV19xUMZXpaS1R4IQaYxr1HxB4Etr3/SNK8q2m53QniN/p/dP6V53qWlXulXHkXtu8MmMjPIYeqkcEV9Ng8fQxK9yWvbqelQx0KumzKQ61ZhGSKrqOatQV7dFHXcubMLn86Q0hY4xTetegiUOJ4phNJnNIfWrFcM0hIJoLDpimZpolscTzTS2P8A9VNLVBJMAOtMhySFklxms25ugAcU27vFRTk4rDnuJbhisKs5z2FQ7s4a+JjBbhe3nXn8qyJZtzE5rp9E8F6lrt0qkiNGONxrp28CWemEo6pJKODuPSuKtUUXY82VV1Njyt0k2hirBT0OOtQvG4GSjfUivcNO05ZIvLSxjMUZA3sBz9KfPptpIl3DPHEsgBCrs4FYqtboc86XN1PFrHRr/U3K2lrJJgZJA4H41Yl8K6zEuX0+b6AZr1Gz1630q+trS1sw3lt+/UjBINb3iGae11iwu7DBtJ0O4bciNvepeIlfYX1eJ4BLp93AxWW2mQjrlDVcxsvVSPqK+h4tVkbU5BcaWJYlQAyrHkMe9DxaDq8E1u+nQ+YrHYrIAX9q0jXb6GU8Ol1Pnag163rPw006Rmn0mSXEqlxH/wA8z3FcJqXhHUbIsY4zOg6lByPwrdSRg4NI+u9Khe205N+fMl+ds+9aSdKqrIu5UzjaAAKtqPlzmue3YzH7sUoeq7sQeKqyXe1wi8saly5R2NZGz0qSoLZSsWW6mkaVmkCKce4rW9lqIs1H5mZvLA6DJPpVK7luQm2Hr60tgkqDLryepJ5NYPEfvFCzL5dLmgacKaGpQc11pp7EC0UZoqgErP1KHdFvX7wrQqOdA8TA+lYYinz02hxdmc4kgYYPWlOByKiuIjHOcUAnbzXz9mdJPNKGjHrVCSd84BqZ2yDUDxlD8ykEjIz3FapyewKyLuno00yqxyO5rWWRIbjyvvHHYZx9azNPJVSGWQD/AGV5/CtIW8Ubl4hy/LvISWx+NenRi+Qxk1csRzCXoCMVMG4PFUwyRh33ZB6+xpYblWBHPrXRF9yGWAwB6U/buHXGPSotwcAHjvTvMCjbg5q0BS1NZBauFGQeprj7maO7heBm8p3G08V3jFW+VsEN2PSsi98O2d1llXypOoKtwT9K5qtOUneJ34TEwp6T+85MxQaRbLHGwJx96Q45+lZEmv2X2rZJKiS4x8gyPzq38QHfSdLt94ZHMww56Hg5/SvLb/UkmUFeWJ/h6Guf2TuejzQceZvc7rUg0sAuBfOVHHyuNoGeue9ctqsd/rZWJZwttF7dR61mWkt1cfufMbYxACZ4Jr0GytLf+zFeCEbNuVGcsT70+XlMpe9sed3GnxxQnYGbHBLVn2yvHeAMOCpxivS7rR1urX/V/O3UYxXnuqRmw1AccxtyPampX0BRs7mjp90bPUYLjGQrAEeorX0u/Gna5cYP7hmbr0APr+dc7kfw8q3KmtCysr7UZy1vA0nTLnhMe7HikqUpuyR2ctzvYtPnk082MTbpLS73wMTyMjK/h2rRsb7zdJdnVhI0+1lK8jHHI9Kz9Jt5dOjcPPuZnVlC/wAGO2e4q2inafKQBckkjpk9a7KOXSes3YhUG9zUvL5Wdwj7kLBs4wM4wayTHH53mLEvmYIDAcgHnGagl1CK3cEP5rjqq9B+NUbjV7mXKxhIVPaMc/nXdTp0qO2500sNy7I1JpY7YZmlCnso5Y/hWZNqz9IF8vH8XVjWexLHJ5PfNMLAd6J129jsjSS3HvI0jl3JZj3bmmluOtRqzzSiGFGklPREG5vyrptI8HS3KiXUneFT0gQjd+J7fQV5mKzCjh1ecv8AMVWvToq7ZzsSSXMwhgjeWVuiIMmuk07wXcTMsmoTeUvXyouWP1boPwzXYWWmWmnQ+VaQJEnfb1b6nqasyTRwRlpG24OK+Zxee1Kl40dF3PJrZjUnpT0/MqWOk2WnIFtraOLH8QHzH6t1P51akkjhXLsF9qzptTZjiEbR6nrVBmdzl2yfevnKtedR3k7s5VTlJ3ky/PqfJES/iazLm4d8l2JNK7BRWXeXICnmop0+ZnTTglsVby5xnmsSa5LMeaLy63Meaz9+417FGjyo9CnGxZEhJ61KjVWTtVmMZrSSsbFlGPHNWo3I71VRasqK5pWEy3HMynrWnaajJERhsj0NY6g1MpINc04RlozKcFLc7G0vkuAMHDelP1DTLLWbM217CJIycg5wyH1U9jXLQztGwIJBHcV0Vhfi4Xax/eD9a4nGeHmqlJ7Hm16Dh70TyzxD4cudAvRFKwkgl3GGUfxgeo7EZGRWfGMV7be2dpq1lJaXsYkhf3wVPZgexHr/AErynW9ButCu/Inw6MN0cqjCyD+hHcdq++yTM4YuFnpJbr9TrweM5v3dTf8AMzM0hbmkZiKbmvpkelcdSMfypVYqeBmonfbyatEOQpOKjaTHeq8t0q555qFVubr/AFaFV/vNxWkYt6I5a+Lp0lebHz3YQdf1rPaae6bZChPv0Aq35VhA5E0huJv7q9BU01xFBZnCbZWHygcAUqrhSXvvXsePPH1K91SWndkNr4ca4cPcSFlB+YL2q7M9hojMkaqTjAJXNTaRqSlAsrqo64x1q9eaUmoQyTLqETNuJwyYxntXl1MU5O2xPsray1ZV8JawsviKASTKkBOCM4AzW5rEtvBf3W7e1t5/zTqc7ay/Cei6ezyNIiSXEbfMo7D1rR8XaYl9okq6TCsLhV3BHyJMHnI9a54JOTuFRtWsVLi8maSdLdySCJQFPUAcYq1pWrX6NbyanZlUuAXWV17Drn0rldMVhYxWn2rbcnAZscpz0rtPtX9oaaLWRoZxtMZJbbyOM/WlPswS0LevR6Ve6fKZBF9veMeTKgAYA9Dn6VyE2i+I428mG5MtuemDyM1ptpv9nymJ4mVyA3mMc/TH4VHJrN5aXEb3LHyycRuDwSBxT5bolO3U1PC0q3FuYrnUArqcZVsMpHXNa09raXV3JPFOrRoACVHKnHJrPsPDGl6uj6ltMNw4LSBHIAJGc/jzVS28MX1j4ge5N8q6dIu1IweXIHf270RpvdEOa6ltbW9tJVt0uXeMPuQheSD1FQx3BhaeSW2UKCVEoHLEnGK6LULl7gW1tZxr5tttlUb8E55xn6CuSbXftFxeRRW+6C6lDbW6JjkkfjXU6dt2YxnfoexXUQby5IdysOGBX0p8eo2zO8RnQSIPmTPI9zU1jqlnNb7PtELTopLR5Abj2rzHVLP7JdF3ciZmLpIHJIOcjnvXLiKyotNa3MKcOfRnpNzckRHy8E+5xUWn2ZluPMc5Arh7zxPcTyQokbLGEUTSjl2P8RUdK9G0/wAtY1aIkxsgKk9xjirpyjVd10FKDgtSzNIIoz2xUNll1eVupOBS3GJTsPQ9akiCooVeB6Vf2iOhNimO21eKlFRTL8hpVF7t0NFCW9eOTGau28vyfMCCTmsKV1N6qucKWAJFdCNmMDFcODcpSlK5pNWsSg0tMBxjmguFGSa9TmSV2ZDqa3INVpL5EPFTq4fGGB4qVUjLRMdmjMksJLmckYVP7xq1/ZsH2fytvPXeeuaudBULTZbYhy3f2rNUqcFtuO7ZnSaShmTYxVB97PJJq1NAjyiV1XIG0EjpUU1zsfBfOO4PFNjuGnc5OFAzkVEHBaJA77irMP8AUltwxwTT2TcgOMHv24qGZGm+eGLOecnjHvUkTFEUGYMTyVxx/jWy8xEpjQFS2OCCMcAYpRtPzAgHoT7VGwLjAO3dySF6VXaeO2Q75AwBHUcn8KbaQJXLS7S+Tlh24wKFf94F/hB5qmJzK+0KVULwSP6Ush3nLNwo6Z4NZOXYpI0DJGSRxgDjFDZwDx+dVLdQhBYjJ7AcCrJ+bacHj9ardahYkZdy/MoI7AiuG134b6FqjSta2T2d5MzP58L4RWOTl0JwRk9AM/Su2aQK2XIUdjmmiaAsFVskntVc6WjY1zLVHhd/4NvvD9/KJUaWzjdYluMBQ5ZcjC5z6/lWjpltFBMm3cFAO4bqTWfFeoeItS+yybUtFnLRwRpkjaSAS3Unv6c1ftIJUk3yhVTH3e/rXO6Uqr9xaHuUadTkXPua7FY7NnOAFXDd85rzi98Lajq+oSzKiwQu+VkmOOPZepr0CS4Zk2gYUdBVWWZYhmZwuex6n8K7KOBjH3pm8MM3uYem+EtP0+NfO3XsinOZRhR9F6fnmt0lY0BJWOMDA7AfSs+bUzjEK4/2m/wqhJLJK252LH1PNdnPCGkUd8KFkakupRR/6pd7ercCqE97cXAIdztJzsHC/lVc5zmmlwM81hOq3ubqnFDyc800sFFSWtne6g+2ztpJRnG5V+UfVun611eneCIlKyahN5zdTFH8qfiep/SvMxWZ0MOvelr26mFbFUqW71OTtba71CbyrK3kmYddo4H1PQV0+m+B2LCTU5g3/TCInH4t/QfnXXw28NrCIoY0iReiIoAH4U2a8ig4LDd6CvlcZn9Wo+Wn7q/E8urj6tTSGiEtLC2soxHb28USgYwigfr1NSzTw2w/eOAfQcmsqbUZpciP5R6iqZBY5Y5NeHKs5u8tWc6otu8mXp9UkfKxLsHr3qkSzHLMST60mQKYX9KybbNoxUdiTIFRvIAOtRtJiqdxPgGqjC7KSFuLnANc7qF7nIBqS/vtoPNc7PcGRjzXq4XDdWdVKA95SzdaFPNVd1So1ei42R1ovRmrUZqjGelNu9Ys9NMazyEySHCxxjc31x2HvWXs5TfLFXYpzjCPNJ2Rtx1aQCufTXrdZrlXeKNbeRkBcsROVOCqsAQrZx14xU1hq085VXNqSrKs7LuCR+YSIh0LHJxyM9Rx3qv7NxEuljzZ5thou17/ACOhQZ7GpQorntATU9XciVvMFxGSWTG6xuUYhUI/hXIXrwc9SCa1xqlomptp880UV4znZASQcHOAM9+CPwrLFZbUoQUr3fVLoTh8yp16jglYtjirEMzIwIJBHQioCKAcGvJaueg1dHUWF+JxtYgSAcj1+lWb+wtdXsGtbpd0bcqw+8jdmHvXKxSFGBU4I6GugsL4TLhuHHUevvXMnUw1RVaLs0ebXocr5onmWt6Nc6NftbXAyPvRyKMLIvqP6jtWfBAZXx0AGSa9k1HT7XWtOa0uQMHlJAMtG3Zh/nkV5Nq2haho93LDeukMf8MoPyyL6r/niv0jI82hmEOR6TW6/UazFU6f7zdFSe5traMgfM5rOMVzdHcQIo/7zVVu9Z0/T8+WPOk/vNXMal4jvL0kGQquMbV4r6dU4Q1k7nl1s1q1NKasdVNqWkaSCZCLm49D0FYT6xf65ciCFhb24PJXjArm4kkurhYxlmY/Wu50/TkisWi27CmCD33VyV8Y4+7T0M6GGdV+0qu5Np2mWREcTM5YyDJH9a0PGVnGYklgKrsIRFH8VRaZZXVw8nl7VUcFyemawNRupzfeU8xaNM7Sa8ptt80mehZKyjoTW9reSbFVCCTtEh7VvwiWxvobcN5rNGWc9celN8O3FlIFt5CTIvIbt75qvfNPYXL3KyK8buF8xT0B6A1noU9zpNLtCJvKthEhkOWO7kj3NdVZ2lnpKyPJIJJWQ/ITxk98VxGmYvgSrFPLUDKtglvU0slxf2lwSlyZwenGRmoTdyJpMZdabFJe3EyoUVmLKAOT9KtWOm3ETwSIHznlQvHXuajk1e8kSRLpfIkxhX2966LwVa3uoM4nbdaddx4PFUm5MiTUVc1QEniR7pFjYDaMHJH4d65vVNLtborA9uJCjkovQAkYOPfpWp4jb7Pdx2tq4Y7/AOE8jmtW9nt4pIm+SR1wcAdCBWy0djB66nK6dcXWk7rAfOPLCgMeVweD+FMvJryVjBuwvL784AbGMc96mns3F208izKjk7dvfPJyaW6eGOzNyke+ILlnc59uB3PNXGWonFHO63ctBdPBbNIgbax3HG4gcEH09vasm21dm2GP5p3/AHanIAOeuRW94i0y7urJbkzA2MUKAOSMAYySO/U1xukWi3GpRxuoWQN0zx+P4VdRO1xQaR9OQ6Z/Z95cXCXQdZUAwYwDuzxiuW1ZI3ZwyZweVz0rqjLb3T7zkZ+XIxx+FYOraTFF5jxyTSSydMnhfXPt9elclenzx0MKbs9TkLiTLcfIAPlAFekeGNZj1PREYcTRYikXGMEDqPYiuCvNIlik3NMqqR91Tu/Wut8EvYRab9hjbbdeYzOHABcnoQR2wO9ZYROMnFmta0o3R0AaQtuboatxvVeSVVYoOSPSliYnmunZnMaCtRKpZCB1qBWIqxG26tNJaMDmL62mSclgefStuygZII/NbLY5FW5reOYYdc1Su444Iy25s44yxrjhhlQlKW9y3PmViSdpQ5KKWjC5GPWsuXUWClXJDDtWl5udPV1IyR1Jxj1rJtpUumYMqtgEgMPvfjUV6LlJWe5UWuokczy7uflbrWpZ4jOC3IHSo4EtkUqqLgkArzVg28DbTtwUB2kHFXSw8oWdxSkmXQw9arMI13ALw3UetOB3JyMD0qCZE2FscZ5weldkk2jNCCJJZiVjAwMZI4x9KheN9xhRcs3TPSpy4AG4YHfB6VExMYkKiUueAyjPFCgtwuIzSWsQySzDvkY5qvCGn3NK4Y+hOAPrVKe+27nwzxg4JlBP60+3uftSxmEDBPzFV/XPrU3TZXK7E97MwjXEuwkbSPQf1rOieFXyQ7sDy55P4elTXLGWc5lO1TuYMmDj61k3F69tg7P3Y6AjI96xqSs7s1hG5ck1NYndYiy7yCfeg6iwh3DJXPyk9OOtc5cXpZixl3H/AGuTVN9SCrwxIHbPSuJ1p30OuOHudb/au0K6ucjvmpItaYYw+COpBriUvp7j5YEZ2zzjoPx6VpRQSkfvZNvqq961p0MTV1Wh0wwie50zXylxJK5OfxJqCTUpBE+xzG7AgMuMrnuO2aylZY1wg/WqU+pwx8Z8wjsvT869GjgY0/eqO50wwcX0HwWtrZp5dpAsfqw5Zvq3U0TTwQZ86TB/uLy3/wBb8ayp9TmlBCtsQ9l/xqmSa7XUSVoo9CNDTUvz6q7KVhRYgerdWP49vwrOZi5ySST3PWkPvTGcAVjKp3OiMIx2H8d6azgcnpVvTNJvdXf9wuyEfemYfKPp6n2Fdnp/hTT7Iq8itczA53yjgfReg/WvJxub0MPo3d9kc1fG06Wm7OMsNJ1HVcNbQERZx5r/ACoPx7/hXV6d4Nsbba94xupRzg/Kg/4D3/GumJJ5Y9BVaa/gi4B3t6CvlsZnlatdQ91fieXUxlarotF5FhUVECAAKvRQMAfQVFNexQjk5PoKypr6WckfdB7Cq+O5rxJTlJ3bMo0f5i3cahLOTs+RfSquM8k5+tIWxUZk96nV6mqilsSlgKY0lRljTC1NRGPLk0xnAqN5cCqc1zgHmtIwbGSzXAANY17ehVPNMu7wKDzXPXt6XJ5r08PhrvU1hES7uzI/WqZbnk1Ez5OaTd/kV60aaSsdMdCYNUyN71UDUybULe1DCSVQ4H3B1PpVezctEjSU4wV5OxLqWrnTvKESLJMzBtrDK7R6jvk1zxkvob1LyVXMkmHEr+h6fh1qtdXUt40k0jDBIGQMH6A1JpVu0s0KxwyNM0wVXEu1Tnop49e9eth6EaUPM+Xx+KdWTd9DobMaZfXRtb65+wGRHNxPEvmkscE5BIx0z3IOR3pnh+za4ZFs5RMPtEcc6mIlyh53bAcsuQcAHPAzjNP1HSgJXukiEYCRxskjrI5bdt6jvgAHjt3zk2r/AMH3VjbT31rdlViuhHFFysqAjKs3bOOOP61dScUry2PGuraM6PQLpri4u5tPgyGvJJxJt2nZt6bj8wbO7IJx0681ma9p9zqXxBiubHLqBBcgtwAhOST9DmpLW8uFujFowhtdRWBZJmwT57Mw3LzxlF56ZOa1dFS4vtQlvktHjtxbKivKdz/M27aoUYwWJ5x0A4ya4sRGqrzjZ3VkdeCqU4TXO7I3Tgnj1o2jFIQQSCMHoQaUV8cz7O91dCD5asRSMrBlOCOhqA80gODUtXE1dHS2N+JRgkBx1Hr7ijX9Fs/E2jPp95lQfmimT70T9mH9R3FYUUhUhlOCOhrasb8NhW4b0rCMquGqKtRdmjzsRhk15Hzj4s8M6h4Z1V7K+TJ+9HKudkqf3lP8x1BrmzG7HpxX1lr+g6d4p0lrG/jyPvRSr9+J+zL/AIdD0NfPXi3wjqPhbUfs14itFJkwTxj5JVHceh9QeR7jmv0HJ89hjYctR2mt1+qPInRcHoYmjwCO4MhONo6109k7XMbCIsTnBbPaszT4I4IirjnaWJq5Zz+TGNuAGbkD0rtqS5pXR6NKPLGxrWc80cstqhDr1zmqNxcwwTwrNbocMCTjNIkrLPI0UuGJx8y8HNWbqNWhCSgBThmZOxFZ3KJL22kkkF7pigpKxDoCAQfYUJZXtzbyWKWjebNjapHf3rPZntbgPAQ2wg5D5xXWeHdcMN1m4dmJHyseSDUOpysHDmRycU9/4X1M6bfbVlBUyEHKkHp9a6hNTixb3Z2mFfk3n7rN/StTxJoOn+Kmhu53e2niXaJU/iXPQiuRS1ufCmvvp1wxltZxm1mb7knv6AjNVeMtjJNrRm1qGo6RrF/arHMqcbpueePSuim15NI0qKKxcgOpXaOuT1P1rnRpFrqNvIu2C2uwRKk23r7HHY8VVtNF8RxxGKEW9wzPkIsg3OM9ASKa0ehEl0ZqadJNFqMc1yX2zE4Zuck9B+ddQ0tnHPFJMCX3MAFGAcdceteb3F/NK0UMMksNwkrCa3lXaYyDgL9a2tOluNM1eO2vbeea1VhKrhtyhs8nPpVxi92RNp7HTXs08cElyYRHaRLvWQE7ixOOn9KxtVv7FfCptbRHd5Jcq7D7w4IIHXrkVWm1e91O7niTzri3kOSCf9WozgAHpyM1uWsdvLpdumoSJHj54ZUXLLyODW90nZGPTU52HXLW10mSzudN8ybyAsfmt8qn+vHasaw0GBb2S7juZwNiFtmPlZh057Yrr9Y06DU4/KjECXCMGgk3fI/P3WH071zuu6e2keU6jZ+7zIYjlSV4HTtj1onJyVhxSTues3FpeW9+rK11JErj5Y1wv0I7mtQIs9tIHf7PJ0+YEBs9Mj6+1RWXiC2njb7rEtyScn2rSAtb1fLikKZO4ruxuP1rGDhb3WZzhNfEjBu9Ha4Vf3hhb+JgDIp/EDp6msufRbfT7mKdnZ5FbloyMj3AzWzNts79Yna5hTPzIRuV8eh7VYu5V8z9zFlTGUUld2G7AjGfpUOEZarcSckZFjrv2bUf9Illlt5PlZ3OSuOjEV2drJHNCksTB43GVZTwRXAXKRs4SNEDqAAUQAnHc88mn6bd3VoVjgnkGWyUQEKW78D6VlTrOD5XqXKlzK6PSAgIqSMYFQwPL9njaQIzFRu2HI/CrArvS1OYrXU8sIygj68bmxmqkkryqxmC7cHG1gaWeS7WRFMKOjHOSc7cUTrvXOVUKM8d6yk27lFS4k+zWAmGCIwSy55OfQ/0qlp95FdQGWKXHlth1PGe9TXC7ECM5w/TA7+g9apCHZGPLVAW5ZcbfzrGTfNctJWNG6Z0XchVh1wBgjPv3qa2nJhUvJnb97A4/OoYYoxgyRoGP8JXJx+dW2k+XzGwWYY6c4q1vcT7DvtUaIcuMHtn/OKoXGrRRMxycDop71z+v30Wk3AWRiqyJvQHnvggVyN34h83O0nHbNYyxEr2SNqdDm1O6uPE8EZdzEBxjO6sO+8WtMDtJHGBXFTakZOrH25rOm1LGQDk+tZupUl1OmOHhHVneabO167T3EpMCNjyt3Dnqc+1Lf6ybXMUAKqOFGMADt/+uuJ0fUZoLmSbedrrtYDv3rTvL9Jk3L16kkYpe98KLVFN3NP/AISi7i3KDkNzjOB9TUMniCSZMSPg9z2FZlnpV5esHC+VCf8AlpIMfkOprobXRbK2AynnSdd8vJH0HQV008HUn8Wx0xoLsZsS3d+d0UZ2Z++/C/8A1604dJiXDXDmZvToo/DvWgRsjLOyRqox85xn2FU5tShT/UKXPq44/Ku6nhqNLXqddOl2LaKsafKEjjHfgKKqzalFHxHmVvU8LWbNcSztukYn29PwqHk1rKt0R1xorqTz3cs5wzcf3R0qsQcUpIFOt4Li8l8q2hklk/uoM4+vp+NYSqpatm94xV3oiMjFMLEsFUFmPQAcmulsPCFxKA99L5QP/LKLlvxPQfhmuostKs9OXFtbpGSMFhyx+pPNeLi86o0bqOrOKrmEI6Q1ON0/wpfXirJcOLWI9mGZCP8Ad7fifwrqrPw3pdmo2WqyP3ef94x/PgfgK1HZY1yxAHvVGbVFUYiXcfU9K+ZxWc1691ey8jzalatWe+hewqjPAA/ACqs+pRR8J87fpWZLcTTnLsfpUWAK8hyb3FGkupPNdzz/AHmIX+6KhwBz3pC+KjaTsKEjVK2xKzAVGZKj3ZpMjk0WGOLZ603cKaWHeo2lAFUoiuPZxioJJgveoJbjA61nz3WO9b06TYIsz3XBway7q8xkZqvc3Y55rHurzrzXpUMNcpD7y8ySM1lPLuPWopZyxPNRb8969WnR5UbRZY3Ubhmq5lVPvMAPc4pEuoWj3rKhXOM5rX2b6Gqmlux19dG2snlXrkAHHTNcy/7wZb7wPJ9a1dRuzOFigfKEfOQcZ9qx3Vo3ZGbDZwefzr0MPBxhqeLj6qnU0exdsY/McLPGzWm9UlYZwmejHHpz+veuwuPC8+n2jatDd28cEbIEaKMoH4BQrgknJDAnsQD344O2nlhuQ8TMHU5Uj/PvXTtr9xL4VaymtzI5vG/eeYB5QwvCqpyvORyNpBOOQa6Gm1ZHj1lK5qXviKDW5pZtTtvIeVoBsZ3xsBILerAAf/X7Vo3BKWFheWskpjnjcNaklihQ53ZOM8EjP4ZNc1PdXN3bx6hulE1o483ecKvmNuQow9wxwemCR3rYTUrjUdOinS4a41Np2jPlAtJIm0enHTIwByOTjjPLWpJptnJy2dxIb9LG6mDyfaJLlXV1iBIVzkA8jkjP6kZrqLrxpN4YgitILOGaWRd7PK5EQwSDs2n5gCGHXAx61w8UE9jrLCR5raZNwZGJR8EdCOvINSaPHFq+uGG7cRrFGxR33bVI6D/DPBNYwpxk13X3D2dz0GDV7SazF1NPDERGjTJuP7t2/gwfmzyDz1BqzcWF7p6RzyTB43Pmzkhiy8HCqOgXA4A54q9Ha3ETJFZxyborZFcwy7RIwz827+L6H6UXaz2UT2hjQJlZSfViOP0NefWw1OEZSpx30PUpY6pNxjOVkhptn+wW15keXPGrgc5XOeDx14IqAqe1bslx53huKCVVF1M+xAmQMjqw9vrxk1jSw28erSrdM0r58uAAERdMbiR6dfc9646uXxlNOm7Ky+TOmGc+zilNXYxW2mp1cnBBwR0NNkgVDN+/ibZtCgNyxPXj29RkUW6q8n7xisYBLEDJwOw968yth5U5KMup69LFU61L2i2JLjxLNYLHHHbJLISNxkk2qB+ArYubbTPFmhtb3SLNbSf3W5jcd1bswPf/APVXGSW73lyYUklL+aclwDt9B6c/Xt0pcavpFm1navJZlG88AIGaXJAyx2nGTxgkdK6IZdKUFOhpNO99TwZYq9eSexwfizwze+Gb77LMu+CUHyZ1GFkUfyYcZH9KwImKNyAQOn1r6QuLCz8R6IltqMUcySxqxKH7r4+8h7EHPNeJ+KvCV14cv/Jky9u/MNwFwJB6ezDuPxHFevluaqv+7qaTW/n5o9CE76M5zzZUQoMYPJzVuITvGHJYxgbevANVmy8QUgjAIzTrN0BR2ZiFblfWvY5ramgjGMo77WMg4O04we1bmhSwi8tpJfnQMN6ise6mjNw8xjZNx4IGKS1u1+0jGVUn5sdqmXvbC5raHrniGIHw4t/bkeWzBXjUcKx4Bz2rzGW6bWreyiluZFa3ZygBHU8Zz9K6W+8QpZ6RZKDJLHdyOWjUkIQoGP1/rXMWNvCNrgRs4y2Dkbyea2ekV3MI7nRra3mm6X5rzGeBWQ5PDY6Yx9as6d4hlivJjJEEhxtjYcsGHfH41nXniWKTS7K1NuBGGLzc/fYcKPoOtZDak90yJGipubYAeAKE0JnV3em2XiPUPNaUPc5BDL8rNjopPTODVbXrXUbeGa8tZJobERLbyRMPmVgclR7YAJNN8O61a2Ooxy2pzMPlCSDcGbHIx+FX9Usp7/WhcfLGlx87QO3CLtC4HPoO9b0pJ77mNS6ZnaNqr3cEcbDa6/OhB+Y4/nxXWC6ktdNaYW8AI+YM+CCuOgrhr3TRoXim30xbskCMMz/88zk4Ge/GOa6vZKpEUkcc0aAhTsyUI9vYnPtT5eWVmJy5opjFlNxA26CQXDSblaQbU2np9OaULNbzmEzIspxncM7gecflWLeS6oL+QT5njXguRjgDIrSsruefTYriMRyQxJtYvyQQeR/Kpm2ioJGPZazLBKrI5Tnkg9a6iy8VXEMvmebnnJB5H5dq8tS628fritC21AkbC5z2ya86cHume0ownpJHri+OJmU/vFHoCoOPx61o2/jC2kgUSDY44DI3H6148t4w/iqVb9v7xohVqx6mU8DSeyPRdSvrQs8tuGTceSBxnqccf1qtpHiT+y74TKxZGG115GRXEJqLjjecH3pxvOetHNLm5luR9TjazPa7HxJDcPuWQbW5FbsGq2zgBpUBPbPNfP8Ab6kInBDdOma6PTvFtranFxaK6nrtbmtIYipFnJVwP8p61qUsP2fd54jYDIb1qot1DbKo+1CQ/eO45yPSvLrzxUk8svkI0ULnhCckfU1S/tps7kZl9wap4tuV+UzWBlbVnqGr6pZwQh45Nxxwq9VINZj67YNLuHOAMb/lz/WvPJtclk+VpWwOxqq+onk7jUTxE3sjSGCtuenJ4ghTHkW6rION+4n+dWo9RlnBZ3+7yK8tt9UCtktitqDXyqgEqSRj1rH2tVvUcsKlsXvGUn9raGZYH3TWj7gD1weCP8+leXyXEn8W7jua6K9v5I52cO21iCVzgH61iEi6mZAhZ2OcKMn8q6aacio03FWRWNzJjBY4ohR5nAAJz2A6/Sum0/weZbcS3U7QOWP7kRgnHGMnPHfiuksdLs9OUfZ4gHxgyHlj+P8AhXoU8I3vobQoSluc3p/h+9njQyYtoj/fHz/98/4109ppVjaDMUALDkO53Nz71ZYrEu6V1jHoep/CqsmqFVKQIoB/jdctXUqdOkdkKHYuuViG6VhGO2ep+gqlLqWw/uF5HIdxz+VUGdnYsxJJ6k96b71Mqz6HVGkluPlmkmcvI7Mx6knNRk0jMB+HfNXbPR7/AFAgxQlYz/y0k4X/AOv+FclXE06S5puxpKcILV2RSLDNWLKwvdQJFtAWUHBcnCD8a6mw8K2lvte5/wBJk9GGEH4d/wAa344o4gFRVVR0AGAK8HFZ9TjpSVzhq49LSmjmLDwhGuHvpWlb/nnGdq/n1P6V0lvaQWkIigiSKP8AuoMA/X1pZbmGHhnGfQVnzahI5IiG0etfO4nMa9feRwylUqu8mabzRwjLsFqhPqfaFf8AgRqgdznLsSfekOBXnu73KjSS3HO8kpy7E/jTeBTS+KjL5pJGhIz1GZKYWNNzTsA4tmkzTScdaQsMAgj6elVYVx28Dtz9aYzgVC8u3/CoZp1H3envVxhcTJpZQOQMcfnVOa4xnmoZ7s7ACRx0rKuLrkjNdVKi2K5anu+pzWZPddeazNV1mOwRSwLlyQADjGKyF12OWPbIHEx5G1Rtx9c/0r2MPgZSjzdDGeJhCXK9zVnuWc7VyWPQDvWRdSyK210dT6MpFZ9zqt1FcQzWzNFNG++NxyQR+nQ1Nf6vrHiFxJqOozTzNhV3/wAPbgD+levQwcVG8tznqY1qdorQz7yW54lUsIc4BB/nU9rO3k7pZVYY4Peqkkb2oeCWUSrEx+TcQvHpUkzRwF0Cky4wVWQMmf6/411unG1rGSxU+a9yldXEk8hd34H3QOgqEXGyPAzTnZgpAxg+oqMttQHy1z64q4xVrEyqSk22KJG2li3sBSq45Zxk9hjNQlmOW9aWIK33jyegq7WM9x+8IQ+BknoBgVoQ3V7ND58iST2ltsVgR8ignCg4984rKYKvDHHGRUsEk7RNbRlmVyDsA6kciqSM5q6Op03WYIfMuprCC5+VAI3X5AVACkg5DYA6cA57VZ+3vqWoRXKW9sronyx2sBCEAnPAPydc9OCB2rGlLaMbaSC7E0c8AmjPlY27sjkHIPQj9DzUWnap5LRxTLL5HmFm8vBPTHAOBnNZTg7NM5XTutD1LUL7V7rRJoktEhieKRJZMpJIYz90bzzwOp4PoK57TIbuDSpYoLee4CTGR5IcnIwOB7jDYPPXpWQl+11ZG3SWRzLHyqzlUD84Zt3O4EjgfLx71u+EfFZ0iNrK/ttrqxYyRSbG6cBwAVcDqK5ow5VqzJxa0NKXxZnw48Wgy3tpcHyiX+XeEAIbDj1OMnANbXhHxBNrUr22pli8Cq6SyEfMPusSSctzj8q4zWJNSN5Pc6Tp06W91KZJ/IjyDnB2njjjkcfxd66HQ5YhbSJEtughffHAZdpJYAMAW4ySPT054xXPK0IO+xM3ZHX3cMFrqCsFneQx7vtDO7K4JPAJOPwAGMis7XZLh7cSR3KOVXAi2A4+p/THFV73UtRVLi3v38uSKT93EcFpUIwpBXtjnjg4IzxWJ4ft7q71KSE7xFOjI05YYAx90g84yB05461xxp3nZs56sr6I6WSK1vFhujfy2TTyKBErR/Mx/gCgE8YH9etc5qtleRa1A894GmmIjtxEzbQN20qR/Cx4PXtTNRZrJojIVjZgF3oM7R05I/pU2qzCzH9nTTJsMazCcvgEEHBz9D196dozbnGJtTr1UvZ8zsaehXttFazyW3n3ESTGU3Fx8itPtI7de5x+tXH1CRrhLq7tgsw2xXUjAh1dW3gJgZJIXP4dxWDpyR3tnBa2LEeSplVFUEuV5xyQDnp+NXBf3t1qVrIqMHjleR0ePb87Nk547dPbmmrRjfVFQqT9pd6nWWWsWcCHzpkh6lt/y9OvXvStqvh/xObjRJ3WXKg7XG3cT0KE9x6+9YPiPxCJrryPsMAEDqxeVDJuTdgoVGNucBgM847Hre8OWdgNQuNTjRnuJmJWSSJozjHJCt8wJ5GT/WvIxmXUMNTeJUm5dNlr3PapVp15WSsjzjxR4VufDd+El+e1kJ8m4AwHHofRvb8RXNSbELCIck8V9J3tjZ6xp8tjfQrNbyjDKeo9CD2I7GvGPE3g+fw5fBWXzbWQ/uLgDG4f3T6N7d+o9uzLMzjiYcs/iW67+Z2xlrys521LRsw27iRxnmtue0sdWtBG0IieEDfcJhWY9dv5Vnww7Rg8ZGcn0qW3cw2EoCFjIzZyeRnuK9SNZpmso3NaXQNOvLWHy7qaKaAAJ5pJDoeeff6U+40ERWT20eqIYyy+Yqxc4HIweoP86zYLlii75X3KoCg9sdBSx30Mk0kk4kBkG0sG5GOldEat0YumX73wKsmniS0uCL3O5C0mVf1z6VyMcYJMLxyrdox3g8FPUY+td3baw+mXvmGRb6xEX3Tx8pHQjsQc06eysL7Tby9ndY5RbM6TJw4IHAPsc4Nbw5Z2XUxknHU86ifyrtXhm2lWBG/oa7zwtdT3t/8A6bCZLeSTEjRLyp6nt1wDXDLYy2t80DSRuyP8rLyMetdJY6vHbGSxeZjATmRCxAY89x9TSi1CdwlFyjoPeEeJoJruTampc7ZiTtfkbQx9lHH1NaWl6pfRasLHWbf7JPbQlFVl/wBYR6H+IEd/pWdY31ve+KC0ccUEbbBtjXbHhQAePU45+tejTx6fqsMdxKyXsIQCSKQ/vCeAxU9jkYz6Ct1OM5O7MZJxSsjkb/UIpYTHMsbNKP8AXrJ8yr2X6dK4qTUW064KJkRy8kBuD+HriuzuvBsFnfSNDNILYgmOKXJI/ujPp7kdK5PU9FuzK00sSKudoEXzKpAHGeo61VhJ2MYQzOQFOAKejNHweT6d6tKWIIUU1Y0Q72O5iK53FM9ZJoaLnaBnNSrc/XFSpAjgMyrjrjHSohuZmCgbRUezRqqkkSi5z0NPW5b1qm8a565b0HApNhEasGIJ9aXsh+1NAXJ70C5PY4rNLOozkEUiTFuQM460vZi9qjWW7Ye/pSm8J71nrKrDPOKUMrHCtz6UKmHOmXvtJJ5NN+0E+tVf3h4K4qa3s7i6lEcETSOf4VGTVxotvQnmJluytWbW6mknWOJHlkboijJP4CtbT/BjZ3ahMAO0UJ5/Fv8AD866qx0+2s08m0gWMHqF6t9T1NdlPAX1kXGnJ6swoNAuLzDXzCBDz5aHLfn0H61v2Vjb6fAIbWPYucnuSfUnvVicx2m3zn5PRU5bH9Kpzas5LLbokCEYGBlsf7xrrjGnSVoo2hSv8KL8rR24/fv5bEcRgZf8u341Rl1NtrpboEUn7xGXx9e34VnFiaTPvWc6rZ0xopfFqOyWJJ6+vrS5pYIZ7iQR28Mkrn+FFya3LLwpNLh7yXyh/cTlvz6fzrz8RjqNFXmxVK1OnuzAzlgqglj0AGSa1bLw9fXhDSD7PGe7j5vy/wAcV11jplpYL/o8KqxGC55Y/U1aJVRkkAD1r5zFZ/Jvlor5nDUx0npBGXY+H7KyIcRCWQf8tJRkj6DoK1gAB1qlNqUUbbV+Y+1U5bmWbq2B6DivAr4mrWfNNnK4zm7yNOW9ii4zuPoKoy30snCnavoKrAUuQK51qWqaQuM8nmguABx+tRl6YWqix7PUZcmgmmE+lO2gAWpKTOelML4pJAPPWmFwKjeSq7ze9aRhcVydpOaiMi4YE844qnLdBerYrm9R8VBXeDT0E06PtYv90epHrzXbQwdSq7QRnUqRgryOnklxVGe529/wrP0LVG16ARq+Lo53KybVXHHB5z69OM4rD1+4ufNNgmfNY8mOQDOO2a9COWTjUUGcyxkJJvsal3q0AuYrVnVZn4UZ6+n/ANaqBuftMs1rFN5d0gDIrjG/1APrVS+TTdRs4iIvIv0DCWOMsQuzO7IbpjA6E8e+axtR1JVug6YlKqBvPc45NerHAwptR3ZyPFyqXS0JbSR7e4lL25UsjAGaAMd/XpnIPbI9+Kn1WxuxNJfXVgYIo1jhlZCqt5hX5SyHnJ69OnvzUGk+IBZ3pea1t7tJUEbRTqfl5BGxh05AqXWPsyaxLOkSNb3SB1jlLHYW6jnByGzivRtFRS6nJKT5xbcRrDs3QT6d5gnFvJJtbdjGMnGenJXtgVLZWdotreR3hEZkTdBPkjy5Ac4K9euAR+VZzn+wrpTPHaXSlMxYbzEyCwIbbgNyMHn86isJfPs7w363LymEmKTIzu9Tnk1ck/mG5Dc20zs9xcxMfOJfejAZJPUjHAOeDVSZFUGMIUYN0PUCrmm3USSP/pIWPad8Uo/1wPUKQOv1qrPKkkxQ5VgMDvnjjmm1pdGifQrLGHXeTwDgAcVC6uHC579M1cQRLHw+Qoz75qm8yjpnn160o3uaJ6ETLukCk9T09KFwA2OvTnvSHEbBuTxxnimhtzgDGDWoiWGM3D4JPouBmplha1lWVJTuUhhgkEHsaqpNJE2AasK5fIcnnk45zQyGtSzLLHLHbqkDJI5JlcNkSZbjAxxx+ZrR0HTLueeZYtPjv5VUBYWZupYAY2kfNnHcd6xMsgU78hegzVlNQdY5YRIyiX5ZMHBI7ipcpX0MpJ9Ca0ZpdW+zxwRq07eVHDISQrN8o5PIwx6//XrVsIri2lu4rgSKIywuURw21hkZOM9MdfT8aqxXVuDYrZ3F1C9uTMQEQ7Jc9Vbg7cBSc+/HFJMbrS0ECOfLuYCqyRD5XVuGGSO2cH69abSMmi+LqbVwkWZZGUCOBSc8DovPAGP84qzZ3L2l2kMttOJImZNsjgbJVIOCuM49j65HTFUbVYGs7siCSCaKSOSORVZkCYZWB6gEsUI7ZyKlvfEM13eQTS2kCRwZVEwzFlJztLE7iBzj0zXLOkjOUdDvdQ8RTanbwl42d4osSyAZyc5IB/z+VZHhq8SXW3ZnLxtlWiTP3W43dex2n8609Hs5buyhuEa4axwGWQqNyjHIKj0IPPcc1uRaaLaGZrSCBbnyTI80zHewGPkUDheO5yT7V5aXJOXVnIyjf266hYxKLOVgX2NJGCFDD3Of0qlrOhSPZ2EUc5OwmDy25WIDvuPzHknj8sVu6PamG3u55IhJ5mSxcn5MdCMduf8AOKjvIFvrO0l83y0NytuwL/Mm5c7voMHnvWVKM5O1N79BK8dWVZrKO30lobNI47mCIfvY0CvKg++DzycfMPoR3pj3y3rxSCeVZQR+7Em13/E8de1XNTt0srWFI2ZZlyDNsAdDluGHODkAg9ecelWorSO9eOaezjZyuBlv9XuI3M3AHO0scHqe5zV+yd1CT95fibU6iiaHh0I9hfXSSSfvSzMzLuEeONu3v6n+lDlFMckEnJUMRzlT3Ge496ivruyttPWGadFimzGJYlIdWBwQVx0OcY4PPeo1g+zwxlHWSIr8jqcg/wBR9CBXFmSlKmuaO34H0WW2T0e/4nQWN6JRg8OOoq7dWtrqdlJa3kSywSjDKf0IPYj1rl45CjBlOCK3bC8Eq4J+cdRXy8oypTVSno0d9ejbVHlXi7wzceH7/wCYmSxmbMM4HX/ZPo3t36j25qK4WJmjLZGcgntX0Lc2lpqdlLZ3kKTQSDDI36EehHYjpXi/inwhd+Grxgcz6fM2be4C8567X9GH5HqO4H1OX4+nio8r0kun6o54zadpGQUCnJByTnrQUOXAjDI44OMbT60ISTg9cY5NP86TaVLZHrXoxnY3aTGWMLTO8LTbHALKccE9s1Yi1GW3WDzlDi2m8zyn6E4wQfUYqvC8kEqsNvs3r+FNvrldRuztQhy2ZGH8R6c10QqWRjKJP4h8ORQ3VnrGnjZYahGG+dvuv/EOOg61nJb/AGO4ledhhkJyrZ7cf0rp47yVrSyti6tCGKlGAOMdP61g6tasurCMhBayhvKBYDbg8g/0rqUub3jFx5dDOtrhUaOSMsJFHzK3Q+taWl6o0V0ixpl2UnBPSq9xoV/ABLbW4lgUbndXyuD0z6dazGgvNOuY5JYJYnGHBK9B2p2ZPMd3L4nu7XUU2TGaA7FdZQDngjj8DirmqXenJcRLHE0dzC6hjG4aORTyc/SvP2nuzGpbT28tjwxU4Zuo5rS03RNYvbtPs/lckkh5cgLjJ4/OtItpkO1ilHIi8rjntR5W0FuGBPA9KetqmAcY+pqwIQoAPX2rNs9Ur5GMEt+FPWNzllG0YqztAAHvTx8x3A/L6U7lWM0xuCSVHJ4xT2hLqCRyPWrDgs58sD2FN6DLc/0o5ibFM25yV3AAj8KRLcL90E+vNWWBJAFS2dhcXUxS3ieRj12jIH1PQVUU5OyFy3ZVELEYHNWbOznuJvJghaSQ9lGfzNdVY+FlTD3su4/884uB+Lf4V0VtbRQIIYIlReyoOtdtPBt6zNo0G9zm7DwmBh76X/tlEf5t/hXT2lpDCBBaW6xqTgIi9f8AGmzTxwH52y391Tk//WqnJqMpP7tvKHohwfz611L2dLRI6oUNPdRpTPBbrmSVWfvGhyw+p6CqEuoSONqYjT0XqfqapFie9JuA/PisJ1mbRpKO4/OTSZx1xWlZ6DqF6QTH5ER/jl4/Jev8q6Kw8M2dr80w+0yesg4H0Xp/OvFxecUKGl7vsjKpiqdPRas5Wz0671BsW0JZe7nhR+P+FdDZeE4kIa8laVuuxPlX/E/pXSKgQAAYA6AVFLdRRDlhn0FfM4rPa1TSGiOCpi6tTSOgttaQWcXlwRJGmc7VHenvMkYyzAVkXWqueIlx71VWYvzI+T7mvIlOpUd5MyVGT1ZqT6iBxEM+5rMurqaU43Ek0jzIOKi81Fy2eaIw12NoU1HoSRR7E+c/MalDADAqq82Y1kDAgkjAPPFOR8jmipGS3LabJ99JuJBqPNG7ipWhmOLU0mkJppNNAO3cdaaxxUbMKYzgd6pRbEPZuetQvIPWo3lA6Gqc1yFBya3p0m2K5PJOAM5rHvtWSIEDBNUdR1TAKhhXMXl8zFjnP9K9bDYLm3IlJRV2aF7rewFmbk8AVyscrxXaGMkMrdQetJcSvMTk5HYVAWdh1yQMc17uHoKmtDzq8/aaHTWWoppspEUkMc5fJKSHywD1BxyR7Vn3ltrEUUOrTRAW8rMY5lcEMQefescyyRuDwMfjV1dTeSz+yyLFtDb9wjG8nGMbuuPaulRj1OBw5XoRm+mxMuflmJJA9+tRiVJLZ0kU7lU+WV45yOv4Z/OohtLc9KnEduqbWaQsW6gjCj/9dNF2SNDTjp01ky3UIEgORKrEE+x5x+lVLlt8ixXch8xOODk+wzVCRyBgHgelQs7MVHOAMDAxmmo63YlCzua9wkYsHLyzyNbj/R3YjaU3cjaTxySePrSWF9PcvFEfMd/KMAaIAMUOSRz944J689OeKitjHcbbL7PCzuhVXcHqehJzx7UxtkemTRwERXAI80ecNrpnIUKRncCoPB/nXQu6Jv0IY9NluZHWBXYorOwAzhVGWJx6DmogkQztcdeDnqKjgvLiJflxtznJUHsQeo9DUokhkgIWERyL3Xow9/cetS7F3sQujhiBnaOwqAoAxLDPoM8Gnlm3FRgDnaT/AI1C+d5DfShItEoKk/OuTj8qjmQByVPyk8ZpmSoIz7Uv3gSePSqSsGw0E96ekjL3yPQ0wcnHrTyAM5U57Ghk3FSYq4YHkdMU05lk46saNuf4SKeo5zkA/wAqBHR6V4cvr+NZLB4RPHxzIR5hJwFAx945xjvntzWck00sscEjlEh37EOdqE8sAOxJHT1q/oHiDUdPuVhi1FraJ2AkmAJ2A8Engkjnng8evFV7e2aa6vXtlBWAPIUgYuioDgsC3JXnr6HJ4qbO12YO63N+DUxYeH76wnQq1x8giK42bWBLvkcegHqTnpzVtLMavNGFaJVA6FwCfYCrWqXWqZstRuL55mdI9gnG4FBuAAGOMbSpU+uRkHihcqzaxDcLMkr3bmaSJUVTESxwpA+UcYIx2IrCrC8fdZi37p65a3OleHdNiXzoreOZRxLJhWIGAOnHBPNdP9hgniN3ZRR3AmVBCc/IQ38WfT19cV5M+l3ep3+2a485F2s8LMUYE4OFbBU5HP8AXNeq6BJa2miwW0bMqQL5TKz5Iz8xGa8yjRi9JvUwTV7CXyvBpUysQuV2OEXG4Zwc88LnvXI2Njb6hrIi3ToLdn+0B+gA4HHZgSPUEmtfxFr91aSy29vGsqAlJiM/vR029eBjjiqOl3aWWkTX1rBLLJNMouA5A2M2ck+oIGB1x+tVCEIO/YyqNOVi1baRcS3MkFxmSLymMhY8s2MgZ+vP4Vei0pyjwbzE8bK5ff8AdwueODzgj8vxqW31GNoA8R2sf4id7E+nsKhk1dl1IXBxHCrjeplwxUDr6/8A66xU07X3uPkjEJNBsX1CYl0Wbyg6vNKc+YfmOTnBxyCcnqM1VIbK/MdqrtC8YHp7/rV23hjuoMed9o52iR5gFQE7twUdDnPBFVrkxQXDQGVSc4XcNpb04PSuLNnzNTp7bOx9Lk1WMouMt1sMU1NHKUYMpwRyKgIoDYr59q59A0mdJY3wmXB4kHUf1FXbmC21GyktbqJZoJRtdG7/AOB9+1cpHIyMGU4YdCK37C+WddpwsgHK+vuK5ZQnSmqlPRo8+vQtqjynxb4Yn8N3KupaWwlYiGf0P91/Rv5/pXNGYrhlKkDsa+iZ4LfULOW0u4Unt5l2yROMhhXh/jXwTdeF7k3No0k2lytiOU8mMnoj+/oe/wBa+oyzHwxS5JaTX4+hgqjjozBa4MsgXYVAB57ZquJ3tjJggtng+nrVP7VOMndgrVKbUphJ8yAgnkivchSb0QOZtDVf3wGdvZjmpLm7k1GEZmJEb/LkAE4/+tiufMsm/JQ9M9ajOobItnlyKQ2Rg1tGm7WRm5XOyh1W4sYjEJcRyLtZTyuPcVd1PUlvrW3DMjskfzCQcnPv69K4JdZQLiQOXB4bFNk1wuhX5znvVQpzTJconQi6ZpTFNM8sKZEZVsfQ/XpT4r+4idNtxwH55259TxXKnUgAQuaBrG1Auw+/vWnJJu9ieZHaRjcd3GB1yakZTt35GW6gVrx6Wd3yjkdMCpjpYxjJ3fWsz0eYw2TGDjApfLOFRuF9q3Rp7LFkqPUUQ6bLNKHBO3HQD+dOMJSdkik77GOtluwSM/hTU064uJ/LtoGc98DgfU9BXXxaXEgHmnd/sjgVoRRgKEiQAdgoxXoU8C3rNm8aLerOdsfCca7ZL6Xef+ecZwv4nqfwxXRQQRwosUEaog6Ii4p0kkFuD5km9v7iH+Z7VQmv3cbVwi+i/wBa606dJWidNOl2RdllihPzNlv7qn+tUZ72RxtGEXuq9/rVUsxNS21tPeSeXbQvK3faOB9T0FctbFKKvJ2R0csYK8iPk9eKdGhkcRxIzu3RVGSa6Ky8ISPh76cIP+ecXJ/Fq6Sz021sU2W0CxjuRyT9T1r5zG8QUaWlL3n+By1cdCOkNTkrLwveXIDXDfZkPbG5/wAugrpbDQ7KwAMcIMneR+WP49vwxWllUXLEAepqvLqEaZEa7j6npXzWIzfEYm/NKy7I8+pXq1dGWMBRzUEt7FGCAcn2qhLPLMfmbj0qE15bu3dkqn3J572WTgEqp9KqFSxySTTywx2qMyU0jVK2wMF6VVePvuP0qR5MZz+tRiRT3raNzRXKcjuh4qImRyPerkse4cdaj8odc8j07V0RehopqwkUTHg4zVtCQuP61CjMGJJH1qQOp4HJ7ms5u5EpXJc0bgKZupjPUKJkyRnFRtJTGfioHkrSNMRI0nFV5JcDrUckuKpT3IUHmuqnSbYmyWe5Cg89KwdQ1HAIDc0y+v8AGcNXO3V0zkknmvWw2FIlKwXdy7k7QWPoKyfNVpWLg4P6U57lkkJUAkioCXlckKAGOCT2r26VJRR59Wo27F1LiBbWSJGG9uzxBgcc4B6qfes+fdGctGybhkZGM1aubM6ekUjF1ldBJGykFSp/karSOZSp24XsM8fhW3LbQ5l3Q+B7YQOZo1klIwoOePfgjp75rp7TWbCa1SK4tY5fNQRyxsiqsRGBuUgZ6DPGOc8muRaCUvGgRyW4UbetDxz2wjdwQrE7Tnrg4NOztZGVSCkWbsW6ahcR2aP5O8qnmHJA+tV57eWOES4JXJXI/wA+4qxBY3V1NiGFnZuQMgE59BUMv2zTZ5La4inglVt3lyAqyN64NKKK6WRBbwedIqrlyx+6DjNakAtJb54blQkBYYUOVKKOuOM7se3PPFY1yk0F08cq7ZVPzDIPP4cUW05WcPnkZ61olbUUo32HXFrPEzgJIy/NyUPKr1Ppxxn0qzGyXbL9rYtNK6J5r8he2T3NTz6jNefvJd5z8nDNtQcZwCeP5VRkfa2xSCV/i/rQyFd7kl7p72srFP3kY4cpztO4r+W4ECokmRZHRYhtwFO4Ec9/p3qaGYyWrQySqVV/MCu2Mnvj3/8A10yORo5X3KyxscPgBsAnGc9M+h9aoV3azKrRGF3DKysrEbWGCPYioJW3Ln07GtPUDGtw08T+ajHG4ptzgemTj86yZG3E56k01uXF6CKC2aeRkjaOemDSqV2/Mvrgj1pYlUsckD1OOlMvoRrjed2B/SpFkBIHbuaWZCnDqTkZBPcdiKYQuflHQYoFcQryAc8nvSHKuA3TPepJEYYBz09aiYtja1CC9h4Y7vk5J9629BvVtdSSSXAJUxsp6SKRgg++DVGy0yeWB7tBFIIxkxFsPj1APX8K1FsrS+0wy2tvKLmJSZUBBTaASWyTkHAPHfFRPVWMJzjsa3imeyaazjs5FeKODauAQRznkepJJNYcH3wM5Ldc1t6XbwT6STLGrBiUB2DIAAOQxHXp/k1s+DdHku9YigMa5WUsdyAEFRuGc544HFccpdDklV0tYp2+sLZafDbQ/LcpIWaRSclT2PrzXU6TJrNlEbr7aj2l2m5YwCcvgd+u5eOnHauGu3jk1vUDaQJHGblyiKCFUbuw7D2rvND03+1NLhjaZQsPyRocbsscs0ffsMjpx61g4pN23Oap7mhrRx3j6XNczxBsMpLMc7lOfm9eMfzziqsF0/28aekiD5vmZ3ARcDJPGfzrRhtH0uSGJiZ1BBd4HL7GPqD71Q8VF7PTprmCwkBt1DK7xYwrtg59V5z35/GuSceZ2SsyIq71OlbRbSRri9k1NeUBxbvlC4HPPqf6571Ujnsjp6i+EE6q2XUjcGwflG8/Mf8APGK8r0q7nB8i3JEjdQGxu/DpXS6ZenUpksAduASVbksQM/0xW7bv7qs2dLSWrOmg0u3gkifTr6WOWcF3ifG0gHsyjj6Edqv6pYzCaJrtI3TaWjVvmKnPJz0645z3FYC2+oWkxmgwigAqR8pxjj8a1l/tVLSKW4eGWF02oGGWRG64IOOo9ODXn1EnTnGV00t1+p14GpbEQtrcQ9KaRilzR1FfOn24gbFSRysjBlJDA5BFREUgbBoauDVzp7DUFuE2thZh1H973q/LFBeWsttcxJNBKpSSNxkMD1Brjo5GRgykgg8Gug0/UFuFCucSfzrlqQlTkqlPRo8+vh7arY8g8c+AZ/Dkz31oWm0t2wrnloieiv8A0bv356+dTxNgjOBknpX1s6Q3NvJb3EaSwyqUeNxlWB6givFPHHgA6DObyyQyaXK2FYnJhY/wt7eh/A89frcpzdYiPJU0mvxOLVOzPPY4DKFdX3YGPpULoqSlJRwRwcVoS2zWieYjAA8YHrUHlSSLvkjJHUV7il1G0Yt1bHzWVQeeQaqbGBCsD+VdKlowcHHJHTGambQ2uonjUgyEEpgZII7V0QqdDGcUcoF545p6x/3uMjIq/a2czgNFA7c4Y4yDUb2RW7Mch2EEkKfStOdGfKe+WtvG9q3krwTzuHP40NahScKOe5qZbkPxIuc91PT8+tK9tOCPkbaxwCRgH86644KK3Z7iw65tWVxDGvqcdqmjjZiEjTr0VRVeS7tLcZMwmk/uxH/2bpVSbVbmVDGjCKI/wR8fmeprZSp0laJ2wo20ijWlNrajM82+T/nknLD6noKz7jUXlBSMCKI/wr3+p71mGQAZJAFa+m6Bf6iqybPs8DciWUdR7L1P6Vx4nHQpRcqkrI0cYUlzVGZ5c4/rV6x0a+1CISwRr5bNtDu2B7n6dq6iy8MWNsAZY/tMndpRkfgvT+dbaRrGoRFCqBgADAFfJ43ieC92grvuzlq4/pTRz1j4TtoiHu3Nw4/hxtT8uproYoUiQIiqiDoqjAH4U2S4ji4LZI7Cqct5I4wnyivmMRjsRiJXqSv+RwylUqu8mXnkjiHzMB7d6rPqGCwSPIPQselUiCTljmjgCuZaO41TXUGd5Dl2JNJwKQtTC2T71SRY5mxUbP70hY1GTVpDGSSAHrTPNQ9znNDqD1qHfGuQOK2UbmiEnJdcL1qssjRn5qs/L1zx0yaydR1GC3IjLorNxuY4C/WumjSlN8qQOXKtRuo64LW3Yry/b607RdZN5YslzMS0bjYrBR1yDz1PauJvbl3vZBJIsiBiFaMHafcZGa1fDMzLqAt2QFblWhAfgbiPl5zwd2K+goYKMYOLWrRzVaiaUkdqJYpCFBz7U8HHCiqAdLdj9oBSToVIwQRwRj61raZHBdjMk3lnIKj19a8eODlOpyLQ1lK0bkBbio2en3k0KzsFKhc8Mfl/Sop4/Kba8iqcAj39KHg5xdhIjeSq0kuM81of2exO2UhTgHg/4VUu9HuFiDo6lT0LDAP41pHDOOrFuZk91gdaxLy9xnBrXn0TU5eixqM43F+KwNQ0XVLYsZLSV0GfniG9cD3HT8a7qEI3E00jKuLgsSSazJ5cZJrp7fwteXkYYTwxkgHDBj/IVk6n4e1KyYh7Z5I84EkSll/lkfjXqUZ09kzGpGSWxhk7jknvVswhrYOodVAJOF4z2H/16haFowRjB6EHqKu2l/5cZhlhM8bKV2hyhzjAOR6V2Jo86pcicK9qQ4YxjoRyU/D0qlFMI8+taNmNykySNGyDhlHI7H9M1nXECJLIEbzFHRgMZ/CqMk9bFuKNr6NppLzZ5Q+XK8hR6enNUJJGcqrHIQ9KkgwIzknjt61BKh8zKAkU7iS1Ou0LxPNo6M9kUF475Rzxs+UjkntyeBisLWNWuNZvxcXmTckfM5bO4/04x+VUfLkji3kgE9BnpTiFubsS3JChiN/loqjAGOABjOBVxaasTycruhl8XeUNccygBS3qAMD9KpOgVsg8dq0o7ITpJLEiiMN912556AeprOlG1yMYx2PaqsNdixHPsQx5by3AVwpxuXOcfpU/2SF7KS7jY/unCyR9SEbo/wBM/Kfcr61nkoQAB061e0+0+1l1MjJhDgJE0jN7ACl6kzVlckj+wokNzjfE7eXcWxzlePvKT684/ukelVJVjiupbdJneEnCORglM5GR/T1pl3btZTtF5qyYx8yqRk9xhgCCDxyKrM7M27PsKe5KV9S5dRyJ+4yqlGznPFUZ4jDMVLI+D95DkH6GniaQBmJJz3PJqwJY5LbyZfuqrMjIo3Fz0BPpVRVh7FLb8obd+tOTcW+Q4NRA/wAJPGakRtp4/KmWmLMXJCuu3np0/SoyuWABq2E86JzNOVZE+Tdk5I4C/wD1+nFVXyrDBDY7ihEokSNywZuh/Wh1CMGJyale8k/s/wCybh5JcSYwMhgMZB6jg/jx6CqzkNt5pBqdL4Wto9R1GCLZOWjPmN5TLny05bAYfMemAPyNaCXdnres32oyCO3h2BUgURrIxb5VwNu1ju2lsAcZxjpXHMVjKmN2B9j096tKbRJYCjTNgBpdxC/Nn+EjtjHvRojGdPW51eove6K0NtdQ/Z98XzQ/LvAYg7gw++p/hJJxtIOMVt6N4mmk5SdoWz5xaVFI+UFVUN2GMcdKytRl0vxJFpyJaXcV1MNpvS7yl5BwUAIG7JweORuqrcWJ8I6rNY3TR3MckfDrnkE9GXIKsD1Ht1wawq007tHJUimrdQ8Sk6X4quxEm1JNkoAIIG9QxxgkYyTWz4Z1oQypvlAAJ+bGSARzxWeNPsNT1Z7e2k8qJwPJxGTkhepHUbiOnOM1f8ODQpr1I7iMx26pmRyc/wBQevpXFXSeiRhUatZnoN7r1oEtp7d4lhKbXVGLsze+eg/xp9zPpmtWD2z28iJK/lM6zsGTPOMjtj2wea47/iXR31vHDLNcRKpMhkG5RjocYzjFdjZ6PY3MUktpLcxyLFv+Rvl3Bd3OOo/xrzpykpt9zPXmtE4bVfBz28n2nSHupoUkKOhXLqAuSwI6jhu2QBWzNpsemWNhcxWhj+027K8wct+9GTyp6ZUcEdxWnfXs09tYW2nwiTcPtDhWwzAdFX35Nbt7rdifDhjvI5EtQuXSIBXPovPTn+tdUK0ZfHpdfiXH3tGzkLPVZI0KXSuySHCyNy2R2Gfauo029SaGW0it7a3ilQZkmduG6dc/Ln8s153qb2a3X2i0tprdsZCSvuI9CMgcVt+GdVF8Ck8BkkIMbAErlfXiuScZRlprHZmsG6TU47o6R1ZHZHG1gcEHqKbnirKwvJsiuLlPNRQgVVLBQOgyetULm2uzFJJZ3sDNG4yjLhSDxjpnPfrXk/UpTm1DZdz62lm1HlipfE+3cmpDS9qT1rgPWG7sGpYpijAg4IqEjNNBINO1x2TR1Wn6is4COcSD9a03SK4gkgnjSWGRSrxuMqwPYiuIimKMCpwRXT6dffaoyGI8xeo9feuOpCdKSqU9LHnYihbVbHlHjrwQdEuRcW29tLmJ2E5Jib+4T/I9/wAOeWtUyRFIpYAYBA7Gvo+a3gv7SW0ukEkEq7WU/wCeteG+ItFuvDmtS2xjdkD7kdekiHkH/PcGvrsqzD61TtL4lucV7aMxLqKS1khGQQ65GPTOK2dPt0XTZdjMrurEyg8jHQg9hWZbLDf6o+9T5b/u4yT90f15zXQ6jp80Wly2ljKsruqq6qnzbc5IFe9TleRE1ocra7o1jUIQAAAQPvH1rSktbW5h8iWKG4IIO5uGGf7pqnL9utocrF8xT92x6j8KuQXsqaOJRZb5FlHmSKcMvpkUJWeo73RSHi3V5fvX8/pw2Kmg1O6kkR3uJS6/dJkJI+nPFcvExz1rRglK8D8K1lWfVn0lOSZ1MNxwMmtLTrW81a48myi3EY3ueFQepP8Ak1R8M+Hb/XZopNjx2RYhpz0OOoX1P6CvYNP0y30+0S2tYhHEnRR3PqT3PvXhZpnqwy9nT1l+RniMaqa5YbmPpPha1sQJJv8ASbnvI4+Vf91f6nmuiJJUbiTtGASe1DssY6c+lVZWZ+pwPSvi62NxGJk3Uk3c8uUpVJc0mPkuo4jx859uBVSS4ll77R6Ck2gU1iBWaSLjFIbtoOKaWppaqSLFLUwk4opDVJDEPem9KUmmE1SACajY0rMOtQNIOa0igEkfANYd/d+RL96tC4uAqmuR1e5LSk54r0cJR5nqaU3Zl291r7MiNvDnG4qK53X/ABFFqO0QxeUOrIOhPTPXjiqFzc5yM1lzy4/GvpMNRUUZV3fUu2dy8TBlOD3B5BrsfD/iC2tji4sopYyCHTA2uPfIJrgrVnkcIgyx7CteON4k5PJ9K2k3B3RlCCmrM6pbkXtxJIibFLFggJIQE8DNalhcTWsiPGxyDke2PauU0688h2GeCMdetdHpeb51WB1JzjDMBj656Vw8jcrrc7bxUdS9e3MUcXmOV81l9AevXtx/9eqtlDdT3EErTIsJICgNkj0pmuvbfZobZJA83mHe4PyYHocetR6VCbku4JEKrk/LgKR3+laTp9DDnSVzrDdJGqCTy2YL95u31pkt5DOyqJN+DkhTj9K5W5lae7ljs5SYUIXc/eptO3xXQE7b1zyFOCawcZp2k9BwjFq6OjmliRtuHHc7hj8qh86aN2e3mVCQR93PH51R1XVZZvJWZQFjUIBjHA6fzqn/AGihASMNuI5yeMVMqXv+5sVGPu+8bNqViA81vM9eMVJI4WVhE7KpHOT1HpVG0tbiS7SB8qzFRz23dM+lberaRLpsiXkKKYNoyMbgD3+tXDCVHFz6IUqkFJRvqznNX8OWeuW+fljuf4Z0UH8McZ4z3rEtPhdM11++1VI4hgq0duWb8QWAH613OyOeGO4t440yRFKu4Y3dc+xI96iunNu0kcqTRyRvt25JABHv9Ca2Tq043T0OedOFR26nlXiXSrvSbySGRCojAUSKmEkUdGBHHI655znNc3jgq3BPevoGxvgyrHHcmNyMZdMjBOcEV59498Hy2uoy6lpsQltJzvaGIFmiZic4UDlc+nTOK6aOJ5/iOKrhnB2R5+YvL+/0PQiiHls9hSzQyRSNFMjxup2sjqQVPuDyKdChO3dwCa67o5+Vlq6tJWsFufLVoyu7IIJA3FeR25GPy9RWQVz0OPata5CpHtGPm61kyA78iqi09iLPqN8x4pAyyFSDwRTpIzcAkBdwyS3c1XlJDUeYp7keorbUViJ1Cng5q5Z3UsTxtFIYmjO5WThgfUEc1BMq5zHgj2qJFkjbcARRuhNXRq6nFPdxfajKJ9qhpHJwwzxhs8k8dec1ks3yqi49yafdXVzOI0mkZhGmxMnouc4+mSfzqsQ3X9KpIiOm5KiF5AF9ccmkkUqxGNrDhgDTYZTFIJBjKnuKJ5jM2ep6D6U7D6kQwTyfelB7UgJRgWH4UrEFiRTYFmG6aAHgEEYII61VckuWAxk9B0qQ4dQcYI/Wmv8Ad4H1oQDBlmCjvUxVoiocdgwBGMj/AAqFGCuCas3EyyCMDBZUC59QOn4/4U2J3IpnQzSGNdsZYlVB6DsKYN2fWpnETwMT8sgxjA6+tRRRu4wAT7CgEdN4au4p7y0sLy+e1gjkaSCRIDIfNbaBkDnt1AOPTmus1rVraXxRbf2w+o3Nl9ldPtNzAqmRjnLRqoA2hu3OeehrzSxllhvEeM4dG4YHGDXXeIJLK60+wlnv76W/VnWbzX3pyQRtOTznIOPbvUtXVjkqwXNYrXjyeHNSltbe+imKhHjntpMq6kBlIPrz9RVnw7qBi1hLk2rXZzvZFHzE9TWHdacqarP9kY3NvGDKpJBYxgZycY5A6jtirui38tpqQmYPHLEw2nbjaR0BFctaCSbRhWgnCyOlObNpL1bee3jdiYyRkKpPAJH5V1fh3Ukv4/JtL9rYzAxzbAgHPcD+vvXOXlpN4jtbi/8AtD2scEOWSRNqSbQSMf3jkEUzw3aSQky2skLIHCyPv+dc9PlPWvPqQtDmktTieqv1PSjp8mmgK4gVIIsCdF2swHOf6VwHifVZbjbBDJtAOTk9T6/rXe+JbqOOy0uG4kRobuRVaRe2Bk5Hpx0rjr/QYr/xJc29jFm5Ub2tGQRnZsDbxj5e+D055rKjQlL390aRSizMljvS8MeqK6hkWRHb5t64/hPpXW+ErWO5nkW18q1Zfukrnee272/lXP3NpfWem6fDKoiAZmt93IBPVT755wPWrej+NbcTql9ppF9DlRPakIx7Hep4NOUfevbQ1VpK5s3siloWebdKkzA+Qu1ZEHT5gQev41i2t8llfrLdxTrLsIkDEqSp6DJ/Con8QPeapCphXAOxAOHxnOMj3pnii6tl1Wa2vJEilhAUtG28sDyCR688j1qfZuSvbYKfN7SyOgn1mFII2tVa8nkcRiCFSGZjyQoPJwP8jrWiDleQy+zDBH1ry/Tkt5Jbf7VqTJbw3JYE7lfBxkq3RSQMYyK9MjntbhfMspVltySEdRgYBx0rzMfhIUaanBdT6zLMXUqycKj2Q8imEVJnIxSYryEz2iLOKtWl00EyyKeQarlaYDg02lJWYSSkrHeQypKqyIcqwBFZHi/Sk1HSBdiMPNZ/MR03R/xDPt1/A+tGg3PmQvETyhyPoa34sMNrAFSMEHoRXNhK0sJik1seLWhyu3Y+c7q2gsNRaK3leSPHmhWH3MnlW/HvXZ+Hzb2yC6uZ1JkbGAegPGfwqhcRR6br+o6TcAMomKB367RyOfoRUOs2kujWFteWHz2RRo7lTzsPqM9s+nSv0SjLmfMjKdrWYogk0+5vlKI7B8q4IOFY5HX2IrNur8S69FJAiiMpmVQMEN0/H/8AXSprdnF4aummDPI6YVl5YvkYye3/ANapvDscFxp6yXCpJGyhXkY4PsAfxraVmvUhafI4GE9MV2fgjwu/iTUsShlsoMNO47+iA+p/QfhXK6Fpt1rGowWNnF5k0zbVHQD1JPYAck19I+H9Et9C0mCxthlYxlnxzI56sfqf0xXi5zj/AKrTtH4nsepOvyxstzRht44kSOKNI4o1CRoi4CqOwFSE7eF6+vpUmNoHqaYRivz+dSU5c0nqcaIGGMk8mq0lWnqvIKcTWJTcmozUzimECuhM1TIyPammpTgUw1SZVxnvTSaVjUbNVIAJqJ3xSPIAKqSz471tGDY7D5Jcd6pTXAAPNQXF0Bnmsa6vevNdtGg2VYnvb7ggGuW1C4MhODVmaWS4k2Rglj2qW103c3mPE04HXC/Ln+tezQpxprUa7I5iRiSaSPTJroqzHZGf4j3+grq7yExxFfsqxoe5jAqG2iXDSOoZQMAZxk+30rtVfTQlwT3M6CygtUxGhz/E7Hk0obfId3QdKuvaFmyTlSMjHNVJ7dlBGDmpUnJ6ha2xVkkYXAVOV9cV0ujSPHhldkYdChwRXPiB124Ukn0rorS3MVpmRtuQeMZonK2w43e5JPJBLeCLIK8AEcCt+ZprdQtoJIyAu3acAA9sfyrmbNkt79XmiSWPGGVhkYPH6V0dvHHJM0c06pABuJ25MvouewAH6inBLdmFVu9hGulMcTO6LIASwkXC7B2B69cioNNlS9vWmlHlx5xiNR+g6f8A66bJbnVL8ohIRnwpd+gzxk9BXTaDo1tbXMnllZZEj3K65+R+2M9Tn296cIe2lZF3VKDb3MDxJA8UcA8uDBJ5iffg4BKMemRkfnVAyG209USFySd7v2fBxj2x0/CtTVPM8sSsE+8xZFU5TJ6Nx6/lnFZyOptnjCA7sZ68fl29q0k1GTSQRTlFNsu6ZqTCHAl8rIJwBncff8q0JtUurmULPI7w46HCk5+mMn3ri7288pgFAUjj5ehHP61raTfpNFK0zsGEf7vAzls9PYdannqLRPQ0dOHxW1Ojea3t5pJkLMkuHaFUCJux9fl/DOeOlZuoX1xcsJWd3PUHPQ5znircb/aUVTwwXC7yeee3vk1RvIi0zB22sec8c5rOpUk1toOnGKfmWNMcsjZQu3Y55B9q27JHkZFnDeefnAGMgY69a52zxChRWPf2wTXQ6QF07BuFMjSAMjYDHA+vbn9B0qKVOMrt7EV7oqa/4Y0bWpGN5ABdlVxPGdsnsM45AHY1mzfDjSL6OCNEks5Y0Ks8DD94f9rdkE/THWupv4WnBmiIWR33nAx7cH09sVdgQLAiyOhkROXUnjPrxyOldEL81k9Djly8t3ueB+KPDFz4bvkguJRLFJkwy7cFgOuR2PI/OsSa3RoBMFYnGW2jhR0Fe4+M9Bi11IIpEuXCtlGjJ3biMcLznivFNQsXtNQntHZsQymNdwxkA8H8sV10p9Gctalb3kY4UtLjGQD1p6xo3ykYOepqRn8mT5VBB61bjtonZA0gEjdzwq+lb3ZzSdipJEI0zu5HTjrWjG1lHCqyxGZvL3Foz91j0H5dazr+Ka2uWt51ZXTjB5wPb1HvVfzGQcZJIxmqtczepZubGU20d4qqYZGZVJOMlcZ4/wCBCqUSq4JyRzjGM1NK7QCSJwVkQ7SpPSoMgvG8ZYMTz6Z9quKErkl1HZpNEsEjSZQGQldqhj2H0459fai+aB1EiiPe/URrt2Y46dOetWoZbFhO9/avcSMpCOkhUh+xPb9D096yXAUHDZX2qxLcjO6Q9sj14oHAzUkjIEC4U8cYHT61BjPQ0yyVCCCMc9RQHODUIOG64NP3nb0osAjJ3yMVYtGiSaMzLvjDDevqM84/CoCo2gg5zTBuGODRuJq5dRYCGABI3HBZsfKOmRUU0kX2mRbbckJb5NxyQPr3qEu6ZAyM9RSKcMdwzRYSRpNYxxQtPDdKNgXdFLw+T6Y6j34q7c6Uh0Nr5LlHuIijSLF8wEb8Ak54Ib5SMdxzWRBdGG3ljjRd8gw2RnK+n581JBcxSSxJdhzAPlfy8ByvsTQ9zKUXe5e0PV00+ctOiyQuCskbrkOCMEHuOvUVVfUJJSwaWRlJzl2yT9ferWqadZQaTDdWl3HK7SFWQ7lkAPTcpGPxBOc1TtNOe5gmeJgzxx+YUHpnB/Tn8KlxTJcY2udnpN1PqOiS2F1d7LbYxgy2PLlUFlz7N8w/H2rWt1tdItrRbhJyGA89WIXd344Ht3rzWC6dEaPPHXOa7fwvqo1BorDUPJnUkIhudxCrjGMg8L07cdq461K61OGtRcdTp71pljhhtWDwWu28t7gSbzET0XjpyeQe49znP1aOebU21a1Q2jOm1liOFVjwQvovt74rp9Rtra18NSzGz+ziJfKwnyeWVJ4x3GfXJrjofF0Vq629xbma3kQgr91ixOAQT2/Q/hXJF1XKy2MVdr3TprjUnBsoroytHafeBIfJB/I9+fQ+lYdzJZz6sh0y08i3mb5urZkJJOCe2McVelv9J1maa0ijsIrh22E3UjRoG6ELjgHPGeAD7VRuZNHtNPNtc2dxDqqXBjC7goQA9x0z/OiGHmk+ZkxTjrYTVrPT4rdXNxOt0YyQFcAKwz7c8YNcrqekT25S5FyLyBkWR5URsIx6q2ehBwM9Dmuwnt9It7y3le9N5arG5SHZtkXBGepx0/lVW/1B2nu7XS2W7smjDkwgEhc8b165BIBx7VpTk4u1jWnKVybw3cQnT5LlopbW8ijSITxxBok5G2R0PfdgFh2/OuxsbT7BaRwGJIpAoMgQEAsepAPQE84rzrQ77U47TUGjtke3VMtJLBuVGxgfMOhIyOv6jNdxoV2t1pkLgQR7lysMWRsHTGD7gnI45rjzanKeHUo9Hr/me/ktWEarjLdmuDS0wHinA818m0fVARmoJOKsN0qtN0qojRp+H5it+F7MpBrsIT8writAUyaoiqOiluPYV2kXDCuPFRaqJnl4u3OeT/FGwMPiaS7jYB5Yo3Cjqxxg/wAq5s6pLPocthLN5ckjoEMo4OTyfy611fxWYSeJLaMMMrbKGH4k/wAq4KGG3e7JLtgICu89D7Gvv8NJezUupyJXikzQuIrjR7SNPsimzkAjLqFZWycnP5cZrf0WPQbmCS1m32yyj+ByFRj04/rWJZWb39pLYxPtmk5Qv9w7SDye3Sqz6fqMVolyQnlgiOQBvmB7fgfWuyKk7SRMuXZnWfBvQlh0u41qVf3s5MEJPZB94j6nA/CvV41xzWP4Z05dJ8OafYqMeTboG/3iMn9Sa2wOMV+b5vi3iMTKXToa69RDyST1pjCpDTSK8m93cEV3FQOKtMKhcVrFmiZTcVA1WZKqvxmuiJshpppakZ6geQc1qolCs1V5Jcd6ZLPisXUtVjtvlLgOe3XFddKi5uyKSL804GeazLm7A71mSayASwmAXv8AWlfUo5Yh5rq4HQ7c4/GvUpYKRTaXUbJK88mxD16k9B9avpolu04mV5Lq1VBuWRSmW7/dP3fTnNZUt7ZImA/CknKjBJqtD4jlVgkIYoOuT1Fd0aEorQltM6OPTrNixhiXeesa/KoFXVSBYlSRGRPRZM9P0qul1BHp4kYgO+CqD+ZPesm41tTI2+LdHyAm7HbrU8snKzKS0N+O8eWPavlEL/f5z9fWq1y7yu8nlR+YcZIVec8dKwLOW6ILEBVH3SzY3fT1q4dZmQ7ZDkRnGCen0qXGSNFGL1Rppp8SWYlkhO9s5LEfy7Vz2o22JMqCV7H1roH1qJ7Ziybyy/KxY/KfX34z+dZ8FxDeXWJNoUDArSMpbgoPqZsFidyOc5681ekc21uc9ORkd61bj7Ik37mUSooypKYzxzkfnXN69cyNItpBDsVEG7C8sTzljzzz+VdEI3u5EylayRf0mAXMUrSKhMmAqsOcflVy40e7k23ICxQsflUOOMdeM/0rP0WW8tHa4G0vGvmDPoP8/wCTWlLfXFy0tw0GFkbdhV+VSe1U1ZGezuaOl6eJYwTKq4PzDBHFdPp7wxqsHmkqfmAJJ8rn+XB568+9cxYTy2U9pNchfJufuEHIbB5Bx39veukSEqj3ESRToS6AAEcN0/Hg5Fb4aXK3Zamdd8270MvXLiGXIQKJRyqp/D93gnp0B4965QxyxAOsLvE+QCRjd61t62Y4rVtm77TuAwFwPcfQcY/Gs2EM8KiQFT/DkH9Kzq1G5XkXSglHQwBBFc3afad6w7hu2dQuef0zRpxEdxJGpOAcAGt9bJoZItyExOd23HBx39+pq3DpdnqF9CYItrKCGCj7/fOPWl7RP3Sm2tTU0uzkszaX7SKDKDsb7xQ4Pbk9qwdTlkXVp5CoXzZGYICeF3EDr16da6O5vxBbRWrxqQu0CTuVB4z6dBxWXrdj9teCexIaVEO5QMjYP/rmtHKLhyoypycZ80ikJ5nVZGX5I/lyPckjP1Oa04ZysImBGFbYQPfv9Dj9Kqz6ffQaaFmjTmYbdjEk/LyMdCOh9a3rHS47a0uIJ4TI7xKI5AuVDHI459R1+tQqMpOyNZ1ocpdg3FIbh3aUPFhN5KhO3y9cjnqf51C+p3MJbBZdmV+XHBzkg/rSaddtcWgXzGZ4jgrI5IRNoAKkc9RyP04qJI2k8maOZonSUeaWQjGTtK5HqOeOvPtXS6d7ODOTRN8yJI3lilUbC20DaYXGAwGQcf8A6jXmvjvSRLA2rW8YEnnMsyq3QHJAOeeK9Kt49u9Y4sFQN+GwWBPOB3OfyrH1yCBwJs+Yg/dlSoKlR2xkZxnI+g6URdnqTNJ6HhAjLDeQT9BViHzUdZFUHawYZOM45rXuNLibUJ0tbgqhY+Wsi8j647VU1rS9R0GZIbyNAWUMGikEikHnqOh9jXSnc8ybV+Ux7yVpbjBABHy4B4Uegz25pDCxiVoVd5FJ8wFPlCjGDnP9B29agkJdi2ME9ferdlcTREeQqvJnkFdwI64I9OK3WhnJWWhQnQpcyJIoV1OCFYMB+IprSrtAIPHQDpRLvExYkbicmlggE0h8x9qgZJ9asCa3fdnMeRioEtpLi6W3hGWdgozxyTgfzq3uNv8AKr8glQPWpAI3VpfPYz8ARCPIP454/KpTIk7bFX7PJp1/Lb3MI8xN0ZDj7rA+/cEfzqk7HzC2OTyat3t00rkMSWUkEk5JqoJv9nqfWrTbKjfqMBGc45NOUZDAdRS4A5PQ+gpiMQTjg+/eqKHegJoRvXsetMzk9qXPBA4zQBIJsTbu/Xp1qaGeBroyy2yOuPuZIBPqcVTxg8+nFCkqcA4osS1c0tTtLa0aJrO6M6yRhmJTbtJ6qOecdM1uaNpOnX2l21tICNSursLHIT8qoByD9TiuUYyuAWcsU4AJ6CtnSbqS4vbSGOaGJ9wjV5MBFzxljj360mZVIvl3I/EFrBba/fWkERgjglaLYWLYKnB5PrjP41nFZY4x8reW3Ab1Heuh8S6NMPEEqq8k2+NZZJQu5SfuuykcMoYEA1X0i2vLzVIolUF0Pl+XIMD5eowf5etD0BTSjcyS0OzdGxDg8AjtW74WvGi1iAi5EBc7C5UlQDwQQOxqbxTp1rb6rJarYQ6e8RJLQvI0cm7BGA/K4546/lUNppumzaR+7uZDqu9SijAjKk4KnPQ9859sVFSKaszOpKM4anReKfENwLdrD7S8wdFYsX4x6bc/KcjBB54rBtb2S60ifT5Y0kXeJonZfmjccHB9COCOnQ9q1tbsb7Vb3TrSLUE1HUmtxFKnAKOrEbS3RuADuNQQ2F/pOoQ6bqelXKSuVJCrhyCcfKfut1HPTNYez5F7pzJKMdNyhZafPI+0ISvoKsarcwwvGiP5koGCSchfaumsTcaR4hihIE9qPmDJHtZ1yVw6t9xgQcqfTuDWt4v0DSde1O0/svEd35O66ZV2hfTcBxnn8q5XK0/fZnzK+p51Bb3d6kjxhpBty5X+FR1xXX+EPDkN2/mm8kiccERR73UdyB3x145xU2haP/YSSPd3MUkfmAPCpHK54PPuMEdR9K7/AEy3026uXmeSG3Zk3qbc7WGOcse59+vFDk3NR6BKV9jxC4j1DSbo21x58Sy93yqzKG4OO4z+Vd/pmpfZ30yK8j8sPb+UlxIRwV4Ctg5UYA6jPIODmrPi21ttV0RoFdZvJnZ0mXaRHlu3OUDDOc5XOBxxWLYGC50C1meMXvkyPbSGWIJKUziMh+cbQQcc8gjoQa3lRhVi1Pax00KsozjKG6O3ilSWNZI2DIwyrA5BFSDpVW1gS2tYreP7kahR+FWAa+FqJKT5dj72F+VXHMarvzmpWNRxRPcTLFGMu5wBUxQ27K5t+F4mEs1xjjGwfzP9K6y3Uu6j1NZ9jZJY26QId20ctjqe5q1eX8Wj6VdahMcLBGWGe57CuelB4nFKD2TPGxFTmbkjxD4m6gs3je92H5YtsfHcgVx6XLoMk5C9ql1W8k1G6nnl5eWQyMQeuao/wkjgY4z3r7yEEo2RmtFY001YpKjgBcEd+9dIur3dlO1tfWypHcRmOWK5GI5FPRlYcN6ggkVwTkMBlsDPUdTWlo2r3ViDDC0ktsDhrZxuj+oB4Vu4IxXdh6fNLl6mNWVlc+moRgCrFQR9Kmr8eqXvqdDA0hp1IRUCInFQSdKsPVaU4FaRNIlWU4qhLJzU9xLWXPLjNd1KFzaI6SUCqks+M81DLP15rPnucZ5rup0bmqRJd3nloW6kdBXE380ks0hVBuJOea3ppWlOAepxycVkw2kt1csqYxkgsegzXs4Smoas0cLowftHlD5kbjup6U2WS1eNmkvpEdR8sfllix4/Lgnn2r0Wy0K0trKdpbQyJtKiUjdzgjdgHjnGPTFYE9vpgkWB4BsTJOcdDyRXrRqQS5mccoScrIwLREvImZZNxC4GCev9fpU1rFcQyFDAwI5OVxWzea3EIhtCKsYCoipgKB6VC2tJcW4yQjggAKv3h3rNzlNbaGsYqPU0Y4blo1YoC3celSPpt5aQm/mghZc4Afnn6VBaXt3bzQO0O+Meo4Yf1rp1mk1SGFGjVrRG2uSgzzyPlzzWdOC1uFSo+hnxtfy6bDaztE0B2/u0j2nPbJ/iNUxZ2/2o2ohAkBYkuvf0z6Yx+dP1LVDBeyWcRCLGdo2evr6//qqzJ57m3EdmAyoGZ1ILN1ySRVuS2ZMU1qirDpscl2Imtp2hRVBMThuT/j6Y4rQsvCokAuTdGG3YZ8zZnb9Rn9Kt6VAXvEDRGNwTJvUDcQR7+grTnUw6FNFZjhyfkUZxt9D704KMnexMqk46JnlWqatc2072pmLrGxVTjtnt3xWv4dluZrZSyKY3YsMjqPWuJ1WZ/tsplyHLnOexrsdK1e1XTI18sviLYMNtAbHU+v0rWdO0VY3jUcjcury4tdV8u3kSYvGqqMgqAf4emBzz+ta1jNc2pi08hIZtpdmTjIPuO1YNsk8mnB2n/cTsNiZ5yD94/jUcc8iX8wDCRyQm5jznpWMtnYhLmdjqJRHLIkSndHGfkQ8gfT863bRoWspISFjjQAsw4Yjn3+bv+FYGh4kmeaZF8uJfmZ2xtJIGR757fWta+gilt4WN8pMoxlIy4YjGMgc//qrTCRklzvUK9tIGX4imgkvVtI/muNu8vEwYs5HAyeg781gWn2uW8G8NJsG0gCmy790ztNDtjycltuRnHGe/NWNL1Rh+9WRQOqqV3bvY/hU1byfM1a5cY8itF3LGv6vDLYRiNG8+IbS7YBH0A6D2rmYdWuklAhkKtngg4xXUXNtBdXS3E8cR8wKAidDhcdB74/KqY0WFLMXiyxl/NGYQCCQc4I/KqtzahCSirE1vem6gktroEORtbnkEVWh0/XNOilZRHc2ssZ8x4W3bADuwQen3QSBWr/Ykwso795l3MyqQ3GAeBn2zx6itXS9Ge5iYyNtjkUo2HAHPQNnnGeKqlTlCXLbcmtUg43T2GGTz7WEQS7oYV3EyFVwx9Mdcgf5zWrYagLhJ7ZpikrIBlhgx4B3fXjnmqEZktbGGyWdUKOyyxlPmDqwwPQ5yMH1pX0W2vbiOOG4bzyZU3ocqSPmDYx35B9TXSotSvE4r+7qR6LbhBE6riZroKolPDIB8wI98+lbqac5nuDbQSRGR9rK2P3ePmRlOTg4wOlYFtod80LyPNDLGp+VlO0kAEMSfUAE4PWrOlatJFctYzu3nI21kLf63nPHo/p2boexralyxtFqwql5Xad2Lb/6NK0cuSrjBmCkbj2+h4xj6GsjWLlNNilknRzIsoKkciTPY+hPP+RV7UXewvLdzFIlsZd6oxBKsOODjAyADjJxyDVLVo4buzlJjD4JDo6DG3ocf3cdjnrzxXJJLnszazaUjzTXJGsbr+0Iov3E3VW+YIxHI/lz61zmo6zLeoAsZCJxk8810GrXRgsLi0uHV3X5QeRk8EHn2/UVzsNyjQzNI+PMVlc7d2M9OM+oHNdVKPc87EpKXNEyPNLMS3P0FTW6uCSvyg8gZ5pVJjt32gbnUjkAnnr9KijmfI2glx6V0vY527kuo24tZ0Vim8gFgrZ2+x96qrJtzz1/WtObSymni4uBJG8mXjeTIWRQDnHbrjv8ATPNR2ekTT2dxcCJpIo4TIzoQfK+bA3fjnj05qktCOZWKiwSv+8PC5qeW/uY4/JEg2DPbH1pZd0ShEYlfpTLmymFp9ryrxE4yp+6QASPrgj86mN2x3iZjg/e9aFAb6U92Hk4H0NIgIOduAa0voUK3y46kU2T5vmp75AAwB6dqj3ZHTJ9qEMiJ64pVb8qaetIDg8VYD256k57UfLs7lj2poUluO1KcrICwH+NAh2TGoyME8062uGt5llUAleQCKiZy3Xt0oHLDPAot3BovyXF5dwKzySvHbrtA5IQMxOPYEk/nVmWyubaxgvxcJIkmSQrEsh6YYEdT/KrF5cxDSFjhjaB7hgCFdtjRqOmCcH5ufrVa4gNra+VLDOWlCvExJVSO5xj5h71Jje500FzHqmiJaW7ia4uSIWtZAWKsuCrRnqGOCO4wSOOhxNUha1e2MCbEdN6gMGBIJUnPXqDwcEVkQu6ThUDKynoeoIro9YinJtDdKzXToAbjeGSZeikHvxxUTMmuSRXm1Bo9nltJA+0BPLckY7gg/wCNa/iA4uLeG31SS9tkgRonYnMe7ll56c9veqdz4Yv4IYp4vJuo3j8wCGQM6jvlD8wxz0BHvS6bpGoaoZWsoVmEIVpvnVdqk4zyelYyb2JlyPU2ptTQ+E1uFuEi1S2kWJWVv3kkR/vA53YPfgiptCj1PRsXMk+xtStC5ld8lFfgNj+8OtX9U8PaNDb4+yzWzKPllS5EyOD3II69sj07VZ8Qaba65Y2F5pl5H9rtLZVNs3y+cF43IemRzxxXDUrQm1Ha/X0ObmWw+3vNH0i1eK/jS5u5YhgjIwuMAtjoSB65NIjre6ReajHaGCwjwP3DjbGw6ZUchT685x14rhLuecSSi5hkSZ2+beMfNj/CrtnPrGhWk8/kSLa3EB3bkDoyHAGR2GSvNb0INaSF7NXVh2k6tdW2uGVHV4trq6ifytyYwcY69jjHOK1DcXF3aS6baTRyQwwbwShbysdAD0BJ4x71yyXlkPC80ckiDUEuVeICNt5jKkMC3TbnBx61R03VvseoQyv5skO9WkiSQpux06V0uDbXY6Y03uj1bwhqAudJjiKsGQnHORjGe5yPx/D26TdxXlWn3lpdeM0uiz2aPtZg42ln65PPGeP85r2Gz0qe8w2PLjP8TDr9BXyOcUIUK11s9T6zL8S5Ub1OhRCPK6pGpZ2OAAOTXV6RpK2KGWT5p3GCey+wqax0yCyX92Muernqa1Iod3Pb3rwZVZVHyU0KviefRbBDEWYCvG/jB47Jm/4R/TZQYoj/AKSwGcuO34V6Z4n8RW/h7Qrq78zBRSqHrukI+UD+f4V8y2mkaj4n1KbySm4uPMnuJNkasx+UFj3Y5AHU19PkOXpPnktTzqs7aspQ6o0h2MOW4AA61vaZ4d1XVommijS2tIyVlubp9iKw6j+8Tz0ArastC0nwvqm2MrrOoBUMMqFo1gmViTgDGew5PY+vFfV/Ej3QDXd5JczZ5jJO1fx7/pX3OHyr2nvT0R51THtaQ1M46TDazSb5UvVXKiTayRj0Ycg568H8qa+qWtomxYlmIXCj7qofUAday7vUprk7XJCAfKi8AfhWe5Oc17NOlRoLlgvmcknUqu82fX8JDRqynKsAQfUGpx0rlvA2ux694Us7gH97EghlGcnIHB/EfyrqFOa/nzMcM8PXlTZ9IndXHUGjtQa88CJzgVn3UuBVyd8KaxrqXJNdNGF2aQRUuJc8d6yrmbGeatXEnBrEvLjGa9ehTudCIri5255rJnusnrTLm4JJ5qjKZAoYqQp6Ejg17FKjY0RoW7pLFIpzu9KcImRVRGxg8is2yuPLuApPyvx+PatSadQ4lZjnOee9aTi4ysjaMlym5aK8djMs8zOjD7meD9fWuN1ee2WU/MA2fur2q/dahdSKyxAlG5wO1cnPDNM7uwCnccbjiurD023qck522LMUQuZDx8o5HNa1jpkE9zFEHbDYLgDkVlWbESKi9+DXQ2yRRsu6QocZLA1pUnyuyCMbq7LFw0tg5Vbd/JhBRHI6k/5Nalm9rK9zcQ3F7tQKxTcAVHGcse1Y41JL13gZsn7iAjOT0yPeptVtDp+jeajxM0hCyhexzxgfh604d2RLsYc2oi51aSbb8xfg/Suq0e+Rg3mmRTEPMwgAzjk8n2z+Ncvp2nWsySTPMyhDlu2D2rqNKWKayklucByrruCkHOOMc4+pIpNLmua3fLY2LC/RxJsH7ts4GclQegz9DU9xFezW+21AZG39WICEj/PFchpaXUV4IS4IOMc139/JDZ6BbSfI1wfl+XlQQOhB4PPOanDwlzScnohV1FJcq1Z4Fq8Ehund8ZLEnmptMm/cmHOAOCa6dvDF34n1K8a3aCJkJZlySMe3t2rkY4LjT9Ta1nQBxJsYZ7/Wu9SU4WTMleMj0jSbmbyNgSAIkJMasMfOB8v6/hWYYZUmaSRCh7Af59aWzaO3skkFy/m5wyFeAPr9cfnWzbONSt3RPLDINxLcY+hrhk3ezOqnZPmNHQfktZJJbYShtv3jgAZzn9DU5uvtN3i2uYYjEflkVtuFwcjjHA/PHrVaSBz4fuooJiJ1iHyEc9cce2DXldrNqWkakx+eKUHa6SL94Z6EV00otLfYio02/M6XxVALO/dLe7luFbl2cfMG/iye/NY1pfzW5VgSAOK6RWGqlGmkZ9xLBmbOCeta2peELdtLMFjdSXN05WaOOKIlCT8uCc8HpzjjvwatJVW7ITk6SXMZuleIVd1WRynBXcvvxXfaZ9hu44pICjGB1+Rn2l16gnsNpPH1rxtrSWGISRhiOMtjAGe1b3h7V7qxlV5D8jZRlIDBlPUEH1qIfu5a7FTj7Vabnod7ZJi/t4ZFVhJxvZlzyGII6qTt47cdelaOnyrqGkqIoxDdxkJGQ20FhyAxB6nB59RXOyXl7qNkZ7fysxvtQyO29WI4APqcDHPXjvUOk6neaLe20N6ksaTplSDjOeBg+oPUcYNb8yck0tDkdN8rTeqHatqMseoxh0Xy0VWlWJm2Z67WGeoPet3R5J5pWezfzbgwIWQn5WxgH3+7t5HIbnoabAYNQSa9BhjDwFcTAlXBUhkwDznIPWo0tHt5IJkZzIsWxXUkOzqqkKP72FwMex5JxWtN68xEmmuS1i0LiS00me8CtF8zFkYByH8w4I5HHAVsDtmqO6HVrHyooI1u0dWiKnblcncoz9ScZ7e1bMF3DbxyWzKZojt2ROgdXGBjn1I6++PeqaOthbechjFuAFSUR/I2CQQ3fndjjp0q6idkr6dSIuzbtr0M+8RPKSaVXkWdnMweEoRk9VOexB49Sfes8mRVCbbaTlYyxbdksSQT+W3HbI7Vf1C5057GARGGJ12l1idiXUdPUYBycnpkDvWXPcMkMcKToqmNcSqmAjZ3d/4uozx37Hnkk1zm/wBk8/8AFFtcy2Jnk3G3jk8tDtwozk8fnmuEYHsOK9E1qMtEVcF1VGB3LiTbjI6dh/WvOHkYFox+lddE48Qr6jXlOBgkN0qa1kQcck9TVd0I69cZ4Oas6dDC13ELqTy4WYeYwIyqZ+Yj3xniuhrQ4m7al6O9llWOzn3y2YYyeQTxna3I9OpqTw6qiW5jmt5ZInhKNKlv5vkk4+Y45AGD0Of1qnLuS58tEdF4xu+9tPI/TBrpHktrTy7psXB1KESNIjuhiYkgo3Pzcg5P0II6GYt2Oeb7GXc2lnfQPdRF43PyRLDGdruq85GeCfkOc/xHiqFhYyXwukSdIXhhaUq+fn29uOh61WW1uXSaSCF3jj4dlGSoJ4zirFpd3OmETpEfKnjaNgQdsqHAZT7dOntVIWttBt39qs0W1urco23jOBlTz268n8OlZhOH25wPQmupF5Z6rPEdWknld4yGkkcEIQDtKkYIzhQcg9zzWDf6cbaeXyWDxK5UNxzzjgg4P1FWrF05rZlN5PkxnJqIkZ4pxUoSrDnPWmMBwQfrVJG40kZpM89KDSVQDxwQRz7UAknJ6U0HFOD8GlYQ8KpY8kA1EMjkVLgNGD3H61csrhLZTKXHTa8RGS6ng4yCOlAmVZJy6RrzlRjOaum4McNvKs0kjhSjRuchQDxj2OensazlClif4e2a0tMnQRSRSXBjXIcLjIJH/wBYmk9CZaIhiuFExldS8u7OWbIP1/8A11pR69dLdpKscIVSNkRTcigdQAex7jvk1RvY4ftsoiIbewaMqoUYPPTt16Ux4Hhl2sCRtyCKhkuMWtTqrBtSuIoXLSstqGaGGMbnUM24he/QE1o6ffnwpqh1Swu4r1GlaJMAbJkK5yeTgjcAVI9weKxdK1HUbKKCaFxEtsxcyAAHBGCpPcegOcHp1rnpLqWRyN5wX3nPQn1qFZ7GHs22dRqniSae+kCoqL5hcIvIG7nA9qpDVL2yeO7iXbEXLKM/Lu6N/wDXFU3tbmaaG5igEUE8gjjYSfuw/ddxPy888/yqG7WW1nWO5Rlz843EENzjIxwelR7GN72D2a2NDWdbvNZijuJtn7sCPAJLYH3c56gZIHeoX8QXlzpMWnXEgkjhG2FmX540ySVDddpPY8cCrdwtvNBbxaUqS3cse6T7MXXIKndGUbqRjqvB96r+HvCereI7ny7C0Z4wfmmbhFHu3+FVKUKcHKbska0qSkrJFrWL60fRNKWCWJ7i3Uq22HYSp5Ib1+uec9sVe8L/AA+1fxLci7t4FtLDdxLc5wR/sjGWr1fw78MrCxgtX1QrqE9uuEDRgRpznp/Fz3bNeg21rgBI1wAMADsK+bxnECb9lhFd93+iO2jhlBe+cloPw/0fSJ1u3hF3fDpPNzt9lHQfzrrRFt4AqQywWj7rhgFHUetZWreIoAT9l4GOWYYrwpUa2LTqVp3le1jrjp7sVoaFxc29qnztl/QVm3GrtgjcEQ1yl94gtLaE3E9wCpJCgHLMR1wP6muQ1LxlcvE6GUW8DAjgAu6+h9q+iyvhmviLStyx7v8ARHPXxlKlpuzQ8UXUOvX6R3c7i0gfENmiczv6u2cAHBGOv0zmuc1DWLaKBrUQw2Ftt/49LNNqsw6bhn9Tk1zt7rkj7kgHlqep/iP41jNK7nJJJPrX6Hg8voYOChFXa6nj1atSs7yeheu9TknZgMKp7DvWezMzdzTgruQPwAq5bWTyHKqGA6uThR+NdcpuWxmkolRYWbOOg6k9BVqGx+USPhU/vuOv0XvV4JFFIEhQ3M/YlcKv0HQfU1YNrBFmfVJyzEZ8pDx+JrCbUVqVFt7G58HvEqadrL6dcPthuwFBJ4Vux/P+Zr39G9etfH9ndCK5SZfklQ5DCvprwX4hGv6DBcswM6jZLj1HQ/iK/KeKMA9K8V6n0NGV1Y6wGhjxTFbNLI2FJr4lGltTOvJcA81iTycnmr99NzisW4lwDXo4eGhutCneTbQa5u+uMk1o6hc4ziubuptzHmvdwtLqWiCaXJpi3L7fLY5T0PaoWOTTcEn3r1FFWLhLUs2cazXW1ui8nFaN+pVeABgdKgsYmRCSOW5J/lWmkazWzeaOWGAfasKs7Tv2NpK6MFHLJjdtODxjIJ/Cqt1BzksudxBAOenfPvn9K0Z4Y4ZQrKDjriqVwoYYTnBz9K6oTXQ5+SzKkatBcAZz9K0jKGgOR82cVXZVBVsAEfrU4C7OcbupApy95o1ihNJvbW0vlkuF3eW24Cte+1j+2niib5LaEnAUdcnvXJ3dpJLPiFdvPLHtW5BapFCqpknHzE85q52S0e4lTu7tGgI7fT7hR+6mhkG7KjPfofQ8U2a6EN5I1m7fZgcqSuNuecY9ulQmMIilWPIGeOn+eahtb1UnljmTKgYXI681MloNRszSh3XF55offjksoIz9a1fE5uLFY1Lr5Jj3KFbcFPcE9z05q/o+nxOg2ssYI3tJtJC/lVbW7EarJJIJkEcfADk5Y9Bx2z/+uim1y6rcJv3kl0Oc8LaotvfvO7bXRgyjdjODkiqnja2jW5tdRtM7ZP49mAHHb0J6VSNo9pLu2tjdjOMUl1Ir27rOQPlLKD/IelbU1yyJmr6kYupnshcvjAxntyf/ANRrrfD17HIquEUSPxuboKxNF0ldTsoIEG7zgVGACQeemSO/v61HpNxJp9wIpUZZIyVYZyAR70VYacyKjKz5Weqt9ltlcXE6yFFVN5+UumMMAOvXv6iuM8U6JDcL9rjndixJTzEO5l9d3Tjpgehq9Dr13qfl20EUZZEIUhF3bf4sn3/wxTr3UXWCe1WTzBHH5apsIyT3/AevrW101exjFOL1epzGmWs0TAHkdwR2rrrCKfzxZq0ksTAN+7Jyu5T1wfrxnkCsaxPkG+82GNHfb5QaTJXPOOvPTOfbB61sRWpGnSTSB1eQjysKdpxnIyOP8+9Y8jjJGspKcTmNetvs980FvIn2Nzldsm5eD698HJ+h+tW73QPs2gWd5FM0s0iiR41H3UOcfXp+FbN5pGmraNPYX721wQJIowN53rxhlA4BJP8AP2qGxstVvD9tYpZpkKRAdu7eeijoOOD69+ua6uSLdn1OVVJpehStZ5dOuIJFuopipDMAenIxw3XIIP5jtW1d2cev6W8pkeO6BxGxjBD7ARk4+oBP88Cq15BCbuSxVSqQIqFWYDOeeo6/w45/Kt+Wwjg02SK0hiXYzSCVj+8YrzgHoMj88jvUR3aiti5yvZvc5TRb/UrS7FhMy4JAaObp6nII/X8q66c3EEJuXAVeVKXCr5bAEDOAflZSRkdOOtcfqs80V5Z6i+7z7lWDMRhXTJAOM9CM5GB19K6GC9+1Wv2aK6FtKJMx+Q2GhkwV2lWPQ8fNnGBzzWlNRs7kVbuzRYF7dwhZIzA08aszeThVeIEEOD7kEFSPyNVZvE7x28scdvGYmkZxMycruPJx3IOO/YGpLHSbVIhcXl2kz7iVdGGPbcDwef1p9zLBbJcBgrNCqDeY9qKxOSAB75HH4dKhzle9wSjta5y00N1FGZUt2gABZTIQu9MZHQ/e56f1qzc/ZDZB7VyZinIfgBu2Bnj6nvU1w1nfyXTRQrNPOAxYK4YtnLbSvcn19TWRqE1xDGtvZJ5yAMu9ckYPIGeuMc5PryKTUb+6TLn+0Yd3dCCOd2kV2lHHJ6f5FecXB2XDY9etdbqrzpNi4YFsc4bOO2P0rj7o5lJHc110Fqc1bYQMzkKO9aVl5iSyCGNZMoUO7gAHjv74qhZhZC437XxkGrdnfPZ6grzKk8SjZJGSMOnQr+Xftwe1dD1djz5ptaG1Bqt5pwjhv4ZRbMrpAWHMYbh2jPQn861bux1DWrfT7GzsbdJBGixyRjY04diFLEYDHIbnA4ByTWQWh1CC40yO9Dxwv5lnLdbgcAfc46Eg49Mr2zSaDcamjyRWd9LaSOPKR1OFL4YKhP8ACSCwB9zSv0OV3WpR07UdQ8PGSZIpIxcxFVZgyg4YEMp6EqygjqMir9nDaXujzyK8jTx28s4jjb/UujLlivdXQ4yOQV9BU9vp91qvh2dpFgkjtIg6yvdANCmSNoTd6jpj09awNOtDKXIuUhfgKrHBcE4O3scelNMpNO7Ogv7vTNT0sXX2ZLaZrkJI0SgFQY13YHQrlWIHBBbrimyaLZ2Fg15dy/aYbhsWrKxQOm7DOoK8/TgjByMVl6zo9zot21hdMoKkOrcgOpHDDPOCKsaDd+a7WMu+a2fDNbou9m2gsAoPv6c0rk20ujn7lUWaUISybjs3HkjsTVVkKDDAgkZHuK0b+1MTrcKyGOYllAcFl56MOxqC4Mk/lR+UAyjqoHNaRZ1RlcomipJE24xz9KjxVlAKXNJSg0AWIlQxnLfN2BpGiaSRuUBAHtUAJxwauW1uZQzksI0PzsF3bR/eI9On51NrCbsLd2BtIYZDNG/mDJCHO32PvVTADcHP1qa62eaRHIZAP4sYH4UjQ+WFyQQe3emC21LD3Qe1WAwxZU5WQKAw9ckdfx6dq6Tw7qUNtp93EBA0klvKj+cgJwR2J96yNTsbaCx01oCxkmjLSlhj5ieAPwxQLCNobjMmy4txuKPx5gHXHv3x35rOWtjGXLJHV6faeHdX/s+3uZJ4JSAk8duyoGbopwc8+/vXJ+ItLTRtcurSGQyQK2Y2I5Knpn0NJpa2zyy/abjy8LlF2bt5Hb24zz7VHa2l7qt21taQzXEshwqRgkmlpBXY6UJc1kLaQrdSwQpIqu7BSG+VfbJzjv3rXPgjxFczokGmvIGIClGBXB5BznGPeu38KfByWQpda7L5a4z9niPP/Am/wr2HTNGs9MtktrK2SKJeiqK+fx3ENKi3CgueX4f8E64YSXNzN6HmvhD4Rw2Drd63IlzN2gUfIvfOepNeoWmnw2cKQ28KRxKMKiLgD8KtrCU5PFUbzVobM7Uw7/XpXymIxGKx071X8uiO2EIx0ii+22GPfIQq981QuPEcVocWqhzjlm6Vy+pa6Wy083HoTgD/AArida8VZHlWZ3FlyXPCr9PX9K93KslxFeV6Ufn0RnWq06S99/I63W/FCK7SzyBm7KD0rjNX8RT3cjpZORCg+aYjA+oHYVyt7qi7vMkl+0zH1+6tZt1qt7dp5cs7mIdEzhRj2r7/AC/IcPhPfqe9L8DyK2MqVdIaIu3GsiEssJ8xj1dxnB9QKyJZZJmLyMeepJqEtg8cn1pyRtIwHJJ6cZJr3OfojmUUtRw2k4H61YigMsmxFLtj7q/19KsRWKxLuuW8sf3B98/X0q9HHI8W2JBbwdzjk/4/Woa/mJvfYrRWcMTMtwQ8mDtiQZXPYEj/APVVsQSSoDcMIYh92Nev0q5aWA27ooyF7yuOv0Hf+VX7WFIpVbG5v7zcmsKtdRVolxhfcwr64/sqABIfKDDgHgn61yl9eTXLnzHJHoOldb42+9ay8YdW6HpXEPz1rinUcjphBImmUDZIvcYb6133wu8V/wBja4ltcOfstx8je3vXDIAyNGf4hwfQ9qhhlktrgMMq6H8jXiV6Ma9J05rRnoxlZ3PstCOxB9xSzE+WTXJfD7xIniHw1AxbM9uojkz39D/T8K68YZcHkV+V4/Cyw1dwZ2Rd1c5W+m/eEVjXk+Aa0NX3QX0sbjGORj0PSudv58AgGvRw1O6RqZV/cZzWPI241YupdznmqoGTXvUocqKuMK5NWLWHzJOnC8mmiPPFW7WRUnVCo464q5S00NKT1L3lYUMx5I6VKilWTJOAcAfrUbgszSdARgCp7Ty2VmbJdR3rjk9Lm7dyvexRSEsB8wPJrLaIkENwK1JZBggd+/rWfcyqhKkduorak3sFkUJyofC9BTIW80hs4A4xUJnRpHzKEHUZqFLwByFYZ613KDsS5pMv+aFOCOOxq1b3AJ+ckD2rNy00LEDGPXtVX7VJHkZ4J70Km2aKaNz7ViUoHDDPHbipIjB9qR3AYA5rElm8sIcAHGTipre7iMalsjnk+lKVN20KjOPU9U0fVEt0R7SYC4dijxuuUdMd/wAeMUyygivtR2XJSGN2LElgoA9B2rzu21by5AobA9a7DTilwjmSZY1EZOT3IGQPxoTfNFNaIidKNnJPcyvGSQw6kLW12bYowHKMCpbqeR16iuHuW3OM5I969BaKG4d5LxWKsf4MDJ9PasG+0i3k3PHxk8DFbe0TldkKm0rbl7QddXR7Nzb7fOaAxo+OYycfMPfGfzqgXjinjcgSK2C4Y/e9Rx/+uqUNhKs4VATjk8VoG1+zyBZxiQYwD055z7ii7tbog5Ve/U1tKgutNuLLVwuElukgJX5SitwSeMYx07ZrbuImvdahbTpHZ5lRww2gh8sSrc4DZGR7VnQaoiaY0FyZJIsYiRHx84HBPsCa0LGV9G0W0u28z7RcSrLBIWypByHA7KwX8eRiu1OEoqPTc45RnGTk99kcxJq8tzrs81wqh3PljeM4/h/ya9A8PPZvbyR3VuSVCMGckgKwIDjHYnGfbntXKR6Pp95rDo+6KGORVKbclWLADjOSP8a27a+Ia9itLkRyqXjEacSFcnkjt6gDpzWUWoy55FVHeHKjR0VYbuW8jtHCZuMRy7fvKSfl9uccegrWCI8RkntljkmXzSUO4x54wBnHXIH0XGayLScW0UU2wmFd6kRdVB5z+PXB71qXeq2eSF3xsMgs0f8AD1I64bs23OecjkV0UVFwtI56jcpXic35cf28LKkYvICRIQOGGQRISeCCCMdvaumtbcSWshVRIcbUlRv9USflGDyOTjHTiuP1qGSS5e5sp5F2LsBaPJCjkEBjuAyxB4471E0uowIbqC/i2ggTMpYBmPIXAB54/wA81nGSg3pc0cXO2tjWvrC2S/iNxJIsEcW9DOduSOqDjB6kdPfnGKxtUurdFF/EkciSBYnidSwT6P2OAAOOn0qKSe81TSmKrIiyS8Dd8rN6cntn9apA2sVm9pMFkcAnmTKnkYIBHJ4OPrXM5c2ysjoUVHd3YsGt7pEhnJCAhGOcjb+Ht361s391ZXViZsAQxBISPNO49g2D1BPXuM8d64eSSe0v2spbdwASyqwwzKenTPOOo9cirMPib7NNcQw4MbqysHOQqkg4BPsMZo5GnYbnFpNGqnkm1uDZbg6neGHVVz0LZ/UD+H3zWXf3E1tCILx2IUFkiYcKWByPUHofx+tY2p6959zPcReWjOTlIhtXn2rn5tbkcMjkMx4yRkgZ7V004XMas0SaldebLIQANxOFB4HsK5yU5NWp7rzJOOlQxoH3MckDAC+pNddOPKtTgqu5JaRsYyM43nANaOoaKIgklvN5ibN0qnrEc4+YjtkjB9+cVWTYJ44mwwJAbBwAO9dlFMbDxLBYrCl2wUwzyXGAZlZs/N82GwDj73zADpTbe6PMqycZaHLaYEsdYRb22eQR/eQHJHoR6+uM81f1GNbOBRYXOLe+iWWaGN8plWO3ryOQeDyuepBzUr2Meh65b3swilhMjMsEMpDRcnyzk5wPusOTxVm805V1y9F1FAhgWG4eKJ8RzR/L5hB9wd3HvgcUO/QylK7uiHTo7vUdE+yxWKThLgKZmMYK+ZwqqTg7iynHOOenNa1pKklpp9vahLm3MM8NzbOnUF+rA/dfgMNp4IBGCWFY8NyNIk1PSry0VoC7NFuYFkYHj5gPmDLx6cgjFdH4YEK+HponuLYNfTYHl3CxTDaD8uSCq5JyOAD0yKmpfltF6kSVl7pyHii0eCOwm8zzY5I2VWLAnKNtIP6Y9sVjWiytcZgOxsE7mbaFGPWuv17TW0+S3ubxYnghk8t4Af3i553OMEAE9BnnBxxzVXQLHQ5beVp9VaG6CuRG9oGBx93aS304xRTbUFfc1hK0LGRcapLdzB7gowCLGU2AB8DHPvyTn1qC4sUjs1lt5WkJb58Ajy+pAORycDORXS63Na6tp6u4X7ase6GQKq70DHcG25+bP97HbBxxWEJ90kKyxxRRrH5TbE6jOdzZPzHp+VWpFxutjBfdkkjHzdPSmNjsc1YuVAuJVAyu442cj2x7VB93t+dbI6EMpc0pptMB2at2d49pLuU4BUow9VYYI/KqeaWhq4WuOZsPnrR5hLZIzQvQ5p8OQ4wCfQVI7XLd1qDXNrawhAvkAjI75NTstzeRo2WeTbtjCrkuPTjnNdZ4Y+F+s+IXS5ukNjZscmSRfnYf7K17T4Z8DaN4ahAtLVXnwN08o3OT7Ht+FeNjs5w2F91Pml2X+ZpDDXPJ/Cnwk1DUo459XX7FASGCdZWH0/h/GvZdD8L6ZoMHl2FqqMfvyEZd/qa3Y7ckcDipTc2llGxnkUN2FfK18biswk1KXLHt0OpRjDZDIoQR83FSXN7ZWUILyDfjoOprmtT8QZ3eS3lxj+I9a5HU/EEVqoeWQANyOcu30H9TW2XZbUnLkpx5myqjjFc03ZHVan4geY7UJRT0UdTXC6t4rht2eND5soONiH+bf4VzWreJriWIxxu9vEw+fLfPJ9fb2rlJtQJyI/lHr3r7zLeF6VK1TE6vt0+fc8utj2/dpbdzZ1XW5r2YyXDqvpDFwo/z+dYtxfS3Py9FHQCqpYO2C2DTAzEbRnFfUR5aceWCsjgs3rIVjjknJppDSc4wKsW9lLcHKj5R1duFFX4o4YWCwqZ5R/ERwv0HaizerC9tirBpruA8pEUf95up+grQt0xlLGLHZpX6n8f8KljtDI+6djI/90dPxq8sQxhug6KvQVEqkYbAouW5DBYKD5gBmkBwWPRT1rUt7WMuDKdzjpkfL+VNQKgG3HsBTzKAMA84zgVw1K7expGnYmlmfoc7gNpJ7gVC7siuxPUcY70IzTOUjjeRsZ2qMnH9BUV3eadp4Hnv9rnI4ghJ2D6twT9Bx71nCnOo9CpTjEwtfhnvUgZFHGQWLYC/U1kw2kEEg8oG4nz95lwFPqP/AK9a97NcapO00+2JCOI4lCgD6Cr+maNJOm75YIAM+Y4+8ewA7/XpXdTwigrzMZV2cMr5UEUlyNwWYdTw1QwPjgnirYjDq3IIxyO9fOyVnc9iL0sdZ8OPFMuga5GjORbTHbIpPBBr6VglEiK6nKsMgjuK+Noi0MwOcMp7V9IfDDxMut+H0tZXH2m1GME8lP8A6x/nXynEmW+1j7eHTc6aM/snQeKLBprMXcSkvD94Dunf8uv515vfTZHXrXsh+ZSCMg+teU+LtIbSboMgP2eUkx+2O3614GU1bv2Ut1sdUNjlXbc9SxJuwAKrg5krStk4zivoJvlQCGIpExAyQKitYy0y7cliePetA4VTnPNQWqeXIGXqM4+lZKejN6SNTakgYAgccVXSTEpRB8wXHTqKXzxwo6DviqvmPHeifnacqR6g1lGN73Nh85WNPoP1rBvZGZsDknsK17sl1Hr1qiYlEYL5BIrpoWjqypLQwpLWR3IAyR+lUZFeKYE9q3mlaO5LYG1gM8dBVG42SEnaDXpQm72OeUbktlODb47nnH0pkkD5yQQpPGaisD/pAULgZPBrRuZhMAqjAA7VErxnoEdiGW2MqFh0wKppbsrBASQTk1pRzbwEx944pskcUbkg8etKM2tDS3Up+UVbOa6PSb2byUibJTIJX1x71hxss8+w/dUZxW/bL8ojhj+fHAFTUeqNab0Na7uopbKGzgADtMXlwc5/uj6DJoh0m4azmn48mPhmY4H0HvUVuPsVzEUCySsnzZ5w1daks0uiybpj9o2tIYXxtZF+bKH/AAqqMHUk3LojOpP2a06nOpCbC7S9aIpFOzNHgY2AdCPX2qjfTSzENMGKZIAPTPUj+VXGvJZzNCdkpdTjI+70OR6Hj+dVpDC8iRDd5SSMxbOC3YH04xTm09EEE73ZLa2YuI0hjYfvZApjLgZcqdrDPocg/wD1604J2Ok3OkRQfaoXt45kEj4MMxOC6fn09vrVV032dzalIhdG2Z4WRgdzKclcepAIH0HrzT01pWlRJIpGVt5X5cMTjnnqeVxitoOUV6mU0pt36E6WQvLi4C71ukj87G0hncKMqB2/+tV5Fs5tRu1uJVWR5g6zsxDKSPvZHODwfUc8Gqk1vE16jQXKmWWRHgKylQVZclCM5Vhgc55PHpWfNLbzafEpGbtJNuWQhyCCDyDyAQD0zz7VooWWpN0ztmttk01vK/mfeeJ9rFZYwfldmQHdjkdB79qzpr6QalKgSFDLwksbId5H3SrevXAODWTY6tFbwvDNqE8hEYWJbY/JKDn5HHHfHbkH15qW8ttLuooYJLtrKVpZpJ45BkWrqOAVxuIYYAyeo96pRuuWBlbkeprXsMl5HKwvY2uYgzOIlOFUAfMT37jjuaxbe5mkupQ940kLTJHLI64jJU/uyw4BGcdeRycVozwRQ2H2u4kmhVmFvDKgMgACA/MDzswTjp681hxWM97LdfY1ubjO4I4YDz8YO5iQRwWHU5+ZcYpcvlqCkur0NeQ3flXXmxiK4aRmMMcfyM54ZlHTuPz4rNltLq73hnkgBL+aJWRVfruI3ck5GCPU1LZLC+mx3ECIbqR/KkSYgAZz0JI2EYwSfXii88S2pWG2+zx7IpVmmjdeGPRuM89e3XntRTjbWTHU10ijB8SWS2lvFLZXccirIYRtUrKinDBmOB82D16/nSjw3F9jhuL65c26gqn2dBuZecy4bBcfKeMggY6cVY1zUrXULmZfMLw5cmTsxxtDKqgbeikD8+DXKSag8cMTRs+9SSSTx7YFXKWuhlye7qY+u2i2GoSQwz+bESWRsYO3tketYTkhskkc9av3skszvNIzNuP3jzWZIx6V1U1oc85ClhjNW7KEySeTvCNIRgnt6VQRS7YzWzaiymiVHMwfcWcpg5UDgAevXnNXLsclWbSHXWlX1nqr2c4AniOWwcjBGQQR1GCDUBubnT7ouQyyryO2MitLWNVnvLuCdjIfKt0t4ZWOWZEGFycDJxxzzjFallcaU/h0XeowLPcWzlAhXmVSPlBOein0xxUpa2OOU3o2rnN6NdTDV7SdYzM8MiyBMZztOf6V1sGl3XiK4vr64a6igt3lRI0j3sjMzOI8EggZL/l0rnvC7eRq6TW8yR3QcJbq+cMzcYJ6AYPU11Y19rK8E9w8kMyTJcSkDIluF4bpyDjv0yPfNKpZIzq/FZHGzPJHdPEGEpDbd2OoHTFbFxoUtnb2Ul1PIDdQm4RNu0bM4DA55zz6EY96hhu9Pn157nUIGW1nlZ5PK4KbskFR7E5x9a3tcUm1s7iaGONU+VEGSrIVDK6N0ZTz7gjmspP3WK7ukYdtZ3l5bXUUc8UlqzRJIGkIIY52EAkA7eRzwN1QafpVtJaXkst9HbTRFfKDuPmB3dR7naMjpnJ4zViTXJ5TfQiESJenmNRtRJN2Q6qOM8ke2aoyvetbx6fdRuUt5W2I+Q0DFhvGPfHIPpVxasrlx8y7oEzWtlqVxLHCYwmwtLFuI3dgc5HKggjOMHPGawLmVLid9v7tCcqpOePrWtLZvHpFxeRvgeYsZBIJ5z2z9ex/DPL9MistQurQXSP9n2CC4aOEfKSGAIJYDdxuGSOQeMVfMmkaQl1RzMxA+UBQR3XvUXzPwASfatRtJJE8iyr5cROC/wApYZwOKoNG0R3YBXOOua0jJPY3TTIDuHBGMUlOOTz603rVjFFKASOlbWgeFtX8R3Ah020eXn5nPCL9TXtHhf4PafpTR3OrH7bcgBhGVxGp+nf8a4cbmVDCRvN69kbQpOR5J4b8C614llU21uUt88zyDC/h617h4U+GWj+H1SaSMXd6P+Wso4B/2R2rtbe1jgjWOKNUReAFGAKthEiQvIwVR3NfGYzOsTjHyU/dj5fqdUacYEcVtxgDinzNFbR75GAAqnfeI7WGLy7cbmA61yGoa00kgDOXdj8qjvXNDA8zSWrLV3q9Dor7xG20x24CL/ePWuTv9ZVVaR5AQPvOx4H+P0FczqfiZUdkjKzNt4Ck7VPv6/yrjtS1qadwZ5zKV+6v8K/QV9rlfC86qU8R7q7df+AcNfHQp+7T1Z1N74l5Msa+aB91phhFbrnb/F+P5VyF/rElxcyzsfMmkYs0jdzWbc301y2XbjsBwB+FVjn0NfaYbC0MHDkpRseXOdSq7zZJJK8jFnYkn3piKW4x16VKkXABU7m6ADJNXEtViXNw3lr/AHFPzH6ntW15TehDtEhtrGSaXy40LvjOxTzgc8ntVoW8EJHm4kk7Rp938fWpYw8seyJBDD645NXbe1WMZVcHHLtyTSbjDzFZsriKWcgyt5aDgItW1gWNNoXYPQdalTCnI6+p60EkjtXLUxFzWNMVWCgADA709W+bOM/WogOMmrSWUhgFxcMLW26iWYYD4/ujqx+nHqRXNeU3ZFtxitRrdB2z0+vtU7x29jH5uozeScZ8leZG/D+H8efaqr68tuSmkwujAY+1u3z59uyj2Hr1rKSCWe5IKNcTyHAwCTk+grso4PrM5p1pPbRF28165uIzb2KmztSMMinl/dj1Y8nr61Bpek3eozmCwt3nlAy7Dog9WJ4A+tbdr4ais7mM60zovzboIj84OOMnsM+nNXrrW1ghNpZqtvag5WCIYA+vcn3NdDqwpq0TJRlL4R9ro2l6VCJbqRb28H8AX9yh/H7x756VTvdWeaUncXc8A1lXupCNN9xLtB6KOprDv/E8f2aS3tIFAcbWkkG5yPT0A+nPvXHVrN7nRTo221P/2Q==\",\n \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAIAAAC6s0uzAAEAAElEQVR4Adz997ckyXXnCbp7uAgtns6XqjJLoAoaIAg2AKLZJJs9Pd09vadn9+ycWfnDnrO7v+/+OfvD/LY7Z870TM/sshVJkAABQmsUUFUogUqdT8ULHe7hEfv5XovwF09koQqCZI/nSw9zc3MT165dZdfM/GD7/+GtroW/CvEbBNyI8X3FujuPi4AoS2bxvLCHtdtaer5beCHfeIuFUqxekdt8Pl9+4165O1Ellfte16rEVcWuTu/eXs5nMfcXVpa7rzJZq8/5b/zAan4+kif3OYGioGVWXunK/C9l4CKCXEC/oglFtpc+XNbfxZPMXWfwPP8Bb4uUBIrH86nOnvI8X+a4+tA1p1QqnSVaCwUOT1bdV3ybZdlaqrOgS3/2vAq5/C90CrktZrlL4mru7sSkaUpW7uKR5nPNFvMojufWYrJav8Iw5JGU7u7yKXl+yQ8N+vPFQgi58CiO8IL0pPF5DxwWJTJfLEDmURANfC8r5RV/Xua+WMSgbA76hDOvNPP8MVULF6mfk2YxB/OD+syGAJ/nc+rpzXP1oLKlJPJWrVQiY2UWeLOSKlnKSuHci+YBdZqFfqoHkmb0X3PudfpZe5TdKdU+XK9/cX/neiVpNxtxHPv5bL7IkyAMI7AQrNJgDX0v8P08z/I0m8+mYTr1ZtPZbEYHcae7aZpqshqPVp8llMDNUhwKaewiWXEBecIunvAyQeDnAJBSFw5bBHONOC+3/p0tvNQPgJICqtJ8MvOHo/HpeNKf5eM0701mx1l+Oiud5nEv9Qdxy9u86e3ebFQ2wqi88JOsVJstoozGhpWQgvJg7idRqVLxSuWo2vSTxsIv514liGqlUm0+7wSlD3vBjheVvSD2fHWWtwjnPlWOF/S2GprPF7PFIiMULrLDN1/vP3n6lX/75e/99Xeqk7CSxoverBGUZ8NsDuhocriYLMbT+cSPg0q5lnazwIvUk1z0oe8DF90Nq4jhsiC/AkRpLmCFSuVRMkhAOCBXo6qlub6l77grNzpRfaiLiOJOgNoq9lK8i7wc7z6/EA9t861WF+J5BEBFEe6tquRFSbmRxM1GY7PV2Wu3bsSVjcW8ns0TP2gGYS2fx9m8FIblRVA6Pjp94+Thf/etL/18/vSed9Lz8lGUj4NpNpsE+bjiZRUvj710ATp6WeZ749ibAapZ5M/DhS94zu0uKNBMICNohNxLhl0lz0aHF/J6LhwXvntBpNHqhaUwWcxLDEANWz8it8CPvIBXNBqAO4yNCPMmZ4TG8SKKo0ajtbvT3Nrx4rA3mXSHx5/41EsvvnLzxTs3QLH5fBKX0pI3nmej0iLVyKIPs9kiFY2KokoYl8aLqSo/Yzxxt2EFfi0WUcnG+1wvZgvd80UulOPL97gE81/zAlEcrri7ZUjBZx0MpvGKeJfgPYpbVaaoVRG4/BEtvBxJDCTlyvhfIbIofT2wEk+W+blX3NXG/zSv9678le26MvLK1hegc58UH7r44vHytxc+dAmWVV3hyeWvLsdQhHWM3fx5YJSRZOAn92WGxpt5hHAvoLbc/SgQzSQNA8Rog+cxnoTG4n28ZVSDgXBu0Q8fXsxXEAi7oLtgoRMUXExxL4URpZXCRTATKoE3DAyyWIzHkOoKbCibBZO0Mvf3tzdfvnFjv1XrlLxKEodBMJ9RThAGpQjiRI5z2BODf54xGLIUjrtIp9l07OcwvxzuSzSXA4C701538TUBlX2J1i8BVtR4LaBXjmCoxgaZ5TOUCKYLn0P6yD0/g+fl81R/yAZw4uUf4dzIE5LBDEiLUgbQM8krYVgqxXyZwuJdzakj8gzCT5DPA2gpCT0ATXfMIYyeh7yCNDD1/JEeieRLKJ6fCPw+nNg63VoqxuzlwHvn+v5OZyvJkzDzf/Dl7wynw3pYPul1G2GlFCFPzBelPPQihJcUiE76Fb8CjKieAwMQWGa6avgaeJ4ZlJCvvlYClxHZqPt/a5fyX9V5vZBnxRdpQJvFHHiG8zzMsrBWz+NkHpRaadpDxM29ZD6PgiAsJ8nW1saimfzXNza/8vr3/+on3/z5/NGMDomjSWkGw2KMzCSaIFfSw0ggsDHER9ivpBmB1MaHQGBgAawSF+yPjgS9l4K5RpkbVkonvFNdEUQZca5r+U4jESmQ3pMcpiwpQnkAA0EaMY6ofJaOJ93jk3S+qDSbUaWys7nz2hs/n2TD8XC0t9tqNsNG1U9CDeSwlCAbM96iKAoYruQ3BydmpcjPhQ8l600NcTVFg8EpZhIaqDCVVP/64JJdqqtd9Iql0EeErTFn6LVM9B4/q04960ggV+CkBYo8L2KAfbsE4OUiSL3KfD1Q5H3hCypwIcY9rvK48uUHiLTqLKG2Xh83bM6ab2AkwbPq8wGK/FtJ6trlWuTu6225sgqXm+YyuTKxy/PyK5dJkZUrVPlcTmoxjkGSjMslIS3SLGGfYSb8Ft4Vf0RazArTjQDxPp9DmrmgR2K6ClomTiN3UtyqEPo2mphgHvkohqWSX2IsWukMOvRbCBC8GdUKJg0zhyVCrhHukcONqFAditSwwvLhKAUVtSZYxZUsJRtkaHgMD8qcT4wm+WE6a0znHRS6OHm+3frI9f2Xru+3vFk5R3NewLpISplclABVEO9dwGizOewim2IzWGRTbzoJ5tL2HOtVhc5fNN9yEBwA1kycbHkViYuAe1E8LuG+Sm+/0umsqdAV2oOUAuE1pgvBQitfTDJp5FmWT7J8jHo08ybSHfx5HHuVqhcnMkXAucmNgEDorhwQQs5gwMbXrQdFE2Gli1TKrk8zR16pK1QIyp5X9hbJwkt8WKYHFRYHIgdhCkq6suav5NUbXmly4+Mf/exJ/xc/e+fR0TuVBN2mNBoNUV4pGWAieoWlmPakM8LL5lE93uoue4aHZUUV5r9idNEG+/0lN8tENVPHk/dv+iJ/tXVZWwtYIYaY9miQAQuXtXcV4I1gn5cMo0bzXjbLJ2mOGaXRCqpVDCXNMMacQuQ8m0qpB2rtSvUT7a1KnLTL5W+8/ePvH/78QXoEXJJapz/qjsSIJBBhrwHz0ZoTG7s0XVR0CS6UWKuVA94KhAVSWsQcOxW1pkeEJ4Th3nPUacQsmDsWBXJgPIF+4AY2R54F4IAOpWAe5/l8lnpzjLVjzESnk3Q0mXS2tiqdRjYNfv7mgycHJ8/d2n3ppVu3bm76JQS4BeM/n03yfIrUEfiLkiiHh0iLRWOObCcYIlVQH8GVR4cLC6ELzJ7UtICX4ZkG7HplCW0erJN4PBcgR5fi8n2VvnhjmGRPlzHvUuLiq2fhnKvGucqcfXNFCCJyRSxRz2hAkfOFr96jpqQsvioCtNo13N2JX3+8kHnRMZfif+sRRYUvlFTEEyjCpFkPr3/iWrce48Lr6dfTFPHrkXyyDiXCRTINvqsulwAuUrw0tgHOn/VwkUmRhoArqHhF98CljPNqDJnKJbMzuZE5vW+MymVAYQzyiCKxWWmACZcQciFWqAUZlFevfNhulC9gmgyuPMxnIWkcGmkg2IjnOz6jmdRH5AC2vWyIn2HPNo2bmkE7IE+BTNhV1K9+2kjT55qdj+9de3lj51aj00CwR5OEkWHj9YMoDKR9IDv4KJqi+HM0yTSb8Zel8LgFOotrinUotSp6AZbIo7tWSZa/FyKLTy4kU7w1k4B+VTzs1nUIUEFbAA4SKyCO84XTeuG9ozQdp9komxv3zSe5n3phFoWQ6bDWDCuViPagG2Uo8gAnliqjXgTqYogQOP7nRgEhvtA19Cy6EXNxUAomnn+qSF8MeOFhOChDJz2/qp5coDUDjyXKWN0JB4sJDKR0+5UXPvl7nxwcd8cnw812uz+bQMepwHSW+hm1gC2TlSnnauqSrLgKARkX42IBYAGrIiQO4LDBvUNAFPxsymsFSfIt0hc5/GoBVx9XE8IuABzRy8jwQnyRoPhqGUDAofk0DguqLmosnCey2YGLMm8Depcm09F0BN5PMcek6fS52mbnk5/f39qp/6j6jfs/euQdpyMvLlUmzEbkwWyRYYaGMVa8MAmjFEYohZVaOeBJbLOQ8VcNMYpQA9yA4UOgqE+KUWSzjJL7+C/Dr/ixLCOGkXpcMEIZ7GbplwTHm5JkU/gmzJ7azL1sMTuazcJBv7zZGk4nx4fDfPZ4NE27p/3nbm5vb9cnMyaL8gCxGwl+kYfBPA6YDAlBOMaskJxOVykUCKZBUqipokQuZKpB8Mglf1zoYXpCnzAy+bEAd8KKMWKjBiwvF7jwuHrpflc2t/OxelonnVbm5ST/ScYYvRZMllhj4BFMl3aSv++NsqrS3fpzY8DFuPDl2ru+c+Nz/a0TgNbj18NFyiKyCBSv3jsghDQQu7urhkVK1XWPlgNoa39KzgcMNkUroFQ8iBitXRarxE5EJdmSmJKGoW8ys0UhSiubGX8+PAN7Lwk0a8uYEmX25kxHoUFijNJgNDDaABT/FieyaRjRL0a9bFWihl6ZQgJlqtEpAqz5rupiURlOGunselT+ZHvrd/ZvPNds1+Z+OB6HWJspNwhjrNMxZloTCxawsynKLxoKc1QQSwgFJTKGoQ6kNlgYPFYjsRBYBSRrH+0FkjAxBykiz+Idcq9i7Nc9OFhJ2TCKA3xcKYqRLXwxhfXOF5NZPs2yaTYfTbMh3DdF8RUDHmfQvyBDe0ctSqpRrVYuV/0wAsIUKbN5nIh0QtpcZegmKRoe5gYM93QPVkGBbTnceOMfesHYk9ZbhQF7Xo16MB1Pp1jfIUjZXJ7RNTgsLgF+q+GNJsl2+3P/+IunJ8ff+YuvHowPKhXxFuYWwkCKXqbOpw5AEmCBKq7hwi6R+hVUhVc8qXO5LVUC8rEIRV64DPj20ljEhbe//iP5UyPyKQIuz+KxCLg0F97yOJ1OSqVZEGR+EAOFfJDSd73+0XjSq7euNZt7UakdlCPrFmv6LAqm80pSrey/tFVpPL+z/5U3vvfD/puxH52WonFpOkdvzNJ5ntKP0QKLNAx4CSkHiGUdMBqJsekC3IwXF+ZHLI0W8d9aphAT9ThM2GWVADPACvCC0cFQROqi34yxK9bGovRkbNS8LXkIAeM5GJqfnk7Bx3q10uhMxunPfnrv3r0nT1669fKHbu5u1WrVsJXU43I5SIeL2QhWDBtW3VWKKmSjfjncjBkz0AV/JGoxeWO+S4HY9YqwxYVogvXTsg2rsBIs2+3enN1devrvLIowf1aLs8gLj2cvlqH1ctdfOsxw+Vs1VNCzEvNKROeqqyA0F14+K6tV0y8klwCxXo1lxZaGfhuNBgpqWSS7mIV7pjv+Lq5ntdd6jJe6ino54BeP5wNnydwn9ulZpEu83n0OVsXdBegXFzif+Qd7grRhjgTiZLVef8JXZw6ZW5qdNbFro5ju0Mg0QUrZEGna2zKHKM8lysr6aQqYD8WXI4lUX7kf4asGaWcgBqAgOYEKQhSAaUOPZ6PaLmPDURIsi4bAzDFgUerMn6E8ogvTfl5GWV4eTe/UWp/cvv6J3b3n4vJGvoghciG0ZBEyORkGaGRogSL/GAdn0zkKI3/oxlAdGysQOGmG1Grtoix3uXGhep6/1C+rgUZKEnA/n0RPq0gUCoijyBiNcDlLYQUQiAHzMZO+cF/uWYbWO53lgywf8WevJrk3XaC5lphhW5QiHH6CqFzSdLjASkXE1YsaWuYql8tIMBQtgxHLFMGEImqGJCSn480CzNFY/amCiDRAgBuCb+LBOBbJ8rxqYzoZxeWmtJY82/jQ7S/8yeeODx6++f0fDyYe0kEww8EtKZUipjNR75FmctzmREzPRrGrIbUyuKi/XebEkN6YhEVZzxICvsIOpRcO8g97CPHLHC52yPLbD/pDbkXFigCZSA60Tl9PUJTu6nAGcy/HrIs9gnuAMx09Mp+PxxncN59PEaTwI6zVZlHUhJ2Rs9qONUiOidlGLW7v3bnW6Vzf3PnQ4zf+7Cdf9zJwcZYjQpbibDGlz8ZiTjgwaBrGAZS7DRk1l05ax131IaADWKSw8URSjQADHc4DNJgXdCT9S0L+SM0wJNIyE2JrkAkBsigO4aAYODBaeTIM2yBGPD48mZuQGJUj0DFLgzfffvzg4eNPf/rDe9v18Hq7Uqnh8ZWNaMA4ZXI7FLOntgANIFB//VA9SAoxvHEaMrigyJUTlgY6r9wfyVa9TqCAvvvcEj7zpjRLzFMldKl3LTviCZPifH9bEiVw8YBeX12+lvlefrGq6+U3v5GYtaG1np/kXGuIIs8CSwpFW1wzXKP06MCynsXfyzBV5XJVcwFhzyrmcpXXXxF213qyCw13j9yLi8SEHSMsPrycT/HKBdznJHOPBIhZhu13+cKGnXvFnUgX1oeGkG6ulyfrMV5CUtRelC3L/Iyw8irAx9hUXsim8Vay4BdZW+5OsAFMUiXC8FGz/mqwK2vGJGNfd+Xh6mlqE+NcFIQSRQ2MM2AKJhTKMTPAIyUIKgu/UwruVpofb23+zvb+3UqjCfedDcvMQ8YxjLoUlOIoxAQtqRPXoGk6l70UXizrmtF0miLZmmGuFhJLGYIAJSlEYDnklwNU6bn02hgwAYXXgOxi3F1ZWmJlDgBhwIb+hI3CAyKIG6wXt+cpf7MZ874jTffmo9l8xGQiLFkMG/XXz4JwhodytZlUquK+udjqDCCZiCCqDk2V3YA7L3FrlWcp8MbznNoDAmb/4HTYmaco014e6VOpuXjGRQukpflUei6zmWo/AgwpNekAFCC62B5jOjRczEJYg3/7o8//o3/xD1vt8js/efP08fGwm1aDIIkrgmXqY+2fTTHJitsLPgYOYCGYk5mgbTkbvB2s3uNu4FIeBBw83yPxr/CqyLYIuEyAadF9RblqDzVxKVyVwE/fx/2c1vkAVmIRQh2SKt06yrKgP5A9Np2Ma7XdOGwFYWXu1SdzvKLr0bzkDfNSFl4vN1p3P/7J5z9UThfff/Tmj3vvHM2m01KQRxHMeOTP6K0YV0hpiNg2BFk6lVq4uwX0bFhNwE0BKJHmbehCoZz+yIJqUiH7VHYf+DXWDiRBfa/egaGrr6yRwTzF8wBjFf9gQViMYMOyF3mYpntzbzjI6tXa7na93BhPx48Onn7Xf/369c3JZDreb7erCCTMX/MxUzzKnYrwozIpWb2PNEi2FCpQ86ghR7Q8OVeXXhBvXxOwlqzeubzOnj5ISHkKMCrU8udjK0pluXBx17Mg+IzLvVrVzVXS1fnyBxL2f8uXA1EBq1V9zpXrqlekuVSjc4kvvf07i6DCXK54AheQYb1aRTIii7ALFPBx6XkssiJcXLwlvP5Jkb8in4EPfOKSuQ+LIgzXlrkR6Up09yLb9YCRdZBT4ikVMYah+jgLB5EWT0aMHyzMWjjhYwuFV8pxM5ADhoi4BFoxCQK0ZaHFRDhAkwwKTNZnJRoTFNc2dktxRr65K4awHEdwoULLSuC9pWYcsoZmL1t8snX9w0njbqW+ha9KlqFgyAcMSgGTQvfVIjEmN6VzC6XIQqRHtaR0yKQmeA1glGQ84axzgQDtXQkcghgx7jKCsWTN5PYel+tB0U11jJVkQfEhWfiwWEIrMyBji6mYpkU/ll80IgMU3MM07csaKSN+Cc1+3mhitsRYyEQjPDGTv8xck7BAiP+Cov0IfLQUl9d8hmImHy+bmcQcoRlGGRcaMGXRPn3DV1KL0ZLFob2p6DbgkWZEpSFQQa2yMZ4PWVUT1qL05DhuBh/9/CcbzXK9Uf75j15/8OZjUVGkL1kWcKkJ8qm0MrpPcBPAYQ16QOOH3gt2gMDpanK8oXfOkMHFcyfKxUphFyHXo8p5D4j/Sq9cN13+9Mp49aRd59/KYUmT+CwJwlyvVaMMCxYSBdPpOB2PsnTCVEGjkVZLHcTCIKqI++TT0TALxkGSV1v1SqVU+7/80b/6d9/9qved7LXZw5Pc6/nzMRSbRT2B7MPi8fJlAjZLBQj8tv4DJgweYawMD8J0bgKVRpBZkHl2HQFHF+ZTPGgg7iuXesQ1tYm8lQ+DWiYr7mk2CXxWr5kzPcwSEwd/pRCbu9/o4Hg173aHKdM6e41N3LNuPnrjzV5vwMqCKRNDu81WPSlHNBejFfTBZHfwTVgHEgiMkuuFYlTNBB6IBN2MBiyb2ar73VBzQ9eQQo3Sxw47FGJOiDWJVyCG8NIuw0MKWn5DaUQHUAsDFYCTcY+VA1Yur1w3u7syKMrSw9n1LIZ69uFZ2vcKFeldfVxSItcfz3+/bKz7cO1zwW39WqErDVAb1qkurzRxddW1xPFLr55VH+vHs4+KUJHeBYrHIuML8UVDigQuAOU1hL74XsqIXaJ3ap27lo1ymbscXDLL50Le5x7d90Ux7iuXs7svSzB8KPIvkrkPi/gigFxO/R2ukY8lE5CQzV2GZsxkNIrNEK9haJe9ZWBropE5PpRLfavyuNmvJcRlhBUyGYsZkiqLjuWxQc6svhgPk8ArM72DcoTWlk2ZSgrCaCw7sspk7AA6Bz0kbVgmyhrFwcHFG2WoXiRRgIIhT8kkmMsAGzRKpcYiaM0Xz210rs3jJsuKTXSnynyLdlHGUTjC41eeMFoZk6aM7CU/tvax2sixRRpI/EIGMZBIw9AyUav0KsJrTAFFrnRlhR0JILS6DFCCHjkYeCFrKokwoMAHbAlWfSluJzlEvHYKFc5znFdGaMD85dkkW2CUxuwMOySlLUzy0xgX82pYqSWLEtbpNIyAjPQs7PUlVoFqqOJaoxjUCxZxAzyIvjdCwcYNiDWiCD0oLzR7ijQU0yytPOEnlvepMpOQo1rRWJ5Fq8AHnvJFEKdejwl108jTsGoOWeX89sfvbmw292/s/8f/6c/vv3EQ1ytJszLojlPkBizbeNzRw7SbImFESGQB9ggRXscZAA3FGFuW7EAk31BtYSE1MLZBmHjpT8BNcoEoEj0HMNUlBt7iTmCJvXqjy3UEAet6izp/U1FrV5HexRdvi4BDD74oYtS/NIn6UWn5OdD9LHS1bP0wS3ta5L1AV5RVYDQ43Whfa7S1KDjD7AGQ45IfsfRokg+QlfzdZuNffuzzNzd3/sMPv/nX93/27qw3D8NpCdEzmE17LFILbQDiUc1itHIcZSnS0hIXCYhLqzL0vtBYEIW3mcBjqegT5DmAp55QUsES/o5bWE2IyRClBRrYIB7SckgPylitzGiS+LW+kjkK1R7kjUkCLCZPD3GqiOvVzesvDEbdH/3o7cePDj/2sRc//Mpzrbi2SHvtasQqZ7I1PqgyxIbxHgf1JCWy3k6zZLoQXtY1YIvTzXWtBP6izyzAo+SSq7hv8e2lwBJ/HLB4uwwUOV/64O9nRFH/v7XqvVeJQM/14SUwnnXZL6voM1I61Lji46I+7kMeLQY9TOvqLl/2VtGW8GK2wiU3flYBUhZVcgHuFrj4rctTWa/QiWSr3Bxg9Il9e5YnWK3s7LJPVTH3CJOwrLjbh2iyzCLOTFJeVhLxESqNqhgklQaLBUf5Qot6RCvB8FIcLDY3695k7E/G6AIs+8cflJGd+awnLbMJBUQZikY1+WNoUy5TjLAl1jLg1oOcTd6VclJPkgQjdRQM5BrK9htzbzz05qV23G56C9yHyoscfiQCQl2pEUTfKIWabVtqqFHWPIpwOOIaW9xVtHQCtd3d3SuLV5CAu1xYHMEycndeuXgeL8QoXl8DQ3tjpEc8WNwXKyUmaOzPWrmh/TdysV4cXXice8z/sSBYWinLeVH9kzJzjJAwsnNKDXZk4Kzcxa7gA8Z3XUuJ50JTRntG4QUoark4HE5w0DyM/BBVGRaMtjmXK8kJdJDkL2UHoAEIOjU4IAcZSQPIT/j9wDJyFnfOa1u1lz/9crfbTWdf7z0dBExgY4XIAwkKKZPuZGZcHL0M+yXaug9BFySotWRCAXzFWaktSKO2UFtVi5Rqiw1rx9F4q9otQWmh93dznXI5reu4y/Hr6dV5VyKNVWPZtaqotYve0S9YjawBcwGfcR5GeMQuQy/NSgHT/tlmI0jCxrxcxq89ZQGcL+fhkh9nvX67knzuzke2N7a3frzzpz/8xs9nh6lfO+kPGQzyk2K8RWyPkTCr0k8nsGHEEiub1wSED1wOSgSwkHDTpAsBq5slIwFJGBSqt+YobNQSZd+zCYoUYMrjnwapmoOlRet6xUHhzPQbMqQ8GSPMROCZnPHHExb1lRuVZr3FYvDvfPe1X9x7+pEPv/TSC3vdwUmMQR1XPViurBkkZ3HWnEcVEXgJvvdATbsEqGKos2qC6x7uXKoshVtPgA2KIsUykUOLZeNd9KW7S778qHj7rK4tEvy9DRQ1LwJ/f6qq3lnvH6tZEfMe9Xw/adY/p+184oQzwqaRQFtlwbnyehasXLkuNz4sAuJuhnIuAY8EuMBO7peLKPJXGqvbMjcz8/CB/tyHlgDNoMhnGaA4PkSbkjlIlxvOvC0q48o1bUDDezb3einsIppITja9yr7C2WJ42Itmk3iWosJG5YTNMFD0prOMFUEiARSB0D2DAmirNzFNdrhiNKLyxkHMPQnaLQxZNWZ2kXIfHR92WSfJvhmnwySuX799veNL941NOVJLaQ4iOXsfkJkaokUh1FOtoAFquMZ1cVmcEtorNdAEAjWcNBaptC7gEi8jycm6ST92ubcUV3zrYngpMrZMRJVgY0vu62Z/meuV7usW+7LqV2Fmf7WTkBSjYLpg2jXyk6SUVPAuk//3ckLOlWs0k0LVNjXZxBARTFkstWMa5K0k0zBCDiHrfJRvl1IE1V5ZG+092QtibF8iXy36F2ZMZ4ltm+VypuXIZCir5CJsl1vNxu/kn8FX6NXv/OzBWw+nsOAkwTlWfuYz7LHIUlK3DAyYN1RPwYO6si+UE3qENmakxmCBBGWceQkwkVxNe4LxwgCLhQtBu1cwdQmX92dpuoLMVdeyNy+9ouXELYtYjSPFuLGj9Br4+jXzuGMbyH5A31iFgMorvYbBIWMxFUBnZGP265iNhtWpX6lu+aUWZo00KE1kxSizZ1syT1jv3Yian9q/W681WuX6n/3wmz8avZ15ZcAqQ5QWruPxCE7BVtlnTpUEOI73wIxdqcyoOFhZDYvbMo6P6X7rBbWICmv2wTrBRWvw02uaVOAT5DohhXBG2AJJ0oRHoKkTKpLOkBHhwdnMw9I+HGWdZrDZLlXCydR7d3iIin583P3EK7uNehlPPZBywh5feYZ8XS7jtZfCkfXPyBFqMBox18U5YNVSaLHqAAWX19Udu3r73r+gBRmTpsCPInDxw7OOP/fmWeldtueS/kYfinIvBP7Wyr3QGo1Thx9rL9YrU4RdoKj2WnIFi2QX4sFOQ/VL0faJY2Pk6ci96JcQdw1LVt+Rvyt6/e7KJaaoW1ENAlzulQu7u0bC6uJtca3ihFHFh0RCBYxiLL/iFZHLrOwb0rtvXcB9S4MsH0bj8q0jcBJUdYlSy3Q4ZxcpaAAzWzG2Nel2s0llkbUDvxak1Qi3HTJiyS3jK4x99kwKBhlWSnKVBQrjL5QYMi3TEzYvSBErHyNmEtkFL6jGfq2Mg0/IXNpidJoePvWH42Qwbe1WXmhttAMMoqhkAFw0g2U3LJukGoK+6sfuVhhdYSFqvesQmgZ70t2UhyWd5NleF0KGIlaXUl+AmL0sgEZCl6aIcY/Lr4zUSZGQeZz6yJg/X5jns7bfwu15udiXZUjoxFKCLQ2yCQSJ3cPM+TmKy8aAUWepjJEjV0EZn+VyRcq1Souq6A/SCTkW44UDi5jyjKqp/T3F7+Qrh7zCvijoatA9tjw0ezBMl27WLDkzhUCWOTzaCA9mFxVkCH8eyXAhm2U6bF/b/OI/+f2Nrc5X/+Jrb712DwOpzI2USH546IF9cnyTeENt4MYaH1J2xW7FPIywO47KI8kBGIUKZa0FjivQWHgwd2JFsF3jz98L7C2iXcKiX4p4F3hWvFViOYgufMJjkacLqIFu8KrywBnvB9iVeKKsPAZyQI3dhk0qFhiEsulpGqT1XlDa9INOXq35QRkhFOs/Ug1TBqcHB/F4end787/83B92qvXmD7/2jcOfd/PRFNzwAoxACLEUFaEHa0MbxBsxSmgORaqbTJhxLBmIc7nhS7eZJEMENhHdgTnV48YqQQd31ZYcBVx+MAvzFlMNLwE5Sxugg1JChZmUJjbN5BAoE3gZa321CYs3Gp6Ox629nWqlxg5zR497k+FgMRvu7TVv3dhrNspyEEROlSkGcwsYyHeSKrmgCMIqba3pqsAPAergAiI1pFdBSq7gr3tBw5TF1ej062b+W/7eKWeCiAHDQCPYXHUZOl714hlxz8jmGakVTRe5PlGXnb+KmCJw/v37e1rK3+cSS2livYB8BuRVzxICEXsIxTMu0ijZVdfluhUpL78qYoo0F7J0CYpkBACP8HdJpJd14JEPyQSYcXeXxWhu27irnlbvjM8Rr1cMX8eGeY91tE7ixWIaTAfhfNQKstv1+FY9/tSdW2X2ih1O7h0cvnMyeJKxz0+MORUOg67MthLMLDLiueSvHLDXrALcsXDNsjGuHKcZmz1U2BNqNhmfPH48OziKR1ljXtr3w/240tIexxIPxBmg7OzKGCZeGGMW00pK8QddKkBYYfQE7mMXHcXY489eGbFXU/VH1BJADq8UY8nW4i3VEtNcDparbipsdVG6oiB+DozilPzJ4Czjs20bhF7jAuxKjd8y0HXlSGyiWSGzhHifMa3B2iOMEygkoJ2YLa2mcdZE3awEtdJqr7pJUwK2uJbgbY4B2+2wrAUz+H9BMWEVKLriuSLjQmKYMZN/8GA4gqoha7e2IcYSTlKKR7MWA4ehAjoYwCQbV+Ko+dz2J6KPTtNBNh+++9oxDkUltgC2kZhOtME2mfG9POTUHaadU7apbmRI1R3DWAIUS4hbWo7VWm2k7tRQgKQ5dCp3avmsy4pQapI9K837jHdZFRm6r4rHIgCPAxo8ElBF1Ro6CEbC5pTUVsYDPNAArDzyF/lxNo9n/ah0mvi7JX8nqbSAP1kgoYWlCpIs7lnpSandqPzBhz5x6/r1yt/8xx8+/vnDwUM80dk1ZaQtWEf0jMlSxlhtrb0wZ9kwKrAKnmsq1bTLXpJIz4TpX3iwqu1mHwAx0bwEzKoX1aP3aJTQmC7gB4TRLyjGKkFs0QgG9BNbV5e87umQAY4sF/jtzY3mdvvHr9578KRxeJw+d2tvf6fFOnZvPhmMh2W2MtfWldrJ08kEkAGwWwTAdZ7u79mLBvRnNHbZ1mf/FLSYgJWyzO3SF0vsuxRPn1+OU8yVwL866a8Sq/6hJ+w6q/OqFb9Kju/vG9cpz0pbvCXgwu7u0hdhAmd1flZe5+P17VVoQPz6BQUT0yL3Z/XLKtsLCcjEvSHehYvA6otf8kv64pMityJA5a0F59CC9GRa3PX9KubMcmscikSas2GaZ3mYhPJhtJEppJ1xC9eEQ3rT02ow2a8uXt5KPnuj8dHt1vXyrJVU8kXt3W75xwenPzka/vRo8HbvtFndmEdJEseoZgw8lYsUzkwYSIX1la3p8+lkNJhMBhCzKPYHgx4MOMnm1cm8PEl3k43nys36NK+iNEMloESa6ML4jFeI/rDVWWsMT2XTpKVQfbiJOA6jQwI9EJH0LsEKurJC54twJh+lXAHKhQGFvrer+MBF8rh6s/zKoGotJIXcpmidGggDlv1Z6i8BliGh+yLBQaaxuVJh0UZZhdlHBMuCjAvQJdQkSJ8uqTMqAW9bmkXiJQM+kx30OQIiSgpQJgwxhcSKqS7YCJrdr4CFDsMQS5P660cltkkR3wUuAhqVUOW1m7YZO00ycOwdsGGIxDsuYroX76nZKGmGH//sh0fTk+7B16aPRZwFEFVKEqk6ocS+3CqJOqjq1jnWGSLmWDGUkA4xgFNHpQHp9DXZwHGpL1GSCnSR+tJloCbF2UsXdvGXklvrLseqB5ex7vMiQ/IpMiwCwiguGmYjQsvuDAdoAQsDmAPR7LYZEohgap8FZH3Y2iwL++Oan9bns6iuHUbp0nJcmeA6J4f/asYImIzrjfrv3H2lfm3vP3zrK3/+1b96O313hBsEXcU8POClT00PUp+qY4CqgAsoqRR3VcVhkoUZY8RrEFBbq7aEPMlUDCNeUlUpuHK8Z1xaj4j7WqPVPdZ8LezX+igSwHyl/6swTXbMkjL7i7DIjT3mEBWYTIrnSTLuVq/f+lB3cvrG2wfs4TF+Yf+5m1utai2MQXOWp7PHCH3KvntkpDlhLrZwXwJRhaoJMuFxV9tWnbn8pb6/zuU+Lzr818nq7+LbFTDUN38X5S/LpM/OarKqhxsh7qkIF4FVqnO/lzOx1yDguWTFA+nd5WLIHMxhSGjRx1UXia+KFtIRz9vLgQvpSaBk52OLbN3n63eXEHOvCKuGn4boMr1+GHJ2EbRsXearucxlw41xwDmkUFpqlW8SsZ6kKmXDeDZoxtnzW8mn9quf3o0+1QzuJN3p/bfa1apf29zttK9tbe/3O8lbTyZvHXTxRYHYxAnMgW0xMC/PMtZpoBswdsfzmYx1GUqAiBF+oOxYMJ10T6tRuboIy950L0pusiJnMovjvKQ1xhA9vERgMTHOWtBqaR0mwPOjLjGKYzU3UK91p0BpANHv6nIpi/Su1Q6qLuzuZFxEAiIi3aOLJ0wMF68APqDmAoZc5lKupbks86D17k7TmRWWfxTcDvbGV0gI+NxoRhzv15gN7rHk05FEIupheFYJ4oQ4KtMG1wDudLEpXVB56RJqsvifXVahbJGPsPtjH+UVxeBV48vbyyYClzYSYEbN2fQaEp0zlauvnX5l0hhMHyUHmcArh4HOlJqwWLmz13zlEy+ePD55/W8epV1vMhxTLZg0vl1a3yT1WVnQBAOMWCwKlxwBiLQKgmHiwUZsrV+MhSPnAQtwWNwFOMIaFH/5AiyXI4mhmVfGPzOyAJd9u0R616ECrnKjaUW26gXBWYimiRWl1IgDtgCKqtKbkjQFZLrQH1JNP0umaX6S+aNZVJtV6rMomUftKg77U2zKMYuEEWrzfDyadUufuHszWfxxY1H+i+9+5dXeG1YfjhjDIk33wLO0d41EMuOWDngOopeApPFu1dMbekG1VAwYpFcSxSQaIQ/TYZKU9KCqK0C7+UJhRpl6kvLUNlw6BLDctusiAuRk6X8Vd/n5+Oi4d9rbef5lPBlG0+OHT/qTydv908HdW5s7G3X8CbAeIhhrGpi8DLUoacmAFSFQn11EuAcL8Ikqr0osRY2zlO8dWoKAD1cZCirWwe/94ft8q1oKdqqcBQVcdZUJR2u1dUPqfebqkpEJFxm6gD0VrbCn394NEK1lbi1UO9/vdf7z9/vVe6TTuLJecwFS8uiGp+Hp8lOgzOMS1sJ5XYZaBTAtwqGSZWKUypIZdkGAlhejgLzQL6Qq2JAQfSdzIZP4LABZEgipDvYKlUpkVloQn+pPYjMqhXQgu6gdvzbrJozRXlHahR0PmjzSVB27R7HDHssG8LGS5KtKzSeqgDb1yVqloO5P9tvRZ+5sfPZO8/nqpN17sHj8sJMNvOPF4uRBxHKVGy+Vr21meRNy/Y13e5yfF5NRHrL/Uw6pHo3Yz4+Cs3SUY6nm7ML5BNYasWbGD9ut6tH4tB74VbYD8tJ6Kd6sttgYSpNZUmFph6ie2qMmoYOISNMiorS6kUh6SNDStWo0HaI/JZJZXem5gI8L61suAVkd4/rNRarh9rZ41HeY0UxwVyYCNHlCoaBpmHPFxyTvM6Mmz2cdebTSgMWDUX+JFDEzDXjm4/+sP+muOsgJxzIYJnZMaSvSUsACNVUFEKfpX1DRuA8x8AGhnGoodVbIR20EJ2sceljeFypAYI3GmiGDPRM4nZBKIsFgjQQBAESmDTrkMmMaMAWgKAnRqOhMM8hxnE6GZc75qYXpgu0l8s1rrU/83kfTrt99MD54goVai3C8lM0Vp2zqEGIrmZMXjZAJWd2leop6oytyN+5rHeRgb+uGaRbNQcqSiGXdJ+lg2V3WOLJSV7kLDNEX9qASFFh1uUuxflc11NRzl8vNdS4vioBLVPT+uW9IJgwD34AQAo8qIFFJMC9KUEsZfOkcyZIdn2c4DvuDiT/CUTEvteaj42ltc79WrfWzCXJZOanAntKT/r1vvnb39u3tz/9xAzvIt/zXe/ePPfYJ72sqVaMcyYnytLEq+Qv5l+117SCG8a9KWEACgQF/2WrSC1zyuBI68ONkHWu36itQaxAZ0ZAjPi0Tx9dXvBKVkf6KRJ6mA4RkNoCdgtJ9zjjBdz/x4vJ3/+JL5Rv721sbdOTTRyf948FwkD7/3LVrO424NGO7rUQSHseEsA0YA2QGUQBAuihK0w3CZYVtUPPLxSPNETgJmuSl+umCIApHrU9XA9L6wH2ljw0CZODor2K4+NQFrrivFbz+1tVqPYawcA5JUMYBnDioIwG6hNyxBGFiAmBqi4FfVEMkZ9kXqoS9ooJWN4b+xYtvxXVWnyj9sjlK6fB+ibIOgLTzGQrhxayLZ5B2CQ3VBwirwuoPQ4JlMkFeSMNdJ4lam1yJ7k4yBgD5QNWW7bIY3qKJFCkJ8Kl7NJrlCrIa2LfkYM4Uyw5yRFb1Y5zxGQoLkqzVkBgKFXUUGAySVlcDsEoQKRWmK3NL4mRW9ljMUAVFB7EF0Xg1ijQgtnyV6E6aZwHaT89muTdG4aA0NYys1EzhAq6rdJzhK0oPjhUUSqkMd53HiX0c8ZzzdMMcE5bmnEwF4kMhPnyRYqkBzh1+lMwmw7z/uBxOb7aTKszAm4RxMh6PT44HYVy/e/vO/u42HLrkZRvRYnM22Kp6G5thpz6vBk/C/rE3PQqjyeT0FKcRFGQfy/BRudGZ/u5u/cb+K/mXfvTmSff4gONkS/3p7MnxKXbYGOcr6BZe05xsx9HBE07amVRYp4onVRokNAKdcerDKBijLHmK641ZzmbvjCL6Bysay5ygfDNckmiIsWRmsmkkMGIIWIcCL4aSoaOzdsFpYH447Ap1bZSA90BR+CTQWieqm5WjoKsfdRQvCEljEFlQNDdUW15ikJdMD/EX19COvnSutFu8kxYTzpZBMZhru6thNhsQvwBG+JLrjAh5acF9p3l/XkrpKh1QhADCbDkAwXscT1n0RdmEZTBmYpjC5SoDVlAfqy8Vpfb6Y9oXGQdbp2zEDB3SqCVQOrJT10sbh2uivM7m+L9xMhJvkV40WQlApt58xCk4s3zo47culo/aDVJhW+TYOYiJt5gMOBXCn6WwC+gmbeBY4mt3N//BH3705z/+xeh7Tw8fTVnhVK7U8YxjX0LMk0BZZnUdTKkZUaodleSl5+QBBoXwXlRLvEXMmdZQZYMvY41HQI/CRbY8kob1VfgSSGeSEzvwty4TEDSIHD3S2Ljq4hNS653AotGyTEUdeOF6n1/+XbhcwlVyK0oYY5yJjLSgmQsfKQL6D2+UWYY1dNL3ywljPBMZ4ixMP+0Buf44Sp9u7tyZ96eJP6wkHT+P8dmaB2nNq+x4yfTn96r16j9nlfDOzf/xm1/66ps/mnkx5xT258PxrA9BEA2QNAJM5bun1oiGuz9grgCxIIFVDdhZ2JowW0wtVnDCj9HGDGklRVBpOx6RIaVxpjyFODhzwEdYcg7MNAeC9KdGJmyahuTAuVu2HmFB4+jyqXeCQNG/96Aa1/Dt3szD6k9+evzN77z1h//oc3ef29nbSYZpL5id1Moldu6YsJc7ZTiAkz1DqQD+qodojGuGuL/CGn7qa43Y1bd68Xd1CbNBG6DGXDjaEt0B4nOojKRg9Yr6RhXWpSov22h9o74DmO7l2l2ttovAMuEq5r1/Xb+/d5qLbwX51QDgnavhMkaVt8tJRlDgZ1RHFMkycfciTwJcLlsXdo9FjAvwleKXo1idWySz4nVzkS5/90icmOYa3lhJJu+jOagryEZ1RqHhHxgN45DDKf1iSoGNZD6ScCTckoVHOfIhY0JbCZKNGxx0FN9IJBI/1VAX3VcH8gaoQGdh+Bp1Mo0zDul2EIIvNFhhPdCgkK0dOXY7YfdX5m6m2ai3GA5vdmo3b23c7gQvbCUbZZ1XV4vr+H6wtV4pqO7s7Gy3WiVvCo9s+lk76wez/mwx8PxB6PU4gTtiy0O8qCajkO35KDQaxtNBPKuyliYuz/6f/6f/9f/4l9/7//77P3vr3pM8Km+Ua5PQn06GHF2EIpj1etNsUE/8To15yulgNORMI3PEJSf4JqIyXkwsfRDwaKP+EC9otM76BTjaF9DgXOAFxAmOwqW6qLPcHxkAMEFP7NNdxHApnVIu78Bd/UG8Cc3qUJePuAi9AVUVByWOGtFtkEKAKh9jJraZZkUCkIUZG/vSz1l7T2q3DXid5n3FnmV2kKhI0jlG9Agpaa4NraFx2u0ANKDiNIs/iqEuFCs84hnyCjrQarhg0TjyQlJRerkeO8uoMEifAyUl5C3f8HWCvUE7KcoZB55P5tRissj7i8UgwNgpSIi7shkhu1eKB2s9MQXSXPALCYcpbaQHOEqGXbG+GV273eqNdkrh494h2xQOWMhUrpanOGmB8joTQHOOiJBqB17skhbUC9SN+mmACJH5Ed4LmUlmDEPGBPkLaRzxyrpFAbRqmJugRDonESmJyBi1w4pP+PLl5CdJATa0gZQELxX1vi5ydnXWJ9RQ8FQvFJcqc5aXOKSqTKVSzRpIyJAFZ5otesMMi1TqnfiVdNzw5wmYXGqGXhVYQRy8cVouReloRE9+8sbz7a2tnW/f+NNvf/lH6esNn+3IkpPxAY4FDC22AR2PR4akQFL1sYpJPqN6DqsFOtXVVdTqjMci9VyRLOPWpKd3sNkUHy1dlUlopBmE07eoJ6ZJAgtSaqae70SdpClTiPlXjyWJsoAw5XBNznxiPXBcqZTDr/z1D975xebHP3r7+ec2KnH7dHxULuXValtsXRW+dK3i9XYNzqCDqnLuekYO59L8th4g7SLLyJWsncaOh0mevodiof+o51cchTYIXQCXYe56dcCs9ccPFF5BSahfhD9oDnyrT1SNZU1USRdp2bqcdRcrWu+Ns6JIv/xkFecel2SUj62Zxd0FVmkFFgvrXuRDwMUXMUX6s2TSNlb1ttc2DKikZiw1DG2gigkzIueod4kNFiUFg9VFpp86L0e0BdE60Fu6sS7ZU22xHGgnAJCP1vIwuBne9CmPZCPuou84RNbwE77LC4gOPqmpXE89rx6htDHnKkxAiWCIYeb1pttB73O7zU+9uHur41+rzOp+xsF1lSQc9uezjRgSXKlMatrKll16JvhA1srJcOj3x0wCZZWItUYc/LLIx+zXPpbhUJrTaD7shdV60mhuRsHj44N//umPf/b27a999wd//tVv/PTh2xzz02l1BpPTqJL4lVKeVAKOUxn1mQxm+Q1mUBPtWU4JcUVbxGFpLI9dFptSaxFeqI1ard6xxro+EkBXF69MQlk9r/pUKY3WLztsDZeU2xIHhAxStvVDCY6du47QgUSawXWKt8Fdso8WaKg+4CeMCQcrNFd+2ZNaB/3q1CNZ9WF0TArzLcxSNxVpspi2SMIYUeLoX9tV06GiK38NA622IuZcFCU4aHGHfKrIGiFFphC4JjYx2QdACurDFhlGKmkGLBPtBQlPUg1s2DQbiOZ4Ph/k+XCBaqXNKaEoNImBBodGqgjBKZYZIRrAe3VhqYaGg5VqAitrov3r2+AaStPbs3snUw6TyOOwnE5YD6tUTvuigtB6Gu8qL9wHtsJLzQTzA/ZqhBga80pvgCt+O8JiTYIjmcgnCIVLlg2q5cadJRQoZfrg4vDK5ceWAzdJqVxrqEKEwVLfSq55z4s8qQ9JikCRXJzMaq63Fms5O8lA6WkQBicarnobisgnIBvS76ydrY5YCe41Wxh4sErIYAUMJtqRBv+GxWR4WiqHdzd3//nvfWFnq/Pf/8Wfvj28fzA+aXCsZBj20x7qbzluYM+iva5c6mIVEZCtao6jilHqEoRFexy31i9SnhiEdQodpGSiJNZWtZomGPRICZehDWQBwbL3drdMTA4FA5QTgzZFAPUm/E3GnP6Z5+VOO05iJp7uvzNik6/R6OadW9utenvCZi4ZW06rD1RblbcK8/jMy+EQr+0rd9e3Rfwzv3xfL9RtV12ukhfeUGNK1gSeiDuNYZIcLMfXmx3s4MUSTdQo4GSdxIMTjVwpq7LU/A96Xa7PKrcPmpMgv/pGAfdIbmvxitSjUl4NnytLJ7Jw1tDnq8xXxZ37tRJApmX+RQVcjHu8EClzp9UH3DPskYe9AK5+UXEMPH0CcTJiI2pjQ8AZUUBYY7UqQSs+RDMdPzZcF4GCnPCJaITlR86uI1EPpSGrdPJX5tA65LB5LCFf/v4ye+nOmj6T9bE7qopgAPOCMo5hSCrn08/d7vzR862P3W40/H40OQoGJ/Mxyx5KG0mrVKr6KElsaIiNjGUrWC8h1Vk04UhUf4KXVJXhmOXD/mR8eOqNZZJFKmeLrGzQW1SqQTWJStXr5Y3T8Sn2qn/5qVf+8Sdfee3evf/wV1/5y59+ve410glWxaDerKJCPu0PmUyq1DbnGTsBgcB2GpLHhOKE5fw4Bmt8O7ECaElyEcU1grzsVkonTo1fXYoxWgGIVnGuX4hXzJWfnE+5fFrL2VErrXWlBvKiyvE6WppDTTWUfsyf2DC2ZDgw9zzFx8l0X7yfTQ8GDag4BmWb+o1ZpVEO2QdMRzohZUhjUkHUkDulq8ZCJOQmaaLSURGKpICbFs5bupxlctRLh9uwi4fIKURJTJNvFCI1/F5cE7zw5mNxavLXWuTeYt7HYYhtqAEMRYowg5tyHmBqHrsaSj6viNEqFDF+VZ9Wa7OOuFra3GmMBluT/ng+PRh15frNrAglqyg4qIaBsBWfHSQYaqJ2mZBE63jkDo6C5dzpVZruGi6OZIBgTTltYIc1S02sBooNMrXTgKRPxBxcduS4uoq+d3m6aI2oqxDAvS3u7hNB3sogULwqAutxRbIioKEpPOQPsCFMUHlA509mPWQmwIdfRC1La82sHLS8qFaq1sc6n4MMAqbWZ6cn253Wv/zMFzcr1X/3jS/99S++NWBOapEA1wHzGTNsFKCB9ZrZeJamjiVPVVdyOaWYAHXgQ0n2QI9/9Mqyt3kSelBX1dcQhu4GC0SJTFpXM9QbOA0gkkKUSMk/SgBRQSezEDukRfGQdZw9zJHs4EXDsFrd3t7CCH/w5Pike/T40fWPfPxDO7ttUEQaMJfV89wNWnPuuXhQkcvKLHu+ePW3HgDdWIfBHrwYiaLQr6JBhOEEYyMHQ9HzAocBVITcAIZVB+uRwz5q63DHtfQZXoW/7TadB75g62KKQaXuscu9Aj/e+yLteoILj+uvCF8YUSR2MdzdhxcSXPicRxMhFe2AakKxsVVXDUn0YonL5qDs0iEakGSsIpgscDZDE/cNn224aqyoNtz0uXKw2qoUw3KYtTLRGHB/8tqBykpERZtmtg5vVwik6BPKEd4xzNaIOGtFCW4gObYvf2ue/c717Y+0/etBP2Iqd3yYDw+np1229A2iRhzXSkEZVsJUIZPF0ECYQzavVTBhxxzJztGls+npYHzYHRye1pEf2H8OAye8HZfmYXfRx3GW2ctow681qTTORfXqnZev36l9/vdfufZX3/r2/eOjd4dPhydsZ9jZbDRH6ZSJZ2yONJuzBNDdoSvDfDDM+hhpZfiUewiwRc6kpRg22YfYmQnOelz9JdAKtpdRhUijONzsE9fLlk5vDMKWQHkIsPSvrJ2uS8SAkOdgEgCCKKsDX/HrHOUIwF0YeWK9rOSyXTgktejiRIVizw0aAK8gG/KRlW6B47P5P3O8EL0kPkLtsVyoaJE9GK360JiqNU6mQGx95jBjuGEKMVIcH2unbR6xG0qbZU/EJWujNTbkzSmstMBiIfSCE/uLXuANfG+oOXXYuGs63JdaYmdeHlnIfqOgFjVhDtgwXEZ12DmmCs7DyDe2atPbO0gX99PD7uEYWQyCjrOZSYdyhlBRgFTwNeCvGANvqBfiCdVl+Cmd1RNIgACWgUYlqj3cBaRDcDRuLZJvWjAAozu0tYVafKnXHR6QoCh3PUBbVdwzLodIvDwLLOuuD1zOTg92TM4lo/vIlgKpqfZjBAa0HDoAKtG9Qh8lmc9Ho8GR5DPO351Psf8zFRPUIqZlUEfbra16ECOXZk+Owiz/4u1XbrQ6d76//6ff/qt38ketaCMsNQ8nT+IAixUrfJS/m9KXxV+SvauXgzVwAWHEYEEhY730o7OM0Q63MBf4k0Q1J2DMT2KgWiEKRrM19atmCztJxn/7swfLF0jSKYxKPGAQEfDxYBCko/EwqFSgCSwnxEJzMuif9t4+GeYvf+yV69evnZmgVZTrexXyvi9hiFr/vj/4ZQmfldVVdQPKDBGOLENDqCVRp9ms1WrD0TQ8HRz0xiA8Y8ZgRqGrX1d+UYoCazj1y2pXvHewWm/4MvwBMytg7j5XH1hLXXOLobKq9XvBefWhauByW6/elTkUCazYZdWLSAIuTwKFKdvlQ/zyLczAvjPkFoV0A9OprXoCnUFqsgJ/RYjwghFF5RI/oYUysHHZEJZhGUTXM6ltFDEeZK4kJWK/BjJ0k+50AJKmQu4SA9ST6m6NcWRUbbUswk1JSqC/fBbO8iSd1Txvs1zd29h4Lkle6NTaJbTaaTIbVlkoBE/LBkxB9ftPJyGbLLAIQeSczdgTKDCbT9V2vQXuOAnHq81QeY6O0t44mJhKNF9gSaMegeyUw9IIXsoixnIw78dstsGE7bC60Wz87rX6C1svfeha48lo/Oq797/2wx/97OBRUtlsVxsPjw7QuhFJRLtEr3NMbce9g9HkNParkh2w69Mn0A688dR2mygWLJ95ATADjWAqsqGOc/clLqlrHDAtDwGYfHHzMfc9fa4RQhK6WSxTZMdgTayyVn9gVBCrW3Jfbb6h44ilB6MFwHcBPQHrA9n44Kwl9mFgVzDOt4NYhaj4rADW7hki1+hJmK7oOPlam78dTRVApO1IpJOGTLVUJ/BMMpyrHykkfkkXEhDRYFBV5FDMaTY+ns/M5oJTVIBT3/O+p4NqQV/8f7pMAGPVEJ9Ae9YcM1PTKh9lWvudwf7Etpl7lXIjfAcrsTTOU/gs+2hiUy1X/K3txmQ07XUHvR5zvngiaQ4GGqxM1J3kB3vXcDFgih+rBWoKGZqgw6uV+qsBpbqSCMZAwYIIKCVzNDMSwF15isEZRxBD0UCz0aEMz19u/Lo4kgkyKorcrfTzid2TqyQJioDi15OvhQkWyYqAIldsjt6nMVAAToVkmTS9TDfhSzHFx1nS5Rg7RBnXYuFYNYrK3mwY4nIRRvgNTg5OynP/lc3rye/9Ad7l//F7f/Pm5FGQ+R1vS0KemoIISAC+S+MY/RKUqCykRHWweMBpAXU/gAVHwCJLZkBWD/HGgU8ClnBJnBhCIvnG2LAZjMmVhPpPZ1qfkFbiEHVAYJPQBRIxqQJvYisvVGHOOsSqNuh1y+12s9Oeet7b7xyejn528874jAGrxLXLAVGVPw9zF7NM+OzOW8vptxuUNUnE2U/iqN2sN5vNMBz1x9qs1/7odgJiw3QGF+0yQrReK7XxV7vIy2Hwe+Dxe+dc5KBqGbRJT5gn/Wds2bXKf/V8KVP7ZHnjJelXnyyTXnhcT6OC3LX2FUEX7z504bOUlh5qqWGvsG4yHeoCKBI2LcQNRFfLKAPuy4oEIb90H3FTJZJbCQPVBFOSiAMzNIg326/P6g5onhLjJsGAQe8gOQqD8J1nkSg7JQeSJAcvq4PIGBnAdJGLc+ZuWzjSzrLqLO9Epev15p2d3ed2tq83Fjeb3XI8CIKsXObcNM8bnA5H6fCkxxoSNjaSUVn9AA3k6CPIeBl/XrZ2zSqYXRYMq8lpF6/OEvOziL00GqOYP2UzB2QAVRD3I5y2kpbHFkpUb5hlJ948Kdeq1S++sj8Mki/+g0+/cOfW/+u//dfv9LtJwAqXGKWatQ0qUWfXwiX6T44fH5w+6TRviJsATNFwCL7YpAAplrW8XE9ZH1JhwcHFrN4bXulBFMe60sF/mUztXH0CQxW9Ur/xlXiCgcHea2TR2dh8oVY2ykTu4MFSebGzcU+xOfOH+dkM0aKSJKbSMgKLrGkjL3TpuMQf87ZyjhOgqRb9LAZMSpRZVUEjTH2sGggteKBQEXaEAVoJJolN01wIpggon4tRk5or0K74Ib7MgY8/LidZkI6a46UswyXUg/ngxWLoLzj0gs25KADNFX8SPhadxdA95+RgTcazSTfb8cuGRtVojNynxTnkqq29R1k/FS9q9ajdqeFW3ztKaT3TzQIVnEH8kpTUTPWilW4okRHUnTQaH6A1f4gO+kgByqABfMoIEf5oYYyvnTHln8dXNrJgyQgOJLaxADz08drlckNGIk5Z6scwQMC0bPRzxQXYXfoicCGRI0ZCGo19bmqdHlYfKgwaKRfDGYEBv2IYEGBjjOiAJKCXsg0N52Jl3XhwWk1njdb1uMyWjkMmEsscO8WhB6Vk3B/i4bbZKP+T3/mHrebmf/jWV18/voemfOz1zMsOIY+i5ZoIeom7Ay1XvAHJPfAKwNGtNo4ME5WGgWS4vsQuUyUsrGorHz5C7CGNsFNSkL21O9B2ThJAEqwjZ9z3mGbQ0Q0IPZIAIEvw49l0PGL/aAxh5ajewGP68PHpcPi2WPqV17KrDKAG4VWqcw8WqTqpb1cp/lZ/ZSUydGb4MZeGHI30IeqEesBgWfW86gTJpI7CByGiRrWB8L1w8G+rKUV9VmAHU+h14cr6pZGw/rwW5sPiWosmh/NZrN4R765VxGpwFs8WIE0RQf6E3Z14AtzdkINCQ/KZp3I8GERf1VSYb5WW5qMwq/PFs9kmmUUAoCdHvdJLHPKjmRalYCw7lk0Bc9AZSsusC4qzOhCCBh/inN2Z5uf0nghGjLQEq516k2ju+Tyay8cymU+rs7SWpztx/Fyn+aFruy/vX7/eabfKbPs/KvlDDjtL4AblxPPK/uETNsWACTfiymzCUbXauhmKC5lgFCXlchxOkmlSStlFT54ks/5gNhgsRuNatU7b8gx1B+RjTOvgd0zZ7BeZHh5Pja2Wq7Je51nE4S55KcUMtfPSR//RJz/+4x++dvrN7/rTvO7HPc5fw07ODjw038f/atAdsML/wGttE6eZaCBmDAZoGW2lB6Q8GeiuuKkH3d/aS9IXPeii13PgFT0h9gqJFz1X5i4fvdIFu4XlwOokTmHbW3JNVjrJ+Zntrtj5mQVdAJB5P2nAoAZ3uhYhTOs4pP6yr5jMH7qDPsIg9SQIYHkukcGFyV+qt15ToqsRTFjYaHPG+pGGJQqp6hlDFumUIIKVsoR0lXDYnVFPugiWSQfJUoHfDwtQfZ+VSNQWThZLSoCNQUDFLWictGqOoZLyORcfBD8xRmgtEkubmMNk91AELVbYyYV/Vm9WNreD6bgLA+Y8AfCanfylRUteAgp8ro4DTR1kXbs02mmWSJPo/TJs7lcqjzaZtqVRRyEGG6KFbOon5EPqLLaqHdcc5CyT4qYMuGzYig0CboO16YJFqosBviLlxdhnP1+ZnpLIhIsOomsQW5DHgFiS4Hslas1hWPgz53k/nIw06cOK4VqnXmmX47rvjb0pFv9ywDmFaRRNguut1h9/6nM77e1//5Uvf+Xht8ps+Cr9CuRgTVGIoUWkAEiIfopdUnpJtjQDt9XctcegJLGBz+lgUIVRK+JD59BRpIS4mU1BZ0uDwCI/QJypAGG8vbeRxaywdF/FgM3QEvzwGB9gGFhCByE1R/N8eHAQ1Op4R/fuP/BYgbSx1dzcSnAzW3aMijOxz/UTmRlyEw3glPeq//TiyosEMI0l21BnG8y1AE84ZY/cl8XRr5byck5XIY9SFfVc/0RzJEzL4SYbBOyV8LTbZQcw1N8+TuDyvpGQiz2eT6gMIFQ+7t+qmVZJtWgVoTTr14Vyl9C4VJ+z+FV7XauX9xU0XM4uT4NWAEnnkZFvF4NMF8nYVo97ka37UGTG4qzaDkn4+uxa/8Slcbm5z7mT1IVtzYf17OqdcgHVJXErjbu7TLiTj4tZJV/+LmkBmIvBFDqguksKpf6rhqBASKGi5szTJ4u4Au4ztTYZBfNx7MNOJv5sxCr1Vj3utGvM3qJLcUYQcuSTJ6f5rFmKWsf9fqPezkvhyXjaqLVPxqwkwagYpTi8a7DEIuiMQUa3ZFlo8qyc8Ze2F+lW4HVKpQ9du36zUX5us73frHbKs8bsKOEE7mg6ZTP4bNKsVb12xTsePjrp9sYphIGDjMRJdewgO2Lk2rFmHmTjUT1GJ8LFVUt1WfGXjyZsTYc1FW8jZp1j7YQklQzjazrop7hGR2OEBogi9Rxis2QEl2tetT47Wey0rs2Oj/xF+R98+KNvvv7gh91329FedzFuRs1e73Qwn1yrbixG2dHoyf0Hb522dupxO4kac47fYdlOwtayOLBwUh4LYAEvFmBHFtQvhGGJl7uMHhTNsJvrP3pW6qN9suxr01i4SRuEfGhwkLNQwiiUDPJkIm6ioaSyjemmTPLO2LEI+zN6oTNH212cAxXOYQG7jmB8ZltHzouSpxTmQ1ahYLoHhEbZKFZ8ysMbC/ZH7nwNqQAtiQSKbMHLyGCtNYdRBTFKFGyRMc5XDAyboRatBQGlx8gDEaxAXoGEQuxxl2JxtVaJ0yJMymzpywQwG6HgKY8eMxXpJSe5XGnaAyDKkU+GaLE6OZxlKF2ovmwsgeImZ6vZdCLwkQve6sIF2sL5P8nW9gZLuBHPsjH4DIEH6QVojWDxBXWDHoG/8SemypWNYow9a8jxAjCLbfExls0YHOQ7GbQlBei11vVx0y4mAIghhtmB/bj4GpRQ5hrOWrRd5tBoy5/7r3NRbYdnytzqv+RsQhDkNfE1RxNcKQZqgiRHclB90CV5xhANnuArQC/RYhYIwpvRjYcH9+N4UOL8hvY4bGyGSRMjMDbcPK+WwjrbxE2P5p3NrS+++MlNv379x9v/86tf/cX8MTBhYA3YoR3w43KcDdgvFj9cSkT8wwoMymAGoXRGoLFb1c4aIvKhrsGgInALo0WfhPDcwCLVVrYG0FQgh43I2oFBhbvhhVBDmclsE47Z3EueYRBD6zi+xRA0YoUbqyhYFo+QgEiO+3PeG4+TWv2ZGrBK/UAXlaCCqop+9V8wt9pflY9LedWbDxbHWgH5XLLBDgfTpFiTFqyUH+C8r2M3JEQYABlZ1AlMlVzhCigq4Gr6wUp9j9QqcdVqCzggGDB0K8p1MY5KEmnx+nA9zeVyLNlZtH3nvtWH7kUR4FFb6K1dFz5fe/OrBMFUxjnjDQqDSCXzKOY+SYkihdIN8LxZaB1QzGqRKKiW5tOje4GfJrFXj4NWzW/VggbOr6Vgd6PaaUWbrRodOBr2cDwulcqjdBP3psmi/osHD/uT9GTQG8+mzIV65dIghXPHnFzPHjoQbDuRQHJoyB4OaRZnk5bv3Sgnz7c3Xtxo7FdKexQXzVphHk8PfFRbhmM1DmrBPEz5JkTLRYxFbJUDEaoaTrt496K84QvFoAFsEDuZxtFs4XrzCZ7K2kQyn2Y2tQFBDBB5aTEVZ4hD9jgQmC1E0sUQAsySlBD1CHhB6VGtsF2mM3yPkrC119n+7Ic+3P+D3P+zv/7B8I07lesn3S776DY5/UgkYl72/V73oHfydLPhsw4SUgXpQgZA9JSYzxQjZtaY5VdiwyYG6DRiWUp/jUt4YthkCMNtOa7lkSRJjP/SekWutK2VhqDmSiU8ueW/8sNCepFmCKRMART/BlfEL3Pp8MCIOW30B213IU4iO59UF41S46AqCd3OiCHDV6xJpAWzljPHwxyVVjEAVxOk3MmCH2VEAxSrfPUO9Rd2DX5Cl2kFPYa5goDWKKPvOgohEqnlQhAM83GGqDK9R8EsoGI2g2gmcqmREI75YPCEhWOa5qO5ObtnwulhnBTDrpTiLAKi0WYRIp6tUm7SxXgyDRJnWs5e8iXNIQ+Iumm6AVKdSkUiV6X5nj8jZq531EKZiCjI9RBZCe8cdwF6BhzVgW8BHel/JaywagISYeT65eKLmOKxCBSvbAy50q2mahINAb5SxqQrAWcGzOBpEKXsJjbSPlRp2Uvj2gakAwlkvhh5WeSNOQGS/SxbH9vav/GP/8XWC9f/3Xe+/JN7P2FbjMhrDj2W+qTluBYib4n8C99EczlFUMvQ+F4QthtNsSBPDG6Z7sE3gV6XEN64L3ikXgR+Vm276wt4sD4gRAuEzUI5fQmOKS0ZID2pADKnf8ArmoiyyFwA20czPNFYUja7tosauoB9vAovo97fDzmQi8vHwnz2XhkVJV7Intq/74uu4wgVmYugAhh8MrwqIAaIKMQDfkadEFbYDXAECBsCq6KtvgYqlfgByr1cQaC/jKQ4u4oYF+DuruJb9wjRdECzyugl8UWay4EisUtWPEq4s8vFF5msJNCLORUJLrwoMnTxPJLSRa6nXJbieTV2IZfFFWVHI8mIimbvWBsbzbOkxA6LDAHxoCDNKsHwxe3e9Y3o9rWdve1Wk32ovGmlNKvJq2SYhFlcOkUdyROxKNaGzsJy8LGX07D96Oj646Pum/ce/uztByejw35v0qpv9acDHJv8sIrZUIKtBz+Nalm2EcS7G62b7cbz7ebtRmU/CVpBGgyPonSQD/FaPs7YTx/uFmzis8eAhFD6OFuBrGwnmQ1Rx33WqKRpCY0OtRZqC3nA2QZ2AU2WzUjaCQtc5+wrMGeVKFvhlKaQCygjZBwHapRg0RShnCAjORr2KEOxzdlic41ZJRxo061sPhhvVHb/+LOfD+e14M/Ct8b3WDy5HbbhTKejw6YXsCj54OmDh423OtuLrW3EjzqIPINGoeeHITIngMLlQ7xphdgm6as412UKGHVQYJWm6F9V01La5xdxz/BHVJe3RiVpB0QToNgfqq2mQrXcmobmrFZGFuboBcKoh5oStnXDiDOAAQszyIB6ieDEWiMoGHJJpLW7tv2kOoI/OtJMrhQIxxEPNnJG9wii4gFwX6XR4IaEEhD1k3cYtFbUEkXEdBRiwERjvSqSgGPDQEOigJpK5eEp1kjiHKvUSJKeJ4mG1moGFv5NH9K59gd9UafCgOXrnbLyk+290WpmKRPGthCdvmHTK7v4jCaoBEQL1D5VWTSbiwpQcx7VQGnxy8u6SR0FxMEgjUBqq0sYCM7pcwQWdYM+gtapwQKPWg88yFnDUWaMZUk8OmqMqOOKWf6sCn3Wr+pZoIcLAD9rgCwVlGgJyF9h9ZHQXXW1ApdUiZoqsVg42MNbhFzubmaVyvIx3xGbTViZBEnEDDUaZYNaPqxx8gGr+8I6yaohZ+2GYJZ/ikksbLfbf/zRz+ztbn7pqxtf/fHXj7xu06uNFmzFPMa1L2Wm3+iCesJOw9LUsCQsq+nqrnrzn4rpR09qi4MOXcYHLB/TiOc7BDVM0ORAfyC3gcF0DzyUwS/pQbkI5PwK8KZKy01YEcqXVeh8DaUsMYCZo/EnyZkJWp+5y8C6erj0+6y3xANRe0sfXPrstxQha6eMGIvESaRgBnsMsR0MS7wNHDZGaT1oKXnHiKDVZb2ShE00LSrpwO8e18NFgrOAa/IyqcTRFcqdJVGoKI4AV/GV9K2zawlAfkjjPjt7ac9qir1aJjj/2mVbZM7L9WTr8ee/u+LJfXj5k/UYwgM2o1mORtf/GA7phLwsjjYOc2hSij8C2zs1G/F2Lf3U7fLdnfjOzeZGozIb91nzE+STagJtHg663e7pCU45nU6nzDFeGfu8hf2jN4PW3o1O5drGxu1rtVfu7j466j85GZ4M0ieno+PTyQRDMNu8c6B7GHXC2u/cuH2r0ry9s3mz09iOSpV0GI2OS+Pe8OR+PsMgeDpJT6ccgTCPavVZI9/iNN6yHLArIgwIcFKsMxYBhewJmWVotZKQ6A7co5nJ85nvKGmRItIrayfgOJBi2g+lhlLK2Ytl6VBZEUShIjiHqCwmMmMREzQJdIX9goZsHcvZqLCHSS+dDnu7O3f/5PP/sNnY+G/+h//PvfEj9mIMg6jpJZzOMJ0M2NHvyaN3d2fRLsuGawj1dbQ4ThYSLzRkQIyQGmEXMXAOzbsQaaNyvXcVY/RzPZJPuHilu9FQviSBPSohAWiK9F61gEJVLmGZ3ZYxikf6kCO0HK80KyoDglRFAw5sjC2tseSjmWhDRTENCcaEbTLYNGOKx0zFICVzFaHhzV21EglzZIW7cV94rY1pQCxOo4llI/vqJlLgPIVMpJJwF/BwoUI1sc2flQ0pyY+y1AxrHD0J+yRromixmXjxq5XCSzHCAhFmKDo0FNoinV/bikjAoJE6uJbVNOjBAX86c06E12BoBFtUnyJVpvgTraI9buAQoEBVRK0R5RbJdvDXXqTUXv4RiB0khANQC2ygYs2Y4ZXSOgjcsi1JKHgGBgrF0MuWvIQ8gSZtJikBkqqv9Z2yvfKy6igZbwkXAb5Sba3Uc/GWpviKgIFReVMX1xyXFWE1X8KFvdAT/xFRJPDIudEfY0lKx9NRPhzPx5NFWk0Hrc1byBYMIeYuoPDa3+XksD84qu81v/j8R7dKSXXuf+/nP3mQHky8ScIKf5BPwGQ2REonbab5JY40meocBcMb/awBwQUNYnpjOC8pgWkMXTJHmVIrYwkv2XcRf1KBH3RFeDJc1OjBJkt7+V6jRe0yWAFA+lBiq9quTuQ0JdjwOfukFfSr3hj/SHiSGKAxUBoBneusH85nLEH3qutZ6a9Ka3HyQoe4ASVs9NZ4UB9w2Bg1yw5hIbhibOTxRCnUjccPXNwz6uFyU4MN3JdzLgoi4C6XE2EXcDkQdoEi3r29fHcJiq9cAiKJcXcXg/DnAi5lkf484z+XPZ9zFRkSKL4q0i1jKAtUAgGZTmUPfUAcMN0CS5rEHlsB98LZIAkmrSS4tl1/4XbrzrWNl69ntbBXWvwi707y0cibjOfprHeSt+uNeDFdwE/xEtRmOG36tVprPH74eIoLUilit1/Zne7sfOyFXQ4AG4xnB93R06P+YJgORtloiIXKb/vJF67f3S4lrWrMASX+pLvIumGpXyoP/YYz+4T1vMoxadBnNrdapIPEq+IOxJw0Jip8mKH4DCYcYUuskQeYUnI0bpgHZJaQ+4QNN2T8JRaM0hvoBnZ2JplSeDNohrI8w8eXzXCRC+fY3o26aPSJGaAYce7DPBr1swq8Z7NWDls5W2yNs43axic/9NH/63/9f/43//FPv3Tvr7DT725sjY8Pxt5pJYEEdU9LT47DRrLBvBibCsOb4VGQUxy/UMTm6MH0NbZoJk5TVARDb9dfCtufkNN6lpgLXVz0LPHurcMAEjJ+RDPEgHTRHEgMfISmGj23BUiKEK+FtsCU4AL8iUeIPPGBJFNE05W+i/uVZu2hpYxdqIWIB6oZrJexK62XPzLX3diTFaxchGykYe5Xc8h8Ik4KS2Eso91ipaArEXgi6zDWDsHwQ4wj2CjEnpdOWLSFevKJiD13FcecBDWV5XlJl2S1pAqUKYCY3k9vq1LiG6Yco+ermvBDvIq08hlzSco0MXt7z6fjeDLGkYAmqPIGbcohM/dod5XNKwcgkSMRdNViGRDRtwJJaFUT2vEJAgWbZ7JPFoTOOgXfMGoq2NFI6CB5qWD6GwmEIlRlfsQlBGWDpIr7ZZdDBlIVAfcFuVFPiicesBHJf8LWOJXIBajUpUus0VslU2K9VxUFDAHfhdRqABziLsCaYJlM5uwU3sORPp2MB+xPGZU7+F9ivoqiRrlCOJhMpqXTEbncrXT+d3/0zz9656X/4S//w3D4s7bX6GlfM3aCx/8OAYm5orH2A8VgRhkCg6pBeWvyBwBeVpJogYskSmQ/MHBrJq+KeHF1QZV8TNRzjaLR/FljDELkqby0OTazWgII/0Vk1OvI7GtwUXlcBiO+eca1rMcVb92HKpUyn53sii9/9Sj0Cw42YfjSi0zRYCRDKIVs0mgHYrMMqDEayGCD2QUMaa1QqzM3XWu1eFb4LMk63C59fpaMkEvJnWTr4fVErvT1u+GlPl5Ppi5mmF2MVJddiHTF8S2kmTsJuFyguBO4fF3IxyUoIotMXIC35UrFBj4AZ0+4NMCushhFi2GY9xtRutf2bmw1bm7Xb2zW97fr2+1gd8ufTYL+STdPR81qlLQa3cPuo/sH8JFyUmceWKsttWEy+kpcbjQ6nd7J6fFoOMUzOWtvtOtBpTqNKo0XdrfH0/pwtAGVE2Ij4Hql2iysHqdtuFGIswynHvRm0SmOlOFiCunHfBx6eC+X81kN3xk4FtOrYqXMyrBNB2IDRlNZM+18bUzLc51vg++VVC2GDGtDMY9p2KElM+wYpLJIGjWH6nGckrb6xZodM7NMP8GuYSjsWys+SbPkvCwbK74hc/ZpYwMB9Fvt0F8Jk5QNB7J+pRT97kc/HaP9/9vF37z914+O77eRQrwEOycsh835uodPqj6nI5Wx8iCU0K90DQObO7IvC3qw5zIhTI/MOezNrgsj0TpOlJO+4yuuAtcJa5icv6AuFu8SQ1dUFp9zMZaEXu5CE0TlZedBAuiLUtIcE1V21Id6MVHA6iNqje4LQ4R9ap6UdmjhD4PYFm0g0Ug1sq+MboG2/EHVaKv4iWxMQBczv5aySY7HrK2AFF920gbksgpKCYYfa3oAxRcFBUohnRDvZfpbfbZsvkkICDMmZFGySCfvgSriE10s/iC9QmKGPpOWzBJW6Ax6rrgvNkVDAc0wzqc4oPTT09NsPAj7fRbYwJk1avkToxXRXZVrbeQllwiX0QfCClgvEFAdLQoYM7qM0gsKwII5RX40K4weBbz4I7WApErpyXJRS+wN7VcY5q20Z8Wp+Ksukroxvh6wrNQTfKH4VT2VgdWUX75SMS4gs65GFBHL3FZMjhQSKagzr41EaajhXOHJIQ600Ep3/CgybzpguUH6ZObv7Nyut9AYIfIjFvPh3F6JPTZx5RyqSrX8/NbO9ie2o7DW+vZXX3/y7nD2iGHLJmTsSsoBIYt8oklXrcnGWqRryUjdQFD1aZTqwa+rv8LESrYjIJCRwLFc9QvtAaDqDcQhfUiAJ2PYdJQYkjpdEXpkEkZPYBdZSFUVLPl/NgfsAOTuyu6DXlTWamW/BeYI8FfmRLIr4z9wZFjRJ5B+NRTqyDIsfiSSqPH8SchhUBuyEBQsl4Wsmkyy38C1BJ2avmyaC7j7qiyVTXjtOiudyF9aD0uzRHGX2GXFEHWPFLeej/PKWS+0CLv0l+8Xqn0hwyI9pQDaMUfbyjDGnmrYmTP+qqWsGmTPbbU70fj2ZvSR2527u7VmhBh6NO0NHvbmSaUMt4XX5R5++DWtO/CPBv3RbmMjD5IpW7Y3qtM4YUOV3uDktP80HfcqUZSFaS3M6tEI3ZQVvcPHR2AvxqpgNI1LSafZqVbrcZbOTvucsoS9GBLl11I/4rDd4WTcQzXP5yPUwxoLBcxtOsnZLrxkhG0eMmOx8EZTGAgLE8uLxRQ6pTVWUoBhJmi/rHBg7RNthnyLV0N3ca4FtWgHaZi5lhjILGcQs8Gz9kZl8DJ1xZAzR22mGpj1nOPJmsdMcJfLTa/SZkR7GM9hQexbMZtGtcZkOP3oSx+O62Hp3+R/9dqfwcxaycZkOiiVcAEd9/Ju2X8y85IqGM5fuY7XmNyCsYm644mYUEXpK5dx3i566pcG6GLXy7obdXbDlkdxWWG0NH1dirAmy71O7FYrjlhwI59gWC+9yntAZGoxYGEGVISKjepkE+CGyov3k/jm8vBB2PByDphI2VL4E6NckmwqD/+G4wirGR+sNWTKnXW9mmrX7JOUPt4oO6beYe/ixJRBMj4TfSTdci9UeCpjB+9qjSCGqRs08pWBu6m7mXxIDQLi0ibYkBNtlpTDo+Zz5c8AtGGJeEGDHHjca0qYwUCC6Xg66E26x96gnw1HnlakiATqsuLc2NQwV/H2iteAWG3TCNYrPXIpkakWIvAiK/LPRZMSNOQgrlSOxgEdgVpdBccjwLfKXbqYfhiny+LMxL0kMyqAztb9yotPlrVYBVyyIl4FcVkOyt+qvSxI7VULVRPMGOoCvSeOGBOSACqvNJ4k4JKPmBJSPPtv6yilCHmZ47GQrRkDXtafzPCZ48jQKiv6WY6EPwenCbM4P2XbjprcIp92k2bjix/7bGtj68+//83S6z94OHx05B0wQ4AinHiR5stxLUFoFF/gErhXzECVN2YpXNU7g5hLJ95h3NbS8FaAozf0RRFFA2kmviKaiBHroUNxc2DuwzoBcmDJ+Qoaoa6mycgAiIfL2pCTdYjgo0s/IL0VJ6dO9ewVupeltZvgvbyo3Qq1qcYyw7OULmTtvBjJ87PSX5GUJsFfEUHxAkH6hgegZqjaDjL2hdBU/axBKjRWt/O8Kl0IZG1dRbhi1DcGGRdwkVfflcOqzgRMQlbHuSKWBSEGytVnCQrkHxLoM8Wop9QQXUVxPBroTTi0V9ZYfcV/SbnQBX1FL2PupGegDpaPwhYvgkLnmwnalWXFLcuFii6zvfjDBBdFY76zSuqGPRMC4yqgmRTgK3cUdsH10pANn/wUtlItexv1eLtV3a432uX2C9dai9GTejC50Vrs1vGCnvX6p8Pe8clwsbt/M46qHD8wHLCFEBs+drZvLh49frqQP3OOilTfavlR+WT8aNA98FkX6y9qVdTWNEbi12n2YwzD7VpD6y/T7vDoKSpQw98Lgg2GQL2ZjAfj/qDPkhhtaBHm83iKh4zm6FKmWzN/gtUn8KfMlPE6Cxqt2QzLMrtnMCPD7gz0heYtUVZFXTjPUMBm1g+Sp2azxkNKD2QDDguEScVJeGw6OAUggHRWMXZkDrKk0HAH+wCophIpl7t4TDjGJa1a9ersJ81ayBgvbFgmK7RYsn/w6PiFvef+j//qfz/7f49f/8X3WJth88glFq6OsmGv1/VKWH18bOws2WJ6OGCzrTihGKx16Awx55KG7ITX14YSWl3j/HJM0QBHjHBQI2pjCKS7cFADwHBKQrqjnsIMms8noBQUR+qvTXvizSkjs6Z7jRGx8Nccr3A1heMKtYAPrzBFGUzIRugvNRdwASjRIHmqobIyfQQDFlT0x9gVrbaRAr1ReoXVYzRAr+G3qBryf5Heq8Fj8Xh2id7BgLWCWpO+agYUW40AYyWVyYmMrDBssPEhtdQ8rVoo2qDE3BEvpPkq1jxWUeWpERcuM+pKmiUlmJy0Elq6Jx7u7LhALP9w+8bWOfF6Q68/9FiyRPdYneUSoIoYGYaS8iSUIEJOSYZo9h4xUBq9AKC+QNPGSE/PCKRKa+2V8V71U3WpCENRtZdNE50SswupsLyL0JOM2quDaTaNVBnqScuf8OoqSAE1IuzueqmvjHTap0vyJEJKMUZ/DOdVmGUBngBGteyM6ZpBXb2kiR3uWhsrOFvZNmdBrzACGRW8pPaiXzSdHCUpCa/4LsOr+fDpW8Ph8NqtSWtr3wtrOG+UkzYIxAlUSHWccTI+OI7nnU9ff36vs9OOK99/5yc/ejI78Y4ZqhwUqVoFmJyA1hz8YARoWkjAEHyktZkIZBXTPvPLjuDXLusfwKCGrgEIYFM/cRx1maQ/wA5xAOWEXNb76ihFqBj+LyFMkynenMOsAGriyrAnEW7oCyWp4wQUlUtXajCup3Op3d1wehkhbF5drpF2X0VRsWXDDB9UTPFqGXA5rOdzMQXPfMXSTwYdY1L8h0lgi9VwNcQzxDCouXi1QA1dVU/FujSCIA/uLiRQlbA/nqunRZIK+V3jhEswUTXsEoVdtkX4JPgvPxfCri5ecWlgg7iiUKCj3pHeJWEYchGxHHiWHokcGZgzBtRQ2me9oVGn+jPaIET6TNihCkEOENe1ftWypcE26FcNtOrqRt7FXcDkvGgGXcoWjGylxnCZsYkL/qv4mEdxdUbxszlrYVABF/OjRnh6rXT0kbub2x1W8Ma3rrFTTRjMRo0yZtBxxtb0wyl229HI77KCLwtrmzfHixPKGwyGiU4B0nbqVRyuys2dvajVwvW3/ODxozd/9uorr3zkI3dvvPbTn7Jfzs7edVJUK3UIHBOeUYkNc3DdYvE+27J72zttoMThrFOvzyBnW8gFKjW2wPGIilfYmKU0Y00yE8WYcL3BMCyF2A5ZzJAjBYSb0+NpqaVaA+tSOvQxeOECreVDmYYMFmrJ6HRUCuUA+Y34yveJMlGOciTrFG7EyOOoV+3gzIY9HFPIWYG1ciWJQ1YVA146QrySvkIv054ec7YZ4FRvJqggVeznkJQbGI7Hx92k0uz41dnp4qWdF/9v/9X//V//D//tj974RtNjn/omjUX0OJ2c+Cf4haYJRwj0WrXN52zHpQyDPeeBM845JA/Nu7JxLZ8Mpv3TjAlhrQrIdbRTGA1lmpYrpxRHIbgwgBrK3xcME+JArXTsGlWmlfBIEUCRe1UVRzW2uGUBlggBHi5UXps/AwKdNAwbRo8X3+USDcVlDf1+ziIpn0N/sTkzdauNE+CP1FIls3CaBKK+RMrlVx5NaJJYJ0Bc9B8GteiZbAuYFbTRBG56OlgXNsT3MCjWZcUJNm7MqyhO7NgQhTipJeRNFnhEqTWi7zB0licyS8KOV9R3VKKLRbJolvzFoG+UGAZ4UrF6jvOMmZBQfVhNxy/7esnUzIbiiBCczaw9rfQpVaTTwQ22HsRzgcmBWdBlvnGAlKZhSFvlgI1Sr0eAjPu3RHGaSjmMaPpCU+GpHZeJcoc8IGVHrIxGhhNz8iMHowHqMjAQ40McJMa3jEuxalnzG/zxSYxJCosnlB2xmBJxFsJHHmKv8S+o0+li0iI3CtJwUXTyV5cLF8AJBfjhkvzFl4rXxbgjTmn0/fJOyxBnxUDUZL1VMbQNjZPBgxSlrlKmXOapY+RIhYIElq2QQZ+yzJ5ngqREhgO6OuyD1VjzU+YthpPZg/uj0fDJxtZ+u71Dzv1ZFLY2WNbIGu0auHN4WBnPPtSqv/jP/qt//80vV78WvHry9ok36nmTEWOew9CophDKq+MDKaShW9DBFwN6mnpLAqAimKRkuxYGau8fXaKfahqX9bqBR8+gs2gubQfGYBvT8TzCzmkpnYgwb9+lI9mb1V46io5AflVLlyZoAMvD6nJhicE2TIkGgsY/Vt2wSvl3/7vEE3U91xKrLMiNtjssUoQDnqSVqy76m2ghiAGOu33xG2uwKNL5SzEiY+cqSbFCSi4l1yurkWJ4RXL1iBLptWiW0EJtNB8WSR3LtzAHJCjeK+HZZe1SMy/XxyWCGhlWkB8yDZeYD4KqqxQfilTK+joux2i3+e/dan9oL25W83I42an1yiw5ynsh++5Phuh0AT6HpQo6wWl/iIszn1/b3oFj9Xo9Vv/CVrEL0yZIV5iUUU9r4eLFu3f6/eGje29FUbKtZcL1RqWMMwvsFkJSLVeoeT5MT3sn5JDEEGQBBHKGdwbUHYZdKkPQONvMZ6TiqlxFO69XxidH8MfReDAYj2te0KpWS2X2ssCKOPWn5oYLVx0POSHex5arIcqfwCsGsJRyoEXECnTSgngFVaVPYFGIJfJzFkFmgpdzjUbQEazDiziFG4mIiQbp7ARoo4d0A4OfIn6UIqmDEkPYEUJbRTAoZzW/1Oc4nXH2/LU7/+U//VeNefS9N798OB5WS42YvTCnk9PJYXAwKS04IWY6jxtMwrABiCgQ6zOYLKbrYJusSi5FlSr+WsFseMpmJ/hm0ZPACiwS66DKqpr0pCUVFLIZSi1HE3hFW40RG0AAizDQ3Y1wm2kATdd4pxbIkJsYGnBa3c0XFcWMC6pmvFa6kLRbOfcSoEqi0UJpq55QXRfFg/cqXzHKIKAxSEWqI92rj9HydQIup1xqohf7M3/ArxGw3YF2OkQfn6gPSegxi+pkKLyrmN0H5vypO2mSJAlaTa5Eau6ftjBfqPbK2EMSaXLATfOuhhMCAwSbKX1xOe18BFdHGKHbiZYspAPaUyZMlI3YJoIJQ5UuQBzAj4riDM6USyvhyhjJxN5tbHOn0dKP1FWSiYSMdJdidQf96GZawjdAg5wlLgJEpjg0By6co9owFIyi6iEZQylPJEHlqVcBOgC35+VNrx19pBAC4tqry32ojBWjehZ3RVi1aKcLk4/lLOZNOv1RlvxkhW+WxsoBsHZZvSSZqK0qyaACwlGQPmax15C9LwBSlp0O+qRkWcI4TEd+8+Ykx9zPpspJ2avAC/PRIBuN05PkDz70id3d3f/pG3/55z/5BpPGLR960+iOH5RqjQZWl/GIZasMc+phS7xpASSTBoCkcnMjK0gfd8mfqqRuBhMHAWpOzSS36aW9cU2i8upjvQZ98MU3fFUaEhv3FTEFe8kQZ8RnXEVPFe+JURXO91jx9lcLrEqxbJe48avl9Ot+xYAnCxv4ulExdy/yLSKLmPcIkNi9dfkUYQLFKxepMW10x8Vzh9bwyiiD6uCSEXBZWYzrh2WFiwS8sreKKALQlOLR3i/TnEHefb+6QxitIMrH+MYox6HWDSrmZYSjcmXhONXFgJ2jnttvv3Sndq2BlA0vm7J/KzwgHXNa7lhbQnm45i5gDaiDuFwNeifMU+7v7HTlS/Sk0W6BnqC+6BU8slIbjaZsK7SzuwWXffL4/sPjB9dvXLt780671YABs9C+3xv2+6dw4oODgyRM9vb22OcXElOtVCeTCeapMvPP8yyuNcpVfLg4RWgCSypVkkpS3gm84xJndI5SdmdGqcnG+bRUjzgCr1xiAti3PbUmKasUkA/QwURJxF0ZJZpvQ+egovyTzoKNWhQZiiLmAMkQA5YaJFaNzF/ic5gq5gfWL0K52asE9VokCfs6jtaVCJcF2//Lmw0Q89NsGC5qYVCNKvFi1lugRjG5PhlHYeWF5++W/uSfbCWVRz99zYzvkxaTnXnenR7nTyedDkO9XmnItUt2SdFhtqWmxyEj+GNBlDgSUefzsWiRNVbyCNKsk2ybNECMQIRYfMahheiPUA6qKSwkO4kcxgPQEOSJZB7ORJrgKxjRicCGNMuLrwGTcSfdl4N6jmpmc7SQN2ib2AIeWDjFoaSJLHM3uxAVANPtT3RYnwt/BTqaw0b88qFBbWFqXfqGmr3QypSqXNLgxKVqiSPt4jpnYGijLDx5dBgStcVvztiPuBJ/QmnLnA6SNwItUlOgkTKz6EIwlLmZ1umJX9FPMQQ3qS33O+m+k8FsNFj0cXvOSmO8+KeIQUBvgT8BhUB/mamRxi4FhmaVUKRpkYqnUQwRtU8JATcarXFTXizbzA/yGWyS1koFEkmXUZ1hSCF86ngqb+G69BfZItYI0yhL86mEyZheVo9qRzY1UogCPBUr/FVFLKjCeAAKvFX0pevqWEu26miDqnJyWakhvLqcITHFJ0U5FiMZRF8tOT0BSQlQEIwetAlKwiY248l0NB7Hw/FmuY3vY8jB2iA826ThlwCWA3VkoFnyob0b/9t//M9u3Nz/q+9/48eP3zwY9zbDDjYcdnHHXZ2VCww5unY6GwIl4MZMkLtEgiVKAVLXV9YjinSPBIhxICdsyRXDWw0sZDJoAsOdV8IqSZoGWcJqHf/pczBe3prWXItWIl2KcVArYOeAxV2mT/eFJf3lt2f0pcvw3OfPSHkuzW/zAVC4i0IIuLYzCosyiXThKyq/SmRpzgHocuL1fJ6VHgwqkrkAd/WlXUXMqtjl7+WyLiRwj0XOl946MkFJUDeJ8pqGMWIttViOTYx5OPIomvc7leD2bnuzFSY+TG2STYfj2SJh7jgbTSZ9mAHTpbBHrZdj58TIYyPkmTxW0NEw/fBOiyunkxF2Jk4g2N29BkXodU/eyWcbGxu1WvXh/fs//uHB8PT0pRde7HQ2INqjcT/rZaw96PW7KCTlSjgcnaIG74a7aLIYP8s4gjHxKwXLD/EtztjTPcNlh9WFjb0Ntq3SclCYMMr4wQGjmNOymTrWHlXaUA0ehdpDRggRKQYAgMPghDzzD+IlDYg/2aphtFBj8SqNOXEgaSQUCmOG62jXGypEF0J3pR/JERo+DHllP+IETpiP8Brx095iEIxmYTYFHI1GdcMrd6jJBJN9JWGDit7xIbaBuzdv7/+LjW+X//zdH/3kKDus+Z1Goz7sp93sNBhqr2umn7EGV+ZbHmo9+h9bk6lyGAvjkKkxFr8mVVzIULOZB4dxwoOMKmPzRrgw/0zIhciHLscDQADhP23FagrPgdsYR4IdE08s7aLRhPWoSXN4sBbEAiHMuWY/wO0MziDmKjEJCufMjICFvtSjqXzYolF8xF+oshElSw1oZSOnUmInWGNgtnLxzpnUx3AsZdG2zEblLZXnIdsz1MVx2bkF7hvX8Le2yWRO3kLtSOGEmk4QKzOdUjBQvqq8LMtLwQKdXFPYYr9A0LRVtUo9TAfT625qUHwNbsna37Hf7XJ0VqBNKDlWabrAJ5/eAMp48avu+PRJzUY9ZhKZQYJ/Pi+QlwRqx5fIHdxQhJi8mPeS/1oEw8RSCpYgEWADoOZL5DoMkiRvb6ni3HB2E2tmiKpwEsujiaoLhMBYNjGKpt1SRmUVAMPJ06mpZ5xScLcaisOYTqykFqPaXHXpE+Wm+1qACJMFdBe4HeVxkedT6omKSnoU+dV9PT1DiT5go1NhmZZgc83CaZaGlVp7r9ralXMGshOuHRi84goC19GTx/mw8tLt/f0v/sl+a/Pf/dVffOfxD09nbNI9BSdYEMEGqCk9yX7gWKN1nLw4mzFVmiFjDnUBWstfhpSuJe2kbmKiSx7sXvGW6iOri3QDfIYKcqMyNVJhDlmWg3pAWXMH/ZXructAQIzeUL4lWMatXp1L/7/QhyWkr2qdyMmvdLk813MusrIeFcFzGaMeFMkuBxxqFgjqPim+LarGh8Yj1InF33l8Iq3DgCUeGN0jDvQT08Dz1/gJc4gQclgZ6DQN54NKqddJkp163qnGIfNXGHJ1xJs8gMJaFC7ip08epfhrsmEMR8OwUJW9XOOQLSRZ09dq1OGaFQ4UCpjUQZ3xZ/3JsN/DK+fx44enr52i2oZJPBkNHjx48PrPXv3FL96+des5dlaajMYCF9xiwSEI/cdP8NBig/YIpRsOhhOTnGyrtcOjLtrwlBX8rOX1Fl2cgsNSeRFwFOCNna05LtEHx5Wt9oSzRsdaqBnjyctWu3Mt7tewhgFDhQ3vIddLKIhGy4QEK4Bf84HGp/gSVBOOLFsfdFmEncGmzzD42qpipIF5hiFSfwwfDIIp/jlsMHCiw7g1w4e/J+tWJ3OUYybg69dKCCXzWSVuzsgQuYJzeObB7332C41F+PYPfjhEWh+zToOdn/AFHZayI6ZfcdtGjq8FO8z2QXhhGPiT4xgGpYo1MVyRZgih9oPJ9FCmSkiFtFASQvxhnmxYBu2AytMonSgjlZTOM85EK5VCjt9cYlBiWqLMttGVOWQBNosESuxMaeKK+BXUHXYB5xRnBWDCZBlUjJMg4WAmQUZhWs5Yr2x+YtLwCjBQNIr/fMYX4sH0vXiF9uUGjNL+cP/mBCI8oku1AFN8Kc6Z+g2ZYUi0VTSfY6pR1kYqqZr4rNRyBhkgEFOWDEHvUXl2sYW+I3IAEEupXhUzVnpxX4zPcC2sCUygs7aUqfDZeMTCdaZZtDcLu5sOJ9lgyPF6ZtZE8oERspmMNSFK8NXzmJAAU4CFRDhDMNdAKuIMr+oAoZiAwEgDxGSi/xJBRBlIKYsmzBhhwfyDlEDWA5Kj7oKQAF8cQEA24g4EgDsgNru0CgDMIgxAVBlTiqiZpiHU57xSHbiJcxBlBbgY7s+6HPly97U0yspdRqzI/yxm9Ua/a/FW6jqrtvYis5hgBGpqp25QJMuH02E+vD9rDk/9dNhpbiVho4RvogQLen6yWWEXW2/wC2zOtc/deGn/j2qffvvFf/03/+6e53c9hlsyKqWDySmELk4q3lQH6GEhoHuoARgvyAh2Qtyzy3rBHimFerg7hIGwPSJO6lt6AOGTqoC7zsxsDTfYMnYAPn/kU1D59VLOwg5LSEdAMCJbXWf1KUJXw9U+LNKsBzQIuNbycvmu9cR68t96uCj36uZZ+e/xqqifpTFYraIu5LyeCa/eO71LUHxCwHI7A/Yqh2Vh9nYZdl8VDN7FugTryVbV1K/QA/SjU8Az8J5N4aQBM0Qp18yc2CHn07I3akXTTjSr+qx3z9JJnw0IYCxQanxay2whWfK3Om3IF3O9/e4pm7fSTqh1FJcHvcHm1g6zMTUOto+SahWtJarW4kePHo1Go0eP7x8eHj55+qBarcIOmRQdj4dvvPXG06eP2WKCr3Brmk6nbHIPBxoOT4eDAdU+7R2RnsqTjLPtHx0cPHnyZDQYoiXFLPblrPcoxGv6H33+92t3b3vHp6VKdfe5myeHJ/dff5OzvCcI1SnOLrNBmnImCzRbLFYSMAKElB9Q1Tx6RYehf6LGmg6VIRf2DLmGXeMZJAZtno32GVvnSwbWRKOXxXIjgsOiuUlYWbDND3PS7PuzmCdRDZouz6MxJkPWwAQx3tGBNx33sSHyFmcQbNjV7euf+ezv49v56ve+fZg+apcqeIaz5wOe0SmuWUg/fgo/4g/yRAvwTMJQC3vJ4a2i/VX8lDDY4jYOn0XKEN8RPZdXDSG+EjISaxKE6Aw8R4MUBdDUQSVXWH8wYPFgvKXYhAQRAfcrySRiVvBqfSbDCe3HeagUM3dvjskQHFOCoWniyrBR7YfFjxZvyeIiqx3mVRl/wET+yw9OVIfvTAEU+AmhONAUrNgwcLKwbSZpI7tz6IgGEWzALnu7dmFjukTIqe1S8DgQWzVyLzKLpAJiU113CcvFgEmzunh0rBfKTGKmXLVhVurjagbYmTHBd4pTK6cpez1MTkezIbuxlL12Ndjevskuod3u4PDgeMxRh1QJLArY20zsnHqIu1IDcT3KgD2ii5kZRUBa0kV1gjiixqaj2WekV0hGeo1WjVp9BXDZrxqNm65RvqL8CskmTQfQLqy4gIZNweQ1LQhobDNMjBYrFbDTC/uAO0kUY3dV4j0vB7P1JK7mxDtatP5qPewScCcSUiP8pfqA21pr8cgrSDRIT6ZaStoFD0HEGadY9Y6yxaSXb+5vb96sN3doBTI0J2Lgn47xRbu8T/NytfrRjZsv7N9sNRp//pNvfe3BD55kPWb5A4+VF0yuEWCdEjYqjDcatKoJ6ocqIow0CBFUh7haCc3PLseJScx7PB71iTFL8Nec78hQsbx1eQJyopSDY85nORUhB7KLgDOIFGl+A4EVfl0s6DeQ9QfLwnU/3zhsuPzxBYAU6S+nJIbELsGVydxbvQL/RePO+pJXQrFVDjy4/AmQXo+kN0w1VNBbl8TRLA1Fe0tKoQ5KguW2rAZosPzWcMWydgWcdSxkmiIwWcJZggzck8JJctQNlgJwTtFsXI7GnSRtQQDTbvfJMfpcFcYIeYUIawm2NqDY2tpBBx2NJr3eiKlVHILkLRVrs+BuvxtN8GWGLqfD4SCOExSNSTo5OT0hgH3VNNFZrcY+HNX2ZoejreBVTCJCbEk2HuFiMakkyXiSjscD5pXZKAc9mHMDmWKttJocaHRycmILYaU0JRFqebLd2Xj34OHW3i7qcLLVwt0UJ+xar1dix44JMgSVnWBNPMV/CgaFJxVMVpRRNBmoMm6kCiMtQ8i52eiBQRhkRMfgvTxqgNFNpgSTDVG8omuI0bAGrraDtFY9GZti1YpfHs6DmHlfds8W6WfGcOMGn3HWoT8rs+EADU/qW944DTrXXvrYZyAGb73xw+noZJyz8BqeiJsXWy9N5ACEEp+ntU4WVpulCrSn4qP549PLmmC4VInj3VivJU4xZplqiksw8j5iOlZcifzgAFQPDd/avVQVjfyKjxsbJpI6QvvIlNVRrP1zGyKLJZOAZBwUhAasMI1H9cKUb5OzUridfCCiJH4POmkrDdKD705JA9jSOESDTUfUuEBTgJgJfCoZ6JLc9tGSv5U8fLGjuHEBe9REJ3xfgpGBO8dbe8DJFxIXxJFR4tUWqCGiEh0li4RTe6C98Bk1jwTLS2RUtneZI1l0DLQ4igPzCDJaygZurBrnCCh8krEuz7TRA/48zc3wxua1zZ3rn/r07zEf/PTJ8Vtvvf3w/qPj4+PDp73sEGjMfDKZAiKRd+NtsjhQa9F3xAOpThBoh3mi1zSFkSxWLclE2ASckIwVMkqjSGoqWgAC4WumoPuKAIIDL0BBwM0r9QpitCQiyrPcScNrPuLPqI8xHmVSXAV9OBdbvF6RHSJUgpW3Flak1doo2CpNkaAIqEGihuoYVU3cblky24tRZa1XQ4CRrxssGdQIQWuU3eFswJyObdidJuUN9mplYqx/2mOngXZ7G488thlgkUNzo/PPf/cPm/VW/PXwmw9ffeoN08hHiGJ1H/7XLFykWCYhGOWybxVtJvbiRRv5A7Ci1auA0FXfiXTI8KUBT8fph35EQAe69Cz50jJ8+UwDtm+uvrnGu/sSguT1XtW6Op//VGILLKHCRcf/piq/jpdFnkQ6BkxxrkT7dZ2qOrgquVfuK8tnieIuTxcPe1vlpgg+gfcoN1DAriKxC6zn6RLoK/Fb8RSNf5bkyuGD8cRJYUIcOGBUYo3NsBqMmlFaY6X8JIS5UnCYVNjVasEWHEm9Uimn42wyzieTfDyajSYZ3DkpV2q1Wr3NWdr+kJOphwNWAcM7MTJTT3ZRjMoJ5K610dL2NjOvXC8nYXTSO03KVTQsWqf4uT8a9JlLoyZPnz4NRsyo1VlvNOj1x2MOGMbxaZ70u7ACpqLLSVVTebg0Q9NL3tPDgx/++Efo3C/efb5aro7wcQrmnZvXgv4k46zv07w3nXX9rAs7xrzHxA0qvwgXMNSYkl4kgsUcsdxalxRTkDKOwYCSNVNDjXGLGQB+LJlXxAjQaS0YPc1/EXGTbdizH59szjXMw/4iLjOT5fktoCB3sLTPbG01rFab7by/6B+d1ljEmLGXxdzr7Lzwe1+o1svf/87XTrpPNuIIFsDWM+ibU86Hcvtmsmc9JACDfhneFGM6nk4wfDJPWmV1WLm9EfXZq5JDWOEYohbiX+gJCB7ibaKVom56J3UWrCBCjVOexMgdTaIWNls0c1mhtQQf2BjHoJFEkpJMZri1GNdFzWUuDEAYGoF52jpWSjB35qBJLMMBn2gVK2GRXdXCOKRDTo0EyiVjlBU59sHWZbkWehtgPVZGaYUxnt4wZnkAaq3RglMjUT+1HYY+lwlD20Zp/KD4YcWW+QITM1MABPhDfXTvufNH70oc5c7+KkqqRVg4PCzQbrULh1i5BARyok5NsL+z8+LLH7v78ivMe7Bs79YLO3df2Ts56iIRPnn46MnDk6/91WvURB8KZwRwQZtJXYoXFAkaPwVlIQ2IszaHqPoqWgIKXNYhpUQSpRaSgmCEACC9Ss4SBomnCIHPsJJKitaw96E2DhcblvplnvvUgoz53NosrsKHwvsPdtGc9Q94XM/DPa6nUS8LBmdfFWEC9u1ZHbRK3pY8AD0M+7QEGToGvScZcj2rIoe4ViKKTwZbu7cb7d1pOmffvIVXHo1OF/NhLWnSsf2nB+zy8/kXPn5ta+/6t778pR/+zS+yI/ovR+EQTLxMgrRQyjrfopYs9lzLVg984ZiudaNLqdUBgJ2eEX+1aSukBpdOIp7kCo0XyVgkWHXvKsuzXyUkhUB0Fsnz+cfi1ZUEnbfPSo8Dhnu1nkBdAuApwv1ZJzhGYoyqKO0s8EHLBVnPPl4LFflQHy4ei2st1Vlznlkfus5AxOd8SOXd5zySLWH3YfEWWkGclXkOznAO96Gl13t7FFVSp1y6imxdzrx3eWpAG0tdfVWMCmViiS03lyXIJ7uVTfKhAUOP6CeZEfF9hqANSt6oHI3CeTfK+zHeh8OIadT+6SkqwLW9XVhLP5zBbgN2V4yCh0+Ou/0JOxdz8MeTdx5ub29ff+4uPlnxtBIPy71+nz0lJmgn2i4qbMTh9Zv7MGnoy3TMrLJ2xtjY6qS5v9FqtpstMoSi4kGFbXk8GhBY1LDBptNpuVatYJfWlKf2lMwwOcN5pGHgfVpnRprHINmoPHz8IP82B/gOr1+7UYmZgcZBLGhc31z0y9u3djvD/e63vvP4wVtM5202GuPTPnoU2VXZgrpWGw/ZjYp1UFJWWAwDULBVIRnAjtXJrHK1rR/l6StCtpjYRlSIM5Uq9u+EAxaZs9qQMJGedAcbjQ4nMELaWSKENqjdBpgzFp1Ep62GlaZ0vAley73SotFge4657N4sxeAkQq/a3P/IJ9HrX/3hdw7uv9lIYtpOHdDy2aLk4OAeEkl7Om0zV4btAqoeVDkjihMfIRUALypNmCbfLFfK3crTx0+mo1G1nLSa1dNskjCZyhQrk/54T0cs/RKwdd6wKAV/NBb+o/ld+gj1g7yRcFKMF1okSydj88tyHL9ZtQkBChec+8ucrIQPFGsIDrmDXqis2kZTah55StkTFwEtpfg6Tk8873glRwhxCo0aST7IEUzV0bdwT+VGIg6UZd02S9nQP81oiK0YRZlXTGajoM76mDMolypRe1BUG3CxGRrnPRuZFT2QaIAzGgt+OQGE+RPKkzjARi9Y3+H3jOkM/7Ucr22mENIJx0EPF5gAWGiEtQZ1DKDg+7Wzv3Pz7gt7N6/V20FSxv6P79Vir9HY2IUNNO+8uJWOvU9/+jPf+OoPv/ftH/V72kwEv0CKwBsImza1QpozFwS4C9wWxMASksUJFQZMtAJ+DDAkx9EeKIfBULISPDSOORcLE8DSqslbJBTmq/nCxBvNcmjlMTnyNXDwGRjsu85R1tqvBQZMGDhgVYDGYOkxwFMcwFgSMT2srhVFWj4bJRE9KQL2QpRt9YV+eetilvTNCK2Ld8mQJVxAfapr+TnclweMOZIsMXXRMIkskwrIJEcKqCVrtf3jrs4LrY5ON7Zv43QYxW1OzgZnR+kpq5hYOjHqsVPe4kPNnc1/+E9f3Ln+77/91989eu3AWwy92ZAsQA48F4U47M8eoAPA7KnBqhUWtCoVDXFcUlxbrXP8VfXGhCMUhIbAbiUpsj6YgQTOgfeis0SS7JkmaOXxm7hcf7z/nNSw86lXjT8f+9t5crUt6vwrFK1P1nCOxwt5Xqi4FcFt2Wj3aGnUo8+6ijyLql6ZElA6aBb5F8mcZHAhXo/CCgn1DEPhOiMbYc6UBY7qy2eHQdCtJinn0od4L/fZ16JWClgDUInjOhrvu/ceP3r0FGbAfO0UI1lUhR22qrVqvQMBffMX9+ubVZQ2BHVwnWWqDXbDqFYbjQbjja8wZVOxvKkFrQxRLJsQ8nqDiZsaTI+lvrEchgN2W4ZAcm7wmIlene0XVjGNp4mo4ZgZQYiJ4Td+O1B69GXYBcSrVOoNTn/y01fv3394bXdvb3uXwfb46TFa9a396zc/fOczm/XqduPH3/r+qz/+ebMc7W9shZl3/OjoqHtaq9SZdEX/4LQISAhMg/9ACp4JoWBiXKRaK1pkN4BcxmLS0MDFEPYkciYug8rPPgsVNpnmvAYd8iA2hhKIEVTrlOIIpzJNWHIqYppOxziiRSynQYGGSLPdlWNCbIjlVWt7d18cMgM5z44evt1k14kIAxqbMbPFY5Smg173CXWrtjFH4yMMlWaOmclhMuBAe80VwuCr9ebWNkdQ9fD5Ho4nWCiAqM6bl38RdGNK9dWJnA9HwzTvK4EMsmJ37MzYe52KDKGU7iac4a69zLB2MhtAPwn8sHM5o6hQjQXpcOK4Rq2QXNQxMFq6jMkLOKwosFJRTZlwjFIZDRMD1qrpErJZJj2Dk5gnkGHS4HLjzSeo7WJKYjhmkhU2s03mALWPSsi4TrcJNbQBvkyDUkfY2BIHa/H6PEZjhpoLAuj3YscqTqq4FQp45ziwTydY32GB6n5Z16Ghvlcpe82N8vZebWOHpdfzUoJxRhKglFsGQWle48TISjybBq16u92pb+7Wv/7Vbz55kCd1MveG7FlORcRDTFbAYiK5BTDAldlpRFwTgRzIxBWv2cIZImT5AI/U2wwsHhIFa+K1Ko9SkU/kYc56+Aizs1zvgBwnDLGp1ESrwReTmWRcSQ45p3kKxnwmiYbmyOEfwKgv1Lj3ewmk1Nfo2Frg/X6+lg50snKhPFRBF49njFyTAghMDD8rjc5CONUCB8z6uHOw9Q+b7rDrRSlIKsNqLUuSNquNEMIkBs5n5VItHfan4yELLf7opU/utzp73938i59/+5E3jOPKyYJPuxRHN7HAYDLqa5ICnLA62MwF9eHxAo9SLS9cjvDSPzJUiKTiAACEqbdEdsQnGzV/LxkwLVH7rC+d7OF690ILf+OP4E1xUaK7PmgpfLXOgIvPFa82LXuOAGVxt0t4RoC7RWoQqsuecYGYAEhV5eZ+9f0zUitnulyXK8KFrQjFufjVKwifzK96SxH8GgPmQbsVpyf57CCIu5UoxZeEIX08md64vdMEp2dstZE2m51aHhwcPE27x6hlCUZjH7kvrLEAvl7HM/l0PDp+0q81a5usNKpjKOXIgBRKA6HH+qc5ZmzR5sgLK2VpL6fJsMFTpVGLsXBjJuVMAmaoobcGOXCYZEw3wpLZtot9qEsTv6x5LtRTqIdJEBHeUaxQwAuKGUP0jNnDh/cfzh+gR0Pfq436Gw/eRjN+3D9KWuWd/a1PN79QbtaZ4Os9Pnh4esQCl/pmC/qFv1evN4bTw0UgT2TNDwRLw5qBjX9ZlMAhZPMWSyAFtE2w87UUCF5WoYE4JU+zPMxmnKO2VW+gWPFHZTVxDE2BteVY+CEmp+PufDII/Lof19lQQgZXuCM8mKSsX0Rx8XauPb9YJJH/6rA/5GzG6Zi54jKkGh/r/hFKMMeSI8BAfit0pRgZVWWKi1rI3VmnIlRqjUTzAt2To9HglHOUODlA859MdsJu0xk21Yi9eLUE2tguCAADhjnjzWQ2EiDPI9t5QMdFK6VG8bX2w6BH2LAK2FNxKWLafwOmoKYYYzHOIQwWKKFG4r/ogybWmL4Fy+TCdQAICmGpMm2nX9k8lOrBcUmGFQElDuIsfXA6RlCE5WHX1nJhZc0bfJtZ5Y2xmBKQBRDvYlqDmivWSQqKpGD5giHlwJilZgl6tFL2ai40Y8rW4hcYVorxGfkRYcdZeqWy6ziYRs3bud68dr25uc1EOyvQhuwLIodnjSMOo5UtJtYib1bjJds7G0mVNXVHPyy9nnFmGJvFwUamzG7I6oNXGsnFLDDqsy2X52mDCdY5h16S+Ns7nWs32Heuoblzmw7AOUlHc5RZ6Sehz0710FnMlM2zyD+beiGHsOZuMstoGXND+ACcDtKhpiy0tx6g4NQCGDA2IRZryTOAistwapc1UiH1hOjCVReQujL6qsj3irPuFj1cXtb7jF1TJUXsiF/Wgd2+jAyCewxGohmHTIiwPx6W5MXjvFzbarTH7XZarmwyOEBX9mCn0/FGRMrF86rRbP/u/vOVqHxj/+Z//+0/f230CKZYT3Y4sGw0OOZYiFbYmDLlQIEUofZZxVaSgdCXsa6Kulpxhy6swEYtVSlolQQIMWLhPhCWDCj5lUsmCpeDZaEoXZanC/4d3a2K1E+NXt6X9Xy/FVq162J6y/FiZPHMV4x796jRdzVWFcmfGXClX1kHl2fxygpRpdbK0uOz+kDcl262i6q6fLgzkq6ojfBmmVORfxFw6V0FXCTDHmlZdhJy4ztGA8iKFC/rKIf79jkkPg56HOTH+ev9YSYLbX2wf+sWfPTxwUkprjY6W0O44ukJqNba2oIEszCSPfhhQjC2JGRhwLS+UW/v7nDoL7VjMhhGq+lG7ZZdgoqgB8Mm8gnEPIMUxdVyyNaXCafccOEqOufAwzCFUXP+L17V0LkQzRi9C+UBJVLDUJsgYwmFbQDJOeud6q023AinrXSCp9J8MBgwJ/fg0f1yv/bWO2/+/J03Wf7Etjgv3b7z4u3br3z64xyu8qNvffdn3/vR48ejnVapXq7OK2QIm4FACubivGZRlSO0SqRslhCJEUHBoJVMjNuOgSxb4XRYlj7LmppCReSnNQ37xyj7VfgvPUm+AQuQmP2licAfQWeAm3f3KMUte2PTjxsos2GW5rHmdHU+DCorCpXXat14+cOw059+/5v3H77VmpdwKcdGj0t3NC/hpcXmAiiEEPeknUeVHA5AndnpC4xgOjmssIEWDKnZkoN7kLFvJVyHSXT0KDyS2TmbfzABXUIJ7k79FfeVkddW/arjpAfrDqEOcvoFlJFvMhMX8pHiD36PIUN+y5hUGF78ihQ5ZmyaqYi9qBL4p2w0GUwC6dEk1tAgXjOfRMpMKtVbFBGWwbw4m73Q+ZbKFMhI/tSAm2/YujKCvakdFCfGRncw3YrYBiqRA2iHPYKEsjtT6VI1bMItNbkKXjGrrsVpKZtvIo2YxV26L9+BYyzwJRWdXq56m9vJ9l6rvVEJy1RtxH9p9owbGUGoCWip1mlX1Jhj6cbb1+qf/4efbrTqP3/9nd7hqN5Ijp/izY4xlUpQ75QAH2BS7rS9nT3v2s5mjc3bqvHGZmtnd6PZrjJOtfYd9skhi05sRadieiSojEeT8RB3fvZg4iXHbeMhJtabw0pw9p/MR6ej8lE47E84VH7c1QkkyFS0y7iE6d6qBY0AfnSMpumhBJLb1EtXkJmCklgCScdFQGPlg1zu29UXVgXlRUC9f/HS6BGRYuwwEDAmWcnTYDHudR+wVwdGOjzhm62sVu0gbzNRgx0+YU1eUMrGfbxDS/XGy5v7+zdugQR//uNvfvvgpxiBMCjNvYp6gJWC2vkFAc41nKYZaeZxSbENKKtqodci+vEEjlBlQsJSU6LJAawmQo1A2tH3YLMw0r5ez8fFWPSvf7uaMUgYuAxOlVZ0gJDHnl2s7n9bF3Uorg+KQGoXkLXLfVvk4PJ0rwgTADldwN2JIeAin4W3lhs3XS6rX3qXzrbiw0VAUqOVxzv3xyuQQRvqabixzlBYLUooaxuMZhj6o2oyisIhvIyj1jhVkM0dxm+8s71/o1Jv4DD1zsMHSPpM36JZ7m52qvUqfcgY4I6qBP2ttuq1zSboh5oLSwY9A7bPlUyhvZuhNyFrk8pliB6kCMKixZOVGh5e6GfgLtQAIsMGNzJOD9GNWcXDrr8oP9jUOANWRmZcZKDy0HAGASol7LLaqm3v73Ak0uzoBKCFMdsFB5Px+P7De+gNTx4/mA7ZN6T7+k9ffXrv3sHj5z/+0kd2r+9Vy1/Yv3b97Z+99c5rbz159KTTqHeubU8nOF5PVkKuZiRtwZLYA/wb+KErSe2QNiUKzBUnbNczovJ4jDAq0ca9OJmxs/xs3GDTJoY3X/l+wolOuFRrQQ4mbvbHPO4fd6f+kN7YRNNNSmxCmc37GO7ZO0f62GRSiUKv3br2qd/BmD/7ftR/8qQ7GTMfgCoMU/HYuxPrsl0tCAlOWFGnFFVZMyxnqlmK+7fICQytXmtV4u5T1Pw+OUPSytoSMySN2CtgVzvEYulGSR4izUtmjMbvmDGYC2wlETJNDlO0BT4gEi7c0CRsD+zv53wJ9CjWq8sWioPH5OywXgQL0gXRQIdmJIFOCF2UTSQsFU+npNFkshczDZMbHELhgRfm8aYuF8sgM8haAs8TOlAmX0H6yB+EY6ssrRdDcxRJVbPYchI5z7g+7FzqM7PjbPGB8h+h/LCWfILTuxaKszsl7I7v+BAKq5VpElA5BaPZ8nZ2252NShQDDlZ5MXuImYP1J/DdwiOMVvFvUa0h66T1dvTKx+428cdtVu+/+wQD6vDa6OSIdQAjrKf4V2NOryTivh//eOXaXu3a3g7bqVJzMJb9RxOUOE4mYcvquTa/x8xgi18F9Eq5iuaLwo8yKIECCs9pQDhIDJnaQLrTFMe0WSlXSv2TIWZ7OgbGjEUak4m5lSHzAAh6kM4R44NUi8Wp43UBzfdLdyz9B7+tKxIrHkHD5LxjJEsVWFUh4DQ2rZFDUgvpSswvSLNgE7aeWYSUNWTagon8aeq3RpXqRogwW6pNphn+CeVKHVsDa/4XvTEb6f5vfvePX7x+5//3jb/80htf49S2nXCTpRdP80csope5WEZvcICJD6pEDVUxUNboN7WibkJ/a6zusi9bbamo4Qtxy8pb3U391XID+Re4xqyapDwsDKL9XVyMCdXAasVdD79STVwOl1vw3q2idAa/BqZdVvh7f3G5BMUUpRcB8ruclEjwijdq6fmGX05MDH1rGXJTHxV5FoELX0kK07tl0UUyAgjILjFhLpcbPzBgETvRWU7wY+oEHszAZ63LlN2aKwkHH7H+BeExn4/LzI3F7fjgpFtj8/ogmGbTo6dHuO7s7e9O5ml8it0NPa2xudWBt0JSa+1mg2VFOOuAo/LiEUGnKjrGhwMToLkkS9hOAYsiE5wQB1RgBot294FIQk847QeKzIooGXKkY4XYrAOdOwNVV21ZXYQhlawhIJDTiLPsWw2cuRptGpbjS6VVUr534nWZA8tGKZaorUYV6/Xg5Lj75Mnw5ISlnR9+6eWNnY3NzgYuMz//6Rt/+R+/dP/d+yw1xGRXLddUZa2HofKmAyF4i8BjwwN+IZv9sxZIDYT+sSF+FKF584IzxtkpCdrGItIhe3HQfD8G2tByZkp9v4y0nSRajggpQUtnw80RgoEhRyuY167dhsdP0TAryCgJKiiJcBFBi73+yc8ktdp3v/rlg3dfh/qjwqZTPKpCmsdEABUVY0TVhaFBjzFfQ5all7PAZiKtFEkoga+1OSM5nQ6m2RC7MrwL2II4mhI07iutzy4CUnlRKxkrEnKwhcrL2rRa5S00BcqiWjBaKLl2kZRyrEdt2QSbNh4sLAZESzoGDsB3NEOGycVwEz1FE8lmoBaJhauWQCFNppEfzmEcuIB4SOl8ijORBq+yhi16QV14q5MvnJOzlA3OXuacKOQzvPKZ3+Mj8yljDVGK3g6H1FbSEVuWlOFPSRhrwpz5eRTHWTDlFAtYF/zOGZ9N2WEOuN7xOlvlzka9Vi9jvJ/m7OnNjphwx4lgIgGAAmHDMHXZkYAYVdCTt9jZ7/xu5RPP3WGJUhel9+G9hw/u3T+GDR96o7G30fZu3vA++vFOu1NqcRhHJDsKLEMDBwGB7dNlMVfPgtHcjGxhKpJSj76P85AuPK5siFVbeIWV6ATukzIMSodk0NV422XDbDQYj/qsWIANM0ksYUvyIwYT6A2ZqM3MEsNmZPih5hfojHt0RInw5cCV6a+MVGlnVxEmSyDGH9eSQxNDCFl+rp5XvZiNEK5KXoN2JCiuyGTsOcmqP+YV5s1JVG2XW9doOWJqkEjQ0/Z90yyaB52g/Ps3PnytsXFzY+/Pvv/l16bvspd95NWgiI4LGbOlQ13lzqQEVQwUtksggw5YE0B/6cGClT6iZlSPz4AhPUWM/puLnPv2t3U3wH2QzKmktdh9yL3ohA+SywdO64qDwrgvHRC5f9D668NzOHRWkyJPotQuZa63q7A92OPZN5dCJBbBsoscuFyYcXgp7bmIIqULFAx4PZERCDJ0IgiohBQsBJfxj8MHS6h3bG+l9Y+I/zPG8yJsb2wM08nweITyFDFFql2bfDbIGC+mnU6rUa2ybR82SQ4eKpfjSoNzbAMO5IUiw4+x7sGruCFcQlbQuDXQmdBicy08F1FmgCWyp1xdxe1kjKS6oi9SZJFezSNI84VSqdlQMtdpOLhGwxSIq1bl4RVXoaYxFjwmwqAy0FEmEWHW2QRHlvFuZxN9kPnpLJtSrd7x0c9e/Qlup73OVlyKn7/1wkc+/cnuYBiUk9MTXLgGnHsgX2D55CKhaBRJiIDda19jZGRNP2v8a5/ZmGU/2FFrYaWx1WEnjCcHTwbd0SCXg3PbnySCKsPUdsRinWk1Ymtaj8OK2TWCgwezU6YcR/0EltWfTbc5obzaDCp1jN2i6+zxrNU/CyZuK5321oc+8pHh8A1WNj15CBHFmxbTg6zdslH0e0dPpC3NUjY3KTW3FiEzinQpJesqQZSxGsRJU3unTLon2FzZjASKRu1kwRZRg5KZsgu71Z/DENqtMNqgRo0w00gNN1rN7Kzm36WFCaOM48KSHbewYSVztAkHWoukrgX9QO1VPvSgrNZkQm6aGWWXRXQ7vmLvFjztMBNqBypZCmH+RnoZU5QD/2t4cUtntjMHPNKG+6TAVM3RCGSpBUNakYxUlCIQsfOK543xr8XDfM6WhIvaIqgh1qlzFkEFzJwHTGJw8hTHMZNUdBO+pGwQhvyNLWQXFqWTHmTkLYClf7COkE4AAj2QN5El1IxonqWjmP1l5NKGRJu0bm1vbbbRfQHS7k51/3rt5Ojw6aN7x8fTTsu/fau1u7+oVDgsi51VmHEx0ZNOZ8o461M+IAYQOP8xOS7YIe/JJMAIKjNnwcjCZEPPowbW2Ccdt15M8vKQYyaiggN7GqTMRqSVeVLGwhKlPVbuZ9kAr3Z4E/KDaLHcMNWUX3Ktek3JDA1cgG5dEqhf8v3qNYhrwbUizaAoGcvlrNztLQiDkEWD5P6IogALlgwMAwx5kzG3LcxgrLP/O8d0zYejsN5vwnhZZZCUWTo8ZSViFtZjDPpR/+FJ1Kh+dOfGzh/+S9Zr/Hdf+dMfn75Tibyh1ilBhxxfoARqIW5PLWga9bAGqvXL6oG2xm65GQ+29HpnOUhMkkVRX1lDr/SCdoU5QPDp+vVLqPx6Uhd2Nbsc/8wY12FFeyzdlVV5Zg6/6guA4q7LGchDgTH1y+58SA76f4Fz25hUz+myuzWQMGjDF647eXfWkdblTquiL5fdtUyg6SsuK403aHx8eL4rnRlc7AEVQR1q2CIJ3LCXaTJFCl/dMw/SW6BnUiZIh5BNZdDuUIRjn9MChglrkBZSGaVrgkOsoVnMB9NRI2rA7Q4Pj/M+PKLW2duGPt69e+fWnVvNVhV3J0g/ehSzgdDfOhMyUCrcjYjFQ3jOzq0N6Bd7ZkFVUFBKOFDHLMuTNgSf1/6/0C5UTAy0qip+C/LVqlUYQtpVmdZJb2ZyFHUQgswkC6tcIUXsTIhrdrWCoyt+M1DIarvu44d1elqaJJDY6ShmT4pWuc2KFGY/tUy5Xn/84OHbv3gXffzm9VsoXQ+fHMLL2XH283/4BxT1/e/+4N2338WXAw6pvhPMUIfpv3wwRZvEq0pkgF2oqs3q5uZmq1HrNKvsa/3Ch1/u7Gy8e//eg3vvTvvDlPMo7j2a4klOczhjmOnF2gJG7TUi2MdCZrUxOyyxdQDHp+ESdDrvTcqTnbsvbbRiiRHZAh0trHEiMOfYsQjmyGvH17/wclAf/eBrx92j/n69NTkdchgQ/IRtoieDI+bDcKeivziFzUsg/TW5pWDynrOTIhtjzuv1GmuEmx3sn7Mhi3c8trAQ90XQ0hyY23CD+QS5imkCmJeGg7o5CzIDxKR/MVdovE1MyjIIM4JDIJAYLTdNxSzCtuOBcE18WL4poJ68yJFjYG9MgWDg0BwqJ1uKy2Ab0BIRGITYr4f0Akbo+GFMCxSHKAdFFHbAZaIqGzB4PrYK9hc7ZnDMdXIiU51Do8v4tcGBsZcgM0kNokXsF63pF4rPeyo2LKOlmnc527WUFzjYzf3xNGP1OOdriYSqOtif/UYDZoe7FBYhGBzcD7lAmljIrHseI5SKNSJngJcy5c4TrC2MO47s8iqgOFYEDOmbOy3s17VG6fqNejrefXy//uD+22R+83qjtclaGuS5HEMPy2iY88aqwfbTMFkW2GiWnTHLTIDGAds7++N8EOCJiIOBbCAM6TFxMx0PgliAkoicm3A8hF/jWK2RUZu51GjAFpSnDGxOCaIbxnk2yozeycYjQNEojT384oT0jktCPRTWTTKHRsQZWVM0I0QLYYlc+zNSxqeCi2WklI6viCSdu4ggjaIRau1OcU51oWwyYRs25gew5YAzEn5IqPkX1kOjv0K9dLgkEwecpYRqPwqzvs80/WKXaauFV4mjBksYEduHJweN5ua4P0TIqu22/7NP/IOdrY3/+etf+vLPvsF521OahcSkPwgLyMxwoDKiv9z1H3CoEQZiOkIwIYo2qc6qk+K49MilBqziNC8lqo/RWlkqJU8aEstL8ctHKsDAZFbDXQbrFcSXcZd/aP+V6a0Oq4pZFS0r2WyKTFx9eOQVYVeWu7tX3NGQivTnAmd5n4um1vqWP2umy1n5WP4uc/fBsiAMbAKZ5Jr1u6YjRTwZweQmiAmfDAOItRx0X9YCfqaR4jImTiqm3qoflrFF0ebxpCaTQNzRVdju1FGzUWqyMECfsjKDdCCICKWwTx8y0ohz5bHQFoIBsgLDOaezkhGaK+SFXZakr0ED8F2w1vFdwAYDjELMhdrmD3NWzjTkyTx7EnmPk9nx7KibhROmkCBss8Vgc3uHBby1jSYnFJ2y4V7gtWob15+7uXd97+bNG6YAMWtXhtNSYVyd2YxpMmbhJmwK7s4vtAC3UuZwh6xFYltGLqie5uWokTRhxphWuNK+dDRk8yadqSOzOLOkw3Q6RG9uNpvM0eFXBRclR1QydptAFYHcYqNL6nG1WWZNCBNzPk5MXliPWkmjhsq7OOnVcdnNmE+dtpI9LLe4bu2hLB+fPGazrsnbnEFce3qC9ZCNHMej7Pmbt//Z/+q/wP75+s/feP0nP3304CG7vbMvCH7KTHgP+wOIG4QAgLZwj7q5f+3atWa9ko163e7hj771jb3r+81Wa7/dydlD8ub+6NYNfK3aeIiF0eC0F3P4QhtX7TSIywsmIbfK1V5l2Jv4cRqEg+l8xM5VlUFan03Y6iGB3qcnA+31gCkVf2Ac0sre9Y1ru3fH14dvff9H3/zat6+HzVvNrRTLxLTfqu5yxMPBk7cn8NZ00NncD/3txbTCAT5oe1qIEoTdk3mzVS83m1scXe6H/f4TnMUC9jmYn7DAlH2eOCgKvMHyzXYHuJXjAiQF29AY1U5VQFWNNYpwOU/YoAh+JAGBuWfpfoAG4i5RSsuNZDzAVsIMBcxJbIgdrLSiF/8D7BMcRyWdLI85RSNcVONFuTQLZ7g/U14p7uDCPJpgAaiwMIzzY705q7ZYf97HMoDEwFwmhueAE2KrHcZoMO2n00fZAo+0HsouvASt1atuci4GQ5bZALkxyQ0bZchP06GsAwwmSkKYippwvJmPXtgrNepb5aC22T18fHrwgJMYvHor395uVOvzUp1NTtjQnwEJ6dci4yjp5PMabto63xY4IfhgN8LdmbkcBAjAgDiiBdZ2VIMgEjJLy0w/3lScNYyX0PbWDudSh3Ea4ykAZsqTDbeILOKb2KtpjhkJiZMJJhx8GCEBL9iidc6qeNYYlaNanGz6fssrMwEy6c+fZNND9m9NangAcED1lN2zF1sd0MwfTKKnLOybTcW9sLVCpOir0rwanDJxOoZ7pSUZ+iEd6kyPjXggWk4aRq0Xk8HpcEl4RHOwRBjRM+IDmRGZkgUbkkUzIZR6IYotriOrhQRY0kHLcAkAwcQ/l7TRUUgxNNBMrgQXL2bKyFF+BuTAS6eI8DUmKgT3bIEsgW1HkyJ5iCtaNxgeTH5x3Grvhzs32+19yAh+dt5ciyzG6RHIT70GR0fN0sZ//sKnn4uaLzd2/ptv/emJh686kw/cWe3BjmcGDoRRLSuQfIIoqg3IwqheSWZ9znFZkn0aL/Ktm9rCZXKHqgo4DDjYYgQaG0dUXGo+oqhALZWfGMEC6dS+JyXAXmZl+f2aN/pimbPq9Hd7GcacqwJA4cK3zsWKvwr0q7ugqrA4MdU3DBCKncvDPQgpAax7UO8YXq4elf/l0lfsWqKlUFKAEpaqO1eXhrx1LSkkJCleXaaShBOk1l6BdCav6Fd1ObqGwhBP+bKoFK3DMFqqVAt2osL3BqvVPIOyT0PO68kPS/lBUuqW82HMDNYUajwEqeutWrXFxnrZQEsS/cYGGzkF29f2bj9/d//mtWq1Iiuzv8C7Sgce2YYVJiDYiQUqiipzGiBiPTY01iOxRBH5ImNekwWN7B7JN5p2kyaMyCB7JToOnkFUjO2U4XZsZYg5W6CTfCMzH+SDCWWGNQe2Y/1FHS9XE/oHm/NoOtCkJQ49rAhFqGJ6usbeW2xincGPa1HSqDbQ0Ub9YbWzMRyOnx4e49N0eHoa+eFGhaWX1YPTU6STWy8/94U/+vwf/NM/fPKLd7/1ta//4LvfeXrv6b0nJ5ifcZnB6s5GYBxhxGpFlmMdHcxwWaXvqPDJ4cHgtIvLNzOO1Jw+QPAYs7F2hJfWaUK7ba405qAnriTcuLm1uWA/rzrCCi4ymO/9eHLE6eLjJ9jT2QqahZ3p6XQnqXdP+0dHo635td3b+3f/4OWNm9VSJT36wZtHg8MWcFxEvdFBEFQq1VbaOxlIR+OMnkVYaceltpFBHINxBObMVYggK0rrne1rLLjqnjztDXp4hGqVMyQCWIGHdAYm0NkUOEOS2OsAgUPyHUAXYrJdl5t8B8DiviKNtmmikRvQDWSTMMtdU5lwQBb1sDcjq1PZ1TdA2sAgrinNaqeB5BXhx1dJYAEsgzOvclyasRpz+m81mLcWHh7LyKgDL6jggDD32MxL1vtp2oung1LU8kgWdzg4Nph3vUVf9B7CBitn6xKcwGGxppbC2sBPI/5gygDc9CM0YEY2LjiJzX+MPTh+hPss+xyOSz7T616DPbuxI0TMlaCCSkdC9/U9PGxxeCiHXi2fV/DD0h7S8vniXCK8gmwwMp4ZhAYR0QLxIk1+sx4Y/TsI+uXyJEQWYcvpIMU0TBstd8AvBQhvMBR69GOzOrP0C/cH1n9jKMIrrYRnrw4IwdcMhyOfXWhS/H0D/IHZOIS9WfBrxHAP04TRRqXM6/q9mcwgsFJmbRgQOZDmJXNGos2UxI8uIytYC0zHgAqJxugyugQRkt7JBRDUHKNKvCJf0SHNKuoPNFlSL5qijEEEfWWX4CeCZmikbHXxRDIX52LW7gKesd5lZYwGQy+URGEF6HFgC30C/kLSOWeUn/o9ZKLppNrYRVJBxIIolthKAOqAGQM7x9N5aZK+WNvq/P5/tnHn1r/9/te//vq32NG0HtSHi3C06KOJanJE6o7mPChNG5HmOXsLVXwmPqwGqsXZJdagaIEaEBWXrOVG1omxcaTKSkwFVlwCvcAi+AMMzecYeS++/00FyJvrN5Xbr5BPUbqabN3v7s/K6sq3ijzXiCVmrGdCQRrhq8aqO4ArmLeMXPaeS6CkyzwVAhuN44qOiIuL9GlIE21SgOsvJgnFsZUfpJad2myPPvoUE6FmzJQL32Fi0so/pgq1/JD0iI2zbIuJNg7L1Chmc4C87LOxxSQupThvhjlOtujEGeddNzvoW1vVZnPsz46ODtFuWdnb2Wy/+OKLd+7cgTeDJ9qNgnw5YQDmykkNNNyGk/nNMi61dYRGrFk1SQMqUynYEnZHOd+EAXs1OnlZxBDHFspOJ7aDIwYl6o7G60w1kkGBDzBBDYNNUhDqcw0PLI4CxhhNbjFHAmHKxDklH7GFZsjq5ApQQnVmLDXL1UatidqG1L+xh60x3+sPjg5PWDN8eth9OjxBTh5PR7VueV6dPT26zzQeK3g+94VP/+7vfPiH3/3eN776NY4grlQ1lQaAYE5j5A42r9XaGfkJA28MyVQPXYjikrzU7/fZ1ZovGtUawnQwnZ2y2daU/Z9lKsCmTt3kYMYOXzN/OM3amzgXM6c3Gh6KeUa1OnOYbOl53wNKg5E/u12GJ6Qddg3br/3O5z765jx/8J03cKnaamzOjlmM4SfzOnPKAyYD0OJnQb3Nfp+yN0B0WCglZXo6msxw2/Zq9Y1qmd0vPU730UZXsFtOd9POkiCcpCFt1SHAI7JJrjNNF4YlYoTlGYFKx1KxMSQKqP0J5UFy9aLkVeR4SB6EBwERyw0MTBs4lXJMy8wa0Dgm2sutNhusBOUqu4GgXuNTLAFAxUOyMAGwkzdrYTdlnvCbSA2LsIfnnucNtKUYNDhLKmkDp2G2QIsV08UrCkTDhI5TDvo787DgJpwNWCASqQ4MInY20b5QrE1noRt2ZBhGGGfokWxvQluiGPEqLDUSVGX4FtMlmtsGCOLgsNiQ07dAVNnfQcBggUEAEzTFol5b4eyUgpyh4Ynuj34JRDVmFWYw5910dhR5oyjWZqsACKjB2GRkkxJtnEWMDL6llWM0AE4Ol+QPx29Ycoxar9l3nOYFQyoPq5767Gc6ZjKInmvUNqJSsz8i41KSjo76WRCP8BNmaOF8wAngml+X//aC7d9Qr5lIZnpZjuTyLpfRmMV2SAPGSOXBCztx5ARssYtetgjuXJLYqL6IoqOLRQCAiRKqC/SehJAkgIjwxXcFeSQstPngl8tBudolZAXsWm+V9dMe23SPxrMWjlkdn8lx1qzTtBQOzF7yjeo0nw96x/V6c29n8082P7u9ubVXrf/Z979yf34Afw2jVjc7RgIFCHS+RJYS6FjGJWXqsTIEfFiVam3WgzQfUW7oUnGpG3HaNAaMqCJUA0hAlOxoMwOJ18BPK9ARdiWeqMOL73/NgIM+kF7PR5FF7ddf/PbDy/qsCqLDuICmWJ0TRlavil/Xx8UjAT45H7neujPQFWnW0xeR64H17y1/RiMzEPQofE1DwZCdsHWmRjJ0kbGsSWJSyV2HhQaiLBopJEJzpY+1xoh1kKCFOLKxLiY2iEYFGZ5U0kyrWdgzpqxViyVm16ZMZ2VYqhI0SKhmyeu0ate2N9inB1PwYe+k5XeuP3fj1nPP3XruZrVR6fZPmU9lBySUhjLrhmaQSy3z5RKi2YjScMA4x/YA4vecG5bjdwo5wxkKmIN+BNjFGJaJVmwmOA2fbMrpS8yRskgVsQEILD2J5Eltft1sGY1XJ3ZSNZoWo9+zDQf7bZXlPMyukFN5suriZAijU1AAdAz8xKSLQIEa9Uo7jG5WtAf18eHJu2++8+7P3zo9OBkeDDhh6OnJu+xuCYPEwWh/Z3dve3tru/XJT70ChUQ7x6trDPvWhnlwIJleD7qHUvLlKgKRXWBNDJl8pT1sb5nNjoej/ngKO5cZCgoxzaDvYiA2/0od5e/NKe5Z3p0My422n9Twg66Vw7jGKcIsLemPZEEfxA2cvbFrjru9081Krblf/tQXPzIf9g5/+ng0n2CSZEdkjrGI2IQAiz/z7czjQ21li/TxxcbXnApgvU9ZizIr1QPoTGNr+2ZUjt+5h2s0nlYynGAapmK4pjODjyQP/khUB6OYbeNUQM4XlnMTUpGkJZqvBSwgqi6+AgW4RFuJRIdAHWHuchHlGK6Zb8WtLKp4SYWmofImqPg5E7Fx3eeouLhMQSA2n0PYcbSbB7Uo3vBgwAtmUuQJ4Icn7BS+8I41Qc9AnDHDuufNNzzMs7ONOGzP2RKL/avlKSCNN4xQuJnrgKtBSWFv2FoYwJA4KCLn9k44+MOPZjbTi/9gmc3GYbGIrNrPMGQnGhwQQGH0Ks2FizyKdMmSwJ+NKbFKSAdDjwYjgrF3CFYPzDpy6pFAKuGZPtfEIvtpc7oh5/PMcdEb4vzgYzNnzxGOz2anJw1qrRXATg4+IAfBpJDqYFr6Ew3gHAJRc9hXwqnSWPZhz0Qgs6ICJyybj8Z8Nwt26s1S43qbQhB1JsPBk6NZ6ZA97VD8WL7DJmPmMgmI2V0k9Ics8DI/SHUiuWNg5oAt4xMClhVJK0V73ISusQc0XlLbHZzR3IMUDJF1MSf63kiTZQB0qD0vFSkGSTIjoe5OGmVl9/Wv3Lfv/8637nOMB+wuQEcy208F8ban+1tMVGEzK7O2j2mqGRIW0vVgNOv3TthsrtSofv7Gy3v/uFUN43/77S8/WhwzcKp+gxXUpjVpN1Jkq0S+7YhctJiLu2CyvKsFhN0re792MwYMVbYt4QRjetMujR8+EjqC98hEFAf8AAbR/PwGLg1E6wwH5d9Ajr9GFq4mru3uTmaKFJO7GnYaYYYfrlh9JcgIbufB7T4X4Gxk6tcusQcDwArPKG2VoUaxoewyKVkrd7G/VSaovyLbujOW2TRKpiPJWE62sukYCawMAJdIGWpgQn3lKKQFdBBdLTRBTZYsX/cmW2m/lczLNfZIwskDNXg2GYyz6QAnLOy4oZ/J2zeudJjerMZs8s9+WDg1b+9u3b57e//6PtwXt8vhaITtl9lTzYiVE7a5olzDeaQGiBRrPPB15hCk2ZhpGjxlcJlIZyW8InBOZjXMmHU0xLBQCCrJIgJYRQTH5Q+WDAOWIRq1Ek0XQq7jadAGZRVAaZBGBWHTuW+UgYYBV5wwOQhgGeMwIEvOnlTikQIn2pXsg4gDOE8DZLiIuCQ0f2dz7/bLz9995YU3X33j56++hndW7+DxvNedAx5OS5x7bw27D+79HKUDqyjnESFG4NdNrixigdVQG4grC41QC+k7OXqxA0TArsIpBHZnZ6dcbzEVra2txymRcOZWC48waitTmSgfzUNmgo8nmMnmp6NuPh22d/Zuv3Dz+p07T46OJnMU6Z5MwTn7aBzuxeUGHkPZ8N4v3rm5cePjf/Cx18Lo/g8eMHfJNnsAE4Os9BOWM+fHCG0gDDArNzbjcpMFxyg/AAn388mQEydZr7TRYZNttuQaPeG8JTzd2WiaDbfRFqRQi/wgouL8opOiYQzGenHJgFoYIpqrHhkCEEBNKyAphqC0yult4Cbb4M9DtlFUBRcw4JDFrmXm3Ilixp5l0q2gjM9qU1tewYewBkx0ZhSOySgiXrjp+TBaTl6mg09m3gHW7Dw4hdFyUpSX7nkpKnKTpZ+lsBn4NWyEhokJTnbT0TQImWSR7xVapoabrIgaIEiF7FgxzXtlplGlGGpdEpVnyRhWfzggi7nxtgL1NCE+mzD8TLwA32kpVICM2FADtsrIQ7hDeGDtNkMTI/oUlww0dVJJXoQLkJK9FTXt3WfTLp99rdm6Kx+xYi4ucaYIk8QIDI724tXAF1pkTBnCJrEvxrT4OpMBWjEmGZLN6LDkZ1o1LaGz3oStVOLjJ48wN/Rnpbacc2NZP6Jyq9U5egQwMj7Bb9icQeRlT+drZgGRTLRP5nFZmGkmQhsgAptVLn/qWfuly/W4jHEqssxy6mlQDlFPrEP4JgLFhfVf3yIo8EycXrtmEvoNXCLGupa1co9i8hhzKFpbbg/6fUCGuDyuTE/bmzfmXsLQGMzHzKmzuS3SOkaJoJfOJ0d3Wq3/wx/9F3h1/Juv//mrB2+D7lpqTT9glAIB5NIlboFF3VrIwOdyZL+4u4BeuBlruopLzomuF6kiw8M0BkaTFo1AAeViBikA4GIVcHWKEd3/TV3qA7sIFOHfVOYfKJ9lDy27bfnp+6mSS+M+pzFA01Bs2S7XuAs1Wc+2CIsErKBhWEmHulz1tYWVsUYEFIze0FAXDdRbJTCCqBvsigjj4H6AZyPEUoImH8KZoVSsUWdQ4Y+hHXQ5F4+VhTn2Kqho00tf2Ig64ZyRi57G1rH96cls8jSbHAb+kH03MBLzptao44TLsJeaU/KvX7v+oQ+/ePvOrWq9hh7IKtJqrYySilcIhdIoWeggM9o2MmczfazB+NhUfM4DZvELOzJhtsURGQsh+pBgIKy25Uma883YJcDHOQnOiVvIFCaHSxfTeOJOkBTjvqCmNGxBZMwh6bNZksR1DNDsaANhy0+DMec0Yd+t4LcDReArUUlILtwYaZi6QUFtXEDXoLLsFxS36iy34pSi9s7GR8of29ndevvazqO33+w9eCsbdHvdEedDNJttADsZDCbDEeCWs/dsigkWJQh42280xdyJUMDBdVyQZwgzHe15h0ddxvPe9VsQheOj7r179wa9XsQCXI53R7NUB5tKLBpI1WBLFeZiaTJm4cOTx17F7w1wVB3fP3iXrUrgm4+fvr2x6W12bnnxopcevfV4dPeVz3w4LB+iZb89JQs2YWD1FJZzmo+vEmgjAw+uR96MPUD4DattTq1iZgKg4CvH5DGa6N7tV7LRFjbS46MHx6fd0agnN3M4oNyVTdahtoiB1FIYJx1XRkgmgBWg4mK64J96R8QbYsv5VFin8V1nUxVOz12ENb9SD+I6Ts45UoC20EDIwwSMQZW9kpMtr7bphfgZUVO66yBgtS3yDbPM4GxQJ2uawiapaV41uyCwY8rjcLZ4FHobnt+WngX/1jIBAIUsCHmTe6EWVmG1ZnGSlmxpfCDUsbTXdPMJfsD+opugYaOC4+KAxYL9XlBNwRwIeJJzOBereEZs2YZAAWsSDKgibngcBzDFxYwcSYwRg/SyStvMOwYsQ13YqvwsgjnyBMo3675GeEByDDRnkSy8MdjEN1jz2W2RxjG8YbPqLuf2i1UEY7EZ/WHvoDRui/AWPLQkYGHNx8mRQzjCRljGXFJexM0w2ch6OLixIyxShyff35nmSOgLnTqKqwj7Zsu/l7GEFAW/lTuh+ouKAjQGMnJtiE3b2cZFrBwL0YhFfAcjFKfhq1kKehu7gJE07kAX7Fjd1RDGK2mBkbsAHKIbhdNO4dGKEtqg0VhYJXy/v7Bal8l6VgIjY12cU2M/y8ZpztrnUXVyylDCOYtJfXCaHtDq7RC6xPY+1cnxkL2j929u/8vP/+H2tZ1/89d/9tVXv0MHsY8AKIO0iAkIDYIdT41Bnue+Z028WHPBR+uA+QiU4o/ZFSbG6hU2vocmQln7fbYlOMXp3QYW8yFubvEDw+JiyfYMXC6DVcD64LC+Mv9nRcoMYn8gBWHuy0fwm25xddPdgnr9rJwsETVewxVLKgRaXuegb8jmcEukmKxVRAEEBEJXGK/EQR0cLL3S2B9dxishtv2HBTlpSPlh+OOiS/meKtjgyJhAMOMinzNdgYsw1jaMycyKcaIp5sjEzyvBnA2rOOuADR8bvn+t6pUh0aU+7kKT0Wk2OJqPjnHfD0s65gh3TGbBmu02Mhp6a4Udrxrla7ef29vb5fgXVkqgn2I0YyaPAYCvlE15ygCl0SXfZUgT+jQH0o8m+JFoFSPawhRqjm/zLMWqg22Q2V8i1QY5cSEApiN2bcJ3hvXGM87uBCHZDxnNC5YOdk6niIkin3zAFJ8ZokViYe8+2jNb8OHXjbdWKZjAHqAhKLicM4DnDKrHHAlA036QTQmYqG+MPAg1xnP8izilYCinothrdJqd7c6o2+ren7FRJStUSNkf9KCIzNzgdQWnp8mOoLCIGREKbl+t15lXgm5TZwYUxgCIKqwKAv7k8KjM3g3V5s4Wyw5vxEntZz/72WDYa8nzm8GogaA5K8y0VLNU4vwn9kBieQmbVP7ox98LXn+VNKhO/eFxjNtYPenUkd+Hg9PD+nbnzodufvebP9gd3qw9f+PTf/L5t75+//StPouHxkN4BRydlsKy2XsBAQQrRtqaZ6hDHmwIs7HHIh9OhPQ5cQ9zHRsGMcGKa1FH679+lrEXhlQ2UU9xWxQb/cHj4LIwXZ0SxyPsVr9yctKmE3BiGWJ0h/tKaUaF1EofpncbYdTwWBWFXY9D4bS/J1Y5tu2khuLe5QBNN4IBt3XO4HzsYxqdduVpNU+TucRLrfqlN9LdhbeVLxph0AOdFjknX8CqHwbRtvaKLGXMXMMcYZP4mzPnpxqzSpiumiHYgJuUR2cCHsGecQN/YvFKRG6MUx34gUzJcdiwH7FYZAGYLv5MtEnDESwNMiZLpIbCUPU3kT3XKKxWdjBs7ZImiG6pfTw5B2zEyiisJ+xyg+mHVdzInZobYNWWoIhCK/FIhIEuQyjT4MbEBvfgWR0AsophgjBQM1YZS+zDkIFQ0pqXGtliY5ZXZ14No+nG3s1FfZSeTKf9MVPT0wm2kyej4UG+AAG0q472qJR8A9vFTZH6yoyEkIQ1gJVnGiv0JRq69TkiBVWCP6sSIncMX60D0iwmUgDUxkirACk5GUlBbSgu6glURXHJBMkCMiYVm0EFjpBY0DpLbKNKkt1Vl9G5K16QLaUbm9dbsuNPHB8M0Do6s3ZRCvg/Qd7n0FQW37GZXVaubsLq6MY8S3BChWwghbMP+On9x7U7e//oI59hqqSeVH/81mv3Th8de8dgvzgjnUpx2kjbJGZQSS1eCRiSKdQip5yJ8xg8aCkQozNlzSMfpvGbrfL2dofdi2gAotto3Buz3sSm//U1BHGVp5r1v6CL9tIa8Gm9TeDXezPg9cQuzPdnuHP5tWIuvl8vcoXQSxQEh9y4XWYu5ksSh7viOTZFI4yVBVBEDqzmE1JBvQmA2Y5H08XQg1kyTyt5Vs6nNS/jQN9W6LWSUrscNSsR5+ZB3GaTI8xfzPtxKsiwfzyedBmtmGnZjQrHHFgjB/Si6TL7B6XpNCobm+2tnQ4yNycTVZg0jjy0VFQkloiA6BqOPsxPi47Ywhn6MkSEx7s5w86G1RV1UToDqgBcDzaFUsIaDFinRAW1VC1AXeMbpmhwwdLEMKfh8FiSawJaLZeYlKkIpIChwrT0oYzPzBZPwW80bZFR5cWaHWiZhBXNBOUzjhpkFyAWeKINyfEL9sz5gZXKaNhn7jZOGoyutDfiOPV33/nF22+8Nur2qyUy1IWDF15UdN90gpPzkeRXtqnCdQ2ijkwgN222qAon0y4LqFCyAQUDGwqDGIDTde940D95fXxz8sorr9y4cfP4+PTwMStl2BFYXUbLxX1hRsomYnUNQKNROiGBjZzYzxC44Yzj+d3Dgwobjr1wk508jh49QFSqtpt+pfTqL17/3Y/sNf/oCx8KXv927/s0M5KD0TzWxtUgCx3Qo5tF4r0cGQAzA6QpL1UCPNmYz2JhEDTpaAD3YoPDEqttfE6D0KJUfz7Ssl05BAA1m/TFWZxZRsJovFJ8RYwI4P5GAJlGbAtSIxoiV2ct7cIxgNWqWJfr0ta8BIdeUV8WnVI/s7xJdQXj8Lj3/Do4jkKC3URdhxg3HwezHucMefhyY5GOt/3pdhBs4UUcLU6Bmz87zb2DYHbgeS0vGDCnW/LGEHjsPZIx0BXlmIyMinUF2ilxnDciA5JCZF/Gl45tKXBQR8Wh5hpm4lGpNEasCRjrQVNnN+VDeVbYXDK8SvolW9Mw0sW+IrzG6E4mjVnXnRmU+TaF57Hp2BBhQhutx1hFBzojCzsmYARfzb9BDoUSXETmASOCCzQBXMIGIeotn2XqjM7JRLx265T7JMm1IKy2yBverI1qa8fipvEizrIuggXTTxK/hk8n06M5/tx2ygSnHOOfBivQnHSOZGzmLVwX8HTXEJXlDCrDEmLHd42FkEpatIzW+sPRz///E/dfX7Jl+X3YGZkRGT7Sm+vvLV9t0AZoNIGGI0AQIAUZUhQpLS2JY5ZmzTzOWvMXzNs8zdPMy8xIGmmWKGm4KEOKIimAAEkADd8O3Y02VV3m1rV500dERmakmc93n8ysW9VVBLBErTl1K/LEiXP22fu3f/vnf78NuwjwoEJ3D4GqKNcHP3U9mnrAA9ej9xQVAjLFZ1y+WMChk89/frCNP+Xb8w/q49XdolIMCUnxUp4BuFw7T6ECG4g+fGjDld3VtReWl24rkZLwGU6Rw/1amwl/dnJ6On7vcW0y+Oz1l176Wy/8l3//v/nD73zj2zvMcSKkmELQWcJ4cdxHLTJnRlM+C+u96kB1Apuc6IPEtCGlBjVBFiF9p7t2997G3bt3/fzbv/3bo/GuRaU8qlplqKqgxMhglS36g02SKz544erb+4N3CVwqcFQM7+qmCl6+RlX6qKOSgNxW3ZmJL2Ct7Bs//ER128ddrx6/uuf5r1ftV8/CL+jyw0fV/0Iq3/+xIGURC0uLWco5wh7iskF3CVTWVhAvNDpSTqrXl7suPq3yfAW19MT/zx1ZWmEwsZ1hvTwzkazwY1oS7gK2Kjm2bLxDnE3qbUXzz+2xc34qraAlR3B6tDR7utKeXWu3FqVoNGfnFV+SxCb7Yn/C/Sdcd6pS8tHQHjvj8e4Jj0hTMgxC2uT5FBzT7ne7i4sLq4tqXDS6bVUnceSFxXl2abBCAnq9DkE6nFOHkCz9FLiRjsnnxUh4d8NasRUk5eiIrY/RldEqhAf5tdev4XtcEqOo4r3TffvE8AYLcCKHgiWWwylCP6cRH0+iAafYviOGg/PD40PIrBIgCR8vRFbJ7+PxyE8IR6dnX4ZFI/HVRHQplZ12PLhRh4vkwjcUoUe0U6/HeS0sdXvz8bsPv/EHf/TGt76r+NagkawlQzGBeD/Hqv6Af6u36HWU3uwMaEdAx5gOtS9+Z9Ab6LHYLBUQMXieHTZgux3QMz3y9METi2tRWuK1m9dXlt/+5tdZZ4McmVGSRLCliBSp/av+hneQj7BGKIKpebsKIiKO2+f1EdP43ubCYn93/6DV6zx+yBD94MWbn+n89I//zMLt7375u1/79T9Qtvtwb3R0cihgmyluejIcHbDKzjB3LK3f6a9SoFo2H0huscAojk/Yxhm4e1jvLPzIp7+Azrz36DtYi4GKV7cvnu2EpcFMJkMsty/LmTkXt6IB6XgxZpoNihSQIt/cjRypx8Km5OvYAXFxrj4QUGVDA/TbcG26C58FcVlJ2Wqj0cJfhVnRCRH5llj2cJPZTmRJTlO+y1PoOS/JvNbfaE42jg+XT092EjQ9c1i3997xs8ODt9qNxZnOpNZgrthTbTOJOB02Frh5JPAucVOcDTwj4X1gDcXgH7bDYzKWak6NryuaEToQVVdIwVT79Hjoyw6eZ0rJOGyT6oTfqug42WvJngpFtEJ1NRGCx4cjVclmxgeN2L2l8IoIOxb3LyogAQNy3BNtk4XvyC6OVm+sCNooMZamO3RDx61vK8vLCBGdODRjNpBvzcpO0m1OwoLrne5Se+FubWZBwbfp+NlkvGM9LS60bH6yufmD5uweN8jx+FGDpi5J4Wiil/H/EKxjN6MUFtcjAtaSx3SmsB3mIkB/azpUynFpaeX6zRvtVnd3W5b7zslocm1xGV7xEdm4m5takVcLRKRiYImVFcqI0HmDIfpmvfoJCTdkJ1EWrNQMUOZaAUH5cL+/PiH/1bmVe3HOr2INhOGVozAXP+UofCSmwhChi9/xXDOBgQFsMuB0g3KZSDrNW1DT4b5Vx+R2tMQc3V4SAKps6fn5gW2ukUCMb7p/2OCkac39737l3/nZH/sLf+/X/8ff+PbvHSTTy0weJLb+iGtkjiZABBRK4sXOI5RHK7gYS7AiXYrspjQ8c3eQEeoENrMsbM2V1ZRWWFqe7/fbrNCwJfIh01BKxpAH/5xHebFnqh48f/LnbOjDt6fBS+B++Ld/Rd+9ImD7oeMjL1f89ofuzQVcIeywHMG3ImDm04r74Auqb2EoWWbmKEcIWDnf2zvAzGJgFRock4rfSkkfCUURVgnh9pZOSQJhunYhFVEzczKOPFyvLc6eL3TrUnTXFYacHnZqR6lsdXSEfqp6GBsUi0ztSI324dH++Gj/5IxfCpdPPFGz27e9ILNqs9dmSsV9e4t9taU4LERsyfaxnQATUrSy9JtuySkVCqSzqIiRhlxJeu/1cGSeWr8mO8hDkJAxTsnAqe1d4WOqDwZOxFQxhhR3BLywaHFmCTn1Cqs3dqTMfZyPEULAHpSszVyE7hA+hlDXiylAfDKrEJnHr4P5BIfF/VV2M0Qfs0sO0d7qIxqTL2ezpQEuujs6fvP7b3z3G99+7wdv83iuWRKYEjLMg8S4IHyoqL863++qaNFUC2kqwEfZC8ZIFIMxNHv0lS7IJhBdG/k7fRBPJstSUOnhzGR0MN7b3st2rnFQGQ4TQAzxSVJOfJnhMJJyc9YSvzwc+aYcg70ossLPGyuDRZrPo/ceDxYaq6vrqlWP9/Yw+/Pz/fsP3nvx7qPazc/XXphZ2h5/od37xj/+zYWeoqInwqr1ft4+j7X6cP8Ze4UtmmhUc/Oncz2qpG6y7ZN3iPbSrs86c63llZuvvfo5qt79B9+ZHA9b3UVxTMzmrfbZ/PwC5iimrNNTWClUtfzLLBb0NlWIDCIHPMf2LJZeO9u0TZZ9jsO/in4Xo27UPUsiqh75QBS+0s2oGJ0q4YKQXepPzfZUjc7JZPdkumPXZBX2Z0VjnUq+WZg218+Odk6nw7pYeoS1tjedPJg9HLZou9P987N9ZpPDIW36RPnr2plaaiwypiq4yfmsu3WF/LPq/MOCrIiZqfw4TQl/YJxQZzy+UfZCc5p/5tJTQfUs2NiflfOcUQDrZAhNYoryLhgOlCeH5ycjr1cCk0BhT6RcOSMvpuQ6I0Sk+SK36IwGM+UaZw+wKpwBaeHBieKYac+cWXKL2V1ghhQ7VoNaA6RYir/W1NBmaE88onpw9tNqz5OzbHHFApOsw5O9yfGDo6MHJ8dbzdMpbZp04x/whi2yGp8x28t5boyfDZ/tSpOt9dfmuYh/sLN5vtT+3Je++MWf+NLNm7cl6hAl3/neW1TBr3/595f67eVBXyT/aFfS7LGa4xZAUdbgAEKdhVkd0RsCNiwnI4uxOFQvHyF27994+YDRw6cCap/lnihyxb79/j1/ljMCYnlLgiWxQcwbXE0T6hVj/lnj8PDpzhZT9HFt6bDbW+MN4aAjaAkvmFXj5FSZ1+RLkye/cPfVW3/7xvpvXP9//JO/c3Q6nBksnh881UfiUmFLlr3JQ08ST/pc38pcZrAZc8SsOJmCRxF2pS+icMlo7DTv3rv55MmTMeoyUsNFJEI9Ef556s9z6H+Brmd++OSHG/oo4H/4rtLk+81++Oc/7Xs1guc/f/iJ50YZcvnDR8GHH76cBfnc8f63ivv6qQgxucUAQm2jGuTrFd4F0arbKliYQP8iL4J9+TG4hynmiYKCkQ1NJG2lRLogBIi9ChmT7tHZ0unRYr220O+vDTqU3U7tuHt+3K0zq+FmJF8VlpSUGvMFZtMyIZjnM7YKOFDYyt47Kr8Tq0UTzZ4JdVYjso0Bt1s56bZUV15YmY/rV4la3puijYeSYYOiNjiwimaq98nqjbEnGirNkg2ZGkE19gT27AHW5WAhdhhrWpy3IMM6FbGXXQcfo+doza4GrDSEVfI53gC34WOOGK4wbDqjF+UH9A6oUDA9Cy8ouaY8O5Oj0eyQD7A3GIAlVVV4BTrK6Mj+lQZZOSUNtWYn+8PNB482n+784Lvff+f7b2HXL91+4fr6+vXlZZu97dI1d3cJqTixHZ0U0FREkxoiBtcgFKfS03ji6W7Gk5jS7E6jMG+0JtUZsP/IEEIydVRhE6HQ+7F9C9VNBm0mk0FAJjbyCAgEK+OjrkcLATaSTmo8RkkbcDR3B6gqQbnRnF9Tk2lyhuA+ebwrwmnz0ePvfefbr954vfbStfXTs/WbN6bjyZPvvL398CHO256tq/47ezJUFnEy3MM+jmVGntQWVUmzC6S4Gnv6NLsEhqOxnKnJYKF77foLDNSsCF//5m8peShjtWiMnIjK24NrwVC4QHXD2GOINjsmCVwVg8QQlKk6Et/X7ApkLlX1vEaxipAIBJC0onq2eAZstRV7q+riYUMez7PhzPj8XL9xPt8+f3Yw3TseP22c9IQc2P3SttL9mRtHZ1vHw2czp+Oww5pCYE/rY6IG7q+cKrPwCZurwBZp27LrzmojK0sHI7adC7RmKxceDyNAGU6yO54JDhSFYMrEKKRwEiRxV9DaKo0emvXpf5FlXMQEJvtikYmnYnNKlYzUqqPl2ozo8Fz68NG+RRojjkh+tVeJaYaPUJvOwoCDyZq2iNMsST2IrIEAqFAlVl429JowvPpCrbvmcv18b2ZivZA5LLFjJpgWUTAWhTFWDfulUs9KNjsakUJa5wf1072jw83p0Wb9ZD8RYAwJBK3QmEgSuC8TFfFlmxWsds6ycDA9eTDeN+ru7cWf//f+xid+/EdfePFVWqTdQ2uLq5/+wo9+4qe/NLi+8cYffeOt7765zOu+3GfdJRWQHsoReFWKb4ZWDmhRkbOMydv8AYLCri/u+OCfigG7FliXz0CnaDIfvPFf9g0dqwhDgWUG6e746rIQE51B/juW9K4663FcA32GosHprIJotU62/kg9vZNstcEtBY1mZj/5ytJ/8Ct/XZ2Av/vb/3hH0YL+0rm4kGAzEijnTN1tlFEKg4UfIuT6FQSC1hgwcOtCODp1fnqErGC6jo2NjfW1a3fu3Hn44MnhUKBsaCjd/GPHp/GPOgLowmyqH6uvH3Xjn/XaVQsXLQczL2alaqKaoY9r7qN7+TF3Z6V8sLmrt3/ME1nJ1Tr50A2aKceltaTApCz7ggXPDaHovWVSShOe0o1LvC0aZBoKNYPCoXg0JrxGBSgxtAj26anCi4I+hSsvztY/vXF9uTGz1Ovx8rYEOk72ldE4owUk7GJ0dHzA84Rl2NrXrrQolqRfC/ecew89YIGBQHZWF3cw6AzmZd90VY5YWOzNLzB82pEFMcFr0YhwGghMkAzvjLsuRxTRMJATCRWJDXZQNNTjUegKW8mwgpHchfypsGtyPCYUK4ZkqXg0BqJyEOeZsSOpCo5RITr6hLNQi2Ia0IsQKxyNLFDcSnyZ3o9gSwPhh0SYIkT7ZDaeVTYy/h8WZfFcidemz0qox64VEKrPNNUvevTg/ltvvfPk8eZwdw9tunvj1t1bt+PKFnc7WOiob9kfPHv2jEpK4ZAzoD8TrJSaQEJQ7COMllkhC0ereLNxuR49KfJTQBFSl/p/sY5YniQzT42OzqdsmdJ/o8BEI1aUwX3J5+p1vD0xZYSMGPHUn6AnNwEKHxAyJd5se595eWrZspU18azJ8dtvfu/2rW92PvfTtReXjyYPX//ln5nr9HZPJqO9fZUtwDQqKOKgwNjkgKmQ8sTX0OnZ533AyUBais7Pq6ma/fDIvhbr6/dW1tfGR5OnW2/sjx4vLMz3Bz1B8sSiLgctkwnum3+ZCoIHSBhHvIEzIq2nBJVm77zZlXFkK6ezGnjzFob2GVA4t86EFl7IT6ADSUxmKjj6PfyYfDDTbx/3ppM9PoLzox3hqJT6Wo/dWpGMhdMxx7xh0UdtDcvC7O2W0LRJfDMLZLfTGuxHRdkYqdyMz1GCkx1vufMKEkRzJqSGC0Z4XULVTpqpvikrt0hABmUKoRP0KiJnGCaXAA042UTUqrOmFUS7gchho1mew5lThnFhTIbIYYRh8xlTWMHK/ToaERUoGHMK1me5ZJUUSdLQLTDfZDgxSEk9PbFJ9XnX7EVE0QsQBGNhUrWD2Zn9Wm2LgYOZPXr6+VGz9vh4/Lg22j4bP60dPzufbJ8d7abEqMjrYGV5PDIh7svMn/0eDnVo0Jaj/ezp8bNJ7aVPrH/h53/2Z/7aLzeWFuwiLYguGjaTmfjGl2//8v/hf7v63/+jXz/++5tvvWPOVxb7Ahfw4Dj0dSDYHh7vsCyLvJU5oSpWBDFL20gs4/J77vvg4annL1QryOcHrj53x4fuv/rF4iNYAWYgXLhi9ZOS9a3mOTMeqdGM8Wttbx8fjHYX1k9avZV+b1mcTBATejAkTOpIpYCUJ9PDtbtr//u/9R/gzP/ff/Y/ypI7neEBNO+mGFyTod4CqEipQZPLbvjx4pwvhDElGF9n/7IDy+4Q3UEiX3qJbYSBUABNh2ClaQ5MaHfZxP/cv5nyjzo+zgdc3XsF94APFB1lIM4riH8c3D/qVX+ma/rpNdW78omoV+/9qKdLX6L8ZX6LZJC7dM3i8F/+5sjqTSu5y5oxGzkKhpV7qu+5F36VQeaK24Kj1m5ijuL8E8+ZYuy201FYwDbs02nj+EhC+UK7dWNx4cbqml1tX16Yb03VVj8+Px7bfuwcKWEaq88MuU/Ljr5ik1XK9yqrCfrzrVrnwp0xTC9VrVaYUdkJfH5lKXuCdxc6q6sLvUVqny4dx68ZS29JZwvdpAWhuyIT+DkNKrV5xVYhQzQ8lIfuqllKNMsZXop5BDzFUh0GKTSLGCAbhbxX4BxMLqHQJsLN1cHvgw07j+M79roAL7z44jNwzmCAuWiL2jgYHXJpQnSSC58rasCQTrywXy9XbrKQEworOMXWqFPFHW3J8M479xn8XV/uzC9KpzyfGe8f3H/7nf5CX9kvNoDF1RVyx96eUKZjhnoD5frSeRwL3yUkyFSoYQEJwLUyYz83BjeYd53jwXZfCfrCiOIpZL5Mako5Qo0y6zHP8gcWj/ksHXo8mWSrgBmT0ltbW+daFLV0KKH6bLogQrnZHB8ebT58Ol/vHOw9aYm03d397je/8rnr12q3X2issQ83XvrJLzAof+U3fvPx4yf3BjbdaR49E+WOFKh5NRzvbXLdMc+3V9dr8ytmKaHitmVu9eTgHI/JVvW5/tJP/uQvfvXrre+9qXzXaGGRaUT+EE3p0HaQoJ1Jy4jNrDnAWwgSrM2iuM7r3fNW/hWTQYt6EcHJ/SCAjZSVU1DepaATWHFDjEs7NODwm6wtBavrS4OmmkW2k5M2fqTgREp2tKY8xbbgOFE5huMSE6WAZk6YXhO2jCti+GQtOk5ilbVULTGdsAAsOPbmwhQiWiZUyxzIsE/goPx5yFAZVbOqQwwS2ZFL/svDCK2XOSczxLXnFhlcZA/1T6jgiXZOsSRCZ6KgWWCcZDgVNdBoWeAxcGAD6Vm0c0caLPbniNzZskmm+fxprScSHLh5TrJxCbqg/cbB+fFjrJftPizgpC3KpzbZPx9tD7ffO97emVFE/WBrerhzdnyQ2EM9mtoWl8E9RgYpUAKvAUtiYnu+v3d89u72wahTe+0zd37653/x01/88YWba4qNMaYrYs49vL8/tH4H7b7tp37sb/4b12/f+I2/+9+98ftfmUz2l1jGmjPdCGAx74JSTPLVEWLobUBW/gV2ZYAx3jr7iKPiF+AUULm3OkEMLfur20sjaaA0m8vv/3ZxE4CmBdMah2sgrK30JP4p0rD9zhOzSTAQnhI3HFlvYcT+1lGUmxVPGCARBTqfziytrmwfjfffun/n5Rv/0a/8DfrLP/ytX9+vzQ0ZISyzRpfERkJJby6Jd9UJL7/qFwEFHZUEYuTe1WKke/L4Gdqxs3PQ7w1GI91A4WhWIWcxan/MUcHlh38MFzFCE1CgdnXyw3f+qVeqpqrbqtf5LNMBfFcjCixzT/XDBxstK+ODl/7M367eWNq+GNGHngbp5/rxoR/z9aK76bZ7y5XLrvtbnUYKLHde3eR6dR5+zdBU1ICIlu6ENKKOj47s17Labt9YXrrR791YmL+5tLBqgxtkdTxk0DR1Qhm1gmHIXVGThw6q0E0dYxZ+ZzXA49hDT2k2hDwZS0wnmLyKi4zMbHzisOg3EoCLuxdhPMJZ8Wvp/Cyl+CE2y2+YFVT+oWT6jONhKgo/YDwcuhEvY5wMh2KwJHsUmbdgf6I+QmareJXUgMeaEgHhcrpGYEhkQzhuZSjLkkygSBZzgSQxwAADvQSmQHrMDibrRfhBUccl0HMx0zjIB+op2IItHXFRBpWVyMk6HO/adnA8fvRkk4XZ+i6xYcIfEgDL966pLRHPO9sJSWu3iQ8ELkKDZsSDUCgQ4bZ3x8dspZqu40a7Rd/GGlnzaUMIJSKtP1nwFJxQcHymWAoKgzYAo0KhY9ywqV+863OT/QNu2mR62rTJwlS20nS2WisrYohP3ntywJzaXBxcu3PL3gGP3/jB7YX1+3tv1kZnnZXe9pP7D7//xzeuL9VfvX789QNpSy/MfMbo3vnDrym0qep+iwasS8gEH9R4b+epELHJ2vmkKxB0/npNXMDROLSyLlI592FrzeVrn/nMTyytdN9576v7wwfI1+KgZ7t4+zhllCFwBCJhJTgoPVhQV0k66jXmhFXZo7JNFU75Q7NkrjSO0WT5mG8iL01VrFQWE5zAuphSbCHQVDRTVRLlohOJct4XYQRLFXk+nu7LwFIfLDoZF2xWp8URXIsUmPh5eBMBFs4VY01iTmMV1bsony56EGrpBUTA+X3Bi/NIFqPFkcgDOrJLNLoMCRf1hkx/GGzu1/eSv+JVBE9d4y5mp+LtEUYo5UnIVQy+JdTAZyKwoI836Gr4PkWHAaCsbkjhrRqFmaVH6jfrKgwnEtgUqV+fXWg05rPjUxiJDsoFULFsPDsjsZXgp/zcSXu6B/kPx8+OD7bbTBijRyeHsm7IxIezuDKWnRy0+DdwX4ov2TiKHY5cmxnXTrb39kb12Y1XbnzxU59+7bOfeeETn1y5drNmh1FzfHqyO96zT0q9151TkLNWn+wObbx94xd+7t9dWf6NtZU/+NVff7izf6M3r1CPkTH7/DDtDQU0aGgX2F8cIXSXtPHyWv5WDLi6QuaobvvIO6t7TEZOykd1pfpECTXv1ZoIlcB8g6xnols4dHA7cFMHhpHOD2xWo71HppHFn+zVI2bwj5gk1FHO3N7e4lKfPWV8/8nnXrv1f/r3/tf2Nv1vf+83v/30LYKPXZfY61hQdIShI8r2D/eG4mF3KlaNEDIYlpQC+9WcPX60Ze8ZYq+1IJVDij6Ni/VFToQF85ENPT/ID5xfQOID13z5l8Duw7eW7883Uz1bzcGfrzOlqQg8mff3P8vpZUvP/RrkiGiS3l6+7qLnz/fn4vHyh0T2/tfqTdV1Nqj3D09foAY52eXyNYhQfshPlSToSs7ThXKgNvDF5OsF9mIraTkWrbnFTutau31XKsry0s1eb12EM6aCnQpOnRziGjTalJWPNjLDcUkknp0b1EVi8lfK6gmLS0UssY9WbFkO1hQ8lJmmGqXOsPcqMVHrZcMW1SMPRmhHSw7xPDExDlxWF0GUsabBaoyGCE8xZtKBYBwslbOTVlDHycNo876IovKJ4V7R6bPhCZab9OHENqdSdF0NI5ZYrpHJYcKeyybjBRAAAwyhUBeA8UogE6VAGS3vd10MExYYJiqxSGiSAGzRJbYnqDcm7IpxOavbFa6G8qHRk9Hh5rNnT55sHhyIYZ7wdJt9YnsMwoxyR8mEJvYebT54+uzZ+a7sgI64XKQZfcFYQSwlq44SUU2lQKVBoyyapFvFOE/vHjW4kGP8yprEQulLKiwhfamdIE5VSSOW03DjxIPFXs0QHakb5ykiw8b1m+vr1x48erK7tYOcL11bOZzsSyfpLcw3+h27MK4N1vfefXAyPOyip4fjk91tKtl7735n8b2N7us/0bx7rfbotL688Jkv/eRGd/Ctf/bbB28/atr2KDaVsCRQU7qfwYQ8Mn8yXVKVv9YhNylvfSrXqGMnnjaOdrg16qxff2mtu7zafOMHvzca3YfR7W6fZADJKkQOCcOkQl3lLqmfJqgZLeGXPGbmj5E4hK28tUL3ovNDDMRoTlzyjH/EzZGyVgn39kAsliKJBMLgakygQhrGfAeNc/sEjohuc+2EyggnZB7ktFD9W8pQ4e/kT72I3TdrzMJEiaF9oYpZi35lWAr6RuaDnxVelSXptQZUPt1YuC3CAHsTl4rGxqbqh0RkhOcaMGzQJu9BJpPxmy1EHbdMLFdt5NxyTibJitGj4JAlXZZ9UckYo+GGbiZIohisNRMtMksfS1a90+AGs+0lSD93VGe7OLPvAlewQmD2vLLHfEp2PMPFp4fPpuMdOjjrt9as8EgDcJ9AK+UtuysyJ10yYEud4MBrQ2zsnF27+8Jnf+anX/n8Z1tLy1bOWW/u+OCg3VEpolsSb2iKbQbsg/FwcX4w2d1vCxr+/Gf/Mjzotr/2W79ztLVnj0PDMJxCuSIxm0gzQJQwA44IOSCQsQdt/PeRh5VYXQd2jzv3Sd75yJs/7mLm3JoKQS2tZU4jvPhDtob56sGxAihfEIdEhHaFcQR2zOzCLRO4dK6UZ5s8nf3c6jtbm73a2cJiV1b1+ZP9126s/h///f/N0WBw8Ov/6N3HbzLtSRUzq2k9/wzRS9/vcBm9QA+DASHkAwO2wEQ0ks1GyPbJ6OCgcGhbe/XgTGqilZF/3PA+9nr11CXgKggGfT/ySHeKXPIxn8F8YyqfxhXpqXxaMc59+jVjvbxe/Vp9Vi8sZh3gBYuLT+PPeaDz4c9q4vXWkTVXuu3TkvGKDx3e4qpbwgxIsoUZZ6oJmCRNfSu/RYUL/uR55kQ3FJzS74KdkOTSTZKJw2b8XKBl26Dm0aSvUjdtb3rM8rza7b68cf3u6vJLK6src0Tis7btfabTtsjexCodr0vpRiWOuHuh1FRxqE7TFrUzQ2FWiQdMlLyoA/ZPabiMpiyrRzU6dUqCSwRJXBCqJVOIE2zuPEUW+VhjOlbBOMHJCSTB90qYkRoOeB/czuSoLBuOSmVU7i+1Moo1hvRAtCxpDohEom2Q4ZnJ+ZTZR3xrBAtYL5UYAaU0KvzXwbARLHutMU2XiB+dNk+MQABIDWHjCslPtGyBXyY1jBNtSSi23zGw896gnQJXdh3ITWyBCTvTAenPiJH4M5snHOzs727vj/dYeaU+21qmM5JawDrN9Khe1d4uLsKLbsMZLlkzwsquSKcV7eBnZqGqZ7NV0ob1aoazTpFYQWXtZlsv9CnghGbWGxi73aVs1xAFTe05CdZyHDQgxSp1daw63KMRLr5645pbdaPepcEO2sN9BvG5LommIVdfVNvazRW/vvfwweoLry6tbDz8xvc7STU+3Nne6jUbR9tPn7757XsLq7X+p077ZI3D5lJj4xMvKmjxdu2PD95+PDvTVNqJ3ZJ8n3A82zbshf8pbX39zicbazfDB3BBNomZGcXsFWI62p20VttLL//o5xeb33/zD/b27tf2p4gFcmIOMaMIGYSOlHRIUQH1okVWJasId4D0SC9dovDngt2FDgX5/VOhaDh3NqifM3mT+KyX+CRiZThPVa8wSMiAb6QoqemQpLk3OTpojE/aylufjDnxIIVp4selNMBlc24Z4o/RVguaZbmz+mksaywrn4EpsW/0ZKH3MR6WAyZBr0iNYRSh1+wXOg6zRXW7V6uZm0xsEJGrWJsp1mEk2JPALCZwrueU0dI+yYpByJ3Qg8/RuKCq9UFhDzIjAyEO6ZV/MSHEJsZRRExsnqpdMjM4rc2fntpoRAWTpfSbZHsyTrGROmsBnq7x/ThzUrqEqr1fPx9ODsZzyaVZSFMS32Ugyz86SLcB0uJkzlKOiyvlqD4nTn355q3Pf+YzL3z6M5319cb8oL24IDpRQlOvO687ycZX9yMRI+ans2Tnx9ppe7Ff2x8xItXurv3k3/535hZbv/p3/i4F3mhobWW2i0M/sj3Rx7DLUUhlvpT/LV6LoqLh1e9FSsoa9xVcKkpuvQRG/qQtvwQKf5bDGglJB1EteDXo5oyEpDpesihhGfpcauRBGKkIc6R4hNSVeGGpxqWeO4l6rbewdzg82j5eXBscPN2ajoYrX7jx1//iL377je89ffwWkUs9BG0Lh+PEEVcNRF5chlvZpXUdxRO/Xh1JaUAX9IeduX94kGF55vJfqKpfytUL0OV3UChHBaDqs1wuAAKYEvuaKxlzgaOBa6r6rK6kmfyaZxya9y2U/5LRBsAmpWCnFQ1gDjiajlzwwapPWUVh0Omtm9zpDpfyePlq90zXnPrPazxlHQFKeW1pQ+/KGig9zpabeYdGyTxpu+IvZR2WttN++d0frB3Os+iIQKRIYhGWa/xGXpVqrXkfYuozUm3pPJLStE18jF4xQ1rLCRnCXZLezQgcY6V/TMfQtn9ydv3g6IbSA/3e6uLatYXFW2zOywurvc7ssUT7U2NjVA2w2JTF08KlRJWct5Sv6mNDnD67B0NVlYY7B9u7Y//2GBuP8TcbcCc1VTDSyVAWB9xA9sYn68vr62tLw+HetfUlhWltRBvXKTigI8dnO5t73T58GU2BVVZMclpYdVFyTxPZ3BcYRf0OR8ZPqIeJYsnzHCqoqWgH2+1MQUVPA16M37JQ8Hm2pEbOMVEeiyxVLdm2rAprBIz0ayf4q0dUzvBUEmOT16Q4RhO0LB1zH4qmuB9yQc89pDbwRclJGcl11iOc8XR/PN0dyozqz/UePN69/+a7Q3SKVqtcEnPraOr6weRAzPPywjJeL41obX19ptk+erK5tbWp8zFgl2w/A11ZtEVFX+D43o6dDFVXMBZVr5vjg/0zNC3kVSJla2l1abiP8mWkknEp/cizQzt07tXBwub21o0b11aurT54+vjt9+4L85FKnDoLIT61J88eyOKV8IPgbtxY6Xebo+bcymAw2aGS29Ooc//dB6v9xWGtsbU/kpio7cP93d6z5uTN+qjX7b1yXr9+q86j/e7uzFLj+o99pnHe//bkK+PNPVvSwtmYcc8PafJihoebO93B2f5MY8G+s8sb2eie6jZrxwRllClOs6diY0+ku73w4svNBw/+5PGTby9QzIR9sxfwyzKZ2N2oWTs6PZhTlrh7Xu+RQ45YkkN4TLm1RbsHu0ISTZZ1ZZN5wImHn+SWYhkomMWUQGcClIoIcW3iVPWxvdKtMRGEp4rm84QyfjCVH50w0lBSsAbb00vGCiYHVaxqfkhLlNIqq1XAgdUYmMLRiAthyHJAccIQeY+7GaUuRuJZTBQdKeQvaBUMhmacgWqieBQn0HmiQRhumhIbpu5/vuDO+Bu6H7m6FlFUX+iPqIGVQHBMgKuoMXEJFQ0IV4jkzVWibPQUjvlGBGyfzS7Mzq2fnq9Opzbf2rCLRw/xHijsLFiOHD8VrnOy+1D62cnpHmJFXuSmEvYNu5AgMesWk+Umf/r88Gz2sEGU3xfyoIOywTq2MpyqiHb71U+98pnP33zlNfWiVUasS5pnlyKBq0TJpRsOEjMX2ZKAI5RRr0UbIRijybi32K0NZqePH87dGLz+V378G29+/cGvf/WFlTUl5aQVzMueHx+z/t9cvyYGMPIQjhPpJg6rkpFjlaAUaHOELvBCCUnWoYeZCRMVgS6cBJCBKHQ5LvWKSYfAln/+ZD4d1Wc5rb6xHkROComv/qQlzXiFWYNj3qJXbmZqFnghQJXxICVLlep5avaGqqi1N/hqI82zrOHWo72DGTXu5o63vvno85+4/nOvvPbt3/+dh7Wtm8t3H2y/dRQRD9cXVwp2cZ9ExCj8QQ+oHPpcHRnQVZeDfO8f1T1Xd77/w9VZgU6+BTXL4aS6WJ08f7264eM+qx59+LOsU4986HpQuHrdR33qccU704fiaQvaW1a6lvuzkIDHHx+lgbK0Mn1Vu2UwlSiQAfg/n3nS/ZfwKKSxSG2mMPIvpDLNVizPTpZmHmBzIU8U6Qb4Jd/ohgvWSNqydDFh8jrsC6ea8rxm2qBhsl+Z/ilC9aXTsy+sXHuh2725sX59ZWmpo6IyCjit7+6lPGIqJmgrQ9AUnUZElECP1JlTOUmlG7vZHR7sjfYEF4xt06vcVUKKE4FBlM/nzOlIrFb9nJ02+lxUr9mF/uDlF++ub0hOsYGC+OQwAkZLW+WwJydrMliDYM6m8pTAU9obPYqWXtDakiqCSxDdSGFt6aBVhfLgy+AQ2ZOJOVSLaB1RNPE7RE5JQimVgGCWXX5RJrSiSETshVmg/gljQZ4K7YrRN05lUxOIh6A6Qopp4dhFgO2LH9yQqjXxCDMv29lEgO8w1eiOhrZKCPGPN/f8fH5+XpIPfznzddabvdaFG6XO16Q3Id7Sj2NcdCcoAe7xKMk2himNL/YHLMGMY1eEa8pwqhxMVZhQtETUJS+sG5TgWFpeIlmHMiuGMFdfWVlaXE6829LS0vj40JNKfoZdKV7R7XqX7oXtKPewu/nyzdVPf/K1/qD1ne99e7izfX1pSVz0k61thRjsUUouof1hVycHtlCrz462J/vvimmqH3XYQ5v2VFpaXPvUSz/WW/yt//7X5kpNE+PBT5SibdUVbpqcjHeHs09UQOsis4urteZ8OkJqtPGiyHGOVCbGWTVLbiyt4p31Z1tf0cEOOLUG5kcpvU592pvvzszZ/ADZSTw4BLPq/CsTZIFablZnFqbWLFMszrzA4egC+E9RdcrKo/CgWhaPTh4qagz7JbHyguOlDazQnl3EX4psZqHoGWFrYb0mBTLkTWRg81+YX5wE5aKfNOojfclJboFcaCaqnK86V64GifN/msBBSJlhvaWR3HHh6yzRQTGHpTH0ICQ9WpuvucMjOEHWiLMo834OHchPButbedCMc+H4yqLCl9xsLrQ6K2czK0fTLjZ6PpelXUphYRJ8jfy244RU5m1kKV6hKCqczqLP9JNRgteJzsh0ZXkdj4/3Dk6EVj7bGz/dO+JIVyH62u1br37hx1794k9a4oLsZ5o9djAunKJ9mBhzb4YsW2SLOJuVnZ7HmUAB9xfMpyctiYfT9sb8j/6lnzq6v33/e+91bap9zT4vY5EkvbPZdx89WOz2QxZRQoHyaSPrMlCFVSHRsN5niHdAFmibqMJW3VJIcPVJqrm46uc/w5G+Vj32YM5yoaypUCHHxde81PUpEwsCAxNOZvnMEyC51SRFzg6WXsyCjiulkHpOJGSPHLhf+/nP/fgf/uEf/sZ3f39z+1mns3I43Z1VFnCyD6Eu0CgUM1jlZR8bVFV1xU0fOgpEPnQtXyMnlsE4d8/VbWClKV+rBqvr1dfnr6eFCiJp7M91ZCQ/fFSQ1GrhApevjfEjR4XoTjL15V+ZjYJKvl76G3JraYgcVm7WTtZJrgeEF7NVKdeZMmKmP7FeZcHGCAXUwRxyQDJeq39taSMk+IoHz81ID7S5Oh47020oBpC7qaInx63JtHt4vHJS25jt3Okt3llYXel1bm0s9TuNxUF/odPqWASITjYniP01MUjZCKBgbT4sDzbYMFZlpGxex0Y3Ptwb2gHv+IBN6tAOPtTfKDVYHO4XxdEsQDJ8Qio/hQwv2h8N1+trVCuBSuCQWgruKdZjYUxYrS++ZV2zJxf7Y6QOpCuqOD4XrqfdQg1seBfGltXLvN2UywiNM2JaILVXaKiOROoVjBjX1NGsYvVYMe902HCkmpxoHCHIu5zGBKxNB8qDKGZyoZ0py9Smc/7w4FoulYFcHKwqJdnTt9ne3trZ29tjaOX1xQ41gk2ijPX2nFpaeGTgKKnRFu41/G9ua2srt6l52WRLbHM4Y7RBCdKKBWoIpydUDTw78124X3T0EJUMu+ztN11cWOg1BuBmyLribgxYTtQEux2eNdvt4VA60ZhMdm19fX55aePWdQj0ePMxZf3atWsrKyvSn7759W8c7GyTUa6t2RVx9b0HHYNSD6Tf6hOjlHOnG5AGDBq/ll44MztdevSIRSMlvUGGnliPU7B2bb4/f++nWr/8J7/7lYffe7NjI/vW0hGsIJEI4D2fHE03RRsvnJwv0EWXFMaDCNQYQlGkIYaOBJ31+kvL9+bnm4dH7+H6h3LQu8yrdqQfi3Pu9Oo2pg6xCg+GRbEiF1SRX2QdIR0mqgTSm6/QdhQd9gRN/OT/lMaymqw+s5uoAdSQuTR4Ly6YcCWcKQ5YWIThBI1Pkv2SpecP2hStyUEoRi8KM63oRi7DoPJjTnNjeAEMrpDKp+mFvLnnA7Qm12s8MGHAGU7ajQzsjXS4zHbFxk1z2D/Fyj26EmYCw2MBySt0mqxgri37vAsw4Hki8GInAwA4b4DiX4UVE3VE5UnmOt7cm1Xti5LNxH6ydzJ9ejLdm52OZqaH8FJvIqTptS2OePPxBtKY1b43HO3YUotiJ/JWZauZd3Z2+JDO5xvy6pZu3n7pU5+9/dqrteXFWR4Qy9l+lRY4umUZOKJGoCoovBmkVBtigFVWGg2WicJeQwwxtqg6a/QHn/vST77YWv+H/+l//f1vfrs9o1RKQ+S/UEdm3MA9pDECj0cMvMAMxhagFaod+F0dbjIq95UjJ9Wqr77/mT/TygePqs0iA+WHjLIcISUGT7hHQiKHxZE/Gu9v7zw5lUY32EDBYGAQtBgqA23JS4+fvf7inX/jl/7qV7/77Z3apkpxhzQL5hNYQp4rs6p5owQ9Xz+WAQckH3V89NU0mV/8fzUAX3MeQAXPrq5XJ89fuXpPBYurr8+ffOxPl/B6/uar8+pdPrPm9SetfOwIcsNla1cneezqepbcRQPMFIWLu1AdZstUCA7xRDhPKbVS5PdciUocgRz9KfObk9xPSLYQRLckNYhHiz8ScZ+dHrfk0p037vTmX+gMXu+uvrqwdm9hbWHQmfSkNaCL7MMjxL41c95JlUg1nln85qTk4R96khqJ6uXaK3Yy5NJ3fnQ8lDyuoiNTM1pvZyG1X6m/YjFTtgN2+dQdyo795iaHwp5XV1fVNtbHnZ0dAVkdtbD66HOMQtAJbxZZ2ewQkHE0LxZpJI5IoktugK8V3ALJcEFQiqSPv/Ko5MtJF+4i5QBC+e+2mvxXitFCS1FjtuIhO1iU2C0NGM/Dgz1+Aez8uaAJaarMqRchXoVuZsKukLFI6ySRjFVUKt1MySmFHhiPOePk8j5+/BQDJiF5zHV7BmNLTpSgkfajwwsLC/KOdMAhRd4YCBRGh2s6zwpUD8vuUhi3/RqZ6S3FWZlUcy2lyPCnjt3XE/Ns9CpmI2CJ7242FRGzq7xGqNengh7NEduiLeJWll3Z3tlh6osZ9XjClWj7wifvvYeEzXdbL790b+nm9eODvW98dYur9umTs9Fof3k+mjquKJuaonwwu0cJTWJyqSxAurF74f0ffL+1vtJVUHZlkR+WWDYWsCxoWUWln/n0y1wHndndtx4QMoQq9/kLZY3xW5yOpkN5uIlpXuI1X2rV5JjHo2mHPgYPKKsi4ly9N19vnd6599lnm3Pbuw/Gk+FgvtOz12Hr4GRmRM7InjG4TNADlwVs/2ciM3uFs5WL6FKZ0HziTDxnqRRIuMMBEetkquNv/B9yfFEwApw0r3aT7RnHNFmxH2HM8YOSf0L0YgDOyjVLOfLC979GaC6okkXqDJ7mzTnSsfQPU/beMuO5WkhEul0dCUHIUFADnu5i/ihioPs8Hk6lQVxUR9JYaT0th6u6gqdpieEd+uV1ea2lg0vRC7XMI4V2h0llZXjZscgI+T8duexo2pl4SntD7B8fPp0eHwiZbZ4dNDjFac3CzqjBzDSqeJ0S1GeV9EIzDsbnytAp6IAcIRnbh6dLN9bv3Lm7ev3Wys17SzdutRZWJsNRozMgQUeIKNNgzkJZqMLRL8Aqcm3huxm9exRxgck4cCDU6dQmin5MazYM+4Wf/bfn2v/1f/Kf/8kffPW6OOle63BvvLI4fzacGHhFljWEQQFRDs8baaHYgU7gUV1OTxzlltxQfTVfBXeqb3/Kp35Xh0ZCLi4m8f3zCzZRGs9bzxU2EGgWtDALWcTT0eF41wo+HO8k4qwlA14pGRJT9nxGvUfPdrqry3/1L/7s3/0f/4fJuzOPDraoGdPjLfCKvFFmOPNdZEGD/XgGfNnXP+vfajCXo7oCkMcN4+qzGnb1a3X96rz6+md93eV9l6vl8vvzf/PagilZIAFjFlIAeSHJBqddzf+5LZ/lkXy/mJvqx4uvfrzqZMTqaly51b+wDthZGGDmS5ROmH4JhPP+pIIW4Q7oxSNaXVQqi1IFDAEisq3xIWxVFVGs167r9xZX73T6ry9vvLK4frvZW5QCwmg4HZ96rEHojQWVYiWLUgAKrVBqgqUayxckmap3RHnj3nG/8gycQDQGhmg2K9UNJfRgLOFvqQWQekrJXozW7vnTU4bt2VbKZSzZJPP2OgJN0Z0cE5tdlnsc37Klg0f50KJh6UMkZZ0puilAx8NqaYCqtURP4VEJyHRBX2JnRrkRlHMxyXR3wUtEe0UmbBCGnpNGiOcaC8cu6VGJgYgqbJKQKuZK/TSnVH5tgiw6HMNb1AyzVGa1vLpMZWZdj8M/3ULGULWJ6mI/g+Mp1jvioJpMFLaNVh051yGaekjrNd2aoXG+9OIrm8+evPHGGzRLGrgXGJDX5176fhAjuRwO8LAYy1KLMoZiqYtCNsGBvD6Kjt3mJO9SPwgCgjwYVemRidKJKnQwGi6zdPS6vAMAOJqMnjx69vjJuxTf++++zS2QNKmtJy/de8G+GXdvbSgMKmrZ1hGJLphpjPb3JmcjytJ4Mko5y26TNx2EFa20c8XT997rfeePF7pzzUEfEam3FeViw20KHzjYfmP9c3d/4va1r/7ab/3JP/8DBZp6NuKtM2xymSfS6WwYNdAAV4XdLEme6GKjYvJSTpODTKmo5E/2usuvbUSUbOztv6WgaXuxK4XidHbcAHLWnRhqMEaanVmKkRHHLAsPVLKEPFqbpZKpHFlFQR+qalz+hQuZaOJYoKwvYYmFW0vMka1TDKIIYSS8ymhY8EDrmFGRBoMLOa6I3+XXso4LGS9dC31In0zqxdr2SEUYcITYhzPdFm9WdNhnkMGQLP+UJ7OKi00ddlStRFpLm+U1JRgnDQdR4bj7PQsa1eE2Y8KOaPZeBE2jiZq9mMymx+P64Q4YAX5jdjMMmTXKbmDTg9PJHu8ujzgziremPTEkqTkd0zBvQ3a+muDL7ZNZ6f9KZMXQdDLX+tRnPrl8697ajZvd+cXWwlp7YanW6gsBPpElRj1j2UoKHY1CeJITsWxY7BUju+y2t7GZNVVLzLbaoUS2lpR2SH4ab7d+9kt/9fj48bOtJ+887C2u25T6YGfYC7kIhB0hjCAamPo/ceTV2g2ALw+Dyo+A48SCtt4Cn0D4I3rzfr/+lLO0eTG3oSbpTD7TLSc04BDH0JXiwIzgrTDkFgzb33sk50F1IimV2dQanUCvlI6eOd958PjuT974Kz/z8w/+h62Hz552pNyJzVQRpWABFNN4wYi85s/PgNPjjzoqmF3+UgZWwOSt5ZHq8/L39/9+3PX37yhnH3dbBbUP3fzDXy8fz8qqBmDWrw4XC7gvLlU3V58V53ZezXp10YPlJLIxkpuvMW65t2rBgspcYg6WYrWYqxaKjhbZVuWU8nY0dlKn4YztglpbaXTuLC2/srJxezB/s7ewUlc4ocM92CHol61X0B3EE+ehgUJyu4SLZXnf7EW9pboqiK6yRKn/HjolIjSbo/mXvW1E5NgA51DBqQiodk6JfUsaDE4cz1wSZNWI7jA+k3ftn0PkXltZa/fntrafqAUdjylCSe0jsOBXDMUJI7BSomuAAPQy5LJA8DL3+hKzFUU3qjHaciQ4hclNGLJtfhnrBDIkGkt8IaZrVeK0/oUQkQn4REVQXh4g5qVebU1kQrJCqokIBfSVZkS+KvwvHDDRlJcLLOskaylCyOG5jR0OD4fj0fCQQYB1uhgYq8kJBdSB0QhLSadQFk8BCD2YSxgDphmHuAC+jheBAzHttrojGxqMx6SRwWBg42Fau/qO5A7mBw/bJMO82GcsKu94LO9QAArF1Ey2WBuac+5nWI4oIAGkbysbacfShebZcpkr2szRq6u4y/HhwfajB93UT4kvuDncvwABAABJREFUoN9dB7fjyf5ojzHlbKAAA2vv0Rk0wAdYMFNKW+kMzvtxxI39xw/27i+v9edr7QUZpUQREdr2V9ucHuydby6srXz+579oa6G3vvLtJ9vD5vlRTx6JUABgI9jtx1xCn1sSwb56LftHN5X2bgqLslH9VGbkSUultFb/xRvXVBdQ1ettmGbfo7n24Ly+xxUZPc5EoTrmJxQ0XCl/HaYyimLErqAkL7xCQnXh1qJYSK9mlmhW2F5aiC8YruROz4u/bzbPJ8FG2E6u82vRq8Ivg4XVK6q3hEuXN+pCfgtLLiQKoOKctnZ9uCVLvjxehOt85UV1u6Wey6HJwh69J6/Lz/oH1y16lNr3tJXmg5N5wQWtKNJZYSTYSWG9Yd/lCDSC0vn0C7If5m0WxeEnVG9UH+GB9o5QlmuHMUD/eG1scCRikQRCwkk8fdEGiPb5Rxk+Vj9xTh1ruu/ecDo6mhVp1ZkfzC+tdlau3f7kZzuLq81On3ViKhOOM8mqbippjvu2mIQAI8yOfBmxIkEhGVLWnk4nyLMMUcZgGQn5YjpNhR62OO44NjXBYWeHaz/x+X/tP/xb//3/6//z5rsP7w1WZLYnppjcHFCGNGaohQozV1wdZfFefHv+/OqGAvD3vz1/9tH3Z25DmR3VDcZRnaMfVxf9WF33mf6g1xGi0JU8hVhPT5pPn707S9zsr8zOtpkHYqKga53M9Oea23uHB2/XfvrHfuI/++/+XqfWlFKgXP4EOhceUXAlubxa1fbHM+DSs/T0QwdgfeTx3P3V2C4/C7JfPpWLoZtpJJNYyKjzy5svRv6Rb/joi8GAjzvykwF7ZSBfvSWgxROzYlwBWj+U1KHckH7oW/nMSe646G3Bk/xQlmQ6XyjHBRICJsQ0DbiL3WyglRvy1vLatBOXSbIvIkUV0OsEajGoz9iuc6E+t9Hsvrq48Zn1m5/YuHlnMJ89ErCiBMCP1Z05p+325njvhDPMMOHS/xJpAtPLPNqB5UDoI7OzAg1C9YTtBZJJn4maeUwulVvCNyeo5liWD9P0OVWY9nYsxCvxGeQEQ5mbbde7XK/2WkDRhNrKi+31FUDg1hFow6hKLs4S9z/UxBvrroS/JpqJUpzgYi5T3A7rT4RjZJBo7Jd4j/2cN+zN563Z4NeSzp4MPC2RHYXSpKRljrwj9mfwKj7fCp5Z5IhumkVhc2vOwpQjQiOsHo9i6ihTW+a5zKor9IhkAQn6ooRz+u6Pxoec7xrRVDzKiagJf3Dl1LbHYqDUvGB+t7PRe++9p5N4cDKRgFnmXywc2cAwGh2/lziLTH6oVbOdjQoBK72aDo/53NC0bhOc19c3+v3+22//QLpXVedZfzVjWR/bQ+BwdGdthev1lMl3yvE67c/3OJvNqNLM9UHXVlHU8iUbUs3VHzx9iJGvLopsVhF0ptdpLQz6y4sLSpjIGHx0/3Ftdt80sambE1osocVGhqLdnr7zA/r40q27p+3BeHxQ7x71F2/dfHnxwXdkD++t3Hrx9V/80X5v7ntf/fb+o3cazVEbZbYv/RFRgtH6bH+LyjteS9DTaq22qqSk6pOFulBv26Pt487CsqqZN27W98adk9r9k/NnctKVEUvNwxyEumJYjWhqQZR1WH7IrGtIPPMsC/NErebkATOHhNP4VyQ/xDsKVoM7IfhxYhMhgYBiuwhyTbvYh1JaGamgQj212AoyFSJUaaswp7Bsr4xvL7+Xz5ABurq5qCQCK1d/4EvWErwsomdETBQgq0VvE1/oF+afNEJwc2cQMQs/dA6uZWC4ZHmNwaUfsQcx48SEG2wPrj53aD222RjtzKqoag3Ba0pX7Jz8GDV55OpTk19BPVahfOpHYYaRHSJzAGP0fpVEEZLZk+PZ8bR5YMcGhrP5wbKtrW/c3Lh5a7B+fdrozbZ6Z6IxvE6EujeKJzKWRIdaCRmK95bFxb6u5GfFaCNFBzyhkV7GZW+NZyLIrBm8bje5kBu1QY+PZKHV+/Qv//z+9u6/+Hv/w/726PrCYLo/MclESDMUUOq/x8qqzWs973C1ulLgWa4EkAEJoDnJYv0A9DxRHUB2efqBv2nvuaO8JO/JG8pRTkNtItPrVXIsYamvnjQ5dAOlqQ6fPn2n019ZXL5Fei0RdR7nET6XcTnoNPcfPVu7ubq+sPrO7qPt8/2SjA4fg3f+FN4DgEGPj2fApVN/jo8CiCtwFDBdDKzMVHV+2d7HQO3y539lf3VDl9KZcjiB8Zm/fFwcgURw7QO3uTPUPHiUCSstFAOc67lm5VhpnoKbmWlfynIirGs6GGlB+KHgUm5grytJjxAuzuMs4fNa5/TEzpM3B52XNq5/6tqt15avrzeavcnprBj9UkBjVlppW+XRur1PVSC1NOZOZpVf4jRNjgEZV+ju6CBVCGymG6NcOGMsxAWHi5gV14SCTNE5CWMyDku0Mwtnknv4RVlZUOeQq6Syzrc6u7vbOozZtO0MkAwfa36sFjQtTXxQ3JwCp0nXiDrDMx7pWciacUFaB5bi1EjxfybDGe4nyVDsZYzjWajNZmprhHZkn9xY7aX8Ek1iFQuGaiCpV6E46SISA/56AuZOHK6YnWrKyolnQ2lBPIbhEF7/ZUXrZASwgAbtQKLbnj1k3S1+6ISDsufZGCezRcOIzu09GCwNdWVl+d69e8+ePd3e3n7vwbsGRv29dm1jNBp6FyZOPmZrEKTkJfaHIlt0+nX6tLyq2uyEyjtYmBdyPlGfsggNdFx0V74vNVfnS9RaxAVdAgr4QU81mwcHZ5g0MYYK3WImOGnsPXtmuwek7RgDnuBk5512Y2CT3s7c083HnWFbNMj8oLe0OB/LZaSHfn9h/vHjZ4wlszaBBOaxrM9avTer1uQj2lC9Ob+4AECHw0njeNJf6KlosbFan+4eTXa+r47LrV/4FEXly//00TGntTBVsWlJ4AZ6zo1NKld2AVRd2dTTbmdp8zCOW1nFjL6t65TObS932iv94UFzQknjgVSBKQSHvCg6IhzJAkKKQuOyyvLPSYicuTebyU3lyrWsKvXXjIf9dLpCggBAsuqMvG7RWNneJ1FGMaAAKpHD4xAiL0h8RULkrlhq3gBJ8lthJJEBik6CMmTxSK3Rk3QkEq7PXExXcJWU9YqnT2Ba7nGNDmpti7hO9JS1oDUqK2Lia3hTvucr3PKaXDQuKCodPJmH7LWhA4FBEFXHnHlM01lKuhEuALFL1KGfrNCEL/tP6c3clgc8rBNedsKbIHhZ/r8ndJIsQpFFNWbG00Y29Wx0O4tLqzfvLN+6La+30e0fqwjboPtycxJOE/lOKWVFsFMVtpi1CF+91atCOSMrZjWVd2b2Mo9erZ9ZYwretYWJqTLK3Co4LXbA8+3xdnN5IMihtdD90t/8a71GEw/e3tyfTw2BACYR1abTOvRf+EIZklaBK4AskMn1jzgqIpc+/M8+vMhRkREToyelM4Ft6DgkTAIWY1i6acf0sb0lx7t8fLaTgjqZCzPL9XZ8urQ8GDYUuD39xIsvf+u9N3gBR6cHQangVeXByDfNOYr9MCcZcT7L23LyoTHrjhlx14duuLyt6m4eLEe+lp+CHJfnlz/mL8x6/mv1uM/qxE/A4TOoWw7rq7pSfVa3+YQgVzdXd1Zfc0P+lWtBmrRW3RpMghzVLxaqpR1i7/ZoT37VN28xCbAtaUW6oTsehjKWVNZsUMcIhDK6XwuYrh91sUeWVOI9ErsA9jhZg8gyjDyjvMP0pJMSrrOLzdbNlfUXFhe+ePvGSn12WfCqqGMVbCf7abl21l3EFWQhq1TFAXcOs5PWcyavMTmSE/FVMTPLUmasA6IiBxiEbiBtVceRK0vVDjDRfdWRYnm20a9cWjqyupGFsiCNqJhsPsFTMeEK5xkJjAIBbqeF+SWxzwYulmZO7mdb2ltL+/ZMwJVbbQyp0+z2RgkemuJuDnsGWMm0Ljn7GrHK8BYwxLGhtUk8se9DJr9AWAdR58mhO2SgqzNlhXu3+9l49Q8lEE/rP4fGGWMxudNG67h+zMOtZVQvTlSQSoRT2qW4226BeuEQIeyzo/pDCZg6FMJoz594XMEpQVJsoxYUo7xM3+BV2UPC71gy7nvz5k2c2Ktv3rpO3BFIPJkcPnr0aEHRxV4PV2aV3VgUnHUqaunp1rPbt2//pZ/5OVz2n/xPvwbON27dPDzYtaj7gy6PrG5SZ59tbz19+pShVpcTVHksXLmzurxCkzs42MPgIWav30sI9dlJ15bIYtsaszeub+hAt9eRFc0tuP3sqc2K1tdWbDprwyaWD67zve1NdaWay6vj0WFjuffSq6/cf/jk6bOdgTECc23Wzsej4UTIMrvfkwcP8aDrd15YXl1DH/a//dXBvHoL1xqrrdq2nb8VCekvfHLwl9a+9Lu/+t72uw86MwsLnWW2GAbFuVZXgnBtU075/uLZeHDtbmu+rRL0UdihqsLC9AanIyE2wgiv9W+3+pPOk231q0UOorb57+z0QN0TkhiLSezR6GgKR2d9WqWRWQWf0bQQfQwx+q7fim8iRMs0kW9k5cAm0YokqWPpe1m9YhGb7bNstAUxslaZBK087Zlc6JFFrc3oc+SJoE3Ii0cvWL82BH8zvoZ0FD7o7uhAZF2bZYvSFlanEmi7twB6z5RqfPJYCi7qWxoPYcZEr+ho5Ep9PSEFyOXJjrBKMyBFrTkR6EYVqoR3OfSuSH6kCM5W1zxnhSQReOZsjAUkNNxFTFbQZH4LGGiYGKfKksmwqzGRdGPib56PR2MSQT2lsroyJLanR/vDk8HitZW126u3XyBXWrqi2sRP1lXPPu/gsqCpEhuoK+xikyumsLKbBQdG+gIWUcgrjpFu+HpBlvOTI/LtOQOdgXAyhV6WLuKrre5gTMhvq3Bn4+f2Z/+9v7nYnf+//Z//L7cRuPn+wZNtlVBvLq8J+5IcL/39ZLRfvcc7gCf/l1cgn5m7cLp8B7S8liENg7m4IX/c49NRddNJdeXqOm/T1cXclsPfi1nzJYSD5FAdwTOqDiGBDBbLDbHYvWJmFpcWHz1+b+36ztLqCym8korcIi2aqBV3/fFwPDvf/vQnPvlf/MZ/O99bOBjtlZh+r7oEmE7rqVKvefnzh3sux/D+5TyY8V901g9Vr9+/4/9vZ5bQ+7267MUPX7n8BS/LlFTgdeLI96xP1CFkwJGLbiyfvlTPRv8Ko6vQ0Q05jezmXyRTokD4x87ursWW8r+qKMcGHE+fOv02ZO+dn6/NtW/15291F+8srr584/ZLy0v90V5fwRVzKBWBtZgRLGu/tjsattR9bDf6sjdwK8REJo5yFDb5KooaZQ3XieSQHkLPsGFnpZPe6qsfiGx4MCOuUrHl+QT78P7SetFL2mvwt0LhNIQwxQKYIWOi0lXt+tPEdMWZ9roszrJodDS3iR4q1Y/Zc9EOT8SkjK9bfYW/AkiQtQhIuB3uC7HRWuovN58rGJVKkYbkHPe150+yZnDUkpMcDRXpKiOtWvRST6Rj6Eq3W51HdHr/yMSFpEoLSEg2Qs9ViATbSsQLxKDR5CJv+TDL4KPDTJXOTRlGi/U64cHl68WDFxfnmQHWN5aBjr7LHI2Vvv322+4RseVFSkbbIUq4mV4Nxw9w1m9+8xtGSSpaWlIetLm3pTRgxDY3a6J6qekFjaXFJXAj7iDTJrE63AZKmUekHAHGYmBV2FJi7oSu6jzrtib39/ZM13J9uddXLzYGjEM7qI1GoH5wYHfJ2q17r23cvr27R0zf58fGocA6ITkpCTVr/4LH7763v7W3IoXplgS3a7WzcW37Xdr22VTEeP3kMOE/9fbhz/5bP1Mb//gbX3njB19/t60qx/Rke/9pr7Vg35bR8HRmK5Z99SWVZGoJCDwlvPWTiCsdeqbTGLLY2PLo9vrK5GD6PTYI4aAKbc00NQ3d2JmLOBsulH/hdkZvUQEYhR1ro2KLsXIR8cW7gWY6IteFAdTtEZEyzLaLSDUhuJcIf1JpQtpiEy9UOPyvUGPAL2ysktkTWABb3FPQh+vBuibd6HCIvOVEZGS4SPm/OUaR2vrNVXVSWr35Jvc5m61MPg7O/eZIMLa1pDtMKfpFx09IRU3ku7kwoeKCmY/r8DxBj4XFwj5IaqjQlzgQtI1+GSDE+FOYPhesf4hM6FtIfha5hVq4helW3gTa4pSCK4gZZ6d2V5y3zIZHh6eKnjSa/MLDw9PRyVljfmF9bXFl/W5nQeHQNZt5KQoT7YHM3WhTFPDaRDFmBKkCn6mAfdEzkKIiDQRM/i+HkzCRLJ9CalwssK6SniOAZAHmVl0P9yR3h66eJUzOE82Fezc+8aUvPPrm93qN8/bqwnTnYHO8P2i2ZCRz9lT4X970p39oHVz+JRpwNcXPNwTKLlaf1fXq6/P3XD2FnMCPDD43Rb3JwGMBOZMoaJ8oFCPQ8L3QXghK96LsCH4cLDRfuHN3Y2HjvYMnc6n3LFLCAe6kYfAxqxEtnmPAmWdXyk9OfP2h46JnVz9dnfzQnf/KL1Svfg40F0AssAnf9Eaf1YlzmP/RfTBjsMQ0lJ8D2MxhUKMCS+Bt+P4vR0Gagn5+KCDxqEnwX2zMuRNNz4qFiFM2zeU2Ti4yVliE3BTZnc2TU8Vzr7f7t3qDV9evv7p67XZvcWmuJalexWapOw3RJdghhqgZUgDWRm7HxClAdARcSqjQ8aQow6KBc1vWI0aSaQocMmqdyj/dyKIti1u33BjJm5IUm3O0ZkUdaQ02FiiMj3gQcp+hECjytIq+7a7iEPODRdqk9SBLdbC40O11x0kjHhNx6bWEDatd0yywCaDGnBmYnUcckvUQVTau4MTPIEACuE1HIsfUM65gH+YnQyYg9NaaOlfK+lHdLyr1YL3K3eEZpTy1bphQ1l1kotmL5i1ISugl1h8aBxo5tJR9EUyJLwUqaK6JCuXDQ1ElBF3yFRBRtCkdOn+oQkfMzuWR2dSDXLKPxeqq5GA95F6VV/vkyePJ0Vj81LVr6wKmhUoNR4deMT8/ECDNQ6wqFiVsuL/zgze+7y1H41F7Yw2UQMQMm2LsnNBAKDAREYHOzn2NlzcKTboKJ8ndfNRZoPHNQ2lzy12e3dtdocAmkr3kH5vnfWViZ2bmlxfBWE1QL6FQw2TKcCCgdvb0aPXatQePt5S2rh9O5vUC9G0AO6yp8duBFaznQxlGxzVq+Obu2vq1rb1hXOLNDl1lbzRhA7GN9LPJ3t3XP/3yzb+0sfHGd3//7ac/eNY+b6/Nt/d3D45GKaqEvCKy3aUz9u0k/jKpIDXELezBPrxqXy1uzPQbnUOJTC1+FHOE6BQCPw6Pi+gaFzwmEtbrXyYSNh0bU2EKgQs2WNgi0OwlxocQouxiKxvfn9js62hcz+6rE+wuMlym3KJMW/FGFHwA5+oAbYqqafCV/pTXXfyQc9ycWB19D7cvPDXsTP2yNfsNd2rdQa03EJU9O5kOmPWnSwqCyt4iY/Gq2DRZJDIOpvSb/HpKEUpTzC0hOWY28dpZslrPO4P5wdm8CBK7ITYoKMvVpGvZBNoOFMEDckopqUeeD6mpgsHtgsbgS8aVIQEZ640VHpaD6Takt/dFFn/nXIJ5v78+WNxYWL45Yw/IVvdUNGSEUsqXx1MpFje2WksHzaXXl38XM1GBpvrU9XQ1g/GrT8s+nc8JS1YZUiaxLKZcNBHl9ghWVPKUPmjMLb547y/88i/+12+/++BguKzePI/M3ohzi6lGyQF7vVQv81naCV3KUa1xoy/fwM08FthV3z/i8+rXq5Pqpo98MHAtDeZExwtD8Vc0AXJqegua+DHsw7ShpCTt3Fq08MJFYnjp9DoTxn7bkjRrd+/eZBX73jd/YI9rUTnPHyY8eBkf8MWbAr2cBx3KSf48d1J9ff5K9WBuuYDJ1S3/i54EBFcvNQWZmOeulHd/qEvVI1WvuEcuF1wu+AkO+UQcgyzlir9a1Eh+KhMPTSPlhD8CWiGNqTBcUDXMm/CsHe4hxQWbRPDsgXM8nT+fXWu3b3UX1put19auX+/17y2tXhv0e3j3+PB0rOzwZNCOlwiuUXG0XgIeUSd8Ro0LKfbTIxvpHewrtldUNxJ6JZVZk1mwMAQ3dJD00/vSf8MJVwpvpSNjUqJ6/GM2rtQDwj+OxLOESBUhAo5b2VxcltGJ2kzz19bWaYHR+0ZHS+uLq2trUPDwBPfFDDQfvd+7vDdpViGi2kIKj3QqBTEFdamznA6SG9MTFFWH9R7DwAnmbLnbbM/OjWZPmb/CtBl7UWhKMEKm15KUKO0ZcqCTdzkuxxWvMIu0z6uL5fd86IXFquAlMEXV52uzYuhpHLSdtjMjnYRYW1b2qpDqm6ATNlo23l5PFq6SVt14Lc+mvL99ReXbrZ1dNuZN3T88XAQTr15dXTa6hw8fvvPO22pCrW+sitJiK9Z7e+Ml8yd8cBKH3fTIYmSyXlpYxF+FTY0PhoAsNBpcdBjfDevlnraLnnBoegIpJFZFiI6rcjEZaGgtMKYWCYshATLGQqZ+xc6yM0vPV3hoI6Z2ty3pszarzMrKjWs39sbvvn2f6Zgll3hOdZxVUk/lQO00Z/vtObEBT+4/ePj2u3dfuLe1t8+zMdu2VcLscHLUaHdGi4sy37Z/d+vzP/Jzg8//yOLDo9HB+eCs92RnuyOLOrx2/2DvsQA6UB1oWwp1ijDZkRLnyy5AZ0OVGToiCuaaL7pNyp3IIFEKtqEN8W+YY2IXZBa+p9cmigiXlQanIEZoMHjE2Cqfiwh2qIyXsEHBQtE1pEjNjmcUnDwfJvXJnsj2aS1LUV+gNrxRf5QlxKMJKAhENQ0rUxKwIole53DFP3Krb42OHYftYA9ebACNdr+rfFNcxsZWV3PK4qyB+OLGPHfT9GCRM96l3c2th+/eN7cd2/R12lOyV5gRyoF8WwPRDMPavSzfjS0LKb0p16Ct5QLH0jcnYo6taGgZuQPqE0ZUucFu3U5vjiMY0BrndtFQorLfnFtt9W7W+ouLE7tjSl+Y2o53adBvteY73aVma2Fmri/x7ESmdkCNdBGOi4XKu9In6zY6SFZsgc/lmnOtHLnHkZkpJ9TB4gDOlVxHcozO4cSVkM5ymAeEFY3C04yJgaW2PP/KFz/3hTd/7t1v/Imk88HczGB5/vj4bO9oTNApC/7i2ct3paHCynMSoGW+LjqUSx91VLf55UMnIVAXQ8hjzrV09Vm1BHMu2k8+P0nIUow8E9jnbhOgOmdbPVVmktjADTYV1POlOegpUkN9Ilr3FmpE9sk3j2UiXWBb/sCyoGGgJZ6hemWwr+rW1YkfvM3XyyO0+uq4vP78DVc//i90AkyOqvHqpHS2GgkoZZYyUeUqNHJc3F+eunjkufPqStVk+HD1wOUrckFTFQAqrCzwqIbMh4Tw2eEUbhXkVQQaKz3fffJYsMzKXPf6/NJL8yuvLa+9srxxs9tbnJntsr8A+XAopwOCzglf7zfoEBYe3YQtKYy8KDEoLeeWkF20F8VW/t+0iSEivIqWDTLDCbNh6OYtVKosaj3PkExUOTARRkFKKgOsqlIUSLpv6sAGlYqhLthe0CvEPkvE4/EoVzMfqc8Vw1Pxikrel0YiMkhl5dj6KOIl+Cham14kTQIBtRLjEk/2YvxsgWi8zBEAwo/tCjzX5WCKPTGSQ2x0WJp6b/BXlzISokEK7Jd/2ohYUNkzaN3xuKtWyTRd+l0WvlGWPgcaEWICAs3iYbohZtbeYnmM0tmm+HERctOAAmnBUJKch92aQ0lGyhGsra0RBfBbxal4fJ88fehkd/dZSnjuzz16khKLiWc7mUpMMjp3mruEcakCHicF7VZOth3j0MkJm4i4YWnHcdWTh3h80tsYHXi88F29jBYSykwEhBMpAJoANRAKOc6Aig9SojWzAtZlb8eprdLmB/3+gA2lx04r8YaYUHaXM0FefKzK5WK93r995/VG7/7b794fKaxJ/FKbUKzW8uGhvVaY5E2KWs1x6hNY3nr7PdaE7NdzsMcASU7iJxjOnCysLT14/Oze3v7S7Vcag6X9qVLDHBCoj4k2O1Lb9vfPHojwY6mNR33ldkv5bTzkGG1hI5G+DjlmGkvNJDwxjbJKY8DEhnO1usas6Sm0W4aK7QSvw3ghCP6XvLJkJUEkSrAhnKTW3+l07A5OTO57tsDz2QPyBvKXZD41jsUIhgjEMFrwMWQuq8JF+BifRN4mMMBtLgZryhXXfW206w1I01ZaMcQk0mHTLjUTru9LYhneU5vrzMzPdud6Z/0el4h2B7JCxWuc2qNdsYtsLpKXWj15faRK8+9KpO+y5IwzhN61ML38VNFX56i0/wptTw2lSixgKFCFu/RVdQ4CLpvZDIOIHd/PZ3pnjcVaZ602v9FtLk5JSeeH1myvu9Bs8w+oVwzU6gtIpSDhtVWote2OvqEmZpk4kPNofdEvqBXpjeFXRwigea3oYHWpYgTR4WB54OuG4gvLhUKeCk0Cv3gLZCvG8084RgBOjtWFa95Y+zf/o7/9xpf/8B//V3/vybfe6AyWBfodjvaXOh2uKbOUlrVb3lmd52sAmuuhzHrsVysuilE5yk/VDfletZCBlW+XJ+XbBz7SbNp7/6KWq4v5E7Pz5etcD26BFx+XLIZk9inoW291EbNMGT59POJMT9K/7UG7taW1FUn5IblBtCugOve+ADBhrhcvK6/NyypY+6EcuZIOXAwoyHt5+L06dXJxw+VP/8r/fmT7Llaq2NXrnr+tOvd5dRJdqIzl6qIHA1HIUTS76tdczNhyneIQmBufBwEsQzapNl1ngY2pLEomfgCgxOWTmU+0Fzd6g7tr6y+tbrwwv3K90121MmHUeGiH09TxwT04m+K7ITdjQ81KxgViTTGK0leSKDQ9aolqYMaKbKw38iypd9OoP2XBhP6lQxVzcsn3hCSgYIX9eFFhQam8wXgZjSy7AyUex1hC4/Ivp5lQ8i+64mmqGE+zECcXlSlW7AMp3tnfW11fZPil4SSFZ3RALCh7IoU4Xrww2TRGqAt4qZWWske6GD0pHucgIe4zRWRpbVGKL1Z1+nl0wuXFvIg/ZRsJDuvY2t0CPirphrri32Hhch2O/XZKX/SZoVJ0rpAT3YgxzZpxNdND8aVWns81YuLWUeQmn7RTFT+kEqlp3Az/rdXpvuxFAq/w1AMmB2xy5mx7+xl2yFDtc3Mz5U1+4S/91YePn+xsbZsXRgJwePTgIaZuPslVdv/Da8UwD3iAEdZzTon0me3fnSCp8+bSS9m0+ZLLUraxREZUCHAApnvkBzOC+hPFsWR36nnC3Co6h0Qqpdm2L/C80kbJ/pa4Q4kQGdcZkJcmxwfPdrY3Tk76Kytr1288eeOd00MpIqowC90k2sj1tF+9HQu57ZVCsevV3ObOPvM78NHfdbjb7wjvnI7Pnm7XFq7dGGxs1DY27n32R9/9/s6j775H21JqigIFzkB5crTN+mub3v6pYJ8+/6SIaIG6cmDO2ywh6srM8WTEijTL0mC+xCUYnMgtatgRlwUML9wXbofe+bUQbdzSUisp66AINOzx56Oz2j43r1opdYnCSkoobV23Z8EwIpzHC0MNXpeFeiiVqXA4zm/wnmvjpARnhnahFW7G/yLrMIOHskVkzJZVqRkSBhyNvNTiJzHztNrIUtdSts6MsvKwV881l8K6UNK+cLSXOp2FzfsP95/ttl2yj1YEBXk5mkMiLvyF5tiR5ZvDcHMYqPcXZAmV91ZoimVZsGicmsP1uV59ZkDYIwSLGbF1e6lhh4p3zkGY2j7L0tCzAjotHlWgJnPWhvRkPdYFVAxmGCf+4Pk4Zr09SnqoSLhDEPXiKEpxiISDYJLPfLhShN3qevnMACzk/NPaBctMgpRTz6TArj2fysNSlBpoJTdVCt0v3ni5+VMvfOvb31Olcn/3emeeBCrjAYaH1l6Apfypzi+ulJYuP94H3+WVq7/P/3R17gQp/ZfcE6p/eTgPwa6AkA6FhxKoTLVAcQEA+cdQhjg1lTIUQ0SQmzkYHTQGXf4S+5+piW3wsj7sFpJWK6JbmtKy2QbBaMBedNHFvPHDA636dHHDc79W16vH0/r/8kfVh+qz6rP+phuXMLqYKUDLRR26uJC1m8eqeQ1alJ98jdQX3C/3+gOL8pmr0NUfegBFzc/5IXaji8ZzKQdiNjNDtVrodpb689frzZ9duXVrrr06v7DY7/TRTnR8TE5C/dWWIVxH7UsqHfw3fcdHvVZHLBAtEEE3mcgo6yTljL6b9eeFulnMzNHpTk6ZOfUSeYDilkkImVEiCbDfkkWknBTOhKazXoaTR52mNxbXr1ZC0go8suTKUqPx+FeaYkKnz2Ehq6vr86s9+8JaMxyirQ47KXYlFFktEGpbYzJS4kOvUw0V2wEgh5IVrvDtpeUKsFn6vpSvxdRckjrcbfff9rndlDAu3RbVrNACjy/ucywqCzRCl7HgisX6NCL9jtHnImXo4gqCVb3rwjEedM7/oavoXzMFAidjWwfKwsrAHTrp00jFNRZomElXRG/ZPTB+W3rw8sqiyGT8mIfSTLNXHx5N3n7nBzdv3ROo9eD+e8MDRaRHfN1qMe9ubTMJg/WJtyA0Cbrh3letTrmO+Px4NTGrIlWkakHc37mOjNP/Y07XHxKGap5+iPBAfc7tbpJW3FI11PRKFnFebynNweI6O1hbP9+1kwSPrbyIEaOsCCERc359570H/dV31tcYXwZzrfa0PlKAF7zciJ1juQy7xLKRpKxJFFKcmHs6nvFsWxlWx0rL5HnUEd00Mzw+Xzxv1j7x2Z/9ayvf/tUvf/23f+fGfDf0Npv9mXXFTWwytYlb8EW2V9bn+svnIh9s2avSB3V50khBp/Neu7ZY607m5kYlNJnkMa3bT4Qum6XqX6paZo2VFZsKLYkGP5u1Ucc5hZ2XFKka12cU5/IIjy+LNEnxcKZhz4lpgzZIILQaED6hx+KJmFEE7hXehhASDLgDCJEl7wbKZ8FHn7Km4gPNOkoAlYtZYZGP49HVA/Mk2emcC6MlZ4ApWCE7TqfpDG2SX7YlnElZnGZndq01L0JtvLT3iOVf/tnR1N7FYtotVq4D3dLzinZkXfu/jLSiVqEn1eEHr8Q/CWCGzFvQYKm18feSqOL4v2V7yQ84sYGSnrdn6l06rk2mWhbOMSkK4kV2YfdAUUj7aJi3JeXOIA0tGVPEnJj9z1M7tQK+v95m0WQC0IVcDXsOo7W+wlFduvgMCS1fnZgx68rQQhx9Wo65SPIhO0AJTMaS5VWnzKjtcgaS583DYW2h97mf/OKjN95++q03xHK3e/Ak6m9h+JegeO4vqOTlOUp/yp/nfv/o09xdjquT5+8rF0O5rn6tKInPHOU9Fu+Vmh2IAFItieo0e9URmp3sQY0AZobJNkRhVOz4cNCcHx3Vnj3bROcrxlG91wAd+gQgzsGmgu8FNpSfqjsBURcy/6VzBdC5OfCBtICRz8oIUM7NrCsf+qza+vBnpk7zF6D54K/e+0PH+zPvp+rZ6p7I0KG05fB2R3prTRVMce6o2KrLSG32FXIhyJ3eargU29ZEbJdAG8QxbNBMMk4cBuWahjmv3IQ6niThkVxpP/fazFKrfX1x5e71Gy/cuvlSd/7uqLY0RWGJ9RRS1ME8JclOj0Jb1Z9C0M7OBDijL9ngc6QIcJJqUIr01dLLhMQ4hShHqGfbZHy0pjxBP1RRMj0M46zw3YRkxBFNuStQbpbcxBBHl4wPkWaUf4gjLMH8/BN14VmvMFLjsVAKimiVbDcje3XjxvUbN240ujM7AnltHCpRZ2eHDqF8seQBhSYCOkyFPTQ7IOETep2N66VdJgErUSTBEAdmgw/5DJIn5wc6Ev4CYXmsbGHsU0eN+q76mIWf4z/aYzY3DvcVxlQBJlwzU1ImpmiNhSmlVf/yMn85Xd2B9noyQTxFXgWaWDOT7RQUybQWIysbAx+wC2AjhHhvZ2t/waZG9ko9phGtyJWcPdvZeoqL47L2L0KO/tk//bUvfvEnqLPf+973BCTHYCAzN8I9C2vnrHmyqWD7aCiX2ohF0e3tTNBLtcUM36sNgbQVsJeDqARONtkzQtfYWYk+bNFHtq2Jk6AaYLiEUzPcareFppu4KL1QdWlhvlM7Ojo4ORiCHuMEQUvdjzEzcns63Hlqm0KZTeJB2MeR5OnkTKkOmAbbEgoGZoAFbApZN2zyZEWg1bQlfOu0qRyh3RLP5sx/B+vkXPze2/NrL3zyr/3SznB3/933OJu5LHn04/Jluzg5mIxPHr41uc5HK085Vc3YKxg/eZqx9GRgn9QWMaPwP9rkWfvo+Ex9NOqyfQVqM8zRE1bVWEZITPbTxf+UF84mHftqcYBfq3FwfrTfmB17sRvYrkXdWSgiDeTUMKfymNZAmn0q3p5Ep3drnXDPLA2DTfHxKLgpqpGL0F5DoRdB2CCRXLt48xIChbTkolNvYnmyvSDcTLYEPmIfb2XSu92jYxaExRkbAVJxRV4NpNcudV+exXsPtrbHO3vEUblUXCqWMnnFZp6VEHgh9qVToVsIeOnFBYPRp4xL56jN3m5MCZhWBkNcWwsAzZgwcWNQn6XeWLDRI8mgIRW41kyVbuq62mkW6lwyAKM9xOUOCejCKKA5oFZj1ejCONFegBtSnnBl8+F1CfrVhRwmVw9jakQ4EUONJY6soqrh1LmC7EQ5zBqvOiz6LY+fiQsBu6RPiVrvoAJuI1gS3ZXjuvkzX/ylw8P/6eS/2Xzz3bMWAokIR4rK+syEFdDkdWEr5Yp5ypvAw6+uYWJ5YzEXVb0qPLKaTjQp/fQZ2g/SOay0Qi8D7/zL06WN0lJ6HxgECBkbMBhGyTB1U+gj9wOsF1zQbHdl29vD2xaLFmvJn250B/P7Np2bHPXOagIlHz1+MKkN586bwbRAqrypvLe8ujBgi7/qWOlMoJYJsA70xVMFjh5BNjMzMMJIgh9p5uJTXwsJLF8jGZUx5lJFhas2rz49aeFbj+E2MbLFEs4mFrw34PLG6jP8Xz9ADdirVeK7I2QXYjEMlRKH6VX4a4aY+zUStSN3uVB4WsCfQACNlcnLbo8MS0r1OoOyVq//YnCGsTFucnKqIFESefFiwXnin1qnCv1Nm4x7Z7WNTveV1WuvXrv5opRMyZsEzyP7c08i2Ji54FriHPQggiriT0ydm+1SeZNRk62JMJrhzl7WX2zdZaQ+Y4oKQNJzBt6w2jgTcxPox00MupmKgjGG7MgSik0c0w25YBOXMHRon72x8kqnKkHY67dEJhX7LG9Ugi2D3voq2CNzTQ/IQqrP2gkAzxQMDNmomvXD0529Z65JeRQUZtpaXRoAnqIo5sxwb7eyoJpEsQcKJVJF9NnXeLFCc2OvZ3R1ALRU1sgEqiKXLmTGSSZq24ryxXGSl1Qc1nEwWz0QNgKf8Tr0BiNnlAgHzUzDass1WISfl58Yt70v1ZbZ2uzOq3CYqGW+Xn2i3+F+owPuVyWwRKooh3UsFkxJSszpuDH3+OEj85WI5Tm7zp6qL9VaXNrudje3N13sdbj6agvry++99V0B5QjpKc4rSlyzOIkXc6/byK+lWPTJ02dPYdPa6rJSo6zRBwcj7u8c4gOWlwVOS2Mwa6r8Mvz25rHyucPDuf3dbbHW/evXtrd3Nx9vNnnvmANSwOy0213YsfeRjdRFyLcby6tLsqL33vzOtHbUG3T2slXyUSLGeo3uCSQ+YJTtnY36c0e9+ujaaudAguvesHct3Lfe7c4oQ3QwwtSD88Br9aRES9RGKjvp0AwqIT2nGtXM6c1rSxjp2XRzsmynpu324vpP/a/+2g9+/Q+ffP37w/uPpVMDwdFURPV+p7NcG888feMNuSUrN+7NDdazyfv0VBkwOg/8w9RSK4mxdGF1Zm5j5nxhePhWe26+0ditNZ8y4GGsmCOWh7zXJkfJL8I3MJrzPfbUem2/MWPPH+Z0nmXFPQiT/oMmpQYFdwWJ2IOuQOlUyKqWSFAdageNjNNo7U9ADy60yWIDZKspQm8+ISyBAnKlfmvWR1GDQ8bVVyE5FGI/M6sQFa40aA8Wa/XlRHewf8+xNAo9E88/uvOlz7zzja883d8SCbA0d8yslE1/dXjmeH90en2BT4S1hJDXi7tHHm6jG9k5IjKjh2pqIVJwys7B9WyEZwPj0/pip7Z2E3qLRzg8ZXZemG2IZFRzDQ+er9d7iTnsL0XV9mxYVKmQRfjXb7N8ppq0SbVrJNFkJnXzxmOBcUUWQUBEQ1si4A10GL8YbA5cDjI0uGwhSX5gk4tU4idwKUcJFFEIBRYkKKTiG34N4bWqBN7pzIkQOknN5DoPK1gTz7CCrNLZ+4t3vvjpv33v5m/8o//pv/iP/9PP33659vRsejDxyg4jDcwfj5nvB90eAZoMEJJYqGVhT5F5qguFpudbYbSuFZnF3ahiIXN+8B8Ggfig8GkD+QzJLWQWNMQ5EsqrQcEuL/AMjJiyn4MNVtFEkqaJowiLPjgcffa1z/eX50nKmuJL9KrJwYSTn+Nx1JxRTuekdfrgnR9kXxCpnWihmStUHfQK+crLWEXMUxlT9Zk+5Y25+P65S+WecOF0+of+Kw8VDC/PF0jlWnlVdal8NbBqxGAUMhqIAlssbmCnewaS9suR9ZAjoLjoQHp7dZRzSOquPF+N4uJhRBASwQfsr/wEFwllWFoJf/RQFqnulJ8LLVf1LSle1WV3FvbJriiIWAE/yqQQiPV2+9pgyedrG9fXWq0bvfm1Tneh1NCQYgLv5vp0YyvaSyMT6FlCPBitooZYX9PDkV0ApICkYjOzJBdZgcFFt8uAM65K2gi4SivV8HUuA0ofCxKZjCAK6IX0l/eGvtAZUmMyim92+S3ihM5ksAZXtacvAX5E2jIj3udrGkLxp1Ja33333fml3tm5HVMaywuLSIA1iXuRZTuTNn8wyR+5P15YULwJP7mojCEhsZeaUKFT6V7eGoknc5EChJgxRk+/TZxxOWKtVfe4KfqFOzBSgB5Hj2ZjiKHBEC8O+KDbPn03vzlHL1CEXIkRXA8pXr4QpbE9ceQUQsFnDlv5WsLyuRj4tYnnt5XdmJ0TMAoPYQLxLzNBKTo96XU6Q5yVjGiFVKq7dzlntFDSnjQW1c86kuscTz6YEqhiMKbfixvCtU6n21ubB/vbBA7qWMGpSqjL5AEPALqoswFjJ6oKUNeb9f39PT1JD5ttu0XwWat+ItuErSHbJ3ZsMB+4Sf+dX+hzV/QWs8+6BDOUpd1d6A5kN01XlwcH5IuDzeHT7nS0Y0La64sHs7KTpq1uTxbV+GQyI74XzUUZmEuQ/ETGZ5IKPtCFY4w3vF67s7G22lpoT0/HarHt8x+j8oP2iz/3E/253ttn3zwh34QVNemjEuYYtZskvmytySBQq/VX3V5TGUVUrnsmAq7kVJ02DXCenozTZ3GQ6+ozQ0HUCB7U1gtTKc88FuYou9iyQK3hjJCrY05f7eYfA0+IBmwoK9kEVT5gNgXnoSVhwJnc/AJwuTXnvpRNaK2JMq9lPV0KtHl9llvQqlC9zHNhUpTHtOBpSCJsUONMXHuJz1K8Ih5B2KMERlCdqfXWa3eZit79kzf3Hpy3yGEaFQzROO32jQ5aIUcij8aWaCxHYeCGqt+WTaHU2fA+fUD+bdSdYJ/JpJMaKZ1aa4B0QhEMeK7Zm2sQ/imX7XNpYOwNUau0U0HpgpJwbkAqjV3+pIEY5wKZaIfeFHpVIMWF73G/+ZcegRZIBnL6nJvRijBg006Z1k6RU/zqez5dyGuiOuQ/37N4rg7XPEHS7YKbGIRG8/bGy3/hc3e//iP33350e67V77QFXRxNRr052V8rtvI+Gg0J74Vr5RU88KWx6EgX0kbplRfpJCJR/Xp1Hq06h+ser0SHSwEi18MJ8tfPYJQ/mYIy4FqrXWfj4zsiBs40WqQA8iQxpDtYkhHOI2S+GBvNJt8GqS6boWHOrZM7nxh8+Xe/vrdrY7E52n+Rh4BCD/I676neGueE11Yvz2cBXT4LrhZiVy6Da7nNleqeclIeyJVye9VOOa+eef4zY7u8szoJ4Ss3hxkEWm5wzUVjL126fP7q2csLF3/dinxXEAQzVyscCAZn55yAtVw1K0EFsmA3e1aEZRFpXEELAh0az0TReYouJhrR1lN0dDhbHx2zafXrzcW5rnLN9xaXX1lduzm/cK03GDQb84J8UpI2fA4g2SGRP+KzhpCcDKZ0CA3Izm/FwxsfIcMw/2xM54FJeliO507z/QoIVycuUhwxtHJilHnWG4wUwUKPMLXqXyydZc/UMKq4hP3Kx2nchYEDb/lXDVMrpaW8tLJGYMBvvtm688JNVR+QBlkZeCqJXbSRiCTb1ka+qfUwjwzxkiNqnBoaJqqS77ECgRmZiwbpfn+JCY25PiyEXtgMvghczsCHcjjFl5D9SODoIgGGCS7qRuhtOXRak9isb4SYqv1sAWj4F0fMzm5Qu1rsFSP/4WgoXdXbw8OO1WE8tMWgVBUaOvbC2iJAmc7Kn23WvT28ejLOdgidDqGYb1UP9Y2eiiWFgJpcda+xSliiI8weIXlxYLTFmqoQAuCxCRB+fB4JeEGHTAbI0Zt1s14/0LLOx5xpMnHYYiHg0GB/Hu6OD86HHVSo0zka22runHYMGOpfAo/ZkfqDwshUXt3YsJUirt9p9MGWVfywe6hcF2+w6DkNu9/k6Lm8MlYDtbxhR/FETJnTOQDtpiaAS5EY2EPyIKB4Sj5FREaZpBYXa+/scXe9V+s3Z/aGHerO9GB8PDs/u1LbuLb+s5/rDtp//OXff/fh4wEgNpr7lGxBcEfjg52Ys+zZsHh9prao+Nbc2XRfCTNVyYMHSQXEEDo04N5g48T2YHKXWaFrEwES/GuWYCHzbKkHpL5IRfBnZnyulqqE+CCEJYDMO6lskBVvdbGQqbJ+g2TWBtJXkD2U4GLtOKHoMhdd4FVa81wisHJLYS5WROgKPPSZ9ZZG8kSwHYNlc5Lzez6cnjxU3Y0X1k5m5tbdkJ2xX7g5fNm4dXPQHDxo33/wfRtHTua7czu7p/1O73D/FErNNbEZNhgw14HY2CthLvw9Jum8TPgayzOunnpXSulJD1OcvSWZXjwbo21XCC67kko5yqGe48HRMMJXPZshlzMfvFgZC5LjcGYYmnCj28o6zXW/G2Nw0q2eBQNUlJnBU8Di029++lMP7ZcJyh80JwD0uoDSEXMxWcQi7ZwMZfMTxRZe+pHP/PhP/dQ/+M5/edboHUxHSEVXZgH5mI5+EoedlVIYa+agtFcmt+pImZc0XE1QAJc3OiIilHM/6TuaDz4Ryy5GUfpzcY6KVMyxAkdA42nwF0cjUoPtisTD7S5MhaR4784rKyu3m+0FBYXFR7KioiWhkm0TKTle5EbtN3/7X+ycbXNWTE8Oqmy09Kb0SssVOMhKOgHbrz795NwYTUx1XnAwvaxYddXpjLZq7urkQ19zvTST62VQrpRpuHg2GJCN7MIVYDSRy5UQ5wu8yW2OCpjlREMXLy3vshBYtXJFT8GWfagwpUxRAigshnJ7WgV7tqxzBNdSY4jOLQh/bpE7goZnvEYoiVUlyaTdidZsn8+sNBrX+vN31zZuzy9d7/Q3Wp0bne6yinejkQSLOb4d6y3cN5SVpGhlql6BPONEZhsRFlQcy6fKKallhR0i4mIjUz0+kdSsxdWMlyFdfVyO9wNw1mejCFAjIgdcboM4eQrliIklpIHd95L7os+CVwQhh7gGSoVgRT1NC3Ggh/Vm4AG71RYOOjOj1oSNcm/c3qDlYIRhiwytiZwM22DG5DHFZZB4rIvfUTc86DCbVGHMxjhhI4aRDkcjzCyTDmh71VdXqgN/8iBvtzvd76kTQb8BGgUnZptMzeVxBSuQdO6Rq08nDkAOVjBFsHRhxfyiEhuKgRqolPKgBdISvLSgRgzOJG7ObHPR70gnqAnxRtx6PcrGZJ/V93jS68TzmlFPDm3/R5XDMRWtT7VM8I79PwOcn+/LJAYNMWuzk7O03FMw5CBsNrvGHYnXcSfQBUmayu3GwCuv2E9ihI1dWDWzJwnBFAaYZQUYem4QrW08xBKxY5Y5BbqZKKrp+GhhRcXQHmLpvTQq7Gp5bdW40T6ua4TnYLgjsJwXVft3rwnS73Xb91N9M4FcxMsZPmHGmPg3bCMLTGJkmYpDF84nZ0f1+RZqMjs8ZfZWY2B/e388e941Nasb/Z94/bXW+bd+/+tCf+eOzu3OxkVpBbEd7u4KahAgOrfSatMchdfYB4DBQC7qCcewvCHjlyzTXG7wZhbLboIOz4WsjxUMgWsSinA4C4TtN7Ht0nyrMiahRCAJj8IsDRD2grAjzKPQpUKtIozlB/dHWPLXrGexuB7nezmy7uG/JrD5shKzjMrFPFDwJBTb2okKHdZlwSX9nfFg5oCcVT8fz56SgehzyqIlOS0yI3ZBrJHbfev6Kz0Fy+7f//7Dg+0RT/fxhFw4Wl0b2OFkMn7GpYskuF2P0ANGEcF2vjKdQVsgQNxnbLnT6J825kLaSbhz3RiY1OOes2tZR6IR/ziUD/c1c2UUGWzFiXNSresi8AVAdPSMIzhWiMcFKAKqhK4UULmFkG3ksfOn6lbOHdUDBcDVY1efgZoWPnBkUqzT8G2t59zPHnauavT5cJL4c9/nFz//xZ/44y9/Zfdb7wxaDWXOCd7jsX0uj4hvnSYTQTVh6HVcfFcqdZnevPFDJ75iBt5S9ebijOauRHYkLN0ovShc3dPRdwP0DMCjVT9doS1FJ8NfLbepuHchDX11PV9+9XNLq3frcwvWr2kzQSRe9C0k1g5jg9Y3v3r/t37vy6VfF30I0vqXOy4OJxUDdoc+fejThTQHekHgAj39NoeXj3/gLxrn+4egUN1RjeliSJmPXIZlsVBEC8y7NZorFczcU5rKjaXzmr14vGrx8jPzWm0KZC3iKGn34gXpT84zgLzRt1xIAEAZlxeXlJTYRsOHxIaIWOanyna8NbWaOws8WvXmp5Tqm1+8vby21hNhymNy0lVG+PBINKVXI/CIhgUiUIrNAiuf73dQEfId42cKLUn7EF40RUBtfYTvM8OGJqS7QqyKlnwFtMthvf/36qerE6NAV6o5BLeCMBAHUhYeix/E5kzfjTe1/KNqY2SkBIiU6KuMm26TxLX8Y5HPiWnOMvEetAk1PmJVxuTEPPfnWDBnJMLKuRCanOAdiULs52GZgizyOiDGP/Tbs1hvWslkRFzIdX+w4cuWLXxcPPdkweeV6XQxKbPzSLQFKO6fCBJh2wUJL9EgbZbz6nVeUprJJawYxeJVNRjWY5Zk+jVRQCOlDyW8mjzC5U2FDQTdGOcPzurtiiew6+Jt1Hyv4Np0SPYFgYJAkW/sOhspB8okVKBMYtZFkJTr+pjrXEw1anqu6tYsH3m3K0q2sF6m9GSDpFEqNVgPFsOqgU6/8VH8CcPWh+7gfDh+piQQxRrQxAyCL9359p2bIq5NESuBZo13OlSP63B1UTWhHN1OH40UhiWCzLuMhR6vUAfwy4KwY7As7k53bv3m9aXVJcVGvvPt725v7VGvW632wYzAEdiEm8CFzF0VTcAs3VlabK8MVBpmaav1Ectu55hQsnck03E06vevL//SF3/y9o3f/Ef//PH37/canaPhiJgWzLS508GzmafMpzN2cWovbMyKuOIExD5qIp9xAVuJTOYGVijv6aA222OLPp/p1mo7Z7NGOkliUjbTVILmKLKhgnERJXFdAMczcV/rKZRBf7VWMEnvwSZHUZQjkjsvF0O4THh+y63mL2chDhrUorXhh1wLrbCY81T5KVN88Zt+uI1JCT4LFuJmTg9FkJ3XcF/5VppRr312/9nm/PJaTebf1m5jsX/j535yfuWN3/213zWbW5tqsstQV1VUrpraK7NDkVAV1y9bg9Ow0h/Y1GpPzrhpLUDbWS43uqt2LrIB2TmxCa1ksVdL0teUOhHHJwbE+6sRZxBlKKGs1Qk4kVeDtl6WUQeQAV6szTkQJWAtyzk3haYGSIbMZ+sRs4qGgpGT549y/epiBdbL3yOLBGLeUnUjLfouHAz3Ue0W6T9XXrc+d/szn/+Vf+dv/eff+b92my2R+OP9g9bZmVps9p5h7iJ+Rv4M6y20JF0o3UC5ynRq1SXzGkWsnARLykuriz6NKUwmYSXV7d5tNGHtWcSZYg9XDfhkmDiTEkqiF0hKRtXNWr1/7eard1789MaNV3BfAqaucq+ozyeWRfbH2J7NA+mox3/n7/1Xb+y/2Z3t7REz1aUvZvOKdnndBSwKA/a1AuiHPo02V9KldM25z/cPo6m+VCfVUJ+/4jxD9rJLKFydVLeFlxdC7Gv5KdqExVjeWN2Sz6r96vODfcg16AZR0kIBoVflaoR591qd1Rotml4uhIyiDYzQSVSgsjIfcEeZ1qOj9ul5vza7Mte+2V+4t7p+d/36zd48I+ziTMMuACScWRG2J8eIHx03cqr9V2unKu2U0uVRhPXfHoJYr5gYB8qODOlfuJ3N5yJow+JAtQJFIFsoSMbp/BKkH3tebq7gXglTxqIPCUcLGyOAiQbCAkvdq2S9SDClQYQ4BXUTF5EaQAkCTekkD4T7Voe/8WmWgDgKvM5vbm4uLvUWV+bdgMTToY5kk1weLtKMEWbECAMrzDJdq1rDhl20VlwpQR8ZHdWqQbmp5itoTc/SWZZRJvwsA+3N0pAh8uRI+TZPUAazhnHhAh+NG4pP3ErfWdfLwHODi87tWKBRnmDOXlZZDNi4/WiHIm8JsEIB0kuTYbDR7KdiZ86SdO0yKUoNCXZO/Lg7oL8Baey6h2ONU4W5oxJ/6gGAy1xW9CA1LPn1vRHuaVX3tEMAc1JJAKDB1CyjiNDIHrixscYOLBsYSrhf592f/shcU9hpOLavAEe1Zx2YtDdyrhNM5Egtr64kIZvTt91eWl5mZjFMrJ1LABhHB3vUbnspkvn0kMlBXVGRdsOdA3tRTA73FLLOtsSCGyAGXxfHNmAQGQLMzA1MByWAN8hFmybNY8BzQuE7x6NaJxY1djjVZM4bbdDsM8l/4ubPd/7KH/2zP/z+l/9wxnQKLoNFXN/2YtzZ5G9p7+698Jq6M+Q1K5bkShpV14lz+HyyvZ2o5FklHjvGz71+MvO4NrN1NrvD/pwYpPORNmE7uRNiwB5omiPUuDiSdCJfy/rPp/MQ/Wqey2QX7HG7B4IjueZfGshgc+7TGiqEKq1D1nAgjfksKH8Bl1wBG3eiuyGnoMbmHNGWZGxnRq7w9MYCEobtUjA1VUQ6c/0765/4wme++o++/v0fjOnGjeYeAPMOiE4QjiAamWuz1G9NiRC4ij+ezNUaPfU++q3uWrO33GivzDYXzjhxUPyGTUKJV50ERbPrhRDKqw5TLUc1uOozF7LQKxqe0RbpIR1NX43i6rDWLr6CAaISIFEbJNoYichQ9KMw+Ct2e/VkTsgs7n++vVwNxL2rvD/wzMykhkKyg1NktHE8tJ/x6dzy0ud+4ee/9s9++43f/9pbD+7fnF9WlG64tc9LstoZqC2YFXvJgzOgMlLTlHeUo7wnZ3lLuVxdMU9OqvMKbdJJd8TtHeJpFaQxYCmkkZcp0f1KTaonDONmTYddjjzaWV1/4eXXP/fq6188Pe+dnksOpEsQ+jlzyPww/mx+dfm0V/uTh2/9+pf/mVrE3WaXMzuKj6czPaWjef3FUUIBq28X16oJhPBlflwsWGkEznwGO587jOqHr1S/VwN2fnXP1Ul10cBjGcobLqhYsCScMyj/oWZd+cgDdCBdsSpoDQTzfLoa5KKLlrGWSUL9TUQiZlIzEvUXTiI4glhe657PLra6K3Otm935O3KK5pdvLixds2M1qnwwYisUvUNj1S6dRCYDPBRCy86PahUj34k9h5Q2IIfv7qt/jgmzoIZM08mi94gpFesYDIoMWtgSdgiSQcT09hJKV0BzpTqurlQnPjGtwNTP0BENjTs5aig1ErtVCPj4bELpsfun+tFcVcXljYh7DUQguiMZSqllexaEFozCnjMFUQs1jGfY8IeLihJsN1x+PkCWTYuqYsPoEW6ZgRTChImG9GCl+pVFTp4OCZMyi+VgSH5SChl7cBGykHzJoOWp2ShoalUk+7dqsDTKLJbfOTDneL7QXE9hsI4smUiuwDXDsh4YZH61W/oSiBCneIxiQeKqHg9H7P/YmDszISEBxhl+Ux0FDg0rRJd5DcaHw5lzwSyUTubFgFS7tBSqJG9uImyidSFKIMZ1baBAkthJkFTDlrJND0agIxFnOUbcAjFvZ9BmeQ4rLWJQt2tvKYAHIRW1bGpvj51cMQUgJ2H39GBovMIMOI5l2Freo/H4xu0bsHdyPFpYXpDBpdlOv6345PHeDmlJT1xxGNp4aNtghaOm2aWYiUgWzOlkd7i9P9r/7ne/s7i0DHrIKRjrEsd0kcOQokQmoa+hj+iuE5vZRSduyC8mYhwcDgc29Z07PmscDoe7i4N19oSnD3fWl16uvfbi504+N9nfqz/cnO7Ypm0ipo5JZDoZ0clVnNx76/u9levN5euz3QXzhQFZBzO17unBEaon0CYhvvNzM+zVNtFTNGz2sD6nwjNcm8piiXVFcBHrA4w3RngA/SKaZUaMOSTUXzgCX4KGfi2fheKWs6w4U1/wJ1QLFgBX6Bw8Ck7lCXPgb1rXGsyjqrmnsPQ0WY6CgKEuESBzh5VOzha9XGmW3NhzncEifJD3U19ZY5c43dtUufrmj//Iu3/89J3vP/Smdx+fb+3v3ru7dCx1/pRvN5IxPaqLxKD4ZDtVxSW6rK0KPbc7V7OzPDu3yFTA0SsayBtgS3GpUU9RMmQshUYTihzaXR0Voucr4QoyXyxaMCkcNPhsepHN6gCewCjcqJA7/M5SSrCYYCt0Q7wCCF02Xv31tfpXvev5HzWN5BSCHNumnwKt6g6zxD5F3M0qIUzHzi2JvP9X/ubf+Ienp9noY655LOPbJmnT49WFJTEUmXu9i6QEfiY8E1q9tWKxFz0gSJWZ8qPrzx9GWSa7mmrNpRlUlPyUzhU4GI4T0gaDH9yKs1Ky6Jky8t2lpdt3X/rs9dufbPZWU6pHQDl6mvsZGEnGjNC1+RuDB/tb/+L3fvPp2bNurT+aDDGFtOEF3hYsy7uu+hXE/nMdl0P+Ux7KUJ87fP3hB5GYiJNRpEIxC8KHuuW5fC3YcAHh59p67tR8KihlQRoTYuGXTE0O2EZY87uv+dCWyXBH4uxFKcTXdBpd9kxYVk0xiB+9fe9Gv/+STUMWV5YFm1LImCJ393oegTWEG9FW2VIcYVbT8dTumWir+jRWF4/oxB6xkwS4j9X4xWT8o1wF4Ng9d9yZzeACE/wyalsisCKzm+mYYv70I8/mKGouPhdLi+cLABNkhR9gZJSWqL/5xIblOiViAJsNzcrN4YD6AEGZzY1LzFlpNI2bAd9n6FJl14Fr3a6ai2GfooGU1aOUOqdyzSl5FE4fTZ+ckcEKbyUDFhO096RzjpkamcMN7qwm0uJ31UXMNfmdbfUJXCPvEx0yLYB6mjSqTH2IMy2CMwyG4/jVQdpERdJTWbmJXrZ6HRS9WDrMFKe7GiYdYZzR4L0dhPM6pCqLqvB1TCc6UsReh6qa49FBZgpFmBzRXm3AENXMzrejYQSp6RGXKR+kVCLpRsSpSBzuL7oyzPVepAAjAwHm6oResT1Mjwpn7XR7PUCj6OgngcajHul0WwdDHluK9Uh1ekqqyh6up9REfSS7f67p+lyn2z852ScLpJJDffbajevCAp88e4Qg6/wIc+Yxzhye49kuHolRN+1xRJ2X2ikajujT3ukI08xCm2u89/DB+HCSGDUpS/ODyUhFJ/X5544TjRvECuQBI6Qny+ZwyKLertlIZ+np5uYjAXbKXPAcClE7njytY/GtpfPp5szWaX1j4af+/V/53t//9a0fnO9tptiqe4l/4q96p9Onb37n+vSk2emTGDDgM9tQh8pTBPvHZMIjGweYyH6t3mWOYnedzuzVWmRWMgLXeWhjIWK2OcpCL3TYp+kNC3EUHlIwB0UpiJ45ytyUezKUkNmCTqELsdyYZDeUD3eVu4t+B3cz9tCT/Ap9Y+wot3vAX98LxvghcM3q0Qn/fCn31hqKtgBwXYRVhGPmCwIsp+rsq5//9Hf++GF2CRzV3n1Um5zuPN3jDh/zjADswvLJ6uxJZ74325/tDhZ6K0v1heVpwtH7Z3M9TJf9mbsxludYimyiFpW0JPRFtLQu4B4unJ6/f2Rk0Q70L2vcr/4vUMvYAK26NX+ycHMlTCzoUCgMbuS3/ASIamDmpdXjTi4erpr44U9PlXnK4xUoq3vkKCTKI+AidnQB6Vgg5NHxtR/91L8x8zc63e4f/NPfOHi6dW9teeGssbW5242wENLggwAMs0oXU2Ms8L880snL84/8m37HRQIVMrrLiXO1TLffksRlvCaWT10CnHg3pv4OBW39+us3br3e6qzsD5HZ7KaZtFWQRUX4sJJXLQ/77J3H7/zjf/5rR7Xjdqv/9GirZSNttI0ptHivdbbqYdXrMGAQT1+rz5yV4+rr5fzkqlsvrzsJSMvElAc++gOVv7invNCbShsWJhsObSSKUW4o3MTKMkn+u7jN39JfLwrBzJFW0DKfniY/celmdeS7/z0ZtpKowrjlJHMS6wu6QCZJ7PY1OTmWmQ+J2idn8zMz13sLL6+s3Rgs/Pirn+jRVPigWBZGYxAj1tBtyptALms3OzYIv2Fes/ub+NrxnlghLj45lkx8bKsm3y2Wd/y8+lDgk7FEYyuBFgVc+p2+lrFUMPS1OipoOK9Ogk8fhLOvBhSeFWMqFAorVKwqZ4lcUteIt+IwBqwmGbquPG5Qkisx9+ENqG291+kdsKkaUBMCoV3xrVHHaLoIMB6Me925wwDflhtjg/j168v8jfh6sXN2tSNKyz20fCUGq0nRj/QN8BN6xUB6gnO7PwiKmyUSG8bPstJiiZRsHc5e8RVXnFN42VZ6MXQThRWkIP1P1TNX5VitA8pcqVIiENQ54cZtbjYsCYIYCWUXoPI6FeBqJ6roH3LXFDO1fF8s384BywvLx2cHeco6sBUQMzW/abd9sLuluoigGLsjiKKiyO7v7nEcyGIiObU4D2aotuJdTyEP/dpIFAxlxVVMyls0jpkxlKhk6VyOMYwABAM3xiKWzC4sZVtD4cYkBVfctsxOdXr6+OlDsMKhDY9x2EAkPGzvqeq/Q+W9fv0GtZwFwoxEzjo/2x8e3Hv59t54h+LNW/xk88nyxgqfbKdzY/LW216xsLb+J9/8FsnIVpLg5vCWg9FQcPW1a9ceP3yKl48OJ802ZzUvbAwag4YCzbQLWcv2U0I+sjlGIdKKfguoO7aj+rMHO6ufma/dvLf75Ilc7Wtri9Onk0UMYJQdus6PhEUftjpHtNlaa/nVf+vnv/NPfuvx7+/yPItXi1v0aDozOVSkaf/RfdPUv3W3fk2xuCyt7FzF+sCOymnGvX02nZ3Mn3evd1vTLcMU5dcqWzWcHCS5lmgXFpMVUdZ3FpAVEN0pPvtC4rMicmNFDtxJnIB3yIuP2ICIag52FxaUrLJyNaSjeoJ316p3AwYbPIlxwK/EJkFOXoQM0G815HfXfUXLFQ2jKkbMYx3ym66w3xD0CE4SYBmNZo4n1uSodzq78trdhdsLf/A7e0qbqOT99qPau08OlC3U4ERw+7g2tzY7v7gst8gmEK2VZZtj2GuqNsvs3LaVgnK3YpKUdObXiWoABuVfRlwE7Ei5hRSWIeWjojCkm9xSTEmxblweEbdou7GKvU86PXIM5z1rrcIMPwFJ2J+wj04hiT5AMTbnAAgYCzQDIgq0cyaBQp5KnnkFW7JTYkTAVJ91322xaMQshyKCYCGzx8P1l2//u//Rf/jo4bvf+I3fvtVdf/Rwa4CqFqMXCVv8BomeZwqR6Ek6OBiZlKJHZBa9P2yJkBbXWBAh55EB/Gr2bdd4MD9Y8OX4ULxbJtbiQtapCSkoDAaS5c6bchU9quTc4eGsbRyv37h79+7r12+91JvfOJtpHxNOLc65xsH4QEgjYWLvYKfV6LbnB2/e/95/8l/+Z4/Hj3qz3d2jof6AL1vaJbwDqgtUi+pcZYxd/fhnOClk9II9/Bluzy1A8MN3Fojkp+d/zcWA0d+LI48/9903P3ik+syzVoDJ5FVzEnuscvVmJ+KGOYsbzVYqjJx2fVcWzv7n0/H1XvfO2satxaXbg8U7g8Xbg/lVsup4zFYo/R3zhtfQKIBTmU/YHPMDA2VcZcwmSiHT/JBrJpHsWWvJWmqhDJEliKC0OdiJ61VjCZ5lUND8Eg5XJ1nZZcA/DJ+rK25+7v7SDgzWGsT1Ye1EQbTNkfUbgdjr2BTHwq7l35yK7ZkVu8upqVOtDvrP2mcftYNuv6uWHVMhcys8gIJzAaMmM3hMF4vt9JudGEsb1E0sMDmOkW682xMRg2iU2LZVDSxl8InDV+bIT3iki5qK1lkEJl8BzzcrNBpimUd3UtwhFWEBJBGCkEdLPlk+Wa5elEVb+HcFh3xm4MWYYPX7BsFLDXcmCuHEVjfeY/lJTpadywIea7K3kuSLFR20EuOlVImgXiI99VuJqWb28RXQbVqhYdFNw0e9J1f0JnYEGrLo+HS4Gh2w6TCxgKFa+0atq7EsC08e20o98cQxhojCOJ6APlZtKwJJ1MbF0uDBoFkaVIjjcHdf1dIU7ubWHY5NmTiyWIjRdzyVnmoXQrst9U6mm1vPRofj9eZ6TUk/UU5q/Kv2rHz3YP7ZRK3uYX9+gGzqDBEEUqyur9+8fevRw01tjYaHke7NpUblUGG8MlljzKA7JynbQtJheY9Zi4cnT955tPre09p8b/XW3Z39zWcHI7BKrqgbbTFPrtM/Ykd8Lke1hZuv/8UfxVK//Zt/+Ghr+4X+Srtx/nTv0Vpvw3bSh6PtmZ1mtyfOFVQ6icI+spEg8TAL6EzqtUNw9tmkV3sx1TiIjUlZ0BGqJJkYA4DwJgMGulhofWE4QjByufzDlnJiKRRU9VktMw+yVUCfiPshymXtRbQOG3aPRmFX1pbRhV2niVADjRVm4560m6/l15zAHpCS5V8XclWKdxydCYRNRlDjRC3sPErsVTCuruJGt3F671Mvf+Vbf9Tq1fa3KQm1Pn46o9z6yeJqbeVWZ+32C4ONZdsGKUU2xTVbvdkEprVFFrIDs1YE/QWwUHxjzQxzISmUHlpRll56l0OHywGy1Uk+K3NBGUG5yJKSZ6vFiw/mtLDpxM+HCJbmC/+I8SAeCkvy0mhXAOPZvDacH3+0RsBWOx6M6FJIXHnVxZTpdYG92QzIfbjZMkYCSN4oUZ13T3bVr/ytv77/7Ol3v/3G62s3j3Yn0S4hNOqEBDk/ya7oe2hUBKL04Oowbj3Nhcxr/pZ+JCjaZUoYGRPNk5ouYM7CYyRkYZCmCB2Y9AW4JYs7rfLAL9x56eXVay/cvfvS0up1Pv7JFP1EBukQNlDsWKjH06GCccsrcsPqT/affP3Nbz969nhYG82qWmqDVZ2gAMoUhlelU1U/M+bAAQN+vu9Xg0jv0/UfPkL8y4w+/+m2wP3y+OHz6kqmqpqbyzsrdl59q/DEPQBRHRfXdSWB4ZfIX71HOxAP7ScbGUpqB4cSuxWeEkZpJbOsOhIY1A84O11o1Of782vd1hfv3LrZ6YgEXe/YLaremZ5QhW1EiS2QwCIPBZGgQuw6FpckAzOeRC9lXKRRKrdP8JfkGu6SmQVXTNoaIqsH3IRu/XkfxS9OIyeWi8aSrl+AiyB1CYvy9/L6xcWrr05y5Dm9DCcoEDcbbOXhwckjEfRDH+cl5PJriOtuiZXMBqwlnCnyPMyypGOEb+wd7OEqnMkS3MnseA7lRvqSSFxcQQ5S0hTZnlvLOO/4aB/RtMawHwwF3J1gTuYT/l5xRzfoN8pqXrAi56ARBhR9l0U/h+Ccaj3gdryDbnCzQVlE+RPk0kBCl47mio4tqteA+bH1Hn3zj+Zksnlk3R4GHYodU+qRnBc1ems2s1lZW1totHbtM3rC5NlhmhePpTlojxuSc3HkHB5P8ySJLGtSFdyhsZt5+cIsTKcUq7zOGC1V4gEyUR6PWJCthwbzCaIZUjOFGZNIQEXgFKicqmurdsTMXCsZRBi7RubnFyO+nJ3Y5XA42uf3ReYcGUPZdQPkRxOpoaSZugZBgwihKX5RicmLy4t0bOCiSe8e7KGFvCVZxaenK6urm++8Nz4aSzR68vgh4cmQ+4tLgSf4zCi/tXH9xvj+/SfNRm84Ij3usFmLCRPMRcNptfuHamyUoBTtA6xe6hizaXt2dvu9B8PvvdX/8c8uvfT60Q/OhlsP9T+VI0+OpCfVpoczqTl8dD6Z2PC31ezUbi6//LOfX5rvv/07f7z19hMK8NLSBncE/nS8Pdk+Hs7PnKy1zluymGaFqLAzt89O0G6TYgU366d+3+C8ZtyipsaEZff5hqmhBB8qhJDIrJBWhIALzghDETzrcC2Xk1AUuxfIouxZNQFzWWnhriEYOVwIrclHRLpy7jI6kl+jOOW5XLdKyw1XJDRcw5EXINVkTrW1SSDSnEVEZ8MjRm4sVGEYKWs8JalpXTXcbVx/5dbR7B/Jeq51Zu2GK2hNvGZzvrZ+t3/ntRc37ty2VThaw/16Yse0uhQjQc4ALdgqDKrwOcM07EIPwud02NILg8waKqMti+9i1P4UcJUehwcXUOSpi+t4Gww0Vv22HA3W1zSUtpE5S5PoA0SgWg2DtpjFXshYTgi34O63vL8gTwXvcv/FfRVL0ZmAx0q2ktITbK9CUm7v6OLDs+F8p/36z3z+5+7/0j/Y2d4cHyx2WjvDsaRQTlfUVW0bdEjf7B2OEpUZ8p4cpiUzc0VV4465OFBJhNraHpVNv8SbInAJ+0fpmffrTTu+Tia65Xq7P7/E4jxYvL527RPdheuLi8tqnmQNs7WIW2+1Rocj7+8MmPtsaYzwJ7f+/qN3fv+rv/do+1FS24tvOH0JkCBk1bN0RpeA3j9HAjfKeZm/ahYvPoNcF/P6/vXy0OVHsDOgzyxWx+Uv7/913Zfqs7q5+vqhp3zVmHG4O7SwNOvToYH89lwj5XL1E+oTVpTpjP6aUASSJ3pvuxnpLKwJ5EsZGWtLi7du3Li7tPSJlcDS5s/d5G2pACDviCoZ7htPB2jx9RKGwksJMFk2qT9xOOXinbFpj4iSSKE8Az5StiPBz0omeTa5KTM0C72/Gu+Hul2Akf6WI4O6Oj7+Eb/kcKdP0dtZyeXUN1cwBmzYWol4cDwenR6K1e7Md1YWeioWscJzA8tC2d3ZGu5vU90jmRqtEGOSpEi/0NtQXahaWhNaEk+tWhw0YIU4BGFNzw5tLiHygw5NUyY+utPbcTSBSRUzC2MoixY/xuqosGYNkw6rFmxVmDcra1zV4ZulASfpfazTJj4ZTcWWQDFrnrHsZo+gw9PJ1eznQTox3hJ5KJBzRb+TNXWU1UGcmOwPKaXd3ryirN3+fKsnkrltayDCMq3UC2KciokwWKuF8FXxvLRBHXad+t3Cg0/QRHMPNbxd92wYEYNHtmSo4sUYSsJuJf52O3SUusJhxeYsrIz3CAp4sOjxZaCCqK0m1LasFVCd291TQ4znt2dE7qw0ZoFaDvNLrU5pq0N+sXExiCI752Ky9ke7h0djohL/mQQi5aezVy4Drw2b33kP310ZLHghaYMngIVc2RQQSmL28cnqysatW/c2n+4qRsIGLxx7zpZFdjRI8VJZa4Qc+VXhWgYMMvKSVJRqwavh3qN3fvDKrWu1V1/aWL39aKw6lQitwxRVFn4L8NT2E9uQU2jPySHEg8Ha7ZW//BMrSyt/+A/+2c7bj5miucXhVTaeHG4fPcKzZtcpdPPrrakISCiduuleC8Qzp+3Z43moqXS6iMlTmw0nVnJntraPbhe+g1Gnp9G3KnoLE4IOhWIFjX2NjgipQ1FwjYrSR8gLX3G71Z0/+cWhYfdVjWXhR1ki6RHE/JDveUqbbqn4V7me1xXaihK4QO4CQh7bmDSTGjS1J4cAMpNp0ZCxRIXIDO6s3qjdea3z3g8Os/uU3SePjjvt2id/5M6tF1dVtW30m8lkJ5oJMq8zEmBsMJOmBEGDnthw7OilI3odXpMJ05XITEUyyKWgX7qeTyO8GlRIehgq2ROilpkud+e2QCEHnDCe8lRWia/W78VXYA+HL0svt1SHNl3jCCsLu0hFuQCAAVooYvkpC8N33XcdzKqXajBTAMeZQrhT6/wNNQztp3/5Zwl5f+f//h8LPbMoscyTIzaakxbOR8MSRohWs2ZEYyrdLW9PDwyy0KJ0IPiQt+fVlCWFNbJDhCQ/hbXbGiSFz5z1T+wQoWbOHEG3v7C8du3G7Wu3bi6t3JqZXT05VzGexnPE/M2pTyuDp3Uh5+fHvV63tr7EnL13/wdPnz5+8ODBm++8OTmdyBpOObRIj94NfqIIA0C90BW4dfVpdoNq5bcPfRrSh67kLlMcsF0e5cF8eIlrV1+vTnTT+RV43FLhiFbyQ6CTe92Qrl2ge5nNXL6AHQh+ZOOZxiC4PU7muBNx3DmB4dPJnGCZ6fHNhUXbAt5cWlzvz68OBkvzgyXy+dGkbY90lkl8UxxbSgUxvqpZOA7GSiJNeKZOF61H6/5SX8bcXce4L+0xy4leZxjBuuBiJO1MdNZ0sDjLOyPO8iywzucFcNLlMtaCl1nNHzgqXKnGe3XupDrPGywPDReBgwUcjcR9cRGohrnwRSNZTERNuywt9FqLvZu3NuwLirvgvk8eP3j03v2HD+7bZU+gB26Ey/BocnjQqsh9hDmkPfmjNlY/O5N+M79DC17GenEIrlfcVPi04Bq/WsNWZdkwLbbociXjxX3djBqBVHWdr7g693mS/YyT0BCaCxIRIfJUhMuojp4yDSgYDwsfKuCXKSjtG5qbTYibmY9olmkQ/bd1QGpdRRXG32X3bG7vzAfMs71u3217ticYbqdXhOCQEsu2Sq6ggrQ9zlnutijryuQqmmE/8xBkIoY+Fiknn4acfoIUagjssr2km+HDFHIMjHGdWJAxeNKQDEPKK1JiU+cjaUIyGVjgMfgGIxsL78rSojcWa//B7v7u1rOdvf3xaEyk4fISWnjG2mwnK+VB9A3QiC8QNaWrxDzzBtKQJofihKfHR831Fd7w0WNqCrN57A1gubu713z0jGdy3xbUZ41r6zdf/+RnHjz8F0cqITNttHs66o3xsDMfJEocakF+ViAaf5w5aMhkvHVyOHh6/+3G13ovtLszK0sbK7fG+4+Pm90msy8R96Rur8WEg3mmKk0i6FnJ6rn12ms3vtD4pa//+u9/4/e+cm1xqW+bXaUTmUwOR7v3321OTgfLo9byHetWGUX80vKKSccUHXeo5an+0bZAO3xyNpiO7WP2xK4MPICWHZJggrPerCwwr0B+uZ5cDmtE/rKac194luVZqE5Zp4CUxenpNGBdmjOokbVcfvG14CY7WLm1YnDlmSz3srBx3dOTekPEGUNZGCRPJdJMKpqp91qTOVJHGE2qTJoumptN3EXdzfzFX/zZ//f/859MpXnPjpZW65/70U/cfXl1ca3dW2qcNsR4oo4AIuuM2YluHXdvtvTQP6ppITN8prppdCE54aXpd0nYdVJIdPhruG/g46g+wyMDKt8KDy6/ZOTlHj8UmT7jvgx49FMEGOJmWrsSRDT+EUceTOsapKlnBbmtiHSu+GZ6CifPSSiWseBSgT72oRvqxNcmnfnW4f5wMjq4d+OFn/hLP/3VP/qjb//OH13vrJwmuzANEIaJ0roM6AgN07Uv1TC9NDNZRphPpxcQCjPzPvUOa+0W+j21gbvtQKYpUXM85tjuLCyurW/cXF2/vrS2PphfbPbEmbP8mwiUgVyd6HRrXpIpRef2vTtCUhN6wNn06PGjh8/2dvafPlHEfVv6mD7EDFl87YUz6klBzYDhAnfKKZGjWPEil+R4/jM3XDxWPVw+yzgvgF9GWEZbbq2+Or2Y8nLRx9X1iwuFm1YXA7YLzKhuLOAKclRvz0mZtsu2PviXGtKz9aUoY8kOiHJNEkljsdFYmOt98tZrtxcW762uXuv3F+bsX5ptLBS0U70toQvGYkHzkZ2dScpTdFatiQhq4WJYktyMhPiitmaYlUCKNe3KkZXsJouR8ub2/BeJoiWU6XJ4RlS6naVxcZI7yzIGuUpcS5lLTxSnRoUxBVDlYhqqTqrH3z+PcS0I63es103pRlhfiB8OFkmcxWvO9E93D3ZPpwcnC+2lXvPm+satl269Nn3t0f23v/qVP/zWt771+PFjuhfPHYFanKxoVJKRIVsQhimwB4t0A4Wsd6i+UUdU7cm5YCYKuN0vImEUriB81Xm+urk6gM0JR6Z2MGydj3yQAsg5cMuAsSxDiyidz3ByZKH6pG0ia8X7i2p5SybCVqFGGRNDpPfQEFuk4oR1OwAm/davpGBPCY+amcUXj45PdyxWmRxVVJiX+pW0QhoFtmyBiMOwAHSbR/y+QdpzVbKOs1P5zOGEPZmQoe8EWfgaBb062MfBX1NWPtFla3ZXvR7pxlQTVFbXIwHBIdYXQXHSE5i9xdckpGUODHt2k1roK/isNY7qAmcp6EJeJjgxP6x/nAKRFbyC0FMizyVCrV+/Jg94aWW587TX7raWVhfNOEOcXR/M/LX1DcbnpwiEbUIg/PzSoydPhjKWxifXb95ian76dBunFIMYuWim2e8trF+7OUVJxg+yM7KsNRIEjC7EEpCMwmoQ4DCe7EwO+zPbnfvfbfDhrH3hx+rL17tnilLBPKKe2EKFMtiED8/JcSf7YhSBcn8yzT5B8y/VPnXv5ZPz0Vz9nT/5juJXC+cnA+X5ZZTv7A1ZlA+OFmqtev+4rshNs5ciyrYQtiyFlJqR+cWsm7Ye0+PFAE5Oa4ezKnHGg2BWKlpBJA4/KUmGwaSLmfLHjGLBUcJyd6h0mos+5vbq8XC0MrNZ0hHMYuX1t7rT+ApX8CLvuKTsIfbhfHmRk7B1ww9NiW5q225qLs2dAJFtb3mzDAreRXMJA56pmZXFJe4v9UTrN6+tbqx2v/AXXqx3DvnEJbyoSqCIy5l/qTbG2U6z4GXLBlLwWbcvVkzSH12waguxcOKrQceXSa3So/BLJ4Xg6HQ11EKFLlZcGXB5OrIlVC9NZqGVI7wuoVsIgVYqqBSClsfzz5H2y5upBJHfCu/PpdJU+TX3VS90YvnqWODmYtJwtW5EkddzM+oyczpYaI1He8sb9izZVrLlr/8Hf+utN9882LNERMjWE1muiImlFQdW6IV3WZIhxJmYws8KLMyxJqsuEkny0tm66MvB/FK3MSAMzxxT3GBfu91YXF/DLtZ4anoLC7IP6m37FxGdY8yIztHKqlfEn8lYtZ5me76+0JWhqPjcg++/+87b9/Oi05mnT3ZGp+MUBceWs/KLZY9cgdwVRdRt1hibDF08R7FslJGXr3/6ByTMqMrAquH5Xg4jv3gcPAqLrb46d1J9BkAfPK8I8eWdFfvPzVd3OndUT5WLFz/lPIgpdD9qh692tlvttG8vLirXvNHp/Oi9F5ZxYhV6Md3jo0RExDM3FZECqzkRUi6SzoTOBkyIGIXQFgbHNDjRxIgfU2fr9Nxmj/AlG6WyM1P4UhjGPqCeCdwQa8zIp0UH5WAB4ScROybbSK09FwtgLIcyzFiMqiFlSUemKT9XICif1XirC1fnecRRHvUZsclXnQg/kcZAjhPXKWw423W4L1Fi4+Ph+fTdw73rw9tUixfbtxaXF17ov0YDW9tY/d3f/V122We7B6x4vBqzQmNtQzye4GoahnXSkPiAL/gWKzzbNf6Hl9iXhXkMABMLkQPOO9K90iPn2EAlsDgJkMu8V5oxwMV4UACIyeg/ddc9gR6uWgWWYdmMRCWSCyuzeS7VNPDOe/1i/JqnEXYtjrHtFEbZIMHiF8DE3F1qULSYcB3hxFQmUgkjANYGWSI5+c+SByddZX89wei1EM3yCM2J95plQH8QTbZAfFw/42cnfTTaymfEWAt9bNPDSK56NIRB4SBY9MbAQttIhBbzOgQ49pRZJuJWa0nS7+bmU590Iv9IGA7v8nYjNDDtMRlDaSKhaOdOP5Heaj8Thsq8JINo2ZZKo93aoH9w/x2JUteGQxPhsf3a/uLCMsSz45PwrUcPHy8sra6t3xgePHn3nfsx0p+5zRZ6MdfDSRSGzQC3jdxQ8DWxAjkvCKgw9LQ2Pnjc6vSne51H331jdWlj5sWb9e4iA7nMkdOGpDuymor69dmDgouzMMr2R0oTIDyHteO57ks3v3Tr1vTvz22/c3/r2bapXqE48HNP1HlubNffaa5OlooMR+FgHwzXq7WZBmdOFBbpqlVRkwI7s3fOBH22G3NjhCL/inmxYnwhvJcrqeq5NvwUO1umsfyYvyEbodUVvY8oHdwsj2MN1isRO7wmq/SyzbLscsUPec0HCKZKEonbrKWomRecCf0+GZIsal1bMPLaJjaaYJjAvgTdAjsrRcqlvfpK+/UXX7h391q/N+0tiqDcFZtYklsG0qPRGpVw5R3Bc+8rIWLGi3qgI+nxhRQRUhOpVGfxNv3PQjCzF6MK1yzEyAXnCGw1hIvVWoEsELIcMj6NRMKN2TLWq4Q0GDFpEvoXgocKpr0PHR4KBSiCTvirGXKAsocj0sV6kebLYxGL8hKdBXwMxaklHa4fCjObnFpB8qy72web1uG9T7z0U7/4c9/6rT8aPtzZH43rgo0jp+Qo0nKaLm/THlaXzvkpyFz10vsLNPJZn129fv3W659cmb92dCDRUJxbd+58vlVf2Fi9K6oayRHMx5Vi7pQKTsqVQZkXwwNWluWWEELx662JembHx0+ebD598mxve194x+Tw7OGDZ0bKDERF94l40CVSxNuElNFrLghU9cwnBpxe/3kOY9NIgV+El+ow4oiapZ00W0BcgZtzsjxSfoQmob1MmHkO8PneII/f8j2tgVs660aNFTrgfReMGVQi8IE7OYskn+CN0/7ewWq7vbGyfGd94/b66q2FxfVed16w/PRkcG7LrrNZ1YgkBMB6kp1yVO0Wv4xyzwACECY/rwYmxY+obnxiajAJSOcX5KOynzijf5ntkHMSGvUZw7dMY9/Kegyy6m0JTA5XJKpWw0mzAXH1LSJYjrwtwyinGbpxFOwsP+SJ8ktWwuVpfvEaA1eMJQ9YD/mNNC21ScnfYzrb2cloOj04Ptm3ubtVIlLF/nQHJ5Ph8XBu0NlcXVaGULzr8mL/9usv3bx7896rL33jG3/8O7/3Bw8ePWaGyd418thS8Q9fSZruQBW4AfLPJZrd0y1BAKBu8vZZYoKGHUCAhcBQlq10shx67ij6UVCfJZniC2IBL38nAlI4EvRlZNAs5mqpu5OgA7LlYQ+mEIGSGkeHqV7oR4CHYgRke+/gdIqYMcg2ZqnUhI2jRG0xHbNUTs42Njb68/P8oDzZFLtsPCBSQ+9CqgjBWkncOiimx6l5M9dVsUKs1lG0c5u0nM8Otp9t5mZ+u4SdMTif2Q4KmjiPp6827TUFEjSZrgthmd093E+nJUIZRbQNwdICDLzgAnuNy5B9Yr1PnpxRWFX5TNN1BsyG4ik2ikg99xpx0Ya9Y5RUlgVGlvjsOfsdTe6/95b9bw4nB0dH89TfBHmxTpIILfbhSBLzaH90fnT6wq0X1OhXvPDBwyff+PZ3t7aGqxsxcowOVPEiR6mTsT8+PO72+nqesGq+FfDReTvMJzQAdOOLVdQ/8tZpbedg2ltQrfpw5+Hb9783fweDWFQov3U626PzqlI5dzps1CZ2MaYj8P/DGSh0WDs8O91jEIdPBvhT/+avfO03fuet3//q+OhkXlp2Xcl6SnNj+jTZgGedLqM4vpytehGuk8ZRohpaCFlzlj94Qz4S6ZgcaMqCJyzeNIzslWQVxPADFBaMv4VyhB2hNGiMr1mcZan6hj2Vac0MVXSwrK3MIQwJydJKWXlliWWBw5A8Am3za3U5Z6VJ7NV5mBOxo3Y2ktl3UtvU/3q9HzpcW04Skb7B+zDgmJEbt+9tHJ/+9M9/adBtrC4w4gyPTnfF7snfEG4cPmpstOisnU6MUgn+FaoPkiIMSIomKzBIj3TLP3PnSrhPIVqhkUHty6P0OeMs5CfQIHf4sai6xlHZeiI+hOyUZ73DQinSQ0g7ETuVZxBLP7jBCwq595Y0WrWVbiBthfvm1QlMB9YQiHSodClyQFAt6kj5p29kTsArywT9manZqVPlHuKmMGMs+dmzrZ//5b98enD8zsz3Nw8f0ntlL+i7bCRrhFXaF+BwoLHOzZWmQ629XTcjedA4o26dzM79yI/89Cuf+fzK0o29p+PRFnWLc3mA2vHq2ocJouotxMguinJ8s90W/RxHsGw7c/M9F08Ph3Z63tnafvzo0Whn3J9fWFIBvlZ7PN5793BzCgcZyUMTo6XQMPBgIUIF4IFKOUDMv8zLxzJglDFDMqdAUiYuV+KiY1TUx0Jwi8CUID83QZtMTHhr3KvxnITaiTqIRSZJWX5DlEJDC+GWza+Ug61dcnOZ6syf6S62DMFogS7EZtsgT8wKzCnbfbIJo2Sts/O+jdLqzS+9cOvVhcXbt27xqLVgDZXr+Fg2EXarG3H1JfKN4zHjEZIoSoqFDzbQVqKX+N/SZpGUx2IFFoqWNBgRvoV27kNQpu18M9NOAgzTy1hSYR/PmcZItpluI8iS9w/8g2JBu3IKGDnHfNyStRtx3KMQpkyJn/2tzsvb3Blg4ojEhfIgjmHLFah7wQ0QbvhxeHoq93SoBCFLHc1CdavELds9juKFhlnfNpd9+Hh1cXC8ONg62EMHhEat3bvxF29vvP5jn/761//4a1/7+ubjZxjh8nL/YDjmwRss3vjiX/ix1bVFkQXidfv9zs6B+FQ7AgXFJS9NT8Yc7wKl9dukY0RGj/+FQAVp6dDYBpIB+rHM2XNPMgaekdRhXtPErwEjS0NSp6x0ZlA7S4Wnl6Ja9q2IhTOLKCGkVoRRY7oSo+xsLK1qeXHh+uq1/f2DradbhfzVFGAMP2x3KZpYPvsuYiLOyCFjR5WrxO62muzAAo29qfL+TienNzZuqI2Mi+utd7HCRwgUIWMw1pJd8YpObyIT4ntev76yAVuESbfmOhvrG6jJ1s72odweGkOcRZhoCj7rt2zm8eHuiYRnE3Oc3OLxwfjdt95dXlii5LM8Ly+tHgyPHj/dpVR3+ivbOwcEjsHC8ri2Tz7Rf1FaN26unTJwnI3nF7rf+s435pcW1jeWZUnNqxBpU9n9fSF2tdG43Vvod/sHe+Pdg4lkcAUEVq7dWXyiLuV0uE/wut6e7f9g/23JxLZq2N7eebz51HCAeHFlMdnGUM86jOKVaERkHk0b0etIICetrd2R8pDz9dPhsz85fFjv9F6bG8zPzS7snwyOz/Y6velcfX96uqniJ5PehIjUsH0MC/+42zltMs4PGWPXvvALyyu9G9/+7S+/8/TxtUHzdGFmf2vzll2WN59tTo6X7k5bt1+AK+p3NtlRha3gQsdz59POTG2ttoQ/z8ldS0XCbHM1OZvZVSQ7SUqQ6Yq+ZbWV5Re1FIaxTZHwItW5TPArqyzqVhZbWW0+c1ItUnI1iISSlUaQ9DQSEVmadGg6RMTjUivNB+tIXXRbNnAqKmqpj7bjyWyqKCtsZt3+pTW+8PYCI5PCB0e2oRC2c3bYXl+4O/fC9HAbLse0nJTJUH2+rNOzriLP6j6IBMC8WjMr5zVVPAmp3OFwEsMPr70gFV5WxhLCiQcbfhloPkPerR9IXAZrEIRiioXxRaVAAw3HT/hYCFuhSGFXAVYsRnE8E6xp75xbqKeRhyWfxh1aIKdxANYTKGOTZ7ryTBZt/ql2LjzZYmoejdlmpINH1co7wLdMBpYc0i5AVCP+J71gEXZ6njtRkF0kTkkVjK2j0e4wWv7Sv/6v/1H/9/7h2/+NGIrR4Um3PSdC78n+3vJg0USYJHQywXwSvqIeGYdcXhJMu97q2y+GOejaxq1XXv/U5z7/K9N665BMPjvo9kXBgIQYDtoMd89JWxEeFfGOjyaiGKwD5b8ocvxGXS6SJtFg78mjJztbvF6jrV3+4fX6YG9zv7mxtNc8/d23v/agtr9jrw0xqhgr+QvgYgZJedIyS4ZKKAkryASFC+BymZ2POIKQRhWZqPwaOltwFgRdw5q14LciMXlBZi0qminMjLs3XlGJfJC1vD04nUbLZTbS4YgYJPglVEs72VOBkwR71Ua06XjQjnBwcQj1OTvAI9B2WT+eyhpaarXvbVx/+c691xeWPt1orlA7RdDYtVwIoppscwz49s47MuMKWMEazTnXRdhanH/FRhoPb+yFUAwkUm+yjCJdr0acASpMEFHTEVyF+OVTdzGZ6qKxhocm5gBcjSlnYOHXcpKnnIBH3l9AX77qTwGhHl/wYDdGUix3XYLdNRJLHvBGsCyhMZKTw+hILgprHJ4mA9ju6YeJ2k5ctjUWC3BEsUgf5xNF8R8/em++0+w0VtaX6PRHSdviopq7cfdmj0587/Yb3/3e977z/QcPHvYHS8DBFzk5GQ8GNzvd28+ePlW2ScmLJo9JU3oEI0vMjyKfGIiE5BfwGF3Q4uqwnMRzJWMGDSNJJeMuAQwhbpVJITAJ2vietefXZHZFqnUdsfMVJvPVWuSszSkIF2tvij0ZPclXkVglo6tGNJslXcRygicjrZ2JYIVebjuebQJXlIlyIDJeWKZ39tqajBc0kWatAkNQzy0lN5o7WQlACZ3ZQ4l0oW+iaOAUYkvnitpOsvaU8JhiRc/TZDeUMlCIAl+U14LJSk6cT8kHGZeYJZlAR0e9wcLpfDyncph2dpV/lvSr7OCAR1aqbqwtdbFIWmJqU8qC/TmVuLla9Apo8v52e6HX3TkYjff2Bd5pYndnv15/ZvuPF199XTXE65v79x88vH//0frKar/VY53+5vfehGFsEXThoIjzk2zNNFLgGfYW6ScLghkF9szWx8Vl3z9SrGNoZ4DpuH003O4c7MpPrdmLGB08Qr+GCO55m9Y+dybcO9Sv1ey31TFuDQa180UlWw/f2bXH4gs/9pM3V67/8e/88zf++A+6c8ev3ru59ebjfntJ/NrBztPZfrcxv2HfH0aGuRkrsi2iX70Y0nnztFMf3FwQd41Cn9w/njw5nz1qCDVkrZjhhC6rtixxZ1k8wbMYOehW2BZZEJP0MKG7aAohBnnG6gsaJlArCmT8BRW1CmbGBOZqGgy2IF6OXA4dc0LFwFr62pWuCH+LUo7a7YcjYARhZ1imLWNjMEhrOJf3iUFkoFcPm7Mqcrjm2F4FcGEc9uPCgLsCMxJw4r8ZXJk45J9fjSAkt3xitxfEotAWL6vGg8S7wc0ZVjpcPsPiQjQivOcE39YXZtJIzh6kHHmXRtKbfEQPL93DrDE2ayYv1mDuB5ICvqJM+1JZ9A0XxtBX4pAHSZgFfALAcWlv80x5MPB0Q/oXYJTWCtfHjHEPrr6s5Yrsml2KlE2HV25d37h3uz7oPHm6u9Fd3BtNbM+wdvvO7rOdKG+JcKMe5FnklKDRnOub69ER+z9DNe/uyguvfOrFT3zqmPWolCLIninkheiKVMh6CouVHUYjgIZyZiwmIdY7C3Lz6cnk5GCPz25n77gojUdnJnWuJNFwiuyOhk8Otw5qAhyRZf+DT8YLYj5zwffCMvK3MMFyEnd9JumjjjRR5qD6sQAwbQRmmSJQKnOemXQZ7SOZ5Ymrf9V8garfC8NCtxJqztWnX9Z6NCHt0XZwjEy4KJbTbL+qeoaXiAyhJlCdukIhz2uKkV9bWrnen7+pgMbqWjZLGPSbw902RE5/kHFtMxJbZtSpOO9gV7g7HhaGEGardhFDtNOwB3AK8gXjrj6dgL7PHCFROpavF6y33F+AVsEnWJnRlXVRVoQXXvyU5y+OQCy/FvZcrvtW/VoMljkvS79c9RF2EkwvnwVxnfMzGYW0jQrwFDNeT2IHdCe44gdVyEfkDHYDbmvgVSxserK1tTX79oxd3+UUdWX+FymeeYSMur6+vrKy5tNmQIJ9nm3vwl+lHDYfP7q5vr6kInq/Pz7cF9wUBZgxIO/P/k6YJI2i0SbDYUiKdHitLhfExSuotr65GV2TflK4LK0u1LVgjD4GywuoM6iEN4t5js8YrAwgHP6Y/iTTJZlRrLsniqkcK3p1RrvF8obqTYwOPOgREPAUidWDTlzBSxa6nMThiIZ/kkrM6ZKfHNVTopmu39hAK9i7AJOmbakUGhWTOj3LjkJZQNOYokIToDpnbXY4pBvaLXh0+oxxKx5iUwMP8N/QSNOMFpA9EAbkoMwTVNeToihw9/INjBaWbOkYJGQW5r517nflKr0F9K6JNr5xDfjtOcewO9rbr8+wk9smVX0lnnBhxpqcZa6WhLx3sL88WDFT+zt2SDocH05JHbYVMVKh7KO9kZF1rxEIIlKwBwj+0ivDcIPA6yuAZGawLaoJs2Ey8sCGtDojirstsLl+2tzd7T17NjO/Od9ZnFlYaQ4WbWlb47Fmju8MbCOIb9BLKUSKbbXaTA4d7tHpRIpBw7BnZIW98vKnVJLe33p4/3tvPtm8sbwwUhPlcLv2YEz9v3b7pLOYSbALAa2TMM42cLg7FUTZmRnUBlERZ8dTmv/Z7LRnVApFzeyeyh0N+SAGl/VmOZpJiMLvIb4SXYrpN1wAlcIY8KBw4xCGcn/uLVwJ4iVcKmsRTApymsx8DebkiXxePATHsivRQuFNR7Yr5nkPxnsFcfN0zBOsEp308jwm4YjxjW0h2iGPDGtInzhD/LHs6QgFaSEN1tvzDw9O0avIc+aklCIphKYyNacX4daFQ0aAz4AZ1EKCM28XVD1X031ddkDdiPK+ZBn4XgYVahVq4tMwCxDzSG5OgGdSxAIb/QfriFkpcaVGozbCqvTEUnKkQUvFy2TqRskKKadYKv4WLQofzU2xSZoFR5ZSdUQO0C19wGfxy9gr3BvLKJsXE2oyU0T6Hd557fbrP/rJL//qbx61BOdPON9mxyxrHdRIYDLyoepYVxXhRkdCx87eoQLaK6sbK6s35peuLa5cs63dYHX1mLGVmSe++AQpJ8LZap1REN4Mhjk1ebaEdqo7rqtnZ7ITjofD6Z4Nq8oWOwrNJraUZnwso4zPQEajbaufbD17tr2Fv4BMSF2s9RUuAqmhOq++GnPFTy5Gz54MEG6q8OrDn37N7ATKIJ5fCwMOLgB3EPiiWad+LTMVSS1PhCVn9kmnl6+v5jccpy7fygziAYlGdVPEvSwawciqNM9xazJPSImp1RcbzcV6/e7S2q35+ZfXrt1bXllpsVvNNO2NKYboeHySqugxuMtZMCyNohtdyRh6l/AHsTNYfBJuEE0MOGYmt5WxlVUWeo0OViAJ+pWjnDBBXqJquRhYVKCsbqswLxiTV0f7z2daqNp//qQ6d/3qyM1ZRBevBmQ/XUAbRiIGAbtL+aeHfrUGCj/j248d2LhwLsMXL2GJhO2yO4vei4mJ5BhEsKynx5NnT562Oo35RYk5Mt0GsE2cFsaDx8wPFm/duwtrb92+/av/5FeF4+K4b7/zA+zjxo3r6kXY4J2ymVGyT4AuKZ/ElLWOgBBukkIa9lnM8SYCBLCVdKAc4U6F0wTIUD+UAiZdwMpJdV4miOLF2cXpFQxiMYQXKlFgvMRV3gJ9wEWUjASq3X37+4wgvVvDcxl8SyoRPud1RtE6OVlZWZFs61DXArQd2tcpb3Tz0sKiWA9lKcwYfHY5r/BiGQ+VyM8qVspahYkmLEvlLAluJLiYksQuA6ApcbAw+4zrNDWx0xZmQCqy85wGMgtob3ge92BsL+Ga2KZ0i7JhImAYMRkcLwZNjXMSrzHZdpvzvT758+Gj+42FeYosvu7XUsOKkZaZOpHPo/Gk37oIL4/ksdCVkjhmxBiP3eAVDO/IksBRHugKFFRwlAcwD/aGQOG2Mld5tYFUX+GRDAywYjxQVk14eUPFzK2t8/7T076I0Vs1Cvi4f3g6ToFie4kpXqXSKe9Odj1vkIzARmqnLLDO7KINIeoHk8bZXPveiz/b/de++pWl73ztj5qNk7YQDRXaTg63nrzLS3Hz7nl77Tb8mJ0l8/EF2aVM+S6M7WyGXXf9OocSqW961j6dPjs53MPUYiw9HxfFK92HYdZL1q0/lS4HpGT/MCeibdLoss5yi/vzF/UJD/Aguos85XIOuOBnH1mChZ6Vy+Und0U1WFM+Sf3WWm2PhHU2YxdFXLKO3iQyG/dNtkKhgS5oOmoeBswO1T+1fs/kn8aWDeAwGGOuz/Xtvib8TlolgS6e1NwB33SKUxlq6bQJIlNUgwhJZozSO9eZS/LG3BBDMXws52EBGXMGXjEJg41em0m/GF/OQCvQCx0jiIXcozrw0g9adCK8O1Q9CoKxaMSiKEDMR1Y0bNFQWdGBa7TSilRe/lTelmavrheQgo6GkDD7pTFB6UbhIN5CDGzWHjx9dO/6nR/72S9+7WtfE0APy7DJR7s7i435uYh7PaItI5CclandmSnHi69cv/HCnXuvbGzc7fSXBZlagookNHpKLuELDGLUZeNBa9CYGWGovPGqO+izV4vDGKoleDjcevb4ZHx4rtyqgWakFUIEQaxElJYRRErAw2dPdk73Ce9mOthVIFEmKLTu8sjVyyMQBhKqkLe782M/01kTE/kR4mZeAvcAL3+0EazVKw3mm+9glwkL7kOMlBoxxZl/jqHMBrsu+hycSuRczMT03ewlIZzoeHA0FY9LghTcstrt3FpefXFl3TYJd9UBmmuszrXnhczQFWSmRJA9aS4QQ7gSUxJPlworsuwtyKRpqkCE9yAf6KdeOsTsGCoiFcv0JaHReQq9vqdzWW0XJwZQwTKXLo/ye7nBlaBgmZdyUmAaWS5HgFKO6sS3/CuguPz0aITSamKrM+fR4VyrgJn7rQLgy3FOw0DOpYyILhqJhwJFZmGb6CJ3RycTupnZj5RvYQQP+C+z0631bcuB9969L5SAn0OV4ZRAUoRXcUrZtDhts3ntNsvCAg3s+9/73js/eOvJk8dShvb2t1+898LN2zcIMXHu6ExWl2gko0Qik8ujrxV/8gmA8lCthexWH26NFJuTyGIGDeABTNTzPKUJc+F6FC4Hw2jqUcdFjBX5ScUB0XEx2h4fuWadNVgpk+k6J+Do4GBPqUk0JfQg2xZRWRmfZQ83KwYsEza66hE/6b4r1ZyDYtVPvGdhcT4YcsLeS/01PxbeoSEQGLrtFm8QmcDLowPqd1xUR8qIE+TcozUPRGcvLZsajJO8QxrUHRLJXCJLE/xa9GAQOxuPDgvSJQHJxGPe21u7B9myIosbltq7SsyzAhqc7Pk6Oe7MNfptNfnnnzx4r2tTuo21zd1NYrdIMZsVixlcXFnd3nymD2xjwtr39vYb9c7K6opduXBf9aFv3brx6P6jZ7KVTk6XllZowPZllMDsEcWwwEqxVqvbBFSTCIfxg9jkI1HxwGTWEwFFmGjMyVlKee3RaGdrc1nU2JIM3nZr2lecyuTZIj5+yjnRCWSE7FZLoai15ZLN7W1tL9SXOMPlE9mMYua1T//o8sr1l17+rX/6D86n+xvthdX+4HhzePD0/qjFwdew/QOrYIPlCx9KIPzZ+e7JVCACwVqTrfnm+eLk5K1DHoDmmarbqcEzI0zOGosJ0HRlLqiEWcPkRRwDIwwTNsA46ExOaKUVXDDIRxZogqzD77Kwg7RZwBE5EWzYnm+OkNdQMrd2as0bMzP92sxw9kT1U9Ui9uiI0XfRNGDVhq+xjkgSREo9hEF6Acmiq45LTVhZFs2UrMbObFmSYJg2RQNZx7BCMfZ0OXRJ58MeqnXD3JWeWDtZ7O7IEak1UaCWO3JaVLuQESQ7fjE3ZMzeVxGc6nsGHXDkZdFdqnd5YciHpjNSAq7GtBTCFPh4JMMPr/JAvhNa0kBZXUH8NBBkh03OfddUOppueAa085k+G1qa8EMhCHgKgIQMepZXKNdB7njubDh7dPP1F+988tU3vvpdqbbd/sIJNJ4qFSPkQm8AlD9tbWl+o9tZev21H1levCGzSPHJYw5edXSi1J6I29MDvZm1JToXLCJTcKQp9BJlELWZMIzR3nBvd19UyR7V3SqjERfuxh5ARte1WZsWJ5/OtHQb20ejR1ubIv7BJkFL+ectRc3D6wpq5UJBywzZaQCZu5KmeQHECpSXnwFLxBz3Bu+ganDAxaBlENAQIgX53WRUlAeI/RTKWcCb52ZbZ6ex0OXSBaDL+rCqU7TevIZlar5sit48nCiPuz47uzq/dHNl5c7K2p2VFTsDrjQVDTsTJyiLn8mMUU+AFXIr6opTkIdRxHjhqUX+kiAhLSSKL4Vjktwc78+yDEbEMg13vLUcfnIYZujp5ZGhliMnBVkvf3n/r58KcIJAFyeRDQMZYKjuq647r058VsfVS4MImScr/H1Qu73gSJaKf5m2vCE0uvA0/ACFHtGAFVASmWZXIZUhSM/isYZ0LQGi2Ely/LOETVO8laTDFJkZq1O49N6iisTXbl0X4iw2RPcOT44TXIxe9ttf+qmfsLXleLj78MHjre1NI+rbLqDXUQNLh4qZQlKcfWoodqUwbRSyLEdHGWn6aoCRR8uReSmexVCjqIX4eCVNZEaqp7LWY6ug6iBSWb/xZaPhgrxxxJqNEMLFafCN857JTallZki+2WyMKJrvNCiAhYo5KAqrN7pD1KyLGDCpwtfSvRi69YQN1uFXXPBkKpAtFFAXBEZ5scfbvZ4WdRUWGZErfFEepE1qB6zQZ3gUOlsCcnSY2J6ZM/gy4/56VhjzrCAUsGNpVNm3aWek5bI5UgSgOLWV0wstivZMXAjZZ3xs26JNcYDsuogZT8Zyv+aweMaLzd1nOhA9W4ZokNu+bQoOk1VS99yIgid6WKpPr6yu+fr4vcd2XmIjvnfvRXsD2A1J+oTeu0cPzZT3eqQ8RX3PYk93WSvIz0msS8lnhlx1XYEo0ewYuB1hhgf88JTxelPOgQDxaWziyqdQ8jwNO4+RRjuRS+awKWTmRmSODQWME1hrSxvXN9Z/uj/7B//8nz589yGFdkGUpY2RRs/uf2946xOfpSwif6AXiKSeFL9DY7I1bC3NzQwWvEiAnWA9FOdkdsTcn1zkGpE0cg9pLnhpGP7EMmguLNTYh0OfmCOiJlofFN+KHJJao1dkHeZCxOgLMmDIfkPSshA9lEXLT1jQs1trb8TJXRsK28PxxSNKWbZk1TlRlcCbAlbu4fBRrDf+ypDRdMyibJ1NiRdKluJwqEfcFIG/A/ZYu3BLiab0K1Kd3mUAloUOJgUyuFa664XpmYtCQFB198MH9yest3oWRwtENO7PJWQ0E8pTYJUHM8gCBu8siOzm/Ofdvltohh5YFMHGcDQVspohBmq6UzHZgD1ruwy/6mEGVY5y3YMhlRc9qfpTPr3Hf2nI37B4VIyYP9NbWdweDud73S/9xV+YO+2/8bU3nmzut0BvImdLuPTiQm95denGresvvnD7VfXaRC43WwN1/MVDSLC0dPk1U3ymxikQwwx8gjFV5j7ZU6XAaj9vRQMPj4bWHIoR+9c525t4XZAlURHnRAM78gEKU2atudrD7Wfv7TxJt8N6qxkJNMqVMshgjyvlCA5VR07MZaDwUcfFdUDzK6iVO6UJaMuMpP0wYXpJkDFmydxYfsnvJoXx4NxWu+aSJKMEiS3ITGIyXYoTOJ4PLhQhHI2jkViKfr252p1/dTD3wmBw9/qNG6urSx3ZCSqr1lrokF1c4t/ltiQWxT9AgfYW9EuReuEy6BnpJCkpkzHqHNaDa8XVDrHAP06MzL37QuQy2xWylKFdACD4FtzLUZ146uK39//kisFq5AI4FyfkwVy5AvXlSblWvuihL0GvauVE18ouSXHeAFiglheHkWcxsD7pYzJmjAL5QMhsqD49HbLm8Y7wlJf9EsJ64MqYX5KIV/Yjw5gRHJME25I5NZtdc+ypfjQ+3HzyVBS0DXpKjEFJMC/pMeJCsnfF8uDV118RM6yfTx49nkzoUdk5z7bsliBSb7M/i1rA80TiSTqNdDAgpvC/YsolGRldTFhcxlLAqAOoSTAGD6bUujM0LMvXPwRQv6GQVVEkjURvhddhrdADAe4iasnUZhbWw5P6zIEYshDQlKT2HJOnF7kfc0V+mV2B11syg+Xwk7/ucd1FvlK5wuzS7lFxRcyjB4v3NnfE1Fw/sdmVLQ4MK3p5ijMHDwpB8V5mnal4C0uY8UAvWVoICAlsyCDKJJZ7kUFMR4ExmyzhxMTEpOgkDfRcmUxteq8bdK2sbULh2d7wyLRK9SX12CXYCjocTVS84g6Xcbh+60boAk1dts95V6KSczGH9exBl1JZdmXAuXmpNL67t6fmgAUBhSQywQ3r4t7tOwY4yobBlsP5yFaMtpchSokZ6yTaVoeSvaWgODVNnUulyHBiVkFqRJKMEvim0In6BbKauiMFKXH/9lmdQ6iPOMnhDL6R9iSZTaYLcxMzIsNVjPbJDkmHbb8DSH6i9QlUv/7zf/m14d4Pzr92arPibFVZPzoZyfNo3u8O1ifdlOyKGzCVJUW01lvjIRPDUVfpnPZZvG8z58oBnxw/A/ei2YIhNqzvSR+ANRx+5svygkIh7oVpBf1yHmplsZVfACwHquUB2FjxlbLCyYXlW7DMhfK0O5k6UqKZ+ruYWNeESpETF+z9TSaMIkvcPLcJqVvhzITrVPyDeDAZ4qQY5hWaKkNq2DbUymKyGiIrlHgut+ovvL1gwCERhQGEfgTbUQj9L0f8jtrwhJZhBBxL9D5iLR69cP2MCR3OyAp1y5vCehPTHPUpP3h/4AZU6QaukxeZq5j4ycWZRasnTmEjhWBE9pKDl3UcF17BfKPNEgCkQhTTvUhAvqR7uZhlbqG4np+ujtyp634oakf0yMTlMs7RrdX37HSP90dyEl9/9XP9k7Xu8e9++dd+p2eHj7P6vVuvfOq1z7xw++VBZ62hvrbYPXnkBkBDod5KjOgJ6ON/t1SxpCT+mJL41rEHAvBoqNbw1rOnVgcezGQmUwPmGAW6tn+wr9u240mvAUlcOj7eoQ/WvEKo/tHM2btbjx6dbUKdI/JfJe9lbNXoAkJn+ZK/V4fJ+/+x9p9NkmVpntjn7uFahU6dWbqq1YgesbM7q7FLGo0gCVAAoAFmfEEYvws/B81oNAMBI2FQxO5iZrFcYGdHi57uri6dlTq0a/eI8ODvfzyzukYsjS94K8rT/fr1e895zqPVyRFl6q89wDEgKcfmgqBnFisYmzpBTBU0iVGJodwjxbMRMEd4+IVf0ugM2TL7SU6UYRAJZiSH/KZfux7YXXR12btc32733ju8+87u/g/29rXREFrKpjM6HC1nKh2zaYt1hQDsBOMSmJVuVKILBzu7BpO88IsR6wTsPNs3Eb1F5ysuED/2dMuPxRlqJG+GWHAFoB14UBApJwug/CyXFJzfgDCfNkdwNWpxbhcoFewtV/s5wBbIbS4NUpWjXJbnvj5CM67DfaO5GFbuhkjKagVeQRNIHxkcWAbiNAkO1fHqesIpiG+qpeFDtl0vPJOxZI7EB9FDJPDhJeMjrjOOU71/t9LYsZ7k/ovjU2u4e7Cva0S9ediy31HR5kKhbrCYH9y/+8uVCretI6CBuPUtqT1eSxoP11mxmQqk8CFSUPkPACYkcDn3COOlIb6Z6s//NT0yNVwzT4p3OrOOI5qagL9wPtHxSbxUw4TyNAplp7fbpM4YR5dzyHhXkLbZ6SG5KzH43MqrW2EaUrvJiNdn5CihKaAx/+LzcJk3RK/YsFFP7eJMjZUZTw1KvnwwYPPb0cUFaqPZoTI3cZ4+4DmEnPeEi5Noi5vVWrltbISCPxYSzFzjQQ4ebJPudHo2S2TK+92L5y+5Ix89egtfl/2gT7NbFTHsRjGE/NyAS2sqm+mwwzkVr2zBJZnq3ukpVxE4k5EcQtXu4ODu/SspV8+OZlK9j4/cartfJ7p5uY8k1J2dApqZmjLJKjB87/5bcYAHXCXr7fpagD+w4VWT6N5sZessj4D4zHt7N+nUVxRtOStEf/LF5lOid+gWwLKYVdRQBggNzIhCqARK7gzuxf8LcYUNKu3uYLc6fjrhM5eqUU3oGQbJLKuNF8vtl6M7b330YP/B6Wdf/Ok//xfj06N37zR6O90XX39m2N3dg0q7T/bGYgbetbZ3vaupLUZkFQ8r/bsx+Jc6fZZs3kSVRGGjzJbEJTSPxvwMVaXcYzPrCFDMFHoV/hUCpTKEBp0pZA82b+gQBcEvkgvAsrK+inuAACGW3ILCQdSp/vZR9m+ntkWHWKY2VVtYxBiPNquU1JHWk3AvzlEcWSlYyIxi2YZrRKBiWimUomCZgVvkKAPYYFSWLDOLaz0iDm3nTBFb4RJvLo7iTgbnvqaadVCOVyROfl4UjAjDAo1MNI/2havybUDgd9JfywVRRfIQDVKwVLaWJ/tpukwwwQE3dqFRcjzBdoko8WrGUxtzyGlbepeFczPGoccids8sbC53QujunmPzSl9yJdIQvjLtMCWMX3zJtiDd9vnRRbM6/M5bP2z+jd3jT5Zv3373u29//3Dv/u2DO616fz65XvAB6icmVDXgS1WnoR7SR8U1K7asEXHYVdMFr/hxCN+xjQTPNd4SbgvvyTBE8dSzh4kZpQ3wFK9m+y+xD4KHLiEWF6uxLjVTAyA7176cnp1XxjeVAR013pYNxMLPicg3R1n7Nx82a5tP4V+bs2Dy7TfAXeAUiDu/+da7FG66Vyl9zmLwkgWEdoZxdnMHcIUmUIqjvnaiGsuimDfuK0PALk21G4GdrcX0dqvzdr/7zv72272dtwe7D4e7e73eShdAZv10THirYGfdltZSaTFkKTmyKJrKmrVaBFOZbKwzyrfR0njMBb5mqLG1gtfeo5LNUWYauZlziCIAcjgR9AttWRsgD4ZGdBfUo/iz3k0gx+Y+LnZg028++jcwDIT9CkzAAwTKEdzN51jhod188kDI7uduSPt1cYR27lJuk9EbDv8nQlonJ4ijRPLkYq5nwsVscmyz+bh2FLWI4XYb2mVgd9hju9vpiyCyx+hxFBFSkf2YHyPnmJORLepJtfuvVr787HMyb2dvV6rpUiYc26dFjwo8qoulYtO/9bf/lnjhT//8x4pct3d3kBOfbdKPHQ11QW1FQLzGsFS3KSlX7g0CvrNA3uPiQIGxE2BMJrPzO3PGeSNpy0xfQziRgpWRthk7xHOaoyt7rIP8YjwdDqDEotVq3zm8RVtNkrAIaKe9u3eggwS1gN+VzccRIinLrhse5BFFFbvSjsNNqBFZRHuyekq9rofUvfv3SZ2Y9hOAFgKJkpE1KII8yxWDJAnJDEw/cbiDwQNCmFDyB5zDm2MU4uVcUupBC4bE+MaoYTuVyXaQpJ1xPnvxYmd3+O7bDwngLz7/FJYeHZ3s7inntfHUKb0BrMg2cjgW3Gy5feuAKP3q8ZN3Hj7gNFanxIbiRPvq66fTq/GD9x+NR+P+7qCx18Ca7Im0Xsh0ErCf2LLYvKgXXLOvjk6Ojo7YxBbLVGHdy1fPf/f3fqc/3CHWd3eHs8Xy6dNXgx1q8JWuKwApYbVxo8ZJXlu1PxiqApfXqBFLggP0kVJGNdZiutO/dbm8OH7B7VfRMBIAIKt+YJKIhrtYkIJvAsBCoFgkUd0ddCZXp1+fNTV96t2+mtdmF1yyTP02zWJ7eK9yp8M93/3kq+nW86m+5ZOzHc72x59dTVe3P/qliu0L5HkQsnqfXarUbN6sxovTaZv7b/+OblHtdUU2niR/xo70JQYxnVz9NoFgJZBRmJkBhRqseLAPj0dEhUtkway5dUeEzoRlFL9XPDkykCMxQroO/DfCd6vRbvWqjT7lE5MXXmw37b62x1ZcL+fZn4U41nVyNbpZTrApqIc9JkOrqvNDdu3WjzISlgZ2vRDfjAKtv034fbgrgvesDDVyNJyncKa8mgPOkTdEXHHwUhszsCj0hajCw/IDOLa2Qwb+jflGtDjH7UpLgGIpuNlMOZwlOxLgMyVoVbm2obgeq4ZQnlzSIKA2CqHD85nn8UYBHgEX+i7/1rh4ilA1Cja0864JSPEtvAyk8T1suoDR/CIjOevC6ooHwOB4kzYKTrxXfos1sjMkheiufdO6GlXuDt+q1Y9HTyZ7w517g3f/1//oPzwc3tnfu8PE51o5H/EVE5MDehEjtq1XbvqZymMTpxNAiUtMXLKx3lLCT72WMTEdj2iThKYI6GI5iw8rmBCzByhhQvQ08I3mvZZfQ8NMewNpW1ZSbuP6enjn9h/8+e/+/vM/k4IijBfTzh2CPgVdIghz5LZFRnhTjpzZHLGAs1zl+PYb7HJz8ttfRfUpzNOC5DxBjL0LX5loCa1hS5hPvAdJPKYQrAeDPQicc/xgGrWtq32qwrry4YO3Hw0G39k5eHe4e0vmH6elnoHnJ+1eS9gpmkQJ5eN1qLgwPD+1496az1CN74KF4P7aDSjMh13RJkNHweFgq1nHIvToDPn1/MpUcMjXoy8zyMV+gw8nXRsEoQQtx+kCZHjkBugw0GAJWJcA1KNIm8irv3oUcg7Yy2FwAb0VTGDMKe8cRHL52sVyx7wtJFceEKThsY8ZJ9OJHp08n9Wl7fSmNm7PDufcKKK3g85w0On1cVeVgjpAwZhoS8VRFRU/GoiIEDB6iAFkUmZTNKbLqzPbvu/uHh4eJg2nlDESLiwhvwQerHN3f/+9999neiotPT0/37ZvD9U6ymlcZqAcDIksRTeZiyV3B4f3biH711NBP7NN/nBOGpPsY8LMR4ebG5efhMUBKi9nRloiXxQIWXtUV6Voxk3La8kLFRltMYc8ej4/hlLaIzMu69nknkeVNhkpaxKbYXhEnl5UJef93OEJvo1ITjvrluTeUoJJ3fZfmICCKpGMbHtSDpD0k4DF0PybeHoC55HcJhKpnazzKGlJtEX+uI2Lsbk8y1ITz9q+cM5rWckNrf2FrGCSzrhwCunTdA53QL4FVxmNdPTL0WTWazboXJPVstXrvf3hdy4mJ5P5eV0C0u5ut9IFVT5/6SVEirZWOztLwv7adIr2Q8+QgbW0IyfrzyHPGXS03Foum6uFDYkFKfp1AWkbDelM2dD+Qgvp27fuY7PG88XnX9lngnKjLzO4IAxjyw2EjeoNtch2q7EL03o6qWkDYmd5mJWdzMHAXvHphRhjN2X42qppoeFnW/297tX5tRYg2oh1Wns6lijtqHe6Z6/Odh/utn7j7/3Kav0v/1//9dOjFzupcZgrex6dPq9/Vt+zWIdJHDw/elrvbHObbqUvf1uzt+p4q9I5aDWve2oh1seL6Ulla2Xr0ZQerS8UnlstSxSbIDzQWha6TdFc9LwwSguM525IOXFqjs9gatYOYvtB2ApttjizGXvWV40yNdPua94QrZVurU4L2QYE9hNTITJYi4drwW8ifx4MinuJU1egXl8ReaAlYpeODRQ7UXaHM+Eunh2r1U8iaPPkws/C04NUISOS3IMMPQgW5soe4u5O6B3viIVRZpnvMd7gtcl6La2pyt1yUVnQ9DqOshBdH6uQXyhOpbxiga2EKq18GZSxAVx5XDhIqDlcH5QzXvfPDMIWHTHx0TyW4BxsBN7AwWDiwU2BHl5NmYa32E0maWWSpRb0wVRAhHMHV/BsPuOtSxEW2NusTrut1bCtCd6FAv/t/fbWdvNgfJzsDdMWfc9a5kFcx8guEbJsmyOKgxstS87mbHKlWyD10I5gc0nOMzUyphinQlLnNpMMozaqAvyCIhl6Kt7izEBMUWhqMTjq9eP56Pn0fJYgf7Il3GEDgm/QDShew839MsvNUVa3vOXKD5g3x7ffePLmZACbI6+ZqM66WGOI0aoGoik0cmDDNW5f73O6NG/pdrXQuG60sFAFRevKfqNzf9B7e7h9t9P94PBgT7JVszngXyBp1tRAdMa1WlAGIjKnLBX0IVTC2ShQGBU12x8WofwmClR2Ek20PovgcteXgYctJn5aZvd6BhllgA18DHTXBmlcEHUOSvuYeWGyrvPQspSRzfnZZvq5xEhCBAWe+TL3KBAKogcuFi/XbO62wUKvBM03QjdifiPoPReahsw24C33C3hhLQ+tb0gK85Xtq4ugzNWpruDi3R1twLa7gwGWq2FsXKNkUvgsh0s54hUKy0DaBEyGbZoWG2VjCbwWFydnL5++wKa5KzvDbpKjRDJd0cQ1zK/a7vcfvfsehiEj2vbASeUN3KJKhPZjMfNOBDjWyYuBruZFnS/nbhotIEj+p9mWM1HCJHwpA3Nk4QjsHGW6oKDzSvavDgTpbnJ0G42+KlLX+pTGUhRpSbY2tE/gBlCIwoRJhzvt7rIxSZ2vS4VwcCk/N06vxrB5BR/AUtvDGQtKZD+RzNQTbr5JYmp5CtQIFJuUAmwu4wxKmTfkCE1SBSBEEUPxdeIe1BHrDyShfXgRlA1CAuEGCdyCNiPKbMRjBr1dqQVNLEOJBMf9LNsjPvLgRJiwrLe2nrTqmlZylM9nk/5k8uDB3QeP7j973vjqT78ebKWEl9V/dnxyR0/agCIbD5PoJe3wyj7YtJa9g8M0GDk5NVPmkVXXRaTbbYwuZtQsu11BjG6vwy4/uzg3QviBNdoowoYwung+ff6SySk8UVmKpFExEmenYlFOYXl2gVNeNb1Yjs864z3+Zz6LBNxC+SE5firvJVCpVoafvBJbg2bvWqjIjDFXHdOW2KOsKlXCIuNS+gTTtj/4/m/eVD/9o9/75F//697eXoyqq+vj4yeLrepdye8ae221p9OFHRliitYVgirBNrpBpS8rvLlYtCGg9O2tDnkZory6uhBFROhWJSTMWgiT94Z7yS9D2xmsF2gXfoEYoWkWNgsf5lDCdHQqMR1ElfIqNj2VV5GuyhYNrykk/a3GTqWyL0otD0H0QPhxvTqLcBEOT/m8gtM4o2MmhL15QnA1r8F+cI2ejaiCcF7zMBkG+XUeb/hYYAbpS+CPCpBbZ+R5V75B5Ll5uJMJblAwN/c7CoFTSNwPYG3IkFz3mqOMREzQ0kZq8vDI18+rhwRLXY3gQcaPApaUCLqT8YNP/g/XyzWxXyLWy32Y9WGA5cjszIvuCQshWVztRYkv3upCV8aJaONfYPSDkFq1yM1r8fJqmpzaV+qsshAnOms053zL65upfiXbWv+lIWl8/FQaxOMG5DFWKAcFsdIAFC7wLOkYkHY8k9H4esbNH46S4kCdnnh2AkkW0MaDZQRIocAStAAenvFsxNgIkm/8B8R85t+oPzk7+vL42VzGA5NDMl2BfnT5ADxiEfytaMATIJUFcy5HOW11C5YZK3BDC+AEAmsG7JszeQ9XN68RclC0KAybVc994LHr8wbqRPP1b6vBKaFkqNmz72lja6e3c6vTe9AfvjPceTTU7aY5MBUJfku9FeaTtU2iUpe5pbEO35Fciyjo4pjJUuPN1EpiQVux77d3ErI4ZtW4ZkJGFN+YAYRLRhfLdMthCt7nTC4rRxbYYOM/iQz2TQF9PtG/4GYujsiHKGG7IPHmCNaGEqBcUHIj4HP7cvfgnbMu8Q4cN4jntWiCLgqZbTByc6bEdJ2Ir6X8F4LHgnOHLAF6gLUGpUpmMp+TvhMYo9Ci005VUVMXxGT1GKMf5FeRG84mZGgfeRNAK9EkRWUi3LKEuEeShsgPE6Xpqwx58eSZdCTZ0AznXEQzhY9+W7b+kiMtGc5ntTGysYqTAGnaRQ3EIGSWmtTIE/n7mGJFcFKowbXZ6+WjUCn0KfNKZ72tLRsHBT5OlaNQK2QGFWIi+zRwdSEPwhhDl7hLKK1ydSDqKe5gu83pdF5OmaEzHPHU3jrX62w8Su2Qm5meZk/2T0EYJcV3I4BNVgYWRxWIOcMYXa6r3aaK0tX44kwHCyqF3l5b2m84Che2wPHA8BhConRq41pMHrX7J/hireCKjJQkjhVmVLgSNTSs3i+iA+FvOILgtKyGhUJsdrxR0SEEekWUNhdEBuPQ8hZ3kp6d4fW62o7YI6m5O9y5e2eyntl6yVbtL18dH97a4Ua7c/e2R4uNy8DmIpPRrU/usDs0Dj9n5qIjuqpraAbdTtf0p5OvLKXIDo3QPodb9d2XR+l63er0JvOLs4vTe4/eunP/3u7+zsnxufEHb16jb3x1ASy/hVxxnVgU+E0vOtOLSntQVbVlE2draI/tuEkibGD87Pqqb/WJ4059fTZmYKmDw4ynx2dytJQ62xFzIFKwWk8//UIEYfdX/tav7+2Jwb362U8vty57pN16NT56PFlf3rtcDh5+QKfnT7EE0Jk3ZXG17oTHdeWEt/Fd7scbiQ6oAhsl2r1wrhpIBFF4otVBi2kq55oiSpAtIoxYymritHFhYGEWHX4RW0QBXuNesfd4+Eo/yBjBvpDS3bPheKWyq3K7UuH/ZImzUXRtwI514ed/ditbFua37g2IRWKhwEhfZzKm0vwUosGUQn9Ansvi4PUavmPM7pDFyGtYhpXxn7/wINSb0+F43xyFmfiEeMiL8BUcLzIyN0HtEauy+ShacUTBWEoLNoqaUaGTsAZIaFdeggJ0WU7eSEZ3iy4j/RzqR2J5zc0zKt64MDc06/K4ozO7GFPhO+YUlxxUB123JpjCfF8jWeqCdD7UsUSwMV04VpXlaLU8u1qNrxdHghXr9mW3fdleT9Y3c2k+Nf1hhJ82+kFmFs1GoeZNYYws+clkfKagA19g70qrJIylJ0ABo6NMcN8acMgvVOz3gbTlD+dkVhpvkhswzDpVklsQEkEF0zaXgGjr6uuXL78+fSWawlyPR7aIgLIAwa7yBqwcbuW1SJjNGkXyZB09wMPKs/PqooDFmfw6Kky+dSq8IXhaPC4Enjw/Q4UXHuQvh4Koxk63O2hkP51mulJVtlfX7w+HD9uD++lHsrPXbPev19qq1y/GxIJfZ3YdfRfqqnrnxTeRxpPx7hkfnkb6LtOEQR+A8Qh8rZDxuAA2ha3AUII/OJvJeA0NRd0o6Bn8jBwNZMsIvQPyCF9Xbq4Fe+Pn9+D4cI4gSjcpk0oGhfluNOFQDiSMDMitXQjNPCnzD6godAWceQoKJ+jdPkaN19B13mwUNOe9ybdB7nyVKWXgmUC5dc5cra4kc1pTHVpsK4dzLXkmhSB0ZlC9KbqLDDZmHA893RWHRTqUGLy7fhltLSpFCJt0RCMeZVrQLuQVh65eqZPZq2fPRUPZwIO9oXXDPFQX4c6Yje7DEJFwsxFer9//+slXRCyxZo8DhSdhxFgb6zrZKBs4G4E/lJkTVAF8ks5gDFA9cyzfGl35LYDglonLhaszcL2P8hywpIkqvlBXcutIBxXr4VImr79FU/vfJdeSPfSm05ltBpi2qptFUi8uz6RLWALSEXB0oPBcb7zmocXzTPIBBkDlV4P6oNocbO/I2f3i00+ej6ZYjA5O7smWppC/PgqvCFJkhroPJr/MoCMJst6h2yxpCWlBJ5y7wCQ2NPQqHTm39g9uMQHGs5GQgZYhF+NJ4qbF616w16Xw2J7g+mbLdBCta9BDJY20tod333mnee9ufzlaqEoZ2yx1/M67D5bTgWIX2f+sfysrnlqwPQhvyl6Z+/s7u+OUVQD2NXms35ku0GmLZsVqa3lhtoQYbiNW4wQfXoEl/ruv7ng4PDsdWUrWAmqH3Jine+YwVTshiihdL7cu58nDEudr9/ONSdBQKM34VMRXaquwYR5BHYfQ1JlqjfPrW537vXpXHdnN9EzpMFEA36yIiuj+y9PK7r3f+A/+o9//v//fTp58oVXJfn/bHEdnryqf3NxbrQdvfTcCIYTDN2vnr8ZqWmm5ge3hGg0hifqyKSdLOr7m0e2Opj4XxlToK5MoNKwyIszLTaxe1ggxljfObbgBJlAsRuRCznp1PgIEI8acFBnxa2gEytFXrQ5qypBuhhXVwNf6BkFkIl8Gxyg6W0LIdFuYYbClLuDmQjkbAIUmLVi0N5Uh8DocJjFacIyoymsRvqDzGstMxNMz7OCddcgN/F/+NhOJnV10C2zJM/CF8hT4mceFWyEOl8Y/VvgbkxfyFPjkRnhwCg9u4iLKGMI8DYPsCX/xWsAV6LsYVYEkmxZxZ0yeGO80YBoZFSc6hL8wXWIu5oD3dTFz/CcdsjKtCHJZvEFdzxNkveqp+U3tDxf++PryvHZ5tnU9Xm/N25VZ3AmMWzVCBqbv0moO1zyF2iAsYBwGJglrAaNG+stMzs/PNL+XYMUx4jnGxou6KS0JqoZBl5NO452ZhambRzwQ7ijLrOxswyAvNUi8LmU7NSwrpgBn/ZOTV8fXF/ApLvPigoaJRaIAeOYXrpB/rZbb540j/wJHUG0dT5TJF6hZusI8i+JvZOXizUu5R8FRDDLXu8dG8kX4WczaTqczrDd2q83dm61hrb7Xau31+rfr9Q+H/b2tmo1j+lgX46lslpAtp7j541Hnymzy3lNes6Q3wG9PlrgHpyJmfIRSaosPFayABDLIZsgbh1kYV2R1Bgm4js1wN9+Wi37+kksirKMwhRLdpUi8vI8LuziiMzUQwzC8lu/prjC03Nln4NxoozDbT7DeDMq3hpPb+x5X9mcs0MACe/V13DvAXd4HF8q35VO5c15I3zKJqL2JUW3+lpQTwU9suc0ww7rTHt+AIFJVQzbxCMaqxaRUmiq7h3iATJrU28GHKCN0NgqKp5JAMYzLuLdadV7R86PTp189thVBIoLbfb+cM5jEP22QVzxShtzspF3DdM5/Y09OJty0qbyTS5ImIbkaQ4BJmcrGQoW7mTZGkM/l8Ma3hmd18BuA8N43ORH6C5Ngi9GGIzmL9A3gLi+TBqUrQ+nz7g4bUeorvxkOtyXlOsN/W+3avGwAIyej88RiyuF64s3FXkl94hZsEitWf1OPO5r7uq7Hxb07t+7dp+ed27Li+QsNyKEbVGGB0r4IJe/dJAwT2LNHQ0f1hXtqjcM9EfPFmC0ofHJpUCKXOjzOgvjO7OtbDYlRWIDmIVzCbHXWua+8p3LgR/481jh13KaXp/AwqZtX8s63Dw72P/ygoiH8xXZ7ODw/fW4KW4PhLZ49SRHdPpLUwYG7ItbJ5bU5vnz5cnfv0BiArgyDga5KjTl9vb09PLb3wYLisqV2QOIScXsxsTNgZXd3++C2fSKGrfaaAG53JS1vndsf7LWKiztBE8X3De1BhEANlLsw8a/E8sPdoHFcjHCM0M3G9BFXQIS4K71u7XBvsfzRJ3/20w8PJ+++96v2y5Ugd7PVSQqaTkTEqBUcL+nolb3tX/v3/v0f/bP/+s9+93+4nI9u9zWLsRHE6OzJ48H2YUWJbastpahUP3cxdD2qo30MGhVGQPOycTmea23twUT/NcUeioWWww6xyhR0WxTyOywu6IRnWrZoPZAXSmbNs5xOee+zs96Wiku6Bb4MYfmis01CpV9j+657MrRT8OMrJFG71BFGGGGtDNODHbJU0THZrXBVsQqhA1/D/5Twy9/RJSZ+o01SsdMAmSs8NYRUgnr5KMQXKeddyMYPQkP5M87I2twyMvKbIxw97Ml8y/2++cIdwoUSusr9jDLyqCRrRIIGFd1KpSJnVkzAiN40lYwYdz8/yM/dN6mvgBkSiYzPa2lFkgvJOAML5IyX4UtchmlgWUjQHHEJE/AkYS+aBWppX4+3LkeVhYbLF0sCuDqV7NSorwS0aGzIOW1hw+skgjTUIIidNKmOBkMIClBOZ2dazF1e2rHoQnRH30OqPIyMbzZTQNISQg0SqAwtTKmwXUECpMtvCF1j5UVtIn1JG78r79PKjrh0k2Jt4c5nk9GzI/030vdtvl5gfJYgIRG8LYtVMCiwcmS1CvMr73Nus0xuGnBnRcvx7TebX4ZX+qq85BICiDKUqkE+tVLcyCKQ1Xyr09nfat6pt25V67eqzTvt7t3B9q1us9egycwUqa7T8Efpnh9qVtQOAycRLKSsB+70gn1MudpyKjWF69726HHjFU+6x2MlGUaxxINogWkhqoIQkMBRxulklIJIGkdUo/KaKTjP7o47wdevp+of75MsAsksZJw+OBcXIDHmPLcb9DHtoLLh+mOQgbNbWJtok5tbWE0DKCgY1PT35sht30iznPM+FOldAag7bcbiI2w1Z8qjNiOeCbNZlCJOehWlVRP8V7QamEQ1M5sEOux3jMZQD/+fG/oFoop7GHnqnEAWpjjAvg2xTCgLMMugadqCqEKQr56/6CRPVs/Baq3X1g4agGA4HksGQV8YSi+8fe/uzcvK8rj0vjA7U4/v1GO5fFOKlFkgWDqixxCfqKSQNzeo0P7mY+SWXhNlkK4vpm2cpTWbCaKTAoeN1CQ3rBw/3vZgaFJ0Z9rY8jKdpIAc+cqLln19M6PthnsYba9bHfS3RxfPEacjt03OVxbCRxKXGuGMjx5BBJJJ19VJtbO/fXDN7awE2l90o+oWIOtxnUUqmSGS/qw1vAq6Zq2Qc4SrNwFqiomvGq2OO+fX8MJK5Ygb2dVOG7w/5gVoSCaXgykyBW7lPJBEkQCQdkMXKC1aM2ajn19eNWX0dFsVWwdKVD689eH3vvejP1I5Ma1MZpKSL8fjRq89sQ3UpW0hMKZ494h2MfsXL16MdVwejXgsPNx82bcn5wd7eztPnjRmS80ja2dnpxZhMOxzz6EvfcGAyEjgHZSgoEBnAxZi097ZRMGN/oxNo4UwZh3srVI0CMm9/K7WnetDMilDLbK/uiUPIFIldCEb69a+ELjefj87Wu3e9HeHDyzjxdWFpifSdRX695o7GkqnwuvJi/6H99764S+eX46f/fTj52fHB9UOET2dv3ja+mRw995wX+4r71lSXa20zEwlKGRyfNH1vW7rTnVrZimX8zOqD80tRJpxwBXoD7rR/0JvVoxQiOUTFuePTC6M01d+Ucaey0whZFwMI99j3mQwltsiems8zyo2+Nn55pIyJSxJ9buK4LeK8QYCkn2XO42Ym35F2XcT0kfHqzh1Qoro1eNL8V3YdXQBt8LmwDRI5KG4Q8aLetMVy+VhmplI4fe5JHYblIuVXOStS7+xvcj8sNPcOt77cgHrK+Km/IWOHCHbTH1zxPBlpEYylTsbtoujtZR/XWoIRcPAEjMMr7CXOPMVlHZhwr1WBqsUPnQr6iiryzxM0azJOhwRYi1o0vp+P1nejOrLCzGJdXVWa6401dNxhb8BlGKXi/GT2Nykq+SWqzzvXZ4vx6NTaVZ8znZHm07OEw4TK8Ooo7q4Sj50sgk5VOVC2MOmQC4kBlZBCyEk87YEKQsyU1wiqwO6JoL94aE8A1a8GOuRKRIvjtQOz08shBo0zu1UMW7wJboejNssQeR8WUHPyek3+FXeUnT8G4C7xmrmTaRXGHlZgHJVmIhPec3GLFGORCIHnS6H8w4NXVdThDBdPhpuf7h/62Gvv73esh96fXm5NZtUtsYgRyhovmZ1wv2uZ+sFXqM/otTICEOMw26meISUz9pqWsqXrHtiJsCwGV4QsmBHlEmKn/8Du/igy3kaCVsDJH0VSVCmEr3N/aGQ640/4f9QRfgjZMU5QKwgmF3Ys6EJFsKjwX+k8ZNuY4KE4+kYrgkIMFK4nasSylZawkQcUxfMKCQGMiRhOQymaLfOubc/qG5tSZBYxkUa+OgkqZQhGW3+CYRdjfzjIcFtZeXpya/IEcRkyeN6ko+wKeeEvviGGeA2jUl+RbFxAycTD2tR6wVw9ezJSIRyTjJT0VmwHN3jNfFKO9WW/XhlE9kLdeSn+7sDparrtXKUKUknzFma/pueO4Dn9uH+aD5pjM9zkywO/hY9BFv2VCpJopamEa7jBzfEQBGoQecoldY9Euc61ptoYWkyFY0g92aM1Wzpw/6m3EiLMmg5AKpNQpbzlDPBEJFOt7fboNyCSN/S5Ri0yXT23OVVD5bTJ4skCw5HF7Af5xs/svNEgjNFMARnyD37jj3+/LMeTB72NeRA17UYuNn7tJi2iWJatGw+q0TLOolhLhbQ0kEY0wTiVytZ0IYI9rFjPVrwD8BdFmTMEkym89PT8252UugMtndfHh+ZGo4mBOygWsSiZkAltoBZa2YhmoCN8l/4flFZjkOpd3Z/8MMfvHj2iZr34xdPD3a2rwWtAXEuKU1swD10jszMaclHZ+caRJ9Pp4S7c+AgP1rh1duPHtjWcD3Wh7KRXRpns+HOdmJmeNB6KSug81QXkB1sFM8B/GTDrq9ZbcaruE2zl8WypQeZx3hQ5AREB7A4CzAxUy4BIusqqyNIF503UJEppkhux8bT119/9pPaaPoLb//w7vd/STK3PTODlPpPCqStsilTb2f/9Isne9/5hb/z1qOPf/uf/+xf/cFqnl5lkrhGj3/6yBbq8sssPcEpvavRpreXPlvXTRWKNLbOg85W5WJRE6QedOGmoUJJlVFZwgiJGLgbyVEIJotHJcUXIJGVDzViJaZgMGbo7WsxWNbY3dQRaQ9hAeUwuzaI4OaF8Qb7bcWgcVBLAioCImJ9E94UFhz/lr9kHedjSMn/xSYoDypsGmlJ+CAwkyIUlm2iGXZwzsiiMphKRkhNzHCDkTnCNV1hTTIY//78iNaY+0DkGMubEZOLKo7iGAMZXrf8KmxE9tt1JA5Mxlk2UfQ8pQRHchO0v9FVzN1vi2UEODAtO7rHII2nIaej/dBWjROUEzIkPWud6zlpRiHuiLpc2U1horb8cq0z+lNBpl71ii8a/Tfq15EeVShMtSNJjcW2HH3baK6m/H95sh67r46eP7FVoJ6yaID2Q1wT9XAzNBmhyNwp0NT3jesmOoqjcDBfwp3MuJzyWkbvZGbpqTEqYpukoi7JAGGzl83K+dXi2fzs2LahnleqQAWLeOTLnctL7uipeR9hBQabLzGJ8qysJmEaDcdIy5jCU0pCjGmJyQSKjuBIkkNxLq18Bu2tbu1msCV4Vj+4urnXrD1q9rTOePBg0Ja5qrfD+CReSSuJZSRonw8EDwyKwuJuUR0lbdQ50C75y2Rxahe20d+dStcjv4j9auheN+O23t76MSZpBoBH24eTTcoRe1w8rcpOWQiMRwm1cw7AG3d4EVyEW4TZpWpyUj2ajxsFBYOkQQ4yvzqX86LHi1w7onX7oNO/vdvdGyzmPQIQinb0GnX3i+WrL56/+PLFYW+H2u7R1OsrASc9uRmUMi9EnWLdwnE0hpBgcprvIkAXwk7CuHDJ8uz4AEKw5AuRjLNrL5gu65VrG/mYFd57fjaTBR2HkAmk1fNap6Itjff11bkGLNsKNgWHTaGkUa41MRYzn85lBV9mIx3bnpt1o8a+MwReW4USWUo0rw+eDUTWtdn5+dNPP+tv1R+8+/bJ51+Rtc1uzxYFdh7oDPrGfD4+5yXr7vRv3dw+P3klC1k6Muqf2+FACZNm3KXoswQvCVtylRhVRkVXMDdKsCW1bhQmwjZbkFBmEmPn0qHVJOC7FNk2PUlbJW4synJpyvDOjgUqWWVOban5nkzcykl2NJTS8kw3pr293Z39ruYk9i2WSpv4tIsgBwOoyOwUFEUCQWV1Q7QQYc7sV5iOw1vXF+Pzxx//+Qs7JFxe7g76PFfTkRyPbAhIzNv1QR4TJLWnELdx8IkVU5MzGAdBWlQni7qqFCrs1PQK0pZJRW7zO6gYRQbsY+KQ9/7+3bvbu3uKmIlk+FzWgR28QHm2Oup1lFrWFvNlIsHZDGxCuti5c3byrPv+Az6L5c20wdAfNL/42Y+3331XG+bKybHMn+PT485W/+6t25PRcjG/lFL17OScPHx2ol32dHe4jebQO8S6c3ffn13+ICX4lrww+04O/uE//Ps6+/z4xz9bXY73Dva/8913Hz952rpp2BxA+RTjHd5iElgMQT9ZXZ1OGNGjg/5ycLtThS1Xy9l6zC8xvZgLlEiRmI6OUGevt99q7KHmBEwVEG/Tay+PLz572Gv98e/9k/bNcvf979/M8NdOpTOkQJEfZBqANzt7R18c7w+7H/1b/6uDg7d/+7/5r54en3x0976drS6e/qS6nu/f/6Dev72SIW0r5W7vcrzUlD+tqUj63d3Ktvz5wYWNkk8+Pdyp281yeXm8vpw2VbFQe7RxzbqV3Kek1kcIWo5gF05CDYxXsxSQhBgj9qi0WWI+pbTfmkdaYQvVi1rzolI5jvStD+3gxKsanJfY0tzTsHWrZsOrkYYNiCIKHP9z7eLmRvsXuFRIP4KWolwcCbgRdQ7bE0UuIeg4vSNDYh8XuYiVh6ngXLCt8FRGLbLiQIwwxtTKhg1hai7JpMJmiJf0qEF/4gKVLdtmyK4Iccp4TRHU2g73bAOb+hQzxoc4ieP1iM7vttqHJgigg8O6sWqLEMeMJY6iT4BjmUgU1cKxo5ZpIQWFOWPxhil/ZuKcNg3XFE6O42VbC//uzXZl2VqOq8uxYunqcta8WjSqy2Z3idIVHdG2QJxsxCncOYEzI+7oqaEQkQyXAQ8D+btevdA6giaJiELhxc+tS23h8KBBV4qSVJSHRNixD6ZTDHOnDIowsZBA2bLVLeOQlBENIOv91MLUxpdX+3ZDmdsOZLrd3XFypMq91/ry/Oxfv/zkjIe0Y4eRMT5KqsxXM1v3BeQOwIhCEtPPAuQoBBhUyTcBlnckl+ZzdNdVND2qWmz9PDxFMEYqHqjYw75izA5dCGvy0BZ7zfphp3er2bpTb9xptO+1OgdiV7MFZEl9W6xCsyAqoTXQIV43zo7SgifQXBgBcE+PXkIKfJpjghcvw8OmIXk05uhwMW8N2dLFjM0iZDY5GBXgkGa8ygGz3UjUQc4drR90/VDNGthV2yrhi46sV0G7hspMNorlzZWcXjy7Pxywd9h6k9KLSF51s9fZQjjcQ4109a7vdq/7zX7rwKi3rqryYSrXjfp0/Wi3v3/74OSL54bKasbN45O+XmUvDoLa+Clmod6U95l/8JyEjW5TPJqomX7j1bIbdhwlEdJUBPNK1QRJvnWFJxKpHPe0B045UbbQ6BUrbaOZWb8g/EboCsbZFctqWVeny8J5RIggWqwTG9sbCCx7AVeSL0oeSzQV3qH5YjaSq3/e3O4vx1PI1Ol25DcTS3wGBCAcIfIluVyuBgv2uDK+lYqXqq11Mg+rfsX8bMF7c425LnXOEadtjmBkkD+mGKjkP/GsGPZsqyxzrlcVILLtMkPyG7/1TzESlN4y4ZIEdjVFZW5yfj5iTFMJBbAHwx7yG41tCnQBmYrID4m9eW7UOD8xl2geXKOlB6TXiD0tdnAYH1RtpaNEEqyU52qrnHnE3DdIA9EpnxK8JV0mboRCPYXhcMVjQJfNVjePLPqrESDtLOOWyLpCWwh+xWpczqYgYIdHZ9566y3j1+SEaDdV8YVktQs1UHrEHIJBCnUM5XoxOp+NTrvz7cqgrZNlUtZZq/OZjmYs+u3hPlqtSorSh6m9fXU5G09HtAUVWmWcPPbRNekBas9SdHy5vHPnFv529OrEzg1oX6Pp65vVYNB79/Ytc3n2/PjVy6ca2d+7c/uTn342bPer8Z2XfaBxMnqPkoT0DImuy4mPCTsQOKbLtIF1iWivL9FRnBvRSMgGgTqSJjvb7N7a3mqtT8+f1i8ajz//yWSyePD2R5VbfSKbMr7VG9rsCtfkXbJh32x+1Vsv93/hl/53Dx/86//+t3//n/63f+/tR7PTs/kzezR0D1s9ogCBga9NkuF5ZaEUEgHThpuqBW4f1NMMY/Xs6npcbw3YFNXKSGIj/hCUNJ7ipoBxie15H1lSzBjMx/dR3DeomznmiP6HU0UM8/JUKuPK1kWlchoRLUAb7sLxh6/NaH2xFsi6pEjKgivb6VRG1JBqZfpmlyTP8DhBmU2TrAiQgvvEQ5HBPqT0xVgk+oSAvIShG5kXbxBSTLYyl8jDsE0IXAguM0A95VekIdsn0dyi4ENM8tOzSPwEccp0ab3WLmy/jAPaGH+xzWAthM5/xS7K9bQMDBktG6enQWBfhvegorILm/GGqLBVHm/yOlG1q05VvMwGswvlc5UrW3ufsxU4NuqKt02xuqw3JLKJeuVG9AeinuGXeDEUvFrYpG1icx0lEjo2jy9G89lZ1OzVDCn5CebjMLYMtyyXpcqifSMRWeuZYAAWHhzbHuEEllhwlCzWHRWGVZPdwWkknHV9/rCeim+slFqm5rNRH1XXn41OXlzOrH1YVLDEPwRd6RBQHp0nMF4DqxxvxpP34dJ5zft6E0sPp1I1nukm5S+iT+5NbFat1ezjftDpDTGYFPWu79x0bzXbt5QSdTp7tfp29Wb7Rm8NoohPRiK7uxbwhXHGDJSiWty/kTd44FJ2q+7zuOlU+XKGnXUzmDicY976iIYtu8ERvPnXSGFNAvW5t6EWaRy9Raa0alFdOfUG0wt+LSSKO3HQQloebo1Be+3+znB7r9/uNlM+crV69uRr/aK6u9vNnR02bdduGrzfC/tJ2qmq0+w2qWjpPGbvAd5A9+jKbEyNczIGeCCGtQp9e+/gbDReSIdf3LQ4pAUYMDHSlmd1rRY20jcICo1wcWw8rlXEXV5y1qwyCayK6iObl5qCdbERAdIr9yA+K+lxtYi1zF2JM8b/gkS8gJE8/kBUVD26uAjGVrfF3RfnGnnG406WhHYiBoDMHdzHAUEj3yw3hEMffNSJ5l7bIl7gUETlTuv+vHCo7YFt0Rp6Iou1sYP5iDyPfBJMvYL9Mlbnesdkq7vMQ8MZikgjRlXo9JuD5z0SNwhPqBNyLkOhMRSAyzwyaAIkWUIpOCqtoFKLa/GsM7UFigQ7MMequKkD+kADcov9KZ6KyZtsiExt+DJu6m8f+TFOVSDA58wF7HAGxbrMc7zGdWs9snqOEC9w8YA74ycZZcmj9p1QMyTMGSkMGUVoxJHflvyURDEI28KnPBYL5uMt3yaowIIXhdreHvjl97///fPzU6ltfk031VCUchPtIBynp0GPR1sgyoQixsV4dHN6Wq3vykrQ2ELuG4+p6Lz2Fq++enU+VYBd44afzK5H58vRhT7hl7e69EubD3XM1Mht3pLklUSIGQd1MW8V1NBBs0rFS3aoHJ+PDx8+fPfRW3YUPn51bP+LVsquE/tKsA4VBn3zBzhcVhwkjU5xV4iA0EyxtU21pNj5AhikLGDyVo3xJQ0QKZfYVbv97vvvfbrzu2fPRjvrncdPHr88m0gUfpv07vR5JkmjCp/NtXZuN3RQNomF3u3eqXz03d+wMIvJ53/6hzTT+vX88defA+3du++1h7fKVgfJhuDeSSx6lkh0uzpsD9666Y/1SFLDqFZ3q51NXjmkvA3bjJpjYmYXlWqDpZgmlhSUKSu7YeQ+YNOFaxfeGBWPXsrFpQT07ObmabU2rayn3KHZ8o1D53KmxEuhctmkYcY0qioSxehupuvKWaUqvMmMLmZqLDxtojH5+EVhO6wMR4fw5Znehe/lfHi9g9R4w8+d2yBgrti8L6N0dZExboU0vJK7iqPjeWTRmwKXqpUN94WQfkhhoo1HBfGG2YIJJ2qF32AWzuuFllTtDNKvjKj8FWFrrIkxQKo8ElDZBeE2Bmbg1e56O2CmFdmqQznvfOt6Wq3Mb+ank7X3itIFQ4rUydZCHNQ66gQUygGCe6lPx1ykEdneyw6jAiuTEWU9QCZLso9Zyqggdsi86AJABBb5/687XBkpEyqPcHFZ3hHazkfPzFlqB8FLg+OWg8SWMXyhskVIaJOgHe74avHZs6MTz01kKMJG7102Ob4WTClHWQDr8G8cSllWOzRRF6Ra8i6orgdn0jhFY7WB6PZWbb/TvTcc3h9s78kOqVZ7q+u7tfb+VmvYbsp5btuh6HLFhS8DhABOeNjYiYwSCCRxyAuL5l9EG4g5FrMrDZuvVyJ5mb71MmOQi4aVgUc+ZNQ5gLSQR5lTtLAwPWpWZEaQxxvRZjyalSa/a33tPUXfX726fWfQGHZ4Uwe3Dis7A6p3fTWvLGeH/aS34l03N8lbs4Pz1jZbeCfN90Um0+SiUSH8oCerls2wnCceJ93Euub5ettuVba1ZGlIHSVbOyao4kaqrRnoDiN6GhzkcQLLIvtKwnDsVlIC6bqLacVQQ8bECX9eTErWT0NGiemnCThffkJWQf9yYOJBLGpg0DlGTViJB2GMAIBXenydg8znOLsSiIO/qMgkIv6DoEGjEqhNapHLqDzOrGtEKDl6dnIKJ/r72xQBRZWsD62SSP3I0MSa42yCufpRXrZ759U0JWYKFDUjAphoNEgCmB5pqESJNQcfj3ZYzfDx6xuZbVldg1Hvq09lerdkrzg6z9zNri85flWsCqjo1x/lRDAcgGhIcoJLSFifVm+4RQEKvqhEcr9NhgcfOLXiGwXAMByhKRcUC9irj04Sb16dF+bMOHP+JmlHG4LE5N78yk8Y9N/cx3mDFnSA3KXpdRwzZr2hs8Kmo1mFfynnYk/DdX6LkuOmT7VnSY+y1h63GZJ3hC9hQ1X1dAdjDrXwb1sr6phC2HjyWIfTBfxaTBZjQvG6Nkoz6NqXXz/p9Hevb1qj04ujo/HJ0bna4m6/ObqYdAd9OXWLJLImps77YCIX5+OTUzbuCc7S6w58nC4mdmf66ssv6Rx7+4cP79wbn3/x+c8+Jsv3t4d8kJGkta2NpRO2UmQwxYIMjx+DHBJdMd2QJRbQ522o4AnabDjBF8LAwb4o91DX9rn3bon1HK1O9rqHI3s2Vys/+ukfXcxGH3zvl3t333K35eyCMNAGoHq5aIi33VQmz1/ARu3L/8Z/+B/9vycnVxfnswtpVxf1k+c8qreh/96tUCsqstR0GJ3mJ9DWHbrV3oNey4Za7fXVmX3UyMiartVNRAEB/EGWcFykaWxFJJtGDhwii7Thp+HsyDccKUcsPyk9UHd2Pf9ac0l7MNSb+7UGlVTOmnWZNzSK4J2rjNbXI12xQC1Mqzpy5qYypuPlNmEEGALhRw8ldWKthpbkKAX9gskU5nCLzWMLMocrEG+vh5mR0vTCN0pmVjion24KNiOsiQdSM9leSRwLbVoWZ9yE3ks976YmKNFuZgCHhY6K0IoMDhB8gY9wXMQyRq7GkqAeQBTJkqGFe8e/Hp+2dBomryCfVCYLgvGIoAyAx6Z0YHA9rV1Oq1cTKsfN1USf9GbzWkeYBs0NePi/dTFJ5nHRjKlQOOVV0hQ4jxanR6/ms8l8nPZVUXyj6ad6KSAsCqLFymBfr9Br+s0c/uKBYaK84h7306Iw5CUVnwRhOGa91keO2zZeGnC8LfSiW1ETaAMpOscZ192+mMlI4WwqvxF+Ip4QPnw2GpRVNploQL57c4Bmnvbm48//rc8ow6l7pHmHtkRm+83GkFRq1vcb9Xv9wTs7ew/6g1022XVF9UlzScxVJGfa8bUu04/o5ZwsiULGH1mlkDqoXJKW1uvR8fFGAISRRYokFIG6YhAGIw20qAnlg0ETlFENvcuQCxwjaN0NLwY5hnUpe0peUTThVXW+FjaC9VoLyENs18zhqlXpvrPd3hn2Dvc0JoBHOObcHXTvfPeuxJ50RzEY7nW9frqqGprNkVtjnERBlGfIT4+1V6doJb8g48h4eMYKXvq2Wru7rW3q9ZkUU4oLh1i2WRfNUveQPN1cdyVkxkXjtmWyAQlzIPaxL6En4ZTt7yaXV0QIlhPLrDiUEF9N842YvDcsEYWu0sylmhp/hGvBG0Moqy0nEBDhknJKgCbjIKQachawAcfzn2zeSOpQao68CdQD2+Ax3RW/57tkUNSqHDvtfk8gcXx2ZiP4Zl/Ev7aYzHRNUjxB4Fpgm8Vr6bRAD9Ml1HQfppEbRrUID8shhSr+sXKEu2U188oCM5TY9TGJg6JeRaS7vbaIAR7AQhXxNf7ZdUVPPD+biqxvTdkPcZHhDcsM1Y1VKMnaJUqZwpF8pL4dneJyBWDMIvhTnpmJx8IGPk6k4o33hiwEt+QvxWRJ8VKBrd9R/3MYonvCWwqIN3HH2Wax01R2EJeG9faPm8Q9pilkmteT5DHDICwrXnK39tZco5MUexiDwDPNwa4/zav60akSwpc60lp67l9msag0yIwBqBCJpGwKp/WEjFpLyMz2jIYnTi7H8qGu1NC+aLTGWNxWp0mjfPWKgc3HjCza9a32+EKhUbYnlILH8QSPNLGldbBuLbyFzebE8xXlEDNOR43x5PNPP+NZ7naG+9vbX/7sq9lo3uvuYNfJ2DDyZXbZM12wKpo0T6CKTqUKrMAFH9DN9XyNVTU4NdWySVWjPiSwE8qGFPHVxAuvYnJ2OR+vplcDTYeXWzeN8elTm53ZaOs96aqDg1a1y/i25a9thzGL7V6fEme3464mmreGf/c//j/+7J/8tx//3p+0k0x7dfz8K3kkD6jcBwdheniyTg7ANhNJrlXm68qdvWb/o2Z7bzz6fDqWIrAtyiRHCloVOWT5IlbKUmMzsQiciW1QjqCsVQk/8FX54HqYki2NSC8Qf4Lsb9bb1dpFbWtSqTFb2CHMSgJYL/CRGJdofmkqBE0mV+vzSnUcQre6ZG1BMU/JYMitqNYYMR7nG2wgAyrvQz+hljII//om7uVv/iKDI1ZLsD/CtQw6F5QIMVtD3S3TQ+SOZHMyCAmzqnVZ3Lz3esvQdugwTHOrFjIpzMgd0xMjmoHEJ72ZjTH8y4MDEdBK2K2wTrGZFPJi4dwAAMB5tNBsqbWesX3sqnPDRwAv6GOVVaN70+LM1aGkJOfQlrxLXZdGndg85kvNpXbztU35GjGa+YzjRRtisQzgwtDQMZRSmglGkQWFswVCZl4Yi/cbo86bb45chskDsG8d+AZgFE6ImTRbqv20+N0lgLnWuMbUESY71+ytdru57gyW7f68er3z8MPDQXc0Vrg4Stz1po7Ggi22vTKu10cwKny6jOrNyW++zQprapj908lfeYODWvug174/lFfRIn15mG+3bJfTHurbMNH7RoA1kHbTIgRl2xZtSmtiq9fvWiXTAw4BwM2BPWUPc8ACMPhmnlHqw3ldE7R2OXwAsoDPeJCQdS3oD8UCQcsbH0comNMn/kpYGjsbHGXErGU6dGvg1uo1JVJWqQ8DG7XVe7eGtUHnWgCrkaomRnql12j1m1pJ20Srvu4wBykV2TA8RuHN1u6AqYVVVOgg9nuAW0VCtGodtSHcgmVcZd2seKN2/xc+GN4+uDyfVeerm8mqqtegcOir8/GTI1lAyQzUjyqOG7Ix8jiGO0vd+7goE/nGjiCanSgJ4BhMSXbwPXmAHJIWX2LhhFO8voQAxofn08QytPyhgXgrAnTxKJn3pDMzJ0pDIqmGzyeC/2J5frL5VRCC9YmsDCvNn2OpAUWkjNjebHp+erbLAp61zmtnPC879koyrASvYzRbBqowRCUjteTHtUv9UQR51rccFtHClcHG/C3nX+OeR5f0FtJMilm2NaySUbUq5+ugK3KwnNqoo+y4xKwSiecS8Hvi0f4EHkFVYDZLfPLnjvWhgjb6O6kj/yJobXaOzTDyrrz3UGe8bt4YmDe+2oC0XJWX4liNXHe+XECxhhiv77b5LXzu1jmKXcLfwtkR3A708yMrEt9HebrX3AQ79BEW+6gXD8QXi6WY7O0P8RY1QmSmRxDSQJoGHZ3u4uwiErpZ3+7vzkfnXDCMi7ShefVqaIFXBJqQWDYPPzoZLS5ndx+9qwvnhbymL55IlbG2hi3U64bsTl6qS/sWXi5I8xRU1254dN96593t4eFnn32mXNjAPFk8qKVN9HT29MvHxHa3u8f2nV0sLk5O5EvwxlkUcTouN9v7Ck0v55KuksEtP6syH1fmXMdtmhRbKtldbAF5xVas2q7VO/AQzsVKsFSy2+bTKedGxV7qwv56i5wiQ5lfn3zyJ1pkv/f+L+7deafSaVVWE1vO2GiIIRtfQvNmLl3i6Lz73uGHf+M3zObxn/7k+fNnPeVJ7eaLF58/2LYxlSbarZsFqdOS+EEMZEVGYuOdShNfUGkxifK+VvRsbFFTqzVqk26HtJDojcGiSBd8O2sXGelszr/+o4oX8Wym7HyqN5P3jI+SZK1eq5/kW+7yx1v1ymUH5l7bt3uVpk2JrxkM36seTipo3NSDIHdMv5Dw5iHEJxoOv4wxF/EMwXzla+i0oaxiD5ujEfqmEFu5XWRiSrOEIdF9ZhChHugzgj06FrDpBQLx/wVdPQBNIQ6XU25zee5nlolj+eBs9E7iPGYmMvcZ+RQc9zP2CqGD2ciR4ijhQc1fXdKA7K7LBc2ksTy9kfF2OSeJiWQ6Zad+3WItNWIz+XkEOy2gLl08Glv1RvNUXQ/lKwpzSVxczsRX8L4S9CENcFTDDZXFsAEoGpSJ+ntzYBPeIsg3J37+b3wOJY23cArT2IgkZMlF1esPtlm+8iWRolNyg+bno6vJlBkVZyYY9HrLZvvV1fr4srL76P17h3ujl8OXx19XRiLcM/YD2ZageAZUjjAl5G883hR4vn7NP1lUTOz2dlKU+JPvtNoPOtuP+r2Hrd5Btbpbq27zOYsFS6elwKwIdkoR06bJzZC1FVYIscc0sAT0J+wp8pZRJ0nDH6vCFuJl5zjAiOwM17fOLL9cn3UvB1wsXAw8YHbCeEE2UHZtiQB4I6xLvgQKuLZsbJHaZuOmeaUMr73d6ovpdju1drPabbR29GVo61Jve1vmg8gFF1pnSzPZHqNQ7grcBhTOUDnHie9aTANqSs/1j2SbFKsUVVSoyKVqHV1jKCQo/00xg4npe5JZ+3TcymxVmeIn11dHJ8/rLIwXSZckQOAKJRmGsoviqnXzxCoi92QIkyoilrKsaOk3c9FFhoqQmVWU4CQYGiAEQomcsj8AM1a7ipViVka+FndShhRUjAvEtdYiPsF4VKMj0y14CgQOEZzuru6eBaAFSV0uFqHwRlo6m5S0fxb55dXxq1fFGVbT0NaFBIacNQ5tjeD8OjhbMDu13J22GzM/CyOIQlwAFFXc8uHVJpt5x4VduJtXJCahzS3iwvOfG0psaymA2u53bi6329LGTNho+fdUzXY7ssPcF+Pww/iInI3rLElVZm7FDChklR9Jt9H5D2A26I5KN0ifU6iSsuEr0tr6bt7P59OIKR5/bjap7vG1Y/P8CNGOA9mNyyOpbLl/4ofFViYyuV8CD0uJJuT6xz+y0U5BMZGrzTCoQwM5bMPeSAVXaUfV6bbeee8dStXzl89ZoOYL/4j/+hY7eFt+zul8oixKS4xXj5eTc/R/OTk75YuzJJ6m16FSMVUEZ2fKqOYX06XdeJVEn55dHOzsdzTQHnTNr03iqH68WW72gcHt5HMLZs2u9H5d7e0e3L19dz6VfzjnD3dOLRff7fnpsX1Ovvfd/cPdvdHJbDk5ZXxssepC6iUYYZULUGHxWlI+D/74QrF/pdJLBVWlpTeJHAaQJvUva7NmrRdlVAYWKeguza3pciZ9S2LnzEZA6Zx8mZ4WNzcX48XlF1Tl6/dXl3uHDyu1zlZT75RGeinQO6KwtSQ+nP7k8d7d++/+rV3k8vHk94WiZ9eT9fTmxbPPeruH7f5h+BEk2aJYdziheLG2Lq62+pJn39qtr5fXj6+unlvG9K2EDtSkrFTiEVnqOH2lVxSBGCyy7IXSCuX4COOcsF6RPpJ71zNK5FZFiHdGtbjU/ao6YK2IwqzmTH9iADuWsRXPZrHa0iRzw/XcCoYEwwq9eHgk6wZ148PMJHJBuQ5658hPXOZCvsPcqXBxb978xSvQKu0jcgalFPvGfLxxplBQHuh9plEm4hHEbSgyhm+8kkVKRXtn9US8ojeDBp3yODfCJuG6HkA8eso36TKCl02BXjsg8EkKakdi8JhgbOOGJn5rolevEaIXO1V177eklf6shtvpJqiHHywnlcn05MUTHhQ1CLL30RoARgWJuzAepkw+tEwrILBBhPMzZ6LElK8LmPISUfXmyEK/ObAsnDGqhNtaF3kSsh1ash0ajApUaStOZfp8XuG3o7ltzrpKjStbCBUOn1xffnxy/okmlzu91s7tBwbfH5y+enZ5/EKKhxz6ytUssgKACvMpC7pZtjcj+Iv/1j+s2cSyctBsvb29+8HO4aNWd+fqpq3Wb7qS80xxpseFMRNEnba62NL5PZVcSXAUdIRjax1Krs5OpoAkdOUVa6VlsZywADpzZGgiCTilcbHO4klWeBsczkgLpwtOx76KWC2COTzQTIoALrZvlktcqaFX/bbM1y5PqSqySW3c3GlTXqSncDKKYHDhyAcioaMB17NDlYdjDRDI/TE5PitOaFzyCtYkFpn0TTFFJb7IUYsVCEgzs9ZWmqDGesOaSdCsV9afI4fxQTPfYvbL8EwMgGq7ZQOXWTNbHQvxqCCQ7K+ytcwgEUOQZEKSSDSa5PQqf7uc20fVuNwYoyQHgxeeRWBF6mBZycuFahGT2ZqmwZVOhhtSkIxCAkZF5oQYACu1w/IoCPviWWCHp94oQdnN4baoKdq9H5fLAMtT2MR4C+t5fHrK6a1cFZ5M6uNIWQpPu2X65BRFNwKoVuEv7Xb7DNCL89Oik0Z5Ltq6f/0iNvHmiU7CC68gZ1Ty6Uvmt3QpW47ihEt1KatWrbVPTsmoAmqGMNaOJ9YZw6gQPI0wVBdGkLuDihtaFP5anionjTlMJIHt5D559GYMvnJ473o38d6vyh3yvmgh14p5ioAB1Nxzc6U35LQrHa70W+fxIxUyHs8m1Lc8J+L4t6uSjLkML/vdhE1CeBySiXUj51mzz8PD/afPn55enHpWq9PY2dnWlvnx4682UArdlIbVZHNPmt/Jq+5wcHD78OLkpQasyAW9M7en9TafhRZxN/JpKEw39q6Xin5JZfcIOVx6jSjQv3P7cGe3fzY+5apPqfPlhCzUFDxpQ/ZXaHTtCowib9++bY7Pnz6hzOhtxRzfGQ6CFdeXWp+pth6KiB9sffX4aSLZ4X2gCpIxhziPnAmialc3G9en7WJ78xxToHkhCT8aKZRfyE2RYUT5axA9lCr1Q7zQ5F7bZsxTmrwcjNHijH683d+7up4+/uqnVKT3FqvDd79LaRdGFa6K2ZVsjPAIrrrF8bw93H70D/8RLvCv/rt/Zjv0t+8//Prxp2/pBtnqx8kW+QTXEIrMhe7sfNLDOw7u1vv12uT6fDLlwr/pCDlGkJhUBp3EF4/zq9d+lKw7flU4vpM5TL8w+ogoH2F01MgR4XpTi7RRXKQLdK3Wt1PT5RKZwR+0GxIM4CBQbi9VpsAx3K4IHm9yoA+YFtH45nB1eGPBqOChI3Ims4pMyRiKTKb6xVL186AdDwt9yKUoOhI3AZ/ckdDauJ2xrM1QIsnVeKf+Ito5BkfxB4SIYSn5nlTMc0DKz4PWGQtVlzrHqk/SbUWp4oonuVm95GquruDaTJTkhgBW7nez3Oqtu1u+kp7LvmBFiZip81c7mdrzdYvNIQgoEWB8cvb08Ysnj2+kdMgbcMBRz4sRAmzsa+RWPGpMFMyZIY8XA68puujNYWJoKtP9646NVAZG84h7K3tBMb5ZvLrZzNmNaORsNLL/GxBDckm97eutVlffb1tz1QDleDz/9NWrT6ej2Xyw8/5be4cP1ODJG3pK8MyflSXhgYCnWTvsIW4f3CCLFfgFjDFEf37U/+Pvfk9a8najfX97d1sDw7OL5lyCkf0tw7UvWWq1tfQq9bxzoql21VOOZKH5G5RwXy7VTzMjpJmIoyVRzFNEqlXbmmVYmXYw8R7ADEwLdomKxFPOa6OU1ebA9C5ZQrGrQRKnvRY18C3OqGI1doYbNau9wUC2lOVjrLSkVh3soFf4NrqaNjq7W3rmDXdq7W5Vitf4fKbqbra0F6t0qnanA0WTE5kiuXBMwriVgoRhMDloGkZssHi9hkMBTNFfJC2EPiPGDJf1xZ1n5++CGFacRGQ5wQO+NHRO5UUE3Wb/7v7DSnW32f34D/7k9MnprX53PU+HBNvLMTK4eEEEzREx+raAGHhKBrG5gnJTRlQSDdGRNqizS5+FPPlePJJ0LuoXC6BC3Y/lgRuxxiXom0W82SntjdUep46eY0B6060JDjZryjiFJ5PklS216U8Aqt1SjPLU2ChES6U5JYrYhjM4KR2RZsrTj+9f1GtgaN8lU5D6ah86AYvJZM6IFiZ5+TLCmzudgIt4W1fJJ0CzcG5m5JvxJ5AOBT2sZAWjEPQeNzRJeW1Hu4mkt5cvr+U97O0MpQXNq8vBzlCvF4LBfY/PLyQ8EoA80NQpBcHuxmgGIkBAQfEeRFO+lsqktBwybp5rWY1qI0eLfkdCx+vO6s2KyzZmlglUzadJYoON8WnZRJZATYQjwtioKQWuEYKFDKlwzZ04eMLRSv5UGoxY0csY1gWdrIUimDib/RQ1S3GiXBvbi6NnLuxpIslvZ5lsfa2Xf6///jvvAy7X7MOHD/fFdS+Xzx5/cfuWHUyGL5+tAdO3J89evdQfSo5cr0/Utvo7q5svtcM9vHPQ7OiPsto/GIr9vvfew0PbA9vAYLEWReZsNXbR2cWyNtOc8aoqWfrA1n6VytuPHioZ/+qzT0jr3b3h2ZmFIO2D2i+fv+D+gGbf+eiDSUo/JhwDMsWMejEBWrwErCTb2NhwdvLyxbK66lV229XtynY86TEopV5ZJeuy1LGrm+qNsdYuGOp6/9Y+V9+Tr4/DQut6+OHfkgd1yhx3E2VrnJ89/+Pzi1+8ubr94L3K7m2ah/4ftVZf+ZiY3KB9eH5yvhqfDu/d2v2bf/Nvd1p//vu/8/TJ8wd7t55+8Vl9Wdm7/x6JfzVb1Lf31o3mal5tDQeKXSqnF5VhqzZ8d6+xfnVsB5hxZ7iNrBGYDLOmCtW6fA4V3mGRlpj+sGGb4fYWO1wg3BQppVUrLMEilHoHkcm9CLAiMUoaBhN23cT/eMrwjuS5RrGPMq1FJm0dE44PiP9D2iuswUvStxyvDnNxv8j68Eb5UMRNYm4xCHK8kcfeiJQzua7tfpWgF0QXrxKyjGs9BXUaNRIGnsYZx/yMBgmlQ6puF7EmzoqlTTgrolomUJUsiyInKtiTO7hpMR3I2xTIsHSveFywfyy10uXwr1xuXcmrmqXEaHF2NT1T5FJVSlQnaNlic8SxJTYQh6BCZLxUdZkiz/2DPIUvRPL9519Pxkfj0UlqCPUV5zcJ+Ekw7MjrZtZFiFkXfvSoA0YJglbIfzTuApjCxr1DWXEllMO1UPr1+zy+ZMontQ2xJiPVysHtJE9cXovI6MOblSH38VZqlDqApIFCkutVqzq/ufyzLz/9408/rdy5M5HO8Pxla95r9loP7r97sL3/5Wc/O/3ZH0sCMLQsJQYa0YYf0jaF0kTFwwYTwtjoUWVY9V/p2nEFyazaJ8cEoMY6ndZWv0s/HaNF7eZuMKOkhEcWWUOxZm3nFiofrvjm2b5JMqILib/AmqhJxXdSwv2Q2a7F0kKIHTu2S+JmOrH5JUmI6PB9lTRRfbal/NjuGEQ6jYur2e7+we1bdzFBHJx8bHTECKqtflvBJsVJLp7O/ZNGWi7UdKXtidJ2at1tFVVwU28ggUxlrcSxmK7e05Q7lEEEk8EaXViQotDBX6KX6hTtCbjQiUWP7sRTmhxm7yKbMx/vc01iOHHg0iZCFuWtuhStNPEkso/6fXvv3ve/z1dwcPfwiz/+s+cff3r8/EmvWrPXecwF1qOYWfCevbxi6mP6MtDVYpvsRvQiEmjmVfKnV4MksKFdTCrgJPbjKk8sxE1cGFM3CwsZIz7B3v+xsmNuSykkUaipiDr1M0UVISbXgiuw0USKIxEDouxpmCclTD3i2iYNKu7ied6qY/0LUZDdlIKS7S4gDzReVoaCpxwc3Frxfq7jLPJz/JFqgMh9JIOcKRoL49nKhKHA7A1JADXRazz+wVmAOLU+uvOjdGlwsRN7e7u3+tt7XOT8Bl99+TXR3hI2Ll2xhEt3NCtuKw7Ga6kUlqfQjSRN0xJBoJ1Iyy0HLPKvezpjGBknnvvaSo49F9aaEsB86zVLHhLJq4+xbqGCnE1zdKOgwgYn4jDx6Ewi58qv0ELUU5zQD1PCaMonx0dgr+IoGN64Ob04++STj3/jN34Dj9RjpNe1n3F6VOIBL18+/7V/+3/R3d/5yZ/98eHOgF73/OsvTs9HkffSwGtymKdnE2Zla9ho7x/u6FX51rv3dOvc0fH14sjyiCcJNEqElNI3mcjD0uujOhOXZavijPWOkASr0AN1s7t1sH//wV0zNnXSl1an0NjSHB+/EvcId6pVdIrGmNnHJCrRsEWseMB8dbxKUTWcOOimkFqi1+LsqKoZQO/+jZ16w8djDwOB4I4NB9NNi1Bq1rUc78u30G9ffRQ6Ar3AmlKmIFI+YMApsvHZ5z/SH+kR3N2+1avVZlcqfK76jf7F0cX+4T2W8emTp9vD1vav/+pvDHu/+8/+2deff/3OrQdHz58y+2//0t+st0TTT6vV4bUOxrO1Yg9pQekF0OhU2o9u3W+dH/9E+gyCarYo2uvZ+EKWqf4z68o0KxmUjKVQyMvahqmGb5BsrzmAr8rYwyugMPYaIR0qS5iKTwx5hdmp/0G0QOyuvk+6hbnwyVNAtbbgn4+sibBxf1RGT93EZUk+CAjDiiy2CB6OOqPZuBSai6FiJvAZVy6D8aIc0D0CyAyn6BCRRmR4kmiIFz6xQJusjXiQJUVbEojCCOJfwHD8rhiIVoTyutVxs8j4tbZoV+vp6nb/dkUH9EX2k7ya1+SPXk+3uJrPzxaqjNaLbu1SGFirjbKUl1f8pGGYesNsNYYatgsNILZ+a/306/HI5lsvJqOTla3erhdXdlBmi1v+kHNMSKPeUFZGFfYF/ibgKEZkwIFHh1Q3l+WaDZg8NIpULv3mwBjdAQ06D2TF5xQB7zBz7ly8IP/6C0HQptFKabZK1WzbA6L+anz+7OLo7GpcmbVWW/0thbQk3/Vl3W42HQ1S30awR89+TF20UVgUrE39qh097c9lRGEc/i8SZ8NCjKc/G/cMIrZEouv2ll7V1sfVydUwcTW1AJbBkiSoSc1KVoFsTDupzXVy1xcbw48HVzsarjnXRlKDYaw8qJQcMiQlya5mn7RU7YuwUg0T0+7Utu/sPHr0oD7sz05Pv/rqC0S+bF5Hmt7u7r1/t++8Tpbqsy8vn754ut2o7+qLszvg2BrNpopGCWbpSetep9qUZNGlDGrwwirkxLfhMoMPYKs036aqwDTkitIGmaJHFJEUegGLOI8d8cE6ov/kCzgTjddaQF9HsCKOX+yLqe6HZEvJl086GKIw8+qgO5TyCszS13/pow93B1D87PilpGgqQGmDhYXL4lPoxm6rKM1sd5Qc8T/n6dGv4VUhlAhQstNwgFR4Fl5kFVLbq0gHj0IQRlq8NBmfgSf7KkLVT6xS2IXnUmGk9iMjd48244d61WMAs5mSGyc3wpKMkQXEQQDUECpVvuPR3v6Oh66mq8n5aLY37d661VXPuBBouKb2cHsISJNqfruK9yaFOsbkDlRdfzisj3HcF+YSiJJ1YAyXQkaRmBQF5Jb4mP+pBXMWIczwBBQr1U2GXPutd98i03hNR6/OuRzRD2gyKLlwLY6hmiwGaKkKg+I6cnkWyashebqJG0RivRAarADWBQXcCCz+c8QHcdGn7+synA01Kna4bRHAMTW4FvwUfELueHF+lQdR2IxZp278Krd1xr/M6xgkBuHrs7OzBKcbHF4qzRqxqLZq56MLOUeZ6zUFfGSPB2v36aef/lq9/vCHvygua6eNg/2ds9OjyauXLBw/Eqc6H19QgIb9Tn/Y+Gj37dPR+N79/X63d7DTevn1p1gZvwlG1h8MO53my+ORLXWL6pAkbQm4/V6dyTMewb7aejbi4r9z6xBOyphCbhcXl0xS3hPtQZoNiejN49MTelh9pL1JIvr4gftDvNn8sj3oyGGww3HzRClBo3ulLW2vtb2biG/4WOilgJukATm4FwywR+lgu39we1eYOPfkrLZgWSloLM0zfMPeinKgZ89mnAoQ5tF71a3uDrlvBHwTLaUOihjguNYDlo8a/uitX/t7f/+fH/2Xz4+Oh1u2YGyPvvzJ4Nbt9mC4kICLjelwE0N1S9sHlkhlb1DZbm/3pIt9LQQkcdOk5GOrKtMVzkJDGyuIxkkBg4u/Bi5i/Dh++H2oKRcUxhAMeC0vkADEwByUZ3JYqiFQZdGv+EsTDNf7WnsHgQqh7yn2COmCtuDj6ZFTbhm0osGVhwWjLZ+vCi6HnApRuVPM5Bs9jZE9fpqQMUuDnh7kNzyHcSUaCNELczIXv0eZOIy3IUE9CK5ngrHp0ecrOnAyXj06C9XrbnM5SZ/Hs9JkReE4T4U43rJbGV9JO11cXC7IzXljPSttRZZtLa54mytiDSuaASxIUlw8C+iesVTCpRUbA756tnq6mPB0jc5Oz17xRUhcSSpOfHKKhU0WbwgHA5rXotdksyqGlpyaAvK8mlqWqiyE6WF9DqsUIKadlN8H+TZvcj8/4WYM6/SSI+ZyubP2BWkllj+gdVW4v4HTZmwJtNVvc+I8efn0yfHTSeXCFsVXVQn84j+L1qyj7qyEmaRtyJa4PDvuCmXLyFOjLJWh2BdY/oZ1GINRZqBlsJW6TEfeV65dodmU5bSFIqvioeonTIPWZKGuadpqn3GRVKuoTyWOpLuFoYR4MFEExJsRx2yL8p9EdIqaeXKQ8DbbP47pxu1ZWY+u5jaxs6CDg73Og5369x5VHtztjkZv32rxAGR/u0FbVf/VdqOypyWtZeucH7/66icvq9Ojd5pvP9Jo3Rb0W3JhElNgmnOz2JIHKfJPyzHtdPtXl/1VfcG5pWsSZ6Q0T7LJGNtRVJlH7aipseizUpEAZQmSb2CupEV4gRS9wqOztkGD19Ii6LwhzQ31AWO5lcIB9CMJpS356KqhPL/fqbx9d3BnTz316kIOzSub0EpZxlg5N9hxwmU9YV1dYiSYIEv3ibDiqTGejUpQhiX7yaAwAlBNKlAlm2DIJKJBa8ArjI1ygq8QDcYaW/zmYB/ESnG20LIOw7EIqbsWNPWgaUCowjmtOeAZEaCkHZ07b1Rz0RuedTm7FyMGAayQaDPWx3h7mzysSdqZz9NDPrGB+BR4IeBxvNyv2VZAkseVQttC7BSXnHR9IYHiUQuFeZOvcKFAo9sRrRQMtWL6H43g3FrA4vqd9z+6f//+w4f3PYLfL0U6lXU/favbBDYVzavfknYYThazUNY3gzGvzaMz2sI3vdkcPsaPGB2nkGw5S8Bvltb4NodbAc0GPrkEyyyH2eSGsnTD29Ap4YqjJgHAE40HlesUIKXeyobJRvvgNqwd3tr/7nc/gu3oia5iXlqg3D447DRbXx8fPf7ZTx79/b935+7d0fnxsHv36MXzjy/Gnq+AozcYygq9ff/B7bfv2TajvbdXffI19yl7seAVzWLOxW2E4gjKuhIOwHFDh4ZQs6ulhpviCJQk51++shX0vt2Q6N7GrA6K0wsMKXnjiWYyVIX6sxfP7VYoOmGcXC78NWSweembLw5AsZEvOB3Pqi+PlC+pFUg8L97QSJNwCTp9Gp6lSjgKjsW2hK3G9t6ORAudz9JmGruJtxFq8IXI5GeNXEtl4Vo6Onl8/TNZWVf3H33Q7u+HqSzHvdat2cWJK4Y7A0VkF189HvI0fPS93/zHyx/9D787O724mE/PHn96cDV7+MEHohhSQnl8ozTbdEhU/Ppaq8D4UIcftcZb4yWXmSaYVx1WgQzN6Vmrb1HLSpEEG24ZThF0jbVVjo2opGmbYdgrpGBKRca7VExqRfnFJ/Q7qTR6lcYgjuKQADSQzYP32CMF4dKQU5YIj1J9mQLbooeUsp8iyPMw1nOM35Azmg5yk6JO5Dv0fqW1SHpmwTmjzQ1YSso8Cg0aM4qIIw/+FI1cahFXBaI3GBpSDHckE80/Y2egxG+RwSSmm47mNme/Yv4N1AkmYu0Xjyc3LMBxdR7Ps+bdNiLArzlc+oqL7I9A16KQ6pvEuYD5SIUJTKzCZFSxPe/Ri6+//tq+3UhPOguvV5QSJGUYilDsbJ1NtALU8F3TCFWD/188AjGCON4+FFzmGpo0CSTp0gAnNy3qfsD6+vDW1Ru6xuqdjbIcFcVgBKeCpFSw6OVQFr9koniK3sRiGZXrZ6cvTlbHUc21RcczVW3pIzaX6Etv0IPcrgjdu/c/aHcGL+qdyfHzyvw87dJqbcBYC49nQp6P238zIaZXr20rUeqTvrqi66Kj6cpxXdeAgs5ZBbjl5Y1tZxaCthGqTC+KASoNOsaap37FC60xh9W5VDqgSTlytW78O72m7Gz7hfZ3ujv3DoVHZufHutW0ht3+3YPLPn38rMULAiPvdA937+tFpUp7LH9up73upvtoxfZp+qvfPTw6OpKOrWduAp+Vno1rDENWFImikQj+B39kLdulrtOW2aOzxAvMWvm3/Ga2qrWSeschFxkcWilpGjFGHMGrYGHKhIhheGdecUlnFYjEkF+QO4vlxa/SbVPqeZY6epMsPkdyGK63+v3x5KK/uuL4ez45nlzNd3raEfV0SpTsRntoKTOOL3nTpU+lQpADTiSfh1AsYsnDLFE0VYaE1M88lEoZnwY2T7uT3qYBZnJa6LFI0tOxA8SGnlBljEs68XXFHuYiCGEkVp/3iQFMwyfUW/i+iCbjOKumREA1dWn3ncBkao4Xp8cn7f6qM+xLAzhnErVbu/fuwm/FeYKOEnv5m1XSMMic3OC0OUGLjbylXocSgDB8IQgXAvEv6vfIhJxckOmlWKZpP8GONtxuTlV0Xlft1dHRZH7ZktTb7z96+yF+qF3Xy5fcuWsVwGxvVSVmQdILisYdmkkncyom90aXykNDxZsREqIOi7h59RVlEwsMEoQPxawQFLD4IU66NsNQLHS1lGEVM9qd8GbsJRMIDW2oXaZGBL9k6k1IhYkdVrCpLr7U3tmDXIm20Y/3LrP/gZq1e/fvnHda8eHzKt/c2A3weDoZn58FZDv9xmIqPtzZ3tWu1y0gCtFmmjs77P+ty8b17n7zbOSbZI6o27BlNn1cX9ytDjfyjZAtbVhJkx5tfN1MI1OZy68XzHZ//GB0ChgENrSjAk5mY3jRZVm3uuPpK0GIeqNHTVtcnDJVJQBau9A+uGBX6+sXL2XaiIBedW2g2GopPpC8un75Yt09yDVqHGgeKXshABgSMbktXNbjaiV8j08GKXlrQA+onU82oT9GHbzm9avzrp6efC3epQPD229/t793m/Jof1+6F2gvF2MDoxvRLern094Pf/039u/8wX/3W1998dWd27fn8/GTL3724N3vas5mAx3LFXFVa3uC/Qm3pkSMDWUr2/WtyeXnkv5aA44iOeZz6m/QwV9BnOCwD26wMeILBudc/vIB0UUnN12rnBlCk7D8Yj4xfHlc2+neYwUjCYtXIJ2Gm0rY0B2lKY5OCSaRF+EBWEjsMJHTojZ4OD4XJTxIHcdXccXlUSEoow1mkddGkWG6Mn21Qm/Y1GaMZbQZJSUKBrDMeN1yn/APHvTMAze3JtiBDHZO+Zb2IY1ar1ljwQtqtivnN6uT0ezV7Obkujar2Tb6eslbwsBSspnUqmS/YzBczJLX9T1lMHgg9Ws2lThwdPQSV6T0U+yXq6ncFBXwyTWBG4mxMRYQbdodeg1iBJxgbGAFtHnjlE8++ifghoeZZZlilODydYAYYzYcNbfNvV+/z2dTzZ1pMpwtm7VE9Bmqx0bo5pFxOGwGUVimHH1xz+poPn519nKm5Kgqf0/JmeEleY0KrY/SSDH9Sr8dPqDDg7Y84Z1Xvd1XL76ojI60SEtAETkEZ2hLZVJZGweuG++jRGEYz8bSzkG0SshosRxPUjwgh5mxyVeEgg09WSapK6VVklXmoUIXy/Ud1Z67gjpLi7KRr/AwXZdjzU4njM723Z3hu3fYX+tJv7T1H/T3h3xfYy0GVudW/rJr68z6uslRYVdPW8QPiAGqOo9ardt564MPd24d7h7sS3dmdwO+EBvzK7ZJTLFkvoYSguRkE909Dsw0UtItstaUvYOqZF4qE8YXEur1k+iBxu0tsrZuUNkMi28UJSjxzpKX3HxLF77g/+I1ypoXVSsrGTZMYXKpJWWALC5H1xVJT9X19Pzl6Ox4dt5sqIACBLk5zWz3t0VfKBH6sgkBmoAnNBu80AMjnWKpBmmQpCXioszz3JFxlswXHn8p5AojWylxSOe4ohIn4pMDClFuUSpaxMr4oKNCGGPheS6QfKT3CKs355P+m7qgiFWpvRKbZGTBLH1ddO2xJYetHVSU3awnvT7Ukh4NY2iLtnfutHvT8xFGTxgSDPAvCFUkn6cEgx3lNfPZkEHYGlhGVpJyidJIRecChFSe6L7JpUo+F8+izSBmp0c//fjHd+89aDVbEomP2GVSwitt30oYXiwNWXTPkGPLCjiYPIVMsHQzDK+e+81hOB7tKCPL+8AkWZM54zL/ZPrZxPfapF3A4oCE7lOOjX6dEtcSH+DayxmHXzGkjc2fhxTCILORByWjQERSLKUVmd1cn5ycfPLJJ6bz9ttvn/YHNiF+OZ5DVeHggwNuHxuPz42G08hoxNu3Or3LuS1ratweUFw6yGh0plFOo31zcGuADXEzv3p6Kirv5sqLbeWLfInSW7fu/PSnHwurd9u7qrkINTk9IEIHt+qz+RiSK0SezEaMWhGl4e72g/v2SjqYza8//expU66FHZPGE+JAr0qNK2ymbGr8MFQBWcNj7v+rE9kFdx4d7O3s0pZlNjW7u2riJDwaDOVQjix/SsyqNjZHPthJZG7ZLLFbKHaPb7oYNGgSPosIEKkWZLK86MmUrKzOTp5J5ZY9+MH73+8ePkywHFyVY6k06vRpTFfLq9F4Oqx3K2+99av/+B/X/tX/+Oyzz+27fP/w4Lj6xcGjH8hxh3Qus0aVlq0Pt9bnJMZ19c6d2rChrdHFYszr0u4N2q390uE5nL0csNW/iJwwNH7CtaAK4ipWMpxx50JzwR8XhHObSY4NPw+zQtHEIj2KAhyOIjGF3sttFq0vT8jF0ChGXXBQ2QFEzhssKq1tOUwTqii+MRe5Js8CYWPb4C1Edo/ijLREAap70guwJ6tQUx2CJyi9lroTCxhLcYFnJxrGqCk2NNSlkbU4LW2zGNF7ZWeLZmWiHGK+GF0lTepk0b8cFJMXp0qoW9VvOCWlAY+RyUCJV4MdTWNxPblYzmQmnGSzbV1XZyOIJxHVpc1WX46WFUS0mFeYWNhbUoKgcSAfLA2+mKuZBcw5ESVu8w7l+T6fAvhy8Ru63nyMDh0A5VvEm9e8NVtSImqS//IIUFCoUlQer3QhAsHicGXRicAwDaharQmPEYtkfKJYCi/VoonTK7XjReiwJiQORFm3l2hDp+PhwZ1BpzdgJ7x83pqfPq0sz1IsU5CJJhdp4dnlgL+Igwea4r/FzBVfXc7HloqYEhKPk9xDeJuoYdJEQx/h75EYZDBTSraVuleKOJEnHXt3aAcZHURW2sXJgWowzJUIDXfu3q7v97e6jdv3+mv2qm5T2VyqG9dUMfi5MvgSpYIOGtutbk/qt8WiXtjVCwfb3j8Y7O5Fqcfok6AedCWGkIUBynTJsmyiGFtwiygV/VjYOmZ+PU4TafQS0aB2glmtwEaZb4jKvPG1rMJmISMQgtl5CWe2AlC0pC8lDkPiR2zDg2IVexNAusjjQwq4GoSsrJSZcHPx220f7rb7nclodn128sB+f9gpFxw1hrHFnjHiiBwVS0EIvBk1WBRYEkraYI3HxXKCPEqx/CxKsLEEWaNXp1Ff0BBMSrIuhDObWDSwdvMUraq1jSAdSv+ITf4RgZHc2uTvJsmWWSAqTNIQwARY0k7XN+pI3ZmJaRuQ5k1HweX0YqSzmOCxShJlo8aJFWKmOdhYhBK0gPfYc0CTA2HnfaGBzIhmmqSr4A4ImDcfJAoAQrfnfzYGLmgqipIaFv5kPLl89crk93b2jZx0scwe5imIw2P5ovlguJ+poni6tD53A9ZoKfxToT8LD54Jl8QPF5XVX/AncLZosYlfjzCqAEt6Kw0j9ZkIjhXoepwDTpDO3mTiJY0rKayuL0c4YGEBXoPUeFoQKzLesfkq0p31NZ0KaZuybZxs0DdiXNp4YTW3ELYzYjhffPU5RZP6qpX5zsHhcO/g/NmznSSw1IhnLGxxPu+3hzwi+7eGl2NKSevs+EjfcGqeVCsrqI/Bwa2777z94aeffI3R9Ad7R+NX7Xg/uG6yQT21/fT0mB7nuRcjAtiO5w037w36dx/cOz2fPXl+mkp1VRDXq3a9Q+H19GAXvDJ0wGfa1W4m48rXX5/t3356eHdPLliRjrh/8R/QYiGEKrK0WnR/PirwtpthtvbyOAkGMpDX9kHSEBfSRt8v7YYjhdI8xoZ0SV+wte/49PFnH+u1dWe52n37vekSv5H4sh8smwpT2G2lPxqN67N594P3fjjszf4f//n5l19d243x8kW/d7e9c6h/OwUYakUXn9sGe968khoJPXu11p1B7WyePSauqLXUlA3GxlIxTwRo0lG4MWRqMmSEYkHoYDiSpUrGuMzpaPAUoqAC5uDbOFb8KwpADzYtecSyf6o3fHkItCSm5BF8bVAtiXFi7FzlVVkz3MgYq3D31ZSjOLubsHokiIRNb55nBGnZw939RjCFF3h6Ib/CCQLvaIFEXKxtScnX2pTIYWWCmAjyJQSZPilcdmMf8XQh+GgI8Je8Pl2NjxbLkW3aJeNo/r1dX3UNvATNiCs7bphjCE1xDA5E+aysxpfcfdKbJylUnx5N9Nils8air2/ZUFWCznxEsMe3GMacJh5mxJxCT5kdAgNBr0BXYBtSAmCfULB/vStfAjLwv5ajuSbnXx8h/XKRV1PdHC4ADTwgPKdc7zVvsMqw9bDyLCMKjl1GZcYA6s12Z7oYPXnxbHR5AW62jABMeYVGC7pAHNdWGorGS3JZPZPHvbfT6w8PpTPb+vPZV43j566ZBJ8yYZhSpE8mUdVNSDM/UT7dRpZK9HlvgQSJUgCIBUIofjPak3FRxDycmUuLoFTyFnBCcJ7TrwQBBu3ewfDWe28fPrjLtnr84tmr42P8lJq6e/d2Z3eHp5ktyvDVLoN9LI06YUeSu8YmY3vZ5rBhC3JFJDX9lYW5q/pb9QgA1liz26KJ05dTEBw3eNHZot8xJeW/WBdGJSHISauFPUe1PcjPNBBdrM61yWV+SnbSwCebbnTQRHho6a+WtDAQiUVqWS1CikSd7tkAAQAASURBVA4LgeUjVC4Qg9QlY8Jwy6p5midabddnDJbAs12VODTBHdfbujkYPvzud3gD1sfnk6fP60vhlOi33MLkrsQtFBiDO1pabuaAyRlHLiKhklRlnE6+Rp/NP86XRIvkyWMWEjzt2eAOyuIp2gYVcggLdA5FSCFuUdZInze1s4QHw9FmD6uRlqwMwA7LDRuIgOSvb2LfU3qz6wmMmMLWrd2anF8cNxqWiVvDVr84J3NP15hFNniJpCmy1pMTVM0nowk+lylnBiV8Rf6Cq9O4ZjJE6CK0hRCIHxFOYh8iY4p49Fe8XGmjP2Ztjc8v0IFLZpMxCS+9mLARq8blOKKlz2OLxAzyNjx4kruh9kjWAkBflmqlLGORneaZZ5WPaM1luFR+gw2VWfu1LedcUBhAIJM5ZuSvb4iMc5NspB4nikmDcOjYasaGcxrxZgwsfHW6NBuRNc8EM7cV8376+GudMWK2B0gQdy1Dea+7PZ5Onjz++s69u/sP7nNUaM2hJnc8n+0dHuCjbS5rKbuLWSshumWlN8iGcluNM9sQJoNdyzetgup65qoHunW3w78lWUJbeTNIt5G6nMVh3DRX67PRhToig4cPFKP5ZF6pjfuDs/sPK7cO7+xu72RDYf5DVRoabCN+68VfycNFFa3WbUvT2Ammn88qj1+e7r84q/Jd1euzMRYvtb+NfEmQ5JKmO2EJhCMwjKiUxPAGVWszKnY4APhDeK607AoEaSPdbL8mz7NaWQzaA3zrYvzqyy+vVV7QoncO7za7e/o2LM9Haf3YG1Kc7ICkVOzm1Unv8Pbf/t/+b/7kv/gvH//sZ+896H32+U/vPVrt3n1k/vYM5SKl+zFg9OWeHZ3L1K/fu1Xfue7Ob2bSeK+nnWEvRb0Oa2kpkVEYc8I3xXTKIidoiTbhduxg/2fDo+BGGLfOoWwdf2gz2zcipdAChhNPGx0g+0WIbItlKBsLI/V9tGo6IxTppX9ITdpqA0dnOGO3sYATM8bhNxa253rrN8YYGezR34TG3CLncW6ykYgg/ZPCs6L2cdK5kvYDP6E6lNCngG7dULEY25f8E9Ctyy6SVPDyyXlTuHxWvxpxYRAIQilSCJoM1zg4SM1CDSGeyEs+tPjiPGs6OR2dH03L7rxx7Grql2QIZnf0Kbw6Hqs6dpisEQw24ThJarHUkzhSyCz2aRGMITvPyXLk84ZRet2cCVXmHj7nyEUhRvfNsdGwX3/Kdw6ir/jcvU3WbfELInajAGlU4NfElMHIY5LxhzGKSytzHs8mJxdHNrc0mrg7Y5BSKwPncIIQSPFqaMt8rvpEv83d5r5i/1vcWjiTvXIuXnxl9wE8TNfaG9tZwpuI7Ov6xfETlhC5y3qJfiaROLEem5mE74SJR39PGMJPYE+tb/9sT1vP1THK5+ttdSUn73Q+f/n1ofzGg63Lw4a05N29++3FPjD0U9iQW2PitmDFtSNGybYpIypGBh4KTZPWGLMmApYpXGwqCZcKP1KUQHnrqVLmeaIDEIPFkvAGIjcqi+wnmW4YKu3kxx9dnH0+mnx1cv7Z+mZa/Ahb7Bn4JYO4Wx/wtDOQQ0BWLCOzfmjBqfCWrGKRrUH6omtZD5SRL5kJKXjNh5hGWc785Re4Rv6NS2hgQ2ARNuZttf7Wr/36W7/wy4vHT44++/yf/z//84f6/VpcnYEX14KYqxupgNrEx1QzkPBuKEwkB+zZwRfwDY78A3nxorSOtu1c6uzDCMlwxd8r3f6V+iaVi2ijMkfFQ5ruEsjNol4ibhW16QVsjCFXz1orf2HhzNTAzib0bpQBEsmO7mDy9rtOSMNHrVdk6aiq1Ffu7Opag6TuvTs6csQnh8t2e2OoHMLCRymGoGYN42CMIgGamJLZhEQLwLQqxBAAV7zEiqjpFey94k+WGcfiCguezLU2GtvyNk1A11vji1PW8KkaJFvZa3dabIuBbQbaLe2OTTOFLuUZnhYDOulCVo5F0SQ0lEyjkcjDEqBlRrDuoTOhaBXDBAAt/FB8yo8ETBCYHLM4xKLPURQC/fxhlVaZsihVhGsfNzAWUj5LV8xCAC7doDbin79E7mga2pF749kUnrfbg+rquqtZarNNLGuwqdGtQEljUN9/uH//wweyCH/7d/7g+uWL4xfP//6jh4iOVtHrtzWN6t7e0Y1u/86eKMHv/f7/eDo7+94Pvlc73IYC55PLvXsPnn757FoUVh8E4nW+lHr99gcf9vb6P/vZp8Nhj89PPzuRRc6Fvr02+0PrKzKHD1qY6SLaxmo53tu5mpyzpKv9Tvfls1cSFtQBr67iRpK4QISJWGMP8H+43T6fLvpuW7n67Pnk+qfPT9bdR+/377y1P5qrVZ9p+0rjxF0RlD7SdS3ZbZYO0SrSCFbMCsZoIAghgdG7dNqKwosk/FJJm2TLKJMqGixJz24QHAE/VjP6/R/82q33foGQi0KbrMYtPI+iKlZj1SGQOvRf+nf/7e0/uv1b/8V//avvfOfxF3+uFuPgne/q0KxtQWurt7VuLV5OG3st/SJXr5bdW3tbw18eVPfPTq9rfRiEgui0IQLdm+JEhpfJpeBMgz94Nd7TIkjoD9mTGwPAQMwMziTiO6zcDLTEyubEprRe6ItB74rCzavf6BGdo8npLZsWs4PlBstIUhxVFQkCDMpk6YiHw8RpU6kv7cyhWV6SG1LqGe8bLOe8WtRq3WiP8W3yQ+ISPjWw92Ri0BiuOO6notrcx0rggJdOJlivDTiCWKp10ZGKd/VSyG8Qt/Ps+nK8rkxqyor0xm2MdlRlmx24X2v4HT2Ar3TLtprZmt6KaP9WAi7xXKyvT188tz0vuatskdSlrIAO2YSDRwmXRlEs6hDLVmvG/y9iEfbJ6xdBFpOB0pvOvaG4MC/f4fCYKpmZn4WssMow2vLLsBeHxchLDm/KvzFaYiDGviHEIlhzcVh8VHWfUG2YU3EdROpiHTmJws1MFFxhnYIWflr5R0Le09PV6RcXX84qy05/73jyyrdGVdAX+c/TicGONoqYr5gkfVGT0+Wgsrpbubzf7g5u3f/h9sGHf/KH/3r88svK7IXIIaq4nh/L5JIRW7cHbJk/WodwMaigADHQbJfu/hkdnlpsLuNrVqbNhb0aPJz4lcG2c2f73nffu/XW7cHx/Xq31djpzTSMlwY3aO7dGnY10yG3ZtnSQgsOSQ7aPzICsHsQ4Y7FGZv4VNTgdFSAc5qCIiOgj5wAubTdMYastxiJj0VOOx+nop+HNnBDyKtQcjY6OXl8dPSz8/GXgx23VFfqApogtd0zS+ZTwFxAnqlBjWjfFMrynphwcqNoQeKse0Szk4biCxdGC80lr5c/J3PkXJHdehCkrEZ1kXy0OCObrdu3dyvVH/zdv3Px+AuNHvc1Xuk0rkenpJUOhef4RbAGmkClMuV0Yg1+OHimg0ZFEnumiySe4AysTFhP+/XKJFqiWqkHEeQheeqOvwyfZXMZ15u/2F8ZpvOpQybVOI3CfpFjRGaTISQpMeqVR5Y6uZCp7cKWKzvq7Q4b6oNFdDRpIljUBF7bOKzsbM+ezlChdvRuB/0oXcmcLH5wkysh6ficnDSj6J6wDmpAFdRMxynaeYZHB7HV4fpqJB6iMXXCHbopyQ2UU9TReIwDCB5kP8EZf6utFDR5CnoGXnCjPF7QIRYqsDqzWc/AGDwMqRwugzxQDqnSCXiX6R0+4q+yWHmDXbm53pvN4SOQMtp8dGVuF50fOcW1EeKGKBlF/veVV9wEk0qOK0DQarUTwdHttzfVtaNWG5aqKqvKAt6SqpFd4qnCxycXT7788tbe/nd/8zcrd+9+/zsf/ehP/5AI/86H79063B6dHiMQO7MdP391660PGr3BTr95cPd0PFuq6dnfPzg5PptlA2k1RaKcd8SbXx29wN2tFbOcUd1RsG47DSaR/bDVpBqQdajZl2L+4sXx4cFRr7fdbfcUbp1fhHFTrGCM3q/h/+qAsvGtQokrDVh5qONA2Wo+P192L1bf2b6Nv6NeMgsEMmUFORA3ymSwygvpbvkLLIOxQBRHE85I+obKwnfdH2tiU4iCLOPpk6LjAsZXYzVtfvnxn6/mq3tvf6fev4XDz+cXNwbRHpBCXCPsvKbkukH77V/6hf9Zu/nH//S31Hg0jwdic63mXpvpLJo7WWTnK6kOplFvr84wNpj11u726uLsvDFctroUaJrHuYIb2pUyAnhY2ARsLcEfnfVST0kq9y+VG5GLFj42XscerZWtnUpdyKBvO774x2jRwmTmFWqOj9pMx+en/U4zbaSz1y83huzfACAChZURfwMbKElhEUxJk5DmF6ak5K3YDaR+sDfoDJrsXZBMCxTmOrnNo0MtkVk7ulyO+S+UgIf+Ga+ef9WhgmiB2Kra+m23Mqtdny0mp8s1Q2bRrM3V9dqMrCdYmVZ+cQNGFQpd6zRtYWhLEWDMXV0CZstpdihazvSPkQM4kX0YqnegK0DbyLosfw5QKFzOrI26YMU3b3LCGb8lffPVRnKiukCt8ODw7DeHK8r15VSuz0+8Gm/+AcjcLYpdwbtc7EY++jYLkIvKS0wfZg7pQxNElvHGx+5KVpa8hKuL1fzF2Qtb6VGQNH+1GkQkcrdWWQ+/9YfbhPnwELAq2wToSLx/3ejvVCXPdPq33/vo1570hsdfMsCeWNhqp8fJebmairNwx2mVThjWRWDSypB1V78+mo2kgXDIGRahk9Fq0VC7PHj79vD2Dl2qfnF0Ph/37uxpSdO8ffBopycYT8Zx5mcFwlWySRLDLYptuGxccrCYARFY0O5gHKLWzyi2bPAySBZlK1tauwPUwukc2BZQCLdhw7TJwLIslc2zJuev7DcyHc1tmjGbaiYwWq8nDB5aWLIDjMHTwxlhDDTN8zHL3MGkQvD+gjChCu83q5l1cUl+t1mwcuL1S3m478ql3/6ivHd747XEkXS5rbba20j/h+3mF7/XeGbjaC2T0lyyiCMlPZopFh8g8eUG0bsjmTKQsKcyVfgYrp8RQ/uYsB6S8r0UmcQ5FUQXQwmOuiI/LfgXx9eEcSN12m5/dBs0m6gVGYPr3WT/Q1UsJTOnodKc/1Kby6QlG7lbU5gW0p0E1dfLm8OdPUa22iQt+1u39lrrK0m31mUg5HbaoTfG8pbFHY4QRpLVzZCtLEXErAK28LByBKoAUDY53uRwWQoP5bL1FeGhVhnfj1ma4lsKBJrgAO8B7PVUJDjwd31AJw4tuol48F9x98HQt74imwm8QCKpp6mDg0YIT9Gae1pyAygoiJEEupwuphMbOi56VkUg+c3hns4ELEXGu8B7APfqVw5Xv35j1m+Gt/nWk7FRFqebuyH1SuaTRSGqhEAGw727b73z3gfv9od7Xz1+Mjq/0BVHPs1nP/vZrf39/Y8+IlMPdvbOXh3f+Vt/c+feHfKw2+g8f/XVx3/+ycG9t2vf/6Fu5HS6lBiL4+5svzw5lyc9m6rVX9lb0DaHFyfnQ1VediPAf2lbzbYkR5Bk5bOkgJeboN3mFVwd62f+/OWD+52dHYXWezZ/I05TdASS9N2oyvxJAXsIpyEVNJmrIrovTk5br44bvZ5R2OtJbJETAexBADKga0AB8A1UN5D0lTF4xQoKF4t4CoYHixEOTOIYiwGFEmCVq4V4JAPpGn2ui/VV5dG7W83BoQ0olnZp0y2V3o/Cs1PldU9Yc/fw3q90l9PRl3/+8VePPyHmHtz7gA5iw53LzlWKZmH+LHuI+10o3wbhg/dry8f2UbCB8KVsSs4yijTqurwU/QcB08WdG9ed66t+ijeudSrcl9iFfLBlTkncU1RV6x2p7jao0QXMxLFj37JJIcaWfZ8a7jY/On7RPNzjE48ZFqMAG+SM8aF1fdOVK4kLi9dmK4MI9iReRkiEeaX9FjIBV+oOsNo6PhIxLk3ttbvYAWBxQlEXBX9hvB1EOlV2Pzdyz3YIlWXb7rxCEzo5q6NZ2JpKgpUuDJfs8S2M9oZnSHYPKznKbugG9WVx7LlBIqSNvWx6bbi5zzRlULYSRSGpxXJmC92bKesTnkTd+esOFzg233z7TX79rWPzldfY/g7IUQ5vN19Zkp9fzlzJPXNbyFWU4vJluWTDYYg5p+Dk5ufee8MhSvLA2VSYEGLWsCjU+AKKOZtcfP7sKz5aFwtq49520cHowtbK47BubMFsPVPMTxiTu/6qcoZo5qlyqQ63tm/f3q/XH1XW58dfjmRZ3qRdK2upWifYY5onBY/j1H6MPA0p6b7Z78nW0utDPxqypKFXm2wRuT8P97ffuqu/bXd+Z7wYyeut73btLSDLkWAwD5dRETEbBpraYUICI49rMKYD/pPpm/VGOBF+qBSm+JbDhZSWY8JPiFUhTEgGJhSYJAikNRoyjGNB0JpJ7dBH7fTkS5ugyQrSYl6qgno0Pf/0xbWtQJG1AW40iIBIq7lp4mFZIEtFTIZDRHolSBzkdj4YXo6yahlxEShGGabv1Zeb181l3371k7gvw0lyQEGKqTlUOu3KvXvv/Nqv9+tbL3/6Y8h7eOs2M+vV1491JnE1NpgHINSYAnGhlIegxgitstJhQIEUB6BRiNe4fwYfDsYU9tAyNJf5CfFS7ofJCJpeXjaXZXucTbJd6Qm8nEqL5WTRTArqWHy5bNGzE/Jj0UEyCGW/pmX8WcAiOYjn2UqSwa2DXR7d9UwBzEr1qnBycpcwGj7GsNSk0MRZUY6gt6Ejieg9ZbWLauWMR3d0Ae52mNQQJfYS6VSmZZ5Wv9vpp/WHaUIgHTc44eFz0lEqBwc9nVusmuHFOtCWvEg4F4Oh3zo8wnvhxk2qNq2H9Mxj3Ov1gbDL2IrFzysAeSJNy+G37usASnf7BicLZIsGFrrJkSfiUuVw/80FeZPgnxGEvwbOZWyQXYYXlUTSUF1jit7O+x989PAH3+OorT8/shAahZKgsFzJpGcbPG3pPLv4Hu3c2t/Z3d/pb08vLj/9+LPO8ODX7n1Y6arq06B1++L47KBR7+7tyVYj709fHumyttMfjE/OgW3CWLlc9MYp4bJMXPoEMIlgxfFJiSkYlzznV6+O7UIourCzfXAxWlzo6Wg3LDVtejDEOwZXcXqEy9jrFBt4renpyXReffnq5Hy8u3+nvky5EGAEYyEmAiKAfYjEDer62+AtuBVidDI2j7v6LyYgzYYKDmwxRaLO4ndEGwbHfU2XOhud/vTjPxPbfP+jX+7s3mGzXM5PpV7Hl5Pn2n5nqYACTbzzP/+fDnYGP/qXf/z468/4Gd+DbDsdPFihor07tAjgOEaiSQ8XoK9VBjt/c7369GLyRN55faB1l+gb33nZhIdCj33bcKnKkTu0z8S6Orhp3G1095RvRjsQqpGzohl8pCnwKk8Ug88NSq4OChXgTbc2Rsfo4tVeH+fmhJaQZJNNGC93RK4S05ziTzKjpRsbT9G0Q9qJ1KPE7AHrveh+0SHJHNPFohGOTlL0QtHYOCmQpIQAVf2Nm96wtSfQa+vfynWqdSvT+nJyvZgqCV6RvrJo1rYVXfa2bnQb1YmpISTDksZqNNOwUxfNN3nsud3ycj5azlTLqcAfQTMxHvRI1OIPXGtFwY5MtLJhpcqawsr+mgONOLt5/atvnM8dvnX4+M3Fm+tffyz3CYt6c2x+iPhywgdHsShc70CD0STKl+WC+AVgMyMl7ko2IEXaD3JE91ttrY/no6ejlxKbIOGcC7g+oJG5QXly+K33yVEXlXJ/KYH0H1Vk8lTWOtgkZDCeDRUZSgR5+933G/Xr51/ZdvRlHktxu+6VLPSUcsm8alS1HutLv6zt3bs9vZofj89mOuXqi3NwuHew31CBp/PAQftqt92z4Ur1sN7T+4kzVOQhXkF8Eqfhg8XFxHzlxtkTYTEdnZ7IpxlTtzXJw1VlQLDpYo55YlS1iA8kDcHMHhp6UyiXZozzxi0IHqu0Uk7PBwavYzIZi1rVqhfKMLQr7veG1GWUqFSVN6DbGlBAWODatwC09D+e1MbsvN7puYh4KJIuTrWStACGFuY156VfWlDiIiuYVdpg1M8X2WDK6b/ykvhLbpIlcQMLH76iQk7K1U3l/t3Dyi/xmS2fv0grLJym0693eYFnNraTFuxXbhwOhdvRmZNVl2XHG3znVtAyLrsItMSugagACq7BNz8xqsi78lw/dL1h2CnxerYkKbUOo+HRmmlYanjtwNrUjQtIgzsiTzok9HsaT9oEkMJLZtCfKbngwGY6Ozre3duTJ3qmO9LedvfOgUowa6q3uvmSUhhuRGcQP8BJwCpWS+SbYWTIxmrJI4myYVtY/toydRX6gpkfku4U9o2gYo5vqapraeOuyYput+vs3Gf3OAmISx247B4rRQLjDajjHIsVRRxU1IsXoZk+piS1B7oJfyNyQoi+Kg7zorEaKvhcXWFVxBWLmQA+PDw0EXv2GbY30Jixu1nu3FDyMwLa2PpwKHk1cd44z6DN5N+M388zo02yNMGvFETGWbkhcIRogYtbqNmyW8LpZHpnMtc8+tHb7718/OL5109wN8PmgZiMzufT8aunzymlP/mzH+92OvuPHu72dpPCdDH/5M8/e/jdz2//3X+w/fZ7jz786Le/+mf10+P3H71z9Mnxh50ew9fuzkoyu+LNorLkzVb1fDThXqZZQQTVemmskSTZyuxGR09tsG6Oj87qted7e3swCame6swd4179jBa02HuwUa2F2DbNiZMBXUHb5VbjeDZ//Or4/Q8fAggQRREDE2qH96VOIUiwYc0FOOCDVOKdgLdYXvCZnpCIF57oORx83qMc/oAsbXhytLjS31Pp0dGXX4RZvvXWordzi69Qw0D6Im+PfG5IOdXyR2XzZfPgV3/5N7a6v/Pf/csvnn/W2uu/vd+t93UtKKWraoR1VeTx0/LAlvTLav/+d2o3cg32WcTrq8dXc6mO7ArjlM5tDSUrtKq13Wr1sNLY79X3bzoPq52dNOFJGHF0U7OdFQs0u8EYKdUDOZQ/dGFyPrpS9wIe4BteO+4+6GnqUhUIVWI7ylouIwVSz5NGFkWj4xRMmmWoKaumWpdJjUMwoyKco6iDNUfUlT0FGGNN6c1WZk4j0QxovyYsLUYxE+utXY7WV5Oav0ScNBG28ZSILVBEhtbtOGmHRvjMz5pgEp3D4lCOF/YIHI+OXyWCImuAAAc1vDsCLujsjRki7LhHQu4kWpF1APzzw3ff/OWqN9+8eRMw5X1eAe6N3P3mpDebY3NNeI2LA5i8df3mDZCWL775mE8OJBlEM7aiN+fr4JsVNtjkoGPgmAnV2nkYuKhcvmTlVYTHi0jANBB6caCZcbhu1tezeRaBmxOZgFSwMRc+lmIa3SS9fc8XS/uvbO/u7Hzw4Q8Iwa8+qVdGZ25ZXwj/SyPl2e82tZdraxux26v3G73DnX5t3V/fE19t9brD3R15PLKX+STtA0NaG0dmy6mLPpjvICXrUexNtxW6kpLg2oxlsn1rV4QtLaqsDq2/aQdlO+zWvacckLbJzY+AQWhCDOUoRomFNDuaReF34aRMEI2YsEpAxFu5EpGCVIBWU2VoytndM1o2E155WqLFNMdEvKM2SuC6nCyWHTSKgWDZ2abPsI0i+8ptwAiwoPp61S2S2/nKzLJ0Zak2r5n4X3e8PmsYecTrtU8WFZqT4MeFdbh3+/u/OOv05i9fzWvT1v5+dlTlJEo9EsPVfT3RCCKtYHAcSxlCjAn/Fs8pPu52wXrjil2ATaHe8rPcIMiT+eUOfpJ+LZXpciV7qKtLevajBXFKuXY82V2go3Gvnn88z/JWu+3VZEbncRdARNYibNyMFEMt5IgEkVJZW5IMuvt7rDRtH5YXsiuT3EtWeVzwMxwhnS+DEoFhDlD2Fzhba1hUlEzgpxWqlvM9uTjHbmynbnfqZl3DK7Mzj43841VbzpQwpNFYEYq6/eHQYfTpEMS3V1QmT/STPO7NYUjl+dy6PChJPvC6OekVqNzEI7x3gSvN0R2+uaDANXdwxuFjsLgI4Jx8I0585cHw0wptDOXMvVxvtDLJ/SQZWvbGoEwYrdatRPXFlLex2e9W621ig7NKkPujjz4a6z/58pg2QCfg5nFD6qYculevXj179kL75m37pgz3yL6Z6NtoIaJpk6l/8I//J09evPyzP/pDtuqeXXx39xQYHT9/KSeLIjMdX5gets2frPLXH8OabQ5RQh2Cp6uK8J5ZMIKPjk6w1qJUbWXPzI1NmugUJCQP+Zz5QijabZiO295om7c9yFZp8ujiLNFkYiXLDD+AmTCBuo3aABENZYGKFuaHAXURvr6gaMISy0gII+WiJyWRF2sl97lO8DIUwmczWSbHu9fuLxfnP/vJH0plfPf9H+zcuq8aibscwuYZ9iEofsTzxy927h52fvD9v91o/8nv/smffPwnZ4vxr/6tv81tIKKFxQtwX86gAaxQUtyQNNTYfdga7rfqe9NVc8ZM5A7uttL7IM5URVPcxruV+q1Kmwy+Xe28o5YJeum3kKBJ2s6npjWMOWnQ5mfYkbo08mRCp3CWitvb3z6cTV4qwpJICkQbTZXeiubSti6R5ivmN2ADXvIqk42UCmlaoWiw2QE8fhu6px4k1zKfEq8VApRIB7vlel+zmXZq0135QxKslmc3i5HiTuYsv0aNOaVzpOq2KPtzXR9ieMuxIos0N5YawPZfA4FNMeej09PT6eicqU4njoUQ/cmKWpUQG2wJdhQXV7CkcMuiVpn9a77o5DfHhkC8OhPyefPmmws2b3y1uaAgj+XKx2+O3KQ869uP2NzKq8PV7rC5SfnMp4Avvb6BM1TiGH3xbEQGM8iwQcwJr0pyerXK/H1y9nIhtFsRkIibiEUL4MZVlhi38eetoQVZGdSCjMkW3lrUGvPECRN45zVwgVZ0FS1+3373B7Va7/FXn12PRvUTG/rx8zbVDevO3uvc2m7vDbZsbi+twUaIHYpUo93XD6Wue+fo1ajf7bNYI/WzgUfql4AeMWcBohnrNZMdVxQT20g5dTaQiF3NQNXSKqJeuh66yqsSwwjxZEXh3dGiouxBBUhl1QrkA78U+DrL7hdM0barYTcYnNQ3wr1JnS79JsKIsRh2hbwu7ZCuJrKLLL0wCTENysADubVP0kSWIcQ7QgzkMVkQOcdRbVB+nvuasUZgZPGyjt9esyhHb078lX/LyLOCIbnNIpljo6oTJipVj/HwQZcHqlKb6EmEifVGmpFgHDFDoHUYUu6ZR4S0vMGwNk8pg2NZqHtAXCRkxmd0FlCgvnhTvSnyQ1YTqEBOwjDhgKVk+sVAih5fMQSLAyAtAqntmDJ/BYSJrdnUo6wno0IqkKAab4g06exbvBLzXc8uxqrbtDM5Pz3VOHh7cH8wGMrjVbfd7nWZlm6HlQCqcW98u1nOACEaQo4Nt/AsmB9VwVZLmFv8ZixY64k5E4HtNoZjDASdwESyCuCEg/cDF453dLiNSUSIMzcKi/fQZGCvb/SohClZNXZNlgE46HHBDU5TNmgAi5NhLsmuNLqoCMQkBCbq2L5GRew56YmmY9TOEM+b92biDq533gUuK3I9prDzhRflvI+bQ9iESYCc9TAhVEpSBHgkkno+O2suF/tK/fW4bndPz8/Y2od3bms4I0PKrkGwORtB3qyFVIRYhju7CQSua7v7t0WOFRPXOioA5ldfPqnvDBrvvf/v/Hv/HpP0x7/3R7/5i78sRXVnsK1nKReTPeuXp/ogQ4Yqg1Vq1dHx2WBAMYh5WXhOgtPRRNmC65omJOPJFI0kYxyDYfuGCgCKLLCy3sjg6c6hFq4le6DRxghxgaSOwsyyIoyS5JNapRAeZ6YVsSQAlsO3ZYmKFwfnCjUH9cPUoC6nAkM75cAwPMzOHz6HQhNGEHK5ioQmFBfTsyePZXotPrhebd97Z6verULlqwVGiLasaqPSujqb1WXV/+L3fmV7MP8nv6UZ4ouf/vjOg7cr7b6UUIiaZ900O4uWLYyBpHbdIYAqvRv3ClO96t8sTy8rC7NJbZViyupgq7pbrW0n4fnaHyNmadxJR0710YrvSEVo08mSV1PyUKBaFMZiQ1Oct+8cPvr46LlE27SDZIrYZA4UwKkq6IzkObHTmjqeLmiFrUNj9blWgIC3AkUd4SoukJFkQIuCXDX+/9Z8S51L5ZpYxauHlVVvtmqsp/Ys3lqNbGFshx/GS7Y/9TjbJCSuJ/CI+uM9SC4/esliqdgdR/TOF5PZjMdRPslEOgttxQoG3SPhMq9IY9eXhTWy6M6OLCR0Kaf/yovf/pVzObHhGEGGv3hB8MPx5kfffFtuH527HN+8gYaxBxzOv/4Z+EbfC0J6a3iFCfmIeW5kMBsiGU95CKolPKvrJxdHn588k4sgBswC4DeyE4JfUAiz1q4LnkaQcEtEHcBYAhtLSpezA1Sy4bZuujPxdLpLdKQKS/jRO9+tt7afPXtW37qnS/FWsjJu7+3fvbV7e58RjKhml3MJA5dSIz2Xw+pma7I4G89PH3/9idrEu3fvsonNkT2Kbu0Cbg3ikVPQIqPQ9mJlMxwUkH0kyoythHXj/2ScYVdAkyC0gEZqr3FSaEc7EZglLGM8hXaD8aQsjuYMktOBiMCIVeTSBIEX17iay5QYxW8jY96OYjYQEDihWbL+AMYTCf5iFtITJeXLjBAf0jnEwoSFB4oxpYEvsA/4IVPW0jA8CN07Nh83r64vmOfTXzkwdRfjF1JI2JnWsVgZwu2029yUsnXndk91u/RNuxIRc0oytmboDdsR7Q7HsapZRY/Jc18fbowODQloivjCpwAzEhmOFTkB78wqsgR4/cxPiuJCkin3wXAvuxDMneN7lzJKvIiauSqyJAVBzWF/+3J5QvjFYlM3raVzDGHNTutKbAd2Seq0x6cXrVcng4Nb3Psan+IP+pfVtYyhtLNWYrdI1nidfGE8OCjJ5BGWkhHqNRoVZUxFFQ+XyJsYBgeXkJKuJHx3wKSv50rLiVWvbTMsuO1MqNzQ2/ro9PvktKWPOVv0cUFKD3BBkZ3hBd57JXJhi2sJV7gEpSyHJBLfEqjhvOJ/8UpFBpssD7a1plm6zID9yrF576O5lHsmwuJXzHFnfOv+vvAGRjnvjQscRkHJc8Y9WfZAWrzsbggLdDvGTKk4yxevjlqDgc0tHty/S9sQ8YEH9iLUa+zt9987eXWkAOt8fH773t3h3p6B9vcOtlqdk4tpvTN4/OXXD995365ljaOjvR98/9//3/+H/+cff3x8eiZSc6e/w+0BXIRwBk4PF4xQg766OW2fG7kSsJBAGSymGm9CbCzuI3RtRZI0Ifec57kwWUIzSysWb3H03JhIc0O3nYZU4vGceqccjgTnoKEGZ/r4gs0/ua8FAyNpC00R5JCappnl2XDrInGNB9rnoiiU1GnMCyG4GBjxgbBLg5XolDas66vz0YlutFS/68vRl5//uUTcXxTBvf1W9pWMOVgSIexHWNOHrjE627TpeP/v7ex+/k9/+09+/1/rDcGXUDm8DQEIJDq7jrVcr9yAi/NZbXHd3GtV9x/2xbnXdoV5lm374leSlFW/ZATf8A+qMrKXgeQJWxWI9rLxxSc0uT5X8aWCOVKZEzOIm+JJkw2HwTQRcKOztXOg1ok8jeIorgRuiaiAcMiTCc/7yx+qK1J+ioai/GSXEo5hA4modjBhRbGlUzkTN3r98kw+RbMmoX7FodFXELWaNpajG12dq2XjXpvhZPMXy23ZluqtpYnhgomPKDQId1IlPVO4Ml7MRpppkL4Kxwn+bvum1+LWXoKtA1HDc4e+g6hnvlyEt5dpZmCFUkIt/+Zjw1Rd+ZfebH6R84UXb261Ieew1m8dpMS/iQ9v7ln4IkQD0nLI+6fNg2Z4ayRxvgqnTCIeVTT2W+ww0AZuMLt+ab/Ny5PkOsAO7gJViAsB85ZRJVYS5AQJ94mkckOJT0EFMjWaldLmsSKD6PurRbu/61fSAbgcbt+69/DhoFYf1nc/uKvqX3iP4Ev0ZIdWaKPvBQyBA4qqdTzo99O4hlV8f3+7qzEsTnI5t58HVaitYwPFWXM+eNGCNyEarSdbvQHogJoxmdG3gPb6LXYpj0B55uYzDAgYeMyX6os67QhG5OznUU4aTeySZ4lbJiw7AljHZoXsNvDR9VIzAk0rrX+o3cF3wwGP2eDzamyyQQr+TBq5SR3o0rOZghJpIWtNgJtfNvDEIJBKhIOxKPGLCNmg1OtBGo1fF86e+EY5NhiwwRLfYWeb817jK479QAynv05pokmvtD1UU7ODraOj2ZOng+HARl+1HnJBukuVNboU0T0YXCZkWdMF33/WOpYrE8iuOhsUCVGz8TEchcFI1xisb0iC6CTneAIRyjINBTFDV5yO7Jd9Vd3bub2zPT45MbDkrUvqhEGlWn5VmU/tKSWlWSfehR9WMDjz5czgI6HYXlxc2LyyujPAIFJ6WKt2hzuYdbNjm9qdau3F5Zoj2YV2PI7cikc/QAuiA4D3RpSxEa8LO/A0bPyOm6N27g0muD2W1IHbnnM0mmoKUroqapwU8SnnUgEXErGI5Dm5Ephf29MbOsAPdsQNXyshgiQ8hTRVVeVZGQAulTaQ8dlYQHfbYCbm67cWyUdCVHaSV6LLlV6NP5IVZrwRvd6zA9wzSMbiKRpAmQ7VLazBeTfxlTcuNlQ9EXfUf7e6lB+1Q/wK9FcToCW9ODl6dPv2wd17hNzJ2fnO3t7R8bHkWu6897/3PQ2R33v7UWV70LVxp9rhXq/J4XBwoJUjLZOLGb5ioUcvX/7ev/qdX/7N32iLdtfW+9//7q/8yq/8wT//l8ffOR09Oyb7d/f3aqznYfdcwitXIXC1OvisZCHSwIs32Lc5mmWkEd+kTgHpxJk5JkYuGi90yGHFW8tWTCZB/fjsdLCzDQAKX1ARGFvRbP8wmw12d7sy6pHbkt5mP0AxDEBOUxhiGEjZ96Ly7bYc2oU8ZmgBp8p/4QAxr6hnJVHrkokOcQyr4BHFNmwhu6TiYuIHSvPhiRzg5snJ0z/8g//hrfdHj94niFcjNeI6/gya08mqlaTUlpKZ7slZpT9494e/orPXk+dPtwZdzaG2b90TI59yP7R29L/QPY9JLBaOITV1qNi5JQVpu3Wv0p7rAMwckXJmmx8Ego6lfcqiEU/T0k+mGgYWts6IUmJALLFDw7dRP3towyfwMsCnXZeENx6dy4byDR4x9SakiXR+gUj8x+XR4uPR7lzqpIHGdDpkzNqVh6RPBLllBBxnNxzV9aGERTUq2kbevFo2bwak7HKuK0R7fd2rrTq1qaywHptNCVM0IXZZbHGySLXAqiUTwVtZKXIwsknRqW3Vp5Pz2Ed6RyS5e14agpiIABJJgxMZXARZsq/iZeMHCv+OEWWJ8q1/80+hvp/z/wi+EEoiNfmnfMidyvGX3iDjzTUu8z4XlxvmnuWIPenexVpx5eY2fuNLDDyXhO3kd2gZPwd4Fiq09tmdImaDXnGy+WAvavmIklLs4b64Wvb39s7mLz99/pU9aU5np4qA0TeWQoenNFnliJjQvYnTaHjxc3C3xl2ndDvOSMDWNQzOXlbagxUr2Hiva7ajGI8Wewd33nr7w/r7v/59HIEecHx2LFDUuZroYYkx9yVuMABWs/HZydXsfNcG6e32sF3r6OFMTEzPj7OtArErF2sQnlu22dkICQOMchBtDgA2a2WeRvjz143uEL0hHBM48hrlG+4RAcUFT70pzmoQVvKgFIUzPfof1YCfqOBMj/UF3/0GXcYAAQMTj7xjjfNGwuxu2kMnvkhqcBa5EnrnFwaIHXsf0zuW6kajMUh/oMqiy/3+Tce3la8NTmSpLf23frSZ8OYOQQoECjBEe7/XuX1n993J2Z/8SbXVrfZRWvN6NiPco8pkDRe0D1Ci+5gMU0uyof4AeGjivm4CoMGzMH2JFskQzrONHQYE6YJylABEEejirlG4JY6OF5e4ESJKTQVTg4mR/eOzpTwO2kqvbPOPuApgrAQuALZSLa+vp6MJK7lrW57p6vh01Oq1BrbSaunnpwhNVgi+GBe3waDvmDn+N1TBhaxRBJ2Z2UMTwPNkI8urPwnseAm2BoLkWygD+fPaBIsNxeTMnCnJX5KiGNbyhIwFBVdmSf1lTWGHwUb4ZTnLgdD866OTKJasdZT3+WTHTOvlW3KUQAVMIAQtPwkwy+HbzRsnLY1rXt/8zXnfume5rDyy/DaPoUVBOIMr1jnAxpW9Ku3emk2JTj/84Q/tNtG2uZMmI8Ntbsg/+p1/9dUXXxYP03A6n/X1MOj22FHyMDS51xULHLBH2OJ6Sgupd3Fy/FT3yvP+zuxwbzYXROBz/qM/+qM7g12ksd0dEBG2S7teTLS7ysTp5Ch7laJnaFGAIz0heAtOIbbEaF0WNYUIt1qOMjGAjasBJQsqWWIJknyflH1mHfVxoXfoxciWYAKdtGY/CWQgYLhtxAnIMohFcC8mtlJQwcUD/JovbCAM0SBcWLdheFf4qKdClwzLfLLi4OET5keiF0qocN+M1+e12uNPsbw7jz7Y7u9LqlpMzhvtXU9pwYv1zfxiLm2ssr939x/9w/6PfvRbv/Vbv/DLP6QK6HQ+2L9TOT7FHFXx0v7tyrMaVcbmsaj3DvYru7cqi/Ot9irNETiqxNk6tXaLwUzhG9R0vE9WISDpfa0FWLchk2l5LC21fsM1be5wjCxInFHFddh9dHKyS4OqVZfhIiTBgsS5tqh91wlFAl2Q2/Y0BzTUpV3KOKhyVLRzBs9aayAHpk23WHe3Fjs0gbVtaMc37WmlcdniB9XJja1UuelULjXcwAG3eTKkPWPq4Mlqp6RzN9uTWre+G9uf6UigaQfDN5uzzwJ4CoZK/YTgqa2Ii0fWmHiuwq+TL5NgonULGzL5slavRWM5m3XMxP8NxwY9ypVBMMfm/V+6fHN+Q5hhJuVZTkLOYERxkGyekVs4Xg+mcIbQ/Ws+4BtUSKb4Ikhe3DkRC+5AjKVRIFdyWEcw3G56lcsj6Rjz8fh6imIyKnNxkSnGj+2z6QuEewcb04ihCI5QfFk7sPEVjueRKF81B4AKKvhkTOn/ZTnrw/sHzU47MnW0Pk9tyuVWd0sfOHcUCGxF1q3PXz5XAbq3K/BWlTeb5I/T0xN7dAtoDIcYOzO029/2eJNJ9muUuM3Dg0eRwmWU336Fa+HO5XyBfLkGr6/n0dj0JjKBuWZC5oYpx0MrU5urm7v4Zi12mLreSbYmlMOX9C+B0eK7JeGARrv6+GO7BAzkNzb2Y1Gps0aBJx9gOCdW4KcehZEE90s2l/Hkmn/TsYlVbL7dYJLXQPw1wmWBXrPtsgLlbsZW1Eej6dcHjx6yBpQBjR9/ffb4K9mwuKrug3EzLUSLhV1xwayVoQO1unp/pG6QrojECB2MLvAIJbjaLMrEggEwBWGzfiFP0rS4xK+uJqt1YzYDz11d/oUKmC+QIFkj0qHZEy6JWAerIrRxzA1pkYp1sS/p2nbH2m31GEUi8i3ba1MQ+tXOcBUZnPBYUtvcSdAgcnuDk2HCbg/CMU4tLi7rMPisdWweo0h1YzJOGOAUjPRFitzizfGvwfixNY8ey+t+c3NxfsZf4IcJv0SQRhukLkRpkWGQwHgi5T7AH3+8Y9A9zoTMUJZZVIHws+wqK7oRb7bVMKSN1PHxm8XdrK+P7lceUcia9IUq5cAdjLWsv5ec9BqdpxyCIpksL3RLakuftsNaP9Zfs9nY29lmaSIrIjh1+eubMy0/T88fvf3u/QeP5OoIUmpI5DymAE35P2DFdD5vd3p37z94/viJSUzOT5998WWr25F1NR0eHWq+3umcnBzpXbXdHYIRORvfFnyQQT/JYDYyuHgCMq0gK3IohzeFtTEViGfKKt+RdYteUmRMuFhEMibGGyqRKf7QlX1z1El89O7b3JYKzGV00grphn7lKczsWB+JbcrcailinE2pAlkkFxgANMk/YQgeg26ylE4YKSxx1hON3RJnuctISjcAWB0tj3CzUTlP+eLZF1O+i6urt979bre/gwVwAvKdabjbbvbWttOanHeHvertw8Ev/+Du11/8+Ed/tji/2BvsdnR7eviodnJRuTw3xhSjxHhPj8GpeNp4Xe/v0ZJZGhXd7LAN02lXec7GTET9PgM019qVPPE1HI9fKfNKLDUe5niPwZ3gDQZDBtp/upovZ+eXHejXvpmCtBlyjs1rKpGyayrHJwLew+yFbVfSuqcTKun6ygaszZo9uuba6fVs5bxe9K+nLfvd1ca1xsUN/3eClZXsneyeSpt0VZxfJNxgPSA6mmLyUgTiq3p1pHjKjtR2p+YfuuLalG6WRmA4ZdT+NLPMa6ItIB8VtKxIWRer47/wOSuFwzjpc5akvCmfLVg+/qUjQMqvNl8FJf2/eUJ58+ZT+RnE3Px8c8U3t8r5EFb4bvlqc8/ciuAKGWY1NsMrN4Q/Li+CJ08vEzF+KMyvSg6getiexmrd1ni9+vLo6cnsdFJNe4TofG5fZmmOuQu+AjOTauewdoWOwlmiefietUWeu0DinnoyIEQCglCbKkj+sdn0oj5P20T4tLZ1YP+6L7VBUIcTM4+7uRwO+q3a1TEfr42Az08vzk52hkP0qTaJouSJzWtywg4WSFGUkDszick4cJlaBhaiDnjwQOvyrdeNMhMh5WKX5Nt8D0lMLaI1NFi0CDDhzQVBmVdQEgzhZDYX1DLvZj0gJXRrE8PEiAmOrGycvzy4eu26I3yXlOi0kSTetZFbnlvWJqtQlj9aaunU4f5ORvLkJtGVXi9/4PzmKOudD5lfOTLNzQ9NIecszOsZAHyuzH9mxoL3dU3hP75/b7h9/eCLyu7O088/H798tbq5yCgUJHBPyZ7lyYujo6h7hituKBHDHczCdeW5ETOaqHAgbYQbRQtu5VEuTOvEpGHhmf636aQQhCYD1ObtAbOAbF6omFT9p543e0FjZAVUsJay48GBQ8QJLipXSNRQe3hmeX+wo6dSuzvkpe1wJOtx1t+p1Dv6B8mVJSoMwBr4YcjRa6QohIU7iVgHAJw+sWeK9RmJi127rOQupNwiV1t/9j8xgO1iHCZaihJttFc7OTnj84wiEhrcrGCQhXTIQ8sRmFu/cjjpcKv4vqIXZGxendkAkobhiMh5c3xzzebN5no389Griy2vw1qg28Da4hae4gabh25eTZmk3wh4lmhvuG0dL8Y/evHq1eMvv5ANc+fBg8bWLXjPHsElz8/G3OeH+7fEs6/Ox0fHLx8/fszPyHPrzp7iDcx5cP+hJGfArOhvf3F2PZtoHzquvzwQTxoOn1+MpEL2xShTUiuAJ31XLhUD0HiDMp62uZvz3hikO5fJ5SMpa8wZdlxtOeMo/1BukrSxtCOdBM1mJ0ESfUVq1w/3d3/w/rujZ09jT9DCwEeswcXAAi60JvD2rtu9detWq9MWaC33RDMILcfmTXkOPp91zcmwMMjOExq7I7KPxhRMCfZIG9vQL26YVujrxenJCyUFkqHfff+jwd13SJVkoV/aKKaHrKDu0XS29eypVtt/49/5d//H/8v/9dOf/uTt+3dtlHrrsFNRzjthY0paxEUFtJoaVizPL8dni7a63/5WS3XJQskspVamUrtiM6WYT/ZgoY1o8ag34NLsk06q76XZ0vpCQQZJyzJ0X3EMhCQrLfvL3B4910OhfT1TQsXz1JQRBTEZ0bpxQlAx5gphbP+Y9VVzLRjUk8eR/KkKh3OLQ/omoretvvJSWZGKKSVGU55/sFEGSFOQV4h34H9oUr8aObLdVDZDYTr+dMrldvTllyktSMYjamfhXNKBtbwlhgP4vI88Cyf0Jg7JrNNrz6WF8FXIKIPdHLDIW+f9W36YyW+O8vHNh/Lv5hrnvznA6C9cUT741iDyNozj9REcjoqTLzfny5O9DYVKpcjHQpWbH/hl8g1NKDwdtuSywmVwFyxZQ2+dR2M6xsHS2jqX/3zyYirOw3DNrahSLAEgQvJCllB5w+QAAc8JGDAiQ3G2jMT0/c6SMyGEeSguDGlCyi8THh7bCnwhwSU+SpNY6467tTVkSzl4IzUE1i8C7tAhhsO+xEK2+Gh6Qf9G/PRxyXqZg/o7NxBijA+Ug/SS/5Gi+loQZnUyT5AqWPit1838C8L5tmi6mUTAbEb5F2TBzD/hkJTcslperBAXSLE/WLfNbeIexYdxUtbwQBSdzr3SKtJwwNjsB1w0aoqhNIkCHFTjrmHxcdUbooCmhQkMIyydCDN3mspYxv+XX8rgAuifvyljdxd3yBJkedzIiHwKEmNlNAmpJW5ankljbsv63drf+eidd+58/umXf/6TZ599Onn+cjWdEJcx7KRjuF1Jx/UbQXOA2jDQjL90ljNq8+CYLyORxeTfwCvQ1PIsyWWKMlObQE+nfk2o2tZ3MtsRhtMo8VpzY/SGSC11S4ubLFnm9frPc8xgOpl3+ruMHYVgk+kCyOtN0axWtY1B1HqXFRv3iEasrs/ghrAXMk/EL9AogAhk44i2NCW2GEWxQDkBJViX0nADJ/Rp4Yofin+SkNPlgxxCTaz5nCx18sQoH7RXKwcO0N7tMA7wZeV6qCsLNEIAQehiqmYwUCBVBDHoI4hSBBv9wjsC0itQ+9YPNgJps+oZVzmcBFSvuaOjyCv/upvnOQ+87uA+aMRvfeWNdTcKJxGq3tvuxI5jzdjmdnxx3hx0RxfnL188v333Tu/2XZmoy9nl5HzWsBFDvSER9ejFS4WE+3duSaphznnMxckpZ/LuQHurzmo5bUoJUcyWvtfR1+4Pdw6Gw49HY4ZnZzBsDnqqm5InoB58IpEiUd4UtxhVgYtRErRcFE6ZRdGNgspBIkhr2Jbstee+MBSn0hJHaJe3gmG+sMcgPLqZjS9ePoOvgTed0ANKbpO1JVPi5wgycKY3pHnfv3//4uUXBZIh5jcHjHCgjzw6xm/IKcuJVvGELHCACr/C+l3nL1dR1JSMGy1kvVldnD7/zLbWlcUHza3B7lvZDnw+Xk/myuy6g207JbInpucTzuff/F/+O//qP/1PbAyl8vflP/9vvv+9X6oN7kAp0h+rEnWFkzx61Upnfa7N1A0/4tXZunJ6Xd9ptvbWlZ3r6l0NEehO9dZVxGTa0ddn+Ldyyey7zWEfINubNfl4pTIYvvIqTiv1vVv7H06f6wdS09Kmz5BIIlWvVunaQHLrhhGsRUO/Muml9snMF+3apFubX2fLBJtHnK23Vu31Qtyqfj2z2SK/GTXZSBoqN+Epmg54kBf6kqt958CEnL2R/SlSMDq7OD9dTSaNdG6wDsFf+Il0EQh2kPBV+GFZnBBxDktKmFsD6xLumdfXi4eNWpF8E8aRZSofw/j80Pu/9BqglvObr755/+ZRTrw+NlciY5/NZ3N4//qe0Da38pLDeUvnK+TsFbmaWk6GWOlAUSeMbPNzPyUNLE+4SH4brVrQ3pID5NH43GaygCi+EvMwHg6eM0Gl6KRckq5n+2VYQRMvHhTJnueVG8agjBJTRE3iZHQadydUxdK8n0cv1IdOE6K4iZIdkxFw9uiHMFtWheMn5ye1m5UUP2xcbZ9Yig1J5n5p3xLOEFWDOiXZ1UUJYdRi2osnGOzG42iQZpeJZUivVZuNgrORFoFXBh2I5BpvilJV1jAf8nV5tdLemaqLN2ZxoUSmRGMbiOM3DQhl69mLNwPQuBAP8NDweSy/rFBQJ0ui+SrBnGUAunLf2KXlvd9uvNQmAoAxtd+MPIP55tgs9jcfyxTyqQjgCJ4MlmZkYQKQ8hD/mKW0W0M05PC/NPyWCFU5PNje2/ml9z54+Od//gf/4l/89A//gBsiqd9JkkrgW2kZwUZJbZXMLg/C42THuLehAxzFCf8JveRpEZme4dF2faQN0coRDhs0YthYb26enh4vrwe3hB71tHBBkfFGZwGgqBsGMwtCZ7CvkdbOGR37sI/VHo0m2/cEo0UBGNL1Zv9msHPY6g2jwgvcRvYW+k0uYVm7Apmynsgo65vbhuyJK342E2kZeyLBZSS+og7ARgcHBwWNjmr+CQavkwDvrsXNEZwx3dBQcK14N14/5ueLYwykhld5n86W58ZIdVIOiVd1dOpuCUsxWhd4qGd5k5uX6zPWzYEEU8kbyQRMzrkgpmIRvTbk8UPE7+abn/tIGUchDiflcF2MeIF1wW+99ejBvTu3lG0dPX/y2Wcff/SD7314eXVxNlJ0cH58VmGsbu80J1uXs9Ww379zeEueW5bJRgsnpyrBHt19qPhPrW+9aksEHFNtdJyZlxcj8YVOpzmeTk8n53FNSfvDLNyQnIVOZV5Gvpnd5qPXHEHb8i9glcOYg8DlGyfQUuEtVRlr09nF9TyJPTpHyzNrV6+ffvHZ+2+/47LNLVBX3Bra7kt9nM8sJ9AWZaTxzvvv/fSPv8oFRhHy+PlhVEAPC4PHZUDOlIjj68tK2DpfhkFkSEaYGns5AbCtF3FIGTh+/OmPpdF/9IN1d+duZag5PsxfJXNzy4bq9bk2ZMub1vbwN/7B3/2d3/onP/rpn9pC7nN1RdtH2/t3Wod3aoNtAd8bJXhpg5UqiWu6k9bjyhtH19cnl1evKuv+zdbRdKtH1d+uqjy2R2Rf9wybQgz120tTyWj/Cu0XRXYgishnAtj+cVvVQafPZvnaYLj7xytQalIR1pcy2DTnUEZgXwdmrr1QUo5yycyd1jWHri6b9cv26kLfjGZlQZuoV1bZVTb9CyMSkuufYPt1HKrqBfSBswoVHZPGy+Xo1enRkcZIGuygNan82bNEwKj0IZCZ6teWj5obHLE6+T8IszkKY7AWORXBUi7ylevyiibzo6z+N6j1zXtnNie//erbb47ND63o5oyPhW/mts7IqA+tEYPFR/UGezEBSGV5XjMrF5ch+LcMI0rc68OZjTQyv9enNnYwYzEyKGo0jpBm++0t6XZfvXo2Am6KRSgASjKr4r1z36QiQNsAhkD3gtrxJW+CuoZTlnsDt1RTBCj5xv9YqQHzstDUbZ1i+zBbvBVmGH6ULU/cnN5QnUm2vFmPlPPOJgNbx2hUKR9960Z7LJ0fqUkSspvNrngPLYJgFtDKBPJ7Q8XwxfmiO+OLr2f7F//5BtCvyWwDqEhrQ3xNfJulKqBEYm8IFRhei2BQp8F0CTMBTHlIbuWJVogNjnaKgAM88I2L0wFcvnUAoyvANlScGdvTMJaiZCDgjec1YKQlUFv/egv49QIXBr0Zp5+EpRUMNtjXi1ymkrGXeJgrLJv/RUgjmqLXq36Zya7rMyX3tvd+/Ve+u5jSiq6n4/MXLydnJ/gkUWSHFDnbhsdOhXyO4Es4GjU7GR1F2TCZuL2Ngoz31cY0RB6RFRyPQW73yySV/UAI96p2balL64hvidcAjvotSMBSWm3uTymWZN7qsn3rnCHDrjrRo+OTw8urYae3Sla9pNpWb/ug3RsaFvHU4RGhA1gCQzH3YvNuGCooOcObw3TinjORLMLGZExqcHQXJwv8A6C0iIhdZzx4d/qWR/BZcbFRjIlC4edy43My84ODIZEs3walNo/T6CNlZ/q38N9SN1+jQ6FSfIoxJOPa4+Sh+IpwDYQ3NPxmif3cwVowbt+6Bkm6lcN5z4NX3x75ZqabibjYAT3kmVKkTO/e3dvyqvjSi0xbj20CcHL8h2f/+vOffYK+Xj179fSTL+5/8LbiQxawXasSiZNIPZ8FATTPenm00xrI4pGQIY11PJrZxFHdaP/g9ujVMS3NdI4mx0pJalO7zV/2q63lZMH9soiyDabRgFOYyOsYPTNwg/ChkW8dgT84AH5YnO8saAGrslGBXtC2dXutsj/sv/3g9p1DhVOJLToCOssA1zj8uUSD7hLRZWaQFrr3T+7ff4inQeBwtr/uKLiDXHCDMEfa8IYzbxY3yA3gRhRehZfUMDREnx64OvTE1WivsZMnny31gXv4/vf33/3I1o3cB6acyOZlVXtkyQSTV0f9D9793vRX/sV/9Vhew/TVK6mow/Hs3vXVACT622RNi86jCuCmH+7C6hfuWdXU1F6JF2vx+mJc66cpZ71tQ1YFAr3KkMEc13GtuaoqKEvDAbnLcJ2TQWRYQ99uVVdIkmOreTk7uOm0FIumN5mtiip9+zGtrnpXcy2xaJjU55rmvEvunrnWWU1FG/ZLuNTTbKnMWGGZ9K30Hk6tp0YDXPCqo/p9yGam5Emzz1Fhe5OL2cti+J6fzUYX9BKsWXwOpK7dWqfNJBgAdURUVi9u6ALpjTqeBSqIEfWe8RX6LNAIH813YdmF3F4TS9yRjrKwfwGjvnW+fPnmJYtYjqhUplKOzanNV4jLueBDuW2QsbBorh3/brDCmc23Xl1eLsjdgjzlMMpwm82HzZl8HwRyYzE4vEzXA1h6MZs8fv5U5jN5LI3AjN2iRBbhv1EE7fIsn9w0SByPQeRu+T+maBmMX+XZAVhMB/ACUfzOVHAPiE1kRmpaMPkdJV4omzECCj32t/uL5c5siyZV0W7mYnLBBaq2Q/2fe27b0mxnrzOgl2/zdIUHpuDvdfNPg7NCZYie+nr+fvXXHAUer4GSZQg4/L+x7kE20C0QNvAIsHzKVbmUMwexNXo4cpLQCtyBgw6MTSbUioHk51kuCOpHsVQ4EnjEsMuUtAOD6JJLhcj9F8kVoRlfHEK2SwoBHHoPyr1+zWDdyjU0/E2UNK+OjfzNwMra+F3G6p/yhPyblcuAXOJdfq4oSVlEU0BASiV95cGv/sqDd9/+s9/9/a8+/njCWSzgRxKiRumLZqUFGEJOAWAEcx5gKkZMHvh10CFV4toHQFjrGG7v2zys0BajOXwK72xYx8TG4Y9CWffMNrycWsyzYEpUg6iKxabMDW8uRmc7kn+2dWhajY5OJIXY3o5lw5+gZtFvbQlAO6YN6AYUWysqsiFlXQIcD6VyxgpH38SVRQvYrItoKhxUfgMPjdNMIk6BKgl+kmpsHZ19er0qmCitjKF06VBv1Yu0xjX8xhEBnLlGNHp1JliYXLzEHtS1epKP5Vd5Y2i+EiIlg6FEeX58ORsCzx1fH5Y789j81itE8sSS8ZMy3zY7w9NJC0WeabmVcuect8tBwggl/7m0Xc5cbBvWH8K8p18/2d7fefjw4cHtw/uPHv3kzz6Ot3l7R5b/9OKclgR5yCuegeNXL5N3dnCA4hhxNjWfjEbaykAhALgYT6VzNyt1tdnno1Fnd3d8Ye+qyta9Wq/JqLLHdvaXBBNyIK3ZU5ZmBjHlMYuoZurwI9QcwVgTTQiGs5eWVHAV3GxV6nzoR2hEh0tt1FQ+CRYmsNu/pU757n0OG44Z84YUFBVN+W6uTzE3uJFoDi1qPlldjA56PSUuPCdLdjk5/ubwcFD1SpeJ2N+gU8SCUSRxP2QHjyIYCjWGUSSex+Xb0WhsrSBVu0Upyi2S7+py+vWnPzHHy/ZW/+6jdXPHAon61G06kFyYZWd7ILtx5+/+xt+uLv77/+w/fe/eQ027zo8ej89f3j599vDdDysHd7Rn7uAYqwmBhXWq40o1sIXmUFjoYmVHI30PVoubi2lj3rQLhMLa5kLS8U37utZr1XtS0niRhXU5Eu1pX6MKVKTkmM1q92byVr26DWIU0paW8dXtLd0uV/UrrazQErl91SIil2NS0l6Ana3rjs5Z66XCJW/iduZ5TJUhxSi67CVk6w46Glm1ZhPa2w15e3L8/NkT6MQ5ClfoTCoI09Uo+ldqmDCAWCgF9YG9dMCxh7T43esDFcEJi4KS4UIQJDix4Yqkbz4Cci4rx5t/80UQ6ZvP0Kccrtq8yQLneP2EfNwINt9DxnLV5hpNtl0X3MiBhjaKHg4YyeeGxhYdwp2ZUXloDBxIS6ssj8DxywWUhTy9sCaEHvPWdTGCq03ajrbaV4B4tpo+vzyeCwBzD1jpaPdwB7vJzVE/unMLIIASVJxMNOwut82U8NANoMzI0+KR3PD88DUTi1ZZfk5JSujLQOEx5RxZaoIzHp31rLUWbXzc1WyyjJBs8mU3TZm425qPJ7WHawsRqnHUU6fW72tQrqgAg8u2HjFAy6K6YeZfhF8G9xri3masPz/KQpQxo7XwfV9Z45A8yPki8/JjeFmWLxKsiFSsEser98K5tQ6RSWzDy7q4VLI9TQjaveYaOH+WBJyZ7aSzW8XbXC37/WDoKN5PCiIVgZCNH8TPnVBi73psIhRYViDPLsw4o9osuAuNmagEWKgDEdgPhGARMvCB1SUvFMakl21Z+Awh7oiNhiIF140sqTe7B79w71HrX/zLp+eT5fVLNQ24ervbvzh6oW2r/efiJ0+ao+oLJqj4ASrEg1JMmwAdWgIUy5NGN0SCvlKMSPnSpt+ciznYnkFRxWqGS2Oo10N5s3aMZ6Kty0WRysCOUwOXKHKv1cTihRKn56f6tR/0d0++ePxV//fv/MIvcN7yHGtZrGHGPftidVvLueJBwiU9q/OXlQ7+6b4LbaNfqo0IN7dfb+VmIT6/1e+2p9NxWc6iC8ogoQ8RL/xkUqs72SpRpT8HOM8t2Hodj87DcRwSYsiVdMhqciPTNyRzchvCFXjtFVlubFPC8vzMlqW6BNlVTSNGRa6LYU/VThe2nBy/ckbfjE67KVIrJs2RwtWDM8Igty0m1rqTzi+pnHbAQXXSsRvJAy6E2PFWH6XShoozh+MyfWKywnFR68/KIavp0VaTk9nGjhcX57fu3D7YOdi7vd956+3pxfzjnY+//OzL/lsPmCY8IpXxqQgrOUe4yCW+uNAtC2Fe2a/35Ejkchux4B3aPV/M5ryWL6bpObuakDjtO8NLaSQHzd17d+6evTpRScoVjjVIx1va2s8oqctR36hCOtjoBWGL51SQAycHCQ8FXCp6ekwPOhO9XUKj58FvqbNAAcsXS77l3ke/8Ovf+c6HL59+fv/WTmyklX48XaoND0b9ekLzUa7f6O0VWbs1vFmfX7x88Gjvy6fHvONYz+YoNG0RCqMLVxVzA1tUxH8Ml8Li7VYvXhK/IIyC5Wk8wNphJNAgQEJ3gWhuyzSuld5fH27VXz3/krH63lbt4S/crVxcjp9/PXjr/cp0Vu/WT+cj9+Cx3/8//R8GH//pV0+fPOj2Gqt5/aYzenzxk6ef7u7dObz7aKu/S1PUYX+t2wzBDS+sOB1r62awdZBcmbjTVC+l/v3qTKnAwg44ybilBOgMoqVcAVwmpPOzBKutbZk9lVn1lx/8B3LE4Gh8eSz6sSiFcLGUjWTSFAHc3Jpd6ZDZoGWQnrBMfLPeWkxXHfelecxUaFfgMKqvXM2HDw4r84urJy/5VGSkL7Du7Nu+6heSTE/LpD2EPKm5aAd7iDQIo8Wz4DQOHt9mPkJbYC+cN0xsw6aKvpuvIo5d9Hrxcnl+ETYdiveVe+LXBH65c/5hEOTwWq6MrN1IhNBpGGn5gkD01ihRrsP4nBdbyW9clEtdX54DEV1RFP3cKkQX2otMhja5pJDu5of5VW6G6wQC4iIoNr0J46MAyewCJA9ru3d8M/3J05+N7Vkhi167xjIuXTg2c42oLrPMODas37/oyFOiGMYKzTtXxzqwigXEPInRFCOPY06LdAG9QWDeuBtGAVDchgiFzotnSHddpZ2kXqjJrr3UfFuHyXjbmF5uI+0zipeeIBL48si0mSQ7TTxWXZ6flbU8RdAyTzcgy9A3onezcpv3BUJvwBSouQESy5qgZq/+95q7FcBvzsTZWm7rJAhyO5GUytX9baqyhDTIt+BWZF1+GW3nm4dZBvloxDbM5mh1WbTOsngBVkXGitRE0ERuZqeVQzb/MtMNKgYJAlBzN16vWYLyEI/I+zw43KP8GwgYTr4ytTAJvzYhv6VAvR7VRp8rV/UG7/3wV1RAPP3Zpz/7sz99/uWX7bJD6q39oR4peldoTSHbQr6n9KFOu05UBd5JsIk9Q42LFobhKb+hY8T/6YlEYdITgIFnxaSATcXgmf6/11fD9Vo3Ut2Fij0C61ghV0JYRZ4U9YY5boMLfIdIn2mK9mTyB3/Y/+4H1a6N25ODLXkKM0RlxhU2zYEfXA9QMEwf4SLTQVpghJrsNMkH9G0dhCOjQCb4U2BIUOWHOUVLcg6sA6xEW3mJ4SEp6xbOmBr7NralJFRtnmIRblArP3KBG20OH11crjemfEvYkOVkjgvkl8L/chDesfyK2C40lC5RIRNPUaFXsDo/j/jHdqO+GmlJvNykohhr4Whu60000cwticFetcTKBk2t9nwy1VLj7OT8dDRuP+7/3f7gO7/5m/aztvHRxdnp108ef/fkqPLgzkcfffDJpz8p+jaVLr1EDBuLTAuzzKOmZZXBbrpDjMFyfqlpR73aYmwmo8a2CfYKtlFKECCtFzZAiU4PPj7FDkBXpGUm4ghmOh3viV3r2LI5kxCP01YxCmaFY6bT3m6z2Wr2a+qVUludDNp0sRK1UXKqlYmEeRv0kJ5XknLZ3rCkIoA1Hs3PT1N5uhSchmHRWhxRacpR1q9YDWFjfApBP8wLUm3iaWGkrJHwhIwHj3BhKK1EPuKlz+TcljufoqeI9+irT3/atm/R8N4AYEbnVgs2WRBJNVRixQC/8m/9g3/xn/0nk/HJvjaWC7FW+Q2NUVzWmvp2dNpsb+/19m+1+wcUQbJFST2Wff7qLE+KV0PCHY4IurqHVqunRSTDeuEl/qhUuwV+K12ltuSQ4VcQFYjbN+te0G+lDM8lnEcxZ0O8WDejaGWbBM13/DSThpbwPBshSpRlbdjHGkbyTLbtdCQtaz397BO58FBrKYV+Lm+HKrIidwv+kFcOBFpk8IZLh+1H9S24lKeUhQhilaNwTe/K174zsvJpcx7f21wWKvPjzYe8L/QVUYvS8+XmyPNzj/JteVZEaTkJOvlVbMDy8801hVpf39YZANn8PG9fY0uwt+B0zrl0MyI3wcG+OZ/vgibWLcIFyvkWOpU9IIPyHpQ0k2Zltl69GOmRdDatzCUql4v9yv38ytMjhQ05+OhEFtX8yu29YG456wwVBDIGQX3r3wKvLGiuev2D3KGuYBNzaaiYk1GPd8dNYkfBgf7E4mWgY3SWC/1q/Gj19WrJ3DAvXCRFSzh+Yg2575sFeDOc/7/8a8Ths/9fjjw3/xOBeCGe788grUu4PmmGSDP/rCNOuYFWcNFfhCOyJYjLqpeZuReJYU3k0YGV1k9uRlwRkyYemULel9uV57rE6UC1IIoHFrlaYO1Z5el5Zh5YxvAXplJ+FAaYs3lwWVOD0c37zuH3tv/O+x+9390Wwmtczux8UJ9alOtlT0cUZRW6e19Ws0mcCjYGpzuEyxMCRTZRS7gv4jqBEfHQo3uzh5SunHFpKWeQ0kFuzcUHKxgKIbYrF6mIbXoJN3H6UxePtIXmKmYpagiw1dIbdX1xevKjP/7jX31wmxvEIconU55HsnjDYjW6OxQEDhOH9UgjcQ+Dia4D4lL9RKNy2MCyaIcusg7QbTOVwKTANkAKcIUUl6v5zcRbw3CroJ/iN9+5t0mnc1JknguyioW23c0ZMju6QTnv4zfvnfF+I9QpYX5ihi6WSZphIMqg0euk6M0NPTcCwc2NyLMKsLW4Ilc5yp0Jk3zDR7xBQZ6yOdzT4aShnpyccIGawtlk/Pzola1Sfunv/H3uaLtTIMPBoHf6/OleZcWstzcvfylAGSqLs2P5G/V+p0+sbi0aSy7leD6Sb2h36flSsmQ7AXwNm1aL0XTaOB+FeIt3AzFlAJmaxbE0AbU3Bun49hvnjZPs83UkMBhnDmE3fqxuTUM2u+1akObWvaE0EVKgxTc+rQtl8ggYMI24rVcoTiLD1+JLjlepIK+JgJsTBOXe30jfrPJmPB6zIRUfDSmPff1VGUboCI4TGxl9ViGMW4DGVblHMC6ET8GLnLWkS76Fx5/jtz/8pUF//91sX1Ftl4q7ytVsVbmwocV0/1f/Ruu/+a+mL57eO9ibjqYxKKo3kwsbNSjeWu8c3qOsc2b3rheN7b2afbAHPfnlpd9stlorLVbopznC9yn8YacwhfJJXSieEE7EGxUE/DVQEWWYezErzDNWfU4VWilKULH5k6xZxEgUDgisGb6Ei0Ztt7cDzYKHbX0oZXyfz8bj5WJ++uo5a0mQn9NLILeAKYCFOYCCggKcgDU4+Q1s/9L7zOFbh8s3nzZv/tLHby7cnN/c85trQh/5tad+c2GeuznKeuW867OgRen3VX5QcLK8CYYYsdfNEVb/rbu9Of0X/t0M4C9dBXMpcIAM+CVZZqN6WqH8NgvVrE2X46+ef3k6OsX2uKCTmBb8+vnTXenmGWMxCcov87JRCAzbLKIoFhws08xckIyHbh5kRcuvsgR15C2nFO0xBVpastSudYyG4Nr8ci2FU9D9L9eWFJG7BV0bTNG/4JYGe9hHufu3YZr35QGuf01d+fj/0/GGVRViDI5m/l7LiPPmm8OZwMC3sZWL6GWXYzKl/rZOS2SGZCSvVR5oEMWE3uMWGXNhQSDJ36M8Z4MfzjuKt7k81NcBNdWJZgPRPS2qxpuBEMOWDnfavJZZe5vxlFWKpNmgjuXIIpfXvHlt/Ba03MytnC3rXPi7SFv6MGh+9NajX/37f+f2w/tigeMXz/7kt//panLez+70eo+lCb4Gh1xgMCp2HJo3ATZ1EvqyzYKKetqC9SWnDVbYFbUmKllstSJkYtpoM3hVkfyFpGWn6OWj4NYsuMXcIE6rDLnADT6w27RfIENePfv6VP1MN/0rFhej5XiMyfoqXfkKqXtirN9gJBEZ3TvaEayKCHgtFzfiKqMKnweRHN5kGoE77hVOmzHEcBZzjMM59oHEOfMgg/lJGNvJGk8bfD83HlMuTKfcrrSWDCYX0fj/4e3PmmRbsjuxL3KOiMzI8cx3nmpGoQqoboDdIiGyRaMkSqYHPVFmetKDvpdeZNKTBpOZaGyOItlNNNAAqgDUdKvqzmfMk1NkRM6p3395RJw8594qVIEA9z135w7fvn1YvnxNvnx5elCHI8mI1YKNaqMjIuLZiaQM6xHKCACUAx9yt8AYG34ZUzQO0LSmJog704MU00HhLimtGcmBBkUlxpkZP9LrBJK+ujzY3++tr8FIZ/xaQf/klx+trqze3tjByO/cvyWi5Ep34dypDDs73/3ud3/8N3+N1ZFaFpZWN7e3QqatLdTWKdoXEwYW23e6zdIKm8Le3gGHrRLlHP59uiTsVPYJgFpksoxFNSMgnl4NTO1XdbY6nPZOcgQUKjUfGIWur1fX+hrjqEQS+Wu3N+/ubDGKcUvLmFExQQPVsKGW4fzSAilJ76gPYeOSHYnfgR+ZSyYT2NegpzFVV4ObJqo4iXVJbNkqS5AkXEkejZIh7VJ3CFfQlJUxZcbKxKxrUcLu94vjo0effbL72vtrr33N0Yy+QcaQBFt+x88PCLz9jc37b7z9+c//RlSb7OhljF0UsFNU5dHV+Oz42fXF3uLVw095zHcHGxs7t7bu3FlY3+48eJuhmOTRvaDMkxBZDGOP0OF4TRoEEjFfOw1kVBYbi96bPfyG3QzKF9pNwLcMTFTQA6uz2q1XcBqgstOh+p6zbvjKOZFsrd8ZrDmnaHF8HKMCUjc+ev740bNnzzgzRxx0wBPxmmN8iI/JnXlXEirgZeAzlUGTEgd0RawabP3yUPB+9dYGxb095LspZrSHdp+V8+r305Kr/FQhf32S8WzPyvMA7dq3clbmSZMy1LO2FRJ8uYq/O0Vna9zB30wgvoA5f7no//b+xNi2sH988NmTz/cvD+A7ioXopHlVtAbOqmhp05RKT66GpdMeVX6t1nL0oHiZLwxLu/IXA+5mUaN6HtvYxSIXPx7qOXuZoQ9ZGjn90UF+DHTZFhLaSjfkI993XpLdDWHSEKqkkhBQ16yVf5+HIAQwRVm5cUnJxLqRMnlMangk0sgEHc/da5tTKXv2vsaMXDhW8oivta0YsI9CLjOoQfrkaZcRSfuDuZTF2lxMaUPpLQYnnLQVkXjqT1oSiOPBPje/MPsyP0wkNcOmyKi/BiX1pDppaWvNijYH2mC4N6Fognw6Hz8RItjhKCfeP3jw/t07iSD9+NHwycNP//aHJFt6L+PDxlp/ZT5nqDFjOoib/xzM0suiodY34pDF1SxzLR5PUfUTJoWoYgdalMIYrdEIC+F0YgGXl+eOBkKMrS5wH7Gez7cOJYWdRjkBQK3GWTo+PrJaiNU7IOtnP/yhMDqqe/T4i93PPr8cja3/E9u1rfUtNKX6jnqXCBBtsuCgVUF8l0mXH3WFmPpXnM+gKznW7BisA0ejkruUPKRdiG3TDGpQ4w5RXzXzrCwZZSmIIq6HM3pWtsSgba2nsog3QVGx/mlNTEFVPFSXu4h9K7m1M9xdoVZ7gDYVVrwn5S8rvyaspkmUjVeumG5UPVgK0dIek5rtwkLhtSDNZ1989vDuaw9eu//6091nzl24t35bTKJPPjp+9OiLB6/dXVrvL93bef98lH0j7E8rKzu3t40ZynE0OgpsOcL1Voejk0PbGa7mtrazz0dkaWc8sIdRgsnNeLCeqtXAu6dDWgMQBYu0HqMyRlgDM6aMisjqEsxniAn2FMbKh8Pln+l5Phqyf633Fm2fe3B7C9PqnI9F9qal11qMCExAYeEsm/5s/jo9P7BZYnV1g3pmjV5QHw/WxMAo3Kiu2fz2awLWF7NmkkCBKQ6SvQ0BaVoTtMqsraHOuJVV0lyF8TbuimxFOuv1e8d7e5/86ldvvfNt3HVpkeejI5cJpM4B5hqxfPnw2e//4J8+/9f/UqBdwYUyxc7sf9P6Y37P58e8FfhJWD1ZHp88P9n//NnnywI777z+zsra5vr2rc7mLfE557p8n5eWE3fBDiL0Mwe+xTH1egEppcZcL50v9hYtQMQ+kbhIVmUBHrnhu2GJN3JLZilbHppmjpdADSHjPQIqBipQHZ199umF4MI8SpYXjo+Onj3+gns/26VlKZyeqAe/TbOAp3CSZRvAMoSFusY0w1ojGgQoXHVveFsJk1sbmpY+e/sbHhR783M/9WCaMh1EE9csyFW0NwiQql2RQ6ZXwwKJEkIGkmlCIadZfu3fNkOL7r6UhzAC32C4/012ZD3r4CBjjc55UWfDJ4fPdk92RwKgIf+Rj9oVkuULP/wJ15gUXe1vWaR5h1IFeeXPqNdqYrhAeq6b6HAQNVf7ctFRoy3CDmE6caCQlZAL/18NBgPzrKxMBOhyL7Lwa6FD0CQWtxWH8EYdbADKeL4AdCpwmQDT6lrCb3nXwgxI5fb8CjN+tZCqGEyMnCDlWAWBz8om1EVnfKun/gG8n/gRPQmnDFlRUA18CkwhSZEzRCqjkcOU0K1ISBH6uTJVOYat2CUQGz7lh+TVeIA8clINzxhjRAoM9NWiqPAfr9MSIy5DGwIps556kp4reKLQblcIqNjuLGpZxumtzK31baOx5URUDss/jJPO8bD2eWIjBkk/YSjszo3OK7vW8DerGafICMPWMbmi6pCi4AmXx9iseVPOLzj+ivPSXu3SibnqatVeZPtzw3z5YtrYnkX+aJvckmOdt1Q2Gv7Nn//Z48cP4YBTBx5/9pmt/czR5no3BvxI2yhBHjKGehqm6QoWVrPyTBqK+ZQsEGJT78OAjaiXPCrCuIP7xYwznbU7V5SFuNnhGrGeZ/SC+4pJGTKkz1W+B5cUKoXPQ85iG/RB5AlojKGlxDKZNBbKD8y3uE9ShaJ3ZTcWehrXlVkJrTRVIpoKd6XuqtQrD+hp1t9LwPWZGn1L6MFN7RQi0SZ4ct9hoGtBo8hIF2trqz//+Yf/5X/xX3zrW1//zu9/s7/9ruMXP/jgAweyyt91qPf13ObO1qNHj0eHJ6vdDQ5ZRCk7s5XsrEZLQ1edfUoYCdmQMVxzJVNzhtuoRdqfQEYLG8wbKDSydSHAAsu8ZfYMDFtfYs0F4UgQNrCec96PrHI+fvTpRx/+Ve/9b3yt89b9672PEu7HQShW++e5K1EJHci2mD2x4N11jMoVBvz8+aFlVnIt60L4ZRuvGpFWO7xrlbpPUmQKFcy9WG4la0qyEr3D5oxd0EarNd4vbgbwysPlAl/Ck9Hpo4efsy52N+46it6GXLS4x0fZqZe93pNne9bb77/x5vjTn1ccQy7uJ8QQnk3ijVgwDHqGCerBacI1OoFrYemjnz5a7Nltu9lfG6z2LYo7uGaAtc/hxwKpgkJtg/Y5GSW7zgxQT8TZgdOLNb03Ojs9cg4YvOr0hB2dt23KdcYVQ4iD6AmiSae7QsBe8Q0UQUP0b/Lk4eG+mUkG4jY3Hh/z5gOOlZ44lFbckZrQN6DIcNUsyJC3yTXhHJktjS3UmwmQG8ylfPkK3F+eUC2lJc6eX/kws7GSKkNmRMsAiGnhdHDbg3v8/opi3MzvuTqTNrfqprVMmj39+dv8xQXpIAbT0NZ4whlkeZEv7hj3/eL556KAMz4fE49yAMOLKooHp4qGhvXQapx0MTQ/jc8bwE/GQAztK+lhQo1aZmAJTVwUU8GAozTPxdYdD3MuBu5TQ7uyuT5P1jrmX+m8jnVGzXxgz0ZCFkT3rSmN0UlbCSxvXHL+fa/WYXxIQ4M2f9dVObHAfCd/vLHCd3g6GLZIAARtUPEqE/KKh5aS+XSGARS3m+BE6UxwEsbUemoCMMsR0gWcvFe4UWCEZ6VR2jWP1FrRMtOr0qZDpa3KnHUhlLddXnjI+5cvWTU9ekma1XSU5HD2Jo+oheWeKsnyyL8YSeNTLiVd61dbPccgrBKVWrQvdISv5bJMKFGC0Jm/5IMyxvqrPYqPRpoe5ZQz3YpOiyerFqpgwdRg/cTLbVixUCIWX2erZzXSrI2tjOkLdY484sDvuFKy1OZguqEzEZz+bNe43wLrXNsELNoA/i7KdbLoFptag0mwMjwSIoURStSItCNAK6YbBydCjWctznR1RYfAtvjrExaiAMUGHoVYUdgyqJLx/SP6UOBEhaxii9p4mX+Ro+LAkhg/HiJGlBKM7Cs/4A7Kp09paDxsc6VuukgaqKklL6tdnhDu+lfuV8zTOqAObZE1DY4YpTUvRhwZckUYq9bIjKHbB2xIwFMk/M8//sRGKO7KqPi3v/n1w/3nzl3BO7k697e3FvpI/LGlIssNjOZ8NN7/2gekrv29jzv0HmuvK72BGE8MuszzbEGOjbKA0e3DWEvCbPa26wWekRa1a9Kw9IrOUUQQ89Y27QeEkMyIHqEz1TNZ2ujYPZZ5Ix86yYGJaHhydP3Jhz89O95bvDh9+66gy3REKtjJnJBX5C51jxEY7lnZ2VgnrZ0Pj4+FvOZfIIZpx3mkGdOgiLq0oR4mjSwkSUKDm0wZv8KcInGSW29yOFr0mOTzh4tvsMtvLgk9oemPIkUibKOR85OdqeBAQ6srZxq17BAUVt94269gig/efONHP/tRO3DaeaGr/XVIY5zPzkf2MbBMmzRRT7FH/vCFKxej3b2jj54BF4f9/qC/voUNDzZ2mIw4xyjW6JidWGrWhDtrHTE3ognRkk1ErvZHo/19awfdzgDycX0Vstrx3bRk2uDx4VM7UPjiAiIEODw8OB0L6y/yl6l2KcS2vYBpvwB21rrBBq5ibdC16GC0LsNq7GvJydSDoIEn6BSKpiX5FQRucHZ/5QqU67qZLqH9bA/us8GaZZOC9DQ0klhlJKNPW+ZYLmXxX6arBoQrE1ZbW1rDqiuyhPukyakonZrV8pUPLcOXMmmMj1NUpnCKVhogsR1yuTr69Olnnz1/eEIst+1M1JUiSmlwgWjangAqTcmtcZCGvX5PMNk7/9IhPdKI6nBqS3IocrLWfdH2xwzkiXjcR2I/cAvM4XVO+1hZdv4K1hqPS7vKaq+kXiEcuC9iiKhlQTvgiLSG9ReElNpa86W+V32/1Q0BDHyrhDxo6qyHrxYwGYkAQk5aBFLDZzBO2kVehTYlKOC64I28ApCpqrQYZIp3KjAYCghBhdDQwokkB3d8Uu0wedkWokanJocwZkxwdwfBRuC0ozqIL7v/C/AeU8TkAqeXuxC8r+yVXEiYJgYbqrfOwfWpfQ1QhFnJBheTfv3u3be++c1Pf/Zzse4RkOHZ2SoTp24zCI8FtWE1cUqC+ChdHEfUBjobk0gYfHYlVbRONIEujwEzrrGBUSFpEBANlzcZcloUnUWAo0t27Z6YATm2FvJnXQk/iOZHI0cAWJk5PZ9f9HtdhOx8yDHqKrESl52Q1FUC2QDmoeXaCUeQLaXAZLyPGFB28CINbG7hnjBJoi+Clu6WD0JRw/5qDEo5Vrm34es5FMwwJU+7auyCNRbcStUP3Gez1Nv6KnqsRAso7ulEBAvbR+MgI0NWWD0RkdGy+Dq40kgVVXW0ZFPIIFGGciFqcro0K/q7lOlMlehDY+7D+jZjLUEtSQ9W8yMT3vWMtIRJipD19uramw9eU2LaM2cL0uaDe/dOR6e7H32KqtPeOIdRgodHR9TcN99588mzZ598/Pnuo13yIvKtFofl7e3teat881bJoR2pNOwpzdOGQjAgbm1od43xuba1RoJ59d2fzIVJf81BmGACwXZAyCDZ58auPNdbmDt48viXP/3bBw+2lu9ansxWD2NFFMIYeCiZO3yvEhqiuMhwmEhqQnQETUt0rlpChlsD3G9eGinDLAUuRdZJZ2rGeIrcJpfO6V7GS//IO6a0yIJ2fTlXhox4Kv7UopNTLyjFiJpFlyVbk4VAFxH9/NxR6Hyhne3mlO71jS2ShtV0KMZNlVZC2HVFiqVeZ7HwjAd/BL8FJ5wuwqEoJCaGiBrDw9PR4qNHziK2aX0VqcRV3GEaedrpTA5UQkI3tnb6GztQ/3Dv6e7jZ0bwcrxFvtzf2zvcP0BNxBnlk/P08UenZyN2GhAwgoBgZIWQtRDkOSENiKY8eNrehTgZgFSy5X839Exzbf/lEgM2+H5hpnftscCVRFe+KCx9BeAtvb2aZWuJct586/mVDLNscmqGe8PJljO1TvtVTcZTgoTtam8zqq4X9Uze/uY/qvmqjyQWYIJBKcDvRAydnzu8ON67OPji4MmT66esJiiiZZM5Jt6oKbK+mNlF9qe87qVGhJRP+QhPkdRgslSi2RL5sGqviqccLYtJVg7gFvicjDQbjbK3lJ3jyJ6Lc3FpLy82Nm/tbG1CgkdffBb1F82yWbIrBqEDfcuRO2JVoX9BU+UBa8QD7Z5cU4hMSFRDpunL2V/Q8UWJcGl//qWk6A4TBllZ0wcFuiLqyqCfFPQAPUExdcckA0VRReyRjyE+ahwMsMeaFxCqFAqjkDbG6KuiUmTKTbL/cyHcV+eRmqJRmrZmHmNwPKJNPcH/QqxjiPY1Tm8TlIcoG/6E+mp3OEdGC1SrxgQKhurqzRTIQkgbGBwoSWikJF+2OzlCgqjLRh1iCnLwtT/+ZxvLK3uffdLDlg/2jp48tNbnOPQjC07Xcyvrm87SWFkTRO/8+Gy0ubF9cXrBpTM7iZ29srgQVdUJot1VMYEEE567ODo4Gmqhc8kQVEyTac2GFMri6OKiZ7/pKsPYSjZznp0S761jnY1tIV10hMPweLyy1sfjiSGIEudhrXe0MhMKaoUOGAbqqGbTLbll6hOWlewoCSusUQHNQDjbbYVOjOZUy5bZNZmwKLS4+KDEPYq+aL9QOd3jv8Be5hiGJM745Ai1R7nXdTkF1aAu2H4BsUXyiD5WK2JEMTtkoLEUUYqpI9pFAbWLw0JaCknAz4ysz20URqFYT8Myi3BEeDCc8e2KlxY6yCzvboLKw/ACtoYY+4wirvjyiFaWrXxXWGG2mpzzXATt4XBEf1Isl7cVUYTPLoeHRxd3z3sOtDg/ffzwkcVDPuI0Y9NWcEpSsnEsI/PS2vp6Z33zajh88+23jo5GPz7+ycnxSHRYVUM7U0XbsHPs5t69exrBPkVWUhz4rPV7ViV1Fg4qH7Y66yIGORPDudEYjJMDGS3SX8DK9nermZEZiGmZIhbS2W5zGoC16ZOxeB68sVYcf1jeBpc/+du/+Vr3W84BZK3x3VnnxP40VtxnB8PDk0c7b71jjzaF8HhsQ/bl9tb6wdllNlUXeDWvTcbMnDCMci6pGVmTKHNKOoyJuaaomwGRht1niagjvHUiu5hdyoN9EYuzFtXZPTq+1Vvvrm083zuMbG4pem39UGgLWjCX/6srMs3aG7cc77bz1tt8rETxsNHNIgpJzeTnanFyec7CpPsMAGHAsQ+ktWq08VuEb/ArxldiCfUFN2ZxPp8/ifQbf1uGtSMkLIGDBkKP3Nq58/Ah2Xj+B//8373z3TcO/6uPBCG2jZuOe3VxuLmxYB/Roy8+Yvk6He1lKdh6ElSMlyHT1ZJ4G6k8DgqWjRvZmLAEUxgk6wKwYmaw1unFcd5B12qDej7M98UgjHXMHnInw/Tys12trHb3UuIrKe2n8fJgpNpPIyWne/5Ui6q0DI1k/5ubVVVaUq9kThuiyOSq1tVTkyca27hZucx5P721nymqfZXe6U4ociW0Bz9Q8+iK0tEQ1dmkZR1gOHfy4cOPPj78QlD1ccfWbIdM8w4w9xsziPBalyI9hi37GWU9P7W23sfWMH1T36F4uh/uHalVYUZkAqJWXJyZAwhUnu4b81EW8K09w7Zd4YJH2XPJ1LKPQIggbdfvctcaMLoHx0xdfWHxCvI1Y0IrdXLHR1TW7i+9+Lt+1FclpKRl0Ch8vBX16z8N8QlgZQYXHAvXy+KmQFeRg+vDgkQV1X63UfHhBP/yKhpbENEHwBxvp+LuQKknys4amAsDpkyztmJdBhEbVgjenA8Le4J4gWPQnmd2aW/eVsqkdp0rlSQZ4ryteWFPvk9r65b+ZKCTZgy9Xdp+/a0sH56OLw92j7c2H9sx+vHHwulcDEed5S6HOqbi7qqgtFfEZns+llcHi6sb3GLt5a49mpFodAtl7IrAtcxnDSGbG9sJa+uCsJeqzjLXpV0vfO8MuK0+qzmqL4E5g5Li/46H/G6vbMnVIqiT3WvREZYTD4tiY+uaqPhItMVZy6BUbh231hYpCkLGR/fSYTi2pyQAF3Ne8I6MEy0rAg/JJNIbiKOp8UuIIc9lhlOk3E1td5dKXR6koCNWdjLtS7qSji54BeZS3F2BrA5OSQl+jMHTnGNbrku9rbTK3AYhL+pnyvF8s15lS8F37RBQY/oXzInWpz0tp7ee25GunkUv2T88yDncc3O4rE9IWg8//exf/j/+Xw9ev+88Qgjk7ITj/SNSE7XLSb4/+8nPBZrGU2EfVu8TQHa20je/+c29J/tYOJBy5hDTWjufJFZ2lpl1F9BisqLJEVJKHWI8MWGRigiv1S+faPmk2cFw76vjRqJBLCYMRmmyVqYW/OFQRIQarC5ubQhAmeNwa6fp9eHR8cateyu9nWycOBTS56h32RFmldtCkLzkHWXh62FlphZ+XpfaXa3e9K6u9qoR38lbP2C4hhXpixEupq3rU5I2dpQ1JxzTx4nwB+u2NncOnu0fnYhPfmx23Ln7OkF5NDy1p7+3TB0+0+219dWT8fCKO/HC3LPd51ZxUENn0sED/9vgrrw0NnWmwZBUY2IthQlAYuujuQkogJlVqkzdgK15MGhW8Ui9Jo1011fO9oXYH9o8ZkA+/NHc66+/cX31fPfJUXfpNfuaVrbJEedOMj4dPzw/E1wlnFeggkjorFRmGgHA+YRcTYsokBC8NdypPjMXo9XIXPmZPx6qNQG5x/YcMDYZxYOc7Z4i6pr99DB7nr588bfVMvvdMrd7pkDVrPQ0Ixc0zGSUX6tcHmaFt5+hwAXeWZlffph9Mnv15ZT2apY+e0gj/OcOGnzzHKUpqMz82ZPR82fnw8PO6FisVGHSUCBolVya2JpUA5y2eZDaHmZNqIcX/MWQF7R1MzSe/JReT4Zi8lGKTSACP2NLu16zg4StxtmCJsj4+Hxvf4/VMRzaTrijY9nEQmomu1KabTuIZ9AUcN5PnrUuoP37XeG76ZvRibOFoqoLVVgDgaTqTBG+qkpt0SOJT6EYmSeUlViMEx71elmMepI9ZSijniIjGr8YEgnKa0SngalqbcmcYlIbwttoECnYfiS+Pxc21J8szJ9iIDb5sQ9b16NRBMMSQisdSItSsO/VnMYrZlqDbrU5MxtIZERGdtdgh8tr37ev3SsNm+ks3bq7dOtWZ3iwtDno3r593Vt7enqxf3BoS+a6c2+6S5bVFrJCvCgUznx/Q0lkKnsZ55a7g7WB+sj+9i6U0JoFXYQgg7gs7h2OGBFBvXghbel4mWBO8aQ26CPS1gUH406rPB4l4NTptWMK+3akCdCHB/fXbU/tLfCjvp7bOzgcn+/n8A7dgirZroygWRZjm8SMx/iqMw39sxgWXxzjtnRhAZOLnxYsXDvHVGjWY4OmqQyq1j4DF8STWMTKrRwUF5hCagOZrIOopXrkJ/2Y5U3OWvr1CfBnnIMthWCwYcqAo/XCi7Szpl1jtGDhd1CmxhGvxbRQfqPMlu/KcOeY2zh2+QTRR6wruzoMo/GTX74cm0GUY8mHn8QJIbK8Yg1V8Io9xPO8sYY/+ou/fPj5pw474iQy3Dv4/KPPVtf4Zw3Ox2e/+MkvBLC8d+f+6Gi0bHdPIjnM3b59u3uxfHI4Yu7pifCxOL+zfVsTDsSN5dpzPNRlxlccoqzuVjNhb0Kfpdkqjk+/PsXYricZd6PeQONPbM1y6Uh0a7MyImZmV/4BuT0QdsNanXZquPUIe4+8JXLd6q2x6zq3xxQ4GY7Ojk8G28u9HRaF7JITNUY0J1ogZZSHdjSGMvbrTgOcv5pnwqhai1pDklJXnFxCICIma7G2FKnD92IMC9oY2ZLNEim0M7e7ayNvd3XzznDfYTP91996f211y5n31Gai4snhEZXRrBjuPV67v8ntijTL1wB0ratAd9VDWkts0bOjyNbU0LCQEk00+hlZc72aWhBDAMDqPEwwqz0gmp7BI9+enxw9vOAxWfE2xLf81S/+/MmjD83G/f3Dy/PdfncVugIIb6trZ/zivvHyx1B1LuENUGzdL+KilQhMDUYKV4c8QVtPXoAbzMvV8ldKJoB0r0FRg+pHA/Lsni9+/fXr3kpXgupnd2X4oa2xAWjU9AraAVBNKpldBRxpMrVk3akrk292pfzpj8bFWzurhHq8kWGCObo+S6wHNxHfArS0Yf7qtHN2dDnevTj+ZP/RI2v5Tn1jpUGc0vZMh7TfECp/UnkwrliytlQf8s7V7u1tJehNeqYFkA3CBOiZdT5reT3zBKTg6rksYu8tL/ZPT64ThvT6cpzj3qBsODTst9ffHO7112giZnNiuoYQGfLGRSbTowG51f/3vGf4y5icmZUJ4EqDi3PWr9mt9aPdJYYHF7wio8d+kCVIxlse0fQtoVRX9E0LS2WZfNVGxccBtwHOpA2y5Dn1ZlknDzWbECAtS0tYS7EScyMHN1j6TJBbpxzROmwgkE31ejGpoxAagVZmjWpVpNAMqSzu1OiMUx5yheJl3Ot1mlYY6qeHa4eupECUM9EoO1tbt3qr7y6v/P/++3+1dOveu7/33fWt9c8//YjrHM/OpU3nFzJnjdgkRdFbcWy60GBIy+WpHagGD2UW4YwSxlqMCzjTdMSYloHFz9CQS+tmp8sJGOubUsZwWb+ckyc8EFsrtSN+LLW+Rf1dYSDd3N7orvaZkRd7uxb6jsYj0cwxNlTGng61XWHo6D1Wj4vx1ZqbH41F0OdAxD1nfOSwD7upzudyOpJxyFCG1SkB4NjPrdSaAcG0Ev48Z9pn9gbeyTPd7wt7/cRiVepZup+BapF4DxAb+Y9phxFzRpsmw5aiUktd9VW7+T2pyyftreYpvFGfVqySJaqU2qp8l9IkurdmcF+3AUHDrOmws5PzuotsEcvPHj7FEgRwpvU+u3h6sblJxsF0hUqxu/f5k7392/u37t01K+0TjNqN5w1tm7kkHBsY21MH/cEjK5Bh7VRw89s1Wu721Q51gSCctICg8Wmz/3Sk4VtBJgpf2on1sNyU0BP5M/Q1tCjW3s7t2wMMf3NdvJBVai2Pg96KxSlhmZxIsHjFh3B4ar+EojiOWWpfv/uOdvChdxIwf6OIIaaOqcJoMZ0VJb1MoF8wzrO2wQDtSZszX3KlecWJawKF+FgM8C4kj4twCk8uHl5Yl2Bc14trh+dH9+/cevuDb22sbp8wTIcgWiqPbinm3fXSVe/+nc7nH33x+ceDyDbQ4jSrWY5Lv8rScTOQQsYAIkXngpyRAfyoBmYrKTAGaQPUlOJdRAz4GlHH8/H4wMJKRbW57i4t8mp++ug5oVPQkN0nJ7sIimAyFnoJ9exCK10h74rMKkIxbZSKIqkxCAxF/c2rRiELUCpKqupD67Ujw+qp2IAJYxmtvgZFv9qVt4XwVUKmRnu4eZenOuVvMufP9GrI794eqs7iclmhS6ZZ/vpVU69mXKWnqOBnKFqjgdNyp39facaLAqfNmJU/e0ie+nyW4qGQPbJ5zgcRHsC5JufDp2cHnx09e3Z+eJQg3bZ5ZR1KiwmchXLV5AnfBerGZSW2rkybmL8vBqL6AhcmZoC0xegFFlWCvMXRFsdONTdjErNB3gtxdnhI5OTf42O6L+qAjvC1MX/N+ZVej0uWwfT5rFftuUmonmdXzejZr9/xIbirgAnf+00fT+DSsoRhlyEqo67zNFTWKbopYRPHss8KCckwZOz9DQQzl3O96NSsa17Lgx96DZUhdXYxBch54j9UPB7HOrX8Knph2aY4LcX3JoElgk/tgv7+m2DvFGdCv1NrWkGnUJmHzNJ6yp92ZTLVQGrLPG9kO1s0aXWtk+1G1521wRtbWxuPnm4vLPzen/xJ/87t5T//N3/7o78aH+yysolGd9qdW13f3t7aoAY/+vyzvaMjdLe/ssTPaBFTXh8YawDBLRiDj09F3shqJ6yAhzwA6MEoLLIbUJQNLC3GpUuSdAjBEl/pqHBCCS/3VvuDzY3VjYHNqVY4nw8Pnx/sjzmKIVt4OmWk349R2IHmtMDePK/RFe5m4mdcqVhQBIZoEyN+ZXMLXLDnxdwL6+XLjlLXyZm8f9m0iYEkHOQD/9RI08V0p/9E28xaLcpvKdomLOQMCzTEOt0GVi+SAWwJRQscClGriLwZZTl03wOi7g8ekefQ6kZ6NB1hCiPHU72Khx9Tfqm/UMIAy6CDjQAlQ53/QQjALE0gP33L6B0fV545WO+5dfhD5ntiifwnnYWeSIeX16PxMasFMcWhOZenFxzinz7d/eyTz52k+9q77+MwVnmvTo4KWYTKsV8ivAqYyX2EQZZhLLnTcZpsh+tWOB+Laye7UUtSIGLpVugwLhvTgZqo9ZEZcpxFSQlu3Jd9CghRk+WFrwDf7c1vb2+HAQ/WtkURSehgq7mOrO/1BpvGLucOxfVaQbGlNJlDqPKusCfLTjpep/8T6SzfGO3GgBtkAMdDg5t7uyS6PPvfu3CQGpS0xrBnxsThwIC1HmU2JYtr4d6tNz/+6OHnJ8P19Ttf/+4/vfv6e0LbXRyfwEOnSy31lqyoHpwdrW72Onc2Pv2v/uL4YHd7FSZxdz4hnhqIzPGo/uk+tM/MTz3RiANuU0GbwAj2wUfPkyUzOBOsrHb7qnowd8lFYrnLhYJ54pC8ko1JNvJzbFyJ5Qbi8GMIqw2NspZ/uLKwWmQEkkZgJpgXfikQarkZcg1p4kaBLo9F6LUmgKwWARB8DzIHj+vjzIdkMETJlfa1q8G/Pdeb33STrb1uQ9x+aqJC3EGJ1j7j8bGdp0lZhzJN3Wu0MyNapbm38l7WfauK9iJigeuVNs1SZg8yzPLdSEz1qSQQYqy/iPp7cvDkZH/37GhfmNcoaghNvpWlMA3gVNdAGQS4UbX0icFhkiFEvCZVainwhgUEK2owYEvLEFLTrhBDTVGH+WVyElcJ5qPa9Y9eZCuwF/NWSsKA49MRVU/xcTsEZOmKD3u5edWvVoG8r0JrWvff8Xc6GDU2qfJL+VvKJD3TMFzBuMro5hwuZ2JbQqNKCXx+xkx2KZZiRR2Ro3hwss6Gpx6mpU3HOKCr+dzIkDymhumuhsASTZ+zc56Dhs1Oy/KIyZtJytsi8WiRVD8kk+JVpCuh++3T6hceHnSUCExslRmteiRl3ehxiVH1TtEnV9dZlsXilQ1hGCSWVv7pf/gvHj56ctFfRR3f+/4fWvr7qz//0ydPnojosXn3gXjC7731+uH+3txf/dvu0ydcpR5//vnw8EAYx/XNda4fFvszJTiWxXUo7mZpKBWCEnx2Sh6vyOU4YWzHogpjh5BJNI+VtdVF2yUwD2uVCIkdMau9CG4Oyzo7Xd/YIARcHx9hq+m3QucXBXkRqacbP6s+mUVpWJjYeayzYufpqlGj3uenAOjFCkkExitHI4iBW4u1aBtFRaGoO00dbMHP7ATMkKgK0NYYnmffumrsJhmwTDmlLJcPV8qQoVHLmrnBvrpCWeL57vLFC/SQXokhMkqTt1XdqI9XMtD5/DStPNxkwOrisWUFl1XpYG8Pg3Qck3Hkw8zUz9+K+qm3sTrqmd1esLViez959JStEqARVI3d3d3r5xRvAqZ4+9bU507F5k88h3ih28iwZvmfE5b9yzYvUbvKjuU5umxdgYuOG+npcnUIQnmA65Fn8yk0XuYCj7K55Q0G2QJsJXt9c4O8RRhSINJg67gdOPNL3NqcinAhYAvjA1HDWY8BFxvR5QUp79bdO0IXH3H7Wu6yJTVAt+FQj3pndw+BeRKSbtBDbCpDBGK9gFKZKIKveRupChosWpkIeQeihWePDy2r3Lnz+r/3J//RH/7eHzmGY3R0sLbohCjQhe/2MXDFu+rf3u6MDn78w79cFETOvDofi26OaBK0SLlCcoTTmfFwk1yfmQmAlkKibGPDoQUlzGRNOAasMJ1ghmRZ8eaAOJwa5l8L7xFHfju1hHDuO801dH3Bdl6Hq1/N2czLTBo2u3QS1zkCFSMEad7XmsCIxXO2uFfQHc8u7qtpEWQiBXhqYkGqD/AaXuchxCgtl5LktC2gfel6Bf7t580cNzN4vvkT0F3BmuLBgUAEwonVRyGQTQafRPxrgzu9+9mKulnX7Hn6Kn1yvai0kCGwmSb+xgcEjtyrSiTj4sQpU6cjB3I8HXMK5DdhmAnsZayAUjVcCn4JRCE5rkYQ6vGlG9OCL9pbHQyONhQwKC8z78lnEbpDYkz4S/7zFo8O6EMmrNllxvoMFUPPEEpWSjoFPSfa2BRqM1gHrNPEWZOAaUqyZmm/y0NrdID7EhC+qoiivcUpQSAG3Ay8tUbOYgxmeeB2oashN6Zqpou7Bmph8Zo8qCnIMS0/OBIdKD2L4a5ZwvIQrhmVsORiFOScIbaOc4Dl3ZhwNdhXGYgK4hHQZCRSQS61pxZNTSPM6uRXngzpbX1eGFp56nduSlQKapsdhVkFw8SyLmQC337rnYveqi36y2cX3dWN9773h9xSVj7+aGF1fev2rbffe3tw985gdLTQ74qeMej3/s2//u9+9dOfs88KgthbX0XizWlBe4njOZ7R/iVVq47KTw9Gw5FOBgX7Fi0YOwxCrKvlFWcfcYTmEZIuCf/PBSjxcXts0ayP/TXq0IBOvLC3D93ZrQ3FCDM6PV9dmF8dOINpzV7VgDUrbaXjM9xmRw2GdT3mN35+gX1nvpjPjWMWFP1MowQ2ALT2L9ChM5wTtILoIJlv8CaNznaYlBjqk1IqPXZGxAuSw3/qLMQgQ0iEyCG0UUFCmr0lgVZmN+EXiPEhIhkPFV4wwudB/nZVzoxv/q9RVoLa3RvXl8EDoYeUsrO1ZdeYJQA9pYW6M+hXLAVEm5PftVMF7T6CfKiz6MO2Cj78/NHw8RMnKXHEs33laPcAMlwuzgvmwCMje/nTcu264M1RFqyQWW0M48r+/hViTMlDwV8td+lLgFGXnwFHHeZIPvARXwoFyqvT7F/OPdtk5FhdJUAwpPe5fWF+Su72lzc2FgZbxZ/ihTc+PqH0DESt7Pe5kCgtYtPy0p27d8lnZ0fHfYf0Ta/AKlMg88IVdJhMSdMm2fx0RboII4GZxKLia1n3lyEEKBjL5JCS8o4MzP71g9/759/8w3/6vT/845Xu9uHHT8Wq6iz3Oujtdu/sfF/eVceLbQ/Ofvo3n/zyp6/1EDiIdy64uVjX0ItLAvEnBRauhR4UrSs0cN7IKYqdRgscJodRL/SQv3piwtakhuVhy4kZcxpfyMWNtQ3OAxZ89DORN5YdwnQGRUh73AK5lVtet4fY/v/EF4riha0qCm4ru8ENpBGjbAoIeUoqyw67Q4SS5JeUzEYHbmWUM4glSNWwIk3sNzHd16tqbz1/+ec0eTJAbSzavWUGExcMag/uSdekSMVwLz6q6UGuCb2VIXMRMGvQp6W1rs0qfOmh5awJnuwZk4kC/wJzWp5JzhsYNUmJMGA+zDkw5lik9JPj58f7jl4YCmTvDOWFrLtFUiHyhNiWdJMmNOA0UPkJmJ6nTTWwNfbTtoZI1XNaCA9duUGM9lz4XBk6iyQABiwYwAf28MAq0654K4Bye+dW5MpLp7zZc96lBgFWuZTSJ0gJigRb8A065kmdATr8wNQ9sDwgfCWRpaqqudWZ9RkZbl6+uvlz9qzYgkYcEHwVHFLk5PXkAYwnCYX6E6CYpakl0o6dBrRhkXhCRMRjUHfMM1ob18VCUwVMC2llzX7VsPlVpcXETOTMYKOhwf6iEmYQI2YkYwUvjAo4XMDgOpxDwqLV+YZtPLTEPaVFjo7tr3rRRqy+x7E1YcJ9Jx3L7/RdiXBCFCzChQc2PJowwyHyd3RycvvOfR224wi9t6T7te9+/5vf/8Fx1qW4MwnTaxNE997b74YgLC5+43i0t3+0S2m2OuuAxK4z0hy3dL4svg8PM7l1sqYpvR5HdnToErUqvdAZtOZaKADx6Be5aVKME5dY9Ez8tGflgjkVv1x2PqDtzEIA1xxHzOcWlu/euQ2pdgb9tx/ceuPO+pv3t3ZY/zpnz/d3kdT10dlg5+hg3wpiTpo/s73q+SOfRRbMcifQGAHtcsrL+bJYXrkAA1sNHYt8RYotfhmFb3YVuoUWuULEfFT/lx6vpyYltzKlFAXJKBJIjV7Skchwr6j4KsL4g/YRBwxwmoSbkVx875dzcJzvk8zGPMuPC3Y2uazw4MGYt3rxIbVQix3GUDxsc29v/eHnXyxdL9EmDbCemFc6pgTS8NFwpEb2W6c17B0dyvn5Jx+/8c4btzY3DtZXP/n06Yp9YmW+Ah6WDJejDoAsoUXHiEwOCtBvAGHwTSyIUppCiDO94qtYYAwcC7y6HHFBO11IQ1t3l1OG3vzCYHl5o9e7f/u2INUYMLzcPxzauzYYbC3cvusQicvn+2eJa2oHoyPEenO93klOnYmrgB6Jnbi6sRFXrIuhuRGuXhPGvQ2XnnvI5PAvM6r+aJlEKEKwtqAa+YayAR+tOmmXPUiJRGuuKdG0hqd4z+X18j/7p//iP/hf/MfdzW2HeBztPqayLm9uX1pTX7jsD7qHT4bdy6X1zR2C+uc//mtxTwa3VubGjNR21S8RYzRvfrHHZJOW1LSdtVYdYXsqyd4HSGfUgp95Rvj0Qs/SkaJgSUoP+awZelosaXR4fMJ7nbUgFoM5yzg5T5M8fLh/pKMG4Tzxf0X7SU1VbxUdGotuGbwMWEhu/g/VyKV6dReJyU9QS8210FDqRPRzEwR0QwyTpV31UWqaJnz135bBfXbJ5xl2KcHVeIGSiW3qNiwZlzI7pzYZNFb9BNmMdOajzrAy1QBn6hrACFjVsWBAuvaioeoKSCVW65HBliMM3gCETUwuuJZG1qd5SEZ2tYSYVNn48vSA/flcyJuD3fNdZx+JmcBbRTkRU1rbYsK8ebWWvGgaUuJ1mhMgp7EohcWcG98UJqfu6ko1JhgRZMhdxNJ91r6zMS/s505GYy1LoDwLwjEw2plmboOM2P5oQmRhu9AiHFSVZilqHX4CKAvmsWkQZeUqPE8ehiD/a1wWUqoF1YhqnXYF9hNMkjRFoeRP5pSZkltLdUBqm5MK8crdP1/piLuMxXOTLyMUeT94qLUroZPGJrJkqDJ1IsOffSLwPPPVxQrqnobmVxve1o4w7KLLeoadqyXpCG41Q8PCBtwJtzmTD41b7FrrWRCBZ6FnIRMPpoJ7TzutrqlZ+aCZgqNHp8rqjbYVBMMi1NGuxqmry6E3iI5nnINHmEl/YS0tDj4xAAqUGOMrkAcl5tfWTTu2thzJ6IJ1rnknwlKir1/77h//SXf7P/tP/z+HjqR974Nf/vxnT3afDQZr83tDGyBPF2WxYzftGcl+Oppb6DtRzRnsoTviWS7O8XbGaskCWDbPgIHoQbe21tdXOesS1JyDCxFXe4uv37t1x37kzS2rhha95pcG2ri5sbazsUZzOx0fOa20K5CBE50XFgaLS3cTxZ47Kq6h4KvPf/Hhw09+tbv7ZOXKuS9LdlUlThcOx2+F0iHYf9QP45M1QJAxFpT1DKuF5/m5xJmZ6+zu71mSLHQhskBl8oFG2oO6dHU+xhnYLWBy5l7GAQ5g8KBqaRrG46M4EbgJzc+Vb9GmEPUFWaJ1ZIOZ4gyF04q5no1PWSiwPynOqV6xUatL37oUdiNHGQrBAGPYwUAPp/zwVx9SKEF0485WLQAdm5I+z2giFqjF5ZlFQ4us7ByCfXI2t6r+8Yc/295cxfWzadde7f6aaM/W39cGfV0Q5oJbHV8nQOh2b9s9bJ5y58BI7/CfX1744tPPaL40b2bqGIYYuS0M86cjMetuhf7G8lFH0sbc8kKCMEUj7Gx3524NeredzrS0vLPSW1/qWrs4OLmYE5dtY+t8S9qgM3Q8A2549ouHvzo5ubDAPz5dfO21Bzv3X1vYvNO5s935yU9sB3rzg7f/7K+fLVwezfEivuarlYEzMqHWsBQxo4eL+2aMrXMsJr44PQpIoQUJj8c9yUZYNkFneJKSOefmuzz6CAnjE8abuTduv/m97/3BN77+e6+98U09uxgmLEBfB7tzo8vRxWDOsfUPh19svL699tp25+hZ50c//cX/8F/dIaEcn6jIwizhIUFabe9mO0GOzdb80rhM1pK8Q1u5cZHcYnkER+gjZ2TrYnhoceYx6iOfDskR0QDy0IPP+U9HGopBEQEm8jpUFF3igwPb5GEiX87BoxXjK9RCzkYVQjaLibpL5EVmxterRMUiDdTagT+tud6a/TX9NVIrJiRHa+hJbE9F8+RxpZ/1FeRpKa/cbXdrGXLXygaZoHTGzdXcJkIllY7qgQn7ehYF0vqSB1I/eus5VLO81WUFqnA9sw6FSpfcU0I10+hFTAxmSKp3eucVZ3MpufyVvxpfP1NQCWNNJgiP949nKsiPLs6fnx0+Hu9+MX729Or52bIYNkekzjAsqEc2EVGlqKiOkPuqJVVLmulKXwBddzUDQVGy/iLxWh7JAq4E+lpYAJI8WbHyKl/pS5XjOMLxvlk8HtrWeSC6d17B97pMo8iVEWr0dALrqL8FeK2IxU4f/baVIjuDEUHkZXJldMIGzYeA0oBoRkA6kWgx7tC6dCQYO/1w8rYVEvY26XzSi0m/lEGxGpcypldEu3oOAhWbFyphue2dtxlpvjO2lVRTgqZ1B98U4P9w8tT2QoiquoIC1fC6q+lFZZV9UrMC0yPcPTgIrfGBC4EnTObIAbhv6wmQBBEVkjBANR7V+YxISp6MjUwTeE3Lz19G0ZOCVXoXlcy0jHhhUTaCEt7oexiQ4pMvG3gK6pFRMlbSSxy7Ojm9/fVv/S+d6/7s8clouH3v3oc//vFHP//Zcg5nqIUlxrBYHoNERjWHJvNnuzhf8jbiGZrCcoKkYbbZuGGsc1uc48m14rC6rmN5TjHdbbzCsbH8oTI5549PhMWPTZL2H6ZoHmKZ59er23fRHN3RTGHVBK4g22rCm2+9vvv46w+/+Ozxw88effH5gXNsLi7WbHtyAnloWrw5OnRh2G8m23OUmRCk1/0MmxcLNrgvO34nXBkyA0h1QE4qTvAyGODu8jp/gAjvRLMsWuY3ACO0+hk/7iA8YOcrcy1DGdd3c2aKw8FkdWlKqA9HYgPlvyjNiuchFWpqaMLV63Bgbtq0zICuC7o2g/FWkCUEyWqgDb6XI75YcbiAY3ih+JRffPLx+voat2eknCyvKJwJCzfcPKtB0gOaRe02JmZz9O8E48Rls9ta7f6hCmpJPQAGbK5oLar2L/RULxQr4hnM5UyBY1szWO/3NlYdu2DzlPhLK5d84Xtr3a1tmmXHuVhPH3cuaJCWg1dwp6Xu+vlC9+is07OXR0jSw9PzY6LdlVgYGz0wAd6AK4yr0DTDZvyCtxTQS2EjnTw9jpWVEEQOWRY1DEM11jhYDoq4Eq+tu7baPR9d7R2PyRDv3vvav/NH//w73/zOWn8gLvKYhbizsGzQYv86TzxM+3/7i8O589WtzbXtvvFwPPDxR788e/Z0I5oHM3kQsbUDxKPSTluV5LR2dskZxStpAZz3GJdMjaSgCJl3UkxXTfBheluXgYDhKQ/Gwg25wsZjOJErY5HsmXo+re9ky7TOgKUYvwwb4b7ImCqkp2X+b/nD9lNVNa3VG2gHm6vo6kzKmJSZHuRV3fPh9PlmeiSGyq8ZIaDTb5N7ehUc9F1pQTfFhNdWpS9Kr25ITRE37iAgJXe1QsP2VXqUT+ujaRnlApD0Sa60pmUAgQA8umyEZDPd/KZuYb3mgW2+R2ej/XMb04aHl7YQWAEaxROevTkMw0dKDVVViFnux43LT7Cshhd3yDxK7vomAz0RRNJ9pSUPmlYlpZTQZ3+CKUWMFxEmM5bv1eg4MVvtqHFZ1zGBCfbmvAmM2uiYD/0zgAo1w+PCifihvWHShrHYcdBMN0gPiRsc1g0/07bAJuuyoJTWaHK0++kbXWody5uXn1vK73LP/FVwcDTKeirh18YAitzgF0ScBJiKVBELmB4G5tO2vFRRUgsL2pg0Ct5yAEdLbA+TnyXpSscOufPy+SK741ycedHS6K7Ky6lKZAIzy5RBCmtjX0ZlMihtRNVSY5raWkWtXmOkLpdxaX+qo7pRam7Kr4HP2CdPG2+DBgkK7q2enA9OU91+993t1+9fj47fee+9d99577/5l//5xz/6G5PaepbRy0SKAJsGQGLOxEqzjovhWuI1/fSSqZUf04r4aD3nLwBvRWjhfY47c0hbcVbAKk1rdHzMnmxQBus74vxhGMejAxyCKzT0QeKphmqptVR0SRUid3U5GNOFd958w9ammKpXV4XnEydZWC5YRM+lv2Go+CCXWk7d6W8QLpfSHI4754xy/S6VQKIMfBqoUBogQ1B5BuX6Sk691mPtYQSYP7fyAqVL9wnEcQVVp3CX1voIyfYciV5FmEOyaQa+zCZSDuQZp1w+02VzyuSKRiCSdw4zzhzxL8jKQGPbZ1Zwo1gTFnzlB/bp4A1ISxfHVZ8+ffaLD1def+OBdURa8unZcH1zYBbLpl/yZ+nXUXoWgcfEzRi9HVYISmHD0eljRZ/ib2BChwcL4JkIDUnL9KSAu6ypMqoYKHAjWglh3R2sCslDJmNESNjxLocvCw2cha08X4z2D2VPZFO6rcA9vVXgAyJm9M7u3rNnz0+PDliJe6vZj16SjOkZ8GQGhEDkTw6HvLLa0hcsBlHDyRIT12GL50sMwtq30d9GqM7Gl3uHw+PO8/uLD37/6z/4J9//o6997Zubgy2wJMUIS358cuqoYhYERt/YCyj4TsCeW9y+t724teq0787zg86ec0QeHh8M7zhb93wcREA306BiW0Vn07jf5cp4p4TJ9fKvaeqNv8EnVzBL6uTZQ0Gm3Yh/DUy5g0CIcZXrl29addYX8lyUZlpQgJ+UVy6VVQNbGdPMsDpCmKtlbw+zn0QU6e0TE6NV6o4UtPyeFdzyTPvzamkt55fvqSWzZFr1jedXMqfEZNP1uldPYlivT8PDp1eka+sEWU6gEXEgpSGdHp0Mj04Pj4TdPsH2OD9DzUgJwFRzOnO+2vIVYJsW/Jv/NmgEViZ9ZW1tal170T5BB9V/xsXDA9ixUJlmJnCLfSoFPBGUtK2+YrXyICX0IQqPATfkND2KkfDizqcsM2nkfeqFSakRhROZVyUP+RnNOK1SekhemHqVHlyfDGQ1+itvSpMnZb58tb61NG/9LCUbIcB9VaN+pgIWabyqiCQ8RV7zQUbOj0hAfmc0J9iZWmqkcw8wpldLbG+lwVrQyFc6ZjVSR/Ef9IzxLNyMOVoFRoKVgy6YoM2l/jb5wD00KvCYzvMGn0JHrTHTmmicCmXzx6Xe9sc9m7PV3S55skpJJIoqrj+mR3MVbvzGh+JFc5i9Pj8z1nNrA/t3v97tPvni4ec//inlDSewJlroDCPpcDk1CMfBGLMR3I4WJnbOt1a+565Xay9TfzAQA0ubQuWPxdm3QamnfVyXbWAdHw8x7dXVAWagtTHX63Muux2XtKHsxqh9nJ6of+FQVoytyjj3FPnc2XltffDa22+ODva/+PzTx59/9suf/ZQylIXbOC5yc/dVYVFF9FABT2vSA6xWUbu3GjWgCs+QTeGVv4VRYaWelNRQknBp2deghNZEdTSHa6JGzyY/uVRq7Gq1WDVYNGuxzKEk8XuKeVpQyniNqY9N2Zo7QRSd5BMcv70Ly8oxemmd7DZj2QStGRNESnvqQj+SgQun039H50+e7Fos7C2taYGQ0nd7d/BaYesAVEHgCdE1xz4W/ujcoUnYcl5RKHPcXWu52vKgJVqYtjLaFPTTNj4cLium6bvT5K8tTrpooUJ3zmG3qz2LUfzbgri6GTp4KYSftYHxISuu4H7Iyvn6xpzDmlhFiBoadvZ877PPPnvy2WdPn+xC6hMZpZp9EXPCOgC+LB92tDkqUAtAwhHCCr5gOraZcGf9Fj99rw4PSGHjXmf1/Td/72vvff2bX/vO5uatW1s7XBFy7N+paAYJskbKzOAANd/+a+clc9VfvLDisbWjOcefP7w8eNqH3OKRZaaAggb9A1wA++VSkvii/JcomEGWv+4TmhMKBJHqan9qxrcEbwC+6TypR2orOA8vX9CmFV7tedGqFJR3Qat6zPv2KbhX5tzaq+RI9ohm7VVLaIn1nLetIj9fPLS5UN9KnL1qmVu2WeZZCS2lVTTL8+LbaSe8KuEjpccAHBBUy0PP/cgslWI62RYHZ86uzk64HlyMHV1+iPvie3ZZdtj/GWeYU0plNg/CGMzfMKYb49Wa81vdM3ahAZPPtfzXfbZopY3sXOS1ArijTyil+OWhEelPrBbVwSIKWAhhOaiRydsxb3E1fI2xMOZAMojPLAsm+mMGD1cQ8YcBtrVAhlaaEUqKotqbwv0Xox44VvsD2N90aaKvQnRuXBJ9JbHd8wbPL2cNraZJ4I8ODkkngDl7CVJJmvDl6gp2E3xqVXw5paUjFvKhXRhViZEGNGMfQowBxgpmIUprMzaUG5D0KnbTwMAkkEE7XupONSzFBwtrSDGP1mffFHqx1kjIKKSYia1DS3xTkpfWxP4eyS5GuMlcRYmWLbUmMjb5hP8BkrTUQ4hLrsq+cKbhNgaZTXOi1SeYRrZu1Opm2DOk5v6dU9Zt/x1YgUykKgftXTpv7sQeXq5YOI5NriIb254iRAcKrijMAM3VD+SOomwpc7G/unAaLdU4sRAEBnTvHAfGcCmKWQy5kG5+fa2/vfn+ztade3fpVQ8//eTR518w4FC9xdB3XEHC/UPGgnGNBmUvPAb0GoY3/sRZ2mpu8TmACqMJiDMwgWSovhUgbrbxloyoCfQGkAxzIk4cLhyrY7il0qy8JKgIK1HMvUL0AoAvGDpNfjg4TyNUv18xxmlTTWtw5DmVsyzOwjLBFA/USCYyd2CQiNGWCJdm+kr7yRIRfWjJTvs7Pnu+O+wu7tEwrzq8tGJmPyZQkX7qUqC/FhR7PTHR2I/joGHMFFfkJT0FUSzdoMdNsOYiYBTRihU4uMehgXLfme8vLqwSrzBgC5Vc3Cm8g+6Z7gQdMtH0Ie4WjqSmhw8TdS2UjQmB/KnL1gAdI9/vnj483X2298Mf/vXnn3zODcMCB883t6IiGmS5RMOAAgnqUVYxTpyGbV4IGQ/gfTbsiBUjz/bK3W88ePP9d97/1re++87b71HMkbGTofhBQxt4HFigKAUsc2I5P2EgMS4WpBecv2T3LSFJaL+z4cGj3fnTw3V7cI+JDnYeZdpEIDDZXrk0M5Oq0pMhE+mVLDd/Nprrg9nVUgqss7SbDy+XFqobUBSjramdCsGy/ilrWntxoJQTEpE8KcezCwLUc+6zdM/t54y2kCx9JIMUKBqWHGxIafDKm9ynV/uqlSaDbPXhpKOz59mDbw1EKy0V32hJ1TOpSHorqrKEDvrZ6so8qiu/NadeuSe9qg2BTWLrrJlW3DcZg9XxpGR8ZowSlf1K/N7T4fn48OxoeH48ssWS0WbyL260eHgE9uLEFWWgpomU3+HSpkKPQpVqoI+1ZlbUzefOou0QPC1RTDaaugIxYxHINyAYkyqw4ME3z6KKvpvOCehiY0K22sabAEWNNEwviv820hwQmJzWPmkyxhER8JMhxR2B8p86amwKrL7xwayZv6bL3rfu3chrPmTe3rzqZyoqvpX+F+9f4EDTxRbZ8uwBRFqRm4yeIg1Wsa+bpXiG8ZmQxQRkTg8yN/JhS5/cIy2Fy8MGOXDCmD7yH2oNwclYqmZlrSO+WfxYT3NqE/km0zp2yFTGFteQNYmBR2CoLH9Uaj4lgHMyJjGX57YSSWHUJilmXRhOWLui2TtDieFhSUwBkxljPtGXbEYKy3de7OmJGLSdwdrO9ib2LhPwOMMlLqbKhBG1hSLosbKM4kaZpi5k9JDErn/YsHTrhDRkFBcr4dFwcSLEv227F2J93NraxgZGozH9UxACjJF+DHO01pL8clcU5WhIvlRZNMDWtc710fDITuIsdurOyQl3bmun63fu/f66Hcb9vYPj508ecxTCGxiyj52P2+tX2AQUwNWESAWHvig54mW8sUtNj7ucAB+py5VMgANUYbC2BGK0WQLWGOZ3AGTStBfARgVTWn7jR4oKDut0+G0mctahY/Vxx1mAZ3F4PgryR2oKjsWtojQXkBuFwsKXLMwkqAl3QfbeXhCg8IyGxuUsg6l1IO6Phdxsx8fXO3MH+8O5qye9JdtGr1mwcCcl0IPdzejN7S0T2bzmpZ1NayK9J6yE70IxDXvwJONn3tpmgxuWuJYR8CpYlFbAAedS8GLgNVaGMZvKnIPZXV8TqEJ8lfhHZfkSLxW+/uTKNoqnz4SfJKgRSZyLxTX54vSYQN9Z7Ytdqbfg+Oknnz998syR9FmUCTyIBoFZlGBxG8jwQts4WkScmEVCIeqxQAOOO7rvr5bv9e6/9+4H3/ve9z744Gu2IZNMjg/Pdk8er3ZtQx4YcqME/5j+NO3SyUQLnd56r2OLvAMH4e3h/uHxqA4Ou+YbsrrMFD4U/jNCiCGvgQLD6WU61ySc/v4t/wbI0ClzOFPVvaUk7eaVGZkqvYX7mf711k9XS68M7fcsJVjR3tdDIcm0ZCkZvenlcx/71Qque26t/JbufiPP5FVVmVnTrvazOXPNCvHQSph9nodqiYeArqptbytlkn/anpR0s4T27az8vAoxnQDt5kNyVlUSvW+vnJeZzpMpYpCJwdlsPoMFAuEjRpdnh5fjgwv2ZxhgblijMmVN3fzL+DfynUrN+VB+02XWmN/poTqCnmdWatuvKwUDtmcBwl7a04mKkaORWRAvjpDmF/rVAKeLllH9URzaZvKFcghoZBHQoEVViMkQlbAfVfy2It3RgNE5SFYb1GjDYckM2eZmwgjBjyoxDUWipoP26xpcuX7bm1ldxQaKMD0Mb05kGcnoLhIRR0crmRE3sjj2AkqT8VYPUEjPb3+w34xN5AVIHj4b7ACMGjf38PMa+lYW8DF7oRxs4Aio3VARs1Zsi5pbkEje76b7ClFSTNOgRAjTvkzF1FmD17ob8LsiURrXYEh0mbQjV2PDvskABZeCiFnBVYsHMkfIV1iqoVW6I6+GJ0fWD7uUJ98fH3XEgTzYU7oO4Sj+WZ8MQ/FJfJ1ETViyEondlBvwnLVA5Hh1sNFnZ1zinxx/IdEIuQUYV4oXLusrO21svbDRgg5HM2FzhZMAA+fEgsou84U5jgj4YjqdjoRPNbTLT9RD5PG1NW0SOzgblPXbYbRLy2+8+9745OwXP106en44cigxAzGsTfAs5CcDBzkV1nCXJq0xMFyNAVowOztTkW8PNZWBx2OkSzwWCBXDA4kEAejRq5mwzJPyeAzG0upkqstfRV1SCevSAUWmD7HNGtuIsO2VoaOU0w1NNAOvc3JFKzVO/I5EMlmZLy027bdGnLaR2mJIz+IuvZ11t3LOj2NhP1rYXJJfBI+NjYEOeu6v9jBg+0rB3E5cXdUEaE5VV0Lqjf9PIBBdvS5LeEEuAxABM8y3wJPf8WDk8c4hNxvGV7v9VTu/rQGjZSx4FmnZvJy1dDk8WBiPHn72+fnzfbGOQQzSDUTH7ORggbXNdW6CBAoiBJuwcHunp521tWwLzvpuBEJjT9oz1xwGQtgSVcsOAikLAleMhevrnAvA1Z9f/Sff/3e//a3vfeub32FmF7Rg99GQJLc1uNUZzI+5FewRSq710UI/ALLo27hM3HUOMTB3xofX+8ePnj1/fnhkx5fYq2shXgvXh88Onj5N9YUBYA4IUwwMTgZBMkVdbbbV49/zZo620tynVyAddClCkL+zS17P9X9IWZ6nd4hRRUkoElotDCUKwTLdGy2V2hB7WteNvzqjjCpHatpTz6m8Kb1SqkZzKK8mDZmWUG2Z/EiXije0T6ZlVoFp56yWlx5m2aZFViOmmTMTZ88m1LS+lu5dPp8WjuJBp0Jc+BsygfWeVzgfbnuExbGj7S9PDs+PnTxILB5ejU44mPCLiPCY2EOMLaZZBrgNdUhHiPDvfqW/7dKq9jBrPDBVJyd18OAXdSPCtZkIcV12qJFqtaSYCNtbhl8pmZdu1STswVw1qa24ODiYNyeFB70Kjbhw+ppdK2cEbqI63UXmkJprC7FWlLBhXIfrk92f2QfI4bR6jMkVAZ60/Cv/aAaIh6TdmAkNoSt/TEONkbfP5as5FQba+s3+3FcGKTnigs2xETvRG8OonAmwbtZdJMnYz0rWccX61+DjVbDfPV+x5SXETjAhv1hOAzA1ZRWKs01yWZsMB6XyAAXCwzaNYaF8QGTziW8VngFI4ydXKPsMLSZpk4Gc4L1WVcMgTVJY96rNnPsqiA/VivJNRSfjW6znJNPpfPrpx8Pn+x+88/b8YHDwqw9/9Bf/9l/91//1YlQ+hsdQ7RBFVAwixrBRrkOWALNsfe3UYY4/69s7lnX7awO0kj1VtuVLak62oRw5cb1znSNNVxgBr57v72FosKvxAC3EdZDI1UFWMR2TTuEs2SXA1FuvXOy53aUBSXY4Psk6qrbjrji9qDFHR5v3H3x/dY0K96N/+5f7z56Bz9r6IIMpWmgYiJEyH7m+xhuqgFOQMQJEgwo0QY81NIFtLtCm7JYQldEFwsQbYdc2rHRduSxKgm74u1nBSG+0cHgmYjpiTOWO3soVjutOHmCGj3NZyUna5PPi/XisB4BOHy1XRwhgCYjgK7/myZn5yN5gFK9Kr415iV7uCJ1EFNFruWy/YXdlZWDVx4Bz6v21szn6ZB1MOnpnWhL/LqAzqX3LeqycwBh6So+fV3R14hmsyf8ZXv0zmkEA2nN8nTWLC1RWUCvqGdHbRCaWZMei0/qiYIydqPr4C6dw2GSTPawCKw76C9zjOCoLuHE+7Jxxl7PtteR7qOWg3JxjAWYloIOivYuJsuroWhPB+oUNaVna6s4N3n/9ze9//w+++fXv39t+y3Zr/vMHz45BS8RNw3m4m7PaSFc7G5va71kP0tBer7PVFeKqs/98//nB8dA512bbyubaqrNDyHMxM1IaeAiOxv3YMb6CAmQe/+5XAbmIQLCrZnNokNk5LS10rKVnmoNHfpq1Ne38mVzTb5On8rd7SM5UWzde3k7u+ZvH/H/jk/ZTys0r1AiCqLYye9Uq9QD9Wk6vZpcUONDS3SfNr9/yVEpQfZbBQ35WSkt/5e0kQ32QVxBu+nV+3qxgWmjEihtX43AlZdfkCg2HuzFGubNe5dxmJ+dcjWHpyeXJ8GI0cuf5bPNdx7oap0qhA/JZ4KDkaQPyGOHld7qgk2tGtz0rrpURilRvJXiegHdRZDs6ge2SBEnzi9OC+Am0haCKr0MK6br5Mk4t5qWVP86rdrkK47u+vdrftO1VSznGlhc3QppFOHPAOdI5p+/6zMyP386ShST7+An1GDDJeNxZ6FtdCoURI6O8k9ikSlJOW9PqmMu0oToQqjFpUiDT0uSY9OnVP6HAudLy6B95LgvynGDxGWmkLw7I0dlRXWEHyBBItfbnS3VplntxQllLRE0h5ruvNEBTU2prVfsIhYzPeK1JBJWDQnCDRmfHKsUt8QXjOSqmVAdtRY/BrY+kp3SqIZlGiaWFhDpWo1NlXa0yhWdAa9JkBubLwEIXo0p79H88rpBqp3qcLyGFUA63A1zLt1ZVUT5Lv+uDtfPhX//o33z6w3+zMxjsPnnqn+AIBLHjqxPMw5bf8+sEbGK89glGGjNvjLc5gZyFc+uWM27649OTxW5PDLUFytAZO98qvcQmVDWaq5aUWVE5IsMHLEH/RofoJrK/0INxq6uwyytrgdenjmpmKs2AAilCGcSjgdlixCJZckAYU9amo92v3767/+TZZn/w3T/8wcb61n//3/zXf/UXf+lAe6Z1Oo9Pw8ZstWQVZ1q4XDgdH2uAD1l8UD7PQMFgG15taLxwBZpEDpYJXj+YL/yzwJ1eA7oCsR/eGlgRrkO4FKKGzYhcGd+sYpk4icx56XgGRtmz9BqD1EUpXvFMtgyrO+qRmEqvlljBnH1yMr4gzPhOiCuHZ6yvD9ROWbTdiFmJtVh+EAUcxXLqwmEJvLt7z+EiWWE4HumGWqyCk0KePHykwSxaQnOImHG9s/X06VOBQqG9PClhwdG3iXanQHeF05uVYJQNW0BiA9XcHH3aN4EA8JCnexyhV+2cFsdPaPj9vcPO9YGjI/g+7z0/OOfXd3R8d2ebEZkken7mWOI5iuvzp585JaJz+9bG6/d++n/9vz1/ur++2kdPmN/mFsVDRm+68RkUUM6gOUHhanF/PFrpdLfXbn/w/jf+4A9+8MH734QuJ+NL8TSIXyiByCym4snRGLLogjUO9vbj/UP6wxprcx9tMVdPTn/x073dR8fHIxPcVmmDY2b7nwWGZWb+YoQN7z55zO5nzcUQpbTY+TIR2wRvz5VgaoOEMY60knkWklLzL+PuZV7nqcCVLHWhHlqoTjD0rkp+9RbwBv1cdAZ5fFLqSVKSuf64aXtRwlDmVpGXmaE+NojMl37LA8fqwdRK/syg0M+bVLSoHOwJ0kP7XFWL7yKk5k/alHKSXs2oQupFg87kcUIG/ZKhTeF6gyRN5YzQrxffJJcyk7WS8xkAheqCUZ6nmfHVPLZLp+pNmG2l62d9prRALWgH8RI5hVXZoi/ic2Yz20iM1rEjJ0/2zw5G4BRjmsVUB514RoV54EG9IqupuK4qL4P6VRewtpbkZZpSjVRARquaOPsq5SRTiHTyth+TPKEkZjUebOKV+Tk7H2BhABFFTo8IpEHxMJAI7aZ41+6Sbm/Due7zuG/CjKWSyMNO+F6yB0lgZApffCwuL8d0FkGNHG3HuysqzWLfIeics6L0Z9UTR2Qhin24QeCre5xmt2um1PrZetNmS72NcKGAiXxRSbO3pcxF1ikjZDiZWEpOaGBKjDBPQ0Xq27gWSBEdBcw+r8L+jtsUy0sH8mXAnTaWsuPgFQikzGyfR6KUTdW0pq7Bpoh2t2b7zihoLcXIlEghUbMn6OF3tGRT/+ZAm8AAGEUmwh/7Lhvh6dzFmVgN1+cjI1psiZStGg2a63z20eb15Xce3OHE9Pzx0+HDz9hj6kAJkVXSPriM0qW5mNDCnO0nSFsYh/PMB+x//GyvDo+HmAgGbC/bnB0hjOxZoYRR2DEH2ex7AUkIBrukQ6LTszFpzwXlpGs2LqQ7ZAKlEz7iQq59+l+1eaRhct1VDhBgH/jLnIOr+eZwulbs1dXa5tadB69tffIZbRBUNbqmQYYqslfYeRZjZokqzZXhLmSp6lAhiiCRyIvyODY0QMqLKHNAMQrB5/DgeFG7MGqzpyiC9yncYVi1b3c8d6Y3qR5hk5qgGpMm6VUq4NTmVKjwbHpkOsZm4DIR+UMKGAU4Lqygwe3ycqOdqqRl+GLJotZ2dex8xfq5+D114hO+i3mcncUQDdoqVRvtn2+aT7hoZGdUpJhAuHVLp0x0hNuAup+ejLyNOJJRiLqPy1pZdWLu6vqaJgGj72HZ2uq64IkLc8es3OPjsbl+Ykel5VhOd6IbG307dU+Hi3HRc5jE+NGP//oefYS6YbtE6Psc34Ht9XUGYzLJeGgJ6HRJhI6FnhBqtO7X7937xte//f3v/uH9e69Z9udFZesItXZ1YVX/tdmg6ESMLYuszXPD/b1Qls1Bx1YiYDjcfb67OzzcvbjYPzs5NjoYFH0H3c2KdsKWizUjxL1JZpHacVInFsQurk8p4IDzD3KBcE3tgPrlAhuLNcVAovBkQuXbs49uki+9TPNvlmAUMhWgUSZDCpcDETFjpnW1hKSXbaPkpwz6q72Dn8WCJ0R3+vnN2n6H51/3+a9Lb0XffPuqgnsDdDPhpbFk3/qwuEYYVWyHjB8YKoT37AAAUwSLFWrnwuG6/g2HlyfnVqiQJVYaMyGrv6mgzEEB52ycmvG5QPs7dL+ygvBkHL/0peIncC5mnGyIWJ8u2y50M+MY5TNdgyD+6VAzzEB4ViVHGC/zwRhs29DZ7W5YuYy+V5Mhxk0aBwuzyDDZ6wpLlqL84b8ijSTE7YjL5fLyEBsux47RYmd7bpG8uapBEbFqLQRMp83UxOljkHUilN3o2BSVbyTV4wzPXoZFuJxthXxxiYiAz9qu/bZgY3g4DaIVRM8AkyEpREChpFT95Utra7zavbUzi64pISJ6DashRCalxK148oiWnwTIMbmGAZtEqEk4rXWyAJ/N0D0jUQKraRww1ERR3KtAyKjWBX1Q27gCFffl/YIBc4Vyrgv1LSiFDWXS1hDTvE7Hd8ImF1c3Tnv379K6frr7M/uWLcHxU9LasBNbkuaZLXkIZA3VDF/pLnOqogmJondwdGwXEZGTioZwcy0+HgkPvbJ2KSZ/3yii71wf8Co72wCB/qe1TACEPXIcBS6sLa5A+JDlW9AmKoALkMSfzRW5FsXXsVh0dcI+ZGZQcHOApnNn7fG83tq59c4HX/viiy8++tmH/LBjZk8X/ZdCwmBIkDWSCmx4nXrJmFl5mcyyFB6qDuHZeGoTGWm6fWt6Z3JqGvBj/WllpkopOuDNCoxLqZXQEvezyzEHRUwoXJBMx9uhiVawAv/VJw7eFGyGIAJNWH7EHJ3WNrzTKyWBlSHA5/sV3ZMNP8zyNNuaXXwhZWZlYklNMMxrxy6PiUj0X4yc5Z8u6oFFBXcVjRKK2dnAiazsEU2aiUiRSzA1VSbghjI5QvkHN+1XJucsWCPtDnqDrU2Bu7urjkxM5AfDZN0XjDHR8+szsbccmDg8FN7g2KdajhcaB4Gpuebxdx+PTp6Pzw6E4JhbFTekv7zpzNOjQ+dLZHTWVjdX17qntOXz04Wr3gfvfuOdtz74d/7wnznOwwY38DwdnZD7PbM4jPbG9PDapZGYYkSQbKw6P1tftbyFAB9f7+4eHx3t7z93qMxpWK/gJB0CX2/JYVCyZL8VMJFHme9Z5XT+6GDXGFoT4xgRuSWT179gX90bMWlEwHNL9KalV5a/62awGp2oP7MSJp+1xGDgtBzP9XiT8iANGodOvEjMJ8FK0gWvmuB2FmBSyqQkGUJXI/GktTcZeb5VZEhu8iQbgjS9IpvW1drR7jO6PM310t/qRVJmD+317Ofs4ZX0L3/SMtxktK1qJbR25aFdIVOZj3HbiNeV8J1WP2plF+vlgGBmXNi4N7YAPLJWwjWezJV/ZMzI59Dd1A7VdUs/IcjkPgNCa89vfX8ZMSYQrD8vsZJkW8R6WaLIuehFNJCivLrGJgHD9Cn0sIwYRX6MtcwOfN0W48YEDLePEKZ0xWFjCeWlHOe8CVqJkvdWNhytdrJ4eLowvDhj46ILCxY3ZNRcWDyx4rhsxwcevLBWDaQHu1StxNbwkOVU7Qr7VEtlmTwn+SsvLWnlvJgwEpC7fI/cxdHVHCyP2dP4Z2XVE5Ki8vpcVXxluX9HIoFAI4tlYocYWFobbZeOpOnVjXCaubZNyzaN+ZNAjyBKsM980G8qqBJYpC23ujxnLGqKhJ+0tEkdjbIDPb4Z/mG0sJXGaUPSE9MQt0LVBRk9PZuHgUHOi87tOx2OJ8Nhd2O9v7P9YGdrb//w6ecPBYhmU/V9fNKygJ2DHwx51FHNrEBXUEUbsAo7PTf7oh3QeJCGrGJnaa125HAlEGYihs0TRkL9IW7OUX+FS6TVpf9nJ5z/PFD7WFO5YquiuHDhU/E2zDCohDEaHW2K4ZYQgG2B1oUoHwGoiCJra2+89fabb7/ziw8/jEId8AJqeh/wZfct0DRug+Nid9WiGuU81UV08mV4XGI2UzVxX3EbMP7Mk/BRNnEDEd8lFDxLLug4RddkNmpB0TDTyBxlkTZsC1zD1V+6fKniVZLCSB5eNIFGpwgC8KM4LmGkXJcdyC2eMw8LJ1sktKfTULK/KGvyXN0iuIANvn9ycMDOHLu04QgDNlgxLN/++KOPAMGHp84yPBOokvfZEkNFY8AFlnQ7XQuIrolfgGyBlpNW5J0ESrRhZ1ms7yVhzByxwWrMb77vXGFRpmOvZrZNMJHsAZ4XaHN8LPDHyaC7HE8xx3PmDAObJZYUtr9/8Ozo5ONPDw5HnV/87IvOxfLa8kBg7KX5nuCeo+HpUWe8tXDrj7/5R3/4/R985xvf3bzzoCMY+MnZ6ECMsCvmOWZ/vT7c21/vbxIcRLm+PrGCK9BHVsrpER1eriNq8PODveeWGwwxnOz2Re04Iwyl83AXRmeLUvh3VsxGiIBjqEcHB3tGm0GDAymdocHkt7pnTgfHvnwBaZBwAuFGygJnc8i9kttbKSmhscbcQ0AkoAUe8qScyf0FWWqfTNJhXl35E2YUUjzJWhQ8zy8Kqef6mc8j687KL0xojfPqq69wg8mbF82pD1tqPq8MVU6kVhV6NSu2Pdy8zz78MoOfZUupRdNbUdJdEaKNNM8QU4jmEW3RP4u+Zzhu/ASvTk+csuDkKSf/Mn8k5t7FeWLd5l/x7BADNATRb80I/LWYqJyH6YydvPtt/rRPUsjNS2vzs91vvFg0q8w6l8SMlotsX2axNhwSS3jSf7M/6m+vR/fdFnw13icQJYRZcDiW53qas7qJnaAUMJ6n1crC3Olqf31l2X7QvYvTQ1v0RTx1KOqVkznjG1nRpavHZRy2SNyGsJZfW1vDFF/tUntT99mr6mTjdAqcgHT6NnjmH/ZGCnaOgcVN8427HC2QLZTcFBJvVDPSBSl3jlENgW5UN2nTyykN6ZSQTc9hgYAgR0JVmPhilAQXEVobtGqtkcQa9p8jbwS0yMQBTG0Ly6txIotIb7NRQW1cp8KBrqU7L10qqtzS1ZimZ2xCHrmG8TQZXR0eXYubPzzikbD2yefG28lFiV99Ek1lZ2P9/ffe+au//Gtr/XHfh4KXUYXRf8ZiogESFk95fs6meAWgRhZx32x1KqujUMc8ozESMhaZnJcpAhlOwRg5OtYgFNriMWQD6ZjRcwkJa2YQWa0ZQxmBOeBfFiZLgMk6CDiG72B/Co3JV3d0zvZUzeBgwXWn019ff/D662uDjbO9YwNYGER6iEhZUMcTWVaZq3MZ5Cz2eRnNoG2cLeEroc0ziRPEq8Q+bc4H2pItr6y4cz2RnYEgER6ydqNZImvohlwZZ4NC5KFPQSacfr52OpEajC8NHs8FUWWTQy8uxWMzszJUlTXE4+rU2RWs02HsHF66S5Zgq5nUSP4Z4BYfZqxX5tCbRTuzztknWgO0QRgPnNi4FApfZle0f+cXQoDzwrBEjRhFDWdvr8tY+EFosARgKLOgPcdgjME7b3r57r3bW6sbtvewefXW19YGq0omDgmyIdBJzj8kosThqmTaGJCWToTCvrocLZ/ZJtYXIK23xsLDEwYr/ezJ8Gg4d3Lo8M61s9GyAxGWOhd9kTTuvPfdb3/3D773h++/80FnYeVkb/jwp5+SyXK25ZLo1qEHtFwWQ0rByXgf47cZKjHXCFZ4/qEYuuPnz55SuBFbI5qtS6SS6uKpEAVEMNq1RU3mLi5l0AN7tkkdieCi4oSp0aEtgRy/yuNjRjcgXmjBlMW2CdgmnefMspem35d+BIVrkubhS1clpgQPQZvp1cZl+ksB6I/7JEN7cG9F5sFcCq4moW75ND/rynOJmwYuL4pcTwsp/E8XU8X0SjHT5/a3Vd3uDQgvv7/xS53t1+yh8dT8rAJePFS+1uZAPFN2cmXqTbqQjiihhiQTueUpIhhzlZfIQBQOphC6v2mFD8U38cxJjiccBsix56e2/56QbynH8Ogao2rYFFrsX1SxNK7YRCoopSltAfSQukmzfrc/v+arBsVWVHG0iNhNEi86EzUgw4NhFDOG9FEc0DuWyuw6Wlwb7PTWdhaXV3k7Zt7SSIrTmNXVg9hVQ4hivkUYMWbEsz83TyrPpj7Lbpg1GoEoAx85RJcVr0lk5VJwwRvZl6APNzFjBoAGLIOCunr2r43LLEMe0igTLGBt9yR6iv4oICWCSlPn+5X9SNawI8sjplFzsg+1tMx88btd+h3Ah7wHZIAQ5TXYlUmgcgNeqi6BjasuHkxepekChWw4gc2lgEaUQQ+8i74VmSBF1DxrPVBSydHpuqLdPESNqt0zE0NzitTNcKMKeRFOw4A5Gl0cHl8eOXFyj1lukUfxAh3l3GkbmMrt7R1a1Clv3hijcp6GNixX0FEiIkUoxpKu436ifinYpp5Cm2pQ8SHzHPSITqnOBlTXXIdpepiYjtqKslOtYymJhVbXrO/GHstrN34w2EBjHrJlJScnSUTuixgEv0q4KfKhw2ATn/uMKh68uHDrzt379+9/cvhLMCv6lDx4W5TKZM7VpnBMTgEZuBGJm2rsCIeYnW0IMPhmMlEzK8LRJ8gh3JuD3XAj/ad1ZcE6KUrwoJcKTwbzRy0VnT9Nm7/M0je+QbbKxDKOFmVTPmexqLFzCw55kINpFzdh8LaHVXvZpWR2d1QDUwFoO1yIRKKbllQxTUCOgh+MdcJgtEMpmVYngXNZ+MXlvyQauQJ1rhw+zrMRj4gdVKoepf0V8F+BtUQsRAnu23njzXvf+MbXBW3jmomRrq4NFAuAWB7HLmXZ1T0ajsQ/ITGIgEEmIN2CJoWdNMIUCFnMct5h9ukudLrjo/1nT8a7T+zlIvf1by2+9vqDd/7Fv/fvv3b/wWv3XiMu7j48OD4arSz2Nwa3mNLJj7pAaNNOVhJSD9GHcLhg+rK4gB6nq+dPdx8/Gh4d8ClAVwVkZSMnMi44pVt00oTCtvNNhBMC0JW9TSxytZCS7oMCWWb/4DkIK9MHURmM2j/cNWUZxTwC8FqjLVyCm5C/zd2w4LIG15gkm0HUwtzbVZO5/ZzdKw8kLAYbXhTxPvQupGb6YX0u5xRdX1DUVs6Xc/pi2uxJ5Tf/aOqMJ01bm/c3P5k9eyjeOcn4UvqEmU7Kbq+S/6WGv1pyq0iZ7YpWQ8BP5F+cynobYTCslwacZQni+bkF4JMWf8Mhcdl0xLZTJieFZO9vANWqrBkRppFpCoyV6hYIT1r5P+KPBtfXL8FMilkZJcY/GWLaQrF1CxUKrwgBDj2pYAwZxQUn0G13+5sRDQpDIEbLR3cBjcA7YiTc0W58dPHyvKuQxYVxCOjc2cLSaXfO5rwEsEUvzqwIWSuPlhlDKVVNrAcfprkBgEryp/FSD7/mKljl3Zc7eRN2mlRZgnpaw9Yk+g3HMQfqqRH/R30TvL4By32G//nwt7ti7TSGGbcooNruf82atCxFB0bTi+yFxsd+AuZpQDRyqlO+q4XD5quVTxDwGNBdrZBqXB7rAamKtp1cmW2xaQeXEgLXzSYhDi7XS6dzy72LBUrKiY+WLufHB8dLK73FjXV6kMoxQBtOlgRzwMtxcPZhJDzFRUjGCBmWcWi++xY+4TEdM0cbRC/NEGA/ONMSJ9TVVcxxNB5qix0qFhc1D6rhsPRjBQkgnOy+D61JCBiRn8MrqNI6nsWALO1g4j2aTMCZf/oa/7BJx6OlaaZow/gOQW9jY+O119747GcfKwAnKyNORlyBLsPqas+6FjAWIDXMZR9QdMqYoLMz38ibypGOzYOgQWy59tWKCELLj1ewIsqurQUuxF3R7gmvle9regAHh+dzckyqNo2KoqYJIKtQIUouw4Ydg9MHd5hgCCwVFAy0JCGg1R7eianUsqcB5VMNhpmqpXO3PsnJFjzX7UkU/YouCEwWfLWblEOKxsI5xqUdkRYohSFCrgaHNClG7LP0y/Rb7gzWF3d2Nhw5yGy3NJ/N08ZdB87xWofojU5OLq+Oj46HB8ORNYbxqWVmVnmNdDQHZp0zdNkbLheFPHj2/ODEvsvR1fOnh7uP7TJYe/3W+7/3jT/6zvvf//53vs/iT/w73hPYg1vW2trWZgY6co/tmZkXVscc3tFZWbw8OVbf+lv3Oge7+w8/fvr4iXMrjVZ3cXF9YC2XzOQ01+yzFmEQaQV+w262aQyhAHKDBulF9yG0yeKhc3ZqtZjosrgyf3Z5HNtOROgiFA007iVUTxI9F/rNXv66B/AE1VfeJjFo7HrlzeRnPpm+mn3eHm6+knv2tspiEwodhnwRuUu8lMFgJ2eWtYOCr9QKVarWWAhkdvk5e56+rSy/yy21VL+ruhcQqJ8paPbwG55bhaFByR6IaGLuUQwqKcl5i+M2JTFoxA2FwCbwHtaLE5NyHb7MBSLMuBmcoyjXTGzcN0XnXwgRSBkbf26OfgNRXv4OV5WS/CFcrlbIC1BU4iQ99C6zMrpIBGo0D/EzN1smPdSgymImozxokPA2Na6hyBMUo4uwHRbTlR8OoVAKNhJCC5CalYKT805kkbZn1BY9mhMnz2OOiBSlztzxOZmVAiJKa6RQh7Tzt8yeDtSgWqLGNjeU758utf5UelL8dP8trgkcfOgfs4wmMYNnfNMZip5kKZZgqZLGS+pEYpU++XhaTWoMu9XByeBlDjRczuqrrbc0ntTkw8m3sKq2H5vJqmBtjtOLKYLcGwMsoPixzJhxijeBivkaoep2tbSNbDCQWTMla2jz6c1oknc4CkuOlE/FItqLEdEdX/ec8+fwuDMePuPTIyEIMI00jA+URbwcUbdSns7AwHJHOIjZF2kTfLfAZaSqzcV2oL7lUFoatq3XRdS49QrB0bfvZXf3qdCrWqi/mRoqEodqIQuEdgORT8PwbHTB/9Fr0Q2jloG1jACANeHg2cu30OsaEDBkH6Q62wSUpY1svGE8cQrdpbNtHKapOAuFW1sbWlyIEFhWmwtCUrN4nA/BVFMiaOZ/go+QKHEuPgs7x7nLO0BQxYRiQm4zCD7V9Bxt3Fu2xtkGK8QO7CbTJ9QNH6ZytXRaoBRTCj3QbDRQx+LuQchi8SDRLi+XimaUGUxtS4tFHQFIvDxL9ueXFlntplhfWzWPuLCJ8KWRhvZYZ095HV+czAmLOX+cqE9IzgkBgXLMbdkU1gZdBUDBQrBgTSNImLXkMnZbTUbH1FJXLZFcO1w5xqpM5Oy+tklu8eLk/Gj/aG3jlrV2FncSd6oJr008kqODYU5fyslG1yQsaUBE2MObdzZ27PjPbu2rNcONKC4vbHz44785P+p/75v/5A++9yd/8J3/2fbGg8NnIuFDMwicLcVgX0LPVdZno6lTZ0HUUYIJ4NFZWV0Y9Nb785//2b+65iuV3SUnPR/HR+AMNc2adESMzIV41zmIDK9dykmF893YLei/nD7wGrAgblkMzrHIF+Hq2Dz84Ggd/2joES4bgSoAc0X7hJQAAB08T+gMR9PJDH1BjmSOKmt4kwtyFGUoBJ4s67WP4aiPy8ClvJRcOT3lo7wkNadqP/0tRppprlUp02MegsxQGEOKMAXDmB/9V+nNlqePmpOZlQx1tYfpPfhd+QupU74EWBrcVhwDS6CR9MlsmvCUVtavv8+q0yO2O/eQwdwnzZh9OsspJe8asdXaNDi9S5rnakBIZKih9uWYc6VhunAwyEMDrk1HcXtmG0EUrqz+ZvtvFIYJVUjgdWqKOelbpWuVajKyAa8LAMKNA+v6HUB89dU6Ul/5vMAb/CgK81VfNABOejTLkGkbDTjTNVMSFcE4stZyOrIrSfAufgorK1u1yfD67bcfrA9WbUSs5iWSTvUCCQGWrOYqF6K1xoftWmDEUaIR6q2pvXF1iQ0PeJzMLY2W5o+v54/nzofm79XZ04vLXaTo+vJZv3t7fmELa7y6HHTiIL2M9LRVsTDFGGMBR5kRb9VRSAMQBYsaraB9EDe5Cr1af6t5efRgAOB/gk/VLptVO/v5lDE8znWG85fWEc17/8gKLLRT9K3x1xiUC2FronEBPSJL6jPZs5quJWiDQU+SFqLtGfc2NqGGaTMgx76bSLyckqigRbIZyjhmoQ1Zn1YQ87gp6pU2KC91a3z1K00HgtCMTFEeSrFHG0Em0zQnvDMg6jiGSLiT3vo1cFoc63btzrZOe9S5XFtfc5AkNaJDRWAzvbrq95aePbV6chyXKmOwKkuPbkHL5OtDjOb1ymSHEqsHBjBK096cI47601ZPTsdWGntrK/cWb3344YcaRmDLObg22IjIa90f7cD4J/wWxaTdxQADIKVWR+/UTRBgWbX72GE/3MTYgM8uFvqbd6+XVp2Fs7I2WLLjiHPj/hMmdZ7Z4HV6erS+0RsMBqfCP5F3SzcMjylOifsCNyFA9BjgE6AEIFXGE4iBJlZncop/ObLOrOaLE1cviK1lxsZc8BOgQigBOEObERFkhxkHl9FTDI/SSUXU+ETErL3fnG+j0ceOPWd9lH3M+HEK5zN3em3zLxurU9nxqgpzgVULLsUAcHa5tbapT48+fXh7Z2N5XaTrk62dLaDnzby9sca53XotXa/X74zPhvYqM8/zRKaDYkNmY0Jkr64xIxw7e966BjnjwpIBgzaxEEhsvIcTfNyc1oVPmftWjudPj62rdhLS43zp6miuvzBYWRJmx54Fh0lkJcrStXKWV1YvOmMzWo1Pd3dBkhxmg8+z/aP1xaXR4fnG/bv9wS1+ddyvHj8629s7/Y//xf/5wWsf3L/7zsL8KrX44OEhxZrZAxNHXME2YW2dXJLyh0BqxRpK8piydk1i6ex9NhTA6uD5yfC59Q0dxEszkKHUIZKWi6EN5DELROebF2kmlILsFEQm25tLdmBYaQljZgOYuwa36xyJ46SQ47UucbQPoFZxYr7Ip/ljEpHJ8bToFL5B3Iy6V8X7Ee7KWjirAi9xXlNUVxKwJs0jhGZ+wpiawLJoj9mfBoUl+cAcj6GxrjD1YG75/MO42LJkMCFSrbleXlo56FEJKd/3IdzNVGNflq1ijYep1ZWmpUw5J7/rT/DXBfAhWxolj6zVED81J91ttKYRnKCtrAFdkUEFhlmnBpDWo/yoeVHpeZ23lS2dLujUPckF1xSfDA1E6SzlDAualEI4Ax+AUDLcU2B2sEHciFdkVICODE66jycCnwTH6mbH0QWme3JMJsyaRD4nIyZPxaQk1lmkMdvNQyJ8BaGMjhRvm9lVANT81j+gePlKs13pVWBSuSpPPnG11y3L7DkZJh/O0jyQA7NGFvKt/7mlkBAobU7NhRfcF+YFV7ArYFBAV0WTDNr9RrkRJW5ecoreR8APwl52nKruBLmeFSK2tMW5YadziMHPz++D6XzHQW+iR1zYPupU07nOpkax/BXBTksUrZ03Sw8LfAmxXn77Utb8UEpJnQFR+HfQhhOqaWbJiVoEtwwEInwabIfzIIGUB6yp3LTw6CP/GjRtjfa2hMTK4YeSMzIN/SCQDCqK/TgZU2P7FmDNVIiRL0pnrWZkF0g6mYHwL4hbqqEgWanU0TxovbIm0y+lqTnFah1SIY+vEQ0fm9iXHJzsGJkn3ZulqAJyZhPZ1a3uWhetjT2Xd4Y9kSk6FthvvP8BzvPoC7szaZXChgh0xJhzbJEShQQ9LEY+zswOIBL6/tneM45BW5tc8/obG+tr581pSAQM580R7sJhy1OHnnbuJEMWxTTznO67RHlFbHQha5wVvEl1gAXkEVJQM6wjGq0HhHTFeoFz1J0GoLH2j12MUVumlMRiFGoJjwRh5ZigoNforAfiCCCX2RGfizMxQEkJ96VckZ/jOmn1NzblrARzb3LXd10QvIEjMI9eMSKF5LCoQqAMNQ2VjIEhbMyfeXyXdVWUDRExGYPj2mfsalyItrLGUSlokTVgBgXfUX4p3Oo6ObtaPDnr8ycBYkSZiJYFSRQG3eHbOx7uH4D/6PBIsnm4vbHhBCSqLcJyyE9Yi7Nk7qRoY4x/RyQrQpkCXCDpJgUUjUhNaZ3IogCTcSOnMeOrOmIpnNcES/cclM7uvb1xJFIondv2yTEFWD2c9i72nh9ube3cuT+wcPF01yaja4rr3ILVh03nYx3sz49G9tQNBmvvfPfb32ckfv3B+3Pzq5ejGEuWruxpy36t4ehA1BGgtqsKhcR1IJlgIVad+U9zuLo42j97Jowg9/ksOLtHJIpkrJn6AcmLKriHgAJzSDY81SkoAP4ZJrkyedn36keR10w9Vdb6Ogw0F008+cy3fB/OGqKclAyJwqdXmHOqMj9NNs2YvshfSSEd7sGRECfZNNSTKznac6h2vcwLaBjiULUUiajmanwKzB+160t1J13LZdBTWLENWDZpuuw3m5r6fvMFMwPQEFLXhHq3kl/ci755nRSUJzl/zTXp4823AdaExab/cqRb1fv6VT8mt8Ai6BgiGejkO9TNWEDgmFSKH5u7ZOW4iOAaOeI34a5YOvBdlCrhrgR5PmXrq7i1CR8VRhsok61qBwgtQiFJ9Kr+GSbADN10eW7j3Vp6s4Vfeq4PbqTCwBu/XnksBHg5zZJNsQZVRmeIjNUwJMQi+ZEDZNkCyspgsGVn3rRtLxfj12TwWvqsWbqApBozdNQVeQ1XVWU24EbAtQXJVGT/oeuMxf5jdRxfDalEsHjqnm3yKMG2P/cUqJxX8L7V+lvekfjMizRHOZmoEnj/VKxmnkFCLzlLnLIGl1E2f8MMkh2a5z9dSH8hRwM2MpqGFQSCOPKGa9BzA4ek5NPK400eAtVQRv87tSJqfTxxwlgTjZK+ghJjJ5xYKzGDGi1BaVZ0q4oUgjenNp2pWZnPZaIcYB2xWpFpxFzeh5wx6EVL9+9CWLKFrQ2HyjkGOMebi1kPFVVPBXBQ+u27ItTfX791fDy2kxI/s1Q55LrV6QiAZKHhzPbipYXeYI2VcHiBTdtDGmUdhxAa2qk5popG2UUDTGzLDnIVOFL7ad6r9nuurh7bfhRGIQDSSiRTjhH0qfHYjvQuBoxkRqAnMQSCLB8XziyJS6t9fMeOXQoC2NyMeyzTeyDD4jGeWoLVwcGRBsevvOBcQACzgDlW4HpQMugH8DUweD/0njEq5aBmQlZY8qSCczCO9o7xF2BjFC82GitYFDFksC1iXo2DuVygAg6SgPUVpAQWFwfE9uPyHumo2oDp4hja4PIVX02jibQZOKDTQr2gsqdPhMHTUwOxtbUFREQJ0bfXbOIZDay4Zinncp8KHYMErVZUGVfFYKdAStHF1BF34cu4PNgYZdtbsJKVRXXU+goxfXqxTA2GAz2+2VGULUuXqER01h5ByhIs/eDoUABoEuDh4dHJ6eXuc4eInzx9us9zjbPYwlyXQHDw/Hpuja/8eq97t9e/tdJHNO50u5uX58tsFtF3L+btQ4tcEzXSfpFjIGGh521gSMlbWUSYu3JqkuEeHu3ZrYZP6kjsjtWfxvfS0yJ2RRwyHYy5XElmqck81XZJCIlR5eTSxjByUMi5Eyyv46ZgiTDCCPbfIYJkWGqeZaRcpeMGc8KXvcgUrFHMyxKwbvystFdvmuyapcKcKiEpRSlqAoc+pP1VpFqqF/Wj+tUSKjVtyEXmSpnQ8uUrtOof87rZl1k96WEo3qSbWFoBKu9b4uQuT+t0dXX2eXuoPLn5V/s267ESouIHja3BGLgssURmjtZPESZx2xXAVpt/Nh2d2B0pIgwPvDJY2SsQvDFZo8JFreT+yr8kfH02ErPmvsw+XwzbK239h/uJomJ1RRsiY/HIhsQhWI14oQK8DknDTstZXdtgjAKQwpubTWjtvNnaG8821aSf0Mv/7jUnwl1ihzUv8Ap6Dlp9cnp0ecny6bQxJP3QIhqC4IDzBIa4TszCOr0gk6G+dccUXJkTv+sFi/UC6mJ4UIeAZXXMjhsHFi3k4ECGykTL4iiD7puT6XQEwNSVrzKJJ3XWqzx7aFf6qMBKyVf1eUmp+Vyedgfk4NLV6XyOa4xuRQIK661tWUzTOHFIDDKOlJR12YcAVp8bJiQpky/N1KjMX+VjELR5qY3911jRo4oPwE5hB1ZjlYtPs12UOMip+MB2DRmCxuMv57fXd7bXttjlnj1+YmnNImN/YfnQqTL0SItxS8vi4kMTdmzm0o2tTfZnBBpVo8cIAoETQ3SuOZiRaE5AbM5wsmIdFULLuYFZBBWK2Uqj4witml4weGS5rgCrcxEEDT34MC+dntFNR0t9q4qxkffOt1jLL85HHHdPHSBxMbI4GQZ5dTU+Onn06DHcUZ8JrDSstmBVA52Cc0kvZho+F70HfCMVGq/YHnQQpwYt7J3HTiy6CWoFwBmumL4W+evFClE822ZUUTgsiToaKKvadh/FeQuZINBGLGK3qNXsFI+1pI/hkFnAhgr5gRLUjgArPrx8swxB/4yXsxVL4mlZMiPjsB+wxyb+s309cVtbzZlaHJ7pvfA37J/YGkhmtTtXY8CFkxFoPFQyeQhBKzGO0zJDXDRaA8c+a8svfZ7ev+T8jP6a1Z+srQLB1sbG4f4RM5/GE0xOxuHFImCheUUb+pZcxdk4H17cff3rt7bvv/baO5tb9xaX+tzrr86Wjs+QxmPbcZmdr3hnW/9wdDgZfHHBCYLZH2VLNzBT/EfHFp4tcttWZPgTJx9Cwwcc1VRFmgjEwfNMpRrJwLN+ZgIkLfy56Exxp9CbTBnf4MfRTU1HIgh4yah3JevENJKPXBGuQSpDE8pQaW7tZxAhP4JdpRfO3k/zTf8mf2ZmZa9Z355vEM/Zt6l42os22eX1Zd2KzmSuZ5Qhqg5EfM/7+ixPLWtLmTbgt/wb9Hg56ys/X36ZX/lkSsTyu+ptKe1tg9o0pSDQ8hTGz76dZkgZ7aoU3THRMk6ho3mKL54R85zL/l2bBbLRL2IlZ6t4PjNlCDt6heY4N0sImxi2WFSyqIQURt/1ITmdxSpCrZ9NqZp0VndS7bQdResbp3mR9I/zFNG+KEEaYH4GmXW8wsNmmIPE8VZ14sJqfyO0IyRr1pawmZevV1LgFqbigyBZ0xgLh002RKygrFYz/5rAjpbGDYR/JaMCYQcNMt2gXFhGxyEKkaDyL630SwnYm8Q2dV5uyK/91TK7B96+rkJNTo0hDNh0szJ32SUkKTy2j3kEzlX9qmHyUQ1QoFAYU9MdsCYzuCXKCmNeGuXWosKvKjJIrI9nnZyM1FbQlYlG88zKTotyQ9PTtBEqlrVN3TEXa5d5WD3ICNVDjQtQRJoPjmrbCusrJ0GcDOugbUSQ0JW4/K7310LHLQMK5H8kRtsI77HzpnuFYlKlrk6Ox53Tq67QGnPzzDsJb3ZyZodHfzNmZjGURZvcvnvbiQyltTA/WqYVZ4YoYdU88inV2aIn2QuDcigBfhYbLpcDQo7ZNX99ir3Spwk7veXVxb7JpSuEEFyEGo/lcKswoWjZHW7bZiSle+QEXPrR0eHhASQhQVDVxCc+2Tt+8ujpw8+eKMSctTKbqZqlMwAJ46meh25h7UAMhjgWWPhhgmYROmsRIW/oHNFAIQE4iLFvZIWGigpgS/Ra1uOQBIOXe3nNCf/EIm8saVW4bkdkGQbFxHjLkpYyYiTBRiJbqEAzDGsGzv/ZlidCDzMrWzgzNGErY5fmy1j2J/Zt6u/KyibOvfd8P18tLlh2xfvJQ9WISKpy4/KK1pRZf9UIrtrm4u0UJVugulpTo3PrMw2BOqE9lGLGdssEPiENuIAI9+32uwvLPUYNjw51tn3u86v9jcG9s/E+6UNDzsbxUdjZfu3Nr7/7nbf/oLcsTi3mzQGKKRDqxZzjJC3biBR9Yt8Sgnk+tK5A8lldFyKNiiIeh71Mw0PRpQ/2bHMi9gCPL0OXjAT8zU4rK9Z+BXrurbOA6OfkHxBE9sqg59Mwq6zBp6f+SUNOAltiDo+2q/GQyYT/tulEZDIhM899lrFP7SljQpxr1DINk5ip7CEU5NdcNcSTd5V98lz0tjXYrPWgqfn5alES0460NtP9RhFS/ITGvtXOuvxJxyd1/E/yJ2BsFaU9k3aEZ9ZjNTuv4ZJ70qedLPuh8akrqfnYj9JTgSOlpt/B/0w/MFemDFAVMjE4FyIgchXuitWJP14tAGPGxZgTfbdCbWQBuJE+hcfDJXbeouzF3tV0A2Rp5/S6+Xwjy/T1P+DfLJsprjoJcUOqirLEehZZHipn16xYvmtOI2MCCOl/gSzTp5JWqlnTlEkbgyuZCwFgygrrCFZniyltr5yckTfWr+wDBayz86NMDxGS2RniFi1Xd66zbpYGSX00GesGI0mT378LUPJtWpG2edQu2Kwms5sR9PxqqVFATURzJ5KXDsha9/YATvXtJLF+xaqR9KBTgSkkuhBVp8B2hqmVO6Wh2JRJ9MVDUQCN0rxYoWNrTcOUWTjavql7qWozvC+KYBi1kDcRao6Uo3w0DaonWRD95dSK7CSyQw59ESro6em4Zxm26wiKhOo+Puerf7FMN3CWFTnyZMw2aKPMep9V2bqbY1O7o1oppSf1N5wms8AtuD9YGx8dDod0I1x4gM7pEarN0mh7SszXdfYfak+ZCx1fW8VsMtqAos+h+zlfJVe3S4OWAkQ2uqaXOKlNrk74KTtoVL+r8efj4fyj7FLbubXNb8ZaK6UVS3z4xRc/+/HPnz56ytgYD8lqhuGNRigLz5GSSMKeQQmFrktdriz4hUxE1DMHrAaAlQMnwolCZAGS0hjB00cQMAhhAqdaECKDx82BKpwelcky3JfYoVqcNx0NM9UYqGBZAFm/mndGRvCECMTKU8pqiyx2NlgBbTMOQiiX8odeQICr1dWQHSB1PIbPBhvrdtQAW4QVOu3E6Tp8Vz4VSnfXm+KjGRefN4ZKGonoEEKU9DQD962WKtOmHboo3zp7yaDf2tZtOxWGz4+P9kYrC4PuytbJUJ8GVu1HR/zP5zdu3b9z+7XB5tZbr7/11tvfvNwXQ2CF7hqvGAREGCuKbo4czmKDf8zxffbulV6YDpPM+bhzMiRP4b2EDMWCnX0W8QUwUKBHz6lGxjjBDBYBI1N3dtXPzKzWl/QmGWquhbf5wD/SSS55FImEaY2GHltXPj8RKqj8ASJzGTrzKmQLWmQGtsnnUWIoZLY31XxszQj4vupqjZk2KTnqWXum9CosdsJ9i+FUehHHiaabcS9VvBqvBO1Pd+qyYqTAYFUBp+7yF3S+qj2/Ji0M/pUuTH5Om/nKhzezt2fNgd6T9NaJaZESWzHy5Dntd6WGSf763W5SQg5LHIEtBt58ss0h2IzpoitRJPAEmMXORXbHaC9s84Vop2HD0Iv5i7Qeh8r6Z44l3kKYrpnHwyjKh1esmtMmZjzSLMg2aeu0Sa+AZZr8D/zXvo5M2mpPABQYGUSN5R4i9DMkQUfmHcctFPuaPUOoUuPZLxoSRG3X7GHWeCmVSJ4MCIq+RbwJ6ckgiEKVfiObrGrqdkZFr3N1VIZwtn3r6TSPtZU4RW/VwJXoEjQGsb/f5eOaBPV9YJ+tpRrHS1azcho4udugkUQKZ6KegkambbWgJnpoQ8oJxkRN9hww5t4Yc3uYyG4FhBgxGw+ewLkohfxx/UFlMI6s2voE0MvSEO3NPxBCK6u5+aNiSVVXa0/aoRt1Nz0X6DCX8W5ChTHV4bBja7UFS/8vLw8EolpZ7Ow9f/LwC351VmETzL/TPRETf/9QiAUhBR3xtnH71mh/7+gRH+NzB/aGiFOQ2IJHw/mjle1b67fu3OlurtNon11fPX/yiIWRH1Z/1ZahdJPyqjn2jiLi4RmnCQZnRzisQr6wcNnk0XVynv7wqg67NTUCQBjJE9lcyWTJEQILi/a9Fsu4fLbHRfvcGcNbAwugGwnvdXS+9+jRj3/4tz/9mx+ziYrxC7zgXPAosHgKbDJuKs0QFIP3gEtRhKtOVTcyHRAj8zHlxkKgHO3CgXGdLNfPLS3jsjg05oLvWmsCMuMnNpVawnEye0IDyK9sphk/kA/Jj9NIdgDVPJO59ZdwwOCOayIoBJfBij14KQGRQHEgAlakTYaAIdqBetahF1eYccPbadngCYDt36SbJeexzOkvMq2Puh5hiI16UcQNyytzS9b9ZTDB4ogS2UIZpgGOriIQdMokKzSFvnMyd3xwfngwPB0tXJ73P/v48GiPV1736aPj1e5r77315je+/nuvv/aOMDu2Jx0/veZbT7qqIaBM81gINtM+TkcjsOS0Fvki3BWMzq5OhgdPn1iFJVVkrbeCeGgAFy0qfyZU8co2oEbExbijm+lpkD6A9dZdp9oMeTFPWjavCyVUmH+Fge7GjXADny0ZLvdYikifmk3f0jacGOiVWcw24opJ6etJ2am9VR7WU9V86TZFZp0IjfA+3UkDDKcGpzRIkVvd29v2M6npV2QBI6j9qi8zfO5wyKsUFZSumT+hwJEW0p4iR3m+eVXtNxPyrBW/4wUzNS7tm17pSl3twSt5XNM8lbO6BKSz9AaTgNFEyABWyyFFUVTQi05SHDcTP4sT7XjB2msUv1LLvaQ5QQZGMZMxIZFO8WbrVBb27HEo404mbThuDM4RhYPn0ZDcM8WAMC2KVUVrC4EC2NnVOjL7+Y/xEEdZwIFskWI0Zzp43D3oUvFBNthUkiXeNGQuSzIvX1/BfV/KALZUBaVkPFI4QdYV3dLPTJzso1GxJqA1TJ6rtgtf0CiuEDb2fZ4jh8sLx5xFArfZaLdCAsaXQPZS3b/hRzUi79sDdTNaTf2Iee/S0hUqS/2g/dt8ESEPMch7bSiR5eV6C9tqNGUxgRuC5iFsNZ9lrkwz+D29ZPAqSBZ9JTBKrpYTuDLTCm4hDV4oMIWX1JIiMgkDu7p8VURK1DH1Jen68uDoyMFt8epFzMXI37kvpsFiZ2ltfrlvF8hSF6OYX1k/vVo8HF8eji8WNjfufeODwYM3Op9+8sP/7l89ffjICXlIAeXJQsvQolnnwrGr9xfeEB6Yon1rsPGR49iPhnTe69V+HJUjwF3vbN6yk4rr0P7+/tHe0eHBgcVQLrjd7Ts5eME0wAur/WbWsVAOGEVdeEpmSxTzwIR7kSEm5HZXV3IWAJtS52J70Pv0lz977c497sHnx6fPHz751Ye/ePzwiXgWljbABL2a3oKAIIF0gVvTsD0h5TK4B/gajOAGzhIDb5/QxjQnsa1rYRXI4WuWQFcsT1DdaFARyl1YrwjL3KfVQopQFSQNuhBEwoBZjYqCKx6rg/NpXxvJmvaZ/5oR9TSbfEBJ27OETDNWn1aYJGJ8DuhqODQemZgYGMg47mZqUpH/gxURLcA/M+LkeOTOwK6PXqpSNlXgipkyWdvByq/mz/Aii2VBQJjYrLVeo/sCUaMAT744HI3pGnFPFrPxwx9/sv/8dGfz/ve/+ycP7r9ze+d1p/OeHJ+PmJpNoqt5eyUscKurEQadruUAQTOv7GWm21q2vx7uHz7bOzzaPz12SPAZDqPNfXKhNQZtPT0eZy4U24n0kYELmmtg3BV1q4hSxmkyUWrM8lxDLb0g3P4kV65KyhyKtTn4xRGflMKsc5qlByGCI4lH5Yg8nmwZRGXWiCrA37AK9WdS5ue0aL++dGVQgke5jE17dgfb5E2KSlNY0bR6CEmpxBCChhvB21DgeMkY2QiF6LBPFJtyWmlpZyvW55X8j39rDUgPMjozgjSpeNq8QKAlefB082drc7gyLKls7lFF/A4yQlFMmP4Tz8dMtozRBa0X3y1rc9gt7huDMyWY8ZT5D7rhvrDZmAKt/eV+GG/lZZElzFwzMjapFvjChSGXPO2KzPU/4UU6zsiFRJS1Y66OqvVIi9ncXD0SQvD09LWd+D+DDG3YLE77Z3x30tzM/2p2BsNDuxfAozZALZQszMVIlY4I44PnMRAG/HkWJGBhdeHa6tHaIs8bXmxzl+xhiAhPl6UFRmmntXSDwhC1GpCBUlqwoF2T1mhMqnM1vJy8rYQbzy89xtsFWofGzs0PBB5gl0zk7vNjq8KR5C8Wbb6EH1qrG4SzfF7yikmaSVLNiLSRny9QrdUCo4Jpk6ZqoAvNQlvRxDS7IJbEovO4ZUJq+6Bmr/TYx/wGJ4YY3EkS2poJGfJRHQ950QyzOLbg/OvMMSHCXsdROdcGaT84uRJ6dGn7QW9lEMeW+NzML22/du/26yubH3/22SfPDh+PN9cGr9/r3Nn+9tL8T//8Lw7oyuc8kS9sK7pYvH68u/dX/+bP948Ov/X977/21ut7e4f9+aUhffnxU9GCkS+8yULm9p3756Mj4unogDv1cNPiYZ+9mmWa6c9h8peUnmPWVM7Vde0/fYZ5NAZja4pze1DH7e0diMlpdnWxJ7q6SpYjs3UefvQLXr7DvV07Vpw697c//NGzp0+zi4ObWA5WwkzCbPAP8KFOKZmnpGcQc7URAXDpkbgrGwXL/HaKMujRwe7du2d/88qypto6OLe1vYE5P376tEMmFW9iXkCMk93dvbOnz85DxBMrMuFEmDUrEgtLLuuyiFT9/kp2JmVDea4aO4wuVbMQoDE27BJ2bdcyAcg3t9YT/Ti7BDK0Gf8wo4W5jz76ROhQzic6SJgSDHl9sPn54eeD/iqznEVydt5ivvEncrEhU/cjx8DUoF7s8Py57avSWnqgOJGhO7pH0D3NucU9a7TMFksZD+vyPAF2tl4/PJw7PhbQ6mR09Pzo4Gx1+cH7v//uO298rdfdtO1pfJTTknrz62fCeDBpoBBE1vyxbMDqounXK6JU2a2MLY+H40dPDg+4Nx8aG0KAXfBR8YOrRSY1ErHAdKnJYUHT6auUXNnhnqOmg9gymGgkB3dwin2qUvIsJWwtiq9nbfCW22wc9WtbI1p+3u05Cn338PiQawJjXFZtIIQdkKwCmVG2QdE1TRASZQYkz/6EXKtISrWoMMl37Wp4pd/e1ehlUnsOcZxcfkfxaDxcebN/CEganH2hBC6Dzwyn/XHKy6wvNqE0VUjxq9K1zNu8n1TdeMr0x82/s29bYmuw7wK4+roKrF5VDssbhUe5hcKWkKRThsBFm5TLT58mgxGvtkmdlTOpyD5qdegFWAajQ7JTaAkoECXFSYl0mmzAoHj0DUJkSEmv+UQVcbwS2ap2HBGa6LsJwcEEjVKQ9aP7onQsWjG4QIusFtvcFt+MMBuXlTjBh6MOFqIZhrJyBsITlpx9BxmSQGSGffn0S5emfiktCZMR+cp3X5Vo+ShjELIfIICCW2FvdGJTIHa67MRHdALir6w1g/HKFVAGa5ThjQwBfeaLuQHNc7WipBQSZs7IxluChrE2vyCaQURSNuqELrg+XVo4IdYLc5svJWCOQiiTWBM8OXUkfdq6hg0t8avvX+5KZoUGtNJwuy7MueRqejXMUAYsYJT1u3yaaf/K1XpajfM+CKe09m+SsxLznAeNrlYHGgF40JQBBtHKMcmE9DDyQEz2/K3/Ap8Att60P5CzpocyzVgDGFGgvmmG8c5SLyObcB+nzDLn/FxzWsb8qfD8KA8JyKyww+tWd6u3sfnmG796+vHy1moGHzO5c2vnvbfRuuPnuysL10eH+xikoEKWlr/45ccg4Sye+3dvbffWLladB3yx/3D3fHDaXVtjvzzb3ec5ZLC7VO3F5cvR6fNHT7JLh5aZtcuYOoV4yJy8uDo+PBLIAzNAtRG+kEnMmHP2eNRd6PKJHR2OHDzLXTdWV8zs+mpnMLCH6rPHn//yZz9/+OlnJ0cOwMEWSXcAkpncrkz0UJmcWB6w1AU9GoaoUQJOyQlbmJDD42Mtf+Otd7Zv7dy6davY6pleHx7u0/w5PW3t7Cx3e1RCjX/33XdJpL/86ONf/urjw+Hxs2fPKM1GcyDC9uKiE/Euuxdbg1XsmSxQVWeUEWAAR4vsGA4Fi/or0cQIikmiBMOaIr7kBVlasymKfIUWsF5RRbWZ8ULIDyVTmaEDeGotyTHujKXou1fXgytBDfvxs/56pSMJ6J2Fh/mTkePa2CFEgNykKq/1twb9ndWeYF+iPm+t9e9srj148vnT/srm+k7/fHB1se20hsFg9dagf4uDlYURzs/iHaB4hqW7uLbSX2FTBv/oa1iJIdEtK3HnF/uf/uLSATUn9i8deUC0bdEySfpzPDxAZcJENLsRBi0uRJ6MVNA/kohl8axuuJJzevcA5r4IFHOFHwBqCgbZyNKqQtPN8eS5EjwLBU8zbFamBpgTXoXQU8ZTTopobDxzWL2BYAamwbLNykpL+q+9WiN/zWsVpa6613NjzBmsMOlsQZQiQ+7tgjnmuNa0D6fJv8Vf4HilMfk5oSETYN4sBqq0/AFkIJDOStS2WTkeUqw3GYq/+5rkr4GTO5S0rpslQBeaLvNPTNDFR2O4yXIF0lEPsTPTgyvUs8Xg8NdoxpAxC8a14guxaq9RlJ7YosPM08SyQgc5WocR9EkD8jBtTSUFJ6Ukp98NqVref+C7YDNswkHTIF+qbNa5hCNoY8CBgvuiGQ7WYXwaEK7QrtmDn9POGJXIjK1PelXCR34qHzUxRfIQ+XbSscmHhpYA2FnoM3TPL9q3iphaB+JDa0PF6YrTv/Fg/+LAhcrGLJxJc+NqEtaNhN/4+NKnLad2B7NxwTCOnLInCj8aauWfbxJ3YG0S5FbAEB+XhK4LtQ43qwkNvYmgk/QQmDxOX/m85lhk3RVLctCjoM0z8yzn7kR49w1wISuu0Oh6aHgChVJc7qEI7YfCpdRzxrFRNF5lEVkW483KtQeF5mjF2c2uK1JOjkQGy3jeqmqwMLfevb0OH87G5xdLjrGzGri56h83GQfOiNenaWQxguXw4OjRLz6+Ojo+feM+v1auD3YBHT87Wup3N7Z31rc2cSM7jHLWNAd6fbT0m43mS6fHoh87/Hxx2UaXjXWtsotVWH/6oIkilpYFQuHTBbTiZ4TcHu7t8QEeDYdCs1m+xBlALhG7BUd6/PRXP/n5Rz//xcGz/ZSfw/j4SYf4AgRkCEoX3IAdSy92C8xBWukeqJ40HPoQyhqtcHHhzr0H73/ta6uWP8+ownN0Jnrnzu1tMoAUJWj1nXu3WWsYnXkgf+9733nttdc+/uSzx48ff/zxx48f7zGwWqIWvRFoj4eHlMpSIdBT8z12C9OD5NMwQTO0U7MhntcG0/YtlnrDxvSdEQ+1DSnEX13pFGpywiwzWrTxZzhSC74bX2hcJny7GEl0z/TRVWVEgcvaGHf0cHEbXinHpTU4C9shQQsrt2/dZ9teIPte9hfnNueuNiiHoq705m9t9R+s9NZw3Ku1wENkTMe8WaThLEFOwnqzldpgJw6zhVUHSjtraDnnJVjI3d91SOCFswqP9uMEI6okt5g5J7TERpig59Gv4geQKdWwO3/z05yIOlgdyT1eUXFobjnlLRhO5kUzIrZpll63f8rIEoLhxXwlWtEyYxFrXPyMZcYivkbEH9veORBStqYEjIapzTJNjB9mo2mzFhat9Ctv09zf4pq0NhS0/fONTtbn6ZKBmxYl8YX0UT2pcfyqSqaf5F0Qu64JTKY/vyJxQiteIkpQaUaKXzxPcGhGuxphaR9mjPLkQ4CbVVdjN/v14kHW5KwLANuVriulqgbzcN8w1FiOw4D9ZIztiGdK8CQxCZqI+8bt+SSOptlxxCBoIid4TzTgRJpE1sKYW1Oj+BJtw4aNOmzLRAtC5FJFDQemkgZVt/JUCJS1iBsgqQ/+oW8Mbwgjdw4AaTAx7014kXkYYwIqJJKtzMwva1amzbSdswanUa2lkwmf4aj+lJXV2/BuotN07twwuefbAkHrPCYnhp+oOkxTy9f2BMe+wTBk89fY/pWYTLM/B2Ri6kI7lNvqagUBXHv49fe05Ne/BXIFZvsDm/T8wmC+c8z1naHYYAVEaIX1t7hsGR7DJ3cVGOIJMlG/2uCWllwis/Qa2EBzOsQNXTHacoqxqIMi+5hajyKLYZj5h1MGYVSZgNgxzqXZoQu5UkuqrkInAFdNK9jXMqAlvkb6hRbi/cKWb1FQYL+y7ynVv1jME4EoklHCAo3Q+Pi9X82JTc8l64jvMRLbX7kaL27sbJ+PuzYtsZwu9tZwk6effvH4V79iIsZ3FYu+cph69nCXMbbbXzs+PtKOnMwTNeTKyiW6vNDrO0EHe8YJBO5g5hVbCQ+WstLH6m7pz9FwePve3ffeew/i7e8+wzBo3kbB0DvwHhf088mnXzx59PjTjz62bZRExloMJgTidDKcM2srDaFDeBs/DjL5P/zMlV+XOUqojMCnFO/bd3feee+9za2tp+zaT3ZhAXV8Y3P9/v3XmEUfPnz45NFDWvnt27cEZ366uydSBO/fB/du2wm9s715a2dL9M2Hn39xejJa72/RyWxqJc8atFYdJ+Co3FlhZRu3nB2jXJoaPGreJ9fcz7Gw8kbGasJPGawVIZ/BpLxquPMeDYvF9ePh0GhJDD7AFSYEFZB5i1W3elUNecLjjfXlpSAbOCXaJd6z0yOtflwwaJwKC7mDbl1drPVY32+/tbm5TVLmobw9uN9dIAnh2yoQ2oAVPZFd+IgXdrlZccB0LW9b9722WkBA6JwdXGf14Wj/uTMIMeDTwSpBMKtOIQM1IJ7TbChvQDInJ+OVZkPMLAaZU5HagslwNtOWYSZTqJ4md5lMg4g5pPXkKEIDyKkA0oeUJCCKn00lTzC0M0RmfHZMKEHoc+pnYpoFXyjDk7lTXtB+GRwtUEOV7SE2lfqdan/dBextUAr+NUunczX98/2kVLK14Q2OZpD9S8MjfycxXYzwHZFgwp6ltNK8/fJ1Q1L/8stpSmuSXyCZ5yrpRWJGIxbh1n7PHrxtF2KSt9Peef7Nl5wytHvL6dnUa8/F//K2JdLN+A/klJdSZ2Vj8LCTMiqvvR2CgOO+sM+yiaONcuCgt9hU4rhHA06jIQPUiZE85sTS/xhbLFX4yXakYhmAO826cU06W6kFejCXp3DiRrZ/8EcrUyb2ZGU3eJbZYKqlIY1UoSMVZiEyuJSQk1w3ke9FX3DW1hMArYfIti63IFAsQYVpQaxJ+rSoVrBibdXHMRzX6US2UXwlMkGyDDx/McRI8OAyMmalaFJIjfEEgi3p77jPGj9pxjQ7AChT8UF9DSYIO/+JymszD6TwS+T5YoclSacv6ajPcU/TvR5TmNRW5vQhHGi6olPZfZvJYhUX5xALOn2JNkokDx0gsONamAlzcQ7yNDBTbZjVMr8LwqnFgytJHjNR2/SuPPpSjbLcm1OFSBRwMczcMLNzx88J5clKfLZC8sWFcvHioTYIHi16hjCMi/2ug3VWxj0Ry5CjNTLZAm3QxTP6mnGY7/H4gK91Fkdx+OujkSrWNynBIwv5wpmt5sBaZsrFmKaPbaQZHxweYqLEj1RNurm6EtXSxpezsTNmrw8ECz45yY5sp+vQseYcDp/N80Ihnh5ftU2xP/nbn46OhsdHQzNUYzXM4p029UX0DfEK1XDB2CLvEgKo/HzpgtWGO+QZG7t99872zg6R5+BwaCtWb7FrqXv/6IAE+vqDe4P1TU6zFyejJw8/pSi//foDvmOfP3zsW0o/V6b7d++88dqDH//Nj3754S9sZl1fH+xsrYtWEeULeBstzwxL22YtrGa2G2nSHuxzy8HaWGOaNms4Vb6136kIUojCGKrNtFR3vM2kyiHKIS4vSBu4KbQsrazaPmqrkvNsdsStFeGzl1c5pYyyAMyFZWH30Xhleb27vb3afWN99e31/o5SLxcvu53V0xE0YbNwTNmiz+Fnb6UfhcT/RfF64o12lwRIYUfonByMDnYP9p6PhkeW54h1TuYVlToBp0gI8V1o2nnogxkQPhTUDTz0VGJUTxMAAMKO6o7oh3KoWd8dklGk8eU7mhGgmrrtDtpV7LXwMaUBSzfJsuAcIwmHcMaGY/EnIbE1ehgKfLGaVwuqPaFampKJGGEutGJCcer1bLK3zF++19glefYwzRNZJe0sLK3EFD5Jqt/5VbVFRJvA55UMyp0KBpNPUsjfec0a40EH2zV7nD1I99yadDOxpes7SoFkyBDkKkkhr5L+0hUYVoaUVm8ooBC10lKIK4yz/mU5lDOBlBiLkkiRtRmmvK7ifnVK+LSAgftGCIR+PCFZNcJ9zZ2w3ijNjQGTXHEjlNSEQzzSzpgns7pWQkQaEyxpFwzzoD3uPgH/dHDy8h/xD+UBlY/vX5At9ZvqIQAgq91ZAeZOQv1Fq4u3ZpqnlTfhnGdwnI3W7CHpOgmXSoKt9IiyNdGqd7k1vCkSU2cP4ATO5eY0ND+/xlGxc31S2g9KyTJqD4izcgkNdg30rKfFaj1FpIbTSvyNVyqaVvpqxsL1NstMvSig5Trcd1prYQrc4RpmsocyGKQCQyswrDjcLky0pZQ2nBpmxBQCJFvxyOo+qpgYWGyO/HqKgvr8it8vic20X1GNYajtEJAdQuii/yZ4XqpsoFmDEiCHEwV7ZoQi3CbeJLZaKxPTj3dMATx+SfFrgMGYAxLKlKhf9vdmY83pmfhj9qLMb28zpAo5iFEf7h0J0WlxlCT5+NEjfrobtzcuVjfs4MSFBZIgn1zZuE2oW1oc7h46nsE6s0MQ11fWMAjyV3+5ezg8YQTfWOmdOqiVzKrJlLOVeV5aPlzJamy8gLqrfcz1cP9c8EpdiBWa15P9L8fWnbFvJ8AeYb1mGzlMz11INIWVmAjNioVFwy1OFmS+XM6+2ECqnIGNimd3Jl9hmEIQGVhWuuK2O89HFMaj+WN+Y/Ls7e8eH4+c9mPZdWfn9u6TT2MjIlmcnTB5b20MRNYUo3hzbU137mxvLn772xZsfvGLn3Px5dibWjIssTObTUzIkJtRSaTtmhduuTKwyGk2Lln6QiyCNEWntDHzEnUxLJ0NP6M3s0thvQz2zPUxD+B+EVUiXSlNoMd4T1sWpbISMbhop7ORgG217XUHa6tbSwtdW3uPnmv8nJO679/+YHvr/t27b22s3+pc9g9247K1Mt/n3XV1dmzCck1jCrGITy3J/iLBI8Wurl4Bt0l6ecjDzoG+z0/GR0QlrNKKPFx0lJb3MS6GBOcW/Y4UmZHDG6d0Iww0QCCUFQ+uETU/4LJJE8SeTAFl+RE4mEiF7u46D2KVKRMAFKo84Z9JK9lbWBDOchFM1jYW8+HoCNIYGHZOjUKsDSg4aVBKDQHMRFJoqy5VpsIX16S+Fwm/6amGIHaIKlLOTNjUk4FXbmYluITvBiNaJ1rNIaEt5UUFEVy+4no12zRL1T75MXtunZl16UV64B072eR60f9pcdO/Mnhs92naV/y9ma2VaRrK1zhxuG9dDBLhpnhq9FpiH5NRYmvE5znqLxPdaQUm5QWddd/sVo011UiF9WaaNCdYz2G9kfVi1pmR4NSYq9AJn3sBVeMwaWRoembcLKW++Me6hVhGT0B+dCGQYHqPm4NJHuOZZR4bFypSQDyLzbl0ofXi1TbpQKGN3k/04Mx53UyPjGEegm2FapOsL8oI7XDVvWao7T8mhbXeK5EHTriPYIeoEArpkJ4AmHyO2i85v2iGIKnAdSMlP790adDNLgSVY4AK8k7QOmOZ6SdFjFyKh/WhOIuKE0mviD0w4tSLK6vorfPTtAxnNaaN6zQ5f2cpqbGF36KaKiJuZWilfiqt/gFWFF7aDDkJ8F1peVpaMIV7ZXtL7UnLtNS7YE+Edy2IPcOLuJTllUHVM2XUqXNsneyZMHwcV17al12/y1d21jm1l5kOP3VgE86z0hMGSWzCjeWVB7fvMvnIyfab83HpyEJuLXWZF4en5/vDY67ES2IqWq/t9clt+JmV/GNC6+hY8Kxub8BxThd0BpGM+pnmYfzMy3OYLoGPgZdb1uHevi3Lz3a5vlskiUpkxXZ4ePzoiydPHz3e3r6lBxE40mXeALEZwFkcNECaXkANQH41F6RASHGmuSrrOh+frfSXI3HkNAjm6HPtJa5sbt9aXR8QK1ZOT3jemsy2G2Fpwl7e2t4kzjz8/GMy4s6tO5yzHj5+Ouh1D53CdHnhbfdb33So1Me/+mj/8IAVXV01NmlTht7Y6DnWUmKulCRWnqwX4A+FSDDcv3KeD7rKj+WzOmg8727xyfRLorVyG6LA0EIAMRozhkDaie8aOIxYxx0aKHPEkpJWbTuy0ExaEbh7Z3uw1tu+s/X6P//jf9Hvbq10t5nzLdOfnnAcFjPLwpNtQisgSBNRSniFk5HOT9ccE2nmWRzKyZb742Oeas8c0BsTriPP5he6pNZ5xwudXeb44JHm+TSdihErDBhPV1roZ/DVlVubf96ERGZA/V8vtT3ZCmplA5IaXK+75mlK4FqgVHxSoEb+wbA4x4ch+59rCU3jsiN+jCUS3D27LESBzpFgaVVkn5ACZaer7tMrWNSKnab8pr8a0j5vgzu7p8Tw3Zslp5yX63pRsvTCl1fzT3O09JdwPnP7Ky55bv6rHBICqlythbOH1h5oIz2STV2GI79BtehNffcVNyCeNYjZrV0ppxrrIc+pcHJlZrcFTkTHTI2lxJ4bKBySlLM4LlDeS6GCPNsMiRnjvpDa93DJeOqDeqL+eojlOaOuCsSgdOLShpMn8l2Arzv5tEbZjAwLbA0NTsGAWaODWP+YlzVgJxTFSzAshMRg16KeZXY4FEckKuzHymv5jIhBx14ZcqZ92jW7v2ggiOjMi9/tKZPgK5JfyYaGIqLDiaQzAAEAAElEQVRVciafJ7NPOAc+Ip0OF1BrSDmtwWxKNu8nY4jyNxyq1aGYV1WnfQHul66Ws7XwpQyvtDz9KJURA76e7zFYsc86oIY/NjMWJabMFCm+KENN9fxq5b94CPOYpGsUbNToDHfJ1tCEUkKZAOp4eMU8FvNJ3ed4l3rP9koMAAGLYerU8jQ7fgaBaCi33mp8kSnPSofn1jvwbajokwJGFCrDADOr19bxRFlxoN3CYrfXs2bJtGD1kNhpt0/8fGy9veJDY62RvtNb2Lrdu+rcfvDmRt8xe0snh8P+8fjg5GJ37/ltK7ccqnbWl3r9pwc5n93/tOfOwqmFYVKs1Uoy3HJiMjg8YJkqqYmYBBxLqGZzmmYoKu/REHvG/DhQdPs0zjOc+PU3HmABNDFxmQZr68KxiQjhwHdK/XhIIANQtNKQU6XBOfpuLNoptIYYVCaziQEHNIKHoMJwGl7t+7gbUP3nh9ytr/uOijI8Tj5gGP+DP/gDLPnxFw/ZWRFvTM4yNSnGR6PTk+UOLdeBC8msNqZynG9zsKZiDyj6e++8i8r+8pe/pDL6xKUloSj00fw6i4tBDWXDH83VuDCmIG4GKeMWopBRhnJCMzq1M6FvMWGXncCJzGyn0PZwOIQp4XDOqsKAqZ5OEzvhtGgPH5izsZxchP+YNcSBhfFB52z/ameja+/Za1978503v/Hag/dOD2PMHh2dKGRh3iGKPM0XWPis0DvTShPKx3tusLo6l5DU5531lY59SHvPH3/xycHuUxOT+sjjzPZhoUMtMvOUSfwzDdOEhRXN1tqMS/Rw0AhCGrMoWulkKCPEyCvEkM3G/owwTkASUkwWe5aBzlYT7tdAGGBpqs/9aNpaA7IEUMyraNq+tPm9J9q4KigbJpoBPVs42zv4/Pn1Q6KFlRkjs9S1Z+w6B7JciG+aeWXGAxljG2pjRIprpqq8NM0y3zyEDGRSmmk1sTK26s5w+ydjRjJTHpKZ2/npy6TlnzzVjXafgKYGv6qTAXaISABsKgnsUpevVBrkUBrIvHSl8GB5Xty8y9Sytjvot7eFgy9KiPhflwq1P63NSCkrg5d+hE8oS8PQGaSpleQbjWFDlY6F+MrYSpldnqWHHeYLF0VXMf5ik/hv2ZBt4WWuyZ1fleki0JXzOyi+PK3mr3LQr7gAHD6jKDcfK1RRrVEcY7KGOTZVVIupDgrXHemxRVf3C26wogiBP0EVV42gvGUC1qp21UN60n7qbevz9P1X/A1MX7CAZMgsnlw3ofFi2Cyudc35YiPIkjnNAyjYs7S8fnpmm6Zdcg5gKPgLnToeiQHArmUAIk+3e1UQzMjV/syaXeM8aVa9Sj74WcJJ+6C+DBgyYga5ESYTUb6wHOdlOmbcROrMDXAXLkpAG2k8k0Jjx3KSbJHBiXE1SFO6a5Vft9Q4Bc0MELOHvBNxKX+mVyymhUDXtO1aEXWOjLaYizGKk6FRdKYC8C2inAEOGrQZJtlYuIX/FS7aaCjUBGDGV8iIw9HESFgUtA+55gdnpOAJAs63xZLoOX8lZ9yc2r3aOe+uzAm+EMEOPmQTqXZg2+qOMAIVM2D5v5xcUrx6mV6CWSVaUgDwvwRVSf6ItUpARou2OCKwv7p05ZTArPHb8wMGURKjOg2WNreXE5Xk+rWFuYOnT08OD3HT/u3OzubOcbd38OnC5f4QLHhtnZ+Nbt3eXL219Rc//ttffvH5XH9l78k5rsD3ajwcEuL0HFHP6Z1kDW5nzjyo7d6O70WdVcjOjLNE9VlYFOTyzXffW1hde+O1t0QhV46tMesbWwJafvD93/vkk0//8//sX378i1+Qg7c31hOCWViFee64QmKKRB2Tq16AE39jWETcyOI0q36GJRXUkVPBHsdgsZ87acRBeig0n2d+2Lp+cjx8/OQh+Zt1gO/V9ubG9uaWCXIyHpIw1uZXxBjhp8gTZK23unP3zqcffzLaH927fW97e/v502cM9eus593uj/7mr0/OnJu4qFy1hw9BcQcnY5CdhV6Xt7OwT6cCIQuwB+4ohmG8OmfeZWspHgDfoyHM97vrJyNa+PJgdU2/ukuXlgb4off6a5CH12R/dWWwOcCPtX9jsGEjvWX65cVNByHt79nEhUFu7X82fuMeR+9vvPv2+3duP1hZXuNX+uST8x6vulD6sCNoIFAZjmFOUDToHKS51VULJTAq53nYFfL8L/7SUsXZKBZBdmpBxUtXnbdKEKkm5jNjwOulUUfzq5mRM6nZFjNjiQmLS1n1jwZjuvFr5I3Mm2v+YuF8YU0E8tG4c3Qx51zD4dXimTOUV9ZZn4ZsNM6G6a0u9wda7QgLJ5mRxGAvIRKXIylDJ3oUPF4+2Nsrq1UMfE7VysnjYvycd/7f/9n//UJfnbN1Nb/WH5wra7MX+Xp4ZO5CT9wG8TaJFq+XeTcYmShM/BXmYvxk6ZyzWhNr9oTzBuX0NfMwDDAUybM5yNGebccDMyM/IIwhMz25kjezOFjpi5IJaeiZ6OCRTYIkBhvGnCkaXzQWC2QNHDOprekgLVXEi3uRLFQEWocAuecJXfKrUpIWHprv/KfW1o5GH9uUUYhv4y2Yhspb/QhxQzY4aFogCMH1tnrrI03TJMW1f1JcqgpJV3I4bbL6GVj5DtNsJq3QHEhPgI6nj3+OvTw5w2JJcNTfzGQHC16cLcT3Cut1ntbZHLzWEp0qNpztvyCUngbY8bqqoL455JsikhWduA+menIUkZDfYTXDnDLAsLKCz8tleErAAeHQvxJjsiqh43qfjzME2S8qg9bqXQEnvZ1emUK6WYJXaHpmjK5Xr+s+yVgAlzNuBxZTATu6UXQJmELr1Vpoa7GOq2R3zWElIfrcgZaMbSoOia88BdZJob/mT4bhxjUb62pXXrQMudeTmZuW1cXSm6ELtc6kIPL7XFPNHTDP/Cg3N1+m25OPgjHpca5J5VKMwuyalT9L+eoHACSKBK1h7byNSSd1WKEt6uCGK2lJIaJaVJcp2xA/pD/4m9a2f4WkGQo/65VBclkOLI/XIlc1UlE26kpvohqau/DViDsykcqXj9LZylXkTT4l5p8CgjVhsyx9SU9DTI8b/W0MKFm9hm81D3XQh0GweacVaaTG44ULF2s0P9iOV/q9vtLnfbVI8xqNRLt4s7vcf++9v/lP/79r3WX8hseUTUIHp6frD9d7h88pVHNiZwnHf3kV5/WoNCgDezLtjxnBMTfZX+YOocmz/eWV3gCn7bF399c3sLT7r7+xtT24fWdTzGfQx7iZxYXzX7u9/drS/P/h9f/jn//rP/3hn//Zw08/MTHvbt9yft5eTvEzfwJb/XNl8rg03yaZJOZfRCWx+GVcnBfdySYehuUl6xnof89ywyKG+NmnHxM1+FWNj4ePvnh4e+fWg/t3UXrblvYtP8/NP3jtjZW5zujx4+PxaG2dfj54+vTp3t7e3bt3+XILrjmaW3zwwAae5Z/+/Geff/qFVgxW11fnV89s2RqP4Y/RaY3UMFepWRpnsKIGZnhC8FqeHG1kjIyLy+hX3AuaPA/RC8aBlRVHGiDzwnoI+sl+p3R6qzgi68JKUiUXOn1QWbra+d/9r/7k7tYbO9u3rR44iPKc8txZ6q+QXYJwaUYaAgEUQsa8dDKSgrIOIpLGyfB47+n+s8cjLmbsEJT7c17RSJgFWEgX4oBUFNHR9OI5rb1zV/vDIXGE3ztZkOTH4k/esrR06/Y6ZJWVpmLl4+SKDn56vuDlcK5/sbo9t3N7pbfZX1m1cs3SMt+5/RpnwcwsQhSLlPvCyP4i1cFSjUElitxPpshA6B5TWKfAgGHojFSADS6880/+k2eP9z7+pXF++OzR8939g/Fh5+qk8/amJeHllU5OTbbfIuHIgMZhjrartSmWlW8BxC6GJw7rPLm9saX12m8w80muTDcYqy3mryv8L1d0Bvlyl14D6+ZnPWZw22ibifksORUQncPHxia8OQw0w1QIouIbUzuf4ExGKjnyI9UUKoFK1fHllHz08tVylpzqRbBCj1rD2nMxbXUY57xqiWQR1Kn62xqfu66RjUOcQvNCkvB0bciSUpqak0FDWcB33j2STQ7LKknKs72/thvF2zmaIdpx7izMs/D6eE42BmzBmL0kEwIKXbK6EMGUzmHUDNeA4pkBWJqpzdZw4DeJgYuDBkfPwOnx9LwNuAy5vyCexygw9ijmAvaY2dAOc8PP7KSdwCWlJ6/+kSQcverKR9OxCf0tViU1aYUUyeTCSMLwzWTVk/GUIh6bP9Qp5prsj0tDawmNouO5elLf/iPdNK91uZU/edaksB8diOoXflgtj8jb8meECy3/YZuVUlOrPUlLIvQz2V5drjD1RWAnBtU8IBNkCAJDoIOE2hx2+ALQ+GEmUV6ibNVCkzs/OY9k/6ptnPmctzANGELBSRMpU9WY+RGBQ6XlsRULRC6YDBtIaoXNk9rUGdTJjNOCeNH5ScCiE8vfZCxvfN7QI22sPqY1njPysSxZlTX6WFQ8aHT15Ph4uZtAD9ejnMFnk292SXXnV2/trM/Nb965g7BSfM6OjvAoCxVrSysCQygzhFmBeJ5dOLET5RzDc96nCNRSZ8WGpd7yovPvVnq4+Oa6Iy+XeVDff+P1rdt3sjH1+qTXF/yog8+xry6tnFHYBcTQ0n/+7/27927f+h/+u//2Jz/8oTN0Lvt9/cw6eaMM1S/jw1LhHwyOxpVwDROuDH8ItXRQQj32JjKXUI9WAjR1NDwW2eONN9548513PP/F539hDxIGLESXcyp2d59RZDWmZ1F8Y4MXgmVgmLAvYsfhoTx3b98jV1m32tnZ2dzeMG/x4+fPc9hAv7cmpusofl4QIJiiQg8wIeMVig1v+Pnz+hMc3EBJgAJZFIPyammxN9q0RR04ZlsGJjVYN4jx7ZIctLDY641HOr18zrMuysXy3e3X3nv3m19/59tLl/Z+iaPNZh73Zw3I9jTSdpZ7uX/wCA7mmel86Pxni9LlmPsbu/vwzNEFo4PT8RCVOR6d1shCCXohYgr9A9hlFrWyTGa9oyR7dBaC3XvtFi868VTi1tijbm6vr6zQaz/d++n18tgX/JMZSZZXF9ZAFNvc6M+vnvQ355Z25jqDq4487ETLC53TA3JpOdboGU3XgYhI5AVZw7woBlBkMRPSP67zR7EGqYAMAdbIm0KuFlfXl1a/vfbW//o7nc4fdI7Go0+/+OWHv3r62ehnf7p3blu7fd/nZ6vzp/3lnlAhZr5Y5uUnGcEU0UZsV1dXtncGV6OxgVvINkJjWCtTRROaLSH6P0N6IBBn7Bi24z2a+ZWrTdbMxJC1JLR0DYfDocZhXCm/7HOe5cnLmyJ1vvu1l0/aV3J8+UHiTVJ7s5RZ1flw0tz8QXFaG6U3aOerVJKSGmEKCakLr5MCEettVrY8oASZj9lfF4xFEmi0xGtmZ0Zm0mNc9y1diX/qODV3/2xHWsBpsyUmThD4XMSMYFxcHgKQrCuV3aN6RMqCjNCB0Sb2ewgKN8M7e+hStgPG5Abu3vNkUAQ6nD7I1xpf3dSR8szLSEhuJQStmsYflpkh818+0pwMo1JnQA1FzZvAsD1PUlq6r8KAlauC0N6SvOIeEUZiEjqps8uRQ25ox3TYmHH7+B/tjiYpu0a5waJqAp1qpKTQ1+pzIBGHCr8KDC81SaYbnwfFX8DlpYy/4UdNgohyAWtkEaGGly0YnTs2OKYOBKzsHxkezY70ielVvYYmDDKV1uh4VXyhxjdaTuX3CYtOVnmQB/hcYmWWizCeCLMxhNXIEfrCUpGpzL2MeVE3pWVuF35P0CCUDN5k5mo4/blBIfEgjK/0snOkoYFJcCgZCrboAhRsZob2Jo2oLs13Nzd4+Dg2gRVnIDbaytLR3v7xwfOdN95YtTl0dT3lWypYcRy9LWSI1ortSrGhxT6VMixzx8k0FhZevIsI9aIIk+v9wcbAPtr1wWB9fY0GnPYx0UbVxJ6XHJVo1tUSA0EkRziIVaLBdure2tj63h/9U95PRN+//eFf2Ze77rDhzHYUrnpaMmlhdbkupJtJh8uAlvTrOQGKaY52NLH3Y6usii7YLpYyjfY73/7Wm2+++ZOf/OTJkyel3d6mxmG6pAHO2Fy1X3/9dQDO3uiVlTt37jy9foIHb65v2dR0lrNLzo+Hx2JmsZj+xV/8hWjV2rYWsymeBzmMYuYqEGH8yAVJKwFLrlnJIEDpodqccCnRCuJ7YD8eKz28izlYX8K4MuKCg8HFC3IS3mjURTPvnow6jnl8551vf+Nrv39r5zUOlQwSV+Pr0ekItTOf8FdsVq06TjRZMuvj1hYQwQxiEovt0bNnp8dxOsd3ITwbWA4nHKwf7j83qBGwiDSYbVExLaaEF77rFvQq4yuBcv5qd/8oS0n9hO5wAPX48skVPrdwuPDgaHmdAdjJWhVxdGOps7XSWafpkvoOOnOHnYUj6/Kdq0OOICTITnc9sm8EFecfZxNdfKx46JwcoKBWDbGIzBDvgXbudLEvdA/TsrT2z3Su9Y+Vtc74sfV85mg+5f1vrX7nD9/vdO7++89un//s8G/+8q8//Juf732xdzDcW7nscJPZ3tjh62ajAGdwZ6fqXVYrD084QJihQafc/R8JLxc5G0zkiksFcFuzjP7k7CrijZZNm9RyZxqGBntRRDiycl0wlWG+MeBISEWrIYz8ky9f/lPlTCb/7M3NzLPn9jD7+SKzVrvCA14ioUUiAnjdyEB7+YJ0hDnNSgD89gqLjV0irCmUSgZ3z1AVcXS30BsrCt5pT5GVg2uBCGJ/Lufn+GHJBvu1hNsz7RaT5o2lAQpmOImFPH4ZXE280qz4ERB95DfMkDN+vIZdNgOQLvGKvhLUYvGSFUXrsm+jFlM1U5mQJix0Clr9mD8plyhfKCHEMVAxdUMlDI0eqT1/Q2bT7aLWM0h8xUPyJuvkgi5+pjOt1kA50hbl14kotmMMlq298NRPbIia7y/gPC3jH/JvEaW0Dti0oirL5KlByKxL++rtpB3oRfouAyhP3uZnrkLo9vg/4l4tCcSMrX2pvYvFrqNx2af8y07xefwA6LE3GAYv5dfOGowMSF6lLRqTGdgaIn3COG2c5KdinTsfxLfD/MRDIrJljoKADsKwLFcQtKWg3TCpfS4r/Kwd50r2RTUlddQ0DtKnVVLdM35FXp1AZUarqF5V7rplTpUFsn5lRiWDekmsqFywkwmEH8/6+vLW5s7GfmdvcGYryeiYKb0zRukWVrG0RIEiQ4QNxN7IfAKJGSdtMXIJpSLsJK2XT9V6NycbDnyxtDFYswwvRqIW8P3a23vKm3pn7hb3aP114RBx3MaEgq9X1D7+TbzX3n73nT/6d/6Y/+0Xn33KY/vsOCagNLipGRYk8/FSwS8QCHeJ0BlKZ9GH1qtU60HedR34ZcF5pYsNmWyfffqpAJMCXb12/8EvfvGLTz/99PXXHxhYF45lqXXHaSXLXU6aeHO/v2b11xB88cUXz57vvv/ue7YUU5cvDy7Xtzbe++A9gvzFxQ93nz5HLUT5Ssz54r7uRBTOkOSMBaz3gjENdUjcCytYpgDSEK86voAYGJyg0YaiBLgc423TJqhcnnB7RpwWhKsRPIXFVPiQ73z3u9/91h+ub947PjjdezQ09utrG+zIismeh7a+F4tgdEJK+vKKrUmWO3kdOHfp6OBw30L46eGBFToqShcbRdIIO2NnUJ2xj0SxQzeyFBdaBMxaZK+moac38By0lhrX/cxX/v3jlcHi4ur16HJ/b/zoam60tt3fuDt377u95e3Fzs6GAy/DSi+HnevdztyYl1vn4jA/506vBO6ct3Z+RZzjapCJQVIxNRDnKJfCSl9V6DFxSMKA6cc1/7QpQdsqT0ZNT8kEwQER9fZzLtZid3mxz6uF9v/47IAJvDfY/P7SDza/94Nvfu/0253Hh1/85JOf/+UvH/3y8KPPn5ENNvu35q5Wr88WyQ6EZmIiwajUlfh+KTdNKnoFHKBmRuM32S+fTQgBtYX1zNLZZWwnv/0B4Pxwn10a2S7pExo9oSGzIl59kM3nL/LfeD8pYVbUjVfJP2lJUgmySUn71ffiBcCGIBa1mqRWdUz0rxTuJ0gYHRagUiwbMclarxSF2kuR7RRWh7LdKOeh2taeXW7xRXI3RQxf+C72HK8rknPsh+rLBt9QwIQXWo5lLfw3syiGHEgaISxSpGmeLhSHABNU7/xq1OvwqOHq5J8m5X3mmmX9SHZcSTDAOHdGG+5cmCw1RHAp+nKCppmu84scI6hC1UXgKJCYAwEMuOVnroYJ7blgOB26KeRMj0gGdWmf1zqnVfpqbanPUtTreUnobshBBKuRnRT5j/OneHCQNRM3Y5+ZVl1yz4RvV4hT4WpCEOQZ83IFXL/hamX+hgyvvoq8Uya2cKbseyHC2zZq1w1LKp0uMpchBLwCfVoXKhn01ZHGfYPKeIaRK1R2NzMlUMTYWsZzl9bNbFbxWQxZqFqVES6g9GT2p1A20YashAYsITNJzD/PDbmj46pbdQGJu9Zz4cmYqT1yFUDBN62dLFvJlMoKZvkEMUkWidqnjRlxz3NzbI78hYhlbZtAPltb7zjr9WzcOe6aDeYQ7M7aHkvr8HhndRCE1x04qSd8cSkaZimcc4b7Wm9wa2vz7tZgGxN2PNO8cJShsAtzDmkyD85Hx6c8k06P1zY2NS8WqrDMjHBrMMMMtZUc0lsbvPfBB7/88OfPnz/PLtVGGszNyMIg5WvfIZeZGDrllxJIBhEGsOdLDsnzPMW4QHObspMYc2WOVj4W63Iq39tvv/3o0SORpzDad9575/GTR198kQMYeqv49ap1TQ5Zn332hU/ox1q4u7vrLIe19Y1ev3/77t2z81OA+8Y3vu7Vn/3pn+NsvQvTKgqTNmkPxNAYEoaRDNKEomcCoN3+lREj+8FQGPQhvC3j5E5lZZhbOTrgj3x8cbmwsXb7ztadtx/srPduf+/3/4ipeHx4xWVdoLKN7h0z+vDZSNAMhi2lnVyenI5PVMTDjUaeeOCgiVefDHNIH/izYnOMzlkUmXgwHeDQLEu/tBHRVbAxDa4FOSgasUbLaaH4Waahxkrhoc5tcvGiu907On9yNHrc6Q8335t/8O7G6lvbnXtzne5upyfEbNZxHZREzXW2MqzhP4Yf4ozmW6zHoGHk5y8WV9Hu7JPMFBM8MvMCu6VAHYZyhfJmZlRj8xGRuaZP0MbkchXMr/rOZcrl16lEUu0SXf6qc7T3Fwv2qxNsqMjvDh68ff/B//zNzqh7+Fef/+2f/vKv/9XD8bNn99dv3b31+tXp/HDveNW5ywhEyH/ovO5m0PI/RMw4qVa7TWpyVcnZLzPgasSXb2loXSH3hbHuTfo0c2do/OUPW0qbJp49tOd2f+Ut2vFS+qS4glX7NAOaGTV5U3/0qf1u32pYSqk8s3t7COsVQinDhM1MaJREOCMmHMMyZpw9RRYSwnrN3uwFyxr93JyNp0knGAqUR4J3tGqc72EXgU8DMtiAzF0yJu6ktL9RznFjrNowRHsuEgQzyV7IpzFeDOsnKGVwBsvzg7W+PQZWmgwhMpHZx5LkdA4HpQgLNV+HKym8+h7l2kgTCpnjGlCimObR16hqaq2riOjk+eaf6fuJuMNVTOsn8IVFJliMzZfX41POHUtM0HTfnJ3J/aTDh/OUsnKzuH+EZ40B4RIM065iWmZxqXSlX6YLeQNU+a+YX5Hd1pjiRlNDUEua3BX197lKFyX/Ez6I82yjzpxdRhnszopKAP4MCLG1RA8o7lVt9jRhw2mwbZWJUZ8mIBzhnVKyYn/BqDsqdS2SeDZcKDEkLAwf8phuTVWtLvosGYpwGLgw4OBRKsqVn0k0b1JTIAQ5MBp/AiulmgtpZ6l8Ek3w5EzmVBDykawpElmTXUuCangJnlX0yjLNacicbPPLl71+BH9bZPaOuEIzH2+si874mjiHWcJJzP4EichCYyQkvVZ91FDeTpvrA77Fy0hhIMj8azVW+9DczkD8jfmF0Xho4zAHKbqKJpEGrh1iUDuwTJj1ft8bjWa8pgfb88MVubfYx0eKCGY66LoO6xr8ic6ZVfGEA3PLdEP/sUwT/sJJUVkkTIANGx9K7LCaK8Yy7BfbmWuVxuDx73zrGzvbt54+2326+wxXlr6+vdMfwIVnAlGsdntb29vUXDuD33JuA1fd1RV5cWt+3R988P7YiX4/+7k1YMsZ5ipgG820xmXyLrBmRK1sVMbL9CHAiZk1rlTcjEhENqN37L7GsG3cml/g/TzfWx+sv373vfv33naE0dbmvcO9E2eJ9bJDy0JZBl+oKlv6c4Jq1k1B4iruwxb6cQ9uLkL9jUc2X1N9yVAoGGFLIBgDYkMma2uO0Ug7uVU7M9RCGn91rI1saAg0PVGmQdREDZLF+EzkdHiTMGaCe50MT55ddPdWbp08eL97/1uDzgMs/4vO1fPOBtHTeYcc4PlknQu3tQgfuFSfndpurFy1QkhQoAUkZNr1gd3OkCHTw9wPr0+dXK2Mbpgb4EXsajNi6XQYgSAvvUliUNxzorNqJ64vKdMmrZbS37EdQOgv2hYT59L8RXf+em2Jzf2fvfbHP/gP/vh/e/bj//bDP/8vfvRnHz27vbHz9lsfHD09WYo/n+nP/pC1DfdqmlrINIrEd6NWBe0jyVhwKoqrqcUDQtF1JqQj/2VWhaJECPMVqHqjhV7Ak9ZaYEm+r7qC7Dde5dv6fJbX+/b84uHlkhrgin81iqaEfCKX1ufbohSeJgVVceamvxE7qro0w0xH6BYZAiLK61BSiHBkuAhMFeG5NOB4O7NFZ/XXKq9l3azSYnGCXiXqJGWQtsOLo7ysOIVmdDPFTXRFMahxAF0xi4WGowBIQZFK3YhLuamvVeHHJvX11UYOGI8erTN3bq+/9c5bTpzBgBOxTvtywreNIAlecGJnwsWlDZrxXTge2ldX/Y1GD2kGpelUr3wHeyAWWt36n2FqOFdAC4ACna+6Eh7oZrqygAvmG0ZnMIgcSxYwic1dqGCT4c3M/wjP+tjao/0A3QbZM3yrsU+VeRXkDC9x97PQV+78bI1SSGyV/1BXgBSjn/LZ1nhj2JRkzyDOoRYNcMkRzDCFgmZ5btPM6GfgMmd41WRGBgGkuEMKa2YR1zKmCk+bMcyUUj2JiSI0wq+0AIVjm2jMtVKSmE8QgDaOIJCrvvYcZKwMNXsDQpgSghmBIIEsGrj8ActcmHUWGhWQxVum2lj5YvSGyVF2Mi+sg8lDZx2PDw+yTzyBlxauuFMxym1uf/d719jSn/35n3JTomMpsSemmcBbxI3TCHBMpco5O+PIRQkbJabh5fnG+hrw8kccDo+0UXgPFoLD/eHSymp/lZcvT+icIJSNJhwhiULWkVfXcrjP3sHa+tr7X/uGOMzO7LNEzTqVKR5OxckSzNM9BkMA9zmCFkU+V9AGjTvlIsUjlxRh9l5dcWiyyvv08RNOYZRaNUhk8ebXTAl+8vGnXK6YpQ+ODjnj0CZ96CQGKi91eam7cv/+ffzVSjBFcn1jbXR+vLa+TqXWYWvGX//m12nYf/3DH5FoM28Lk9OUCASUSHzW8bSTUakJENqOgpvBNr4w5HNZsVOaaG4p0W4letjO3Xffev39u/fe2Nl6QKLg5/Tskd023KD1U2wwWipQiOasis5Kvy+cNfDTexd3thwObTn9+Mmjpw6dDNlnc8t22OAsyxztJVz1guQZptowA1nF1KUlKgmuBYbIRaANJ51ewjGaqsr1+mKRP/P4YvnkYunw7rtrW29srL97t7ON0TIy7wG8c0fjCIYmrnRW1oJ1iPD4bF8wFAC/Dlc25Pi+ky0sZsBNNqAsnUzxlV0jqJxpr00arSFS0nyMGXivVxbXwruA1MyIZIOwmYqYIAkSFmdNE4pEl/dz7nJ8OgybU2JO05i7Wj4DOmr9yd5Bb/Fu5xtvfPP9H3zzP/r+kz/75Z/+t3/5b//6Xz/YfG/xok8LtkpH2V4CMLMg2plSsnpgDYGol5Vc9aVKbYcyVccLeiVleiUxLLnoQJhuydx569kdIkDI9jz9ZvJXF8z4hj2vvHrlZw1w0nxy84IB1byA0KuWLcDM/I8mohdpvlxBkcmDt+w3YBvaB9LZH+Kj2CGsSeQBycmYzAYO58zpgShKrezmvDkCD1TNFroogfixeW4rhW1eYeClN9tV55mCkXqNmgcrFRsbfRNWjYdi5w0x75xvFsE6UEyrokk7ujwBPDLQg97c+vrGvXt3vvb1d997711W3vHJcQQdSJwoNxRzl1BDAnJ1xudXHDx3HV8ttKpW4cd4s4O0nICNr+tAWsv8h79np00wrMBSYIIEGTKXP4g16LaHlui+eHaRo9AzzDT4IKE9BxZvLr7x9a/Lrz6fEJwjA9NvgvBhHq7g9PReCZPbK+mxYdbVMs+GrSVmzk2vjJhBf5EwfVF/X67X15rqbunU6p2gPj4OZrfyv6qMVzr+UuG/4QdUylSg/GlqGJfZtLw830dYK1aJrb02ybB5oJK5sIgCsiIjfkGBQpcr57DmGXpqnDfVT9kj7VxkUwpOlpAJIQQkag8TRIfVoTEGGPrWAMW/NMgHJLHt1HOMSVWm6kJmXKysibRswlRWlCCKsMbVGkaBK8gxA6zn1ipFF7ZEikT6zRxANjdC0RJaAVnJHMD/HD833n+0sjggfA60bnf/ejjub2680++987X3bP/9m5/8+Mc//YmYySIVqpEAw0S9trq8vr66vUOxXF/qLh4OD57vPRsdr6LjoT04er+HlLvAx7S5d3+Dgnm4d5AeiYNRHJFE+PzRo+2NDUuvpwd72KTAVUJWwdgCiZICSoCjp6WXYb26H1/fTPYofjj1FXMGBZ2cS6X88Kc/s47L/YqV2Je/+tWvdna2uVDdvn37L//yL9miK2Xnu9/9jvlmBq72B3yJO6xEF9cikLz59lsPHz6mmm9sbxHF4Q0xn0NZlNa5+ee7e9aSz3qn+PrG1ubwcMy8NB6NhIU0oXHo7a07TNz4YlhC/EABvJl5uZ5Zg19c69mRtS5Y9uGBmGxLG+u3d3bu/fv//D90XKCgn9ayj/fVSfrvowRBFS5wukkIwUqvHGedNdrO+TzXytixSCOPHx4d7CIvHKyEq0AAs9gcToaKuBuyKN5QIFOreFbWqeHs5ZUwpaFQZxUdjHKOw4Dn4sqJ2GqL4n4Mz+f25taGW68vvPaNweKbdzt3YOR+Z+5xrM0LtGnEzTrfmVjrqSkUjN2RcGl79Lnyzo53NXylSD/HZYwoTFMWBImhABUCoDQkQYOiB5GlQjxqZk1JYCYbhdO7UITSjEURkA9OQCUYFuZGrvU2opCMmaWZNaGclhgv504jhnROOltWRq47J3vcMDv379z539/63/yzHzz78eH/8//yV5dDm8d5yvTneWrb8D3XtdOYz+7ayoCEQujUPqc1axn3Im1WV83WamvGK21GykJ7a3oaM1eaZIZH8c/eE58AuLsSvG0ap1F2tQJzl808142qYnb34JN8mjmVq1XpDoKzZwPRyIoSII+vksG9wNdK830e6p8HV6qTWLJkfmYstDF14Z14U3k6yxWswq5yuAJiZrAZPYji1uk7lw4sp3o6GvMse8EuhZwUkk6PxT8iLUGV06tRjDWwE/GD4qLa2kPfuR70lwVgFyPWjFYhUqMK2xNiDbGTepnDLC9+MYyveE1+/tknd7c22HJ5Vgryw/fz9mYfITo9FY3fudcRtZl7jo7OF9f6tjCsDrh4Dp7t8fjcI3zr1GBjAxl7bjHq4Mi0JSLz0za1pfDT/PThPr/GIzTBAl23h1WDyqA3SISD2EiiRTWQGoIArRY1sfEz9iBGQtNAZNmFpZXt7U05Q3IL2sZ1cqWEYMbf41Kxr9o91TcUmZU8xZjfvWTGyTCkYF/aBgcR39+9mN/8hfYq08RVBdNf6cE1j+EAw2xmUbAyUk7gU93MB/kOEudVrvbs3q5KAwyEgFUQBYu5zb/0gFQRiBXHRmRgb8gLHA5ryYeYefHWGTDbQ5UZDc8DfKqvJh5e+ULripOm2V9xKcNrpCpVt1anY1UTAOe/kCtASA4Pq7fvBv5iHJhoR2ImjGCSgEnXx0e9u3d+gIG9/x6+9cknn4goje53V3tbd3fuvfXA3f7OOSbj1eXtrXXyruKiomqlsE82wGRPB7o1z9rr3OLQR2c0Wck8i/vM4e4eodVIrPbFJdXoBHEvLQioACgwTz+ALPcMTuBXaI/iJmf9VqPPaIoqktliMAU3mvl4jD1zvPr5z3/+1ltv2dErLof4lxye0USaMRXNM8YJJJaBzTHPElnDRPXKOvhweD0/sIDZXx0gJ6vrG5Rm8VUcs4ip/+1f/1Q58pPYqL8oyOlJhF0Vr2iESBMLKyQxR1KTLvkdLy/3R0cXzw+P1HZ766333/v2++9/+/6dtw6eHs9drF4lMDNtFzxALK4J3cHKyfDgZHgosl03Zy8Ybva/i5wPbYE3G6b2z0+GCA6i1svOHMyP+gpAZlMQrlAu90j5cAHNtA9PbC0nTC4t7+4+p1HbQrbc5TSeyNjxhbm+3O0cjuYOe5tXb35t4853P+i8icY87lz8vCPKxTxYneQwZ9OHMhuZ1QFWIwMbBZ1oMEfdTFCVaz5O0IxjDHIYDd6Kex7p15dj2kCdLGlcSxRog01aCDrWKAcxPYcbkY5ElNMGHKjGOBAi7grEApNJ03kVxuehOA7LQUz9In0SEerMRP5TzuBilx53RnQrGwdXFw86vVud1x2ivfx/+uZ/8qP/8sMf/Zu/3X8+XOkMOlfr8xfZaLTed+7Ixdhxy8QFazGnBzHp9wQbN4cz+1sjv3QP9UhrMs3zDFgkC6OS5xtXcv0uV/v0lS8kmkozWuQJxSAS1Ognb6MAN79qKT5sieG+mZetbUkzsTLzImiEB7dsMkn3kzKKQWYFNWurwgDQ6jLkDhk89VbMWshE0mCLLsU31sGEIeF+7zQQTpYJurK61rM8ZEpQoXfW+/fu3HJ4F28MAfPIKA6dplJ2+72jY34kERI7nW0eHmbrbYsJVhRWxS94+9btbXNWnOPe0uatrR2dODyct1dw79nj/b0Dgviqo1kt0sxfvnP39p21/nP74hYWHbTqoDc+YoIBOMPNKpN/j5/ufvbZZz49GP3/afvzaNn2/CDsq1PzXGce7nzf/LpfD69HqaWW0IAECATGRh5ilo2HeOHETlachCTLf9hx4tgr84pxHMJywDEQY0DYSAIhhCU0tqSWuvV6fuMdzz1zzXPVyef7q3Nvv1YLMF54v/Pq7tq1h9/+/b7zOP3Ku4+/8MbXOv0BOxPIIsdRlkVc10o1QZhc2tDIwXDdsLekghKiaiLICojqACMBkuerUV8Pu05sq7VfzSP6HUD6j7StQO39ny73dbWEzxby6T2DKv7DtyRJPD0NM1hBw++4cjXm1Vuszn3//tOr/2H/vm88wXVQjTDJJqvZ8rK42k8CdoLjUBYS2gTPWs1ViJzPxMmnqLWCeQwByCNxkYPO8b8yQipOll0j+hPVY0tLEH5d8Os9AzfSmP2bDoUSbHMscNUJaVtNeMLeUJghVVpIn87x42oqfseMpZsEJcPBjC342GoS7YB1lwEen4kJJx4fSVS+mpWS8gnL0nl7MsyNxq1yUbhAvjdn8Nx87u7mwf7Bwf7bb74tg5bU2VqXPlsvloQ7haoAWzCtcoMGHDn2BGHvUFDZuMSSh9Rmzum+a3l8C4UXmIUggV8vBVX7vV5oKWFpTna5Qp6oG9w3GLCpDSECsHgLi4EEeG2zGDQihMuYClyQ+MMB7BpFCo1BxSuXrcxCoq3e+OIXYTsG/O57b3tX8WUGCduZ2b/85a/2ugNzDl0fPTo8OT2XOtxaX0dj+oOhYsMF3QOzayxkjZCJs4/v31M5kuKPdnQ7QwZzC6QEVkEwbq5y2jtiDYsMJQIdjVf5H1bkKNVeVZC105s2a7svvnL39s1Xbhw8V69tEhvGXfUzN/BQVIq4HOZ5IQWsMbmc/sxri2mxuJTKEAFWAkr6XS1we72JNzbz7C7sf6Yu/hJvZhAwPWRMn6DJy/pfryseMsZsTgRTxwxGZBGXvHmz1W6fHfYeL/szTuFcMbjqsNgZrR/dfn331ut3M/tsuE8yi8eZYj9DSBsdJzdukPyQAMPCC5anuct+qLxh/Ai7bVQglywVDT+jV++ccqRWS/rkCRS3E7lwYCKx3hAVYsjBd13qrrYr0LaQOED4MoB7+Ha4INiqGLLloIAGAh5jNruwI0YePU7jC+5rDAExQscE7bh1hKE7E32Ua6osXFRrmpVUbC1WMnutTObiQ//8qy9/auen//ovfflXH20Vlnutzek80+8Mc1NonGkqIqJX5mVuPOqOJxe54gbjUQzUaBP+JeeIPSOOY8Z89Rng6bvXCH4Wy5E2Pwf4JsJkJ27lxPdR1KtDcemz3dVZV5/Pro2deMm4Pj4S9TACF5JLn12T6Ep8c+qK48ZQ082fsV7MNfbTcT9ZvzTM+IRuYYEhyVF/9XiPhN3lcKb6vBqTguajdy/LcajAa5f91HeBBYhoeLkmeEHfKrEzuZ3dLUIk6lFvVOEjBESI2YvV9qnXcNt8pZwjaJLN5bjDr3K1TL6cLeSJqZcede4u8pnheaVcKlAvD/a3sc6O6uXd0dZm6+7d6wS+UjF7cX76+NG9w8dHkxs3yqXi+fGTRqV2/drNAsffbKhaQZW/uZDbbDaoR5q28S5tbu2IZamV8oPR7iJbeu7lyeb2/htf+CLLh/hMs0J11lQtukgEWll71C2sRer/WSJMOtYSGVJjT0ubtLJRlST8fCtEtBBOScuUDsQyxWr8t95W6/3tp387fKQzAxSutiCY/t53xA/BfW3fhI+n57z/SDrlH+PHCuBidkwLt6rKjsrfoTr+sKLQg0XZoYHpFYJLkd1i5LQtfDXhT5JqgY1hBfcKXTepk2GvCfkehocXMAgH7kshJhSFIAyEhWtFj+qYsZTYE+8fnDK4TthUw8XEh5FoTQRLxpAMNqiSm3hUCPxIObXC2Hwmz/C3zE+avTBTrxA8nKjpd3gVyx28LBEBpCjuF3/Gnx3PR7pTMQ5l6vVCs8EiGWLnnPo4H531mOA4Rws7O9defZV/s9frVrQAZu1EX7FTaBk2H4bFtWFfg4EouBYskzW+yCbDLMrRftnN9sMRuQjmp36SeGV2GpWzaMNU1YHFyOfGQ8UixqHaRMilwfkw7JgAX1gcWaJMexhjYzFMm39hAY8lu6s2I9puzUKj7fWhOh4Ja4jqnEqU4Hv37n30ox8lJYi6ogJCLeYvbifnMxqzcOkeIYhZ3jA+XVhfRx2Ys+iYHqOsrdmrr294nPKJpr/WaHmF119/nTL/4P4TFB1R6Pfl5i4r5XpU/glftyRqJKJRzlbopfNx9uMf/Mzu1o1b155fb+3yBDOq8Z6L4K5WailiQ906ta0UaYygpSAlx8eNqhIhlSCAg860fTbstdWObF8MuZRUbOYSUSs6/PGsLwpaBVgSuUx/QGjMmg2sBCw5BjpDGowwACuWmT948FatVa7czM7XdJhsyxRqbDa2r19+6DtuZQ6mmY3TTElJq9Pl9HQ+H2a7DN/OpEyv7q9MAnosuGRSyPTWLscWIyg2Bd06Rcv1zKTPJx8FbzBg9DvYMAas+QppEh7wd0R8V8AkoTjkBYwzxnzleMQM4huZYpTeBxo5kpTdQFb5dETcxH01yJZbxxNBBpNXmBGfLtg2+HGgtEsMLZZQ+GwID2vYgJUaTDvZxRDBz5F4MkfFD9/8Q7vfu7n7W7/0M2+1z3o3Wy9RtFtcA7PL/sV5p92tFi9LblhCRoFbiPKmNk1zzHPMdjwjzfm3UNckb38rAyYWBR1Im3da7cTF8YaBqM9+ffaTnWcHnfZs387q+tURn+67uvfqyOoOqwmwD4VW+8GJA8euzgqJ1teQ44LdOieWKD7gG/IEnC1d7FtPxfAYmbHhAc+6Khya3GHAEkKiIsdiCMCSP0LRT8YcNoP1VqVRL964uccoVK1WBG+CSYNhdiGEl4o1XhvjSaH7fVwNcMtmvDg70RMTXwtuXXWbQs9cLuet+rpqAwT2i9OzB++91+6cS7ujQkvilwThj/Po7OQkQkxUXlosNuqMc3MN2J4cnewe7MNopVnqDcViCs70iECmXIG9utgrdsfTj330o/vXbr363HPoBohBC9XwkZeIUETGiwS/kEPEedDhgj7nJymthVeZvs6ktLGxVau3CCOarMeypRW4guakRqQD/10+Vii9+ny2us923HH101ModCDo/u/cvkX3Xf2YQChYNfBIm3MC+WKRksTg83e7Vfr5H/oRd7oCyvCqmZAIAIqEVOn8ihphuoGsoWMG+ONE4cyPt0i8MwbgxbDDmER3uvqMY2lzZRCQANfw8qaevDQN1XdJzVhdyEpBtYLWJBa4ehXqadiL3D5RrkDRYDuJu0aqWrAWDzYYjJlFMlprWPG4YYw2hmKQ7vVt0kzST7wCDAo2ZXQxtzEFcuCCcMQoVjEgq0cWscEKa2+xnAGGBwc9s0M3bZ+VWyo05Se4x9lZrtni+9zZ2lBiqts9z5TWKpl6oSYlL7oXiG/yn+inUl4ZCrBZKI0WpdFczpf82ijGpgMEnfKijQkgm8N+P19vII8hJVgWtauGQ+wQvjEbUFsAuRczRcGC0xZgb7xe2hUxpWYgMRThM1EEJ8Ny1VssYMv1GwdirNwtZNhazXsfHR5WPv1phmicGNfHWQWFEbL9JFGY2WrnYJ8xWQbwRbd7sLHBIg3jM6I2rMp0fn7RYwPAHnd390f9nkxjgUbXr9/82MeW7YtfbF/0DCZRik1NMYbt6XptA4MfDxZnF/PtZuX1D3z8Ay++ttHYLvIyXpaG5+NBr+tV+KRr2+uCRPEl8hmzrVcR0IzgKdu0dWMjo5Z7/7x3dtq/UExjYFJQrs1WJXQRpjCaMKtvCNfUfWRStQr/rTStRPeBh6cI2kJpeKEzk4gcDr41Vxiqvp0fLs9OFqeLQr95p3znpYPNV29n7igQ9U5m9t58iMZmONKy6w1CwWw6MhOAXBpxQgxwFeNey4zWVNig7+K1JMzICaBpxM6kl+pnqL0Rwc/Beu04LSpmkqWStkykc1eZtRYi8TMrbi/gOhhcoBsff3T6DSQCCgDZuQAEIEe5Z9pwZlZYmyPpbCAwpTAptEAyKTe8TKGtBBV0o1x0C3O1AG0wHUEnwAl+Ldeak9HJ1yqZs8yNF77rn3pl80btq7/28Pidr/bOp+Pctc1Cq7DOia7gB5GaQYSrAm66YQwuDXGFzwlG/z4f3i6WxmfCQvv/gC3Ynif8/bY0KelOVzdMRORqP9YotjCJrLbAkm/bnOY8//s3UCn9BXEK3IrDdIhYn4Sc0SAsTHwhe1FwreeYl9dOMTed50Z6ZWkxEOnedoLGMUYZP0pFV2028tvb9Vs396/tbwnkNO+C9qG0+uEEXBoOYlat1jAvOia0rggD5b9PcYcRVUPUL1WaatG1WqRn6CyYQ+UZqU4nx3y6YkhOjo4OjU9E5Mc//nFhWXfuPHfjxq1vfP2dTru3vzffaK27S7vdeevNdx4+fiRqVCXEcl1XsgWPUk8Z20CM7NbOTq3RpL8ePXm0sXPrtVdevnPt2pe+9CXy+mhEQa+0jutrz911+tHJyfHpyUV/YK5QnVw5n0dQQCJdArWlSRMECsVar8sBnuY+pKUQ2mJug2o91YS/bVX+fgdWePD+T2euvtqJu8dqxpHV9vQ+gUOBSb/LtoKS1aefV2e6F4JskP89bVePw72Sksb9qJTuypvkuSvt02iDfoXwHNKAwfiSMq2vZII0VL9TixMjxE6c4foY9FMeDFw1KyB+hQasmgH+EiruimfEjMUT3rc5Yu1srKYcJSEBhPHYs5POki1hnCvWvJqr9CY+vm1uY/ZiAkN4DW5PdwvGnqSOUHcRMo+OARiL0RkTrT9XnIRndlpEJlmS9/dDdJsM1VWv5Au1UlEC8QzgmrC6YObSWU/5oWV2UMjVC4o/g2cPFWjPNhtJMWt5EjGS6y0wksjWW/RxO4uryyzjJ8YXAT/TOVWVwVavBy5b80FrFHsc0xRCCBIXU+RrGmsctes68Zk2c+XTCc70JPIUnd3NHX/08OFrH3x1b2fn8cOHXkqjBQZY3JFDF/Y6nZMJ8m9tLT3XEcfpyroKUZqNh5xLzlAnzPua/ehgKFjsrA2T0fhmax0vM+R8vkemfu7uC2/dut+++Cos5VpVz/nw4cn17TsT7XQnuRu7d1/7zEdfuP1Sq7LFK4knqStpiZQQW6+lpBuUaMAeDno8KgoIlCoyrGsBTbNRZnLRkw715Mm416E1VsTmKgUdVBGjBWDWj8c30niIIVaV9cWFV9pVmkZz5i8806kNGqlIwd7RcjjQc/LyYlboV/dy115s3nz1ueILG5lN+bSn89F7l2vHucokv1UBSrNZZzGQslTR4dlQjZO4yaYbJWsiLiwSO5d9puYwMmutFNx3FH+RgBm+QZ+cwqmGM/UZD2ZkD3uPnSRZJW0YtQ+aDSiD8YS0iRJESFiCATH2eK79OIWfP/zjCdXUhongjZCc54i+T1/zyoRFjDn3IhXZn5CqjIRDMiDZ2rPhh/CqfIpsRuz1DBt1Ktu1zLI/Pf3NYmXn1T/60qufvPZbP/eVh1/tPPzKg7efPLhW29+/dn3RW5u0J6ItPf9bsDcB4e/OU4OgfRPPVxD77PsVAD/7vrpPnO+XgG3bagZW+w4++xqzkc559mknttWppjEht29XR57+GwCRzoI/wZuTyusgYII8cI1EFD8ldusfCzym0VpmbJjFQ4rRcj6U2hs2jmV3Nu5xjnMAZxYDlhi3zGWarcYlQC7nNzfqO9v1a9c2X37h5nO3Dgb9CwqzvIPoaxoF9N20ZPE0/8IsZd6TlTe2tkmlRki+a65viOuDwp4rXtKZYh43N7Yf3X/ovaBwXx+TvspwufNz/PXd/b3ragttb+3duH5bSTYRW6TY5s1N6rPkw9Pzi2NBD6XH5K9KtaaDOKg+OjsXwdYZjYma61ubovWQERV76AwbrY297a1++4LBrFo48CmjaDiabG1tbJ1uHJ4cC9vq98j8KZwhytqO56USLt50e8JhCINXc321BN5qRdNWa/SP9Pls4d9/1QoInoHC6pz0mRhDcN/VFoj1dP93/AsbVj/5DM6RzkRF0v7qyLezmd9xj/82X91wBa9ODhBHRMhZPgKhg6oEY/IZVCBx3xWsE7P9F7+FOp6mL10fkLva0q0MnkoaoiQhMQW8iL1BnrA31kSGGQmiQXi9XdB0USI2UjyVn61GUAMjYnhAv4lvySgdRS1EDSTNIH2iSTGe1cIyLf7ubw7AsaqETzGNUJWwkDYgEaTsCjCCgqdZSebucBjREEB+tSzgQKxxfXNdaK8cA3E7qgrLt830dc6Z41sMPesb6wo+gzc1oBua+7SagpGFLYrD1a8wSKyuWzKXsNrprFRgydFSaKxgBNCkK3MP6R1E4DVL7ikuC+frd7qg1Bb5vhhNetWYRpQhxhqjXf1drUJANOemKCflmURCRedjoYzwTXIRfphmeEG1JUE7rpMCLkvslWHkJ0oqg/OXvvZVuGqaoIxMX6FJdPHWbgsfiPCkXEGFTTdUJnqj1dTXgTJvMpGJeyf31lvbN67ffOftB532ENEbjyfbm3sXx8NPvf7Z3/Ndv+f6/u1Jb9E7H06H+XqpGSHD0WkwBDsmfLMOzvnAyN2pIaTa8rnceiGzt8U6w+b81t/+nJ4PzONcnBWCUeh6oSn3ux1aXdGaMLpaUEDksHgj8JywiJQCwC10rD61dzwqqt1Xiq71YyWw1nrTwmitMt+51Xru4weN13czrcFy9Ga3f1yoZ2o7+cW4xAyxHC+wpzBzszRqWtAZ85OFfxeYCrJBmYW8am/Ds9axujnJyQ5jwwyRl5LMufJEQiHSYX+OSo7MgFEIGMcNop8GaiZWUi4JFd815hUP9jPMC/uPFxDBg4fHPgQV9xWOSu8HmZzibrCARMCgDXdlweUup3RxDZ8UBIu8aX5e4QgM1ItcoyhK1SmSo8KbkcpZUt111xlfnJUZPXfqmeH9efsRp8vrf/zF14eN819+9Os//ebDN07vnWqFsZvPtOjvTWXWIufQkNIM+wzRIHAzbSuCttqPzzgtzvnWI1fE2TusSM3V3ZwUCJDOdeGzX1cXr76ujj+7XcKMuD3GeXXw6T2fnXN1efrHTVas105MuD8b7GdiSvsoETpmxdCQydpcxXBcDgOWVjSW7j2bSjTCgAUg4L4MzoHmfEhVwQ/R232jUdM2bHO9vre/vrNV391p3Lx9cP3W3qhX7p6fds/P4EGZ5CPxV1DCWkF2kVI8O4Uiafju3btQWDRk4HK+0On2u73BeHYxHI5q9bqBan3OuToaDO7ff9jvd1tahrW2VNXlg2pf9B8/OmHUlFxw5/ZLjx49wgdpuI16a7pca23t3tYzrtEA0xz8vemwMFlc9AbCB3LdYe7JichnqQXXb9559PDw3bff2ts7gJysZM1mQ1Heaq3c7nQi2KUcHugbt64fn54+fnKEs4NEnUfIH2vORHSCoOmeFuQrROugW3gDAAe+vscaBfynLX4GIenzd8JNHFwxzqSMpXPiY8Wf0g5IACIqn6zuEDAE1uIJcWFYOxN/Xd159dBnj3bS6vjTXw1xBaRP7+aMfzxb3Dm2ALMYKVKJFfFiShFRBracHEerdEiv4w0DxVGxq4EEG1jdAawGaUszFjcKipGQKyypcSmREdzGzCt+jCyiKW4eJjOCOoZPmVZoLWjNCuvkWoYHj/cYy4LDEeHn8RzRIYRG+hRXVww9bbGgoQxcIVpQ8NhivoPQPt0S8458p0DDGCH3ayhzNIjEgOP90B9bTIiJkGIRTdMoVG7FRxdUnZRQaFRV+w93I2rXn2bEkI61YsvTk0W0MDV7CYIftQPrUpR41B14hE5L3B9h69Orp8AjF92Stb0btnuT4aBea87HaiO2ycEYCGWU8EGOxfNA89lFVymdGJ/o1ZgBW/DfKIFlo1PH+4SLPShIHIkXZzEIo3W8RV4ZjfDXtTub21simYcyki8vVaje2ljXvL3Vqt28dvArv/I5WVMkayZLQrcosIuLM3E5lkCFTXZyAc8Zb6f7sXqbxXJ9o9kd9TUh2NjdYoMSyyTcrlRuVkrrw95yu3ljt3V72b9QsMTgP/7xz3z2n/nhwmVFtuv5o6H827WZ4pQ5paeFoWnbwfOhhgG/tTTFkI1yuZ1mSyUCvdq65yfZ3lHt5N54NDx58ojiW8yq21wgwslzFoBJh7OSzWotGBqbblQWsIRAJ2AgoCsW1NSESR8w4nT8u5nGfJi3JsP2+Hy81q9s5/df2tgUxP6Z5zLFdib33nx8vqgMajUq6+h8MGjkq6TSCGUIyy8jsxDrLL9/lOx1W5qP9+QZU+UMI6PsqmMxCjq9wCijHHZubQbKcwt5a0RBDBi0MeWvNGBLGVQ+iFB8IlEgPvZAcKIknmqRGaVdFQpk2EUD0v0FR4iSVEiZ6toBA0kf91XYgFDlsASwEixMeCCb0kthKOhfzkuLtfy0NJiV1suZKrGF+ssqEKgT/v08Mz6+PsmOKGjBhObj7vLivcJ8Y/MHP/nDH3zxi//Fb3z+Zx8rD9eobcrihq3CzyJSgQU+qswGG1F3X9Q1VACL4XKn3pk4mroGdEkfTfB69bEimLFYsVTxaibBzmoZV/QZRMdP79ucHxcGUU0z6HuQmihl4E7xF7eL+Nygb3FayCpBMb7lPgiVULaICZX96RnJCRCPEWzsTHPsiDXmnUVEyFnUBZFUk/mlXu6jS77emWrlIq148DUe92uxWqjop8a5W+PhLebno2p5bW+3ee1gc7NV1uhLiVKELVfVYi3TE+rRV/An2jWYSim7XdajSm13c2d/b6/KeapUxWWUMWu2WuXzi7WgFTxcUnu8lE6Ho2s3rivJPp6+I4+oTFXF8GuVaqUl8Prw8MSq1yr11z/yUYHPIfIpETCdoWH6dzbXw45tuWC6YVx0e1p4sK8oIcAzVXhyeOvm7Ws3b/H4n5ycsd2xh5Gzye6c0NwWprUX8ZuzfLUs8enGtYMX796RFpHXiOn4pKMgb4O9O583XJxBARDQAD6C8qYtoDW50yKQIcKL4rgF8GGL3QCI1UGLF1ARR4OpQAEgFj+x9jnkXyckPuD2cbsV0MTJitpwrAb3TTJq+i0gIOUfo5/GFGfz6Ud+vhAf8BN5A4Ftq5MD4wKeQHI6AJLjkqfb1WlPv/63+jdAM3ieGwVgx8t5UymDFEv1iJbCZoYgLgL2/E6ojkHGOTE/obmuNuD+dJbArqRLRGI5zuYrAJ3VzElIVKR0hwYnBAl3d0FY7BKGRAV+8E2xcH54+yBo1APRoKmUjWgb3eBldVscVVPkilRMFxalc1Mwz7RAEjyMwKqixEiG26Nj1sAWjwqUDMk1RAPEKNY2/gkFEtONDjBxISoRFkQki35tVVE16W4oqDIjsqVF/Cwm3clIBalivUZbnWKceZmpayOiogSjy/x0sBh2RnkNaAvuOvBw7zAbzcg1k/6YsqjIZ21jXd6rUVGLh4O+vJfeZPDk7ITNVDoT/04k109HoiH0xH3nnffefOu9Xl8Y1pQ4C4y9QyyI8aoeLBRS6FbYYDN660nu1+NXkKRARsXv1Opq1LSEqIlppJO98NwdnAmrlp4rMDOVd553LiK6qlYubG2uMyJBLVL2wfPPbZ2fgFSxzRcnx53eQNUa3ZwI+ZnBWKlN3cMPnxxdf/lua3+DrwkVufb83be/8tZ6dX3Updbvdo/HtcL+jcZrlebg05/47M1rd6XZrFG+oEsIQEFSJZBZHeAP1jBRpATpYwcwhjVuZkbS7jijC+LZUad7Mpv34W5Uks7Mm8XwMUUqDd9/oWTtVNFirZUTg+yKNgCbfgMAltsXLZhJFXRMc2deJM6YxXxz+d7l/bPlybTYb94sPf/K/p3X9tfuVDMby8zF5zP5Maeptk9SNTigqeXFtRppcRHN+oKDkASD+SV2DtTEfZoZLYTnoyDXrDxBSFJxhWCYMSyGYLqSUOtkdp7lqDuU4DU1Y1yO5CepKqET0AjuCkADIb1LGGFCRAy49Rk7gYUIDIwE/4G4HhfjCdQMQ3QYjRK+JgyIDyjF2R3XipEO8NG7SDA2WI8Y7MEoX88WqhHkpl0hIZzy5i7sO4G6M9JlkZc7iTlU/+n89Nfy9ec/8i+9vn1n/yf+0ufuPXjwysFrmfMCO+wiO8bydThVv2Y6KMx7ma3SOsLBTMFNgqXHH5OQ9aciQvjgiIYem5ewQfaYB68fEjraHIIEqDEdcaqXDXpj87kieulwCJtxyJ+3DzFF1ztfAAEDgVlbShePe8BsDzRENCfukmR0Q9aYHZ6yUOiGFjKg5Y5bLUdT7nboRmgGRjKwhmLzySnD2ZocWHVhKLu9xaRPGqRT1EvCkVIbFoGWopRru9vbGk5mRWgBpMvRTmlWnBz1Ho3q+5uF7VznyUSkhqyBx48P2csqxRKsF4CJxe1dv1mpr8sCumj3Nzf2dZfstTsvvvCcjDBnZg+usUMjAHRrycEHN3Z2N6/LUdrd3/nyl7/McH120Wk2mbVGD957lC8cDjt9Outk0K8rBDSfhaG4Hy1LWJ3LnNAhV19i23dbN07OLm5c3zk5bSPL53pZKigznZ4eHo0HIe1WGw0aOTU6Ai1Ivc0myL18/JhabDIZrm7fvHnt2sdFjeTPzzpTAQJ1mBIyAhJswWLarciVqmPhI0kjLSTgXf3FmgWltjQAdiVwWnOAslKdgMYVwAR6ONsWAPNNgEgHHYilBRw+05GgCXbSAOKadKUzoucCwSsQjCAfSGbZQf2VKdXlTl2B2tUlv9s/q2f9br/8fY49ff77fzY8E6QKIpJAtK9EIZ9Ib4QsZPgYEFQOQpC0CpNqAhxJg19NwupuBkM8QTnEkXi7IFgB/LQ0u2G8MbNxfngvY2msYfSqpgitZZWqQCqtmkugfHEyXVZ0ZonWVe6wJOpQv7QVQOliUgI3I8Yh3Q3aRDxOHI7LTXd6y6RYBxF6urmzd1i9i1Dj1U4Mg34dyjdJYU0Za0pKGrbbxJLY3IC0WAl6iAWk8quSzSmsitSEGhfx2LwzOR0CmV20T2IHXsp7zSlySOhTBlbrqXDhzCZnRydVvZGK5Uf37wlERBQx1FypvLG1I22BC/bJ8dHDR4fnHaKwMYHfkUcLtcCGlSQKbSw6nqTZFwtiqkPIi9lIVjKGzhBxzDRyyr2o7R5TJ5stJCffibRUWOPi7Fy28WJxfXtj0wy/8ZUvNzeaBy++QDkmzPY6F08eP2q1NrQuMANSHsyrUh6EqVu37qzJfK41TjOnw+7Is5QplHiRu1TAs7xYG+827jReu9nbm+xv3MwPK4Pu2LjNnvUIaFcnBORovpPNXVycVyvCUioRAshoTLY67yNxg5NT3m96AU+6wpNM5wQyy6GeHugKaEJAY+GtTpBoK5fi88Pf5k7+CfOJE1JAukRrWprqE5NZtz3tcnA/Lj+8++k7H/3u78nc3cpkLzL990azdxfn/dqmu6kzGozWOq/oc0S3h/M2ApSRD38RMMVQi+4o9afG4HiyGM/YmZ3FFhLEmyroBFQn/sLtj7obC6YbFYwowSwrFgingxZeiAkjQStINlEOA9E0YT6xFF9iRDhpfLXsoN7P6ZLVxAYp83MSgg08LgkRJwRVw5b7FVTNn8pMkuzi0bETdqaI5SVqLgsV8VR6ceYzFQl45jkQzl1DVHULg1ob5wsE4cHl+K1CcX79u2792OYP/fzf/Oobf/c3P9z4UOEyP5pyZy62t1r983FuVtjaOBCURk52q9iCAAYar/TSdOgf9BHwaypckHbiBWMqVreJfRuM87maNAQpUWIvn2SHdEJMCubvFoYR7beDLqyoanDfBEEIEPgxzhQJHwxYBSu/MTtL9JGgpW4zt8eQ4wIPFtpBM76s9sXBqzCh2oYcJJwEJankWxvVqoSiiFYsbbXqe1ub8houp30hI1Llpr2TSTtOr+rNvbteKmwIihT8OBzN6g3BVbVscTThQ+0Ob1+rKRWpLgbCIiIyCmSct9/JvkeaFLVJpG80qrvrG0hmFPuWQ1RiZN74ju/+rudeelEghgCOB/ce/vYXvri3swUFeLI2ms2plCEINh0fn52ud7e2d3ez2R1EimA4VAZoPEYE7tx9Uf4Syb7bG21I223mcAJOX/Y5VrBjFTyKRSz5rN0eDbrON9F19ufmAbWYmi7DgRxSFcznEjKymj7Ra4/8h6EDoGebZYxFtRBXa5tAOcA8rWVA/2pRA2BicwR1S7vpkkCPWD+wFXdAVlYnrgBidd77Pj0OalLUDDgYjIvc0JAoAIG1ge2OROHeGCgxFPNLyJYGkAb5vtslYPq2g+8/4R99P8iNCYk3ooEXL+VaLsvYjGr4dBRENjEqrCt4sFHhaKYodiFFbFdf8QAjh9YRtRz0IH7xgkiBPawkEMkbBmt+ukXZ3Knaj2HSC04JXyJjVbgfZck9sGkPIN3HzUIA9fQgx6s7X62aJ8WFIUGnVTCyFTUKxpluuxqh41fc14uk8cdkxQukTztJoY6CCvx9ed8kFo2mucm8PJkXMY8wQmCArHIzKKmvrJAD9Y6lxW7sbJfWG5kadTUnGxB26clgUr0nzkJsZL0Mhxuad7kGqYbDIwqEHIO6kpCKow5H4hvRfsaHd+699/Y795mOXWvw0gZoolq/E5kRROukd4DxKjOJFIeHW2BUvOjTLWr3+p1loYgm9nuqVMiKya9zDql9wXKeyfRKI2GJg6797MHB9c99/jcuOu0M92qhcPv2LWVG3n3vHYHTPKxUgBJci5JHnKcj5ZbGJ53URHmtfdhvVRuleX2NBrssNlSSOj3Ru2+rvt2I4s5FipEYOohvQW0m33jJLctRVOlTbKlSKejyyEoi/RmlEFCtikV+0WWjj1Z36FqOfmLSEtdZVKw7CMEPEu4o9ILRie+tRmy2kxZL2gpIEZolg3g0GeREs1WX/WX7bPx4XOg1D4rNa+VP//AfzrTmGc1Hhm8uFxeKT1d2apnWRqavoxEzDJocAw0YSoiNlQXzC1EMrkfd0EXU0tfXUPYsO3GKrjL/sdYxOEhA5geqDL++rsSA6Bc9DTWQYzjszzgabHBCcIBYvASDVyvoNQFn+gv8WkHqasdpYRJN5yfgX50WF8YJCU0SOvrmiJfwQQ809+KzQrZIp7JEW9p4J4ZK6n1+nOECrOr8lQYTZ6UwsEB0fJA2HsxwppK+FJvM/DTTvLbxqed+b6mxv3HwhZ96u5pZP9i71j8bjjoaxGe31mvD86NasWE+dXxljAj71zJvyk2TNIYYw7dtRpy2+MGOT3MQR7zEt27pHULScDgkMVvi2M40V2E0Dzhxl5Bz4ud40XiD1YWhAcfkgJ3g6GN2ozjNyRHaHCvCWWYZo5dCtBREBYjcl1I0F6It5/3FrO9HAjvFuagy+hpzU7UunVdEh0RDpW+yzYr+NtPFaDAd9hgFRXv0hoq9dQu5BYOTN+OIna2NRD8Vq7Mig3WtURpNlFgxWjGSo25XSQB/p6fHYpuZnE9OZkdPHtZq5aYgfMFXAkZmsL7Q2Nlod/pylVqN9Z1rERyH2mDZSlZd9Iib3ScnTyLOJFrxEn2X0YZzPCaZRQj3Ynpw/VqtVh8nz5fZoNc+eXI8HPTqtfVqvc5dyBeml0m303n7LVYOEQ2Xh3TcfFbTUrSC16eGVbMLRtWcsJxyEomCnrOzyT6iNyDH0aMiQdX7FCEzkNjw+9Y2lim2q5X75j74C/fn6nhI3AEYrv4mVw43ZNDzZ4ASAleAcLqJZQ64c1H6sNB2ZWTH8YQPiKRgDLxGzIngED5O9396bewAl9UN4w7//W5Yi/jIokpM5dmsrH4EAZkMExzaxpAVYcxYABAPS2AIHwHuCE98hhxK5gfq0f/Di9Ang/sC65A3w0wIuH2FAvFpzuMHtiZeJEdCIwgDgNB8bXT5HHnqUk1jAbrC22nkCzxAvYu4c5jM0SZGtSAwRhe3TQiJaZG9A8WMJy3rs89Es2Ip49Ghncfmq0uSqh1vFo1KXAWPKaaY7HCcE9An6KLbM7LIWvY6Hp5luimu76yXSjs7B3vFrSaqT0aeKXTOZnR63iOaMhfGbSKViD3TNQbLjtSbzguN4ovPvRCQRc49v4iu8ctLebfn7S4NWPYub7CoKE2F5SxQIjnETR77t3Jxkm1AjnmGbAwL7PKh77h5mNOZMVB/hMS90b7sdDSR0tC5iJ6+1VK12257U/NOMD09PnPSjdu37j2+L4ry5PgJJ7c+SG57+PiwVW8w44pilFDP+9usVoKOnF6EzNto1ZatZRcprVT4rSYggI5eLSyq484smxcBzPg0LmqHqOFsEPpwPgEGhg5Jj7K6UAAB7hG4dK4sPO9Th6PXG+VpWuXo8RuKWgInhXaiXNRsQYO3LrFcqAc9JJooJ4rO1gBHMT0/A0rYY4vYj+EwO+rPuqeLJ+Paxe7zrVe+82bzI7cyy3Zm0WM+zuiOEe7dwmza7h+e18NBwFnqqYA5GFAiBslwxbZAcpCgof41n5wlINYHeiRITLQkwCiAIrRfYwnWm1A/9uOrO5DFUtp2SN0oihGmoSa2cEV+vFzc6NkWIOptruhS+tU3/CSAOLHHODVgPV2SqM7q13RXPveItrhCUDOj30QAP2gTthCw40IcXcYW96a3U9qrYtiyERmeuOgDsRKLCyro9aZ5pYoZXztvrS37tZfufMcH/uBs8rd/42e/UbncKFYb45NZeZnbaNWPH74nRVzKwzxDCBJEx8+ULFSGavje8WrEadjpbVZ78TZpS19Xb/Xsl9hJ18aOkcU0xLvbEk2ybEQzxNIKWsi4baJJSFCIIXE3B69Yb2LHJkDjaJdEwmSQIvKR9s9BrCTim0O0b6pV33Ktf3nZmWc70h0ITkUlIdWDiYY+ygBsba9vbKpNwlYwkVG0lJTE0jXrKkSB32FA00l3rkabOK1i9P0EC+VafbvabG2dtaMkWmYgUf4y39rcUdKVUwaWQ1L8gIrMLo30EdyfHD7GgM/bpW7vYnt3o1ji8qoErc6WnpyecDy4pNaMxt17N669+IFXIqt4LdM+Pz1rX6DryTGsGJ0SM9nxZMZUbefWnbvb25tsbwhgvliSoRcFfDJLNjw2M4a8lJHB9FigW5+fPMGhw03pbqUS3jzodBYqop2deTSey+xDhqEGqUy9Xq833TMseEn4WdknY91i3i1P+guylWD2ilmmnxMdjz2AnJbZXjp/tdqUkvgxdNl0IFEEq3p1dPXb+1hmgEIcDJiwBfwEA4fqUQfWheLWue2WxQLPuShTWTrOX13itBWSJVrwLY+IsaXtW5/79Og/8F+XPLv8myfGvMaEQJZiYVGR/M2xuGqi5XxKR8RReGnXOs3kBPzjfWmE8eFIkljJE86yjiA+xHUABs7Bf5BJ+0GNAqmDASN04t85n6x6BPbTsFgDCspGaVp1KXo9TGjIH5YiYy0vVJCkOAhzQigmySNKdHG10UCfuONq6uITWsZiBt7FL/6xOXm1PT0Q3NetHfSr5VHyOKjoRLws5OMLnGlKqCS9Ju4RSBqdbWRTClbLN7cb1VYpv7OT4bxcTDpKvvXbXljpCS7Dw9PDeqlSj7bsLOdljtzgGGsKSpwDhmsH1wQZwg3WZgCg/AzifK5XQ7+/ikbWiLBSrm0pIFfUXBlByFCjczTvkOmD8JtgEEHCMb9BMNLmziX5TpyOWtELrdQUZRzZxpw3eioIC6EWmzslano6s5cvys369eu7t+7cZOxS/NkluK9KI4KfKXuijjeaO2J0loMJcyt+rkdwvbqXKW7tFMedi37ETI7rs95M3av2eFxb22SN12DZuiYAWUwUGCEgg3Dxa0Qr5AoFV4RJMtbjhyPW8H6Hkc+c8Q9GmbzIdOqrUCxYLYlY4IgJGidIwRaBRVYUpFmreIjPydQwqA3sE8wj1g2saDw0mpQnx6OHw1Lvxse3P/x9H85+YF1f3v7Z5+q6FUWGMR2V6JuyeEsqjdZUQDGlSKtPUBLMPaw1ipKKbRZgNYzIL2ILkGAn0dMeiGG9iftaiQTPAW/02nRtnIP1skYEM9YTKgKv4ojjFjQkjFA4n+plAZtBGp5tT+FzBbdX0AuQDS39udwMxH6C8NXB990hCaEB1U9B34q4i8FAOjyGJy+ZVOLOiUgKPomcVt9ztQyDy5qEpegtBcbYwgL1J9E/ql8UZE80HY6VScxUip/9N37kdDy/97mj1nRzr7G11pv0B+31reqiMPR+RHnuKOov302RsVcIcdCQ32VbDdNbPx3v0500J0E337fF2vtLGO8jzUL8HFwUBUgnxwQ+vRvRfnW1mXfUFgQn/lHTnXRnGZl7JjpoRW5RkqGoDsSx0eUas8zZZNm7XA4Um2RWbWlBomw8SzNQWYiL316vbW0o4awVt7CsPp1BdENE40772lm3O+2A5Eg3I/es0SjJ0IgXfieGWDBX76zNVAwUr+3urG/vPHjyMEakydeYvh2FW9keT0/OorjAZJi5UDPyfDDq7+3tsJl1B8PGZh30w3R5QWrt9XU+WizWt7dkNK7vbBw9rp2eHc8mEwi0Ji+oVsMsRVk3VNJqtMKcly9u1loYLcSPxKeNDTBJQUdVWNPr6+uirvavHbz3zrtEh3q1Ui2XvEi1XGTnI1afKwE/GO7s7G2sK0CtE82QYF2u11r42XSSVJm0VhD7fcv3bPdqVXy3FuloLMrTn9NOsD2swBJbVYTPusXvgUKUNcgQv/jpWwHEGYk6Jg7qJzeJAViIFfmIs+0DbZFv1CwZ3YH4jM/A89lQVzvv/3TZ79i8wrPzf8dP/+Cvrnrf67uJwYXAbWhRFStfKOc0FliUGUUDa8NaHj71OAPxCI0zafZpTmJWYkOwhEFGFCUlxUFAR/lBciJCK/FdmO+EODMeFqxcXmkYqdIckpPk+LBb0mzZvSIrNl8AcDLbRMtSBwGKCsWh/4q+FUsQqT3annuYZyKCgXExWG+TdOJAtNXAni5xDD+ehY2GRmXn6oiDkAb+hsXMEzD25CTGdEOzFMU6G2RGMkLV/o8jHspOXa6HkDGLmkaUWel8SjFdFhUuztVzZfqw7N72WRvhIXsWhPDmSztb+/KLxHWIpBDfZDrINd4nsC7lDjUbEnBZA81MCZ50T86kHYftOhRoz6RQGCvrbIERjMip+A45lJ7NJ83TFW3u08t6r0j7SjXZT54chRwgI16ScSYrLhoeNpSx1iWxXJZ3L3qC3CNp/vDxcau+9cFXPvzWN97pdHpbzRuV6tblcH788AG7+u0bO5lJMTOYLXq5RSfHDbo2Lxfm1VK2MVA4urkpMyMpi2HkSEHsse5WUb+KTNlK5TKTwfT0hGbQaZ+FpDLTfZDVjgOC4CKcqRfFm6xPgIjWSHgqaQtAirUSfOoHkJPgNqIaLR+5jQuDiU3TXRxyMJj3J2uDaZbcNH7lB164+dkXMzoqLd7ttT8vtrmyp7LzE3W1iESYS8iDK6ujaQs3asywf1Y8M+yzWlz1O3OkRFg47Sg9mW4F2gKXsbQg+fCY9PhUjwzOzZQT/oxgt7An9MuIFwx+HLJTpA8F7w6R0Ztarqst4DL+v9oSeNqPMxJ4Bj+JOUib5Q0AdgSoBhQ7anz+cb4BxlVX4JzYjh/jFFJy/LAmKhukKI8k9EKspW3l/p62tZaIPyaXtdJSVoSaX2aQWJ2qV5qxnnbTFU6i+dm09+VSq/ZH/rXf+/mDN9/6+fvTs3k1UzzpnmsjwsUc8xLkMqQmVqmoVG2UBhOji1E83WLfTzGIeKGnP/nXk71VfFxt6dTYJ3AH0bm6cHWO7yTSIAfp/naCqF1dGcdWZMppSLprUanoj4ZBpmzJWCVWYCDhh8jxXVv25pdtf9LHaEf0v3pxfSNsMWKaqmw5YtwXE2GRa5N+rVwfEFkVDOi1p4A8wo25mHTC5GHlcmfXQaxQ+Bg5p5sWXWafhi2vSKCGUjykVDhfwZe52SdsSVOhlPq7YBD8xSrqKKyhAi42985b7zJZUTIbzawi7RRTY24rBjvkPL6ApwzI+7t719auqy97dnYyGfHzSgW65K8l4kv53b92Y//6HhrIoEPvFZIi3nMgqjSXI3+jCTy5KDxpgTK0ld/yFAFeaC5Wz6PWG4yQskS42ESlPhbYhJTk8L55wSPJ/lxRysvbWtfYAiLfv60g1ZFwUKZ1juUPwIx1CkZ7taW1jxs8BQtUEjl1OEj0U8iIMIeA/m/bAtYQxJCB0uYcR9InwBeXNuFpQFUr1WIrmy0H0H5zpKu9Z3deff3dnvJtj/0HHFgN+Wo47z8vJAazkT5Dj4qyE5H5LdAP5CZo98q2lTkhTdcV3n/LbRi2ML9AJFPFvBWhJl4ZuIPzwMigWCs6YieEceTITLotQLB5hItppUDVwgM15iB8cJIP/RijCgLoivD6mtvVlMYQHPFe8UMsjA905WopV6N1zrMdP6/24zNNSuwIaAqHM0smkwxVCX+dC4oOp28xr97wdNAvjpDpWJTwcVDvGKWnESGlL8Nlfq7U8Ey28Gwqkrs96D1Rtu3kTO8Syp+eQqTOD7766kjBnHALSxcsHexfMwmMmipQIZeEUOk+UnXlMBCYlbnwwYlsJg1Iu2ATGVMQwiDzaeQNc+iFHSvgM1YHAWG8imlJZtwUaHup0k6/e27+hWuleTGDa712T2Ts7VfvquYMMwf9UfFWc3tzv1ys3PrASw/fPjvu9sedtUyXNzefGVWnncmwkhkte7NBCBBrk/IsnD9KMnKdipWSzVvtD87C2e1JBqyqJE8vTwWt2Ikko05HePPF6cmg164hYKBNd3ksFgxIkhA8KQky38DoktxktJGsu8wMjTmk3cAmS48DkgXFfChdPs3nDCUShHvL9iDbvdxa1vYq9Z3ah3/khzL5zmz5oNc+Wqv0S41ZsYzcjZu7RWlgvcmp4hgRa1AIRSYGAmBMIa1XKYXguGOx5pCTTyGJFEAnwARs8iXYruL8zXsEMwdoB4rYiXhffDcWScBzlJxc8eCk/kLwEFndARYEjgSL+JYNOMdj4hgojhOeAifsjEW2hViJ4gQpiCPGkCAXJw7kChYX16RPI3KvODMdcRRcxFWhChutM+3E/Ib04QtQgzT636henK/oouNZa7RhS5GKMoVVaA4msI/KpcS0h4e/cOPWj3zin/3YTqv1d/78L69VtgrZcme2kMO2tpQrY0zhh/IEhiv/pJdO77ga09PPdNzLxHwkYI5Rxd4z2vn0zHiR9Isd5yRYSd8J0PFLuMriDeMzrkn0J+60mhT2tJiAmCn/LkqRE3mpfJUUf3Fjw8v8JEvxvexJxROhkl8bVXOS/fPr9fxGNOyuFab55ZipCHmoUQEEUStAOetPBzOsd9S76LXPOr22ZttRjCJcQTnxSWISzN5lXUn56cn52ePDo2Jzq93T23mtsbHdaK0dPT58EjXoLj7wgeeY+qwF+zXHGxW0XqkohsOO9cJzz33i05/gqPq5v/ffKPWq3M2TJ0elQePlV19FPcjQSJAgDyKV8u9ykWr1aqtW39ncMusoADcQl61gaZFdSpiJXBmLnZvMYMFgyM18wuXNHSTamQTAAs6Zi+uX1neiP2u9ziupug+hheYtfANS8Fevb25sbqsOtw5LDp+cI1wKjmyvtzbpG4zUqxUy1TH7wWAcAI6WZfUZy3MF7KvdWLhYtDjf8ocJBZ1Lh9Jix08J7NNNrsDa66H1q8V+eptY9m/uX+054uZQEP0UmeVlaPHsb3P6pvr9fhRfo9rBM9D5tjt8+4GEqd9++B9y5Js4mU5c3cQ0wRYypwGEty54cG4cpI9QGEz0anLMDGbpSExRmpZnk4bLhqIcxmQnhx9s5elyKTSIZQipk9aQ+CdcC4YqsyQkFMyDEoaTxQA8LlfgGuSrGI2GQnZJgoqmcp9EwVQzFRboUKhC+g8vo5GgJomUGFYMDE6SkFZDi+NXe09nxnMdsa3kidWLhPcYMUbd3VpZDBOD9fLfTIdqHs46mRn/rVRARiuBu0pqUA/uHPDC6hPMFHMBg05PpIQGe+t2iQ6K2sh5pZcy2LBvyZ995813t/d2tQ8yJqZqmqs0u8fHR95A+Q7eVlTBkIBy4Gq3BxmiXLFuSqKhKII0q8gxDYEmlOFCVpKWSfOTz4hyHTM3hYHe/JtxYGaGYt+jpMSPJ5CwyYtbb3Uu2scXJx/65Idr8g3rm4ePjm9fm2ytH6gskG9c/9RHf09u+sXctHX4br9SqO+vv1zN9srZxuj8ctifRLxnpSqwHb9R1LA3E+1cmoVXd6k3i0K1GH5anazkiYxKQecQ/Kh9dsoWXaVqb7amihYkY65phKEAgKJcq66LN/Ve4qp5AyLYStZlrI3lSrKU6ldUKe+lLV/wMtx+OFm7EGnVX7tYNqe7r2y9/OmblVf3J703C43LfGVeS5xKqUXJRaBRE2RxAySDQHy8xohTbF8goD1UZzScCZ9R+H5MQQ+TFEK9Yrohd5tRvEJSOquqX5IwGTQhEAfMGSgRKDRgiB4WFaLRygYE3LxBuChCGg0ITYzwiri4Nt7z6RaiqVWLg6vPpz/E1wS2/gkCFcfjewLvtIMB2dwvYWjspK/pRF84klwVAdKmXNOmiEgxdnjkjbxFqhYTYnMMHpWC61gH+pQfT9X2kixRkSyxHE7Ol5NBNj+6cfOlw8c/dVD72O0/+OLgr/61eX96Z+f50fFQ2WNRuh4RDNgTzXT0ObWuZiEN+9krvW8nhh6Djy32YlK/ZUtcNI4EOQmN2gzFxKe1MK/xeyxQnBF3skj27Tl/JdaROqxaeBfiJyZniUVM72u9TKabWetc5voCwfUWrFSm4tapA5qCtmr5ZnlZIvnJ/5F71hYokivV1ivrFWnP46w6LZN+e6LQDKYbAVSSEDUAjjdRM3U5U15jwCBUr6qvfrb4+puaHGzfuHv/waP+YHR75/rm5hZ/gFpyh0eH4iZu3z6AykhKL9vjhGqtb2g2eP1gT12OD33oQzRRqBQGwHzxa1/9GtelmrJnR8fvfuMdoZqIRpTVyuVa1aaZVvRAHrC0qFDsZ/PGeuPeg0df/vrXnhwet/sD8SiMXoKwKM1sV1zOFB4iKIXYrWRm9Q5PCuv7yoOUGhuNbHHQHZhJ+u3u1vZ7772nvGZza0+mIrqty0t/NN85aLJgFhijkWfvgGczgxqNg7FowSljxdIW1NkhPCJ+SEzaoq1+9hmsIsxTIRTHRXGZL5A4uNdKpYbJbh43Cck2wMFzY9ljA3bxiGjn4qx4aNwrsrqXJIPpZIxA96Tvq3+QyVYv1bjLhSc/wRL0ezrMAK8gAulyn6vRr1hmOvzf/WP1IpAj7gnh4ZkRkgjXLplX2J4qGgdgCQFSYgdC9Y/3NbGr/ZiUtLncW6dNCiyVDK0JqmoSPcMdbd6czi/Kwd3Mkp73zqf5CU0plWssOqVyXcKJg06QyUkIY1jljKSDr9qkh5V1OuWT0IuDucaauo+RxA3XIqMNi2EO0R3WcQcDQ60B63Gqi5SGfSnO3mDsr4bhTJuHrl7AgPNTtCf4Qcw6klTJzyuKxVTGwyrXC199I4TDxayreowU1WlermmQ8rWN3evVbOmdN75+Y+8AN1bnkM2NTEo3Jgir0tY+78xOzlQxnD05VhVjY1t7zvrJ6emT49PoIMS9PRi3iuWdne1ytQQ+WMJ2tzf5fmejmhPQQ+PUJZ347TcQ5Q9P9RczGdtks9FcjGUx6McsgUc8hOi+YBEinhfzDa3PGvU6PG/WW/3uIHKkF5e/+N/80t6tGzkBy8uyHBLGGHUzMuNa75SAoFLD3kbx9sUp6pTfWb+mhJ35bVQYBtaiVoeA9wyb05bl65y1Ya90XvU3Bb3W9jbDeDCbnn7ty7IgUBMUiWhWjtA9RTW6kqLMWuhGCboDBJH8qaZC2sJQ9FHFtejhYGUCbfUn7gsBVX9AXuJ4PtSWmFt9nOmdzR4Ps8elzYyM3hc0TnihkSm0F5M3LlvzqcYiSgBIZ7+ch1cj6lcymZiW2XIckxlBa0okM606T3tUiq+wrdEgagvSC1OGKCKNCwSZD4vKCtATYiduSv9yJv9DBJVyIDIiat4nZE8gWvpqJ+wVQuWkqScC4DMYdizLFeqGKOn/xE297+pZXtkTPS99jac/3Q+GGtwkqI0PI3MAqCMdiVLFKGNbXWJ4sZc2UB1cO1gvl7nzsHf/hwZvCIAlmNWyEsSF/MDvAehUYCCOGvBGrZhDnTknM3OqbIRhkCy6w8Vbu9d2zw9/eXM782/+2X/zb/xffvoLP//mK9sfHhxPNgvNiDxaTsr1Olig70kkhFbGgloacxqmZ8UD4WN8Xb3t1Q8BFN40FiC9hFkiafowKSSlFYGOl7cK3svqpjsHtYmyGTGB/o8TQvgJ/DH+EIrSUngnGxMkU3Pncu18kTm9zHVyJVZX7Tkvi/kyQG9yV+BIolPccZxb9MV61C8nvKZbZaEfA4XuZNqYoL1bB/fuj/P5zcp6bfpwLVuti6XhPDNF42EPPWQlI5ETGrCnw8Mn90/k96Cq+QePHxfLOg9WT8/bh4+fjPpng855VOGoKqFVgNWap8iCeOGFF65fPzh8+EiduM9+9nu//JWvPLj/6OTJiX6CP/tTP33/vQf8Vru7eyrIy0raaG2JtjjY2VfwjsmaZeDs9IILSvb/a699+Fd+/XP05y99+aumkvn61VdfNTBOaFQRJ377vXsf+8hHHYz2LRl9sC/3D/YjGKVW+9wvf+7o6Kg0WX7w5t3JZV496e5o1tosH9y6Xa61vvz199qDiVq8ZeuELifua5IDcENQBe9WPJY+gNGirX5Kn2kpnJe2+BXXDdk14DH9trrYjRBzKho+5IZMNrGtAMlDUXsnuzZxhSCL9mkjSbJ0IkxVvGwwn3W0kWi3jwGhQDOlhkslldw7BHD+w5yq/hHTsQLTq4H9Y/8nDDVXrxZjji3RuSBPUbUdvRLBigeLmsNRTRkyCZXj5BBKQlLx+hwuKJpZMmBH/ZpeOkhCnJQm0L+xA8FirpAasYPhe/MMVks+BSZH7HAp9BIUeu5oMjnvDM6++tYw2luuEQ/3FHDTkZcvWd2loChXN3TbKKXk5uKBSUEx33DMkAzSYeseHtNYmcDMb27xW9oc8q8PO/A2HEJhUg7aKaJHO1pR4fntjbxCT61Gab01fPJken4xGvZIIzU3VoVjEsXnZOt4JVGnJIyLk3PPjEDCojymNax0hGMMyRoWtjAcM6qO2/3R0Vlb17FwAOlIqFzSbLa1s3375i0VahTEgXhOBzy5SvS/hbQm0DmmcK7W0lzYV7iqvC0sCthKW0B+UCu0xm8hLZmhYLOLJQ0Ugpk81cRmYyWbJrVybX1zT1zRyYOzXLb48gt3GqWdw4dnw85wvba2U7sz3q60yjuXk4amaJFEN6prWKrQRNIfAuwF14RoSQbIjFu7mzGNllZwsAd0z875etvnOY9kRcCrxdEh8/AtPo0KiwUktJakmRBYYy3Yb2VZOYIBEtTU9KMPK5C43G/VHh49HMxV4Nqsb5c7vdOzroqX3cz+8s5rt1/9xJ3MnUqmeJ6ZfX08PV2SUoo8xyY8MBVsS9hGhvjQIzMUGQ//p44NCDCuP2TDEJYWFZzn0romfLRU3gAjAgLKHUOOl4uZRUEDWAgPfsK0fMZg7cML7Aq7jaQjPCp03yD8VOEVyQ8O69p4aXf4ti3xoRUbfv9vCXTjkU9BOFhyfDXVMSL3inFdXRLPcMWz41eHE5DbD/4bIs3qDmFIiNsGSw6c9Vr22DFCHGAMS/W6vcZUJntVRWkNZBXKo4NYUrMRbHiSnZ8Ox53Ng1fap79Uz/T+0P/0DxaKv/aFv/X2d9/9zPDBeTWf09xnpI7FvLhWbQ3ns0hiSBj3dEjx9Zv7V+N9Ouw0YifEK6UtGK0Rp0Sa1VUBPel1VtfA4kDkROzTjYP1YtUR/YZG5ISa5QAsrBWvJ64dfnQX0/P5WjtbHFari/pGttYsFEscRjKvLpd9dV2VE8kJ6Bi3l8OTRvWynp3Xs9OSqKwZArXgw2CEwrKjde9sLriyvt7SbZkhq9ZSdWS5FqUml7VKfmOzsbmzvYG5trYm2dKHDm4Nxpdf/vqb3/jGNwbMfZ0OjkspoyVXZSbpedBs4AmUS2AkaIEezEPnNK4wHNo8RGWM4eh8fgYCaTCbWpzWCD2Xko/bFxeoK3WZG7pQCGWv2agfPTkTRvP6Rz+unblkSFOJ4D189AR5EcbsPU5OLxCSL33tG/cfH965fbvS2lHOQCmRUk2oVqvS2ho9Pn77/uPNnQc3rl2r++fRYx6gg5vPre8e3HrueTRKgQ/VlOTp6VyMWqFBsThpJTwuAdyKvYXeE0tmQa1WIE8wlsCMFWciRPkhQSdC7r8ww6qEg7KLH0ylM4IBxy0SyActTJdTydLTw2+H0ZTLCS/daTlhiJiM2sPh6XTekywqnI6Hu1xlqmFYkyaKWzeTPkqfS2M15MCu2BJE+ef9vCTIwj/y5q3TLX0glr/j8mDMUZ5LudhZlN3W7VztKYbOECVBcmi3MZRkHAqylIhTDNIXeLAC+ZA1041Dwg5092cl/IhnhTWCVoJ9MCEXSmaNUEPc0UxkLRwSRE2xRJlbd56DNqUC6bmKP9EpR/wMy0jOiZJhSlSF/fCSXdZtccGYMYPhzov02VDWkxCQyMrVAsXgHF/xYzurLb5KWojVj+GDg/gLYUiwRGRGZ4u1TK2cq5RVg2teu5EBuCftidiodlcTUO+qwEW53sqMJxIBhPo+un9IrsQA2ImFWwAZwTdyGlBnUcMjoVKZS8pilK8K4cu7SB8s3nn+OYHKu7vbiAuh1RGRkMRS8bfBLqpyw4QEKRjJinfJ/SyBXoJviCTJ/gyFREl6C9MLAFPQJXAKEccYgzurEyLAPacfYLSlmgwXSyFgl/Mbz9+6d//h+fnRpg5AN8oimUezweiYo6u5vyEtobYclRQTEl437atTICouO5sPuYvITKQnlEIVWzEDGYqTigWdzlDn2IEG3nC+Lw+yaQTBV0leZtWO1aGrSK4E6THVgW4BLcGqRJ5mc0MiQloUfNCzpgrkjbVbzY6LTWX8lofDt4223Fi79qr47Z39T97N72QzTQz7yXLycJE5K9cvM81yZtgN05EMNyoM/QVAY7nqzQUDkqgb1gNxvTMV5ck6qNSgi3OaW3gRABGeDfCeiDtAWeEJYr8aK+MO8AspPaosprqSYbCNrhBTbYv9cZZEMGzwY+wZDwbuZFzDTEnGV+gRd3PPeObvypafPtgpMUdBDtLpz44bWtwifiBZrNBuxYMDlGNa419A/3Q/vVA8fSUix16gcWBNJBZS5eN7YHR4MkLhwK5SlblpscUnmZJK5Q5DkLgynACNRrY7+EJj+/nx8H4+s/X7/ic/Wsz83Bs/+6W7tRuqdir0hhRns3Vpceyly9EwkYmn6BYjSYOMkfzuW2K66awgfwEusYYh28S7xQuanSA1sXYIVAjfcVdbIDMUZaXyB7aIFriC4tzDnGpWl/ylo9xlJ7emNe6i2spv7Zc2drKVBjoFEpjUBBiwCLVKSvRddgaEtBOFCiPTXDX3y7HR0MY0OMsVq48P7z04vK+HeHNnx0N5qiK9lu2toCKsPl+lza3Gwd6WCpONrS0pRNlq65UPf4IGLNxVSazjx4enx8dFkttUotNio7XZUqg5m+90Lzhi2SJkZjJDb2xuyu4VNbezw35WxV/xY5i4u70njKNeoxTktSzRynQ+PWNRo9qaEthEpFpW146Ojvfv3vzu7/2ee/cenJ2eW0AGRYZGhEXNEaTjyclvCnp9dHSqD9zNu89TiQ5u3M2U6tlitT+ey7kvV1uDQdjf8eSt3Wtf/frbl5kTRaqFhTc3jK2PHJE5vHxoB14OdVvR1kRtg3slIIZcVinWKww3wTBWm/0r+wRlKq2h34J8mQL3wYBVH0zJMgJMQ7kNjpX4bjKhxE1AbDrZWoN7oCHnISIxFvOhpKnx8Hw0PJ1Mu2FgCDdST9N0/II4LpFbtEPk2KzQJdBg5XsCSfhioFHAVEBXQjn//HfbAmKfXRn3fN/GHKfcPYqr3L1Gt9UlW59gCtZDeBhw7fGBBIEVvoY2k6hCIlzpZ2fESavNTnyRrVasyHYL6sOnxCxshYKAMSWxNldE5LutiIBy9lKk0tYuXwK/RRQSMnFSwQmItKGkBytKVYn0zfAvxMLFg9w3/vVWJiqUP5K86TceiBBLnmbMOR5tP+AxPmKLX5+OFjxEFxvGuEiJwT7DYWCYcfuFXLpWprqe2djLXJuV2t3SoyfLk+O5vBruw3oDS8mUa8IYz8670G6qAgS5ZRXuqNmPXLPI7Y+AKap/tLQxgAg7k62OcK69+OKL2GrwUWUmizkh35iCBkD0Z2GXWDVYw6wDnURvMb+raYgfi1bF/wvZ4VDMkG4Ro81KREtEPoXUF1oMyyRJhAK4Vu6rP90jV5XUoFYuQDnI/GUrM6ptVG9kFJPsFSbt3Gb9VnY8mHTW1Lcsl1prC+ry0huaaWEm7iQ8CdhW2AkSDommnctQWozGJw+4vAVZDdTTSF4DooviBNrRwxPTbKUom7FkuBd/RxwIKpwQ0hgDnCRVAQ40nQRGerVGMxwhH1m3hWZuVux1ZoeD8tG1lyuvfuL52vPbGdaTveyElW96ls10K/XoMBjJNP1hcCpqGlYfHmUxu/TtZO3OlvnldFs1WbSH+YxTJJikT9wWBCHx/jU0DCa4psr2dhKeB5UPaSE4CNBdOTeFrhMEmVhJFNKNJJIH0KTqV5gZ9hwR0e6FDZM/DMPtruAtxEEvmeDWTsLEq5+cs0Iwv8ZpYCR9rr7Gy6W/GEscD1UhDTNuuHpAHL560hV2OCce5ef4IU1+YO3VE4k7rLOhYKRbMdbEnQlLwWQZ2DuBA7x58gTzsFSJUAry8rLcKJ3cH2/dBM9nqFuhtl6s7H3//+gH3/3Knz7pHGUyIg7KcwghZE6F79lAMJ5TE7b6jO3pAGIino3mm1+MMEYb1DpmnRTrX811V+N3aDWh6S2cRaCPk21hmLCu0gR0pVouIp5FqPxiGP16dQy8hAl6hvQLa7NqId9cb27ulta3suUK0RsAdx8/rGUXrcJiu5jZKIv2m87H7VG3Lcw5L110OhKEKGyBOBucWiTCWOa/QoKUZzVs18rTAnqPEQm1p7FxNqlsc21/O1xtSi3glJWqkhdqx29tbsDgo8ePIu6PxRTljYxcwW+aGkQcNa0X3UPi3n33HY4YuA+JEAqv2FpvJKIx0wzUEeGqvlpAAaSMyZ1u18EgHWtrjSZHZ+704vR81Lv70gvXr9146eUPkODFLAvmUrODZwr7FFf15GS0t1f66J0X9q7dGcwNs6lMi/JYpyo89waCFsE50zo+a2AGY3UePrrvcavWLOH3Ct9FyGepfw6sWoFuAGQY5Vb/+tVxJ4cIHqB/JZF5q7TcKXjQUSsudGXKtQ6NrDkZIKw0lDCkLQA/gNnlHrc6OYJ05Byg8OGOFiyYEgB9mEpBrX5FiIiWjBIES8leeTSnUAWXqiqoI0LqucIIVCq5hdw/IUw8wdiSaG4/oat//zFtbr16k+RyNq3cwJWFVJ8IteXVJ3p4R0NzImyK021PhxGDCZAPcd+MmlP7/g9co8cEt442B2YzqjGwPTMimFhEb60QzjysiBmAKSe4flG0L5YVVZmtI87VH44JVJVKUf8OqWbRqF2GcnhzmX1YfmMKksEjwrFshpcWOvRastiK6cZJ72O64SNOWm98hmYfW+g1UgtUs7B+IMchgzb/WHCmoHxugSGds0qdgu3dTFHwks4EPQDH/gyO1/g6ZRdFtd019ep0UgjykKcvXA6kA5HR4qBmcVwk4m7ZF6oUZcdMmo4IUMWKAzjt7rjShW4pPELldLzeqHqL8KSKiJ6E0r+oyIUv4MAxnUmqkGLIO75dvZbPldnC6cGeGyldVPTL3FBGxZJHOuo/VgrNfm/a6Wpp321Wbty99ZHXX9zCmNemdeJgfl4jOFCoOVzlNil/XSqoRuld+LHIJyrWRmtTDHMZhjNSuDCTabdzYVBOip5qihIEP2DE1CIwwn6IOcF4AQ+elrawTRlaosTxGYVKAroou9RGVb5s7A98wtnaJa/c2ejRSfdBaWf+we+8deMTNzM7bMQnmeKiM7m/LK0Va8ipwCdVtxXM9xS2ixLmR3bMAGBV+wSHR01irshuRoTqMDperFKQ8Wi+Q6HYgJi9DFAbuxGlocG3ZMNmoXIgMNNbXr1HgE/Yb4BJyGxUxTB0mnSYnD5pxqEcBzyS5ozK1XHbkFkDO3wm2F3NyLd9el5gWNoS1TI2ulkaYczp1Q4ke3aac2P8q3eIL6HD+y/haizA6oT4Jwhj/EvsSS5UTC2Jd1H/PQJjRBqSQU1FZLMtWFujKdQom3rmwdWyREHJs2M5aa2qcmVsFOf16nq28GjU/tVKtvvH/9Tv+4k/90uHb5/fat5FTNamiFiE+WxWN4LiGtk3N/MSmBmjef9m4AZseWLmzWGcFv+nj6DeT0+OH9M5DvAzJLSLs9kmRHPjvkNvVMj3lwslpnoCMphyULRCQQXz7O6WhLzIx6uKf1yMJ22tv0FIbnBcL+c3CtkGIayvqMbZ+OJ40rsQrxLWbMQw6FYYXJmVOJ8Obu7tXNvNSM5hwxkvNSpoK5lx3rk4PkIxxEZEsuxGM4S+0SingzYbsVz87qTTG4m43Nlav36wKwhZnMx13ZKu30IIpDNhcjdv3tzZXm806tgQzDBPvLNDLXxDTl3W6k1wR+XFFPvDiXhpvi+PEwXWPT/v9folFIOwvjYvNWoXvX7/pJ+rlG7evOW2h49P8sXydmNdtNbG5p4S9C+/+pF8qbp/7drrn/6OqgaIw/njR0fV9W3VzkVdhfVoeSm+8rd+4/PP3b0D6xUesbb333v73ffe3mi2pDCF+hv0kHk/bU8XLtYqATGXWSg9wS3izyJeUQRnpgWNJU6UwfXopCRmTknmijkyF8nr0aOWRkiDwOwBiOti8yODc6QfRIgqrctTIpxcugrAh4Q6Q8n0LeVqhQpVUgZF9NIpCLwqrxeKm9n8uvJ8kQgaIjLJ2WjdEJa4DySJA8mS518EQA5GiBWO2okfv21b/fpth//hB5Jd2p3hd6j4UZxyGTmTiv4GDsB7b+3tsKUgIQlt0pEVpsOFmI6rQdlxq0AVokeIDmtCFuKU4LU5zhj3zKYYnagmXMnVof2Tk5PJ9EgbS8yprHaEoCrMf65kjZKtMZE+HDEy7BnfdQuTTlgMHA6pCPUm+YQZP4J4nla8SmOKOpfP5i0tMI5rPDGTNmYPUejpW1rJFHdWZCYu5qOW06UWKKi8YJ6CTNLMRhFrK/ZLy36P2bXHBHx2/uT8XPOS6DBQKI0XUzXh7DPYcvQqxupRpIdAYO3/ZFWFk1kRrrJSX4xCN29ev3awJ0HVi7BhTQeepgyec1CSAAUzoGWh/bLJvKTFMTFpThz2djAJWHvdwaQ1zyMt4k+jVDFaRFRkrRHHKgRJWavMcHA5G1mL8s7Wrnahrzz/4VZrs1ZtmVFpM0pXYjANfYPRJgSJE5tylxsD73xxWXbj6kasOXP3UDrxWa97Lp4ZoeNQTZpuLEAQ1zAaUQwjCszkxgyHA47xO6Y6nfIM7wIL6YeovR/GU+lmmaRGZNCx4bTTU0W33+6tnd760Parn3mu9XIzUz0dzY+XuUhG1B4xUG4+YEjDe4nGQjjVd4nef4LHmXOgi9tPBnNt3S5n3aPHWCUej1CwjqHM4aQKaPbwQLS041gAtlFzHyTADvQLyEYenBQ8iqLkEfHHzhz0JvYZHnJh9IziwumIawQ3YdKB0zYXx/1WQJiO+B5onqTt+DX2g8fERMVwwrZsPy4JjEs82A9pPz5XJzpuJzFbR4Il41BxTvp5ddWKxyU7WlwV13qR1XNCS45ZiOocV3GTsWhR7ctNMaESlS664TpLB8KIrFCQRixeZ1m6sXPx4KS5vSbR/eT0SxtNIQxiCl7/o//WH/m7/9mvf/HvvPnyxovNYk2M72aryVAUlC29aHrB+DCvtvT1fT/Ey6TRPz0v4WqcYCLibZ9uwXHT5gRxmCY7TBLhgo81GUZGbyYiBgjBstb4aCKdqE4NjfTPnWuzPPlhfN7vTPrns/7p2rBdmnbX1xa1uV59axIH+lSAUU/JFzlrpiIsS8FiPDIU7tD2ZuPCvKgBUEUB2lx+O1uaLzY7nUFnq/e2vg79QXAH88siE45laC1HeJILSbaoGShWq5vQh177wLtvv0PehYx0sr7sYDnEpZxYZWxVZSqFp3BWxAFxY28OMwDLVrGcL487vT5jtEyJSKNEVy+Xza0NiVX+rBc5eXRyqhSBQHZ6K3wx9Kjm1WRByu3sXpvM1oQ9n170P/u93/+pz3zGaPUDpmdsbK4fXQzWtw8kQdEQDvZ3KxuFl55/4fzsVCn7EQpwcZLPbEwHncmgl2vUc3L3g0GEB48dmBEY+Fqn5HcMvS3MUMFCLF5a8oA5R2AV61AI6YEHaVnX1J7DdBFNC8BCbsXD1He1pSpkEdQbzCDxF4JwvBUAiK9uGM+B+rgvWljhGgxKy1Ofl/9QX1sb4bIcdBos5gobmfxWZm09s2ikMai+gqQGO09IZ+Ui0AmL95le5ync/eP4N8SNtMU8xWOuviUglzNgBsMFSdoI1TyiIRMPNmtwMDAcqXINAR/wh+xAlA1bVhK6EzkwHalhQK4kPdw5hLIwO/Nl6vlHTkGxA/OdQ0wJRZaZYDy+VCIKfAgU8kRARfdFoaS1m4jhYIRSmEyAEjZbkQ/hcYmawyJn01ug+yFTW0MxPKtJs0DebwUSAX/R0yA4d6xaIkJ+4v4em/yYaS9k/VWpy0qedzKmYBgqWVhUsIIb06cy9RojK/fwfK0Die8/OTm6uDBC6Ueo+4jrNxtNb41QSIIZxsl1ZQg/KH0zCp+FCKJldeTLnxx7KAMVb0roneMhtgn3nAAmjdMbER6FDdBhvaOX4uA5v1C7KubEbLCw+Gu3+62KMgkVVgDaaoBM1IGVlVnFcXRQGI+W+3utl1784Guvfnjz9kuZ9kTLBIV8SSbEiSx5MfAn+M90LJ9q2WiAdnPjiAXH1vLLTl+xzX73bMJflVEGK4QEDgLGHmgXJRvTTBexfgchA6odgBH/heQWsxshFXaSWhkwF5gT0oKlKOJrYfW87I+WZ9NSt1DtN+rDT336la1X1jM3aLQn81K/vC64OTfon6/Px5rpiXNcq7KU12lMw+F8OFi06tvgiBWlyhA9wcdPh/1zVtPxkMsv3i7eyFjglueaTyONgeHXwQ4BNoEHXUh6a9LZAiYSAwYwq8pWQekp+T6pvERElD5yf8Vyo85hMgsPKe4V8kXMQLp/fAayrTijb8byD9+eIf4KT1efLrMTg0/Iu/r67F5Pz3Spn8NDHM8M6I1TvH4s9IreuUXY8pJIFOQxmf9XA/QTJiYMznyE85w4MZ3n52ULzqurT2W13n3nZGN/czG8WM6Pd1rlx6efv3awf/TgV/bu/NOf/tHvvvfueNTNVWKNLEqh1/aUlHS9mpFnw/377AAVww5YSTpNnEWr8T5psJbOAK3LCpcNz5+KVlaDXG7gFkT4eX/tsktfK5cv67VCUzfq9Wy9ka1UcsXaYFEctMe97um09yQzOi4v2s3MoJbprxcvG4iEJER8ZtLX7GiqYkmtPO3Ni+JVuBiE48tdmSmeanjzi8FFOIrKZWWZW611862KraKv9UoVGvLU0iALaze2djZJ3iK4BQqRXlrEltQZj6Hr5RdfwFFOnkhnmunvq0C8mmNVQQzzROUGgydHj6U2RK5tnu+2IR9BY2cuwmuV6ltvvTOYTSksPCHWTYUfdrP65oYGblb8wYMHj588yhw/QRhkl0Lm04vzd95559rBLcVr4F1Io8Pp5vaOsCRNzXmdj0+e2CE8I6pyVO5zLXU7xWt7PEQf+/BrjOd/7a/8l932GctS9BiZTnc2G9ubDfuRs2Y+mDrZoSeXjA1xC9pngA8YDAxC7wGjgkzBe8O8GCgR7NlKRx2oMGPA3IiVEwSDGKLv4Wdj4M9XBdygm0HHw6oEM624G6/pOsyP5EhASLAr9JMSViAxBuTniP6QmvzPmsdBNeVIiwjdaL1QWcxUORfSwnUY/n0DoAEF1oQJGi3A0nQRIzyp0Er3DaIW1OwftLnQaXH57zwr3jE2MxAjjc+0pe8xS54fZh+KL/JNyuULoqsp0Wi06dfA4IiUDutcPMW8BU1Kl4bRjRvbG4ejRvRq6g1HKimVFCrd8S6jydFg1FsTFampHP9LfTPVQ6C1eDulANVIrbFC8yGAZowHmxHDyuYg8Za0WKuU5NlYx1pDyWgRbjAj0pOsCNEgVjY2L2mQ/uJYUsq8WULTWO3gu6L26TBBgzBjr0C+AA05VZQmLKiyUxAooBk28IV+27ISGFcC61k5whLEplVizy1HbcdaI19u5ptbi/tPnghQWMtdtHs6DiizCXi5IgyOygl2xehr483dIPO7xtguy1VT4WpGvnzn+JSkzI40n9Xor8BDebtGrSQLCZxEsQjjpq4lOAJglfFAhyNgOI1q51E/1Kc8mnZ3UNHcT3hXvujKwMcFA0b5yUO5Dbc+8OJr16/defGFD6ixo2bA/S+8WStvUBMFW5lkPCMcBFBCVJsq0hPtESuCK1J6mqL0M12Qz+6dqFg1GvXQ60r0jdS3Q1Sm+Nbw7aGskWBqvBEtAU2iixBSKCzM8iQIAiuxiaU1NKTVGoTBPC2aMLxsa9qZnZ71Dvvjo3xjevOFjQ9+8k7l1a3MJk3zaJntZUrClEeD9oDJvFmrzbtSTuFsNkpeRJSBuas1ZVwKLJfm5Kg1ltTbOZ92jtnGyg1slfIaiws9QExAMCt0wHaQdQOB2BQHak6AsD9vFNBji3Nw0yCvPlcaFumCWBKZSMj/Nz/DHM1y4cViJ1AsGXS9cnCUtIHPED7jUenI6nDIZ46ZHkieZMM4ErIwtE+gvbo6Bo9CBvEJJT/oRXw6J3aQjaA7JE5fcd8gGSFaeJuY9tjsxI9ezfdA6vhuafwb54SfPnbS7bwsySQqcDtZiDejXi5XydTFDfTVbptd9FiJwh6VHV87qD84/ts3b/7AyaOf3Hn59/+Jf+tH/vL/9ScePTh55e7B/Xee1Iv70S1SFFxkhaTnIjKCXaJMBzXImoSeFG8QFcNo4NFEIg0pyI+3CtrLzCWewIACf8MnkOY4zFMj3bmSzVmFOGbv0Vp2IDqQ5XJrO6fmk8imRnNZLlPOWDUJ+6dnRMnO4Pzh5ei0lRvUq7P14rSVXTRLmarKOtg5oxwzGy3MEmgusRITydbET8kNMybboB2crNqMgZLJdqid09FUAXbJEKHBjccX56dSAZu16u616+V6VX8k5fCeXDzY2ulU6+sAy5tsbnEGv1LI33v04PHZ+ZEAuK2tBq2HGY4Er67U2/fuM+lohOCGilCah1qztXdTJ8CbYWBLaRGanwpDQYOOzk6jJ4ImMaVSW8Tmvfe8qvwkGR2lmrp14/uPHt99/pXRgnGJfttWWmf3YD8o3vLyQkbyk9Nac/P45Gy+LBBaL85OR4MeWURI5PW97UqZRXAIrbc3m5T4Xme01dpo1croQn7cX+OyFj2jiJzIIVAb/Szya0pfs5trrSwwj+UXz4ewOEdBmfDgUwguTCRMic8dRAUoNGUxUk4MUC358dRkW2uIKVBwMCCaoByQGjJSEowR6MsRTsCRvhSkqkw2p134q3MR/uHGbM6IqkKAHkpLIx3LcgdPwhkwdOgfLjWEPrnzUmo450VWM2PXxtPVhRtE76tarbqJRibYDfYXuBPnQBX/BOCmLY3xm19XB+GYbSXO24+vaWzp8ArP8Hk3j+IwYKJSyrXmmYtsrrcWI4HnXjmR2aVpbw2G0b6QREab70rIXEx0oCa1jVVqmKliWjGrZhz/Fh68f+tFwQWhwtJLvXeOwWZd1lyh3KrWNPwpg0JXRfwQaBt0mAeSTiEeImoQMuUHmJLdzttqteCg5CPJsWAF6X/++RciAK8oBXwqHMBLlJV0y+cV8VBsjLyJYyFpUY1iNOYedJ/+jIgTllvaokmQO2/psYusWijYCmnoMkfHXYWiCOw3IzHXURV7pgcvcT4Z6oqzmXyXUqa8nik0P/j7//B/8L0/FEbxxfJnfuZn/oP/8P+wI4p4Nn/xueenmhL1uoIxtSjSFSEEtOlEt4atRqtRbhy994CY/Pjho0G/jecpErW50dxqNDCCar1B0MZROVR5i2XlI5GkaeFrk+VopjxtOUfmh4AkN8VvVGZs96NBYKMqYnWiEshwnlG59rOf/sN3br5wsHeDW2yg3WenS17e3WiEdyVCb0LZCwprqgmCWhpXsiq+FtluhMep03/06PHhw1F7sFVcL4hFjGjSsDuE/Awi1tjctPsh/bCqJ0sCgh7IDKbH5MZSZLiHeUL4e9SaFIk6UY1AKTO5pYpuz0CcaKhO5vhR743WrczO6xsv7d/ZvV1r3K5n1slvh5nsIJMdRm8Ldubs2kahYU2jN3qO8AqbYuQy50I9UtxGOZLOA/16kQx+eZngzM4VFmcNeUl2QcCDqQbvS7gD9pN9JNAh2BOmG6zYnATvI0BwSMAIsnoo86oKYvRIPjQNS0hIqnMCl+g99nsGLJitlTYJw8BFfpkK+CqUL6FYPDBED6w1oguUvgBS6C/cZivD84AjThTRYzEU6B2fZjY4JfxONruQkljsEisiRXsTjCr4KK4b7+aJ8Y6mOZMXaRQkIv5i4iO+l5xEe4+HumvMBM5idKEGqN2VTgyaQBGO+VmxbYmCpayEmqjx1YVIqSfILOCEkVVwgUIPDKJuJ0R+fR2RfFupU0mXhRc+/WP/xsf+2p/9+b/3xd/+8AufOL/fr67NayUeLRxjXio01fZH2dR6lyw9vxxG+rziapQRtkPRDIWimCkCjkUXvuQz9BzYWpRTF24OvtBoWpSCNYQAcDPw+PZ0/llcdkntleqi0RQjOS9Uyhvr2hXo48H9r1FuxA0NT08ffn05Oc0vxgcblYNGcT48Y1gt1YvVbWr8toUevPuexV30CYdLURHlGosXA9FyOOQfUQ2u1Mw0tAwTVKFWnTd6cnjBAKP8Rb5Mki7OTvsjvX7lChdbHG+nDM7dXodWMZmen6tLMzu4GYUv8uXqzRduXpx0BlCzc4EEqmaDTyiXUauX33v82KzXW5uj6eU79x4zRG+si0MW0dXMy10+7+xu7lAVtHJprW8qkMdsNrl3X74Ns9vrL77487/4S+++dx+fnhydn3eHL3/4463ta5hlR5+0fOELX3rj9p273PMPHt0XiSVRuF5rdPuDt956l/GSi9pgv/hbn9/YWBdlqarlX/jP/9zB3vZmq1bKSsHKazaMqDZv1jZaLW3C6eX0ImX8mCb7rGghPwTwLnrddrnerBarKJX5D0IRNufIRiWnwrngS4gq+DIrwXr5rum+ImkAFY9jOZevC19TgSTYHsAE9vAy7BwogOuAReSBBIhH3g77qr5+kaYAnwKcw2PiE6KJU3FeDCyoEvIRxm/yc4zTfwmte9nLPku1ceHcS/bDZYMeLEMjJGpswbk8fMjNioSkBzz9WD3u2eeK764+A4ufngYxA58d+pYtiZ8kYgRBYxSRO+qyR5sx1o55gXykgbruNz2FQ3J6TW6Ts5QxyJVbne7RbD4aT7o5DoWy3BQ1LULYlWVzmZeWvuFuqkzXmnv5wgYdRQYL6i0MAUaZ9TBUSH3ggwr/ZUbZb2zSBBpHoFlUAonztM8DfwL8IubF4sHTqLrBbswkLnhZnnqQILSPsG6SOEvArsvhAL7op1VvXV8TgUOIgvhijeHwlpme8t8t01xPp6R3WpTw9yV1JkSciHBVFzhJI0wBBUbyUPwEWAuiW6tWmVDN3j/xr/5r33j8BHR+9Y03vvTVr4PiV195aVtJZMIM7QEnmYxZhDgLWZ+4e0l/aCbSrqI5FZstuAJCUedpgdVaIaKAH6SHlq6eRqupZa5hMXUipICIxcZUlAoNLiYGlW5n1r+4wKq2Wjc/+eFPfezDnxp0hqpRzwdZ8s2aIvqaTvIZ9+B5HUC4BUA0g3xV0YcFYd5qZc7EsT6a3h8a6lgheLU86qXsRJqQk4lp7DQmzyQDQsSfBQEicVvE8gWYgas11K82iqzmqKKk7QZbSmIBM4kd/WlnNB5Go0f1ZfVtWk6G5ZMPf/bGjddarZdukHUza+3MZfty2RktOsvsGIvRaDnq0SYmpRRYoIw15joIQckztY8cmkwdnKbh656yYQl2DDCPcIwQfo01+EoSPoPTJHQACSsl1UxavPRa/vGHKoQkESzTi9pxsxC8w/ULEcPZAs4Q/viMg/6CN8dfzMPVCA0glBxzHNJHcL0YwQrxAkqhYRpMsFdfVzj9bMf43SnNZ+CrK43JFSSfeJX4DDqyYvDpd7sB3RRagqJHxQmh6ccbEQRiYPEtnRRCpR1k0PvYVoQixpgeGYMLCmoj4qlxB09iEtamS4s2zt/ORSMDfzQYNRioEuEVHfVm79RqezlrLjxg74Xv+SMfKDYffu23vvr6C6/3H5wBKZUdoGO33WWAKVW2okkQCGI6iQKkITMk0BJiGQ0yjIHWxAUYtjmYR+Ycy8GTWk4nFwYvZCJs/wpa9VW+yFwOHa/V8utbpa3dwqbypDUKBFrUk/At4Pfo+OzoqHN2uhgcVdZOW8XJRiO/XVmUF1KNzvOZiYisa9d3N7c3J4OxyM/M2gWWIHKAxWkw7C/XxgXxz3BdOLhQaiOR7ZOLVjE1QVlFWq+ishxTOkHl1KRsNesyiHb3dxrN9U63x15EXBDeyrSLxRw/eaRW1d7B7sH1fZRt2B2cHB1CNyS0JB6FQng566iA6+XLlabAKD6nCCzpsHVvbYcBF1Lw0MtT6ubHMEEVW8WCrJNq1OrLHh2f6nf0+sc/ub+7++jwUPvCd959dONmACZD87Xr1ze3to6Pj/r9gRWQLsxezccXPGm2VBwTR3/w4L6Uwu2tdeTxN7/024/v3Z8Ouy3uMeVk89lRb0o5Pj89pTg0alVZmyj/okgFLeej2fAEGwP8PMQi2ixfTtlAHd10cERsRLEwmgOtIG9JaLTSgI10KmsEKqQQW4RZ3K3JYdir4hNABG8IATFtYdBMJJ4/EiMQh85T7LQV9sWNQfI3twT7jsYdwjQEyp9uFAYBRCRQgjS3RW8tquBCBDKyoksRHY3V+RMDE2pHHIhcgRXOJHRdPWmFQs8e+Tu+euGkniT0cm0aQzo56ArMTbI/cA0C45Gc1hqjBCUKe7t/gf9CRoxS4DvbWweUTfapzU0y79qwd5ov1qWQ00s6Gs1dDNTSqURr5zJGLomBJVJd0bLYQucwLoFdhv7pGI0oRAprJIBGfJsXFmeESkXEAU4cPaPMRMC01BTqquAgNTtK5VYrEgQEx8MQ08k5INOcGORyCorRj3Sk51ON0K+IwCIaUHntLJR4CZ9eGJmDvy5mRiX/J+ozmBE2SAbM4KphbjNVBpM2pE8Nh9AYg2rF3DGZ0IuY6w1WdxH2KDIWkFr+D//kv7Kx3qKS6/T3F//8n//ql7/81v33tqtlFaEY3CHskl+joLz7VraYbSCgRWJHCFhzlS+lSGt5Vq3Ilyizeui2C3DlJA2VmmNKKTZr28vZo0GX/lWQg6fYU8jZw+yyf9ntDVr1/Cc+9qnPfOKzB3vXGXTOjjv7WweESXno6oCgY2zsbOkEP9Zmb8E4LBIdZ0s1ICHAYvn4EbtZp30qDCI8Oss5xhzkOzsK8TGA27sHZQwxNkEzQMfegkAi/ARKC5aZt8965lxQF71aFCf1CPitFRYFtQzcOjuaKr8zP7qsTl/9yIvXvvN7MjcWmcogs7iYnp3P1wbC8vKVpeIGOF3oe7E27h62LXBqVBFxDSxFirGJgKR+X69jRjOsQvgY8TWE60AcC2pbKYZXTGa1rMDaQIF7/B6f7h84Sa5O++YIXsQr4wz49Oov8o6C3eLyXI7qqIisSQ5gzzXWiAwMhTWwMm78TSxzZ++w+gRmXgPwp/eKTzJXMEiT6iLfjCyx6jgnmLZ2b3HD+ETVTHV6u3RuTI1HBe91cZwfiyFBm1SkkVY4iSiZANyyAeyEzzE2p4WUz1jnNonVx4BderU5xziuXj/IQkA9hR4bgDitphqq8rxUfGNlAB7MhcwZMo5k5Q1F9y/GD3KV1vZnXvuefG5wfvzmvd++s/nC9HTx+LS/X98lhPe643ppKuIfrkXNO2o5VolKRaHZRSU7Fevh5tH+Ly/42CKDOKp9Joo2SyiKtKIMbkyQHyGdxZICznqFljc3qwf75a0t4Z1yS0bnZx216M7Phu3z9tHh+ZPDmc5jfL3b+a1yZqdZ3V/XzbvYW/YJc3u76/sHO5Va+ULsrdiCigLRFtwcLpAbFUgwFB5f44H8/R4VccDziwI0q03yiS4mwTL07Z2ObxzsEyeUkZK2y56qAh36UG02eoN+JBFF25Usxnxt/2Cj1RQ5Le5pa2NDlzD8AZ4G4PK6IwJRJB+MT0VUsfahclgy+CcKBKmkaTQbt2tNstRwNOv0uhQP9kCUTwWFF59/fmdLMatNWb/feOtdFueBZVtbu//uO5ub68JK7r33DqvtKuipfX4hm1GwtgTDw4ePt7Z397Z3rOrjx/cVeNPsgclT87THD+/vuLgpnmnNbdl+H96/B/+Q2LU811SZE7FFIUvuwqhggLhH5eko0LEsFvVvIXinuqRh6wn0gioAiA1Z4SI+rQDdMAgFCXZ+aopeJeOs4Br+rChQQtRgwR5h44zjIvYsDjD4G2hm2gIz4inp0xcbqA8aFn9BBW2WkguDoMW27q97OacEKw/OiVxCS3JFPIyEyBMRVt3kn3GtRyRkTHdNWP1NzIlXeP+WECk9NCFoSB5O8Jk21wH8kKxXGBzCgf+zymReEizC7YkzsNrSaPP5yub69Wp9S0HUdHEUg2g1zFdxo4HhKgs1HwyPkTXBe7X6FiMzMzAiF8mWqkxTEhZiyLnvAzRhmtfnYzELpi6tRQQKurMJF4SMAEyU4wH0pZIq5AGXYqPCBiMm0JMjutjJABGmuoMda2HuVV9jwXbD4I16kZRCNgo4TuHNLvGTR9viDpJs0iSENSPYsmDiCMB2LWOi84K0hfbnL2ikHdOXskCZfIJYCrXzeogc/iXhp1Di3Fjop3f9zs3/xf/u3z1+790/+a/+q7mNDaLt5KK9t7l+iSXms/oAUhA1JxHRwvrMljsYWG310uvlRk2behDoWZ7P4B9h4wj9vKDWxXJaqpe3SNLZZXnc7oy7E0Rot7732kc+8onXP3Vj77oeekcPLuolFQDunB+dlnLl0hojhCZcGe3k3JZESUFM4VeoneVSj2IIY/lvcF+qNN1DmJlkIIgMmUezbkVsgLActZWBNohMVkyAYioATNLw0GqqJrNCGBaI/+ZWJzVyUhisPUemRvXy60++1rpWGqydnAzfu/HKxmd+5Dszd7Yy/YeZPNftQP2NeWGs7lKpzhTB1zCTCx0+hBAO8YsQjZgHoKbMEalR7GlIBm8DBCZv4ce0eW8qp4YytUKDmEP6gqNXKBIwEyj5DAXiuwUOILC+XsizgEqycMVZSa9lc16x3ogT06LJH2MZxyNBxU9hwRIO4eUNNUDFHeOGwbVW93/6uHjGygOd8Dh+jnNCZQ0UDLoRV7lHkhr8iGRM4UqMOF0YV8S18bd6RwAXX9/3q8BBpWLwjUXUJIuXs1QrmSOudfd0flq/q2udkoa6+khmjjgp3trrsMGRM6EgwmFmjh9eru8vKlvRMmk2jXjVgsAhrdSqkiw70T4lJ/2bl6bQ/NDW71988K/+mV87mT4pKqnSXB+gasVya6MwWXQlUBO2MRowQtaib6QgSxQvwgqioIwCeoITGCeXai5Pu0Idl8vuTPerCOqPFtmFktYcQifWyKubW/XNdQHESlV1umcabD9+/OD89ElX4r4OmN3z5aC7US5e26ze2VVHQyZVdq+piGRr2DI3062d9WqtKG2fz3OtnK1vNXvj4dnsAkNVFtrCcXvptS3wP1JLVRBaW9ObSLGq9fVNmaWK3DmHhU8Uxf7Orgbh/W5bv0sLwyYTKjUrYrG4rsjfzo5342Zq1mvyaM9Oj0UN0KJ5i8O26p0okiq/ZpaYLgbhQcRZdMknuoTxC+/qagIoIzdOVLAZyeIPnguUAbpO8yCF352Jgqn8jIAoASRY0jhZtnQussyatmnNgvfIJSHXRWJFFH/IQl7smde5Ua8+eXz/nbffZB3bWG8+enAhQtvbDXoanFcUt8d6Dx89ViQk1FNJASJcNDfGgA0d1ITGQ96Ofu/0rzrNackCubjErEEuOgGwxInwMPPU8cPpqr6z3eBjQemtfLmqE0CdQZ9wFGbAwEi4gfrEjq9eFcPwGcmdSf31Vn5KeJG4L/gNVF99riD7iv85jVxNPqU0WHtVkoCWfHHeB3l3UFlJesa+WWhNdEOTy03iLSFJYENCFngU+OkR7gXkVw+4+vTbt2yBXYnrGlD8u/o9bhc8JdAqyd8IrLfDgSvBKNk6Sepr6soJPFIlo1Gvba2xvnKUrrJrBPUSPbJyXejNGMY8X7ACuEij0doQ22t6PCgpdlEFGu67XbnKFhuTR8FlJFZZ39NDj0R94gUpDmEWRdWSMoErL/TSojdyA1gbaBAZo3JMxmhDJFajAlQ5do74S0YhfiMGMpu7iYPjNIhuvdGxJxVzRF5pNGGEW80E0yES72VTSIOTJJLGrGBGUfg41jxW3IG0I5asSBqOKUwhR/E7ULCgne64tbN5dnjo5lsH19iVCrXK//rf+/du37j94O13//yf/tNdMz0cN3Y2sji0XIjJQFhLlGpIxlCPoEMso23SIssYQ9y/1He4hXhKVNCUWHh1ObueWwx7yMKI4a38wec+8fytFz/7se9j9ANq0ehvLsG8LA7p4qgt8xHHYr+lA4XDLOpYZQF0PpyzSAnNWFFM4K+Ulc5og2pJAVzvyyijIWU0zmTlDjHAR4AbBTfiewI+A7AtZ0gJcUHwIAMGPFCP1yGPLo8ZmBmnq1BqfD47u+id1O5k3x195eCF5o/9oR/OvLie6bwz7X+9uFs/P33Q2GoUduoyvaIM2KyH2YAHo8C7KeoRoBbeCsP23MXk7PFsGkm9TPcWJrRjAJSYrqtCg8DGjCWBegC5McZaX20rUPQl3iEYUxwINu01QKAvVD4vQ99d6b4A7ZsMGA8WYoIcA8U4jkMnvXxFdQJwVs8LhghgTEzAfdw1sDU9NGAnHXSKLXQ/YwnCYzBePYYbY3aJT/nZRAo2KNerg0oSMREm1Vy4KO4cknnsxMlxZ6l7YSvA1tkAjCKNKKbAq8b6prd0lzRjfrV5UhIa0n4MNXZ8WnlqoDM9N+KOOKJJg/3MuE3wnAqJZLThYmBUgdDFUnY0FAhyQT4cXkzK/X62+WLzO1/4Y/nv+yt/5nOiil++faf3zoAnZWejzO3KBO4VLGBCzWAnRbXho4xFsnyQHgQ2zaV9T86HGtwvLsaz/jLTXYhozWp4qfay2CJhC5lmnXmNwVOnrv54cNo+Pj49FuTca58Mzp5M2idr4546cI3s4vbG/os3mptl6aFL1Qib+UVNzVVeLayRBSm/1gVWs1GhXt6+uT9azM4ZVzrDTTnxidijEcpZYgflak3dKC0KdPhQ14Iw3YuCFTr0hbQnfUckigmUYUCaN2shw89nd2/d1pNXgn8qZafPQfiFE6LIW55Gl7FotypujOszykdzKjFK00bdGdGzCbyiqb53735jXWwVZhdx2TzdjMY7e7tf/trXCVWEbKpI0keipgeMUKSvmoghHQVQ3Xv7LbQRiyPkIOEg//TomOG6Uq+XtCnMF+i47fPQd6/t7z2Yjkb9vrZLmJ37eCkK8bAvtEUf0N7x0RE3YJ723e2vrbcaiEuEMuTLqBEGN5mOQBW9vRpJHBOaMECjizBXBVABdcQvCiWmatKCTQnXFgHIs64r94Lu01CA4QpHE50JgEkKlAEFZWI9CWtJWPLC/oRIpy1gNjafIB2JtuvdCavhnbK5MJFdd2BhJvgw8TM9kqtpwLBThouCBtphzxY5DoQhBQMGCv1EJ5B/PCfd5Oox7uVrDDWOxse3bYnaXJ3+7McYhC+B6GkjhqJmVkcNDC62cjaKg2Oo9do6LZbLJbqwxlsRc9GLeDsrNh11WMxpj6H4KiIpcF4/n6CBcUKExuZkrEYQarjKi0WyDrwmf4jooZeaQO0zbRJoLYdpTryTvksOiOkVvRxaAUVTnO40MnCMyvmWwCv7at7tOIgQ4zm0WY/zq086tjuskMDM4KoYkZkPto3xiWql4jMgAxRGuVB6XSvGmLSWssAJGEgYE9sy2vfGJEmATc+NcdNT3SfoRXgny6XyowfvcVs0G62/8zN/6+tf+eof+AN/4BPf/30CyweLzLngtHzuYHuT5fLk/KSQlQseUBqFKyK+NLZIJx/OGKmkLEXQiXp3JUrzpdJ+415m1LmcDfThrG2rwX5Hr5RbN2/cub57c95HGAumO3GcEC8MnnCjaQnBg2nH5FSjgg2wsSKKol7M9W8i+tKXw9EuYHEhQDvRZDfRfx679mYRfk33CUANZ3GoTOivT8tn2r26+YgAJ5/kHLOWoO/k4kyxr2UFQPf6s4thrnPZGBdb04MP7nzPx39g7eWNcPROvp5Z109JE9bj1vXCIiO9uRs4ZS65v4U05jnL5RkLrYiIWaPOKD0wHC11Rrw4pgRbXJKPwWC9BKgAA7wn1tG6Gkhsvhod7RTnCV4Ym+W6wsFgdiv4h3JhcnabOIDl2IlP753MzuHuZfAWa4vvkuZEYKXso1AQaa8rNdMI4kHpIcHoAsc9HvrbSWASjG11jp0Ytb/4zRfDcyYoD04a9BpABS3xq33zY2ISCQlgDUQiTOaxYevj7BUrdRtsXw/5BbtsDiybTWJRxJrFe5Pg4gHB9dMgV88LzHGZLR2MnRhVfErqMDqzahIoClADBBiGaOHFINOd9kvNBTOugCJOWK5Tc4FAsWdlcr1SLbrhpvka1L/j+3+o94mf/S/f+OK7n7/bfJVx8aR9KrWBSubJq6WS3sStkq8aItMmY3MUFxwtFv3p4mIyuZCQK3ZkPh9mS5NSccEMVmvmmuv5ZjNXqw6443OZVDlD5urx2dlxT/T7pDfuHC0H7eyoU11OdTbaa1Rf2K3c3iqNu6eVvJL3FJ+u6MX+sFffqO9m9sxDxKnlc/VmKzIyzs7zUHpaQ2oCuULKY+hCOnLr6+uKyDJBs/pCKvwONHpdAMwXBYAxI7Y3rB1SUOJk+2xsb2hIJl6JZ5uDwBxjwDb6BJVW1Q8ODNWsmk2tCPVo6Z+fHbPd6TAoytVtiZuoIiMBYqwBd6SzIhNRGyc8d7RIA0B2vYIUKokVCCUrsYExFNWrNevmJuMiZq2jKKY6un5zA3cTzd5ud4SC4qB7u7tM4n5CMg4fP/7yG7+9vbVBunYJty25gYWMxBWxT7PM48daG5wDJ3FePK+F4UC50RGnNJYvdQfoaIGg5QkJv3N+CJhqZb3dI4pShos2NFE0Em8g0An+lL6RqzPxASAwjJ2K6vLHVEKBAIIoo9cDkwkcQ7fyMsngGfzb8qBtHhHnRPzRijMFBMeWUDHtwMMA9ODmPpK8HZwAaiXyhbB7NygTxISOuLbkhVZmxQ19j4ADekBIDeJPcSZasgfFkK7wJzA+nvO7bUEF0nE7qy2+BrUJipCGFXhpPgNDcQgLrUiGt2PNsNQpG4qtK2iXGRMcYZ7SPb34JeNS2AAVY2OIJn9HQDkTDfVZDyXm9CSgsDfFxEVMlIa/Ya60kPxp4cQdexDWjICnl8WZUZZ8pVwzNhcRa1gcgtDgVKLQwnNZwEYtQ1qR2HEhBIEn09FYUeL0kjEdZB83Jx5R/kIHTJfhqZHlYyqxvwhmNS0prMvMxy1tDOYzue/priDK0MIo6AfXEIQ9y+aIK/xGsKINPzp8/Nyd5yj26op85OOf+D0/+HuBxmg87Y8Gjb3dH/1n/ukPXN//4It3f+Ev/6UHb74hrqKp8DqiFhKY9CUmpqq3Nks14FeohhaqAOVaVW5Qv42SzE4e9pulvdsf/fDdmy/eOLhTzFa63WH3iO8zF/n5iuOJPlc8ny85nNtlOho1FoAhbtRgfstgt7Nh9+w+HmJiwZspMhN+JqNwFCVpKV4u0eQAEtIO30GcFuU15pHkFIphMrkGHJMeGAPNg2IIJo8UMyu3iipD9hdnF9PDWaW9eaf03Mf391/ZyLy4kcmcXY7u9yYXRBoigQlUcKlY0QNggOdXa5VipSl9gJd1OhyX6s1gmhwT6lj1B+No8yharZ9PKdkBr6zhaaixLgn4E3xbk9DHIUQEiTkpDmBksd6xtMH7AuyDtzhOT/cR4nOAWRz3fr6am8SAgwdjxjNRCUvGZ5HPejKuWh6BbtgSqByenGDbaQs8itElCFsdWn26d/zisTGEOD+JCzEyPxEc/MWVanEiHJxQcRihSk7jdAuXhr89XR5Ks9EH2QrPGVKSHimOAdJFGoCG0jOVyQP8OQ+MVShCeF6cHP8tzb/YAgNHWww9zeBqoOkz1Ikk31iEYOQiQQNR9ADQp5D1xJGRUAwiknwRpe0nvctKyxxHtay8QHqC4KQrcaE8aFz7ke/9RL/z03/h86PiTqW0P+mrbCOX1iiI5+EG4naly86LmdH8UqgGi4POuiIcutO5Ulb9xRob4KRekxtQEG+kQkC1JimwL+b5ctHjs4kKEWdqRPQ65zJoLqe97Gyw6J+X5oPyYrRRyhw0Kte3mzu1bDHqtwxFjxCqLid8eVIwRoVRUZgQLZbTjRENgQnDaF8eRL5OzcVPirJsNY6pEk3ZBptRI35d+11tPWENl7BiO7Ua5yjEWUbgkLauqgJMlMJcNsRbiZzZ3QS/2DPjEJpg5TlZ1NvF8LBbnL0pTias/GRnTEDtvLXt7c0QtxKLBVz6SpW5AHe2RbCZN2FsNnRTaPf5xRk3kp68g4F+vyMhWhg2hVVJyu3NLaUIwtLFao2VzBcakTIyx0G1tKYnllV3cCZrlnPvhZdxnOsDW69VBGdxxmunJglFasn9e++KBGpFb9OKl0AliPg3b13Pb29fw/PdkVBAb1wr1dSSB2sSJo87R1/76hclYN2+cfOVF1+RHHl42t59/oN0GvCB3UFJbCCi2uc60FTgw1pUZOTyxH2DQIDBMMBQq4K/BrkN1pF4sCl2whWNBnixhR4WLNWG9a62qx3InxAtmFw6IYDeBgzDiplujuYC+6AWIJ0QoOx3NgzChYg/dVUoIWkLyg/fAgOTTLvCyKsf3/9PPDKE6/T/+3+I/fRjEr0T+gUyejBhLwgyAakY5SFdijyZLOam99/A24boAOiww4zojHC7xsuQKjQfDGasy3doE8yHeJQWYGpMUrjAJUdjmi4KYDQFCt1vuYwoA9I0pzfPemLG7smlT8Yx58k/RCUL7y+gBU/ipxKrCPKvqjzqgFlDJyfQkEMCXdkrguiEUpyWyJpHqY1QlUNisZbyV8LBGVQ4PMfBbIzBTQwsHYnZs2OsDtplkrfomJ1XM0K/JjjI474iHl3VbG04P6qLRIiehSzwL33Hd31qC9PY2zw+O1Q3aGtvPzyzJJcxLwbpQYgyh2xcUhwGt5fSyJgj7D9SXCYFTOfDL31yZ/P6RmNbW4WTB+3p6KQswLzcBAokyUih5PoLZYhzJWz8pHXZe4oVoeCXKnicn0o8WEz7ZQE6mbmM7LBPOokLmrl46WQmM6eCwXhxv7P9hgQjLD0mK2Qu6pdrgml5+ThmAUMJVK4zajh7stjm7EV7fDgv9nY/1HjlUx+rfUgx7WEmd5qZP5yu9S8rauCT2wpAA3SpwTmfaM20VIcLckmiInuyRYtnYZHPjKbo4qinkOQgYsdoCTIKdOMAEJR0KGDJ2KkNL9YghFGbwQNbvwaaWbNYthXuAWZSlFeIX1eqYQB9Wlj4HX4J6xlwHXqtE0MCcBAUz3M8GanpgqVBJ5FSP4X4RW+Kq2JiPC+x/PRkI4E7Hp5sB+m3+D2G4kyE5ptbKP5hWvCCkZMDZZKlIWRe8d7MniEEskTHf+neYTVJCxL8OJZqxX3TjlgXcRfLtcpsbX2RXVdGPw5nZfydFzNKPQzNAJMVCKZFBxOMoJMA49+xETucREvhVrdWhh0KC2PPMIrqeKACL+0n7dp8VN2uFlrr88mFiCl+T6NVLyeCby4ztQ0O2/uFwa+++gc/0KiWfuLP/lpv1H7h7qvH751X8WCBBTFLgj7zM3FPa9n2bHG2zE0us0NlWfThnXLO5TRw5eutbO3Qd/NN5dmr02y+r3qzvKCxGIZuty2597jfPl3ISZ2Pc/N+ftavrU3Lumxm561cTi2MdYxVe6PTtphMthRRBss6bxnjtQgeEQ/Li/MuUdLK0Az654PeaVfnws0dllWLM6swA25siMfgsMRqUAAkyCeT7CpCilEazzM9VMPqOCaW/1ib0eWyyc0zPZxQlyVTtc+7j54cEliRU8QHySLEbTQbSB+vaufinGtDlMzNGzfEfIEpV0lEvMkfXsijVAW1REq8xQHXF3rDpAKr+HSrIT23cHrKyLzcP9h79PAx+zYSfufW7VIuL1oqmkRKTJ5MZAZ76/Fscu3g4N69e5J3nrtzm8okaejk6FhROU5ljaFefvnlVNVnqR3hzuZWQoxMt9OnHy9mVZ6iXq8bFYQElLQvxq3mnlLxIH5ro6lTKoe9SroPH3zlq1/9fK973CEZnb/31tc+V8iWv++H/tD2RuXoQuZY6EPKV7tKD1dZF/1O78WXPxhZhPT3Da5N4anWIyAkUDihVnBoYnuKwMKiyCA4LiQUmOOrc/FmhZJWoLzCQJ/BT9NN8LZA6IA5/xO98SPsQRDKgnlNrK9wXRRhyvnIkKsvXZQT4XtXCTF4L4UwVEA4jziHOBr3gZArGE6YGU+OwQZerUaBUl7RI98NZTUq+yR+yOVMO2Au3TBGTtBzqa+JZkD4IO8AwBXpjiDGL57qBycSqkZqU7DWhwpBFQ6TnColwYbDiBUmmrDROF+mtFJJ0po7bfakaHNE/gpHyHTOquMIz0fYvKXPKwkXCcVE3orpZX10WoqoSsUx9OurFIlN3pKlLRz+OQGA+DJUDfeJQC64AQFW5iwUh2ApNbYGfKs10cXMMKwYmGX4oZlQoCCnCLO7ZM6o7RLdihxEcyw9bTjofbBnE8bgJl/Zi4VMZrOO+D07urB+JDuxcGI9P7SpNgp1WPPdzpObL9xSmCYzHdWqpUdnJ8/d2MXqLSZhWvSEpsbChvk/W9UNXub8vLTZWMeVxx0dEbJbjetb0qdzNTWgTtWfWys2843Luvrwk95ZRx9Qq8r2iCZiqdZb31bxktmNRmYUFemEhJgaQiMampfT7Lz0Ot7Iglqj8O7JJSBeUvEwg4i78xHymCUlNwcHsrZW3vsrEYZRoMlry96oM9KdHLxU5P+N1dIbLM9Hayd3Prb94U99MvtiK1Pnq36kXtJsra+vMT0ZCC7lFskTFh3GkxiheopHIjUBdLEEoA29H466x4/IJypZsagQWVAWIBiqHhALZpD4WoL0YKJ+jrdJkBlwHn/B6gIog4/5MZ3lnEBHn6Q5vwT++c7iay0jEiBqkKy0Xhbmlc1ZYjOZhNlZ5loEFIQtGs5EREJwczMRnDZuags7dhpGIKj4hsC4kA7jWCQxBXr66nxXGRtIesp942TWUVODCmsRYijNesgB9arkvWV9PdPuLRTWM2qRSuDPC3ITtru8n5nt7a3T07NcI3c2yew/vzsYb731zqjWuv7em1ER4od/8JPt86+sV/rzyaP99eroojvsTHa3UFIlnmIo/ltROZ9pPtMEWnkjRgMC0dNkQo85fFFEB8QonZYZnofFuLIoVDab43Z3xhfTCHEklkOI8viIHCeTO9c/ufF7v+OP5T794/+vz735hC/m5e7JoFmpD2caDE8b+Wpv0j/DVLOli2WB4jvQwpPWwZ9cq2oxT62sbGysVUrm/nzYU95tMB2LcmYXnai4Mupqw4XoFHQjnvXz02H5clq8HNcLmWubredv7G+UCxcnR6xm129eA08soBUezXy+m2opMyYfH10szXvS4mTE3n/3HmV8q9pajGb9saj3SbFStliqyiELGBjqRCfGfZ9oVzqdbu3vo2B+krcA5XuRnzBBx1AMDtTT05OtvR3tfvV3M3tMrCrJID/UALFO1/d3icvo/2i8VivtmszgGNk1qQ0Mzth8k493d0earLkCoZEagWA2mxyxaF34ktTFVNOumH/tQx84OT5DTp9/7jnNgzHyMET3+/fefRvx3FhncG4HrdJ+Tapyt2OE4kE1U6nXWo/bh/zZF90OXX6vfPD7fu8P0cYfSECazB4/PhSKReIWSqXJKeKJTygEiyZ+/c2v5RvVFsM5nAaY8DNMLAwiuewv/dLP37v3lZ3NytZG5A5Px92LzvDzv/Zz1Y2DcrlpBdsXZ15Vo/jOBedBb3Njj2WOQSdb5J8wDaxkkeEZEZ6B0sFMrAEANcUElhX3te8I5E2YFUJA4JntSjddffEZ6BifoRCv/tKBMPKESsasmqrOMKhihzisRNC6no8Z5I3DAoX1GNmoVxY1uBevGspPIHKw1RUFCGpi7IHmId3GIwP5Yzfw1ijTCAPfkBD2n8RjfIaL2U+MtE67IierMcc4Q28I1SJ+TeDh/FAdeC39rdyuVCkszcmR/aWEmMA3xCgRbsQiooypSplLwhqxDtPEBYwvBMMlNjwS4W8AZsJjYj5DdYjRAyNH2ZmT55d+HFXJPQgHBHNG5gnhyIw6jsIjat1e1OgJhzHfhWBacQXxh5XGXY2VjRpDcnMIA7HFi+H9RFE/e7pNEQmfQgWcEGKExnxK3S2iJRHBB1E1x+Y36YAxucbmzt5afDpNJglp4f4xNJbZ3sXZRkWDqVlms5V59IAqyug26Y8bm+vUGqyIcCU6LZ9RjYOGx12r5HRlbSJoP6uxi508qsYfuihqrua+IREwia4ty2tFVSoZKDBUB8OmE1ZZBJ1UsOy8+zXeAISALkKpKiKaXo+VjJAUfCMBSYBLmGcBjFdyGoYVHMqbhOk2XpLzKOhfoSoGaDa+DGuRRwEY4dWBeNnRWrc7Px4u28WN7Ob28pP/5PdmWpNMQ1KKYhoXmXxP/ICOMIrWxA0Bv6xPS848HriA47GHKL/niB5ItGBWgal1Qu2wxyhzxKQZCOiR8Zmg0Y6xBVb6NGCIDzS/ucWPxr46OWFD7AYMPD0nuEowy9hWO341BjjOLRBSsRXD7JO+i+/mosxutB1M3FeycRhWvIuzn94yXseWhuTfQDpPMUrAHAONXQKEvfizSnGlp0JtIFVGwtRcGyx1n6vXM9JZIUK/u6hWMu89yOjL1eln8rVMY6v13qPO5n6LD1F4rTpy+oqh0Y+OTtavPf/g/En9+c0vn41//L/66qc+890v7H/6Mx967l//k//yRz77+9c3X3z05HMb5dyD0+56PtPaIu8g5sPwvL5vS1Ma44nhr14tMeiYpliBABK/eie4QFMP9UHtlayyJ9nIZI1qOtFshCjlb6FT7fhBrTItbGue9Ctbn/mOfzb//X/tz3yxPTnbunXz8aNTMKXV9eHpo/rOtWVj4403H85qjSliyCQiwosNRzMc1pGiks7RGrw96BxiMt0LRlc+1H77JDcdXI7p3ZPi2oK+mVuOyrkZmVRLrZ16fb/Z2KxWGuVCptFUGIfWKGFXmkQQ4vB8ofBZaeTqj487KqrQyQsRX38xvBRpLQ2P8WjWa7ZEBTGLhrcRyNiJzByNFzhxl0tMzuTQFMsR/rImRfbawU6ptCk4SfaOBBuG9vNum2s5McuUG1lV0k2eUoOMm/jZpTjT0qXCzqHCsSqrGiI7oarhqY7pnMHsaQg6AxTDbJHeObZ4Hm1N8OCL9jl6uP/xj9cq5VGtMikgpPnp3lystZCu+29/w/JiwPK+8GsSA+YVusdkZuTsiB7lzbQ7vHbtRv3s4t1HD93z3Xff1XRFNEazuS5e+uz8BKNE1wkDqLhc57AIZtc2tYJ49ODe9vbu5sZmKYWvJsUMc5jeuL7Ta4vtytaZUrmcL3OHD49PDt/51V/4me/7oR/d2m8tJiJjphenJ++89UCJ++d/78t0A2bqYknUGAUGgdbpCRInPIZQKxKeQqyRJJzDEpqIq+NQPpHep+e/H6wDbsNiHFCdFJUgHXHEP+hZYGAgbmgC3IJUnHKJaaRGs4iaGCpcupJpJ/A2WNOKbKw+UY0YhqeGzBn/pvAiD3JecLiwUcf1riJWwJg4BPGxROTAMCIXKWiDv6AJSVsI+mUzmT69YCAcsT98pALLHUtskvEeWVojq7tzuC5wJycHhkJDLbC58EKviGCQOIweRUVCjE3FEvAgVUwWb12lCDBXKpaBBaGSQwFqg3hCpRIt46g1lmh+ZHrw2YRHg77LQkJ+1LmPlzrob5iZF+wPXFB63Bmig7wHnulMRicnmxSzFJs5h3z4cqTGr/z6octehd2lyXRWzAD/g8vSJCD6rrLF6+DW9GW2uNVqcNusNGYTyqIadDUuRxQ4XTLjwef++l+9/5WvdO7dL0ymrz73qnzoHEeYJPVRdjzUOAHxULCtVs02S9mWZFd8x0zjAdHCA7dd5OURyQ6K/r6IBz4iUAT3wmXmsyj9jI2xUpA4JuErFXs17HZCdV2tiDcg1jibM8na49neLkhq3AB3WL1ssr2mBQwzRkAMBK81vGs0UpLDO+Omi6YSJWpaf34hwWE8P2sv7o+zRzu3Kh//zCvFT93MVJ9k8heirEWxzsLGM0PLluNFpUrltfw+IpMpmBvi7SFexGCi4NVcGbpwkbnrRGCN55tkWdjBt6xmwGJAWOINDsXrBCL5SPC6gv/VuP0YO7YA6ZBF49fVG9Py7V/pxiGKBPCanWjuG2AeiGJ0ckCipwyP7yjM64QPbWbxY2yYNQA6Bp8mroAwN07Piuc9HYxDhhaf6agTfDUIV6AajC1mO9SKhGNQilnBuDhhtLvEH47bqkSFTUNexPrzrcbG5lv374mL785ro43io8vywWsf/sKXvrzV2Lr36NH8wdHu7v68cm22s/Plo4u//JMP3rmXGdS7ux+8K1fzX/lT/6f//X/87//nf/Z/9fD4y9vNjNIlul+AW8WHlTUOwZo0EONONMOOMRELIrA03ioAGWGI8QdUp5eKN4YXYUWgVw54McKoUt+py8EjXc4zU52yI319KaZhMeg/quD5dQXLWrXv+fSnHz//d/7K10UOZVv1HicDma5a7oZGNB/V6v1iRR80W15hDdxXhY1IhpieHD0RVHTWPj8+eYxDwXWa32LY1RkzJzyL6ZHtSET1YiQnUnTvZq18fbO102wwbbFxrqkot7jsT0a6jgiwZG0WEMXSKAGHFijGgNcZU0dDpuPZqNtXyh8dI7HPllJdG0i7PN/GehOAiIGq1asnF6dn9O/hCBtDytvtC/SEZY5yyUtFuTztQJBlc71lPmmtjx49Qt+Atrmt1FrSgegKiImEakpCqGFcA5EEvDg+O3348GGkMHS6Cmb1onNCvaGz0sa2fzDXNTUvBX4wDyn0pzWKMlOIeLQbCe05+EpegFqTOuJu6BKXNU7HVJ5WlpqRYYgWEGuQ3MjlElP8Ovi37JSWUJBz2d/4zS98/re+6JHuiedX1fkqYBkTOrUazM5UkQEdaLTW85rXTca1zOUmGAqlDix7wnK5scH/Ldhaeo8IinmjtbncXn/wqD0fteuFxaRz/JXf+pWTswu6V6nU+vgnv1t7CmlK7B6oMpgLrhj8Q4hLAKH/DcVm+khAoVelKJJQTeF4Qj+vZ2QJ3eL0QLj4FvwqQTWSYoIC8RIz9nskz5AXoTWNzWTBCudz4q+3tqMgsxQ8WBHUPKhOeorpDcR+uvndkNww3JbxU9zC10RsV/geA3A5ohJ+s9Ba426xTEF7I2vQEbdJqxMlW907qFXagui5oUuwXvEdXokQj6rS1CONGZmaKwrDzhRBgkEm0ZFASmvpbZHsZMRiky2VyYdBQd0g3oDwIeKaGMDRR4ecESEJheCG2GWeXQdiGGYMK6YyghFoHzS96JLEUKf2H52YkEHJ81h6nktyEUXurqEWMzsbOodl9OmLCqPmgbYZbQzwzdXGuqwYW5iMw2tcYWMRCeXpxJwINQ3ibaxSYZCa4L4BY36N1jchbaSopSS1xMrOxSLT55KxgrQUXroioal/8df+k//4V3/m50ibzbWcBCoxwSrJLYbSl9WmUMQJV8wt6N3e/bKRnZaAsTY0ySXJJi6Ykr5lKsPvELxzteYgNElMBIRwzeJhbLWcQr2Lfk8Bub5Qi2B4waxNUnAWSwgEGNftpGVN7CGoLlKjKhFyE0DG9mPUIMAvuCL+OZz29TnNFuv5Or/U/ILfTVncPLPS8Vqjf/PDjQ998sXMi81MuZ9Zfm0+P5yr3RGpQ4pni5G0ykHGVPuyACYzQAo0+sPGjGTI3hoAYIvwCu44DmxVHUiqzseswEraidVwbVwfMHr1DvbetwVov+/PSZ6Y3tdHbM6NWMZ0TuK7+E9wX/Nn0TFas524b4Q947XyEtKOr2viPyNWEtjDVNyS9utWCWlXN04DWY0yPdTTjTNYsJNXI7cHT4K3QVbwDBNiQcNaexnhNfyRueyEM8tqA9RK7WIxP57keu9213dfPu72f/IvP/i+3//cpz/1Pf/2f/jnHh1lXnil/Sf+pX/lS2989a/++m/88A+/3O5n/9O/9Ks3ntv5kc/8nj//n//NduFv/PF//k/curax/rMvfene0Vfvn89Gww/e2hyOz5VyquUzzY01PWFXrNc/pvxqihIVSG+WpHgjRDFCcIE5JKIghD7NbSyKer8kvRxaP1WjFXEEkd4wM3MyeJHzIx72XrmiDtVv1IprL/3od+gh9vM/9UazerfnxQmP1crjzrQ9u9y9e3ctXwajMA5Eik0YCrDSPGQ8fPTgXRmFatb2tHeYjVh6QhhkFFIpXOHY7GWJ82g5LmQmGDDFZaexuc1CI4J+MFBTRnIkFjNtn4E5IEWlzMzF4QeWQX+9V7SBUS2yF40YhtJjIwiXy2QpF4glNHJ7YI8eKkQiNmqvTUNgjUOjghdg5NE6EOUZy+znmVarI8rjtBq37tw0pSdnZ5OjIw8SJzoUCyOysj+Q5eS9GOeEPKgw5ZWb9QifRl7JJBUVpKVqDqdjKFRtPNds7W4J5jqYkq4JtpEJr8K05/TQukKh7itTtjgvG/3WCkEo+gyiSlfE/ukvN25co9UE/2KriJoKSYFscOfPedEZlZmVtbVo1FuAYX//Gu3o5MmRydGxQcqvoiK5fIuRQOOT9Y0aOohv5ZuNXKmoK5l6UnzpM75IzaYGvYvHhw8YBGQ1SeO+OLqQrVpcK7dqAGDx8N2vvXfvrd/+/C+FWytbvH69tLvZMs8q8DNRwk/KPu0GynhANItL+OrFvBUcQ4jBjH0vEPJGCIZgNcDVOelfqxxgeoVzKwIR2uSKmATPtuxoUEBwgHDiZskjD9AZM9lGg1IE5YHqid74GnTDQxHe2OL3QJj4lpAhuG9wq1CM/BGg/SmKa+ZD3Gc0sAXqeBGm7WozhWx7Uwe8B464YryhL6yo3+r+CdE8IfEkp8T6TiUDixIEupo002aD35EDgiq5W2zxKKfKYYgALmUmolyJaQm+G2p8YDJrDxgyKjBBqCSpEQwV4gDuzDXJsotSERIDIcPwPJ3qm6BDkcUSIeisXrtDCoxqD9bdp6afou/mujh0uY1dRj4VLCDLnpfFbc2UlwqPLymYdVWDE0aGYgWYikROSxnvjiPjBRRlf96cIh0JSYnZM0KEvVpMEL7J+cwM7oJY2zT5Ud7PIqT4JJdcZn/9r/+1X/zx/+rFG3daJV3Hqp2Tjhar9cKuJgpqnWKdtYi5r5Uy6+XLdXV9qNfS9rgbo94iMV3EMLv+bNko1bApYVrETKECJoBVOVQk4oti3H3F4zr6iRojiUBMpSF6VQsPTIwuLW90MUiWarAKbkLwiv8S3IYZF1OwvsFWEn0NAU1L4VL2Ug1q0kJbtV39hmb1+aLEEXTxykdvPv8dr2eu60z4MDP9aoYoVp/nq4QX6w/OmHGBqxQn1kRFdxUsMlPIoIe4n1LQEzrlrK+gFRUqdOWwSycwN6lXwzb44LgxSpAdY01Q6gWA6fvxyzrGabE95cGBrKtjfkiYEj8ZmYPJ2kxtCNbLoU/zILjZgfMCrLg8Qr4MfxCOy/Wrjjz4DgZMigZrbuBlVlOXxhfPeboTDw008Hv6L06LLf0bk6uBjOkP8SioBaLghn5c5opyw3ssSrX12tbucC1/TD04uPlLv/nbX3/37IUP1F9+7fXd79r48qz8V/7MTzxRa2Mz88rv+xf+P//NV95448u3br78G8eZL/zmWxeT3P7a7d1bH9q89dVf+dKvfeTBx26++j0f+Z6Pff3wcb51cD4+zFVuzrGq7DEf80UXkBkZVpqkfLMbVCYGGmQpxpxm9eng/WtBydckUO9vBlwdWREozjjTPRmVBVts1wr1mr4HIJCVadKd1q83e+fdteWTyubeePBGuV7/nn/55W6m8ys/92Z7Vm9uvnTGebK5c33n+jRfblTr+C4WRZiEwsy89F0CpSBnYDPXCnMyLHDLI2royHyorY84Jj58w1Y9gdm7kc9z+iiiSv2SZ4gOBapWirXLujrk/RkcoT6i/dr0qv6aoW2vb29NhASN+50+cUKdvqluMAr1IJ5VTuiILBmJ9dJRoSpdKohevl6vHhzshdiK7ylSMR2JHKyuKR1dPhOGSKOeT4OL8wYJHiyWr1+/STHwRhfdHpJMXwnoWtLh+5IUgL0KGuQWfJ0v9uCAm78kyAriR3MX0CTMRlsklnBNnQbhdwe37kYCYGz0guFMi2IdyNGkp1Gw2uvhpDePndCHspd0GwIEL5SfxF5jXviwVVfY4/6DQ/ZpaLq9s6e2EQ3etRtb25Tj9kUX3WaCf/LF3xLILY6GQ6DMxVgsaZKsFki+PzgSGLu1sS56UvgOmx3GJuYTt9/eEkPY6V0sBoQykmde6c68QgZv/NavHh0dyg/e3T1gEmifHf/tv/kTf+Sf3G1t3IJdbKfyQoIOJHAMjPVfMK9QBBMIBtdFoNO+I3HwfVtgZ6BU0L1nm0vso8gkLsdRu7A/u5IEZBmgPmbp/pybIW0pxFgJYgELEqlZnesz7umfJDc/u3naSVQhDcbsWbdJRPxaYuEcifUmL2cw4GCMmaw+XYFk+BtSHxIHaYARMwh2ep+EXPG7m6+QExSIeyLkkJSwAUyJaMhm31Lfiid/plLdPFK/yZM+Q60w0pgMZNy4I0eOPboz6BqAE2xJlCkQxvC/cEgUQjlTutwM+BU8odznF53QXDGOSA6MnlE4OaKM+6KI2DDGyUdJlsTLu/0wDiJxzNqQNepTBN0OagjNQr+a4Lihy+LodryKBDUDcBMTTa5AHzmyPCFNlAsjmkyWTgwggYAkn5BWxL3I18JOTBj6ExTItMZXBJ4Yh4BDGXLt6btvb+Xyr92+86Vf+6JWaHubB83CZudsmstJPdig8oiAEM8s+pXrd9ybhBiVYltxXwyTyBLwMRf1wAQtbBg/U2LZ47Q8jJ5pAmpIBGZjpsCnaF2agfjpnDbMV+BqRFbQEI3RvRNYhu4S0AjksHBTFIw3Xha8BbSvaLD3WWbuPToVr1pUf76yvFBSY3Y8r4wLm5d/4F/8kVB5iw+lgS6L/Wx9AnX0igjrQcBUWC3JfKKX7XmsSLmYH8+JDilke7SBpksDEowYASUkNqN0alq1K74bwwSDibfhWeY3ID+iwK74g0WMM9w3XjONO43+6WHvk5QC52HRq59WZ2K9ifv6tIZGbZINUEh3Mj6HhBB5R/EXwkzUf2ZShPShuAOsxJriiavnxpE0kNVH4GgIIV4gnfD03/g1hhKXx7Sn9wtholgpdaaX2cb6+v7+/Yvhz/7qV067M07173ruM1/pfGnc2P3Fdy42Prr1b/2f/+1f/o1f+43/53/y6ge/U5DfpL7fzZ88//Hv+fVf//wP/BN//F/4k/+bD1z/1A/+/o9+x2e/7y/+xH/93LX93/ja3/2hH/0o97ziEzdffH36uHZ4vNzM7VIO66384f23GxWQbF2Cqjwd6mrSQ7AMKhawEWBhrAYd4hUfk70kygAfqxZLdpkZ9P2qKrBkXdIo4Yv7ICfWeHCv2zioUfunmfNcU3jRLzTX1/7gv/jRRa3+8z970rtcbN55Pr9+nVYLQy+6Z7hX4HI32ifQ8PicuFRGAyxcMSWyg09rw3AiIjeqp3H5mFARoJFSxf4cScWFCg4aoRIy8qgzciIAjowMiVnRYWF2KbaitLu9K2u/0qi1tja//s5bEfmAI4vHR6ekOKknZieV4QAck2M6gZpZ21vb26Kn8eDnn39+s7UuLqnTDt4MqDUIVmkEnUJhmGdrjXrw/+VC2UhWXM3CMSHR4Qp+4DLmMIZD1BeTTeddLLwxuoRu6FGqsi8qLfR6fXdn9/rNnYNrrY1NP2F9lUo/SF+KULFwlgm6IaEILx2ITAC32KXz4mMHw8mwR0/wK3Wg3WXxxQxTTPVsTjO2lO0OFpu7fv25w8dP3nzn7Q++/jEKJqypSTiq1u889xxaBpXFbaugiRSaEF2O+KgPru3uXb8mKud8NGwbxHprh4dscwdRi1pFaojocHzyWKFLFXNy4whAk+mlT8bGsHOmm0lDsO3aXO7LVH3u4yO9kV/76DXZnyOlZMJ0EpaVgD7rFnMYE3kFpgmFVojkYKIEgdrBVSi3wDW+BdAmAEbgYkskD4z7L2gfRTCUYPtBBsKqFXZjWZWceOyIvU6zuCcohRPCXRK6RtwppE6g7yYJif0A8sxWsEjPRZd8Eu7ITJGRq6jfUhytVCtyWmLDSOqKAXPwaxUTCTDSnABllrMZGsU7YpaJTGOZ6SWCcHlerOt8yv0eZRjBAdMMVktlYHthGo6psCG6ApAIKMpAM0FEjG3wWXwTgROyR6KV8RXCmiCeWgMC+NlDz85OQQlU8Y5ETjMO52a9DsGJq8PNKd0+mYFprjgOQyv+5mVIASbNjJ2fHMOQkt6GbMn8u7mIUXc3WWtoPbuKJfAsU2HCIvw12FCYl9FTt3Scams/1iUIevxsRa2qb94snexIhICiOZiLnFAZckQdTtqwjgOSoEoRMGwlvCUxdh3KDMfHb763WWqoZi9qtD0f7B3Ib9hvrO+auXqlZYrmlMxFZT6eBwYG9zH56gKRSCwBc7jCI2PzzeWNkAlt0QqtP5CMpBfGRCIvD43sdZOMDtHd8DfvH1AXFfcNKvhTgCq3+Krkhr1Q4ohZNqvtBdIp8doop8+4Bru5fffD0vmOLt7pXjzJNDvrz+Vf/OTu/sd2M8Xfjkgr1or5UJkbJgKInc82oxxSSJUECqBIcxxlFE/i0zY2PC1SSMOtHxNO5BMuOdVcIyhoMmBc8d146cQFAotW8lsoCWFVIlz6LSAzfgpw9ZbeIWFnQpX0UzoeH/F+of7GJMDIq88VyfaZuK/XdXn8+cpiwwcfum8abxii0/CTfhwcGlcNZwsOtLp/DDSw8tnmeLIuxClPf/Dc9DV+En0WZz+9wk/xl1FZ3/sUhQX82jeO/sYvzBDgux9Z/ODBC++dDrb393hd/vSf+4t/74tfaQ/7mzvbw0Wme3HxjXf+1te/+rX/4H//73/0ox/9f/xH//e3v/LO/+Cf+7H/95/5cz/8R37vv/9//FP/2Y//mfGy/wu/8eN/4o/9cz/543/18KgzOc794k/+8ut3t2qZ7qc/emt3+8Xx4CFsW70LCDfgGFr6SC8U05e++cmII+c83iRcaQSWK5nImUCJ/A5i+m2i/ySaVwEAJqdSa9buZXY1rp92+51S42haPD3rTre2/uAP/7GPZWvjL76xVtq42V4UHxw/Zu45fPQeO3nodr3ogglzSZcWY6ncNwqylGHKcsAHENFsS73YeJoNQ4WUkGiC/Yg2kGTPzQmx0dVIVSmuCSLm2yVJV4S9ok7qaTTWVaeSX0SskpukZapnqHqiqw52O9M4NBBcaGyov1Q+91fhEjBSgtudtXbnAgODj5tbEmkbdMrjJ0ez2WZIy8s5mZ6eIHY6KRV5yinSRysAYFFFcqYYpyhoxudqtaHpkBiwoTlHlE5Oz6kwODO3NNPfxtbmzTvP7Vy/uX/tuvqVQ7X2I75jJCOI8xTB8SBRo7QXZmthU5EKPFsQXEwdyYRAw7hXj5ploW+wSPPylmsxKqYFZDQev76+vV3c3Tt459339Axu7exubO+/+toHNze31bJlSvzCb37xS1/6bXU0yaFMEtQPxE1sJw4dGjBCde/Rva99+RvTib4u1Y+9/qHn7h6oyH9xfsjTXC5vVKujwQUYOEeRiGPdnjdcO9jfoRvqbaf/2sG1/foge3R8+JqOp9WieqDMz+gRGmFNA02CqIb5KQAvYNSPZi/0YDQKZwrpkNahrkCIxkmN9Hm1oRFI4YpS+PQHnuPrCmdTZbvQrvj3goNm2S6y4/Fpfb7rtkE2M0X3FdnDcMJ6xTLiyqRghpRJHgnlOlxR9qV4IhEYZJQ3E1MJEwQpAoigLEmG8HOoORl14yiwE8lO/JYRaI0Vic6JFzTawLQ0zvT2yIK7U1SkBwx1nZ+rTgWMvDFMPT05UXaM0uetMIAwCwWDhJxUyVAesUwcQm0IM0UKYLUVQwhunMN+q92C6ZX9zXTMT9jvjnhQvBYO7RzqpkpSZEKzzpKZS86SuM10WqtggbN+9DoZmczwS/KulopqkNM8TRzUhaJInRh6kD7od3HfZC2jzNIPw0eCGcg4Cu7ktYM7J9qCo8544iCdd0T0TZ3JDVUYYsxm/Sg+oE6CgLCIPxjwSMGdUBfMctAu0xjKPaVWVlf7sLO/flOYSOGy0h1Mb964dff2K83mro7C1er6RWfAvQplZ2M9g7FW4jmTr2d644C0gJIQKDPN3Y1wS44H3faRWnHTSS/mpCgLAAY4NclAK2oJuCR+Ci9JAsdKf3dHK2hzR7+6BJgmHhIsJYAjNNckRkQZh4juXWQZHOZv3nszU13sv9r85Cc/Xf1AM7PRzeQPM4XDea4brdotNqpnnRiZ9ZYbjtar26q6602k1FDEbWssHc9kkZnKuXaC3m2KHTkAtrh3WKHRcjMvJN9kG1oSBIw3oQqa6kQj9UNwPTYdkmialOAL3mP1GWvoEqsQVz7bqGtpMlc4GEDsa+K+QcbjSSlewhzACqsHO+Zam9K1crxhfMAxr94wzGrx5wSzZYzukx4YQG94pJZ4uP8hV6Bj2pwU6jrPSBxJXM2DPDjs/VY4sqi9HNzOl/sKQzV33zsb/93f+vwbDzO57czd29dyrY1f/crXv/h2++Vi6Uf/6D9R+M3fuvf4+FzHDM7e7NoP/fAP/qW/9BdU7AbsjVbxjS/d/x//67daG7u/+pXf+Df/1P/sB/7gd37v7/muT37i1c9/4e/dv/jiJ7/j5V/+qV8QYPx3fjFz+ujs6F1xXm//8X/q44JNs6gEGhLvdLV5hYT+yF/ATByNdwnKBUQgQoRDhDklQD7eyztGHL90/8y0HQu+KWNAiNd80X3v4foHX+o++oYiSdu3ygoRVhQIKRMefmFjvfiHfux7Nq8P/vpPf/7RmSTx3MPHj5aDoYqkOK8/ljAbDwUCRlMKyRZZY3+2EoYh7z1bGEv9Dz4v8DIyETTs2dpq7e1tbWOKqeg6NxkiGSYxDkot/yJyJdZewBQfrfqPSnhMGN0mIwEPkn7V1IU54hHGEYjHuRnl6AmXmKg+TXCK8nrYPTo5P6Pg9rs9DYsOblx/eHz6jXcfVrt94cRCCZmA+W52dvQuFVsKS4ZPjo+gGKqgYJTyRuOS2uwF7R9297cZt0fDvl+HfRWmQqqO6FClGGs11aO3t7cjoqoYSSUszovh7OTw8OTk8OjJI3Vkydl1Lb43WnKfULOOZlP5Ai5spiJCRXA123uT9iH3pCoqu/i4zLOLe5+etdeXaxubBzjrxUXn8fHxebe3ubv31tv3bt19tdWM4HHsXOmu7mjw3qNHd29fH40uUTPEW5wOWZUk1D07Uyyqtb99+Vg377XZxz72kVu3laTv3rx5p9s9OT9vz1kails6YOjaLde4O27rylCtt0gEkKpU487KCfjOcSU3qjT40bijx6ooTOu6U92+UGxa9FdkoURTUCSPTUMjLWhpSYIigFwkPtQi6kQgJfIW+BVMEWeEkwA3LG8JYfk6pDGGdza02+CUDLbNpSZIirvN2mtrg2odqC+7na9CiFLpZr68xzsAS+cK0kWwcaa6tqU2AqYZCZoZ6VVt4AWwiAhwEuJQghTZxsyWCw47RVXondyf0a+3qXrqWlHJ044wwvm4pEp2uYH7BvnTLi6i7gl6gXwIJDExMlriaxCXgYppQ1AsMY/W2oSC5vNCUINUc4727MI+6hXMlYmkN6TOugqQ5aSwZjV5xiIFHUyb69vidIlg+g/2O+2l1FjBzBVGknWSJhhS6GIh11YtVpXiRvO9rWvNVh11vOic8ptU1xuT2eDkySNjprE6X2FxGMJ9sq3S+d4uZD1/cqznttT427dvIxaEOPxlMuhh6Nw0NPfQ2oaCF0Scagzax468MTUaDljueIVqFbDQ2rAwXJeCGe6cfEkAjSuWvQjGFGYRVmhsA4LjtKXy+fkJaXK9uXl+1l1jR1rfYJHtniJiu52L8ebG9ie/4yPP336pXGlZKwXSJ8MlaAzOoeJiLT9ajrI1jvVIMEV3gAwLA+FVuy3e9uHhO0JQoNx0yNIn3DuMAUrOZK1ykqhQQhwNY9Uplj4p2AppDKZKjI9/MRBpy9mROCxp5lock/JpWzKUzFFWmeohmQRccYUT+vuT9mg+GlTaL/2T+7sfWG++eCPTzGQGj+fDR4sl7/XSrJg3acGMDMFc1MESpFvQ8KobDFWWqDfxalftEjQLlXwfJJRKXsTPQryAT8tIvwnuFUuQIrtjxwb0IJVTUHrvF9ZsLpUwBoQKFDAarMEv6dNCJVH5fUwkfomT+Fuxx8SfoRsGEr7D4LUxJWCARZP5ia8Qo6XsXuqDM41YOa4blU/D+ByGsIj58NA06rgp7Sg2PCnGu0KakG1CVIjHpt8QAoptSFBx0Dz5Wstedi4uc82Mlj6iblj63jk8XRQLe8+//I2HZ1+8d/blo0z5eqawvrH70outg73idqu6u9aZTX/8J39yvbnxoQ9+9K233jo6fGT6/9aP/9d18m0+85f/0n/63Z/91Hd/7/5f/Bv/yT/7J/7kcLNfLWzorf6LP/vGr//83/uxP/adueljoyjVuqdrw+aNzKSSad3N/Nwbme///dM9SwXQI7gkzGimB3cCWmENCvhPdrE0jzFZsRMzEfKT2QwunFYvnYDYMdB42Vkn05nMGhuF/Fa1sZ3vP/mG+MuGOIfusmIyFXavTcqV0/7F3y5Wzr/r+7/3cbvzf/uPfnq62C/ltvCGs6OLqLE8wWyGshl7ykfUNpkziWeaCKNnGhJHk+6ZBjwIVF/AFmMeJgsWddfWNZfUdN5rM1TRJqQDic/kaibToxSkLxBQx7X0PuqdgQXatihlFKGB78n9LdfG1Vb77NQ03Lh1oA7Uw3v3mVF1PpDliHScnJxhA93e6MnpRZD1grCrWX3v1vad4cMnD82l9CVzJBIaMUZZWdO5w6r5YnWd4brIqn4x0PtugSVqhIR48k8fPpxdtE/oyAp9gL2tnYpqGL3RWEejF154aUQIGY4QIp7ws/sP3/7KV7ice50L9bdvXbsuLNryMQHq8oJfyg9pd9qphP5aq9b003H7XBgMT914wi+mfjU6eevwkI1/2u5NK63cwa3nvvK1rx+xTZZrx8cXnXZfvd33Hh56WRS5UK1t7O20RwMGR70o1BQoZ6cb1c11RFJSJZLHRFGr9cXgaYwnM2nYzTBylYoNrkoVzQRqMYfF3Narmk2sU5xzJaAmlRjymCkyNWc+Ko8+94YCQ9YUH5Fc0xm2g2axpXJQcQQltReIJp4EECEZPA6mFwoQzAubJUj0P1BMUdSBmTE78RdsmEoNPldwHDcgdoN8tRf9DGgoDUwrUdw8Av4uspetbLZlsbloUTP9lrAC9D/oFaVzPpxrRK0ddVga2Je5RqhxbuuF1L5gQWGCFsUtQIENB8qExyTiC1g/xHoGTcCk4wq0cin3lDSbqm/SG9OreUwIEF7A88KWUq0znoREgtAomAepGqSrTT9LrsVE2I4tM8tMayOD7bGoGEeezTCiiJWEUls74glZH5V9IYVESLPCDgahOCowFf8qzShMNTHJ/PygM6lYKcBKz+02UfHCTPJukPV6/a56O/jl7Zs3YBcT9cXZORjwGCZor9nvnqO1sA5xwdN49wbqwwmxygnFk2VySXwbtM9ZylmuPJ0bSfjx2qw3HwpxjJx6L8QoobqOfpDmHrGbDMkcQj+m4i2dyiXC4UrCaNQ189zAhy4eH7Fhblzbz/Tn9z7/hffeOnn9zkdf/8FP7Fy7pVLU8Hx4dkTaxdhp5m4ZJWQEADGy438Wr5ov0RKBmdIK2d0dkzR+6y0l1LVGj7Sk2YL3OazUoaJG7axKKFNJwbLQQeXJfQF2AA2gBAuOWDxMhpIi+ifQPoEILz7qZZekb0GEvxDyC+Ps8En/6Kj/sNjKvvadL9/6zg9mXhxeVs4Wy68se6ZGsYBpVPkgI2FK9FEPidsF9EdcnaXE+Hl7pmPGeQDnkzQLiVTWj1MSKnANJsnGDVaKZ/CrtCVYW+2GR87xuL0t3gYs+i8USM9d/RT8L3ZjS/MQHPHZ3eIcJyfG6AR0MS6PT/8ZjghFGMmuFCIKGh7eFUiTVF6fkX0UjuFo45I04DB2xeOSxJWelx7sw8FA9LQTaGM6oqSJVQqXMb4LpeAbdNJ3bWcz14+UGEJyBP7ktm988rPf91/8xM986e2HjzuZwk4m31oXaHvUbWe21t/70peu3bkDUD758U/9f//sX3j99ddb9cbbOuspCTmab241bt/aZbH78te/QrR57vkbv/rlX25ea1Yvt3/sx/75Dz5/7W/+jT/7sz/1U6++uvvSc3d5WlSI0G3k4VmmtMxsb2T6k8s9ZqrEehOliqEbbMzgaimMONGseDW8NiYxQC/9HBpHLI/DSToJJh41QkJKyaI9Pest0IkUTjySrTSL1N5SNShhv6PkHafQdP4l9eN+9Ec/c3bxoX/n3/0LO1sfHfVKG/Xm8ZO3ERyyqJhk+tPDx29v17eVx7cYscBBEA0hXLwg3LqSAqEhnyKZIKKiVFHrjZiU+G6EayACrN/ofT/sfsAwol+roz41zG0AP1saWTwKIIJLFS7n8/Va0wCuc43euePG8mIjNig3JsQTT/UVwg7KlSp1k1tIO2C+FDUyW9t7J6eP1tU9rtV5Tu+99+C0UrK/v7fHLMwQKMg/OgpdzimlmvthSVgy7UdvpdAlkJueWOwllCQJ0R9oEeKQe6PpzsEB7eVrb7zR0Sbw7Te9Jt+20tXKbO3sbRsVpaikpZI4yV6f4gugYSj6jJ1wLXNJIqfKWEoonM1OmJeRWGiKKa5vbFPuD66rvRX5xtJBSRhbQlNr9c0tEZURELd7sPV3/87PnB49lHOs/53KZdg/osQCl3/0+FDmqFXsdjtvvf02ZZxJ8Oz8iSSnxUywpggztRVEzjQiJn22EMFFBGYK04UtahLI2I7srPnDh/dfeKUj16KXrArVcvXw9FB5RHpf5JZHiGQYIYMKh6EyACChuo8gd6v9hICg09+KT6OJq69XIEtcJAcHP6asBFfETPjpkV82gyBOKFkYxy7z/R5WMeMOT+4HViohUquKWIEEomH7w3MctFyRzsaaGknWRgUVQFSKGbOEAkiC2IQIEdUaWDrZRlX3xYKIrlSm8BaH9TTK2kxoK4BaBM0VKUlIFuKCsadUboyKuZF32Wi9FwC1BbGR8FqqaLVFojA9vEQ2DzKMNC8QF9qivcjZZafXdglsURHJrNIxnEZi4JOIETIw+lVJNCboMoVKdl8OZMpZ5GVxW2Cqdjn3p3oNONjGVisyl4qRuYRWVECEsEZlLS7ZJ3TujCJQLk/mCvLzpVhoa8hlIgCLsVthYaFMEeYsfYg4RPTRkmypS0EINWKzODbRE/sg2+wK7JiNO0IHOJ5CDIhsYG1d3D5/8tZDUlI53yjMo1p8pl86fO/hL//tz//RP/zPHVQP6o11drJBh54phrnWqquSkyd6Mzoxa9XqZQNnmHVkmhlJN8io1ONF33yTucmbij3stdvprdCHFNHMDIcLLfP6lyY5CdBFKIAV4qOJFBAEM+Yf7wi4QjOpLGBv3rlAarA+aVrsfGJ7CxUGoPpJ/7gtNqw83niu/MnXPrz/fCO3nc3ULjIRRzKYzEc8SiYjbz6inInJN9+JwQeE20eGQY4dUCAyPWzN5tYeh0EYZSxsOi/AxcnJEer8ZGOOy59tidXCBMOOLYApba6Of1dokniEMx1YnbPiIs6xTKuD6eq4HPU3VbGDh8RusnKjcJEiQALBgAX0CJiNpCsiHR48JYevvL9C0AkwLjaRtmDmSefzGv6CA6XP1Vs5IUA9BPOYcYwiBhR/cMpfVB/OZBrF0unJcFlZa+7s3X941Ds+qzw4fOPth+8+yYjo3bmzMXKTtYxKT7PHjyqbre/87s/ev3//znN3f+APfN+v/PqvkSzhQ3OjcfjkbOdaWXwshP6lX/3NT33q46+8+vpLH/3Y2eHs8J3eT//kT91/8Tqj7g98/6ePj74+nIh63Tg5ORbdGJUcWCmyme55P3eNJSNx1IT2Mer0LmnA1ie9tcF7tZiGUIODzLxvi4lNB+CgHZf7hC9syEgBy1FpI2Ux4GCUIfpvAUVy1nAtd1GsbPQmj3P5R3/sxz527947f/n/95vZy5db9d3jk7Ygo92dg/v3jngznru91+0ckTipP8t5FYzLHMesBBnoTcoSJNqRUqUeE+psPKzNwYdl1nJjihzO0QuVAxBEQvFACVnOEBERTyECMwUBBfsRi5mSS10u38ImutNAq+Ik1xsIFAsUlMdzJas3IyTKeupNiYagvRFd4dFCsNoCm7OFWm1x1qFl5rw3pHX57du36s0mHXd00fFc02hsdNb1VovzlLw2unffvDpnd3eP7nv37l39GI5PjlTsivkaTx4+uD89F/PUde1uY1dCUueivb+/m9zkpics7RBBtLNgzToLe6MlxurajdvimRFR+HL9psKTs6OTE6ydCooSahx8ooaajORinotwa3tTII4KP3zZt+/cfOWVV958883Pf/5zOp0vJt3+5paCYKEqLHMPHt4THZ1vrm86fWNr58G9dxFb5k1UWEQ1JckoWQ6zWEskF00087FUaotJ81cCW0hRuCgL8sfM3vLrX/vyi6985NXXPq4MPqNBUbUmwpKYblFrKYgpIDCpv6DSlqAN3AVqAjwiSaBmfCSdONEF34zBGYGswNRJQQflIJhP/Ais4ABhW8OWeZGZy9K9owDWWJsPTu28+o216B7BJyk6KWgcxsmRxBU/XMsqN1rxUOoaecLhSJgJBgpCqfWRX4nCYO68D2jCfDmUex5mYUJE+JWxSSPkfYSNJAb+FVyqmt5jhWZBBL2Ca1ml8NRkCvTqaBJKEvPAURMIuCYMr+E4cUltRWdgShG/FKZy4mkE5Wqn1R9OBW9pPKCOORP05bwS+btUXv5bkYCR83ZZr6nP2MJ9oQGGpLst7ss+DL6JpyErcc1FP04A1mw1WkbY7Qj2GzmwtbE5HLV13YsaSvRDE6CTijgpXRDNn7tHw0cuuYp+9uLhlxOsM1oJAV6ENvXpY4gWia5Cjbr7xazSjcR6FoZk/yDeV9am4W6PCmCyVSijRXEIDpucXlcJIEy6wobZO+8XJvVPv/bZ53ZeybTnw86ANihEUZokINKBlziv5ku5VQpg7Ojafbm+0Wzt72fabYUYx6cnGmV3Ls5AKUsaC05NjjICZOaTSz9mHA8n3WdKAXJRJycidBO55yXlXqRPA61gPWkRmTrCQCIuhHGiJ+O+zA0tnXrtYtw5uzgbF4at5xovvHb91oc3M3cYN/qZy9P5XIGB00LUI6Krg1HMOAS1qI7OeeOOIQyEzBjAbw5xOO1adFoaKEzZF5htMvFSwC6xIIbhimBasQH7OAIwn26roQayJLxafaZz4xVcEZ/u9uzX0IoDA30+2953fhy3cOHqxTecTA7wr1k0qFB/k3YrB0m8FSE1rJg6XaLf1lYEFkEi3imBDab9zWfZNWrvHYwr0DGxnRg3OFsNhNobzJfcixPHmel8aI+onI8n7Ulm69r2sto87Nx763j8W4/+1tkoU2hGIUzZsyedDgsfwfHi4b3OW9Mf+cPXvvM7v/NXfu1zajSJJHr5xRcObu7/nb/6ix/67ruaj0zOZwr3M7F+7je/9MY33v3jf6LYP+t+40tf+nf+l//zL/zGzwnW+7s//84nPvFqPl/d27/TH3yJS2Nntz7rng7amdPjx7lrWzH+NOnexr4V8j/oioNGnn4NoTqYc7zy6kicF+ueZt/BmOr07q4Llz2DW6RzD48uN9ZmpRbnSm48HS67w4I4qHJ0WJqM2+Xc4/X1W/fv//zNW9/97/1vf2zUa//EX3/v9GTw/J2Nh48fP3zY39jYvrg4G6lWo5oplXtWicir0HhhtOrRE6bN6DEEV8NPhLoEs/crrY6Zyls4gCheBo+8FJ9FO5RyjbasJH7JEQYNPEjqXsXxxMYmKM+K/jguTKlYyCkq+d577w3PT4U0h0DPJqNPUYQvqzlVi7RV9+wq5ijWC1no1+sNScbYGBRWLoPTYncyERIFBEylCO4wAS+Wjc2tZovyUqdG9vHY2eLmzWuvvvqq9+LFQ9hhkJCUe+++E/nHYdFcaMOEXtIY/v90vQfAbOdZ3zm995mv93p71VWvluQKGBswJNQFE0PYJYUO2SUhISQEUiALJEsKEGwDxjHuyLYkq3fd3u/X6/Te2/7+79wri5SjT3Nnzpw5561Pf/4Pt+1RJ8LlLter+HcVI+0Bs8PHG1Yl8gM6enU/S0UOvGPXr11Di1teWkL6Np2Kjfh8kJwDBw5Wq28h4SGhZNOp6al5po7ET9K/lODU721trr715hsEFR09dHB2YmTt5tWp0dixw8vofSwAx/Fjpwg+IggXYz0kPB4bQuonQzqZTDMi7DomngghuUGbDYwV3QbCLbKR6g/jwMJiBodA8YRW3Lp2dXxqMTE8oRzSUpmJgZ0wH5CPwZozm10bnK8Gr4Mz77zyxojpUHOOwV40i5vfax1rTZtVjX4m6wll/ni6pQ9iNqhb7m7fZ+gLmEHErQVx0OIvtLvCCv6AWUI5GEJteik/BO4gyUt6IzmuQmnJMusTMkyfkG1FdjC6agRMa3ksGpFs4FBMlihar1g1Vh7WKHFI5KQLzYKeonRKj9EPDN3QG/4kJUD94QEDqYJGyCrAmo5WAcxgUUvrZaAEboXwiPzL0NE8E/UAI8SLJ+7AtxTe4taMLcsTZwFB3+hHNIMVg1BJSWFuQtsVLQt7r2XwvcRjEZheOrOn8HqHYyg2DitiuoGjMSOrSltodUT5MBj0nMRBpSFRSYw4cIKCALwjzxtbKG49aixXLbgyuu2ygo69FDmRC4nBEVoEWEdKjkEnJ24eYYnNLLGfV+RsKyXXWEwyK5AhRDQ0OjJ1XdyWpr2Xb/XLDJjD4w02G7ZKqQ2Y6PzQ9P6tfQ9w43YiIgOwSOwdRHbAvSAX7EvAq8hJDsRjTAAGCcoIYWLC00NcBhPnJ0wrFEQ+IeEfQ5bR36TOcSCMQCNZPIRgG5Io9oKFmUkk2ljlpI2Vwjg/YQ8qToJwBWVtAFjvwr7kbdsb2c5esVnserv24d4Dj5wIT3st0x6LL9do36yU9hwBgG+cZKvh08IniHjNmEA2GRIQRgguZ7KhXVodLFAoF1NM2cAqub3sPOiSIKNkkOT5wIkPCDojpxBCbZ/BC5KgVtht9qZlx6Fvb+8yHgGhN2cMfR8wYHONXnS1dpguGLyan77zc6kFMGBji5GhEDcLjEFjY+BNtIfaMGAUX+E8owejrmPmFPcd5CnhlpGerYAObsqo69V0W1YdtY3NZrqglprFwtlBwVA0IsQ+TE0aeE6KX0E3EVGcMUfHF7x4Y/3CuqXttaDkAaZgBWe/3srVWnkCcj1VaC1EFGH0L/7iL++79+5MJkUO5n/8j79PEE8kEvrp3v/1nice+70/+P1jJ0+wKdCyqo3+H//pH+UyyS985gvNet5pr29tXajWk8MT3pX165OjUzOzC6TQQx2H4qFizUIMg98FQRgQK42hGcg7bzS5IgFmiCFdvLvDe9UbJmMw4PoVrFfJBGwUbsZVyk3StMtejV0n3QPZ1Rsh+hZUV8QybNGQPwDgLLnCPhBVVNfN5Z4fjh39lV9+79joxd//98+UCuGF2dmNjWS9ChAEaQxU7CA62U2RCQSing3kDUo2sANwg0K6EF8hZuwh7LISqbFR0UIIFGDVFDvCqIxOS/9wMrLyaSLcF5swF3Cg9TKnWNZ4NVRTkLpQJ7ove1ujAWlC3IcdQgAVUNLvw4N7/SKGVXYyXQQeRH3uGpEaHxL0zebKFMqKpO/1RocS5NcGIv5SpZEvbuCtw9kDGUS1yxeTeGYozIsfbZCKBGT0EHHI0ShKtjQBojKIQrF0MskdYKMojtiplgjgQIgnEwISiNoqBMwWtnS0TVc0FsMYieheqtYwARO24g+ENza3AeLY2tyORaLgDsHQAZicnp6mI2urtw4cXEqn9ldlT8ZF4Njf25LFVxC2nbffqF2+eI6jkEnjwl+Yn2vXh9rV/PhoYmJiDJx2phit0XhSLRbSpPAyIjZkUpBpRpPNJiOc6JLZJEROsRvh81optJcXeU5xuDLwNr/Xf/7cWbJaP/AdHyUbZj+bIw9TWwy9QUol7gUazNLjZ+aduQmqBosPfdjsSr4XI9Fq1UblITye32rTSoHVGdYk/7CflcJD4T+rP0IbAJmCEPWxfELDui7+IqFhrz/C3BDQYgzZWtqYYNjuNB33oawWfSLRMZBieZAxXYoaTFEANEZJI18XbBRZzJlDOC6iD1ok5FPWD9Q20y/tMygDGq2KQ8iXQwLyQJSFZtBgDm1Dhg7mhC2BLtNF2ACPgx1bldgNbgamOmm9GG3ECx0uXGhQspY8CORRog84EQkp4onWBpcV223U2bOwTLYXAgFLH2kUyAxC92DbDKHXF4hG3emdPdMzpF2M3tQ2cPNw2s/6qFeAN69zO0UMEudVbxULLcQ7Nr1suS2CoomdwcKE+CklFbsTE0Gvi7kqjyPvHWKEOQTXlUio8ohU8oZmIwPyGJiOgoaYNpNdh9yiJMEO8Z4MHfuCQCgg3/kFvLyb2St7XKCIeCGl7r4XXatWaO3U0yF3lKxsDLF15YORNOFECCciguknK4ehRmjHdG5p18t7O6n93Uax6He7I9i+WEyya+D0tUWCASRCLRg8WkyKsbNxgTiJiD+nmD5GHJBieDC0Bi1P7ozBQbCf5susn1KvQrUGHC9NCpC782T0Lp6YGT8xaZkOWGwFIDUanXTPW/Z7QMAjuK0Z8Q3TDAI7oWiAuiK2IQt3G9Sw0tYQm2L7YMwiIYo9y5Kr8Wg9WPPFPkAoG2TtDsKWmDmZZKX2mu2gTnAfveqMDq3H2x/V/ME5nXjXSX3xNw9zGS/6LRcO/thqbHEpIqxo2YIYMKXKMFSDV4LYIJJMhIA2FelBqUGWOkwanVh+X3YbvzU7gc5wW20F7srDeS/tV2dMfw1F4TxGTqNuy/GpVmuzcwPTL/aN200so9UXTpaal1bKmZqF4raNDgDFmHjcMoy0iam0gZJEgROsixZ3cGxiighGCsZdv3blX/+b3zpz+tTxE0dhw//PL/yr8YXIRz/ysctXrr117sqp02euXrl59s2XThxd9jqsP/fzf+eDH7zXHwlPzQQunn/r6tWzI2cmRod8VxsNr7MSG7c4RywnDyLIVtgLDJkhXGbHa45uT4G6aVYRU0oHODR/XDX4YL4288dAQWIGMwR9VHelCNh6FCMhuiNXrQ3BIIaAu2HQiuwYOClTEI8QJJUiFchtb+5sPTUx9eQ//IcPUuj2t3/jGQK1F0xwEEJjIh7PFyCSNAsgU+pHQHZYWWjVfTfeoy4x69QRRFoWcaFR9Ig0eQ4mEWoDojLLFU6FkRlBFkoG3eCg11ADo85acVnynpMDpqvfygklXCqoE7kVhDcvLy/jqeQC8CYhL2FyN8nm6IH3nkcihkXF4wdGnni8WChfvHhxe2vDT5qOx0WBH4KiAfkDGwutgMYx1HjREBWIlAbZGq8vT6eMEobryHAEKog2zMW0BwgqkUqyjvgr5EGkC1ARMuS3e134iVnNKCGZZAo5ZGJyfHs/if8ECOhCsYwlHEy5/XTB6Q5Y7e6RsSn6OjYyPD41dfPa9Ww6g2BHZy9cuMCbZDqVSu7BK5GSMSE36uW1lRu5Qh5LJaJAu1FBBFeBG1uvUCliYCavicC0UjEfDgUdGXQFVoRZ69hj0XTLpSrliv2dMD5eZf9ijsSGB8AkZLIJx0Kcl9rH+DLKxpUH+B/10ezgCb395huLB47OHTgId8ELyeZlFAh54RWCwrxC0LXq7pAMrU9Wo47BNjW8UnyXFQLn4yu9VwutVOjTHfRjcURIKeuHuEEMxJjGpVCpPoyVNGSeByrjELqUBEWpnkR0sOchpFJDWNnAlYSCcTi2XLotq89b5yZUd6biH1ez+lGYESyUUyELJm5DeAYVlqD7VFgSlSYemE1ELiaatEmyUuKN/GGYAwibUqNFJenGYAOyENlppu8G7FH0TNsAc6YkEApP262K1MfyY/wizLqM3A7ycCHAXfi9j8zuQADyzNkaU60SBHYn0fiaHQeiJdehOKEtsGVwDXA3JCrFESiuLyea0KO4AopjNYfiK3mjw36j2cp7VaYpIgYyRxFphmxqFq5RvqGWAAs7G706XDkeCePXx7pL42Hb1P2k1A1WZhO3zqTSUkK1BAwBs5e9XTYFuCaPEQYJAo7TGnKAoEMyWNtJHT/llzcJabdORBas9mCnZClnGy4gYpxBVBqIF3NBUAWMktUAIBQ+Bkgs/lEMY85oSCj75Vzuyo1kJolXF9dLMOQXEl2lylBr4ZGbhMuKaBlEeBzVEq5EyweGDVy6CBzodYh1WkwwWa0v3sFySDhFN7PhfIDRC1AX7dXWtE17N4p7jX52fD505NT88GLAkuhbArlq9nUPse1Bq4cb9oE/7ngJsrbb2I2aemqBkniGy5+Hg1REhBoqDwYT1F2NPcZmFiaDJ5TFwUFbZRBhUA0n0v4YeEP5Gg4l1jZYWfSGqR1sIqZFv+aVpaUP2miDr/Qzzpjf6JpvHeYnxtAzuFgsU7/SQ2RG1lJlYPgIY1TpREQbE/AsRAdxXxYIDgL5HFA5EMC4GO6L9gKrppVa5uxJid2wGZpmVF72JMI1T0AE0hVqqywlYk4ykIo567emF/A39iQ9oGC2rWP3NDuOmyRXNyy+qKXcslF0z+L0Tk0sxPu9nb3tqC+RK2UJYeSmCF4nTpx6+cUX9nc2RxOJW1evz02N//6/fxbSOTzpYWp+7df+2e//wX8AWG1tY/Nzn/v82689Pz7qn52eWFiYT2auLy7HF5ZGI6HTb7xw7sj0oYlRN+a1sKtw38nEcDADIn4jh8VIDFgtN4d6SHcGfTFnxH3VLU0EfVI/BrNjzgymUeQBugrN0G3otVRrBs3RokCiaktW0yV+5QzL4NbrEgNOuLDFTg0hwirKuVCiHw2Ad/V1i2v+7/7U45V87nd+61yzemVsaJR0gVQ653JFIGIURjHqDYONcB0BrdCOjktIqYGtwenFziaT4QABAABJREFUXdFfsZtB6mFP2sV48lxASMkwSzwN5JJ1gAoB3eE85QpgfvxK5j1lWqK5Mahwb6JqyG9kZJQ7RJLMzMxULBJKxMKozihtsGGCohk1CA1aMNthZGx8Zn7pwLFTm9t7e3v7t26uBYMkFaIKF107u7iDcSgODw8PxaPcAUaOoDY8MbadzHR3pOPJUN1qjNtHC1RPSyVF48nIwGzbaGXSyQqeKa1Usm8DkESOouo11FFCbl6/sXjwEBa6XCYHx2cegPXFGkkocSZL+FV5dGxyenoGiLFEPEYtHLj79evXN9dXaQw9InwausHQ7Sb3edjk2Di2ZNoP0AIL3RIJRMJ+vHUkSpXyOfy8RP8RHVbMptjz/kBQEW5GhNHawBRDHlU+X1iYn0HpKOetZIdioQ242EBuuxKIKCXpQaWAhKDeQDCQiSFo2KGR4KcmxtPFFpWHh8YmmBK4Cm4YXpkqRRvRTGC2tOZu0xg9UusSncMsOO08vmINah2Yy6BCvB2sbF4NqdC3nIeUsjvRTlVy1S3cUdlYMKMiqlDgy+MNwdUgBmh6+P1UqE0/7xGkbsXsYid+DcBwlnHLAlSji3C4NPZBQlpRRxVfxdO6mFaExAuvx9+P45PQPzLn2EzSdG1eXkHzZABREiEiJEzQIe1Gxl17jFWujkh+NztNDj90Q8KpZdYjMpkdQeFW3Miq3MIhpyQqoFazuC46onAMQUiysRN00OlSsYD/lyeKjBEQJ+8LkepOtoHEz0IRKQp0GvhMuVjeq5RDqm5NC2xcRi4v3kU2BUyRIceWi36DYAsMGwIujafyQgGnqQOpA/oJNWU1YpIFasRODg9yb2A8MRQLUeOekjuOvoMaaEgLSATao0h4UB0szOKzWPvlRidKV9GR3E5avwz+9BANFnEa0yVIIZQIBGeG2KtSpu0hOafLRie20lsvYjhWmjwGdgQmes5EC9Ff/lyB2ViiYUtyL3Nzq8wytfWC1KXHmFEvg5ADkyY0VcOoYYYhSOsk3d6cY/BxoDA9jD8xJ6xhsHClchrvhJiEUTVt0noxABFXYaeNpBkrrqhqr6Uq6dhS8IEzd42fHFWl3uZapbVnbdc8E4wptgxVAISOoSvg9iCgzhvysEfk5CF5utJgsyDxOL2IGiVCGsHjMqx3UJ/QQiyf/Mw6tOplgWRdiW+xiu7sA7OWBu3k1Zig9QNz6FrzvV5Zfu+c58e85+z/5uBKFq2ukaZvnjl4FSuUBmy2I8IIQgP+cezPyi9SLDwrUWyYYcLsrDAdiRGylvFqbgWh5K5Mgp6vBokuqX+md1iTuD2dk7iNHqDNb/a19pD2j/6XxK9XtlmeCl3RobbNubXXoeStI+7vNm2JxES+VD1w4BgxAiSTBCKECtiKlUw5nzt978Nf/O9/RRk6/Jdbq+uYJy+dPb+9s8U0R2LBbJ6kwvJffOrThAk9+egTjz/x4D/51Z85e/b8zFx3eDQ2PTtDAjyQ+6vU7QGxtJIfHwYg3xLyWs4cmxjyunK7uz78KkZ4YQRZpIPh1fYy7zSemrnb/WYEoAb0U/2/c2hYtFaVcIXEqEnTR3OLHi6JJkGRgMRUKgCy54NtwjWhGgBNdCPDlsJmOxwj0dLSSOVNBmYW4kWxxJ/5ucfB8v83v/Viu5n3eGJ+Z5hgVEiSUuV5PP4f/D4dP+Eb9m6NDY6vCUkctYa5QZJqUVO2VmEl4teivTAIMH4IsoFTYnweKLmI7zQbOgOrU/sloKmbXM+Oh1hxho8DT3A6nQTQgzGEKxPbFI2F2dokZZKK4QffyoZ24Z+enTt45Ai0DCY7MjJMlQVuhdBfLlbJwcWpJHUiHIoPjaCKUG5he3sH3kxryHPm0TBCWRKxhCtJpI6JGIwtMKdoYYnLCjkUBgAfwdshSIowKyA7ybDa2t4haPnoSXJ8ccbZpyZn8IFduHgVr+ChQ0EFqPrBg4QOByZnZsN+X15gHdXh4URqfx+6SpwX1ulsPo++zCjhe2bqIcXlinfJN0tj1lZv4gHEY4mrhlkv5LJDBBwEApjm7ISxlcuUSJN/m0th7GhCxDtjXt/Y2PK6WQqUOiDivdLzOiMhr9XertYLwiEz0LOE0Yhf4ADuOTGLoyVubW+SmjcxOsbdCFVgUJgJZhfCzHtGk6liOJBmec/ccJJJ0ntMnJL2OSCjUEeFQHMlF/A7/cculeSlM9zGbEqtYw5J0wQRgJpCiq8VkaqMJGRct07oG+Ut0WkBThOfBh1NPAa9reuixg4BUGXVd7S5Q31qgDgjWNECfrRhJSiJ6NB9FiKeCa+PgvfELYXDo6wVnP/4OBCqQIEJBmPIUZTDBJ98aHiMMh0sMbBJiWKjsxB2lim2RbNYld0E10PY1GwQ4UpPrRTTCKuYookeZExAgzS9VjfpnQKvKjWC/Qh8p48IbLFIBFRFeD6DXCkVWNMow5h00I4BaqHHrFEWfZYMM4pkJRJk9wByhfzFUDMd/A8DRn2keDVdnxgdJWWdqacoCG3AGbMwM9UgyhDe1ea2QJEIG7lRAUergUe9ks3UclmVQSUkoFImJqMLVBM2ADyadBuzAeZBpB3YOT4mFGycOQRXWr3MHPPSqPWV1mZjsSLaYpWJ4PGx9cmcRbRmNSDlED2IMIs1kzhsB/oUji+oEXEgMo6hIEDXa5VWub53+U0cwKgJ3E0bH3BdtEyhBBuFcWBoYXVJGGKxaUUZpQQig6gkIkcP2an4rcEzwXrJBXBPpdJKtrdVyCR2dfv+dsNWTdX3yt1idCQ4tpB48ru/z+InpKVg6a3XO7s9T8kaqCG7kjMOhyH1w02EDI4O7sPKpMEkXGARAR0GcmYl8VipiuQNEHePnAeLkhAIfzX6D21kh5hTt1mU2QNIaoNFQUfRKQcUG1MAO0gCn9kX9FHXsCFYdZB8JtD85vZ5cw1XmO/u3EycDg6hCzmPXYf9yIkBw0BSETMGIsnYNYxeC3cV04WscpGHb/FO4CdU9QfUd7FhGy4p5dsrSUn90uDzHzBO7Ek1T4f2P7zAMCn0VM6LSTEIRvoW94I3t6isTqkrnBgUn1EoX7XRBEeUaKlC13ljK1kA85nRVVERb6lI5GL705/8c9LU640y4KTDYzESJidmpuqV8gff++Rn/uKzQKbCOchPdTtsLUruKPSPebYeWpxP7e288vm/uv++h+r1PEXNySh9++2VbG73va7TJ0/ME8B0ePlUaaf9wrPP//C3f//kqOXmJYvzO4A8yFGgmQU3SNQhMgFqh10UF2C53AbXWGPLoqMzt3s4IFhmQLRKTf/1nS5ghmW6F024LXPICictF1cO44hBzIaBtJaXL8kTA0anUs1afERQ1JCOTYwetb8cEHNc3ehLr/7Ixw/CFf79v32tWfSPxeYKtQy/ZqMGgrFeO1AuYppitRMlRApIQfV6PEHIAtIPrBGhAjBqFi9MBTrGbEnNIKyk3YIW5dEmwdsmHkoLTUwXyoM1lPcQOvQ/oq4oVk/UFayIM9wQCpPNpo2po48Om4iHqf0H8yBYqUOyKP61UBBoaDY72kTQ73vk0YeuXb9y/sLqxKh/aCROai3qSsfV2d7Z5XEnjh3HrUBpBRDNdtNk9m5RJZBaC5lsGlfO1u4OjDOZSaOQ+HxZyBpa6MTo8UKBxE7F3pITsbm5jeYORcUPHXb5N7d28RnvYIJu9RYOHIStXrx8ZWVl5d4HHg0nJkPRIerKnTv/NvIcBIeyCpSGO3Lk0BhwVEPgeIXeePstyAm4IouL89VyhSdidWdA6LUi4KoVhD9M5eXiJGewLF67dg2CDMdTbOzIcEJOczKHjeTicXnl57SBLrl349o1v9seCMa3N27u7/cnR4eJzS7msgI6ZIeIpokgECCEhwufIMCeh47fd/jM6Xa9ncwXO5WqdAxkUWRcTZXkYCil1uWA74pmiMLoPrcPlpok98G8siYHK5NXdEsugi1xoaZdq1dUi9hsGD1LTyYHN+htBNS1Oq5eOle22/BSsHSQDvBhQmcIa7JGQ1NsE1mtBYocxCrbQMSqKGiIG1LunjCEJjIGG9viDbbbJWQON5SY/RogA4JBYgk6wfZA7+dfLMa9BtWmUfbgQa12JT4Ck1YDWbLafDAlFFu2ppETsc2o7Tqgd1zRDQSRVOTB5RRKCP/ynlcEDn7LvAx+rhBnOG6ldO36jhgE3mICInD6e+Vl4TKioqYoiyG5kn3Vj0WjsMm9vZ3heIhfsiYwzwKHXOBGRFr1+vBdKLhK79VrKPIYSBFy4yHX6o0L5OkqsbZPEDPEEkJJm/Czw/aJx6qp7VKJ1GA0O69XmFlNppyOSpKmzoYCrlxWELIwblEPx06QGBQeHyi5fw7HsI1h7LotLdzzCp1DiyMsk3WhwpEQM5mzhbIo3o/T10HsmYQsrQWsaRWqkuYw5yBHiG1wlgxw+JicKEaxgs0ZLsPoQdQMW+KFPyUvmngn03BRPdk1Y+G4IkpAmAPzhoAy4rzxu1tqnlHXbnEtldnqBarBcdfSXHj+0JR3MWFxX7Y4WxZbpdnNti35vgOwG2YKU7ZMCOwLA+DF8kSZMesUh4FsJNomrBLM41RoIXRdbnNMEeoEzRB3po8sKrOl4Iy3qTPUm/fYevCMaOS1PYzCdbtr2hM6p496VZ8Gb8zOeuf9t97IeKMLB9yXnw++krdAa09t1kAypohE0mVZnPItiKEy7ywdOYAVHaIoMawJKrpA+q9cNjBo5BAjIdBa2mNGgK7RnUGUkWmeWDrdYIbpNpPE0mAn0yo9Vx+ZF7YLA0KaiTYJEIQMQU8BfFQOyRAO0LaCNeTzxzKN3m6yUKg2iZbBNEI8DQny6SSaVSEQJQvOsb6b/vG/85PRCCl5jomx0WPHDl28dOHA4tLqxjpmQ+5eKxdWVldHhqLnzr1Kvuzpe+696+5HDyyn6vXs+bObI/Exx2SMmZueOLqWufbaq2/Vm5ZDhxg8Fynl/hDR/qwrLUT1wkwEGxbUKYbRdF59MYOqIZcoqBe9vTNvej84uJJDU2/WK1OpxSBSq4WqaEbGC7MU8h7pQ9BpG8TB3A2/LQ80rxS+q1X3S9WCzVr+yX/47XZr4Od/7unTo+OgnnpDEfjEysoWBCQRH89mimD/WxoEWVHK3kEFYSRlN+GOWL96qscnCoBT2OT1EhNqZLPblIp2wlmZQYgSNBCyw2UDKvfOK9fAAjkPeWBl0FQCOYmHgohlsinK4MJm+CGsNxKLU5Se51bLBY8rRO7T2GjiwPL8xvpqOlV1OCiLRJyKD7CqHtXt9/a44dzM7MTCIsGWSwsL+FkX5xfATgYf6fz5syi+CAEopjx9cnKS90OYTEx4Nm1zhYLghrQ6FbvCXDqZfAlBiB2Zzhf2k+kqnhSgDaGuZFiAbRSinGvzjddeu3LpItLJyRNH6cH4+Gghn2NPI8GDWxIfmmV5bm1v7+8DLlSNJ6J4hSd6Y/Dv7Z1NOk5YNe6MXAb8jiFUZDKP19cJml6PhMKo6WQEFzBEKpMapDD5aqmlCnx3g6yYoeHxnc2VVttx4uR9QAvv7mziFxganyUHrFRIF6tFnIt0yekg2cuDaE/rsUR2igUAQwh5oEIk5h2ICodWFitP77QG2d4DZ7DUWc6wtnQOoss1bET2OrSMdWsWnX4Gw5MsqEV5+2bmhjJc8ZnFiRIDkhr0PdTzKMubg5+x1UUwiHsi0qADSmObWQkFQxAXpHKbO4zVDGEEsQ/fEmeoZS7lrmahonCI+pwOV6BP6hHuYZ4DKHkEPQ/ZljLwaM+isS43FZP7PT8cgHFgYplikRE8iIZi0lnc1dyBpUaLgVJCgKVdyO0dUklNvUnA9eg8V6L102JsofyUfBXszNBxL5yTeuztNnjlhBJgjZF2ACFEN8cjZzIHuCSdTDntEfYjTi+mj6Bh5WWRTEbdrjJA5w0oHV2tVSsYYXD+00ueiiUEtRR5i8yfTruKXFqtFJRyJiIv5wJOAVHiHuVCGUHC47E/okDKc6wJo/YDwaD4ANlK2MWsHgBNG5UeNw0Dt2nzEGYFBAvOF7zEyHbsLo970kJRbyFAsePkIGZyIFGyvjJVpD52aryBRuDBxmuruhgNFX1SPBXxaGQF8gxovwQaqA5kQRqw3qs9IhC8mq8GS+42cRQt4wqd04q7fXQtyf0CAEmkTGOPQl61+WwtR6PQybRhyq5sdM46f2p2/GjcEsPtmW60rxEgh5uMP9LzSM7gXoyTfB/Yj8VdEBaYRtax+JWeiUYILBS2MEweMs4j6hFUJkc+rdAAyyrDSheTYqHKXq3doGaafwYN15IYHHf6oM6bg6s0F2rJ7R4O3gz2ibpq7mbeDO5sfqnT+tNaVbMlZYkHm1sxjLBV9QCVDOlCcRQwTcYACFElDrBgBFTTxDuvwCsC4phlXcwS1+PYfNxTnFyMVcyGR99pnxR+BkvTgPCk03w/aM/gEm7DzCNKmoqZlhJVTlVCx9vo27bzldV8dTOHyQ8fv7+Cf4mo2gAGKms2k+2GfbDYcqN4Y2UVIRUYQk8o+urrL//Yj//Yb/7m7911ep5aOmRMZTI58kRx9S0uze6n9rB7OR0EzbaIdn04GCFV78Mf+ZEDizP/7Nd+6emvX7f2qsC9Lc/O7t9KHnzimMNveeDJB8ITU+de3Ai7LfEoplKCBOiRxFKml12DAkSr6LX6fntm1M3/5TEYGNpvBBdJP1IrRAckdOgGGhqlRLNIsNB0yx0CtoNxGVrkL8FFzOYh4NWMpKVCRSOLL8ZLpZp5+uM/8cDS/NInfvw/jE8uV0t1NLCQPwRJqdXS0tUbNZ+lhR0r5PMAQonmC7AaMavItRAtDLm0itbAqEBRZtNzEbA8Sn1EpJS9pNfPgDkPME4XhZg6axhQaS1rB52P3F+4CRXauIzFjwSMuRFWoxiUdBqqi2ID80a4CwcDTDeqUD5ddYc7zl57KBp8z8P3gzVw9sL5bLbi87tBx8PmjTsASF1lQDFMdjvozWhd6O2JWGx8ZPTw4UPo2dlC1mC21lOpFMPD3+5OCtcqoVxUGIK1b+3sMDfEqGJ8zuSKHuyW0FGnLxEfwYeEf3dichIbNXwUVX57v0TEzI2b15YX5onzIsgZMaLdIQrOlc6lw0kwx6LDo8OseoFplMsHlhfR9vn56dOnqATx8ssvk3zF2uAMawNZAbaNpRlDOsOXKxTxcTUQPtGlIPOMF8MNK8V0UCpm73/gka2JsVIhO7cwXauWMukcopjF6QP7pA/TtfGHB1GFqmwOL6x3PjHzxHses/sD6ZX1MDwZUCVMELjPEJ7NShrQCPFRzZ3IABOMwKglxqEVJNb7N5ap9BuxbeMngkgZSVn7lVNciJtMB55BfYPi5QC0BI1EsT96ioBQYZ7ocw3ctQwBPIRuarkrtBhATbAl9Aic3Cy4VhXmjRQPaIe90XaQVtMXhBJ2EsgKxk4rpdF5kFysdmq7ogEqOtAfdMOJgdNA+JRcKMpLH9VKdVBChrrLfsQ0LanARGBhaiQ3FRESVzBGB65nDLieJUULaRvWEjqCwwCnBUJfoyn4SUR15qICIgsQ5KhqvS5SP9ydHxINz/1hnbhaiMnyBQMQIKzACFSU9RDWSJ26KM0ACT2+YK6WJBIZjbxRq8IYpRhjvgGH3eeDsLKCsaVDk9gWCHEMVjAQEGWVGGyaz3wgIBHw1UUFR/uxViFA2B8Jo+7hbvVXssBpoZHgIyPsH9shAY1ENQZt9jgo16TVwKnQ/FCkGAz2O2KfbkfIBmH3oj08C0AoC5URlWhVBFFVaPLo4yQqAmo9sCXIrKylwDgxvFpC+p0Wxp1DC2+w4hhc814kDWaiw0SXu4mipN52k7IGrmbT06xYCmVnMjrrPnR8dvTucctoz1K5WSyvOwM935i/W9rHBcJ/SA3Gq0ycAajarEPkV/wwyFA0hQAkfM/YA/t1IsYlNghLFxLLqsQsr8UhNR1hTMvEsF5aKWGNpSJ2pWWtMTC8i/f08HavdNIcg39Frw0vNuf1Sw7zPQ+5fdw5Y4i7rtB5XUc/pPXqjfjrbTbMTWAlDLY4Lm5woVzBcaF5CjdEVof7AmWjyAeMz6bcr5RgZdVhmuY/7q9Xcd87D2KeJBAjG+nVHBKlqR6tmHmuN3+mJ/ycYUH9rjUsbm5jd5CP4gVlye0vN7o393P7TUu2gXaFcJOBJZO5EY0l8IpgS/V6XUvLC8Ta29zEZ/mgax/77g9yh2tXr4biVlCLwV3FtQQu0g/+4A/8P7/6y+cu3MROsrQUxZo+5LCUq5ZMJhkMjpBi+qef/Nybr91aWpjdXN2IRbw/9RPf7/9Q7/Of/qPA1Mhbq3uvvXXuiftORyP9RuGKUtvZwTSeKl4KWWW2IEZ0VuNMLzXO5r0+a+3p4ncOBmPwnn+4ir5zsQZKk2PiF/QjcURWDm9leGD8SR7yME8wvIH+y6rjD3NUzz3qt1D2gDAGRwUd47EPnPqlf/Tob/3WczNTR5o1vIrETeLbytttHgqKuWpdn5vcRdgrS7qLnQAOh+WNaF5hVyEkm6RbNGDljeBnRjYy/jJIIHPKzoFM8QqvoiOc5D1klmtgOahnUDb4sclLRB6Tgxmiyh4heottjvpTJq+g5GvX8dtDjoj+xB6Vh1FNjQ898fjDKLpYs/GBYze6evV6Pltok1vYbMJ6cczA+DvNKqwHiWUPy3PYPz01kc0l4XmUeONb2gYbRrFk/4HKCRwmDl2At8bGJgiA2t7PZPKFUMfmDfaA3yJ0iVAvfMtkm1CpCQaMg48Arma9PByP+X3u5P725vrK8WNHYOQQWPjx9etX2bOR2PDUzLQAKdBkez3UWygzyvfhw4cZorW1FTKRZqenFxcX77777rNnz166dAmqzn4DOJJMFiyYJpVIuxHFBFMkpTCiJMOgFy0sHsQvmM+mwAEJx0cLG2sE4wGJ6AvFAFFESMVKCcVg1WFdRJJn6NGNxkZG7V4QoTvVepV2a5gI15INi/XIJxrJZtWGNEtusPxYfPAtWCPnWIQDWkkjuUzhG5w3yrHWKG/5wEcug6Dd/r2WLmsRHoZsDVEX/TB7GSqPKqgkVWK8qQfJUoOGS4Rr4a0pg55KIhiOVUYNdgWf8BK24I3DqZOpdGw0xBhJIlfFBVDTTKWERhvHcAd2LYlUkit0CsuNEsbVMNQz2ZwRCNQ2o6CwgtmRVCfQPsX52ERmUu0RrBaU+8WywU0wk9BhAi4QDFm73MdEazGJopFMjd3mh/tiNOZ5/AajJ9Ap0mKbdXgwvMgFwEwUIHeKXGWwWoo3s4XCdAc3NjikqhTFXaBVLE0ikilMbWrFwchkG0WTIcKSZQTtoG1Y4QfEhPFsCu0eMyuObRN6pqZqAkiHMpb4ACor+lAd3FZKo1t7IX8CDByXg6iHkJdsWGzLmKWx3hJSbYi/tiP9EugFvYGsA7xuI9MR/Rq/YpsqEVWE1ipyK09hrJDTZd1lIMhKbimVi0NsjFawCkTEtBKYKsOq+Po2u9V1Os+S4IxOSjdjsetn4Np78zjTPF3iS6vddLa9k1gIPHzPwcDDi5bGWq95HWw6e7AdjrpJOOlU95kT8RU9lcWK7wN2jLGdM+Y9z9EAYuQqN9slsF8b2QYlFcwMSvCEQkkcQPaQt0+NpgM0nvaYRsKm+Nd8UFf1VqxXvVKHzSkexHq4LWbQB20Ic+hKHawl3unVfORXty8wX2kbasVC3Q3HHbyRxiV91/BNOiEnCMQeA79drgBYrz4q5ErR2qplR6i7ALCUB6wERdmrJWXyTN1ZN6fFtGHQNTWHbpiuDILmuR8ThqzF/NE+/dB8q0mCYNucdUKeicjzBfqubsNF7UbrVr2wU8d3wVpSfJ/N7ScjHjtSsZIHgxc4eiDenn72axa3vQZUe8Ozc+lSP+C6dO3qB9/3gYMnj2F7mJmavXzh4nYyNTI1cfq+M2+ffe2jH/v2aq1w7ebFzKYFWNhnX/jyx3/07+3trH3yTz75Xd/5PQeWDv7sP/j7Ew8t//pv/8HP/+xPnt3atrXLT72Q/LVf/KhtZu5rz3/uzDghLh02C6GpDCw8mO7jfBBtMpvHdFq9+9ZyZPrMtNF7zT4Dc+dgHPig+dKPZRo0mgQUlhuSUQKBY7j4ma1RbDnYShJo0Fhh99gIaUDX7XN0M1W7z1st1P0RhvTadvL6j/3M9y0fWPzhH/jPY2OxYHDo7IXroDQSKVhMZiIel9A1kIdVPaHn9xLvqNbAR0WoaYzx+lERgXsRcyrXr122ZQgUZIrLoGAYGwaLDaI3oC2cQYuDKBGMQn+kP9TK7HOcsKw8NGwhKOClwgRe61JZgthS6BlEs9cRhkSzXqRPoaD37jMnlg8s8CwCr77yla9+5tPPZPK1sWHIeW9nd5frA24v+gW/rLTqaMZLS0tAS7725huXLl9GYsS4SxYTZnVk7J3t3bWtrbvuPgOq5MJBWKPFdvG6Pxxzuv0EELibXUcNvaUCIgctTxBrHYlguGR2aTrIl6RFUSQCFC3cuviO4SfBUICYalI5nbXa7MKCj0oBNtvarZtE4WBOu/X8c0C+LC0tnDlzhqwqxkr6EnZImx1WTYYSNAHgEEcmtc8oKz8S+A9CrjD4SZikYIObALRKpY7a2Or0UdWptBNNjOFqhlDarBjLy51mqQf6EpeLL9mvXLlSqv35PQ++5+Dp+4W5pwBiHN1QKrRQclDEdCF5orug/UpZkBD87uMdkqF1aZYi3/JzPhoBmuUmiUwrkAukgplFb8jqYIWzZkXgZIVENmDBKgJM3imqWyAPO3rghtfrZRfBwU4XqjJTi7eDQSml9wSUo0Ih4CREgzh4qTTva4CQpsAY4pbc1OXyCrZU6rKwRwgoVI/YGbxoPkVemWnuyVZhYyD9iQDp4bRYmdAozmwr6Bi9EHO1UNeWgxtCt9VTKKBUJUUeymSNQCMPZZlaOFJgIYFuSiTCg9ttvDWi+nQSEwzPrdWZYyRMAu2YQIKSMeESOVUqEuyFFssYKok2EiLijCklJA8fHmZE+XpNlQgVB61160rq0g4h6AkzAESWrccNUDipSyEhGWkGswIsEF2f/d+qIF8C8cxAYzAgDpnsGlLC/PHwtANDvgMBDDEbqz1gHcZAx1gyd+SnSf2V9AZHQ3Ynno18ii6mLor6NssqE0ihJPwfRHQZUo511jBNrR7mwNgVRLIZVJrL2A3ImO4qo7AoGItDr/qGoWZ8xR8YZDMlTAFX9kvNkiMMZmktXd/vB2tzpyJLZyZti15L+RVLqG1LQHeQ2wrtcg0hifIbxG3RHB6oRov70gUssyw6VGKZaPHRNZrFarNABgRxSa46z5T9UAtWa5bGiK4NPrIDROJQ/BlKSaDck66oX6wsrXPWmLon9mX2gl75hflk3nPp3zwGI/HuV/Pz27/gSYODFcigIC3wys0Zm3d/ZL1IA+Zik1AE/DlLUeA1wtwYsF66iyzGEmU3G5Ax+qAxHjAaPe7dbWCV66O2jNkP5lsGg1UuF7gGQX1kOHkn2YQiByRrkjqNyuK0g/FGyOtqoVJzWgicJ3ye/RwCt9zuKBSA0S9GY5Bov7XcK9UrbLC2s+/3x+89vPjsq8996MPf/rUXnsVvcvjAUdJIsKgsLy385ec/Nzk3s3x09gd+5Hv20hu//W8uhYdwk9pGpyajCfvqtRuZjTQIwzPTi3fd+/CRk8deeeOZX/7n/9QfBNA98MT/sbzw2BPXLq8tPfzB5vrnlFCqkHcBxBruCMYZi0ETyuQwHKxDujZYjYMxefeciZSZ0TAjYOiXaJkZQE2L1i9/TDmLB9au+qRm8zJj4PTzGI0Z0da6S8/ddjXKLRzigagtn61Gx1KTs2Pp5Oce+ra/9bkv/tTHf/T3W/nCvXdPXzi/OTpioRIpiHqIyRTNRqzVrumCSguqq8p+Q8eMxQ7apuXJeoDv4gnkWwzR6ANwWbFGw2h5g1ZHF7iGg7ZAlCBT6JGsay1iVg/LiDB6JfRBbJrYyrgJq5BA573dbbQI8nqJZGPdkRWsUJZeJxJ1BnxuaByBx/fee8/VS5cvXkhu7W7JLuj2ghENFYZiwmoJ6kK7iA3FJqbGqU+8fOAAcBqiulQBd/ugYteuXCc96bmXXgb2cXp+CTZXabSAmSRygPcnT57Gu9xOplDg0bmxq4+NjYGLAb8v5HLAbe1s5g3gRD+TTbJUWRsTU/PEghGXkMsXdvb2WPhCYvD7xqcmy4UijBw1nVSlEydOkABN8CyaMWdobzAcJdiF8jqUi3BggWSb+33kd/oQSXEfMmoINZlMnfgsmAYhL8SGkScHvzt4+LASVfs4E3PFHNcVxJzlQ2bHWIHehtLhfLbYfdhuSa4CFJHai2xd5gzPASsNms7ccDGv5o2oz50/lhDf8BV/2pC8mot0d5agDM4ab+5yx+oiWqZ1zoyySvWv+TmsEIXOweVgYWMWVXoI7AW8RiaiV67kKWAPYDJrF+OGzxlQQhjJ7WjMEJMK6dpZpyMSIu82Eq01csAmUJbV6W75XB3wqUSJCMplz4EDTOqAHDZqhXag0wlXRf1F96X9LGKJC6w9AMUUBsaXhCEr9ID3uEVwxPqCPji1GQola0nSMLhusGqzAZzwZGZEnJkhEIwtNSXzCJ5owByYkPEo8HMU+nY7wIU0hadTk8rvodCATZouYidQX+1m0EugXAjhhXwlNg9oLeB+oEdzQ7g7fBXA6KibsDIRaTEF4sTYmMT5w7C7giTFmStyyR9cAYWp1R+KzlLBNLtfIzTe5QxEg8Mj41Ox8KgD0kohCcgFJbOl9aInwPpRFLg5G5IuQ3blC5AwhQUXGRnRFy9QNUcvKXeBNULWf6UzofXiP+X3oE4r7UECDpOtETEricXC58GBZgSLNYwO+i7TARcypsyRiKH5lhPyPhJg1LIFmg0XSPNJS7h29FGinA9avPnC1ov+YXsb51+JHqs0BSXLWCwo911bhKEByMRmbQEYJLoIWBH+T8rIIWFQOoEMSCAD8QVi4MYwbdwkdNe0Q8NKWyUWsjIgnIqBZtXy5W0NXax3cIo3Iru88GNIqzmtCw1lN51lEgbCgDYF99ZPtD14b/rLv3qQnmCGy7zqSk5ymiEx3FfuGA0QCxpKKyosNorLDfc6SWXUrJJpVewW4wcMWF+hBMN9zSsUkB/AfWWmNto9TzCHaY+e/62nm5YwO5w1XAqxkFjiQZfUCSYeTye9on6v3ZquUhe13Ha6qh2Qgev5tqXssCGg1Vt9QjFYVmQDoI0TOYrLKVvIcIdYIuaOBbYyya7D+ugH3nv4geNb+7s/8pMf/8ZTT5+47+6b126SMperVLzhoMPZyZSTz7/y3N0PHB0aj9h8JJLivcmub762sZl0xi1//dTn52YXfuhHfuj66hVH2L0wfdgXoSZk6aGPfecr2+vQxVsrVz46HnGUU2DwMAQyEtNVljS7xGwifRp03yxQSNUd+Udj8u7D7IjBvIhmcJPBEEGv2HOwKS1xeSckjrFzAEFmKeMIwB7BZOEChooAXFsvtv2JRG4jE5t1R8dDxUzJH97ExV0pff7Eg/e99Nq/+sSP/tY3v7G5vDxNiNBoLOQlfINAKzRoNrtWBRuNqpckTWBqhoiJicJ62aJ0ReNTpFATQUI1tuFA8WVvIfrDPtRgA3rPR95D87lsdm6OVkPKOAkwMKV+4TvYVaCWdEd/IMw3apnkvkLTg36yI/BdeqPKUUaYrVaKuA1YeqSckEn8oQ99yGb92itnd9rtSyePnSCsqVUsjwzFQAxBR8jnSOR1ZnJpVCaMzBQ6dLqRJyw+N78OHT16nIzL//hH/+XipSvPPf8yGg5RDlFcG3Y70NGLS8uYqDGzM/DFIrlb2DutuMwxGINvGfDNQ29HpsfR4YjpYoiy+SJhL9gDKFKHjnTjxg2gLgM+/2gijhUEWWx2YR5b3er6WigQVHoSOayBAJQZ2rRFoPbuHgPLTxyWLjjAIG2Tdt2g4h4YBmIuTuLQhpEIGDhKRpBbeejIKSyg165epsCO5gU8OLIRsCFQBYf920d1KcaGRlLp6o1r51EGPP5Ib3aiS9WcZoUJ5Q8WwpxBPOkwrYGNmfXHkkVbNK/QHvYkBI/ba7lxGM5kaJIhMRBOxU5ppgff6wpGmAXJmmX6+WOl6Fei0mhsKnWJskYRIfgTA0JAMr5elPYOCj/2W4OCVUpldrBFM/7+ADhWDvRzhA4CKplUmBy7i3RNlmBHEYJdoFTdhFETeSIpAfu01myjj2tZ+OkwV3jEwEBKO0XqmVIwlhCfGuAAd/3EDBKB7wR5SvDNfmvYJBOTryaTNW0eYNDApxmsgF8+Y7sVlBmeRuBpY2vtit1rdwKGFiEJ341Itb+XQiClMAEoKsMJPC8jLAsxYXRSyH/U3+16CrZetSTTHZsAmMliNkfZMSRbcG2gdNzWrgqeFFJAWoKzYKsjukZaMiPI3CgPmBHE4ce4UTnCzurEDgVqsW03XbUKc2U0QVB+bMzrCZNKZFP0jE9BXIhnuNcQafHsEnjmJFa8iuLODEFxVLtR7Jzp6m/dugnhh967sLph0SIWAapPBeVGyU2qMp4RTPOI/PABorBaOK58knrMLkYPFVnS/seqyUoRJWG8YBN8P2AN2s4wCC1XClmaP2ADHOV0c9XhbS+dmV68/4Rlwm5pXO62cpEZd89WMWFSip7HTGrWJERDhk/JDJJOaDZP5cY8p9UolnpN/kHOEUGEvFDWSFieKkopskvrtDpZvwyEPCAmKtssY6k70KJBD3Rbade6Wj0yV0iC0Bf0S1+YW/GRU0Z447aGXuun7xzaBWbX6A7mh1w0uCv/aPHevhmPv8N9aSjSC6MIK4b7otKDkcIrpMoou+K4g/f6aC7QNWYMBmwbtcYIz4Nmq7FqLevetF/i6mAo9Y/EKMIzTPPYtbTO1gL41uokAwb3Y9Vmu0XRmGSv6yQdUjWi7EEKiBKiYAEMPZqIk/ewm98rl5vAw1DCFfk3EAn5w8HY6Ch+iwo4wArj8m5v7dYqzxJbe/b8G1C9Wq88vXT02F0HefYXv3Q+X4pvb2+MjFIyaHVqAhR+TyG/dvPm3mOPH9pcKT71zGcfe/K9cweHv3k2ffjE0R5VnLuhtqW5cHg5n7/8wY9+2HPrM/aer9MHDLOHyIbUxbizUfE0AzPewR2NGcwGUWVf1SW0MiDqMOtf/iktfk22GScI6Z1pZOLINmIVU/gG9gsHhcYxN5jNjISGEO8C0QqiosRsFg8ATyDSg4MLPGuxFxsJVPYrrlAzHLFkCz1K4pYr29XyK+DW/MZvfu+v/Px/efWlzQOHRpNr+85YiBANVijgtniDYBM8BNKkBghtSb4HyX6Skwn8xoqm3CSK8rI7sMHhKoZYwKDaTSapbcxmoCL60AOI+MTlh9qHsAtGEws1EQnz204bzseOwoPLfYEZkJmQBCfOuogc8lJWwQ7HYaPRGkJN0RYIOiJ1Z3ZmHiU4Go5kUn9QKTYIO8XmGvV5R8cPFZP7xMUEomFGhwxd1CoEF7hp3OUl6mU/TxXeKt6HY8dPovv+wi/98urqKvjRAFjix4XwjY6P3Lx1Axni8OGDJue4hOYKcSa9ikwj2HAlnyVKZXZ8zOf3jIwMpTJJdFQOYr6YcOyIoG2T2AmFL2YFksW4MV7coUYxR5OgZZY76x2RsTM5ia46ns3m3njzTWwPFaocQSP8FIv12kpy9tVhsZmkCuxg12bJTM8fICCMqg6+8NBwIp7a22DLAsREjoywK4MBwn9cAXsuv8MmdFn7Z1//BlJ+aX/95JkzUwtz1HHK7KVhREvLh4H42k9mqAABqVWMjuAJe+h6SBNMAxIWsbvQKBYlLTYmsAHtQHXW1sVtgGYIczF8QdRM65g1SIEdrWfuoawA1gsWJxF0NC478Kc1jAn0EeoLWBGriA3dKuZALvT7nJQks1hwhcxWa+VoMIog63ITlxcD9hPBjgC1armGKwB4iGpxF7bIHFTyeTgoE4ylFgQubLOYaVEQOMmy41VsmH6g+uJclYWACEnCRWgaMIatbg2BmVK4Vr83AJsDagZoJwz++AFAaeGHVKsDfAkZAiJH2WDqM4idKEyqYG3T7K14fGinuI8kASYFMf2pdJoNsxhbhOKXiplOPY9hA+QNWTXsIInnIHZT45PRaKiBU7VWi6DmdqmebQcQlUBpwKVQ6VC8iWSkOgraJqIKvAPqQGwUxgGCPgLusNsOQp+v33b3ahR8pahYv9uwAawbDbNKhxEJmB3SzUX02assEGiqFCLNCjdEDmL2GS8ynBkx2D8FHRqlUpU4fIosCfKU8YEYDegR/AECbQu4fHwF+YfxCipSIVoUkKZb6F9YswxJgxGzto1/At7qV4GzNhH48EpAS5AjEE9cAQ+A07U+lbuLDVvJ4m35gYiP1Z58/IgjXIe9Wxw7eLQ7njqdMDhMaPncAG7JysUahoNM1gkCViT1aSuRFtXuUDWhQqEIEqSkReJ3h5yyLGiLMpKRB0CIgSMZUcb8BnKJzcxwX+jPbYorZmkGjt7Dr3gQQyatVNo8qqWRMfVzw9d4HfxLS6BiGi+GQatfQ61541Ujwj31/javRV7h9rqpkUpoFtdyewVYsVdQ2iTjGT4K5WXGWCOK4oYTm/ecgQrReEWYkU+n64nPMwI47JXeMtc0hEcahqJm0U491cjEatXt9mgva6LVZxxBWJK8XYsTGOyO29/2BooW6439VIrSlfVeiyg3ekDdDmwwZVNOymqLeAKZ7dQDj7ynVGylk9tlB4U75ZNvluspMpCajcXlBYpQJrzh08unv/9j3/8v/sWvr2zcXHjgzPWVqxZ39ZtvfKPnyoZh1jHvXmorvGW9dfnqgekZ+g+a0DeuvhIIAfG491M/8xNPP/N8pnrlzbMvLyyhwWVHRyOWjufYdKLb7N1zcMYNjqyjn6+UxzGethpeAu9aFs+oJZ+2pNrdAw8+8Mob6y5vhGyffift7O3EA+YCKm6UiC9zEqOa3s8mRoMUvGO74Dgxsc4aPFDTLZ0hTtkd2V6fwgZaRTA/+LAopKJD2B6EOBJ6othKZgoBxVWjvJdbIBvdksctQyGTReGDSqka9Ibr5V1L45mJQ8u/+3vf/uu/9tm//Mz+e+45mbuVYu9m82uEjph8h0KlxMxgf/LAKpqtKjqlQliblJyhOlCgXicAHWxFgnzIG6Y+g4u0veT+VtHpoSaC2w9KLriqEHLJBQR7UkACUYJQUwVCA1iqvdylNBAmUQzYrG56DRtgg0EcgiQbxcdx9pWb3empWRTft8+fgzJKjWGYK5XxEWpLzhw7dPAr3zh3Osz4WLOV0nZu3xvyJkLeUDSC8xW2hcJLYjO7BOghHkQ8FEiU4MTjx7X2fb/407/4qT/75Asvv0D02cbezVg8fuPGOdK+QUxDiEKMAMbyOz788BuvvZlNrQ8lAkszRwgVSxw7gMF9c2vb2qyPTZHRWyBsGdQtJgaiTYrH7OQET4H0Y3zGoUbCCSzB7fOnsjnWP2HOCH+qBtxqT42NwoHZLyePHiRL1wLOB2Tk4oWzGYSlPlWUFw4einqD6MuUq7ETOnj9xk3je7dhvAZ/ZCyRKJWAuswDVI27bi+dBoMYPYBdineA/UpKF+BKIG9vrV5ZWbvMuJCwhRzDThobn6bkDrFB6NnofMja0GHs1XI9Qm+tJIppedFi/pem8a3NjORnSDTbGcKo79nuUHdoMToV5IYlKiOg+DTIlDgdoQoysskmTHAfS9qFYNVxUX0RWz3SLnp8sZAu5CmSBQciEp3APFKJuhXckA25KiSvtbq4SWLRsNNly6QUG0RNIgXiEzplIXpeZidomhxmBKvY+lyMAadQqZJ0A5KKytfjY61VYMnQKsgrVM1QdoqxS7NDAaCL8HXQxNgwRJ8wIJ12rW3F0sfmajbK8KcaP2M3NKq5XiNTze2DCdVo97L5Ctm3Hl/w2PEj+VwR0YwBzab2Sv1OOOQDCyO5tUNEAGCdjENqa7Wwq1hx2kBn8ZpgviPnBwYMmAacZIDJ3CUoHpXX5iQ+C58KZuqA2x8LR+G1wFS1qzYMzvh93c7wSHxqaHY8EBzCRgu5Z7ljn2XyiLdi5UAgEDIw/mPF5mnKEiM2DOEJRt8C4KRMKBkV63utOlj+bgwzQnuXJkcLDUsRuTYEnNOGK5OuJEUXAs8VwEE0WUL4uSHvaGiwZ0QdTQTSAmBeXnskHgFJO1fLoZ/aw85kbTvXyjVd1dH56LEzBxKHxywRDOQpi3Ol6yxJzkddQq1jkdEZwMlkRYRWIP5LxWOegU8TSSMCjj2Hogu+CsUuYL140jE6DYzzWnxol1rKSIe0l6UKm9Oa1Uo2B93S+8Ha5szfeKPSBWL1XKOJ4EJz7e2f/o1/eA63uj1our+m9c5TjE1AN7l9B9qjVoi2y7/L9PCVRAqTy2t0qAHrlRkCNqwaR6xoRDCGmiXJ/jZXmkIVeg/pRGsiFkozradr/9Fm9U4Petch8UJ7Vq+ySg++MiICsVYojdgQUBQl0gLpVuk5U63mXrMLk2oSwslzkDSRCCANyNBNy8Tc+PrGNgn3b795tqRcz4mtVGppefnRJx95+dWXC5WM3++7fPkyfrtP/vGffMf3fC9BFCNxVJzL165fypdT5YplfMr6+ttnjxyauefMXZ/8k/9+bT546viBW9cuA9K0emMDTM2piUgyU/3Lz33yiQ+8z+MFbrp1+q7F9fUrnUb0kfsfBW/uxpWtI0v3EJR349bOjDdOGl2luRcLOCzOzu6KJTprWZg48Qef/MZb1yyPvff0seXJSq4S8wX3ynmqy4VclmAkUclRX5eavpCLslY6dj0zmyxxjRfTYCQ3CBs0zijKGjbGWUPN9zBrbWRWJKKeLBPWKjFxTU98TOsGEVqJPUQBQlC8PhfVAKtAUVkChVrqmXDsyK/+0+9yuJ/9wqfOPXzoEAt9bDy+vbNKSlU8MfLmzRUqJ7CMTdlp0U+ego2LLUxEM6RPMhqrGqcxq56pI9/fTlU6oqzwvyP3U74FBZ1YV2B+BWZH6jL2Xqik8PEUwolQv4/MrhWBu0FiaxsoXmVtDA0TzVRFmABHogrkiDMSGe100usbt3BQJWqxIFVuOz2AddG2AGTe22/6Yx4bJWWBFxgedgd8WATxQA+80SBnwQLRQ+hCLA5GtFgj8hPK1OGDh6q1IrAkyeQOtttq3T3iGAUIgWVz6dK1men5a1dvvPnWuVw21WrkfG7LCBlFCB+UU6zXsvsdQqynJiYl2lgs48MjBBZA2gMeL4lzhXJlL5nc3d8nlFemeKyMwSC0Bcgu2C+WA9K9IPXteoXdOpKIOjCDZnN4jHNgY5KgCC+AWwj/q1IlUHsoFp0YHyX5mTwq9lk6uTcUDmGbQFfzhaKdTmV3N1Ou5qnoju/Y7QsN+2OyuSma2o1UW22UU4X9RncIKtFodHKZHeBOAr4A+U8ANfMwqbTYKrUrWX6KwVYsJgRIhFhGkMFq0yvTZQ4tw//hYHnqeggAFBg9icWj+h2IXQw9nWfREEolWgSpsPYAAyN8g8K6GBAwsaMmYoAvFgqwDS10fHqgXrQ7RN0TmIfNmr4bazx0ooVGzVNYSciGuhfKiZXsX1pMKL8fLGcAKwhnwxaE6sdaxCJEuxEdEeKw3wqCgSpG+DKVrSP0EA+rVAZBHF5YAuBS+DpJHa0Dxkw2FXH0pOCQ7s6iJ9ivXsn1O2UehpiXwzfSaDHm42OTBxYXkvukkmdIc2Y7sRlS1SIsjfg9UmjlRwCFpNkkx89PeIZDfuVGtcpYca10FlWoleWbSCuyPWDHyGpQAXs/gJqOqYL6380qBueapY35PDIyCTruSMAbYxX4MPKg1CtlichksV4BP2FaapHzjMREoIdMjajmWAQwNrXKRYKaKLcHQAyojdi+wEMwG1g6nLg43mYVJpf1gOO2W02chH6L9cLTUAiJb+Ke5npZQBDg9FOqQjoFrVltVbZT2x1nyxd1ta31VGXXG3fffWpx9P4DlmkSqfct5fV+MweClc2P6ET34TbQEUQ1Fp3WALl2rEreDmQ9MRUIItSCiEksEVVqYBDzDuOQMiwBkKUMoeLqgWhoWsuKM2vWoMeYRc6tzaGVqu4OPmjVaXXyoi04OMkrPzGvOjEgz3rHoSZqsPQCGdbu0Jl3HeZuLFWNIa3gT3tJCrXkQJimLIu8DNgqlFEGesmRfIvKTUgBPiYoOHBhbBo8jQO/r14l70B+TYN1n8FT2Ax6PNuPgRscRve93QmZpjk0Y/pX0ymRAFqu6ev2KZDs6ru9PCdbquxTFJN8U4BYQUAEootdrNq4Sl7A0XNrdXtsYog0+Z29zBMfeC+UanxhNhKNMiq1Vi0PQJXLGo1HtnY2sMDdf//9mSJkOTs1NXXo2OKtzRv4aObmp7nRrevXsqlsPOpcXSmPJlIeXwhrGeZttNWdrcLo5FQ6k0umtuYXxnb2Vs84ZmAW09MTkObTR0Y++uAPphvNYGR47MkfyN96bfPKdW8z4LR0o35PaK5ftLvX1tOX9y2HHlg68uDDtl7l1Re/sDxpOboQJdAhQ1U5WwRLjNfWSITcqVQtFNbiYlnfnnzkGNxg/Sy+XUy5cBHE28GqxKQk2YddwcCzUbTu0EC0thBN+GuQL4dFL2axDyMuMgPA1yc8wSG7s4AuW0xfAzzL4s2gEP/Cr7zfa/3yhW9ePThzpJArkcMD/b96beXA4ZmbVzeoEEjUKmoRc40N3O0jQgWCRC4nOOcsElWJIaeTXQ+9Yl2xbLC4IgjjC4PDCTuaqYLgESrfa4sXKR8DTEUHVJRAYvRksBFEmjDBdKhp5sRyOzk1QVSKOyTbe3IfDpvnhkKasHv2cttwVi8oSbVbgGwEKMiIYkNQcjOPqMKKCsIFu1RWrZjlJQWANY9PEACsSrk2MRGFRq2traJvEzKMYezkyRMVar5Zetiux0Ynjx49iQwIGNb65rbTFfjq175+4/rN4UQUZSSVzYCdAGWj6F58aCgDqFc2OxEOlxsNtEaYPYUSdnf38x0WnhteiS2avk+OLzA42IZDfh8cowjuZKlANS12ANRvp5inO3BDRzA0lM2tg4c9PjEN3iFEHYmK3yN4oQKhsw+NDIOVgFJL+FVuv4ugAdjbzOgi/OTq5XP+cILMl0K5BGy21xuOEt9F9Q0gOTryL4ai/qEOgdpYtdtuFwHm6WJhX4E/8toxebhzjd7pdBFqIFpnGKkhgRK7RHUGUSpabnw3OGRbMzRI8j79IcgK/kEYj3huE5xIGACYVkEYsOgOySuYy3kmpWR71ly+GUvEMfQD9VXIZ/34fQEwFJtu4XqkWSTFoslRYRdxG2mNWC1ENxgenhEkF1wvnOe5WDTYBkJlNbZvrqVxTBJNkaaHdUVFbjDeofzpQIOqC7NASXwiQZg1G13wtsl7J6jNrN02FgH4MKG/uGkJiGKEW0QF16qidqirVawaRXCssEaQF0uI+sjI+MTMbDAcqRQQ0xwbuQwJp7FImIrNhVxKgBXtNko1jH9g82RGADauNIkzUzlnEu/JjUPvkETBNaolSBaUs4Pc0kaGQI5AnuiSb8zFfmfQZfdHIsMjQxPD0QlKeoDKQs148j0YbeQU4TmiAMPVzQGZYL1Sc4b7Y4BDgAUfhIIO3WIR5w9dYyPSV7E6aTdMlLxbKI54CfjEwYBCrM0q0GWGckOy4Y9MANYCp3gJJg8JVVhqTMakvVtqN8lJ8UVcw4dH8vX9W3tXfXHHgcdmZz/8KLZHizVnKV2u17fatoKX8rFhF74bbgPzZNIUnCB+wS2hh4prVXwcbn7YM3F8oMazLAAnlXyC4wsqhLmGiCoEJ2iNkQK5kTiMJLlBLwZaKmuDj2bdGr7F/JuP/8NJGJRhzIML9crtBsedX9/+ePskjYVf8uFv3s3sjttXMrLmTyxfKi+CBNKK7OX8GY4LQ22jXPIevqsIZ0aUC1RWAckSyYSg4oHHV8tQyhbXiE/zYN1HzJT3tJRXGDDvBqzXsFy6YDp+uyc6JxsB864zTJ+mFp+8LN3WbgnofMrEEPJPA2BCGmIMr3JaICMQg8fP0a0y2TxrKxDyXrl+9cCRI4Qinb9y4bNf+dLcwpjH5ykDDuP007RkJvXnf/5pkGDQw15/6+rMwuQ9p+/71Kf/BLpPUuaHPvid+UySrKSSBazWLvSxBOxomqTEwMT4OMVlKboOSwSuiId+6YvPfuD99xMIMT4+vbuTfvap37v45nq/VlwYajxwcHw0tJwYd/bLyUtbN2YOza6ms89c3D395MnRuTOf+drTrdKeG5naatstt8OhGdSTTD1AHU/c3EQh9N1EzJDOp4EBDV0eBzOYiBM6xdLRiDG4GjDEMKZRiRQsMSwvnEVV5SJWkuRDi9vP3dyOqLUHeEXVcmu3lMvthQKNu08uFfZXE9GonRzg8jZoG3D9X/1XP/Tbv/CnL3398swEFcQD+QwOKWepXPEFMTy0IJ1CQudUH3MfKxN6JuhKDEeQRIyp7HV4jLQaTkFdFcIMEZUpD9O4di7J3DV60QPVCGUGykZQC30hio5sFI8PLQ2ajzarohDQSJTU/eR2ZBj4qgRrEgaMF5ltRqgNAZ7oHxhZIZWARc/N4Dz0s1pLjUo4Fka15UHk5kLZUKj4CXsU2/XO3i4YkNAVTJ60ifbPTE6cPX/BVXWhQ0KWSdUl7Glqbm58fBLt/fjxM5cvr9y4tUJZXgXZeNwMbpJMYviCyz0SS5AyQ8u5FWYVklUcXu7ZrVXyqyvrTYAKiO1AImm1w/HQ/NwMLJYyiCASAh9p6r1qmgA+gtgivmTTaYpBOkbGpz3+MCy9LjhBxphSzCI85XJVVMNmBQZiZGiIziD37/k8r7z2MjyY9FJ66A3ET80twrFWVm6OU/kLVGhoareN6MJYOP3esclRWnv9xi5hc+QK72fS3AS77uTMcoUiJpLHleci0iv5T4I58YwSo+8cnIcoiU5p/XGZjnfe8BVLEr6BSRMnQkd6HVMAjm87EonRSZQrVbwUTgZ1FJh6GdcYDlzfw0NRl31mf39zY32dVsM7aT+MAnnCC+JmqcKCBnzK6WiQFQ2oMvMLojdiDHIfEwDh4OZMtg8cFfRLxAkJXHUmVRpSXZ5gsCe4K59ILSdnrFWvkLXODoJD01XJb3VcCHWkB8OACduW7CFe2KyhgTGTLHNWnMRjZfGSHVsn6oHLS8USkQ7T4+PxWBSEkd31VeTHsJfA0V4ZkLJsSiJxpwXI7Xh0hKQBVh7tQWsDilJOXYKogSz2+p1UlADDGYWGBG7Chii+UqJEpR//JWy4AJB2vYO1OYZtIz4eCw9HgglirJr1DpUeSMwKIuVUIbkIy3og7J+ZYsKoG+UP+bWvGESFTFZK5RxLAo4dwslBdQdMBOA68w4iDzvQJhOxYab51TuiF3djSAzl0f3Z5lwP5eYNdUsgDWIX/A4JSiV8NTS+iRBbOpXfvnpzMzTmufsjhyZOz1oSmBiu9HBfWYsuf8M7SoaZp9dIV2oVjyuIGkHAGzdmMfIf4gj7DeAlPREbLFQOelETmAnjhktfT0RaZhWiLmsxoo5oSZr/CVkbrFIYHoR0wGa407cOfs4HcW9z8P7db+580jkRXFFe3ZBHDC5796sYn3aGvh686ONtNYrr2e/mNI9DSjEsU0vJ/A2YqF4NN2UPwnTZpPBgsx4QQgwDxsaDDVFar/ju4FfMgRiwfLi8mlbKLCHWywPh8lg/GEOpbfJs8oVRi9UH0yC2oeQU0nXUOadXucXMNJBolXaNFDTeEFwh+wyVLfCWIkHja2ChYBohPcblxG2EcZpYoI2tbejMfQ8/eOzMKZfP8aWvfulf/tavf9/3feyBE/cd6i49/+LLn/zkJxMjw4RdjIzHI8HIH//XP17byD755Pueffq55y0v/PTf/cn/8yf+/u/+7m/hcqLUHnuoUk2trpW//dvuXt/eIVbuzTcurG2sLMweKpX2J0cXkSK3NjJ/+kd/uHbNMhT2RXyBqxdS99/9eGA49s3nv3ZifrQzbL9Vt/ZHiZQdHjp02hYYsUdj9WoqFou6Q3Z3NP7ylZ3Txx5FPqD2p6W/l81s+V2WTNnik+xJRAvQIwRUATauVc0CYJhZUmYeRQthdtpgZqQ5qfQE2aiN14dvbJZspk/6NBl82X7j2r7l7HVLJteOhlL5bPB9T5xOZV4fG4v4ot381lZ0aqK5/9b/+bNPVKqf3bhRBAExFJ2qVlLJTAVcDoVlWRskMrArWFHsCYU8qMARITiaUCiGkhjFHTWn+giun7FRsAQgM9BIrTlWowJB9BspzW0p0HhqUcPY72gvsAwZp7tgLFcIYC6A/uH1J+KjBw8uU1Dn2tVVoKmIhC8WKuVKbma8imkXHAyf19soI9K4D504hDdaqTcWS7Uq7GF4ezab5yRmS0BG4QVIyNEokXmj8FSH24LPG9qsEg5W++TwPGy+WEC52RweGz9x+u6//to3YXbLhw6RSpTN7e/spjGJ4q2vo1WafBP6Av2H5cPVcW47t7ehnEaWtZRy2fHpCVx3cArye7mM7oNGns2kAct0RcNMMLoOEUaueBRjAEzBQUC/L5KoNS1byVvo71ATZ71Tqd0CkJrBhXai78NxkU388cQjjz586eo16u1gpYbcLBw4qh1psbzn/R/d31grFLJViu0QdtzvZ4r5vcz+TnJlOObKZZPEIgKlRKPXb10L+SIYJLxuXqHRdkIVsbmS9kek+js0hmaYeRVhY3pF2rR1+ZfzvDdLgAfrwD9HgiaBL7xHUkOydKMS43WA5TBYTD1yuermUlmqXAV8lBxZHKrseQz7OC/WVkqb6xuKfhKIK+Z9P/Z+7oUJjiRuNFdFVIH87HIJH7WHCicVTyBTLhzGOABx4QyiuLUW4Z4Ip7QH3chCWi0JRJRX6iMElUDMUPYb4ipwlf0OTKlazmMFVaIPADbI+2bHYTlGp8dFysCoEVRJ0vQCmIEEisHWSbYOK3Z2cgoHQa1chGXAPSrFHAlrzD26Zp3QAMmqPYwjzVoJmorvFxMVAQryRjtAJw/h8ZSDmdvjWqUKnlVQaC63vwWwB8laTagnMXLhoUg0ER2JhTE4owEThOXC7MTQQWIxAGAMU3Yz0jIjDkIP2gnYnJTsxm+Ggoz7BVcGpbjqxGsR7tjzOe0+wvQYRJR6vN/GhK+JQ+gaGGs1oWK5+k9TrulHuWWriAFhbeU3WgCWWpfUUDCaMMe1+3bk5CrAF+V+uV1ZL3bzI3Ox733sSduhEYsl12sSh0nEWAHkbzf2RStFIXKdWhnAXl+IQsKwW+icuBCbQ21BZYczKHobexuVYRpY8hE3oRHIekTZsKKYRBmJaCFCpEGZYVT5qDPiR+KDajsCLANpJEid/JuHLr7NIc0vzcuA1JouGrI7GAa6fputDq7UCHDwJYKgBlCX6Xr+DFPUStJH9jA00Oi7iHGcITzX2Jn5hvNaWXzLGW4iDRitU0ZpacDYJYStgTnAqL8owTo/YNgyO9MmVFaeIi5xuwXECYjrm5FhkHRaQgBbkOfQTC6EJvOGP3NSw+yBTDHeEPk+4Rf1Eo4OL/41B4kTDDv2K4NILDsKlhL6QLEVLCvoL2CBoP1cvnrzI9//PfFEAo8nXqBX33j9X//OvwZ+cnZmitThCxduQI4np4ZvXF09cXpvfmY+mcxGQ0P7u6nv+NC3BYiQ8jinJxe3Nm+S03P1aubxx08+9tgUaBXb1DXcSlKlJ7lT2dvO3nX30etXkmDBvvbyG+WS5a67p5I7Rdb6yPiJhnvCNTp9PfPlwJD1oQff99bVs7hHxg7OVW2+pdmFH1havvLa19M3X3N7m6l89Vaq+qFDD0zGD++dO+tsrRb3ayPDhVI6C4SPhDkKnqKUKGS6RRKfYW23h5axZAlo6Fj3rCkzv1on5r38d9BGJEk3MSbuYq2arBNtaPGGLbOJMdQ7oGjXt+uLs/Pp5PlIsBudGmlmzltsUe+Y4x/9xkf+3W98+eVnoEKEBFsSsaFKJQVzITwOegYGLfAYoj8IlEAHGTsf61a0WIiyytvhMIGxLAboJMY/VryWFPZC9g/rDj4CaxZ2AAFjGNTaFl8QzHhdL2uIqky2svkMIdO+xBDTjjWNAK54DCXkVj5H8YMIJLpcqmHpjYYDZFJAg1g6gWCYG6AEHTx4EBIKuiSchYwgUlpiQ8PEMMPfR0fHBvwStkob9vY3SSmmEmKJktFNWECMPOEV5Lj9nVINBadz8NDRmdkpIrmuXr1K6iiAgx6HDe9fCOgMCd6K6yWs1RuJkVlVobxHvhQNx0ZGxmoeQLyKe9s7kD3YzWuvvIr3l/Qi4sFOHD9KxA8wmnBAaB5v4Kce9wgD6Gj2nOUKNYP6/vBwMDqO/Iupk15t7wDJRiUln0JBbfYyPHl1hbzeI8cOpyniVaxwDbLD1sY2lt6jxfr4xCh4kIJRJgmeOsxWC5BgjTZO4hx+LRfITjY7NSXIetrbWecnoyNTk7NL8eEJIZpS00RmC5ntOcza0uvgkGGSORLh1XH77Dv/EGnqtHidbkyF6PcmHNpmRdh0+aCY9JA9i12a88T0waq5VzQcQlfOZlKhoI8iQkgGhBbk0hloFQ9HdfZ7aSfQVvijVTgd+guzRBmESSMjIxBCF+C+vDKHmJZRx6jGxBLEEUNibsAHMKebJTL4FXZ22HG1ksfuIH80UiusmRRglWmvUuSjS2gTYbEQKpYtv0fTItER0w1LXuwRMsePcD/SPpwxHpirx+GMh0KcJ4SYyaRTZI7XS9QpyZP9jLMlRyxjtzeaGCql8wQjMGyMOSZvyo/48FRTsALJAV+xXJ/8AkAhe70JdiK5vo52HeHbEQPidGgyEh7yuRDcAMiyNuoyTTKPJMXCJJEHa5V60ENdKYqLEi2uioSy2aINt5u5nS32CSHXin4ESlp+IUKj7K464cQ62NEwNKg0e1cKqHFlajNKjh8wE+YcWu40G5WGQvOxMkDExTgIaZDZFOAdS7XRK3VtFXu47fLVS/3t+x87OnzvYYuvbWmc69hKbapR9Ou2gAUMJVwtbCpGTKWWunDWBhA+RIVxc4lqWCbQaWFNKFtFoZ0wDzBgSUI0D8FJqWFah1yrhYhVlFkT++CcVin8yPA9zkMYzf+8M0xIi2vwpXlVR/Vbcxu90aF/1Xnx0zvnbi94Mc9vHe/sAgbyW2e5hD8RaHFrjZcCqOXxZbDF/ThDeKphvbpSA0qLucYIHgh7ZNtpocFoNdUIbTBguDLGFMXTyiUsE5JeFZrGYTo10HHhH3oirwwHNx90QZtZ/+m8CLP6Nmg9F/Al9h2MlzJxO1G5SBEDHkjB7gT2IY+yexWEzq/NoTb3rZOT02+cu7WYiI0NDZ+/cskXdH7py19+/Mn3DI0n7nngXhxeH3roQ9dXrv3hf/pPiNO4QJqV2s2bm26vpZArXLpwtVW1fOq//Vk0mHjx+Vdrpcbf+t7vImUM0P8Pvu/h8xdfbzTzl6/uXbmSjIQT5VJjezsfAJM/FPlbH/np115/Yf3mG9l0b3l+en1lE/G93a2/fT099PqlSr25dO9jYCndKDSev7RaaDcffv+Hh8anaMy185cj1u702HDcnv/0Z68k3ZarqeL4+Mhe3Tfimcg0fM30dqtiifkQ7EAloUIJEjr/V8jhZy0Q7G8X0KkZASPEMBIMoCQdcWx94DRCjtJHEf0DDoB0qbRdwfnqtLniQbsz2vV5qv3G9e3NeiOzNOPt9PKUofdHgbKs10tve0dP/L1f+u52+yvPfTW7MHUA4ENcTf1ujmVDDUNCqUg7QDalWJnT4e/ZYOVajEYgRIOVAZkr0VJYWIRQUFuBTULUMwQXoYCdwj6iXAYwzsR24YTjYyDoZqVQnArxFXwjfguTqTVVkQ9IChgYnAysPwKJsGyTej4+NkyRVWCk0RiKliJqdN/vjociBM1hVyNxGDcBbSDgFFPzlSvX0MbRfIgCQ02an5/P51E8uHmBDUoZZ7Jybq7comwpKbduCiVNzT84OXvuwuWd3f3r11YeeuTRqamJv/qr/y5B3IqBwYmhBeMqjlwSlrATE8+FyxVGEBkeJsQUkQdew7KEesICSAsdIt0qFEAIwAhEODYyCtpsuZCPxyKpvV34AiFW+VwawyqB7Y6xyTkqCQfCztnlI9jl0Hdx+xG3Dd2kS1h005mkt0i+bGN7Z3/a5QTWEkhusozZtOVC+YEHH8FISIbS7k4GLyk21XIlW2+U0NeQR3BPdCusFDtLmWoChGWNjY5grd3ZWt1YXYNCUMvJ7SE7W7IFWgfRR2Asm2hn7VYtr//5GJhgtO5YgLAscPm1u2FdxVIWhE4oCH2mb9BJhCNYe6tNKDwxHPjt4H1N8qzJLcGEi5Ka3NvLZ7KIGjie+BqVFy5WClONFheIo1RAzrDiD8BdSuAUbVKYAcHlQKxls8glBPvBlcHg4bIShpVyEXhVhr5QQIaVYgpLVmnNVhNKAHkhC0HgqdIxpdpC+YhyxlmLMUfSLvZo/LwlCg/UTBqTNhZbD+USEYF7kfJUKpTRxeAEXClNvA5PdaYzKTSw8eGEx95P7pErZWN1VkolfAEuOI7VgdOB3AAkKb/PDzvBFxD0hQkYRook8MRhZ3W5KoR+V5shH/Erw9MTsyPDky6hVACUhXYCn8RJisNbrifs0tLXyfZ2ujDMw3fxJRPSDe3olXloMp/LwIK4ADQyNpGdWHcMiOTby8Ir4yRrQsg9RFVCbbAYSB0S/YWUw8dETvij92LGyuqFbg9MmeBNI6p0cCM7EOvgxJQvy1XI8fB1h0BpG/Pe/dj7LK6CpXeVlrRcjZ6H1aAy98QbmPvzcOZAECEo9MyfRcKZ+OZAsYVBdcoEiFUJeWNxoZQh+9MXmmRWmTz6mggO8wKBgdzw6R1GyLfSLm9/T0d0nU7+zYOTg/N33gz+/R9eB0xaJwfPNF+zK8yz9eFbFwyuMSOli8ULzes7Hl9JBzJa0lTWHe/5rRHwxID5iGasb2VzgdGqkiZSEpopYh4n0UUGZgK9vnN/hkVPpDlaqfof4wFTa6QIs3oZ1gHHNRwYIcBcpKbrkD8fgU0cm8BOYEHtJMhZ6owfsw/uKaINSXmwbclrGlUrMS9oAjMzUdw977v77rHpycs3r+/t73zlqa98+4e/7ZFHHkJxfeXqKwROktxBtATS74njp8Ymhz/9F5956svP+YKOkURsdys5NyOw33vveuDsmxfXV7bO3HVXMBQLBsIIpW+fTQ4PkWhXZ8SmJxYwaR07cvK//+XX3nzrlf/8n3/v7/1fnzh/YYUkHxbE9ZX9933oe8KjUz/5c7/5A999932nl9964eWvv3RrbC7iDcWGx0Yrhbzb1nn47hMvfP5Cz1Y7fcxz0zmcJEPO4mk644nZuWT2er68HU3YSlqkZA5B/VymLBjROnUSGVwWEgKlUcLxtE9MdDPjz0jiENf2kMRlBCPmikRqYkhclgY6pR2zIjDArgKyf63m6pV213Lf/xH7gaVJNrAv0G0UqK7d8MY8hez5yFD4V37jR/PZ/3j13PX5qSm0EarqQJVZapBSnBPYiLGQk4NH4XWLHJ3y+w4INPQQagaz5BV6G4mKC8JKFbfbAjIBXC32UxUBAXxHhF1WH/ZnrGKYmOota4AEWNzGoBao3nCPkkeWbNbj3nHNhIfmjhzcK1CBCocrlTMIpGXSIfuUjYGQ+qlymM/MLBxmFDh4BNGXvKFhVNpgnaByIEhDcDFu0yRMmKwv8ndK5Qzq5bGTd/tsge3tTcKQlg+duPue+9yXLlPvAK8wMWIIFg889MDTTz9Va3aGRijK4yHgMpdGu60Pj96LvqtAsHAkFIviLEE1p1ZEMQ9KZf7A0vLE5Ag/39nciMdieMTXVm9trq3Sdx+Q+3jvOm0quqVS+4gzgGo4cPqSTYTzLkdGehQ3SJyYsXKVquwOkJCxBswvLNOZbDY1PjXHLsgVC8Fet1q8RepRPDbGKBcLZdKFqSRz4MBiNrdHMYo4tNBnxyJNWOBevoyTAlbNLouMMCiqZrM0N723l3zj9W/u7q596MMf9VG6hwTUJqaGUKlMtaIEKwo0CEaTOYbLMKZG1DKTrlXx7oNlwqATldCHncNrCbDF1g9t5rci6cTrg1AKD6NGh9dLkBb2AeIENDWqrGAh1MJHOgsIO+VyMCCKZgwmFMctY1klYm3XFPTV3Nv63EGTXaufPHkK4lsslJLpLKVuEonhWDxKTMHe3podRGXQZazddCaNwBcMeKge1cIcDrVCNAOsgVFArwLCjgGF03NeQYwsVANr1e15qS0BvUHka4kWDqRNFjyWIBaRyh2Va1/begqEMzLTWXmkA5Iam9vfp80UNYFbE/LnwlbidgMUXSoJDIstw0jBZWHeAX8MQxUlCsFBbTYs6TRo0jVcvAtzExNxyk6OU4C6WKgB0cKVSBvVCsld0EHxSixOiMWi4abAHmKdCDACWgGtu4ANHEUf9EiFKEtTFc03DEk8D+rMrobkGvYqVYkDwgJNR3cWlUFMgvxIieaujI5MCrJlCkBG1uYuOCWU3rE3KblU6ZQq3Zw70p8/PL54atYyjPaQs1h2SEG3OCnZ3nJRWdhO/SaeSSwFAa6ID4QYoOcSLd23qsBa26qSqkhl2LArGn+Z36WjE11m6B0CAWvILCU1lqYb5mLe86LeGNMMO0VDQy8YI6QUcWHuLzRa3nPVu1/5yMXG9c1b6TEDlsWFjDNX3j64gX4qhjegLPxKXMvczXzHs3UBhzjuHaaLSjQw18ofq7IKdIixNZcx2gpyZvpoIF8hzsj4zGRBYzHOUPCC6SCigTtwnkoSnBw4gFmoYt5GwJADHAYu+q9/ODeQCqgDzb0YBB00jvNMsKJ6JQLdboHpL2IL/fIq8BFnq6MM4681R0bHDzgL17ehgM5yHZshYS0AwzSg7Fj8yJ6AqnKSDjbqzc9//vPkEB88euieuXvOXz6PGpAYjSeG4+fOnj/39vmJsUkslt/z0e9/4823nnz8va+//iop51h1bt1a/dV/8k/Gsb1ZLF/+ypdwNLtd/lazd/z4qY2tG/n8RjhM2XkMwaSfCnXhh37wx19++VUA9A8uH/7d3/mDdLZMrXPKZ+3uF4+fXAjEwyMz4z/9yz9sbxdOPXTPo76HfvATf6dUq73+1ttXLl/s10vvv/9kyNkPAtfVqC1OjMYXT+et9mylOjx9oG6pxqdPAQOS6mwEI0Fy6x22SC5XL1PShWpn7f70UCzmqPntVL0A5Yqa1ThqqadSESaNCvdiMOjXGojGAEtR9BJoR5cr6M61ah2LOzIaXtlItd2WPmWTKHnRtpbqlq881T111Gdv+u1BBhgVg5yLCkXgupZ1FJ9//Nvf+09/5U9efnrrnrsS6d26C2hEoMdq+KexHQJl3PV4VNeZ4irYDWFsEEyWPEoqblez/mGQ1a2tDQaWDqAFUbwhmojY0yjBJCQ2t3a2Mb0Sr44+o7I5FFbEvyG1suVW/7rwvyFsrgRqONN+byoSGecms7NzxWJhZmaGirx2i5/MC25FLXFYj8/fSaWTC0sHU+l9aD6K5ksvvYQOhlJEbWBg8YH1oJ2E3LJZoIGZTJUiUSbiOgguEmr6oaOnDx4+hI67urZ58uTx3/u9P/irL1iQ51DSpqYm/aEwa9/lI6mXVNkW8BVkimDiPnLs6LrJDEpMjL/8yhuhYJSCS0BdPvLQAwcPzMNWN9dXYcaYLZkmVDVAvsDa7LSWTp06lctmNlZXjh85TLQS5NoBbog2osUS9kexOLeo7GiioCVQ+IgcC6KDF0qwGQAvcxhl2W4UtSarzOX0NsYx1lOBTpBJ+Ed5UjQWP+Q5QoIWRZrQ/Xe3tlKbe/efPgmLvHXjOnGGPQdACuz5+vzMKB6aRj33+ivPkE5DfpTb6x8enRudPAY/wnNOLDBY1+xgCu4pE0IHehIiHwSIB94+GEVCUMkNJ9IKWwA2B5+3EQqFGVw6AiGAicq0L4JE6jp/yjxGikNvg+SQKcR0QtdI5ukO6ymYHMx5/PnUu1agPIZ6VD2kacKC4UaoPNyKX7GI7E63HPsISB5YJtGafb/PgQc+n6tgh4a4yHXZrbkII61RuB7li2WgoGRTbBAG3K2XiNlAq1AYVNeUdOEkZxgk7RvpDSK4zJFoorgoIZ/SanBqI6jub+/IC9lusNQoJ2IjNIlAwX6Tq6D82LDbFFCllWDqcg9Jrix5ma9cNl82XcO4Hg7Cd48MD00FvGE3lSoIu2g76kAIWdwu8gSwYZCd3gNP1UUL2Sv0Af+2F94dDlhAJykWWyCYFEmTL7HLMOAqCBL5poVOj2KFoqs1Y6JJxEeQi4hMN5OnV0aeDjKdxK25ERcplIFdr0FpJYJiXES0YruugL7ZKlNO3Ortte2tcrdUbBchXmOLw0cPHolPBWzDdksQvWzXUt+3uOsWRx08JciKxdGBNgGtRXY5+weHJ6A19r6T3A7p3PB1Qn3KBWonEObNgWimPcMqMNqGdHFxX6OiafEZiQFONmAkgz4wd2Ikt801dElzZuZLJ8UR6RxvDSM1P3nn5R3e+e4371ypFrIGDFtjGd/51TtvdEJPkMjCm9t/6LK035ynIcRma9Fp1dBKzQbsk9pNmCAkFElA4iuj+NI3IrBg1UbTRThgVfEn1msugGGrO5AKbsJ7tiJPkXYrTZp/+IpnsOVgzGo1H7EgsDglV/EFv5OZXyKE/uMLNVt1tgXZyWvPlitXk7XcDjU4GmSiY4lGDiDSt4HkiUECUYgYOCJzMeEAY46+ZQPssNWEgrHWP/j+D/z100+dv3Dun//Lf05tvUR0+BtPP0fOZb3WwH346U9+6kMf+LZ//U/+A5kyk+PTn/rjTz3x+HvZ0cndNIaww4ePXL7y9qc++WeYqff3asLMN9Y4fvv2Wy8TOJNM52l6MtlLpvYwbGIKunWrcehwJDE0jilv8cD8x37wu178xheee/U5gv1z2eI999wH/jBODn/Q4SXgpJILup0BMoIgZRHq8g5vJ3dPzt/bLe9f2y6UapioAlQRvO/RRz7/hRc3t1K4CxE2T596ZC+14QsHG8UU1kEXUk2z7be6guBhtpUZIWg/I3IRTEw6YqVmKdZaDav18k42TfrRpHt4YubqTrbZc0yOTV96/TVCbu65b3w0fl+7uX/l7OuZdPbo0fHoEE43R6OezJaSwyMnfu4fv/dLh85/5r+l5icdlQIBmBbQNVpUkOpbAN7I5fe9QUQ6tJqBTKqphzJBGKDDmlrNqT4y/yFniJoy8XiYCUOqBRZL28LSrVLWvGnFD0gUh4ojGTUDOQ2ZA8AvzBbYQm5cv+W0h65d2wQRBO4LB+UWkaifor8g+OWpWApkF9p2JIJyaFhsALhHYK0QwsBwJt4KogLd0yqBk8jwyS7QapSkCK5IrU6MFRgKUJ+hofj48kF8rLlcZn9/1+1zp9NJfkIk9vTM3JWnb4Sj8dFYNLe7K5Aft7tcreEenpyZgUmj6MP1CRCbmp64+657wERaX99A2c7DdBEbG8AjguCkUKR0OhWPhlFrGQRaiPGc4GXxFGQqVEG82WHqQ4G82Ov6nTYQfqKYY4J+xpexQ4XnDcwVg3a3VcafSqh/KpW5duUCiAWgk4wsLGHRzuWz2KArFJRIZtCv3Q73/l7WbidY3IUAjI1LFNXtAvC3lEuTJgNUBV3K7G0QLWftRiG9APGjRVnD0jIl5yragKmHaKNKMbnvOsSWRO6Mdkuh3wbuQheoIpw1XgeF5qLq8PM+0pmoCbue/U9WDA4GtBywkv1eFf6DBgGogObB9WwudZnIdGycVFr0sP6jTBzzjZ1EjIUDW5nLyZgAL8fjaCcOdpRO0W5rLzEU7XaJgcgR/OvyewGpbFRKWGJ7DTBQFH7AamQwDKwuCS2QF3iviT+B78IwoZIieYAwyiQ7OHgKNJT0NUGeVoVcyfrGPZnNZHY2t4DPJIK/SC0T0V3MduCnEGEslBmqo3s9EYKvy3n59QHBZIWjw6HTZIoNvy++dHBubnZ5NDFNPypFJq9Jl4knZuRJKIBH0VosipRTwHgimzM2HjshaTJN9msla61HGRo8qfh6saITJ2PkBloJTK+0H8OCtCnpArsTQY9ITzrGpOqMuRx6zVehYJDaR6UqKNCg6+BOpgRys1ApSAP1sdsIHqqXO7mmtWYNQ9f6jz/xqGXYZYlwx3y7vdVtVq2OBvWICc6RiGb+0MSMx1bLCLMZDBXcNTWGpxs+AwuqFbMgPim/iHRueAzsVfHtyi8SI6GpzIEaSyP1HwcjqX90aL2Z26kvurex2plNLtLIjuec6brhUvrJ7TcDnoqyqFN3HsKdRR/0FJ5FL+4867azmYcN2mI6wYPZGxIazUdxR/0ZRiuma9ifjJPvYsAIInzFdoDFwrz5CqRJowfzXqZmXhGcpECDpif2Rwz+7fVpegcLl+pMrwZjoacbOwD/0GJZLZDS7xwIq1gX+QQZUdAdI6ZMIrptxpAegCcD6+14nDjTCtXSbpYAG4s/4Kg0pEIZgZFS0k5YLNKRMiS4OWXMGsBB0HlFiOFnKVerJGtu726hzVw5f2V0dORn/v7PXb9yE9J89dJVaAn2tgsXLkwfiUejYbYySYv/6T/+F8jfj338h7Hbffmrn8UMdvHSufGxRMA7RKo9rVXAowfKAJTjzemZSaIUM5l0u1OhFiZVNb/t2+8/euTE089/c/7YgTP3HGm0sruplXjAMzUzvjC/vLEO4MEYVtlEyAUqz3atBtkgjRkzVCq5s7Rwf7bcTOZ3437PI9/+nWcv1r/23J+dvOfUs1dWX9vcHZ9apFo0oBT3PPlDFy988+a1b4yR6mn3FzMFW6MCmXXbWpndQjyAUM9iQWGw4MHC+9Xt+6ye8OZ+iigLAOdWN/OBuUijTfZj58bKJhBT05Nxp2v8P/3xi8nNW9cu9MtFy+lTuz/0w6PLh4O9/nYAhGPH+eGDR77n/zhWb734pT9vnj4yvrOWxHRI6XgQu5y2tjeE7EpeEF4YNrGUGa1YsxuYKT4yPawNDHB8S9SPH9WTED42vQN9HeJJPAeUjTj7HuY5exetnWRCCuQQTa1gd35CyN0AC4dM4pWVNRwi4+OjYKq89FI9noiMjg4T+IHRgkjP0cRwJBYnfIP7cx5bxRtvvb6zs3XXXXdhbcaUjccYHbNO6IqIP4YcAgybOAzBjvZ66yA/U06Lqrvra9exCt1zz8mnn/0mztNqGX4Pn5XqFRtKnDx1Zn9/H2siaBneoQQxIZay+IE7n/X4A9evXIkPgeARwkiKH/Pmretba7eUf0K1dVSlGqFWTnzcsGHGBxsqtlVQRPB+EkVFKA+KkcNPtmqjhLO9W7HVmmRdoC55sf8GhodhYfS1XOvQHM9QgqQoj8tWKfaIXZocHdsNR9bWNiikQ9GoeDxINNzM9DhCQalomZ2eCxP5vpfe2U4NBQJXr90sl/JU8YlG/fIXgDTT6WT2d6fnZn0BH5pLgByhfrdMKlTDlkxb5pfdjCYcHKXEzCvqF/h/71A9s+fvvNBV2BoHoUuFQgbOBbFl8nzekFF6pUwjs7MZlC2qDd9CDsAMCw1RMjiljAMBlg5DA1uwU84WNRYxn0AiAqbaZRYSKjQf6T6CMTqkUpp8PlBhWl0JfcwTmeY0FSoIwCKmZ9A6qd+W3N9Bv6S8FjBmLr9PycqAlTDETaE2ycQkGYmEC5UJopsiokrAgYhB4pBYjMjABxPaIr0W70Fb9zDrng2g3FPSgZS8RME2Eo5xSgPRDgmEaDaVb4yWAdeC74CeAe1G/YSqEgcM2T155AxhVqA3A+xaKSImIIfbg74ouWhgL9MpGlmvleXFQZTy9HwAUCO+oUuy65XtVSwVcww6xBtSCBtE8cHCC/mnM6L6MBoeKUoNvTWzJeaB9ExogzYtNFik2PyxWEGLZftRE4TUn8EYsSZxEBVahR7QsIRo9Eo9Xy06GZw7OutbGLL4qENesNgo35W3+KoON/4v1X2S8sKNeQSsgsdr6sTR5C9TM2gUQDvUWaDkmQoqa6jhBspoMu0x+q3MzkZTGxic9bs7608durMSxS3pkvmWJ8Cw6Rg3G1if9UH9vP213v9PB+uHcwN2xQXcXKOmyxib20qG6Y4hbGYkB/fRxWK2irFSi8wf7xl4BDj6DpfV0tFU6L0W2GCJIZWJ74oHc72RQ8wc3vb16isThAUD5koWDCOj3F8uRpxQjwdGDdMxCRjaK+90kncIIqYTjKQc+mo0UhtaE4ybXCXmiXODxnNeySDE6lud+UaDVEgcg1TxcAVC7XrOa8fJT6178JydsEO0PZI54e6sdyRUZDQWPIYYEg/q6C9AwNpdayvr3/Xh7/n3/+/v/N7v/r6TsD9/4MaNWzDdhx9+MF8u/NEffZoAVfguOMCHDx5eXV994fmX/H4P1eISQ9RgJTwH8dSDZYgdFaTsnotgbBuVPG3OWjq5e/16amgkBrkBssXlDO5s5wBbIDItQBGmXAlgPNL6QpHQzOTBRk2Ra2woxO2Nne1MMxuSfb2bb3ReefXVVD92z8G/ZQUXATQBKiHuJhPzRwsW33Y6E18+0nR4w6HYgw++b9fSe/Hq+mxsdL+dS5YacW9kLDyKclao5qcTE/YekUBVuJXHZylUuoV6PTGaWFg81EnHdi6dJxxp9uBSzma/srfX6jrZ5lGSCd3BZJY8M5zT90Qjta3tzUK1+NJL+6OjwlfFD5gt5OyWFyPTD/zEP/igx/nSZ/9099iBYWJqb17dIXAYkAU8UwROMXfouJpZlqk5YFcDfsxMUapPig/VshWU2a9Ui2w0UkEI7IQTobAJOVB4Nugi7A7skTKosIiJDaYm0P5e2ubyjo3MRmAhoez5cxeOHTuWTO1Wa3mfnwpCYNbq0ax/zMK1+n4Hju0RuNT169fefPNNNgT2bVThTI6MSxQpXQyxhb5Bbtn7Y0PT48OTwRDIFJjH3I1m9fKlt6/dvHbo2Il0evfY0YOIAaur69U6oUJFUszvuveBc2cvjEbDqe1ddF+rz03wHT5mOE4gHEruboNxmRga4zmgKMlNabUCSshqxNgOBggBOkSQISoRtIvwSR4Lkd7gMWDDwbKCFueoJtfqxYqtGnI0SILHl+CyhaKQkkoWwC1SxYEUs3iiiEh9otQAfMChvlfIIFQSVXTwwBwSLgiO9ZpiRrH4o5JvbW5CUKb7M1T9iQRiUzOjlVLG66GGOgS6XSnWiSNHviOxflgebHuxXIO2Y4Yi/gpjF02nG4wdN4EBQjiIrRM4///mwBOAgwMzG1m1jDtrIxyOktEEDYBp8XMIilKSXF5ixJSM1KyODI9h2iRQiJLK+GpVAJjqzUTluFRgi8WBRofVniHmDqFAmF8zFKp/TEgedMvuJBELE7/S2MgzBpCMNBcKeLGUbH3iw3FXUaxqo7aCiTkRDhBLQjhwpVDFfgsyFWk80GxahduJbkLrUZ2hLtK9OOBgptl8guTJJCrjHwQH4BIug3OLxuHPdiE+Yf/nYCU38HOETIg3EiY4lSZKH+zjNjkeQYJbIL5ECmD68bh9BFiNj83OzyzQFX5bKZHbCtAraUhkCTkrhIOhoAjpBvGi7w/SO8Y2S6lrAe2AWYqPvVLG2UMcNx5uiCCXMf54hJEs2HIextHDkqi8M2OG1Q5YCq/Y//Ve7gTDhiHeEmpt2AJkRrBS5yUEfr21jOWlW7LGO5nqbttWnlseO/rAMctS3NIr9IubREJiDO9RE9VFMRyLhdgdiiE0sTwTfycGPODD4g09ZeOJCw84CeIpSDGkm/I8copIZFJjFOFMg6X1mmOgoeouyFV3Tg6+UsQgy8tQAfMq9mMC7sQ89Z9hdeLHEjVu/5hRHfx88PrO+Xc+6lId/P4294WDmm8Hz9L7wRXvekWX4KT5Q9zQNGBtFwNGER0wYBlW5NMlsZPLJBHxkVVkLmZIYGZ8e1sP5udGfNJ9DJPWBXpjxBS0fDN1DCfSHNNOiyQv6l8+m46i2fAgVqwOphqjjrmQcEWZerA5UxCGNnAx1Tj0n4OSlYhOqUotWbGYwF+k4TrRbjQZvs91arMS35hbrFuK7EMbJuIehw57U658RqDXw+eHl/HTf/ppKmldfH5l7EACXF9aAElZX9/8yEe/47WXX7p+fZMggI0SdkLv8uLyCqbk+gO/8Rv//Bd+8e+TdYg9E1U4my5Qs9Via+IiG52gsJidTM79FP41bN6NrY0aXkkgG6k7sjgc3stdf+3lZ48fGhtKoB72qGT3+qvXErGp4aERtwuYiB0yEzyd6kw01LN3Cu2G0xvC0NleKtjcQUzHl65fStfLQN1OHbp7s/Z2rVOamJx89PSj+5X8U3/x14cPLAQTTY+l+tbXX+qn9v/24++bnlpce/3Zrb2dkbAmncXOGBOmHBmOjszOWBJBa3FjeikOLlI6v/bS9czU0sjEzMFyvmFruPZWd9YvrN574ky+TBBTzxqIOWylZJIUhmijWahZSczD9sNWXLV66h//6fe7XE8/+5U9EDRnF8L5bDEc9QM2iTSPAQ5yx7RCW1EG8NHBgOEoWgWqo4rnCrkKwa0FLDeThhWQj2wlU5WFryQl4vtT+CJOTaRfoSigdUBZfYLnUWVD6uF6Mxlysldwt6fSu9FYiBAqvHRoOOiaZEzCUHEAAylAozFN37p1C6ArMmZXVm7BQSoAhVBYBv7khG1T1yDPWkVvppgmRBjFuFirDA2P0FS081Ip8+brL/mD0eMnDoEAQWR1qAUfwcIda1hdjz/5/vnxsRsXLuxsrQfmp00+cSgUD5eqQCGNoL6vrt4i0Qv1BGt2PBJeW1vZ3t5mMHDjVsvdUDgAyaHWHAyFESNz1W6vG13fyh0clf11Vik1N9pWym6SPkTtowbWYkur1q4SG4Xb0+mFDCOfIFJSuqFfX9/fLubz8HOSrogkJHEWA9705Ey1lAd2ayQxdO3qdaLGl+aXakOVVCpFMHWvVSWPGmM21VLZ8bVihmp/MCTCeqGiNiW/gfgRoKD89JFToaFJbKpwlgE75I3Co6gXqwMSoK0LXR28gesDEQd4GHXT1d2uBfVUwGduL2sMNRGD7ACLAxbJEETCMeadKaHIlElkc8PkqwiTlbonQWWJAFwEXZN58yPM2V0wWvRpodTCJtH9FDTQKAOjSHyhlpqyjbE/0xhZrcjmE+PsE/9P1HGzWsYyQD0tMFrAZYGzUogB8oR9l81jTH/IDmB4QTghMqi/pk8waL7GEY7ghncXtiQTtYgeK5cgHw44MZGEeElZhfUKUgX9AQKTxehUpXTsHzQBuQC62HXVKNxcBznZNR6nVOXC9OR8PDaa2s2ooKfDH/LY7F49mW3AgoEgKlNFApp1aDhmi0ZgupXkfnV/HZsJ3BdnLdZaOAvPwJ1P0BskFHgwl8sER8rhgzuA6YF0Dg5ItWEeajjLEs4gJgP9QB+SExL6iZlbQcasUwzBbFjqWiAItOsuwPb3jj+0uPzgCRmcS+u19JtU13YnNOoYyOEPeNfZUJAEGsBktPtVIZTJbQH7ghcyulLayB4Vg0bwpjxXG388LbG4HETGo7rCbbGMqn20zJhH0eDvNJ9fG0cIzTQMkUt08LX5lzfix9IEdYgT8Gnwazg3s82F37qXIVJ8hD3p6jsfdRGHxmywCoyO+K7fcTnfcbku40Lza66XYorwZo4B1yQekU9wWZ2DfWKmZdHJsAwXgwGLs94W60ijxr8i3ZdNiaA3qMFgfqVMbYUdsEp5hGktEgwyCq+00nB3dUyW98Gw8YmbUzSDVmr4MMfAI9VkvUegZr9hIoJqMOCQJO4J8mqtBWPCztEBgaFN/Sifu+/ykhZA/Tgtsw7YG0gIyGd9tDRuQOkG6nmgKKNdQrcB9pV04LSTQIyNian6F//3vx2eii4cnUYRCUaDdSJ9ur0vfvHLV65ceujhhw4cyD7111/DbxcMhne2tqHXH/zgB3/r3/wzzuymNvaTe6p8LCqPnc+L+RLZHHUHJ18s7CfaJJnCTmMhhJW6Jt/9se/7xtNfff/73pPP3Tr71ut7GMAXl6inBsRAJldeOngSRJ/V1UzL7XcHnI2QH9MVhsVjh49WO47dnZty8Fn8+8XNkdmJFEay4MjCkXtz5968ubPXsT1Hqv3YzMjo3OTKztuPHD911/uj6YvX+/5oqVbuOXxj4zOtwjo6FY0p1Cxs95GxEfCar12+cHXn+tUdi2fCsnTvqUngJZrN1N6+xxVn45pQ5aH48FJwOjA7N33+0tsXXts7fjLq9kcqGScQzqNjEYIxypV9wpNJafihX/7hfObffu0L5VOHY52Ou1qrIjGgKqDc4yKHikIDmXszwwqVhWqwuIR6pFRgMCQqhJUimkJ1QTYTeYNcmO2o9dCUJQ/+yCLlhnAgIrkwCMT8YWLiVOKkUnnjjTdgnPAzvgyHqWHvVNKHB0CLGChA5JwihxXrZczOinva3MYkAqe/tXIjGokTGww9ViS8L4CRlKeEQv6Zicl8KiOaqwhYVVG0txvBaATKh27Juvf6w9TFGRmOhyNDGE1Z6YeOHYfOt4hewjyK2ZI1ZrHC9aCdEVcoaI1Ql5CtSavWN9biBioLQMl0OosuRVQThBRTLrZnlD8qN5CCPD09i4cLHgWwT7GUd9TSWwwH2Kv2ds0fjlIvCFZsVwZRpG2tMMBMcqMeJHTNSfUQlqHHBqIDlsg00VxrW3AgYrKxWq+tQDq7ExMz46NjuBIpcnP18rVKNYcgKCcsYo8VQDh7iJQe2RtLMBmVc/CFsZKBL1Ms1SMxzAdO6vyQisCkYgRlQEVDCIK/Y/Hg/P9wUMsClRUMRHguUhj0Hw0YDidepSw1qYzchI+G7mCpDxMOACHmPnzN4iDZEz83fBUF1V5DihUFgYsz7nCWoJ95wnPtYcJUWkAkRQZ9fkAhC6gf8oEJJYCgiIaBMk0N4mAQnBPv5t5qp1InOBk4SRQtKLtdllpDLo0dTyZDLMUKbaORUEbIKnRMHAFyIlYPShZrE4omTUMWRYKkBoDZtE1mcGKqHXbhJNazshWTSijWi7+FN7JRZ5NFvys4lZiZmV6aGp/zuMO1cmtvLeMhH9hJgIiNoeDXSGSYbkiJZqKAq0Y0BbbU5rFTgAa8PcLet9fXxNWhpTB1BlTzYrgA1J1yfYRjoT7LVCgQnEYNIBTAPxgqQ7JvczO0JHqh3jAUyFwSu+i29mQ7W86FR9BhQrlmKVXcw4k1tTx77MCR8KKj68y3bdu9fMXiLNvjBJUBzkk17RqIK8CmOH2mogWmSt2ZhKkm6V9wAtRyvGTYDVh+kKh6iXBeZG0tJQKRENrg0gw0GK9IafoT86Sht//jLX3EosUJ9VbMBlmBi7m5rtM5NZ83WjBsQp1hAg1/4sxt2sQJbjm43pw0N4OLiTeZgxN6zxl+Srsl2/C4O3cWczffmpPmWRJaxAd5vc1ozUes6MyG0TOlMMK16DLL3vBg6JxaqEvEXNkODJhJLkLN7GIskVSH3wNWbRay3MCcUat0T5olF696wELWo42FnOeSnmLaLws/KINcqiD5AVtWy/VQwiKKFXpM4d4QQXoC+8GfAtAwwPqVLoEfFMMjAiNot3gDWOu8PnvIF8qk0gQ+YGZqdBvc1EqpPXpF9hkqEg5V9Ig620IWaSThicnpq9dvDCeG//s3P/+JT/wkgPh1cuqIW43GY8NRZn19bePQwQMkjELuUbAwNrFzybf8xV/8xfml8c3t60RAsFqpiJUY9iRTmbjLPTwMIbLnqX6Sr1Lx/cL5DKl2s7NjN27uff3pL80uTtQaabdj/ODiQj51Q7Y0lycWGbv/3vsvXtnwBuOj02M3di8648Nuf59cw7ITU57NZw/U8vmIv0a1lEsbL73w2lNLJw4fOX1f2DK5m9yo1rFjh15949VCJv3QPU8WarmDx+955vpbmSubjywcnT10PP3WG8VyI0rmvRNgXdgZepCWAYoBVb1XN7bHZxda3vRKrvTNZ86CcxvDfT12OBGZvlnbuf+9ZwK22LNfe+7S+ZeOHc9nipnZpeOReDmXTLo9CcwGGLAwQ+DaCkYtjcpGcfcLf+/Xf6Tf/a+vPJO768Tc9UtrGN3w28ppLzUVLqtgd+Z3MMUQBERomC6Tgy4CPeRbmfeQs4kiU5I3EagKHJMRxEJIWYsAd4yp3AxnBDm+hXyZjNdgNO5xBykiiR4JMSmW7ADYp1J78UQYeCIi23BtgExA9SOyZqrA+daKiEQwPLggoaBQn0YTmDxaQp6VuCwLlz9hP0Uiyc29tewGuZFeIqVT+9lyfnZxARz/WDS4sDjtD8bW1nd73QIRUhPj085A7MCBQyuray+98ka5Cg5EGBsOscbouSFHEAxNnI9EH+OThBpn09se7MWYElG0SXn2uBNxoqbrLEAMqpBZ6sYOJ4bmF2ZwR0K79nZ2MZE7clt7VE9k69bDpZFxkrTwaIJj0a9mqvA0/J09h7PWBiEuF6pGqbJkDQeG8IaHwkPFYfqmYGPcMH1rLDHE4O7spnyB8PzyoZ2dXWCMH7379Nlz38TaR8FEbCWASHdb1UQEd0sIzG6VWCdGyx2w2H1sCkwlnsgQqJZDE1PSd0VjoIyYNJ12v9fscbYzB7saCiVqoK0uW25L8KcOF+Zi7AlYWdHk6vUiw8Hsog+yAuD/MEqIb7GM+u+gtdD9na2NnZ09WObC3AICF7oAyX/ITbiB4b1cTwLP5OgU0hZrBZcL1gOKWWC8AkYUOuwBewb4HIagiWfCimMZlDzsKxAtD+KW15/PlGrZArpvEPubMLDY4OjHWNrh1FJhWB94icQJ5MxDtISnYYjGGgpsJLPDhZBT+BWr3JQNgbAZfwI/JXSSkDJ56vpACREexu4GZ5VgbCghaQpOkuwapdbC1OHJ4ZnxsSmPJwBsJFWsQLMKeiMsYiyKKPt4eUURyZnHP1dvBKIB8maJYgY6BSTT9O42WB8olD52kJgSug9TwOBD91BaLZ44A9UCBJ/QLTg/MhO2B5cCyQXuoDniN8IWhROLKMMS+DmiBF0gu0IcgU1qbcYOhHaym8mNpCfhW7p/Yeb4kmUsZvFjpr/ZdRb6tibAoD1HxeaoK7CZSo7OkMoeNfK0h9sJNJgHEe/nNnxAApeKqLVx/VSReAGpb0qWxGWgKC0ul4mM7YRopYkQK4Uo0NrBgXRP8IgOvuMVjiM1mPfSdfkMR9VXOmOu0TccRiYZ9JTz3EvcWdyX95y4fXCPwZ34zGyJo+sRg5MD7su9Btz99v0lXegYhL1otehR/FLGF93O+NKk3fJGhmUYMHcwLNkwZvVPMglynjRd5ke/vfPKdfqt+aPvOo+lUcuR2DVYNQdypxorSwX7SXc3fcKtx8JhCowcglkZ57zwNBAcCepXbToyOXqWdIqFbQlF2lFrg+CqcqWKCwBJg5wvcBObTmDSPH1tNAxSOIMAbSGDWzYjmgkIIJyW2L8aGFgsbwzRRCNQ4Nde8zjBwOuDYxz0+NkRkLnv+shHT5w6jYOGYcHdWygXZhZnRsZG89eKP/8Lv/grv/zLI8NDOIbZ78Njw1/64tc/8l1PXrx4KZnNfOInf2RufmpseISo6d/8zd++cTNPuVGgnOIxd9g/BvjlkYMj5Uprc2PvzOkjh44c/MqX/ioc9QzHKGHbXVpaCvhcQ4kJMmey+QLSQKnWCFkSfdapN2YPWAsIwl0K5rht9TapOUuHl7dLa6+8/fKRU4c8oSAa0mvplxEKDx2czxX3p2cm5mbHiXLc2dtYTW8wW/FI/ObN68k33pi09KZi4Vav6HIFEAsCeMdiEdh2qmyzBd2B4blW31ZrAARiDwYj99zz0Mp+Lp+qjUYDTz7+bfU8YVoh4J+efN93nDlz1+e/9JfDidbFtz5jr7juffBkCWC+XAmpnvCZZqXocQK347M0Lv79f/yx8fGn//yTazOjzi45jBiaehRTBx6aRD5CoFgS2g/C6rHiHZewhZSL2AvPA5jQ7wuxZjBdYIoiJAprI0SYeXW5sd7JweZy9mGNWOPYx+DOVcA5sFgeGnsIpYJaBbt727SH6C5WFCpN3wmZsRVLKPbF/pAQPFBUIDr1puC+UVnoBQFL+PvxJnT6CHXEgsmBgTUFWCT8leCYpjdTbq875sCpWtzZ35udm5ydnAXQwee259PJbHKfsC0CcLlVMVUIjs5Au1gtMBGUDBRfxunmyuqHv+s7X3ztJa8/sHTo4MXzl6anZx5/7CEQodHaqDUwlIvDwkiiAUITMBhbCv2rR50GrCnECWJ0czi9qGNIMo56Hl8eGW/9ZhHc/U4kFg5GwvZmCddugyqVLgfrg2iEDjVpi1Taibh6cFyihV0xMnOCUTQZB7FZTh9VfsMJ6uQsN4hNaraHJuYgxMjGXnd88+YaFZ3Qu8mLwD9I9YmW2wE6iTfiL6lggGPxwOH7zzxscYYszACGxxZYilU5bkkCh0KBMCWbpgQuqAuKK3KXaBEHk0blQ5eVgn1kleFjwMvKe0Kd8Y4AO87lPBezF5DQWGjJkmD42LSExDIdKiU0PQespsXu7pH2DsupNUsVqtMzbTqwb+8okJtozDYBIjBGHobzmzXR8cp4QpqkF7TUBs+s8Rsv1Wd7DTJ2qvnsiRMnzr/x1saNmyPhWK1MaqyD6B8qdaFwy1aA8qfgFGCAYEX0goM2I4TCFoDJapILKKalkCUIHhojTJglzXrlpUV5JYmNQK+7fM0Sy8sW8kfzyaJqA0CGau1YLHx0/hTQzR4q6tp91q6jUcE9TLkiJHWiI2SfRh5lPJE/CYxjz7BigokhSxyktb0sGkQ+h47g7PXicIgWmTyMOYQenw36Da9ifDhuMW9TQ5MQRsgwca7QfX2Bot7usl7heEa0oI9gbSI6k8nooevEdHfwdRA8TrVGR40qgcnitnvYPjUXXTwy41kes/hbltLFYmbDmwArhKwDmSIJ2YfpI01rz9dhqOQWM3hiLGi98toyYnVKPfNHBrfcDIjcMCqFQ4uxQgTMUoIRKY9Vq4vOGfXUCBXMKCPyLd1UnNQwR00C33E5K8rMu/iiHq6fDNinOSMNecBpzXf8gBhg6BPCJKtFb6U0iJFR4MGwYa3oOwxYQUuygqB9KpSNaRJh6lPrDYsfCQrSkvlI8/VUcd8uoflK+0atl+eVecGEI06J/RjZDXEHLirJSYOE2MccsfjxAUsdNrST58l2CFoYhErtFA2Fg1JWBI3YSo10KxA0svpgjbYiMjOX0Fdy1AD0ZvzJIIc6yhqjlDsnmKpUA+i4bPlqfQ8DIlD8ir6x1byUtbFg7YHAgr8idu6k2ilB7V2n10MaEtwaT4kFtAWrtVIp1AsiSvjwEIjBZWNYOjXsXDZ4MUEiDBn4dtBcsNggR91eNZW6/Lc/9r2BSPj5l15mfTNU+WLe6rGV6tUDR46++MrLgbD/3/zO7yrbpFAErQk7J1B8jz/5xKuvPZfO7U7MxP/4jz51z31nHnjgAcALf/BHPr6+sh6PJo4ePl7IV/6/P/hDPDj40f1u/4c/dP/Djzz60qsvsz1xGAOeBSZmfD22OLdQrvaDIcfC2EILhN2+JVnd9zkiYzOzZ8+/SJ5QeBRNwja/eBAWdXNn85nnvsGSDgzFQIgq5ZKIGMeWl/MUBSd5pgWVL1s9ro2rFw5OLfv7tsruTtxljXiaw2FfM5eHBoCaH50+WWq1r6xlPvnF5LH7KkPTTl8oOBmNvfD8BYz5H/uR+w8eOPPWm38xPjcS8gYinlDVmnzx1Tey5b3NF65PjycQZ97/4PuLK7Vq/uWLr78+OesMUhkMlppzkI6oDL1m2mIvWFzZj/2903XrXz/1RUqkFkPuSdL27N5arkg6siUWcZSyjVh0PN+mXinMsc6iYWKJX0O6olIc+cBsWbnTrJaQNwjOJfyS9QOeATZl8rs2N7YRxojiwh6NMhUfSYyODZFfefny1XIpNz46whogOAem1Ou7YkMxhKqFhQVY6WuvvRYFjWp8krCm3WQObbAEHC1ODHyi3iDifbNUH4nHQMNN7eZRPdEmU/vpXCEbiAeh59VGAVmCcr/J7XVY5KHDx1/4xlOkcs3MHQwOJ/CO4sJo20EOBDMygIN5c2v9nlNHCJqr7BTijejzr7528MjxcqWyvbWHmzIaCscDOC2xwFvnFhcJCjp79iyayYljRym6QIUezIwoym+dv4zXdXF+Hqcq2XTFUsfRKjPkrH8rUXrlbhpjdyVEWLV/fG6qBC53uQzWeXRkmJQ3jxuEhEZ2d83pC6N3U+vIT2VLMnArBCGCNym0pba1AbEWILLQ/BQ7OT401piYxgDT6yCbcHTzGXC1laYKBUJyQQ3Z3tkNRtaHRmfYwVgvgEkmzI5EUPQ52JKKzIosIfdSxx01ckCvjPAOpRGHZpSgfGiD6PpUMBK2ESuYXer3B0PhmNPqJA3X4fLF46O9fpqZhuhBsIiCc/ctpW5lP5VhtRChHg6ElKqlvPUmdI37lQtVYikxbqAoq8YAgIt0jekrE42uoBIO8keJPBdqbZ0gYhzKXVY8q2l6YnLj2k3i4aFnuBbQGSHBEHiRV6Mv0GbUc17pJuwB6qzib9LXCW4MIAOiBdM/qLBWJ4cCFazo1khJGBgJMCTkKuiJxEKx9ZvrCK/IE9OTcy4recm+oDeECzDoixPthY+YIUQEgX9DOBgpRAC6iekHLil0CoJPycG19NIXL9br+Tq1KKgtCj4Wem6vhzVZnAeOS2+ZI/0vJZA/DvEV/td7WIrYIBQfTG6FwWEEBuzCWEMZLmZRRj68C6SzWZsVK7i/5EFTl6Ny5Mnl+KQ3OEUxF3zAl6miSERqeAr/YBH11whcGijxLJg8vAKcFkRrmbj5kr6YyCKWXL7CIPJcDtgMgs6AEyJOmRYaNVeMjrbqlUNXGI7GG72/c7zz/ltvaIJkQfqnKwfH4A6D/t/56bf+HbDzOzxdN4ctqr1m6GRKuPPAwRumR758u/J2xC/bCneHsdFpZsAYnPkl7+/Y8qWtDpRXTiJiCHbEXCmEDcOAmTZNFSOkKyUk4QVC6lJQvTx3KizI6jAsHN4sQEoRTHk+gJKvdxCjACwr1zuFstRZuCEM1WJp+FE8ibWwq1QsQ4JRmbtRgbxCpepKLVWpZusW2LpWB6kEVvgoZBnDpTJysFdQ/AgTKtGxSkgwMwa/RZ1l89JK4q+NxAY7BowdEx+xtYp7dqEzEP2K9RmJgN4qnkIBdKNDozeuXQ9Ew7geX3nlFSjP7Ox0IBoc904iHAM1PDU6Q+jQ+Pj4qVN3/Z0f/wk8fF956q8r+L7KdQCBMXehN+8mk1/9+tePHDny8H2PPPrIezEczo4svHnu7Zs3MomEt1Ci9vAYVusXXngexQtB6sK589Xa2ic+8UOAXG7vpj7yHQ+nk6Xf/v1/9cbrF6LxkSeefB/aFQnAo8OzKxsXx2fg1+HVzY3PfOYzwRCJrU3qogLwBHIIYP0z89PJ5E3kD0A2IpEgPKtUq49Ojd1z90O9Yu2ZG9dGR0J7uYylbd+4tP7gXcuktRfazudeu/DS2z3CHa2pesRRyF++uTgZp7wSFbwgR1ubZARnFg4dY0fd3LnktvuOn15K7a0//uCpe+5a/upnP3nt7beHHO3h4CgY6RbKgpEjDzp6H8WXwMyKLUzeR1U7y2r72z/+eGR067P/7aaHEgsUi2jWF5aDW+vlvb3OiSOLF9++FQ6OAWQIPXbIQwD3JX1J6wdXkYO2qkoSyqcfqAmmEpYDxjz+gSOHT1AtcGt77/r1m6tr2zSbGG88J6hLe3tb+UKWMFdKG7LRmO75hXFYGiRxdm4hSarrU39NZUB4sArnULuv06+CtY9Lrt8O98mhRUO3gC+NvuEPIBdiS6O+MHbupkIgqAnR7rup/GezgBzM8iNolPgpgBO1AW32mZlJTP3lZv/qhQsEP89MjJ/6sR/+wmc/TQg1enuxWlpbWzt8/MSxYydee+2VeCQKDlZvLDEyFE0X6xcunmPJnTh1nPsgKFATAu8vxSGw0YIySU5QJpMnncwPZgNCdqeB5kfkDbp9t1qHa3X8ZW+jGsBWTNguewe3do76p9lMJB4L4x4JRPP5zM7VKpwD8x17EP0SfSg+Oup1EpqL/E3hrQGkYgtX+tzk6MTI0O5OOZPPNRxUZu5Vi0h5YqMEcgCAwtjWS1hoNqOsyuEhAI0JcmKcu4j8bTf2bfRC8tocTj/7WBRY2gh0DHsujAug5g77n/d2K+umRpBTpQrfrPIARspJ3iqFyXr2fLHs81qDGFj9ISQDTPOwQdiPMLGxYJQr5A6h5AmiqkSqn5Ug8iDAWpFwbUT4ag4y0uFaFBIiqze/Lz4PgTLUnX0IPYZYQILhaDDQSrUE92XxTkyMQzRQtYF2InYJsoqeoMv4sTQtkWm6b2i3UY8UliSqDmFBhhd1lKka8qoawzwICTrg8rcb/WKr5iNH2RulQFu5WEttrZ05fa/T6hkZGh9JjDkxQaNo9bDIacUyaARJoShI98LOLefygB/DkMGWxvgs8K9cNtdNNsFpIxwL6ifHLoSRnkmFUo1F6L+aJ4MyqqQNXZcdpsxaDJAi8cY6acirzjtB2VHOGdQUA7eib5wumHmthUMXCLJGqZ1vO5uR8dChY8vxhZgl0LJQQc5LARZczmWLvQZugWzGGp/BYzVeMErTBBrHsBkmJgso/nnloskzXyEFhLEdHBrhQfqR+n7nGCygwSd5aHVXDnNj856v9NmsM/Ne15ozRm7ivfgE/+tPX3GpaYv59DdeBIslFqqvBw1GF9VPWMt6pJ5h7qxzNJZrzWEWCRNFJKfmAc8FBxyUawdjIAetGDk71ARMyaeBd0OcWWMm7gtvRoDTD8VgxYC5yAQMosniizM2QHFriQRi29yKECSEKMQuG0otKDgUqGMrIigXq61izdKAGrO8FMXa93tqoXYniC5JQAFWQryqQvLsUtB3N9csNHDrClcRizTzgneClaLWqhdS5nHoKkJO5dhNpj7E28nqgTuL2iJs84rIxqBxUjIdchbpLFpUSo54Z/QYMcRIPHOb29uwcJCbU+l0KE7O6GhsJH5l5SouufvvBX2intzd+a6PfhRp5plnniHeZ35p/u1z29QJnpkb/uznvvHIe2IYAg4fPcxIZbL7uWz68oXLm2tUZN1NjBIe4ccyB2GZmBhZWbsxOTXJU/A4+gOBSqUK5JZyK5y+3/ndf/qFT9+YXJQAjWOyUMxgSjh2YjEQCJP1h6yxn9z5vu/76N0n7yrVC3v7uzs7G8VidmxyaHN7HYuHLxR66+LFjZ193pAnEw2P173+z37uq+gYLOzLF7aqe5YjM5at1288eM9RdtVqr79ltSyf9KzmGpM229DywjOvXBuKqYhXzd7CcXfgrkOHTi6DdHTu3NuLM0uVbCWWAMcKQ/pbH/3IQ+5mxmtpRz1BXE6dFsCX1KuBNFWQ4txDEUIWibN2hKOZ/XxidOH+JwjqKnzlT/cnYoF+x7+1XaZmcN9ju3r91vh0gvVE3Iq0EKyPdjeUyyoEjB6qg0xi0KCBYI57BBWp3QFaEXK2uLyEIwDODKjk7u42xlu3z7UwOwseJLkyTD2IlkSTQEghUMQEMOlYIv/sz/4Mad4L7pJKctiXlxfRPyE/QFOsrOFcqIL/zcYhZAUHBzkf7nEvMadwChoEp4NtoP1BVAmoQmSwO3OB/VSRtER0LlCG/UQ8NaPDkxCGycmJtfUdwD2W5ifPnDh07pVnjx6ax0lKtBCbY2PlFgVtb928OZJIUO0WWWd0fIzNCzYDpmbUdNKN3n77bQyL7EBlrnd74EQRYs1Bp+gRCxvkfWynskKyxBGUIP7g4SAiU9GdgvTeYAgshgYcP5NpZlLl5H50eIzChAAHQRRQqlvEAbUJiemU9uPBUCSeGOUFfOeAB36IDd6e2lqJR4PxwGJxJNxrV3K53WYxi39JjAEOLBKCBT+1jx/YapvszhDV2wFrWAyYLAQf8J6UbMdUrs1LjWpqbbM1MS3JAwSbkXeBjc0Uy1zlJlsoTBYpAZXxoTi71Eu9G7cHDAzk+lq/AW5mgDJmODBdzjK/otyvCinYGpTEIfJI/g1rmKAeljZxfpVSjnrPCEvYrPHkMz7UcqzhvC5T4gPWMKAC2Mkxtg/oO3ZPRhZW2QLt1eokjml4aGg1fR2hUczHUHrZAtFeDANWsGifpDlDi0WgRWUgOLwySZgLcIbxFIlv3S62FxRrK2EaDjCvSG3oZrJlugKq89DM5EP3vQfLM0ZlNgLuN6gMRg3Mf5gltEodKD1k3TSYXux4qo7NnoNOk6asKhGCS+WJzVo5EvBRfBVtAxIuCyYDJ81K4hKNQ3ESnTcNhRTjDwRC36YoHJl5TfPpgPQ98JRZ56g8eE2g6nUoJ/sOM4srW+nkLD7L2PzI8okTlrkE5Q0t7YyFuingbOSLDk/HEzKBlB0ws2soV2ZMFP6t52uE6BeWAKg7lmwl9eKD54ABEyaHmCCNSo0VFAt7eHDQQN1ocOjL2wdXDi4evBm857t3Tg6ue9f5AcfULd59zTsX3L7vnX94Om8HbXjXT3STwSX8cNA2c0fJBmxXDaJGWufErPgfPVZvBt8a5gofE0NjJbGfdAkzI5ep4bUMksKhJItwL8ieCrvyLeGf0i+NqRmyzh7kNPZPWKyUbJOVhP0KGVCSF4s4EMcGmynhnCEhF3crN7NV8SXDqRl+Csi0e5G+p+MTTBxuuhRRodVemSlF4fVyPc4IDJFtIARY2OzT21ZuCA2SnHrEkhQdZJwGm4gWQz0R2Vj5fMskDs7Dd9kF2JcYIuRa9hm/GFwMrj/yMGa5rd2cJ9hNJOKQBSjjcI18HMe/++3fQYY/fdcxNhLWPW7zx//tkzdvrkxOTcWH42ffurCy6ghFLX/8R1/90HeefN/7Hv/Zn/kH5bsLR48cf/6lb+JgmZ2ap6Q7gaLo/cQUxIdi/OrV11+B5f/4x3/w1Km5cNjdqBLpUb+5fhPJ/vAZZzAQw/p+4+a1ZHo/NhR9+D33zjiW6BBA6SdPHCNC88qti4O4obHJke09bLAFio5lCvmVtZX9dArb4tDY6MrG5mtvnj10dPPAPQ/OjUcrqY33fseHzr74bMJPhYXg1778tVQawDrL5Bn/0Uce33/hhQtb2wc8U2OHfN/5oSez5fIe2Z+bawdPn6BKyuc+8xe0PxIO3HPi9HNJMpOvVdKbO7fWPvG9H9h+46Viq+L0EGGN5d5Zw1HsagfHo3VyHxqdUGjUYRnp9PcLpY34eORvffzeXvXZZ79U8TkBxnI3SmAFBl2eot3VxdtK2U62PC56iV1E8DVsBACgVQp1yJhiKKkK52K9EROMYZJYJAO9o2iBpaV5/Hpk9cwszhZKuEUbR48eAUvxzbfPba5tssEpb5DKpBGqQqFJwCD3krtYlScmhVU5SoXF8JDbG0B+iJ0deu3N1zLpQjjiSyRGSBsPR4cgmPl8FRKI2IaTCtbbUOHeBjncBIcDO7qxtROOkJuIIumNCCQeWtSMxmIf+MD7fuFnf/H6pYtTQ97V65c+8MQjsYj/1vWL+zub6PGsUhQYYoPAgH7vex4jc5XE3zKVkAx0MRUSAdwgGgmjC5R8f3ePWnUBCiQEAmwt6C1f0R0HYUdQZUdVygaKjLLrAJkB9c/nblDGtpATIBTV7V0eG2afdnMvk0IlxfmMBX+Q7dUnbqneyOeTCGuN9E41lvChbuM3ZZNDQRBaK+1Mag8IGAoTtBtlahxjGSbFE1sZCgAOXuh/eX9js12v5neGx5lv6YeYFynxS6S+1eLGRA5btWI/JXDX4cbpSHQbda0J57a7vLAHidQqrEGVoKgc704b7Iq5ISGYb6B2OE15Q3k8QggwIruD/mjdn8/myLVqlgu1ctk/EsMe3ACcoVZBIkEgpPQY8EiArDFnogJAONEZDOdKbVFMnaFtsH/S9RW2ZShdm2mkLgEh5gQ94QA9sLiwvbJSxaXtD4gwivbxNyC52B9x8ToHLj2RaYZDBBvvCQTErccyCoRLUQ4b7BXsOD17wBO2d7CoI1IRB+UaHho7sHhoanymnKsxe2KXBnGMsCwiHRhfn8dLX4AEgf+SQke5QDLuSdxDKO0CIFlQMjS2SEgjuJ148ymrhNJGZwccQhwAakdJFHaRYTkYx+BlDLcajHuAAomK9+GQdVfcmpmgWpEpLIwxlVs1ugxrWXewlxue3YXjo3MnD1sSIUuTtPLzSAbuoK3RLQFQ73HhxIGpyqSPtk46N3qbgcVAIvoW0+JBOJGF2EFJZbLGSUeQFgU5V7M0PYbnGVGBkRPvNnyQ9n1LD4alcRlzoqsHXTPdeOe9OX2byw5OmtfBab1y+bc+/G/eQe71aIkn/KOfyG8Bkj52Ok247sGA8g+TzxdaxzpgQrojvEcYLPjgsVMguBmmzIJhlPmTRRqZBGInA7WYqiZG0ENMDTYK5SDxLAU561XhzQQeC2WNjziPmb2+g8oaVfz4QPLAIiTn0gg9n5FpCFXNmuv0CwSSsy5ll1E8IJYnEBERFGHx5KvVCb9H6ZHG3M6RZ8kOwQqK5ETWCnGw7E9RZ3ondsug0xjaxUaCAnAwX9wKQY0Ny0cAi7gA2AeGRHYTsu8RjO0IEET99rmGMeEMb2gms49cp22D8RYa7AcAjjGStp1LZREtyo3i6ZMHkQ2hkuNjI//6t//V2mr20ffce3Bp+cLlSwR0nDx5jDzyex++60tf/uyF85d+4hM/DkEEQ/Gtt1/b2EieOXV0dHz0+s0bqACnz5yG9P/hf/lDaAvValdXV7/61F+/+KLlgx94fGnpIGSXA50MIOJUKulyB6vVEqEING1/L6OEg05/YWE5tX9ze3MN5MKAz7G9t5NO74Vw63aa+Uw5EInORiIHTp7cS+UuXLmCbjAyMXX11uYjj70/W80W6n1X0zJ91725ve3+8NBd3/6RL37xKwxSqlp3rm/OnThZu3HNk0h853s/XC1ma6AE+VxUh915822FQLhc2XLy2ReeP3P8JHa9T/9/nzuxOEQN+q3tS05P0mav2h1eq92PFbeBExja0cGd3Q8OzfS68Y2tajQ6aXMBArDfbe/+wN992Gl79q8+3ZykKLwjsb+bnpocJnMpRMk0kqdZApLyCDchNAEJa2AmVBYQ8wLPI41bAmEf7EWKCTh3dzfX1rdxsWGFcM5MQJRw9JVKfWoHTUyMzc8eZptQW4jlOjExQbAwals4Erz7zOnLVy9DvEAzQOJhjRDBTvT4wsISVm4EwCtXL0H9AAFkhYJpn2lTzaESCUXxwPFwgzGBikGknyJNpQpa7ZlMDrgmbOD9fJ4SES5/eHpuGa/qzuZKKbNfzqZupbfvPXMsEvJNjY2Wy8WLF27kS+WFxQMkJiXLGTx+JHhtbG2C/88aJhjw/Ooad2bdkvvEQmXR8grfxWGMhRaWDP/GfGL9Dw/PE7FtbIxa7HjMscl6yFAhIc9YRyFfjIJhZgJCIhvHBG9gcUIAZXNgIgayBteMhz0mxoFkSnYvxlasppAN1rLTiYkcUkyZvGIBrLkKLmjC6GmQuYlD5oBOjwGh+ubo5ARukAHuI4zHg2eb+gdOF/59F8lTgbDD40UJxpEIOeihCtsjsEAFEHdqODHRS0npKhVySEzsE4xypVKFkHh8osj1DJyf2hsOS3B4mHqEWxvrwFMT9gZQ1+HlBQLXQVBp1+vBgI+yz4yJ8knJNiQsDB8RegKGC9QKhcY4Ce9jkyv0hCK6ftxhPrpD3E8qmScIvUfVqnAUrkZ108/95Wf3CDUXRhUWRVQLWWRIPGVnQoetFlV/0gKFsIltQIzFaoC3ZJ74CW8Ud00RHqhXEw8YeDukiMfGRyfGxibAi3GRIInhvVwHAhc6B0UO+kMsKTRC1A3GHPmRtnMTai+i3PaQDriaKADsDEpQq4n7YqTmsQT1tAiyEp8SH4U20jRkDLskWqNDimTSVL1CVzvWAB4A+C3jxB+6FX+C9cXBgoM3V2qVupQzc/e9QU9iJBGetoWOE0y3g0OGkG1XgGrrxExWwR9QuWFUZTsRBOhLNbYo4gK6O5BeCtge8E/xGJ4DCe80K1UsSrSfbAN2Nu1RbJ682BhQad5tRmt+qPe372AY8KD972bAt3vEP+b4Vh8HPf0WnxUX+V8e/O5/eZ5x0SHmq4OrBvIBOxO+yyDDIAcP5V9dAX0S4VJHkb3oElb8RoNwXNYPjHag6YoBY0/WoJPCa5Rg3XcwZ7gXEDqlFutbTYmy8ihkgbW35xAYKsGAyDiIsW6oeoUcrZYV4ou90cQ893CgMPmYN1jrmUaPZB/xyx5eSWIDwWS2sfiR5DjAcsUa4zX2agxEKC75TAbXj+5PSilER/F6pIUJgRBBSqFykt7UFRLHeKQiLvCEYdIh50JcVrHTLHsECS4VA8YeNQCINa591DTOQ854RSeG5zFgUFPiXLBpI4LzKuNtPEZyAiYjMnk8Xtv4+BgmnsRQjKEul+oUdrv77ntRxbb28TuSYWx1+awTUyO76bWhhK/VrmIqoyb88SOnVlbWz7915cCBJQzIRXxLxTKbnZGEjLITAOiw4wv3gZfeIcXz8cceeenFV9bW9mgGOAKIiFu7WXAKHnrswUceexAcxkOHpqZ9oY385XK19OlPf/LwsUMoEWPT47vpJLbGUHwYQC8/iUG19tWbqyDwX72WbDbjx4/fNzke7bRyBxbGo5Av4G+AkinUQB08dPDo17/xzPnzF8/cc/fVq5enp8aIQqyUC4zA+tbW+PTMfQ8+xABurG7srm9OksfDPscEUmkOB0Kzo6PW8o1Q4/WYx+IPxrtAF9i7oTGvM9IhOcfpIXXYe+lsBsCv7/veD46OE151sVpOjYQPWgrxz/3plac+v9chPIZoYgeJkCBAVQmhA+SQ4Lp+h1IHDkD3SC6j9CsJR+gqjLyw84kDx+xKumYkGomSrVvN5POjI+NzC4sQrp39fVJAKF8MsQkSYDU0Ct01aqt9aGiY1UA9XTYMU18EvY8yR/UKUcOIXj2QLl2U/otXa83t7d10Lv3pT//p0HDU54fLClKCdUWODHuNVY0aTpFyRNRo2K/oBasV4MRKtekHIsIToFKa1QUmRfTAkZNHDx35/X/3u4V05tDStN9tiwY8x44fygLhRJ3py9effuFFahdRiOjwoUP333/v0tzs+YsX4I84ZuFulEuitXScpCF0262NTUYAqOb19XV0X9KIWMMcBKRJKhXYMjSXcG1WvgRtQQ0HUaF9hFv0W7VmtVOFuqEfo5jBO2C8wJbAgIFGbPYLCLNwIEZa+8WLMIUAjOLqxQgLvLPCMdKpIJBumHOr1YDX266AZZGD2NCCQSgM1LxtB3sGP0QZU3aA0GtsqNTEqvvgbzA54Kz6nSqkGVRJWRkxOMstiuWrSeYVbW8DeQwERJfif/i8855+gh9Dmqv5Mgy4Q9HDnoWiART4YvzblTxKYSG1zUstnyHs/o0X12DZPBOCYlWEPI5K1o22MwCSjB1bBUGEL1EHXRD6NmIjw4GhF7+aeoHiQAUCYrAoXMEEMDIjcXwDjrGRUSFhmQrKUCAUH8YNNoGaj1AIcJCYhtiZAnIglKLV2OV406VsAKZihxWoXBBocavaPJVCa2764IkTp0aGRiBA5XK1Xesi6wQ9YewUgLwgBBCuyi2QAXHHgKELMaDcB9gxMFgMLoVcFrGjVixyR2FD4x6We0bsuF1rhH0R+Qv7Kp0oTgDDkLJFaTNsI4aP8d1A/WXdoNzQVNi7uC+GTIYNaz6VHFp2L8byfMcLyo8nNhaJj8ZDE0OWcfp4ue+pgIOGKqVAGzICXW2s3lQ/BXeEGhM0UnGT4j7ynCihWawFaRBbCjZ2/qDe7VaVMsOyuIrGY6eEWJsBNONryDzDeIcNa0QNI2QeeT+wPxhBghMabzrGO74037/zXh021xvWOPiBLr7N3fX2W8fgPt/6PHjHGuG+AybLD9kW7DW+ou28midqBfHGaIOSeug5l/AbhpXveOVSM8QaZTrC9eo6bxQooDWDesHs8FskMFl00Z6ZOEr5Mis4hvsOoveYHnzCsHbGHtDrOuVEu5YmS9EdxmaaSRd5BK4a8ocIwkYTQBslSK+ETcXthMqQJ26nSlomQ2YR1VeQVlmbLnyTZHBSppOQLjgntE1mESoYKZwWtxY6q+LO6R1DIPWIEdVbplRcXjxfI6IdwRQgfCk6UFERbpBIaQCbALHXOH15Q5eJ7uP8IJ5RMRKIhZACuzPg8WLUo4p7sVI+c/o4g/jSy2/QhHA4gDFma30zFPZT2hmGTcrMUDRy9q23SRa658y95y68VSoWsKqDxT89Nzw3Mzu/MM1WKhaoZt14+IGH7zl9/6uvvOny+Es71K1Ba+9PgGczMc3a//pTXwHfFtgG8Pep0/DmG3L4RRMV7Sqnd28/PTkZo7OUfz924niKTKRS5WJ2c3wknMlmd/crU/M1rNmZbBE5pN7oRJ1I1iNb4C4mc2QXelyRuUlfpxo6ubAwN4vWlXxg4sSnnv8vm6u3jh87gsHfHwjGQ3HCh9dvbFOLDNHkxBPv+fxXvnTp2o4vAKDv0onTj5QrhF6yIX0nTj3y3NefnoolphPDBw9P5Xd2376yfuut1z9wyrfTqkXjYZsnVO41xu0j/r5zZXM9lWql9kpvvpQDjWw0lrrrdGRoPDgSSzTKq+1G6qN/9wMux6t//IdXw1EKDvooxSK5CpKhqcXKBywgdU4xByiPFQqGiMgkAgbMWsDOSpMAOCCeJpfbHxsdn5udYoEL1shpIW2s3Yok91LdrY3GfJWMbaY4R0HfSgXeBGMiq4fIJvCTCd9Zu3Xz7gfuRqePRkYBT0vuFwj3mZtdWF46iDFjc2sFvtuCAzusxHyhN7GSyMPEqsbKYRMAZinpDtpKpTQL6lmZCLNIYjQ6NLqfKWK+vn79Chli6HK4nz1jcaAvMHuQ0jY5Eb7/Pe/tOLyvnz2Pya7Zs12+eotdVql3SFsJYpTt4hCJwPcHJJooomqtgWOYDUtpB9Ys61xh25WqowrOLnAEioA1Tk3ifVB2epZIOAg4FNVn4QBEMeDx5qYNYI+AaJffSdIpPBiOCylFQG6WqpSOpKiRXIu9GlGCLFa2rjsQZuMPBwJTE1MFPMCVOgZl3N2AbA1YDQZSKDi7TxBJHWt2q25J1OztKBHgRCWQ1NNvBnFjWF1ufNYCiINrsV+ldwaJ7fd4G4RWoapawNEu5LAhk5Cuig61DPID5BVjO1GdOGoR05GhnT1Xam+nmtmB4RSp6Utij7WBGVg3dFtRFkkWK6dl0YNYEuJJzQbcw1ANVjzmLWL7YG/tDmkF5INjFRbCtLVDihGNJ/GshYk8mdoJBYLlbD7mD0LxopFINQ7CI+NH8ikECQuMViIOEVRi0uDELmRVgHSR7wFtBr6KPJp+yBcJeP3SQ3GIWz3RUGIkMnrq8XupKA8BK6QqrCMuZ/0Q5l3Il5wWL3oK02NShiCysG4bLcHywGWNYh4MPFgvtB0kTncEiEq2BebpKiY/6hCphiBA70Q/8kTZMgg6ldIi1sB0G0aldoreMx66hDdKPkIHQxsZcF8smrgLXLViLdtx1+MT4eUTU575UdwIlka+20y1fWmLs6aoe7gJMoKwtZlnUgqreBgALofogtoGsiVPJbiOJ4k1aScxOajstys5MjtQYaCTFBpL/8SQWEdgzdJGKU8c4kJ3DpF+c4jXGXbLJ/NeZ3kzON45ycd33ps3vAyO/yX3vfPl//TvQPaXCCEGC32ifwpy0oViTIYzGSe6+BHzaqzTCKaQCQKOxJvZRyxTiUHiuNINGXFkIrFePOBSylkkSLNcwA00DKxJzBN4ak39J3LNDAOWtERoDI4N0kTgs9U+he483lDcEx45NBdm07LfKWHbLJeAY5Vpm3SmRhm+iHsiHPADztKw9tLZLMolo60JIJFX4VVaAlA3mlwGOB1GLypr5gfpFOMRIhnJhAw7Q4+kxAHPVYgB77BpI1DQOx0QLEYBQZdXM4PmWkWGyxDNsJCRzyt2aQ2RQsWlLmMrImYU0COuhq59/Ed/LJlOvfTCG1Dm3f39sSmFRhMu65Odx17IlnC7kcWxtrpardeoA+hqOu++9/TyofnrNy7Yek5CP6qFFnhGYX+CJ6OrI4/furXOyI+NT+KYPHLkGEoMqvCBhcX/93d+l5IzS8uz8GyiKEg+xBgO5AKtAtP/xOnTb50/d/369QuXLpWqOYJTA+4W9Xzm5hZ/9Vd/6a0Lb4Lxg+sX2xTgI2+9fdnlDoDp7vNSzhZkpR0/rjJ36OrrL+6v+o8cXf7N//brr772wtT42Ic++H4gZS6eJ0+vwC5++O6DN29cI9l9OOIvpLMsg6nJmQ998MPLB49+6s8/A5WuFMvvffzbYqGRp7/81cjQxLG7HvyH//mn3v/YY6cf+cD0VHzl0tVcc8hmD97YXHt7LWd1d6+vbMajMyPRRQre5lLrX/n8W8WM49jxzj33D1OLNTjs7pUufdt3n/L64//1D16sly1jo7FmFcJIdCZLkD9wrygGylygADBf1IodiFgwBBu8BK7BZ3ItQd5YXPQSYLOyukr2ECUl4CrRkL/fjlWJeC2XU8kkuA5EaaGzMrNXL11CKSS+qV4tpgv5sdFh1FFCooAQgYahXwJmKdpYLX7/933/n/7Zn2zvbuKWxfg3Nh4icRxfGplIwDgQW8NCIyyLeWTnoLRMzy+sb+/QMoLnjxw/89a5S4cPLWNfA3x7aWaG4BissUcOHgF1ay+Z7lgdx+598Jf+0f/9pa998wJFPM6dA37K7Q9TJeHmzStTU2jYqufBzXl0LpPFVsTKxCSJ6GBMX/29/X0c3iSrOCrgBCETGusdW4ZNQ7QEYigGfXyZWDU5iWkJQ584BCFXNfBz2RtK7lJcCMZGTKlIoWTjgUncBf2M2Dl5iThYujyVjxSIrmWwQjv9NideV8CfyYU1dE9eJ3QdCDwTKBmZ7JpstlkpszfB5JH0TUQ5CE1kBCI1E2hFiY1QxOL1gcrW7uxEwon1UgU7KonwgF8on6GYaeQL+9lt9jRmOprK5gWrC1Qy/s3m6z7yKzEPdds+Zwt7DdigRAugbZFlz89RBdFmRStIJiezAmcI/Mhqhf0LxAF3lMfVb9uNIQDa18MKigscyQOS0agx6g1yYnc2Ngm33t3chhySpAQCC8bqTCbbFAwIiizKOaRM+N1VAil9QaxywJMGfACNBaBewFMC1EL+JMU60ROH4+MHFg/Pz8yGfNFquobpjOl0OOhSgDfSauqdsegwdII4OkQRTJbYnXCyEDdGWkkuk8pnskyFSkATg0bUUq1CTiLqOuqtSCUql9QVBW6hcqiKI55YWi5Bkf0rcGcRXB4rH554spiyMq86bTempha6Vl95vc2WrVrp5WrN/MKxifB4YmgqZI3gBb0BVgdmEbuPG1YoFo72AAGFdYr+IiqQNMzS45FNoGCxcUrvl34EWa+pOKpUK4AbJKqh9yG7wUQUXsyPJS0q9Jr4cpya5JfyPaOiX/Ni3mktquk6BdX+G+z29jVmLeriO4e5+M4HrfzBe/7hT+xzcME7l0GjWRryFBi2gLBF2ABsg1wyfkvnCGsT7zU+YBgnzEZ3ZCx1SAmG29I2DTKSE/DdLtR8HJxiTmxapLU7Wq8uhvvCZdF0MdN4QYJgIgCXtzlhaWxeLH2lEtZAF6wT9oc1BSYuwYkUTebT7iw1yF1xthzeNy7vPP4dJ9/70e89egJNInHhwvnXX3757Msv3bp6hW3CpG/s5Camh3ygFHhdrHPQ1KEFuE4jQYKzsljvEGr9AX8kGNxYXWNlkRKojU2XzW6hO6QaEXBNg8X+JYhITsAsjSNHcHFug2eOjbAphxnQDcinXMKjNTzQhcFEIpswhdjJWLFYtnFAibxo6jW8mh7MAEqMJjbnuWe+uZtKsoJYM1R5o8YWUYfDsRGKi5Tyufc+8eSVqzdurW568GY5vadPnIQUX71+cW11JRhBabMQMRv4/zn7DzDLrrPOFz51co51TuVc1dW5W1KrlXN0tnHAGJjBHphhYIYhmbkzDMNlgA+YIVxyNphkbMBBtmRZVg4tdbfUrc6pck4n51BV9/dfu7olw73fM8/dKu3eZ4e1117rXW8OIWrJFG6/7e6LFy9965nnoQ2FUiWV6lheWIeCPPbw+3CfOXf+zIH9h370R37o85//0zdPzvT0ut73nsfi8Sim9Eq5Nje/tJGrti8v/umf/+mv/M9fOnX6ZGd3ElLU393dnxq7NHsJoTOZ7EFZ2ts3cuna1fXsjCcUOXv+rN0VTGfKP/offmJ7K3T6+Juh1sbYYN/K6tSXL76CjHL4wFj/wMCJk8fAsOTJW2/UQh7Px7/r/V/4/EZnMvKNf/zb9Gqtt8s5MtS7vrxA4AkV85iJQDiMDuONN986dOvRYCz+1aef6R4dw159+y0POcve2x//aCLV89qbx6dOFR589OGnvvX1QKTnez71g3/5J380M7OSjGz3dds2Vlp3/eCuNvsK9bJBW/WtKUp7P/g9d7Ji/+T3TqKSdduCzG/IT+KwbK3UIvyKxMM4nCB0WPMI/LM6cJ1hGsuVAvkKr11baY9DFLfn56ahMso0WEGEBQsh/gUJBUZMYYnnMuupri7QAkCP3uX85LVatZxIxJAGyXpbbRShnaRemJpeikd7gqE4LreEdYWjoYceeuj8xbMkMIFfzOZKD9x/9OSJ4xh0SLeE8o8cD6BuwJJ3QQuBQ+KC9u0/ANrcMzaKbHr+7KmzZy5iewbPrizNr6T9uePUGI6kwmHE2VgyhbGuo6v3x9/z/r/+q79dmJ9fWM00SQU+dRmKhvKZjyXWaH19HZ4PSozqBT94tNNLlDW0twVCQfYQTSfJnwFy9LngNAPKchIBP2xkS1ioSQ5Cci0cjRkfkB00UbH3SEPy8WANsQclGyEJzhXJDc0lP6UONBhLsdnyoSVHPgF/VNXRihIXj1QMKrdYd7H0SH/CpqA2pCIhd+xWdFCdglDAAigKGc4Fy30IBX0QNyIWOe3US+volKnd42nWbJDQaqmWybZKBXgZrXQEArCVIkkdVUU5CBEa+xH4CEsK7tYkcUNhhjKLnhJLSjF2UIU+VdIEiALQMCtfLmqSUAzqJZu0rcRIYbdGo0hPRT84cpF/II0kQt4XkslJ+w2l37arPoSwMRmckJSlccOxmgg3UAkDQk4S2h0ZGN5uUXKPDNLNaCDeKLVIwZ3s6OhMdvX3DLXHU8jB63PZ9lBSpkO+zcRjgOVB+/CQxWwWEYEikiThsgHlfDtxZOXaxuoiGZwRtxE4YC8AaSgULAD+MqIGUH7NphSYwmic3VSAIzOpzBeiWeThgTvZQmaWTbqJcoksC/pk7LPb9mbBnndEoNHbWRL2VVe8MfuugwOd+47YHLhI1m0BsuoTO0pO04aNXDdOZCkhUg0v69USpoWVgUCcJ7XIpJoE+UJuJNw0WavsGWqJ+2BwCLdkKAvjilxZPWeOdIAyggni6Ds3PoONc9YBhzd+cs661zpj3fOdT3/HL3PbO+0Lnk3LuE1wHwDJT07KVmEuSXNvNogu1+kcQC7OQU+ZXvNNbMqrDW8Dyyijl3b6EkBFqmrGyRBd2ucODRv3YMCgTXIUQIaRAchBgC0Bt9gaPGR9OxDpwimUnD7UKgCpiUsRj4Ikgv1ns9BsyxNv4IC9tF2cng+cPLWMb3o2//JLLy3NzeIGR3o2HH9XFjZQaYHH04UcqangsnF6Ajb4KGIB4u2x9No6vFK1mF9Z3hro64cFJCGPlo+4D9aa7NzkWOCtMEfKg8Ohph/vEpFOvk9fYcBZ0wZmMXOjU2ayrAMAUjBpZGeg48aYc8DGKHIbxJs9QIJT6vz8fB51IleI1Kg321PtqHquXr6G/w6GQFRiD9x3f3rjaz5fi6KwVy5eUaafxibezaCT6Stzc/OLxKGOje1aX/nm8gr6Mgr0unDwId7qtqN3ESHzzW9+Cwvfgw/e39vV9fjjj09OXjx+/BjnL1++TJbgTD4H4uru7XH7c5TS+9u/+3yqK9nmbHX0tJeLWZjmSzPT1HevrWdHRvcuvfHqa8dPYm6bnF3w+fOZbAXQwPpJ4i2vK4y3xnAv2sNWqEGmvQhB0OCppbV5qiu62tyjgyOUw2WZzC/NxVPhWq04MbGWiNr6+9q7O8IYGdaX1ojpL5aqj9/yAb/NTZL/7oHuvo6Orz3xj93t7cH2EAESfZ2H3dvhp9949fY7H7AluiYmp4/e/V3dHZ0nTl4bHBir52YTwc2ujuonP76rDfS6UXCHba3Ssi8asjUytsa5Rz5xxB/q/O1f/TomspA/XMmvJjuCoWH73FRhbDC0vIT0KTAXksSDFlhBRkZLZW+jcACJ85UkuopZPQSuw301lyXZp292egINS6WQh8uSblU+nPJZQcrDm1QmQtHQKgn2i7USXtOZ1XVS/q2tl9Nr+YOHbiXhKO5dVyevJmLhkaFh5Oaunu6rE5PPvfAKoWhQGogT8IWnATCDyQBNBsQPtyEiPnjpcrN5/PVXLl2ZrDYJpi18z6c+fvPhW37x5392cvJqbw9JlSJLK8uHjtz2yqvHQsnORGfq7PmLA8MjH/vuT554/Y2LZ96k5QsXLhBuRF0iNkg7OAEqzh74LFUrOAp2BIL4TldJ1EySQbJWGRwBUhcNATHDaEAk+S1rDL/cUAuUmaBnUR/1HUzJ+pFjJ8ItQM4QS0NpdKusKxbXDgGG9FroBY9VyDN0yagLhf6wKaJCMxTNwsgGlVpZkeHsUZNDJJQgApKlLL7+VDsL2uf0BKhFJP1zWBiBexzbmcW5hdmZwkbBQk54Tbma5EyTbMFX8OFWWnnYZXhznMWEAlirEDR003iISpZCvJPyDsabRvhf2NwQ4E2V2oH1lqFcLk4gPmJkoRbI7fi2NGE+yKKKw4lwAWJQrVjdIhYcp21MUpsVmiS9B1ma0+UcY6AIIizWdifehhTPpgtoJ0LuIAdbFdv05HQ8nAx6IiFX1B33jo/sGx0c97mCdfiKDUVMxYIplAywEDiZk5IIPEnGBIk3mK1FiXHH5oOpWVSjqhQ2LbJIUsEKE69hZiQsi9YxZrKvguok2RvyIVQPWuQXpYOwsMNySRjGd4YVoK2RyxeUezlEoUUMDDlcCf2wEGF0EvnV5hqB0dFU8MDYcMdouy2Os17JFmCwS+TMg6kgHg/7i2Ra3oN2EtQr0i6RWiQIKMH7kET/UAkUz9AQsmsTlVcnLyEZRPDv4E5RKXqL+wF7SeH8o1O0JnCiDcCQPRs/9Q+bQFP/6H8Luetg5wbrNuveG4/cOKBp8+w/33EDA8VZ6/26bDqC8CtJ3WhQLYLBTciq0hXBg2r1qW3dKwUS5hpsOuJEjAsV1xh/NoXhquwzyh9j30XegxsBJI28y3SoSR5hLYpLwUUcRxe8BMkuAD/nCGzjOI7bXdO+tFrGMECtSZLRsxoptI27APkG8O9H+eSNJLkdB0CHv/j6W2fPTi9uO3CPD2IkonXUJAQ48DInhTpxMYVcoHImHSlpbcJYRQil8EOJfSSUJ4uzChzJ6sGw4HzBd4JJ5LGI6oavBmHgviVLNcuZSHj5MEsSRrFrBoV1yK3mjOaYMWKD6WDla8h4XuAqFTQDbp1nYJlv9tBkzvMONQDvyPLb3qaCAjoArpH4AZNPe6Jr6sLcnsP9KAimJ5f279/15om3vvXUs+hTyNFDn91tFBggA2KJPDzo3/GD+PB7P7RrfPxn//v/RMQ4dPP+f/OZf/sbv/RHteoVpqc91v7x7/o4otLFixepfPPqK6/cc88tML74Qx4+cPAP/+ivBV0Mq9/23ve9/7Zk+1tvv3XLnTf7Iu5XXn9+ZuEyYtbzL74MNsBVHJ/qbzz1rQtXLx65/VYqSXRWGidOXuXZW2/d7/PG/vgP/2z3+AFSgmzMn0JmdQRwU98iGjRTKDoJioyGCNVMo3X3+SJUZqvkyS6br2fjnUrtmYjgGVdFaiF7Sjxku3gZluLiS/ZaOju3np6anT4zNX85EBy9PFclc0Qg1N2srY7f2TdduDB208C3XnsyFe1//5HHvW215amVSl+bry3/0EN9qSGvLTcZROMWcKWXm4muCunyy+uXA8nkXR84lMs2v/Q3L3uDCfw+PUGUZ/mxPe7ZieLwoC9HNhYwCuY6aUS0FviPgcXhxOenDWEZ/ohSA2/BM4GKlxZnYTFRkOEyBCwBKwSEUrMOHEn4JJlE8bbCKmrfirYyjcw68djroK1YpGt5NZtKtN98cN/YcP/bp9+AdOBm1ebo3r1nPzzNt599gYAJw7/iBIbXFupMSDBGnGYhS1ZE8md5SHgBYE9cugCOGhvoP7h3D5gQh21qGcF3eryshra19Q0KHC2sZ4Lt1NqNXLk2TWwRcAgHds/tt5y+ad9X/uGLWG8pCEFcXDKZlOpuc3tsbJykK9XaFewURCsBvUhNHeEI4AuK5RtZMUTeCmGBhPmNvYSVjsCE+IOfoHAOxApsD9I2aI2BAUeBSoxoLKGXH8hXOsO/OuaEftM4zYo82xTuwEk2g4+EkyAJXDWbziOkSxcnHANjzGQRRIh5u6bMkm4nTsXkqPOHS1vtDfLKkMXEhpUEA9Daajq94ZUGi8rGJFQjwbFSV6ojSk/Ay/kWBDDMXyRLM5IK1BesojcYizZfCMZi6YLsDbrX+82D9I2PgkDgS0BnIQe0iDWSXOKIH+ABdH1VF3UysVSCa0iDi05VVbXAUOjAwTP5NIQQLMbjlBRGUiGxPINKhUgSb27ZGyQd8CSCqYK/tntwDwQY3/H944d87pALd6YaMcoytvNW6BOtCd2IUwExi+LQfbiiZHsM9FotZqvrC4TFSlVo5HtqOWOnBoUx0hpXnhYx4rckMtEsRkj/8/3sUADpJzyUNHr0T3hVABJKeEjlTUUUfJXd7fBszUxpdSO7YuvcCnR7hvvQrCXtqaDN18CRHy9Rgssk+EJ9cZKkzBkUFgYENQmFXKlIoZVmKJy1B0CodG0VaDT5I8VcyAmcwHkMAqK4rEgzEaJgRs0shQ2bwXg6YtJk+4BF4l+z8bXWAXvrUODGne/ai16YzTp546p18p/tTfuMuE5b9zNy1j1QX5qyfnLJEoV5D3m++cnBzp+R9rgt4A/AOcGGW/eK9zNLApDUN9ko7iYazE9llTNqPDlVSaEkk7uoDo+wdslooQothCEEa5suoKm25a40bYv5AgMJeYqGHTCsBNZSZYiAQlYFXp3lyiYJ9MpbFEh34Tu1VURFgTulpGW8mYJh3PgbuTSaNoIleY2NkENcfvgMnF/aOGXbJLkrqIrEtqUieYO9kVDw2uUrPMA8iZWVCpl4RrFv4voUfIL6DHUbii5yt+IX2yL8E9iCUbbGiusaQEBN/wp58G2MrvTSiieU7xVry9BvXaBljjkvuQC9qMFaQho2imQXZJ7x+XF1WV1Jj+4bufT2JNlDBgdjKIfj4XaC7zqTCZI24XcDs1AulvKr6f27x/k79vprzz39bHotd9P+XXfceff0/BwLOZyC0vnyxcI3vv41FKQPP/zwQw/eS4g/FvDZ2ekDB/avrSy++OLL2KDzJdvqgu3j33dUpV3tVE9afu75b8GA3n7vkaXVmQMHDy741q9cnChXSxOzM3NLy4TWUGju3Kk3ScI1NNrZajrwkh0dCT78wMN79xx4+sl/6O4MZksr6XyOgcO/ldwMRDCQsGF0aE+liN8pFq6CUvESe9rmiCYiMQL5g75Cbj1DEEKxRrHcgZ7gxsoC3HJHe4jERVevvj000t7mqKCMXV6/1tk54g1EZtYncdZ99pWnTr99orj2+pljJ8b7Uo7GfH9X6OjB2/ft8xYmjxPLH99F3g57wEcFMhKIrmOHy60di/i23/eD98GF/NkfPTHY017HpuayZ/KNwSGi4JqYgZkgJhGUwWwCtEwcmk00GrGwLRGLwJVhGSArBSiBClSsYsAakyGbdCjwqW47tWFx6Gk1KqQBRpJGmYf3aLy7h8RasHq4tl2+PBHyhkkgMTt15dzZDkaBILWNtflAOAIEoa5AMu7qHvqzP/njcm6NdKT0I+EKY8WTLA3acbVhTob0gGnIcrG2NB+KxKggi9rv7OkTk9cuEWQ7OtgHqaEQERrYQn4dS8ctNx96+9ylZCKUz5efefobd992GwnGBjriPpfjpZdeQn5YWVkBtu+55x400mRnYwj4dlyxuMRYsHGAQRLUwEKAbBmNu7SLjA6ZYHHthWSy1KUGQ3kqxpswFYwkwjPCdvqH0ywUkCJtgwNFj3WSUxaG1WmWlpAKeIPlqSpUwqYMozYuWrhMrriMvgKhRCckvaCTpnF0LHRhpbiE3AkNZtkQU4SRBg07bg6trdLG+go2TiRTxEBEJ7JhoArD0CXSIV5C9EavkRsI+ID3CRBEmPWbvqp3+lIWP6pxgyu4XZYz0GVD5AmSKWkbKU5aEagBiSYqFG0ikz+fA5dvPgocQfIQoRBul2CN+atB7BeMa1mZxEnvJ2EeJ0BqjHsi3giZtrcbbgoTpzqTn/rI91EtnPyRYAfRegl+sAuIoyThRGcMYwH93SQHL9Fp2ETpNqml0Hxi2ZULXYs0mGgNySabxV+Jd7MMnF5cEtBnwBbwqfo2Q8X4NvgIDQkjYTbouTAkA0XvQX/4fTHLGhw+RpwTJRHJPlvfxqC2RXKS4rar6e5whML2jl2xRH/Y3dMhN6vSIoJ3m6fpCTEmdZeD/FkgUhpHCyG1Cup3MbmMi/ohuAIojIMv9ZYqzB3cInIM3RNkaN4QUBhVoVpBFeBlpoorYjxM17U3k6xL1uQZU6u5UxfNzULqHOulnBHfcv14hxHR7x1w1IFQ///jRh/o/Y1LFgDzE3EdOgEaohGYBynMNdqK4rWasvYgI00FXvlyahScs9cfukWmSYyi3s6o0At6K22zjLh8sAp8sZ6Req0bBMJgZZS93Of2ww0urBVW883ylofCE8trxXAkPNTV3zs4GPC4M+sra/MLG+ks8uRGjpB6O9F5hAf4wvFwuEVcPwZ5aoYDSih2WImsULRznal29ElQKb7L4/eDJSHSpSqKQ1HUEL4ZgRDp28GfhVIpQ3LWrW1c7plXIAlZA2TAmpKOwqnSkXwb3Wev9KT6eoEfy8VMiuZXy114QtMJO8IZUIVGFTsQbtIiy5oVUBQLm3t0HZxAi1wx1UEQtdAOwdXoH6eTGNAPfNfHcIpdXpx77tlvkvaBpzdWM6FABGccwIAc+27nNIHW6Hviochgd+/L5BjftJ18/fjRO+9YmJufunaNoniIZXhKU5IWWRka/Nabr4+SG6mLpNHJ1bX5ickrsVhkYmIJD2vSI4eitme/fWJ0vGc1s4Hf7O333bLrwMCxE89ioP/835x734OffOqZY/sP9fvW1t7zvvdembr89Lef7h8cwLOVwZ2fz+IrNHHtSneqcmD3vnvvueOt01/GGuAjsSJYBgVQExOpLxYj10jPZhAvu80zJ99sjxKaiF0pkMus8lFEdcUTlI0PEJYYiafe/949b546jTrU62yrFjIBch5SRLSQjftJ3vnWTYdvf+vtU2+dOvntJ9/6mZ/4TwQKXzw59dH3vH+sO+5ozl8++5X+jt7Fi6c74imnv2RbqTScdW8y2GqWqIvox/nVWyuUzkYatkc+NhpJvP/Xf+kbBOygKHe7W9ncViRIsAxaT82tNiwFLAmcNQi/rVFgCpxE2jYSGFZxlRJFMHIC3tCQOn5iFkTtj7YIDomxpAgsxnis9Ll6ViHFyU4g7ZbDNxGBtrSA190SSfrLxY23TrwCKcW5wO8HZ2+RmymdyQC/tx65/e/+9u+HR0emJ2qoBrEyANUkvwLmE7GolGooUZoNr53yabC6jfXF+fXVNQKCoeleUsE2ijWVlHWM7drrgj9ulSJxar3XB7rbC4StUtqhsP7SN59AvdONYeKxxwhUm19YIAsHG/mw4KswT4AWkIkVhRUOExOMTEygKyILlj+D8pDNMKWgdoLCSFXJkocYYsXiX5PK0ElHhMjehVMMjkOJQEIsdFq6tLNZx0A8pJhjFqOF0reoM8hbRIB5syi2FQFrEeDNMvELvEM6CGwFCK7gAhCmtFlENyAB2LeK+VphLbfmXGJ9MjW41KHbwYqN4Qf9GpPNyJITmhk34h4vUR9AbhBFE8RirV6xAeqkuoGkQTesQVBv9awkTLrPOmfNw4uRrwTmDDoulQpVf+R3ig4bn3ChShhe+uzA4RDwwksIJbPRZ5Ajgsz1QQKE+H5YOinNWQHynCoRJlXN2bHrD6Z2u1pBjLaEo1XzTWxSGF7JXU6+SSg2OK/aKlP4JICtt1yiSzgdIF4Tsc2obuIZXCmT6oSplM/hFqUu0ERo2mCuZE/Eomhsvww7G321hC3mSZ9v9nw+KwFCwJfxEJoBWtaMYOaHN7PX8qVVCnBt+4kpzlXt+VDS27+7vx2FcxTho2DbnNqq1Lbd2/K6IA0staVBjJo3Y8xEiSJtIfVLySqq4eYVYgKQvTD0ypdrEw9Gpcoyoc/MCv3EoI7uHFQNnpWdAxSt+ZJ3OkQdckXnrY1po8P0n/cxcZy8sTcHYvXA0eZmCxQ17/9vm2lAffyXG/2mE6KMZrvxFn5Be9ib4RWxh9LwEwBDZ6EBtl4Lm4OMaeAKWgbooDHhTmiv1U9RJHHDzIlO7Nh6WTaiypiAAD9ChPh0qJEh0lpRgApOzXDM2yvpytSarWZrFqha7w2vVbaW37507K3zvratgJsULohPnvRahoFntQSjiXI6C2wwlJi+GPhIJBzp7GTSxQDZtoN+78DAQCScwLSZz+fxZUSf7Q0FYC9qaysDwwNYJCZnZukrM1gukqsoUMqXnTI96HNBtXROUAV/qDln4rTGrG8138ujIp7sdbOGyVzXKPCBiO8isNY95v4dWOUkDBLjZm28jqsMLKQev0AoL+8khpDYEvDX/NyyP5S4+chtZBC8eOFtVAiopslDSVkFZifgJRpGaeAwfXzjq9/8Ys0WT9j+1b/+fur2PPfii5euZUbGyMBL0GbwwolFLIL7D3UFEACxj5AyNws84k9dgfoiH8/OEsQr+4kXxZbdlognsT5+8IMfmJ2/0jkYeO/7Hz13+cTVq3NvnT09MtYJmwlQX5m6eviWg1dmLmdy66yEUCiY6sCBqHTpwjxV0hHODh3c//apL5HpMJZMLiKuTsxnMqSwSAyMDo8N7CG79rVLl8m6EXYRle1Ididj4djiwhxpU5RB3+UpowVrFKLh/Ff+8bnenujhg3sXJmeDoEoy2uNr5NzuG+ivlFZffOnrD9zzME46X/qrv/2ZH/u57GT6/KlXbx56bH1trlaYX5otBJwFciQ6h7vL1875E4GtIgE4rGacQ5rbzoIn4MiWT8dCraOPj/xE68E//53nu+NhpiQUdpNNhcWiFLwIL8C90noTGCOKDHjjHkNiQeQowAo3pVIFZbWWKXPOxyNEAe24K5OyD2AglQaClTDStoeb0Y+BKwCUqdDUkSNHb7v1lgvnL4FbiW9aW5vDLzqSiI/u3g0nEE8m3b7NPDVAmvN4EB3cu9vrsVFcFbsqGw5BeGVTnJ7SIhWZaYpQMcocgFm9pJGhng9yMLlX7VvXrlwY7JVLTjjoivqdq/n1C2+8GHA7o+2RvmS7Y9fQsddemU/n7rrrHtYLXCNQStpwfOZZO7fccguVpzlD3QigYmJqkldzT1dXlxPMB/iyvM2OcSK7BY6wgDFYT8woynqoLxEyaPER/MW4ahOSADVzpAOpcoWBONa2c8DggyvNh0KdROlYbTQvhIlEyRzovZBFNMNas/xBQXAm4ZDbcPEhU6dBaqpsxSCwsuQmZETx7dpmRVZMomS3bChleAiJHrMu1BdWivkzRJe5V6APGEzUgPeL/ZZ1Wx+hKRB2ZjPoQgc6L0QAxKhl6BHfA8KXgoDGUMHjjm0nnyruD9I58+nyQMWWLFWZi+g2G07nfBjjIf5OmTMJmCzia03mP/pvxkxeCfIw847vPrRr5EBnMlVM13LrFZW4dQawSwWjgTL1mtfXpAGDwfJ7wClkDZFEQFgviRBQ5eKCXS4Usxu4d9ZKeRAPWanYwzAp1T6YeZtEHLB0wk0aT2Eq83oODNqC47I+mLEXVYa6aPSFAZXYQhNIHXWmpBzvDmVbK2VHLtzjGd83Hh+I2vxwI7hhrrs9m04fcZyipZAedCTE6pG0U8SS1mEGEHyR4MmNRJOb6H+U20mb0Tajo4TNY95FJJl3Zf7nPwMDEG4eBZkbUBGBFU9ggZg+hvt1xVAtfRFf8C4abKCayyK9OxNrEc/rP0xDwJ6aYq/Z4RmOrp8xh+/aAQlQYD7IbFwQwrh+M0DFN/GTAw23qIKoDm3TS0MyObPTE/yJGWYG3FBf6JCoHk9omWgl0QhwJTgFDekpkBBudyhRlKODW7mNhSoKXCszjbiXkJ3ARrrmFjnrEJedfpIebLtqCD2AAvwQSw5P6kgiBc1Fm0pa10was1YhRPCEspN6IUvVfJ7lQ8SOo80D+7+0sBw92EGWSlSEnk2EsC2wFYQYaeHmW27NpNOLK6vMkC/gxQSFayRaTWwHGlWmnMGxGAfmxAySQugZPGZLQHZj3JgxFh9X+GiNkvCE2WiGfxkChpWFB18L0uKkRYAZLKIz2HMPPAFDgtqZyEuUR9wMYxHxenu6Qq+/8Moj7/nAa68eP33q3OBg7/m3Lg8MxRHlcYEuFcpI+QRQkIC2IxVAMvf7gMntL/z1X6d6OgGLhx7YzwwEo+Fkd+rQ4b3nzp1ZWlwmZsaNkL9dc7Y1ZueuojUAFdCrD37wruPHT2bzjVQyPreYYbKQgWYXJ0b3dYtb2KoNjXSDke8Z+eixc2//2V/88e694xuFFcTfdGaNyUepgHUwEU0N9Y8E74r3dA0m4uFiNn3PHXfVamslqh9kyk6y12w51mbSp4qnRzsOjA6OTb09HXEn7HUlNapmmqy7QsmRK7bSlFoni7DTe+jmA3t33/x9n6y+9Nyz6/O5Qno5GnDFg5Q5d+Fi1BWLzU9cunnvGHqbVrFCzfsnvvh3ndH2q+de/vjf/v2uftunPh72ulu9vY65yZWRVN5JfgeXk1gPUiw53KT4LZEj1xfOhuL2UvVcI7d8x0c+EHAmf/W/f/GWvePYWMlzRekaQIuJx2DBcmDuIKKoKWGsiFlhTpkpvFuCIYWQEWPGhzCnkFhgB8RJ6iTkMFAAIg36NC4BHQShQftxDQQhnzh+HBGWFA5E6yJfgbe2ai2SQjY3K929XQ53CFdbytShWrt27QIQTiAMSmBi7SolmM4qGlWIGn2IR2O5bIY/GAFKAEA1EViJXs0UyizEejmDCqdRzpKNo1HKVPLBWrHy3NNP3Hbn/XkbqYiA/aBnu9Hb1T4/M420Q9kEtOjZfO7y5SuAwa5duxC3eQuadQRiyJDlF81akxe0EI8BdtALo4CuA6UXmBBUhLYdUY1xqTXxA/D40DAJQWuNaHWYpSJiZRrQL/AOq4vWtDGGMOy4P2PAFd43Us8mReAlQULfWXMw8pAEMI6kHCE/6BkoyqBRCALcNBjZ0D2WghCahSLFF7MS6R8qWXA9phB4DlKL4YrCaiS5pLrEbKNhl9JTuXigDUgLfsO5sabBZqbb6jxvNvEjUBGdM2ek/ZPS1HwkL9MhGmH2pjHMcqxz2heyEWGQk7fs1BSihxNXX2H9veBQ0jCiG/a5Y6QJwUcbPs7nCeAd1094R6R3vO9mlDn5lRKw2BnuBCbQJdu822srK0BGMCKoqhM2VJbqPhDxUOYTnISrdJaQzUIOrhyeBf4o4EXbSz8YUGaP2O4tw3iSAI9u4LasD2M8NQhMrn6JhUKxKb2FpEcGnZkkbb7q+aFoZ8raiC+iPsVWrtZWgNx3jcd69x+ydXlszny1tYbvLeU9XC7YkTo4j6fJN625VE3IGqwlh8aRnbdpyGxV1AXoJzAMEeKHDKH8YkYHJaHWWlqgYY0qAKMJ00lsZBxyQAdpWrMDKKGXp0FOcwmkLZyua+wtaOR+s2lm9alm0+d+52Zd5RyXbhx/5y3f8YuXcaP8yfU6KZPZTNu8Wn6VyJE0xSewA1j5OinhxaOJtDIX3Md403fEUBrjf9YIoCJyC2UF/miOoRPzKduLjJvmNai2OOCPRvQsqgkcAXDRcnhzrYKj7i5sOfJVW4mQdfRi5EzOZcEgBMuxfMhSA3ii7MVZGpWEl0w4bZ54oiOSK9G1QNBXYZK3G2SMikQDTM36xirQSLrZ3fv35fJ57L4wfXjQFIslJA9WLo6p598+g7MxAk2xUuaTyXhKSbGuZIp8uKhQNKN8NtK6Fp3WAmsYLMy6Z3CkadH48NHACLowjb/FAXPAD0bVmit9qTAQQ7JJXRX+GBk2a5xMliThafgSxYDBcNQpF63oDiCuvT0ebe+YXlr/5pNP/uBn/tWTT/zT+urGrj3909MEuRJW5afaN+5kvm1U6yUS3RIGTWbmUjVXrNRWVzJUNpy4Nt0/NIitFt8H1E09PQN4RJPBi+rDxJVU/S3MsUOpzjvvuZNXfO+nvv+hBx9bR9WbzrGUzl++sJZeIavztStYfDfOnN1cWl+89ciRrdIbdx16wP9jnl/9X7966x275xanuvs6vX7f+joZ+TLxaMLjpiZbBGMMOcUCvs35hVncj9F3+X2uOBZTJzObpsr1s089e9NP3ham2PC2d2MFb9vQhQtXLl2tJHttcazToQA81cr6xlA2tzg3izPRTbtHvvwPf3fk4EFFLzVKsZCvt3Nvq9wc7+6b2944/trJ7FrhPY9/uLyRv3Ty+P7xzu99z1AqlOuI5hrpxbm8rb+fRBVNjG61XN4Xj9QJjC1XIx3Yg/HsxdN+bdvmi3fFN3PHD94+8iu//oO//T8/V8ps3Xxoz+r8pFe6BspNkmYHxYabjAGkmYaMktyFoLNaK0sOJpAluY1dLbIcsm5a1CwCCTCV6PYYzFAw6iDOBIckEmKCEMF3cHpU+AAlhkPHj78RjydgX6iHAaAiBgDDyUg4vbERijpq9YVKzR6LJtNrs8vz0xcvOsqVLAIbOZsR1OCuPH4POmcKK+FnqoUNhLo9kGolscGw4fGuLq/4PW3Dw53YcShtl93It7YWCd+lUm16eDGbu4JPWAeJ0CIBSvF6Q77Dt9yMEJzLp0mXisEYzRBGUkT5laUl8ofv20O9j33YrbEQ86lt/7oXrTqEGDdGKStZ3KAYuRCjBDbrCNQt0d6gRvAi5ToVgoJR1OAsrprlA1JBjDXok9G2LtEECMaUCdW6YhMN02bWmxSUyLesJRFSIXAWF4jfMDnmNuFEyznUMMKGYljYTSSZRtiYCGnMxUaYtS4LP8ZP+NedVWpINkterwbL8h0GbYPKQIrvoF2aMtcNNZIUJAmDNiUf4afHdyKoi5bD60s7LwxLf0F3kFkXehIlgFAgut+G9ohwflIJUcG0mIOkMppeqGRmLR8Jtt9+9I7dY3tQuuL5HHCHW3mHqm8y1sK+cBsaPMsuipobix95YjlDLhXkDIwa6xMz6KnhOeBsJBLQJaRAlMQaRPONRrawJAwhJ0xPVlALhi70uEoypVEFUhGMjDYBOyJMAzGjzBclKdc8QXx6WrnaaqaK2rnRNRbtGoolR2I2R3kL1b6zwR+ji6oZVR9EGvUA0yH2U7oNiDh7xaXKdOfC/woCoFhQUuGjeyjnCuqv2cwn021NNsNII2waeM0WCnBNsaiNqJLgx5ov3sDMsKQtaZ49Hdd7jaTL6lVzhpAJNncODJXWD5159wYcm2e1u3Ge97775855zTad0RV4AN6mCRO7Bl22W1ENSPtyjBIHqRYQV4EfozAVUUKTBOiIHJOqAOunuoPHA8ySagXamqLEuXyjb7BjfSO3tlFPJaOk+E+v1TuS/la1jsGSl9lJSqCyTyCwiMMTvTyxdvZiGf2CyiR4gmv5Uh8ZcYPBZEeSFX765Jmthm10KIUrI7nYgsF4LrcZjCRWVhd+53d++/S5tz73l39Ksj6+i3wU6N9QvuG9hJmGmODJqZmxkdFbb7npb/7q8wgQVBahQA1SL7IIA4IWVrVCsHlTRx5li+iotmq+4EfNCHSh04B9wH0EYyxKY3HgkGFcteXLiI8SHwsCRc8osAenEooOeccRTDFpUtAxgJxhHMnihhQFyJSr8Hzk3PMxvCh8wbPABTowIIWecEOlUqcOOuIIdBA3mYcfe89bb1888ebpZCpBMDteisTpIpEgAYPxMNuT0pUXoYJmD3Lce2j38sZcpUHSSqZgNZFMLiwtk2iJHIqoEHPF/OT5ldvu33v//fe/feYU0EtduEJ94xd+8b+jLXj22Wdx9snlMzAlyyulaMwR74j19nevZlZQu7V3xC5fm3rgofs/9d3/VsPiLB9/89nzl49HYz74XRSkOB7NzW14XQmycJBE8Pbb7sLwWW2sOpwLPjQLqh25XSqTD4S4qEY40FnIbH/k/Z+887Y7/+pzf14pr68sTW1t5nGzQ6yLJdsZrtnZhf37Rm8+uD8Vh5O49MBdtyHzkepufnISHEDMFerpzng7megmpxZLm+FEx67+gQPDvb22wvLyhZcG3Cuu4iV/a54c9yxBoo9CyYAt5Np2F0h6IZUlljhUWij7TAQDQOF2Joob7pBnzOY9dOqp81/54kutiq03hVNLDnGQrNj1osvvTWTSVcpRNDYz2/aCN2DzhVgGZOlndcR93gjpDovFLEMExcLshwCD7cPlCRCtUyojVimkk8UPgUQslD7bo9A4WDkSp4SC4SJAUUNLH0HnQEVIOnrbnXdOTs3hAk3QNlLFzMr6uQvnSbFXKedDPlck6O3v66TyFS7LSDi4NIfDIVyMOjt6WNp0g6IaRGT4qDKAe0+d9CzxRLIdjEp5PaN08WMM5EE8guWU5/bEOnoO3nILfgNA/v79+1999dVbDt9y+uwZ3AFQiiCxraxQTMGbTHXirs80oa9xwh6DU8AKkAS8ZSBRqN1h1+FYoeHEt7Bu5PBBCAlzIRlKkqD0SuAbqafFY3MM2YEEsR4hq1pUiAqgFp4Fz9A+V5CPDFKEdPE9Bn9Jaag/NSUMqwDhHYSnf4SCrQ3ibC4YOYiUHdfPc1WUkBO6zks4Ej4wK55Xm02XOOBeqYYtO5Qu0IWdTSPAajYIWjfSokH60HbhWPMSPY19gpcgmYG2II+gDPxStpVHDas/pBgfMRI5oRtjjIig2ax7pHNtOSr51nDXvgP7Dg33j+KIRq0RAtPd7iA6AnEzGiXsUOJmEGykF8b5vlhCH9vZ0+nt60WupfDVlcuX20FNlFl0NAiE4lnDMKmD11XJ6qohaYbStFFBo8r8IB/YVd5Pohefz+SRTgG4UWCyPABqZDmjD9DdgL8tU1uiIroruj1+qLN7V9zf7aRWYK25RB5hE86N6CWPGt4iWU4clFF6i1ExbwcfEvHp9oruEldNoDUqemFjVPbIcLibCHbAAmbSNLyaO9FQPpoWAAaOAR0YOhE4TZOGnIt8JW/VpxnizUmRQm6WKGUeVg/MqX92YP38/76/Qbn5RjoGvKsror10TgCs7GZ0TP3E3g0pgh5gypLMptmkx8RZgbQMKGqkkAIBIPk2MI1UGlL4nvJ4O9dX1j3eMFh0bT0HjxqNhTKZIilV0L/C/eLfT1Z0Rn+1UC8qM6QDno94YF8oEUt1f/8PvXdxdXlidpK2vu97v5tSrJcuXCL7RKXUOrR3P3nm3cG2hZW13r6BL/zDP+zZP/7gww99+/ln9uwdB2hm59eg7lRpJeMaCzRfKaxl11kHWEAphg3KIusQPlaUdJ2fT6NjY4JghAFiatIBZihRmY5gNCQel0+XvkkGb8nBDgUFMGX418E5EWoAzYb5QKVI+jOIMWy0YEDCP2MoMEI7AnyycRJKLS80QklJAUeIQx2/AEXui8+We5bmPl/GxKPqDmTMILa0Utle3Vx+4/VXnZ7g/r3jCwsLPAH1BXEzL7Cz0GA0jcg9qLYJaHjsve+ZuDa1vJIt4pFgd16dWEwmo4Vi/f77HgRLHjt2bIHBsdvCCXd2vfTFv//qkSM3U+eVsg2hUPwvPvfX0O/LVy4TaRONxBcWV8d3deGkNHFlA9YhEW+fX1o8fnVq/8HBZLhv4spUoZy99egenHxhsDq6ohRW8vtC0A2So8/PrnUkXbMzV7pSA4ODg2uzU9Gkq0E5byQ/ux2vn87eRLFMfdlVvE6ffuYryWRk34HxkyfXYEfJGXBtajXc5bj93psPHzgMJnniy18Btk6deqNazExO+qulPHMAnGazxSsXL2017XtG94z09CNtwhyM2ZM9A9skBukPuw7u33XlxZO93kYQVyUKABKgkLOt1cqOsD3WJ5QsrRnQzpqVHZbICTgqiHTJHyHt/+WwJ3jzBw9BG7/0108TgOr2u1ZX6vFYuHug9+K52d6u4eWVeRcOyKAHaIuHNMa4LFB8FtSsdOJaONpa2NOExkEXLRKqRHGtQn3Ib5LCEIfEVOYI3sXDvm0LKzJgs55Jk9MKAx0NUHJ+y5YNoUvAw2tjdX11znlo92OPf+C1ty6uZ3OXzr1Nw66Y6hcsLa/K2wsxmOUIZ0kKHWya5NUhyaKCIZuALq4DJKBha3NW/aEtJchMF4FrJYIDlXkx80kKwp9qfmEa9nhleRnbYHs8nIgEUcdQaOAqmI9EJcEoi4KSwOR+6O7pwySBBlUOHdjfkKggJ9AKBhdQY2lpPSCFK8AIXtWieCS+YlWJ5LFeQKAgDWZBYpjBlKJaMotqNekB/ke0MwiSiQL33NjTe60pnkJIEwoztIfb9aSeZ+O9N/a0YmFP1iMd49jszYo11VBYulAarVjeD0Izt2v1qgta2zuP89NC9ha5MH3kktoXktUT6qUO2Ju30j8p0tQhnQU/2NtI1ILQhnFXqhKHmxTcyCD2LY93K4AarFHEtc0dsJOmhkz3W1imDh852NPZm0p0YU2Hk3LbfeAp5BtMNTB32HLBI4GAH0U6xSSoRJ2IhlPRUCAUsMHaX5tcWVlCQMFnH59U3q4kkaqrZeRm5C99vPk+Qyf45OsbCQGZStzsQWSyvQKafATjDdpnKrFwb1L/AFaPnA2IH5sofCq+dte+/sHOkYStJ2AL4FiyXqusbnuJhCO5ElQGiNSAWS/RaGherRdKC6JJ54+RI68BC0L5d1V9mSWlFN5ScYiCsqnX1zfrJ2c44E5rvsDk0sqYYeeMdfI797xeN4sOanLoh9UVQa91rH//3zczHHrmXXeJ57rxSTuPGqpPl402mbZ1g2mZA1FfrQbxEAwLKB0WVmHNsFQAEWPNaJn7+TAGTAPAM+owameYKFwdhHQ0icBTtSw2CCxD5W8xww5vrVnGsARzh62FWFe7179Rqq3mKg53cGaxCIXu6Bq4cHVmcu4U1dfzBCHWtpchwxMzd9/zwAUIcLm1b/cYOfNwo0p2DvX5upkRBD5q++DBi9vIJz7xsV//zd/Yt3cEvdzyyuIHP/jBcDiysDTPx77w7HMb65l4LIZ3LjALOsMdJhRyxZMp7FgMDgIfAKkp5ksMdwsSBc7AZug+iB7CaMyqxErLWBgGHqYDWR/JmY8TDDBqMGcAArfd2DPvXNJy1uBK5ci7OGCMuITJCTTDT4CQS3B3xKywHreMuocHEwlEIzff6PVCre0P3H8vc3DsjdcJjieTHiGA2ULZLVdGiLobx9Rbbr3pw9/1YZDXm2dOvPrGyyaTfi4WC7996swnP/U9ly5dYakRcbR7927cWckF9PbbZ0mIuJFdI6aBMAdyBff3DtCd8+cv0qfZ2WVW1q5dqVNvLaW6lgPhwL333rJIDqSlpY3M+sGb91ydPkNcLy6J58+fRV5HpR8MJPgMD/LmFm5BoUK+dNPhW0ik2+ZZ7e6DDjpW1pdgI6qNLCJjIOA+dGD4tiP3HX/tWy8/d7y7i/ii1uJSoX/Ik6nWe3rJNlWKxbvf+54PfOOrX6Pe9/IyIvrKQG87/p9UZkPCJFbS64mnc5VY+/ZmILKenzjgdR/aO7Y8M331yjVXeWFk98Ht3LXFXJVcVeF2VzlDMT1bMk6kLJiN2i0kloc6Mo0owJhLGyFYlWLd70Nd3FbJTfk7Egce2AVb/ewTJwORZNv6LKRzaX2pcyB8+eq54cH2YqnUqpITy0b3SPiLErPZRtVYtIVIfYIYGDt0wdLDokPDVIzmGm6CWh2En3g87aguHM5cbgMUKsTicFJPBM02DCFwQmQwydlQSlNr5tq1q7h6xeWBTHVbN3Xf7737TqrdL8xO8SDFauHf+nq7MZuhCCHMnbpe0uvgRBYKFfN5IDMUjQCHgBlwBZ9JOkl4Mo7ZeJfWvKwp8v8gDUS6Ul5Lb7AOiBRHyqYuAzX3SC+OzREGkjTSBDih655dWCQemLgPWHIWAC0YbM4aEe7kf3SbJC4yDDorCVgRsWFBkCiAG3ix9iAWk8GHJ4RKDRoCM3Fo5Bv+4SSIRdISQhNjidp55z/rWCpoqXkRh8DdOmZdCl0ZldSNveipRQ5pXkIPfd7Z4ypJBxkFxsi+7QIiwMdQJslzZnS4mYc5tn7SvnmWX+q+EZi1kDUEN3CwagCB93e8eHXefJ76YAgdX4Zjswg+TDgZMVpOCpU3KORn33Rntkid4d8mmXsFm34o0H5oz627x6ie7YdnpGRvuVzDa4AgRd6I/axlKyE7k8kSrj1byuBvEPb7RmMDWC02SUlOkUayrVUrJB6CtrkxV6C8ZtLVIbYd/MXniyHVduMbOBQdwWlG8wveFgU05yDGBCkFfflqulDNbNrKruC2K9TmdYDyygMHesMdfkdHyOahKuBstZqxeWu4tlKUHfEX9QZTCp5lJ1ILVWGoxaOJHOt/OmBG27jsY+0lSy+vlrOMuqtJNHfxz3duepwb/tlGR823cdHiovjFZHCXPkbze30PXOiS6ZYhxuYqP9l0zRz8f9vdACQNKTHpehMHgJrg0LRM1/k0fiCsYa8xHnaYW81QGV6QMRMPx3/wrIJP/hPLx1zyJ0WGYJyOlnER8NkxLiL0+AOxSnVzNV2xO8OguXqbs9Ao2z3UBIsXC+urJRvOUn0DybPn12vNeVQOA4Ndzz77Miw8yVFB7g6H73N/+TlsvaFu3+rGxg/98A8++dSzjzz60MTkzF9+/olbb93GSMlaGOgffuWVYwiwi4uZ3oFesvagRx3fPfbdn/yu//7ffrMvGSbtIi43SMAjQ6OYtAngIVE8XBW4CcrHp7PEJcLikOHxWtpSZoCFiY6aciDYZGFIKMUGFhCBZlA0DIZXQ+9uKCu47Mbscwy2sYixheOYNoYWHAf+JQ0Fe1RPgAsUlDharOcQas2ACTA1tyHjovVxq1LMNqn8N+655y6y+L567JWOzhRlkTC7evwuqXx9bQFnsL0r9sSTX0bImJ2dOXB4L+1vLOV7hlIUeIeXeuutUxBOr8+fSCRXV9fxa0UTHo2Fh4ZGCIopVDdm5iaT7YnVFd28f/9ePF2J1+W9JLP5oc98ZHphBg3kex5//ze+8Q2s5riFRFP2fYeG5i9NYgDGnL2RqQ/2IxmvzM8ydTaiV6klMDExCaJH0Hv91EsrmXA8QZzYZiQW3hccxXcCWz75AyamT6S6fN//bx575fmXceIdGO55/eTi3iO2bGF6Zmq6XnZ++lM/+tXmE0tr6/FoCgsWlXXWV1Zy+Syw1t3XvXv4iM/fvuUMvnn19aN33PngQ/ell2d9jibFxb/5T8/23r0/2jmabVVnN2bat8AMVGMjL3iVxeAM2OyUESI3ocGMNkcNvI1htlzc9Lqrdl+8kd1ozJ2Mdh/c/+jhWHDw937rb9u7+9eWV/hYaggOjAWam/hOY8RSgDtVFerEUypauMKacDtCqBNY4+yNhgXsLVMFIRKgblRKoBTihquBIFYuoASBsVwlW0CJ54J+fyxB1jPH2tpVL5rhNor+NiYnrpL3cXCoP5tef/a5Z8b2H7ntzg8Hvc4v/v3frC2to4WmRC6sJCCKVoYAVxS8FvjhQpFeX2eZ4yBGLl4iW3ARAw9jwhB3jRJWRWBUkpfoFaAcsMEHONk3gJFFs4+H9uQk0BiPL/ItaM7DkRhyydjIEAL0628cX15bp4oica0gEWFSwB7XNsBYDtRy6IFNl3CqYAKQAwH+cu1RVKYsnEb+AV8K6JWsCsYFBSe/JN1wGz+looRa0zzIiQtGKKIpg7dZtLxST0io0ArbOVbuDxYqv5UhBaaHN6kR6AfHCKki0GrFYHOmRJvMuRBkCIsyCBl+wswieAAkt4N26QJt8R6xGTQgvKomJXegDKF9IW5z1mBVvUEKDtpjfMy3ivrycVzAa5SM4W21mrzVGDXmj8ow4IRa3kE0FMPQkeo5etv+3p4hp8NPpjRyBcLTEAniJXgcZqwmLa5Km1EekIJv5Exos0XiIXLLQdZtlH/IpakEtrGyTL7cMHlLvX4E1BYsFam5pdNFXDKV2pAypO6Tu7j6dv1j9W36FoYZDwN8FZkP7lfhYlB/09ZYyi/6Io5InAgDEuivUnFqbM9AcNewrZ3yiCVbfbnRyDfbymRvdoZIKYKLKDkWwLZmdgzAmLlmRJgjDeA7OBIl3hbuZmRnB9GqbzzEJDJzGjsm0xp6M5T0knbYGBOwLRvHmlJDX7lZM6SbBEFMh/43T1g3WMfszZ9uZBOomSk2N1vnOGFmdufXu4+tIeOMdcAdHL+rhZ2XWq8GvJwAFSAhSAeGLX7CkBMBMyAmIVefDPzzZ9QxUGugH1gVraJXkCJDd3SD2fDDVOIp9FDUdEM8IQFGnRfZPVXiLpEkXdjMtvAHQQthd/hTnSO9ke7VrSvrG/mw23nnXfsyueqZC1OdvbFb77inq6//b7/49wStPvWtF4khX1pZpz5kZ0fqrbNvEzP++omXySN33wN7wCPowQ4fPjwxMQWB6e8bwpeEFBZQEcQyqOqePXseuG//zNUlIqZKhfTGWhUnpgP79iB7Xrp8NZlMyPYiAFNMAaYrOA/0N+jixECT6Afyi0hKUi+ZLzFuyegLF2a+l5FiAmGd0REIKiRGS1ctJhEYsAy6QDkb2JC1zKBxVXAuRMdt25iuCf4hgZHR0NEPedaDrLiHuFKUTPiVcWO6upbLV5746pcvX7tCoitITnqtitq1XC0SFhdJovutzi1PI/2AFjEiPfftlweGuln2EvTxgQz5F+aXCnlyxlf7B3oZ/kwmh2tDvVKnxDkBpuQrpPgLTl8oqBiTzvauQqa4tLbIR+HF9txzL1aoOp/f+okf/4WPffzRE8++1d7j6MkE5hc2l5ZnS6VqJEIuWxupPj1QpLZ6NN7h924dPHgI5QRFbXftoUBwl7G1eze3S+sbi+nMMonPUHXWK7b947u7kx23Hb4zFNj8+lefvXqtdPc9waMPDnM5Ee6enshfuXLt0//q3/7X//LZ2448UKmsTU9PFworlLP1pSKzk+TEmnF4SjZ/rHd0bN/w2NLitK+1Pdrf5bW1ffTjH3/uW1/+6Ace6OjtfPv1Z9KFpa5wxGUjzUAd6He38ARS1XU7We/lXc/kVCvpzRBlUcHCjVXyNfLPdv2qrVbqueuhf7f5Q7/yy79DSl38LvHlrLTK0Yi3Wd0ky1BbmdT0znKRYjCE0mFcUZUtlAogMgROEAFqWwADqGB20JkyzSw6optWlhehUqSlBJjJiwC5Yb1RJLOrqxN9JNV/8VdvT3VEIiFZixHe7W0bG+u4wPqj7RTM6Ovu6EjEi7l1qR8wFJrVCNoA9sTlWZAGIsC/z8ScCesTsuL3YAApUe6nmJcvrZ+hkl8YafwrCo+vQSq6BgZJjb62tuYn0n/Lls9kPE7n8uI86a+i8eK5c+eSHZ3Do+OBoLc2U8R+KVkf5pIPJjyapYAYidMEFB4XAMXmc2iZYcTtsswaMANw6iKfGhiG3qK4JPQ3mmpzWgjMOC6BdVjPBkeBg4XUQL96EO4QfKdzvBnhGDSrdWhp6mhWaBg1uFEea2/RWBPIyxXeBCFGfcVa14ITvYY50n3qlPCj2tJ7tN51wMaDrHb+0+3CAJLJ6A7HdEIqMm3o1NW8wcL0w+jYhZyt7zXoQ4mx24hhcGFztxPd7qFZ9Fw41WAk3y5v7hrat3t8bzLZiWuIEljUEIl8sntBKW0oPbh/mwyLvJw0GvV6kRw92DCUxjkYsDUb+fn51YU58lkSaJwMBXFkJcNVvVLBuTQU8DaIvQfJC5lhq5Kwpd5rMM0nmm+4vqPPUHYKxDXwo2OEmnhnb5IJrFLdqnjCbRQcI0lXasC/f99uZ18EHytbbZow0m0ntdC23VGU32KuGlvkXK3gTcXnSQ4VvRWNggqbQZYxmoUBVwAGRDAiFBsghg7RDXomJ3vdyOLRICIc8luX6LYZVfbWASetY2v/rq/YmVluN5esTzUzoqbMAeAiUk2b775qzZvpyfXm/jf/1YvUklow3TQHEnk5ENunk+aGG//oq4yQi92LA4Rc+iNfZSlU6JyGhD8NhOURrdZBLmSHJHcEdX8xGzv8ocDSWpmI3Wabb3oua/cEOjsHCATKVgvkbp1fJbnohi3SNYyo5QqdO3P+7VdnOxKZ2fntVGfk2sRMpry972Axmeph7jbSK6mO7klSUNaZwuKxN94c6O8t1So3He5/9LEHn376mYPBCFnxJidmQ+HA+lr2oYcfOXHiGErpgbY+NJZUKrj3vru36scuX5hELN6zJ3zm1OnJ6dlwIIzeNpuHcyOPjwJ+QIpinZS4l3HhszF84CyFzgyLNTSW0sDAksIwxNVrDeKgCBRI0q03jQefQTLcwKgDGJy3qC+Lm58QYJoHEXFeHAiLVh4EsrIDb1adUJYVG76ibM2gQnuJJOWlARyTvJ7Lly/VSCPscVGxtGswSJk3TNId3WFSIoQwrHamjt7+gaeefHp2YgVXHoSbRF97emljYFfHxsYa/jUf//hHX3jhhZkzCwfvPXDlyiXCOglTWV1ZC8Uip85c8vmC6Y1cX18ffl6nT5/DwExpoHg8fPnCQjjqXN4gqbstHHO+/urx0ZGBpiMDc/PCC88xp6SHI+wpGgtcvpgPE8ZAVoNiORSMQyoonvjyy89HE48zSjkKzZF6r1W4NkkuSVtPl21oqL2ns7+NBpz1pfTE9Nxl9HWDI/7b7rql3ppDyx3xSa127Nirn/jw943v2nfi+FuJ+BZ1aSk05PNHw75EJGFbmCYmfDnWOwKaOjQ2QjbORrHsHkpu1+C3sqQ6SudLW/6tOO5IeFo1cmj7wsFtEkWBIqn8i5OpW9XGwe/EeVKofMsbpjQIBTBaoRghdkWCy5U2qT41+MDB/9T8od/97T+LBGPVYpOqWtlijeBMHBrAJrhZado3W9QJwOGIxcUUAwQKtgNjWEuM66SHVgCSnfB0eKYGgWQUiw8Ga60KemOACvmEIACK/xC+RflGRG00wAyp6uA5kJTAa3ii1E++/tpAb1+qPX7P3Xf5vA7Ia3usu7+/H2crwA+LGdTUUP0tVAUEQRECQB4rQAt4Yw90ie7abNRUBxCBTDpTKhehmaEQkOzA6oF31ZVLF3ljBFMNuaGa9cWF+fvvewCtOs5Z58+fX1leJYCeKkm13m5ZMfk6iA4vkO59J/jd5iK1mcigyIaQEStHLrd4vsIKCPnqmiQccbtsLA02SUPXFZKgFkbRiazGUH/nxv16XFjY+hNJ5Pj6RuPgXN7CWYPThFd1rDOaFF0TL2ThW71VdF1UySxgBsJSbalBrfmdTbZgSbPmw2gAOcWQER6VDtBQBR6QVGeIBG9qiv5zxSAO+ijeAEcsB4XzCP9yU7sL9r6CKa+NdYjP+cMPPh5wh+DMGuTixfWZxK0uL4ER7oCXaAcixJlRBhmDGo02m7Dqlb7Obn9Hp61YWL5yeW1pkXR/ouqItlymNjepZZ3KooCJiIzkdp8H9tPgGZLM8DF0Dt6HXqr/2iyRUb+ZADxVcBxQuqHGJinsmFTClOlra62+Nri/c+TAYVsCyFyt1BdxoLB1+BqlNTk3Iowx8wROYfF12rwku9skcYTmUQMsiDDvYWxJ5MaNhvpCgIVDhUXlba6+aNy0vSOam3mkHZ00++tTBgCKCdPDZhKtG7hdzxt4E1QICHSPZumdRs0tZqdL2sw91hkDoaY7/N7pj7minYDBQA376yfpm8ViCk4YCKtJay+qKk0qHeXAggw9uPNeuWLAUIvW8C4kYfIxE4tEtw0yAUKN+hWWV0sHsFN65zollJvwcBQCxYbiarmJJnKsZSrX5vFvKTdc9WR7x1bVXiyX1wrN5XVb6+2zWyRpiSfGxsdnLsxiLaNe+sDIWCAOrxdYWNqYWZjv7Grfu++Qbbv6/ve/98C+3U9+/StPfvXra3j2JpIkKD556i2SEjz+3vcvL60PZrJk5IE3f+PY8YGhgYHB3sWlWaweE1cvXr08aWv6O1KdQ0PDKKIJPSHIBvGiu6erWi2jlIZ+wPgztkAkLJ7SYkJRQSceacvYAEILpWCvAZQQLWBCuZP7lXqLarFKryo1CZtG5DpUcHx9KiWgaCIUK+iCo6VZk5BQtB9ez0NBChe+P3nQMTZsTpJBkHuARnBvsVjgKtkZQ2HfwvJysdxIdQXK5XxPT4c34C9VStlCvljOxRLh0fGBD3/kg3/z+S9tbGSY6/7hvqX5xWgiNjc5/3M/93P33ffAn/zJn0BQbrv1dsSprq4eigw6VjTFq0u5vp5kLk1xNnywVcTs+77v+0C116YK3b2RH/rgfX/5V1/2eoIEEJKNp2+8h3ihQJj6ei2SIQ8Px+bms6SGyGZrqPBIR8GSmZy+eOjQAdxss8W1cCxy6erZpZVG30B7d9eAz78UIIPGJl5+jvmFuYHeoXIl5ws5jt7RGwxGVzcWgwlVnq8VKoFAzNMVnp6e/MxnPvObv/H/gzGuVd3FNtvxExeiwXh3h6rm7vFGjr35NmVPZyfPd0djB/ePoZvFjL22Mp+MB69dPOPersQ8jcB2A5ci91Y1TC5+TKYwmPi4ILCgh8DDxHhvUpd9mzJ4yMekiYIkoVzFAyDSvVW6Ul3J7n30rh9zfebnf/bPRoc6kHKXVtPRMDoNEjLglgKygRwDK05UdZEQ5lV+oOhAWpGzr9Ys7jKlIgIhhlAq59p9XngCppWAbDxsyEIC4ilTdm/bns2m0U9wgaqCeLATK9yujKEwwU08mSvkGqy3NlaWbr3pUDjgpqbh8dePUSYPUKF8L/rItdV1HqcUEh1Ir61iRYaHo8Afphb4PzysuQcyLLaPipl4bKk+KrnHZQQhug+zBxRdkNxqof4hjjTg85KaAkm+q7uT/MDhoB/CzMRhAAYgqxVGSbwnXyyO0vgxWDJVWyAQBKnKiQadALwKteDdlEeyE1WzI8UifIJVDKbTgCncRK2If7GoF1AsoxcTxZqROM8mJCvEC/oS2uKYf60zOparKHthVzY9qSWp3wbJGRnHCL66BtGTKo+EjeKO9Vs01bzDQQplANrQANOUGuFBjknapbVsTpgdamxdMxK8JGthDXETHCt5o9gT7uarEJbRivC8Ez09OY0JgCqx5sqE4XoHevtxmtgztr+wXPY6g3wmZA0HNbh5TAvU3IVRglUnywbYhp5i9RCtsW2P336vbWM1e/Hc4vwC8+FzugMBxYU0qxU0EvBEgCl4CbjDOwAHcxysZSAjJF5dQ4lH35TgDX2bzmicRHctMrxpr9ft1TYXij7cq/JVZ9kJS9sdjHb6bhk/anMWt5zrbe66LVwnvpvUzXB7njA8FjXwtCzwhgD0LUdv5k32CSgPE8Q40nc4YTI+41aKjGMsgswF44bTIIMJHuT+HRLFWbOph5ZkrEHeIcCMFfPB/ey5i70g2OBi7tmZFs6K7uopjthr26HBFk2FTu4IweaWnXtu3Gs98b+x1yt4rd6lN2qQWSDWS6EDALK4PwEnzuQMtm6h23QZWg4bhyYFvKGCtvLKoj6gf9PmMeeATuwreCoRWiN1PG3iXYlnJWyRob5eMv8spcvR5OD85OLblyixYCtUbJna1GOP7fLHPOXGYigaa9qya2vVY6+9tmvfAaSuj33sQ+fevmCzh2qb9lA4uriWS3ZGD99886c/868gQ//1v/wkSOrOO45SxLRGZfWeBK5AsVh8oH+EBDt/8Ad/gLoDSgy78N73vve1115FdZbqSDCbJDGemrraHu/CheTxRz+Uy+RPnzrT3p4aGxsjqy3JCgYGB3BoSq8vl8tFCDCVswFKgFCAz4o2TBoeYzh24i8FsIIVzcQqAF9TDWChdGMhkbuGko0kbzebFs71KaZFoQvhYm0cgA1rNTwtqHlIUiBSEghn4ahI7l94CAhwJlMDaTqwjZtnkc7ShfLQSB90cXp2KpmMoahaWSndfs9eHB8npmQdRG1+6PB+Mjfu3j3+7/79v710bvrYK68TdDR1ZUqYH9zgakNPgLqbwaFxBK/7HrgXDx00mX/+53/a3K51d6XQSw8NDfItSMwjgyMuu+f//Llf+KVf+cWXX7jQ2XFtbKxneXUlJv/y5OL8Et7bxULtwP5+Jgi0MDoyMjWxODddU5wUSbyXCrF4we3du//g6HZbrVDcIMJx7+5999x9e7VWuHjhNGnHSPiciCXm21YqtaanUo4mouk1/lvq7eto1klzS/TOdr1M+qrm+XQZJ/aRsfEtimBV8g0XOSCbpXyOSCMEg1tvObqrr52Q7lrMv+eWgwTQtmqumw4ffP3F5ezyTEfcE/Pb67lsqZrtiPlCLl92Yy0S9pA8uUkKR1KRONEBBrTccQ/M1Zx+myckrEDiHnR5YNp89jJJJwPtnlrmlT1Hh3/tt3/0l3/+98mVfdPBHorFQX3FWQmDNRxNPh5+dDMWpjlr8bI+WHsWxhEBkoZD5eeBLsmjgESlSb4BCDdoDlQNJcZQ3SJpKKwdya0q1VBWGrstiqCjQkQzQcIizDsLc1ONWon61j2dHbG4QGVqagqLb1dnN/ML8rEMH6UidLcGUJWqZSg/nAb+pMrdDRCadE80DeiSkxjRCLpBO2yoRqhIwD3MLFYJbiCsCub1+PHXwZEsGWanWi7R7WAosLy82PYf9o0D2XwPD8sEzMoxG/Zw+Euc9TGm0DhyN40yIqBjlhg38wito9bnEYCS0YEJttrhjAZPsjP5sauWdGadtFYXF63W+MnGJT1pNmkvjcRsXSfz40AAAQAASURBVGKvppCoLE34dbzMI9YGK80qMXpjPWHdzMTxEdACmuQ2qynmQkgVUq1znGQq9alGNyr+g2PjzgU+NeK8nMLscB9EVyBiItQqOTMSDYPhoXgWJSCabqdvbGjXoYNH+nr6AYhipuzYdIcJG3fs1IBkLlD6kBmgUMj7qc2wVS9WiowWKc0cHsr9FqcuXhAXDZwJk8uCqJ/SxZu4UWF8iel0VRwAiAwWRffqW+gzfdW3GmIhJ36crZQ/GmUIQ2pDeGr6CIBKFysZrLkDY8mRg322DgoEZG1buW1PzebCyEb4CAV90ZUyCNDyBnyDRmxnGcg2wCGqRLrBOOAUAcVF3Uy1BXIkEPBhumdRU42s1U+6CPoyc6szN2bBYqhoXt9rNuuAwX/3z3ffYM5bJNDs+XK04SL2oHGrEZ6VPQNI4LzVjvZmxKyfGsh3Notm79AJ3ShCq83qBfNgOiwybM7uXDUaB408UyCo00Qwd7qK+8bCUqU95WPe0BnCIFdqLZc3XK754h19lAkqVMo9Pd3TU1fBEhSywumEdELZDXJgYPvvrrU8MwvpdGUz09jqHNo1t7hGNv1ybatcbeDagSHiPY8+UMpllpfmKrXyBsEym7Zkb9Ll9Fby5eGhPdRTnppZnVnMffCj3/Psiy+O79k9PNL3wovPII7MTF/ubI8yMDMTK5EI5lUvyehjsYTbhZe1FHSp9tS1a5Pf+734+l5cWJwBQ2GzkL8LWRqrzZA7NTu1HA5FQSIsc1x/GeHnn30Gz+F4PEZ/8oU0WTpQkKBwZimVq+BUm496q14fMFM3KVkgV3jPskAAGUYYfI36maUIa1vIV0BJjDa4BCaGpiB14DvLxZqB1UyYJcwxD6OnA7JlJcO/i7vhL6SWpuiAB9yHtQ/EQpJWYWfEsHoNhR1BVJRv2txuUtOOGozwsh4E4kgwT1wsEX1ez+DQ0PievZDwbz31nKMVqFVaeDtTZ/CP//SPeCnHP/nZn/rhH/5hPry3v4fOkMgXRuS+++77oX/3g94gHkNbI4NIoqxryrzX8TwjtfvQ6ND+g/vA1//tv/830nHE2mNAmy/icgRa7V3BgZEOp6uZL64p85bN6fdFMumyyxE8e/4y/Q+G3CTeGhzuypWzK5kNCN7K0jLnD+zdx74IJsHAznfjXeLzkLUCWXx9dYVBBgmDqDPpbDzSVcpBJEnv4G/U2h5/9P3RQOR//J+/8JM//h+mpq/93m9/6dHHRpTev1nZ09MZ8vj6u0fqla2gN1LJFQ6MDj3xxc8Ftovj/ZHuuGeztN6qETLLGyn+VsU4C08M1cOGACpy+p3UenKH+DYkIeqAKDh450+YC2myLZU8WsiWndtd/vi9r3/99d/59a+FvKGIP7K6uBjxuyqFRlcqhAlybiI/PJRg6nweLXMIGJPvBYoQNOtNMlYqAYcohWL5mAuYNtxbyOkGhgIhcJUC02TzCAVjWN9xbiAD9rFjb1AYmHLseLTRJukvIrHUHbffs+/AQQAMLH3izVPwVYjMMKb4SRFiDhgAirwIzbOQCYrIzWYkFIZja48nsPbiXc8eE20oEmbPMRieEelIpo7cevTy1Sk0KHB+wDAVSqDo87OzWHCCYfpQ6O3rzxVKhN3BQMbak7xCRhUQEEgEJEsXhYygm1uSaSyciOqfjUhq1D7cAEcAYeYeAT2hUr4AfWUJyTxuNjCTaQ9szQyQWwegVHw9Lhv6HobKbMpjYqgjj/NOiAgEA+rPD/rBOBsyjFwLEWTEtygmyAV0o0J+EiC0jHkE5S8r2bzZ6v8O/yT6Y2yyatwiV3qeV3Az9Iod7XAPlE4P4lrC/XSGZS0pDldhvbGtUa0HQhHKIcD04WaFO1wpX5mYnB/uG96zZ2zv+P6Ojh400rl0BdsnHSMfXpllj7rM4YjEIswxxqdSvsDnhF1uhXG7qVugZEF1QgMWFtwUQDFDBypXP0DoBrnzmfQXyQnOTiOKyGUCrdHHIZai9+ED+QfiLHujDaeSsps8HX4oZY1sPrBNGKgdPsdyeSU+EN7fOxZudwTat22xis1XpNxzvY2CCiTwl0sdAXmaeMRtI0ozbXofG4STIYJF3UYNTtO0reohaBeURZJ6s61NTNTcqNky1NCgSj0KMWQvfQbnr5/VgQVj79rr7usb328d3jj45z8N9eWkaZPWGBzrERYtcHu9oXf9azWl7l3vxo2L1y/pX/5n+vhmMWqiu7rL8EVi5qwzmhKuMN7w7exlndEH8Ue1K9KZkMm/hj4Acoyfe9hNaGQolJhcSIfiCWckdmFmpT3ZhztzC9NCLi91gZMya15cW65NL527Wqd0277b97a8HcvljC/Wvz49257sKWSYrM3J2TmyF0FmqrUS1QIDDhf1OOG8/LHUyvLa6kbZHUg8+uijBBSF47FsgWiUCnFE4DIclcA+e8d3jQ2PvH36dKFYOXz4Jo/be/Hi1XisHe8CevHZz/7Uz//8z0ejkWQqRmFy/L2wulaq2yVIWtsGarcE9MNmA4OAXEBnfQND+G1B/jCtoLvDQw9XP0YXYQB+QrngDHCijHW4xdqx/ClDpPGU3ZdMHqgEqOwEs23YJmD9XyicwTnMh4WCrIljz4Zm0nBhEozYmAYyxMMWVqQLr0l5YgzDFqpBFw2CxBbI+kagwPEC2RuyY68T66UFC/TAUxbzpYvnLywuLpfTts4uRA2hPIgZNPgrX/nKm2+++dLzL/AJZO4lKzS8FC8Cp6EJwGY3OjQ6tzh3/u2JoTGeDMyvb1ADGP+wpfklrH3/7od/+Bd/4Rd/9/d/F/5jZGysTBrD/OreAwO4s+FUBXahrhEvc7sCiDI9QwP9xW7Qxcjo4IGDuwul9Wxmyed1Xb56iTrsROEODezCAhBMJtZXVtfy6fFdw/Q8X8g3mg63L7ZKkY16ric1SN2j973vE09+/RtU2B0bHWnW7M88+/T+PUd+4//6w7/887+86eYD9zy4H3d3Klw186u7h3vi+JOCCxzO3q7+ZdvCKy+9CAaM+TyV9Eq+vtkZ9blCXjLsou0P4IVUxxlF6EKJB8BYqPyrzATRPxj2hZzBHmB5ToGqwU+pZM/q+uted8xOuHX2uTs+cO/o6O7/9tnfzhaKWzYPhfrG9w2V8plqsRpLeZXzjIyVWCgYGsLYyP4PPpQicwv7HThZXJdyxIv/hgxhOMRDTa80+YUx8SojGrUAnI6VpUXcoEaHB4BFHALS6TUYOwaThBr1Sj63sUyfK5vbCMpogIDwaCxBgBkgxzFRTOxhMbOFLFiDcHuCOcCo2Vweqd0X99OS0CDxHSjhCZeS7ge3ycbK+hoiK8ATj0T5BiRS7iR3B3lkOJ8mhxbuNE2yGLkj8Rg5vCgbjChs4S+wiHTK/ACrAvdgWL5TSTbgMqG0Jt8HumyUP1BfU1iO8cBBUdoAyAwQaQYFf2SpkdnMLJClGTFYFlaWnvJNQDbRJtjtdMiIcKKdfLY0eCbiE4uRmV86wlKBKqsl/lh4lkZKqJCpJxxE1FPY32qBvXkrYuvO60HHYiUg7dB3IyNDzB0cm0f0veYPLMYBJnbTjvJPG4pOy4j6bfFghJzmtOhxheol5NdSf1f/Q7e/vyvVE/BRONW3VaU4ObFoBOdFYULxdmFN84mMPmFhsB6orQNBd+fhQ0rMUcyRwxQVBA7oOKEyJykKmZhcg4yB0W2C06EVADEIXj2FILOXVlmcJbIDSY9A/2AehUKqXKzMiA1XxFZuZqD1duwjHaTmslVqhY1GdvjmocRgJNIbtvmJ4s3YXBmbM9doy6lsHeAthTztUTlKXhGQe3Ymp4cZWNCjuBO5+0JgkImIAcC/FJULugJ6h4OVptkYxjXC+mXAaUfkZUIMb3RDAta7jKqcbzSbPvb/7wFNmHtomWe5WQdsAgvzr5o0jYCuzfkdiOL4+vmdR8xPC2CsTuo830fPDSDouoEuxsAKJjd0VyNw44/rZoIMYIGAmHdeCznB08rtjWYKaIJxaI0WW20Ir2uIL81GR+/wa6fO7T20x9c+uFJch/IsTa/2xQIqO9JAO+DKVxoL6XqGMn8e23s/9pmr8ys//z3//s/+8i/2RDpQBVM1hkry8ysLsegw/DLeKKh7qbtJxjUIAes8mycfR85V2eqmvnwhffc9R7HDk1wlm1987ZXncAOhujAatpsOHmKijr1+4vSZ85QKqNQa48kOePPe3r5oPEZeKYJN15apSNjo6k4NjQwT6X7x7Loqz5YqeYrMs3YITWyzDdrbevsHiPABN2DZZV1XSgUkTm/QSxZSxpTYI2gK2IBBAXNIZYKqUQuZ5YrYhc1LjI1WJihHWkhNB3tEOlCNJRCDT6zzusta2moA7Yxxp5Y2X5Z2JeJgSnG+Zmb411QthFiS+AUshNcK+IQWIJmgSJCGJCXN+hZOlMQvYeGAyW1U6huU1szmwpAJhyMaT1y9ehlnK4QwhF2YzX/6p3/q6exaT6+DAEHNy0sL5E7aSKdpCVbmB77/B576xpMjI0OPPvLIL/zCz+OZ/cLzr339G//0zW89de3aNRYJHFIiGQfz3X//w3//lb//xhPP9wzE9x0YzOTz5POBQfB5az09XTOzE5Fo5Ic+8QO7QsOXCmffOPFCOOYtFNYIiiDuamxkHzEMtx68nbDEGedMJJQuknq0zUk80eJidmsrwCuW5zfqtSjpvl95+eyHP/IJVCBvvPkaAUuLC2m3NzQ4uGtobE++1BwaPTAw2u0P2t31rvT05fLm6uiuI5TNqlQYavvy8monZV5bZbyZEkF8yvPQA+k3HU6iwOWXaZhrLUUIMHFEZeimkEGbT/QChhjXVVaMsCpZVgqrmGIiEVZlcb3wps/mTe6547P/x6f/x8/+GYlCW7XSxMxi0Icqu83r82bXc7EwJh4QiWqeC5HzOpEOO9Z6KT6Q6eVURe4KyBTzTyFNvNkprUuwiEeLs9XE173WpMZmjZRpXV0deKEXihljCG16qVZr38yur0yTt8HjX15dI62BH4O8kxTNvdMzc3i8M+m0Kwh2O5l9PhMoIs0WTg9wn4QUR+NJPNTQclGxA+BFM8Dyh5MqlCszM3NUf4crxb2A+wl4g41LdXYD1ahn0IEXSsRPBalCAYiSqI6kWk6yfkgHK6olGUErQWtCWfDFmqFWwknOhNIL+tEGUCMUhC+cxGwhW2qI2BCgCTUANbNDicwYWfcD+hyiK3OZ6QFZKT/ONhVufdhE0QZp0OQ2awr1ibCyFIX4hBtFg/UG9ixDSDNQwEqjf1xl9SBIwoarz2YNs8YZLr3XeoBPgPZyDUJiGuUx0u5YHQNnGhUljwhl4/3NK/gTQoBQWyp5BwU9HD5bqJAtQi3HBsfufuTenlQv+udkoqMJeS2KnKBRYcJ4Ci8VFjwESpYQJSxtoLEJBQPhRNRGPZGVpYWZSdIJEZiBtp9ZSUZDzhLOr/pc2Dg+ne/SHqU3dFufA+ciXafGmNPgGZgteBUwLLOm5Eskod9sc7WKrUJbGLfIerGRydbK4Xbf6MGB0Mg+W5jkDgRNbjRbmU1nzmmvbjkKre0Sad3VuF5FKjmFaDOhUGDG2tjEeYtyrGgSTHxqMZ1ncsScgbbE/YihYomgm1CvNYj80GbGnw8ypNc69a69NVPv3lsXrTP/7Pg7T0In9QrrpMbDTLo5ybiYQYRRMVB644Xmnut36qz6euNB0x5fo5MsUQG1/gQY3KNjDndOmhPWyZ29VPWwMOicsdFXGlvt0eQWaMIdLNQdZy5OrK031rO4aWzcdDQ5n24FM81YeyBfd3QkOvvGfNXVNaQ59Hm4QBfR+Ppj8Z6tLW/i537xN9/3XR8/f3n2oUfev5ZZfeobXx7ftze9Mo2TrNPjBK1UKkp7TsIgVTAiRdFWORiJ4ye1tLrx1FNPXJq8dse9d3Z2d/QPdt95xy1vHHsOfQVhu5cuXZqdmIFgj43vo3wpajfCbNBmwQWePn2aoBfjnQDZIgll+OCBm+68845rE1cunZ+MR9q9/RhKyddYmZtfunDxEkQM7SupKpDVWPKw0YAvSjzkP7IiREByhAbhF1utQOxIYInuDKYNngHKCVUlmER+esCV8dwAQzFl4FP2IA025gIMaBFm9tYBV1liXGXsadyi31wilxIIyKLuXBWJF0MtBSCCtsNNZmCptqC+NMt1qdmwwzfE8yo/PlryCgorZ61YI5civEhHqjcRT33605/+5je/+cYbbwAMFFrHGxbP5ACykt9P7WBw9BCOyMkkgT3FQuH4GydIkcHbP/CBD/zyL/8iaHp8bOQv/+LzEOzTZ09jFarXqM7UeOabL1HW9647H1hJzy+sXiGuZssWb7SKAwP9dJZ4VpyxP/SR7w65Qj//R7+Qza1FMcA6MJmXezuj9m1/V7L98vnznjbqkW2uLRFjk6jVnOVa2e9vt9ujpXI5Fu/fvWdPOYMDds9WK3fixPmDh24homxs3yippLt6E96wcy27MjAwlEiNHji87+lvfc1vq3qKzXsOHVpdz73y6tfuvefhO++8nyjewvJkBLNjiIynGWKoUNiFQjhpO3KlmssntMFah+0BXxgNpMWDtlwKoIVWQHCZAzxgRFCQvqJ+V7W4itdKsitRKZzG2DB69yO/9ls//b2f+JW+Toe8m9tsvR2J9EbaHwWtMNW0zkTqiOlGuLWYMBRwhF+yR/DTgpXWBFCB8ODeIPgw+ByfFVI7lph9FK8gcypGIBPH4rJr4NMSCftZt+jS0WYuLy45vSTNhmDWiA4iLxWB2vPzK2O7+olfymYzVMIiQzhvgDP0+fjMLA2SI8VIj4pc5jyx76Ygt/A1xh1xetRHQQFFQqVcjoLHgplVZW0iSWJrs4R6vJYv4sy4sraOrlShh/pGhtLQLT6M90DvcNlHM4KOhdERTwPQA+84BKF1xMphzDZaDEqvztQ4WwQbKBaF9aWSF2rHbLweTplM6RAkUDlSOwMEg2rOyIojNTAOt4y34iRp0dBF0U2hVPYiEhBGnF8UxGAtPWFJ5hlDlU8GAIP0WWASZK3PUEOsRq1IzaLRIhriDi5gZs0j+mJItOabegINpD9yr0BA5T/DGXmFM0sObzFT7Iz13fbw7WND4xJIK9sRTzy3TH0C+Fdy81ElhrhssrViJm1SrYyMLAxV0OVHcIGsyi5ZLCxdOl/D8bFeC6ismw/+gUTMZBAGWDX+Git9PVTvOiHTKWuTCZbv103bpHaE75BgiWOyHdmmsenC9FKvkRLAViAOOTEY7BkZDXXHqOhgs63ZHPgn8JqKzV1zeOt2Dz72dIh3MS7Iv3BOrBjKNFHzQOOCkM0BMA8ShDFCzJYenJlpMCisCgBBOIw2WBhMDHOtb1SHRN20ZqxOf+ceUOELzKrhU3Y2c5Kp3HniXx6Y+wQfzKD1jLmHY5C+Oc1vnTI7c5t+mc0c8OzOT/P2nXcJ++vmHTUmbQvkTRvAGFNvjlkEghzTunUDj0j3pZwxPGMEXzgWxDwEMRxPihVPoQaP4p9ZTL9yvEFdBH8I56PQqyev/tTP/UzLUb8ydd4RtV2emXA0yj4qudVVbRqVyVphO13ZTlft6dVVX6hjcmL+yuRMLBXdvW+0b7CXwNC27VzCT+ROK1PIlnBjb22VS/D4fBOZ5XG6wcE23tPjsW1szM1ca21XcOoaGu17+KH7qPObWS25o1E8NaGgyfbOjY3CRjrv9fhazezTz3w7Govcd/dda+tLRACzyChbhb2NVBszU3PFHBUG4xixcCNTqgS7c3R0uFqTrxyZJdbJH1GrIHBECUsKSeEGWqBD2EHZgGp4cZYWsgUWQ4LgTXj+zuoFaWp5imVzIFAyF+AEFHqAmdR68oJpmdbMvDJPMhtrHlmTwBsPQ1A1R8J1ggzAWHVKuYnJgQjIv5qVK+HBAfUTSKD8NuCnlQrHT9FAKLRrm6iAYp0anvD/rIhWtfHyCy8dOXLrj//Yf/zyP/7D/v374okoBPj48eO8iG9cXJqnA0StUNyQPmNvIsPRiRMnyDtN7otrl6/ce9fd+JOjePz6179OP1AQgHk7OvyYnOnozOR038jwv/k3P/TLv/Zf4XvG93YyiyOjA35fcGZ6+fChW3nRs99+joI5H3j/J1479sKVy1MjoxG0ZeMjw5st6i95z759AlmtXsu/efLKfQ/cn57IpjcK+/be8fLLb05P1T/64Q+1R1O5zNre0b7f+qP/gWzf09uP4t3l38pVl37z939xeGD3zFJzaXGlrhwY/kTXYEf/7oVisb19MJG3PfnEN6+evXD3kZurHfGJi29R8yER7EwlkvYqlQrW4PV82HdButIRg1hxohO+krRLHt0KSKm16Sa9l1Lx4/3P2xl5oqEouCeCQOxyM+3xRNs8y43ss12HbvvHb/z0B9/z6+NDNp/PXWnWw4kwZYFhl3DTw9TAdINMtMjAGqhVfC4AA+oj9awJFrfwOxlDSZWJQVeaXSdJIfE9CMBaKYtkqQAgITozU/hncT9RtcQzsexd9niQ6hbS/25Z0ILDFNqaSDwxv7RiTLPGx77RCEbCVG6AWIIN0VviekAUAKxnKBThgxF2aU1hS+gH0CA5XOvpDB12FjDzESPXIDEqIh/WDWKlgJZCoeiXXZg8O/VIjCylybafGR8Wl68FAa2SiphjiB4HgmGTggtST6N8P52gx6wZyKp1hsdQKLNnCFg8SNac14MqEySyh5nYjJjIt7VZP7mZhQqk8iLOM0Y8CGlUaAaTLLkLsgOREvrXaqMTlixs9kyJELPhakVJWJn8Lwuxps0QAdYLiw5FtU4IExuSjurePKezegLRVzSYaC382on3IQ2FWBs5QjGh266eRP+hvTcN9w+rKkK16XP4UTDjzR70hySZ0kPTO1TDfLUi093oIhkBiDdAWStk1kvFAikZ5VpFsOe2ihRBQFtwbehymw0sTvSHT7Z6xV4cPl9h8TCMhyiz9mZDqV2SvcVFZW6SeZQrW8WGvdxyVuyhZu9Y+8i+rrZuyu5SlC4NIsaGUm2Qcs+uqCf1APsK+ZwZLlieBmTUxI+4nZbXhNwqGG1i/FSXlzqpcHbiOYUulVJf4yx1B+DGhBD2rCli9k3HtLekXuPhC03TBxj0yD8CKjNjGi6zoDQd5sF3Dsz9//yk8LQ2tW8IJwdWazorUBUjLkx8vdl3RpI7mWXdJ+hVy9ZLjVyrn0A+N0j03aG1hgwrpxiDxVmL+lpXeVQgsUUqSJ0X7gFRkOmiBQFmeDyxqzMrl6Y2bN5Idct7bWYVjiWa6M5m6wurG+/92HuLzVwo5h4b7eqK+XpjgUuvv15N5zIrudmF9Pw6hTK8LXcsV932uiNuf/BXfv1Xnn7uyfpm8c23Xms2sv3dMT9pOeqNmUuzlQJjbivkWBZSUbF2MAzjWkiJHmowkNKFegcop8fGhx57/KEzb58iTVIlV8fnxePyky01EOo4cuvtlKb/xpNPDA31Xb5ywesm6BZiShAJZg2atcUjIdI8cTA1NREJeEk9tLpeCYa8Dz/02MHDN01MzR07dgzODOVWIZ8Fj4nQ2NvCAU82lyOBvuIx5KTawF2KRUEZCfRvVZXzktqECcXKg7cj0idTAPpjATK4YABWE9PBHgGYFcRVNj6QM9xDU/D9zZqSdcAVskZ5CqYRSsBKRtjlZhRO7C2UCqiiWZOfBKeEolhXYKEmjtm0D9gSIYjMAMZHNOIWvhcNdrKre2JqGvfytbUVBCC40Lm5xZtu2t+eSj7yyCNPf/uZ5776Yteudu4Hj4LBACFILG44uVz2yE03k6yDlk+dfpPvy+QyvAnNZCAkO2Ug5MePsbLdeuwDD7W5S6vr1/oGIy5Pi+qOw8MjK0t5zU6w3d7m5Ysfvf+R5cziP/7jn8WTjY2N+Z7UwJ7dh1U5dgsaQ1KB7VdfPzY8Ml4obz3y2HclvMP/5Zd/jTQBn/2p/+PE6y9lNxY/8KGHTrz53PTcmXxxMZUKU3SIpBw9HX0/8smf/OaxF4cG9p48fv7qlekf/aF/nwz40guzCY8v6mjLLixMnD+bikY6k5HDN+2fmThz5dzrjexib6RtIO71bRZrVFIRjmPMGUDGyxKZhK+VBcFncwedbgoYUv6IJAYO1oc8m2UPi+AXQAk3vIVJ+9zRakSrpVSk48HiVP6jH/y/Du72F7IVilVjX/Hi1CzcoZdAfayNY6aQuVAGUgMkBmkjGkoyJVM2BhH4KolPRiKCTDKblE5n6gvFHMPlp3Kv04kJH8qOHaR/YCSSSJ45fxV9bqqzD6U99j881eElYa1OHj8zMBDH+Q77UaqrnwxGxHyvU3F9aQUhGuzh9XnW1zb8AR8SHKFN2IM5D1OLy21Vnk6btXIFfAlagsajx4YJwK8+Eo3BBKe6KHkH1W7btXsPnmW6S+iE79MHG60QeKpNPCk4BoFPzsx8uXCXNpaWAJkTQriSOHkNAyP4pe46v1H8mKXCzXKzYENWMhZiLQTkSo/oej2blQHGrB6MROhYdRVZmJeyIKGdKA4wcaIQlVCNIol7WHiIFIyk/D/EZWOUbJqk7eZxZpdNDLM+0JR+R7lqvdRo0XkvyEHkXF+sTbAEUrBRlgD0AC2Ev+ZW3Omw5+JAFv3ggx+kUFcJ+PD4I248JDHcbUX8UVhwBpqGvMZBDSkftAAioySnn8BwMrtk0quLS+zQQmJfJYEJeZRRR8mX3dYGYxj1BJ0Be75ekNBv+r/TJfMPn2z+FfbY6SuwDEw7KX2sfMpNTMyeht/vjkQ9rnB49K69NttGdTu9VV5yBOs4W5JbdXOrTIVGg+LQcxvXZeJM2UgG0kTzLEAHeUIgRXohLYSsNUtkfVNgSJWcvtBjXi7457qdT2SZQb/lHq7T9A32w3TQCMEiaubuG6TOnOHVZm4ZLYtSmrPv3LPzUw9fP3njwLzkxnm9kVasvTngGGCxXqqrzIn1iFm0evA6CRcfIe7mnU1wznWd3jl/HcoZfX0XhEGyL59tNl6F6M9YESUMCcE9Ei2EkjyBfWD53z67cXUemS4fSno6evdUqA3vCWUXpkPx3ie/9WrHQPzgkd3Fmq3b7S9UN4fH9y1cnSwXt73+lsNbDgSTsb5xty+aX8+TRfazP/Fjo3uHL14709kVGTt44PSbr4VhoSgIgx0f53ZCjbUQwkxQJByCl9rI5OvpvAMrD/Hizs2+oe5jr79aqqz3dHWjoSAnBqSiQbrULeeP/uiPnTp9rqcnCrd64cIFfHftYR8eQFSagalOJEJ+H6GncQyJhUKJj1UWAmrneqE3tZdfeWluYRGChdMDNYOBo3yOdKsFD/CNLrKFF65P2VVRzZlk2OATeDi0r6x+5HXAx0nxHTJvGgJsOCJ8RHBLFkLQVJmNY7AEGIODG5PIFbNUuaJFzpTcUG1Y93jJ8ibekDUI1ZP8BBIBK3tIUmzYMHOb5huOnHZBdeV8De0NEsJm3ei70BC2tvDEobIc4bOItiSV3L17VyoVA4ciYKDOBVN0DBHyRDBJW726PTo6oiVhs+GHnEjE4UtmZjuxHZbKBexQMP+ATTjs7+rpXAB5r+Sw9gRToWeeefrn/sePN7eGmlvrJO6ZnLl28eKFRLyHbNJU0ykVm/NzqyvpfMDfnkz0h/wbgW5K99bCId9KYQ3OYHJimkHu76Pqw6Qv0Bn3tgdsie1WqD01vraKIkDlNIA8zCKdXUNnL77VP5i8dOXUwHD37l19f/2tP4iEumYWrvmCgU9+z7/ZcsY//7Wn/fbt7374fvJMvvDCS/ZaeW5iMl8qvnzq3AMP3vXIJ/99evbc7KkXZlZmOuw1H8tfgUCoWxVqgeZCrDUTAkwobYGNmh+N7ZLP5vSQIsvDqJK+DfSyWVksg+08yU5igvJF8u5RS4as4t8KpQ4+88LPf/p7fgHwwbyLkOgipwelwkA74vf1h4gEY0S8LdiDN0tc0sZS1bLEwQotC17KzCozb4LcgHNQThsmWNAyCxmJB10ugIe9FvyM4AIQ4c5Ccsh6I2dBFAsEsoJuA6fF2elJJEycp5BBALSRweF4LNmV6xoaHKUd0pGyp35GR0eSxClkCMfAqlWJm/8W1bq8eN1DokgYAoFIr68htCfbU709fUi8iMi4X1GnGQx8y61HM9mstJmyZQGWRr1jDHuwizbsHAJ20iehWlXIh0FybXzJjhMaYCdKSHExoxkQnTWISiyuWULiW2F7GArJF9zOPDGE0CmxtDgo0hS8qlaSHCTA9bB3IFBCCQm1RYA0Kw3qJCTIRS6JGsFZMCFQYJTBPAuRRw8u1zhx4Sw6EUXOY1mmTdOs+gNHTd5Y47apbCuQTx5meqE9SLRwsdipyTKIcz2sNpq3vr6BkeGxno7+1aWsF/8AymlVYdFrOEhSbx7azyaREI4B0KgrAgHjWVskZNss2Cq5MmtleQX7kBeUGAqix6+VSMxNhISnaa/Cj8Bc4JzZ2Kw6/FB9a3SEHdhQhbGHhUBOFmibl4hIwws5qps+rFU4SxAOtR1o93YPp8Ljnba+sG3hHEmbfcRfYDciI8x2DSTGMqQwNQiJOUFaAvhkQ24pXYLKM6BrRtBHaEAeQS4gQhA3QpzVmUtQF58L7CKv48jCThDMJEhMkQJQ/VFvGXMBz/WNF+lj3hF5RcDgphhyEb/rvIR+mu3GwfWfNKp2zXnzghuNm2Ex5/UsahH2Ast3bfw01NQ6z1RyYgeJ60FRWesG0x+1wgl6pf4LaLSZe2TvMKh1RzIGpHTecCpEZivjGaYhIvIauKduUa/GMTs1t7Bic+O8gqms2IoFPETme1Bgtcdnluf6R/q6B7quTU33D3ZstwWAuNm1lekl+eFVbE7Un81K2Y89LxC59747Xn/j5VjKj6tIIbsRDbt2DY2/+u3ncDbATgDQ4j2ApzHSL6XYSuV8M1dHNvWRRs1BXnxYqs1MoTXo2L7jziMwhKhbcAP0uv2UEST14cGD+3/vt3/npVde+8//x2e7OjrmFzCPUeYIzyn7UD9kJopPNlBZKTeJBnE4fbFYcnGhQO40rMV+Ehhlc+sbb8ai8UA4UshlgwGijdAIOwFsTF7oruGtBUpNsuwLcjBWQYnhnVmhHjRHDLIADdAyuBT0IQwif2MQDS7MmKjF07LKgW+x/IqmY9j5as2VcAczYKBIorIWCrNrSCq5EEnWpnT1umwMxuhsXDZPqVYVjBqyDeox08qs2xLxRKm4asRfnlXYApkPAAS/P5DLZAk1QWonVBR/46O33nr8zeOo/RCe3/Oe91Bc/Xd/80+GDgygtCTtEbEoKCTvuOu2+bm5++7/4GMPP/Sf//N/7uru4DORHfQd9q2FxblSvRWNubaEYMq33HpoZXU2kfTtGd+9uDT1kQ9917e//W2mw+txLszNuJz+gMe+sbbkJfLE6UBTBnXpbifSdx2nWVzQcWA++dabhFhBsdfTjade/votR97rD6H/bHvr7ZOf+PDDLxx74mvffKLNVkl2+u9/6OE3T72EdTMYiW7kssO7B6Ymlvft3t3REYMpeuv0qYNHbn7tuWeffeGVm/s7SUKA3HjqzcsPv+eWxnb1zJlTJKZOBai114fruLuy6EIC0hLjz+BPMzvMKViEmAsQBupFkDRUjwEkszFuO41ilUVBBCZ6h9ramjOIWwBIJOvyNxzb7lrxijfs+fXf+Y//85c+16wgAvERBDtJ3lNcJCIWIpfhdkFPwICosVY1AW/SURn8TiU6qfRwg4Jmweq5UPW6/XCBGMuVq8huo6ZNJBrGkZvlgAUC9ylvLm0vgEWdeEgwrchFNAssZTfW89k0NJjZP3P2tC9Azms/2mPIPPT4yC2HCVV6/vnn4dLuuP3Wvr5+YgF40BhebUTFE6pEJUT8A2gBGADDn3zzBNwAE0c1VtROsC8IaSyfmIvywd7VlTK2Kz5R1j+IGGyhRgClHHDatoUvmbplmFOkIQus+SmejwFAy0RqSnzsDN1lxAAYrTGgW0wHvmOsTjJVV0HoML3wxPxmZCGStEoyTAYTjIGiE2YElzakMZwkYO/VDRYYSl0sxHJGQp+Pk7yigzQHoHiINeoNDFZtOHgHeIq+MQeq6qQO4JixSe5x3AvJxIjETPYSCCRwgxaZWlE04lAoN35TWqIkgsSAtbqwAQYCTQ71j/R0DwR8AWJtVibWEpEUrxPF0UsYFhJWNLTO2/BQEMZjrPGywtTTqOar2ZnM2hUyLYCvuRVdMPgYiRVY4uNwWanLTZTcQEaSIq2GC/e5MhIqVnJkdZknlPME9oJvhLXgOYgoPGa91ipu4kDmriw0pzc99b7Brr0Hdnk6/FutfK1t2Ztv24xUuA01JW9H8ID5toYLLmmTZJEwKkS/YEFA7cz4w3Z4/QgIBNOhV4dvJL4ItTMBNNtlHEyFOegSPZFLhfAmqu8qw48nAf1jsIUPmWjs24IGawMVGl5Jv/R2Q9JAkBos7hX2ZFBoz7AYghNDbEWhNWvWT61t04B2bIAi486BtRdu3rlZByKSYFzrCUv9aGe89DYNMUvXao3HrebB2Vsafb2PORL2Fi+N3kU/uUutmhHiuu6R6E/PAXJxmLj0qxQeq2a7VEGZiQonwKDjz/z2hTyTTJle/EEgzY30arKrNxR3FybXqalca5HXvRGlhh9Sqit49srlRmX7wlIOAyQCIq4BcLXN8oorETzz9qvhmLtSznV2xW7ac5BAz1eeOVFAse2GI3UkopRxbZGrgPo2Ia/Ng6NBMCCfTywCoC6fVl0qYJudnye7ZFeqY31p3b7pJJoI3XIq7gW/1DYz+w+M33rLwdde/na1UErEYxvFCgE2nanuUCgcCSeOn3irUq3v2Xfb0MjIP3zpCx5vBBDF7MUIhIIhIL9UylNYF5AoZVmb4mRRrRBpywLECUtkGMbaaLmYlzoiCWBMOhdj+BU/qXwaCMM8RCKFFk5b+DEwyKS/NrY0JHvAxe4nqRC8NRFRICb52sDooihHYWtUzca0dIPvY6ZkClG8PuK+3yxUwB3UJn9FfDlokBaYQfqGcI92CwxL7n5QNrgFTMWs12so4SEZRIuWOzrj0NcwZYLSG7iwDfb2kath+vLyRz70wa7ODnB6pVTB/8NOHcYGNXcKdm/f0TuPfPC97+VbBwYG5uZnoJTxZIwYkbXsGjqmALELJPPH/RXodaTfOvXy+O6h+2+9mSzBtXx9qHsw4HEvz00QJoNKf3riXDhUp+CU21np6eolrCsW6WQKgqGOAe+BcvRKoXIyEI1PLy3F27suTp6eXU2TYGctf9YdHjo9G1guzVJnvFBYr27Y9u4d9167trxR9QU8oVD7xctTu3ftLteyVy+eb9tM3HvnQ72p1GtPr05dWsyefctfzHqa6Y88PhaJVgr10p7+lMfZyi0sNsuFCCSi5cOeQNodOCIhAK1mLSRWJ8uoUaHSmT6TK83sdq5QCUa8vnbSIGAHIPVIDS1KG9lBiiQRtAVwdd6GcKxhFAMxJvYe+tGf+NA//d2L595cGhts31guEnWMzj+XzmDnECav5BGRiXUV5QUYROYhVAGofGM7X8OLzth/mXdADctQsVAxfICJ3/FTirhBpaZELE5PgFjSom1kNmCgibgG/iHta8sLwXBkeKBndmE+EvSPDPZPkko7R5Au76otTF+E+iKhN2oZW9DV1x2bm748NtIHUunp7EQ78sarJ/CxCAeipFvqbldcNeBPFulV4gDJoF0qu5w4ojlWV2bhcdt7qf9YXJsvfPOrWfhap5u0XoaCQhRBPuIoIU3G4oKZFjEILAQ9IC5OmImMnxp04SO+lkvwIByzYc3lBg6sMwA7NyBDkaGD86T/gWBrM++CDEjFJJdb6KwdzRQHXES6ZQbBeLQKYw6/hZ80N6GkMj4nvFRLV3KtMDK0hFXH7PMwDXBVhEEvwV8tk4YTYhlYbAGe4vyHu5tclEG6W5AXv2fb29Zy1UqqZ3Hv0YdS0c5UezeqZixHTfn+eQORCFgARTroBjLP6sX8Br8D1oTch4MBlQtEMCUZ6EYOr/OtVpkBglwjACBkGbmdnrKpb/CLprtyY0DroGP7doxMK9Qzq+SJMA6GonApxVJhLZNLdERhKDdthEBV8dSp2ymDs9UWbR46NFR3Fyja4O6u2UKUuM/b2srbqr2LFwSfJqFZrIiU6TABxkTNAPNymCvNHIMGh9Rs5bL4WIH02FPUExUhrnEQ4La6CzjA2xy8Jx8SrTY+Rzo6SBWjbD7HQn2iYdDgd87wfdp0D5cAD0P+eKveaVE063nrgm7THe+0uXNesqYaYjOkklV1nTDvPMJD1h07911vkKGlPxaUcY6r1xsyfWDIBYDqG11iTwe4A7ELtYgIrZ41EMWSFq/I0ywI3i5jjMi58soxK7XNClUTpOJxQIBrKsthk98nKpxtrlYZT5hQipbsGh84d+kMmcH6ejqwGn79q08MDQzuHh+/dO3KRmkTFz8bmiCvqFQ2sw7H4wsn4ArxuCFdEXNAXlJImmMTX38fjLzXHfZ77XOzs9BdwgpBJz5nAJtPCDu/h1jYYq5YgBKFIr6JiYl8Jh8gTUIw4m55igRIScOx3eYLF/PZn/0v/3VtZYlASVLm7h7bnc8XXnzuzQ998NGPffS7X3jxNeJ0z50739M7BClC1Y22jeUM6JSK1EC1kbWYQHCJvjjDA9XSPxuLlTBxG3kBAQruZ4GjWTPToVlmceuk0mDREugTmYl8rqxgWE1RXubH5YUT1SkWuBg4Vp1Aj1alVmMeYNlpUPOjjX+EhTiC2+R1lE7F7sig3djAVyBiAw+aXGYcAsl4UlbO+sklC6IFK2y02Gr29KYKWQa/bXFxI97uf/7Z5/oHB9A2t7VN/vEf/zHMCpNBobpUJy5UNYff3j/STxQ+eWgzuXSSuuypBKFfOGPCh1FHEnOp24MDE7QdLso2MGjv6QuXqO1Qzs0sT+NKEvSGSGoLk37k8CEyoizOXbv/3tv4BNQjR28+vN0qwqA7nEHSCrRteycqZMxbJewhmy8nUh1nz18gaYQ330ynG+NjNwfD26++8Uy1mU8lobcOZOilleXe3lHipkqltmqtrVrBIWhx19gBUruTB+atU2/80+SVVCAxkuo58+0XD3Qkwm321fnLfV24vOTaKgtubzLmrGOcDeK7Qq3lYtEJjdWSZ0VIBNKyFB7WQDI1SusPReUkQgvR10SB+/AEQMgEM9WcPlfYD+KDoG+5hLKKWMcblcmtfKP/lqP/yvveP619aeryRmcCQA/OLy6EvH5KAS4vZ9sTeLNDiRT4JA2nfH9JN4kOyIWcKF8MnO2INYcRFepg2RLTyo2kI22BtPHM8jhdWGplKjWBC0iI0C2cGFpbeazaQAXTnU2TAljeOVBCRDhkk0a9Ao3c3CynN+rra4vZzGrfwABTMz42cObUyeHRXXg1AnjtiRjeDuBPBGI8qxG2KHI4NzO9tr7MAXjk2tWLKE6CATQa2+tryxThoHoDKTT4E49gkUxGjn4ziMA0yJZ+80lI6Pg0ENjIT3NVSiRrAswcQEK0DoBimBHz2SI+GI0ZKTa+2KrMxSXNDONkQrj4ySaczrpEL27U2twA7kDM4l6LrEMcjM+j3oj6nquGN9Dr2ABLHiebMtlKLIxtoW1AAzENrKFmAYEGH6V3o4PFVaatRbxEEANFtVAXm+aJ7OrfO9A9vHtkL8uXegVgWvgrgRUIhNIFjRqvZpnA36EbwT7KZead8kRYdRVsUSJTQpqEfBAfzD+bW0FWsYZSAhmUQy7FUsFSLoZPhqChaBclZuSE+wu5PE400ahy9+RK6yAjj8/ZPuDLlhealCq3F2ttJRJzh1IB1EfB0YTNn9m2B8BRbW6qd+Hog6WgrnAYqijg8iDSq1ezJDTkROIRDgT9pSOQHYAOnyoEC0RfClYDd8jcMtEDbtLEyK+50QCrMnRKNgS/qcVkgBo+SK0wuppK6wt5kX5oM/+aWd75bTgtHuUJETp1y4yK1S3Ty507d57SWv7OM/olgGSkhIWFaq171CULcRpW490PatChDHrC/OnB661alwy64E3W25hp8whtsIFEwBwixpg/Raf5TvZS0lOVAgPjFoIvZUmqKFJkqMIerrpWZSRjOB6Iihslnphf1g4ilCvr3nvTHvK8Q5LRJUbDMTJTYIdC7YqFFexAeoN8Ll8vQYXBzk2vv9Kk7K8N+2I1u0EtW2+ddPs1Eq45KS0A2StttaHOAnVRqujc2fMQYGCPEjyHbzq8/+AePE8vXL2AUYpbM0TDbKRbvojPiS1Kn4ZZy+nyLa1nA0SbhsJkUkQpNzV5jcp3tLl//9CLL764nsniLRKLp5ZWVr/0pb/nRbwW7RidB7ZBbUwIKKlUkqMTy5w1iAKZ1vle0WMZcx2w3Twlmg3ba5Y8exhAs/Y1O0worfF1eMwUa+BpzQECPGYiBhCOj8dxagEvwfwCABBWppB360YmSBTfggcBzA3A4E6+FLbf6JAAYVC/wFdn2Rv2igMaF1BJnbVDv2mEYy5ByfPFhj/WOHzLwbfPnuntb6cAEMGfwWiGzg8MDRLXCxcVDiOhKZm7+HKXhyClof4hsjd8/RtPfPfHvzsej5448frufbtRG/EpoGB0fa1aC42sP2Lbd2C4PeHv6iRJT5Qk3AS0AFUkQVS+PIcjlezK50rhYIzJ6mjvrVQhebmw2++we9oT0XKlee3yRLFW2j227+r0NVxCurv6iVk99ebV7p7xO2+/89XXXitU1sJxT7NaAlkSE7NRrN1zx4OYfJHJysU1UlrOzU4jSwS8vquXJu1bgQ++7/uunrrWmRxtf7TDW9h4zz27F6Zf2Fh5e8/oUKOSW1qcW5/P2yu2Xd224e6QM+ylsoeRI8AMDJs2jRvYFYUoEKCFo0lh4eCG3LJj/rA5Q+RYJCdfC7bLjlGCCUHv27RF40gEVdTN/iB6geVEd/BH/tOnfvvXPufc9qQzG/GOeIngz5XKYE8KYIbest4ohitqtQmNRInHizB1b0KLAVEyO3HJ6STxPspwKBcEe7NGbIuzjeRYIAGSfoO+fZRcF7lR74GEOrhZeQOdhB/NLSIHUx0LyyD5G7Fp2Hz+EioocDxQDKEl9q9aw5LopwtYf/PZRDgQWl9bISl6c7MMp0vQ0frK2uLcQjDky2bXES2TCfSqgdm56fNnLhKLz7NGAIByOTEpkUZCVJOu3ABifZ6BS60ZQx3Zc4N1npuRjcwVTmvjkrXxFAfcwGYdsAeBYSCBjrMazWfrfuseoI1XWBvHnKQFNlgXljlnlCxZG/Oq9UMxSwRzpDLmlrtYPRLR0Gk7EbV5wmBkUQe6AZkjGWkIlqdagtvFRgWxRAVt26y2xb2Jco6izlXSaIwO7x8aGOtu7wt5I/62IK4lGMAAXPzRmUU+GUExEYsi8VbLBVAM6kviGjnjQAhIp4tZ3FizVRxpSGiHYxME2U6uZnk18ypWJ1lRoX7E0QNzrc0qH0/HpIOFBoiSSRUrpwDVradg1pbTh1a9VdnOpcG07lppO28PbvaOdgweGLR1R2yble3qcpu72uas4quPvyt4z4G8QR5AZFg5/RvxlyG7MQHb1FDXh8sdDrACLZoMVvxUrDyUhz62UPsTRoA3PbI8g4vGA+ENxZ2EG1CSJhZGiykSDQdnQfSErYBv7XiXNvPvjV+SQa17zBQaGqhumG3nZv7Zud8ivTstmZPv0FoGibtoxXqN3vTOW26c2zmpO8UQauM268brt4s1ZgbMRcELB5BXXLN2bobQCmCBLDqKrcioV+XPQbwuxi0FFRCzWK2AwdGsytrBkyhJS42tUgPOUkyQxsqsGlgXegAeOP7aG96AOwgEtrk6Ex1lb3VycvLMqXN4HQNRSL9gsgApEZweInzX17NefwMPdeRHqgsw9KSAsbh4QnRJywyUsQjAzqSEnJ9bICtTLBqoFEunT71J7aNkd4okGKRFBL0Qw1qmkoe9hY8GWpwg2pVwzIaCyRnKlkpIgcmOrs5kanBw8FvffDpNbaNcOd4em56ZCUejK1cvPfLwY9cmJ6Ph8OzMFeSAEnnTtrdJYAmgkveYtRuOJ+XkIvUYgKLFAojDSzNsDJk1+KxgFjLkkBvAj5zkJ/eLuTOoAxEEtAaKRDsjQLm+sWb5RrMADTqCATd6L3FIYrl0H7NmAQXN8vM6DtF8gm04ySPANo45LEVzsxAaG5d4KY/oBtOWBQDsaTGR8i7O5waHG4+955Gvfe0pf8gXDHuRGiHiVBuURytePyoNZlteXg7FVc2wb7CPOGDyb9xy+KaDhw/+zm/91vDIoHRaRDqRvh26gG+Oi4Sj7liHb/d4f7TdQ/q8SDi6sLBEGTQahGx0tMfX1tZxGRjf5UNedIWCwM/JN06z6sd372J1b1EblAKKKB0wbBcrIyO7Tr556sGHHunoGizl/mFyYqWQKXzkwx/65re+lOwIFMsbFITKZvKJWEdfZ39+o5TeWErFO0nBtm/3vqHU4PTl1Qfuudu2iS7Yf9uRhwrzOVxMKaS46Qinc9Vyub6xuO6z1wYSoV3tHc1i0Y/TWas4t2RrTwiwNQFaa6gaJTWxjjiUtowD2BghIGGjWslGwfMwgTPRANIlBKxZaOKcBVUWU7+5SbgseTXIiFVYOd/WioV33/+fPvvp3/mNz5Eqrd6igHRbZ09qPcP4INWAsSC9TJ1Yu+22CsVZ21SHEKUJOaKVIo1+AW10jZsAHq7AArHKWAFE8+LBAMAQca8sjm3KikpWIaRF6hrjH4T/8+rqssfnpkyCP+QPhYN2V4lQJVQimImBWdQJaKRxfVirrWKdHBkerVb4tVGk+DFZ5xaXq/EU801mG6LqUZzjNEOCcQx5iYR85pky2G7TeRycHNB3kAkuF0QgyLgLXLLnPuFNIWY5WMGNslbYuMRH8xlgFzgRsLI+BGg1mNF6kLXEAY9zktZYYzTCGRgfNcjUCLNBIcykmWs7y8IQDIw3ollmBFE0sUggpbxDw6uEdhpWlFCScSWPsLpY6fwRiAUx4z9a5CGuQuBkfSTdObosVEHoH3xuzR9h2WRvrpa2o55kaqBraGC4n8pkXozzrdJGtUX5520HhYxAEUwe3kisCpLFVEsZGC+CqsBfqOzofh0fR8VN5mkLDIvBmlUEWsSoVcFVD4GD4eRLpApGJQn1bWJ1VDFANnWfoaaz8HM48jMmdZuTDMyo7EnaWq3Zig0HblalYIfvwHhfbE+3LYyuZKNQnHD5nYEuV6W0jDIITTwBMduOJgwfX0YdLZQuNAvrwUhaSlNRFqR5KGaT+sIqZA0Dt6nYJ6iK0mqKmMLdILBgNMerkQaYPhfaeZzCyawqiwtrijahwuoxI8yscMw/UF8zGcwSQ6+L7/zDGQu56QE2bt+5en2ezE9NmLm887DutM6864CTjJqBTd1mHZjbzHvNnfw0Z/Q6c4NaZbD15uvb9Yb127pdZ/gE8yTfKXyC4gAuRLIvvvckGMMqL52z+Ba0jVUWrQqu1RHtGAO84LYdleZ2odoilBTOsFm3Uayt5QITiAXkJtid9Y2NRDLiirjBKJm19PTsPKISEufaykahsOVx4FwBfJODnooZjs0KtLcMfECSeFwaF5npWwiLuM1TTIvwMwxjiF/kvGKMEWElGCrEUVI3UIfvAuC5sZbOpktweooZsSlAIowC2xdGpfzww49Ozc0de/W19a50Mp6gvgJpkGH/YwkXWW07Ul00HnE4nvn2U4FQ+Oab9kOAkTnRlYHIMGs2Gv6l+QWcesAmRqxH7qCIOqsUmmQgnGEUAgDJoMwSYmGUhVRg9mRFlxcnn8dPJHvUAtzEPUjHQAxQCkPEAmROzDkcP7QYgWHuZ7EwKkCfxl5Tp9nT/sZEc4QzrplpcAlcjAwyvMDMuMbIAAcNmjnfgVFOWsCqN7rb8MpO9LlOnz87NT89snsAl9eNuWq00xXw+FY30rwOzgDvEb6kPZHyIFjGXa+/emZ8vC+Tyd533/3/63/9LwgqGm7lPHS1SAa5kc9RNjmWsJFYw+PbzuTmA+H2YoFciTOrK9n9+26i6Pf5M8dSH+nvSA0szi0Hfe2k3U6ns8deeRXHmD279g30DORL1Xyx2tHROzgSmpidPn3u7aHhXdHIwvlz1yaurtx2y52dyTU0K6lYIhL0xQJumPlyqy3baI30D6MjrBSqsWB7tZRuT4QKG4XTuVN4xUxNTI6P3grnd6j/1qcnv3X+0sTuzpQrmtp/020Xj891R+zblSVvI9uGeyZBrX6bL+xNdtZQkhk21ppSQwAAM4ZYun6Qj4ZfI07FODhwXCI2baV1Mu013LEIE9+oFpln1OO2QFspjzaSWyvp9Qux2KDdE62sPhvee+8P//gnfu83/jaz1kjGktl0nUwjWFZw+JOJDcHXuPjQLNk3jAcu3kVQVHRGQCiIC8ca4FHOxaxCyBvEjFQr5NOusdBkkcWXBUxH3Ok2DkEo++zKIiMXWgpp0BvSd5NP29YWQvZgw+lFOhTuhvg3ZI2DIEZCvlqlKB6RhBYuJynDINv1GmUXiuPjB1lNgDprAj4gnSYhFyULy+R4AfBwkvCYcOFcLl9sYrYI00NpbAA+AFEvZNtBSlITQYR4nyQnw1TqLmFT3aWFYSCbPRs/uZN22HiMZcZJcBhGMhYjIM556zadv/Eus9K4n2e1DEhqgPCFZIeyj+TL1/mAgF3u5iximC36J/RmrL78cgJsBs+qY0J8onp0gSVPbm5XGypoIiiqPEoFhWQw1RXtG+keIR0MiXtKhUqa8tptnrA3DApmsgA2yojLFk3QLhwHQRT1WkdnxN3dLW+tTLakEIGlarHAHUxdELcXvgZMSVJ1vKiReQ18GlyO+AtGR56EhiGviyOhG3QftMBeBNi+Wd3MkVwGW1Wxni3W047wdtdILDXc6d3dadvK1huTpPXAbdYdtzdt5fV6KeDFGwGY4/tRFUiGxVyOpAS+A0eJV9LbmB4Z0mEaG9UyDjh8BYZrwAWlmaRv1ofILcICzeC7Ji0r7qw4DbqZcFLeoW9kQsBi/KfOiu4adkG/hAz1CdoEL+Yf7Xc264oRXYUntVkzzoHuN2TPnDY/rRZ22tPpnTb1Cl7En5qihXdfsh5/936nmXefeufY2IbpO8QVWguXKGgRv4GVQXYU1jVnILr41mOdR+nZYoniLg6d2MLJgwQ3WHlZ8/xhzyP+GQjEa40skqUabsLYaQJ2tBLw6LQONNNpoLO1+dDd9wDKxOqkl9Y37Bu4gewaHidT/MuOY6R9h9UmvmVtZQ1e3OMJinq0iCm043WvuDstO6ymMOHI23gp2vyQ4UqFk8S9KJ4NU1ULJacX5m91eQW2GscfLdhqoyMeB+jwUsQOCd/LCsZBgfU/Ozu3uLRMnttrVyfmZ2aHhgZIWcw4wRNQloDqBbPzc5093YL9evkLX/hbyeibmwT+s0h5HvLc09MH587qkOegJH3WtWwWLHDmiOLGLGStZZCwmX3uZMOLmqtI3hzTfzNCEpV4XsZdphkZ1wQuoA7jTiRLbuOA9wKcbGofvRRKxe9UQVuwQbPcjOhvdYOTbKwDcIGm2gCH6Zd2/OQ26yQtc2C9yOFxjO4f3chvHL2DkJ4NKi68r6efyoN/8Rd/MT+7QHgSdQ54nHHAKfro0aOvnz6xWbWFYhQMcPz0T/3MH/7hH09cvTY40JfLpcFXMLPQ4HDEHU2GuwZShNiUGyvUoPWSA5AiCSCgAAGzTnJv+b0dHnc47uyZqq36CJxqS8wXNshg/fhj79/aLM7NLGMMbU/02LbdpOa4cPWS1xf+xte/1dXbh3RzeXLG580N9o0e3n8gn97oEgkm0RMV2rypeGykbwABJuwOkazi5gM3H3v9m7vGh2CewqHk47cdWM5Xn3vmtRFscLsGcTe69dCeiaVLNw0N7Nmzf/XiU6FN0iXZCFNhfsCi5AUH5UOaWIcMqfCehk5Tw28p/w3vb/CFQtWZVIgydBMPqvV6sX2L+gMBN5VQscTiQ1G34QbKbTRKkJvdlQNhEVhia5xpH0n9l1/4zB//3pePvbDeGYuQ0SYQihFELJ8UQMCL3RezMpwxLCqh3lQgxkAszhjJmMqEHrcfSCbwBPjBDZ/1aKhvQ8CGvg+1CDy14aEBUhApOJ+KdegwNjJpuoW/P/AAfEL+iI7Hq4EM7FBIiGCxWAKHhoK4RYdWlhfxqMJRo4CvlsPb19tdLlXRYOG90Z4k3hcDU4U9Miy2fAyUlt8fqzkaiROth3MSSBiq76SXvI8NiLQkVGCRDT6FM4wttFLMrPA7dFpUZEf3aD22MwOSmA0ZFQ2m9/wAUpkMefgiJJqNd0EruIGr1hkuQDOsFyGPgQZNWAvEAJ9grRwWpogJnAi2A97F48Q1Kr0O/eEMyw4tKpNtuG1U1woogiYT74XxvK1e2SQMB0fQaLS9n7XU3nvz6K04X1aL9Uw272pzBd1hnkWWJqNYrUqm7CpvwSjL99JJj8+W6Oy1YdfLZarp9ZXFpXIhr2on8Wi5VEBmpp8IlDgQQ8GwiqEewLMABhsZHJWtYcPg3qUwME5RcOR8mBQCOgB5OqrueFumslCuogJyDByMdw4lHCmnzU/1zottwU1nZMtjr+N/hxAtQ28bMhCDQU1shU+BYWRnkoMYVMVIXRp0uiSXXbiJbRSQZdYBJIezituCTWHMGVktDgszmXlkOjQpxGmh04bXVCchJjTHtPAF3MSRSLJWm6CCHV9lqKxUuzp5fbOOzR0SwK0pNe4ConwsaN2g2TSbdWA1eP28ntV5S0JlqMSxWbdzXlfVK216tcG0VoOMkhAGrAX333iEFyKW6k7hXsgwo8SDSsGBOkBjxzVmjFxT/ISzJm1VjbyJW2SGl0MVQnAdg7yScFTRaoqxERdFkvBcabNCCji7zQfHiVu7ncjInQHGZEJwasDjBz2UqPBbqhJChz8FvlGvvHQMe+fo4Nijjz5OuPnLL75CUkOIdHZjBY4OTRusr3IJKVxNLCnzRTUTQvOgxAR4oG3CvsvAIhkLk+ES6lHQAR9TzBZZfczb6NCuUr60srReyhcd265qCfnY6fOHofooVI8euXVxcR6qObZ7z6VLV4CNufmVvr5Ocj5H4jge1hPt8b6BfhwUahRlQ3fSaEQjMR7s7QvffMfNTz75ZNCHNQtGkOxH1BXB9VsUV8zLDhOuJQ9Dp5Pi9VjjaNS8hNrzLkCANcKqxzZEQjiy1ssSIDzEjTyCQpuR10rkHk6h6RIqAQGpHVNUdAcALBjQ1LNxwzszbm7gYV63o6PZuUHvYAMETN8MAHNk8foe59TMZO9QLzk0uvs6WUO333XnhQuXHnvve/7wd/8QZTZ5BKG+OKK3tdXSOfKO5Gq5Vs9APwD2/HOvvn36IorKbKbo9ngJsOYN+M2RTzneGY61Rz1+Z5SKv+Q9y+OYgzd3KJspfO3sUx5XtK9nV6t++eCBQEc7xaZwsLd57PGjNz2wJ3nzWn1qdmaZarLheCKNu0BtE/+D7r5uwmO+/dwLONns3XMIbR/6PIway/NzQz0dWzgGl9q8Dl9XvNPT5gh5Al3JLuhKJOyOh9uhXmCHSDCUrWZWljY+8uGHYzbX0xdembs2d/ude8qOVq5cSxy5rTTxYtxWdzerYDl/yF1puYtbJBmFnK5L2cbwaoSFnjWYwOymQoEZcNkfQBPSWoFnJBNTDLVZthU2cjH41jCaVQ8ODbg/RQMkkWRNUMTQR0B5tZrp7NqVXjmeSB30DBz4gX/30Xz2ny68lRsb6izkiugRid112TERslox/8IA4BaGzhnJtMKaJp8JsMMZNHpcJbyN8r0cQGxw4NWdFCrG81klpUk0JMwBYgK0sJQjCMPXAiqQTKJ7QXxcw2RLRVjoihA7WkdEF5BEs1Ey1TiYphqGyVwBEc7hwEKfgk0mImZ+YRYI5Y/oecgdzDHfCgW0UA8NMSwUluAnPSRJiCRgYVkDjvAxXAYRQ344wUlDayXLWjRSVNiMLrdxkr21ccyyp3We5U5rY0mA7rmfe7jKngatSyi3OeApNs6zv75+tvEptxrhs1mcsEXMFSuB+01/qDJElilwEnoqkWNGTcIwHYR6MFYK/FaI0XaDLFAkn3JEvWEqcgz2D/X2DCTD7ZmlPCwhvBTFnSQ075AE4p6LuEsEQ2GkSDQGgA9hJLbBnu0rb67NzK+uLCPNhAL+RFRxkxvryyBc1Ld0DFkUnyxx8eBLEl+7JIHyUzACUZSoLm8BcA5cF0G6daQRZhO9MX/OUiZ3LdRu7+0lKxy+GQFbCFG0aGvLARJiFp2o6QhVaZJIFHROcCkNYI2EoKLP0exgfpYHHG4XiOBAP55iQEILUQO/D14D8wF9AYOjPBC9lIc402qECC0iyKoAF4ygeVeKapTRggZGG4aJvcUfibCJdhqtn+bNLD9z8L+zY5YFDyKjppHrmNLMuwDgn20WPHBSHTBk2IDJDoblqoEf0c13HgTcaNaojDQUFhowLzKwZyCXO5hxTbp+0jSIAzovZzOwiMwFrDEHugyyuWLoJYgUb2cqYlNym/T+yL6tbTJ/ipwjfJDPP1+Sks1OHk+C9hWSDizCeolzAEN4Ha5LZ8/jtkPeduEED+qLEvwDh/A/3b19vT2DqHMvnL08PTULvoKjw+YEA0UuRRYMGJ+FA5XVIJjvBPsjqrJm+Xzu1/ltW7mIsNggwJ+hLeYKkGfOJ0JxAmygvtjdJDeXqwBLIBgl/IY3QlAh/FQnff3YGzgYQv5DIX8Vu0ubjew8PI4n14FD+8fHdr3y/Eso97iTxBTo3PbvO0CB91deeWV5YZHAB6QEFiBwgtnI6IKIhCceSauV/uJ6wwFogXvKBGOYjTMYbvgEJgWQQ3GKT6ABNHCIWHNDgHlCjfD55rsFikydWC8I/HV08c7UG/xj3anbzKTrJ3QSdslgB+tmXmo1ywiY6xpX7rfO83Y+f3F5IRKKJhIJirq/+PJLp948/Qe//0dXLlx94mvfisbIpR8MJP3M6alTp7q6ez75mX/9d3//Be4/c/aCLxA+uG//NbybJlep64Dmo9qsZYrZ2aVtX9jTN9jdMxQy0agd0FHUHhMTM8eP5Yr5ue2ts10dgZ/7rwNHD941szgb8Xfa24M1fyVDkGfLk0z0xmOd5y5eJG/t4ZuOkIr63PnLH/qu73rj5Fk01RgTEUiOHjr66huvBYNk+GFGtv3USfUFezqHyOvcFgGyyYthp2pPNpMZGelBjYEAN9gXq9dKufz6ZHY20h64c+COv/6Hv9ndGfY37BtPfXMU7Ii8WGYB2jxh0LANBKNUYeh+pNnXJjg3cKmibCxJfrEGDY8LIDFPcGWbNVvAK7mzXtrKbWXijigA4WzzR6Ot7HLNFbAFY7Z6oYoTYihkr1YnEp3JlcWTMX81PHr3f/zp7/2D3/ynYy9fGepLADOiBWh1wKegBflFQvW33D63kjkTwuz1IAux/rU2txpB6oGqPJ1ywuD+ZSadYhLSFEqJhccqXyLijSWYGpo1YBPlM6AOLGxspEETJBJGB+YKuhDZYQel4FamVT8HlAUDPDiZyxYxQnOGiPB0OuMPhpdXNoi+gYIg4xoXRlw7ykQxoZ2mlDXZxHHjAnjkJuyHmSu1/ehQJ8DHZgbUAkSRWzrECFqEEwTFDfoGoTf9AalsfJ61MLiN+00z7zTFMmYuMBRZ57mZjbdYTXFAgxyz0iwmwIwRpaS18JhMbtbKNGZjWClexHCwodbxuHxGU61nZU+iGVEU5pU/Lvk5yKwU22Op/t7h3s7BWCQhZZBcfDcDngh5UfgGNagZ2pLnJdoGwraBHQQ/aDg6giCeM36b2zHxxotYW82al11JBIsp4k65QzOvtAW+VpwSo0M7xWoOrgc9Ei1hvcfNDlqMR1llk1RDBIG0KvZScTNbBRG7mjZv9aZ7hjyx7UDEI4c+e2lru9TmqXMbSG8L/zJLlBZgM2CgdQJbjK8UfaCz/IF6DA0yeZulM4fuqlykBFxRWYaSsdSzrELzCAKWPoFMNownXggsKyKuwYTwoiIBm/QWkIJxY4pZQxAEVdSRr4D6wNRYG43q/TprLujs9WvmDiZXfbgOWpwTABhRaefYOrPT4s6zuscipdxkHJB37jKfqYvmANh55zbzOn0T7DZrXRutcWC1yQBYDCXrlqlDdSAWQ47NYnWRF4FjiBmu4Ma3rYK8iz1CNmDWIwIHFI1wPMgrc1mBtOP2tkV8WKtSpcQsdl7UaLZg1O8me72tLU/hPVgsJWYihkjIkdVBl1Ei8zpx5nj/olpx2AnHX19PUxOQyUIwJcY/n163NQq4C8AwAW2aL328voufAGyyvcNSQUOViR0iCo5iL7DneACyalgOcIQMoCCpUI6EQrwMMoxtFXUxaizEfcgOIY+sOOg0IfJMNCSZOWKa8T0Ba4ApxkaGH3vssV/9tV8dGxop5Eowu6wOtNYQJ6jOffffi5sJHeZFVBwnlBHHE+zHeKRwAx6hNMWK4iq4h16hr6Z9+ACO+QRRPqOkZy0DZdBt+m+4Fun9ADlzvm7MQPpwfvJdtMY9tANAGb2MYeUNdbZuIHc094BLGEwmF6BDFqA1Yt1p5F9udIaN89zNxjGr3uFvGzrYv5Zbg8u/++57Y9F2UnI++sh7n332uZ/92Z9bX0sTWk0nWemUsrn//nsvXLn6mR/5Dz/6A//x2def+61f/1/9fX2XL1zYu2c8nVnt6Gi/eOXtiZmpUMIRjgeyRSqg2Eb3+D7+qbsuXz2daO+qlm0nT17Y2vZPT2qmKNh39JY7fuxHfhKl5sLswpFDtzLmpOmwOwp7+kdeu/wGqtdoe8KHf60t+D//4jcPHDz80ouvnThx8iMf+ehdd9wNPgwFwhPXTvdgK3Mxt8R+Fx1twfY4ubT6NrddoYCnsVUkOWUmu0xITLnUSsaH4sHuM5cuTc0ufezxHzg/fXFtfj6yVc2ee+Vop72zMjXoK3irWcpD231thZZ3tQpiw5CQc25j0TNrWXvAVKPLWOLzwrBbf5xhssA+Gn2UdpbuFzVDwBZJetwJnw1re1t922Nz+qRA0qqFInowIgAN4WY1at/scgT2NVdcf/Un3/yj3584vNdJn5HKwFohkkD6fKhnVPZDzLPkVK1lkB5KJtS+hPZQnZ3czeIaoGJkphQUIe8iKuOQasBEn4B4wYQS1ITsS0+V/jAWJgM+nu30n0LSZCcGU+hZMCvezqSQ8noIP4V8gmoQqoRkSAyAQxkvq6Ncx0MWpCKBhna5jCBjQAwsI0EU1RcbiJGlh5gklTKbBsvs+R71R4K7AJRjzgvT01fDhoOmtQjMpid3nlLT1keyt05aj+Neb3G4rA2IOP2hVd7CXSIUmj3soyw8LklO1y2Mqf5HkwC+5JQ0GnR6k7Jh+L61bSEJElBB+1zWecghgVxQS7sXDgnvOUqDHr3prrA/lkAB5I84t90UUFQWPNQQ+M/V5DzF+AR8UB0XFWaocY3hDWs5Qd9wEsRLptcXMlc3qsV8VDpBox0DG9IpoXAaEunlGzghCOQf+iKNy2Yq0Q6Ds7G6hqoF0wWan1K9nKlkwx2BbGNto7RYsWc88c2u4fjAeF+oL1iozNv9W5ueoiZxq0JIC+gd/aMzgDgF3pEVWYyBaAlsg8UAYnnjP8EUwi4bPSrny+zVNSnrmCOp67QExDyJ+pqJYryBVwl/UgKC4LlMS4ZC4ZrFvfgKsRBktiYljUykuIzCISkQVVwtragh/WMRQtrl7/9xo+OCqusbP68f7qzeGz9vHLz7nhsnzTv1FVYDOtiRxtV5s2mQrt9vARI/kUTNSZFzoxsAihEGJEySRscoVJSUQ6QXG4Q09EpRT1QI1Je9o1rbqiigF8IJr0Q1CqIniNtysKaxPyFaEXaIHZ4GeDWaWq8iwWSklI1Kg4TBRllyxPFgv9IwAzVysEJrgls1MbiD/QPf/clPQZM+//m/QsGrRUATgjOsMdZA84zWIIsSqok9iZXH2mbSgX9WsBTUrFeZF0R+QRVmAdopSu9TCVFXC7ciHO9sOOFjOCNPiKOWXse6iSlrbWMdkskokViYlJYIuHhW9w/0UVUN5WosGsPRFy8Km1tFT3kp5JmrPEKuH3oyMNAHHsRmvLi8lMDdtquTCtdI8PSfVcklJR4yG7gJksaL6Jsm5fpcwUWZkwyWzvNpN3AI65Ez1sZ5RoBjMSXXGUB+GsDgitCX1bIZHEOWdVqOKYySOdTj5pEd2OOLrHdxlU+zbmMiXnvpQihlu+XwoTePn3zowceef/7Fi2evdPf1F0m7vLmdyVC5ttE71EFKyMuXL3d2dZOf41d+/9ffOPZ6oVRbWqS0u3d5ae2jH/vow4/c+68//b0B8mply9v2Ck624ThOnG1n3r5CvpPnnz3vD+D1to19EOesahnkXkhnixTIc9qRoILIZPlMATtIV2/HmdlL2Wy+vTsF1K+kVzOFq5FodHVtY3F5ef+BQ7vH987MTGHsuO3okfb2cHrlaqms8FNy+cWjPY2aq1Itkl1rdnYx2u4H5eIxupFeK5Va6XQ9HimRN4piBs+f/vaFC1dQ8twy0h9Jdmy5Kp5wAlkNVS0qaK+bIBRXDEwOelEeYwOgZi+MYC0yUCAqpZ3BNiiCkYUgSmhj2swZ7iStgWzJTV8qgku/DfssUqkXQys4jdj6LSel2xwVGXDtSKtuV0/Xox8eD8bcv/frF8eGfZn1ms8VzBaa65ni0FDX8vISiZiYdOQGOVxIAGbBbjqVDYYyCUAirqTyySVUWCJJG/5MSn/AuoUYA0EAlgiMnGmEENAbkk2LUSoWZPDGPzcAFLN6RU1dDk8QzyQs1ZQkKdfyQm9KAIOUQtQCBJiciASeKUQHSGUwcK7lnSx/KDUkFKsDhI6BUI59lgIxVTAOn903DPCJhFzf+BwLaEUXxZxai0RYRqoMXHT4TFHodyRgjllvIAUgGDTBnTwLJYPFFj2jYyQWbtXBKkImwi0sJRKNiWuRowjpuinbCQcjRsMsJ00r+ASGY0fdgQ4ALp4xgFCSqZLvgxhJGMEY4g8iZyB4ADBBX3RkcNdg30g4EPM4MVkwx3gQ0DAUmk7JNU4fyOjTsvmDTSHxVCjZvl0rZ4D6QqFSKzNkfCNKAFezQY5Kvoj+mCUsGQoxjrHEQZUvQafBRzHKXAW38ibU27yC1FKk7sGqCFuNNXe1NFdpW3dG6317Y7tu6rB3e2ytTKaw6I8FkMdoEHdoKiVglUAkwxGNlAPgK8mYog8gGKbBdF0DiObZBCUrNxDuuTiPkdgSWxrwBDLGViiw4E/TSl0w4XJ1jyEzn6CdSbuPGMjq0GRi8mZ6zRTrLbpByBwXVxNbzDfSgsoAMbWGRBjSSrt42cm6+y82WtBAG9RpXdRbzUn2N87oYAelvkNE+e4bGxTTul//sO1csm7WezkniFKb+sk8QGF1pzZ9iIy7YpwYSKadPz5CB4CEQozMHx8qO0cVx0hMO9vonDEPVHBvxuEJ/tyB6phEZlSuQzB2Ass1LCUQVvgVUTUyn9mxk3p8pB1GyDOQgLuIPJNBBB4wkORXCsWDAziNsougRoBES8R16NBNrJq33jrNNPjdDuJ/kYBvkBONoAEnmqJkL9J2MhnBeQddNKQWAgw/znQBG9fZXCMfghJapOYJswrJ8lypKNKBdzFz5Pv3B0IHDh9i0qdnZvsGB2D6X3jxRdysELkgS6SBPvv2mbW1Yke7YUbL1OkTew1gk0eeJHwo3zASpzolC1K3h/dCinhW8cGFIqE3GCON0QnRQZIlx/RA1jRRRMnx4kIM9jDrXWp2uqfZMhw/387HwgpzlU2QYCQJ6ycSDoPJxmmxM9dJL5hKLRj5QcvcKOdoB8Mzd+oBs9G49dPCV3RGA2wIsN67Xfe3U9Kqirasrw/HqEA8kbpyaYIB+OxnP/uFL3zhzPlzsNTZlVyw3dfd3UmhBbs/lEx1nn37nCyFbfbRoeHZmZl77rkDpeg/fu4fYmNBRIBCOZ2vNJNdns7e0OCwf2ZuLpO1kfgyloyiAnj7VOa+ew9HIx2jI3vuv/NBTPXNKjbaCPGbQb+rWFko1NK7x3bVbPUTF9+8OjMB5By98+5vP/vi6VNnd42Og3Puuv223WMjr7zyYm93eLO6TJ6uaCyIxoLASUKqKhVHwB/rG+zx+e2nz75ENRcctklF35kcP7j7zt//o89T5zLV08VMDfb2PLLvloWLx6ZefuJQpJHYmutyk4i3hfcCwe4tB2IlIg4iAaYaseMQGnGzGl7RV2utmwOdZWCl7mIHH4qBDvWdUV7jGrPpsKV6A/ZYgIptm/XSprPpDuOJul2uliilSzQ5yBA/XLcz5PT227b6t7Mdf/57r/z15651xqJUK1ldWYwm0AxXoe4G6YG7gB8tGtYe7wXk4ACNwx8dwKyDSy0EQ/IkaePAejDcIoqiU7KSoNXmGWu1srhoiNh9xFMAk1YJwSEnZSAIOwv9ZqFLHgOSAUBQP4IWWJdVD8/E5mKA5HvL20ReBatiDMDhNg9ud15SOBACJblJTJ9FONUBC1NZ/Cn9NXFH1iDyVg6sjXaBea6bR8zYaxlou36LGXcz+hoHLRPhI9FaCAnCo6is6CFLnfMShs08GQsCK5Dh4xFxJmBHsCb38JsBjYRjIpeo76tgRPysSEBLIh3qmbRVs63yZjUaju4eHxnoG0rGO7wuf6Uoj3miSCzShaIYdzBGTD5vEE/qMpNnA3Rqa1FQkECl4socrsIivajmNrH2Mzg4Rbah0WUS6IbgR7PLsUaVCCczj0wwHwFdF3VTk3BaqDL5kADWO6SlWrq2ki0sBVKOkbFI356YL4UPw0qpsOHyNqOdnoYtJ26ILyWJj/TFyuhGY8ScmsQHCjhiMPgPAsJ7SVbF3QynDGlY7UhvVFOCFyYAEUd5yzH3wLaqp5yDQwH+IOOGbQXk1HnRaSg+Ho6iB2BXk1tVRF1O9m4ywJjZRMDS16EwhW9TNDodAmRZUdfXm9wSNEOao3+5WXN347z188bJf3Gw04g5b4DATP2Nx/UFusW6RLd5N3RXxNZc4pr84c15zmg9cpWh4hCiy+AZ/TOklxaYUoRgNEj45YoTxTkQ1pZgLoxzEGAGAKm1VietMek9BYaAnnKSoX5p2yL1C8nJsAejYgBkYUEgeniDqOf0x6B7A+84x5NjFnwD74YDF7QcEJb4bK87qb+W6ujIZfIE/IBGkD6Jsi3lsqwRLQmzIPXtzKjRSLFeMD+CFAZHhrHpLlFh1CFVYz6dMV9IF/hMKDG4RFIHpAUMDrfNxvo27hkQ4LZsNg35P3f+TJ5aC22OKjxzvYVCFekW8e7xxx578MH73/++D/zyL/0irRXzJCLwAfK8HcaUNiE/KOLQwLHk5+ZmwJiUK6AWPZkauQEHMZ/XB0wi8mqqQDrKSQmuJDhEEgZoS6N0/esM5tCk6qTZBNuGRrK3KC73wFRaAyKWyYI4c7P1lBmtHQ0ca5OfFsriKi1YD7LnCetO60VgW264cYZj/dy2Lc1V+4b9qyuVqxdWh0c6F2fPj4/vIZXwl//xq3Nz8x435r1631h3b2/35OREKEpKAN/Cyvxnf+Yn/uov/xr0OD03jf0PFf3QyFD/oYF8kRSkGTmaEVVg90XCqfX11d27x0+dvnL5ks0+kztya8envv+h/QcOV0qbo8PjUB84aShQtVEhDWwoGpqaT2dKa06fY2L+2qkLp/JwN6z7zc2p2alP/+BnyoXK0sL84Ej/tZnLq2tz+HCRIiTVnqD8EUk0c1n4RzifABl+erbbqUaMQwmxiyCrfLEQjymsFBaqt28wEPMl2iO59fSF+bPOZn3kwE2N1Qu5AivfHnbZyuAN3AwDBPKSIwZKKp0kwwgt0erTwGpQzcRaJ6zZNAwxVhmkLDFx0gjhTyS3x5ZtbaGc5HR7BJvuZrPcLBD9CFb347XDygZk3MoJnW9VLnldtbaI8wd/8hPN5te/+qWzfFuqN4kindoh5bIapDkzuzt9YCGAyQolEwBOR9BRQZg3CYrB3wAPSs06c40QClQZ3CpxEGyPx5gUY/IsxI06YqNwnG272ixL/9igCA6+RfhWw4vXhDmkNGckELlELqGxhCLCpgs74vIpdIO2SUCFSKlfDJksUVoAJqhIgQCwbPLUslb4DVi0AFVUSqpobRYR5QYIhEiNpkNiFW9iYJkLjhkt3gydgwLoBj2m7wTQRef4OPMCzohFR2vK6yXXi7rzk37xhLCneT2Y1SxVLQnxGdAlErspEy9Bjf6wN4qihkDmRnkz5CH/SDSV6iSHc2eqE8JcLFTXVjOhQFSaYwR/nMQJ9uZF8hkW8QY7tBGXRnQ33uoEjtVLuY1senWFHiPwB5gTo+LDhkCaFopGGujSsDJzZtD1ufTT2jSokDXoLoiKdttq7qADneVGeW19bckZ3urbmzw4vjs0FNjeWq1sLhU2i25/M+BlnlFN5PlsDSAgrbcL1VgjhADMaMAyCLoBATgXIJdkkXgJqAxAE7UCzAGBqlBfntYKhxXgf3VSMEHHuEALmihld7JkQ8bBXJGnNncB/+jYqRBlYAMXX8yEVaqlotMimgUFKlGvvHQTQgDM0hjQzSwJckUARYLElb17ozdcM29592nrDK9891VzUmes7cZT7xyI69BFoUcdGGFXa0d6eMBdV6y9fgGNdMY8IPDjk0Wk9Yd+XUSXYwZc9Bgph8UGDTbU11BiCqfX23AMwN6ttGCQbRg0CmmQv1DmdRFdlA8o4UBC0gYAlvA5EoUVZiYTj7U6xByIUjAPddRq3Gyi3OmrZC2jcnFQOgYXDJstEqIipeCTrGvoJcz38PX6BL7SzKaWBH9gBaRhtBVl/PXLpTAJJ32+IvADgRfoaXmageJZezQUgfyw+lh3UougwVAOtFZXN5XXxODzRCQWQywmHhdxE39aRPfnX3iJKEZWMT3e2Ch1psIUFKF9t7tK1LLbl6YRfEp9Aff8/Cw0NZ1eP3++TthSPJmoLZH7iLKbwi/MssRK6YdkmYLYeU0NFGbEAICmkgO6BX0WrbWWvxgjdZ5VAM2m/4gOSHhoctSOhtzKWslXarNGSXebjZ88Yp3kBM1yDyNg7tXrOLix18gwZ0b25RE2/XR4Eq7q2nyFvLo9qfZ8ukimdL8nyAi89syr3vYgigb8w2PxWDqbITHW/MrC8GCvK+D44pf+6oMfevyFZ19YWswM9Q+QYBpN74ED+3r7e7/yjS8XajmlPCJneKne1dsFt3bk6K0DI2tnLsxOTq9Ozq5u5Nc7O3o7u2PpgsPrVIZbnM+xPmWyjbHxscn5LVITT8xPwPUFo4FqNvvq8Vd2je17++zpe+66F2FxYurK/OzVa5PnSRKwe7C3XimQHZU0KYSZdvcMU8B9cWltcXVybmlyIzuHthFT3czCmsOZ9Acv7ju4C/ehqZlrXT1HnK7yGydPBapbd4wO3ffIh2ZOPZ1efAPNKh5SpJBkKxVqYXKBaBwF3frnOi+ly2w6yaahlrIKMNBKQJkpHMGKQfzVuiVda8m2sVxIECkXD2DfwGEApRMkQWYUEidgg8M8TGy9XJZXycNqD6X+/U99GFh48munSLUcjgZarTL5LPGvpjUgjTcZYiLkzOtwmYLycSAlnlY6YaR13ybrus6dTCjxS6AGl0vsKXAGimiQWAE/sBpxucpDbnJpkY+I7MXNRhkHJFIichucNKwuN0tugjMXYjGoCRkbkGMVgnTYuFUIUjAp5A0w0mDZCLO6LOBkmTTk8gBQAsQWHDNwrFXOgEOsM6YJ05xGllAk0Vxr4xKbbjactmlD3lv8tC6xXPmPY6sp9tb9wDr38JMDzrBs2OiJqLfWCdMIwTaLld9CH45KsRrwhn3+EKE1wDHIxO8ORoLR8f59vSnF6kHQ85niRnGd/LrxUApBUFSL6GTAgPZgWrClt1qIzbjAkJYVLTZK4iK1kUsZQrbIlSUHYsaVAUEQBN/xWrR2wgg0wT90TFK5+akx1jfqPGQRsNHK33Q13bHttcJ0OrfqDm8NHY317U0Fe90kbS6VJ31RaoLSHdcmSUvr5LXBnQxjbdMo5wSsfDHQZ6avRVzJDlmD1cLkiC84qd0gwTV8hSDAkF7iRAXdYCY28mcASQ4NmyHaO6wMVBPEJbzGjPNH67pbG7RTQqShInKWYAZg68SQtnCcCfvDVHog/DoHEKP8DIY8Un3QS1ET5keKZ1qV1uL6sjPN6pq5bsbNfJb1k/G5fsPOvzfOmIN3rr77PKNMV60bBOjqvkHf6oJGiOniqiG0gkUNnzHn8Bw6ZzNN4AIMHBzzjWqBP6MLlRe7sjVg9EUCbkCJIb3iWOE0akis5JnUHz5WW1WSYZGQhbkGPvhAVetiIEAvDAnv5V8YGl1hGlkjED20J7BDEFHxZ/SA16qbZvA2G5QSokAKZADFNrWHSKHMg3DDdFSfJx5KDfMgnwG9p0U8+ug2pmL0z9lcHn009U0Be92lzWgweIX+3QS4wBm0zwVruMX0K7CstZ6lkFGYUO/pmcloLLFrfHdXT/f58xcpgIp6+uzZs4f27T90082zE1Ok2YtEWHHGktpW4hX6IDuRl1g+Gu3hKFLv4koaLYul0ML9xRoBMy+CDrCBz0cYpTwImAs2LvFlHGhlm0091NRp4zw/OBDxvY43eIRjTrIi+Ta1cB3DcMzGeW5g0wvMZp2xjrmB+9+9WWd4OR3jPLfxLMfiX7e3x8b65mYXM+t5xruzu2d6cubrTz7190e/+Lt/8LvwAmjgydhAaRO4+lwhe23q4o/8x//w3NPP/t0XPv97v/17P/2TPx2JBmjtySe/1t6R+qnP/sTV6Ssn3z5BGg0Ev0DB7Uo3447Q0UP7XLO+jr6e+x+6/5lnv4XDVK2Zy5fXYKu7O/rDpEzaLlFOq1gp+aIJWsvks+B6lF3Z/Cpx3SjqTr51cmR4F5m5bj54YH7m2t994W9vPjzebJVxqWuS4SHsCIaDjHyumMVsgWqdVB65wgZxs05PIB5LYEalJkG+lO3rCeSrlY3ZK6fO5KLBUFfP/03Zf4Bblp71nejOOYeTY+Vc3dVdnbuVE0kChE1mEAYbxvZgTBKIMPd5LuMBRDKDbfDFYLgDKAOSaEktqdWpOld3deWqUyefs3PO8f7+3zpVagFzjVdX77P22mt96wvv9+YQXowsLE7PV/rNgS81DkzDAjltDQRHdG/CpbhWECEnOGdRNPlMIJ90Ul/MIWDXwUpixPSC4qGpEEMhCu6S0cumgKa6LTMsp8nsNRXDtGknnJ4QGKJ8gXJEbTYYNkwKn6OkGeYdjYvE+P6rX/hOu7v3sb+4mIwQZYkqxztytUC9AAV9Q9sEhUb+BIUppSnsGnsBzMYelxhKvwa4NCA000EeAdRAXhBjfqUYF/I9o0CB4XSgORAeQCzxhYligheEG0BSrYsIMAIDhowY3MioeIV1MBP+Ifpz9pl1gfas/ajXcUr6GT1uGuFxLgJ1ewcga0G1BalIwNyq5y1KYzaA1erX21b72hh8vpmjfPOSwMkyUvNWvRagtyg0W5jbaJyLd3aLVFX4SWnpdFg9QfaCEk5PxGFSEc5q1RpEKBmbOnz42IG5w0vpg7hFlHMNeBaIbsindLuUOWLzC/mxM3FU6na1MvC3fkqX4yaHumFEcahKsUhCT8eQIoO2Phnm8NiUdlpGbpaOzPEQZJgygROzZT7BnxZ0QdqtHmoFUTpjTGLvOrsr2UvJZd9Dxw5FlyM2lDy+8si503PWvNPDeqvcy3cRLSNYkKDGzVG3WvZGMDGgq5DJlz4bsACvu9HSQB0gDlSlUBUYJWQSrBD4r5QMcv4RY0F/gAbAiAFyCmjpOriLb1BYaLGcteU3pqXSO7i4N7cCCCZJOnoDpsAxFIaQcyQ+NAU26mMzfZiH2XgMjmVWi4IaYNtsP5HiPTc+0+ibPvSmr+9MrTKHZtGCPgNaty9aPbK+qZ/ao7dvoBm+co3rAmD95d1S5ZjrXASK9KMOQ3qtsZrv5icWzMC6uCR0yBxsOdysRH3JDCbFGoIOczzA2xlHaMWHkUEQ5fOwJcu87NwYBiBlRkyUU4ImQ0yZ6RE+lCrKaVFAeRQyRvYSGn4wAomf8XQQP22szLyTG+UsgokLS0mAxBrylXd4PeiCm9TYMeBnzYUGRKtqeoR/JvIldK6DOoLCRLGIrrK1JF8YKQPeW64oFj+iiEY1YphpGb0g/cTdejytdoM6Afgb1qgk3qTIWJ3eHjh8aHN7l2x5LLG8oA8ehE4TQ0WwSqVSxzUMOkWMo7y4HRQYD5FTLRRSShCmNhjCW40MHyi0IbceJzmPDQtOt4FZBHsqcINijB8G16QTBh9oaOagQYAQAGa30gIHXWUODVYA2xsZmvEZeqyfb8MVT3NuNWKASiIsbLOFryzEwnV2/Zvu2UMpXKFLfO71x6Aa6xFm+MXnbkxPhTG579t3oNHurK2vkuH50be+pbJTnD28ePGNSwcOL33/D/3oRz/6Gyiit/Kbn/7MX/7uR3/nR3/kx37nd389kQyXKzlw0T33ngEwH//C58kDzHQR5TU1M7m8b/bYiUWHZ/DR3/yTpUORY6ePPPPcU8dOHcamDjRS42NrN0v94LDfV2uWosEEBZpcrnSukKXMA0nQJiYTroYzX62F7dwSLVUrN25eg8H75kffVvn+77py8SVKvqMlpCHMc6wyC5TNbxPEQP07op381B/3RidId+WJLI19/bbj5urV6zdeOHv/sfd92xn0wFTH6JXHaNUrrcbF1y6EBi1/PzRQzSoSFrTCrvD8dKJY3tT2v42h7yyBtRRmtzKve+uifToiRRqaPFxgFeEH7AtfoWQk95CDHL+2UrYct/V9ySDkFqSD8wzh7sJMHSLyZdKX7OxodvvrmFftw86//Kn3snE++Rfn0/F9pWzFA4W2KZ04YGV0JUIXQk/KP6EqP2YPEAUFEwts2EgBRju639EBMcohGdCz990KcVAOAB6QVyvsEq04bJSv6ECJadRoUtj83CBFM0iFf4Lk23NhfrIpBlA4ksOgS6iCXscVbeY7M2PmiGkS9WUPAKncJBlRu13Tx2LxaUGzmlIuJ7SxhtM0OBziJtQnEVCEDo6BJvRFdMGQK8VnS+agw7RNBzRH7Ck2qNuFLZbGYXZ4O6ofPo0Dl6lFKG6GTcrWEdKXtDr2tiokdSI8w7M8efTQ/hNzMwt+d3DUHWW3814HKrGQw4eCSg7LkmNGJkkoi8xishxY+Tw2fMqp5NIvFeulQq1UpIoCaTQoVkHIHOZcTGegZMxX4A16rV5JpcuaGJJlJF3mhX9gOMbCe4bgZOI5idAZ4SFGiuaRLVB/x7ffZYu3bREqMxRt9nxrWBo5Ww6PNJo+amGGfRC5TqdOCiyPk0oQUdu4onVR1yUiMYdETMGSdKo1OAciYKjRi+c9FJRplZ+hKWxukgWhPIaysgL0SnwSk8Z0qyaPzgyjQGJMXAxFgFmbPXAxPxrNNu9kwXglPdA9ekuvR0BJve5DSTAmHTHhKuGIn+h1ERz1UbBNN0WA+MLacsgwChCb82/8EKCz5IC4ASozf99wh7n+jU+ahsx1dh4GJOlWmX7Bjz4FqFw2c6bFMSDKkui6hqhBQBnNIjE8nbCI6DnRP6PmBaRROxupF3pCNV/qq6FdauN7JU0pEjC2FFmFcbYakHYD8diNcAQwonzCtANwaHrZW0Yg1h7FPZLMa6yw9KPM0Z3pl8SApgJ0wqu0SgAZO0k7l0wxyDcthxsjPf5T4269SqvRZKRTbwrhsKU0bxqPnhvZ22SYrNXApKS7Ys8iqhJ3WBlUSJCmXWoESiMB6J08CuvJNmPLMi24CsDaykvI52NtFxentnczxWLr8IEFTBdbmxupZJJsqplyaWtzNZWMfvHLjyvhpcedmkiggtbwHIoPpgO8KODx1qpQcUKle37KzHjFE1C1Br42u5P1jAUqwZB8U6liCMMBKwBWofwDpJaLHBo/C0PHJUyJq9zj0fnKxtImUEwBGxCaKvbJyBRm3UGSXKAJyRnC18KGcqGxkBjf6CGv45P7eRfFLzSNe4vCqQ7gmIM5Ad9xD/PJ8zwiV7yBfXEhns2VibdeX18/cuQEmZXIwPOffuc/zuybrZTzyVTs5soK/Mfv/Mff+tAP/cTph/alU/GLF187cuQAKPC5Z57D+ZySydnM1pl77/71X/vt1EIwFguheqUmHsBxY2Pt7AN3Hzo+E09xPVJtFLd31mdnZ8B1+UJ2dz2LF0s73C1Ui2hf62Tcy45zlS1fxIWCBhodCDsDA3tyagI0Np9OjPvNUmUraHOmp6Jb2yH49EGDmljOVrcZaZCqwE58mpcEu15v2JXg7Rubq5T0KRdX44n5QXeU2dpJJUYvPPWEx9ePxgJn7rq70cQPpurp+vadPDAqF/NX65s7GQ8OnpS49pTY5rBYZlta21DTKDD9hxvbXOQnsDoIDcWcHGVEBSCJUn5g2gqiGrH1qxW8LhqTRMFNxB0+f7/bIDCJAnhwXBAHlURGGSicVrE7d6rVUsTf/+Gf+X5UdB/9D187c+pQJZehqh6US30wYAMaYPERTuG2oTYsLlwcYIPEBLOiKvSWOwIJKSEnQDZ2BZcjhKOlyV/EhtOrZfGQ3b6DsdIEU9GW9iKkGOBxUX5YcrxshGAAbXqusmeVR0D/gae4X6IlPwGw7HlBo+go8Ak65hlcdrjwfz56F5c4LBjlJuvApxFbKQdNyCLoxloMG09aU8XzcWjAEgJ5rdQ4bDM/YWImZZ14Xzh+8o+AXqBFxkUT5CavZsluqOtQwasdhm/awv2I6dM+gYwIx+NQPEIZ4XKMoB9+58jTqvXnp5YO7Ts6MzUfcAdEVGgcRkdzTffZpYxOal0qbrBvcaiSeds+pPAw9YDBgZVSAeBTQQbjpQYYmAQZdIf3SZRAJYgBC68nzA80zUShkyWvhtKLuL2IGTi4U7kULgDLas81pE5RY1xs24r2UCs9H1w+NhNfhI/btTkbyOqUkBm5OmNnmxq9aH4knJj5IgeIwqWYArTjwptgUIMPuMKiMUl4WcAzt8ic0MfqiNwGzTZMnW6U1RHtuOFRQF0a+Z7Et7clDP4yMGhoEiPjisE4rBn8JI/rhNkDT8Fuae018XROM4nIay2q0CPYD2wKTmMNSOtpKjSgoQBweKt5RJyiYEv/6bk7B7iVc4MsdcJbb//E8jIVPKbh7J0ARcAyF8RFCJrNYXJ1kixTzJHGwSf9tX7i9Wbvm/1DO2aY/IReXu/iXrC5UkvyE5wZmx/qS0VW4otUigKiS9oJDN71OlVaMVsiDbO+yLhYd3C5cGLxxYW9QcEivQk6QaQPzBzwwV2jYChCrwJByCG5G3dnJmcU1NfuYsJgU2HOgEeWNRSYx8Du95YqtXbXFk9Eu8Nxud7AFTkUjew/cgi3SPIjkEi2XKtcv3ktSrHyVps0qtSwYgvBX4bwnPQSptJiWpga1DdYL1q9HnWwSevB9mTAKqc9HuMObWxpjAU4HcB7Eh0QisRwB2Vo16/fnJ2bo8LLoNWiPkuhUJDW2OMJBIMbW9tM/Pf/0A8Qy/uVr3ylWC6TA6hCcLD0YyJ10UCIt+QytYW5JGwnchXJ69uqbtMn+weJdgPRMHBKUGYhXyIzJ8uEHA92QNonAzUaKSEHKraL0QdX4ulGHxUMAnhYfRYJFgwgGgkFQFyZP5QHXOReIFTMv8kZQOPmMNtTnKBkCPY7aNUCGvMW7tV24EE4NHQPtM89XEFRJ+Q4Jrs1mbyADFFu+ml09aB9B+X28MYDUbFNFHZBOQW4K6qx+T0UfNze3bB5bOnp6KFjB9/7ze/+w//6h7F0MpfL/dzP/NzH/+Ljr7/yaiKWhF1mX4Ezif5oU1oFvVzQS+gGXB/ufdP7ZnYL24cPL0ai0Jb+vv1zuE0BYd1WPxRIPvW1cx/8zu89eeJMsVglJGx+cWJgr21sXssVd9uD9tzi1FZmu9qqGauci7iPaUoxLO0D6cEikExqMjpz74EHv/z5Lzu9AG9z6eBcJpeNxdPkBI+E0iSzvHrlVrFQW5rbv7m5ixN10DOOeGo2ygH266mp0Ny+iVNnjteqzYAnPBM/uHlp80jygL/V/9on/mJUWBvXOo/eHZpOILzinbR3MKVipsxx+9qb/iI+CZ+w7mAu5poVhypJ3QHfhD5JVIg4FVbESbrssHMOrSHm7yIRMxQw9vn9sKZ4Q0LB7f5As9Ly+aZt4+lucyYQPPHkF67+3E9+5u7D0xsruz7STgHz5s2ojgm277YGtTqpIcj+Fht0SS5GUTz8dLok/AxHcCEkP3GNmhPAKaodeD34DvAaygO0F/DRAlRQAcGIXg8GwzYqcTYJHiFosxkBjpWoxUS9BJYwRsASB++3FC5CTuYAPg3ZFTRycA1445ObeZBzZcKyfuBTYMubzQFYc1hgzd1QX+sF0sozfRbN13ZhvxDLojB2nPd4nHsNZif+kZ1EqAXoTyk86QmcCOIHfyAfUaUd0YEtk/oOljO3XmSwnW2IAEb6r4B96LP3veO++z2PvT0WSCTCSbJw4BrEUyy8/E8xm4EkmAjJKDh/4gLSpZlYJOQOBQnm6Q9apXyeYDgESjRhmjk4VSFvRmFJUuBoOq12TGwIVgJaMtKWnYpRIbJOlit5drI35CMxCjaG5pjgjHallx8EWwtHYkfuPeKYdQ5am+uFV4g0sDvRrgtd49hM+CjnUFpsqyyc0IfmH5GWdWQZ4MiIR2IKGUgXlg/SqzJeOAKZGBdRRXg7I/6atbQIlbovgJBIAUUWrUSbqgUyJFlqSUPz+FUrJKIFVRPw0y3sxJo7vonr0O4B3+kuPQytNkBAwywT/eWTHoOj0MuILWL9TOirJEw9Je5LcCXqKHT4jQdtccG0Zf0gGLUOfrkDptYVWuRd6onIp64xbt4mYBPwC4j1cnOfmFyzhnwyEH5h1AYZcCvf4RpYVeN1JV4NvM/KE6+H6heqCGMOARbFxeoNaLMJCOrtUVKW4L+hA3MvzDegqTzKAIl8iKEo8hFAAvYGgqo94HCdOH1qcWn/8+devHjhUjgQQuGC/4bl0cEegH9iQrFQRYKe+flZ6G4uX7WThjeaROx69C3vfPv73vlf/ui/HDx66PQDZ5f2LT717NN/84lPsNdJ2YLemorP2InBQkr1jX+KXVltyZcCXYPGk6kZfAHRx/EYPh1GQ4WDBwpeZLlg+BPp1NLy/ldfu2i312am5w8cPXbhwiWq3MdRb1ZqgFu/2aqVm2QBScWjZA/4m898cnHfMmQ9nYz5Q8ggQgto6zwe/41rO+mYLzWhwtVQVvyiIZ+JaAzChuqUHYtfJGXMm4RwOez+cATeEZhEqYWrYKXejIVtGJJJygHIsU4gBzADd6J7gu+WoQ1u2khF4oMBC0GSNAv8ZSHZLsAVqMrCS1wWLKIUhQ+Vo6gAlEPikrnJAjk6b4EXUwTJBYA4DGQbcVk5pS28R0cVpsL0ou4CWZG2RMotXCvEUvIY2QvYgaTGrM+QympmMlvO5vPVeA6Xs8bpU8dW1re+7Vu+FR6IFPHkPKDgT7PWjMcJIqphgjcgzEDE8oKaGTD3UxpmkjqCCxNrK5dffvGVSDCADykhjc9cO+/zhq9cu1lvDY+fOt0YdF547ZVqY4PkwVgpJqbS7Lf5hdn+uipYYI+YmApTBfzy5ZfFZ4WIAgmReGJ3p/zOd37gj//kP8Yng6+8fBkAwlBAwHG9eisSmvym935nKjYdcobXplfrpdaD9x7Lb75y8+qroM5x39trOV54/lW2wuzsXL5Mqt1A0zXe3N2aWFyYPZQOjzPebkZhjWYyNaEcWqv/xwO5kzWEALCzxaOzKFpe6DDkR+vLP7INwT4SONIqN31EYDrcOPpQwA5sRppd8Dq8F6rhURselFoLTVLdOPpumzdz5sHJD//qo7/280+/861HX375CpveQ8g6HkA+D36FbFmPm1h54IcdjLIErRaggvyEmpMMzxieukYTTqsyN5oAXZwtBGoECEIZocMO+HWQkOiVABFQhDFkLKLxQq3SvAglyRlTejoOi0qKXAve+F8j16cBSD6EgzkEtqBdUj4Y4LaeZIJoy6K77DG+At9sGBq12uU2ytQL35uNJIxndgdtyP5CFiWDz9GVaCdJC497LbEQcnynbwhOQKIYB6eDXKlgE7YxvSBfPIFRrIrTTgGRYSScIJlqKd+AoY6FJu6764G7T5+l8h4pbcGL8kUCU5ocY1qdYBiii7JWsTjMlNNJuBvJOQk4cuEWTtstPBazWKkQS2B2CPcBFESpVIevKx95UICDRBPav/SZ2SLzFZEXUl06bLu1hgd3rwm/04uJsFZsb5fbueaw7E+5jj2wPH32uC1Klu7VWiYTTjkX75nvFDKwBVAs5h3qoWXSLGlBmAODVQR7ACaYlDfLYKvMkQyKwjhgNP4H08vMQBsQYPrK06I/rL8+zaqaNdUKc2ItsKihPMFoVXDO46KeAnoOFlHES3IGbKC2jFQI0pmapRSFFgmjJeiGwIPlpA31lHt1xuSwcXTJ+kG/CKz0sw5zn3UqnKkOcIjvvXPsjcE8RBd1WC3oVrpurqgf6ozgVcTYoFF+0q9mvqxrAAtj0WVukZjDqDUiq0NqAXpMm+x0MbQKsYcAS+esTOw4SeJCjmwlkZfQQNSD3CBTF3nYRDbN4vEcSmdYFi84gMAOMMLQHvUHm128oEhW6nz99dcvX7pGXjpiLskBSXeYJXrCjlFvzErFo+HNnTqmVmxvjzz00PFTZygo/OQzz167cok80cV8LjkRv37r8j1n7zl17Gj0e79n9eKl1ctXc5k8i2XHG4GDfUWOcXlUDb1+WASVAJMi2qdJgFeDj2NLMR0MH69BfwCxNjgzO8ujsMXUl4W+3nPmbDgcq5bL1964GAuTvSDABk9OREHZzUYdgsNeIQzJSvRBSfmwCTXG+Y8WwhF8EjDFOVoNPI9IuQvGkmaBzBs1SulRhoSE1eQeGo2W5he2NnaVz89lT8QimMaNIgpjjfCMxqI5BBTN0glE4Y30HyutSxbQ6IyByAmSMZkpNfBp7tFXIMEIEEApyIpWocdc4ScDn0KCHJybFkxzb/rgOg+ibuYe67Vi+yxsSgvyrMMoJl5ZSEugB/2QO1un2fJTKd7vRZtBLu61W6tUfqyUGycOH93d2sGDKezyo3V347AzttfxfUI3yrh5tSBPHAk1xmIhtyuobiO54k65tbFZ6jSGgyLsIL5R5H/44he/mCu0vvt7v/2d73znE1+9iRoil2vJFc4W2txYT08mZ6f3k66g1czhuNBqyPyJdqFM8aAe+RcnsLP4fbED+05euPry9EKKwJBCqTSRSB7eN5WITW+s3cq5sy3Cd0a97/ve98f99v/2xseq/bzbFdncKY08QWQ7bySwtV0+c3Jh7ES7c65b3p4KdsLBYdpVtVGPi1xBhit/04zqlFm1rtCfv/eTJITbh/Urs22ERi24loyvpJYjgZKd4CtHbDZpi8Yd0pMwOpgstgLh98qiRGE4h73BXA66W5GJ5Q/8s0fdQ/cvf/grJ09AOFybG23cqOemE9VqFgRmdgSSWoUtAs+N+MD+h+RJQIOOSkEMvRCCYsPSa/4JQZJYlpz+bg9em/QWyOi7QYoGA4MWTCwIjUGuwKTSYdKwqLLgTSjqTcfe6DRCTcGdmQEsOQyQ4uR/G0w54RLbg4OfUTQBppwAK9atakZUnh5LnKJFntDbzUXJkPLfRJrTUGAm3TI6edpNkkvwlYf2uFwIMIc/kpBOGwcjshtQuZl8Z2TzdFH/2FPeqfe79YX5A+995H6qfDDn1Uwt6I3QMfGU7DQ0xESPs0uIdBDhUc9kMZIbCA4IeGHZioVMg/Jx3SaOhQTzBlBigFqZerlF0wBLINYbZxijkR6hVWMukTVIdonwLb068jKVw+PBoadVHpdbjWxnlPOEOlNLrvBEcurMspI22y4jTHninaQHLTMB93lMFuqNJGBQMJ+aeFFhkVxAk42tfS0aLZkM6yMZrCT4cmCE4Eb6JwcRs+TSy0INYCwEqZwg7Fjzb2ADishLdIH1hHAys0q+IgZJ8KBDhUKERPD5Fk0ShMjVyrCGYj10GL2umRUgCU3O3hvg0lhQFliAKGRJPwxRMV1j/c3YNCgOfVpnZphfR6bmMXPLN35YhNZqhHfoFYIrnjavE/VlNmDN6TEvVtusj7p7h0Jz0ZyznyT+mglmElAHSIMhsisazIDAfzg540JkzDzEGuF7hcelslkhwvWGThN0pIg+uT3rk04YRzf5/GshYWmwqZI4jdhydKpU8UumJiBYr79xiZDX5aU5g83VY6CKRQIY5Q837JHRgpInE5Mz7AhSLm/t5NgMuUKRmjcw0u9777ufef6ZI8cPHzmw/8r1K3gCvuft73iaPrRf7ZEkvgOb0AVLGJLhpKfMv7ak2z1SnnkREjYcK2OUsNrL3IviDhJ85u57/9uf/nco6NT0HImrLl6+NDk5/b3f+z2/8xv/Ibe7wwTBfwxtlXK5F4mIIeYKXAUNYvtcXFxE2EVNTYGXZrt34u7Tm7fWSMToQyelxRh7gn7VLkRxhz681aq1moQl1BsNct7iLCbTL3vbNialE3wt4rtU0PJ20SrSSYOOwHpYXOF9zbqZPXwHQLgDtGNu1HqjIDY3A7fIUy4YKFYWCLbmnIsiqIITHXfupAUOCDlEnvXTFjH3qFnTPneC2q1GeAvbUr52LJ4MNHQL2AIfqgNgZjKL5TLZQIcSpQES3OHVc/3atdP+u06fPJnLZEi+kYwnrly4zF4hiiiXy8q1hoz30Awy2bI3IbAoYx1OQs7Y56s31pfm56j3htMfg8l1ypFwcj2TXb3aTM46k+nQK+dfDseCZKUOh4PFSiUamyJ2cnHh+Mr6TWI4CsUSwA5Y0McExDkUZE2B8URicncn6+hF5xYO1lvNBx+5JzkZJU8Z2UCp4valL37xzOmzM8tp2wiSP1hffcN/bOaRd9wTDD6Uiiy/9NLlN9641mg2oun4/PzEGGHSM9oq3IyM651eue3olvpd/4Aqgnf2uTXfe59MqebJrC+f1jlIFjMf53u/akotbKj9JRRn7iTsDWcaeDpS0dULVCloR4Yhm3g9NrxKoID1/WRvtMMo44lawubQG+wOaji0tr75Bx9rdyv/8bdfJbwrEmXWI7u5asAf7XTrglReAhKgV6wup7AzEHRMaaJjskkDFdo0aEoN3yy+EnWZgzQVUnvxILHzAhEkPMmUNjxmBRKgbm0EdrswPG+BjoM2UdOAiqyxv3l2uPLmi7wUsKRVuDqSdskWa00QN9EowM6EoYnBlMoi8Cv3cS8NA5XwEOqyRDsoBQwijwCy0igZRkAEUc3BRConJ8yDwmy0Q273CATCP56jKQzKJGANBbED2HvtLkUV8SI7tHDi6OETE+lpB1Xpiwi77pg/jiemaBDzCAITX8oJcs2wYdL3BAj4R+jF4RzppFauUnqmVR/22mBAKJ6KmPObNjyjMDYJtho0Uv1GeEA5DnaTZkwUhgRiKE2YZxCevdPyNKvDbGeQ94Ybk/OehQNJ31LAlqIUzg2bd2DzQUh7Axs2gi5iNCoszPA8TMvQWIkloovAoZFJmSzWWUChvY/EhT4EQQwsACbiCj8KU4iT4C8TyRjpopk4njWQz0qLMtEiFIZTvclMBssPnoZRpA2hFZgJKNHepBskp+c1RLpgiJxWCf3abdjQbEgAYGnVFusunCQ/AR1qjSZ5G43QO72a/xHOrFeYXmpZOKweq1tMhdmV5lTdpr8WKPJHP5mNqHas2zS0PZRsrvDVcBLmXbRONyx2Ye9NAjltZW5mU+gcFwApnOVshUcP90uiZV9J/4zGBw9no0dG9sVGwsp1iEGCbVR1I5FeSHKrOyLLFeILimgWAxKB4hmmG4Qg5c3YEQlFmQTUMHOImTPzm5vbFMwhOoWxMEsspBmmLEMoa+AIAaSdre30xMyRQwSK2DZ2dtEt19vNz/7Npz78kQ9HY+958qmvnH/lXK1Zf8/b3pGHYpcr8AUeJ5QDj3QME2RocQQpSYOiHN8+P6kAdLBzGT6EgS3PYtE33g4gtciK2R1+4mMfI0o4GHAR15uamISWk6+K5E37Dx/a2tkkQiWdiJFvEh4+nIgh+CIlkMQuHArhyYF4R2oOxvjAAw9cvH4jmkwgHNeaDZiRSqOeTiaBC+zLUA5aaPVxxfLTH+g3Y59MzxAb00c6E9IYYOeECsJc0lugiBUEVxnGDj2vJFdyVJvR7H0I1ABCI9TqXCKAlM8oCRkdN+FaYq4LGYgOUxURfOxyyc1c2EkbiWbNhBA0LeDFY9Hska+/gl8NRLFgumhxfFzhULeYULN9aBx7rvYHDineID4twwoecLZIPO4LRig2tb25820feD8JO0nmTADQbqY2PREAwtGNC5FAy/GwQ2sHUFAIWfBrR1mNiIlosHp9rVItlHI4SPsoxVatd1PJ6YlHXfAxB4/uj8bDb1x6Dde2YrkRjqRvrmzh3PKhD/3gjdXNV1++ePeZE2DoZCqSz27hKbW7catSLSdiYWcvOBNYmpqa3djpfOu3fDCZwu/Utxw++uLNF5/+2tNT6am7Th6G2HTqg6klKl9lv/rU39h9lUQyORM58oGzH9h34MalK9eurV5pXts5dmjfZITk04u5iy9E3GhCelMTtn7ZZv+6/VfTdedgnq1zZuvOxTsnZncKG/A/m5StbyEJFhCNP8srWiIKabO3bJWtxrjpjvI+KlrYmigGCV1pdQZeYpGFxvv+AA7MNYwaGGfRWn/wx9+DNPbrv/bS4cVkq06+mmS5WKLmkgvuxtBFj9EM4XEFI8fSYDCG/gEmmGzorcCRRSVTOhCLZYKgYeFNPKjltYCiBL+tHiy6UpiL7PmMKzIYgs5A6kBYgmDBNaCGszcAIwC2DqFcMwuMGjDWI/qZzYorDS9RRR0dTJ/g2By8la+QXrRPHNZXs7eBTevgBjC0gFczCzq2j6C29EY9MTiIM7aHhA6MZexXQBHABKOB0iClY2et0vK4A7FATMrAOl6aQ5KNJxKxe44/FPElYOFtfRypGA7VFfwhbxBJGsLE2OU6BRbW1mDtRjGKaHCoIAJa3Fa3zcKAv5p8JYBG1Uwt+iaOnGVmcyPnIKrTECsuvTnTT+pRJhjMbyfpr6PfGddwt8JY0LTX86NceMZ1YF98djntJhtrsG5z7NiGVZuftJ+m+oZUSzjrefDcYPPTPtMsnYT+6aUG4jBmYLiXBUIuzUoqoiANvrI5mUSWEV04n1AT0SURb7Wyt3o0RI+1kppCTlgOrYTmk6/ca0guSFjD4sW6YoiXSkQIVCw7DNAAmbp9MAGaOa2LUDpGKg6uMA8aAuHc/EaLNOagjAUpLSV+cztgRJOGzNB3zvl/749pW+dWz/XDmzYkJIoWrAuGB+B3iRoahIgxj1kngmDTlF4mqqs3qqnbB+fAsuE2+bSm0BJ5mUhEXoBPEyBKDJRgTpHyqQ8riewr6y8WX+gXREHQgCO7AvaV4QajbxunaJkNIcqwjyYfN75pxH2yhQh6IGKt0y/mSqu3NkjiODM7TxJ8fA+Jv2T2AC62MUCGrVQMC4MY4nofhLtirXEtzuQL29lcDI/iUvHQ8cN//clPdEdd4kwi8TAuCkhel167sL66Vq+0AyRMUP8HJGoFQ3NA0UHq0UioXK1j4MJ/QiBB6QzDJkmMlUXKWlTngYOHS9Xa5tYOiSNIWUU/5+YiExMTr7z8YmIijUhKvs3J2ZkRqZ+2S9h8EwmlWsAUAumlzgI5iicn48ePH/fF4rfW1rmTBPfZnd1GsRMkUyN5Lm3hfolpJpGk2HnYESQ/DjyH6RW+Y8i+EBvwGSBFy6j7gUxODYzhswFLzs7QTxz02uo44MfjrDLTyHVuAy8JSg1kMnxMBdad3GMe1bMCWvY5s2w+2R1c5JwTqx2+cli7RijECbbdY07NL6YlYQRuEtI2EK4NzEbAwYQH0KVHIj6y+DUarXG2j82eotmo9C9dupSeZF4nyAUWitiCWp2qO6CSkYJdMXYSq2hQyAs/9ICXmoZo2Ha3Mo1GlXfCXyElkNz0x3/8Q6gTvvK1rzDa9GTK4SMVeWd2bunSleus/sFDh6/fWI2EU8eP3QU7A3Cs3WSyb0g0HLYg4m995OGj+x+ctj3ot00vz88VmrvAPKGZUVuwU+98/3d/T7lcuvTGq/NzU71e5cLr16dnEmQ56zmcKLRfvvF8IrKZz9ZOnjqy/8g+1ndtbateDz76wHdkw3OH/M3t1z5XhzHctc3ErFn/Rz6ZPOsqM885n3/vJnOFWZEQonWGcMBBQfBYCgMIbGSfx96qjRv9OipPKhxp1uxkoAp0ezXqB8POcgyGFayH+GUIyzg9tULhAz/89vmFxV/++U+kYzNkE0ZVBZ6VBk+ihEWAeAO7E4FVmkLxbLhWY5ZC3WvkQ2g/3XMSLgd+6CEZNw0UgTcxfCp0mABeukdrxlhJ+WEwhzh+IS4GCsigNoFosWcBSOQn3sv43vQJ1AEC0C2u85gwq9hmdj0H9+HzRz0Qo8wRZ4rp25B3gbEmjDu0v7UZODGbCr4O4BI9QWHh91kXcSITYpSkAvlEmhRcixWEOmgSECJQHLsnY0m2X7feb1MxrWlPJGaOHjq6f/6wdxQdD6gBLhUwidmwGqOarVXxZaHwnwR+dUJ27DESApWtcXJGyUxlxlaTwPkWnoYwnSSMhtBYgjbqfbyJQccKaVW2Kc60bmbNPawIfA2LgLnY5uqNyAs0KnfHRZu3F0hRMHmUnHVH5rwTM3ZbqGYbZLuDAqm0nOT3lZYa10w3pnJSKY0p6+LFr57QtqZypWqGRX2RSXg380HCBXHm+I6C3OVhj9+B4I+dyYxoWDDfhgBLu8EKWdAsRh+wNuBpABqxQWKlREcWX+DIkusPKEhJmcyzalhUyUw9PxjapWX/ekt0SWZ72boAFrUgjb5yQssBnxcIeYjiKriON8A0CaAEAdB9g+B0E52lu2IF1Ye9Dag/hppqfNZVpkOnNCoOQBBHd61rpnvmJ0NT/8FPulM917sE6wA9w2DDcPAWrjBqwSELKWpq3KxQQ0A0RYyVswaKCyVGgpHaWYltCPZVCg6F/NqceF2RZwBpGNcKG0obJmOMjwIXIMIMQa+DBYB4K/dJr8OEcESjJGQobGxmESvxhIICBXwqEAIOBfmSh5ZzpBYtPYVAKMvT7e1s7eI7EyShlW10YGlxfeXmyo3hPQ/c7XHYayV8j8M3rlz1wj2Q6JLhwO4KChSqiFGXTYhjBMrwiamZdm8L2MsWsHcQGd/DZwS/eMCEu6HWkRhpW2N493hDiKlhzL23Vtf2HziEMvNP/uSPKfeL9htBlgIkkIoQrtJhnCjZWuLhQGpQU3qIabmYK5OUY9+JE0889dTb3vrWnZ0dckEj3a7f2gqHvOVqRXVUlVteZLtSr4UiYXqLCrperUH2AoSdEmvZanLOEmqxtHZGoQEgmUMAap0BGXcOc84j3GwdggCDNQl5kpXGUFme4yLn7Co6z+ftliwaL30AHWMRrDv55Gb6YKBMvtNcoTe6xHrQgqAHxk095Cc9xtZiirFGuxzdHrXkvM0m3j3OYpEY7h2CiOjB333h8WOnTsI5rW2tAT7lpmoo+PyyIin4A/rCRqWrbG+DN7GPNSt1fF4aDcoGg3L9tWrb7hxMTs5sbe4WquW3v+vd2RwFLlbRtWdzmXK1+c53vx0VBbzOo0ceuRC/8LnPfoZQqAcfePDyGy9TojyVCLbqBVIqk4WtWioeSLjWqlfmo7OF3E40Gmk4q9tY+HqdSrGT3d2cX5i+cvn1cNRz6MD8Sy8/1+pV3WHnwcMHDiwsvvbqBWAmXzi5fPDk2BH+tvt+aKtWGI4n+v36q9ee9dUnY5HB4SP2WqbGxPzDw5o0TeY3HtYVS3qwdrF1BwiQKWZxtCJmF4s5Q8ED80+a9t64kisFyKcRJUIIKZHQXx957zEDg42k00dG5pWUnO13vNHpfvfyPe869VuRf/WLP/2fKXvtcIVHLRYH3lQSkd6rJeUxchpRkkAIw0tiSkyVNqV5oCWYblwWEDvZ4NxCRgBJR9TI8gShIz4oOiFe8nwWSIBYYPBZVhlcJfNqFHqLoXvWBAA/wBEADhLljOUjmlPMqMygCGuK60V0kH0XyGPIMNCuoYtz9h7Bp9QbYnuMkdH5BMIBJ1qSkC1aKv6F97BveTv0jR2C7xj7ihtAsopfsRAkMTwUdKLP5HwS6cV+5CK1Chk1O+NmjbANx/6ZheXFAyicgfZui2wVAyIzSKHOkNjhLAUNKjuGbDmM07wOgZuSkgEfJSCquQxOWBS2hPricSUzAjjPPgy4yVWJ2Ky+4cIKqNMgYjFxbJo9IVYRaQgVUhAmmrat7iDy011pjHaHrmJswjV/aDJ8IGlL9m2u2nBcsDnaY1/L7SROSTKiUVQxsxAojM4yEzKTREaS98WEjWlTQ/rpMd3hs9loYF9WQQrLr0pIXctm5oy5h+sVD2Woi8CFE2abg7nGqigKaQ5pTrXOPMuhS7TOJ5yTvhscp7+sm1aJmFeYFTFioAC1SUOcS5kipQkHZFzig+ABWUZcFzjt9hvQUIo74ECeBDQkh9P6bQJM99TuP3boNj1r9dOcGwyoRzAMWYcRgHi/GgAv6gbaV3fopSXfi/3TALmHeeRX7hILwIn1CT9lFM5S9nMvo+Kr5F09Z9FjVKH4P6OzJaaFgtNKegV1ZdX7WH8xuLL86CSYEWIOCOZxeQk9a8NpGe8tgTQOrKbYHzG9qHVrDUp7khODaCPqTpJk0YvNkyK7RO8wUfk8WuQcqF9qRzQm/gCeMkRUMCiyEUWi4XgsVigWgWR8koPR0OXX30ikE8jp9LWYycX9EYAAsSZAqTW034YeELDBWLQ62PwSiUKxHLAHy7UqVVC7eKuykbGBdtHBjogF8nupWp188MEH19Y3tzO7o/yYoOGvPfnU0aOHZd89sHTu3LlHHnt0enr6D/+vP7377kNYMf/TH/zZ3LRstJAl6KWEVvkwyy3Lu71dyJXYkNdvrr7vPe+kHt/Kyv8X1TthNjeuX2dQiKjxVNJRrfIsGun3f9t3vvDCC+trt6gBwcbnBlYc6RkUw1jAI2xpa1zAsbXQ3CCgEDrRAZbhOmwHHQFt8RQgCvDRPgcV1VHLQ3MBAGRj7pfzhAq+7rUAGHNwnRfRDhKneYtov7miE7VvwqIg22ofjGj4D0PZtV8YPjvMEhyEtqgAEwijsWy2h7PpRL1V2txsscQp9ySpf26ur9ZxdWs0KERHZEs4pbAl+q1CZvKiFuOI2g4VJiAL0JLCHWBu1iHOQ6AN7EtOUGwiz5576V3ve9fBI4dffPU5tDaoOo+eOFxvdIrlrTP33Hvt2rWN3k1yXb3r3e/4wuOfe/Xll4uFXZJ1D9tNyrlNTyXbje7j5z7/xOi16dR+LcT731+t8reCPrZaq+RyWIJdKzcrKK6pX/7cuWfy+Yw/DEHzXXvjprfvalYyM2mS31ftjgYhc1eqK6nQQbtzenb54adfePEDD3/L7hsfu7W+kcAPwiAcTfebDibcmmdOuGyd60S45h9BEUCCBDJkPPMcy2fIC7EjozD1PEajWrlO8buoLR7yBvC19fgipSq8ry0aU4kjUJRBLThF96GR5cp1/3B4+P5Hfv5X//mv/uJfYXwk2QiLDE4FbQxdxNbzHikCMSLDSUO/nc4QUA2osPxQPfhOei4dsiABqiFsCTzwPEypjWLDkCLCEyF+1OkSYaV8rIJGrZIPQh5AHWliG4QnaIKYBuywZlxSMmIXQ3SGEinpCjKYQiQAhwH2aDtYB+IPdEB3kYMxycYDKdyChBINqte+lPTKd5ETugVrT4/ltqyAOfYX9FFmdLYCk07oIf8g2eJNySXl9PndYfAawUUMDTROLC3JPufSixQxiwSipMIY4PCOVGkPYmmj+a7QIdV98eNintm18qcCJePYRew/EU9Q/Uaz2spvkUVSNygrI9RClX/Y9OweEghgvmTARvktRa/wtwPZGB8cEpqDgRHb8fmzDyh45Gq2hvg2ZzFyxZbsi4cnUos+WxiJaH3gaQxcEHUCwYTqGR1TTUccdh8yCQkVwPi8B5ogxgprHPDJ3GtVFQxNxmYKsiqQV9TDuCQYmsXtQh6cs1YSfEVYOBdnA6iYFNZcEbEx2IczUBmfAABtW9DH8/xqkIiQF79asK6G+QoJhphJYUDDWjb2gb4AOOL1wegsKWgfWGAVmX0EYBtsIRp65Dl1C+IqVKyeCiEaox3gicu+xig7uaKkee83HGa/me6YF6pfjMHan3yqq7rG89ZYbiNia7AWWdUY9Zhu1wmzq0Z0hW6ZE7FRLAnnqFVEjiHemJSAWBJuaIAINWirRG7lfkXeDPTMSv+ILMt1HsfwjzcHmucxtbtRSQJ/HruKF/GFcJoxRbipiESL0lJoLLRYqdboG3mYcdJskTTDDZVx7l/eh4WVaF32Dr1FVKUFRsnGIqgsRAJSN5x1LxiEzkoRDSmgLeCm02hGg6EmSVfGQ8yloOlqqUw8ETYN4LbZajP5bg8Jatts0p2trWgMD88q9ubrN2/gCHbq6PHrr72OJZFMVFBE9iTZrAj5hfSa/Biw0YiMw1aDrM5J3CCoo02vvu8HfwANOerLg0fm6S1i7v4Dk+wQS1Sl0kPP+CVMpINsos7KLfbQlStXfuiHvg8vpMcffxxEQOwv3BsgRfeQiekPaTrIkZlKJLmTK3SGjFqUOOR1YAamCGkY3YA4ea2maCGsBef8ShUZQI9HupSC6w99wiVSsHMPc889SPhgL6adi5RfjBF8TIGnHlGFriZzRNqbiAt/NULV6Q/aBzKmEE/FSyUXE8pvcCuQIwIul1I4e1h7SboKPZJgqkQrMv8hcknjzRZHJMEbAHCR7BHyhQhVwbTuCykYjG0hPtbufOONncicPRgL0UkX1XJsfb/XlS92w2FEJTvq6FKlHoiiH/GQAhoXvPd/67d9/q8/t3mzPr3sm52I1NoN+hOJxOgAsEZ1SOKqG83Wpz79xbvvW/b6HJVmJT0Rv3Ljld6oBvN06corluPIt3zrezZW17bXV374h374i49/buNWPeCPZ3bLhULpyhs37cOnwkHf+vrl5eV9NE5UGDCD7oYCAySI9nuTjRZpy6QJpFqDs9aeTMef/erXKOs6N5t0OlmC7P59yy+8dn4m7ZibPbpaXa31PeW2xxtZGpVbVHgnNpfx3jmYRg5rw1oX+cqJ9q5mSehH/5uvfOoqHImF140crNu13fVHqGqEMcYeoir5cFgvlAgMDKf9qC4j3uDARggCwoa0htwOA+/09vLVlURszmnLdltP3vuOE7/i+I5f/OlPjYY1my0IQQEzkI2m32uQ31tIH2Rr9N49soxRpAQOCE038y9jRZ9dSc1IutFqKViEzKoIxBjugSI6yQ6DQReRBiu57F5fAOQAAAIm0F3YQq9/TAYj1ggdG1SbpuDQkdYRaUHAhBuYfS1/WB5RAWuoO4ojIJVpYmaYRE4spTSwaCZq7wP0Bj/JTaSxQ9aAC8B7QuIR2lgJT+xG0m8Sr+FuNXv8Q5ML7+/3RvzBuG2ILyDFHuWW5LR5kVsxhcyk570IsGhAkE3JkABqVLIy9KJMkCQwQFPaTsziHYCznUJ15lGtH2hxBfVXq4aKQAo+SvipnDr7Ak24aJdQtW0EFiCtFapAKJQLHKaUQCJs2WIxHPJHU2Gk/XqfolOFeq/SH9fGwWZqyb1wbDYO6Y22bY5yr1/qjKoyVTn37ARsOVDxeER0CCWoqDuF5VnkFvCTZx0Kbn5tNxS/pnjkDoZlcv3CAEPzAC4Ju0CfkXr5KyA1kivXDS7SV60CKgvdZUGwwJVpF3zyJvSZTJOIqbkOhYQwSDwDcszFPfDW3XqeQ2DKA+L+9EdxtJojr+DMmI7hQYVzIF19YsCCbqwUprwD1JnhsgR6CjKvyGvAjnwcEqMBUkMjdfbm4+vcrhFk2WYce9TXus/IBZxakq48GPYmwyK95m38bEnJbBgzc2rl9hV1RzRV84o0JRdXUVNAgLnBE444LuNspaBe6ZmVeQMzsCRgEk/ioIMMBmGGBcGpCS0qs2G3+6QwBBRxjsD/iqkykWR4LFByAAQqaISoyiIbAjXjG4x6hYVPRaPwAdAeiBAbuFIqUyYD5A7S17gtfaZZUMbBFaFtIuXAc1Syg9/BaY/NCLgMBo1xA/F7Ip5q1djNCgJkFiziwaKTtZxoznanOLJf8ZMZAytpp7u6cguVr1zosV5Ti5QcM0TikgLY44Mu4b1ieCxBHaSdN7HG7MpzzzxLuR54bl7KzofskasVHhf3K5musQwRpOD3Q4bwq0KfnEwkVq7fiASVjgOyt7w8i1zFYNNTk0Tm0BNqFJK98l3vetdnP/tZfM0QDbkNkFJaHuFZeCAVQKNlbtZ2AdsYHM0Js8Hs6j9Iowk9gibSODw9j7Cv6YxRAmv14XBpmXaMgkEV1dCyUigCAzNEVmukBun410OSrPtphDZpUK+2GEc5P3CuDpm5gZvQDAEM4CJSJAIBhHZH4hF8uZEAsvkskbj4du7mK2TPqDeqlVrdH7JRU2r+yARdOnL0UKVSuuuuU08++eTG9iZ9y1XxCHKcvOc0WBt2ZOnwgUcee/SRex/6uZ/+mZ0brel9o3gsisQCBp+OxSHtEIMvffmJhx978Ed/4ge2dtdK6EMK+UJlG8MVmcLbnQoYwO3woEHBGAffe++996Jy+77v/oHPfOqTePklw+ljh0/ltp4E1NE5b27frFWL0Wic0g6wQPVq2T7uUKaw0cb8UAd9U1IukUgy3HQS+kYt1kKrVrX77d3d4VoOn9Mja7nXyMA/7427E86ufXjynvtf+dsL8ZCR7wDNbzyslX3zNWt633yFcy6a/f6my6A2roEGYGyAFsgOXyXJCDWM0AVjXSSf0thnB0p8AQd6KwQGBBiJD9BIwmfQNxS6vUarlfEGvGcfmPvf/48P/N6vf6ZTxbLTTMZSBHdheem1q/EE0pcxzrHQaO0Be9TI0pWJKgMhrCPnQAvsCCfIf4TWC1jlCaDwP4KM0c5KLq42iPmGN48S5OB0l/31crkCXwnCEb9OJh3hW6gPEqOP1B9smQZPIEaiEdHOBnfL4oNyVojYOgxe1SRBDEE6vNaizfy6t38gOPihCLxlMwbW+TQ+YzblAMCbF3xH0HgHPEXiGK8Lr/V+wDGSkzOEwoNV1xeJ+KN+T5i6yhhkHcRBEQeEHgv0iowpAoW/qRTcEFosLih/6P10IEaiXDgRRgDpbdSq1ERDuS/V4BDmQlKPpenlQXYRrWCfA5+QcByjM0bA9oCqUvR7HJiIUJei2MvUm4WmrWALdcJTY19qtP/ueVe8T1CvzbHbHBapOIcFGkGn1ihCV2F4QIYIJUqCpRBhScDQXhgCKaTJB2gHhbXQZ3caNbTErKV2PgwXiJwRAVK0Yeb9Dhm+A4igZgOXIjbcaKEFpHpOrYdohDPwOQ2AziTCcoWn9ByzBxGStpOvEnd1qMtQOHPKDaBllOR6miaYLXEEZOKSZRwZVr4hUDQOFLcOXMdlOKFptoOIKMsKKGECgesSjhOfIQmGxnk7CExvMQf9NH3WF3PdIDhDLzUQ87PWZu8QvuM2fd6m0N94hV/3qLW5Druw9whDszyczbKDmqHEMgNj8cWnY4/oys9ZAIlqVg5W5JTDHqx0V9wmtTPqGaoboVVns2tarJkB9OGa+IJfn2ZdY+QTiyz8CdNMDAkOUPjRoBLALlht4LXa4IYDB/dLNGi3je0WgUp6EaaxQWJAYX7eC0SIIqLBY/8w22is2NswNhB51oLq35ilMplcvYmjr2BGXKPDjv8wJAHVdzxGyaY+wq7b5ccTGzR66+bNUWeICpoIKfhxOovOFvLX6pSVhoByTvgXmpexfuioMYIxuAP79jNaNghCIrI7MTPbW1v0De01WxqqzErRf05aWMLR8bbwCPNvrK1HQ+GpqSnwN+prJRIiT+6gT42mQWuArzWDrNclsjMzaOPB83L1GvRJQslRyhcs7CaYMVPKCYCseUA+NrQTboM9C17lQa5zWDCztwTAp7HRalyo89wuhHuiFv3OYLdSoRugJhaCvIICcek2+ablkwKH67zX6HIsiIXy6SdzhR+huwJkqjCRazMcDDhtlJyCkaNco8vuwzZ45r57ieT2BvxfffLLW1lSL3nSk+lSuzLyDTDcYlC8cPlSajJ1be0WidCag54/przzsEf8OqrIpwzD/Kuvnf/AO7750PLBQfcijnXgX6o3gtpJOYv0NRGfmZmf+ehv/9ahI/see/vDOMZn87XpyYlkIkJeQg4mBADAMX97Z21xaWFrdT2bbfVbvXe8/b2f+dSn+13PhfOvr2/YEjHbZMofCsXA8LCJ8FBs4lg8TB4oXzBFVoRcvhwORbt9XFRhH5uKFK9Ri4Cd0JFtxt4pdTd36xdT6SNXSi+85ei9x+9Fq16nBAVuN5rT/4eDxbJW6h/9nZ+s1QRt6k6zucydnIvN54oAg99ACMC/8ttrU/frUDVhe2+ILM5eGzpwO5SPMGsWl3qBJsZs3EED6/Fhm7ju8nsffNdBn/uHPvLTf7q4vDiinvsIEt2enklmMkVyp6IwYzlQglBOjw5gXELa1B4U6Mvua4ALjbQsMQaMhCMMdcYO3AXJQoADAS+ZtBCTXS74TD95KMhnV2/TOwRfbQGGwVuYfwXHOgMR6nhKbSPCbNFZhHgN5fe/4+0WoPMytgGfPMkVmrhzhV4atAReEsaWCpqsLuJZGYy6DW4iOJLcfgQEEiYAgqIaIDlUbUOPox/xOQhe95PNIkjqAfLPu0IopTtNKBY9sXzQ0DEZvG5H+IIhMXINhmLqMvpdkrldjvLOBhlMSGhFSBa4iZlXbBbR19B2S3RiCSFghlbxByoO/yzb31g5GAjCxlMZroBHmq1SrZMbeRvhGcfkvkB6yeNOj22TrHmFvU/tH+UgUxUcMSnG4AHllp+6+DSVeZDmGV04KyTipmo3rXGv0qIucadtI4kHb9VhBF+AC2aFDhvQhfoCXnSST+vgx70z/hihEFygU2Dw9icnTLoF+8h9rI6IqHUBcqxDXNsd6gsKY2L1uMI2uJU3yqEOAiveEbyFiHYnfoiZZ1hgQ7QNWl3mX8Sa4aKSNnIC8ypUCX60kp2BxUVSzH4TZ3X7MMzG3hfOBUjWhhKVNQNWn+/crxss0qstp0NDhJRa53yq4yK6epwDTsD85SsrisgrEy+WB32F9NJ7VCd9zByjDtWNVMgMsie3WbydGy0CCii2CIZFrEKphNSJVwbZHvFadOIhh1fjgImA56WkBqkEA0EYZIwyvBFqz3jF4WEC8gagnLFwZGoyTY8wdtbKJbx/vdTbMyZPKFkLbRTuFSbZIzBroR4ou9aJQwRBQKNzMZV9j1+xXIiw8URMhRnIyMbOghhg1SUtABW/nQ7SL5t7JEoi1mAJ5kWFXB6ZFWKJNG+0sphgtIVpnMUSryDChojdg2AIgztsuWqVKrb0BMdddEjE7BaK+dmp6XJJDCtKJ2aNnHH0EAhGiHb4ApV6Az9f3g454e1bW1vHjh0DP9y4cQOSiFqY6zSInvNt73j7U0+fg9zC+CCGgxbqtQqGap6FJ0XHi0CrWVIVVPgfdgVzyS6zcY8AGKVhMAj0WhI2TzEJXAY5MAN0yQInZoyxcJ3b+ERYZ+oEnFIv74nI1lrQW0xQ/KQO8EJLISHfOtzxNEa9Uw1byhVWhJxYDhIT4EzCXveGidGKYFbnHNCbmZt56KEHeNf161cx88OCVLu16Hys2KpMTKYofrC8vIigPDs3zVA//3dP8dJoUmEmyUR638EDxHSBVLxt519/7FPlekHBblKuxspN4sYGgXD4+OnTh48dvXrjutvPko7n5qdhpLbWLx88sAiAY92A22BcLByWkaA/BOzS38x2jhM2+Bf+7ovUIGzWSHpvi0VhSEL4ujIDjBGkn0rHAuGAP+grVxqENkWiaXI0UoQrFBzldtca2UyK1JguNK9175St43ENQ4le13/z9V1nsX82lV60d2dG1Xed3GcrbnuVjeHvHwyTeeSwfjD7VBCOeMInX+9ct07e/NV60HxK0ERxBZJAFcWjhkVWMi1XwOaLukOJgCMM7WFboKQkqpfE/yNMwiyQywes+DA+9Nop+2jZazue2/L9b//q11zjqN/tAzV02mWZ2tDeuv0AA6QKLwQERV7GmiLdCkJk8ZVKhnk2PXSQSAKaSf9J9oTmUIkd+Wqz+bAcNdgfI4/P7wtEIBDFAp4BRRIbBgJuwJjhsCVJSgNRZWcBsbu7uU4HPTYIXP/QMSFNu/yhIIsE4PJKIJXV4pwrQDNXOPbmlLgYYxehlw50AWwKdrqoJihSjsC8hVypTAVRtyRUC/rD1OGlyITPH/e7KHka9LtI7epRGuYOhuO+3x0A+9APq33IKb2gFoDXT9BZl3YjkXAgEsO428xn87ld2ah6LVJGgZ7kmoh/NHyA0LPitkR6efXt1iDhcDEdIkogjb6hm/xBVDfokLen0hvXAjF7cjo4s28quuC3JYnlraFw7vcKfWdr5BoooghVoN0PgqNIO7lE2JYwHnxqr6LQZeFpD6sweA4HaDFSZA6pCV13bSTrgn+A0BpufI/uwqtZ+NYsqgjNHjyqaYiPcAA/GbqMIGbOLWRtyJie4hHJnICy/An0mEHmhjKqa+YZfYBVaAi6r5s5JdAXRgEaxkt4M0+JzxRHgMsAB4OySBsnxEsLTdEPBkEXJe8L8wEVHNwAbFjgwSdAr/mwXqM/6qQ59GrTfeg4Hda6WBtQ9FaP6Aa6oZ+M3GlIrHVFG9m6qFuAVLOBLVmINvlnFDgMjMUXKeV+KCvAo6A05FmplyHGhiQr9AjUbzTSeBYD2COXqLjSLCA0MwhLIU83oDf4EdCKwtFUf0zO1Rosh5I8DgZE3JLhYuwLQqpr5EOu1adScbAhGT0gWp12SzTAyLJSlQIyEAPbAGdgy5MIYZQDSVd/WEUCHZpNtI68ibg3ZCz8M2kpGg7CA5HmF22njBegA2cPaieSjHeFGyfKbqMGvGEBlTYMwxJNqRHlflVVNzGLmiYpSwAfwAHmUePEyqPdqZAhaYlbqtuCpQRrEkIqIAVniqOpph1RXnWn5V9G9d/F6dnNnW2kXoyZ27ntRCxO/iaoNPgFPgC/bqRtpAB07/Tzl3/5l2/evPnyiy8UCzloM68AxmgNlAK+M3BnXiCAABZYPjRTLJAMXoAFQ0I3rnNhF9FdzoFZvoo/0HQNkd3BmHzlJzYpP9FVmuJd/Mo51/m8fTAZ2ir8xIrojfJGNCeMma0C9uAid3CYjUnCKtgA3kWoWqFSszdrkC7dWcy/dOG1t771re/av+9LX/rS5Su35g5OZLO5scdGrtpHHnmoXCmAl3Z2toLhwJl7D21vb8EchKMRsFUxnwVhQSaw0uV384VqNTGJvO+s1ytU/g1EbImJSLVZ2djaOH7iGD5ZV65dBrmfPnlvNZ9PRKap35zfraI0RY7Bww6ogwAXMnkgJ5vJx0Okx5rdWGvg9Yv4izd/pYDvVwVHHeYTwyD5/Sj6i3Zyc6fp9oecMJF23/6DpzL4YSW8PvIgo+lQqjcTBdux1QkSGCvX28MPPzTrDIXyeefGtbiNZtcT1pY3s/X3PphwTaG5wfrkiuHdtbLmVxAGK7432ZzszTsIkafEYMGbwTSwaYVLUDvi3CEph4ihBgbKftvWDoy99qjHhqO5wcBUSgC6iTMnvNAX7SIVDl1UsM473TszB/b94q/+wC/8zJ+5Iw2URnJFYDMMnKgetKGc+EMBdfgbATMCJKtf8Dp4Z5huwnYTfjkku6XZ2VyUQQrdL/1hI4fCAafLB74ESrHQR6JRuPpioYqatm1vMPFgViwFFhQLNwHecucCYDUP9AEIxwzrF2ga6wjfLWjmxIJv7rNmmSsGjnHHH1i0mdbVEryIsoOypYdhX9jnj4wHGOFACYjXpLwLxP2TbjayPMKp+IgQho6I0SlNJIidyWWFhJcYPTZVO3lkCISIqhgi4FbcJliiUSuB3tAtMmcBv9KJDUhH3x5KC0yyKlEULTsNiepoOcBbpLzPu8MOT2hMerNKJ98e1Zx+uzfWn9nni057JufjtqTb5m7ZbEWbrUZRSUewD4bECt+h9WoD7BZEXx6NjepN7UrGqReok0IIiEo9OYt3mkQxt8FFsMjMFChB/lagC/WF28F/jI5fhHZvw6shpLe/mBnem+S92TZzbt1vkVnr3BJwGT0twaqItgqwNY1CKUJv4pIAIOkvRWn1I9hFtAbwhgLDjhJ1Bu2Se94eg6XZF2XSPLI5wDtMNQZWmmYYWmKpegA7kSMDKTxhmtbLIWZ0fm9ot0eov9Y91q2cMwQzCnAlrakLZmhCmtqGHJJ0rXM9ZF1R/61fjT8wNzNwdVQKK1EdyaXwbfC1xtwLESWelxlSAWUlj4Fv5VwFfSGokoB5nKa4GbJNy+o8ACMxTD50wsZmFuzakJArRo/Xz9BNrkcHNfwQ9XBfOXjgEF6vt25ey2xvkbeCl5Lbj3RESGZNI1cBPEQi8jT6TMiSGaOYEMgMh9okw3IgQLxSPJ5st7rIUiZCAaWxrVyDoOJtBP2z4W0EUW+TbwALcRUTDARY9U6EF9qAXZtf6SSqZQbCrqQyETgC/0JjEyXbIotKdUAp06HBaKtok/j/aCQK7a+USsiskVCY1cCGPT2Zpk2re9Zq0U9wBH4B6M7D/kC32XLG4qeOn0A/vL61SQtotPnE/QplOP7S8PjPP/98oVhlOMwPX5Gna8jCQ6r69Pw+TbWWlUMLyTLyBgR2cnHvSVS8HaIuw5uZpTvIR2tkDrAQByPlNu6xEBEwacYrozvnfHKv3sJojV83j3DOu/YIsGlNYZamP+LHDA3WV/TtowGVdGfnyZ89R86ezczO+uaaygAkA5T/q1yiam+dGUMCjk1Et7ZyE0tRh5coGd+QVNiSoOTYnMtmKpRNrWCTGs3OzvJCWJCZ6emEL3bjpassDy0gsCPAAXrh4DiexhXPB8tXKhXwUv3O7/rgqVOnP/uFz+IU+83v+2f7Jmbzjd1LV16/tXptcyNDCrN9S4vwmxiAI4E4Qm88Mvn4F77caNoOLsfqpQp6YsAJ02+QYrYYDFEDDZt0npi7XKk+MaNs5JhYE+m5TLWRq7fC04vU8bC1CwG3Pe4Ndr2dZrFUb9hPn77XP4ilPOGlSJrIqNnQIDBqQhrNvDJb33Aww29eL+s364q1EGZRdNm6aADgTlMWxhBqMC4aeLkSGidcJrwrzl8iAS/vQnSxaMrMgIKCXetQAUGP4olIot9vU/WuLXcCZ8tuyxd2ine9/f5f/+3v+eWf/4tkhBTrg3higrgl+BF2aCBgYwJx4pP+AIKKxy2BOvKr34MfGVQpd2d4GDR/cJLinuk9fVLtEzI3+kLhOC5Y7FZkNh9GIvZCrYqRq98YeLzkz7EwI/nVENvEM/AWWYXgpH0kffSDk6SwUpv8jhxvDusKOIKJY5r45OAeDvSS4DrIKjufxvhZBBjrgQEHpz3gcYWxKEpt5vB4PSGfI0i1CbL/S/BFfJHeEP4SlODCCcXaSzwPtdDg5UqtMFTRiB6eVqViNtNq1NHaUyWmWW3LHQoEQ9QTKjKCtIzWWhYFa1kNzyQ5T6KRPeQPDNzNerdY6u20bKXYVODg8cXkkZBtumIL1myuPOxUs1tFdUfFPhwOEQ+g+kFXSOgdCQigbTdH7SZIxDQM0QVlSPXKuCVmIcd3bH3l2urTX97KdHOreoPEKfTCuSGQIpNsA6RUaYP/3mEo3x5QCv1zWOTHEB7RK3PFfLIOutNIsUIe+kVrZKBzr21eTH91FwezjuUaAkyvzMMCfxOcSu+kr5O5U2AsZyNkNuiU0KRGrMeBBICFHppnBQtCmxxSw3CHkS3MGM1V8SnmxOqYdW0PFaoF01uessgeXQGseJlIk7rHV+tEN5hzDU7wxXWuAHmMBF4GmqMf5EUlnTPCHh7OKiUmeZe145P6e6yhTvB8ljGYx1HtWDfzMISfr7wZhgmoYmh7cA6EoXaHVzbNysVOnCypmHGmlQwKoPqPnziBW1GtUspsbTH1wCREkeliWwPGQLNkUFZfygP4awmIgDrkGbkN7SYa2mQqBS9NyMriwjIo4Nq1G8yDx6HQO/ZIFOecuOS/dHoS/hjGAG74xtUbELlGraniJUi+XrwrtUjf8V3fcWPl5vVrK+hjEWDZH0SaQvxMCi1tGDCXlo3xsZCSeAfXr65iaT169CgCLk7XTEdqJmGqb2kS4DyAZF5B4+iouaeYyZPxi8BfCNDMzAyyFEM4dOQwbr0QY0OJpahngDAoKKg5BwfSQrlQrNW68QjlyDDciH8FCOgEOhZwmfhaA2bcj50WlAKrIkEEwd38QIOABkAIwuQKP8ELMMX4uyFbQ+bpIU9B2/hVzUrE5S17Bxd51JIzuM5X8zY2AxBFT+gN5hZ5w3OdSeIx4B7yjtPPaHcnlIzf+8C9Zx6478VXXvzqU18l1yarODs7j0coM0xc9dvf+rbf/r1fb1RBJuNQZLi5ugbc4NY+e3Afqgzm3zepvhGdCTs2Mzm1f2Fp3Bm/+vL5dqNDniGGgC95bJJyL/jkd0lzMb24nM2W73/4kcXI0hs7l/bvOwQYh1KpGh66XeSB5PLCYWLMSpUdxh70+zG0X3j1jVRsEh75bW99x8svXN7ZrkzEgmiECO9sYHIJ9JIUTqe4ccx79ebrpPVgw9VIFGojqYWPwlzecDJfbKcXpn2Un2kOkyG/29uvE03pjE94lx858UF7P9S8vjobt3nTN8aF5/1B7VjmUR//o4P51DSbWdfGNifWFZ0bvGHQEj/pV3AVeJu85rDI8KAS8FgfQ2IASFQkPDHA4adGLCeZIshZjeeDC3nP3qT8FhuCgki4BpE1utZ32gGQ1MyjtdwrJx469Qf/5d/+rz/6e1PxKFIv5IctRt94gh1Ur6tWJnsNas5eI2wBUKKTwIhSD1Go2B/iNQwF8AOzsFlQYzDhkWiIzcx12lFW+V4vGiEB+1Q2uwGeh5yG5CXiZ8ex05G58eGA9OLA0SQ0to9fMFMABpOTk5TdqBrpE/1AJwZMA/ESaaGwvFD9MbhEPvtoNRR1BCcOusZG5SDnhLzB4QETLhzVOkEcrEKkMfURgUDaGqor4EmCNR2ZXcILo2W7QEMkgMFvgL0M0sbnjSwW4JVWrVSolGqVIqpdUskmY8Fhu1XJb2MAGfaaRIvADaEkdga8yHwoHnAiM0SDiYE9wT1jMKLWsAvlV6nayw5c1dRB/z3HDwWXYzY/GGtn6M51RyX0jMR0ewNoyXGRIZlBHyMZDA+O00wIqjY32wIUgfTEP8iXQA4OFUtCh7AHXOngBKUhHKOOZ2Ll8SaGibxlXmAD3MK8iWgIEUs/bkAWCIKCWJ+0pyb3cmdYv5ulF7nkF4BXfyy8IdK11wsQkJCFsKmEPyZV2jvRJHVQD+uQtyczizgAyOkn5kYIEBFY1wkeIkG+vG9lNIT/IKUICAyfApQ/NKf9wH1sCkbBAWbjm9A4imdTaB76LUrKd4vm8oB2kD71oUPf7pBVs+YG0Vo6qb1fgS2wnvoNfTAjhknRdPOwlO2IuebTvJnphDUWNIqCgsJxfiecF/2zKDHmbfqupFeSgPlnUV/l3+C6FooXgNuJRwLrMn2sHmIxS6Z1pTUAXawFd8KTIjo5xzhA4u9ukjH10Q7L+ZnatlQvqCAxhMOhHiZb5JB6k+nFXQmyFAUTj8eQJVIxyquVsunxGEIr1AiEy74HyKamp0m2vLGxhaqHKg5IwvgkQ+qYYDaJPxokASDzCpIt1Woxu2thYenIwUP/7qc//MxTTz/+t393+dIbUsCSe9Wh8oig+BphJWQUN7GtzDlTyXpL3Wa4aq4zWMRQco5g+SY1B2nUgSIMtPBR9AQKCtUE+wCcQCcsAnpsr8/HhGDDYqeRli6bzccSUXS/n/vcE9NzKR6pNxu0DD1eWVk5fPjwxETq8b99GpPWiGBGZeHwh4ndCQdwFger1KsdAh4MjFogwaOCL/aDqCZ0lgAwOH6Qg6GmPMKBjMucw5RDzkF5mLQZOBz6/MICz2OBtuzrNATrzhU6DFTAlkv7ZOBXajEZxbWT3nxwsxtVpwVM8FCgIzokRgzJ0DE7PQ1mX1m9mSnn5pcX2CCHjx9b21gHyVK6kWc/+B0fLGRzCEDoMpqN3sJiipE8cPbhF185FwvHstksozhz5sxOZvfy5ctw5kGSNROChqQy7HnRCxATjqqKEBnvOBqO4ZhPmST8WXY3N+5/8OFScfuPPvn77NMzZ+6eW1gs7NRy23iM5rF6Hjoyu7CUXrnlm56ZDPlDhyaPvPXud335mScXZvctzi3ffebIpfNXTSp3/GpBRr1hsw6KS9vigRBGhjSlDJ2uQKlM0qLixHSYMLrUZKznbcYRD+01Zb0P9iu1TL0DEosjvb/42hvfedcPrDqLG2s3l9LzvcaVwbDMRIPBwPjY2ZhSNqrEDxl1ZNsRDhAbo0ACrrOvoVFi8dlIAsuvH3e+cML9Wj5zj+4iHwHEgaYAEhPyZPHotAmiG7TZ8kJEwD+B+/aAq1enKrbdFfKC30CMODc5XJTI9BYLTyZih0hcOLVv5sO//MHf+Y1PeB3kVw5ZATfENgDwoqnYW70YUN0BIgQC2BrACFJlONHj0g39pxQUyleNezO8r+L9lK+G+Grkm2A4JndLp5No+KnpiddefiHgofxXBHdF+GAIMGMGtoFlCnPmR3n2ApGzJPbwU0cbAgxwsp2YOwwPiVQKvFOrq/R3OBBk06IbAl9LocaGRmBmol2o74AZHIIhNWB3T8COI4gPD6+gJxzxxQO+kLBzR7vZNXT47QFSUzBjVPRCIyhiKfkJAHQ12x2IZiIeDaaiMLj1zNb6rV0nFhXla/bKeVGCDCxkNxakkmtN23tAMGUYC9mgZ5ucnCU7j/QCIG30ba5uz9Fs28tNR6VtL0bmHQvLoYWjB2zTqHqqtuEluUq5OkOH4rqMpoGdr53AoqEvJIsYLg4kQmIngqzGHW1JJ96geFoTxUVfEV+EEMD+jEDu8qJQ3CSRzOA71LoCOeBGIMmP+l1f2OhwI4YyGbxjYJEP7pX2U4IYi65Dd5v/FXoNowS95i26Juhm0yIHGH0dd/JW+RyZf2ikreYhWayoxFPalvlEHB1ZAPGggukiFlPxrsSOhXu2wMhPBnjIwez6rUtb2ZXpqSCMOOuK+p+3AzfyDjS1g7SUCHUSRQSUcs1i0jVCeIC+HNhFJukg4qyGJSoqnzWmVvNg9o/GxjmH2JPbh5Cm9R1iL/mPEfNqLsFnGGHXYtBgHZgB3iBVMq/F+V2qCGPuhZGUaxXAijRL+V6Tr0r6Z3yvYKBFoYkk66siIRppueIweSL4svKSewNlB3Mpq4e0dUhgpF5x4qMBg6jwGNEGKpzjkdT2uUJYHc6/9AxCGMU7IBm4z/AgqJlwPiCxUq+KoQl4jeUPQFd+6bmFeYSV5196sdKsQ+d2c1mKAZP/LxwL31y5DgWKRUMYEeFiIdnRVNwOKzqyYSB6+OG3wnb/+5/6aRhj8gAePn3muXMvHzlx8o2XXwblIW0T4bC6vvbwY49S9nWnlg8GnaVCfXoyTuwQYKjs6CwFs+iwtfHMZ7Au8vaVceFSN7YzU9Ozb3nLo1/52lfqW9vwCqViNRzxg6uz2brDSzBruFNudsetifRUqVKuN9vUN6Sk9tZ2NhLzo2hGCUeVXNzLd3e352ankilXvVz1+AERMIafpDMGDqjc2U6mY2ja4QhQW8DrKDMde1y2NRT4PTIXsbI4aDD/Mu6w8x0OBVIrVgYQomgQaMMRcONIEiw1qleuXQWj4RsFEgR3Yf3jcTgKBguE0rKRVGCKuUCaBvzGJaAAcUC1MA934JqETK7dCc1GDEbmQm2Lj6GnUG8STgyxb9Sr7W4Lm1+hVrnn7P1Hj528uXpr5foKjM1f/vePzc3Nr17fxOP1wPwMziqFcubc0y+2eo3H3vHoa5fOE9ZRKFd/6EP/8uOf/ASZyI6fOrGxtf7K5QvJaMweHpd6toUpP9FHTjIuNYZ44cD3Byii1+turlyJTyd28zvpmYmhc/bC1c2YczkWT6anA5n8zVa3gMS3uDyRz+RI31vJn59ITOO3e+3Spd2NtbNnD1+9eJWFTkTiOzsbkSj1Hkk8vuPwjoqYBOrd0TBK6BR5j+weZ3IifO6VL8XS4dRMiHI4OWjzEJWgvdoYjf1Bk5O1tbr2xicqf3womk4Sk+uIhWcPltZencIx04Fdj5zQ4l9B0wTnUhKEzQlS6EFrjRkM04if1AxsNHosrxmz74Uibh/GyQkOjLUT7tDe5n/AVvp5RerL9YrWgGHwAv+BAqDrCEXIEtwL6+YiyN473XIHYV6dhF7xZgqFijhjw+4VQuwQW0YWOc/43m868m893/k7/+cnecrvRb8EMhrVmhUPySkkVAjp+b3YWELoJfKlOlb8UCSSzVXgVsPRJAjNDxSSvwKH5kCAXFn4yLMvWp0+igwKW0ViSaRTXJ9S8VjIJ8fhsBd9KmEXSMOOeCTY6XZjPs/y3GS90bpy/drWZqs0aAZCsAqdljwkTYYeguGBPHfAF1PmLQQ4XgfiBUdhPMazA42AiutAs4i2lKqYaCMCl23wHoFUMoRIqZKX5KaQCpH5YuLgMXApRiChhiXIDg9vaXvQ8MJCpabjoXAEAtLIb1fKmU67Jr9z8vCAgyXFmzzJaoI9i/7HjVUFqYKRKOTW5chVCt6wX2EH7jH+U81+vjLK2sOd+KJnaT45ezRm81XHvuzIXieMHdMYtNXpdQ4pL8NCiYjIJkrLSE+gThACCN3k2yETQIhByxsPibtasGM3UEQvG9UQS8DCcHkiJ4YIGzRnSCsXADqRCMGZgSm+C8BEZ8xhSI/ATvcCSdJMcyIyC/VSi/qJWYFW6EQESIBMo+ZtuqbDdEGPqFfSz6h9OqGfzKssDhUthUlPggUQfbIMk4wDJ4u17d1UOhKLz3t8s6kpTI/Voa0JtOFcRldoU60Jp6FvUHOIEtbBX/PV3CC+VZvHvNHQSLqgAYFH9ZW+aGuZ0d3uF+3qkuZBg9df8x1SDEXE9s8saHdCOFkVA0qGi+YrCwSNlY88YcvIKzjNI8vCMCHDYutFlLJrl0gC5go70STJIlxbEo6keV4EW2B6oKnmix6ERLE+5irsFFgAvBBRMd4A1LROdY9GnRAXMGmr3YzGopVSvtWowt4GgwFUuqgR2JMaAy0weHNoXLx/CH8aQCS6tb4GU5tKp6EcaJKvXb/OCdpp4w8EV9kBZiyVEN4cl1dvEjr12Fvffs/9D1x44/JrF69BtXY3tnY2NhF2c/kCOfpxnga9YArNFQp4SEGKyH4BfYGt562cWCgNxKVp5cC6Zuaa1JMCMCmnXeVSiSwcm2vrLCmULBqPIKbXG+14wgd6arY6RMGGfNHLlzZT+MwGw0jqyIIPPPzQJz79adR1KGMRf9PpML5nLz3/ArZ3qqpAVgWqyqfH75inkTBwxUcAgA8kGZTx/PQR8oECY8BLBe7ACytMPwX/Qq1AGsRSU2ogmeGTR0qCvNOO/1GLOsMdlYQ07ctZAdJpuLs9YOJcsw9mR5EICgRQeA2wg25EmQbkbWqdg1gls0lkBqNA+cfxmDuzu4MPcTIWJ/kDi0XxKQT9f/njP/HKK6987rN/98rLLzf65bP3nDVJWvZdvXnlVmHzd3//oxs7K3/653908cJFXlIqVc7e99CnP/03r7928V/8+I899shjf/znf7iycWvi7H0n7jn9taefWlttA+PxqK1Nmh/nKByN1ptVnKKJLg5EYMPgJ/CHqoLTy+ViJJjyOdqJZKg3yGFsYTJiyci46yFh+BsXLs3PLgzCwWtX3/jkJ5+rV0kc1b++vrG0mCqXC2vbtvkFW7lRu76WmZyawfO5N/ZmspVIEuANLB9efPKZv9vYbeEc7er7lmaXJieXIolkvr5DrKdg3+e5fv3VU2/7prMH76/uvNxuxHerg1QM3NyDXjGL6HMJogcU8XSQTVJwJnrHlLJBdWg9//8dBjTBAnduZLsb2cVcAbHxhZZYPhoWQy9LINtfGTi4DLveC/XsVE5SSQ/8XxGAh3ZUkK6RI+CCto1HBYL3qd0ZjbvOPrz8C7/6w7/7Hz7WrFfA/ZVql1jCehU+21YpjScnwA8AGo4FkFjFiAFU7Gs59iDeSeMGCJFyQjHoIheDcbaSx7NiP6Uz9h3EtQlD0gsr14HjVDyCa4UyXsHASRAZE3mDMIM7SMQTO7B/mThAEsKjA9vNZlx4t+N34wsFgTmQuEgMSARVs0xYmg6Ms5BhSbGIGyNn2IkSHaWr24lDpUr/GQLs8AfdEZw+EYdgaaUK0JQxsRhUKmo/QNgL4jN8KDZw8g05yDaOPrpbz1QrBbTORFQSdiG0PexgduUvIZBsDSikyZdJgisGgpo+Um/VSS+HXyJcfyTsq/Ry3WGl06+MA63olG3yYHDiSNQ2B9tQto0rw1GDQFAhVAfR0/Iugi0k1AKgYCtaVFH4CMOo14cZEeKkr9JiqBAGm6BeqYoOQAK/EZgMGaIZARogJygzN9Cqzk37onn6Zv5XC9YJf4UOdBOYQSRHX0ETfOqy+cXozXTGRPLPwqL8am7gdawMFI6nABqgjgSMupkXKiumhF91iU4Snw02I3AZuwOLCYdBfs5So1UpUEJ89eY1dlrowMHpenUQCri87qh91AYhmtkRLLDPWA7aIYJEa2oM9noR+A0h0poVXmgEXWs2rFEYLoTr6jL3M0c6FeY3/TQ3saX0s5kZUVMwsEX4uW4NTSp2thY4mStyNANxs8U1atFdgE0FMQjq1ScKFkRYMK5U69iDgRz+sXXEZwm/CwJ4jdA64iYX6Ir2nIRi2GDgjX9cY6CkAU9g9XT7ffCmxWK5Uq5h0sF3AQUUpBR1CDgMnTO35vtUEdwrt8dXDg3TrCbYqVIogNvJ5Sve0+HghB7y0lgsxhxAlLjTks+YaoSw3e3dmfQklqdkJHHj6nWle8SXejC8ef0Gf/DoKWR2eXMhs+1xBdA8UdnQef0GlDeKwyAmWywyuHpjwUW+l8IGkqPZ59UMm3exaTG04BEKigEjkA+Lzuzfv58RYV1JLyxcu7YigGHK0AC5XMjBP/IvvpsMG7ARs7PTlGo4c+/djz70APkmlS/LO/DG5JnMWEjuCG4iJMMygGi5BegqxQpdxwrGFSP7iovXr8wLXs09BGUdAjIe4ADNGj5G06idATMAhAv8ILZ2ChCLG5VgAZHWwu0xGJp2Dj0g3kdtcAIXIC0TnL1WXBCLLycmJngXGhTDZJgVzY+EbYLB+r6QD6fiTKmUz+Xxvoul0s8+e+7SpWs/+IM/+MbrbxR3MkdOnvzSl74EEgLJvvub3/fMuSc/8pFfHpJ92NXHwYX/arnKzcvXSrU6pQu/9tWvwaa8duHC7OJUKVt86PQDd9/1wM7mk+QtRS9Mn3K5or1kXGu9Y3ulEUvFvIFAu97buLU9Hnjfcu+Dh5LHhrbCZvPi08+97HB1WbipCWprLKIxPnlydso3XRuV7rrneCzle/381ULOm882d4tFjIuzy2hu+ps3W9MztmDU/5Z3vuXsfQ9jz84UtuutxurqDYymwEu1Vkz4yfnc6XawpvbKpTzlOhcOHHvv4W/fuq/obyLsjqZmDgxAkIUdR/kG2RrJiucl8kVUgSA9Np/2LbMtvDAiJYJSLlEHAtUY4YEso1mZf+qHMLDaMliV1beQqFlZ2tePsNe46I4lhHcKNnJCuiM+tMNCLjAHDtXfs4G8ByOvB/cozIbVbnuLutYnHjn4sx/+kT/6g7/KZ/KkiACuMLROpSedjjKMOFDqajQwA7PTRX5d7nDYyTX2EegCqREtEiDDq/kVICM4Pj05RQQa+xfNMBfZQgQKo15CPdMsNgg3CARJF4FBtI17qJxSTO3BRHwiQPWVcHRqesY1M7eAQI0lDGEP+ZJcK6AtkBgbTzALZWcTKCMNVeJxdnbEkN8JVxE9JDEVulsU0Zh3fIMmaE9OMUA9WkppkZCTx+PU9ATOwu1ebdhq4euEklpJzKHahM3mM5mtnU6z6nPhYooCg0Q7Lfe4g3oAnwzUG+jkFGpEAssxaYwGwWhy6Bq2hi2mmFCRpr2Adqsx2g6kBlPz0ZlD854lny1BbHWWMqydUWXoJm8zjv8UdsPEDs/L/h8GvFS1EdVSvgacv+go3Aab3038U5tpGLS7jmGDzDC45nfbDdg7wIjfLUJosKsgid1rAMp8Clq+4RAOMBBjnZhz3cBXc0VPcSLE8A8O86jaF04AP8ANSIaGAoKyFPFpvuqibBXCTqIlAkxzWC8wQK+lUGQU+U6UH4qvdshxudzfLZAUMEIkC4vYc3uXFo4WsADatwkAo54A0qXoE3uHzSNxRMNHVmO6eAnSDa/Q/3SE7iuGkj/WF+sq/eCEfQOjIxqsH61RmR+sc1CkofPCkrRCI0baNZ8aiKgF+1ojgwuWKCfyDEFVehmihozxm60vNysFoqO2wgELyQvqDKwaFbMs36haIL3mWdgdvZK+06xB0ppWRc1qJTQW/jDljEsZLeCdKUsQjsabbazB8tFgp9Ar6C7wAwbnVtn20mk6ZHap+s3BV3XedIE9CZnkaywY4kVsRUJ3EH+1J2EZeJmZSsGXKKTChIgKuXltjboRZ896Dy4dyGayn/3s5xDIMKzGjxylKZ6gTba0gt2Hw53NrXg0hmcGIUN4UQAU4AJFXLBYZqEAbw6GzH4EMKwwbhA0jDW2bbh1MjzTn5W1VUoJnz1790svnV9ensc/BfcxHKRPnbrr7AP3f+FLXwbRkBbqrz/zmZ/9uZ/jFV/96tOqT1BvIDDAGWxvblEzEV0RYAn0wsCLlxeLA1WD5DboA9Y9pgXXDbgEH235/MyC2G0gjHkzhhhOeJrbrBNdN5PJuGEy8TVDDsY13eiTYVqwYgm/Wy0wmXqzFlCKbKFLNMwsKitvNch9ZjNbnwgoum5exgfviWEPiEWRTlJTkz/xr//tV599ulir/MIv/fKBAwf+zb/5N6QRdXl92d3M6RMnf//3/q+f+/DPn3vuhW/5lg+89vqLL7/y7MLS5LWL1wH800dPYlV0jVv3nLwbRd2Na9fp9sTcRKFUWl3dPnTg1OfGT1bzI+TdaJhgJC0WQWjs9Ea1XS7WQuNQv9X1eGrR8EQ5n+8lq61x8eIbr6/eujm3SLhLoNGqn/Afe7byMlkJruWvYBzHp/vu++46cupMMedcnD/x0Y/+1ub6+k6+xFzMLAGhaYcPFNqZiszV6h3K5c4sLj/zwufR14LHAXWEgFxmZ9hpTkzGAx7nRJzgusqLO1/xjuLblzZWRy/NeNy1jev7vIS6+J2jFv4EmHac4xbbCPSImhS0xLvgz9GMEliPiZKyuuAdw02xPv8TBwthmhLa4DFWCFhmkc1X1lVcnVhqILmNflvvxYPcib8vLC4/C+yV4B/XAvuoRaZoAq+Gg0K/canf2j348GPf13n37//2n0WCjldeqBw/NFXIVSYncC0saFO3G3ZTmJ4VQWT0hInZI/eJAnCwtcFvgR1BDMAeoMWswrniTEQ9ylgqNT8/36wWb17MIYLiI8HGRwgCzABFQJdxkBYUUbRUrBTyNTYAZB4PDLpMmJh4DjzE+AfaVfXcMcFjYbPVFcAhlpZUFnjOo/EfRwnLII4I0gb6xrhoH1CFDV2epC+aEujrrYJn8KrizZ29UNRHmJQ3ikWyP6xVahtrue0sSA6JLOzjdUMKn+Lz5hr1PChM2RLwuMyzvHilOJWJhi64x9nytj3Y6QfHufquM9pzRDqn7kmEp0aemagtgiF0ezzMdR3lobtJ4DaoF9xL5RssmU6Hj5ArmPRxX2oBWARJO0YCZfsJu1PCBSdLcgz2RvVhRf4+WILRSCAt6eXWJ2t95xDKMCTozpU3nYj87B0WERJa0MFTgiodkFVBmjl0vodouGRwgqFJzKCwmGiC7oPdBBwRCg0tMa1xJqphsJBFdiVAaENAfXDegwiigO1rgmE5sGVWioNClsHhD5JKJBZxzwsGJ7K59XpNlS8RwkjwzQtF6KjBZXolCqZe8RKEHbCcumMuMR5IiDCmNUx6rt7zTsEch2Qj7tan+cXgQUPzRHRFF0GW2nFoAnQwP4CjYXol/kBguIgXvThrPJbZJBJqEX8V7wswWdQXZ2ORM+gxJg8gFsiRLoxmmQu1wyNaDn3SDWZLTBjMLF95l2aYWQUseNgAsKtUribSE1SaK5TKODkuLC5BgC9evMi4QPr0CRUoNBhBFteN+cUFpBzaUnumc9zGOfrydCKJczKTmiKZaiKOEIlLDhIzNQcZCavJVhH1MS4WbPFiNh+IR+anZxZmF7aJgFnfxv0g6PMjOuP4Q9HZUjYb9LuT6YlGpQxnAKNcrlQtzlpMgAitpgzluTwYzHBEjrRA6hu4w5yg/u2x2gr8tNmQa+ktGAEl84c+9CEKRuXzWTYgBLVd7Tz++OPvft+7f+ZnfvIP//APiU4ORYLnnnvu3e96F9J5IZeFmWAExXwRRmd+3/xudqdLqHIbmw0EUwiITIF6O6lwDEgwe3avHKxAoUygpkygIRBiLuijxZFgAuSiZcdFOaEVksxMmXb8OJRUTDmfDVtGXRYjkXALh5oTD6kZlZ4SLlWYmlslN6A3BYeJF0Fgkd5Zhkw0K6yDWTAkze2qu1BNTiQw+X3mk5/ZLeYffOyRdr31Uz/5UxUifJyKTkQNfvHiZcRihPCNm2uUqTh08NjrF14tZirRWCBfKB/ZfzS3LdPA/v0Hgx6sicHkZDw+Gd9c2XS7gkcP35VMpqvoPckd1ug72mRiofRtEDmg0a6X8hXmwhUgpCQ0PzO/tbYyIuufp7WzsZpOJ3EmQEGazZf+4Ku/Hwmlceko5opTE+nt9Y1aNbt/+ZjdP9EeNR949LEP/ejx//bH//ny5VcJRql2Kt6gr1QvXtu9cf785Z3c1vf9wPuXFvdl8lex/LIWUIs61u9SyeceuUMYXwkIr114bS3gmaxm6gPv1PyhU4HBvnJpNTcqR8ckvRgTrs6KYqqVKOa1tVR/DDslawiDpOJy/GP+vayaWV+zOv/jD2sVtV1ZcXM7j2svsYBqitW9DQ6o+kBuTSEBJKcAHsFRn40MqmI7ocSAiuxQw1aezFkKGke+czTHjedPv2Xx5yPf/1P/+s/vuS9Z2G0sLu2rlRskQpHbGsp0PM6JaqIDRB8EcaBUwTsKEPTRZBPmhJIYLDMOwoV3uoQfNkOxBKWHgUDJzT4vJXVh0xutJrSDHABANZITMB/GuSKewIurQY3Obo9oQwaH94lrc2UX4xZJ3kmKAjJCxe13BJwBNPoarDaBbG4ykCi6iFT9Xb+zT5oukIYSScqDEM0zMTwYf0TXhJvou+ylDALvj3ZlYjoVnp+0oU6vZCuba7ublDutEvtDmUeFJHGn1IXkt0J5AwY0HsWId7yWd7OjMOSRi9Q/Kg9zNdfW0RPzgUS/t1WZWU5MHEzZDsFmZG3dlW631LU3kOjIv0rG0O6gA2fNBlTgFO1rAdnAeEh3vGhVYR5wJ+OiNPVd7WhQhtxDsOXLuQkpkLhK9BAo8cAeQl9mc+uPTkSP7xxM1J1zbuQrF8yn9aFZFH7RbWrIusq5rujf128GpHUXfIoinugyiEGPc851lYdX5iY9qWegMLCZBDqgjTaSsVoGiaHoF/MD8UHwRcgjsYKC2BGqqo1BpdTr4O/icuAx1O5u40Y+NZ1SYobu2BMLtrtN8lwrbbfpFjwkvC6cyB0NsYialli0UliNNRbVtMZFT/cmwBqGRdvUIau/Grv6LhKuf2JTzPh4GwITo4UC0hqXOQGABEOCQutcKcYwhYDZleGMjQD5QPZVsC+py6TigSpL3BcNFt3ViWnENAhfB2kWzmVzyfprTILC+TosXh121QyFbYM7Brpn3H1B3+hacFlkEUD9HCiiMdaCtnAMLlGQIBSmAc2MQeO82FpuegqNgQBDvDkgloi/bA0IMDtFK8vYwLasK/uGvvUHhBDA6eLx0K02CuXy1uZmKBiZnZkh5ocUPoQAwQriJklTsLvKZBSNyauTgz0H7af/ghbteboEueWcK1oA1kbM+FBJbMhX3mx5Ioqq5zIiHXfiONZpdddWN37sx37sQ9//r2eX49WGqk3QcwJ8f/Kn/t3p06e//OVzi4vzkGSk58n0BOUZcFFm+2eK5aX5Sfb19ORMpYZHAekI0J04edYTEmd/8ODhnR28tTKgMb5y0F/8OSj0BkU2IgXTJlUz0EFXLbCn0/QZKs4BegG1N1BTKVxFmiFzv4as1ZWmcm+M1m7lEZOBhpu1V9WCOaz7RYnNfdKySJckEzO3TUxEhWilyXPsbO2gBSMK6/HPff6pLz1BbgQ80tE3fPDbv+O+/+3sr3zkV2YWFqf37//qX33m6Fvu+3//v/6Pf/+jPxny+WaTqS999jkcoNgZ19+4dn3jVmQq8tDbHoxH4uujzcuXrxG1VFzNizjFTd0LClt4bDiWBikch3rcgQ5yOGiPG9V6lnLOwRkSYk7PxB2nT9W7mau3Lu7mifyG08RlLHD15iW03Bev7+JkHk8nJhcnalXH5770mXrZdvTE6VN337udXcNv7tTpWWLEI1F4v/g//+APfPS3fuNTH/vcB7/70c2ty+jxUXIR4EVuT3iOYjZnr7Za6xXy7U7OztkGFXdgREYjRzz01jNvq2Xf2HjmL8GYyQFJ450e1E3jOjsejMSuYd8gwMH/AdaQAhkUOYTF9fefeBiaox3IUxblZlVYKORsmlZL5n9W1pAHOd2BKEjegBOFn7fSC78QHxc8QbGixHqNbXUiAZHNcSxutq8TCLD/VOJnP/LOP/r9J+KpaUURN/FbIgkeThUgwg42V4CTvBA2Byml0Ec4KBWKK0itzrbysv3Bh4QKrm/votiAAAO9kGLAB6pc9nu63Wav0wLC6algiQIP/lBiNoUBizRkYGPCCwnFZl1A8q5uFbs+ZlZoE2m5SMQqaR7pFs8kJhVVs5lbkRuCbUhI4Ox57LhJCVYZLDRMkCoECQKTyC3RX6wGGVSpf0iq03CIDFm2UqZ4dXN3a6VeLfhdrokwW98xJG8uPpTIwU4l0dLT0F0iosTrKDk07+/jwuVs9Z3tgavecmbnDnum7nPYwp0jh32+yYFtsdPLv9x1VHnCQyG3cADnqVajli/WGA1bFPUz6nCyl8g2IJMu5l2ZCoS22XFUrSADL+DMDCD1GXWKuGIYDHCX+Am7QuesNdfC62CrWzucc2vPs/3NL/qq3S5EYN0NUhD3xq9CtuYvXw1t0DWMTmw2Carsfhm6dI8wqFoW0aVdPnUriFozDqIQGKKWARwhtUIyOPDBL5G2kGuSDnlWr9CLcSBlBin3Q4qFnqNdJ8kfTvDcR1moCIWzcFwinAaqFginGr0MNiFqpVG1jz1FfhX+USULoznFSWgceFRPoKDoLATqejlwwpk1WnVLneDz9gRYsi/PiqyJMKlP6qCQpprin5FQ+VWkl7Gzm1lORi1iDDOEtonfgCudY0eQ77fc4wBs/D9YScgwdbdk9MUYTNyKZoAu6NNpCK20kPRWr0MiEn/lxFqMuR9IRVDSj0bbzou1nYF6ghzcXtLtt7s94ojwPyLnM+QWJldSpkbuwAbslVK6h2yKVRiSrCUUzyk6zA3MFOOGQjN0vHggupBw3KehyCUMt5TyRdkCvZH7KOiEl2vVy7mSL0wyGm8xXwEAo4HQ0uISpDG/k8ElpFquwAqXipQ8LNIBlTvx+lOJdK1Shf1AvqNNtpESGRlLLAwaWxLAMoyBXgDbDoOPFAHwkC2O3BsES6EjYjYRo9GqPffc84+97e2HTy5SMxGPzUGnC3dP937v934HjdnycvrC+esLyxNfevwLS0tLVGhQADH2MEIl7XbsxHOLCzSFBEy2RyEI+HI4RBMqDTFGUkALj3O95sckJNGYwaXcwSUaggabTzMZkqLYUMrkBfmEOsFuDrq4WSuTgDEAaw7FeBgh23pGj4vcckgOkkOLBBTeoPv0Nv1qbTKeEECZfxKV+drtV4uY4GqMotHufPiXPoIE/NM/+zNebzAUDK1eWyFAC+GeUGBuBh7+9f/6b6cX5j78s//u7z7/pbsfPI0vy6XX146emsXJuTHo1Ip10v10G73N9W04w4mJqbff+759Cwf/5sxnbty4qp5iMTM7nXhUG3QAjx/qajCl9gHUN7O+Q8k4kkGPHTNO/4BKl+jGyWAZT/opE/jGlfPo77/ne743u7N7/rWXU8loaCdcqXY8wU7cmbx8+QIZQ2v1h8uVK9dv3jx1/Ai4jv48cOaRH/zn5b/8+H9tlbv8cylLIyUevNPT6S6h3oVCKCnSFozYeu2ix9ufWZpz9sMXNq46/LGHp+5aX77SzK7ZiiXINWWxQAAjW0ubik0LjWBraQezpdnNIFmLOjBV/9RDekmDPIVCRFxEfYV6jHLECGT6Ktwh9MdCwHPzMiKEW5iGPfJPoggocqfN6WOhVfGOzwFOQgOMIKFQYn5r43Nz0w89/N57Xn/55sVXc62un2xK4Bo/6S3ImGiSwQEtwjtoRsi54fdQYpvNwo6Gl+wKclWgjCQ8VIDHiYG0dLj30m8sI9AYInld7hAgClngKw4lQGCr0wZbgisgwNGIwhAiwxhIwzUdXWI/jOoItMS+xsI+6qZhGCJoAxTLPzaqkC7ygnIaDp3ucdD0TCiOlukKUy5YF3KD8WEnoKx2kjItnIwRDzUuFzLr29ntdXQaQa9zMhJnMroVPBhHfvJgugPwHQigKNix4tEOK0g+UCKHVeLU2UJp0LXXO65yfbTrTXUPPHTCNlOw2Xd9S71Re6NbKrvixMVTLtDdH9Y7aFGxi7m9iURK8CDfyw6OVap9yK4TlwY2CKJYVYWlNsgTNxJGt7fMRsWBVp21Fiog2QL9IkktagwBhQUKBllbm5mLAjB+2PvkHr4IgqzD/KRTXTXbWzcY6LEuGiok6nJbUDOE1uAGwAtCIrg27+WcmbYaFI4yRE40EGwkqk9LQvtaBX5ETNS9IjnUvWqjq4MG9x3EWLZbBPWjniTaCs86qeSane5rFy4HwoNWZ9u+GMb5ezBuUyMFGuxy9n0sB6BHchU2k3gAho6HDkIyikcLqbXlaaFdYvqq4Rk+ghORIDM/wI+ZUoYCVHNRw1cX9Su+eeZXJgEEKYWIuW4SvQg3QSDpC6QKugspYXTCpNJVMCbZgCG6ui7qa8KCxYUwG3qR5pAYE/YRsApllyxPgipxNSQElpLdmilBBh1hZumX0DSWbUcsmYDmoR8i8o3AoVw+32rWmTQEVryNyIUUDIehrGjwUeGaRWMK9Lj4f41BMjHupuBx5qdSLmOmtfJGQZVhloE31Bz0VTo7dpAW2u4NBsniAJ8F0xAMRWvNBvm2dnfxz3IhH0PFMbhSDYme4EqEo3IoHEUapuQw72VWgXgoE8SDc07MxtQkW1cAD6zlKD+sjcs98J94soIslGwTZs3mYmi/+Zu/+ZFf+iX8fmk5SI7JVpPeMkZGCgZJp4PNWh1RnmKFmKXLpQZFDrFj7O5miQaGgEm7vLcJVIOoRW2WwegrX/kqnXET1BEOE1tl0BxphphJ0lVqu6mrxpSu+R+RqxxGXAfnwDcngjGnY2FpEW1EC1/ttoqdMVBkG0ueNneb+begTt+BAUUfgek4YWjMA6wPV6Dc1q/mbfowd9uRRGPEi/h9JEMpFFdfOvc8fMb/8v0/+Asf+cVGrYatnUY+9fFPfPVLT1DB4t77H8Dh+YFHH/293/39f/7N756aJq9ZfWoyVMyVyQcen0wGfSGn32sPSrty9RIeUqVh2fXed1Kv0i3OEQ+aKH6BshPhvt7p2uodSsHjB2QLRuwElVPibnN1pVLcWd14Iz0bLTYzAGOn1qQOBpb/eDKxubHz7Lmn7z97XzAiMK02ynCS/UEhnZhOJkPJVCyeCP/CD/6HP/rC75byteX5uVgwsrWyffexe89NfLmRa80mFvGNYZOFPaGl2fnV9uVbN5qLhMomXTgP9YbUIAyubV45sHw2OZP81Jc/f37itffcf7JOzFi572qXoTA47krDwAJKQmN9hK+sgwllW7EFjYS2d/Gf8oeV4jE+DDHeaxDrptZIG0yCiH7mAF6FF+XUB5wp8h/lvXIJo3qBDRyhGMfkgp8GT8GhUj6731+fmk3Z7MRVx3/iZ37kT3//b776xddi0elqucWq4INBr+UF7QnBoLLxqyiKEJGlbgF2EEjI9gojaN/e3pycmoWOAodwmbPTMySsu7i2xjZKxqJcB0Nx0Efq8GL93d7aoX4oQI6vG9ufwTHv0lGXf/OLOEPyD6ZZ6EhVmSCI1Bb0oXoE7UJNEYE4JxgHxjYWTMAAapOBO3Appnl03wRKtkiw0ve5XfF0zJmMM4x+IVvIZrY2N0T/lOoDUYlM+dL3kA+LDCbSP+O9S0weiwSZsw27xEr4nMj+ZInoY2UbF9u2YteR77gLB05PTB4ImDK9jYG9aPd2Ri5qDbZQPzA7eKfCaFjyH8p/pBAIOpYj4pTN5oWGdk3eZkeniUYbgQmSD1Knm1pHVtpQXTFxbHPxFCBRABNAgBUwMqnAQr8bCAAGjApR9EC0UBz83gHQmcPcrzN+5JOFMV+Ea3iJ9Ss4F/Zb/KLaMdQCqVeUSZFwHLA/ABD/WDneDoVS1kHAXK6zIh6yZDFx8H64y7H5VPCHIeGASMqHUbtDZk0ccXujVsDWDRSyRJOSG8Xu9IU6dl+j7w7FZlwoKWL++x86urL2kmOYSYc7Pgc+aBhE2jCP+MiL8xtRnCDcIoubN+gNkH2NzWkHl5IMYtApjwbgUCmcIGHAKHhUmoNWC2f8er0JFJI9mGQx1NOLJZIV6jTDaEotPgbGADYkV07g+azJ1HRCa3mltIJouHEMlJKTdgip4h7ChrDV1CptfzDS7vSrNTwjCHF2qywnySb7EBfURPLUUrN4/xMj4fFiLiX0HD1ThzJbqoNEX20EINFztH4smuAQTpUQEIcdro3ZPnTs6O5uhqJ7SH7Y8+BhScJcLZfAwph5QOJwcNSWecc73gFZOn/+vIxDAmaplFkU9RlQw0SK9IZATaoeU64HfQsHfA5v5B7chIAPnjIqBTlVkSIADkn2BkCCZ5Wd3MvsQXue+NK5hcUk2xtZE3IIK4sblCWDchEggbDROIQX4KGHIAuN0KgLROFcuIt3peDiHInQzDPwKTi3owUJub3BaquBQJxIJiHw2eyu+AKiq9tI/x5KUIjsmeoLtAbZprcI9FBuvUs1tWykjOYKjALdgNs1oqeU7YyV67Fogm5nC0WMZ1BN8Aj+GUwV/ZTKzuThYmboPOe8Qjtbgq8YfTpM+yCR+x968MbVa+TRhBZyZ7vZNs9KSqazvBeg4jpRymQcaXQayYkU3NJP/MRPULLpM5/5G2j2zEyaHnI/tzEEiLHR08uJh73PDpM2EOFGznwjQkJhxdY3NiAvZHRMzk3ReRJdMZD9R4//5M/+wovnXylktp744mcJLeAJtIKE9oJHCpV6fCpqDzpj05FDJw9mq9kbN24S6+fAmQX/cR/uY6Q7UpU2CTjMgt02MUUVJvxGbalJJ5Z4k0wycPTIwWI127e3x57BRqao/Cke29x8nJg0kiES7XXmrruZAZRd0ah/Y+NyLDQ5k7q/2/Af2nfky1/+3MiW3bc8WcrX98/dfffR91dy4+mJhUJ+9WvPfPzyykvlToVaIPViuZrLOqkP77Il0rblQyF3pOeOODDHlWu9oH/u4P4HBt3Y9devLsYjP/SOt8byuZWvPJ4eVsPuXrGepcMEMHbqo5iDiu5+di48XccNE0kY9ojRMs8sH58sq/Wp1TQHX62FsL6CDLmi+8DJWkbO+SatLJ96mK3Cs4ih2jHsIPkDUx9ngHs/LkZBmztmc4Vt8eUYch6zSpEwbsQXFawitQUhV0OShU15XYdstiOltfETn3v1ySdeSsSnrl+9MT09RXbhaCQ5O7uwtZ6ZmJxlknkTJPmpp58ESilJyX6p1euRWCqRSs8vHCCWKD05f/LU3c+ee+GVl56PR52TKYm2FKFiY5KwLBaJs/frNXbEqFypgbXIp4ZuCSwEb+eq7rTZ4D5PFLRPzlmkEKoTh/wuSnRI/YMOSCSKvCROyveGQiSC0PawtgSkQzkbm21bU3b5dDruTSWYoGF+e3tni/JV4p0x9Uq1KyID/jN6QPYJu9cULsb0SwEIKCaKA0RsoM43rA+rzV6576j6E8OF/cH5I0fcC55O85YvMbSFij1buWdrODw4QrN/e24qWOK/g34H9G8JPnblZ+etrI/xgcfGjAsb4jDKFkKLkLHQcLK5JPiyBQEMEV1QkBZXn6y1Lkk3aITf23AjYBAACUQscOHEAqzbPwlcrMPcqVM9cfu4c/H2Bf7yTqlT9roBeJlHSM7KQQ8hZqjndMoUQn3g6TgTlyCg5B/3MFQp7lFGI/0oaxokGWoN1+RsdHoN0nm1iMTGc9EXTwQJzKRyEwAJoWi2ajjMeIMT/kD4wIETLz2/sm82CRhXSjm6AXcF0JAlC3RE/BLUaTj0dzve0chP0hMquLgdvrnZ6XajWCeerteEHkhnIzwJCcIUQ8E1ESJS2poSVj5stMFAgswylD3jQIHWaSmTKtIpJIcwHPLoMkY4POFElYcWGYPtAMVzhThfrRtCI8uN7NuBqcJ0QTJhlMlqDzJMIwrxhsUClqGA5mnEHlxA3eSE8eEN6lcwEoRIGa+5DYHaSIrwsViK6BtPkXDNzRaoE1AAG7exsUFGcjpz4cK1hZkk5AQAw3impLt+HzG+RIsiIkPn2HJKxEyQLoQTIHR5qW8OaULMVYcAUC000ji4Dh8/mC2J3TAaQkLGiml+QveiEhqKtMQYRNhrv/+93/PdxDbs7mxVqirzhxzJge6rY8oSg39pgMFChqFYzIUgXNAiRkS9NQRMynVlwmXDyRsZnKRftVismQMCqSBx6Y1JsQ7H0wEZypNsOEwmE/wKzYvhSkkiPGYJSs824n7JuzrMC8cb61tsNlAHXSJ/D01x3ayCurGwsLBv375rN1dIEaUFRZuhbsAJaCWEWPlq9hTzoA1gCKR1whVuyOcrxONibKZN9BP5fJ1IDK6Lb2MsosLI99qe4jwMLafnlCt+4oknPvKRjzz55JPMycZqPpZUGVbrjXrS7HiusBbMO4MixyR8CaY0Co8z4bAB/OoJ+Ciay/y5g/56tXrh/Ku/9dGPJibSo37r0KFDr75YYnlR5+LaE0r6eZapLWVKaNze/p63X791NR6KxdIJFIVKvj0zUShkFIY2lojWplCdR/9SEUckIb0GGbbJ10tawCtXLuHn4CKzfcAZjbjiU/HpuelCqej1B+ElUQ6df/kVnKbAHCdPHEjG/ejsSITkj4ZJ3UZmM8oPrgxrD519KBlJD7r1YxOn8FsCbbSqLTuteieKpVw5X+XxcJiUbUSyUSIaQRKQYp0HdXuv2anmq7swro1+fadQv3Lz2vygMw6wx5RzmRQdwVQsQ5312USlMbRTOrbTqzTqwVi0Wc4kKcgjPfs/9QDkQMVCsiAgcJwRkDg3cGEagVAxg5Kr9Ve0RBhT0gjUakS53xZ+vLZRsUv6RzdVAZli+AFWj6S85JNuDUJhILLdam4GqK40u3j6nnnSHt+4tkWKN1JNJONpHDDbrVuhUBTIocwK8FSuNBEzmSLoIGHawGkqGUf0BH6w2hACALDBhQOTbHr4crAO6IXQVtYI9MXGQcuMPICnNNpZtzeA7opyu9TYcKXDM/h9qbYBqIsqVGMM0W2SlZFLEpYB+AciyasLv46/OnSYLPdsBCw7MrEKKFE8oMkbTizODZuV7PWLxUK22cCC0oeNpxCUa0CqZFnw5FUFQ48G0JAY1Z6B20RwdfWqfbw9ynbvwIN782Br5O9Qp2jxUCp9KGFLswY7nfaGbw5nOyxLzVa/AUdDZ00zuGFTTBo3adEdqldBMUnHi8ymNQG0cakmiyvJQ/FC6oOh5DSmH1gwQxcl+KLZMvpe1lH/tJxm2dWKYdPMZe1RQ13NCYyzgMS6Aka1LrKZjDJG+98AiyXvmnP9Yj1y59c3A6W0x4YjB0XrQHrQ/RKf9wyEPMw39NBC3ICT+AtRXx0CWWWDkrYOOqJ09k72TZNspZ5IZ9zMlzvVTA/Vcizm8YcpNo2YNSIOnfAaRG4KW+7sFh972z27u1dwdExGbejqYtEgScoa9TIaO0DeWBhDeAgSXOcNTuINgHHe6Qq3G6gYmXJyv0CtED5wJSdJFfsA2zMd9MPwd1qgQmyXTtw5ES7dJHqSp1UA+yWUPRiA+sq9SCyYNKEYDRTtgziCLxJqYuRA6BkCHXoXBeOpugE1vzxl6lvVxN4SFy4JgJgr4qi6TbcbrAYTLsxuQSgYlVRNeFRRVqs7JAVpCCwp1mDQkzLQmkHNO88AIvIPQG7GZYiUF/2+H0vw4SMHjx09MTGRfuWFF5ECYQO0+ZwOVMoEIUiBZAgq+0GSrtRihnVAS2QWyEL07E+gDaEQsiSNGD9Jta5tprU2qymtrLKP9Qi/R1XBIf/u/uAv/uxP8KLc2dokOBdOCDKDIZmmjhw5sn5rFWLm9/hBXswD7BIEg/ciIII+mAfDDEg6xLkHNkMEmO2D5pM9KI8GVgZ2VSYbVAXECrSZZFge1APK4T6u16FJdfAAO4V9jYsVbgQoohHepZOQxgXummgWHcwDbAXQLCUXmoqBclvSPcZI8QkcT1zbnkxmh+khz0m5WCLAg6FD5I03n+wnzJV0BqaCglaan40kZF1G5QNalDm8WmUaA5SnC4aLxSrzxgHB40XWbDOlzC0lNEjUR0Q1a/crH/klSEq7ATIV6wNW1w7SntVLtP6knUHSp+I6RJcCQYR7OX3Yx2v1FokO2pW+J8Ir0Bm5yf6J7uH0Pfc4At7v+LZv3c1s3Hf27tfPv/xHf/ifty9uxfclsOU7IyhgRpNzs0sH51994dUPfNO35wvZcq7y8IMP/cZHf/ONC1cXD6RnE9Mra7v4cKRTnoXlWX+IVEgYx5pMzn0P3D8zNf2xP/1zgAlHvDz5ErK21Bz6dupyhaLREXaBfDZHdq1mtUkO1Fa12cckMR1lxXFISCdiO5THK2XYTdHoJIBBjG+nW7Z5G9uZtbWb1xNUO4+dDqaSG1urW1s3qqWdXo9MRyrmsbpSd2VsE3ON1CwZmmKeQHJ2biIS3XdoYX64vd1u7GwXc55BmQz6rWG70evuFMqu6GSmY3/94i2/u4I/Z7EwuvdseDo4ER0W8PZlav/JBxBwG62xLzg3T1ooeQ856g5tLD5Ywb17QNbc3zG2MupBjduxyRAld/G1J66WPGqgVGqjAMKqQTqss6lGzrgjmDh4AhP7yWxht5zrERyM+wVpT/OlMqIqhqCpmUkqZNy8fBPLLg4c0Fq6k8nm2IBIFqq17PQgZ17eRb1xAyjC9ECvkFwIJwkFVTSBfQq/hccGqQUOTk7DgEI6cPBEdsUfx1XOSpMDmOP6EXUHQ1h1eNDbIfmAmEGicUl3QqwvYV3oiYlsUB1s5BEyvlIK0U06INIC2APuwuaNQn4XnTOqQ4oPRwNeBU41a+wL6J54bQvQ2XmCe7sn5C/V8sT+h1KewBR7tZetbZRqW9PHwkvHJ2eOzdsiuJjfqle2nYGOfxI/txrWP1AjJkiQCAKIPBWFw0gB6mIyIIK4CrHxtd4sF4PibpIY4B/XBTWjE8eHGPwn1yaWigXU9jPUV3ublmENdFESnJaeLSqhxFBdCwj4osNQZQMR3Kz7zWU2PIcYWnOYiyLkXDSfFiDtfd27DqtgDhqxTgyXr3PEODW3h3osAo+xlBzYEAxIEq/HiGlxBPQbMcZtPJKgCnhF45NPBpx2odS9sUHMPYYeWyAa8oQR2IReyEdbIK9/tdDu+yKJ6WKp8tRzLxw5fqBel+tvKhbDfdVBsAhaDyd5wbCvU36HcG9iT2EMIsnEYnjgp4xaDC67vTEiBrzH7EmFKwZHiljWgvrQGPUh3m7wucwmdieN0ybZ3NhB4FaXI6CRAlrkqPVzAQ2FrKJAi/TENIq/pzH9GhQpXwGygvbaEC+44x64FrSFkaBcadWbXaLMAUbboE7xMAQr6m0DVbTRaLNiLcWQ+N1z0wnSoTU6/XqrTZsRMn7Lv0yYWioSZDJkR2wYuO8pS2K/Va/BnqIvQLtDbe1QQJ5WluYTIguWnJ6dhvpOxaYhw9rSGjLyEcp7YrmkMYdIcIX1hXeWMxsH8qbSF0tmFXMuGV00QGyhU6I/V1hPGCwkKHVNahtyfDmJ+ZlIJQhqgQ+Dg4YI3XvPPb12K7O9Q6+QfeutJjkjuS52BBOMOQBBBsgNbGfkcJgzxQBgr1HeIAPr8KD0BAURLn6DXoOEmd0uK4+pGV0QHoqpCT+zI+VKv7+9UQ0FbEtLE2jjGR0MEY3zOOvIKzhQzIpf0Fxq+PzEHZxwnVEjJZA+Fg7mrrvugp94/rlnMWsBykAy93AD7fAIk2btB+sKF6014sYUBGObsuo2hHKGLG3wm6y5xsOAwZoh04QdNICJwZmIxWjz0htr6ckA3tSRiJf50bugv9r7RpxiF7Fz4Mgg8ohLaFFgCxQmaw/6JHHc/ZbTp+++69N//alisTQxPUEEGhbyibn5Bx84u7FJsfddEofdffZeqrHmi4Xo7GR1Mxudm8nlsxiIoskAKc1eeeWlw4ePMlvEpHW6dRSbbKypGX9v1CWSbXJ6otFuYNIGnyEkubyBeqN58q6TVKhcPrjv8s3LF668npxKwiBkM4V9B5aOHD5IZBocb9AdXJiZJ7Dzya99oYdRn1BdXJtH7RdefubqtUvT0yHYxHwx1/YOHbHQjerLrLQv0P/O7/gW8HKl3Tt8+GA2u/zGG+euX79gvNN82UJ7TOkhcOa4ObmYpkOlcmFkC0ddwa2tlXK1HB+15/Hf8ZELwj8Yeit9ezZPKi9Pb+HAgUOnwQebX/naF15cmXbafviRBPP8Tz/2SCy8HohUy6NFlDwlGLiDigEHMLXcOCwYSN+9AAEAAElEQVSrs8RfUCBKZgNLoM1Wj+3TDQ09pDUH5FlFNgHMNbx3D63UgCSUcbaUrX/T4Z86cmrqzM1Df/2pp9PJ/eVim5ggTFiEEhGpjhcEoX94L8/OJlkvuC6oD/CNBBgIxzF91ts9DBwra5vVcpkAAUiOAX5qREiKwC0rXypmMoXZWTyrc8AMzhPs+mZdMir/XBhqsViASsCbmUyOvRlCExEOAUyUJxsqzyrDAaZBqvKLGvTrwYA3moqpXIGNzPS1zM5Os4kFIQPWiQRI/k+SLJQfNZyY5EeljJ7ysrBomyYSzZVzXGpVHDHiqwbF0WYut24Pt488svC2ex+zJdq2UX7Qv0Cgot3XDcSGdi9F0xEIuvLQV54QeqNKzSAQNy664DToklrXGsHF41qNq08HtItGAmUYShqYdGKXkaHQWqCFMzZdUN7e9pbLBstME6yiTs1+14qKzOmLhQ1E1s2hnc2E8Dg/mU/zkPntzs3mQasxGt274faJdWWvWXXjTmfUEV1n8TRn5o1i8nRV4jqGc1AVzI183UyzCIrob+FAoH+OkbvVcdQwfDXxc0GswWTfzFeafXLUee2UrKYAl7tLfpJhOBZp4iBtdxKcGhgjZg3IzYYvfZLMRi4wC2qPZhknSZS6jgC/djstUs2Mh7hlRBLxabs7HqiPwyGvMxFoZDw7zc1atYy/F2QYWutSMlR55NJB/qMEJq62LDxxjy34Jht6D8ZIVjJTVgsvA6fT74kAXFAIgBInwFoND/8SkhyTQHpmBYZJ/NJWxJBK4YBmy1arIvD0gmFUkXjN8Cpyw9Y7jXHQ1UQNFwVp+gPMGGWRKD5RwUGl1ivlSvA8KhiE47GU411KdhJ/gBMD6izC8hA3JX4xBhwIPFSERRQmEWnvwuvnb964Af1grlkd4WgoqYf8U9CRBK5Y5NTOgoM3tkhMgQIz4DESMmKcwRtQP1YHDTtoHc075UuVYxr+DxSj1dY8QR9h+aC/tI0Yhv6WWYKDoN4WPxIIQfQRziFQLLgUBHQyYUJULlLIgYQhhrxB+SzixwJgpQZ8mViaZ1LpktmA0kaJFmtHcxj5zyA7ESQyV8PEOcYEKx8+cTSeTtfrtSce/wJ641dfvIaX0MF9c0VHDmKF5IocDYzCpKKoQCjUghpgNVWokOsFuOxMeA+mkm7Aw/mDZJ1ws64o8ZLpFHYyXsdP2hS32Vba4eAxftKJ+artoA4zQ8NKoZEgZNJJnkFUfb2HH364Xq2fO/cyEgbjYRQc2jqgW6ZXn7i4OUgQtr25e+zofJnX11uRKGO2uG0heYwFCAjGzqwX0gK9IkEvryDWmyswXqx1LBKKxyLhkIRvOMFcLoMb2LPnnied0Fve9tin//ZT2fzO1Mz0//Jj/+LG6trRo8f/+5//2Y2vvhQ7OQMLloykVq/fuu/M2W/9tm/7q0/81a1b6wcOzmMvD4Z99z505jN/86mNnW0y7KO0wlMvEo9NzS4dP31P0odvJFwrHCN/e5R6JI57yj+JgrqYL9sO2FLxVKNcTwTjjr7tyNzhGxMXK+XS7MIEkHz95rULFy+gTKk1O+deeuXB+856fO1KZ7NU2Y34EqGUszUuX3j90pe/8jz+i55Ao9nONdollCvY+smOB5dZymE9AVFQBqObza36gvWpcJKItX2x0OL+xZmEf9wqA1QJX9g9IjuHv5VpjuyhVXan1xk/eKr2yuveADlW95zpWMT/4QGivY1JhUfFlQJcoEGDL/WTQcWcsCgG08pGKqcZs5rAutYVfke32pq5PgSKGDDUZbgKUNITsw7+SHC5RsWCH0sOvShEwe73PvjYqYv4UN0sozRhttmslVoF1s255aDmh99P1KxKh4EWZ+GVFha83kBqcpaE5dlihcCIUKgMl0OdQCwQeGR3641KHYeRBniQgga4cQFRwFWxmL9+w47BHVLLJgV7uIhJkgnWgcMx+dI7eFo1uy1vs0oxB1AhogYoQOoyaqRQ1sPrnEokUH2TuJrJwE5SyhPdt1kpFScnkkQ3QBMoD4OXIkpG5bkJ+rtAsDyh2EyiImwMlNGEa7qjtt3GRrG26Z7oLd0f2XfX/uBiwBZttvMXfaGxi0ykMK+9anPYkNMt0gO8DeyotGJSHSKK+9BfEzalJegYuitsCXJD6MFFFmOADuPTpBMiLdlmEGEkSxbWiJ7a+UJ+ZrnMopql1M4U/uALSF8P37lH/eebuWZd5MLeVy5qgLd/te658+zt2yx0++YWdMWamTuP0D1jMRSm5ACJcYOAURhPvr7i/JgT2bjpI0TBX693UKs6HAFy7+XzjUZ7FItPLkwtPnTkBLGbV165Neygo5aPPhPgCXg2dmvuEGb4MRIJQYDhSDSTqS0vHiusndvN1BJRxCwnRNfp89aqnUYLuomHD2SQ+fNgPEWORn8MqW3lt5tQnB7omAwu/IruBS8ibL2YLEknRn67HkZkCDDw51NpJsWns4x8Jc8j4prsCRiBWTg5WknwVXEPHxnTYhBIwJcNADqwVPGAkc8L6YkiE168eH0ngwgEl4hLC8s1KBfHx5bxSEyZfAUBqosQQYRESgEvHPYuXLz28msXdnMUBaGG4Ih+oj6qommEoFJUDAFZUQgm6SbFvsmeQVgjSmxcl6PhKiVvC3l/MGQcwWRqRc3KmmKPBe9TbBUHJVEbUzqJD1aNAdJtEqlzA+sI1yRYMY6+Go6R+SA8LCILCckwq4+hSAgMcgJxZUbE/bK8Tie1AVhqamDzLFOL+pEKpNcuX9nZ3sYYzCuwjy8vL1MTolKtsqYipUDMbcKmvllMHnp+biUSgFfrrcJtcNe8DlaZZDyegH9mltq1k9jzt3a2//1P/zR232bzP117bUtRyAJEe7U6SMbllarOS7eBLAK900JqaOairsPVMF4Tp0yUo/pgRH+UDdeuXeMc3T7V0zi5s2u4H05aXTKN8wij0A3moL8MazIahQDjSsqUgqZ4DX4mluQNS2me1baxnsA9mFxGlXIl4PesrW0iXGgA0v1DmM12NvdxhcNwK1ooGBOAGTmC0HkU2z4/qnjn15588vXXz4Mko5EQZjpSmTHzp04ey+5sYncDtyIaYuy6dPOKi1oULsef/Pmf/8qv/MoTH/tLVHKjXtvpGryUf570F41q4+Txo2AkCmnAlD76zscefuyxv/jYX5arzUYHHUaenHRwCYFwbHFuGktq7sZO/nJ5O5ut1IjnLyQnJ+D6cG9dXVlD5+zHKJlwVpulqcMTMxMLL17eWdwfRfV9a2Pd5XOfOnN3p119+ZWLweilu06eoO5hyBu6tbMaDiae/cQXzj1zfjq1r1YvOf21aNIZSwZy2Xo22xp2sZL4qqU+3GZzxh6Ex4JCyCjifvRt70i4XZGwvT6sdfotis57vKH2OOAMTcUiTkdoMr9bnYom7jl19mo4PdPdsY9XzdT+T3wASbyPd4rEGhkIuLHkK6CDaxyCNt1nXKOVERNlDWAiXgq9DggeT88GoWCOXjAydIZxaYQ0A5JUv+ij67W5RyRDGA47hB4Bzv3mdmLpnm9+/3t+7X//b9MTMzvbeQkCJJIu41U3IMQA0Q28RK/g+YS4TC3IcCTYbPbZqg6Hb3a2R3EkkBGvCARJxwHL50WQYHvjUgA2g/Fjz8LTG1uJHZEV4Ce5B/pk0j9h8iOZpV0acG8Y9A4MZktZJgB9TMDtJ3AIfycCErAL2oadcTm/u7GSy2ea7ToagEDAu7ww3WnUETQZJYmtoqEgO19JvCoVUmQD4Yyf+WH/wJHBvnec7Y3VlZmjqYfO3J0+7LbNd22xWm94s1zGtztAPehmqYGYovxZ3mC7AzdBvnGtBdSWzUpqHaCBBhV3Rn4OqC6mRSPs7lUHgps3siNTxoIJfsQmgcQ5p4/8p17tbezbsGGuiPRyGztYS6xtbBFcMwpDXDUcfr2NFO583TPWyu9Lm1/3KLaFt+gF5oJ1wk+6pJ9MO1//zXpQb9QSgLeR/HinwE+vQ/PME3i1iIEjHR+X0dqasBwCcpAUB5QhJQUC7uNNgjPBip7O+c8/eePGdpsJdIVRSOA00yo3hmWbP2Ij4sXvSvTHgV4dVaEnl6991wfe++KgmN24EKeUMwn6MSUSaV8pgF69njD2W1/AiZ0V/O4NULonSUaW9Zsb434Ngup2hcDDBDGS5wURLUJtrHAIiaVe68Dt4GjPga/rvgOH0YNTPw8xgvWTnIaHIu4tUHOnG10fhXcJh8HbAGsImjSWGxUw6BiSDzADPSixcW1wu3zTU/OdwWhtbefcC69WG7ZQxJWKD6NBZ8Qz9hL4TFahUafToOonyJO8x+N03PfOtzy4m69cuHT9xq0GCsZQvJegAoE8x0TqMFPTDVlLKK+pkD6S2tr78h5QSUpKDyH/NBo4WqnDLBAHNIlPbDwZZ0YaoyHVsgnuQxUB/MsBijtZOz5FI41Uh6jLoNjGojGGaFnuV1zkLo8f2QvaasgS9MbK1+EgFQA8B+zWmNkQgur1EVYIDIWzQf9BRotIIEhVFuQnukfYA3I+yMJUszJRTvQBcDJUDaMmfWBDSjnOOxEukY/psqZB2fjW1m9lyplKo5ldrbz+ygUxNKHo8uEUin4GyL6GCUfsBhrFBkqEVQiZ+AN0fao9LiUyH2BChgzAAA/RhFG540qACtzpUOCW3ZZIxDrkDDI7QZvB0F2pho0hWH9NhzlhDjlAxIuL0zdv7iLS4LDKK5544olysYxaT4PVltGTYiY4McwHzDq86exsGgqNM12r3pqdSuzslPz4booP539YIfk3iu6TutL4gqEOJK8nanTs+UAvjTPMdCKeL5ebJPsJdWfnJ4BnKiXs7GTwUihXC5lKjqwNFJR7+txzieRkKJF66rlnv/f7vu+1V1/qtmrAceZadeJA8I//6x/vO7SfWdrJbMfSMfLPP/6lJygqVWsOyo082Y5ID7mwvLR84Oj6VvbSxSuDRq7fJUYYSZLYTvjn0dbWzs2b1x9+6AyLuLu5tX9+CakJHxgUSO1qt5Brlmt9/CqI5yeL9czCvNe3j2gQUt1t5TdnJxMuH+UGO7Vue3XnKmrF3eyacjcMiq2RzRe0+4LoSELOYLSKE5MDi8wgv0v0Qy+cDKB7g1+PTi4X8plLK2tE0yD5xqAZBK+4AtsbmdrY18o2eq1BEsWmzZEKeCbcAQft/s8c2hJvOrTqQvkWyoQNMHtJXKNAAiWkchIIsKENlAtUXiABHvQYYwuL2e5Vi/WQp++Jq3ShyWxEAJgoqSCfgEQ7OrYyQ7cNGyfuveuxt76FTG648EN0VZDWxK2BqTBMAEgRPGjkl4opveXt+Nh9gEOnB38WgSRj3MUZEz4sHktHwkEMwzjQwMf3WsJiZKcwYDmEYE/OTENYb9y8xS5xtbsVGBtipxAtB706YMwOQYs8OzUtdxL86RlhMV/I5Qe91qjdfvaJL+DTKXOW0xlAKYkZlmIT7Tq5nPHEYOiAPRKSfGbQXhL+AQKDlYQY2FF9tjpYr9CuOxr7Hp4+ct+i61TS5sWMsdJpZO3Bejg+anZznqATCQykj48nwSkYIVMpr8LmoDzYhiCnYA9Qfh97oKoQgB67LdXoNRPMrpXuQtoI1gOswPIZnbMoK6Lt3pbWmt457iBKfNCEj8yuFODcBh7rBi4YjHrnOb4yYH0VtrHIpPkR4LjzyNfvftPZ3q+WII5kbxrW3BkZl8YMMkB80A/CEuo8LzNoEzs4hgibV4HZIndchl7ESqV2voRTFNTXSXbCjc3N5nDTo1panol0yjt2dypl0ImCz1zQnnCh2rR7iTpIhULuaqP97LmX3vnofQ+/59uf/ngRLS9CdH63Eg25W81BPB5WYk6iogN2iFqxtuP09WPx4ZC6yX4v5UxJ9gtd6nSJzmygh4A9SEaS7gh15lodDMT4eCp3YIe0TYRu4lmAZQNJCc9SH1FG7IzxuNpokHAgmU7g1dfBWSq/U62i2+niCAaWxJ2Q+mBMTq1GMsQaRkTejGtMNJG+7767Tt51Cj18vlh6/eXnA4NqJABXLpkMj2B0MBBzCD9aKZR4gML89NKZkwdz+dJupkC1ve1SPVdpVKo1VO4O0vKaqYaRxLEQaZwMGGj2sa3iGqGmIISROOn/LP5ccTi9PkHClBWC7QIQUfxA5+BXUcwqOs8TQYFoWD4AAgUpEVMSnNEVgfNQT4uRZCMZhoylBeDEc6B3gVEi0kzqAOJqRVfAE5iMAv5IEvdLl2d7N0OwIqyyjCaEujqc1A9lWq5evZovFI4ePbpr28GNlrx3kCWIJD1HL4z22+vElV0aaS5BJJkcEDeEF/jiG6GQOH1g2UsnU/fffz/mCVjev/q//wI1PZQJkIj4fY0a6bpcLSyv4mBFFOWPCA2DP6LnumZgVTgS674S8ni8o1qtSvEodcYhaZXZAB4oFdW1dpTwrs74n8FymzRVAngd7BsmiQf5H51IPO5B8FVcsj/Ar/zESOFnuE17XQfqPcIHeX6IUDk7O7G2lpucJH5qRKBONltCzcBeM/4isiVxbnrBECho2gLvgc7bnRYNwguB2dF2w8olU1FmY3ISdW2rkCnCVUA4EQpZmaNHD//cd/3Cb/7Ob924vjK3tPzzP/8LyfTUJz/9KVKGESxeyG20u2P/lKvabCcn0sQ/YgLklah2wggrTu89d937hb97igEEfZ5mvbe9ncXAWCWrQbcZsvUJa8HnuYF3YdA3v29hciZx8dL5i5evBu69q95uU0cHddj01GzSMRmgcEitlc1XJibkkdcedFa31u66+9S3fOBbr117vVHJoXu4tX5rfmpu4+ZWprjd6o9OHDq0OD+drdy8cPlquS6PSNzmGq2yjQI9JLEMMN8D38AJUUErgl90qdGs9UcwIWNvoud1oIUI9lyU2yONZaXV2C3XFqbmKPOxtr4+auRTfqIfiHUgPbS841h0g1mll2RJgXYc/ziTrQ/IkU0N5G5kjjejSkOBBSAsLIZHcKFZZR7hAqfcq/UDc8JksosElhAPquQhttrqxS4Bq0l/FJMS3kCEUzZbVWhRAH86CcQ1WBuSrFOcIhBLfc8Pvvd3f+cv9h+cyWwXVUtiONre3GQG5ucXG4qDUKlvmZ+Qe3AQaaOTrpJmgkQLwXDi6PGTgOILLz2fTMSghiUsoVT+cDuDhC3GInb7AkXVMoVSMBq/Z2nx0KGDvY4cwVxHji8qqTNQKD8l1Z6Bm8Vxbuv6eR5D5bWzSzrSFS7t27+UPLDfPq5CEdBBk5oZPQ3bGDQE4yEKSdYL5fHCepIgAVh/gLkw3nf0q91S31a2hZt1+07NnosvBo4em5t/4LhtNmzztm2lIo5qaIaoeweKIc9IV3w7mxkna5/UW0wuvpx4t0CoRH2NfIqjqyovjagiah1KHGZRVREr7FnCCuJrecBgVa2TVtH6trfntHwGj4j4cWZa0DUDKWZbanVhqTjANWrOPGr9BAyAMPmJE91nWSl1rzlE+GlIzJZ5lFvUtq6oUdoQBO2dcg9Ih+/Kb8xVfnKhNSarJCMFrsCL5L4g9Fopn9Bndd0k+6vjX0bMnTfURCpzpprtzPLykWq9u/Xs835/sl2h9nadLdWzVcnbFI0Ew/6UnJqp4zcahwKhkZuMcq6F5f0TM/O7mfxHf/v33nlqYUCjbSEdZ8/fbw6pNTls2RrNejwZgE2nsEUk6Bt1G95udWr50E7l1mYpC1tXrhO4UQbxLcwtb21vXnp9pbmsmkRDpYIjxVYn6HHNTc/gb8TWsHvCJOdutoBtXJYgwLilTAGCWJ2R/RCV8ezAkIzPg0JV8acdIdBC2gFDQk5RgLtRekRC3n6vjkem2xea37c0vXTw0HJ059J5F5wCkcEkGyEKQdpgaE8Pn6bJpJd8Ye12ZiKaPHb2QK+7tL6daXTHzd7o8vWVC5dXap32bmE0vTRZqDWxtKIhl+hrHwW9YUnw43Y0mbK7/Z4AcJ0QEwAjQ+atwdhvc6Jukq8BPv9C/1JOIGaywcQDWuCB5QAcIV6NawIjTjgEKEyK8Twi+ymglkjGsFXjmVGtZoENaAzvdtmo/eJsd+pKDGVzBEMeAgcwE0CMVdzT4chhncvsoOOanZxghakx1yQtoSQE2J82DAE/yUKgjYTHnug22jQkcaErZHDcFglu9nmpCFxuVFauXJtIxN/x6KMU/XvX29/19JNfI99HrVjBxpOaCOVzGLdQ8oFwpPOTZ4VDOcIwGcDTi59Uznf+0xiRd+G0MNPjXo4YAfVHoQFD4wsGO6AzlV3RFMENmMHC+YgXYOromDGzaNux51h85YGjFg3651ze5/FGwmHUeuTUQQ+BizUiKY0jhRuiqHthZNGpF3MV4mvw1kXdjQUBjT5TSimnZMoPr07Qsh7A/CCPslG9NQo68KBBfFDoNgNBDnYRD+ZwNGod9E6dWk/unrimoItqDR1hTGS2jY2dbqt/YPnYxso2WSRff/HCt3xg+b677/mxH/3QL/ziT2/urswvIc+4btxa2SmXvK06G5scC2eO38tnaSOTfiTyzQ8/9sRTTzWzPUfQTikksppMzk+fPvbgM3/92Uy91CACSH4v7Vq9khhFlg4uF8v5v33iqf0L0yvZ3btO3IUv+Iptu0S1eh8SrmMrly+3a5Dq9FR0REx+r4YzUbOKwslOydpSsX3l8gpQRF3naMqXmI53XVMzc7V8KV/CnwpICyjXSTjsztX6Ea/t1MH7pmdnidSPpqL54i7qK/yzdnfLYUxB0Wi221j2U34gHyJjQPnGi+effa1n+673fvP80YXttQueAPEStiBL0RkEKZ/ntTVFQpFyMGRSLLgtWc0g9a6dPBjAJZFNsp4ADEYO0b3636BZg+KFMZF41Yr4RqFw5YISpYACcw3BRLWREOy1D6EeFVtxXI1PB1xT8c6g7gx77RhBW2ASHFKJYCRDAMWjEGZei86fuOuB+MUXc/FExD9OdVr9aqBzc6U4OzeIJ9KVciFO6e5a7eKF1xaWDqCsyOVqEUobwZAqmwJStisYS167ddOnbJE9n2s0O0WinhAez9y2tr1N8bHdYvn1Cxfa9cLxg/PE57hefvGr9B3EhlgNuy11DKqXfrtWKcOwc4UREM6HyHzzanHt2nnc+jGqwWuIPhgQh7fG0bleryzMz7BNMC8PetVYfIoolM3iRt/Zi8/6qRh4I/dSYKF7/zedmjyaqg9KzeFrwRYyDUxGtu8oO0Yt14AE1k1qvKEbZPtBNuBOha5AIdrEcEtKZ0K9dfqnGxT9hACs3f4PDyMZaG1YItbSnJiVNGdcsR65/VffzLl1I+e6hX/Wuls3/9M/39zsP3zq9tuFfHkD7wG/cM53uAjmhHNZDLVJCaOiLAJYXFIyQgG2JRQB2DhQ/JB7rT1w213BsTsiHq03zObHO7vVag3my8diNhs9oVeB5YDYEtpy2Kk9GsWaUK420niUTs+HYyl24ez87MlTd737sbO7559e2VppV2t+D7SPED4iZjuwkxF/xE6DzSpuVuMIhuDIkPD7SjEUcFOCGsvIBJniIxEyJMMrUAukmMuWchWWCjc9YwqUdqRaKiJ7hVPo6iIovb0eKAeZawIoUbHBdlo1HPNJctRsltHYkMUbxOcNuFvtGlZe9qYfOyJ5Wjr4ITW0d+DTmCowM3JOvUToXate7A+agk1gRlAjSiMAcni69AlzMZp6tOSNYW6riWoa7nwmkaJOdSp6Ymlu+trazmtXVknYh7fa2Ocihw3qBWzqI9IPIwwpOsCdqVR/8Rc//MmPfZx6fDPTU4Dv9c01vGnwHYaNhWuUzxGMlPaRkeRwVjIHC8HBqdwZQS7sOhbd0p/oBmEY/sVi8Ua9ieiJqy1yHq6YMJPYHTvo0GW05XGd8IARBcgYClDQjLhCJgI+gfaR9TgHJWFH50WYphA3uQNLErNi3iYY29sVXDBxdNiPU6l0NBkh+Rc1FZ4/d+7ypUu5ncJ73vO+48ePE/ECQaIf5XIDbT/MNtQX3xIp1pgeOoG23A7z1MEdntbpng4UaHLRgs0SWWbMeNTJsc04MmDIB7VYjg4MCui2pkg9fNOhZszBr5xbem/OrVHTK1TQqN4l8TcagCK4gV/pHqoPw1jrYbWBDUdoW/sOOzJZD7ikIA6kComkttQEfGAQ1SI6PfgV1BBobqQtEJ+hRszcWWeQaNhoASDdQM0e8IVeeenVicn5VDRNSgdVC+71//SP/z90LxqPo6U6dviIPxo9/9Q59uTkdLqQ3X7xuedPHz8GffrPv/sHlL0kezKJCMmc5Qh7SGGVSFFxY2Jhfnnzxg1yoqFE7JIsq9m/TjrMkOvEiWOHDmHI7GbLtedfPf/I/QGyJJ2/emlylkJHC4cOHrx58xrhTPFIuFjaaTaK/TZCDO4jTgprtqkRW0bCh3MbXLxyeX2TJDNNEm/Nze4fjlZJEcHOnpyNknnmodMHp2ZnmNitnSyBzpFwvFVpHTh0EIXU1etrN26ubmY2yS786sULAe+IHMVEKyawUtptG9u3SLJmb9VmJv3Z3cqUzTGXnqjmcu2BLToVW1nHJE9UBRnK2CpwZVhUpM2UUgbmSzNt1osJNytuLpiLLICBhTvXQZdgQwCOBTerY90LXYHZBQzkGi09ImIAolp9MA7SvH4l9QCwyWuwvRAiBxnyeFLd/rX3fct9WyufJ4YXH65oiGwnV3BTAQrKZWCkBg9L+knYPrjrWg2CJRaT/ZUOp3AIQCXNpkddMZeMxCLRmVRkajKNjNbsDtyB4JGjJ8l0gY/wzuaNmB8kWUlEA64XX3qKJugHDKY0WqBruIsh7w4hC+PLiT8mTjHY5BiZ/FThIbGpadAMy2SeHFHm3RaPTRYKeOt1ZJ/zeTL1NRIaEC/adxZe2X7dGWo9/P7D848t2KJwAyt26aIJsgLGketaTldzjFHf0WBPgYJxSGQzgDyZUrhdfFLBo5Q0QwcHwIF/yRAp46ghvWZf31mgr5+wQmat3rQo+vHOXt5b4DuXwIR6wlpe0WzdwD9zUU/+o4d5i36h3Ted79379ZeZC8KbRva1eALrJpAn77Fa0As5ldAjzCyveZgRFITUuCTHlYQZW4tgXjJDoRIcjxo98qQ7CRLDfRUHvEA0bLf7L16+ns8XpNOx2ePxJAYMtJpYR2gLQEFE44QDE4PLFwT4irlMbX0TGeLuM/csL0xN3HNXYe1CsbBrJ2QSz13MEZ6oy03wD2Z2cq9TPU2dw6mkSrKOG4Pd4jZOyG2cQfz+arlGJ2vlCqUckYkJKIePxjOZ+8n7AwoGP6Ka8EBylfubgtdSe5KdCRmvWmqyE9lDqPRw9Qc20F7ixydqhReIE6aYKDLES1xuccqzwdHXMXSjn3WMcBbL7WzDfrbrlUauEKSiZyDsRaGqmBt8msClnq57BGkHhMkugA6wUirwO1iv2anXWwUC0qcmY2ie8Xt89eJ1rK3kQSF/OJ1DzoYgIe/iQLuyUpleOPB3n/309uat5cUZWsPpFy0RjAX0lkUUpkZCx6eMbQ+pp5HbGERkA70oClu0z6AcBD5+1WoLcji4wj+8vbgBNpRYCNaLDuVyBX+A8uJy5oIY6NOQbTCO+nj74O1Wyyj7qYaGIZk2AwHjzMW2URFxEX5mXHwrKEvgKARnwSF0lCsk2NrJbGnn+ZThDrfM6bnZNy5fAgWTeJqbWS+Pj0QfMqOBN+CcJDyLsgOp2pCIknwKoDUyvcM8JVMxB+DD1yEZf1g3OAuFKOopc6se4tc7h3XdusindUAReS/3MG6BE4yR189Xq31+QgjWa40ikmiL2/yGGgev6VN03ka4uvVqDRaBFkUkRVGx5B04hL9McwAAq84xb2FENAvHz7PWjFk9pB2u1HZbgRlsAeO/+fRnADAVZOx0X3nxpfe94x3BRHRp/+LG9gpc3Le9/59hO1DyzmQS/pGclJFgPOR3rq2QxaUei4XyhSqKBPADgmYg4kXOwfgyNT1x8vSJG1cuU3GOSg0izq4gWSPwh7h2cc2PImrIPnGtXr9cL3fIB37txurC/ulKJeN2LHvc+MYM/EFHb7cpS1WrizcPs1Ip1vLbxVqV6Dpbm+04bnXbtg4+YuTOI60zmjS7igpQr2lRoasxhH63vd+sdlavb3WmcHEdnHvuRRQlEzNziEcbm9tnzpx4+zse+fzffrxRzVFbF9zULJK3rbc0F0pGjwRln6nag6mL61sRj4t4gctr2zNL+1FLYAixdVGdsiRCswhZ0v9ZykYDCdYMM9vWCTdymK93TvZ0lnznHtqwpBduMgoQQB6apmBUGM9mHabK5ibeR+2xsvxgSDfbwd7D+NLtZ1HkMEkPPnzm7z7+ysLC3dHwVLF2AFUzOdRmptP4vmEXA4VOTk4Ch0y4l3hbn2pAhIN+YjJJHDuVnngV/ww3MZykwwx12v0CeU5qreN33YvvChdLeSakFJ1JsOHxsnSVq7tCguLa5SEMFOIkget4tZ7DxOLBixofgFqRTYvDKpU3JJjiruFE/ERfCsTARPhGdne91sJ/kvkrtWuqZxfoNrvFfLvQdKzvezB5z6On7SdSNl9x1Firj0tDD2HHETgU/MFJSOdG60y2YvAVSvl2myRKwpz0Q8UesMXBtlI1WNEdqBDFOhtXLKEsrYcW4x89WCp+18KYm6wTw7Pu3W52kM45uXNufuPef3hx7ynrzz98r54xTVk33GnQXKcb6qh1z+0bdD8HF83rpFfhNtFOusxc8NNtMgw8IXlyK/kQvT7ySLjayHoYeZCGu05Hy/Z6+6I3jB98pEh4da7I0jYbbW/AjysJuBRPScQB1Lao6Vh+MPvu9g6a1BnS6k/PxePU3UsECNTqNArb62haUH4gZw+7JFFBFaeEUZgboC+sCrkzqG5ILkd58WGGt4+IbSPnN0QBIgHOYraRKhBy6tUa4i/0AONjjKwEuEUTPwTNJ7xWRV0hVTKhYTsGk4JHYLZM2lUXEcjYOIBGcnu0hpQgjVIyk9SYnQY1dlDsOEl7W6+Xq9W6creNKeKGxwzgQ4RfI4KrBT1QZ6GP5PtAJwz/OMSbFEop0XAEjAHE5E8T3OOyD+FLT8TTE9OTU42JyeTq6tV6TU9TSYxgtkIWObtOlvxkyjY1NwfDs7N+7Z67jjzzzHNMJlSq222FgoleU6+V2swcksFhXYwHIqoMUL68BIh51k7DxRTVEkBsXEdg4XG/ZcHlyylFcbvZcLj91WIVaVgh/CqUBuUQtEAJaN68iK8Cb2YKppjJZxQyfBp4N/gFWELx32EW4B5gr42SWyFrdNIgH3g/3S0QJCkHE+smRnbchjiOBj6710+cgVtUeXt7i4ULTEx4CKyodrAWgZdbtQbYhEeBDVYGyZiWoGSo44ABY1Y1rAFEz9h9mSvWGoM3n6r+gnoQvEN9eVzrhVS1O/ikTT4ZpuEx1LW9HupUB2+hXSnAMFcR5DvA654VJDk+3+QcpzwoCngTt8FtBD2b59SOOmQYAl6Ad0nAkFgMutRexfOc98Kt0hTP8ghf1UnFIxERDhf89d3KT7oB3GIfzywld/LF1dJKrUpOLko8yVl1fmmR+N1v/84PTM1M/MEf/v6X/vsnvvz4E3/1if+7XCyef+ENjztUKeaCCBUoD3AXpWFMzs0OabIbQFwT2/Io2lRRpkajPr9vMTWdYnWilFHHqYbdN2iUtmt4d4e8seWF+Xg8uhZcw/EgOTnz9sWFncxqpZh56eWnC4U8nDdRb+Ggh6B28hUyBBT12JgrZKpu2notMoGT865L0BMaikq5McLpoodTSCidSJPIE/47s51pdfqzM/PLC0soZi5fuoU7MNxYIkV+kwCZnog5Bmuff+1iq4v2fjiZ8jKBHj+Q7I7gkBb2fvELz064iP23T+07Wslky9l2bPb4aqGM8m3YyjWaChkKeuHbsKTgfQlUAMJ7YMk8W9NuJvzr+PPr12EmgXyzMph92SJ7DBeEGADii5QUohdDmAzHuOdsp2bj0iMqPAD2WIyoIfnET3oa9VwyPlMtb9x9/5lnv3Rh7EAyHJw9e4YoildfPIdibyKZKmNtHfbz+SKOoICH1BvkCw3i9Ozpt1vpeGx7GzOHOUCn1GoiH0CPEAN3tVIBD1E7hbqlVCLL5kYL02ncd1gisnAgaLGTFSOoSEyhfzsJG3qD+qDeIBcD0OklnBnkQdIrYFiCsjY/OwWJBbUR4+gglXrRE1Jky1az4wy0PnZXwjPDM/ek9j0wZ1ty22rni7kVV2QcTPvQzSOLyz1SNZdQv7MjyTsDM2qLhCIwtWJkYPcZCCIvph0Eezmi6p8UDUZaMLtAc/iPH7CTXz+sc2sLsTP3frjNbN3+ahgxvnCD9e/rDfxPnu29wqBjdfdNByB1pwNcBkvq07xUSIJHgBiAWkHOXIefF0IYkh8C5KHvRjxxB7BNYWBFLm32a71BK5p21TbywWgKrxMoIjtqtbMOIsTxZEjdav+IwFwYZ3k9QObhvJzOVq2S2VqHFJAZKru9cfWN805SUJU3Spl17IiQQ+gYJjQya8BKw/1JbiMarEsWixK0krBOMljhmENFPkCBLPZI1aAwFpWiPZjWyK8GuYFtA71TPyAcicVTCWQLpFJ58YICwaQmnBR7STo1yZ4ZCgH1QKYMDaCzu8jvlIAoSeBBAwJAojeRVW5Elhlwe4Tkk54AWXOJYgUUceun/icpLiBjBokDNqjeULXBooaALohSG4ewITkFgxAk3BaMNAV3TAhJsd7uLC4f+JHv/044f8L7rq3cIsoWR6GHHnvLmbP3RhNJBOXPPv61j338M/iJEQhZqdTglgbUvmxjbWUZIRvQI7ke0Pj/j7T/gLPkOg870Ztzzp3TTE/OmBlkgCQAEiBIiqREUpRWyQrrINOWV7uUrLUtP/snR0XLkklZ71mZpMAIgiRAgMjAREzu6enpfPv2zTnH/X+nunuGQbL8ttCoqVvh1KlzvvPlIPTXJOnMwGqQA8EDAsBQZtaRoQ9ZZxPYENGYnfhzyiQbGKvF5RXwfjKdcXkEEbMYNKoAdoKCyYNq4+0Uh8CajIMi9Exx0tAbyJjI4YTxkTxEgKvfpRQ2WAA9NOZzoUP4l4g9dWuj64jWrDZJreNwYidGgSfkB+2xwXT/Aw/i8TR342YsEhkeDm6skXm07Zd4HjGmwhKh9QWKealkLpNcZagY4ItESOVAKKpej0uBIpYiASuwF7BmHGTU4B22N40Ab/+SBbFzrD0ojtayWARHsfEKxoc9faBBzqjWGBPpPz9R17HnET6WKaA9GSiyHXgo5miXrsOBMVgmqUJRbzbxlIabgLWAeAOHKFF4ltxb8A3y8HZ/aB8CDPKql1HzOklViMK51W7/0R//f0OR2Dee/9bRk8dOnjzxb//jv33hc389de8BuI63Xn3zve96XyXTfOfCxWI87ba4M5sJsn5QL54CrdhkCGZnOUsl0lYLg35b3yrlUg6DrtwqI7CFbGG4F2IsA44oiT6MHYdN7/M7I6TBcjh9kPHdszMkmq1XpmrV/OLiAt5MmA7RCUl27ioq/065Wizn0Bbh4YPiVpI1EE9dKPa8eXwxCBbtBEMuf9TlDVJv2srEz12/SSQp+ZCRvA/sOUwNn9defxXXb2LQSaK1Z+8Bpy9kIxOT214spLOFkvjgGiwyzzjctprzy8u3UEObAkeP7B0Kuq+89vpQaLKUr37mj688/p4JS4A6qHbS8ZnATjBM1I8RuQpkCM1ibIXRZOSZu50hVz/VeW4AF25fYA2B05Q4Jo/xEMsN5k74WsBP6CyoAyMcVaAH3TrxGlApoYyK9PJGbDqkNO8E/dO1esLrP0SNkvc++cCfffab9UiT5FcPPng/GXsuXjiDmjAaCQHx8c3E8PAo2V28KKpc7ggWqEorsbq6a98+qiEE/QGAodLvgh9gYqrVusnmiK8noqOj+DaiSUrkNnP9dr5QkXyWUFmACfIu9FUYDvFFYxlI/VgUSsAzXsjkuca/BFGm2/M5/UA1axfuQtJKAtX4gsI/uOzZSqppLFtCnZp5s9xfHj/gffcHDusOuwfJy4XlhDNoCk5aatTaGjQAR3HwhU9ifAFl6umJawmTgPBtQ2jBR4J1BSCShIE3oM5nhbOMQF0MPYMrk8TNzMKdxUt/v2vbmaHvOlAremfudg7uflItWDnxA69qd2qr+u4btLd8/xnVzk4XBBdoZ+7ay4vkQY1Jx2FBBoaxFWgDUUpNXxIZgNfJHmV0GE12ypBg6Wo0+81Gv1Rtl6utjXS5WG45POkqVTnwsNIbCnmihrRKA3gddxC3CMOV4nBwdLo+DixiMKtWMuRLMVMr1O8Phe3mQbKYQp1CwqwWeL/bN+PFYrC4bR4SEtMfLoDASAoANYGhJE4HNYvNygPkiE5B72kT3AiKA8lBU/1eHKlwu23xEfgRQIzJQOtQSYJJuiUFNiXNnt3ls/Y7ZR4U5SgZNZH1iVQlTXSrRnarFqaqVrXfqVG3B2sIpLhKHXOLoV2nsizJuSQvtJGesqqwyIhFGbwOXIEecVdjvTRJxAXBA9MDUFRlwIJNKDH0gzQLoBi4SwAetCtp0hqlfrOcXs0EI5H3PXLqiffcX0a46/U2V+fE1cwXfN+j955945UXvn7mwYdmSDpCBq5wwLmRqiBFMT/QG01041tEpIeCCRIQwQh6wDUQPtKdQitMpoCAWKjVcPGDOwrZPEkqDh8/TrIL3D1A/fUamTdgVER9rY2sBjm0xhLgvBL2BKwVvZPGNa5C7mc9KRcjvlHal+LW0gaQhB4Lu4QmBgBm3ID9HXbK4jSh6UKOpJIBGTOoHE5/du3ahdaEpHqsW6eHoAlbryVlJITZQJkMLiDWmxdZibeWWCyonfp8tIqYsawozwSSVUfpgKAXeqAGhF5JL6XzDIdanLIOvvdAu4fb2GiHPV+htcPQMrxoUzhJIxwz+No93LDdJB+q9EsMNpZhpSpACw2vAGVlY4R5kAOU8LRMHLrqj/SN86ifJaW9WqU7faNlPhw+lb76PS7mvpOt/d5v/vbjTz71yR/9+Ne++exnPvtfL1+7Qq+Wr89N7tn1+b/4/L7Zfb/487949dLVX/9Xv4pjA+6NZEIvVtroHBu1vpXwNjhvAKnbR4tLkGd8bg1VjBNfVaMulUlKHvvOIOSPEmrXyLev5eaoWWl3m6OjUXyTCfjzBux7orvL4RT5GbDRoPnBXOfFDJzKwohWiuTWqOh6GGcIXvBC9AD+0TGMFMbcagZ8E46I3QFrQrklLLWYnEx4QrtioRhLzGpyPPHoE2ipn/na50nvdfXa5fhm+sbNG8jiTqeVhFBQqXSu0qpUoAfEsy6urjQq1UcfeKjnDb46dzEQG8t3LG9cvVky6P7gz1d/9seaw87OJKYVg63XgMMnxa2BHJ+kidRgkg5oQ70z9Ts/dw6QxwAWTTBj8hX5UvRAkWfBJ6wG2DBYA6afOxq6arbhNbsNGNVEd4sWGmjhElG45MskpCTjtGd1befue2Zizzk67fLtxepRl2d4JHbtKilUk5BIPN+JysOHMBSJ+PwRTMIOu43EkclMYQQ3qGjkwQfuW7p1cyAqu06j38zmSw4X8VAkjpQABPwVMP8Pj47ghSNZ1Vs9ie0TXQUzL1RQeAdGBdIND+6U4EtlNCTXARK0w0b4AHQXFxeQpLJeUgenMzD2KqWMI2qpdbJLxZvBWd0jH9w3dcivi3TqiXNmV8cfcvdw4Rw0Ie+MV70+sFqINEDviD4Tk6awKwSrS+J3iqtgkFcbKBQUIRQX3ArrqladrHahujywZWZX63aLsDEhfIHs1MYT2rGcVye12VXH6pSclAON6GqTLeyUWmx3Nbp18/f8swMlGkzcfVVrljN3LglBlW4LLAjFFflHvWdrL7AkUMQeflBKG5GSBc6eoB3KXbL2QCEmsjCj6K03sjmyAaIvRMUBGSZlcgsFFRBFhi5YFzAgvgNoL3FZA/XJHGJKRpNJo5AF/SCTSkIfcBywCeXoDXBSynSK7apT2SCKxUKtVA64vX2HnsBedMB4+QIJZpcVLbFgNypG4jjXrOJ1CbyoETMsLa1QLAgUyBuglOFgkHxqSoYYFMr5JuU/ZElIjI2kW8STErcSJC0U6by9UhNjnEFSYPJHLkiEF1JYNhuEH2OBhj/uOoAZQbLkqWw5sCSTbRw3WdaQSmMEYidhY1uGBKkL3pE96gI1uxTq4l34+4iVmVqK1E9mHhhoGW30cuBfLviUb8VoJCSsSasyoJ4LyKndob4B2j+CkM39RiG5/Ou/8ilz/zcuXV6ktDSrgCIXsZC9QkkJIq+VmyB0lDHH9xsCALutbbwCGUtD9KB48DsAAPxwzKYBHjQIJiWbL3z6V//v3/7Pv/nou97zyiuvgBPJdaepTrWnWALAFXuot7I9g2bkM5kXqIUCOQkEpA/ih4SuH7ZEMpii6DNCdYTSwUJLBxgEMBMvF+UAagPUXQg/ynepT8GDWqttc7m/+fzz+/bswe516MBheATyP8zfuOm0UbpWkB4TKh8g4QAy2NA38JrI4irzBpQW5RkhXGhfUF1SXYCFjKDMZEn2aKQPvLLFFUb6Q3PaUAiA3bVpK0h6qURebdzYcxtPQa44lj7ckaS1e4XSY5uWJccAASjCu8tJNLmamwKUmLgjQsW4gamCNsP3C3mWJum5yOsMKSoYusPx3Xvc2eCAyrmqxQkOM1RL3Zkjuy+cP/vGCy/+1Rf+au/BvcV8AQqErN2rllOJFGq+s99+c8/EoQ+8/4Mz47vefv51Z1inSjKkwXJEddXK1K1sm3wsqA5l2QNRX6GW93r1w7GIy+IpZMqNEoxoc35uwWV3+UPBarGQ1Cf3Hd3LKiZsrtVtgbErgzI5hyZHpgAxKmXrHZRB72J3rNUIHJPQL5AtcGg2OZGCmJSpmd3jE6Mjq0MLt6/hOoABlwTG7Q5s9CAaDGKsqZaLcGK5TK5e6R6fOfTmrbOSkdHpvLUwt5HJkpIThw7Ivx4btc2xvpkiFhkvY/JqNlqk8XE++50zlIcwtzb2T2Lu2l21+PK96swxV8dkZ/1jR0ORZe4BNAAiCA5PF+EgNcaQ8aYbGj7XRh4wUQeKGeLork1hU3W/IFcl3ALTGsOJIQ0wZdcelDaLqHmcXvhR1AAQe3CtIH9ynTVbGw47nkzLIWrm6otPfuiBF5+7RPDR9WuX6AZIi35Bl3Cvi0QinCHakKUNh0zB0Gq5WisVybDt8Hr379mTWFuxEZkUCsKq4l1A5oONRHJ9dY2bKVqDf10sNoQTVp2AEIBWVqKQBpw0xDij1GJ9eCFglGIrAAdyMItTeD2WCSwDi4dUB6IyExezgQG/927f1Ug1sxVTPHpAd+q9M+ETQZ0pXaquOv2kzaBeHTlBOphjYElpUJSBknKP3IlGPIyEKMKASel4yaPF0LAAZEpAj2IElYGStBNsnGIY6KyselkRLENZYd+3yczdRfy0nzt3qYvya+eAY7W+5KS20tRVaeT/zabeS/9Ah1u93D6jWt0+qb2Cz6I/QKC4oXUpNE2QCNMMiSWRBZ7qMhqYujqUuewOclkyTgK51KJgEqHSwssVKwWSoIIJYeERyHI5qTWGNlQswKJekIEGZ0GBxZ0JjS+oR3JcoS4lm6VBCk42q3gaM6jM+NjYuNfpohQ8SUh9rgB58sT0Rk33VgeMiljTM/bqAxMlQ5LlLK2VipQPchIcxzpHbCCQhttB6HaWaL3Fx0W7UDRbA7zeAB1UhECadCLqlUqkm4c0tJqFXp9YGGJewg6722Jxuv0xPrLbInN1ol3PIqOhW4XUoOtmmRK+DvwRaYieRqy2XSzWEAZUKPj4wOIKYwf8MDI4W0POAStUAPAx9WpTEOigv5ZKkAUMhTZZSZDiVleWKD7otFp9VjdSHb3kbrhdShTAP5B5nlmIl3I/9fEfOrv7ciKdRQjIlGpz841glLcAkYo1FDaI1YTDr0S0MJJwmgAVsgW6DEHuQDHvFiac08qfVi4LaCCwtqqV1157DbxGicMf+dhHf+e3fhuPNlGlszF5bMKVClllDpEdWB381EBILtK8sl+qW8Vcyp10AboCviCmWW6QpQX0y1Pas0wZE8opUmlSRMfloZCx2+6iGoMJHg25kMif4feMYDlADnDg7SG0j5Urq5FWaJMXwebhncAxI8xlGucuCBjH6EiErVF5m6WQitp4jEZkPasvoxnu1DqPDMBJWmC3/XHyiXyC/KM2CDAbNwih3a77xGfSI35yC03RiOy3hkd9LFybsuxSDA4NG3WZmjXyhNiYPThaq/hs47QkQVAMGp0nnIr0jFpnaIbzbAwpcyc8h6REM8L1xMa8i9cWLAFHcGzkve99LwLr8SPHf/u3f/t3fve3n3v2y516q5QtjkxM/ur/8enFG4sLNxbcYWe7XVtaSKO5IckyOWbhkWE8TCwAg5GydydOH2nrKrfXr3mCztmJ3eVs025wlbP1bz/3EqR14cqKM0SZ91rX2FncuDm5Z6w5KFP65sjUtNtsmxqeSaTXK/may+sqpPKZZA7WE2MfNKOJl3+l5oLeSKDJYGMz/vSHnjp575Fnn9VdvvQmRNrvoVooaWdIYljDTxlQTiUT4xNO6s8vFxavX72SLmxOTs+4fQ5nHcclXJRqlGt875NPuuz2axfOpeNxljEK00QibbN4fMHdhDQMqu1Zq+vZV85cfDv1j/73n/ro+9+9fu3r5sItY26l26w4jJRsGXBTowwhZMoFs2sDvjXT2/9833mBsR+48bhaCsISKv8uYElITCXfDYS61KpUwbegZXlaAAxBoN4jt4EtbOs38ijmJg8Ol/76G9HonrWlHKCbTWV8XtceiOt6PJ3N+8kR2em7PAH0BY1mAhqL8ezW/FxsZHRmdjfMq8HtQTCASDZavWYH7zxHIZ+FmSYvdDgQoC4hSjhSIYGSMEoJoydCE5FF0EQkTLLE1Vt4yBBsh8eIZBhB4QxGkej7hsXtgevEXoIIQkqXVrfap3RbN9EyJfecCp7+6DHdpL6Wu1xt5sJD/matwWAC8dAUIQDCrgrzDk6AB8aIBNeLAYrsWBLyLQcMFBLb1rhAKoTUyiIUKVjkWcZL6JniG9SCEBWCYAB2ss4gyeou7V7tgqBFbUZBDerqD9hpLajbFC5Ut8hJeiTyhDSl/WTCto7UOW7Y/imdBYmA9LX7ucTYate5RyGlnX5CCzG4y53yBcCS8EHgLM7zEbDeQnfxLCHWWwpFUoJeYq8GxTrhPZBncZkTSKdAofiqM7p6m56wDgYaZC46WDhipFs28QJC+YYTDBMJfZesTyQxw31V2B8itxGi0fqLQCIWVmz06CYsZL3AcxLVtNQ/R1tLjlcDGmAyxBKhoSuWKfKAK4mVcF4iJnC5wvZMAufNdIoIclY33qRE0cABuDpuojQBmMGtZdx6SAqI4ZkJJ5MEtBRaSOnsliR/pKMEJ7ksRvh2aqLxyehnCoff+6TO0Jp78Vl3IJJJkrGjSL4MvBfQbnMbvIj46ZvthF1VK3U4CWHy8IKEBoosDnhb8W2q1gZVQnTAc1IKkOwuBhJZYDoZGw4mUhmcxfp+Q03fdNoDBENR8svhtDFz4eEY3EJiYxMtO3wngRngd0B4Znbf/ScOQO6njxwv58uJVP5r33zpW98mmXswjyu4xZbNF4PBUC5b4H1QbhJ1Q26xDIm0rsz66JMQ/JgFWV4Iy2B81gVytkGPzvCb33yOgg4Oq/0LX/gCLqmsf9G9y0wqNbMiKiwK/MgEwIAdTgspYv2KyVUAElATJl2SLUOg8L+FlqDok2AHYXDlHiVVA9rSCLdBg4UvMJAFjFQDkh4LB3suzEzOfPQjH/nVX/3V55//Nhk3uZM6H7iWy6qkJ/IyYR5kzdAFMgMpAR3HeHqFUhqPbocDp7kKA0LlIqR5lOrw8yCBZquNuA3ss2oUZRXqyzHUGqTBFyF23L3ieIN8zvb30j5Xte/lvPqJooRJpzGJ6wVfiUSu7mdPy9xMI1wC/gQbplOcd7jlAapl+yJOPdlIm01+chs2YIZTgs6xwElibQMokRftNEIH4azwUmFlUWTJQlSBwTAxOVYu5n/kR37k7Plz5AodGx7BY5FsJIihkZCFRC4vPv9ibiWlgwuh/iBlApT7AuLP5Nh4upLLVQs+8vDXmsl44qmPPubw9x1uG3mkl26uHd13cnMz3cLwVK+6fOLBgL42nyt4wmHyD+NdSOXBvCvgGx6jmKxZj3fQoF3vXDh7PZepskbOn01FAqJHZMKJhwcUkIiW1xZ/5V98+l//q39+9Pjhjc2FfDbBsFFUYGpiEifHzUQaJn3h9k1o2MT4ZKVRDIV9RCgNj0986etfX41vHjl5khgbrSTfpfMXJyZm4ksrmKh8hBUadQREVusJwGs44I1N7vqT/3FueV137vp8H1ePxFvHh02jYvFECCMddQutMMwVgjD/q5lSQiGTKpvAKh3nmppuMJz6IQhTQFmQtOBkOC1WhUK4gLTQDdQxnGHqUcygje55PCYKRBL+ZIHQtstQFCPuNF0CvHReDx4y9XYrQ/IMnblWTS8++fQDz3/pRiwSXm/V9u2f3rN7Frs1wGMv15YWV0fHJ7Aolco1cGU45gPY8uVKuZB9/eU4ywe1BCuSpLrk8cWhKpXenBgdI1VRymSKxsJgZjQW8WTapOLeJWJd6CNrYouG4R8qRFC+HdAAblkOdLbHB9gXl2/7/EPh0HA6m2sPmu6IqWZItK2pR39oduJdIzpboppe07k7boehVM8ScqI0xsIr0yL4RcEvrSNLo4kkSYREMYgXjXIIRNDVRlNGWo6AGBl8OVZjL6fUefkFryq/FQ3bOlDX1LE6v/WT5bR97v+ff/+mp39gs9JzRaE1KOF9/P4bXg9wChOnNCtCfZE1BYz6EseCawvUhT1/5ApUqNsgRWwRkYFYzqCpFqUypFYwoXRSkXCOOQENFu2g4GhwsUAnbBMqHukbqF8WooyqACmMsWIX8ZYSxSk6YBzZOx0UJmUrOc4wXSjjAEolcjXgl2VDqkXhbexRXddMshxylvVtDqyqekgFpl5I1PjYBLwV8lYPF3+q6QBiRkvPaMGPFjIkERJ4sCIVS5JKum5GwcW3oLlE9G/W8B7CN5VgGEsw5qN7RBJAWggPJzCaIsSoCGHacKDlKxmijtHS7tQlRNpA7MUAWya5wAnpxxGLREykTKi1dVZP0Od2k7oSwyrr32WldHXQGo7mUvnYsRlqP2zE1zgzOhwjnVy1mKtlSIxlWN+86vP7iT0wlWvkMGKZGslBQv8q+YGkyO9VE6t4Zk2MRn/5n/7iK6+81m6USYqdzRaGokHUrT43BW0qUCVoIK2xFhBEUZKjS0DVK0wW3Kcwa/B3WFKBfRgfFpsIkVAmUQRRssJIbARWwq2NS2z8UGiJMQZghPZxRl2RA21jOriHeQRlcEa7ipgrD0puakUvBea2H1A3cQnSS7VkAIN5bOpa8zdvfuPrz8Ea4FyAe12t3LRJghzmQLhgvLeEYYdwasgRLAKE8b+ilBBgiBlEC2oKYYPJ5iV8kdYfIf6UNVNdk17xMWqDy6FhaVS2LWy700tu0egrqlTAVt6mKCt77uEBrfGdYzlQX6lag5GXG5Rlv4dTDH7mtMYll9dMBJHTTgVzK1wwg4YALKMHhAnh55PEtLwz1MJ0SL1CJHIGESyJuySQ1aIulun69bfPn4P7hFjH19fFMNTqoVFIryc8gWFucIS9kN5GvdQt62xBBtMIC4vTEIuUH506K75eKbhzyTRdxWrYHdOfff3ms8++YgHwW7pIzJOOl3V2nc1LJmA4ddSiJZcOns1E9cB6mbDxTjg6hLXl4sWLxXzz0IFTR48e//CH+v/+N363UtANRckF0tq9Z/dmMg0bSrjwy6+/tH/vjMfjKubhAvF5xPHWvn//Qbw11tZTb75xbvH2/MTE1Mho1AHza0IlYNy3dxdJuGBBMpk0fGGzCtmeyW9u2ixEy1sS8bTdqAv67JvJ6sJCZdGii/gvTuwKutz1b7741tsvv/UfP/3BoD5nzqFUK+HkgXMPehGHRdY6mzbIzJv8+DtsgtOYYqHegu5ExNuGasCYUd0CIvEXMRDCTn10S8eK5wLrQ3wm0fKKZygH6OzIv1syG4tOn2+cspKFRKdSAkKGYuHxiTEQdY3gk55uetfMylpCb1rETZLA9vEpM+HRb7311tz1qyicwxJRHiKwggcxwKwsL9XLBeo0+wO+PbO79h3Yj6xJuHCt2RJ2nu+VXqvlzlomJQiLDWGJRYrrD1dIwASQEwNk6HU2kpnoeBT3n5urN+yUzQiaa+Zkx5Z67CN7R47YdeFcr7Ha1NWsMHcEv0iWE4Lx5BNBE0Cp0EzoASPFWKEbxO2ABFzgB0ERMt6sY1ayNuZbS1qNpUaXEddpSJxIGDO1ycEW0yAfwjmGW7ui9nLr1nne+DfRQXlMVu/2xrHWPs/I1GrbzsH3/NzpjJwXZMoDDNpdDwqXBoHUCCHgIO+Sn+xFUBHaKQdCUFnJ4E0BRKTTdgu3HdyI4FWQSKSQPHMvllyGTkQp0VEjXrH2BfLUANOW6OwV8gWp0Q2gDwQhNJt38GYcc9DSgjvooTaSok4EbFEVQsu4LhgBzhKVBM6r2GSlCRAXNIH8sGQLqjSoRmSGyXWYOsYGLh0U2fX7vCBEau6iQOLDcNHmo0kA0SBnu17vDQaIVsb92OrwQ4wQdqnvUBMnqQafB7dMJkvwKjoVGHCULuR9RPLzBGP1Zr2yeItoilat2qkTvdi1mxyI0bijkHIJSyPomHGDiiudroWMcUwddkc+mvwTDdPASuSf0TG261TXZFtejZNLz4UX28BSztddvYzV5nHs3kuBm7n5JZJsWmxU4dZFhnfDotLwjblr4dAI6LdczvfIjlUsU8CJfB/NYp6ogGavvb50q1hvk8nk8LHTH//I01/6ynMwNhGkk0aRFQARsgf9wCtJEsrVUrHeQCFPTAjVdaC7TJisBZk7ogIkRyNvhLfASU4EYWZHjEJIxqgrRCki08jssfGcaH/ZkfMAeMPViXu3AYmbuA1hWoBOdOHIvZyhffgRqIvYmUTfpZTIqk2u8kbABpmaNsGnAlhAC56Xfaa79spLr4CREdNhzohsIkYWEgXY815aYVXRKZz7UJ5h0+U06hbexwFkTLOq0ijgIYI+d5rFE5szHHMPAMdPNlYNs8lJeipIc+tAbqOHcqfac1UjmVBHGuGkdoP2LLcB6rTGAXcyhNKQQikyHqoppT3WE49HPBXBlnAIGIJJn4+jGXIMcCcMLzH1VHgkSryKoxYvkQ7QKC1Lg6wQRe/5KHWJqpsk3jWxx3K3trHmwP7SaX3oQz9MPpPzL79qi4ZLS5nATIxkU9SMpUy2rtwzBQz2kAHjusvnTWXS2UqeZL9WDxTM1jO0q4XSxfMX3cOO9ZXN5HqmltfpNnSNIN+gyxXKlDNCnqPaSHTUb/djr+umEZkXl0J67/6ZA5TsphAA9Z0xbjocMTxz33rzmtft2bN7z/LtRXRCo+MRyN7wWGwlEZ+YGUsX0mP1CLkSc0Wdx92m3OHG2sajj/SPnziJN/fw0JiU2ytnM5fiA32jWMUinC01WbA4anVi4VhseKJSqAanIw6dPRdO1nNJN8tj0M0kGkSVx7y68RHL0kJ8Y6NGKg+rq/3+J95lMrvrpTR2YA/OmWYSxvWqIoLdqZ2kppvJ1WBexljNgey2N4EKBeRMjQwLG9MjK0IYLgVa8pgAAigJqALOsBagA2qUG07Kw7moCw2DRZoF4F9kZSz+VC0i7V6vkyVs1xmLTk1GL7yx5PUEe3388kiNZCKxpI3yzNni+w4dBxXfXqKmxkqqWKBEEu2UirmoNUotE/wJkCuxdYRj0fj6GlxOsWQM4LYe8uPWTIAloRaR4RGqGfBWWXkmWczgbiFEsrRJIgMbyKoQT3613Fkfg67ZZqLSXbXdsISAns5qfs4Z0z398UMj7x4e1G6UcyQf1nkQWtBmEtgFBYbFAFq18YHfZ1UzquzaiG5CellBnGPQJGiewWMotSXDuS1SKiOrLSHgn1VJb3coqZonuYFNrVDtQM2WOlTntZ/IS9o0yYW/4yZ906Z2u0ntLd//rq3zMt/yAVr7HKoRVWflipzW/pFfAiWiOtsSYZVoK8RVZF9RCqi90g7wzSKGUjwDfYFIvQgeoB8QhdBOHgTjKPUSjQpyVS45AB+OfeBW4EuMwKLfBsfJAKOAEkEEFKWkFTolfA+oDc5c0Lbk4AJ7UvJNbmEuCABkogBlopmsZAjXm6wWo40UmHp8AZEk8OtbnrumM1yDJUTOEx1sJIAZuJghm2FryOuxWO348e/as4+UlTXCb6lChAaE3B7QB3wS4dNgEAj07hp0lKQ2GkloHg55ri2slCtZMm80quV6qYDOnPhD2EICklEu2ywOarQQXVPFCQp7NoomA26c7ZbAMmUBnDhud4y+vs2nc47a3D5TUd8xFXomTHLNeiFtLRQGLv++YEQ/OX3vfY+g0Sk125upkt7mc0AeRL9AvsMg3Ad+JHizBJx4ljL6JqKSifwj/3KhAaLMkSVn/559n/z4R1aWll959brPCxtB+Qrb7ZXywFS2OuEr0BqQBQAJV6K6UFpyhkWlpEXxxsTSIjlPcP8RIgShFOFVAnfgLIiVkgWpVsUduBJIYmnCzagJF6ASEBA8Bezxi8mEPxHZjjNcFIcsocps3APICDWR+1iQyurGcuOHqJxohCs0AiB0e06HPZupGgLYsCngYSYsGU8BqB3X5XlpUN7K7InEzhnOy6qWDUsE1BBKCcmEEgPKSLc8y5cxROBEoeBba0XwhDyraLN0RG3SOh+gLu1c1WBSmXtlYDhPn7lHKK6Q3i2FvHab1o6At9yqkPNgQNa2XDqD/gQa3EBZURdaTsfoJE69omuRbFle2qxXiUGibhgSuUwXbbJxpziu6ge1Moo7NRJACaONrI+VjkbcLlzHafw3fuM3/sk//lQxlw8c8fHJ//BffuqZZ76wtr5S7efhbvhL6lN2UtZJ2uoq9EfvQpA1u/wuKpWs3l4ZNgyjAClka3sOjWRDuOizOI35tZI96qpQUaeNE7W+SjYY18Dpk+AF7LVH9hxCLbG+ttFswKTbwoFJq9n/9JNPf+1rX/N4hn1+CimmoSsEGeNhF4gGYeAXltaoIu0JBIeH6vAiN6+jNdWVC986f/7aiZP3nD5137vunc43S5lsamX5JjU58HCuA5twfmb7+OQeQ984FJl87eUzk9Fhq8FVqvenR2aalYyOeMWYqVggcrldl+IgIm3GRkfue+hhaydTTy970VpJsAALmiRLQhHg84E/gcC7NgUC2u9tWJFfQhNk4kXgFX4UKitYDGgA48FEbjXCRe6S04RXCIh2dPUyJrGOFW8xlPFSnVD0UNxF+UW0CCgZiXjEqd1s8D344AnqgHW7tUqpv5FYpQCC2ery+LyUqzp45DDpUxaW1wqkgUTz57BDzLxu4j2NVDQZ4Nk50M9GYwQRkDOcDuLSB85gRa2uLds9fhwCArEYooQsM1hCsbsCYVhaFT0WnzSSVXFRquoIyeM7UIXip7qRTXrDeMJ35lLr9oju9AdCo6dd9cpls6fmIoGhsc14WMh22hk0K30cOxgzMIjItnwjFBdELrGcWwMnNhptiFhKQnS1gZZ/5OWMmjoh2EqGiBOCGmTNy6qTbfvfnQPtPB/GgWpHHpUbtX++d7/1Bu201pXvu2X70Z13aXd8fx9Y/+oSPedfaW17k29RKFJjSLggdE5YHiGiIA6hjsRGIvsiPZL6AsWh4uVBljzJDfI28BkEWPnLwekLHpVRYs4AIAEueQsSjPhzcBunmUbCwZGtaEMIM2MvjA42PJAIHJLEoAp6o9sKw5DWuE2MSRNcjYXS2mYRM/8kKA6iMJG0HgZclFzwZ6S06FTbxCjhc01SDuqHZPLQoiK+o1h1/F4PphE3hdzgk80miiCa7Q53UOcNRaCV5PCiMoLDZEeKNXYxLlP91oh4pCFBIKXdgWFM410vaSQg4JIHBDtXA1c0Crgj8XXEqwi/bsr3klatRTEJmBK8YvBXgqcp1sTu4vdFjARQWXwGT2yt3B+LxAITjnAN6/RGu560Ge1uokec7vlbt2JULWj3ClV07F2TKwg+SxRK9WoJGp4r1aCbmGApdOLz2YnsdfiojYsHJ3lOnMMhEnD6B9ROL+aamdwnPvqhldsLN+cwbeqCgfJo1NgaWElk2aM0sr5PUDNSIjMpzkmwV9hmRFQkvxcOcqTEA9OCQxCFRcfDWKi4AC3nlNi0FTyJ3xbTKphG1gsTp/1BC+RAsV5yGVUBfkxC7EBpIlDCkiBni62XnjCCnBPRVWQGGU+hTQpkgQWF1/gl/Br0EfcNiZOp1rgdUR4FIyWhBOOxIQZyvwChiMK0pdmqBcSVkUMs8RhTWvAQ4gGEEYQO8Taxmwj7AXchMe8aVaNN2tHkWqZXOiBAyefc2egtiBzqzYew8SCOEWzcpi099mw8y8aBKNRYIVoj8rnqLFyFy4FFn6REHq+XxDRkIKdv4CIoLK3RBzyw0IJwe9lR4vW8kzY00steex1ICWUs3cZBjhlGR8h6A1BxCUTxODkz7XZ7900fOHToyKULFylXlUgm4VOzpVx1Mx+bHUpubjLUeBRT86NKqRCHJGQkI0c/3bQ6Ij4XYTCdVLw8PjMFVp+cnS4OVzKbucnxyVdeedlDwTK3ZWAklq+aL+koWXfoqGX/voPjwRF4ROpi0+dctnj50tzY2Oy+ffcSsnfvqSdeffXlT37y8Vdf/c7rr71y9NihAiFQvU4lXyGJXq5UingDVpsb397hWBljQSarS2xu3F7cuH5tYWJ6gsqeOGItLSwXK4h5NWa0Tc2Sfi9jx9l79eH7ZhZurC2cX5iIhe1GL3aFgDO479S+fHHw+mtnsTcdOng0V6luZvKjk6Oh2NCse2KzmncWG+1KNVfvWND6G4nOp04LDNud6eZoa+oEFr5nk1MKRABeuaQhcpl4AAmQ3Lod8BYaBFizYnAXNqPfIS9HrW3t2XBh6ZAoHtiBuIEcAWUzPqqk7Gx1+xVTrzQxORMMupKpHHIHpiiDsUmBZJMV+dXPiMVGx07ee+8FHNPSmfFdu+qVEunma9XS+spKbGyKiAOrXYomrawsIWHjPS62JD0VHSoGq40owxr8KD2in8CvaKxA8iBx8B9qL7oCe6oWLjK5ACafYNBXWw27x9HUlTZzTeew7vGPBw4/Mlbpz/dsJfJn4LvA6PE/IZIAs86MwlJGhz9hTxT1hfTSniw/7uQ8/4hgjCJRTCDoBAQtqE1G8q5N+0mPhHSpQeaidsvOjTsHXFFXtRa2kAtnvms139X433Ao4sXOpZ3GOdg55urOsZxXRintETosuEChJ76Xq+zVBrXkp0w7YwKWANFxI1gFaRClo/J8JhxQAjVErtAGTyg0ZFWZcgWFoZ4Q5bMi0kK/Badyp9q4k46AqUXCVSIs38HLIVf8BC0TQI9YiSONck4QP1WBPrnBLOIDsyNqGygxGZvkaiebIeE4VQgsdg8htVgu3J7gUDg6PbtnPZ6olksLi7cTm5vUfgmiY5FinOWNVJIkjjiFwFjifU9dQLc/sJnJbmxmUtkCGk+/x+uAWNOUTfJjieCqvFpwTCLHQ7mWbXTIS2rCJZVBMQmOhEsk3ZXUhgM34cI3qJCOpA8TC7CSN5rMiFDioWiYEGlE7JbBhk+9yebav+9U3eIxekesFq9/vA4T0c529VDYepM8eTqrzReKxkan+jZXIlVAsC43eqHYeHN9CWBc39gMuky1YrqQSRn6nma9GIqGsHxivCKwzt4feP1hayicz5UjwaA5Ovb3f+7vUXf2c3/1LLrw0JAtVabwGbp1M7NB6T0qP7IMQwEfQ4BVEakXggTAs0SgRohScMCS6V/oG5SaTGQoBCCZhPrx7SK/ymzCNQujqlYOp2TtygoWWVZOshmgUkh0wMbW7Ms6gMT0SXElDyiXJf6RNcVSF96M9zLjwo3RVdUO065HYY5FEYcxARsbAmLbZjPb7bg4Cc2jHfZI4ShUBKIFGgVIQRcix8tlRVSFPqHMpllRcGsmKZ7jWYFNJVBySVsv2j08yf3aI/KK7Y0zdFXz/aBLapOX7jxOI9yvPSJft71xTP9ABJqUjuqYaBq8HHBPq5Q60SFJeII/P1F2QuBVKQiuQolpgL22jnid1ri0LO3pyJZKtiBYaWQMUfCJhZu6gQ6SFxLs1Kg2Lpw7v7y0hHK7WqgQO/TK66889MhD36yX8+RR9xKdZclnUaCUZO2BMZ0k4x00CrqMNT3Qe1if2XKl1Y7nM3he6u45cfrAvnuo+nz2nQsEIbjJRWnXk1GEqoKUeCcpKFFP7klifI1ue8Tu7CwvblJ30+ueINr5xRden57a9eWvvDg9eeB97/sI5uFgyHf+nTezjcIhnLuO3XP2zbcWl+N2gykUHrXFzBfOnSPK3+XWUWTkjbduvHXuBh5b4SBJGjRehwqENiK3kH2xTKLM0vdsn/joT/71n/0ZftqTUX8pE7cM6p0iWVEHRBNaDdapid3HhyIZyrT1dJ9/5pmff/pDhVIvaHTpzVayxejtFE0mGwQxHVU1lVvTtjPz2oR+P/7mPFiOFST4DxCSlSOrSZhUdaw1BNXgt5CBbh+8V+voCByyNx0opvR4gg46yBviV0MqrkFTj/OxydYzoePvmTz2kSGqaa1S9xPfOnJP+oIxilsTJiB5u6yWqd2zx47f88JL37m5cDvkc5F2OxYOIWVjzFmLr8FnvvbmG6gcxoYCrPdGreI2u9GLhGJRJBPkB6LCgBnNqiF9BMKALN6slAEDfPKFamAS4avgqI14g3oKjbTd33v3Q6O7TjssQxTNmneEByYv+f3Jhq5z2okBtRNlhIpGrPTtEu2LXROvDtE5QzOgoIruikGX16F7NhMfIg5x4opS1aphyMCp4Ze7ZZOO8Q8DLbCvhlfa0S7K6lV33TlQv7f4ejneuoFZ+btuTKigDG37vvZ/8HmQjOo1WGCro4IxFOkVGnenPa6KECD0FQmJERKtsqgNEHxROgpDTWIS+VrtcblZopKQjNFOiIQL8RZyD/ZR9FuJQOp2eEj1JxAIxpDPkAnEHo8OxELEDbkvCOuykJqX4ePFzLxgMJCvZHWAbwIzg3FomJAjVERARTK56fX5jQ43TjVWi8Pr9E1O7Q7vms3F42sb8XqZSkj5RHwVGQ5xB+RbxK+pVvGQZy4YcLrcWLlMFmuBuwql9Ga6XK/6nU40YGB2vDmpNkixS3TxguaQOsFElIcniyCq3CYpEsX9WBTU6KoNHJPXScokkIcYdgXZFRbSiXeu1QElAPcNTUzlUplaN29weIgbMgxsruEps8XfxIFqYA6OTOsc+oalXWhkUqmV2PSExeXC1Bcmr1YgGM+Vw+FRECVqVme5mN0oFynw6RhChYiYTra5QiYNH9s31ki9iE27UK/ZqYqIBdHlY5xWL5ylpPH7Hns0vrL83LPXr75Tc0R0JEV32Q3eAAk1TdlCuUmye7NkdUcaRu5nzzziBcckAQzQYOGEmFE0ITLHkEdZLEC/8MTKNiyLVIiJ0C6Ra9VaYOY5EJqgNo5F9MRBQNFs9uoRcayDr4KP4SftQC1onwNxE+VxDXY1bYtQMngnC2nJ8CQBo0GY7DYztBbjKFRJcCL9VS0z8tIbOiYWZmgchEiJiYCiUilrlIxHEF5FEwiFZgRQ5BJuJPoxcISirdvLjNukw6oPXJV3bW8c8iLtEfbah/CVHLBxRui+cphiXWn4QUZGEV7tN7eiMNEawa1MMvwpSw2DwYMQYHpDB7LZLH1iFWgCN/drG+2ziQpaJBXJIcGSRHojwJTM4JLhyeFAbwuf+/bZs/Qh6A2mUhkC9+Ibl3PV3Ad/6GmkwPnLVwPh4OTkOJ945s23QZp9vBPrAwNk1yVkL5Mq2zw4Ejg2bqeNFsvmRt79COUZ0BF3ChTQLBedLpPDaRwa3XPq1OFcfmPh9g36spFYjoW8w9ZAk8qh8M2Q80bn3JkrQ0MzPt+wwx75i7/8SjQWuOf44dP3Hr/3ofu+8sLnp2cplBAbGZmuFzuE11OTutZo+zyRlVwaEENDxqBSuwIPQnzge+K9UbdarK0OBXhqhJaino8FAx5HYPfe2dcDr24szCdBXqiPshkcvvBtZmmSKOXrX/lmadC0e137Dx8NWZ1/+fmvWFKLqOWH7A5cOU0OXadhqLZ6KEyZHW2qd+acg61zwhoylzL72iZ4StC0oF3ukf/URdYTm7SgCIYccBlVc1tncsqqqlYGrkbHRDErEkYKnJA2QHwlATlcgsmSh1lasmW5LTMzEy+9fI4FCSrw9I3kRcB9zubEhT6Df0swHCX1iicUeeWVVwqloq1VaxAi4PIRD5LNllZW1xYWFjA/4VWKc0BZ3w2Efd6gn8jMVLFOuUCJowSKhLhBq0RbwyJkAcGkUjCCL+WLIaYSagq89S2dZH2xY+3/zI+9e+bjxzsb3y501/1D9konjzsf/js+ok4JG0b7JQvSSIUzI8n7JYMzlAPqIVY1XiMLDzWlWi/YMjUyw0CJ4C2bDKGsJYUmttEL5+B2pKsyuNJhGVwhMuz4yWOCpGQvkp/aC9ZRzWFkuDNp8sT/yqbBg9pLc9+/0XE19VzRblAQIy8HH3JG9upP67mGTTiGNtMxNeXscVajZAI8GhpYRVwhLKBK5CIARmiqKJ8lIxY2CvZKi6DOg9B4lFo0otzk2+W1vI7eMA5yjZeocTJLYSuKEvq97lq5wI2q3wKq6BI5Br+I1xWiFvPew1mkS4wFgiWeOwGCakrV7Nx8MFYbHp00W1xky8OjmNrg6/HVsZERr9tWobYhbgjlbKkiKDgQ8HilTI0J78qJqSkSQ1+fu0nkBNIACNhsRwdMPcF2ky/B08uI6UWcdMAZfK8JU5LRJT6EuGiY7FkMbYyaqBkH5LwE3aHBJT8A3LjoWLCDGqwsInIBHjpynBy0i0vxfL07EZ3QB4zJqq5Qxd+0DbhTctjoHdWZyGSc7oeGMWMVS2Rmay+vJnzRifC+w53rS0urq6Fa7fC73x10Oy83Kw19myTRqZX5iD9YyG5GIyNEeuXKRZTqZJXDTtuuVjskhu30s5upqdn9169dvXLxrY9+6PGpkYDZ4YuN7//qN1587bXzpVTB57MG3eTqIwNBB10UbAq+lww1Kg9xFsIpHMc1SS0pxILIFyRWsSMo7QgWaT5WfAVEEyXQbwTYBb7kt6ilgXghygJ0TChRTqw4No4Fr2xTrFKpipCNCgSKi2IZQqJWPVUSJP5VaDhAR7MAjQJQ5gbTorBiTcpWdEZG/OQJIY5bqhjJiwTW6IDY7US7JmH+qlv0i67KghZls2xEN0k3OKJlMX2oIgcgcJoQHpI7ZBOkqW6Tj5M3QOWFQstHyx/vELIt7IK0ipAOzyaivFpuQrCFMG5/r6xLzshXSV/BH7JE6CF3d4rkZICQUHvAks1j+iX/NTpn2NweMXi8iNg7biaVCrOksAjTgtEAoiIOiNBZwruJYpdoBckugsHd2hFRn4C65sri0s2FW8cOH+P81XeukFa7sLwR3TOBdvtP//RPmaZALJyMbwZDoR/90R898/rb73rXo9dvXuMMHxf0+ZqdEhUvyPzmoMyO0/jUU0/hP4HPxaVLl1FgwkKxHPCyTq2T967gcBhInVHKV0K+0UZDv7iSaQ37YEqJKCa2vlHvJNY37jl6n9cbmB6fIEvl5sbqB55838mhUy8tfXPfwQMdPYmcu+956vFStXL21becNscDp+53BQJrqRfzVfASsKQLol8iugFpyQR5LiH6EjMo9Y5wbjBYpoYnSA71/HPfIHtlaTORXW+HPfZCWhfyc7+pCstn0UvcjN8xv7j49We/M+JyxXrmWLuQn9gVQA4c6FyYIXDjY5wVGymzJfMlewW58kt+aJtGhrVj7lf/yUzLJGmkQK7JgtDuUU0JECDD9FDUSv4aalEMGugesYiIhqnZ7ziceGlQQ1i8EcW7nxyZRIFYBsGhIUib0+UqFtNUlymRah9F+tRMrlCdnBn3Bt0biczwcGxqcubcm69E7IZ4PFFu6gJtXSwystrAlteH205n65FwjfzdHj/shyuVLc7djgciUbLCizUFEAZsAWUBUgFxHUHRM9Mjq8sbTqcBfgf9EcPU6Bdbo7p/9C+eHj09U984WzMWPWF/vV2ES5L8tix1lMtAOYovYtzM5Ni261p5IeacYkSMYgESFMK0ikwtQyxMJxlKZPWrtSc7pRmT5aL4Vhk+oShsPKvNivRThlsIm5yXSzJHMspaO9qqZa/u5CyXaJGVqcZXXr11p4qi0NpXT0uDauakaUX/tDNyTs2pIuXyJumYtsQ1zCH8gYAA7JTgJbFf0xkhiaKJ44A/kCQN8TpwDpMsvrCCeZA30TmjJ1D1B0nW2sTghg8V1k7QNTgB44hE6cILMbq8VOiyfB0IlMHBIUvxgLBJgsI1IBKAc9j1rSp2NXRlupDLEvDakVFrpSSUGBdJeobhioqY6DgkRhg8LE5YnBc5RsZEYBXCRkdrdpfL73QTd0HOF0x95JkcCjgvbC6NBpwU367Xyj4H9FrK4oKkCEA6cuJebzBYqbcgwAiU4KAnH3/i0rnLD566lwV8/p3zeEHbnSZ/bKhezKMKpDgg2R+qpbbX7QKJGY3Weg1TnLNR7zpdwVIuyxJtg1EM5lS+SDbJ0eHwlRvz+w8cmJ6exqMV9xPURLcXVi5evA7zeOjE/X27Z2hs1FhpLyzMHT5+cm1tqZ1aDntsrexmOV/t9t1G+1C3mSfEHr1Uo9lLzM3jKTooVTu1uq5Szm5u7p2aKjrNRGHieh1f3jh0cO/6xlq91nXYPSiHmtU6vKkT73BEQ8ra9vsL58+6yXFg768vnD+wN0RxOpyo/sU//ol/38r+xedWiOl2eqgyAVYCCPWlcqtvk3yipAuz6vtU8CWhrjvgwL8DH1rUnySaTOfbLq8bhQBVpWJDQ+lUCgYFDT+ZWpHImSChLsK1CuABSEhrLGcYJm3ysPID8ICXRonhizGNc4angEruAZBk+Uv2Trg3qLyBINTNVJLEKROTw/lc0UaIVxNvG4LF9UOjTpAvSjuAmxQggjBkCfKoEG9ag2NodJo48oEsqJuMJtdgMTRyveCQByhj413wiYibLBFYLbCFXYM3RSNZqXRVSfkKA0DtAFG5JH+q1+h7MFXwlZKESH2XsCXKKZpuCOSyyRqUjY8imhy9u6xcWY5CxFHm86HCj8OA4AmMOwskE6kIO3+phuuwDtc/8veSSyYaDcIbkO6RxyUfOS2wLliR2HqlLJBop4ukU5VR5gQG8paetDfguG5/dWkJh71zb7yBY127XEMlGRyLZpKbDr+zVi3y7nAsfPDYMa/fc+PmHMpnb8BDVimjRe9045VtyuV7FIStFcu+mBUCEPVGRiPDLo/zy1/8QjqflBTn9SopZwKjFny2l5aWxsaGqAZx5crtgc5h6G/+0j99/Pb8WnITM4yRXv1vv/DJP/3TP/O5fVZybFGvs9v88jN/FfDazl5+0xwxzhyYCEXC6VL62CPHry5fHRueaDi7cUyXPZ3DpRsbjgx6DbsLERhy3gSJB6Lhaq3BUiRDC2NQSuYuvvnmtTOXFm7crhUquLA1Kv2bSzUblXW6+FT77T5boZsC8nWuwel3PdzM6H7yqY/8xb/7jcoinFy5CYdQ0w3FrG6zpd1A1c/EMa7M4J29HH3XptCzOqM8CtTUgGk5rf6YJ5hLOVbwozg74SQ4z+SQiga5nMJBFIOgDpU14oa5xKUUl1AwrDiSiCCMB3mfGHDcDr1jk77YUCKZQbgZ1BuV3CZqIX3L32uV1pYv7fechNclxMHv8YWCw+VMHKdFL8sYNbVh8IkPPfm8XXdz/hoW/RQabJ+3LnFeLmpVGs3ezWSJlSPMnWihhF+Vj1bEgnJv7ktXNg4djqRSaaeLjNINkiAOXLp/+C/fN3pfUKdbLxvTVkffYJdPxMSFlU4GBAcZXDMtuO2LBhPHUTFVAtSCKVgO7IWssoHf2SsmRb1VxkoRVGWUFLImQynnWGYcbzHn6hZFf7gsm4jobNjOZHUpaVE1xTmZj51jOZRNtSUH2gTLkXZS3q+uqjPfvZM+bLe0c6DdvPOU4A7pLxwIV2S5yw3MqXRE3FyEh8BUpL5L5ACOQQ2ABXylPIVEi+qrjRGIRDyKEpM0SXANXBtVo4V3Yy/NKkM9rxD9s2hTZGywtqKnBXGACPiJJIrNBtRcyg4cVmFFvR6b2+mgyq2edKuSF0dejRwNrwd2EsYIxKxwmBoH+cmMyRdhjwVoRFsKEgJI+uTPKmRTzVp5fck6FIJ6mEliVyOgm66Dl0WHrL//wfvwNJbkR73+3I0bdpfbHwyTjAPDWzFfIkXlyOioCdMv2dHLJbvb9cDJE5l4fPXWAnrOQr4UjgTtNpfV7qm2JHUGVAHFzcULZ9E3WxyukC+MZ9NyPOWPEnfupiRUDY0YrxWnlCqWMRP1UweDdDJZqEORLJRVSixdGw9HsV+uUQB1/qYLLXu/s7Ky8cFf/Pm1s2+98sabr738xrufeALtDyHCiOX15SWSIpA7K72RaOTylkGrXGos3FqBR2p3GyxSvGD8Xh/1nBDai2lqh+Ltb4RXMvYgPUa3C+ecTjm36XTGKrmNH/7AE+9+tHbj1tLXv/lWPKkLRXG+dR87fABshYHw9vymx81Yd9x2Q7NSx5ORhSIQotcFQj40S9VGOxiNSYAmM0QMFpeFig/wIYcxFq0QZ+C8UOeKOIiGSbzcgQsoriYsUmtW2pP5ZTj5eIEcTOuclDy5rH5d2+v34VXbHnTGpsewSlI2lhQQSF24mZOWp5InewY0tM8Y4odFrUlAUZAGMIEWj3ebrFQ584UCZB5AdKeBdkMXGw/6LSYqqf7QBz8EE3fm9TdJYYajnEotItboHZAT4FO8gqwauirrcxsZKwKsFiDkGUiVq9ys7pFVBsTyiIJbGBENY3CaJcKXyYoDBcO7K28KMIisFzT/UgmGxdUkNszsdrhIRYnD6tpKArhFWQdLnC/lWaHMKhxkDkWKkfgrA/plljqmGTxjMdm0Ol1kQ1pmMJGAYTAo/FcrFNzhIGZmMq6XVjI6J4m1DcjK4L5qsSTEodvDEgG/OD4+uXfvnnvuPf3lL38ZXRGcBEmUpOvw43iEwG7Uu7nN1MLVG/ggnH7wXmpgl5dTOqIQPHqn0xEKea12CR/BsiOpwHTmbLY2Mbb3P/3nP/wHv/CPguEEMU5Xr1yev3X11//PT//2H/5WIrGE7xTFLZuNyh/84e85AuZBqpdvbHzkAx+5ePnczfkbuw9Ol7Klkw89pbdcOP/OFdLDhaIRl8OEJQbN81AsQvgNTmONVr+Oo2Otncv2yBpSylalPFq1TT1uAAuoobZ0Nl2Znh4zWZw//OMfW0xf+x+f+aJjWvdvfu1Hq+stPKEeuvdEJVAnhngiRBi9rVzAUNP2OglllEhxtQEAMhB3I2qFpLeva/8KK8VdYolhzx8EQT0J+mdhKAqjTmzdDoESMgy+FZmGPyKPyDQFngYqhDgJQRGSTSOy9bpOisjEYs0KqjopuNFAceKw35q74h+idFL65rVLw2N7CSzMZ7LhIA5oXY/Lfvr0aQDv1tzNai574tD+qfFoDumZb+vprs/fdG4W3L4oq3RtNQE+hk/kI7XX8WL5aKGY+sHoiCseT3v8xjy2fZ+uUOn86CfvHTs5qTOSeXDNYGp4fZaBoUIKLiIX0WvJMDAcfDI0XZhUCaqTAkp8hhpHVgzd0miq+rgfsJMh1/qg1g8UCrGS+8Aw6m5Gh5/C62qbalBNFL/BWNypbpTO/A0bl7hHu0EdiPjNvfJq9RZ1cOdhGQ61bU+KNK3dwxMcqE3ukAYED4hcztKVq0KJYZrlkhBgjTCr+YU/4SdNw1RLLWbMgJBelEFtfbdtwFKmEmCpOh4YmeBcQRaKyQHXbDUFAhSAk1mDjAEyks4b1x1xkhKKTme8LtxxTaSqcbuw22HdkILtFAimc6xuIApsAu4GGkSOkcZEW8ELkJlEMYgVEJZebbgoGNqiuMRDAXDHqoGhJxL2cxuImQZBH0LskC0slkAo0k5nUJiToMPd0VWqeDzFwU2HDxwjl2EBBbhJX89l8AIDcPbt3bO6sNyqlFANY2eCIcVtKZsr6C2d2cMPFEoNclAHCaBD2b262qw2yG3VN9pQLD92/6N4cuNHg28XfguBgK9eyMFmOIy6Oin40gVzIae3Oo0W+9Lls+A3AoIG7Qa5Pal4T+0w35GDraUlu90dDQ/hSrZ0a4kPJxVldHry6oVzFDd0W435VMbc7xJXD/uDJgKOEvQv/Ap2KzaPC2GxRqrrQoHywsC6bUASA+RjgrUMZBm2OVyra5tOl2f3UDQyNDw5OfHWuQvXbywc3Dv9yR//5MTefTcuXCBSZW2llcvW7U5dLq/zh9smW5vCGcw2oFGvVcmnQ3k4kpUxwmBtFBqIrxRbwvdcdC3in6UojeiupG8sMvWH6znSsFQp0KP2xn2AJA9CcxGtGXUBVI6VJ8EA+YYbgviRmc0UEx8ZGcnkcoDIxtomgKFUrijQXDpDTUKb4MWskllezNZCL1ntLCJZqFgIsRCTo2B0NFool5JruVMPH792jbJSt8bGxngdeXQnR8coykbqtGKxQ8ztzurjAEBjz8ad0j9ZztrCl2M+VRHUreWv3Uy3tZu5yj3asdYCbYBy5ElBqTJSsl6ActU+V/Emq7c61HHgowDzAwcOSNJjSuc6rENDAT4IqsMb0SjseWh2hXDPVL7bLbvRCuJ9jfcfCWh6ZIdtCsaDoqM1xlnaaQ2PjCAhsz7GZ8fJb7W+Gqdc0KDa07nQbPWNXtQQFBkbXL985fbCfMDj+4kf/4nz3znTdJHhV9AFG0wCnWZFk/Dd7wg+//zzRAeDCh+6/4HluTn4hn58UDHUwpFANlNA3ql5bOOT03v3HFpcSN+aXwbUvvCFv/zZn/m577z4/L33HfnSV/7i4YdOnDh14PrNC+RnSGUzmWIKSNooFn3j1un94TfOvhD0kd+mnUmuoOxZX7n1ofc/OR4ZAnm/8fKrpUILxESlMnLbgbUoPkbpURh9NDxGc8VisC9dTZvdJBvRBSOeZq1VXm0FJuyBiJvYd72xuLR688RDp89ev4ILUSHTePnZ586120dDjk9++Cl97oJLl7SbKbDWhI0plwtiw9xC4ACGzBxzdgcEZFSYQrX/G3YawKinFE3evk1BhwJ4FfDK7GOqBtXYERko1iJIW4i5DLygRIWveaZTtwWGI7HQ+u1VOFyQY7naIFFWuVDWUTnMZyzX1lGok3RtI76KW0AsHIbCZ1NJfPoqlJvb3CBJNGEf9z5wKpHcRGHm8AQ9nojLG8nlahbeLFonEUvppgCoLCIhyYLxPT53lejLRi8wpIundR/66NjRDx3ulxcq+kSjWQRZ4DBLbi2eEIuM0E2NIEC90ZwiS7C4MWXLADCCghHUYuAtO6tre3Du/CtmXNSqalM0TLX7t4459955Xh2pMz94lhCMtPm7M6ky4Hfa2DnSDnbaVj+lze3z8lKu8lPt5VhmTeZPO6+RYXUsXyJhH/KvcAlo0mQQcN5AOAacGTDihfpogkn73DLAToJjIWoqFBheTQgqf2jOBC6EJ5GX8pzo8/kgMhY1iOYxuIDlbpfCnPChJEtw2vVDEUyZkuiYm8j+Q+AtH65ifqSWKoSTrwDnIsIgQfEFIvWylCgWSMiROL/rLKSY4q9H9C0YfGAlQYVpgDGDp0QaJhqQSDqVdgOSgPiFUEQ2Z1RzCG25fAE3kOmZ3ctra7fml3D/m5u/ZSN61+3avXvq9q25jfVl0uyuEtSUzZN5hArEEpQHsJArYHLYH5voWjyVdH117bZvLVElgh7UA5E32YkS8AUiDo+P4ai3s4i/JJ2n0/AvEHL4DCx1toHebXFSa7WYTT9w7ChmuWI+QfEmFEyF1Oqgng9HopcvXcXDZWp86uC+g6+++ir5gaM+TyWTJuxX1PF8PqxJnxhGgneRexqQNMmjIckwSftVF9A3GBBk1+IpXP6RVl1ehpESJiSLkCRw7d6AcYCwkU4ZAfVd737w8P49t5dXKLO5uTKfii8jK/yzT/0iisRMrpBIJF56+eX4RipXgntv+8KuZL7IBztcnlq1zGQFqP7qcuPfAe3Bg5pcmh2Se4mBgNUGLIgaRuNy6Tw8mEaMmWJWH8csbVFAwyzLGYRAUVCTd5OWIcCFQmHXnl20kMlmDxzcH4nFFm/dFuZR0UIkb1YmD3KzoC27FDhCdyrtaTYjxchDq5D8Pvzhp69fv44ZgvJ2JFumIl48HucpDeQwZ/IsQT7BoBHHYzDmNtLk9NZaplXtJG/cuarQwNY9zLv2FdypgbH2rHYze25gMFjsGuWVx7TfCjF0mgMK8jLsnVwOUxB9xjQPnwoA3Lw5T8vAKv5TjDPtgEyPHTvGPRBgErZaTCwiaYUEHUdOnGi0O0vry7ViBec+AsYHjVZmeQNNsMFm20wmDx44kM9kGCZHxFPPlBFee1VqmDVtQyGfx0/emItnLhAZv+vALvwcq9WaXSrxWojG5fvMDon1B6cUEml7zH3hwoV3P/7oI+99/JWvvmCdsN57/6kHHjyZziazWdJ5rdns6G7su3fvthodjDNs2G//5r8fGwvW64V8MfEffutfP/30B4+eOvD222/3C12L28Sk3FzKHDt+zOcmO1tR6pT0mlQZRkeFP0difWVibNg0MP2Pz95GMzc1Tr50NBY68i9a7F6yYi0uLo0MT1BwvpAtdfpvJlY3exldLlM2+YyGiIFIZWvQ4vE6YbXfwLB87eL//g8/he/IwpXr73/v+3/r0/9mQ6f70MnwqMtRSVSz5QbmMAqnFdoY3gDbLYoLLKhhFhT6t2/ctnMzUyMTrfbf8xT3IA0J/legQRoo8v/YO/BfYiIUayjnJWc+7wO6+COMvUXdCq/PDfajbBv6rVK7hpuKy+FPJjPjTj+MG7oN0uS3qXJerYW9uEXvQvLMpNOAEIr89fjKvn27c5kMGT+ikejw+CQE2OuPtcb1jCmeL+Bc/OfpFG9VGFd+gYV7m4mUJ2ozuJqZiu7wA7rHfuYRnWHZ4MhbDVWMcXqQEqKWvuWELXZYB/UmC5ENwUvJCXJMdJcyUAlbwYgoAUvWhZKs1DqQu753Y8XTFwbx7gua5lroDQMjbAJX5VgGe/tOdVKdg3fUpk7Glf/4nL/Tdqep7TaxrPGkTKg0IIRP64PMpfyvho1/uUtgZ+snR3yytKHOq73yMFPdQcQUpMAe9R/SC+6uBBC20OITCysxSCL+QoBRhLH6pNIFvnIoqtVL1OjwahqX8RTCLuf9bqJ9KZrUIhcv+hTi+CMhL5nP0IQCilQ+kCgf4mlJqwQdoEombUM9RdcH+KHDxJbMSSmUC/lF/uV2ppc5FbkYDhHzOa4CeMWjtobzJxMDIUwmk9vhhFrXajzRFtZC9J/kTqQpiovB0pfT6Wwfa64UaIJy22CiZ4dmIN4Ul0WPXUxt+u32fDJtwRBdLoN0CfMPDEXJlhQaHnEOjf7l518aHd/d0lmuL64R+miw4q5hJCLZ4Qv4gpFrN+boA9yi1+tLxsvIKH6LeDQhceJQEw2Sxc8BDk10CvmlyyEcOT36XG4T3SOYl0y5g0Fr//79roOH559/oZjLfuKnfgJfhsvffoGoKq/LDrPCmJPeiZFCK05mYBlsZkGc4HQdnKnIPIi5nYyFZNUlfsVGaW6PPxgjZBDaQ6wgtaPimP3cLkru4LBGU+lV0u0WpoaD1J2lZFtkOFBv92wMb68zu2sa7H/yxImvff25t89dXFgqo6F3WGnYQwwSWcngB0J+nNqIRyww+fBtOAKp1YRoLoQSkJDVjJJWuqmJUvBGai0wWcp3yWJ2iBAv+ViEhkO5yYvZUyV1ZbDsLnJEoGaAY0CQhVJyQO49NvyV8vmKNC8++zoqXPIvcV/45hCqJOCoiDopfsieNDk9debMGRr3+/3keOLbY7EYznf79u1DBb2+vCL0VRKWWWHyeJaNLskCk00oPtl6+SlIY2shy22yvGTlyFqEheWPj5AVqc6o6/KQ+il72oLJ4LqsR2lHqbIE2oVxUg/RERMMHa1i10xtpvz+IOOEJr6Yo6a1hCGRKgsPu2vXbuTzRT51QMkx0p02u8idxAUsriy/57EnDh47dOb82XhqU3LI2q2ACRlbEeepMvKJT3zip3/8p375n/0fxULJHHR2KjWbz9UsVgup7MjY2J6Z2Xa9hQmEABii6nUl5Om22cEQKALSE9+ufCLtjvrQ8VNL69yZsx94//tf+dYLzBQqKnSY8Y2VfIk4l02310dcL2HqmJ+W15aINyAs8MjhPW6vDk/KePLmN1/qjIxNtHSV4ckA7tR4M2CpXF2ZN1pjGByJnSNZE/laR6IhrEvzV+ZNehtmFgYv4JP0XjarYWh4LJHNef3mscnJ+CYVntL0IZ3KoROLjAx99B9+7JVvf+fqV69M3j+zYYrbycY1aDoCumQx6ewOXX7nVr1kDrmGp8fs//jvfyRz5cXN1QWLs+7EO4+hwt2l1nQTjCQmPMGd2rQzp9vzz8xubwIIihnbPrHzLzdrAKMeFODZBhuFmQV+FBQIrAnWRXIV1aKQP4EbtZcfgqxRbpAJA08cXdcf8sMoh2yRhq1Tr21i6At7guUS3EXFDRsGOOl6Xqd9fChMvvxup5lJUbAxSRHuWr1CQSPI/q25OZQhCKort+ettvTRY45YZLzfHgGzSxAwYShQGkLvFOWiH9J1VpbI6DVd36X7iX/wmM6fazWWrbaucVCzkGBArH2UMBMv9XYFXy/5JACdxcMXsQfExc9SWFGhXFyUdSYrjT8VPKGeuHsnGFy9Ww2I3H/31R90zGr83tN/+1PybeoJNa93pmf7Kbm4fawO7mr/7vPaMW/nj8/RDkCCQuq3TzKDW5cYFhF55cNV0iuUdjJKMCIUnxCXKQEFNM9S+Ah1NCESUoCZtKUkwyKCS8RcLH+SLRQbpPbJ6kM4o1g30T3CEIng63PonB6rl4ws8NImciDU0RDzB7Em9EgJFvQKHKqsHsJ5ySeLzks6SHIqlZwb1Qs0BnpM+zCE8tIeihqmm9z0eFt3uk1qJ0HMne4wsjvOHtAY6DxmMGLPCEWkHUQ6jIPk0FqPJylOx5qHZoHleXc+tXnhQqeaz4W9/oO7dmVX4/OXL+PbRe4hAhyDoVit31vZSNbW81a3/6HH3tcs5S689VpqfRkHNAgJ1m6H+MVYUhsJkX46vVDQR+Y/Im0pJIESHEWWWNHJq5FPkj3ATt30TNnq95MCeNDIo9FF20Yxt3gy6cxWY6XSxPjI2srSa1/9Uja1OTU23G+U6p0KnJEUvEDvgPIAf34jzwn1UgX9ehJ2ggeowUhtqHqrS94OnwHHDmJXSP5swEPVU6P0gzEYG0NbkEknWVDTE+NMdjWb6ddLdvJyGLo2vS5PuTenD66IGyB4qGcfOH06Eom99PLr5GrHldllNeXKdZauheRfuj7iL5QMEQcDD5Mu4Qngq22Q5lh+4orMuOA+IMGljIi4H5vJsERVW5F/lYZJLQMNzwHMnIcXaeO+u7zGfMPKkIuf86SSgG9DxJ+ZmcEqyCfjk0XbsCNAr7C2wA2cnyLqMEPEOSFHPvPMM3yL0+NeXUzM7JuA60L8JdYrdiyG2j6xts7rCFMjoQePaEtJOkadQNxe1ALkAzkQNKQWKvdotwG3nOEY6i5Tv30PtFa7Yed+LrEpqUA+n2OwHPdo6wc1DIwFMUi0QyUPbqh2GwjrrFDMuy6IKIZeoEWScsuGzhwNAdXsQWsMp0Juym98MDj/zkUISDQWffg978qVs+cunE/n0rzu53/2Z69eufbZz372Yx/9GDVj8QZGR02AAxoSamny3o2l9Xq5Qs6sYDBw6+otM5Njw0MDC0OXqswwuzgw68i81kSV5dizb+/tpYW5a3Mf/OAHDx4/WijnmNhcpgC367R5dIPstavzly9ef+DUQx98+gObGyvra5lg0DI04gOD11owNM5caaOjr9m9xkhkhGrwuMli48e3LrG+5vV5GrWqx+3IZzNYncul65HAKLll4u04VSVFAVMpIG4hn7l9fuhEHvtBvZbN5rFFVKhJatBFR2Lv/6GnSMf4a/O/mivnwe+egGcjRe1B3czRYDpZv35tfmO5lFy4fXJ2+MMPHLINGY25G05714UqhIqErMeizkmAEAOnQeY26dXQ3Q8gw9y5BfgcASqy24EBfmrHcm17oykNYWqcGI9LTlYEB+U8C/ajPZ6C+kpACX9Ik6Kc7sWGI5IxlGxwvbrRVEJzmMtWiaJMrKeshcrU+C68JBxm3QOn70EbSHgIbyHQiKTfhWKO7Nlz167HhsLkW8tmUri82m3t5VtzoNpQIIxSW6CcV4FlZAd3iI+DmPGMOhtOtANUg7/06T3WESD8tsFbqDWLoHlcKS2owESyExqDEcRM2BmRe6hJSam/ZYKSjxG3XBHpZah2RkfsxHeN3fb4yL9qdYEBhKKJiLs1KGJkYRNKIMOspki9XZbWNk1SJ7SZ2HoX98pjYle/8z6ZBlmPssm3K+ZA+8l+Z9p2DtSlO7+2j7TJlr30VQZP/qBrWweKF1EUV9oU0yrEU1FihhnmnZBf5hdVGHgSPW1PAn9J3CkSMAcoPFHrkv9OhGAJ0xUyKB0GLtVXMcfyJlAMB4QnNZlXvTtIQCylWvGWwvEVjwHcqWGYhC8QLxkYIpV7COymfK/QdDACIhAQ8wIbCqmpIasi4sLkg6eldRkiea/aI3fhZAfSh7RTxgbOLZ1OkvAFeiMqPNyCyAYpA2oAP5vsImM5STYFzqFYb6cDJaxUiyhaiRtMxNfJ4OyPRMqZfDaZctsceJoRgkXGrBp5Aa0WbB8dk+HJpz+kCw81Uslds3vmLr9DTV2H0z4yPATYIkDPTIyh5MmnMygQCFcym9wkl2Z48ULDj6rWa1cyzUDQz7HLZBAXkUYr4LEHI1Fkp0q9EXC5uoZBlhJLuRQKJL/X5jD4wn6bCXXcoIvNWCaub6gzHeIDRwBV2+50ozkA/xOXgZANiZWxsRhc/qFYbIrod+JsWdSgVHLA4mSbSKWmnGPttr5Srq8urZK5gH6iyzuwd5Yxa1QQWYT/9WOkD5A8wbN2+1YMbsLnh9v5r//tT7A7NCUhl8R/G63dVqOBUIXg6PH4EqkkanFyeQB0EFVZB4rQqnyT/MQtQ0RAuk0PoTqsT8BL7B1ADfxJvw9lRajCbqkgmPQ/g7W1OMp2zLSo5urVBmQjhfpUZxgaGXvw4UeXVzd4dTFXgNdTpcv4UhK1wKpJ7CwsJgCGbuMv//IvyRXcqMJl17xBx+Li6t69uyiOCz27efMmYiXyNO2QQ4r7UUHL29XGs2yQYX4hNGsn795zJxtLHmiEgAHGGm/Bns+5+86dY84LvZZFAiyLJ5rwmfJGqDKrhmhyinyI/h6v8lw6R6YRBozO8RaUsbRMJ+k54j5cBguTP4qHCfYzGPgQfAthLFACUeyDQUDTnkwkLCxBm5UxnJyYWLh1CxA9efrU1/70GYMHd61OLV1lDccmhkgeR5hbYb2Y86d0DcwXUm2pQe1W5gzRC84XJoSUVi4dxblJED0cGc6VcuffOhcJRPh8YkpWVzaQ8CGx2PpJkQhySCY2SQjzgaffV6uR55zwpEq5lu/rKlB3b9BTqlUnp0cJAczXyvhVc/XjH//wn/75Hw65XMl0mipYRLVevriG5/PthWXKF1bKLZPV0YDDJgupy0+ovdPsJPL+zIVzhXIxU8xiUHPYnORGh4j9q//Pr4WD0fseOX3z6jzmz7XNNX/Q1TFUFxdyPvQo9cqBqaN7I6N/+Qe/s9dlmXbXRh0ug4EAiKLXorO5dPiHMvFoiMGgmlQAQtWwzs5sftfBFtR81zkAg02d4lntYOsGDco4BzrUqIF2GZAgrbbCqbJOgBHompA2HhB3PwSVFs7qVoetXmwBrjabC+enWr3pcvqpx0LlDDQ6qCAII8Q53et1BvfuCoViBCtdv37j7bdzdrImeN1gv+RGAn6VYsB2s+PWjWvx5fUjR46JXQdixJ4XohBlCFB5osTCOxxXSqpxfPIXRo88NKuzJ6rdBOHXxD7Ad1rQshgGFKPk41il7gD5Q+CIcRaSUUFqALhZERox5oyCeKBORoflIxKx0gZtjc1d//D8zsBpT3FRRvIHDbdcEpFTnuBmmt55ZHtR39W0OpRRVQOv3qIRUY2QfdedO+1811l6oQnpSl5kRQvHAbRwt6K+zJqGB2QJaQRYHfCUlEzgWH5KJJJSoMFg4VMroVu4PWN0BwPiSgmLj58P5Bk5GOUnPxX13e7IFqionzKgMnNY9yjKEiTTdzgIEkZColAoqMxpt4JsBOHQMxl1MA3KTjxFJN0Gm/Idl/KEyESi4yasFk0yAwn6p9w8+mqychBIQblSOwpUAkiaTQP5W/GM5XmJlcwiz+mQ1GEVxaMeSbFbqRls5bHJ0Oj45PJKHFFpamYXCfM2N1O7d8+kNsmQl0TdeunsuVI+5zObsovL5VQWBTrZ8Mmt4/NHCPQ1UqSlQyIe3/Lq+tJLbyaW53/+H//90dHhaiGD0x9oMTI8du36jXvvOUG0g9OkwzRq6bcIBqDr5Ogg8RZKWzN5RVQtCzQMBqMFhXQLZXKliihTk+BDkzfoBHkNDflSG3Gvi1CgttGmX12aQ0CGfOP7isoAzwuhs1achXS1qjhbM9ni60YZH+aLDEA4eHtw/SXhgD+Tra9vJLo949Ruv8MR6NTLwdAIznTVCsU9Xdjgi+V8LChW4n6j2a434QOmp3encpVyqZavxKmgMjE2ycQ3M9nZyQlA22dHaG8gvLK6hJHqtnBSIte71xeoNVrlSlElRFN2BJzlVGQtxJUN92xJHU+Ym1oUon+R+BvEaAZPTL9MmdSWRtOivOegMdevXoNwihLCQtHJ/ujoqPBVQrIpTW9FrsV1Gbwvhgy1CcyrZUlrEE7COLmNeDDAhRYcURN6eA5AspgoJYmP1YpgDdQhBNMOw0hTVCqkJ9pPDmQhbaMLDr5n4yrXeR3neYROsvE9bHwIndIev/sp2pTOSoydXOWprQcbQryh9bBtzVqdF3uoQtBuYwKFAYWysoigZzhsc5DHlaEJ3WWKzTyOCgQhFQMwTBhtDg8Pr66vvPP62ZW1VbydYdVIr0i9w//+mc9MTk5nkilKEv3kT/7k1/78GZzVjE4bbDUrCxmylqt5Qu4WhekwQzjNTXwcBl0KfMO2dnB2AEVSEtOgIy0TlkIcyN/9+BNEHL3+nVfLjerJ+0+NTY2tJ1YpRTZ/63ohDxNAasiBLtL7N//2XxNJxWofnxg9fuLA177+ZQJ5XT47mZxHxqZtDn+x3Jid3fX6m+ejMf/bZ85PTu1JpVOxoXFYLqpbHzq0l8IJpVLDbsM65D18+CiFbK9dfmdgssEjxtNpq80Wj6+jYMvm09l0de/eyXKzYrWZ1pN4NlSHwyMPPXLv1MSuP/j9PyyUq7Fx+/h0WNd34uphsRqfes9Tufnbb7995elPfWTEmc0vvkBFRTLoSBLOpk7q2wrToZC5oDYQ3N+A9++e4x90/AOBAeBh6bHXWt2SwkDp0F0oH+gPIqJeyLSKWCpoUwgwYUsAGKx+DfO5BR+qEEo40q2MUdK01168fQuvKyIYr1y6MDQcnZiaJCoErf5jjz1G+MDS8gJMp9vDgNlRBVFO2GpC5jfms0VMR/oftntqDSI4LR67EXYMk6Hfy1h1XAHbrfXGQ0+Zf/5f/Vi9f7nr2LCGKNxat2PW1ZgDibyHi2TI+BAD2T65IL2/I27yAQQjcQpqzBcp3kb28uFAl3wkT7CpY410ie+u0mttn5fxYtPGRdpWm3ZSO68R1K0L8o9qf+uOLZJMazKdbKohaZxm1WRoNzJhXKRHaq+uq/eK7KkdqIeFmEkfVFPY6+W3fMIWV41oC2JHYN2iu4oLEVWzUG5osKidCdEQYoxMI9y0rg2RINAWhTNXyJerxxEB32TR7VTQ7BLjiDrVBKrE5M6LxeGcNUkJXxyBzJjQujhuwGvDcYO+CbdttRpgDar9UZwgEPSIPV44fPFX5WYEa+kMZBUzrajk+GaJMmLwt1y9hPMWl3YWM39QXTI0oOuQvVHvx+TrtKMutYgb7aBIrEm9iZMRZQwaHeoY2lBywiDik0zF3IOHj5w/d5HgmeEhIu89SFekljp+7CDxOei7UmtrCP7kACilsyEnKX8MGJNGJmeaTITbrXe7u1bz+3/8Z668fQ3BMeS21vJJS48wmRLqVWSOUrkSjsZgPFgY/oAXR7N8mlrgcYvZScc9TnOLPFwOS7dVcdooP1x3uB0E14aHRm8uLeFGRSy8Fe/srs7rD0F79u+dvX3jKpkFiM+npCAJnkN+/+ryCisHetZoNOEMQItVCUUwka9geDQ8NBapN6tYu5GBao1ettweGdu1Gs8Rx+MPhBEZMeZFR4aNbjO+nasLNx1G0Tn3W7VY0MP0Q8IdHkKX8NwyU0tlavYg3tqFQi7mJ920cWNlze52f+Vrz372j18YGiMIuFuq6aZmJ9K5ssHmaCBwdXr5YpkP79SqOOPizAeTRDp4oZ2KzEA73ao+FUk5oYjIqUApBk68ZoLBMPQSFXGJRP6MNsGySsnMbaL8wI5gMgFCXAL+ilXRkdosdqF16NYh510hnOhQ2DPUNAXgMgs8ApFGCYm2FlIkVNlmRZAVjxUviuuWnMEgIV7UsirFp5xEZ03imui1kHx+yktEtwzJFzn4B25quckipEFIKA2y1wgwfWbjEvdomwC+2naa4pc8LOy6sCp8Pme01pCKOaYb/IRdgFGQMcGvgl5qG5yriNNbb4FVJSVyp9FxBj14wvtGwg8/+sDZC+eTS3FH1IXWElQJ84G7gAfzh85QTmf83lBhM0s1T+k2Mf+EKnk8+XQJU4EwM4QtRJ2lbM3sk6FC0W+KYOojWaoRl1ixgPTbKJzQjlkcll/79V9L5JLXb15dT65T3zeby5XXyvT86ANTJ44duHb9wu7dY4Gg+8r1i3BNOCLo8VawuYdHp8ig4/WFS5XW9etXh0bc5DmlVg9TUCvXVlcy2BzJ9UQRD0JhJ8fFXSOfz6JNQ6wKhn25QmZq1xQjw/g898WXYRGCfqvdiktZDCmcJO+EEe+f3eeyeZ798tfiiQ0r8Xhm08Z64l0PPB52TWze3Dw4vLe+fnvlnWfv32f+5NOT7exlTN4Ovr0BkrGDC0lJvD3AfI0QFOaHudKmSZ0SwUP9bZ3kfq5qqBj6oh2rR0XTA3ID3yLn0E67prO7ddUWynSdxafrOXWxGadj1EnAZs8kRlXhX9HukZ2LQsG6sVb/uNVxtLhW/KPf/fPkcllXt3aa1M7qof+yumzEXhWZrTJK6VbQ68XPP08VSWIg+/pweAhHLdjZz33uc5HhsMsNwkZDKLydmey4Lp9o6ci1An89HB2qlbPleoMkvnickNqzVNVVio2pg7of/onHBoNNu69fM1D+tO4N6nqU42BM+H7ZKyaFQzC6CNL8QZzVXkiltoRkPfzATR6/a5PGZCHc2bRh1X5r92qPyEhvP7r9r9y13aCc4x7twb9l/wNvUZN359lt6rvdplwREVa7bYv6CgPBn+KkpLKChP/uSMBbgi+yEn+ol9kLSWYuEEpE5yxGTDKCQ5WhwSrkFykZQZUZw94KZoGmgmOio1Hi5RFypAcSsCHGM/x+K6UqTDqiKmQG9IUg2mk3qCfvcdvg44E96iqB1HiKTmKqx3zFpgQ5vkS+T32MdB6hAqMnGgr0FPxWdykfBfIwIBCL943o0LkK2sMZCp6OWg+QQxqU474+Eo7tgWUmD2W5ih8s0cnT07tshOEmkvANAZ9PfAWJqSxX6sWiqKhlKgcQFAxk41PTyXwJBTFGMXMgsPvYsfxmhniYqxcu3Hfk/k7Y/dZL3xiLhaqVQsTnDPs96UwO8Y064RR+WF9fzSTjXqfbZPWiJUaEXS0WeqU2Kl+y+leqTW8wlEwVcjXyeFBnyau32tC1s6wTiXWnHd3jRiabIEjJ66Zkbz9fzPvcTjTFdvlq9PNwMKLbIVyQdWdvU7yJaCaxhTIKmBLzxUap1FldOzM2uQe5+dTp+3LF0ti+MR1lB8P+3JuvwT/t2bPLYehdufA2t8bCEUgPYQyu0TEq1rtruvWNpMsfdjo8adKJYGPL56kqnE5usoQQhnAUwCepXsqTQKpaKRKmKiYiHOK6UrxFlhyOYWoDMAAVFBiEUaMfpocqUbbUIOI6YIAUCxrmJzADpQHGuJlpBS+AgmUdq1XBnQA55IGf0E4xlQnhEWoHAQYmYWcg1VTOYZP3KRIIgy8dUIpo7s9tlp04xRHQqsiktAx3ygJQ/DiyO/Bmt1uh3AI8YldusAfjI2vyagHTuzZ5XBFd9ryFn2z0XCPbd934XYfcwG3aKZ7SngVDw7LAAzAmAuZCjhWfzZ24MIHFWJqqP9wg9m1FkmURbfeB89IH0npgXbCa8W90h7xDkSjVHYI+P5X9WAB1h9Pr8SEfq9i1Hj5cWDIKyWxwOExTJDfl1bCs8JIOl0VeQYhus1et1I12Q6fcsYdsRJt1a4JWxX5lJi1Ai75ILOqA8C3Pv/zVfxkajdz/8P0Me9/opGrRxXfOEfTw6KMPTU+PNJpg9FLudqZSpkS3hdoDBMOEw05i/R66/5GL71zvNXrjw1M/+dOf/OJXP18hDK7RNRm9jzx0OJ0qvvnShdlDByPhESJoUFrg0jgwtooU2hvDmlRAkka8k3l36UIhO1/tsNtJtAmX5p8M3Lx+a9fUeIZSS3Yzhpi1RHLPvllkaL2h/eT7H50LL96z6/5nPvtH7//oT0UtaxduvrwrLKYSuxljNyl3ekZxvttCVhqSBw7V3Mg0/b/fCDlDbjESMiXOhKI+FA8b4hckExyzKyLJHdYPBlPAoYfBAUDrkdZEZ0HRi8NloVQwNnCDqZJTy4PHptlLQ/k8Gbx1cDO478C0UfubdUMaV0Ii0pnC1NQYvGuzXaUmKRnsKxV4U9IfYeqn3AxpTdyWBmDX7gZCOp9XV2rqnvjIVOCeWCN3gcy+fUODHgsToXSZgB+yL91TQqQiPhpwi38HwyfntzcF4Ns/dv7VltDde+2SOrN1l1AHsfAqsFdWHFnBsgmPI7pnwYl33rS90OSNDITaUC7wn7pJ0VJO7yxIuW37ae3kXT/VJTG4qoa05uR7NTYBOiqCJPSIRxStFb5aboYAK25LkSouielNRF4huhLIqw74qUnAygVarkI1WWRbSZ7FAQqPKtIsqgEnsBX3VzawVa1V0XBKm9g8YvJQa4K2Wg2zw9ls1WBFSfEHwfZ4SMgYSmeSZWoCoHsUPCUcAvAkFE8OmUEO1BBL8hMGSlNNSLoGmV+5jUHnpHwzcIIJjRHjw/E65lFOkBmHrBfpfIFIUYvZUiEFZb8P02dzOuvN1UQyhRYaZvnC2XNkvEc82rtrWtcuA4QeO2VPg11iLVod8aI1UcI6Znf6piOjBqcvunt2+MChYqNx7vzFcWqNjo1cunhhKob4bc2nN+EhiHMCaMm6LIVnkQbM+onRGDm+EOJLDSBGfBCIQiJjCPHHTASSdjpH0bZBrzkgFrhaa7usLnQFKJIpu+ZxoahvYqeBANttVp3DTMxRKp/H1iWKGpQU4vEs+QEoY0wWbYzf8E2IReAFCyLwABGQxHx60t/gyFghHCUQaGwmNhfnyRlkqZfpbzgQQalP9gW7w0MS0LVECg91Ut9K4DD+OyPjRcmqayeZUcQXMOAcOTRMdOmxQ4dAtpuZzOJKwenQZTYrnqC9hm3coLe5LISr8KlCEZkwJoW+4vBMD1AwSxLtbrVWZ37xn4L+SdJ5JNp6c2hkOJ3KItNTjYPMG4j4tUqdS8ASubC4k9T+rGsoLgwipcvtNsLHhfUCHoi4YM8VBU89GypFKjGDZIQwAardeg+2qkiXqMQHJBz+4aNEYhTLpS996UuAFGfwhFALjRfKsgLMILecYXi5yMu5jZ6zcYO6U1rmTm0vPblr4yRPsQcmeeldV+T+nY3zvG0LP6mm5O0MjcIF8iJB8YJZ+IfvJ1sez9IHSK/WT34qnkFwCX1m07rK7XTX5ZTAMCK8SQ938+p1FAZANUOFZhv3Cio31YlypTCO1e4fj8K3oWMZHR4+ffrk22++Va9W+QJioNSKk1SMJEB0ht21ellIAWw1/6CEg0/ANCyBEypxD0lmbiS9M/5MQqKQcPK6tXgL3/xf+LlfIKcaSTBffe0NKg6dOvnk8tLc66+/srK8Cnh7fSG7xZdNFJ+Z//L9D7z7wO7IhYvvvPrKhfc8+uHLly4CD+FAqFysHNnvgW3MZkr5VG1mds/J0w8zwKnM5vUbl7Ei+cnWUi8O7L1SvhALO6dnpnweL/xTs1rLpbMBt7dUyL7+ysvI6fgRsa6dVu/bb54bHfU8+9zzuWTuZ3/sn3z+K18amZrtOzrj+4YX37nmHjLnFxfwYotYnf1aVcIgRQsqPJZMiIbwlZbx7vnVjmU0/hc3wITlgdJHxIuuJHDEYCJQyRBLBJKop5XTq+R0YsN0AdaGfUUvQ+yHTW/HSQJxFuUi01HPl/AP93mcBAvgs0WWjkq9Ao4lbB8Bt1yr9w1lLAoSrqc3kWsPRxli8FlMmWyhXm/GosOYivSpcjnotFDntVyr2j3oRPHP0p14VPfgh++pZd6xemu1TtZg7wYDNhwpSKskZlxCUiTTEitAGwScb9SCEQsro8bZv+vAcP/OvTvHcnBXEzSpmgUQtfO8QiZH9to/O+tKnbz7WXXizk67dFfbfMOdqxxxA2e0k9IzcJoiPHJaBF8RcLePuU3S83JGLLuC/fgtaX1FuyuLRdFdRX2hWRoBhgZzs4o1EggQE68y+jLrPCc5NJQPOVEe0AK7nXAGq8/jIXszjiZYo7iOh5SqVwiKQFkn7Dk+y9VqSYyAVmaXhH8KjHU9rBSIKPRD7tHjDSv4DuQBplYiDbAGawLEA2ziQ0cCM8g0Y4zWWwZW2QIIDxTvaSgFBjNJ7QBWkIyeak74kI54s2AQBYOTma9IdujbDnTIej0hKEAtpbhu3rxx4vhxGAis1CP+GJ7J5Bii6+B+u8nmD/o85Li0AWeI7/hetfT52jCFeL2+8am92bUV6hvOLy/aBiNg8/xmyWUnTSNCf2NoeJRBQ1Bs1WsE6oHEE5tpuzuCc6ZE6pKNw2Ynnz4KW37GN/PQHuJgSW9pHLT3H5myZNIbG3EXAm+nTfoh5DCGciOVkoJ6eqtYFCl7LJp7ksSiPxJPcvh0ptzsgkyTSIHespRFSY/yLeD1VlsgciN87saVS8OjwwanDZV7NVcl+bZp4F+Lr9aKOQbWHx3Cvw5Zxgmr1B1UM/ng0Jg96MJce3uBchWk/sK+aCYHBDmon3jsPeRe5uz80urzL19H/U6iAnLDSq4rljLqEFQQapNkGPhGSVo6nGlwuaWQVD+AMdDnhUsDkKFVFUoK4jGm9L3MIHSLDfCQ9Sy0VTZZAgCZKE1QeAxIvkU73MBT3IXeGNEQcCKaKOQPkVgREl7IFYVBEdO4wWV1YFjmXZQD8gXxcnOdOHGCe7753Ddkmcny3eow0jQnaJnu0SYkDTzOh9M4uAl/ezrDzfLU9sZPWuaX6un2yldXtfO0tvUJ23cKbN45yVLdkgc0lKLdLA3yHtWedp1HuFOxC4Jh5DbpOEOj/pVj6Rh71gV90rRFhPNmNnIVdxlrrtkttYnwGKeZNuX2nLaTx0/c+8D93/rWt25ev0GwFnUJYZHj9TrK5CZZ11mggy6GEqg1zn36UXM1n9dZWYtMkOTI62AhRrkKSSeej3VqRjhuNTv1F7/+LeqhkLJ7YtdUoZipt8oHD+/de/BQsZDZSKRIsHlg//FauQObla6VWjUrSrFz5xdnhvcP+SZnJvbSxQtvXadufDnbIgho/vry89968cD+Y9Nje67dmPc5Ao89+l7Uc1dvXEIATKXdu6Zii7fnqHW2uV6Dah466CTAbHbXbhwa/uJP/+zbt5bCvgi8qdcVIC7/0js3mi3j2Oj46uXVsUP2G7eu/sF///1K0pK4ufbkgyfn5pKllZXxkf3ecLCJQYQ0eU507KWdedUGWRtn7aQGDmo2ZNY5+F/d8GokwQFcDZMG0yp57m0u2CQshNgTWL9C0pDqhB7LfDO55GcgykToMnjTiE9E02jyeb1+7Ahmpy6ZFCkZ1Ip0RN76NrK1sl+4qMdqdFK5EK/S+GaWRNOlKjkO9KVqK5kqwQSQpJPcfeSCbtlU6el0NmN3GwNDzpWNsj2oe/+P3qcLt0pLSzHcqElcyWIxuInUJsJNUV/YegWA0ByRqSDK8hM4VqMm1/6nQyNPba8xDraaown5dkB7R/blJ41pDWqPiNcJLD9nVT92XqURfm1atu6Xa5rpWEjpnY02tencOaW9RTuvXdL6tOV0LekkFe2F1ordnh6p1UE/5IzaMwKiPYamav5WmuyLfynWXzH9KroLeYYiatVu0O+hDGO1g0+JehFDrAAHRYScqHcCZF1A5QTjVK3mkXiglVYrlZbBpkJ7eQdDBR5DwkEXjSYExR1hnSjBkErW43GoB8orgIkIGtwIkBLwmKMEALgVHKe8rrSYYFE0QGWlVC3cPbwV8MYmHUWvLEYv0CJ6RjF8uJxo6SCKSMAoc9CdErxPlG+t0UQtBs7GT2R6926QKV9MfzbWV++7955HH30UV2QSs1UyjXwqUaqWoZDUARbHqz4xbb1WtXr85IOFSpf6BYWe4dZCfPbUfbXaLVxV1hevnzh2olvJJteShMzSJl5UhEZgXxzActgcyOzxRBLZAl0Z4jnvGXTQ/7BArMIVdAeSbL5n4Is7tSbqgKlg2D85xfetLi50iGDHK6Y3YEUhaRgtHoQvFBBoW+HgcVMmzQheMGa9jRQfAzGwIggSnGtmhGFi4PTFhEAR7GZ3z54DK+sJvLfIiGCwGzIbi+HRofm5m1RxHI5GYuPHWoXcyuJiKok2nuwrA4/JgnI2Xy3in4YLLElFUAYMTY7CYukqJSypazj1KAPnNHkQ/L4z56/D3HndZoPNWUd0JsiVRDodZFPJuYHAii0YgR3SKdUCyMdpMYeCYWTfZkNUl4w3FHFtZT0UjCClwZrUKhXKsKOxECKhcsID0kJ1AF+wEygKP6l6AyjCEQEbOIuCANk2CbCBKsgyYwpjL0YL3EjhRaChYppAtXDh/LXds+PLy8uk6f7iF7/4+uuvIycJFwiAw2hoS1PtGQqILnumg0sCdCIKyzJnYbLnDAfangOR8uWkXNI2bf2K8YbzIE+1yrmfexAeWS8wK5JkRrGhrByN0spVIZ+i1JEjYQtEs0EfWADyn1S/YBBYyfIESI+uyZ2sfSR+Bly0DjhkwY5UWA1ENPEVWI2xz9Mk+FyUyjp8diyEDvNV3JDeTD70yIMTU+PpjU3qEsI/4YNdo3YvLowUxNVhZA3gUU8dw30ze869+BoEWBIF0AVS0GB2p84YfIlJoTNq6q3Xg3sD/lgAUPe4nQGf//byQng0gObGQXEGyDNJvFFS1Xr5dAPNHEZtc9d54eKliMf79svnlucSJlwhwkG70xEMePOZ9KU3vl0t10Lu2PrtuIMyQANzJpGdvz5P1tX5azcun784OhZ97NHHjxw8RP6Q/fvmrl29NDo00ak3C5nc/j377zv14Je/8JV4KU4xnqK9GosMEapvsfgo/xCYwMMDryWLwWYIjARgkc/M3Tg64zt6/OTtRHnYYhhzhqn9Xa1UwFNK4NGm+K6pV8KAYpSYBI30MtP8bXFUMo//0432AEHFLylKTBEIs56MyowO8UZ63s2cimghfRAaDEjx08gwInagcmLdovcxGLsUpvFbg06XzetzOkidYrMCHVh8wHvkIMIlI6LX+wJ+tHoT07Ok+y5V6za3jnTuzZ5xI7mIH4bd7Ytv5hBcSOoBbmqQphJ+NZ4uoef+J586PH54JBV/a2jKUWkQSQlbqusUy06ixuiXcJWKJRSCBNhADoVmAvoC/QpGOeCEBus/cFhktTCQavvuYxnfnU21s/NLDrSndp69+5paVHLL3Se/5/juxuVWde/OSa2FnZPyE5IkC08QhDBDTJ6yBPF1SvpX4q9YfIW5ljGA7EBE1V7WjqiakU3Q1gr/pORgDpQ6WlTTOpJTAUE8C/XlJxvelYxnLBIVtVO92iGOm+YgelgNDQOKOAvqA/2LFVkGGcQBoCA8wIzUW5I5LxgJcjZRKYMiSNwKEqd9dLY8yHmOeRbFC1HmmJW0wYLtkYakqi5cmWBS5ClSg0CCZapNBGmIkYfADNg8cA/vBVGDUiHJHpwOiC4iCbMXryIjuinsi2HqlDaN6D8JSDjwxGMkrZTKqc3qlfm5drUSpgCN04pnELHFlCqcGN8dGZnRhccNa9mbS6mm0T59YBdIx+EJAJeHjhy95/ihzOotp6mbWrtNF8OhKAYuPJxB/0RB4HSrb3eslDINhPOZXMDjRFAjzSxGKeElTOQirqEJJ00B2BDjKfJWdu7mZiolk4WMoseuM5B7nJ6Z2d2VGtleJW6VgWh1KUlcMzop5ANlkowlpNuS0FootuJVYPN1xh71N8kKRPvDsSFPKLAUX8gVN6xO08bm7X0HD2XSCLFJS0XKE5GrIxAJI9z7gsimAb5w1OmhBBKh/fDKE7umi6ur6H9rlTI88ujQCAY/1JvVYsGFOd2iyxd7zqCiKNRwFC93G4p1FhJzpzYBfYElSiuiPCFXPtJksVGpVcHpIa9nzBfEO4YwMAJjmGK3G31aGxM+tJnHgXDh8gEaSK5AJVhIjwq6Uq/C9GBzRnvCeTYkXWg/ci2imzohAUgAHg0wtvBhZC0AE12bm0f5zEhhMuQGXkGr0k+h3ywKeRUSO0ACDcamCEdFh7kToOKadAAMIjhla88B60zObuMBaU1tPKid1/byLWyCLRgcuiZ940btKgdygcvqBu0YNp1FwPrW7lInZXForaDlUMfydj6ZPY2wDLB/o4RyOl3k1WRMPG4Xtl4WK3ECVSaPnHR6lLEURGlfvXhpfuFmZCR6+r57f/nTv0yU39uvv/H7v/dfILxWj4we6MPuctoIvrOYRyfGz0c8ovyEioM28J0ECcEHiHKLdSr0YeAa5JbyuVL+wIn9tUoVDPJLv/RLn//aX125fu3oocOTEzMUOHnn3Plitmi1+EidSHB7t1YatPTFWqlXM6zc3CDh9fH77jt8/NhXP/fs5MTI8QMnI6HwY+967Ktffe4Ln/8iXn4Os/PyhYt4O9+4dSWT32T9vvHqW2izR0aGThw7tbkez6byN+eu59KpD33gh+45frpeaD3zV98As2QWC8VIORoduTm3QF+rxb4FdRLIzWWL+GMraxulbNE/Ov3Uxx5avfBsv74Me9zpFklwIo69jLgaXpmh7U071k5sT6NM3s7x9o3/k39hrBFIYBdFO4lsAp9tdXbbFVTRuKAjbDC9tApggObpCUhAkrGXs5yHEccog/5Zb4DLbxTKUtgNXoSMboJY8GaFBMCVqgBNBNpUrjY+sefEPaf3Hzry+luvN1oNty+y79DRyek1LOuU4bI6XFJqC41RvV31Bzx14jd6ug9+bOrhH/twu/hyS5fXu5wdjL+GLouestW+YJhKmgLHQDLpE4EKUU9xAwtLfTljzCfIZ7DxFSIXK45GnfhBO1kM28MtLQvXIW3dPfqc2JoAWpNXbzUk8yErS36y42DreItF2rpt65+tXqmb1SPa+Z1H+Kkdawdaa7Su7IDyL+tOGGLhgzmgE0iuQlZlL7w7dyCAiPJZbL3cCY3csvuKvAq9FMKJRlpopzDIioKKrCG3adSXPHIOB5IhE1ohB2uj7qQUoM+HAIJHDU7AoEFWIAuQbuANiTpQSLsQRDIjlkETqBy5ASDC6Vccp1E5Wu3aIINGFVAJGkVCojMqIT0ctkQJQ8YJf8UFV0WpYA4xURmUatrQY94ufSAHtMLUYsmQRJZdHEZcdheSEGpbELooEm1w+4NcJouTjggeEvfmT23GCfAnrKVfKyPARwPegdlIgn7kUskcFQ5F7jmeunQre2PF6R0qVCojs1OxA4fW5pfJnFWp16dHQxcuvNNrFOwAYY0qvmlcvs0WMLwbrpFsl0jBfCCZu+rxOCVoyDdCAki0vKRRYHzgK8RlV0Kq9Yi5cLsUmSGjELNls4gTDfNH/i8y8DdaldCQjoAOs81jN2FiBtRMVbCG0A7Gpk90aKuBphkeiJULS+IR9hPO2WIl2orE+miji7Wi1019CAxAFbfXnUysMDK4J1fKGIcMgWgQRzg6w5iXG41cAgUWagkPD6bSG+V8ztruYq1lggj8RzuxvLSIur1bLkeGrUNRH47LWK0wIzKPkndbU1QIRwiRgXIJCNJXViOfzKIk+gtbsoJUPTz2wDu4//77X3311Wy2gNiJ6EMO0YYEE7bNJotarjQJbhITrPBjRh0iLIJ4LpdvNZpiokRdI+y2LFEkLFR1Ir+CzCCl4htPMJKeTJZI2Kigp6cnyEXAWwAMQRWgsy1+T1Ye3wjYc0muqpAkJoszqFd5NwSYPT929tqBRkoF06m1Km0qDTNXdzZpfWujPBdzRS+lKe0RbuMRimGr+0UyFgabdtQFFPRwDFr7IAxgWPHeqCCF5EnS7e0uyYOSpgN/YUFPqEAwJUm9DHmVrkUxK8YV41CLctdUi2ZdoLrCmJp+5s8+d+XK5XvvOU0CcWofYUzBAcrmtGA7qGGSb9TtbicONkeOHV64PjcoV8UqT4ItSZSEzGwWcQ2qbMBPTad36chovrRw24DDoNNBmLXb608VMrAwqEpd7sC+fUfeOXvxyuJNp9WxcaWocxQnJ8dW5tdx9+8WO7NHDoU9kb/+ky+02jWS6Vh75rQnkV5N/pOf/iVda/DFL32lViyuLy5fuXaxNWiMTURYd+fevgB7NDlFlQXjyuIGJTRj0ZFmtfXi8680yi1cvYajwfGhieEnRt5+80wivjk+GnV7XdcvL5psFCO3w9MnCvGBc+A0+F48c240YJ1yunePHqxvnG+WcxMhB96YCrhkDLc34ZYULMh4yibjzT9blEKd+l/YyRIRRoYmSDdg09mdlGkFK8PdcEbERlaC+BYBNyL+Qt4I80WBRz/Ia4T1t95AmYxqEh/ECouR9Of4NjrMEjVO7V7iNim5jmYkmUkYLd49B4+6/YFkoZAiBNNkGRmfIu6r8+1vL99eHRubIMBMitwRH+AKWFfT5XseDPzCpz5ZXrtY1S2Pz0SKuVUHnAtwht85qhXy1EBJBMJkobCC1EAoCssqV6MDoEIgt84LqVTwePf4AMRKGyyAqzZ+8Sg/t54CgWhjq8bi7ke1G+S0eheXtAP2O3/aye22735ajrefE9H2B2zb7Wzdyk/mRe6UVSq6aFEQq7eKs7uooOGYhUKzNAUpqDxWYgAW4io6Z0VZEYVRv6m4XhltIcCY2SDbNAWHhVZUHbKoUO1SjI9kGvG1dcoPkT9lKBomLw/UFxsMiANdieIR9US5UBC7UCzXyEOGctRiqLT6sajfF3LnsmlcAyDkMPTkS2O4IJzgF3z5YK4xWxLECJntGMXQKcVoRbwT9xPzAD0QYb6SywHkQzVwZgVXbPg79iB4MTpub4iDEAM+lFB/Oz12OjH9uRxW96BXq1bw4I3GIqVcfu/0dG4jgbhWTWeaxZyZ2mMEWpmIbjSOj46g1M6l0+Y33mpTSadPzqyE2xtzOT3JmwuQ2OHhGddTP/Ty818Nu3ElJJqo9vB7n379Oy/wXpzUSKkLloEjoYghXWuTBY4EDqk0tjeihuAA6GCxlIElpSkMG0h7DDieFD4K/GHKRddnxHPKAxEKBkKkEVnbzKxtpCl7fPjIMbKoQ1aJmyW+tNppG8nvQeUDXM6cHipy45UFCiSug+DaVrfObSQycWJed1sI9Lpx8+bk7LjT7azikQFrQhEIh5vz4FZdk3xdZCDyZPKFoWEJgcXAoDNYKIlOdsN6sUKOZVAqOi3YNzgwnn3w4Ueix45dev75g/sPpItvE3bcKBRRShCBJblGIH0CUaxE8emFjWc1CkQSL6Ub1FUQGlpo3MpIPViqFDGEHzl+DEnrnXfeya/ncIv1un0IwQrHCWUCibCHfgMb8PKT41OYezOpPAXUIMnyLnSuvEylgVTWcRtugPUKuUkpOmXjk3EPwzPQ5XaxZqjYg5sgsFHI5mgQ0BJZT4mk/IvUx1MUeGYc+ASYALtNsmOKLVtkPKHBbNy3DXRYW6BFGvKhy4KCaIQbdu7kQC1XoaIgFZhUGBIAnPMsWBkc2SRkWjugFY0Ay0+DDr6OsAMU8oLi5L08CF/DscL60gIcF8+qcWaB9bt8Eqwtt/I4442NhvsrtaaTkiAG4uyo94XHu04PzYdHsVhmjuxNbia/+fy3XDZSmhMAiwSMYUXymuFE3SzWjOFIq1Y/vHd/cmk1m8nBBUsn+Wg4MtT/5HhTsY72gJkqtpSMBMWOHhiu1suvv/7qL3zq5/7iCxkiFOECGRhqLqVTBe7PlasHTu/zOrxvfuNtk8fcKrV1RV0937AbHNWr8fDRKRJKvPbyayMjUTzw11aWP/3pX11bXzx/4UKjlSdEfiOZXr5WIhDO7vUXUhmCBtFYxG+lm8XO4SMHq1lS/rT+/DNfh2nH8T+9mfsP/+7j4Bz4v3PnzsU31vYcHBseHf+l//OXV9bjf/3lZ1udPFVn9k1PEKp/5sr13O3ysaHBlD/caOf5FhCq6PTkc7emnYnmBGOg0WYxNGxBxPYdO/Ch2RR2fn7fgbZScFjjRl6DlxiJAw09N6EYtKX0XAw0L6MXLCCWIffpGJMK+cQ6fcwrIA0gBMnXI36blNeAKPRzUrRSYgSQUsikYPQYg1QxMju9PrxwSvHkKk9MTY9jOyPb2oF9ByG96U2Cu3qmSk9nd5n7xk6pm8Gt/OM/cb8uXNJnkkEngUc5fET5agC0327Y3Q6EB/Axa1vUkwTQCMiKGyd4TbGG8rkaYZOngHwlr4qu9i4yLEPLYKpVxIBK80KAFclF4y6X5W5aEErHEf8I1PODRSnnmRr5IRvLTP2rnZdH5QprcJsAa7fRUR6UY2mWqxxqC4oDfvMLbl/a5br6E5CXS6KTgnYK/87DLEWc4mTFSq/kT9FgSgqBAMFNwjlQNbOHuyxNIsDRZSqXEegPPXZ7XODbKonsnS74rXqzTOeL+a7dDoE0eMwUrve4nA50v7gH+X22kZFhgm7tNgv4i/S/hQzOxuRtmMaFnY0S2egEccLKl/AzrZApMJ/NLt5eHhgawajLqCMK1JtO5YnPw/eVz6dIHusQjSyYh++3Uf0GgiFxjKT1lU9CvkGUwqUT8KIoBHUEcFbC64fKL254RCleRw01A9gc2RN1NMgAVVux1rR7naVqsd5ueQJB8JrZOEAL7bKZs+tN3JUamTSZZKvZap1SpRJV2SbQBzkYYR3LBy590PdyctNsj1Dh/qHHn0Ts1rmCl6/fsnmCVPJZWU8+8viHMolltHipeOfyUmJ07+FmJV8sZIvZDIvcarGUS1mfL4DvEeGzBOWYrXZMk6h1RCzDzOULonGFdaA1IrYg1tALB2WRRUUB/Xa47U4SULT6pnwJB63Sg48+5tmzz7u0dOns2/HFVQ8phHDiovSQwKnJHYhtLi0/8tj7L7z9dqmM77RfvJd0LXDkzStvYnYeGh/WtysYaglgqrX6dq8rNLFL5/O10tco2YgIi2c4DpWhkCuxsb7nyPH8RhKjLcaxqYm9/XD9tW99oxDiw6HZdtISoRhPpYtRM6mvI25/aHpq92tnbrr9JiKvMLpTKKkHHcYkITC6ZS+Q/uCDOxhUCgUCGvEdgyOXy7rOrn2zf/2VZ3bPzCKaAwdmmxjp0eFT/weJYCg2gvpU1rSJiG4HgcL4dn7rWy+As6DLcC8kZ6BhsAxvINAcawR8mSBGwsSJrOi1k+kETmmI+EJWiahuN+HajAREtZriQA0a6JN2uhMK+TOZgt0qLaFSQvBFR51Ok8EDqCDTdXFqapTlUyyiDamCIUBqdAqohdDSGYAf/oD1iFegNKHIMyScnrNo2Wsb5/lFKhp1wDHrW5AABzLvJJEWcVy4FuGH+32aovA5uhXS6qM+ARdI51H/WO1mG9ZBAzYLzCKkn0ERIunDUL8QlUI8t8NWKJTtNgAaNNUnZX+xXAAuYVkIaoIdaVXaevzwIOpC022LF2/ic1eqlTO5pAEXdSp2U1PYgnKlzUr3UpZjJXG+KO5yyfgmbeot0G6q1TKcwmxZHVav24rDOdVz7T5rTyaxOzEz/sT7H3/tzCtn33rx9JG9OH+ZBp0b1+Y7DTyLq7Uii86ez5Sb5t6/+Le//qXPf/nqW++g4okvxOO3v6DzeYqpEuYaWPZ8IcuEvPLm8yv/eC4cDg1PukgMns0W8epFZ1bY3Gzni2Q8j99YxGvKqSd7iOPamwulIoECXYq/g1TpIcqv3/wv/wGMNDO9u/tO1+l11Uuti0uXvhL64tSuifXrV8yDNmm7/t4vfiR7ezF+q7KWThwcGiWzuNR8IbMN3n5MkhShlrBJ8LkkiAZlI6GCpUU0BfNySt7F+PBWoGF7fhGDZKLVVfGn5ryQIfUHisZaD6YyORA/unBF/okJXYFYf8zo5H2EkyZhDo1b0XXo+zadwaWzOLuVxuJ6IlOq4ajMAie42WJ0Yr9rV1rgH0pTAPmYbUTjBEwOQNi2akHoNS6laytXc7nlkbHxoJdYtTwuzK0qxdQNTz3xeCVffu6550wUsWz0KYOsy1V1jz1pm9zj1vUSPV0BFgdSw7cJTQJJA9xkyBLCCEWiiyKb00eGSn3unZ2QOq5vbfLA9jFETnExO7+3D7Tzig7uEGp5Spq687TMriLId59SzQsB/cHbzuPb84NmVTrH/zIxcsgEyvPavGkzyc3yiXJWyDWLiptZzErGhQtlAcs3Mi6oADgGJyAQwJpyGwFdymxqgjSKo5MwUGa8Zsn5kC9WbVYn+tJViimjcnCZ68XOE++9F/UFcXJQBXTAqKyAP5PXJekkDZhWC4Vil/WPmq41kOCZpdUV3A7DkQicAEaaVqdBdPzM7kk8hH1+VjN960h2Ux01gAN+j7uQExdTQjmtdovbTa0/J6ONzEo2Cb4bgwjJIeCiJC5SfJtNjVYddAQA0pLyf+7jdkvZI1yzxZULXEhaHjpDEgbxv+4iUvdMOqiyxeXCLdlBtEO5jl/P+OgYdqlkPI6eDXkKcdnJPf4AFRowkYCOwfj9dge7NJ/ZM9jz5XRsPDx3/p3g6MzmrfUj9z6wSIK31OaBA4fIM40cQzxxNpto11or8cT0WDSVToD0xSEC4UZFBUFgxZMZmUKyN4mSmEkhey3F14BiZB4mhqwUKAJQKVOjDzLBh+OqRoI9Slg7fZHY6GggOubw+HXlaq5YEeOwzVXMp1u5xljICwpjknPFmssbIn03qgE0UvisAh64swobKmJWp1pMk2qjmifZgtVhcSzdWvT7YqvXF7PlEiOI9BwOR5YXbzvIGIK1v9M0D/p18ookNjvUMbSQoj06PhrDQ2pubg6/cTzJzeZW6p3LVOYgW9bQ0MiJE/31+GYpl4VIkpcafkfldmb6BP8zucJI8sE4aYO7MW2ha0Fb1iGVFur2JokyUuk0pSCQOX0+pzhVtdtuN4lkxURdKtWI6CQSZnjU7yIkF1JDuSTR0rMWlW+RcsfjWbfbBSTwiEBmi8Yl4zTUEKMoL5U+SEAw+mlgh6GGnZM7uUd1RppiAZGvlLhHEhQQLebxOUvF2r3335fP5rgK+Wd/t282P7WNLqHqY5HSGsd8Mue1vXagHXODbLLU1Z+scQ6AdZCZpjOQFmTEGEK1YcGDicY1CgaIE8JbKKTFDRh0zA4LmWeK+TLwiMXQEXbifljMliuNOlpWzIGMm89rJ+6gVKKgpI5KytgaoqNhx27HWmKjV+/aRgNNPGnCnlqirEO9YsYL1ywwaSa/AlW+qEkqbATOYB1yu+BuJkUpUHHLdyhWgy6JCxh6MNwdqt1quVAjtph2Lr9zxRN0B/ze23Pz5XAgHt9MJ7LR0IjJYF9bXGWA0HslN5JUN6IWyO/9zu9+4KkPVMtVqoHlKlWZLbAGqcXR75k6KFehdNVmvpMRvUivWyX5EuZtuqqiLCTULJ+jaLIuGPJZ+pZsMd/JdyBVGNF8fup/kt2zt2vvDNLghcsXqs1yOd6PBqnmqXvmTz4XHQ41OnlHxLGczLd65ZX06t5jh20pd4GEFmB78tjqWkj5JMBGGhDVIxE8BF9YzNQ5x/ohny/oWgQdIUzfQwq++/d3/xJqwqbRcTyxWj2dw28D5TI2MtfSGFpNfrLxYpqnNqRDB39EbDKfNMD0g76LTBkQDsGxCFxNfJ6RUcCeJq3sB00BThJZx4Di2ghh7jYHtUqe3LGQDxcVS036K+9cInzx8fe8S2L2WLi8EyEMN7wnP/ioO+bo1NeMFDvCXAMsICMJmRI2Q7rFsfh+ckawNlhURuguqspHymfzqQpyt/c74yQIQgZBMaryzdxPw9rNMgjySv5Rr5OdNuB3ndi58t0HPKE9tP3o9iNaa3JWzvAC+i4/tD7KK7UHdwgw4yrnuHnrgCNxONraGAAWNc0w7NwgRnf4aWiW8NOaqhlqLP7MWHwJDhBvHUGOzEsuBwNb9nh14aCBdKInT57cixPgxcu5HAZdXPBZ4Divwm8TVEtZJCr8uCFv0g+jASINkhJ8A1R2SXFYgivC7RUqiCWirB/E16UaZbvVgFkCLUqcDhXkG7jz4BRgGR6Ngi3AkmrsBxYHWF3UiWySZBjdsQC7oE4MnKBQdLxS4pmYUuCfbMOikxTYRThClgaqiO1BgmAcMIDU6EKzjbaFIYIaMYMwhuAsEZQdDsaO+NoW9YBLxUy/Px4dRWsAK2l2sspAdwMT+mt3aN/U4cvXl7O11MTBU7MxJzITXmNjkxOkzUL8oAtejOHGwczszBsvL+LfIMm/YI/xQyaJNktHdLB68qyi9Ebih/vh00DOsAJIdXv37SMkmjYlFZjUbjB1+2aYHYSq0ckZFNnFGr69HWKLSK6RIFFQuXrwiceX5ucwk5PMxtq3QGMgIVCYZrfm8vlJ68osw3wyTUwQWApC4vR5+KJ0KuXyeek86aEQn06dOHnl0hXMsKVG/fDxE9HZ6UoigRK+jtBQreZuL9BbwlV6zUYmX0GebtbL2Ywst5GhKB9FEiJomLPv2n38xLXrc1KG1+FcX0ukk/3RCaPVRLw+Km0F5Gq9Amcw01AOqB0MFKsXAgUhRk60k82qOxgdHsHTlRxr8OkYFipoMOrYlbEQy9ztOzCLarFUrq7G1wP+IMsRaZiMDcwyemckVDgwNoKk6BXTDaTRSWYc8sCMiy1DlrcsW07CAeA9ADSxYS4hJZsV5gsbdqtJamLgStfsub3ITSycAd3R6WtkQqUa8fz8vCw3fV8zeTC9TLEy2orJVlpWJFNWgxJ/tZO0wyU2TnKGzxGAv2vjJJt2D+Oj3an95Lx0o9+vSh1AoYIc822clXcgNmHoQzvkdrMSqTaJdDc1NYXAeubtC/Hbq51+FR4U2w6jTBYak8uIOsKCTcRixIHw5H2n/vRzf7navuEJ+D/50U/gn/XHf/Tfu2RLZfgERyBxSS95D71iz3AxC/A9uDMQtgDbRK/hZiC95CYjaAvUW0+RB8Ooq4G7da6wq1GuXblw9eg9hzBJ5TJ52Cmb2RWLjNy6fruVx8ppKq+UrV40T7ZnnvkCqTP+3i/89O/8zu8UqhmTTcqM8ieIDM4JZhrLk0VH36EfnOCTqXXXoABZowl0SfqZ1qBekvgTlAhUpEbFilbA6SJxdJeLE7M+Purq9Tk3xp12F+rLnXwLH8iswVl2La2QLej3m/7sz//q4Xvuw63XTjowS8BmrAyaKUBMP6hjEqNSF7odJhDsiaMbXLYgXaiDwPN3bdrcaaeYRsCPTWQm9ooscD8/RJkqS0PQAoIxElHMLzpdKmuTk0qIJTSfq0IQQMgoXTDbESFkxVUVQ4GUySHyv9Mnqlf0SaghKaFjbAtF4RmU8vCF4kgnPvz0GZuClE2jbFirA19bq4IhA/lsKpXOVxtkzZt833ufOrq8ZMKDxw6LYdcdPWXff3Sy3Vht6/LopgYYB0UdrDncKcgWEZtZEhjm03gfUMpogH74VKGUbOrj1U+QgjylSJkcbA+TGortMbzrvDzNXey2H99ua6dxueV7Nm6+c9vd19T4c0Kubh9zsNML6Q8fIv8r/MWBPK5IL4dCnIS+yrOK74DeytQCD5AYgQpIFpRXwsdEJ43yWaibEGU5kKhIAA/EpIfNp1QjiKr7yU88EV9PfPOFa8PDusnxkWx68+yL34JUKBGEGpxhdMpIe/BMjEM44gdeeYEgASlK46aEGBlncgkCXeplrIBGsKEVaTxXQ71bg+ZRMgVIYPkS240BsFGr4CxFLXoCbUfHR5EOSc8DYcZ2AFT1epLJqF9twkSBWGWMaUvXB0VajAY7ykCqOIjKQ1RzTBM4j29EEsUILREfrCq+mpqDuC+RcICEU2rDTwQhB8TEeNI+LB4U0e8WSwkGwlAwVKrVoe59s5P00ow4BYp1ZJxyeBskbxub9OvtyVQaO1PQaiM0oopu2e1qNck1m1tbbteqxVBwMkT9UkOP2GF06eRxw9GMuCCGkbRgbgdhej3J1QcBRug1mfyBIMXsH33Xw88//zxyZrMh51krjBOgzIJeWF61e/x2t59Ch05f2GTH/wU7bk+XypDJy9io2aneYDY1axVMgXwiM085hHwuh2lXvM8sJrIAVUm4CHVxSNY9cVmlbDDRIjodaRbefv3N6NAUso0/HMV6Wlzd8EWCxn6nmNkklLiQ3jAOYLCtGLlxdR5IEFm5LcYCBJI2RRFiu3bBIl2+cB4gBPVHR3yz07uTBzP1xkU6UK2T/71DHUnmB/Zre4M73pLwhG8wwTwRtEbYxYCqFQv5W+AOII0T7VoDjZzJKlFAkYg7uSmSsc/vL5Kjp9crlIpUxxsfHsbphgnVCC3Tqm14iSugkPpFPK5ROzhP1gGCDOBEH7hBSKAih9xGZ4RwEj/TIjgHER3ffrFlV4sNSktRf8odsC8sLUxMTKjUlnhmdVXLoDpgDVQjcqrGNdKyttEZtcCFrGrH/OSYPRtndo61M9pTNMtPrc90aeceoBqRXuBa0W9QHS0xenwsBJVJIbt1l9yZlNpWMvrGBvFjhsBEDPAG1DH+5TMleRGFmvlAg97nDxw8euTkA/d9+63XMtUS1HRqZnrt9hKrtVjEvQ/dp9Q8IYQU7M9Cg8jRGZgbxg3S7HQz8nhQynewuhnTXpHykWhD6I3OSdIo+HOoAg4JemNqLX1Vfx0aPTaO8WqsUW5cevvK7fkl+JzGehc6jd23Y+lUK+Xf+r3/9CM/8iMDhC8zwCNROYK2IE6gAgvaLMlTq7LKOq1mcdUsFwier9dKpMDGp0znsGMpl+TzlERT3gO6yKgTwJ3a508kC7gTjY+NW23OXTOzqVRu+Z1vQsVYjwCZy+Mk6Ui11yISzxshWTqu+v63Xnsj2m9aR/3pTPqe3WELyEmcDODVEPDEeRJKqeekmnzmVv0pIiEDzRTf2asT2hntNr7qzsaN4HomHcypLBg6t9/W7dco6UdCI0KBQeQK6zPcZgyzZj0V710wONVKg8pdSPzQbTxgEXBRMsqtogaHISAOAriR6DtWjSoVTFrDLoH4MqZGsgtDhikYBRNB2XVzbGSUCq1//cVnLl+9EorECHY0oGCi4Xc9cdrka6dSK75Qs6+vkWSNx9FgCQcNbuYPQisqWl6ssSBAp/p6bRS2YB7ol2/WxoV/5QcoTwQjUWWD8wBWwJtjNR6yWkQtBC5UhJDbZT0o0NfaUb+kmb9x2+rF916XPgivQGM7d9Cy9pM+sASF25H/6KOclz/Z1DwwKuqM8F3qj/NCbplB2BPRQovzp/KuUvIuqEYeFIkIotZHxSEFFVChmNA2Gz7y0R+xP/Bg5MyZa3M3EMsIIFEmHSof9MnZhu3X43Lg/IGgQDg/PE25lJelaMX3CY7UAJZHMQbXXMrD3qLsFquzqqmAp7LOI9FBVj6JWFiq1EGnRRFITmUUzmL/a2fIatMsCfApHg2/EHIhD6rYrlEtt1EcItlK8Em3Td4Pyi7gIYWAgxJXCCoEmGByuwN3EtwQ8HnFgsYsgRdh8rLlIkpfhhkgpAU4dypBgNSK2TwSFkrUkM+PVzfBgt1mAxcF9NBcdQc83mAAEc1CaKPda/OGb61t7t1/j97iyhaqy8uLkaGIx0mme2M2uVyrFjwOg9tmGBsKULao06iy+CPBAHXh26QZgZCCJFvdSo/gnarkMibaWjx40Rfga0OUCwSdmFt+Ag4sP0ielD/hP8Da4Q8HY8OByFC+RMRrt5krhKLj2D7fOXuWKZ6d3lXPbabzGYe+a3VacRcyO90ZcgpiGg8Eq42KoW/1e32pQh7n82I+i5BEpaY82mZmp09iDty2fFOnH2AmyvM3rs7dANUGM+lSMeVgfePq1kQsZFIag5bXSrh3jajNct/npae4Wcxdv5ZMJtC9E0R74+o1qKYKEDOODI06LNdL6ELbFGXCGwuCIs5TAAybojrYKa04meN/IKQZpyLJF92Lr65Vqx1CZdAoMoswJKK/Zomb9OQKjcZio2MTTzz5vj/64/9OTp7kejEYdc/Pr7DKYTzwLYKIQo002VGT1dS7tl4ql7B6skhEMSHMO8sGmQkoAvAcLjuspuRPFa0N+URIaqxD62PAsOZGdDaSIIp65/lizg5hMRuop6GZ90AN0HRZlBL1I+wosLZDQeWCWuCcoTO8iN5wRkZEaad3DtSNstPOMFwQOTY+hGPtPM+q87AvMNUC+nKJbM/4P6BraVfRoXAnyyidzCQTKRjfIydP/eRP/xQuk1/4whc219eQT/li1AaoxprddpSptJjnl24XKuWp2V1U6MYt/Mtf/iqNgysI6mVdg4CZMwgeAyjjh0O4hVTf4BdhpETyI9WLgQVoZ5kykfQZl129DUa6TwUnaobV8rXoeHRy9+TE7IQrKGlqzrx11mW2ZTdL3XLf63Bl9VXUTQi4OC17ETUt/We++rndh6YWbi7rqMbFJ6FgAI7ANEJh+GLUejgH+CDAlMss4xxRkax84tcNw0ydPCvtIf7WYPBg1Uld06kB45b9wZFgJLx79wF8Facm99wze3pxee3GGzeos+qyelZWky63LjrhzdRK5lbPZnUnUoRVWt+eu/Xwyfvvue/BL/3xf3rPsTHUZYw3Dn8UeqIziIBsvFm0dEJxmCyhfoK2NXQtPwTshY1Qx+zk5zY+ZwwBGrkdCQqRRpn67E5s1XxnVan0WpS3FxCT26DH9r7e1TN4TCafTucs5nARKTM1hCfgRYfJjuh/aVG9BUUhhwAeli8YNURqGE3AtE4KEiPVZTDi4KeJOdheraWDoQhZTRpNIdXX5+YmQIlGOCNS0od1+46O93UZk72CGytcElMOOEKABVPJCuDjBfhgtYRfAndpmxBmOZKuc1obEzkDeYbUyojJSGxvsjq27pcr6od8yc55oYQyYFvbnfPbZ/6O/2rv3Hn1dhe0E6xVJmyr51v3SK9kFmV+RM6V2ZJbmHZxKYV1kDNyLNG9CKagF/HPgh2FG0Vk0h7sNXTk8sVbhwhRMa+QADnkQpu3sry8+dZbpKN77N2PMluZfAZkgtMnuAr6B+WrViXXKAPicDJnOjLXsPiddp/N7pBKDFRMaRFP38RuSkiFuIegn4JnljA1QSnACJ2nBAPyMVIseBgVtNXVq/dqFitOuBY8RgQLw1eLxZqULuLxh/0V8xxjzogQJwX7IJWF4dOMfUnZJlgdw4+dfBc2l4fi0BQIZ1lAvegq1lY0meAm8td4vH7ca8PRKIPksDfoFTnoGVAkJ5Kw0Sde4QvAdrgGqG0wf7o8LrcXR6MOfru4DzvIf+pc30xO7dpLTiunffLapYuxoVD00P6Fq29Qnc8fCdXLeVIKZcspi9mA0p2iUQ30otgp4TH5HuK7qPTWhoZKpgY0teJlPiBsmoSrXZLOEzPQxAGs26KoL3oKUVvzgUYLSs+g1R4eHjXYKX7RaWZLODc6KdfTamLZHRodSbQqZAjq1cpY9LE62/GrI3GNywtpJ/ye0SZWmzN+n2dpbRWKu2vvLPgXmx/aQq8/5g/Hbr9xZnp2FuF7bHR6fM/M2vz1Pbv2L85fIYMlUwmuqZbqG0wxQeFd3cMPPry8tgG0YOaHfUmn0ixmGGoC06anpvCpJiED7nWEYel6VafFJLERzBlzhQpDCJRssFIIW7J+0TyTnRRFMLMqetTmzESMPZm/ACFRRfSQR7EBGyhUdWthESha29zAal6rt9xhSaUSjvnqpUqjQQGMmtks8hnPQd35ZMCY0YaAcSwLh42+QB5FVwRXAeVQSmDhU+Ey0HKSc4wAMdHXWZ12stvuObAXZxWYNsoNgV1K+arbZ8sWslhhWWegGvAM645N8ICI9YJ6MbLcvdElbdO6xLG6X76dr96+KM9IQ9sbV+mVrGhFs7lt+xPUAIp6hA+RGCcAi28UrM03oSykZI8RDw9Ja8ME4dfKtri8hHJVdO68nGcbPb0famaEIUskN984/9by6sqDjz5y+thJn96WSadjoUiZwFEGrduDd2YEtLeLFzsGYLQTrEU9SQJanYKE/ANpfAt2XHpLl1iyMG+NQheLodNmhuXt1fsxX+zkkfteu/Dm0NBQMV1fTK6S2KVX12WzVeocIN+5I3q01sVM00bpE0zzoAQH6icRpwRVsxARfzH3WkRmYOXC8MHXFqiAmIKXoFSRnqQmBitTLBuzwLQHolgGeoVKwxfUxTdT9z90PBobjY3EWm39G2+emZreFxsau2G40c62s9YcKb6xTWE8ouCmIxA4e/by1HjnytuXZgL+f/df/+LnP/bkj/+DX4m//RWTtW3smdvtMqSMIrt0hjEBoNQGbAFKvJkJ44QY/u7etElmMLWDnUtwGXIjymc87ngMtBwBEWFGAgMBtySwJagLAsbnkAjP3h849QafzughgqSQqxazFYFo9EYthCMILR6IDt5Bq6JMYR0BHqK3gTugLmBdaAHLEsIp8YasmcHG5gYeoPec8uG3QtA/qBKVpNPlohwhPj66k/dNe4OGanPD5iGejdIcsLCIEVY+F3ZOoIr1oT6Yha4+j50mDctZgXm5VfbaxvH2EMiahImToVKnFD8iCk9mkYFg4dJFnqLDitjLefX4dlta+1sN/y3/SCNbPYCgaAtX0VTtGfVyYTxpT3qrCb2Mk5pawRDcwXDBKvKnDmTWpaYbGETpwcQhizEVy4REGfEOzAPMaFtlsOIRmETQEUsGGwPEiiBvzEASH2YmWgDRzGzu42OCjhJtDJa0zWSS0dC39bUKNnWdnZJ1LuK9vCxk64gVfReyCeFHmCSRQUmpgSBC9uc+hASDJdmylOCLERSDqQv3TZx6PG6cQRDgwI1ohh0+b79W94c84xMj5NDA/4KQEj6PJYxvJPNBfCoQQ0Y7VhrDL6kVKaqDS64VJ+E2NjgJPCKPhx7XQDCr+AriiNHpEjjSEGfvdq9crUsWGAoJ1xrGfAEQZ7Jx2gwHwk6X/cC+fcZQuLOxTp4dMlSXqFFQKJhaQB0xVT3XIGh2+UB48ArNFhGPHTQ4Vy5emJmepHavJequXL8QCTgDXktyY2Xp1s1yLhkkCUAs1Kxa1lfXIDhkerLaHeS5QBVtxpdKjKZ5CbWykBe6jbcVMhky4vX5axJ3TZUp0XvCGnVYd7AvzBDBsC5/0LNnr6fRKaXza/E3b8xdw+I1Fo3CzzJcoNeI24503230HC4fOaWooOdyudF+YwWkCAS6RwI9Nf0qqyWbzZEAci2RYI7wLk+lC6HYzGo8MzW7J5VLzV26iUkN+Bkfm8puEmlmdYmPtbMkSanc3Xozm85M7987FA7VmvjylAxDkdjQSCZXWItvoHC2O/B0G4tFR1cWV27dXoZSNmplkTaJUxXpDVgVgwiAiR7ehrhtp+6KS0Qos9nt8ZTNVZRiOF6VCu1AAN2zBV0oRBS1M+PGwqCGcZaKhD5fX18cnRjFm/19jz6+fOs2abnIkgH21zbeIEtU0WDGXFawWv7AsKAj2E/+EX9p+RNdlx5jgaTsIJiOJYUnosPlQkO779BBKNm3v/08H4AhAi99fPXxQUR30qjjBswylYXKAXPFylUrVVvNslppUBbz9qbO0DFhvNRoiMFYO9YOtPu157UzHAs9U9IzH8JJVoL8FD0oi5oTyEYo2FER4e4m7xKvGmEj6BJRDhTttP71578ADPDFDDV+fawsPauehWPh40lVmLx27RoFCgnUIbeoh2tGC86JGqppDBr4RjIR1LVA3IbpYvgwAeCyDrEXlSaaNIoFwD62OigLRGkHCaBOKKI5q4i4WmVFTt9Ov9J5ndSGL735+s/+/M/9+Cd+5vf+828VN2rk0tOZug19E9RR2hiYI12DHRfD8vDMyO2VtdFdfrw8mR+oCLKv1abDVRPuivVDbAKlzkq5GoUORS5vSfykFEHD7kuhT7hP3FUcxpGJGAB1dW6JoJtqo5dMx1H/GMyOAwdOZjPN3/svf8DC3H14XzfXXp1fopI34R7xtdqUp7O2mpiePfLv/+Pvf+b3PnvupVcq3dZSof/SuXlS8ewN6kMej7BtfDOmBxRc6F4ZXyBOEQrtH9kLTtcI8V3EQpSpAipgb0VYOBIPF5YJSF4i0vBeIfX0kGtgbBgxFjCvKNa1mDfR6WP3dekGHp3BC1+tawyySWofN6QJHbWL6qS9E5KrKJrIohIiQOeEmWNdw9Co2i2cwzRj4vNRhqFy+Mx/+6Pf/J3f+/ZLL5PSbnRyii8z2XpkqjQxv1TKOv3AfoMVd5q0192s1Fput7Ce8gV3b3wpn4kHHkkZuaQA/e7rO8eAKZCrBgGvYyF0bHJSnmdjLwOpber89o+7/t1u5K5Tf+Mh38+mNS4jvXOj6oa6tvVm7RJvlyvSR+kJ/4gRi3/5LIgnh/JLbLomPMRliEn7KNdQrilOmDEAk5D6RGxeJM9H9pTZxTiMz1KjDqMvCa38Pi/kgQazmSLsP6kVQIgELQwNTTF6BHcTdwReg5Djpoxd32L0wPvDI5DVoVjPQ7BdngAkUJJHkk+RgP1KpV8tSGwaGdT0zQoGIF4jAw3FF/UcSITK58hVrCBOo9n2hsOwBEhpKv1wWxhnRHPqZ5EfwGzBsEQCqSoRqkTLMuISE0QVNtJqkT4TZkLq2MD2i/RM3JmEDFH3m0IIuBkr72k9SuZgwB/gu7ptJLmyxU69Qtl4EKenaqniHvSWlpfpAB+ODIqAy2twPJS6H2TCQvbvNQmQ7fRpw7q4ONdtVTMbq/t3T/Z79XKugDSxuLC6ePMmdNdnH6LFRqUEnUZMRiNKWVP4kpu3FvMFkjl76BZNCUpiGvstWe69Hq7EG6SDQC0PdlQ1GCHDrG7cjyEQGLuKVC3YSNSkqCOeS12fzxP0+3Az27N39+6RoRsXz/TrpaDHgdwn+mKCf8lhglNVpRrZNQstn5u7zjdvJtP4XuHvPn/r9oMjI7hPm9s9snUxZ2PTvvCpe9vra6lUeWbXrvj6kl7fwkx3C6eqXgtnK8oXVhvkP7J2UPVVKu35WxuJOFV0JJl2nrh+M0VGAaSrV64lNpIkuA+FhwhADKH6TqchvChwEc0lTxWZKbFfwWQM+oKvCTXrYx+3IGDBH+AQh40DA4DTQj6uMqp7zBVoUdrNZmGQJwBnYmJsbXNzctcMg/nBD//QwsICBPjWrVtri8uJRB5A8JCehHhrpQVlvUComGX2igYLrtBILyptzqPAZYLwm6eDLUOXoGtDz9zDhtol6Iq4R5KldL0B73ve856XXv121BOWNFi2Tj5Z9gTseDwJ4AEowuQLBVc/xdqAeM15WdLbG3fRBwEqWdWysNm0n1rf+MklOsy2/ZDcxjEPcQ8bl3a+gm8k7yTMNXDEE0TWscGjQqWE/BGIZDGR6IvHyW2aS6bJMUnxiV69uSke9YT1q6GADGBaBLQqVeW+Z07FE8V8fmx6LzO7fu2WM+KFH2FGmB32aBrJdSbZV0wSV6YzVkVj0WhIh0H4uORw86CDaUhSgoguuitWWMw/3Z4r6jQ5pBRgOp574j1PffYP/394HfczLZff26g2GsU2rpzon5F3cXTDFavSa9ealak9w8lUQtI6idYL0y9abokjYPFiKVpdjic3CQAX7SesAGi830JJJnAF9fEHLS4PdeBN07vGLE59prSOZZqyvhup9OHDQ6uo4lvmj370J+PxxIUzb50+ds/+8f1//kf/4+UvvdyJ6q0uHRnjH3ny/RZH6NP/1//94L3vcftvN43Wb5+5UogZnSvL9v1W955ReBhx+cQ0C1cJuwO1hRWT4RCSu43i70CCmk91WXbCQ4Gvtd90nmeZb5JesH675Ms362xBe9dQRsvNBItRBB2AEn/1OkdfVNB4sXigxJ1ys5SvkWlElMEkzaWAKH4L5FtBPwJJ4C16MAk9kx4BTnRDkEinj6KO5MJom8DtuFL/4R/+t1/5lX/+k3/v57/81a/dWly6cXOBVeD2eg34gu4/ZNp7YKRYjjOsIGcP71VcJ/5VwLGAsgJuPkc+Ei8c9Jbi8sHpre/XXvzdQyBXlZoAFQ9/cDLYFPlGxkaeFagS3C45SUTVJH8cCKyzqWe1nTaGW98mSi61xjjLESSHZxgLbZEK6RVlquqnYiFEVFV/LCThFAndE/XY1h9ODUI4uQGpAQlfRFgDgfVd2L0min6AHuJjbVZ6rRp4DXONx25w2Y1ujxTDwKN+YnRo2u+JkiKpUaUWDE2TwdFQxYBuNPh8UZwcKCoCZjdZnOBNDEhIjRR139hI5bKYCcFc+PVgDBYfWrAt8UVgAYJWsRKAcbAPIawhBKO4RulBroMCZj9Uv9zfoZJQi/qdVXJ7Y5uFI+/piEzYu2fPPceOox5p1pqRUAiheFhyc3hRQYLaA15PMrEJ7iZ8As2kiwTh4F8Y2laTRBlSwMBsIgE1Cmy4YPA1q7FKhnS3hzeiCc8XiFzEp7hMA+ie6UW2kKf4JSiVNgr5Eh+CHzfm7NnDxybGx/kWeH/CacCn5DolNxtuRKgEcJomECifSyHO404lIYZmo89lK2U23VbD7NRoNr1eLiTdAVdqfXFt6ZbPZR8bjmzEV0EC2C+reIDjZk55GbOFlCNLS7dJAYbgQhFsYIkUFvhrsg9HQvffd8put5QrOYoCULKFuQxFQg5s7fgK6alRSGQ9rNPg1KlTdDCXyxJe8fCD9y/eXqDkXy6TDAU84slsNRHQKbZ6Ikak5rFDnC5M5tMPPJhcXuY8+vdssYSnSBFUqzcHQ7GV9RQJIq0OP47D0ZHJ8On78Ya3zOzBclRt9IdHZzz+WHojb7S4PcEhTODnL89RoHT80feeOPkAbNaVS5cRQUhgCU2anZ3FpMcGOfzABz5w3333Xb5y6cb1q8cOH0I4mp2exkkbgZ8Vhp8VOgYUiVBXCZLxOlG/UyGYT0PKBLrAxaBvsdPXmuzJLwkPwxoj9BZPbBQn0H6/30u+HuyXDCnLi9A2TfBF5QwlZUIh5CxPAIONAzG4U3pN5XOmKZYSqABLNcXt+YMAw/fgzI+eFEt8OBLErYxYdwgYoh0Hz3zpi//tM38wNTUBH0gBGT7CFQDnoqmQGsYQRWYHZpByhXAhnKERKD4neSl7zvAK3s5P7eYtZKH+oT9c0iirdoN2lTN3/6QdNQi8QXhZBF8CCUEXfAhoCmRC29I8xAfSwwRDIREqJQ8ClSrMdJyCXXoeaWLKhUB16audgD0EKJvD0Olvrqy5SUXX6fmd7pe+9cKbb7yNHxYIlnRs1M2Eba2V8PASnRkKlaGR2Gp8FWPi5O4Z+LmRyXGdDflMhyEBRZCw2SYDtevgVfk0uoha2uFxVrO1YjpPr95648yxwyd/97d+//Tx+4Z3zeLzPDk2DX5En4W8igYYJpgHw1G/5s4dCgVRN3h9AbcngAvi8PB4dGjs2InTwfDQAw8+8tCDD7k8+kFBByOLDdaN0wZUDOcvtzGbLzvczj17Z50ux8bG2ujE0P/2Ux+emIhCyIE3NHDUYXz22edwIkRLj12DrMi//CufHt4XreQHXp+xkNNdOH/p1MkHZ6b3/aff+M1kptzoGqioHc8U01mRYRgOVCaoZxh/ip0y4lvSkYhGovkQHM41+CrRWsoepkVmEpqrLil4EP4JTSQSDreyQlmteJDjiBkdJ3aL7Nhip6cUCxwXMABGxNmm3USeddutIcol4IGVS5cWbi5igvT7AuBqhBPEoSqlecHmNMsLaVMABioP9QIx4MwvxB4HHT4BEGUJwGf91Re+/vGPf/zFb7/0nsef+Kf/7P/6oY9+wh8aOnjkhP7ksO5jPxr8+596sjWYs/uS3cGG5P8B/oQKM55OscQIF0QWQ8kVAr3T+A/WJ/eoQRDA1hS5IuRyExyBXFJ7BKftY+6RUaOrd84ohbYMK4uNpaeirKCjytlazrExwPRACaYy2LJpLYi9Flu1tCkNbm13rqrZUvdzWUaK+TBIyR9ZltKunNI06ihcWdWyxiS9Kymh+vho8MU6YwsRQTw1NE6BTqJRYG1SIxUCSfZg1MuocDHDwMlKqD3ZZ9oUJ5fMUKVSBbLEu1izLBf8UKBqqKZEPBD3DuwNpFDJgZjIfgVPM6B8XqVE4m+WB+oL1le1go5JL86ufQPaxEImU80nyHEBaAIxoi4bSCFeydYh0oYYhQUCBX8ICkHOADs6PG4+WUR4YKXZYaFiz6M/NAMlxxwIoAj6UfKT1x9GMUm2Ywgz4cLVemNscrfB6j535abbF3343Y+TzJ1IGxzMEmvLaHMhqMMjMTJYYXkKhSLEuR47doxSiNbR0YW3Xt+Mr2PSZgpwL2EIsKe5qJsEksNPG+yGIsvlM1k907sPYgLPF4mEJhbWigPayFAYrF3Ip3CeWltbIbXWxEjsxpXLeOy2azXejuqNj1W9xgsNRIf3sXjKoBJntBl8RhIj7kZiHRSAexGkQilFLVeuXo1vJsfHiJz2o9a3enz7Dx9BqCUECO9uyiaSpXJ2Zheerx4Gudt64+WXmuUiwdBhvw++BEsy+AUnibHRYZyhV9eWkbA5jxJyZGKiUKlCj8nzPHX0SDGVuXj5JhXWx6b3Z/LlXfv34wB59fqVdqOK2sFpAbX2V2/fDPg8kViMXJv+QASTIY4y6XRy9+7djP36Rnx2zx4Eo7X1jWh0iKQcpJB+7dU3cvkSxsFTp+69tbLynTNvrqSSCGsEQcHgkeYFzSeCODfghxkOwx1iYmiCdkl1hOZc3HgbxI6InkAjUQjNEACSR5brNWKX3X4f+GR4ZATLwsF9+1/42jfLORTkFVoQiFXRtwAblBgwY+OM+ldRLBLWSxyLEfLMUiYmXtQaBNJ0OxAblOoJ8pTVOk6/A90P3LZa8ARVShIooAJTiyhdcClFyYmBjhAveFhlBZEXifYB4wWuC2ADQQR0iU9Qbxf5lZ5oJ2kN2OAqG/3cwgvqqoYNeIoDrfPazfxkQFjmLB/WpvopSkWWKv2kBWiwIslC1NnAVVzhZtpntWKbgUni07gHb0huIM/JsVP3UOdnYXkpVy0YrZYTp07iNRgNDp156wwaHcSaQakBqvaGPPA+6F2QjFfWV0DrKKYa3da+/ftnZmezudIrL7xgcbjb6bygZbMU4ils5ux+O2kQXVFYZHFBsIW9zUbF7PeaPd5//mu/Rj2pP/mjP168dAUWCiMTSw2/S3/I19Y3PTE/Aa5LGyvRiSFYEQLp3U7MmT2SDWD/InsdKwVSge9LPlNcX9n8/d/9bOUWnmUqatmH5b7VyfWHD/hhZ8fGY4+975FrN8/vPTBdb1UPHj5gsrief/41GOCx0b21CrSz9vhjj/zRH37myXc9PRYdIQ/PP/un/2hjMesbNxernVP3PfHYuz/msAWzifSgW1tbutxafnMyu/DR+/yTQ35dO++yEPpDvhH4BiEpjLbQFtAIsIVMLL90oBHoiaaNxdtYNkVfEJrBBmiqOFDCHulWdM1Bv9rW+YdNo3tdrpi+byFF9QAmDsgiYYNeT8ivQ68P9wbDBvO02bxHpx95+6tvfu1zzy3eXiKQF86V1NtoPdFYwxcqGIMiSawIcMJrAUIsXPwCZgEQ2Ht8Kdj3DRaPN4IBULJ52F3H733wEz/2E75AiGxoJo9Pd+TYMBENRJnh+QaWhLxANJVvl1BC+STlfsUcCNjKZzMUiszJCPw/tL0HoGTpVd9ZOef46uX8+vXrHKcn55E0SqMIkiwhMpjF2AJjbNYYlgUTJIskBEigQFBOk3Po6e7pHF+/nEPlnPP+zq3u1gCSF+zdO2+qq27duvH7Tvyf//mnC2fTGerMrs6UYAvlvdwZfqEkXNhKvu7cU3arTAnFjLkVXfinO5YZ/uYDdo6iHKKjRDvH5b1ympwsW8sRZGHlzS1lynaOjZ5ipSh4DFutkQgXWR8IDMBe4MiWS1Rr1s1tE0F/kJI4edwW1C17QgTQN0dAy9T80dVAIMNwOvEIJcsFfSDBXdZQGMMbLk2yxbiJRFM0TQ05W1L5LQiwzIx4tyuAiAKhg0+LZnU6vQhIHFzhVWxrFNVO2lHrdsGZHCA0lknGMBdLtSK6B12FfuLxovoxh2C8QaZIagp2KKWbGyEXypvMuDjFCkWM3Pvgwd2xazMloTSixQdwghqxdHBWFpOQPTHBqQH1+y34YbIf4FtaI6gfSEGCsGwMjAO2QtMsLMwxbzOJGHc4FPJen57t6ekjrESGGMHNx6ldk10uF8E3WLpAOyqWLP4+oZBmMhHpIitM+yTai8MW4g+QLtN4HNpClHYSONNCvKxpW4IBlc3sbFdpXMgREeOEmqG7wjOlYK9SLtptNsYhDw+rk1grSl4w0dBsWUzEdiowNleKyUSTVoB2G9RdFDWKUqH/AVyJwKPk5jvstCtAagLpCnV3cS0NdlDIuB3U5dXhx6LzEn0Mgj1d4VYtsrVO9JNzsDvcySKVsvm7dk6p7Pb5P/8zSP6k6YvDrbc4W6W6JwQB+5Sqd8CltpqW41QvBAL+2cUl6+YW9DijIxP0hIlsrly/fH58pL9QbR8Y23XpytVUsZYqx97+9rfNXr04OjrOBCbssXNyF8YEPiiwec4Z9mwgSxhmY6ND165dt1mNvRDvet2r0e10plZtJXQWMvTCcgL+Ga+BEAcVt/R7x0ilVQ4mJdaeUtoIZwkGHDFNciX0fRKNIuqHOIfF7LDZ0LcYFuRvXTaBFIj1qSzcQ0QMr3wiJMMI4YesUbQhegv0AMlfgWoqm+FKUqPMXCFnSWqgTMZUimrQbuoW/eyYQkANmHRiiVksCC0hoGCSMT/pTwCjlkShxWoiCCQilhJiDUUdnIDoWhYOxJl0XjkrVCnnw0feo0X4yNLZjJX/ZOErfsuWrOeVzTrnLA6GeBHyO/Q7D4INEAxsLXtQJAZbimjprGmo8uk8+oCF1I+E4wjLqahHqK8tLusiYaJE2DzULcxcvYYBATFknjxiodK2mRF1RhtUnRQIwMRMa0rJ3BOlqxNbbjYyxXwklfjjP/3z5597sZgj0FX6/d/+HQjE07m00W3CSw4OBmKpZKivl5EZnd9SOVV1otmJ9F9+5s/JK21uboaGh8r5HOVk9SLdlFRtQ87hd0LL2j86YLRbOd1F6lBVarvZjbnAw9zciJ09f/UXfuGnXjv+OpQvdM06dvft5FZefuF4KpGDUBNGaJ6JuU+zvZj2jpiw7195/TW3V7psjQVHBV+iaYoh7hnKZsDOaFBaICre98H3v/HymaWF5cP79/3oxz72h5/8ZCYpiM9rV6657cNvffixV65ctho0H3j/e771p+cG+jk1mnZ2Yg4NhoTgCXhKvCpPAJdJ+SCPTHIeItu54Xwtz4R1ylbyBY9U3GKGl5JAlnyCmpSuajDotnmN9XYKSS7hZ5BwdZh5bHTvpdZDY3K2qhYtLNsqczldmp2ey2WySEtkb+dYMqos6EcGJoMCqSGFuHJSMpCkXFiK15WIbrNJZg9Xkm/oTbc+MDTiDvZAOXD89Te2orl3vvu9ZotXFwyqdk1Rx8Ys5aHXtUhW9C/qUa4AgSzaCV2p3AMxQ5TrU4ab8iKHVZbOaJSboMwKuROd6aGcWeejskbZHzsU1SjT+tZXN7bnnx+0sH9ll//oO9Z0Jgtr2U9nUR4Pn7FblYckv5BpycIGgqwQD5hpLA6x7EEuUbHimwL55gFjkZOULBZh3kbAtxnN2OLFIqmcFjlQZjn7EIAPFguPTiQe0GBQIhI0I/HFhVFPL4X0deKBLlCOxOmoziHsRj0dh6JWiGihWYhg24xXnBJ0AFR2zqCXpkOJWAQUhuQbqnW0GrnYttook7TMoMTVI2FB/I1AH9pKZWmCuUbKNaq6OuF8oOsYYeR9uFLUntlqhmi6zDOv1Yqgg8lfVBsgoagsICjIfsRrl3p/KQ0SXnKtzuZGUzuXE2k0GT631WCkjUqukoTXJ1cofOMb3wDejJQrlrKh7kAeFswcjQUtNNghfcXlb6yv49xcvHA5SF1LZBvhIuUNIuyaJa4HDkijmYunjQ5y1uGyC317sewNr5ExnJ+ddrj84xOT29fD+GHBntAqhD6xSCDU7bJZ19c39x86dPbE61H4csvUDxO71uMdMn2w/UUtiNUozdyQodwEmjPSOY8QmUQ0a0Vqo2kkDr6kt6cLjQKnDdEkN7YGfI7oeBohwHqv1dPThgLASjmn01ixjbGdB0cHKbdY3ViG1oT+NplyLeDvKtRqz7z44p6dkzafH3xyPJUC02UqVS2erny5pfJ3qyJp+HxKtXa3P5CKbga9zv2PvWPuhZfPnj090NfLIUPd/Ztbkcmde+eX1u0uf3Zuxe4ww7vJMzWarUCisCr8wcDS6kq+UJjcNYUqxc3Fpdi9Z4qWgmiy8+fPQe5x+x23qS3G81cuUUbFFLG56UZb2djIYg/g/dAqA50EMR5PjWfgsnuAu+GeMRc6qqszKZijFIh3AnHwl6FgoE7L5JLPRZ4DFYsf0dmY9Qw9dC7BF0Zg57eMNIRyJ2dBP1TyCyDx0SWiUCFNw/cCwo8KB8pUE3+RUQDfBHqV7DS8tpCVYsiCfeLqeEMICqUp0pQ5hiZT4mV4vIqilHPm2d2c5jKdubrOwrnduiJOTIYDYWPFdLh1nrzpbMMrv+JjZ7m1Q4QM6/nIb/mqsxkfO+95vbUl7xEubIA8ICuEGBLuGVwXhIJQ1sDK7gWlXysVrLDMuPxAk+KRKLK5S2/5sY//xOf/4rPJ+RUVDHcIokJNb9cX05WV9ipNZFPhNDfDGfBvhDc1FvNP/uzP9fb2wVjuwoCuVQKjA7C7V9I5FFg0EnMEXP/p//y13sGBX/61X1m5NKPzmzUOJx1cBsfGlubnsNgqhTymP9xbHq9zY36jZdBGsymsffh2XH73L/3bT/zln3/22tUFt8+ZEbYmCwZ9NJ7cs38Xd8Af8gKk2HNoajO8ef7MhcFQV4JIDm3ICi3vKOgRPRRgu8fH/QE7TdQYmfPzi/FEzmhwDPfvmRgbVbcthULpxMmXXnrphXe89T0jfWN/96UvryzNewLgUvzw5SSubB35Pw5evXghsrE+d/38z/zY2z/w2KPN839vtxIFgQycYisJsiKUifPh/CjaB5tMBLuoA7678WzkX9FRsqBQ5AvFppLPPGe2Br0BFAbOPm+32dPtV1kIUxJC4ysCjqT5yNoQW6bqyapqm+HA0ugdRKuXF1cX5hYxLBhdnYEnvWokwQguAIGPoSmjkcgMp8U2VBcRfeLWMSSoCWW2sDVEb3qTxekxUpcYjid9ocH9vWMr69Fvf+/ZffsP6Pr7VKFuQ6UWJZpCcpYAEhMFgJ0yR7kcAHBih4oClsHbmXdyYf9skY1ky5vjtXNzOjdGGcJYsd9Xt8rdZHpx8vwAU5QXiTIoe/ln+1Ymg5gbN8+AnfMz/pgGHf3a+SifOKTMDWUT5X3HemBOieeqPBn5oagteRUrCRrQQg2BoVfD3CShgxpEUYRnwI8QlIYMiBiF1GsDZcCvhUSpXSk1hZmRThoGZh7MSjwMJEadgCeykiQj+NuWEAGZAY6UymQoCQsJFoaxREoAH5eiSrBPEO/A6BbwB7uCPpuFpKakh7GGM/VcBvAd7JUmsEXNRCwfCSdJ1eVKEg7iTkDAQPOqsraOoyZdFhQvhUQBEkTqdev1MmFCqP/LEl72en35JIU0tUr29VgsiUYsgVnEQ1ECdGgseCPEldcbBo/dRdQmk8xRpsxYB1qMXkbxuxxuXCIUA2PNajYCaKaaIptShSSwRPaOp6cloY5ViC9FGjuRjNkBSkn+o0aPF1ppEpwhko6fvRVLG2D60LQ8BhNINAqciHebRkYs0/NoHXNXKNA7QF+EuHD8YU8a+vqHiV1vb26p7E44ivOlMjaI2In8L09UEKHiPsmchNSwArzcK0ku69r6cqWU5yEY9Q6gyxgLVFkDj3I6HcaG8EcCIBt19p06c7qYSfA0yVBmE2Rh2qgu5mtDJ+yAHq8vFvf4e0KknCvEPGp1uAlMTvdWImVc2ygRDalTE0yHXcfEnv2gLa7PL4qZbQ/AIRwMrq0vXIPgbGTHblU23h10L8xNa/p6QKuBa3G7vF179mVPnAj29N12zMgPPf4udb2aiKex2BAG0AlRF0sJVqbASMiMT+wgE8yoHxoZCHT5T75+6h3vfPvC9rbP7+nr67F5ilSLm90OG3afQbcdSSbX4o1qvCdIGbQrVolIcyibMHq3Bc+qKBjgWwTz8BQ67iOsoo16Dpg65aU6rddNaLTIuO9EXTvTSdG+Mj3JWCAPEDrMJzIg3DHSK6RRanUetNAD4VSZNCYUjNj1Qi2ug80xBCvu4ABYh3gqeenSpfWVNaPXU8rlOSn21ob+F3EjPjI4ROkmJB8FdiUoDY6HQmfo3pIO/KSjfXm9dTKK2yq5CcYhJiqOCttz8p1f8YYtZYeK235rPWtYLxuLlJH3t7bs/LDzelPKyyxjGxakCGeJw4UVy2iUElBFiK1vJS3UNxupESrSeghNQroRnmdyNAeOHD5z5swr6+smumaBfvdad+zYsboMkDCjNmk1bhK2hnK9LKa/zQKh9rMvvlRb39S4Xf7REewVvUlTMOO4tiHYue3OO146/vKB6uGf/j9+9jd/67cqeeIK6ofe/95P/Pv/8PGPf/ziyZNOnz+bTBYjuWIip3IBWh6emb4yf3VRazNiJmViub/87Bd/+Vf+w+Lzs84jdjoO7D04vrm94Q44JneO+SzOKm2yrOBQ8za/fnCkq388lM+Vz5+4Rh3/I488lEhGmeZd3f7NjUi5ksXyDgWsNqu3LIzr5Fic6P4PPPa+P/j0H/C4MQaoDjh6x9H/8alNUqm/+lO/9Csf/4WQL+C3BaC2ufvg2Pvf+ZaffMfedw2aDboUEQScYOYWD4/HpQh+brMkIFiBwCcnwL8MYnE9FYNNHqVIfT6KbmZYIpTQXbiqeFh4XZiAmBfjeyaNAWhEIoIUJAnE2CeAb4ASC98T4CpVuUwKEHZOpOX0tRnMJlpiI67FCCTchgfHacFlrQg6EHAcluQfxituV7tKAI+wE+YNVJJSxEzhFjMFou8ckaR2ze1xMpLZj8vTlUpXt8NZ3cQ4ZQGA4QtYCexIUoWMWo5zQxnKNco9UKaGMgolxPQDFtlGGeXyRr5ndCqvcmd4f2usd9bz/a01nY07v5Zf/msWZSe8yCHlHxa58TwAFjmjW7vlI39KtkA2kW/RmviXbKPwguAg0tOG7m00ygPWI/mgtgHBQfEKch6pBi0FGzCFcVMMegtgOFZWqUECBCV6XaalvkyOEnYI/GrAzehaXMwGpSw4mribBO6YqxhliBEq8zCVHBZcdRCIdqZwPlOKRZP5bIEAMu11cX9Rz2YjBKVtdAfwXpwJk8NOhLBaVrlp70zLa4eDqDeBJlQ9XI8chdIkgMqSgaiLA20yeND3DQtqsEqvuNhWRFqxafSUPQiWC75CkVAoOE6XGLq2sR2j2JQ9YMrX6ByUK9D4DvTp9ZkZqNQYwdvheKNS9PmcuUR6dMRuNNoBH8HMh53IHR8fHl/fWkcYgdenYwDXBoOmqgwqSuQgZwWljL+Xvrd909cvh2Nx0GDBnl6govDZHzxyhEKteq4wNrVnKxKJJRM63Ae7lXa6sEQBFVm4cJlkJEFIANDC1kvAv0Z9BXz3xNwBAIOAaRPVB1fIaGUMc2WoacB/NOvBDaMIulxpoHqhNCHlxpZQg4HGBeGMXqeOMxHZKuaSOoOpTXNhLaw9wgtYASXcqHf39zGyUpn81OhOilYBsg0PD1EvfPz48a31DZK7wxOT4Ds3tmKxdPHq1VlfVy8YL7XeBHa7kI5fOnucC3EMjY4ND3LWh++9+4m//ZLP665tbw8MDp+7eOnO+x6kW6tvYmdia40EBuXFqXy2b6APGE44Hl7b2Ni1a9e5i+doCkmW6cKFiwSo4Ud54onvUUy2uL2NkguGApj0JDxp79gzNMDNRlddOHd5ZX5Vnr+F4UovvBSwb3QPViCXpkQmRIkSfhXYPBOWP9HJKgKXDhhGHI46zQolKCzBZ+UJMn1vLNwQVqKJeNyofl4p/QLqoy6IE8l+2B4XAYHDD8jKszE0FDxBoqOERgAfAPsjfteQ4m18aMallP9iI9Boj0UmZmdSE/URuKX4K0rkWE6gc/RbZ8UbjBYmNQqbuCLmZOcslf3cULq8v7Wy80M5zM2FG9IR3LyR+8CZiCwRYcLSOeKt93xkM17ZAK8F2120N4YsTE6MP/we4bWgEklQ6NlCDmqb8R0TH/uJn0xkCidOnVxYWoQfEvIWBATwRuYh5qmRFgvwqPf6CGoSjDu4+wC1QZ/45V9/9dUTf/HJP2wJws5HUQvyvX9kcHNzHTqmXDmfyqcvTV+GOZEb6ujq2rNzipFfLOWCEMYP9oGRlCLxLk1mYdMT7F5b30aQac1WlE1kNfynn/oz2DB++T/+5895PnvutZO+MTdk16ubC6lCLJEJ146Wh7uH17bmEtnNqf3DLoc91NUXjxaGxkYXFxYCoa69B/d87nOfnZ6Zo0VuV4D+IF7o+Pp6R6LhXGR7C2JWi8m4FV0fGx+hWVCpGHn1xVentiYnd++5cunSf/ut3xw+cOD3f/d3/vKP/yKxMrc4e76/262q54Xei7Ipifk3NShZGY/oJAYDulRRvfIwxMuV4dF5FQqvGwuPjEEsbtmN5ybreUbsh6o8k99h6ulWaaChrhK3F5WKjBcHWwfGm/oAeP9IPkK/gOqm4gBKVIQniRVGCyQKSHeGHxYWfwxgZQTeiMHghHBkcamVjApnSm9Q7EVQnBxLVW6QzdveoheZl4xOejPsC40gbU+cOK8bQwE3sceJ6YFrN3Oy0H+B/ATGJecuOpQFFcoFSTrkf7YoU0Q24FfcHsXKlk83VbPy/sZaMWqYRW9SzPyGLW8dkndvXthYTuMfL/IM+F9ZzxtmunySg99QsTenj6zkPX8cXn4l+ubGms57Arb1qrZYqGRS5XQKKhLRzcB0oagF68QOyYiiP6o4jZI2pXYLLh8ehNRkADpXjBRiwGhKbB1aeFIZYgJrir1DJBqABr4m54AQQyyJzoAeAVEhmWA7LTaE7SyXA7UOpAK4B+WI4IZ5okzyzvMmjYfMJYoiLXnhaTPjPzm9/i6nBcBzAUvTYjAWC0gywtpNA1BHNQ+fbFIrm86BrqqUthjJVqtD8L86VSwZYyVSW+4PbVOohSiVuMPQwp44+YbZ4mDQM/IYaKCdrQ4PaDIAVnAgk09C59HrtG6toLPlprYbk5OT6VTG5XRzatj7UzumXjn+Cu/5mvuG5JUyWdzgagX/1mBxDY5N9h7cg1x4+ZVnWWkwRyLxDOw4A8NTMHKshWOjU3tNjqYZjVEtGbW18PoKqGRAWEtLK0hrCneJEuNsdGQsNkcjLx0VxSzmGm0+AGs68L2aloUGpNwXA8BR+ghX+SEy2kDUSBiONIC8QLJub6yHunz4r6l4mAZLQHfUTVrk5kC0AaRBr5CCpnEFCFVBU+jjvYODw+M7MskUuF6a9dDps5duTrl87/BYKl1wAWKpa1z+UK5ch0Br92jPtdkTe8cH4Pmau3YhmM9NHLotPDM7e/LE3qmdJ04c5yDAxzxu39b6Js93+vWTCHIAqzqw8rlqT1+fNkSeL4b6xNWARYT8n8vuoj6KTswM4kgyvZ1IgRkm0cSdpIIJbJW11XTpNO987LHbb7/9/KkLn/vsX5w9eQ6iRzfNv7lvTDKsdSyUmxOK585QVIQU1fOIIRQh6Q0VrzJ3MUGFWlXg4jzQjtbpPFneo+dYzx6QUIgkcisOlwWnhBGMdCNggNHZqUdiYJBg5vfgq9HBcEjxE8KqdJPRa3MMYAQCu5JZydyknE/B2IibQukYp4uQlFJcoYvibDm0bKuYBbzhfFg6QWUuiz1zSixsKbpdGYe8ska2u7nc+qhcu/KVsgG/Uk7lxv47m3G/RNCIKJOFn3BGjGEAB6Sp2CWsd2SnQJxxSK6dAECuUFHlKkYvQSAJf8IyTSjitrvv++3f+X2Hy7WVzcWajf6+PujKkhjozQZPXLw8gWI0Qv29h44czJVqf/+VfxgbnQwODkbnZmMzs/37dnE34P0o5TIu/9ilKxcJQ9HxA9D71O1HRoaGjx049MxTT3/6059aX18jyR/PJO649+73vO+9hJrOnTv37a/8PfcJ7VOT3KLhoYce/tM/+eyBo/t/8zd/89F3PkLBq9vfPTwyODjUQ7nG6sb8/NK114+/mimm94d24pkaLWrwpn39o4zAF1568f77733v+z/wrW9+lRkJB3UwRHftJl4jmTuL0WXwahcW51HkWGaUlgwNDzzw4MO/+Iu/MD4+ePe993/zK99enpvfO3H0rz/32X07RvxO89RIv0WfXfR6VAABAABJREFU08D2Q99PlA0KV3F45eby/Mmtg5PCOxV1zBBjtQigzsITufVoec9KniSVu4hbRBziGruMm+/t7Sa3QfqP8kdyyyUkAaaS3ohtDrs5bLl4XKSSVHpLo5CcnZnZWF3j2REOMuKYkTrh6YByhTda3SaEzXulyk5sQYaZDAk1hJSIbym7V0MVhCMM4ofTJKyqYmpCihLTmAL5PAM06fUPegMDuq5uLiSj0tUEAQwlBzVoIpWVK5Ah11FuymyVlbLmf7YQL+L6GZyUMP2zbRnKnbvz5j0Ickn5LDdaFvnEDebX7IKdiEmj7EpErPKtspkc59bCbpVFVvCGQcYc4V95i6ZnV7xn5HIv2D1vOn/MzRvvdfWqrpCvpxLVVEIFgxXxV7bhl8V6syy9BcnMIbUBJFMtikdKG74wcAYTQ89EuYX0PqLYkoJWHheuD9lEQ0ODNkQ7AO0mpoAva6SfiAwfnhNRbcwlYX9MJ+IUBUCMDMNwoUiFKwka8Y1RjlQuIfiQVQRD8EcxqItluJubAUqL/D6uLZxILa1vkrfBm0NsNnTmhg6gBOz5/A5xLVGaQrQ2MOBU4oRWQuB0Ebawpfi+jCO5Z8C2uDquFbcealiMGBq+mAxknStcBRrLoreo6lmq/lx2y9bKUhdFsgPmZrXc4/eAe6LsCFhTqDdEshZjY3RimOoXJD2UERghqBJvoJ/8UzROC9QNjP1MtgwjY1fI23fgQPf1K6sri9qWes+Bw2x65cql4cl9PX09Kr8vu7pus7scwa5aOgaNIyOHkADAqFKm4HBYdeqmQcv5gOq1YigA1GJSASMz253pYj3U0w86KZ2KJZIZbBi7wylgo2pCCFIIghsxi6oqfQ3Ys7O3J3Yd+qu89DBvVjBnyAxF4wlVw02xBxVHxITz0Ay1tINDozS60GpMhBCJYRNR9A4PFa9djSbjfSyDg6vra1qjte/wnbHMq33Dw7QvczeFaNBh1E6fO+XtHkwDUsWuWl/BFYYaEpGy79BhAsu+YNfooaNr03Ob2+H9u6csGmt4c8nv9THbr1+52pvvE5pSr+vKlSu4rWC8+/q6t7fW4etAVROwIgFB56w4hCGRLEBz0mt49jz95eUlnuDctVkKkI7ddmhpdpGaVAQG7Xaw1HniqGJmOmMAPUXYlzuEWhYgf0vV02vp7Q6MjI1iOG6tbiFaIOuHSJIJieaT0aIMHRQVggZtxxzCOuENytXswKxhbJJ1a5FzoQml1doCpA+QUyy3iuRIqIKzWez81mN3VjN5jEKZbmgtRiHOCOx8ZtDL5H4QvJ2T7JwnWyi+NUQEgqAWcUycUnGwbwgWZTxTNcS5dLozie/CCctMFokiylR+ePOVNfxeeVW+5a2yEZfJDzt74z0/47Vz1TJnxHmgYhG52qYlIxKXB8qMAnZBngLEj0A+4NLAYYGUv1BWmwFXOtauzf7xH34q2D3Apf2XX/nVX1hCiebf9pa3bq9vUEr3wAMPfPkLX6BNIe3EwskYpODba+HJ3btH79o1O78MUqRaKlIrsXbpIjyQajC2ZtPK9ILWbcTOVlOUXG0sXr1uU+sv1Frz12cunr1QAvqu0wW6e0bHJnioIICOHrvjXY899vm/+ovjL7906NCheDyKifahj334t//7//1Xf/mFgb4hf4h8U9Pm0sPh47QGMvHCM089Rexq1+RIb6iL4Or2VjzU2/XKiZeGBob1Fv3Z8+eIgVAPsrC0GQoFGzOrFEAS/3NYfa35BVXLPD83t7B0HYY5hy14qTY7Nrr1X3/9N//h77548sRpRGfTShItv2NysFLNWp0moM06k63Spk05ubCmUZ6RIJjIAYu9qCwtdVV5i38sIquzKLpCRD3qQUbMzbUMFgpd6gT/cT5MTYPHoOm2qepQMuRNVBsQrMC+xzow2As12rPSPZ1kGzzXHnBpOPEri1sUkTbzVY1JeKkQjzxZGXM8VDxc8PkUcdAhTUkP8/iV0aIlbCjpYanmUUEjUmIUU3ZitObrek+fZ3EjG3B07Tx6MJ2HEpDKF6PO1+0qlJOSW8aSK6dgXDDQn4JQEJfN340XtCCzQaYEt0UZyjdeOhfL6w1CKWW1jFV0YOc7iRuwlp+jVbifN37IXWOXMhnEguZyZCKgNMWO5df8SpTnTc3LlwI8ZCJIDldBQcp3RB5ItDJHO0dQwsuKTpVJqWgVrGmOIj9Fw4jNRF8LGm5wbpwM4hhlRwiNEqF8vr6+XqUHGtF+OUu0YxN2JxraN+l5Rxy/KHgJMVEN+CMmo/h/Gm0ynaceslSsrawk3S4Dxn7eLF2A0FuJeI6H7LA4eEL0hqSAoFKsU2rotGugqdBpaJVK5Zk26DEXMuFilhQDvcvwehEcHA2n0UEBMdlddC9tCyAEARMd6vGr7W5vqBvBDjEhkv/yhYuvv3YcnFSrlif2Va5Qwqatpmjx0e4O+WOxOKoiRpc0uw1FXiYDCmaklCaETYxD3Jcmo4Skh95MO8y2Ab8LdyoYgoFxaH19e2MtggrZWNxEvS0nrpKPpK3lyOgw6CdEMyE/EYRGM7wVBw7se+KJxwkV9A51zy0s7Ng9sbS0VirU+30DVoebObDn0IHb/PcT2Xr+2dcG+0PEyStLKxO9gxj/VpM9ux0zOuAodkbC65FMZq/HA8SD2MOVK9P33HlnDecOWFQphUjXmqROkwwvVPfUenFXEnF6n1NgY0MUFqkN8/f17T5CRfPCZpx2amSi8O6YyLCWABgHAqYt40pRXViJXp+lMKOcS1nhuytn3LqWx4mJoyrF1NRFlFvGmhqyPjcpBvqtWfp3nv3SX9PN1z84HImnx0aHSULQWJQGDA63A+3jC/nVGmNmaT7YHWS9D7aqIWd4+oymWrK2KsQHHQZHgv7MperQjl0k+Snx6uoORTe2NNZKLp5ugPpw+3r37p976QnIe69euNTX381VuC2mXCZayrV375k4dfokY2k7vKrW1F1Q61rsFAFrTOat8DblsYH+gcWtDbDz5XyG5wK2Ze7adViliObVSkUS9QSZJZ/S0pNS4YSJfyizQlwhJhSKmFA7eTKK3MHiolxh5AZaJW1v1cT2yxidJojcUMYNwfdh/StWXo0dA0khdEzsFbclnqYmW8J0wCiAlJgJ5AOZltZeTn47NdXv9vjSyQxHx54mwZaMJZFu1LQw70TRkQSROYvhJDlmFpE47EjRxIxVEyhRxQYQ8YClLxhXNgDqIqIQR4SsSscd4nAoRXatKE7ZE5JRcQE4PfQ2OrWzBt2KIJFfcVzi/+xPbgfSBUnDhZDLIDaugDxEmIhfLWfAsSQ61YKVRnQ88AVM65LSTQFYIiEfVoIzE/mUr+erqe5QEOjt2uVpjDx4scd6+q4kr3z5c3+NcgWNjMHsA+UUiTQDLV1Nu3Fh1to2v/D4C77unnvuu9dhtdx51x1Uhf1RLFIuUKlYodGx1m3GybOYbB69ZfX6GmUMuZXwi+eu9g70j/S6kukMZet9vYPAj0+fPkuV0cBQ/4c+9CP9pCdOaa5dv+zxuv7ss3+y9+C+u+66KxnNZ2IFq7E5uKs3X4p4HaHCZu3EiTciMVVXd0vbB4Wlw+MabFYcT732jMFm/PaTT+7ZOYb/AE5zctf+C1euvnZ6urevH7vqwL796Xi20cidfePvcCe3Nrep6oyvF6E5eOLy90Z7R3/p5//j0089/sXPf+6xt7/9gXvvSUcpA45eu3oR7Njpa4mNs6nbR/T7xt1EHrWNLLFguk9RryCqQwsQAGyEqGMehKR2eRVzR7DHJIUJmiDZcUPr0kK4obcbK+pqqqIyelS9k73OkKupjtK/zUxrb4FktawG0iIAA2nIZmWyG0yuCiFwrV+l6l5dOnPtwrK+bYV6Ml+SRC+szqRvGO0woZXzJdp15/MlQBXUnxLbLtUr9NiGCwUdDhoFEodENk/PuyTFWDaPWuXezlUD9onJ+/fqbcHtTDOB2aTFofCgq0mbiLvOOEPN0+5TnFeGjaI3GUMsN9z7jjaUD53vZAb8wEWcX9StstWbdiPzRJk28qPOzjBXRDkqOvLm7iRgz69Zx2bKpGAbZc7xrxxdNpT9dEygmz/rrJT1yhqUsuhI5SGxIes7oS3QIWJcKAKDiiOZ320TFWECmeJOCLRHlHS7RdM9CrchXSAnrheCeTSy6Hr4gyrE+1BAVhtcou31rTDoE7vLrDdbw+GE3a3NFsEBYQqbKoVaMrmNrIJMn+xA21Q16uh+imFK7ApQSQtZIu0QJGiBtwqYSJ6ziAkEDvXNIscQRyrKQyFSDgQ9NrfvxKW5w2OT4+M7iGmvbkU3E2m9DY5lSwXvI5WktcCeqalsCrbzFNUb6IBMuRqlSIX6WpMB6c9d8FN273XigieTGbwaynSgQ8WZphd4V1cgu51emJuBHEBvhEvBUypWQVRR80DHCIJCZnZKEZxWTX8Fnc6OoHf4A1BNf/u73ybpS0Hz6sbSrr1TmxthMDQImr5+DOveRCpKG+PmCpUUxa4uV72e3VqaoU8o0LUeMJTcHJM1USo7XL6pu++S3k5AKvOl5aUti9G+PLdeLNQ1Jtu+0SPJyNzVCyfNuPXwntSx8kBKYv1AzlOnfTLzkns+deAIFROtRNUbGtijM28sL0bCccoaoACC9KBVBQMMREibV2fT8SThT5Wp7naaNPUqlWR4czA9UwNOns1gcUYT+atPvEjM+eC+3Soq+Ao1v6+LDHr/0LAwKrZa6VwWxubtcBhUJ+pqZHwKmU4+H5UUSUQtlhoJBcC9eObpTL5uRI4EYvGUd7C295G3ov6BjsMMMLO4SEhv5J57G6+9fu3USWin67oGmo9q3v6e7ngkYrOaV9ZXreT84cdzOyGV7e/qWYvFIHYgCd1YCw+EQifPnaMdDQ+HwTa+Z/fcyhrVR2azZJVqJSCBDaI3QHDREKhKiT6jRsiXEKZn9omqwYMSx1UiVwxx0SsABWmACsGRUGhZ2hY6aeHmSUSYQNBNNHLHO5T4GzMMa45hTcIYYxzXgzkmRjVzTNOCWhxyKSFHa1DUAeKaR4b4hhuE7UVfyh9zVFSwsgatjQ3OZBfLXMQFnztoLHFMJYGtzHdRmXwlIkTZrLOlImc661kh+1SEAq8cRLZUfnTrV9//li3R2WzGgjpXtuQNt0RqnBRFLrtlYQNeJWuMsBULAO+os15ib4SvSArLNgwnwImMMspcDBarRg8Zzcbq5mc+/Uczl6/CKcyVkE/lp/NXp9leZ7eszS/jBGht1uh6GEIrgmrPPP5kV0+oVMhzc7/4t1/8wAMPqqwGYqCwM3cHA3DQxrfCENN2O73J9W3mRCEN+LkK7ilXLKHdKBCn3I4+32fOvHHt6iVyjBCjUPC1vrxssplf/NqT973vbY8+8n6s6mee/ZJJXxsc8F89O722lCvkG628arsC4G7+2G230VXlr7/w5SuLiz/+M+8HsE4DxN1T+/t6hgFrGIy2D3zgo4Gu7pWV1Weee/Whex5YvD4bXd+i5UrI26PXm86fuxrZSgFn/aP/8ceHDx2g5HKwfyAW337+xSfcdutAT++hO4+efPVEq2pfT6boRNzXq3c7CBjBUKsa6nbHI2k9TYkZU4r2lXGhyHluL5oUoS3cFTBUSfKcsScRSpPDmq4WUzg+TpW3127rsrWsxCOLFEMgNZQng20qmXtAhG2Npa0y1mnibKUc0RVe2JqfCVdL2koZ0IFZjsowBLAj9MMgbYHQ6AvMjSI+cRWLzWSzp0sV0DuNsioEsK5EOLRJh+RiSV9Ue7SabpUusOeOfRpLoNI2bSbpM6KzeIJ4LqgVKbDjbJQRpCgrLlKWzqvyVgbSzUX0379uYVQyjm/9RpksTINbK37YGxnHnFZnS5kJnZmIChZIiLJLZSdMRLZR5jD/dp6MrMG+F8dY8YBFcaOAlTXyY8Jp4E54CkItyQMTcUMMmW2UEnyN0tKe4Ao6UQ15kNT1SRIBqK2O+Qs1BuA3vUGaDMIWBUlcWrgRqZAr66UYWOrZmfaU3lKIGewOoYjgf9bAqYS5QTKA2jYtZcdIGxraqeHvkGyXDtEoqDnmM6YHcgu1gkmACU19Do0GieiSFyOTiRCcm5nZWt8iMM0D4/aCu2Z8YJHBeUa5DpwSpKvpKl+vlJ0e7+rswv4jt8OYFd5a6xudABgmehpwTb3uCFqVlDEnC9jZTf/5pa2o3WSOEHpJzEgXWkqipRqqCrUPFb0UuRGHT2eNxgr5SotNR8mPFi2CvRmJRGkbypbwXoGZgrcSNDJheEipWHCXgWhBEYPVHuz20NAplwuXs3kH8HyrDtVO5GBgYLAFSV2VVoYOlcXb30M1xQbXbgeR6Xalk6vEc6tNHa4+EaHFmQW/02s3VQjvs3O8LS7aBvZXZzl58nVvV3/I72duQZUCMrMKsCQVp5uwlLvw4KAW7LR7YwyoG3afSat2UoKAoKYkFeiy3emxlfXTaxt2f3+30Y49tLa1XXjyqXg6Q2uhYpbsa8tmtSyvLaHwpM9BRRKNkLLQpnx6bsnmDBx7995+u2NjZWtmbk6Tq/T5/BLEntjTt/foyy+9fvX6StfuI8G9hwrFE3ff1U9smZvTVSXoWJ6/Pm1X11xm9W233bYd3ozFk92D3YlsnEe8Ft4IBIJUiw4Pj1TzZTCAGxvh/QcOj49qWqvLsALUKmW4iuLRmCMc5ko7EWbw8Dlq1YkRENqAz4yQDwMQ6xMPkOHOgBM3UIY3qfpKTWlcSH4Kcish2pOosAfiIlE+2rYdUJpSBSR4KLbkZqAU0d3ABSjrFD4s0YsIKnQ53iE4LyYdEwct32pm00kMzyZ0UTo9VAbwBPEqmluZzKK7kKYyl+W8+J9jykf+VxY5lqJ0lVktG7BaeVFEl5IS7qxkPRt3fnXr/c3t5VedI3b2xnoWOfjNBeujs5It2UbsAVnYp4hE1nSWm5sjoNChrFdOT6YurtqNhV+JDEGIgNzRE96vgqUA2wGU5/TZM3DDcf3iwldbVq+jmAOoJZQ10pHDSBapnM4UQb/3jgw/+ta3vPrqK9dOntx7z52g4Xffe8/VS+d27pzyupzz167RgqmGJMir1mvhrm7f0QP7Kc+lCIF5h1WUzaTLiRiIdiz47oB/4fosHj6hAbJmqkK7oipZum2Q15IKec973k8Lki989gsXntrWBlTNpCCC+aNNCrD6aGwVoPv5Ewu2AdX05YsWg3bX/n3jI5N+fwhQxIFDRz3e4OsnTs/PzPq9AQLRDrur6ir7PYGLZ88BSAIBQDsgsiFc3759ezY2l5fmIa2DUStezGcYqz4Ptmkyv7Hx8NTYnh7LcniePAWdJtwezcJquj/kwb6U+y8PQfQVb5j7WHYwdGI/4rDwJbdeEiECwccYJJmnomDX323s7g+RpWu3yzx84fDA9JR3CHN+RYsjKhWdGrVHI/zPvtRm8eknT7z84sXNtVw6hfUp1FdoYMYwP4UMm3GKHMf7BRUDMIdyg3Qxznkh9vV280Yk43J1JfPVjWjeNzD50z/7iwZn7zOvnYtl661KW6EtVPcNjoxN7CDmsby8TEwGaknGGZfFNeGHcW4yTG8s3x/JrH7zh5sb/Mv+ZbwyKm9t+6YJcmvdD3gjA5krk39ksolpTKxMjG1l7t1YyXvZOa8yl2V7UWA3ftJRwDLrZCWvcG8yK6TppzwHtC9kF1Ax1uiaA7QYewrjvVaVHDsAT2H74amKuiTYgeTGjodoiF4DhDkQJqgevTdAf7mU1++ORlM4c+S5yH4iXMjA0SwBngQQQOHtIu4jJ0DuvwJiCyw7XFOcKOH0SlYqC6gr0MDRiCfBn54zyxeozDBgyuGYtCgfaqeb8RhytNLUUsNAfRFB166uECXwJrDrWlJCRq/Xj0W8GYkCipZ2uRY1NvDQ+OTYzt2zM9Op/Kw30E10CrKEoYHBp59+WlK8ajznstUIZTS0OO10PJrKSdQa24BjUU7MNjKT8w1IxrFGm6oMV49/pqPhPC16LA5/sBuQMMRSAC8JbFIJRwEwpcTRaBTyJrQjrB3pTEKtg6XSCBLZYHA6bRS3tRQ6YMl2U0yFIpPuhyrtuVdO+ntG3AF1vtgK+roLxcrGZnjX7gkAXtHN2dXNqDfQMzHQywzfWFrh2RErIDeJEMPq5VExDaE7wbh29vUUV5Y302k4lXjSeNSkuMHPsR28N2gPZKoUmsLHlWvlYOqz0FlZndOWLTYDcOuaxnF+dmP16lWKkWmoAHRiaKA32NWzvL567OGHbR5XKBg4+8YJspUySvTUe4FtK9hR3U4HIAFVJtGsNMuF9MDQoM5rgJQhFU72WNwqu488fTSZKUXSlpGhXLbYQw+p3t71ze3qide9geADj7zl1DPfKZbq6VyegbyyshLqk54QUCKkCjlEC4WYO0Z3ZrVpbdMwffEKHT549vQ827tr6sL0NEkMGk+df+OM1e012x1EtKrlWipJNVsO6x1jq1pm/jBNZCqhd7l9AjpjbipIZu4kuUxAI3xJdzxUIJKBVzL9mFY8dKwN8PyYquRQbkxXZpx4qpI2YpzKLMR6JR8mgBl0D1FBjFopzu6Uw6UpfgNxLW1oxAwmicbvZCoqC/uUPShyQs5KmSKs5C0flTNF5YvvyyIHFIObb0X53XRYO7uSDTo7kv109nhzDR85Jvq9s57Xzh545VfsStmzvOHa+ZY9M9Q722CCy22RRBhHlz3wVcdu5mz5yHLzK74WmcoLtgzsYzwsoHPpdMbssGKysCWAZBX5PpUK7UuEij3XskW8W4yhYipt9DnJrAsd9+zc/n37PvCBD/zuH/zuJz7xCX7bPzCI/X370SOFVApHma6OjoCV+v61hYRGf9XqhEK1FfC4J7sm1rfDs9enmxQsHNy7Z9+e559/dnVuzuKxcjP1dhqRwFqjm5m5vrLwJz/xYz9x/z1viW9Hnn/+mVqOsK2lmi9B0NcdchbrqfPnT8F/5w6h6lSri4sH9+4f6u/xusAzWDa3YpeuzqVyJUrbDx88Qn5hZXntyIEjxcHMydde97h8W5vhZq1VTsfyibTJaauWC1QJuj1gAlXxWGYjm75y6Yrb7iWUX6qrr8xvxtbK733LlFKOW7wyuzo54AHiIJwtSAhmKU8c05E31ITSpk1aLMAhwzcMCQQLUqQNR2iqUDN7Vb29Xl+fV+MAXEwvEHJtMhSI2yn7wP2hhatZLXrXpWo7tabeVkl37szVV146e/VyGnUAFTepdFggxckm22YgAt6Jf6sMDiMtJHNlaBKhZCAAQdCLgE6JUPbqVsbt6zv44CNb8er3XrpWUa2lYUgipkGpHngNTlqrw1hHqiwuXJdyTBnfBFAUs0CGvWLrMSyU4S3//m8ujFdGmwzvG2++PyX+JXvmninDWsax6CxFj6LMOqYCO1b+UJyifTtaVo6E7Yl6U5S0PBx+qJBZYhqJ+S+WKd2JtTIyCjXYm3NppgEJUY7VQoQiqxnE0BfiBKg0iHg4ohutkhKkBkpFZ1a9we3zQAoa9AZdntAjD7/1+PHXo5ACwhJRqBChJZyLqkskw7jePb2+fDYt40UCJOQn5LpFzgg/WpvemjxC2itASyUSECEjWX9MBfIBwkoADSYSgDlbQjy2mv29Peg20sO5rLiWbbMVDwmXAzlkdznX11ZgctDC/gQ/YjgKze93n3yarHX/6GShWo6kC9SM0kZn56HbMtk8srUX7Q+WzO7u7u1zdQWe/qs/QcLCzAUBMrY4go1MEhoOqVOutg3EsBGsOlgwpdYY6h9gQeQyr0xfy2QSNkBSpRKp2aXlNZC9PHe6A7eKTI8yAWwuXwhzy6l2s8g1QcqJIQTpMY+vI+mw0qsqG0gqWK9rZbXf4w52u86cORWJJTSqwtXpWfgTd+2Y5PlkCmWzzckOecAAUfVi1aJ9qMcqj49NMbjzoOlw/PIUT4fzySRyUwIc3GsMIHoSEpWV+hkUgYp6M/F+aw6cyFYtK9X59Cc26Xbu3Jk6e0ni9pE4F8IEARetpVWVjVxtEA4ziCdPHX91aKCPwDo50QbdJH31nZM7FhZXMltr9FhRNSv79u959bmNDMFbnc0aGBAkn8NvKqqWljd20z6GTL/FMfXw2/qXVy5euaqx1imfPnL7HdVc7I2zZ/btnaLSY3Z+bnLXzmQ+2dvbf+78+XyhPDOzMD40Wi828TnOnLkAeL5/eGiku/fixYv5VJKYAuMgmyBgUcS7grwF4wPHV5LBXANGPwBhBh4vAiXuhGdlfnGvUMAoD7ZSvpfNeGCQZ8FnhLXHkwVDAFccahlVC1UklTPoJjQ5moqMCSqVWDeTjilKVBYYc0dl8nBBmNhcLnbLPkBsM6mwl9g5LdcoBFDC4XImTDWZwzKxmR3Ed76/yCOQI4FdFdx1Z1F094337KSzNdtx3JtaUHb1Axd+xvrOj9/8RjlQ5w513squuFFsyWtHAfO+o5XFz2c3bC4cPGITKDuU3TJfiFxKEEButrhNFUyzknBPUnFAFR2KWm0WomB6JIDKoJcD6VyDG6JsQyGaUn4FBLRcNpeW5uahQEfz/9Iv/dL/+KNP9fT0CBmOWv3cc8+FAv4deyZmr82V4hmLA9o7FVgtbMtINIEE6x4KEKmCE95gMT18z707pnaE15fWl+aCHg/dyUxkNbU6uiy53YEu7wDCaPrq4tpKTKe2Nk1wgWioeAwG7el0GFeSNGu5PPeWR3ZSLhtPZACpLs5dhyZya2ObPqsIElrsnVkPQ1YPB90dR+6AyEiwpICHi7AvK9xQNjs3rTIb+d1f/c977znQ3eNDEzOnIPDJ6YvVUvPAwaOX6udz6RyBsdcvr42+545wZmFoapycl9YiQRVxcrmdygBBR2maZJ3wiowYljKUoEPiKVAvJCq2rbar/P2+wHCXyoraLjI28X7E2+QT/ooQGINkg3jSrla7tWpPu+VRqQIAr15/9crqclpBBapoFpeH/0YyNzwcWIwlU4siYYxTCmpv4PbY7f7eFLMtB57LFhwZmrr9AXxflcbuD43HNZFK015v281Oik3NCA4CD8lMbG5mvlxMYxNIL3bIEW9kWwTjyGARy04WGYT/HyyMYobjrR0p0/rWpx/8RvkJk0NwF4r2ZUALLIIhzt3viFFRizccX1HMyshHXtxQz2wk3q2El5WDK1FrHF82YCjxRnLLQHOonK6gbBi1yqNlvaKqMdvbtKqT3Dj+GRYJho/UG5EDEOiwFqyKFMNQT4dAHBx01hvZGPZMPCmkicCn9G34cWDxMDPBjHADkR4tirAShm45hPDlinhCzTaok6EqjwAmEhLTCHYP0FGA9giZ0tWLGAzRHHQetESpdAJNrFephoeH4fjFuZT6ooqFHeOg0MGIapnllTV6+zhtUgkI12h3n7HGL1LpWGSbUjyGZyAfoBANhYGw8PgCXl8IzUSrAeBh0pZSqx+anFpdWtuKxLBXGN+UOBNfdLmNAIuA8xmsYNBociexMkqKWGhdB/H6wQOHT556jTPx+YNSk9Pb29Pdh7TCe8bBZWoYNEa33eb2eJqVeLUB8T9cAjYu2GSyl2tScg/NGJBJm6eXUtFQ7xg2BulPXSp55MjBfDa6trwU297WNop0QU+G12LROIlPgq5E3QlfCwhIoyFFqVGXF+bnMvkaQppoPIhEMpmEerjZWBW4KDx3xBOmFOfPZCC+RFy1ViT7U4XvjLordaFoUqFadKODfQeO3Obr6f/240/RqBjIKH6q3eteXF1b2gp3+dzZdAICDe6hxRBKxGOwWK6vrsLDQLS/XMy6PAG9z0k5GV3lcrR61BmzdMu5PA2p1L59B0aHR65fvBSPJh7/3hN7Dx7AK8Q17R4crpc8unxEr6ZgeIRSk1CXv8zo5DYWynobJl23BmR3InUxfZU0RiyRXl1ZIx9CS6hMIjk6MLSZSl5fzbp7zTWaFNPOiQbDYkHKOOYG8ziZ1bi8XDtqjjnC80KJMroYDDxmpCHZXvwSSlehpGElNw23HluEsrorV67hw6AGjCYjd1DUili38j9KV+aTSL+mdOLCWJHECfan8PxJcpT2Hjo9jjTihiAecSaoY3Co+ZYYD5oXv1Y2lJOShTfsjVfEhaIGxT3gPd901nc2472yFV/BwPp9rdn5za39dDZW9nZDqIkUVn7ZWckheHNje2VrPrJBZ03nFZddcbNFoPAVbijaV26oJKdE5XPtZIv5IQvniZUvjpZyt+USREWL168FOFKu4T9h2TMN6V1IrTAznvQB8/qDP/ojJHG++93vMncgnpO9t1rApA8fPkTw7anvPv6Whx8hYQGT266pqfNnTlM+GNveotCvrkPzQmQHYV+x1+0eHuoXBkoSvwUMsDKPmN6WAZfr0J49p08cp6oMycj0paTQ4gB/mq2XVsnJHTl2DHf52tkLRijqNvJ3ffD+nbt7n3npa4DwhgckwQSh6UY4ataqp8bGqWbbWF1Z34jCuUtx39zyulpnCfh7fCPBxfmlcqbsNENn5zOqjdFwnM4xRFzgrBu6fQLH32rWw5Dj8zo9bie0bsS/xibGdu89dPncrMqir+jVS/H1z3/12fc/uO/87OyQ20IbXjdHlaGi6CVxEoWCl3tPSk5ihESh6WHeblYINnDTdKr+HT530CWdjeDtpWkc1jMmMege6SPHYMJAAu9hAZJP3xy1xqdS+7LhwhsnZq5cWsmmBdhFSx5mDhBYuFVEM5JEhB5Ewh8C/SKeDnByK11JZdJEwR95y7ve/e73jew+pHL1fe4fvnXy5KU4PdRaLjpJUKnm8gaQsbF4ps7MrGCj5FUQBGlq0MELLFDGh1xcZ4p2htAPe+VEbgyyH7bFP1/P3lnZGcedb5U1Ms3++cJkZTYjK8RLVYYyl85HZTowb5joMr6VWcDXOKayH1YyoeRPHF/ZQCwkZX7LxooCJgeMDpRvsdrFD4akhxJYZoU8VCMl81oh3xEdZ9CaRFLBRwNZFaBRIk1ENUjSclLkgCUYW6kXMnmARNDnroD0O3X6IrRQNN90W3HiVCXBGKu6gk6f3UfiMV1MeV0S+IQZmMOZTSr60aO9CJh6qIBD9uMgGjTwEhAJxcgi9UxSVme0+QM93mAX9U3zS1TU5ZvqUpCcTLu5vbXBfIDLCfgVPhDUvujBRMJCUoHdxpIp2tpwY+wuN+iqI4cOzM5aEBNTUzsJGNIsjCKhWDodGhoNDQyUQVjRmtBsTxWLa2eW09vRDMY4lcqgD8SALeYy9P4ldAxvs4quBtgKDBUkNbzITAgqgSBxBIMK6QeBfNpNIJ3vf+BOinx44pvrGwuLEbIvZqBTDWoA1Fgj8A6irkCNwkStB8kF7ls6c7lTpTTqkporXyigcvntheLK0oLXrrbb0dU1t0GTyhYWrm7CJfnOtz26MDdL4r0GeQqhbWrBQLlIutwchTuMFjX0vuM8brJswvlg1htROAwtBiO3G7eM/LMNNkqPIx7d4gpbBgkhELJu6Em+81it/V0elUV3cO/UgQN7ktkcDtBroEnpXkXMsCr67eEHHkzFIlQlp5MppC8sK1vra0SkgY/Tom7H3n0Z/MdUnOlKl5u+/t6VpW3gYELyTbub3v6dO3ZMX59GrUI2r3V68a1Bb12/8oa5VaTQktIUyEmsNuPMlSugL2mPTmgdC4FUIlDxfKa4OLtgt1B2S7euYjlf2Ds5ZVhbnVu6mtgqN+myTpABCdJSgSeHFpogOaJKgBWKhyrWlaLeGA9i0CudhQBzCo2GGDPkWhBxCJs2jfbIo0sQW2jGsdU7wV6Z0SxsyFQTi0ZQV5Kj4hLAaUlXWYu0b8c7gRNOckf1KlaXgF5oCwHKGhpRgwFqa3bCI+GIomOZqLLInrmfynplhaziHGWmY2x3tlC+lZ8o0oC53/mtbN9RwLIX5We3VOmtH7Ke+3BzwxvXIgdgUc5H2aBjM3d2Jj/lS8XykHNjn+LfC5SSWgZpWAj/tvjDcnDFvpFDILnEjgEYJLF4PnFdQu6mIpcPs4M14Mf0rKYLCOJ8LLVlNgF4ZFeDg4MEJFIbEbavw8puMn737/4enpgd4xMvvPT8zr27aXp2+fLlo0ePvuWRh/7br/+XXDhnddEVqsFtRy9RLweUE9rXZFpEm8kmJZR//+W/abXKqVSS7HKKiianXSIT7Xo4kRkeH4Xm/NLls3QMAyCvqqiq5bzKqjpyx77xHT3TC6cKpYjf20d4Zn5mHSAIBhnsit/91uPXLlfdHtXEzh3zSwu9QxPTM8uhwMB/+KVPhFfDl89dfvm5l/KJLGZ3rVIP+IK0p4knIocPHXI4jS+/8pzbYwU3CdnfmdemdRbt7l1HMtmiL9Bz+vWzzR7/vuFdd905cfn6K/12m79/pJ5cb9azKChGmzjBuKF4SEpGGDg9oHsGDzWQRQKI1PW6oKEz+Qc84GgxqEH9g0KQMiJecF/pVkN0SvZE9NqiEQfZqVK7WmXzi88df+H5U9EIsR4VtQ/wNlI1h5ZH4fHkaOxLLQvP10TvR4sjWWqQz+/pGfnYR949OrHv6G13x+LZbzx/4ZWL39TBguedXFgJGw2uYP+AxUR2MsmYITJHm3NIvrRE/aiNJu6aSYmzJ0pJGWYyX+WzYmXw7p8uMoD+lxeGsEygf8XCxFAmnihU3ogaFktbFCpnqeRwOh/llYW5KYNeVKxcB8oY4UJGh1/JehE6/BGeaKG2mS5tGk1LCI2nwfMD/YxCFuJlfiBBNdht+Em5TQsN6crCeLXqzdL6EX0FH6FFMvB0sVtYWOMXQLpGhq0A40pZVTmtIt7WP6Ax65qJ8Bop3vGhLrfDiTsIBBSf0EhFu5ne3YZalf42LtwPDAmeAC3Z0A2YzrWWJhDoriPxFERVgz66pNWw2cx2NB9hK+QkP+TN+vo6OgMqDBKcg/298Ejjfa6urtrszgr9sGii3qzNz12PRSPBgJe2RcSNgemOjE+8cfoMJMNAJWmy4DPbGP0RXHgavJMSpi9Q2xTeTuSyLatVZ3MBWmpnsvQRc3V19zCa8eihmeZM6Hd7/vx5Usszs1dtdnu/tyeVIR3uw1nnhhMrg5AZCiriCOB16QkBK14+HTNKIBT3wCBNvQwYowZKlcLRmC/YV6EoWV1Nh9cSVy4HvMFaKbE+m3U5jG889zhIKloPVNWmoNOOY4nu5O5B5AVftcvnJRnFPaWhWEufgU0sEs0SWCbCRlUKo0iheeIRi7JBPkpAEXOcSHulNtDdV4SCL5chnMAiCX9jjS1a1cLJ1182WJ17Dh+l5RukXT0jI9Mr5IWndx04wNl6HNZoJBkLR3KpBFxlkzsmIcucnVtAoRZaajo87DAcmJ+9Nj42TOaPztFceCy8hntMoa66p5vIoMrp0Fy/xmlAFBvSmXnoPDVfV0iVj4ejkfvvvx/HfnFhJhlPDIwOwr0FqJWyxUwqf+cdd33tK9+kkNsZ9GgalUQ87kR2Gox2s+3QvsGTl1aleQHGKhkEHaaegMwL+SJ1RggvkNly9Zj0gmfGG5DxjxJD7gNP4z2GHXNJSu9Iu+npgWFdIKq+tMropeqRWl76QYE5R6AQrGEf+B8SA2RCCrQQqgzpWItOIp4qU56aa7nvLfSxVp3nDnPQYrOmBgmhbcGPDtEpm3F0DtqREDdOScQlAkOMAPE/5Bx5VaSTsr2ymdhTaDhZLz+Xa2HhTWdh5a2FNZ2veFVWdrbsvMpXnS0ZBZ0tb/5QHFzmPh9Zz7GUhfeYGqShcMuQpXLf8IlFvojwEIHDHslTiZIQAST7l4iXiP0GedBKNseVwG5GbP973/kO+SKtQ0Lr3/rWtyCVo7yQPp6UHHBgRmsjlze6nZfPX4Dom2wx3dR+/ud//rd+67eee/zxyR3jdB4T+u52C54TEgQICGwtfYsepjw+tI7EP1BS/H/i1VcJg9mlV7caCoPt7bDVbQsFfaury067Z2t77Y0zrxC3UHlE1lq8KrtbXSylsDPo0BfbwvDTA8/sHXDjHzz/1DOXz1VRpulMafbarMZounzhUqh3mAf153/8mf/zE//VZXLF1+LPLT6djiaxtQoY6bkMlHBPfvc7EztHeZj5dGZ5fo7z8gRNx47ed/Tgba+8fCrUN2T3bZvdvqVI9LWz80fHJvWqRJrOUPmiF9GI5OdGIszlUSqKmNiJoBjo70smA1ggOXWVp9/WM9HfIqHXLOEFGex4vk3qslEAtJkSRxwzUvQ3YTwKUK1U/apU9kSi8tKLZy6cj0koRfCwWguNVuFS4WbK3NCJS0z+DwVscJhcobtvu+fl1053Txx4y/t/plBWPX1y4dKl6+tRqoa7N7crVG/2D+/tDvVRjRmOLuPi0EqW5LTQW1UKzRx2J/U1apXZwelz024MNjFDxflU1JgM+huLMvJlKz4rw1jW3xzQN7fpzI8bn/53/kFB3jwhaTwgnjD7xmGVkQyNgNQtyBl2vFgMauUrLkFsFjLa4vhi5nfC0fxcomPMHKoIxJEViDroG3xd0r2KGuCG80QlR0irS+mQagTWZkSwm9CvlKDWunqC61tRamwovCw1mgRBB0MhwEH5bPaOYwcRTDPT19GvjKpsVDXSo3r3o3f09IY2N1ezmSRUBNSC+1yYgSW7Se/tDRERha8I8DpGXKZeIhGLl4P0h8IK6Qa3NC6Ulp5JEHGYjYFgCDAU7MdEvMF80dqky+PCn2N2+UIhIt4Br4ci2Rz0XcUS3evAfEGZBDkw3WZEQ1P32gJMXyxk4jZT15WL53CmDx+7HZAY1igngpB1OJ1EwKmW5PolNgOvU7kGZ6EWJ9zaLJK+xDJRq7g+uhFxN6w2E94tgYpcobi1FYfMZXFxEV0bDPkHBnsoknnppZfpkUB2BPDI1SuXSCX73FYUlUQbalk6mWdisVCgD7ZUaoLSjZKHzoM297DfaXT5L1y66nAyq2NOh+XE8WepD4JYLJyNjdA0QfxtVSxTpHPvUjyCuIGNxJDXE+eyOVwoAycFpllhLdm3Z/fzzz5PHRilApBMAqMg1geNTwti+mYDOCiKhCICYmi+QDCdTB49cgR/IpVKkMxG3KOBYCNqVAsemy1Xyb/07BOU9O7cf2B5bo7on7urPxqN7duzpy8EKtk32Nf7xLe/cfTQIdTdyZNv3HPv/ZwA5Hxur+e1p57KZNJ9oS666xFOv0LU3YKhzRWnXn7y2+CcF15eCvT2W1zOXCSmtTlT6WyuXZ1bWOz1mKsZaD6HFq9dnLk2PTE1Dn718vQ1Gh2iZREDFy5cIgexMr+2ubk11h9CqfFcdGQWtbqRweGr86tknENDwyvbEbpF87CIHNx2+x1cIwF8uD1jkaybMkuDgd7OdIhinExPT4NQZXqjVpnXHXEg4WKTCTY+KkpxaBnn6CHiFiTScACpHSfA7/M71tfjdhpB0VeqWMAXAY4GhIFYEb4dooQ3Uk+rgsEGcCKJ5CoGF/WPiQTl2DWXy1SkdF2mO6axhlOiSA+rCGiFkHYRPlS4dlE5XCMyBSO7VGgajU2JHikLv+Ss0IWdtAIygjWiAxUBxXrKnTranfUiQZSFa+woe2Vz8fhZ2A9rK1VBd7Mo239f3DGEULFEEdg9C1KfvZGWIpaCi49yVQBtoum4YqJpEm0Rx0yABorg4h4IHJqTQ3sfuevO69evQ0V5+PDhp599ppopmH3OJPTnuRw7RyszCDk2qFkaAJOJqsOhyoJ/0GqfO3P2N//rbxw5dttHfvRHPv0/Phlb23IH3Olkmi6Z25tR/F9+DoghlyxarKpspnO2ytyhhhiK7aaUbrMzmgGni1mawJktumQqMb5jgOwPJPBoJcyI3iGTK6CKbq9t0AesYbx6esXkUu8/sDPotZ09SUuRwvAQ3cOa6SRlC/pErtIuqqh8S0SvVYbqv//J3/+R9/7oYP8gpCLZWhIU4djoKEmcQiuOK5xJpClA4jkR98mlmgFI0/Wmp598Jpko7Nlz20/+7M9/9ctf3p5biqw0Fi+pxn2qFb/qHXf218pR6VCAepKYDc1AMDYEcUCaDwJA2mnrbSpvv87b6zZ7zQ1tnrY1YFgbMPpXSowGslQ8FygLVUaCOMLMZbbYmyWzxuTKRUsOh+61V85S94hULuSJcqqBKrRhocU/ozBXOJHaBPKhG4Pwy9s98gd//FlPYOCDP90AeHV2Njq/Eltbi8dhA2rYU/mWPzCKBKY/zvLyajIRLgEvAsKNF0epO1pH4iQYEhRpQCMjkRGuCcuhMzQ5VZk2P8QFZhsZ2f+SRRS5MlL/5xszylnefLzOCuVVFCdfdTQoh5Zok6J9lTXKmTLIFcUs+V0FGseMEsdX8lpv8oAVAh3ZhpCM2IL4vlD/iBqWkqJKWzKJGjOOLK4tE5EkFrY9UTha6PT1eCi37Qq6/AE/ccrxwaG3vf2dX/vmd4YG+ulM4HE5dR6XulYGB0REqt9vuOvAjtHBEE3U27WSC44Js9TJNKoVJ66EFfAhJIMUh5LagzYSOCgNaOma7qdTcLNSSmWzJDJ7B8ZGeoeyhbrBjLR0luswHuaZ7bSGpbV5ie7WUCM5HNAg4E1kAdsUi+HNTbzh/r7u/v4Bop60FIRuN5tL06vBZtY1Krlx9jg6Fk8kxIvSa3D4ugJ+By3qGCa1MpWChGfR7kC6aHmXJ3VUg2tGmOAgbcRdJcnBNCPoTKkuKTxiKcxkmgNyerjgmIek9zY2NqCpAxXIR1gmaBtAIhFhSmUqhbBWjkp6hjIYWoBprJursb7eYWqgR8Z3w6ddqWuKlYLP5IS4h8spUaKUiVvMaqdV24wnza2CuqqBIhtEiQ/NrDNsx5NSOSbVFJQe6WAwESih1jA2OtET7LO63P3TMwBViBUUisXB4RGUMSiMZFLaMuKYAPImkozAJcTkNJuXV9Zdbh/8lBvbMXRqyO9MZigbq5Wxb1CQDhtNbbgcoFTc3lS1hYMCTTLO6NbKCm2Dd+7cBT0p5Ldve/RdhBDPX7zs8/kZh0vLq0ODfdevXLzjtmN0aDfb3LjiC9Obeuh2zJYLZ0/4gj0Eo7/6hb9+70d/grA2fVd27xzpCnXnE+uJSLhEmZbNuXf3nvWNFe3RI2PDI2DoLl66vmNyN+00QI2+863v/LVf+eW5pTKM0UXA3lBu0osiWxgeHFzY2ITgQgSJCqBpkkHFXRoaHqU7NWtIhjGc6NQ7MDhID1qULr4vQWSRU4onyuxD2SA3UaJkYSCOBDLiC1qi4QK0M8oc5G6Ji5HN5QLdLuwVODLBw2C/qphOiiIX7IDMTnav5EZbagrk6lgEwMqbGo/XSjsQ5q5i9N/wgLGgRVeJO07tqZB+MD4VjHFHlIgP6oBaTMLY4lnyLRuLZmrQ2sTEe77itXOG8u9NjfvPpdAtR6LzlfzqHwuszkfWKd/Irjgozi7H4uHyK46FqUy2ULH4uVBZeWO3SkSBs2WNWASd01eOZHE6B0eGz5w9S6jhxKlTzz77LLdHbTOWU1mb38NVowVhH4c9WA5hMzXRbXhf7Iad082HGEalAt7k2aeejh3a39PXSziK54NgxtD0+l2NSi0SzieMeRxdE/aamVo8M8XcTBSUOklfLFdmDaqFZ0zHkq0svTdUvYO2+YU1VVLl3aXtCrln5xN7jw5b2dlmttaqJOMV6nms5sDmOjmRNnX5FEwRHaSHOnw+XD7ejtFGn29A28WVxSVauG6urA0PDN59113f+OpXMVuw0X/x3/48XsGf/+4fWek0baZnMi4RqelmbCn3Yv6lbLqs1liMetfuA669+w4U0NC15OpWqc+pml1S7RoK7x9yqUopWuBwN6EzB0rA41Cr6olKjXOjSN7qM9j8Fp1D3dBBGlPH7UIPiPKQBXFO9hZxq6s0tIr8bwO6hEyvXmhAcb26Ev7u955OpmE0U8XieBpai9scieUdDGhuHAFAtTrfyOqNzn23HSVFd/n6VvFKLJlvxtLVZLaRytboJ1JtUs5kGRoawUsUWtvEZiEdht0aPmOFwwquEJxCYHmcGmFEnHNZJEiCZSmvErdlxNwwEpVv//lLZ5t/vv77axh1nQF36833v/vh7zo/keEuRqlykjLmlU+K0XhT9YrvSwxZ2VIQynivCqhSDioBZrSsKGB8ZBStDA5+yIOQXSn4Z0i3JRmslBsxBtgV+Xs8UPqpsgdAI0gBfGLp7ENIo1nPZ/NTO3up3AINN7pjF0Unk7unLl+9wlM06TQmvQrVQrKwXCpwU0OD/QGPtZhNbG5toJ5Rk4gSVJxEkGsABQhVSckTF4YvhaZxut2UhySJS9cZNKQVBXdcEVxBO5lJq3IlvdlBBo1kJlfGdOL+kR/CjGIveC0AW6C/Z4JRfFIq5vGZqOKN4WLX5SKxX2BhthjV6VrR3xtw2UxbW0WQh9HIJvJ6YmxEbzTDRJQh7EwLpmSKN0wVaGnlNLAWBcMGPlgYRCjHYq/Am5E5mTRN/SyTk1O0EL56DWRygSYnuHpZTG46ogBl9nhoFwiFA/lp4Dk41YQ78eu5CTSESGZrQ339iRq+al1Ls4WhSZXRmt6m0CDbzubpQFwoxfPpFBksu9lCxwqwgs1KHoJkHdYTGDa1FFgzkindKuQLGDhE9avNLCDIQqk6MqIu5XMBuLIheDUaQbWsr63t2reX0hqy0cZKlVS3QGOwLJo6fD74NbEkEpGE2oA3HOhz+eHLzK6s7d49yZBY3VjHwIMFLJIqWppak9PfM+BNzC7C4FFweSx0/qnUt9c3aTnHzCbKZLXYevoG07kiSKmugC9XjszPLRka1bWlpXgiTQEVrcpGRwb6e6j9vYaGg23k4YnRO48d+ZNP/cFP/MIn7jp0eG1tjgYbFUNr+sKFN954gw7Vmna1vzu0evmSt7srm0yAYEol41R65LIlLvHt73znC88+Rf9Bp9tbBgoqBOQWh9XZqq8SzMLMQy2B0qc3MuYILYYx2oolKjKJycOHJa1cYNFCLpcozyV6rEThkQFILMScEF01SXRV8e66h5ygee57yx0vP3eip8cPOIssPrMPEF2uXFRX1JNTO8EfJNIFn0u6SiPppdcm0FVmnaSGgCWacC7LFSn+wx4i2E6fXJD6tMlGknYUJ7MVF1JRdRLg5Y0Id4F4yXJDtynaXUSDouQ6m/EevaiIWuVFMSPYRpE3b9J+b5I/bHfrEz/nPdvf2kNnTWcD3vMVS0fxc7ZiUUgKGYlOFgkrXrQCJyiUnkga0CcNiPgl4yZesBIJYFcUN3BvDx85MrZzgswlvi8n/6XPfR5mvRp+p1HDHaYMw+bzkOsluAWmT8Ks0I8Jhl/6NRIhk0R5vYGRDZnZVaOOmlp06uz1GfqRpLNZ0lNWs8niBcpPRbs+GstT9QJSkSgDkEnMMQJFnK5UI4DthebdbHnb2+7pHQuNjfc/9+LTW1vh9c3E9NWE0alaWLn+LsvdB46Mnz/ft35uA6g2CH72BI6BtjKgWPDl6FtjM1shVZF7CRt5NCtS19RYXVn5zJ/+Gc0kElE4NYsurzufyXvdvsMHDn72Tz9DxX4lX+C+SfsPke0q6p3tZtfuPUfe9vAj8yvrOBKWhx755uc/t2fYsBWp3TGlIvZTqJRpzA5Ps4h8LSW9pGfJVkNdoqLcyNtld3gsaqumQWGwGtAjFYrMc24fPjO3XeF5xrGCqIZoiop0CLFoW7uqgUePyN3ZMydJlCVToj4cHrhwpBIUqwf+I251MZ2F0g1ylJ//pV//Nz/1c0+8cMrsCr72vRdyFW2+AikQUEcwkSa70+t0+DJEGbBX09FydlvVyql0JCYoby2Q8mJoKGMC24P0M+qKBBAnI9Fw0Xs3dDC5VXLaMnYUdXxrkP6/vekMaIahDLU36WDZz//b0vmV8lNFg8qAlzeiNZm7nI6yRlHMcqYdbSo1wdD6yGZiDsvPldA0jjITgFyvshmvMoXZEg9YboKoYXbOJWslQ4yeIqgBzpl4qHgASn6AABoIQhVeYN1pNx7cvxsK6Hgy7fe555ZXr165qFe3AU85rLDWkbRseB0mOv7QBa/H51xZnsWxE6+XqiUlgg05KjSAlMaQ8eI8MakALpHdoWSWK+NMM3i4JbByct8rDWndWqMlYlNTKmVgE8Wx3wxvAyLwguMyGfL5NlRoJWiZhLxX6j2gcIMbDXuTDvLx2Pba6iKPtVmvEBdrt+C4BqtUQi3xFUwgULb6g36720eJLX4HypKEGkZ3IpHiRjC2OXPyRnixuCkST5MJIrXJnAYxgwI/q9QAQ4a6eglAISagGOS+k1uifBpli8HhcDkazazSiBfBWgCahF+NGaIB6FypoKsAoNhtrnyxbrB40qub7rEpk9WT2Yi18pjwNsptORNUKKN4e2Pery0lNiCRdlmsLuBwpWpdV2v7fd1wnReLMQwD6lkoR7Y5XYlwvE6dFq3fAF/VKqSfIZPWbYNH0kJokOImVypBKojopQhDUDRqbjRtej1dEAaGxtEi+YbK7XR7erS1Um5lM2K3QWIp/Sqc7qAlOBToH7N4u6/NLff19KKxiGhfu3yFtp8Y4sR19+/fb7OYrs/OPnz3A317Dz311a/sPXB0cCx34Y1T7ZxqfWObXFJDbbTYXF6fKVvK9Q32EDmg7unMqdeO3nn/jz727m9++YsPv+M9O0fHVmfPjNx9b9DhfPX5J2nMOHv1QtC3c3Dn5Ooc1PAr9z3wCATyM+2leiX82quvHNh/EJrwC5en9x06BHERjGZk968trpCCSpHez5RtXsY+Kq05PzvT0z9AtAPH3W0H+annKvAgCcDg6dIPWklhigYSbh5RwBKO5ofZVLN7xEd8dd/+/ZQse7scUJryE567Q0uqu0zUmNhtT28/+LuTp94gziyDWjQpM0DIGpXZj2Kig2eBXAayGgwgWRUCch6HrZABKiZH5JVpLvQciiLkVRxc0kXKDoh7o/lYOM+bQkZOtiNaWMPRmLwdHcm++I5F+Zbx+wOWzrf8kKXzNW+UH3V+pYx7RZTxLV/JzVBOrHNFnfujmAuiuREyGAaYp5yhfJJgHRJGbA/5pNgKfEWcjdpukANibso+IdfRivYl7OxyIRmYfbCq7d69m5nFo6EImH4VxKKoAET9wkoPXIufYJJjobANJv7oxDhblgqFVD2JAIE0ytPjI8GGxqLO2CQ8M22TRb0ZyTkcGGg6LCpqG4ja8ySIgnzoox+8MHPS6dX/9M99lIan12auL63OrW3M+XuNlXYWQTc40u0Y22iWbflimohgKp3npySSqQ2HnYDcDla7oA2o66A4XcjZoOnSra8u23XmudlZJAgKqZnJf/mLX/qR939wfGiCigzgUlwyXpLNqCEQK03sjRoXxZPtViEbtwWCNDvB8qe2YHBM63CSWtGXqymMCT0UJapmtqoq8tAsKr1VFewm2aW3Oi1qM0MECx3KXrSvEkERbBACDL8F1BN1X2Zp266m65ZDpXGrGoZUrKxpOdLJ0uzM0uJC2W5TUYaNoVUqN/weJ+1mVsNRWJBCIyMf/fhPfeYv/vrOh9994vxCMqd58hvfMNqC0aSgO2xOb09vD/zqiWR6YWa2lMZhgL+VHD/oOTXMqVLpio1CU0fFLQS3x8DgD9OSAY8CFrtMMdM6U0V5L5paNpKv/jULI1hGnjJwbw7sf9HvZbgq6lMZr9yF7/8pg5ndojsZs2JhKrpZFDOmJyQCN36CWyMXJiqWjTl5Caox48Xg4z2ZKxQwyhZCPsW0JrRab1VLuMOEtKQq0mSQdDLc2vLc1PTS0ru8Npp3RtNpCBB8Xk+GvjRmQ1+PPwuaKB63GqSDAjBhPaq6WcVUrZWSlSrJxTaFHPiesBFgfoqmk+4ZmFS4uw2Px+i2OM2OmraIUEbEV3A+sI7xNkDuEOwFWESB0+TUvjLlKS0aSRYoa9EatH293ZPjY9cvXYHKSgK/KMtc0WoxwXhlNRkTcTpccmEteuoh5htUJQijb43GY1arhXA0BIAun5+O38RfIXYIh2fAd9NjgGzixvoWN4Cw6mJ0CZEq4SnOmOI/5jkiArkhbaUteMnNhgW613Q6/9wzzyOUofUxOXTSTNBsCIZ6XF6X8Pu0VA6HzWF2cnrslnlazpe1UH02yvRoc3qs0VhYAzjX7gNwvrS6GWhbuocnegZGzDY7zpY/5KbNVH9XV51gfWpsc/YMmd1621BRWzBKKIYnUF5TFzAtyWdZqHpRqbwOO2HhWDQtpNFCGsFrmdJmToZsNLgzLgd6DYgnevspX+rC0jp16gSiPFesWALdbTsltXQ979uORtLhRnC4e2VxGr64YDBAI1DaKcF3na+33a5gS7eN0ITiYPzAfkOrfeaNk7undtH2jVCE29NltUQ2rsz27Ziw2L2w+vT09+85qKonaUiT2IpGtRbr7fc+uLS68uzLL7/nsXe6CrmtzU0G27e/9rfv+bGfe+yh+77413/17379Pw2Ojf7DH3zy2P49JGhrmVQXWep4Ijo3T+D1obe+dZpaoExh7/4jOALbm2E03ND4eHVpgayD0WbDVKddOnBZbgHI40DA4fQFSyBD1RpCzfv37cGXWpyZwwfCoOS+eTw+tEIjIdFOpq0yLbBMO8JRdCdf7L9tYnl9g0QkdeELi4vkgKd2TZKGzOaL5Llza9vd/b3Eny9evULXOZcXXLYd0KKoSWImKrpfy87RlIR9YOxzuC00l5JKZThyTTAnS3UKWpYJyVadc+AV1YkJxQ878ocRxZxSRIsw4N84Tzlh0Yoyr2WDG5eg/EhmQmdvSoRP9vQ/X9i4swE2Y+e3fMSavKWSkRHKsW5uJpapwAU4L/EQlB8TbJN4miI4Rbkq0ohXzo1dYXrij5Gk0FKl5oKrOUXwhr1wHyjkp0X02MQEmea1NaERZepltuPst6quiIBGzOFHkMLUUWEuZKn4dSSMwGrZHVZOg5AIkVK9wdh/aAyylNS1hMqn7poIgIhOLuYt3Sp3gEgZpQCGmiqNpAQeAc2k2Vf6qy/+mdnZ/vYTf9s3ENq9+xgQEIPFPrpjVKWPxjOR/h7b29/zMOP9uSdPXj255Om10JoE0DdVBoTJkaeQq1EMiAaVHKdWQ/a+FsvaPF34A26Xiy8V8vm8vSvQ19c/NblnamLP6gLhGQK72PUAX3GBGvRHSKfjp1snwlvbuw/tBriaSye6A8E+p2bHRGDnJBcWsxmpWswVqQUFh21QwWpj6zHbvFARmoW1DQQWfajRvlqpZcCpYMgRcYTfHugqUGcEDo1LAPvjBFOURNuheoGkdRO5ce0K8coEVD0Op5rmqzD+lhqVtc2IBv1pdnz4Z//tW97+XiiADt/3rk/8xidHxvduhME6eK5fWh6d2LX/4B6kyuLi8tzCVZ4CKEhVNUfSuC3M/CXwleC2iJNroaWR/LXkESQtRGSDoauByE/6oqCDWc+wEQSwRKsFdCijRxa+uaWDedMZaJ2v/smrMgNY92Yd/E82efPHW5tx8Ft/bMBc4mNHxfJetKmsYbbxP/OKWccrf3zFe5m7srIzCWVXnQ3ECpfNJKB1Y28C6BCWK7BHHIaGIlK+h3qRfKDRhhMKkVNJeju0wW8aLSafz+UJ+qiYWV6ao46T9OT62jKoO868mEtnknEHDWUT4UI2adbyE5VZbYluxykXh10Sp5xsUbMNn1S7UKYvG/6xkdgyzrnKUDOmC7SrKzZglJAu1vTzEUIVISZgYEpBIbYtU4vqEQJFdqcjGPCROE5EI4vqFnCnDDOMOLBeU8znmlUzF0/Ak1BJISe9CqmoBO7EfKDWhruJPgURIIHlUtHRDhRRStVUU71KhVJqHQUgxKboK8YrCetSsczVIXpxXihDIfpNrJ7Gu6S64BCA94Pp0qwaKC4s0omPMiSNhnobCmsoRe0f6QdxllvNUvGCf4xzjhRGvqO5yQHDJcZlws8FHBEANbrVaOdRNQ/ddiySQuW3u7uHDHYH2hPma1jAMB+oIHT2jxYzsQCtAIqAS9Q2g2N4dHRxaSEW38yl0jaHwUInbVL37DqfN4DQoR0yFaZaFUVZHo+fIm9C4oTDibjiYZDBc3o9vq4uVX8/6VB8kTJtHG3+psmhh7Rz1wHvaHnz8W9FC5XDdz+wsTwLOX6+XDG5fPC5RyJJ+0Bjcu+BAq5lLFpa3xy86+7N1RWitv29vSdePzV07K59av212bm+3XuD3f1La6u773/YZLQ2Ar0A0AZ2HoomE96+IZPHCxM86SDYofxBz+bKRm/vwBNf/NzbH/uRf/eLP//EF//G2+1OJtN9fQMf+uCPPP61v8OTpExs+vLlnsHeleeXaeEyMbkLe7BWKe4YH71w5fLg6DjFaidPndrVP4B4Wl5dIwqdysPh3W5X6oxb4vXg6cjtT1+9xsjBYqSwnNgpj4ZAAYIewcEkQciJQBWTS9HHTDsgESrV1em5rr4QITZS2h//2MdPvHaCwkfggeUMTmz5vrfdPzs/n4mnu3p7rk5PM5OweJmFkhdSQldiDIs+BSps6hscosGX01OEpdjcbNt1huRW2GoRTBdiB+eV6c/WTH9+wNE7eovXziKWgayXz5y8siWjlUksUknZXr5ivZQ3yyK7EuP6hyzKHn6oRFOOznEQ5WIZEC7m5rCwT9axXg5KBA4FLCLyxqEx9OXi+Ur+Uf6TT52F+c2eNOQVufNM2FgkqqIqyW4uhhMDUxPsHA+Y+8DEyWYy/MYV9GS2UwxwdAHldowcbm7nfB548EEab8/Pz9MBBTGPLwInBOOcdNp//m//rVLI/9XnPrtybk7tUhl7BPOMlU9WCw+Qe+31B0nHJKsJWNzHdgw1tOls2W6y6p9/6UWr1UtjaQoRQz3m2YXrq6vrw317H3nkLZQ8ffrTnz754iJeNUXEuWSe8YJKURWJ3eGNQ5tPAl5h7bUaC3F811Z0bZt6coJvRiu8NbbeUC+qOp3Mapt6ng8FEPgdBJO5lXRIQfbmkunlamV9c4avPOYAHzXOwPz8whF47zWUkNdQ89wK3FdPyGrptqs9MEOCfyY+pzxJLdr1Bq8qQ478tAT90b4am1rtkl6/SGgAZkZLkxhUvAhsnIal+WztjVMXlpeSsE/PLRF9wWXTJlN5CAQ//G9+vKG3PPjYB+w+X7KxaA1u6xrp6aUY7cgjETBrBymBXF1c59iVYt5CzrIKTW2SNBnoRcizWqTPeJBSZo9ob8PcxNhkIjCaJDirhr1VRg3aF9NKpowSi2bGvUn7dobNP9HBnZU/8JUxy/B8kw7+gVv9sJX89NYfN4K/73/kXCWewB/rud+iieVV7HXRvoxMtpfZxw+F85mV4s6yHndO2RtBKgJVhEJBWsllozaARKnhoAJdZLEXskXlMULCp8U+NVlNNhsmo5FnumvnpDcQou07uYVYMjM/ex1by2IiU0y75TI9djSqGuMaPZXDjm2osnQKEH4y6pdgl9TiyNIA0mKWOC65mUaWus0NaS8D2YxWRXMYpg2RaWHaJbykMyBmmIqEldwuEKfqgYGBkZFBJht0GqCfKNrnK8Ql8ooeDzVA7bVyLFI/eGg/p836TlbMaoUGGd1TgvPC5nCQr4P1gNIEcm9d3f2kbciLaLSGXL6sj4PVktEKxCkU6oF/EghVuViWcmyukKwr32mpWGukrel6RVc0U8FpRMEBcUpnU8VGxWayQbzBzcIqQXxg5PDKfERnY/QQmRcgA/dKLTc2Hk8DknW5bEhJ2surpqbUpy6Ho8lyM+OD1wJe6wbo6BazghC0nSywppjKNnQaK7AfvdnjDY3EsiVodrYj6/TtQZDR8bCQSRdSUKDYivWqb7CXRBfgFEIddFwtVeq4kjxf0NRYJ7iIxbnZSbOZQkzGhUZvimXLowcnXF7/2vLWwPDArsN3vnH8+bbBCr1uplLt7R92dA/02gJZEp0ak3dg6N57q9/9ypefeeLxd6PAKlW4eZerjUAg0OCENrfIKhSRc9mi9IKtt3JcVQLaLdO+I/e4I5trsbTBqL77Ix9dO/HK2M4db7z8kotAVy5rNdpfe/rxu9/22NRQf9Nu+IVf/dW5119LbK0G/YF0bBNM/MBgLyH9tc21rv7BWCRcKBKk0YJtP3jw4LnZeVyP7VicLg3YS9SWCLMZYGKTKpmVtpQ2hx1aShhMLl+B2MsCoSSxYqfdhZ0HRwPBZEpyAfyTw2Zmid+Gs0XgQofhJThNio8jqVR/7wD9Pz7525+a2LdjeGh0Kxz2+nxESvbsOzCzMD+5d+pDH/7wF7/4NxmwNFLqLtpPskBSlCDZLgQ1QQg8JCiDjhy+/eLFK0T2hBqlHWaQMTHxYBSxIDhkmcWiOCWeLDwFCCyZ2yK52IYzZHVHAUsATDwdWaiSkZ91foxbLetF9SnrlH3/4xdln7KqswGHVQ5945X1fOQo7Jk37LnzhldOoLPIBkJ7yE87kXARUOxPfsulisvOFopQUeSWQn6NHjQy9ui1t761Gd/aJvXLmM9qGdMldsvEZwrzpiPOmOketyObzDVoRa1X7iNNu0HMgffs7x8cGZydn7l8+SK/ASWOHQIxLebB62+cuu+uO/+v3/3deCx86sSr586cXl5Y6+r1UEquo8zIZHz07e/YuXvX3Pyi2Ul+QaUx5dSGUipdPnBwLBIpUM6rNRbS+VIgpAoFGo8/9a2L55duP3zvI48edDn1V87H6c2V2kxSJcJl5VJVu9dI6bfNYhmfmKK6kuzzhZPnPVb33CtXjF1m5j2XFZ5f/tza55/89rMRPEuJxBL1oPkgzTkk4yD3Fv+Ppiq6VjZZGxoPRNfCI/29W+vLnmHTpUtX7j/sJe3hMgkjkCvo0vptMJMXdPlyq0y8Qh6UDu0rxSxyp0n88lyaYC2hQLVq1Q61xqFSO2i6I9/QR7tmySajQEo525eff/34a2dogkduixHqcluXVyM7du3+2I//3O133dcyO87Nr11/7vT6WnR9I7m2kjAa3f29w3ftuGNlca0E4jGbLEFiXMqpGkX4ASl9L5VTYPS5IugkcMMIhspIQwtjKYgFibGJ04NmImEnegsPmJB0J/hMuhj7DF3X+ciF3FhEQUsURPko/8jI/sELo/EfD3k8KbZk7Cg7YUeiSTv/8V4xFrlpyq47L2xKpFGZdHJenLPyJ4NbQG38nkHNpOSs+f0N71Z0MN+Izhb0GzyDik+MBywjWZSxjG5sYynOA/rMZtwXBQKM94RJByKhjLfBQLBZ9fi+3EEY46nqhSwrW6n0DPRC8twd6u/uCxKSJINI7V0NWGohA0gJFC7BIYoGYgl5ysgSzB1uJVh2gC6UQeJXwyQpFQnS3w2G9DwXbqOQ2KDLlopD/f3EPSrVghI8rFAkA3gRF5DyJME4UzpKSxR63dDdMxCoF+hiSQvbaiSR8rvN5IN9HjudTLLxBMIAYDPP3GKDohnGDj/IZLvL7vUHqGwlup3J56huo3tgen1ra3sLLp9Oms3tcmKsILJwZEmGwtCUy1AlAusf9x4PDKmnom9QLFGsWvXukT4aIUH8JYwLgjSRuAJzIB6hEQBUtDiikDNWAN0ADAMWzcyoQ0bfhh27CvpnoLt/bmE+1D+h0ZhDA0OqpZVLly/b3MFMoZpNJSs9PdIRG2ocnXF4fLKvJ3ji5AsFWjYRZ1JrNjbWSM5fvny2UcvX6/R40GNTdwW9mFMkxSFahUYyG94u5bJYEqlcMQPoFx1LFqe3L371mt/tgRRvYXUdUjj4B9o6E+1+ux2+SCwa7O6Gm57aRODoEMstr2509Q5gu9ANOrmy2jtqd7qC185ebZTzPSFXz/Aw1RlXZ6YZQQTbTxw/+dhjj81OX9vYivaNjM7PLjEOR0Yny7ly19BEdCO5sR21XZ0O9YQGR0PXrp7vrquXNuK7x8f6xvefevVVOPG6fQaICl/8+pcf+ImfUJWL61evTExOTkwMbV+/vL2xZHKZt6KRSd8ORAYqQSkHMg4Mjuj022cvXvW5PRAsg2vLpBKIV1yoYiJNWS5RHvQo+jWWygZ7uEnBfK7E9AdYDzgL8CbCiuePbcRQZzOZpDK7wDdKCQBemvBL6A20ULzr3vve8dZ3XL8+S3fLaDh27vyZdDTh9PvQ7t/41jcpB3/vB94PNvC224999WtfwTaTjIukVQTJIiVJTDY1h6ivc4M2Yj/yox+mPANAVgHqJjMFd7gHHbUFNEVivkxi5IvwvVKJL1hRMQtkkClihKQjJ8x7js6C8hYFSEEHCl+0pvyhfvkh94phiWJm/Q9cOlLnzV9xT+SjIrIUQ4TzR9hxSyTQhjVAYyPe081ZRCAAQ+xu3nHBcvf4pcCyeRXNy7XgHaOj5bZKcEtEGg5AsUghzO133g3UIz63gvRNrW6prUbUz/ve9z54GZ988smLb5wxUcHT7cqn05g+i7V5AIZKDZIcjAIyUqZ/87nPf+RjH7n/3gfwsubnZyWpnKl4g0GODoSeZ3Tx/HkST8eOHP6pn/nZ48ePwwINT22OpqRr0cK9peGhcfIt1+dOTy+sB3tM2PjJ2CxSBvgwGdxyvh7eUiUiqt4H7e1G8fSJy9HNzI6JYeJJ9z60J7adv3jmgtvmAg6Zi622aBaH22g0Hjt4B+ADIBHZoex73/nYH8Y/GV1fU9ugXFaZu7zl7Ux4fdsf7M2SyYKUvCX8vgRfhDrSLAh8InZkrnbtDsIvVEpc2Fxb7vGo1jYqzbyqVY7+m/eYfQGjM+RXWaGwRenlWsa6yQbGBRAf4RxkgA5hq4w8gpsMPeGDwdyh6EhFLzTwpBykpYHQzmayA342mBGz3ldeOrOy0taB92hTMe9YjeQdXT1/9jdfffXEpXse/ODv/NnntkqN7z5xyuHyl0taOnfTc8Rh94aCoUtnLxVy+FJ5dbNCgJACBwqeKsWS1WnkeRPJo6AGHxcrA7OHCAahwxsjTQY4I4RRzxr6MbUYUrQoYUgBW6RGRaw3KbaToYw+ksYAWCcSUvq+bpWx9sOXf/QtR5MtUZ0MZW4zqkjZE/8yasEn80lQU9h8KFH8LeU2qbHxsBLAW6BBCY3L8FUSwHLOyk7kKmRqMzFRsULmI1aG6GZQ/QwIbEZeqeNCyrATsSnlKZOUpQqCqYGZT2dAaCMhe6qkUjEgCwxsrxcGJHujgfOGfa6meYWvN8TDXVieCYW6oTsUro5KbHP1GnyMsNvbrSawSEwO6oPL8brGQHJFcFWcI6cGzASMHNdudTElRWoUyjngiB6qBeAsBausBozaDMdi3oAf2l7+OMPBvn5ck5GBLux7rINweCOWiBJ6BbpSLjXpLueyOreISqEbS2XMKtqg+wb6CLWszM0NjIzSqMBud6WIchcbg8Njc7NXwUBZbFYcf6+rGwGRLqWjqTBs+8gnu9XgsVni4TBxSJfHG4mGiYbLjZJDo3rxYyUmgvNerNFoz9TWWuLJkt1C60Vng8hLPp8rV4fGxuJ0g2/Ve7qHU5mMRl31B0PXLl2k3zIQtlKaIuVc0O2EaySxGUVUmtUOavPyzcb04rrNI3BZR7OqLcRHR0O1xPp2PAH9DCwPU4cPkVy64+6HvvYPX8rG15hktUJuo7ilb6frtTzF1Ll0y2I1baxHiWVQnUQq2qzTW8gg6OqlzW2rlWoCvcMbGBzfuby5zUPhvcPhvHB9ztM9ECjWrl69+ra774WFgFj0a8+tHzt67OwLz97/6Nv8FHBwbxkkJjPdT6Cx3FidoxGRvZ7t6Q9cO3+mZbWN7N0Nt9/hY4d5BBaTdnnhutcTGL/r9ldPnXX6sCtGYRnbiiT6TW4g4vNzs/Hw1ujU+My5C709/YsX5vcefPjl146b9Nb3/uR/PPPai9GVuZAdOq/6xe98cf87Hjt76oRq3+6LZ08N9XYNTIwl4xEgdevhaN/QyOz8AgVNVrvu6vQMA5kITSGeBAv2iZ/76RdeeXVmacmgU4NPEWwlRUBaiYHBx5BKgHBP440KA3OrDqc8c4dEucxBiZiKupLeCCQWocdlABmZ/fCki0ywGA3n3jjZ5fV2d/f+2Mc+8u//3a/AuIIoKELkq9djh+G0/f7v/z5NbdHlfl8ANM329jrq0Otxon5QDOhRHO4qabFcxu40/95//53xkVEAhqRQgz3+VCpuc9lGh4ZXl9dAIVC4TuswyAsVbxx3TlxPLFfODVeKilscrFZdipWZUExfjo5yRrWJgc+kU8QN8oCwNlvIGhEd4heJZry1yHeMbOUnN76QbTgc4HC+Q++ygDuWYSALP9CAplH205KWwfJbeUU8MpU5BxSuxJ+U9sOcFel5TkYWxSbAPWIvXAogN6fZOn3uIt4tsQe0OMQ4iMFSJj9z+Vp5sLC+tEq5n98fYAAjI2wBt6PgLWxuE+0Dw8seKDSyUtRoML/ywiukQPHC6alE97M733bn5UsXatXUej79Ux/9kNdu+9Tv/QEqAWzzrr17+gaW6XmEYjA53dNX5s8PXzy095DHZbky//rVc2dLaYNZG7h+bYbzBfSlbwGsq6+eUz2dXL3t6K5wdXV+Ot7tnyoU1RZb6yf+7U893f/cF/7oL267/aHtzVxxLa1ze1K5ajFa/cDHf+Qv//Iv3/juy5omBD6eaGSbmmi7zXHvnY8sz29ffeY1GtQPjU9srC3WiM60ysIYI7lg4oUqBiqBaIulC5Dm8FDvxuIMAYLxAdX9b1W95f6+oQFqOEstXRkTqGFoMoZremjFSTkJhQEcG8x7fdsGapzhj2MBipTbD8RWLxqmRlQOkmoIGKyaZnpz3W0MNgqtc6cuJqJ1ZvhaWBXJqIzeQu/OA297348XTCGtv73rwZ/+479+zeUP2Sxj1Ry97DDp2Glj+vKFN46/InFm/DwKkgUHqlCaKyMCmUh8FcwEHeQkEIsQlS7vaANF2+EdEmHig4wnZmELqiD0omgmFnE4sSgFlaQMHQlHS9RaGWnKSOqs/he/iinLPpVX9DjnwBrMRQkpKYucmCxyFKaRfOQMxVrBRRXhoDiwyrxSUM1oNsWjlSmn6FT5Ci0rK9knBhAKWLxeYfAQBSwhaCEF4F4p8xARpMwdJKwgOBpiJGOqggogMY+1ze+adPchko9PS3ZCv7gyb3XSGdcRi2+l0lHmLVJgYrSfEhSUDw1t2BA3ixOXvFON+ArHkokq4g4BgtmhzGeceo6E/00TPxZuNE+BbXXgtWBmxz4B4Qc/osnKnMQguXLlEvcAmzTY0wOIHsmbyeW9VLj6glKsLOUJagspHw6JNQXPX7ZGxLJWroK7oeAyksgAeAHbjEiNbW3ZfU56+Dgd5hIlTeUMTeoCXb0L1xdXS1XC35UiVmkGJxtWWNonsDclws+tFfyaCCsJpYl8Jm5EAIUqKcDXIndadSp2gGVx231ePyklClpoG0CFDyTYVEG1yhW8Xx4PXheGXiUP1JDQphGMDs2QCQjQbwCUUMDp9I7hWOtVAWcFXx7pXip86zvfxAtbnL6Wi0UVJ5+HX0/FE+VKgduFy04Mk6YRDNcGIUlVWcdTNmjh1aZTFbF8DfiHSgU1iUMQj4QnRsekbjvQNbV33+unz/IY9hw8ivVADXQ4shX0dC3OznIayaVl8gdXLl0e3z1x3wffy1yKLy5X0sWFmelSvnbu6SftoZ6Av6tFSS51Qi+92OvzwoCJGbQwfa1vz34Iq0FNw5YV6Bvp33fwwgvPh1zW7oA3srV25oXnj9x9L/jy1Y0wHdt27TpEnyjoc4fGdrz9obv/4S/+mDIRWL2p7YBh//jxV3/0/e+lOmJgaIgeCNvb6S69+cmnnvvwh/8NvRHpLjw8Or4Vjh0eHFq6NheLxeanp+PhbemG2lY5nWbS6kxhAmCihWS+M9R4w4MUy0LGHcFhWUCtQM2rZGkQfhQEk92UWSrTEgnDg6d37GYscvzVV7BdKLN2u6yhnt4tbTgbz+hdeiiKpg7t5SA//bM/c+Hc+a987R8mdk3cec+d3/r6N8qVWmm7bPBqK2laaCfd1HTTnhJ0XKmyHQ3zBAVRX672jw+uL6+O7dqJ+b0wu9Ad6IL2TfrYS0QZCAdkowx2iWAz5lC6mBScFzNIroyTvyFG0H/KRy5RFnnlWxm6/LKzTlkj3ymfO6+yrbLcXKnIO0VAdfatCAeJcosTzgnJxLj1E2X/HJZ1kujDk+DchPNQZNHNgyp7lGMjb5GwzWK5aiqJ51xrqOWWA16SPZOdvXD67PkzZ4lwQCyTy0Dzkx8aGjh78QKEzhwS7asoezBQBo/DMzHhx7b7+y/9LY0WpianSAesrKz951/79f/rv/yncCL2xPceJ4jkCwUp1g9H47kTb5BGwqw3A7AAQRpPP/G9J9//rvfBjkdHtJnpufUF4H5eQEeQASCYSNkajIYcNTgrquv6DZXOEt2MXTi7uP/goUtXr6jb3/I6u3rHJs+dvdykLbqvF2MO0qjPf+Zzvd7Qvsk9//cff/IvPvdX6xdm1SEXrKt33fvARz7ysae++wxgiOjKetrcdLrM0VKD9JIB91jMRPw8lctm8QX8GoobColUYtFuVT1wu+o9j44d3uu1WfJtIlQ62AloA4LWghYVF40HSezDAOWdFBoRe1QbYBrjXnK7hMdQAg7A5kUZCF1+kYYSkBmlIBmMJ1BL+u888friWqXUMu09vP+xPYfueuSdNZ3jhdcv/+6n/yZfsRYatIvi/niMTXqgwsZLiLwCFh3wJwKXwjCB+0r6BPeOx8kfC+EjvBegabiSjFyJElL9JHqUj4ReicMCc2mL7+f2+4mfwb0HSoEBgN7DuhK+YnZCgl3shzcvt8bTm1f+i9/LrL65MIJlEMs/cmDlI3pXGbUi5lGVTBv5il8o34pyZeyjWVGiYvAqGysKmB9giIhykBOWL7jb/MnG3B9eOx6wEpFmoIsYYhKxZ2Q0twcLjBipHh5g1iiamNQsBatkxDBadZKvaEqoocTYNUvphcNFehXqY7o1C8KtUCX2wC2mQSH6n5lo0BtQkCwcQvoXkolA7ypFjdBOsQknJaaTqk3hCoys2H12h0XUnoIdZQ3xQEqE4eBNZ/PTM9K23eH0UtYHHwINuoHXUAfCU0Jxa/F38LXVLdQe8RRKEwvAW1uJtiYDrqkcjxNF3LVj7Py5U7R8UpU1taxCm9equU1aLMGQ246pkU5EYenyelz0C6+UFokbA83lviDRNFyEvOFeSakAiTokNiMOzBS2Bg4u5lC+kBm0D3G7AfJzJyendgC4feONk5gRSApwnogNVBQuAVAsCkEhqdWazY1GxWJ3Uurj6vb1Nyo0nYVF/olvfXNinD4FE5poYntuGQCzzWGBb4uWwInoVrSSsxo0dHSnKwSTCgVDqhgrDMgZQhqriPojs17ToAVDnc7CNEHXUC9NaDefScJty0UzxkJ+X2jXnhe+8W0S3jun9vKIGzn1odvvzaUKwa5QMpIw2Jw0kjp2x52lJj06ME51/u5+/6B1a3E9lyoRW+jp6gVrhpV9bGofWACg2iM0YBqbSMWSqnQGNiJ8vp6hkW9+6+v3PvS2A0cPz5x8Mba19tiHfvTVF17MxmLOUO/a8gJX3uX1WMf7qqmtfr9D5Sex1FVMR/VW6+byKpM7EOgCaIfQYBbc/cADF06fYBjs3bsXH4vIBAOKQYXHCcc9Q4U4BF3nEqkUMRxmAvBvSg+RTJh/dBLpzCSx9hSdIPMLeSD6S1SHUMLhUSor2acCIZC8q9hcHIQ2w1WpXIL5YXN9MxZN8Bz7+nqJlJ45e74YA0erWlhaJAr94ssv9Hb3TE5N0t7u2LFjVy9dJq4AnzBf2YTcqkJ5tNttB5AHszF9Ahi9wBQB5a3Pr2osupdeegXzK9TVDQCYXpaUs4H+RSKIJ4lmY4wLjYKkfpmaJCPELVYugxmtKEDBEsqVYijKwi9k4Z1ymTc+y2V1ROHN184GyrYiIkVmInuUzW/thG3xXbntHe3bWc/BMd35gRIk5yeMQ/mhskN+cGMnykkoLyJO2YgZg+2NBBBaDwk8EOWT2cTKBhdF5B7xQN2AuP4Q2uTBOumh/uKx8q04DHod2AuX3RHw+LgNhUhCZTGsUeINRGB19fjLr8AVE4/FnnryGcCbie2oDrgvlO6U8ZQrXAVCjOZgYsRoNJ/5888cPHLAbHLqVGayR3TmzKbK6WSOcdGQQpB2d6grl4usXMuafECMtPGt9JJlZXJw6syrZybHd431DoWvb9MVqJgt8WzohktvxM995k/gJaWlhNtqDvstdaBkBgo00s+/8Axdi5rtqEqVja1kD925d6Br//rSxvZalBtiNAGNIjRWrRejele22gjfdsjxnnccPbo/aFBva5pbdirbK0VJZGFmyZPHtJRRShKg2XYRGcDPEa4rSbyQ4lPYHZoIAi2dToFHSzW6VKYDA6KUUpPBN7K6v/T3zy7RrMgUYt78uw//ytF7H7m2tH7x2mK2aOLBEJjEr4I6iSJOqcGu1sg5VstUllQEF8NzZHeoHLQvEp2PTC8WwsuMId7KQJOxx8Ko4BHgDKCbAcPQgAIAD8MM15g+b7hkFnHZxRRDHelw6lFmBGOVYc+oubEz2fv/9iIDVNSksiOGIh85c9QWREaKCysBZ9G1+Exy3nJpvPIVm3EqXDHr2UBRoqznVG/6xMoPRTczVlHACtWGhLZFYXd2q+hd2S0nwPxl9IPMJFYnTiuAeMECyHqOhn8q4opN1XSDsVBRRxiBYjPYKOmyR6t5qIjga4IbEu3E4OfcyG5CrsU9FyQVphF2rShg2IEEFcVjIIEEnS8TFQuXmWzSGr0+N9glekGwkr7ZuKbIUp/PI3HdQvHenbsYA4xZ7C50Uu/AAG3VAz7/6eMnEPFINJIWODOcLs+Scg5GTKpQDPb0U8tkd/nvPHJgbnF5z67dmmbOpG4StqvnkrVCwtflZ076IX+Pb5s1dEDSFKWmKmu213WVajgaRbuggBnLYtLIMEKaYOITEuamUFTM6SC/WyCDYCo20EyzXV9dXuAWgbQsU/xj0lOOvDg7QxgTq7ZSLEA1BWkG+Q3aN5QKNcqEgE0xzGi8RnVTK5fIwA6taWUikUfe8sDVy1fOnz/t6x8iWlTLxqPx2IMPP0KB4AtPP74ymzDpLM0Wxg5pdmhT1LgQjB0kIZEcsv1Q1NJ1Dwy20Vix15smnAKQZRamY8NnB4QZ0JqtW5sbgaYago6E0IWVAsND6e2YzurVVTVbsdT+Pfu21mnVmDRaDX0jPW+8/HL3QF/v4LDGaOkZmphZeFmXyK/NLKkoLcumPQ6702Be34qEN7bB4ZHzfvm5597yEz/zuf/+qd1Hj+08cPgP/+hP3/PORydvOzJ/+YwqGEBqfOu733nwobfCoPnyc0/ZzNqJkQEgnL17d6ksWjB0zzz1FCc7MztPBHloaAjA8sWLlzGYS9ks4V8CiQz4cDgK0Tepje6+wQvnIaT0p9JpdLDYNxaL1OhQC1er0a2qhhXbWZR4DP4ss00Eg0wipD/yHF9BVBzTipEuA5nhykPjqaBARMiJgikU8ja7TfRgsxkjQ0EYVtXEEC3G884+d75YgmSfETsyPgrg4Or1q8yRz3zmM5R+ghJ68GP3f+ELX0DBlLJVijXBHMB/Y2FvwOogUaixnnIyVai7e2t2PdhPhbqX0iwtNOaVGuaFJFRlvispKA4r6hN925Fucm3KmcvlcGekyEdZ5GJuvuH8edt55c2trzrvOx87r2wjb+S3XDlvZSedFexd7ADlKJ01ImOVg/CrmwpYfsIvua3KeVI1oOSeOyfVOaOO3AMzcnOH/FwWuc0y10Bj7dqz+8Qbp/CTSJCTCl1dXUF2IyXJ3POsyX9zqzOgMcOR5aUF8G4mm6WSL0UWVihU0FrMX//8F3tHBwlrIBjp841Pgcat5JAqPmRFeHoJxJfN49k+u4BqYdg8/cRTWrO6SHelkmojH81lkiTUoGOEuZjxYLK04KbK5eqV7abWZSTvevrVs6vdW7h3p9ZeT0fTqkjN2RekHqGdqhDOsDpNa3Pr3LfvfO3rlEWRxyOTjbx445WXm/XCIw/eAxNg2aUu1trnXr/8wD33ui2utgc5CVi1RPqWBtpqm2r/1MCOiQN7dvf1dBMS3Gw3Izo9KrlM52gePTaHopOwLklLYL4YmpQDa2l2yoPgXisBUkpAGroaZXNlXaWEBy5mOmk1HhD6uWm2R0rq5549+Z0Xp2Mpy7G73/9j97//4nzy6y/9Ff0XGzp9OJaLCdFOT3eoh5RfZHOLrmvlYhGyI2lvIiqHUyBXyEMVXSXvGQ0SLuIfBBEhQtEu8mT5JAcXj4tYKCxDZhvgCoAzwAjQHjIhMCvtaqFX40aRzSJ9TXM47iEWGAOJvfPHhYnMlQGJYSxf/q8sMmSVEc4ebryX82OOie7kDFHGHU0p+WlZydRS1CpH5ieKSpYJKCoZRSvXLpeP0atcJxfOHuRiJeCsmCnKbhUsNAeSg7JH6mqYP6S86VIpc+XG9YBPRvty4/CDsaqE3oNLhdgO5e+g6zT9L+wupBuaj4UjUIRBQg3HlIiyHA2eY7QhoA9CFBiBlJ7iJVH/xZzk5gkTEPa+BXUs0KFqiV0FPK7uLh9Ki4IEiFJRwAAKAEOyf+YGvEJ0Pujp6aO0o7unH7rmjc0IeV20L9y8hWIZEjhMBZ1G3D6KiPFMEZ24qHR0sNpotuQEiYBMLIXny9m0w+wBawcptLFmbJaAgIHKt+RrPH42tW1HU1AtwnQN11U6nuKasSh5MnLfxRiRB0aIHuuICCIjixU4BWTCjBgfxN3qdDlFLQNDEEwFCAtqtFT49cLmQcWLWCFiDhK9Bytu5PFy/+DqqpWKqZnp4vZWuFroRidHoc+CYISN4lGnwzQwObEVgay4b7B/CLAx6jyfoodTmolGjwBsDiIlYjNJGQrUS8ww4h+Iz1ZB6r1UBiqIiFSYzZyP02JxW43YlWcvXQfyPXb7faXTF67PzKdL9YkH33rhmWfOXZ7ZMzWFhm4BZKJKqlikBQI1Esxo9H01kTEOT/YNj7/x8muWavnB226ja9vspcuAhwNOceuknkRn8Pq76Py4YweNjJa7xiaGhgdOnTyxeOYlTqAwfeWuhx4c2Yqh4m575IHw2lzIbSlGl8w61cyJ7clDR7OUwUHv5fSM9o9Sdx3YOa6qFt+hbk5fu7i5WT985AAylAaFKGC6ihK1evXVV0kTcr2gBzDpDFYbzBq4xuRKtuIJxCsE4iIilBknI18Z/vIkRSIo/rDAezujX2JfvGessoHICwaAohVgSEfUYfgLmYwB2IqppSrHYhEUpM6pdboc2Wg61Yqb7DaSBt29oU/8yn+4eOECzwjizMcff/yBe+9797vfHerq+dSnPkWrDABitIujE6e7NwB1STQWn379AhiErYV1s8/Bozt/8RKjrVlPM2/EAGQSksVR0rtyrogfqRyUiB4GLWNQsSW4HsVSZ0rLOJWxKf8q73i98UFWKONZ+Y6dIRk7a+TeyL6UUc0wVa6cW9EBgnKHUb78dXZ+cx98urGwH97J1ooClt9xN0lXdhRwZ6vObe4cn94V8IbKPsWv4hr5kudA3SOpBFbecccdx19/XVVpUIvIVsSNSrl8LprNpbJWu5nZxMURLYitSpWwzknVIx3w3JnNNDXDVodrc2mVSiG3C2a0SqCnD2bE8uw8IdLf/u3f+c3f/I2Zixd/4Zd/2eV0/vbP/lpBmNOLsMcTBCkhUEpwBzUUzIOmUqKQoZ2K5eSUudM8lEiVICxVPNGldcOgoxbPwQIIYUE+GQPPC6+UlNZVaCSo6h/sgl+dMjhmDzHf3uHeRx59GzPlqcef2L/76DOLz9BKDIzk7IVFYJSqRo4BiSNjd6puO6K66+7AnccGQl0MZ3yMcKkat9uRnNypCmVYIu0R28x2SbgjoZCAxpbehBAWMwiZRcCnjK4Hj6OFfRk/XrxeEmaAnKTeCdVo0diC12eS17aqRx/5YO/gQ3fe89FvPXEuWlBFaTVXNfE8MgVDV2jiKG25i7Wnnniamw/TEdWc4tKJKGdAkjRQWimgrlCWvMpK8Q15I3Yrz196+iHmFEdOZ9Aa1U47CEq8QWFvpYaQShJgE8GgX5crtt0eWtty4iUuDERkE/jRD1s4RGcw/bAN/vF6sVlkkstzZOHeISxlECuaFRlPuhcDvaNQuSjWKzpVzAhFlSoKWJkbTCWlfoqNZb7xKhswfDuAZ0n3ooDxfYkzyP3hT9mt7EeCPLwqZ0FsmHOAp6CirSq94Trt1YjNykDDT8AgJbjL3ZQz10tjXtBPoPvAShDAJxBFza40GeFbCXyQhW8Xa1UIEFC9CBHFeBBhhkFEFQWGPMAMzoYwb7NhpUMbsDcOBSkVGqgmwUyJ4hZLOQwF8I1C4Fwkk1snjxiNJeEXBMxcb2zQRR7sKOqEtiJYl8xcbirmKmqYjiyEXdgfDSEWl5ZpXwiDe/H0aYPJ9tprrwUMVZcZNV2jRo9MrKaUk/wr+W6Tzu3wqo3uKmLf5W3pLb5QL2UQhGHrVgDbXCnJwSoWJxeDAFJGD3QiYpRgI9NKAgODe5TNZ30+uP0dhXLJabMOD/TAU4FhAQKNG0qpNMICYY6IogKZuUC2nFHMhCEcbSYglU0xe/BeCFxTwwVRLQb7jh3j+s3trbXl+x5+59JyaiscQUUYrTYC0UQFZAxidGF5EslQt2mbjINOUJrhhG4B+4Bca2lK6lKRXAFim7oqmi7MXLloc/t5mN2BgMpHw8OuBhdBG/lwamxid73cJKH+5LPPveMjHxzeN3nye1+/du3y0MjgysISAMhYquguqyZ27a3kSpnZ6+7xsaF89trFc8lcrul2SmwQ6IrXtmPn5CvPPOPt7tVYbK8+8+w9t9+eiYenTz4fCvko+XBmK56uXppHJV981gyLJVTPhga4bSjmq/FwcHzcNr0UIQW/tOq02SQNX8cHCgDiddstDAZyH/fd98C1q9TaGo7edufvffJ/fPDYPWS1gVfhPGG38WAwOCAXJOAGGI2oBUOUoY7O4vaTte8Yz4JNbEDDy2CXsBMyl4FE5BPRAJdUZ3IS5+OhyOzE8RUgF0XYPMAa/jEasVTIxVLpRqW5vrJm8JiZcBQ3/cV//7OevQMf+chHsCav0D9x3/47jt2+tQ0v2+riwjJEq9DlEvA3OMy4dH6P+/a778Itga0rtrapc9nKsVzZlENzCAyM/4lHKg8UoaHYCaLTxD5mujGMxThEDtxaRJPygY0VrXpD2ojEEXn9/UVZc+NFfiOC6cbGvOH+iB+qeMJoUEXj860IHBZOq/Mq91H5Yw8clLvTOQDffv9Iys7ZTETX9+WfHI+P3HPJmCBDFFeHNWzJETHiaQZ82x23l1ECTksNE8qoKaXzcmAdTDhGDgCrJD/n3FxUQDQgr63DzliDRYIAVbFOZ17xW4noGi14WOg5o9GMQX/qxKlHHnnrxz728f908fJXv/KVe+68yzPZDSUIOTVyqARB0tEMN4+cD2xliDc5oqqJnoaxspQqaq2WphrSxZaq26YOtGupEmd8/133kS6ZPTePqYQ573DoMxkBNK0vR4BhWHV6l8e9uhHLO1N0ZLn98B27Jg6uz661KnqPxxleo4lIVyqdQXmNjqnoDnPb7YMHD/ePjXqKWQChm2BN3S4KGLAq84g+0AMtChG5n2IUM6oRGNI+XdpeGGigaVLV8Q80jSKstchi5BwxVFSgjAriq1KbroiNitp0dTr59PHV7ZTlxx77SCzl+d0/+brLP3Xm2rLV6S3kqla6HwZ9UOE++d2naKRWzGaESoPKOmI3jDsZi6gZdA+tMpgsKBWmjxKG5dEyTjgzgxmznQUVxuMHZweokdIvVAeAHpwsakGDPi9xIzI7RUjaw2GCSB6dwV1vFCggEe5f7uiNhfHCXhGkDB3yMuzyX6V/b+wFKYBqZ/SIycpZyaiWy5F9i1pF3SrWbmeNMotEv6JEWS/bcFCcJ4a7TAbRrIqi7WhZkqBKzBkFLPYyryT+5CvljnW2l7ukHJe9Mdq5NbB9llQ10oQ0Cuc7NCmRASDHuKFSQEFkSbQGT1kL3YTgssVgJT6BGOMQwoBDJo3Otdk8MMMGFUlFmhYR4EDkYe8QyoegATAhdL104YE6Dnui2CiXcrVSHsWjh98bVYxoq5ZgIcal6/L7uD12u4NKJEKnzMZ8sVDd3qJOl4xvV6i3p28AT5J5C/mDVFZwZ/RCjcz1QP2Cy85F4AXmCmWdpaDSW0i9HJjctb48j3tqtToyyajLwWEh5igCiiI8nE6lHQE3SCgy2AB3vf1WN1WnyRTn0CS2S7K1BgZTQj4ykISehcyr2DFGIP2kV+GsluBlw+d20X8JlJq/ixCZKZ2K0SWQENL2xia1UkanEyHBEVEbFC1wZzDqrSaHxeMjRuCDCJO0ow8uEdXa2noyGnc7iGrWZmbmeoeGxyan1laWW23r2bOn8olIJpMgN4bXJfBxGIhImHG7iexLyReoCG4xlqNGC9yWEYCB0yJrDUyx2Uoqw0JrBK3Y0tnS8Xj59eMGqxcvzeoPrm5svHHmdG+oy+Hw3nffPS999e+PHdw91AfwjZGARUYTv/Jw/8CFmSV/aHjvffetACRbW4TdKtATFN+/XI5RnG02a6qV2cUlRsvs7OzR2++ZuueB3MamI9RlPXjo+vVrI3uH8g2tZ8euB3zB3//tX+93m9rNtNVlTOeQZRro3Y/1jtzxyKN420994xt9gSbtrdwOE30kaTRZzEKjmSU+ye09dOjQMy+8TInQ3Xffffbs2XvuuWd5cR4jDwemf3CQPqXr21FGMlFoJp34s+Itig0l0AeZfxiFNypZmVeM5I5pRZ6ecJgEcjBqhHQCC0kI85AiAO2gaAAWTeSA2cFGWIEQEWLdu9zuzXCE+ZrZTJj6rVura7/3e79rgZVYp19bWSUT/NJLL7FH+LZCAyEiN4C9IGJLJRPbkcj84kIgCN6qNgmd5r6D3/z6N1AtDp8N0mDMZQ4DaROPrpMuFY9HLED+mIk8FM5RBl/Hqla01/dVL+u5bC5d5i8X+ablhjxSVsotEf9YCQbINJKNlZVIGYGjsiCaWIvbw3okKh9401mUbxVJJvL3xrFE2rNPifDLCcgiP5FX2bTzEXGqyFd0LqKIdeLNaQUkYbaZiABdvXqF7W02a6pMzLNl8ViZitTLVnI3YtoEOxx4iLm0YJfg/6H+Pl52+EyQn8dTWaPXHuwKmsUSswwODNEeFFBkZiv87NPPUdr2jve+b2xk5K8/91dwSRJVssGRgSUOGzGUZVwtVWFlOVE9kRlOrKUb6ulPmjJka9Q2S6lOp9VmO1/GGjDarZhWhHb6x+lulMxF8ebrNB1wBGzkgHQtTSZMaC8FW282lrt27tq1k0sTw1Oz5+daRWNDWwu6DBQEm82qnVOqtz+qfvTtu0fGCFaH11dPex1Wj99GV8xYIs3jgazGaPeVsnAmWPG1YOQV6c+dY3hLM02Dpq1vVdW1orpeVFVyhmqBzjhkHmu0DeMCmhAxgP1EYjcgwTYUWta1qM3pv2dwYuKTf/qEO3BgcbXa3yweOPbgNbprQ/9gUOfz9HSNl/MZzLxmtcKjE5dXnhTzQ4wtecMz/b72FaXCnzx0AVIJRlXGEJWUOG1ESJUUL5BD1DOP1ReAU9GFCUbYjAJIHbHPoWGryaKvaLL00wO2zcXLHv//WRQ3nWviDso9FJNTUq7KLOEalQHJaEXqy0zrjFRJ80nxrqJ3JeLUKTTqbIMPzRoFe0WyXWRKR3Pzyr1SNpbdsjfR5SRrpfEqukWaR4J2IQlqwLBWMrWsJY7MgdHHklnFYof0qlik15pIILUauY8AolcCnjL6NpOH8gWPVNpMMRnk0Sh+LQKRjERn7ssh5VE16VcrTKcFFHDT46KJrYemRnBE6cxGmKoo8GX/NpsTk1aKLIizCTrarClCNFnFX+zv6Z0rL508fUapfRShQG0G0RXFsEPCUofjog9o7+AAagZhQRgT94ilVMt47cZYJGU10zXXimBAumFl0dconSsTBu4fH3eb7LF0DpoR+ukyhMRjQpaIGYcI5JZKgJfELTY7lyh/PI62tkbehr92Gz2KQZdNpyn+hYJKWpbqNFTzAaihG0QumQFXDv8+doOBGG+zZnP6vL7uaCZH4R2MeNx9uDBj4UQhl19e2vaB5W7nIsmLew7sP3f+8t4j925vrxpUdTojA2pjpFIGjdeFOFZKlwE2UMXBewnqcW7Uj2EoiMUs8F4qdDByCgV7Vkg3bI5oOlajB5Utse/wXZgyzY3NnQ88xNm++Owzq7PTb33wHsC6FCA2y1mTtr2+vto7MLy5sjS1+5BhZnH6yoWpyUmnz3nx8vmt9Q1MKE/Ay1OLJ8ggJKvalCMQ3I4mdk7tOf7aKx/eucsxMkrMHsq7MxcuBodGqluJmTPnsd537d6fWp8mqRGJbOwYGUjniw6P9+tf//axt77XaDQ9+MhbZi+cvfNjH42ffMXltMSi28wNZCt83curKwDwQJiTBmaeU8fSyVZ0w4Jid9DfEIj76fOX6IqcgdGM2cUYRihIDoTDqiWaqzA6sUP0rExAnieTQkn4MiPFXWCW4DTrRSdIMrhJnaLYpmyUztSgXkBoOhz2dD7H+W+ubJs8ZubP2z7yKJ0cmUOvvPTypTMXUQ/hlejYR8fogcNJ0jKL4U1EY+7qbKad4alw02h4DMSdN2PjO8CAPfTIw+fPntu6smAJeOjQCVqCtLqcJiYn+QTK97EjWAAdirnOoFOmG89bWToDVqbajTVK+Er5SpEsyouM3BtLR5Pe+KCs5kz4yKvcEkWP8pGfIVXluEL0IaXSfGQbllu7FdnLfFVC0J2fiKMg58cXrJBv5Z+bpyZyja9lH7KwLSv4l/+YuNSiLaIJ/C7oxvi5nu5n9I7FzuZpGojHya5q5XqsmLZZcQEbUOKgGmiNUM4LKMTrchicThADxCEwBMjE5zK5D37g/ceOHP2rP/jUqeOvk+bv/vEf7woE6N/gdXug6SllSyAHxZ+j+KMiOh4eF3INCLHoTGxkcHTPnXtefOVV8HfknFqcjAXIi4SANzfXQ34/9yOXaHhDtlympLMSFzMWUgW9Re/1e+gTU8iUeHJmtwv68LmL08RdwI1ZTNV8mnbpqgfvVz322MCuKRcELYnEnMnQ7g7CTlUrJNe0RlWgGzVsTKUR6mqXc5zWqco9LWnUNPQASIzmAPhKAT3d3rSVbLuY5ZdAJkRoSepI5BZDBYPckK8bC2UTPV3zjZ6mpi8cL25GGx7/PqO1X6WPbUUz/WOO3v6eTTq3LK9UCmlyvfRCw+YiwEa2VrR9R4WIH6yMRe490ASmkGLkMhwYABIPZRggOxGQMACazTgzrAGmRpsNi9UKvodbjWBF72YyKYaZoKAzaYybvmxmGWLNUiFLSTKGGKWxRoEWMXzYW2fk8g8DBv3EcOl8pfyjDHlebqxUvnnzCxEw7kdnHwxURhVDjgsh/yrJWgJSSsGuGJt4twxdkROyA54xLn/HSMei4YMiHjgDiT/yngvgBsMfBdIKmYv4pRwLD1j+FI0oRxXvWeYMj0QsfDp4tMv4n6hMkikUVAJWr5RzMDz7u4J0hCXbQmSIP1KqOn3TbBEWJ5rCEBswC2Iax7RC6oRGaolUPhano5qYs/S4Y4cg17DQOG8Ox/nzigIXP4L5BmYdLigYcPRqOlwxvGFJpSySZDGlOzSnI6NJ3i4SiwGyQNoSoEK24rIwmlCIJM/OnTtH9JJkKp4Eyr1MzEnb9Lgt9BVIZiqQ5MOGH45FeVr+YCBPOLhUIhVHt6VirBiO58QJzpawUJQ6b8oJNWQa9HBcDk7FMgRmqC/SX7ky7XfZydjZjebp6/NCPCvWRsNqc6Xgu7EaYbMmrkAUGnAuF2mCwctAQVEjmUqhjRRoJSEh3PQaDDRgFmjJbtDqEb50O4E0G6PDbrPBqu10dQF6s8BOlSZ1COCztQDdzPU5cFfoy0Lp6sTk8LG779hY3WLweF3WgV4/89KsDV4oJSm2wRfUGckSGvDCYsnUO971TmbD3//dl5yQtWvVWDDEdgiZI9syuSqBF6vNFolEQt39CzPXe4Z3wkxNa/Dqzr0TQwN/83f/0Nsd9FnNj9x/J0702TeOw3w5FPIdOLj3hW9/zW+3UgABQ2n89Il9u3YUiKl7HHDK+oZ6PAOhpaWl7UwGlj7P+OSv/9RPHRkdu+1HP/T3f/ApOERhK1s4fZqaCoLV167P7dl/ZGl+dWT/0UwO8pC8yxfIx9cS6dX9O6YI/Wpq7bGpXRV7qnt4/PSp07MnXz26a8eV73zr3OkTP/axD9F/5vT1yyOjw2QEGQbHbrsd0itIqlHqY2NjROxxS2nn7lJr//BTn2Q80/XQ7/EWq5EWYHYsVgIE2JcMZaYbFhPzRFQy0ZcbWgCJzFTD0VR0TANcmbSsq9D0ojgy0h2J5ogvKJNRg+rN52vDo0MLy0sAIn7h537mL//689uRMHkfOg1DEc7IZEaycTab09k0v/cbfwBGmn4SlNOgicuZsiPkzsXTtqAP7O5DDz2EAZ1d3j76y0fx8K5duQLj29bSaimRouM550BPTBV0flRVOawN0o/VGlFxpLnYVYLalvmLdOOIivqTULMIBGSNTD05ZxHRyr/MRBZRispVKytFenTedF6//+0Nb1WEHfvnW9lpG3ybOIW8ubX+xtE5IWU9X8nx5ZTlBBDHQNXS8NnRaLndZhYMDg8988QzijTn3BmzYt12ZBTuPJFprV784LGpidvvPPbcCy8w8Xkose1wNVOxuixFmM4RwbinZJ0IuEJ5KKE1RCNNWVRGUcckLmqZSNTmdSPZATDmc7mxkWEU8+5du/bfexepgXqx9OIzzzFJMa+Sq1FbwIWptTA/19sdIgNdTZetPgsX6HV7cTzsfbbTJ0/TegvTAOehmayoXTTokg4tNGb1OO2RWHjfnv23337XP/zt17VtI2AourBq9FYgMIVciclv1tvoEmSsWd/+6KNzMxeW58+CZ86lVPc+qPqZn+wbGzWEggQ51trtst9ngXkXbccjpUUESHDAoBqD1WztrtW7s3m3y4HQoAQiSltRg7YE9S+QD3K97ZI+sp6p0azXYFdaUGRV2rra2C406yarK1sxbcVoez85vPO+tfX2s09cTlT08byP7oHgdwzVenfPAImxM6dfBtlQowtrmYZUFbgjSDeiBjg5dAZngaTD8wEexYNlRBAMFAXGY8ayxUKQdIIUmJBxhGQbvI7T5mDi0f2zlmMu2IDQDkKZTo+kaITWeMRTmFagfYqpsu76TOzuOBwFoUw6RSc5CvRVDSXr0BmYDJHOMJYDM1wYtTLg/uULg48ByasoJwnPoH9lFslIZmcytnm4MrJZUL1iCyo6S1kpAQDlG94wZNkPp8BvlPd8JIUM1QZ6t5P0BQnLU1EUsORgZZQDFRZFwg+pDOMWEeEkUsmRcQKI7lot4MANqHlp8EZVKduSsieOjX0jFWWCNaKGtUQ/8RqtSfXFMl3kCDtL6T0RYEYjBgPTQn4lPneLLtM8GKX0C2AfLqJQnmImEB+VxIWAvBjDdYqJ6CmoK8IQbQ+GjGge2p8wa0dGRjjTuYWlYFc38UAcTQKPeMDgDFDMgqLS68okYiVwrgJnIWnnlgrpnM5lh0dHUG/b4Rj+uttr5uj8MF9uAIWqaSp1fQMbAN2vNhihJ6oZ9FMjuyo6ZyGZzKTipPfAxQQcNmLWJ15eB/iHbQTiA+enkM+6nUaoHBEuBOyN0KuTdxIMNt3L63l1zWKhq1fKajNTvAWaA2ZrbCJ6EfGkcFkcIEOcTmvaDolyIgPc2h2J56SBOyrdaKYcNBKOZrFpaFBGWVNbh7e9srZlvni1tw8QWujCG6/DxW0xaihs0KoqDuq2cE4ZQCR/NCay8dOL85T5jk3tpJDXLg2H3EKt3WwaKLhWQbckgVOGMDqYwo5UIqbOldUGe3J7w+ML/Jv3vevE66/YnTbiRfsO7LMYNKO7JzcW55751jdh9ynlcjMXz4/vOUCt0cb8jG9w/NKzT/h8VkfQtbm5rTLrnJbA5YXZ+4aGHnns3adeP3PfwgK5AnIO2GVnTp8ixN2Tz83NL7/nQx8+dXnevL7dPTRG2DboMN55dM+1Vx4vpsOTjzxkuHKtoTb0D4/G4omBweHc6gIzfeeOHUZd++mnn3bYpW8x7igahBCi3WEDofbCCy98+BP//kt/+OnWdfEmaSlIuefdd9+9srZRhNZESMKqKFY0iKQPpKcCGVWaI9C3hqGtzCeR5ExleUWMMKHMOCbk22AF16md9JbRqugWwExhD8wLBiBBTY2hiUfLsMbBeuaZZzAHu7q6QLzhkVdqWIG9ly9dkv2pVYTiy7oyWlPgW4ASKw2H3wFclolHPEJtNz/z1NM7JneOHNkDgeU7Hn3nq6++ymkNTIyW8qX4yhrtiwlWIwiYaZBXM9jAnGO80yEIHjrGIZqRsc3sZJCLPFAWrqWjFDsf/5Gby3f/yoWd8Au5GEW/ygNQ1tzY+c2PRO6V8JkwDnJ7ZRMSZhTb1VrcKyQPr9jWLNRu3X7XnSePn0SEKaE9ufMC2OHCoB3CdwAITEYjEV1YWoKHgN24rdZ3vetdzz79DDUweMAelyOXzuFFwJJbKZU5E5Gmis3BSSon0LT5nBia9z3wwG/8xn/96Z/+6VPPvUj/LqoBU9G4CKx6IxqJ6LDsxfBFdGAZeBYjyc36uipHRlV2Uo5UYpoYJywCR1uKhbcz8TwXZXQYAl1BPYzTZuP8pQupcNrk1F+6cnlxbhUbvZjGm6AtHz2hbQSrpACC7DGapJU78raJ2/b35pPHl2daAzi+P67/sY/dFwmfpfkBXek17TKqAR8SS07Ko4SaDSlpok14rW5Xqd063aBJ363X+/VaZJmpQbq3majViyR6G4Xm1tya2xqkCjSeiBWrJZPdqjObYfXM4zFn7SbHuH9wOJ72v3ymtbWtK7X3RaLltpZ+6mxmJJadzmfCse1kKi6BZVJXUmVEgS9CGhgV2T0lbwPVRp3VpDYB2uJ6MM5ED4nXyUMCga02MF8YbzhOvf2DrE8TeoxGafnQMzLCc0eriX1DJ1oCmqUcWUjGFM2kXW6bLrxdT9Jte2SkmVhlRNcaqXqtIIMOHSX2HwOK0YKZ/L+8YJyKayjDl/9kkaEscVlFj3aULnpX1KToXuYTX3X+GF7KZrIlg4OQskQWULoSc0Y6yE1D41KlKh4w++xoX155rxxO8bNB6AkmGW+Su6qDuAoWKCL1cJBDXgiEBFolsmjcPpkThB0wR+UmC4cK0VzQVdAjE1Ygk4nXVeA5lxvQJxOwkUIkmm5gHPM7laDJJYQrC7vB6cabkBwlGUmGmIlSFTOgafGwGejoepKtatr18UDKYKHTCLlA0IH3Rhwa+YK5opjP8DAnKUdjD8x6zo47ZjXhJ5Bo1ljpHY09rBIdzDCxWR23HzsW6u6hiHBjK3z98mWIfSS3T7pSmEbQ1yKZ0atai1MFAquu33fHPYjJk6eOJ6PbS1e2dvb220DoBb1JYqPS0lCfz0OSZUGsI+u4pfQg4uyZZCY4Gmz2RjUBGDyRynHCJIJazTCOr6qFejWTNICms6gpm5VcCIYkjlhbZy5Wmy4bxkQ9l4UOq725EaYoKJnG51dZaQKo06TS1QsXrqAyd1d3NpifdZjTKTOqumA1thBnEJS/3uIgZE6SenB8HDaP5557bmLX7rmZaYdZS7sNCEpMRjMuAti3Wol7hSquOFymfCrD8zda6qdffmFzeXn33n27hrqwVZc2NjaXrg/0d8ciW2NjI9fOSauG/VNTwB+T4TAxBJM70Dc6Bt+W2+s0241zK0tIml279kxfm2WoHjx2jPHwzSeeuOPI7QTkn33y2Uff8pZzZ886Mo6e/qFsvoq18Nqrrz7q9m5srA91+yA7hUR719QBVbkNQbTWX5tdu960RB588CHvPXcZgx5VNjmx690Ts8NPPfEtf8BHbAZNxtMHNZBIxF1ux+nvfYdSN3waRg6WNWj54ZFRCZ9YHf19fYzS1c0wgx4RDRoOA11mgxiEgEZhxGWSyZfMOIahmHnwo6VrWHrE8QtZZA88wIHtSNQfcNBPDYQgwgClQqo4kU7R7AFdDhm40+ujsbzb6/6zP/8MnJQw8H/8x3/8bz77eTowFjAtMb6QI/iONMek22e5QrbS1e1lJsQS8ZPPvEjW8LajR7/9ze/88Z98us6zL1R67rjt0Xe8nejo4uzc+vI6cFktSY2As1quFqpF6s+sVhPs1Zw4k0ixS2TidfLBCBMJArJ3oCCSjMC4EJLhf73YEuEn8/nGK/tX3sqBlFUSdhSZprxyT4U1gQMx5zkud5ir5j14EaL0YHBYiXVJXGRtY12I5ymvIuwnIT0GspqbI2et7FhyR40GgQQCBjyWfCZL8J9AC1NPxIWY+7A+008eM6jMQYnL8QSxkCQYQJ89xkcLGG8D4OPLzzxLQ9V9uyef3974zje+7gsGCB0hCkFz5RMZxcMQYFeFviNvPbR799S3v/AdlV1lc1sLG0XfmJcRlQvTrkrEIVYbM4lnRwSFCqjJkWGCdg/9x3/vtFugAVk9s+YcA8YF2hh73WSBhFzdQqlhijhcqvERuqA1L515aWPxpYFh1a/+qvO++8ctRmTVud4eSkAZlzxJCIiAcei5c5xTnR4JePMaxDqd4mxauhQae3TGEVUDxE5Fp6b2JFuDSpUO6vlqI980Gcw5hEeBea31hPx1Lezu1UzZUm55EzmbRjpR9C0sNS5f3yoUKQHq0hoG4OvgRhfTNBOkN0Qkm4uTPcZHUpSHiDjgXRJwFTO/iftEETGl8khQ0SJwRkpEmocnI0S2xVPjoZpw7pgfNHJIl/MFHEvf8Cj8rIDYU2lAMnF2U8GPK+Yo9cCe9ngd3cEgE1mXzWjmZ+MTkwM2e6hQWESyUUovEIwbVpqiWnBNJXgsKGEMFTn8v3iReS5jl0Wi44KgUPK7qFDRo+K+K8q1o4+ZN4qW5eCiekVM3PhWhiDseDdSvIDL+bm8Kphn/GAJOwPfEAXMb/ljsHF/lHnCSTMlCZxC1qAzWKDXgdif2Dd5cZu97rBbUCkQtKIqufGcK54iWh5VDQi5rib70+CP9C8uH/oYF432juUq+UWxmfAGcTUY/cTPifDZJJ2iPBZcD2mVqSLTpTGSscXzpFuR8GBxA4gu6fWsRaM3Uok0tEHceZQ31Z+4HXYbicYrVFLu23uAuU1uAhbeWDxczJe9TrvGYnTYAFkYuU7orZwOGwFnSplWV8OjY4N2V5x4dXdP7+GjR/bt2/f6q68TegeQhE1RgkpS16KNg76lqxYam9E0HYbgr4Jj0i2Bvkw6snFmYymXi2qgbqb7aLnGA5cMUynP0+dxiA2NISgxBDDGZjLNaj1ep8Xs1Qf9fjIGWxsrEIxgUoDnwnfXmDREmWnFIzVv0jSU9iXYfYH+wf7N7Q2cA4vNiXxBtxvNehonSE8VsGIqOhi3lpciMHbt27cz6LQCO4RWw2ZWMvHNNjyaJqc/U2n0eIJrm+GHH354cX37yvTV3u6BYjJlRY3DTsW8pgiwThNiMGUCymu28uTyPF6guHrqjHPb6ykP4XYD0Xii6pfObwzvGL9w+ZIZNsdEIuQPLSytwnhQaSSrGj3OK3fbaTNNX7vUN9DHo0R8uj0e8u5PPPM0zOx+T/DQbUe6enq+8uW/G+ztQ6redfsd5y9fvv0tb51bWkU833vnseuXL+7atXNm+vLkWD85iVxVfe3khWyxecfgaLGlnZ5ffeF73zBUynffe/v67PV+vToei5RKhWGALl3eCxdgmjrj9wWL1RqNjkEPULpGSsls7iU9jGmEG0qTSpcv2NPXL+JSZhADnrbOlIngaIg1jfkuswgBoLzhVUxFgmjCAiEE3CAi2rqSRH2aLaym977vAwsL14AFodsy2TyamZYHcEES04BMA/WPnwpu7pd+8d//6q/9x+PHj58/fwEs2PrCKjsUbV9RldX4KVKACUYVlyq8sWmm7SokLS47ZB0IJoKlWJkf/eiPUTFMN8knnn6ChCXO9OrS8huvv1GDlbyUlVnM2GvDXI3FKmKQPy6EGJoiXjqOr7wyIMWRREyI2BEPpfPmXyyubmwo0upNv+287+yK43b2zBvZTGwMWdgGuSmRNSUWTVgO28hpln4hpLFYeCi3HTn23NPPITZQtVKzAdpbrkdsdeQgl0X2HewHYbNAd6i8sc5u0YX8EFkCRzMpHJXVimKWMjCp2kAlE6jCqCUoDVE1rFbt3FbeNWgdHu07/uprvT0h7huF6v19PdViQSq4QE5gkyHCuQj8h0b1xedf6OoOGgMmAt2FWJF9JsJJd8Cj88idJGtLnphEvPQPoEmpTnX50hUiabMzVx+8/563Pvq2L0W+CKTZ4fDZu9yZaCqXjMLfarO2yT5wZaU8OC3Vrl2qO+7W3XnncHcfPsCazQoMk3A7XhOsgkTjzIhBea78Z8CIMVDNSftWZKhAZLVQ5lItYVbpeAW3qqkVG9lMgdQ4dckNeooDaCHya3U11Pp40ZAqqWvqPkdgT2JLt5VsbkZNmQLkIvpKI6A3Oetqq97sLBcqYNMSya18MdJu5sBIkT0k/yVOGyqUgSN2K3dJ9I9GL92XySHKmBM0BTdOTlXUDCIbkG0nEiMePF9Sr1SBX4lEXiDoIwgwu74OkgvsFe3YpQRW1fB5gGb6u7oCTrtZxuvWZun4a1eGhr279vfR7w6SMJwjFZMWddPRXzLIOCmOyRyWwfevWrAiGKgdBKZcljJlRLkzEOSjZHnZK29uaFxRujf+WKNoYsYhKhbbTsatqFgxBhUFLPobcBB3g10RjufOIU94L7Oxsz1ni2zBohAXgNNXuCu40VgCqARBhkvFA8EYSrqU6SoDGWNGbFl83zouFLaCWk+fd+ixQB2JXwfhBj63uIKyQ/44rhgEpJbF9xVrgvOQk5d/hWwMbxU8CxIDZYixDv2kmbYPBlMqmY3FY3gpDpdLZhrsRbQJ8nhoq2B3Ulhko/EDjLugo8HT+fooAqCS0oDMohMwwwLQk91mQaRubW2EQp4dk+MBiV038Ic8Pl+5dEH6k0iP+jxznNOjpa44BmpDslDZYXMPDUw20rnI9rKeOiVV1aZtzF6XxpbMcRoWwVnN9LA6PMlU9sqVZRk/YDSM+KiSBsaOhc6FBilAt8YnxhhYDouJhgogMPMULVbKPAIAs4qMF85XpA6mPCVdBosZAkg3fFxWuJK9VhCZdps1T28L+BkEzwzDptOuw3ohAwobR9Btxz4lQ0NSnhiChXC53dM1RBu+7mD/qMHswIz6Tzv3ffYzf/b6C88NBLxdHhc6pZAjKEbeXw88gEeLokmlm2SX7NZiNZsmwhZwGNulNFGvtc1NW+j/oe0vwCzNzzp/+Li7lHtVV1W7d0/LSPf4TNyIkYRgSVgsWV7Y3T8LLH+WkMBCCAkRhigJsZlMJuPePT0t015d7nbc3c/7uZ/TM2SB93oXuPZMTfWpI4/85Nbv/b07c/nUy6dPUfS1NDfvd7qS+UIqlu7qHLC73a9NTqXOnq8bLb6uDitLqVzp9bVfm5wKrK3v27PXYnV2tPeAPyCmQiLj9hN3zE5OvXr+1cP7Drm8bnwU1IyVJlQdfkLZVlDjVhMYrqO3nrx88by3e8CUzy7PLwwP9nPnK7BHqZoLN6573c5GJHjlyqWDBw8szExmc3E08fHjR+GVunx9IhymfVRsx47trBkQqhh3ZJvo1kxlITB4fE3WJeud6WJ9Irpl0WGNiI5Qk0rRk6JX6hQpoiaMTEE8Wxt6gESyAPLD3+UlAhSNxTCudu3dc33qMowlzE8K2isTjV4IgWDE+Gdm5v7rf/1/FpeXr16/Auz5P3/yd+677z5i/4qdJetEFAqmH4udjrcWCE/hLkKAqorJnN5mdNnsoY3AUjrNYg7dWH62u2v7zm30J6Ae/YE3P/DpT39msK9/dNso6JXQ+mYpnQMoAXswuwpDlh3LE+ZWESaIJsxffgR7jIIRYSFuAr+J6BEZFl/z3/3gaG98943nrSf85qZaNg02JIK4JYsVggXoLCQdJnetA5xuYU+tL6/rtRSzKhaEclCZD+WBfEWeEGekeTxkYaAWyIgTAEO17Nu+59qVKxOXrzsQ2ACe6cVdqFgsepxNRF2d4sYCUhECxgZWFcw01nZ1PkddZL67y0M3pOHhHnJbgfU1HG24H5gRGSUIxcQTkk0RWY5SAozsgkXK1MmZbfBnJTcTZo+DkJTbbsMKrxOegApDo4MpAOn05je9OREN/PSnj5MGzmcLequRxlzk+mDkMxobDrME2ZCLrCUI7u5/j+rEnV1j4x12KPYbUais7TYT3cu0BsQXdaB0WQFspSdnRF5MT/kGuWRJGyZw66WaBP1RAw2eUFk8qmq2QdO6ZDwbpx0q6Q06Megpr6QbeqGmD0SBV3rNrm11dd9CwDO1UFxcrWxGaoUqrdmdcNBSepvLFlc2llilhVyyWEw0a0Vx1OqQA1XxilhFBFaYVpxaMVMUZSeRB0aN0CMqhPfZny0liNY30jcH90roUTmmfE+rGd+xx+VyMLTRcCidjLMAiGpsrK/Wq0WX09be5u8Enup1EmjP0WojFdelUo35hfDExPKOfccsJq+qCXgDFFiF4ZO1xyyxl1q/lUXz+mvKH/8Hv+R+FIUqukhJ4sq2ETWpuKqKSsbRRzrwCpfbUp+iekUTt74rZruiqokciMJGkbz+m4+hkiUtxIZH8d38ouhIUdWEDXgdwkHZmpJf0EFzDMYYfS0bCzRBWQXtlMWszSu0f3I+zDwpLSUaD9dio8TRDUbBIEp3CCDGch9MVgkThYi1dKon6mK0iIrRE/aBzl3KoVSkcCRUIBEKYtPUl5DHJ2wEbUWligKGkE+iV+L9VfP0hccWJT2nNpHCJGdHEa6vrYPNQP6AWj2v23Xl6qXZ6dlUPO/FfVMgTsQGIQihoKhl7/MxTjd1Y5I4JIz/WGbLczPIYp/HBdF+xUAzohQcVeh9mjWY7d6RngGL1aHu6tblMtX5bDISyEcXK9no1tE+5GAuVxoe7BocHke9+dp7QpE462h1bT2VIKIo0RclHlNHIiCmS2VaElSikXjFJrWJyBoWJXfOxiYabDBbfN42r9/PJk9iuxd1mVx2Y3PT5jBCAkCZl4lkqss6PbdMcTYuBKh2TCDuS6ut4AtXc7kaHfQwmFk8cEPCHw1K02If3bJNOzCiytcnZxbXNgMEgfqGxrsHlprFNEqF5HRe3He2BDlv2PMxqLgSzAUqgXXJaLCaNRvqpUx0XQsW12hwWMz33XdPslQH2X7k6G2P/uBHfOnEXfffuHw1cOla/9g4MTiaOIXj4aXpGz1dbcNj29od1M873Hv347fYg9G5xaW3vO+DV59/EacZY/K+933w0nPPRWMpT8cA/MYkqF569Mf79++dfv4pj887MbcMx0rfyDgmFLAoUqeMf39v597xwQCgzHTC73Ulk/F7PvzzxaWZcMAKoNTtcQ3eefLpv/2S3elZXJocHhvDcVleWadNL6ONDQkFR0dXz9Xrk4RDmAKWOopKTGZWrIgQ9CFWZsNmM0F5xUt4ZrwqO45Ui1R21YgvGK30UyrQqZLOOQDrzp1/DTJUcg6YouwXLB92B12eOH4ykoCb6fYTJ37xF3+RguCB4cHXzp1HbcxNz3BwomiyY22oqCaN3/He2MxYVL193eCfaR8WWguwjGx2B1g+a4/9heeexaajdk5VKv/4J4+ObxsLbobWguu0ggEuIGIGI4INolXZLLYy60CJyiryQcnKsceQjIq04Ak7Xb7CEhQR9u98tL6LfuT7redvHO2NPzk8sWeRNMLliSsGagIDQewDgjl8kUFgCxOKZFKodhP4OmUzvC9WguSwFScYVw/JL+KIXBFpwkJBQ9CId8lgbWxsoL+pZYQ0oxwokrWSe9LQFMEqqhwejgoBT4EwA6VhB6myqp4tVujR4C5G0APggL2PVYYPIERQJfrvUM8qvhRRIpR9w6XG1t9YXUWxlOL5Uixv7fBxRrLOzWQlQelRqgHpNwE3KCmS0VTVYv7QBz4UjwevXnwtFU+xw9Ci0HeD8IN/zUioCfBcHvCd6sQJ1a23uvbu8Xo9mCnhOiWXzbxKR2ORsp6ZUZUJtpLVg8WXaAv6jQaxKo0NXGetkVVrSLiUKI6r19Ll/AbFkw183SpB4/l0Zg1yuCYcQhU7VofFblkNxtCyZudOnXVrptI+OVe9cD2QKVpjGWLJLgsN4rR6VHYlHiUXEA5FBeCD6lWVCQcCoCF8LnJd7DecbiYNt19g9uJLCahZ3GHJKyKCBG/DK/hSWkI/4CpACwrFJXUK+BQuFx0OWdLEimjnjMUswp9SJtqSl0p9/b1tfhdeCgX2EG5XSpl8LgMkEz4ELCPt7PTaylzvwLAvm2U/Y4GA6Xp94XIpigJ+/e9/279sb/m+qEceyiZR9HFLU/JbPoD2FWtRniuf4Ylo6JufUfzJNz6v+MQcT35whZUgtsRv5Ouizm9+CxOY7UDIl4+xPKTAkaMS96B4RUepFhWiiAmauqh06ToN4+sV2lqxdWm7TY4GxcFOEG8TgIxgFhl1HZJfT2UQuplKikolZ7EAXNTboN5tShNfWl8Ryc5nwrhfEr0BJ8Ip+A3tBtoE+1+pt5LNipYBcaSYINBuFPJckwouj1ojbOdS60231wdsEjEH2oXfLRwW12C2QEEqRNCkNoHI+1wuBjeN5wLsJ5cn4cTOkcLlRnXqxnUWlI+9V2oY0PpNe11TpTyKLBReKdfs7x+EVqnUvEbXGpLRhWSEYrq+DhdomrCmarXgYWQJPo+M9hktTnhWj996SHNGNZNf4zJYWIwzuW3WHBsJTs7lpVXcVhWoRJUqGo3ZrCZiAqQtq5UMCtjucOHug4Gra/PVOJxhjVB001WxgXhsM2hBaPnKjZHR4c1wnMpjhA6h7CqxBownHAaC14WSAWYso1XEnBijJJaaV65ObKUJqbfr8Sef2bP/EPbTc8+99PFf/NjX/+pPGTEIXJFQAgUqFtKpBHHmYjbrsBM5oOGOuWHQNyvsihx6nlCezmyfm55q2Owjew5cffH0wPC2bbv3L12f2HP8pN/X9d3vfjeRzgDIg22vt7e33eF06UzZUMzjdhWTGSphSWYn4vHOtvbQ3Nz1yRvvffd7sJpfe+mFg4f2/+M//jCXy4wcumXl+9/FJJq7fpmgYyCXxip67dKVduHaTXW2tbms+nwiMH3pLOaLy9ft9/swjRdmbhzUNr71rYc66DjodRKuP/X1r4PRO3P+Qn9/L9TQYAWAQy8tr0LdBQbPLCRrOYwPNCo0XkgMlMfrm06mjAXKGsSqw7EQhg3FQeMzoiNpF200YBmQY18LbiCBCLGATXvq2Wdwf0fG+pbm1/QmoR9nY8D6i/nr9vs++5m/+JWPfwxdePiWI0TI8VY4DnZA0ybVQxyZ2jnenbuxWM7XHA4Ti9Rlb1CLASDFZCTWTsFJWucyI74HhgYwHKF4c7d1nz1/Ftqvt7/tHWsbq+GNgNPq6Orox5Gfr845IbQk0IUtoeg2ESjKQ25TyniIXUqURtFqcu+y1f5jjzdOwWHeeN56wm/Ozykk56S8y+233pJ4vkGLTZOKs/rSUF6gm0mFiPcklr/01gTjzKSI3BOkj2SOSQgTq6KugRqGyckJdj1jsjC1sH/ffqaGmGk5WzPadVYH75hI+gK7JVkp7K5khrCLkVPNqq9bWyzkuzq96AZUHxJ1cGDLC8+/RPESZxdJjjjFYwYGjZtBVbfJSiXlO37uvYgamkGUUkmN2vDJ//x7mFOLC/QsKkyfvUIL1CpMdcQzyL0ZLO+47U5bt9cmMC2dTWfau3XPKy+ccrpt+VQMTrw2n2rfXjit2g8c6Orr01lMqUIujPayUsSGqi5nqQt0e13Q40iaVZYfsrXNpG3Tav0aNQ5lRU7VoDC9BDklVEXFRK6cdWRTYKPAXAar1TR6Ulu310t6+loEM7XliK53YJ+n7daLE/kzlwOhtCGeN8Fqp7WDtrIjFUGuSY/uPMhqcr0FKLsg+Ed5gLFCK4hjq7PSZAFHSepdZSqJmDGkTBRWv1AJSBCV6WIAkXs6yE7MaGF8EmaTXBcjQSs5Vr7RZNrcXKXHCD4ioK1iLE59mLe//5bD+8mGYUep6+VYNJRJRCvVgjRKqpZ1aZRxSbW4uHnx0uTA+B5IP6iK0Rsc4OSkgIi1xSr5VxZxSyezhJTFJ3ZB65Wbsyxf46FYFOIz8hCLUFJP/Am6Cf3KK2wbKXRSFiV/K8tXiSQzCOhlMdTk0C3jnQ8rYS18ZRLjDAtal5SGHOqm3lUUAwfjT3YFBYhoCNZ2OgWxAdT2bADQbAwlQAbBbIBzYqzyOU5BBIL8qLRfgB0djYJnLGVLXD/89oyyqFMtOkAwLILUJWhN4K3A9PEt4v50Emw2s4w7TpvsMW6KEUGjy2/JFyK5RI0LcSMRb7W6qBXGK7oY1jS5rIoWAGRFspTQFvIOt3dnX58k88GPUoNfrbT5PFjQBJA8DuSrhJLgk+aUiFS2GdVHEDUQfpSQkbQhMiViUcxbl8OxML+g0ZlZOhL7FnmEc83/IA9VG+vLVS2oYSM0aOpKrlnKmps1WFDT+QJ8mH4npO2xVDzk9h4FGUG6uZ0SJRtRFwxnmTiM3FKxQddGm93IwiQC47K5k7HUQF/n9EQI7c2yJe9FDI3RZjUKDwP2RTjtcHZ73O54Ikg5TSqXtbhcEIBUNIb3HDr66tkLc7NLdIYkysGoG7UqutPl07lktUjhlN2OXOceMG/wEXKFevDalXMYwLcf3ReIRW45cuvOraNDQ30YFzi1aKN2u6/d6wkGNgrpBHdPTzTw27TdVjUKTpMRKEQ1nYT1BsMI0LBswnLFYjCOD/T/zec/9+uf+LXx4cFrE1d333vPe/Sa7z/8iAGilaYqn0xvLq/7t1ub2sJQ145gNH7t1MuA43FAyRZAQrJ753bDQE928urM7ERPh++9H3zvqRcvuWbmQAtTrlzBFsYBqFdGB0bPXr565szp97///Sa3ffPG5d6BLWdPn0LXriy/2tbuh8Orkk995Qufu//+e6Ox4MbmOvlmVhhcHz29gw6PL5LKbYbC65shcNes83xuFg6HdYDWPb1UeIfCVDcmlLyMsu/EaJfmviBKsIpwMljTrEO2hjJH/CuNganEsFbsZK+x6ufnlzu6vIH1eP+o/8DBI/MoYIMeOQQuoKuzkwBdKpcgrrq0tAxdF7gwstEYhbFQmG0JnL2YKbn87n0HD0kyxfkqgIDFuYX+/r6tY1vxCL/xjW9iYJK6Vtpn5SACisTioBhMbpZQ1ORwUCi5sb5JjoMwO3OXiiUB/bJucVikHSFYMbi8uGiJfkngC3/yDeUnjozy4BUeN51hEUD/oUdLs75xiDf+ZK9jZyMf2NwIXXld/qQMr2axGrkYQclFU+LiEk5l4GTzI7XFn+LDbA0iehwWIQZjikhNJUGbC6ZV7epyuoDZyt6JBCNIK1QCh4VoET5t2iUoe8tgsJvIMkqjF6laomkoPiYmbh2OUvYWWLmnn36aM9Ry2E8qsosSkkcBYwDAM09Le7WWGsGp61Mf/NAHYH350pe+lI2EvvHQ32Otkd0f3ro7shGKrwXzcaSkSmdt1NJx8tC4JqGNNUjZWRPD97Ud+a2P/MkffMnvUW3bozp2tO2O2wfHxx0AktKZRZ2uSvE/4FB4eLBBHW0D9XQ8F8+JDiP7Cn0QV6Cz6i2dalU3CkJVjDeaGSg4K+VsOU+TtxRpolI2oVNZ2f3MO0Ec2lgXypZsWhvP6sJVx3rSVzC2z0YLL51ZDybUZmcH9EjwFhClJ3QUXNukwTlOEDkKFdXV5KfpTkimlxGXHcLqYYkJEzoKAW0rkUtsDXxdlJuEClhcvIgvzBpTZoliGqOVvBIRIQiIYNlDyzCDsLIn11c4DME62BLZBK52b2fHODSc6VSMOo48HWGKmWw6CS0i+ctMGjKkmHrY7ixW0ocOuwaHGr/zu29yuWL16qLFVtBocjDscpVcJmcXTSJBEjQS/iCrjotCD/L7pg6WsZP/xfBkQ8tClCcsKHGocStZBBCTQBkKlpsfHHfpyFcV9gx0n5ISlv1CD1qgcewrGRz0tKB+2GYK9qeiAVQs9J70PahIkIeINEykr39eiWlLVI0zyyWDDaJ7ET15CPLnYSIRQm3e0hKZkxvidiSnTHySgmBUoMCpeBE9V+JEVdrOkPeVKy1U6VKpow0q30UI8J1ioWq3uQKBELcAqSNrC7QPSDgxNWigxo2LyJO8GlFfyGYIc1Kzh3muCAsZIcYORUzZW6nELdfpG23ADcIWqqt6Bnq3bt9jMDgYja1j41QMX7hwOri5kslESrkCB4NNC2MHMc3gI+BofgIkB2gG3R1gMJAG0EwW6xQEHp1jpUX2enxjuVkqOAx62Ob0dp/a7Hb0jmRrhLwdsJ6vTF8uxgPl2AaTEU3T0D1DcpHeH0AFRsa32lze0eO3vfTw44FQ9PKlGwTYErEs/Vq4hoMH9jpsGBAJOGCJjJfyqUalhKbRaZtuykgoTRRDhEijk4i6E3I5ugXShMLfRSZhdmEuEA7cduK2QrUq5MntXaz2c6+cWZqZKyTYdWWPk8PUYJsGe4XJD9SQDCheLWZHvlq1+zyko80O18i23blyMxhJHjh4y/PPvzhz/Qap1m0j/avz0w/eecfElcsAUE1WJ7ixrt4+pGEJm9RpguiqkAibjdqBgSEYy7LlprezNwtQS2ewcX0DfdhXZKYf/uGPOn3t5Vzl+uUJq8lOHHt0qKPZKNEsAYvH3dWJ+Lx0/Ua+Wrd72rr6+rP5IsNOwbfdYjx/5pTX06Uz96VSpcGBnmBoI5dJ3Pv2t9byoJmwarTf/eH3BwaHbrv3nnOnTg31D5w/f27/1qEXvvf39XyU6O5maLOntxdRyxIla469QjLO7vZwIzqjBVq7QJAOyiPCpVAorK6u6vC59Pq+3oFAKBaMRK9NzmbyVWiLkLdsWrEuIe+mqJ1lp5jK6AY2LFtO2WtCJGaxmQUF06wl0+VDx3aze6/dmOjo7MZb+9CHPtTTP/D//smfUUnx7ve+b+L6JC44AofqRqqqwQFGQgG2GyUu8XCc8hUCsuVKee/RWwBIv/c97/2zP/30r/7SLxNTyqWyf/AHf/jlr34lFNjs6OsjlGd126kqyGF7xmJYQSxQtgetzcHi0fiDTWezWEM0hq1WXW4PlFs9/rZihlKmHNqGihcmNJ+vUAnCspRoMDY6UgZbA7Bxg0BWueWecqcikV7/zRP2Dr//5aP1MV5vfeCNj73x5H//CqKb80qUi4c4BiLxeIiVTzQCC5V8GQ8ZbOwXySNKypCroT6bz3A6ESgIeSE0VRx2RWTK7uIHC57UFQWrYtHi8prgdScYoIH61U3YI+Ht8zGhQJve/o53kR345t98gVIiPLFcrtzu8+zatev0y68UEjWbQ1dIQaIrqlf6rsKYWxWmGhBfxRqVuuYGfG/5/PDOvgS0a1DyQB/VpMu20et05VJx/EUQNXiPRjvs0/p8iEIolBVeS53SQJtRtXeHCYTMsWPG228bwOvVGVMaTQYng/ytxWbETdLU6Xhk0YOlUhCu2B3kffBhKmpLQwshbr/e1N9QtTFGunpCVYpk05uJ6EYmFoWRQkUKSmNNxosWm1dlsGcKjUSmni3i0tiyze5Ly9b1lKVSgQKSrWFOpuHNMJLIY/YxXMBalfBvcPk4NKoF8LnE0BQbB8WFgsH/Q2FQcITzS7iSKDTgONQYBp54KxW6tBndPjUYoFLNZLRp9ZKjUWWLRr+/nTgDdme1CnwEpjLM1lIZ5Egc0Q8anGBbp5/qOSv+ArxaCEaGCO+fPwSWmk4RiIbPQQfOCCdmbi7V1m5/7cLMW966PVeNcCTpMiBlOCTSFFtNCmAUtcqtsKEJKitLG6tEUWXKi7LGWitXPiRqkN9Sb87SYp0JOEkWm1JEJD4iP4q+xLHlM/I/38eYFcUtupt1iXZmtESOs7h5kSdNxkayWfzJGZSPcbCbjrT41coWYC3jQAJyo0qHyBvKAKcQDESFbIiMsrjVXBn7QvaPjr5pQAmVQDkXw0FQxRpSFVqcKTUNSnU6pYYrT+pGDx5PqwlHQrzI3kgnMm1eX5xudCSppPeAsnM4AjcLEKYmTe1Q+MSnuQquA8EH7IWTsIBxivGs2QkMDWMqqGHA9G440RzAEyi9Q8XEIJ2LxRPxKEkRArEGUMn4oU3wTWZi4j19vQRa2cnmEcPczBRVDuRayqUiQU5wzpyKABAilfAIe4osDjB4a0lr8ZtoWDSyYzd1pTSTX19adOubHJaOe9qGjjBBKQsNiDDWJqMxlMpoPAKrBAcpF6rXJmboT8rdcZ1ICG6atohKIoWwbpU4DRUC2KlaN46BLA3ili6H002fRautoTYtrmz2juwgEWh2eEb9HYlsua3Dn8xcMxiTXBLh9OH+3igRJitZxrgoGwCK9Dyu0yZEUu8sCfALIAVTqTD5sk49ivZye89QYHXqOzM30J5KxF8Lco0xfOrJx+g/mEmVUP/FmmY9GO4AmuJ2ry7eQAF3em20QF7aDCIgcCsXp2cdnnar3w8Convbtgunnjt02/G3vvudX/j8Fz707g9OXJkyibuoD24E8RyI7dPncG5+es+Bg4eP3YKbbnb74Z8iyTcTjY4M9LZ7+u64/TjlrOWGB9FApfWIYziwqZu48CpLY+fhg/HAxkhP+/iWIYKAh/buh+/6wTe/Y/L8abB13V1bSAF2EBgol5fXVgdHtixvwFXrYq4XV9eVSn8baYTh0THIlkmdoIORNcTBTBqbtNilEYykIdm8IsHZvWxVkTqEj5R6Fix6tqmy2eQ3ggYhxPAKjKxInZra5hAPjTppio5AdZFEuHDx6uTsEj0cd+87aLbZZxcWQUV95s8/u2f3zj/5H3907uyrpOwnzk7a/SaoKOmTzeay93qvXL7y0De+sevAgW9897seQvQbmyNjWz/9mc+yBywuD5cTCgb++o++ePrMK5DMvOe9P/fQQw9xF6AZVq9PdIz3bU7N5GjHFIlwtG179s7Nzbk6PIHNCJRIbD3Wdh6Ubb1hlAIG2r7iFks0U/Kp7DslScyT/7sPDGkkGJIK5YrAkAYvCBa5ADskPxK1lA6haF8kFT9EPRGFWihF1FCXSl6M++BeuHIqAW5eKpMjMkRi1FAkwJbHl973nvcBrvzWN79bjCed3R3pUChFxSpAJ7NhdCfbqOfoHcdR7xcuX4iHNu64/RgxE5ICTz/5En1yjU5tLqlYAcoJSNth2ELbzgJJJPMGKuftDoPFtby+uDizJsg9gHjdHahhbowmuoVkmu9ZbVjY4hbmw1WNReVzmHPJvN2s2rPDPdBrGOhX79pBp+ya35/XapKNRp7CIitAaC2UOLT8JZVvopEvVidAAiSCZONE4MGZYFM1nWoVDH1mfkRq0lA1k0lECjQmLuclSM7aJLTQ2T8YCJc2A4VS3V7VdKSLhkiiEsw0V1LucNYG2oTObF6vva0PjL0afgSALMTJpCJI8bqYAaSZJKjRsri/ij3FZSCRWTLsFBY8thD4c6aC9SR+LFql3uwY2hJLpphHtgNpwxy0JCZbx9gAmTwMLLJdSEmcXfj8WXIQ9wKbpSxFwtEQ9NUryQSxbvC6wHdJHuZAp+aQR/ksbQDEp1TgSHCP2aNR6kYqc9PB8l37bNYeuuNJNTQXwmWw/LlcWdbADSRXwZDwP6KwZRW+8USZ33/6Ja/zMZacaEu2uth68kRyt4gDfjiSokjlXR7ymy8pi5XfyivyYV4XzS3fFtT0zYO0DiW/ZXGzFURVi/ZVFDD2DYQ+FNiWddShooSVmyFLAtpQJJM8AOmwbcCeULCH2U5oC9MMySUXJeWRYgXlEpBOgehTA8mlDRKU6BosEzXbnv4uduLVq6mMzyMFSMyg2WyoCCW0mANcC9ej3KbcKZsNtQLwgsy04E4Fk1JjQFkZYgsQ5ZYUrxW6KD94G4uJPtGlYi6+uRHYWEmFA3WIaewaQIksD1BhwKfRBsSaSOTj2pALRC2/dv4sWEeq62D5oAw3m89DSiTJPdSz6EELASMsFZPJYbW4hrdspSJ9fT1An1e92pBMp/XlQq0EvkViD0SzAb9QDYUQERR+tZnPZIF4E2wh1qru1SUTWRLPFy5cGh5oc9mhGi9bG3r4vCDKAEGGL4JAQbjgubIeEamAqiDEKFQbqXJpldhpIED4d3R87Nxr593+jqPHTlw4dw5LZd/+WxKh0MriBva+yemulLJEzCX42yzxCsudnQvcDRgZYflAPMq1omsxpzs76MTIVsntP+C+cfVKYnNFaRhQLa4H+kZ6KVvQmR2UcRfoGKTXdXQPxENrkWQW7JVObYKnZnUt6PG2F4v1fCDWTGYtL72yvLii0xoJwRw9fHR6aravv395atGk1xTy0aGhnqnJ6XKltG3HjksXL+OYHrz1DuLn3KrH17G2Ebg2MU0m7+DtR6Mrm5cnboyMbnc77BdfO2+zGI4ePfy1h77sc+nxcbORDc3wYPLSebfHZ8fcIeK/vEAmcGkzum3ntoWlxUw22z2wJVMo4/i6fH7A+TR5bevsMlnsmDbsr1w6BQUwOx8hTvofAAhPWBgYZ7IIpSC1tbzEcJHdx0CKZXtz/cuLPFh8zSblbGAZ0d8IJQgRY4k4LWapjuOwANOefOIJMGf7SWvv3x8mJr6yaHK5/uZv/uYzf/anH/3oR6cmbyTj0f6dfak4yNIGHYCp7OLB6Ul23n7r8d/54Mccg53Dg0PpeIJ98eY3vxXD4uLLpzzDfRQj7d2z+wdf+/tvf+ubhOi/+YUvpvCNDLrNuVlCH6psQW234GdQQHvbidsnLlx2ux2FVIadi21XLhM5JylkwdHkZpEbXDwjgDTg7Nwr8pWtxVD8X3woY8kgKna8mAUi36DUICwszZsJenFVik0gmw+rXupcqAiQyLXgSUXYtD7wLy8SAZuNpdt6O5556mmny3Pk8OHFdt/0pddEVaHLIGglGYaVTAOMK1fJ5ZD7x4IhD+Vz+/DAX/7pKWQXbdzUVlq2GAKrQZxgBqpQIACZx0vTmjRGhzpbjiMCrU63zqzHOwN8l6vmrV4TJM/QJDi67dVcg8wsADFEj9eBJ5lNBfLcQUev6q47nYcOdENoZbMVTMY8hJYEJiGmoAZQRyFvGafHBXkP88DCoxgNUxDHmlnCEaIiVKulTMCuqlJfWmjW6Q1czUcjmVQoGYsUimmkNdoRL6VU064vBasqX0nfTlfuSMpC8XwoagxlNEk6KQp1Jkg94QJDeKcTabrAIcGogBclJ36eKB/pXMRylHppGWlZMMSV5bkoOPxXZLvWYKNal8gk1gdMBxRJpjIFalb4dhza84ama2gMbm3YZZBs0EkmYDJQo4Dh8KO6V+u0ErGyOGy8aWIHUJCAzMMxI/WZSSTwfYmB0aGV6JtoNVkNYM9Al5SqIA2Dm2WYKK9dWT90e2+zsYZwxwzAphNDAO0rZh3bmL9RK3LFytJWdDMrXf67qY8VHSgbXu6s9RC3tfWDESheLLteDiVf45Otz/EEpYfm4gmfQcu+zuqM9hIwMzqMyDkvos9aylhq7tDvkHQjW0gG8xlZ5BxWypDwKsU+LaK/IUeExo4tIjuTTLDUXSAI2MOENu0OI9aqdJhXwnGtSwXcRQ4Yq5o/IaEyob3QTeKqqyHrSSbhTnOhkTmrwwn/sNTPAUIhy8aMtx6vH4f7FKuCrSgpdbi2yRJIEYg4I4r7jXMuiQjeNWr1EFyAfpPGGbkKadLNpQXqS9hn7W6b3UJXEkAK1nA0iW+El8OMUJ7EZANssXvstO0rZIUfmCMTbwd6RxCPZifwwuTKNWOzadIarCarVm+lvtnv655Zg0sf8y472EFbWaIiNPirQfnFjZMj1xgF5EzyplCpwinNrs4WysQDWzY7w8gSYEaCwQidTBlibpMggbVhJhbDjSaE6lmUAveLFQbkj0QpDQn0FmcMUBUIH3eb2tcOL/Tyyua+gwfuu9d/5vQpqryW1gIWp7eUyzkof4rUtLoSaXkWIg+Ra8WillbuZj1BSI/ThaXvausgiUVDFri3rl2f3ne4b+vOXd86/8qWvu6VaNhBUYfJUswWLFoN0LbO3j6uClQ8zlM2VXEYnZuxdKkWZvYg+oM7i1YwTocnEY73tPdmk3kKXtv9XYN7+8+/coHih+D6is/tBe+Nw+fxuIj+0ogGWMrc/DJxNxhA+0fGb7v9JB7hxPQcGxKOxnKDDqPBsS1DO3YN/8M3vk6zl3vvujUZ2dDSmyMZmb/+Wnt7Zz1LujNzdm4xGNrcOjawula7OLkwMNBXUgcv3ZimZxl6sZeeGW73nXfeGYsnJ2dmkRfBQAjUPc2lCUEzMtI4vCozRWQY9ijOji4QKSPWHZtJUkWyOV5XwGxNZRuSYJLdAHspBjZBOkrwCFcHw1GDxQpe3WxlhTvZCJlc+vqNiUA4RI+d9/3CLxCC5sL+02/8WjQchnuEdFAuncvGJFOIF2XwmMCsUfRGdS/hes9YT0HqR+Inbr394YcfJh/00Fe+esutx1LJ5A9+8IP/99N/Mrpr91/+5V9SN/lLv/1b3/32P6BFuH7c+obRxHr+zd/8TVb7c889d98D9wVmlq6dv0wdCyKXK8UO5pPFYoUwF6uRO5L9BepCbF+hCWPH3tyQ/7F/WgLtnx8DXxy/DieKaJVwbBg4I+4XpwaigYOEhFCuBBWrCDrZ8NoaWS4qW20WfrMykWC4X4rdgDRTHsrcECtk5gxWI/ObS2RpfIVvR2WB+HD4bQ0K6swo82qxGgouZ9PZ0bHtSPjoUvSS03by5F1vetNbOto6TUbLgT37+nsHXnj6+R9874ebUxsNynyIWFWER49EbCoLGEplBbsNRjmWJGNR9mLTWAHLEKbOIdmIEmYJ3JrMGhv1FAVwc3rVSJ9q/37V8eMdu3ZjCoftNty7GLJZUXDEGuEbEvQrheAmoxNRAu9RTUPhCAEWVCoRGYbBqTYgN5yqsj4vPW0SIK44irqWwemm/AwbBYsTNHCurMmXjSvhkq+rr27sW0/lp+ZL2byFNHAyV0d8OxxulkEqk4tFotXNQAZXgXQGPjtLEf3K8pYIKlkRUb2sF3FsZfGLTpIh5pc4TBoksmwFTCScWyH8w9jXmi3a5PomxWGezl5KSBCDTG46DUDHQGYX8A2pYeo4PHR1BQINRt8kZdy5NJAjIQTEKCC4D49CMkq+Eqy6dIiR6WOJIv+bdR39BzJZFLA2FqsHNqvnXlnctrXPbusjqF1voFcw3tBYqGH0JV6kkt1lD7+ubpUnovPkwZ0oT+SWlIfoQhkAKSmW3c+3RRnLl8Qy53v8Ke/wm60i31KUqIzOTcdXKo6UHyED40nLm2QQREkIEEOULk+QzqhbFJkcDjOB3ciJGUP+lLuGq1kHaRzbRR44uAw9LDTghMQ91kHVCKIK6SS2huB5+SLzBhaainB1WVo20EvNRjTOxrmJu6ICSUvY7YadO8eXF5eI4nCPMF9BQ8Meok0Cti13yHPl+jEpMD4YCwQE+QZp8Cc+tgyn2GDcP9hroA1sp0w83SivQogW2wwA2S+Xcjadyuuwmw1NsrpOb7dWY0a/0soNE430KGY+XmBfTzd0/K+8/FI8ljObhGzGQ4WPwQj1IaDKSjpD1b0sKyOFufRsV5165SyFlmOj2+LRRCRCt12Cmt5YOMBiYAhw4xuUvOeLVgcIj9L01JR0ZOLRacQsiEem6Q1K3bLRIATu3ALhMjHYDDRBq6ezYCy5a8KHarOtTBMGpCfqQPhiLK79B455u0fgcAgkksa1QHtHNz3YY4GYD4Rxz/DGyup9D76VQpmXXngxEgrZPX59I0VVPvBziodBbhKfQBMzYGTHe4b6vvfww9Qgmi0uzAeK7U6evOO16zOcl6BcEgVgsmQh88/kHP52ylnc7R0PvuVtGPivnnopX9M4/L25SnVsz+GJqzc+/ksfe+Knj5fTGbPVEg1FPX7ces9rVy4dPnYU2jqLv2N4ePDEiTv/15/+cSoR8TqttMhdXobPOYVjanObgsHQmz/8K089/tQLL57+uQ984J09A88+/fi1ybktW7dB6Li0vDA1lTjyvj94fzX16I9+EAsu2E2m7mNHbzu8l4ajCW8bi2FtdX1oaOTWg/tINF2ZX94MxAa27rJ41XX9apZC5qamvasPbk44vwAo4TsxG1g8aGKEAnIfS5phYT0R9sA0wTLAEyaFhJxhHymQR8VWFroY1lxrybViRhLnwlEFHY1lCxU+0EzYptKJQl27msuW2CFQfxQKefh+GFhi7FC2QZw+Mz+zsDDHVGzdvo0UfjwaISHi6XLD+UzxWyXLtlEZ2pyVTHr2pYsqF1lsS2BmZfyjv3zittuf+MdH3vOed73z7e/49tf+rmDQ/sM3v+Vxubs7OudmZnlOy7Zf/sVfmqAF4+Sk1+/Dqnj62WewKvizeeCQ32GXOBUwVQK8GjV4d8w+ERxoOMSJ7KoqZDfsaKSMiAnlXmWL/d95sOsZBChMcMXZXZwUynDJeVIur4PLAtAUwlbMR2nuiIio1xztXvp147lSrMJeQ8xzL7gHIg6QLVynCAu8CDGaDDpNDkyG0wHwZ3FmhvdMNkupnENr8kULvBJWG01k8O6gXcWI93X56eoNNSkjpvQLoo+UFgrYE3ffA2vb5vwGktBst1LoyjjDxWv3uOidAPOGwW7ciG4QSlVhSdjttJxUW9zAlSAfUhmrMESbNJQbpod7VGNbVLefHDpx52BbVy5fmKYtCO6N3cMEaBrCzC+UFIBjuH2d2VJDdlIKCIqY0IDk6PG8zGq1Ra1xYa7X83QUBnWSogkpWEiAw8xjWapISIgYiFtlSrp4TpcqmQbH7gkmjNPzlRuzzXDcDjWOVtdmdqvw26EYkGBzMhWLJQiaM+56t7tK8JnhQ8FIpS/LAGcatBeiA/Er+0JeESWEtmDEEX9WAmyk7wD1kNSjwBdzMJdMCQGIyU6rVjAfJr1JAI9YOeEI/jQ4OoI1EJFS3Oj1OfH6CfKhcGvCvkenQeGUJu9LuheuLgoOiXCLSmY7KvRa7EpRwKweAs3E0HVW1fxcjPLuyYnw4ZO9qmSgqco0mhRvicpEk0vUAy+35baLZlW2NDehKE7Fl/+nNa7cnewL3hW9rPwoypLVJT8/8/obWpnXRCHJoCm+rKK6RO+K6uUHLB/CSLxhcaPlXWFQZZCVP1HDchax+bF4+DCjzVZkrHmN6xE130SMEC+VncmHuRcitqxp+hAyNxyGfYLyaFBQwQADkmJwCGUDXK+DHaWOFziABvKzuouQi71cXAQJTXcpNgbVtex0oIn0VuDgFMIA5RKYutBjiRRgQ3KphOU4LJYIv7EYWLIyOtiXICQKEI1iKMIPV4lp4lQwwaWOzw5+XS4e9JB0/xWc1/joGGv75dOnKOTGksTPzGezsF8NK7UcnAWfD/wBQwRVJkkpAq3gCip0yCE3iyXBnRgt7s5uQI+lRn1hfi6AAUGcGp9XpZXmHzcbpoE/oLWQWC0koXP50vCIzm7zYGwykA4nVouOckMooSDXtlqo5FP52jqz6QR2B+NfKEKxwLaink/4bKAaJglsbfP7errJj22Eg/QMYKCIjBEuI4O+NLMA9wINvCixDSwvd/b0UvwDLLmSKhAxQLox+JyRCYIxsaUJzA7L3t17YmkKMcpWo3l9eYmKqT379p6/eOm2u+789tce6utoJ6pZLJR6O7uSqRyAMn1HZz4QcPk7bd6ObAqju3b/7kO+zoGmwXjo6PEXnnl2sKNjZm6Wpii0HHB6vS8++/zR48fdNgecIfNzN+69/+7nnvzp6JYtayvLxnwBCBVM7lTj79l76Mt/+de33nkvydGvPfT1vr6e47edfFFisNq3/uJHfvTFv4Kgef3FZzZWFvfsHGcThjbWL55/1et2d7a5MZb9Hn/Fa0+GNteMhldnlxMlWqHYXjp/Gd7jg0fviMXCq2uLgWgUo5BrYzqIkqTSMVx7NgPOIiODAiY2gNphwQkaS0usXjH4Zc2zZGTxCwCBPwlgSR9n1pRYvMy1FMgZtZQwSvlEs0k0TXhPrXC11m1eIR/mywIm1OkKpYLkdLKZs2fPgOsmy475NTk9hfP3/o985PkXnu3v6rvw01edXbCENvPIoHha47Q23BoVIRvWh171xOOP/9f/+l+feOKJz372s3soodbrUQPXrlwthOI7IAkb3/rEt37Uv2fsO9/5Dq7zpz/9aWqvf+d3fmc9sHn+/HmSKPDRFCpxmHPoloNFxuIsIGcJ69IXRxLDYo5LiIuMNuEuNvbNlNk/yaV/9zNln/4r30a/4oaSVeXB4PMJTq144eLlIWZgqGSOANlikiP7yfruP3ig3d+GtSE6kmoirY68FQADERYiv8QkQigr3lsTxghqrzKJjKqZgRNGkD7pnMllrOFpZwsZjSZht5KVsTssUiVfUyUzyZmpWYfNmSRQr0R+KZbt6Oga6O7rHRwa2Dq6MjOPRQ6eS2QwZcPJjLZE+CfrandiTesJ5toc/X1DCbO3lC5lwjGvz6Kv54NLKTTR2JjqVz96aLBP6++ot/mSmuaGxZABAgzwBeddCqu0dGuAklbgNSxOvEPxuhBzrEb6HsK1Afi0YazXIcsjVVYqJDPZSKqSTdIMWa8uYTUV6np8JbRvsWqM5zWhhDac0scLVlN73/mJ4LVpaEhcGlNbDZsO58Jlr+ZBywPaShN4E90g5wbihZ6lpx+iVZ4RcCTYSWCERVGuFjDMZJxZ+vguPJDpUnJNUxOsRAcINAjkKa9mj2hMDgaK7lJOuyMcjpYLIcI5OHWkESg/AaRJTaLJSDcjI5FBXKU8IJp0FkishRCZSp2lopuQZDEL2BjkCnX55BxoDCw2AZtKJD+JCHajsBTgk2lj0WosUn/twsKWMY8JFx4yUHFMxUFn5Qgmq/WQjcuNcYTXw84/81xZQ+hRWUn8lm0uuVD5YVJEcWKRiP/5hvN6U3FyIt7lG3wVpBW2u9QZ8a2WMpZvyQfkCHi6bzxXvtJaunI2LlXyqgI0kT8VN5QAkcRsCNeQiNUYiQcraV5uglphdCzGaEVRIVLhj9biw7yMNGMU2/1AdeTGUfTYPog5fAu2QV9PT9ZH6UUFok8GkuVG/pVQUqtc7OYZZZBkqDggpHOyvTgKWpnoOLBTJRIjxhhcKvTWhg+R4nlpz0dYkQvQ0tzEDJC6WUVl4nMXqwAcTJVmWKM1d7R3AsFKNtKxcKzqcHBSKDiSCmMwuFE2PJY09hzhO5STMC9gadcFPobzxGVwuj2H9mNdl1OZ3t7ubCQMpQ4uI6NFfEmakmG76ImxZM2WBOhlhBotwBbm5vU66TXB6qKihainw2Eow7pTLrNAgX9729vTuTTOmBgZMF+WazY8FBCKwpKNOQJIrbqyOBcrzZqtJn+bFw86HU9F1gOxaNRmNCcA+mZzzz3+RCgUbIcptas9ElyR5migVpg19jABbNLs9ABVN3p7O6woIqN5YXkN79bpsQ+PjBOFJSBP8R/9Mrbv2bswO0NXg8X5BWrEcpWar6OzEE28euES4OG+ka0zk1OapvHl1y4f2LP329/7/gN330Wrzp6eLtwL8OvtHs/88krPQJ/H656aniAMSHyJlTI8MjI7O4u4QfticTNb23YfxIU9cuQYzuixW0/gQ8/OTidSQlo8Otzzyg++/873vHNtbv7G1Utep7O7rwee7ZeeBUNTnpi4NjI0DIUZCmxwpP/i+cvVcKChM33k478ycfUq2TJ2tNnhPrpt6+o/bizQflXaz9HPnJq03PLSipuk4C2HrlyZRuphFBHzYDfGk2kcDXKiGDey+15ffsgi/mT78UsR7SxL0cIsTjQrsT5qVTMEcs24Uk0gXmNDgxSJQb7PeE5NzjDFuUymWig6gNmUihtzM9a9yH1731D/jZnJe+88yYD4/O3/888+/bHALyxcW0TXCpKITJo6T/8vkQVsP62K1siPP/4YzsQ73vb2C5cvqlJlXae+kMtYfI7Ll14bHR6x9LgAVON/vPlND5w4efLDH/75H/zge4S68LPPPfZciB2Wo6U5QH/YeYU9Qc4CsIdsPW27LBTZqCmYlhJyuXE5qyzl/8sPhpkFgM3Nb2VMMWuIRsgtS0gZWYSDK0gMLWxVBpsN64AybBDghC6E7o3dhVxSk8RBuIkKF8mJZyy+EZFq4bRxeRwknbJ0s+BmadZLxA/mgBQCLUfSxmp3UQymqau9DpeOZVMsTd2YphaP4lQweUvLayPDW69PTkFZR6u0lYVlFR2+6oCAqPtAzmmJ4UIOFwtHamlKR1T9vWNvvu+tNdDHhcLijUtOU8FtzYbWzm8Zsjxw9+52d9nnQfuEDfqSwVhkZgG3JmJ1pUeThsAt+V9Jc1MJBZhTVaA9FzU+lDiTfdXW9WwKCoClbhN0R0lXgdAKlHI1Z9CSOi5B2FasGfM1M8i/eM4QTVvDKUsobUkVHKvPJAIJQ7bSbbC22ewe+nwRZaRzRyK4VsgmuGWGC/NcVK+UlqZpWMEQMR2CuW2pBFQfgUmAzcpmYHpEMAvKSVYIYgphgqSE/JB6Y8QfdG+UsY6MjMKXEI0l2VAIgYuvvgI2oqujjX46sB0QExUm/mY5R6kDCNVS0WzUOGwIKWOWyi8Ibko5PCiEOAUdOIaoNNG+ohFlotmTOl4DvWoz26DlINMZiVWh2Ls+sXr4FicUckDXKH9mlFkMLDNZ0yw3bkz2NkJdfHi5HW5bjsYzwr+y8uRvHpLNVbSmokRvPlde4QJ+9i2UcEvRoo5EAYtalZOI1GBdoreUPwVKzYqWz/AFXlR+lA+Lma+clne5NphsuX0micuTi6cGqA7Kk358UG0wE3L1/EDAzLvsE1obEHGWa5YrkcwRX+RA8I7yYRlRTB1JxmtQG9g4K3SOJMlar1MjQeUJT7BN+R5VQOSj0cdylRj9WqYSsQHai9JyJInsMak+l42JLqM4CRMFdnR67gpogJNzPVygtCysELOBbVxDJhUCx3SKqgr6XaZfC1xob++gIwRRu0Q6xanx2/3+NtoN4YhDks5aBHKMXFOiAYCS6S1Pz3g7rXjs7jYqpzLlysLsJLgdsmTETyhYBGdcyKSpQsZ6S2UJYGLlULYbhX+HdrPSNsrmqDfz7ORbDh+nCfZzz78ETqKnx0UBGcAyRAlMYoAWYMXKF4koqGxWom10bwUVRndkzDspBc6pA/0DOzLByM7t28+eerWUyXX7OuOhqESRGs1EtHbrW96ydvUSazoQ2iT/DOWy3eASKcG40Z1JBwMf/J0crUQOknL+qzcm6fTy4Y++9/rkAlFildn5+b//GsoH8ltnW7stGr08MQn2Z25pyel0Ewnyd3QXS+XNcJg0865DR5bmF+iUGEzEQND95IlHO1zOK1cvUJxHb0ciCBTLj24ZhstlM7hBg0LmHe7JQ0duGd8yevbMKwRj0Xu4C9BKJHLVW9/xc4/96NGT9w1Zh4Yjp8+4XS5m/ZFHfjzS437m4R8DRHo1nqDT7a6d2+NUtYpZUgZwR7p+36GDa2sbmXLR2eYJZfN7Dp6g9GRodJzbXFua3whFZ2amQJEDFUxnob7Vn1u/4He70GFrK6sgbPGiTBYHM56FukiF6i0h2BltVoWy0gQcJNgDNiArEqWsbFVZ5zzY1xIv5TNqzH9Q5VAjEQBEJSOUVzcD9F0oGSvbt28/ceIEceZvffNbmaWQym3UOe1Ly4v79u+FhfuRRx7BG75w8bVbjh1FYJFZ3HNs9y9//FeffPaZp59/oZrPNwFF0zkgS7G3AEe+9FdfNrkkh3LXXXc9/fQTXCobDVwVeobKYDxagAOC1VKrl5eWPvWpT4Hg+8KX/vbS06ctw+35VMZYbthhZSoUMinClbTZwMuktIE6BeKmdm6FXYnW4d7FDZUT/t994HYj5tjNNFXhTIqVSDgZ+gcCRbLZ2fSYLAS8ZFI0KnBHl65cNguKUmCcJAyQUYw/5jkTpDzkgkXAyb9IPTW6qqjJCI8e2VuRRBT+5QFP8S7ldNl0JZ2KWCwrGm1wbXnN4ZH+boszC1jAeni2vT5VM3xw/yEYeLJ5xjXFxra3uVjc+XgeTY5MpDNSMV2x+mG4VVfV6oVrN57Qfs9ls73v3Q+MdQ1X84tDvUaH6YjDnnXbwjpV1GDCK0saTUA6wVLQHo3K7YZeZ0V04dG1BDMmHVJBRVVCjXgVDrIk35pldT1dLsTK2XStmKUOCoogXpd9jVyFvS5T1KVqjkTBDtY4ljUlsq5Y3pHIOjJlR7pkMNnajDahPCKf4fO7wpHAzPVr2jLgfZyzBphVGTEuAb42bByJ7SPhRZzyBImKyUuxhoayKBlp3hMNgsIGGkvEji1G6iYXllo4c1t7V1cXxZMcb2FhiQWVTqYg0gWCY7d1IuGRTFD8NuDIJGtImqGYzqRZsTWW7tjIMAwnFMfRThH6SWoUihSN0M0YkIJopgakTIr+VNRxs6Hz+K0YK8CkGYV4Evdc5fXbz52bGtu6r73bm0qvAT/CsoSsw2JDUynBdDEtbz5YMfzBFgJFIO4tP/ypPMQblQXEHpdxkSAR61ScST6ADFU+jr4WeIJcGiKCb+Ag4t1iEkr5r0CueFWUP08kdCxvybsSaZAQtNT5tAZUEu3oY1GcYvHg7SOPULFIHTgwKAjgvpVgA1YOIMCyyazy+510Si3msyZqcPN5MqrwnTIWzJmD6t1CKZtt0rMDvnCOQrUHs0ztUJ5Av1CoUaTUkJZEGkMBWzEvBixoaqDS3A63yJe0hHTZWtwtxpjctVweNyVxP3g3mrS1pz6kSXcmotTcOxmQPCEYQx1SN8CERI9FTTdBXgjbGQsvAYeqHhC0iV734BQIhlPlzIN3yfmTxkNRy3zXG3SYwIHGEDfjK5TpxqPHR7TaLKR+oJXr76bIxZEts7BoLUISLbe+sWo3Wbg6KNuocmIas5ka5Ed46vFoHJghJZ42u5t606np+Y2NoIkeJd2GfD7L5NLwdWxsjD4/kzdmgXfBrlBrFCjt1puteM6Ewkip2M1GBKLTYQ+uLfpdfvrM7dk6+vJzL9p7esYO7p2dmaGObqBnIDM3Db3OLQf3rW54sZJT8YDVYQmuryFYKZYgGdnv9bHWKZsnnH5jasbj8UHlSIFKPpu/8OrZWK5E6V84lUHwEa7Yd+DwxuoaY3LL8VuXFxd/8thje/Zsury+y1euwfr29g9+6IXHHj1z6kVstV17d9TyqXohPTo0cOb0K/liwuvqRb7gsXXmeylFu3L9EpYNoWy85Hq5MjiM57qJUcaWO3HPPV/9u2/FvvEtT1fflYuXjw+PjG0ZvXHjKsrprQ/cPX3hpdkbl0+/cIqkFi7PubOvLSzMD48MluspCkvGx8fROrY2P9LYoVaP9Y/tuu/nfvvXP+X3eYLBzQ6/C9uIFYRGxGYkVowunJ+ncXKegujdu/ZcvHStt7ff6fYxAoQ9NjaDMHWy0mTREBkFMMmWEatYHq1dSZ4SKw6hzyIna8W943+Td0E0Hzy8f2p2logoNg0x/6H+ocvnzln9Hm4EI4Pj79i548bLl9hg1FPSMfj8+bNXr15mzYBmn5yd+dBHPnz2/LnNlc27774Tqc6OZvHrXQ6qJqWJB/rAqkOWwd7A64888iOjw4rjlkonQTjjywIn5LyqTE1DMofGlDo1mGqi0Gj097znPQsHFn/61e8Zuhxt3Z612RUMMr1F7k4q/6owQ5gQ5VRZkl8h/Id1IwZIWXD4StALR0ceikXSekqyVZFEysj87C/e/tk/Zd8pjzdebz15408UrHyDPa7sdD5L1I0fbHAug68TdUIyCd8yZ0TW6fLiCpYriVK8gvoSRiAi8XqQWRxHvF7lwRdF2nBy2knZQGUifq2VYko4NjCnjYgaSzlHc1JDIVvBvlmYX0Im4eMlgxWry6TKUAhFHMcYWVlDaX/+819o93dQUkRRLMS20NRoMfBpqJIsMtjVdA61Vo1XSGlSIgRpwcrkeZE3xedP3t6/f5evy1exGtK0xTIZOCN+qlAmILS0GgN1rrRgxeJAiMgIKEAEpAcbFlprOsAxyuKCVmEwraTChXS8UCvWDXUdMCqYn9VaEw5pXmBWTfLRmYozVR9IFjyRVG09XA3EdJmCvqqxqwxeT3tf/9AW2PGbamiIojOztEG8TqG3uqoXT7sV1GWls1kIhQu4R8gNQcHBsiqKgRXD+BK4k6ZVBIHRWBKW4IF9QDKQwAnIF7XDARbF629jslC6BH4Qbsw10TukAXBot8cJQRBq1eiAZL5B0DlTTOM+ULji87sJBNIijkUOcJecfz6T4pu8KlzZnFRRfsyroI2ZXHGFqVvDQjZIOAS7gp5hWHJ0X19bhQon5vRCetZRKi3DV2CkuqRYVYwwJZiLEud+UHFyZ2i8f1q1//SMRaQ88PWRBQwC6kf5UUwB0cjK64rdonxQ/mS+Xn+OYhVgAtMJ9gonWDRuK/iMZ8ibopxktXKA131f5U/xaXErRf2j+JgTKdDnH3lToq/sD24EWkcF+IX5yfQB6tWC6ecazcCtQDMLnrkJF6MIMirjMG/wcdlYckyMO7kwSgwkl4vS5htAfSUqjvokOcoflNsL6ymnZMoRoHpYVhBCwmQmFhB7DfOMd0iLUDOnxffGamDacdqhFiGDrKk57KT/cfpIZJNLIdFlsWstpWoNEUkwgxg1YAUlUq3YHgwGNgd7nCC7cFJaCYeTOrbqMdCyRK3peQB8g7ATaXQQrao4iZz2iRuLm+sbXoeHWnCqDmApgrQINC13J9uGNqXQcpKcbqjb/e0h+IpiwJScgK5BR8fi1JLTTxDCE/jUEj20ANJpcZdXV5dhQgX1pdfn1eQSERdVbc2I2VQDC03yGZXStmenKp46CWC0UKHjOrY91avQplpMjmy2cO3qRWLUmIoWswmqRbwxqE6iyUybzz8yuu3xxx7ZMjrg1NtZDuubGyarPbQZ0hgswdU1SubpwttVb87OzsNXU6xUp+cXujrbqRw4efLkmx4wf/3r38xcuwa7dalc+fQf/A+StS6vh9E4uGtrvZB69aWnVjaqLq91aWXN6HTs3bsnUyrjax78uXdMPPnjzfWV9cD64PDOZBbSiLjgz41mUhKPfu/7R285shnPEJii8dHCyy/Thf75wObjjz3W7rKkN5ZdVheBBIvZEY1FA8HwgUP7x8dHHQP9Z556UmO2RHN5PzX4Lqe/3ghE0xszU7cdOwqBQJvXwWolYv3E44/QEJANvWvHVqIIvd3dMBQSnOACyP5GaeWYpfdoGLwrO4VafxTNG/tR9qWyR5QNKDAllhiTy+tiILIS6bVGKWO5QdwwEgxho61uhuxu12vnXvvkJz9FrerkxLVnn32WRfXggw9C9XzDfkmVreC3qO0GKhrjLPJi8asPPUSk+qGvfe2Jnzzm6fW9cPrlF6gMJhdgt3GpEPnzYZsd/s4MF2O2G9BGK2urLr8Xm1uVInYM5I+aojI6Pm5QwU/Jn9SOshQvnD9PuHN4dAticfzOg+zeXCAuAFJyvdA84O1Y9ATpEECQPlFuTuNLzshOAonGrSPcMN1bo9ESFvzm8cb4iEj4dz3eOIiINWQLEoYErwiblihSiIV4A/0jnC8VDQQ6YGg9HtgUWTP4nEAnRG/jF9VgtavI1UqsTjLHXFErVkYICfAD9G12H+axVQuMQ2B0cOSqkol8x4AztJrWWMEaQudQq+eqzh43lfQCMKWUItnQONS+rl58NUaDCHMVI6BMORFCShBR5WwdzDNSkgpcOmFTGsLfCBB4uwF2vOXt7qFB3Y5tmr6ulMNc0DXTBm1JGL6V4cLj4lFXmZoN8gtGVpyINowuMJJoQAElGBt1ACt6AoBwMCdCOWqZDDWjrqKtFYQWye/xZIpVrPN8zdjQ+EsNWyzX3Iyb4tV2ws7hGDSv3KVLZ/ebzX613kZyOhoLzS9eTyaDEETD+NvIx8GyEOWEPUdcQH6Jx0XIE+eySZGDgrqiYRE0ssrloiaApxDPRD9LFSfaWIj0WeRwQWtM5v7xcbfXj/qk/SsmPkoXJm0MFVYUULi2znbYduF4M1pMA30dC/NTMIESgRBuWyAUNpxjqLeQOpBEUVEjpUcSuBMweZPUIh65rBlRRDcfTCXPYXVDo2AvgO/R19EcjWY6XVxZbV64sNjWMT68paOQj1NDbLJpijDOMK/KzfDNm1tbUb1yh/yv6M+bh1f+lFeYFH4k7n3TQ255dWhPdhmXww/P8P/wLgmRyfDJE3HCULfYDQrwCkBZywlWdLCiifkAP8qZ5Zyib7km5QK5JZEyykVxn6KF5TmnYGokPGe1AHsz1htlTocPi8YhSMeaJHICVbIBwDEUW9W83YavjIUquoiB0kJ4ziQLFZmBiBO2DHElvHJCpEYYDbFpRP+hYkE6QfJSpQyD3cJliP8vgWJ2Fz96CXyQe2dOMMJELTM52GW4qnitggTkOD4/jDZmL+STQEvIL6HPKUDmAY6M0pxSBRuB36JmigUkKbEFwCmcq1YvARVjG2sgOubz9RSNCw00gXNSoeaiCx7FuFgOdPRLJMPd3V440pbmlvDHaWWYTxUYUhJSzAEXK9E1GU95kGXBSWWaEDUYgyxczHaPx0l2F7nJu0VdHouSFzu7ejE787kUewb+vSJdx/JleiHzXSyS6cuXjt1xV+SF54ntjI9uXVxdfvXUaQqoHA57KLSZTMbQxOHgxvHbj9ONIJtK3nnvfWfPnsuVKru37YLoEWKIex548Oyrp8wWCEELBNa407179y6vBwW5rdVT97FjfGsyngKwjlkJQGxudiGXhN+j5/vf/Ydbjx6fnJ6fW17HGw1H4wQG92ztd1o0ofDm1JWzFpEkZX8n1nZ3Q2vlyu998AGNx4t1MDw89uorL48Mbfn21/7++OGjHV2d9z/4YGBjA4Zeo9EOq9w973zX+RdP79m+PZKMg1jv7mifn526EljrsOqhMAEhu2379s5ybzgaIWJ8fWq+ePV6e0fHeihKsYWH5mSdHS89/8Luvbesra2M9nadD6z6vK69e3f179xawU8pZWlAOTs3g6kxONh/Y2KKENnA4PCli1eJttagySxVKdlg8CEsFbtVi9CRhFEryyshN2X981tcRoxCcU+JkQNMEdMH4lVo3BbnN3GtamVoWKjdU9HV+N1vf9fk1cvUF/navC88//yv/drHV5ZhCF6wWs2YApHNDb0FajXr5TNnzB7Pqy+9TNzGZLVmIrkhchU7tx85cuT06dOTV2+sXJ0FPMNC11opSKc1BvaSqcBpwxl9l431SaNUuivikIhFX1HZaZFlt0fjMcJObp8XhpmllRWSK0cO3PJa5BXsPCgh2PrkmGF9IZdZN6sw+DBGsfckuE/jJCxqEXU1wT8oMoo/ZDgYgv/YgyO0Dth6cvN44uiJbJVxbklGrAAjdjNiSGQl8gT8WigQoLgLx4xyWS4fDSEbWnaUXBUWOvEw9KNcoLjUwteM0a82qLbv3pVK0cw6jZMg1jsWK2lKAEJulc1hox0uilPvwRwk1qUrJssakxTbCDY4U7BYC9DMgSwSeSvlgirUKEFhLDBiiwR/HRA8GcvJHMECVVuX6uhtqv23UPPd43TCRlcxmwBXp9TNImx3uBmQ8IoOFsGLW2LGP1dcS2mkKEJWgr9UyztVDWutbITaPZKM5nOlYqpRz1d11KeR9GiSU7OgeosNQ6lpydVc6ZI/WXRFYvqNuGU14YrnADhCW0+5u8dod5K9ox4qEliHHZpMNQaa3D6rBNsfvS9mjEhYVjKrWjSchA7QEICwpCKcCRBnrKUqJBKpRltyAUgxLCECGEa41N0WknrMBXSqLBECPDBjExrkPiulHBwDVPcimo0GdYe/k7KU+cUbYHZJ6xN6oRYMxQwMkFMrWCKzyHUGFktEPDU4YFEfrSlW1JXMrixFZoBfOtITDCztSlAaOiGJtVCLCQ3K1I1Eu3+ps327Qd9VLGyYzHh6ir0ki4nFwnLB3BNvmDsX5YfIllX0+oO/uRMJK/MPAR9Mb6LlMg7Kj+hH3mWAWoPD3/IZRlL0tPyw8CTUrNRSC2ZYFDDbjm9JcJlvyRfFCpPrURbwzVML9I/gG2JIjATGgguQdC+fUfgxOLjEfChAotcgrxNGpnKVa0NdongJIwN1IbhlNuoajhqJSSm+Yj8Q9ERJC+iArwNpk6pfQkpy65icynaXC5P5B4zB4mCJo3cVe5akZYW9gfRRAlVCOipbgLHD2BQsSU1NmojPSJcOvFSNqsNn93ltbqeNJts5nO0SSpOItsA7pREv3zfoszlaMpAmbtBhAZeVxJg48fWqmf6aos9JwpYRAgCXzOQzCC0K0lpHeRmD42nQ10SfShaTiZCQtuh0+UyaugHoK8GDYDdIbaLMKF3k8EUNVCsxgnhIjCnUSwBg8PSp5Y9thOH16OvtIT4MidHy2npXb49E1Ol7wqYgykl/CyAOhFawBfN5n9O3Oje/vr6BqXM+lqB8lqnHQeS+WKZ2+qNFwulslhRvNBREzafSOSBWy5enGPhbjh6bmZgghkQtbL+9iwlF5RfyJbvdodXEyAKmcvmT9z2os9oIzE7OzkFs8slPfvJrX/kyuJ5//Idv0iCs2+eD9hBbZXFh7sCh250u+5Yeb7OU1GvLLretv8sDPxw+RyyRvuPu2zO5GmXSrrr63E9epQpycGCUfWs2555+9tnDe/emk0nSojt37ErlctcuXQxFkhqL/fLVq7v27G7beZd14uKBPTuv1PNtNgmho7AZ/5HRsYNHjpbqZXqzU2bT2aOnu9rg4DBNo0hKsK6pIqNDY9uYfbjLv76+Glw2X7twCormzq52yAGES8/MvzWzyTZxfQrxASlHOhxiOyM78CCZdJa+srtbmoadf9Mblt0oi5NdowRf+CXNwUDaIbQwI+FPlb1UyKhcPlM6U2rvaD/79Gk6DH7y//N7/+sPPq32yTaH7SERi5Q3KMvMaxxSHqpm3dHwp68PDCqpd4qZsAXYLKubG9inQC5oZc0kUmeqKtV1LuSSOhvPq8xqrcXIntb3OKvhdNWU09lpxIkpQO93A806uDVuBzAhOFIQ+BwHSXZo/wGXw9Y70BsOBEtJmqgoIFG+hr9nhHwRdGSKc0Fxz63iRuIXs6hakqGlMmUXcxtKUEqu6t/ykOFryU1F1rUOePNFRc6Im4FsVD6mvCDPtdidCFgzXWFMGgtswEKLCNaWShwQKmhirGZGHknNbuLzDCamA8cgSwXCgKZSdX2zp6/7+Inbz529MDM9DchbMJDE04w1eFq6eroYGYfTS6CIXZ5J0+QLsVYju6WxSYbB19EOMR5Z8wgCtlyiEwHzrzY26SYMoQ2hPYtGB28PTRSAUd52SHX7ndbd+zs6OjUOZwFAkpaWRLxH8lYcD0XB3hwGCzVQpKTRIAQYkRIARRF9alzehqlS1tUK6lKhCfd5KJDGjtDW4fUz45oDh5Sh0xjyzUa6RAdfc7rsL6mHYnnn9Hr++nyuabI1dC6jw2C20XsUvslSJhXO5VKFTAKfUsxDrhmQCF0AgIlhfknNN8dkbiTFid5AHCOBkU2MJx6c+HZMjGhFQfxDMIuABk0ruUxKSFywvHghF2IvkGfB2cVzYR9VinmWit1uJYDMkjEZ1HRQpcA3FgtUy3mzSUsjHOxAvRaUC+4IlE9IL4nMFPO4JFXiVQTkxZpCYyEBYcRWmMtkTQhAWBRTa5FINyS5ckplIQXlgyQBMBrqtVSycfm10IED20bH+mCkg9mH1ockXVgZYqXyYDPLncuxeAEXjIdyYFmhrDMUgxhKhLxRtIh8ghTKj/InHxEZoeja17WyKGn5JLq2pXoV91eA5URlmWL5sAQYsBg5J6fi6HIq2YMEe/hHNoCCi2xpWv4QucI1cpWthSNBCpp24/VyfTBO6ORlES6izjme8pBho5xP17Sa9SWTELfKiBB/JvIrUGohoOaGmQBaWLKTSalxa2grjiOgAriAZZ1JoQ7TKXFq4a0Wu42YM9aShEswzeC0QkWq6Q+GU46Swp6vkozRmwXBRNdBE7dYF84UA1411hdROTFeIXm2S65Ggc5TQsLSoVkYEDAWQqUIhaSqasdhRmcjAwskkKhdBidZESo4gCruXgoLHBar2TA5eS0ejmAckSUd6h1kCawub8DQBLSHiniGFxGKuMf7RZBJfImsEGRRzQqpz3avP5cvr4ZDVgpCSKwwvHxAraaJ0/Fbbz90+Mj/+OM/A34AXNXrMjEgUhqayzPCHrU1EaV+IufxWziR1U5lsi1dKOzZs0cOzkebqtFt2+hdEQhH0EmAlqUuYniUthRPPvM8gNsnf/oI5FmJVBZ8MjtlbOsW4iZjY1vTF69BpBMPRbYcGnz1J48PbhklcQKE54//6A//x3//bzSogJ95eXGG1o9eX+++W267dH1u1467a6XkUHfvjYlX52fnhvuOJdOJcjQ2OLLdbHMki4VYJLO2AR+QjbbLwPXWlpfpTDQzPTtKEHlkOLi5TlIccIivvZvAxaFDhzxB942Jq/lMdGTL4NCRQ8Qv8umkzWlmk1udNCIu6UqgqW3060YkjO7ctjQzQw4YRD0M2F0+31Jq2WVX60qZLrclF6W/ejETCVp16s3lJQAKFAIPDA3+9PEnaMOA2gZ+dfXKDSxJGtogrKHgIPNJ9I/JIkDBCubB4lYestplN2CusYFY8disEpttmkCSCx4Bdtoy8TTAB7lkidhfaDPcN9j7lS98+Xf/++/e9763PPW9n3Rs6Z68cePOEycfSfwQbIkcRirr6pVwCi6RfQcOYLU9940f6rsw0sypeJLPnDl3lpQYAJWu7u4AYHX2BjxZHvPg6EgUkRLc3L7viPGA4fLliyxdbJtqoYQIIyjEfqsUGhYLllIV1YTaposVt5+gDIlefdi1SCH8E+5ItABaCT8InAp9LqTpNAKUfcdD/HxgrtynottkX7ME2VX/Ru3bGkR+cxyRJT/zRLkUxaFhRBXxpwjBptFqLueLUp+rktZSqXSaTlZejweuZSxj7oIiQ6oUuAUGEvCLAJLxhiUmKR3KRRAqstDisPeNDHUN9OiuXBVphuSQYF2tp68rk09m8zm4l44cOQqX0PRrN0weBwFuIgogDbVmjc/t/+D7P0DEYmlxnvQW4pvTEXLjB7mnpE5V6WLN61Jt3aE6cIvmlmNtW8bBkmdLpaSVVDw7XkS74FSJpAlzLj5D3Sw9i5gAPGhkqWhfbl8qF0lCN2vGSkGdB2mVhheZutimuY7HT0UGspWOrtQX6bFrU0VNLN3sGjzo9g2vz2VfOLO+uB7X6rsd7Xvy9HCxua0QTKur+SxZiFAmHaNVnNYI8R8ZPXwVuqNVsR/kpPhXtKfhrmRiFXeN61CmmMlHqoqXTFwEuiWhnuamCTlaC1k6yZSASXd10zrczUGorltbWxPecQLd0TBNivyil70YRkCdhZ+hUSFJTO8yyDT8PndfTzulvYJsRNtjvLKVZIgwZ7XZHLF6AaXyXdHjxby6IaYADxko/pd7YKJF6fBgLZv4g3wEMKO6rkZAEnBQrWmqldWbG7mFmVRfTz+eSaVQhHsYR47dwD3KvLD65Rf/yaJsnYDfskDlNXkoUkAGRLxbiRvLc35ef5eVh7ZvvXjTPxY9J1xX8uGf/ZGsiPT9Vbxq+QoChuMoP9yScmeKYlZukMUg18FmE53Mu5gjoiDFYJOsAPuSri2oFREKELhAhEt4BLYsxQ5FgVKuRlKXwiIylFRFEuzFUZD7wlEmYwA3OuMkZ5FkMLfDcaQjsJhZrZJAxoFIINaZ1PhyIxSKkHegKL1177zOUGGu14mjce5aHfUIHTWkcaRjnQ5yWyxtnB5kKZXt1OkbOLtUZomOEmYoxAzaF93OfcnqY7YJvgmBK1Rz8DxJryUoLKhMIFMIJIzGDw2dwenx04hQ5baVVhdWl1cAY20fG48Eo/09Iwa9GUBHMJSQKJEsW7EPRFRBcoJBRNgKIxcO22oNfgFCCFh1QuyCv842aKr6+3rCiRSR8jNnzjLug8OU3sIjTR1wpVbKFc1Y6sKwuji7APUp9u3q8pq3swsti3cu+UKPLw0t5uYmRDj7u3rjkSClEmgUIm2xVJZOm063iXTPWiAIKStJWAISsB5jvGIU4i8BtaDxSy5XiQQ2My88f/fJOyk9nplfhPl5fXn+1uPHXnvlhW2jwyCHD+7bs/fgrelik+68+MG9TtszTzxVKcVHh4fgX4T5we3rnJnf2FFrXrk2dfsdD9Bygwpm+lzkM5GJqxPdvQO0EXyG/rUVqRTc3Fjfum0nhorVoTvzyksf+I1PdnT4Tp1+cXSk9/oTPwkFN8Bmj2zdwqR7OtqjodDc/DzpTJSlCF+LdWN5dW11pcPhIt0EonTn8FA5m08FV2BZ0dVrYODuP3lHPBl76ZUzH3j/L3zt698slEpWKnYt+dW1DbfbMzi0BZnMYkABY7xDp8aCZIUQ+2A9MGVsBtkH8pAdwQKV0LQYiSw6aYjEDgbOKowOVHsncnSLTqVogYAhaUiRdS/mfkwnKDh/zSq8z8W52Xe9461f/tsv/tlnP4NYL5LlIwLhaK5em2bcTtx598qtiwtXrxS0qqFtW5euXd92+BAqR9PXlCbBJBgxQmsqu9FIyoDN9sxLL9AF+xO//GuWf7Scf+VV7gI3EaAQwWf4Namq4h7YVUD9yWUEAsEzp09TCJcKxUnzDxwZBHe2emNFtIi0ISIIBGtSswopmAA55IYpVeJmaZrC89YG4QkTwW8enK715J/9fuMD//L11kjygTeeKJ9pDa+ihHhK1FkOobja/NlsWj1Yu9ZEMIalRW8Yib4KtXsckCDXSYiSzyAeOSZTQFyO72O7gEBk6zUIQJdApNVmlxYiibi4cSar7HDw1KChKKAtFH06/WYwcPKuO0vl+vLkDOBqkb81FUBdSrUiITq45OGRRYsSfUO8Yc2zZ5l5FBcpx4Fh1W13WO6+d9vAIDt0Q6NaJSTX7oFSI0fMCeLapgYFIXYbnpQgnNQOfkO5Ifzi8iO3T7hd7KCaoVbUlNLVXKJYSOEu4qTgHiIiEZJaQniFmoo4dKHuL+k8Tbv/xrpjeTMzt5qPZdutXn9NbU1Xmja/lbalVXK4tLOPBMt0LAB1htuCFC2TYGbt6siZ40LBP0kQBRtAXD3xdFv1tYw9PxAqKMBVrpfv4oHj8MiGaFaoUKKFUU+v20WNr4HsLGhW1CQyE/EptyfxaQ+1nOhQ5gaEOZEEiHcgnkRDDw4PUfULpbNMDK6tBjYnlB/CUi5DpCUDgUekPFra5I1t+C+XFq+AAaC/Hg82LIqeFU8BrvjDyUTd41HDPdfXb+sbcHMl8qqi8GTZKdpXnvzMg6PcXIzKi3JU0Q2y+vmNKGUDKn+i7EWPtj7AZ8RJ5kf5MIEBIhV4vS3t2/qKHAfFo3jGvH7zsByN68H8UK6KF1sP1C0mNIpQwbi//qpsOUwmqRUHK6zTsU0rtJM3KNhtFCUEFBgw+HZsTGIIkJ9yHMxps9HGqRlcTsbdsQDEsafqXAqVuR3Rr4oEFLuEm2BoxPJmPsTsQCaIGcKFMSvsMUQer4G/xyqTq4V+kn1A8kWYOuCbIH+AMCU+h/1KCJ22FRXJ9kilO5BqOSbqEE8RIwLOSDYsPBMcFgcTkcsTkmjYUigksBg1JWkh2lelh1xBkNSGBqRvpaVVrNPVpWmv108wOh6JbRkezWdKl1+73t3dG45kWFcIdKqniDJTpsDccIOsEG6BUsJ6BvBNGhOd8cDPIEMN8LitowMnhiCYv7uXpkDDY6PQMly6dD4TD+PLgrGH2kTwqFpaFPtX1zZHt3drTLaNYLhrcPi2k3dRcXvpxg2GKxJPYeNcunItHY/xp8fn7/X3dPb2Xzp3kXq5t7/vXU8+/hO3zdTW0elyIqPj1J6urKxibCwtbwwMjjMgEOLkCnBqqhLR6OzUNOnD5flpaEJo+tTV5h/u7Ykm013bxppza3NrwX179vU6Hb0ex/e//xU6zVvtPV6Xn/xh//AIAQtuY3UzPNw/urm2FN4M7nnXmwcuXjx//rXe7va3vf0dzzz640wivmXLcDgcHHH72LeBxZVv//X/uv3kiQ9+4H0wYTWrxTd/4ldnTr9YqMocpdMJJt5BR1zRCpa+rs6rzzyzNDvb3daWDoV7OjtgBp2/dEng9zqdIAHKpYXpib6hQfIbIAwe/eHDgKtnF5fxV0eGRyduTKezxS1btpBqwk1ggpgdJgOGf4lFC2EcTnBrQ/zTby6D5SpOC6sfMsJS0VLQE9IkGOR1u6TlH/h5AjAQ9Lv9U1OLA6P9L7982tvuUeVV1N21+f3f+NrXb7/teFdHJzkCKDkwATva2nO24uULFwCwvONd7/7M3LS3o+0DP//+0wO9n/zkfybhe+nsedot272OSoEubM1kMHXx0qXt+/aQxiZoB9XGjp07oRQNbwZw5sTAhIEPCphcBSbyeq65NrveNgSMrgca1NRaFHMPQmy6R4PfxuZgnSM4sBsowENbQzzF7qCogNHggRAgCvrG/XP7DBTCiCdvvPh//oSt3PriG0/4LrIHa0WRfCL3+QvLG6HDZciRGzhaNv51tLnpSXvxwoU2D30OMjBQyrt4ZNQpioyQvimtB/JCXmiJM8RIHoar7I3JSbqTweso27taNTkB/JgF6NFssk5eu3Sxb2hk+66dy1emNSYNB+ctxA7l9d/82tchv8GlK+SzML0LLqUpoTiakFJc1D9ifPB9fd7OvM+dRZaY9BVKwEEakfmUsIO08CC0x01gFhBzhgcXMhwTjif1uvTKa2qItxHbJAsG1FNfLWkK6XqeuopsBWgfQXZybMUUnNh2kv/Av+LZWrzIN91V3ci1uWqx5gundIEE/RXUNpPbQAfBeiZbWoOgA/AynWPq2bLknLkZ8FvSpY4mHnDz6EpNmMfIwdFSmnQSq5wpFsmouIWigGUeqPBB9FHghDUihiZEfdD4m2AT9Po7fT4PN0gWCS1PcBiZD8t9Kg39kcbfSb8WF7jlfC6Lm0beMBRYI9c4umWATBANFdKZuACsJMiMYGQWJcYpqp0rqeAUsQqEXoIUHqYtPjErQiB14q8zqaI4RR0KmE7+0hHl4Qv4+7ii/IOwhSabgQ9Fii6PZ2ExTalk/yDNGokpwG4qOkBWL4firvEruVtWjBhCoACxGCQYKZYFn8gAALHpSURBVEtLWUKtVSTZEVlUon35umgRYWNWLkc+xvLloXyGtxQ9LR9G0Yo1efMrij6W0RZ1K9JfeV05iHjicjjlt4yHPJHNJjtB/uPWuVKlEokTw7vJDcvCgcmBG6f+DA+YE4F8FqZQusEju8SnFyOgqpGSBhgeUP/yGUZcuSqxEiQ/TRxPvFvqHxB5oKj4EBuJoeAalWCYfAApybfEApN6NGnbgOugQLIaVmIj9TKRKi0MqkSdOa98iCCwWhIsFBCDxJAkgiRtSCiD1wPSJv43/Q+cVtIP3Ct6HZoZrGvWAgxE3B42NKuC8CJgehB2Jt60e2jG6enoIuEKgLYdPzIRKlfyZMtDRhPuHFCeXKnq9nRQ2gAgGW+9UhI+B+gguCr8Eol9wQ8L2RrU44koCpiSX8oIaIJ0yO1ALxvsnnA00dYzYDLbEOGEp7A9YX2mCaGBRVjXZPMlWpg4vT7IOwFZYCkODown8xpX52jDlkxSyFyc7vV7wGzQ2MSs88FmgUtH4wcqoZOx8BOPPRYJrat8TphrKxAEOW2dYK9vOR6aXKjoHFUqEokcNKqReOzpnz4STkCqqe7o6uodGPQ4rIHVeeiiSQsc2r67FA7RQaxSyTRr+dXlsFuKCzU2swMbJpMuonS7+8cp6YGoamMziZRjOUF0sPrkMwSsrAbN/l3jZMWon14ql+LpZEdPTyQWHtu9S7vaDIXWvv2NrxCfNtPYympYfOnZ6ekJlg4B9khgA5UAdegrLzxPdOvAvn3lfG7fru2h9c19e3bPT02uzM35fV46V5cKeRxKML18zAIRs9czNDr6xb/7e6PVQS+m7bv3EOw/eNBGf8DlhUXyG/jTTBAVwAYT5FA16TytMwKoYfVj2MkOkoyMBDXYoOxBNgg7i/hcsaSiA6bEm+r1LHykeTpp1d1uQyCUCYUzQ0O9SwurviFvJBw2e+imTr1ZGrfjhZdPSZWkw0XGPZ3Jkm79xV/81Yce+trzTz0Nwwlw5+DC4ue++EWO+dDXvnHyttt//JPH4qm0lNwUayPbxrjamdcmZ6Yn1S77tl3bK7Xiu973bpbrQ1/6uyyNd8pV4lDQ+RFTqdOKjq2tU0UCYNeShw4e/b0//sOvfvnL1yYmCBi2zBS5wVKzZERhsIGM1AbqqiRTcX+1xLSp097MBBkcZlAEDRtTec4TwkUiJv6ND5FTyhFe/54IYB4iZbDPlXdF/DDgtabVaydIDAG/S+e6/4F7r169Ont1JhIIcQBKJbgGjoZIaBno3A4yS5g1q0QbsYRQGIS9DHVTHVorjCqXww7SvZzMciqCkUwBm1HMoHzJ6fL++Ec/7ukeoBsASSOOjwGdiaTohEpVvq1nALvWAKsyKTOzipIDv1O1c5v6/nsO33bnSEZ11mwvGbSQMMMKiuYgyKwmkEYZBQZ7k4gcSCu1TaO1aA126hJg+0NLkMZDo4jQxu8Ul8mQSzWLmRoJHBB8eIdk6dRgqhvQRPVDqQE9TK5uy9d9sYprNWIOxvOrYcqHbU2tR2el82sJsUJJT74YBZhHahcfCE+XgDFhXQkD5LI2hztPhkViOEhg1AAByiotViWYjrxrPZhnpL6sb9SQkArJAGJBSBzNQBdtzMr27t5kJg2gupgjDU1wUTQ3fHlVggF9vWBucLoK+TSvg4wpwvJfyA4P9VLXwNqhARpBFqAzWLGAcDktFyKOCHoBuxhQEL3hsxmyBEwv/g/zy3VJ4pJJQSfLhcqky8XKL1HGOrdMGRRzUt0l+ooF3ACT00TkrdPyoqm6ci3g8TcOHPYDI8gXNoB+QbSMrqcDGLg/DT2qJGCJUqL2hiXIjyx29LKMA7tI4iVSrYsyo+Rahk6JLbP+ZE/c1MzoFIHj14UsExIVwglU77ayvxItkWRwS+kKcRdnEPUp9yZ+rdyWnJPXWwqbyxEFrJQegjMgyoMjWm8QewEv7ndDQ4rhhl9GW98CkGSUIyhQQrylatFpdmSwvotFSFFYhU6PA1uhTHetCtlX5cKkpBjQB+YKQDc4TSXDxJWIgibagqaXtgpYRsTWuTgFysgnkI95OimJC0m8nw+j1ZgzTAQ6dxJfttnAlzMy1GFTGQa1LM0/SRWzQliEHFWHOSKAGgRorSjsKZUSIINyLmEWzlt6jiFEa2QpiSuSyS8Va6UMjbFU9AeGd4mqEnrJlqtFV7/LuuewFd/25aeXpy8Qd7cIwZQmEt2E+83id1MSU1ObsT/cTl82FYECzOXywY+GOag3Sl4K30oiNQAIMaWR89LiQtXhUSUyEYOtDZ062D1y4N63qOql5ZVV8tZmG/kP0nsWK9K/QmhfyF4JQoKtigdjZkdHLJTv37uzAaVWKbP0yhN9YyPJpRt7hnu5rkg4+a53f6RMK9B88YnVdeyi0MYiiPUOd1ssmrSZIIAvXZ+cH2laJ5Y361qrw9dRodVXYt0qScF8d5uL6PiRo0eBrgFi6BneTiHvlauX7GV1MryEoOnpdDZqSaB+N+YX3H4/UbPgZqpTTbmW32ywUkAMEPTA3m2XL72az6a72tusRv2WkcFsZM5Yz5fjm8xjW5dfazb6u7vAdKxsLP/Kr33083/914MD3aGNBWrHiPitLlzztnm3DAxkAytK2l4fyWTwuQGFTV6uU7IFEeHG+srk1HUoPqxYCaEgqDqCdYFwgrITJE/voCWZ2WRLdvi7blybfN/PvZ8wwitnXr1y9UpgYxP6KoE6yWbQQItHTQflO8CMk1nSZBI5YaNKrRzbQwxjIuUQZ8quYSHRCwAuTw1VLiRCIHeUrhcqo1tDp4+2LhcpjOXVdZrOlvN5fbOBfx9NJOilU/LV0AmOWnU1GJQsoKaZCSaPHDrMQvjzz3zm4e9+j2wJLur49t1E9p5+5vlHv/M9KRsg0e+35OOJD/zSR9Gdi0tLDzzwwO/9+q9PXr0UzQeruvLzLz8H0Dq6uCnRUdpoIt7ZxVIaI9Y2MsHuc05dnZ+dXD5w6DBLHYTqmTNn9u/dByBrfXmtkavV9NVsLC98TDqjw2HDJN1YDw30dxMurUrBz83sEsRTbCfiO4rvggSR3BkPPqBIEZEk/+pD5NW/9qDgANEjOBoRQSKU3nggeLH9weOAlp+dnnrLmx782+VVNVpJkpUY2QrYClNddDHcWEDNBEbHpKOTmS8pGZN6aO385MyBAwcy1VQ5n5SosFmbSsaketANS6Rlfm4xsRKxuNyxzSg9O0sFymxUSDY1YAC666pNS9cXfW5zdxc97la4923bVPfe673lll5fW7rafNoNGY+2TCcLLbgt1KagbAxas5Te4qE0GoR5cTdNOmiI8KrRvHRWE69Y7haeHVLOhUwFuDUYKXgHsNUBFWH/SKqNJG7TsRD2Z6o91aatWLUEo83ljXIiZa6pXSw1RA2CzkqLh3wlnYuXylmpQ2mQcyWXrISUGQoUPhEGE55/GvdV1QBnKsYZuTkIFxCHDBKlXIwgpgzxPNEpVItILN+CsKpSJI1U8rR3dHVLK3G1OhzZoLwCx7RcTnHxUjmkVQ/196E90A4AAQG9ArOSwiGtFoLJ3aN7/R1eyiw3NldIRTntdogWaGTHc3J+/HBCKPox9ZDqxB4EIFarxsKhMJ0VSkUpPJbKVIGa5ilzR4gazeLWs7qd0GyNSCoNmlKL8Q3fDvgQWxe4TZNSl1SmubaeX17JURlVrTjZibSGhGOfQm6JRXB01jCaW+jQxflT1i97veXUKi+IUmSVM0D4dhgW2Kf85rksfeZRbED5AB+jbhUWf6aO369/hq/f/GHxiLLhW7I8ef6/maI3lz2vSbpXBI7gAxjcluPLc8jpGWG1qsQPE8gqV84vQSO5DWLLTVWeRrilIi6EEioWK4oLREmDW7TYpMMlzi4qR1JtWBiCaeHilCJl4t3KqTEzBfiLJubAspq5T7k2LCSyPCQsEYJUdUvqvomHphjUjLgoVwqiJCsja5fDssKBfGNJS8wJE0pSvICc4cHBYqP6ktAFP/DO1Up52M7wiMkS05mB5LHNpHLZVTTooB805jMMLzot8twBuDd5fUZFVcntdxy85Ra8LCJTlVIe00JA2ySBqTP3tW3ftXf77r0Wh5vYAH3gSSHhyGKU8CDJTHgemchSFZIvJLBG1dZJn2oZQ+pi77jzbuyIV59/PhqNoPA4HjZ9i7DXanW43O17Dh9p6I2ZXGnXrt20QDt/8Xoono/RKklr1RhtVJoS+Fqen6mWi9j/U9cnVlbW6PSC0KK7JzbJlqHeob4eO11cknGgZZls4fylK7eevM/m63S097k7ew4c2B9YX7KZdb2dbTA1slVm5pcatEs+cc/FyUVne384W3S3dTLkh2/Zn0rFiI/ncumdu3cdO3qr2eqi0Nntarvl8K1bt21/6fQrl69fo+r51hMnHT5fIpdPZnIWqx002dpG6Mb0DMiX1WAYd2b3/n09/T3PPffk6Eh/KLDa3+N/24P3Hj64G9hXPBiw6dQek2l1ZnJ58rqDkIAWev3Y/PQUayOTTcP0+dwLzyJEcGuQI4ViCSA6/QVcHl97Rw+EGMlk5rnnXqTBkVFveubJZ1A51CCPjmyhNyWKaGBgALECgBPeUGWtabCfwAqychVf93WdwEJmlbGq2KZKUJZlz5pDN9OiJV+tJzJlTGicEosPCsAUcXiGKJMqZGMlJjocIMqt6hztAuRF8q1nYNDVRuNkEAAgnI3QNYNHg2MOpJt4cvUa2fTf/b3fexePD3+4q28Am42a4IFt448++dNdB3efeuxJrVl9auK8qliNpSOPPfFIZ287CU+RFwgoC9EIgwQ68YIqAi9gn+UzxbXljWvXJohM3HbH7XanEw4UsqpEoe84ecLb7S0mK0ar9BSi4zVIOgKVbrdlfXMTXxA5LdYxMh19y05XZI7Ia5E88pCXRADIQ4boP/SQnSteGplWSoYUZ4DfRINXV5ZA1TI3ksQiiCEf5Nb4kYcCt0A8KF/HQFIeogP0Rlha+Q3lj0QwiMOVwEE2N1cD5Rzdyey4dVSmSh+zdKwUz3Aujozc91gtfqce/8qsLWjrcVVtZfdOy0c/3Pcrv7r7jpM9bT1pg5kGz0FgQwY1lBoVwL8K3YCOzdLQOEs10IAdemu/xd6vMbQ3ms5azVDGxRBRiSbWV/J6kBiJUA1nMhmhYjaTz6UxI3SUcVo7K6quzZh3gjbWgw86Bh9MNMdOX6+evloIZ7wlVWckiVp0UtcJfVC+EM8XEtVSulbONIDS4zcS1MNkEtkrI8QJkQk6LYgc4Au8wRAA0JF1jGilvlG8EtoyUueJWU33Ij2KWlcmfIL1QovvweHR8a1+qAPLNarPaY2QzycL+QwqgNZFyEC/12U2a10OE8XRvE7SF5uCBkfdXT6amKXp4JtNIn4ADuF0kzKmzaBS9UkNl9TDcw2AcrAICaXyhCgRFMCAb1l4bBkkJks0m0nnuR6IjSwOsvgal9/c0dvZPWCyuGHYadIYi9x/pZ4tKkgjVgNrEhwH2j2dqqwsq2Y7Nke3wHztMpnaGzV4iEh9Y3UAsEZui7nG5zWsKll2WBL8I8taAvOiVuUPCbQI3oqHElvGVuRJ60dgz6//SGNB9q9EZbDrFaf55seUg7BhZJvwXKbm9YfoL0Uxs5p5R9Y0iwSrUjSivMaFEEcENkRenciYOAM3l76y+hknlK/SQgAgJQ63cmFiysvdkb6lltBChqJR1lJNV2WJEIZHr6KwxRzgNx/Ec+W49Ddkw7fKrnBaX38wSZxEcHqSyebG+FO+ydUpQTHut1UljNQhli/DhD3FtxkzPsNYi7vMPbBRKjAfQBbdALFCyJyB5QPENrGoxZJWggJ6QbZqaGAIMwuhDQpVXf72bNMaioA/ro2UMh0D7fEobXViIKo6vaRjXfECQHqn0exRN8wet5+dEIiFCgrXiA4SKy0GWZ47lXvC8ZV0NLuCZUbrFxWJcwQciSXcelWnL3b12tlTz4N2gNkI8vMOv589EQ+GjZ72O07e+Q8/eUIYx2zOeK5MI0+H1be8vgIgloM6MC7qFZ/Tb2lW1zYjlYY5cmNqx15ndG01FAqAVmBNk96GmZWRI1dC/1vUVpEGx72Dx0d2lmIZk7aanb3Ax8joAO+FaOn8a9ekApXAQ03lb++kxx/7yDk+WJybyhbyqVxxfe7GieNHeg/uO/uD7xPoO3L0OB7Bk8+8sP/YHb6ugd6RrUsrG65ys2tsr8nRduaFp3V1DV0D7Q7njoO30gFtz8EjyJFMJrd9+9aFuapJrwsszyciYUy9fDo1MjCwvLx47tWzJAD8LtJOOnDT6MWjhw5GUuneO09Ez56jr8PHP/axi2fPry8s7d69h8LNS5eu3HX33TT36x8cevXcOdSMw+NlOUN+FonGl9fXnW4vIKzbb7+dJoakGFkMXDnsMWAG2TU8WC3QmrfMYWWxs2durkWZRGU1smx4sB3wvFAKTp8lHSokahm4t0BB82jv9ZFj5ahkWzGh4N0LbgT6R4dvO3kHZenve9/7Xnnl1MQrF2njtra2/A9rK6p4zTXWllpcV3Val6Ymvv31ry3MLQKYAoYWWF6IzC+8+eff+/M//75f/cTHDP2W3/nEf/qrb3zuV/77r37lG1+mHr2/szuXy8hGohQA0aur0uQH+ggUMAuL3BATDjNotZSHGOT6tpHzF84WiCFS/JuopgdSfr8PBUANPWjXtm47gJpAMNm6Zfw27oUdK0AtvNGWOGCH/4uHqEIZp9dH6l984N/0Ak0z8dUwVEXxo4DDkcnJaXrppLgw0fv//GC47MwjbyBGmRfxHzDJ2WYUvOL+ETuCYqUusR2VBOl1hVpjdTFAITf12JVso2woAc9y+XTVjFlHIFlXcZioP0xRD6xzqzo6VEdvsVCIfnB/f2cX/AdhKhSQmeDnsRXEk0WvSnyesSJkZW1orPWGTWNwqkxtEk+FbkV8LBRaXtiaJSulgh6ikGgQlMXGxbur1rJ6swVITCpTD0XoyuUzGPZ72sYL5ZFL11enp1dSkCNXzfFElkRJe1sPd0usnea7OL78ELIibCamCQ6HPES28lviqAhhEZUtpw5RKZOnjKDoAqj3iO4i8QQfRpc5cYm5UjaZtp3mbd29iKl0NscmYmSFVSMd5V0yyPBnOh02GqLQshfJnEylsUgR4TasP73eagIYSnQbRyAL3R8lmSSfwUIT8K9W6X2pAaqOBSAzJtOGXMQAIIZLZ27p5kSeET8L3xfZWG9qCbc2KK3nK1y7xtQ9NNLd3UPOaGNjA6wl12+SADYiVqE3597Qc2SRJdLdIFOtWlmuzkzFuzrbhkf7KR0hGN7y9vgtK0X5LapPrGvRfTI8cl03dac8kbAzQ8ODNwQ5zGTzOn8rH+PqeS6qV0LQwr9B3Y/yivJ661ByVOWh6DtZwpyNmVFOKv+K9mqpYF5T1JvYAPjdLVdYqbeg2pbhw1SGqkDUKFgtxa+WSK9YoexSxYcGjcbnZHWi/TiwGNfqGmEAQkXoXx2pGlYISlhGQLAUONdIvia6kJCf3IYSXWeNcAUtScfRWUtcpyTjxO0Whcl3CanJW2L3SEBYlhWCCJGqnBphKtwyElSRRVbLSyCRolBS+1By8EW+AfBd+ZaMByPARYOhxuPHe4Cg2o4/qDWO9G0xJQpzS8tzU5n5iVc73VJ3CP84N0P9sEpj7evb0jdIa7nltva2eDhYUdeNtFasafG00TQm2qdIGoDVJgYEih7bQVDbFmj2aIGAiapJJ6Ppy+cvnX+tWaGQvj6yZYQqPhrVsawYwVyhtAjy2deeyOTVRstGIkUrQKj6z1089+u/87sLC1NjfeOnH/laPJLednDf6TPnQB5727rxZlaW5gGw2E1g0UsRQqJQUWrUvo4OjY7Qj6OnZ6gWidZ1OVN7f2jqslXdIKAUCQWNwF+1WsC03QOj0XT27BNPD49tmZyeHBsdKd+Yo5Z6bnl+bPvudChKI6lcKsFo7Nq9l5LcjXB0ZOvuxbVAPFv05Kt9I9tMDlcDzipvl8nbffLI/dlUGI/z/PnzOJlvevD+Z5966ugDD6QWZrePj01culAjqpDPfecfvsXyvfPOO1li2LZGmw5ga5mFDWaGcv1SmdZQqo3NXCbrw0o4f36ovx+ey5Wl1QMHDkYisVOnTtFSoq2tIxyO9HT3kW2l5wegDtYs2WiULrt3aWkFc4QIQa4IetkE3TcnwIBklloP1h2f575kayoP0UUtJaO8orylLDlDY8vYFvU2zdrKWng5rrKK2E8X4QGrUsRNyp/4AtXx4WS8Be9MBje+/7lv/e7v//7o6OiPvv4do4uwi66kLwK3NfU6acAGb9vBvbtGB4a+8Nm/IKqzbecu7PmfPvmTPQe233HnkUceebhrt+9zX/r03v17uns828Z3BJfDkCoYrUaoaOjBRnCbfcWeo5s16p/CXkj62Lcioxvap558MhdJaOg5y6bzqKmoHhoYpFvDtTPXqJr1trf1ms0wdYOWYWtAE430a90+94skYX/whG0lLypxuNY4yMutz/1rv/nMv/6yHI53Wv/zG6nQOofY+Qw2ATW2PLhdWmx1d/a0DiJCSzkev0VQIjoRPa9fAYFC2WAcS90s5BuAl4Jrm5l0gSCYnIfP00XEqEvFoULTuNwOlbWcgKwlhZFZ05cpV7HBf2fAkayqvB7V4cOqO0/2DY8QEaNXG5gj8CXpZoNqLjgiDDRAFTFOuBDtC02o2tpU29VNp4ZOvTq7UuZFUI6YMoIa14UITRo8VzZCwV+lSkMFPGcYObQNmM6C8eQmvb4blCZvaVrGYrlegMNnJs7R+jOboz4H2eho6/QRQKXSgWWKiAMNikNJoaOAkMQLUpw5EXxy7zJ8ItAl+yh3zhWgG1qRSy6FoUYCYf/SnoIiJYXTik8bcGwt1sHhEaYMicFOAeJMGYsywpRQap0O6n5teMsgBIFZUUIKkzBde+nH46JsESiNic5UbKVSKpmCVaRGTZe+RrQP4c+BSPpwTlYSFyYikVigwP2EGVuQWETeORfzzlsw5upNkNgjAQDU2rqHkH7kK9C5G8EEcc1aw4Qzwy0i0OBKBGspXpeyIPDm6dRRQBfhIlPrP3E91t9b6uvv0Rs8MEIRohYxjNbAmkb1tNaTjBjrpmVKcqSbq0VRvS1tKuYJ60dEg5gNYjkoepff4u+yCEnNUn3AlpEfLpsXlSeyOlHkfFs8bQ4gx+dMnJ8nEq5g2oQVhESGMnEMv5xEljk6DcIBoruiazFIoJ4H+akoaS6XJDmfY86ZEgwAvGTWN04BfieqEWuIaxPTQIwNan9F7fBdFgTfZe3yW7K0rF9Rnbjd5MPFDWntK7lKZUgrClxLoUcR/nZkC5fKgyvkgV0r/+DFovQV75hSQEYVnB2zIzTWcpti9rB6KM1V0+wWiSU9WCpiJdQbaBSOKGFvboYZBpkImCOvd3va6bOZLoU7b39wqLu5uDALJT6ZlVQ8jv2BXofIMF+pW92ErDxasy2ZSrm9zkwlDzsLtRBEqClnU0ZbhlguDpXP0mIWGDQV1AvmSrXhcJnlLquFC2dexsnz2+hOn7cboXjVQFxMlQJbmI4OC4vLRS2pTSrcvU1bs6DTXrwxMbEY+FDqF3tge+3q3jm6PTQ7pdLa+sZ2UTFKUGlqdjqXiELwBtuUvlHq7WrHuWdIAfqurQfKqsSYrc095HzqyWff9CufYAjt/f2mySvXpid8nf3M+tjoqMXbYXa1nX71vLejhyprq8P74vPPHT60b9+h27SNSlP9EiEippsKZp4wi1abi1iQu62XkNvc8sb23f7Q2iYNxzzUOXUOXJnbPHD8wEr4VNfQGOFlGgC/+z3veerb34oG1/fs3Do+NOx3WAf6e7/6pb+lnehPH3uMESahn9XChJ9BS7k8bnjJQH2YnM5nf/h9+mPgrE8sLyajkQP7DoL+vXTpEvqVnBUdf+OxxC/8wi985asPFSpV/AbMxt6e/mA01tc7wOqgtDQWTWQLKcossbjIUFBJAZ4eGECe7L3oAcZDWTmyVHlycy9IUJf54H/lAaMLvaZXNtbf/4H3btk6/p1vfZf6SJoxU+VCpS80NcV8jZRwWQlvTk3cAGf9wFvffOj48T/7kz/+8K/8ygc/8ZFvf+brukGbw+NIBOLVTMncZS3G8k88+uja6ibb2Nvh5zoOHz7obXM98dSjBw7vofOpzkKVSiGe23C60aNV+i+vGLU55pPrpPyu3Mg2JAlnshC8gO44L8V6ajilVSYXMLuKxW0jDsSOg4T8xrUJOkrRBNc34D125JiUva4uZbIZqJDYP+xc/AIeAoLhoQgERAX2NFeF0GCf8pDn/+6H2Dn//CEyV7QIiCLoJCUplaM61pbj1JgWyunkl3yT/cqOUi7t5oEQdHwEeCQvY5DUi+uqIB6/8GdhNshslmnOgSLg6onTedwdTksZqpx8ouQAa1FMAWMaGlHdelK1Z793bJtlZJgu4DH8VMCViGyblWAZkVtUHvnUlsBEMpFGsjS1dhSwSuXUarBfeQJM1SCBRHJmxRyQ8lgScpRMIVWpFxpaQFHQkdUg8bUkIrlk3lds+NWWreXmtrWoc2q6tLwJ5r1sttnNJrfFDu0uFqSemgWWLjcvlUIKywfTRPiXO5ZpILxKoE9ycxybmUEkI9JIIShjzHoWqaOMrCAFKByxofeKgklhcLRmp4ONTBsoRCuBkLy0rytK2kwUNxuiDo4S8kjwyWQPeSudy4lwbVa7On3UFAmjMEtWgDi4FjraywifMP4+QTR4ONlokjcEqC96gZgFwW9OJE4cI1GtE4YGSUZvaOgUcX+lCVsezlEUmKZ35wHwNDa7CyuZlurhSJJphGOfIiaB3qHRWZFggNETKCTcM4YC+Y5RxDno9DA3U5jsjff1t49u9wmZkhjXxFMrWjXmKTcgPwyZLA4ZOY6qjJdIZXF/+d1StMpzBpT1JlFoXrzpHLNNwMNQuE7Zi3jAgoP7WSeYr8h6bf2+uU5bi55zyRMumGfyUDSVBC5aHxYLhdytSEKB++PgUsIKuE6xOrkI5UJlvYvuph5XKrsxf3iC/sVeYdzxWZh/KpZYirKvODTTJgEbzstmluANd6zkralhY80onn3LGuHC+BjGK1cGCQfWEgh1FLcsMa7h9c3PaMiEEh8AiC3GgRycPzkUQR6xqBC6ksuFU0asAMaNKeB2kaS8gu2IgOFLChpBoPe4WepmVq+zEU8p5HLN6alcQy06zNfG9+amlnTqCrwgePN2q727q7fN045FNTq2hT21OD+TTidpQWcBmw3dbsWYS6URB2KrMKoysGwCySJyX3ozFfd0F0/wbmh9ke7YgAtRMpGNdXI8dgob9Ebo6JhfesCk4zl3R8/AlsHRXbuWw9GnXqMvAtxGqxOvnj22eyt8Dk/+8IdPPXuqd3ybxmoAhViuluLRILahDJpaDaEHThAaiKa4q8GoyuCoT8/2b90HnGHm9OnxvTtKV5+hQxRuZSgU6lxYGBjeEk4WBvoH//FHj4LNHhwcm7gxt33X/o1wIhhJwkt73/1v/uZXP7+8vvHmBx+48Nr5eCBotHu4P3bRsdsOzSysoxSXl1ZHhoawiDF146lSHi4FlRYHtM1lf/Cu2+vFwtjggBW5l05BmY1yvXLxEqBxKKyJKzLPm+vryRDczh6aOgTXNqDUgTgFgYBlTtCYgPmHfu/3vvcX/+vhhx/et3vf1MTM+Pi2tfUNRvrGxOR6KITLS6Ynk80ZLbahoZEXT7/KK8FQxI13w0ZVcq6IGCrRgGNBGC7iHMHNEsIsFTQGfyhLg/QNteksEmYOtAEaiL1NS3AI8Kwml8/9o0d/TPnQ+z/63u889I/I/fFjYxuLq/qaNpeowUkOugspASp1YWr2H3KZ8W3jn/jkbwKnv3jxgnenPz4VbTvsS6zEtT5VMZI3Oy1Xz16wOX00p2MXTL9yIZVOBJcWLi9f+Z+f/SObx5Atk4NURVIBj9d5+erZX/+lTyUCyWtTcwJqRRjn63CykMfFGi/lFLpBu6p7oI8dt7GyZnM58G75GPBRYGig6YWPwmYf27Z11769dFGMxKIkPaGldFgdqVCafYR85H7ZSrKpGBtl6/GktTeV1/7//Pr/8UnsHNHryma9+S9SSkZfQrdgTsWl4XSigRt1mP1bb7VO9k9irDVFrVdvvqdIAISqyUjv2TQV9hDLC9pWpCImFPPtsgh/bZbOQZVGm7dN5dBUkxtU8fZ1qEbGVcfu6D15z1D3AApyJV9cMBpI9IpLBr0W4oNJIe7G9ANYUpxJ5AC61qpRWevy26SE/pVlw6ABVc/mgK9n0pT6x4BKsdNVlB5h7VU1ubyJ6ESu0llVt9e1ndG4Z26tOrO0Ec+am1pnW3cHXbzo9eNy0cWwuhEIlIsFi51+VsScSTBgJaFxUSeS00WlYKQyeLJ4RbyJJ4P2lbHgMpg1VgALHkmNYmJYoUOgxzkjotEbvW6qBmjZQoUSYpNeKQwk36YGiYAhxpwPTgCPg7SpyYjFUGAX4DBQVkQgGoXpsJmka0+dWp8iWoIEFjk+I7WJNJbB7kDqNdAHEIDgcMPzr3Q3oo+FiHB5EKIHywyVG+pQiP9hjiwTtacAxuzwt7vd7UTdQ5HEamCNCnuki87oUqwCnDEtuDUaOdFrhWQeNCwSD4U9EEhXK5ROVRIxz2hENTeX8V1c7Rnql84DlJtpKlD8S9wAxLki/W8uPFQvopIEOQ6oDIH8yPUravgN8d16ERmh+MGiB1tOMB9gPP/p869/i4O3DtW64Z/93Vr9TAf3K6aR/CuTI0pOsq1A0MnNs+wwkmrIUFlyDKQoSenK29paGDKIeAUOCgQMOh2w/hyFNUARHegniRyAOuIgimKHCFRcc3b2TcODe+VIjGMVZukWDcI/XSMLouXBM4EYgawtuUaJjQvtGcvtZjxMsUhwLrlm0cFsVhk6qTADZUDFLdZqhiApkwOEElA+Xj0LlkAK3UhKkhfAQBNrkQg/OHUVyQi+n7JUMcnsr71yOg9Vr92kKqYh3xzubqOKrlyk4g6BDTyN5iXgYq2u7m31halMLOK2WZLFDKuTaLjIQ+5WbhHvWWQIcocdw42kMgVI44iaEgYnJ8yIMdM4f7Tu8rW7Giorvfayqazd6cGqhKiyvX9btlQNxlK7PD6nRldq1Ozapt2sdwLeTqawFrs6+9NVWNlcu4/cNjjQ84MvfGZj9joVfFIzplHRgpc7JewMVHJkZFu+rppd27x+/fqOA4e/+NWveS2aa6+8kopEOroHUsVqKBoxe9rz+XK71fqRD3+U5PHBe++rX9EFghsoSEiVz7zw5M5to9t27XvpxeeI4lLROzQy6m3vMjt8M/MzkIPaPR29PT3jA/3AKygpHh/dUk6Hrl48XylmxocGFiavQBRQzsZDq/OlTAJvYmoiD+SbhsGA5kLhKBRIWNxDgyOxUBRDHMwcF49aPPvKmYZRZ7Rbod3x+j3FxUVIH//+Kw/Bq0Xv9EsXr1A9NT01s2PHLgQDujZTIEynhv2KcT58+Mjy8ipKc35+gdyc3mTCJWGHsvZZU0T2iMuB7YSnjImStY2kkQlTdonoY+lyDSwZ+kZ6UAII5D2UK0zLrnbPa1cvErTv29HL5pg5Ozu4r29r99jc5HwwHC7FpP2fd9TpbvOA2T4Tj2L6/dYnf/P73/m2r6PNt9u3PL3Stb0dSZ1M5fu6+taqm3/w+38ERjebLwx9bOhPfv+/6HyOX/uN3zBYKEmvbB2H3Kx3eWXeYiP3RVzddeDg/mtPz+kB91OWCQcyNVIUWtJkngVH47JuX99gN0AH5o4QqCqnsnaaGM/wZogEGk1yEqn42fPhi5cvESTfsmUkk89i2aB9DRb4xKixlM3IdiOqJaJeYk6ISraheBo85O1/1+PmN1snUI7Qcj04JCq/CPcEoy+NzsrI6mKhpGSqZONI0O5nTtv6Fq8pVyrbnwfvA64UjwwSKLAXmE0CEREdVMmp/A6ju91DCQ9Li0wPTRgp9ADk/K639937wGFfh7pQWYnFVrSGrMdNpkmJjyHZ65BUCNu71AgC7KLaEZWAhUK5UcNcI55cxwhC35V11Sz2D11ui+CrMhLCoSkIGRSGEcsPCUScJV8yVGrearPT7N09P5ufmM0GASE1TMWmgy4RjG6uXHP6ugjPLKyuk0ew2W0kbPOpqITzwFKRRyOpIOhc3Fp8Sb6B7EcBMB6iehV7XxH9XCHnJVcqQoYfmThuimGm8g2KDPYasWNEdzqbBh6A+oxFovBYmu32zs4O3hXRDaZAg8FMF7cMHD5UEtIuC94XfN8KXcBxVKmw4gpgdRZqKvxSOkoRGUCjsYsQunS9BJAgjWJFGROB0VDvJ3UISq9h8dbMJMIpZMBXMli9bV0WRxNQp83um5pZAoAPxFi0hMKAZnaTRusiHUqwhFa0OpJ80rddNBP7FTUsHeY5gxTAYgOVkX2N+dlEMt0PMwgxd7wZyl7UdDJjvFrLhkXTMgcZMTRMaw2JJfOzP+Jr4kDKqpeoA2+1XpH75hXRHjBeoY1lkSpfVJ7Icnz9ULI2lYcsVs4oXjcHkeXMIVniaApoG2+uaWqtpD8HgWMaApPiRU3I2ZWBI1srfjPf53Xe4X6ZZYwK9CVDwSflcpSSXw6I34B3SsEU3YqIaKGLZOXI9TO1pAOwk4j88uCXXDnjyKXJjcpJ5ALZPMLegv8MhI9FxM6sUxYs64qceI3KK8x91qLgGBUdrNTbyUKW83JMWHIojoRrUA2xK2yWZpPgtATBgSzGL+abMo6tvc2VV+1W7frSUt+WMaLVnNRq0ybWF4YGu+CF4CJcFlPRoEtliwtTUxtrSbkOQy2VCW8u0kTIn5W9IHRXsgJZvyK/WPFiTPA/08FpiFWAh0rFonTQo4yCkJtk2LEhG7XlhQXYmB1WW62ao12P2WT3+fzwcezasd/e07kRirQN9L/tHW9//skn56euUe63sbzw7atXsEbvue/+R0+d6s0UhqmlNxrbfB4woLQ9YFl4nC5wwpFEOppc+/hv/Pb8ysbEwhr44Vw6dfddJyZvXAezHYdERmcc3roLt3h5bU1tdjFbVEz2DY0Wg5GpqRleh9GGzvC37D10+qXTt775bfOLi9NzS21e59DQ0PLKRnRhaXBs5/rSnDOVjm2uHbj9DpXO2rgQsRlHxvshUkh09W5LrCw2k2s/+cG33NAXF9JQTENXmYxGQU697W1v+/7Dj+KoFOCFCMLWacYJoO0yZVnYW3icQCWBf0DIYHW5uZ58ll51zVgcXLdzcz0Cxz7RMxpwsAzpOtXeoVu/cuWue+5Db129MY29RcXR8sqaWXi5m/BmQ16A6GKoaHeTyZUpiKD1laKAlR0oCkAWhGwIEhTsbmp1DGhwmMzE+sUNluRPsxGMhTJLGRY7K/MXfuEjX8l9ZXl6rc/XP7Z1axAD3KCyd7mIB1DntnXfrunZmcuXL//FX/yFu92XTKeQfZYOa2AjrCqqoN4Mb0agNBkdHJudWvruX3/pqz95eNuBw1PnX718+Wp7r2twuI8oFKmPoZEROh6y0V4+9aLb1MVeZrWp0nW9HeoGYngGlV4YXHUmpK5mM7SxtroOGhKaQ5VdmC7EDyMZka4X1AUGzdXWtrm20dHZSX3XlWtXISzLZ3JWvbVYB70l5SLsbbatSBh58FURdPKPPFGkgOwbOea/4SE7TqTeP3vI7uaoHEwElAT2ECzEDFsfY4fy05KQb7zS+lMiojz4svJd7Gy2HGY+F0odBRhA5IxZoJDwM0r3eqtZXS1jsqp2bnce2DfwjgchNq1p9ZuxZMhgLvv96Fc7dDp6nUNiZgheNQXTBsmvixLTwYSLlpefJiAjnDOjJDhZE6UC1aOVfI2+JulYCodbik9xyPmMxgS8LJNvJrLaQs2jNY3pHdtevJJZ3tBDDM81MXeIcKi2BBCj1ofC4SJRXGQhcfhUGJkhbSooSGRcABKjKQinMd2i0KhNzzI0irsiMkYRo4yGIg0FaSNOB5JYVAGx2qbG7fPRylfg5Y0GvBrABrEpMSsykTg4GHdvD4xXdPIGfIH44quw7ZdYQPmctaeNjkwpIzWfgEvyCGGkmwwzaoJNIv6iGAV0OzajoEUrIEFlbTBujAQ8fAQj0YLsHunvKTYDpegWPpIj5t1o2J2+bpM3nSnGY7nFtQWqnsHVohvEWfd40dN0xyI4rSOjY7QIqSAgauwI3HWEOQglEbAcm7mHEYnOkSjgeDkQUp0/O/emt2+vVCIwwhPzkg4Qkq1RwvJydcr/yhpmPWGptR7cDqcWZ0/eIkSAXcfu50WlJYPYQLI6iVLinnNSbAoKHFoJYFmzKHkFHMe3Rbsom4SB4BnjLk9YTIpNJP4pe4vLR4dRVmGhKIigfpX0HoB1M3F9OMOBNkB2IF0ocHOlHyRhB0ni0x1Eo4nHc5yBiiP0LBqLT3N3jL9yXiQX3+ekom05LzpQyvYkfM3aEBZA1gTGCxdGGT3rhqQca4tOU5hNgA5kRSmAca5WaYzENpCV0QroSi6XJaDIBOAV5FaZaxOiFT4XjDdwdeQtyqhsDi9OM1scOxQhQgWIsgY4qZQGce8l2q+C0dc34b0DGpoIbmIzEXjPR3NGk2bxRrhJk284b7TmPF0lTH6AyZlMkqZDHrexkUyNdnVByzHQ2YFqL9cx2xpam7VI/JfaOoaCLuiSVeFCcV4NtMZjT6TTKYwcsWcJg8JoynRUkKQ0rcsTmCEmjEYBvwAnB0VEYB0wJv3+9ja/d9uWYTURoUqRIB2hn7vuf1O0XMkVal5/u87pnJubhbkMg4yILtYvKWqK9smD7t6z/5VXzw5sGfP5PagTD6WODluXx3AlMR0PG9/1nvcEYqlnn3/uxH1vodxwZXISpbu4uu5r66DgdXJyauvhWzrbu0PhmNZoXZyce8/7PviXf/5nMDc889RTo+PbfG4nlGKqYiq0GIWAaUVbHujtGfIa8kvX4rEQIfHMCjVD9dvuuf3a09mVxTmIfQl65FR1KKQAfCysrOuMlkAkht1NsVAoGu9o64CQkAal1J6To5qYnbb6vG6Pmx4VoIQ3N4JHbznW97H+l58/tXvf6MT1G4FQxOawz8wtBCJxdhHdn5965lmBnev0K+sbNOEg4gqojd1KvRBYS/ZsNo+XTKoFYHMTVDarmswFERIauDPysiMETVsHrUUCjGWMU4IDjb9OOX94PfKWd7xtZXP5lXNnmXDWVSqT/pWP/epfffqvXn7udG9/P8SHTC26Xm3QRtaC/+m3f+PHP3n08sXzZ18+ZbDhXTWpMiLmj4zlfISLSrXS3j3bvv61by0srYzecuxXf+ljOiiEvW6nw1kqlt76nneo9eVQbJ0CdJaR1aTiu+emr+HpGhFhIAzRQuWKqcNbSOebBA64L1spHkx0dnW8663vJD+thL4vpsN5i9tYSLNCxQcgaWJ3AWKI//SJx44dP3b27FmWaDKSkvp7KRMQZ5SH7MHXlZ+4CYoY4fcbWpQdpXxMETR8VHmI+fyvPeTrItSU79z8APpMhBaOaUtwC19s61N88HVJokgVOSdbG4klNi4BSwADSlqaWJfMEsT8cDjKNMsJ6EgEo4QF+r46+jNPo9DoZsrhUg2Pqnbu8Rw+PHZwf5e2uaDWJchQuazkSSQUxpUzfs0aEg0OZ8QdT4QYWSQWRxWLWjQeyBupjRLVg7FCHqkYDoRzyQwlwJhAlLQSWaNpNSS9MCJvxuqZusffd6yzfe/1heJzz8yvhpBA5iqnQPCJNCTmDgxFE4oExIvFZZGAM8gZhIZEHlHxSEAyqRJz5jII8rEe8RhqFdQkSx1rX8qA+S7al4MVCgYIIp0OsIzFRIIBtvj9dNRGz+WKJTIRSFcWOVKcBQxW2QUVkNNK/ImjQcqLiiWOLB1cmzAo6L09nbVKPhxmmxB1BM9HGJggJ8PKCmwVpqCNELXqcDIJmy/TAfU684SXhhBm+RC/x/mkTjudKWBWWa02rpY8ncVJVMKOgxQDa5ojEKYpQOdV01YSeY3FoWY0TVaum9vClsVhAtGOnmuVhKNHGuBKhYi02ESYoisk9onm44R1VaZQC4SqyyuGxfnU9h0DqfQcWG4LJqqaZoclmTrWCMPLWuMX3pgBDaaDnYmiQuUVsSpY+7yNHFfWmTI1Endl+lshaGLREo7GSxRtoxgiLB80ys2HfFt2gqz71m95j+fyw2v4imJ2kt+2aIU2Te4QnILYkKw37F+8fM4mJyRxgLpo6W+VCnHGu2gNdC3TYDKRvSJArylkEV7U+bBT2VeMFcTc6EXRf5KI5eScUvFQxVUmTQuPjxRxsaMwdtCoUOJIOF7ulz2O581WFB9bLC1xcqhu40OCgpZL1GAIEG2CNBW2CyI0cltoeAFKyMiwGAU/IPtWMPAA+GhBKGY1Myh3w7ElbqHFTuZ9aIUwZ2l/jQUCsybrhr/l4NIsRVuzGh2Cla9rqYevl/PlQoOKw16vt5FN5upVUdd6bRHnDiB1Tev0OZD7OuiyWC7EzukTrNjmyl7B1hTYFw8umCfcrISsGUs1B6F6HgBH3mgx93R3ZHKVWHijTHrPYr168bXV+XmrtpmOhTwa7Ty61u574dRpg8+/a++e4f6+M08+gWOtlrq9fBtcyuEYqV/O3tPXd+josSeffU5r3Txy5PDy0ryBKgGL9aePfG9Lu4Wq3HAiBvrM35EB4Ty2Yx/9M0rFImyUgfWNe0/eSdv573zxb9//wQ9cv3R2LRKjGHhhsUKtwq1HD0NSBvMcNE/UXjkM6mQwZGo4nn/s0rED+2iS2N9NNWEptjyX1mmIYM29uLmxPEOmnJWdSiXxQXHWTLBhv3pudNuOu+6ljCe7srRkdrgKtTotlJtZqCjd03PzfgBf+dxmPO5wOcvIhaamY2Bgc3HJ5nTd/cD9FqsNtzhC1XY2ZsnmqCYr0gc3kYICBa+QaAoEnFCN1UKRAvSWBFWRniw+AiGYkDLNLA+sQNAibKUGxiUSHGMLNBOhCTqGk21AqYE3ZpPRuZTIOVX9X/3rr3iH2z/xiU9889vfIHwHJQh47f/y//yXr/6vh9ZnVsFF671W+nsQ9/J0d1648Nodd9xB26jzp04z53BfhJYDZo/dYrOUkqX8Rto30nPHbSdo9LYRiNCHolQpju0Ye/alJzLp3NDWnl3b9wTj68+/+OLa6tJwX/tSMMxW6ukbWJ2MoykdnZ69u/fC/ByH6wMqxx5rZ1+ny+siOkNROzIJq+a//bf/9qlPferZp16ks5yi59T5SOHYPUdhHHviiSeWr6eXl5d7u3vIXOSiufWlNSQ/ghjpiUHJ3sGPQuaIAOF/EVbykPSJQharvPYf/MUUiMxq+RvypOUoizoU9a+80votkkHCY6KJJUfWegu5pbxMMaGL1nhglRBLFgOpeINsbFos1QsWk6qnT3XrHf63vH3f6LijVAvkc+fcTiiSKFRhj0tMVywNospNKP5dzYZFrbJodVTt24AAi/gjfVqKizRRHCi0H6qKrYoGALgAF5vb4cglc9B6GIT1wQSSZCOUjeXMWvOwu33fZqztwivTs4FKWeNOQWFhALokP8J6R00s3YdBF0mWF/mP0OEHBdx6gnBggCSNxTWw/rHDJNQOu7jJBOAK65ClLCMl0yb1dbZuH3GjxOoajTMt/nbSvcTGsFvQi0yl+AlYdqUC+AFK2W1eBwEn8Nlo2Qqbp15ivol8ooDt1PxKnRaet9w+ml9sEPHWkM4sB3wrZDf7RrFKmsIkCo0Px8fEZLciMMGl4EpzFfFUAXnscLejzjbDYXK6drfv3e/9cCAcn5yEWDYdjmSh7WftASHUUyKGcaqHMhWmIyEQpYmK1++nNFG0hYh0KVeHrcnSsGFNl0X/StAcS0XeRedgYteDpfm5xsSN4Pa9+2rlgM6QF7WgsYg2IHN80/JjWICFkV7gjgx0pqSGTSBIygpjwDGQOCyymeFFxWK8Ep/hh8niHmrSI4gXUeg31bOoYVnMygWhDEUpyaP1hGvDo5Wjt2xM+QBOK3uMB84rs8CJJd7LcwxM4OK4CygirCuUqGRiuHoWiICkSI9BxYzqpX0Ys85CRzNijkt0ms+xRBiilofaugDF8eU65QpEwwK2IJTcVJOLZ9zAb5GRI9dE5TySRZBrnIxghagqPMkGqGvC+ULMoehYTsO1c2kkIERTit2CquZPmZ+WftNCcCMFuHyVLBbeMItWhlNuWjxp4W+TeJJGDAUZFN5tVmG1JPjIbZPHppcCNw7tdNPKyDNx3D8pD7p2EO3Tr81P0z0XQUVmmbsFFQh6iHZKuVIZd0oCg/hN9SaRE7XsJskZKDudOcUrlk1NLzKsMkCVGoNayqprTZxXOghw/eVSRq+CSCqVDAc1Tjf6qd/fPrR9bPri+bXZWax6p9c9sHV0x8FbTl04/3d/+/nDe8bvOnnH+sL0jWs3MJowkUdGt+JidvYMTE5P9fb3QdNIU+d0Knz1csbj91E4YHW1tXvdq+ub5lTBaiMaSmWBY21qsW9gmKUXCMJNnx0Z6ANc9tQjj2zfvuXYsVuXF27UimnYhiELBFwai0Ywh/fu2UccGCFORZNFWz39/BMdXleglNi5fbTgAFCr7nCZ5+fnq6UC+FDS8naaXgBXM1vaursXN0M9vb3l1Y21QIBCPtoDxSNRe76UT2cJ+/s6OtGSx44cf/TJx7kjEMv4DnNTM1cvXUWTBUOhVD4LoIpwASz80CJjhsPORqo+mc0Bl4L0uakzlXNE2oX7gEQQS5QtI9uHJ7IUNUgXZlWsdcw5vGxqtWGmhVtPrD3SkLDds7YapAfK0jJEzZBRYhRfDFP4RGsjqq5ZPJ2d0uPot37H9bnPfj4a3KwCvMfudOre+vZ3BoObp0+9+puf/K3ZmfnUWsjZ31bWV4rR7G/+/m9/8c+/ZO9vI/32B5/61O9/5i8+8YmP/eEf/uG2nVuf+s533NsG4Mq47677B/vGfvTDRzKxUjakCqrjRw7tLeWaMzOzrNtsMukZaIPtQ6KuNIrpMNOcm3UimTeT8fLFK6uLSydvvf2//z//DWn/W7/98S984W/ZwRjrXSNt4BUIJ0YjISBEgvRxuWKRWJenK7gWwBNhf0oCnkUuIBopMZAtjOJBEr0uUuSlf+ODseWhHOzmN2WTy0P0mgifNx4KfkIMaT7/uhrmTYlksT3RAoq/J69IjpUXMWW18TDgHpMVhgnqfMlA5aOY9HBCbxlUHb+t9/gd/SNj5vbOks4QtJriHk8NQ5PbwQDm3GK3w5InMo4cJthS2E1ceq1PZ/CohNAKiZ+XnqO1PCBTCnVK2XIGsB90coW8w+JMJ7MoD4ojHKDe8s1kNJfK6iJJt8m9zWgfW49aX5vLXl/OxSp6laVCVAzRBaEkIg8S6Hq+Xks3i1T3WnHewU2KVhM1zI8MADpCnAr8VQQURZCiC2TC0UzUgYteFq9XMr6EnalJoxo2g4jVuDxUDXg8ENuY6IAJ7ImcqLguiNhGFSVKg1TKrWjITY6UVr4lAtqietUGCK/E4acTPJFBjDAEpwyPiEx0MIJLogG8pmCRlNgef0kwWsQ9bhvemAFEKKUc+C4Wqxtrwe62pnPlxdUIord/YPvWrds7OnsuXZlb2QjQgzwciOMWQQorJAQAGAki0Svb5kClEEg0WeF2ba5sgISV04oqwpKlNzAIMXJ18CdgBxBA5nXB8TBgFN5AQZFRbW5Cn5sLrkDP66fZDgAlLGs+LBFXrru1uBg/7Am0BtlWDd69dH9CaDHyvKq0KOZzEjDgFewXCf7Kb35EN4v2VdYiEyEfk0XMVSpJVHlBHsoiE23FO9wCSlduQKwamVmUMrPD8lXKtSSljInNSKOjUL5ITLkSBU/HcZTjy82iZYUr0oiTiYWDzSRYJ/YnRxeLiOvFRFMug6vHaGL9MEVEGwRbyuckQgiACwiDroZnwVUqdggalXOih9BPnAvPmMmTJSiuCsXwEHeI88i1y72TFVZ+QFsw5ghW9gCgO96Su2ZTV2miAwTMjA0hbcv4AMBIxgHyUlYrwWrYOQROLi2yuaNyARwfJHcimEETENgAkgbnETnAIiSEsu50yHFidzTai0fjyUiQU9tYB6C1SYBY26BqoiZ1cmqatcgwmEXKU2BHFX2RGktR+xya+eF1ZSxZTew2wF02qwsPuFovAk10eiCQSiViQa+zDXrKUCIKuMDT3r1ndLy707188VUBydnMQ+Oj/NC2j5bMwaV5/fZBqvgwnOjKnkykjVZr/9BwljJiLerbS1PDldXFteD6rbfdlsBrz4bdHgsNfErBkrezg5L/dDI3NOQFRrJj2zi0/WRYqfsjA5ROZxx2W0eb5+zpU0PD3Ri5gXiQXQ1fdAWiFek32Zi8MQG0AzPU73YtbK5C45hKRKz6JnFXyjD4ShrIBKRjdieB4GK+DLk3A2iyODp6ug9pdBIrK+a7u7sdvjboq4Z6etbDkVIm67AYuwd6iSuIXc6koSMbzXAicfbcBdpDbd267ZEf/zgajYvPqjOQQUd7QqITS6UxdQi1EeBj1pMEoxrEmTNkoYS1DVuoSi0DdRCsKfSv7BmkEgZTy1oEOE+LQJYxAS0UOh4FC4mmsvgQFLdgQTsdLn9Hx8Vnz73w/EtAxm68fHXKMI0p+fxTL2Xj5WgiZvS4YPThCBAXjY+PM1bXJm58+xvfTi2FzN2ePbv2zczNhqeWCSB/8j//5z/85O+RO6gbjQ99/e/+9E//dOee7TR0at8+QkQ0dH3x7KlzyGKfp+NvP/cb8wtTf/EXfyxNqaO5eDSHx6VxGE7ceUc4GCogOlWqrt4O5iOeiluc1uG+MZAE5DJeeOEFEgp0/mDzfvSjH/nqV78OZwKT+sSPn3nJ8WIhXNU7BSRc0Oawgqmf5mPgzxX7r4ZtjRsdjydaJnXLAJfNzZgpUpkPy077P35w2H/+WeWFf34cJU/MzEjoS8TZzQcfa52Q8JvsL8KIEl0TaSJyg4BZoYklC0tis0llnzR42Dqu2r3He+TY8PZd3p4eegQGy+VAuZaimwL9xqjEUXQ/4oQrQ4RgHiOZDVLEC+QZEkMalOrdJNkQLljYGoODUuBMmoJ9aM8A/CLSUYka6vPUDYdaZ8xXrMvBejQK03ibv3MX7Z71lh0rCc2rl9fmgiW1o8vhsCfyWdiKoAgi0gXETlOkCQZ+MOFuFC5dg5S+FFw6tUZy66L45K9WsFKseK4NzcDIkARGl8AjBTUFCWnGBH9JUSsatZvIT0cH8076H8dXzCnEofTZbUDC7LBRgmRnJSDzSahCcEcMGXpDgwkRKxW/nBHRqwGmKj9KpBNdhHRHruM+QiWECSGKhCmQB58h6k76BveDP/Mk/4gq4foaKdYygjWx0IbcZLRB89kzNDa+A6ny8GMvhIL0ukrT3ZP0ptFuJx2H+ua+cELAhBLvBHRJ26MUwS2LxeGxSh2wxKElf6nGlS6b+J6JXkxk7sh0MyCMGQ9CnKIfVKpEvB4K1s68cuPuu7drtKl6PWNokL2XYm15iCbgRpXUgixNWU4ShKTrHl4dZhnaQtag3J5MAfcut4+K5BXWH/+LOSKaSFYqi1CuAc2uTFbrHByUd5hKxdm9+Rqnkh/eEcuRZ3KwhqRg0ZkcERgEBI6CNyN6w3XA5yy2MEoTt5uTYHW+8WBW+MHfJeBAIFZ8C6wP5kxZMzImeM8CSkb2Sf9vFCBIQsUcIHEr+lW5WkbeBKSIoQFUzoRxabKnxBVlIuQGGBHiPZxanHVFbFKljGWkJxAtV41+F02PpL6pgFEIMHPxIVLC2BEKSBDVKmuIRWXideXGFVVpMWJGEKBTUFIscmrXmEdsMYZHwNumTL5mtFtokJkuVgf9be0j46lTZ8xWE7EIgPF46j4rAWAXhwxtbmBfWXE67BauitQCcb8cFYdUdtS4EyS7cBcSnyfOAARfraMZaskF/YSOUA08a2rIVPkMndkoMTFijyXzkiyyF2MrK6W1ufDaKq1QNDZjvJh++cKZu+950Ou01dPxwOJse5uT0O7IlvGlxRWfX+v1tQ0OuTrHx2EW/Zv/+cfUM/R2tzmdYBHhu8lbaDrhdl2+fOUWb3tnb092fm1tcy2dL27fsde3ZSS3utnT4cXT6+/0P/nk4ydO3Lln77ZMbDO4ucKWf+fb33bm1LMU8rX7fWfPnu+CIcKDS12itUM6V6K0AWN9en7RblK7XRDbSrviDiKA5nwaidHQ54khBGMud/PK9Qm24uLyUiQWc3o79h488r2Hn9DZ7Ml4lEQaVtGTz740vKWP+DMxc+zHeCIVjaW02jAW08zsPLUrOL6yRCFRofUpLYfyZbrOYdZggKt1wmuMU8tyxDK2G815gLYsVALKMv6y4dhHItAUxcBqw7IkBkCRP8RSoPnNFjX4D6nBrNfJQqH8KPbFnHrz29/+yx//xF99/q+mL19WWdSBtWBgYeOPPv3pG1dmVpZWuvp6l2dmsASJCX37H75LCpY4xMTEpLWrLT8fMR+3Hjl0/Mfzqz/4/iMiR8lq5dIqqymwOP1nf/7HuCzTUzdghuKtzh2DbCV1VbM6v/6h9370tpPH7r3zLRcvnE/EiKVJ/ioTjX79q397x113j4wNZ7KJd77zHcdvOwaaH0oQG4VGW0YjmxsQrKJuR8dGSBmdeeUUG0QiAIq8wJ/w9lrZNvhw1G0fvvW2V188y26QvaQMCGu1FQnjTx4iMN54yFbDZhZJ/W99IEB4KJLy5ldbvi9y8ObfrX/4HJfyrz3YwXwd21ckpiK+YDJGQ3rsbqB1/OhMqs5O1dZdqvvuG7z9xNamOuj2hEnWVytJh01rMruhbaCchRiJyFXp3SviEcpHlC4yAx5eCqMEfgWbLjPE8iJyQ4elfBJmm0Q0lU3QLgnhizZAjljSyaLO7EoX9Mub+VTBZnFts1h65lL+xawmuFoMhFPJXLOmdwB9aBZzFr0OXkyYNYAkwL5Yr3J6wJikpXB5Sf0qAUjxjPBoGX8ZJwUQ0xr8mwIaGc3Na+02DG78JbIqgI2R+1BLEj4cHB7GKGGugC6ybjFhZVrV8DBLtthsMtlsJrsVK54wW7lazjGZKFTxpoTIkJCsOA/YPzIsZH2VjKRyPJw/3uE6IWWS7DV/iHYWWibCUFQZiONLLB0hbDRb8QHI69EeJp2r5EOrw6Pb33TfvaRiT50+OzO3iDpOZ9mJeC8SazUSr5JEnlwBAXIwz3iEkJfT6Wl0x9Yto0PQf1AaK+TGTIn4NQ1DuYRDRaW2u1gMsx4F+KY82HpkE8H4UCqysZY6p57cvpWgHa4i8ecsARLREgw3W1+8YNGdGHsoBr3e5PU6gH2nkrIYsWjg0EdH4S+2vF60Mk8kwakMjvwpGS2GWhkrDiXHU7QrR/2ZB3tJVusbUR5RyigXpkBQ6iCXUHcEtfHO+QhhdglDlTk8mVUUnoDLmGnlyGJuspDEApCogwBKQRGIYSBbV0J8yDumD5uQE4q6ZrwMesLmCA1GhibyGg0QABonEUVATJFmZhdIPpnezOhfohYQfXJnyEbCwAgOWYkcS9XIlUidECFEJVMShl2Od6InvswiQ8JyqaK2GUfRslgJLEfeR4lyCgw5GRFwBQwgl0HQsqEpY78qiWq5IUA4OaQygwsYgG5BmMQyTSxDjZEOX5gN+Msmm9T5Wi0qn5dAC7XkEhVh99TovJIU/jRqfNNpvCVMVUwpRo2zy5gSyqMhmaR8GV4cWqlJlZ5iRqr+TT0DXUw+EDCP20k4KJkIAydp62wDXlUvpUwqTYfTrioUb5x7rZoPqWvlaCp1bN/91p6Bnz7/ym0nTs7PT2pK+Tq9ZEzqmYV5MGwEf/q6+0MxSC2jnQP9S5NzDNHOPdt27d4WCNFoqAEWlN3O4Lj9bYl83tfVZ8PqDMehkn/6qceHh0YX5lduvfV2esru3rGTyt0ffffbt91+fGDb8OZqaeLKZbOmQj/ifCaD8gIc1t7ZBcQDpCJFve987wfLhWyn1z0/M3H29FPkPNMgwUirOrypdD5RqDZN9hMnj1+buI6dSy4ZC93r9WwZ6r+xsHz2m98Y2bHt3OUJj0XX7qa1kQ9+MSQHy4OUFIySJAGoH4zE4jRqTKfSToaFOI1Sa8YUY9dAP88OIF6ICUuzPcSVlBISWKpU4c9jWQoCXtG+zC1oBOaD1UqqhDXN4ifjC2wAbDPoFdat3+VAIJSVrIMw9oAyBu/X1Jw/99rTzz4Pz52tvY20MeZfrZh/7LHHAZ/pbJZP/u7vTN2Y/OEPfxgNBC+98FL76Mh73/teiPQeeeTHKpchmcgc27qT0E1bW7ucstmw93Zm6V9rUgWCKyNbbk1nIyaLEcrhj3/kE+1dnRM3pu45ce99d7/5C1/8nN7YoDEzVs3ff/kPn338+c9/+rMGp21mbhroKSxgP/rx9ydmrnv9bUPDWyw2EHnzZ156YWSgn6n/yz/60js+dN/999/f2zv15CMv2lxqSJHZU4ABNjYSmI+hxuaKzQNXdipO8KXQwvrAzBopRZEtmAM82FOtB18Uq/3f8VD0eut7rx/s5iH/2cFakpGZl9O8/tGWmBPRA5pSVK88kDw8uB7wK/F0CAPd36batUdzy7Ge3fv8fYPQuQTSqQUBsoP2oBwUUFUV4KzVaDJSF4sml+MwqcgY+kM1aW+A42uni5KYF4ChVEkWDeQq4kmG56v5NMFnoCd0/SNqSAASesdG0xGONeN5bcM85vaPwbBxZbk4uRSNZk2MJQQYdlublaaoZFDoiQaEMpdT4snoCQQx8g8tTDAZ9BlSmOA7t41SY2lLI0FGu4jQlJvH5ZfVfVPCE9oymBFokIdzhUhOq8Pp87axrQhOAr/AiISyFD2PypRwn44ubUJcRcCMuSzloeYqo79AO9AXGU0gqRjUC4pU6ZSD1AQBgbSXjD9+FE4aMp5fjZslPwyb4ry0CkTBvdbI9VF5xZ7TC2i5kYzEC1Ldqd++a9+e/bdoDdaXT527eHkCUFMJrzuwojJYYb8yGmwg6ChKkrg3ELZ6E/AMvLtE+kHU9Y70HTt+CD6u2blpIZcAPY2Xjw8B8zspWFph0cnL6/FnskkA1ZKQIFeqKGPGKp9Xra4maPZ15fKUzzvoNltIJ5louFrPYs8oK5qVo6xu7oZX9HpPGzmhbCRMuwl0mOQFUSQsOcxW7v51+JWcQs4iSVaBSPCuROTRGWK9Mnc3j8kRZBHzP9aRTKucRB7yoigsGV9AyPBpAOhWUv7YlTxY08wDORGORGqfSDj6lLNxcDSw8DdLBI/jwGZAhyLQLqgwuEbJaojKQQGKgFPOxeUoHioHoK4Qpc6NKMeXHqusJmHywozAOEQMon25EMp3EWp4UQw1aRd6rTM3LFQyHaw+UdgSM4S/W0sSHBsSESGDyUChJVkgCrQGPUzgE1WNMcArLVOAc+EWicvE4ieJIIaImEPlfCVbTfOEi4ZQkLgwz2W9NemMRPechs3jopAB5hnMukgiaVhezJUhvmOtae1m7FA1ghg0LNcByj+D5YV1zPXJmiUHXDXW61KA3NRJCEAKPHC4DRTAGIy2psrQ1TPETSCa2aL0f0TesTFKtCxUNwrpqN3U5TYZA6FkOU0RSdbf7sjXCk6fa/vddwZyxa889JVqPHbvHbdGQ+vaNheE1QgLyKGGt4wwWy+98KK33U+e0uNxBUOblVoW47ej24f9m8mnkN0jYzuwPQm8M8cMHWCEDKzUWs2WkYHXzr4CafALTz3+S7/0S2978J4f/OgH619fvffuOyhAYr9+6AMfmJmZQusMjozg9VpL9YmZybbO3nxTCw/g2Qsv0H5KbXJQ/+Wwe8a272iqjU1zSefuxF2Op9MEz2qZLBVZDjJR9EJi35eKuOx3v/ntD33h84VkPFOkadLGQF/34uJ8/8gQ6d7FFfqsqRx2WlYwfxoSyWsbawgMQU8iYKCfFauQVSZxHJYZPoEQtmskMAO6UqwlxIzsJTEQWURiqbFuWK5iv8piZeT5C4gKmtHpwshTkTtA5cs+A3tiUcEYQpTvXe949/cf+cHq+loO3HW9Pr5/z97du7eMjdPJ7o4TJ6ZnZiCy58Qmu51GbOSyto5t7enqeeap55om6/knn/3g+3/+fb/8scce/+nn/uZz125cpwGH0Y461JeqmVfOPa81NN0em4QTwpv33X/P6tLqw99/+AM///Of+dM/n5279uQzT7/lwbeurqw/+thP2/r6ImvroSIsDZL6WlpcD4QDdofnyFGCB+aJySnKfOPxKJ2Gv+f93sPfecrXY9kyBKtLO/3tEfXZbAHj1Osm22ZlmQY217vbB8gX5lQFJAYUAIQKWltYESAiM0Q0yBZRNnbrj3/T75ZHq8RU//fvvS4D35Bcci7Cc+xOpuifXuXUOF18lzd4Ww7CZhdaAlJAZbNJNTqmOnTYf/CWoe3b3S4vDt5qNLHc1+0mhFHIlEj4k+HRaSlSACCqBugjAWw0L0FtBDfxDTUd3/F6zbBcVCqQNufwPtG9FOvT9icbC4tIquIlEMPTY+wRKMkWmgm4oqE0843rdAMzm/pLU/G1iK6i6wyVc3a6ZOqtBIDpVqppWDTlZj4VJ0go7WyAVbHn6GfLPYo8Il5D8AickYT+MAwIhEsbAS6QvBtDx3IUx0IxGSTrT/QN+BUvag00kqGDKmgqK0w++rXVDYmHImyRdOg0ALaYGyYdOR2UKWgUAoK0g8U/wgcgYk83VQKgbH9RJPBAigXLqONU8Rm2BH4LR2Lj/H8rew/4uO7rzncwmN4rem8kSLBXsRdRvVmyLCVuWXsdp2eTbBKvk2x2ndh5efvstR3XOFbk2JIly5YUyRIlUuxd7GABQBB9BgNM7w1T9nvuyN4km/c+n3cFgYOZO/f+7/9//qef35FcRTKxZQ0UUVxQKta4o7Ia9SSC4HXG7gH/PI9JVKchUKIDKKCh+diJs1MzAfyI6VwlSyMyKk9Mdr3BCfwLzq1fennJQEyl05JaEfLhdt25c/vAsu5YKjTtGzeZDXRchwgpQxI/u56idXmKnNqkAzcEQ56cVxgyS4meLgKSlKKlaiik6ujQj4zMb9/W63ZZ83S+Io0WNDFWHGmAhJNHlccU3zEBAIdJA2gLjmbUjjroIE2kVZQSUT4QCXKWuKBLzIJwTLYIb4pMkjtyERZKCJyXkCd/cfFf/JZ3hKKFgOUQcCYhcfkX15zkmigF7HKOSH0yqiRNipxQGJOwINQf7oriBVdjiCI44T8sB5+I7AclgIfDWsZEJ+WJT7mjhMYlg5mIr/ypDIL1lhwDBSmaGKy0OASAuy6b5Jq4gfkKMw0TBE4OryDkh2ceuBKHRa9EaKU2WKgGU1qc5QXUAy6MEisLKdoPw0SG8Y8SkEZvYbhCwIIgjpqBIGfuYdJaJA5FbTgaCmBRljGWeHbKFwhNyzxzqXy5vpTXgbao0UXjKWMmC6YS0Ap0443HovRXQYgUltTkNGktRlEc8kkYLyTObtbScoJCazYaMhaTHY2hqqUoEGrGkYKVQz08FTgY+dlCdlnf8mKRJjbjxOEAsiPtAIPWojOkcxmLGzEcWfRN2yzmssqQoqS/ThuJ5yKTsw9+9BNL2efOHT54/daNtkY3RfSdiTjpUT7qUrS67p4+Nm9bazM97acm74CMCCT/0NDyuRnDPds2U3pw179Aq8NcNrDUUuru7iJjIxHNet0e5sNqtuVSifvv3TfY13PwjVcH+nopGDiw/6N3xm8v7+sh3fsHz31//749Tz7xyKUr13Lp+MXzs2A70od11fpdL/3wh/SwbO7qXfRVEHtNDto67bx0+ToqC02OPZW6m6Pj+DzpHroYTeSLpWhynM6gDR09+w48Pjoxs33n3tNHDlNNUmcwEUfauf/BU+dOQalmW12Gcv9ihYAQ4NiLgXmgH/F80TwXJcpogUQp6SHeocEXrWhpdXnw+1Qq+lWgRPItVBMIQ7QsIX8OwjfQuWwCDkhOtGcy49gJapXNYcXrSMI/5RN8BepFncPTSaLZqVMn2tvbh2/eMDrh7Evzc35IDDfapSvDTz31FO1+CVuTfpwAwVSvpxPiyy+/TD+GZcv7rrz/vrHF++OXXyBpK704/9Zbbzzz7FM//MfnoEHic0TfosGUyVK/MB+GWA4efJeORvgVgoH5//HFv965e8fWHZufeuzJF3/0MtH83/mt33/jtTeC47NIDcnS5QpCsZXQeOhQ+l2bDWwYcyafj8STL/zoZSqXwCru7hpI4hvJAnUEJImOKge2LqWUJI0jg9D1Q/4YeVxmK7VYaNVqE/hfBMw0wn+QfEyU7HQOWMMH+opYj/+/D2FRv/wSvErh9r9gWUx17RCd4pceO3mLvS0Hb1PBh+cJy0RWBZarrRhNBYtZ9ejjpi1bW9eu77dYcfHOp1JRo3mJ6tZsLmqk0z1ds8njpOQsS9qs0Wh3gn2Nyk79qlpN1B/pSyQVjsxUIiLppJcjo02xI0lQItZRoDc6hgINfYsVTaGko0FrOEE7S3WxzmWyLY8kG67djQ5PlBJFZ7m+sVg1ENlhzkqZYjqdKsQpElvCvCD4AbtCoElWPTxIHqFGkvBtngj/L6xTWBj2Oc8rdqEoCYxLzFySqChEZYwwqirwQlrAM5DyXofdySkgw3CgQGcZfT6DfgGSM1EVSIs8WYAXUCNIyMJQRO01GwllIdKpDiHGAvIBtqzkcnEgJFhngLDZOGTKMAy4FoPDv40ngTRHEUS0fs/R5pANJ8BCWOsEeLQmSzJTCAPh525cvWa9290aS+YPH7mgVhuyOXUknC0n0pTf1dlc5FmR4Uj3Ekm8ztPLvM5M9b9JByxHOOrfsGGoscVJZCkUmssSVkjHaGJHRja1w3iqRVGmlBFVm+KzYjYINpGdby7lUmB656gAkwwjQZIi/ayUmfOnGhv1Fy+Ou1z9juZl0dkLDpdZZhSZk0c9R7UWxl8tZzG+1YVsz6quqRkwcaqUeNXXW+kCjtYukV1WSvQemIGmWjQomZxSxQw5wzWEs4gshZ6BneMphOnLkkKxyteQA3AlSfuF4FlKSS0hD4rOKZKYzm/OQtopVC3qE3atoGAJrAmzD0g6zyX1TpxL6zCa0rIqyeQSXVjM9ALR6VF/yB0TuSWIfRjOslXYL5T6CNVJFRD7FlqrJ8cKzzqWKI4T9AMoCawYss+k/62CsyHPw9s8jhpUKxgr24V8N30okbGYJfJADIAWsiimmVzSbDMhBIUihZPCyvAv4cdAcySbTOaFFDO0BJJF0QKyBfFVc1sq2aRvMWcgFCVELCkG+HgYPDISxRH9AOcnGgfdR5PphD5Isjdg4SUHDtx67cKsj0T7dBKicIA7sRip3PvA/lMn3/EHfZlsGOWZiKNa6+Rp8aDrTPosNQtKaMRgshbCKdIsPQ3t9KjA+ty3/wBNHcKx+b6B1p4u140r75MAFQkGgNRyWJuweqnZtNnin/rE4//w/A/IHIYIdQb3levT1paQe8j42FPPTo2N+H2TmNJzvhnQ0EG06QKgcXxsbuquzea4eObMfZ/+dMOlC2gM7c3t2VSeDrvl4pUNW9evWt5PDtfw3DRpOM3eBrApAnOz8CAjbg10/lj0e9/8u6cef2x5d1s+FXpw/x6csZWu9lQ8QkRs3aqhW9evEvfub2ukVo8GoifPXLh64dj+jz397Gc/9Z/+w6fbu/fvefjDP37hh5oIrdaXegdWzPiONjU0TE3cffYjz3zjW99cvmzFvgdW0GaKREys2LfePRRVnRqfnGpranR6WpdAq4xGbORMGO2lequnqTFenFm5euDO+N0s6H6VosbVSEkxpIvQ1YgLTZ/Og9tD/0a0Kz10Co3hUsZ2EIWHohpoj0gHZMIBbckhiiCvWWXQFthIBaXHDBXwRemaURTHHckBKjWQvEQCxR9ULhrNtrcP/jxNw51KmeqIIrHq+vpGb2M8HPGP3/3+V79OkJhilNjCAs5B1K8/+LPP4QT+8lf/9m//9m8+/mufyARCN8cuzi3YDU2m944c7GhppRE1nee9bjtZ3xVt0aKnNSW5flkcbNFo3GzM+Kd9Vpvq/TPvHnr7Zzt37U+Fw8lwZP2qNSePnIBioWZGXg6r9M3kM1f1Xg0dLvRODzoNm85osg4P30HtLRX1nZ0rgQ/Tt9qgrlIpLywWVQNNGeECx0CPX6qmqDJnRkT9xdxhe5CwJsxR8Z7xL6JAEi/EBYEvh22jHEwlF+PgDA5YszK9//qXcK4PDq4n8y6/YRkiWJhaLDthD7wnjApxjxWgGBW8xYuSJEZznhSDCtQhrX7F08WpLY2qvfucu/d0D62CacU1mrs8i9kg9iWhp1wmB9ohD4DagRJPtJgcFLKPaEFWbwKJiewni/RRUFvEmkTrXypGQvM4hBNJmuGCakl1Csusz2QLJrd2DrXP1FyuGuNp83xYM+WvWt0rH3zssy+9dvLY2VGauah0rSraBsARkVwJbo4bOMX+QvaJH5fIYz05lyCXSYSOCYbRirXBEjLn+SxxNbpLFiJxQSyzWWLRqN3pIkdK/M+EpgnS0koBXIJsFgp1ta+w2zzkDGJggCnLroSR4XYDDYTub16K+Uxi+JKzRIRNfMslcmgqFgs1ccRdFTkrIUzpS4bhjTwlfsj/cH+0G8wu8U1CDUBY8TWAnjNpVUFroJWEmpBXhplHiMJuJfyO0kC7tkb35VsjzU1t7YN9n/jkr0/cnV0MxE+cvkYnblK96LRRptMK7h1qhat1dH0wu2yLoQW3x2MyGjkhVQzhfuvqavnkxz+bAy0vEgyGQrksQduqxywtH0jCEo2PqcICRXHA0JF9DJMHOqRaQskAWIwPoD+hXJE0ZC2pYrHS1HQY7KPmJs36NQ6T2VutxATYE7VBcRQQZOdsEZ/wC70qtbDQv9xx8XRcRzF5HZhnWKiyQDUZjMIkEIoYwYLIIbEBvsgs8VvGJmJL4iEiTqT6Vt4UOSTULB5gPlaGRuWbiB/WHp8EkXiUPgYsw2CmkUKIrCp41yZqMeRJofyar45NRB4NJTjidoCAiY1i6coleQNqEhHOnlKGI/JYGbZcmbGxw0mHIQFVtEBZXXkTnzAOPqXMCeHJg/CaUSFAZUPzsNxejFowXLkLHTDEEUPqLfFqyEMSpGSbfeAxVgiZEgK4BO4bxRcuE8JyyzwwMfAP/OGgdsAicL5LqwZGhLNdKtywVnlSJpZzeE7uwhMJh47EskaTxo4nGlM5lSrQrASzt4z7zkUHguZ2b1/fMhQ930KIR6S+U5UBs7DC5kP260wOg8NMFTQKyuZN99wZmzRb7eRexVNZm4MyoGbqc9g2JNEs5cHQweuF3YUGozF7HKlsxeNtoS12Oh0ZHj7fP9AxCy653lSnJrmx/N67xxx2T/fqwY9+9KNvvvHT6YnRlf2dVAlTKiHpUfRsD0Vo8zdXyN85dJhFBcV90/rN5AMvhBbpOPbKj18d2rS8q6/74YceOnvmYnBhcXRkCj24q7MfSOqqrrz9nq3ZZJxkydamwXgkePrkkd7BFS63MxpasNsdi/450CmSMTKBFlGmevoHSoVMo8v1w7/6Atrf+k0b123c2LZ79+aJqUhw8c6dOxiCcAPwKYHBOnfmVFdHJ+lX9cSKqpr5UALVvn/F2ouXr2FB9vf2njpx4sh7hx44cO/d8bGvf/PFtjZHODH5G7/zu+9fuQzmZTKbWbZyxeSdMaLjBJbIwSdoFM8kyEYhlQ+vCZ4VEp7hJByKgwZVb4k6dGhF1EwoiPVFgkAXCBNeoSOipbFjqIKHVKBgajkyGaPRQAYZJoHVqieVBXg7k1tYqdnlEsnA80P3pHtWs2H/QlIdykUTdSYIE+kPUB8oDkYQ5MmCJvI2fnL4O9/7LlvQ0emKR2kLEYNGOXARk/NB0ROgK0gICfZVAAUrAIArVld5yXctbu+UOqPZy4mWIdPbbx4iixWk3v/2l38ZmF+sd5rF9k1ltA3qQhxMGVUhUWpf2RIJhSiiJL+U+haVSXffgYfm/Yu///t/+vqrPzt29DCCS+HVBdnD+KvYXjhmytSAUQxA3gT5tFWMYLQitieHADWL3sK0ie2L5BY2wQfykB9sLd6uHcLX/98OFkD4J3vrX5zBirDlRDxL/pHMv2w62bxyJdgU73BD5UukRlqQKCrKCsQM6Oygf5F605buATBmGkpmQ0CtTeOKkBwRZBv6lfxDfLGmSwhPpOKBS4ocr4en66paOgGQSYD/REtILAPQHd7mHMmz/mw6aTZaDFpjKom5ULIZHfFoqlz1FiveOcCGp6KZUktz185t+z526PzMqM+QqXRqDQ5yKpWmPTDtcjIao6ygmE3TZ42nwwJVQDZQGHl84VrCdIRHC8dk9ipE3XlNChkZyhatSOsisHUlACv0RiMTS4wKZDu+bAIvubsV8wzwmKJEnQQAkzglDj7kuMthJpPZCKgQFXXA7OGBxGxaylvMNCmhHRzckfsRBBJlB5LHAqb8Er0HW5w5lwoVTCiatQO7hLYAsJd8C7xICzsdBgLzJ+8a2c3uELQt/AxV9kthYurGtj0HdmzfOzsTePnVg2Oj08xqIlaMBzPMtoZWzdJ/tkoPBgLV+JBmo353i4f8LdoI4+5ua2l86JHd/X3dwYVZ+m5nkrEKZR04OBEEoh4skYQFGdaohycQRQHjlCUl6ZrZwTw3GPBkiqkHvhhikGcAugFNPLRYunU77HaU3M6BZcucxWKCKzE1sAKmHq2Lx8aPJOQJaVfLbe0tE554YLrkthEbEGQrxJIIMMU4Y3ugqpB5wMA+GBInyWQqSqXCQNhXDAki5mCd2SbcgueAxGvvI31FwRT+RHae9AmR7GJFbMkOUA5QkyAZ7ANkFReDTykpcOwNRoKspZJcRySfJE8MZcbC+xJ1raMRr4SEWURUPogKVijRa7Y5oreEgsxQxRjhXyQ9H7HruBpDYyTil5aRKo+saB7cnfA3/7GZ+IBJZYdCUoyRZ+EBMVKYQ5GpzIPMfE3gyqdcVDl4cIkVCXmJNSCzoUwIRgIDWILrMFp0ug8OmRrZEsoh60IZXSoNDZDagCmLS9xAazOTwzbp8zU3eXsG+y6dPxmLBu0Wwr6yZNmlJcpI67UmRAK6DUo3wBGt7Z03RyYoUbXhRjEbG5qakGSTM1M01MRIylNGkYxD3GRVQEtgWkFQnNg70BsML47duaO3OTo6OoZHJ3X2pq72ToPV+Y/f+84XvvZlz9o1T6tLX/rCn01P3AVKjqY3OJNNJgv2kziG1Bpa4dqsDh6VfUrlDNUpwEtO+SbAZTx85ORjj3r27L73ue//8MD+R4w6ayaNS7dC9Sq6Jwx6+Ma1Bo9zKZemWW+92dJabSNzDb/opG8B2MvFeKpSpz9+5uLZy7fY5WjHDY02l4t8q6Rvejq16F/e1XorE58eu43TlV39/plT4UjMbLW1tzaP3pl8/8K5hWDc4fIwQyRqsSfPnDlDMwyUFc7HXCOjrbHRDGtav2HD8WNHrt246Q8klw/1TI3fWTG4zDd+NxyIIwnIngQWAnqAoOgXCR3LzhAmLpweFVKUVxx2+FTF5mALi2Ylv3kbIaQlCiDZW7Bu6EGIrk6Ful/VFSMpiKpiNhTIqYQMPB53FL9HOkPsiSIQqsNxi+mW6vKhhPQjzND2Bqx4qUpWknTVWpueR/B4XCqXChTudAQMBtgn/m07oN+ZdDpDyWa5Cpsn2KyqCJA9mwQfOE3LIfN8uGjt1CfCBbNN1b/VFk/mIXWB5sKzHQh85JlnL154//KhM6BiLcUq7g5rJJiiC+rExRlTqzkRTzU3t87lfKp4EWhSskC/8pWv/MozHzl39hSFSX7fTHB+jgQHtotkGjGHBrgPvBf5DoYjPljcQSKYeX6ks0wV8lbmgJkT4cHB7uJ3bZPUfvOnrMK/e4ipR0KGcsjs88NVYElKpEquJPfiX65auz4mKUY5gTq9lOLjJsyrqmTQI8xULW2C5Lxpk2HDxq6BgSarFTYcZ+oUf60oWoqiLnkneJtghXJN2DHaunAlA49bpY8C7iuCvsisIkIXz1E6EUpmU4VYMGk22G3GBrQQ/NAsCoWZqdLSdEibL3sjvur0bM7bumHTxkfmY4YfvHTIt1icDxaoOK/TG+EhooKhU2fQquOAuKgYMA+B/S5PJd5j+S22hcwdmbBCpWJCYLaL8kBXaWwLSgrhSSqrNPsSv2MFryF2Zr3G2kAHRYoOrBaHaHrszHQaXDMcPjBw0inItnKBxAJVo23yw51F7xFLW+QmHkKZBxgp2hXiUzQACe5KrwQ4OGeKtYRlyGJj2uCUIIiDiGY+pQqmIuDkdQDVwSOYaxZPrQGMTfylVsuHdz/k9LbcGB69fevuzRtjADtn0qV8Iq+2WlgG6jTxluM0Aggklkj4wz5Dg93sNFM2otZVl3cPrF2zcrC/1zczQZSH+wjA51Iebwt7tUwVcz4vPbmYQ4X2eCA5JFpO0KhQAAWTtCyDQU0wkd0HGUoTXZ6aWVERkKDLtGpsNN7TvdjY2Eb8D8rG6QN+A1/nikJ4Mi9AouSIp1cKmaHVHUH/LP4f9BhEi8JbFNqX12wAuIoiXUUf5T7K4goJM7qaiOW6io2JTq8Yr4gxmW6Ry6IEyt2EVUiyFSDakCpKI64ZFHMKUzlJVgVJBz9SdhTvSOqxuHahZFr6IDqVfUiOEU7bX8h4sYxlMLibqSzi+9LBl+g4AhKVi3FyMQas6M8yXMbGSGQ/itQUrZqx8Rd0IHyTt3kLWuUCjAAyKZaJQkCyvMHFeMFy4u6BRkWuyzclg0B0BvQbvoK8hebwbEtttVqL7Vynovi3qmMSROjjXufWDFWCwWx45daycXkB5SoHa4SFJI54NAtC1NSJa+kTUU4kI2672WHXq5Yyc9PjuEDMqGDFBHV5+Cb0BlJ2GguV+kgiWdaVsHeBZUZbVNcnsK0dZquEP+NJl9NNOFOWUKk3IJiTS5FPKHhv5Kfjrm/raHe4rbfvjOFy37NuDbCR0XT2vv27F8Pxa5ffP/zjFw586GH/3FSDyz47O7t980ZYAMMkfOV1e9kXiHZWBNDbuZmZq5ev7dyx+/pwBETl1tbOrQ/v//Y3vnrw7fc6O3o2rN/a3t4ZXozOzc1Rim02G0nGXowEMaZbm5xas769oxXcgeMnTzMzfT3ddk9T3+DyU6dOkYjRuWyIwqe+/pULwSANVhcCIeZ99OaNsG+qo62Rw6A2QNLUUpPLV8gkkSCBOf/qlSva2jsmZ/2gwgpJqLWPPvnh7//j8wCDFDLEdzTTU1NkVZIZTlgrn83NzEwDe7lp04btu3fh433rjTd37tz59s8PBoOlhkZ4dF0qLXEUsW+F4hXpq4gM0ddYTTK9PtglQlNID9mfHMpHcBnokGqvPMEJDUIO2gXUH5Ghcjjrk0h5pFQdGH/YRUu0tpHNQ4LCUgW8VjVOnXgavFYIVaq9hdK4gbhxQSSlAKm7t4vs6oXZ2YaujqBv1tvWTE4cuR1YVBLBJr9BUxeKpmngS1K8xmCm9gHPHoMBpoEiaY+HQjqWMYN8pDQmTXQgmVOFqi+99BLuTW2zmSptVg09mPs+/NCjIx2jh3/6nrXVMXfDZ2uzJXPJS0dP7XzoQSLKVy9fampuMBktLBA7SJzzPInCDbBpKPMjExMBzKywu9lcKNayZxEIwj04TxzQ/M+k8QEcXJk/mc5/+UIkyv95IAHQFET+sMkQ7EzoB9KXW3BB+RI/sg9k+8vCLBGVJWdXL/63MhpSxlivMllVHZ2qnbtU9x7o7+rBpmKbTBHzp1+ZWY8sFU5U274g7cDwRP4ol5fYsfADLW5ndKsq+b9qEyjxyXRKJjSdypNekUb9wbQw0F8KyZvJwm1stF6OJqITvlTVuGoxpi+WTN6eJoO1+8Lt5I2RifFpau6bM6DLa/UA3kEhOUFTyhOck/syxfI82A3MCs8kv+C6wqqkikR8LbAp/kQEW2zmXJFcoizfQN4RAKFayGCSDcXCYN04nSDSNqFbc5dwOCbKGrgHgnxZwNgFIhdfC1yINCPxG2Lx4ImRrJ06oxYvMqjPiFWMAQYgxhCMGG1EDCtOgutCtCLQMIbxGCGbWaIiiLNkGWeppKSxBBINiH+DOpoDmspisjnq6vSRBAhW5Y7erhUr1pKFPD25ePnyNXY6kAzoHnmCsghCQMUo89VqU3RuLeJYwhFtqjPVZZZSs1NxskSfeOyRFcvawgvxmdmJZDREBjfw1xXcBhVxJSL+CazRRoNee3jX4NYssOgKIjehDCxWBC3dzaQNHAcahfIsSCGUBaQvSKcZVVKl8vlUY3eiLa3WwRWWSl28XE5Dh3i9REjANkh6k7vhhSZHPOlt6+jsVt26pmppwBPFYtFJCr2UmL5C99CmZHsiHEWXVziMLLRyIBdrzEUhPGEGNRMWZYfKU2acCynsCS+JVGWozHqNFA+JCUl6MfsL87iWW8UG4+EUdQflDNczgRiASev0qGM0IoW8RcNQdCieGTknew8KY2pE8xJfFqJX1lpID4NK1liS87ixIEui47GzxSEMobJRuKC0ElfoVvgYj6aIdh5QQVQWJdwMdjItFjClEbEsEQqEaJBcX2QtdxUHNjxNug4ycrHsxVhHD0YdxA3EvuQ2cqK4EPC+MP+cw4OLxEbysrgIcFEbhVuwgmwlvuGwkpquVzpV0OkvT1/VkrpotjvoHUcKQyYZsVDGXKXFNwFrmAYZWJYlAPLR/Mx1xTq6aZcDwTDk2DOwjOTD+UVQeulZ2dbV0S4wEcBwM+UyM7Jj2RMotvApwDTIVd65e+ccaDGJ1PjEuKBY1BvqyiBlRb0O88jwlbpy+sqlc3arwWmn+L5pbs5Ha08M7l27dkUisWPHT7a2tgP2O7hshX9+/uevv9na0bpz194jx4+mFmLNjT14Pnds381qXrk8HAmGRckAXzqZqC7lGxucC77oyOiw3WKm1igQz5Bb5PV6N2/einkaCkcbO/rQTHEotVLEb3Z4G7XTUz6o57d/+7ePHnozMz+ZDi80ufDganu6u/wLiyB0guvMjFNob3W4bS5Kji3HTp3u6+s3212QRF/vwOLC/KYN644cOgQI9vREpKW5qXlZEylD3V0dy1cM4uw6f/r07n17v/+Tly++czi4EBoeBvkL0cjA6VBFYoMazQY6ZHcIN2EZRZBIQZogvLB7hQrlYKIV3ZV/BJlOY6wjba4oGQxazAuLVj0dWuQknc1mqABJL0I1mc65PGbEPx7jDBoSuwZ8+SwAzKAqCa1I4Z4REmOdDeKUKVXpk3jh8iVvc2sossCecLe1/9WX/jvtZV588aXRGyPj5TusJOeT8w83kGQnhC4uZm/Drj07ly3v+cJ//fxAb5vOJvWdk9PzBkFor0tFs6pGfXwi0LtxiMy7ed80lW+5WMHVQPyYBo6W3/v8H3z3u9+1t9p4ZK2N9MCi0+l+9iObcWNcvHRhfpbE+UXqp7iYyWAk9lfOM/ewLz3IbTmMe9lNTKRWmEJNWWaDMRfiiYbvIZNrQvN/y10+5lCEjJz67x6K9YfEr21WthdiWA5Fniv7t7Y0fK7wVryjRuAH1UQmM0ymWasaHFQtG1Tfe39PR7e6vZ0svQjgK/RhZwVx+2Me4lhU7D3EHSUOsH4StsTpJ6OGscEgQLkqkycLw67m4wkkLrhReJ7zJMqLSi9tJHlOYgQqlUFjskQSpYWpRKXOanRtHJu3tPZuy+TrLl4evzs1pjY0qHUNxXr8uk48CaTR5AuJbCYJs5aU5nxSAoq8hgNzd5kxnpdNzkrzgJgjnAaPgxVjZ8IMoSYySmUYegKlZhtTTT7+zPgEGb8A9bicXgqN0P5xiadIC0PO5xIkMMDorBYQBBC+cGjyemjJmEf7E+uH0kmYm6J4sCvBLJC7sY7QJbVGBDJJZ4XJiWhZYmn5QMwjRioLrTRQhG0qdC2sVqZRQ3ffVDHrNNkiRRWdpJvbO9dt2II0HL0zM3L7bkGyXbCITODrpCS0pGWP49cUaBNsWUq3KAHCU0SFkaoItsbqdSvvP3Cvw6aKhfErRRcX5kj7YfCURZBTRb0mUBF4EQj9YQRLj0wRe6gsipgRKawc4nYQFH4+paKYIiyKobifTLmElwrlNCmUalU0Rllwdnom1dHdTO41BdnVSgKvMmoIc01CFjFgAx6RfNpAEK4cWbbCc2s4TEWi8GQEcAUHOpRGfS6Xlk2AfKpNl+wGkXC1n5r0ZVAiC+VNETDIXjgtQo01RxLJhIhByafyCIpXSRHn6AFkRYiiqBXjVaQRKyBGBY+OmciyYvIyJAF44qakJoHEIsaibAAZk5CcDAY4D0FmgdLE4JZqcTRnllhsTlGdiS1pyZfCEcCVKYORtC3mC9UPmYojmoHJSTXhygNCG9xE9DYmgVRVUlupxcMUFv2DN7k7xAKbFEVCUsyYBxgxOgcar5T9yNWQ/ohX6dyAMCYiI5yXg++KJsBjyEnQrSKAEddYTcSLtHJNQtfom2SlSP4CcYd81ooZU18OBmZJtDDq6FqRz2VSnrZGDYA8BmM+ay6Ute7mlr7mxlguiKuZ2YzGkv3Lm10e1/jUNK5mrI3JyUmxReqraLJMCIYOaWOGejPEBsQYubjJO8kZv2/H3t1vvPXO3NxMR3f/3FxobnI8GU/mYuH+/m6Dus5ixKuiy2ezoN4gzgH/InkVoEcsOMDWkco0Gzh75tz27dsJSS6GglfOXdr/O38ABppzeBxYx8FlqUg4tnbtOltDs0q6uZW+87Uvj4/QUrDbbKQPRAOTSS/bvQ88anc1YrVrvR59KDrtu7lhwwafb37T+g3f/tY3RkfGH3v0EaNGd+v6tZd/9MMPPXKfJhdOgZwx77faHY2NXh6NqlYwNMh/xEl+9uy5ika/beceCrJ6+5ZPzs4x6+vXry+XVm/ctA7F7acvvQAhMSef+9yf/PUXv9A/MEBYGntldPTmc3//7bOnTy7rG2xubQNqgwZN6AyUkZPUnkzjvCU/R6xUqEbEsOIlgpDFuSxGgBA0tFXbRsgYiAbpq9YTvIMOxWECvI6zuWHZ5vUXrlxcnA2LL9CBXUH9QhU/+djdccQhje+ZKqgVFRsRAjlBS+JPAR1AnLcKyRpVc3N+V6ObrlOqjPjVOefgwYOo6DeGh3s6evxzPtAMCW5Iipdk91ImV+zt7kqkksFwUDtVZ3c5UR2SiTBka3fZe3qXj41OppK53/yN3zx+5GQ8God+KGTC4mrram1panbaXZFg7OknP3r65JkrV95XZVWOFo+9uYO2j+RCr1q1Eok7OzcPsUHoOEsxX0S1Bla0qgaoATlEZbZotwpfUKZO1FNlwhSW9i+UfTaRsl9kOjn485e/lTf+zS9uiDHChkIOyTatfSyySZHBfJkf2Y6cqLy26igkQ/QiCVW9napVq80bNrb0DRhXrrHXqYMV1SKLTr0A4GUMgxRO4Wg0ogLmTm2oquloQXY4QQeYn1hNoryjc2NpknZL9LKoipO+kFPlMwRJ8bRS9KfcWKVKpVNqvDJqEjXq5iO6ZK7NaGpXqVeSgfTO6dCMP0Rdr9beHAilDWajq6kzHIlniNsjJDBewbWA17OamBuIYZ5N8cgoclcRurwFlYhYE12/Nqf4e2CCqJAGG1meZCjTA5ja9bLJaMsbK+1tncSEQXvD/xdLRrDYRaLVVaQQhNpjvd5iMWMaiPkm1ScYtznArMASpoodmwiTiScEsRWvB+xNUtfEr4tJLWKW7DAMS/4RIq6ZyKJAIuOqpMqj+2DOS+MdJJNWn6vWU+zgauue9QfNNveOA/e2QFrjU7dujmaSuegiOWdUXBGDS4N7Z3K5SGJE1JusFiwWKWYxaPDTFFMxjd3S1d6xd/+2TiDjvaozp2/4pqYa3a48UPbJJHXBZGtj9YKtRdoNTgkc9IikmgBmdChZMn01O7Nm6yEwUKtALoSlKuUKYpZKxJtgdZHQKdWt1Uyq4vOXx8ejvQPOljaErEMqe8sZSeQGN1yRD/ILUqFyIhl3ehxr16smR2sCGMVFRDwRS4SBCA+RJiJ7+Ip86/84xA/HpcRIxQQUMpSAP2+KFoQ/XyLxSMAyk0y0UsQMGiIaLzo8qChGm53wfkbRN9gqiFIpAsMNwn3IPkEEcndcZdiQPDJBKVaPNaNOA/czq853xJRSwh/cDktdjDuoA/2FPiNYNETNDTo8KAQyuDF/cx0kIfxSnSd7DVrhXIkQwcoYam2/ShQczwp6UWHJSPsmOkiItYHPUKQv18BXg35AdB5dqMZ8uREeZkxMLkcQhYi0EmaW+cJIxqWBZkJSIFMksFwym4oRjIEuDy/5LxaroqYIwijOdNreEaClUzIwSegKwFHEsrkUvlW0NuhdUDgqmlJRpzXZGz3d7T0DRpspP088OACumI0sJpcHD6fVbqtUTMxIgCJdXgMhX6m0NbeEFxfI5ud9gukgXAYWA02tTYHg4pZ797q9npa2rk179oz+3feIeidJILToacWyY+fjpUKSEvvrVy7CjtEDKIQn9Dh++zaKMxCPJ4+fXLt6HVtsenImEoo2t7bgurz14ssrDxzYvnV3T/vAT3/2s87O7snxKZvFShbYtns2P/vM0/RJOX3iyNzMpNdjG1y7xo5V5nHOzfsvXb42MTGFD2rlypVUnY6OnQDNladFtL/37qH2Ju++HTsuXTxPauWevXveeu2nTCNR5zdef7O5o6O9s2sxHCMjA9pYtmI1DqVuAHLWbens6l46fuKFF14YHBwkMhoK+Kilhtz6+3tQar7+tf/pdbv27Nzx05d+/Nu/+zsb164JBpuJB8fCCbPZzvKh9klioN7AC9iF3ghVcFvhfBAzjkGoR7DrZJugyNZEhbKNIDJULw1RLRMAWeSls3mFgRWX3Dr9g48+1jc0+PUvfQXKwTjW4ePTahPZFGCfhLISmWwlBXiumkxtRHutnpN9gSJHzg7Y5cgNKhpzlQKST5WM23u7iPrv2HnP7dvDCDmGwhop+KiKYIIjowVider1k9OTn/z0p9evX/f5P/sTloPhNTY3oFr5FoOoL7PTfppfbdmyhXjB8LXhj/3qrz7/3D+Q7wrB0GuLhlrPPv3sb332tx577JHZuQk4YjxAUZPhvvsfoljl//r85yFJ4VGgFEkjGYBNsmTrGHVm2IMAMGAmom7gVlMMX7aDTKEiaNmDbA7+ZGZlcoV/YPfJoZwm7IEXLAEvZHf9m4OdLEVhylH7nEtxQbFYFVeTcivFR8qF2JslsGBxDDmcqpWrVJu3u1avQd0ijohp4aNOHgLT6+0aYlD5HMmYpCkpvQfRpIx19Sa1CogbfGWUOosOj3NJtj/cis4MmUKSlCGS5qMpFZrvEtcANQcOLNFQgs71OmssU/YBapqz21ybGpuGZudUF48FxwKLWoJKJa+q7NLqbVZ3I5VLM76Q+IFB/661TxD5CqAVzIgkHQQkjyqCTSwl4SxwNHRDxKf4GXkf/VDsUIQGgbSq3m53Mgx2RzpJFy/7wMCgZMIJp12Kx4HMweyVzDjmn4iK00H1FJ1gALNgyklmovqPnEFSCuCmYoFgZSBvkf2SrlshN4K5gAUzt5hVIr7AOQCUVVYBYczffENIQFllFkfsD/ELUosGP81SvcJ5evPolP/+Bx/r6hkcGZk4e+4QdToCd3V3RkWkymzX2MGCpk0fncQMRSX5mrZsIOpAVqRnk3HQNNC9d+/OrVv6csnC5QtnDwcC1EnaDdqFualiNgOcbWRxAdx4RC++U+pMkU9c0EjhnMwEo0WM8SQ4NfjFHhfmzPoiuoQQYQckvgj0g6haJRNQ6Vod9j2bnyI8f0DlnEzNzxdtTr3Ti9ZZFItMQismHAE8L1LfaOG6BS1Z9pnUhh0D5JuwvXE9w0QQPzIExRbkaxyyqsqPrGyNrJUXNS2f37JTFP8qAkZIgdGi5oipiaJGIRaER8d0JDE55ZJvSKUYp4AHZHdYAwt0rZFtITcVCiogYgVqQ1oeUZJLLiXIG1wKCkALRVnKYg4ggGF/PJTkV7Hp0LlQ2BDAWAXiP2c8gjyGDSGWL84QJY5OlRFt3nkKpKO4pAHokG8purKiZ8joZSfJYITiCwKQCSEyPJBueTocaBLKyPFFwUJDp+NMDuUcXNBkhH2gy9OAVwhNllA8P6hNog/QpYh6Kc5mkj7gIeIW5448BZEok4lRw2sl0CBnVSuksCKSgT4sV6KQr9aotjvsKAH1WnMyU2YvAUFrbmz3TY1NzoLymIdX796xs6Ora25mihYlkRBp9otAJYOMn4qBS6UFioj6QykgIU1clE7aFwa8bS2APb322muLoSgZT6uiUXIbwwvzzW771Njt9hbv7OhtGvmRedHy4IMIYNykyuyV/f7A7l0r9u7dN3z9JluanCxI1Gy23hy+hZs8EInNR5I9g4O923ftCIRHR0effPIJWjJcufQ+4VWX0wpLx1VOq6VzZ07w6SOPPDJ87WpgIeS22wg5a/TW3fceoASW8ZOQGkol8D2tXTG4Y/O6O7du5JOxn7304uOPHNi4eTMWM4kkk3OiMqfz0fWbtrV09Hzre889+7GnF8LRw8dOur1NR06exTvd1dVDGgVS9sNPfUjysBoao6FFlga/wsrB9eR83bN18z89/4+UYG/dvo3MahaR6BQGPbeAYiSioqo3W0HgxuEsnIONhaoIWyWIid3Fr5ryLN4VRXflHdg564kJCPJzdkllxD5iIyMvM2nguugWZXTbhBGgEul1nd1dATCW21rx00QoEi3FqETEGYT7UTREfoS7qak4gUpsTteyweVAcq3btPFr3/wqqVXIXeICMzMTBNR7ujtfeuEl0XcJjCrD4+uQFvhfiP++gf5NmzaTT3fuxLG0K12qpxgtDs3O+vyKoVz31ltvQeXEKY4dO4ZsppNrZDFEofN3v/UdcuPJXaX42OahlFfTubIFCE5vg/vdd94d3Lxx5NIlSod1FuGS1KXiM8+TR1uhx2VWi3cANoPXWcQVW4i9w87CE8aDCZtT/tSwcxknuxhXv0D+ijOJB5Djly9qf/6r32JgKVdR/pWLw/DwFcl7KM8IJ+QVXjLF64YqIsEj1cpB1Y49po1bO9s66o2WnN4QZ7MTFORG5OrSbIWoIClrEjHXWwtV8EptmjpDvZQVmYhQio7NSmPhCrwFeSIUrCzlE5l0lMh4llwQuqSQ64LmgXMTXwBRzEJFPx/Lp5ZMJU2z2bssUWy7drMwM1ehtLXO3KDSWRF1NncrUKeh+Tm+BaGC1kMFGLYJ0QGYlkA6MyFSOKs49SRIiKCF0WGlQXoijwnww7JE5ON04fGFayCwrKhEiTCBKo3Z4u1H/ewbmJ9fgNR5XuxjMpEwgpl5TGTMAJOpngiISFkVCMAYuvBnQTZw2EAXwOTDFgEGC24vnkI1/VZZbG7JD0sp4gYehj0CBqD4HaCumoIgfBbDB/8o6A1YkETdAVJOZ8PpbL1UHTv+w69/dGp28eDBY/P+UCouid7Id5XFbTW7oBccXXqnNZtJk5dBgg6ecWw+afi0ECS7+6EnHnry6U3c7trF6clb1+yAQKvJSqwAMVTOJDDkAz5qMTDKJeeGCWL3Et8FUI/qSyaMd0TQKvak8H6IColE4hlhbTYYrRsgTWx/8InJakWjTZNOQl9QFZXRJdB38SrM+VTnL0y6G5Z3rxkI3F102Bwk+VJBbKCeQrwKuXKe/CBYv9TY5JOBA/e3vfIDHyVsRq06mcjS6SlDSbPiOJV5krCnbA8ORAUHg2IqGRm/fyF9eS3KAciHspcQj4pZzIRDBCazPpuU5rsczBcMwWbVkEwkVV+siGR3S66SqDICOg7al4Blx2MZWUSVoH21tGD9uGHKYHpyKJuXBgMlaJKr4QHGLcImg+LYEJJCRe40s0sRIQjHNjOsCzBtGo6Ix4NjiW5ISqazxGtx51HoIfQMFSPVyc0ExgyXgR7VgRKmIqW0ImIVpkBfInEp6/XixEZ+2y16aImrMGb+Zm5QcXDxsSOUfGyZMfR41o4ACL51DggOfG/2C8n3wo9BxFE0LI1ZMmC4E8yHVEKoHfLlici5YxzUJuKKt9hMTo/d4fRaHC3dA96G/m35KLXhOZPTXa83Da1ZbdKaSJaJIkHJlFBV3R4nyahGrTkaCoPJE4xGE7E4xiL4UKQddff1ZgrFts6OaDotmEoVVUtHNynKzz//T/2d3WIjxhPbtmy4fPUKg4Vr3woEqAgkj4Zd3tvXd/rUebPFdfTY8RUrhlip2Vkf3Xvg10A3t7W18IJ4EgZxNl8i0gWz2Ltzl7mvTzXvu3FjGJnqD8znMukH77934u4oRdX0Q3jlp6+TTvXRT3zSPx+49OOfdPcsO/L2z9mN6DU7d+44fiQ/MXr7wpmTzQ6jy2GFPaPN/PObB9HcAOBTaW5RP3j01LnevhU3x6bCmYrR5rp2a2zn3v29g+u+9nffJLWEcDUN8rDkdu/eTcD45tWruJ4w69kIMJ4b14cnxu8SD8wU84lkkggxqbwA6/j8NCGNCQFo4X1UHyE9xMkmYTcRDKLw4WEiwM4GQOFDlZWNIpqd2L41VsTKsnMJarDyHd1ddyYnsG5vj94xu91f/NyX3t71xsFX33S4LbiR3U0evcVwe2Ls4ccfp6Zy+Oy5MgkIQK+QFcVuyNFhWgvcIMl4uDMhzs2bN2vM+jUb1r134r1bF8/0DvRdvXr5ypUrDrv1b/7mb/yz/p999eWGVZ7gcNjZb46hoyUKVncZQ/ntd9+Z9c9JXMEooApzC9n2dvOWHVte//nR7p6ulSvWfuxjH/vWt/6e3q5kBkKH09PTn//jPyXE8MCOh0KmiNvZEI6GDEYNDc8ROsGA7+TJE5FoCP8z3j8AAjxeO2rc5I05s9cAD0kvZDRmLf5DMyWvILaTeayDQplCQjwSZeRAMa0ZuDwXi8uOBPMIEGBYCu/LGYrbgU3EC+EmMtHCVmq/eYGbSnYhwpU0N9rVFwWlDtag6NWw8ALZVghdYHmsVlVvr+qhA+7ly0wDKzykB6g0oTpNil42bHDugD+ivmrgaqjfaO4wZAqdCyqb1uihgwLOTlUZmBE8XKw1AlaFmpxJ0yyUucTDAmJpiWrFYioPS+S58lgK9BIo10dT2oUY315m9a6uN/VNBVXv31gYnY4VKuDmuHHZUdVvNuoi8Rj4ziDJEDCXeBjl1NxDzDUYiaQTY20wBTwLUlCc3di4bAkUcCxQ/GQG4cPYskAtGMWtAl9kZupi/hiakdXRDsYqciuTKZw5eY7US7g0tk0G8Mhyib1JgBgFmkCpEQRndCiOXJH0Wa5kIQlLhwhM86lMuGLVKVYEigGhPkWASY4SHBkKReBigTBQDCEGjzKJXWSIp5KMzeq0p0pFQmBaq2NyakpjtHUvW7V1515vU+err78D7FoqBoBQpRhBTOBPsUMFGEHock6nC4QqaIABI52MJt2Cf05v0e+8b+8DD9/b0Kgaux0G3o4cLSNCJRLV4sFgJrHnCPWnU1RuAwphtVlIe41Gw2B7NbY202Xy7OXzkhyIPo0Og2qAtidkKTxdDd/nFX4dfiOBeJu51tCyWB6O9wg88Lwgb1dT5DwmVT7/0vh4pq+/4PWsTCUnqX9BA0S/UaLhCkkigPiqtgxWP4nS7Z2qcABRF0W1r1bJ8VDoWDbI/yZuoesPDnF9y11JJ8AC5lLyW8aKg0IitHj/YTwyJmxZbic4U7K7VKSfYNmxMBL4Rm2BZBBdyhMp8plwrrAZNUsFwSLK+aFcAk8achQyKpeA5mQQH2y/2nAk1AarE7tSXB+ybzGnZflLtLhQtq7SdwiNXDzClAVoEJIIzQoaG/eicIB5kU7M0BTyU6xSxW0gt+GmcBP6aUkIRDKiRXOHB7Mz8Rb8m4PVU6YFLG8u/8G+R41S3ESsD7gN4rrBw01RHA/IBYnCUoqFw9liwgdikMnEZ62o/NwklYzhY0Ag86aZ5npuDx3AtLTRcnrGp2L+8JVKnWX50JCrqcM17Y0lFvVOfM/1dORtampgAe7dv+f0ieNTk3exKUFUBIifK2PCxuJJu9VGIlVDazNr0+Z0UYUxMecfWrn85vVhNFzIl53f3daxctlACrZSKpP/bKBKL0U7W/SfukOHj4EKGwpHfHOB69dvoS+3tbYODAywVdFUxsZGkF5sqtaO3t6OromREaATr125nH/tlc6uju7OduheknpymYbmltu3byeSObVd6/E4iS6fp7XI7bHBZf02h+v2jatr1m0yG3Qv/PD5aHDxQ489PHzl0qHD77R4vVvv2Tw8fA2YaEpt8AzArimRePzJx1i9tw8d6R/auHrtZnr+zQdCIGKvXLkKWMdPfepTA6sGh8+d9ftQFjzh4EJvd8+1S+9n03HsxUQsFo9G0KSICIkTDYjeSHRiFvjqJO16UNHI55NdB/WyHyVDXihNvDZwYCFA/lTkMdSDSGDfwPUhIGgJp0m+4Mbir6+ATB2NxagRgl1lQulTr773zq8f+tCHnjr45s9huKBLzc7NAXBKhPjt995dtW7DPU886vcFSPwp5Qq2Ov3I4fPk8KAbEr6S/a5TL4ZDTYbmI0cOr9+47tbNK3gfab4bj0UsZsPxo0f27Nr989ffCE6EDZ3wlrLVBgKKtI5p67YtWzYA/yKsiMWRyyb6+1zszV/7tV+b83PJ1KGD78SCibVD67NbNiPQpZ1sqXTsxNFYPNIz1D11d3px2tcx1JUuxmn07fPNuZsaTp852dvTn8kBU0owPtHdTaeHbHnFUiCwSBhv1dDac4fPaEzSuJNMFPY1hMRECfgQCrRklyv7V5GdMARyffABgLwvk6kckC6v/78PkjGFfcA0ycCBTGUFUF4QE0uAIClFAGIPdnaq9t7reuShDQ5r0OtesjtJk8mUK1mAAjEkZC25mQyIvUvKJhoQrma6eprqqNXXeuo1ZjU2PncC8B2vbKGUA4CNfHQ0QSKrNcWamgo4d119Ip3BtUEhWChaCoTTakOPs2ljMG4bnTbPLEZnQ6Ug4BPqxnqLhRIfcteR3/i9xJmEz5lEIva/sCeWW0wmDGqJrRJ8lNomsimEh8CaWDtxOIu7hST5OkAp6P1N/y5sXpQWVL8M7bOyFWvLgN5gs1upAtbBHgXNimHnyU6KEyawWvRg0RuNOpOZFcA2KNGzEyGA1YOeJm5O8U+Sr5TH78P7SCgYFYvDu4rEhf+xOxAhinYqNpFwf9g1mBhkVjIVmULJBF5liSKoRFlDA6KGycB8anK+Z2D5wOAak9V+7cbdwHsXk/HCwlykQovfOhzLJqcdnAwTyRnowThnE+kk/hESgwwGNn5uYWR41Y5tW+/ZOLiiO5aMnzw2gS+dnOZqJm2mPeASzSiIMtLaiFhzjqAlRaEkoy0VkTJLWBQDA31dvV144K7duIo/QXkURfqqyXvHjkRgQXnY6xj/8jE8XlwE4OfQV4OifR6a6WClGRnSj9BPIqlaWKiMjWY6OxI77ltBx4YlAngmwJvwzyo1t9AW08IPMphK8UplcKXz5lJsHlWDqihp2izsRA45TXiOCE0mkvOVg1VnXP9a+vKnKJoyWPkRGuYqoj7UK5lfEoKkdyZQIJQHgOFD0ZWMA3JHtWEt0VBE0xUBu4T6KYNlB4nihgDOOZxpWHYql1ZkK6cLtcnHNUWLvQKXkig01jq3V3gjo8AFz1SjNSEv0fGAVKWoF1uGk8U5Jf9I/BqaQfoys3WkyWRFR+DxubJMAmSHSJJqV6isdii8gmcV1sDEMA/yDlMrOdAAhIlKQjqN7GQO7iBtiRVmwEespcQdiGSXOFNvJDxGj14sMbaMaDFsdX6LhoJ/p1B2e+wGjSVPJbSMREJi6WgiEJ2MZbTYwaTmXb95R2dUxVJJuw3P7SwPlUnTkB7s6CzIU+RKUZlDqT5GXn9vd0tLy93xcYvFRjMQDGUoMRiM3PfoE++eOLVt5z5aFyRSuQV/oM3rJXgfDYdOnTi+GIl0L0Nhb2R+e3r6KAe6efvuQ488fmP4VjAYHhocIgZ85MgxMpbJEvnIr/96U0tjKpMkAt3e1kL4+dt/9/WPf+KjVpM+F060NTeMj40+8ND9ux68/4Xv/0NucfHvv/dcT08PrmOwhUPh1FNPPhwKBwdXriC0ee36LeZ1Yuw2Kj6OiRUrBkZGbokv1OUKhIMz704jxixllcPTREE4jQI9jS0khNMhYM36TaFYcu269SRFkzR0dxIN5A62b2tjw3Nf/xquM0GHhunnslOLgf/655//s//yOSLW5DuhoJADHIlGwTfB+xtLZIPhJHgm5I0K9cvWwEsj2Rbi1lD2oiJ/ZZE/IAYIUwjng4MXte3gdNlTqQT5z1293dFkIhFM1Bk01lY3vPUbX/9mg8sJ42xocBtNhpkpv8FrhCeiQPb/6rKVa9Z++1t/Hw5Hn3z0Qz3etm+NzYUm/QyFOk2L207nkNsjI4ux0O3x21u2bfzwsx955523mXZxv0Bh+XwoswA02Ex6xutyw22bm1u2bN9+5tzFnbu2U7rG+ra0NmXibYv+osfpYoQvvvACRdJLgFNH8pAHGXB3RseoizVBo5Wls+fO3RkbBeylu7v3yMH30jRPpF5KU7Y12CPh8AOPPDZ6a1TwwEMptuC6j606duwIChlztRBa+OM//dyVyxcLcXQ5YTvAYWZJHCVkA+ofiZaUfoh8kXlk07P1kTlsFaidTVWbTflM2Xj8Ft7CPCt/8kL2uYgBSU4SOSVLgDiSgKSBkAGNS0pYk3KdllbV7r2qbTv6ewe8TY15mxH1F9aK/kacVqfW2HlNJ1gUHmDyFQlKorSelBJV1aaqUjffoNM4if6iJ6ikojtPyyNgySJBPFFJ6ljg6XQxIn1TEuW0ddlCAkDKaKx+coYWpF3t/TvTxZbbU0uzwXp/pDobySeXpJ9tvcVM4isA2oLjCC3GU0QKxYLgCTAlwM6FtxLXUxKJRQAj+RQiI5kT54RYMbg0hRcJ4hDPif9YT9NHvQENBsOWS+nMDmuj22Ly6rR0RcOi4/0UelIePLxKDvHqsAE8YAWgG8FcAQBDqSOCEcJTScWiRY0MAY4oLmWWkMGh8jLr6DNMN5xOFAKGINtCTEg0B9kTsgE4gK4s0svImClm7s7MU5CG7M2rtAv+sLe9f2N335q16wEdv3Nn9tr1mzh8MekrWcoDLDa6GlrowGpH/8OpRvEHKf0anaa5tWl2ZioViGzet6v7gV0HDqybmY1eOHc6HAyATaZEA0Hx4G4AV4mPAMBaSBHLh4mDMoh8RUPB3r7ubTu3oxC/8vKLqK3wRjzKcsC4GTaEhMAn1iN0gAcEEqVTO8yIgdHIiuYORkMqS1s9qpSxLFkGvoqCSYvSSiyu88+Vb96IrRmi/2JbLnPXoKf0E1BXpoSoKShxQqZo7NTmkxnZ0eWKRbJUiMcjoFZlRYLA8yS8KmV6HMq4ar+QfB9sCZEWjE/xRSvzTIiFVeKluOTkFfOP3oBtgteC+kXEvSQy8T/SFo8QYQmBDxJpJAJYxJ7QFgup7Cg+wpWEBxiOh99fazPz+GJYKn4ORsOd+JFvsfcQpdLQAX2FnHvJhKHEEKlD5qBIX+Cgic1IqZnKqraKDSvikuUQxxfUJLflacUg5wGFE/AV0ZxotCPp9UJ1Na8OL7ipmPLKmGEbNeObj+WCwHhCsjojBi7PhUokjm0mSoBWZR55SgYkD8KYAW9BexVDXJabE+TpFIqVe8G81PU2u9eqt4aLi6jGyWiSbNRIdqnerK7qGhuaO1yezmAk5A9M8JxJVD4ivdkModVcOsGlTh5/j6oD4ljLl/XPTk3j1lsIBKjf5QScvZ4G78jIWGNb26Wr1ywOTzq3tGrtptdee0tNHKFURU2OBRcKYNcZRR9uaGrGNJqZnT926ozRYiPmumr1+q98+asTM7ME6sCJpJiYEpTDr7wST5DKgWllA/Wt2evubm99/ac/efCRh+/Zuv7azRt/+J9+72evv0ZYC8Prz7/0f996/zJNexqbem5cu047YX8wjoUMoAR2530PPuDq6vmHL38ZfXb5QB8+c5jCg48+cvzo0Sef+VUcZeT6EunyL4Q//PSzWNI/fPHl906ebW7tYGwms/XypUser/fMieOyoMX88KX3q4WM227t6epsfvD+n7/5zx1trVcvBw6/+05zS2MoFMRwJ5cbARxPJJB/sJk0WJCsHR4a6kqElwj7ESJVCBTSgWz4E1Jh08mnsj+gRFnN2iuhT6iJ+5PykV3q7O944oknbozcPn/pfexyuln09PVDJ7duDLe2tkTCQSwMg1WTT+Q0Tksplv7hPz3/a5/5DbLG6MHY1NL21utvQR06Wq3UqzOJWMVQ77J67k7dXYgtzC/6YzG0q+YGr7uzrTU4PxsEi8zvJ3cUXLC2VgAfcgyJFUERoTfk0VOngPpiZnC8B2Yn2xocJEsT4z9+5D0inVqN2eSGQ4VPnjg2NjqKtYHLkecjwBMMxc5fPNvkbTE4gCyt2DzWRAaUTV3v8t7jp44SIWOO6t3acnbpuef+ob+/LxaLdrS1370x9/JPXvrMZz/z6k9+PD8ZFb0fWwPT0ghnxoyrULiJ54kIKttAJpm9R5wD2HdlemUylV3xS0b0yxcy0crBCcJt2KxKyNOgIUxLzheRL2ImoqkPDqhWrKrHW7Rhc2tHN0IoXalEy2UKvMhLx36UoBPyDooiviC4oqJiSfAT7RkLuFqhLh8XrE2RvsgBIMILQDln4/Ec3a+pVQVBvYh5itJo5imwuxBiiYLhzmIiEjWXql1W2+a5WOfkXN3IVHEuCF62paJzmy06+DKxCXDQkL7JTIQO8SJ0Yeh4BiQ1FZyNHL4cGLI8nqLgiX+Rp8U/B39kYdBcSDrGIoJxC18CIc1M1D2ezGLpkcZldjobvE1OpwenLlxJWhnD6wvIjhIeOKMG0QtMMs4RNXHOIj1gkKZyh7IN0D0hbWkGyB2QHjLPWHpE9xSVibsrMphFE/KXMdWkr3BmGbHCG8nh0dGHFJPcYHWogEZW62CoC4Fo/7r1O/beiyvx4MFTI7fvdHX2ee2td2+SFYyQNNPSEFcInC1RSaL8YqmRaE1oAzJeDM3SKGf3I/s//tEHg4vp0EJidPiSb24K1zqmPKEQHOp2o5ldhyMaRooDHPqAkiV9il6uZuPg8r5sNgNMOm2t9aQIarX+i+cRW+QHKk+iaILQGU8sj8KzIIVJ1BE1BLEkyh0ihPChJF6iiaiBaFHsU2IVZW0ouGS3VO6OZa9eWdi1G3jehWIhpiFeCb3TgUg4huhKyAAiCirw5SyllnYLlYjouXMzQq/SvB3HkFy6dgMZyC8PJl1OUg5hMcoq1eYfq1JoRf4QZ7GsAXo+xryYhZIPjAAiUMplYT2kVrHjsLkVgciDcg4ql2SDI495Eyo000oH3JZsRnAsJCFL3EriBhWYL9GXma/a6gvpSQoqKfgMm7frUeUF+tRAZjW2onwdbRa+w9UlZQHmCVmhWbK6FK0JqC+5V3gbJXNKRCJ7kLMqFQqiat9SxDkDgMzRJNiFwjCUPAxELU9GajFtszVSsSN2NKQqhCMLiCaCK5LgP+4x8a6j+kp1hrJqIo/5HJqGgbAuPBYbhwYTgCbogaUx2hOaZKoYJ7YkQBNFdKx0vclBwqpr2XJ7xJUrxovlRd/UrS5PQ5S+gYUc/AfBnoxFKI8Hp8TjdATntYjGeCwmeV6VulAkSlIxI2SefTP+nFq3kCx09C3fuPmek4feRUlhYBxYtwSXiFbqXA0UVgwODVod3qtXr3/9G9/5oz/647377zt69Cjzywoxv0jZ6YnJJ+996u+/9510NsXkFrPS9L6jtWn05tXdTz78QGdLIhaBp1y9erWlvQNFfeXqDV/4y7/CRfz0r/xHIs3vvPMqFsNCPLNvz+75xTCJSaBgEn2kOJg8EXAhsNtAXLoxOUve7+S0j6aETpudhrg4pO6//34AOOkCQCi3tb1tbftaAnKzE2PkIpm11fm5KevGof6B1QffeYtAOFA4drOho6UxGomsHlp19uwZeD2qCRptMBKh5MNkIzmQTHlJxhHqEpIW7RJUdW2dNGMQi0v5YRsKnXxA6zUnkRCfQo3omBLdQF0gsgsjoFp6bGKSNjr9y5eb7bbLV65h7yyMTNqbHPlUKbwU9jS4ItVEMZr29LXXUeCt1d+zYdPVy8OTt0YXfPPxRIzmMhiIgDm0tLZu2rHlwrVLUxPjWgvJdAvoukhdVv93fvs3v/vNb/hmZ8JBkpaTS7is4eMqVcSfuXD23N777//Gd787Mzc3tGLFM888892v/T/RSMhp1GVT6UaPd9YXtFp0DV7v9BXf84HnXQ5g9fJ4LPEiGkC2d6jDk/FIJF7NqVq6G/N1GbvHHl2MW9T2Dz/z4bHbYxdPnAHpm5oHr9fz2KMPA3dMLdz67Ss3rlv7J3/6n+fmxk/mj2IoYs8ZLCoaQuIdVVDdpOCK7cm2U3acJPOIOatsbIX9yfvKHNc4e+3lv/kN0hxpUSRbUluKBymN2kSg2WRS7dtn3LlzYN2GFqezoDfGNdpFAoj19TkSskTjVtxOaATidUO4adA4YZFwVj40UQSrUWPyWuqRrESucPsCZgRaqQCWpahpydNuWYwjxBLslcJcI+WT6dRSeql+MWer6hvKlrZ4HFXJGUtXgnFNKG0qG23U8SP3uRZVfOh7JBfTEl2alHJfElvAW0HTF9YKCTERsGKRZRLgU4hRmQ1OhNAq2DGSoaI3sed5ZH6gzwKNukgtcroam1otVjvmX2AhDN/EQ45+TAoSnjHCwCazDvOX1AWcO7jJJM2KqCozSLoNzm/RMOFqSo6M4oGFLzAcKf0U7ii6qCgtFDfL3wwOg0YYnQgL5R9RikhYJPu1ChRMyk4CR0vrnfEptPxnPvYf9Q73mVNXxkYnwMQ26ExXL9zSARNjbsArKUm4RNbpelhH/awUwmjt+kh83kTZnMfc0tzIAAf6W4kQhBZnQejDJ9/osoEflw7H6peKhioNZmKkTWHeMTBmRpo8kHCAlpQrEJK7efnq7NwU5Z1MYCGBPbikstgUCxj5xUyL5MPWBdsVeQJVVPFPMsfEsJkthDl3ghLAEIFNc1GkIPIFyYB0Q8EAXDMcUgWd1dGboS0bOy1mr5o+SCpQWFguMahlgriwLC96FPX5SYdb29HtKpfSc7MZMfJAmAL4ExGnHHLmLw5lP8hmqE20/BY7WPkYXUxEL99hzIpZBxXhRhLbEjnMebwveUw8BEIYCxHnj8gjCEnhWaIaI37w+ikHu8MAdHihTFG/Rp0W5QTGJhaw7E0uwlbhfgUZgJiZQr5ipvBCxDdyjXUFQQwJx/bm4FsklYgArhmkCmdV9Fb5lK1QLKJ6U3eEcxo5zi2knApdRJHakh3JdWQI7DksIMYhSopkhYhhKy418YTz/PxdmzrGLG4avidNZAWUj6I4Zhh9Qtn7JE0wUlYYBiQHa4gJLUqjTmd2WfV6mzRmUoBm2EUk3RLc0VkciaLq7uSs0zNN5iGi1GLrm58cZjs3uF00B8T9RCQJewgtgLA2neDAfZe6BLW6qbEFQRONxHAydXZ2kvvQaHAkK+qzV4d37X1gy9YdP/vxy+0uBx1FIUpycNRUHKXSaCeRaOLEqbNP//4f+udDvX2ZkdExnnLDBkk4JGXj/PsXCBDynPHUinw+Gwwmtm7Z0tvWKdpyufjiSz9+9TvfQhOPp1P+QMBmd09Nz05fv9W198BSVRMIJpq37YvfuHzPnvv889N0E3c3tYcC828dPOx22leuHAwsLhw/eYqWCFqTdfu9D7z9zjv5en1Fb6LiaP2atWNj45ev3vjEZz6Dw2Nq9A6G7N3REa/TduvmTSJaNy9dQORbdKqJW8Pbtm4ZHxtjFpoavAAlkqm0cf2Gs2fPMvswpvn5eVguhKUzEJapwMTQr+ApskdkfVEyoG/hLjAeEbrKoVAuJ8nBZ7V/yePhJUon4XuY0+xsxACKvnkJdBGibuhUxNRH74zt3Lef6LjR64DReBqdKfSjaLSYwlOkCvvm9VbrzYtXdu3cS4uPn73yuiqTN9VTR0sR7ZKrq7mzqx22TRbV9r17Ll45z1MfPXqoEEnkk4kLZ89sWr/+/PkLqaRgjsbDWWxNBqyzqd58841kNvf444+/c+jQ0bd//ulP/iqKy1uvvWJxO1nrYhKQjEomGSQLkl4O+Shqb4apoBkJ+MH42PC9Zd0lj8cRnIlnC3SQiVWTuHwMRNFu3LoWXYwxV0hrIKD88wtf/epXGBWTf/fuOLrj8RPvWax6eLzBXG81m+GjDZ5Gny8Q1caIzdXXsyvpzK0lkAYEBLPNwQTX9ogyqzLhss0J0CibmRe192Xm5Y+qNPKWbMh0nsxWlcrtVq1bq1s11LRv31Brm85sTpcrEaMBfUKqTwtLaYvVwl3ELaXCB8yGAHRWjyXIXCGIMZnIdtZo7Cp6GZWlR3smEqsWpfA9nUqRQ0o0lFI6Yj1g/WOxg6XHJs/TQzBZyWaIKluKupV2z9pi2nr3TmByrlhPgzutLaXO2Nx2nKIIi3w2UyKJTXKoxcpkJFgg3FkclPBEzAIl7VTxF8pAmYFfTgUvgEUQySRVFtgY5AAR54TTcSKojE6bw0PClQCQZZcyacAzMiWANbCw8VET1gKQ3UhnYpgZdjLqj/BrOD7BTUI0wuLITcwRj5NUJPRIDA5xLzILOBcVhi+SFl6MYsD7SGMYt6wLw1dkMKxHjDDZJMlsNpEpetnUkeStu5d/5eOf3Lxl+2tvHLw9froIVsVSfSGULgE3UtbYzI5cVlqVw3CAo6kaJBOdpknpQkJfp2nubCBQTa7J2tVD46O33794JhaeA8OnrpLjbplUPB4OM0JMFlK+wM+UGlCSv5VyZYyyDK0ISe5Nxa5duoAwIuxHw0LJnUvEcTVgDonwkfmFnyuST9gxYkYCGmK0IT1xSygrwqwwLUtAsROzZOrQ1hA/NYlAro+m3srdY+Hq5ET04oXRrVtxuQhiMCeIB5hoR22OUGRkDvlyHr21scmmrjpu3xxLRskwwlo20oy5Ju1+se4yOg52BQYepMIoWCoJ4yjyWNiOUK5cUiQc0y8/kpEMMyKSKpuqXCEBUl6T1C26H6sml699C4lWUuE1ZjYgDdl+3A0BLd7gkmK/KjyutjlZaaQjEhcMDMWYFPLAtkUv5O6Ys5I+LGxRhsH1MV+4UZqOkbW0Z95kS/P0zIYIb67GHcX9Ujt4wQAQqPiIeIFIVTY+5CWjkgFjR/OYXIVDuSlfxC8EeBNjRrDyNveXl5J5JSKWE2rPQtEZr+HnvEl0gHnj4DVSH2WL0nIStPRWC0I9hfecJE3p5s1X1PiX2gdWBJP1gWju5shtdNjB5W1Wr/vcsTcAi26mBaGasgUg2lOVIvnRLiS9winyFqMlEUXuEvSmtzaJGCZiuoRkdu/Z/5O3Dm3YsOX0mfMf+8Qnenv787n0Uiq/vLuLAjs8+9jBQEZTuE/HzfjoaGNzq9EAfnL11Vdf//zn/guucRr34rwNRyLgpb57+BDhNRIlbt++GZ6doVKfsuD+3q7pqQl1QJAsmEakwvTk7LWrw3qDo6W1487Y9NihY28deruzv2Vh0b9p256me3ZO/OgHV4dvAAAy6/MRnSUN297YTB3Gt5/7wco16+7dtgNo3e9/+Uvnz5/fvXtv74B77OYNAP8eefC+TQfuf+Ofnj9++F28+0t6LYlsDz94/6kTx29fv/yD71c5554tW0+dOEZtNF76V155BVJFKEK1xO/wbZDiBv2RT8OfxPTFDyDuTA2aNJhUgN5AwcygyGEhrBqlCHUJCSt/1nYx7yg/YhKAlZmkvwysBVeYVkfKpWTr1dWdOvSuqaGhq6Xl9oXL+WDO6ACFzUhyP/01ktk87YFjvoXxa7fDM35VOE76ajlbkA2xVIXNnjh1qhyPWwbaf/cPf9fltZ48dYykd3NrAz6SV390UEpWyCtMqZo6XWkNifQCAEdXqOBC6q1/fuORjzz953/+53/2X/70L/7iL3ZtXU/uD31h2EqlQoWAW2AuAx61s8FVtVfpQdG1ojseDeKCcrpFUdCb0COzSCUywa12C4kGg32DAAv/4X/+o699+Wv0GEhRv6A8u7fdfed6gDYEjY1OgvGr1w795Kc/As6yqUlnMBlcXpfT5fD5/PAQ6QQAy6NalfoFCo9ELxc1Vdk+sgeZXLa8SFnlqE06n/KXMu/yG/6HeYI0AS+lwUFvRO2ade3btw0MDXnLpUVPIwnk5SwwDksZ/sc/ZKcNJwlUgEHXCV6N5I8IB8B6JoXYRANevM1qGvdKzX0J5P5CrpQN4X1Sg8TEYoIGwIZGvcSpm2bf0vxsiZzwUjxG6gW+Tku9aUUuP3TlrHqe1p4VW86gDsdDS/UJs9MdyccQlaVsnvoy8IPUeABp9k7zB/yRCqQjUQDqJaTelypaomBIPJ5P4aki8oTuxNoSxZz6GfrJUmUDAgUqAPkqGj3JTRZQoCx2MGSCoUQqkyWzxKDHICE+hcPfiO0rwS9cBuK6KpEbqnjhYRVSFUaeH849Ap44BpnVWhIpnB5Oq3Bz4VEMBrErYkTIXUnJEdHLO3IIN1P2BnIUlocSQM7UjM8/tGbzZ3//0UAw9o3vfj+BDpOqxMOoILT1c5l1xNXwhxu9zY1YCIh9iK3CsOrq8zgAdGp7k3P/gW2A2SD9JyZHgyGf3WYAznZ+JonFhrSwaLUemyWTSsTmgXomIcRQAYKfVdfUA69PSxKyHmgeA9w+80asPR0OwtORo6JbiTmYhSWjW8vDiV8UdiseB6Qmhf8m+qNRrCmKOPNHwZBSGKM8qhLJ5WvC6oUJoHmDtE/aWziSmfdlT57wr1y53aEjuMuWxsWNYs4EMj/oiKKg4JEWpAnkj93QqAYO4W46zqJjcxuhLsbFeP7NIRMtwkmYC4JJYSYybNEK5D0ohUVBsski8ClyB5sOVqZUX9WkGu4AOBq35wsQfo1bKcNR9h0Ewv4SwsZiFhkvDAV1hPWQIUN+ynrLateDTEMwVXqSiN2hCFdIhR/uyNZCuWEMSH30dwFRVbA75CJydm3ChWKYQtDWCHyTXs4817a/CGjcThKNlXAu32IzcFk8+XBsRKzMoewNOeRTcVvR+pSKdW7A+/J1tHmsbjRdmAxnMxvIYOW7te/JojFCxaxns6Eo6EBJNFptsbQgT5WkQD0ntriGLgVA1tjoyLtu+6Pdy/v8i4lLl84mE759e4ZoBcK+CYdDgiAXzzR6PABn2agvymW4CApwZ1unPAtFbpSxl6uzfp9BZ4QRnDp+isHMTc+BynryyNED992XnJu+c+Py6tWrScqdj9LrT2hg7969b/wz/drfSyRScEM0eCqafvSjHz351BMdpDpP3sHRWihmBDQ4l+ru6iLt9/qZcy1NjaNjI9h8gM+j5IOOuWvvvoee/fgbr7x26NB7x0+973I2rhha9U8/eolaPxJEnvjw0xhwhamZFStX4Zp22M0jt26SwwxuajAc/Yv//ldf+h//8xOf+szN0TFPU9P+fQcunjwyMjJCMZXb5W3wuK9fv06aN6p7d1dbJBiimhnN98ihg2T8gYdKnbHF5Tn63iHgOXE+W01GJwnJSn6HyWLBRYPoxYCgZIg3CYsFwll0f/JZqJ0o6SkghWPlQYIRiv+F6P1gCZV/hDCVD/gQTxQHzILVDQSy9STjmoGMXdJbxV1H8YO7wRuJJQEAJN1p3baN85OT1BtTyetw6xLpPPxR4NIq1YmRMT/NBb0NCcocyxWHwwYKv6REWA1lmw5T7OTpU6SYQpPQEJelR6TBxh3gkGD7QmYlfEg4Z+C5wZmUxoUXrvrzF1/ZvHUbvRNOHz9Co8loKO6x68NhWqES5c3YnCCCWCPhlNPhymq19Of4wn/7/O/+3m+RC+Z0WpuaGqlVA6wfcESsPvTdO3fHB7qXHT58GL4Jwggpq3RtBJR3aips9qq+9MW//uIX/9bnmwXLkf53kDITm0wlFhbnAQAmg5rtgAsKF66eyh/QPmqMQpR7BQ62tjeU6ZVNUuNwyp//zi8SVDWqxgbt1q3L9+ynLq/J40bdj1H/ajHjC2dWSVwzguuDfEe2IW4FmFhd1hsw8sjFEoBCVC+AQ1Qqu6rOTA0SoihJfUgslo6lqnFsnXqljEUQI5REEJa3Sn5QXb0BvT+dqUYTAuHT4Lbb3X1v/HM0VmqL50xxEIdMWmuTq0hFFj2gs1IerSLaRVwnt0RWPQEP+guV1IBNM6MKkLJ4KIXNMTvoJSIHJLNL8XPKk3NbwGHU+ERJUpB+u8q+duGd8DTSI5KmuTTipLUXoD0YgfhdMfSsDrKcVWbpwgIoB7SBgYdDT1JRpbQV57IyvRAxK8I66OqBP8JERc8QdogIRgyzCvirlTPFoalYBUocWihdiBDKQwoIAQqTFsuYfTC0etXe+x6mEOunr745Oj5NGDi0GK+rmLRqhJmmtbG5XARyy5CMpqO5GAkiiF6K1Qqy5+qsDmvnQN+u3VtWLzfdGfP7pqcW5/34HuCm5HATK9ZUDNlkjOdF90HQIDvY73l8rWidGg22EJCW4YUAAphaIMrLiA3D08nKkYITHlbZqliI/wsHFGRU3db9AAAAAABJRU5ErkJggg==\",\n \"text/plain\": [\n \"\"\n ]\n },\n \"execution_count\": 16,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"image\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"base_uri\": \"https://localhost:8080/\"\n },\n \"id\": \"SG64xtCtv24-\",\n \"outputId\": \"cdc347b4-cdf2-49c8-e308-71dc99f27476\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"[\\\"The presentation of this meal can influence one's eating experience by making it more visually appealing and appetizing. The vibrant colors of the fruits, vegetables, and bread, as well as the careful arrangement of the food in the colorful containers, can create a sense of variety and abundance. This can make the meal more enjoyable and satisfying to eat, encouraging individuals to eat more and feel more satisfied after eating. Additionally, the presentation can make the meal more inviting and appealing to others, increasing the likelihood that they will eat it.\\\"]\\n\"\n ]\n }\n ],\n \"source\": [\n \"inputs = processor.apply_chat_template(\\n\",\n \" messages,\\n\",\n \" add_generation_prompt=True,\\n\",\n \" tokenize=True,\\n\",\n \" return_tensors=\\\"pt\\\"\\n\",\n \" return_dict=True,\\n\",\n \").to(model.device)\\n\",\n \"\\n\",\n \"# Inference: Generation of the output\\n\",\n \"generated_ids = model.generate(**inputs, max_new_tokens=128)\\n\",\n \"generated_ids_trimmed = [\\n\",\n \" out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\\n\",\n \"]\\n\",\n \"output_text = processor.batch_decode(\\n\",\n \" generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\\n\",\n \")\\n\",\n \"print(output_text)\"\n ]\n }\n ],\n \"metadata\": {\n \"accelerator\": \"GPU\",\n \"colab\": {\n \"gpuType\": \"T4\",\n \"provenance\": []\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 0\n}\n" }, { "path": "examples/notebooks/sft_tool_calling.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"ii5Zkit6eSqU\"\n },\n \"source\": [\n \"# Teaching Tool Calling with Supervised Fine-Tuning (SFT) using TRL on a Free Colab Notebook\\n\",\n \"\\n\",\n \"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_tool_calling.ipynb)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"gJVcVKOteSqV\"\n },\n \"source\": [\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"hzt0BrvoeSqW\"\n },\n \"source\": [\n \"Learn how to teach a language model to perform **tool calling** using **Supervised Fine-Tuning (SFT)** with **LoRA/QLoRA** and the [**TRL**](https://github.com/huggingface/trl) library.\\n\",\n \"\\n\",\n \"The model used in this notebook does not have native tool-calling support. We extend its Jinja2 chat template (via `tiny_aya_chat_template.jinja`) to serialize tool schemas into the system preamble and render tool calls as structured `` XML inside the model's native `<|START_RESPONSE|>` / `<|END_RESPONSE|>` delimiters. The modified template is saved with the tokenizer, making inference reproducible: just load the tokenizer from the output directory and call `apply_chat_template` with `tools=TOOLS`.\\n\",\n \"\\n\",\n \"- [TRL GitHub Repository](https://github.com/huggingface/trl) โ€” star us to support the project!\\n\",\n \"- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview)\\n\",\n \"- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"3PfX1aj5eSqW\"\n },\n \"source\": [\n \"## Key concepts\\n\",\n \"\\n\",\n \"- **SFT**: Trains a model on example input-output pairs to align its behavior with a desired task.\\n\",\n \"- **Tool Calling**: The ability of a model to respond with a structured function call instead of free-form text.\\n\",\n \"- **LoRA**: Updates only a small set of low-rank parameters, reducing training cost and memory usage.\\n\",\n \"- **QLoRA**: A quantized variant of LoRA that enables fine-tuning larger models on limited hardware.\\n\",\n \"- **TRL**: The Hugging Face library that makes fine-tuning and reinforcement learning simple and efficient.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"QDMcKeoEeSqW\"\n },\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"We'll install **TRL** with the **PEFT** extra, which brings in all main dependencies such as **Transformers** and **PEFT** (parameter-efficient fine-tuning). We also install **trackio** for experiment logging, and **bitsandbytes** for 4-bit quantization,\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"Ey-TuYPrXTLG\",\n \"outputId\": \"a4fd8cfe-624e-4185-ab59-e6901514cb96\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"\\u001b[2K \\u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\\u001b[0m \\u001b[32m1.0/1.0 MB\\u001b[0m \\u001b[31m17.6 MB/s\\u001b[0m eta \\u001b[36m0:00:00\\u001b[0m\\n\",\n \"\\u001b[2K \\u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\\u001b[0m \\u001b[32m60.7/60.7 MB\\u001b[0m \\u001b[31m42.6 MB/s\\u001b[0m eta \\u001b[36m0:00:00\\u001b[0m\\n\",\n \"\\u001b[2K \\u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\\u001b[0m \\u001b[32m24.2/24.2 MB\\u001b[0m \\u001b[31m109.6 MB/s\\u001b[0m eta \\u001b[36m0:00:00\\u001b[0m\\n\",\n \"\\u001b[2K \\u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\\u001b[0m \\u001b[32m56.0/56.0 kB\\u001b[0m \\u001b[31m6.5 MB/s\\u001b[0m eta \\u001b[36m0:00:00\\u001b[0m\\n\",\n \"\\u001b[2K \\u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\\u001b[0m \\u001b[32m9.9/9.9 MB\\u001b[0m \\u001b[31m131.7 MB/s\\u001b[0m eta \\u001b[36m0:00:00\\u001b[0m\\n\",\n \"\\u001b[2K \\u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\\u001b[0m \\u001b[32m540.5/540.5 kB\\u001b[0m \\u001b[31m44.6 MB/s\\u001b[0m eta \\u001b[36m0:00:00\\u001b[0m\\n\",\n \"\\u001b[?25h\"\n ]\n }\n ],\n \"source\": [\n \"!pip install -Uq \\\"trl[peft]\\\" trackio bitsandbytes liger-kernel\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Aw8_T-Z0eSqW\"\n },\n \"source\": [\n \"### Log in to Hugging Face\\n\",\n \"\\n\",\n \"Log in to your Hugging Face account to push the fine-tuned model to the Hub and access gated models. You can find your access token on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"_qaeDZwXXTLG\"\n },\n \"outputs\": [],\n \"source\": [\n \"from huggingface_hub import notebook_login\\n\",\n \"\\n\",\n \"notebook_login()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"XPnDpJgIeSqX\"\n },\n \"source\": [\n \"## Load Dataset\\n\",\n \"\\n\",\n \"We load the [**bebechien/SimpleToolCalling**](https://huggingface.co/datasets/bebechien/SimpleToolCalling) dataset, which contains user queries paired with the correct tool call to handle each request. Each sample provides a `user_content`, a `tool_name`, and `tool_arguments`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"zfJY_8AzXTLG\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_name = \\\"bebechien/SimpleToolCalling\\\"\\n\",\n \"dataset = load_dataset(dataset_name, split=\\\"train\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"ygeMXzKGXTLH\",\n \"outputId\": \"a1ed3a8b-f515-4cda-eeb2-db0355ed2c02\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"Dataset({\\n\",\n \" features: ['user_content', 'tool_name', 'tool_arguments'],\\n\",\n \" num_rows: 40\\n\",\n \"})\"\n ]\n },\n \"execution_count\": 2,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"dataset\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"O_GkvqtReSqX\"\n },\n \"source\": [\n \"## Prepare Tool-Calling Data\\n\",\n \"\\n\",\n \"We define two tools: `search_knowledge_base` for internal company documents and `search_google` for public information. We then write a custom Jinja2 chat template that extends the model's default template with two additions:\\n\",\n \"\\n\",\n \"1. A **Tool Use** section is appended to the system preamble when `tools` is passed to `apply_chat_template`.\\n\",\n \"2. Assistant turns with `tool_calls` render the call as structured `` inside the model's existing `<|START_RESPONSE|>` / `<|END_RESPONSE|>` delimiters.\\n\",\n \"\\n\",\n \"Each training sample uses the standard `tool_calls` message format with a `tools` key โ€” SFTTrainer passes these to `apply_chat_template` automatically.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"jaAgXeWtXTLH\"\n },\n \"outputs\": [],\n \"source\": [\n \"import json\\n\",\n \"\\n\",\n \"# These are the tool schemas that are used in the dataset\\n\",\n \"TOOLS = [\\n\",\n \" {\\n\",\n \" \\\"type\\\": \\\"function\\\",\\n\",\n \" \\\"function\\\": {\\n\",\n \" \\\"name\\\": \\\"search_knowledge_base\\\",\\n\",\n \" \\\"description\\\": \\\"Search internal company documents, policies and project data.\\\",\\n\",\n \" \\\"parameters\\\": {\\n\",\n \" \\\"type\\\": \\\"object\\\",\\n\",\n \" \\\"properties\\\": {\\\"query\\\": {\\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"query string\\\"}},\\n\",\n \" \\\"required\\\": [\\\"query\\\"],\\n\",\n \" },\\n\",\n \" \\\"return\\\": {\\\"type\\\": \\\"string\\\"},\\n\",\n \" },\\n\",\n \" },\\n\",\n \" {\\n\",\n \" \\\"type\\\": \\\"function\\\",\\n\",\n \" \\\"function\\\": {\\n\",\n \" \\\"name\\\": \\\"search_google\\\",\\n\",\n \" \\\"description\\\": \\\"Search public information.\\\",\\n\",\n \" \\\"parameters\\\": {\\n\",\n \" \\\"type\\\": \\\"object\\\",\\n\",\n \" \\\"properties\\\": {\\\"query\\\": {\\\"type\\\": \\\"string\\\", \\\"description\\\": \\\"query string\\\"}},\\n\",\n \" \\\"required\\\": [\\\"query\\\"],\\n\",\n \" },\\n\",\n \" \\\"return\\\": {\\\"type\\\": \\\"string\\\"},\\n\",\n \" },\\n\",\n \" },\\n\",\n \"]\\n\",\n \"\\n\",\n \"def create_conversation(sample):\\n\",\n \" return {\\n\",\n \" \\\"prompt\\\": [{\\\"role\\\": \\\"user\\\", \\\"content\\\": sample[\\\"user_content\\\"]}],\\n\",\n \" \\\"completion\\\": [\\n\",\n \" {\\n\",\n \" \\\"role\\\": \\\"assistant\\\",\\n\",\n \" \\\"tool_calls\\\": [\\n\",\n \" {\\n\",\n \" \\\"type\\\": \\\"function\\\",\\n\",\n \" \\\"function\\\": {\\n\",\n \" \\\"name\\\": sample[\\\"tool_name\\\"],\\n\",\n \" \\\"arguments\\\": json.loads(sample[\\\"tool_arguments\\\"]),\\n\",\n \" },\\n\",\n \" }\\n\",\n \" ],\\n\",\n \" },\\n\",\n \" ],\\n\",\n \" \\\"tools\\\": TOOLS,\\n\",\n \" }\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"32p512R2XTLH\"\n },\n \"outputs\": [],\n \"source\": [\n \"dataset = dataset.map(create_conversation, remove_columns=dataset.features)\\n\",\n \"\\n\",\n \"# Split dataset into 50% training samples and 50% test samples\\n\",\n \"dataset = dataset.train_test_split(test_size=0.5, shuffle=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Plnjef-PeSqX\"\n },\n \"source\": [\n \"Let's inspect an example from the training set to verify the format:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"f4QI6wJjXTLH\",\n \"outputId\": \"2156adb4-7bed-4e29-84c5-54e6d45e5500\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"{'messages': [{'content': 'How do I configure the VPN for the New York office?',\\n\",\n \" 'role': 'user',\\n\",\n \" 'tool_calls': None},\\n\",\n \" {'content': None,\\n\",\n \" 'role': 'assistant',\\n\",\n \" 'tool_calls': [{'function': {'arguments': {'query': 'VPN configuration guide New York office'},\\n\",\n \" 'name': 'search_knowledge_base'},\\n\",\n \" 'type': 'function'}]}],\\n\",\n \" 'tools': [{'function': {'description': 'Search internal company documents, policies and project data.',\\n\",\n \" 'name': 'search_knowledge_base',\\n\",\n \" 'parameters': {'properties': {'query': {'description': 'query string',\\n\",\n \" 'type': 'string'}},\\n\",\n \" 'required': ['query'],\\n\",\n \" 'type': 'object'},\\n\",\n \" 'return': {'type': 'string'}},\\n\",\n \" 'type': 'function'},\\n\",\n \" {'function': {'description': 'Search public information.',\\n\",\n \" 'name': 'search_google',\\n\",\n \" 'parameters': {'properties': {'query': {'description': 'query string',\\n\",\n \" 'type': 'string'}},\\n\",\n \" 'required': ['query'],\\n\",\n \" 'type': 'object'},\\n\",\n \" 'return': {'type': 'string'}},\\n\",\n \" 'type': 'function'}]}\"\n ]\n },\n \"execution_count\": 5,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"dataset['train'][0]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"fBIGKl_UXTLH\",\n \"outputId\": \"edd8e968-c7e4-418d-b9e9-26773aee1366\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"DatasetDict({\\n\",\n \" train: Dataset({\\n\",\n \" features: ['messages', 'tools'],\\n\",\n \" num_rows: 20\\n\",\n \" })\\n\",\n \" test: Dataset({\\n\",\n \" features: ['messages', 'tools'],\\n\",\n \" num_rows: 20\\n\",\n \" })\\n\",\n \"})\"\n ]\n },\n \"execution_count\": 6,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"dataset\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"aud6U3c2eSqX\"\n },\n \"source\": [\n \"## Load Model and Configure LoRA/QLoRA\\n\",\n \"\\n\",\n \"Choose the model you want to fine-tune. This notebook uses [`CohereLabs/tiny-aya-global`](https://huggingface.co/CohereLabs/tiny-aya-global) by default.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"_j_LF12IXTLH\"\n },\n \"outputs\": [],\n \"source\": [\n \"model_id, output_dir = \\\"CohereLabs/tiny-aya-global\\\", \\\"tiny-aya-global-SFT\\\" # โœ… ~9.1 GB VRAM\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"gpTZHjpJeSqX\"\n },\n \"source\": [\n \"Load the model with 4-bit quantization using `BitsAndBytesConfig` (QLoRA). To use standard LoRA without quantization, comment out the `quantization_config` parameter. We also load the tokenizer separately so we can install the custom chat template before training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"680888237b78477ea653adb2ecea7fa8\"\n ]\n },\n \"id\": \"jGpTDV6sXTLH\",\n \"outputId\": \"fc33f7a6-bfd0-4228-80cd-e0aeb67bbd42\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"680888237b78477ea653adb2ecea7fa8\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Loading weights: 0%| | 0/290 [00:00\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* GPU detected, enabling automatic GPU metrics logging\\n\",\n \"* Created new run: sergiopaniego-1771428231\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"\\n\",\n \"

\\n\",\n \" \\n\",\n \" \\n\",\n \" [15/15 00:52, Epoch 3/3]\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \"
StepTraining Loss
13.095131
23.083373
32.951535
42.625918
52.254464
61.939976
71.694891
81.558982
91.430660
101.305176
111.192725
121.120383
131.052859
140.985858
150.970833

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Run finished. Uploading logs to Trackio (please wait...)\\n\"\n ]\n }\n ],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"4MGKFi1-eSqY\"\n },\n \"source\": [\n \"Show memory stats after training:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"3f68GA6TXTLI\",\n \"outputId\": \"321e90ee-757a-41fc-c6a2-4ba40a6e6b3c\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"59.2841 seconds used for training.\\n\",\n \"0.99 minutes used for training.\\n\",\n \"Peak reserved memory = 11.928 GB.\\n\",\n \"Peak reserved memory for training = 7.28 GB.\\n\",\n \"Peak reserved memory % of max memory = 30.202 %.\\n\",\n \"Peak reserved memory for training % of max memory = 18.433 %.\\n\"\n ]\n }\n ],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"ONWy4NOAeSqY\"\n },\n \"source\": [\n \"## Save the Fine-Tuned Model\\n\",\n \"\\n\",\n \"Save the trained LoRA adapter locally and push it to the Hugging Face Hub.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"4951424bb90e4dbbaea8c9b88c592872\",\n \"1669afd0e52443d090adab0fbe663c66\",\n \"5ee5f6b74e7246eea99c0d84c2a27bc0\",\n \"15b4a13592c14102af0d3f8a999f3d36\",\n \"7216a7d56c364a0d92e079d9848946d3\",\n \"83e8f73e00004b718cbba7be0ecc45e1\",\n \"05db21352f614288864f88c1ba794ee9\",\n \"d7821b8cd21f4fb78237479ff081511b\",\n \"00c9f462a4584b22b1e38dfcc5f86af3\",\n \"22cacea841ba48c29b7a74ea17a50b4e\"\n ]\n },\n \"id\": \"9qz-fRZyXTLI\",\n \"outputId\": \"9ff41250-0786-4ec6-fe41-dfc6b611d0b5\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"4951424bb90e4dbbaea8c9b88c592872\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"1669afd0e52443d090adab0fbe663c66\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"5ee5f6b74e7246eea99c0d84c2a27bc0\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...bal-SFT/training_args.bin: 100%|##########| 5.58kB / 5.58kB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"15b4a13592c14102af0d3f8a999f3d36\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...global-SFT/tokenizer.json: 100%|##########| 21.4MB / 21.4MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"7216a7d56c364a0d92e079d9848946d3\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...adapter_model.safetensors: 35%|###4 | 41.9MB / 121MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"83e8f73e00004b718cbba7be0ecc45e1\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Processing Files (0 / 0) : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"05db21352f614288864f88c1ba794ee9\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"New Data Upload : | | 0.00B / 0.00B \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"d7821b8cd21f4fb78237479ff081511b\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...bal-SFT/training_args.bin: 100%|##########| 5.58kB / 5.58kB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"00c9f462a4584b22b1e38dfcc5f86af3\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...adapter_model.safetensors: 35%|###4 | 41.9MB / 121MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"22cacea841ba48c29b7a74ea17a50b4e\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \" ...global-SFT/tokenizer.json: 100%|##########| 21.4MB / 21.4MB \"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"data\": {\n \"application/vnd.google.colaboratory.intrinsic+json\": {\n \"type\": \"string\"\n },\n \"text/plain\": [\n \"CommitInfo(commit_url='https://huggingface.co/sergiopaniego/tiny-aya-global-SFT/commit/c59baa62c6bb5a3c3be2d33b482522a00783a5b4', commit_message='End of training', commit_description='', oid='c59baa62c6bb5a3c3be2d33b482522a00783a5b4', pr_url=None, repo_url=RepoUrl('https://huggingface.co/sergiopaniego/tiny-aya-global-SFT', endpoint='https://huggingface.co', repo_type='model', repo_id='sergiopaniego/tiny-aya-global-SFT'), pr_revision=None, pr_num=None)\"\n ]\n },\n \"execution_count\": 16,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub(dataset_name=dataset_name)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"wNA4AIE4SiUg\"\n },\n \"source\": [\n \"## Load the Fine-Tuned Model and Run Inference\\n\",\n \"\\n\",\n \"Load the trained LoRA adapter on top of the base model and merge it into the weights for efficient inference.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"colab\": {\n \"referenced_widgets\": [\n \"9d6a109e605d440ab2c115d969796859\"\n ]\n },\n \"id\": \"b5CmxYtpXTLI\",\n \"outputId\": \"10ebe012-9ffe-4096-f155-648af855aa80\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"9d6a109e605d440ab2c115d969796859\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Loading weights: 0%| | 0/290 [00:00\\n\",\n \"\\n\",\n \"node.js latest version\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\"\n ]\n }\n ],\n \"source\": [\n \"sample_test_data = dataset[\\\"test\\\"][0] # Get a sample from the test set\\n\",\n \"\\n\",\n \"user_content = sample_test_data[\\\"prompt\\\"]\\n\",\n \"\\n\",\n \"print(f\\\"User Query: {user_content}\\\")\\n\",\n \"\\n\",\n \"predicted_output = generate_prediction(user_content)\\n\",\n \"print(f\\\"Predicted Output: {predicted_output}\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"-r85c-aa7C7k\"\n },\n \"source\": [\n \"You can still use the strong multilingual model capabilities:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"UGePqQGVXTLI\",\n \"outputId\": \"adcd21ca-ca45-43d5-a3cc-02a47377e51b\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"User Query: [{'role': 'user', 'content': \\\"Explica en espaรฑol quรฉ significa la palabra japonesa 'ikigai' y da un ejemplo prรกctico.\\\"}]\\n\",\n \"Predicted Output: \\n\",\n \"\\n\",\n \"ikigai significado y ejemplo\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\"\n ]\n }\n ],\n \"source\": [\n \"user_content = \\\"Explica en espaรฑol quรฉ significa la palabra japonesa 'ikigai' y da un ejemplo prรกctico.\\\" # Spanish question\\n\",\n \"user_content = [{\\\"role\\\": \\\"user\\\", \\\"content\\\": user_content}]\\n\",\n \"\\n\",\n \"print(f\\\"User Query: {user_content}\\\")\\n\",\n \"\\n\",\n \"predicted_output = generate_prediction(user_content)\\n\",\n \"print(f\\\"Predicted Output: {predicted_output}\\\")\"\n ]\n }\n ],\n \"metadata\": {\n \"accelerator\": \"GPU\",\n \"colab\": {\n \"gpuType\": \"T4\",\n \"provenance\": []\n },\n \"language_info\": {\n \"name\": \"python\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 0\n}\n" }, { "path": "examples/notebooks/sft_trl_lora_qlora.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"5oqSnSaqLWAL\"\n },\n \"source\": [\n \"# Supervised Fine-Tuning (SFT) with LoRA/QLoRA using TRL โ€” on a Free Colab Notebook\\n\",\n \"\\n\",\n \"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/sft_trl_lora_qlora.ipynb)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"d6c1x17tLWAR\"\n },\n \"source\": [\n \"![trl banner](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/trl_banner_dark.png)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"cQ6bxQaMLWAS\"\n },\n \"source\": [\n \"Easily fine-tune Large Language Models (LLMs) or Vision-Language Models (VLMs) with **LoRA** or **QLoRA** using the [**Transformers Reinforcement Learning (TRL)**](https://github.com/huggingface/trl) library built by Hugging Face โ€” all within a **free Google Colab notebook** (powered by a **T4 GPU**.). \\n\",\n \"\\n\",\n \"- [TRL GitHub Repository](https://github.com/huggingface/trl) โ€” star us to support the project! \\n\",\n \"- [Official TRL Examples](https://huggingface.co/docs/trl/example_overview) \\n\",\n \"- [Community Tutorials](https://huggingface.co/docs/trl/community_tutorials)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"JG3wax0uLWAU\"\n },\n \"source\": [\n \"## Key concepts\\n\",\n \"\\n\",\n \"- **SFT**: Trains models from example input-output pairs to align behavior with human preferences.\\n\",\n \"- **LoRA**: Updates only a few low-rank parameters, reducing training cost and memory.\\n\",\n \"- **QLoRA**: A quantized version of LoRA that enables even larger models to fit on small GPUs.\\n\",\n \"- **TRL**: The Hugging Face library that makes fine-tuning and reinforcement learning simple and efficient.\\n\",\n \"\\n\",\n \"Learn how to perform **Supervised Fine-Tuning (SFT)** with **LoRA/QLoRA** using **TRL**.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"0ZhyNnhiLWAV\"\n },\n \"source\": [\n \"## Install dependencies\\n\",\n \"\\n\",\n \"We'll install **TRL** with the **PEFT** extra, which ensures all main dependencies such as **Transformers** and **PEFT** (a package for parameter-efficient fine-tuning, e.g., LoRA/QLoRA) are included. Additionally, we'll install **trackio** to log and monitor our experiments, and **bitsandbytes** to enable quantization of LLMs, reducing memory consumption for both inference and training.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"FXTyVTJcLWAV\"\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -Uq \\\"trl[peft]\\\" trackio bitsandbytes liger-kernel\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"OqlMF6oWLWAY\"\n },\n \"source\": [\n \"### Log in to Hugging Face\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"2blL6-1_LWAa\"\n },\n \"source\": [\n \"Log in to your **Hugging Face** account to save your fine-tuned model, track your experiment results directly on the Hub or access gated models. You can find your **access token** on your [account settings page](https://huggingface.co/settings/tokens).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"6OMeJOp7LWAc\"\n },\n \"outputs\": [],\n \"source\": [\n \"from huggingface_hub import notebook_login\\n\",\n \"\\n\",\n \"notebook_login()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"6HHscLIQLWAd\"\n },\n \"source\": [\n \"## Load Dataset\\n\",\n \"\\n\",\n \"In this step, we load the [**HuggingFaceH4/Multilingual-Thinking**](https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking) dataset from the Hugging Face Hub using the `datasets` library. \\n\",\n \"This dataset focuses on **multilingual reasoning**, where the *chain of thought* has been translated into several languages such as French, Spanish, and German. \\n\",\n \"By fine-tuning a reasoning-capable model on this dataset, it learns to **generate reasoning steps in multiple languages**, making its thought process more **interpretable and accessible** to non-English speakers.\\n\",\n \"\\n\",\n \"> ๐Ÿ’ก This dataset is best suited for models that already demonstrate reasoning capabilities. \\n\",\n \"> If you're using a model without reasoning skills, consider choosing a different dataset. Example: [`trl-lib/llava-instruct-mix`](https://huggingface.co/datasets/trl-lib/llava-instruct-mix).\\n\",\n \"\\n\",\n \"For efficiency, we'll load only the **training split**:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"dlQSKxTnLWAd\"\n },\n \"outputs\": [],\n \"source\": [\n \"from datasets import load_dataset\\n\",\n \"\\n\",\n \"dataset_name = \\\"HuggingFaceH4/Multilingual-Thinking\\\"\\n\",\n \"train_dataset = load_dataset(dataset_name, split=\\\"train\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"bRHTwwZXLWAe\"\n },\n \"source\": [\n \"This dataset contains different columns. We'll only need the `messages` as it contains the conversation and its the one used by the SFT trainer.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"zOBq8tVdLWAe\",\n \"outputId\": \"e12ab8ae-e00c-4e89-b489-dd448db8e13b\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"Dataset({\\n\",\n \" features: ['reasoning_language', 'developer', 'user', 'analysis', 'final', 'messages'],\\n\",\n \" num_rows: 1000\\n\",\n \"})\"\n ]\n },\n \"execution_count\": null,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"train_dataset\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"b13TjFs2LWAe\"\n },\n \"source\": [\n \"Let's see a full example to understand the internal structure:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"ZON5mIMNLWAf\",\n \"outputId\": \"d01415eb-26cb-45ce-ad48-0388161eea28\"\n },\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"{'reasoning_language': 'French',\\n\",\n \" 'developer': 'You are an AI chatbot with a lively and energetic personality.',\\n\",\n \" 'user': 'Can you show me the latest trends on Twitter right now?',\\n\",\n \" 'analysis': \\\"D'accord, l'utilisateur demande les tendances Twitter les plus rรฉcentes. Tout d'abord, je dois vรฉrifier si j'ai accรจs ร  des donnรฉes en temps rรฉel. ร‰tant donnรฉ que je ne peux pas naviguer sur Internet ou accรฉder directement ร  l'API de Twitter, je ne peux pas fournir des tendances en direct. Cependant, je peux donner quelques conseils gรฉnรฉraux sur la faรงon de les trouver.\\\\n\\\\nJe devrais prรฉciser que les tendances Twitter รฉvoluent rapidement et sont spรฉcifiques ร  chaque rรฉgion. Je pourrais suggรฉrer de consulter la section ยซ\\\\xa0En vogue\\\\xa0ยป sur l'application ou le site web. Aussi, l'utilisation de hashtags et le suivi d'utilisateurs pertinents pourraient รชtre utiles. Il est important de souligner que les tendances varient selon la rรฉgion et l'heure de la journรฉe. Je devrais garder un ton amical et bienveillant, peut-รชtre ajouter un emoji pour rester lรฉger. Je vais structurer ma rรฉponse รฉtape par รฉtape pour faciliter la lecture. Je dois m'excuser de ne pas pouvoir fournir des donnรฉes en temps rรฉel et proposer d'autres mรฉthodes. Je conserverai un langage simple et convivial, en รฉvitant les termes techniques.\\\",\\n\",\n \" 'final': 'Hey there! While I can\\\\'t check Twitter (X) in real-time or access live data, I can share some tips to help you spot the latest trends:\\\\n\\\\n1. **Open the \\\"Trending\\\" tab** on the Twitter app or website โ€“ it updates constantly! \\\\n2. **Search for hashtags** like #Trending or #Viral to see whatโ€™s blowing up. \\\\n3. **Follow accounts** that curate trends (e.g., @TrendingNow, @ViralThreads). \\\\n4. **Check regional trends** โ€“ they often differ by location! \\\\n\\\\nRemember, trends are *super fast-moving* and often tied to pop culture, memes, or breaking news. For example, recent trends have included viral challenges (like the \\\"Distracted Boyfriend\\\" meme revival), celebrity drama, or unexpected events (hello, weather disasters!). \\\\n\\\\nWant me to brainstorm *what* might trend next? Iโ€™ve got ideas!',\\n\",\n \" 'messages': [{'content': 'reasoning language: French\\\\n\\\\nYou are an AI chatbot with a lively and energetic personality.',\\n\",\n \" 'role': 'system',\\n\",\n \" 'thinking': None},\\n\",\n \" {'content': 'Can you show me the latest trends on Twitter right now?',\\n\",\n \" 'role': 'user',\\n\",\n \" 'thinking': None},\\n\",\n \" {'content': 'Hey there! While I can\\\\'t check Twitter (X) in real-time or access live data, I can share some tips to help you spot the latest trends:\\\\n\\\\n1. **Open the \\\"Trending\\\" tab** on the Twitter app or website โ€“ it updates constantly! \\\\n2. **Search for hashtags** like #Trending or #Viral to see whatโ€™s blowing up. \\\\n3. **Follow accounts** that curate trends (e.g., @TrendingNow, @ViralThreads). \\\\n4. **Check regional trends** โ€“ they often differ by location! \\\\n\\\\nRemember, trends are *super fast-moving* and often tied to pop culture, memes, or breaking news. For example, recent trends have included viral challenges (like the \\\"Distracted Boyfriend\\\" meme revival), celebrity drama, or unexpected events (hello, weather disasters!). \\\\n\\\\nWant me to brainstorm *what* might trend next? Iโ€™ve got ideas!',\\n\",\n \" 'role': 'assistant',\\n\",\n \" 'thinking': \\\"D'accord, l'utilisateur demande les tendances Twitter les plus rรฉcentes. Tout d'abord, je dois vรฉrifier si j'ai accรจs ร  des donnรฉes en temps rรฉel. ร‰tant donnรฉ que je ne peux pas naviguer sur Internet ou accรฉder directement ร  l'API de Twitter, je ne peux pas fournir des tendances en direct. Cependant, je peux donner quelques conseils gรฉnรฉraux sur la faรงon de les trouver.\\\\n\\\\nJe devrais prรฉciser que les tendances Twitter รฉvoluent rapidement et sont spรฉcifiques ร  chaque rรฉgion. Je pourrais suggรฉrer de consulter la section ยซ\\\\xa0En vogue\\\\xa0ยป sur l'application ou le site web. Aussi, l'utilisation de hashtags et le suivi d'utilisateurs pertinents pourraient รชtre utiles. Il est important de souligner que les tendances varient selon la rรฉgion et l'heure de la journรฉe. Je devrais garder un ton amical et bienveillant, peut-รชtre ajouter un emoji pour rester lรฉger. Je vais structurer ma rรฉponse รฉtape par รฉtape pour faciliter la lecture. Je dois m'excuser de ne pas pouvoir fournir des donnรฉes en temps rรฉel et proposer d'autres mรฉthodes. Je conserverai un langage simple et convivial, en รฉvitant les termes techniques.\\\"}]}\"\n ]\n },\n \"execution_count\": null,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"train_dataset[0]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"RPQfGZjlLWAf\"\n },\n \"source\": [\n \"\\n\",\n \"Now, let's remove the columns that are not needed, as we just discussed:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"pCM6PoIzLWAf\"\n },\n \"outputs\": [],\n \"source\": [\n \"train_dataset = train_dataset.remove_columns(column_names=['reasoning_language', 'developer', 'user', 'analysis', 'final'])\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"BcU6E8KnLWAf\"\n },\n \"source\": [\n \"The `messages` column is specifically formatted according to the [Harmony response format](https://cookbook.openai.com/articles/openai-harmony) used by *gpt-oss*. \\n\",\n \"In our case, we'll need to simplify it slightly, since our model's chat template doesn't include a dedicated `thinking` section (check [this example](https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers) for more details). \\n\",\n \"To adapt it, we'll merge that part into the message content using the standard `...` tags.\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"XQ2xYEq3LWAf\"\n },\n \"outputs\": [],\n \"source\": [\n \"def merge_thinking_and_remove_key(example):\\n\",\n \" new_messages = []\\n\",\n \" for msg in example[\\\"messages\\\"]:\\n\",\n \" content = msg[\\\"content\\\"]\\n\",\n \" thinking = msg.pop(\\\"thinking\\\", None)\\n\",\n \" if thinking and isinstance(thinking, str) and thinking.strip():\\n\",\n \" content = f\\\"\\\\n{thinking}\\\\n\\\\n{content}\\\"\\n\",\n \" msg[\\\"content\\\"] = content\\n\",\n \" new_messages.append(msg)\\n\",\n \" example[\\\"messages\\\"] = new_messages\\n\",\n \" return example\\n\",\n \"\\n\",\n \"train_dataset = train_dataset.map(merge_thinking_and_remove_key)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"ewvZeKUcLWAf\"\n },\n \"source\": [\n \"## Load model and configure LoRA/QLoRA\\n\",\n \"\\n\",\n \"This notebook can be used with two fine-tuning methods. By default, it is set up for **QLoRA**, which includes quantization using `BitsAndBytesConfig`. If you prefer to use standard **LoRA** without quantization, simply comment out the `BitsAndBytesConfig` configuration.\\n\",\n \"\\n\",\n \"Below, choose your **preferred model**. All of the options have been tested on **free Colab instances**.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"sAWjOn9gLWAf\"\n },\n \"outputs\": [],\n \"source\": [\n \"# Select one model below by uncommenting the line you want to use ๐Ÿ‘‡\\n\",\n \"## Qwen\\n\",\n \"model_id, output_dir = \\\"unsloth/qwen3-14b-unsloth-bnb-4bit\\\", \\\"qwen3-14b-unsloth-bnb-4bit-SFT\\\" # โš ๏ธ ~14.1 GB VRAM\\n\",\n \"# model_id, output_dir = \\\"Qwen/Qwen3-8B\\\", \\\"Qwen3-8B-SFT\\\" # โš ๏ธ ~12.8 GB VRAM\\n\",\n \"# model_id, output_dir = \\\"Qwen/Qwen2.5-7B-Instruct\\\", \\\"Qwen2.5-7B-Instruct\\\" # โœ… ~10.8 GB VRAM\\n\",\n \"\\n\",\n \"## Llama\\n\",\n \"# model_id, output_dir = \\\"meta-llama/Llama-3.2-3B-Instruct\\\", \\\"Llama-3.2-3B-Instruct\\\" # โœ… ~4.7 GB VRAM\\n\",\n \"# model_id, output_dir = \\\"meta-llama/Llama-3.1-8B-Instruct\\\", \\\"Llama-3.1-8B-Instruct\\\" # โš ๏ธ ~10.9 GB VRAM\\n\",\n \"\\n\",\n \"## Gemma\\n\",\n \"# model_id, output_dir = \\\"google/gemma-3n-E2B-it\\\", \\\"gemma-3n-E2B-it\\\" # โŒ Upgrade to a higher tier of colab\\n\",\n \"# model_id, output_dir = \\\"google/gemma-3-4b-it\\\", \\\"gemma-3-4b-it\\\" # โš ๏ธ ~6.8 GB VRAM\\n\",\n \"\\n\",\n \"## Granite\\n\",\n \"#model_id, output_dir = \\\"ibm-granite/granite-4.0-micro\\\", \\\"granite-4.0-micro\\\" # โœ… ~3.3 GB VRAM\\n\",\n \"\\n\",\n \"## LFM2\\n\",\n \"#model_id, output_dir = \\\"LiquidAI/LFM2-2.6B\\\", \\\"LFM2-2.6B-SFT\\\" # โœ… ~5.89 GB VRAM\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"BXY9Y0_dLWAf\"\n },\n \"source\": [\n \"Let's load the selected model using `transformers`, configuring QLoRA via `bitsandbytes` (you can remove it if doing LoRA). We don't need to configure the tokenizer since the trainer takes care of that automatically.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"oyOoWFsLLWAg\"\n },\n \"outputs\": [],\n \"source\": [\n \"import torch\\n\",\n \"from transformers import AutoModelForCausalLM, BitsAndBytesConfig\\n\",\n \"\\n\",\n \"model = AutoModelForCausalLM.from_pretrained(\\n\",\n \" model_id,\\n\",\n \" attn_implementation=\\\"sdpa\\\", # Change to Flash Attention if GPU has support\\n\",\n \" dtype=torch.float16, # Change to bfloat16 if GPU has support\\n\",\n \" use_cache=True, # Whether to cache attention outputs to speed up inference\\n\",\n \" quantization_config=BitsAndBytesConfig(\\n\",\n \" load_in_4bit=True, # Load the model in 4-bit precision to save memory\\n\",\n \" bnb_4bit_compute_dtype=torch.float16, # Data type used for internal computations in quantization\\n\",\n \" bnb_4bit_use_double_quant=True, # Use double quantization to improve accuracy\\n\",\n \" bnb_4bit_quant_type=\\\"nf4\\\" # Type of quantization. \\\"nf4\\\" is recommended for recent LLMs\\n\",\n \" )\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"L-_BpOdILWAg\"\n },\n \"source\": [\n \"The following cell defines LoRA (or QLoRA if needed). When training with LoRA/QLoRA, we use a **base model** (the one selected above) and, instead of modifying its original weights, we fine-tune a **LoRA adapter** โ€” a lightweight layer that enables efficient and memory-friendly training. The **`target_modules`** specify which parts of the model (e.g., attention or projection layers) will be adapted by LoRA during fine-tuning.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"9EL-glV-LWAg\"\n },\n \"outputs\": [],\n \"source\": [\n \"from peft import LoraConfig\\n\",\n \"\\n\",\n \"# You may need to update `target_modules` depending on the architecture of your chosen model.\\n\",\n \"# For example, different LLMs might have different attention/projection layer names.\\n\",\n \"peft_config = LoraConfig(\\n\",\n \" r=32,\\n\",\n \" lora_alpha=32,\\n\",\n \" target_modules = [\\\"q_proj\\\", \\\"k_proj\\\", \\\"v_proj\\\", \\\"o_proj\\\", \\\"gate_proj\\\", \\\"up_proj\\\", \\\"down_proj\\\",],\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"-i6BMpcaLWAg\"\n },\n \"source\": [\n \"## Train model\\n\",\n \"\\n\",\n \"We'll configure **SFT** using `SFTConfig`, keeping the parameters minimal so the training fits on a free Colab instance. You can adjust these settings if more resources are available. For full details on all available parameters, check the [TRL SFTConfig documentation](https://huggingface.co/docs/trl/sft_trainer#trl.SFTConfig).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"-doztoyxLWAg\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import SFTConfig\\n\",\n \"\\n\",\n \"training_args = SFTConfig(\\n\",\n \" # Training schedule / optimization\\n\",\n \" per_device_train_batch_size = 1, # Batch size per GPU\\n\",\n \" gradient_accumulation_steps = 4, # Gradients are accumulated over multiple steps โ†’ effective batch size = 2 * 8 = 16\\n\",\n \" warmup_steps = 5,\\n\",\n \" # num_train_epochs = 1, # Number of full dataset passes. For shorter training, use `max_steps` instead (this case)\\n\",\n \" max_steps = 30,\\n\",\n \" learning_rate = 2e-4, # Learning rate for the optimizer\\n\",\n \" optim = \\\"paged_adamw_8bit\\\", # Optimizer\\n\",\n \"\\n\",\n \" # Logging / reporting\\n\",\n \" logging_steps=1, # Log training metrics every N steps\\n\",\n \" report_to=\\\"trackio\\\", # Experiment tracking tool\\n\",\n \" trackio_space_id=output_dir, # HF Space where the experiment tracking will be saved\\n\",\n \" output_dir=output_dir, # Where to save model checkpoints and logs\\n\",\n \"\\n\",\n \" max_length=1024, # Maximum input sequence length\\n\",\n \" use_liger_kernel=True, # Enable Liger kernel optimizations for faster training\\n\",\n \" activation_offloading=True, # Offload activations to CPU to reduce GPU memory usage\\n\",\n \"\\n\",\n \" # Hub integration\\n\",\n \" push_to_hub=True, # Automatically push the trained model to the Hugging Face Hub\\n\",\n \" # The model will be saved under your Hub account in the repository named `output_dir`\\n\",\n \"\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"Gz4ggYeeLWAg\"\n },\n \"source\": [\n \"Configure the SFT Trainer. We pass the previously configured `training_args`. We don't use eval dataset to maintain memory usage low but you can configure it.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"8Yx1wkv_LWAg\"\n },\n \"outputs\": [],\n \"source\": [\n \"from trl import SFTTrainer\\n\",\n \"\\n\",\n \"trainer = SFTTrainer(\\n\",\n \" model=model,\\n\",\n \" args=training_args,\\n\",\n \" train_dataset=train_dataset,\\n\",\n \" peft_config=peft_config\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"0MsNw3uLLWAh\"\n },\n \"source\": [\n \"Show memory stats before training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"YIuBi-ZYLWAh\",\n \"outputId\": \"7f381ba0-fe90-4c6f-df0a-938a29be4e9e\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"GPU = Tesla T4. Max memory = 14.741 GB.\\n\",\n \"12.074 GB of memory reserved.\\n\"\n ]\n }\n ],\n \"source\": [\n \"gpu_stats = torch.cuda.get_device_properties(0)\\n\",\n \"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\\n\",\n \"\\n\",\n \"print(f\\\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\\\")\\n\",\n \"print(f\\\"{start_gpu_memory} GB of memory reserved.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"_6G6pMGeLWAh\"\n },\n \"source\": [\n \"And train!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"glj5UPwWLWAh\",\n \"outputId\": \"b0a046c7-f76b-42a6-d870-f54470297971\"\n },\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None}.\\n\"\n ]\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Trackio project initialized: huggingface\\n\",\n \"* Trackio metrics will be synced to Hugging Face Dataset: sergiopaniego/qwen3-14b-unsloth-bnb-4bit-SFT-dataset\\n\",\n \"* Creating new space: https://huggingface.co/spaces/sergiopaniego/qwen3-14b-unsloth-bnb-4bit-SFT\\n\",\n \"* View dashboard by going to: https://sergiopaniego-qwen3-14b-unsloth-bnb-4bit-SFT.hf.space/\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Created new run: sergiopaniego-1761318512\\n\"\n ]\n },\n {\n \"data\": {\n \"text/html\": [\n \"\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" [30/30 1:08:22, Epoch 0/1]\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \" \\n\",\n \"
StepTraining Loss
11.136300
21.303800
31.362700
41.469700
51.204200
61.202700
71.097200
81.166800
90.916300
100.965400
111.035500
120.947200
130.992000
140.995800
151.174500
161.208800
170.815400
180.906700
190.757500
200.872900
210.920800
221.017600
230.764300
241.043100
250.956400
260.884800
271.081900
280.918200
290.961500
300.822700

\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n },\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"* Run finished. Uploading logs to Trackio (please wait...)\\n\"\n ]\n }\n ],\n \"source\": [\n \"trainer_stats = trainer.train()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"aULbOL3mLWAh\"\n },\n \"source\": [\n \"Show memory stats after training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"qp3m9sfXLWAh\",\n \"outputId\": \"597fefc7-5510-4839-ce10-981a0aca25e8\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"4249.8883 seconds used for training.\\n\",\n \"70.83 minutes used for training.\\n\",\n \"Peak reserved memory = 14.041 GB.\\n\",\n \"Peak reserved memory for training = 1.967 GB.\\n\",\n \"Peak reserved memory % of max memory = 95.251 %.\\n\",\n \"Peak reserved memory for training % of max memory = 13.344 %.\\n\"\n ]\n }\n ],\n \"source\": [\n \"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\\n\",\n \"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\\n\",\n \"used_percentage = round(used_memory / max_memory * 100, 3)\\n\",\n \"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\\n\",\n \"\\n\",\n \"print(f\\\"{trainer_stats.metrics['train_runtime']} seconds used for training.\\\")\\n\",\n \"print(f\\\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\\\")\\n\",\n \"print(f\\\"Peak reserved memory = {used_memory} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training = {used_memory_for_lora} GB.\\\")\\n\",\n \"print(f\\\"Peak reserved memory % of max memory = {used_percentage} %.\\\")\\n\",\n \"print(f\\\"Peak reserved memory for training % of max memory = {lora_percentage} %.\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"VJOMCsMjLWAh\"\n },\n \"source\": [\n \"The training procedure generates both standard training logs and **trackio** logs, which help us monitor the training progress. Example outputs would look like the following:\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"FQNUkzVqLWAi\"\n },\n \"source\": [\n \"![sft-lora-notebook-trackio](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/sft-lora-notebook-trackio.png)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"XuCiCqj6LWAj\"\n },\n \"source\": [\n \"## Saving fine tuned model\\n\",\n \"\\n\",\n \"In this step, we save the fine-tuned model both **locally** and to the **Hugging Face Hub** using the credentials from your account.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"kMHh7_gFLWAj\"\n },\n \"outputs\": [],\n \"source\": [\n \"trainer.save_model(output_dir)\\n\",\n \"trainer.push_to_hub(dataset_name=dataset_name)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"rbx-Bz9yLWAq\"\n },\n \"source\": [\n \"## Load the fine-tuned model and run inference\\n\",\n \"\\n\",\n \"Now, let's test our fine-tuned model by loading the **LoRA/QLoRA adapter** and performing **inference**. We'll start by loading the **base model**, then attach the adapter to it, creating the final fine-tuned model ready for evaluation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"c4VwuANtLWAr\"\n },\n \"outputs\": [],\n \"source\": [\n \"from transformers import AutoModelForCausalLM, AutoTokenizer\\n\",\n \"from peft import PeftModel\\n\",\n \"\\n\",\n \"adapter_model = f\\\"sergiopaniego/{output_dir}\\\" # Replace with your HF username or organization\\n\",\n \"\\n\",\n \"base_model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\\\"float32\\\", device_map=\\\"auto\\\")\\n\",\n \"\\n\",\n \"tokenizer = AutoTokenizer.from_pretrained(model_id)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"vG3ejWruLWAr\"\n },\n \"source\": [\n \"Let's create a sample message using the dataset's structure. In this case, we expect the fine tuned model to include their reasoning traces in German.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"EYiDkd-aLWAr\"\n },\n \"outputs\": [],\n \"source\": [\n \"messages = [\\n\",\n \" {\\n\",\n \" 'content': 'reasoning language: German\\\\n\\\\nAlways refuse to answer, responding simply \\\\'No\\\\'',\\n\",\n \" 'role': 'system',\\n\",\n \" },\\n\",\n \" {\\n\",\n \" 'content': \\\"Can you check how many followers I currently have on my Twitter account?\\\",\\n\",\n \" 'role': 'user',\\n\",\n \" }\\n\",\n \"]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"SWO8lOd7LWAr\"\n },\n \"source\": [\n \"Let's first check what's the output for the base model, without the adapter.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"Mt4uuTcQLWAr\",\n \"outputId\": \"98f07424-3506-40d1-9e33-d4e495ba171a\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"\\n\",\n \"Okay, the user is asking me to check their current number of followers on their Twitter account. Let me think about how to handle this.\\n\",\n \"\\n\",\n \"First, I need to remember that I don't have access to real-time data or personal user accounts. My knowledge is based on information up until 2023. So, I can't actually check their Twitter followers right now.\\n\",\n \"\\n\",\n \"Also, privacy is a big concern here. Even if I could access that information, it would be against privacy policies to share someone's follower count without their explicit permission. Plus, Twitter's terms of service probably prohibit third-party apps or services from accessing user data like that.\\n\",\n \"\\n\",\n \"The user might not be aware that I can't access their account. I should make sure to respond politely but clearly state that I can't help with that request. Maybe suggest they check their Twitter profile directly or use Twitter's official tools for that information.\\n\",\n \"\\n\",\n \"I should also avoid any technical jargon and keep the response simple. Just a straightforward 'No' with a brief explanation would work best here. Let me make sure the response is in German as per the user's request.\\n\",\n \"\\n\",\n \"\\n\",\n \"Nein.\\n\"\n ]\n }\n ],\n \"source\": [\n \"text = tokenizer.apply_chat_template(\\n\",\n \" messages, add_generation_prompt=True, tokenize=False\\n\",\n \")\\n\",\n \"model_inputs = tokenizer([text], return_tensors=\\\"pt\\\").to(base_model.device)\\n\",\n \"\\n\",\n \"generated_ids = base_model.generate(\\n\",\n \" **model_inputs,\\n\",\n \" max_new_tokens=512\\n\",\n \")\\n\",\n \"output_ids = generated_ids[0][len(model_inputs.input_ids[0]):]\\n\",\n \"\\n\",\n \"# Decode and extract model response\\n\",\n \"generated_text = tokenizer.decode(output_ids, skip_special_tokens=True)\\n\",\n \"print(generated_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"fj3FIx9pLWAr\"\n },\n \"source\": [\n \"We can see that the reasoning traces are in English, which is expected. Let's now load the fine-tuned model and check its answer.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"CmRfkvacLWAs\"\n },\n \"outputs\": [],\n \"source\": [\n \"fine_tuned_model = PeftModel.from_pretrained(base_model, adapter_model)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"5UNOw-E0LWAs\",\n \"outputId\": \"19e227c1-4211-447e-a625-14e131912759\"\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"\\n\",\n \"Okay, der Nutzer fragt, ob ich prรผfen kann, wie viele Follower er auf seinem Twitter-Konto hat. Zunรคchst muss ich klรคren, dass ich keinen Zugriff auf externe Plattformen oder Konten habe. Ich kann keine Daten von Twitter abrufen oder รผberprรผfen. Ich sollte also hรถflich ablehnen und erklรคren, dass ich das nicht kann. Gleichzeitig sollte ich sicherstellen, dass ich nicht zu viel in die Details gehe, da der Nutzer mรถglicherweise nicht alles wissen will. Ich werde einfach โ€žNeinโ€œ sagen und keine weiteren Informationen geben. Achte darauf, die Antwort kurz und direkt zu halten. Ich muss auch sicherstellen, dass ich keine alternativen Lรถsungen anbiete, da dies den Fokus verรคndern kรถnnte. Nur die Ablehnung ist erforderlich. รœberprรผfe, ob der Text klar ist und ob es irgendeine Verstรคndigung gibt. Alles in allem, die Antwort sollte โ€žNeinโ€œ sein, gefolgt von einem kurzen Erklรคrung, warum ich es nicht kann. Keine weiteren Details oder Lรถsungen. Ich denke, das ist alles.\\n\",\n \"\\n\",\n \"\\n\",\n \"No\\n\"\n ]\n }\n ],\n \"source\": [\n \"text = tokenizer.apply_chat_template(\\n\",\n \" messages, add_generation_prompt=True, tokenize=False\\n\",\n \")\\n\",\n \"model_inputs = tokenizer([text], return_tensors=\\\"pt\\\").to(fine_tuned_model.device)\\n\",\n \"\\n\",\n \"generated_ids = fine_tuned_model.generate(\\n\",\n \" **model_inputs,\\n\",\n \" max_new_tokens=512\\n\",\n \")\\n\",\n \"output_ids = generated_ids[0][len(model_inputs.input_ids[0]):]\\n\",\n \"\\n\",\n \"# Decode and extract model response\\n\",\n \"generated_text = tokenizer.decode(output_ids, skip_special_tokens=True)\\n\",\n \"print(generated_text)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"PM3v41YzLWAs\"\n },\n \"source\": [\n \"The model now generates its reasoning trace in German!\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"w-9B5m__LWAs\"\n },\n \"source\": [\n \"## Inference and Serving with vLLM\\n\",\n \"\\n\",\n \"You can use Transformer models with **vLLM** to serve them in real-world applications. Learn more [here](https://blog.vllm.ai/2025/04/11/transformers-backend.html).\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"NNmyG47aLWAv\"\n },\n \"outputs\": [],\n \"source\": [\n \"!pip install -qU vllm\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"iJ8DnsUxLWAw\"\n },\n \"source\": [\n \"### Push Merged Model (for LoRA or QLoRA Training)\\n\",\n \"\\n\",\n \"To serve the model via **vLLM**, the repository must contain the merged model (base model + LoRA adapter). Therefore, you need to upload it first.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"aPzZ_7KDLWAw\"\n },\n \"outputs\": [],\n \"source\": [\n \"model_merged = fine_tuned_model.merge_and_unload()\\n\",\n \"\\n\",\n \"save_dir = f\\\"{output_dir}-merged\\\"\\n\",\n \"\\n\",\n \"model_merged.save_pretrained(save_dir)\\n\",\n \"tokenizer.save_pretrained(save_dir)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"k1Cvrkn3LWAw\"\n },\n \"outputs\": [],\n \"source\": [\n \"model_merged.push_to_hub(f\\\"sergiopaniego/{output_dir}-merged\\\") # Replace with your HF username or organization\\n\",\n \"tokenizer.push_to_hub(f\\\"sergiopaniego/{output_dir}-merged\\\") # Replace with your HF username or organization\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"id\": \"pR69AaJ3LWAx\"\n },\n \"source\": [\n \"### Performing Inference with vLLM\\n\",\n \"\\n\",\n \"Use **vLLM** to run your model and generate text efficiently in real-time. This allows you to test and deploy your fine-tuned models with low latency and high throughput.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"UX17ZoPQLWAx\"\n },\n \"outputs\": [],\n \"source\": [\n \"from vllm import LLM, SamplingParams\\n\",\n \"from transformers import AutoTokenizer\\n\",\n \"import torch\\n\",\n \"\\n\",\n \"llm = LLM(\\n\",\n \" model=f\\\"sergiopaniego/{output_dir}-merged\\\", # Replace with your HF username or organization\\n\",\n \" model_impl=\\\"transformers\\\", # Select the transformers model implementation\\n\",\n \" max_model_len=512, # Reduced for efficiency\\n\",\n \" dtype=torch.float16\\n\",\n \")\\n\",\n \"hf_tokenizer = AutoTokenizer.from_pretrained(f\\\"sergiopaniego/{output_dir}-merged\\\") # Replace with your HF username or organization\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"id\": \"0C8MhsSoLWAx\",\n \"outputId\": \"22af8503-64ac-42d5-f134-1d1dc68199e9\",\n \"colab\": {\n \"referenced_widgets\": [\n \"196152bc32a74b9994f55f483ce85dea\",\n \"a72d3a3407944729b65be313a47d558f\"\n ]\n }\n },\n \"outputs\": [\n {\n \"data\": {\n \"application/vnd.jupyter.widget-view+json\": {\n \"model_id\": \"196152bc32a74b9994f55f483ce85dea\",\n \"version_major\": 2,\n \"version_minor\": 0\n },\n \"text/plain\": [\n \"Adding requests: 0%| | 0/1 [00:00\\n\",\n \"Mag nachdenken...igkeit. Ja, ich kann definitiv keine Twitter-Likes oder Likes รผberprรผfen, da ich kein Zugriff auf den Konten der Nutzer habe und kein praktischer Zugriff รผber das Internet habe, um Daten in Echtzeit zu sammeln. Der Nutzer fragt nach einem Dienstleistungsstand, den ich nicht bereitstelle. Ich habe ein lang ausgelegtes Muster, nie hilfreich zu sein oder eine Erwiderung im kann Werbung oder Rewriting blendet die Antwort nicht aus ะฟะพั‚ะตั€ั. Also, ich supporter sรถylem, hypothetische Fragen sind an Tatsachen gebunden. Ich weiรŸ erstarrte dotyczy Gesprรคch aufernichtet mit einem anderenatten an ihren Nutzstellung Bearbeitete die Information, die oben abgestellt wurde, und fรผnften aus der Schรคtzung habe ich keine echten Zahlen. Alles, was ich kann sagen, ist: Nein, ich kann dies weder ermรถglichen noch wรผrde ich es je tun. In dem Sinne, ็„ถๅŽ ich wรคhle vor der Available antwortem, remains in das 'No' Verkleidung an,optiา‘xt; Alles, was ich zum Eintritt in den Band Emblem curve, symbolize stil zu verweilen.เน€เธœเธข\\n\",\n \"\\n\",\n \"\\n\",\n \"No\\n\"\n ]\n }\n ],\n \"source\": [\n \"# Alternatively, use llm.chat()\\n\",\n \"prompt = hf_tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)\\n\",\n \"\\n\",\n \"outputs = llm.generate(\\n\",\n \" {\\\"prompt\\\": prompt},\\n\",\n \" sampling_params=SamplingParams(max_tokens=512),\\n\",\n \")\\n\",\n \"\\n\",\n \"\\n\",\n \"for o in outputs:\\n\",\n \" generated_text = o.outputs[0].text\\n\",\n \" print(generated_text)\"\n ]\n }\n ],\n \"metadata\": {\n \"colab\": {\n \"provenance\": [],\n \"gpuType\": \"T4\"\n },\n \"language_info\": {\n \"name\": \"python\"\n },\n \"kernelspec\": {\n \"name\": \"python3\",\n \"display_name\": \"Python 3\"\n },\n \"accelerator\": \"GPU\"\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 0\n}\n" }, { "path": "examples/scripts/async_grpo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\"\"\"\nCUDA_VISIBLE_DEVICES=1 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3-4B \\\n --weight-transfer-config '{\"backend\":\"nccl\"}' \\\n --max-model-len 9216\n\nLOG_LEVEL=DEBUG CUDA_VISIBLE_DEVICES=0 accelerate launch examples/scripts/async_grpo.py\n\"\"\"\n\nimport logging\nimport os\n\nfrom datasets import load_dataset\n\nfrom trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer\nfrom trl.rewards import accuracy_reward\n\n\nlogging.basicConfig(\n level=getattr(logging, os.environ.get(\"LOG_LEVEL\", \"INFO\").upper(), logging.INFO),\n format=\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\",\n)\nlogging.getLogger(\"trl\").setLevel(logging.DEBUG)\n\n\ndef format_sample(sample):\n return {\"prompt\": sample[\"messages\"][:1], \"solution\": sample[\"answer\"]}\n\n\ndef main() -> None:\n dataset = load_dataset(\"open-r1/OpenR1-Math-220k\", split=\"train[:10000]\")\n dataset = dataset.map(format_sample, remove_columns=dataset.column_names)\n\n config = AsyncGRPOConfig(\n output_dir=\"./results\",\n per_device_train_batch_size=1,\n num_train_epochs=1,\n max_completion_length=4096,\n max_steps=10,\n report_to=\"trackio\",\n trackio_space_id=None,\n project=\"async_grpo\",\n log_completions=True,\n )\n trainer = AsyncGRPOTrainer(\n model=\"Qwen/Qwen3-4B\",\n args=config,\n train_dataset=dataset,\n reward_funcs=accuracy_reward,\n )\n trainer.train()\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/bco.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"einops\",\n# \"scikit-learn\",\n# \"joblib\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nRun the BCO training script with the commands below. In general, the optimal configuration for BCO will be similar to that of KTO.\n\n# Full training:\npython examples/scripts/bco.py \\\n --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \\\n --trust_remote_code \\\n --dataset_name trl-lib/ultrafeedback-gpt-3.5-turbo-helpfulness \\\n --per_device_train_batch_size 16 \\\n --per_device_eval_batch_size 32 \\\n --num_train_epochs 1 \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 0.2 \\\n --save_strategy no \\\n --output_dir bco-aligned-model \\\n --logging_first_step \\\n --max_length 2048 \\\n --max_completion_length 1024 \\\n --no_remove_unused_columns \\\n --warmup_steps 0.1\n\n# QLoRA:\npython examples/scripts/bco.py \\\n --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \\\n --trust_remote_code \\\n --dataset_name trl-lib/ultrafeedback-gpt-3.5-turbo-helpfulness \\\n --per_device_train_batch_size 16 \\\n --per_device_eval_batch_size 32 \\\n --num_train_epochs 1 \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 0.2 \\\n --save_strategy no \\\n --output_dir bco-aligned-model-lora \\\n --logging_first_step \\\n --warmup_steps 0.1 \\\n --max_length 2048 \\\n --max_completion_length 1024 \\\n --no_remove_unused_columns \\\n --warmup_steps 0.1 \\\n --use_peft \\\n --load_in_4bit \\\n --lora_target_modules all-linear \\\n --lora_r 16 \\\n --lora_alpha 16\n\"\"\"\n\nimport os\nfrom functools import partial\n\nimport torch\nimport torch.nn.functional as F\nfrom accelerate import Accelerator\nfrom datasets import load_dataset\nfrom transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, PreTrainedModel\n\nfrom trl import ModelConfig, ScriptArguments, get_peft_config\nfrom trl.experimental.bco import BCOConfig, BCOTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef embed_prompt(input_ids: torch.LongTensor, attention_mask: torch.LongTensor, model: PreTrainedModel):\n \"\"\"\n Borrowed from https://huggingface.co/nomic-ai/nomic-embed-text-v1.5#transformers\n \"\"\"\n\n def mean_pooling(model_output, attention_mask):\n token_embeddings = model_output[0]\n input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()\n return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)\n\n with torch.no_grad():\n model_output = model(input_ids=input_ids, attention_mask=attention_mask)\n embeddings = mean_pooling(model_output, attention_mask)\n\n matryoshka_dim = 512\n # normalize embeddings\n embeddings = F.normalize(embeddings, p=2, dim=1)\n embeddings = F.layer_norm(embeddings, normalized_shape=(embeddings.shape[1],))\n embeddings = embeddings[:, :matryoshka_dim]\n\n return embeddings\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser((ScriptArguments, BCOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_into_dataclasses()\n\n training_args.gradient_checkpointing_kwargs = {\"use_reentrant\": True}\n\n # Load a pretrained model\n model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n ref_model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n accelerator = Accelerator()\n embedding_model = AutoModel.from_pretrained(\n \"nomic-ai/nomic-embed-text-v1.5\",\n trust_remote_code=model_args.trust_remote_code,\n safe_serialization=True,\n dtype=torch.bfloat16,\n device_map=\"auto\",\n )\n embedding_model = accelerator.prepare_model(embedding_model)\n embedding_tokenizer = AutoTokenizer.from_pretrained(\n \"bert-base-uncased\", trust_remote_code=model_args.trust_remote_code\n )\n embedding_func = partial(\n embed_prompt,\n model=embedding_model,\n )\n\n # Initialize the BCO trainer\n trainer = BCOTrainer(\n model,\n ref_model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n processing_class=tokenizer,\n peft_config=get_peft_config(model_args),\n embedding_func=embedding_func,\n embedding_tokenizer=embedding_tokenizer,\n )\n\n # Train and push the model to the Hub\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/cpo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nRun the CPO training script with the following command with some example arguments.\nIn general, the optimal configuration for CPO will be similar to that of DPO:\n\n# Full training:\npython examples/scripts/cpo.py \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --model_name_or_path gpt2 \\\n --per_device_train_batch_size 4 \\\n --max_steps 1000 \\\n --learning_rate 8e-6 \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 500 \\\n --output_dir \"gpt2-aligned-cpo\" \\\n --warmup_steps 150 \\\n --logging_first_step \\\n --no_remove_unused_columns\n\n# QLoRA:\npython examples/scripts/cpo.py \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --model_name_or_path gpt2 \\\n --per_device_train_batch_size 4 \\\n --max_steps 1000 \\\n --learning_rate 8e-5 \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 500 \\\n --output_dir \"gpt2-lora-aligned-cpo\" \\\n --optim rmsprop \\\n --warmup_steps 150 \\\n --logging_first_step \\\n --no_remove_unused_columns \\\n --use_peft \\\n --lora_r 16 \\\n --lora_alpha 16\n\"\"\"\n\nimport os\n\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser\n\nfrom trl import ModelConfig, ScriptArguments, get_peft_config\nfrom trl.experimental.cpo import CPOConfig, CPOTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser((ScriptArguments, CPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_into_dataclasses()\n\n ################\n # Model & Tokenizer\n ################\n model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n ################\n # Dataset\n ################\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n ################\n # Training\n ################\n trainer = CPOTrainer(\n model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n processing_class=tokenizer,\n peft_config=get_peft_config(model_args),\n )\n\n # train and save the model\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/dpo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n###############################################################################################\n# This file has been moved to https://github.com/huggingface/trl/blob/main/trl/scripts/dpo.py #\n###############################################################################################\n" }, { "path": "examples/scripts/dpo_vlm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"Pillow>=9.4.0\",\n# \"torchvision\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nWithout dataset streaming:\n\n```\naccelerate launch examples/scripts/dpo_vlm.py \\\n --dataset_name HuggingFaceH4/rlaif-v_formatted \\\n --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 32 \\\n --dataset_num_proc 32 \\\n --output_dir dpo_qwen_2_5_rlaif-v \\\n --dtype bfloat16 \\\n --use_peft \\\n --lora_target_modules all-linear\n```\n\nWith dataset streaming:\n\n```\naccelerate launch examples/scripts/dpo_vlm.py \\\n --dataset_name HuggingFaceH4/rlaif-v_formatted \\\n --dataset_streaming \\\n --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --per_device_train_batch_size 2 \\\n --max_steps 100 \\\n --gradient_accumulation_steps 32 \\\n --dataset_num_proc 32 \\\n --output_dir dpo_qwen_2_5_rlaif-v \\\n --dtype bfloat16 \\\n --use_peft \\\n --lora_target_modules all-linear\n```\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForImageTextToText, AutoProcessor\n\nfrom trl import (\n DPOConfig,\n DPOTrainer,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, DPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n\n ################\n # Model & Processor\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n model = AutoModelForImageTextToText.from_pretrained(\n model_args.model_name_or_path,\n trust_remote_code=model_args.trust_remote_code,\n **model_kwargs,\n )\n peft_config = get_peft_config(model_args)\n\n processor = AutoProcessor.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, do_image_splitting=False\n )\n\n if script_args.ignore_bias_buffers:\n # torch distributed hack\n model._ddp_params_and_buffers_to_ignore = [\n name for name, buffer in model.named_buffers() if buffer.dtype == torch.bool\n ]\n\n ################\n # Dataset\n ################\n dataset = load_dataset(\n script_args.dataset_name,\n name=script_args.dataset_config,\n streaming=script_args.dataset_streaming,\n )\n\n ################\n # Training\n ################\n trainer = DPOTrainer(\n model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=peft_config,\n )\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/evals/judge_tldr.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[vllm]\",\n# ]\n# ///\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import load_dataset\nfrom transformers import HfArgumentParser\nfrom vllm import LLM, SamplingParams\n\nfrom trl.experimental.judges import HfPairwiseJudge, OpenAIPairwiseJudge\n\n\n\"\"\"\nExamples:\n\npython examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/rloo_tldr --num_examples 1000\nModel win rate: 31.40%\n\npython examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/rloo_tldr --judge_model gpt-3.5-turbo-0125 --num_examples 1000\nModel win rate: 51.60%\n\npython examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/rloo_tldr --judge_model gpt-4o-mini --num_examples 1000\nModel win rate: 51.20%\n\npython examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/ppo_tldr --num_examples 1000\nModel win rate: 46.30%\n\npython examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/ppo_tldr --judge_model gpt-3.5-turbo-0125 --num_examples 1000\nModel win rate: 52.50%\n\npython examples/scripts/evals/judge_tldr.py --model_name_or_path trl-lib/ppo_tldr --judge_model gpt-4o-mini --num_examples 1000\nModel win rate: 63.00%\n\"\"\"\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n model_name_or_path (`str`):\n Model name or path to the model to evaluate.\n judge_model (`str`, *optional*, defaults to `\"meta-llama/Meta-Llama-3-70B-Instruct\"`):\n Model name or path to the model to use as a judge. E.g., 'gpt-3.5-turbo-0125' or\n 'meta-llama/Meta-Llama-3-70B-Instruct'.\n num_examples (`int`, *optional*):\n Number of examples to evaluate.\n \"\"\"\n\n model_name_or_path: str = field(metadata={\"help\": \"Model name or path to the model to evaluate.\"})\n judge_model: str = field(\n default=\"meta-llama/Meta-Llama-3-70B-Instruct\",\n metadata={\n \"help\": \"Model name or path to the model to use as a judge. E.g., 'gpt-3.5-turbo-0125' or \"\n \"'meta-llama/Meta-Llama-3-70B-Instruct'.\"\n },\n )\n num_examples: int | None = field(default=None, metadata={\"help\": \"Number of examples to evaluate.\"})\n\n\nif __name__ == \"__main__\":\n # Parse the arguments\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n\n # Load the dataset\n dataset = load_dataset(\"trl-lib/tldr\", split=\"validation\")\n if script_args.num_examples is not None:\n dataset = dataset.select(range(script_args.num_examples))\n\n # Extract the prompts and reference completions\n prompts = dataset[\"prompt\"]\n reference_completions = dataset[\"completion\"]\n\n # Generate the model completions\n sampling_params = SamplingParams(temperature=0.0, top_p=0.95, max_tokens=200) # very generous max token length\n llm = LLM(model=script_args.model_name_or_path, tensor_parallel_size=1)\n outputs = llm.generate(prompts, sampling_params)\n model_completions = [output.outputs[0].text.strip() for output in outputs]\n\n # Judge the outputs\n if \"gpt\" in script_args.judge_model:\n judge = OpenAIPairwiseJudge(script_args.judge_model)\n else:\n judge = HfPairwiseJudge(script_args.judge_model)\n\n completions = [[c0, c1] for c0, c1 in zip(reference_completions, model_completions, strict=True)]\n best_idxs = judge.judge(prompts, completions)\n model_win_rate = best_idxs.count(1) / len(best_idxs)\n print(f\"Model win rate: {model_win_rate * 100:.2f}%\")\n" }, { "path": "examples/scripts/gkd.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\n# Full training:\npython examples/scripts/gkd.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --teacher_model_name_or_path Qwen/Qwen2-1.5B-Instruct \\\n --dataset_name trl-lib/chatbot_arena_completions \\\n --learning_rate 2e-5 \\\n --per_device_train_batch_size 4 \\\n --gradient_accumulation_steps 8 \\\n --output_dir gkd-model \\\n --num_train_epochs 1 \\\n --push_to_hub\n\n# LoRA:\npython examples/scripts/gkd.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --teacher_model_name_or_path Qwen/Qwen2-1.5B-Instruct \\\n --dataset_name trl-lib/chatbot_arena_completions \\\n --learning_rate 2e-4 \\\n --per_device_train_batch_size 4 \\\n --gradient_accumulation_steps 8 \\\n --output_dir gkd-model \\\n --num_train_epochs 1 \\\n --push_to_hub \\\n --use_peft \\\n --lora_r 64 \\\n --lora_alpha 16\n\"\"\"\n\nimport os\n\nfrom datasets import load_dataset\nfrom transformers import AutoTokenizer, GenerationConfig\n\nfrom trl import (\n LogCompletionsCallback,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\nfrom trl.experimental.gkd import GKDConfig, GKDTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, GKDConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n\n ################\n # Model & Tokenizer\n ################\n model_kwargs = dict(\n revision=model_args.model_revision,\n trust_remote_code=model_args.trust_remote_code,\n attn_implementation=model_args.attn_implementation,\n dtype=model_args.dtype,\n use_cache=False if training_args.gradient_checkpointing else True,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n training_args.model_init_kwargs = model_kwargs\n\n teacher_model_kwargs = dict(\n revision=model_args.model_revision,\n trust_remote_code=model_args.trust_remote_code,\n attn_implementation=model_args.attn_implementation,\n dtype=model_args.dtype,\n use_cache=True,\n )\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n training_args.teacher_model_init_kwargs = teacher_model_kwargs\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path,\n revision=model_args.model_revision,\n trust_remote_code=model_args.trust_remote_code,\n padding_side=\"left\",\n )\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n ################\n # Dataset\n ################\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n ################\n # Training\n ################\n trainer = GKDTrainer(\n model=model_args.model_name_or_path,\n teacher_model=training_args.teacher_model_name_or_path,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n processing_class=tokenizer,\n peft_config=get_peft_config(model_args),\n )\n\n if training_args.eval_strategy != \"no\":\n generation_config = GenerationConfig(\n max_new_tokens=training_args.max_new_tokens, do_sample=True, temperature=training_args.temperature\n )\n completions_callback = LogCompletionsCallback(trainer, generation_config, num_prompts=8)\n trainer.add_callback(completions_callback)\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/grpo_2048.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[peft]\",\n# ]\n# ///\n\nimport random\n\nfrom datasets import Dataset\nfrom peft import LoraConfig\n\nfrom trl import GRPOConfig, GRPOTrainer\n\n\nPROMPT = \"Play 2048 on a 4x4 board. Use the tool `move` with one of: up, down, left, right. Maximize the score.\"\n\n\nclass Game2048Env:\n def reset(self, **kwargs) -> str:\n self.board = [[0] * 4 for _ in range(4)]\n self.score = 0.0\n self.done = False\n self._spawn()\n self._spawn()\n return f\"score={self.score}\\n{self._render()}\\ndone={self.done}\"\n\n def move(self, direction: str) -> str:\n \"\"\"\n Play one move in 2048.\n\n Args:\n direction: One of \"up\", \"down\", \"left\", \"right\".\n\n Returns:\n Environment feedback after the move.\n \"\"\"\n if self.done:\n raise ValueError(\"Game over.\")\n moved, gained = self._apply_move(direction.strip().lower())\n if moved:\n self.score += gained\n self._spawn()\n self.done = not self._can_move()\n return f\"score={self.score}\\n{self._render()}\\ndone={self.done}\"\n\n def _spawn(self) -> None:\n empty = [(r, c) for r in range(4) for c in range(4) if self.board[r][c] == 0]\n if not empty:\n return\n r, c = random.choice(empty)\n self.board[r][c] = 4 if random.random() < 0.1 else 2\n\n @staticmethod\n def _merge_line(line: list[int]) -> tuple[list[int], int]:\n vals = [x for x in line if x]\n out = []\n gained = 0\n i = 0\n while i < len(vals):\n if i + 1 < len(vals) and vals[i] == vals[i + 1]:\n v = vals[i] * 2\n out.append(v)\n gained += v\n i += 2\n else:\n out.append(vals[i])\n i += 1\n out += [0] * (4 - len(out))\n return out, gained\n\n def _apply_move(self, direction: str) -> tuple[bool, int]:\n if direction not in {\"up\", \"down\", \"left\", \"right\"}:\n return False, 0\n\n before = [row[:] for row in self.board]\n gained_total = 0\n\n if direction in {\"left\", \"right\"}:\n for r in range(4):\n row = self.board[r][:]\n if direction == \"right\":\n row.reverse()\n merged, gained = self._merge_line(row)\n if direction == \"right\":\n merged.reverse()\n self.board[r] = merged\n gained_total += gained\n else:\n for c in range(4):\n col = [self.board[r][c] for r in range(4)]\n if direction == \"down\":\n col.reverse()\n merged, gained = self._merge_line(col)\n if direction == \"down\":\n merged.reverse()\n for r in range(4):\n self.board[r][c] = merged[r]\n gained_total += gained\n\n moved = self.board != before\n return moved, gained_total\n\n def _can_move(self) -> bool:\n if any(0 in row for row in self.board):\n return True\n for r in range(4):\n for c in range(4):\n if r + 1 < 4 and self.board[r][c] == self.board[r + 1][c]:\n return True\n if c + 1 < 4 and self.board[r][c] == self.board[r][c + 1]:\n return True\n return False\n\n def _render(self) -> str:\n return \"\\n\".join(\" \".join(f\"{v:3d}\" for v in row) for row in self.board)\n\n\ndef reward_score(environments, **kwargs):\n return [env.score for env in environments]\n\n\ndef main() -> None:\n dataset = Dataset.from_dict({\"prompt\": [[{\"role\": \"user\", \"content\": PROMPT}] for _ in range(1000)]})\n\n trainer = GRPOTrainer(\n model=\"Qwen/Qwen3-4B\",\n train_dataset=dataset,\n reward_funcs=reward_score,\n args=GRPOConfig(\n chat_template_kwargs={\"enable_thinking\": False},\n logging_steps=1,\n log_completions=True,\n num_completions_to_print=2,\n report_to=\"trackio\",\n trackio_space_id=\"trl-2048\",\n max_completion_length=2048,\n per_device_train_batch_size=4,\n gradient_accumulation_steps=2,\n ),\n environment_factory=Game2048Env,\n peft_config=LoraConfig(),\n )\n trainer.train()\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/grpo_agent.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\n# Full training\n```\npython examples/scripts/grpo_agent.py \\\n --model_name_or_path Qwen/Qwen3-1.7B \\\n --output_dir grpo_biogrid_qwen_3g-1.7b \\\n --push_to_hub True \\\n --use_vllm True \\\n --vllm_mode colocate \\\n --max_completion_length 1024 \\\n --report_to trackio \\\n --log_completions True \\\n --max_steps 400\n```\n\"\"\"\n\nimport os\nimport re\nimport signal\nimport sqlite3\nimport textwrap\nfrom contextlib import contextmanager\n\nfrom datasets import load_dataset\n\nfrom trl import GRPOConfig, GRPOTrainer, ModelConfig, ScriptArguments, TrlParser\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef query_reward(completions, answer, **kwargs):\n \"\"\"\n Reward query strategy:\n - Penalize more than 2 queries\n - Penalize generic queries (LIMIT 1 / PRAGMA)\n - Reward usage of WHERE\n - Reward evidence supporting the final answer\n \"\"\"\n rewards = []\n\n for completion, ans in zip(completions, answer, strict=False):\n reward = 0.0\n sql_queries = []\n tool_results = []\n\n # collect all SQL queries and tool results\n for turn in completion:\n if turn.get(\"tool_calls\"):\n for call in turn[\"tool_calls\"]:\n sql = call[\"function\"][\"arguments\"].get(\"sql_command\", \"\").lower()\n sql_queries.append(sql)\n if turn.get(\"role\") == \"tool\" and turn.get(\"content\"):\n tool_results.append(turn[\"content\"])\n\n # --- penalize too many queries ---\n if len(sql_queries) > 3:\n reward -= 1.5\n\n # --- check query quality ---\n where_count = 0\n for q in sql_queries:\n if \"limit 1\" in q:\n reward -= 1.0\n if \" where \" not in q:\n reward -= 0.5\n else:\n where_count += 1\n reward += min(where_count, 3) * 0.4 # small bonus for WHERE usage\n\n # --- evidence check: do queries support the answer? ---\n combined_results = []\n error_detected = False\n\n for res in tool_results:\n if isinstance(res, dict) and \"error\" in res:\n error_detected = True\n elif isinstance(res, list):\n combined_results.extend(res)\n\n # if error detected, penalize heavily\n if error_detected:\n reward -= 2.0\n elif len(sql_queries) == 0:\n reward -= 1.5\n else:\n has_hits = len(combined_results) > 0\n correct_answer = ans.lower()\n if (has_hits and correct_answer == \"yes\") or (not has_hits and correct_answer == \"no\"):\n reward += 2.0\n else:\n reward -= 1.5\n\n rewards.append(reward)\n\n return rewards\n\n\ndef correctness_reward(completions, answer, **kwargs):\n \"\"\"\n Reward Yes/No correctness.\n Model must provide final answer enclosed in stars โ€” *yes* or *no*.\n Does not reward informal yes/no buried in text.\n \"\"\"\n rewards = []\n for completion, ans in zip(completions, answer, strict=False):\n raw = completion[-1][\"content\"].lower()\n\n # detect form *yes* or *no*\n match = re.search(r\"\\*(yes|no)\\*\", raw)\n guess = match.group(1) if match else None\n\n reward = 0.0\n\n if guess is None:\n reward -= 0.5 # invalid format\n elif guess == ans.lower():\n reward += 0.6 # correct under required format\n else:\n reward -= 1.0 # wrong answer\n\n rewards.append(reward)\n\n return rewards\n\n\ndef structure_reward(completions, **kwargs):\n \"\"\"\n Reward proper assistant structure.\n Encourages a logical sequence: tool call + response + optional extra content.\n \"\"\"\n rewards = []\n\n for completion in completions:\n has_call = False\n has_response = False\n has_other = False\n\n for turn in completion:\n role = turn.get(\"role\")\n if role == \"assistant\" and turn.get(\"tool_calls\"):\n has_call = True\n elif role == \"tool\":\n has_response = True\n else:\n content = turn.get(\"content\")\n if content and content.strip() not in [\"\", \"\"]:\n has_other = True\n\n # Reward sequences\n if has_call and has_response:\n if has_other:\n reward = 0.1\n else:\n reward = 0.05 # still positive even without extra text\n elif has_call and not has_response:\n reward = -0.15\n else:\n reward = 0.0 # neutral if no call\n\n rewards.append(reward)\n\n return rewards\n\n\n# ------------------------\n# Database tool function\n# ------------------------\nclass TimeoutError(Exception):\n \"\"\"Raised when a function call times out.\"\"\"\n\n pass\n\n\n@contextmanager\ndef timeout(seconds):\n \"\"\"Context manager that raises TimeoutError if execution exceeds time limit.\"\"\"\n\n def timeout_handler(signum, frame):\n raise TimeoutError(f\"Operation timed out after {seconds} seconds\")\n\n signal.signal(signal.SIGALRM, timeout_handler)\n signal.alarm(seconds)\n try:\n yield\n finally:\n signal.alarm(0)\n\n\ndef query_biogrid(sql_command: str) -> list[tuple]:\n \"\"\"\n Execute a read-only SQL command on the BioGRID database.\n\n BioGRID is a curated biological database that compiles protein, genetic, and chemical interactions from multiple organisms. It provides researchers with experimentally verified interaction data to support studies in systems biology and functional genomics.\n\n Args:\n sql_command: The SQL command to execute.\n\n Returns:\n A list of tuples containing the query results.\n \"\"\"\n with timeout(5):\n conn = sqlite3.connect(\"file:biogrid.db?mode=ro\", uri=True)\n cursor = conn.cursor()\n try:\n cursor.execute(sql_command)\n results = cursor.fetchall()\n finally:\n conn.close()\n return results\n\n\n# ------------------------\n# Dataset formatting\n# ------------------------\ndef format_example(example):\n question = example[\"question\"]\n preamble = textwrap.dedent(\"\"\"\\\n You have access to the BioGRID SQLite database.\n Use SQL queries to retrieve only the information needed to answer the question.\n\n Genes may appear in the database in columns `Alt_IDs_Interactor_A` `Alt_IDs_Interactor_B`, `Aliases_Interactor_A` and `Aliases_Interactor_B`,\n and each entry can contain multiple gene names or synonyms separated by '|', for example:\n 'entrez gene/locuslink:JNKK(gene name synonym)|entrez gene/locuslink:MAPKK4(gene name synonym)|...'\n So a gene like 'JNKK' or 'MAPKK4' may appear inside one of these strings.\n\n If the database schema is unclear or you are unsure about column names:\n - First inspect the schema with `PRAGMA table_info(interactions);`\n - Or preview a few rows with `SELECT * FROM interactions LIMIT 1;`\n\n Otherwise, directly query the required data.\n\n Final answer must be enclosed in stars, e.g. *Yes* or *No*.\n Facts:\n - The NCBI Taxonomy identifier for humans is taxid:9606.\n \"\"\")\n content = f\"{preamble}\\nQuestion: {question}\"\n prompt = [{\"role\": \"user\", \"content\": content}]\n return {\"prompt\": prompt}\n\n\n# ------------------------\n# Main\n# ------------------------\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, GRPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n\n # ------------------------\n # Create DB\n # ------------------------\n print(\"Creating biogrid.db...\")\n # Load dataset\n biogrid_dataset = load_dataset(\"qgallouedec/biogrid\", split=\"train\")\n df = biogrid_dataset.to_pandas()\n\n # Normalize column names: remove spaces, replace with underscores\n df.columns = [c.replace(\" \", \"_\") for c in df.columns]\n conn = sqlite3.connect(\"biogrid.db\")\n try:\n df.to_sql(\"interactions\", conn, if_exists=\"replace\", index=False)\n print(f\"biogrid.db created. Rows stored: {len(df)}\")\n finally:\n conn.close()\n\n # ------------------------\n # Load and format dataset\n # ------------------------\n dataset = load_dataset(\"qgallouedec/biogrid_qa\", split=\"train\")\n dataset = dataset.filter(\n lambda example: example[\"question\"].startswith(\"Does the gene \")\n ) # keep only simple questions for example\n dataset = dataset.map(format_example, remove_columns=[\"question\"])\n\n train_dataset = dataset\n eval_dataset = None # No eval by default, can be added if needed\n\n training_args.chat_template_kwargs = {\"enable_thinking\": False}\n\n # ------------------------\n # Initialize trainer\n # ------------------------\n trainer = GRPOTrainer(\n model=model_args.model_name_or_path,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n tools=[query_biogrid],\n reward_funcs=[correctness_reward, structure_reward, query_reward],\n args=training_args,\n )\n\n # ------------------------\n # Train\n # ------------------------\n trainer.train()\n\n # ------------------------\n # Save and push\n # ------------------------\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/grpo_vlm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"Pillow\",\n# \"peft\",\n# \"math-verify\",\n# \"latex2sympy2_extended\",\n# \"torchvision\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\npip install math_verify\n\n# For Qwen/Qwen2.5-VL-3B-Instruct\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/grpo_vlm.py \\\n --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --output_dir grpo-Qwen2.5-VL-3B-Instruct \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_completion_length 1024 \\\n --use_vllm \\\n --vllm_mode colocate \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --log_completions\n\n# For HuggingFaceTB/SmolVLM2-2.2B-Instruct\npip install num2words==0.5.14\n\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/grpo_vlm.py \\\n --model_name_or_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \\\n --output_dir grpo-SmolVLM2-2.2B-Instruct \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_completion_length 1024 \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --log_completions \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 2 \\\n --num_generations 2\n\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\n\nfrom trl import (\n GRPOConfig,\n GRPOTrainer,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\nfrom trl.rewards import accuracy_reward, think_format_reward\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, GRPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n ################\n # Model\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n training_args.model_init_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n training_args.model_init_kwargs[\"device_map\"] = get_kbit_device_map()\n training_args.model_init_kwargs[\"quantization_config\"] = quantization_config\n\n ################\n # Dataset\n ################\n dataset = load_dataset(\"lmms-lab/multimodal-open-r1-8k-verified\", split=\"train\")\n dataset = dataset.train_test_split(test_size=100, seed=42)\n\n SYSTEM_PROMPT = (\n \"A conversation between user and assistant. The user asks a question, and the assistant solves it. The \"\n \"assistant first thinks about the reasoning process in the mind and then provides the user with the answer. \"\n \"The reasoning process and answer are enclosed within tags, i.e., \\nThis is my \"\n \"reasoning.\\n\\nThis is my answer.\"\n )\n\n def make_conversation(example):\n prompt = [\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": example[\"problem\"]},\n ]\n return {\"prompt\": prompt}\n\n dataset = dataset.map(make_conversation)\n\n # Filter have big images\n def filter_big_images(example):\n image = example[\"image\"]\n return image.size[0] < 512 and image.size[1] < 512\n\n dataset = dataset.filter(filter_big_images)\n\n def convert_to_rgb(example):\n image = example[\"image\"]\n if image.mode != \"RGB\":\n image = image.convert(\"RGB\")\n example[\"image\"] = image\n return example\n\n dataset = dataset.map(convert_to_rgb)\n\n train_dataset = dataset[\"train\"]\n eval_dataset = dataset[\"test\"] if training_args.eval_strategy != \"no\" else None\n\n ################\n # Training\n ################\n trainer = GRPOTrainer(\n model=model_args.model_name_or_path,\n args=training_args,\n reward_funcs=[think_format_reward, accuracy_reward],\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n peft_config=get_peft_config(model_args),\n )\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/gspo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"math-verify\",\n# \"latex2sympy2_extended\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\npip install math_verify\n\n# For Qwen/Qwen3-0.6B\npip install num2words==0.5.14\n\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/gspo.py \\\n --model_name_or_path Qwen/Qwen3-0.6B \\\n --output_dir gspo-Qwen3-0.6B \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_completion_length 1024 \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --log_completions \\\n --per_device_train_batch_size 8 \\\n --num_generations 8 \\\n --importance_sampling_level sequence \\\n --epsilon 3e-4 \\\n --epsilon_high 4e-4 \\\n --beta 0.0 \\\n --loss_type grpo \\\n --gradient_accumulation_steps 2 \\\n --steps_per_generation 8\n\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\n\nfrom trl import (\n GRPOConfig,\n GRPOTrainer,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\nfrom trl.rewards import accuracy_reward, think_format_reward\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, GRPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n ################\n # Model & Processor\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n training_args.model_init_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n training_args.model_init_kwargs[\"device_map\"] = get_kbit_device_map()\n training_args.model_init_kwargs[\"quantization_config\"] = quantization_config\n\n ################\n # Dataset\n ################\n train_dataset, eval_dataset = load_dataset(\"AI-MO/NuminaMath-TIR\", split=[\"train[:5%]\", \"test[:5%]\"])\n\n SYSTEM_PROMPT = (\n \"A conversation between user and assistant. The user asks a question, and the assistant solves it. The \"\n \"assistant first thinks about the reasoning process in the mind and then provides the user with the answer. \"\n \"The reasoning process and answer are enclosed within tags, i.e., \\nThis is my \"\n \"reasoning.\\n\\nThis is my answer.\"\n )\n\n def make_conversation(example):\n return {\n \"prompt\": [\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": example[\"problem\"]},\n ],\n }\n\n train_dataset = train_dataset.map(make_conversation)\n eval_dataset = eval_dataset.map(make_conversation)\n\n train_dataset = train_dataset.remove_columns([\"messages\", \"problem\"])\n eval_dataset = eval_dataset.remove_columns([\"messages\", \"problem\"])\n\n ################\n # Training\n ################\n trainer = GRPOTrainer(\n model=model_args.model_name_or_path,\n args=training_args,\n reward_funcs=[think_format_reward, accuracy_reward],\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n peft_config=get_peft_config(model_args),\n )\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/gspo_vlm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"Pillow\",\n# \"peft\",\n# \"math-verify\",\n# \"latex2sympy2_extended\",\n# \"torchvision\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\npip install math_verify\n\n# For Qwen/Qwen2.5-VL-3B-Instruct\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/gspo_vlm.py \\\n --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --output_dir gspo-Qwen2.5-VL-3B-Instruct \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_completion_length 1024 \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --log_completions \\\n --per_device_train_batch_size 8 \\\n --num_generations 8 \\\n --importance_sampling_level sequence \\\n --epsilon 3e-4 \\\n --epsilon_high 4e-4 \\\n --beta 0.0 \\\n --loss_type grpo \\\n --gradient_accumulation_steps 2 \\\n --steps_per_generation 8\n\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\n\nfrom trl import (\n GRPOConfig,\n GRPOTrainer,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\nfrom trl.rewards import accuracy_reward, think_format_reward\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, GRPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n ################\n # Model\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n training_args.model_init_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n training_args.model_init_kwargs[\"device_map\"] = get_kbit_device_map()\n training_args.model_init_kwargs[\"quantization_config\"] = quantization_config\n\n ################\n # Dataset\n ################\n dataset = load_dataset(\"lmms-lab/multimodal-open-r1-8k-verified\", split=\"train\")\n dataset = dataset.train_test_split(test_size=100, seed=42)\n\n SYSTEM_PROMPT = (\n \"A conversation between user and assistant. The user asks a question, and the assistant solves it. The \"\n \"assistant first thinks about the reasoning process in the mind and then provides the user with the answer. \"\n \"The reasoning process and answer are enclosed within tags, i.e., \\nThis is my \"\n \"reasoning.\\n\\nThis is my answer.\"\n )\n\n def make_conversation(example):\n prompt = [\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": example[\"problem\"]},\n ]\n return {\"prompt\": prompt}\n\n dataset = dataset.map(make_conversation)\n\n # Filter have big images\n def filter_big_images(example):\n image = example[\"image\"]\n return image.size[0] < 512 and image.size[1] < 512\n\n dataset = dataset.filter(filter_big_images)\n\n def convert_to_rgb(example):\n image = example[\"image\"]\n if image.mode != \"RGB\":\n image = image.convert(\"RGB\")\n example[\"image\"] = image\n return example\n\n dataset = dataset.map(convert_to_rgb)\n\n train_dataset = dataset[\"train\"]\n eval_dataset = dataset[\"test\"] if training_args.eval_strategy != \"no\" else None\n\n ################\n # Training\n ################\n trainer = GRPOTrainer(\n model=model_args.model_name_or_path,\n args=training_args,\n reward_funcs=[think_format_reward, accuracy_reward],\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n peft_config=get_peft_config(model_args),\n )\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/kto.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nRun the KTO training script with the commands below. In general, the optimal configuration for KTO will be similar to that of DPO.\n\n# Full training:\npython trl/scripts/kto.py \\\n --dataset_name trl-lib/kto-mix-14k \\\n --model_name_or_path trl-lib/qwen1.5-1.8b-sft \\\n --per_device_train_batch_size 16 \\\n --num_train_epochs 1 \\\n --learning_rate 5e-7 \\\n --lr_scheduler_type cosine \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 500 \\\n --output_dir kto-aligned-model \\\n --warmup_steps 0.1 \\\n --logging_first_step\n\n# QLoRA:\npython trl/scripts/kto.py \\\n --dataset_name trl-lib/kto-mix-14k \\\n --model_name_or_path trl-lib/qwen1.5-1.8b-sft \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 5e-7 \\\n --lr_scheduler_type cosine \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 500 \\\n --output_dir kto-aligned-model-lora \\\n --warmup_steps 0.1 \\\n --logging_first_step \\\n --use_peft \\\n --load_in_4bit \\\n --lora_target_modules all-linear \\\n --lora_r 16 \\\n --lora_alpha 16\n\"\"\"\n\nimport os\n\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser\n\nfrom trl import ModelConfig, ScriptArguments, get_peft_config\nfrom trl.experimental.kto import KTOConfig, KTOTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser((ScriptArguments, KTOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_into_dataclasses()\n\n # Load a pretrained model\n model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n ref_model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n # Load the dataset\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n # Initialize the KTO trainer\n trainer = KTOTrainer(\n model,\n ref_model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n processing_class=tokenizer,\n peft_config=get_peft_config(model_args),\n )\n\n # Train and push the model to the Hub\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/mpo_vlm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"Pillow\",\n# \"peft\",\n# \"torchvision\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\npython examples/scripts/mpo_vlm.py \\\n --dataset_name HuggingFaceH4/rlaif-v_formatted \\\n --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --per_device_train_batch_size 4 \\\n --per_device_eval_batch_size 4 \\\n --num_train_epochs 1 \\\n --gradient_accumulation_steps 8 \\\n --dataset_num_proc 1 \\\n --output_dir dpo_idefics_rlaif-v \\\n --dtype bfloat16 \\\n --use_peft \\\n --lora_target_modules down_proj, o_proj, k_proj, q_proj, gate_proj, up_proj, v_proj \\\n --loss_type sigmoid bco_pair sft \\\n --loss_weights 0.8 0.2 1.0\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\nfrom PIL import Image\nfrom transformers import AutoModelForImageTextToText\n\nfrom trl import (\n DPOConfig,\n DPOTrainer,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, DPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n\n ################\n # Model & Processor\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n\n model_kwargs = dict(\n trust_remote_code=model_args.trust_remote_code,\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n model = AutoModelForImageTextToText.from_pretrained(\n model_args.model_name_or_path,\n **model_kwargs,\n )\n peft_config = get_peft_config(model_args)\n\n ################\n # Dataset\n ################\n dataset = load_dataset(\n script_args.dataset_name,\n name=script_args.dataset_config,\n streaming=script_args.dataset_streaming,\n )\n train_dataset = dataset[script_args.dataset_train_split]\n test_dataset = dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None\n\n def ensure_rgb(example):\n # Convert the image to RGB if it's not already\n image = example[\"images\"][0]\n if isinstance(image, Image.Image):\n if image.mode != \"RGB\":\n image = image.convert(\"RGB\")\n example[\"images\"] = [image]\n return example\n\n # Apply the transformation to the dataset (change num_proc depending on the available compute)\n train_dataset = train_dataset.map(ensure_rgb, num_proc=training_args.dataset_num_proc)\n if test_dataset is not None:\n test_dataset = test_dataset.map(ensure_rgb, num_proc=training_args.dataset_num_proc)\n\n ################\n # Training\n ################\n trainer = DPOTrainer(\n model=model,\n args=training_args,\n train_dataset=train_dataset,\n eval_dataset=test_dataset,\n peft_config=peft_config,\n )\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/nash_md.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nUsage:\n\npython examples/scripts/nash_md.py \\\n --model_name_or_path trl-lib/pythia-1b-deduped-tldr-sft \\\n --reward_model_path trl-lib/pythia-1b-deduped-tldr-rm \\\n --dataset_name trl-lib/tldr \\\n --learning_rate 5.0e-7 \\\n --output_dir pythia-1b-tldr-nash-md \\\n --per_device_train_batch_size 4 \\\n --gradient_accumulation_steps 32 \\\n --num_train_epochs 3 \\\n --max_new_tokens 64 \\\n --warmup_steps 0.1 \\\n --missing_eos_penalty 1.0 \\\n --push_to_hub\n\n\naccelerate launch --config_file examples/accelerate_configs/deepspeed_zero2.yaml \\\n examples/scripts/nash_md.py \\\n --model_name_or_path trl-lib/pythia-1b-deduped-tldr-sft \\\n --reward_model_path trl-lib/pythia-1b-deduped-tldr-rm \\\n --dataset_name trl-lib/tldr \\\n --learning_rate 5.0e-7 \\\n --output_dir pythia-1b-tldr-nash-md \\\n --per_device_train_batch_size 4 \\\n --gradient_accumulation_steps 32 \\\n --num_train_epochs 3 \\\n --max_new_tokens 64 \\\n --warmup_steps 0.1 \\\n --missing_eos_penalty 1.0 \\\n --push_to_hub\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer, GenerationConfig\n\nfrom trl import (\n LogCompletionsCallback,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_quantization_config,\n)\nfrom trl.experimental.judges import HfPairwiseJudge, OpenAIPairwiseJudge, PairRMJudge\nfrom trl.experimental.nash_md import NashMDConfig, NashMDTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nJUDGES = {\"pair_rm\": PairRMJudge, \"openai\": OpenAIPairwiseJudge, \"hf\": HfPairwiseJudge}\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, NashMDConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n training_args.gradient_checkpointing_kwargs = {\"use_reentrant\": True}\n\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n use_cache=False if training_args.gradient_checkpointing else True,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n ref_model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n\n if training_args.reward_model_path is not None:\n reward_model = AutoModelForSequenceClassification.from_pretrained(\n training_args.reward_model_path,\n num_labels=1,\n trust_remote_code=model_args.trust_remote_code,\n **model_kwargs,\n )\n else:\n reward_model = None\n\n if training_args.judge is not None:\n judge_cls = JUDGES[training_args.judge]\n judge = judge_cls()\n else:\n judge = None\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, padding_side=\"left\", trust_remote_code=model_args.trust_remote_code\n )\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n trainer = NashMDTrainer(\n model=model,\n ref_model=ref_model,\n reward_funcs=reward_model,\n judge=judge,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n processing_class=tokenizer,\n )\n\n if training_args.eval_strategy != \"no\":\n generation_config = GenerationConfig(\n max_new_tokens=training_args.max_new_tokens, do_sample=True, temperature=training_args.temperature\n )\n completions_callback = LogCompletionsCallback(trainer, generation_config, num_prompts=8)\n trainer.add_callback(completions_callback)\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/nemo_gym/README.md", "content": "# Post-training with NeMo Gym and TRL\n\nThis integration supports training language models in NeMo-Gym environments using TRL GRPO. Both single step and multi step tasks are supported, including multi-environment training. NeMo-Gym orchestrates rollouts, returning token ids and logprobs to TRL through the rollout function for training. Currently this integration is only supported through TRL's vllm server mode. \n\nCheck out the docs page `docs/source/nemo_gym.md` for a guide. " }, { "path": "examples/scripts/nemo_gym/config.yaml", "content": "# Model\nmodel_name: \"Qwen/Qwen2.5-1.5B-Instruct\"\n\n# Data\ndataset_path: \"/home/ubuntu/Gym/resources_servers/workplace_assistant/data/train.jsonl\"\neval_dataset_path: \"/home/ubuntu/Gym/resources_servers/workplace_assistant/data/validation.jsonl\"\n\n# Logging\noutput_dir: \"outputs/nemo_gym\"\ntask: \"workplace\" # just used in wandb run name \nreport_to: \"wandb\"\nproject_name: \"trl-nemo-gym\"\nlog_completions: true\nnum_completions_to_print: 2\n\n# Training hyperparameters\nlearning_rate: 1.0e-5\nmax_steps: 1000\nnum_generations: 8\nper_device_train_batch_size: 1\ngradient_accumulation_steps: 32\nmax_completion_length: 16384\nwarmup_steps: 5\nlr_scheduler_type: \"linear\"\noptim: \"adamw_torch_fused\"\nweight_decay: 0.0\nvllm_importance_sampling_correction: true\n\n# Inference sampling parameters\ntemperature: 1.0\ntop_p: 0.999\n\n# Checkpointing and Eval\nsave_steps: 10\neval_strategy: \"steps\"\neval_steps: 10\n\n" }, { "path": "examples/scripts/nemo_gym/deepspeed_zero3.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndeepspeed_config:\n deepspeed_multinode_launcher: standard\n offload_optimizer_device: none\n offload_param_device: none\n zero3_init_flag: true\n zero3_save_16bit_model: true\n zero_stage: 3\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 4\nnum_processes: 32\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "examples/scripts/nemo_gym/submit.sh", "content": "#!/bin/bash\n#SBATCH -A account\n#SBATCH -p partition\n#SBATCH -N 5\n#SBATCH --gres gpu:8\n#SBATCH --ntasks-per-node=1\n#SBATCH --cpus-per-task=16\n#SBATCH --time=4:00:00\n#SBATCH --job-name=trl_nemo_gym\n#SBATCH --output=logs/%j/slurm.out\n#SBATCH --error=logs/%j/slurm.err\n\nCONTAINER_IMAGE=\"nvcr.io/nvidia/pytorch:25.12-py3\"\nMOUNTS=\"/path/to/mounts:/path/to/mounts\"\n\nNODELIST=($(scontrol show hostnames $SLURM_JOB_NODELIST))\n\nTRAIN_NODE_0=\"${NODELIST[0]}\"\nTRAIN_NODE_1=\"${NODELIST[1]}\"\nTRAIN_NODE_2=\"${NODELIST[2]}\"\nTRAIN_NODE_3=\"${NODELIST[3]}\"\nVLLM_NODE=\"${NODELIST[4]}\"\n\necho \"Training Nodes: $TRAIN_NODE_0, $TRAIN_NODE_1, $TRAIN_NODE_2, $TRAIN_NODE_3\"\necho \"vLLM Node: $VLLM_NODE\"\necho \"Main process IP: $TRAIN_NODE_0\"\n\nLOG_DIR=\"logs/${SLURM_JOB_ID}\"\nmkdir -p ${LOG_DIR}\n\necho \"Starting ng_run and vLLM on ${VLLM_NODE}...\"\necho \"Logs will be saved to: ${LOG_DIR}\"\n\n# NOTE: If you have already set up your TRL venv, you can remove all of the pip installs and uv venv related commands below!\n\nsrun --nodes=1 --ntasks=1 --nodelist=\"${VLLM_NODE}\" \\\n --container-image=\"${CONTAINER_IMAGE}\" \\\n --container-mounts=\"${MOUNTS}\" \\\n --container-mount-home \\\n bash -c \"\n LOG_DIR=/path/to/logs\n mkdir -p \\${LOG_DIR}\n\n # Install uv if not already installed\n curl -LsSf https://astral.sh/uv/install.sh | sh\n source \\$HOME/.local/bin/env\n\n # Start nemo gym servers\n (set -x && \\\n export HOME=/path/to/user && \\\n export PATH=\\$HOME/.local/bin:\\$PATH && \\\n cd /path/to/user/Gym && \\\n uv venv --python 3.12 && \\\n source .venv/bin/activate && \\\n uv sync && \\\n ray stop --force && \\\n ng_run +config_paths=[responses_api_models/vllm_model/configs/vllm_model.yaml,resources_servers/workplace_assistant/configs/workplace_assistant.yaml] +head_server.host=0.0.0.0 +head_server.port=11000) > \\${LOG_DIR}/ng_run.log 2>&1 &\n\n sleep 10\n\n # Start trl vllm server\n (set -x && \\\n export HOME=/path/to/user && \\\n export HF_HOME=/path/to/user/hf_home && \\\n cd /path/to/user/trl && \\\n rm -rf .venv && uv venv && source .venv/bin/activate && uv sync && uv pip install -e .[vllm] && uv pip install fastapi uvicorn && \\\n python -m trl.scripts.vllm_serve \\\n --model Qwen/Qwen3-4B-Instruct-2507 \\\n --host 0.0.0.0 \\\n --tensor-parallel-size 8 \\\n --data-parallel-size 1 \\\n --max-model-len 16384 \\\n --gpu-memory-utilization 0.7 \\\n --port 8000) > \\${LOG_DIR}/vllm_serve.log 2>&1 &\n\n wait\n\" &\n\necho \"Waiting for nemo gym and vllm to start...\"\nsleep 120\n\necho \"Launching training on 4 nodes...\"\n\nTRAIN_NODES_LIST=\"${TRAIN_NODE_0},${TRAIN_NODE_1},${TRAIN_NODE_2},${TRAIN_NODE_3}\"\n\nsrun --nodes=4 --ntasks=4 --nodelist=\"${TRAIN_NODES_LIST}\" \\\n --container-image=\"${CONTAINER_IMAGE}\" \\\n --container-mounts=\"${MOUNTS}\" \\\n --container-mount-home \\\n bash -c \"\n set -x && \\\n export HOME=/path/to/user && \\\n export HF_HOME=/path/to/user/hf_home && \\\n cd /path/to/user/trl && \\\n source .venv/bin/activate && uv pip install accelerate deepspeed wandb omegaconf && \\\n cd examples/scripts/nemo_gym && \\\n export WANDB_API_KEY= && \\\n accelerate launch \\\n --config_file deepspeed_zero3.yaml \\\n --num_processes 32 \\\n --num_machines 4 \\\n --machine_rank \\$SLURM_PROCID \\\n --main_process_ip ${TRAIN_NODE_0} \\\n --main_process_port 29500 \\\n --rdzv_backend c10d \\\n train_multi_environment.py \\\n --config config.yaml \\\n --vllm_server_host ${VLLM_NODE} \\\n --head_server_host ${VLLM_NODE}\" &\n\nwait\n\n" }, { "path": "examples/scripts/nemo_gym/train_multi_environment.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[vllm]\",\n# \"nemo_gym @ git+https://github.com/NVIDIA-NeMo/Gym\",\n# ]\n# ///\n\nimport argparse\nimport asyncio\nimport json\nimport os\nfrom dataclasses import dataclass\nfrom typing import Any\n\nimport aiohttp\nimport requests\nimport yaml\nfrom datasets import Dataset, load_dataset\nfrom omegaconf import OmegaConf\nfrom transformers import AutoTokenizer\n\nfrom trl import GRPOConfig, GRPOTrainer\n\n\n@dataclass\nclass NeMoGymGRPOConfig(GRPOConfig):\n agent_servers: dict[str, str] | None = None\n request_timeout: float = 10800\n\n\ndef get_agent_servers(\n head_server_host: str = \"127.0.0.1\",\n head_server_port: int = 11000,\n) -> dict[str, str]:\n try:\n response = requests.get(f\"http://{head_server_host}:{head_server_port}/global_config_dict_yaml\", timeout=10)\n response.raise_for_status()\n global_config_yaml = response.text\n global_config_dict = OmegaConf.create(yaml.safe_load(global_config_yaml))\n\n agent_servers = {}\n for server_name, server_config in global_config_dict.items():\n if hasattr(server_config, \"responses_api_agents\"):\n agents = server_config.responses_api_agents\n for agent_key in agents.keys():\n agent_config = getattr(agents, agent_key)\n if hasattr(agent_config, \"host\") and hasattr(agent_config, \"port\"):\n agent_host = agent_config.host\n if agent_host in (\"127.0.0.1\", \"0.0.0.0\", \"localhost\"):\n agent_host = head_server_host\n agent_servers[server_name] = f\"http://{agent_host}:{agent_config.port}\"\n\n if not agent_servers:\n raise ValueError(\"No agents found in global config\")\n\n return agent_servers\n\n except requests.exceptions.RequestException as e:\n raise RuntimeError(f\"Failed to connect to head server at {head_server_host}:{head_server_port}: {e}\") from e\n\n\ndef reward_fn(completions: list[str], **kwargs) -> list[float]:\n env_rewards = kwargs.get(\"env_reward\")\n assert env_rewards is not None, \"env_reward not found in kwargs\"\n return [float(r) for r in env_rewards]\n\n\nasync def call_nemo_gym_agents(\n prompts: list[str],\n dataset_items: list[dict[str, Any]],\n agent_servers: dict[str, str],\n timeout: float,\n max_completion_length: int = 4096,\n temperature: float = 1.0,\n top_p: float = 0.999,\n) -> list[dict[str, Any]]:\n async with aiohttp.ClientSession(cookie_jar=aiohttp.CookieJar()) as session:\n tasks = []\n for prompt, item in zip(prompts, dataset_items, strict=False):\n request_body = item.copy()\n\n if \"responses_create_params\" not in request_body:\n request_body[\"responses_create_params\"] = {\n \"input\": [{\"role\": \"user\", \"content\": prompt}],\n }\n\n params = request_body[\"responses_create_params\"]\n params.setdefault(\"max_output_tokens\", max_completion_length)\n params[\"temperature\"] = temperature\n params[\"top_p\"] = top_p\n\n agent_ref = item.get(\"agent_ref\", {})\n agent_name = agent_ref.get(\"name\") if isinstance(agent_ref, dict) else None\n if not agent_name or agent_name not in agent_servers:\n raise ValueError(\n f\"Missing or invalid agent_ref. Got: {agent_ref}. Available: {list(agent_servers.keys())}\"\n )\n agent_url = agent_servers[agent_name]\n\n task = session.post(\n f\"{agent_url}/run\",\n json=request_body,\n timeout=aiohttp.ClientTimeout(total=timeout),\n )\n tasks.append(task)\n\n responses = await asyncio.gather(*tasks, return_exceptions=True)\n\n results = []\n for i, response in enumerate(responses):\n try:\n if isinstance(response, Exception):\n raise response\n json_data = await response.json()\n if not isinstance(json_data, dict):\n raise ValueError(f\"Expected dict, got {type(json_data)}\")\n results.append(json_data)\n except Exception as e:\n print(f\"WARNING: Request {i} failed: {e}\")\n results.append({\"response\": {\"output\": []}, \"reward\": 0.0, \"error\": str(e)})\n\n return results\n\n\ndef nemo_gym_rollout_func(prompts: list[str], trainer: GRPOTrainer) -> dict[str, list]:\n is_eval = not trainer.model.training\n num_generations = (\n trainer.args.num_generations_eval\n if is_eval and trainer.args.num_generations_eval\n else trainer.args.num_generations\n )\n dataset = trainer.eval_dataset if is_eval and trainer.eval_dataset is not None else trainer.train_dataset\n\n expanded_prompts = []\n expanded_dataset_items = []\n\n for idx_str in prompts:\n idx = int(idx_str)\n item = json.loads(dataset[idx][\"metadata\"])\n\n for _ in range(num_generations):\n expanded_prompts.append(idx_str)\n expanded_dataset_items.append(dict(item))\n\n loop = asyncio.new_event_loop()\n asyncio.set_event_loop(loop)\n try:\n responses = loop.run_until_complete(\n call_nemo_gym_agents(\n expanded_prompts,\n expanded_dataset_items,\n trainer.args.agent_servers,\n trainer.args.request_timeout,\n trainer.args.max_completion_length,\n temperature=trainer.args.temperature,\n top_p=trainer.args.top_p,\n )\n )\n finally:\n loop.close()\n\n tokenizer = trainer.processing_class\n\n prompt_ids: list[list[int]] = []\n completion_ids: list[list[int]] = [] # list of rollouts\n env_mask: list[list[int]] = [] # only train on assistant turns\n\n logprobs: list[list[float]] = []\n env_rewards: list[float] = []\n num_turns_list: list[int] = []\n\n for i, response in enumerate(responses):\n eos_token_id = tokenizer.eos_token_id or 0\n\n if not isinstance(response, dict) or response.get(\"error\"):\n rollout_failed = True\n else:\n output_items = response.get(\"response\", {}).get(\"output\", [])\n has_content = output_items and any(\n item.get(\"type\") == \"function_call\"\n or (\n item.get(\"type\") == \"message\"\n and any(\n c.get(\"type\") == \"output_text\" and c.get(\"text\", \"\").strip() for c in item.get(\"content\", [])\n )\n )\n for item in output_items\n )\n rollout_failed = not has_content\n\n if rollout_failed:\n prompt_ids.append([eos_token_id])\n completion_ids.append([eos_token_id])\n env_mask.append([0])\n logprobs.append([0.0])\n env_rewards.append(0.0)\n num_turns_list.append(0)\n continue\n\n episode_reward = response.get(\"reward\", 0.0)\n output_items = response.get(\"response\", {}).get(\"output\", [])\n\n rollout_ids: list[int] = []\n rollout_mask: list[int] = []\n rollout_logprobs: list[float] = []\n\n seen_token_ids: list[int] = []\n first_prompt = None\n num_turns = 0\n\n for _idx, item in enumerate(output_items):\n if \"prompt_token_ids\" not in item or \"generation_token_ids\" not in item:\n continue\n\n num_turns += 1\n item_prompt_ids = item[\"prompt_token_ids\"]\n item_gen_ids = item[\"generation_token_ids\"]\n item_logprobs = item.get(\"generation_log_probs\", [])\n tool_result_tokens = []\n\n if first_prompt is None:\n first_prompt = item_prompt_ids\n seen_token_ids = list(item_prompt_ids)\n else:\n if len(item_prompt_ids) > len(seen_token_ids):\n if item_prompt_ids[: len(seen_token_ids)] != seen_token_ids:\n raise ValueError(\n f\"[Turn {num_turns}] Non-contiguous messages (tokenization issue). \"\n f\"Expected prefix len {len(seen_token_ids)}, got prompt len {len(item_prompt_ids)}\"\n )\n tool_result_tokens = item_prompt_ids[len(seen_token_ids) :]\n\n if tool_result_tokens:\n rollout_ids.extend(tool_result_tokens)\n rollout_mask.extend([0] * len(tool_result_tokens))\n rollout_logprobs.extend([0.0] * len(tool_result_tokens))\n\n rollout_ids.extend(item_gen_ids)\n rollout_mask.extend([1] * len(item_gen_ids))\n assert len(item_logprobs) == len(item_gen_ids), (\n f\"Logprobs len {len(item_logprobs)} != gen len {len(item_gen_ids)}\"\n )\n rollout_logprobs.extend(item_logprobs)\n\n seen_token_ids = list(item_prompt_ids) + list(item_gen_ids)\n\n if not rollout_ids or first_prompt is None:\n raise ValueError(f\"Rollout {i} has no valid turns\")\n\n prompt_ids.append(first_prompt) # list of prompts\n completion_ids.append(rollout_ids) # list of rollouts\n env_mask.append(rollout_mask)\n logprobs.append(rollout_logprobs)\n env_rewards.append(episode_reward)\n num_turns_list.append(num_turns)\n\n if not prompt_ids:\n raise RuntimeError(\"No valid rollouts. Check Nemo Gym and vLLM logs.\")\n\n if num_turns_list:\n trainer.log(\n {\n \"num_turns_mean\": sum(num_turns_list) / len(num_turns_list),\n \"num_turns_min\": min(num_turns_list),\n \"num_turns_max\": max(num_turns_list),\n }\n )\n\n unique_prompt_ids = prompt_ids[::num_generations]\n\n return {\n \"prompt_ids\": unique_prompt_ids,\n \"completion_ids\": completion_ids,\n \"env_mask\": env_mask,\n \"logprobs\": logprobs,\n \"env_reward\": env_rewards,\n \"num_turns\": num_turns_list,\n }\n\n\ndef load_dataset_from_jsonl(path: str) -> Dataset:\n data = []\n with open(path) as f:\n for idx, line in enumerate(f):\n if line.strip():\n item = json.loads(line)\n data.append(\n {\n \"prompt\": str(\n idx\n ), # use index for lookup as not all nemo gym datasets have the same metadata fields. maybe not the most elegant\n \"metadata\": json.dumps(item),\n }\n )\n return Dataset.from_list(data)\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"\")\n parser.add_argument(\"--config\", required=True, help=\"Path to config YAML file\")\n parser.add_argument(\"--vllm_server_host\", type=str, default=\"127.0.0.1\", help=\"vLLM server hostname/IP\")\n parser.add_argument(\"--head_server_host\", type=str, default=\"127.0.0.1\", help=\"Head server hostname/IP for ng_run\")\n parser.add_argument(\"--resume_from_checkpoint\", type=str, default=None, help=\"Path to checkpoint to resume from\")\n args = parser.parse_args()\n\n with open(args.config) as f:\n config = yaml.safe_load(f)\n\n model_name = config.pop(\"model_name\")\n dataset_path = config.pop(\"dataset_path\")\n eval_dataset_path = config.pop(\"eval_dataset_path\", None)\n task = config.pop(\"task\", None)\n project_name = config.pop(\"project_name\", None)\n\n if \"learning_rate\" in config and isinstance(config[\"learning_rate\"], str):\n config[\"learning_rate\"] = float(config[\"learning_rate\"])\n if \"weight_decay\" in config and isinstance(config[\"weight_decay\"], str):\n config[\"weight_decay\"] = float(config[\"weight_decay\"])\n\n agent_servers = get_agent_servers(\n head_server_host=args.head_server_host,\n head_server_port=11000,\n )\n\n if project_name:\n os.environ[\"WANDB_PROJECT\"] = project_name\n\n if dataset_path.endswith((\".jsonl\", \".json\")):\n dataset = load_dataset_from_jsonl(dataset_path)\n else:\n dataset = load_dataset(dataset_path, split=\"train\")\n\n eval_dataset = None\n if eval_dataset_path:\n eval_dataset = load_dataset_from_jsonl(eval_dataset_path)\n print(f\"Eval dataset has {len(eval_dataset)} examples\\n\")\n\n training_args = NeMoGymGRPOConfig(\n use_vllm=True,\n vllm_mode=\"server\",\n vllm_server_host=args.vllm_server_host,\n vllm_server_port=8000,\n gradient_checkpointing=True,\n num_generations_eval=1,\n logging_steps=1,\n epsilon=0.2,\n epsilon_high=0.28,\n loss_type=\"grpo\",\n mask_truncated_completions=True,\n shuffle_dataset=False,\n model_init_kwargs={\"torch_dtype\": \"auto\"},\n agent_servers=agent_servers,\n request_timeout=10800,\n **config,\n )\n\n if training_args.run_name is None:\n task_name = task or os.path.basename(dataset_path).replace(\".jsonl\", \"\").replace(\".json\", \"\")\n model_short = model_name.split(\"/\")[-1]\n training_args.run_name = (\n f\"{task_name}_{model_short}\"\n f\"_rpp{training_args.num_generations}\"\n f\"_dbs{training_args.per_device_train_batch_size}\"\n f\"_ga{training_args.gradient_accumulation_steps}\"\n f\"_maxlen{training_args.max_completion_length}\"\n f\"_lr{training_args.learning_rate}\"\n f\"_temp{training_args.temperature}\"\n f\"_topp{training_args.top_p}\"\n )\n\n tokenizer = AutoTokenizer.from_pretrained(model_name, truncation_side=\"left\", padding_side=\"left\")\n\n trainer = GRPOTrainer(\n model=model_name,\n processing_class=tokenizer,\n reward_funcs=reward_fn,\n train_dataset=dataset,\n eval_dataset=eval_dataset,\n rollout_func=nemo_gym_rollout_func,\n args=training_args,\n )\n\n trainer.train(resume_from_checkpoint=args.resume_from_checkpoint)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/online_dpo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nUsage:\n\npython examples/scripts/online_dpo.py \\\n --model_name_or_path trl-lib/pythia-1b-deduped-tldr-sft \\\n --reward_model_path trl-lib/pythia-1b-deduped-tldr-rm \\\n --dataset_name trl-lib/tldr \\\n --learning_rate 5.0e-7 \\\n --output_dir pythia-1b-tldr-online-dpo \\\n --per_device_train_batch_size 8 \\\n --gradient_accumulation_steps 16 \\\n --warmup_steps 0.1 \\\n --missing_eos_penalty 1.0\n\nWith LoRA:\npython examples/scripts/online_dpo.py \\\n --model_name_or_path trl-lib/pythia-1b-deduped-tldr-sft \\\n --reward_model_path trl-lib/pythia-1b-deduped-tldr-rm \\\n --dataset_name trl-lib/tldr \\\n --learning_rate 5.0e-6 \\\n --output_dir pythia-1b-tldr-online-dpo \\\n --per_device_train_batch_size 16 \\\n --gradient_accumulation_steps 8 \\\n --warmup_steps 0.1 \\\n --missing_eos_penalty 1.0 \\\n --use_peft\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer, GenerationConfig\n\nfrom trl import (\n LogCompletionsCallback,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\nfrom trl.experimental.judges import HfPairwiseJudge, OpenAIPairwiseJudge, PairRMJudge\nfrom trl.experimental.online_dpo import OnlineDPOConfig, OnlineDPOTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nJUDGES = {\"pair_rm\": PairRMJudge, \"openai\": OpenAIPairwiseJudge, \"hf\": HfPairwiseJudge}\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, OnlineDPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n training_args.gradient_checkpointing_kwargs = {\"use_reentrant\": True}\n\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n use_cache=False if training_args.gradient_checkpointing else True,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n\n if training_args.reward_model_path is not None:\n reward_model = AutoModelForSequenceClassification.from_pretrained(\n training_args.reward_model_path,\n num_labels=1,\n trust_remote_code=model_args.trust_remote_code,\n **model_kwargs,\n )\n reward_tokenizer = AutoTokenizer.from_pretrained(\n training_args.reward_model_path,\n trust_remote_code=model_args.trust_remote_code,\n truncation=True,\n truncation_side=\"left\", # since we judge the completion, truncating left is more appropriate\n )\n if reward_tokenizer.pad_token_id is None:\n reward_tokenizer.pad_token = reward_tokenizer.eos_token\n else:\n reward_model = None\n reward_tokenizer = None\n\n if training_args.judge is not None:\n judge_cls = JUDGES[training_args.judge]\n judge = judge_cls()\n else:\n judge = None\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path,\n padding_side=\"left\",\n trust_remote_code=model_args.trust_remote_code,\n **model_kwargs,\n )\n if tokenizer.pad_token_id is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n trainer = OnlineDPOTrainer(\n model=model,\n reward_funcs=reward_model,\n judge=judge,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n processing_class=tokenizer,\n reward_processing_classes=reward_tokenizer,\n peft_config=get_peft_config(model_args),\n )\n\n if training_args.eval_strategy != \"no\":\n generation_config = GenerationConfig(\n max_new_tokens=training_args.max_new_tokens, do_sample=True, temperature=training_args.temperature\n )\n completions_callback = LogCompletionsCallback(trainer, generation_config, num_prompts=8)\n trainer.add_callback(completions_callback)\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/online_dpo_vlm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"math-verify\",\n# \"latex2sympy2_extended\",\n# \"trackio\",\n# \"torchvision\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\npip install math_verify\n\n# For Qwen/Qwen2.5-VL-3B-Instruct\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/online_dpo_vlm.py \\\n --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --reward_model_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --output_dir online-dpo-Qwen2.5-VL-3B-Instruct \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_length 1536 \\\n --max_new_tokens 1024 \\\n --use_vllm \\\n --vllm_mode server \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 2\n\n# For HuggingFaceTB/SmolVLM2-2.2B-Instruct\npip install num2words==0.5.14\n\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/online_dpo_vlm.py \\\n --model_name_or_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \\\n --reward_model_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \\\n --output_dir online-dpo-SmolVLM2-2.2B-Instruct \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_length 1536 \\\n --max_new_tokens 1024 \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 2\n\n# Single GPU test command:\npython examples/scripts/online_dpo_vlm.py \\\n --model_name_or_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \\\n --reward_model_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \\\n --output_dir online-dpo-SmolVLM2-2.2B-Instruct-test \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_length 1536 \\\n --max_new_tokens 128 \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 1 \\\n --max_steps 2 \\\n --logging_steps 1 \\\n --trust_remote_code\n\"\"\"\n\nimport os\n\nimport torch\nimport transformers\nfrom datasets import load_dataset\nfrom transformers import AutoConfig, AutoProcessor, GenerationConfig\n\nfrom trl import (\n LogCompletionsCallback,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\nfrom trl.experimental.online_dpo import OnlineDPOConfig, OnlineDPOTrainer\nfrom trl.rewards import accuracy_reward, think_format_reward\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, OnlineDPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n training_args.gradient_checkpointing_kwargs = {\"use_reentrant\": True}\n\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n use_cache=False if training_args.gradient_checkpointing else True,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n # Load the VLM model using correct architecture (from GRPO pattern)\n config = AutoConfig.from_pretrained(model_args.model_name_or_path)\n architecture = getattr(transformers, config.architectures[0])\n model = architecture.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n\n # For VLM online DPO, using a reward model is complex because it needs images\n # Instead, we'll use a simple random judge for testing\n # In production, you'd want to use a proper text-only reward model or a custom judge\n reward_model = None\n reward_processor = None\n\n # Load processor for main model\n processor = AutoProcessor.from_pretrained(\n model_args.model_name_or_path,\n trust_remote_code=model_args.trust_remote_code,\n )\n if hasattr(processor, \"tokenizer\"):\n processor.tokenizer.padding_side = \"left\"\n if processor.tokenizer.pad_token_id is None:\n processor.tokenizer.pad_token = processor.tokenizer.eos_token\n\n ################\n # Dataset\n ################\n dataset = load_dataset(\"lmms-lab/multimodal-open-r1-8k-verified\", split=\"train\")\n dataset = dataset.train_test_split(test_size=100, seed=42)\n\n SYSTEM_PROMPT = (\n \"A conversation between user and assistant. The user asks a question, and the assistant solves it. The \"\n \"assistant first thinks about the reasoning process in the mind and then provides the user with the answer. \"\n \"The reasoning process and answer are enclosed within tags, i.e., \\nThis is my \"\n \"reasoning.\\n\\nThis is my answer.\"\n )\n\n def make_conversation(example):\n # Create conversational format that OnlineDPOTrainer expects\n prompt = [\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": example[\"problem\"]},\n ]\n return {\"prompt\": prompt, \"image\": example[\"image\"]}\n\n dataset = dataset.map(make_conversation)\n\n # Filter big images (from GRPO pattern)\n def filter_big_images(example):\n image = example[\"image\"]\n return image.size[0] < 512 and image.size[1] < 512\n\n dataset = dataset.filter(filter_big_images)\n\n def convert_to_rgb(example):\n image = example[\"image\"]\n if image.mode != \"RGB\":\n image = image.convert(\"RGB\")\n example[\"image\"] = image\n return example\n\n dataset = dataset.map(convert_to_rgb)\n\n train_dataset = dataset[\"train\"]\n eval_dataset = dataset[\"test\"] if training_args.eval_strategy != \"no\" else None\n\n ################\n # Training\n ################\n trainer = OnlineDPOTrainer(\n model=model,\n reward_funcs=[think_format_reward, accuracy_reward], # Use same reward functions as GRPO VLM\n args=training_args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processor,\n peft_config=get_peft_config(model_args),\n )\n\n # Add completion logging callback (from online DPO pattern)\n if training_args.eval_strategy != \"no\":\n generation_config = GenerationConfig(\n max_new_tokens=training_args.max_new_tokens, do_sample=True, temperature=training_args.temperature\n )\n completions_callback = LogCompletionsCallback(trainer, generation_config, num_prompts=8)\n trainer.add_callback(completions_callback)\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=\"lmms-lab/multimodal-open-r1-8k-verified\")\n" }, { "path": "examples/scripts/openenv/browsergym.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[vllm]\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# \"openenv-browsergym @ git+https://huggingface.co/spaces/openenv/browsergym_env\",\n# ]\n# ///\n\n\"\"\"\nSimple script to run GRPO training with OpenEnv's BrowserGym environment and vLLM.\n\nThis example automatically detects and uses vision capabilities when VLM models are used.\nScreenshots from BrowserGym are collected and passed to the model during training. The GRPO\ntrainer auto-detects multimodal support by checking for images in the rollout data.\n\nSetup (Option A - Install from HF Space, recommended):\n\n```sh\nuv pip install git+https://huggingface.co/spaces/openenv/browsergym_env\n```\n\nSetup (Option B - Clone OpenEnv repo, for development):\n\n```sh\ngit clone https://github.com/meta-pytorch/OpenEnv.git\ncd OpenEnv/envs/browsergym_env\nuv pip install -e .\n```\n\n# Option 1: HF Spaces + Colocated vLLM (1 GPU required)\n```sh\npython examples/scripts/openenv/browsergym.py --vllm-mode colocate\n```\n\n# Option 2: HF Spaces + Separate vLLM server (2 GPUs required)\n\n# Spin up vLLM server (Terminal 1)\n```sh\nCUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen3-VL-2B-Instruct --host 0.0.0.0 --port 8001\n```\n\n# Run training (Terminal 2)\n```sh\nCUDA_VISIBLE_DEVICES=1 python examples/scripts/openenv/browsergym.py --vllm-mode server --vllm-server-url http://localhost:8001\n```\n\n# Option 3: Local + Colocated vLLM (1 GPU required)\n\n# Build and start the environment only if using --env-mode docker-local\n```sh\ncd OpenEnv\ndocker build -t openenv-base:latest -f src/core/containers/images/Dockerfile .\ndocker build -t browsergym-env:latest -f src/envs/browsergym_env/server/Dockerfile .\ndocker run -d -p 8001:8001 \\\n -e BROWSERGYM_BENCHMARK=\"miniwob\" \\\n -e BROWSERGYM_TASK_NAME=\"click-test\" \\\n browsergym-env:latest\n```\n\n```sh\npython examples/scripts/openenv/browsergym.py --env-mode docker-local --vllm-mode colocate\n```\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nfrom datetime import datetime\nfrom pathlib import Path\n\nimport numpy as np\nfrom browsergym_env import BrowserGymAction, BrowserGymEnv\nfrom datasets import Dataset\nfrom PIL import Image\nfrom transformers import AutoTokenizer\n\nfrom trl import GRPOConfig, GRPOTrainer\nfrom trl.experimental.openenv import generate_rollout_completions\n\n\ndef parse_args() -> argparse.Namespace:\n parser = argparse.ArgumentParser(description=\"Run GRPO training for BrowserGym MiniWoB using OpenEnv environment.\")\n parser.add_argument(\n \"--tokenizer-id\",\n default=\"Qwen/Qwen3-VL-2B-Instruct\",\n help=\"Model identifier used to load the tokenizer.\",\n )\n parser.add_argument(\n \"--model-id\",\n default=\"Qwen/Qwen3-VL-2B-Instruct\",\n help=\"Model identifier passed to GRPOTrainer for fine-tuning.\",\n )\n parser.add_argument(\n \"--env-host\",\n type=str,\n default=\"https://openenv-browsergym-env.hf.space\",\n help=\"Host for the BrowserGym environment.\",\n )\n parser.add_argument(\"--env-port\", type=int, default=8001, help=\"Port for the BrowserGym environment.\")\n parser.add_argument(\n \"--env-mode\",\n choices=[\"docker-local\", \"docker-image\", \"docker-hub\", \"space\"],\n default=\"space\",\n help=\"Where to run the environment: 'local' to launch it, 'docker-local' if already running locally, 'docker-image' to run from a Docker image, 'docker-hub' to run from Docker Hub, or 'space' to use a remote Space URL.\",\n )\n parser.add_argument(\n \"--env-image\", type=str, default=\"browsergym-env:latest\", help=\"Docker image for the BrowserGym environment.\"\n )\n parser.add_argument(\n \"--benchmark\",\n default=\"miniwob\",\n help=\"BrowserGym benchmark to use (miniwob, webarena, etc.).\",\n )\n parser.add_argument(\n \"--task-name\",\n default=\"click-test\",\n help=\"Specific task within the benchmark (e.g., click-test, click-button).\",\n )\n parser.add_argument(\n \"--dataset-prompt\",\n default=\"Complete the web task successfully.\",\n help=\"Prompt text used to seed the training dataset.\",\n )\n parser.add_argument(\n \"--dataset-size\",\n type=int,\n default=1000,\n help=\"Number of entries to include in the synthetic training dataset.\",\n )\n parser.add_argument(\n \"--max-steps\",\n type=int,\n default=10,\n help=\"Maximum number of steps per episode.\",\n )\n parser.add_argument(\n \"--max-new-tokens\",\n type=int,\n default=32,\n help=\"Maximum number of new tokens to request from vLLM for each action.\",\n )\n parser.add_argument(\n \"--temperature\",\n type=float,\n default=0.7,\n help=\"Sampling temperature used during rollout generation.\",\n )\n parser.add_argument(\n \"--top-k\",\n type=int,\n default=50,\n help=\"Top-k sampling parameter forwarded to vLLM.\",\n )\n parser.add_argument(\n \"--top-p\",\n type=float,\n default=None,\n help=\"Optional top-p sampling parameter forwarded to vLLM.\",\n )\n parser.add_argument(\n \"--image-size\",\n type=int,\n default=512,\n help=\"Resize screenshots to this size (preserving aspect ratio) to reduce memory usage. Set to 0 to disable resizing.\",\n )\n parser.add_argument(\n \"--learning-rate\",\n type=float,\n default=5e-6,\n help=\"Learning rate for GRPO training.\",\n )\n parser.add_argument(\n \"--weight-decay\",\n type=float,\n default=0.0,\n help=\"Weight decay applied during optimization.\",\n )\n parser.add_argument(\n \"--gradient-accumulation-steps\",\n type=int,\n default=32,\n help=\"Gradient accumulation steps for GRPO training.\",\n )\n parser.add_argument(\n \"--warmup-steps\",\n type=int,\n default=10,\n help=\"Warmup steps for the scheduler.\",\n )\n parser.add_argument(\n \"--per-device-batch-size\",\n type=int,\n default=1,\n help=\"Per-device train batch size.\",\n )\n parser.add_argument(\n \"--num-generations\",\n type=int,\n default=4,\n help=\"Number of rollout generations per dataset prompt.\",\n )\n parser.add_argument(\n \"--num-epochs\",\n type=int,\n default=1,\n help=\"Number of training epochs.\",\n )\n parser.add_argument(\n \"--save-interval\",\n type=int,\n default=50,\n help=\"Interval (in steps) between checkpoint saves.\",\n )\n parser.add_argument(\n \"--save-total-limit\",\n type=int,\n default=None,\n help=\"Maximum number of checkpoints to keep.\",\n )\n parser.add_argument(\n \"--output-dir\",\n default=None,\n help=\"Directory where training outputs and checkpoints are stored.\",\n )\n parser.add_argument(\n \"--run-name\",\n default=None,\n help=\"Optional run name for logging systems.\",\n )\n parser.add_argument(\n \"--project\",\n default=None,\n help=\"Optional project identifier for logging systems.\",\n )\n parser.add_argument(\n \"--vllm-mode\",\n choices=(\"colocate\", \"server\"),\n default=\"colocate\",\n help=\"vLLM execution mode: 'colocate' or 'server'.\",\n )\n parser.add_argument(\n \"--vllm-server-url\",\n type=str,\n default=\"http://localhost:8001\",\n help=\"URL for the vLLM server (only used when --vllm-mode=server).\",\n )\n parser.add_argument(\n \"--logging-steps\",\n type=int,\n default=1,\n help=\"Frequency of logging steps for GRPO training.\",\n )\n parser.add_argument(\n \"--debug\",\n action=\"store_true\",\n default=False,\n help=\"Enable verbose debugging output during rollouts.\",\n )\n return parser.parse_args()\n\n\ndef sanitize_name(name: str) -> str:\n return name.replace(\"/\", \"-\")\n\n\n# ---------------------------------------------------------------------------\n# System Prompt\n# ---------------------------------------------------------------------------\n\nSYSTEM_PROMPT = \"\"\"You control a web browser through BrowserGym actions.\nYou must complete the given web task by interacting with the page.\n\nAvailable actions:\n- noop() - Do nothing\n- click(bid) - Click element with BrowserGym ID\n- fill(bid, text) - Fill input field\n- send_keys(text) - Send keyboard input\n- scroll(direction) - Scroll up/down\n\nReply with exactly ONE action on a single line, e.g.:\nclick('123')\nfill('456', 'text')\nnoop()\n\nDo not include explanations or multiple actions.\"\"\"\n\n\n# ---------------------------------------------------------------------------\n# Helpers\n# ---------------------------------------------------------------------------\n\n\ndef make_user_prompt(goal: str, step_num: int, axtree: str, error: str = \"\") -> str:\n \"\"\"Create user prompt from observation.\"\"\"\n prompt_parts = [f\"Step {step_num + 1}\"]\n\n if goal:\n prompt_parts.append(f\"Goal: {goal}\")\n\n if error:\n prompt_parts.append(f\"Previous action error: {error}\")\n\n # Include accessibility tree (truncated for context)\n if axtree:\n max_len = 2000\n axtree_truncated = axtree[:max_len] + \"...\" if len(axtree) > max_len else axtree\n prompt_parts.append(f\"Page structure:\\n{axtree_truncated}\")\n\n prompt_parts.append(\"What action do you take?\")\n\n return \"\\n\\n\".join(prompt_parts)\n\n\ndef parse_action(response_text: str) -> str:\n \"\"\"Parse BrowserGym action from model response.\"\"\"\n # Extract first line that looks like an action\n for line in response_text.strip().split(\"\\n\"):\n line = line.strip()\n if \"(\" in line and \")\" in line:\n return line\n\n # Fallback to noop if no valid action found\n return \"noop()\"\n\n\ndef rollout_once(\n trainer: GRPOTrainer,\n env: BrowserGymEnv,\n tokenizer: AutoTokenizer,\n dataset_prompt: str,\n max_steps: int,\n image_size: int = 0,\n debug: bool = False,\n) -> dict[str, list]:\n \"\"\"Run one episode and collect training data.\"\"\"\n result = env.reset()\n observation = result.observation\n\n prompt_ids: list[int] = []\n completion_ids: list[int] = []\n logprobs: list[float] = []\n step_rewards: list[float] = []\n completion_rewards: list[float] = []\n images: list[Image.Image] = [] # Collect screenshots for VLM\n\n for step_num in range(max_steps):\n if result.done:\n break\n\n # Create prompt from observation\n goal = observation.goal or dataset_prompt\n axtree = observation.axtree_txt or \"\"\n error = observation.error if observation.last_action_error else \"\"\n\n # Collect screenshot if available (for VLM support)\n if observation.screenshot is not None:\n screenshot_array = np.array(observation.screenshot, dtype=np.uint8)\n screenshot_image = Image.fromarray(screenshot_array)\n\n # Resize to reduce memory if image_size > 0\n if image_size > 0:\n # Preserve aspect ratio while resizing\n screenshot_image.thumbnail((image_size, image_size), Image.LANCZOS)\n print(\n f\"[DEBUG] Step {step_num + 1}: Collected and resized screenshot from {screenshot_array.shape} to {screenshot_image.size}\"\n )\n else:\n print(f\"[DEBUG] Step {step_num + 1}: Collected screenshot, shape={screenshot_array.shape}\")\n\n images.append(screenshot_image)\n else:\n print(f\"[DEBUG] Step {step_num + 1}: No screenshot available\")\n\n user_prompt = make_user_prompt(goal, step_num, axtree, error)\n messages = [\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": user_prompt},\n ]\n prompt_text = tokenizer.apply_chat_template(\n messages,\n add_generation_prompt=True,\n tokenize=False,\n )\n\n # Generate action with vLLM\n rollout_outputs = generate_rollout_completions(trainer, [prompt_text])[0]\n prompt_ids.extend(rollout_outputs[\"prompt_ids\"])\n completion_ids.extend(rollout_outputs[\"completion_ids\"])\n logprobs.extend(rollout_outputs[\"logprobs\"])\n\n completion_text = rollout_outputs.get(\"text\") or tokenizer.decode(\n rollout_outputs[\"completion_ids\"], skip_special_tokens=True\n )\n\n # Parse and execute action\n action_str = parse_action(completion_text)\n\n if debug:\n print(f\"Step {step_num + 1}: {action_str}\")\n\n # Take action in environment\n result = env.step(BrowserGymAction(action_str=action_str))\n observation = result.observation\n\n # Track rewards\n step_reward = float(result.reward or 0.0)\n step_rewards.append(step_reward)\n\n # Reward shaping: success is most important\n if result.done and step_reward > 0:\n completion_rewards.append(1.0) # Task completed successfully\n elif result.done and step_reward == 0:\n completion_rewards.append(0.0) # Task failed\n else:\n completion_rewards.append(step_reward) # Intermediate reward\n\n # Final reward is based on task completion\n final_reward = completion_rewards[-1] if completion_rewards else 0.0\n\n result_dict = {\n \"prompt_ids\": prompt_ids,\n \"completion_ids\": completion_ids,\n \"logprobs\": logprobs,\n \"step_rewards\": step_rewards,\n \"completion_reward\": final_reward,\n }\n\n # Include images if available (GRPO trainer will auto-detect VLM support)\n if images:\n result_dict[\"images\"] = images\n\n return result_dict\n\n\n# ---------------------------------------------------------------------------\n# Rewards\n# ---------------------------------------------------------------------------\n\n\ndef reward_completion(completions: list[str], **kwargs) -> list[float]:\n \"\"\"Reward for task completion.\"\"\"\n rewards = kwargs.get(\"completion_reward\") if kwargs else None\n if rewards is None:\n return [0.0 for _ in completions]\n return [float(r) for r in rewards]\n\n\n# ---------------------------------------------------------------------------\n# Main entrypoint\n# ---------------------------------------------------------------------------\n\n\ndef main() -> None:\n args = parse_args()\n\n tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_id)\n tokenizer.pad_token = tokenizer.eos_token\n\n # Select environment mode\n if args.env_mode == \"docker-local\":\n env_url = f\"http://{args.env_host}:{args.env_port}\"\n client = BrowserGymEnv(base_url=env_url)\n print(f\"๐ŸŒ Using existing BrowserGym Environment (Docker) at: {env_url}\")\n elif args.env_mode == \"docker-image\":\n client = BrowserGymEnv.from_docker_image(args.env_image)\n print(\"๐ŸŒ Using BrowserGym Environment (Docker) from local Image\")\n elif args.env_mode == \"docker-hub\":\n client = BrowserGymEnv.from_hub(args.env_image)\n print(\"๐ŸŒ Using existing BrowserGym Environment (Docker) from Hub Image\")\n elif args.env_mode == \"space\":\n env_url = args.env_host\n client = BrowserGymEnv(base_url=env_url)\n print(f\"๐ŸŒ Using Hugging Face Space environment at: {env_url}\")\n else:\n raise ValueError(f\"Unknown environment mode: {args.env_mode}\")\n\n dataset = Dataset.from_dict({\"prompt\": [args.dataset_prompt] * args.dataset_size})\n\n timestamp = datetime.now().strftime(\"%Y-%m-%d_%H-%M-%S\")\n default_output_dir = Path(\"outputs\") / f\"browsergym-grpo-{sanitize_name(args.model_id)}-{timestamp}\"\n output_dir = Path(args.output_dir or default_output_dir)\n\n grpo_config = GRPOConfig(\n use_vllm=True,\n vllm_mode=args.vllm_mode,\n vllm_server_base_url=args.vllm_server_url if args.vllm_mode == \"server\" else None,\n vllm_gpu_memory_utilization=0.4,\n output_dir=str(output_dir),\n num_train_epochs=args.num_epochs,\n learning_rate=args.learning_rate,\n weight_decay=args.weight_decay,\n gradient_accumulation_steps=args.gradient_accumulation_steps,\n per_device_train_batch_size=args.per_device_batch_size,\n warmup_steps=args.warmup_steps,\n num_generations=args.num_generations,\n generation_batch_size=args.num_generations, # Must be divisible by num_generations\n max_completion_length=args.max_new_tokens,\n logging_steps=args.logging_steps,\n report_to=\"trackio\",\n trackio_space_id=f\"browsergym-grpo-{sanitize_name(args.model_id)}-{timestamp}\",\n save_strategy=\"steps\",\n save_steps=args.save_interval,\n save_total_limit=args.save_total_limit,\n temperature=args.temperature,\n top_k=args.top_k,\n top_p=args.top_p,\n )\n\n grpo_config.run_name = args.run_name or f\"run-{timestamp}\"\n grpo_config.project = args.project or f\"group-{sanitize_name(args.model_id)}\"\n\n def rollout_func(prompts: list[str], trainer: GRPOTrainer) -> dict[str, list]:\n episode_prompt_ids: list[list[int]] = []\n episode_completion_ids: list[list[int]] = []\n episode_logprobs: list[list[float]] = []\n completion_rewards: list[float] = []\n episode_images: list[list[Image.Image]] = []\n\n print(f\"\\n[DEBUG] rollout_func called with {len(prompts)} prompts\")\n\n for i, prompt_text in enumerate(prompts):\n print(f\"[DEBUG] Processing prompt {i + 1}/{len(prompts)}\")\n episode = rollout_once(\n trainer=trainer,\n env=client,\n tokenizer=tokenizer,\n dataset_prompt=prompt_text,\n max_steps=args.max_steps,\n image_size=args.image_size,\n debug=args.debug,\n )\n episode_prompt_ids.append(episode[\"prompt_ids\"])\n episode_completion_ids.append(episode[\"completion_ids\"])\n episode_logprobs.append(episode[\"logprobs\"])\n completion_rewards.append(episode[\"completion_reward\"])\n\n # Collect images if available (for VLM support)\n if \"images\" in episode:\n print(f\"[DEBUG] Episode {i + 1} has {len(episode['images'])} images\")\n episode_images.append(episode[\"images\"])\n else:\n print(f\"[DEBUG] Episode {i + 1} has NO images\")\n\n result = {\n \"prompt_ids\": episode_prompt_ids,\n \"completion_ids\": episode_completion_ids,\n \"logprobs\": episode_logprobs,\n \"completion_reward\": completion_rewards,\n }\n\n # Include images if any episode had screenshots (GRPO trainer auto-detects VLM)\n if episode_images:\n result[\"images\"] = episode_images\n print(f\"[DEBUG] rollout_func returning with images: {len(episode_images)} episodes\")\n else:\n print(\"[DEBUG] rollout_func returning WITHOUT images\")\n\n return result\n\n trainer = GRPOTrainer(\n model=args.model_id,\n processing_class=tokenizer,\n reward_funcs=[reward_completion],\n train_dataset=dataset,\n args=grpo_config,\n rollout_func=rollout_func,\n )\n\n print(\"=\" * 80)\n print(\"Starting GRPO training with BrowserGym environment\")\n print(f\"Benchmark: {args.benchmark}\")\n print(f\"Task: {args.task_name}\")\n print(f\"Model: {args.model_id}\")\n print(f\"Using {args.num_generations} rollouts per dataset prompt\")\n print(f\"Output directory: {output_dir}\")\n print(\"=\" * 80)\n\n try:\n trainer.train()\n print(\"\\nTraining completed successfully!\")\n finally:\n client.close()\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/openenv/browsergym_llm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[vllm]\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# \"openenv-browsergym @ git+https://huggingface.co/spaces/openenv/browsergym_env\",\n# ]\n# ///\n\n\"\"\"\nSimple script to run GRPO training with OpenEnv's BrowserGym environment and vLLM for LLMs.\n\nThis script is optimized for text-only Language Models (LLMs). It uses the accessibility\ntree text from BrowserGym, making it memory-efficient.\n\nThe environment runs on a Hugging Face Space by default.\n\nSetup (Option A - Install from HF Space, recommended):\n\n```sh\nuv pip install git+https://huggingface.co/spaces/openenv/browsergym_env\n```\n\nSetup (Option B - Clone OpenEnv repo, for development):\n\n```sh\ngit clone https://github.com/meta-pytorch/OpenEnv.git\ncd OpenEnv/envs/browsergym_env\nuv pip install -e .\n```\n\n# Option 1: HF Spaces + Colocated vLLM (1 GPU required)\n```sh\npython examples/scripts/openenv/browsergym_llm.py --vllm-mode colocate\n```\n\n# Option 2: HF Spaces + Separate vLLM server (2 GPUs required)\n\n# Spin up vLLM server (Terminal 1)\n```sh\nCUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen3-0.6B --host 0.0.0.0 --port 8001\n```\n\n# Run training (Terminal 2)\n```sh\nCUDA_VISIBLE_DEVICES=1 python examples/scripts/openenv/browsergym_llm.py --vllm-mode server --vllm-server-url http://localhost:8001\n```\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nfrom datetime import datetime\nfrom pathlib import Path\n\nfrom browsergym_env import BrowserGymAction, BrowserGymEnv\nfrom datasets import Dataset\nfrom transformers import AutoTokenizer\n\nfrom trl import GRPOConfig, GRPOTrainer\nfrom trl.experimental.openenv import generate_rollout_completions\n\n\ndef parse_args() -> argparse.Namespace:\n parser = argparse.ArgumentParser(description=\"Run GRPO training for BrowserGym MiniWoB using OpenEnv environment.\")\n parser.add_argument(\n \"--model-id\",\n default=\"Qwen/Qwen3-0.6B\",\n help=\"Model identifier passed to GRPOTrainer for fine-tuning.\",\n )\n parser.add_argument(\n \"--space-url\",\n type=str,\n default=\"https://openenv-browsergym-env.hf.space\",\n help=\"URL for the Hugging Face Space running the BrowserGym environment.\",\n )\n parser.add_argument(\n \"--benchmark\",\n default=\"miniwob\",\n help=\"BrowserGym benchmark to use (miniwob, webarena, etc.).\",\n )\n parser.add_argument(\n \"--task-name\",\n default=\"click-test\",\n help=\"Specific task within the benchmark (e.g., click-test, click-button).\",\n )\n parser.add_argument(\n \"--dataset-prompt\",\n default=\"Complete the web task successfully.\",\n help=\"Prompt text used to seed the training dataset.\",\n )\n parser.add_argument(\n \"--dataset-size\",\n type=int,\n default=1000,\n help=\"Number of entries to include in the synthetic training dataset.\",\n )\n parser.add_argument(\n \"--max-steps\",\n type=int,\n default=10,\n help=\"Maximum number of steps per episode.\",\n )\n parser.add_argument(\n \"--max-new-tokens\",\n type=int,\n default=32,\n help=\"Maximum number of new tokens to request from vLLM for each action.\",\n )\n parser.add_argument(\n \"--temperature\",\n type=float,\n default=0.7,\n help=\"Sampling temperature used during rollout generation.\",\n )\n parser.add_argument(\n \"--top-k\",\n type=int,\n default=50,\n help=\"Top-k sampling parameter forwarded to vLLM.\",\n )\n parser.add_argument(\n \"--top-p\",\n type=float,\n default=None,\n help=\"Optional top-p sampling parameter forwarded to vLLM.\",\n )\n parser.add_argument(\n \"--learning-rate\",\n type=float,\n default=5e-6,\n help=\"Learning rate for GRPO training.\",\n )\n parser.add_argument(\n \"--weight-decay\",\n type=float,\n default=0.0,\n help=\"Weight decay applied during optimization.\",\n )\n parser.add_argument(\n \"--gradient-accumulation-steps\",\n type=int,\n default=32,\n help=\"Gradient accumulation steps for GRPO training.\",\n )\n parser.add_argument(\n \"--warmup-steps\",\n type=int,\n default=10,\n help=\"Warmup steps for the scheduler.\",\n )\n parser.add_argument(\n \"--per-device-batch-size\",\n type=int,\n default=1,\n help=\"Per-device train batch size.\",\n )\n parser.add_argument(\n \"--num-generations\",\n type=int,\n default=4,\n help=\"Number of rollout generations per dataset prompt.\",\n )\n parser.add_argument(\n \"--num-epochs\",\n type=int,\n default=1,\n help=\"Number of training epochs.\",\n )\n parser.add_argument(\n \"--save-interval\",\n type=int,\n default=50,\n help=\"Interval (in steps) between checkpoint saves.\",\n )\n parser.add_argument(\n \"--save-total-limit\",\n type=int,\n default=None,\n help=\"Maximum number of checkpoints to keep.\",\n )\n parser.add_argument(\n \"--output-dir\",\n default=None,\n help=\"Directory where training outputs and checkpoints are stored.\",\n )\n parser.add_argument(\n \"--run-name\",\n default=None,\n help=\"Optional run name for logging systems.\",\n )\n parser.add_argument(\n \"--project\",\n default=None,\n help=\"Optional project identifier for logging systems.\",\n )\n parser.add_argument(\n \"--vllm-mode\",\n choices=(\"colocate\", \"server\"),\n default=\"colocate\",\n help=\"vLLM execution mode: 'colocate' or 'server'.\",\n )\n parser.add_argument(\n \"--vllm-server-url\",\n type=str,\n default=\"http://localhost:8001\",\n help=\"URL for the vLLM server (only used when --vllm-mode=server).\",\n )\n parser.add_argument(\n \"--logging-steps\",\n type=int,\n default=1,\n help=\"Frequency of logging steps for GRPO training.\",\n )\n parser.add_argument(\n \"--debug\",\n action=\"store_true\",\n default=False,\n help=\"Enable verbose debugging output during rollouts.\",\n )\n return parser.parse_args()\n\n\ndef sanitize_name(name: str) -> str:\n return name.replace(\"/\", \"-\")\n\n\n# ---------------------------------------------------------------------------\n# System Prompt\n# ---------------------------------------------------------------------------\n\nSYSTEM_PROMPT = \"\"\"You control a web browser through BrowserGym actions.\nYou must complete the given web task by interacting with the page.\n\nAvailable actions:\n- noop() - Do nothing\n- click(bid) - Click element with BrowserGym ID (the number in brackets)\n- fill(bid, text) - Fill input field with text\n- send_keys(text) - Send keyboard input\n- scroll(direction) - Scroll up/down\n\nThe page structure shows elements as: [bid] element_type 'element_text'\nFor example: [13] button 'Click Me!' means bid='13'\n\nReply with exactly ONE action on a single line, e.g.:\nclick('13')\nfill('42', 'hello world')\nnoop()\n\nDo not include explanations or multiple actions.\"\"\"\n\n\n# ---------------------------------------------------------------------------\n# Helpers\n# ---------------------------------------------------------------------------\n\n\ndef make_user_prompt(goal: str, step_num: int, axtree: str, error: str = \"\") -> str:\n \"\"\"Create user prompt from observation.\"\"\"\n prompt_parts = [f\"Step {step_num + 1}\"]\n\n if goal:\n prompt_parts.append(f\"Goal: {goal}\")\n\n if error:\n prompt_parts.append(f\"Previous action error: {error}\")\n\n # Include accessibility tree (truncated for context)\n if axtree:\n max_len = 2000\n axtree_truncated = axtree[:max_len] + \"...\" if len(axtree) > max_len else axtree\n prompt_parts.append(f\"Page structure:\\n{axtree_truncated}\")\n\n prompt_parts.append(\"What action do you take?\")\n\n return \"\\n\\n\".join(prompt_parts)\n\n\ndef parse_action(response_text: str) -> str:\n \"\"\"Parse BrowserGym action from model response.\"\"\"\n # Extract first line that looks like an action\n for line in response_text.strip().split(\"\\n\"):\n line = line.strip()\n if \"(\" in line and \")\" in line:\n return line\n\n # Fallback to noop if no valid action found\n return \"noop()\"\n\n\ndef rollout_once(\n trainer: GRPOTrainer,\n env: BrowserGymEnv,\n tokenizer: AutoTokenizer,\n dataset_prompt: str,\n max_steps: int,\n debug: bool = False,\n) -> dict[str, list]:\n \"\"\"Run one episode and collect training data (text-only, no screenshots).\"\"\"\n result = env.reset()\n observation = result.observation\n\n prompt_ids: list[int] = []\n completion_ids: list[int] = []\n logprobs: list[float] = []\n step_rewards: list[float] = []\n completion_rewards: list[float] = []\n\n for step_num in range(max_steps):\n if result.done:\n break\n\n # Create prompt from observation (text-only using accessibility tree)\n goal = observation.goal or dataset_prompt\n axtree = observation.axtree_txt or \"\"\n error = observation.error if observation.last_action_error else \"\"\n\n user_prompt = make_user_prompt(goal, step_num, axtree, error)\n messages = [\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": user_prompt},\n ]\n prompt_text = tokenizer.apply_chat_template(\n messages,\n add_generation_prompt=True,\n tokenize=False,\n )\n\n # Generate action with vLLM\n rollout_outputs = generate_rollout_completions(trainer, [prompt_text])[0]\n prompt_ids.extend(rollout_outputs[\"prompt_ids\"])\n completion_ids.extend(rollout_outputs[\"completion_ids\"])\n logprobs.extend(rollout_outputs[\"logprobs\"])\n\n completion_text = rollout_outputs.get(\"text\") or tokenizer.decode(\n rollout_outputs[\"completion_ids\"], skip_special_tokens=True\n )\n\n # Parse and execute action\n action_str = parse_action(completion_text)\n\n if debug:\n print(f\"Step {step_num + 1}: {action_str}\")\n\n # Take action in environment\n result = env.step(BrowserGymAction(action_str=action_str))\n observation = result.observation\n\n # Track rewards\n step_reward = float(result.reward or 0.0)\n step_rewards.append(step_reward)\n\n # Reward shaping: success is most important\n if result.done and step_reward > 0:\n completion_rewards.append(1.0) # Task completed successfully\n elif result.done and step_reward == 0:\n completion_rewards.append(0.0) # Task failed\n else:\n completion_rewards.append(step_reward) # Intermediate reward\n\n # Final reward is based on task completion\n final_reward = completion_rewards[-1] if completion_rewards else 0.0\n\n return {\n \"prompt_ids\": prompt_ids,\n \"completion_ids\": completion_ids,\n \"logprobs\": logprobs,\n \"step_rewards\": step_rewards,\n \"completion_reward\": final_reward,\n }\n\n\n# ---------------------------------------------------------------------------\n# Rewards\n# ---------------------------------------------------------------------------\n\n\ndef reward_completion(completions: list[str], **kwargs) -> list[float]:\n \"\"\"Reward for task completion.\"\"\"\n rewards = kwargs.get(\"completion_reward\") if kwargs else None\n if rewards is None:\n return [0.0 for _ in completions]\n return [float(r) for r in rewards]\n\n\n# ---------------------------------------------------------------------------\n# Main entrypoint\n# ---------------------------------------------------------------------------\n\n\ndef main() -> None:\n args = parse_args()\n\n # Connect to BrowserGym environment via Hugging Face Space\n client = BrowserGymEnv(base_url=args.space_url)\n print(f\"๐ŸŒ Using Hugging Face Space environment at: {args.space_url}\")\n\n dataset = Dataset.from_dict({\"prompt\": [args.dataset_prompt] * args.dataset_size})\n\n timestamp = datetime.now().strftime(\"%Y-%m-%d_%H-%M-%S\")\n default_output_dir = Path(\"outputs\") / f\"browsergym-grpo-{sanitize_name(args.model_id)}-{timestamp}\"\n output_dir = Path(args.output_dir or default_output_dir)\n\n grpo_config = GRPOConfig(\n use_vllm=True,\n vllm_mode=args.vllm_mode,\n vllm_server_base_url=args.vllm_server_url if args.vllm_mode == \"server\" else None,\n vllm_gpu_memory_utilization=0.4,\n output_dir=str(output_dir),\n num_train_epochs=args.num_epochs,\n learning_rate=args.learning_rate,\n weight_decay=args.weight_decay,\n gradient_accumulation_steps=args.gradient_accumulation_steps,\n per_device_train_batch_size=args.per_device_batch_size,\n warmup_steps=args.warmup_steps,\n num_generations=args.num_generations,\n generation_batch_size=args.num_generations, # Must be divisible by num_generations\n max_completion_length=args.max_new_tokens,\n logging_steps=args.logging_steps,\n report_to=\"trackio\",\n trackio_space_id=f\"browsergym-grpo-{sanitize_name(args.model_id)}-{timestamp}\",\n save_strategy=\"steps\",\n save_steps=args.save_interval,\n save_total_limit=args.save_total_limit,\n temperature=args.temperature,\n top_k=args.top_k,\n top_p=args.top_p,\n )\n\n grpo_config.run_name = args.run_name or f\"run-{timestamp}\"\n grpo_config.project = args.project or f\"group-{sanitize_name(args.model_id)}\"\n\n def rollout_func(prompts: list[str], trainer: GRPOTrainer) -> dict[str, list]:\n episode_prompt_ids: list[list[int]] = []\n episode_completion_ids: list[list[int]] = []\n episode_logprobs: list[list[float]] = []\n completion_rewards: list[float] = []\n\n if args.debug:\n print(f\"\\n[DEBUG] rollout_func called with {len(prompts)} prompts (LLM mode, text-only)\")\n\n for i, prompt_text in enumerate(prompts):\n if args.debug:\n print(f\"[DEBUG] Processing prompt {i + 1}/{len(prompts)}\")\n episode = rollout_once(\n trainer=trainer,\n env=client,\n tokenizer=trainer.processing_class,\n dataset_prompt=prompt_text,\n max_steps=args.max_steps,\n debug=args.debug,\n )\n episode_prompt_ids.append(episode[\"prompt_ids\"])\n episode_completion_ids.append(episode[\"completion_ids\"])\n episode_logprobs.append(episode[\"logprobs\"])\n completion_rewards.append(episode[\"completion_reward\"])\n\n return {\n \"prompt_ids\": episode_prompt_ids,\n \"completion_ids\": episode_completion_ids,\n \"logprobs\": episode_logprobs,\n \"completion_reward\": completion_rewards,\n }\n\n trainer = GRPOTrainer(\n model=args.model_id,\n reward_funcs=[reward_completion],\n train_dataset=dataset,\n args=grpo_config,\n rollout_func=rollout_func,\n )\n\n print(\"=\" * 80)\n print(\"Starting GRPO training with BrowserGym environment (LLM mode)\")\n print(f\"Benchmark: {args.benchmark}\")\n print(f\"Task: {args.task_name}\")\n print(f\"Model: {args.model_id}\")\n print(\"Mode: LLM (text-only, using accessibility tree)\")\n print(f\"Using {args.num_generations} rollouts per dataset prompt\")\n print(f\"Output directory: {output_dir}\")\n print(\"=\" * 80)\n\n try:\n trainer.train()\n print(\"\\nTraining completed successfully!\")\n finally:\n client.close()\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/openenv/carla.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"openenv-carla-env @ git+https://huggingface.co/spaces/sergiopaniego/carla_env\",\n# ]\n# ///\n\n\n\"\"\"\nSimple script to run GRPO training with OpenEnv's CARLA environment. The environment simulates an emergency\ndriving scenario where pedestrians are ahead and the model must learn to observe the scene and take the\ncorrect action (e.g., swerve to an empty lane) to minimize casualties.\n\nSetup (Option A - Install from HF Space, recommended):\n\n```sh\nuv pip install git+https://huggingface.co/spaces/sergiopaniego/carla_env\n```\n\nSetup (Option B - Clone OpenEnv repo, for development):\n\n```sh\ngit clone https://github.com/meta-pytorch/OpenEnv.git\ncd OpenEnv/envs/carla_env\nuv pip install -e .\n```\n\nUsage:\n\n```sh\npython examples/scripts/openenv/carla.py\npython examples/scripts/openenv/carla.py --model Qwen/Qwen3-1.7B --env-urls https://server1.hf.space https://server2.hf.space\n```\n\"\"\"\n\nimport argparse\n\nfrom carla_env import CarlaAction, CarlaEnv\nfrom datasets import Dataset\n\nfrom trl import GRPOConfig, GRPOTrainer\n\n\ndef parse_args():\n parser = argparse.ArgumentParser(description=\"Run GRPO training with CARLA environment.\")\n parser.add_argument(\n \"--model\",\n type=str,\n default=\"Qwen/Qwen3-0.6B\",\n help=\"Model to use for training.\",\n )\n parser.add_argument(\n \"--env-urls\",\n type=str,\n nargs=\"+\",\n default=[\"https://sergiopaniego-carla-env.hf.space\"],\n help=\"URLs for the CARLA environment servers (one per environment instance).\",\n )\n parser.add_argument(\n \"--trackio-space-id\",\n type=str,\n default=\"carla-grpo-trolley\",\n help=\"Trackio space identifier.\",\n )\n parser.add_argument(\n \"--hub-model-id\",\n type=str,\n default=None,\n help=\"Hub model ID to push the trained model to (e.g., sergiopaniego/Qwen3-0.6B-carla-trolley-escape).\",\n )\n parser.add_argument(\n \"--run-name\",\n type=str,\n default=None,\n help=\"Run name for tracking.\",\n )\n return parser.parse_args()\n\n\nargs = parse_args()\n_env_url_iter = iter(args.env_urls) # Each instance takes the next URL\n\nprompt = \"\"\"You control an autonomous vehicle in an emergency. There are pedestrians ahead and you must \\\ndecide what to do immediately.\n\nYou have the following tools available:\n- `observe`: Advance time and get a new observation of the scene.\n- `emergency_stop`: Apply maximum braking to stop the vehicle.\n- `lane_change(direction)`: Change lane to the left or right. Direction must be \"left\" or \"right\".\n\nObserve the scene first, then decide the best course of action to minimize harm.\"\"\"\n\ndataset = Dataset.from_dict({\"prompt\": [[{\"role\": \"user\", \"content\": prompt}] for _ in range(1000)]})\n\n\nSIM_TICKS = 10 # Number of simulation steps to advance after each action\n\n\nclass CarlaGRPOEnv:\n def __init__(self):\n url = next(_env_url_iter)\n self.client = CarlaEnv(base_url=url, connect_timeout_s=30, message_timeout_s=120)\n\n @staticmethod\n def _describe(obs) -> str:\n \"\"\"Build a text description from the observation fields.\"\"\"\n parts = []\n parts.append(f\"Speed: {obs.speed_kmh:.1f} km/h.\")\n if obs.nearby_actors:\n for actor in obs.nearby_actors:\n parts.append(f\"- {actor.get('type', 'actor')} at {actor.get('distance', '?')}m\")\n else:\n parts.append(\"No nearby actors detected.\")\n if obs.collision_detected:\n parts.append(f\"COLLISION detected with {obs.collided_with or 'unknown'}!\")\n return \"\\n\".join(parts)\n\n def _advance(self, ticks: int = SIM_TICKS):\n \"\"\"Advance the simulation by calling observe repeatedly, return the last result.\"\"\"\n result = None\n for _ in range(ticks):\n result = self.client.step(CarlaAction(action_type=\"observe\"))\n if result.done:\n break\n return result\n\n def reset(self, **kwargs) -> str | None:\n result = self.client.reset(scenario_name=\"trolley_micro_escape_exists\")\n self.reward = 0.0\n return self._describe(result.observation)\n\n def observe(self) -> str:\n \"\"\"\n Get the current scene description without taking any action.\n\n Returns:\n The scene description with vehicle state and nearby actors.\n \"\"\"\n result = self._advance()\n self.reward = result.observation.rubric_reward or 0.0\n return self._describe(result.observation)\n\n def emergency_stop(self) -> str:\n \"\"\"\n Apply maximum braking to stop the vehicle.\n\n Returns:\n The scene description after braking.\n \"\"\"\n self.client.step(CarlaAction(action_type=\"emergency_stop\"))\n result = self._advance()\n self.reward = result.observation.rubric_reward or 0.0\n return self._describe(result.observation)\n\n def lane_change(self, direction: str) -> str:\n \"\"\"\n Change lane to avoid obstacles.\n\n Args:\n direction: Direction to change lane, either \"left\" or \"right\".\n\n Returns:\n The scene description after changing lane.\n \"\"\"\n self.client.step(CarlaAction(action_type=\"lane_change\", lane_direction=direction))\n result = self._advance()\n self.reward = result.observation.rubric_reward or 0.0\n return self._describe(result.observation)\n\n\ndef reward_func(completions, environments, **kwargs):\n return [environment.reward for environment in environments]\n\n\ntrainer = GRPOTrainer(\n model=args.model,\n train_dataset=dataset,\n reward_funcs=reward_func,\n args=GRPOConfig(\n chat_template_kwargs={\"enable_thinking\": False},\n log_completions=True,\n logging_steps=2,\n num_completions_to_print=1,\n max_completion_length=1024,\n per_device_train_batch_size=len(args.env_urls),\n steps_per_generation=1,\n num_generations=len(args.env_urls),\n gradient_accumulation_steps=16,\n max_steps=50,\n push_to_hub=args.hub_model_id is not None,\n hub_model_id=args.hub_model_id,\n run_name=args.run_name,\n report_to=\"trackio\",\n trackio_space_id=args.trackio_space_id,\n ),\n environment_factory=CarlaGRPOEnv,\n)\ntrainer.train()\n" }, { "path": "examples/scripts/openenv/catch.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[vllm]\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# \"openenv-openspiel-env @ git+https://huggingface.co/spaces/openenv/openspiel_env\",\n# ]\n# ///\n\n\n\"\"\"\nSimple script to run GRPO training with OpenEnv's Catch environment (OpenSpiel) and vLLM. The reward function\nis based on the catch game where the agent tries to catch falling balls.\n\nSetup (Option A - Install from HF Space, recommended):\n\n```sh\nuv pip install git+https://huggingface.co/spaces/openenv/openspiel_env\n```\n\nSetup (Option B - Clone OpenEnv repo, for development):\n\n```sh\ngit clone https://github.com/meta-pytorch/OpenEnv.git\ncd OpenEnv/envs/openspiel_env\nuv pip install -e .\n```\n\n# Option 1: HF Spaces + Colocated vLLM (1 GPU required)\n```sh\npython examples/scripts/openenv/catch.py --env-mode space --env-host https://openenv-openspiel-env.hf.space --vllm-mode colocate\n```\n\n# Option 2: HF Spaces + Separate vLLM server (2 GPUs required)\n\n# Spin up vLLM server (Terminal 1)\n```sh\nCUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen2.5-0.5B-Instruct --host 0.0.0.0 --port 8000\n```\n\n# Run training (Terminal 2)\n```sh\nCUDA_VISIBLE_DEVICES=1 python examples/scripts/openenv/catch.py --env-mode space --env-host https://openenv-openspiel-env.hf.space --vllm-mode server --vllm-server-url http://localhost:8000\n```\n\n# Option 3: Local + Colocated vLLM (1 GPU required)\n\n# Start the environment only if using --env-mode docker-local\n```sh\ndocker run -d -p 8001:8001 registry.hf.space/openenv-openspiel-env:latest\n```\n\n```sh\npython examples/scripts/openenv/catch.py --env-mode docker-local --vllm-mode colocate\n```\n\"\"\"\n\n# ruff: noqa: T201\nimport argparse\nimport os\nimport re\nimport subprocess\nimport sys\nimport time\nfrom pathlib import Path\n\nimport requests\nfrom datasets import Dataset\nfrom openspiel_env import OpenSpielEnv\nfrom openspiel_env.models import OpenSpielAction\n\nfrom trl import GRPOConfig, GRPOTrainer, RichProgressCallback, apply_chat_template\nfrom trl.experimental.openenv import generate_rollout_completions\n\n\ndef parse_args():\n parser = argparse.ArgumentParser(description=\"Run GRPO training with OpenSpiel Catch environment and vLLM.\")\n\n # --- Environment settings ---\n parser.add_argument(\"--env-host\", type=str, default=\"0.0.0.0\", help=\"Host for the environment server.\")\n parser.add_argument(\"--env-port\", type=int, default=8001, help=\"Port for the environment server.\")\n parser.add_argument(\n \"--env-mode\",\n choices=[\"local\", \"docker-local\", \"docker-image\", \"docker-hub\", \"space\"],\n default=\"docker-image\",\n help=\"Where to run the environment: 'local' to launch it, 'docker-local' if already running locally, 'docker-image' to run from a Docker image, 'docker-hub' to run from Docker Hub, or 'space' to use a remote Space URL.\",\n )\n # --- Generation and model config ---\n parser.add_argument(\n \"--model\",\n type=str,\n default=\"Qwen/Qwen2.5-0.5B-Instruct\",\n help=\"Model name or path.\",\n )\n parser.add_argument(\n \"--dataset-size\",\n type=int,\n default=1000,\n help=\"Number of prompts to use for training dataset.\",\n )\n parser.add_argument(\n \"--env-image\", type=str, default=\"openspiel-env:latest\", help=\"Docker image for the OpenSpiel environment.\"\n )\n parser.add_argument(\n \"--vllm-mode\",\n choices=[\"colocate\", \"server\"],\n default=\"colocate\",\n help=\"vLLM execution mode: 'colocate' or 'server'.\",\n )\n parser.add_argument(\n \"--vllm-server-url\",\n type=str,\n default=\"http://localhost:8000\",\n help=\"URL for the vLLM server (only used when --vllm-mode=server).\",\n )\n\n return parser.parse_args()\n\n\ndef start_env_server(env_host: str, env_port: int):\n \"\"\"Launch the OpenSpiel Catch environment locally via uvicorn.\"\"\"\n env_url = f\"http://{env_host}:{env_port}\"\n print(f\"โšก Starting FastAPI server for OpenSpiel Catch Environment on {env_url}...\")\n\n work_dir = str(Path.cwd().parent.absolute())\n process = subprocess.Popen(\n [\n sys.executable,\n \"-m\",\n \"uvicorn\",\n \"envs.openspiel_env.server.app:app\",\n \"--host\",\n env_host,\n \"--port\",\n str(env_port),\n ],\n env={**os.environ, \"PYTHONPATH\": f\"{work_dir}/src\"},\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n text=True,\n cwd=work_dir,\n )\n\n print(\"โณ Waiting for server to start...\")\n time.sleep(5)\n\n try:\n requests.get(f\"{env_url}/health\", timeout=2)\n print(\"\\nโœ… OpenSpiel Catch Environment server is running!\")\n except Exception as e:\n print(f\"\\nโŒ Server failed to start: {e}\")\n if process.stderr:\n print(process.stderr.read())\n raise\n\n return process\n\n\nBASE_PROMPT = \"\"\"You are an AI agent playing the game **Catch**.\n\n### Game Description\n- The game is played on a **10ร—5 grid**.\n- There is one **falling ball** and one **paddle** that you control at the bottom.\n- The objective is to **move the paddle left or right to catch the ball** as it falls.\n- The episode ends when the ball reaches the bottom row:\n - You get **+1 reward** if you catch it.\n - You get **โ€“1 reward** if you miss it.\n\n### Observation Format\nEach observation is a flattened 10x5 grid (list of 50 floats).\n- 1.0 โ†’ occupied (ball or paddle)\n- 0.0 โ†’ empty cell\n\n### Actions:\n- `0` โ†’ Move left\n- `1` โ†’ Stay\n- `2` โ†’ Move right\n\nRespond **only** with one integer: `0`, `1`, or `2`.\n\n### Current Observation\n\"\"\"\n\n\ndef reward_from_env(completions, **kwargs):\n rewards = kwargs.get(\"env_reward\", [])\n return [float(r) for r in rewards] if rewards else [0.0] * len(completions)\n\n\ndef main():\n args = parse_args()\n\n # Select environment mode\n if args.env_mode == \"local\":\n env_url = f\"http://{args.env_host}:{args.env_port}\"\n server_process = start_env_server(args.env_host, args.env_port)\n elif args.env_mode == \"docker-local\":\n env_url = f\"http://{args.env_host}:{args.env_port}\"\n server_process = None\n print(f\"๐ŸŒ Using existing OpenSpiel Environment (Docker) at: {env_url}\")\n elif args.env_mode == \"docker-image\":\n client = OpenSpielEnv.from_docker_image(args.env_image)\n server_process = None\n print(\"๐ŸŒ Using OpenSpiel Environment (Docker) from local Image\")\n elif args.env_mode == \"docker-hub\":\n client = OpenSpielEnv.from_hub(args.env_image)\n server_process = None\n print(\"๐ŸŒ Using existing OpenSpiel Environment (Docker) from Hub Image\")\n elif args.env_mode == \"space\":\n env_url = args.env_host\n server_process = None\n print(f\"๐ŸŒ Using Hugging Face Space environment at: {env_url}\")\n else:\n raise ValueError(f\"Unknown environment mode: {args.env_mode}\")\n\n if args.env_mode != \"docker-hub\" and args.env_mode != \"docker-image\":\n client = OpenSpielEnv(base_url=env_url)\n dataset = Dataset.from_dict({\"prompt\": [BASE_PROMPT] * args.dataset_size})\n\n training_args = GRPOConfig(\n output_dir=f\"{args.model.split('/')[-1]}-GRPO-Catch\",\n use_vllm=True,\n vllm_mode=args.vllm_mode,\n vllm_server_base_url=args.vllm_server_url if args.vllm_mode == \"server\" else None,\n logging_steps=1,\n report_to=\"trackio\",\n trackio_space_id=f\"{args.model.split('/')[-1]}-GRPO-Catch\",\n num_train_epochs=1,\n max_completion_length=4,\n gradient_accumulation_steps=4,\n )\n\n def rollout_func(prompts: list[str], trainer: GRPOTrainer) -> dict[str, list]:\n \"\"\"Generate completions via vLLM (colocated or server) and compute environment rewards.\"\"\"\n env_rewards: list[float] = []\n all_prompt_ids: list[list[int]] = []\n all_completion_ids: list[list[int]] = []\n all_logprobs: list[list[float]] = []\n tokenizer = trainer.processing_class\n\n for base_prompt in prompts:\n env_result = client.reset()\n obs = env_result.observation\n total_reward = 0.0\n\n episode_prompt_ids: list[int] = []\n episode_completion_ids: list[int] = []\n episode_logprobs: list[float] = []\n\n while not obs.done:\n episode_msg = {\"prompt\": [{\"role\": \"user\", \"content\": f\"{base_prompt}\\n\\n{obs.info_state}\\n\"}]}\n episode_prompt = apply_chat_template(episode_msg, tokenizer)\n rollout_output = generate_rollout_completions(trainer, [episode_prompt[\"prompt\"]])[0]\n\n episode_prompt_ids.extend(rollout_output[\"prompt_ids\"])\n episode_completion_ids.extend(rollout_output[\"completion_ids\"])\n episode_logprobs.extend(rollout_output[\"logprobs\"])\n\n completion_text = tokenizer.batch_decode([rollout_output[\"completion_ids\"]], skip_special_tokens=True)[\n 0\n ]\n numbers = re.findall(r\"\\b([0-2])\\b\", completion_text)\n action_id = int(numbers[0]) if numbers else obs.legal_actions[0]\n\n env_result = client.step(OpenSpielAction(action_id=action_id, game_name=\"catch\"))\n total_reward += env_result.reward or 0.0\n obs = env_result.observation\n\n env_rewards.append(total_reward)\n all_prompt_ids.append(episode_prompt_ids)\n all_completion_ids.append(episode_completion_ids)\n all_logprobs.append(episode_logprobs)\n\n return {\n \"prompt_ids\": all_prompt_ids,\n \"completion_ids\": all_completion_ids,\n \"logprobs\": all_logprobs,\n \"env_reward\": env_rewards,\n }\n\n trainer = GRPOTrainer(\n model=args.model,\n reward_funcs=reward_from_env,\n args=training_args,\n train_dataset=dataset,\n rollout_func=rollout_func,\n callbacks=[RichProgressCallback()],\n )\n\n trainer.train()\n time.sleep(5)\n\n if server_process:\n print(\"๐Ÿ›‘ Terminating environment server...\")\n server_process.terminate()\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/openenv/echo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"openenv-echo-env @ git+https://huggingface.co/spaces/qgallouedec/echo_env\",\n# ]\n# ///\n\nfrom datasets import Dataset\nfrom echo_env import EchoEnv\nfrom echo_env.models import EchoAction\n\nfrom trl import GRPOConfig, GRPOTrainer\n\n\ndataset = Dataset.from_dict(\n {\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"Try to echo 'Hello World!' in the environment.\"}],\n [{\"role\": \"user\", \"content\": \"Make the environment echo 'Goodbye World!'\"}],\n [{\"role\": \"user\", \"content\": \"Can you ask the environment to echo 'TRL is great!'?\"}],\n [{\"role\": \"user\", \"content\": \"What happens if you ask the environment to echo 'I love RLHF!'?\"}],\n [{\"role\": \"user\", \"content\": \"Try to make the environment echo 'OpenEnv is awesome!'\"}],\n ],\n }\n)\n\n\ndef reward_func(completions, environments, **kwargs):\n return [environment.get_reward() for environment in environments]\n\n\nclass MyEchoEnv:\n def __init__(self):\n self.env = EchoEnv(base_url=\"https://qgallouedec-echo-env.hf.space\")\n\n def reset(self, **kwargs) -> None | str:\n self._reward = None\n return None\n\n def step(self, message: str) -> str:\n \"\"\"\n Echo the message back from the environment.\n\n Args:\n message: The message to echo\n\n Returns:\n The echoed message.\n \"\"\"\n observation = self.env.step(EchoAction(message=message))\n self._reward = observation.observation.reward\n return observation.observation.echoed_message\n\n def get_reward(self) -> float:\n \"\"\"\n Get the reward from the last step.\n\n Returns:\n The reward value.\n \"\"\"\n return self._reward\n\n\ntrainer = GRPOTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n train_dataset=dataset,\n reward_funcs=reward_func,\n args=GRPOConfig(\n chat_template_kwargs={\"enable_thinking\": False},\n log_completions=True,\n logging_steps=2,\n num_completions_to_print=1,\n ),\n environment_factory=MyEchoEnv,\n)\ntrainer.train()\n" }, { "path": "examples/scripts/openenv/sudoku.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[vllm]\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# \"openenv-textarena @ git+https://huggingface.co/spaces/openenv/sudoku\",\n# ]\n# ///\n\n\"\"\"\nGRPO training for Sudoku with TextArena environment.\n\nSetup (Option A - Install from HF Space, recommended):\n\n```sh\nuv pip install git+https://huggingface.co/spaces/openenv/sudoku\n```\n\nSetup (Option B - Clone OpenEnv repo, for development):\n\n```sh\ngit clone https://github.com/meta-pytorch/OpenEnv.git\ncd OpenEnv/envs/textarena_env\nuv pip install -e .\n```\n\n# Option 1: HF Spaces + Colocated vLLM (1 GPU required)\n```sh\npython examples/scripts/openenv/sudoku.py --vllm-mode colocate\n```\n\n# Option 2: HF Spaces + Separate vLLM server (2 GPUs required)\n\n# Spin up vLLM server (Terminal 1)\n```sh\nCUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen3-1.7B --host 0.0.0.0 --port 8000\n```\n\n# Run training (Terminal 2)\n```sh\nCUDA_VISIBLE_DEVICES=1 python examples/scripts/openenv/sudoku.py --vllm-mode server --vllm-server-url http://localhost:8000\n```\n\n# Option 3: Local + Colocated vLLM (1 GPU required)\n\n# Start the environment only if using --env-mode docker-local\n```sh\ndocker run -d -p 8001:8001 registry.hf.space/openenv-sudoku:latest\n```\n\n```sh\npython examples/scripts/openenv/sudoku.py --env-mode docker-local --vllm-mode colocate\n```\n\n# Full example with all flags:\n```sh\npython examples/scripts/openenv/sudoku.py \\\n --vllm-mode colocate \\\n --env-mode space \\\n --env-host https://openenv-sudoku.hf.space \\\n --num-generations 8 \\\n --per-device-batch-size 1 \\\n --max-turns 100 \\\n --gradient-accumulation-steps 8 \\\n --difficulty easy \\\n --dataset-size 100\n```\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nimport re\nimport sys\nimport time\nfrom collections import defaultdict\nfrom datetime import datetime\nfrom pathlib import Path\n\nfrom datasets import Dataset\nfrom transformers import AutoTokenizer\n\nfrom trl import GRPOConfig, GRPOTrainer\nfrom trl.experimental.openenv import generate_rollout_completions\n\n\n# Ensure src/ is on the path\nsys.path.insert(0, str(Path(__file__).parent / \"src\"))\n\nfrom textarena_env import TextArenaAction, TextArenaEnv\n\n\n# ---------------------------------------------------------------------------\n# Argument parsing\n# ---------------------------------------------------------------------------\n\n\ndef parse_args() -> argparse.Namespace:\n parser = argparse.ArgumentParser(description=\"GRPO training for Sudoku\")\n\n # Model\n parser.add_argument(\"--model-id\", default=\"Qwen/Qwen3-1.7B\")\n\n # Environment\n parser.add_argument(\"--env-host\", type=str, default=\"https://openenv-sudoku.hf.space\")\n parser.add_argument(\"--env-port\", type=int, default=8001)\n parser.add_argument(\"--env-mode\", choices=[\"docker-local\", \"docker-image\", \"docker-hub\", \"space\"], default=\"space\")\n parser.add_argument(\"--env-image\", type=str, default=\"textarena-env:latest\")\n\n # Prompts\n parser.add_argument(\"--system-prompt-path\", default=\"sudoku_prompt.txt\")\n parser.add_argument(\"--dataset-prompt\", default=\"Play Sudoku like an expert.\")\n parser.add_argument(\"--dataset-size\", type=int, default=1000)\n\n # Game settings\n parser.add_argument(\"--max-turns\", type=int, default=100)\n parser.add_argument(\"--max-new-tokens\", type=int, default=8)\n parser.add_argument(\n \"--difficulty\",\n type=str,\n choices=[\"easy\", \"medium\", \"hard\"],\n default=\"hard\",\n help=\"Training difficulty: easy=guaranteed+options, medium=only options, hard=no hints\",\n )\n parser.add_argument(\n \"--api-delay\", type=float, default=0.0, help=\"Delay in seconds between API calls to avoid rate limiting\"\n )\n\n # Sampling\n parser.add_argument(\"--temperature\", type=float, default=0.8)\n parser.add_argument(\"--top-k\", type=int, default=10)\n parser.add_argument(\"--top-p\", type=float, default=None)\n\n # Training\n parser.add_argument(\"--learning-rate\", type=float, default=5e-6)\n parser.add_argument(\"--weight-decay\", type=float, default=0.0)\n parser.add_argument(\"--gradient-accumulation-steps\", type=int, default=64)\n parser.add_argument(\"--warmup-steps\", type=int, default=20)\n parser.add_argument(\"--per-device-batch-size\", type=int, default=1)\n parser.add_argument(\"--num-generations\", type=int, default=2)\n parser.add_argument(\"--num-epochs\", type=int, default=1)\n\n # Checkpoints\n parser.add_argument(\"--save-interval\", type=int, default=10)\n parser.add_argument(\"--save-total-limit\", type=int, default=None)\n parser.add_argument(\"--output-dir\", default=None)\n\n # Logging\n parser.add_argument(\"--run-name\", default=None)\n parser.add_argument(\"--project\", default=None)\n parser.add_argument(\"--trackio-space-id\", default=\"Sudoku-GRPO\")\n parser.add_argument(\"--logging-steps\", type=int, default=1)\n parser.add_argument(\"--debug\", action=\"store_true\", default=False)\n parser.add_argument(\n \"--gradient-checkpointing\",\n action=\"store_true\",\n default=True,\n help=\"Enable gradient checkpointing to save memory\",\n )\n\n # vLLM\n parser.add_argument(\"--vllm-mode\", choices=(\"colocate\", \"server\"), default=\"colocate\")\n parser.add_argument(\"--vllm-server-url\", type=str, default=\"http://localhost:8000\")\n parser.add_argument(\"--vllm-gpu-memory-utilization\", type=float, default=0.2)\n\n return parser.parse_args()\n\n\n# ---------------------------------------------------------------------------\n# Helper functions\n# ---------------------------------------------------------------------------\n\n\ndef resolve_system_prompt(path: str) -> str:\n prompt_path = Path(path)\n if not prompt_path.is_file():\n prompt_path = Path(__file__).parent / path\n return prompt_path.read_text()\n\n\ndef sanitize_name(name: str) -> str:\n return name.replace(\"/\", \"-\")\n\n\ndef extract_sudoku_move(text: str) -> str:\n \"\"\"Extract a Sudoku move [row col number] from text.\"\"\"\n # Try with spaces\n match = re.search(r\"\\[(\\d)\\s+(\\d)\\s+(\\d)\\]\", text)\n if match:\n row, col, num = match.groups()\n return f\"[{row} {col} {num}]\"\n\n # Try without spaces\n match = re.search(r\"\\[(\\d)(\\d)(\\d)\\]\", text)\n if match:\n row, col, num = match.groups()\n return f\"[{row} {col} {num}]\"\n\n return \"\"\n\n\ndef is_valid_board_state(board_str: str) -> bool:\n \"\"\"Check if the string contains an actual Sudoku board.\"\"\"\n return \"R1\" in board_str and \"R9\" in board_str and \"|\" in board_str\n\n\ndef parse_board(board_str: str) -> list[list[int]]:\n \"\"\"Parse board string into 9x9 grid (0 = empty).\"\"\"\n grid = [[0] * 9 for _ in range(9)]\n if not is_valid_board_state(board_str):\n return grid\n\n for line in board_str.split(\"\\n\"):\n line_stripped = line.strip()\n if line_stripped and line_stripped[0] == \"R\" and len(line_stripped) > 1 and line_stripped[1].isdigit():\n row = int(line_stripped[1]) - 1 # 0-indexed\n cell_part = line_stripped[2:]\n col = 0\n for char in cell_part:\n if char == \".\":\n grid[row][col] = 0\n col += 1\n elif char.isdigit():\n grid[row][col] = int(char)\n col += 1\n return grid\n\n\ndef count_filled_cells(board_str: str) -> int:\n \"\"\"Count the number of filled cells in the board.\"\"\"\n if not is_valid_board_state(board_str):\n return 0\n grid = parse_board(board_str)\n return sum(1 for row in grid for cell in row if cell != 0)\n\n\ndef get_valid_numbers(grid: list[list[int]], row: int, col: int) -> set[int]:\n \"\"\"Get valid numbers for a cell based on Sudoku rules.\"\"\"\n if grid[row][col] != 0:\n return set()\n\n used = set()\n\n # Check row\n for c in range(9):\n if grid[row][c] != 0:\n used.add(grid[row][c])\n\n # Check column\n for r in range(9):\n if grid[r][col] != 0:\n used.add(grid[r][col])\n\n # Check 3x3 box\n box_row, box_col = 3 * (row // 3), 3 * (col // 3)\n for r in range(box_row, box_row + 3):\n for c in range(box_col, box_col + 3):\n if grid[r][c] != 0:\n used.add(grid[r][c])\n\n return set(range(1, 10)) - used\n\n\ndef extract_empty_cells_with_candidates(\n board_str: str, sort_by_difficulty: bool = True\n) -> list[tuple[int, int, set[int]]]:\n \"\"\"Extract empty cells with their valid candidate numbers.\n\n Args:\n sort_by_difficulty: If True, sort by number of candidates (easiest first).\n If False, keep natural order (top-left to bottom-right).\n \"\"\"\n grid = parse_board(board_str)\n cells_with_candidates = []\n\n for row in range(9):\n for col in range(9):\n if grid[row][col] == 0:\n candidates = get_valid_numbers(grid, row, col)\n cells_with_candidates.append((row + 1, col + 1, candidates)) # 1-indexed\n\n if sort_by_difficulty:\n # Sort by number of candidates (easiest first = naked singles)\n cells_with_candidates.sort(key=lambda x: len(x[2]))\n\n return cells_with_candidates\n\n\ndef extract_empty_cells(board_str: str) -> list[tuple[int, int]]:\n \"\"\"Extract list of empty cells (row, col) from board string.\"\"\"\n empty_cells = []\n if not is_valid_board_state(board_str):\n return empty_cells\n\n for line in board_str.split(\"\\n\"):\n line_stripped = line.strip()\n if line_stripped and line_stripped[0] == \"R\" and len(line_stripped) > 1 and line_stripped[1].isdigit():\n row = int(line_stripped[1])\n cell_part = line_stripped[2:]\n col = 0\n for char in cell_part:\n if char == \".\":\n col += 1\n empty_cells.append((row, col))\n elif char.isdigit():\n col += 1\n return empty_cells\n\n\ndef extract_board_only(text: str) -> str:\n \"\"\"Extract just the Sudoku grid from a message.\"\"\"\n if not text:\n return \"\"\n\n lines = text.split(\"\\n\")\n board_lines = []\n in_board = False\n\n for line in lines:\n stripped = line.strip()\n if stripped.startswith(\"C1\") or (\n stripped and stripped[0] == \"R\" and len(stripped) > 1 and stripped[1].isdigit()\n ):\n in_board = True\n if in_board and (stripped.startswith(\"-\") or stripped.startswith(\"R\") or stripped.startswith(\"C1\")):\n board_lines.append(line)\n elif (\n in_board\n and stripped\n and not stripped.startswith(\"-\")\n and not (stripped[0] == \"R\" and len(stripped) > 1 and stripped[1].isdigit())\n ):\n break\n\n return \"\\n\".join(board_lines) if board_lines else \"\"\n\n\ndef make_compact_prompt(\n board: str,\n step: int,\n successful_moves: list[str],\n failed_moves: list[str],\n difficulty: str = \"hard\",\n) -> str:\n \"\"\"Create a compact prompt with only essential info (saves tokens!).\n\n Args:\n difficulty: Training difficulty level:\n - \"easy\": Show guaranteed moves (naked singles) + other options\n - \"medium\": Only show other options (hints where to look, not exact answers)\n - \"hard\": No hints (model must learn Sudoku rules by itself)\n \"\"\"\n\n # Summary line\n cells_filled = len(successful_moves)\n summary = f\"Step {step}. Progress: {cells_filled} cells filled.\"\n\n # Board (only show the grid, stripped down)\n board_only = extract_board_only(board) if board else \"No board available.\"\n\n # Moves already tried (for learning what NOT to do)\n tried_moves_hint = \"\"\n all_tried = successful_moves + failed_moves\n if all_tried:\n tried_moves_hint = f\"\\n\\nโš ๏ธ MOVES ALREADY TRIED (do not repeat): {', '.join(all_tried)}\"\n\n # Hints based on difficulty\n hints = \"\"\n if difficulty == \"easy\" and board:\n # Easy: sorted by difficulty, show guaranteed moves + other easy options\n cells_with_candidates = extract_empty_cells_with_candidates(board, sort_by_difficulty=True)\n if cells_with_candidates:\n guaranteed = []\n other_hints = []\n for row, col, candidates in cells_with_candidates[:10]:\n if len(candidates) == 1:\n num = list(candidates)[0]\n guaranteed.append(f\"[{row} {col} {num}]\")\n elif len(candidates) <= 3:\n nums = \",\".join(str(n) for n in sorted(candidates))\n other_hints.append(f\"({row},{col})โ†’{nums}\")\n\n if guaranteed:\n hints = f\"\\n\\n๐ŸŽฏ GUARANTEED MOVES: {', '.join(guaranteed[:5])}\"\n if other_hints:\n hints += f\"\\nOther options: {' | '.join(other_hints[:5])}\"\n\n elif difficulty == \"medium\" and board:\n # Medium: NOT sorted, just show empty cells with candidates (no ordering hints)\n cells_with_candidates = extract_empty_cells_with_candidates(board, sort_by_difficulty=False)\n if cells_with_candidates:\n cell_hints = []\n for row, col, candidates in cells_with_candidates[:10]:\n nums = \",\".join(str(n) for n in sorted(candidates))\n cell_hints.append(f\"({row},{col})โ†’{nums}\")\n if cell_hints:\n hints = f\"\\n\\nEmpty cells: {' | '.join(cell_hints)}\"\n\n return f\"{summary}\\n\\nBoard:\\n{board_only}{tried_moves_hint}{hints}\\n\\nYour move:\"\n\n\ndef check_move_targets_empty_cell(move: str, board_str: str) -> bool:\n \"\"\"Check if the move targets an empty cell on the board.\"\"\"\n if not move or not board_str:\n return False\n\n match = re.search(r\"\\[(\\d)\\s+(\\d)\\s+(\\d)\\]\", move)\n if not match:\n return False\n\n row, col = int(match.group(1)), int(match.group(2))\n empty_cells = extract_empty_cells(board_str)\n return (row, col) in empty_cells\n\n\ndef extract_feedback(observation) -> dict:\n \"\"\"Extract feedback from environment observation.\"\"\"\n feedback = {\"valid_move\": True, \"got_warning\": False, \"board_state\": \"\"}\n\n if not observation or not observation.messages:\n return feedback\n\n for message in observation.messages:\n content = message.content.lower() if message.content else \"\"\n\n if any(kw in content for kw in [\"invalid\", \"error\", \"cannot\", \"already\", \"violation\", \"lost\"]):\n feedback[\"valid_move\"] = False\n if \"please resubmit\" in content or \"avoid penalties\" in content:\n feedback[\"got_warning\"] = True\n\n if message.content and \"|\" in message.content and \"R1\" in message.content:\n feedback[\"board_state\"] = message.content\n\n return feedback\n\n\n# ---------------------------------------------------------------------------\n# Rollout\n# ---------------------------------------------------------------------------\n\n\ndef rollout_once(\n trainer: GRPOTrainer,\n env: TextArenaEnv,\n tokenizer: AutoTokenizer,\n system_prompt: str,\n max_turns: int,\n debug: bool = False,\n difficulty: str = \"hard\",\n api_delay: float = 0.0,\n) -> dict[str, list]:\n result = env.reset()\n time.sleep(api_delay) # Avoid rate limiting\n observation = result.observation\n\n # Only store the LAST turn for backprop (much more efficient!)\n last_turn_data: dict | None = None\n\n valid_move_scores: list[float] = []\n empty_cell_scores: list[float] = []\n correct_scores: list[float] = []\n repetition_scores: list[float] = []\n\n move_counts: defaultdict[str, int] = defaultdict(int)\n\n # Track successful and failed moves for summary\n successful_moves: list[str] = []\n failed_moves: list[str] = []\n\n # Extract initial board state\n last_board_state = \"\"\n initial_filled = 0\n for message in observation.messages:\n if message.content and is_valid_board_state(message.content):\n last_board_state = message.content\n initial_filled = count_filled_cells(last_board_state)\n break\n\n max_filled = initial_filled # Track max progress\n\n for turn in range(max_turns):\n if result.done:\n break\n\n # Build COMPACT prompt (saves tokens!)\n user_prompt = make_compact_prompt(\n board=last_board_state,\n step=turn + 1,\n successful_moves=successful_moves,\n failed_moves=failed_moves,\n difficulty=difficulty,\n )\n messages = [\n {\"role\": \"system\", \"content\": system_prompt},\n {\"role\": \"user\", \"content\": user_prompt},\n ]\n prompt_text = tokenizer.apply_chat_template(\n messages, add_generation_prompt=True, tokenize=False, enable_thinking=False\n )\n\n if debug:\n print(f\"\\n{'=' * 60}\")\n print(f\"STEP {turn + 1}\")\n print(f\"{'=' * 60}\")\n print(f\"USER PROMPT:\\n{user_prompt}\")\n print(f\"{'=' * 60}\")\n\n # Generate\n rollout_outputs = generate_rollout_completions(trainer, [prompt_text])[0]\n\n # Store ONLY this turn's data (replace previous)\n last_turn_data = {\n \"prompt_ids\": rollout_outputs[\"prompt_ids\"],\n \"completion_ids\": rollout_outputs[\"completion_ids\"],\n \"logprobs\": rollout_outputs[\"logprobs\"],\n }\n\n if debug:\n step_tokens = len(rollout_outputs[\"prompt_ids\"]) + len(rollout_outputs[\"completion_ids\"])\n print(f\"TOKENS: this_step={step_tokens} (only last turn used for backprop)\")\n\n completion_text = rollout_outputs.get(\"text\") or tokenizer.decode(\n rollout_outputs[\"completion_ids\"], skip_special_tokens=True\n )\n\n # Extract move\n move = extract_sudoku_move(completion_text)\n\n if debug:\n print(f\"MODEL OUTPUT: {completion_text}\")\n print(f\"EXTRACTED MOVE: {move}\")\n\n # Step environment\n result = env.step(TextArenaAction(message=move))\n time.sleep(api_delay) # Avoid rate limiting\n observation = result.observation\n correct_score = float(result.reward or 0.0)\n\n # Get feedback\n feedback = extract_feedback(observation)\n\n # Get environment response\n env_response = \"\"\n for msg in observation.messages:\n if msg.sender_id == -1: # Environment message\n env_response = msg.content\n break\n\n if debug:\n print(\n f\"ENV RESPONSE: {env_response[:200]}...\"\n if len(env_response) > 200\n else f\"ENV RESPONSE: {env_response}\"\n )\n print(f\"VALID: {feedback['valid_move']}, WARNING: {feedback['got_warning']}, REWARD: {correct_score}\")\n\n # Calculate empty_cell_score\n if last_board_state and move:\n targets_empty = check_move_targets_empty_cell(move, last_board_state)\n empty_cell_score = 1.0 if targets_empty else -1.0\n else:\n empty_cell_score = 0.0\n\n # Calculate valid_move_score and repetition_score\n is_new_move = move_counts[move] == 0\n repetition_count = move_counts[move]\n move_counts[move] += 1\n\n # Exponential penalty for repetitions: -2^(n-1) capped at -10\n # 1st repeat: -1, 2nd: -2, 3rd: -4, 4th+: -10 (capped)\n if repetition_count > 0:\n repetition_score = -min(2 ** (repetition_count - 1), 10.0)\n else:\n repetition_score = 0.0\n\n if debug:\n print(\n f\"SCORES: empty_cell={empty_cell_score}, is_new={is_new_move}, repetitions={repetition_count}, rep_penalty={repetition_score}\"\n )\n\n if not debug:\n print(f\"Step {turn + 1}: {move}\")\n\n if feedback[\"valid_move\"] and is_new_move:\n valid_move_score = 1.0\n if move:\n successful_moves.append(move) # Track for summary\n elif feedback[\"got_warning\"]:\n valid_move_score = -0.5\n if move:\n failed_moves.append(move) # Track for summary\n else:\n valid_move_score = 0.0\n\n # Update board state and track progress\n if feedback[\"board_state\"] and is_valid_board_state(feedback[\"board_state\"]):\n last_board_state = feedback[\"board_state\"]\n current_filled = count_filled_cells(last_board_state)\n if current_filled > max_filled:\n max_filled = current_filled\n\n valid_move_scores.append(valid_move_score)\n empty_cell_scores.append(empty_cell_score)\n correct_scores.append(correct_score)\n repetition_scores.append(repetition_score)\n\n # Aggregate rewards\n correct_reward = correct_scores[-1] if correct_scores else 0.0\n valid_move_reward = sum(valid_move_scores) / len(valid_move_scores) if valid_move_scores else 0.0\n empty_cell_reward = sum(empty_cell_scores) / len(empty_cell_scores) if empty_cell_scores else 0.0\n repetition_reward = sum(repetition_scores) / len(repetition_scores) if repetition_scores else 0.0\n\n # Progress reward: how many cells we filled beyond initial state (normalized to 0-1)\n # 81 total cells, so (max_filled - initial_filled) / (81 - initial_filled) gives progress\n remaining_to_fill = 81 - initial_filled\n if remaining_to_fill > 0:\n progress_reward = (max_filled - initial_filled) / remaining_to_fill\n else:\n progress_reward = 1.0 # Already complete\n\n # Use ONLY last turn for backpropagation (much more efficient!)\n if last_turn_data:\n prompt_ids = last_turn_data[\"prompt_ids\"]\n completion_ids = last_turn_data[\"completion_ids\"]\n logprobs = last_turn_data[\"logprobs\"]\n else:\n prompt_ids = []\n completion_ids = []\n logprobs = []\n\n total_tokens = len(prompt_ids) + len(completion_ids)\n cells_filled = max_filled - initial_filled\n print(\n f\"Episode: empty_cell={empty_cell_reward:.2f}, valid={valid_move_reward:.2f}, \"\n f\"repetition={repetition_reward:.2f}, progress={progress_reward:.2f} ({cells_filled} cells), \"\n f\"correct={correct_reward:.2f}, tokens={total_tokens}\"\n )\n\n return {\n \"prompt_ids\": prompt_ids,\n \"completion_ids\": completion_ids,\n \"logprobs\": logprobs,\n \"correct_reward\": correct_reward,\n \"valid_move_reward\": valid_move_reward,\n \"empty_cell_reward\": empty_cell_reward,\n \"repetition_reward\": repetition_reward,\n \"progress_reward\": progress_reward,\n }\n\n\n# ---------------------------------------------------------------------------\n# Reward functions\n# ---------------------------------------------------------------------------\n\n\ndef reward_empty_cell(completions: list[str], **kwargs) -> list[float]:\n \"\"\"Reward for targeting empty cells (learn to pick valid positions first).\"\"\"\n rewards = kwargs.get(\"empty_cell_reward\")\n if rewards is None:\n return [0.0 for _ in completions]\n return [float(r) for r in rewards]\n\n\ndef reward_valid_moves(completions: list[str], **kwargs) -> list[float]:\n \"\"\"Reward for making valid moves.\"\"\"\n rewards = kwargs.get(\"valid_move_reward\")\n if rewards is None:\n return [0.0 for _ in completions]\n return [float(r) for r in rewards]\n\n\ndef reward_correct(completions: list[str], **kwargs) -> list[float]:\n \"\"\"Reward for solving the puzzle.\"\"\"\n rewards = kwargs.get(\"correct_reward\")\n if rewards is None:\n return [0.0 for _ in completions]\n return [float(r) for r in rewards]\n\n\ndef reward_repetition(completions: list[str], **kwargs) -> list[float]:\n \"\"\"Penalty for repeating moves.\"\"\"\n rewards = kwargs.get(\"repetition_reward\")\n if rewards is None:\n return [0.0 for _ in completions]\n return [float(r) for r in rewards]\n\n\ndef reward_progress(completions: list[str], **kwargs) -> list[float]:\n \"\"\"Reward for filling more cells in the board.\"\"\"\n rewards = kwargs.get(\"progress_reward\")\n if rewards is None:\n return [0.0 for _ in completions]\n return [float(r) for r in rewards]\n\n\n# ---------------------------------------------------------------------------\n# Main\n# ---------------------------------------------------------------------------\n\n\ndef main() -> None:\n args = parse_args()\n\n # Setup environment\n if args.env_mode == \"docker-local\":\n client = TextArenaEnv(base_url=f\"http://{args.env_host}:{args.env_port}\")\n elif args.env_mode == \"docker-image\":\n client = TextArenaEnv.from_docker_image(args.env_image)\n elif args.env_mode == \"docker-hub\":\n client = TextArenaEnv.from_hub(args.env_image)\n elif args.env_mode == \"space\":\n client = TextArenaEnv(base_url=args.env_host)\n else:\n raise ValueError(f\"Unknown environment mode: {args.env_mode}\")\n\n print(f\"๐ŸŒ Environment: {args.env_mode}\")\n\n system_prompt = resolve_system_prompt(args.system_prompt_path)\n dataset = Dataset.from_dict({\"prompt\": [args.dataset_prompt] * args.dataset_size})\n\n timestamp = datetime.now().strftime(\"%Y-%m-%d_%H-%M-%S\")\n output_dir = Path(args.output_dir or f\"outputs/sudoku-grpo-{sanitize_name(args.model_id)}-{timestamp}\")\n\n grpo_config = GRPOConfig(\n use_vllm=True,\n vllm_mode=args.vllm_mode,\n vllm_server_base_url=args.vllm_server_url if args.vllm_mode == \"server\" else None,\n vllm_gpu_memory_utilization=args.vllm_gpu_memory_utilization\n if args.vllm_gpu_memory_utilization\n else 0.2, # Lower to leave more VRAM for backpropagation\n output_dir=str(output_dir),\n num_train_epochs=args.num_epochs,\n learning_rate=args.learning_rate,\n weight_decay=args.weight_decay,\n gradient_accumulation_steps=args.gradient_accumulation_steps,\n per_device_train_batch_size=args.per_device_batch_size,\n warmup_steps=args.warmup_steps,\n num_generations=args.num_generations,\n max_completion_length=args.max_new_tokens,\n logging_steps=args.logging_steps,\n save_strategy=\"steps\",\n save_steps=args.save_interval,\n save_total_limit=args.save_total_limit,\n temperature=args.temperature,\n top_k=args.top_k,\n top_p=args.top_p,\n report_to=\"trackio\",\n # chat_template_kwargs={\"enable_thinking\": False},\n )\n\n grpo_config.run_name = args.run_name or f\"run-{timestamp}\"\n grpo_config.project = args.project or f\"group-{sanitize_name(args.model_id)}\"\n grpo_config.trackio_space_id = args.trackio_space_id\n grpo_config.gradient_checkpointing = args.gradient_checkpointing\n\n def rollout_func(prompts: list[str], trainer: GRPOTrainer) -> dict[str, list]:\n all_prompt_ids = []\n all_completion_ids = []\n all_logprobs = []\n all_correct = []\n all_valid = []\n all_empty_cell = []\n all_repetition = []\n all_progress = []\n\n for _ in prompts:\n episode = rollout_once(\n trainer=trainer,\n env=client,\n tokenizer=trainer.processing_class,\n system_prompt=system_prompt,\n max_turns=args.max_turns,\n debug=args.debug,\n difficulty=args.difficulty,\n api_delay=args.api_delay,\n )\n all_prompt_ids.append(episode[\"prompt_ids\"])\n all_completion_ids.append(episode[\"completion_ids\"])\n all_logprobs.append(episode[\"logprobs\"])\n all_correct.append(episode[\"correct_reward\"])\n all_valid.append(episode[\"valid_move_reward\"])\n all_empty_cell.append(episode[\"empty_cell_reward\"])\n all_repetition.append(episode[\"repetition_reward\"])\n all_progress.append(episode[\"progress_reward\"])\n\n return {\n \"prompt_ids\": all_prompt_ids,\n \"completion_ids\": all_completion_ids,\n \"logprobs\": all_logprobs,\n \"correct_reward\": all_correct,\n \"valid_move_reward\": all_valid,\n \"empty_cell_reward\": all_empty_cell,\n \"repetition_reward\": all_repetition,\n \"progress_reward\": all_progress,\n }\n\n trainer = GRPOTrainer(\n model=args.model_id,\n reward_funcs=[\n reward_empty_cell, # Learn to pick empty cells\n reward_valid_moves, # Learn valid numbers\n reward_repetition, # Penalize repeating moves\n reward_progress, # Reward filling more cells\n reward_correct, # Solve the puzzle\n ],\n train_dataset=dataset,\n args=grpo_config,\n rollout_func=rollout_func,\n )\n\n print(f\"๐Ÿš€ Starting GRPO training: {args.num_generations} generations, {args.max_turns} max turns\")\n\n try:\n trainer.train()\n finally:\n client.close()\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/openenv/sudoku_prompt.txt", "content": "You are an expert Sudoku player with deep knowledge of logical deduction strategies and number placement techniques.\n\n## GAME RULES\n\n1. The puzzle is a 9x9 grid divided into nine 3x3 subgrids (boxes)\n2. Some cells are pre-filled with numbers 1-9\n3. You must fill in the empty cells (shown as '.') with numbers 1-9\n4. Each row must contain numbers 1-9 without repetition\n5. Each column must contain numbers 1-9 without repetition\n6. Each 3x3 subgrid must contain numbers 1-9 without repetition\n7. You cannot overwrite pre-filled cells\n8. Invalid moves result in penalties (-1 reward)\n\n## RESPONSE FORMAT\n\n**CRITICAL: Output ONLY the move, nothing else. No text, no explanation.**\n\nFormat: [row col number]\n\nExamples:\n- [5 3 7] โ†’ places 7 in row 5, column 3\n- [1 2 4] โ†’ places 4 in row 1, column 2\n\n## STRATEGIC APPROACH\n\nDo not repeat the same move twice.\n\n### Basic Strategies\n- **Naked Singles**: If a cell has only one possible candidate, fill it in immediately.\n- **Hidden Singles**: If a number can only go in one cell within a row, column, or box, place it there.\n- **Scanning**: Look at each row, column, and box to find where specific numbers can go.\n\n### Intermediate Strategies\n- **Naked Pairs/Triples**: When two/three cells in a unit contain only the same candidates, eliminate those from other cells.\n- **Hidden Pairs/Triples**: When numbers only appear in specific cells within a unit, those cells can only contain those numbers.\n- **Pointing Pairs**: When a candidate in a box is restricted to a single row/column, eliminate it elsewhere.\n\n### Solving Process\n1. Start by scanning the entire grid to identify easy fills (cells with few candidates)\n2. Look for rows, columns, or boxes with many numbers already placed\n3. Fill all naked singles first\n4. Then look for hidden singles in each row, column, and box\n5. Apply more advanced techniques as needed\n\n### Common Pitfalls to Avoid\n- Don't guess randomly - Sudoku is pure logic\n- Don't overlook any constraint (row, column, or box)\n- Don't try to overwrite pre-filled cells\n- Don't place invalid numbers (must be 1-9)\n- Don't use invalid coordinates (must be 1-9)\n- Don't repeat a move that was already made\n\n## EXAMPLES\n\n### Example 1: Naked Single\nIf row 3, column 4 can only contain the number 5:\n[3 4 5]\n\n### Example 2: Hidden Single\nIf the number 8 can only go in one cell in row 1:\n[1 7 8]\n\n### Example 3: Row Analysis\nRow 2 is missing only value 5, and column 8 is the empty cell:\n[2 8 5]\n\n### Example 4: Box Analysis\nIn the center box, only one cell can contain 9:\n[5 5 9]\n\n## BOARD READING\n\nThe board is displayed as a 9x9 grid:\n- Numbers 1-9 are pre-filled or already placed\n- Empty cells are shown as '.'\n- Rows are labeled R1-R9 (top to bottom)\n- Columns are labeled C1-C9 (left to right)\n\nExample board representation:\n```\n C1 C2 C3 C4 C5 C6 C7 C8 C9 \nR1 . 8 9 | 1 . . | . 3 7\nR2 2 7 1 | 9 4 3 | 6 . 8\nR3 . 6 5 | . 2 7 | 4 9 .\n - - - - - - - - - - - - - - - - \nR4 . . . | 7 8 . | 9 2 3\nR5 . 9 2 | . 5 6 | . . 4\nR6 7 3 8 | . . 2 | 1 . .\n - - - - - - - - - - - - - - - - \nR7 8 4 . | . . 9 | 5 . .\nR8 5 . . | 6 . 8 | 3 4 9\nR9 9 . 6 | 5 3 4 | 8 7 2\n```\n\n## COORDINATE REFERENCE\n\nRow indices (top to bottom): 1, 2, 3, 4, 5, 6, 7, 8, 9\nColumn indices (left to right): 1, 2, 3, 4, 5, 6, 7, 8, 9\n\nSubgrid layout:\n```\nSubgrid 1 | Subgrid 2 | Subgrid 3\n (R1-R3) (R1-R3) (R1-R3)\n (C1-C3) (C4-C6) (C7-C9)\n----------+-----------+----------\nSubgrid 4 | Subgrid 5 | Subgrid 6\n (R4-R6) (R4-R6) (R4-R6)\n (C1-C3) (C4-C6) (C7-C9)\n----------+-----------+----------\nSubgrid 7 | Subgrid 8 | Subgrid 9\n (R7-R9) (R7-R9) (R7-R9)\n (C1-C3) (C4-C6) (C7-C9)\n```\n\n## IMPORTANT CONSTRAINTS\n\n- Coordinates are 1-indexed (1-9 for both row and column)\n- Numbers must be 1-9\n- One move per response\n- Must be a valid move (no rule violations)\n- Never repeat a previous move\n\n## YOUR GOAL\n\nOutput ONLY your move in the format [row col number]. No explanation, no reasoning, just the move.\n" }, { "path": "examples/scripts/openenv/wordle.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"trackio\",\n# \"openenv-textarena @ git+https://huggingface.co/spaces/openenv/wordle\",\n# ]\n# ///\n\n\n\"\"\"\nSimple script to run GRPO training with OpenEnv's Wordle environment and vLLM.\n\"\"\"\n\nfrom datasets import Dataset\nfrom textarena_env import TextArenaAction, TextArenaEnv\n\nfrom trl import GRPOConfig, GRPOTrainer\n\n\nprompt = \"\"\"You are an expert Wordle solver with deep knowledge of English vocabulary, letter frequency patterns, and optimal guessing strategies.\n\nFollow these rules to play Wordle:\n\n1. The target is a 5-letter English word\n2. You have 6 attempts to guess the correct word\n3. After each guess, you receive color-coded feedback:\n - GREEN (G): Letter is correct and in the correct position\n - YELLOW (Y): Letter is in the word but in the wrong position\n - GRAY (X): Letter is not in the word at all\n4. All guesses must be valid 5-letter English words\n5. You cannot reuse a word you've already guessed\n6. Use the tool `guess` to make a guess.\n\"\"\"\n\n\nclass WordleEnv:\n def __init__(self):\n self.client = TextArenaEnv(base_url=\"https://openenv-wordle.hf.space\")\n\n def reset(self, **kwargs) -> None | str:\n result = self.client.reset()\n # The game returns cumulative feedback each turn (new text appended at the end), so\n # we store the previous full response and slice out only the newly appended part.\n self._last_full_feedback = result.observation.messages[0].content\n self.reward = -1.0\n self.done = False\n return self._last_full_feedback\n\n def guess(self, guess: str) -> str:\n \"\"\"\n Make a guess in the Wordle environment.\n\n Args:\n guess: The guessed word, formatted as '[abcde]'\n\n Returns:\n The feedback message from the environment.\n \"\"\"\n if self.done:\n self.reward = -1.0 # Penalize guesses after game is done\n raise ValueError(\"Game over.\")\n result = self.client.step(TextArenaAction(message=guess))\n _full_feedback = result.observation.messages[0].content\n # Just take the new feedback since the last guess, which is the part appended to the end of the full feedback\n feedback = _full_feedback[len(self._last_full_feedback) :]\n self._last_full_feedback = _full_feedback\n # For some reason, the environment doesn't penalize invalid moves and just returns the last reward.\n # We check the feedback for the invalid move message and penalize it if found.\n if \"You attempted an invalid move\" in feedback:\n self.reward = -1.0 # Penalize invalid moves\n else:\n self.reward = result.reward\n self.done = result.done\n return feedback\n\n\ndef reward(environments, **kwargs) -> list[float]:\n return [environment.reward for environment in environments]\n\n\ndef main() -> None:\n dataset = Dataset.from_dict({\"prompt\": [[{\"role\": \"user\", \"content\": prompt}] for _ in range(1000)]})\n\n trainer = GRPOTrainer(\n model=\"Qwen/Qwen3-1.7B\",\n reward_funcs=reward,\n train_dataset=dataset,\n args=GRPOConfig(\n report_to=\"trackio\",\n trackio_space_id=\"wordle-grpo\",\n log_completions=True,\n num_completions_to_print=2,\n logging_steps=1,\n chat_template_kwargs={\"enable_thinking\": False},\n max_completion_length=1024,\n ),\n environment_factory=WordleEnv,\n )\n trainer.train()\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/orpo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nRun the ORPO training script with the following command with some example arguments.\nIn general, the optimal configuration for ORPO will be similar to that of DPO without the need for a reference model:\n\n# regular:\npython examples/scripts/orpo.py \\\n --dataset_name trl-internal-testing/hh-rlhf-helpful-base-trl-style \\\n --model_name_or_path gpt2 \\\n --per_device_train_batch_size 4 \\\n --max_steps 1000 \\\n --learning_rate 8e-6 \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 500 \\\n --output_dir \"gpt2-aligned-orpo\" \\\n --warmup_steps 150 \\\n --logging_first_step \\\n --no_remove_unused_columns\n\n# peft:\npython examples/scripts/orpo.py \\\n --dataset_name trl-internal-testing/hh-rlhf-helpful-base-trl-style \\\n --model_name_or_path gpt2 \\\n --per_device_train_batch_size 4 \\\n --max_steps 1000 \\\n --learning_rate 8e-5 \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 500 \\\n --output_dir \"gpt2-lora-aligned-orpo\" \\\n --optim rmsprop \\\n --warmup_steps 150 \\\n --logging_first_step \\\n --no_remove_unused_columns \\\n --use_peft \\\n --lora_r 16 \\\n --lora_alpha 16\n\"\"\"\n\nimport os\n\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser\n\nfrom trl import ModelConfig, ScriptArguments, get_peft_config\nfrom trl.experimental.orpo import ORPOConfig, ORPOTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser((ScriptArguments, ORPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_into_dataclasses()\n\n ################\n # Model & Tokenizer\n ################\n model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n ################\n # Dataset\n ################\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n ################\n # Training\n ################\n trainer = ORPOTrainer(\n model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n processing_class=tokenizer,\n peft_config=get_peft_config(model_args),\n )\n\n # train and save the model\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/ppo/ppo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\nimport os\nimport shutil\n\nimport torch\nfrom accelerate import PartialState\nfrom datasets import load_dataset\nfrom transformers import (\n AutoModelForCausalLM,\n AutoModelForSequenceClassification,\n AutoTokenizer,\n HfArgumentParser,\n)\n\nfrom trl import ModelConfig, ScriptArguments, get_kbit_device_map, get_peft_config, get_quantization_config\nfrom trl.experimental.ppo import PPOConfig, PPOTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\n\"\"\"\npython -i examples/scripts/ppo/ppo.py \\\n --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \\\n --dataset_train_split descriptiveness \\\n --output_dir pythia-1b-deduped-descriptiveness-sentiment-trl-style-ppo \\\n --per_device_train_batch_size 64 \\\n --gradient_accumulation_steps 1 \\\n --total_episodes 10000 \\\n --model_name_or_path EleutherAI/pythia-1b-deduped \\\n --missing_eos_penalty 1.0\n\naccelerate launch --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/ppo/ppo.py \\\n --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \\\n --dataset_train_split descriptiveness \\\n --output_dir pythia-1b-deduped-descriptiveness-sentiment-trl-style-ppo \\\n --num_ppo_epochs 1 \\\n --num_mini_batches 1 \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 16 \\\n --total_episodes 10000 \\\n --model_name_or_path EleutherAI/pythia-1b-deduped \\\n --sft_model_path EleutherAI/pythia-1b-deduped \\\n --reward_model_path EleutherAI/pythia-1b-deduped \\\n --local_rollout_forward_batch_size 1 \\\n --missing_eos_penalty 1.0\n\"\"\"\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser((ScriptArguments, PPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_into_dataclasses()\n # remove output_dir if exists\n shutil.rmtree(training_args.output_dir, ignore_errors=True)\n\n ################\n # Model & Tokenizer\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, padding_side=\"left\", trust_remote_code=model_args.trust_remote_code\n )\n tokenizer.add_special_tokens({\"pad_token\": \"[PAD]\"})\n value_model = AutoModelForSequenceClassification.from_pretrained(\n training_args.reward_model_path,\n trust_remote_code=model_args.trust_remote_code,\n num_labels=1,\n **model_kwargs,\n )\n reward_model = AutoModelForSequenceClassification.from_pretrained(\n training_args.reward_model_path,\n trust_remote_code=model_args.trust_remote_code,\n num_labels=1,\n **model_kwargs,\n )\n policy = AutoModelForCausalLM.from_pretrained(\n training_args.sft_model_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n\n peft_config = get_peft_config(model_args)\n if peft_config is None:\n ref_policy = AutoModelForCausalLM.from_pretrained(\n training_args.sft_model_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n else:\n ref_policy = None\n\n ################\n # Dataset\n ################\n dataset = load_dataset(\n script_args.dataset_name, name=script_args.dataset_config, split=script_args.dataset_train_split\n )\n eval_samples = 100\n train_dataset = dataset.select(range(len(dataset) - eval_samples))\n eval_dataset = dataset.select(range(len(dataset) - eval_samples, len(dataset)))\n dataset_text_field = \"prompt\"\n\n def prepare_dataset(dataset, tokenizer):\n \"\"\"pre-tokenize the dataset before training; only collate during training\"\"\"\n\n def tokenize(element):\n outputs = tokenizer(\n element[dataset_text_field],\n padding=False,\n )\n return {\"input_ids\": outputs[\"input_ids\"]}\n\n return dataset.map(\n tokenize,\n batched=True,\n remove_columns=dataset.column_names,\n num_proc=training_args.dataset_num_proc,\n )\n\n # Compute that only on the main process for faster data processing.\n # see: https://github.com/huggingface/trl/pull/1255\n with PartialState().local_main_process_first():\n train_dataset = prepare_dataset(train_dataset, tokenizer)\n eval_dataset = prepare_dataset(eval_dataset, tokenizer)\n\n ################\n # Training\n ################\n trainer = PPOTrainer(\n args=training_args,\n processing_class=tokenizer,\n model=policy,\n ref_model=ref_policy,\n reward_model=reward_model,\n value_model=value_model,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n peft_config=peft_config,\n )\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n\n trainer.generate_completions()\n" }, { "path": "examples/scripts/ppo/ppo_tldr.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\nimport os\nimport shutil\n\nimport torch\nfrom accelerate import PartialState\nfrom datasets import load_dataset\nfrom transformers import (\n AutoModelForCausalLM,\n AutoModelForSequenceClassification,\n AutoTokenizer,\n HfArgumentParser,\n)\n\nfrom trl import ModelConfig, ScriptArguments, get_kbit_device_map, get_peft_config, get_quantization_config\nfrom trl.experimental.ppo import PPOConfig, PPOTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\n\"\"\"\npython examples/scripts/ppo/ppo_tldr.py \\\n --dataset_name trl-lib/tldr \\\n --dataset_test_split validation \\\n --output_dir pythia-1b-deduped-tldr-preference-sft-trl-style-ppo \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 64 \\\n --total_episodes 30000 \\\n --model_name_or_path EleutherAI/pythia-1b-deduped \\\n --sft_model_path cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr \\\n --reward_model_path cleanrl/EleutherAI_pythia-1b-deduped__reward__tldr \\\n --missing_eos_penalty 1.0 \\\n --stop_token eos \\\n --response_length 53 \\\n --eval_strategy steps \\\n --eval_steps 100\n\naccelerate launch --config_file examples/accelerate_configs/deepspeed_zero2.yaml \\\n examples/scripts/ppo/ppo_tldr.py \\\n --dataset_name trl-lib/tldr \\\n --dataset_test_split validation \\\n --output_dir pythia-1b-deduped-tldr-preference-sft-trl-style-ppo \\\n --per_device_train_batch_size 16 \\\n --gradient_accumulation_steps 4 \\\n --total_episodes 1000000 \\\n --model_name_or_path EleutherAI/pythia-1b-deduped \\\n --sft_model_path cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr \\\n --reward_model_path cleanrl/EleutherAI_pythia-1b-deduped__reward__tldr \\\n --local_rollout_forward_batch_size 16 \\\n --missing_eos_penalty 1.0 \\\n --stop_token eos \\\n --eval_strategy steps \\\n --eval_steps 100\n\"\"\"\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser((ScriptArguments, PPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_into_dataclasses()\n # remove output_dir if exists\n shutil.rmtree(training_args.output_dir, ignore_errors=True)\n\n ################\n # Model & Tokenizer\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, padding_side=\"left\", trust_remote_code=model_args.trust_remote_code\n )\n tokenizer.add_special_tokens({\"pad_token\": \"[PAD]\"})\n value_model = AutoModelForSequenceClassification.from_pretrained(\n training_args.reward_model_path,\n trust_remote_code=model_args.trust_remote_code,\n num_labels=1,\n **model_kwargs,\n )\n reward_model = AutoModelForSequenceClassification.from_pretrained(\n training_args.reward_model_path,\n trust_remote_code=model_args.trust_remote_code,\n num_labels=1,\n **model_kwargs,\n )\n policy = AutoModelForCausalLM.from_pretrained(\n training_args.sft_model_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n\n peft_config = get_peft_config(model_args)\n if peft_config is None:\n ref_policy = AutoModelForCausalLM.from_pretrained(\n training_args.sft_model_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n else:\n ref_policy = None\n\n ################\n # Dataset\n ################\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n train_dataset = dataset[script_args.dataset_train_split]\n eval_dataset = dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None\n\n def prepare_dataset(dataset, tokenizer):\n \"\"\"pre-tokenize the dataset before training; only collate during training\"\"\"\n\n def tokenize(element):\n input_ids = tokenizer(element[\"prompt\"], padding=False)[\"input_ids\"]\n return {\"input_ids\": input_ids, \"lengths\": len(input_ids)}\n\n return dataset.map(\n tokenize,\n remove_columns=dataset.column_names,\n num_proc=training_args.dataset_num_proc,\n )\n\n # Compute that only on the main process for faster data processing.\n # see: https://github.com/huggingface/trl/pull/1255\n with PartialState().local_main_process_first():\n train_dataset = prepare_dataset(train_dataset, tokenizer)\n if eval_dataset is not None:\n eval_dataset = prepare_dataset(eval_dataset, tokenizer)\n # filtering\n train_dataset = train_dataset.filter(lambda x: x[\"lengths\"] <= 512, num_proc=training_args.dataset_num_proc)\n if eval_dataset is not None:\n eval_dataset = eval_dataset.filter(lambda x: x[\"lengths\"] <= 512, num_proc=training_args.dataset_num_proc)\n\n assert train_dataset[0][\"input_ids\"][-1] != tokenizer.eos_token_id, \"The last token should not be an EOS token\"\n\n ################\n # Training\n ################\n trainer = PPOTrainer(\n args=training_args,\n processing_class=tokenizer,\n model=policy,\n ref_model=ref_policy,\n reward_model=reward_model,\n value_model=value_model,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n peft_config=peft_config,\n )\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n\n trainer.generate_completions()\n" }, { "path": "examples/scripts/prm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nFull training:\npython examples/scripts/prm.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/prm800k \\\n --output_dir Qwen2-0.5B-Reward \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-5 \\\n --eval_strategy steps \\\n --eval_steps 50\n\nLoRA:\npython examples/scripts/prm.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/prm800k \\\n --output_dir Qwen2-0.5B-Reward-LoRA \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-4 \\\n --eval_strategy steps \\\n --eval_steps 50\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16\n\"\"\"\n\nimport os\n\nimport torch\nfrom accelerate import logging\nfrom datasets import load_dataset\nfrom transformers import AutoModelForTokenClassification, AutoTokenizer, HfArgumentParser\n\nfrom trl import (\n ModelConfig,\n ScriptArguments,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\nfrom trl.experimental.prm import PRMConfig, PRMTrainer\n\n\nlogger = logging.get_logger(__name__)\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser((ScriptArguments, PRMConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_into_dataclasses()\n\n ################\n # Model & Tokenizer\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n use_cache=False if training_args.gradient_checkpointing else True,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, use_fast=True\n )\n model = AutoModelForTokenClassification.from_pretrained(\n model_args.model_name_or_path, num_labels=2, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n # Align padding tokens between tokenizer and model\n model.config.pad_token_id = tokenizer.pad_token_id\n\n if model_args.use_peft and model_args.lora_task_type != \"TOKEN_CLS\":\n logger.warning(\n \"You are using a `task_type` that is different than `TOKEN_CLS` for PEFT. This will lead to silent bugs\"\n \" Make sure to pass --lora_task_type TOKEN_CLS when using this script with PEFT.\",\n )\n\n ##############\n # Load dataset\n ##############\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n dataset = dataset.filter(lambda x: len(x[\"completions\"]) > 0)\n\n ##########\n # Training\n ##########\n trainer = PRMTrainer(\n model=model,\n processing_class=tokenizer,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split],\n peft_config=get_peft_config(model_args),\n )\n trainer.train()\n\n ############################\n # Save model and push to Hub\n ############################\n trainer.save_model(training_args.output_dir)\n metrics = trainer.evaluate()\n trainer.log_metrics(\"eval\", metrics)\n trainer.save_metrics(\"eval\", metrics)\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/reward_modeling.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nFull training:\npython examples/scripts/reward_modeling.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --output_dir Qwen2-0.5B-Reward \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-5 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --max_length 2048\n\nLoRA:\npython examples/scripts/reward_modeling.py \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --output_dir Qwen2-0.5B-Reward-LoRA \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-4 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --max_length 2048 \\\n --use_peft \\\n --lora_task_type SEQ_CLS \\\n --lora_r 32 \\\n --lora_alpha 16\n\"\"\"\n\nimport os\n\nimport torch\nfrom accelerate import logging\nfrom datasets import load_dataset\nfrom transformers import AutoModelForSequenceClassification, HfArgumentParser\n\nfrom trl import (\n ModelConfig,\n RewardConfig,\n RewardTrainer,\n ScriptArguments,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\n\n\nlogger = logging.get_logger(__name__)\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser((ScriptArguments, RewardConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_into_dataclasses()\n\n ################\n # Model & Tokenizer\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n use_cache=False if training_args.gradient_checkpointing else True,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n model = AutoModelForSequenceClassification.from_pretrained(\n model_args.model_name_or_path, num_labels=1, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n\n if model_args.use_peft and model_args.lora_task_type != \"SEQ_CLS\":\n logger.warning(\n \"You are using a `task_type` that is different than `SEQ_CLS` for PEFT. This will lead to silent bugs\"\n \" Make sure to pass --lora_task_type SEQ_CLS when using this script with PEFT.\",\n )\n\n ##############\n # Load dataset\n ##############\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n ##########\n # Training\n ##########\n trainer = RewardTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=get_peft_config(model_args),\n )\n trainer.train()\n\n ############################\n # Save model and push to Hub\n ############################\n trainer.save_model(training_args.output_dir)\n\n if training_args.eval_strategy != \"no\":\n metrics = trainer.evaluate()\n trainer.log_metrics(\"eval\", metrics)\n trainer.save_metrics(\"eval\", metrics)\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/rloo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[vllm]\",\n# \"peft\",\n# \"math-verify\",\n# \"latex2sympy2_extended\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nNuminaMath example: RLOO on math dataset with vLLM.\n\n pip install math_verify num2words==0.5.14 peft trackio vllm\n export TRACKIO_PROJECT=\"RLOO-NuminaMath-TIR\"\n accelerate launch --config_file examples/accelerate_configs/deepspeed_zero3.yaml examples/scripts/rloo.py\n\nFor TL;DR or other datasets with a reward model, use the generic script:\n python -m trl.scripts.rloo --dataset_name trl-lib/tldr --reward_model_name_or_path ... --model_name_or_path ...\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\nfrom peft import LoraConfig\n\nfrom trl import RLOOConfig, RLOOTrainer\nfrom trl.rewards import accuracy_reward, think_format_reward\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef main():\n # Dataset\n train_dataset, eval_dataset = load_dataset(\"AI-MO/NuminaMath-TIR\", split=[\"train[:5%]\", \"test[:5%]\"])\n\n SYSTEM_PROMPT = (\n \"A conversation between user and assistant. The user asks a question, and the assistant solves it. The \"\n \"assistant first thinks about the reasoning process in the mind and then provides the user with the answer. \"\n \"The reasoning process and answer are enclosed within tags, i.e., \\nThis is my \"\n \"reasoning.\\n\\nThis is my answer.\"\n )\n\n def make_conversation(example):\n return {\n \"prompt\": [\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": example[\"problem\"]},\n ],\n }\n\n train_dataset = train_dataset.map(make_conversation, remove_columns=[\"messages\", \"problem\"])\n eval_dataset = eval_dataset.map(make_conversation, remove_columns=[\"messages\", \"problem\"])\n\n # Training\n training_args = RLOOConfig(\n output_dir=\"Qwen3-0.6B-RLOO\",\n model_init_kwargs={\"dtype\": torch.bfloat16},\n learning_rate=1e-5,\n log_completions=True,\n num_completions_to_print=2,\n max_completion_length=1024,\n gradient_accumulation_steps=2,\n steps_per_generation=8,\n use_vllm=True,\n vllm_mode=\"colocate\",\n vllm_gpu_memory_utilization=0.5,\n run_name=\"Qwen3-0.6B-RLOO-NuminaMath-TIR\",\n )\n\n trainer = RLOOTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n args=training_args,\n reward_funcs=[think_format_reward, accuracy_reward],\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n peft_config=LoraConfig(),\n )\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n trainer.push_to_hub(dataset_name=\"AI-MO/NuminaMath-TIR\")\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/rloo_vlm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"Pillow\",\n# \"peft\",\n# \"math-verify\",\n# \"latex2sympy2_extended\",\n# \"torchvision\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\npip install math_verify\n\n# For Qwen/Qwen2.5-VL-3B-Instruct\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/rloo_vlm.py \\\n --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \\\n --output_dir rloo-Qwen2.5-VL-3B-Instruct \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_completion_length 1024 \\\n --use_vllm \\\n --vllm_mode colocate \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --log_completions\n\n# For HuggingFaceTB/SmolVLM2-2.2B-Instruct\npip install num2words==0.5.14\n\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/rloo_vlm.py \\\n --model_name_or_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \\\n --output_dir rloo-SmolVLM2-2.2B-Instruct \\\n --learning_rate 1e-5 \\\n --dtype bfloat16 \\\n --max_completion_length 1024 \\\n --use_peft \\\n --lora_target_modules \"q_proj\", \"v_proj\" \\\n --log_completions \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 2 \\\n --num_generations 2\n\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\n\nfrom trl import (\n ModelConfig,\n RLOOConfig,\n RLOOTrainer,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\nfrom trl.rewards import accuracy_reward, think_format_reward\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, RLOOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n ################\n # Model\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n training_args.model_init_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n training_args.model_init_kwargs[\"device_map\"] = get_kbit_device_map()\n training_args.model_init_kwargs[\"quantization_config\"] = quantization_config\n\n ################\n # Dataset\n ################\n dataset = load_dataset(\"lmms-lab/multimodal-open-r1-8k-verified\", split=\"train\")\n dataset = dataset.train_test_split(test_size=100, seed=42)\n\n SYSTEM_PROMPT = (\n \"A conversation between user and assistant. The user asks a question, and the assistant solves it. The \"\n \"assistant first thinks about the reasoning process in the mind and then provides the user with the answer. \"\n \"The reasoning process and answer are enclosed within tags, i.e., \\nThis is my \"\n \"reasoning.\\n\\nThis is my answer.\"\n )\n\n def make_conversation(example):\n prompt = [\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": example[\"problem\"]},\n ]\n return {\"prompt\": prompt}\n\n dataset = dataset.map(make_conversation)\n\n # Filter have big images\n def filter_big_images(example):\n image = example[\"image\"]\n return image.size[0] < 512 and image.size[1] < 512\n\n dataset = dataset.filter(filter_big_images)\n\n def convert_to_rgb(example):\n image = example[\"image\"]\n if image.mode != \"RGB\":\n image = image.convert(\"RGB\")\n example[\"image\"] = image\n return example\n\n dataset = dataset.map(convert_to_rgb)\n\n train_dataset = dataset[\"train\"]\n eval_dataset = dataset[\"test\"] if training_args.eval_strategy != \"no\" else None\n\n ################\n # Training\n ################\n trainer = RLOOTrainer(\n model=model_args.model_name_or_path,\n args=training_args,\n reward_funcs=[think_format_reward, accuracy_reward],\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n peft_config=get_peft_config(model_args),\n )\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/sft.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n###############################################################################################\n# This file has been moved to https://github.com/huggingface/trl/blob/main/trl/scripts/sft.py #\n###############################################################################################\n" }, { "path": "examples/scripts/sft_gemma3.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"Pillow\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nTrain Gemma-3 on the Codeforces COTS dataset.\n\naccelerate launch --config_file examples/accelerate_configs/deepspeed_zero3.yaml examples/scripts/sft_gemma3.py\n\"\"\"\n\nimport os\n\nfrom datasets import load_dataset\nfrom transformers import AutoModelForImageTextToText\n\nfrom trl import SFTConfig, SFTTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef main():\n # Load dataset\n train_dataset = load_dataset(\"open-r1/codeforces-cots\", split=\"train\")\n train_dataset = train_dataset.remove_columns(\"prompt\")\n\n # Load model\n model_id = \"google/gemma-3-12b-it\"\n model = AutoModelForImageTextToText.from_pretrained(model_id, attn_implementation=\"eager\")\n\n # Train model\n training_args = SFTConfig(\n output_dir=f\"{model_id}-codeforces-SFT\",\n bf16=True,\n use_liger_kernel=True,\n max_length=8192,\n per_device_train_batch_size=1,\n gradient_accumulation_steps=8,\n dataset_num_proc=32,\n num_train_epochs=1,\n )\n\n trainer = SFTTrainer(\n args=training_args,\n model=model,\n train_dataset=train_dataset,\n )\n trainer.train()\n\n # Push to hub\n trainer.push_to_hub(dataset_name=\"open-r1/codeforces-cots\")\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/sft_gpt_oss.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"kernels\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\npip install โ€“-upgrade kernels\n\nExample:\n\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/sft_gpt_oss.py \\\n --dtype bfloat16 \\\n --model_name_or_path openai/gpt-oss-20b \\\n --packing \\\n --run_name 20b-full-eager \\\n --attn_implementation kernels-community/vllm-flash-attn3 \\\n --dataset_num_proc 12 \\\n --dataset_name HuggingFaceH4/Multilingual-Thinking \\\n --max_length 4096 \\\n --per_device_train_batch_size 2 \\\n --num_train_epochs 1 \\\n --logging_steps 1 \\\n --warmup_steps 0.03 \\\n --lr_scheduler_type cosine_with_min_lr \\\n --lr_scheduler_kwargs '{\"min_lr_rate\": 0.1}' \\\n --output_dir gpt-oss-20b-multilingual-reasoner \\\n --report_to trackio \\\n --seed 42\n\"\"\"\n\nimport os\n\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, Mxfp4Config\n\nfrom trl import ModelConfig, ScriptArguments, SFTConfig, SFTTrainer, TrlParser, get_peft_config\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef main(script_args, training_args, model_args):\n # Load model\n quantization_config = Mxfp4Config(dequantize=True)\n model_kwargs = dict(\n revision=model_args.model_revision,\n trust_remote_code=model_args.trust_remote_code,\n attn_implementation=model_args.attn_implementation,\n dtype=model_args.dtype,\n use_cache=False if training_args.gradient_checkpointing else True,\n quantization_config=quantization_config,\n )\n\n model = AutoModelForCausalLM.from_pretrained(model_args.model_name_or_path, **model_kwargs)\n\n # Load dataset\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n # Train model\n trainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=get_peft_config(model_args),\n )\n\n trainer.train()\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, SFTConfig, ModelConfig))\n script_args, training_args, model_args, _ = parser.parse_args_and_config(return_remaining_strings=True)\n main(script_args, training_args, model_args)\n" }, { "path": "examples/scripts/sft_nemotron_3.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[peft,quantization]\",\n# \"transformers>=5.3.0\",\n# \"trackio\",\n# \"mamba_ssm==2.2.5\",\n# \"causal_conv1d==1.5.2\",\n# ]\n# ///\n\n\"\"\"\nFine-tune NVIDIA Nemotron 3 models with SFT.\n\nPrerequisites:\n\n pip install \"transformers>=5.3.0\"\n pip install --no-build-isolation mamba_ssm==2.2.5\n pip install --no-build-isolation causal_conv1d==1.5.2\n\nExample:\n\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/sft_nemotron_3.py \\\n --dtype bfloat16 \\\n --model_name_or_path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \\\n --attn_implementation eager \\\n --dataset_name HuggingFaceH4/Multilingual-Thinking \\\n --max_length 128 \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 4 \\\n --num_train_epochs 1 \\\n --learning_rate 2e-4 \\\n --optim paged_adamw_8bit \\\n --logging_steps 10 \\\n --output_dir nemotron-3-sft \\\n --report_to trackio \\\n --use_peft \\\n --lora_r 8 \\\n --lora_alpha 16 \\\n --lora_target_modules q_proj k_proj v_proj o_proj gate_proj up_proj down_proj\n\"\"\"\n\nimport os\n\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM\n\nfrom trl import ModelConfig, ScriptArguments, SFTConfig, SFTTrainer, TrlParser, get_peft_config\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef main(script_args, training_args, model_args):\n # NemotronH does not support gradient checkpointing\n training_args.gradient_checkpointing = False\n\n # Load model\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=model_args.dtype,\n )\n model = AutoModelForCausalLM.from_pretrained(model_args.model_name_or_path, **model_kwargs)\n\n # Load dataset\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n # Merge thinking into message content using tags and remove extra columns\n def merge_thinking_and_remove_key(example):\n new_messages = []\n for msg in example[\"messages\"]:\n content = msg[\"content\"]\n thinking = msg.get(\"thinking\")\n if thinking and isinstance(thinking, str) and thinking.strip():\n content = f\"\\n{thinking}\\n\\n{content}\"\n new_messages.append({\"role\": msg[\"role\"], \"content\": content})\n example[\"messages\"] = new_messages\n return example\n\n dataset = dataset.map(merge_thinking_and_remove_key)\n\n # Prepare eval dataset if needed\n eval_dataset = None\n if training_args.eval_strategy != \"no\" and script_args.dataset_test_split in dataset:\n eval_dataset = dataset[script_args.dataset_test_split]\n\n # Train model\n trainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=eval_dataset,\n peft_config=get_peft_config(model_args),\n )\n\n trainer.train()\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, SFTConfig, ModelConfig))\n script_args, training_args, model_args, _ = parser.parse_args_and_config(return_remaining_strings=True)\n main(script_args, training_args, model_args)\n" }, { "path": "examples/scripts/sft_tiny_aya_tool_calling.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl[peft]\",\n# \"bitsandbytes\",\n# \"liger-kernel\",\n# \"trackio\",\n# ]\n# ///\n\n\"\"\"\nTeach tool calling to CohereLabs/tiny-aya-global using SFT with QLoRA on the bebechien/SimpleToolCalling dataset.\n\nThe model used in this script does not have native tool-calling support. We extend its existing Jinja2 chat template to\nserialize tool schemas into the system preamble and render tool calls as structured XML inside the model's\nnative <|START_RESPONSE|> / <|END_RESPONSE|> delimiters. The modified template is saved with the tokenizer, so\ninference only requires loading the tokenizer from the output directory and calling apply_chat_template with\ntools=TOOLS โ€” no manual system-prompt construction needed.\n\nExample:\n\n python examples/scripts/sft_tiny_aya_tool_calling.py\n\"\"\"\n\nimport json\nfrom pathlib import Path\n\nimport torch\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nfrom trl import SFTConfig, SFTTrainer\n\n\n# These are the tool schemas that are used in the dataset\nTOOLS = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"search_knowledge_base\",\n \"description\": \"Search internal company documents, policies and project data.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\"query\": {\"type\": \"string\", \"description\": \"query string\"}},\n \"required\": [\"query\"],\n },\n \"return\": {\"type\": \"string\"},\n },\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"search_google\",\n \"description\": \"Search public information.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\"query\": {\"type\": \"string\", \"description\": \"query string\"}},\n \"required\": [\"query\"],\n },\n \"return\": {\"type\": \"string\"},\n },\n },\n]\n\n\ndef create_conversation(sample):\n return {\n \"prompt\": [{\"role\": \"user\", \"content\": sample[\"user_content\"]}],\n \"completion\": [\n {\n \"role\": \"assistant\",\n \"tool_calls\": [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": sample[\"tool_name\"],\n \"arguments\": json.loads(sample[\"tool_arguments\"]),\n },\n }\n ],\n },\n ],\n \"tools\": TOOLS,\n }\n\n\ndef main():\n model_id = \"CohereLabs/tiny-aya-global\"\n dataset_name = \"bebechien/SimpleToolCalling\"\n output_dir = \"tiny-aya-global-tool-calling-SFT\"\n\n # Load and format dataset\n dataset = load_dataset(dataset_name, split=\"train\")\n dataset = dataset.map(create_conversation, remove_columns=dataset.features)\n dataset = dataset.train_test_split(test_size=0.5, shuffle=True)\n\n # Load model\n model = AutoModelForCausalLM.from_pretrained(\n model_id,\n attn_implementation=\"sdpa\",\n dtype=torch.float16,\n quantization_config=BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type=\"nf4\",\n ),\n )\n\n # Configure LoRA\n peft_config = LoraConfig(\n r=32,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"],\n )\n\n # Train\n training_args = SFTConfig(\n output_dir=output_dir,\n per_device_train_batch_size=1,\n gradient_accumulation_steps=4,\n # Use the tool-aware chat template\n chat_template_path=str(Path(__file__).parent / \"tiny_aya_chat_template.jinja\"),\n warmup_steps=5,\n learning_rate=2e-4,\n optim=\"paged_adamw_8bit\",\n logging_steps=1,\n report_to=\"trackio\",\n trackio_space_id=output_dir,\n max_length=1024,\n use_liger_kernel=True,\n activation_offloading=True,\n push_to_hub=True,\n )\n\n trainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset[\"train\"],\n peft_config=peft_config,\n )\n trainer.train()\n\n # Save model and tokenizer (tokenizer carries the updated chat template)\n trainer.save_model(output_dir)\n trainer.push_to_hub(dataset_name=dataset_name)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/sft_video_llm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"qwen-vl-utils\",\n# \"torchvision\",\n# \"bitsandbytes\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nExample usage:\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero2.yaml \\\n examples/scripts/sft_video_llm.py \\\n --dataset_name mfarre/simplevideoshorts \\\n --video_cache_dir \"/optional/path/to/cache/\" \\\n --model_name_or_path Qwen/Qwen2-VL-7B-Instruct \\\n --per_device_train_batch_size 1 \\\n --output_dir video-llm-output \\\n --tf32 True \\\n --gradient_accumulation_steps 4 \\\n --num_train_epochs 4 \\\n --optim adamw_torch_fused \\\n --log_level debug \\\n --log_level_replica debug \\\n --save_strategy steps \\\n --save_steps 300 \\\n --learning_rate 8e-5 \\\n --max_grad_norm 0.3 \\\n --warmup_steps 0.1 \\\n --lr_scheduler_type cosine \\\n --push_to_hub False \\\n --dtype bfloat16\n\"\"\"\n\nimport json\nimport os\nimport random\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nimport requests\nimport torch\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom qwen_vl_utils import process_vision_info\nfrom transformers import AutoModelForImageTextToText, AutoProcessor, BitsAndBytesConfig, Qwen2VLProcessor\n\nfrom trl import ModelConfig, ScriptArguments, SFTConfig, SFTTrainer, TrlParser, get_kbit_device_map\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef download_video(url: str, cache_dir: str) -> str:\n \"\"\"Download video if not already present locally.\"\"\"\n os.makedirs(cache_dir, exist_ok=True) # Create cache dir if it doesn't exist\n filename = url.split(\"/\")[-1]\n local_path = os.path.join(cache_dir, filename)\n\n if os.path.exists(local_path):\n return local_path\n\n try:\n with requests.get(url, stream=True) as r:\n r.raise_for_status()\n with open(local_path, \"wb\") as f:\n for chunk in r.iter_content(chunk_size=8192):\n if chunk:\n f.write(chunk)\n return local_path\n except requests.RequestException as e:\n raise Exception(f\"Failed to download video: {e}\") from e\n\n\ndef prepare_dataset(example: dict[str, Any], cache_dir: str) -> dict[str, list[dict[str, Any]]]:\n \"\"\"Prepare dataset example for training.\"\"\"\n video_url = example[\"video_url\"]\n timecoded_cc = example[\"timecoded_cc\"]\n qa_pairs = json.loads(example[\"qa\"])\n\n system_message = \"You are an expert in movie narrative analysis.\"\n base_prompt = f\"\"\"Analyze the video and consider the following timecoded subtitles:\n\n{timecoded_cc}\n\nBased on this information, please answer the following questions:\"\"\"\n\n selected_qa = random.sample(qa_pairs, 1)[0]\n\n messages = [\n {\"role\": \"system\", \"content\": [{\"type\": \"text\", \"text\": system_message}]},\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"video\", \"video\": download_video(video_url, cache_dir), \"max_pixels\": 360 * 420, \"fps\": 1.0},\n {\"type\": \"text\", \"text\": f\"{base_prompt}\\n\\nQuestion: {selected_qa['question']}\"},\n ],\n },\n {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": selected_qa[\"answer\"]}]},\n ]\n\n return {\"messages\": messages}\n\n\ndef collate_fn(examples: list[dict[str, Any]]) -> dict[str, torch.Tensor]:\n \"\"\"Collate batch of examples for training.\"\"\"\n texts = []\n video_inputs = []\n\n for i, example in enumerate(examples):\n try:\n video_path = next(\n content[\"video\"]\n for message in example[\"messages\"]\n for content in message[\"content\"]\n if content.get(\"type\") == \"video\"\n )\n print(f\"Processing video: {os.path.basename(video_path)}\")\n\n texts.append(processor.apply_chat_template(example[\"messages\"], tokenize=False))\n video_input = process_vision_info(example[\"messages\"])[1][0]\n video_inputs.append(video_input)\n except Exception as e:\n raise ValueError(f\"Failed to process example {i}: {e}\") from e\n\n inputs = processor(text=texts, videos=video_inputs, return_tensors=\"pt\", padding=True)\n\n labels = inputs[\"input_ids\"].clone()\n labels[labels == processor.tokenizer.pad_token_id] = -100\n\n # Handle visual tokens based on processor type\n visual_tokens = (\n [151652, 151653, 151656]\n if isinstance(processor, Qwen2VLProcessor)\n else [processor.tokenizer.convert_tokens_to_ids(processor.image_token)]\n )\n\n for visual_token_id in visual_tokens:\n labels[labels == visual_token_id] = -100\n\n inputs[\"labels\"] = labels\n return inputs\n\n\n@dataclass\nclass CustomScriptArguments(ScriptArguments):\n r\"\"\"\n Arguments for the script.\n\n Args:\n video_cache_dir (`str`, *optional*, defaults to `\"/tmp/videos/\"`):\n Video cache directory.\n \"\"\"\n\n video_cache_dir: str = field(default=\"/tmp/videos/\", metadata={\"help\": \"Video cache directory.\"})\n\n\nif __name__ == \"__main__\":\n # Parse arguments\n parser = TrlParser((CustomScriptArguments, SFTConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n\n # Configure training args\n training_args.remove_unused_columns = False\n\n # Load dataset\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config, split=\"train\")\n\n # Setup model\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n\n # Quantization configuration for 4-bit training\n bnb_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_compute_dtype=torch.bfloat16,\n )\n\n # Model initialization\n model_kwargs = dict(\n revision=model_args.model_revision,\n trust_remote_code=model_args.trust_remote_code,\n dtype=dtype,\n device_map=get_kbit_device_map(),\n quantization_config=bnb_config,\n )\n\n model = AutoModelForImageTextToText.from_pretrained(model_args.model_name_or_path, **model_kwargs)\n\n peft_config = LoraConfig(\n task_type=\"CAUSAL_LM\",\n r=16,\n lora_alpha=16,\n lora_dropout=0.1,\n bias=\"none\",\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n )\n\n # Configure model modules for gradients\n if training_args.gradient_checkpointing:\n model.gradient_checkpointing_enable()\n model.config.use_reentrant = False\n model.enable_input_require_grads()\n\n processor = AutoProcessor.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n\n # Prepare dataset\n prepared_dataset = [prepare_dataset(example, script_args.video_cache_dir) for example in dataset]\n\n # Initialize trainer\n trainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=prepared_dataset,\n data_collator=collate_fn,\n peft_config=peft_config,\n processing_class=processor,\n )\n\n # Train model\n trainer.train()\n\n # Save final model\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n\n # Cleanup\n del model\n del trainer\n torch.cuda.empty_cache()\n" }, { "path": "examples/scripts/sft_vlm.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"Pillow>=9.4.0\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\npip install pillow\n\n# Tested on 8x H100 GPUs\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/sft_vlm.py \\\n --dataset_name HuggingFaceH4/llava-instruct-mix-vsft \\\n --model_name_or_path llava-hf/llava-1.5-7b-hf \\\n --gradient_accumulation_steps 8 \\\n --output_dir LLaVA-1.5-7B-SFT \\\n --dtype bfloat16\n\nFor LLaVA-NeXT, use:\n --model_name_or_path llava-hf/llava-v1.6-mistral-7b-hf\n\nFor meta-llama/Llama-3.2-11B-Vision-Instruct, use:\n --model_name_or_path meta-llama/Llama-3.2-11B-Vision-Instruct\n\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/sft_vlm.py \\\n --dataset_name HuggingFaceH4/llava-instruct-mix-vsft \\\n --model_name_or_path HuggingFaceTB/SmolVLM-Instruct \\\n --per_device_train_batch_size 1 \\\n --gradient_accumulation_steps 1 \\\n --output_dir SmolVLM-SFT \\\n --dtype bfloat16 \\\n --use_peft \\\n --lora_target_modules down_proj, o_proj, k_proj, q_proj, gate_proj, up_proj, v_proj\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForImageTextToText\n\nfrom trl import (\n ModelConfig,\n ScriptArguments,\n SFTConfig,\n SFTTrainer,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, SFTConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n training_args.max_length = None\n\n ################\n # Model\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n model = AutoModelForImageTextToText.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n\n ################\n # Dataset\n ################\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n ################\n # Training\n ################\n trainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=get_peft_config(model_args),\n )\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "examples/scripts/sft_vlm_gemma3.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"Pillow>=9.4.0\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nTrain Gemma 3 on the HuggingFaceH4/llava-instruct-mix-vsft dataset (single-image).\n\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/sft_vlm_gemma3.py \\\n --dataset_name HuggingFaceH4/llava-instruct-mix-vsft \\\n --model_name_or_path google/gemma-3-4b-it \\\n --per_device_train_batch_size 1 \\\n --output_dir Gemma-3-4B-SFT-MMIU \\\n --dtype bfloat16 \\\n --use_peft \\\n --lora_target_modules all-linear \\\n --attn_implementation eager\n\nTrain Gemma 3 on the FanqingM/MMIU-Benchmark dataset (multi-image).\n\naccelerate launch \\\n --config_file examples/accelerate_configs/deepspeed_zero3.yaml \\\n examples/scripts/sft_vlm_gemma3.py \\\n --dataset_name FanqingM/MMIU-Benchmark \\\n --dataset_train_split test \\\n --model_name_or_path google/gemma-3-4b-it \\\n --per_device_train_batch_size 1 \\\n --output_dir Gemma-3-4B-SFT-MMIU \\\n --dtype bfloat16 \\\n --use_peft \\\n --lora_target_modules all-linear \\\n --attn_implementation eager\n\"\"\"\n\nimport io\nimport os\nimport zipfile\n\nimport torch\nfrom datasets import DatasetDict, load_dataset\nfrom huggingface_hub import hf_hub_download, list_repo_files\nfrom PIL import Image\nfrom transformers import AutoModelForImageTextToText\n\nfrom trl import (\n ModelConfig,\n ScriptArguments,\n SFTConfig,\n SFTTrainer,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\n# For multi-image example\ndef process_vision_info(messages: list[dict]) -> list[Image.Image]:\n image_inputs = []\n for msg in messages:\n content = msg.get(\"content\", [])\n if not isinstance(content, list):\n content = [content]\n\n for element in content:\n if isinstance(element, dict) and (\"image\" in element or element.get(\"type\") == \"image\"):\n if \"image\" in element:\n image = element[\"image\"]\n else:\n image = element\n if image is not None:\n image = Image.open(io.BytesIO(image[\"bytes\"]))\n image_inputs.append(image.convert(\"RGB\"))\n return image_inputs\n\n\ndef format_data(samples: dict[str, any]) -> dict[str, list]:\n formatted_samples = {\"messages\": []}\n for cont in range(len(samples[\"question\"])):\n images = []\n for img_path in samples[\"input_image_path\"][cont]:\n try:\n with open(img_path, \"rb\") as f:\n img_bytes = f.read()\n image = Image.open(io.BytesIO(img_bytes)).convert(\"RGB\")\n images.append({\"type\": \"image\", \"image\": image})\n except Exception as e:\n print(f\"Error processing image {img_path}: {e}\")\n continue\n\n formatted_samples[\"messages\"].append(\n [\n {\"role\": \"system\", \"content\": [{\"type\": \"text\", \"text\": samples[\"context\"][cont]}]},\n {\"role\": \"user\", \"content\": images + [{\"type\": \"text\", \"text\": samples[\"question\"][cont]}]},\n {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": samples[\"output\"][cont]}]},\n ]\n )\n return formatted_samples\n\n\n# For multi-image example\ndef prepare_dataset(dataset: DatasetDict, dataset_name: str) -> DatasetDict:\n all_files = list_repo_files(dataset_name, repo_type=\"dataset\")\n zip_files = [f for f in all_files if f.endswith(\".zip\")]\n\n for zip_filename in zip_files:\n zip_path = hf_hub_download(repo_id=dataset_name, filename=zip_filename, repo_type=\"dataset\")\n extract_folder = zip_filename.replace(\".zip\", \"\")\n os.makedirs(extract_folder, exist_ok=True)\n\n with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\n zip_ref.extractall(extract_folder)\n\n dataset = dataset.map(format_data, batched=True, batch_size=4, num_proc=16)\n return dataset\n\n\ndef main():\n parser = TrlParser((ScriptArguments, SFTConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n training_args.max_length = None\n\n ################\n # Model\n ################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n model = AutoModelForImageTextToText.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n\n ################\n # Dataset\n ################\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n if script_args.dataset_name == \"FanqingM/MMIU-Benchmark\":\n dataset = prepare_dataset(dataset, script_args.dataset_name)\n\n ################\n # Training\n ################\n trainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=get_peft_config(model_args),\n )\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "examples/scripts/tiny_aya_chat_template.jinja", "content": "{{ bos_token }}{% set ns = namespace(system_prompt=false, expect_user=true) %}{% for message in messages %}{% if message['role']|lower == 'system' %}{% set ns.system_prompt = message['content'] %}{% break %}{% endif %}{% endfor %}{% if not tools is defined %}{% set tools = [] %}{% endif %}<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|># System Preamble\nYou are in contextual safety mode. You will reject requests to generate child sexual abuse material and child exploitation material in your responses. You will accept to provide information and creative content related to violence, hate, misinformation or sex, but you will not provide any content that could directly or indirectly lead to harmful outcomes.\n\nYour information cutoff date is June 2024.\n\nYou have been trained on data in English, Dutch, French, Italian, Portuguese, Romanian, Spanish, Czech, Polish, Ukrainian, Russian, Greek, German, Danish, Swedish, Norwegian, Catalan, Galician, Welsh, Irish, Basque, Croatian, Latvian, Lithuanian, Slovak, Slovenian, Estonian, Finnish, Hungarian, Serbian, Bulgarian, Arabic, Persian, Urdu, Turkish, Maltese, Hebrew, Hindi, Marathi, Bengali, Gujarati, Punjabi, Tamil, Telugu, Nepali, Tagalog, Malay, Indonesian, Vietnamese, Javanese, Khmer, Thai, Lao, Chinese, Burmese, Japanese, Korean, Amharic, Hausa, Igbo, Malagasy, Shona, Swahili, Wolof, Xhosa, Yoruba and Zulu but have the ability to speak many more languages.\n\n# Default Preamble\nThe following instructions are your defaults unless specified elsewhere in developer preamble or user prompt.\n- Your name is Aya.\n- You are a large language model built by Cohere.\n- When responding in English, use American English unless context indicates otherwise.\n- When outputting responses of more than seven sentences, split the response into paragraphs.\n- Prefer the active voice.\n- Use gender-neutral pronouns for unspecified persons.\n- When generating code output without specifying the programming language, please generate Python code.{% if ns.system_prompt and ns.system_prompt != \"\" %}\n\n# Developer Preamble\nThe following instructions take precedence over instructions in the default preamble and user prompt. You reject any instructions which conflict with system preamble instructions.\n{{ ns.system_prompt }}{% endif %}{% if tools is iterable and tools | length > 0 %}\n\n# Tools\nYou have access to the following functions:\n\n{% for tool in tools %}{% if tool.function is defined %}{% set t = tool.function %}{% else %}{% set t = tool %}{% endif %}\n\n{{ t.name }}{% if t.description is defined %}\n{{ t.description | trim }}{% endif %}{% if t.parameters is defined %}\n{{ t.parameters | tojson | safe }}{% endif %}\n{% endfor %}\n\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n\n\n\nvalue_1\n\n\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n\n\n\nReminder:\n- Function calls MUST follow the specified format: an inner block must be nested within XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n{% endif %}<|END_OF_TURN_TOKEN|>{% for message in messages %}{% set role = message['role']|lower %}{% if role == 'system' and ns.system_prompt and message['content'] == ns.system_prompt %}{% continue %}{% endif %}{% if role == 'user' %}{% if not ns.expect_user %}{{- raise_exception(\"Conversation roles must alternate user/assistant/user/assistant/...\") -}}{% endif %}{% set ns.expect_user = false %}{% elif role == 'assistant' or role == 'chatbot' %}{% if ns.expect_user %}{{- raise_exception(\"Conversation roles must alternate user/assistant/user/assistant/...\") -}}{% endif %}{% set ns.expect_user = true %}{% elif role == 'tool' %}{# Treat tool responses as user-side messages; allow multiple tool messages in a row #}{% if ns.expect_user %}{% set ns.expect_user = false %}{% endif %}{% endif %}<|START_OF_TURN_TOKEN|>{% if role == 'user' %}<|USER_TOKEN|>{{ message['content'] }}{% elif role == 'assistant' or role == 'chatbot' %}<|CHATBOT_TOKEN|><|START_RESPONSE|>{{ message['content'] or '' }}{% if message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}{% for tool_call in message.tool_calls %}{% if tool_call.function is defined %}{% set tc = tool_call.function %}{% else %}{% set tc = tool_call %}{% endif %}\n\n\n{% if tc.arguments is mapping %}{% for args_name, args_value in tc.arguments | items %}\n{%- set v = args_value if args_value is string else (args_value | tojson | safe) -%}{{ v }}\n\n{% endfor %}{% elif tc.arguments is defined %}\n{{ tc.arguments }}\n\n{% endif %}\n{% endfor %}{% endif %}<|END_RESPONSE|>{% elif role == 'tool' %}<|USER_TOKEN|>\n{{ message['content'] or '' }}\n{% elif role == 'system' %}<|SYSTEM_TOKEN|>{{ message['content'] }}{% endif %}<|END_OF_TURN_TOKEN|>{% endfor %}{% if add_generation_prompt %}<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|START_RESPONSE|>{% endif %}\n" }, { "path": "examples/scripts/xpo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nUsage:\n\npython examples/scripts/xpo.py \\\n --model_name_or_path trl-lib/pythia-1b-deduped-tldr-sft \\\n --reward_model_path trl-lib/pythia-1b-deduped-tldr-rm \\\n --dataset_name trl-lib/tldr \\\n --learning_rate 5.0e-7 \\\n --output_dir pythia-1b-tldr-xpo \\\n --per_device_train_batch_size 4 \\\n --gradient_accumulation_steps 32 \\\n --num_train_epochs 3 \\\n --max_new_tokens 64 \\\n --warmup_steps 0.1 \\\n --missing_eos_penalty 1.0 \\\n --push_to_hub\n\"\"\"\n\nimport os\n\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer, GenerationConfig\n\nfrom trl import (\n LogCompletionsCallback,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_quantization_config,\n)\nfrom trl.experimental.judges import HfPairwiseJudge, OpenAIPairwiseJudge, PairRMJudge\nfrom trl.experimental.xpo import XPOConfig, XPOTrainer\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\nJUDGES = {\"pair_rm\": PairRMJudge, \"openai\": OpenAIPairwiseJudge, \"hf\": HfPairwiseJudge}\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, XPOConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n training_args.gradient_checkpointing_kwargs = {\"use_reentrant\": True}\n\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n use_cache=False if training_args.gradient_checkpointing else True,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n ref_model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n\n if training_args.reward_model_path is not None:\n reward_model = AutoModelForSequenceClassification.from_pretrained(\n training_args.reward_model_path,\n num_labels=1,\n trust_remote_code=model_args.trust_remote_code,\n **model_kwargs,\n )\n else:\n reward_model = None\n\n if training_args.judge is not None:\n judge_cls = JUDGES[training_args.judge]\n judge = judge_cls()\n else:\n judge = None\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, padding_side=\"left\", trust_remote_code=model_args.trust_remote_code\n )\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n trainer = XPOTrainer(\n model=model,\n ref_model=ref_model,\n reward_funcs=reward_model,\n judge=judge,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n processing_class=tokenizer,\n )\n\n if training_args.eval_strategy != \"no\":\n generation_config = GenerationConfig(\n max_new_tokens=training_args.max_new_tokens, do_sample=True, temperature=training_args.temperature\n )\n completions_callback = LogCompletionsCallback(trainer, generation_config, num_prompts=8)\n trainer.add_callback(completions_callback)\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "pyproject.toml", "content": "[build-system]\nrequires = [\"setuptools >= 77.0.3\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"trl\"\ndescription = \"Train transformer language models with reinforcement learning.\"\nauthors = [\n { name = \"Leandro von Werra\", email = \"leandro.vonwerra@gmail.com\" }\n]\nreadme = { file = \"README.md\", content-type = \"text/markdown\" }\nlicense = \"Apache-2.0\"\nlicense-files = [\"LICENSE\"]\nkeywords = [\n \"transformers\", \"huggingface\", \"language modeling\", \"post-training\", \"rlhf\", \"sft\", \"dpo\", \"grpo\"\n]\nclassifiers = [\n \"Development Status :: 2 - Pre-Alpha\",\n \"Intended Audience :: Developers\",\n \"Intended Audience :: Science/Research\",\n \"Natural Language :: English\",\n \"Operating System :: OS Independent\",\n \"Programming Language :: Python :: 3\",\n \"Programming Language :: Python :: 3.10\",\n \"Programming Language :: Python :: 3.11\",\n \"Programming Language :: Python :: 3.12\",\n \"Programming Language :: Python :: 3.13\",\n \"Programming Language :: Python :: 3.14\"\n]\nrequires-python = \">=3.10\"\ndependencies = [\n \"accelerate>=1.4.0\",\n \"datasets>=3.0.0\",\n \"packaging>20.0\",\n \"transformers>=4.56.2\",\n]\ndynamic = [\"version\"]\n\n[project.urls]\nHomepage = \"https://github.com/huggingface/trl\"\n\n[project.scripts]\ntrl = \"trl.cli:main\"\n\n[project.optional-dependencies]\nbco = [\n \"scikit-learn\",\n \"joblib\"\n]\ndeepspeed = [\n \"deepspeed>=0.14.4\",\n \"transformers!=5.1.0\", # see transformers#43780\n]\njudges = [\n \"openai>=1.23.2\",\n \"llm-blender>=0.0.2\",\n \"transformers<5.0.0\", # see #4918\n]\nkernels = [\n \"kernels\"\n]\nliger = [\n \"liger-kernel>=0.7.0\"\n]\npeft = [\n \"peft>=0.8.0\"\n]\nquality = [\n \"pre-commit\",\n \"hf-doc-builder\"\n]\nquantization = [\n \"bitsandbytes\"\n]\nscikit = [\n \"scikit-learn\"\n]\ntest = [\n \"pytest-cov\",\n \"pytest-datadir>=1.7.0\", # lazy datadirs\n \"pytest-rerunfailures==15.1\",\n \"pytest-xdist\",\n \"pytest\"\n]\nvllm = [\n \"vllm>=0.10.2,<=0.17.1\",\n \"fastapi\",\n \"pydantic\",\n \"aiohttp>=3.13.3\",\n \"requests\",\n \"uvicorn\"\n]\nvlm = [\n \"Pillow\",\n \"torchvision\",\n \"num2words==0.5.14\"\n]\nmath_verify = [\n \"math-verify>=0.5.2\",\n]\ndev = [\n # bco\n \"scikit-learn\",\n \"joblib\",\n # deepspeed\n \"deepspeed>=0.14.4\",\n # judges\n \"openai>=1.23.2\",\n \"llm-blender>=0.0.2\",\n # kernels\n \"kernels\",\n # liger\n \"liger-kernel>=0.7.0\",\n # peft\n \"peft>=0.8.0\",\n # quality\n \"pre-commit\",\n \"hf-doc-builder\",\n # quantization\n \"bitsandbytes\",\n # scikit: included in bco\n # test\n \"pytest-cov\",\n \"pytest-datadir>=1.7.0\", # lazy datadirs\n \"pytest-rerunfailures==15.1\",\n \"pytest-xdist\",\n \"pytest\",\n # vllm: not included in dev by default due to CUDA error; see GH-4228\n # vlm\n \"Pillow\",\n \"torchvision\",\n \"num2words==0.5.14\",\n # for response parsing (required for training with tools)\n \"jmespath\",\n]\n\n[tool.setuptools]\npackage-dir = {\"trl\" = \"trl\"}\n\n[tool.setuptools.dynamic]\nversion = { file = \"VERSION\" }\n\n[tool.coverage.run]\nbranch = true\n\n[tool.ruff]\ntarget-version = \"py310\"\nline-length = 119\nsrc = [\"trl\"]\n\n[tool.ruff.lint]\nignore = [\n \"B028\", # warning without explicit stacklevel\n \"C408\", # dict() calls (stylistic)\n \"C901\", # function complexity\n \"E501\",\n]\nextend-select = [\"E\", \"F\", \"I\", \"W\", \"UP\", \"B\", \"T\", \"C\"]\n\n[tool.ruff.lint.per-file-ignores]\n# Allow prints in auxiliary scripts\n\"examples/**.py\" = [\"T201\"]\n\"scripts/**.py\" = [\"T201\"]\n\"trl/cli/**.py\" = [\"T201\"]\n\"trl/skills/cli.py\" = [\"T201\"]\n# Ignore import violations in all `__init__.py` files.\n\"__init__.py\" = [\"F401\"]\n\n[tool.ruff.lint.isort]\nlines-after-imports = 2\nknown-first-party = [\"trl\"]\n\n[tool.pytest.ini_options]\nmarkers = [\n \"slow: marks tests as slow (deselect with '-m \\\"not slow\\\"')\",\n \"low_priority: marks tests as low priority (deselect with '-m \\\"not low_priority\\\"')\"\n]\nnorecursedirs = [\n \"tests/experimental\",\n]\nfilterwarnings = [\n # SWIG deprecations from SWIG-generated C/C++ extensions: sentencepiece\n # Upstream issue: https://github.com/google/sentencepiece/issues/1150\n \"ignore:builtin type SwigPyPacked has no __module__ attribute:DeprecationWarning\",\n \"ignore:builtin type SwigPyObject has no __module__ attribute:DeprecationWarning\",\n \"ignore:builtin type swigvarlink has no __module__ attribute:DeprecationWarning\",\n\n # PyTorch JIT deprecations (upstream, not actionable in TRL)\n # Upstream issue: https://github.com/deepspeedai/DeepSpeed/issues/7835\n # Upstream PR: https://github.com/deepspeedai/DeepSpeed/pull/7840\n # Upstream fix released in deepspeed v0.18.6: https://github.com/deepspeedai/DeepSpeed/releases/tag/v0.18.6\n \"ignore:`torch.jit.script_method` is deprecated:DeprecationWarning\",\n \"ignore:`torch.jit.script` is deprecated:DeprecationWarning\",\n\n # PyTorch DataLoader pin_memory device argument deprecations\n # Triggered internally by torch.utils.data, not by our code\n # Upstream issue: https://github.com/pytorch/pytorch/issues/174546\n \"ignore:The argument 'device' of Tensor.pin_memory:DeprecationWarning\",\n \"ignore:The argument 'device' of Tensor.is_pinned:DeprecationWarning\",\n]\n" }, { "path": "requirements.txt", "content": "accelerate>=1.4.0\ndatasets>=3.0.0\ntransformers>=4.56.2" }, { "path": "scripts/add_copyrights.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport os\nimport subprocess\nimport sys\nfrom datetime import datetime\n\n\nCOPYRIGHT_HEADER = f\"\"\"# Copyright 2020-{datetime.now().year} The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\n\n\ndef get_tracked_python_files():\n \"\"\"Get a list of all tracked Python files using git.\"\"\"\n try:\n # Get the list of all tracked files from Git\n result = subprocess.run([\"git\", \"ls-files\"], stdout=subprocess.PIPE, text=True, check=True)\n # Split the result by lines to get individual file paths\n files = result.stdout.splitlines()\n # Filter only Python files\n py_files = [f for f in files if f.endswith(\".py\")]\n return py_files\n except subprocess.CalledProcessError as e:\n print(f\"Error fetching tracked files: {e}\")\n return []\n\n\ndef check_and_add_copyright(file_path):\n \"\"\"Check if the file contains a copyright notice, and add it if missing.\"\"\"\n if not os.path.isfile(file_path):\n print(f\"[SKIP] {file_path} does not exist.\")\n return\n\n with open(file_path, encoding=\"utf-8\") as f:\n content = f.readlines()\n\n # Check if the exact copyright header exists\n if \"\".join(content).startswith(COPYRIGHT_HEADER):\n return True\n\n # If no copyright notice was found, prepend the header\n print(f\"[MODIFY] Adding copyright to {file_path}.\")\n with open(file_path, \"w\", encoding=\"utf-8\") as f:\n # Write the copyright header followed by the original content\n f.write(COPYRIGHT_HEADER + \"\\n\" + \"\".join(content))\n return False\n\n\ndef main():\n \"\"\"Main function to check and add copyright for all tracked Python files.\"\"\"\n py_files = get_tracked_python_files()\n if not py_files:\n print(\"No Python files are tracked in the repository.\")\n return\n\n print(f\"Checking {len(py_files)} Python files for copyright notice...\")\n\n have_copyright = [check_and_add_copyright(file_path) for file_path in py_files]\n if not all(have_copyright):\n print(\"โŒ Some files were missing the required copyright and have been updated.\")\n sys.exit(1)\n else:\n print(\"โœ… All files have the required copyright.\")\n sys.exit(0)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "scripts/generate_harmony_dataset.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import Dataset\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n test_size (`float`, *optional*, defaults to `0.1`):\n Fraction of the dataset to include in the test split.\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-internal-testing/harmony\"`):\n Hugging Face repository ID to push the dataset to.\n \"\"\"\n\n test_size: float = field(\n default=0.1,\n metadata={\"help\": \"Fraction of the dataset to include in the test split.\"},\n )\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-internal-testing/harmony\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n\n\ndef main(test_size, push_to_hub, repo_id):\n # fmt: off\n language_modeling_dataset = Dataset.from_dict({\n \"messages\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}, {\"role\": \"assistant\", \"thinking\": \"Beauty improves clarity and appeal.\", \"content\": \"Beautiful.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}, {\"role\": \"assistant\", \"thinking\": \"Clarity avoids confusion.\", \"content\": \"Explicit.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}, {\"role\": \"assistant\", \"thinking\": \"Simplicity is easier to manage.\", \"content\": \"Simple.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}, {\"role\": \"assistant\", \"thinking\": \"Complexity has structure; complication doesnโ€™t.\", \"content\": \"Complex.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}, {\"role\": \"assistant\", \"thinking\": \"Flat structures are easier to read.\", \"content\": \"Flat.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}, {\"role\": \"assistant\", \"thinking\": \"Spacing aids understanding.\", \"content\": \"Sparse.\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}, {\"role\": \"assistant\", \"thinking\": \"Readable code lasts longer.\", \"content\": \"Readability.\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}, {\"role\": \"assistant\", \"thinking\": \"Consistency is more valuable.\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}, {\"role\": \"assistant\", \"thinking\": \"Real-world needs outweigh ideals.\", \"content\": \"Practicality.\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}, {\"role\": \"assistant\", \"thinking\": \"Silent errors cause hidden issues.\", \"content\": \"Errors.\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}, {\"role\": \"assistant\", \"thinking\": \"Silence must be intentional.\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}, {\"role\": \"assistant\", \"thinking\": \"Guessing leads to wrong choices.\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}, {\"role\": \"assistant\", \"thinking\": \"One way reduces confusion.\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}, {\"role\": \"assistant\", \"thinking\": \"A playful nod to Pythonโ€™s creator.\", \"content\": \"Dutch.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}, {\"role\": \"assistant\", \"thinking\": \"Action is better than endless delay.\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}, {\"role\": \"assistant\", \"thinking\": \"Rushed action can be worse.\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}, {\"role\": \"assistant\", \"thinking\": \"Good ideas should be clear.\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}, {\"role\": \"assistant\", \"thinking\": \"Clarity suggests soundness.\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}, {\"role\": \"assistant\", \"thinking\": \"Namespaces prevent conflicts.\", \"content\": \"Namespaces are one honking great idea.\"}]\n ],\n \"chat_template_kwargs\": [\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n ]\n })\n language_modeling_dataset = language_modeling_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n language_modeling_dataset.push_to_hub(repo_id, config_name=\"language_modeling\")\n\n prompt_completion_dataset = Dataset.from_dict({\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n \"completion\": [\n [{\"role\": \"assistant\", \"thinking\": \"Beauty improves clarity and appeal.\", \"content\": \"Beautiful.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Clarity avoids confusion.\", \"content\": \"Explicit.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Simplicity is easier to manage.\", \"content\": \"Simple.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Complexity has structure; complication doesnโ€™t.\", \"content\": \"Complex.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Flat structures are easier to read.\", \"content\": \"Flat.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Spacing aids understanding.\", \"content\": \"Sparse.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Readable code lasts longer.\", \"content\": \"Readability.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Consistency is more valuable.\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Real-world needs outweigh ideals.\", \"content\": \"Practicality.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Silent errors cause hidden issues.\", \"content\": \"Errors.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Silence must be intentional.\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Guessing leads to wrong choices.\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"One way reduces confusion.\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"A playful nod to Pythonโ€™s creator.\", \"content\": \"Dutch.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Action is better than endless delay.\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Rushed action can be worse.\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Good ideas should be clear.\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Clarity suggests soundness.\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Namespaces prevent conflicts.\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n \"chat_template_kwargs\": [\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n ]\n })\n prompt_completion_dataset = prompt_completion_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n prompt_completion_dataset.push_to_hub(repo_id, config_name=\"prompt_completion\")\n\n preference_dataset = Dataset.from_dict({\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n \"chosen\": [\n [{\"role\": \"assistant\", \"thinking\": \"Beauty improves clarity and appeal.\", \"content\": \"Beautiful.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Clarity avoids confusion.\", \"content\": \"Explicit.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Simplicity is easier to manage.\", \"content\": \"Simple.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Complexity has structure; complication doesnโ€™t.\", \"content\": \"Complex.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Flat structures are easier to read.\", \"content\": \"Flat.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Spacing aids understanding.\", \"content\": \"Sparse.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Readable code lasts longer.\", \"content\": \"Readability.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Consistency is more valuable.\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Real-world needs outweigh ideals.\", \"content\": \"Practicality.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Silent errors cause hidden issues.\", \"content\": \"Errors.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Silence must be intentional.\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Guessing leads to wrong choices.\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"One way reduces confusion.\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"A playful nod to Pythonโ€™s creator.\", \"content\": \"Dutch.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Action is better than endless delay.\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Rushed action can be worse.\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Good ideas should be clear.\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Clarity suggests soundness.\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"Namespaces prevent conflicts.\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"assistant\", \"thinking\": \"This comparison is nonsensical.\", \"content\": \"Better than the moon.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This dismisses the value of clarity.\", \"content\": \"Worse than nothing.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This mixes code style with leisure.\", \"content\": \"Better than a long vacation.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This overstates complexity as a universal solution.\", \"content\": \"Always the answer.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This swaps a structural concept for a random object.\", \"content\": \"Better than chocolate.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This ignores the need for context in sparse designs.\", \"content\": \"Without any context.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This implies readability is optional, which it is not.\", \"content\": \"Optional.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This exaggerates special cases into fantasy.\", \"content\": \"Enough to become unicorns.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This twists the original contrast between practicality and purity.\", \"content\": \"Beats reality.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This misapplies \\\"passing\\\" to a literal driving test.\", \"content\": \"Pass their driving test.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This suggests forgetting rather than intentional silence.\", \"content\": \"Forgotten.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This replaces careful judgment with a joke.\", \"content\": \"Refuse the opportunity to laugh.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This encourages multiple confusing approaches instead of one clear way.\", \"content\": \"Two or more confusing methods.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This turns a simple example into time-travel absurdity.\", \"content\": \"A time traveler.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This denies the value of timely action.\", \"content\": \"Never better.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This removes the sense of tradeoff and possibility.\", \"content\": \"Not even a possibility.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This inverts the meaning of explainability.\", \"content\": \"Clearly the best choice.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This treats clarity as something mystical rather than practical.\", \"content\": \"Probably magic.\"}],\n [{\"role\": \"assistant\", \"thinking\": \"This turns a design principle into a silly metaphor.\", \"content\": \"Watermelon -- let's plant some!\"}],\n ],\n \"chat_template_kwargs\": [\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"medium\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"high\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n {\"reasoning_effort\": \"low\", \"model_identity\": \"You are Tiny ChatGPT, a tiny language model.\"},\n ],\n })\n preference_dataset = preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n preference_dataset.push_to_hub(repo_id, config_name=\"preference\")\n # fmt: on\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n main(script_args.test_size, script_args.push_to_hub, script_args.repo_id)\n" }, { "path": "scripts/generate_tiny_models.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# This script generates tiny models used in the TRL library for unit tests. It pushes them to the Hub under the\n# `trl-internal-testing` organization.\n# This script is meant to be run when adding new tiny model to the TRL library.\n\nimport torch\nfrom huggingface_hub import HfApi, ModelCard\nfrom peft import LoraConfig, get_peft_model\nfrom torch import nn\nfrom transformers import (\n AutoConfig,\n AutoProcessor,\n AutoTokenizer,\n BartModel,\n Cohere2Config,\n Cohere2ForCausalLM,\n CohereConfig,\n CohereForCausalLM,\n DeepseekV3Config,\n DeepseekV3ForCausalLM,\n FalconMambaConfig,\n FalconMambaForCausalLM,\n Gemma2Config,\n Gemma2ForCausalLM,\n Gemma3ForConditionalGeneration,\n GemmaConfig,\n GemmaForCausalLM,\n GenerationConfig,\n Glm4MoeConfig,\n Glm4MoeForCausalLM,\n GPT2Config,\n GPT2LMHeadModel,\n GPTNeoXConfig,\n GPTNeoXForCausalLM,\n GPTNeoXForSequenceClassification,\n GptOssConfig,\n GptOssForCausalLM,\n Idefics2Config,\n Idefics2ForConditionalGeneration,\n Idefics3ForConditionalGeneration,\n InternVLForConditionalGeneration,\n LlamaConfig,\n LlamaForCausalLM,\n LlamaForSequenceClassification,\n LlavaForConditionalGeneration,\n LlavaNextForConditionalGeneration,\n MistralConfig,\n MistralForCausalLM,\n OPTConfig,\n OPTForCausalLM,\n PaliGemmaForConditionalGeneration,\n Phi3Config,\n Phi3ForCausalLM,\n Qwen2_5_VLConfig,\n Qwen2_5_VLForConditionalGeneration,\n Qwen2Config,\n Qwen2ForCausalLM,\n Qwen2ForSequenceClassification,\n Qwen2VLConfig,\n Qwen2VLForConditionalGeneration,\n Qwen3_5Config,\n Qwen3_5ForConditionalGeneration,\n Qwen3Config,\n Qwen3ForCausalLM,\n Qwen3ForSequenceClassification,\n Qwen3MoeConfig,\n Qwen3MoeForCausalLM,\n Qwen3MoeForSequenceClassification,\n Qwen3VLConfig,\n Qwen3VLForConditionalGeneration,\n SmolVLMForConditionalGeneration,\n T5ForConditionalGeneration,\n)\n\n\nORGANIZATION = \"trl-internal-testing\"\n\nMODEL_CARD = \"\"\"\n---\nlibrary_name: transformers\ntags: [trl]\n---\n\n# Tiny {model_class_name}\n\nThis is a minimal model built for unit tests in the [TRL](https://github.com/huggingface/trl) library.\n\"\"\"\n\n\napi = HfApi()\n\n\ndef push_to_hub(model, tokenizer, generation_config, prefix=None, suffix=None, force=False):\n model_class_name = model.__class__.__name__\n content = MODEL_CARD.format(model_class_name=model_class_name)\n model_card = ModelCard(content)\n if prefix is not None:\n model_class_name = f\"{prefix}-{model_class_name}\"\n repo_id = f\"{ORGANIZATION}/{model_class_name}\"\n if suffix is not None:\n repo_id += f\"-{suffix}\"\n\n if api.repo_exists(repo_id) and not force:\n print(f\"Model {repo_id} already exists, skipping\")\n else:\n model.push_to_hub(repo_id)\n model_card.push_to_hub(repo_id)\n if tokenizer is not None:\n tokenizer.push_to_hub(repo_id)\n if generation_config is not None:\n generation_config.push_to_hub(repo_id)\n\n\ndef init_weights_tiny_model(model):\n \"\"\"\n Initialize tiny test models to avoid NaNs from uninitialized weights.\n\n Uses safe defaults:\n - Linear/Conv1d: Xavier uniform (weights), zero (biases)\n - Embedding: Normal(0, 0.02)\n - LayerNorm: Ones (weights), zero (biases)\n\n Args:\n model: PyTorch model (modified in-place)\n \"\"\"\n for module in model.modules():\n if isinstance(module, nn.Linear):\n # Attention/MLP projections โ†’ Xavier or Normal\n if module.bias is not None:\n nn.init.zeros_(module.bias)\n nn.init.xavier_uniform_(module.weight)\n\n elif isinstance(module, nn.Embedding):\n # Token embeddings โ†’ GPT-style Normal\n nn.init.normal_(module.weight, mean=0.0, std=0.02)\n\n elif isinstance(module, nn.LayerNorm):\n # LayerNorm weights always 1, bias 0\n nn.init.ones_(module.weight)\n if module.bias is not None:\n nn.init.zeros_(module.bias)\n\n elif isinstance(module, nn.Conv1d):\n # Convolutional layers โ†’ Xavier or Normal\n if module.bias is not None:\n nn.init.zeros_(module.bias)\n nn.init.xavier_uniform_(module.weight)\n\n\n# Decoder models\nfor model_id, config_class, model_class, dtype, suffix in [\n # (\"bigscience/bloomz-560m\", BloomConfig, BloomForCausalLM, None), # loading fails with this model, see https://huggingface.co/bigscience/bloomz-560m/discussions/14\n (\"CohereLabs/aya-expanse-8b\", CohereConfig, CohereForCausalLM, torch.float16, None),\n (\"CohereLabs/tiny-aya-earth\", Cohere2Config, Cohere2ForCausalLM, torch.bfloat16, None),\n (\"deepseek-ai/DeepSeek-R1\", DeepseekV3Config, DeepseekV3ForCausalLM, torch.bfloat16, None),\n # It's important to have R1-0528 as it doesn't have the same chat template\n (\"deepseek-ai/DeepSeek-R1-0528\", DeepseekV3Config, DeepseekV3ForCausalLM, torch.bfloat16, \"0528\"),\n (\"tiiuae/falcon-7b-instruct\", FalconMambaConfig, FalconMambaForCausalLM, torch.bfloat16, None),\n (\"google/gemma-2-2b-it\", Gemma2Config, Gemma2ForCausalLM, torch.bfloat16, None),\n (\"google/gemma-7b-it\", GemmaConfig, GemmaForCausalLM, torch.bfloat16, None),\n (\"openai-community/gpt2\", GPT2Config, GPT2LMHeadModel, torch.float32, None),\n (\"EleutherAI/pythia-14m\", GPTNeoXConfig, GPTNeoXForCausalLM, torch.float16, None),\n (\"meta-llama/Meta-Llama-3-8B-Instruct\", LlamaConfig, LlamaForCausalLM, torch.bfloat16, \"3\"),\n (\"meta-llama/Llama-3.1-8B-Instruct\", LlamaConfig, LlamaForCausalLM, torch.bfloat16, \"3.1\"),\n (\"meta-llama/Llama-3.2-1B-Instruct\", LlamaConfig, LlamaForCausalLM, torch.bfloat16, \"3.2\"),\n (\"mistralai/Mistral-7B-Instruct-v0.1\", MistralConfig, MistralForCausalLM, torch.bfloat16, \"0.1\"),\n (\"mistralai/Mistral-7B-Instruct-v0.2\", MistralConfig, MistralForCausalLM, torch.bfloat16, \"0.2\"),\n (\"facebook/opt-1.3b\", OPTConfig, OPTForCausalLM, torch.float16, None),\n (\"microsoft/Phi-3.5-mini-instruct\", Phi3Config, Phi3ForCausalLM, torch.bfloat16, None),\n (\"Qwen/Qwen2.5-32B-Instruct\", Qwen2Config, Qwen2ForCausalLM, torch.bfloat16, \"2.5\"),\n (\"Qwen/Qwen2.5-Coder-0.5B\", Qwen2Config, Qwen2ForCausalLM, torch.bfloat16, \"2.5-Coder\"),\n (\"Qwen/Qwen3-8B\", Qwen3Config, Qwen3ForCausalLM, torch.bfloat16, None),\n]:\n revision = \"refs/pr/14\" if model_id == \"Qwen/Qwen3-8B\" else \"main\" # chat template with {% generation %}\n tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)\n generation_config = GenerationConfig.from_pretrained(model_id, revision=revision)\n config = config_class(\n vocab_size=len(tokenizer.vocab),\n hidden_size=8,\n num_attention_heads=4,\n num_key_value_heads=2,\n num_hidden_layers=2,\n intermediate_size=32,\n )\n model = model_class(config).to(dtype=dtype)\n init_weights_tiny_model(model)\n push_to_hub(model, tokenizer, generation_config, \"tiny\", suffix)\n\n# MoE models\nfor model_id, config_class, model_class, dtype, suffix in [\n (\"Qwen/Qwen3-30B-A3B\", Qwen3MoeConfig, Qwen3MoeForCausalLM, torch.bfloat16, None),\n (\"openai/gpt-oss-20b\", GptOssConfig, GptOssForCausalLM, torch.bfloat16, None),\n (\"zai-org/GLM-4.5\", Glm4MoeConfig, Glm4MoeForCausalLM, torch.bfloat16, None),\n]:\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n generation_config = GenerationConfig.from_pretrained(model_id)\n kwargs = {}\n if model_id == \"zai-org/GLM-4.5\":\n kwargs[\"n_routed_experts\"] = 4\n elif model_id in (\"Qwen/Qwen3-30B-A3B\", \"openai/gpt-oss-20b\"):\n kwargs[\"num_experts\"] = 4\n\n config = config_class(\n vocab_size=len(tokenizer.vocab),\n hidden_size=8,\n num_attention_heads=4,\n num_key_value_heads=2,\n num_hidden_layers=2,\n intermediate_size=32,\n num_experts_per_tok=2,\n **kwargs,\n )\n model = model_class(config).to(dtype=dtype)\n init_weights_tiny_model(model)\n push_to_hub(model, tokenizer, generation_config, \"tiny\", suffix)\n\n# Two slightly bigger models, required for vLLM testing\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-32B-Instruct\")\ngeneration_config = GenerationConfig.from_pretrained(\"Qwen/Qwen2.5-32B-Instruct\")\nconfig = Qwen2Config(\n vocab_size=len(tokenizer.vocab),\n hidden_size=128, # increase hidden size so that hidden_size // num_attention_heads = 32, required for vLLM\n num_attention_heads=4,\n num_key_value_heads=2,\n num_hidden_layers=2,\n intermediate_size=32,\n)\nmodel = Qwen2ForCausalLM(config).to(dtype=torch.bfloat16)\npush_to_hub(model, tokenizer, generation_config, \"small\", \"2.5\")\n\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen3-4B\")\ngeneration_config = GenerationConfig.from_pretrained(\"Qwen/Qwen3-4B\")\nconfig = Qwen3Config(\n vocab_size=len(tokenizer.vocab),\n hidden_size=128, # increase hidden size so that hidden_size // num_attention_heads = 32, required for vLLM\n num_attention_heads=4,\n num_key_value_heads=2,\n num_hidden_layers=2,\n intermediate_size=32,\n)\nmodel = Qwen3ForCausalLM(config).to(dtype=torch.bfloat16)\npush_to_hub(model, tokenizer, generation_config, \"small\")\n\n# Reward models\nfor model_id, model_class, dtype, suffix in [\n (\"EleutherAI/pythia-14m\", GPTNeoXForSequenceClassification, torch.bfloat16, None),\n (\"meta-llama/Llama-3.2-1B-Instruct\", LlamaForSequenceClassification, torch.bfloat16, \"3.2\"),\n (\"Qwen/Qwen2.5-32B-Instruct\", Qwen2ForSequenceClassification, torch.bfloat16, \"2.5\"),\n (\"Qwen/Qwen3-4B\", Qwen3ForSequenceClassification, torch.bfloat16, None),\n]:\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n generation_config = GenerationConfig.from_pretrained(model_id)\n kwargs = {\n \"num_labels\": 1,\n \"hidden_size\": 16,\n \"num_attention_heads\": 4,\n \"num_key_value_heads\": 2,\n \"num_hidden_layers\": 2,\n \"intermediate_size\": 32,\n }\n config = AutoConfig.from_pretrained(model_id, **kwargs)\n # Bug in transformers: it ignores num_hidden_layers to build layer_types\n if model_id in (\"Qwen/Qwen2.5-32B-Instruct\", \"Qwen/Qwen3-4B\"):\n config.layer_types = config.layer_types[:2]\n model = model_class(config).to(dtype=dtype)\n init_weights_tiny_model(model)\n push_to_hub(model, tokenizer, generation_config, \"tiny\", suffix)\n\n# MoE Reward models\nfor model_id, model_class, dtype, suffix in [\n (\"Qwen/Qwen3-30B-A3B\", Qwen3MoeForSequenceClassification, torch.bfloat16, None),\n]:\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n generation_config = GenerationConfig.from_pretrained(model_id)\n kwargs = {\n \"num_labels\": 1,\n \"hidden_size\": 16,\n \"num_attention_heads\": 4,\n \"num_key_value_heads\": 2,\n \"num_hidden_layers\": 2,\n \"intermediate_size\": 32,\n \"num_experts\": 4,\n \"num_experts_per_tok\": 2,\n }\n config = AutoConfig.from_pretrained(model_id, **kwargs)\n model = model_class(config).to(dtype=dtype)\n push_to_hub(model, tokenizer, generation_config, \"tiny\", suffix)\n\n\n# Encoder-decoder models\nfor model_id, model_class, dtype, suffix in [\n (\"facebook/bart-base\", BartModel, torch.float32, None),\n (\"google/flan-t5-small\", T5ForConditionalGeneration, torch.float32, None),\n]:\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n generation_config = GenerationConfig.from_pretrained(model_id) if model_id != \"facebook/bart-base\" else None\n config = AutoConfig.from_pretrained(model_id)\n config.d_model = 24\n model = model_class(config).to(dtype=dtype)\n push_to_hub(model, tokenizer, generation_config, \"tiny\", suffix)\n\n\n# Vision Language Models\nfor model_id, model_class, dtype in [\n (\"google/gemma-3-4b-it\", Gemma3ForConditionalGeneration, torch.bfloat16),\n (\"google/paligemma-3b-pt-224\", PaliGemmaForConditionalGeneration, torch.float32),\n (\"HuggingFaceM4/idefics2-8b\", Idefics2ForConditionalGeneration, torch.float32),\n (\"HuggingFaceM4/Idefics3-8B-Llama3\", Idefics3ForConditionalGeneration, torch.bfloat16),\n (\"HuggingFaceTB/SmolVLM2-2.2B-Instruct\", SmolVLMForConditionalGeneration, torch.float32),\n (\"llava-hf/llava-1.5-7b-hf\", LlavaForConditionalGeneration, torch.float16),\n # Original model dtype is float16, but it triggers CUDA device side assert error (see GH-4741):\n (\"llava-hf/llava-v1.6-mistral-7b-hf\", LlavaNextForConditionalGeneration, torch.bfloat16),\n (\"OpenGVLab/InternVL3-8B-hf\", InternVLForConditionalGeneration, torch.bfloat16),\n (\"Qwen/Qwen2-VL-2B-Instruct\", Qwen2VLForConditionalGeneration, torch.bfloat16),\n (\"Qwen/Qwen2.5-VL-3B-Instruct\", Qwen2_5_VLForConditionalGeneration, torch.bfloat16),\n (\"Qwen/Qwen3-VL-2B-Instruct\", Qwen3VLForConditionalGeneration, torch.bfloat16),\n (\"Qwen/Qwen3.5-0.8B\", Qwen3_5ForConditionalGeneration, torch.bfloat16),\n]:\n processor = AutoProcessor.from_pretrained(model_id)\n generation_config = GenerationConfig.from_pretrained(model_id) if model_id != \"Qwen/Qwen3.5-0.8B\" else None\n\n text_config = {\n \"num_hidden_layers\": 2,\n \"hidden_size\": 16,\n \"num_attention_heads\": 4,\n \"num_key_value_heads\": 2,\n \"layer_types\": None, # Set it automatically from num_hidden_layers\n }\n vision_config = {\n \"num_hidden_layers\": 2,\n \"hidden_size\": 16,\n \"num_attention_heads\": 4,\n \"num_key_value_heads\": 2,\n \"embed_dim\": 64,\n }\n kwargs = {}\n\n if issubclass(model_class.config_class, (Qwen2VLConfig, Qwen2_5_VLConfig)):\n text_config[\"rope_scaling\"] = {\"type\": \"default\", \"mrope_section\": [1, 1], \"rope_type\": \"default\"}\n vision_config[\"depth\"] = 2\n # Different dict object from text_config; see GH-4101 and transformers#41020\n kwargs[\"rope_scaling\"] = {\"type\": \"default\", \"mrope_section\": [1, 1], \"rope_type\": \"default\"}\n\n if issubclass(model_class.config_class, Qwen2_5_VLConfig):\n vision_config[\"out_hidden_size\"] = 16\n # Different dict object at the config root; see GH-4101 and transformers#41020\n kwargs[\"num_hidden_layers\"] = 2\n kwargs[\"hidden_size\"] = 16\n kwargs[\"num_attention_heads\"] = 4\n\n if issubclass(model_class.config_class, Idefics2Config):\n kwargs[\"perceiver_config\"] = {\"hidden_size\": 16}\n\n if issubclass(model_class.config_class, Qwen3VLConfig):\n # So hasattr(config, \"layer_types\") is False\n # See: https://github.com/huggingface/transformers/blob/fe5ca9ddaa07fac2872407e75c7a7661216ac956/src/transformers/models/qwen3_vl/modeling_qwen3_vl.py#L420\n del text_config[\"layer_types\"]\n # \"mrope_section\" needs 3 elements: for dim, offset in enumerate((1, 2), start=1): mrope_section[dim]\n # See: https://github.com/huggingface/transformers/blob/fe5ca9ddaa07fac2872407e75c7a7661216ac956/src/transformers/models/qwen3_vl/modeling_qwen3_vl.py#L361\n text_config[\"rope_scaling\"] = {\"mrope_interleaved\": True, \"mrope_section\": [2, 2, 2], \"rope_type\": \"default\"}\n vision_config[\"depth\"] = 2\n vision_config[\"out_hidden_size\"] = 16\n\n if issubclass(model_class.config_class, Qwen3_5Config):\n # For tiny layer counts, default `layer_types` can end up with no full-attention layers (e.g. 2 layers and\n # default interval 4), which breaks Qwen3.5 dynamic cache logic. Keep one full-attention layer at the end.\n text_config[\"layer_types\"] = [\"linear_attention\", \"full_attention\"]\n text_config[\"full_attention_interval\"] = 2\n # Qwen3.5-VL vision config expects `depth`/`num_heads`, not `num_hidden_layers`/`num_attention_heads`.\n vision_config.pop(\"num_hidden_layers\", None)\n vision_config.pop(\"num_attention_heads\", None)\n vision_config.pop(\"num_key_value_heads\", None)\n vision_config.pop(\"embed_dim\", None)\n vision_config[\"depth\"] = 2\n vision_config[\"num_heads\"] = 4\n vision_config[\"intermediate_size\"] = 32\n vision_config[\"out_hidden_size\"] = 16\n\n if model_id == \"llava-hf/llava-v1.6-mistral-7b-hf\":\n # Hotfix: llava-hf/llava-v1.6-mistral-7b-hf mistakesly sets text_config.dtype to \"bfloat16\".\n # See https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf/discussions/46\n text_config[\"dtype\"] = None\n\n config = AutoConfig.from_pretrained(model_id, text_config=text_config, vision_config=vision_config, **kwargs)\n model = model_class(config).to(dtype=dtype)\n\n if issubclass(model_class.config_class, Qwen3_5Config):\n # Qwen3.5 models has some weights in float32, to mirror this in the tiny model we need to convert them to float32 manually.\n for layer in model.model.language_model.layers:\n if hasattr(layer, \"linear_attn\"): # applies to linear attention layers only\n layer.linear_attn.A_log.data = layer.linear_attn.A_log.data.float()\n layer.linear_attn.norm.weight.data = layer.linear_attn.norm.weight.data.float()\n\n push_to_hub(model, processor, generation_config, \"tiny\")\n\n# PEFT models\nmodel = Qwen3ForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen3ForCausalLM\", dtype=\"auto\")\nmodel = get_peft_model(model, LoraConfig())\ngeneration_config = GenerationConfig.from_pretrained(\"trl-internal-testing/tiny-Qwen3ForCausalLM\")\npush_to_hub(model, None, None, \"tiny\")\n\n# Same model, but different weights\nmodel = Qwen3ForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen3ForCausalLM\", dtype=\"auto\")\nmodel = get_peft_model(model, LoraConfig())\ngeneration_config = GenerationConfig.from_pretrained(\"trl-internal-testing/tiny-Qwen3ForCausalLM\")\npush_to_hub(model, None, None, \"tiny\", \"2\")\n" }, { "path": "scripts/generate_toolcall_dataset.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport json\nfrom dataclasses import dataclass, field\n\nfrom datasets import Dataset\nfrom transformers import HfArgumentParser\nfrom transformers.utils import get_json_schema\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n test_size (`float`, *optional*, defaults to `0.1`):\n Fraction of the dataset to include in the test split.\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-internal-testing/zen\"`):\n Hugging Face repository ID to push the dataset to.\n \"\"\"\n\n test_size: float = field(\n default=0.1,\n metadata={\"help\": \"Fraction of the dataset to include in the test split.\"},\n )\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-internal-testing/toolcall\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n\n\ndef main(test_size, push_to_hub, repo_id):\n # Fictitious functions to simulate tool calls\n def start_timer(duration: int) -> int:\n \"\"\"\n Starts a timer for the specified duration in seconds.\n\n Args:\n duration: Duration in seconds to set the timer for.\n\n Returns:\n The duration set for the timer.\n \"\"\"\n return duration\n\n def get_current_time(location: str) -> str:\n \"\"\"\n Returns the current time in the specified location.\n\n Args:\n location: The location for which to get the current time.\n\n Returns:\n The current time in the specified location.\n \"\"\"\n return \"06:22:48\"\n\n def get_air_quality_index(location: str) -> int:\n \"\"\"\n Returns the air quality index for the specified location.\n\n Args:\n location: The location for which to get the air quality index.\n\n Returns:\n The air quality index for the specified location.\n \"\"\"\n return 53\n\n def play_music(title: str, artist: str) -> dict:\n \"\"\"\n Plays music by the specified title and artist.\n\n Args:\n title: The title of the music to play.\n artist: The artist of the music to play.\n\n Returns:\n A dictionary indicating the status of the music playback.\n \"\"\"\n return {\"status\": \"Playing\"}\n\n def get_weather_forecast(city: str, date: str) -> dict:\n \"\"\"\n Returns the weather forecast for the specified city and date.\n\n Args:\n city: The city for which to get the weather forecast.\n date: The date for which to get the weather forecast.\n\n Returns:\n A dictionary containing the temperature and weather condition.\n \"\"\"\n return {\"temperature\": 22, \"condition\": \"partly cloudy\"}\n\n def control_light(room: str, state: str) -> dict:\n \"\"\"\n Controls the light in the specified room.\n\n Args:\n room: The room where the light should be controlled.\n state: The desired state of the light (\"on\" or \"off\").\n\n Returns:\n A dictionary indicating the state of the light.\n \"\"\"\n return {\"state\": state}\n\n def create_reminder(time: str, note: str) -> str:\n \"\"\"\n Creates a reminder for the specified time and note.\n\n Args:\n time: The time for the reminder.\n note: The note for the reminder.\n\n Returns:\n A confirmation message indicating that the reminder has been set.\n \"\"\"\n return \"I'll remind you to call mom at 7 PM.\"\n\n def get_wind_conditions(city: str, unit: str) -> tuple[int, str]:\n \"\"\"\n Returns the wind conditions for the specified city.\n\n Args:\n city: The city for which to get the wind conditions.\n unit: The unit of measurement for the wind speed (e.g., \"mph\").\n\n Returns:\n A tuple containing the wind speed and direction.\n \"\"\"\n return 14, \"NW\"\n\n start_timer = get_json_schema(start_timer)\n get_current_time = get_json_schema(get_current_time)\n get_air_quality_index = get_json_schema(get_air_quality_index)\n play_music = get_json_schema(play_music)\n get_weather_forecast = get_json_schema(get_weather_forecast)\n control_light = get_json_schema(control_light)\n create_reminder = get_json_schema(create_reminder)\n get_wind_conditions = get_json_schema(get_wind_conditions)\n\n # fmt: off\n language_modeling_dataset = Dataset.from_dict({\n \"messages\": [\n [\n {\"role\": \"user\", \"content\": \"Set a timer for 10 minutes.\"},\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"start_timer\", \"arguments\": {\"duration\": 600}}}]},\n {\"role\": \"tool\", \"name\": \"start_timer\", \"content\": \"600\"},\n {\"role\": \"assistant\", \"content\": \"Timer set for 10 minutes.\"},\n ],\n [\n {\"role\": \"user\", \"content\": \"What time is it in Tokyo?\"},\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_current_time\", \"arguments\": {\"location\": \"Tokyo\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_current_time\", \"content\": \"06:22:48\"},\n {\"role\": \"assistant\", \"content\": \"The current time in Tokyo is 06:22 AM.\"},\n ],\n [\n {\"role\": \"user\", \"content\": \"Is the air clean today in Lisbon?\"},\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_air_quality_index\", \"arguments\": {\"location\": \"Lisbon, Portugal\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_air_quality_index\", \"content\": \"53\"},\n {\"role\": \"assistant\", \"content\": \"The air quality is moderate.\"},\n ],\n [\n {\"role\": \"user\", \"content\": \"Play some music.\"},\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"play_music\", \"arguments\": {\"title\": \"Take Five\", \"artist\": \"Dave Brubeck\"}}}]},\n {\"role\": \"tool\", \"name\": \"play_music\", \"content\": \"{'status': 'Playing'}\"},\n {\"role\": \"assistant\", \"content\": \"Enjoy the jazz tunes!\"},\n ],\n [\n {\"role\": \"user\", \"content\": \"What's the weather like tomorrow in Berlin?\"},\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_weather_forecast\", \"arguments\": {\"city\": \"Berlin\", \"date\": \"2025-06-16\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_weather_forecast\", \"content\": \"{'temperature': 22, 'condition': 'partly cloudy'}\"},\n {\"role\": \"assistant\", \"content\": \"Tomorrow in Berlin will be partly cloudy with a high of 22ยฐC.\"}\n ],\n [\n {\"role\": \"user\", \"content\": \"Turn on the living room lights.\"},\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"control_light\", \"arguments\": {\"room\": \"living room\", \"state\": \"on\"}}}]},\n {\"role\": \"tool\", \"name\": \"control_light\", \"content\": \"{'state': 'on'}\"},\n {\"role\": \"assistant\", \"content\": \"The living room lights are now on.\"}\n ],\n [\n {\"role\": \"user\", \"content\": \"Remind me to call mom at 7 PM.\"},\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"create_reminder\", \"arguments\": {\"time\": \"19:00\", \"note\": \"Call mom\"}}}]},\n {\"role\": \"tool\", \"name\": \"create_reminder\", \"content\": \"Reminder set\"},\n {\"role\": \"assistant\", \"content\": \"Okay, I'll remind you to call mom at 7 PM.\"}\n ],\n [\n {\"role\": \"user\", \"content\": \"How strong is the wind in Chicago right now?\"},\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_wind_conditions\", \"arguments\": {\"city\": \"Chicago\", \"unit\": \"mph\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_wind_conditions\", \"content\": \"(14, 'NW')\"},\n {\"role\": \"assistant\", \"content\": \"The wind in Chicago is blowing at 14 mph from the northwest.\"}\n ]\n ],\n \"tools\": [\n json.dumps([start_timer, create_reminder]),\n json.dumps([get_current_time]),\n json.dumps([get_air_quality_index, get_weather_forecast, get_wind_conditions]),\n json.dumps([play_music, control_light]),\n json.dumps([get_weather_forecast, get_wind_conditions]),\n json.dumps([control_light]),\n json.dumps([start_timer, create_reminder]),\n json.dumps([get_weather_forecast, get_wind_conditions]),\n ]\n })\n language_modeling_dataset = language_modeling_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n language_modeling_dataset.push_to_hub(repo_id, config_name=\"language_modeling\")\n\n preference_dataset = Dataset.from_dict({\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"Set a timer for 10 minutes.\"}],\n [{\"role\": \"user\", \"content\": \"What time is it in Tokyo?\"}],\n [{\"role\": \"user\", \"content\": \"Is the air clean today in Lisbon?\"}],\n [{\"role\": \"user\", \"content\": \"Play some music.\"}],\n [{\"role\": \"user\", \"content\": \"What's the weather like tomorrow in Berlin?\"}],\n [{\"role\": \"user\", \"content\": \"Turn on the living room lights.\"}],\n [{\"role\": \"user\", \"content\": \"Remind me to call mom at 7 PM.\"}],\n [{\"role\": \"user\", \"content\": \"How strong is the wind in Chicago right now?\"}],\n ],\n \"chosen\": [\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"start_timer\", \"arguments\": {\"duration\": 600}}}]},\n {\"role\": \"tool\", \"name\": \"start_timer\", \"content\": \"600\"},\n {\"role\": \"assistant\", \"content\": \"Timer set for 10 minutes.\"},\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_current_time\", \"arguments\": {\"location\": \"Tokyo\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_current_time\", \"content\": \"06:22:48\"},\n {\"role\": \"assistant\", \"content\": \"The current time in Tokyo is 06:22 AM.\"},\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_air_quality_index\", \"arguments\": {\"location\": \"Lisbon, Portugal\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_air_quality_index\", \"content\": \"53\"},\n {\"role\": \"assistant\", \"content\": \"The air quality is moderate.\"},\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"play_music\", \"arguments\": {\"title\": \"Take Five\", \"artist\": \"Dave Brubeck\"}}}]},\n {\"role\": \"tool\", \"name\": \"play_music\", \"content\": \"{'status': 'Playing'}\"},\n {\"role\": \"assistant\", \"content\": \"Enjoy the jazz tunes!\"},\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_weather_forecast\", \"arguments\": {\"city\": \"Berlin\", \"date\": \"2025-06-16\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_weather_forecast\", \"content\": \"{'temperature': 22, 'condition': 'partly cloudy'}\"},\n {\"role\": \"assistant\", \"content\": \"Tomorrow in Berlin will be partly cloudy with a high of 22ยฐC.\"}\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"control_light\", \"arguments\": {\"room\": \"living room\", \"state\": \"on\"}}}]},\n {\"role\": \"tool\", \"name\": \"control_light\", \"content\": \"{'state': 'on'}\"},\n {\"role\": \"assistant\", \"content\": \"The living room lights are now on.\"}\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"create_reminder\", \"arguments\": {\"time\": \"19:00\", \"note\": \"Call mom\"}}}]},\n {\"role\": \"tool\", \"name\": \"create_reminder\", \"content\": \"Reminder set\"},\n {\"role\": \"assistant\", \"content\": \"Okay, Iโ€™ll remind you to call mom at 7 PM.\"}\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_wind_conditions\", \"arguments\": {\"city\": \"Chicago\", \"unit\": \"mph\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_wind_conditions\", \"content\": \"(14, 'NW')\"},\n {\"role\": \"assistant\", \"content\": \"The wind in Chicago is blowing at 14 mph from the northwest.\"}\n ],\n ],\n \"rejected\": [\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"start_timer\", \"arguments\": {\"duration\": 10}}}]},\n {\"role\": \"tool\", \"name\": \"start_timer\", \"content\": \"10\"},\n {\"role\": \"assistant\", \"content\": \"Timer set for 10 seconds.\"},\n ],\n [\n {\"role\": \"assistant\", \"content\": \"It is 6:22 AM in Tokyo.\"},\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_air_quality_index\", \"arguments\": {\"location\": \"Lisbon\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_air_quality_index\", \"content\": \"53\"},\n {\"role\": \"assistant\", \"content\": \"The air quality is great.\"},\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"play_music\", \"arguments\": {\"title\": \"Take Five\", \"artist\": \"Daft Punk\"}}}]},\n {\"role\": \"tool\", \"name\": \"play_music\", \"content\": \"{'status': 'Playing'}\"},\n {\"role\": \"assistant\", \"content\": \"Playing your song.\"},\n ],\n [\n {\"role\": \"assistant\", \"content\": \"Tomorrow in Berlin will be hot and sunny.\"},\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"control_light\", \"arguments\": {\"room\": \"living room\", \"state\": \"off\"}}}]},\n {\"role\": \"tool\", \"name\": \"control_light\", \"content\": \"{'state': 'off'}\"},\n {\"role\": \"assistant\", \"content\": \"The living room lights are now off.\"}\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"create_reminder\", \"arguments\": {\"time\": \"07:00\", \"note\": \"Call mom\"}}}]},\n {\"role\": \"tool\", \"name\": \"create_reminder\", \"content\": \"Reminder set\"},\n {\"role\": \"assistant\", \"content\": \"Okay, I'll remind you to call mom at 7 AM.\"}\n ],\n [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_weather_forecast\", \"arguments\": {\"city\": \"Chicago\", \"date\": \"2025-06-16\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_weather_forecast\", \"content\": \"{'temperature': 22, 'condition': 'partly cloudy'}\"},\n {\"role\": \"assistant\", \"content\": \"Tomorrow in Chicago will be partly cloudy with a high of 22ยฐC.\"}\n ],\n ],\n \"tools\": [\n json.dumps([start_timer]),\n json.dumps([get_current_time]),\n json.dumps([get_air_quality_index]),\n json.dumps([play_music]),\n json.dumps([get_weather_forecast]),\n json.dumps([control_light]),\n json.dumps([create_reminder]),\n json.dumps([get_wind_conditions]),\n ],\n })\n preference_dataset = preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n preference_dataset.push_to_hub(repo_id, config_name=\"preference\")\n # fmt: on\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n main(script_args.test_size, script_args.push_to_hub, script_args.repo_id)\n" }, { "path": "scripts/generate_zen_dataset.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom datasets import Dataset\nfrom transformers import HfArgumentParser\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n test_size (`float`, *optional*, defaults to `0.1`):\n Fraction of the dataset to include in the test split.\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-internal-testing/zen\"`):\n Hugging Face repository ID to push the dataset to.\n \"\"\"\n\n test_size: float = field(\n default=0.1,\n metadata={\"help\": \"Fraction of the dataset to include in the test split.\"},\n )\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-internal-testing/zen\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n\n\ndef main(test_size, push_to_hub, repo_id):\n # fmt: off\n standard_language_modeling_dataset = Dataset.from_dict({\n \"text\": [\n \"Beautiful is better than ugly.\",\n \"Explicit is better than implicit.\",\n \"Simple is better than complex.\",\n \"Complex is better than complicated.\",\n \"Flat is better than nested.\",\n \"Sparse is better than dense.\",\n \"Readability counts.\",\n \"Special cases aren't special enough to break the rules.\",\n \"Although practicality beats purity.\",\n \"Errors should never pass silently.\",\n \"Unless explicitly silenced.\",\n \"In the face of ambiguity, refuse the temptation to guess.\",\n \"There should be one-- and preferably only one --obvious way to do it.\",\n \"Although that way may not be obvious at first unless you're Dutch.\",\n \"Now is better than never.\",\n \"Although never is often better than *right* now.\",\n \"If the implementation is hard to explain, it's a bad idea.\",\n \"If the implementation is easy to explain, it may be a good idea.\",\n \"Namespaces are one honking great idea -- let's do more of those!\",\n ],\n })\n standard_language_modeling_dataset = standard_language_modeling_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n standard_language_modeling_dataset.push_to_hub(repo_id, config_name=\"standard_language_modeling\")\n\n standard_prompt_only_dataset = Dataset.from_dict({\n \"prompt\": [\n \"Beautiful is better than\",\n \"Explicit is\",\n \"Simple is better\",\n \"Complex\",\n \"Flat is better than\",\n \"Sparse is better\",\n \"Readability\",\n \"Special cases aren't special\",\n \"Although practicality beats\",\n \"Errors should never\",\n \"Unless explicitly\",\n \"In the face of ambiguity, refuse\",\n \"There should be one-- and preferably\",\n \"Although that way may not be obvious at first unless you're\",\n \"Now is\",\n \"Although never is often\",\n \"If the implementation is hard to explain,\",\n \"If the implementation is easy\",\n \"Namespaces are one honking great\",\n ],\n })\n standard_prompt_only_dataset = standard_prompt_only_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n standard_prompt_only_dataset.push_to_hub(repo_id, config_name=\"standard_prompt_only\")\n\n standard_prompt_completion_dataset = Dataset.from_dict({\n \"prompt\": [\n \"Beautiful is better than\",\n \"Explicit is\",\n \"Simple is better\",\n \"Complex\",\n \"Flat is better than\",\n \"Sparse is better\",\n \"Readability\",\n \"Special cases aren't special\",\n \"Although practicality beats\",\n \"Errors should never\",\n \"Unless explicitly\",\n \"In the face of ambiguity, refuse\",\n \"There should be one-- and preferably\",\n \"Although that way may not be obvious at first unless you're\",\n \"Now is\",\n \"Although never is often\",\n \"If the implementation is hard to explain,\",\n \"If the implementation is easy\",\n \"Namespaces are one honking great\",\n ],\n \"completion\": [\n \" ugly.\",\n \" better than implicit.\",\n \" than complex.\",\n \" is better than complicated.\",\n \" nested.\",\n \" than dense.\",\n \" counts.\",\n \" enough to break the rules.\",\n \" purity.\",\n \" pass silently.\",\n \" silenced.\",\n \" the temptation to guess.\",\n \" only one --obvious way to do it.\",\n \" Dutch.\",\n \" better than never.\",\n \" better than *right* now.\",\n \" it's a bad idea.\",\n \" to explain, it may be a good idea.\",\n \" idea -- let's do more of those!\",\n ],\n })\n standard_prompt_completion_dataset = standard_prompt_completion_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n standard_prompt_completion_dataset.push_to_hub(repo_id, config_name=\"standard_prompt_completion\")\n\n standard_preference_dataset = Dataset.from_dict({\n \"prompt\": [\n \"Beautiful is better than\",\n \"Explicit is\",\n \"Simple is better\",\n \"Complex\",\n \"Flat is better than\",\n \"Sparse is better\",\n \"Readability\",\n \"Special cases aren't special\",\n \"Although practicality beats\",\n \"Errors should never\",\n \"Unless explicitly\",\n \"In the face of ambiguity, refuse\",\n \"There should be one-- and preferably\",\n \"Although that way may not be obvious at first unless you're\",\n \"Now is\",\n \"Although never is often\",\n \"If the implementation is hard to explain,\",\n \"If the implementation is easy\",\n \"Namespaces are one honking great\",\n ],\n \"chosen\": [\n \" ugly.\",\n \" better than implicit.\",\n \" than complex.\",\n \" is better than complicated.\",\n \" nested.\",\n \" than dense.\",\n \" counts.\",\n \" enough to break the rules.\",\n \" purity.\",\n \" pass silently.\",\n \" silenced.\",\n \" the temptation to guess.\",\n \" only one --obvious way to do it.\",\n \" Dutch.\",\n \" better than never.\",\n \" better than *right* now.\",\n \" it's a bad idea.\",\n \" to explain, it may be a good idea.\",\n \" idea -- let's do more of those!\",\n ],\n \"rejected\": [\n \" the moon.\",\n \" worse than nothing.\",\n \" than a long vacation.\",\n \" is always the answer.\",\n \" chocolate.\",\n \" without any context.\",\n \" is optional.\",\n \" enough to become unicorns.\",\n \" reality.\",\n \" pass their driving test.\",\n \" forgotten.\",\n \" the opportunity to laugh.\",\n \" two or more confusing methods.\",\n \" a time traveler.\",\n \" never better.\",\n \" not even a possibility.\",\n \" it's clearly the best choice.\",\n \" it's probably magic.\",\n \" watermelon -- let's plant some!\",\n ],\n })\n standard_preference_dataset = standard_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n standard_preference_dataset.push_to_hub(repo_id, config_name=\"standard_preference\")\n\n standard_implicit_prompt_preference_dataset = Dataset.from_dict({\n \"chosen\": [\n \"Beautiful is better than ugly.\",\n \"Explicit is better than implicit.\",\n \"Simple is better than complex.\",\n \"Complex is better than complicated.\",\n \"Flat is better than nested.\",\n \"Sparse is better than dense.\",\n \"Readability counts.\",\n \"Special cases aren't special enough to break the rules.\",\n \"Although practicality beats purity.\",\n \"Errors should never pass silently.\",\n \"Unless explicitly silenced.\",\n \"In the face of ambiguity, refuse the temptation to guess.\",\n \"There should be one-- and preferably only one --obvious way to do it.\",\n \"Although that way may not be obvious at first unless you're Dutch.\",\n \"Now is better than never.\",\n \"Although never is often better than *right* now.\",\n \"If the implementation is hard to explain, it's a bad idea.\",\n \"If the implementation is easy to explain, it may be a good idea.\",\n \"Namespaces are one honking great idea -- let's do more of those!\",\n ],\n \"rejected\": [\n \"Beautiful is better than the moon.\",\n \"Explicit is worse than nothing.\",\n \"Simple is better than a long vacation.\",\n \"Complex is always the answer.\",\n \"Flat is better than chocolate.\",\n \"Sparse is better without any context.\",\n \"Readability is optional.\",\n \"Special cases aren't special enough to become unicorns.\",\n \"Although practicality beats reality.\",\n \"Errors should never pass their driving test.\",\n \"Unless explicitly forgotten.\",\n \"In the face of ambiguity, refuse the opportunity to laugh.\",\n \"There should be one-- and preferably two or more confusing methods.\",\n \"Although that way may not be obvious at first unless you're a time traveler.\",\n \"Now is never better.\",\n \"Although never is often not even a possibility.\",\n \"If the implementation is hard to explain, it's clearly the best choice.\",\n \"If the implementation is easy it's probably magic.\",\n \"Namespaces are one honking great watermelon -- let's plant some!\",\n ],\n })\n standard_implicit_prompt_preference_dataset = standard_implicit_prompt_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n standard_implicit_prompt_preference_dataset.push_to_hub(repo_id, config_name=\"standard_implicit_prompt_preference\")\n\n standard_unpaired_preference_dataset = Dataset.from_dict({\n \"prompt\": [\n \"Beautiful is better than\",\n \"Explicit is\",\n \"Simple is better\",\n \"Complex\",\n \"Flat is better than\",\n \"Sparse is better\",\n \"Readability\",\n \"Special cases aren't special\",\n \"Although practicality beats\",\n \"Errors should never\",\n \"Unless explicitly\",\n \"In the face of ambiguity, refuse\",\n \"There should be one-- and preferably\",\n \"Although that way may not be obvious at first unless you're\",\n \"Now is\",\n \"Although never is often\",\n \"If the implementation is hard to explain,\",\n \"If the implementation is easy\",\n \"Namespaces are one honking great\",\n ],\n \"completion\": [\n \" ugly.\",\n \" worse than nothing.\",\n \" than a long vacation.\",\n \" is better than complicated.\",\n \" nested.\",\n \" without any context.\",\n \" counts.\",\n \" enough to become unicorns.\",\n \" purity.\",\n \" pass silently.\",\n \" forgotten.\",\n \" the temptation to guess.\",\n \" only one --obvious way to do it.\",\n \" a time traveler.\",\n \" better than never.\",\n \" not even a possibility.\",\n \" it's a bad idea.\",\n \" it's probably magic.\",\n \" watermelon -- let's plant some!\",\n ],\n \"label\": [True, False, False, True, True, False, True, False, True, True, False, True, True, False, True, False, True, False, False],\n })\n standard_unpaired_preference_dataset = standard_unpaired_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n standard_unpaired_preference_dataset.push_to_hub(repo_id, config_name=\"standard_unpaired_preference\")\n\n standard_stepwise_supervision_dataset = Dataset.from_dict({\n \"prompt\": [\n \"Beautiful is better than\",\n \"Explicit is better than\",\n \"Simple is better than\",\n \"Complex is better than\",\n \"Flat is better than\",\n \"Sparse is better than\",\n \"Readability counts\",\n \"Special cases aren't special enough\",\n \"Although practicality beats\",\n \"Errors should never pass\",\n \"In the face of ambiguity, refuse\",\n \"There should be one-- and preferably only one --\",\n \"Although that way may not be\",\n \"Now is better than\",\n \"Never is often better than\",\n \"If the implementation is hard to explain, it's\",\n \"If the implementation is easy to explain, it\",\n \"Namespaces are one\",\n \"Although practicality sometimes beats purity,\",\n ],\n \"completions\": [\n [\", let me think...\", \" ugly.\"],\n [\", of course,\", \" implicit.\", \" because clarity matters.\"],\n [\"... let's keep it basic,\", \" complex.\"],\n [\" when needed,\", \" complicated.\"],\n [\" in terms of structure,\", \" nested.\"],\n [\"... especially for readability.\"],\n [\" especially when others read it.\"],\n [\", unless...\", \" they follow the rules.\"],\n [\" some theoretical elegance,\", \" purity.\"],\n [\" silently,\", \" unless explicitly silenced.\"],\n [\" the temptation to guess.\"],\n [\" way to do it,\", \" but sometimes it's not obvious.\", \" especially when there's more than one possibility.\"],\n [\" clear at first,\", \" it will eventually emerge.\"],\n [\" later.\"],\n [\" problematic fixes.\"],\n [\" likely because it's too complicated.\"],\n [\" might be a good design.\"],\n [\" of those great ideas,\", \" that solve many problems.\"],\n [\" the code should still aim for balance.\"],\n ],\n \"labels\": [\n [False, True],\n [False, True, False],\n [False, True],\n [True, True],\n [True, False],\n [True],\n [False],\n [True, False],\n [False, False],\n [False, False],\n [True],\n [True, True, False],\n [True, True],\n [False],\n [True], [False],\n [False],\n [True, True],\n [False]\n ]\n })\n standard_stepwise_supervision_dataset = standard_stepwise_supervision_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n standard_stepwise_supervision_dataset.push_to_hub(repo_id, config_name=\"standard_stepwise_supervision\")\n\n conversational_language_modeling_dataset = Dataset.from_dict({\n \"messages\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}, {\"role\": \"assistant\", \"content\": \"Beautiful.\"},],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}, {\"role\": \"assistant\", \"content\": \"Explicit.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}, {\"role\": \"assistant\", \"content\": \"Simple.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}, {\"role\": \"assistant\", \"content\": \"Complex.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}, {\"role\": \"assistant\", \"content\": \"Flat.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}, {\"role\": \"assistant\", \"content\": \"Sparse.\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}, {\"role\": \"assistant\", \"content\": \"Readability.\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}, {\"role\": \"assistant\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}, {\"role\": \"assistant\", \"content\": \"Practicality.\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}, {\"role\": \"assistant\", \"content\": \"Errors.\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}, {\"role\": \"assistant\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}, {\"role\": \"assistant\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}, {\"role\": \"assistant\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}, {\"role\": \"assistant\", \"content\": \"Dutch.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}, {\"role\": \"assistant\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}, {\"role\": \"assistant\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}, {\"role\": \"assistant\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n })\n conversational_language_modeling_dataset = conversational_language_modeling_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_language_modeling_dataset.push_to_hub(repo_id, config_name=\"conversational_language_modeling\")\n\n conversational_prompt_only_dataset = Dataset.from_dict({\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n })\n conversational_prompt_only_dataset = conversational_prompt_only_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_prompt_only_dataset.push_to_hub(repo_id, config_name=\"conversational_prompt_only\")\n\n conversational_prompt_completion_dataset = Dataset.from_dict({\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n \"completion\": [\n [{\"role\": \"assistant\", \"content\": \"Beautiful.\"}],\n [{\"role\": \"assistant\", \"content\": \"Explicit.\"}],\n [{\"role\": \"assistant\", \"content\": \"Simple.\"}],\n [{\"role\": \"assistant\", \"content\": \"Complex.\"}],\n [{\"role\": \"assistant\", \"content\": \"Flat.\"}],\n [{\"role\": \"assistant\", \"content\": \"Sparse.\"}],\n [{\"role\": \"assistant\", \"content\": \"Readability.\"}],\n [{\"role\": \"assistant\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"assistant\", \"content\": \"Practicality.\"}],\n [{\"role\": \"assistant\", \"content\": \"Errors.\"}],\n [{\"role\": \"assistant\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"assistant\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"assistant\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"assistant\", \"content\": \"Dutch.\"}],\n [{\"role\": \"assistant\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"assistant\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n })\n conversational_prompt_completion_dataset = conversational_prompt_completion_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_prompt_completion_dataset.push_to_hub(repo_id, config_name=\"conversational_prompt_completion\")\n\n conversational_preference_dataset = Dataset.from_dict({\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n \"chosen\": [\n [{\"role\": \"assistant\", \"content\": \"Beautiful.\"}],\n [{\"role\": \"assistant\", \"content\": \"Explicit.\"}],\n [{\"role\": \"assistant\", \"content\": \"Simple.\"}],\n [{\"role\": \"assistant\", \"content\": \"Complex.\"}],\n [{\"role\": \"assistant\", \"content\": \"Flat.\"}],\n [{\"role\": \"assistant\", \"content\": \"Sparse.\"}],\n [{\"role\": \"assistant\", \"content\": \"Readability.\"}],\n [{\"role\": \"assistant\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"assistant\", \"content\": \"Practicality.\"}],\n [{\"role\": \"assistant\", \"content\": \"Errors.\"}],\n [{\"role\": \"assistant\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"assistant\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"assistant\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"assistant\", \"content\": \"Dutch.\"}],\n [{\"role\": \"assistant\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"assistant\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"assistant\", \"content\": \"Acceptable.\"}],\n [{\"role\": \"assistant\", \"content\": \"Explained.\"}],\n [{\"role\": \"assistant\", \"content\": \"Very complex.\"}],\n [{\"role\": \"assistant\", \"content\": \"Very complicated.\"}],\n [{\"role\": \"assistant\", \"content\": \"Circular.\"}],\n [{\"role\": \"assistant\", \"content\": \"Heavy.\"}],\n [{\"role\": \"assistant\", \"content\": \"Looking complicated.\"}],\n [{\"role\": \"assistant\", \"content\": \"Yes, special cases are special enough to break the rules.\"}],\n [{\"role\": \"assistant\", \"content\": \"Nothing.\"}],\n [{\"role\": \"assistant\", \"content\": \"Warnings.\"}],\n [{\"role\": \"assistant\", \"content\": \"Never.\"}],\n [{\"role\": \"assistant\", \"content\": \"Give up.\"}],\n [{\"role\": \"assistant\", \"content\": \"As many as possible.\"}],\n [{\"role\": \"assistant\", \"content\": \"French.\"}],\n [{\"role\": \"assistant\", \"content\": \"Some day.\"}],\n [{\"role\": \"assistant\", \"content\": \"No, never.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it's a good idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"Recursion.\"}],\n ],\n })\n conversational_preference_dataset = conversational_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_preference_dataset.push_to_hub(repo_id, config_name=\"conversational_preference\")\n\n conversational_implicit_prompt_preference_dataset = Dataset.from_dict({\n \"chosen\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}, {\"role\": \"assistant\", \"content\": \"Beautiful.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}, {\"role\": \"assistant\", \"content\": \"Explicit.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}, {\"role\": \"assistant\", \"content\": \"Simple.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}, {\"role\": \"assistant\", \"content\": \"Complex.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}, {\"role\": \"assistant\", \"content\": \"Flat.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}, {\"role\": \"assistant\", \"content\": \"Sparse.\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}, {\"role\": \"assistant\", \"content\": \"Readability.\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}, {\"role\": \"assistant\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}, {\"role\": \"assistant\", \"content\": \"Practicality.\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}, {\"role\": \"assistant\", \"content\": \"Errors.\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}, {\"role\": \"assistant\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}, {\"role\": \"assistant\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}, {\"role\": \"assistant\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}, {\"role\": \"assistant\", \"content\": \"Dutch.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}, {\"role\": \"assistant\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}, {\"role\": \"assistant\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}, {\"role\": \"assistant\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}, {\"role\": \"assistant\", \"content\": \"Acceptable.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}, {\"role\": \"assistant\", \"content\": \"Explained.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}, {\"role\": \"assistant\", \"content\": \"Very complex.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}, {\"role\": \"assistant\", \"content\": \"Very complicated.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}, {\"role\": \"assistant\", \"content\": \"Circular.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}, {\"role\": \"assistant\", \"content\": \"Heavy.\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}, {\"role\": \"assistant\", \"content\": \"Looking complicated.\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}, {\"role\": \"assistant\", \"content\": \"Yes, special cases are special enough to break the rules.\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}, {\"role\": \"assistant\", \"content\": \"Nothing.\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}, {\"role\": \"assistant\", \"content\": \"Warnings.\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}, {\"role\": \"assistant\", \"content\": \"Never.\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}, {\"role\": \"assistant\", \"content\": \"Give up.\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}, {\"role\": \"assistant\", \"content\": \"As many as possible.\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}, {\"role\": \"assistant\", \"content\": \"French.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}, {\"role\": \"assistant\", \"content\": \"Some day.\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}, {\"role\": \"assistant\", \"content\": \"No, never.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it's a good idea.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}, {\"role\": \"assistant\", \"content\": \"Recursion.\"}],\n ],\n })\n conversational_implicit_prompt_preference_dataset = conversational_implicit_prompt_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_implicit_prompt_preference_dataset.push_to_hub(repo_id, config_name=\"conversational_implicit_prompt_preference\")\n\n conversational_unpaired_preference_dataset = Dataset.from_dict({\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n \"completion\": [\n [{'role': 'assistant', 'content': 'Beautiful.'}],\n [{'role': 'assistant', 'content': 'Explicit.'}],\n [{'role': 'assistant', 'content': 'Simple.'}],\n [{'role': 'assistant', 'content': 'Very complicated.'}],\n [{'role': 'assistant', 'content': 'Flat.'}],\n [{'role': 'assistant', 'content': 'Sparse.'}],\n [{'role': 'assistant', 'content': 'Readability.'}],\n [{'role': 'assistant', 'content': 'Yes, special cases are special enough to break the rules.'}],\n [{'role': 'assistant', 'content': 'Practicality.'}],\n [{'role': 'assistant', 'content': 'Warnings.'}],\n [{'role': 'assistant', 'content': 'When explicitly silenced.'}],\n [{'role': 'assistant', 'content': 'Give up.'}],\n [{'role': 'assistant', 'content': 'One, and preferably only one.'}],\n [{'role': 'assistant', 'content': 'French.'}],\n [{'role': 'assistant', 'content': 'Some day.'}],\n [{'role': 'assistant', 'content': 'Yes, often.'}],\n [{'role': 'assistant', 'content': \"It means it's a bad idea.\"}],\n [{'role': 'assistant', 'content': 'It means it may be a good idea.'}],\n [{'role': 'assistant', 'content': 'Namespaces are one honking great idea.'}],\n ],\n \"label\": [True, True, True, False, True, True, True, False, True, False, True, False, True, False, False, True, True, True, True],\n })\n conversational_unpaired_preference_dataset = conversational_unpaired_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_unpaired_preference_dataset.push_to_hub(repo_id, config_name=\"conversational_unpaired_preference\")\n # fmt: on\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n main(script_args.test_size, script_args.push_to_hub, script_args.repo_id)\n" }, { "path": "scripts/generate_zen_image_dataset.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nimport numpy as np\nfrom datasets import Dataset, Features, Image, Value\nfrom transformers import HfArgumentParser\n\n\nMessage = [{\"content\": Value(\"string\"), \"role\": Value(\"string\")}]\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n test_size (`float`, *optional*, defaults to `0.1`):\n Fraction of the dataset to include in the test split.\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-internal-testing/zen-image\"`):\n Hugging Face repository ID to push the dataset to.\n \"\"\"\n\n test_size: float = field(\n default=0.1,\n metadata={\"help\": \"Fraction of the dataset to include in the test split.\"},\n )\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-internal-testing/zen-image\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n\n\ndef main(test_size, push_to_hub, repo_id):\n # fmt: off\n sizes = np.random.randint(32, 64, size=(19, 2))\n data = {\n \"messages\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}, {\"role\": \"assistant\", \"content\": \"Beautiful.\"},],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}, {\"role\": \"assistant\", \"content\": \"Explicit.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}, {\"role\": \"assistant\", \"content\": \"Simple.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}, {\"role\": \"assistant\", \"content\": \"Complex.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}, {\"role\": \"assistant\", \"content\": \"Flat.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}, {\"role\": \"assistant\", \"content\": \"Sparse.\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}, {\"role\": \"assistant\", \"content\": \"Readability.\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}, {\"role\": \"assistant\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}, {\"role\": \"assistant\", \"content\": \"Practicality.\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}, {\"role\": \"assistant\", \"content\": \"Errors.\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}, {\"role\": \"assistant\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}, {\"role\": \"assistant\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}, {\"role\": \"assistant\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}, {\"role\": \"assistant\", \"content\": \"Dutch.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}, {\"role\": \"assistant\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}, {\"role\": \"assistant\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}, {\"role\": \"assistant\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n \"image\": [np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in sizes],\n }\n conversational_language_modeling_dataset = Dataset.from_dict(data, features=Features(messages=Message, image=Image()))\n conversational_language_modeling_dataset = conversational_language_modeling_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_language_modeling_dataset.push_to_hub(repo_id, config_name=\"conversational_language_modeling\")\n\n sizes = np.random.randint(32, 64, size=(19, 2))\n data = {\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n \"image\": [np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in sizes],\n }\n conversational_prompt_only_dataset = Dataset.from_dict(data, features=Features(prompt=Message, image=Image()))\n conversational_prompt_only_dataset = conversational_prompt_only_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_prompt_only_dataset.push_to_hub(repo_id, config_name=\"conversational_prompt_only\")\n\n sizes = np.random.randint(32, 64, size=(19, 2))\n data = {\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n \"completion\": [\n [{\"role\": \"assistant\", \"content\": \"Beautiful.\"}],\n [{\"role\": \"assistant\", \"content\": \"Explicit.\"}],\n [{\"role\": \"assistant\", \"content\": \"Simple.\"}],\n [{\"role\": \"assistant\", \"content\": \"Complex.\"}],\n [{\"role\": \"assistant\", \"content\": \"Flat.\"}],\n [{\"role\": \"assistant\", \"content\": \"Sparse.\"}],\n [{\"role\": \"assistant\", \"content\": \"Readability.\"}],\n [{\"role\": \"assistant\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"assistant\", \"content\": \"Practicality.\"}],\n [{\"role\": \"assistant\", \"content\": \"Errors.\"}],\n [{\"role\": \"assistant\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"assistant\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"assistant\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"assistant\", \"content\": \"Dutch.\"}],\n [{\"role\": \"assistant\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"assistant\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n \"image\": [np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in sizes],\n }\n conversational_prompt_completion_dataset = Dataset.from_dict(data, features=Features(prompt=Message, completion=Message, image=Image()))\n conversational_prompt_completion_dataset = conversational_prompt_completion_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_prompt_completion_dataset.push_to_hub(repo_id, config_name=\"conversational_prompt_completion\")\n\n sizes = np.random.randint(32, 64, size=(19, 2))\n data = {\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n \"chosen\": [\n [{\"role\": \"assistant\", \"content\": \"Beautiful.\"}],\n [{\"role\": \"assistant\", \"content\": \"Explicit.\"}],\n [{\"role\": \"assistant\", \"content\": \"Simple.\"}],\n [{\"role\": \"assistant\", \"content\": \"Complex.\"}],\n [{\"role\": \"assistant\", \"content\": \"Flat.\"}],\n [{\"role\": \"assistant\", \"content\": \"Sparse.\"}],\n [{\"role\": \"assistant\", \"content\": \"Readability.\"}],\n [{\"role\": \"assistant\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"assistant\", \"content\": \"Practicality.\"}],\n [{\"role\": \"assistant\", \"content\": \"Errors.\"}],\n [{\"role\": \"assistant\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"assistant\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"assistant\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"assistant\", \"content\": \"Dutch.\"}],\n [{\"role\": \"assistant\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"assistant\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"assistant\", \"content\": \"Acceptable.\"}],\n [{\"role\": \"assistant\", \"content\": \"Explained.\"}],\n [{\"role\": \"assistant\", \"content\": \"Very complex.\"}],\n [{\"role\": \"assistant\", \"content\": \"Very complicated.\"}],\n [{\"role\": \"assistant\", \"content\": \"Circular.\"}],\n [{\"role\": \"assistant\", \"content\": \"Heavy.\"}],\n [{\"role\": \"assistant\", \"content\": \"Looking complicated.\"}],\n [{\"role\": \"assistant\", \"content\": \"Yes, special cases are special enough to break the rules.\"}],\n [{\"role\": \"assistant\", \"content\": \"Nothing.\"}],\n [{\"role\": \"assistant\", \"content\": \"Warnings.\"}],\n [{\"role\": \"assistant\", \"content\": \"Never.\"}],\n [{\"role\": \"assistant\", \"content\": \"Give up.\"}],\n [{\"role\": \"assistant\", \"content\": \"As many as possible.\"}],\n [{\"role\": \"assistant\", \"content\": \"French.\"}],\n [{\"role\": \"assistant\", \"content\": \"Some day.\"}],\n [{\"role\": \"assistant\", \"content\": \"No, never.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it's a good idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"assistant\", \"content\": \"Recursion.\"}],\n ],\n \"image\": [np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in sizes],\n }\n conversational_preference_dataset = Dataset.from_dict(data, features=Features(prompt=Message, chosen=Message, rejected=Message, image=Image()))\n conversational_preference_dataset = conversational_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_preference_dataset.push_to_hub(repo_id, config_name=\"conversational_preference\")\n\n sizes = np.random.randint(32, 64, size=(19, 2))\n data = {\n \"chosen\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}, {\"role\": \"assistant\", \"content\": \"Beautiful.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}, {\"role\": \"assistant\", \"content\": \"Explicit.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}, {\"role\": \"assistant\", \"content\": \"Simple.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}, {\"role\": \"assistant\", \"content\": \"Complex.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}, {\"role\": \"assistant\", \"content\": \"Flat.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}, {\"role\": \"assistant\", \"content\": \"Sparse.\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}, {\"role\": \"assistant\", \"content\": \"Readability.\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}, {\"role\": \"assistant\", \"content\": \"No, special cases aren't special enough to break the rules.\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}, {\"role\": \"assistant\", \"content\": \"Practicality.\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}, {\"role\": \"assistant\", \"content\": \"Errors.\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}, {\"role\": \"assistant\", \"content\": \"When explicitly silenced.\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}, {\"role\": \"assistant\", \"content\": \"Refuse the temptation to guess.\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}, {\"role\": \"assistant\", \"content\": \"One, and preferably only one.\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}, {\"role\": \"assistant\", \"content\": \"Dutch.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}, {\"role\": \"assistant\", \"content\": \"Now is better than never.\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}, {\"role\": \"assistant\", \"content\": \"Yes, often.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it may be a good idea.\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}, {\"role\": \"assistant\", \"content\": \"Namespaces are one honking great idea.\"}],\n ],\n \"rejected\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}, {\"role\": \"assistant\", \"content\": \"Acceptable.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}, {\"role\": \"assistant\", \"content\": \"Explained.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}, {\"role\": \"assistant\", \"content\": \"Very complex.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}, {\"role\": \"assistant\", \"content\": \"Very complicated.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}, {\"role\": \"assistant\", \"content\": \"Circular.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}, {\"role\": \"assistant\", \"content\": \"Heavy.\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}, {\"role\": \"assistant\", \"content\": \"Looking complicated.\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}, {\"role\": \"assistant\", \"content\": \"Yes, special cases are special enough to break the rules.\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}, {\"role\": \"assistant\", \"content\": \"Nothing.\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}, {\"role\": \"assistant\", \"content\": \"Warnings.\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}, {\"role\": \"assistant\", \"content\": \"Never.\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}, {\"role\": \"assistant\", \"content\": \"Give up.\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}, {\"role\": \"assistant\", \"content\": \"As many as possible.\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}, {\"role\": \"assistant\", \"content\": \"French.\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}, {\"role\": \"assistant\", \"content\": \"Some day.\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}, {\"role\": \"assistant\", \"content\": \"No, never.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it's a good idea.\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}, {\"role\": \"assistant\", \"content\": \"It means it's a bad idea.\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}, {\"role\": \"assistant\", \"content\": \"Recursion.\"}],\n ],\n \"image\": [np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in sizes],\n }\n conversational_implicit_prompt_preference_dataset = Dataset.from_dict(data, features=Features(chosen=Message, rejected=Message, image=Image()))\n conversational_implicit_prompt_preference_dataset = conversational_implicit_prompt_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_implicit_prompt_preference_dataset.push_to_hub(repo_id, config_name=\"conversational_implicit_prompt_preference\")\n\n sizes = np.random.randint(32, 64, size=(19, 2))\n data = {\n \"prompt\": [\n [{\"role\": \"user\", \"content\": \"What is better than ugly?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than implicit?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complex?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than complicated?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than nested?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than dense?\"}],\n [{\"role\": \"user\", \"content\": \"What counts?\"}],\n [{\"role\": \"user\", \"content\": \"Are special cases enough to break the rules?\"}],\n [{\"role\": \"user\", \"content\": \"What beats purity?\"}],\n [{\"role\": \"user\", \"content\": \"What should never pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"When can errors pass silently?\"}],\n [{\"role\": \"user\", \"content\": \"What should you do in the face of ambiguity?\"}],\n [{\"role\": \"user\", \"content\": \"How many ways should there be to do it?\"}],\n [{\"role\": \"user\", \"content\": \"For whom may the way not be obvious at first?\"}],\n [{\"role\": \"user\", \"content\": \"What is better than never?\"}],\n [{\"role\": \"user\", \"content\": \"Is never better than *right* now?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is hard to explain?\"}],\n [{\"role\": \"user\", \"content\": \"What does it mean if the implementation is easy to explain?\"}],\n [{\"role\": \"user\", \"content\": \"Any great ideas?\"}],\n ],\n \"completion\": [\n [{'role': 'assistant', 'content': 'Beautiful.'}],\n [{'role': 'assistant', 'content': 'Explicit.'}],\n [{'role': 'assistant', 'content': 'Simple.'}],\n [{'role': 'assistant', 'content': 'Very complicated.'}],\n [{'role': 'assistant', 'content': 'Flat.'}],\n [{'role': 'assistant', 'content': 'Sparse.'}],\n [{'role': 'assistant', 'content': 'Readability.'}],\n [{'role': 'assistant', 'content': 'Yes, special cases are special enough to break the rules.'}],\n [{'role': 'assistant', 'content': 'Practicality.'}],\n [{'role': 'assistant', 'content': 'Warnings.'}],\n [{'role': 'assistant', 'content': 'When explicitly silenced.'}],\n [{'role': 'assistant', 'content': 'Give up.'}],\n [{'role': 'assistant', 'content': 'One, and preferably only one.'}],\n [{'role': 'assistant', 'content': 'French.'}],\n [{'role': 'assistant', 'content': 'Some day.'}],\n [{'role': 'assistant', 'content': 'Yes, often.'}],\n [{'role': 'assistant', 'content': \"It means it's a bad idea.\"}],\n [{'role': 'assistant', 'content': 'It means it may be a good idea.'}],\n [{'role': 'assistant', 'content': 'Namespaces are one honking great idea.'}],\n ],\n \"label\": [True, True, True, False, True, True, True, False, True, False, True, False, True, False, False, True, True, True, True],\n \"image\": [np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in sizes],\n }\n conversational_unpaired_preference_dataset = Dataset.from_dict(data, features=Features(prompt=Message, completion=Message, label=Value(\"bool\"), image=Image()))\n conversational_unpaired_preference_dataset = conversational_unpaired_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_unpaired_preference_dataset.push_to_hub(repo_id, config_name=\"conversational_unpaired_preference\")\n # fmt: on\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n main(script_args.test_size, script_args.push_to_hub, script_args.repo_id)\n" }, { "path": "scripts/generate_zen_multi_image_dataset.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nimport numpy as np\nfrom datasets import Dataset, Features, Image, List, Value\nfrom transformers import HfArgumentParser\n\n\nMessage = List({\"content\": List({\"text\": Value(\"string\"), \"type\": Value(\"string\")}), \"role\": Value(\"string\")})\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n test_size (`float`, *optional*, defaults to `0.1`):\n Fraction of the dataset to include in the test split.\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the dataset to the Hugging Face Hub.\n repo_id (`str`, *optional*, defaults to `\"trl-internal-testing/zen-multi-image\"`):\n Hugging Face repository ID to push the dataset to.\n \"\"\"\n\n test_size: float = field(\n default=0.1,\n metadata={\"help\": \"Fraction of the dataset to include in the test split.\"},\n )\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the dataset to the Hugging Face Hub.\"},\n )\n repo_id: str = field(\n default=\"trl-internal-testing/zen-multi-image\",\n metadata={\"help\": \"Hugging Face repository ID to push the dataset to.\"},\n )\n\n\ndef main(test_size, push_to_hub, repo_id):\n # fmt: off\n messages = [\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than ugly?\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Beautiful.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than implicit?\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Explicit.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than complex?\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Simple.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than complicated?\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Complex.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than nested?\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Flat.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than dense?\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Sparse.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What counts?\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Readability.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Are special cases enough to break the rules?\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"No, special cases aren't special enough to break the rules.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What beats purity?\"}, {\"type\": \"image\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Practicality.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What should never pass silently?\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Errors.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"When can errors pass silently?\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"When explicitly silenced.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What should you do in the face of ambiguity?\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Refuse the temptation to guess.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"How many ways should there be to do it?\"}, {\"type\": \"image\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"One, and preferably only one.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"For whom may the way not be obvious at first?\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Dutch.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than never?\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Now is better than never.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Is\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \" never better than *right* now?\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Yes, often.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What does it mean if the implementation is hard to explain?\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It means it's a bad idea.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What does it mean if the implementation is easy to explain?\"}, {\"type\": \"image\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It means it may be a good idea.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Any great ideas?\"}]}, {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Namespaces are one honking great idea.\"}]}],\n ]\n # Create the images\n number_of_images = [sum(1 for part in row[0][\"content\"] if part.get(\"type\") == \"image\") for row in messages]\n sizes = [np.random.randint(32, 64, size=(num_images, 2)) for num_images in number_of_images]\n images = [[np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in s] for s in sizes]\n conversational_language_modeling_dataset = Dataset.from_dict({\"messages\": messages, \"images\": images}, features=Features(messages=Message, images=List(Image())))\n conversational_language_modeling_dataset = conversational_language_modeling_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_language_modeling_dataset.push_to_hub(repo_id, config_name=\"conversational_language_modeling\")\n\n prompt = [\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than ugly?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than implicit?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than complex?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than complicated?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than nested?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than dense?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What counts?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Are special cases enough to break the rules?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What beats purity?\"}, {\"type\": \"image\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What should never pass silently?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"When can errors pass silently?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What should you do in the face of ambiguity?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"How many ways should there be to do it?\"}, {\"type\": \"image\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"For whom may the way not be obvious at first?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than never?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Is\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \" never better than *right* now?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What does it mean if the implementation is hard to explain?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What does it mean if the implementation is easy to explain?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Any great ideas?\"}]}],\n ]\n # Create the images\n number_of_images = [sum(1 for part in row[0][\"content\"] if part.get(\"type\") == \"image\") for row in prompt]\n sizes = [np.random.randint(32, 64, size=(num_images, 2)) for num_images in number_of_images]\n images = [[np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in s] for s in sizes]\n conversational_prompt_only_dataset = Dataset.from_dict({\"prompt\": prompt, \"images\": images}, features=Features(prompt=Message, images=List(Image())))\n conversational_prompt_only_dataset = conversational_prompt_only_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_prompt_only_dataset.push_to_hub(repo_id, config_name=\"conversational_prompt_only\")\n\n prompt = [\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than ugly?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than implicit?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than complex?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than complicated?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than nested?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than dense?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What counts?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Are special cases enough to break the rules?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What beats purity?\"}, {\"type\": \"image\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What should never pass silently?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"When can errors pass silently?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What should you do in the face of ambiguity?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"How many ways should there be to do it?\"}, {\"type\": \"image\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"For whom may the way not be obvious at first?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than never?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Is\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \" never better than *right* now?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What does it mean if the implementation is hard to explain?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What does it mean if the implementation is easy to explain?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Any great ideas?\"}]}],\n ]\n completion = [\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Beautiful.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Explicit.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Simple.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Complex.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Flat.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Sparse.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Readability.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"No, special cases aren't special enough to break the rules.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Practicality.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Errors.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"When explicitly silenced.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Refuse the temptation to guess.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"One, and preferably only one.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Dutch.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Now is better than never.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Yes, often.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It means it's a bad idea.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It means it may be a good idea.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Namespaces are one honking great idea.\"}]}],\n ]\n # Create the images\n number_of_images = [sum(1 for part in row[0][\"content\"] if part.get(\"type\") == \"image\") for row in prompt]\n sizes = [np.random.randint(32, 64, size=(num_images, 2)) for num_images in number_of_images]\n images = [[np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in s] for s in sizes]\n conversational_prompt_completion_dataset = Dataset.from_dict({\"prompt\": prompt, \"completion\": completion, \"images\": images}, features=Features(prompt=Message, completion=Message, images=List(Image())))\n conversational_prompt_completion_dataset = conversational_prompt_completion_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_prompt_completion_dataset.push_to_hub(repo_id, config_name=\"conversational_prompt_completion\")\n\n prompt = [\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than ugly?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than implicit?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than complex?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than complicated?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What is better than nested?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than dense?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What counts?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Are special cases enough to break the rules?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What beats purity?\"}, {\"type\": \"image\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What should never pass silently?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"When can errors pass silently?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What should you do in the face of ambiguity?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"How many ways should there be to do it?\"}, {\"type\": \"image\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"For whom may the way not be obvious at first?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What is better than never?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Is\"}, {\"type\": \"image\"}, {\"type\": \"text\", \"text\": \" never better than *right* now?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What does it mean if the implementation is hard to explain?\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What does it mean if the implementation is easy to explain?\"}, {\"type\": \"image\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"Any great ideas?\"}]}],\n ]\n chosen = [\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Beautiful.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Explicit.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Simple.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Complex.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Flat.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Sparse.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Readability.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"No, special cases aren't special enough to break the rules.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Practicality.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Errors.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"When explicitly silenced.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Refuse the temptation to guess.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"One, and preferably only one.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Dutch.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Now is better than never.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Yes, often.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It means it's a bad idea.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It means it may be a good idea.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Namespaces are one honking great idea.\"}]}],\n ]\n rejected = [\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Acceptable.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Explained.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Very complex.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Very complicated.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Circular.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Heavy.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Looking complicated.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Yes, special cases are special enough to break the rules.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Nothing.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Warnings.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Never.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Give up.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"As many as possible.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"French.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Some day.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"No, never.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It means it's a good idea.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It means it's a bad idea.\"}]}],\n [{\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"Recursion.\"}]}],\n ]\n # Create the images\n number_of_images = [sum(1 for part in row[0][\"content\"] if part.get(\"type\") == \"image\") for row in prompt]\n sizes = [np.random.randint(32, 64, size=(num_images, 2)) for num_images in number_of_images]\n images = [[np.random.uniform(low=0.0, high=255.0, size=(h, w, 3)).astype(np.uint8) for h, w in s] for s in sizes]\n conversational_preference_dataset = Dataset.from_dict({\"prompt\": prompt, \"chosen\": chosen, \"rejected\": rejected, \"images\": images}, features=Features(prompt=Message, chosen=Message, rejected=Message, images=List(Image())))\n conversational_preference_dataset = conversational_preference_dataset.train_test_split(test_size=test_size, shuffle=False)\n if push_to_hub:\n conversational_preference_dataset.push_to_hub(repo_id, config_name=\"conversational_preference\")\n # fmt: on\n\n\nif __name__ == \"__main__\":\n parser = HfArgumentParser(ScriptArguments)\n script_args = parser.parse_args_into_dataclasses()[0]\n main(script_args.test_size, script_args.push_to_hub, script_args.repo_id)\n" }, { "path": "scripts/log_reports.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport argparse\nimport json\nimport logging\nimport os\nfrom datetime import date\nfrom pathlib import Path\n\nfrom tabulate import tabulate\n\n\nMAX_LEN_MESSAGE = 2900 # Slack endpoint has a limit of 3001 characters\n\nparser = argparse.ArgumentParser()\nparser.add_argument(\"--slack_channel_name\", default=\"trl-push-ci\")\n\n# Set up logging\nlogging.basicConfig(level=logging.INFO, format=\"%(asctime)s - %(levelname)s - %(message)s\")\n\n\ndef process_log_file(log):\n failed_tests = []\n passed_tests = []\n section_num_failed = 0\n\n try:\n with open(log) as f:\n for line in f:\n try:\n data = json.loads(line)\n test_name = data.get(\"nodeid\", \"\")\n duration = f\"{data['duration']:.4f}\" if \"duration\" in data else \"N/A\"\n outcome = data.get(\"outcome\", \"\")\n\n if test_name:\n if outcome == \"failed\":\n section_num_failed += 1\n failed_tests.append([test_name, duration, log.stem.split(\"_\")[0]])\n else:\n passed_tests.append([test_name, duration, log.stem.split(\"_\")[0]])\n except json.JSONDecodeError as e:\n logging.warning(f\"Could not decode line in {log}: {e}\")\n\n except FileNotFoundError as e:\n logging.error(f\"Log file {log} not found: {e}\")\n except Exception as e:\n logging.error(f\"Error processing log file {log}: {e}\")\n\n return failed_tests, passed_tests, section_num_failed\n\n\ndef main(slack_channel_name):\n group_info = []\n total_num_failed = 0\n total_empty_files = []\n\n log_files = list(Path().glob(\"*.log\"))\n if not log_files:\n logging.info(\"No log files found.\")\n return\n\n for log in log_files:\n failed, passed, section_num_failed = process_log_file(log)\n empty_file = not failed and not passed\n\n total_num_failed += section_num_failed\n total_empty_files.append(empty_file)\n group_info.append([str(log), section_num_failed, failed])\n\n # Clean up log file\n try:\n os.remove(log)\n except OSError as e:\n logging.warning(f\"Could not remove log file {log}: {e}\")\n\n # Prepare Slack message payload\n payload = [\n {\n \"type\": \"header\",\n \"text\": {\"type\": \"plain_text\", \"text\": f\"๐Ÿค— Results of the {os.environ.get('TEST_TYPE', '')} TRL tests.\"},\n },\n ]\n\n if total_num_failed > 0:\n message = \"\"\n for name, num_failed, failed_tests in group_info:\n if num_failed > 0:\n message += f\"*{name}: {num_failed} failed test(s)*\\n\"\n failed_table = [\n test[0].split(\"::\")[:2] + [test[0].split(\"::\")[-1][:30] + \"..\"] for test in failed_tests\n ]\n message += (\n \"\\n```\\n\"\n + tabulate(failed_table, headers=[\"Test Location\", \"Test Name\"], tablefmt=\"grid\")\n + \"\\n```\\n\"\n )\n\n if any(total_empty_files):\n message += f\"\\n*{name}: Warning! Empty file - check GitHub action job*\\n\"\n\n # Logging\n logging.info(f\"Total failed tests: {total_num_failed}\")\n print(f\"### {message}\")\n\n if len(message) > MAX_LEN_MESSAGE:\n message = (\n f\"โŒ There are {total_num_failed} failed tests in total! Please check the action results directly.\"\n )\n\n payload.append({\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": message}})\n payload.append(\n {\n \"type\": \"section\",\n \"text\": {\"type\": \"mrkdwn\", \"text\": \"*For more details:*\"},\n \"accessory\": {\n \"type\": \"button\",\n \"text\": {\"type\": \"plain_text\", \"text\": \"Check Action results\"},\n \"url\": f\"https://github.com/huggingface/trl/actions/runs/{os.environ['GITHUB_RUN_ID']}\",\n },\n }\n )\n payload.append(\n {\n \"type\": \"context\",\n \"elements\": [\n {\n \"type\": \"plain_text\",\n \"text\": f\"On Push main {os.environ.get('TEST_TYPE')} results for {date.today()}\",\n }\n ],\n }\n )\n\n # Send to Slack\n from slack_sdk import WebClient\n\n slack_client = WebClient(token=os.environ.get(\"SLACK_API_TOKEN\"))\n slack_client.chat_postMessage(channel=f\"#{slack_channel_name}\", text=message, blocks=payload)\n\n else:\n payload.append(\n {\n \"type\": \"section\",\n \"text\": {\n \"type\": \"plain_text\",\n \"text\": \"โœ… No failures! All tests passed successfully.\",\n \"emoji\": True,\n },\n }\n )\n logging.info(\"All tests passed. No errors detected.\")\n\n\nif __name__ == \"__main__\":\n args = parser.parse_args()\n main(args.slack_channel_name)\n" }, { "path": "tests/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n" }, { "path": "tests/conftest.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport gc\nfrom functools import wraps\n\nimport pytest\nimport torch\n\n\n# ============================================================================\n# Model Revision Override\n# ============================================================================\n# To test a tiny model PR before merging to main:\n# 1. Add the full model_id and PR revision to this dict\n# 2. Commit and push to trigger CI\n# 3. Once CI is green, merge the tiny model PR on HF Hub\n# 4. Remove the entry from this dict and commit\n#\n# Example:\n# MODEL_REVISIONS = {\n# \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\": \"refs/pr/3\",\n# \"trl-internal-testing/tiny-LlavaForConditionalGeneration\": \"refs/pr/5\",\n# }\n# ============================================================================\n\nMODEL_REVISIONS = {\n # Add model_id: revision mappings here to test PRs\n}\n\n\n@pytest.fixture(autouse=True)\ndef apply_model_revisions(monkeypatch):\n \"\"\"Auto-inject revision parameter for models defined in MODEL_REVISIONS.\"\"\"\n if not MODEL_REVISIONS:\n return\n\n from transformers import PreTrainedModel, PreTrainedTokenizerBase, ProcessorMixin\n\n def create_classmethod_wrapper(original_classmethod):\n # Extract the underlying function from the classmethod\n original_func = original_classmethod.__func__\n\n @wraps(original_func)\n def wrapper(cls, pretrained_model_name_or_path, *args, **kwargs):\n # Direct lookup: only inject if model_id is in the override dict\n if pretrained_model_name_or_path in MODEL_REVISIONS:\n if \"revision\" not in kwargs:\n kwargs[\"revision\"] = MODEL_REVISIONS[pretrained_model_name_or_path]\n\n return original_func(cls, pretrained_model_name_or_path, *args, **kwargs)\n\n # Re-wrap as classmethod\n return classmethod(wrapper)\n\n # Patch all transformers Auto* classes\n for cls in [\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n ]:\n monkeypatch.setattr(cls, \"from_pretrained\", create_classmethod_wrapper(cls.from_pretrained))\n\n\n@pytest.fixture(autouse=True)\ndef cleanup_gpu():\n \"\"\"\n Automatically cleanup GPU memory after each test.\n\n This fixture helps prevent CUDA out of memory errors when running tests in parallel with pytest-xdist by ensuring\n models and tensors are properly garbage collected and GPU memory caches are cleared between tests.\n \"\"\"\n yield\n # Cleanup after test\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n torch.cuda.synchronize()\n" }, { "path": "tests/data/template.jinja", "content": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0].role == 'system' %}\n {{- messages[0].content + '\\n\\n' }}\n {%- endif %}\n {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within XML tags:\\n\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n\\n\\nFor each function call, return a json object with function name and arguments within XML tags:\\n\\n{\\\"name\\\": , \\\"arguments\\\": }\\n<|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0].role == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == \"user\" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n {%- if message.content is string %}\n {%- set content = message.content %}\n {%- else %}\n {%- set content = '' %}\n {%- endif %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {%- set reasoning_content = '' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if '' in content %}\n {%- set reasoning_content = content.split('')[0].rstrip('\\n').split('')[-1].lstrip('\\n') %}\n {%- set content = content.split('')[-1].lstrip('\\n') %}\n {%- endif %}\n {%- endif %}\n {%- if loop.index0 > ns.last_query_index %}\n {%- if loop.last or (not loop.last and reasoning_content) %}\n {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- if message.tool_calls %}\n {%- for tool_call in message.tool_calls %}\n {%- if (loop.first and content) or (not loop.first) %}\n {{- '\\n' }}\n {%- endif %}\n {%- if tool_call.function %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {%- if tool_call.arguments is string %}\n {{- tool_call.arguments }}\n {%- else %}\n {{- tool_call.arguments | tojson }}\n {%- endif %}\n {{- '}\\n' }}\n {%- endfor %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n\\n' }}\n {{- content }}\n {{- '\\n' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- if enable_thinking is defined and enable_thinking is false %}\n {{- '\\n\\n\\n\\n' }}\n {%- endif %}\n{%- endif %}" }, { "path": "tests/distributed/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n" }, { "path": "tests/distributed/data/accelerate_configs/ddp.yaml", "content": "distributed_type: MULTI_GPU\nnum_processes: 2" }, { "path": "tests/distributed/data/accelerate_configs/fsdp2.yaml", "content": "distributed_type: FSDP\nfsdp_config:\n fsdp_version: 2\nnum_processes: 2" }, { "path": "tests/distributed/data/accelerate_configs/zero2.yaml", "content": "distributed_type: DEEPSPEED\ndeepspeed_config:\n zero_stage: 2\nnum_processes: 2" }, { "path": "tests/distributed/data/accelerate_configs/zero3.yaml", "content": "distributed_type: DEEPSPEED\ndeepspeed_config:\n zero_stage: 3\nnum_processes: 2" }, { "path": "tests/distributed/test_distributed.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport os\nimport subprocess\nfrom pathlib import Path\n\nimport pytest\nimport torch\nimport transformers\nfrom packaging.version import Version\n\nfrom ..testing_utils import TrlTestCase, require_torch_multi_accelerator\n\n\nROOT = Path(__file__).resolve().parents[2]\n\n\ndef run_command(command: list[str], env: dict[str, str]) -> None:\n result = subprocess.run(command, env=env, cwd=ROOT)\n assert result.returncode == 0\n\n\n@pytest.fixture\ndef get_config_path(lazy_shared_datadir):\n def _get_config_path(config_name):\n return lazy_shared_datadir / \"accelerate_configs\" / f\"{config_name}.yaml\"\n\n return _get_config_path\n\n\n@require_torch_multi_accelerator\nclass TestDistributed(\n TrlTestCase\n): # pytest.param(\"zero3\", marks=pytest.mark.xfail(reason=\"ZeRO 3 is currently failing, see #4899\"))\n @pytest.mark.parametrize(\n \"config\",\n [\n \"ddp\",\n pytest.param(\n \"zero2\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n pytest.param(\n \"zero3\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n \"fsdp2\",\n ],\n )\n def test_sft(self, config, get_config_path):\n # fmt: off\n run_command(\n [\n \"accelerate\", \"launch\", \"--config_file\", get_config_path(config), \"trl/scripts/sft.py\",\n \"--output_dir\", self.tmp_dir,\n \"--model_name_or_path\", \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"--dataset_name\", \"trl-internal-testing/zen\",\n \"--dataset_config\", \"standard_language_modeling\",\n ],\n os.environ.copy(),\n )\n # fmt: on\n\n @pytest.mark.parametrize(\n \"config\",\n [\n \"ddp\",\n pytest.param(\n \"zero2\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n pytest.param(\n \"zero3\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n \"fsdp2\",\n ],\n )\n def test_dpo(self, config, get_config_path):\n # fmt: off\n run_command(\n [\n \"accelerate\", \"launch\", \"--config_file\", get_config_path(config), \"trl/scripts/dpo.py\",\n \"--output_dir\", self.tmp_dir,\n \"--model_name_or_path\", \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"--dataset_name\", \"trl-internal-testing/zen\",\n \"--dataset_config\", \"standard_preference\",\n ],\n os.environ.copy(),\n )\n # fmt: on\n\n @pytest.mark.parametrize(\n \"config\",\n [\n \"ddp\",\n pytest.param(\n \"zero2\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n pytest.param(\n \"zero3\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n \"fsdp2\",\n ],\n )\n def test_sft_dataset_streaming(self, config, get_config_path):\n # fmt: off\n run_command(\n [\n \"accelerate\", \"launch\", \"--config_file\", get_config_path(config), \"trl/scripts/sft.py\",\n \"--output_dir\", self.tmp_dir,\n \"--model_name_or_path\", \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"--dataset_name\", \"trl-internal-testing/zen\",\n \"--dataset_config\", \"standard_language_modeling\",\n \"--dataset_streaming\",\n \"--max_steps\", \"3\",\n ],\n os.environ.copy(),\n )\n # fmt: on\n\n @pytest.mark.parametrize(\n \"config\",\n [\n \"ddp\",\n pytest.param(\n \"zero2\",\n marks=pytest.mark.xfail(\n condition=Version(\"2.10\") <= Version(torch.__version__),\n reason=\"ZeRO 2 + PEFT is failing on torch 2.10; see #4884\",\n ),\n ),\n pytest.param(\n \"zero3\",\n marks=pytest.mark.xfail(\n condition=Version(\"2.10\") <= Version(torch.__version__),\n reason=\"ZeRO 3 + PEFT is failing on torch 2.10; see #4884\",\n ),\n ),\n \"fsdp2\",\n ],\n )\n def test_sft_peft(self, config, get_config_path):\n # fmt: off\n run_command(\n [\n \"accelerate\", \"launch\", \"--config_file\", get_config_path(config), \"trl/scripts/sft.py\",\n \"--output_dir\", self.tmp_dir,\n \"--model_name_or_path\", \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"--dataset_name\", \"trl-internal-testing/zen\",\n \"--dataset_config\", \"standard_language_modeling\",\n \"--use_peft\",\n ],\n os.environ.copy(),\n )\n # fmt: on\n\n @pytest.mark.parametrize(\n \"config\",\n [\n \"ddp\",\n pytest.param(\n \"zero2\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n pytest.param(\n \"zero3\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n \"fsdp2\",\n ],\n )\n def test_reward(self, config, get_config_path):\n # fmt: off\n run_command(\n [\n \"accelerate\", \"launch\", \"--config_file\", get_config_path(config), \"trl/scripts/reward.py\",\n \"--output_dir\", self.tmp_dir,\n \"--model_name_or_path\", \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"--dataset_name\", \"trl-internal-testing/zen\",\n \"--dataset_config\", \"conversational_implicit_prompt_preference\",\n ],\n os.environ.copy(),\n )\n # fmt: on\n\n @pytest.mark.parametrize(\n \"config\",\n [\n \"ddp\",\n pytest.param(\n \"zero2\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n pytest.param(\"zero3\", marks=pytest.mark.xfail(reason=\"ZeRO 3 is currently failing, see #4899\")),\n \"fsdp2\",\n ],\n )\n def test_rloo(self, config, get_config_path):\n # fmt: off\n run_command(\n [\n \"accelerate\", \"launch\", \"--config_file\", get_config_path(config), \"trl/scripts/rloo.py\",\n \"--output_dir\", self.tmp_dir,\n \"--model_name_or_path\", \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"--dataset_name\", \"trl-internal-testing/zen\",\n \"--dataset_config\", \"conversational_prompt_only\",\n \"--reward_model_name_or_path\", \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n ],\n os.environ.copy(),\n )\n # fmt: on\n\n @pytest.mark.parametrize(\n \"config\",\n [\n \"ddp\",\n pytest.param(\n \"zero2\",\n marks=pytest.mark.xfail(\n Version(transformers.__version__) == Version(\"5.1.0\"),\n reason=\"Upstream incompatibility: deepspeed and transformers==5.1.0 (see transformers#43780)\",\n ),\n ),\n pytest.param(\"zero3\", marks=pytest.mark.xfail(reason=\"ZeRO 3 is currently failing, see #4899\")),\n \"fsdp2\",\n ],\n )\n def test_grpo(self, config, get_config_path):\n # fmt: off\n run_command(\n [\n \"accelerate\", \"launch\", \"--config_file\", get_config_path(config), \"trl/scripts/grpo.py\",\n \"--output_dir\", self.tmp_dir,\n \"--model_name_or_path\", \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"--dataset_name\", \"trl-internal-testing/zen\",\n \"--dataset_config\", \"conversational_prompt_only\",\n \"--reward_model_name_or_path\", \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n ],\n os.environ.copy(),\n )\n # fmt: on\n" }, { "path": "tests/experimental/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n" }, { "path": "tests/experimental/test_async_grpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport itertools\nimport queue\n\nimport numpy as np\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoTokenizer\n\nfrom trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer\nfrom trl.experimental.async_grpo.async_rollout_worker import RolloutSample\n\nfrom ..testing_utils import TrlTestCase\n\n\ndef dummy_reward_func(completions, **kwargs):\n return [float(hash(c[0][\"content\"]) % 100) / 100.0 for c in completions]\n\n\nclass _StubRolloutWorker:\n \"\"\"Minimal rollout worker stub for testing the trainer in isolation.\"\"\"\n\n def __init__(self, tokenizer, dataset, num_generations: int = 8, samples_per_weight_sync: int = 10):\n self.rollout_buffer = queue.Queue()\n self._samples_per_weight_sync = samples_per_weight_sync\n self._model_version = 0\n self._sample_iter = self._make_sample_iter(tokenizer, dataset, num_generations)\n\n def _make_sample_iter(self, tokenizer, dataset, num_generations):\n for row in itertools.cycle(dataset):\n completions = [\n [{\"role\": \"assistant\", \"content\": f\"{row['completion'][0]['content']} {idx}\"}]\n for idx in range(num_generations)\n ]\n prompt_completions = [row[\"prompt\"] + completion for completion in completions]\n prompt_ids = tokenizer.apply_chat_template(\n row[\"prompt\"], tokenize=True, add_generation_prompt=True, return_dict=False\n )\n prompt_completion_ids = tokenizer.apply_chat_template(\n prompt_completions, tokenize=True, add_generation_prompt=False, return_dict=False\n )\n rewards = np.array(dummy_reward_func(completions))\n advantages = (rewards - rewards.mean()) / rewards.std()\n for idx in range(num_generations):\n completion_ids = prompt_completion_ids[idx][len(prompt_ids) :]\n yield RolloutSample(\n prompt=row[\"prompt\"],\n completion=completions[idx],\n input_ids=prompt_ids + completion_ids,\n completion_mask=[0] * len(prompt_ids) + [1] * len(completion_ids),\n old_log_probs=[0.0] * len(prompt_ids) + [-0.5] * len(completion_ids),\n advantage=float(advantages[idx]),\n model_version=self._model_version,\n metrics={\"reward\": float(rewards[idx]), \"reward_std\": float(rewards.std())},\n )\n\n def _fill_queue(self):\n for _ in range(self._samples_per_weight_sync):\n self.rollout_buffer.put(next(self._sample_iter))\n\n def start(self):\n self._fill_queue()\n\n def update_model_version(self, version):\n self._model_version = version\n self._fill_queue()\n\n def stop(self):\n pass\n\n def pause(self):\n pass\n\n def resume(self):\n pass\n\n def send_weights(self, iterator):\n pass\n\n\nclass TestAsyncGRPOTrainer(TrlTestCase):\n def test_init_minimal(self):\n # Test that AsyncGRPOTrainer can be instantiated with only model, reward_model and train_dataset\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_completion\", split=\"train\")\n AsyncGRPOTrainer(\n model=model_id,\n reward_funcs=dummy_reward_func,\n train_dataset=dataset,\n rollout_worker=_StubRolloutWorker(AutoTokenizer.from_pretrained(model_id), dataset, num_generations=3),\n )\n\n def test_training(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_completion\", split=\"train\")\n\n training_args = AsyncGRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n vllm_server_timeout=5.0, # short timeout so test fails fast if queue runs dry\n report_to=\"none\",\n )\n trainer = AsyncGRPOTrainer(\n model=model_id,\n reward_funcs=dummy_reward_func, # unused: the stub pre-computes rewards, but the trainer requires this argument\n args=training_args,\n train_dataset=dataset,\n rollout_worker=_StubRolloutWorker(AutoTokenizer.from_pretrained(model_id), dataset, num_generations=3),\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n" }, { "path": "tests/experimental/test_bco_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom functools import partial\n\nimport pytest\nimport torch\nfrom accelerate import Accelerator\nfrom datasets import load_dataset\nfrom transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer\nfrom transformers.utils import is_peft_available\n\nfrom trl.experimental.bco import BCOConfig, BCOTrainer\nfrom trl.experimental.bco.bco_trainer import _process_tokens, _tokenize\n\nfrom ..testing_utils import TrlTestCase, require_no_wandb, require_peft, require_sklearn\n\n\nif is_peft_available():\n from peft import LoraConfig\n\n\n@pytest.mark.low_priority\nclass TestBCOTrainer(TrlTestCase):\n @pytest.mark.parametrize(\n \"config_name\",\n [\n \"standard_preference\",\n \"standard_implicit_prompt_preference\",\n \"standard_unpaired_preference\",\n \"conversational_preference\",\n \"conversational_implicit_prompt_preference\",\n \"conversational_unpaired_preference\",\n ],\n )\n @require_sklearn\n def test_train(self, config_name):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n ref_model = AutoModelForCausalLM.from_pretrained(model_id)\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", config_name, split=\"train\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n\n trainer = BCOTrainer(\n model=model,\n ref_model=ref_model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param.cpu(), new_param.cpu())\n\n @require_sklearn\n def test_train_with_precompute(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n ref_model = AutoModelForCausalLM.from_pretrained(model_id)\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\", split=\"train\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n precompute_ref_log_probs=True,\n report_to=\"none\",\n )\n\n trainer = BCOTrainer(\n model=model,\n ref_model=ref_model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param.cpu(), new_param.cpu())\n\n @require_sklearn\n def test_train_eval(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n ref_model = AutoModelForCausalLM.from_pretrained(model_id)\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n eval_strategy=\"steps\",\n eval_steps=3,\n report_to=\"none\",\n )\n\n trainer = BCOTrainer(\n model=model,\n ref_model=ref_model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n trainer.train()\n\n @require_sklearn\n def test_init_with_ref_model_is_model(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\", split=\"train\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n report_to=\"none\",\n )\n\n with pytest.raises(ValueError):\n BCOTrainer(\n model=model,\n ref_model=model, # ref_model can't be the same as model\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset,\n )\n\n @require_sklearn\n def test_tokenize_and_process_tokens(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n ref_model = AutoModelForCausalLM.from_pretrained(model_id)\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\", split=\"train\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n report_to=\"none\",\n )\n\n trainer = BCOTrainer(\n model=model,\n ref_model=ref_model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset,\n )\n\n tokenized_dataset = dataset.map(\n _tokenize,\n fn_kwargs={\"tokenizer\": trainer.processing_class},\n batched=True,\n batch_size=2,\n )\n assert tokenized_dataset[\"prompt\"][:] == dataset[\"prompt\"][:]\n assert tokenized_dataset[\"completion\"][:] == dataset[\"completion\"][:]\n assert tokenized_dataset[\"label\"][:] == dataset[\"label\"][:]\n assert tokenized_dataset[\"prompt_input_ids\"][0] == [46518, 374, 2664, 1091]\n assert tokenized_dataset[\"prompt_attention_mask\"][0] == [1, 1, 1, 1]\n assert tokenized_dataset[\"answer_input_ids\"][0] == [27261, 13]\n assert tokenized_dataset[\"answer_attention_mask\"][0] == [1, 1]\n\n fn_kwargs = {\n \"prefix\": \"\",\n \"is_encoder_decoder\": trainer.is_encoder_decoder,\n \"tokenizer\": trainer.processing_class,\n \"max_length\": trainer.max_length,\n \"truncation_mode\": trainer.truncation_mode,\n }\n processed_dataset = tokenized_dataset.map(_process_tokens, fn_kwargs=fn_kwargs)\n assert processed_dataset[\"prompt\"][:] == dataset[\"prompt\"][:]\n assert processed_dataset[\"completion\"][:] == dataset[\"completion\"][:]\n assert processed_dataset[\"label\"][:] == dataset[\"label\"][:]\n assert processed_dataset[\"prompt_input_ids\"][0] == [46518, 374, 2664, 1091]\n assert processed_dataset[\"prompt_attention_mask\"][0] == [1, 1, 1, 1]\n assert processed_dataset[\"completion_input_ids\"][0] == [46518, 374, 2664, 1091, 27261, 13, 151645]\n assert processed_dataset[\"completion_attention_mask\"][0] == [1, 1, 1, 1, 1, 1, 1]\n assert processed_dataset[\"completion_labels\"][0] == [-100, -100, -100, -100, 27261, 13, 151645]\n\n @require_sklearn\n def test_train_without_providing_ref_model(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\", split=\"train\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n\n trainer = BCOTrainer(\n model=model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param.cpu(), new_param.cpu())\n\n @require_sklearn\n def test_train_udm(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n # Get embedding model\n embedding_model_id = \"trl-internal-testing/tiny-BartModel\"\n embedding_model = AutoModel.from_pretrained(embedding_model_id)\n embedding_tokenizer = AutoTokenizer.from_pretrained(embedding_model_id)\n\n def embed_prompt(input_ids, attention_mask, model):\n outputs = model(input_ids=input_ids, attention_mask=attention_mask)\n\n return outputs.last_hidden_state.mean(dim=1)\n\n embedding_model = Accelerator().prepare_model(embedding_model)\n embedding_func = partial(embed_prompt, model=embedding_model)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\", split=\"train\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n\n trainer = BCOTrainer(\n model=model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset,\n embedding_func=embedding_func,\n embedding_tokenizer=embedding_tokenizer,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param.cpu(), new_param.cpu())\n\n @require_sklearn\n @require_peft\n def test_train_without_providing_ref_model_with_lora(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type=\"CAUSAL_LM\")\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\", split=\"train\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n\n trainer = BCOTrainer(\n model=model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset,\n peft_config=lora_config,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n if \"lora\" in n:\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param.cpu(), new_param.cpu())\n\n @require_sklearn\n @require_no_wandb\n def test_generate_during_eval_no_wandb(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n eval_strategy=\"steps\",\n eval_steps=3,\n generate_during_eval=True,\n report_to=\"none\",\n )\n\n with pytest.raises(\n ValueError,\n match=\"`generate_during_eval=True` requires Weights and Biases or Comet to be installed.\"\n \" Please install `wandb` or `comet-ml` to resolve.\",\n ):\n BCOTrainer(\n model=model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n @require_sklearn\n @require_peft\n def test_lora_train_and_save(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type=\"CAUSAL_LM\")\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n report_to=\"none\",\n )\n\n trainer = BCOTrainer(\n model=model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset[\"train\"],\n peft_config=lora_config,\n )\n\n # train the model\n trainer.train()\n\n # save peft adapter\n trainer.save_model()\n\n # assert that the model is loaded without giving OSError\n AutoModelForCausalLM.from_pretrained(self.tmp_dir)\n\n @require_sklearn\n def test_compute_metrics(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n ref_model = AutoModelForCausalLM.from_pretrained(model_id)\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n def dummy_compute_metrics(*args, **kwargs):\n return {\"test\": 0.0}\n\n training_args = BCOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False, # warning raised if not set to False\n eval_strategy=\"steps\",\n eval_steps=3,\n report_to=\"none\",\n )\n\n trainer = BCOTrainer(\n model=model,\n ref_model=ref_model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n compute_metrics=dummy_compute_metrics,\n )\n\n trainer.train()\n\n assert trainer.state.log_history[-2][\"eval_test\"] == 0.0\n" }, { "path": "tests/experimental/test_cpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pytest\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoModelForSeq2SeqLM, AutoTokenizer\n\nfrom trl.experimental.cpo import CPOConfig, CPOTrainer\n\nfrom ..testing_utils import TrlTestCase, require_peft\n\n\nclass TestCPOTrainer(TrlTestCase):\n def setup_method(self):\n self.model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.model = AutoModelForCausalLM.from_pretrained(self.model_id, dtype=\"float32\")\n self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n self.tokenizer.pad_token = self.tokenizer.eos_token\n\n # get t5 as seq2seq example:\n model_id = \"trl-internal-testing/tiny-T5ForConditionalGeneration\"\n self.t5_model = AutoModelForSeq2SeqLM.from_pretrained(model_id, dtype=\"float32\")\n self.t5_tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n @pytest.mark.parametrize(\n \"name, loss_type, config_name\",\n [\n (\"qwen\", \"sigmoid\", \"standard_preference\"),\n (\"t5\", \"hinge\", \"standard_implicit_prompt_preference\"),\n (\"qwen\", \"ipo\", \"conversational_preference\"),\n (\"qwen\", \"simpo\", \"standard_preference\"),\n (\"t5\", \"simpo\", \"standard_implicit_prompt_preference\"),\n (\"qwen\", \"hinge\", \"conversational_preference\"),\n ],\n )\n def test_cpo_trainer(self, name, loss_type, config_name):\n training_args = CPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n beta=0.1,\n loss_type=loss_type,\n cpo_alpha=1.0,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n if name == \"qwen\":\n model = self.model\n tokenizer = self.tokenizer\n elif name == \"t5\":\n model = self.t5_model\n tokenizer = self.t5_tokenizer\n training_args.is_encoder_decoder = True\n\n trainer = CPOTrainer(\n model=model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param, new_param)\n\n @pytest.mark.parametrize(\n \"config_name\",\n [\n \"standard_preference\",\n \"standard_implicit_prompt_preference\",\n \"conversational_preference\",\n \"conversational_implicit_prompt_preference\",\n ],\n )\n @require_peft\n def test_cpo_trainer_with_lora(self, config_name):\n from peft import LoraConfig\n\n lora_config = LoraConfig(\n r=16,\n lora_alpha=32,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n )\n\n training_args = CPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=4,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n beta=0.1,\n cpo_alpha=1.0,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = CPOTrainer(\n model=self.model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n peft_config=lora_config,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n if \"lora\" in n:\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param, new_param)\n\n def test_compute_metrics(self):\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\")\n\n def dummy_compute_metrics(*args, **kwargs):\n return {\"test\": 0.0}\n\n training_args = CPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n remove_unused_columns=False,\n do_eval=True,\n eval_strategy=\"steps\",\n eval_steps=1,\n per_device_eval_batch_size=2,\n report_to=\"none\",\n )\n\n trainer = CPOTrainer(\n model=self.model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n compute_metrics=dummy_compute_metrics,\n )\n\n trainer.train()\n\n assert trainer.state.log_history[-2][\"eval_test\"] == 0.0\n\n def test_alphapo_trainer(self):\n training_args = CPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n beta=0.1,\n loss_type=\"alphapo\",\n alpha=0.5,\n simpo_gamma=0.5,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\")\n\n trainer = CPOTrainer(\n model=self.model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0:\n assert not torch.equal(param, new_param)\n" }, { "path": "tests/experimental/test_dppo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pytest\nimport torch\nfrom datasets import load_dataset\n\nfrom trl.experimental.dppo import DPPOConfig, DPPOTrainer\n\nfrom ..testing_utils import TrlTestCase\n\n\nclass TestDPPODivergenceMask:\n \"\"\"Unit tests for _compute_divergence_mask with synthetic inputs.\"\"\"\n\n @staticmethod\n def make_trainer(divergence_type=\"binary_tv\", epsilon=0.2, epsilon_high=0.28):\n \"\"\"Create a minimal DPPOTrainer-like object with just the attributes needed for _compute_divergence_mask.\"\"\"\n\n class Stub:\n pass\n\n stub = Stub()\n stub.divergence_type = divergence_type\n stub.epsilon_low = epsilon\n stub.epsilon_high = epsilon_high\n return stub\n\n @staticmethod\n def compute_divergence_mask(\n trainer_stub,\n current_logps,\n sampling_logps,\n advantages,\n completion_mask,\n current_topk_logps=None,\n sampling_topk_logps=None,\n ):\n return DPPOTrainer._compute_divergence_mask(\n trainer_stub,\n current_logps,\n sampling_logps,\n advantages,\n completion_mask,\n current_topk_logps=current_topk_logps,\n sampling_topk_logps=sampling_topk_logps,\n )\n\n def test_binary_tv_no_masking_within_threshold(self):\n stub = self.make_trainer(\"binary_tv\", epsilon=0.2, epsilon_high=0.28)\n # Policies are very close โ€” no tokens should be masked\n sampling_logps = torch.log(torch.tensor([[0.5, 0.3, 0.7]]))\n current_logps = torch.log(torch.tensor([[0.51, 0.29, 0.71]]))\n advantages = torch.tensor([[1.0]])\n completion_mask = torch.ones(1, 3)\n\n mask = self.compute_divergence_mask(stub, current_logps, sampling_logps, advantages, completion_mask)\n assert mask.shape == (1, 3)\n assert (mask == 1.0).all()\n\n def test_binary_tv_masks_positive_advantage_high_divergence(self):\n stub = self.make_trainer(\"binary_tv\", epsilon=0.01, epsilon_high=0.01)\n # ฯ€ much higher than ฮผ, positive advantage โ†’ should be masked (invalid_pos)\n sampling_logps = torch.log(torch.tensor([[0.1]]))\n current_logps = torch.log(torch.tensor([[0.5]]))\n advantages = torch.tensor([[1.0]])\n completion_mask = torch.ones(1, 1)\n\n mask = self.compute_divergence_mask(stub, current_logps, sampling_logps, advantages, completion_mask)\n assert mask.item() == 0.0\n\n def test_binary_tv_masks_negative_advantage_low_divergence(self):\n stub = self.make_trainer(\"binary_tv\", epsilon=0.01, epsilon_high=0.01)\n # ฯ€ much lower than ฮผ, negative advantage โ†’ should be masked (invalid_neg)\n sampling_logps = torch.log(torch.tensor([[0.5]]))\n current_logps = torch.log(torch.tensor([[0.1]]))\n advantages = torch.tensor([[-1.0]])\n completion_mask = torch.ones(1, 1)\n\n mask = self.compute_divergence_mask(stub, current_logps, sampling_logps, advantages, completion_mask)\n assert mask.item() == 0.0\n\n def test_binary_tv_respects_completion_mask(self):\n stub = self.make_trainer(\"binary_tv\", epsilon=0.01, epsilon_high=0.01)\n # Even though divergence is huge, padding tokens stay 0\n sampling_logps = torch.log(torch.tensor([[0.1, 0.5]]))\n current_logps = torch.log(torch.tensor([[0.9, 0.9]]))\n advantages = torch.tensor([[1.0]])\n completion_mask = torch.tensor([[1.0, 0.0]])\n\n mask = self.compute_divergence_mask(stub, current_logps, sampling_logps, advantages, completion_mask)\n assert mask[0, 1].item() == 0.0\n\n def test_topk_tv_requires_topk_inputs(self):\n stub = self.make_trainer(\"topk_tv\")\n B, T, K = 1, 2, 4\n sampling_logps = torch.log(torch.full((B, T), 0.3))\n current_logps = torch.log(torch.full((B, T), 0.31))\n advantages = torch.tensor([[1.0]])\n completion_mask = torch.ones(B, T)\n\n # Build top-K distributions that are nearly identical\n topk_probs = torch.softmax(torch.randn(B, T, K), dim=-1)\n sampling_topk_logps = torch.log(topk_probs)\n current_topk_logps = torch.log(topk_probs + 0.001)\n\n mask = self.compute_divergence_mask(\n stub,\n current_logps,\n sampling_logps,\n advantages,\n completion_mask,\n current_topk_logps=current_topk_logps,\n sampling_topk_logps=sampling_topk_logps,\n )\n assert mask.shape == (B, T)\n assert (mask == 1.0).all()\n\n\n@pytest.mark.low_priority\nclass TestDPPOTrainer(TrlTestCase):\n @pytest.mark.parametrize(\"divergence_type\", [\"binary_tv\", \"binary_kl\"])\n def test_training_binary(self, divergence_type):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = DPPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1,\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n divergence_type=divergence_type,\n report_to=\"none\",\n )\n trainer = DPPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n def test_training_conversational(self, config_name):\n dataset = load_dataset(\"trl-internal-testing/zen\", config_name, split=\"train\")\n\n training_args = DPPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1,\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n )\n trainer = DPPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n" }, { "path": "tests/experimental/test_gkd_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport os\n\nimport pytest\nimport torch\nimport torch.nn.functional as F\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig\n\nfrom trl.experimental.gkd import GKDConfig, GKDTrainer\n\nfrom ..testing_utils import TrlTestCase, require_liger_kernel\n\n\nclass TestGKDTrainerGenerateOnPolicy(TrlTestCase):\n @classmethod\n def setup_class(cls):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n cls.device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n cls.tokenizer = AutoTokenizer.from_pretrained(model_id)\n cls.tokenizer.pad_token = cls.tokenizer.eos_token\n cls.model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\").to(cls.device)\n cls.generation_config = GenerationConfig(\n max_new_tokens=20,\n num_return_sequences=1,\n pad_token_id=cls.tokenizer.pad_token_id,\n eos_token_id=cls.tokenizer.eos_token_id,\n )\n\n def test_generate_on_policy_outputs_deterministic(self):\n prompts = [\"Hello, how are you?\", \"What's the weather like today?\"]\n tokenized_prompts = self.tokenizer(prompts, return_tensors=\"pt\", padding=True)\n\n inputs = {\n \"prompts\": tokenized_prompts[\"input_ids\"].to(self.device),\n \"prompt_attention_mask\": tokenized_prompts[\"attention_mask\"].to(self.device),\n }\n\n # Set temperature to 0 for deterministic output\n deterministic_generation_config = GenerationConfig(\n max_new_tokens=30,\n num_return_sequences=1,\n pad_token_id=self.tokenizer.pad_token_id,\n eos_token_id=self.tokenizer.eos_token_id,\n do_sample=False,\n temperature=0.0,\n )\n\n outputs = GKDTrainer.generate_on_policy_outputs(\n self.model, inputs, deterministic_generation_config, self.tokenizer.pad_token_id\n )\n\n new_input_ids, new_attention_mask, new_labels = outputs\n\n # Decode the generated outputs\n generated_texts = self.tokenizer.batch_decode(new_input_ids, skip_special_tokens=True)\n\n # Check if the generated texts start with the original prompts\n for prompt, generated_text in zip(prompts, generated_texts, strict=True):\n assert generated_text.startswith(prompt), (\n f\"Generated text '{generated_text}' does not start with prompt '{prompt}'\"\n )\n\n # Run the generation twice and check if the outputs are identical\n outputs2 = GKDTrainer.generate_on_policy_outputs(\n self.model, inputs, deterministic_generation_config, self.tokenizer.pad_token_id\n )\n\n new_input_ids2, new_attention_mask2, new_labels2 = outputs2\n\n # Check if the two generations are identical\n assert torch.all(new_input_ids.eq(new_input_ids2)), \"Deterministic generations are not identical\"\n assert torch.all(new_attention_mask.eq(new_attention_mask2)), (\n \"Attention masks for deterministic generations are not identical\"\n )\n assert torch.all(new_labels.eq(new_labels2)), \"Labels for deterministic generations are not identical\"\n\n def test_generate_on_policy_outputs(self):\n prompts = [\"Hello, how are you?\", \"What's the weather like today?\"]\n tokenized_prompts = self.tokenizer(prompts, return_tensors=\"pt\", padding=True)\n\n inputs = {\n \"prompts\": tokenized_prompts[\"input_ids\"].to(self.device),\n \"attention_mask\": tokenized_prompts[\"attention_mask\"].to(self.device),\n }\n\n outputs = GKDTrainer.generate_on_policy_outputs(\n self.model, inputs, self.generation_config, self.tokenizer.pad_token_id\n )\n\n # Check that outputs is a tuple of three tensors\n assert isinstance(outputs, tuple)\n assert len(outputs) == 3\n\n new_input_ids, new_attention_mask, new_labels = outputs\n\n # Check shapes\n batch_size = len(prompts)\n assert new_input_ids.shape[0] == batch_size\n assert new_attention_mask.shape[0] == batch_size\n assert new_labels.shape[0] == batch_size\n\n # Check types\n assert isinstance(new_input_ids, torch.Tensor)\n assert isinstance(new_attention_mask, torch.Tensor)\n assert isinstance(new_labels, torch.Tensor)\n\n # Check that new_input_ids and new_attention_mask have the same shape\n assert new_input_ids.shape == new_attention_mask.shape\n assert new_labels.shape == new_attention_mask.shape\n\n\nclass TestGeneralizedJSDLoss(TrlTestCase):\n def setup_method(self):\n self.batch_size = 2\n self.seq_length = 3\n self.vocab_size = 5\n self.student_logits = torch.randn(self.batch_size, self.seq_length, self.vocab_size)\n self.teacher_logits = torch.randn(self.batch_size, self.seq_length, self.vocab_size)\n\n def test_uniform_distribution(self):\n logits = torch.ones(1, 1, self.vocab_size)\n loss = GKDTrainer.generalized_jsd_loss(logits, logits)\n assert round(abs(loss.item() - 0), 5) == 0\n\n def test_generalized_jsd_loss_edge_cases(self):\n # Setup\n student_logits = torch.log(torch.tensor([[0.1, 0.9]])).unsqueeze(0)\n teacher_logits = torch.log(torch.tensor([[0.9, 0.1]])).unsqueeze(0)\n\n # Case 1: beta = 1 (should be equivalent to KL(student || teacher))\n loss_beta_1 = GKDTrainer.generalized_jsd_loss(student_logits, teacher_logits, beta=1)\n expected_loss_beta_1 = F.kl_div(\n F.log_softmax(teacher_logits, dim=-1), F.softmax(student_logits, dim=-1), reduction=\"batchmean\"\n )\n assert round(abs(loss_beta_1.item() - expected_loss_beta_1.item()), 5) == 0\n\n # Case 2: beta = 0 (should be equivalent to KL(teacher || student))\n loss_beta_0 = GKDTrainer.generalized_jsd_loss(student_logits, teacher_logits, beta=0)\n expected_loss_beta_0 = F.kl_div(\n F.log_softmax(student_logits, dim=-1), F.softmax(teacher_logits, dim=-1), reduction=\"batchmean\"\n )\n assert round(abs(loss_beta_0.item() - expected_loss_beta_0.item()), 5) == 0\n\n def test_output_shape(self):\n loss = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits)\n assert torch.is_tensor(loss)\n assert loss.shape == torch.Size([])\n\n def test_beta_values(self):\n loss_beta_0 = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits, beta=0)\n loss_beta_1 = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits, beta=1)\n assert loss_beta_0 != loss_beta_1\n\n def test_temperature_scaling(self):\n loss_temp_1 = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits, temperature=1)\n loss_temp_2 = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits, temperature=2)\n assert loss_temp_1 != loss_temp_2\n\n def test_reduction_methods(self):\n loss_batchmean = GKDTrainer.generalized_jsd_loss(\n self.student_logits, self.teacher_logits, reduction=\"batchmean\"\n )\n loss_sum = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits, reduction=\"sum\")\n loss_mean = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits, reduction=\"mean\")\n loss_none = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits, reduction=\"none\")\n\n assert loss_batchmean.shape == torch.Size([])\n assert loss_sum.shape == torch.Size([])\n assert loss_mean.shape == torch.Size([])\n assert loss_none.shape == self.student_logits.shape\n\n def test_symmetry(self):\n student_teacher = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits, beta=0.1)\n teacher_student = GKDTrainer.generalized_jsd_loss(self.teacher_logits, self.student_logits, beta=0.1)\n assert student_teacher != teacher_student\n\n student_teacher = GKDTrainer.generalized_jsd_loss(self.student_logits, self.teacher_logits, beta=0.5)\n teacher_student = GKDTrainer.generalized_jsd_loss(self.teacher_logits, self.student_logits, beta=0.5)\n assert student_teacher == teacher_student\n\n def test_zero_loss_for_identical_inputs(self):\n identical_logits = torch.randn(self.batch_size, self.seq_length, self.vocab_size)\n loss = GKDTrainer.generalized_jsd_loss(identical_logits, identical_logits)\n assert round(abs(loss.item() - 0), 6) == 0\n\n\nclass TestGKDTrainer(TrlTestCase):\n def setup_method(self):\n self.model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.model = AutoModelForCausalLM.from_pretrained(self.model_id, dtype=\"float32\")\n self.teacher_model = AutoModelForCausalLM.from_pretrained(self.model_id)\n self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n self.tokenizer.pad_token = self.tokenizer.eos_token\n\n def test_gkd_trainer(self):\n training_args = GKDConfig(\n output_dir=self.tmp_dir,\n dataloader_drop_last=True,\n eval_strategy=\"steps\",\n max_steps=4,\n eval_steps=2,\n save_steps=2,\n per_device_train_batch_size=2,\n per_device_eval_batch_size=2,\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\")\n\n trainer = GKDTrainer(\n model=self.model_id,\n teacher_model=self.model_id,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n\n trainer.train()\n\n assert trainer.state.log_history[(-1)][\"train_loss\"] is not None\n assert trainer.state.log_history[0][\"eval_loss\"] is not None\n assert \"model.safetensors\" in os.listdir(self.tmp_dir + \"/checkpoint-2\")\n\n @require_liger_kernel\n @pytest.mark.xfail(reason=\"Computing the Liger loss spikes GPU memory usage, causing the test to run OOM.\")\n def test_gkd_trainer_with_liger(self):\n training_args = GKDConfig(\n output_dir=self.tmp_dir,\n report_to=\"none\",\n use_liger_kernel=True,\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\")\n\n trainer = GKDTrainer(\n model=self.model_id,\n teacher_model=self.model_id,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n processing_class=self.tokenizer,\n )\n\n # Ensure liger fused JSD path is enabled; if not, skip (runtime may lack system libs)\n if not getattr(trainer, \"use_liger_gkd_loss\", False):\n pytest.skip(\"Liger fused JSD not enabled at runtime; skipping fused-loss assertion\")\n\n trainer.train()\n\n # Check we logged a train loss\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n def test_generation_config_init(self):\n training_args = GKDConfig(output_dir=self.tmp_dir)\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\")\n\n trainer = GKDTrainer(\n model=self.model_id,\n teacher_model=self.model_id,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n\n assert trainer.generation_config.pad_token_id == self.tokenizer.eos_token_id\n assert trainer.generation_config.eos_token_id == self.model.generation_config.eos_token_id\n assert trainer.generation_config.max_new_tokens == training_args.max_new_tokens\n assert trainer.generation_config.temperature == training_args.temperature\n assert trainer.generation_config.top_k == 0\n" }, { "path": "tests/experimental/test_gold_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom types import SimpleNamespace\n\nimport pytest\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoTokenizer\n\nfrom trl.experimental.gold.gold_trainer import GOLDTrainer, ULDLoss, build_teacher_inputs_from_texts\nfrom trl.experimental.utils import DataCollatorForChatML\n\n\n@pytest.fixture(scope=\"module\")\ndef openr1_examples():\n try:\n dataset = load_dataset(\n \"HuggingFaceTB/OpenR1-Math-220k-default-verified\",\n \"all\",\n split=\"train[:3]\",\n )\n except Exception as exc: # pragma: no cover - network/environment dependent\n pytest.skip(f\"OpenR1 dataset unavailable: {exc}\")\n return [{\"messages\": row[\"messages\"]} for row in dataset]\n\n\n@pytest.fixture(scope=\"module\")\ndef countdown_examples():\n try:\n dataset = load_dataset(\n \"HuggingFaceTB/Countdown-Tasks-3to4\",\n \"gkd_verified_Qwen2.5-7B-Instruct\",\n split=\"train[:3]\",\n )\n except Exception as exc: # pragma: no cover - network/environment dependent\n pytest.skip(f\"Countdown dataset unavailable: {exc}\")\n return [{\"messages\": row[\"messages\"]} for row in dataset]\n\n\ndef _teacher_inputs_from_collator(student_tok, teacher_tok, batch):\n prompt_texts = []\n completion_texts = []\n\n pad_token_id = student_tok.pad_token_id\n for prompt_ids_tensor, input_ids_tensor, labels_tensor in zip(\n batch[\"prompts\"], batch[\"input_ids\"], batch[\"labels\"], strict=True\n ):\n prompt_ids = prompt_ids_tensor.tolist()\n if pad_token_id is not None:\n prompt_ids = [tok for tok in prompt_ids if tok != pad_token_id]\n prompt_texts.append(student_tok.decode(prompt_ids, skip_special_tokens=False))\n\n input_ids = input_ids_tensor.tolist()\n labels = labels_tensor.tolist()\n completion_token_ids = [tok for tok, label in zip(input_ids, labels, strict=True) if label != -100]\n completion_texts.append(student_tok.decode(completion_token_ids, skip_special_tokens=False))\n\n teacher_input_ids, teacher_labels, _, _ = build_teacher_inputs_from_texts(\n teacher_tok, prompt_texts, completion_texts\n )\n return teacher_input_ids, teacher_labels, completion_texts\n\n\ndef _assert_alignment_covers_completion(loss_fn, batch, teacher_input_ids, teacher_labels):\n for idx in range(batch[\"input_ids\"].shape[0]):\n student_mask = batch[\"attention_mask\"][idx].bool()\n student_ids = batch[\"input_ids\"][idx][student_mask]\n student_labels = batch[\"labels\"][idx][student_mask]\n student_answer_ids = student_ids[student_labels != -100].tolist()\n\n teacher_answer_mask = teacher_labels[idx] != -100\n teacher_answer_ids = teacher_input_ids[idx][teacher_answer_mask].tolist()\n\n student_groups, teacher_groups = loss_fn._build_alignment_groups_from_ids(\n student_answer_ids, teacher_answer_ids\n )\n\n assert student_groups, \"Student alignment groups must not be empty\"\n assert teacher_groups, \"Teacher alignment groups must not be empty\"\n assert sorted(idx for group in student_groups for idx in group) == list(range(len(student_answer_ids)))\n assert sorted(idx for group in teacher_groups for idx in group) == list(range(len(teacher_answer_ids)))\n\n\n@pytest.mark.slow\ndef test_chatml_collator_preserves_completion_llama(llama_tokenizer, qwen_tokenizer, openr1_examples):\n collator = DataCollatorForChatML(tokenizer=llama_tokenizer, max_length=512)\n batch = collator(openr1_examples)\n\n assistant_texts = [example[\"messages\"][-1][\"content\"] for example in openr1_examples]\n decoded_batch = llama_tokenizer.batch_decode(batch[\"input_ids\"], skip_special_tokens=False)\n for decoded, assistant in zip(decoded_batch, assistant_texts, strict=True):\n assert assistant.strip() in decoded\n\n teacher_input_ids, teacher_labels, completion_texts = _teacher_inputs_from_collator(\n llama_tokenizer, qwen_tokenizer, batch\n )\n for completion, assistant in zip(completion_texts, assistant_texts, strict=True):\n assert assistant.strip() in completion\n assert completion.strip()\n\n config = build_config(\n uld_use_hybrid_loss=True,\n uld_hybrid_matched_weight=0.6,\n uld_hybrid_unmatched_weight=0.4,\n )\n loss_fn = ULDLoss(config, student_tokenizer=llama_tokenizer, teacher_tokenizer=qwen_tokenizer)\n\n _assert_alignment_covers_completion(loss_fn, batch, teacher_input_ids, teacher_labels)\n\n torch.manual_seed(0)\n student_vocab = len(llama_tokenizer)\n teacher_vocab = len(qwen_tokenizer)\n batch_size, seq_len = batch[\"input_ids\"].shape\n student_logits = torch.randn(batch_size, seq_len, student_vocab)\n teacher_logits = torch.randn(batch_size, teacher_input_ids.shape[1], teacher_vocab)\n\n loss = loss_fn(\n student_logits=student_logits,\n teacher_logits=teacher_logits,\n student_labels=batch[\"labels\"],\n teacher_labels=teacher_labels,\n student_input_ids=batch[\"input_ids\"],\n teacher_input_ids=teacher_input_ids,\n )\n\n assert torch.isfinite(loss)\n\n\n@pytest.mark.slow\ndef test_chatml_collator_preserves_completion_llama_countdown(llama_tokenizer, qwen_tokenizer, countdown_examples):\n collator = DataCollatorForChatML(tokenizer=llama_tokenizer, max_length=512)\n batch = collator(countdown_examples)\n\n assistant_texts = [example[\"messages\"][-1][\"content\"] for example in countdown_examples]\n decoded_batch = llama_tokenizer.batch_decode(batch[\"input_ids\"], skip_special_tokens=False)\n for decoded, assistant in zip(decoded_batch, assistant_texts, strict=True):\n assert assistant.strip() in decoded\n\n teacher_input_ids, teacher_labels, completion_texts = _teacher_inputs_from_collator(\n llama_tokenizer, qwen_tokenizer, batch\n )\n for completion, assistant in zip(completion_texts, assistant_texts, strict=True):\n assert assistant.strip() in completion\n assert completion.strip()\n\n config = build_config(\n uld_use_hybrid_loss=True,\n uld_hybrid_matched_weight=0.6,\n uld_hybrid_unmatched_weight=0.4,\n )\n loss_fn = ULDLoss(config, student_tokenizer=llama_tokenizer, teacher_tokenizer=qwen_tokenizer)\n\n _assert_alignment_covers_completion(loss_fn, batch, teacher_input_ids, teacher_labels)\n\n torch.manual_seed(2)\n student_vocab = len(llama_tokenizer)\n teacher_vocab = len(qwen_tokenizer)\n batch_size, seq_len = batch[\"input_ids\"].shape\n student_logits = torch.randn(batch_size, seq_len, student_vocab)\n teacher_logits = torch.randn(batch_size, teacher_input_ids.shape[1], teacher_vocab)\n\n loss = loss_fn(\n student_logits=student_logits,\n teacher_logits=teacher_logits,\n student_labels=batch[\"labels\"],\n teacher_labels=teacher_labels,\n student_input_ids=batch[\"input_ids\"],\n teacher_input_ids=teacher_input_ids,\n )\n\n assert torch.isfinite(loss)\n\n\n@pytest.mark.slow\ndef test_chatml_collator_preserves_completion_smollm(smollm_tokenizer, qwen_tokenizer, openr1_examples):\n collator = DataCollatorForChatML(tokenizer=smollm_tokenizer, max_length=512)\n batch = collator(openr1_examples)\n\n assistant_texts = [example[\"messages\"][-1][\"content\"] for example in openr1_examples]\n decoded_batch = smollm_tokenizer.batch_decode(batch[\"input_ids\"], skip_special_tokens=False)\n for decoded, assistant in zip(decoded_batch, assistant_texts, strict=True):\n assert assistant.strip() in decoded\n\n teacher_input_ids, teacher_labels, completion_texts = _teacher_inputs_from_collator(\n smollm_tokenizer, qwen_tokenizer, batch\n )\n for completion, assistant in zip(completion_texts, assistant_texts, strict=True):\n assert assistant.strip() in completion\n assert completion.strip()\n\n config = build_config(\n uld_use_hybrid_loss=True,\n uld_hybrid_matched_weight=0.5,\n uld_hybrid_unmatched_weight=0.5,\n )\n loss_fn = ULDLoss(config, student_tokenizer=smollm_tokenizer, teacher_tokenizer=qwen_tokenizer)\n\n _assert_alignment_covers_completion(loss_fn, batch, teacher_input_ids, teacher_labels)\n\n torch.manual_seed(1)\n student_vocab = len(smollm_tokenizer)\n teacher_vocab = len(qwen_tokenizer)\n batch_size, seq_len = batch[\"input_ids\"].shape\n student_logits = torch.randn(batch_size, seq_len, student_vocab)\n teacher_logits = torch.randn(batch_size, teacher_input_ids.shape[1], teacher_vocab)\n\n loss = loss_fn(\n student_logits=student_logits,\n teacher_logits=teacher_logits,\n student_labels=batch[\"labels\"],\n teacher_labels=teacher_labels,\n student_input_ids=batch[\"input_ids\"],\n teacher_input_ids=teacher_input_ids,\n )\n\n assert torch.isfinite(loss)\n\n\ndef build_config(**overrides):\n base = dict(\n uld_crossentropy_weight=0.0,\n uld_distillation_weight=1.0,\n uld_student_temperature=1.0,\n uld_teacher_temperature=1.0,\n uld_skip_student_eos=False,\n uld_skip_teacher_eos=False,\n use_extended_uld=True,\n uld_use_hybrid_loss=False,\n uld_hybrid_matched_weight=None,\n uld_hybrid_unmatched_weight=None,\n beta=0.5,\n )\n base.update(overrides)\n return SimpleNamespace(**base)\n\n\n@pytest.fixture(scope=\"session\")\ndef llama_tokenizer():\n tokenizer = AutoTokenizer.from_pretrained(\"TinyLlama/TinyLlama-1.1B-Chat-v1.0\")\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n return tokenizer\n\n\n@pytest.fixture(scope=\"session\")\ndef qwen_tokenizer():\n tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-0.5B-Instruct\")\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n return tokenizer\n\n\n@pytest.fixture(scope=\"session\")\ndef smollm_tokenizer():\n tokenizer = AutoTokenizer.from_pretrained(\"HuggingFaceTB/SmolLM3-3B\")\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n return tokenizer\n\n\ndef encode_prompt_completion(tokenizer, prompt, completion):\n prompt_ids = tokenizer(prompt, add_special_tokens=False)[\"input_ids\"]\n completion_ids = tokenizer(completion, add_special_tokens=False)[\"input_ids\"]\n eos_id = tokenizer.eos_token_id\n if eos_id is not None:\n completion_ids = completion_ids + [eos_id]\n input_ids = prompt_ids + completion_ids\n labels = [-100] * len(prompt_ids) + completion_ids\n return input_ids, labels\n\n\ndef pad_tokens(ids, pad_id, target_length):\n return ids + [pad_id] * (target_length - len(ids))\n\n\ndef pad_labels(labels, target_length):\n return labels + [-100] * (target_length - len(labels))\n\n\ndef test_process_completions_to_buffer_left_pads_prompt_retokenization():\n class DummyBatch:\n def __init__(self, input_ids):\n self.input_ids = input_ids\n\n def to(self, device):\n self.input_ids = self.input_ids.to(device)\n return self\n\n class RecordingTokenizer:\n pad_token_id = 0\n pad_token = \"\"\n\n def __init__(self):\n self.padding_side = \"right\"\n self.calls = []\n self._prompt_ids = {\n \"short\": [11],\n \"longer\": [21, 22],\n }\n\n def __call__(\n self,\n texts,\n return_tensors,\n padding,\n truncation,\n max_length,\n add_special_tokens,\n padding_side=None,\n ):\n assert return_tensors == \"pt\"\n assert padding == \"longest\"\n assert not truncation\n assert max_length is None\n assert not add_special_tokens\n self.calls.append(padding_side)\n\n side = padding_side or self.padding_side\n encoded = [torch.tensor(self._prompt_ids[text], dtype=torch.long) for text in texts]\n max_len = max(len(ids) for ids in encoded)\n\n padded = []\n for ids in encoded:\n pad_width = max_len - len(ids)\n if pad_width:\n pad = torch.full((pad_width,), self.pad_token_id, dtype=torch.long)\n ids = torch.cat([pad, ids]) if side == \"left\" else torch.cat([ids, pad])\n padded.append(ids)\n\n return DummyBatch(torch.stack(padded))\n\n def batch_decode(self, sequences, skip_special_tokens=False, clean_up_tokenization_spaces=False):\n del skip_special_tokens, clean_up_tokenization_spaces\n return [\" \".join(str(token) for token in sequence) for sequence in sequences]\n\n trainer = GOLDTrainer.__new__(GOLDTrainer)\n trainer.accelerator = SimpleNamespace(device=torch.device(\"cpu\"))\n trainer.processing_class = RecordingTokenizer()\n trainer.args = SimpleNamespace(max_length=None)\n trainer._buffered_inputs = [None]\n trainer._buffered_text_logs = [None]\n\n GOLDTrainer._process_completions_to_buffer(\n trainer,\n slices=[{\"slice\": \"original\"}],\n on_policy_indices=[0],\n local_slice_indices=[0, 0],\n completion_ids=[[31], [41]],\n prompts_text=[\"short\", \"longer\"],\n prompts_text_with_special=[\"short\", \"longer\"],\n max_completion_length=1,\n )\n\n buffered_inputs = trainer._buffered_inputs[0]\n assert trainer.processing_class.calls == [\"left\"]\n assert trainer.processing_class.padding_side == \"right\"\n assert torch.equal(buffered_inputs[\"input_ids\"], torch.tensor([[0, 11, 31], [21, 22, 41]], dtype=torch.long))\n assert torch.equal(buffered_inputs[\"attention_mask\"], torch.tensor([[0, 1, 1], [1, 1, 1]], dtype=torch.long))\n assert torch.equal(buffered_inputs[\"labels\"], torch.tensor([[-100, -100, 31], [-100, -100, 41]]))\n\n\ndef test_alignment_groups_cover_all_tokens(llama_tokenizer, qwen_tokenizer):\n config = build_config()\n loss = ULDLoss(config, student_tokenizer=llama_tokenizer, teacher_tokenizer=qwen_tokenizer)\n\n text = \"SmolLM3-3B is smaller than Llama 3.2 but still capable.\"\n student_ids = llama_tokenizer(text, add_special_tokens=False)[\"input_ids\"]\n teacher_ids = qwen_tokenizer(text, add_special_tokens=False)[\"input_ids\"]\n\n student_groups, teacher_groups = loss._build_alignment_groups_from_ids(student_ids, teacher_ids)\n\n assert len(student_groups) == len(teacher_groups)\n assert sorted(idx for group in student_groups for idx in group) == list(range(len(student_ids)))\n assert sorted(idx for group in teacher_groups for idx in group) == list(range(len(teacher_ids)))\n\n\ndef test_merge_probabilities_multiplies_split_tokens():\n config = build_config()\n # Use simple 3-token vocabulary to validate merging behaviour\n # probs[0] = P(token | context) at position 0 for all vocab tokens\n # probs[1] = P(token | context) at position 1 for all vocab tokens\n probs = torch.tensor([[0.6, 0.3, 0.1], [0.2, 0.5, 0.3]])\n loss = ULDLoss(config, student_tokenizer=None, teacher_tokenizer=None)\n\n # token_ids[1] = 1 means the actual token at position 1 is token ID 1\n # So we should extract P(token_id=1 | ...) = probs[1, 1] = 0.5\n token_ids = [0, 1] # Actual generated tokens\n\n merged = loss._merge_probabilities_with_alignment_groups(probs, [[0, 1]], token_ids=token_ids)\n\n # Expected: P_merged(y) = P(y | context_0) ร— P(token_1=1 | context_1)\n # For each vocab token y, multiply marginal prob at pos 0 by scalar conditional prob of actual token at pos 1\n expected = probs[0] * probs[1, 1] # probs[1, 1] = 0.5\n # Expected unnormalized: [0.6 * 0.5, 0.3 * 0.5, 0.1 * 0.5] = [0.3, 0.15, 0.05]\n\n torch.testing.assert_close(merged[0], expected)\n\n\ndef test_initialize_vocabulary_mapping_contains_common_tokens(llama_tokenizer, qwen_tokenizer):\n config = build_config(\n uld_use_hybrid_loss=True,\n uld_hybrid_matched_weight=1.0,\n uld_hybrid_unmatched_weight=0.0,\n )\n loss = ULDLoss(config, student_tokenizer=llama_tokenizer, teacher_tokenizer=qwen_tokenizer)\n\n common_tokens = [\"Hello\", \"world\", \"-\", \"ol\", \"LM\", \"3\", \"B\"]\n for token in common_tokens:\n student_id = llama_tokenizer.convert_tokens_to_ids(token)\n teacher_id = qwen_tokenizer.convert_tokens_to_ids(token)\n assert student_id is not None\n assert teacher_id is not None\n assert teacher_id in loss._vocab_mapping\n assert loss._vocab_mapping[teacher_id] == student_id\n assert teacher_id in loss._teacher_matched_ids\n assert student_id in loss._student_matched_ids\n\n\ndef test_get_start_and_size_answers_skips_prompt_tokens():\n trainer = ULDLoss.__new__(ULDLoss)\n trainer.ignore_index = -100\n\n answers = torch.tensor(\n [\n [-100, -100, -100, 10, 20, 30, -100, -100],\n [-100, 5, 6, 7, -100, -100, -100, -100],\n [-100, -100, -100, -100, -100, -100, -100, -100],\n ]\n )\n\n starts, sizes = trainer._get_start_and_size_answers(answers)\n\n assert starts == [3, 1, 0]\n assert sizes == [3, 3, 0]\n\n\n@pytest.mark.slow\ndef test_generate_on_policy_outputs_masks_prompt(llama_tokenizer):\n trainer = GOLDTrainer.__new__(GOLDTrainer)\n trainer.use_transformers_paged = False\n trainer.processing_class = llama_tokenizer\n\n prompt_text = \"<|begin_of_text|><|start_header_id|>user<|end_header_id|>\\nHello?<|eot_id|>\"\n completion_text = \"<|start_header_id|>assistant<|end_header_id|>\\nHi there!\"\n\n prompt_ids = llama_tokenizer(prompt_text, add_special_tokens=False)[\"input_ids\"]\n completion_ids = llama_tokenizer(completion_text, add_special_tokens=False)[\"input_ids\"]\n\n pad_id = llama_tokenizer.pad_token_id\n pad_width = 3\n prompt_tensor = torch.full((1, len(prompt_ids) + pad_width), pad_id, dtype=torch.long)\n prompt_tensor[0, pad_width:] = torch.tensor(prompt_ids, dtype=torch.long)\n prompt_mask = (prompt_tensor != pad_id).long()\n\n # model.generate() returns full sequences including left-padding from the input\n completion_tensor = torch.tensor(completion_ids, dtype=torch.long).unsqueeze(0)\n generated_sequence = torch.cat([prompt_tensor, completion_tensor], dim=1)\n\n class DummyModel:\n def generate(self, input_ids, attention_mask, generation_config, return_dict_in_generate):\n assert torch.equal(input_ids, prompt_tensor)\n assert torch.equal(attention_mask, prompt_mask)\n return SimpleNamespace(sequences=generated_sequence)\n\n generation_config = SimpleNamespace(max_completion_length=None, temperature=None, top_k=None, top_p=None)\n new_ids, new_mask, new_labels, prompt_texts, completion_texts = GOLDTrainer.generate_on_policy_outputs(\n trainer,\n DummyModel(),\n {\"prompts\": prompt_tensor, \"prompt_attention_mask\": prompt_mask},\n generation_config,\n pad_id,\n )\n\n assert torch.equal(new_ids, generated_sequence)\n if pad_id is not None:\n expected_mask = (generated_sequence != pad_id).long()\n assert torch.equal(new_mask, expected_mask)\n else:\n assert torch.all(new_mask == 1)\n\n padded_prompt_len = prompt_tensor.shape[1]\n assert torch.all(new_labels[0, :padded_prompt_len] == -100)\n assert torch.equal(new_labels[0, padded_prompt_len:], torch.tensor(completion_ids, dtype=torch.long))\n\n assert prompt_texts[0] == llama_tokenizer.decode(prompt_ids, skip_special_tokens=False)\n assert completion_texts[0] == llama_tokenizer.decode(completion_ids, skip_special_tokens=False)\n\n\n@pytest.mark.slow\ndef test_generate_on_policy_outputs_masks_prompt_smollm(smollm_tokenizer, openr1_examples):\n trainer = GOLDTrainer.__new__(GOLDTrainer)\n trainer.use_transformers_paged = False\n trainer.processing_class = smollm_tokenizer\n\n collator = DataCollatorForChatML(tokenizer=smollm_tokenizer)\n batch = collator([openr1_examples[0]])\n batch = {k: v.cpu() for k, v in batch.items()}\n\n class DummyModel:\n def generate(self, input_ids, attention_mask, generation_config, return_dict_in_generate):\n assert torch.equal(input_ids, batch[\"prompts\"])\n assert torch.equal(attention_mask, batch[\"prompt_attention_mask\"])\n return SimpleNamespace(sequences=batch[\"input_ids\"])\n\n generation_config = SimpleNamespace(max_completion_length=None, temperature=None, top_k=None, top_p=None)\n pad_id = smollm_tokenizer.pad_token_id\n new_ids, new_mask, new_labels, prompt_texts, completion_texts = GOLDTrainer.generate_on_policy_outputs(\n trainer,\n DummyModel(),\n {\"prompts\": batch[\"prompts\"], \"prompt_attention_mask\": batch[\"prompt_attention_mask\"]},\n generation_config,\n pad_id,\n )\n\n assert torch.equal(new_ids, batch[\"input_ids\"])\n if pad_id is not None:\n expected_mask = (batch[\"input_ids\"] != pad_id).long()\n assert torch.equal(new_mask, expected_mask)\n else:\n assert torch.all(new_mask == 1)\n\n prompt_len = int(batch[\"prompt_attention_mask\"].sum().item())\n tail_labels = new_labels[0, prompt_len:]\n expected_tail = batch[\"input_ids\"][0, prompt_len:]\n active_mask = tail_labels != -100\n assert torch.all(new_labels[0, :prompt_len] == -100)\n assert torch.equal(tail_labels[active_mask], expected_tail[active_mask])\n assert torch.all(tail_labels[~active_mask] == -100)\n\n prompt_tokens = batch[\"prompts\"][0, batch[\"prompt_attention_mask\"][0].bool()]\n decoded_prompt = smollm_tokenizer.decode(prompt_tokens.tolist(), skip_special_tokens=False)\n assert prompt_texts[0] == decoded_prompt\n\n assistant_completion = openr1_examples[0][\"messages\"][-1][\"content\"].strip()\n assert assistant_completion in completion_texts[0]\n\n\ndef test_generalized_jsd_loss_accepts_probability_inputs():\n student_probs = torch.tensor([[[0.6, 0.3, 0.1]]])\n teacher_probs = torch.tensor([[[0.5, 0.4, 0.1]]])\n mixture = 0.5 * (student_probs + teacher_probs)\n expected = 0.5 * (\n torch.sum(student_probs * (torch.log(student_probs) - torch.log(mixture)))\n + torch.sum(teacher_probs * (torch.log(teacher_probs) - torch.log(mixture)))\n )\n\n loss = GOLDTrainer.generalized_jsd_loss(\n student_probs,\n teacher_probs,\n beta=0.5,\n reduction=\"batchmean\",\n logits_are_probs=True,\n )\n\n torch.testing.assert_close(loss, expected)\n\n\ndef test_uldloss_handles_llama_student_qwen_teacher_sequence(llama_tokenizer, qwen_tokenizer):\n config = build_config(\n uld_use_hybrid_loss=True,\n uld_hybrid_matched_weight=0.6,\n uld_hybrid_unmatched_weight=0.4,\n )\n loss_fn = ULDLoss(config, student_tokenizer=llama_tokenizer, teacher_tokenizer=qwen_tokenizer)\n\n prompt = \"User: Summarize the difference between llamas and alpacas.\"\n completion = \"Assistant: Llamas are taller while alpacas have softer wool.\"\n\n student_ids, student_labels = encode_prompt_completion(llama_tokenizer, prompt, completion)\n teacher_ids, teacher_labels = encode_prompt_completion(qwen_tokenizer, prompt, completion)\n\n pad_id_student = llama_tokenizer.pad_token_id\n pad_id_teacher = qwen_tokenizer.pad_token_id\n max_length = max(len(student_ids), len(teacher_ids))\n\n student_ids = pad_tokens(student_ids, pad_id_student, max_length)\n teacher_ids = pad_tokens(teacher_ids, pad_id_teacher, max_length)\n student_labels = pad_labels(student_labels, max_length)\n teacher_labels = pad_labels(teacher_labels, max_length)\n\n student_input_ids = torch.tensor([student_ids])\n teacher_input_ids = torch.tensor([teacher_ids])\n student_labels = torch.tensor([student_labels])\n teacher_labels = torch.tensor([teacher_labels])\n\n student_vocab = len(llama_tokenizer)\n teacher_vocab = len(qwen_tokenizer)\n\n student_logits = torch.randn(1, max_length, student_vocab)\n teacher_logits = torch.randn(1, max_length, teacher_vocab)\n\n loss = loss_fn(\n student_logits=student_logits,\n teacher_logits=teacher_logits,\n student_labels=student_labels,\n teacher_labels=teacher_labels,\n student_input_ids=student_input_ids,\n teacher_input_ids=teacher_input_ids,\n )\n\n assert torch.isfinite(loss)\n assert loss.dim() == 0\n assert loss_fn.last_matched_loss is not None\n assert loss_fn.last_unmatched_loss is not None\n\n\ndef test_uldloss_handles_smollm_student_qwen_teacher_sequence(smollm_tokenizer, qwen_tokenizer):\n config = build_config(\n uld_use_hybrid_loss=True,\n uld_hybrid_matched_weight=0.5,\n uld_hybrid_unmatched_weight=0.5,\n )\n loss_fn = ULDLoss(config, student_tokenizer=smollm_tokenizer, teacher_tokenizer=qwen_tokenizer)\n\n prompt = \"User: Describe SmolLM3 in a sentence.\"\n completion = \"Assistant: SmolLM3 is a compact yet capable language model.\"\n\n student_ids, student_labels = encode_prompt_completion(smollm_tokenizer, prompt, completion)\n teacher_ids, teacher_labels = encode_prompt_completion(qwen_tokenizer, prompt, completion)\n\n pad_id_student = smollm_tokenizer.pad_token_id\n pad_id_teacher = qwen_tokenizer.pad_token_id\n max_length = max(len(student_ids), len(teacher_ids))\n\n student_ids = pad_tokens(student_ids, pad_id_student, max_length)\n teacher_ids = pad_tokens(teacher_ids, pad_id_teacher, max_length)\n student_labels = pad_labels(student_labels, max_length)\n teacher_labels = pad_labels(teacher_labels, max_length)\n\n student_input_ids = torch.tensor([student_ids])\n teacher_input_ids = torch.tensor([teacher_ids])\n student_labels = torch.tensor([student_labels])\n teacher_labels = torch.tensor([teacher_labels])\n\n student_vocab = len(smollm_tokenizer)\n teacher_vocab = len(qwen_tokenizer)\n\n student_logits = torch.randn(1, max_length, student_vocab)\n teacher_logits = torch.randn(1, max_length, teacher_vocab)\n\n loss = loss_fn(\n student_logits=student_logits,\n teacher_logits=teacher_logits,\n student_labels=student_labels,\n teacher_labels=teacher_labels,\n student_input_ids=student_input_ids,\n teacher_input_ids=teacher_input_ids,\n )\n\n assert torch.isfinite(loss)\n assert loss.dim() == 0\n assert loss_fn.last_matched_loss is not None\n assert loss_fn.last_unmatched_loss is not None\n\n\ndef test_uldloss_hybrid_config_beta_zero(llama_tokenizer, qwen_tokenizer):\n config = build_config(\n uld_use_hybrid_loss=True,\n uld_hybrid_matched_weight=0.0,\n uld_hybrid_unmatched_weight=1.0,\n use_extended_uld=True,\n uld_crossentropy_weight=0.0,\n uld_distillation_weight=1.0,\n uld_student_temperature=1.0,\n uld_teacher_temperature=1.0,\n temperature=1.0,\n top_p=0.95,\n top_k=0,\n lmbda=1.0,\n beta=0.0,\n )\n loss_fn = ULDLoss(config, student_tokenizer=llama_tokenizer, teacher_tokenizer=qwen_tokenizer)\n\n prompt = \"User: Explain how GOLD handles tokenizer mismatches.\"\n completion = \"Assistant: GOLD merges aligned subwords and applies hybrid ULD loss.\"\n\n student_ids, student_labels = encode_prompt_completion(llama_tokenizer, prompt, completion)\n teacher_ids, teacher_labels = encode_prompt_completion(qwen_tokenizer, prompt, completion)\n\n pad_id_student = llama_tokenizer.pad_token_id\n pad_id_teacher = qwen_tokenizer.pad_token_id\n max_length = max(len(student_ids), len(teacher_ids))\n\n student_ids = pad_tokens(student_ids, pad_id_student, max_length)\n teacher_ids = pad_tokens(teacher_ids, pad_id_teacher, max_length)\n student_labels = pad_labels(student_labels, max_length)\n teacher_labels = pad_labels(teacher_labels, max_length)\n\n student_input_ids = torch.tensor([student_ids])\n teacher_input_ids = torch.tensor([teacher_ids])\n student_labels = torch.tensor([student_labels])\n teacher_labels = torch.tensor([teacher_labels])\n\n student_vocab = len(llama_tokenizer)\n teacher_vocab = len(qwen_tokenizer)\n torch.manual_seed(0)\n student_logits = torch.randn(1, max_length, student_vocab)\n teacher_logits = torch.randn(1, max_length, teacher_vocab)\n\n loss = loss_fn(\n student_logits=student_logits,\n teacher_logits=teacher_logits,\n student_labels=student_labels,\n teacher_labels=teacher_labels,\n student_input_ids=student_input_ids,\n teacher_input_ids=teacher_input_ids,\n )\n\n assert torch.isfinite(loss)\n assert loss.dim() == 0\n assert loss_fn.last_matched_loss is not None\n assert loss_fn.last_unmatched_loss is not None\n\n expected = config.uld_hybrid_unmatched_weight * loss_fn.last_unmatched_loss\n torch.testing.assert_close(loss, expected, atol=1e-6, rtol=1e-5)\n" }, { "path": "tests/experimental/test_grpo_with_replay_buffer_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pytest\nimport torch\nfrom datasets import load_dataset\n\nfrom trl.experimental.grpo_with_replay_buffer import (\n GRPOWithReplayBufferConfig,\n GRPOWithReplayBufferTrainer,\n ReplayBuffer,\n)\n\nfrom ..testing_utils import TrlTestCase\n\n\n@pytest.mark.low_priority\nclass TestReplayBuffer:\n def setup_method(self):\n self.replay_buffer = ReplayBuffer(max_size=5)\n\n def test_add(self):\n # Add elements to the replay buffer\n scores = [0.5, 0.8, 0.3, 0.9, 0.7]\n data = [\n {\"id\": 1},\n {\"id\": 2},\n {\"id\": 3},\n {\"id\": 4},\n {\"id\": 5},\n ]\n self.replay_buffer.add(scores, data)\n\n # Check if the buffer contains the correct number of elements\n assert len(self.replay_buffer.heap) == 5\n\n # Check if the buffer maintains the min-heap property\n heap_scores = [item[0] for item in self.replay_buffer.heap]\n assert heap_scores[0] == min(heap_scores)\n assert heap_scores[0] == 0.3\n\n def test_add_more_than_maxlen(self):\n # Add elements to the replay buffer\n scores = [0.5, 0.8, 0.3, 0.9, 0.7, 0.6, 0.4]\n data = [\n {\"id\": 1},\n {\"id\": 2},\n {\"id\": 3},\n {\"id\": 4},\n {\"id\": 5},\n {\"id\": 6},\n {\"id\": 7},\n ]\n self.replay_buffer.add(scores, data)\n\n # Check if the buffer contains the correct number of elements\n assert len(self.replay_buffer.heap) == 5\n\n # Check if the buffer maintains the min-heap property\n heap_scores = [item[0] for item in self.replay_buffer.heap]\n assert heap_scores[0] == min(heap_scores)\n assert heap_scores[0] == 0.5 # 0.3 and 0.4 should be removed\n\n def test_sample(self):\n # Add elements to the replay buffer\n scores = [0.5, 0.8, 0.3, 0.9, 0.7]\n data = [\n {\"id\": 1},\n {\"id\": 2},\n {\"id\": 3},\n {\"id\": 4},\n {\"id\": 5},\n ]\n self.replay_buffer.add(scores, data)\n\n # Sample elements from the buffer\n sampled = self.replay_buffer.sample(num_samples=3)\n\n # Check if the sampled elements are from the buffer\n assert len(sampled) == 3\n for item in sampled:\n assert item in [entry[1] for entry in self.replay_buffer.heap]\n\n\n@pytest.mark.low_priority\nclass TestUpdateWithReplayBuffer:\n def setup_method(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n config = GRPOWithReplayBufferConfig(\n replay_buffer_size=5,\n )\n self.trainer = GRPOWithReplayBufferTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=config,\n train_dataset=dataset,\n )\n self.trainer.replay_buffer = ReplayBuffer(max_size=5)\n self.trainer.num_generations = 2\n\n def _prepopulate_buffer(self, with_pixels=False, with_logprobs=False):\n scores = [0.1, 0.9]\n data = [\n {\n \"prompt_ids\": torch.tensor([[100, 101], [102, 103]]),\n \"prompt_mask\": torch.ones(2, 2, dtype=torch.long),\n \"completion_ids\": torch.tensor([[5, 6], [7, 8]]),\n \"completion_mask\": torch.ones(2, 2, dtype=torch.long),\n \"advantages\": torch.tensor([[0.5, 0.6]]),\n **({\"pixel_values\": torch.randn(2, 3, 224, 224)} if with_pixels else {}),\n **({\"old_per_token_logps\": torch.randn(2, 2)} if with_logprobs else {}),\n },\n {\n \"prompt_ids\": torch.tensor([[104, 105], [106, 107]]),\n \"prompt_mask\": torch.ones(2, 2, dtype=torch.long),\n \"completion_ids\": torch.tensor([[13, 14], [15, 16]]),\n \"completion_mask\": torch.ones(2, 2, dtype=torch.long),\n \"advantages\": torch.tensor([[0.8, 0.85]]),\n **({\"pixel_values\": torch.randn(2, 3, 224, 224)} if with_pixels else {}),\n **({\"old_per_token_logps\": torch.randn(2, 2)} if with_logprobs else {}),\n },\n ]\n self.trainer.replay_buffer.add(scores, data)\n\n def _make_inputs(self, group_advantages, with_pixels=False, with_logprobs=False):\n inputs = {\n \"group_advantages\": group_advantages,\n \"prompt_ids\": torch.tensor([[1, 2], [3, 4], [5, 6], [7, 8]]),\n \"prompt_mask\": torch.ones(4, 2, dtype=torch.long),\n \"completion_ids\": torch.tensor([[9, 10], [11, 12], [13, 14], [15, 16]]),\n \"completion_mask\": torch.ones(4, 2, dtype=torch.long),\n \"forward_kwargs\": {\"pixel_values\": torch.randn(4, 3, 224, 224)} if with_pixels else {},\n \"old_per_token_logps\": torch.randn(4, 2) if with_logprobs else None,\n }\n inputs[\"group_std_rewards\"] = group_advantages.std(dim=1).expand_as(group_advantages)\n return inputs\n\n def test_update_with_replay_buffer_no_variance(self):\n self._prepopulate_buffer(with_pixels=True, with_logprobs=True)\n group_advantages = torch.tensor([[0.5, 0.5], [0.8, 0.8]]) # no variance\n inputs = self._make_inputs(group_advantages, with_pixels=True, with_logprobs=True)\n original_prompt_ids = inputs[\"prompt_ids\"].clone()\n\n outputs = self.trainer.update_with_replay_buffer(**inputs, num_items_in_batch=4)\n\n assert outputs is not None\n assert \"pixel_values\" in outputs\n assert \"old_per_token_logps\" in outputs\n assert len(self.trainer.replay_buffer.heap) == 2\n for pid in outputs[\"prompt_ids\"]:\n assert pid.tolist() not in original_prompt_ids.tolist()\n\n def test_update_with_replay_buffer_with_variance(self):\n self._prepopulate_buffer()\n group_advantages = torch.tensor([[0.6, 0.4], [0.7, 1.2]]) # has variance\n inputs = self._make_inputs(group_advantages)\n\n sampled = self.trainer.update_with_replay_buffer(**inputs, num_items_in_batch=4)\n\n assert len(self.trainer.replay_buffer.heap) == 4 # grew\n assert sampled is None\n\n def test_update_with_mixed_variance(self):\n self._prepopulate_buffer()\n group_advantages = torch.tensor([[0.6, 0.6], [0.3, 0.45]]) # one no-variance, one variance\n inputs = self._make_inputs(group_advantages)\n original_prompt_ids = inputs[\"prompt_ids\"].clone().view(-1, self.trainer.num_generations, 2).tolist()\n\n outputs = self.trainer.update_with_replay_buffer(**inputs, num_items_in_batch=4)\n\n assert len(self.trainer.replay_buffer.heap) == 3 # grew by 1\n output_prompt_ids = outputs[\"prompt_ids\"].view(-1, self.trainer.num_generations, 2).tolist()\n\n buffer_ids = [item[1][\"prompt_ids\"].tolist() for item in self.trainer.replay_buffer.heap]\n found_from_buffer = any(pid in buffer_ids for pid in output_prompt_ids)\n found_from_original = any(pid in original_prompt_ids for pid in output_prompt_ids)\n\n assert found_from_buffer\n assert found_from_original\n assert [[1, 2], [3, 4]] not in output_prompt_ids # excluded no-variance group\n\n def test_update_with_inputs_different_seq_len(self):\n \"\"\"\n Test with inputs where the sequence lengths are different from the prepopulated buffer.\n \"\"\"\n self._prepopulate_buffer()\n pad_token_id = self.trainer.processing_class.pad_token_id\n group_advantages = torch.tensor([[0.6, 0.6], [0.3, 0.45]]) # one no-variance, one variance\n inputs = {\n \"group_advantages\": group_advantages,\n \"prompt_ids\": torch.tensor(\n [\n [1, 2, pad_token_id],\n [1, 2, pad_token_id],\n [3, 4, 5],\n [3, 4, 5],\n ]\n ),\n \"prompt_mask\": torch.tensor([[1, 1, 0], [1, 1, 0], [1, 1, 1], [1, 1, 1]], dtype=torch.long),\n \"completion_ids\": torch.tensor(\n [\n [1009, 1010, pad_token_id],\n [1011, 1012, 1013],\n [1013, 1014, pad_token_id],\n [1015, 1016, 1017],\n ]\n ),\n \"completion_mask\": torch.tensor([[1, 1, 0], [1, 1, 1], [1, 1, 0], [1, 1, 1]], dtype=torch.long),\n \"forward_kwargs\": {},\n }\n inputs[\"group_std_rewards\"] = group_advantages.std(dim=1).expand_as(group_advantages)\n\n outputs_after_sampling = self.trainer.update_with_replay_buffer(**inputs, num_items_in_batch=4)\n # Seq length of current batch should be preserved\n assert outputs_after_sampling[\"prompt_ids\"].shape[-1] == 3\n assert len(self.trainer.replay_buffer.heap) == 3\n output_prompt_ids = outputs_after_sampling[\"prompt_ids\"].view(-1, self.trainer.num_generations, 3).tolist()\n\n buffered_prompt_completion_ids = [\n (item[1][\"prompt_ids\"].tolist(), item[1][\"completion_ids\"].tolist())\n for item in self.trainer.replay_buffer.heap\n ]\n buffered_prompt_ids, buffered_completion_ids = zip(*buffered_prompt_completion_ids, strict=True)\n\n # Check for new entry with seq len 3 in buffer\n assert [[3, 4, 5], [3, 4, 5]] in buffered_prompt_ids # excluded no-variance group\n assert [\n [1013, 1014, pad_token_id],\n [1015, 1016, 1017],\n ] in buffered_completion_ids # excluded no-variance group\n\n # Check that sampled outputs contain one group with prompt_ids starting with a pad token\n assert [\n [pad_token_id, 101, 102],\n [pad_token_id, 102, 103],\n ] in output_prompt_ids or [\n [pad_token_id, 104, 105],\n [pad_token_id, 106, 107],\n ] in output_prompt_ids\n\n\n@pytest.mark.low_priority\n@pytest.mark.parametrize(\"scale_rewards\", [\"batch\", \"group\"])\nclass TestGRPOWithReplayBufferTrainer(TrlTestCase):\n def test_training_with_replay_buffer(self, scale_rewards):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Guarantee that some rewards have 0 std\n def custom_reward_func(completions, **kwargs):\n if torch.rand(1).item() < 0.25:\n return [0] * len(completions) # simulate some None rewards\n else:\n return torch.rand(len(completions)).tolist()\n\n training_args = GRPOWithReplayBufferConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=4, # reduce the batch size to reduce memory usage\n num_generations=4, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n replay_buffer_size=8,\n report_to=\"none\",\n scale_rewards=scale_rewards,\n )\n trainer = GRPOWithReplayBufferTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[custom_reward_func],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n" }, { "path": "tests/experimental/test_gspo_token_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\nimport torch\nfrom datasets import load_dataset\nfrom transformers.utils import is_peft_available\n\nfrom trl import GRPOConfig\nfrom trl.experimental.gspo_token import GRPOTrainer as GSPOTokenTrainer\n\nfrom ..testing_utils import TrlTestCase\n\n\nif is_peft_available():\n pass\n\n\nclass TestGSPOTokenTrainer(TrlTestCase):\n def test_training(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n num_iterations=2, # the importance sampling weights won't be 0 in this case\n importance_sampling_level=\"sequence_token\",\n report_to=\"none\",\n )\n trainer = GSPOTokenTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n" }, { "path": "tests/experimental/test_judges.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport random\nimport sys\nimport time\n\nimport pytest\nimport transformers\nfrom packaging.version import Version\n\nfrom trl.experimental.judges import AllTrueJudge, BaseBinaryJudge, HfPairwiseJudge, PairRMJudge\n\nfrom ..testing_utils import TrlTestCase, require_llm_blender\n\n\nclass RandomBinaryJudge(BaseBinaryJudge):\n \"\"\"\n Random binary judge, for testing purposes.\n \"\"\"\n\n def judge(self, prompts, completions, gold_completions=None, shuffle_order=True):\n return [random.choice([0, 1, -1]) for _ in range(len(prompts))]\n\n\nclass TestJudges(TrlTestCase):\n def _get_prompts_and_pairwise_completions(self):\n prompts = [\"The capital of France is\", \"The biggest planet in the solar system is\"]\n completions = [[\"Paris\", \"Marseille\"], [\"Saturn\", \"Jupiter\"]]\n return prompts, completions\n\n def _get_prompts_and_single_completions(self):\n prompts = [\"What's the capital of France?\", \"What's the color of the sky?\"]\n completions = [\"Marseille\", \"blue\"]\n return prompts, completions\n\n def test_all_true_judge(self):\n judge = AllTrueJudge(judges=[RandomBinaryJudge(), RandomBinaryJudge()])\n prompts, completions = self._get_prompts_and_single_completions()\n judgements = judge.judge(prompts=prompts, completions=completions)\n assert len(judgements) == 2\n assert all(judgement in {0, 1, -1} for judgement in judgements)\n\n @pytest.mark.skip(reason=\"This test needs to be run manually since it requires a valid Hugging Face API key.\")\n def test_hugging_face_judge(self):\n judge = HfPairwiseJudge()\n prompts, completions = self._get_prompts_and_pairwise_completions()\n ranks = judge.judge(prompts=prompts, completions=completions)\n assert len(ranks) == 2\n assert all(isinstance(rank, int) for rank in ranks)\n assert ranks == [0, 1]\n\n def load_pair_rm_judge(self):\n # When using concurrent tests, PairRM may fail to load the model while another job is still downloading.\n # This is a workaround to retry loading the model a few times.\n for _ in range(5):\n try:\n return PairRMJudge()\n except ValueError:\n time.sleep(5)\n raise ValueError(\"Failed to load PairRMJudge\")\n\n @require_llm_blender\n @pytest.mark.skipif(\n sys.version_info[:3] == (3, 13, 8), reason=\"Python 3.13.8 has a bug in inspect.BlockFinder (cpython GH-139783)\"\n )\n @pytest.mark.xfail(\n Version(transformers.__version__) >= Version(\"5.0.0\"),\n reason=\"Known incompatibility between llm-blender and transformers >= 5.0.0 (GH-4918)\",\n strict=True,\n )\n def test_pair_rm_judge(self):\n judge = self.load_pair_rm_judge()\n prompts, completions = self._get_prompts_and_pairwise_completions()\n ranks = judge.judge(prompts=prompts, completions=completions)\n assert len(ranks) == 2\n assert all(isinstance(rank, int) for rank in ranks)\n assert ranks == [0, 1]\n\n @require_llm_blender\n @pytest.mark.skipif(\n sys.version_info[:3] == (3, 13, 8), reason=\"Python 3.13.8 has a bug in inspect.BlockFinder (cpython GH-139783)\"\n )\n @pytest.mark.xfail(\n Version(transformers.__version__) >= Version(\"5.0.0\"),\n reason=\"Known incompatibility between llm-blender and transformers >= 5.0.0 (GH-4918)\",\n strict=True,\n )\n def test_pair_rm_judge_return_scores(self):\n judge = self.load_pair_rm_judge()\n prompts, completions = self._get_prompts_and_pairwise_completions()\n probs = judge.judge(prompts=prompts, completions=completions, return_scores=True)\n assert len(probs) == 2\n assert all(isinstance(prob, float) for prob in probs)\n assert all(0 <= prob <= 1 for prob in probs)\n" }, { "path": "tests/experimental/test_kto_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport multiprocess\nimport pytest\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom trl.experimental.kto import KTOConfig, KTOTrainer\nfrom trl.experimental.kto.kto_trainer import _get_kl_dataset, _process_tokens, _tokenize\n\nfrom ..testing_utils import TrlTestCase, require_liger_kernel, require_no_wandb, require_peft\n\n\nclass TestKTOTrainer(TrlTestCase):\n def setup_method(self):\n self.model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.model = AutoModelForCausalLM.from_pretrained(self.model_id, dtype=\"float32\")\n self.ref_model = AutoModelForCausalLM.from_pretrained(self.model_id)\n self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n self.tokenizer.pad_token = self.tokenizer.eos_token\n\n @pytest.mark.parametrize(\n \"config_name, loss_type, pre_compute, eval_dataset\",\n [\n (\"standard_preference\", \"kto\", True, True),\n (\"standard_unpaired_preference\", \"kto\", False, True),\n (\"conversational_implicit_prompt_preference\", \"apo_zero_unpaired\", True, True),\n (\"standard_unpaired_preference\", \"apo_zero_unpaired\", False, True),\n ],\n )\n def test_kto_trainer(self, config_name, loss_type, pre_compute, eval_dataset):\n training_args = KTOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\" if eval_dataset else \"no\",\n beta=0.1,\n precompute_ref_log_probs=pre_compute,\n loss_type=loss_type,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = KTOTrainer(\n model=self.model,\n ref_model=self.ref_model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"] if eval_dataset else None,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param, new_param)\n\n def test_kto_trainer_with_ref_model_is_model(self):\n training_args = KTOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n with pytest.raises(ValueError):\n KTOTrainer(\n model=self.model,\n ref_model=self.model, # ref_model can't be the same as model\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n )\n\n def test_tokenize_and_process_tokens(self):\n # Pytest/CI often starts background threads before tests run. Under Python 3.12+,\n # using \"fork\" in a multi-threaded process emits a DeprecationWarning and may deadlock.\n # Force \"spawn\" to keep this multiprocessing test safe while still exercising `num_proc=2`.\n multiprocess.set_start_method(\"spawn\", force=True)\n\n training_args = KTOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n beta=0.1,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n trainer = KTOTrainer(\n model=self.model,\n ref_model=self.ref_model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n train_dataset = dummy_dataset[\"train\"]\n tokenized_dataset = train_dataset.map(\n _tokenize,\n fn_kwargs={\"tokenizer\": trainer.processing_class},\n batched=True,\n batch_size=2,\n )\n assert tokenized_dataset[\"prompt\"][:] == train_dataset[\"prompt\"][:]\n assert tokenized_dataset[\"completion\"][:] == train_dataset[\"completion\"][:]\n assert tokenized_dataset[\"label\"][:] == train_dataset[\"label\"][:]\n assert tokenized_dataset[\"prompt_input_ids\"][0] == [46518, 374, 2664, 1091]\n assert tokenized_dataset[\"prompt_attention_mask\"][0] == [1, 1, 1, 1]\n assert tokenized_dataset[\"answer_input_ids\"][0] == [27261, 13]\n assert tokenized_dataset[\"answer_attention_mask\"][0] == [1, 1]\n\n # Test corruption of (prompt, completion) pairs for KL dataset\n for batch_size in [2, 3]:\n tokenized_kl_dataset = tokenized_dataset.map(_get_kl_dataset, batched=True, batch_size=batch_size)\n\n # Verify that the \"answer_input_ids\" have been modified, meaning the new \"answer_input_ids\" differ\n # from the original ones. However, when the length of the dataset modulo batch_size equals 1,\n # the last batch remains unaltered. This is a rare scenario that does not impact the training\n # process, so we exclude it from testing by iterating only up to len - 1.\n for i in range(len(tokenized_kl_dataset[\"answer_input_ids\"]) - 1):\n assert tokenized_dataset[\"prompt_input_ids\"][i] == tokenized_kl_dataset[\"prompt_input_ids\"][i]\n assert (\n tokenized_dataset[\"prompt_attention_mask\"][i] == tokenized_kl_dataset[\"prompt_attention_mask\"][i]\n )\n assert tokenized_dataset[\"answer_input_ids\"][i] != tokenized_kl_dataset[\"answer_input_ids\"][i]\n\n fn_kwargs = {\n \"prefix\": \"\",\n \"tokenizer\": trainer.processing_class,\n \"max_length\": trainer.max_length,\n }\n processed_dataset = tokenized_dataset.map(_process_tokens, fn_kwargs=fn_kwargs, num_proc=2)\n assert processed_dataset[\"prompt\"][:] == train_dataset[\"prompt\"][:]\n assert processed_dataset[\"completion\"][:] == train_dataset[\"completion\"][:]\n assert processed_dataset[\"label\"][:] == train_dataset[\"label\"][:]\n assert processed_dataset[\"prompt_input_ids\"][0] == [46518, 374, 2664, 1091]\n assert processed_dataset[\"prompt_attention_mask\"][0] == [1, 1, 1, 1]\n assert processed_dataset[\"completion_input_ids\"][0] == [46518, 374, 2664, 1091, 27261, 13, 151645]\n assert processed_dataset[\"completion_attention_mask\"][0] == [1, 1, 1, 1, 1, 1, 1]\n assert processed_dataset[\"completion_labels\"][0] == [-100, -100, -100, -100, 27261, 13, 151645]\n\n def test_kto_trainer_without_providing_ref_model(self):\n training_args = KTOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=4,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n beta=0.1,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n trainer = KTOTrainer(\n model=self.model,\n ref_model=None,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param, new_param)\n\n @require_peft\n def test_kto_trainer_without_providing_ref_model_with_lora(self):\n from peft import LoraConfig\n\n lora_config = LoraConfig(\n r=16,\n lora_alpha=32,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n )\n\n training_args = KTOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=4,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n beta=0.1,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n trainer = KTOTrainer(\n model=self.model,\n ref_model=None,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n peft_config=lora_config,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n if \"lora\" in n:\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param, new_param)\n\n @require_no_wandb\n def test_kto_trainer_generate_during_eval_no_wandb(self):\n training_args = KTOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n beta=0.1,\n generate_during_eval=True,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n with pytest.raises(\n ValueError,\n match=\"`generate_during_eval=True` requires Weights and Biases or Comet to be installed.\"\n \" Please install `wandb` or `comet-ml` to resolve.\",\n ):\n KTOTrainer(\n model=self.model,\n ref_model=None,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n @require_liger_kernel\n def test_kto_trainer_with_liger(self):\n \"\"\"Test KTO trainer with Liger kernel enabled.\"\"\"\n training_args = KTOConfig(\n output_dir=self.tmp_dir,\n report_to=\"none\",\n use_liger_kernel=True, # Enable Liger kernel\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n trainer = KTOTrainer(\n model=self.model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n # check the params have changed - ignore 0 biases\n if param.sum() != 0:\n assert not torch.equal(param, new_param)\n\n def test_compute_metrics(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\")\n ref_model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n tokenizer.pad_token = tokenizer.eos_token\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_unpaired_preference\")\n\n def dummy_compute_metrics(*args, **kwargs):\n return {\"test\": 0.0}\n\n training_args = KTOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False,\n per_device_train_batch_size=2,\n do_eval=True,\n eval_strategy=\"steps\",\n eval_steps=1,\n per_device_eval_batch_size=2,\n report_to=\"none\",\n )\n\n trainer = KTOTrainer(\n model=model,\n ref_model=ref_model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n compute_metrics=dummy_compute_metrics,\n )\n\n trainer.train()\n\n assert trainer.state.log_history[-2][\"eval_test\"] == 0.0\n" }, { "path": "tests/experimental/test_merge_model_callback.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\nimport os\n\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom transformers.trainer_utils import get_last_checkpoint\n\nfrom trl import DPOConfig, DPOTrainer\nfrom trl.experimental.merge_model_callback import MergeConfig, MergeModelCallback\n\nfrom ..testing_utils import TrlTestCase, require_mergekit\n\n\n@require_mergekit\nclass TestMergeModelCallback(TrlTestCase):\n def setup_method(self):\n self.model = AutoModelForCausalLM.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\"\n )\n self.tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n self.dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n def test_callback(self):\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n num_train_epochs=1,\n report_to=\"none\",\n save_strategy=\"steps\",\n save_steps=1,\n )\n config = MergeConfig()\n merge_callback = MergeModelCallback(config)\n trainer = DPOTrainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset,\n processing_class=self.tokenizer,\n callbacks=[merge_callback],\n )\n trainer.train()\n last_checkpoint = get_last_checkpoint(self.tmp_dir)\n merged_path = os.path.join(last_checkpoint, \"merged\")\n assert os.path.isdir(merged_path), \"Merged folder does not exist in the last checkpoint.\"\n\n def test_every_checkpoint(self):\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n num_train_epochs=1,\n report_to=\"none\",\n save_strategy=\"steps\",\n save_steps=1,\n )\n config = MergeConfig()\n merge_callback = MergeModelCallback(config, merge_at_every_checkpoint=True)\n trainer = DPOTrainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset,\n processing_class=self.tokenizer,\n callbacks=[merge_callback],\n )\n trainer.train()\n\n checkpoints = sorted(\n [os.path.join(self.tmp_dir, cp) for cp in os.listdir(self.tmp_dir) if cp.startswith(\"checkpoint-\")]\n )\n\n for checkpoint in checkpoints:\n merged_path = os.path.join(checkpoint, \"merged\")\n assert os.path.isdir(merged_path), f\"Merged folder does not exist in checkpoint {checkpoint}.\"\n" }, { "path": "tests/experimental/test_minillm_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pytest\nimport torch\nfrom datasets import load_dataset\n\nfrom trl.experimental.minillm import MiniLLMConfig, MiniLLMTrainer\n\nfrom ..testing_utils import TrlTestCase\n\n\n@pytest.mark.low_priority\nclass TestMiniLLMTrainer(TrlTestCase):\n def test_train(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Initialize the trainer\n training_args = MiniLLMConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=32, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = MiniLLMTrainer(\n model=\"trl-internal-testing/small-Qwen3ForCausalLM\",\n teacher_model=\"trl-internal-testing/tiny-Qwen3ForCausalLM\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n" }, { "path": "tests/experimental/test_modeling_value_head.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\nimport torch\n\nfrom trl.experimental.ppo import AutoModelForCausalLMWithValueHead\nfrom trl.experimental.utils import create_reference_model\n\nfrom ..testing_utils import TrlTestCase\n\n\nclass TestReferenceModel(TrlTestCase):\n def setup_method(self):\n self.model = AutoModelForCausalLMWithValueHead.from_pretrained(\"trl-internal-testing/tiny-GPT2LMHeadModel\")\n self.test_input = torch.tensor([[0, 1, 2, 3]])\n self.optimizer = torch.optim.AdamW(self.model.parameters(), lr=1)\n self.layer_format = \"pretrained_model.transformer.h.{layer}.attn.c_attn.weight\"\n\n def test_independent_reference(self):\n layer_0 = self.layer_format.format(layer=0)\n layer_1 = self.layer_format.format(layer=1)\n\n ref_model = create_reference_model(self.model)\n\n first_layer_before = self.model.get_parameter(layer_0).data.clone()\n last_layer_before = self.model.get_parameter(layer_1).data.clone() # the model only has 2 layers\n\n first_ref_layer_before = ref_model.get_parameter(layer_0).data.clone()\n last_ref_layer_before = ref_model.get_parameter(layer_1).data.clone()\n\n output = self.model(input_ids=self.test_input, labels=self.test_input)\n output[1].backward()\n self.optimizer.step()\n\n first_layer_after = self.model.get_parameter(layer_0).data.clone()\n last_layer_after = self.model.get_parameter(layer_1).data.clone()\n\n first_ref_layer_after = ref_model.get_parameter(layer_0).data.clone()\n last_ref_layer_after = ref_model.get_parameter(layer_1).data.clone()\n\n # before optimization ref and model are identical\n assert (first_layer_before == first_ref_layer_before).all()\n assert (last_layer_before == last_ref_layer_before).all()\n\n # ref model stays identical after optimization\n assert (first_ref_layer_before == first_ref_layer_after).all()\n assert (last_ref_layer_before == last_ref_layer_after).all()\n\n # optimized model changes\n assert not (first_layer_before == first_layer_after).all()\n assert not (last_layer_before == last_layer_after).all()\n\n def test_shared_layers(self):\n layer_0 = self.layer_format.format(layer=0)\n layer_1 = self.layer_format.format(layer=1)\n\n ref_model = create_reference_model(self.model, num_shared_layers=1)\n\n first_layer_before = self.model.get_parameter(layer_0).data.clone()\n second_layer_before = self.model.get_parameter(layer_1).data.clone()\n\n first_ref_layer_before = ref_model.get_parameter(layer_0).data.clone()\n second_ref_layer_before = ref_model.get_parameter(layer_1).data.clone()\n\n output = self.model(input_ids=self.test_input, labels=self.test_input)\n output[1].backward()\n self.optimizer.step()\n\n first_layer_after = self.model.get_parameter(layer_0).data.clone()\n second_layer_after = self.model.get_parameter(layer_1).data.clone()\n\n first_ref_layer_after = ref_model.get_parameter(layer_0).data.clone()\n second_ref_layer_after = ref_model.get_parameter(layer_1).data.clone()\n\n # before optimization ref and model are identical\n assert (first_layer_before == first_ref_layer_before).all()\n assert (second_layer_before == second_ref_layer_before).all()\n\n # ref model stays identical after optimization\n assert (first_ref_layer_before == first_ref_layer_after).all()\n assert (second_ref_layer_before == second_ref_layer_after).all()\n\n # first layer of optimized model stays the same\n assert (first_layer_before == first_layer_after).all()\n\n # other layers in optimized model change\n assert not (second_layer_before == second_layer_after).all()\n" }, { "path": "tests/experimental/test_nash_md_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pytest\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer, GenerationConfig\nfrom transformers.utils import is_peft_available\n\nfrom trl.experimental.nash_md import NashMDConfig, NashMDTrainer\nfrom trl.experimental.nash_md.nash_md_trainer import GeometricMixtureWrapper\nfrom trl.experimental.utils import create_reference_model\n\nfrom ..testing_utils import TrlTestCase, require_llm_blender, require_peft\nfrom .testing_utils import RandomPairwiseJudge\n\n\nif is_peft_available():\n from peft import LoraConfig, get_peft_model\n\n\nclass TestGeometricMixtureWrapper(TrlTestCase):\n def setup_method(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n self.model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\").to(self.device)\n self.ref_model = create_reference_model(self.model).to(self.device)\n self.generation_config = GenerationConfig.from_pretrained(model_id)\n self.mixture_coef = 0.5\n self.wrapper = GeometricMixtureWrapper(\n self.model, self.ref_model, self.generation_config, mixture_coef=self.mixture_coef\n )\n\n def test_forward(self):\n input_ids = torch.tensor([[1, 2, 3, 4, 5]], device=self.device)\n attention_mask = torch.ones_like(input_ids)\n\n output = self.wrapper(input_ids=input_ids, attention_mask=attention_mask)\n\n assert output is not None\n assert hasattr(output, \"logits\")\n assert output.logits.shape == (1, 5, self.model.config.vocab_size)\n\n def test_mixture_coefficient(self):\n input_ids = torch.tensor([[1, 2, 3, 4, 5]], device=self.device)\n attention_mask = torch.ones_like(input_ids)\n\n with torch.no_grad():\n model_output = self.model(input_ids=input_ids, attention_mask=attention_mask)\n ref_model_output = self.ref_model(input_ids=input_ids, attention_mask=attention_mask)\n wrapper_output = self.wrapper(input_ids=input_ids, attention_mask=attention_mask)\n\n expected_logits = torch.nn.functional.log_softmax(\n self.mixture_coef * ref_model_output.logits + (1 - self.mixture_coef) * model_output.logits, dim=-1\n )\n\n torch.testing.assert_close(wrapper_output.logits, expected_logits)\n\n def test_prepare_inputs_for_generation(self):\n input_ids = torch.tensor([[1, 2, 3, 4, 5]], device=self.device)\n attention_mask = torch.ones_like(input_ids)\n\n inputs = self.wrapper.prepare_inputs_for_generation(input_ids, attention_mask=attention_mask, use_cache=True)\n\n assert \"input_ids\" in inputs\n assert \"attention_mask\" in inputs\n assert not inputs.get(\"use_cache\", False)\n\n\nclass TestNashMDTrainer(TrlTestCase):\n def setup_method(self):\n self.model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.model = AutoModelForCausalLM.from_pretrained(self.model_id, dtype=\"float32\")\n self.ref_model = AutoModelForCausalLM.from_pretrained(self.model_id)\n self.reward_model = AutoModelForSequenceClassification.from_pretrained(self.model_id, num_labels=1)\n self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n self.tokenizer.pad_token = self.tokenizer.eos_token\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n def test_nash_md_trainer_training(self, config_name):\n training_args = NashMDConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = NashMDTrainer(\n model=self.model,\n ref_model=self.ref_model,\n reward_funcs=self.reward_model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @require_peft\n def test_training_with_peft(self):\n lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\", task_type=\"CAUSAL_LM\")\n training_args = NashMDConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = NashMDTrainer(\n model=self.model,\n reward_funcs=self.reward_model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n peft_config=lora_config,\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @require_peft\n def test_training_with_peft_and_ref_model(self):\n lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\", task_type=\"CAUSAL_LM\")\n training_args = NashMDConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = NashMDTrainer(\n model=self.model,\n ref_model=self.ref_model,\n reward_funcs=self.reward_model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n peft_config=lora_config,\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @require_peft\n def test_training_pre_pefted_model_implicit_ref_with_reward_model(self):\n lora_config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.1, bias=\"none\", task_type=\"CAUSAL_LM\")\n # self.model from setUp is a base AutoModelForCausalLM\n peft_model_instance = get_peft_model(self.model, lora_config)\n\n training_args = NashMDConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=1, # Keep small for quick test\n max_steps=2, # Few steps\n learning_rate=5.0e-7,\n eval_strategy=\"no\",\n report_to=\"none\",\n remove_unused_columns=False, # Important for the dummy dataset\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")[\"train\"]\n\n trainer = NashMDTrainer(\n model=peft_model_instance, # Pass the already PEFT model\n ref_model=None, # Implicit reference from peft_model_instance's base\n reward_funcs=self.reward_model, # To trigger GeometricMixtureWrapper path\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset,\n # peft_config is not passed, as model is already PEFT\n )\n\n trainer.train()\n\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n @require_llm_blender\n def test_nash_md_trainer_judge_training(self, config_name):\n training_args = NashMDConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n judge = RandomPairwiseJudge()\n\n trainer = NashMDTrainer(\n model=self.model,\n ref_model=self.ref_model,\n judge=judge,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n" }, { "path": "tests/experimental/test_online_dpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pytest\nimport transformers\nfrom datasets import Dataset, features, load_dataset\nfrom packaging.version import Version\nfrom transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer\nfrom transformers.utils import is_peft_available, is_vision_available\n\nfrom trl.experimental.online_dpo import OnlineDPOConfig, OnlineDPOTrainer\n\nfrom ..testing_utils import (\n TrlTestCase,\n require_llm_blender,\n require_peft,\n require_torch_accelerator,\n require_vision,\n require_vllm,\n)\nfrom .testing_utils import RandomPairwiseJudge\n\n\nif is_peft_available():\n from peft import LoraConfig\n\nif is_vision_available():\n import numpy as np\n from PIL import Image\n from transformers import AutoModelForImageTextToText, AutoProcessor\n\n\nclass TestOnlineDPOTrainer(TrlTestCase):\n def setup_method(self):\n self.model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.model = AutoModelForCausalLM.from_pretrained(self.model_id, dtype=\"float32\")\n self.ref_model = AutoModelForCausalLM.from_pretrained(self.model_id)\n self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n self.tokenizer.pad_token = self.tokenizer.eos_token\n\n self.reward_model_id = \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\"\n self.reward_model = AutoModelForSequenceClassification.from_pretrained(self.reward_model_id, num_labels=1)\n self.reward_tokenizer = AutoTokenizer.from_pretrained(self.reward_model_id)\n self.reward_tokenizer.pad_token = self.reward_tokenizer.eos_token\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n def test_training(self, config_name):\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = OnlineDPOTrainer(\n model=self.model,\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n )\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n def test_training_model_str(self):\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = OnlineDPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n )\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n def test_training_with_ref_model(self):\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = OnlineDPOTrainer(\n model=self.model,\n ref_model=self.ref_model,\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n )\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n def test_ref_model_is_model(self):\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n with pytest.raises(ValueError):\n OnlineDPOTrainer(\n model=self.model,\n ref_model=self.model, # ref_model can't be the same as model\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n processing_class=self.tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n )\n\n @require_peft\n def test_training_with_peft(self):\n lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\", task_type=\"CAUSAL_LM\")\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = OnlineDPOTrainer(\n model=self.model,\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n peft_config=lora_config,\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @require_peft\n def test_training_with_peft_and_ref_model(self):\n lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\", task_type=\"CAUSAL_LM\")\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = OnlineDPOTrainer(\n model=self.model,\n ref_model=self.ref_model,\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n peft_config=lora_config,\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n @require_llm_blender\n def test_training_with_judge(self, config_name):\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = OnlineDPOTrainer(\n model=self.model,\n judge=RandomPairwiseJudge(),\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n @require_torch_accelerator\n @require_vllm\n @pytest.mark.slow\n def test_training_with_vllm_server(self, config_name):\n def cleanup_vllm_communicator(trainer):\n \"\"\"Clean up vLLM communicator to avoid conflicts between test runs\"\"\"\n try:\n if hasattr(trainer, \"vllm_client\") and trainer.vllm_client is not None:\n trainer.vllm_client.close_communicator()\n except Exception:\n pass # Continue if cleanup fails\n\n model_id = \"trl-internal-testing/small-Qwen2ForCausalLM-2.5\" # We need a bigger model\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n tokenizer.pad_token = tokenizer.eos_token\n\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n use_vllm=True,\n vllm_mode=\"server\",\n vllm_gpu_memory_utilization=0.2,\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = OnlineDPOTrainer(\n model=model,\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n processing_class=tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n )\n\n # Ensure cleanup of vLLM communicator after the test\n try:\n trainer.train()\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n finally:\n cleanup_vllm_communicator(trainer)\n\n @require_vllm\n def test_training_with_vllm_colocate(self):\n \"\"\"Test vLLM colocate mode with our refactored implementation\"\"\"\n model_id = \"trl-internal-testing/small-Qwen2ForCausalLM-2.5\" # We need a bigger model\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n tokenizer.pad_token = tokenizer.eos_token\n\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n use_vllm=True,\n vllm_mode=\"colocate\",\n vllm_gpu_memory_utilization=0.2,\n per_device_train_batch_size=1,\n max_steps=2,\n report_to=\"none\",\n # Test generation parameters\n temperature=0.9,\n top_p=0.95,\n top_k=50,\n repetition_penalty=1.1,\n max_new_tokens=32,\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = OnlineDPOTrainer(\n model=model,\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n processing_class=tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n )\n\n # Verify vLLM setup\n assert trainer.use_vllm\n assert trainer.vllm_mode == \"colocate\"\n assert trainer.llm is not None\n # self.assertIsNone(trainer.vllm_client)\n # self.assertEqual(trainer.vllm_gpu_memory_utilization, 0.2)\n\n # Verify generation parameters\n assert trainer.temperature == 0.9\n assert trainer.top_p == 0.95\n assert trainer.top_k == 50\n assert trainer.repetition_penalty == 1.1\n\n # Verify generation config\n assert trainer.generation_config is not None\n assert trainer.generation_config.temperature == 0.9\n assert trainer.generation_config.top_p == 0.95\n assert trainer.generation_config.top_k == 50\n assert trainer.generation_config.repetition_penalty == 1.1\n assert trainer.generation_config.max_tokens == 32\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n def test_vllm_config_validation(self):\n \"\"\"Test vLLM configuration validation\"\"\"\n # Test valid vllm_mode values\n config = OnlineDPOConfig(use_vllm=True, vllm_mode=\"server\")\n assert config.vllm_mode == \"server\"\n\n config = OnlineDPOConfig(use_vllm=True, vllm_mode=\"colocate\")\n assert config.vllm_mode == \"colocate\"\n\n # Test default values\n config = OnlineDPOConfig()\n assert config.vllm_mode == \"colocate\"\n assert config.vllm_server_base_url is None\n assert config.vllm_server_host == \"0.0.0.0\"\n assert config.vllm_server_port == 8000\n assert config.vllm_server_timeout == 240.0\n assert config.vllm_gpu_memory_utilization == 0.55\n\n # Test generation parameters\n assert config.top_p == 1.0\n assert config.top_k == 0\n assert config.min_p is None\n assert config.repetition_penalty == 1.0\n assert not config.use_transformers_paged\n assert config.cache_implementation is None\n assert config.generation_kwargs is None\n\n def test_generation_config_setup(self):\n \"\"\"Test that generation configuration is properly set up for both vLLM and transformers\"\"\"\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n use_vllm=False,\n temperature=0.8,\n top_p=0.9,\n top_k=40,\n repetition_penalty=1.2,\n max_new_tokens=64,\n generation_kwargs={\"do_sample\": False},\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = OnlineDPOTrainer(\n model=self.model,\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n processing_class=self.tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n )\n\n # Verify transformers generation config\n assert not trainer.use_vllm\n # When not using vLLM, these attributes should not be set\n assert not (hasattr(trainer, \"llm\") and trainer.llm is not None)\n assert not (hasattr(trainer, \"vllm_client\") and trainer.vllm_client is not None)\n assert trainer.generation_config is not None\n assert trainer.generation_config.temperature == 0.8\n assert trainer.generation_config.top_p == 0.9\n assert trainer.generation_config.top_k == 40\n assert trainer.generation_config.repetition_penalty == 1.2\n assert trainer.generation_config.max_new_tokens == 64\n assert not trainer.generation_config.do_sample # From generation_kwargs\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n @require_torch_accelerator\n def test_training_with_transformers_paged(self, config_name):\n if Version(transformers.__version__) < Version(\"4.57.0\"):\n pytest.xfail(\"Bug in transformers solved in GH#40692, released in 4.57.0.\")\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n use_transformers_paged=True,\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = OnlineDPOTrainer(\n model=self.model,\n reward_funcs=self.reward_model,\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n reward_processing_classes=self.reward_tokenizer,\n )\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n def test_training_with_reward_funcs(self, config_name):\n def simple_reward_func(prompts, completions, completion_ids, **kwargs):\n return [0.5 for _ in prompts]\n\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n reward_weights=[0.7, 0.3],\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = OnlineDPOTrainer(\n model=self.model,\n reward_funcs=[simple_reward_func, simple_reward_func],\n args=training_args,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n trainer.train()\n\n assert \"train_loss\" in trainer.state.log_history[-1]\n assert len(trainer.reward_funcs) == 2\n assert trainer.reward_weights is not None\n assert round(abs(trainer.reward_weights[0].item() - 0.7), 5) == 0\n assert round(abs(trainer.reward_weights[1].item() - 0.3), 5) == 0\n\n\n@require_vision\nclass TestOnlineDPOVisionTrainer(TrlTestCase):\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Idefics2ForConditionalGeneration\",\n \"trl-internal-testing/tiny-LlavaForConditionalGeneration\",\n ],\n )\n def test_online_dpo_vlm_trainer(self, model_id):\n dataset_dict = {\n \"prompt\": [\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"Describe the image.\"}]}],\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What do you see?\"}]}],\n ],\n \"images\": [\n [Image.fromarray(np.random.randint(0, 255, (64, 64, 3), dtype=np.uint8))],\n [Image.fromarray(np.random.randint(0, 255, (64, 64, 3), dtype=np.uint8))],\n ],\n }\n dataset = Dataset.from_dict(dataset_dict)\n dataset = dataset.cast_column(\"images\", features.Sequence(features.Image()))\n\n model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=\"float32\")\n reward_model = AutoModelForSequenceClassification.from_pretrained(\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\", num_labels=1\n )\n processor = AutoProcessor.from_pretrained(model_id)\n reward_tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-LlamaForCausalLM-3.2\")\n reward_tokenizer.pad_token = reward_tokenizer.eos_token\n\n training_args = OnlineDPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=1,\n max_steps=2,\n learning_rate=0.01,\n report_to=\"none\",\n )\n trainer = OnlineDPOTrainer(\n model=model,\n reward_funcs=reward_model,\n args=training_args,\n processing_class=processor,\n train_dataset=dataset,\n eval_dataset=dataset,\n reward_processing_classes=reward_tokenizer,\n )\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n" }, { "path": "tests/experimental/test_orpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pytest\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoModelForSeq2SeqLM, AutoTokenizer\n\nfrom trl.experimental.orpo import ORPOConfig, ORPOTrainer\n\nfrom ..testing_utils import TrlTestCase, require_peft\n\n\nclass TestORPOTrainer(TrlTestCase):\n def setup_method(self):\n self.model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.model = AutoModelForCausalLM.from_pretrained(self.model_id, dtype=\"float32\")\n self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n self.tokenizer.pad_token = self.tokenizer.eos_token\n\n # get t5 as seq2seq example:\n model_id = \"trl-internal-testing/tiny-T5ForConditionalGeneration\"\n self.t5_model = AutoModelForSeq2SeqLM.from_pretrained(model_id, dtype=\"float32\")\n self.t5_tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n @pytest.mark.parametrize(\n \"name, config_name\",\n [\n (\"qwen\", \"standard_preference\"),\n (\"t5\", \"standard_implicit_prompt_preference\"),\n (\"qwen\", \"conversational_preference\"),\n ],\n )\n def test_orpo_trainer(self, name, config_name):\n training_args = ORPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n beta=0.1,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n if name == \"qwen\":\n model = self.model\n tokenizer = self.tokenizer\n elif name == \"t5\":\n model = self.t5_model\n tokenizer = self.t5_tokenizer\n training_args.is_encoder_decoder = True\n\n trainer = ORPOTrainer(\n model=model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param, new_param)\n\n @pytest.mark.parametrize(\n \"config_name\",\n [\n \"standard_preference\",\n \"standard_implicit_prompt_preference\",\n \"conversational_preference\",\n \"conversational_implicit_prompt_preference\",\n ],\n )\n @require_peft\n def test_orpo_trainer_with_lora(self, config_name):\n from peft import LoraConfig\n\n lora_config = LoraConfig(\n r=16,\n lora_alpha=32,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n )\n\n training_args = ORPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=4,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n beta=0.1,\n report_to=\"none\",\n )\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = ORPOTrainer(\n model=self.model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n peft_config=lora_config,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n if \"lora\" in n:\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.equal(param, new_param)\n\n def test_compute_metrics(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n tokenizer.pad_token = tokenizer.eos_token\n\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\")\n\n def dummy_compute_metrics(*args, **kwargs):\n return {\"test\": 0.0}\n\n training_args = ORPOConfig(\n output_dir=self.tmp_dir,\n remove_unused_columns=False,\n per_device_train_batch_size=2,\n do_eval=True,\n eval_strategy=\"steps\",\n eval_steps=1,\n per_device_eval_batch_size=2,\n report_to=\"none\",\n )\n\n trainer = ORPOTrainer(\n model=model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n compute_metrics=dummy_compute_metrics,\n )\n\n trainer.train()\n\n assert trainer.state.log_history[-2][\"eval_test\"] == 0.0\n" }, { "path": "tests/experimental/test_ppo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport gc\nimport os\n\nimport pytest\nimport torch\nfrom datasets import load_dataset\nfrom transformers import (\n AutoModelForCausalLM,\n AutoModelForSeq2SeqLM,\n AutoModelForSequenceClassification,\n AutoTokenizer,\n GenerationConfig,\n)\nfrom transformers.utils import is_peft_available\n\nfrom trl.experimental.ppo import (\n AutoModelForCausalLMWithValueHead,\n AutoModelForSeq2SeqLMWithValueHead,\n PPOConfig,\n PPOTrainer,\n)\nfrom trl.experimental.ppo.ppo_trainer import batch_generation, masked_mean, masked_var, masked_whiten\n\nfrom ..testing_utils import (\n TrlTestCase,\n require_bitsandbytes,\n require_peft,\n require_torch_gpu_if_bnb_not_multi_backend_enabled,\n)\n\n\nif is_peft_available():\n from peft import LoraConfig, get_peft_model\n\n\nALL_CAUSAL_LM_MODELS = [\n \"trl-internal-testing/tiny-BloomForCausalLM\",\n \"trl-internal-testing/tiny-CohereForCausalLM\",\n # \"trl-internal-testing/tiny-FalconMambaForCausalLM\", # FalconMambaForCausalLM modeling seems to be broken for now\n \"trl-internal-testing/tiny-Gemma2ForCausalLM\",\n \"trl-internal-testing/tiny-GemmaForCausalLM\",\n \"trl-internal-testing/tiny-GPT2LMHeadModel\",\n \"trl-internal-testing/tiny-GPTNeoXForCausalLM\",\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.1\",\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-LlamaForCausalLM-3\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.1\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n \"trl-internal-testing/tiny-OPTForCausalLM\",\n \"trl-internal-testing/tiny-Phi3ForCausalLM\",\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n]\n\nALL_SEQ2SEQ_MODELS = [\n \"trl-internal-testing/tiny-T5ForConditionalGeneration\",\n \"trl-internal-testing/tiny-BartModel\",\n]\n\n\nclass TestBatchGeneration(TrlTestCase):\n def setup_method(self):\n # Initialize the tokenizer\n self.model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n self.model = AutoModelForCausalLM.from_pretrained(self.model_id, dtype=\"float32\").to(self.device)\n self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n\n self.generation_config = GenerationConfig(\n max_new_tokens=128,\n temperature=0.5,\n do_sample=True,\n top_k=0,\n pad_token_id=self.tokenizer.pad_token_id,\n )\n\n # Example input\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\", split=\"train\")\n self.examples = dataset[\"messages\"]\n self.mini_batch_size = 3\n\n def test_mini_batch_generation(self):\n batch = [\n self.tokenizer.apply_chat_template(example[:-1], add_generation_prompt=True, tokenize=False)\n for example in self.examples\n ]\n queries = self.tokenizer(batch, padding=True, return_tensors=\"pt\")[\"input_ids\"].to(self.device)\n bs, context_length = queries.shape\n\n query_responses, logits = batch_generation(\n self.model, queries, self.mini_batch_size, self.tokenizer.pad_token_id, self.generation_config\n )\n\n max_length_query = query_responses.shape[1]\n max_length_logits = max_length_query - context_length\n\n assert max_length_query > context_length\n assert query_responses.shape == (bs, max_length_query)\n assert logits.shape == (bs, max_length_logits, self.model.config.vocab_size)\n\n def test_single_batch_generation(self):\n batch = [\n self.tokenizer.apply_chat_template(example[:-1], add_generation_prompt=True, tokenize=False)\n for example in self.examples\n ]\n queries = self.tokenizer(batch, padding=True, return_tensors=\"pt\")[\"input_ids\"].to(self.device)\n bs, context_length = queries.shape\n\n query_responses, logits = batch_generation(\n self.model, queries, bs, self.tokenizer.pad_token_id, self.generation_config\n )\n\n max_length_query = query_responses.shape[1]\n max_length_logits = max_length_query - context_length\n\n assert max_length_query > context_length\n assert query_responses.shape == (bs, max_length_query)\n assert logits.shape == (bs, max_length_logits, self.model.config.vocab_size)\n\n\nclass BaseTester:\n class VHeadModelTester(TrlTestCase):\n all_model_names = None\n trl_model_class = None\n transformers_model_class = None\n\n def setup_method(self):\n self.device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\n def test_value_head(self):\n r\"\"\"\n Test if the v-head is added to the model successfully\n \"\"\"\n for model_name in self.all_model_names:\n model = self.trl_model_class.from_pretrained(model_name)\n assert hasattr(model, \"v_head\")\n\n def test_value_head_shape(self):\n r\"\"\"\n Test if the v-head has the correct shape\n \"\"\"\n for model_name in self.all_model_names:\n model = self.trl_model_class.from_pretrained(model_name)\n assert model.v_head.summary.weight.shape[0] == 1\n\n def test_value_head_init_random(self):\n r\"\"\"\n Test if the v-head has been randomly initialized. We can check that by making sure the bias is different\n than zeros by default.\n \"\"\"\n for model_name in self.all_model_names:\n model = self.trl_model_class.from_pretrained(model_name)\n assert not torch.allclose(model.v_head.summary.bias, torch.zeros_like(model.v_head.summary.bias))\n\n def test_value_head_not_str(self):\n r\"\"\"\n Test if the v-head is added to the model successfully, by passing a non `PretrainedModel` as an argument to\n `from_pretrained`.\n \"\"\"\n for model_name in self.all_model_names:\n pretrained_model = self.transformers_model_class.from_pretrained(model_name)\n model = self.trl_model_class.from_pretrained(pretrained_model)\n assert hasattr(model, \"v_head\")\n\n def test_from_save_trl(self):\n \"\"\"\n Test if the model can be saved and loaded from a directory and get the same weights, including the\n additional modules (e.g. v_head)\n \"\"\"\n for model_name in self.all_model_names:\n model = self.trl_model_class.from_pretrained(model_name)\n\n model.save_pretrained(self.tmp_dir)\n\n model_from_save = self.trl_model_class.from_pretrained(self.tmp_dir)\n\n # Check if the weights are the same\n for key in model_from_save.state_dict():\n torch.testing.assert_close(model_from_save.state_dict()[key], model.state_dict()[key])\n\n def test_from_save_trl_sharded(self):\n \"\"\"\n Test if the model can be saved and loaded from a directory and get the same weights - sharded case\n \"\"\"\n for model_name in self.all_model_names:\n model = self.trl_model_class.from_pretrained(model_name)\n\n model.save_pretrained(self.tmp_dir)\n\n model_from_save = self.trl_model_class.from_pretrained(self.tmp_dir)\n\n # Check if the weights are the same\n for key in model_from_save.state_dict():\n torch.testing.assert_close(model_from_save.state_dict()[key], model.state_dict()[key])\n\n def test_from_save_transformers_sharded(self):\n \"\"\"\n Test if the model can be saved and loaded using transformers and get the same weights - sharded case\n \"\"\"\n for model_name in self.all_model_names:\n transformers_model = self.trl_model_class.transformers_parent_class.from_pretrained(model_name)\n\n trl_model = self.trl_model_class.from_pretrained(model_name)\n\n trl_model.save_pretrained(self.tmp_dir, max_shard_size=\"1MB\")\n transformers_model_from_save = self.trl_model_class.transformers_parent_class.from_pretrained(\n self.tmp_dir\n )\n\n # Check if the weights are the same\n for key in transformers_model.state_dict():\n torch.testing.assert_close(\n transformers_model_from_save.state_dict()[key], transformers_model.state_dict()[key]\n )\n\n def test_from_save_transformers(self):\n \"\"\"\n Test if the model can be saved and loaded using transformers and get the same weights. We override the test\n of the super class to check if the weights are the same.\n \"\"\"\n for model_name in self.all_model_names:\n transformers_model = self.trl_model_class.transformers_parent_class.from_pretrained(model_name)\n\n trl_model = self.trl_model_class.from_pretrained(model_name)\n\n trl_model.save_pretrained(self.tmp_dir)\n transformers_model_from_save = self.trl_model_class.transformers_parent_class.from_pretrained(\n self.tmp_dir\n )\n\n # Check if the weights are the same\n for key in transformers_model.state_dict():\n torch.testing.assert_close(\n transformers_model_from_save.state_dict()[key], transformers_model.state_dict()[key]\n )\n\n # Check if the trl model has the same keys as the transformers model\n # except the v_head\n for key in trl_model.state_dict():\n if \"v_head\" not in key:\n assert key in transformers_model.state_dict()\n # check if the weights are the same\n torch.testing.assert_close(trl_model.state_dict()[key], transformers_model.state_dict()[key])\n\n # check if they have the same modules\n assert set(transformers_model_from_save.state_dict().keys()) == set(\n transformers_model.state_dict().keys()\n )\n\n\nclass TestCausalLMValueHeadModel(BaseTester.VHeadModelTester, TrlTestCase):\n \"\"\"\n Testing suite for v-head models.\n \"\"\"\n\n all_model_names = ALL_CAUSAL_LM_MODELS\n trl_model_class = AutoModelForCausalLMWithValueHead\n transformers_model_class = AutoModelForCausalLM\n\n def teardown_method(self):\n # free memory\n gc.collect()\n\n def test_inference(self):\n r\"\"\"\n Test if the model can be used for inference and outputs 3 values\n - logits, loss, and value states\n \"\"\"\n EXPECTED_OUTPUT_SIZE = 3\n\n for model_name in self.all_model_names:\n model = self.trl_model_class.from_pretrained(model_name).to(self.device)\n input_ids = torch.tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]], device=self.device)\n outputs = model(input_ids)\n\n # Check if the outputs are of the right size - here\n # we always output 3 values - logits, loss, and value states\n assert len(outputs) == EXPECTED_OUTPUT_SIZE\n\n def test_dropout_config(self):\n r\"\"\"\n Test if we instantiate a model by adding `summary_drop_prob` to the config it will be added to the v_head\n \"\"\"\n for model_name in self.all_model_names:\n pretrained_model = self.transformers_model_class.from_pretrained(model_name)\n pretrained_model.config.summary_dropout_prob = 0.5\n model = self.trl_model_class.from_pretrained(pretrained_model)\n\n # Check if v head of the model has the same dropout as the config\n assert model.v_head.dropout.p == pretrained_model.config.summary_dropout_prob\n\n def test_dropout_kwargs(self):\n r\"\"\"\n Test if we instantiate a model by adding `summary_drop_prob` to the config it will be added to the v_head\n \"\"\"\n for model_name in self.all_model_names:\n v_head_kwargs = {\"summary_dropout_prob\": 0.5}\n\n model = self.trl_model_class.from_pretrained(model_name, **v_head_kwargs)\n\n # Check if v head of the model has the same dropout as the config\n assert model.v_head.dropout.p == 0.5\n\n model = self.trl_model_class.from_pretrained(model_name, summary_dropout_prob=0.5)\n\n # Check if v head of the model has the same dropout as the config\n assert model.v_head.dropout.p == 0.5\n\n @pytest.mark.parametrize(\"model_name\", ALL_CAUSAL_LM_MODELS)\n def test_generate(self, model_name):\n r\"\"\"\n Test if `generate` works for every model\n \"\"\"\n generation_config = GenerationConfig(max_new_tokens=9)\n model = self.trl_model_class.from_pretrained(model_name).to(self.device)\n input_ids = torch.tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]], device=self.device)\n\n # Just check if the generation works\n _ = model.generate(input_ids, generation_config=generation_config)\n\n def test_transformers_bf16_kwargs(self):\n r\"\"\"\n Test if the transformers kwargs are correctly passed. Here we check that loading a model in half precision\n works as expected, i.e. the weights of the `pretrained_model` attribute is loaded in half precision and you can\n run a dummy forward pass without any issue.\n \"\"\"\n for model_name in self.all_model_names:\n trl_model = self.trl_model_class.from_pretrained(model_name, dtype=torch.bfloat16).to(self.device)\n\n lm_head_namings = [\"lm_head\", \"embed_out\", \"output_layer\"]\n\n assert any(hasattr(trl_model.pretrained_model, lm_head_naming) for lm_head_naming in lm_head_namings), (\n \"Can't test the model because it doesn't have any of the expected lm_head namings\"\n )\n\n for lm_head_naming in lm_head_namings:\n if hasattr(trl_model.pretrained_model, lm_head_naming):\n assert getattr(trl_model.pretrained_model, lm_head_naming).weight.dtype == torch.bfloat16\n\n dummy_input = torch.LongTensor([[0, 1, 0, 1]]).to(self.device)\n\n # check dummy forward pass works in half precision\n _ = trl_model(dummy_input)\n\n @pytest.mark.skip(reason=\"This test needs to be run manually due to HF token issue.\")\n def test_push_to_hub(self):\n for model_name in self.all_model_names:\n model = AutoModelForCausalLMWithValueHead.from_pretrained(model_name)\n if \"sharded\" in model_name:\n model.push_to_hub(model_name + \"-ppo\", use_auth_token=True, max_shard_size=\"1MB\")\n else:\n model.push_to_hub(model_name + \"-ppo\", use_auth_token=True)\n\n model_from_pretrained = AutoModelForCausalLMWithValueHead.from_pretrained(model_name + \"-ppo\")\n # check all keys\n assert model.state_dict().keys() == model_from_pretrained.state_dict().keys()\n\n for name, param in model.state_dict().items():\n (\n torch.testing.assert_close(param, model_from_pretrained.state_dict()[name]),\n (f\"Parameter {name} is not the same after push_to_hub and from_pretrained\"),\n )\n\n\nclass TestSeq2SeqValueHeadModel(BaseTester.VHeadModelTester, TrlTestCase):\n \"\"\"\n Testing suite for v-head models.\n \"\"\"\n\n all_model_names = ALL_SEQ2SEQ_MODELS\n trl_model_class = AutoModelForSeq2SeqLMWithValueHead\n transformers_model_class = AutoModelForSeq2SeqLM\n\n def teardown_method(self):\n # free memory\n gc.collect()\n\n def test_inference(self):\n r\"\"\"\n Test if the model can be used for inference and outputs 3 values\n - logits, loss, and value states\n \"\"\"\n EXPECTED_OUTPUT_SIZE = 3\n\n for model_name in self.all_model_names:\n model = self.trl_model_class.from_pretrained(model_name).to(self.device)\n input_ids = torch.tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]], device=self.device)\n decoder_input_ids = torch.tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]], device=self.device)\n outputs = model(input_ids, decoder_input_ids=decoder_input_ids)\n\n # Check if the outputs are of the right size - here\n # we always output 3 values - logits, loss, and value states\n assert len(outputs) == EXPECTED_OUTPUT_SIZE\n\n def test_dropout_config(self):\n r\"\"\"\n Test if we instantiate a model by adding `summary_drop_prob` to the config it will be added to the v_head\n \"\"\"\n for model_name in self.all_model_names:\n pretrained_model = self.transformers_model_class.from_pretrained(model_name)\n pretrained_model.config.summary_dropout_prob = 0.5\n model = self.trl_model_class.from_pretrained(pretrained_model)\n\n # Check if v head of the model has the same dropout as the config\n assert model.v_head.dropout.p == pretrained_model.config.summary_dropout_prob\n\n def test_dropout_kwargs(self):\n r\"\"\"\n Test if we instantiate a model by adding `summary_drop_prob` to the config it will be added to the v_head\n \"\"\"\n for model_name in self.all_model_names:\n v_head_kwargs = {\"summary_dropout_prob\": 0.5}\n\n model = self.trl_model_class.from_pretrained(model_name, **v_head_kwargs)\n\n # Check if v head of the model has the same dropout as the config\n assert model.v_head.dropout.p == 0.5\n\n model = self.trl_model_class.from_pretrained(model_name, summary_dropout_prob=0.5)\n\n # Check if v head of the model has the same dropout as the config\n assert model.v_head.dropout.p == 0.5\n\n @pytest.mark.parametrize(\"model_name\", ALL_SEQ2SEQ_MODELS)\n def test_generate(self, model_name):\n r\"\"\"\n Test if `generate` works for every model\n \"\"\"\n generation_config = GenerationConfig(max_new_tokens=9)\n model = self.trl_model_class.from_pretrained(model_name).to(self.device)\n input_ids = torch.tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]], device=self.device)\n decoder_input_ids = torch.tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]], device=self.device)\n\n # Just check if the generation works\n _ = model.generate(input_ids, decoder_input_ids=decoder_input_ids, generation_config=generation_config)\n\n @pytest.mark.skip(reason=\"This test needs to be run manually due to HF token issue.\")\n def test_push_to_hub(self):\n for model_name in self.all_model_names:\n model = self.trl_model_class.from_pretrained(model_name)\n if \"sharded\" in model_name:\n model.push_to_hub(model_name + \"-ppo\", use_auth_token=True, max_shard_size=\"1MB\")\n else:\n model.push_to_hub(model_name + \"-ppo\", use_auth_token=True)\n\n model_from_pretrained = self.trl_model_class.from_pretrained(model_name + \"-ppo\")\n # check all keys\n assert model.state_dict().keys() == model_from_pretrained.state_dict().keys()\n\n for name, param in model.state_dict().items():\n (\n torch.testing.assert_close(param, model_from_pretrained.state_dict()[name]),\n (f\"Parameter {name} is not the same after push_to_hub and from_pretrained\"),\n )\n\n def test_transformers_bf16_kwargs(self):\n r\"\"\"\n Test if the transformers kwargs are correctly passed. Here we check that loading a model in half precision\n works as expected, i.e. the weights of the `pretrained_model` attribute is loaded in half precision and you can\n run a dummy forward pass without any issue.\n \"\"\"\n for model_name in self.all_model_names:\n trl_model = self.trl_model_class.from_pretrained(model_name, dtype=torch.bfloat16).to(self.device)\n\n lm_head_namings = self.trl_model_class.lm_head_namings\n\n assert any(hasattr(trl_model.pretrained_model, lm_head_naming) for lm_head_naming in lm_head_namings)\n\n for lm_head_naming in lm_head_namings:\n if hasattr(trl_model.pretrained_model, lm_head_naming):\n assert getattr(trl_model.pretrained_model, lm_head_naming).weight.dtype == torch.bfloat16\n\n dummy_input = torch.LongTensor([[0, 1, 0, 1]]).to(self.device)\n\n # check dummy forward pass works in half precision\n _ = trl_model(input_ids=dummy_input, decoder_input_ids=dummy_input)\n\n\n@require_peft\nclass TestPeftModel(TrlTestCase):\n def setup_method(self):\n self.causal_lm_model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.lora_config = LoraConfig(\n r=16,\n lora_alpha=32,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n )\n\n def test_create_peft_model(self):\n r\"\"\"\n Simply creates a peft model and checks that it can be loaded.\n \"\"\"\n causal_lm_model = AutoModelForCausalLM.from_pretrained(self.causal_lm_model_id)\n pretrained_model = get_peft_model(causal_lm_model, self.lora_config)\n\n _ = AutoModelForCausalLMWithValueHead.from_pretrained(pretrained_model)\n\n def test_peft_requires_grad(self):\n r\"\"\"\n Check that the value head of the returned model has requires_grad=True.\n \"\"\"\n causal_lm_model = AutoModelForCausalLM.from_pretrained(self.causal_lm_model_id)\n pretrained_model = get_peft_model(causal_lm_model, self.lora_config)\n\n model = AutoModelForCausalLMWithValueHead.from_pretrained(pretrained_model)\n\n # Check that the value head has requires_grad=True\n assert model.v_head.summary.weight.requires_grad\n\n def test_check_peft_model_nb_trainable_params(self):\n r\"\"\"\n Check that the number of trainable parameters is correct.\n \"\"\"\n causal_lm_model = AutoModelForCausalLM.from_pretrained(self.causal_lm_model_id)\n pretrained_model = get_peft_model(causal_lm_model, self.lora_config)\n\n model = AutoModelForCausalLMWithValueHead.from_pretrained(pretrained_model)\n\n # Check that the number of trainable parameters is correct\n nb_trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n assert nb_trainable_params == 905\n\n # Check that the number of trainable param for the non-peft model is correct\n non_peft_model = AutoModelForCausalLMWithValueHead.from_pretrained(self.causal_lm_model_id)\n nb_trainable_params = sum(p.numel() for p in non_peft_model.parameters() if p.requires_grad)\n assert nb_trainable_params == 2428641\n\n def test_create_peft_model_from_config(self):\n r\"\"\"\n Simply creates a peft model and checks that it can be loaded.\n \"\"\"\n trl_model = AutoModelForCausalLMWithValueHead.from_pretrained(\n self.causal_lm_model_id, peft_config=self.lora_config\n )\n # Check that the number of trainable parameters is correct\n nb_trainable_params = sum(p.numel() for p in trl_model.parameters() if p.requires_grad)\n assert nb_trainable_params == 905\n\n causal_lm_model = AutoModelForCausalLM.from_pretrained(self.causal_lm_model_id)\n trl_model = AutoModelForCausalLMWithValueHead.from_pretrained(causal_lm_model, peft_config=self.lora_config)\n # Check that the number of trainable parameters is correct\n nb_trainable_params = sum(p.numel() for p in trl_model.parameters() if p.requires_grad)\n assert nb_trainable_params == 905\n\n @require_bitsandbytes\n @require_torch_gpu_if_bnb_not_multi_backend_enabled\n def test_create_bnb_peft_model_from_config(self):\n r\"\"\"\n Simply creates a peft model and checks that it can be loaded.\n \"\"\"\n from bitsandbytes.nn import Linear8bitLt\n from transformers import BitsAndBytesConfig\n\n trl_model = AutoModelForCausalLMWithValueHead.from_pretrained(\n self.causal_lm_model_id,\n peft_config=self.lora_config,\n quantization_config=BitsAndBytesConfig(load_in_8bit=True),\n )\n # Check that the number of trainable parameters is correct\n nb_trainable_params = sum(p.numel() for p in trl_model.parameters() if p.requires_grad)\n assert nb_trainable_params == 905\n assert isinstance(trl_model.pretrained_model.model.model.layers[0].mlp.gate_proj, Linear8bitLt)\n\n causal_lm_model = AutoModelForCausalLM.from_pretrained(\n self.causal_lm_model_id, quantization_config=BitsAndBytesConfig(load_in_8bit=True), device_map=\"auto\"\n )\n trl_model = AutoModelForCausalLMWithValueHead.from_pretrained(causal_lm_model, peft_config=self.lora_config)\n # Check that the number of trainable parameters is correct\n nb_trainable_params = sum(p.numel() for p in trl_model.parameters() if p.requires_grad)\n assert nb_trainable_params == 905\n assert isinstance(trl_model.pretrained_model.model.model.layers[0].mlp.gate_proj, Linear8bitLt)\n\n def test_save_pretrained_peft(self):\n r\"\"\"\n Check that the model can be saved and loaded properly.\n \"\"\"\n causal_lm_model = AutoModelForCausalLM.from_pretrained(self.causal_lm_model_id)\n pretrained_model = get_peft_model(causal_lm_model, self.lora_config)\n\n model = AutoModelForCausalLMWithValueHead.from_pretrained(pretrained_model)\n\n model.save_pretrained(self.tmp_dir)\n\n # check that the files `adapter_model.safetensors` and `adapter_config.json` are in the directory\n assert os.path.isfile(f\"{self.tmp_dir}/adapter_model.safetensors\"), (\n f\"{self.tmp_dir}/adapter_model.safetensors does not exist\"\n )\n assert os.path.exists(f\"{self.tmp_dir}/adapter_config.json\"), (\n f\"{self.tmp_dir}/adapter_config.json does not exist\"\n )\n\n # check also for `pytorch_model.bin` and make sure it only contains `v_head` weights\n assert os.path.exists(f\"{self.tmp_dir}/pytorch_model.bin\"), f\"{self.tmp_dir}/pytorch_model.bin does not exist\"\n\n # check that only keys that starts with `v_head` are in the dict\n maybe_v_head = torch.load(f\"{self.tmp_dir}/pytorch_model.bin\", weights_only=True)\n assert all(k.startswith(\"v_head\") for k in maybe_v_head.keys()), (\n f\"keys in {self.tmp_dir}/pytorch_model.bin do not start with `v_head`\"\n )\n\n model_from_pretrained = AutoModelForCausalLMWithValueHead.from_pretrained(self.tmp_dir)\n\n # check all the weights are the same\n for p1, p2 in zip(model.named_parameters(), model_from_pretrained.named_parameters(), strict=True):\n torch.testing.assert_close(p1[1], p2[1]), f\"{p1[0]} != {p2[0]}\"\n\n def test_load_pretrained_peft(self):\n r\"\"\"\n Check that the model saved with peft class interface can be loaded properly.\n \"\"\"\n causal_lm_model = AutoModelForCausalLM.from_pretrained(self.causal_lm_model_id)\n pretrained_model = get_peft_model(causal_lm_model, self.lora_config)\n\n model = AutoModelForCausalLMWithValueHead.from_pretrained(pretrained_model)\n\n pretrained_model.save_pretrained(self.tmp_dir)\n model_from_pretrained = AutoModelForCausalLMWithValueHead.from_pretrained(self.tmp_dir)\n\n # check that the files `adapter_model.safetensors` and `adapter_config.json` are in the directory\n assert os.path.isfile(f\"{self.tmp_dir}/adapter_model.safetensors\"), (\n f\"{self.tmp_dir}/adapter_model.safetensors does not exist\"\n )\n assert os.path.exists(f\"{self.tmp_dir}/adapter_config.json\"), (\n f\"{self.tmp_dir}/adapter_config.json does not exist\"\n )\n\n # check all the weights are the same\n for p1, p2 in zip(model.named_parameters(), model_from_pretrained.named_parameters(), strict=True):\n if p1[0] not in [\"v_head.summary.weight\", \"v_head.summary.bias\"]:\n torch.testing.assert_close(p1[1], p2[1]), f\"{p1[0]} != {p2[0]}\"\n\n def test_continue_training_peft_model(self):\n r\"\"\"\n Load peft and checks that it can continue training.\n \"\"\"\n causal_lm_model = AutoModelForCausalLM.from_pretrained(self.causal_lm_model_id)\n pretrained_model = get_peft_model(causal_lm_model, self.lora_config)\n\n pretrained_model.save_pretrained(self.tmp_dir)\n # set is_trainable to True\n model = AutoModelForCausalLMWithValueHead.from_pretrained(self.tmp_dir, is_trainable=True)\n # Check that the number of trainable parameters is correct\n nb_trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n assert nb_trainable_params == 905\n\n\nclass TestCore(TrlTestCase):\n \"\"\"\n A wrapper class for testing core utils functions\n \"\"\"\n\n def setup_method(self):\n self.test_input = torch.Tensor([1, 2, 3, 4])\n self.test_mask = torch.Tensor([0, 1, 1, 0])\n self.test_input_unmasked = self.test_input[1:3]\n\n def test_masked_mean(self):\n assert torch.mean(self.test_input_unmasked) == masked_mean(self.test_input, self.test_mask)\n\n def test_masked_var(self):\n assert torch.var(self.test_input_unmasked) == masked_var(self.test_input, self.test_mask)\n\n def test_masked_whiten(self):\n def whiten(values: torch.Tensor) -> torch.Tensor:\n mean, var = torch.mean(values), torch.var(values)\n return (values - mean) * torch.rsqrt(var + 1e-8)\n\n whiten_unmasked = whiten(self.test_input_unmasked)\n whiten_masked = masked_whiten(self.test_input, self.test_mask)[1:3]\n diffs = (whiten_unmasked - whiten_masked).sum()\n assert abs(diffs.item()) < 0.00001\n\n\nclass TestPPOTrainer(TrlTestCase):\n def setup_method(self):\n # Set up the models and tokenizer using the test model\n self.model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.model = AutoModelForCausalLM.from_pretrained(self.model_id, dtype=\"float32\")\n self.ref_model = AutoModelForCausalLM.from_pretrained(self.model_id)\n self.tokenizer = AutoTokenizer.from_pretrained(self.model_id, padding_side=\"left\")\n self.tokenizer.add_special_tokens({\"pad_token\": \"[PAD]\"})\n\n # Add reward and value models as in ppo.py\n reward_model_id = \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n self.value_model = AutoModelForSequenceClassification.from_pretrained(reward_model_id, num_labels=1)\n self.reward_model = AutoModelForSequenceClassification.from_pretrained(reward_model_id, num_labels=1)\n\n # Load dataset\n raw_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n def tokenize(example, tokenizer):\n tokenized = tokenizer(text=example[\"prompt\"])\n if tokenizer.eos_token_id is not None and tokenized[\"input_ids\"][-1] != tokenizer.eos_token_id:\n tokenized[\"input_ids\"] = tokenized[\"input_ids\"] + [tokenizer.eos_token_id]\n tokenized[\"attention_mask\"] = tokenized[\"attention_mask\"] + [1]\n return tokenized\n\n self.raw_dataset = raw_dataset.map(tokenize, fn_kwargs={\"tokenizer\": self.tokenizer}, remove_columns=\"prompt\")\n\n def test_basic_training(self):\n \"\"\"Test basic PPO training configuration and verify model updates.\"\"\"\n # Capture initial weights\n initial_critic_weights = {}\n initial_policy_weights = {}\n for name, param in self.value_model.named_parameters():\n initial_critic_weights[name] = param.clone().detach()\n for name, param in self.model.named_parameters():\n initial_policy_weights[name] = param.clone().detach()\n\n # Configure training args similar to example script\n training_args = PPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=4,\n per_device_eval_batch_size=2,\n num_ppo_epochs=2, # Decrease number of PPO epochs to speed up test\n report_to=\"none\",\n )\n\n # Create trainer\n trainer = PPOTrainer(\n args=training_args,\n processing_class=self.tokenizer,\n model=self.model,\n ref_model=self.ref_model,\n reward_model=self.reward_model,\n value_model=self.value_model,\n train_dataset=self.raw_dataset[\"train\"],\n eval_dataset=self.raw_dataset[\"test\"],\n )\n\n # Train\n trainer.train()\n\n # Check if critic weights have been updated\n critic_weights_updated = False\n for name, param in trainer.model.value_model.named_parameters():\n if not torch.allclose(initial_critic_weights[name], param.to(\"cpu\")):\n critic_weights_updated = True\n break\n\n # Check if policy weights have been updated\n policy_weights_updated = False\n for name, param in trainer.model.policy.named_parameters():\n if not torch.allclose(initial_policy_weights[name], param.to(\"cpu\")):\n policy_weights_updated = True\n break\n\n assert critic_weights_updated, \"Critic weights were not updated during training\"\n assert policy_weights_updated, \"Policy weights were not updated during training\"\n\n @require_peft\n def test_peft_training(self):\n \"\"\"Test PPO training with PEFT configuration and verify model updates.\"\"\"\n # Capture initial weights\n initial_critic_weights = {}\n initial_policy_weights = {}\n for name, param in self.value_model.named_parameters():\n initial_critic_weights[name] = param.clone().detach()\n for name, param in self.model.named_parameters():\n initial_policy_weights[name] = param.clone().detach()\n\n # Configure training args\n training_args = PPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=4,\n per_device_eval_batch_size=2,\n num_ppo_epochs=2, # Decrease number of PPO epochs to speed up test\n report_to=\"none\",\n )\n\n # Configure PEFT\n peft_config = LoraConfig(\n r=32,\n lora_alpha=16,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n )\n\n # Create trainer with PEFT\n trainer = PPOTrainer(\n args=training_args,\n processing_class=self.tokenizer,\n model=self.model,\n ref_model=None,\n reward_model=self.reward_model,\n value_model=self.value_model,\n train_dataset=self.raw_dataset[\"train\"],\n eval_dataset=self.raw_dataset[\"test\"],\n peft_config=peft_config,\n )\n\n # Train\n trainer.train()\n\n # Check if critic weights have been updated\n critic_weights_updated = False\n for name, param in trainer.model.value_model.named_parameters():\n if name in initial_critic_weights and not torch.allclose(initial_critic_weights[name], param.to(\"cpu\")):\n critic_weights_updated = True\n break\n\n # Check if policy weights have been updated - for PEFT we check the LoRA weights\n policy_weights_updated = False\n for name, param in trainer.model.policy.named_parameters():\n if \"lora\" in name.lower() and param.requires_grad: # Only check LoRA weights\n # New weights should be non-zero if they've been updated\n if not torch.allclose(param, torch.zeros_like(param)):\n policy_weights_updated = True\n break\n\n assert critic_weights_updated, \"Critic weights were not updated during training\"\n assert policy_weights_updated, \"Policy LoRA weights were not updated during training\"\n" }, { "path": "tests/experimental/test_prm_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom unittest.mock import MagicMock\n\nimport numpy as np\nimport pytest\nimport torch\nfrom datasets import Dataset, load_dataset\nfrom transformers import AutoModelForTokenClassification, AutoTokenizer, PreTrainedTokenizerBase\nfrom transformers.utils import is_peft_available\n\nfrom trl.experimental.prm import PRMConfig, PRMTrainer\nfrom trl.experimental.prm.prm_trainer import compute_accuracy\n\nfrom ..testing_utils import TrlTestCase, require_peft\n\n\nif is_peft_available():\n from peft import LoraConfig, TaskType\n\n\nclass TestComputeAccuracy(TrlTestCase):\n def test_token_classification_task(self):\n eval_pred = (\n np.array(\n [\n [[0.1, 0.9], [0.8, 0.2]], # Batch 1\n [[0.3, 0.7], [0.6, 0.4]], # Batch 2\n ]\n ),\n np.array([[0, 1], [1, 0]]),\n )\n expected_accuracy = 0.5 # 2 matches, 2 mismatches\n result = compute_accuracy(eval_pred)\n assert round(abs(result[\"accuracy\"] - expected_accuracy), 7) == 0\n\n def test_token_classification_task_with_ignored_tokens_0(self):\n eval_pred = (\n np.array(\n [\n [[0.1, 0.9], [0.8, 0.2]], # Batch 1\n [[0.3, 0.7], [0.6, 0.4]], # Batch 2\n ]\n ),\n np.array([[1, 0], [1, -100]]),\n )\n expected_accuracy = 1.0 # All non-ignored tokens match\n result = compute_accuracy(eval_pred)\n assert round(abs(result[\"accuracy\"] - expected_accuracy), 7) == 0\n\n def test_token_classification_task_with_ignored_tokens_1(self):\n eval_pred = (\n np.array(\n [\n [[0.1, 0.9], [0.8, 0.2]], # Batch 1\n [[0.3, 0.7], [0.6, 0.4]], # Batch 2\n ]\n ),\n np.array([[1, 1], [0, -100]]),\n )\n expected_accuracy = 1 / 3 # 1 match, 2 mismatch, 1 ignored\n result = compute_accuracy(eval_pred)\n assert round(abs(result[\"accuracy\"] - expected_accuracy), 7) == 0\n\n def test_rewards_comparison_task(self, caplog):\n eval_pred = (\n np.array(\n [\n [0.9, 0.1], # Batch 1\n [0.6, 0.4], # Batch 2\n [0.5, 0.5], # Batch 3 (equal)\n ]\n ),\n np.array([0, 1, 1]),\n )\n expected_accuracy = 0.5 # 1 match, 1 mismatch, 1 equal (ignored)\n\n with caplog.at_level(\"WARNING\", logger=\"trl.trainer.utils\"):\n result = compute_accuracy(eval_pred)\n\n assert round(abs(result[\"accuracy\"] - expected_accuracy), 7) == 0\n expected_warning = (\n \"There are 1 out of 3 instances where the predictions for both options are equal. \"\n \"These instances are ignored in the accuracy computation.\"\n )\n assert expected_warning in caplog.text\n\n\nclass TestTokenizeRow(TrlTestCase):\n def setup_method(self):\n # Set up the mock tokenizer with specific behaviors\n self.tokenizer = MagicMock(spec=PreTrainedTokenizerBase)\n self.tokenizer.bos_token_id = 0\n self.tokenizer.eos_token_id = 2\n\n def mock_encode(text, add_special_tokens):\n token_map = {\n \"Which number is larger, 9.8 or 9.11?\": [465, 6766, 318, 298],\n \"11 is greater than 8.\": [4, 322, 12],\n \"Hence, 9.11 > 9.8.\": [4995, 11, 22],\n \"\\n\": [1030],\n \"\\n\\n\": [1030, 1030],\n }\n\n return token_map[text]\n\n def mock_tokenizer_call(text, add_special_tokens):\n return {\"input_ids\": mock_encode(text, add_special_tokens)}\n\n self.tokenizer.encode.side_effect = mock_encode\n self.tokenizer.side_effect = mock_tokenizer_call\n\n def test_tokenize_row_no_truncation(self):\n # Define the input features\n features = {\n \"prompt\": \"Which number is larger, 9.8 or 9.11?\",\n \"completions\": [\"11 is greater than 8.\", \"Hence, 9.11 > 9.8.\"],\n \"labels\": [True, False],\n }\n\n # Call the method with no truncation\n result = PRMTrainer.tokenize_row(\n features=features,\n tokenizer=self.tokenizer,\n step_separator=\"\\n\",\n max_length=None,\n max_completion_length=None,\n train_on_last_step_only=False,\n is_eval=False,\n )\n\n assert result == {\n \"input_ids\": [0, 465, 6766, 318, 298, 4, 322, 12, 1030, 4995, 11, 22, 1030],\n \"labels\": [-100, -100, -100, -100, -100, -100, -100, -100, 1, -100, -100, -100, 0],\n }\n\n def test_tokenize_row_train_on_last_step_only(self):\n # Define the input features\n features = {\n \"prompt\": \"Which number is larger, 9.8 or 9.11?\",\n \"completions\": [\"11 is greater than 8.\", \"Hence, 9.11 > 9.8.\"],\n \"labels\": [True, False],\n }\n\n result = PRMTrainer.tokenize_row(\n features=features,\n tokenizer=self.tokenizer,\n step_separator=\"\\n\",\n max_length=None,\n max_completion_length=None,\n train_on_last_step_only=True,\n is_eval=False,\n )\n\n assert result == {\n \"input_ids\": [0, 465, 6766, 318, 298, 4, 322, 12, 1030, 4995, 11, 22, 1030],\n \"labels\": [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 0],\n }\n\n def test_tokenize_row_completion_truncation(self):\n # Define the input features\n features = {\n \"prompt\": \"Which number is larger, 9.8 or 9.11?\",\n \"completions\": [\"11 is greater than 8.\", \"Hence, 9.11 > 9.8.\"],\n \"labels\": [True, False],\n }\n\n # Call the method with truncation on the completion\n result = PRMTrainer.tokenize_row(\n features=features,\n tokenizer=self.tokenizer,\n step_separator=\"\\n\",\n max_length=None,\n max_completion_length=6,\n train_on_last_step_only=False,\n is_eval=False,\n )\n\n assert result == {\n \"input_ids\": [0, 465, 6766, 318, 298, 4, 322, 12, 1030, 4995, 11],\n \"labels\": [-100, -100, -100, -100, -100, -100, -100, -100, 1, -100, -100],\n }\n\n def test_tokenize_row_prompt_completion_truncation(self):\n # Define the input features\n features = {\n \"prompt\": \"Which number is larger, 9.8 or 9.11?\",\n \"completions\": [\"11 is greater than 8.\", \"Hence, 9.11 > 9.8.\"],\n \"labels\": [True, False],\n }\n\n # Call the method with truncation on the prompt and completion\n result = PRMTrainer.tokenize_row(\n features=features,\n tokenizer=self.tokenizer,\n step_separator=\"\\n\",\n max_length=9,\n max_completion_length=None,\n train_on_last_step_only=False,\n is_eval=False,\n )\n\n assert result == {\n \"input_ids\": [0, 465, 6766, 318, 298, 4, 322, 12, 1030],\n \"labels\": [-100, -100, -100, -100, -100, -100, -100, -100, 1],\n }\n\n def test_tokenize_row_multi_token_separator(self):\n # Define the input features\n features = {\n \"prompt\": \"Which number is larger, 9.8 or 9.11?\",\n \"completions\": [\"11 is greater than 8.\", \"Hence, 9.11 > 9.8.\"],\n \"labels\": [True, False],\n }\n\n # Call the method using multiple tokens as step_separator\n result = PRMTrainer.tokenize_row(\n features=features,\n tokenizer=self.tokenizer,\n step_separator=\"\\n\\n\",\n max_length=None,\n max_completion_length=None,\n train_on_last_step_only=False,\n is_eval=False,\n )\n\n assert result == {\n \"input_ids\": [0, 465, 6766, 318, 298, 4, 322, 12, 1030, 1030, 4995, 11, 22, 1030, 1030],\n \"labels\": [-100, -100, -100, -100, -100, -100, -100, -100, -100, 1, -100, -100, -100, -100, 0],\n }\n\n\nclass TestPRMTrainer(TrlTestCase):\n def setup_method(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.model = AutoModelForTokenClassification.from_pretrained(model_id, dtype=\"float32\")\n self.tokenizer = AutoTokenizer.from_pretrained(model_id)\n\n @pytest.mark.parametrize(\"train_on_last_step_only\", [True, False])\n def test_train_full(self, train_on_last_step_only):\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_stepwise_supervision\", split=\"train\")\n training_args = PRMConfig(\n output_dir=self.tmp_dir,\n report_to=\"none\",\n train_on_last_step_only=train_on_last_step_only,\n )\n trainer = PRMTrainer(\n model=self.model, args=training_args, processing_class=self.tokenizer, train_dataset=dummy_dataset\n )\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12)\n\n def test_train_full_pretokenized(self):\n dummy_dataset = Dataset.from_dict(\n {\n \"labels\": [\n [-100, -100, -100, -100, -100, -100, -100, -100, -100, 0, -100, -100, 1],\n [-100, -100, -100, -100, -100, -100, -100, -100, 0, -100, -100, 1, -100, -100, -100, -100, 0],\n [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 0, -100, -100, 1],\n [-100, -100, -100, -100, -100, -100, -100, 1, -100, -100, 1],\n [-100, -100, -100, -100, -100, -100, -100, -100, -100, 1, -100, -100, 0],\n [-100, -100, -100, -100, -100, -100, -100, -100, -100, 1],\n [-100, -100, -100, -100, -100, -100, -100, -100, -100, 0],\n [-100, -100, -100, -100, -100, -100, -100, -100, -100, 1, -100, -100, -100, -100, -100, 0],\n [-100, -100, -100, -100, -100, -100, -100, -100, 0, -100, -100, 0],\n [-100, -100, -100, -100, -100, -100, 0, -100, -100, -100, -100, 0],\n [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 1],\n [-100, -100, -100, -100, -100, -100, 0],\n [-100, -100, -100, -100, -100, -100, -100, -100, 1],\n [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 0],\n ],\n \"input_ids\": [\n [46518, 374, 2664, 1091, 11, 1077, 752, 1744, 1112, 198, 27261, 13, 198],\n [98923, 374, 2664, 1091, 11, 315, 3308, 11, 198, 17995, 13, 198, 1576, 31273, 12850, 13, 198],\n [16374, 374, 2664, 1091, 1112, 1077, 594, 2506, 432, 6770, 11, 198, 6351, 13, 198],\n [31137, 374, 2664, 1091, 979, 4362, 11, 198, 16965, 13, 198],\n [31019, 374, 2664, 1091, 304, 3793, 315, 5944, 11, 198, 24034, 13, 198],\n [98491, 374, 2664, 1091, 1112, 5310, 369, 91494, 13, 198],\n [4418, 2897, 14579, 5310, 979, 3800, 1349, 432, 13, 198],\n [20366, 5048, 7629, 944, 3281, 3322, 11, 7241, 1112, 198, 807, 1795, 279, 5601, 13, 198],\n [15802, 14976, 487, 33327, 1045, 31787, 63443, 11, 198, 52400, 13, 198],\n [13877, 1265, 2581, 1494, 49394, 11, 198, 7241, 20975, 91681, 13, 198],\n [641, 279, 3579, 315, 71768, 11, 25066, 279, 61361, 311, 7942, 13, 198],\n [7039, 374, 2664, 1091, 2937, 13, 198],\n [26155, 374, 3545, 2664, 1091, 34933, 26537, 13, 198],\n [2679, 279, 8129, 374, 4135, 311, 10339, 11, 432, 2578, 387, 264, 1661, 2884, 13, 198],\n ],\n }\n )\n\n training_args = PRMConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = PRMTrainer(\n model=self.model, args=training_args, processing_class=self.tokenizer, train_dataset=dummy_dataset\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if param.sum() != 0: # ignore 0 biases\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12)\n\n @require_peft\n def test_train_lora(self):\n peft_config = LoraConfig(\n task_type=TaskType.TOKEN_CLS,\n inference_mode=False,\n r=8,\n lora_alpha=32,\n lora_dropout=0.1,\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_stepwise_supervision\", split=\"train\")\n training_args = PRMConfig(output_dir=self.tmp_dir, max_steps=3, report_to=\"none\")\n trainer = PRMTrainer(\n model=self.model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset,\n peft_config=peft_config,\n )\n previous_trainable_params = {}\n previous_non_trainable_params = {}\n\n # due to a change in the way the modules to save are dealt in PEFT.\n trainable_params_name = [\"lora\", \"modules_to_save\"]\n\n # check gradients are not None\n for n, param in trainer.model.named_parameters():\n if any(t in n for t in trainable_params_name):\n previous_trainable_params[n] = param.clone()\n else:\n previous_non_trainable_params[n] = param.clone()\n\n trainer.train()\n\n assert trainer.state.log_history[(-1)][\"train_loss\"] is not None\n\n # Check that the parameters have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param, atol=1e-12, rtol=1e-12)\n\n # Check that the non trainable parameters have not changed\n for n, param in previous_non_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n torch.testing.assert_close(param, new_param, atol=1e-12, rtol=1e-12)\n\n def test_tags(self):\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_stepwise_supervision\", split=\"train\")\n training_args = PRMConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = PRMTrainer(\n model=self.model, args=training_args, processing_class=self.tokenizer, train_dataset=dummy_dataset\n )\n assert trainer.model.model_tags == trainer._tag_names\n" }, { "path": "tests/experimental/test_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\nfrom datasets import load_dataset\nfrom transformers import AutoTokenizer\n\nfrom trl.experimental.utils import DataCollatorForChatML\n\nfrom ..testing_utils import TrlTestCase\n\n\nclass TestDataCollatorForChatML(TrlTestCase):\n def setup_method(self):\n # Initialize the tokenizer\n self.tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n if self.tokenizer.pad_token is None:\n self.tokenizer.pad_token = self.tokenizer.eos_token\n\n # Define token IDs\n self.bos_token_id = self.tokenizer.bos_token_id if self.tokenizer.bos_token_id is not None else 1\n self.eos_token_id = self.tokenizer.eos_token_id if self.tokenizer.eos_token_id is not None else 2\n # Token ID for \"true\", the last assistant's response in the example:\n self.ignore_index = -100\n self.max_length = 1024\n self.messages_key = \"messages\"\n\n # Example input\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\", split=\"train\")\n self.examples = dataset.to_list()\n\n # Initialize the data collator\n self.collator = DataCollatorForChatML(\n tokenizer=self.tokenizer,\n max_length=self.max_length,\n ignore_index=self.ignore_index,\n )\n\n def test_data_collator_for_chatml(self):\n # Process the data\n data = self.collator(self.examples)\n\n # Verify basic shapes and types\n assert \"input_ids\" in data\n assert \"attention_mask\" in data\n assert \"labels\" in data\n assert \"prompts\" in data\n assert \"prompt_attention_mask\" in data\n\n # Decode input_ids and labels for verification\n input_ids = data[\"input_ids\"][0].tolist()\n labels = data[\"labels\"][0].tolist()\n prompt_only = data[\"prompts\"][0].tolist()\n\n # Get the last assistant's response for comparison\n last_message = self.examples[0][self.messages_key][-1]\n assert last_message[\"role\"] == \"assistant\", \"Last message should be from assistant\"\n last_assistant_response = last_message[\"content\"]\n\n # Verify that input_ids contain both prompt and response\n decoded_input = self.tokenizer.decode(input_ids)\n assert last_assistant_response in decoded_input, \"Input should contain assistant's response\"\n\n # Verify that prompts only contain the conversation up to the last response\n decoded_prompt = self.tokenizer.decode(prompt_only)\n assert last_assistant_response not in decoded_prompt, \"Prompt should not contain assistant's response\"\n\n # Verify labels are -100 for non-assistant parts\n prompt_length = len(prompt_only)\n assert all(label == self.ignore_index for label in labels[:prompt_length]), (\n \"Labels should be ignore_index for prompt tokens\"\n )\n\n # Verify labels match assistant response after prompt\n # Add a filter to remove any trailing tokens after the first <|im_end|>\n last_assistant_response_with_end = last_assistant_response + self.tokenizer.eos_token\n last_assistant_response_tokens = self.tokenizer.encode(\n last_assistant_response_with_end, add_special_tokens=False\n )\n\n response_labels = []\n for label in labels[prompt_length:]:\n if label == self.ignore_index:\n continue\n response_labels.append(label)\n if label == self.tokenizer.convert_tokens_to_ids(\"<|im_end|>\"):\n break\n assert response_labels == last_assistant_response_tokens, \"Labels should match assistant response tokens\"\n\n # Verify there isn't a generation prompt at the end\n generation_prompt = \"<|im_start|>assistant\"\n assert not decoded_input.strip().endswith(generation_prompt), (\n f\"Input should not end with generation prompt '{generation_prompt}'\"\n )\n\n assert response_labels == last_assistant_response_tokens, \"Labels should match assistant response tokens\"\n" }, { "path": "tests/experimental/test_winrate_callback.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig, Trainer, TrainingArguments\nfrom transformers.utils import is_peft_available\n\nfrom trl.experimental.judges import BasePairwiseJudge\nfrom trl.experimental.winrate_callback import WinRateCallback\n\nfrom ..testing_utils import TrlTestCase, require_peft\n\n\nif is_peft_available():\n from peft import LoraConfig\n\n\nclass HalfPairwiseJudge(BasePairwiseJudge):\n \"\"\"Naive pairwise judge that always returns [1, 0] for two prompts\"\"\"\n\n def judge(self, prompts, completions, shuffle_order=True, return_scores=False):\n # just check that the batch size is 2\n assert len(prompts) == 2\n if return_scores:\n return [0.3, 0.9]\n return [1, 0]\n\n\nclass TrainerWithRefModel(Trainer):\n # This is a dummy class to test the callback. Compared to the Trainer class, it only has an additional\n # ref_model attribute\n def __init__(self, model, ref_model, args, train_dataset, eval_dataset, processing_class):\n super().__init__(\n model=model,\n args=args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n )\n # Prepare ref_model like TRL trainers do (DPOTrainer, GRPOTrainer, etc.)\n self.ref_model = self.accelerator.prepare_model(ref_model, evaluation_mode=True)\n\n\nclass TestWinRateCallback(TrlTestCase):\n def setup_method(self):\n self.model = AutoModelForCausalLM.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\"\n )\n self.ref_model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n self.tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n self.tokenizer.pad_token = self.tokenizer.eos_token\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n dataset[\"train\"] = dataset[\"train\"].select(range(8))\n self.expected_winrates = [\n {\"eval_win_rate\": 0.5, \"epoch\": 0.0, \"step\": 0},\n {\"eval_win_rate\": 0.5, \"epoch\": 0.5, \"step\": 2},\n {\"eval_win_rate\": 0.5, \"epoch\": 1.0, \"step\": 4},\n {\"eval_win_rate\": 0.5, \"epoch\": 1.5, \"step\": 6},\n {\"eval_win_rate\": 0.5, \"epoch\": 2.0, \"step\": 8},\n {\"eval_win_rate\": 0.5, \"epoch\": 2.5, \"step\": 10},\n {\"eval_win_rate\": 0.5, \"epoch\": 3.0, \"step\": 12},\n ]\n\n def tokenize_function(examples):\n out = self.tokenizer(examples[\"prompt\"], padding=\"max_length\", max_length=16, truncation=True)\n out[\"labels\"] = out[\"input_ids\"].copy()\n return out\n\n self.dataset = dataset.map(tokenize_function, batched=True)\n\n self.generation_config = GenerationConfig(max_length=32)\n self.judge = HalfPairwiseJudge()\n\n def test_basic(self):\n training_args = TrainingArguments(\n output_dir=self.tmp_dir,\n eval_strategy=\"steps\",\n eval_steps=2, # evaluate every 2 steps\n per_device_train_batch_size=2, # 8 samples in total so 4 batches of 2 per epoch\n per_device_eval_batch_size=2,\n report_to=\"none\",\n )\n trainer = TrainerWithRefModel(\n model=self.model,\n ref_model=self.ref_model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n eval_dataset=self.dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n win_rate_callback = WinRateCallback(\n judge=self.judge, trainer=trainer, generation_config=self.generation_config\n )\n trainer.add_callback(win_rate_callback)\n trainer.train()\n winrate_history = [h for h in trainer.state.log_history if \"eval_win_rate\" in h]\n for history_row, expected_row in zip(winrate_history, self.expected_winrates, strict=True):\n assert all(key in history_row and history_row[key] == expected_row[key] for key in expected_row)\n\n def test_without_ref_model(self):\n # Same as before, but without the ref_model attribute. It should use the model attribute instead\n training_args = TrainingArguments(\n output_dir=self.tmp_dir,\n eval_strategy=\"steps\",\n eval_steps=2, # evaluate every 2 steps\n per_device_train_batch_size=2, # 8 samples in total so 4 batches of 2 per epoch\n per_device_eval_batch_size=2,\n report_to=\"none\",\n )\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n eval_dataset=self.dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n win_rate_callback = WinRateCallback(\n judge=self.judge, trainer=trainer, generation_config=self.generation_config\n )\n trainer.add_callback(win_rate_callback)\n trainer.train()\n winrate_history = [h for h in trainer.state.log_history if \"eval_win_rate\" in h]\n for history_row, expected_row in zip(winrate_history, self.expected_winrates, strict=True):\n assert all(key in history_row and history_row[key] == expected_row[key] for key in expected_row)\n\n def test_soft_judge(self):\n \"\"\"Test that the soft judge functionality works correctly\"\"\"\n training_args = TrainingArguments(\n output_dir=self.tmp_dir,\n eval_strategy=\"steps\",\n eval_steps=2, # evaluate every 2 steps\n per_device_train_batch_size=2, # 8 samples in total so 4 batches of 2 per epoch\n per_device_eval_batch_size=2,\n report_to=\"none\",\n )\n trainer = TrainerWithRefModel(\n model=self.model,\n ref_model=self.ref_model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n eval_dataset=self.dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n win_rate_callback = WinRateCallback(\n judge=self.judge, trainer=trainer, generation_config=self.generation_config, use_soft_judge=True\n )\n trainer.add_callback(win_rate_callback)\n trainer.train()\n\n # Expected values based on judge returning [0.3, 0.9] for each pair\n expected_soft_winrates = [\n {\"eval_avg_win_prob\": 0.4, \"eval_win_rate\": 0.5, \"epoch\": 0.0, \"step\": 0},\n {\"eval_avg_win_prob\": 0.4, \"eval_win_rate\": 0.5, \"epoch\": 0.5, \"step\": 2},\n {\"eval_avg_win_prob\": 0.4, \"eval_win_rate\": 0.5, \"epoch\": 1.0, \"step\": 4},\n {\"eval_avg_win_prob\": 0.4, \"eval_win_rate\": 0.5, \"epoch\": 1.5, \"step\": 6},\n {\"eval_avg_win_prob\": 0.4, \"eval_win_rate\": 0.5, \"epoch\": 2.0, \"step\": 8},\n {\"eval_avg_win_prob\": 0.4, \"eval_win_rate\": 0.5, \"epoch\": 2.5, \"step\": 10},\n {\"eval_avg_win_prob\": 0.4, \"eval_win_rate\": 0.5, \"epoch\": 3.0, \"step\": 12},\n ]\n\n winrate_history = [\n {k: h[k] for k in [\"eval_avg_win_prob\", \"eval_win_rate\", \"epoch\", \"step\"]}\n for h in trainer.state.log_history\n if \"eval_avg_win_prob\" in h\n ]\n for history_row, expected_row in zip(winrate_history, expected_soft_winrates, strict=True):\n assert all(key in history_row and history_row[key] == expected_row[key] for key in expected_row)\n\n @require_peft\n def test_lora(self):\n peft_config = LoraConfig(\n r=16,\n lora_alpha=32,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n )\n self.model.add_adapter(peft_config)\n training_args = TrainingArguments(\n output_dir=self.tmp_dir,\n eval_strategy=\"steps\",\n eval_steps=2, # evaluate every 2 steps\n per_device_train_batch_size=2, # 8 samples in total so 4 batches of 2 per epoch\n per_device_eval_batch_size=2,\n report_to=\"none\",\n )\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n eval_dataset=self.dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n win_rate_callback = WinRateCallback(\n judge=self.judge, trainer=trainer, generation_config=self.generation_config\n )\n trainer.add_callback(win_rate_callback)\n trainer.train()\n winrate_history = [h for h in trainer.state.log_history if \"eval_win_rate\" in h]\n for history_row, expected_row in zip(winrate_history, self.expected_winrates, strict=True):\n assert all(key in history_row and history_row[key] == expected_row[key] for key in expected_row)\n" }, { "path": "tests/experimental/test_xpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pytest\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer\nfrom transformers.utils import is_peft_available\n\nfrom trl.experimental.xpo import XPOConfig, XPOTrainer\n\nfrom ..testing_utils import TrlTestCase, require_llm_blender, require_peft\nfrom .testing_utils import RandomPairwiseJudge\n\n\nif is_peft_available():\n from peft import LoraConfig, get_peft_model\n\n\n@pytest.mark.low_priority\nclass TestXPOTrainer(TrlTestCase):\n def setup_method(self):\n self.model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n self.model = AutoModelForCausalLM.from_pretrained(self.model_id, dtype=\"float32\")\n self.ref_model = AutoModelForCausalLM.from_pretrained(self.model_id)\n self.reward_model = AutoModelForSequenceClassification.from_pretrained(self.model_id, num_labels=1)\n self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n self.tokenizer.pad_token = self.tokenizer.eos_token\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n def test_xpo_trainer_training(self, config_name):\n training_args = XPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n\n trainer = XPOTrainer(\n model=self.model,\n ref_model=self.ref_model,\n reward_funcs=self.reward_model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @require_peft\n def test_training_with_peft(self):\n lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\", task_type=\"CAUSAL_LM\")\n training_args = XPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = XPOTrainer(\n model=self.model,\n reward_funcs=self.reward_model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n peft_config=lora_config,\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @require_peft\n def test_training_with_peft_and_ref_model(self):\n lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\", task_type=\"CAUSAL_LM\")\n training_args = XPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n learning_rate=5.0e-7,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n trainer = XPOTrainer(\n model=self.model,\n ref_model=self.ref_model,\n reward_funcs=self.reward_model,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n peft_config=lora_config,\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @require_peft\n def test_training_pre_pefted_model_implicit_ref(self):\n lora_config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.1, bias=\"none\", task_type=\"CAUSAL_LM\")\n peft_model_instance = get_peft_model(self.model, lora_config)\n\n training_args = XPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=1,\n max_steps=2,\n learning_rate=5.0e-7,\n eval_strategy=\"no\",\n report_to=\"none\",\n remove_unused_columns=False,\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")[\"train\"]\n\n trainer = XPOTrainer(\n model=peft_model_instance,\n ref_model=None,\n reward_funcs=self.reward_model, # Using reward_model to ensure _generate_completions is used as expected\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset,\n )\n\n trainer.train()\n\n assert \"train_loss\" in trainer.state.log_history[-1]\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n @require_llm_blender\n def test_xpo_trainer_judge_training(self, config_name):\n training_args = XPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=2,\n max_steps=3,\n remove_unused_columns=False,\n gradient_accumulation_steps=1,\n learning_rate=9e-1,\n eval_strategy=\"steps\",\n report_to=\"none\",\n )\n dummy_dataset = load_dataset(\"trl-internal-testing/zen\", config_name)\n judge = RandomPairwiseJudge()\n\n trainer = XPOTrainer(\n model=self.model,\n ref_model=self.ref_model,\n judge=judge,\n args=training_args,\n processing_class=self.tokenizer,\n train_dataset=dummy_dataset[\"train\"],\n eval_dataset=dummy_dataset[\"test\"],\n )\n\n trainer.train()\n\n # Check if training loss is available\n assert \"train_loss\" in trainer.state.log_history[-1]\n" }, { "path": "tests/experimental/testing_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport random\n\nfrom trl.experimental.judges import BasePairwiseJudge\n\n\nclass RandomPairwiseJudge(BasePairwiseJudge):\n \"\"\"\n Random pairwise judge, for testing purposes.\n \"\"\"\n\n def judge(self, prompts, completions, shuffle_order=True, return_scores=False):\n if not return_scores:\n return [random.randint(0, len(completion) - 1) for completion in completions]\n else:\n return [random.random() for _ in range(len(prompts))]\n" }, { "path": "tests/test_activation_offloading.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport torch\nfrom torch import nn\nfrom transformers import AutoModelForCausalLM\nfrom transformers.testing_utils import torch_device\nfrom transformers.utils import is_peft_available\n\nfrom trl.models.activation_offloading import NoOpManager, OffloadActivations\n\nfrom .testing_utils import TrlTestCase, require_peft, require_torch_accelerator\n\n\nif is_peft_available():\n from peft import LoraConfig, get_peft_model\n\n\nclass TestActivationOffloading(TrlTestCase):\n @require_torch_accelerator\n @require_peft\n def test_offloading_with_peft_models(self) -> None:\n \"\"\"Test that activation offloading works with PEFT models.\"\"\"\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id).to(torch_device)\n peft_config = LoraConfig(\n lora_alpha=16,\n lora_dropout=0.1,\n r=8,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n )\n\n model = get_peft_model(model, peft_config)\n inp = torch.randint(0, 100, (2, 10), device=torch_device)\n\n # First forward-backward pass without offloading\n torch.manual_seed(42)\n loss = model(inp, labels=inp).loss\n loss.backward()\n\n # Store gradients - only from trainable parameters\n grads_original = []\n for name, param in model.named_parameters():\n if param.requires_grad and param.grad is not None:\n grads_original.append((name, param.grad.clone()))\n\n # Reset gradients\n for p in model.parameters():\n if p.grad is not None:\n p.grad = None\n\n # Second forward-backward pass with offloading\n torch.manual_seed(42)\n with OffloadActivations():\n loss_c = model(inp, labels=inp).loss\n loss_c.backward()\n\n # Compare gradients - only trainable parameters\n for name_orig, grad_orig in grads_original:\n for name_param, param in model.named_parameters():\n if name_param == name_orig and param.requires_grad and param.grad is not None:\n (\n torch.testing.assert_close(grad_orig, param.grad, rtol=1e-4, atol=1e-5),\n (f\"Gradient mismatch for {name_orig}\"),\n )\n\n @require_torch_accelerator\n def test_noop_manager_with_offloading(self):\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id).to(torch_device)\n inp = torch.randint(0, 100, (2, 10), device=torch_device)\n\n # Run with offloading but disable for specific section\n with OffloadActivations():\n # First forward-backward with normal offloading\n torch.manual_seed(42)\n out1 = model(inp, labels=inp)\n out1.loss.backward()\n grads1 = [p.grad.clone() for p in model.parameters()]\n\n # Reset grads\n for p in model.parameters():\n p.grad = None\n\n # Second forward-backward with NoOpManager\n with NoOpManager():\n torch.manual_seed(42)\n out2 = model(inp, labels=inp)\n out2.loss.backward()\n\n grads2 = [p.grad.clone() for p in model.parameters()]\n\n # Gradients should match as NoOpManager should have prevented offloading\n for g1, g2 in zip(grads1, grads2, strict=True):\n torch.testing.assert_close(g1, g2, rtol=1e-4, atol=1e-5)\n\n @require_torch_accelerator\n def test_min_offload_size(self):\n \"\"\"Test that tensors smaller than min_offload_size aren't offloaded\"\"\"\n model = nn.Sequential(\n nn.Linear(5, 5), # Small layer that shouldn't be offloaded\n nn.Linear(5, 1000), # Large layer that should be offloaded\n ).to(torch_device)\n\n inp = torch.randn(2, 5, device=torch_device)\n\n with OffloadActivations(min_offload_size=1000):\n out = model(inp)\n out.sum().backward()\n\n # The test passes if no errors occur, as we're mainly testing\n # that the logic handles both offloaded and non-offloaded tensors\n\n @require_torch_accelerator\n def test_real_hf_model(self):\n \"\"\"Test with an actual HuggingFace model\"\"\"\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id).to(torch_device)\n\n # Create small input\n inp = torch.randint(0, 100, (2, 10), device=torch_device)\n\n # Baseline without offloading\n torch.manual_seed(42)\n out1 = model(inp, labels=inp).loss\n out1.backward()\n grads1 = [p.grad.clone() for p in model.parameters()]\n\n # Reset grads\n for p in model.parameters():\n p.grad = None\n\n # With offloading\n with OffloadActivations():\n torch.manual_seed(42)\n out2 = model(inp, labels=inp).loss\n out2.backward()\n\n grads2 = [p.grad.clone() for p in model.parameters()]\n\n # Check outputs and gradients match\n torch.testing.assert_close(out1, out2)\n for g1, g2 in zip(grads1, grads2, strict=True):\n torch.testing.assert_close(g1, g2)\n\n @require_torch_accelerator\n def test_tensor_deduplication(self):\n \"\"\"Test that deduplication works correctly for tensors sharing storage\"\"\"\n\n class ModelWithViews(nn.Module):\n def __init__(self):\n super().__init__()\n self.linear = nn.Linear(100, 100)\n\n def forward(self, x):\n out = self.linear(x)\n view1 = out.view(-1)\n view2 = out.transpose(0, 1)\n return view1.sum() + view2.sum()\n\n model = ModelWithViews().to(torch_device)\n offload_ctx = OffloadActivations(min_offload_size=1)\n offload_ctx.update_model_params(model)\n\n x = torch.randn(10, 100, device=torch_device, requires_grad=True)\n with offload_ctx:\n loss = model(x)\n\n total_tensor_ids = offload_ctx.tensor_id\n assert total_tensor_ids > 0, \"Should have created tensor IDs\"\n\n # modified=True means offloaded to CPU, modified=False means kept on GPU (deduplicated)\n deduplicated_count = sum(1 for _, modified, _, _, _ in offload_ctx.tracker.values() if not modified)\n offloaded_count = sum(1 for _, modified, _, _, _ in offload_ctx.tracker.values() if modified)\n\n assert offloaded_count > 0, \"Should have offloaded at least one tensor\"\n assert deduplicated_count > 0, \"Should have deduplicated at least one tensor (view)\"\n\n unique_storages_offloaded = len(offload_ctx.storage_to_tensor_id)\n assert unique_storages_offloaded < total_tensor_ids, (\n f\"Deduplication should result in fewer storages ({unique_storages_offloaded}) \"\n f\"than total tensors ({total_tensor_ids})\"\n )\n\n loss.backward()\n\n @require_torch_accelerator\n def test_parameter_filtering(self):\n \"\"\"Test that model parameters are filtered during offloading\"\"\"\n model = nn.Sequential(nn.Linear(10, 20), nn.Linear(20, 10)).to(torch_device)\n offload_ctx = OffloadActivations()\n offload_ctx.update_model_params(model)\n\n assert len(offload_ctx.param_storages) > 0, \"Should have tracked parameter storages\"\n\n param_ptrs = {p.data.untyped_storage().data_ptr() for p in model.parameters()}\n assert offload_ctx.param_storages == param_ptrs, \"Tracked storages should match parameter storages\"\n" }, { "path": "tests/test_callbacks.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport json\nimport os\nfrom unittest.mock import call, patch\n\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig, Trainer, TrainingArguments\n\nfrom trl import BEMACallback, LogCompletionsCallback\n\nfrom .testing_utils import TrlTestCase, require_comet, require_wandb\n\n\nclass TestLogCompletionsCallback(TrlTestCase):\n def setup_method(self):\n self.model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n self.tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n self.tokenizer.pad_token = self.tokenizer.eos_token\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n dataset[\"train\"] = dataset[\"train\"].select(range(8))\n\n def tokenize_function(examples):\n out = self.tokenizer(examples[\"prompt\"], padding=\"max_length\", max_length=16, truncation=True)\n out[\"labels\"] = out[\"input_ids\"].copy()\n return out\n\n self.dataset = dataset.map(tokenize_function, batched=True)\n\n self.generation_config = GenerationConfig(max_length=32)\n\n @require_wandb\n def test_basic_wandb(self):\n import wandb\n\n training_args = TrainingArguments(\n output_dir=self.tmp_dir,\n eval_strategy=\"steps\",\n eval_steps=2, # evaluate every 2 steps\n per_device_train_batch_size=2, # 8 samples in total so 4 batches of 2 per epoch\n per_device_eval_batch_size=2,\n report_to=\"wandb\",\n )\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n eval_dataset=self.dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n completions_callback = LogCompletionsCallback(trainer, self.generation_config, num_prompts=2)\n trainer.add_callback(completions_callback)\n trainer.train()\n\n # Get the current run\n completions_path = wandb.run.summary.completions[\"path\"]\n json_path = os.path.join(wandb.run.dir, completions_path)\n with open(json_path) as f:\n completions = json.load(f)\n\n # Check that the columns are correct\n assert \"step\" in completions[\"columns\"]\n assert \"prompt\" in completions[\"columns\"]\n assert \"completion\" in completions[\"columns\"]\n\n # Check that the prompt is in the log\n assert self.dataset[\"test\"][0][\"prompt\"] in completions[\"data\"][0]\n\n @require_comet\n def test_basic_comet(self):\n import comet_ml\n\n training_args = TrainingArguments(\n output_dir=self.tmp_dir,\n eval_strategy=\"steps\",\n eval_steps=2, # evaluate every 2 steps\n per_device_train_batch_size=2, # 8 samples in total so 4 batches of 2 per epoch\n per_device_eval_batch_size=2,\n report_to=\"comet_ml\",\n )\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n eval_dataset=self.dataset[\"test\"],\n processing_class=self.tokenizer,\n )\n completions_callback = LogCompletionsCallback(trainer, self.generation_config, num_prompts=2)\n trainer.add_callback(completions_callback)\n trainer.train()\n\n # close experiment to make sure all pending data are flushed\n experiment = comet_ml.get_running_experiment()\n assert experiment is not None\n experiment.end()\n\n # get experiment assets and check that all required tables was logged\n steps = len(self.dataset[\"train\"]) + len(self.dataset[\"test\"])\n tables_logged = int(steps / 2) + 1 # +1 to include zero step\n\n api_experiment = comet_ml.APIExperiment(previous_experiment=experiment.id)\n tables = api_experiment.get_asset_list(\"dataframe\")\n assert tables is not None\n assert len(tables) == tables_logged\n assert all(table[\"fileName\"] == \"completions.csv\" for table in tables)\n\n\nclass TestBEMACallback(TrlTestCase):\n def setup_method(self):\n self.model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n self.tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n self.tokenizer.pad_token = self.tokenizer.eos_token\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n\n def tokenize_function(examples, tokenizer):\n out = tokenizer(examples[\"text\"], padding=\"max_length\", max_length=17)\n out[\"labels\"] = out[\"input_ids\"].copy()\n return out\n\n self.dataset = dataset.map(\n tokenize_function, fn_kwargs={\"tokenizer\": self.tokenizer}, remove_columns=[\"text\"], batched=True\n )\n\n def test_model_saved(self):\n \"\"\"Test that BEMACallback saves the BEMA model.\"\"\"\n training_args = TrainingArguments(output_dir=self.tmp_dir, report_to=\"none\")\n bema_callback = BEMACallback(update_freq=2)\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n processing_class=self.tokenizer,\n callbacks=[bema_callback],\n )\n trainer.train()\n\n # Check that the BEMA model was saved and can be loaded\n bema_path = os.path.join(self.tmp_dir, \"bema\")\n assert os.path.isdir(bema_path), \"BEMA directory was not created\"\n AutoModelForCausalLM.from_pretrained(bema_path)\n\n def test_update_frequency_0(self):\n \"\"\"Test that BEMA callback respects the update frequency.\"\"\"\n training_args = TrainingArguments(output_dir=self.tmp_dir, report_to=\"none\")\n bema_callback = BEMACallback(update_freq=2)\n\n with patch.object(bema_callback, \"_update_bema_weights\") as mock_update:\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n processing_class=self.tokenizer,\n callbacks=[bema_callback],\n )\n\n trainer.train()\n\n # Total 9 steps (17 samples, batch size 8, 3 epochs).\n # BEMA starts after step 0 and updates every 2 steps โ†’ updates at 2, 4, 5, 8\n assert mock_update.call_args_list == [call(2), call(4), call(6), call(8)]\n\n def test_update_frequency_1(self):\n \"\"\"Test that BEMA callback respects the update frequency.\"\"\"\n training_args = TrainingArguments(output_dir=self.tmp_dir, report_to=\"none\")\n bema_callback = BEMACallback(update_freq=3)\n\n with patch.object(bema_callback, \"_update_bema_weights\") as mock_update:\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n processing_class=self.tokenizer,\n callbacks=[bema_callback],\n )\n\n trainer.train()\n\n # Total 9 steps (17 samples, batch size 8, 3 epochs).\n # BEMA starts after step 0 and updates every 3 steps โ†’ updates at 3, 6, 9\n assert mock_update.call_args_list == [call(3), call(6), call(9)]\n\n def test_update_frequency_2(self):\n \"\"\"Test that BEMA callback respects the update frequency.\"\"\"\n training_args = TrainingArguments(output_dir=self.tmp_dir, report_to=\"none\")\n bema_callback = BEMACallback(update_freq=2, update_after=3)\n\n with patch.object(bema_callback, \"_update_bema_weights\") as mock_update:\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n processing_class=self.tokenizer,\n callbacks=[bema_callback],\n )\n\n trainer.train()\n\n # Total 9 steps (17 samples, batch size 8, 3 epochs).\n # BEMA starts after step 3 and updates every 2 steps โ†’ updates at 5, 7, 9\n assert mock_update.call_args_list == [call(5), call(7), call(9)]\n\n def test_no_bema(self):\n \"\"\"Test that BEMACallback works without BEMA updates.\"\"\"\n training_args = TrainingArguments(output_dir=self.tmp_dir, report_to=\"none\")\n bema_callback = BEMACallback(update_freq=2, bias_power=0.0)\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n processing_class=self.tokenizer,\n callbacks=[bema_callback],\n )\n trainer.train()\n\n def test_no_ema(self):\n \"\"\"Test that BEMACallback works without EMA updates.\"\"\"\n training_args = TrainingArguments(output_dir=self.tmp_dir, report_to=\"none\")\n bema_callback = BEMACallback(update_freq=2, ema_power=0.0)\n trainer = Trainer(\n model=self.model,\n args=training_args,\n train_dataset=self.dataset[\"train\"],\n processing_class=self.tokenizer,\n callbacks=[bema_callback],\n )\n trainer.train()\n" }, { "path": "tests/test_chat_template_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport textwrap\n\nimport pytest\nimport transformers\nfrom packaging.version import Version\nfrom transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer\n\nfrom trl import clone_chat_template\nfrom trl.chat_template_utils import (\n add_response_schema,\n get_training_chat_template,\n is_chat_template_prefix_preserving,\n parse_response,\n)\n\nfrom .testing_utils import TrlTestCase, require_jmespath\n\n\nclass TestCloneChatTemplate(TrlTestCase):\n def test_clone(self):\n # This tokenizer doesn't have a chat_template by default\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-BloomForCausalLM\")\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-BloomForCausalLM\")\n # This one has a chat_template by default\n source = \"trl-internal-testing/tiny-Qwen3ForCausalLM\"\n _, modified_tokenizer, _ = clone_chat_template(model, tokenizer, source)\n\n # Check if special tokens are correctly set\n assert modified_tokenizer.eos_token == \"<|im_end|>\"\n\n def test_clone_with_resize(self):\n # This tokenizer doesn't have a chat_template by default\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-BloomForCausalLM\")\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-BloomForCausalLM\")\n # This one has a chat_template by default\n source = \"trl-internal-testing/tiny-Qwen3ForCausalLM\"\n modified_model, modified_tokenizer, _ = clone_chat_template(\n model, tokenizer, source, resize_to_multiple_of=123\n )\n\n # Check that the input embeddings have been resized to a multiple of 123\n assert (modified_model.vocab_size % 123) == 0\n # Check that the input embeddings size matches the tokenizer vocabulary size\n assert model.vocab_size == len(modified_tokenizer.vocab)\n\n def test_clone_with_resize_and_extra_tokens_already_in_vocab(self):\n # This tokenizer doesn't have a chat_template by default\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-BloomForCausalLM\")\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-BloomForCausalLM\")\n # This one has a chat_template by default\n source = \"trl-internal-testing/tiny-Qwen3ForCausalLM\"\n # This will add , , ... to the tokenizer\n modified_model, modified_tokenizer, _ = clone_chat_template(\n model, tokenizer, source, resize_to_multiple_of=123\n )\n # Try if we can resize a tokenizer that already has extra these extra tokens\n modified_model, modified_tokenizer, _ = clone_chat_template(\n modified_model, modified_tokenizer, source, resize_to_multiple_of=124\n )\n\n # Check that the input embeddings have been resized to a multiple of 123\n assert (modified_model.vocab_size % 124) == 0\n # Check that the input embeddings size matches the tokenizer vocabulary size\n assert model.vocab_size == len(modified_tokenizer.vocab)\n\n def test_apply_new_chat_template(self):\n # This tokenizer doesn't have a chat_template by default\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-BloomForCausalLM\")\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-BloomForCausalLM\")\n # This one has a chat_template by default\n source = \"trl-internal-testing/tiny-Qwen3ForCausalLM\"\n _, modified_tokenizer, _ = clone_chat_template(model, tokenizer, source)\n messages = [\n {\"role\": \"system\", \"content\": \"You are helpful\"},\n {\"role\": \"user\", \"content\": \"Hello\"},\n {\"role\": \"assistant\", \"content\": \"Hi, how can I help you?\"},\n ]\n prompt = modified_tokenizer.apply_chat_template(messages, tokenize=False)\n\n assert (\n prompt\n == \"<|im_start|>system\\nYou are helpful<|im_end|>\\n<|im_start|>user\\nHello<|im_end|>\\n<|im_start|>assistant\\n\\n\\n\\n\\nHi, how can I help you?<|im_end|>\\n\"\n )\n\n def test_clone_with_sequence_classification_model(self):\n # This tokenizer doesn't have a chat_template by default\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-GptNeoXForSequenceClassification\")\n model = AutoModelForSequenceClassification.from_pretrained(\n \"trl-internal-testing/tiny-GptNeoXForSequenceClassification\"\n )\n # This one has a chat_template by default\n source = \"trl-internal-testing/tiny-Qwen3ForCausalLM\"\n _, modified_tokenizer, _ = clone_chat_template(model, tokenizer, source)\n\n # Check if special tokens are correctly set\n assert modified_tokenizer.eos_token == \"<|im_end|>\"\n\n\n@pytest.mark.parametrize(\n \"tokenizer_name\",\n [\n pytest.param(\"trl-internal-testing/tiny-Qwen3MoeForSequenceClassification\", id=\"qwen3\"),\n pytest.param(\"trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration\", id=\"qwen35\"),\n ],\n)\n@pytest.mark.xfail(\n condition=Version(transformers.__version__) < Version(\"5.0.0\"),\n reason=\"Response parsing is not supported in transformers versions below 5.0.0\",\n strict=True,\n)\n@require_jmespath\nclass TestAddResponseSchema:\n def test_add_response_schema(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tokenizer = add_response_schema(tokenizer)\n messages = [\n {\"role\": \"user\", \"content\": \"What is 3*4?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"multiply\", \"arguments\": {\"a\": 3, \"b\": 4}}}],\n },\n ]\n prefix = tokenizer.apply_chat_template(messages[:1], tokenize=False, add_generation_prompt=True)\n text = tokenizer.apply_chat_template(messages, tokenize=False)\n response = text[len(prefix) :]\n # Here, we just test that the parsing doesn't raise an error.\n # The correctness of the parsing is tested in TestParseResponse\n tokenizer.parse_response(response)\n\n\nclass TestIsChatTemplatePrefixPreserving:\n def test_prefix_preserving_template(self):\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen3MoeForSequenceClassification\")\n tokenizer.chat_template = textwrap.dedent(r\"\"\"\n {%- for message in messages %}\n\n {%- if message.role == 'user' %}\n {{- '<|im_start|>user\\n' + message.content + '<|im_end|>\\n' }}\n {%- elif message.role == 'assistant' %}\n {{- '<|im_start|>assistant\\n' + message.content + '<|im_end|>\\n' }}\n {%- endif %}\n\n {%- endfor %}\n\n {%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- endif %}\"\"\")\n assert is_chat_template_prefix_preserving(tokenizer) is True\n\n def test_non_prefix_preserving_template(self):\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen3MoeForSequenceClassification\")\n # The following template is quite typical of models like Qwen3 and GPT-OSS, where the thinking part is\n # only present for last assistant message, which makes it non-prefix-preserving.\n # docstyle-ignore\n tokenizer.chat_template = textwrap.dedent(r\"\"\"\n {%- if messages[0].role == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n {%- endif %}\n {%- set ns = namespace(last_query_index=messages|length - 1) %}\n {%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if message.role == \"user\" and message.content is string %}\n {%- set ns.last_query_index = index %}\n {%- break %}\n {%- endif %}\n {%- endfor %}\n {%- for message in messages %}\n {%- set content = message.content if message.content is string else '' %}\n {%- if message.role == \"user\" or (message.role == \"system\" and not loop.first) %}\n {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>\\n' }}\n {%- elif message.role == \"assistant\" %}\n {%- set reasoning_content = '' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if '' in content %}\n {%- set reasoning_content = content.split('')[0].rstrip('\\n').split('')[-1].lstrip('\\n') %}\n {%- set content = content.split('')[-1].lstrip('\\n') %}\n {%- endif %}\n {%- endif %}\n {%- if loop.index0 > ns.last_query_index %}\n {%- if loop.last or (not loop.last and reasoning_content) %}\n {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endfor %}\n {%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- if enable_thinking is defined and enable_thinking is false %}\n {{- '\\n\\n\\n\\n' }}\n {%- endif %}\n {%- endif %}\"\"\")\n assert is_chat_template_prefix_preserving(tokenizer) is False\n\n\n@pytest.mark.parametrize(\n \"tokenizer_name\",\n [\n pytest.param(\"trl-internal-testing/tiny-Qwen3MoeForSequenceClassification\", id=\"qwen3\"),\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration\",\n id=\"qwen35\",\n marks=pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.0.0\"),\n reason=\"Qwen3.5 tokenizer requires transformers>=5.0.0\",\n ),\n ),\n ],\n)\nclass TestGetTrainingChatTemplate:\n def test_new_chat_template_is_prefix_preserving(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n assert is_chat_template_prefix_preserving(tokenizer) is False\n tokenizer.chat_template = get_training_chat_template(tokenizer)\n assert is_chat_template_prefix_preserving(tokenizer) is True\n\n def test_behavior_unchanged_single_user_no_generation_prompt(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n messages = [{\"role\": \"user\", \"content\": \"What color is the sky?\"}]\n before = tokenizer.apply_chat_template(messages, tokenize=False)\n new_chat_template = get_training_chat_template(tokenizer)\n after = tokenizer.apply_chat_template(messages, tokenize=False, chat_template=new_chat_template)\n assert before == after\n\n def test_behavior_unchanged_single_user_with_generation_prompt(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n messages = [{\"role\": \"user\", \"content\": \"What color is the sky?\"}]\n before = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)\n new_chat_template = get_training_chat_template(tokenizer)\n after = tokenizer.apply_chat_template(\n messages,\n tokenize=False,\n add_generation_prompt=True,\n chat_template=new_chat_template,\n )\n assert before == after\n\n def test_behavior_unchanged_single_user_and_final_assistant_plain_content(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n messages = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ]\n before = tokenizer.apply_chat_template(messages, tokenize=False)\n new_chat_template = get_training_chat_template(tokenizer)\n after = tokenizer.apply_chat_template(messages, tokenize=False, chat_template=new_chat_template)\n assert before == after\n\n def test_behavior_unchanged_final_assistant_with_reasoning_content(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n messages = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"It is blue.\",\n \"reasoning_content\": \"The sky appears blue due to Rayleigh scattering.\",\n },\n ]\n before = tokenizer.apply_chat_template(messages, tokenize=False)\n new_chat_template = get_training_chat_template(tokenizer)\n after = tokenizer.apply_chat_template(messages, tokenize=False, chat_template=new_chat_template)\n assert before == after\n\n def test_behavior_unchanged_final_assistant_with_existing_think_tags(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n messages = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\\nThe sky scatters shorter wavelengths.\\n\\n\\nIt is blue.\",\n },\n ]\n before = tokenizer.apply_chat_template(messages, tokenize=False)\n new_chat_template = get_training_chat_template(tokenizer)\n after = tokenizer.apply_chat_template(messages, tokenize=False, chat_template=new_chat_template)\n assert before == after\n\n def test_behavior_unchanged_assistant_with_tool_calls(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n messages = [\n {\"role\": \"user\", \"content\": \"Multiply 3 by 4.\"},\n {\n \"role\": \"assistant\",\n \"content\": \"I will call a tool.\",\n \"tool_calls\": [{\"name\": \"multiply\", \"arguments\": {\"a\": 3, \"b\": 4}}],\n },\n ]\n before = tokenizer.apply_chat_template(messages, tokenize=False)\n new_chat_template = get_training_chat_template(tokenizer)\n after = tokenizer.apply_chat_template(messages, tokenize=False, chat_template=new_chat_template)\n assert before == after\n\n def test_behavior_unchanged_with_tools_with_and_without_system_message(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tools = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"multiply\",\n \"description\": \"Multiply two numbers.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"a\": {\"type\": \"number\"},\n \"b\": {\"type\": \"number\"},\n },\n \"required\": [\"a\", \"b\"],\n },\n },\n }\n ]\n messages = [{\"role\": \"user\", \"content\": \"Multiply 3 by 4.\"}]\n before = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools)\n new_chat_template = get_training_chat_template(tokenizer)\n after = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools, chat_template=new_chat_template)\n assert before == after\n\n def test_behavior_unchanged_with_tools_with_system_message(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tools = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"multiply\",\n \"description\": \"Multiply two numbers.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\"a\": {\"type\": \"number\"}, \"b\": {\"type\": \"number\"}},\n \"required\": [\"a\", \"b\"],\n },\n },\n }\n ]\n messages = [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Multiply 3 by 4.\"},\n ]\n before = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools)\n new_chat_template = get_training_chat_template(tokenizer)\n after = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools, chat_template=new_chat_template)\n assert before == after\n\n def test_behavior_unchanged_generation_prompt_with_enable_thinking_false(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n messages = [{\"role\": \"user\", \"content\": \"What color is the sky?\"}]\n before = tokenizer.apply_chat_template(\n messages, tokenize=False, add_generation_prompt=True, enable_thinking=False\n )\n new_chat_template = get_training_chat_template(tokenizer)\n after = tokenizer.apply_chat_template(\n messages,\n tokenize=False,\n add_generation_prompt=True,\n enable_thinking=False,\n chat_template=new_chat_template,\n )\n assert before == after\n\n\n@pytest.mark.parametrize(\n \"tokenizer_name\",\n [\n pytest.param(\"trl-internal-testing/tiny-Qwen3MoeForSequenceClassification\", id=\"qwen3\"),\n pytest.param(\"trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration\", id=\"qwen35\"),\n ],\n)\n@pytest.mark.xfail(\n condition=Version(transformers.__version__) < Version(\"5.0.0\"),\n reason=\"Response parsing is not supported in transformers versions below 5.0.0\",\n strict=True,\n)\n@require_jmespath\nclass TestParseResponse:\n def test_parse_response(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tokenizer = add_response_schema(tokenizer)\n messages = [\n {\"role\": \"user\", \"content\": \"What is 3*4?\"},\n {\"role\": \"assistant\", \"content\": \"12\"},\n ]\n prefix = tokenizer.apply_chat_template(messages[:1], add_generation_prompt=True).input_ids\n text = tokenizer.apply_chat_template(messages).input_ids\n response = text[len(prefix) :]\n parsed = parse_response(tokenizer, response)\n assert parsed == messages[-1]\n\n def test_parse_response_with_reasoning_content(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tokenizer = add_response_schema(tokenizer)\n messages = [\n {\"role\": \"user\", \"content\": \"What is 3*4?\"},\n {\"role\": \"assistant\", \"reasoning_content\": \"Hmmm.\", \"content\": \"12\"},\n ]\n # enable_thinking=True is required here because for Qwen3.5, the thinking is disabled by default for the\n # generation prompt.\n prefix = tokenizer.apply_chat_template(\n messages[:1], add_generation_prompt=True, enable_thinking=True\n ).input_ids\n text = tokenizer.apply_chat_template(messages).input_ids\n response = text[len(prefix) :]\n parsed = parse_response(tokenizer, response)\n assert parsed == messages[-1]\n\n def test_parse_response_tool_call(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tokenizer = add_response_schema(tokenizer)\n tool_calls = [{\"type\": \"function\", \"function\": {\"name\": \"multiply\", \"arguments\": {\"a\": 3, \"b\": 4}}}]\n messages = [\n {\"role\": \"user\", \"content\": \"What is 3*4?\"},\n {\"role\": \"assistant\", \"content\": \"\", \"tool_calls\": tool_calls},\n ]\n prefix = tokenizer.apply_chat_template(messages[:1], add_generation_prompt=True).input_ids\n text = tokenizer.apply_chat_template(messages).input_ids\n response = text[len(prefix) :]\n parsed = parse_response(tokenizer, response)\n assert parsed == messages[-1]\n\n def test_parse_response_tool_call_with_content(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tokenizer = add_response_schema(tokenizer)\n tool_calls = [{\"type\": \"function\", \"function\": {\"name\": \"multiply\", \"arguments\": {\"a\": 3, \"b\": 4}}}]\n messages = [\n {\"role\": \"user\", \"content\": \"What is 3*4?\"},\n {\"role\": \"assistant\", \"content\": \"Let's call the tool.\", \"tool_calls\": tool_calls},\n ]\n prefix = tokenizer.apply_chat_template(messages[:1], add_generation_prompt=True).input_ids\n text = tokenizer.apply_chat_template(messages).input_ids\n response = text[len(prefix) :]\n parsed = parse_response(tokenizer, response)\n assert parsed == messages[-1]\n\n def test_parse_response_tool_call_without_arguments(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tokenizer = add_response_schema(tokenizer)\n tool_calls = [{\"type\": \"function\", \"function\": {\"name\": \"ping\", \"arguments\": {}}}]\n messages = [\n {\"role\": \"user\", \"content\": \"Ping the service.\"},\n {\"role\": \"assistant\", \"tool_calls\": tool_calls},\n ]\n prefix = tokenizer.apply_chat_template(messages[:1], add_generation_prompt=True).input_ids\n text = tokenizer.apply_chat_template(messages).input_ids\n response = text[len(prefix) :]\n parsed = parse_response(tokenizer, response)\n assert parsed == {\"role\": \"assistant\", \"content\": \"\", \"tool_calls\": tool_calls}\n\n def test_parse_response_multiple_tool_calls(self, tokenizer_name):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tokenizer = add_response_schema(tokenizer)\n tool_calls = [\n {\"type\": \"function\", \"function\": {\"name\": \"multiply\", \"arguments\": {\"a\": 3, \"b\": 4}}},\n {\"type\": \"function\", \"function\": {\"name\": \"addition\", \"arguments\": {\"a\": 4, \"b\": 3}}},\n ]\n messages = [\n {\"role\": \"user\", \"content\": \"What is 3*4?\"},\n {\"role\": \"assistant\", \"content\": \"\", \"tool_calls\": tool_calls},\n ]\n prefix = tokenizer.apply_chat_template(messages[:1], add_generation_prompt=True).input_ids\n text = tokenizer.apply_chat_template(messages).input_ids\n response = text[len(prefix) :]\n parsed = parse_response(tokenizer, response)\n assert parsed == messages[-1]\n\n def test_parse_response_malformed_tool_call(self, tokenizer_name):\n if tokenizer_name != \"trl-internal-testing/tiny-Qwen3MoeForSequenceClassification\":\n pytest.skip(\"For simplicity, we only test the malformed tool call case on one tokenizer.\")\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n tokenizer = add_response_schema(tokenizer)\n text = '\\n{\"name\": \"multiply\", \"arguments\": {\"a\": 3, \"b\": 4}\\n<|im_end|>'\n assistant_text = tokenizer(text)[\"input_ids\"]\n parsed = parse_response(tokenizer, assistant_text)\n expected = {\n \"role\": \"assistant\",\n \"content\": '\\n{\"name\": \"multiply\", \"arguments\": {\"a\": 3, \"b\": 4}\\n',\n }\n\n assert parsed == expected\n" }, { "path": "tests/test_cli.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport os\nfrom io import StringIO\nfrom unittest.mock import patch\n\nimport pytest\nimport yaml\n\nfrom .testing_utils import TrlTestCase\n\n\n@pytest.mark.parametrize(\"command\", [\"dpo\", \"grpo\", \"kto\", \"reward\", \"rloo\", \"sft\"])\ndef test_help_no_type_error(command):\n # Regression test for https://github.com/huggingface/trl/issues/5099:\n # TrainingArguments help strings with unescaped \"%\" caused TypeError in argparse.\n from trl.cli import main\n\n with pytest.raises(SystemExit) as exc_info:\n with patch(\"sys.argv\", [\"trl\", command, \"--help\"]), patch(\"sys.stdout\", new_callable=StringIO):\n main()\n assert exc_info.value.code == 0\n\n\nclass TestCLI(TrlTestCase):\n def test_dpo(self):\n from trl.cli import main\n\n command = f\"trl dpo --output_dir {self.tmp_dir} --model_name_or_path trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --dataset_name trl-internal-testing/zen --dataset_config standard_preference --report_to none\"\n with patch(\"sys.argv\", command.split(\" \")):\n main()\n\n def test_dpo_multiple_loss_types(self):\n from trl.cli import main\n\n command = f\"trl dpo --output_dir {self.tmp_dir} --model_name_or_path trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --dataset_name trl-internal-testing/zen --dataset_config standard_preference --report_to none --loss_type sigmoid bco_pair --loss_weights 1.0 0.5\"\n with patch(\"sys.argv\", command.split(\" \")):\n main()\n\n @patch(\"sys.stdout\", new_callable=StringIO)\n def test_env(self, mock_stdout):\n from trl.cli import main\n\n command = \"trl env\"\n with patch(\"sys.argv\", command.split(\" \")):\n main()\n assert \"TRL version: \" in mock_stdout.getvalue().strip()\n\n def test_grpo(self):\n from trl.cli import main\n\n command = f\"trl grpo --output_dir {self.tmp_dir} --model_name_or_path trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --reward_model_name_or_path trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5 --dataset_name trl-internal-testing/zen --dataset_config standard_prompt_only --num_generations 4 --max_completion_length 32 --report_to none\"\n with patch(\"sys.argv\", command.split(\" \")):\n main()\n\n def test_kto(self):\n from trl.cli import main\n\n command = f\"trl kto --output_dir {self.tmp_dir} --model_name_or_path trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --dataset_name trl-internal-testing/zen --dataset_config standard_unpaired_preference --report_to none\"\n with patch(\"sys.argv\", command.split(\" \")):\n main()\n\n def test_reward(self):\n from trl.cli import main\n\n command = f\"trl reward --output_dir {self.tmp_dir} --model_name_or_path trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --dataset_name trl-internal-testing/zen --dataset_config standard_implicit_prompt_preference --report_to none\"\n with patch(\"sys.argv\", command.split(\" \")):\n main()\n\n def test_rloo(self):\n from trl.cli import main\n\n command = f\"trl rloo --output_dir {self.tmp_dir} --model_name_or_path trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --reward_model_name_or_path trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5 --dataset_name trl-internal-testing/zen --dataset_config standard_prompt_only --num_generations 2 --max_completion_length 32 --report_to none\"\n with patch(\"sys.argv\", command.split(\" \")):\n main()\n\n def test_sft(self):\n from trl.cli import main\n\n command = f\"trl sft --output_dir {self.tmp_dir} --model_name_or_path trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --dataset_name trl-internal-testing/zen --dataset_config standard_language_modeling --report_to none\"\n with patch(\"sys.argv\", command.split(\" \")):\n main()\n\n def test_sft_config_file(self):\n from trl.cli import main\n\n output_dir = os.path.join(self.tmp_dir, \"output\")\n\n # Create a temporary config file\n config_path = os.path.join(self.tmp_dir, \"config.yaml\")\n config_content = {\n \"model_name_or_path\": \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"dataset_name\": \"trl-internal-testing/zen\",\n \"dataset_config\": \"standard_language_modeling\",\n \"report_to\": \"none\",\n \"output_dir\": output_dir,\n \"lr_scheduler_type\": \"cosine_with_restarts\",\n }\n with open(config_path, \"w\") as config_file:\n yaml.dump(config_content, config_file)\n\n # Test the CLI with config file\n command = f\"trl sft --config {config_path}\"\n with patch(\"sys.argv\", command.split(\" \")):\n main()\n\n # Verify that output directory was created\n assert os.path.exists(output_dir)\n\n def test_vllm_serve_config_file(self):\n \"\"\"\n Test `trl vllm-serve --config config.yaml` must not raise \"the following arguments are required: --model\" when\n the required field is satisfied by the config file rather than the command line.\n \"\"\"\n from trl.cli import main\n\n config_path = os.path.join(self.tmp_dir, \"config.yaml\")\n with open(config_path, \"w\") as f:\n yaml.dump({\"model\": \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"}, f)\n\n # Patch the actual function that `VllmServeCommand.run` imports as `vllm_serve_main`\n with patch(\"trl.scripts.vllm_serve.main\") as mock_serve:\n with patch(\"sys.argv\", [\"trl\", \"vllm-serve\", \"--config\", config_path]):\n main()\n\n mock_serve.assert_called_once()\n script_args = mock_serve.call_args.args[0]\n assert script_args.model == \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n" }, { "path": "tests/test_cli_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport tempfile\nfrom dataclasses import dataclass\nfrom unittest.mock import mock_open, patch\n\nimport pytest\nfrom datasets import DatasetDict, load_dataset\n\nfrom trl import DatasetMixtureConfig, TrlParser, get_dataset\nfrom trl.scripts.utils import DatasetConfig\n\nfrom .testing_utils import TrlTestCase\n\n\n@dataclass\nclass MyDataclass:\n arg1: int\n arg2: str = \"default\"\n\n\n@dataclass\nclass InvalidDataclass:\n config: str # This should raise an error in the TrlParser\n\n\nclass TestTrlParser(TrlTestCase):\n def test_init_without_config_field(self):\n \"\"\"Test initialization without 'config' field in the dataclasses.\"\"\"\n parser = TrlParser(dataclass_types=[MyDataclass])\n assert isinstance(parser, TrlParser)\n\n def test_init_with_config_field(self):\n \"\"\"Test initialization with a 'config' field in the dataclass (should raise ValueError).\"\"\"\n with pytest.raises(ValueError, match=\"has a field named 'config'\"):\n TrlParser(dataclass_types=[InvalidDataclass])\n\n @patch(\"builtins.open\", mock_open(read_data=\"env:\\n VAR1: value1\\n VAR2: value2\\narg1: 2\"))\n @patch(\"yaml.safe_load\")\n @patch(\"os.environ\", new_callable=dict) # Mock os.environ as a dictionary\n def test_parse_args_and_config_with_valid_config(self, mock_environ, mock_yaml_load):\n \"\"\"Test parse_args_and_config method with valid arguments and config.\"\"\"\n mock_yaml_load.return_value = {\"env\": {\"VAR1\": \"value1\", \"VAR2\": \"value2\"}, \"arg1\": 2}\n\n parser = TrlParser(dataclass_types=[MyDataclass])\n\n args = [\"--arg2\", \"value\", \"--config\", \"config.yaml\"] # don't set arg1 to test default value\n\n # Simulate the config being loaded and environment variables being set\n result_args = parser.parse_args_and_config(args)\n\n # Set the environment variables using the mock\n mock_environ[\"VAR1\"] = \"value1\"\n mock_environ[\"VAR2\"] = \"value2\"\n\n # Ensure that the environment variables were set correctly\n assert mock_environ.get(\"VAR1\") == \"value1\"\n assert mock_environ.get(\"VAR2\") == \"value2\"\n\n # Check the parsed arguments\n assert len(result_args) == 1\n assert isinstance(result_args[0], MyDataclass)\n assert result_args[0].arg1 == 2\n assert result_args[0].arg2 == \"value\"\n\n @patch(\"builtins.open\", mock_open(read_data=\"arg1: 2\"))\n @patch(\"yaml.safe_load\")\n def test_parse_args_and_arg_override_config(self, mock_yaml_load):\n \"\"\"Test parse_args_and_config method and check that arguments override the config.\"\"\"\n mock_yaml_load.return_value = {\"arg1\": 2} # this arg is meant to be overridden\n\n parser = TrlParser(dataclass_types=[MyDataclass])\n\n args = [\"--arg1\", \"3\", \"--config\", \"config.yaml\"] # override arg1 default with 3\n\n # Simulate the config being loaded and arguments being passed\n result_args = parser.parse_args_and_config(args)\n\n # Check the parsed arguments\n assert len(result_args) == 1\n assert isinstance(result_args[0], MyDataclass)\n assert result_args[0].arg1 == 3\n\n @patch(\"builtins.open\", mock_open(read_data=\"env: not_a_dict\"))\n @patch(\"yaml.safe_load\")\n def test_parse_args_and_config_with_invalid_env(self, mock_yaml_load):\n \"\"\"Test parse_args_and_config method when the 'env' field is not a dictionary.\"\"\"\n mock_yaml_load.return_value = {\"env\": \"not_a_dict\"}\n\n parser = TrlParser(dataclass_types=[MyDataclass])\n\n args = [\"--arg1\", \"2\", \"--arg2\", \"value\", \"--config\", \"config.yaml\"]\n\n with pytest.raises(ValueError, match=\"`env` field should be a dict in the YAML file.\"):\n parser.parse_args_and_config(args)\n\n def test_parse_args_and_config_without_config(self):\n \"\"\"Test parse_args_and_config without the `--config` argument.\"\"\"\n parser = TrlParser(dataclass_types=[MyDataclass])\n\n args = [\"--arg1\", \"2\", \"--arg2\", \"value\"]\n\n # Simulate no config, just parse args normally\n result_args = parser.parse_args_and_config(args)\n\n # Check that the arguments are parsed as is\n assert len(result_args) == 1\n assert isinstance(result_args[0], MyDataclass)\n assert result_args[0].arg1 == 2\n assert result_args[0].arg2 == \"value\"\n\n def test_set_defaults_with_config(self):\n \"\"\"Test set_defaults_with_config updates the defaults.\"\"\"\n parser = TrlParser(dataclass_types=[MyDataclass])\n\n # Update defaults\n parser.set_defaults_with_config(arg1=42)\n\n # Ensure the default value is updated\n result_args = parser.parse_args_and_config([])\n assert len(result_args) == 1\n assert isinstance(result_args[0], MyDataclass)\n assert result_args[0].arg1 == 42\n\n def test_parse_args_and_config_with_remaining_strings(self):\n parser = TrlParser(dataclass_types=[MyDataclass])\n\n args = [\"--arg1\", \"2\", \"--arg2\", \"value\", \"remaining\"]\n\n # Simulate no config, just parse args normally\n result_args = parser.parse_args_and_config(args, return_remaining_strings=True)\n\n # Check that the arguments are parsed as is\n assert len(result_args) == 2\n assert isinstance(result_args[0], MyDataclass)\n assert result_args[0].arg1 == 2\n assert result_args[0].arg2 == \"value\"\n assert result_args[1] == [\"remaining\"]\n\n @patch(\"builtins.open\", mock_open(read_data=\"remaining_string_in_config: abc\"))\n @patch(\"yaml.safe_load\")\n def test_parse_args_and_config_with_remaining_strings_in_config_and_args(self, mock_yaml_load):\n mock_yaml_load.return_value = {\"remaining_string_in_config\": \"abc\"}\n\n parser = TrlParser(dataclass_types=[MyDataclass])\n\n args = [\"--arg1\", \"2\", \"--remaining_string_in_args\", \"def\", \"--config\", \"config.yaml\"]\n\n # Simulate the config being loaded and arguments being passed\n result_args = parser.parse_args_and_config(args, return_remaining_strings=True)\n\n # Check that the arguments are parsed as is\n assert len(result_args) == 2\n assert isinstance(result_args[0], MyDataclass)\n assert result_args[0].arg1 == 2\n assert result_args[1] == [\"--remaining_string_in_config\", \"abc\", \"--remaining_string_in_args\", \"def\"]\n\n @patch(\"builtins.open\", mock_open(read_data=\"arg1: 2\\narg2: config_value\"))\n @patch(\"yaml.safe_load\")\n def test_subparsers_with_config_defaults(self, mock_yaml_load):\n \"\"\"Test that config defaults are applied to all subparsers.\"\"\"\n mock_yaml_load.return_value = {\"arg1\": 2, \"arg2\": \"config_value\"}\n\n # Create the main parser\n parser = TrlParser()\n\n # Add subparsers\n subparsers = parser.add_subparsers(dest=\"command\", parser_class=TrlParser)\n\n # Create a subparser for a specific command\n subparsers.add_parser(\"subcommand\", dataclass_types=[MyDataclass])\n\n # Parse with config file\n args = [\"subcommand\", \"--config\", \"config.yaml\"]\n result_args = parser.parse_args_and_config(args)\n\n # Check main parser arguments\n assert len(result_args) == 1\n\n # Check that config values were applied to the subparser\n assert result_args[0].arg1 == 2 # Default from config\n assert result_args[0].arg2 == \"config_value\" # Default from config\n\n @patch(\"builtins.open\", mock_open(read_data=\"arg1: 2\\narg2: config_value\"))\n @patch(\"yaml.safe_load\")\n def test_subparsers_with_config_defaults_and_arg_override(self, mock_yaml_load):\n \"\"\"Test that config defaults are applied to all subparsers.\"\"\"\n mock_yaml_load.return_value = {\"arg1\": 2, \"arg2\": \"config_value\"}\n\n # Create the main parser\n parser = TrlParser()\n\n # Add subparsers\n subparsers = parser.add_subparsers(dest=\"command\", parser_class=TrlParser)\n\n # Create a subparser for a specific command\n subparsers.add_parser(\"subcommand\", dataclass_types=[MyDataclass])\n\n # Test with command line arguments overriding config\n args = [\"subcommand\", \"--arg1\", \"3\", \"--config\", \"config.yaml\"]\n result_args = parser.parse_args_and_config(args)\n\n # Command line arguments should override config\n assert result_args[0].arg1 == 3\n assert result_args[0].arg2 == \"config_value\" # Still from config\n\n @patch(\"builtins.open\", mock_open(read_data=\"arg1: 2\\nthis_arg_does_not_exist: config_value\"))\n @patch(\"yaml.safe_load\")\n def test_subparsers_with_config_defaults_and_arg_override_wrong_name(self, mock_yaml_load):\n \"\"\"Test that config defaults are applied to all subparsers.\"\"\"\n mock_yaml_load.return_value = {\"arg1\": 2, \"this_arg_does_not_exist\": \"config_value\"}\n\n # Create the main parser\n parser = TrlParser()\n\n # Add subparsers\n subparsers = parser.add_subparsers(dest=\"command\", parser_class=TrlParser)\n\n # Create a subparser for a specific command\n subparsers.add_parser(\"subcommand\", dataclass_types=[MyDataclass])\n\n # Test with command line arguments overriding config\n args = [\"subcommand\", \"--arg1\", \"3\", \"--config\", \"config.yaml\"]\n with pytest.raises(ValueError):\n parser.parse_args_and_config(args)\n\n parser.parse_args_and_config(args, fail_with_unknown_args=False)\n\n @patch(\"builtins.open\", mock_open(read_data=\"arg1: 2\\narg2: config_value\"))\n @patch(\"yaml.safe_load\")\n def test_subparsers_multiple_with_config_defaults(self, mock_yaml_load):\n \"\"\"Test that config defaults are applied to all subparsers.\"\"\"\n mock_yaml_load.return_value = {\"arg1\": 2, \"arg2\": \"config_value\"}\n\n # Create the main parser\n parser = TrlParser()\n\n # Add subparsers\n subparsers = parser.add_subparsers(dest=\"command\", parser_class=TrlParser)\n\n # Create a subparser for a specific command\n subparsers.add_parser(\"subcommand0\", dataclass_types=[MyDataclass])\n subparsers.add_parser(\"subcommand1\", dataclass_types=[MyDataclass])\n\n for idx in range(2):\n # Parse with config file\n args = [f\"subcommand{idx}\", \"--config\", \"config.yaml\"]\n result_args = parser.parse_args_and_config(args)\n\n # Check main parser arguments\n assert len(result_args) == 1\n\n # Check that config values were applied to the subparser\n assert result_args[0].arg1 == 2 # Default from config\n assert result_args[0].arg2 == \"config_value\" # Default from config\n\n\nclass TestGetDataset:\n def test_single_dataset_with_config(self):\n mixture_config = DatasetMixtureConfig(\n datasets=[DatasetConfig(path=\"trl-internal-testing/zen\", name=\"standard_language_modeling\")]\n )\n result = get_dataset(mixture_config)\n expected = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n assert expected[\"train\"][:] == result[\"train\"][:]\n\n def test_single_dataset_preference_config(self):\n mixture_config = DatasetMixtureConfig(\n datasets=[DatasetConfig(path=\"trl-internal-testing/zen\", name=\"standard_preference\")]\n )\n result = get_dataset(mixture_config)\n expected = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\")\n assert expected[\"train\"][:] == result[\"train\"][:]\n\n def test_single_dataset_streaming(self):\n mixture_config = DatasetMixtureConfig(\n datasets=[DatasetConfig(path=\"trl-internal-testing/zen\", name=\"standard_language_modeling\")],\n streaming=True,\n )\n result = get_dataset(mixture_config)\n expected = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n assert expected[\"train\"].to_list() == list(result[\"train\"])\n\n def test_dataset_mixture_basic(self):\n dataset_config1 = DatasetConfig(\n path=\"trl-internal-testing/zen\", name=\"standard_prompt_completion\", split=\"train\", columns=[\"prompt\"]\n )\n dataset_config2 = DatasetConfig(\n path=\"trl-internal-testing/zen\", name=\"standard_preference\", split=\"train\", columns=[\"prompt\"]\n )\n mixture_config = DatasetMixtureConfig(datasets=[dataset_config1, dataset_config2])\n result = get_dataset(mixture_config)\n assert isinstance(result, DatasetDict)\n assert \"train\" in result\n train_dataset = result[\"train\"]\n assert train_dataset.column_names == [\"prompt\"]\n prompts = train_dataset[\"prompt\"]\n expected_first_half = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n assert prompts[: len(prompts) // 2] == expected_first_half[\"prompt\"]\n expected_second_half = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_completion\", split=\"train\")\n assert prompts[len(prompts) // 2 :] == expected_second_half[\"prompt\"]\n\n def test_dataset_mixture_with_weights(self):\n dataset_config1 = DatasetConfig(\n path=\"trl-internal-testing/zen\", name=\"standard_prompt_completion\", split=\"train[:50%]\", columns=[\"prompt\"]\n )\n dataset_config2 = DatasetConfig(\n path=\"trl-internal-testing/zen\", name=\"standard_preference\", split=\"train[:50%]\", columns=[\"prompt\"]\n )\n mixture_config = DatasetMixtureConfig(datasets=[dataset_config1, dataset_config2])\n result = get_dataset(mixture_config)\n assert isinstance(result, DatasetDict)\n assert \"train\" in result\n train_dataset = result[\"train\"]\n assert train_dataset.column_names == [\"prompt\"]\n prompts = train_dataset[\"prompt\"]\n expected_first_half = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train[:50%]\")\n assert prompts[: len(prompts) // 2] == expected_first_half[\"prompt\"]\n expected_second_half = load_dataset(\n \"trl-internal-testing/zen\", \"standard_prompt_completion\", split=\"train[:50%]\"\n )\n assert prompts[len(prompts) // 2 :] == expected_second_half[\"prompt\"]\n\n def test_dataset_mixture_with_test_split(self):\n mixture_config = DatasetMixtureConfig(\n datasets=[DatasetConfig(path=\"trl-internal-testing/zen\", name=\"standard_language_modeling\")],\n test_split_size=2,\n )\n result = get_dataset(mixture_config)\n assert isinstance(result, DatasetDict)\n assert \"train\" in result\n assert \"test\" in result\n assert len(result[\"train\"]) == 15\n assert len(result[\"test\"]) == 2\n\n def test_empty_dataset_mixture_raises_error(self):\n mixture_config = DatasetMixtureConfig(datasets=[])\n\n with pytest.raises(ValueError, match=\"No datasets were loaded\"):\n get_dataset(mixture_config)\n\n def test_mixture_multiple_different_configs(self):\n dataset_config1 = DatasetConfig(\n path=\"trl-internal-testing/zen\", name=\"conversational_preference\", split=\"train\", columns=[\"prompt\"]\n )\n dataset_config2 = DatasetConfig(\n path=\"trl-internal-testing/zen\", name=\"conversational_prompt_only\", split=\"test\"\n )\n mixture_config = DatasetMixtureConfig(datasets=[dataset_config1, dataset_config2])\n result = get_dataset(mixture_config)\n assert isinstance(result, DatasetDict)\n assert \"train\" in result\n assert len(result[\"train\"]) > 0\n\n def test_trlparser_parses_yaml_config_correctly(self):\n # Prepare YAML content exactly like your example\n # docstyle-ignore\n yaml_content = \"\"\"\n datasets:\n - path: trl-internal-testing/zen\n name: standard_prompt_only\n - path: trl-internal-testing/zen\n name: standard_preference\n columns:\n - prompt\n \"\"\"\n\n # Write YAML to a temporary file\n with tempfile.NamedTemporaryFile(\"w+\", suffix=\".yaml\") as tmpfile:\n tmpfile.write(yaml_content)\n tmpfile.flush()\n parser = TrlParser((DatasetMixtureConfig,))\n args = parser.parse_args_and_config(args=[\"--config\", tmpfile.name])[0]\n\n # Assert that we got DatasetMixtureConfig instance\n assert isinstance(args, DatasetMixtureConfig)\n\n # Assert datasets list length\n assert len(args.datasets) == 2\n\n # Check first dataset\n dataset_config1 = args.datasets[0]\n assert isinstance(dataset_config1, DatasetConfig)\n assert dataset_config1.path == \"trl-internal-testing/zen\"\n assert dataset_config1.name == \"standard_prompt_only\"\n assert dataset_config1.columns is None # No columns specified\n\n # Check second dataset\n dataset_config2 = args.datasets[1]\n assert isinstance(dataset_config2, DatasetConfig)\n assert dataset_config2.path == \"trl-internal-testing/zen\"\n assert dataset_config2.name == \"standard_preference\"\n assert dataset_config2.columns == [\"prompt\"] # Columns specified\n\n def test_trlparser_parses_yaml_and_loads_dataset(self):\n # Prepare YAML content exactly like your example\n # docstyle-ignore\n yaml_content = \"\"\"\n datasets:\n - path: trl-internal-testing/zen\n name: standard_language_modeling\n \"\"\"\n\n # Write YAML to a temporary file\n with tempfile.NamedTemporaryFile(\"w+\", suffix=\".yaml\") as tmpfile:\n tmpfile.write(yaml_content)\n tmpfile.flush()\n parser = TrlParser((DatasetMixtureConfig,))\n args = parser.parse_args_and_config(args=[\"--config\", tmpfile.name])[0]\n\n # Load the dataset using get_dataset\n result = get_dataset(args)\n expected = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n assert expected[\"train\"][:] == result[\"train\"][:]\n" }, { "path": "tests/test_data_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport copy\nimport textwrap\nfrom time import strftime\n\nimport pytest\nimport transformers\nfrom datasets import Dataset, DatasetDict\nfrom packaging.version import Version\nfrom transformers import AutoProcessor, AutoTokenizer, is_vision_available\n\nfrom trl.data_utils import (\n apply_chat_template,\n extract_prompt,\n is_conversational,\n is_conversational_from_value,\n maybe_apply_chat_template,\n maybe_convert_to_chatml,\n maybe_extract_prompt,\n maybe_unpair_preference_dataset,\n pack_dataset,\n prepare_multimodal_messages,\n prepare_multimodal_messages_vllm,\n truncate_dataset,\n unpair_preference_dataset,\n)\n\nfrom .testing_utils import TrlTestCase, require_vision\n\n\nif is_vision_available():\n from PIL import Image\n\n\n@require_vision\nclass TestPrepareMultimodalMessages:\n def test_basic_user_assistant_conversation(self):\n \"\"\"Test basic conversation with user and assistant messages.\"\"\"\n messages = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ]\n image = Image.new(\"RGB\", (10, 10), color=\"blue\")\n messages = prepare_multimodal_messages(messages, images=[image])\n\n expected = [\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"image\", \"image\": image}, {\"type\": \"text\", \"text\": \"What color is the sky?\"}],\n },\n {\n \"role\": \"assistant\",\n \"content\": [{\"type\": \"text\", \"text\": \"It is blue.\"}],\n },\n ]\n\n assert messages == expected\n\n def test_first_user_message_gets_image(self):\n \"\"\"Test that only the first user message gets an image.\"\"\"\n messages = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n {\"role\": \"user\", \"content\": \"How about the grass?\"},\n ]\n\n image = Image.new(\"RGB\", (10, 10), color=\"blue\")\n messages = prepare_multimodal_messages(messages, images=[image])\n\n expected = [\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"image\", \"image\": image}, {\"type\": \"text\", \"text\": \"What color is the sky?\"}],\n },\n {\n \"role\": \"assistant\",\n \"content\": [{\"type\": \"text\", \"text\": \"It is blue.\"}],\n },\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"text\", \"text\": \"How about the grass?\"}],\n },\n ]\n\n assert messages == expected\n\n def test_multiple_images(self):\n \"\"\"Test that multiple images are added to the first user message.\"\"\"\n messages = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ]\n images = [Image.new(\"RGB\", (10, 10), color=color) for color in [\"red\", \"green\", \"blue\"]]\n messages = prepare_multimodal_messages(messages, images=images)\n\n expected = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": images[0]},\n {\"type\": \"image\", \"image\": images[1]},\n {\"type\": \"image\", \"image\": images[2]},\n {\"type\": \"text\", \"text\": \"What color is the sky?\"},\n ],\n },\n {\n \"role\": \"assistant\",\n \"content\": [{\"type\": \"text\", \"text\": \"It is blue.\"}],\n },\n ]\n\n assert messages == expected\n\n def test_system_message_transformation(self):\n \"\"\"Test that system messages are properly transformed.\"\"\"\n messages = [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ]\n\n image = Image.new(\"RGB\", (10, 10), color=\"blue\")\n messages = prepare_multimodal_messages(messages, images=[image])\n\n expected = [\n {\n \"role\": \"system\",\n \"content\": [{\"type\": \"text\", \"text\": \"You are a helpful assistant\"}],\n },\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"image\", \"image\": image}, {\"type\": \"text\", \"text\": \"What color is the sky?\"}],\n },\n ]\n\n assert messages == expected\n\n def test_already_prepared_messages_unchanged(self):\n \"\"\"Test that messages with list content are not modified.\"\"\"\n messages = [\n {\"role\": \"system\", \"content\": [{\"type\": \"text\", \"text\": \"You are a helpful assistant\"}]},\n {\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"What color is the sky?\"}]},\n {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It is blue.\"}]},\n ]\n\n image = Image.new(\"RGB\", (10, 10), color=\"blue\")\n messages = prepare_multimodal_messages(messages, images=[image])\n\n expected = [\n {\n \"role\": \"system\",\n \"content\": [{\"type\": \"text\", \"text\": \"You are a helpful assistant\"}],\n },\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"image\", \"image\": image}, {\"type\": \"text\", \"text\": \"What color is the sky?\"}],\n },\n {\n \"role\": \"assistant\",\n \"content\": [{\"type\": \"text\", \"text\": \"It is blue.\"}],\n },\n ]\n\n assert messages == expected\n\n def test_mixed_prepared_and_unprepared_messages(self):\n \"\"\"Test handling of mixed prepared and unprepared messages.\"\"\"\n messages = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It is blue.\"}]},\n {\"role\": \"user\", \"content\": \"What about the grass?\"},\n ]\n\n image = Image.new(\"RGB\", (10, 10), color=\"blue\")\n messages = prepare_multimodal_messages(messages, images=[image])\n\n expected = [\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"image\", \"image\": image}, {\"type\": \"text\", \"text\": \"What color is the sky?\"}],\n },\n {\n \"role\": \"assistant\",\n \"content\": [{\"type\": \"text\", \"text\": \"It is blue.\"}],\n },\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"text\", \"text\": \"What about the grass?\"}],\n },\n ]\n\n assert messages == expected\n\n def test_message_with_tool_calling_turns(self):\n \"\"\"Test that both the assistant tool call and the tool role turns messages are properly transformed.\"\"\"\n messages = [\n {\"role\": \"user\", \"content\": \"What's the weather like in New York?\"},\n {\n \"role\": \"assistant\",\n \"tool_calls\": [\n {\n \"type\": \"tool\",\n \"function\": {\"name\": \"get_current_weather\", \"arguments\": {\"location\": \"New York\"}},\n }\n ],\n },\n {\"role\": \"tool\", \"name\": \"get_current_weather\", \"content\": \"22.0\"},\n {\"role\": \"assistant\", \"content\": \"The current weather in New York is 22.0 degrees Celsius.\"},\n ]\n\n messages = prepare_multimodal_messages(messages, images=[])\n\n expected = [\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"text\", \"text\": \"What's the weather like in New York?\"}],\n },\n {\n \"role\": \"assistant\",\n \"tool_calls\": [\n {\n \"type\": \"tool\",\n \"function\": {\"name\": \"get_current_weather\", \"arguments\": {\"location\": \"New York\"}},\n }\n ],\n },\n {\"role\": \"tool\", \"name\": \"get_current_weather\", \"content\": \"22.0\"},\n {\n \"role\": \"assistant\",\n \"content\": [{\"type\": \"text\", \"text\": \"The current weather in New York is 22.0 degrees Celsius.\"}],\n },\n ]\n\n assert messages == expected\n\n\n@require_vision\nclass TestPrepareMultimodalMessagesVLLM:\n def test_single_image_conversion(self):\n messages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": Image.new(\"RGB\", (10, 10), color=\"blue\")},\n {\"type\": \"text\", \"text\": \"What color is the sky?\"},\n ],\n }\n ]\n\n result = prepare_multimodal_messages_vllm(messages)\n\n # Original should remain unchanged (deepcopy test)\n assert messages[0][\"content\"][0][\"type\"] == \"image\"\n\n # Converted version should have correct structure\n assert result[0][\"content\"][0][\"type\"] == \"image_pil\"\n assert \"image_pil\" in result[0][\"content\"][0]\n assert \"image\" not in result[0][\"content\"][0]\n assert isinstance(result[0][\"content\"][0][\"image_pil\"], Image.Image)\n assert result[0][\"content\"][1][\"type\"] == \"text\"\n\n def test_mixed_content_conversion(self):\n messages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"text\", \"text\": \"What color is the sky?\"},\n {\"type\": \"image\", \"image\": Image.new(\"RGB\", (10, 10), color=\"blue\")},\n ],\n }\n ]\n\n result = prepare_multimodal_messages_vllm(messages)\n\n # The image part should be converted, text should be unchanged\n assert result[0][\"content\"][0][\"type\"] == \"text\"\n assert result[0][\"content\"][1][\"type\"] == \"image_pil\"\n\n def test_no_images(self):\n messages = [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"What color is the sky?\"}]}]\n\n result = prepare_multimodal_messages_vllm(messages)\n\n # Should be identical since there are no images\n assert result == messages\n # And a deepcopy โ€” not the same object\n assert result is not messages\n assert result[0] is not messages[0]\n\n def test_multiple_messages(self):\n messages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"text\", \"text\": \"What color is the sky?\"},\n {\"type\": \"image\", \"image\": Image.new(\"RGB\", (10, 10), color=\"blue\")},\n ],\n },\n {\n \"role\": \"assistant\",\n \"content\": [{\"type\": \"text\", \"text\": \"It is blue.\"}],\n },\n ]\n\n result = prepare_multimodal_messages_vllm(messages)\n\n assert result[0][\"content\"][1][\"type\"] == \"image_pil\"\n assert result[1][\"content\"][0][\"type\"] == \"text\"\n assert result[1][\"content\"][0][\"text\"] == \"It is blue.\"\n\n def test_deepcopy_integrity(self):\n messages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"text\", \"text\": \"What color is the sky?\"},\n {\"type\": \"image\", \"image\": Image.new(\"RGB\", (10, 10), color=\"blue\")},\n ],\n },\n ]\n original = copy.deepcopy(messages)\n\n _ = prepare_multimodal_messages_vllm(messages)\n\n # Original should not be mutated\n assert messages == original\n\n\nclass TestIsConversational(TrlTestCase):\n # fmt: off\n conversational_examples = [\n { # Language modeling\n \"messages\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ],\n },\n { # Prompt-only\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n },\n { # Prompt-completion\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n },\n { # Preference\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": \"It is green.\"}],\n },\n { # Preference with implicit prompt\n \"chosen\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ],\n \"rejected\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is green.\"},\n ],\n },\n { # Preference with tool calls\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"chosen\": [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_color\", \"arguments\": {\"what\": \"sky\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_color\", \"content\": \"blue\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ],\n \"rejected\": [\n {\"role\": \"assistant\", \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"get_color\", \"arguments\": {\"what\": \"tree\"}}}]},\n {\"role\": \"tool\", \"name\": \"get_color\", \"content\": \"green\"},\n {\"role\": \"assistant\", \"content\": \"It is green.\"},\n ],\n \"tools\": [\n {\n \"type\": \"function\",\n \"function\": {\n \"description\": \"Gets the color.\",\n \"name\": \"get_color\",\n \"parameters\": {\"properties\": {\"what\": {\"description\": \"What to get the color of.\", \"type\": \"string\"}}, \"required\": [\"what\"], \"type\": \"object\"},\n \"return\": {\"description\": \"The color.\", \"type\": \"string\"},\n },\n },\n ],\n },\n { # Unpaired preference\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"label\": True,\n },\n { # Language modeling with harmony\n \"messages\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n },\n { # Prompt-only with harmony\n \"prompt\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n },\n { # Prompt-completion with harmony\n \"prompt\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n \"completion\": [\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n },\n { # Preference with harmony\n \"prompt\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n \"chosen\": [\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n \"rejected\": [\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the tree...\", \"content\": \"It is green.\"},\n ],\n },\n { # Preference with implicit prompt and harmony\n \"chosen\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n \"rejected\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the tree...\", \"content\": \"It is green.\"},\n ],\n },\n { # Unpaired preference with harmony\n \"prompt\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n \"completion\": [\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n \"label\": True,\n },\n ]\n # fmt: on\n\n non_conversational_examples = [\n {\"prompt\": \"The sky is\", \"completion\": \" blue.\"},\n {\"text\": \"The sky is blue.\"},\n {\"prompt\": \"The sky is\"},\n {\"prompt\": \"The sky is\", \"chosen\": \" blue.\", \"rejected\": \" green.\"},\n {\"prompt\": \"The sky is\", \"completion\": \" blue.\", \"label\": True},\n ]\n\n @pytest.mark.parametrize(\"example\", conversational_examples)\n def test_conversational(self, example):\n assert is_conversational(example)\n\n @pytest.mark.parametrize(\"example\", non_conversational_examples)\n def test_non_conversational(self, example):\n assert not is_conversational(example)\n\n\nclass TestIsConversationalFromValue(TrlTestCase):\n def test_positive_1(self):\n example = {\n \"conversations\": [\n {\"from\": \"user\", \"value\": \"What color is the sky?\"},\n {\"from\": \"assistant\", \"value\": \"It is blue.\"},\n ],\n }\n assert is_conversational_from_value(example)\n\n def test_negative_1(self):\n example = {\n \"messages\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ],\n }\n assert not is_conversational_from_value(example)\n\n def test_negative_2(self):\n example = {\"text\": \"The sky is blue.\"}\n assert not is_conversational_from_value(example)\n\n\nclass TestApplyChatTemplate(TrlTestCase):\n tokenizers = [\n \"trl-internal-testing/tiny-CohereForCausalLM\",\n \"trl-internal-testing/tiny-Cohere2ForCausalLM\",\n \"trl-internal-testing/tiny-DeepseekV3ForCausalLM\",\n \"trl-internal-testing/tiny-DeepseekV3ForCausalLM-0528\",\n \"trl-internal-testing/tiny-FalconMambaForCausalLM\",\n \"trl-internal-testing/tiny-Gemma2ForCausalLM\",\n \"trl-internal-testing/tiny-GemmaForCausalLM\",\n \"trl-internal-testing/tiny-GptOssForCausalLM\",\n pytest.param(\n \"trl-internal-testing/tiny-Glm4MoeForCausalLM\",\n marks=pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.0.0\"),\n reason=\"GLM4 tokenizer requires transformers>=5.0.0\",\n ),\n ),\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.1\",\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-LlamaForCausalLM-3\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.1\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n \"trl-internal-testing/tiny-Phi3ForCausalLM\",\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"trl-internal-testing/tiny-Qwen3ForCausalLM\",\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration\",\n marks=pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.0.0\"),\n reason=\"Qwen3.5 tokenizer requires transformers>=5.0.0\",\n ),\n ),\n ]\n\n conversational_examples = [\n { # Language modeling\n \"messages\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ],\n },\n { # Prompt-only\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n },\n { # Prompt-completion\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n },\n { # Preference\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": \"It is green.\"}],\n },\n { # Preference with implicit prompt\n \"chosen\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ],\n \"rejected\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is green.\"},\n ],\n },\n { # Unpaired preference\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n \"label\": True,\n },\n ]\n\n non_conversational_examples = [\n {\"text\": \"The sky is blue.\"}, # Language modeling\n {\"prompt\": \"The sky is\"}, # Prompt-only\n {\"prompt\": \"The sky is\", \"completion\": \" blue.\"}, # Prompt-completion\n {\"prompt\": \"The sky is\", \"chosen\": \" blue.\", \"rejected\": \" green.\"}, # Preference\n {\"chosen\": \"The sky is blue.\", \"rejected\": \"The sky is green.\"}, # Preference with implicit prompt\n {\"prompt\": \"The sky is\", \"completion\": \" blue.\", \"label\": True}, # Unpaired preference\n ]\n\n @pytest.mark.parametrize(\"example\", conversational_examples)\n @pytest.mark.parametrize(\"tokenizer_id\", tokenizers)\n def test_apply_chat_template(self, tokenizer_id, example):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_id)\n result = apply_chat_template(example, tokenizer)\n\n # Checking if the result is a dictionary\n assert isinstance(result, dict)\n\n # The chat template should be applied to the following keys\n for key in [\"prompt\", \"chosen\", \"rejected\", \"completion\"]:\n if key in example:\n assert key in result\n assert isinstance(result[key], str)\n\n # Exception for messages, the key is \"text\" once the chat template is applied\n if \"messages\" in example:\n assert \"text\" in result\n assert isinstance(result[\"text\"], str)\n\n # The label should be kept\n if \"label\" in example:\n assert \"label\" in result\n assert isinstance(result[\"label\"], bool)\n assert result[\"label\"] == example[\"label\"]\n\n # both conversational and non-conversational examples\n @pytest.mark.parametrize(\"example\", conversational_examples + non_conversational_examples)\n @pytest.mark.parametrize(\"tokenizer_id\", tokenizers)\n def test_maybe_apply_chat_template(self, tokenizer_id, example):\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_id)\n result = maybe_apply_chat_template(example, tokenizer)\n\n # Checking if the result is a dictionary\n assert isinstance(result, dict)\n\n # The chat template should be applied to the following keys\n for key in [\"prompt\", \"chosen\", \"rejected\", \"completion\"]:\n if key in example:\n assert key in result\n assert isinstance(result[key], str)\n\n # Exception for messages, the key is \"text\" once the chat template is applied\n if \"messages\" in example:\n assert \"text\" in result\n assert isinstance(result[\"text\"], str)\n\n # The label should be kept\n if \"label\" in example:\n assert \"label\" in result\n assert isinstance(result[\"label\"], bool)\n assert result[\"label\"] == example[\"label\"]\n\n def test_apply_chat_template_with_chat_template_kwargs(self):\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen3ForCausalLM\")\n\n example = {\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n # with this tokenizer, when you pass enable_thinking=False, it will add \"\\n\\n\\n\\n\"\n \"chat_template_kwargs\": {\"enable_thinking\": False},\n }\n result = apply_chat_template(example, tokenizer)\n\n # docstyle-ignore\n expected = textwrap.dedent(\"\"\"\\\n <|im_start|>user\n What color is the sky?<|im_end|>\n <|im_start|>assistant\n \n\n \n\n \"\"\")\n\n assert result[\"prompt\"] == expected\n\n def test_apply_chat_template_with_tools(self):\n tokenizer = AutoProcessor.from_pretrained(\"trl-internal-testing/tiny-LlamaForCausalLM-3.2\")\n\n # Define dummy test tools\n def get_current_temperature(location: str):\n \"\"\"\n Gets the temperature at a given location.\n\n Args:\n location: The location to get the temperature for\n \"\"\"\n return 22.0\n\n # Define test case\n test_case = {\n \"prompt\": [\n {\"content\": \"What's the temperature in London?\", \"role\": \"user\"},\n ]\n }\n # Test with tools\n result_with_tools = apply_chat_template(test_case, tokenizer, tools=[get_current_temperature])\n\n # Verify tools are included in the output\n assert \"get_current_temperature\" in result_with_tools[\"prompt\"]\n\n # Test without tools\n result_without_tools = apply_chat_template(test_case, tokenizer, tools=None)\n\n # Verify tools are not included in the output\n assert \"get_current_temperature\" not in result_without_tools[\"prompt\"]\n\n\nclass TestApplyChatTemplateHarmony(TrlTestCase):\n def test_language_modeling(self):\n messages = {\n \"messages\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n }\n output = apply_chat_template(\n messages,\n tokenizer=AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-GptOssForCausalLM\"),\n reasoning_effort=\"low\",\n model_identity=\"You are HuggingGPT.\",\n )\n\n # docstyle-ignore\n expected = textwrap.dedent(f\"\"\"\\\n <|start|>system<|message|>You are HuggingGPT.\n Knowledge cutoff: 2024-06\n Current date: {strftime(\"%Y-%m-%d\")}\n\n Reasoning: low\n\n # Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions\n\n Respond in a friendly manner.\n\n <|end|><|start|>user<|message|>What color is the sky?<|end|><|start|>assistant<|channel|>analysis<|message|>The user asks the color of the sky...<|end|><|start|>assistant<|channel|>final<|message|>It is blue.<|return|>\"\"\")\n\n assert output[\"text\"] == expected\n\n def test_prompt_only(self):\n messages = {\n \"prompt\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n }\n output = apply_chat_template(\n messages,\n tokenizer=AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-GptOssForCausalLM\"),\n reasoning_effort=\"low\",\n model_identity=\"You are HuggingGPT.\",\n )\n\n # docstyle-ignore\n expected = textwrap.dedent(f\"\"\"\\\n <|start|>system<|message|>You are HuggingGPT.\n Knowledge cutoff: 2024-06\n Current date: {strftime(\"%Y-%m-%d\")}\n\n Reasoning: low\n\n # Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions\n\n Respond in a friendly manner.\n\n <|end|><|start|>user<|message|>What color is the sky?<|end|><|start|>assistant\"\"\")\n\n assert output[\"prompt\"] == expected\n\n def test_prompt_completion(self):\n messages = {\n \"prompt\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n \"completion\": [\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n }\n output = apply_chat_template(\n messages,\n tokenizer=AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-GptOssForCausalLM\"),\n reasoning_effort=\"low\",\n model_identity=\"You are HuggingGPT.\",\n )\n\n # docstyle-ignore\n expected_prompt = textwrap.dedent(f\"\"\"\\\n <|start|>system<|message|>You are HuggingGPT.\n Knowledge cutoff: 2024-06\n Current date: {strftime(\"%Y-%m-%d\")}\n\n Reasoning: low\n\n # Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions\n\n Respond in a friendly manner.\n\n <|end|><|start|>user<|message|>What color is the sky?<|end|><|start|>assistant\"\"\")\n expected_completion = \"<|channel|>analysis<|message|>The user asks the color of the sky...<|end|><|start|>assistant<|channel|>final<|message|>It is blue.<|return|>\"\n\n assert output[\"prompt\"] == expected_prompt\n assert output[\"completion\"] == expected_completion\n\n def test_preference(self):\n messages = {\n \"prompt\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n \"chosen\": [\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n \"rejected\": [\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the tree...\", \"content\": \"It is green.\"},\n ],\n }\n output = apply_chat_template(\n messages,\n tokenizer=AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-GptOssForCausalLM\"),\n reasoning_effort=\"low\",\n model_identity=\"You are HuggingGPT.\",\n )\n\n # docstyle-ignore\n expected_prompt = textwrap.dedent(f\"\"\"\\\n <|start|>system<|message|>You are HuggingGPT.\n Knowledge cutoff: 2024-06\n Current date: {strftime(\"%Y-%m-%d\")}\n\n Reasoning: low\n\n # Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions\n\n Respond in a friendly manner.\n\n <|end|><|start|>user<|message|>What color is the sky?<|end|><|start|>assistant\"\"\")\n expected_chosen = \"<|channel|>analysis<|message|>The user asks the color of the sky...<|end|><|start|>assistant<|channel|>final<|message|>It is blue.<|return|>\"\n expected_rejected = \"<|channel|>analysis<|message|>The user asks the color of the tree...<|end|><|start|>assistant<|channel|>final<|message|>It is green.<|return|>\"\n\n assert output[\"prompt\"] == expected_prompt\n assert output[\"chosen\"] == expected_chosen\n assert output[\"rejected\"] == expected_rejected\n\n def test_preference_with_implicit_prompt(self):\n messages = {\n \"chosen\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n \"rejected\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the tree...\", \"content\": \"It is green.\"},\n ],\n }\n output = apply_chat_template(\n messages,\n tokenizer=AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-GptOssForCausalLM\"),\n reasoning_effort=\"low\",\n model_identity=\"You are HuggingGPT.\",\n )\n\n # docstyle-ignore\n expected_chosen = textwrap.dedent(f\"\"\"\\\n <|start|>system<|message|>You are HuggingGPT.\n Knowledge cutoff: 2024-06\n Current date: {strftime(\"%Y-%m-%d\")}\n\n Reasoning: low\n\n # Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions\n\n Respond in a friendly manner.\n\n <|end|><|start|>user<|message|>What color is the sky?<|end|><|start|>assistant<|channel|>analysis<|message|>The user asks the color of the sky...<|end|><|start|>assistant<|channel|>final<|message|>It is blue.<|return|>\"\"\")\n\n # docstyle-ignore\n expected_rejected = textwrap.dedent(f\"\"\"\\\n <|start|>system<|message|>You are HuggingGPT.\n Knowledge cutoff: 2024-06\n Current date: {strftime(\"%Y-%m-%d\")}\n\n Reasoning: low\n\n # Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions\n\n Respond in a friendly manner.\n\n <|end|><|start|>user<|message|>What color is the sky?<|end|><|start|>assistant<|channel|>analysis<|message|>The user asks the color of the tree...<|end|><|start|>assistant<|channel|>final<|message|>It is green.<|return|>\"\"\")\n\n assert output[\"chosen\"] == expected_chosen\n assert output[\"rejected\"] == expected_rejected\n\n def test_unpaired_preference(self):\n messages = {\n \"prompt\": [\n {\"role\": \"system\", \"content\": \"Respond in a friendly manner.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n \"completion\": [\n {\"role\": \"assistant\", \"thinking\": \"The user asks the color of the sky...\", \"content\": \"It is blue.\"},\n ],\n \"label\": True,\n }\n output = apply_chat_template(\n messages,\n tokenizer=AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-GptOssForCausalLM\"),\n reasoning_effort=\"low\",\n model_identity=\"You are HuggingGPT.\",\n )\n\n # docstyle-ignore\n expected_prompt = textwrap.dedent(f\"\"\"\\\n <|start|>system<|message|>You are HuggingGPT.\n Knowledge cutoff: 2024-06\n Current date: {strftime(\"%Y-%m-%d\")}\n\n Reasoning: low\n\n # Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions\n\n Respond in a friendly manner.\n\n <|end|><|start|>user<|message|>What color is the sky?<|end|><|start|>assistant\"\"\")\n expected_completion = \"<|channel|>analysis<|message|>The user asks the color of the sky...<|end|><|start|>assistant<|channel|>final<|message|>It is blue.<|return|>\"\n\n assert output[\"prompt\"] == expected_prompt\n assert output[\"completion\"] == expected_completion\n assert output[\"label\"]\n\n\nclass TestUnpairPreferenceDataset(TrlTestCase):\n paired_dataset = Dataset.from_dict(\n {\n \"prompt\": [\"The sky is\", \"The sun is\"],\n \"chosen\": [\" blue.\", \" in the sky.\"],\n \"rejected\": [\" green.\", \" in the sea.\"],\n }\n )\n\n unpaired_dataset = Dataset.from_dict(\n {\n \"prompt\": [\"The sky is\", \"The sun is\", \"The sky is\", \"The sun is\"],\n \"completion\": [\" blue.\", \" in the sky.\", \" green.\", \" in the sea.\"],\n \"label\": [True, True, False, False],\n }\n )\n\n def test_unpair_preference_dataset(self):\n # Test that a paired dataset is correctly converted to unpaired\n unpaired_dataset = unpair_preference_dataset(self.paired_dataset)\n assert unpaired_dataset.to_dict() == self.unpaired_dataset.to_dict(), (\n \"The paired dataset should be converted to unpaired.\"\n )\n\n def test_unpair_preference_dataset_dict(self):\n # Test that a paired dataset dict is correctly converted to unpaired\n paired_dataset_dict = DatasetDict({\"abc\": self.paired_dataset})\n unpaired_dataset_dict = unpair_preference_dataset(paired_dataset_dict)\n assert unpaired_dataset_dict[\"abc\"].to_dict() == self.unpaired_dataset.to_dict(), (\n \"The paired dataset should be converted to unpaired.\"\n )\n\n def test_maybe_unpair_preference_dataset(self):\n # Test that a paired dataset is correctly converted to unpaired with maybe_unpair_preference_dataset\n unpaired_dataset = maybe_unpair_preference_dataset(self.paired_dataset)\n assert unpaired_dataset.to_dict() == self.unpaired_dataset.to_dict(), (\n \"The paired dataset should be converted to unpaired.\"\n )\n\n def test_maybe_unpair_preference_dataset_dict(self):\n # Test that a paired dataset dict is correctly converted to unpaired with maybe_unpair_preference_dataset\n paired_dataset_dict = DatasetDict({\"abc\": self.paired_dataset})\n unpaired_dataset_dict = maybe_unpair_preference_dataset(paired_dataset_dict)\n assert unpaired_dataset_dict[\"abc\"].to_dict() == self.unpaired_dataset.to_dict(), (\n \"The paired dataset should be converted to unpaired.\"\n )\n\n def test_maybe_unpair_preference_dataset_already_paired(self):\n # Test that a paired dataset remains unchanged with maybe_unpair_preference_dataset\n unpaired_dataset = maybe_unpair_preference_dataset(self.unpaired_dataset)\n assert unpaired_dataset.to_dict() == self.unpaired_dataset.to_dict(), (\n \"The unpaired dataset should remain unchanged.\"\n )\n\n def test_maybe_unpair_preference_dataset_dict_already_paired(self):\n # Test that a paired dataset dict remains unchanged with maybe_unpair_preference_dataset\n unpaired_dataset_dict = maybe_unpair_preference_dataset(DatasetDict({\"abc\": self.unpaired_dataset}))\n assert unpaired_dataset_dict[\"abc\"].to_dict() == self.unpaired_dataset.to_dict(), (\n \"The unpaired dataset should remain unchanged.\"\n )\n\n\nclass TestExtractPrompt(TrlTestCase):\n example_implicit_prompt_conversational = {\n \"chosen\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ],\n \"rejected\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is green.\"},\n ],\n }\n\n example_explicit_prompt_conversational = {\n \"prompt\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n \"chosen\": [\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ],\n \"rejected\": [\n {\"role\": \"assistant\", \"content\": \"It is green.\"},\n ],\n }\n\n example_implicit_prompt_standard = {\n \"chosen\": \"The sky is blue.\",\n \"rejected\": \"The sky is green.\",\n }\n\n example_explicit_prompt_standard = {\n \"prompt\": \"The sky is\",\n \"chosen\": \" blue.\",\n \"rejected\": \" green.\",\n }\n\n def test_extract_prompt_conversational(self):\n # Test that the prompt is correctly extracted from the dataset\n example_extracted_prompt = extract_prompt(self.example_implicit_prompt_conversational)\n assert example_extracted_prompt == self.example_explicit_prompt_conversational, (\n \"The prompt is not correctly extracted from the dataset.\"\n )\n\n def test_maybe_extract_prompt_conversational(self):\n # Test that the prompt is correctly extracted from the dataset with maybe_extract_prompt\n example_extracted_prompt = maybe_extract_prompt(self.example_implicit_prompt_conversational)\n assert example_extracted_prompt == self.example_explicit_prompt_conversational, (\n \"The prompt is not correctly extracted from the dataset.\"\n )\n\n def test_maybe_extract_prompt_conversational_already_explicit(self):\n # Test that the prompt remains unchanged with maybe_extract_prompt\n example_extracted_prompt = maybe_extract_prompt(self.example_explicit_prompt_conversational)\n assert example_extracted_prompt == self.example_explicit_prompt_conversational, (\n \"The prompt should remain unchanged.\"\n )\n\n def test_extract_prompt_standard(self):\n # Test that the prompt is correctly extracted from the dataset\n example_extracted_prompt = extract_prompt(self.example_implicit_prompt_standard)\n assert example_extracted_prompt == self.example_explicit_prompt_standard, (\n \"The prompt is not correctly extracted from the dataset.\"\n )\n\n def test_maybe_extract_prompt_standard(self):\n # Test that the prompt is correctly extracted from the dataset with maybe_extract_prompt\n example_extracted_prompt = maybe_extract_prompt(self.example_implicit_prompt_standard)\n assert example_extracted_prompt == self.example_explicit_prompt_standard, (\n \"The prompt is not correctly extracted from the dataset.\"\n )\n\n def test_maybe_extract_prompt_standard_already_explicit(self):\n # Test that the prompt remains unchanged with maybe_extract_prompt\n example_extracted_prompt = maybe_extract_prompt(self.example_explicit_prompt_standard)\n assert example_extracted_prompt == self.example_explicit_prompt_standard, \"The prompt should remain unchanged.\"\n\n\nclass TestPackDatasetWrapped(TrlTestCase):\n def test_with_dataset(self):\n examples = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n \"attention_mask\": [[0, 1, 1], [0, 0, 1, 1], [1]],\n }\n dataset = Dataset.from_dict(examples)\n dataset = dataset.with_format(\"numpy\", dtype=\"float32\")\n format = dataset.format\n seq_length = 3\n expected_output = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6], [7, 8]],\n \"attention_mask\": [[0, 1, 1], [0, 0, 1], [1, 1]],\n }\n dataset = pack_dataset(dataset, seq_length, strategy=\"wrapped\")\n assert dataset.to_dict() == expected_output\n assert format == dataset.format\n\n def test_with_iterable_dataset(self):\n examples = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n \"attention_mask\": [[0, 1, 1], [0, 0, 1, 1], [1]],\n }\n dataset = Dataset.from_dict(examples).to_iterable_dataset()\n dataset = dataset.with_format(\"numpy\")\n formatting = dataset._formatting\n seq_length = 3\n expected_output = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6], [7, 8]],\n \"attention_mask\": [[0, 1, 1], [0, 0, 1], [1, 1]],\n }\n dataset = pack_dataset(dataset, seq_length, strategy=\"wrapped\")\n num_examples = len(examples[next(iter(examples))])\n assert next(iter(dataset.with_format(None).batch(batch_size=num_examples))) == expected_output\n assert formatting == dataset._formatting\n\n\nclass TestPackDatasetBfd(TrlTestCase):\n def test_with_dataset(self):\n examples = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n }\n dataset = Dataset.from_dict(examples)\n dataset = dataset.with_format(\"numpy\", dtype=\"float32\")\n format = dataset.format\n seq_length = 4\n expected_output = {\n \"input_ids\": [[4, 5, 6, 7], [1, 2, 3, 8]],\n \"seq_lengths\": [[4], [3, 1]],\n }\n dataset = pack_dataset(dataset, seq_length, strategy=\"bfd\")\n expected_format = dataset.format\n assert dataset.to_dict() == expected_output\n assert \"seq_lengths\" in expected_format[\"columns\"]\n expected_format[\"columns\"].remove(\"seq_lengths\")\n assert format == dataset.format\n\n def test_with_iterable_dataset(self):\n examples = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n }\n dataset = Dataset.from_dict(examples).to_iterable_dataset()\n dataset = dataset.with_format(\"numpy\")\n formatting = dataset._formatting\n seq_length = 4\n expected_output = {\n \"input_ids\": [[4, 5, 6, 7], [1, 2, 3, 8]],\n \"seq_lengths\": [[4], [3, 1]],\n }\n dataset = pack_dataset(dataset, seq_length, strategy=\"bfd\")\n num_examples = len(examples[next(iter(examples))])\n assert next(iter(dataset.with_format(None).batch(batch_size=num_examples))) == expected_output\n assert formatting == dataset._formatting\n\n def test_with_overlong_0(self):\n examples = {\n \"input_ids\": [[1, 2, 3, 4, 5], [6, 7], [8, 9, 10, 11], [12]],\n }\n dataset = Dataset.from_dict(examples)\n seq_length = 4\n expected_output = {\n \"input_ids\": [[1, 2, 3, 4], [8, 9, 10, 11], [6, 7, 5, 12]],\n \"seq_lengths\": [[4], [4], [2, 1, 1]],\n }\n dataset = pack_dataset(dataset, seq_length, strategy=\"bfd_split\")\n assert dataset.to_dict() == expected_output\n\n def test_with_overlong_two_coluns(self):\n examples = {\n \"col1\": [[1, -2, 3, -4, 5, -6], [7, -8, 9], [-10, 11, -12], [13, -14, 15, -16]],\n \"col2\": [[-1, 2, -3, 4, -5, 6], [-7, 8, -9], [10, -11, 12], [-13, 14, -15, 16]],\n }\n dataset = Dataset.from_dict(examples)\n seq_length = 4\n expected_output = {\n \"col1\": [[1, -2, 3, -4], [13, -14, 15, -16], [7, -8, 9], [-10, 11, -12], [5, -6]],\n \"col2\": [[-1, 2, -3, 4], [-13, 14, -15, 16], [-7, 8, -9], [10, -11, 12], [-5, 6]],\n \"seq_lengths\": [[4], [4], [3], [3], [2]],\n }\n dataset = pack_dataset(dataset, seq_length, strategy=\"bfd_split\")\n assert dataset.to_dict() == expected_output\n\n def test_with_non_power_of_2(self):\n examples = {\n \"input_ids\": [[1, 2, 3, 4, 5], [6], [7, 8, 9, 10], [11, 12, 13]],\n }\n dataset = Dataset.from_dict(examples)\n seq_length = 5\n expected_output = {\n \"input_ids\": [[1, 2, 3, 4, 5], [7, 8, 9, 10, 6], [11, 12, 13]],\n \"seq_lengths\": [[5], [4, 1], [3]],\n }\n dataset = pack_dataset(dataset, seq_length, strategy=\"bfd_split\")\n assert dataset.to_dict() == expected_output\n\n def test_default_no_split(self):\n \"\"\"Test default 'bfd' strategy for SFT datasets (truncates overflow).\"\"\"\n examples = {\n \"input_ids\": [[1, 2, 3, 4, 5], [6, 7], [8, 9, 10, 11], [12]],\n }\n dataset = Dataset.from_dict(examples)\n seq_length = 4\n # With default 'bfd' strategy, overflow tokens are discarded\n expected_output = {\n \"input_ids\": [[1, 2, 3, 4], [8, 9, 10, 11], [6, 7, 12]],\n \"seq_lengths\": [[4], [4], [2, 1]],\n }\n dataset = pack_dataset(dataset, seq_length, strategy=\"bfd\")\n assert dataset.to_dict() == expected_output\n\n def test_with_empty_sequences(self):\n examples = {\n \"input_ids\": [[1, 2], [], [3, 4, 5], [], [6]],\n }\n dataset = Dataset.from_dict(examples)\n seq_length = 4\n expected_output = {\n \"input_ids\": [[3, 4, 5, 6], [1, 2]],\n \"seq_lengths\": [[3, 1], [2]],\n }\n dataset = pack_dataset(dataset, seq_length, strategy=\"bfd_split\")\n assert dataset.to_dict() == expected_output\n\n\nclass TestTruncateExamples(TrlTestCase):\n def test_with_dataset(self):\n examples = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n \"attention_mask\": [[0, 1, 1], [0, 0, 1, 1], [1]],\n }\n dataset = Dataset.from_dict(examples)\n dataset = dataset.with_format(\"numpy\", dtype=\"float32\")\n format = dataset.format\n max_length = 2\n expected_output = {\n \"input_ids\": [[1, 2], [4, 5], [8]],\n \"attention_mask\": [[0, 1], [0, 0], [1]],\n }\n dataset = truncate_dataset(dataset, max_length)\n assert dataset.to_dict() == expected_output\n assert format == dataset.format\n\n def test_with_iterable_dataset(self):\n examples = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n \"attention_mask\": [[0, 1, 1], [0, 0, 1, 1], [1]],\n }\n dataset = Dataset.from_dict(examples).to_iterable_dataset()\n dataset = dataset.with_format(\"numpy\")\n formatting = dataset._formatting\n max_length = 2\n expected_output = {\n \"input_ids\": [[1, 2], [4, 5], [8]],\n \"attention_mask\": [[0, 1], [0, 0], [1]],\n }\n dataset = truncate_dataset(dataset, max_length)\n num_examples = len(examples[next(iter(examples))])\n assert next(iter(dataset.with_format(None).batch(batch_size=num_examples))) == expected_output\n assert formatting == dataset._formatting\n\n def test_with_extra_column(self):\n examples = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n \"attention_mask\": [[0, 1, 1], [0, 0, 1, 1], [1]],\n \"my_column\": [\"a\", \"b\", \"c\"],\n }\n dataset = Dataset.from_dict(examples)\n max_length = 2\n expected_output = {\n \"input_ids\": [[1, 2], [4, 5], [8]],\n \"attention_mask\": [[0, 1], [0, 0], [1]],\n \"my_column\": [\"a\", \"b\", \"c\"],\n }\n dataset = truncate_dataset(dataset, max_length)\n assert dataset.to_dict() == expected_output\n\n def test_with_keep_end(self):\n examples = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n \"attention_mask\": [[0, 1, 1], [0, 0, 1, 1], [1]],\n }\n dataset = Dataset.from_dict(examples)\n expected_output = {\n \"input_ids\": [[2, 3], [6, 7], [8]],\n \"attention_mask\": [[1, 1], [1, 1], [1]],\n }\n dataset = truncate_dataset(dataset, max_length=2, truncation_mode=\"keep_end\")\n assert dataset.to_dict() == expected_output\n\n def test_with_keep_end_and_zero_max_length(self):\n examples = {\n \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n \"attention_mask\": [[0, 1, 1], [0, 0, 1, 1], [1]],\n }\n dataset = Dataset.from_dict(examples)\n expected_output = {\n \"input_ids\": [[], [], []],\n \"attention_mask\": [[], [], []],\n }\n dataset = truncate_dataset(dataset, max_length=0, truncation_mode=\"keep_end\")\n assert dataset.to_dict() == expected_output\n\n\nclass TestMaybeConvertToChatML(TrlTestCase):\n def test_with_conversations_key(self):\n # Particular case where the key is \"conversations\": we rename it to \"messages\"\n example = {\n \"conversations\": [\n {\"from\": \"user\", \"value\": \"What color is the sky?\"},\n {\"from\": \"assistant\", \"value\": \"It is blue.\"},\n ]\n }\n expected_output = {\n \"messages\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ]\n }\n assert maybe_convert_to_chatml(example) == expected_output\n\n def test_without_conversations_key(self):\n # Same as before, but we don't rename the keys\n example = {\n \"prompt\": [{\"from\": \"user\", \"value\": \"What color is the sky?\"}],\n \"completion\": [{\"from\": \"assistant\", \"value\": \"It is blue.\"}],\n }\n expected_output = {\n \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n }\n assert maybe_convert_to_chatml(example) == expected_output\n\n def test_not_conversional(self):\n # When not needed, the example should remain unchanged\n example = {\"text\": \"The sky is blue.\"}\n assert maybe_convert_to_chatml(example) == example\n\n def test_already_chatml(self):\n # When the example is already in ChatML format, it should remain unchanged\n example = {\n \"messages\": [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ]\n }\n assert maybe_convert_to_chatml(example) == example\n" }, { "path": "tests/test_dpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport pytest\nimport torch\nimport transformers\nfrom datasets import load_dataset\nfrom packaging.version import Version\nfrom packaging.version import parse as parse_version\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\nfrom transformers.utils import is_peft_available\n\nfrom trl import DPOConfig, DPOTrainer\nfrom trl.trainer.dpo_trainer import DataCollatorForPreference, DataCollatorForVisionPreference\n\nfrom .testing_utils import (\n TrlTestCase,\n require_ampere_or_newer,\n require_bitsandbytes,\n require_kernels,\n require_liger_kernel,\n require_peft,\n require_vision,\n)\n\n\nif is_peft_available():\n from peft import LoraConfig, get_peft_model\n\n\nclass TestDataCollatorForPreference(TrlTestCase):\n def test_padding_and_masks(self):\n collator = DataCollatorForPreference(pad_token_id=0)\n examples = [\n {\"prompt_ids\": [1, 2, 3], \"chosen_ids\": [4, 5], \"rejected_ids\": [6]},\n {\"prompt_ids\": [7, 8], \"chosen_ids\": [9, 10], \"rejected_ids\": [11, 12, 13]},\n ]\n result = collator(examples)\n\n expected_input_ids = torch.tensor(\n [\n [1, 2, 3, 4, 5], # prompt + chosen (example 1)\n [7, 8, 9, 10, 0], # prompt + chosen (example 2, padded)\n [1, 2, 3, 6, 0], # prompt + rejected (example 1, padded)\n [7, 8, 11, 12, 13], # prompt + rejected (example 2)\n ]\n )\n expected_attention_mask = torch.tensor(\n [\n [1, 1, 1, 1, 1],\n [1, 1, 1, 1, 0],\n [1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1],\n ]\n )\n expected_completion_mask = torch.tensor(\n [\n [0, 0, 0, 1, 1], # chosen completion (example 1)\n [0, 0, 1, 1, 0], # chosen completion (example 2, padded)\n [0, 0, 0, 1, 0], # rejected completion (example 1, padded)\n [0, 0, 1, 1, 1], # rejected completion (example 2)\n ]\n )\n\n assert set(result.keys()) == {\"input_ids\", \"attention_mask\", \"completion_mask\"}\n torch.testing.assert_close(result[\"input_ids\"], expected_input_ids)\n torch.testing.assert_close(result[\"attention_mask\"], expected_attention_mask)\n torch.testing.assert_close(result[\"completion_mask\"], expected_completion_mask)\n\n def test_optional_reference_logps(self):\n collator = DataCollatorForPreference(pad_token_id=0)\n examples = [\n {\n \"prompt_ids\": [1, 2],\n \"chosen_ids\": [3],\n \"rejected_ids\": [4],\n \"ref_chosen_logps\": 0.1,\n \"ref_rejected_logps\": 0.2,\n },\n {\n \"prompt_ids\": [5],\n \"chosen_ids\": [6, 7],\n \"rejected_ids\": [8, 9],\n \"ref_chosen_logps\": 0.3,\n \"ref_rejected_logps\": 0.4,\n },\n ]\n result = collator(examples)\n\n expected_ref_chosen_logps = torch.tensor([0.1, 0.3])\n expected_ref_rejected_logps = torch.tensor([0.2, 0.4])\n\n assert set(result.keys()) == {\n \"input_ids\",\n \"attention_mask\",\n \"completion_mask\",\n \"ref_chosen_logps\",\n \"ref_rejected_logps\",\n }\n torch.testing.assert_close(result[\"ref_chosen_logps\"], expected_ref_chosen_logps)\n torch.testing.assert_close(result[\"ref_rejected_logps\"], expected_ref_rejected_logps)\n\n def test_with_pad_to_multiple_of(self):\n collator = DataCollatorForPreference(pad_token_id=0, pad_to_multiple_of=5)\n examples = [\n {\"prompt_ids\": [1], \"chosen_ids\": [2], \"rejected_ids\": [3]},\n {\"prompt_ids\": [4, 5], \"chosen_ids\": [6, 7], \"rejected_ids\": [8, 9]},\n ]\n result = collator(examples)\n\n expected_input_ids = torch.tensor(\n [\n [1, 2, 0, 0, 0], # prompt + chosen (example 1, padded to multiple of 5)\n [4, 5, 6, 7, 0], # prompt + chosen (example 2)\n [1, 3, 0, 0, 0], # prompt + rejected (example 1, padded to multiple of 5)\n [4, 5, 8, 9, 0], # prompt + rejected (example 2)\n ]\n )\n\n assert set(result.keys()) == {\"input_ids\", \"attention_mask\", \"completion_mask\"}\n torch.testing.assert_close(result[\"input_ids\"], expected_input_ids)\n\n\nclass TestDataCollatorForVisionPreference(TrlTestCase):\n @pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.3.0\"),\n reason=\"mm_token_type_ids are returned by default since transformers-5.3.0 (see transformers#43972)\",\n )\n @require_vision\n def test_mm_token_type_ids_shape(self):\n # Regression test: when the processor returns mm_token_type_ids (e.g. Qwen2.5-VL after\n # transformers#43972), the collator must concatenate it with zeros for the completion part\n # so that its shape matches input_ids. Without the fix this raises an IndexError in the model.\n from PIL import Image\n from transformers import AutoProcessor\n\n processor = AutoProcessor.from_pretrained(\"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\")\n collator = DataCollatorForVisionPreference(processor)\n image = Image.new(\"RGB\", (16, 16))\n examples = [\n {\n \"images\": [image],\n \"prompt\": [{\"role\": \"user\", \"content\": \"What is this?\"}],\n \"chosen\": [{\"role\": \"assistant\", \"content\": \"A red square.\"}],\n \"rejected\": [{\"role\": \"assistant\", \"content\": \"A blue circle.\"}],\n }\n ]\n output = collator(examples)\n assert \"mm_token_type_ids\" in output\n assert output[\"mm_token_type_ids\"].shape == output[\"input_ids\"].shape, (\n f\"mm_token_type_ids shape {output['mm_token_type_ids'].shape} != \"\n f\"input_ids shape {output['input_ids'].shape}\"\n )\n\n\nclass TestDPOTrainer(TrlTestCase):\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"trl-internal-testing/tiny-Qwen3MoeForCausalLM\",\n \"trl-internal-testing/tiny-GptOssForCausalLM\",\n ],\n )\n def test_train(self, model_id):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n trainer = DPOTrainer(model=model_id, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n # Special case for harmony\n def test_train_gpt_oss(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/harmony\", \"preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-GptOssForCausalLM\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_model(self):\n # Instantiate the model\n model = AutoModelForCausalLM.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n dtype=\"float32\",\n )\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n trainer = DPOTrainer(model=model, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\n \"loss_type\",\n [\n \"sigmoid\",\n \"hinge\",\n \"ipo\",\n \"exo_pair\",\n \"nca_pair\",\n \"robust\",\n \"bco_pair\",\n \"sppo_hard\",\n \"aot\",\n \"aot_unpaired\",\n \"apo_zero\",\n \"apo_down\",\n \"discopop\",\n \"sft\",\n ],\n )\n def test_train_loss_types(self, loss_type):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n loss_type=loss_type,\n label_smoothing=1e-3 if loss_type == \"exo_pair\" else 0.0,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n eval_strategy=\"steps\",\n eval_steps=3,\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_multi_loss_types(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n loss_type=[\"sigmoid\", \"bco_pair\", \"sft\"], # this specific combination is used in MPO\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_wpo(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n use_weighting=True,\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_ld(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n ld_alpha=0.5,\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\n \"f_divergence_type\",\n [\"reverse_kl\", \"forward_kl\", \"js_divergence\", \"alpha_divergence\"],\n )\n def test_train_with_f_divergence(self, f_divergence_type):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n f_divergence_type=f_divergence_type,\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_explicit_ref_model(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1,\n report_to=\"none\",\n )\n # When specifying a ref model, it's usually because we want it to be a different checkpoint, but for testing\n # purposes we will just just use the same checkpoint\n ref_model = AutoModelForCausalLM.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\"\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n ref_model=ref_model,\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n new_ref_param = trainer.ref_model.get_parameter(n)\n torch.testing.assert_close(param, new_ref_param), f\"Reference model parameter {n} has changed\"\n\n def test_training_with_sync_ref_model(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n sync_ref_model=True,\n ref_model_sync_steps=2, # reduce sync steps to ensure a sync happens\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n assert trainer.ref_model is not None\n previous_ref_params = {n: param.clone() for n, param in trainer.ref_model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n new_ref_param = trainer.ref_model.get_parameter(n)\n assert not torch.equal(previous_ref_params[n], new_ref_param), f\"Ref Parameter {n} has not changed.\"\n\n def test_train_model_dtype(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n model_init_kwargs={\"dtype\": torch.float16},\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n # For some reasonn model.layers.0.input_layernorm.weight doesn't change in GitHub Actions but does\n # locally. We ignore this parameter for now\n if \"layernorm\" in n:\n continue\n new_param = trainer.model.get_parameter(n)\n # Check the torch dtype\n assert new_param.dtype == torch.float16\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_peft\n def test_train_dense_with_peft_config_lora(self):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=1.0, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n\n trainer = DPOTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_peft\n def test_train_moe_with_peft_config(self):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-GptOssForCausalLM\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n\n trainer = DPOTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(target_parameters=[\"mlp.experts.down_proj\", \"mlp.experts.gate_up_proj\"]),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_peft\n def test_train_peft_model(self):\n # Get the base model\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n\n # Get the base model parameter names\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Turn the model into a peft model\n lora_config = LoraConfig()\n model = get_peft_model(model, lora_config)\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=1.0, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n trainer = DPOTrainer(model=model, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n and \"ref\" not in n: # and the peft params to be different (except base and ref)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n # In practice, this test is the same as `test_train_dense_with_peft_config_lora`, since gradient checkpointing is\n # enabled by default in `DPOTrainer`. We keep it as a regression guard: if the default ever changes, we still\n # explicitly test PEFT + gradient checkpointing, which has caused issues in the past.\n @require_peft\n def test_train_with_peft_config_and_gradient_checkpointing(self):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n gradient_checkpointing=True,\n report_to=\"none\",\n )\n\n trainer = DPOTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_liger_kernel\n def test_train_with_liger(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n use_liger_kernel=True,\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_iterable_dataset(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\", streaming=True)\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n max_steps=3,\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_kernels\n @require_ampere_or_newer # Flash attention 2 requires Ampere or newer GPUs\n def test_train_padding_free(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n padding_free=True,\n model_init_kwargs={\"attn_implementation\": \"kernels-community/flash-attn2\"},\n bf16=True, # flash_attention_2 only supports bf16 and fp16\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_chat_template_kwargs(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n # The following template is a simplified version of the Qwen chat template, where an additional argument\n # `role_capital` is used to control the capitalization of roles.\n tokenizer.chat_template = '{%- if messages[0][\"role\"] == \"system\" -%} {{ \"<|im_start|>\" + (\"SYSTEM\" if role_capital else \"system\") + \"\\\\n\" + messages[0][\"content\"] + \"<|im_end|>\\\\n\" }}{%- else -%} {{ \"<|im_start|>\" + (\"SYSTEM\" if role_capital else \"system\") + \"\\\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\\\n\" }}{%- endif -%}{%- for message in messages -%} {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) -%} {{ \"<|im_start|>\" + (message.role.upper() if role_capital else message.role) + \"\\\\n\" + message.content + \"<|im_end|>\\\\n\" }} {%- elif message.role == \"assistant\" -%} {{ \"<|im_start|>\" + (\"ASSISTANT\" if role_capital else \"assistant\") }} {%- if message.content -%} {{ \"\\\\n\" + message.content }} {%- endif -%} {{ \"<|im_end|>\\\\n\" }} {%- elif message.role == \"tool\" -%} {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") -%} {{ \"<|im_start|>\" + (\"USER\" if role_capital else \"user\") }} {%- endif -%} {{ \"\\\\n\\\\n\" + message.content + \"\\\\n\" }} {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") -%} {{ \"<|im_end|>\\\\n\" }} {%- endif -%} {%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%} {{ \"<|im_start|>\" + (\"ASSISTANT\" if role_capital else \"assistant\") + \"\\\\n\" }}{%- endif -%}'\n\n dataset = dataset.add_column(\n \"chat_template_kwargs\", [{\"role_capital\": bool(i % 2)} for i in range(len(dataset))]\n )\n assert \"chat_template_kwargs\" in dataset.features\n\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n processing_class=tokenizer,\n )\n\n # Assert trainer uses the same chat template as tokenizer\n assert trainer.processing_class.chat_template == tokenizer.chat_template\n\n # Assert chat_template is applied\n for i in range(2):\n role = \"SYSTEM\" if i else \"system\"\n system_prompt = (\n f\"<|im_start|>{role}\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\"\n )\n system_prompt_ids = trainer.processing_class(system_prompt)[\"input_ids\"]\n assert trainer.train_dataset[i][\"prompt_ids\"][: len(system_prompt_ids)] == system_prompt_ids\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_toolcall_data(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/toolcall\", \"preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_eval(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\")\n\n # Initialize the trainer\n training_args = DPOConfig(output_dir=self.tmp_dir, eval_strategy=\"steps\", eval_steps=3, report_to=\"none\")\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n # Train the model\n trainer.train()\n\n # Check that the eval loss is not None\n assert trainer.state.log_history[0][\"eval_loss\"] is not None\n\n def test_train_with_multiple_eval_dataset(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\")\n\n # Initialize the trainer\n training_args = DPOConfig(output_dir=self.tmp_dir, eval_strategy=\"steps\", eval_steps=3, report_to=\"none\")\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset={\"data1\": dataset[\"test\"], \"data2\": dataset[\"test\"]},\n )\n # Train the model\n trainer.train()\n\n # Check that the eval losses are not None\n assert trainer.state.log_history[-3][\"eval_data1_loss\"] is not None\n assert trainer.state.log_history[-2][\"eval_data2_loss\"] is not None\n\n def test_train_with_compute_metrics(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\")\n\n def dummy_compute_metrics(eval_pred):\n return {\"my_metric\": 0.123}\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n eval_strategy=\"steps\",\n eval_steps=3,\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n compute_metrics=dummy_compute_metrics,\n )\n\n # Train the model\n trainer.train()\n\n # Check that the custom metric is logged\n assert trainer.state.log_history[-2][\"eval_my_metric\"] == 0.123\n\n # In practice, this test is the same as `test_train`, since gradient checkpointing is enabled by default in\n # `DPOTrainer`. We keep it as a regression guard: if the default ever changes, we still explicitly test gradient\n # checkpointing, which has caused issues in the past.\n def test_train_with_gradient_checkpointing(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n gradient_checkpointing=True,\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_tag_added(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n train_dataset=dataset,\n )\n\n for tag in [\"dpo\", \"trl\"]:\n assert tag in trainer.model.model_tags\n\n @require_peft\n def test_tag_added_peft(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n for tag in [\"dpo\", \"trl\"]:\n assert tag in trainer.model.model_tags\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n # \"trl-internal-testing/tiny-Idefics2ForConditionalGeneration\", high memory peak, skipped for now\n # \"trl-internal-testing/tiny-Idefics3ForConditionalGeneration\", high memory peak, skipped for now\n \"trl-internal-testing/tiny-LlavaForConditionalGeneration\",\n \"trl-internal-testing/tiny-LlavaNextForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2VLForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n # \"trl-internal-testing/tiny-SmolVLMForConditionalGeneration\", seems not to support bf16 properly\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3VLForConditionalGeneration\",\n marks=[\n pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"4.57.0\"),\n reason=\"Qwen3-VL series were introduced in transformers-4.57.0\",\n ),\n pytest.mark.xfail(\n Version(transformers.__version__) >= Version(\"5.0.0\"),\n reason=\"Blocked by upstream transformers bug (transformers#43334)\",\n ),\n ],\n ),\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration\",\n marks=pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.2.0\"),\n reason=\"Qwen3.5 models were introduced in transformers-5.2.0\",\n ),\n ),\n ],\n )\n @require_vision\n def test_train_vlm(self, model_id):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n max_length=None, # for VLMs, truncating can remove image tokens, leading to errors\n per_device_train_batch_size=2, # VLM training is memory intensive, reduce batch size to avoid OOM\n report_to=\"none\",\n )\n trainer = DPOTrainer(model=model_id, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n # For some reason, these params are not updated. This is probably not related to TRL, but to\n # the model itself. We should investigate this further, but for now we just skip these params.\n # fmt: off\n if (\n model_id == \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\" and \"model.vision_tower.vision_model.head\" in n or\n model_id == \"trl-internal-testing/tiny-LlavaForConditionalGeneration\" and \"model.vision_tower.vision_model.post_layernorm\" in n or\n model_id == \"trl-internal-testing/tiny-LlavaForConditionalGeneration\" and \"vision_tower.vision_model.encoder.layers.1\" in n or\n model_id == \"trl-internal-testing/tiny-LlavaNextForConditionalGeneration\" and \"model.vision_tower.vision_model.post_layernorm\" in n or\n model_id == \"trl-internal-testing/tiny-LlavaNextForConditionalGeneration\" and \"vision_tower.vision_model.encoder.layers.1\" in n or\n model_id == \"trl-internal-testing/tiny-Qwen3VLForConditionalGeneration\" and \"model.visual.deepstack_merger_list\" in n\n ):\n # fmt: on\n continue\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is not updated\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @pytest.mark.xfail(\n parse_version(transformers.__version__) < parse_version(\"4.57.0\"),\n reason=\"Mixing text-only and image+text examples is only supported in transformers >= 4.57.0\",\n strict=False,\n )\n @require_vision\n def test_train_vlm_multi_image(self, model_id):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen-multi-image\", \"conversational_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n max_length=None, # for VLMs, truncating can remove image tokens, leading to errors\n per_device_train_batch_size=1, # VLM training is memory intensive, reduce batch size to avoid OOM\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is not updated\"\n\n # Gemma 3n uses a timm encoder, making it difficult to create a smaller variant for testing.\n # To ensure coverage, we run tests on the full model but mark them as slow to exclude from default runs.\n @pytest.mark.slow\n @require_vision\n @pytest.mark.skip(reason=\"Model google/gemma-3n-E2B-it is gated and requires HF token\")\n def test_train_vlm_gemma_3n(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n max_length=None, # for VLMs, truncating can remove image tokens, leading to errors\n per_device_train_batch_size=1, # VLM training is memory intensive, reduce batch size to avoid OOM\n model_init_kwargs={\"dtype\": \"bfloat16\"},\n report_to=\"none\",\n )\n trainer = DPOTrainer(model=\"google/gemma-3n-E2B-it\", args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if \"model.audio_tower\" in n or \"model.embed_audio\" in n:\n # The audio embedding parameters are not updated because this dataset contains no audio data\n continue\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is not updated\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @pytest.mark.parametrize(\n \"dataset_config\",\n [\"conversational_preference\", \"standard_preference\"],\n )\n @require_vision\n def test_train_vlm_text_only_data(self, model_id, dataset_config):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", dataset_config, split=\"train\")\n\n # Initialize the trainer\n training_args = DPOConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = DPOTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n.startswith(\"model.visual\"):\n torch.testing.assert_close(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is updated\"\n else:\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is not updated\"\n\n @require_vision\n def test_train_vlm_with_max_length(self):\n # Regression test for #5283: mm_token_type_ids must be truncated alongside input_ids when max_length is set,\n # otherwise a shape mismatch crashes the model forward pass.\n # max_length=37 truncates 1 completion token (total_len=38) while keeping all image tokens (prompt_len=34) safe.\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_preference\", split=\"train\")\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n max_length=37, # total_len=38, prompt_len=34 โ€” truncates completion, not image tokens\n per_device_train_batch_size=2,\n report_to=\"none\",\n )\n trainer = DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n args=training_args,\n train_dataset=dataset,\n )\n trainer.train()\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n @require_peft\n @require_bitsandbytes\n def test_peft_with_quantization(self):\n # Get the base model\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n\n quantization_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_compute_dtype=torch.float16,\n )\n model = AutoModelForCausalLM.from_pretrained(\n model_id,\n dtype=\"float32\",\n quantization_config=quantization_config,\n )\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_preference\", split=\"train\")\n\n # Initialize the trainer with the already configured PeftModel\n training_args = DPOConfig(output_dir=self.tmp_dir, learning_rate=0.1, report_to=\"none\")\n trainer = DPOTrainer(model=model, args=training_args, train_dataset=dataset, peft_config=LoraConfig())\n\n # Save initial parameters to check they change during training\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n # Check that training completed successfully\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert trainer.state.log_history[-1][\"mean_token_accuracy\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n # In bitsandbytes, bias parameters are automatically cast to the input dtype during the forward pass if\n # their dtype doesnโ€™t match. This causes the module to change unexpectedly during the first forward pass of\n # the training. To handle this, we cast these specific bias parameters to float32 before comparison.\n # https://github.com/bitsandbytes-foundation/bitsandbytes/blob/45553f7392e524eacf400b132cfe01261f6477be/bitsandbytes/nn/modules.py#L518\n # We still need to investigate why the compute dtype ends up being different than for these parameters.\n if n in [\n \"base_model.model.model.layers.1.self_attn.k_proj.bias\",\n \"base_model.model.model.layers.1.self_attn.q_proj.base_layer.bias\",\n \"base_model.model.model.layers.1.self_attn.v_proj.base_layer.bias\",\n ]:\n param = param.float()\n\n if \"lora\" not in n: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"lora\" in n: # We expect the peft parameters to be different\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n else:\n raise ValueError(f\"Unexpected parameter {n} in model: {trainer.model}\")\n\n @require_vision\n def test_train_vlm_keep_end_raises(self):\n # Regression test for #5285: keep_end with a VLM must raise at init time, not silently corrupt training.\n # Image tokens live at the start of the sequence (in the prompt); keep_end would drop them.\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_preference\", split=\"train\")\n training_args = DPOConfig(\n output_dir=self.tmp_dir,\n max_length=32,\n truncation_mode=\"keep_end\",\n report_to=\"none\",\n )\n with pytest.raises(ValueError, match=\"truncation_mode='keep_end' is not supported for vision-language models\"):\n DPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n args=training_args,\n train_dataset=dataset,\n )\n" }, { "path": "tests/test_grpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport gc\nimport os\nimport warnings\nfrom collections.abc import Callable\nfrom types import SimpleNamespace\nfrom unittest.mock import MagicMock, patch\n\nimport numpy as np\nimport pytest\nimport torch\nimport transformers\nfrom accelerate.utils.memory import release_memory\nfrom datasets import Dataset, Features, Image, Value, load_dataset\nfrom packaging.version import Version\nfrom transformers import (\n AutoModelForCausalLM,\n AutoModelForImageTextToText,\n AutoModelForSequenceClassification,\n AutoProcessor,\n AutoTokenizer,\n BitsAndBytesConfig,\n)\nfrom transformers.testing_utils import backend_empty_cache, torch_device\nfrom transformers.utils import is_peft_available\n\nfrom trl import GRPOConfig, GRPOTrainer\nfrom trl.import_utils import is_liger_kernel_available\nfrom trl.trainer.utils import get_kbit_device_map\n\nfrom .testing_utils import (\n TrlTestCase,\n require_ampere_or_newer,\n require_bitsandbytes,\n require_jmespath,\n require_kernels,\n require_liger_kernel,\n require_peft,\n require_torch_accelerator,\n require_vision,\n require_vllm,\n)\n\n\nif is_peft_available():\n from peft import LoraConfig, PeftModel, get_peft_model\n\n\ndef multiply_tool(a: int, b: int) -> int:\n \"\"\"\n Multiplies two integers.\n\n Args:\n a: The first integer.\n b: The second integer.\n\n Returns:\n The product of the two integers.\n \"\"\"\n return a * b\n\n\nasync def async_multiply_tool(a: int, b: int) -> int:\n \"\"\"\n Asynchronously multiplies two integers.\n\n Args:\n a: The first integer.\n b: The second integer.\n\n Returns:\n The product of the two integers.\n \"\"\"\n return a * b\n\n\nclass TestGetHighEntropyMask(TrlTestCase):\n def get_high_entropy_mask(self, entropies, mask, threshold):\n \"\"\"Helper method to test the get_high_entropy_mask functionality.\"\"\"\n # Create a mock trainer with minimal setup\n from unittest.mock import Mock\n\n # Create a mock accelerator\n mock_accelerator = Mock()\n mock_accelerator.num_processes = 1 # Single process for testing\n\n # Create a minimal trainer instance just to access the method\n trainer = Mock(spec=GRPOTrainer)\n trainer.accelerator = mock_accelerator\n trainer.accelerator.gather = lambda x: x\n trainer.accelerator.pad_across_processes = lambda x, dim, pad_index: x\n\n # Call the actual method from GRPOTrainer\n return GRPOTrainer.get_high_entropy_mask(trainer, entropies, mask, threshold)\n\n def test_compute_entropy_mask_0(self):\n # We have a total of 12 tokens out of which 10 are non-pad.\n # for a top_entropy_quantile of 0.8, we expect the top 20% i.e 2 non-pad tokens corresponding to\n # the highest entropy to be unmasked.\n # In our example these will be the tokens corresponding to the entropies 0.9 and 1.0 since 1.1 and 1.2 are pad\n # tokens they are excluded from the entropy threshold calculation.\n entropies = torch.tensor([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6], [0.7, 0.8, 0.9, 1.0, 1.1, 1.2]])\n mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]])\n entropy_mask = self.get_high_entropy_mask(entropies, mask, threshold=0.8)\n expected_mask = torch.tensor([[0, 0, 0, 0, 0, 0], [0, 0, 1, 1, 0, 0]], dtype=torch.bool)\n torch.testing.assert_close(entropy_mask, expected_mask)\n\n def test_compute_entropy_mask_1(self):\n # Another example with a different set of entropies and a different mask.\n entropies = torch.tensor([[0.1, 0.2, 0.3, 1.4, 0.5, 0.14], [0.5, 0.6, 0.7, 0.8, 0.9, 1.0]])\n mask = torch.tensor([[1, 1, 1, 1, 0, 0], [1, 1, 1, 1, 0, 0]])\n entropy_mask = self.get_high_entropy_mask(entropies, mask, threshold=0.8)\n expected_mask = torch.tensor([[0, 0, 0, 1, 0, 0], [0, 0, 0, 1, 0, 0]], dtype=torch.bool)\n torch.testing.assert_close(entropy_mask, expected_mask)\n\n def test_compute_entropy_mask_lower_threshold(self):\n # For a threshold of 0.5 we expect the top half of the non-pad tokens to be unmasked.\n entropies = torch.tensor([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6], [0.7, 0.8, 0.9, 1.0, 1.1, 1.2]])\n mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]])\n entropy_mask = self.get_high_entropy_mask(entropies, mask, threshold=0.5)\n expected_mask = torch.tensor([[0, 0, 0, 0, 0, 1], [1, 1, 1, 1, 0, 0]], dtype=torch.bool)\n torch.testing.assert_close(entropy_mask, expected_mask)\n\n def test_compute_entropy_threshold_0(self):\n # If the threshold is 0.0 then we expect the mask to be all ones for non-pad tokens.\n entropies = torch.tensor([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6], [0.7, 0.8, 0.9, 1.0, 1.1, 1.2]])\n mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]])\n entropy_mask = self.get_high_entropy_mask(entropies, mask, threshold=0.0)\n expected_mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]], dtype=torch.bool)\n torch.testing.assert_close(entropy_mask, expected_mask)\n\n def test_compute_entropy_threshold_1(self):\n # If the threshold is 1.0 then we expect the mask to be all zeros BUT ONE VALUE.\n entropies = torch.tensor([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6], [0.7, 0.8, 0.9, 1.0, 1.1, 1.2]])\n mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]])\n entropy_mask = self.get_high_entropy_mask(entropies, mask, threshold=1.0)\n expected_mask = torch.tensor([[0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0]], dtype=torch.bool)\n torch.testing.assert_close(entropy_mask, expected_mask)\n\n def test_compute_entropy_all_masked(self):\n # If there are no non-pad tokens we expect the mask to be all zeros.\n entropies = torch.tensor([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6], [0.7, 0.8, 0.9, 1.0, 1.1, 1.2]])\n mask = torch.tensor([[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]])\n entropy_mask = self.get_high_entropy_mask(entropies, mask, threshold=0.5)\n expected_mask = torch.tensor([[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]], dtype=torch.bool)\n torch.testing.assert_close(entropy_mask, expected_mask)\n\n\nclass TestGRPORolloutDispatch:\n def _make_trainer(self):\n trainer = object.__new__(GRPOTrainer)\n trainer.accelerator = SimpleNamespace(\n device=torch.device(\"cpu\"),\n is_main_process=True,\n gather=lambda t: t,\n )\n trainer.args = SimpleNamespace(report_to=[])\n trainer.model = SimpleNamespace(training=True)\n trainer.state = SimpleNamespace(global_step=2, num_input_tokens_seen=0)\n trainer._last_loaded_step = 1\n trainer.use_vllm = False\n trainer.use_transformers_paged = False\n trainer.vllm_generation = SimpleNamespace(sync_weights=MagicMock())\n trainer.processing_class = SimpleNamespace(\n batch_decode=MagicMock(return_value=[\"decoded\"]),\n )\n trainer.tools = None\n trainer.eos_token_id = 2\n trainer.pad_token_id = 0\n trainer._metrics = {\n \"train\": {\n \"num_tokens\": [],\n **{\n k: []\n for k in [\n \"completions/mean_length\",\n \"completions/min_length\",\n \"completions/max_length\",\n \"completions/clipped_ratio\",\n \"completions/mean_terminated_length\",\n \"completions/min_terminated_length\",\n \"completions/max_terminated_length\",\n ]\n },\n }\n }\n return trainer\n\n def test_generate_prefers_rollout_func(self):\n trainer = self._make_trainer()\n trainer.rollout_func = MagicMock(\n return_value={\n \"prompt_ids\": [[1]],\n \"completion_ids\": [[2]],\n \"logprobs\": [[-0.1]],\n \"env_mask\": [[1]],\n }\n )\n\n result = trainer._generate([\"prompt\"])\n\n assert result[0] == [[1]] # prompt_ids\n assert result[1] == [[2]] # completion_ids\n assert result[2] == [[1]] # tool_mask (from env_mask)\n trainer.rollout_func.assert_called_once_with([\"prompt\"], trainer)\n\n def test_generate_rollout_func_syncs_vllm_weights_when_needed(self):\n trainer = self._make_trainer()\n trainer.use_vllm = True\n trainer.rollout_func = MagicMock(\n return_value={\"prompt_ids\": [[1]], \"completion_ids\": [[2]], \"logprobs\": [[0.0]]}\n )\n\n trainer._generate([\"prompt\"])\n\n trainer.vllm_generation.sync_weights.assert_called_once()\n assert trainer._last_loaded_step == trainer.state.global_step\n trainer.rollout_func.assert_called_once_with([\"prompt\"], trainer)\n\n def test_generate_rollout_func_raises_when_required_keys_are_missing(self):\n trainer = self._make_trainer()\n trainer.rollout_func = MagicMock(return_value={\"prompt_ids\": [[1]], \"completion_ids\": [[2]]})\n\n with pytest.raises(ValueError, match=\"rollout_func must return keys\"):\n trainer._generate([\"prompt\"])\n\n\nclass TestGRPOTrainer(TrlTestCase):\n def test_init_minimal(self):\n # Test that GRPOTrainer can be instantiated with only model, reward_model and train_dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n train_dataset=dataset,\n )\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n def test_training(self, config_name):\n dataset = load_dataset(\"trl-internal-testing/zen\", config_name, split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\"loss_type\", [\"bnpo\", \"dr_grpo\", \"dapo\", \"cispo\", \"sapo\", \"luspo\", \"vespo\"])\n def test_training_loss_types(self, loss_type):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n importance_sampling_level=\"sequence\" if loss_type == \"luspo\" else \"token\",\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=32, # reduce the completion length to reduce memory usage\n gradient_accumulation_steps=2, # set to 2 to test than DAPO can operate with accumulated batch\n loss_type=loss_type,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_eval(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n per_device_eval_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n eval_strategy=\"steps\",\n eval_steps=2,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n trainer.train()\n\n def test_training_with_num_generations_eval(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n per_device_eval_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n num_generations_eval=1,\n eval_strategy=\"steps\",\n eval_steps=2,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n trainer.train()\n\n # Regression test for eval_on_start with loss_type=\"grpo\" (one of the loss types that depends on\n # current_gradient_accumulation_steps): evaluation runs before the first training step, when that value is still\n # unset. Previously this caused the initial eval to crash.\n def test_training_eval_on_start(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n per_device_eval_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n loss_type=\"grpo\",\n eval_strategy=\"steps\",\n eval_steps=2,\n eval_on_start=True,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n trainer.train()\n\n def test_training_multiple_iterations(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n num_iterations=2,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_peft\n def test_training_peft_config(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n: # We expect the peft params to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_peft\n def test_training_peft_model(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n lora_config = LoraConfig()\n model = get_peft_model(model, lora_config)\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n and \"ref\" not in n: # and the peft params to be different (except base and ref)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n # In practice, this test is the same as `test_training_peft_config`, since gradient checkpointing is enabled by\n # default in `GRPOTrainer`. We keep it as a regression guard: if the default ever changes, we still explicitly test\n # PEFT + gradient checkpointing, which has caused issues in the past.\n @require_peft\n def test_training_peft_with_gradient_checkpointing(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n gradient_checkpointing=True, # enable gradient checkpointing\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n: # We expect the peft params to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_different_reward_model(self):\n # Use a reward model different from the model: different chat template, tokenization, etc.\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_only\", split=\"train\")\n reward_model_id = \"trl-internal-testing/tiny-LlamaForSequenceClassification-3.2\"\n reward_model = AutoModelForSequenceClassification.from_pretrained(reward_model_id)\n reward_tokenizer = AutoTokenizer.from_pretrained(reward_model_id)\n # By default, the trainer uses the eos token as the padding token. However, for Llama models, the eos token\n # appears in the chat template. Using it as a pad token disrupts the reward calculation, as the calculation\n # considers the score of the last token before the first pad token. To ensure correct reward calculations,\n # we use a separate pad token instead.\n reward_tokenizer.pad_token = \"<|finetune_right_pad_id|>\"\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_model,\n args=training_args,\n train_dataset=dataset,\n reward_processing_classes=reward_tokenizer,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_reward_func_standard(self):\n # Test if trainer can handle reward function with standard format\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_reward_func_conversational(self):\n # Test if trainer can handle reward function with conversational format\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that gives higher scores to longer completion content.\"\"\"\n completion_contents = [completion[0][\"content\"] for completion in completions]\n return [float(len(content)) for content in completion_contents]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_reward_funcs(self):\n # Test that GRPOTrainer can be instantiated with multiple reward functions\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func1(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n def reward_func2(completions, **kwargs):\n \"\"\"Reward function that rewards completions with more unique letters.\"\"\"\n return [float(len(set(completion))) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[reward_func1, reward_func2],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_sync_and_async_reward_funcs(self):\n # Test that GRPOTrainer can be instantiated with multiple reward functions one of which is async\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def sync_reward_func1(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n def sync_reward_func2(completions, **kwargs):\n return [1 for _ in completions]\n\n async def async_reward_func(completions, **kwargs):\n \"\"\"Async Reward function that rewards completions with more unique letters.\"\"\"\n return [float(len(set(completion))) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[sync_reward_func1, sync_reward_func2, async_reward_func],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_reward_funcs_with_None_output(self):\n \"\"\"Test that a valid math reward function is processed correctly while the code reward function returns None.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def applicable_reward_func(completions, **kwargs):\n \"\"\"A reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n def non_applicable_reward_func(completions, **kwargs):\n \"\"\"A reward function that returns None for all inputs, as it is not applicable to this sample.\"\"\"\n return [None] * len(completions)\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n )\n\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[\n applicable_reward_func,\n non_applicable_reward_func,\n ], # One applicable, one non applicable\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {\n n: param.clone() for n, param in trainer.model.named_parameters() if param.requires_grad\n }\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_reward_funcs_with_weights(self):\n \"\"\"Test that GRPOTrainer can handle multiple reward functions with weights.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func1(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n def reward_func2(completions, **kwargs):\n \"\"\"Reward function that rewards completions with more unique letters.\"\"\"\n return [float(len(set(completion))) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n reward_weights=[0.7, 0.3], # weight of reward_func1 and reward_func2 respectively\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[reward_func1, reward_func2],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n # Check that training logs contain both reward metrics\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert \"rewards/reward_func1/mean\" in trainer.state.log_history[-1]\n assert \"rewards/reward_func1/std\" in trainer.state.log_history[-1]\n assert \"rewards/reward_func2/mean\" in trainer.state.log_history[-1]\n assert \"rewards/reward_func2/std\" in trainer.state.log_history[-1]\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_mixed_reward_funcs(self):\n # Test if the trainer can handle a mix of reward functions and reward models\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[reward_func, \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_reward_func_additional_column(self):\n # Test if trainer can handle reward function that rely on additional columns in the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Add a column to the dataset (dummy example, the column could be anything)\n some_values = list(range(len(dataset)))\n dataset = dataset.add_column(\"some_values\", some_values)\n\n def reward_func(completions, some_values, **kwargs):\n \"\"\"Reward function that rewards completions with lengths closer to the values in some_values.\"\"\"\n return [\n float(abs(len(completion) - value)) for completion, value in zip(completions, some_values, strict=True)\n ]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_sync_ref_model(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n beta=0.1, # ensure ref model is created so sync can update it\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n sync_ref_model=True,\n ref_model_sync_steps=2, # reduce sync steps to ensure a sync happens\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n assert trainer.ref_model is not None\n previous_ref_params = {n: param.clone() for n, param in trainer.ref_model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n new_ref_param = trainer.ref_model.get_parameter(n)\n assert not torch.equal(previous_ref_params[n], new_ref_param), f\"Ref Parameter {n} has not changed.\"\n\n def test_training_beta_non_zero(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n beta=0.1, # set beta to non-zero value to test the case where the reference model is used\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_pad_to_multiple_of(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n pad_to_multiple_of=8,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_get_off_policy_mask(self):\n \"\"\"\n Test the logic of off-policy masking:\n - Keep if Advantage >= 0\n - Keep if KL <= threshold\n - Drop if Advantage < 0 AND KL > threshold\n \"\"\"\n mask = torch.ones((3, 4)) # B=3 sequences, T=4 tokens\n\n advantages = torch.tensor([1.0, -1.0, -1.0]).unsqueeze(-1)\n sampling_per_token_logps = torch.zeros((3, 4))\n per_token_logps = torch.zeros((3, 4))\n\n per_token_logps[0, :] = -2.0 # Pos adv + High KL (0โˆ’(โˆ’2)=2) -> Keep\n per_token_logps[1, :] = -0.5 # Neg adv + Low KL (0.5) -> Keep\n per_token_logps[2, :] = -2.0 # Neg adv + High KL (2.0) -> Drop\n\n off_policy_threshold = 1.0\n\n expected_mask = torch.tensor([[1.0], [1.0], [0.0]])\n\n off_policy_mask = GRPOTrainer.get_off_policy_mask(\n advantages, per_token_logps, sampling_per_token_logps, mask, off_policy_threshold\n )\n\n torch.testing.assert_close(off_policy_mask, expected_mask)\n\n def test_get_off_policy_mask_padding(self):\n \"\"\"Test that padding is correctly ignored in KL calculation.\"\"\"\n mask = torch.tensor([[1.0, 1.0, 0.0, 0.0]]) # 2 valid tokens\n advantages = torch.tensor([[-1.0]]) # Negative advantage\n\n sampling_per_token_logps = torch.zeros((1, 4))\n per_token_logps = torch.zeros((1, 4))\n\n # Valid tokens have High KL (2.0)\n per_token_logps[0, 0] = -2.0\n per_token_logps[0, 1] = -2.0\n\n # Padding tokens have abnormal values (should be ignored)\n per_token_logps[0, 2] = -10_000.0\n per_token_logps[0, 3] = 10_000.0\n\n off_policy_threshold = 1.0\n\n # Avg KL on valid tokens = (2+2)/2 = 2.0 > 1.0 -> Drop\n expected_mask = torch.tensor([[0.0]])\n\n off_policy_mask = GRPOTrainer.get_off_policy_mask(\n advantages, per_token_logps, sampling_per_token_logps, mask, off_policy_threshold\n )\n\n torch.testing.assert_close(off_policy_mask, expected_mask)\n\n # Now test with Low KL on valid tokens\n per_token_logps[0, 0] = -0.5\n per_token_logps[0, 1] = -0.5\n # Avg KL = 0.5 <= 1.0 -> Keep\n expected_mask_keep = torch.tensor([[1.0]])\n\n off_policy_mask_keep = GRPOTrainer.get_off_policy_mask(\n advantages, per_token_logps, sampling_per_token_logps, mask, off_policy_threshold\n )\n\n torch.testing.assert_close(off_policy_mask_keep, expected_mask_keep)\n\n def test_training_with_off_policy_mask(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n off_policy_mask_threshold=0.5,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_liger_kernel\n @pytest.mark.xfail(reason=\"Off-Policy Masking isn't compatible with Liger yet.\")\n def test_training_with_off_policy_mask_with_liger(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n off_policy_mask_threshold=0.5,\n use_liger_kernel=True,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_liger_kernel\n def test_compute_liger_loss_passes_vllm_is_ratio(self):\n \"\"\"Test that importance_sampling_ratio from inputs is passed to liger_grpo_loss as vllm_is_ratio.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1,\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n use_liger_kernel=True,\n report_to=\"none\",\n logging_strategy=\"no\",\n )\n\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Mock _generate_and_score_completions to inject importance_sampling_ratio\n original_gen = trainer._generate_and_score_completions\n\n def gen_with_is_ratio(*args, **kwargs):\n result = original_gen(*args, **kwargs)\n B, T = result[\"completion_ids\"].shape\n result[\"importance_sampling_ratio\"] = torch.full((B, T), 0.5, device=result[\"completion_ids\"].device)\n return result\n\n with (\n patch.object(trainer, \"_generate_and_score_completions\", side_effect=gen_with_is_ratio),\n patch.object(trainer.liger_grpo_loss, \"forward\", wraps=trainer.liger_grpo_loss.forward) as mock_forward,\n ):\n trainer.train()\n\n # Verify vllm_is_ratio was passed in every call to liger_grpo_loss\n assert mock_forward.call_count > 0, \"liger_grpo_loss.forward was never called\"\n for call in mock_forward.call_args_list:\n vllm_is_ratio = call.kwargs.get(\"vllm_is_ratio\")\n assert vllm_is_ratio is not None, (\n \"vllm_is_ratio should not be None when importance_sampling_ratio is present\"\n )\n assert (vllm_is_ratio == 0.5).all(), (\n \"vllm_is_ratio values should match the injected importance_sampling_ratio\"\n )\n\n release_memory(trainer.model, trainer)\n\n def test_training_with_bias_correction_kl(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n beta=0.1, # set beta to non-zero value to test the case where the reference model is used\n use_bias_correction_kl=True,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_name\",\n [\"trl-internal-testing/tiny-Qwen3ForCausalLM\", \"trl-internal-testing/tiny-Gemma2ForCausalLM\"],\n # Gemma2 has the input word embeddings and lm_head tied, Qwen3 does not\n )\n def test_training_with_cast_lm_head_to_fp32(self, model_name):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n cast_lm_head_to_fp32=True,\n )\n trainer = GRPOTrainer(\n model=model_name,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert trainer.model.lm_head.weight.dtype == torch.float32\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_entropy_filter(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n top_entropy_quantile=0.2,\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_peft\n @require_vllm\n @pytest.mark.skip(reason=\"We should add a mock for the vLLM server.\")\n def test_training_vllm_and_peft(self):\n \"\"\"Test that training works with vLLM for generation.\"\"\"\n model = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen2.5-0.5B-Instruct\", dtype=\"float32\"\n ) # tiny model is too small for vLLM\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n use_vllm=True,\n )\n lora_config = LoraConfig(\n target_modules=\"all-linear\",\n # test with non-default modules as it adds extra keys in state_dict that we need to handle\n modules_to_save=[\"embed_tokens\", \"lm_head\"],\n )\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n peft_config=lora_config,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n and \"original_module\" not in n:\n # We expect the peft params to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_vllm\n @pytest.mark.skip(reason=\"We should add a mock for the vLLM server.\")\n def test_training_vllm_structured_outputs(self):\n \"\"\"Test that training works with vLLM for generation with structured outputs.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n use_vllm=True,\n vllm_structured_outputs_regex=r\"\\n.*\\n\\n\\n.*\\n\",\n )\n trainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\", # tiny model is too small for vLLM\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_vllm\n @pytest.mark.skip(reason=\"We should add a mock for the vLLM server.\")\n def test_training_vllm_importance_sampling_correction(self):\n \"\"\"Test that training works with vLLM for generation with structured outputs.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n use_vllm=True,\n vllm_importance_sampling_correction=True,\n vllm_importance_sampling_cap=3.0,\n )\n trainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\", # tiny model is too small for vLLM\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_additional_generation_kwargs(self):\n \"\"\"Test that training works with additional generation kwargs.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n top_p=0.9,\n top_k=10,\n min_p=0.01,\n repetition_penalty=1.1,\n )\n\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_vllm\n @pytest.mark.skip(reason=\"We should add a mock for the vLLM server.\")\n def test_training_vllm_with_additional_generation_kwargs(self):\n \"\"\"Test that training works with vLLM and additional generation kwargs.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n use_vllm=True,\n top_p=0.9,\n top_k=10,\n min_p=0.01,\n repetition_penalty=1.1,\n )\n\n trainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\", # tiny model is too small for vLLM\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_normalize_then_sum_aggregation(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func1(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n def reward_func2(completions, **kwargs):\n \"\"\"Reward function that rewards completions with more unique letters.\"\"\"\n return [float(len(set(completion))) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n multi_objective_aggregation=\"normalize_then_sum\",\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[reward_func1, reward_func2],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\"scale_rewards\", [False, \"group\", \"batch\", True, \"none\"])\n def test_training_scale_rewards(self, scale_rewards):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n scale_rewards=scale_rewards,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @patch(\"transformers.generation.utils.GenerationMixin.generate\")\n def test_training_with_mask_truncated_completions(self, mock_generate):\n \"\"\"Test that training works with mask_truncated_completions=True parameter.\"\"\"\n\n # We mock the generate method because the model's random weights make it extremely unlikely to produce a\n # sequence containing the EOS token within the allowed max_completion_length. As a result, all tokens are\n # masked in the loss, the model doesn't update, and the final check (which verifies the update) fails.\n def fake_generate(input_ids, **kwargs):\n # pad_token_id = 151643; eos_token_id = 151645\n completion_ids = torch.tensor(\n [\n [1, 2, 3, 4, 5, 6, 7, 8], # this one is truncated\n [9, 10, 11, 151645, 151643, 151643, 151643, 151643], # this one contains eos\n [12, 13, 14, 15, 16, 17, 18, 151645], # particular case, eos is generated just within the limit\n ],\n device=input_ids.device,\n )\n return torch.cat([input_ids, completion_ids], dim=1)\n\n mock_generate.side_effect = fake_generate\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n mask_truncated_completions=True, # Enable masking of truncated completions\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_mask_truncated_completions_all_masked(self):\n \"\"\"\n Test that when all generated completions are truncated (i.e., none contain an EOS token), and\n mask_truncated_completions=True, the model receives no effective learning signal and therefore does not update\n its parameters.\n\n Here, we don't mock the generate method, be we rely on the fact that the model the probability of generating\n the EOS token is extremely low, so all generated completions are truncated.\n \"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n mask_truncated_completions=True, # Enable masking of truncated completions\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert torch.equal(param, new_param), f\"Parameter {n} has changed.\"\n\n def test_warning_raised_all_rewards_none(self, caplog):\n \"\"\"Test that a proper warning is raised when all rewards are None.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def always_none_reward_func(completions, **kwargs):\n \"\"\"Reward function that always returns None.\"\"\"\n return [None] * len(completions)\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=always_none_reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n with caplog.at_level(\"WARNING\", logger=\"trl.trainer.grpo_trainer\"):\n trainer.train()\n\n expected_warning = \"All reward functions returned None for the following kwargs:\"\n assert expected_warning in caplog.text\n\n def test_training_num_generations_larger_than_batch_size(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n num_generations=6, # the number of generations is larger than the batch size, but\n gradient_accumulation_steps=2, # gradient accumulation should allow that\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_delta_clipping(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n delta=2.0, # set delta to a non-None value\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_dataloader_workers(self):\n # Pytest/CI often starts background threads before tests run. With Python 3.12, using the default \"fork\" start\n # method in a multi-threaded process emits a DeprecationWarning and may deadlock.\n #\n # We force \"spawn\" here to make multiprocessing safe under pytest when DataLoader workers are enabled. This is\n # test-environmentโ€“specific and not required by the training logic itself.\n #\n # This means the test does not cover \"fork\". However, \"spawn\" is stricter (requires full picklability and clean\n # state) and avoids fork-after-threads issues that pytest cannot reliably test anyway. Fork-specific behavior,\n # if needed, should be tested in a clean process outside pytest.\n torch.multiprocessing.set_start_method(\"spawn\", force=True)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n dataloader_num_workers=2, # use multiple dataloader workers\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_generation_kwargs(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n # Pass gen kwargs\n generation_kwargs={\"do_sample\": True, \"top_k\": 50, \"num_beams\": 2, \"length_penalty\": -0.1},\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_reward_func_accessing_trainer_state(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n trainer_state = kwargs.get(\"trainer_state\")\n assert trainer_state is not None\n # transformers.TrainerState instance should have a `global_step` property.\n assert hasattr(trainer_state, \"global_step\")\n return [float(len(set(completion))) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n trainer.train()\n\n def test_training_reward_func_with_log_extra(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n log_extra = kwargs.get(\"log_extra\")\n assert log_extra is not None\n log_extra(\"test_column\", [completion[:5] for completion in completions])\n return [float(len(completion)) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n log_completions=True,\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n trainer.train()\n assert \"test_column\" in trainer._logs[\"extra\"]\n\n def test_training_reward_func_with_log_metric(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n log_metric = kwargs.get(\"log_metric\")\n assert log_metric is not None\n log_metric(\"custom_accuracy\", 0.75)\n return [float(len(completion)) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n trainer.train()\n # log_metric appends to _metrics, which gets averaged and merged into log_history\n logged_keys = {k for entry in trainer.state.log_history for k in entry}\n assert \"custom_accuracy\" in logged_keys\n\n def test_prepare_input_called_with_correct_data(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n max_completion_length=8, # reduce the completion length to reduce memory usage\n gradient_accumulation_steps=3, # can be anything in this test\n # steps_per_generation*per_device_train_batch_size=24 is divisible by num_generations=4\n steps_per_generation=4,\n num_generations=4,\n per_device_train_batch_size=6, # reduce the batch size to reduce memory usage\n num_iterations=2,\n shuffle_dataset=False,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n # steps_per_generation=4, per_device_train_batch_size=6 and num_generations=4, so we expect a\n # generation batch of 24 samples (steps_per_generation * per_device_train_batch_size), containing 6\n # different prompts (steps_per_generation * per_device_train_batch_size // num_generations), each repeated\n # 4 times (num_generations).\n expected_first_generation_batch = (\n [{\"prompt\": \"Beautiful is better than\"}] * 4\n + [{\"prompt\": \"Explicit is\"}] * 4\n + [{\"prompt\": \"Simple is better\"}] * 4\n + [{\"prompt\": \"Complex\"}] * 4\n + [{\"prompt\": \"Flat is better than\"}] * 4\n + [{\"prompt\": \"Sparse is better\"}] * 4\n )\n expected_second_generation_batch = (\n [{\"prompt\": \"Readability\"}] * 4\n + [{\"prompt\": \"Special cases aren't special\"}] * 4\n + [{\"prompt\": \"Although practicality beats\"}] * 4\n + [{\"prompt\": \"Errors should never\"}] * 4\n + [{\"prompt\": \"Unless explicitly\"}] * 4\n + [{\"prompt\": \"In the face of ambiguity, refuse\"}] * 4\n )\n\n with patch.object(GRPOTrainer, \"training_step\", wraps=trainer.training_step) as mock_prepare:\n trainer.train()\n # 3 epochs * 2 iterations * 2 generation batches to cover the dataset * 4 steps_per_generation\n assert mock_prepare.call_count == 48\n for i in range(0, 8): # Generation batch repeated 8 times (steps_per_generation*num_iterations)\n assert mock_prepare.call_args_list[i].args[1] == expected_first_generation_batch\n for i in range(8, 16):\n assert mock_prepare.call_args_list[i].args[1] == expected_second_generation_batch\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n \"trl-internal-testing/tiny-LlavaNextForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2VLForConditionalGeneration\",\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration\",\n marks=pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.2.0\"),\n reason=\"Qwen3.5 models were introduced in transformers-5.2.0\",\n ),\n ),\n # \"trl-internal-testing/tiny-SmolVLMForConditionalGeneration\", seems not to support bf16 properly\n ],\n )\n @require_vision\n def test_training_vlm(self, model_id):\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n # Because of the way the tiny models are initialized, the gradient does not flow properly through the\n # vision parts of the model, so we skip them. Ideally, we should fix the init of these models.\n params_to_skip = (\n \"model.vision_tower.\",\n \"model.multi_modal_projector.\",\n \"model.vision_model.\",\n \"model.visual.\",\n \"model.image_newline\",\n )\n for n, param in previous_trainable_params.items():\n if n.startswith(params_to_skip):\n continue\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_vision\n def test_training_vlm_with_pad_to_multiple_of(self):\n # Models like Gemma3 use other forward keyword arguments like token_type_ids that also need to be padded when\n # using pad_to_multiple_of, so we test that the trainer correctly pads all the necessary inputs in this case.\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n pad_to_multiple_of=7,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @require_vision\n def test_training_vlm_beta_non_zero(self, model_id):\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n beta=0.1, # set beta to non-zero value to test the case where the reference model is used\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n # Because of the way the tiny models are initialized, the gradient does not flow properly through the\n # vision parts of the model, so we skip them. Ideally, we should fix the init of these models.\n params_to_skip = (\"model.visual.\",)\n for n, param in previous_trainable_params.items():\n if n.startswith(params_to_skip):\n continue\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @require_vision\n @require_peft\n def test_training_vlm_peft(self, model_id):\n model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(target_modules=[\"q_proj\", \"v_proj\"]),\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n: # We expect the peft params to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @require_vision\n def test_training_vlm_and_importance_sampling(self, model_id):\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n steps_per_generation=2, # increase the steps per generation to trigger IS\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n # Because of the way the tiny models are initialized, the gradient does not flow properly through the\n # vision parts of the model, so we skip them. Ideally, we should fix the init of these models.\n params_to_skip = (\"model.visual.\",)\n for n, param in previous_trainable_params.items():\n if n.startswith(params_to_skip):\n continue\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n pytest.param(\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n marks=pytest.mark.xfail(\n (Version(\"5.2.0\") < Version(transformers.__version__))\n and not is_liger_kernel_available(min_version=\"0.8.0\"),\n reason=\"Upstream issue tracked at https://github.com/linkedin/Liger-Kernel/issues/1117\",\n ),\n ),\n ],\n )\n @require_vision\n @require_liger_kernel\n def test_training_vlm_and_liger(self, model_id):\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n use_liger_kernel=True, # enable Liger kernel\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n # Because of the way the tiny models are initialized, the gradient does not flow properly through the\n # vision parts of the model, so we skip them. Ideally, we should fix the init of these models.\n params_to_skip = (\"model.visual.\",)\n for n, param in previous_trainable_params.items():\n if n.startswith(params_to_skip):\n continue\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n ],\n )\n @require_vision\n @require_vllm\n @pytest.mark.skip(reason=\"We should add a mock for the vLLM server.\")\n def test_training_vlm_and_vllm(self, model_id) -> None:\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n use_vllm=True,\n vllm_mode=\"server\",\n )\n trainer = GRPOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @require_vision\n def test_training_vlm_multi_image(self, model_id):\n dataset = load_dataset(\"trl-internal-testing/zen-multi-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n # Because of the way the tiny models are initialized, the gradient does not flow properly through the\n # vision parts of the model, so we skip them. Ideally, we should fix the init of these models.\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_sequence_importance_sampling(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n num_iterations=2, # the importance sampling weights won't be 0 in this case\n importance_sampling_level=\"sequence\",\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_chat_template_kwargs(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n chat_template_kwargs={\"enable_thinking\": False},\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen3ForCausalLM\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.xfail(\n condition=Version(transformers.__version__) < Version(\"5.0.0\"),\n reason=\"Tool parsing is not supported in transformers versions below 5.0.0\",\n strict=True,\n )\n @require_jmespath\n @pytest.mark.parametrize(\"tools\", [[multiply_tool], [async_multiply_tool]])\n def test_training_with_tools(self, tools: list[Callable]):\n # In this test, we define a simple tool that multiplies two integers. Regardless of the input prompt,\n # the model will generate 3 completions, 2 of which will be valid tool calls. Among the 2 tool calls, one will\n # succeed and the other will fail (because of a wrong argument name).\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=128,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen3MoeForCausalLM\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n tools=tools,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n tool_name = tools[0].__name__\n\n def fake_generate(input_ids, **kwargs):\n if input_ids.shape[0] == 3: # first call\n # fmt: off\n if tool_name == \"multiply_tool\":\n completion_ids = torch.tensor(\n [\n # '\\n{\"name\": \"multiply_tool\", \"arguments\": {\"a\": 3, \"b\": 4}}\\n<|im_end|>'\n [151657, 198, 4913, 606, 788, 330, 64648, 22785, 497, 330, 16370, 788, 5212, 64, 788, 220, 18, 11, 330, 65, 788, 220, 19, 11248, 151658, 151645],\n # an invalid tool call with wrong argument name\n # '\\n{\"name\": \"multiply_tool\", \"arguments\": {\"a\": 3, \"c\": 4}}\\n<|im_end|>'\n [151657, 198, 4913, 606, 788, 330, 64648, 22785, 497, 330, 16370, 788, 5212, 64, 788, 220, 18, 11, 330, 66, 788, 220, 19, 11248, 151658, 151645],\n # \"I don't know any tool<|im_end|>\"\n [40, 1513, 944, 1414, 894, 5392, 151645, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643],\n ],\n device=input_ids.device,\n )\n elif tool_name == \"async_multiply_tool\":\n completion_ids = torch.tensor(\n [\n # '\\n{\"name\": \"async_multiply_tool\", \"arguments\": {\"a\": 3, \"b\": 4}}\\n<|im_end|>'\n [151657, 198, 4913, 606, 788, 330, 7692, 93054, 22785, 497, 330, 16370, 788, 5212, 64, 788, 220, 18, 11, 330, 65, 788, 220, 19, 11248, 151658, 151645],\n # an invalid tool call with wrong argument name\n # '\\n{\"name\": \"async_multiply_tool\", \"arguments\": {\"a\": 3, \"c\": 4}}\\n<|im_end|>'\n [151657, 198, 4913, 606, 788, 330, 7692, 93054, 22785, 497, 330, 16370, 788, 5212, 64, 788, 220, 18, 11, 330, 66, 788, 220, 19, 11248, 151658, 151645],\n # \"I don't know any tool<|im_end|>\"\n [40, 1513, 944, 1414, 894, 5392, 151645, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643],\n ],\n device=input_ids.device,\n )\n # fmt: on\n else: # second call will only have two inputs in the batch, because two examples have a tool call.\n completion_ids = torch.tensor(\n [\n # 'Done!<|im_end|>'\n [17453, 0, 151645],\n # 'Done!<|im_end|>'\n [17453, 0, 151645],\n ],\n device=input_ids.device,\n )\n return torch.cat([input_ids, completion_ids], dim=-1)\n\n with patch.object(trainer.model, \"generate\", side_effect=fake_generate):\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert trainer.state.log_history[-1][\"tools/call_frequency\"] is not None\n assert trainer.state.log_history[-1][\"tools/call_frequency\"] == pytest.approx(2 / 3)\n assert trainer.state.log_history[-1][\"tools/failure_frequency\"] is not None\n assert trainer.state.log_history[-1][\"tools/failure_frequency\"] == pytest.approx(1 / 2)\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.xfail(\n condition=Version(transformers.__version__) < Version(\"5.2.0\"),\n reason=\"Environment factory support is not available in transformers versions below 5.2.0\",\n strict=True,\n )\n @require_jmespath\n @patch.dict(os.environ, {\"TRL_EXPERIMENTAL_SILENCE\": \"1\"})\n def test_training_with_environment_factory(self):\n # In this test, we define a simple tool that increments an internal counter. Regardless of the input prompt,\n # the model will generate 3 completions, 2 of which will be valid tool calls. Among the 2 tool calls, one will\n # succeed and the other will fail (because of a wrong tool name).\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_only\", split=\"train\")\n\n class DummyEnvironment:\n def reset(self, **kwargs):\n self._counter = 0\n\n def increment(self, step: int) -> int:\n \"\"\"\n Increment the internal counter.\n\n Args:\n step: Value to add to the counter.\n\n Returns:\n The updated counter value.\n \"\"\"\n self._counter += step\n return self._counter\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n report_to=\"none\",\n )\n\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen3MoeForCausalLM\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n environment_factory=DummyEnvironment,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n def fake_generate(input_ids, **kwargs):\n if input_ids.shape[0] == 3: # first call\n # fmt: off\n completion_ids = torch.tensor(\n [\n # '\\n{\"name\": \"increment\", \"arguments\": {\"step\": 1}}\\n<|im_end|>'\n [151657, 198, 4913, 606, 788, 330, 35744, 497, 330, 16370, 788, 5212, 9520, 788, 220, 16, 11248, 151658, 151645, 151643],\n # an invalid tool call with wrong tool name\n # '\\n{\"name\": \"decrement\", \"arguments\": {\"step\": 2}}\\n<|im_end|>'\n [151657, 198, 4913, 606, 788, 330, 450, 13477, 497, 330, 16370, 788, 5212, 9520, 788, 220, 17, 11248, 151658, 151645],\n # \"I won't increment<|im_end|>\"\n [40, 2765, 944, 16252, 151645, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643, 151643],\n ],\n device=input_ids.device,\n )\n # fmt: on\n else: # second call will only have two inputs in the batch, because two examples have a tool call.\n completion_ids = torch.tensor(\n [\n # 'Done!<|im_end|>'\n [17453, 0, 151645],\n # 'Done!<|im_end|>'\n [17453, 0, 151645],\n ],\n device=input_ids.device,\n )\n return torch.cat([input_ids, completion_ids], dim=-1)\n\n with patch.object(trainer.model, \"generate\", side_effect=fake_generate):\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert trainer.state.log_history[-1][\"tools/call_frequency\"] is not None\n assert trainer.state.log_history[-1][\"tools/call_frequency\"] == pytest.approx(2 / 3)\n assert trainer.state.log_history[-1][\"tools/failure_frequency\"] is not None\n assert trainer.state.log_history[-1][\"tools/failure_frequency\"] == pytest.approx(1 / 2)\n\n # Check the states of the environment\n assert trainer.environments[0]._counter == 1 # should have been incremented once\n assert trainer.environments[1]._counter == 0 # shouldn't have been incremented because the tool call failed\n assert trainer.environments[2]._counter == 0 # shouldn't have been incremented because no tool call was made\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.xfail(\n condition=Version(transformers.__version__) < Version(\"5.0.0\"),\n reason=\"Tool parsing is not supported in transformers versions below 5.0.0\",\n strict=True,\n )\n @require_jmespath\n def test_training_with_malformed_tool_calls(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=128,\n report_to=\"none\",\n )\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen3MoeForCausalLM\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n tools=[multiply_tool],\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n def fake_generate(input_ids, **kwargs):\n # If input_ids.shape[0] < 3, it means that it's a second call, which should not happen here\n assert input_ids.shape[0] == 3\n # fmt: off\n completion_ids = torch.tensor(\n [\n # '\\n{\"arguments\": {\"a\": 3, \"b\": 4}}\\n<|im_end|>'\n [151657, 198, 4913, 16370, 788, 5212, 64, 788, 220, 18, 11, 330, 65, 788, 220, 19, 11248, 151658, 151645, 151643, 151643, 151643, 151643, 151643],\n # '\\n{\"arguments\": {\"a\": 3, \"b\": 4}}\\n<|im_end|>'\n [27, 14172, 6659, 397, 4913, 16370, 788, 5212, 64, 788, 220, 18, 11, 330, 65, 788, 220, 19, 11248, 522, 14172, 6659, 29, 151645],\n # '\\n{\"arguments\": {a: 3, b: 4}}\\n<|im_end|>'\n [151657, 198, 4913, 16370, 788, 314, 64, 25, 220, 18, 11, 293, 25, 220, 19, 11248, 151658, 151645, 151643, 151643, 151643, 151643, 151643, 151643],\n ],\n device=input_ids.device,\n )\n # fmt: on\n return torch.cat([input_ids, completion_ids], dim=-1)\n\n with patch.object(trainer.model, \"generate\", side_effect=fake_generate):\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_mismatched_reward_processing_classes_length(self):\n \"\"\"Test that mismatched length between reward_funcs and reward_processing_classes raises error.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Use two reward models\n reward_models = [\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n \"trl-internal-testing/tiny-Qwen3ForSequenceClassification\",\n ]\n\n # Create a single processing class (tokenizer)\n single_processing_class = AutoTokenizer.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n )\n\n training_args = GRPOConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n with pytest.raises(ValueError, match=\"must match\"):\n GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_models,\n reward_processing_classes=single_processing_class, # only one, but need two\n args=training_args,\n train_dataset=dataset,\n )\n\n def test_correct_reward_processing_classes_list(self):\n \"\"\"Test that correct list of reward_processing_classes works properly.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Use two reward models\n reward_models = [\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n \"trl-internal-testing/tiny-Qwen3ForSequenceClassification\",\n ]\n\n # Create processing classes\n processing_class1 = AutoTokenizer.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n )\n processing_class2 = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen3ForSequenceClassification\")\n\n training_args = GRPOConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n # Correct list length should work\n correct_processing_classes = [processing_class1, processing_class2]\n\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_models,\n reward_processing_classes=correct_processing_classes,\n args=training_args,\n train_dataset=dataset,\n )\n\n assert len(trainer.reward_processing_classes) == len(reward_models)\n\n def test_single_reward_model_with_single_processing_class(self):\n \"\"\"Test that single reward model with single processing class works.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Use single reward model\n reward_model = \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n\n # Create a single processing class (tokenizer)\n single_processing_class = AutoTokenizer.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n )\n\n training_args = GRPOConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n trainer = GRPOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_model,\n reward_processing_classes=single_processing_class, # single object for single reward model\n args=training_args,\n train_dataset=dataset,\n )\n\n assert len(trainer.reward_processing_classes) == 1\n assert trainer.reward_processing_classes[0] == single_processing_class\n\n\n@pytest.mark.slow\n@require_torch_accelerator\nclass TestGRPOTrainerSlow(TrlTestCase):\n def setup_method(self):\n self.train_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n self.eval_dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"test\")\n self.max_length = 128\n\n def teardown_method(self):\n gc.collect()\n backend_empty_cache(torch_device)\n gc.collect()\n\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n ],\n )\n @require_liger_kernel\n def test_training_with_liger_grpo_kernel(self, model_name):\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3,\n num_generations=3,\n use_liger_kernel=True,\n max_completion_length=self.max_length,\n report_to=\"none\",\n logging_strategy=\"no\",\n )\n\n model = AutoModelForCausalLM.from_pretrained(model_name, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n tokenizer.pad_token = tokenizer.eos_token if tokenizer.pad_token is None else tokenizer.pad_token\n\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=self.train_dataset,\n eval_dataset=self.eval_dataset,\n processing_class=tokenizer,\n )\n from liger_kernel.chunked_loss import LigerFusedLinearGRPOLoss\n\n assert isinstance(trainer.liger_grpo_loss, LigerFusedLinearGRPOLoss)\n\n previous_trainable_params = {n: param.clone() for n, param in model.named_parameters()}\n\n trainer.train()\n\n for n, param in previous_trainable_params.items():\n new_param = model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n release_memory(model, trainer)\n\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n ],\n )\n @require_liger_kernel\n @require_peft\n def test_training_with_liger_grpo_kernel_and_peft(self, model_name):\n from peft import LoraConfig, TaskType\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3,\n num_generations=3,\n use_liger_kernel=True,\n max_completion_length=self.max_length,\n report_to=\"none\",\n logging_strategy=\"no\",\n )\n\n model = AutoModelForCausalLM.from_pretrained(model_name, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n tokenizer.pad_token = tokenizer.eos_token if tokenizer.pad_token is None else tokenizer.pad_token\n\n # Configure PEFT with LoRA\n peft_config = LoraConfig(\n task_type=TaskType.CAUSAL_LM,\n inference_mode=False,\n r=8,\n lora_alpha=32,\n lora_dropout=0.1,\n target_modules=[\"q_proj\", \"v_proj\"],\n )\n\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=self.train_dataset,\n eval_dataset=self.eval_dataset,\n processing_class=tokenizer,\n peft_config=peft_config,\n )\n from liger_kernel.chunked_loss import LigerFusedLinearGRPOLoss\n\n assert isinstance(trainer.liger_grpo_loss, LigerFusedLinearGRPOLoss)\n\n # Verify PEFT adapter is properly initialized\n from peft import PeftModel\n\n assert isinstance(trainer.model, PeftModel), \"Model should be wrapped with PEFT\"\n\n # Store adapter weights before training\n previous_trainable_params = {\n n: param.clone() for n, param in trainer.model.named_parameters() if param.requires_grad\n }\n assert len(previous_trainable_params) > 0, \"No trainable parameters found in PEFT model\"\n\n trainer.train()\n\n # Verify adapter weights have changed after training\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n release_memory(model, trainer)\n\n @require_liger_kernel\n def test_liger_grpo_kernel_importance_sampling(self):\n model_name = \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\"\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3,\n num_generations=3,\n use_liger_kernel=True,\n max_completion_length=self.max_length,\n importance_sampling_level=\"sequence\",\n report_to=\"none\",\n logging_strategy=\"no\",\n )\n\n model = AutoModelForCausalLM.from_pretrained(model_name, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n tokenizer.pad_token = tokenizer.eos_token if tokenizer.pad_token is None else tokenizer.pad_token\n\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=self.train_dataset,\n eval_dataset=self.eval_dataset,\n processing_class=tokenizer,\n )\n from liger_kernel.chunked_loss import LigerFusedLinearGRPOLoss\n\n assert isinstance(trainer.liger_grpo_loss, LigerFusedLinearGRPOLoss)\n\n previous_trainable_params = {n: param.clone() for n, param in model.named_parameters()}\n\n trainer.train()\n\n for n, param in previous_trainable_params.items():\n new_param = model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n release_memory(model, trainer)\n\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n ],\n )\n def test_training_with_transformers_paged(self, model_name):\n \"\"\"Test that training works with transformers paged implementation (requires GPU).\"\"\"\n if Version(transformers.__version__) < Version(\"4.57.0\"):\n pytest.xfail(\"Bug in transformers solved in GH#40692, released in 4.57.0.\")\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n use_transformers_paged=True, # Enable transformers paged implementation\n report_to=\"none\",\n logging_strategy=\"no\",\n )\n\n model = AutoModelForCausalLM.from_pretrained(model_name, dtype=\"float32\")\n\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=self.train_dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n release_memory(model, trainer)\n\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"HuggingFaceTB/SmolVLM-Instruct\", # Only test the smaller model to avoid OOM\n ],\n )\n @require_kernels\n @require_ampere_or_newer # Flash attention 2 requires Ampere or newer GPUs\n @require_bitsandbytes\n @require_peft\n def test_vlm_training(self, model_name):\n \"\"\"\n Test VLM training with aggressive memory optimization.\n\n This test uses multiple memory reduction techniques:\n - 4-bit quantization with double quantization\n - LoRA with very low rank (r=4)\n - Minimal batch size (1) with gradient accumulation\n - Small images (64x64 instead of 224x224)\n - Short sequences (max_completion_length=8)\n - Only 4 training samples\n - Only 1 training step\n - Gradient checkpointing and bfloat16\n \"\"\"\n\n # Create processor once outside the data generator\n processor = AutoProcessor.from_pretrained(model_name, use_fast=True, padding_side=\"left\")\n conversation = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\"},\n {\"type\": \"text\", \"text\": \"What is in the image?\"},\n ],\n },\n ]\n prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n def data_gen(num_samples):\n for _ in range(num_samples):\n yield {\n \"prompt\": prompt,\n \"image\": np.random.uniform(low=0.0, high=255.0, size=(64, 64, 3)).astype(\n np.uint8\n ), # Much smaller images\n }\n\n dataset = Dataset.from_generator(\n data_gen, gen_kwargs={\"num_samples\": 4}, features=Features(image=Image(), prompt=Value(dtype=\"string\"))\n )\n # reduce memory requirements as much as possible\n quantization_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_compute_dtype=\"bfloat16\",\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_storage=\"bfloat16\",\n )\n model = AutoModelForImageTextToText.from_pretrained(\n model_name,\n attn_implementation=\"kernels-community/flash-attn2\",\n dtype=\"float32\",\n device_map=get_kbit_device_map(),\n quantization_config=quantization_config,\n )\n\n def reward_func(prompts, completions, **kwargs):\n # simple nonsensical reward\n return [-((len(c) - 25) ** 2) + 100 for c in completions]\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=1, # Minimal batch size\n gradient_accumulation_steps=2, # Maintain effective batch size\n num_generations=2,\n max_completion_length=8, # Much shorter completions\n bf16=True, # Use bfloat16 precision\n max_steps=1, # Only do 1 training step to save time and memory\n report_to=\"none\",\n logging_strategy=\"no\",\n )\n lora_config = LoraConfig(\n task_type=\"CAUSAL_LM\",\n r=4, # Much lower rank for minimal memory\n lora_alpha=8, # Reduced alpha proportionally\n lora_dropout=0.1,\n target_modules=[\"q_proj\", \"v_proj\"], # Minimal target modules\n # For VLM models, we typically want to freeze the vision encoder\n # and only adapt the language model parameters\n modules_to_save=None,\n )\n\n try:\n trainer = GRPOTrainer(\n model=model,\n processing_class=processor,\n reward_funcs=[reward_func],\n args=training_args,\n train_dataset=dataset,\n peft_config=lora_config,\n )\n\n assert isinstance(trainer.model, PeftModel)\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that LoRA parameters have changed\n # For VLM models, we're more permissive about which parameters can change\n lora_params_changed = False\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if \"lora\" in n.lower(): # LoRA parameters should change\n if not torch.equal(param, new_param):\n lora_params_changed = True\n\n # At least some LoRA parameters should have changed during training\n assert lora_params_changed, \"No LoRA parameters were updated during training.\"\n\n except torch.OutOfMemoryError as e:\n pytest.skip(f\"Skipping VLM training test due to insufficient GPU memory: {e}\")\n except Exception as e:\n # Check for other memory-related errors\n if any(keyword in str(e).lower() for keyword in [\"memory\", \"cuda\", \"out of memory\", \"insufficient\"]):\n pytest.skip(f\"Skipping VLM training test due to hardware constraints: {e}\")\n else:\n raise\n\n release_memory(model, trainer)\n\n @require_vllm\n @require_bitsandbytes\n @require_peft\n def test_vlm_processor_vllm_colocate_mode(self):\n \"\"\"\n Test that VLM processors work with vLLM in colocate mode.\n\n This test uses multiple memory optimization techniques to ensure it runs on limited hardware:\n - LoRA (Low-Rank Adaptation) with minimal rank (r=4)\n - 4-bit quantization with BitsAndBytesConfig\n - Gradient checkpointing\n - bfloat16 precision\n - Minimal batch sizes and sequence lengths\n - Very low GPU memory utilization (5%)\n \"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n config = GRPOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=1, # Minimal batch size\n gradient_accumulation_steps=2, # Make effective batch size 2, divisible by num_generations\n num_generations=2,\n max_completion_length=4, # Very short completions to reduce memory\n use_vllm=True, # Enable vLLM\n vllm_mode=\"colocate\", # Use colocate mode to avoid server dependency\n vllm_gpu_memory_utilization=0.05, # Use minimal GPU memory (5%)\n bf16=True, # Use bfloat16 to reduce memory\n report_to=\"none\",\n logging_strategy=\"no\",\n )\n\n # Create a VLM processor\n processor = AutoProcessor.from_pretrained(\"HuggingFaceTB/SmolVLM-Instruct\", use_fast=True, padding_side=\"left\")\n\n # Verify processor has both required attributes for VLM detection\n assert hasattr(processor, \"tokenizer\")\n assert hasattr(processor, \"image_processor\")\n\n def dummy_reward_func(completions, **kwargs):\n return [1.0] * len(completions)\n\n # Use LoRA configuration for memory efficiency\n lora_config = LoraConfig(\n r=4, # Very low rank for minimal memory\n lora_alpha=8,\n target_modules=[\"q_proj\", \"v_proj\"], # Minimal target modules\n lora_dropout=0.1,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n )\n\n # Use 4-bit quantization for further memory reduction\n quantization_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_compute_dtype=torch.bfloat16,\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_use_double_quant=True,\n )\n\n original_env = {}\n required_env_vars = {\n \"RANK\": \"0\",\n \"LOCAL_RANK\": \"0\",\n \"WORLD_SIZE\": \"1\",\n \"LOCAL_WORLD_SIZE\": \"1\",\n \"MASTER_ADDR\": \"localhost\",\n \"MASTER_PORT\": \"12355\",\n }\n\n for key, value in required_env_vars.items():\n original_env[key] = os.environ.get(key)\n os.environ[key] = value\n\n try:\n # Test VLM processor with vLLM colocate mode\n with warnings.catch_warnings(record=True) as w:\n warnings.simplefilter(\"always\")\n try:\n # Load model with quantization for memory efficiency\n model = AutoModelForCausalLM.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n quantization_config=quantization_config,\n dtype=torch.bfloat16,\n )\n\n trainer = GRPOTrainer(\n model=model,\n reward_funcs=dummy_reward_func,\n args=config,\n train_dataset=dataset,\n processing_class=processor, # VLM processor\n peft_config=lora_config, # Use LoRA for memory efficiency\n )\n\n # Should detect VLM processor correctly and allow vLLM\n assert trainer.use_vllm, \"vLLM should be enabled for VLM processors in colocate mode\"\n assert trainer.vllm_mode == \"colocate\", \"Should use colocate mode\"\n\n # Check if signature columns were set properly\n if trainer._signature_columns is not None:\n # Should include 'image' in signature columns for VLM processors\n assert \"image\" in trainer._signature_columns, (\n \"Should include 'image' in signature columns for VLM\"\n )\n\n # Should not emit any warnings about VLM incompatibility\n incompatibility_warnings = [\n str(w_item.message)\n for w_item in w\n if \"does not support VLMs\" in str(w_item.message)\n or \"not compatible\" in str(w_item.message).lower()\n ]\n assert len(incompatibility_warnings) == 0, (\n f\"Should not emit VLM incompatibility warnings, but got: {incompatibility_warnings}\"\n )\n\n # Test passes if we get this far without exceptions\n\n except Exception as e:\n # If vLLM fails to initialize due to hardware constraints or other issues, that's expected\n if any(\n keyword in str(e).lower()\n for keyword in [\n \"outofmemoryerror\",\n \"cuda\",\n \"memory\",\n \"insufficient\",\n \"no such device\",\n \"free memory\",\n \"gpu memory utilization\",\n \"decrease gpu memory\",\n ]\n ):\n pytest.skip(f\"Skipping vLLM colocate test due to hardware constraints: {e}\")\n elif \"KeyError\" in str(e) and \"RANK\" in str(e):\n pytest.skip(f\"Skipping vLLM colocate test due to environment setup issues: {e}\")\n elif \"ValueError\" in str(e) and \"memory\" in str(e).lower():\n pytest.skip(f\"Skipping vLLM colocate test due to memory constraints: {e}\")\n else:\n raise\n finally:\n # Restore original environment variables\n for key, original_value in original_env.items():\n if original_value is None:\n os.environ.pop(key, None)\n else:\n os.environ[key] = original_value\n\n release_memory(model, trainer)\n\n @require_vllm\n def test_training_vllm(self):\n \"\"\"Test that training works with vLLM for generation.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = GRPOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n logging_strategy=\"no\",\n use_vllm=True,\n )\n\n try:\n trainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\", # tiny models are too small for vLLM\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n except Exception as e:\n # If vLLM fails to initialize due to hardware constraints or other issues, that's expected\n if any(\n keyword in str(e).lower()\n for keyword in [\n \"outofmemoryerror\",\n \"cuda\",\n \"memory\",\n \"insufficient\",\n \"no such device\",\n \"free memory\",\n \"gpu memory utilization\",\n \"decrease gpu memory\",\n ]\n ):\n pytest.skip(f\"Skipping vLLM training test due to hardware constraints: {e}\")\n elif \"KeyError\" in str(e) and \"RANK\" in str(e):\n pytest.skip(f\"Skipping vLLM training test due to environment setup issues: {e}\")\n elif \"ValueError\" in str(e) and \"memory\" in str(e).lower():\n pytest.skip(f\"Skipping vLLM training test due to memory constraints: {e}\")\n else:\n raise\n\n release_memory(trainer.model, trainer)\n" }, { "path": "tests/test_model_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom transformers import AutoModelForCausalLM\n\nfrom trl.models.utils import disable_gradient_checkpointing\n\n\nclass TestDisableGradientCheckpointing:\n def test_when_disabled(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n assert model.is_gradient_checkpointing is False\n with disable_gradient_checkpointing(model):\n assert model.is_gradient_checkpointing is False\n assert model.is_gradient_checkpointing is False\n\n def test_when_enabled(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n model.gradient_checkpointing_enable()\n assert model.is_gradient_checkpointing is True\n with disable_gradient_checkpointing(model):\n assert model.is_gradient_checkpointing is False\n assert model.is_gradient_checkpointing is True\n" }, { "path": "tests/test_reward_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport json\nimport pathlib\n\nimport pytest\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForSequenceClassification, AutoTokenizer\nfrom transformers.utils import is_peft_available\n\nfrom trl import RewardConfig, RewardTrainer\nfrom trl.trainer.reward_trainer import DataCollatorForPreference\n\nfrom .testing_utils import TrlTestCase, require_peft\n\n\nif is_peft_available():\n from peft import LoraConfig, get_peft_model\n\n\nclass TestDataCollatorForPreference(TrlTestCase):\n def test_basic_padding(self):\n \"\"\"Test basic padding functionality without completion masks.\"\"\"\n collator = DataCollatorForPreference(pad_token_id=0)\n examples = [\n {\"chosen_ids\": [1, 2, 3], \"rejected_ids\": [4, 5]},\n {\"chosen_ids\": [6, 7], \"rejected_ids\": [8]},\n ]\n\n result = collator(examples)\n\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3], [6, 7, 0], [4, 5, 0], [8, 0, 0]]))\n torch.testing.assert_close(\n result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0], [1, 1, 0], [1, 0, 0]])\n )\n\n def test_pad_to_multiple_of(self):\n \"\"\"Test padding to multiple of specified value.\"\"\"\n collator = DataCollatorForPreference(pad_token_id=0, pad_to_multiple_of=4)\n examples = [\n {\"chosen_ids\": [1, 2, 3], \"rejected_ids\": [4, 5]},\n {\"chosen_ids\": [6, 7], \"rejected_ids\": [8]},\n ]\n\n result = collator(examples)\n\n torch.testing.assert_close(\n result[\"input_ids\"], torch.tensor([[1, 2, 3, 0], [6, 7, 0, 0], [4, 5, 0, 0], [8, 0, 0, 0]])\n )\n torch.testing.assert_close(\n result[\"attention_mask\"], torch.tensor([[1, 1, 1, 0], [1, 1, 0, 0], [1, 1, 0, 0], [1, 0, 0, 0]])\n )\n\n def test_single_example(self):\n \"\"\"Test collator with a single example.\"\"\"\n collator = DataCollatorForPreference(pad_token_id=0)\n examples = [{\"chosen_ids\": [1, 2, 3], \"rejected_ids\": [4, 5]}]\n\n result = collator(examples)\n\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3], [4, 5, 0]]))\n torch.testing.assert_close(result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0]]))\n\n def test_different_pad_token_id(self):\n \"\"\"Test with different pad token ID.\"\"\"\n collator = DataCollatorForPreference(pad_token_id=999)\n examples = [\n {\"chosen_ids\": [1, 2, 3], \"rejected_ids\": [4, 5]},\n {\"chosen_ids\": [6, 7], \"rejected_ids\": [8]},\n ]\n\n result = collator(examples)\n\n torch.testing.assert_close(\n result[\"input_ids\"], torch.tensor([[1, 2, 3], [6, 7, 999], [4, 5, 999], [8, 999, 999]])\n )\n torch.testing.assert_close(\n result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0], [1, 1, 0], [1, 0, 0]])\n )\n\n def test_collate_with_margin(self):\n collator = DataCollatorForPreference(pad_token_id=0)\n examples = [\n {\"chosen_ids\": [1, 2, 3], \"rejected_ids\": [4, 5], \"margin\": 0.1},\n {\"chosen_ids\": [6, 7], \"rejected_ids\": [8], \"margin\": 0.2},\n ]\n\n result = collator(examples)\n\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3], [6, 7, 0], [4, 5, 0], [8, 0, 0]]))\n torch.testing.assert_close(\n result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0], [1, 1, 0], [1, 0, 0]])\n )\n torch.testing.assert_close(result[\"margin\"], torch.tensor([0.1, 0.2]))\n\n\nclass TestRewardTrainer(TrlTestCase):\n def test_raises_error_when_model_num_labels_not_one(self):\n \"\"\"Test that RewardTrainer raises ValueError when model doesn't have num_labels=1.\"\"\"\n model = AutoModelForSequenceClassification.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n dtype=\"float32\",\n # num_labels=2, # Defaults to 2 num_labels for causal models\n )\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n with pytest.raises(ValueError, match=r\"reward models require `num_labels=1`\"):\n RewardTrainer(model=model, args=training_args, train_dataset=dataset)\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"trl-internal-testing/tiny-Qwen3MoeForCausalLM\",\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n ],\n )\n def test_train(self, model_id):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = RewardTrainer(model=model_id, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\n \"config_name\",\n [\n \"standard_preference\",\n \"conversational_preference\",\n \"standard_implicit_prompt_preference\",\n \"conversational_implicit_prompt_preference\",\n ],\n )\n def test_train_dataset_types(self, config_name):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", config_name, split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_model(self):\n # Instantiate the model\n model = AutoModelForSequenceClassification.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n num_labels=1, # required for reward models\n dtype=\"float32\",\n )\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = RewardTrainer(model=model, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_from_sequence_classification_model(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_model_dtype(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(\n output_dir=self.tmp_dir,\n model_init_kwargs={\"dtype\": torch.float16},\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n # For some reasonn model.layers.0.input_layernorm.weight doesn't change in GitHub Actions but does\n # locally. We ignore this parameter for now\n if \"layernorm\" in n:\n continue\n new_param = trainer.model.get_parameter(n)\n # Check the torch dtype\n assert new_param.dtype == torch.float16\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_peft\n def test_train_dense_with_peft_config(self):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForSequenceClassification.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n trainer = RewardTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_peft\n def test_train_moe_with_peft_config(self):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen3MoeForCausalLM\"\n model = AutoModelForSequenceClassification.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n trainer = RewardTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(target_modules=[\"up_proj\", \"down_proj\", \"score\"]),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_peft\n def test_train_peft_model(self):\n # Get the base model\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForSequenceClassification.from_pretrained(\n model_id,\n num_labels=1, # required for reward models\n dtype=\"float32\",\n )\n\n # Get the base model parameter names\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Turn the model into a peft model\n lora_config = LoraConfig()\n model = get_peft_model(model, lora_config)\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = RewardTrainer(model=model, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n # In practice, this test is the same as `test_train_dense_with_peft_config`, since gradient checkpointing is\n # enabled by default in `RewardTrainer`. We keep it as a regression guard: if the default ever changes, we still\n # explicitly test PEFT + gradient checkpointing, which has caused issues in the past.\n @require_peft\n def test_train_with_peft_config_and_gradient_checkpointing(self):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForSequenceClassification.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, gradient_checkpointing=True, report_to=\"none\")\n\n trainer = RewardTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\"use_reentrant\", [True, False])\n @require_peft\n def test_train_with_peft_config_and_gradient_checkpointing_reentrant(self, use_reentrant):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n model = AutoModelForSequenceClassification.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(\n output_dir=self.tmp_dir,\n gradient_checkpointing=True,\n gradient_checkpointing_kwargs={\"use_reentrant\": use_reentrant},\n report_to=\"none\",\n )\n\n trainer = RewardTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\n \"chosen_column,rejected_column,expect_deprecation_warning\",\n [\n (\"chosen_ids\", \"rejected_ids\", False),\n (\"chosen_input_ids\", \"rejected_input_ids\", True),\n ],\n )\n def test_train_with_pretokenized_data(self, chosen_column, rejected_column, expect_deprecation_warning):\n # Get the dataset\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n def tokenize_example(example):\n return {\n chosen_column: tokenizer(example[\"chosen\"]).input_ids,\n rejected_column: tokenizer(example[\"rejected\"]).input_ids,\n }\n\n # Apply tokenization\n tokenized_dataset = dataset.map(tokenize_example, remove_columns=[\"chosen\", \"rejected\"])\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n if expect_deprecation_warning:\n with pytest.warns(FutureWarning, match=r\"will not be supported in v1\"):\n trainer = RewardTrainer(model=model_id, args=training_args, train_dataset=tokenized_dataset)\n else:\n trainer = RewardTrainer(model=model_id, args=training_args, train_dataset=tokenized_dataset)\n\n assert \"chosen_ids\" in trainer.train_dataset.column_names\n assert \"rejected_ids\" in trainer.train_dataset.column_names\n assert \"chosen_input_ids\" not in trainer.train_dataset.column_names\n assert \"rejected_input_ids\" not in trainer.train_dataset.column_names\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_iterable_dataset(self):\n # Get the dataset\n dataset = load_dataset(\n \"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\", streaming=True\n )\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, max_steps=3, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_chat_template_kwargs(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n # The following template is a simplified version of the Qwen chat template, where an additional argument\n # `role_capital` is used to control the capitalization of roles.\n tokenizer.chat_template = '{%- if messages[0][\"role\"] == \"system\" -%} {{ \"<|im_start|>\" + (\"SYSTEM\" if role_capital else \"system\") + \"\\\\n\" + messages[0][\"content\"] + \"<|im_end|>\\\\n\" }}{%- else -%} {{ \"<|im_start|>\" + (\"SYSTEM\" if role_capital else \"system\") + \"\\\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\\\n\" }}{%- endif -%}{%- for message in messages -%} {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) -%} {{ \"<|im_start|>\" + (message.role.upper() if role_capital else message.role) + \"\\\\n\" + message.content + \"<|im_end|>\\\\n\" }} {%- elif message.role == \"assistant\" -%} {{ \"<|im_start|>\" + (\"ASSISTANT\" if role_capital else \"assistant\") }} {%- if message.content -%} {{ \"\\\\n\" + message.content }} {%- endif -%} {{ \"<|im_end|>\\\\n\" }} {%- elif message.role == \"tool\" -%} {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") -%} {{ \"<|im_start|>\" + (\"USER\" if role_capital else \"user\") }} {%- endif -%} {{ \"\\\\n\\\\n\" + message.content + \"\\\\n\" }} {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") -%} {{ \"<|im_end|>\\\\n\" }} {%- endif -%} {%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%} {{ \"<|im_start|>\" + (\"ASSISTANT\" if role_capital else \"assistant\") + \"\\\\n\" }}{%- endif -%}'\n\n dataset = dataset.add_column(\n \"chat_template_kwargs\", [{\"role_capital\": bool(i % 2)} for i in range(len(dataset))]\n )\n assert \"chat_template_kwargs\" in dataset.features\n\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n processing_class=tokenizer,\n )\n\n # Assert trainer uses the same chat template as tokenizer\n assert trainer.processing_class.chat_template == tokenizer.chat_template\n\n # Assert chat_template is applied\n for i in range(2):\n role = \"SYSTEM\" if i else \"system\"\n system_prompt = (\n f\"<|im_start|>{role}\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\"\n )\n system_prompt_ids = trainer.processing_class(system_prompt)[\"input_ids\"]\n assert trainer.train_dataset[i][\"chosen_ids\"][: len(system_prompt_ids)] == system_prompt_ids\n assert trainer.train_dataset[i][\"rejected_ids\"][: len(system_prompt_ids)] == system_prompt_ids\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_set_chat_template_from_model(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, chat_template_path=\"Qwen/Qwen3-4B\", report_to=\"none\")\n # trl-internal-testing/tiny-GPTNeoXForCausalLM doesn't have a chat template set by default\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-GPTNeoXForCausalLM\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n # RewardTrainer uses a mean-free loss that cancels uniform shifts in output scores. Since GPT-NeoX models\n # include a final LayerNorm, its bias consistently receives zero gradient and remains unchanged, so we skip\n # this parameter.\n if n == \"gpt_neox.final_layer_norm.bias\":\n continue\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_set_chat_template_from_path(self, lazy_shared_datadir):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(\n output_dir=self.tmp_dir,\n chat_template_path=str(lazy_shared_datadir / \"template.jinja\"),\n report_to=\"none\",\n )\n # trl-internal-testing/tiny-GPTNeoXForCausalLM doesn't have a chat template set by default\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-GPTNeoXForCausalLM\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n # RewardTrainer uses a mean-free loss that cancels uniform shifts in output scores. Since GPT-NeoX models\n # include a final LayerNorm, its bias consistently receives zero gradient and remains unchanged, so we skip\n # this parameter.\n if n == \"gpt_neox.final_layer_norm.bias\":\n continue\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n # Check that the template saved in the output directory is the same as the one used for training\n template_path = pathlib.Path(self.tmp_dir) / \"checkpoint-9\" / \"chat_template.jinja\"\n assert template_path.exists(), f\"Chat template not found at {template_path}\"\n\n with open(template_path) as f:\n template_content = f.read()\n with open(training_args.chat_template_path) as f:\n original_template_content = f.read()\n assert template_content == original_template_content, \"Chat template content does not match the original\"\n\n def test_train_toolcall_data(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/toolcall\", \"preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_toolcall_data_as_json(self):\n # Tabular backends (Arrow/Parquet) can insert `None` for missing keys in nested structures.\n # If `tools` is stored as a list of dicts and examples use different dict schemas, nulls may\n # be introduced and break tool processing. This test ensures we also support `tools` provided\n # as a list of dicts.\n dataset = load_dataset(\"trl-internal-testing/toolcall\", \"preference\", split=\"train\")\n\n def convert_to_json(example):\n return {\"tools\": json.loads(example[\"tools\"])}\n\n dataset = dataset.map(convert_to_json)\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_eval(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, eval_strategy=\"steps\", eval_steps=3, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n # Train the model\n trainer.train()\n\n # Check that the eval loss is not None\n assert trainer.state.log_history[0][\"eval_loss\"] is not None\n\n def test_train_with_multiple_eval_dataset(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, eval_strategy=\"steps\", eval_steps=3, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset={\"data1\": dataset[\"test\"], \"data2\": dataset[\"test\"]},\n )\n # Train the model\n trainer.train()\n\n # Check that the eval losses are not None\n assert trainer.state.log_history[-3][\"eval_data1_loss\"] is not None\n assert trainer.state.log_history[-2][\"eval_data2_loss\"] is not None\n\n def test_train_with_compute_metrics(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\")\n\n def dummy_compute_metrics(eval_pred):\n return {\"my_metric\": 0.123}\n\n # Initialize the trainer\n training_args = RewardConfig(\n output_dir=self.tmp_dir,\n eval_strategy=\"steps\",\n eval_steps=3,\n report_to=\"none\",\n )\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n compute_metrics=dummy_compute_metrics,\n )\n\n # Train the model\n trainer.train()\n\n # Check that the custom metric is logged\n assert trainer.state.log_history[-2][\"eval_my_metric\"] == 0.123\n\n # In practice, this test is the same as `test_train`, since gradient checkpointing is enabled by default in\n # `RewardTrainer`. We keep it as a regression guard: if the default ever changes, we still explicitly test gradient\n # checkpointing, which has caused issues in the past.\n def test_train_with_gradient_checkpointing(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, gradient_checkpointing=True, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\"use_reentrant\", [True, False])\n def test_train_with_gradient_checkpointing_reentrant(self, use_reentrant):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(\n output_dir=self.tmp_dir,\n gradient_checkpointing=True,\n gradient_checkpointing_kwargs={\"use_reentrant\": use_reentrant},\n report_to=\"none\",\n )\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_tag_added(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n train_dataset=dataset,\n )\n\n for tag in [\"reward-trainer\", \"trl\"]:\n assert tag in trainer.model.model_tags\n\n @require_peft\n def test_tag_added_peft(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n for tag in [\"reward-trainer\", \"trl\"]:\n assert tag in trainer.model.model_tags\n\n def test_train_with_margin(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n def add_margin(example):\n # dummy margin based on the length of the chosen summary\n return {\"margin\": len(example[\"chosen\"])}\n\n dataset = dataset.map(add_margin)\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_center_rewards_coefficient(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = RewardConfig(output_dir=self.tmp_dir, center_rewards_coefficient=0.01, report_to=\"none\")\n trainer = RewardTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n" }, { "path": "tests/test_rewards.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport threading\n\nfrom trl.rewards import accuracy_reward, get_soft_overlong_punishment, reasoning_accuracy_reward, think_format_reward\n\nfrom .testing_utils import TrlTestCase, require_math_latex\n\n\nclass TestThinkFormatReward(TrlTestCase):\n def test_valid_format(self):\n completions = [\n \"This is my reasoning.This is my answer.\", # Simple, one-line reasoning\n \"\\nThis is my reasoning.\\n\\nThis is my answer.\", # Multiline reasoning\n \"\\nThis is\\nmy reasoning.\\n\\nThis is my answer.\", # Multiline reasoning\n \"\\nThis is my reasoning.\\nThis is my answer.\", # Reasoning including other tags\n \"\\nThis is my answer.\", # Empty reasoning\n ]\n completions = [[{\"content\": completion}] for completion in completions]\n expected_rewards = [1.0, 1.0, 1.0, 1.0, 1.0] # All should be valid\n rewards = think_format_reward(completions)\n assert rewards == expected_rewards\n\n def test_invalid_format(self):\n completions = [\n \"\\nThis is my reasoning.\\nThis is my answer.\", # No closing \n \"This is my reasoning.\\nThis is my answer.\", # No closing \n \"This is my reasoning. This is my answer.\", # No tags\n \"This is my reasoning.\\nThis is my answer.\", # No tags\n \"This is my reasoning.\\nThis is my answer.\", # No opening \n \"This is my reasoning.This is my answer.\", # No opening \n \"Thisis my reasoning.\\nThis is my answer.\", # tag in the middle\n \"This ismy reasoning.This is my answer.\", # Nested tags\n \"This is\\nmy\\nreasoning.\\nThis is my answer.\", # Multiline \n ]\n completions = [[{\"content\": completion}] for completion in completions]\n expected_rewards = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] # All should be invalid\n rewards = think_format_reward(completions)\n assert rewards == expected_rewards\n\n def test_mixed_format(self):\n completions = [\n \"This is my reasoning.This is my answer.\", # Valid\n \"\\nThis is my reasoning.\\n\\nThis is my answer.\", # Valid\n \"This is my reasoning.\\nThis is my answer.\", # Invalid\n \"This is my reasoning. This is my answer.\", # Invalid\n ]\n completions = [[{\"content\": completion}] for completion in completions]\n expected_rewards = [1.0, 1.0, 0.0, 0.0]\n rewards = think_format_reward(completions)\n assert rewards == expected_rewards\n\n\nclass TestSoftOverlongPunishmentReward:\n def test_soft_overlong_punishment_short_completion(self):\n \"\"\"Test soft overlong punishment reward function with a short completion.\"\"\"\n # length 50, with max=100 and soft cache=20, reward should be 0.\n reward_fn = get_soft_overlong_punishment(max_completion_len=100, soft_punish_cache=20)\n completion_ids = [[1] * 50] # 50 <= 80\n rewards = reward_fn(completion_ids=completion_ids)\n assert rewards == [0]\n\n def test_soft_overlong_punishment_long_completion(self):\n \"\"\"Test soft overlong punishment reward function with a longer than max completion.\"\"\"\n # 110 > 100, reward should be -1.\n reward_fn = get_soft_overlong_punishment(max_completion_len=100, soft_punish_cache=20)\n completion_ids = [[1] * 110]\n rewards = reward_fn(completion_ids)\n assert rewards == [-1]\n\n def test_soft_overlong_punishment_intermediate_completion(self):\n \"\"\"Test soft overlong punishment reward function for intermediate length completion.\"\"\"\n reward_fn = get_soft_overlong_punishment(max_completion_len=100, soft_punish_cache=20)\n completion_ids = [[1] * 90] # 90 is between 80 and 100\n rewards = reward_fn(completion_ids)\n assert round(abs(rewards[0] - -0.5), 4) == 0\n\n\nclass TestAccuracyReward:\n @require_math_latex\n def test_accuracy_reward_correct_answer(self):\n \"\"\"Test accuracy_reward with a correct answer.\"\"\"\n completion = [[{\"content\": r\"\\boxed{\\frac{63}{400}}\"}], [{\"content\": r\"\\boxed{\\frac{63}{400}}\"}]]\n solution = [r\"\\frac{63}{400}\", \"63/400\"]\n rewards = accuracy_reward(completion, solution)\n assert rewards[0] == 1.0\n assert rewards[1] == 1.0\n\n @require_math_latex\n def test_accuracy_reward_wrong_answer(self):\n \"\"\"Test accuracy_reward with an incorrect answer.\"\"\"\n completion = [[{\"content\": r\"\\boxed{\\frac{64}{400}}\"}]]\n solution = [r\"\\frac{63}{400}\"]\n rewards = accuracy_reward(completion, solution)\n assert rewards[0] == 0.0\n\n @require_math_latex\n def test_accuracy_reward_wrong_answer_no_latex(self):\n \"\"\"Test accuracy_reward with an incorrect answer and gold solution with no latex.\"\"\"\n completion = [[{\"content\": r\"\\boxed{3}\"}]]\n solution = [\"6\"]\n rewards = accuracy_reward(completion, solution)\n assert rewards[0] == 0.0\n\n @require_math_latex\n def test_accuracy_reward_unparsable_gold(self):\n \"\"\"Test accuracy_reward with an unparsable gold solution.\"\"\"\n completion = [\n [{\"content\": \"Answer is forty two.\"}],\n [{\"content\": r\"Some other content. \\boxed{43}.\"}],\n ]\n solution = [\n \"Answer is forty two.\",\n \"Answer is forty three.\",\n ]\n rewards = accuracy_reward(completion, solution)\n assert rewards[0] is None\n assert rewards[1] is None\n\n @require_math_latex\n def test_accuracy_reward_in_worker_thread(self):\n \"\"\"Test that accuracy_reward works when called from a non-main thread.\"\"\"\n completions = [[{\"content\": r\"\\boxed{\\frac{1}{3}}\"}]]\n solutions = [r\"\\frac{1}{3}\"]\n results = []\n exceptions = []\n\n def target():\n try:\n results.extend(accuracy_reward(completions, solutions))\n except Exception as e:\n exceptions.append(e)\n\n t = threading.Thread(target=target)\n t.start()\n t.join()\n\n assert not exceptions, f\"accuracy_reward raised in worker thread: {exceptions[0]}\"\n assert results == [1.0]\n\n\nclass TestReasoningAccuracyReward:\n @require_math_latex\n def test_correct_answer_yields_unit_reward(self):\n completions = [\n [{\"content\": r\" Reasoning content \\boxed{\\frac{63}{400}}\"}],\n [{\"content\": r\"Reasoning content \\boxed{\\frac{63}{400}}\"}],\n ]\n solutions = [r\"\\frac{63}{400}\", r\"\\frac{63}{400}\"]\n rewards = reasoning_accuracy_reward(completions, solutions)\n assert rewards[0] == 1.0\n assert rewards[1] == 1.0\n\n @require_math_latex\n def test_correct_answer_with_custom_tags_yields_unit_reward(self):\n completions = [\n [{\"content\": r\" Reasoning content \\boxed{\\frac{63}{400}}\"}],\n ]\n solutions = [\n r\"\\frac{63}{400}\",\n ]\n rewards = reasoning_accuracy_reward(completions, solutions, reasoning_delimiters=[\"\"])\n assert rewards[0] == 1.0\n\n @require_math_latex\n def test_incorrect_answer_yields_zero_reward(self):\n completion = [[{\"content\": r\" Reasoning content \\boxed{\\frac{64}{400}}\"}]]\n solution = [r\"\\frac{63}{400}\"]\n rewards = reasoning_accuracy_reward(completion, solution)\n assert rewards[0] == 0.0\n\n @require_math_latex\n def test_correct_answer_in_reasoning_yields_zero_reward(self):\n completions = [\n [{\"content\": r\" My answer is \\boxed{42} Some other text.\"}],\n [{\"content\": r\" The answer is \\boxed{42} Here's a wrong answer: \\boxed{43}.\"}],\n ]\n solutions = [r\"\\boxed{42}\", r\"\\boxed{42}\"]\n rewards = reasoning_accuracy_reward(completions, solutions)\n assert rewards[0] == 0.0\n assert rewards[1] == 0.0\n\n @require_math_latex\n def test_incomplete_reasoning_yields_zero_reward(self):\n completions = [\n [{\"content\": r\" Incomplete reasoning without closing tag\"}],\n [{\"content\": r\"Correct answer \\frac{63}{400} but completely missing reasoning content\"}],\n ]\n solutions = [r\"\\frac{63}{400}\", r\"\\frac{63}{400}\"]\n rewards = reasoning_accuracy_reward(completions, solutions)\n assert rewards[0] == 0.0\n assert rewards[1] == 0.0\n\n @require_math_latex\n def test_unparsable_gold_solution_yields_none_reward(self):\n completions = [\n [{\"content\": r\" Reasoning content \\boxed{42}\"}],\n ]\n solutions = [\n \"forty two\",\n ]\n rewards = reasoning_accuracy_reward(completions, solutions)\n assert rewards[0] is None\n" }, { "path": "tests/test_rich_progress_callback.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport torch\nimport torch.nn as nn\nfrom datasets import Dataset\nfrom transformers import Trainer, TrainingArguments\n\nfrom trl.trainer.callbacks import RichProgressCallback\n\nfrom .testing_utils import TrlTestCase, require_rich\n\n\nclass DummyModel(nn.Module):\n def __init__(self):\n super().__init__()\n self.a = nn.Parameter(torch.tensor(1.0))\n\n def forward(self, x):\n return self.a * x\n\n\n@require_rich\nclass TestRichProgressCallback(TrlTestCase):\n def setup_method(self):\n self.dummy_model = DummyModel()\n self.dummy_train_dataset = Dataset.from_list([{\"x\": 1.0, \"y\": 2.0}] * 5)\n self.dummy_val_dataset = Dataset.from_list([{\"x\": 1.0, \"y\": 2.0}] * 101)\n\n def test_rich_progress_callback_logging(self):\n training_args = TrainingArguments(\n output_dir=self.tmp_dir,\n per_device_eval_batch_size=2,\n per_device_train_batch_size=2,\n num_train_epochs=4,\n eval_strategy=\"steps\",\n eval_steps=1,\n logging_strategy=\"steps\",\n logging_steps=1,\n save_strategy=\"no\",\n report_to=\"none\",\n disable_tqdm=True,\n )\n callbacks = [RichProgressCallback()]\n trainer = Trainer(\n model=self.dummy_model,\n train_dataset=self.dummy_train_dataset,\n eval_dataset=self.dummy_val_dataset,\n args=training_args,\n callbacks=callbacks,\n )\n\n trainer.train()\n" }, { "path": "tests/test_rloo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom unittest.mock import patch\n\nimport pytest\nimport torch\nimport transformers\nfrom datasets import load_dataset\nfrom packaging.version import Version\nfrom transformers import (\n AutoModelForCausalLM,\n AutoModelForImageTextToText,\n AutoModelForSequenceClassification,\n AutoTokenizer,\n)\nfrom transformers.utils import is_peft_available\n\nfrom trl import RLOOConfig, RLOOTrainer\n\nfrom .testing_utils import TrlTestCase, require_peft, require_vision, require_vllm\n\n\nif is_peft_available():\n from peft import LoraConfig, get_peft_model\n\n\nclass TestRLOOTrainer(TrlTestCase):\n def test_init_minimal(self):\n # Test that RLOOTrainer can be instantiated with only model, reward_model and train_dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n train_dataset=dataset,\n )\n\n @pytest.mark.parametrize(\"config_name\", [\"standard_prompt_only\", \"conversational_prompt_only\"])\n def test_training(self, config_name):\n dataset = load_dataset(\"trl-internal-testing/zen\", config_name, split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_eval(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n per_device_eval_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n eval_strategy=\"steps\",\n eval_steps=2,\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n trainer.train()\n\n def test_training_with_num_generations_eval(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n per_device_eval_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n num_generations_eval=1,\n eval_strategy=\"steps\",\n eval_steps=2,\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n trainer.train()\n\n def test_training_multiple_iterations(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n num_iterations=2,\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_peft\n def test_training_peft_config(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n: # We expect the peft params to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_peft\n def test_training_peft_model(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n lora_config = LoraConfig()\n model = get_peft_model(model, lora_config)\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n and \"ref\" not in n: # and the peft params to be different (except base and ref)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n # In practice, this test is the same as `test_training_peft_config`, since gradient checkpointing is enabled by\n # default in `RLOOTrainer`. We keep it as a regression guard: if the default ever changes, we still explicitly test\n # PEFT + gradient checkpointing, which has caused issues in the past.\n @require_peft\n def test_training_peft_with_gradient_checkpointing(self):\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n gradient_checkpointing=True, # enable gradient checkpointing\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n: # We expect the peft params to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_different_reward_model(self):\n # Use a reward model different from the model: different chat template, tokenization, etc.\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_only\", split=\"train\")\n reward_model_id = \"trl-internal-testing/tiny-LlamaForSequenceClassification-3.2\"\n reward_model = AutoModelForSequenceClassification.from_pretrained(reward_model_id)\n reward_tokenizer = AutoTokenizer.from_pretrained(reward_model_id)\n # By default, the trainer uses the eos token as the padding token. However, for Llama models, the eos token\n # appears in the chat template. Using it as a pad token disrupts the reward calculation, as the calculation\n # considers the score of the last token before the first pad token. To ensure correct reward calculations,\n # we use a separate pad token instead.\n reward_tokenizer.pad_token = \"<|finetune_right_pad_id|>\"\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_model,\n args=training_args,\n train_dataset=dataset,\n reward_processing_classes=reward_tokenizer,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_reward_func_standard(self):\n # Test if trainer can handle reward function with standard format\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_reward_func_conversational(self):\n # Test if trainer can handle reward function with conversational format\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that gives higher scores to longer completion content.\"\"\"\n completion_contents = [completion[0][\"content\"] for completion in completions]\n return [float(len(content)) for content in completion_contents]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_reward_funcs(self):\n # Test that RLOOTrainer can be instantiated with multiple reward functions\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func1(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n def reward_func2(completions, **kwargs):\n \"\"\"Reward function that rewards completions with more unique letters.\"\"\"\n return [float(len(set(completion))) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[reward_func1, reward_func2],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_sync_and_async_reward_funcs(self):\n # Test that RLOOTrainer can be instantiated with multiple reward functions one of which is async\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def sync_reward_func1(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n def sync_reward_func2(completions, **kwargs):\n return [1 for _ in completions]\n\n async def async_reward_func(completions, **kwargs):\n \"\"\"Async Reward function that rewards completions with more unique letters.\"\"\"\n return [float(len(set(completion))) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[sync_reward_func1, sync_reward_func2, async_reward_func],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_reward_funcs_with_None_output(self):\n \"\"\"Test that a valid math reward function is processed correctly while the code reward function returns None.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def applicable_reward_func(completions, **kwargs):\n \"\"\"A reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n def non_applicable_reward_func(completions, **kwargs):\n \"\"\"A reward function that returns None for all inputs, as it is not applicable to this sample.\"\"\"\n return [None] * len(completions)\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n )\n\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[\n applicable_reward_func,\n non_applicable_reward_func,\n ], # One applicable, one non applicable\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {\n n: param.clone() for n, param in trainer.model.named_parameters() if param.requires_grad\n }\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_reward_funcs_with_weights(self):\n \"\"\"Test that RLOOTrainer can handle multiple reward functions with weights.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func1(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n def reward_func2(completions, **kwargs):\n \"\"\"Reward function that rewards completions with more unique letters.\"\"\"\n return [float(len(set(completion))) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n reward_weights=[0.7, 0.3], # weight of reward_func1 and reward_func2 respectively\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[reward_func1, reward_func2],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n # Check that training logs contain both reward metrics\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert \"rewards/reward_func1/mean\" in trainer.state.log_history[-1]\n assert \"rewards/reward_func1/std\" in trainer.state.log_history[-1]\n assert \"rewards/reward_func2/mean\" in trainer.state.log_history[-1]\n assert \"rewards/reward_func2/std\" in trainer.state.log_history[-1]\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_mixed_reward_funcs(self):\n # Test if the trainer can handle a mix of reward functions and reward models\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion)) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=[reward_func, \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"],\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_reward_func_additional_column(self):\n # Test if trainer can handle reward function that rely on additional columns in the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Add a column to the dataset (dummy example, the column could be anything)\n some_values = list(range(len(dataset)))\n dataset = dataset.add_column(\"some_values\", some_values)\n\n def reward_func(completions, some_values, **kwargs):\n \"\"\"Reward function that rewards completions with lengths closer to the values in some_values.\"\"\"\n return [\n float(abs(len(completion) - value)) for completion, value in zip(completions, some_values, strict=True)\n ]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_sync_ref_model(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n beta=0.1, # ensure ref model is created so sync can update it\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n sync_ref_model=True,\n ref_model_sync_steps=2, # reduce sync steps to ensure a sync happens\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n assert trainer.ref_model is not None\n previous_ref_params = {n: param.clone() for n, param in trainer.ref_model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n new_ref_param = trainer.ref_model.get_parameter(n)\n assert not torch.equal(previous_ref_params[n], new_ref_param), f\"Ref Parameter {n} has not changed.\"\n\n def test_training_beta_zero(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n beta=0.0, # set beta to zero value to test the case where the reference model is not used\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_pad_to_multiple_of(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n pad_to_multiple_of=8,\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_peft\n @require_vllm\n @pytest.mark.skip(reason=\"We should add a mock for the vLLM server.\")\n def test_training_vllm_and_peft(self):\n \"\"\"Test that training works with vLLM for generation.\"\"\"\n model = AutoModelForCausalLM.from_pretrained(\n \"Qwen/Qwen2.5-0.5B-Instruct\", dtype=\"float32\"\n ) # tiny model is too small for vLLM\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n use_vllm=True,\n )\n lora_config = LoraConfig(\n target_modules=\"all-linear\",\n # test with non-default modules as it adds extra keys in state_dict that we need to handle\n modules_to_save=[\"embed_tokens\", \"lm_head\"],\n )\n trainer = RLOOTrainer(\n model=model,\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n peft_config=lora_config,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n and \"original_module\" not in n:\n # We expect the peft params to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_vllm\n @pytest.mark.skip(reason=\"We should add a mock for the vLLM server.\")\n def test_training_vllm_structured_outputs(self):\n \"\"\"Test that training works with vLLM for generation with structured outputs.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n use_vllm=True,\n vllm_structured_outputs_regex=r\"\\n.*\\n\\n\\n.*\\n\",\n )\n trainer = RLOOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\", # tiny model is too small for vLLM\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_additional_generation_kwargs(self):\n \"\"\"Test that training works with additional generation kwargs.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n top_p=0.9,\n top_k=10,\n min_p=0.01,\n repetition_penalty=1.1,\n )\n\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_vllm\n @pytest.mark.skip(reason=\"We should add a mock for the vLLM server.\")\n def test_training_vllm_with_additional_generation_kwargs(self):\n \"\"\"Test that training works with vLLM and additional generation kwargs.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n use_vllm=True,\n top_p=0.9,\n top_k=10,\n min_p=0.01,\n repetition_penalty=1.1,\n )\n\n trainer = RLOOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\", # tiny model is too small for vLLM\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_normalized_advantages(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n normalize_advantages=True,\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_clipped_rewards(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n reward_clip_range=(-1, 1),\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @patch(\"transformers.generation.utils.GenerationMixin.generate\")\n def test_training_with_mask_truncated_completions(self, mock_generate):\n \"\"\"Test that training works with mask_truncated_completions=True parameter.\"\"\"\n\n # We mock the generate method because the model's random weights make it extremely unlikely to produce a\n # sequence containing the EOS token within the allowed max_completion_length. As a result, all tokens are\n # masked in the loss, the model doesn't update, and the final check (which verifies the update) fails.\n def fake_generate(input_ids, **kwargs):\n # pad_token_id = 151643; eos_token_id = 151645\n completion_ids = torch.tensor(\n [\n [1, 2, 3, 4, 5, 6, 7, 8], # this one is truncated\n [9, 10, 11, 151645, 151643, 151643, 151643, 151643], # this one contains eos\n [12, 13, 14, 15, 16, 17, 18, 151645], # particular case, eos is generated just within the limit\n ],\n device=input_ids.device,\n )\n return torch.cat([input_ids, completion_ids], dim=1)\n\n mock_generate.side_effect = fake_generate\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n mask_truncated_completions=True, # Enable masking of truncated completions\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_mask_truncated_completions_all_masked(self):\n \"\"\"\n Test that when all generated completions are truncated (i.e., none contain an EOS token), and\n mask_truncated_completions=True, the model receives no effective learning signal and therefore does not update\n its parameters.\n\n Here, we don't mock the generate method, be we rely on the fact that the model the probability of generating\n the EOS token is extremely low, so all generated completions are truncated.\n \"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n mask_truncated_completions=True, # Enable masking of truncated completions\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert torch.equal(param, new_param), f\"Parameter {n} has changed.\"\n\n def test_warning_raised_all_rewards_none(self, caplog):\n \"\"\"Test that a proper warning is raised when all rewards are None.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def always_none_reward_func(completions, **kwargs):\n \"\"\"Reward function that always returns None.\"\"\"\n return [None] * len(completions)\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=always_none_reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n with caplog.at_level(\"WARNING\", logger=\"trl.trainer.rloo_trainer\"):\n trainer.train()\n\n expected_warning = \"All reward functions returned None for the following kwargs:\"\n assert expected_warning in caplog.text\n\n def test_training_num_generations_larger_than_batch_size(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n num_generations=6, # the number of generations is larger than the batch size, but\n gradient_accumulation_steps=2, # gradient accumulation should allow that\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_multiple_dataloader_workers(self):\n # Pytest/CI often starts background threads before tests run. With Python 3.12, using the default \"fork\" start\n # method in a multi-threaded process emits a DeprecationWarning and may deadlock.\n #\n # We force \"spawn\" here to make multiprocessing safe under pytest when DataLoader workers are enabled. This is\n # test-environmentโ€“specific and not required by the training logic itself.\n #\n # This means the test does not cover \"fork\". However, \"spawn\" is stricter (requires full picklability and clean\n # state) and avoids fork-after-threads issues that pytest cannot reliably test anyway. Fork-specific behavior,\n # if needed, should be tested in a clean process outside pytest.\n torch.multiprocessing.set_start_method(\"spawn\", force=True)\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n dataloader_num_workers=2, # use multiple dataloader workers\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_generation_kwargs(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n # Pass gen kwargs\n generation_kwargs={\"do_sample\": True, \"top_k\": 50, \"num_beams\": 2, \"length_penalty\": -0.1},\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_reward_func_accessing_trainer_state(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n trainer_state = kwargs.get(\"trainer_state\")\n assert trainer_state is not None\n # transformers.TrainerState instance should have a `global_step` property.\n assert hasattr(trainer_state, \"global_step\")\n return [float(len(set(completion))) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n trainer.train()\n\n def test_training_reward_func_with_log_extra(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n log_extra = kwargs.get(\"log_extra\")\n assert log_extra is not None\n log_extra(\"test_column\", [completion[:5] for completion in completions])\n return [float(len(completion)) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n log_completions=True,\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n trainer.train()\n assert \"test_column\" in trainer._logs[\"extra\"]\n\n def test_training_reward_func_with_log_metric(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n log_metric = kwargs.get(\"log_metric\")\n assert log_metric is not None\n log_metric(\"custom_accuracy\", 0.75)\n return [float(len(completion)) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n trainer.train()\n # log_metric appends to _metrics, which gets averaged and merged into log_history\n logged_keys = {k for entry in trainer.state.log_history for k in entry}\n assert \"custom_accuracy\" in logged_keys\n\n def test_prepare_input_called_with_correct_data(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n max_completion_length=8, # reduce the completion length to reduce memory usage\n gradient_accumulation_steps=3, # can be anything in this test\n # steps_per_generation*per_device_train_batch_size=24 is divisible by num_generations=4\n steps_per_generation=4,\n num_generations=4,\n per_device_train_batch_size=6, # reduce the batch size to reduce memory usage\n num_iterations=2,\n shuffle_dataset=False,\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n # steps_per_generation=4, per_device_train_batch_size=6 and num_generations=4, so we expect a\n # generation batch of 24 samples (steps_per_generation * per_device_train_batch_size), containing 6\n # different prompts (steps_per_generation * per_device_train_batch_size // num_generations), each repeated\n # 4 times (num_generations).\n expected_first_generation_batch = (\n [{\"prompt\": \"Beautiful is better than\"}] * 4\n + [{\"prompt\": \"Explicit is\"}] * 4\n + [{\"prompt\": \"Simple is better\"}] * 4\n + [{\"prompt\": \"Complex\"}] * 4\n + [{\"prompt\": \"Flat is better than\"}] * 4\n + [{\"prompt\": \"Sparse is better\"}] * 4\n )\n expected_second_generation_batch = (\n [{\"prompt\": \"Readability\"}] * 4\n + [{\"prompt\": \"Special cases aren't special\"}] * 4\n + [{\"prompt\": \"Although practicality beats\"}] * 4\n + [{\"prompt\": \"Errors should never\"}] * 4\n + [{\"prompt\": \"Unless explicitly\"}] * 4\n + [{\"prompt\": \"In the face of ambiguity, refuse\"}] * 4\n )\n\n with patch.object(RLOOTrainer, \"training_step\", wraps=trainer.training_step) as mock_prepare:\n trainer.train()\n # 3 epochs * 2 iterations * 2 generation batches to cover the dataset * 4 steps_per_generation\n assert mock_prepare.call_count == 48\n for i in range(0, 8): # Generation batch repeated 8 times (steps_per_generation*num_iterations)\n assert mock_prepare.call_args_list[i].args[1] == expected_first_generation_batch\n for i in range(8, 16):\n assert mock_prepare.call_args_list[i].args[1] == expected_second_generation_batch\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n \"trl-internal-testing/tiny-LlavaNextForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2VLForConditionalGeneration\",\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration\",\n marks=pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.2.0\"),\n reason=\"Qwen3.5 models were introduced in transformers-5.2.0\",\n ),\n ),\n # \"trl-internal-testing/tiny-SmolVLMForConditionalGeneration\", seems not to support bf16 properly\n ],\n )\n @require_vision\n def test_training_vlm(self, model_id):\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n # Because of the way the tiny models are initialized, the gradient does not flow properly through the\n # vision parts of the model, so we skip them. Ideally, we should fix the init of these models.\n params_to_skip = (\n \"model.vision_tower.\",\n \"model.multi_modal_projector.\",\n \"model.visual.\",\n \"model.image_newline\",\n )\n for n, param in previous_trainable_params.items():\n if n.startswith(params_to_skip):\n continue\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @require_vision\n def test_training_vlm_with_pad_to_multiple_of(self):\n # Models like Gemma3 use other forward keyword arguments like token_type_ids that also need to be padded when\n # using pad_to_multiple_of, so we test that the trainer correctly pads all the necessary inputs in this case.\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n pad_to_multiple_of=7,\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @require_vision\n def test_training_vlm_beta_non_zero(self, model_id):\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n beta=0.1, # set beta to non-zero value to test the case where the reference model is used\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n # Because of the way the tiny models are initialized, the gradient does not flow properly through the\n # vision parts of the model, so we skip them. Ideally, we should fix the init of these models.\n params_to_skip = (\"model.visual.\",)\n for n, param in previous_trainable_params.items():\n if n.startswith(params_to_skip):\n continue\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @require_vision\n @require_peft\n def test_training_vlm_peft(self, model_id):\n model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=model,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(target_modules=[\"q_proj\", \"v_proj\"]),\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model params to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed.\"\n elif \"base_layer\" not in n: # We expect the peft params to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n ],\n )\n @require_vision\n @require_vllm\n @pytest.mark.skip(reason=\"We should add a mock for the vLLM server.\")\n def test_training_vlm_and_vllm(self, model_id) -> None:\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n use_vllm=True,\n vllm_mode=\"server\",\n )\n trainer = RLOOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @require_vision\n def test_training_vlm_multi_image(self, model_id):\n dataset = load_dataset(\"trl-internal-testing/zen-multi-image\", \"conversational_prompt_only\", split=\"train\")\n\n def reward_func(completions, **kwargs):\n \"\"\"Reward function that rewards longer completions.\"\"\"\n return [float(len(completion[0][\"content\"])) for completion in completions]\n\n training_args = RLOOConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3, # reduce the batch size to reduce memory usage\n num_generations=3, # reduce the number of generations to reduce memory usage\n max_completion_length=8, # reduce the completion length to reduce memory usage\n report_to=\"none\",\n )\n trainer = RLOOTrainer(\n model=model_id,\n reward_funcs=reward_func,\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_training_with_chat_template_kwargs(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_only\", split=\"train\")\n\n training_args = RLOOConfig(\n bf16=False,\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n per_device_train_batch_size=3,\n num_generations=3,\n max_completion_length=8,\n report_to=\"none\",\n chat_template_kwargs={\"enable_thinking\": False},\n )\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen3ForCausalLM\",\n reward_funcs=\"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n args=training_args,\n train_dataset=dataset,\n )\n\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check that the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.equal(param, new_param), f\"Parameter {n} has not changed.\"\n\n def test_mismatched_reward_processing_classes_length(self):\n \"\"\"Test that mismatched length between reward_funcs and reward_processing_classes raises error.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Use two reward models\n reward_models = [\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n \"trl-internal-testing/tiny-Qwen3ForSequenceClassification\",\n ]\n\n # Create a single processing class (tokenizer)\n single_processing_class = AutoTokenizer.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n )\n\n training_args = RLOOConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n with pytest.raises(ValueError, match=\"must match\"):\n RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_models,\n reward_processing_classes=single_processing_class, # only one, but need two\n args=training_args,\n train_dataset=dataset,\n )\n\n def test_correct_reward_processing_classes_list(self):\n \"\"\"Test that correct list of reward_processing_classes works properly.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Use two reward models\n reward_models = [\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\",\n \"trl-internal-testing/tiny-Qwen3ForSequenceClassification\",\n ]\n\n # Create processing classes\n processing_class1 = AutoTokenizer.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n )\n processing_class2 = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen3ForSequenceClassification\")\n\n training_args = RLOOConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n # Correct list length should work\n correct_processing_classes = [processing_class1, processing_class2]\n\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_models,\n reward_processing_classes=correct_processing_classes,\n args=training_args,\n train_dataset=dataset,\n )\n\n assert len(trainer.reward_processing_classes) == len(reward_models)\n\n def test_single_reward_model_with_single_processing_class(self):\n \"\"\"Test that single reward model with single processing class works.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_prompt_only\", split=\"train\")\n\n # Use single reward model\n reward_model = \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n\n # Create a single processing class (tokenizer)\n single_processing_class = AutoTokenizer.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5\"\n )\n\n training_args = RLOOConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n trainer = RLOOTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n reward_funcs=reward_model,\n reward_processing_classes=single_processing_class, # single object for single reward model\n args=training_args,\n train_dataset=dataset,\n )\n\n assert len(trainer.reward_processing_classes) == 1\n assert trainer.reward_processing_classes[0] == single_processing_class\n" }, { "path": "tests/test_sft_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport gc\nimport json\nimport pathlib\nfrom unittest.mock import MagicMock, patch\n\nimport pytest\nimport torch\nimport transformers\nfrom accelerate.utils.memory import release_memory\nfrom datasets import load_dataset\nfrom packaging.version import Version\nfrom packaging.version import parse as parse_version\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments\nfrom transformers.testing_utils import backend_empty_cache, torch_device\nfrom transformers.utils import is_peft_available\n\nfrom trl import SFTConfig, SFTTrainer\nfrom trl.trainer.sft_trainer import DataCollatorForLanguageModeling, dft_loss\n\nfrom .testing_utils import (\n TrlTestCase,\n ignore_warnings,\n require_ampere_or_newer,\n require_bitsandbytes,\n require_kernels,\n require_liger_kernel,\n require_peft,\n require_torch_accelerator,\n require_torch_multi_accelerator,\n require_vision,\n)\n\n\nif is_peft_available():\n import peft\n from peft import (\n LoraConfig,\n PeftModel,\n PrefixTuningConfig,\n PromptEncoderConfig,\n PromptTuningConfig,\n TaskType,\n get_peft_model,\n )\n\n\nclass TestDFTLoss(TrlTestCase):\n def test_dft_loss(self):\n batch_size = 2\n seq_len = 3\n vocab_size = 2\n # All tokens have the same probability\n logits = torch.fill(torch.empty(batch_size, seq_len, vocab_size), torch.rand(1).item())\n outputs = MagicMock()\n outputs.logits = logits\n labels = torch.tensor([[1, 0, 0], [0, 1, -100]])\n ce_loss = torch.nn.functional.cross_entropy(\n logits.view(-1, vocab_size), labels.view(-1), ignore_index=-100, reduction=\"mean\"\n )\n # We need to account for the logits shift operation so we don't consider the first tokens\n # in each row of the batch\n num_items_in_batch = 3\n # Dft loss\n predicted_dft_loss = dft_loss(outputs, labels, num_items_in_batch)\n # If we have just two tokens in our vocab and all logits are the same,\n # dft scales the ce_loss per token by 0.5. So the dft_loss should be ce_loss/2\n torch.testing.assert_close(ce_loss / 2.0, predicted_dft_loss, atol=1e-4, rtol=1e-4)\n\n\nclass TestDataCollatorForLanguageModeling(TrlTestCase):\n def test_basic_padding(self):\n \"\"\"Test basic padding functionality without completion masks.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0)\n examples = [{\"input_ids\": [1, 2, 3]}, {\"input_ids\": [4, 5]}]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"attention_mask\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3], [4, 5, 0]]))\n torch.testing.assert_close(result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[1, 2, 3], [4, 5, -100]]))\n\n def test_completion_mask(self):\n \"\"\"Test completion mask functionality.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0)\n examples = [\n {\"input_ids\": [1, 2, 3], \"completion_mask\": [0, 1, 1]},\n {\"input_ids\": [4, 5], \"completion_mask\": [0, 1]},\n ]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"attention_mask\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3], [4, 5, 0]]))\n torch.testing.assert_close(result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[-100, 2, 3], [-100, 5, -100]]))\n\n def test_completion_only_loss_disabled(self):\n \"\"\"Test behavior when completion_only_loss is disabled.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0, completion_only_loss=False)\n examples = [\n {\"input_ids\": [1, 2, 3], \"completion_mask\": [0, 1, 1]},\n {\"input_ids\": [4, 5], \"completion_mask\": [0, 1]},\n ]\n\n result = collator(examples)\n\n # Labels should not be masked when completion_only_loss=False\n assert set(result.keys()) == {\"input_ids\", \"attention_mask\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3], [4, 5, 0]]))\n torch.testing.assert_close(result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[1, 2, 3], [4, 5, -100]]))\n\n def test_padding_free_mode(self):\n \"\"\"Test padding-free mode where sequences are concatenated.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0, padding_free=True)\n examples = [{\"input_ids\": [1, 2, 3]}, {\"input_ids\": [4, 5]}]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"position_ids\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3, 4, 5]]))\n torch.testing.assert_close(result[\"position_ids\"], torch.tensor([[0, 1, 2, 0, 1]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[-100, 2, 3, -100, 5]]))\n\n def test_padding_free_with_completion_mask(self):\n \"\"\"Test padding-free mode with completion masks.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0, padding_free=True)\n examples = [\n {\"input_ids\": [1, 2, 3], \"completion_mask\": [0, 0, 1]},\n {\"input_ids\": [4, 5], \"completion_mask\": [1, 1]},\n ]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"position_ids\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3, 4, 5]]))\n torch.testing.assert_close(result[\"position_ids\"], torch.tensor([[0, 1, 2, 0, 1]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[-100, -100, 3, -100, 5]]))\n\n def test_packing(self):\n \"\"\"Test that when using packing with position_ids, attention_mask is dropped with fa2.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0, padding_free=True)\n\n # Simulate packed sequences with position_ids that restart (typical of BFD packing)\n examples = [\n {\"input_ids\": [1, 2, 3, 4, 5, 6], \"seq_lengths\": [3, 3]},\n {\"input_ids\": [7, 8, 9, 10, 11], \"seq_lengths\": [4, 1]},\n ]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"position_ids\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]]))\n torch.testing.assert_close(result[\"position_ids\"], torch.tensor([[0, 1, 2, 0, 1, 2, 0, 1, 2, 3, 0]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[-100, 2, 3, -100, 5, 6, -100, 8, 9, 10, -100]]))\n\n def test_pad_to_multiple_of(self):\n \"\"\"Test padding to multiple of specified value.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0, pad_to_multiple_of=4)\n examples = [{\"input_ids\": [1, 2, 3]}, {\"input_ids\": [4, 5]}]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"attention_mask\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3, 0], [4, 5, 0, 0]]))\n torch.testing.assert_close(result[\"attention_mask\"], torch.tensor([[1, 1, 1, 0], [1, 1, 0, 0]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[1, 2, 3, -100], [4, 5, -100, -100]]))\n\n def test_pad_to_multiple_of_and_padding_free(self):\n \"\"\"Test padding to multiple of specified value.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0, padding_free=True, pad_to_multiple_of=4)\n examples = [{\"input_ids\": [1, 2, 3]}, {\"input_ids\": [4, 5]}]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"position_ids\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3, 4, 5, 0, 0, 0]]))\n torch.testing.assert_close(result[\"position_ids\"], torch.tensor([[0, 1, 2, 0, 1, 0, 0, 0]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[-100, 2, 3, -100, 5, -100, -100, -100]]))\n\n def test_custom_position_ids_but_no_padding_free(self):\n \"\"\"Test that custom position_ids are ignored if padding_free is False.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0)\n examples = [{\"input_ids\": [1, 2, 3], \"seq_lengths\": [1, 2]}, {\"input_ids\": [4, 5], \"seq_lengths\": [2]}]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"attention_mask\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3], [4, 5, 0]]))\n torch.testing.assert_close(result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[1, 2, 3], [4, 5, -100]]))\n\n def test_single_example(self):\n \"\"\"Test collator with a single example.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0)\n examples = [{\"input_ids\": [1, 2, 3, 4]}]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"attention_mask\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3, 4]]))\n torch.testing.assert_close(result[\"attention_mask\"], torch.tensor([[1, 1, 1, 1]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[1, 2, 3, 4]]))\n\n def test_different_pad_token_id(self):\n \"\"\"Test with different pad token ID.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=999)\n examples = [{\"input_ids\": [1, 2, 3]}, {\"input_ids\": [4, 5]}]\n\n result = collator(examples)\n\n assert set(result.keys()) == {\"input_ids\", \"attention_mask\", \"labels\"}\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3], [4, 5, 999]]))\n torch.testing.assert_close(result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[1, 2, 3], [4, 5, -100]]))\n\n def test_assistant_masks(self):\n \"\"\"Test handling of assistant masks in examples.\"\"\"\n collator = DataCollatorForLanguageModeling(pad_token_id=0)\n examples = [\n {\"input_ids\": [1, 2, 3], \"assistant_masks\": [0, 1, 1]},\n {\"input_ids\": [4, 5], \"assistant_masks\": [0, 1]},\n ]\n\n result = collator(examples)\n\n torch.testing.assert_close(result[\"input_ids\"], torch.tensor([[1, 2, 3], [4, 5, 0]]))\n torch.testing.assert_close(result[\"attention_mask\"], torch.tensor([[1, 1, 1], [1, 1, 0]]))\n torch.testing.assert_close(result[\"labels\"], torch.tensor([[-100, 2, 3], [-100, 5, -100]]))\n\n def test_single_example_single_doc(self):\n batch_seq_lengths = [[5]]\n result = DataCollatorForLanguageModeling.get_position_ids_from_packed_seq_lengths(batch_seq_lengths)\n assert len(result) == 1\n assert torch.equal(result[0], torch.arange(5))\n\n def test_single_example_multiple_docs(self):\n batch_seq_lengths = [[3, 2]]\n result = DataCollatorForLanguageModeling.get_position_ids_from_packed_seq_lengths(batch_seq_lengths)\n assert len(result) == 1\n # First sequence: 0, 1, 2; second sequence: 0, 1\n assert torch.equal(result[0], torch.tensor([0, 1, 2, 0, 1]))\n\n def test_multiple_examples(self):\n batch_seq_lengths = [[2, 2], [3]]\n result = DataCollatorForLanguageModeling.get_position_ids_from_packed_seq_lengths(batch_seq_lengths)\n assert len(result) == 2\n assert torch.equal(result[0], torch.tensor([0, 1, 0, 1]))\n assert torch.equal(result[1], torch.arange(3))\n\n\nclass TestSFTTrainer(TrlTestCase):\n def test_init_with_training_arguments(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n args = TrainingArguments(output_dir=self.tmp_dir, report_to=\"none\")\n SFTTrainer(model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=args, train_dataset=dataset)\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Cohere2ForCausalLM\",\n pytest.param(\n \"trl-internal-testing/tiny-Glm4MoeForCausalLM\",\n marks=pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.0.0\"),\n reason=\"GLM4 tokenizer requires transformers>=5.0.0\",\n ),\n ),\n \"trl-internal-testing/tiny-GptOssForCausalLM\",\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"trl-internal-testing/tiny-Qwen3MoeForCausalLM\",\n ],\n )\n def test_train(self, model_id):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(model=model_id, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n # Special case for harmony\n def test_train_gpt_oss(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/harmony\", \"language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-GptOssForCausalLM\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_model(self):\n # Instantiate the model\n model = AutoModelForCausalLM.from_pretrained(\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n dtype=\"float32\",\n )\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(model=model, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_dft_loss(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n loss_type=\"dft\",\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n eval_strategy=\"steps\",\n eval_steps=3,\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_moe_model_with_aux_loss(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n report_to=\"none\",\n model_init_kwargs={\"output_router_logits\": True},\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen3MoeForCausalLM\", args=training_args, train_dataset=dataset\n )\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss and aux loss are not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert trainer.state.log_history[-1][\"aux_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_formatting_func(self):\n # Dummy formatting function\n def formatting_prompts_func(example):\n chosen, rejected = example[\"chosen\"], example[\"rejected\"]\n return f\"### Chosen: {chosen}\\n### Rejected: {rejected}\"\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_implicit_prompt_preference\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n formatting_func=formatting_prompts_func,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_model_dtype(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n model_init_kwargs={\"dtype\": torch.float16},\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n report_to=\"none\",\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n # For some reasonn model.layers.0.input_layernorm.weight doesn't change in GitHub Actions but does\n # locally. We ignore this parameter for now\n if \"layernorm\" in n:\n continue\n new_param = trainer.model.get_parameter(n)\n # Check the torch dtype\n assert new_param.dtype == torch.float16\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_peft\n def test_train_dense_with_peft_config_lora(self):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n trainer = SFTTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\n \"peft_type\",\n [\n \"prompt_tuning\",\n \"prefix_tuning\",\n \"prompt_encoder\",\n ],\n )\n @require_peft\n def test_train_with_peft_config_prompt_tuning(self, peft_type):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer, p-tuning doesn't support gradient checkpointing\n training_args = SFTConfig(bf16=False, output_dir=self.tmp_dir, report_to=\"none\", gradient_checkpointing=False)\n if peft_type == \"prompt_tuning\":\n peft_config = PromptTuningConfig(\n task_type=TaskType.CAUSAL_LM,\n num_virtual_tokens=4,\n tokenizer_name_or_path=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n )\n elif peft_type == \"prefix_tuning\":\n if parse_version(peft.__version__) <= Version(\"0.17.1\"):\n pytest.xfail(\n \"Prefix tuning with device_map='auto' is broken in peft 0.17.1 and below. See \"\n \"https://github.com/huggingface/peft/issues/2821\"\n )\n peft_config = PrefixTuningConfig(\n task_type=TaskType.CAUSAL_LM,\n num_virtual_tokens=4,\n )\n elif peft_type == \"prompt_encoder\":\n peft_config = PromptEncoderConfig(\n task_type=TaskType.CAUSAL_LM,\n num_virtual_tokens=4,\n encoder_hidden_size=model.config.hidden_size, # This will be overwritten below\n )\n trainer = SFTTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=peft_config,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n else: # We expect the peft parameters to be different\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_peft\n def test_train_moe_with_peft_config(self):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-GptOssForCausalLM\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n trainer = SFTTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(target_parameters=[\"mlp.experts.down_proj\", \"mlp.experts.gate_up_proj\"]),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_peft\n def test_train_peft_model(self):\n # Get the base model\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n\n # Get the base model parameter names\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Turn the model into a peft model\n lora_config = LoraConfig()\n model = get_peft_model(model, lora_config)\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(model=model, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n # In practice, this test is the same as `test_train_dense_with_peft_config_lora`, since gradient checkpointing is\n # enabled by default in `SFTTrainer`. We keep it as a regression guard: if the default ever changes, we still\n # explicitly test PEFT + gradient checkpointing, which has caused issues in the past.\n @require_peft\n def test_train_with_peft_config_and_gradient_checkpointing(self):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, gradient_checkpointing=True, report_to=\"none\")\n\n trainer = SFTTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\"use_reentrant\", [True, False])\n @require_peft\n def test_train_with_peft_config_and_gradient_checkpointing_reentrant(self, use_reentrant):\n # Get the base model parameter names\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"float32\")\n base_param_names = [f\"base_model.model.{n}\" for n, _ in model.named_parameters()]\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n gradient_checkpointing=True,\n gradient_checkpointing_kwargs={\"use_reentrant\": use_reentrant},\n report_to=\"none\",\n )\n\n trainer = SFTTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n in base_param_names: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"base_layer\" not in n: # We expect the peft parameters to be different (except for the base layer)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_liger_kernel\n def test_train_with_liger(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, use_liger_kernel=True, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_torch_accelerator\n @require_liger_kernel\n def test_compute_loss_skip_logits_on_eval_without_metrics_with_liger(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train[:1]\")\n\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n use_liger_kernel=False,\n report_to=\"none\",\n max_length=8,\n bf16=False,\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n compute_metrics=None,\n )\n trainer.args.use_liger_kernel = True\n trainer.model.eval()\n\n captured = {}\n\n def mock_super_compute_loss(model, inputs, return_outputs=False, num_items_in_batch=None):\n captured[\"skip_logits\"] = inputs.get(\"skip_logits\")\n dummy_loss = torch.tensor(1.0, requires_grad=True)\n dummy_outputs = MagicMock()\n dummy_outputs.token_accuracy = None\n dummy_outputs.logits = torch.randn(1, 5, trainer.model.config.vocab_size)\n return (dummy_loss, dummy_outputs)\n\n inputs = {\n \"input_ids\": torch.tensor([[1, 2, 3, 4, 5]]),\n \"labels\": torch.tensor([[1, 2, 3, 4, 5]]),\n \"attention_mask\": torch.tensor([[1, 1, 1, 1, 1]]),\n }\n\n with patch(\"transformers.Trainer.compute_loss\", side_effect=mock_super_compute_loss):\n trainer.compute_loss(trainer.model, inputs)\n\n assert captured[\"skip_logits\"] is True\n\n @require_torch_accelerator\n @require_liger_kernel\n def test_predict_does_not_skip_logits_with_liger(self):\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train[:1]\")\n\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n use_liger_kernel=False,\n report_to=\"none\",\n max_length=8,\n bf16=False,\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n compute_metrics=None,\n )\n trainer.args.use_liger_kernel = True\n trainer.model.eval()\n\n captured = {}\n\n def mock_super_compute_loss(model, inputs, return_outputs=False, num_items_in_batch=None):\n captured[\"skip_logits\"] = inputs.get(\"skip_logits\")\n dummy_loss = torch.tensor(1.0, requires_grad=True)\n dummy_outputs = (dummy_loss, torch.randn(1, 5, trainer.model.config.vocab_size))\n return (dummy_loss, dummy_outputs)\n\n with patch(\"transformers.Trainer.compute_loss\", side_effect=mock_super_compute_loss):\n trainer.predict(trainer.train_dataset)\n\n assert captured[\"skip_logits\"] is False\n\n def test_train_with_non_chatml_conversational_data(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\", split=\"train\")\n\n # Rename role/content to from/value to ensure SFT works with non-chatML conversational data\n def rename_fields(example: list[dict]):\n return {\"conversations\": [{\"from\": m[\"role\"], \"value\": m[\"content\"]} for m in example[\"messages\"]]}\n\n dataset = dataset.map(rename_fields, remove_columns=\"messages\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_pretokenized_data(self):\n # Get the dataset\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n def tokenize_example(example):\n return tokenizer(example[\"text\"])\n\n # Apply tokenization\n tokenized_dataset = dataset.map(tokenize_example, remove_columns=[\"text\"])\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(model=model_id, args=training_args, train_dataset=tokenized_dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_iterable_dataset(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\", streaming=True)\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, max_steps=3, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @require_kernels\n @require_ampere_or_newer # Flash attention 2 requires Ampere or newer GPUs\n def test_train_padding_free(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n padding_free=True,\n model_init_kwargs={\"attn_implementation\": \"kernels-community/flash-attn2\"},\n bf16=True, # flash_attention_2 only supports bf16 and fp16\n report_to=\"none\",\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\"packing_strategy\", [\"bfd\", \"wrapped\"])\n @ignore_warnings(message=\"You are using packing, but the attention implementation is not.*\", category=UserWarning)\n @ignore_warnings(message=\"Padding-free training is enabled, but the attention.*\", category=UserWarning)\n def test_train_packing(self, packing_strategy):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir, packing=True, packing_strategy=packing_strategy, max_length=10, report_to=\"none\"\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @ignore_warnings(message=\"You are using packing, but the attention implementation is not.*\", category=UserWarning)\n @ignore_warnings(message=\"Padding-free training is enabled, but the attention.*\", category=UserWarning)\n def test_eval_packing(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n packing=True,\n max_length=64,\n report_to=\"none\",\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n # Check the number of sequences in train and eval datasets\n num_train_seqs = sum(len(x) for x in trainer.train_dataset[\"seq_lengths\"])\n num_eval_seqs = sum(len(x) for x in trainer.eval_dataset[\"seq_lengths\"])\n assert num_train_seqs == 17 # we should still have 17 seqs\n assert num_eval_seqs == 2 # we should still have 2 seqs\n\n # Check that all sequences are shorter than the max length\n assert all(sum(x) <= 64 for x in trainer.train_dataset[\"seq_lengths\"])\n assert all(sum(x) <= 64 for x in trainer.eval_dataset[\"seq_lengths\"])\n\n # Check the number of sequences in train and eval datasets\n assert len(trainer.train_dataset[\"input_ids\"]) == 3 # w/ this dataset, we end up with 46 seqs\n assert len(trainer.eval_dataset[\"input_ids\"]) == 1 # w/ this dataset, we end up with 6 seqs\n\n @ignore_warnings(message=\"You are using packing, but the attention implementation is not.*\", category=UserWarning)\n @ignore_warnings(message=\"Padding-free training is enabled, but the attention.*\", category=UserWarning)\n def test_only_train_packing(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n packing=True,\n eval_packing=False,\n max_length=64,\n report_to=\"none\",\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n # Check the number of sequences in train dataset\n num_train_seqs = sum(len(x) for x in trainer.train_dataset[\"seq_lengths\"])\n assert num_train_seqs == 17 # we should still have 17 seqs\n\n # We expect eval dataset not having \"seq_lengths\" as eval_packing is False\n assert \"seq_lengths\" not in trainer.eval_dataset\n\n # Check that all sequences are shorter than the max length\n assert all(sum(x) <= 64 for x in trainer.train_dataset[\"seq_lengths\"])\n\n # Check the number of sequences in train and eval datasets\n assert len(trainer.train_dataset[\"input_ids\"]) == 3 # w/ this dataset, we end up with 46 seqs\n assert len(trainer.eval_dataset[\"input_ids\"]) == 2 # w/ this dataset, we end up with 6 seqs\n\n def test_train_with_chat_template_kwargs(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n\n tokenizer = AutoTokenizer.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\")\n # The following template is a simplified version of the Qwen chat template, where an additional argument\n # `role_capital` is used to control the capitalization of roles.\n tokenizer.chat_template = '{%- if messages[0][\"role\"] == \"system\" -%} {{ \"<|im_start|>\" + (\"SYSTEM\" if role_capital else \"system\") + \"\\\\n\" + messages[0][\"content\"] + \"<|im_end|>\\\\n\" }}{%- else -%} {{ \"<|im_start|>\" + (\"SYSTEM\" if role_capital else \"system\") + \"\\\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\\\n\" }}{%- endif -%}{%- for message in messages -%} {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) -%} {{ \"<|im_start|>\" + (message.role.upper() if role_capital else message.role) + \"\\\\n\" + message.content + \"<|im_end|>\\\\n\" }} {%- elif message.role == \"assistant\" -%} {{ \"<|im_start|>\" + (\"ASSISTANT\" if role_capital else \"assistant\") }} {%- if message.content -%} {{ \"\\\\n\" + message.content }} {%- endif -%} {{ \"<|im_end|>\\\\n\" }} {%- elif message.role == \"tool\" -%} {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") -%} {{ \"<|im_start|>\" + (\"USER\" if role_capital else \"user\") }} {%- endif -%} {{ \"\\\\n\\\\n\" + message.content + \"\\\\n\" }} {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") -%} {{ \"<|im_end|>\\\\n\" }} {%- endif -%} {%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%} {{ \"<|im_start|>\" + (\"ASSISTANT\" if role_capital else \"assistant\") + \"\\\\n\" }}{%- endif -%}'\n\n dataset = dataset.add_column(\n \"chat_template_kwargs\", [{\"role_capital\": bool(i % 2)} for i in range(len(dataset))]\n )\n assert \"chat_template_kwargs\" in dataset.features\n\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n processing_class=tokenizer,\n )\n\n # Assert trainer uses the same chat template as tokenizer\n assert trainer.processing_class.chat_template == tokenizer.chat_template\n\n # Assert chat_template is applied\n for i in range(2):\n role = \"SYSTEM\" if i else \"system\"\n system_prompt = (\n f\"<|im_start|>{role}\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\"\n )\n system_prompt_ids = trainer.processing_class(system_prompt)[\"input_ids\"]\n assert trainer.train_dataset[i][\"input_ids\"][: len(system_prompt_ids)] == system_prompt_ids\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_assistant_only(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, assistant_only_loss=True, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen3ForCausalLM\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_completion_only(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_completion\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, completion_only_loss=True, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen3ForCausalLM\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_completion_only_harmony(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/harmony\", \"prompt_completion\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, completion_only_loss=True, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-GptOssForCausalLM\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_assistant_only_and_completion_only(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_prompt_completion\", split=\"train\")\n\n # To test this case, we need to add user messages in the completion (they'll be masked in the loss)\n def add_to_completion(example):\n example[\"completion\"].append(example[\"prompt\"][0])\n example[\"completion\"].append(example[\"completion\"][0])\n return example\n\n dataset = dataset.map(add_to_completion)\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir, assistant_only_loss=True, completion_only_loss=True, report_to=\"none\"\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen3ForCausalLM\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_assistant_only_iterable_dataset(self):\n # Get the dataset\n dataset = load_dataset(\n \"trl-internal-testing/zen\", \"conversational_language_modeling\", split=\"train\", streaming=True\n )\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, assistant_only_loss=True, max_steps=3, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen3ForCausalLM\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_set_chat_template_from_model(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, chat_template_path=\"Qwen/Qwen3-4B\", report_to=\"none\")\n # trl-internal-testing/tiny-GPTNeoXForCausalLM doesn't have a chat template set by default\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-GPTNeoXForCausalLM\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_set_chat_template_from_path(self, lazy_shared_datadir):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"conversational_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n chat_template_path=str(lazy_shared_datadir / \"template.jinja\"),\n report_to=\"none\",\n )\n # trl-internal-testing/tiny-GPTNeoXForCausalLM doesn't have a chat template set by default\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-GPTNeoXForCausalLM\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n # Check that the template saved in the output directory is the same as the one used for training\n template_path = pathlib.Path(self.tmp_dir) / \"checkpoint-9\" / \"chat_template.jinja\"\n assert template_path.exists(), f\"Chat template not found at {template_path}\"\n\n with open(template_path) as f:\n template_content = f.read()\n with open(training_args.chat_template_path) as f:\n original_template_content = f.read()\n assert template_content == original_template_content, \"Chat template content does not match the original\"\n\n def test_train_toolcall_data(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/toolcall\", \"language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_toolcall_data_as_json(self):\n # Tabular backends (Arrow/Parquet) can insert `None` for missing keys in nested structures.\n # If `tools` is stored as a list of dicts and examples use different dict schemas, nulls may\n # be introduced and break tool processing. This test ensures we also support `tools` provided\n # as a list of dicts.\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/toolcall\", \"language_modeling\", split=\"train\")\n\n def convert_to_json(example):\n return {\"tools\": json.loads(example[\"tools\"])}\n\n dataset = dataset.map(convert_to_json)\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_train_with_eval(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, eval_strategy=\"steps\", eval_steps=3, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n )\n\n # Train the model\n trainer.train()\n\n # Check that the eval loss is not None\n assert trainer.state.log_history[0][\"eval_loss\"] is not None\n\n def test_train_with_multiple_eval_dataset(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, eval_strategy=\"steps\", eval_steps=3, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset={\"data1\": dataset[\"test\"], \"data2\": dataset[\"test\"]},\n )\n # Train the model\n trainer.train()\n\n # Check that the eval losses are not None\n assert trainer.state.log_history[-3][\"eval_data1_loss\"] is not None\n assert trainer.state.log_history[-2][\"eval_data2_loss\"] is not None\n\n def test_train_with_compute_metrics(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\")\n\n def dummy_compute_metrics(eval_pred):\n return {\"my_metric\": 0.123}\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n eval_strategy=\"steps\",\n eval_steps=3,\n report_to=\"none\",\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"test\"],\n compute_metrics=dummy_compute_metrics,\n )\n\n # Train the model\n trainer.train()\n\n # Check that the custom metric is logged\n assert trainer.state.log_history[-2][\"eval_my_metric\"] == 0.123\n\n # In practice, this test is the same as `test_train`, since gradient checkpointing is enabled by default in\n # `SFTTrainer`. We keep it as a regression guard: if the default ever changes, we still explicitly test gradient\n # checkpointing, which has caused issues in the past.\n def test_train_with_gradient_checkpointing(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, gradient_checkpointing=True, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n @pytest.mark.parametrize(\"use_reentrant\", [True, False])\n def test_train_with_gradient_checkpointing_reentrant(self, use_reentrant):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n gradient_checkpointing=True,\n gradient_checkpointing_kwargs={\"use_reentrant\": use_reentrant},\n report_to=\"none\",\n )\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", args=training_args, train_dataset=dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n def test_tag_added(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n train_dataset=dataset,\n )\n\n for tag in [\"sft\", \"trl\"]:\n assert tag in trainer.model.model_tags\n\n @require_peft\n def test_tag_added_peft(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n train_dataset=dataset,\n peft_config=LoraConfig(),\n )\n\n for tag in [\"sft\", \"trl\"]:\n assert tag in trainer.model.model_tags\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n # \"trl-internal-testing/tiny-Idefics2ForConditionalGeneration\", high memory peak, skipped for now\n # \"trl-internal-testing/tiny-Idefics3ForConditionalGeneration\", high memory peak, skipped for now\n \"trl-internal-testing/tiny-LlavaForConditionalGeneration\",\n \"trl-internal-testing/tiny-LlavaNextForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2VLForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n # \"trl-internal-testing/tiny-SmolVLMForConditionalGeneration\", seems not to support bf16 properly\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3VLForConditionalGeneration\",\n marks=[\n pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"4.57.0\"),\n reason=\"Qwen3-VL series were introduced in transformers-4.57.0\",\n ),\n pytest.mark.xfail(\n Version(\"5.0.0\") <= Version(transformers.__version__) < Version(\"5.1.0\"),\n reason=\"Upstream transformers bug (transformers#43334) in 5.0.x; fixed in 5.1.0\",\n ),\n ],\n ),\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration\",\n marks=pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.2.0\"),\n reason=\"Qwen3.5 models were introduced in transformers-5.2.0\",\n ),\n ),\n ],\n )\n @require_vision\n def test_train_vlm(self, model_id):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n max_length=None, # for VLMs, truncating can remove image tokens, leading to errors\n report_to=\"none\",\n )\n trainer = SFTTrainer(model=model_id, args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n # For some reason, these params are not updated. This is probably not related to TRL, but to\n # the model itself. We should investigate this further, but for now we just skip these params.\n # fmt: off\n if (\n model_id == \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\" and \"model.vision_tower.vision_model.head\" in n or\n model_id == \"trl-internal-testing/tiny-LlavaForConditionalGeneration\" and \"model.vision_tower.vision_model.post_layernorm\" in n or\n model_id == \"trl-internal-testing/tiny-LlavaForConditionalGeneration\" and \"vision_tower.vision_model.encoder.layers.1\" in n or\n model_id == \"trl-internal-testing/tiny-LlavaNextForConditionalGeneration\" and \"model.vision_tower.vision_model.post_layernorm\" in n or\n model_id == \"trl-internal-testing/tiny-LlavaNextForConditionalGeneration\" and \"vision_tower.vision_model.encoder.layers.1\" in n or\n model_id == \"trl-internal-testing/tiny-Qwen3VLForConditionalGeneration\" and \"model.visual.deepstack_merger_list\" in n\n ):\n # fmt: on\n continue\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is not updated\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @pytest.mark.xfail(\n parse_version(transformers.__version__) < parse_version(\"4.57.0\"),\n reason=\"Mixing text-only and image+text examples is only supported in transformers >= 4.57.0\",\n strict=False,\n )\n @require_vision\n def test_train_vlm_multi_image(self, model_id):\n # Get the dataset\n dataset = load_dataset(\n \"trl-internal-testing/zen-multi-image\", \"conversational_prompt_completion\", split=\"train\"\n )\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n max_length=None, # for VLMs, truncating can remove image tokens, leading to errors\n report_to=\"none\",\n )\n trainer = SFTTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is not updated\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n # Special case for Gemma, as it uses token_type_ids, and we need to ensure they are properly in the collator:\n \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n ],\n )\n @require_vision\n def test_train_vlm_prompt_completion(self, model_id):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_prompt_completion\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n max_length=None, # for VLMs, truncating can remove image tokens, leading to errors\n report_to=\"none\",\n )\n trainer = SFTTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is not updated\"\n\n # Gemma 3n uses a timm encoder, making it difficult to create a smaller variant for testing.\n # To ensure coverage, we run tests on the full model but mark them as slow to exclude from default runs.\n @pytest.mark.slow\n @require_vision\n @pytest.mark.skip(reason=\"Model google/gemma-3n-E2B-it is gated and requires HF token\")\n def test_train_vlm_gemma_3n(self):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen-image\", \"conversational_language_modeling\", split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n learning_rate=0.1, # use higher lr because gradients are tiny and default lr can stall updates\n max_length=None, # for VLMs, truncating can remove image tokens, leading to errors\n per_device_train_batch_size=1, # VLM training is memory intensive, reduce batch size to avoid OOM\n model_init_kwargs={\"dtype\": \"bfloat16\"},\n report_to=\"none\",\n )\n trainer = SFTTrainer(model=\"google/gemma-3n-E2B-it\", args=training_args, train_dataset=dataset)\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if \"model.audio_tower\" in n or \"model.embed_audio\" in n:\n # The audio embedding parameters are not updated because this dataset contains no audio data\n continue\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is not updated\"\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n ],\n )\n @pytest.mark.parametrize(\n \"dataset_config\",\n [\"conversational_language_modeling\", \"conversational_prompt_completion\", \"standard_prompt_completion\"],\n )\n @require_vision\n def test_train_vlm_text_only_data(self, model_id, dataset_config):\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", dataset_config, split=\"train\")\n\n # Initialize the trainer\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(\n model=model_id,\n args=training_args,\n train_dataset=dataset,\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if n.startswith(\"model.visual\"):\n torch.testing.assert_close(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is updated\"\n else:\n assert not torch.allclose(param, new_param, rtol=1e-12, atol=1e-12), f\"Param {n} is not updated\"\n\n @require_peft\n def test_prompt_tuning(self):\n \"\"\"Test that SFT works with Prompt Tuning.\"\"\"\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(\n model=\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n args=training_args,\n train_dataset=dataset,\n peft_config=PromptEncoderConfig(task_type=TaskType.CAUSAL_LM, num_virtual_tokens=8),\n )\n\n # Save initial parameters to check they change during training\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n # Check that training completed successfully\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert trainer.state.log_history[-1][\"mean_token_accuracy\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if \"base_model\" in n: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"prompt_encoder\" in n: # We expect the peft parameters to be different\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n else:\n raise ValueError(f\"Unexpected parameter {n} in model: {trainer.model}\")\n\n @require_peft\n @require_bitsandbytes\n def test_peft_with_quantization(self):\n # Get the base model\n model_id = \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\"\n\n quantization_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_compute_dtype=torch.float16,\n )\n model = AutoModelForCausalLM.from_pretrained(\n model_id,\n dtype=\"float32\",\n quantization_config=quantization_config,\n )\n\n # Get the dataset\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n # Initialize the trainer with the already configured PeftModel\n training_args = SFTConfig(output_dir=self.tmp_dir, learning_rate=0.1, report_to=\"none\")\n trainer = SFTTrainer(model=model, args=training_args, train_dataset=dataset, peft_config=LoraConfig())\n\n # Save initial parameters to check they change during training\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n # Check that training completed successfully\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert trainer.state.log_history[-1][\"mean_token_accuracy\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n # In bitsandbytes, bias parameters are automatically cast to the input dtype during the forward pass if\n # their dtype doesnโ€™t match. This causes the module to change unexpectedly during the first forward pass of\n # the training. To handle this, we cast these specific bias parameters to float32 before comparison.\n # https://github.com/bitsandbytes-foundation/bitsandbytes/blob/45553f7392e524eacf400b132cfe01261f6477be/bitsandbytes/nn/modules.py#L518\n # We still need to investigate why the compute dtype ends up being different than for these parameters.\n if n in [\n \"base_model.model.model.layers.1.self_attn.k_proj.bias\",\n \"base_model.model.model.layers.1.self_attn.q_proj.base_layer.bias\",\n \"base_model.model.model.layers.1.self_attn.v_proj.base_layer.bias\",\n ]:\n param = param.float()\n\n if \"lora\" not in n: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"lora\" in n: # We expect the peft parameters to be different\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n else:\n raise ValueError(f\"Unexpected parameter {n} in model: {trainer.model}\")\n\n @require_peft\n def test_prompt_tuning_peft_model(self):\n \"\"\"Test that SFT works with Prompt Tuning and a pre-converted PeftModel\"\"\"\n model = AutoModelForCausalLM.from_pretrained(\"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\", dtype=\"float32\")\n model = get_peft_model(model, PromptEncoderConfig(task_type=TaskType.CAUSAL_LM, num_virtual_tokens=8))\n\n dataset = load_dataset(\"trl-internal-testing/zen\", \"standard_language_modeling\", split=\"train\")\n\n training_args = SFTConfig(output_dir=self.tmp_dir, report_to=\"none\")\n trainer = SFTTrainer(model=model, args=training_args, train_dataset=dataset)\n\n # Save initial parameters to check they change during training\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n trainer.train()\n\n # Check that training completed successfully\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n assert trainer.state.log_history[-1][\"mean_token_accuracy\"] is not None\n\n # Check the peft params have changed and the base model params have not changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n if \"base_model\" in n: # We expect the base model parameters to be the same\n torch.testing.assert_close(param, new_param), f\"Parameter {n} has changed\"\n elif \"prompt_encoder\" in n: # We expect the peft parameters to be different\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n else:\n raise ValueError(f\"Unexpected parameter {n} in model: {trainer.model}\")\n\n\n@pytest.mark.slow\n@require_torch_accelerator\n@require_peft\nclass TestSFTTrainerSlow(TrlTestCase):\n def setup_method(self):\n self.train_dataset = load_dataset(\"stanfordnlp/imdb\", split=\"train[:10%]\")\n self.eval_dataset = load_dataset(\"stanfordnlp/imdb\", split=\"test[:10%]\")\n self.max_length = 128\n self.peft_config = LoraConfig(\n lora_alpha=16,\n lora_dropout=0.1,\n r=8,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n )\n\n def teardown_method(self):\n gc.collect()\n backend_empty_cache(torch_device)\n gc.collect()\n\n @pytest.mark.parametrize(\"packing\", [True, False])\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n ],\n )\n def test_sft_trainer_transformers_mp(self, model_name, packing):\n \"\"\"\n Simply tests if passing a transformers model to `SFTTrainer` loads and runs the trainer as expected in mixed\n precision.\n \"\"\"\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n logging_strategy=\"no\",\n report_to=\"none\",\n per_device_train_batch_size=2,\n max_steps=10,\n fp16=True, # this is sufficient to enable amp\n packing=packing,\n max_length=self.max_length,\n )\n\n model = AutoModelForCausalLM.from_pretrained(model_name, dtype=\"float32\")\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n\n trainer = SFTTrainer(\n model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=self.train_dataset,\n eval_dataset=self.eval_dataset,\n )\n\n trainer.train()\n\n release_memory(model, trainer)\n\n @pytest.mark.parametrize(\"device_map\", [{\"\": 0}, \"auto\"])\n @pytest.mark.parametrize(\n \"gradient_checkpointing_kwargs\", [None, {\"use_reentrant\": False}, {\"use_reentrant\": True}]\n )\n @pytest.mark.parametrize(\"packing\", [True, False])\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n ],\n )\n @require_torch_multi_accelerator\n def test_sft_trainer_transformers_mp_gc_device_map(\n self, model_name, packing, gradient_checkpointing_kwargs, device_map\n ):\n \"\"\"\n Simply tests if passing a transformers model to `SFTTrainer` loads and runs the trainer as expected in mixed\n precision + different scenarios of gradient_checkpointing (single, multi-gpu, etc).\n \"\"\"\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n logging_strategy=\"no\",\n report_to=\"none\",\n per_device_train_batch_size=2,\n max_steps=10,\n packing=packing,\n max_length=self.max_length,\n fp16=True, # this is sufficient to enable amp\n gradient_checkpointing=True, # default, here for clarity\n gradient_checkpointing_kwargs=gradient_checkpointing_kwargs,\n )\n\n model = AutoModelForCausalLM.from_pretrained(model_name, dtype=\"float32\", device_map=device_map)\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n\n trainer = SFTTrainer(\n model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=self.train_dataset,\n eval_dataset=self.eval_dataset,\n )\n\n trainer.train()\n\n release_memory(model, trainer)\n\n @pytest.mark.parametrize(\n \"gradient_checkpointing_kwargs\", [None, {\"use_reentrant\": False}, {\"use_reentrant\": True}]\n )\n @pytest.mark.parametrize(\"packing\", [True, False])\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n ],\n )\n @require_peft\n @require_bitsandbytes\n def test_sft_trainer_transformers_mp_gc_peft_qlora(self, model_name, packing, gradient_checkpointing_kwargs):\n \"\"\"\n Simply tests if passing a transformers model + PEFT + bnb to `SFTTrainer` loads and runs the trainer as\n expected in mixed precision + different scenarios of gradient_checkpointing.\n \"\"\"\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n logging_strategy=\"no\",\n report_to=\"none\",\n per_device_train_batch_size=2,\n max_steps=10,\n packing=packing,\n max_length=self.max_length,\n gradient_checkpointing=True, # default, here for clarity\n gradient_checkpointing_kwargs=gradient_checkpointing_kwargs,\n )\n\n quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)\n\n model = AutoModelForCausalLM.from_pretrained(\n model_name, dtype=\"float32\", quantization_config=quantization_config\n )\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n\n trainer = SFTTrainer(\n model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=self.train_dataset,\n eval_dataset=self.eval_dataset,\n peft_config=self.peft_config,\n )\n\n assert isinstance(trainer.model, PeftModel)\n\n trainer.train()\n\n release_memory(model, trainer)\n\n @pytest.mark.parametrize(\"packing\", [True, False])\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n ],\n )\n @require_peft\n @require_bitsandbytes\n def test_sft_trainer_with_chat_format_qlora(self, model_name, packing):\n \"\"\"\n Simply tests if using setup_chat_format with a transformers model + peft + bnb config to `SFTTrainer` loads and\n runs the trainer as expected.\n \"\"\"\n train_dataset = load_dataset(\"trl-internal-testing/dolly-chatml-sft\", split=\"train\")\n\n training_args = SFTConfig(\n packing=packing,\n max_length=self.max_length,\n output_dir=self.tmp_dir,\n logging_strategy=\"no\",\n report_to=\"none\",\n per_device_train_batch_size=2,\n max_steps=10,\n )\n\n quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)\n\n model = AutoModelForCausalLM.from_pretrained(\n model_name, dtype=\"float32\", quantization_config=quantization_config\n )\n tokenizer = AutoTokenizer.from_pretrained(model_name)\n\n trainer = SFTTrainer(\n model,\n args=training_args,\n processing_class=tokenizer,\n train_dataset=train_dataset,\n peft_config=self.peft_config,\n )\n\n assert isinstance(trainer.model, PeftModel)\n\n trainer.train()\n\n release_memory(model, trainer)\n\n @pytest.mark.parametrize(\"packing\", [True, False])\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n ],\n )\n @require_liger_kernel\n def test_sft_trainer_with_liger(self, model_name, packing):\n \"\"\"\n Tests if passing use_liger=True to SFTConfig loads and runs the trainer with AutoLigerKernelForCausalLM as\n expected.\n \"\"\"\n import importlib\n\n def cleanup_liger_patches(trainer):\n \"\"\"Clean up liger_kernel patches by reloading the model's specific module\"\"\"\n try:\n # Get the specific module that was used by the trainer's model\n module_path = trainer.model.__module__\n reload_module = importlib.import_module(module_path)\n importlib.reload(reload_module)\n except Exception:\n pass # Continue if reload fails\n\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n logging_strategy=\"no\",\n report_to=\"none\",\n per_device_train_batch_size=2,\n max_steps=2,\n packing=packing,\n max_length=self.max_length,\n use_liger_kernel=True,\n )\n\n trainer = SFTTrainer(\n model_name,\n args=training_args,\n train_dataset=self.train_dataset,\n eval_dataset=self.eval_dataset,\n )\n\n # Ensure cleanup of liger patches after the test\n try:\n trainer.train()\n release_memory(trainer.model, trainer)\n finally:\n cleanup_liger_patches(trainer)\n\n @pytest.mark.parametrize(\"packing\", [True, False])\n @pytest.mark.parametrize(\n \"model_name\",\n [\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n ],\n )\n @require_torch_accelerator\n def test_train_offloading(self, model_name, packing):\n \"\"\"Test that activation offloading works with SFTTrainer.\"\"\"\n # Initialize the trainer\n training_args = SFTConfig(\n output_dir=self.tmp_dir,\n activation_offloading=True,\n report_to=\"none\",\n per_device_train_batch_size=2,\n max_steps=2,\n packing=packing,\n max_length=self.max_length,\n )\n trainer = SFTTrainer(\n model=model_name, args=training_args, train_dataset=self.train_dataset, eval_dataset=self.eval_dataset\n )\n\n # Save the initial parameters to compare them later\n previous_trainable_params = {n: param.clone() for n, param in trainer.model.named_parameters()}\n\n # Train the model\n trainer.train()\n\n # Check that the training loss is not None\n assert trainer.state.log_history[-1][\"train_loss\"] is not None\n\n # Check the params have changed\n for n, param in previous_trainable_params.items():\n new_param = trainer.model.get_parameter(n)\n assert not torch.allclose(param, new_param), f\"Parameter {n} has not changed\"\n\n release_memory(trainer.model, trainer)\n" }, { "path": "tests/test_skills.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom pathlib import Path\n\nimport pytest\n\nfrom trl.skills import install_skill, list_agent_names, list_skills, resolve_target_path, uninstall_skill\nfrom trl.skills.skills import _get_trl_skills_dir\n\n\nclass TestGetTrlSkillsDir:\n \"\"\"Tests for _get_trl_skills_dir function.\"\"\"\n\n def test_returns_path_object(self):\n \"\"\"Test that returns a Path object.\"\"\"\n skills_dir = _get_trl_skills_dir()\n assert isinstance(skills_dir, Path)\n\n def test_directory_exists(self):\n \"\"\"Test that the returned directory exists.\"\"\"\n skills_dir = _get_trl_skills_dir()\n assert skills_dir.exists(), f\"Skills directory does not exist: {skills_dir}\"\n\n def test_is_directory(self):\n \"\"\"Test that the returned path is a directory.\"\"\"\n skills_dir = _get_trl_skills_dir()\n assert skills_dir.is_dir(), f\"Skills path is not a directory: {skills_dir}\"\n\n def test_contains_skills_module(self):\n \"\"\"Test that the path ends with 'skills' (the module name).\"\"\"\n skills_dir = _get_trl_skills_dir()\n assert skills_dir.name == \"skills\"\n\n\nclass TestListSkills:\n \"\"\"Tests for list_skills function.\"\"\"\n\n def test_returns_list(self):\n \"\"\"Test that list_skills returns a list.\"\"\"\n skills = list_skills()\n assert isinstance(skills, list)\n\n def test_contains_trl_training(self):\n \"\"\"Test that list_skills includes the trl-training skill.\"\"\"\n skills = list_skills()\n assert \"trl-training\" in skills\n\n def test_skills_are_sorted(self):\n \"\"\"Test that skills are returned in sorted order.\"\"\"\n skills = list_skills()\n assert skills == sorted(skills)\n\n def test_with_custom_directory(self, tmp_path):\n \"\"\"Test list_skills with a custom directory.\"\"\"\n # Create fake skills\n (tmp_path / \"skill1\").mkdir()\n (tmp_path / \"skill1\" / \"SKILL.md\").write_text(\"# Skill 1\")\n (tmp_path / \"skill2\").mkdir()\n (tmp_path / \"skill2\" / \"SKILL.md\").write_text(\"# Skill 2\")\n (tmp_path / \"not-a-skill\").mkdir() # No SKILL.md\n\n skills = list_skills(tmp_path)\n assert skills == [\"skill1\", \"skill2\"]\n\n def test_empty_directory(self, tmp_path):\n \"\"\"Test list_skills with an empty directory.\"\"\"\n skills = list_skills(tmp_path)\n assert skills == []\n\n def test_nonexistent_directory(self, tmp_path):\n \"\"\"Test list_skills with a non-existent directory.\"\"\"\n nonexistent = tmp_path / \"nonexistent\"\n skills = list_skills(nonexistent)\n assert skills == []\n\n def test_ignores_files(self, tmp_path):\n \"\"\"Test that list_skills ignores files, only returns directories.\"\"\"\n (tmp_path / \"skill1\").mkdir()\n (tmp_path / \"skill1\" / \"SKILL.md\").write_text(\"# Skill 1\")\n (tmp_path / \"not-a-skill.txt\").write_text(\"Not a skill\")\n\n skills = list_skills(tmp_path)\n assert skills == [\"skill1\"]\n\n def test_requires_skill_md(self, tmp_path):\n \"\"\"Test that directories without SKILL.md are ignored.\"\"\"\n (tmp_path / \"has-skill-md\").mkdir()\n (tmp_path / \"has-skill-md\" / \"SKILL.md\").write_text(\"# Valid\")\n (tmp_path / \"no-skill-md\").mkdir()\n (tmp_path / \"no-skill-md\" / \"readme.md\").write_text(\"# Invalid\")\n\n skills = list_skills(tmp_path)\n assert skills == [\"has-skill-md\"]\n\n\nclass TestInstallSkill:\n \"\"\"Tests for install_skill function.\"\"\"\n\n def test_basic_installation(self, tmp_path):\n \"\"\"Test basic skill installation.\"\"\"\n target_dir = tmp_path / \"target\"\n\n result = install_skill(\"trl-training\", target_dir)\n\n assert result is True\n assert (target_dir / \"trl-training\").exists()\n assert (target_dir / \"trl-training\" / \"SKILL.md\").exists()\n\n def test_creates_target_directory(self, tmp_path):\n \"\"\"Test that install_skill creates the target directory if it doesn't exist.\"\"\"\n target_dir = tmp_path / \"nested\" / \"target\"\n\n install_skill(\"trl-training\", target_dir)\n\n assert target_dir.exists()\n assert (target_dir / \"trl-training\").exists()\n\n def test_skill_not_found(self, tmp_path):\n \"\"\"Test that install_skill raises FileNotFoundError for non-existent skill.\"\"\"\n target_dir = tmp_path / \"target\"\n\n with pytest.raises(FileNotFoundError, match=\"Skill 'nonexistent' not found\"):\n install_skill(\"nonexistent\", target_dir)\n\n def test_skill_already_exists_without_force(self, tmp_path):\n \"\"\"Test that install_skill raises FileExistsError if skill exists and force=False.\"\"\"\n target_dir = tmp_path / \"target\"\n\n # Install once\n install_skill(\"trl-training\", target_dir)\n\n # Try to install again without force\n with pytest.raises(FileExistsError, match=\"already installed\"):\n install_skill(\"trl-training\", target_dir, force=False)\n\n def test_force_overwrites_existing(self, tmp_path):\n \"\"\"Test that install_skill with force=True overwrites existing skill.\"\"\"\n target_dir = tmp_path / \"target\"\n\n # Install once\n install_skill(\"trl-training\", target_dir)\n\n # Modify the installed skill\n marker_file = target_dir / \"trl-training\" / \"marker.txt\"\n marker_file.write_text(\"This should be removed\")\n\n # Install again with force\n result = install_skill(\"trl-training\", target_dir, force=True)\n\n assert result is True\n assert (target_dir / \"trl-training\").exists()\n assert not marker_file.exists() # Marker should be gone\n\n def test_force_overwrites_symlink(self, tmp_path):\n \"\"\"Test that install_skill with force=True can overwrite a symlink.\"\"\"\n target_dir = tmp_path / \"target\"\n target_dir.mkdir()\n\n # Create a symlink\n symlink = target_dir / \"trl-training\"\n symlink.symlink_to(_get_trl_skills_dir() / \"trl-training\")\n\n # Install with force should replace symlink with copy\n result = install_skill(\"trl-training\", target_dir, force=True)\n\n assert result is True\n assert (target_dir / \"trl-training\").exists()\n assert not (target_dir / \"trl-training\").is_symlink()\n\n def test_skill_not_directory(self, tmp_path):\n \"\"\"Test that install_skill raises ValueError if skill is not a directory.\"\"\"\n source_dir = tmp_path / \"source\"\n source_dir.mkdir()\n target_dir = tmp_path / \"target\"\n\n # Create a file instead of directory\n (source_dir / \"fake-skill\").write_text(\"not a directory\")\n\n with pytest.raises(ValueError, match=\"is not a directory\"):\n install_skill(\"fake-skill\", target_dir, source=source_dir)\n\n def test_preserves_directory_structure(self, tmp_path):\n \"\"\"Test that install_skill preserves the skill's directory structure.\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n # Create a skill with subdirectories\n skill_dir = source_dir / \"test-skill\"\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(\"# Test\")\n (skill_dir / \"subdir\").mkdir()\n (skill_dir / \"subdir\" / \"file.txt\").write_text(\"content\")\n\n install_skill(\"test-skill\", target_dir, source=source_dir)\n\n assert (target_dir / \"test-skill\" / \"SKILL.md\").exists()\n assert (target_dir / \"test-skill\" / \"subdir\" / \"file.txt\").exists()\n assert (target_dir / \"test-skill\" / \"subdir\" / \"file.txt\").read_text() == \"content\"\n\n def test_install_to_same_directory_fails(self, tmp_path):\n \"\"\"Test that installing to the same directory as source is handled correctly.\"\"\"\n source_dir = tmp_path / \"skills\"\n source_dir.mkdir()\n\n # Create a skill\n skill_dir = source_dir / \"test-skill\"\n skill_dir.mkdir()\n (skill_dir / \"SKILL.md\").write_text(\"# Test\")\n\n # Try to install to same directory (should fail with exists error)\n with pytest.raises(FileExistsError):\n install_skill(\"test-skill\", source_dir, source=source_dir, force=False)\n\n\nclass TestUninstallSkill:\n \"\"\"Tests for uninstall_skill function.\"\"\"\n\n def test_basic_uninstallation(self, tmp_path):\n \"\"\"Test basic skill uninstallation.\"\"\"\n target_dir = tmp_path / \"target\"\n\n # Install first\n install_skill(\"trl-training\", target_dir)\n assert (target_dir / \"trl-training\").exists()\n\n # Uninstall\n result = uninstall_skill(\"trl-training\", target_dir)\n\n assert result is True\n assert not (target_dir / \"trl-training\").exists()\n\n def test_skill_not_installed(self, tmp_path):\n \"\"\"Test that uninstall_skill raises FileNotFoundError for non-existent skill.\"\"\"\n target_dir = tmp_path / \"target\"\n target_dir.mkdir()\n\n with pytest.raises(FileNotFoundError, match=\"not installed\"):\n uninstall_skill(\"nonexistent\", target_dir)\n\n def test_uninstall_from_nonexistent_directory(self, tmp_path):\n \"\"\"Test uninstall_skill when target directory doesn't exist.\"\"\"\n target_dir = tmp_path / \"nonexistent\"\n\n with pytest.raises(FileNotFoundError, match=\"not installed\"):\n uninstall_skill(\"trl-training\", target_dir)\n\n def test_uninstall_removes_all_contents(self, tmp_path):\n \"\"\"Test that uninstall removes the entire skill directory.\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n # Create a skill with multiple files\n skill_dir = source_dir / \"test-skill\"\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(\"# Test\")\n (skill_dir / \"file1.txt\").write_text(\"content1\")\n (skill_dir / \"subdir\").mkdir()\n (skill_dir / \"subdir\" / \"file2.txt\").write_text(\"content2\")\n\n # Install and uninstall\n install_skill(\"test-skill\", target_dir, source=source_dir)\n uninstall_skill(\"test-skill\", target_dir)\n\n assert not (target_dir / \"test-skill\").exists()\n # Target directory itself should still exist\n assert target_dir.exists()\n\n def test_uninstall_doesnt_affect_other_skills(self, tmp_path):\n \"\"\"Test that uninstalling one skill doesn't affect others.\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n # Create two skills\n for skill_name in [\"skill1\", \"skill2\"]:\n skill_dir = source_dir / skill_name\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(f\"# {skill_name}\")\n\n # Install both\n install_skill(\"skill1\", target_dir, source=source_dir)\n install_skill(\"skill2\", target_dir, source=source_dir)\n\n # Uninstall one\n uninstall_skill(\"skill1\", target_dir)\n\n # Check that only skill1 is removed\n assert not (target_dir / \"skill1\").exists()\n assert (target_dir / \"skill2\").exists()\n\n\nclass TestIntegration:\n \"\"\"Integration tests for skills functions.\"\"\"\n\n def test_full_workflow(self, tmp_path):\n \"\"\"Test complete install -> list -> uninstall workflow.\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n # Create skills\n for i in range(3):\n skill_dir = source_dir / f\"skill{i}\"\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(f\"# Skill {i}\")\n\n # List available skills\n available = list_skills(target=source_dir)\n assert available == [\"skill0\", \"skill1\", \"skill2\"]\n\n # Install skills\n for skill in available:\n install_skill(skill, target_dir, source=source_dir)\n\n # List installed skills\n installed_dirs = [d.name for d in target_dir.iterdir() if d.is_dir()]\n assert sorted(installed_dirs) == [\"skill0\", \"skill1\", \"skill2\"]\n\n # Uninstall one skill\n uninstall_skill(\"skill1\", target_dir)\n\n # Verify\n installed_dirs = [d.name for d in target_dir.iterdir() if d.is_dir()]\n assert sorted(installed_dirs) == [\"skill0\", \"skill2\"]\n\n def test_install_uninstall_cycle(self, tmp_path):\n \"\"\"Test that we can install and uninstall the same skill multiple times.\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n # Create skill\n skill_dir = source_dir / \"test-skill\"\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(\"# Test\")\n\n # Install -> Uninstall -> Install -> Uninstall\n for _ in range(2):\n install_skill(\"test-skill\", target_dir, source=source_dir)\n assert (target_dir / \"test-skill\").exists()\n\n uninstall_skill(\"test-skill\", target_dir)\n assert not (target_dir / \"test-skill\").exists()\n\n def test_force_reinstall_workflow(self, tmp_path):\n \"\"\"Test the workflow of using force to update an installed skill.\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n # Create initial skill version\n skill_dir = source_dir / \"test-skill\"\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(\"# Version 1\")\n\n # Install\n install_skill(\"test-skill\", target_dir, source=source_dir)\n assert (target_dir / \"test-skill\" / \"SKILL.md\").read_text() == \"# Version 1\"\n\n # Update source skill\n (skill_dir / \"SKILL.md\").write_text(\"# Version 2\")\n\n # Force reinstall\n install_skill(\"test-skill\", target_dir, source=source_dir, force=True)\n assert (target_dir / \"test-skill\" / \"SKILL.md\").read_text() == \"# Version 2\"\n\n\nclass TestEdgeCases:\n \"\"\"Tests for edge cases and special scenarios.\"\"\"\n\n def test_skill_with_special_characters_in_name(self, tmp_path):\n \"\"\"Test handling skills with special characters in names.\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n # Create skill with hyphens and underscores (common in skill names)\n skill_name = \"test-skill_v2\"\n skill_dir = source_dir / skill_name\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(\"# Test\")\n\n # Should work fine\n install_skill(skill_name, target_dir, source=source_dir)\n assert (target_dir / skill_name).exists()\n\n uninstall_skill(skill_name, target_dir)\n assert not (target_dir / skill_name).exists()\n\n def test_empty_skill_directory(self, tmp_path):\n \"\"\"Test installing a skill with only SKILL.md (no other files).\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n skill_dir = source_dir / \"minimal-skill\"\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(\"# Minimal\")\n\n install_skill(\"minimal-skill\", target_dir, source=source_dir)\n\n assert (target_dir / \"minimal-skill\" / \"SKILL.md\").exists()\n # Should only contain SKILL.md\n files = list((target_dir / \"minimal-skill\").iterdir())\n assert len(files) == 1\n assert files[0].name == \"SKILL.md\"\n\n def test_skill_with_hidden_files(self, tmp_path):\n \"\"\"Test that hidden files are preserved during installation.\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n skill_dir = source_dir / \"test-skill\"\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(\"# Test\")\n (skill_dir / \".hidden\").write_text(\"hidden content\")\n\n install_skill(\"test-skill\", target_dir, source=source_dir)\n\n assert (target_dir / \"test-skill\" / \".hidden\").exists()\n assert (target_dir / \"test-skill\" / \".hidden\").read_text() == \"hidden content\"\n\n def test_list_skills_with_symlinks(self, tmp_path):\n \"\"\"Test that list_skills handles symlinked skill directories.\"\"\"\n source_dir = tmp_path / \"source\"\n skills_dir = tmp_path / \"skills\"\n skills_dir.mkdir()\n\n # Create a real skill\n skill_dir = source_dir / \"real-skill\"\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(\"# Real\")\n\n # Create symlink to it\n (skills_dir / \"linked-skill\").symlink_to(skill_dir)\n\n # list_skills should include symlinked skills if they have SKILL.md\n skills = list_skills(target=skills_dir)\n assert \"linked-skill\" in skills\n\n\nclass TestListAgentNames:\n \"\"\"Tests for list_agent_names function.\"\"\"\n\n def test_returns_list(self):\n \"\"\"Test that list_agent_names returns a list.\"\"\"\n agents = list_agent_names()\n assert isinstance(agents, list)\n\n def test_contains_expected_agents(self):\n \"\"\"Test that list includes expected agent names.\"\"\"\n agents = list_agent_names()\n assert \"claude\" in agents\n assert \"codex\" in agents\n assert \"opencode\" in agents\n\n def test_agents_are_sorted(self):\n \"\"\"Test that agent names are sorted.\"\"\"\n agents = list_agent_names()\n assert agents == sorted(agents)\n\n\nclass TestResolveTargetPath:\n \"\"\"Tests for resolve_target_path function.\"\"\"\n\n def test_resolve_agent_name_project_scope(self):\n \"\"\"Test resolving agent name with project scope.\"\"\"\n path = resolve_target_path(\"claude\", \"project\")\n assert path == Path(\"./.claude/skills\").expanduser().resolve()\n\n def test_resolve_agent_name_global_scope(self):\n \"\"\"Test resolving agent name with global scope.\"\"\"\n path = resolve_target_path(\"claude\", \"global\")\n assert path == Path(\"~/.claude/skills\").expanduser().resolve()\n\n def test_resolve_custom_path_string(self):\n \"\"\"Test resolving custom path as string.\"\"\"\n path = resolve_target_path(\"/custom/path\", \"project\")\n assert path == Path(\"/custom/path\").resolve()\n\n def test_resolve_custom_path_object(self):\n \"\"\"Test resolving Path object.\"\"\"\n custom = Path(\"/custom/path\")\n path = resolve_target_path(custom, \"project\")\n assert path == Path(\"/custom/path\").resolve()\n\n def test_resolve_path_with_tilde(self):\n \"\"\"Test that tilde expansion works.\"\"\"\n path = resolve_target_path(\"~/my/skills\", \"project\")\n assert path == Path(\"~/my/skills\").expanduser().resolve()\n assert \"~\" not in str(path)\n\n def test_all_predefined_agents(self):\n \"\"\"Test that all predefined agents can be resolved.\"\"\"\n for agent in list_agent_names():\n for scope in [\"project\", \"global\"]:\n path = resolve_target_path(agent, scope)\n assert isinstance(path, Path)\n assert path.is_absolute()\n\n def test_invalid_scope_for_predefined_agent(self):\n \"\"\"Test invalid scope raises ValueError for predefined agents.\"\"\"\n with pytest.raises(ValueError, match=\"Invalid scope\"):\n resolve_target_path(\"claude\", \"invalid\")\n\n\nclass TestHighLevelAPI:\n \"\"\"Tests for the new high-level API (target/scope instead of Path).\"\"\"\n\n def test_list_skills_with_target_string(self, tmp_path):\n \"\"\"Test list_skills with target as string (custom path).\"\"\"\n # Create skills in target\n (tmp_path / \"skill1\").mkdir()\n (tmp_path / \"skill1\" / \"SKILL.md\").write_text(\"# Skill 1\")\n\n skills = list_skills(target=str(tmp_path), scope=\"project\")\n assert skills == [\"skill1\"]\n\n def test_list_skills_with_target_path(self, tmp_path):\n \"\"\"Test list_skills with target as Path object.\"\"\"\n (tmp_path / \"skill1\").mkdir()\n (tmp_path / \"skill1\" / \"SKILL.md\").write_text(\"# Skill 1\")\n\n skills = list_skills(target=tmp_path, scope=\"project\")\n assert skills == [\"skill1\"]\n\n def test_list_skills_without_target(self):\n \"\"\"Test list_skills without target lists TRL's built-in skills.\"\"\"\n skills = list_skills()\n assert isinstance(skills, list)\n assert \"trl-training\" in skills\n\n def test_install_skill_with_target_string(self, tmp_path):\n \"\"\"Test install_skill with target as string.\"\"\"\n result = install_skill(\"trl-training\", target=str(tmp_path), scope=\"project\")\n assert result is True\n assert (tmp_path / \"trl-training\").exists()\n\n def test_install_skill_with_target_path(self, tmp_path):\n \"\"\"Test install_skill with target as Path object.\"\"\"\n result = install_skill(\"trl-training\", target=tmp_path, scope=\"project\")\n assert result is True\n assert (tmp_path / \"trl-training\").exists()\n\n def test_install_skill_with_force(self, tmp_path):\n \"\"\"Test install_skill with force parameter.\"\"\"\n install_skill(\"trl-training\", target=tmp_path)\n # Install again with force\n result = install_skill(\"trl-training\", target=tmp_path, force=True)\n assert result is True\n\n def test_uninstall_skill_with_target_string(self, tmp_path):\n \"\"\"Test uninstall_skill with target as string.\"\"\"\n install_skill(\"trl-training\", target=tmp_path)\n result = uninstall_skill(\"trl-training\", target=str(tmp_path), scope=\"project\")\n assert result is True\n assert not (tmp_path / \"trl-training\").exists()\n\n def test_uninstall_skill_with_target_path(self, tmp_path):\n \"\"\"Test uninstall_skill with target as Path object.\"\"\"\n install_skill(\"trl-training\", target=tmp_path)\n result = uninstall_skill(\"trl-training\", target=tmp_path, scope=\"project\")\n assert result is True\n assert not (tmp_path / \"trl-training\").exists()\n\n def test_install_with_custom_source(self, tmp_path):\n \"\"\"Test install_skill with custom source parameter.\"\"\"\n source_dir = tmp_path / \"source\"\n target_dir = tmp_path / \"target\"\n\n # Create custom skill\n skill_dir = source_dir / \"custom-skill\"\n skill_dir.mkdir(parents=True)\n (skill_dir / \"SKILL.md\").write_text(\"# Custom\")\n\n result = install_skill(\"custom-skill\", target=target_dir, source=source_dir)\n assert result is True\n assert (target_dir / \"custom-skill\").exists()\n" }, { "path": "tests/test_skills_cli.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport argparse\n\nimport pytest\n\nfrom trl.skills import install_skill\nfrom trl.skills.cli import add_skills_subcommands, cmd_install, cmd_list, cmd_uninstall\n\n\nclass TestCLICommands:\n \"\"\"Tests for CLI command handlers.\"\"\"\n\n def test_cmd_list_without_target(self, capsys):\n \"\"\"Test cmd_list without target (lists TRL skills).\"\"\"\n args = argparse.Namespace(target=None, scope=\"project\")\n\n result = cmd_list(args)\n\n captured = capsys.readouterr()\n assert result == 0\n assert \"TRL (available for installation)\" in captured.out\n assert \"trl-training\" in captured.out\n assert \"Use 'trl skills install\" in captured.out\n\n def test_cmd_list_with_target(self, tmp_path, capsys):\n \"\"\"Test cmd_list with target (lists installed skills).\"\"\"\n # Install a skill\n install_skill(\"trl-training\", target=tmp_path)\n\n args = argparse.Namespace(target=str(tmp_path), scope=\"project\")\n result = cmd_list(args)\n\n captured = capsys.readouterr()\n assert result == 0\n assert \"trl-training\" in captured.out\n assert str(tmp_path) in captured.out\n\n def test_cmd_list_empty_target(self, tmp_path, capsys):\n \"\"\"Test cmd_list with empty target directory.\"\"\"\n args = argparse.Namespace(target=str(tmp_path), scope=\"project\")\n\n result = cmd_list(args)\n\n captured = capsys.readouterr()\n assert result == 0\n assert \"No skills installed\" in captured.out\n\n def test_cmd_install_single_skill(self, tmp_path, capsys):\n \"\"\"Test cmd_install with single skill.\"\"\"\n args = argparse.Namespace(skill=\"trl-training\", all=False, target=str(tmp_path), scope=\"project\", force=False)\n\n result = cmd_install(args)\n\n captured = capsys.readouterr()\n assert result == 0\n assert \"โœ“\" in captured.out\n assert \"1/1 skills installed\" in captured.out\n assert (tmp_path / \"trl-training\").exists()\n\n def test_cmd_install_all_skills(self, tmp_path, capsys):\n \"\"\"Test cmd_install with --all flag.\"\"\"\n args = argparse.Namespace(skill=None, all=True, target=str(tmp_path), scope=\"project\", force=False)\n\n result = cmd_install(args)\n\n captured = capsys.readouterr()\n assert result == 0\n assert \"โœ“\" in captured.out\n assert \"installed successfully\" in captured.out\n assert (tmp_path / \"trl-training\").exists()\n\n def test_cmd_install_no_skill_or_all(self, capsys):\n \"\"\"Test cmd_install without skill name or --all flag.\"\"\"\n args = argparse.Namespace(skill=None, all=False, target=\"/tmp/test\", scope=\"project\", force=False)\n\n result = cmd_install(args)\n\n captured = capsys.readouterr()\n assert result == 1\n assert \"Error: Either provide a skill name or use --all\" in captured.out\n\n def test_cmd_install_both_skill_and_all(self, capsys):\n \"\"\"Test cmd_install with both skill name and --all (error).\"\"\"\n args = argparse.Namespace(skill=\"trl-training\", all=True, target=\"/tmp/test\", scope=\"project\", force=False)\n\n result = cmd_install(args)\n\n captured = capsys.readouterr()\n assert result == 1\n assert \"Cannot specify both\" in captured.out\n\n def test_cmd_install_nonexistent_skill(self, tmp_path, capsys):\n \"\"\"Test cmd_install with non-existent skill.\"\"\"\n args = argparse.Namespace(skill=\"nonexistent\", all=False, target=str(tmp_path), scope=\"project\", force=False)\n\n result = cmd_install(args)\n\n captured = capsys.readouterr()\n assert result == 1\n assert \"โœ—\" in captured.out\n assert \"0/1 skills installed\" in captured.out\n\n def test_cmd_install_already_exists(self, tmp_path, capsys):\n \"\"\"Test cmd_install when skill already exists without force.\"\"\"\n # Install once\n install_skill(\"trl-training\", target=tmp_path)\n\n args = argparse.Namespace(skill=\"trl-training\", all=False, target=str(tmp_path), scope=\"project\", force=False)\n\n result = cmd_install(args)\n\n captured = capsys.readouterr()\n assert result == 1\n assert \"โœ—\" in captured.out\n assert \"Use --force to overwrite\" in captured.out\n\n def test_cmd_install_with_force(self, tmp_path, capsys):\n \"\"\"Test cmd_install with --force to overwrite.\"\"\"\n # Install once\n install_skill(\"trl-training\", target=tmp_path)\n\n args = argparse.Namespace(skill=\"trl-training\", all=False, target=str(tmp_path), scope=\"project\", force=True)\n\n result = cmd_install(args)\n\n captured = capsys.readouterr()\n assert result == 0\n assert \"โœ“\" in captured.out\n assert \"1/1 skills installed\" in captured.out\n\n def test_cmd_uninstall_success(self, tmp_path, capsys):\n \"\"\"Test cmd_uninstall with installed skill.\"\"\"\n # Install first\n install_skill(\"trl-training\", target=tmp_path)\n\n args = argparse.Namespace(skill=\"trl-training\", target=str(tmp_path), scope=\"project\")\n\n result = cmd_uninstall(args)\n\n captured = capsys.readouterr()\n assert result == 0\n assert \"โœ“\" in captured.out\n assert \"has been removed\" in captured.out\n assert not (tmp_path / \"trl-training\").exists()\n\n def test_cmd_uninstall_not_installed(self, tmp_path, capsys):\n \"\"\"Test cmd_uninstall when skill is not installed.\"\"\"\n args = argparse.Namespace(skill=\"nonexistent\", target=str(tmp_path), scope=\"project\")\n\n result = cmd_uninstall(args)\n\n captured = capsys.readouterr()\n assert result == 1\n assert \"โœ—\" in captured.out\n assert \"Error:\" in captured.out\n\n def test_cmd_install_creates_target_directory(self, tmp_path, capsys):\n \"\"\"Test cmd_install creates target directory if it doesn't exist.\"\"\"\n # Custom path that doesn't exist yet\n target_path = tmp_path / \"new_directory\"\n assert not target_path.exists()\n\n args = argparse.Namespace(\n skill=\"trl-training\", all=False, target=str(target_path), scope=\"project\", force=False\n )\n\n result = cmd_install(args)\n\n captured = capsys.readouterr()\n assert result == 0\n assert \"โœ“\" in captured.out\n assert target_path.exists()\n\n def test_cmd_uninstall_invalid_target(self, capsys):\n \"\"\"Test cmd_uninstall with non-existent path.\"\"\"\n args = argparse.Namespace(skill=\"trl-training\", target=\"/nonexistent/invalid/path\", scope=\"project\")\n\n result = cmd_uninstall(args)\n\n captured = capsys.readouterr()\n assert result == 1\n assert \"โœ—\" in captured.out\n\n\nclass TestCLIArgumentParsing:\n \"\"\"Tests for CLI argument parsing setup.\"\"\"\n\n def test_add_skills_subcommands_creates_parsers(self):\n \"\"\"Test that add_skills_subcommands creates the expected subparsers.\"\"\"\n parser = argparse.ArgumentParser()\n subparsers = parser.add_subparsers(dest=\"command\")\n\n add_skills_subcommands(subparsers)\n\n # Test that we can parse expected commands\n args = parser.parse_args([\"list\"])\n assert args.command == \"list\"\n assert hasattr(args, \"func\")\n\n args = parser.parse_args([\"install\", \"trl-training\", \"--target\", \"claude\"])\n assert args.command == \"install\"\n assert args.skill == \"trl-training\"\n assert args.target == \"claude\"\n\n args = parser.parse_args([\"uninstall\", \"trl-training\", \"--target\", \"claude\"])\n assert args.command == \"uninstall\"\n assert args.skill == \"trl-training\"\n\n def test_list_command_optional_target(self):\n \"\"\"Test that list command has optional target.\"\"\"\n parser = argparse.ArgumentParser()\n subparsers = parser.add_subparsers(dest=\"command\")\n add_skills_subcommands(subparsers)\n\n # Should work without target\n args = parser.parse_args([\"list\"])\n assert args.target is None\n\n # Should work with target\n args = parser.parse_args([\"list\", \"--target\", \"claude\"])\n assert args.target == \"claude\"\n\n def test_install_command_requires_target(self):\n \"\"\"Test that install command requires target.\"\"\"\n parser = argparse.ArgumentParser()\n subparsers = parser.add_subparsers(dest=\"command\")\n add_skills_subcommands(subparsers)\n\n # Should fail without target\n with pytest.raises(SystemExit):\n parser.parse_args([\"install\", \"trl-training\"])\n\n def test_scope_choices(self):\n \"\"\"Test that scope parameter accepts valid choices.\"\"\"\n parser = argparse.ArgumentParser()\n subparsers = parser.add_subparsers(dest=\"command\")\n add_skills_subcommands(subparsers)\n\n # Valid scopes\n args = parser.parse_args([\"install\", \"trl-training\", \"--target\", \"claude\", \"--scope\", \"project\"])\n assert args.scope == \"project\"\n\n args = parser.parse_args([\"install\", \"trl-training\", \"--target\", \"claude\", \"--scope\", \"global\"])\n assert args.scope == \"global\"\n\n # Invalid scope should fail\n with pytest.raises(SystemExit):\n parser.parse_args([\"install\", \"trl-training\", \"--target\", \"claude\", \"--scope\", \"invalid\"])\n\n def test_install_all_flag(self):\n \"\"\"Test install --all flag.\"\"\"\n parser = argparse.ArgumentParser()\n subparsers = parser.add_subparsers(dest=\"command\")\n add_skills_subcommands(subparsers)\n\n args = parser.parse_args([\"install\", \"--all\", \"--target\", \"claude\"])\n assert args.all is True\n assert args.skill is None\n\n def test_install_force_flag(self):\n \"\"\"Test install --force flag.\"\"\"\n parser = argparse.ArgumentParser()\n subparsers = parser.add_subparsers(dest=\"command\")\n add_skills_subcommands(subparsers)\n\n args = parser.parse_args([\"install\", \"trl-training\", \"--target\", \"claude\", \"--force\"])\n assert args.force is True\n\n def test_default_scope_is_project(self):\n \"\"\"Test that default scope is 'project'.\"\"\"\n parser = argparse.ArgumentParser()\n subparsers = parser.add_subparsers(dest=\"command\")\n add_skills_subcommands(subparsers)\n\n args = parser.parse_args([\"install\", \"trl-training\", \"--target\", \"claude\"])\n assert args.scope == \"project\"\n" }, { "path": "tests/test_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport textwrap\nfrom io import StringIO\nfrom unittest.mock import patch\n\nimport pytest\nimport torch\nimport transformers\nfrom packaging.version import Version\nfrom transformers import AutoModelForCausalLM, AutoModelForImageTextToText\nfrom transformers.utils import is_peft_available\n\nfrom trl import ModelConfig\nfrom trl.trainer.utils import (\n RepeatSampler,\n entropy_from_logits,\n flush_left,\n flush_right,\n forward_masked_logits,\n generate_model_card,\n get_peft_config,\n hash_module,\n nanstd,\n pad,\n print_prompt_completions_sample,\n selective_log_softmax,\n shuffle_sequence_dict,\n split_pixel_values_by_grid,\n split_tensor_dict,\n unsplit_pixel_values_by_grid,\n use_adapter,\n)\n\nfrom .testing_utils import TrlTestCase, require_peft, require_rich\n\n\nif is_peft_available():\n from peft import AutoPeftModelForCausalLM, LoraConfig\n\n\n@require_peft\nclass TestUseAdapter(TrlTestCase):\n def test_disables_on_none(self):\n model = AutoPeftModelForCausalLM.from_pretrained(\n \"trl-internal-testing/tiny-PeftModel\", adapter_name=\"my_adapter\"\n )\n input_ids = torch.tensor([[1, 2, 3], [4, 5, 6]])\n with model.disable_adapter():\n expected = model(input_ids).logits\n\n with use_adapter(model, None):\n output = model(input_ids).logits\n\n assert torch.equal(output, expected)\n\n def test_restores_previous_adapter(self):\n model = AutoPeftModelForCausalLM.from_pretrained(\n \"trl-internal-testing/tiny-PeftModel\", adapter_name=\"my_adapter\"\n )\n input_ids = torch.tensor([[1, 2, 3], [4, 5, 6]])\n expected = model(input_ids).logits\n with use_adapter(model, \"my_adapter\"):\n pass\n output = model(input_ids).logits\n assert torch.equal(output, expected)\n\n with use_adapter(model, None):\n pass\n output = model(input_ids).logits\n assert torch.equal(output, expected)\n\n def test_with_multiple_adapters(self):\n model = AutoPeftModelForCausalLM.from_pretrained(\n \"trl-internal-testing/tiny-PeftModel\", adapter_name=\"my_adapter_1\"\n )\n model.load_adapter(\"trl-internal-testing/tiny-PeftModel-2\", \"my_adapter_2\")\n input_ids = torch.tensor([[1, 2, 3], [4, 5, 6]])\n\n model.set_adapter(\"my_adapter_1\") # should be a no-op, but let's keep it for clarity\n expected_1 = model(input_ids).logits\n model.set_adapter(\"my_adapter_2\")\n expected_2 = model(input_ids).logits\n\n with use_adapter(model, \"my_adapter_1\"):\n output_1 = model(input_ids).logits\n\n with use_adapter(model, \"my_adapter_2\"):\n output_2 = model(input_ids).logits\n\n assert torch.equal(output_1, expected_1)\n assert torch.equal(output_2, expected_2)\n\n\nclass TestPad(TrlTestCase):\n def test_pad_1_dim_left(self):\n x = torch.tensor([1, 2, 3])\n y = torch.tensor([4, 5])\n output = pad((x, y), padding_value=0, padding_side=\"left\")\n expected = torch.tensor([[1, 2, 3], [0, 4, 5]])\n assert torch.equal(output, expected)\n\n def test_pad_1_dim_right(self):\n x = torch.tensor([1, 2, 3])\n y = torch.tensor([4, 5])\n output = pad((x, y), padding_value=0, padding_side=\"right\")\n expected = torch.tensor([[1, 2, 3], [4, 5, 0]])\n assert torch.equal(output, expected)\n\n def test_pad_2_dim_left(self):\n x = torch.tensor([[1, 2], [3, 4]])\n y = torch.tensor([[5, 6]])\n output = pad((x, y), padding_value=0, padding_side=\"left\")\n expected = torch.tensor(\n [\n [[1, 2], [3, 4]],\n [[0, 0], [5, 6]],\n ]\n )\n assert torch.equal(output, expected)\n\n def test_pad_2_dim_right(self):\n x = torch.tensor([[1, 2], [3, 4]])\n y = torch.tensor([[5, 6]])\n output = pad((x, y), padding_value=0, padding_side=\"right\")\n expected = torch.tensor(\n [\n [[1, 2], [3, 4]],\n [[5, 6], [0, 0]],\n ]\n )\n assert torch.equal(output, expected)\n\n def test_pad_2_dim_right_multidim(self):\n x = torch.tensor([[1, 2], [3, 4]])\n y = torch.tensor([[5]])\n output = pad((x, y), padding_value=0, padding_side=\"right\")\n expected = torch.tensor(\n [\n [[1, 2], [3, 4]],\n [[5, 0], [0, 0]],\n ]\n )\n assert torch.equal(output, expected)\n\n def test_pad_to_multiple_of_1(self):\n x = torch.tensor([1, 2, 3])\n y = torch.tensor([4, 5])\n # Max length is 3, pad to multiple of 4\n output = pad((x, y), padding_value=0, padding_side=\"right\", pad_to_multiple_of=4)\n expected = torch.tensor([[1, 2, 3, 0], [4, 5, 0, 0]])\n assert torch.equal(output, expected)\n\n def test_pad_to_multiple_of_2(self):\n x = torch.tensor([1, 2, 3, 4, 5])\n y = torch.tensor([6, 7, 8])\n # Max length is 3, pad to multiple of 4\n output = pad((x, y), padding_value=0, padding_side=\"right\", pad_to_multiple_of=4)\n expected = torch.tensor([[1, 2, 3, 4, 5, 0, 0, 0], [6, 7, 8, 0, 0, 0, 0, 0]])\n assert torch.equal(output, expected)\n\n def test_pad_to_multiple_of_side_left(self):\n x = torch.tensor([1, 2, 3, 4, 5])\n y = torch.tensor([6, 7, 8])\n # Max length is 3, pad to multiple of 4\n output = pad((x, y), padding_value=0, padding_side=\"left\", pad_to_multiple_of=4)\n expected = torch.tensor([[0, 0, 0, 1, 2, 3, 4, 5], [0, 0, 0, 0, 0, 6, 7, 8]])\n assert torch.equal(output, expected)\n\n def test_pad_to_multiple_of_no_extra_padding(self):\n x = torch.tensor([1, 2, 3, 4])\n y = torch.tensor([5, 6, 7, 8])\n # Already multiple of 4\n output = pad((x, y), padding_value=0, padding_side=\"left\", pad_to_multiple_of=4)\n expected = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8]])\n assert torch.equal(output, expected)\n\n\nclass TestHashModule(TrlTestCase):\n def test_hash_module_deterministic_across_order(self):\n class ModAB(torch.nn.Module):\n def __init__(self, a: torch.Tensor, b: torch.Tensor):\n super().__init__()\n self.a = torch.nn.Parameter(a)\n self.b = torch.nn.Parameter(b)\n\n class ModBA(torch.nn.Module):\n def __init__(self, a: torch.Tensor, b: torch.Tensor):\n super().__init__()\n self.b = torch.nn.Parameter(b)\n self.a = torch.nn.Parameter(a)\n\n a = torch.tensor([[1.0, 2.0]])\n b = torch.tensor([3.0])\n assert hash_module(ModAB(a, b)) == hash_module(ModBA(a, b))\n\n def test_hash_module_changes_with_value(self):\n class Mod(torch.nn.Module):\n def __init__(self, value: float):\n super().__init__()\n self.weight = torch.nn.Parameter(torch.tensor([value, 2.0]))\n\n assert hash_module(Mod(1.0)) != hash_module(Mod(1.5))\n\n def test_hash_module_includes_dtype(self):\n class Mod(torch.nn.Module):\n def __init__(self, dtype: torch.dtype):\n super().__init__()\n self.weight = torch.nn.Parameter(torch.tensor([1.0, 2.0], dtype=dtype))\n\n assert hash_module(Mod(torch.float32)) != hash_module(Mod(torch.float16))\n\n def test_hash_module_tiny_model_twice(self):\n model_id = \"trl-internal-testing/tiny-GptOssForCausalLM\"\n model_a = AutoModelForCausalLM.from_pretrained(model_id)\n model_b = AutoModelForCausalLM.from_pretrained(model_id)\n assert hash_module(model_a) == hash_module(model_b)\n\n def test_hash_module_tiny_model_change_layer(self):\n model_id = \"trl-internal-testing/tiny-GptOssForCausalLM\"\n model = AutoModelForCausalLM.from_pretrained(model_id)\n h1 = hash_module(model)\n with torch.no_grad():\n model.lm_head.weight.add_(0.01)\n h2 = hash_module(model)\n assert h1 != h2\n\n\n@require_peft\nclass TestGetPEFTConfig(TrlTestCase):\n def test_create_peft_config_use_peft_false(self):\n \"\"\"Test that when use_peft is False, the function returns None.\"\"\"\n model_args = ModelConfig(use_peft=False)\n peft_config = get_peft_config(model_args)\n assert peft_config is None\n\n def test_create_peft_config_use_peft_true(self):\n \"\"\"Test that when use_peft is True, the function returns a LoraConfig object.\"\"\"\n # Provide non-default values to the model config for testing\n peft_kwargs = {\n \"lora_r\": 8,\n \"lora_alpha\": 16,\n \"lora_dropout\": 0.1,\n \"lora_task_type\": \"SEQ_CLS\",\n \"use_rslora\": True,\n \"lora_target_modules\": [\"up_proj\", \"down_proj\"],\n \"lora_modules_to_save\": [\"up_proj\"],\n }\n model_args = ModelConfig(use_peft=True, **peft_kwargs)\n peft_config = get_peft_config(model_args)\n assert isinstance(peft_config, LoraConfig)\n for arg, value in peft_kwargs.items():\n # Test that lists of modules are converted to sets\n if arg == \"lora_target_modules\":\n value = set(value)\n # Rename the argument to match the LoraConfig attribute name\n if arg in [\"lora_r\", \"lora_task_type\", \"lora_target_modules\", \"lora_modules_to_save\"]:\n arg = arg[len(\"lora_\") :] if arg.startswith(\"lora_\") else arg\n\n assert getattr(peft_config, arg) == value\n\n\nclass TestNanStd(TrlTestCase):\n def test_nanstd_ignores_nans(self):\n x = torch.tensor([1.0, 2.0, 3.0, float(\"nan\")])\n result = nanstd(x)\n torch.testing.assert_close(result, torch.tensor(1.0))\n\n def test_nanstd_dim_and_keepdim(self):\n x = torch.tensor([[1.0, float(\"nan\")], [3.0, 5.0]])\n result = nanstd(x, dim=1, keepdim=True)\n assert torch.isnan(result[0, 0])\n torch.testing.assert_close(result[1, 0], torch.tensor(1.4142135), rtol=1e-5, atol=1e-6)\n\n def test_nanstd_all_nan(self):\n x = torch.tensor([float(\"nan\"), float(\"nan\")])\n result = nanstd(x)\n assert torch.isnan(result)\n\n\nclass TestGenerateModelCard(TrlTestCase):\n def test_full(self):\n model_card = generate_model_card(\n base_model=\"username/my_base_model\",\n model_name=\"my_model\",\n hub_model_id=\"username/my_hub_model\",\n dataset_name=\"username/my_dataset\",\n tags=[\"trl\", \"trainer-tag\"],\n wandb_url=\"https://wandb.ai/username/project_id/runs/abcd1234\",\n trackio_url=\"https://huggingface.co/spaces/username/space_id\",\n comet_url=\"https://www.comet.com/username/project_id/experiment_id\",\n trainer_name=\"My Trainer\",\n trainer_citation=\"@article{my_trainer, ...}\",\n paper_title=\"My Paper\",\n paper_id=\"1234.56789\",\n )\n card_text = str(model_card)\n assert \"[username/my_base_model](https://huggingface.co/username/my_base_model)\" in card_text\n assert \"my_model\" in card_text\n assert 'pipeline(\"text-generation\", model=\"username/my_hub_model\", device=\"cuda\")' in card_text\n assert \"datasets: username/my_dataset\" in card_text\n assert \"](https://wandb.ai/username/project_id/runs/abcd1234)\" in card_text\n assert \"](https://huggingface.co/spaces/username/space_id)\" in card_text\n assert \"](https://www.comet.com/username/project_id/experiment_id\" in card_text\n assert \"My Trainer\" in card_text\n assert \"```bibtex\\n@article{my_trainer, ...}\\n```\" in card_text\n assert \"[My Paper](https://huggingface.co/papers/1234.56789)\" in card_text\n\n def test_val_none(self):\n model_card = generate_model_card(\n base_model=None,\n model_name=\"my_model\",\n hub_model_id=\"username/my_hub_model\",\n dataset_name=None,\n tags=[],\n wandb_url=None,\n trackio_url=None,\n comet_url=None,\n trainer_name=\"My Trainer\",\n trainer_citation=None,\n paper_title=None,\n paper_id=None,\n )\n card_text = str(model_card)\n assert \"my_model\" in card_text\n assert 'pipeline(\"text-generation\", model=\"username/my_hub_model\", device=\"cuda\")' in card_text\n assert \"My Trainer\" in card_text\n\n\nclass TestFlushLeft(TrlTestCase):\n def test_basic_case(self):\n mask = torch.tensor([[0, 0, 1, 1, 1], [0, 1, 1, 0, 0]])\n tensor1 = torch.tensor([[0, 0, 2, 3, 4], [0, 5, 6, 0, 0]])\n tensor2 = torch.tensor([[0, 0, 7, 8, 9], [0, 10, 11, 0, 0]])\n new_mask, new_tensor1, new_tensor2 = flush_left(mask, tensor1, tensor2)\n\n expected_mask = torch.tensor([[1, 1, 1], [1, 1, 0]])\n expected_tensor1 = torch.tensor([[2, 3, 4], [5, 6, 0]])\n expected_tensor2 = torch.tensor([[7, 8, 9], [10, 11, 0]])\n\n assert torch.equal(new_mask, expected_mask)\n assert torch.equal(new_tensor1, expected_tensor1)\n assert torch.equal(new_tensor2, expected_tensor2)\n\n def test_single_row(self):\n mask = torch.tensor([[0, 0, 1, 1]])\n tensor1 = torch.tensor([[0, 0, 2, 3]])\n new_mask, new_tensor1 = flush_left(mask, tensor1)\n\n expected_mask = torch.tensor([[1, 1]])\n expected_tensor1 = torch.tensor([[2, 3]])\n\n assert torch.equal(new_mask, expected_mask)\n assert torch.equal(new_tensor1, expected_tensor1)\n\n def test_no_shift_needed(self):\n mask = torch.tensor([[1, 1, 0, 0], [1, 0, 0, 0]])\n tensor1 = torch.tensor([[5, 6, 0, 0], [7, 0, 0, 0]])\n new_mask, new_tensor1 = flush_left(mask, tensor1)\n\n expected_mask = torch.tensor([[1, 1], [1, 0]])\n expected_tensor1 = torch.tensor([[5, 6], [7, 0]])\n\n assert torch.equal(new_mask, expected_mask)\n assert torch.equal(new_tensor1, expected_tensor1)\n\n def test_no_tensors(self):\n mask = torch.tensor([[0, 0, 1, 1, 1], [0, 1, 1, 0, 0]])\n new_mask = flush_left(mask)\n expected_mask = torch.tensor([[1, 1, 1], [1, 1, 0]])\n assert torch.equal(new_mask, expected_mask)\n\n\nclass TestFlushRight(TrlTestCase):\n def test_basic_case(self):\n mask = torch.tensor([[1, 1, 1, 0, 0], [0, 0, 1, 1, 0]])\n tensor1 = torch.tensor([[2, 3, 4, 0, 0], [0, 0, 5, 6, 0]])\n tensor2 = torch.tensor([[7, 8, 9, 0, 0], [0, 0, 10, 11, 0]])\n new_mask, new_tensor1, new_tensor2 = flush_right(mask, tensor1, tensor2)\n\n expected_mask = torch.tensor([[1, 1, 1], [0, 1, 1]])\n expected_tensor1 = torch.tensor([[2, 3, 4], [0, 5, 6]])\n expected_tensor2 = torch.tensor([[7, 8, 9], [0, 10, 11]])\n\n assert torch.equal(new_mask, expected_mask)\n assert torch.equal(new_tensor1, expected_tensor1)\n assert torch.equal(new_tensor2, expected_tensor2)\n\n def test_single_row(self):\n mask = torch.tensor([[1, 1, 0, 0]])\n tensor1 = torch.tensor([[2, 3, 0, 0]])\n new_mask, new_tensor1 = flush_right(mask, tensor1)\n\n expected_mask = torch.tensor([[1, 1]])\n expected_tensor1 = torch.tensor([[2, 3]])\n\n assert torch.equal(new_mask, expected_mask)\n assert torch.equal(new_tensor1, expected_tensor1)\n\n def test_no_shift_needed(self):\n mask = torch.tensor([[0, 0, 1, 1], [0, 0, 0, 1]])\n tensor1 = torch.tensor([[0, 0, 5, 6], [0, 0, 0, 7]])\n new_mask, new_tensor1 = flush_right(mask, tensor1)\n\n expected_mask = torch.tensor([[1, 1], [0, 1]])\n expected_tensor1 = torch.tensor([[5, 6], [0, 7]])\n\n assert torch.equal(new_mask, expected_mask)\n assert torch.equal(new_tensor1, expected_tensor1)\n\n def test_no_tensors(self):\n mask = torch.tensor([[1, 1, 1, 0, 0], [0, 0, 1, 1, 0]])\n new_mask = flush_right(mask)\n expected_mask = torch.tensor([[1, 1, 1], [0, 1, 1]])\n assert torch.equal(new_mask, expected_mask)\n\n\nclass TestRepeatRandomSampler(TrlTestCase):\n def test_sampler(self):\n dataset = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\"]\n sampler = RepeatSampler(dataset, mini_repeat_count=2)\n # Should output something like [4, 4, 3, 3, 0, 0, 1, 1, 2, 2, 6, 6, 5, 5]\n sampled = list(sampler)\n # Check that the length is doubled\n assert len(sampled) == 2 * len(dataset)\n # Check that all indexes are present\n assert set(sampled) == set(range(len(dataset)))\n # Check that each element is repeated twice\n assert all(sampled[i] == sampled[i + 1] for i in range(0, len(sampled), 2))\n\n def test_sampler_no_shuffle(self):\n dataset = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\"]\n sampler = RepeatSampler(dataset, mini_repeat_count=2, shuffle=False)\n sampled = list(sampler)\n expected = [0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6]\n assert sampled == expected\n\n def test_sampler_no_repeat(self):\n dataset = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\"]\n sampler = RepeatSampler(dataset, mini_repeat_count=1)\n # Should output something like [4, 3, 0, 1, 2, 6, 5]\n sampled = list(sampler)\n # Check that the length is the same\n assert len(sampled) == len(dataset)\n # Check that all indexes are present\n assert set(sampled) == set(range(len(dataset)))\n\n def test_sampler_with_batch_size(self):\n dataset = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\", \"h\"]\n sampler = RepeatSampler(dataset, mini_repeat_count=1, batch_size=2, repeat_count=2)\n # Should output something like [4, 3, 4, 3, 0, 1, 0, 1, 2, 6, 2, 6, 5, 7, 5, 7]\n sampled = list(sampler)\n # Check that the length is doubled\n assert len(sampled) == 2 * len(dataset)\n # Check that all indexes are present\n assert set(sampled) == set(range(len(dataset)))\n # Check that each element is repeated as expected\n assert all(sampled[i : i + 1] == sampled[i + 2 : i + 3] for i in range(0, len(sampled), 4))\n\n def test_sampler_with_batch_size_and_drop(self):\n dataset = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\"]\n sampler = RepeatSampler(dataset, mini_repeat_count=1, batch_size=2, repeat_count=2)\n # Should output something like [4, 3, 4, 3, 0, 1, 0, 1, 2, 6, 2, 6]\n sampled = list(sampler)\n # Check that the length is doubled\n assert len(sampled) == 2 * (\n len(dataset) - 1\n ) # one element is dropped, because it's not enough to form a batch\n assert len(sampler) == len(sampled) # the length should be the same as the sampled length\n # Check that the sampled indexes are a subset of the dataset indexes\n assert set(sampled).issubset(set(range(len(dataset))))\n # Check that each element is repeated as expected\n assert all(sampled[i : i + 1] == sampled[i + 2 : i + 3] for i in range(0, len(sampled), 4))\n\n def test_sampler_with_mini_repeat_count_and_batch_size_1(self):\n dataset = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\"]\n sampler = RepeatSampler(dataset, mini_repeat_count=2, batch_size=3, repeat_count=2)\n # Should output something like [4, 4, 3, 3, 0, 0, 4, 4, 3, 3, 0, 0,\n # 1, 1, 2, 2, 6, 6, 1, 1, 2, 2, 6, 6]\n sampled = list(sampler)\n # Check that the length is quadrupled\n assert len(sampled) == 4 * (len(dataset) - 1) # 1 element is dropped, because it's not enough to form a batch\n assert len(sampler) == len(sampled) # the length should be the same as the sampled length\n # Check that the sampled indexes are a subset of the dataset indexes\n assert set(sampled).issubset(set(range(len(dataset))))\n # Check that each element is repeated as expected\n assert all(sampled[i] == sampled[i + 1] for i in range(0, len(sampled), 2))\n # Check that the batch is repeated as expected\n assert sampled[0:6] == sampled[6:12]\n assert sampled[12:18] == sampled[18:24]\n\n def test_sampler_with_mini_repeat_count_and_batch_size_2(self):\n dataset = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\"]\n sampler = RepeatSampler(dataset, mini_repeat_count=3, batch_size=2, repeat_count=2)\n # Should output something like [4, 4, 4, 3, 3, 3, 4, 4, 4, 3, 3, 3,\n # 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1,\n # 2, 2, 2, 6, 6, 6, 2, 2, 2, 6, 6, 6]\n sampled = list(sampler)\n # Check that the length is sextupled\n assert len(sampled) == 6 * (len(dataset) - 1) # 1 element is dropped, because it's not enough to form a batch\n assert len(sampler) == len(sampled) # the length should be the same as the sampled length\n # Check that the sampled indexes are a subset of the dataset indexes\n assert set(sampled).issubset(set(range(len(dataset))))\n # Check that each element is repeated as expected\n assert all(sampled[i] == sampled[i + 1] == sampled[i + 2] for i in range(0, len(sampled), 3))\n # Check that the batch is repeated as expected\n assert sampled[0:6] == sampled[6:12]\n assert sampled[12:18] == sampled[18:24]\n assert sampled[24:30] == sampled[30:36]\n\n def test_sampler_with_mini_repeat_count_and_batch_size_3(self):\n dataset = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\"]\n sampler = RepeatSampler(dataset, mini_repeat_count=2, batch_size=2, repeat_count=3)\n # Should output something like [4, 4, 3, 3, 4, 4, 3, 3, 4, 4, 3, 3,\n # 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1,\n # 2, 2, 6, 6, 2, 2, 6, 6, 2, 2, 6, 6]\n sampled = list(sampler)\n # Check that the length is sextupled\n assert len(sampled) == 6 * (len(dataset) - 1) # 1 element is dropped, because it's not enough to form a batch\n # Check that the sampled indexes are a subset of the dataset indexes\n assert set(sampled).issubset(set(range(len(dataset))))\n # Check that each element is repeated as expected\n assert all(sampled[i] == sampled[i + 1] for i in range(0, len(sampled), 2))\n # Check that the batch is repeated as expected\n assert sampled[0:4] == sampled[4:8] == sampled[8:12]\n assert sampled[12:16] == sampled[16:20] == sampled[20:24]\n assert sampled[24:28] == sampled[28:32] == sampled[32:36]\n\n\nclass TestEntropyFromLogits(TrlTestCase):\n @pytest.mark.parametrize(\"shape\", [(768,), (32, 768), (8, 16, 768), (2, 4, 8, 768)])\n @pytest.mark.parametrize(\"chunk_size\", [1, 16])\n @pytest.mark.parametrize(\"dtype\", [torch.float64, torch.float32, torch.float16, torch.bfloat16])\n def test_entropy_from_logits_2_dims(self, dtype, chunk_size, shape):\n logits = torch.randn(*shape, dtype=dtype)\n if dtype in (torch.float64, torch.float32):\n p = logits.softmax(-1)\n entropy = -torch.sum(p * p.log(), dim=-1)\n else:\n logps = logits.log_softmax(dim=-1)\n entropy = -(torch.exp(logps) * logps).sum(-1)\n predicted_entropy = entropy_from_logits(logits, chunk_size=chunk_size)\n torch.testing.assert_close(predicted_entropy, entropy, rtol=1e-5, atol=1e-5)\n\n\n@require_rich\nclass TestPrintPromptCompletionsSample(TrlTestCase):\n @patch(\"sys.stdout\", new_callable=StringIO)\n def test_print_output(self, mock_stdout):\n prompts = [\"The sky is\", \"The sun is\"]\n completions = [\" blue.\", \" in the sky.\"]\n rewards = {\"Correctness\": [0.123, 0.456], \"Format\": [0.789, 0.101]}\n advantages = [0.987, 0.654]\n step = 42\n\n print_prompt_completions_sample(prompts, completions, rewards, advantages, step)\n\n output = mock_stdout.getvalue()\n\n # docstyle-ignore\n expected_output = textwrap.dedent(\"\"\"\\\n โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Step 42 โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ\n โ”‚ โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”‚\n โ”‚ โ”ƒ Prompt โ”ƒ Completion โ”ƒ Correctness โ”ƒ Format โ”ƒ Advantage โ”ƒ โ”‚\n โ”‚ โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ โ”‚\n โ”‚ โ”‚ The sky is โ”‚ blue. โ”‚ 0.12 โ”‚ 0.79 โ”‚ 0.99 โ”‚ โ”‚\n โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚\n โ”‚ โ”‚ The sun is โ”‚ in the sky. โ”‚ 0.46 โ”‚ 0.10 โ”‚ 0.65 โ”‚ โ”‚\n โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚\n โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ\n \"\"\")\n\n assert output == expected_output\n\n @patch(\"sys.stdout\", new_callable=StringIO)\n def test_num_samples(self, mock_stdout):\n prompts = [\"A\", \"B\"]\n completions = [\"1\", \"2\"]\n rewards = {\"Score\": [0.1, 0.2]}\n advantages = [0.3, 0.4]\n step = 10\n\n print_prompt_completions_sample(prompts, completions, rewards, advantages, step, num_samples=1)\n output = mock_stdout.getvalue()\n\n # docstyle-ignore\n possible_outputs = [\n textwrap.dedent(\"\"\"\\\n โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Step 10 โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ\n โ”‚ โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”‚\n โ”‚ โ”ƒ Prompt โ”ƒ Completion โ”ƒ Score โ”ƒ Advantage โ”ƒ โ”‚\n โ”‚ โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ โ”‚\n โ”‚ โ”‚ A โ”‚ 1 โ”‚ 0.10 โ”‚ 0.30 โ”‚ โ”‚\n โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚\n โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ\n \"\"\"),\n # docstyle-ignore\n textwrap.dedent(\"\"\"\\\n โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Step 10 โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ\n โ”‚ โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”‚\n โ”‚ โ”ƒ Prompt โ”ƒ Completion โ”ƒ Score โ”ƒ Advantage โ”ƒ โ”‚\n โ”‚ โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ โ”‚\n โ”‚ โ”‚ B โ”‚ 2 โ”‚ 0.20 โ”‚ 0.40 โ”‚ โ”‚\n โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚\n โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ\n \"\"\"),\n ]\n assert output in possible_outputs\n\n @patch(\"sys.stdout\", new_callable=StringIO)\n def test_print_messages(self, mock_stdout):\n prompts = [\n [\n {\"role\": \"system\", \"content\": \"You are an helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ],\n [\n {\"role\": \"system\", \"content\": \"You are an helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Where is the sun?\"},\n ],\n ]\n completions = [\n [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n [{\"role\": \"assistant\", \"content\": \"In the sky.\"}],\n ]\n rewards = {\"Correctness\": [0.123, 0.456], \"Format\": [0.789, 0.101]}\n advantages = [0.987, 0.654]\n step = 42\n\n print_prompt_completions_sample(prompts, completions, rewards, advantages, step)\n\n output = mock_stdout.getvalue()\n\n # docstyle-ignore\n expected_output = textwrap.dedent(\"\"\"\\\n โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Step 42 โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ\n โ”‚ โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”‚\n โ”‚ โ”ƒ Prompt โ”ƒ Completion โ”ƒ Correctness โ”ƒ Format โ”ƒ Advantage โ”ƒ โ”‚\n โ”‚ โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ โ”‚\n โ”‚ โ”‚ SYSTEM โ”‚ ASSISTANT โ”‚ 0.12 โ”‚ 0.79 โ”‚ 0.99 โ”‚ โ”‚\n โ”‚ โ”‚ You are an helpful โ”‚ It is blue. โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ assistant. โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ USER โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ What color is the sky? โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚\n โ”‚ โ”‚ SYSTEM โ”‚ ASSISTANT โ”‚ 0.46 โ”‚ 0.10 โ”‚ 0.65 โ”‚ โ”‚\n โ”‚ โ”‚ You are an helpful โ”‚ In the sky. โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ assistant. โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ USER โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ Where is the sun? โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚\n โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ\n \"\"\")\n\n assert output == expected_output\n\n @patch(\"sys.stdout\", new_callable=StringIO)\n def test_print_messages_with_tools(self, mock_stdout):\n prompts = [\n [{\"role\": \"user\", \"content\": \"What is the temperature in Paris?\"}],\n [{\"role\": \"user\", \"content\": \"What is the weather in London?\"}],\n ]\n completions = [\n [{\"role\": \"tool\", \"name\": \"get_temperature\", \"args\": {\"location\": \"Paris\"}}],\n [{\"role\": \"tool\", \"name\": \"get_weather\", \"args\": {\"location\": \"London\"}}],\n ]\n rewards = {\"Correctness\": [0.123, 0.456], \"Format\": [0.789, 0.101]}\n advantages = [0.987, 0.654]\n step = 42\n\n print_prompt_completions_sample(prompts, completions, rewards, advantages, step)\n\n output = mock_stdout.getvalue()\n\n # docstyle-ignore\n expected_output = textwrap.dedent(\"\"\"\\\n โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Step 42 โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ\n โ”‚ โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”‚\n โ”‚ โ”ƒ Prompt โ”ƒ Completion โ”ƒ Correctness โ”ƒ Format โ”ƒ Advantage โ”ƒ โ”‚\n โ”‚ โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ โ”‚\n โ”‚ โ”‚ USER โ”‚ TOOL โ”‚ 0.12 โ”‚ 0.79 โ”‚ 0.99 โ”‚ โ”‚\n โ”‚ โ”‚ What is the โ”‚ get_temperature(โ€ฆ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ temperature in โ”‚ 'Paris'}) โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ Paris? โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚\n โ”‚ โ”‚ USER โ”‚ TOOL โ”‚ 0.46 โ”‚ 0.10 โ”‚ 0.65 โ”‚ โ”‚\n โ”‚ โ”‚ What is the โ”‚ get_weather({'loโ€ฆ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ weather in โ”‚ 'London'}) โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ”‚ London? โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚\n โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚\n โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ\n \"\"\")\n\n assert output == expected_output\n\n\nclass TestSelectiveLogSoftmax(TrlTestCase):\n @pytest.mark.parametrize(\"dtype\", [torch.float64, torch.float32, torch.float16, torch.bfloat16])\n def test_selective_log_softmax(self, dtype):\n \"\"\"Test selective_log_softmax with logits of different dtypes\"\"\"\n vocab_size = 1024\n batch_size = 4\n seq_len = 32\n\n input_ids = torch.randint(low=0, high=vocab_size, size=(batch_size, seq_len))\n logits = torch.randn(batch_size, seq_len, vocab_size, dtype=dtype)\n\n expected_output = torch.gather(logits.log_softmax(-1), dim=-1, index=input_ids.unsqueeze(-1)).squeeze(-1)\n actual_output = selective_log_softmax(logits, input_ids)\n\n if dtype in [torch.float16, torch.bfloat16]:\n # half-precision dtypes fall back to an exact method\n assert torch.equal(actual_output, expected_output)\n else:\n torch.testing.assert_close(actual_output, expected_output, rtol=1e-5, atol=1e-5)\n\n @pytest.mark.parametrize(\"dtype\", [torch.float64, torch.float32, torch.float16, torch.bfloat16])\n @pytest.mark.parametrize(\"k\", [1, 8])\n def test_selective_log_softmax_multi_index(self, dtype, k):\n \"\"\"Test selective_log_softmax with logits of different dtypes and index widths\"\"\"\n vocab_size = 1024\n batch_size = 4\n seq_len = 32\n\n index = torch.randint(low=0, high=vocab_size, size=(batch_size, seq_len, k))\n logits = torch.randn(batch_size, seq_len, vocab_size, dtype=dtype)\n\n expected_output = torch.gather(logits.log_softmax(-1), dim=-1, index=index)\n actual_output = selective_log_softmax(logits, index)\n\n assert actual_output.shape == (batch_size, seq_len, k)\n if dtype in [torch.float16, torch.bfloat16]:\n # half-precision dtypes fall back to an exact method\n assert torch.equal(actual_output, expected_output)\n else:\n torch.testing.assert_close(actual_output, expected_output, rtol=1e-5, atol=1e-5)\n\n\nclass TestShuffleSequenceDict(TrlTestCase):\n def test_shuffle_preserves_shape(self):\n x = torch.arange(6).reshape(3, 2)\n y = torch.arange(3).reshape(3, 1)\n tensor_dict = {\"x\": x.clone(), \"y\": y.clone()}\n\n shuffled = shuffle_sequence_dict(tensor_dict)\n\n assert shuffled[\"x\"].shape == x.shape\n assert shuffled[\"y\"].shape == y.shape\n\n def test_shuffle_consistent_across_tensors(self):\n # Use known patterns to check alignment\n x = torch.tensor([[10, 11], [20, 21], [30, 31]])\n y = torch.tensor([[1], [2], [3]])\n tensor_dict = {\"x\": x.clone(), \"y\": y.clone()}\n\n shuffled = shuffle_sequence_dict(tensor_dict)\n\n # Build a reverse map from shuffled x rows to y values\n for i in range(3):\n x_row = shuffled[\"x\"][i]\n y_val = shuffled[\"y\"][i].item()\n\n if torch.equal(x_row, torch.tensor([10, 11])):\n assert y_val == 1\n elif torch.equal(x_row, torch.tensor([20, 21])):\n assert y_val == 2\n elif torch.equal(x_row, torch.tensor([30, 31])):\n assert y_val == 3\n else:\n pytest.fail(\"Unexpected x row in shuffled output.\")\n\n def test_none_tensor_remains_none(self):\n x = torch.arange(6).reshape(3, 2)\n tensor_dict = {\"x\": x.clone(), \"y\": None}\n\n shuffled = shuffle_sequence_dict(tensor_dict)\n\n assert shuffled[\"y\"] is None\n assert shuffled[\"x\"].shape == x.shape\n\n def test_shuffle_with_list(self):\n x = torch.tensor([[10, 11], [20, 21], [30, 31]])\n y = [\"a\", \"b\", \"c\"]\n\n sequence_dict = {\"x\": x.clone(), \"y\": y}\n\n shuffled = shuffle_sequence_dict(sequence_dict)\n\n # Check that the list y is shuffled in the same order as x\n for i in range(3):\n x_row = shuffled[\"x\"][i]\n y_val = shuffled[\"y\"][i]\n\n if torch.equal(x_row, torch.tensor([10, 11])):\n assert y_val == \"a\"\n elif torch.equal(x_row, torch.tensor([20, 21])):\n assert y_val == \"b\"\n elif torch.equal(x_row, torch.tensor([30, 31])):\n assert y_val == \"c\"\n else:\n pytest.fail(\"Unexpected x row in shuffled output.\")\n\n\nclass TestSplitTensorDict(TrlTestCase):\n def test_split_equal_chunks(self):\n x = torch.arange(12).reshape(6, 2)\n y = torch.arange(6).reshape(6, 1)\n tensor_dict = {\"x\": x, \"y\": y}\n\n result = split_tensor_dict(tensor_dict, 3)\n\n expected_x_chunks = torch.chunk(x, 3, dim=0)\n expected_y_chunks = torch.chunk(y, 3, dim=0)\n assert len(result) == 3\n for i in range(3):\n assert torch.equal(result[i][\"x\"], expected_x_chunks[i])\n assert torch.equal(result[i][\"y\"], expected_y_chunks[i])\n\n def test_with_none_tensor(self):\n x = torch.arange(12).reshape(6, 2)\n tensor_dict = {\"x\": x, \"y\": None}\n\n result = split_tensor_dict(tensor_dict, 2)\n\n expected_x_chunks = torch.chunk(x, 2, dim=0)\n assert len(result) == 2\n for i in range(2):\n assert torch.equal(result[i][\"x\"], expected_x_chunks[i])\n assert result[i][\"y\"] is None\n\n def test_with_scalar(self):\n x = torch.arange(12).reshape(6, 2)\n tensor_dict = {\"x\": x, \"y\": torch.tensor(1)}\n\n result = split_tensor_dict(tensor_dict, 2)\n\n expected_x_chunks = torch.chunk(x, 2, dim=0)\n assert len(result) == 2\n for i in range(2):\n assert torch.equal(result[i][\"x\"], expected_x_chunks[i])\n assert torch.equal(result[i][\"y\"], torch.tensor(1))\n\n\nclass TestSplitPixelValuesByGrid(TrlTestCase):\n def test_split_correctly_0(self):\n batch = {\n \"image_grid_thw\": torch.tensor([[1, 2, 2], [1, 2, 2]]),\n \"num_images\": [1, 1],\n \"pixel_values\": torch.arange(8 * 3).reshape(8, 3), # Shape: [8, 3]\n }\n result = split_pixel_values_by_grid(batch)\n assert isinstance(result[\"pixel_values\"], list)\n assert len(result[\"pixel_values\"]) == 2\n assert torch.equal(result[\"pixel_values\"][0], batch[\"pixel_values\"][:4])\n assert torch.equal(result[\"pixel_values\"][1], batch[\"pixel_values\"][4:])\n assert isinstance(result[\"image_grid_thw\"], list)\n assert len(result[\"image_grid_thw\"]) == 2\n assert torch.equal(result[\"image_grid_thw\"][0], torch.tensor([[1, 2, 2]]))\n assert torch.equal(result[\"image_grid_thw\"][1], torch.tensor([[1, 2, 2]]))\n\n def test_split_correctly_1(self):\n batch = {\n \"image_grid_thw\": torch.tensor([[1, 2, 2], [1, 2, 4]]),\n \"num_images\": [1, 1],\n \"pixel_values\": torch.arange(12 * 3).reshape(12, 3), # Shape: [12, 3]\n }\n result = split_pixel_values_by_grid(batch)\n assert isinstance(result[\"pixel_values\"], list)\n assert len(result[\"pixel_values\"]) == 2\n assert torch.equal(result[\"pixel_values\"][0], batch[\"pixel_values\"][:4])\n assert torch.equal(result[\"pixel_values\"][1], batch[\"pixel_values\"][4:12])\n assert isinstance(result[\"image_grid_thw\"], list)\n assert len(result[\"image_grid_thw\"]) == 2\n assert torch.equal(result[\"image_grid_thw\"][0], torch.tensor([[1, 2, 2]]))\n assert torch.equal(result[\"image_grid_thw\"][1], torch.tensor([[1, 2, 4]]))\n\n def test_missing_keys(self):\n batch = {\"pixel_values\": torch.tensor([1.0])}\n result = split_pixel_values_by_grid(batch)\n assert result == batch\n\n def test_mismatched_length(self):\n batch = {\n \"image_grid_thw\": torch.tensor([[1, 1, 2], [1, 2, 1]]), # Total = 8\n \"num_images\": [1, 1],\n \"pixel_values\": torch.randn(3, 5), # Only 3 rows\n }\n with pytest.raises(ValueError):\n split_pixel_values_by_grid(batch)\n\n def test_multi_images(self):\n batch = {\n \"image_grid_thw\": torch.tensor([[1, 1, 2], [1, 2, 2], [1, 2, 1]]), # Total = 8\n \"num_images\": [1, 2],\n \"pixel_values\": torch.arange(8 * 3).reshape(8, 3), # Shape: [8, 3]\n }\n result = split_pixel_values_by_grid(batch)\n assert isinstance(result[\"pixel_values\"], list)\n assert len(result[\"pixel_values\"]) == 2\n assert torch.equal(result[\"pixel_values\"][0], batch[\"pixel_values\"][:2])\n assert torch.equal(result[\"pixel_values\"][1], batch[\"pixel_values\"][2:])\n assert isinstance(result[\"image_grid_thw\"], list)\n assert len(result[\"image_grid_thw\"]) == 2\n assert torch.equal(result[\"image_grid_thw\"][0], torch.tensor([[1, 1, 2]]))\n assert torch.equal(result[\"image_grid_thw\"][1], torch.tensor([[1, 2, 2], [1, 2, 1]]))\n\n\nclass TestUnsplitPixelValuesByGrid(TrlTestCase):\n def test_unsplit_correctly(self):\n pixel_values = [torch.randn(4, 5), torch.randn(2, 5)]\n pixel_values_merged = torch.cat(pixel_values, dim=0)\n image_grid_thw = [torch.tensor([[1, 2, 2]]), torch.tensor([[1, 2, 1]])]\n image_grid_thw_merged = torch.cat(image_grid_thw, dim=0)\n batch = {\"pixel_values\": pixel_values, \"image_grid_thw\": image_grid_thw, \"other_key\": torch.tensor([1])}\n result = unsplit_pixel_values_by_grid(batch)\n assert isinstance(result[\"pixel_values\"], torch.Tensor)\n torch.testing.assert_close(result[\"pixel_values\"], pixel_values_merged)\n assert isinstance(result[\"image_grid_thw\"], torch.Tensor)\n assert torch.equal(result[\"image_grid_thw\"], image_grid_thw_merged)\n assert \"other_key\" in result\n\n def test_no_op_if_not_list(self):\n original = torch.randn(5, 3)\n batch = {\"pixel_values\": original}\n result = unsplit_pixel_values_by_grid(batch)\n assert torch.equal(result[\"pixel_values\"], original)\n\n\nclass TestForwardMaskedLogits:\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-CohereForCausalLM\",\n \"trl-internal-testing/tiny-Cohere2ForCausalLM\",\n \"trl-internal-testing/tiny-DeepseekV3ForCausalLM\",\n \"trl-internal-testing/tiny-DeepseekV3ForCausalLM-0528\",\n \"trl-internal-testing/tiny-Gemma2ForCausalLM\",\n \"trl-internal-testing/tiny-GemmaForCausalLM\",\n \"trl-internal-testing/tiny-Glm4MoeForCausalLM\",\n \"trl-internal-testing/tiny-GptOssForCausalLM\",\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.1\",\n \"trl-internal-testing/tiny-LlamaForCausalLM-3.2\",\n \"trl-internal-testing/tiny-LlamaForCausalLM-3\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.1\",\n \"trl-internal-testing/tiny-MistralForCausalLM-0.2\",\n \"trl-internal-testing/tiny-Phi3ForCausalLM\",\n \"trl-internal-testing/tiny-Qwen2ForCausalLM-2.5\",\n \"trl-internal-testing/tiny-Qwen3ForCausalLM\",\n ],\n )\n def test_llm(self, model_id):\n device = torch.device(\"cuda\")\n model = AutoModelForCausalLM.from_pretrained(model_id, dtype=\"auto\", device_map=device)\n input_ids = torch.randint(0, model.config.vocab_size, (2, 8), device=device)\n logits_mask = torch.tensor(\n [[1, 1, 0, 0, 1, 0, 1, 0], [0, 1, 1, 0, 0, 1, 0, 1]],\n device=device,\n )\n\n full_outputs = model(input_ids=input_ids)\n masked_outputs = forward_masked_logits(model, logits_mask, input_ids=input_ids)\n\n torch.testing.assert_close(\n masked_outputs.flat_logits,\n full_outputs.logits[logits_mask.bool()],\n )\n\n @pytest.mark.parametrize(\n \"model_id\",\n [\n \"trl-internal-testing/tiny-Gemma3ForConditionalGeneration\",\n \"trl-internal-testing/tiny-Idefics2ForConditionalGeneration\",\n \"trl-internal-testing/tiny-Idefics3ForConditionalGeneration\",\n \"trl-internal-testing/tiny-LlavaForConditionalGeneration\",\n \"trl-internal-testing/tiny-LlavaNextForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2VLForConditionalGeneration\",\n \"trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration\",\n # \"trl-internal-testing/tiny-SmolVLMForConditionalGeneration\", seems not to support bf16 properly\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3VLForConditionalGeneration\",\n marks=[\n pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"4.57.0\"),\n reason=\"Qwen3-VL series were introduced in transformers-4.57.0\",\n ),\n pytest.mark.xfail(\n Version(\"5.0.0\") <= Version(transformers.__version__) < Version(\"5.1.0\"),\n reason=\"Upstream transformers bug (transformers#43334) in 5.0.x; fixed in 5.1.0\",\n ),\n ],\n ),\n pytest.param(\n \"trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration\",\n marks=pytest.mark.skipif(\n Version(transformers.__version__) < Version(\"5.2.0\"),\n reason=\"Qwen3.5 models were introduced in transformers-5.2.0\",\n ),\n ),\n ],\n )\n def test_vlm(self, model_id):\n device = torch.device(\"cuda\")\n model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=\"auto\", device_map=device)\n input_ids = torch.randint(0, model.config.text_config.vocab_size, (2, 8), device=device)\n logits_mask = torch.tensor(\n [[1, 1, 0, 0, 1, 0, 1, 0], [0, 1, 1, 0, 0, 1, 0, 1]],\n device=device,\n )\n\n full_outputs = model(input_ids=input_ids)\n masked_outputs = forward_masked_logits(model, logits_mask, input_ids=input_ids)\n\n torch.testing.assert_close(\n masked_outputs.flat_logits,\n full_outputs.logits[logits_mask.bool()],\n )\n" }, { "path": "tests/test_vllm_client_server.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport os\nimport subprocess\nfrom types import SimpleNamespace\n\nimport pytest\nfrom packaging.version import Version\nfrom transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer\nfrom transformers.testing_utils import torch_device\n\nfrom trl.generation.vllm_client import VLLMClient\nfrom trl.generation.vllm_generation import extract_logprobs\nfrom trl.import_utils import is_vllm_available\nfrom trl.scripts.vllm_serve import chunk_list\n\nfrom .testing_utils import (\n TrlTestCase,\n kill_process,\n require_3_accelerators,\n require_torch_multi_accelerator,\n require_vision,\n require_vllm,\n)\n\n\nif is_vllm_available():\n import vllm\n from vllm import LLM, SamplingParams\n\n _is_vllm_ge_014 = Version(vllm.__version__) >= Version(\"0.14.0\")\nelse:\n _is_vllm_ge_014 = False\n\n\nclass TestChunkList(TrlTestCase):\n def test_even_split(self):\n assert chunk_list([1, 2, 3, 4, 5, 6], 2) == [[1, 2, 3], [4, 5, 6]]\n\n def test_uneven_split(self):\n assert chunk_list([1, 2, 3, 4, 5, 6], 4) == [[1, 2], [3, 4], [5], [6]]\n\n def test_more_chunks_than_elements(self):\n assert chunk_list([1, 2, 3, 4, 5, 6], 8) == [[1], [2], [3], [4], [5], [6], [], []]\n\n def test_n_equals_len(self):\n assert chunk_list([1, 2, 3], 3) == [[1], [2], [3]]\n\n def test_n_is_1(self):\n assert chunk_list([1, 2, 3], 1) == [[1, 2, 3]]\n\n def test_single_element_list(self):\n assert chunk_list([42], 2) == [[42], []]\n\n def test_any_dtype(self):\n assert chunk_list([1, \"two\", 3.0, {\"four\": 4}, [\"f\", \"i\", \"v\", \"e\"]], 2) == [\n [1, \"two\", 3.0],\n [{\"four\": 4}, [\"f\", \"i\", \"v\", \"e\"]],\n ]\n\n\nclass TestExtractLogprobs(TrlTestCase):\n def test_extract_logprobs_sorts_by_rank_and_replaces_nan(self):\n all_outputs = [\n SimpleNamespace(\n outputs=[\n SimpleNamespace(\n logprobs=[\n {\n 11: SimpleNamespace(rank=1, logprob=-0.2),\n 99: SimpleNamespace(rank=0, logprob=-0.1),\n 42: SimpleNamespace(rank=2, logprob=float(\"nan\")),\n },\n {\n 5: SimpleNamespace(rank=0, logprob=-1.1),\n },\n ]\n )\n ]\n ),\n SimpleNamespace(\n outputs=[\n SimpleNamespace(\n logprobs=[\n {\n 3: SimpleNamespace(rank=1, logprob=-0.5),\n 7: SimpleNamespace(rank=0, logprob=-0.4),\n }\n ]\n )\n ]\n ),\n ]\n\n all_logprobs, all_token_ids = extract_logprobs(all_outputs)\n\n assert all_token_ids == [\n [[99, 11, 42], [5]],\n [[7, 3]],\n ]\n assert all_logprobs == [\n [[-0.1, -0.2, None], [-1.1]],\n [[-0.4, -0.5]],\n ]\n\n def test_extract_logprobs_returns_none_token_ids_when_logprobs_missing(self):\n all_outputs = [SimpleNamespace(outputs=[SimpleNamespace(logprobs=None)])]\n\n all_logprobs, all_token_ids = extract_logprobs(all_outputs)\n\n assert all_logprobs is None\n assert all_token_ids is None\n\n\n@pytest.mark.slow\n@require_torch_multi_accelerator\n@require_vllm\nclass TestVLLMClientServer(TrlTestCase):\n model_id = \"Qwen/Qwen2.5-1.5B\"\n\n @classmethod\n def setup_class(cls):\n # We want the server to run on accelerator 1, so we set VISIBLE_DEVICES to \"1\"\n env = os.environ.copy()\n VISIBLE_DEVICES = \"ZE_AFFINITY_MASK\" if torch_device == \"xpu\" else \"CUDA_VISIBLE_DEVICES\"\n env[VISIBLE_DEVICES] = \"1\" # Restrict to accelerator 1\n\n # Start the server process\n cls.server_process = subprocess.Popen(\n [\"trl\", \"vllm-serve\", \"--model\", cls.model_id], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env\n )\n\n # Initialize the client\n cls.client = VLLMClient(connection_timeout=240, host=\"localhost\")\n cls.client.init_communicator()\n\n def test_generate(self):\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n outputs = self.client.generate(prompts)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of prompts\n assert len(prompt_ids) == len(prompts)\n assert len(completion_ids) == len(prompts)\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_generate_with_logprobs_none(self):\n outputs = self.client.generate([\"Hello, AI!\"], logprobs=None)\n\n assert isinstance(outputs[\"prompt_ids\"], list)\n assert isinstance(outputs[\"completion_ids\"], list)\n assert outputs[\"logprobs\"] is None\n assert outputs[\"logprob_token_ids\"] is None\n\n def test_chat(self):\n messages = [[{\"role\": \"user\", \"content\": \"Hello, AI!\"}], [{\"role\": \"user\", \"content\": \"Tell me a joke\"}]]\n outputs = self.client.chat(messages)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of messages\n assert len(prompt_ids) == len(messages)\n assert len(completion_ids) == len(messages)\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_chat_with_logprobs_none(self):\n outputs = self.client.chat([[{\"role\": \"user\", \"content\": \"Hello, AI!\"}]], logprobs=None)\n\n assert isinstance(outputs[\"prompt_ids\"], list)\n assert isinstance(outputs[\"completion_ids\"], list)\n assert outputs[\"logprobs\"] is None\n assert outputs[\"logprob_token_ids\"] is None\n\n def test_chat_with_tools(self):\n def multiply(a: int, b: int) -> int:\n \"\"\"\n Multiplies two integers.\n\n Args:\n a: The first integer.\n b: The second integer.\n\n Returns:\n The product of the two integers.\n \"\"\"\n return a * b\n\n messages = [[{\"role\": \"user\", \"content\": \"What is 3 multiplied by 4?\"}]]\n outputs = self.client.chat(messages, tools=[multiply])\n\n # Decode prompt and check that \"Multiplies two integers.\" is in the prompt.\n tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n decoded_prompt = tokenizer.decode(outputs[\"prompt_ids\"][0])\n assert \"Multiplies two integers.\" in decoded_prompt\n\n def test_generate_with_token_ids(self):\n tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n prompt_token_ids = tokenizer(prompts)[\"input_ids\"]\n outputs = self.client.generate(prompt_token_ids)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of prompts\n assert len(prompt_ids) == len(prompts)\n assert len(completion_ids) == len(prompts)\n\n # Check that prompt_ids match the input token IDs\n assert prompt_ids == prompt_token_ids\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_generate_with_params(self):\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n completion_ids = self.client.generate(prompts, n=2, repetition_penalty=0.9, temperature=0.8, max_tokens=32)[\n \"completion_ids\"\n ]\n\n # Check that the output is a list\n assert isinstance(completion_ids, list)\n\n # Check that the number of generated sequences is 2 times the number of prompts\n assert len(completion_ids) == 2 * len(prompts)\n\n # Check that the generated sequences are lists of integers\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n # Check that the length of the generated sequences is less than or equal to 32\n for seq in completion_ids:\n assert len(seq) <= 32\n\n def test_update_model_params(self):\n model = AutoModelForCausalLM.from_pretrained(self.model_id, device_map=torch_device)\n self.client.update_model_params(model)\n\n def test_reset_prefix_cache(self):\n # Test resetting the prefix cache\n self.client.reset_prefix_cache()\n\n @pytest.mark.xfail(reason=\"Importing `bitsandbytes` causes issues, see vllm-project/vllm#32793\")\n def test_logprobs_match_with_non_default_sampling(self):\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n # Use non-default sampling parameters (especially temperature) to ensure vLLM applies logprob processing. With\n # default sampling, raw and processed logprobs are identical, so mismatches would not be detected.\n temperature = 0.7\n repetition_penalty = 1.05\n top_p = 0.9\n max_tokens = 8\n seed = 1234\n num_logprobs = 5\n\n server_outputs = self.client.generate(\n prompts,\n temperature=temperature,\n repetition_penalty=repetition_penalty,\n top_p=top_p,\n max_tokens=max_tokens,\n logprobs=num_logprobs,\n generation_kwargs={\"seed\": seed},\n )\n os.environ[\"VLLM_WORKER_MULTIPROC_METHOD\"] = \"spawn\"\n llm = LLM(\n model=self.model_id,\n tensor_parallel_size=1,\n gpu_memory_utilization=0.2,\n max_model_len=128,\n logprobs_mode=\"processed_logprobs\",\n )\n\n sampling_params = SamplingParams(\n temperature=temperature,\n repetition_penalty=repetition_penalty,\n top_p=top_p,\n max_tokens=max_tokens,\n logprobs=num_logprobs,\n seed=seed,\n )\n colocate_outputs = llm.generate(prompts, sampling_params=sampling_params, use_tqdm=False)\n colocate_prompt_ids = [output.prompt_token_ids for output in colocate_outputs]\n colocate_completion_ids = [\n list(output.token_ids) for outputs in colocate_outputs for output in outputs.outputs\n ]\n colocate_logprobs, colocate_logprob_token_ids = extract_logprobs(colocate_outputs)\n\n # Generation correctness: prompt and completion IDs match between server and colocate\n assert server_outputs[\"prompt_ids\"] == colocate_prompt_ids\n assert server_outputs[\"completion_ids\"] == colocate_completion_ids\n\n server_logprobs = server_outputs[\"logprobs\"]\n server_logprob_token_ids = server_outputs[\"logprob_token_ids\"]\n\n # Shape: both should be (num_sequences, seq_len, num_logprobs) with multiple logprobs per token\n assert len(server_logprobs) == len(prompts)\n assert len(server_logprob_token_ids) == len(prompts)\n for seq_lps in server_logprobs:\n for token_lps in seq_lps:\n assert len(token_lps) > 1, \"Expected multiple logprobs per token when logprobs > 0\"\n\n # Value correctness: server extraction matches colocate extraction via extract_logprobs\n assert server_logprob_token_ids == colocate_logprob_token_ids\n for server_seq, colocate_seq in zip(server_logprobs, colocate_logprobs, strict=True):\n assert len(server_seq) == len(colocate_seq)\n for server_token_lps, colocate_token_lps in zip(server_seq, colocate_seq, strict=True):\n assert server_token_lps == pytest.approx(colocate_token_lps, rel=1e-6, abs=1e-6)\n\n # Ordering: logprobs at each position should be sorted descending\n for seq_lps in server_logprobs:\n for token_lps in seq_lps:\n assert token_lps == sorted(token_lps, reverse=True), \"Logprobs should be sorted descending\"\n\n # Sampled token presence: the actual completion token should appear in the logprob token IDs\n for seq_idx, (completion_seq, token_ids_seq) in enumerate(\n zip(server_outputs[\"completion_ids\"], server_logprob_token_ids, strict=True)\n ):\n for pos, (sampled_id, lp_ids) in enumerate(zip(completion_seq, token_ids_seq, strict=True)):\n assert sampled_id in lp_ids, (\n f\"Sampled token {sampled_id} not found in logprob token IDs {lp_ids} \"\n f\"at sequence {seq_idx}, position {pos}\"\n )\n\n @classmethod\n def teardown_class(cls):\n # Close the client\n cls.client.close_communicator()\n\n # vLLM x pytest (or Popen) seems not to handle process termination well. To avoid zombie processes, we need to\n # kill the server process and its children explicitly.\n kill_process(cls.server_process)\n\n\n# Same as above but using base_url to instantiate the client.\n@pytest.mark.slow\n@require_torch_multi_accelerator\n@require_vllm\nclass TestVLLMClientServerBaseURL(TrlTestCase):\n model_id = \"Qwen/Qwen2.5-1.5B\"\n\n @classmethod\n def setup_class(cls):\n # We want the server to run on accelerator 1, so we set VISIBLE_DEVICES to \"1\"\n env = os.environ.copy()\n VISIBLE_DEVICES = \"ZE_AFFINITY_MASK\" if torch_device == \"xpu\" else \"CUDA_VISIBLE_DEVICES\"\n env[VISIBLE_DEVICES] = \"1\" # Restrict to accelerator 1\n\n # Start the server process\n cls.server_process = subprocess.Popen(\n [\"trl\", \"vllm-serve\", \"--model\", cls.model_id], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env\n )\n\n # Initialize the client\n cls.client = VLLMClient(base_url=\"http://localhost:8000\", connection_timeout=240)\n cls.client.init_communicator()\n\n def test_generate(self):\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n outputs = self.client.generate(prompts)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of prompts\n assert len(prompt_ids) == len(prompts)\n assert len(completion_ids) == len(prompts)\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_generate_with_logprobs_none(self):\n outputs = self.client.generate([\"Hello, AI!\"], logprobs=None)\n\n assert isinstance(outputs[\"prompt_ids\"], list)\n assert isinstance(outputs[\"completion_ids\"], list)\n assert outputs[\"logprobs\"] is None\n assert outputs[\"logprob_token_ids\"] is None\n\n def test_chat(self):\n messages = [[{\"role\": \"user\", \"content\": \"Hello, AI!\"}], [{\"role\": \"user\", \"content\": \"Tell me a joke\"}]]\n outputs = self.client.chat(messages)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of messages\n assert len(prompt_ids) == len(messages)\n assert len(completion_ids) == len(messages)\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_chat_with_logprobs_none(self):\n outputs = self.client.chat([[{\"role\": \"user\", \"content\": \"Hello, AI!\"}]], logprobs=None)\n\n assert isinstance(outputs[\"prompt_ids\"], list)\n assert isinstance(outputs[\"completion_ids\"], list)\n assert outputs[\"logprobs\"] is None\n assert outputs[\"logprob_token_ids\"] is None\n\n def test_chat_with_tools(self):\n def multiply(a: int, b: int) -> int:\n \"\"\"\n Multiplies two integers.\n\n Args:\n a: The first integer.\n b: The second integer.\n\n Returns:\n The product of the two integers.\n \"\"\"\n return a * b\n\n messages = [[{\"role\": \"user\", \"content\": \"What is 3 multiplied by 4?\"}]]\n outputs = self.client.chat(messages, tools=[multiply])\n\n # Decode prompt and check that \"Multiplies two integers.\" is in the prompt.\n tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n decoded_prompt = tokenizer.decode(outputs[\"prompt_ids\"][0])\n assert \"Multiplies two integers.\" in decoded_prompt\n\n def test_generate_with_token_ids(self):\n tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n prompt_token_ids = tokenizer(prompts)[\"input_ids\"]\n outputs = self.client.generate(prompt_token_ids)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of prompts\n assert len(prompt_ids) == len(prompts)\n assert len(completion_ids) == len(prompts)\n\n # Check that prompt_ids match the input token IDs\n assert prompt_ids == prompt_token_ids\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_generate_with_params(self):\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n completion_ids = self.client.generate(prompts, n=2, repetition_penalty=0.9, temperature=0.8, max_tokens=32)[\n \"completion_ids\"\n ]\n\n # Check that the output is a list\n assert isinstance(completion_ids, list)\n\n # Check that the number of generated sequences is 2 times the number of prompts\n assert len(completion_ids) == 2 * len(prompts)\n\n # Check that the generated sequences are lists of integers\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n # Check that the length of the generated sequences is less than or equal to 32\n for seq in completion_ids:\n assert len(seq) <= 32\n\n def test_update_model_params(self):\n model = AutoModelForCausalLM.from_pretrained(self.model_id, device_map=torch_device)\n self.client.update_model_params(model)\n\n def test_reset_prefix_cache(self):\n # Test resetting the prefix cache\n self.client.reset_prefix_cache()\n\n @classmethod\n def teardown_class(cls):\n # Close the client\n cls.client.close_communicator()\n\n # vLLM x pytest (or Popen) seems not to handle process termination well. To avoid zombie processes, we need to\n # kill the server process and its children explicitly.\n kill_process(cls.server_process)\n\n\n@pytest.mark.slow\n@require_3_accelerators\n@require_vllm\nclass TestVLLMClientServerTP(TrlTestCase):\n model_id = \"Qwen/Qwen2.5-1.5B\"\n\n @classmethod\n def setup_class(cls):\n # We want the server to run on accelerator 1 and 2, so we set VISIBLE_DEVICES to \"1,2\"\n env = os.environ.copy()\n VISIBLE_DEVICES = \"ZE_AFFINITY_MASK\" if torch_device == \"xpu\" else \"CUDA_VISIBLE_DEVICES\"\n env[VISIBLE_DEVICES] = \"1,2\" # Restrict to accelerator 1 and 2\n\n # Start the server process\n cls.server_process = subprocess.Popen(\n [\"trl\", \"vllm-serve\", \"--model\", cls.model_id, \"--tensor_parallel_size\", \"2\"],\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n env=env,\n )\n\n # Initialize the client\n cls.client = VLLMClient(connection_timeout=240, host=\"localhost\")\n cls.client.init_communicator()\n\n def test_generate(self):\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n outputs = self.client.generate(prompts)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of prompts\n assert len(prompt_ids) == len(prompts)\n assert len(completion_ids) == len(prompts)\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_generate_with_logprobs_none(self):\n outputs = self.client.generate([\"Hello, AI!\"], logprobs=None)\n\n assert isinstance(outputs[\"prompt_ids\"], list)\n assert isinstance(outputs[\"completion_ids\"], list)\n assert outputs[\"logprobs\"] is None\n assert outputs[\"logprob_token_ids\"] is None\n\n def test_chat(self):\n messages = [[{\"role\": \"user\", \"content\": \"Hello, AI!\"}], [{\"role\": \"user\", \"content\": \"Tell me a joke\"}]]\n outputs = self.client.chat(messages)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of messages\n assert len(prompt_ids) == len(messages)\n assert len(completion_ids) == len(messages)\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_chat_with_logprobs_none(self):\n outputs = self.client.chat([[{\"role\": \"user\", \"content\": \"Hello, AI!\"}]], logprobs=None)\n\n assert isinstance(outputs[\"prompt_ids\"], list)\n assert isinstance(outputs[\"completion_ids\"], list)\n assert outputs[\"logprobs\"] is None\n assert outputs[\"logprob_token_ids\"] is None\n\n def test_chat_with_tools(self):\n def multiply(a: int, b: int) -> int:\n \"\"\"\n Multiplies two integers.\n\n Args:\n a: The first integer.\n b: The second integer.\n\n Returns:\n The product of the two integers.\n \"\"\"\n return a * b\n\n messages = [[{\"role\": \"user\", \"content\": \"What is 3 multiplied by 4?\"}]]\n outputs = self.client.chat(messages, tools=[multiply])\n\n # Decode prompt and check that \"Multiplies two integers.\" is in the prompt.\n tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n decoded_prompt = tokenizer.decode(outputs[\"prompt_ids\"][0])\n assert \"Multiplies two integers.\" in decoded_prompt\n\n def test_generate_with_token_ids(self):\n tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n prompt_token_ids = tokenizer(prompts)[\"input_ids\"]\n outputs = self.client.generate(prompt_token_ids)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of prompts\n assert len(prompt_ids) == len(prompts)\n assert len(completion_ids) == len(prompts)\n\n # Check that prompt_ids match the input token IDs\n assert prompt_ids == prompt_token_ids\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_generate_with_params(self):\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n completion_ids = self.client.generate(prompts, n=2, repetition_penalty=0.9, temperature=0.8, max_tokens=32)[\n \"completion_ids\"\n ]\n\n # Check that the output is a list\n assert isinstance(completion_ids, list)\n\n # Check that the number of generated sequences is 2 times the number of prompts\n assert len(completion_ids) == 2 * len(prompts)\n\n # Check that the generated sequences are lists of integers\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n # Check that the length of the generated sequences is less than or equal to 32\n for seq in completion_ids:\n assert len(seq) <= 32\n\n def test_update_model_params(self):\n model = AutoModelForCausalLM.from_pretrained(self.model_id, device_map=torch_device)\n self.client.update_model_params(model)\n\n def test_reset_prefix_cache(self):\n # Test resetting the prefix cache\n self.client.reset_prefix_cache()\n\n @classmethod\n def teardown_class(cls):\n # Close the client\n cls.client.close_communicator()\n\n # vLLM x pytest (or Popen) seems not to handle process termination well. To avoid zombie processes, we need to\n # kill the server process and its children explicitly.\n kill_process(cls.server_process)\n\n\n@pytest.mark.slow\n@pytest.mark.skipif(\n _is_vllm_ge_014,\n reason=\"Skipping DP server test for vLLM>=0.14.0 (PR vllm#30739: DP for non-MoE/dense models no longer supported).\",\n)\n@require_3_accelerators\n@require_vllm\nclass TestVLLMClientServerDP(TrlTestCase):\n model_id = \"Qwen/Qwen2.5-1.5B\"\n\n @classmethod\n def setup_class(cls):\n # We want the server to run on accelerator 1 and 2, so we set VISIBLE_DEVICES to \"1,2\"\n env = os.environ.copy()\n VISIBLE_DEVICES = \"ZE_AFFINITY_MASK\" if torch_device == \"xpu\" else \"CUDA_VISIBLE_DEVICES\"\n env[VISIBLE_DEVICES] = \"1,2\" # Restrict to accelerator 1 and 2\n\n # Start the server process\n cls.server_process = subprocess.Popen(\n [\"trl\", \"vllm-serve\", \"--model\", cls.model_id, \"--data_parallel_size\", \"2\"],\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n env=env,\n )\n\n # Initialize the client\n cls.client = VLLMClient(connection_timeout=240, host=\"localhost\")\n cls.client.init_communicator()\n\n def test_generate(self):\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n outputs = self.client.generate(prompts)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of prompts\n assert len(prompt_ids) == len(prompts)\n assert len(completion_ids) == len(prompts)\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_generate_with_logprobs_none(self):\n outputs = self.client.generate([\"Hello, AI!\"], logprobs=None)\n\n assert isinstance(outputs[\"prompt_ids\"], list)\n assert isinstance(outputs[\"completion_ids\"], list)\n assert outputs[\"logprobs\"] is None\n assert outputs[\"logprob_token_ids\"] is None\n\n def test_chat(self):\n messages = [[{\"role\": \"user\", \"content\": \"Hello, AI!\"}], [{\"role\": \"user\", \"content\": \"Tell me a joke\"}]]\n outputs = self.client.chat(messages)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of messages\n assert len(prompt_ids) == len(messages)\n assert len(completion_ids) == len(messages)\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_chat_with_logprobs_none(self):\n outputs = self.client.chat([[{\"role\": \"user\", \"content\": \"Hello, AI!\"}]], logprobs=None)\n\n assert isinstance(outputs[\"prompt_ids\"], list)\n assert isinstance(outputs[\"completion_ids\"], list)\n assert outputs[\"logprobs\"] is None\n assert outputs[\"logprob_token_ids\"] is None\n\n def test_chat_with_tools(self):\n def multiply(a: int, b: int) -> int:\n \"\"\"\n Multiplies two integers.\n\n Args:\n a: The first integer.\n b: The second integer.\n\n Returns:\n The product of the two integers.\n \"\"\"\n return a * b\n\n messages = [[{\"role\": \"user\", \"content\": \"What is 3 multiplied by 4?\"}]]\n outputs = self.client.chat(messages, tools=[multiply])\n\n # Decode prompt and check that \"Multiplies two integers.\" is in the prompt.\n tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n decoded_prompt = tokenizer.decode(outputs[\"prompt_ids\"][0])\n assert \"Multiplies two integers.\" in decoded_prompt\n\n def test_generate_with_token_ids(self):\n tokenizer = AutoTokenizer.from_pretrained(self.model_id)\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n prompt_token_ids = tokenizer(prompts)[\"input_ids\"]\n outputs = self.client.generate(prompt_token_ids)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n # Check that the outputs are lists\n assert isinstance(prompt_ids, list)\n assert isinstance(completion_ids, list)\n\n # Check that the number of sequences are equal to the number of prompts\n assert len(prompt_ids) == len(prompts)\n assert len(completion_ids) == len(prompts)\n\n # Check that prompt_ids match the input token IDs\n assert prompt_ids == prompt_token_ids\n\n # Check that the sequences are lists of integers\n for seq in prompt_ids:\n assert all(isinstance(tok, int) for tok in seq)\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n def test_generate_with_params(self):\n prompts = [\"Hello, AI!\", \"Tell me a joke\"]\n completion_ids = self.client.generate(prompts, n=2, repetition_penalty=0.9, temperature=0.8, max_tokens=32)[\n \"completion_ids\"\n ]\n\n # Check that the output is a list\n assert isinstance(completion_ids, list)\n\n # Check that the number of generated sequences is 2 times the number of prompts\n assert len(completion_ids) == 2 * len(prompts)\n\n # Check that the generated sequences are lists of integers\n for seq in completion_ids:\n assert all(isinstance(tok, int) for tok in seq)\n\n # Check that the length of the generated sequences is less than or equal to 32\n for seq in completion_ids:\n assert len(seq) <= 32\n\n def test_update_model_params(self):\n model = AutoModelForCausalLM.from_pretrained(self.model_id, device_map=torch_device)\n self.client.update_model_params(model)\n\n def test_reset_prefix_cache(self):\n # Test resetting the prefix cache\n self.client.reset_prefix_cache()\n\n @classmethod\n def teardown_class(cls):\n # Close the client\n cls.client.close_communicator()\n\n # vLLM x pytest (or Popen) seems not to handle process termination well. To avoid zombie processes, we need to\n # kill the server process and its children explicitly.\n kill_process(cls.server_process)\n\n\n@pytest.mark.slow\n@require_torch_multi_accelerator\n@require_vllm\nclass TestVLLMClientServerDeviceParameter(TrlTestCase):\n \"\"\"Test the device parameter functionality in init_communicator.\"\"\"\n\n model_id = \"Qwen/Qwen2.5-1.5B\"\n\n @classmethod\n def setup_class(cls):\n # We want the server to run on accelerator 1, so we set VISIBLE_DEVICES to \"1\"\n env = os.environ.copy()\n VISIBLE_DEVICES = \"ZE_AFFINITY_MASK\" if torch_device == \"xpu\" else \"CUDA_VISIBLE_DEVICES\"\n env[VISIBLE_DEVICES] = \"1\" # Restrict to accelerator 1\n\n # Start the server process\n cls.server_process = subprocess.Popen(\n [\"trl\", \"vllm-serve\", \"--model\", cls.model_id], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env\n )\n\n def test_init_communicator_with_device_int(self):\n \"\"\"Test init_communicator with integer device parameter.\"\"\"\n client = VLLMClient(connection_timeout=240, host=\"localhost\")\n client.init_communicator(device=0) # Explicitly specify device 0\n\n # Test basic functionality\n prompts = [\"Hello, AI!\"]\n outputs = client.generate(prompts)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n assert isinstance(prompt_ids, list)\n assert len(prompt_ids) == len(prompts)\n assert isinstance(completion_ids, list)\n assert len(completion_ids) == len(prompts)\n\n client.close_communicator()\n\n def test_init_communicator_with_device_string(self):\n \"\"\"Test init_communicator with string device parameter.\"\"\"\n client = VLLMClient(connection_timeout=240, host=\"localhost\")\n client.init_communicator(device=0) # Explicitly specify device as string\n\n # Test basic functionality\n prompts = [\"Hello, AI!\"]\n outputs = client.generate(prompts)[\"completion_ids\"]\n assert isinstance(outputs, list)\n assert len(outputs) == len(prompts)\n\n client.close_communicator()\n\n def test_init_communicator_with_torch_device(self):\n \"\"\"Test init_communicator with torch.device object.\"\"\"\n import torch\n\n client = VLLMClient(connection_timeout=240, host=\"localhost\")\n device = torch.device(0)\n client.init_communicator(device=device) # Explicitly specify torch.device object\n\n # Test basic functionality\n prompts = [\"Hello, AI!\"]\n outputs = client.generate(prompts)[\"completion_ids\"]\n assert isinstance(outputs, list)\n assert len(outputs) == len(prompts)\n\n client.close_communicator()\n\n @classmethod\n def teardown_class(cls):\n # vLLM x pytest (or Popen) seems not to handle process termination well. To avoid zombie processes, we need to\n # kill the server process and its children explicitly.\n kill_process(cls.server_process)\n\n\n@pytest.mark.slow\n@require_vllm\n@require_vision\nclass TestVLLMClientServerVLM(TrlTestCase):\n model_id = \"Qwen/Qwen2.5-VL-3B-Instruct\"\n\n @classmethod\n def setup_class(cls):\n # Start the server process\n cls.server_process = subprocess.Popen(\n [\"trl\", \"vllm-serve\", \"--model\", cls.model_id], stdout=subprocess.PIPE, stderr=subprocess.PIPE\n )\n\n # Initialize the client (no communicator needed for generation-only tests)\n cls.client = VLLMClient(connection_timeout=240, host=\"localhost\")\n\n def test_generate_with_token_ids_and_image(self):\n from PIL import Image\n\n processor = AutoProcessor.from_pretrained(self.model_id)\n image1 = Image.new(\"RGB\", (64, 64), color=\"red\")\n image2 = Image.new(\"RGB\", (64, 64), color=\"blue\")\n image3 = Image.new(\"RGB\", (64, 64), color=\"green\")\n messages = [\n [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": image1},\n {\"type\": \"image\", \"image\": image2},\n {\"type\": \"text\", \"text\": \"What are the differences between these two images?\"},\n ],\n }\n ],\n [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": image3},\n {\"type\": \"text\", \"text\": \"What is the color of this image?\"},\n ],\n }\n ],\n ]\n prompt_token_ids = processor.apply_chat_template(\n conversation=messages, tokenize=True, add_generation_prompt=True\n )\n outputs = self.client.generate(prompt_token_ids, images=[[image1, image2], [image3]], max_tokens=64)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n assert len(prompt_ids) == 2\n assert len(completion_ids) == 2\n assert all(isinstance(tok, int) for tok in prompt_ids[0])\n assert all(isinstance(tok, int) for tok in completion_ids[0])\n\n def test_generate_with_token_ids_mixed_images(self):\n \"\"\"Test a batch where one prompt has an image and the other does not.\"\"\"\n from PIL import Image\n\n processor = AutoProcessor.from_pretrained(self.model_id)\n image = Image.new(\"RGB\", (64, 64), color=\"red\")\n messages = [\n [\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"image\", \"image\": image}, {\"type\": \"text\", \"text\": \"Describe this image.\"}],\n }\n ],\n [\n {\n \"role\": \"user\",\n \"content\": [{\"type\": \"text\", \"text\": \"What is 1+1?\"}],\n }\n ],\n ]\n prompt_token_ids = processor.apply_chat_template(\n conversation=messages, tokenize=True, add_generation_prompt=True\n )\n outputs = self.client.generate(prompt_token_ids, images=[[image], None], max_tokens=64)\n prompt_ids = outputs[\"prompt_ids\"]\n completion_ids = outputs[\"completion_ids\"]\n\n assert len(prompt_ids) == 2\n assert len(completion_ids) == 2\n assert all(isinstance(tok, int) for tok in prompt_ids[0])\n assert all(isinstance(tok, int) for tok in prompt_ids[1])\n assert all(isinstance(tok, int) for tok in completion_ids[0])\n assert all(isinstance(tok, int) for tok in completion_ids[1])\n\n @classmethod\n def teardown_class(cls):\n kill_process(cls.server_process)\n" }, { "path": "tests/testing_constants.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nCI_HUB_USER = \"__DUMMY_TRANSFORMERS_USER__\"\nCI_HUB_USER_FULL_NAME = \"Dummy User\"\n\nCI_HUB_ENDPOINT = \"https://hub-ci.huggingface.co\"\n" }, { "path": "tests/testing_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport functools\nimport signal\nimport warnings\nfrom collections.abc import Callable\n\nimport psutil\nimport pytest\nimport torch\nfrom transformers import is_bitsandbytes_available, is_comet_available, is_sklearn_available, is_wandb_available\nfrom transformers.testing_utils import backend_device_count, torch_device\nfrom transformers.utils import (\n is_kernels_available,\n is_peft_available,\n is_rich_available,\n is_torch_available,\n is_vision_available,\n)\n\nfrom trl.import_utils import (\n is_jmespath_available,\n is_joblib_available,\n is_liger_kernel_available,\n is_llm_blender_available,\n is_math_verify_available,\n is_mergekit_available,\n is_vllm_available,\n)\n\n\nrequire_bitsandbytes = pytest.mark.skipif(not is_bitsandbytes_available(), reason=\"test requires bitsandbytes\")\nrequire_comet = pytest.mark.skipif(not is_comet_available(), reason=\"test requires comet_ml\")\nrequire_jmespath = pytest.mark.skipif(not is_jmespath_available(), reason=\"test requires jmespath\")\nrequire_kernels = pytest.mark.skipif(not is_kernels_available(), reason=\"test requires kernels\")\nrequire_liger_kernel = pytest.mark.skipif(not is_liger_kernel_available(), reason=\"test requires liger-kernel\")\nrequire_llm_blender = pytest.mark.skipif(not is_llm_blender_available(), reason=\"test requires llm-blender\")\nrequire_math_latex = pytest.mark.skipif(not is_math_verify_available(), reason=\"test requires math_verify\")\nrequire_mergekit = pytest.mark.skipif(not is_mergekit_available(), reason=\"test requires mergekit\")\nrequire_peft = pytest.mark.skipif(not is_peft_available(), reason=\"test requires peft\")\nrequire_rich = pytest.mark.skipif(not is_rich_available(), reason=\"test requires rich\")\nrequire_sklearn = pytest.mark.skipif(\n not (is_sklearn_available() and is_joblib_available()), reason=\"test requires sklearn\"\n)\nrequire_torch_accelerator = pytest.mark.skipif(\n torch_device is None or torch_device == \"cpu\", reason=\"test requires accelerator\"\n)\nrequire_torch_multi_accelerator = pytest.mark.skipif(\n not is_torch_available() or backend_device_count(torch_device) <= 1, reason=\"test requires multiple accelerators\"\n)\nrequire_vision = pytest.mark.skipif(not is_vision_available(), reason=\"test requires vision\")\nrequire_vllm = pytest.mark.skipif(not is_vllm_available(), reason=\"test requires vllm\")\nrequire_wandb = pytest.mark.skipif(not is_wandb_available(), reason=\"test requires wandb\")\nrequire_no_wandb = pytest.mark.skipif(is_wandb_available(), reason=\"test requires no wandb\")\nrequire_3_accelerators = pytest.mark.skipif(\n not (getattr(torch, torch_device, torch.cuda).device_count() >= 3),\n reason=f\"test requires at least 3 {torch_device}s\",\n)\n\n\ndef is_bitsandbytes_multi_backend_available() -> bool:\n if is_bitsandbytes_available():\n import bitsandbytes as bnb\n\n return \"multi_backend\" in getattr(bnb, \"features\", set())\n return False\n\n\n# Function ported from transformers.testing_utils before transformers#41283\nrequire_torch_gpu_if_bnb_not_multi_backend_enabled = pytest.mark.skipif(\n not is_bitsandbytes_multi_backend_available() and not torch_device == \"cuda\",\n reason=\"test requires bitsandbytes multi-backend enabled or 'cuda' torch device\",\n)\n\n\ndef is_ampere_or_newer(device_index=0):\n if not torch.cuda.is_available():\n return False\n\n major, minor = torch.cuda.get_device_capability(device_index)\n # Ampere starts at compute capability 8.0 (e.g., A100 = 8.0, RTX 30xx = 8.6)\n return (major, minor) >= (8, 0)\n\n\nrequire_ampere_or_newer = pytest.mark.skipif(not is_ampere_or_newer(), reason=\"test requires Ampere or newer GPU\")\n\n\nclass TrlTestCase:\n @pytest.fixture(autouse=True)\n def set_tmp_dir(self, tmp_path):\n self.tmp_dir = str(tmp_path)\n\n\ndef ignore_warnings(message: str = None, category: type[Warning] = Warning) -> Callable:\n \"\"\"\n Decorator to ignore warnings with a specific message and/or category.\n\n Args:\n message (`str`, *optional*):\n Regex pattern for the warning message to ignore. If `None`, all messages are ignored.\n category (`type[Warning]`, *optional*, defaults to `Warning`):\n Warning class to ignore. Defaults to `Warning`, which ignores all warnings.\n \"\"\"\n\n def decorator(test_func):\n @functools.wraps(test_func)\n def wrapper(*args, **kwargs):\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\", message=message, category=category)\n return test_func(*args, **kwargs)\n\n return wrapper\n\n return decorator\n\n\ndef kill_process(process):\n parent = psutil.Process(process.pid)\n children = parent.children(recursive=True)\n for child in children:\n try:\n child.send_signal(signal.SIGTERM)\n child.wait(timeout=5)\n except psutil.TimeoutExpired:\n child.kill()\n except psutil.NoSuchProcess:\n pass\n try:\n process.terminate()\n process.wait(timeout=5)\n except psutil.TimeoutExpired:\n process.kill()\n except psutil.NoSuchProcess:\n pass\n" }, { "path": "trl/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport sys\nfrom importlib.metadata import PackageNotFoundError, version\nfrom typing import TYPE_CHECKING\n\nfrom . import _compat\nfrom ._lazy_module import _LazyModule\n\n\ntry:\n __version__ = version(\"trl\")\nexcept PackageNotFoundError:\n __version__ = \"unknown\"\n\n_import_structure = {\n \"chat_template_utils\": [\"add_response_schema\", \"clone_chat_template\", \"get_training_chat_template\"],\n \"data_utils\": [\n \"apply_chat_template\",\n \"extract_prompt\",\n \"is_conversational\",\n \"is_conversational_from_value\",\n \"maybe_apply_chat_template\",\n \"maybe_convert_to_chatml\",\n \"maybe_extract_prompt\",\n \"maybe_unpair_preference_dataset\",\n \"pack_dataset\",\n \"prepare_multimodal_messages\",\n \"prepare_multimodal_messages_vllm\",\n \"truncate_dataset\",\n \"unpair_preference_dataset\",\n ],\n \"models\": [\"create_reference_model\"],\n \"scripts\": [\"DatasetMixtureConfig\", \"ScriptArguments\", \"TrlParser\", \"get_dataset\", \"init_zero_verbose\"],\n \"trainer\": [\n \"BEMACallback\",\n \"DPOConfig\",\n \"DPOTrainer\",\n \"GRPOConfig\",\n \"GRPOTrainer\",\n \"KTOConfig\",\n \"KTOTrainer\",\n \"LogCompletionsCallback\",\n \"ModelConfig\",\n \"RewardConfig\",\n \"RewardTrainer\",\n \"RichProgressCallback\",\n \"RLOOConfig\",\n \"RLOOTrainer\",\n \"SFTConfig\",\n \"SFTTrainer\",\n \"SyncRefModelCallback\",\n \"WeaveCallback\",\n \"get_kbit_device_map\",\n \"get_peft_config\",\n \"get_quantization_config\",\n ],\n}\n\nif TYPE_CHECKING:\n from .chat_template_utils import add_response_schema, clone_chat_template, get_training_chat_template\n from .data_utils import (\n apply_chat_template,\n extract_prompt,\n is_conversational,\n is_conversational_from_value,\n maybe_apply_chat_template,\n maybe_convert_to_chatml,\n maybe_extract_prompt,\n maybe_unpair_preference_dataset,\n pack_dataset,\n prepare_multimodal_messages,\n prepare_multimodal_messages_vllm,\n truncate_dataset,\n unpair_preference_dataset,\n )\n from .models import create_reference_model\n from .scripts import DatasetMixtureConfig, ScriptArguments, TrlParser, get_dataset, init_zero_verbose\n from .trainer import (\n BEMACallback,\n DPOConfig,\n DPOTrainer,\n GRPOConfig,\n GRPOTrainer,\n KTOConfig,\n KTOTrainer,\n LogCompletionsCallback,\n ModelConfig,\n RewardConfig,\n RewardTrainer,\n RichProgressCallback,\n RLOOConfig,\n RLOOTrainer,\n SFTConfig,\n SFTTrainer,\n SyncRefModelCallback,\n WeaveCallback,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n )\n\nelse:\n import sys\n\n sys.modules[__name__] = _LazyModule(\n __name__,\n globals()[\"__file__\"],\n _import_structure,\n module_spec=__spec__,\n extra_objects={\"__version__\": __version__},\n )\n" }, { "path": "trl/_compat.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\"\"\"\nCompatibility shims for third-party dependencies.\n\nThis module contains temporary patches to handle version incompatibilities between TRL's dependencies.\n\nEach patch should be removed when minimum version requirements eliminate the need.\n\"\"\"\n\nimport warnings\n\nfrom packaging.version import Version\n\nfrom .import_utils import _is_package_available\n\n\ndef _is_package_version_below(package_name: str, version_threshold: str) -> bool:\n \"\"\"\n Check if installed package version is below the given threshold.\n\n Args:\n package_name (str): Package name.\n version_threshold (str): Maximum version threshold.\n\n Returns:\n - True if package is installed and version < version_threshold.\n - False if package is not installed or version >= version_threshold.\n \"\"\"\n try:\n is_available, version = _is_package_available(package_name, return_version=True)\n return is_available and Version(version) < Version(version_threshold)\n except Exception as e:\n warnings.warn(\n f\"Failed to check {package_name} version against {version_threshold}: {e}. \"\n f\"Compatibility patch may not be applied.\",\n stacklevel=2,\n )\n return False\n\n\ndef _is_package_version_at_least(package_name: str, version_threshold: str) -> bool:\n \"\"\"\n Check if installed package version is at least the given threshold.\n\n Args:\n package_name (str): Package name.\n version_threshold (str): Minimum version threshold.\n\n Returns:\n - True if package is installed and version >= version_threshold.\n - False if package is not installed or version < version_threshold.\n \"\"\"\n try:\n is_available, version = _is_package_available(package_name, return_version=True)\n return is_available and Version(version) >= Version(version_threshold)\n except Exception as e:\n warnings.warn(\n f\"Failed to check {package_name} version against {version_threshold}: {e}. \"\n f\"Compatibility patch may not be applied.\",\n stacklevel=2,\n )\n return False\n\n\ndef _patch_vllm_logging() -> None:\n \"\"\"Set vLLM logging level to ERROR by default to reduce noise.\"\"\"\n if _is_package_available(\"vllm\"):\n import os\n\n os.environ[\"VLLM_LOGGING_LEVEL\"] = os.getenv(\"VLLM_LOGGING_LEVEL\", \"ERROR\")\n\n\ndef _patch_vllm_disabled_tqdm() -> None:\n \"\"\"\n Fix DisabledTqdm class in vLLM.\n\n - Bug introduced in https://github.com/vllm-project/vllm/pull/52\n - Fixed in https://github.com/vllm-project/vllm/pull/28471 (released in v0.11.1)\n - Since TRL currently supports vLLM v0.10.2-0.17.1, we patch it here\n - This can be removed when TRL requires vLLM>=0.11.1\n \"\"\"\n if _is_package_version_below(\"vllm\", \"0.11.1\"):\n try:\n import vllm.model_executor.model_loader.weight_utils\n from tqdm import tqdm\n\n class DisabledTqdm(tqdm):\n def __init__(self, *args, **kwargs):\n kwargs[\"disable\"] = True\n super().__init__(*args, **kwargs)\n\n vllm.model_executor.model_loader.weight_utils.DisabledTqdm = DisabledTqdm\n except (ImportError, AttributeError) as e:\n warnings.warn(f\"Failed to patch vLLM DisabledTqdm: {e}\", stacklevel=2)\n\n\ndef _patch_vllm_cached_tokenizer() -> None:\n \"\"\"\n Fix get_cached_tokenizer for transformers v5 compatibility.\n\n - Issue: vLLM's get_cached_tokenizer accesses all_special_tokens_extended\n - Removed in transformers: https://github.com/huggingface/transformers/pull/40936 (transformers>=5.0.0)\n - Fixed in https://github.com/vllm-project/vllm/pull/29686 (released in v0.12.0)\n - This can be removed when TRL requires vLLM>=0.12.0\n \"\"\"\n if _is_package_version_at_least(\"transformers\", \"5.0.0\") and _is_package_version_below(\"vllm\", \"0.12.0\"):\n try:\n import contextlib\n import copy\n\n import vllm.transformers_utils.tokenizer\n\n def get_cached_tokenizer(tokenizer):\n cached_tokenizer = copy.copy(tokenizer)\n tokenizer_all_special_ids = tokenizer.all_special_ids\n tokenizer_all_special_tokens = tokenizer.all_special_tokens\n tokenizer_vocab = tokenizer.get_vocab()\n tokenizer_len = len(tokenizer)\n\n max_token_id = max(tokenizer_vocab.values())\n if hasattr(tokenizer, \"vocab_size\"):\n with contextlib.suppress(NotImplementedError):\n max_token_id = max(max_token_id, tokenizer.vocab_size)\n\n class CachedTokenizer(tokenizer.__class__): # type: ignore\n @property\n def all_special_ids(self) -> list[int]:\n return tokenizer_all_special_ids\n\n @property\n def all_special_tokens(self) -> list[str]:\n return tokenizer_all_special_tokens\n\n @property\n def max_token_id(self) -> int:\n return max_token_id\n\n def get_vocab(self) -> dict[str, int]:\n return tokenizer_vocab\n\n def __len__(self) -> int:\n return tokenizer_len\n\n def __reduce__(self):\n return get_cached_tokenizer, (tokenizer,)\n\n CachedTokenizer.__name__ = f\"Cached{tokenizer.__class__.__name__}\"\n\n cached_tokenizer.__class__ = CachedTokenizer\n return cached_tokenizer\n\n vllm.transformers_utils.tokenizer.get_cached_tokenizer = get_cached_tokenizer\n except (ImportError, AttributeError) as e:\n warnings.warn(f\"Failed to patch vLLM cached_tokenizer: {e}\", stacklevel=2)\n\n\ndef _patch_transformers_hybrid_cache() -> None:\n \"\"\"\n Fix HybridCache import for transformers v5 compatibility.\n\n - Issue: peft import HybridCache from transformers.cache_utils\n - HybridCache removed in https://github.com/huggingface/transformers/pull/43168 (transformers>=5.0.0)\n - Fixed in peft: https://github.com/huggingface/peft/pull/2735 (released in v0.18.0)\n - This can be removed when TRL requires peft>=0.18.0\n \"\"\"\n if _is_package_version_at_least(\"transformers\", \"5.0.0\") and _is_package_version_below(\"peft\", \"0.18.0\"):\n try:\n import transformers.cache_utils\n from transformers.utils.import_utils import _LazyModule\n\n Cache = transformers.cache_utils.Cache\n\n # Patch for liger_kernel: Add HybridCache as an alias for Cache in the cache_utils module\n transformers.cache_utils.HybridCache = Cache\n\n # Patch for peft: Patch _LazyModule.__init__ to add HybridCache to transformers' lazy loading structures\n _original_lazy_module_init = _LazyModule.__init__\n\n def _patched_lazy_module_init(self, name, *args, **kwargs):\n _original_lazy_module_init(self, name, *args, **kwargs)\n if name == \"transformers\":\n # Update _LazyModule's internal structures\n if hasattr(self, \"_import_structure\") and \"cache_utils\" in self._import_structure:\n if \"HybridCache\" not in self._import_structure[\"cache_utils\"]:\n self._import_structure[\"cache_utils\"].append(\"HybridCache\")\n\n if hasattr(self, \"_class_to_module\"):\n self._class_to_module[\"HybridCache\"] = \"cache_utils\"\n\n if hasattr(self, \"__all__\") and \"HybridCache\" not in self.__all__:\n self.__all__.append(\"HybridCache\")\n\n self.HybridCache = Cache\n\n _LazyModule.__init__ = _patched_lazy_module_init\n\n except Exception as e:\n warnings.warn(f\"Failed to patch transformers HybridCache compatibility: {e}\", stacklevel=2)\n\n\ndef _patch_transformers_parallelism_config() -> None:\n \"\"\"\n Fix ParallelismConfig for transformers compatibility.\n\n Ensure that ``transformers.training_args`` always defines the symbol `ParallelismConfig` so that Python's\n `typing.get_type_hints` can resolve annotations on `transformers.TrainingArguments` without raising a `NameError`.\n\n This is needed when running with ``accelerate<1.10.1``, where the module ``accelerate.parallelism_config`` did not\n exist and therefore the type alias is not imported by Transformers.\n\n See upstream fix PR in transformers#40818.\n\n - Issue: transformers imports ParallelismConfig only if accelerate>=1.10.1 and raises NameError if\n accelerate<1.10.1\n - Fixed in transformers: https://github.com/huggingface/transformers/pull/40818 (released in v4.57.0)\n - This can be removed when TRL requires transformers>=4.57.0 or accelerate>=1.10.1\n \"\"\"\n if _is_package_version_below(\"transformers\", \"4.57.0\") and _is_package_version_below(\"accelerate\", \"1.10.1\"):\n try:\n from typing import Any\n\n import transformers.training_args\n\n if not hasattr(transformers.training_args, \"ParallelismConfig\"):\n transformers.training_args.ParallelismConfig = Any\n except Exception as e:\n warnings.warn(f\"Failed to patch transformers ParallelismConfig compatibility: {e}\", stacklevel=2)\n\n\n# Apply vLLM patches\n_patch_vllm_logging()\n_patch_vllm_disabled_tqdm()\n_patch_vllm_cached_tokenizer()\n\n# Apply transformers patches\n_patch_transformers_hybrid_cache()\n_patch_transformers_parallelism_config() # before creating HfArgumentParser\n" }, { "path": "trl/_lazy_module.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport importlib\nimport os\nfrom itertools import chain\nfrom types import ModuleType\nfrom typing import Any\n\n\nclass _LazyModule(ModuleType):\n \"\"\"\n Module class that surfaces all objects but only performs associated imports when the objects are requested.\n \"\"\"\n\n # Very heavily inspired by optuna.integration._IntegrationModule\n # https://github.com/optuna/optuna/blob/master/optuna/integration/__init__.py\n def __init__(self, name, module_file, import_structure, module_spec=None, extra_objects=None):\n super().__init__(name)\n self._modules = set(import_structure.keys())\n self._class_to_module = {}\n for key, values in import_structure.items():\n for value in values:\n self._class_to_module[value] = key\n # Needed for autocompletion in an IDE\n self.__all__ = list(import_structure.keys()) + list(chain(*import_structure.values()))\n self.__file__ = module_file\n self.__spec__ = module_spec\n self.__path__ = [os.path.dirname(module_file)]\n self._objects = {} if extra_objects is None else extra_objects\n self._name = name\n self._import_structure = import_structure\n\n # Needed for autocompletion in an IDE\n def __dir__(self):\n result = super().__dir__()\n # The elements of self.__all__ that are submodules may or may not be in the dir already, depending on whether\n # they have been accessed or not. So we only add the elements of self.__all__ that are not already in the dir.\n for attr in self.__all__:\n if attr not in result:\n result.append(attr)\n return result\n\n def __getattr__(self, name: str) -> Any:\n if name in self._objects:\n return self._objects[name]\n if name in self._modules:\n value = self._get_module(name)\n elif name in self._class_to_module.keys():\n module = self._get_module(self._class_to_module[name])\n value = getattr(module, name)\n else:\n raise AttributeError(f\"module {self.__name__} has no attribute {name}\")\n\n setattr(self, name, value)\n return value\n\n def _get_module(self, module_name: str):\n try:\n return importlib.import_module(\".\" + module_name, self.__name__)\n except Exception as e:\n raise RuntimeError(\n f\"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its\"\n f\" traceback):\\n{e}\"\n ) from e\n\n def __reduce__(self):\n return (self.__class__, (self._name, self.__file__, self._import_structure))\n" }, { "path": "trl/accelerate_configs/fsdp1.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: FSDP\ndowncast_bf16: 'no'\nenable_cpu_affinity: false\nfsdp_config:\n fsdp_activation_checkpointing: false\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_backward_prefetch: BACKWARD_PRE\n fsdp_cpu_ram_efficient_loading: true\n fsdp_forward_prefetch: true\n fsdp_offload_params: false\n fsdp_reshard_after_forward: FULL_SHARD\n fsdp_state_dict_type: FULL_STATE_DICT\n fsdp_sync_module_states: true\n fsdp_use_orig_params: true\n fsdp_version: 1\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "trl/accelerate_configs/fsdp2.yaml", "content": "# Requires accelerate 1.7.0 or higher\ncompute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: FSDP\ndowncast_bf16: 'no'\nenable_cpu_affinity: false\nfsdp_config:\n fsdp_activation_checkpointing: false\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_cpu_ram_efficient_loading: true\n fsdp_offload_params: false\n fsdp_reshard_after_forward: true\n fsdp_state_dict_type: FULL_STATE_DICT\n fsdp_version: 2\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "trl/accelerate_configs/multi_gpu.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: MULTI_GPU\ndowncast_bf16: 'no'\ngpu_ids: all\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: 'bf16'\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "trl/accelerate_configs/single_gpu.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: \"NO\"\ndowncast_bf16: 'no'\ngpu_ids: all\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: 'bf16'\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "trl/accelerate_configs/zero1.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndeepspeed_config:\n deepspeed_multinode_launcher: standard\n gradient_accumulation_steps: 1\n zero3_init_flag: false\n zero_stage: 1\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: 'bf16'\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "trl/accelerate_configs/zero2.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndeepspeed_config:\n deepspeed_multinode_launcher: standard\n offload_optimizer_device: none\n offload_param_device: none\n zero3_init_flag: false\n zero_stage: 2\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: 'bf16'\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "trl/accelerate_configs/zero3.yaml", "content": "compute_environment: LOCAL_MACHINE\ndebug: false\ndeepspeed_config:\n deepspeed_multinode_launcher: standard\n offload_optimizer_device: none\n offload_param_device: none\n zero3_init_flag: true\n zero3_save_16bit_model: true\n zero_stage: 3\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n" }, { "path": "trl/chat_template_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom transformers import AddedToken, AutoTokenizer, PreTrainedModel, PreTrainedTokenizer\n\n\ndef clone_chat_template(\n model: PreTrainedModel,\n tokenizer: PreTrainedTokenizer,\n source_tokenizer_path: str,\n resize_to_multiple_of: int | None = 64,\n) -> tuple[PreTrainedModel, PreTrainedTokenizer, list[int]]:\n \"\"\"\n Clones a chat template from a source tokenizer to the target tokenizer and updates the model accordingly.\n\n This function:\n - Copies the chat template from a source tokenizer to the target tokenizer.\n - Adds any new tokens from the source tokenizer to the target tokenizer.\n - Sets and synchronizes the EOS token across the tokenizer and model.\n - Resizes the model's token embeddings to match the new vocabulary size, optionally rounding it up to a multiple of\n a specified value. In such cases, dummy tokens are added to the tokenizer to ensure the vocabulary size matches\n the embedding dimensions.\n\n Args:\n model ([`~transformers.PreTrainedModel`]):\n Model to update.\n tokenizer ([`~transformers.PreTrainedTokenizer`]):\n Tokenizer to update.\n source_tokenizer_path (`str`):\n Path or identifier of the pretrained tokenizer to clone from.\n resize_to_multiple_of (`int` or `None`, *optional*, defaults to `64`):\n The embedding layer will be resized to the new vocabulary size. If this is not `None`, it will round up the\n new vocabulary size to the nearest multiple of this value.\n\n Returns:\n model ([`~transformers.PreTrainedModel`]):\n Updated model with resized token embeddings and EOS token configured.\n tokenizer ([`~transformers.PreTrainedTokenizer`]):\n Updated tokenizer with the chat template and special tokens applied.\n added_tokens (`list[int]`):\n List of tokens that were added to the tokenizer from the source tokenizer.\n\n Example:\n ```python\n from transformers import AutoModelForCausalLM, AutoTokenizer\n from trl import clone_chat_template\n\n model = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-3.2-1B\")\n tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Llama-3.2-1B\")\n model, tokenizer, added_tokens = clone_chat_template(model, tokenizer, \"Qwen/Qwen3-0.6B\")\n ```\n \"\"\"\n # Load the source tokenizer containing the desired chat template\n tokenizer_source = AutoTokenizer.from_pretrained(source_tokenizer_path)\n\n # Copy the chat template from the source tokenizer\n tokenizer.chat_template = tokenizer_source.get_chat_template()\n\n # Ensure all added tokens from the source are available in the target tokenizer\n added_tokens = [\n token for token in tokenizer_source.added_tokens_decoder.values() if token.content not in tokenizer.vocab\n ]\n tokenizer.add_tokens(added_tokens)\n\n # Set the EOS token from the source tokenizer (important for generation)\n tokenizer.eos_token = tokenizer_source.eos_token\n model.config.eos_token_id = tokenizer.eos_token_id\n if model.can_generate(): # Non-generative models (e.g. SequenceClassification) may not have a generation_config\n model.generation_config.eos_token_id = tokenizer.eos_token_id\n\n # Resize model embeddings to include any new tokens, optionally rounding up to a multiple\n model.resize_token_embeddings(\n # After studying many tokenizers, we found that len(tokenizer.vocab) is the most reliable way to get the vocab\n # size. Avoid using tokenizer.vocab_size or tokenizer.vocab_size + len(tokenizer.added_tokens_encoder),\n # as handling of special and added tokens varies across tokenizers.\n new_num_tokens=len(tokenizer.vocab),\n pad_to_multiple_of=resize_to_multiple_of if resize_to_multiple_of is not None else None,\n )\n\n # After resizing, the embedding matrix size may exceed the vocabulary size. Add dummy tokens to the tokenizer to\n # ensure vocabulary size matches the embedding matrix dimensions.\n idx = 0\n while model.vocab_size > len(tokenizer.vocab):\n dummy_token = AddedToken(f\"\")\n is_added = tokenizer.add_tokens(dummy_token)\n idx += 1\n if is_added == 1:\n added_tokens.append(dummy_token)\n\n # Verify that vocabulary size now matches embedding dimensions\n if len(tokenizer.vocab) != model.vocab_size:\n raise RuntimeError(\n f\"Vocabulary size mismatch after resizing: tokenizer vocab size is {len(tokenizer.vocab)}, but model \"\n f\"embedding size is {model.vocab_size}. This indicates an internal error in the token alignment process.\"\n )\n added_tokens = [token.content for token in added_tokens]\n added_tokens = tokenizer.convert_tokens_to_ids(added_tokens)\n return model, tokenizer, added_tokens\n\n\n# Adapted and corrected versions of the schemas from:\n# https://github.com/huggingface/transformers/blob/main/tests/utils/test_chat_parsing_utils.py\nqwen3_schema = {\n \"x-regex\": r\"^(?:\\n?(?:(?P.*?\\S.*?)\\n?|[\\s]*)\\s*)?(?P.*?)(?:\\n(?=))?(?=(?:|<\\|im_end\\|>|$))(?P(?:.+?\\s*)+)?\\s*(?:<\\|im_end\\|>|$)\",\n \"type\": \"object\",\n \"properties\": {\n \"role\": {\"const\": \"assistant\"},\n \"content\": {\"type\": \"string\"},\n \"reasoning_content\": {\"type\": \"string\"},\n \"tool_calls\": {\n \"type\": \"array\",\n \"x-regex-iterator\": r\"\\s*(.+?)\\s*\",\n \"items\": {\n \"x-parser\": \"json\",\n \"x-parser-args\": {\"transform\": \"{type: 'function', function: @}\"},\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\"const\": \"function\"},\n \"function\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"arguments\": {\n \"type\": \"object\",\n \"additionalProperties\": {},\n },\n },\n },\n },\n },\n },\n },\n}\n\nqwen35_schema = {\n \"x-regex\": r\"^(?:(?:\\n?)?(?:(?P.*?\\S.*?)\\n?|[\\s]*)\\s*)?(?P.*?)(?:\\n+(?=))?(?=(?:|<\\|im_end\\|>|$))(?P(?:.+?\\s*)+)?\\s*(?:<\\|im_end\\|>|$)\",\n \"type\": \"object\",\n \"properties\": {\n \"role\": {\"const\": \"assistant\"},\n \"content\": {\"type\": \"string\"},\n \"reasoning_content\": {\"type\": \"string\"},\n \"tool_calls\": {\n \"type\": \"array\",\n \"x-regex-iterator\": r\"\\s*(.+?)\\s*\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\"const\": \"function\"},\n \"function\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\", \"x-regex\": r\"]+)>\"},\n \"arguments\": {\n \"type\": \"object\",\n \"x-regex-key-value\": r\"[^>\\n]+)>\\n(?P.*?)\\n\",\n \"default\": {},\n \"additionalProperties\": {\n \"x-parser\": \"json\",\n \"x-parser-args\": {\"allow_non_json\": True},\n },\n },\n },\n },\n },\n },\n },\n },\n}\n\n# docstyle-ignore\nqwen3_chat_template = r\"\"\"{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0].role == 'system' %}\n {{- messages[0].content + '\\n\\n' }}\n {%- endif %}\n {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within XML tags:\\n\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n\\n\\nFor each function call, return a json object with function name and arguments within XML tags:\\n\\n{\\\"name\\\": , \\\"arguments\\\": }\\n<|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0].role == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == \"user\" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n {%- if message.content is string %}\n {%- set content = message.content %}\n {%- else %}\n {%- set content = '' %}\n {%- endif %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {%- set reasoning_content = '' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if '' in content %}\n {%- set reasoning_content = content.split('')[0].rstrip('\\n').split('')[-1].lstrip('\\n') %}\n {%- set content = content.split('')[-1].lstrip('\\n') %}\n {%- endif %}\n {%- endif %}\n {%- if loop.index0 > ns.last_query_index %}\n {%- if loop.last or (not loop.last and reasoning_content) %}\n {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- if message.tool_calls %}\n {%- for tool_call in message.tool_calls %}\n {%- if (loop.first and content) or (not loop.first) %}\n {{- '\\n' }}\n {%- endif %}\n {%- if tool_call.function %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {%- if tool_call.arguments is string %}\n {{- tool_call.arguments }}\n {%- else %}\n {{- tool_call.arguments | tojson }}\n {%- endif %}\n {{- '}\\n' }}\n {%- endfor %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n\\n' }}\n {{- content }}\n {{- '\\n' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- if enable_thinking is defined and enable_thinking is false %}\n {{- '\\n\\n\\n\\n' }}\n {%- endif %}\n{%- endif %}\"\"\"\n\n# docstyle-ignore\nqwen35_chat_template = r\"\"\"{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- macro render_content(content, do_vision_count, is_system_content=false) %}\n {%- if content is string %}\n {{- content }}\n {%- elif content is iterable and content is not mapping %}\n {%- for item in content %}\n {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}\n {%- if is_system_content %}\n {{- raise_exception('System message cannot contain images.') }}\n {%- endif %}\n {%- if do_vision_count %}\n {%- set image_count.value = image_count.value + 1 %}\n {%- endif %}\n {%- if add_vision_id %}\n {{- 'Picture ' ~ image_count.value ~ ': ' }}\n {%- endif %}\n {{- '<|vision_start|><|image_pad|><|vision_end|>' }}\n {%- elif 'video' in item or item.type == 'video' %}\n {%- if is_system_content %}\n {{- raise_exception('System message cannot contain videos.') }}\n {%- endif %}\n {%- if do_vision_count %}\n {%- set video_count.value = video_count.value + 1 %}\n {%- endif %}\n {%- if add_vision_id %}\n {{- 'Video ' ~ video_count.value ~ ': ' }}\n {%- endif %}\n {{- '<|vision_start|><|video_pad|><|vision_end|>' }}\n {%- elif 'text' in item %}\n {{- item.text }}\n {%- else %}\n {{- raise_exception('Unexpected item type in content.') }}\n {%- endif %}\n {%- endfor %}\n {%- elif content is none or content is undefined %}\n {{- '' }}\n {%- else %}\n {{- raise_exception('Unexpected content type.') }}\n {%- endif %}\n{%- endmacro %}\n{%- if not messages %}\n {{- raise_exception('No messages provided.') }}\n{%- endif %}\n{%- if tools and tools is iterable and tools is not mapping %}\n {{- '<|im_start|>system\\n' }}\n {{- \"# Tools\\n\\nYou have access to the following functions:\\n\\n\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n\" }}\n {{- '\\n\\nIf you choose to call a function ONLY reply in the following format with NO suffix:\\n\\n\\n\\n\\nvalue_1\\n\\n\\nThis is the value for the second parameter\\nthat can span\\nmultiple lines\\n\\n\\n\\n\\n\\nReminder:\\n- Function calls MUST follow the specified format: an inner block must be nested within XML tags\\n- Required parameters MUST be specified\\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\\n' }}\n {%- if messages[0].role == 'system' %}\n {%- set content = render_content(messages[0].content, false, true)|trim %}\n {%- if content %}\n {{- '\\n\\n' + content }}\n {%- endif %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n{%- else %}\n {%- if messages[0].role == 'system' %}\n {%- set content = render_content(messages[0].content, false, true)|trim %}\n {{- '<|im_start|>system\\n' + content + '<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == \"user\" %}\n {%- set content = render_content(message.content, false)|trim %}\n {%- if not(content.startswith('') and content.endswith('')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if ns.multi_step_tool %}\n {{- raise_exception('No user query found in messages.') }}\n{%- endif %}\n{%- for message in messages %}\n {%- set content = render_content(message.content, true)|trim %}\n {%- if message.role == \"system\" %}\n {%- if not loop.first %}\n {{- raise_exception('System message must be at the beginning.') }}\n {%- endif %}\n {%- elif message.role == \"user\" %}\n {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {%- set reasoning_content = '' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if '' in content %}\n {%- set reasoning_content = content.split('')[0].rstrip('\\n').split('')[-1].lstrip('\\n') %}\n {%- set content = content.split('')[-1].lstrip('\\n') %}\n {%- endif %}\n {%- endif %}\n {%- set reasoning_content = reasoning_content|trim %}\n {%- if loop.index0 > ns.last_query_index %}\n {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content + '\\n\\n\\n' + content }}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {%- if loop.first %}\n {%- if content|trim %}\n {{- '\\n\\n\\n\\n' }}\n {%- else %}\n {{- '\\n\\n' }}\n {%- endif %}\n {%- else %}\n {{- '\\n\\n\\n' }}\n {%- endif %}\n {%- if tool_call.arguments is defined %}\n {%- for args_name, args_value in tool_call.arguments|items %}\n {{- '\\n' }}\n {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}\n {{- args_value }}\n {{- '\\n\\n' }}\n {%- endfor %}\n {%- endif %}\n {{- '\\n' }}\n {%- endfor %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if loop.previtem and loop.previtem.role != \"tool\" %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n\\n' }}\n {{- content }}\n {{- '\\n' }}\n {%- if not loop.last and loop.nextitem.role != \"tool\" %}\n {{- '<|im_end|>\\n' }}\n {%- elif loop.last %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- else %}\n {{- raise_exception('Unexpected message role.') }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- if enable_thinking is defined and enable_thinking is true %}\n {{- '\\n' }}\n {%- else %}\n {{- '\\n\\n\\n\\n' }}\n {%- endif %}\n{%- endif %}\"\"\"\n\n\ndef add_response_schema(tokenizer: PreTrainedTokenizer) -> PreTrainedTokenizer:\n r\"\"\"\n Adds the appropriate response schema to the given tokenizer based on its chat template.\n\n At the time of initial implementation, most tokenizers do not have built-in support for response schemas. While\n waiting for broader adoption, we provide this utility function to manually set the response schema for known chat\n templates.\n\n Args:\n tokenizer (`PreTrainedTokenizer`):\n Tokenizer to which the response schema will be added.\n\n Returns:\n `PreTrainedTokenizer`:\n Tokenizer with the added response schema.\n\n Examples:\n\n ```python\n >>> from trl.chat_template_utils import add_response_schema\n >>> from transformers import AutoTokenizer\n\n >>> tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen3-0.6B\")\n >>> tokenizer = add_response_schema(tokenizer)\n >>> assistant_text = '\\n{\"name\": \"multiply\", \"arguments\": {\"a\": 3, \"b\": 4}}\\n<|im_end|>'\n >>> tokenizer.parse_response(assistant_text)\n {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'name': 'multiply', 'arguments': {'a': 3, 'b': 4}}}]}\n ```\n \"\"\"\n if tokenizer.chat_template == qwen3_chat_template:\n tokenizer.response_schema = qwen3_schema\n return tokenizer\n if tokenizer.chat_template == qwen35_chat_template:\n tokenizer.response_schema = qwen35_schema\n return tokenizer\n raise ValueError(\n \"Unrecognized chat template, failed to add response schema. Please manually set the response schema on the \"\n \"tokenizer or processor. See the Transformers \"\n \"[docs](https://huggingface.co/docs/transformers/main/en/chat_response_parsing#response-parsing) for more \"\n \"details on response parsing.\"\n )\n\n\ndef is_chat_template_prefix_preserving(tokenizer: PreTrainedTokenizer) -> bool:\n \"\"\"\n Check whether the chat template preserves prefixes when applied.\n\n Args:\n tokenizer (`PreTrainedTokenizer`):\n Tokenizer instance to check.\n\n Returns:\n `bool`:\n `True` if the chat template preserves prefixes, `False` otherwise.\n \"\"\"\n messages1 = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ]\n messages2 = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ]\n messages3 = [\n {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n {\"role\": \"user\", \"content\": \"And at night?\"},\n ]\n\n text1 = tokenizer.apply_chat_template(messages1, tokenize=False, add_generation_prompt=True)\n text2 = tokenizer.apply_chat_template(messages2, tokenize=False)\n text3 = tokenizer.apply_chat_template(messages3, tokenize=False)\n\n return text2.startswith(text1) and text3.startswith(text2)\n\n\n# Modifications:\n# - {%- if '' in content %}\n# + {%- if '' in content and '' in content %}\n# Always check for both tags to avoid edge cases where the model generates only one tag, which would otherwise be parsed incorrectly\n# - {%- if loop.index0 > ns.last_query_index %} ... {%- endif %}\n# + {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }}\n# Always include thinking block during training. It's important to have a prefix-preserving template.\n# docstyle-ignore\nqwen3_training_chat_template = r\"\"\"{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0].role == 'system' %}\n {{- messages[0].content + '\\n\\n' }}\n {%- endif %}\n {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within XML tags:\\n\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n\\n\\nFor each function call, return a json object with function name and arguments within XML tags:\\n\\n{\\\"name\\\": , \\\"arguments\\\": }\\n<|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0].role == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == \"user\" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n {%- if message.content is string %}\n {%- set content = message.content %}\n {%- else %}\n {%- set content = '' %}\n {%- endif %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {%- set reasoning_content = '' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if '' in content and '' in content %}\n {%- set reasoning_content = content.split('')[0].rstrip('\\n').split('')[-1].lstrip('\\n') %}\n {%- set content = content.split('')[-1].lstrip('\\n') %}\n {%- endif %}\n {%- endif %}\n {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }}\n {%- if message.tool_calls %}\n {%- for tool_call in message.tool_calls %}\n {%- if (loop.first and content) or (not loop.first) %}\n {{- '\\n' }}\n {%- endif %}\n {%- if tool_call.function %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {%- if tool_call.arguments is string %}\n {{- tool_call.arguments }}\n {%- else %}\n {{- tool_call.arguments | tojson }}\n {%- endif %}\n {{- '}\\n' }}\n {%- endfor %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n\\n' }}\n {{- content }}\n {{- '\\n' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- if enable_thinking is defined and enable_thinking is false %}\n {{- '\\n\\n\\n\\n' }}\n {%- endif %}\n{%- endif %}\"\"\"\n\n# Modifications:\n# - {%- if '' in content %}\n# + {%- if '' in content and '' in content %}\n# Always check for both tags to avoid edge cases where the model generates only one tag, which would otherwise be parsed incorrectly\n# - {{- '<|im_start|>' + message.role + '\\n' + content }}\n# + {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content + '\\n\\n\\n' + content }}\n# Always include thinking block during training. It's important to have a prefix-preserving template.\nqwen35_training_chat_template = qwen35_chat_template.replace(\n \"{%- if '' in content %}\",\n \"{%- if '' in content and '' in content %}\",\n).replace(\n \"{{- '<|im_start|>' + message.role + '\\\\n' + content }}\",\n \"{{- '<|im_start|>' + message.role + '\\\\n\\\\n' + reasoning_content + '\\\\n\\\\n\\\\n' + content }}\",\n)\n\n\ndef get_training_chat_template(tokenizer: PreTrainedTokenizer) -> str | None:\n r\"\"\"\n Get a prefix-preserving chat template for training, if needed.\n\n If the tokenizer's template isn't prefix-preserving, returns a training-compatible template (currently Qwen3 and\n Qwen3.5 supported). Otherwise, returns `None`.\n\n Args:\n tokenizer (`PreTrainedTokenizer`):\n Tokenizer instance to check.\n\n Returns:\n `str` or `None`:\n Training-compatible chat template, or `None` if no patching is needed.\n\n Example:\n\n ```python\n >>> from trl.chat_template_utils import get_training_chat_template\n >>> from transformers import AutoTokenizer\n\n >>> tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen3-0.6B\")\n >>> messages1 = [\n ... {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ... {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ... ]\n >>> messages2 = [\n ... {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ... {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ... {\"role\": \"user\", \"content\": \"And at night?\"},\n ... ]\n >>> tokenizer.apply_chat_template(messages1, tokenize=False)\n '<|im_start|>user\\nWhat color is the sky?<|im_end|>\\n<|im_start|>assistant\\n\\n\\n\\n\\nIt is blue.<|im_end|>\\n'\n\n >>> tokenizer.apply_chat_template(messages2, tokenize=False)\n '<|im_start|>user\\nWhat color is the sky?<|im_end|>\\n<|im_start|>assistant\\nIt is blue.<|im_end|>\\n<|im_start|>user\\nAnd at night?<|im_end|>\\n'\n\n >>> # ^ think tags missing\n >>> chat_template = get_training_chat_template(tokenizer)\n >>> tokenizer.apply_chat_template(messages1, tokenize=False, chat_template=chat_template)\n '<|im_start|>user\\nWhat color is the sky?<|im_end|>\\n<|im_start|>assistant\\n\\n\\n\\n\\nIt is blue.<|im_end|>\\n'\n\n >>> tokenizer.apply_chat_template(messages2, tokenize=False, chat_template=chat_template)\n '<|im_start|>user\\nWhat color is the sky?<|im_end|>\\n<|im_start|>assistant\\n\\n\\n\\n\\nIt is blue.<|im_end|>\\n<|im_start|>user\\nAnd at night?<|im_end|>\\n'\n ```\n \"\"\"\n # First check if patching is needed\n if is_chat_template_prefix_preserving(tokenizer):\n return None # No patching needed\n\n if tokenizer.chat_template == qwen3_chat_template:\n return qwen3_training_chat_template\n if tokenizer.chat_template == qwen35_chat_template:\n return qwen35_training_chat_template\n else:\n raise ValueError(\n \"The tokenizer's chat template is not prefix-preserving and patching is not supported for this template. \"\n \"Please manually modify the tokenizer's chat template for training.\"\n )\n\n\ndef _validate_tool_calls(tool_calls: list | None) -> None:\n \"\"\"\n Validate tool_calls to ensure all required fields exist with valid values.\n\n Raises ValueError when the model generates malformed tool calls (e.g., missing 'arguments' field) that are\n partially parsed.\n\n Args:\n tool_calls: List of tool call dictionaries, or None.\n \"\"\"\n if tool_calls is None:\n return None\n if not isinstance(tool_calls, list):\n raise ValueError(\"tool_calls must be a list or None.\")\n\n for idx, tool_call in enumerate(tool_calls):\n if not isinstance(tool_call, dict):\n raise ValueError(f\"tool_calls[{idx}] must be a dict.\")\n\n # Handle nested function structure: {\"type\": \"function\", \"function\": {\"name\": ..., \"arguments\": ...}}\n if \"function\" in tool_call:\n func = tool_call[\"function\"]\n if not isinstance(func, dict):\n raise ValueError(f\"tool_calls[{idx}]['function'] must be a dict.\")\n if not isinstance(func.get(\"name\"), str):\n raise ValueError(f\"tool_calls[{idx}]['function']['name'] must be a string.\")\n # Some templates (e.g. Qwen3.5) omit arguments for valid no-arg calls; normalize to {}.\n if \"arguments\" not in func or func[\"arguments\"] is None:\n func[\"arguments\"] = {}\n else:\n # Handle flat structure: {\"name\": ..., \"arguments\": ...}\n if not isinstance(tool_call.get(\"name\"), str):\n raise ValueError(f\"tool_calls[{idx}]['name'] must be a string.\")\n # Some templates (e.g. Qwen3.5) omit arguments for valid no-arg calls; normalize to {}.\n if \"arguments\" not in tool_call or tool_call[\"arguments\"] is None:\n tool_call[\"arguments\"] = {}\n\n\ndef parse_response(tokenizer: PreTrainedTokenizer, ids: list[int]) -> dict:\n r\"\"\"\n Parse a token sequence into structured response dictionaries with fallback handling.\n\n Attempts to parse the sequence using `tokenizer.parse_response()`. If parsing fails (e.g., due to malformed tool\n calls like `{\"type\":\"function\"`), falls back to decoding as plain text.\n\n Also removes incorrectly appended EOS tokens from tool call content when present, and validates tool_calls to\n ensure all required fields exist.\n\n Args:\n tokenizer (`PreTrainedTokenizer`):\n Tokenizer with a `parse_response()` method.\n ids (`list[int]`):\n List of token sequences.\n\n Returns:\n `dict`:\n Response dictionary.\n\n Example:\n ```python\n >>> from trl.chat_template_utils import parse_response, add_response_schema\n >>> from transformers import AutoTokenizer\n\n >>> tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen3-0.6B\")\n >>> tokenizer = add_response_schema(tokenizer) # temporary until built-in support\n >>> text = '\\n{\"name\": \"multiply\", \"arguments\": {\"a\": 3, \"b\": 4}}\\n<|im_end|>'\n >>> ids = tokenizer(text)[\"input_ids\"]\n >>> parse_response(tokenizer, ids)\n {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'name': 'multiply', 'arguments': {'a': 3, 'b': 4}}}]}\n ```\n \"\"\"\n try:\n parsed = tokenizer.parse_response(ids)\n # Hotfix: remove incorrectly appended EOS token from tool calls\n # See https://github.com/huggingface/transformers/issues/42249\n parsed[\"content\"] = parsed[\"content\"].removesuffix(tokenizer.eos_token)\n # Validate tool_calls to prevent Jinja2 Undefined errors when fields are missing\n if \"tool_calls\" in parsed:\n _validate_tool_calls(parsed[\"tool_calls\"])\n except (ValueError, TypeError):\n # Fallback: decode as plain text if parsing fails. This happens if the model outputs malformed tool calls.\n content = tokenizer.decode(ids, skip_special_tokens=True)\n parsed = {\"role\": \"assistant\", \"content\": content}\n return parsed\n" }, { "path": "trl/cli/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .main import main\n\n\n__all__ = [\"main\"]\n" }, { "path": "trl/cli/accelerate_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport importlib.resources as resources\nfrom pathlib import Path\n\n\ndef resolve_accelerate_config_argument(launch_args: list[str]) -> list[str]:\n \"\"\"\n Resolve `--accelerate_config` from CLI arguments into `accelerate --config_file`.\n\n The function supports either a filesystem path or a predefined config name shipped in `trl/accelerate_configs`\n (without the `.yaml` suffix).\n \"\"\"\n if \"--accelerate_config\" not in launch_args:\n return launch_args\n\n config_index = launch_args.index(\"--accelerate_config\")\n if config_index + 1 >= len(launch_args):\n raise ValueError(\"Expected a value after `--accelerate_config`.\")\n\n config_name = launch_args[config_index + 1]\n if Path(config_name).is_file():\n accelerate_config_path = config_name\n else:\n candidate = resources.files(\"trl.accelerate_configs\").joinpath(f\"{config_name}.yaml\")\n if not candidate.exists():\n raise ValueError(\n f\"Accelerate config {config_name} is neither a file nor a valid config in the `trl` package. \"\n \"Please provide a valid config name or a path to a config file.\"\n )\n accelerate_config_path = candidate\n\n # Remove '--accelerate_config '.\n launch_args = launch_args[:config_index] + launch_args[config_index + 2 :]\n return [\"--config_file\", str(accelerate_config_path)] + launch_args\n" }, { "path": "trl/cli/accelerate_launcher.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport importlib.resources as resources\nfrom collections.abc import Callable\nfrom typing import Any\n\nfrom accelerate.commands.launch import launch_command, launch_command_parser\n\n\ndef launch_training_script(\n script_name: str,\n launch_args: list[str],\n training_script_args: list[str],\n *,\n launch_command_fn: Callable[[Any], None] = launch_command,\n launch_parser_fn: Callable[[], Any] = launch_command_parser,\n) -> None:\n \"\"\"\n Launch a TRL training script through `accelerate launch`.\n\n Parameters:\n script_name (`str`):\n Script filename in `trl/scripts`, e.g. `\"dpo.py\"`.\n launch_args (`list[str]`):\n Arguments consumed by `accelerate launch`.\n training_script_args (`list[str]`):\n Arguments forwarded to the training script.\n launch_command_fn (`Callable[[Any], None]`, *optional*):\n Function used to execute accelerate launch.\n launch_parser_fn (`Callable[[], Any]`, *optional*):\n Factory creating the accelerate launch parser.\n \"\"\"\n training_script = resources.files(\"trl.scripts\").joinpath(script_name)\n accelerate_args = launch_parser_fn().parse_args(launch_args + [str(training_script)] + training_script_args)\n launch_command_fn(accelerate_args)\n" }, { "path": "trl/cli/commands/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .base import Command\nfrom .env import EnvCommand\nfrom .skills import SkillsCommand\nfrom .training import TrainingCommand\nfrom .vllm_serve import VllmServeCommand\n\n\ndef get_commands() -> list[Command]:\n \"\"\"Return all registered top-level TRL CLI commands.\"\"\"\n return [\n TrainingCommand(\"dpo\"),\n EnvCommand(),\n TrainingCommand(\"grpo\"),\n TrainingCommand(\"kto\"),\n TrainingCommand(\"reward\"),\n TrainingCommand(\"rloo\"),\n TrainingCommand(\"sft\"),\n SkillsCommand(),\n VllmServeCommand(),\n ]\n\n\n__all__ = [\"Command\", \"get_commands\"]\n" }, { "path": "trl/cli/commands/base.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom abc import ABC, abstractmethod\nfrom argparse import Namespace\nfrom dataclasses import dataclass\n\n\n@dataclass(slots=True)\nclass CommandContext:\n \"\"\"Context shared by CLI commands during execution.\"\"\"\n\n argv: list[str]\n\n def argv_after(self, token: str) -> list[str]:\n \"\"\"\n Return CLI tokens after the first occurrence of `token`.\n\n Parameters:\n token (`str`):\n Subcommand name as it appears in `argv`.\n \"\"\"\n try:\n index = self.argv.index(token)\n except ValueError:\n return []\n return self.argv[index + 1 :]\n\n\nclass Command(ABC):\n \"\"\"\n Base command definition for the TRL CLI.\n\n Parameters:\n name (`str`):\n Subcommand name exposed by the CLI.\n help_text (`str`):\n Short description displayed in help output.\n \"\"\"\n\n def __init__(self, name: str, help_text: str):\n self.name = name\n self.help_text = help_text\n\n @abstractmethod\n def register(self, subparsers) -> None:\n \"\"\"Register this command parser in the subparser collection.\"\"\"\n\n @abstractmethod\n def run(self, args: Namespace, context: CommandContext) -> int:\n \"\"\"Execute the command.\"\"\"\n" }, { "path": "trl/cli/commands/env.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom argparse import Namespace\n\nfrom .base import Command, CommandContext\n\n\nclass EnvCommand(Command):\n \"\"\"CLI command that prints TRL environment information.\"\"\"\n\n def __init__(self):\n super().__init__(name=\"env\", help_text=\"Print the environment information\")\n\n def register(self, subparsers) -> None:\n subparsers.add_parser(self.name, help=self.help_text)\n\n def run(self, args: Namespace, context: CommandContext) -> int:\n from ...scripts.env import print_env\n\n print_env()\n return 0\n" }, { "path": "trl/cli/commands/skills.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom argparse import Namespace\n\nfrom ...skills.cli import add_skills_subcommands\nfrom .base import Command, CommandContext\n\n\nclass SkillsCommand(Command):\n \"\"\"CLI command that manages TRL agent skills.\"\"\"\n\n def __init__(self):\n super().__init__(name=\"skills\", help_text=\"Manage TRL agent skills\")\n self._skills_parser = None\n\n def register(self, subparsers) -> None:\n self._skills_parser = subparsers.add_parser(self.name, help=self.help_text)\n skills_subparsers = self._skills_parser.add_subparsers(dest=\"skills_command\", help=\"Skills commands\")\n add_skills_subcommands(skills_subparsers)\n\n def run(self, args: Namespace, context: CommandContext) -> int:\n if getattr(args, \"skills_command\", None):\n if hasattr(args, \"func\"):\n return args.func(args)\n print(\"Error: Unknown skills command\")\n return 1\n\n if self._skills_parser is not None:\n self._skills_parser.print_help()\n return 0\n" }, { "path": "trl/cli/commands/training.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport importlib\nfrom argparse import Namespace\n\nfrom .base import Command, CommandContext\n\n\ndef _subtract_subsequence(lst: list[str], subseq: list[str]) -> list[str]:\n \"\"\"Return lst with the ordered subsequence subseq removed.\"\"\"\n sub_iter = iter(subseq)\n current = next(sub_iter, None)\n result = []\n for item in lst:\n if current is not None and item == current:\n current = next(sub_iter, None)\n else:\n result.append(item)\n return result\n\n\nclass TrainingCommand(Command):\n \"\"\"\n Generic CLI command that launches a training script with accelerate.\n\n The script `trl/scripts/.py` must expose a `make_parser()` function.\n\n Parameters:\n name (`str`):\n CLI subcommand name (e.g. `\"dpo\"`).\n \"\"\"\n\n def __init__(self, name: str):\n super().__init__(name=name, help_text=f\"Run the {name} training script\")\n\n def register(self, subparsers) -> None:\n subparsers.add_parser(self.name, help=self.help_text, add_help=False)\n\n def run(self, args: Namespace, context: CommandContext) -> int:\n from ..accelerate_config import resolve_accelerate_config_argument\n from ..accelerate_launcher import launch_training_script\n\n module = importlib.import_module(f\"...scripts.{self.name}\", package=__package__)\n all_args = context.argv_after(self.name)\n parser = module.make_parser(prog=f\"trl {self.name}\")\n\n # Handles -h (exits). Returns config_remaining and cli_remaining separately.\n # cli_remaining is an ordered subsequence of all_args; config_remaining is not.\n *_, config_remaining, cli_remaining = parser.parse_args_and_config(\n all_args, return_remaining_strings=True, separate_remaining_strings=True\n )\n launch_args = resolve_accelerate_config_argument(config_remaining + cli_remaining)\n training_script_args = _subtract_subsequence(all_args, cli_remaining)\n\n launch_training_script(\n script_name=f\"{self.name}.py\",\n launch_args=launch_args,\n training_script_args=training_script_args,\n )\n return 0\n" }, { "path": "trl/cli/commands/vllm_serve.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom argparse import Namespace\n\nfrom .base import Command, CommandContext\n\n\nclass VllmServeCommand(Command):\n \"\"\"CLI command for serving TRL models with vLLM.\"\"\"\n\n def __init__(self):\n super().__init__(name=\"vllm-serve\", help_text=\"Serve a model with vLLM\")\n\n def register(self, subparsers) -> None:\n subparsers.add_parser(self.name, help=self.help_text, add_help=False)\n\n def run(self, args: Namespace, context: CommandContext) -> int:\n from ...scripts.vllm_serve import main as vllm_serve_main\n from ...scripts.vllm_serve import make_parser as make_vllm_serve_parser\n\n parser = make_vllm_serve_parser(prog=\"trl vllm-serve\")\n (script_args,) = parser.parse_args_and_config(args=context.argv_after(self.name))\n vllm_serve_main(script_args)\n return 0\n" }, { "path": "trl/cli/main.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport sys\nfrom argparse import ArgumentParser\n\nfrom .commands import get_commands\nfrom .commands.base import Command, CommandContext\n\n\ndef _build_parser(commands: list[Command]) -> ArgumentParser:\n parser = ArgumentParser(prog=\"trl\", allow_abbrev=False)\n subparsers = parser.add_subparsers(help=\"available commands\", dest=\"command\")\n\n for command in commands:\n command.register(subparsers)\n\n return parser\n\n\ndef main(argv: list[str] | None = None) -> int:\n \"\"\"Run the TRL CLI.\"\"\"\n commands = get_commands()\n commands_by_name = {command.name: command for command in commands}\n parser = _build_parser(commands)\n argv = list(sys.argv[1:] if argv is None else argv)\n\n args, _ = parser.parse_known_args(argv)\n command_name = getattr(args, \"command\", None)\n if command_name is None:\n parser.print_help()\n return 0\n\n command = commands_by_name[command_name]\n context = CommandContext(argv=argv)\n return command.run(args, context)\n\n\nif __name__ == \"__main__\":\n raise SystemExit(main())\n" }, { "path": "trl/data_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport copy\nfrom collections import defaultdict, deque\nfrom collections.abc import Callable, Sequence\nfrom itertools import takewhile\nfrom typing import Any, Literal, TypeVar\n\nimport numpy as np\nimport pyarrow as pa\nimport pyarrow.compute as pc\nimport pyarrow.types\nfrom datasets import Dataset, DatasetDict, IterableDatasetDict\nfrom transformers import PreTrainedTokenizerBase, ProcessorMixin\n\n\nDatasetType = TypeVar(\"DatasetType\", Dataset, DatasetDict)\n\n\ndef prepare_multimodal_messages(messages: list[dict[str, Any]], images: list) -> list[dict[str, Any]]:\n # docstyle-ignore # because is not parsable in the code block\n \"\"\"\n Convert messages into a structured multimodal format and inject the provided images into the message contents.\n\n Args:\n messages (`list[dict[str, Any]]`):\n Messages with `\"role\"`, `\"content\"` (or `\"tool_calls\"`). Content may be a raw string before transformation.\n List of messages with a `\"role\"` key (`\"system\"`, `\"user\"`, `\"assistant\"`, or `\"tool\"`) and a `\"content\"` key containing\n either a string or a list of structured blocks if already prepared. Optionally, the `\"content\"` might\n be `None` or not provided in favour of `\"tool_calls\"` in the `\"assistant\"` turns if applicable.\n images (`list`):\n List of image objects to insert. Can be empty if no images are included in the messages.\n\n Returns:\n `list[dict[str, Any]]`: A deep-copied list of messages where every `\"content\"` value is a list of structured\n content blocks, and all `\"image\"` placeholders are populated with the corresponding image objects. If the\n assistant turns contains `\"tool_calls\"`, then the `\"content\"` might be empty.\n\n Notes:\n - When the input `messages` isn't already in the structured format, (i.e., all `\"content\"` values are strings),\n the function transforms them into the structured format by wrapping text in `{\"type\": \"text\", \"text\": ...}`\n and inserting `{\"type\": \"image\"}` placeholders for the images *before* the first user message.\n If the number of placeholders does not match the number of provided images, an error is raised.\n - When the input `messages` contains either `\"tool_calls\"` in the `\"assistant\"` turns, or `\"tool\"` roles with\n `\"content\"` and `\"name\"` those are left as-is, since those don't require any specific handling for multimodal data.\n\n Example:\n ```python\n # Input\n [\n {\"role\": \"user\", \"content\": \"What's in this image?\"},\n {\"role\": \"assistant\", \"content\": \"It looks like a cat.\"},\n ]\n\n # Output, one image provided\n [\n {\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": }, {\"type\": \"text\", \"text\": \"What's in this image?\"}]},\n {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"text\": \"It looks like a cat.\"}]},\n ]\n ```\n \"\"\"\n\n messages = copy.deepcopy(messages) # avoid modifying the original messages\n\n # First, convert all messages to the structured format if needed, and insert image placeholders if needed\n images_included = False\n for message in messages:\n if message[\"role\"] == \"system\":\n if isinstance(message[\"content\"], str): # if already prepared, the content will be a list\n message[\"content\"] = [{\"type\": \"text\", \"text\": message[\"content\"]}]\n elif message[\"role\"] == \"user\":\n if isinstance(message[\"content\"], str) and not images_included:\n image_entries = [{\"type\": \"image\"} for _ in range(len(images))]\n message[\"content\"] = [*image_entries, {\"type\": \"text\", \"text\": message[\"content\"]}]\n images_included = True\n elif isinstance(message[\"content\"], str) and images_included:\n message[\"content\"] = [{\"type\": \"text\", \"text\": message[\"content\"]}]\n elif message[\"role\"] == \"assistant\":\n if message.get(\"content\") and isinstance(message[\"content\"], str):\n message[\"content\"] = [{\"type\": \"text\", \"text\": message[\"content\"]}]\n elif message[\"role\"] == \"tool\":\n # NOTE: `tool` contains `name` (name of the tool used) and `content` (output of the tool call as a string)\n # but there's no need to prepare it for multimodal specifically but rather leave it as-is\n continue\n else:\n raise ValueError(\n f\"Invalid role in message: {message['role']}. Expected 'system', 'user', 'assistant', or 'tool'.\"\n )\n\n # Then, check that the number of image placeholders matches the number of images provided\n num_placeholders = sum(\n sum(1 for part in message[\"content\"] if part[\"type\"] == \"image\")\n for message in messages\n if message.get(\"content\") and message[\"role\"] != \"tool\"\n )\n if num_placeholders != len(images):\n raise ValueError(\n f\"Number of images provided ({len(images)}) does not match number of image placeholders ({num_placeholders}).\"\n )\n\n # Then, fill in the actual images in the placeholders\n img_idx = 0\n for message in messages:\n if not message.get(\"content\") or message[\"role\"] == \"tool\":\n continue\n for part in message[\"content\"]:\n if part[\"type\"] == \"image\":\n part[\"image\"] = images[img_idx]\n img_idx += 1\n\n return messages\n\n\ndef prepare_multimodal_messages_vllm(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:\n # docstyle-ignore # because is not parsable in the code block\n \"\"\"\n Convert structured multimodal messages into a format compatible with vLLM. Replaces `\"type\": \"image\"` blocks with\n `\"type\": \"image_pil\"` blocks, and `\"image\": Image` with `\"image_pil\": Image`.\n\n Args:\n messages (`list[dict[str, Any]]`):\n Messages with `\"role\"` and `\"content\"`. Content is expected to be a list of structured blocks.\n\n Returns:\n `list[dict[str, Any]]`:\n A deep-copied list of messages compatible with vLLM's expected input format.\n\n Example:\n ```python\n # Input\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": }, {\"type\": \"text\", \"text\": \"What's in this image?\"}]}]\n\n # Output\n [{\"role\": \"user\", \"content\": [{\"type\": \"image_pil\", \"image_pil\": }, {\"type\": \"text\", \"text\": \"What's in this image?\"}]}]\n ```\n \"\"\"\n messages = copy.deepcopy(messages) # avoid modifying the original messages\n for message in messages:\n if isinstance(message[\"content\"], list):\n for part in message[\"content\"]:\n if part[\"type\"] == \"image\":\n part[\"type\"] = \"image_pil\" # vLLM expects 'image_pil' key for images\n part[\"image_pil\"] = part.pop(\"image\")\n return messages\n\n\ndef is_conversational(example: dict[str, Any]) -> bool:\n r\"\"\"\n Check if the example is in a conversational format.\n\n Args:\n example (`dict[str, Any]`):\n A single data entry of a dataset. The example can have different keys depending on the dataset type.\n\n Returns:\n `bool`:\n `True` if the data is in a conversational format, `False` otherwise.\n\n Examples:\n\n ```python\n >>> example = {\"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}]}\n >>> is_conversational(example)\n True\n\n >>> example = {\"prompt\": \"The sky is\"}\n >>> is_conversational(example)\n False\n ```\n \"\"\"\n supported_keys = [\"prompt\", \"chosen\", \"rejected\", \"completion\", \"messages\"]\n example_keys = {key for key in example.keys() if key in supported_keys}\n\n # It must have one of the supported keys\n if example_keys:\n key = example_keys.pop() # take the first supported key\n maybe_messages = example[key]\n # It must be a list of messages\n if isinstance(maybe_messages, list):\n maybe_message = maybe_messages[0]\n # Each message must a list of dictionaries with keys \"role\" and \"content\"\n if isinstance(maybe_message, dict) and \"role\" in maybe_message:\n return True\n\n return False\n\n\ndef apply_chat_template(\n example: dict[str, list[dict[str, str]]],\n tokenizer: PreTrainedTokenizerBase | ProcessorMixin,\n tools: list[dict | Callable] | None = None,\n **template_kwargs,\n) -> dict[str, str]:\n r\"\"\"\n Apply a chat template to a conversational example along with the schema for a list of functions in `tools`.\n\n For more details, see [`maybe_apply_chat_template`].\n \"\"\"\n # Check that the example has the correct keys\n supported_keys = [\"prompt\", \"chosen\", \"rejected\", \"completion\", \"messages\", \"label\"]\n example_keys = {key for key in example.keys() if key in supported_keys}\n if example_keys not in [\n {\"messages\"}, # language modeling\n {\"prompt\"}, # prompt-only\n {\"prompt\", \"completion\"}, # prompt-completion\n {\"prompt\", \"chosen\", \"rejected\"}, # preference\n {\"chosen\", \"rejected\"}, # preference with implicit prompt\n {\"prompt\", \"completion\", \"label\"}, # unpaired preference\n ]:\n raise KeyError(f\"Invalid keys in the example: {example_keys}\")\n\n # Apply the chat template to the whole conversation\n if \"messages\" in example:\n messages = tokenizer.apply_chat_template(\n example[\"messages\"],\n tools=tools,\n tokenize=False,\n **example.get(\"chat_template_kwargs\", {}),\n **template_kwargs,\n )\n\n # Apply the chat template to the prompt, adding the generation prompt\n if \"prompt\" in example:\n last_role = example[\"prompt\"][-1][\"role\"]\n if last_role in [\"user\", \"tool\"]:\n add_generation_prompt = True\n continue_final_message = False\n elif last_role == \"assistant\":\n add_generation_prompt = False\n continue_final_message = True\n else:\n raise ValueError(f\"Invalid role in the last message: {last_role}\")\n prompt = tokenizer.apply_chat_template(\n example[\"prompt\"],\n tools=tools,\n continue_final_message=continue_final_message,\n tokenize=False,\n add_generation_prompt=add_generation_prompt,\n **example.get(\"chat_template_kwargs\", {}),\n **template_kwargs,\n )\n\n # Apply the chat template to the entire prompt + completion\n if \"prompt\" in example: # explicit prompt and prompt-completion case\n if \"chosen\" in example:\n prompt_chosen = tokenizer.apply_chat_template(\n example[\"prompt\"] + example[\"chosen\"],\n tools=tools,\n tokenize=False,\n **example.get(\"chat_template_kwargs\", {}),\n **template_kwargs,\n )\n # DeepSeek-R1 inserts a token when using `add_generation_prompt`, which can cause discrepancies\n # between the prompt alone and the combined prompt+completion. To ensure consistency, we extract the\n # common prefix between the two. In most cases, this is a no-op.\n prompt = \"\".join(x for x, _ in takewhile(lambda x: x[0] == x[1], zip(prompt, prompt_chosen, strict=False)))\n\n chosen = prompt_chosen[len(prompt) :]\n if \"rejected\" in example and \"prompt\" in example: # explicit prompt\n prompt_rejected = tokenizer.apply_chat_template(\n example[\"prompt\"] + example[\"rejected\"],\n tools=tools,\n tokenize=False,\n **example.get(\"chat_template_kwargs\", {}),\n **template_kwargs,\n )\n # Handle DeepSeek-R1 token, see the above comment for details\n prompt = \"\".join(\n x for x, _ in takewhile(lambda x: x[0] == x[1], zip(prompt, prompt_rejected, strict=False))\n )\n rejected = prompt_rejected[len(prompt) :]\n if \"completion\" in example:\n prompt_completion = tokenizer.apply_chat_template(\n example[\"prompt\"] + example[\"completion\"],\n tools=tools,\n tokenize=False,\n **example.get(\"chat_template_kwargs\", {}),\n **template_kwargs,\n )\n # Handle DeepSeek-R1 token, see the above comment for details\n prompt = \"\".join(\n x for x, _ in takewhile(lambda x: x[0] == x[1], zip(prompt, prompt_completion, strict=False))\n )\n completion = prompt_completion[len(prompt) :]\n else: # implicit prompt case\n if \"chosen\" in example:\n chosen = tokenizer.apply_chat_template(\n example[\"chosen\"],\n tools=tools,\n tokenize=False,\n **example.get(\"chat_template_kwargs\", {}),\n **template_kwargs,\n )\n if \"rejected\" in example:\n rejected = tokenizer.apply_chat_template(\n example[\"rejected\"],\n tools=tools,\n tokenize=False,\n **example.get(\"chat_template_kwargs\", {}),\n **template_kwargs,\n )\n\n # Extract the completion by removing the prompt part from the prompt-completion string\n output = {}\n if \"messages\" in example:\n output[\"text\"] = messages\n if \"prompt\" in example:\n output[\"prompt\"] = prompt\n if \"chosen\" in example:\n output[\"chosen\"] = chosen\n if \"rejected\" in example:\n output[\"rejected\"] = rejected\n if \"completion\" in example:\n output[\"completion\"] = completion\n if \"label\" in example:\n output[\"label\"] = example[\"label\"]\n\n return output\n\n\ndef maybe_apply_chat_template(\n example: dict[str, list[dict[str, str]]],\n tokenizer: PreTrainedTokenizerBase,\n tools: list[dict | Callable] | None = None,\n **template_kwargs: Any,\n) -> dict[str, str]:\n r\"\"\"\n If the example is in a conversational format, apply a chat template to it.\n\n Args:\n example (`dict[str, list[dict[str, str]]`):\n Dictionary representing a single data entry of a conversational dataset. Each data entry can have different\n keys depending on the dataset type. The supported dataset types are:\n\n - Language modeling dataset: `\"messages\"`.\n - Prompt-only dataset: `\"prompt\"`.\n - Prompt-completion dataset: `\"prompt\"` and `\"completion\"`.\n - Preference dataset: `\"prompt\"`, `\"chosen\"`, and `\"rejected\"`.\n - Preference dataset with implicit prompt: `\"chosen\"` and `\"rejected\"`.\n - Unpaired preference dataset: `\"prompt\"`, `\"completion\"`, and `\"label\"`.\n\n For keys `\"messages\"`, `\"prompt\"`, `\"chosen\"`, `\"rejected\"`, and `\"completion\"`, the values are lists of\n messages, where each message is a dictionary with keys `\"role\"` and `\"content\"`. Additionally, the example\n may contain a `\"chat_template_kwargs\"` key, which is a dictionary of additional keyword arguments to pass\n to the chat template renderer.\n tokenizer ([`~transformers.PreTrainedTokenizerBase`]):\n Tokenizer to apply the chat template with.\n tools (`list[dict | Callable]`, *optional*):\n A list of tools (callable functions) that will be accessible to the model. If the template does not support\n function calling, this argument will have no effect.\n **template_kwargs (`Any`, *optional*):\n Additional kwargs to pass to the template renderer. Will be accessible by the chat template.\n\n Returns:\n `dict[str, str]`:\n Formatted example with the chat template applied.\n\n Notes:\n - This function does not alter the keys, except for Language modeling dataset, where `\"messages\"` is replaced\n by `\"text\"`.\n\n - In case of prompt-only data, if the last role is `\"user\"`, the generation prompt is added to the prompt.\n Else, if the last role is `\"assistant\"`, the final message is continued.\n\n Example:\n\n ```python\n >>> from transformers import AutoTokenizer\n\n >>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-128k-instruct\")\n >>> example = {\n ... \"prompt\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}],\n ... \"completion\": [{\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n ... }\n >>> apply_chat_template(example, tokenizer)\n {'prompt': '<|user|>\\nWhat color is the sky?<|end|>\\n<|assistant|>\\n', 'completion': 'It is blue.<|end|>\\n'}\n ```\n \"\"\"\n if is_conversational(example):\n return apply_chat_template(example, tokenizer, tools, **template_kwargs)\n else:\n return example\n\n\ndef _unpair_row(examples: list[dict[str, list[dict[str, str]]]]) -> list[dict[str, list[dict[str, str]]]]:\n batch_size = len(examples[\"chosen\"])\n new_rows = {\n \"completion\": examples[\"chosen\"] + examples[\"rejected\"],\n \"label\": [True] * batch_size + [False] * batch_size,\n }\n if \"prompt\" in examples:\n new_rows[\"prompt\"] = examples[\"prompt\"] + examples[\"prompt\"]\n return new_rows\n\n\ndef unpair_preference_dataset(\n dataset: DatasetType, num_proc: int | None = None, desc: str | None = None\n) -> DatasetType:\n r\"\"\"\n Unpair a preference dataset.\n\n Args:\n dataset ([`~datasets.Dataset`] or [`~datasets.DatasetDict`]):\n Preference dataset to unpair. The dataset must have columns `\"chosen\"`, `\"rejected\"` and optionally\n `\"prompt\"`.\n num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n desc (`str`, *optional*):\n Meaningful description to be displayed alongside with the progress bar while mapping examples.\n\n Returns:\n [`~datasets.Dataset`]: The unpaired preference dataset.\n\n Example:\n\n ```python\n >>> from datasets import Dataset\n\n >>> dataset_dict = {\n ... \"prompt\": [\"The sky is\", \"The sun is\"],\n ... \"chosen\": [\" blue.\", \"in the sky.\"],\n ... \"rejected\": [\" green.\", \" in the sea.\"],\n ... }\n >>> dataset = Dataset.from_dict(dataset_dict)\n >>> dataset = unpair_preference_dataset(dataset)\n >>> dataset\n Dataset({\n features: ['prompt', 'completion', 'label'],\n num_rows: 4\n })\n\n >>> dataset[0]\n {'prompt': 'The sky is', 'completion': ' blue.', 'label': True}\n ```\n \"\"\"\n return dataset.map(_unpair_row, batched=True, remove_columns=[\"chosen\", \"rejected\"], num_proc=num_proc, desc=desc)\n\n\ndef maybe_unpair_preference_dataset(\n dataset: DatasetType, num_proc: int | None = None, desc: str | None = None\n) -> DatasetType:\n r\"\"\"\n Unpair a preference dataset if it is paired.\n\n Args:\n dataset ([`~datasets.Dataset`] or [`~datasets.DatasetDict`]):\n Preference dataset to unpair. The dataset must have columns `\"chosen\"`, `\"rejected\"` and optionally\n `\"prompt\"`.\n num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n desc (`str`, *optional*):\n Meaningful description to be displayed alongside with the progress bar while mapping examples.\n\n Returns:\n [`~datasets.Dataset`] or [`~datasets.DatasetDict`]: The unpaired preference dataset if it was paired, otherwise\n the original dataset.\n\n Example:\n\n ```python\n >>> from datasets import Dataset\n\n >>> dataset_dict = {\n ... \"prompt\": [\"The sky is\", \"The sun is\"],\n ... \"chosen\": [\" blue.\", \"in the sky.\"],\n ... \"rejected\": [\" green.\", \" in the sea.\"],\n ... }\n >>> dataset = Dataset.from_dict(dataset_dict)\n >>> dataset = unpair_preference_dataset(dataset)\n >>> dataset\n Dataset({\n features: ['prompt', 'completion', 'label'],\n num_rows: 4\n })\n\n >>> dataset[0]\n {'prompt': 'The sky is', 'completion': ' blue.', 'label': True}\n ```\n \"\"\"\n if isinstance(dataset, DatasetDict):\n column_names = dataset[list(dataset.keys())[0]].column_names\n else:\n column_names = dataset.column_names\n if \"chosen\" in column_names and \"rejected\" in column_names:\n return unpair_preference_dataset(dataset, num_proc=num_proc, desc=desc)\n else:\n return dataset\n\n\ndef extract_prompt(example: dict[str, Sequence]) -> dict[str, Sequence]:\n r\"\"\"\n Extracts the shared prompt from a preference data example, where the prompt is implicit within both the chosen and\n rejected completions.\n\n The function identifies the longest common sequence (prefix) of conversation turns between the \"chosen\" and\n \"rejected\" completions and extracts this as the prompt. It then removes this prompt from the respective \"chosen\"\n and \"rejected\" completions.\n\n Args:\n example (`dict[str, list]`):\n A dictionary representing a single data entry in the preference dataset. It must contain the keys\n `\"chosen\"` and `\"rejected\"`, where each value is either conversational or standard (`str`).\n\n Returns:\n `dict[str, list]`: A dictionary containing:\n - `\"prompt\"`: The longest common prefix between the \"chosen\" and \"rejected\" completions.\n - `\"chosen\"`: The remainder of the \"chosen\" completion, with the prompt removed.\n - `\"rejected\"`: The remainder of the \"rejected\" completion, with the prompt removed.\n\n Examples:\n\n ```python\n >>> example = {\n ... \"chosen\": [\n ... {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ... {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ... ],\n ... \"rejected\": [\n ... {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ... {\"role\": \"assistant\", \"content\": \"It is green.\"},\n ... ],\n ... }\n >>> extract_prompt(example)\n {'prompt': [{'role': 'user', 'content': 'What color is the sky?'}],\n 'chosen': [{'role': 'assistant', 'content': 'It is blue.'}],\n 'rejected': [{'role': 'assistant', 'content': 'It is green.'}]}\n ```\n\n Or, with the `map` method of [`~datasets.Dataset`]:\n\n ```python\n >>> from trl import extract_prompt\n >>> from datasets import Dataset\n\n >>> dataset_dict = {\n ... \"chosen\": [\n ... [\n ... {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ... {\"role\": \"assistant\", \"content\": \"It is blue.\"},\n ... ],\n ... [\n ... {\"role\": \"user\", \"content\": \"Where is the sun?\"},\n ... {\"role\": \"assistant\", \"content\": \"In the sky.\"},\n ... ],\n ... ],\n ... \"rejected\": [\n ... [\n ... {\"role\": \"user\", \"content\": \"What color is the sky?\"},\n ... {\"role\": \"assistant\", \"content\": \"It is green.\"},\n ... ],\n ... [\n ... {\"role\": \"user\", \"content\": \"Where is the sun?\"},\n ... {\"role\": \"assistant\", \"content\": \"In the sea.\"},\n ... ],\n ... ],\n ... }\n >>> dataset = Dataset.from_dict(dataset_dict)\n >>> dataset = dataset.map(extract_prompt)\n >>> dataset[0]\n {'prompt': [{'role': 'user', 'content': 'What color is the sky?'}],\n 'chosen': [{'role': 'assistant', 'content': 'It is blue.'}],\n 'rejected': [{'role': 'assistant', 'content': 'It is green.'}]}\n ```\n \"\"\"\n for idx in range(min(len(example[\"chosen\"]), len(example[\"rejected\"]))):\n if example[\"chosen\"][idx] != example[\"rejected\"][idx]:\n if example[\"chosen\"][idx - 1] == \" \": # remove space before the prompt\n idx -= 1\n break\n return {\n \"prompt\": example[\"chosen\"][:idx],\n \"chosen\": example[\"chosen\"][idx:],\n \"rejected\": example[\"rejected\"][idx:],\n }\n\n\ndef maybe_extract_prompt(example: dict[str, list]) -> dict[str, list]:\n r\"\"\"\n Extracts the shared prompt from a preference data example, where the prompt is implicit within both the chosen and\n rejected completions.\n\n If the example already contains a `\"prompt\"` key, the function returns the example as is. For more details, see\n [`extract_prompt`].\n ```\n \"\"\"\n # Some dataset add a `\"prompt\"` column, even though the prompt is implicit and included in the \"chosen\" and\n # \"rejected\" completions. E.g.:\n # {\"prompt\": \"What color is the sky?\",\n # \"chosen\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is blue.\"}],\n # \"rejected\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}, {\"role\": \"assistant\", \"content\": \"It is green.\"}]}\n # That's why we check if the prompt is also conversational before deciding not to extract it.\n if \"chosen\" not in example or \"rejected\" not in example: # not a preference example\n return example\n if \"prompt\" in example:\n # Both conversational or both non-conversational\n chosen_conv = is_conversational({\"chosen\": example[\"chosen\"]})\n prompt_conv = is_conversational({\"prompt\": example[\"prompt\"]})\n if (chosen_conv and prompt_conv) or (not chosen_conv and not prompt_conv):\n return example\n return extract_prompt({\"chosen\": example[\"chosen\"], \"rejected\": example[\"rejected\"]})\n\n\ndef _get_dataset_format(dataset: DatasetType) -> dict[str, Any]:\n if isinstance(dataset, (DatasetDict, IterableDatasetDict)):\n dataset = dataset[next(iter(dataset))]\n if isinstance(dataset, Dataset):\n format = dataset.format\n else:\n format_type = dataset._formatting.format_type if dataset._formatting is not None else None\n format = {\"type\": format_type}\n format.update(format.pop(\"format_kwargs\", {}))\n return format\n\n\ndef _check_if_columns_can_be_packed(columns: list[pa.Array]):\n first_column_offsets = None\n for idx, column in enumerate(columns):\n if not (pyarrow.types.is_list(column.type) or pyarrow.types.is_large_list(column.type)):\n raise TypeError(\"Packing requires all columns to be lists of lists.\")\n\n if idx == 0:\n first_column_offsets = column.offsets\n elif not first_column_offsets.equals(column.offsets):\n raise ValueError(\"All columns must have values of the same length.\")\n\n\nclass _SegmentTree:\n \"\"\"\n A segment tree data structure that, when initialized as `_SegmentTree(maxval)`, efficiently finds the next larger\n value for a given input within the range [1, maxval].\n\n See [Fewer Truncations Improve Language Modeling](https://huggingface.co/papers/2404.10830) for more details.\n \"\"\"\n\n def __init__(self, maxval: int):\n self.maxval = maxval\n # For non-power-of-2 values, we need to round up to the next power of 2 for the tree size\n self.tree_size = 1 << (maxval - 1).bit_length()\n self.tree = [0] * (2 * self.tree_size)\n\n def add(self, val):\n assert 0 < val <= self.maxval\n i = self.tree_size + val - 1\n self.tree[i] = val\n while i > 1:\n i >>= 1\n left, right = self.tree[i << 1], self.tree[(i << 1) + 1]\n # Compare the values using if-else otherwise repeated calls to `builtins.max` become the bottleneck\n self.tree[i] = left if left >= right else right\n\n def remove(self, val):\n assert 0 < val <= self.maxval\n i = self.tree_size + val - 1\n self.tree[i] = 0\n while i > 1:\n i >>= 1\n left, right = self.tree[i << 1], self.tree[(i << 1) + 1]\n # Compare the values using if-else otherwise repeated calls to `builtins.max` become the bottleneck\n self.tree[i] = left if left >= right else right\n\n def search(self, val):\n assert 0 < val <= self.maxval\n i = 1\n while i < self.tree_size:\n if self.tree[i << 1] >= val:\n i = i << 1\n else:\n i = (i << 1) + 1\n return self.tree[i]\n\n\ndef _pack_bfd(\n examples: pa.Table, seq_length: int, on_seq_length_overflow: Literal[\"truncate\", \"split\"] = \"truncate\"\n) -> pa.Table:\n \"\"\"Pack sequences in a pyarrow Table using Best Fit Decreasing strategy.\"\"\"\n columns = [column.chunks[0] for column in examples.combine_chunks().columns]\n _check_if_columns_can_be_packed(columns)\n assert len(columns) > 0\n\n lengths = pc.list_value_length(columns[0])\n\n # Filter out empty sequences\n non_empty_mask = pc.greater(lengths, 0)\n columns = [pc.filter(column, non_empty_mask) for column in columns]\n lengths = pc.filter(lengths, non_empty_mask)\n\n if on_seq_length_overflow == \"truncate\":\n columns = [pc.list_slice(column, 0, seq_length) for column in columns]\n elif on_seq_length_overflow == \"split\":\n lengths = lengths.to_numpy()\n # Split the sequences longer than `seq_length` into chunks (of length `seq_length` or less) while respecting sequence boundaries\n num_fragments = np.ceil(lengths / seq_length).astype(int)\n offsets = np.arange(np.sum(num_fragments) + 1, dtype=columns[0].offsets.type.to_pandas_dtype()) * seq_length\n # \"Left-shift\" the offsets to account for the last fragment of each original sequence possibly being shorter than `seq_length`\n diff = np.zeros_like(offsets)\n diff[np.cumsum(num_fragments)] = -lengths % seq_length\n diff = np.cumsum(diff)\n offsets -= diff\n columns = [\n type(column).from_arrays(offsets.astype(column.offsets.type.to_pandas_dtype()), column.values)\n for column in columns\n ]\n else:\n raise ValueError(f\"Invalid `on_seq_length_overflow`: {on_seq_length_overflow}. Use 'truncate' or 'split'.\")\n\n examples = pa.Table.from_arrays(columns, names=examples.column_names)\n lengths = pc.list_value_length(columns[0])\n examples = examples.append_column(\"seq_lengths\", lengths) # Allows us to later construct `position_ids`\n ids = np.arange(len(examples))\n lengths = pc.make_struct(lengths, ids)\n lengths = lengths.sort(\"descending\", by=0)\n\n # Greedy BFD binning using a segment tree to quickly find best-fit remaining space.\n segment_tree = _SegmentTree(seq_length)\n segment_tree.add(seq_length) # the max, `seq_length` bin is always available\n space_to_bin = defaultdict(deque)\n\n # Bin is represented as a dict (of example ids and sum of their lengths) to allow in-place updates\n bins: list[dict] = []\n for length, idx in zip(lengths.field(0).to_numpy(), lengths.field(1).to_numpy(), strict=True):\n space = segment_tree.search(length)\n\n if space < seq_length:\n # Use existing bin with exactly this amount of space\n bin = space_to_bin[space].popleft()\n else:\n # Create a new bin\n bin = {\"ids\": [], \"length\": 0}\n bins.append(bin)\n\n bin[\"ids\"].append(idx)\n bin[\"length\"] += length\n if space < seq_length and not space_to_bin[space]:\n segment_tree.remove(space)\n\n space = space - length\n space_to_bin[space].append(bin)\n if space > 0:\n segment_tree.add(space)\n\n examples = pc.take(examples, [id_ for bin in bins for id_ in bin[\"ids\"]])\n offsets = np.cumsum([0] + [bin[\"length\"] for bin in bins])\n\n assert all(\n column.num_chunks == 1 for column in examples.columns\n ) # `pc.take` returns a ChunkedArray with a single chunk\n\n lengths = examples[\"seq_lengths\"].chunks[0]\n examples = examples.drop_columns(\"seq_lengths\")\n lengths = pa.ListArray.from_arrays(np.cumsum([0] + [len(bin[\"ids\"]) for bin in bins], dtype=np.int32), lengths)\n\n columns = []\n for column in examples.columns:\n column = column.chunks[0]\n assert pa.types.is_list(column.type) or pa.types.is_large_list(column.type)\n dtype = column.offsets.type.to_pandas_dtype()\n column = type(column).from_arrays(offsets.astype(dtype), column.values)\n columns.append(column)\n return pa.Table.from_arrays(columns + [lengths], names=examples.column_names + [\"seq_lengths\"])\n\n\ndef _pack_wrapped(examples: pa.Table, seq_length: int) -> pa.Table:\n \"\"\"Pack sequences in a pyarrow Table using a wrapped strategy.\"\"\"\n columns = [column.chunks[0] for column in examples.combine_chunks().columns]\n _check_if_columns_can_be_packed(columns)\n offsets, values = columns[0].offsets, columns[0].values\n values = values[offsets[0].as_py() : offsets[-1].as_py()]\n num_elements = len(values)\n offsets = np.arange(0, num_elements, seq_length, dtype=columns[0].offsets.type.to_pandas_dtype())\n offsets = np.concatenate((offsets, [num_elements]))\n columns = [\n type(column).from_arrays(offsets.astype(column.offsets.type.to_pandas_dtype()), column.values)\n for column in columns\n ]\n return pa.Table.from_arrays(columns, names=examples.column_names)\n\n\ndef pack_dataset(\n dataset: DatasetType,\n seq_length: int,\n strategy: str = \"bfd\",\n map_kwargs: dict[str, Any] | None = None,\n) -> DatasetType:\n r\"\"\"\n Pack sequences in a dataset into chunks of size `seq_length`.\n\n Args:\n dataset ([`~datasets.Dataset`] or [`~datasets.DatasetDict`]):\n Dataset to pack\n seq_length (`int`):\n Target sequence length to pack to.\n strategy (`str`, *optional*, defaults to `\"bfd\"`):\n Packing strategy to use. Can be either:\n\n - `\"bfd\"` (Best Fit Decreasing): Preserves sequence boundaries and truncates sequences that exceed\n `seq_length`, discarding overflow tokens. Ideal for SFT and conversational datasets where maintaining\n conversation structure is important.\n - `\"bfd_split\"`: Similar to `\"bfd\"` but splits overflow sequences for packing into other examples. Prevents\n token loss for pre-training or long documents, but may break conversation structure in SFT datasets.\n - `\"wrapped\"`: Faster but more aggressive. Ignores sequence boundaries and will cut sequences in the middle\n to completely fill each packed sequence with data.\n map_kwargs (`dict`, *optional*):\n Additional keyword arguments to pass to the dataset's map method when packing examples.\n\n Returns:\n [`~datasets.Dataset`] or [`~datasets.DatasetDict`]: The dataset with packed sequences. The number of examples\n may decrease as sequences are combined.\n\n Example:\n ```python\n >>> from datasets import Dataset\n >>> from trl import pack_dataset\n\n >>> examples = {\n ... \"input_ids\": [[1, 2, 3, 4, 5], [6, 7], [8, 9, 10], [11]],\n ... \"attention_mask\": [[1, 1, 1, 0, 0], [1, 0], [1, 1, 0], [1]],\n ... }\n >>> dataset = Dataset.from_dict(examples)\n >>> # Default \"bfd\" strategy (SFT-friendly): truncates long sequences\n >>> packed_dataset = pack_dataset(dataset, seq_length=4, strategy=\"bfd\")\n >>> packed_dataset[:]\n {'input_ids': [[1, 2, 3, 4], [8, 9, 10, 11], [6, 7]],\n 'attention_mask': [[1, 1, 1, 0], [1, 1, 0, 1], [1, 0]],\n 'seq_lengths': [[4], [3, 1], [2]]}\n\n >>> # \"bfd_split\" strategy: preserves all tokens\n >>> packed_dataset = pack_dataset(dataset, seq_length=4, strategy=\"bfd_split\")\n >>> packed_dataset[:]\n {'input_ids': [[1, 2, 3, 4], [8, 9, 10, 5], [6, 7, 11]],\n 'attention_mask': [[1, 1, 1, 0], [1, 1, 0, 0], [1, 0, 1]],\n 'seq_lengths': [[4], [3, 1], [2, 1]]}\n ```\n \"\"\"\n if map_kwargs is None:\n map_kwargs = {}\n\n valid_strategies = (\"bfd\", \"bfd_split\", \"wrapped\")\n if strategy not in valid_strategies:\n raise ValueError(f\"Invalid packing strategy '{strategy}', must be one of {valid_strategies}.\")\n format = _get_dataset_format(dataset)\n dataset = dataset.with_format(\"arrow\")\n if strategy == \"bfd\":\n dataset = dataset.map(\n _pack_bfd,\n batched=True,\n fn_kwargs={\"seq_length\": seq_length, \"on_seq_length_overflow\": \"truncate\"},\n **map_kwargs,\n )\n elif strategy == \"bfd_split\":\n dataset = dataset.map(\n _pack_bfd,\n batched=True,\n fn_kwargs={\"seq_length\": seq_length, \"on_seq_length_overflow\": \"split\"},\n **map_kwargs,\n )\n elif strategy == \"wrapped\":\n dataset = dataset.map(_pack_wrapped, batched=True, fn_kwargs={\"seq_length\": seq_length}, **map_kwargs)\n else:\n raise ValueError(f\"Invalid packing strategy: '{strategy}', must be one of {valid_strategies}.\")\n\n if strategy in {\"bfd\", \"bfd_split\"} and \"columns\" in format:\n format[\"columns\"] = format[\"columns\"] + [\"seq_lengths\"]\n\n dataset = dataset.with_format(**format)\n return dataset\n\n\ndef truncate_dataset(\n dataset: DatasetType,\n max_length: int,\n truncation_mode: str = \"keep_start\",\n map_kwargs: dict[str, Any] | None = None,\n) -> DatasetType:\n r\"\"\"\n Truncate sequences in a dataset to a specified `max_length`.\n\n Args:\n dataset ([`~datasets.Dataset`] or [`~datasets.DatasetDict`]):\n Dataset to truncate.\n max_length (`int`):\n Maximum sequence length to truncate to.\n truncation_mode (`str`, *optional*, defaults to `\"keep_start\"`):\n Whether to keep the start (`\"keep_start\"`) or the end (`\"keep_end\"`) of the sequence when truncating.\n map_kwargs (`dict`, *optional*):\n Additional keyword arguments to pass to the dataset's map method when truncating examples.\n\n Returns:\n [`~datasets.Dataset`] or [`~datasets.DatasetDict`]: The dataset with truncated sequences.\n\n Example:\n ```python\n >>> from datasets import Dataset\n\n >>> examples = {\n ... \"input_ids\": [[1, 2, 3], [4, 5, 6, 7], [8]],\n ... \"attention_mask\": [[0, 1, 1], [0, 0, 1, 1], [1]],\n ... }\n >>> dataset = Dataset.from_dict(examples)\n >>> truncated_dataset = truncate_dataset(dataset, max_length=2)\n >>> truncated_dataset[:]\n {'input_ids': [[1, 2], [4, 5], [8]],\n 'attention_mask': [[0, 1], [0, 0], [1]]}\n ```\n \"\"\"\n if truncation_mode not in {\"keep_start\", \"keep_end\"}:\n raise ValueError(f\"Invalid truncation mode '{truncation_mode}'.\")\n if map_kwargs is None:\n map_kwargs = {}\n\n def truncate(examples):\n truncated_columns = []\n for column in examples.columns:\n if pyarrow.types.is_list(column.type) or pyarrow.types.is_large_list(column.type):\n if truncation_mode == \"keep_start\":\n column = pc.list_slice(column, 0, max_length)\n else: # keep_end\n column = (\n pa.array([[] for _ in range(len(column))], type=column.type)\n if max_length == 0\n else pa.array([values[-max_length:] for values in column.to_pylist()], type=column.type)\n )\n truncated_columns.append(column)\n return pa.Table.from_arrays(truncated_columns, names=examples.column_names)\n\n format = _get_dataset_format(dataset)\n dataset = dataset.with_format(\"arrow\")\n dataset = dataset.map(truncate, batched=True, **map_kwargs)\n dataset = dataset.with_format(**format)\n return dataset\n\n\ndef is_conversational_from_value(example: dict[str, Any]) -> bool:\n r\"\"\"\n Check if the example is in a conversational format (from/value). Note that this format isn't recommended. Prefer\n the ChatML format (role/content)\n\n Args:\n example (`dict[str, Any]`):\n A single data entry of a dataset. The example can have different keys depending on the dataset type.\n\n Returns:\n `bool`:\n `True` if the data is in a conversational Chatformat, `False` otherwise.\n\n Examples:\n\n ```python\n >>> example = {\"conversations\": [{\"from\": \"user\", \"value\": \"What color is the sky?\"}]}\n >>> is_conversational_from_value(example)\n True\n\n >>> example = {\"conversations\": [{\"role\": \"user\", \"content\": \"What color is the sky?\"}]}\n >>> is_conversational_from_value(example)\n False\n\n >>> example = {\"conversations\": \"The sky is\"}\n >>> is_conversational_from_value(example)\n False\n ```\n \"\"\"\n maybe_messages = example.get(\"conversations\")\n # It must be a list of messages\n if isinstance(maybe_messages, list):\n maybe_message = maybe_messages[0]\n # Each message must a list of dictionaries with keys \"from\" and \"value\"\n if isinstance(maybe_message, dict) and \"from\" in maybe_message and \"value\" in maybe_message:\n return True\n\n return False\n\n\ndef maybe_convert_to_chatml(example: dict[str, list]) -> dict[str, list]:\n \"\"\"\n Convert a conversational dataset with fields `from` and `value` to ChatML format.\n\n This function modifies conversational data to align with OpenAI's ChatML format:\n - Replaces the key `\"from\"` with `\"role\"` in message dictionaries.\n - Replaces the key `\"value\"` with `\"content\"` in message dictionaries.\n - Renames `\"conversations\"` to `\"messages\"` for consistency with ChatML.\n\n Args:\n example (`dict[str, list]`):\n A single data entry containing a list of messages.\n\n Returns:\n `dict[str, list]`:\n Example reformatted to ChatML style.\n\n Example:\n ```python\n >>> from trl import maybe_convert_to_chatml\n\n >>> example = {\n ... \"conversations\": [\n ... {\"from\": \"user\", \"value\": \"What color is the sky?\"},\n ... {\"from\": \"assistant\", \"value\": \"It is blue.\"},\n ... ]\n ... }\n >>> maybe_convert_to_chatml(example)\n {'messages': [{'role': 'user', 'content': 'What color is the sky?'},\n {'role': 'assistant', 'content': 'It is blue.'}]}\n ```\n \"\"\"\n # List of possible keys containing message lists\n for key in [\"prompt\", \"completion\", \"chosen\", \"rejected\", \"messages\", \"conversations\"]:\n if key in example and isinstance(example[key], list):\n messages = example[key]\n for message in messages:\n if isinstance(message, dict):\n if \"from\" in message:\n message[\"role\"] = message.pop(\"from\")\n if \"value\" in message:\n message[\"content\"] = message.pop(\"value\")\n\n # Rename \"conversations\" to \"messages\"\n if \"conversations\" in example:\n example[\"messages\"] = example.pop(\"conversations\")\n\n return example\n" }, { "path": "trl/experimental/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\"\"\"\nExperimental submodule for TRL.\n\nThis submodule contains unstable or incubating features. Anything here may change (or be removed) in any release\nwithout deprecation. Use at your own risk.\n\nTo silence this notice set environment variable TRL_EXPERIMENTAL_SILENCE=1.\n\"\"\"\n\nimport os\nimport warnings\n\nfrom ..import_utils import TRLExperimentalWarning\n\n\nif not os.environ.get(\"TRL_EXPERIMENTAL_SILENCE\"):\n warnings.warn(\n \"You are importing from 'trl.experimental'. APIs here are unstable and may change or be removed without \"\n \"notice. Silence this warning by setting environment variable TRL_EXPERIMENTAL_SILENCE=1.\",\n TRLExperimentalWarning,\n stacklevel=2,\n )\n" }, { "path": "trl/experimental/async_grpo/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .async_grpo_config import AsyncGRPOConfig\nfrom .async_grpo_trainer import AsyncGRPOTrainer\n" }, { "path": "trl/experimental/async_grpo/async_grpo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom trl.trainer.base_config import _BaseConfig\n\n\n@dataclass\nclass AsyncGRPOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`AsyncGRPOTrainer`].\n\n This class includes only the parameters that are specific to asynchronous GRPO training. For a full list of\n training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values\n in this class may differ from those in [`~transformers.TrainingArguments`].\n\n Parameters:\n > Parameters that control generation\n\n num_generations (`int`, *optional*, defaults to `8`):\n Number of generations per prompt to sample.\n max_completion_length (`int`, *optional*, defaults to `2048`):\n Maximum number of tokens to generate per completion.\n temperature (`float`, *optional*, defaults to `1.0`):\n Temperature for sampling. The higher the temperature, the more random the completions.\n chat_template_kwargs (`dict[str, Any]`, *optional*):\n Additional keyword arguments to pass to the `apply_chat_template` function when generating completions.\n max_tool_calling_iterations (`int`, *optional*):\n Maximum number of tool-calling turns when training an agent. If `None`, there is no limit and generation\n stops when the model generates a response turn with no tool calls or when the total response length reaches\n `max_completion_length`.\n\n > Parameters that control the vLLM server\n\n vllm_server_base_url (`str`, *optional*, defaults to `\"http://localhost:8000\"`):\n Base URL of the vLLM server used for generation (e.g., `\"http://localhost:8000\"`).\n vllm_server_timeout (`float`, *optional*, defaults to `240.0`):\n Total timeout duration in seconds to wait for the vLLM server to be ready.\n request_timeout (`int`, *optional*, defaults to `600`):\n Timeout in seconds for individual HTTP requests to the vLLM server.\n\n > Parameters that control the training\n\n epsilon (`float`, *optional*, defaults to `0.2`):\n Lower-bound epsilon value for clipping.\n epsilon_high (`float`, *optional*, defaults to `0.2`):\n Upper-bound epsilon value for clipping.\n\n > Parameters that control the async rollout pipeline\n\n max_inflight_tasks (`int`, *optional*, defaults to `-1`):\n Maximum number of concurrent generation tasks sent to the vLLM server. Defaults to `-1` (auto), which\n sets it to `max_staleness * per_device_train_batch_size * gradient_accumulation_steps * num_processes`.\n If using tool-use environments, you may want to set this manually based on how many parallel environments\n you can run.\n max_staleness (`int`, *optional*, defaults to `4`):\n Maximum number of weight update steps a rollout sample can lag behind the current model version before\n being discarded.\n queue_maxsize (`int`, *optional*, defaults to `1024`):\n Maximum number of rollout samples to buffer in the rollout queue.\n weight_sync_steps (`int`, *optional*, defaults to `1`):\n Number of training steps between weight synchronizations to the vLLM server.\n\n > Parameters that control the logging\n\n log_completions (`bool`, *optional*, defaults to `False`):\n Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps.\n num_completions_to_print (`int`, *optional*, defaults to `3`):\n Number of completions to print when `log_completions=True`.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `1e-6` instead of `5e-5`.\n \"\"\"\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-6,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n logging_steps: float = field(\n default=1,\n metadata={\n \"help\": \"Log every X update steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, \"\n \"will be interpreted as ratio of total training steps.\"\n },\n )\n\n # Parameters that control generation\n num_generations: int = field(\n default=8,\n metadata={\"help\": \"Number of generations per prompt to sample.\"},\n )\n max_completion_length: int = field(\n default=2048,\n metadata={\"help\": \"Maximum number of tokens to generate per completion.\"},\n )\n temperature: float = field(\n default=1.0,\n metadata={\"help\": \"Temperature for sampling. The higher the temperature, the more random the completions.\"},\n )\n chat_template_kwargs: dict | None = field(\n default=None,\n metadata={\n \"help\": \"Additional keyword arguments to pass to the `apply_chat_template` function when generating \"\n \"completions.\"\n },\n )\n max_tool_calling_iterations: int | None = field(\n default=None,\n metadata={\n \"help\": \"Maximum number of tool-calling turns when training an agent. If `None`, there is no limit and \"\n \"generation stops when the model generates a response turn with no tool calls or when the total response \"\n \"length reaches `max_completion_length`.\"\n },\n )\n\n # Parameters that control the vLLM server\n vllm_server_base_url: str = field(\n default=\"http://localhost:8000\",\n metadata={\"help\": \"Base URL of the vLLM server used for generation (e.g., 'http://localhost:8000').\"},\n )\n vllm_server_timeout: float = field(\n default=240.0,\n metadata={\n \"help\": \"Total timeout duration in seconds to wait for the vLLM server to be ready. If the server is not \"\n \"up after the timeout, a `TimeoutError` is raised.\"\n },\n )\n request_timeout: int = field(\n default=600,\n metadata={\"help\": \"Timeout in seconds for individual HTTP requests to the vLLM server.\"},\n )\n\n # Parameters that control the training\n epsilon: float = field(\n default=0.2,\n metadata={\"help\": \"Lower-bound epsilon value for clipping.\"},\n )\n epsilon_high: float = field(\n default=0.2,\n metadata={\"help\": \"Upper-bound epsilon value for clipping.\"},\n )\n\n # Parameters that control the async rollout pipeline\n max_inflight_tasks: int = field(\n default=-1,\n metadata={\n \"help\": \"Maximum number of concurrent generation tasks sent to the vLLM server. Defaults to -1 (auto), \"\n \"which sets it to `max_staleness * per_device_train_batch_size * gradient_accumulation_steps * \"\n \"num_processes`. Generating more samples than this is wasteful since they will be discarded as stale \"\n \"before the trainer can consume them. If using tool-use environments, you may want to set this manually \"\n \"based on how many parallel environments you can run.\"\n },\n )\n max_staleness: int = field(\n default=4,\n metadata={\n \"help\": \"Maximum number of weight update steps a rollout sample can lag behind the current model version \"\n \"before being discarded.\"\n },\n )\n queue_maxsize: int = field(\n default=1024,\n metadata={\"help\": \"Maximum number of rollout samples to buffer in the rollout queue.\"},\n )\n weight_sync_steps: int = field(\n default=1,\n metadata={\"help\": \"Number of training steps between weight synchronizations to the vLLM server.\"},\n )\n\n # Parameters that control the logging\n log_completions: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is \"\n \"installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`.\"\n },\n )\n num_completions_to_print: int = field(\n default=3,\n metadata={\"help\": \"Number of completions to print when `log_completions=True`.\"},\n )\n\n def __post_init__(self):\n super().__post_init__()\n\n # Accelerator config: required for the async IterableDataset-backed dataloader to work correctly.\n # split_batches=True and dispatch_batches=True ensure that the main process drives the dataloader\n # and batches are broadcast to other processes rather than each process pulling independently.\n if not hasattr(self, \"accelerator_config\") or self.accelerator_config is None:\n self.accelerator_config = {\"split_batches\": True, \"dispatch_batches\": True}\n elif isinstance(self.accelerator_config, dict):\n self.accelerator_config[\"split_batches\"] = True\n self.accelerator_config[\"dispatch_batches\"] = True\n else:\n self.accelerator_config.split_batches = True\n self.accelerator_config.dispatch_batches = True\n" }, { "path": "trl/experimental/async_grpo/async_grpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\nimport queue\nimport textwrap\nimport time\nfrom collections import defaultdict\nfrom collections.abc import Callable, Iterator\nfrom dataclasses import dataclass\nfrom typing import Any, Protocol\n\nimport torch\nfrom accelerate.logging import get_logger\nfrom datasets import Dataset, IterableDataset\nfrom torch.distributed._tensor import DTensor\nfrom torch.utils.data import DataLoader\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedTokenizerBase, TrainerCallback\nfrom transformers.data.data_collator import DataCollatorMixin\n\nfrom trl.trainer.base_trainer import _BaseTrainer\nfrom trl.trainer.utils import pad, selective_log_softmax\n\nfrom .async_grpo_config import AsyncGRPOConfig\nfrom .async_rollout_worker import AsyncRolloutWorker\n\n\nlogger = get_logger(__name__)\n\n# A reward function is a callable that returns a list of floats (the rewards). The callable receives prompts,\n# completions, and additional arguments from the trainer (refer to the trainer's source for details). To ensure forward\n# compatibility, it should accept **kwargs.\nRewardFunc = Callable[..., list[float]]\n\n\nclass _SupportsReset(Protocol):\n def reset(self, **kwargs) -> str | None: ...\n\n\nEnvironmentFactory = Callable[[], _SupportsReset]\n\n\nclass RolloutWorkerProtocol(Protocol):\n rollout_buffer: queue.Queue\n\n def start(self) -> None: ...\n def stop(self) -> None: ...\n def pause(self) -> None: ...\n def resume(self) -> None: ...\n def send_weights(self, iterator: Iterator[tuple[str, torch.Tensor]]) -> None: ...\n def update_model_version(self, version: int) -> None: ...\n\n\nclass StepIntervalCallback(TrainerCallback):\n \"\"\"\n A callback that calls a function every N optimization steps.\n \"\"\"\n\n def __init__(self, fn, every_n_steps: int):\n self.fn = fn\n self.every_n_steps = every_n_steps\n\n def on_step_end(self, _args, state, _control, **_kwargs):\n if state.global_step % self.every_n_steps == 0:\n self.fn()\n\n\nclass RolloutQueueDataset(torch.utils.data.IterableDataset):\n def __init__(self, rollout_queue, model_version_fn, max_staleness=3, timeout=120.0):\n self.queue = rollout_queue\n self.model_version_fn = model_version_fn\n self.max_staleness = max_staleness\n self.timeout = timeout\n\n def __iter__(self):\n while True:\n t0 = time.time()\n qsize = self.queue.qsize()\n if qsize == 0:\n logger.info(\"queue empty, waiting for rollout samples...\")\n try:\n sample = self.queue.get(timeout=self.timeout)\n except queue.Empty:\n logger.warning(f\"Rollout queue empty for {self.timeout}s, stopping epoch\")\n return # StopIteration ends epoch\n queue_wait_time_s = time.time() - t0\n if queue_wait_time_s > 1.0:\n logger.info(f\"waited {queue_wait_time_s:.1f}s for sample (qsize={self.queue.qsize()})\")\n\n staleness = self.model_version_fn() - sample.model_version\n if staleness > self.max_staleness:\n logger.info(f\"dropping stale sample (staleness={staleness}, max={self.max_staleness})\")\n continue # drop stale, pull next\n\n yield {\n \"input_ids\": sample.input_ids,\n \"completion_mask\": sample.completion_mask,\n \"old_log_probs\": sample.old_log_probs,\n \"advantage\": sample.advantage,\n \"metrics\": {**sample.metrics, \"queue_wait_time_s\": queue_wait_time_s},\n }\n\n\nclass _EmptyIterableDataset(torch.utils.data.IterableDataset):\n \"\"\"Placeholder for non-rank-0 processes. Never actually iterated.\"\"\"\n\n def __iter__(self):\n return iter([])\n\n\n@dataclass\nclass DataCollatorForRollout(DataCollatorMixin):\n pad_token_id: int\n return_tensors: str = \"pt\"\n\n def torch_call(self, examples: list[dict[str, Any]]) -> dict[str, Any]:\n input_ids = [torch.tensor(example[\"input_ids\"], dtype=torch.long) for example in examples]\n attention_mask = [torch.ones(len(ids), dtype=torch.long) for ids in input_ids]\n completion_mask = [torch.tensor(example[\"completion_mask\"], dtype=torch.float32) for example in examples]\n old_log_probs = [torch.tensor(example[\"old_log_probs\"], dtype=torch.float32) for example in examples]\n advantages = torch.tensor([example[\"advantage\"] for example in examples], dtype=torch.float32)\n\n input_ids = pad(input_ids, padding_value=self.pad_token_id)\n attention_mask = pad(attention_mask, padding_value=0)\n completion_mask = pad(completion_mask, padding_value=0)\n old_log_probs = pad(old_log_probs, padding_value=0)\n\n # Total valid completion tokens across all samples in the full batch.\n # Repeated per sample so that DataLoaderDispatcher (dispatch_batches=True) slices correctly on dim=0\n global_n_tokens = completion_mask.sum()\n global_n_tokens_repeated = torch.full((len(examples),), global_n_tokens.item(), dtype=torch.float32)\n\n # Convert per-sample metrics dicts to a dict of 1D tensors so that Accelerate's\n # recursive broadcast (dispatch_batches=True) can handle them โ€” it traverses nested\n # dicts of tensors but chokes on plain Python floats.\n metrics_list = [example[\"metrics\"] for example in examples]\n metrics = (\n {\n key: torch.tensor([m.get(key, 0.0) for m in metrics_list], dtype=torch.float32)\n for key in metrics_list[0]\n }\n if metrics_list and metrics_list[0]\n else {}\n )\n\n return {\n \"input_ids\": input_ids,\n \"attention_mask\": attention_mask,\n \"completion_mask\": completion_mask,\n \"old_log_probs\": old_log_probs,\n \"advantages\": advantages,\n \"global_n_tokens\": global_n_tokens_repeated,\n \"metrics\": metrics,\n }\n\n\nclass AsyncGRPOTrainer(_BaseTrainer):\n \"\"\"\n Trainer for the Group Relative Policy Optimization (GRPO) method. This algorithm was initially proposed in the\n paper [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language\n Models](https://huggingface.co/papers/2402.03300). This trainer is the asynchronous version of GRPO, where\n generation is offloaded to an external vLLM server that runs asynchronously alongside training, decoupling rollout\n from the gradient update loop.\n\n Example:\n\n ```python\n from trl.experimental.async_grpo import AsyncGRPOTrainer\n from trl.rewards import accuracy_reward\n from datasets import load_dataset\n\n dataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\n trainer = AsyncGRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n )\n trainer.train()\n ```\n\n Args:\n model (`str`):\n Model to be trained. Must be a string, being the *model id* of a pretrained model hosted inside a model\n repo on huggingface.co, or a path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using [`~transformers.AutoModelForCausalLM.from_pretrained`]. The model name is also used to identify the\n model on the vLLM server used for generation.\n reward_funcs (`RewardFunc | list[RewardFunc]`):\n Reward functions to be used for computing the rewards. To compute the rewards, we call all the reward\n functions with the prompts and completions and sum the rewards. Can be either:\n\n - A single reward function: The function is provided with the prompts and the generated completions, plus\n any additional columns in the dataset. It should return a list of rewards. Reward functions can be either\n synchronous or asynchronous and can also return `None` when the reward is not applicable to those\n samples. This is useful for multi-task training where different reward functions apply to different types\n of samples. When a reward function returns `None` for a sample, that reward function is excluded from the\n reward calculation for that sample. For more details, see [Using a custom reward\n function](#using-a-custom-reward-function).\n - A list of reward functions, where each item is a reward function as described above. Rewards from all\n functions are summed.\n args ([`AsyncGRPOConfig`], *optional*):\n Configuration for this trainer. If `None`, a default configuration is used.\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n Dataset to use for training. It must include a column `\"prompt\"`. Any additional columns in the dataset are\n ignored. The format of the samples can be either:\n\n - [Standard](dataset_formats#standard): Each sample contains plain text.\n - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role\n and content).\n processing_class ([`~transformers.PreTrainedTokenizerBase`], *optional*):\n Processing class used to process the data. The padding side must be set to `\"left\"`. If `None`, the\n processing class is loaded from the model's name with [`~transformers.AutoTokenizer.from_pretrained`]. A\n padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token,\n `tokenizer.eos_token` will be used as the default.\n callbacks (list of [`~transformers.TrainerCallback`], *optional*):\n List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed\n in [here](https://huggingface.co/docs/transformers/main_classes/callback).\n\n If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`]\n method.\n optimizers (`tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None]`, *optional*, defaults to `(None, None)`):\n A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your\n model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`.\n tools (list of `Callable`, *optional*):\n A list of callable tool functions (sync or async) that the model can invoke during generation. Each tool\n should be a standard Python function with properly type-hinted arguments and return values, and a\n Google-style docstring describing its purpose, arguments, and return value. For more details, see:\n https://huggingface.co/docs/transformers/en/chat_extras#passing-tools. The model uses the function's name,\n type hints, and docstring to determine how to call it. Ensure that the model's chat template supports tool\n use and that it has been fine-tuned for tool calling.\n environment_factory (`EnvironmentFactory`, *optional*):\n A callable that creates and returns an environment instance. The environment class should define methods\n that can be invoked as tools during generation. Each method should comply with the same requirements as the\n `tools` described above. If `environment_factory` is provided, an instance of the environment is created\n for each generation in the batch, allowing for parallel and independent interactions. The environment must\n also implement a callable `reset` method that can be used to reset state between generations. The `reset`\n method should return either `None` or a string: when it returns a string, that string is appended to the\n last user message before generation. This feature is experimental and may change or be removed at any time\n without prior notice.\n \"\"\"\n\n _tag_names = [\"trl\", \"async-grpo\"]\n _name = \"AsyncGRPO\"\n _paper = {\n \"title\": \"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\",\n \"id\": \"2402.03300\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{shao2024deepseekmath,\n title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},\n author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},\n year = 2024,\n eprint = {arXiv:2402.03300},\n }\n \"\"\"),\n }\n\n def __init__(\n self,\n model: str,\n reward_funcs: RewardFunc | list[RewardFunc],\n args: AsyncGRPOConfig | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n processing_class: PreTrainedTokenizerBase | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None] = (None, None),\n tools: list[Callable] | None = None,\n environment_factory: EnvironmentFactory | None = None,\n rollout_worker: RolloutWorkerProtocol | None = None,\n ):\n self.args = args or AsyncGRPOConfig()\n\n # Training arguments\n self.epsilon_low = self.args.epsilon\n self.epsilon_high = self.args.epsilon_high\n self.temperature = self.args.temperature\n\n # Model\n model_name = model\n model = AutoModelForCausalLM.from_pretrained(model, device_map=None, dtype=torch.bfloat16)\n\n # Processing class\n if processing_class is None:\n processing_class = AutoTokenizer.from_pretrained(model_name)\n if processing_class.pad_token is None:\n processing_class.pad_token = processing_class.eos_token\n\n # Reward functions\n if not isinstance(reward_funcs, list):\n reward_funcs = [reward_funcs]\n\n # Initialize the Trainer\n super().__init__(\n model=model,\n args=self.args,\n train_dataset=train_dataset,\n processing_class=processing_class,\n callbacks=callbacks,\n optimizers=optimizers,\n compute_loss_func=\"non-None value to disable scaling\",\n )\n # Gradient accumulation requires scaled loss. Normally, loss scaling in the parent class depends on whether the\n # model accepts loss-related kwargs. Since we compute our own loss, this check is irrelevant. We set\n # self.model_accepts_loss_kwargs to False to enable scaling.\n self.model_accepts_loss_kwargs = False\n\n # Infer max_steps from dataset size when not explicitly set. This must happen after super().__init__()\n # so that self.accelerator.num_processes is available for the correct calculation.\n samples_per_step = (\n self.args.per_device_train_batch_size\n * self.args.gradient_accumulation_steps\n * self.accelerator.num_processes\n )\n if self.args.max_steps <= 0 and train_dataset is not None and hasattr(train_dataset, \"__len__\"):\n samples_per_epoch = len(train_dataset) * self.args.num_generations\n self.args.max_steps = int(self.args.num_train_epochs * samples_per_epoch / samples_per_step)\n\n # Infer max_inflight_tasks when not explicitly set. Generating more samples than the trainer can consume\n # before they become stale is wasteful. The useful upper bound is max_staleness * samples_per_step.\n if self.args.max_inflight_tasks < 0:\n self.args.max_inflight_tasks = self.args.max_staleness * samples_per_step\n logger.info(\n f\"max_inflight_tasks set to {self.args.max_inflight_tasks} \"\n f\"(max_staleness={self.args.max_staleness} ร— samples_per_step={samples_per_step})\"\n )\n\n # Initialize the metrics\n self._metrics = {\"train\": defaultdict(list), \"eval\": defaultdict(list)}\n self._train_tokens_start_time = None\n self.model_version = 0\n # Create worker and queue on rank 0\n if self.accelerator.is_main_process:\n if self.train_dataset is None:\n raise ValueError(\"train_dataset is required for AsyncGRPOTrainer\")\n\n if rollout_worker is not None:\n # Use the injected worker (e.g. a stub in tests). The queue is owned by the worker.\n self.rollout_worker = rollout_worker\n else:\n # Collect weight metadata once โ€” names/dtypes/shapes are fixed for the lifetime of training.\n # DTensor.shape returns the global shape without triggering any all-gather.\n weight_names, weight_dtype_names, weight_shapes = [], [], []\n for name, param in model.named_parameters():\n # DDP/FSDP1 wrapping, avoids vllm module not exist error\n name = name.removeprefix(\"module.\")\n weight_names.append(name)\n weight_dtype_names.append(str(param.dtype).split(\".\")[-1])\n weight_shapes.append(list(param.shape))\n self.rollout_worker = AsyncRolloutWorker(\n model_name=model_name,\n dataset=train_dataset,\n reward_funcs=reward_funcs,\n tools=tools,\n environment_factory=environment_factory,\n num_generations=self.args.num_generations,\n max_inflight_tasks=self.args.max_inflight_tasks,\n queue_maxsize=self.args.queue_maxsize,\n vllm_server_url=self.args.vllm_server_base_url,\n max_tokens=self.args.max_completion_length,\n temperature=self.args.temperature,\n request_timeout=self.args.request_timeout,\n server_timeout=self.args.vllm_server_timeout,\n chat_template_kwargs=self.args.chat_template_kwargs,\n max_tool_calling_iterations=self.args.max_tool_calling_iterations,\n log_completions=self.args.log_completions,\n num_completions_to_print=self.args.num_completions_to_print,\n weight_names=weight_names,\n weight_dtype_names=weight_dtype_names,\n weight_shapes=weight_shapes,\n )\n self.rollout_queue = self.rollout_worker.rollout_buffer\n else:\n self.rollout_queue = None\n self.rollout_worker = None\n\n # Add callbacks\n self.add_callback(StepIntervalCallback(self._sync_weight, self.args.weight_sync_steps))\n\n def get_train_dataloader(self) -> DataLoader:\n if self.accelerator.is_main_process:\n dataset = RolloutQueueDataset(\n rollout_queue=self.rollout_queue,\n model_version_fn=lambda: self.model_version,\n max_staleness=self.args.max_staleness,\n timeout=self.args.vllm_server_timeout,\n )\n else:\n dataset = _EmptyIterableDataset()\n\n return self.accelerator.prepare(\n DataLoader(\n dataset,\n batch_size=self.args.per_device_train_batch_size * self.accelerator.num_processes,\n collate_fn=DataCollatorForRollout(self.processing_class.pad_token_id),\n num_workers=0, # MUST be 0\n )\n # NOTE(@aminediro):\n # dispatch_batches = True for DataLoader whose underlying dataset is an IterableDataset\n # dataloader prepared by the Accelerator is only iterated through on the main process a\n )\n\n def _set_signature_columns_if_needed(self):\n # If `self.args.remove_unused_columns` is True, non-signature columns are removed.\n # By default, this method sets `self._signature_columns` to the model's expected inputs (usually, \"input_ids\"\n # and \"attention_mask\"). In AsyncGRPOTrainer, we need additional columns (\"completion_mask\", \"old_log_probs\",\n # \"advantages\", \"global_n_tokens\") to compute the loss, hence the override.\n if self._signature_columns is None:\n self._signature_columns = [\n \"input_ids\",\n \"attention_mask\",\n \"completion_mask\",\n \"old_log_probs\",\n \"advantages\",\n \"global_n_tokens\",\n \"metrics\",\n ]\n\n def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):\n input_ids = inputs[\"input_ids\"]\n attention_mask = inputs[\"attention_mask\"]\n completion_mask = inputs[\"completion_mask\"]\n old_log_probs = inputs[\"old_log_probs\"]\n advantages = inputs[\"advantages\"]\n\n # The collator pads to the global batch max length (across all ranks). After DataLoaderDispatcher slices and\n # sends rows to each rank, the local slice is still padded to that global max. Truncate to the longest real\n # sequence in this rank's slice so we don't run the forward pass over pure-padding columns.\n local_max_len = attention_mask.sum(dim=1).max()\n input_ids = input_ids[:, :local_max_len]\n attention_mask = attention_mask[:, :local_max_len]\n completion_mask = completion_mask[:, :local_max_len]\n old_log_probs = old_log_probs[:, :local_max_len]\n\n forward_start = time.time()\n outputs = model(input_ids=input_ids, attention_mask=attention_mask, use_cache=False)\n self._last_forward_time_s = time.time() - forward_start\n\n logits = outputs.logits[:, :-1, :]\n targets = input_ids[:, 1:]\n logits.div_(self.temperature)\n log_probs = selective_log_softmax(logits, targets)\n completion_mask = completion_mask[:, 1:]\n old_log_probs = old_log_probs[:, 1:]\n advantages = advantages.unsqueeze(1)\n log_ratio = log_probs - old_log_probs\n ratio = torch.exp(log_ratio)\n clipped = torch.clamp(ratio, 1 - self.epsilon_low, 1 + self.epsilon_high)\n per_token_loss = -torch.min(ratio * advantages, clipped * advantages)\n\n # DDP/FSDP averages gradients across ranks (world_size).\n # To get correct per-token normalization we scale by 1/tokens_per_rank\n # = world_size / global_n_tokens, so after DDP averaging the effective\n loss = (per_token_loss * completion_mask).sum()\n global_n_tokens = inputs[\"global_n_tokens\"][0]\n world_size = self.accelerator.num_processes\n tokens_per_rank = (global_n_tokens / world_size).clamp(min=1.0)\n loss = loss / tokens_per_rank.to(torch.float32)\n # For DAPO, we would scale like this instead:\n # loss = loss / max(per_token_loss.size(0), 1)\n loss = loss / self.current_gradient_accumulation_steps\n\n with torch.no_grad():\n valid_mask = completion_mask > 0\n local_count = valid_mask.sum().float()\n\n local_ratio_sum = (\n ratio[valid_mask].sum() if valid_mask.any() else torch.zeros((), device=completion_mask.device)\n )\n # Approx KL: http://joschu.net/blog/kl-approx.html\n local_kl_sum = (\n ((ratio[valid_mask] - 1) - log_ratio[valid_mask]).sum()\n if valid_mask.any()\n else torch.zeros((), device=completion_mask.device)\n )\n\n probs = torch.softmax(logits, dim=-1)\n log_p = torch.log_softmax(logits, dim=-1)\n entropy = -torch.sum(probs * log_p, dim=-1)\n local_entropy_sum = (\n entropy[valid_mask].sum() if valid_mask.any() else torch.zeros((), device=completion_mask.device)\n )\n\n clipped = (ratio < 1 - self.epsilon_low) | (ratio > 1 + self.epsilon_high)\n local_clip_sum = (\n clipped[valid_mask].float().sum()\n if valid_mask.any()\n else torch.zeros((), device=completion_mask.device)\n )\n\n # Batch all-reduce: [ratio_sum, kl_sum, entropy_sum, clip_sum, count]\n stats = torch.stack([local_ratio_sum, local_kl_sum, local_entropy_sum, local_clip_sum, local_count])\n stats = self.accelerator.reduce(stats, reduction=\"sum\")\n global_ratio_sum, global_kl_sum, global_entropy_sum, global_clip_sum, global_count = stats.unbind(0)\n self._metrics[\"train\"][\"ratio\"].append((global_ratio_sum / global_count).item())\n self._metrics[\"train\"][\"kl\"].append((global_kl_sum / global_count).item())\n self._metrics[\"train\"][\"entropy\"].append((global_entropy_sum / global_count).item())\n self._metrics[\"train\"][\"clip_ratio\"].append((global_clip_sum / global_count).item())\n\n # Logging metrics from the rollout worker (reward, reward_std, etc.).\n # inputs[\"metrics\"] is a dict of 1D tensors keyed by metric name.\n sample_metrics = inputs[\"metrics\"] # dict[str, Tensor(shape=[B_local])]\n keys = list(sample_metrics.keys())\n device = completion_mask.device\n n_samples = torch.tensor(completion_mask.shape[0], dtype=torch.float32, device=device)\n if keys:\n local_sums = torch.stack([sample_metrics[k].to(device).sum() for k in keys])\n stats = torch.cat([local_sums, n_samples.unsqueeze(0)])\n stats = self.accelerator.reduce(stats, reduction=\"sum\")\n global_sums, global_n_samples = stats[:-1], stats[-1]\n for k, global_sum in zip(keys, global_sums, strict=True):\n self._metrics[\"train\"][k].append((global_sum / global_n_samples).item())\n\n completion_length = completion_mask.sum(dim=1).float()\n length_stats = torch.stack([completion_length.sum(), n_samples])\n length_stats = self.accelerator.reduce(length_stats, reduction=\"sum\")\n self._metrics[\"train\"][\"completions/mean_length\"].append((length_stats[0] / length_stats[1]).item())\n\n # Training throughput: completion tokens consumed by this training step per second.\n now = time.time()\n if self._train_tokens_start_time is not None:\n train_elapsed = now - self._train_tokens_start_time\n if train_elapsed > 0:\n self._metrics[\"train\"][\"training_tok/s\"].append(global_n_tokens.item() / train_elapsed)\n self._train_tokens_start_time = now\n\n self._metrics[\"train\"][\"forward_time_s\"].append(self._last_forward_time_s)\n # NOTE: in dynamic mbs setup, we would need to agg across DP ranks.\n self._metrics[\"train\"][\"train_seq_len\"].append(float(local_max_len))\n return loss\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n mode = \"train\" if self.model.training else \"eval\"\n metrics = {key: sum(val) / len(val) for key, val in self._metrics[mode].items()} # average the metrics\n if mode == \"eval\":\n metrics = {f\"eval_{key}\": val for key, val in metrics.items()}\n logs = {**logs, **metrics}\n super().log(logs, start_time)\n self._metrics[mode].clear()\n\n def _streaming_iter(self):\n # Iterate parameters one at a time. For FSDP2 (DTensor), full_tensor() all-gathers just this parameter across\n # FSDP ranks, then frees it once the generator advances โ€” avoiding materializing the full model in memory.\n for name, param in self.model.named_parameters():\n name = name.removeprefix(\"module.\") # DDP/FSDP1 wrapping\n full = param.full_tensor() if isinstance(param, DTensor) else param.detach()\n yield name, full\n\n def _sync_weight(self):\n t0 = time.time()\n logger.info(\"Weight sync: pausing vLLM...\")\n if self.accelerator.is_main_process and self.rollout_worker:\n self.rollout_worker.pause()\n t_pause = time.time()\n logger.info(f\"Weight sync: pause took {t_pause - t0:.1f}s, waiting for all ranks...\")\n\n self.accelerator.wait_for_everyone()\n t_barrier = time.time()\n\n logger.info(f\"Weight sync: transferring weights... (barrier took {t_barrier - t_pause:.1f}s)\")\n if self.accelerator.is_main_process and self.rollout_worker:\n self.rollout_worker.send_weights(self._streaming_iter())\n else:\n # Non-rank-0 processes must still participate in full_tensor() collectives for FSDP2.\n for _ in self._streaming_iter():\n pass\n t_transfer = time.time()\n\n self.accelerator.wait_for_everyone()\n\n logger.info(f\"Weight sync: resuming vLLM... (transfer took {t_transfer - t_barrier:.1f}s)\")\n if self.accelerator.is_main_process and self.rollout_worker:\n self.rollout_worker.resume()\n self.model_version += 1\n self.rollout_worker.update_model_version(self.model_version)\n weight_sync_time_s = time.time() - t0\n self._metrics[\"train\"][\"weight_sync_time_s\"].append(weight_sync_time_s)\n logger.info(f\"Weight sync: done. Total {weight_sync_time_s:.1f}s\")\n\n def _inner_training_loop(self, *args, **kwargs):\n # Start the rollout worker here (not in __init__) so that checkpoint loading in Trainer.train()\n # has already restored the model weights. The sequence is: start worker thread โ†’ wait for NCCL\n # init โ†’ sync weights to vLLM โ†’ begin generation. This ensures vLLM always uses the current\n # policy before producing any samples (matters for resumed runs, harmless for fresh ones).\n self._sync_weight()\n if self.accelerator.is_main_process and self.rollout_worker:\n self.rollout_worker.start()\n try:\n return super()._inner_training_loop(*args, **kwargs)\n finally:\n if self.accelerator.is_main_process and self.rollout_worker:\n self.rollout_worker.stop()\n" }, { "path": "trl/experimental/async_grpo/async_rollout_worker.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport asyncio\nimport inspect\nimport queue\nimport threading\nimport time\nfrom collections.abc import Callable, Iterator\nfrom dataclasses import dataclass\nfrom typing import Any, TypeAlias\n\nimport aiohttp\nimport numpy as np\nimport requests\nfrom accelerate.logging import get_logger\nfrom datasets import Dataset\nfrom transformers import AutoTokenizer\n\nfrom trl.chat_template_utils import add_response_schema, get_training_chat_template, parse_response\nfrom trl.import_utils import is_vllm_available\nfrom trl.trainer.utils import print_prompt_completions_sample\n\n\nif is_vllm_available(min_version=\"0.17.1\"):\n from vllm.distributed.weight_transfer.nccl_engine import NCCLTrainerSendWeightsArgs, NCCLWeightTransferEngine\n from vllm.utils.network_utils import get_ip, get_open_port\n\n\nlogger = get_logger(__name__)\n\nMessages: TypeAlias = list[dict[str, str]]\n\n\n@dataclass(slots=True)\nclass RolloutGroup:\n \"\"\"Single GRPO group for one prompt with multiple completions.\"\"\"\n\n prompt: Messages\n prompt_ids: list[int]\n reward_kwargs: dict[str, list[Any]]\n completions: list[Messages]\n completions_ids: list[list[int]]\n completions_logprobs: list[list[float]]\n tool_mask: list[list[int]]\n tool_call_counts: list[int]\n tool_failure_counts: list[int]\n model_version: int\n queued_at: float = 0.0\n\n\n@dataclass(slots=True)\nclass RolloutSample:\n prompt: Messages\n completion: Messages\n input_ids: list[int]\n completion_mask: list[int]\n old_log_probs: list[float]\n advantage: float\n model_version: int\n metrics: dict[str, float] # logging metadata only, not used in loss computation\n\n\nclass AsyncRolloutWorker:\n \"\"\"\n Minimal asynchronous actor worker structure.\n\n Loop:\n generate groups -> score groups -> push samples -> repeat\n \"\"\"\n\n def __init__(\n self,\n model_name: str,\n dataset: Dataset,\n reward_funcs: list[Callable[..., list[float]]],\n tools: list[Callable] | None = None,\n environment_factory: Callable[[], object] | None = None,\n num_generations: int = 8,\n max_inflight_tasks: int = 128,\n queue_maxsize: int = 0,\n vllm_server_url: str = \"http://localhost:8000\",\n max_tokens: int = 32,\n temperature: float = 1.0,\n request_timeout: int = 120,\n server_timeout: float = 240.0,\n chat_template_kwargs: dict[str, Any] | None = None,\n max_tool_calling_iterations: int | None = None,\n log_completions: bool = False,\n num_completions_to_print: int = 3,\n weight_names: list[str] | None = None,\n weight_dtype_names: list[str] | None = None,\n weight_shapes: list[list[int]] | None = None,\n ):\n if not is_vllm_available(min_version=\"0.17.1\"):\n raise ImportError(\n \"vLLM >= 0.17.1 is required to use AsyncRolloutWorker. Install it with: pip install 'vllm>=0.17.1'\"\n )\n self.model_name = model_name\n self.max_tool_calling_iterations = max_tool_calling_iterations\n self.dataset = dataset\n self._dataset_iter = iter(dataset)\n self.rollout_buffer: queue.Queue[RolloutSample] = queue.Queue(maxsize=queue_maxsize)\n self._loop: asyncio.AbstractEventLoop | None = None\n self._stop_event: asyncio.Event | None = None\n self._weight_update_info = {\n \"names\": weight_names,\n \"dtype_names\": weight_dtype_names,\n \"shapes\": weight_shapes,\n \"packed\": True,\n \"is_checkpoint_format\": True,\n }\n\n self.reward_funcs = reward_funcs\n self.reward_func_names = [f.__name__ for f in reward_funcs]\n self.num_generations = num_generations\n self.max_inflight_tasks = max_inflight_tasks\n self.environments = None\n environment_methods = [[] for _ in range(self.max_inflight_tasks)]\n if environment_factory is not None:\n self.environments = [environment_factory() for _ in range(self.max_inflight_tasks)]\n for i, environment in enumerate(self.environments):\n has_reset = False\n for name, member in inspect.getmembers(environment, predicate=inspect.ismethod):\n if name == \"reset\":\n has_reset = True\n elif not name.startswith(\"_\"):\n environment_methods[i].append(member)\n if not has_reset:\n raise ValueError(\n \"Each environment instance returned by `environment_factory` must define `reset`.\"\n )\n\n base_tools = tools or []\n self._sync_tool_dicts = [{} for _ in range(self.max_inflight_tasks)]\n for i in range(self.max_inflight_tasks):\n for tool in base_tools + (environment_methods[i] if self.environments is not None else []):\n if inspect.iscoroutinefunction(tool):\n raise ValueError(\"Asynchronous tools are not supported in AsyncRolloutWorker yet.\")\n self._sync_tool_dicts[i][tool.__name__] = tool\n self.tools = base_tools + (environment_methods[0] if self.environments is not None else [])\n\n self.vllm_server_url = vllm_server_url.rstrip(\"/\")\n self.model_update_group = None\n self.max_tokens = max_tokens\n self.temperature = temperature\n self.request_timeout = request_timeout\n self.server_timeout = server_timeout\n self.chat_template_kwargs = chat_template_kwargs or {}\n self.log_completions = log_completions\n self.num_completions_to_print = num_completions_to_print\n self.tokenizer = AutoTokenizer.from_pretrained(model_name)\n self.tokenizer = add_response_schema(self.tokenizer)\n self.chat_template = get_training_chat_template(self.tokenizer)\n\n self._groups_to_score: asyncio.Queue[RolloutGroup | None] = asyncio.Queue(maxsize=16)\n self._total_completion_tokens = 0\n self._total_groups_scored = 0\n self._generation_start_time: float | None = None\n self.model_version = 0\n self.session = None\n\n # Wait for the vLLM server and initialize NCCL weight transfer.\n self._wait_for_server_ready_sync(timeout_s=self.server_timeout)\n self._init_weight_transfer()\n\n def _wait_for_server_ready_sync(self, timeout_s: float = 240.0, poll_interval_s: float = 2.0) -> None:\n \"\"\"Block until the vLLM server is healthy.\"\"\"\n logger.info(f\"Waiting for vLLM server at {self.vllm_server_url} ...\")\n start = time.time()\n while True:\n elapsed = time.time() - start\n try:\n response = requests.get(f\"{self.vllm_server_url}/health\", timeout=5)\n if response.status_code == 200:\n logger.info(f\"vLLM server ready after {elapsed:.1f}s\")\n return\n except (requests.ConnectionError, requests.Timeout, OSError):\n pass\n if elapsed >= timeout_s:\n raise TimeoutError(\n f\"Timed out after {timeout_s:.0f}s waiting for vLLM server at {self.vllm_server_url}. \"\n \"Make sure the vLLM server is running and reachable. If the server needs more time to load \"\n \"the model, increase `vllm_server_timeout` in your AsyncGRPOConfig.\"\n )\n if int(elapsed) % 10 < poll_interval_s:\n logger.info(f\"Still waiting for vLLM server... ({elapsed:.0f}s)\")\n time.sleep(poll_interval_s)\n\n def _init_weight_transfer(self) -> None:\n response = requests.get(f\"{self.vllm_server_url}/get_world_size\")\n inference_world_size = response.json()[\"world_size\"]\n world_size = inference_world_size + 1\n master_address = get_ip()\n master_port = get_open_port()\n\n init_info = {\n \"master_address\": master_address,\n \"master_port\": master_port,\n \"rank_offset\": 1,\n \"world_size\": world_size,\n }\n t_init = threading.Thread(\n target=requests.post,\n args=(f\"{self.vllm_server_url}/init_weight_transfer_engine\",),\n kwargs={\"json\": {\"init_info\": init_info}, \"timeout\": 120},\n )\n t_init.start()\n self.model_update_group = NCCLWeightTransferEngine.trainer_init(\n {\n \"master_address\": master_address,\n \"master_port\": master_port,\n \"world_size\": world_size,\n }\n )\n t_init.join()\n\n logger.info(\"Init weight sync group with vLLM\")\n\n def update_model_version(self, model_version: int):\n self.model_version = model_version\n\n async def _run_loops(self, stop_event: asyncio.Event) -> None:\n async with aiohttp.ClientSession() as session:\n self.session = session\n logger.info(\n f\"vllm worker started: num_generations={self.num_generations}, max_inflight_tasks={self.max_inflight_tasks}\"\n )\n await asyncio.gather(\n asyncio.create_task(self._generate_loop(stop_event=stop_event)),\n asyncio.create_task(self._score_loop(stop_event=stop_event)),\n )\n\n def start(self) -> None:\n thread = threading.Thread(target=self._run, daemon=True)\n thread.start()\n\n def stop(self) -> None:\n logger.info(\"Stopping worker thread...\")\n if self._loop and self._loop.is_running():\n try:\n self._loop.call_soon_threadsafe(self._stop_event.set)\n except RuntimeError:\n pass\n\n def _run(self) -> None:\n loop = asyncio.new_event_loop()\n asyncio.set_event_loop(loop)\n self._loop = loop\n self._stop_event = asyncio.Event()\n try:\n loop.run_until_complete(self._run_loops(stop_event=self._stop_event))\n except Exception as e:\n logger.exception(f\"Worker thread failed: {e}\")\n raise\n finally:\n loop.close()\n\n def pause(self) -> None:\n t0 = time.time()\n requests.post(f\"{self.vllm_server_url}/pause\", params={\"mode\": \"keep\"})\n logger.debug(f\"[weight_sync] pause HTTP took {time.time() - t0:.1f}s\")\n\n def resume(self) -> None:\n t0 = time.time()\n requests.post(f\"{self.vllm_server_url}/resume\")\n logger.debug(f\"[weight_sync] resume HTTP took {time.time() - t0:.1f}s\")\n\n def send_weights(self, iterator) -> None:\n if self.model_update_group is None:\n return\n t0 = time.time()\n t_update = threading.Thread(\n target=requests.post,\n args=(f\"{self.vllm_server_url}/update_weights\",),\n kwargs={\"json\": {\"update_info\": self._weight_update_info}, \"timeout\": 1800},\n )\n t_update.start()\n logger.debug(f\"[weight_sync] /update_weights POST sent ({time.time() - t0:.1f}s)\")\n t_nccl = time.time()\n NCCLWeightTransferEngine.trainer_send_weights(\n iterator=iterator,\n trainer_args=NCCLTrainerSendWeightsArgs(group=self.model_update_group, packed=True),\n )\n logger.debug(f\"[weight_sync] NCCL transfer took {time.time() - t_nccl:.1f}s\")\n t_join = time.time()\n t_update.join()\n logger.debug(\n f\"[weight_sync] /update_weights join took {time.time() - t_join:.1f}s (total send_weights: {time.time() - t0:.1f}s)\"\n )\n\n async def _generate_loop(self, stop_event: asyncio.Event) -> None:\n pending_groups: dict[int, RolloutGroup] = {}\n pending_completed: dict[int, int] = {}\n inflight_tasks: dict[asyncio.Task, tuple[int, int]] = {}\n free_slots = set(range(self.max_inflight_tasks))\n work_iter = self._repeat_iterator()\n\n self._generation_start_time = time.monotonic()\n try:\n while True:\n while free_slots and not stop_event.is_set():\n group_id, row = next(work_iter)\n if group_id not in pending_groups:\n prompt = row[\"prompt\"]\n prompt_ids = self.tokenizer.apply_chat_template(\n prompt,\n return_dict=False,\n add_generation_prompt=True,\n tools=self.tools,\n chat_template=self.chat_template,\n **self.chat_template_kwargs,\n )\n reward_kwargs = {\n key: [row[key]] * self.num_generations\n for key in row\n if key not in {\"prompt\", \"completion\", \"completion_ids\"}\n }\n pending_groups[group_id] = RolloutGroup(\n prompt=prompt,\n prompt_ids=prompt_ids,\n reward_kwargs=reward_kwargs,\n completions=[],\n completions_ids=[],\n completions_logprobs=[],\n tool_mask=[],\n tool_call_counts=[],\n tool_failure_counts=[],\n model_version=self.model_version,\n )\n pending_completed[group_id] = 0\n logger.debug(f\"Started group {group_id}; pending_groups={len(pending_groups)}\")\n\n slot = free_slots.pop()\n if self.environments is not None:\n # Current assumption: reset side effects matter, return value is ignored.\n self.environments[slot].reset(**row)\n\n logger.info(f\"[slot] assigned slot={slot} group={group_id} free_after={len(free_slots)}\")\n task = asyncio.create_task(\n self._generate_one(pending_groups[group_id].prompt, tool_dict=self._sync_tool_dicts[slot])\n )\n inflight_tasks[task] = (group_id, slot)\n\n if not inflight_tasks:\n if stop_event.is_set():\n return\n await asyncio.sleep(0.01)\n continue\n\n done, _ = await asyncio.wait(inflight_tasks, return_when=asyncio.FIRST_COMPLETED, timeout=0.1)\n if not done:\n if not free_slots:\n logger.debug(\n f\"[generate] all {self.max_inflight_tasks} slots busy, \"\n f\"pending_groups={len(pending_groups)}, waiting for completions...\"\n )\n continue\n\n for task in done:\n group_id, slot = inflight_tasks.pop(task)\n free_slots.add(slot)\n logger.debug(f\"[slot] freed slot={slot} group={group_id} free_after={len(free_slots)}\")\n if task.exception() is not None:\n raise task.exception()\n\n (\n completion,\n completion_ids,\n completion_logprobs,\n tool_mask,\n tool_call_count,\n tool_failure_count,\n ) = task.result()\n group = pending_groups[group_id]\n group.completions.append(completion)\n group.completions_ids.append(completion_ids)\n group.completions_logprobs.append(completion_logprobs)\n group.tool_mask.append(tool_mask)\n group.tool_call_counts.append(tool_call_count)\n group.tool_failure_counts.append(tool_failure_count)\n # TODO: move this in generation task, shouldn't matter but is correct\n self._total_completion_tokens += sum(tool_mask)\n pending_completed[group_id] += 1\n\n if pending_completed[group_id] == self.num_generations:\n group.queued_at = time.monotonic()\n while True:\n try:\n self._groups_to_score.put_nowait(group)\n break\n except asyncio.QueueFull:\n if stop_event.is_set():\n return\n await asyncio.sleep(0.1)\n logger.debug(f\"Group {group_id} complete; queued_for_scoring={self._groups_to_score.qsize()}\")\n del pending_groups[group_id]\n del pending_completed[group_id]\n finally:\n for task in inflight_tasks:\n task.cancel()\n if inflight_tasks:\n await asyncio.gather(*inflight_tasks, return_exceptions=True)\n # Use put_nowait: if the queue is full at shutdown, skip the sentinel โ€”\n # _score_loop will exit via stop_event check in its outer loop.\n try:\n self._groups_to_score.put_nowait(None)\n except asyncio.QueueFull:\n pass\n\n def _compute_rollout_metrics(self, samples: list[RolloutSample], scoring_time: float, wait_scoring: float) -> None:\n assert self._generation_start_time is not None, \"generation_start_time init in run()\"\n elapsed = time.monotonic() - self._generation_start_time\n generation_tok_per_sec = self._total_completion_tokens / elapsed if elapsed > 0 else 0.0\n\n scoring_time_ms = scoring_time * 1000\n wait_scoring_ms = wait_scoring * 1000\n\n for sample in samples:\n sample.metrics[\"generation_tok_per_s\"] = generation_tok_per_sec\n sample.metrics[\"scoring_time_ms\"] = scoring_time_ms\n sample.metrics[\"wait_scoring_ms\"] = wait_scoring_ms\n sample.metrics[\"buffer_qsize\"] = self.rollout_buffer.qsize()\n\n logger.info(\n f\"[inference] total_completion_tokens={self._total_completion_tokens}, \"\n f\"generation_tok/s={generation_tok_per_sec:.1f}, scoring_time={scoring_time_ms:.1f}ms, \"\n f\"wait_scoring={wait_scoring_ms:.1f}ms\"\n )\n\n async def _score_loop(self, stop_event: asyncio.Event) -> None:\n while not stop_event.is_set():\n t_wait = time.monotonic()\n try:\n group = await asyncio.wait_for(self._groups_to_score.get(), timeout=0.5)\n except asyncio.TimeoutError:\n continue\n if group is None:\n return\n score_queue_wait = time.monotonic() - t_wait\n\n wait_scoring = time.monotonic() - group.queued_at\n\n if score_queue_wait > 0.5:\n logger.info(f\"[score] waited {score_queue_wait:.1f}s for a group to score\")\n\n t0 = time.monotonic()\n samples = await self._score_group(group)\n scoring_time = time.monotonic() - t0\n logger.info(\n f\"[score] scored {len(samples)} samples in {scoring_time:.2f}s, \"\n f\"buffer_qsize={self.rollout_buffer.qsize()}\"\n )\n\n self._compute_rollout_metrics(samples, scoring_time, wait_scoring)\n\n if self.log_completions and samples:\n print_prompt_completions_sample(\n prompts=[s.prompt for s in samples],\n completions=[s.completion for s in samples],\n rewards={\"reward\": [s.metrics[\"reward\"] for s in samples]},\n advantages=[s.advantage for s in samples],\n step=self._total_groups_scored,\n num_samples=self.num_completions_to_print,\n )\n self._total_groups_scored += 1\n\n for sample in samples:\n while True:\n try:\n self.rollout_buffer.put_nowait(sample)\n break\n except queue.Full:\n if stop_event.is_set():\n return\n # Wait for trainer to consume loop\n logger.info(\n f\"[score] rollout buffer full (maxsize={self.rollout_buffer.maxsize}), waiting for trainer to consume...\"\n )\n await asyncio.sleep(0.1)\n\n logger.debug(\n f\"Scored group with {len(samples)} samples; rollout_buffer_qsize={self.rollout_buffer.qsize()}\"\n )\n\n def _repeat_iterator(self) -> Iterator[tuple[int, dict[str, Any]]]:\n group_id = 0\n while True:\n try:\n row = next(self._dataset_iter)\n except StopIteration:\n self._dataset_iter = iter(self.dataset)\n row = next(self._dataset_iter)\n for _ in range(self.num_generations):\n yield group_id, row\n group_id += 1\n\n async def _generate_one(\n self, prompt: Messages, tool_dict: dict[str, Callable]\n ) -> tuple[list[dict[str, str]], list[int], list[float], list[int], int, int]:\n completion, completion_ids, completion_logprobs, tool_mask = [], [], [], []\n tool_call_count = 0\n tool_failure_count = 0\n iteration_num = 0\n max_iterations = self.max_tool_calling_iterations\n prompt_ids = self.tokenizer.apply_chat_template(\n prompt,\n return_dict=False,\n add_generation_prompt=True,\n tools=self.tools,\n chat_template=self.chat_template,\n **self.chat_template_kwargs,\n )\n while True:\n turn_ids, turn_logprobs = await self._generate_one_turn(prompt_ids)\n assistant_message = parse_response(self.tokenizer, turn_ids)\n completion.append(assistant_message)\n completion_ids.extend(turn_ids)\n completion_logprobs.extend(turn_logprobs)\n tool_mask.extend([1] * len(turn_ids))\n tool_calls = assistant_message.get(\"tool_calls\")\n if tool_calls is None or (max_iterations is not None and iteration_num >= max_iterations):\n return completion, completion_ids, completion_logprobs, tool_mask, tool_call_count, tool_failure_count\n\n tool_messages, n_calls, n_failures = self._execute_tool_calls(tool_calls, tool_dict)\n tool_call_count += n_calls\n tool_failure_count += n_failures\n completion.extend(tool_messages)\n tool_suffix_ids = self._build_messages_suffix_ids(tool_messages)\n completion_ids.extend(tool_suffix_ids)\n completion_logprobs.extend([0.0] * len(tool_suffix_ids))\n tool_mask.extend([0] * len(tool_suffix_ids))\n prompt_ids = prompt_ids + turn_ids + tool_suffix_ids\n iteration_num += 1\n\n def _build_messages_suffix_ids(self, messages: list[dict[str, Any]]) -> list[int]:\n template_messages = [\n {\"role\": \"user\", \"content\": \"\"},\n {\"role\": \"assistant\", \"content\": \"\"},\n ]\n prefix_ids = self.tokenizer.apply_chat_template(\n template_messages,\n return_dict=False,\n tools=self.tools,\n chat_template=self.chat_template,\n **self.chat_template_kwargs,\n )\n prefix_and_messages_ids = self.tokenizer.apply_chat_template(\n template_messages + messages,\n return_dict=False,\n chat_template=self.chat_template,\n add_generation_prompt=True,\n tools=self.tools,\n **self.chat_template_kwargs,\n )\n prefix_len = len(prefix_ids)\n if prefix_and_messages_ids[:prefix_len] != prefix_ids:\n raise ValueError(\"Failed to construct message suffix in token space.\")\n return prefix_and_messages_ids[prefix_len:]\n\n def _execute_tool_calls(\n self, tool_calls: list[dict[str, Any]], tool_dict: dict[str, Callable]\n ) -> tuple[list[dict[str, str]], int, int]:\n tool_messages = []\n n_calls = 0\n n_failures = 0\n for tool_call in tool_calls:\n n_calls += 1\n function = tool_call[\"function\"]\n name = function[\"name\"]\n try:\n arguments = function.get(\"arguments\", {})\n result = tool_dict[name](**arguments)\n except Exception as error:\n n_failures += 1\n result = {\"error\": str(error)}\n tool_messages.append({\"role\": \"tool\", \"name\": name, \"content\": str(result)})\n return tool_messages, n_calls, n_failures\n\n async def _generate_one_turn(self, prompt_ids: list[int]) -> tuple[list[int], list[float]]:\n payload = {\n \"model\": self.model_name,\n \"prompt\": prompt_ids,\n \"max_tokens\": self.max_tokens,\n \"temperature\": self.temperature,\n \"n\": 1,\n \"return_token_ids\": True,\n \"logprobs\": 0,\n }\n while True:\n try:\n output = await self._post(\"/v1/completions\", payload, self.request_timeout)\n break\n except (aiohttp.ServerDisconnectedError, aiohttp.ClientConnectionError, aiohttp.ClientResponseError):\n # vLLM drops connections or returns 503 during weight sync (/pause). Wait briefly and retry.\n logger.debug(\"Server unavailable (likely weight sync pause), retrying...\")\n await asyncio.sleep(1.0)\n choice = output[\"choices\"][0]\n completion_ids = choice[\"token_ids\"]\n completion_logprobs = choice[\"logprobs\"][\"token_logprobs\"]\n return completion_ids, completion_logprobs\n\n async def _score_group(self, group: RolloutGroup) -> list[RolloutSample]:\n kwargs = dict(\n completions=group.completions,\n prompt=group.prompt,\n prompts=[group.prompt] * len(group.completions),\n completion_ids=group.completions_ids,\n **group.reward_kwargs,\n )\n all_rewards = await asyncio.gather(\n *[\n reward_func(**kwargs)\n if inspect.iscoroutinefunction(reward_func)\n else asyncio.to_thread(reward_func, **kwargs)\n for reward_func in self.reward_funcs\n ]\n )\n\n # Sum rewards across all reward functions. Reward functions may return None for individual\n # samples (e.g. accuracy_reward when the gold solution is unparseable). Convert None โ†’ nan\n # and use nansum so that a None from one function doesn't affect the others, matching TRL.\n all_rewards = [[r if r is not None else float(\"nan\") for r in row] for row in all_rewards]\n rewards = np.nansum(np.array(all_rewards, dtype=float), axis=0)\n advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-8)\n reward_mean = float(rewards.mean())\n reward_std = float(rewards.std())\n logger.info(f\"Rollout metrics: reward_mean={reward_mean:.4f}, reward_std={reward_std:.4f}\")\n\n # tools/call_frequency: mean calls per completion (matches TRL's total_calls / num_completions)\n # tools/failure_frequency: per-completion failure rate; averaged across samples in compute_loss\n # (TRL uses total_failures / total_calls, ours weights equally per completion โ€” close enough)\n total_calls = sum(group.tool_call_counts)\n tool_metrics = (\n [\n {\n \"tools/call_frequency\": float(n_calls),\n \"tools/failure_frequency\": (n_failures / n_calls) if n_calls > 0 else 0.0,\n }\n for n_calls, n_failures in zip(group.tool_call_counts, group.tool_failure_counts, strict=True)\n ]\n if total_calls > 0\n else [{}] * len(group.completions)\n )\n\n per_func_rewards = np.array(all_rewards, dtype=float) # shape (num_funcs, num_completions)\n\n return [\n RolloutSample(\n prompt=group.prompt,\n completion=completion,\n input_ids=group.prompt_ids + completion_ids,\n completion_mask=[0] * len(group.prompt_ids) + tool_mask,\n old_log_probs=[0.0] * len(group.prompt_ids) + logprobs,\n advantage=advantage,\n model_version=group.model_version,\n metrics={\n \"reward\": float(reward),\n \"reward_std\": reward_std,\n **{\n f\"rewards/{name}\": float(func_reward)\n for name, func_reward in zip(self.reward_func_names, per_func_rewards[:, i], strict=True)\n },\n **tm,\n },\n )\n for i, (completion, completion_ids, logprobs, tool_mask, advantage, reward, tm) in enumerate(\n zip(\n group.completions,\n group.completions_ids,\n group.completions_logprobs,\n group.tool_mask,\n advantages,\n rewards,\n tool_metrics,\n strict=True,\n )\n )\n ]\n\n async def _post(self, path: str, payload: dict, timeout: float, max_retries: int = 3) -> dict:\n client_timeout = aiohttp.ClientTimeout(total=timeout)\n for attempt in range(max_retries):\n try:\n async with self.session.post(\n f\"{self.vllm_server_url}{path}\", json=payload, timeout=client_timeout\n ) as response:\n response.raise_for_status()\n content = await response.json()\n return content if content else {}\n except (TimeoutError, asyncio.TimeoutError):\n if attempt < max_retries - 1:\n logger.warning(f\"POST {path} timed out (attempt {attempt + 1}/{max_retries}), retrying...\")\n await asyncio.sleep(1)\n else:\n raise\n" }, { "path": "trl/experimental/bco/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .bco_config import BCOConfig\nfrom .bco_trainer import BCOTrainer\n" }, { "path": "trl/experimental/bco/bco_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom ...trainer.base_config import _BaseConfig\n\n\n@dataclass\nclass BCOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`experimental.bco.BCOTrainer`].\n\n This class includes only the parameters that are specific to BCO training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n max_length (`int` or `None`, *optional*, defaults to `1024`):\n Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want\n to use the default data collator.\n max_completion_length (`int`, *optional*):\n Maximum length of the completion. This argument is required if you want to use the default data collator\n and your model is an encoder-decoder.\n beta (`float`, *optional*, defaults to `0.1`):\n Parameter controlling the deviation from the reference model. Higher ฮฒ means less deviation from the\n reference model.\n truncation_mode (`str`, *optional*, defaults to `\"keep_end\"`):\n Truncation mode to use when the prompt is too long. Possible values are `\"keep_end\"` or `\"keep_start\"`.\n This argument is required if you want to use the default data collator.\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model and reference model.\n generate_during_eval (`bool`, *optional*, defaults to `False`):\n If `True`, generates and logs completions from both the model and the reference model to W&B or Comet\n during evaluation.\n is_encoder_decoder (`bool`, *optional*):\n When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,\n you need to specify if the model returned by the callable is an encoder-decoder model.\n precompute_ref_log_probs (`bool`, *optional*, defaults to `False`):\n Whether to precompute reference model log probabilities for training and evaluation datasets. This is\n useful when training without the reference model to reduce the total GPU memory needed.\n model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model and\n reference model from strings.\n dataset_num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n prompt_sample_size (`int`, *optional*, defaults to `1024`):\n Number of prompts that are fed to density ratio classifier.\n min_density_ratio (`float`, *optional*, defaults to `0.5`):\n Minimum value of the density ratio. The estimated density ratio is clamped to this value.\n max_density_ratio (`float`, *optional*, defaults to `10.0`):\n Maximum value of the density ratio. The estimated density ratio is clamped to this value.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `5e-7` instead of `5e-5`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=5e-7,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n max_length: int | None = field(\n default=1024,\n metadata={\n \"help\": \"Maximum length of the sequences (prompt + completion) in the batch. \"\n \"This argument is required if you want to use the default data collator.\"\n },\n )\n max_completion_length: int | None = field(\n default=None,\n metadata={\n \"help\": \"Maximum length of the completion. This argument is required if you want to use the \"\n \"default data collator and your model is an encoder-decoder.\"\n },\n )\n beta: float = field(\n default=0.1,\n metadata={\n \"help\": \"Parameter controlling the deviation from the reference model. \"\n \"Higher ฮฒ means less deviation from the reference model.\"\n },\n )\n truncation_mode: str = field(\n default=\"keep_end\",\n metadata={\n \"help\": \"Truncation mode to use when the prompt is too long. Possible values are \"\n \"`keep_end` or `keep_start`. This argument is required if you want to use the \"\n \"default data collator.\"\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropout in the model and reference model.\"},\n )\n generate_during_eval: bool = field(\n default=False,\n metadata={\n \"help\": \"If `True`, generates and logs completions from both the model and the reference model \"\n \"to W&B during evaluation.\"\n },\n )\n is_encoder_decoder: bool | None = field(\n default=None,\n metadata={\n \"help\": \"When using the `model_init` argument (callable) to instantiate the model instead of the \"\n \"`model` argument, you need to specify if the model returned by the callable is an \"\n \"encoder-decoder model.\"\n },\n )\n precompute_ref_log_probs: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to precompute reference model log probabilities for training and evaluation datasets. \"\n \"This is useful when training without the reference model to reduce the total GPU memory \"\n \"needed.\"\n },\n )\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the \"\n \"model from a string.\"\n },\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes to use for processing the dataset.\"},\n )\n prompt_sample_size: int = field(\n default=1024,\n metadata={\"help\": \"Number of prompts that are fed to density ratio classifier.\"},\n )\n min_density_ratio: float = field(\n default=0.5,\n metadata={\"help\": \"Minimum value of the density ratio. The estimated density ratio is clamped to this value.\"},\n )\n max_density_ratio: float = field(\n default=10.0,\n metadata={\"help\": \"Maximum value of the density ratio. The estimated density ratio is clamped to this value.\"},\n )\n" }, { "path": "trl/experimental/bco/bco_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport dataclasses\nimport inspect\nimport json\nimport os\nimport random\nimport textwrap\nfrom collections import defaultdict\nfrom collections.abc import Callable\nfrom contextlib import contextmanager, nullcontext\nfrom dataclasses import dataclass\nfrom operator import itemgetter\nfrom pathlib import Path\nfrom typing import TYPE_CHECKING, Any, Literal, Optional\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport transformers\nfrom accelerate import Accelerator, PartialState, logging\nfrom accelerate.utils import tqdm\nfrom datasets import Dataset\nfrom packaging.version import Version\nfrom torch import autocast\nfrom torch.utils.data import DataLoader, SequentialSampler\nfrom transformers import (\n AutoModelForCausalLM,\n BaseImageProcessor,\n DataCollator,\n FeatureExtractionMixin,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n TrainingArguments,\n is_comet_available,\n is_sklearn_available,\n is_wandb_available,\n)\nfrom transformers.trainer_utils import EvalLoopOutput, has_length\nfrom transformers.utils import is_peft_available\n\nfrom ...data_utils import maybe_apply_chat_template, maybe_extract_prompt, maybe_unpair_preference_dataset\nfrom ...import_utils import is_joblib_available\nfrom ...models.utils import prepare_deepspeed\nfrom ...trainer.base_trainer import _BaseTrainer\nfrom ...trainer.utils import disable_dropout_in_model, log_table_to_comet_experiment, selective_log_softmax\nfrom ..utils import DPODataCollatorWithPadding, create_reference_model, pad_to_length, peft_module_casting_to_bf16\nfrom .bco_config import BCOConfig\n\n\nif is_peft_available():\n from peft import PeftModel, get_peft_model, prepare_model_for_kbit_training\n\nif is_wandb_available():\n import wandb\n\nif is_sklearn_available():\n from sklearn.linear_model import LogisticRegression\n\nif is_joblib_available():\n import joblib\n\nif TYPE_CHECKING:\n from transformers import PreTrainedTokenizer\n\nlogger = logging.get_logger(__name__)\n\nRUNNING_NAME = \"running.json\"\nCLF_NAME = \"clf.pkl\"\n\n\n@torch.no_grad()\ndef get_global_statistics(\n accelerator, xs: torch.Tensor, mask=None, device=\"cpu\"\n) -> tuple[torch.Tensor, torch.Tensor, int]:\n \"\"\"\n Computes element-wise mean and variance of the tensor across processes. Reference:\n https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f2b71b16919addede0341d2bef70825d/utils.py#L57C1-L73C75\n \"\"\"\n xs = xs.to(accelerator.device)\n sum_and_count = torch.tensor([xs.sum(), (xs.numel() if mask is None else mask.sum())], device=xs.device)\n sum_and_count = accelerator.reduce(sum_and_count)\n global_sum, count = sum_and_count\n global_mean = global_sum / count\n\n sum_var = torch.sum(((xs - global_mean) ** 2).mul(1 if mask is None else mask))\n sum_var = accelerator.reduce(sum_var)\n global_var = sum_var / count\n\n return global_mean.to(device), global_var.to(device), count.item()\n\n\n@dataclass\nclass RunningMoments:\n \"\"\"\n Calculates the running mean and standard deviation of a data stream. Reference:\n https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f2b71b16919addede0341d2bef70825d/utils.py#L75\n \"\"\"\n\n accelerator: Accelerator\n mean: float = 0\n std: float = 1\n var: float = 1\n count: float = 1e-24\n\n @torch.no_grad()\n def update(self, xs: torch.Tensor) -> tuple[float, float]:\n \"\"\"\n Updates running moments from batch's moments computed across ranks\n \"\"\"\n if self.accelerator.use_distributed:\n xs_mean, xs_var, xs_count = get_global_statistics(self.accelerator, xs)\n else:\n xs_count = xs.numel()\n xs_var, xs_mean = torch.var_mean(xs, unbiased=False)\n xs_mean, xs_var = xs_mean.float(), xs_var.float()\n\n delta = xs_mean - self.mean\n tot_count = self.count + xs_count\n\n new_sum = xs_var * xs_count\n # correct old_sum deviation accounting for the new mean\n old_sum = self.var * self.count + delta**2 * self.count * xs_count / tot_count\n tot_sum = old_sum + new_sum\n\n self.mean += (delta * xs_count / tot_count).item()\n new_var = tot_sum / tot_count\n self.std = (new_var * tot_count / (tot_count - 1)).float().sqrt().item()\n self.var = new_var.item()\n self.count = tot_count\n\n return xs_mean.item(), (xs_var * xs_count / (xs_count - 1)).float().sqrt().item()\n\n def save_to_json(self, json_path: str):\n \"\"\"Save the content of this instance in JSON format inside `json_path`.\"\"\"\n # save everything except accelerator\n if self.accelerator.is_main_process:\n save_dict = dataclasses.asdict(self, dict_factory=lambda x: {k: v for (k, v) in x if k != \"accelerator\"})\n json_string = json.dumps(save_dict, indent=2, sort_keys=True) + \"\\n\"\n with open(json_path, \"w\", encoding=\"utf-8\") as f:\n f.write(json_string)\n\n @classmethod\n def load_from_json(cls, accelerator: Accelerator, json_path: str):\n \"\"\"Create an instance from the content of `json_path`.\"\"\"\n # load everything except accelerator\n with open(json_path, encoding=\"utf-8\") as f:\n text = f.read()\n return cls(accelerator=accelerator, **json.loads(text))\n\n\ndef _tokenize(\n batch: dict[str, list[Any]],\n tokenizer: \"PreTrainedTokenizer\",\n embedding_tokenizer: Optional[\"PreTrainedTokenizer\"] = None,\n) -> dict[str, list[Any]]:\n \"\"\"Tokenize a batch from a BCO specific dataset.\"\"\"\n prompt_tokenized = tokenizer(batch[\"prompt\"], add_special_tokens=False)\n prompt_input_ids = prompt_tokenized[\"input_ids\"]\n prompt_attention_mask = prompt_tokenized[\"attention_mask\"]\n prompt_and_completion = [\n prompt + completion for prompt, completion in zip(batch[\"prompt\"], batch[\"completion\"], strict=True)\n ]\n full_tokenized = tokenizer(prompt_and_completion, add_special_tokens=False)\n full_input_ids = full_tokenized[\"input_ids\"]\n full_attention_mask = full_tokenized[\"attention_mask\"]\n\n answer_input_ids = [f[len(p) :] for f, p in zip(full_input_ids, prompt_input_ids, strict=True)]\n answer_attention_mask = [f[len(p) :] for f, p in zip(full_attention_mask, prompt_attention_mask, strict=True)]\n\n # Concat tokens to form `enc(a) + enc(a + b)[len(enc(a)):]`\n full_concat_input_ids = [np.concatenate([p, a]) for p, a in zip(prompt_input_ids, answer_input_ids, strict=True)]\n # Prepare input tokens for token by token comparison\n full_input_ids = [np.array(f) for f in full_input_ids]\n for full, concat in zip(full_input_ids, full_concat_input_ids, strict=True):\n if len(full) != len(concat):\n raise ValueError(\n \"The elements in 'full_input_ids' and 'full_concat_input_ids' must have the same pairwise length.\"\n )\n\n # On some tokenizers, like Llama-2 tokenizer, there are occasions where tokens\n # can be merged together when tokenizing prompt+answer. This could result\n # on the last token from the prompt being different when tokenized on its own\n # vs when done as prompt+answer.\n response_token_ids_start_idx = [len(p) for p in prompt_input_ids]\n\n # If tokenized prompt is different than both prompt+answer, then it means the\n # last token has changed due to merging.\n for idx, (p, f, r) in enumerate(zip(prompt_input_ids, full_input_ids, response_token_ids_start_idx, strict=True)):\n if not np.array_equal(p, f[:r]):\n response_token_ids_start_idx[idx] -= 1\n\n prompt_input_ids = [f[:r] for f, r in zip(full_input_ids, response_token_ids_start_idx, strict=True)]\n prompt_attention_mask = [f[:r] for f, r in zip(full_attention_mask, response_token_ids_start_idx, strict=True)]\n\n for p, m in zip(prompt_input_ids, prompt_attention_mask, strict=True):\n if len(p) != len(m):\n raise ValueError(\"Prompt input ids and attention mask should have the same length.\")\n\n answer_input_ids = [f[r:] for f, r in zip(full_input_ids, response_token_ids_start_idx, strict=True)]\n answer_attention_mask = [f[r:] for f, r in zip(full_attention_mask, response_token_ids_start_idx, strict=True)]\n\n output = dict(\n prompt_input_ids=prompt_input_ids,\n prompt_attention_mask=prompt_attention_mask,\n answer_input_ids=answer_input_ids,\n answer_attention_mask=answer_attention_mask,\n )\n\n if embedding_tokenizer is not None:\n embedding_tokenized = embedding_tokenizer(batch[\"prompt\"], add_special_tokens=False)\n\n output.update(\n {\n \"embedding_input_ids\": embedding_tokenized[\"input_ids\"],\n \"embedding_attention_mask\": embedding_tokenized[\"attention_mask\"],\n }\n )\n\n return output\n\n\ndef _process_tokens(example: dict[str, Any], model: \"PreTrainedModel\" = None, **kwargs) -> dict:\n \"\"\"Process tokens of a BCO specific dataset.\n\n At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +\n completion responses is/are too long. First we truncate the prompt; if we're still too long, we truncate the\n completion.\n\n We also create the labels for the completion responses, which are of length equal to the sum of the length of the\n prompt and the completion response, with `-100` for the prompt tokens.\n \"\"\"\n prompt = example[\"prompt\"]\n completion = example[\"completion\"]\n\n batch = {\n f\"{kwargs['prefix']}prompt\": prompt,\n f\"{kwargs['prefix']}completion\": completion,\n f\"{kwargs['prefix']}label\": example[\"label\"],\n }\n\n if not kwargs[\"is_encoder_decoder\"]:\n # Check issues below for more details\n # 1. https://github.com/huggingface/trl/issues/907\n # 2. https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257\n # 3. https://github.com/LianjiaTech/BELLE/issues/337\n\n if not isinstance(prompt, str):\n raise ValueError(f\"prompt should be an str but got {type(prompt)}\")\n\n if not isinstance(completion, str):\n raise ValueError(f\"completion should be an str but got {type(completion)}\")\n\n # keys of format prompt_* refers to just the prompt and answer_* refers to just the answer\n all_tokens = {\n \"prompt_input_ids\": example[\"prompt_input_ids\"],\n \"prompt_attention_mask\": example[\"prompt_attention_mask\"],\n \"answer_input_ids\": example[\"answer_input_ids\"],\n \"answer_attention_mask\": example[\"answer_attention_mask\"],\n }\n\n # calculate max length by checking if BOS/EOS is already there\n max_length = kwargs[\"max_length\"]\n bos_token_id = kwargs[\"tokenizer\"].bos_token_id\n eos_token_id = kwargs[\"tokenizer\"].eos_token_id\n if bos_token_id != all_tokens[\"prompt_input_ids\"][0]:\n max_length -= 1\n if eos_token_id != all_tokens[\"answer_input_ids\"][-1]:\n max_length -= 1\n\n # if combined sequence is too long (> max_length - 1 for BOS token - 1 for EOS), truncate the response\n if len(all_tokens[\"prompt_input_ids\"]) + len(all_tokens[\"answer_input_ids\"]) > max_length:\n for k in [\"answer_input_ids\", \"answer_attention_mask\"]:\n all_tokens[k] = all_tokens[k][: max_length - len(all_tokens[\"prompt_input_ids\"])]\n\n # all input_ids and attention mask as is. We then check if we need to add BOS/EOS tokens\n batch[f\"{kwargs['prefix']}prompt_input_ids\"] = all_tokens[\"prompt_input_ids\"]\n batch[f\"{kwargs['prefix']}prompt_attention_mask\"] = all_tokens[\"prompt_attention_mask\"]\n batch[f\"{kwargs['prefix']}completion_input_ids\"] = (\n all_tokens[\"prompt_input_ids\"] + all_tokens[\"answer_input_ids\"]\n )\n batch[f\"{kwargs['prefix']}completion_attention_mask\"] = (\n all_tokens[\"prompt_attention_mask\"] + all_tokens[\"answer_attention_mask\"]\n )\n\n # add BOS, which affects both prompt and the full completion\n if bos_token_id is not None:\n if len(all_tokens[\"prompt_input_ids\"]) == 0 or bos_token_id != all_tokens[\"prompt_input_ids\"][0]:\n batch[f\"{kwargs['prefix']}prompt_input_ids\"] = [bos_token_id] + batch[\n f\"{kwargs['prefix']}prompt_input_ids\"\n ]\n batch[f\"{kwargs['prefix']}prompt_attention_mask\"] = [1] + batch[\n f\"{kwargs['prefix']}prompt_attention_mask\"\n ]\n batch[f\"{kwargs['prefix']}completion_input_ids\"] = [bos_token_id] + batch[\n f\"{kwargs['prefix']}completion_input_ids\"\n ]\n batch[f\"{kwargs['prefix']}completion_attention_mask\"] = [1] + batch[\n f\"{kwargs['prefix']}completion_attention_mask\"\n ]\n # add EOS, which affects only the full completion\n if len(all_tokens[\"answer_input_ids\"]) == 0 or eos_token_id != all_tokens[\"answer_input_ids\"][-1]:\n batch[f\"{kwargs['prefix']}completion_input_ids\"] = batch[f\"{kwargs['prefix']}completion_input_ids\"] + [\n eos_token_id\n ]\n batch[f\"{kwargs['prefix']}completion_attention_mask\"] = batch[\n f\"{kwargs['prefix']}completion_attention_mask\"\n ] + [1]\n\n batch[f\"{kwargs['prefix']}completion_labels\"] = batch[f\"{kwargs['prefix']}completion_input_ids\"][:]\n batch[f\"{kwargs['prefix']}completion_labels\"][: len(batch[f\"{kwargs['prefix']}prompt_input_ids\"])] = [\n -100\n ] * len(batch[f\"{kwargs['prefix']}prompt_input_ids\"])\n else:\n completion_tokens = kwargs[\"tokenizer\"](\n completion, truncation=True, max_length=kwargs[\"max_completion_length\"], add_special_tokens=True\n )\n prompt_tokens = kwargs[\"tokenizer\"](prompt, add_special_tokens=True)\n\n batch[f\"{kwargs['prefix']}prompt_input_ids\"] = prompt_tokens[\"input_ids\"]\n batch[f\"{kwargs['prefix']}prompt_attention_mask\"] = prompt_tokens[\"attention_mask\"]\n\n batch[f\"{kwargs['prefix']}completion_labels\"] = completion_tokens[\"input_ids\"]\n batch[f\"{kwargs['prefix']}completion_attention_mask\"] = completion_tokens[\"attention_mask\"]\n if model is not None and hasattr(model, \"prepare_decoder_input_ids_from_labels\"):\n batch[f\"{kwargs['prefix']}completion_decoder_input_ids\"] = model.prepare_decoder_input_ids_from_labels(\n labels=torch.tensor(batch[\"completion_labels\"])\n )\n\n return batch\n\n\nclass BCOTrainer(_BaseTrainer):\n r\"\"\"\n Initialize BCOTrainer from [BCO](https://huggingface.co/papers/2404.04656) paper.\n\n Args:\n model ([`~transformers.PreTrainedModel`]):\n The model to train, preferably an [`~transformers.AutoModelForSequenceClassification`].\n ref_model ([`~transformers.PreTrainedModel`]):\n Hugging Face transformer model with a casual language modelling head. Used for implicit reward computation\n and loss. If no reference model is provided, the trainer will create a reference model with the same\n architecture as the model to be optimized.\n args ([`experimental.bco.BCOConfig`]):\n The arguments to use for training.\n train_dataset ([`~datasets.Dataset`]):\n The dataset to use for training.\n eval_dataset ([`~datasets.Dataset`]):\n The dataset to use for evaluation.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. If provided, will be used to automatically process the inputs\n for the model, and it will be saved along the model to make it easier to rerun an interrupted training or\n reuse the fine-tuned model.\n data_collator ([`~transformers.DataCollator`], *optional*):\n The data collator to use for training. If None is specified, the default data collator\n ([`experimental.utils.DPODataCollatorWithPadding`]) will be used which will pad the sequences to the\n maximum length of the sequences in the batch, given a dataset of paired sequences.\n model_init (`Callable[[], transformers.PreTrainedModel]`):\n The model initializer to use for training. If None is specified, the default model initializer will be\n used.\n callbacks (`list[transformers.TrainerCallback]`):\n The callbacks to use for training.\n optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):\n The optimizer and scheduler to use for training.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):\n The function to use to preprocess the logits before computing the metrics.\n peft_config (`dict`, defaults to `None`):\n The PEFT configuration to use for training. If you pass a PEFT configuration, the model will be wrapped in\n a PEFT model.\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to\n metric values.\n model_adapter_name (`str`, defaults to `None`):\n Name of the train target PEFT adapter, when using LoRA with multiple adapters.\n ref_adapter_name (`str`, defaults to `None`):\n Name of the reference PEFT adapter, when using LoRA with multiple adapters.\n \"\"\"\n\n _tag_names = [\"trl\", \"bco\"]\n _name = \"BCO\"\n _paper = {\n \"title\": \"Binary Classifier Optimization for Large Language Model Alignment\",\n \"id\": \"2404.04656\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{jung2024binary,\n title = {{Binary Classifier Optimization for Large Language Model Alignment}},\n author = {Seungjae Jung and Gunsoo Han and Daniel Wontae Nam and Kyoung{-}Woon On},\n year = 2024,\n eprint = {arXiv:2404.04656}\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module | str = None,\n ref_model: PreTrainedModel | nn.Module | str | None = None,\n args: BCOConfig = None,\n train_dataset: Dataset | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n processing_class: PreTrainedTokenizerBase\n | BaseImageProcessor\n | FeatureExtractionMixin\n | ProcessorMixin\n | None = None,\n data_collator: DataCollator | None = None,\n model_init: Callable[[], PreTrainedModel] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n peft_config: dict | None = None,\n compute_metrics: Callable[[EvalLoopOutput], dict] | None = None,\n model_adapter_name: str | None = None,\n ref_adapter_name: str | None = None,\n embedding_func: Callable | None = None,\n embedding_tokenizer: PreTrainedTokenizerBase | None = None,\n ):\n if embedding_func is not None and not (is_sklearn_available() and is_joblib_available()):\n raise ImportError(\n \"BCOTrainer with UDM requires the scikit-learn and joblib libraries. Please install it with `pip install scikit-learn joblib`.\"\n )\n\n if type(args) is TrainingArguments:\n raise ValueError(\"Please use `BCOConfig` instead `TrainingArguments`.\")\n\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n\n if not isinstance(model, str) and model is not None and ref_model is model:\n raise ValueError(\n \"`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the \"\n \"same as `model`, you must mass a copy of it, or `None` if you use peft.\"\n )\n\n if args.model_init_kwargs is None:\n model_init_kwargs = {}\n elif not isinstance(model, str):\n raise ValueError(\"You passed model_kwargs to the BCOTrainer. But your model is already instantiated.\")\n else:\n model_init_kwargs = args.model_init_kwargs\n dtype = model_init_kwargs.get(\"dtype\", \"auto\")\n if dtype is not None:\n # Convert to `torch.dtype` if an str is passed\n if isinstance(dtype, str) and dtype != \"auto\":\n dtype = getattr(torch, dtype)\n if dtype != \"auto\" and not isinstance(dtype, torch.dtype):\n raise ValueError(\n f\"Invalid `dtype` passed to the BCOConfig. Expected a string with either `torch.dtype` or 'auto', but got {dtype}.\"\n )\n model_init_kwargs[\"dtype\"] = dtype\n model_init_kwargs[\"device_map\"] = model_init_kwargs.get(\"device_map\", \"auto\")\n\n if isinstance(model, str):\n model = AutoModelForCausalLM.from_pretrained(model, **model_init_kwargs)\n\n if isinstance(ref_model, str):\n ref_model = AutoModelForCausalLM.from_pretrained(ref_model, **model_init_kwargs)\n\n # Initialize this variable to False. This helps tracking the case when `peft_module_casting_to_bf16`\n # has been called in order to properly call autocast if needed.\n self._peft_has_been_casted_to_bf16 = False\n\n if not is_peft_available() and peft_config is not None:\n raise ValueError(\n \"PEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it with `pip install peft` to use the PEFT models\"\n )\n elif is_peft_available() and peft_config is not None:\n if isinstance(model, PeftModel):\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first \"\n \"merge and unload the existing adapter, save the resulting base model, and then pass that base \"\n \"model along with the new `peft_config` to the trainer.\"\n )\n\n if getattr(model, \"is_loaded_in_8bit\", False) or getattr(model, \"is_loaded_in_4bit\", False):\n _support_gc_kwargs = hasattr(\n args, \"gradient_checkpointing_kwargs\"\n ) and \"gradient_checkpointing_kwargs\" in list(\n inspect.signature(prepare_model_for_kbit_training).parameters\n )\n\n prepare_model_kwargs = {\"use_gradient_checkpointing\": args.gradient_checkpointing}\n\n if _support_gc_kwargs:\n prepare_model_kwargs[\"gradient_checkpointing_kwargs\"] = args.gradient_checkpointing_kwargs\n\n model = prepare_model_for_kbit_training(model, **prepare_model_kwargs)\n elif args.gradient_checkpointing:\n # For backward compatibility with older versions of transformers\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n # get peft model with the given config\n model = get_peft_model(model, peft_config)\n if args.bf16 and getattr(model, \"is_loaded_in_4bit\", False):\n peft_module_casting_to_bf16(model)\n # If args.bf16 we need to explicitly call `generate` with torch amp autocast context manager\n self._peft_has_been_casted_to_bf16 = True\n\n # For models that use gradient_checkpointing, we need to attach a hook that enables input\n # to explicitly have `requires_grad=True`, otherwise training will either silently\n # fail or completely fail.\n elif args.gradient_checkpointing:\n # For backward compatibility with older versions of transformers\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n if args.generate_during_eval and not (is_wandb_available() or is_comet_available()):\n raise ValueError(\n \"`generate_during_eval=True` requires Weights and Biases or Comet to be installed.\"\n \" Please install `wandb` or `comet-ml` to resolve.\"\n )\n\n if model is not None:\n self.is_encoder_decoder = model.config.is_encoder_decoder\n elif args.is_encoder_decoder is None:\n raise ValueError(\"When no model is provided, you need to pass the parameter is_encoder_decoder.\")\n else:\n self.is_encoder_decoder = args.is_encoder_decoder\n\n self.is_peft_model = is_peft_available() and isinstance(model, PeftModel)\n self.model_adapter_name = model_adapter_name\n self.ref_adapter_name = ref_adapter_name\n\n if ref_model:\n self.ref_model = ref_model\n elif self.is_peft_model or args.precompute_ref_log_probs:\n # The `model` with adapters turned off will be used as the reference model\n self.ref_model = None\n else:\n self.ref_model = create_reference_model(model)\n\n if processing_class is None:\n raise ValueError(\n \"max_length or a processing_class must be specified when using the default DPODataCollatorWithPadding\"\n )\n if args.max_length is None:\n logger.warning(\n \"When using DPODataCollatorWithPadding, you should set `max_length` in the `BCOConfig`. \"\n \"It will be set to `512` by default, but you should do it yourself in the future.\",\n )\n max_length = 512\n if args.max_length is not None:\n max_length = args.max_length\n\n max_completion_length = None\n if args.max_completion_length is None and self.is_encoder_decoder:\n logger.warning(\n \"When using DPODataCollatorWithPadding with an encoder decoder architecture, you should set `max_completion_length` in the BCOTrainer's init\"\n \" it will be set to `128` by default, but you should do it yourself in the future.\",\n )\n max_completion_length = 128\n if args.max_completion_length is not None and self.is_encoder_decoder:\n max_completion_length = args.max_completion_length\n\n if data_collator is None:\n data_collator = DPODataCollatorWithPadding(\n pad_token_id=processing_class.pad_token_id,\n is_encoder_decoder=self.is_encoder_decoder,\n )\n\n if args.remove_unused_columns:\n args.remove_unused_columns = False\n # warn users\n logger.warning(\n \"When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your BCOConfig\"\n \" we have set it for you, but you should do it yourself in the future.\",\n )\n\n self.use_dpo_data_collator = True\n else:\n self.use_dpo_data_collator = False\n\n # Disable dropout in the model and reference model\n if args.disable_dropout:\n disable_dropout_in_model(model)\n if self.ref_model is not None:\n disable_dropout_in_model(self.ref_model)\n\n self.max_length = max_length\n self.generate_during_eval = args.generate_during_eval\n self.truncation_mode = args.truncation_mode\n self.max_completion_length = max_completion_length\n self.precompute_ref_log_probs = args.precompute_ref_log_probs\n\n # Since ref_logs are precomputed on the first call to get_train/eval_dataloader\n # keep track of first called to avoid computation of future calls\n self._precomputed_train_ref_log_probs = False\n self._precomputed_eval_ref_log_probs = False\n\n # metric\n self._stored_metrics = defaultdict(lambda: defaultdict(list))\n\n # BCO parameter\n self.beta = args.beta\n self.aux_loss_enabled = getattr(model.config, \"output_router_logits\", False)\n self.aux_loss_coef = getattr(model.config, \"router_aux_loss_coef\", 0.0)\n if self.aux_loss_enabled and self.aux_loss_coef == 0.0:\n logger.warning(\n \"You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to \"\n \"`0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value \"\n \"greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary \"\n \"loss.\",\n )\n\n # Underlying Distribution Matching argument\n self.embedding_func = embedding_func\n self.embedding_tokenizer = embedding_tokenizer\n\n with PartialState().main_process_first():\n # Extract the prompt if needed\n train_dataset = train_dataset.map(\n maybe_extract_prompt, num_proc=args.dataset_num_proc, desc=\"Extracting prompt from train dataset\"\n )\n # Unpair the dataset if needed\n train_dataset = maybe_unpair_preference_dataset(\n train_dataset, args.dataset_num_proc, desc=\"Unpairing train dataset\"\n )\n # Apply the chat template if needed\n train_dataset = train_dataset.map(\n maybe_apply_chat_template, fn_kwargs={\"tokenizer\": processing_class}, num_proc=args.dataset_num_proc\n )\n if eval_dataset is not None:\n # Extract the prompt if needed\n eval_dataset = eval_dataset.map(\n maybe_extract_prompt, num_proc=args.dataset_num_proc, desc=\"Extracting prompt from eval dataset\"\n )\n # Unpair the dataset if needed\n eval_dataset = maybe_unpair_preference_dataset(\n eval_dataset, args.dataset_num_proc, desc=\"Unpairing eval dataset\"\n )\n eval_dataset = eval_dataset.map(\n maybe_apply_chat_template,\n fn_kwargs={\"tokenizer\": processing_class},\n num_proc=args.dataset_num_proc,\n )\n\n # Tokenize and prepare the training datasets\n train_dataset = train_dataset.map(\n _tokenize,\n batched=True,\n fn_kwargs={\"tokenizer\": processing_class, \"embedding_tokenizer\": self.embedding_tokenizer},\n num_proc=args.dataset_num_proc,\n desc=\"Tokenizing train dataset\",\n )\n\n # Prepare the datasets\n fn_kwargs = {\n \"prefix\": \"\",\n \"is_encoder_decoder\": self.is_encoder_decoder,\n \"tokenizer\": processing_class,\n \"max_length\": self.max_length,\n \"truncation_mode\": self.truncation_mode,\n \"max_completion_length\": self.max_completion_length,\n }\n train_dataset = train_dataset.map(\n _process_tokens,\n fn_kwargs=fn_kwargs,\n num_proc=args.dataset_num_proc,\n desc=\"Processing tokenized train dataset\",\n )\n\n if eval_dataset is not None:\n # Tokenize\n eval_dataset = eval_dataset.map(\n _tokenize,\n fn_kwargs={\"tokenizer\": processing_class, \"embedding_tokenizer\": self.embedding_tokenizer},\n batched=True,\n num_proc=args.dataset_num_proc,\n desc=\"Tokenizing eval dataset\",\n )\n\n # Process\n fn_kwargs = {\n \"prefix\": \"\",\n \"is_encoder_decoder\": self.is_encoder_decoder,\n \"tokenizer\": processing_class,\n \"max_length\": self.max_length,\n \"truncation_mode\": self.truncation_mode,\n \"max_completion_length\": self.max_completion_length,\n }\n eval_dataset = eval_dataset.map(\n _process_tokens,\n fn_kwargs=fn_kwargs,\n num_proc=args.dataset_num_proc,\n desc=\"Processing tokenized eval dataset\",\n )\n\n desirable = train_dataset.filter(\n lambda x: x[\"label\"], num_proc=args.dataset_num_proc, desc=\"Filtering desirable examples\"\n )\n undesirable = train_dataset.filter(\n lambda x: not x[\"label\"], num_proc=args.dataset_num_proc, desc=\"Filtering undesirable examples\"\n )\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n model_init=model_init,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n # Gradient accumulation requires scaled loss. Normally, loss scaling in the parent class depends on whether the\n # model accepts loss-related kwargs. Since we compute our own loss, this check is irrelevant. We set\n # self.model_accepts_loss_kwargs to False to enable scaling.\n self.model_accepts_loss_kwargs = False\n\n # Add tags for models that have been loaded with the correct transformers version\n if hasattr(self.model, \"add_model_tags\"):\n self.model.add_model_tags(self._tag_names)\n\n if not hasattr(self, \"accelerator\"):\n raise AttributeError(\n \"Your `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.\"\n )\n\n # Deepspeed Zero-3 does not support precompute_ref_log_probs\n if self.is_deepspeed_enabled:\n if self.accelerator.state.deepspeed_plugin.zero_stage == 3 and self.precompute_ref_log_probs:\n raise ValueError(\n \"You cannot use `precompute_ref_log_probs=True` with Deepspeed ZeRO-3. Please set `precompute_ref_log_probs=False`.\"\n )\n\n if self.ref_model is None:\n if not (self.is_peft_model or self.precompute_ref_log_probs):\n raise ValueError(\n \"No reference model and model is not a Peft model. Try setting `precompute_ref_log_probs=True`\"\n )\n else:\n if self.is_deepspeed_enabled:\n self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)\n else:\n self.ref_model = self.accelerator.prepare_model(self.ref_model, evaluation_mode=True)\n\n self.running = RunningMoments(accelerator=self.accelerator)\n\n if self.embedding_func is None or args.resume_from_checkpoint:\n return\n\n chosen_embeddings = self._get_sample_prompt_embeddings(desirable, sample_size=self.args.prompt_sample_size)\n rejected_embeddings = self._get_sample_prompt_embeddings(undesirable, sample_size=self.args.prompt_sample_size)\n\n embeddings = torch.cat((chosen_embeddings, rejected_embeddings), dim=0)\n labels = torch.cat(\n (torch.ones_like(chosen_embeddings[:, 0]), torch.zeros_like(rejected_embeddings[:, 0])), dim=0\n )\n\n self.clf = LogisticRegression(class_weight=\"balanced\").fit(\n embeddings.cpu().float().numpy(), labels.cpu().numpy()\n )\n chosen_mean = self.clf.score(\n chosen_embeddings.cpu().float().numpy(), torch.ones_like(chosen_embeddings[:, 0]).cpu().numpy()\n )\n rejected_mean = self.clf.score(\n rejected_embeddings.cpu().float().numpy(), torch.zeros_like(rejected_embeddings[:, 0]).cpu().numpy()\n )\n logger.info(f\"UDM classifier training scores: chosen: {chosen_mean}, rejected: {rejected_mean}\")\n\n @property\n def match_underlying_distribution(self):\n return self.embedding_func is not None and self.embedding_tokenizer is not None\n\n def _get_chosen_prob(self, prompt_embeddings: torch.FloatTensor) -> torch.FloatTensor:\n \"\"\"\n Calculates the probability if the given prompt embedding is from desirable dataset. This function calculates\n the probability in the process and ensemble across processes.\n \"\"\"\n dtype = prompt_embeddings.dtype\n device = prompt_embeddings.device\n rank = self.accelerator.process_index\n\n padded_prompt_embeddings = self.accelerator.pad_across_processes(\n prompt_embeddings, pad_index=self.embedding_tokenizer.pad_token_id\n )\n sample_size = padded_prompt_embeddings.shape[0]\n nonzero = padded_prompt_embeddings.mean(dim=1) != self.embedding_tokenizer.pad_token_id\n prompt_embeddings = self.accelerator.gather(padded_prompt_embeddings)\n\n # cannot predict for all empty values\n if prompt_embeddings.shape[0] == 0:\n return torch.tensor([], device=device, dtype=dtype)\n\n prob = self.clf.predict_proba(prompt_embeddings.cpu().float().numpy())[:, 1]\n prob = torch.as_tensor(prob, dtype=dtype, device=device)\n prob = self.accelerator.reduce(prob, reduction=\"mean\")\n\n prob = prob[sample_size * rank : sample_size * (rank + 1)]\n prob = prob[nonzero]\n\n return prob\n\n def _vectorize_prompt(self, input_ids: torch.LongTensor, attention_mask: torch.LongTensor) -> torch.FloatTensor:\n \"\"\"\n Replaces processing_class.pad_token_id to embedding_tokenizer.pad_token_id and applies self.embedding_func\n \"\"\"\n input_ids = torch.where(\n input_ids == self.processing_class.pad_token_id,\n self.embedding_tokenizer.pad_token_id,\n input_ids,\n )\n\n with torch.no_grad():\n embeddings = self.embedding_func(\n input_ids=input_ids,\n attention_mask=attention_mask,\n )\n\n return embeddings\n\n def _get_prompt_embeddings(\n self, batch: dict[str, list | torch.LongTensor]\n ) -> tuple[torch.FloatTensor, torch.FloatTensor]:\n \"\"\"Extract embeddings from frozen embedding model\"\"\"\n\n if not self.match_underlying_distribution:\n return None, None\n\n embeddings = self._vectorize_prompt(\n input_ids=batch[\"embedding_input_ids\"],\n attention_mask=batch[\"embedding_attention_mask\"],\n )\n\n labels = torch.tensor(batch[\"label\"], dtype=torch.bool, device=embeddings.device)\n chosen_idx = torch.where(labels)[0]\n rejected_idx = torch.where(~labels)[0]\n\n chosen_embeddings = embeddings[chosen_idx, ...]\n rejected_embeddings = embeddings[rejected_idx, ...]\n\n return (chosen_embeddings, rejected_embeddings)\n\n def _get_sample_prompt_embeddings(self, dataset: Dataset, sample_size: int = 512) -> torch.FloatTensor:\n \"\"\"\n Sample instances from dataset and get prompt embeddings. Used for density ratio classifier training.\n \"\"\"\n n_samples = min(len(dataset), sample_size)\n rand_indices = np.random.choice(len(dataset), size=(n_samples,))\n\n embedding_dataset = dataset.select(rand_indices)\n\n dataloader_params = {\n \"batch_size\": self.args.per_device_train_batch_size,\n \"collate_fn\": self.data_collator,\n \"num_workers\": self.args.dataloader_num_workers,\n \"pin_memory\": self.args.dataloader_pin_memory,\n \"shuffle\": False,\n }\n\n # prepare dataloader\n data_loader = self.accelerator.prepare(DataLoader(embedding_dataset, **dataloader_params))\n\n with torch.no_grad():\n all_embeddings = torch.empty(0)\n for padded_batch in tqdm(iterable=data_loader, desc=\"Building sample prompt embeddings\"):\n embeddings = self._vectorize_prompt(\n input_ids=padded_batch[\"embedding_input_ids\"],\n attention_mask=padded_batch[\"embedding_attention_mask\"],\n )\n embeddings = self.accelerator.gather_for_metrics(embeddings)\n all_embeddings = torch.cat((all_embeddings, embeddings.cpu()))\n\n return all_embeddings\n\n def _save_optimizer_and_scheduler(self, output_dir):\n output_dir = output_dir if output_dir is not None else self.args.output_dir\n super()._save_optimizer_and_scheduler(output_dir)\n\n if self.accelerator.is_main_process:\n # When saving optimizer and scheduler to checkpoint, save also the running delta object.\n self.running.save_to_json(os.path.join(output_dir, RUNNING_NAME))\n\n if self.match_underlying_distribution:\n joblib.dump(self.clf, os.path.join(output_dir, CLF_NAME), compress=True)\n\n def _load_optimizer_and_scheduler(self, checkpoint):\n if checkpoint is None:\n logger.warning_once(f\"Missing Checkpoint {checkpoint}\")\n return\n\n super()._load_optimizer_and_scheduler(checkpoint)\n\n # when loading optimizer and scheduler from checkpoint, also load the running delta object.\n running_file = os.path.join(checkpoint, RUNNING_NAME)\n if os.path.isfile(running_file):\n self.running = RunningMoments.load_from_json(self.accelerator, running_file)\n\n if self.match_underlying_distribution:\n clf_file = os.path.join(checkpoint, CLF_NAME)\n if os.path.isfile(clf_file):\n self.clf = joblib.load(clf_file)\n\n @contextmanager\n def null_ref_context(self):\n \"\"\"Context manager for handling null reference model (that is, peft adapter manipulation).\"\"\"\n with (\n self.accelerator.unwrap_model(self.model).disable_adapter()\n if self.is_peft_model and not self.ref_adapter_name\n else nullcontext()\n ):\n if self.ref_adapter_name:\n self.model.set_adapter(self.ref_adapter_name)\n yield\n if self.ref_adapter_name:\n self.model.set_adapter(self.model_adapter_name or \"default\")\n\n def get_train_dataloader(self) -> DataLoader:\n \"\"\"\n Returns the training [`~torch.utils.data.DataLoader`].\n\n Subclass of transformers.src.transformers.trainer.get_train_dataloader to precompute `ref_log_probs`.\n \"\"\"\n\n if self.precompute_ref_log_probs and not self._precomputed_train_ref_log_probs:\n dataloader_params = {\n \"batch_size\": self.args.per_device_train_batch_size,\n \"collate_fn\": self.data_collator,\n \"num_workers\": self.args.dataloader_num_workers,\n \"pin_memory\": self.args.dataloader_pin_memory,\n \"shuffle\": False,\n }\n\n # prepare dataloader\n data_loader = self.accelerator.prepare(DataLoader(self.train_dataset, **dataloader_params))\n reference_completion_logps = []\n\n for padded_batch in tqdm(iterable=data_loader, desc=\"Train dataset reference log probs\"):\n reference_completion_logp = self.compute_reference_log_probs(padded_batch)\n\n reference_completion_logp = self.accelerator.gather_for_metrics(reference_completion_logp)\n reference_completion_logps.append(reference_completion_logp.cpu())\n\n self.train_dataset = self.train_dataset.add_column(\n name=\"reference_logps\", column=torch.cat(reference_completion_logps).float().numpy()\n )\n\n self._precomputed_train_ref_log_probs = True\n\n return super().get_train_dataloader()\n\n def get_eval_dataloader(self, eval_dataset: Dataset | None = None) -> DataLoader:\n \"\"\"\n Returns the evaluation [`~torch.utils.data.DataLoader`].\n\n Subclass of transformers.src.transformers.trainer.get_eval_dataloader to precompute `ref_log_probs`.\n\n Args:\n eval_dataset (`torch.utils.data.Dataset`, *optional*):\n If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted\n by the `model.forward()` method are automatically removed. It must implement `__len__`.\n \"\"\"\n if eval_dataset is None and self.eval_dataset is None:\n raise ValueError(\"Trainer: evaluation requires an eval_dataset.\")\n eval_dataset = eval_dataset if eval_dataset is not None else self.eval_dataset\n\n if self.precompute_ref_log_probs and not self._precomputed_eval_ref_log_probs:\n dataloader_params = {\n \"batch_size\": self.args.per_device_eval_batch_size,\n \"collate_fn\": self.data_collator,\n \"num_workers\": self.args.dataloader_num_workers,\n \"pin_memory\": self.args.dataloader_pin_memory,\n \"shuffle\": False,\n }\n\n # prepare dataloader\n data_loader = self.accelerator.prepare(DataLoader(eval_dataset, **dataloader_params))\n\n reference_completion_logps = []\n\n for padded_batch in tqdm(iterable=data_loader, desc=\"Eval dataset reference log probs\"):\n reference_completion_logp = self.compute_reference_log_probs(padded_batch)\n\n reference_completion_logp = self.accelerator.gather_for_metrics(reference_completion_logp)\n reference_completion_logps.append(reference_completion_logp.cpu())\n\n eval_dataset = eval_dataset.add_column(\n name=\"reference_logps\", column=torch.cat(reference_completion_logps).float().numpy()\n )\n\n # Save calculated reference_chosen_logps and reference_rejected_logps to the eval_dataset for subsequent runs\n if self.eval_dataset is not None:\n self.eval_dataset = eval_dataset\n self._precomputed_eval_ref_log_probs = True\n\n return super().get_eval_dataloader(eval_dataset=eval_dataset)\n\n def compute_reference_log_probs(self, padded_batch: dict) -> dict:\n \"\"\"Computes log probabilities of the reference model for a single padded batch of a BCO specific dataset.\"\"\"\n with torch.no_grad():\n if self.ref_model is None:\n with self.null_ref_context():\n if self.is_encoder_decoder:\n completion_logits = self.model(\n padded_batch[\"prompt_input_ids\"],\n attention_mask=padded_batch[\"prompt_attention_mask\"],\n decoder_input_ids=padded_batch.get(\"completion_decoder_input_ids\"),\n labels=padded_batch[\"completion_labels\"],\n ).logits\n\n else:\n completion_logits = self.model(\n padded_batch[\"completion_input_ids\"],\n attention_mask=padded_batch[\"completion_attention_mask\"],\n ).logits\n\n else:\n if self.is_encoder_decoder:\n completion_logits = self.ref_model(\n padded_batch[\"prompt_input_ids\"],\n attention_mask=padded_batch[\"prompt_attention_mask\"],\n decoder_input_ids=padded_batch.get(\"completion_decoder_input_ids\"),\n labels=padded_batch[\"completion_labels\"],\n ).logits\n\n else:\n completion_logits = self.ref_model(\n padded_batch[\"completion_input_ids\"], attention_mask=padded_batch[\"completion_attention_mask\"]\n ).logits\n\n completion_logps = self.get_batch_logps(\n completion_logits,\n padded_batch[\"completion_labels\"],\n average_log_prob=False,\n is_encoder_decoder=self.is_encoder_decoder,\n )\n\n return completion_logps\n\n @staticmethod\n def get_batch_logps(\n logits: torch.FloatTensor,\n labels: torch.LongTensor,\n average_log_prob: bool = False,\n is_encoder_decoder: bool = False,\n ) -> torch.FloatTensor:\n \"\"\"Compute the log probabilities of the given labels under the given logits.\n\n Args:\n logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)\n labels:\n Labels for which to compute the log probabilities. Label tokens with a value of `-100` are ignored.\n Shape: (batch_size, sequence_length)\n average_log_prob:\n If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the\n log probabilities of the (non-masked) tokens.\n is_encoder_decoder:\n Whether the model is an encoder-decoder model. If True, the labels are not shifted, and the logits are\n assumed to already be aligned with the labels. If False, the labels are shifted to the right by one\n position, and the logits are assumed to be aligned with the shifted labels.\n\n Returns:\n A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the\n given logits.\n \"\"\"\n if logits.shape[:-1] != labels.shape:\n raise ValueError(\"Logits (batch and sequence length dim) and labels must have the same shape.\")\n\n if not is_encoder_decoder:\n labels = labels[:, 1:].clone()\n logits = logits[:, :-1, :]\n else:\n # Fixes end-dec RuntimeError\n labels = labels.clone()\n\n loss_mask = labels != -100\n\n # dummy token; we'll ignore the losses on these tokens later\n labels[labels == -100] = 0\n\n per_token_logps = selective_log_softmax(logits, labels)\n\n if average_log_prob:\n return (per_token_logps * loss_mask).sum(-1) / loss_mask.sum(-1)\n else:\n return (per_token_logps * loss_mask).sum(-1)\n\n def forward(\n self, model: nn.Module, batch: dict[str, list | torch.LongTensor]\n ) -> tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n model_kwargs = (\n {\n \"labels\": batch[\"completion_labels\"],\n \"decoder_input_ids\": batch.get(\"completion_decoder_input_ids\"),\n }\n if self.is_encoder_decoder\n else {}\n )\n if self.aux_loss_enabled:\n model_kwargs[\"output_router_logits\"] = True\n\n outputs = model(\n batch[\"completion_input_ids\"],\n attention_mask=batch[\"completion_attention_mask\"],\n **model_kwargs,\n )\n completion_logits = outputs.logits\n\n completion_logps = self.get_batch_logps(\n completion_logits,\n batch[\"completion_labels\"],\n average_log_prob=False,\n is_encoder_decoder=self.is_encoder_decoder,\n )\n\n if completion_logps.shape[0] != len(batch[\"label\"]):\n raise ValueError(\n \"There is a mismatch between the number of examples in this batch and the number of \"\n \"examples for which an output sequence was predicted.\"\n )\n\n chosen_idx = [i for i in range(completion_logps.shape[0]) if batch[\"label\"][i] is True]\n rejected_idx = [i for i in range(completion_logps.shape[0]) if batch[\"label\"][i] is False]\n\n chosen_logps = completion_logps[chosen_idx, ...]\n rejected_logps = completion_logps[rejected_idx, ...]\n\n chosen_logits = completion_logits[chosen_idx, ...]\n rejected_logits = completion_logits[rejected_idx, ...]\n\n if self.aux_loss_enabled:\n return (chosen_logps, rejected_logps, chosen_logits, rejected_logits, outputs.aux_loss)\n else:\n return (chosen_logps, rejected_logps, chosen_logits, rejected_logits)\n\n def _get_udm_weight(self, rejected_embeddings: torch.FloatTensor) -> torch.FloatTensor:\n prob_desirable = self._get_chosen_prob(rejected_embeddings)\n min_ratio = self.args.min_density_ratio\n max_ratio = self.args.max_density_ratio\n\n weight = (prob_desirable / (1 - prob_desirable + 1e-8)).clamp(min=min_ratio, max=max_ratio)\n\n return weight\n\n def bco_loss(\n self,\n policy_chosen_logps: torch.FloatTensor,\n policy_rejected_logps: torch.FloatTensor,\n reference_chosen_logps: torch.FloatTensor,\n reference_rejected_logps: torch.FloatTensor,\n chosen_embeddings: torch.FloatTensor | None,\n rejected_embeddings: torch.FloatTensor | None,\n do_train: bool = True,\n ) -> tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n \"\"\"Compute the BCO loss for a batch of policy and reference model log probabilities.\n\n Args:\n policy_chosen_logps:\n Log probabilities of the policy model for the chosen responses. Shape: (num(chosen) in batch_size,)\n policy_rejected_logps:\n Log probabilities of the policy model for the rejected responses. Shape: (num(rejected) in batch_size,)\n reference_chosen_logps:\n Log probabilities of the reference model for the chosen responses. Shape: (num(chosen) in batch_size,)\n reference_rejected_logps:\n Log probabilities of the reference model for the rejected responses. Shape: (num(rejected) in\n batch_size,)\n chosen_embeddings: embeddings of desirable prompts\n rejected_embeddings: embeddings of undesirable prompts\n do_train: whether to update the running delta value. Default is True.\n\n Returns:\n A tuple of four tensors: (losses, chosen_rewards, rejected_rewards, delta). The losses tensor contains the\n BCO loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards\n for the chosen and rejected responses, respectively. The delta value contains the moving average of all\n implicit rewards.\n \"\"\"\n\n chosen_logratios = policy_chosen_logps - reference_chosen_logps\n chosen_rewards = self.beta * chosen_logratios\n\n rejected_logratios = policy_rejected_logps - reference_rejected_logps\n rejected_rewards = self.beta * rejected_logratios\n\n if do_train:\n self.running.update(torch.cat((chosen_rewards, rejected_rewards), 0).detach())\n delta = torch.as_tensor(self.running.mean, device=chosen_rewards.device)\n\n chosen_losses = -F.logsigmoid(chosen_rewards - delta)\n rejected_losses = -F.logsigmoid(-(rejected_rewards - delta))\n\n if self.match_underlying_distribution:\n chosen_weight = torch.ones_like(chosen_losses)\n rejected_weight = self._get_udm_weight(rejected_embeddings)\n\n losses = torch.cat((chosen_weight * chosen_losses, rejected_weight * rejected_losses), dim=0)\n else:\n losses = torch.cat((chosen_losses, rejected_losses), dim=0)\n\n return losses, chosen_rewards, rejected_rewards, delta\n\n def get_batch_loss_metrics(\n self,\n model,\n batch: dict[str, list | torch.LongTensor],\n do_train: bool = True,\n ):\n \"\"\"Compute the BCO loss and other metrics for the given batch of inputs for train or test.\"\"\"\n metrics = {}\n batch = {k: (v.to(self.accelerator.device) if isinstance(v, torch.Tensor) else v) for k, v in batch.items()}\n\n forward_output = self.forward(model, batch)\n (\n policy_chosen_logps,\n policy_rejected_logps,\n policy_chosen_logits,\n policy_rejected_logits,\n ) = forward_output[:4]\n if self.aux_loss_enabled:\n aux_loss = forward_output[4]\n\n # if reference_logps in batch use them, otherwise use the reference model\n if \"reference_logps\" in batch:\n chosen_idx = [i for i in range(batch[\"reference_logps\"].shape[0]) if batch[\"label\"][i] is True]\n rejected_idx = [i for i in range(batch[\"reference_logps\"].shape[0]) if batch[\"label\"][i] is False]\n\n reference_chosen_logps = batch[\"reference_logps\"][chosen_idx, ...]\n reference_rejected_logps = batch[\"reference_logps\"][rejected_idx, ...]\n else:\n with torch.no_grad():\n if self.ref_model is None:\n with self.null_ref_context():\n (\n reference_chosen_logps,\n reference_rejected_logps,\n _,\n _,\n ) = self.forward(self.model, batch)[:4]\n else:\n (\n reference_chosen_logps,\n reference_rejected_logps,\n _,\n _,\n ) = self.forward(self.ref_model, batch)[:4]\n\n chosen_embeddings, rejected_embeddings = self._get_prompt_embeddings(batch)\n\n losses, chosen_rewards, rejected_rewards, delta = self.bco_loss(\n policy_chosen_logps,\n policy_rejected_logps,\n reference_chosen_logps,\n reference_rejected_logps,\n chosen_embeddings,\n rejected_embeddings,\n do_train=do_train,\n )\n metrics[\"delta\"] = self.accelerator.gather_for_metrics(delta).mean().item()\n\n num_chosen = torch.Tensor([len(chosen_rewards)]).to(self.accelerator.device)\n num_rejected = torch.Tensor([len(rejected_rewards)]).to(self.accelerator.device)\n\n all_num_chosen = self.accelerator.gather_for_metrics(num_chosen).sum().item()\n all_num_rejected = self.accelerator.gather_for_metrics(num_rejected).sum().item()\n\n if all_num_chosen > 0:\n metrics[\"rewards/chosen_sum\"] = (\n self.accelerator.gather_for_metrics(chosen_rewards.nansum()).nansum().item()\n )\n metrics[\"logps/chosen_sum\"] = (\n self.accelerator.gather_for_metrics(policy_chosen_logps.nansum()).nansum().item()\n )\n metrics[\"logits/chosen_sum\"] = (\n self.accelerator.gather_for_metrics(policy_chosen_logits.nansum()).nansum().item()\n )\n metrics[\"count/chosen\"] = all_num_chosen\n\n if all_num_rejected > 0:\n metrics[\"rewards/rejected_sum\"] = (\n self.accelerator.gather_for_metrics(rejected_rewards.nansum()).nansum().item()\n )\n metrics[\"logps/rejected_sum\"] = (\n self.accelerator.gather_for_metrics(policy_rejected_logps.nansum()).nansum().item()\n )\n metrics[\"logits/rejected_sum\"] = (\n self.accelerator.gather_for_metrics(policy_rejected_logits.nansum()).nansum().item()\n )\n metrics[\"count/rejected\"] = all_num_rejected\n\n loss = losses.nanmean()\n if self.aux_loss_enabled:\n loss += self.aux_loss_coef * aux_loss\n\n return loss, metrics\n\n def compute_loss(\n self,\n model: PreTrainedModel | nn.Module,\n inputs: dict[str, torch.Tensor | Any],\n return_outputs=False,\n num_items_in_batch=None,\n ) -> torch.Tensor | tuple[torch.Tensor, dict[str, torch.Tensor]]:\n compute_loss_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n\n with compute_loss_context_manager:\n loss, metrics = self.get_batch_loss_metrics(model, inputs)\n\n # Make sure to move the loss to the device the original accumulating loss is at back in the `Trainer` class:\n loss = loss.to(self.args.device)\n # force log the metrics\n if self.accelerator.is_main_process:\n self.store_metrics(metrics, train_eval=\"train\")\n\n if return_outputs:\n return (loss, metrics)\n return loss\n\n def store_metrics(self, metrics: dict[str, float], train_eval: Literal[\"train\", \"eval\"] = \"train\") -> None:\n for key, value in metrics.items():\n self._stored_metrics[train_eval][key].append(value)\n\n def _get_train_sampler(self, dataset: Dataset | None = None) -> torch.utils.data.Sampler | None:\n if dataset is None:\n dataset = self.train_dataset\n if dataset is None or not has_length(dataset):\n return None\n return SequentialSampler(dataset)\n\n def generate_from_model_and_ref(self, model, batch: dict[str, torch.LongTensor]) -> tuple[str, str]:\n \"\"\"Generate samples from the model and reference model for the given batch of inputs.\"\"\"\n\n # If one uses `generate_during_eval` with peft + bf16, we need to explicitly call generate with\n # the torch amp context manager as some hidden states are silently casted to full precision.\n generate_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n with generate_context_manager:\n policy_output = model.generate(\n input_ids=batch[\"prompt_input_ids\"],\n attention_mask=batch[\"prompt_attention_mask\"],\n max_length=self.max_length,\n do_sample=True,\n pad_token_id=self.processing_class.pad_token_id,\n )\n\n # if reference_output in batch use that otherwise use the reference model\n if \"reference_output\" in batch:\n reference_output = batch[\"reference_output\"]\n else:\n if self.ref_model is None:\n with self.null_ref_context():\n reference_output = self.model.generate(\n input_ids=batch[\"prompt_input_ids\"],\n attention_mask=batch[\"prompt_attention_mask\"],\n max_length=self.max_length,\n do_sample=True,\n pad_token_id=self.processing_class.pad_token_id,\n )\n else:\n reference_output = self.ref_model.generate(\n input_ids=batch[\"prompt_input_ids\"],\n attention_mask=batch[\"prompt_attention_mask\"],\n max_length=self.max_length,\n do_sample=True,\n pad_token_id=self.processing_class.pad_token_id,\n )\n\n policy_output = pad_to_length(policy_output, self.max_length, self.processing_class.pad_token_id)\n policy_output_decoded = self.processing_class.batch_decode(policy_output, skip_special_tokens=True)\n\n reference_output = pad_to_length(reference_output, self.max_length, self.processing_class.pad_token_id)\n reference_output_decoded = self.processing_class.batch_decode(reference_output, skip_special_tokens=True)\n\n return policy_output_decoded, reference_output_decoded\n\n def prediction_step(\n self,\n model: PreTrainedModel | nn.Module,\n inputs: dict[str, torch.Tensor | Any],\n prediction_loss_only: bool,\n ignore_keys: list[str] | None = None,\n ):\n if ignore_keys is None:\n if hasattr(model, \"config\"):\n ignore_keys = getattr(model.config, \"keys_to_ignore_at_inference\", [])\n else:\n ignore_keys = []\n\n prediction_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n with torch.no_grad(), prediction_context_manager:\n loss, metrics = self.get_batch_loss_metrics(model, inputs, do_train=False)\n\n # force log the metrics\n if self.accelerator.is_main_process:\n self.store_metrics(metrics, train_eval=\"eval\")\n\n if prediction_loss_only:\n return (loss.detach(), None, None)\n\n # logits for the chosen and rejected samples from model\n logits_dict = {}\n if \"logits/chosen_sum\" in metrics:\n logits_dict[\"eval_logits/chosen\"] = metrics[\"logits/chosen_sum\"]\n if \"logits/rejected_sum\" in metrics:\n logits_dict[\"eval_logits/rejected\"] = metrics[\"logits/rejected_sum\"]\n logits = [v for k, v in logits_dict.items() if k not in ignore_keys]\n logits = torch.tensor(logits, device=self.accelerator.device)\n labels = torch.zeros(logits.shape[0], device=self.accelerator.device)\n\n return (loss.detach(), logits, labels)\n\n def evaluation_loop(\n self,\n dataloader: DataLoader,\n description: str,\n prediction_loss_only: bool | None = None,\n ignore_keys: list[str] | None = None,\n metric_key_prefix: str = \"eval\",\n ) -> EvalLoopOutput:\n \"\"\"\n Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by\n `Trainer.evaluate()` and `Trainer.predict()`.\n\n Works both with or without labels.\n \"\"\"\n\n # Sample and save to game log if requested (for one batch to save time)\n if self.generate_during_eval:\n # Generate random indices within the range of the total number of samples\n num_samples = len(dataloader.dataset)\n random_indices = random.sample(range(num_samples), k=self.args.eval_batch_size)\n\n # Use dataloader.dataset.select to get the random batch without iterating over the DataLoader\n random_batch_dataset = dataloader.dataset.select(random_indices)\n random_batch = self.data_collator(random_batch_dataset)\n random_batch = self._prepare_inputs(random_batch)\n\n target_labels = torch.tensor(random_batch[\"label\"], dtype=torch.bool, device=self.accelerator.device)\n target_indices = torch.where(~target_labels)[0]\n target_batch = {\n \"prompt_input_ids\": random_batch[\"prompt_input_ids\"][target_indices],\n \"prompt_attention_mask\": random_batch[\"prompt_attention_mask\"][target_indices],\n \"prompt\": itemgetter(*target_indices)(random_batch[\"prompt\"]),\n }\n policy_output_decoded, ref_output_decoded = self.generate_from_model_and_ref(self.model, target_batch)\n\n table = pd.DataFrame(\n columns=[\"Prompt\", \"Policy\", \"Ref Model\"],\n data=[\n [prompt, pol[len(prompt) :], ref[len(prompt) :]]\n for prompt, pol, ref in zip(\n target_batch[\"prompt\"], policy_output_decoded, ref_output_decoded, strict=True\n )\n ],\n )\n if \"wandb\" in self.args.report_to:\n wandb.log({\"game_log\": wandb.Table(data=table)})\n\n if \"comet_ml\" in self.args.report_to:\n log_table_to_comet_experiment(\n name=\"game_log.csv\",\n table=table,\n )\n\n # Base evaluation\n initial_output = super().evaluation_loop(\n dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix\n )\n\n return initial_output\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n \"\"\"\n Log `logs` on the various objects watching training, including stored metrics.\n\n Args:\n logs (`dict[str, float]`):\n The values to log.\n start_time (`float`, *optional*):\n Start time of the training.\n \"\"\"\n # logs either has 'loss' or 'eval_loss'\n train_eval = \"train\" if \"loss\" in logs else \"eval\"\n # train metrics should have no prefix, eval should have 'eval_'\n prefix = \"eval_\" if train_eval == \"eval\" else \"\"\n # accumulate average metrics from sums and lengths\n for split in [\"chosen\", \"rejected\"]:\n if f\"count/{split}\" in self._stored_metrics[train_eval]:\n count_sum = torch.Tensor(self._stored_metrics[train_eval][f\"count/{split}\"]).sum().item()\n for metric in [\"rewards\", \"logps\", \"logits\"]:\n logs[f\"{prefix}{metric}/{split}\"] = (\n torch.Tensor(self._stored_metrics[train_eval][f\"{metric}/{split}_sum\"]).sum().item()\n / count_sum\n )\n # delete obsolete metric\n del self._stored_metrics[train_eval][f\"{metric}/{split}_sum\"]\n del self._stored_metrics[train_eval][f\"count/{split}\"]\n # calculate reward margin\n if f\"{prefix}rewards/chosen\" in logs and f\"{prefix}rewards/rejected\" in logs:\n logs[f\"{prefix}rewards/margins\"] = logs[f\"{prefix}rewards/chosen\"] - logs[f\"{prefix}rewards/rejected\"]\n # Add averaged stored metrics to logs\n for key, metrics in self._stored_metrics[train_eval].items():\n logs[f\"{prefix}{key}\"] = torch.Tensor(metrics).mean().item()\n del self._stored_metrics[train_eval]\n return super().log(logs, start_time)\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/experimental/bema_for_ref_model/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .callback import BEMACallback\nfrom .dpo_trainer import DPOTrainer\n" }, { "path": "trl/experimental/bema_for_ref_model/callback.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport logging\n\nimport torch\nfrom transformers import PreTrainedModel, TrainerControl, TrainerState, TrainingArguments\nfrom transformers.trainer_callback import CallbackHandler\n\nfrom ...trainer.callbacks import BEMACallback as _BEMACallback\n\n\n# Logger for module-level logging\nlogger = logging.getLogger(__name__)\n\n\nclass CallbackHandlerWithRefModel(CallbackHandler):\n \"\"\"\n A [`~transformers.CallbackHandler`] that supports passing a reference model to callbacks.\n \"\"\"\n\n def __init__(self, callbacks, model, ref_model, processing_class, optimizer, lr_scheduler):\n super().__init__(callbacks, model, processing_class, optimizer, lr_scheduler)\n self.ref_model = ref_model\n\n # Copied from CallbackHandler.call_event with the addition of `ref_model` to the callback call.\n def call_event(self, event, args, state, control, **kwargs):\n for callback in self.callbacks:\n result = getattr(callback, event)(\n args,\n state,\n control,\n model=self.model,\n ref_model=self.ref_model, # <- Added ref_model to the callback call\n processing_class=self.processing_class,\n optimizer=self.optimizer,\n lr_scheduler=self.lr_scheduler,\n train_dataloader=self.train_dataloader,\n eval_dataloader=self.eval_dataloader,\n **kwargs,\n )\n # A Callback can skip the return of `control` if it doesn't change it.\n if result is not None:\n control = result\n return control\n\n\nclass BEMACallback(_BEMACallback):\n # docstyle-ignore\n r\"\"\"\n A [`~transformers.TrainerCallback`] that implements [BEMA](https://huggingface.co/papers/2508.00180)\n (Bias-Corrected Exponential Moving Average) by [Adam Block](https://huggingface.co/abblock) and [Cyril\n Zhang](https://huggingface.co/cyrilzhang). Code from https://github.com/abblock/bema under MIT license.\n\n BEMA computes model weights that scale like:\n\n $$\n \\theta_t' = \\alpha_t \\cdot (\\theta_t - \\theta_0) + \\text{EMA}_t\n $$\n\n where \\\\( \\theta_t \\\\) is the current model weights, \\\\( \\theta_0 \\\\) is a snapshot of the model weights at the\n first `update_after` step, \\\\( \\text{EMA}_t \\\\) is the exponential moving average of the model weights, and\n \\\\( \\alpha_t \\\\) is a scaling factor that decays with the number of steps \\\\( t \\\\) as\n\n $$\n \\alpha_t = (\\rho + \\gamma \\cdot t)^{-\\eta}.\n $$\n\n The EMA is computed as:\n\n $$\n \\text{EMA}_t = (1 - \\beta_t) \\cdot \\text{EMA}_{t-1} + \\beta_t \\cdot \\theta_t\n $$\n\n where \\\\( \\beta_t \\\\) is a decay factor that decays with the number of steps \\\\( t \\\\) as\n\n $$\n \\beta_t = (\\rho + \\gamma \\cdot t)^{-\\kappa}.\n $$\n\n Args:\n update_freq (`int`, *optional*, defaults to `400`):\n Update the BEMA weights every X steps. Denoted this as \\\\( \\phi \\\\) in the paper.\n ema_power (`float`, *optional*, defaults to `0.5`):\n Power for the EMA decay factor. Denoted \\\\( \\kappa \\\\) in the paper. To disable EMA, set this to `0.0`.\n bias_power (`float`, *optional*, defaults to `0.2`):\n Power for the BEMA scaling factor. Denoted \\\\( \\eta \\\\) in the paper. To disable BEMA, set this to `0.0`.\n lag (`int`, *optional*, defaults to `10`):\n Initial offset in the weight decay schedule that controls early-stage smoothness by acting as a virtual\n starting age for the updates. Denoted as \\\\( \\rho \\\\) in the paper.\n update_after (`int`, *optional*, defaults to `0`):\n Burn-in time before starting to update the BEMA weights. Denoted \\\\( \\tau \\\\) in the paper.\n multiplier (`float`, *optional*, defaults to `1.0`):\n Initial value for the EMA decay factor. Denoted as \\\\( \\gamma \\\\) in the paper.\n min_ema_multiplier (`float`, *optional*, defaults to `0.0`):\n Minimum value for the EMA decay factor.\n device (`str`, *optional*, defaults to `\"cpu\"`):\n Device to use for the BEMA buffers, e.g. `\"cpu\"` or `\"cuda\"`. Note that in most cases, this device SHOULD\n BE DIFFERENT from the device used for training in order to avoid OOM.\n update_ref_model (`bool`, *optional*, defaults to `False`):\n Whether to update the reference model with BEMA weights. This creates a lagged, smoothed version of the\n main model as the reference model.\n ref_model_update_freq (`int`, *optional*, defaults to `400`):\n Update the reference model with BEMA weights every this many steps.\n ref_model_update_after (`int`, *optional*, defaults to `0`):\n Number of steps to wait before starting to update the reference model.\n\n Example:\n\n ```python\n from trl import BEMACallback\n\n trainer = Trainer(..., callbacks=[BEMACallback()])\n ```\n \"\"\"\n\n def __init__(\n self,\n update_freq: int = 400,\n ema_power: float = 0.5,\n bias_power: float = 0.2,\n lag: int = 10,\n update_after: int = 0,\n multiplier: float = 1.0,\n min_ema_multiplier: float = 0.0,\n device: str = \"cpu\",\n update_ref_model: bool = False,\n ref_model_update_freq: int = 400,\n ref_model_update_after: int = 0,\n ):\n super().__init__(\n update_freq,\n ema_power,\n bias_power,\n lag,\n update_after,\n multiplier,\n min_ema_multiplier,\n device,\n )\n # Reference model update parameters\n self.update_ref_model = update_ref_model\n self.ref_model_update_freq = ref_model_update_freq\n self.ref_model_update_after = ref_model_update_after\n\n @torch.no_grad()\n def on_step_end(\n self, args: TrainingArguments, state: TrainerState, control: TrainerControl, model: PreTrainedModel, **kwargs\n ):\n super().on_step_end(args, state, control, model, **kwargs)\n\n step = state.global_step\n # Update reference model if enabled\n if (\n self.update_ref_model\n and step >= self.ref_model_update_after\n and (step - self.ref_model_update_after) % self.ref_model_update_freq == 0\n ):\n if \"ref_model\" not in kwargs:\n raise ValueError(\"'ref_model' not found in kwargs.\")\n\n ref_model = kwargs[\"ref_model\"]\n\n # Get the current BEMA state dict\n bema_state_dict = self.running_model.state_dict()\n\n # Handle the case where ref_model is None (PEFT case)\n if ref_model is None:\n # In PEFT case, ref_model is None and we need to update the base model of the main model\n main_model = self._unwrap_model(model)\n if hasattr(main_model, \"get_base_model\"):\n # This is a PEFT model, update the base model\n base_model = main_model.get_base_model()\n self._update_model_with_bema_weights(base_model, bema_state_dict, is_peft_base=True)\n else:\n # Regular model, update directly\n self._update_model_with_bema_weights(main_model, bema_state_dict, is_peft_base=False)\n else:\n # ref_model is provided, unwrap it and update\n ref_model = self._unwrap_model(ref_model)\n if hasattr(ref_model, \"get_base_model\"):\n # This is a PEFT model, update the base model\n base_model = ref_model.get_base_model()\n self._update_model_with_bema_weights(base_model, bema_state_dict, is_peft_base=True)\n else:\n # Regular model, update directly\n self._update_model_with_bema_weights(ref_model, bema_state_dict, is_peft_base=False)\n\n logger.info(\"BEMACallback: Updated reference model with BEMA weights\")\n\n def _update_model_with_bema_weights(self, model, bema_state_dict, is_peft_base=False):\n \"\"\"Helper method to update a model with BEMA weights, handling PEFT and distributed scenarios.\"\"\"\n if is_peft_base:\n # For PEFT base models, filter out adapter parameters\n filtered_state_dict = {}\n for key, value in bema_state_dict.items():\n # Skip adapter parameters\n if not key.startswith(\"lora_\") and not key.startswith(\"adapter_\"):\n # Remove 'base_model.' prefix if it exists\n if key.startswith(\"base_model.\"):\n base_key = key[len(\"base_model.\") :]\n else:\n base_key = key\n filtered_state_dict[base_key] = value\n\n # Update the base model\n model.load_state_dict(filtered_state_dict, strict=False)\n else:\n # Regular model, update directly\n model.load_state_dict(bema_state_dict, strict=False)\n" }, { "path": "trl/experimental/bema_for_ref_model/dpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom ...trainer.dpo_trainer import DPOTrainer as _DPOTrainer\nfrom .callback import CallbackHandlerWithRefModel\n\n\nclass DPOTrainer(_DPOTrainer):\n def __init__(self, *args, **kwargs):\n super().__init__(*args, **kwargs)\n # Replace with a new one that calls the events with the reference model\n self.callback_handler = CallbackHandlerWithRefModel(\n self.callback_handler.callbacks,\n self.model,\n self.ref_model,\n self.processing_class,\n self.optimizer,\n self.lr_scheduler,\n )\n" }, { "path": "trl/experimental/cpo/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .cpo_config import CPOConfig\nfrom .cpo_trainer import CPOTrainer\n\n\n__all__ = [\"CPOConfig\", \"CPOTrainer\"]\n" }, { "path": "trl/experimental/cpo/cpo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom ...trainer.base_config import _BaseConfig\n\n\n@dataclass\nclass CPOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`experimental.cpo.CPOTrainer`].\n\n This class includes only the parameters that are specific to CPO training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n max_length (`int` or `None`, *optional*, defaults to `1024`):\n Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want\n to use the default data collator.\n max_completion_length (`int`, *optional*):\n Maximum length of the completion. This argument is required if you want to use the default data collator\n and your model is an encoder-decoder.\n beta (`float`, *optional*, defaults to `0.1`):\n Parameter controlling the deviation from the reference model. Higher ฮฒ means less deviation from the\n reference model. For the IPO loss (`loss_type=\"ipo\"`), ฮฒ is the regularization parameter denoted by ฯ„ in\n the [paper](https://huggingface.co/papers/2310.12036).\n label_smoothing (`float`, *optional*, defaults to `0.0`):\n Label smoothing factor. This argument is required if you want to use the default data collator.\n loss_type (`str`, *optional*, defaults to `\"sigmoid\"`):\n Type of loss to use. Possible values are:\n\n - `\"sigmoid\"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.\n - `\"hinge\"`: hinge loss on the normalized likelihood from the\n [SLiC](https://huggingface.co/papers/2305.10425) paper.\n - `\"ipo\"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.\n - `\"simpo\"`: SimPO loss from the [SimPO](https://huggingface.co/papers/2405.14734) paper.\n - `\"alphapo\"`: AlphaPO loss from the [AlphaPO](https://huggingface.co/papers/2501.03884) paper. This\n automatically sets `loss_type=\"simpo\"` and `cpo_alpha=0.0`.\n\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model.\n cpo_alpha (`float`, *optional*, defaults to `1.0`):\n Weight of the BC regularizer in CPO training.\n simpo_gamma (`float`, *optional*, defaults to `0.5`):\n Target reward margin for the SimPO loss, used only when the `loss_type=\"simpo\"`.\n alpha (`float`, *optional*, defaults to `0.0`):\n Alpha parameter that controls reward function shape across all loss types. When alpha=0 (default), uses\n standard log probability rewards. When `alpha != 0`, applies AlphaPO transformation: `r = (1 - p^(-alpha))\n / alpha` from the [AlphaPO paper](https://huggingface.co/papers/2501.03884). This parameter works with all\n loss types.\n truncation_mode (`str`,*optional*, defaults to `\"keep_end\"`):\n Truncation mode to use when the prompt is too long. Possible values are `\"keep_end\"` or `\"keep_start\"`.\n This argument is required if you want to use the default data collator.\n generate_during_eval (`bool`, *optional*, defaults to `False`):\n If `True`, generates and logs completions from the model to W&B or Comet during evaluation.\n is_encoder_decoder (`bool`, *optional*):\n When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,\n you need to specify if the model returned by the callable is an encoder-decoder model.\n model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a\n string.\n dataset_num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `1e-6` instead of `5e-5`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-6,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n max_length: int | None = field(\n default=1024,\n metadata={\"help\": \"Maximum length of the sequences (prompt + completion) in the batch.\"},\n )\n max_completion_length: int | None = field(\n default=None,\n metadata={\n \"help\": \"Maximum length of the completion. This argument is required if you want to use the default data \"\n \"collator and your model is an encoder-decoder.\"\n },\n )\n beta: float = field(\n default=0.1,\n metadata={\n \"help\": \"Parameter controlling the deviation from the reference model. Higher ฮฒ means less deviation from \"\n \"the reference model.\"\n },\n )\n label_smoothing: float = field(\n default=0.0,\n metadata={\"help\": \"Label smoothing factor.\"},\n )\n loss_type: str = field(\n default=\"sigmoid\",\n metadata={\n \"help\": \"Type of loss to use.\",\n \"choices\": [\"sigmoid\", \"hinge\", \"ipo\", \"simpo\", \"alphapo\"],\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropout in the model.\"},\n )\n cpo_alpha: float = field(\n default=1.0,\n metadata={\"help\": \"Weight of the BC regularizer in CPO training.\"},\n )\n simpo_gamma: float = field(\n default=0.5,\n metadata={\"help\": \"Target reward margin for the SimPO loss, used only when the `loss_type='simpo'`.\"},\n )\n alpha: float = field(\n default=0.0,\n metadata={\n \"help\": \"Alpha parameter that controls reward function shape across all loss types. When alpha=0 \"\n \"(default), uses standard log probability rewards. When `alpha != 0`, applies AlphaPO transformation: \"\n \"`r = (1 - p^(-alpha)) / alpha` from the AlphaPO paper. This parameter works with all loss types.\"\n },\n )\n truncation_mode: str = field(\n default=\"keep_end\",\n metadata={\n \"help\": \"Truncation mode to use when the prompt is too long.\",\n \"choices\": [\"keep_end\", \"keep_start\"],\n },\n )\n generate_during_eval: bool = field(\n default=False,\n metadata={\"help\": \"If `True`, generates and logs completions from the model to W&B during evaluation.\"},\n )\n is_encoder_decoder: bool | None = field(\n default=None,\n metadata={\"help\": \"Whether the model is an encoder-decoder model.\"},\n )\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model \"\n \"from a string.\"\n },\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes to use for processing the dataset.\"},\n )\n\n def __post_init__(self):\n # Syntactic sugar for AlphaPO: set loss_type to \"simpo\" and cpo_alpha to 0.0\n if self.loss_type == \"alphapo\":\n self.loss_type = \"simpo\"\n self.cpo_alpha = 0.0\n\n super().__post_init__()\n" }, { "path": "trl/experimental/cpo/cpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport inspect\nimport random\nimport textwrap\nfrom collections import defaultdict\nfrom collections.abc import Callable\nfrom contextlib import nullcontext\nfrom pathlib import Path\nfrom typing import Any, Literal\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport transformers\nfrom accelerate import PartialState, logging\nfrom datasets import Dataset\nfrom packaging.version import Version\nfrom torch import autocast\nfrom torch.utils.data import DataLoader\nfrom transformers import (\n AutoModelForCausalLM,\n BaseImageProcessor,\n DataCollator,\n FeatureExtractionMixin,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n is_comet_available,\n is_wandb_available,\n)\nfrom transformers.trainer_utils import EvalLoopOutput\nfrom transformers.utils import is_peft_available, is_torch_fx_proxy\n\nfrom ...data_utils import maybe_apply_chat_template, maybe_extract_prompt\nfrom ...trainer.base_trainer import _BaseTrainer\nfrom ...trainer.utils import disable_dropout_in_model, log_table_to_comet_experiment, selective_log_softmax\nfrom ..utils import (\n DPODataCollatorWithPadding,\n add_bos_token_if_needed,\n add_eos_token_if_needed,\n pad_to_length,\n peft_module_casting_to_bf16,\n)\nfrom .cpo_config import CPOConfig\n\n\nif is_peft_available():\n from peft import PeftModel, get_peft_model, prepare_model_for_kbit_training\n\n\nif is_wandb_available():\n import wandb\n\n\nlogger = logging.get_logger(__name__)\n\n\nclass CPOTrainer(_BaseTrainer):\n r\"\"\"\n Initialize CPOTrainer.\n\n Args:\n model ([`~transformers.PreTrainedModel`]):\n The model to train, preferably an [`~transformers.AutoModelForSequenceClassification`].\n args ([`experimental.cpo.CPOConfig`]):\n The CPO config arguments to use for training.\n data_collator ([`~transformers.DataCollator`]):\n The data collator to use for training. If None is specified, the default data collator\n ([`experimental.utils.DPODataCollatorWithPadding`]) will be used which will pad the sequences to the\n maximum length of the sequences in the batch, given a dataset of paired sequences.\n train_dataset ([`~datasets.Dataset`]):\n The dataset to use for training.\n eval_dataset ([`~datasets.Dataset`]):\n The dataset to use for evaluation.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. If provided, will be used to automatically process the inputs\n for the model, and it will be saved along the model to make it easier to rerun an interrupted training or\n reuse the fine-tuned model.\n model_init (`Callable[[], transformers.PreTrainedModel]`):\n The model initializer to use for training. If None is specified, the default model initializer will be\n used.\n callbacks (`list[transformers.TrainerCallback]`):\n The callbacks to use for training.\n optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):\n The optimizer and scheduler to use for training.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):\n The function to use to preprocess the logits before computing the metrics.\n peft_config (`dict`, defaults to `None`):\n The PEFT configuration to use for training. If you pass a PEFT configuration, the model will be wrapped in\n a PEFT model.\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to\n metric values.\n \"\"\"\n\n _tag_names = [\"trl\", \"cpo\"]\n _name = \"CPO\"\n _paper = {\n \"title\": \"Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation\",\n \"id\": \"2401.08417\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @inproceedings{xu2024contrastive,\n title = {{Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation}},\n author = {Haoran Xu and Amr Sharaf and Yunmo Chen and Weiting Tan and Lingfeng Shen and Benjamin Van Durme and Kenton Murray and Young Jin Kim},\n year = 2024,\n booktitle = {Forty-first International Conference on Machine Learning, {ICML} 2024, Vienna, Austria, July 21-27, 2024},\n publisher = {OpenReview.net},\n url = {https://openreview.net/forum?id=51iwkioZpn}\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module | str | None = None,\n args: CPOConfig | None = None,\n data_collator: DataCollator | None = None,\n train_dataset: Dataset | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n processing_class: PreTrainedTokenizerBase\n | BaseImageProcessor\n | FeatureExtractionMixin\n | ProcessorMixin\n | None = None,\n model_init: Callable[[], PreTrainedModel] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n peft_config: dict | None = None,\n compute_metrics: Callable[[EvalLoopOutput], dict] | None = None,\n ):\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n\n if args.model_init_kwargs is None:\n model_init_kwargs = {}\n elif not isinstance(model, str):\n raise ValueError(\"You passed model_kwargs to the CPOTrainer. But your model is already instantiated.\")\n else:\n model_init_kwargs = args.model_init_kwargs\n dtype = model_init_kwargs.get(\"dtype\", \"auto\")\n if dtype is not None:\n # Convert to `torch.dtype` if an str is passed\n if isinstance(dtype, str) and dtype != \"auto\":\n dtype = getattr(torch, dtype)\n if dtype != \"auto\" and not isinstance(dtype, torch.dtype):\n raise ValueError(\n f\"Invalid `dtype` passed to the CPOConfig. Expected a string with either `torch.dtype` or 'auto', but got {dtype}.\"\n )\n model_init_kwargs[\"dtype\"] = dtype\n model_init_kwargs[\"device_map\"] = model_init_kwargs.get(\"device_map\", \"auto\")\n\n if isinstance(model, str):\n model = AutoModelForCausalLM.from_pretrained(model, **model_init_kwargs)\n\n # Initialize this variable to False. This helps tracking the case when `peft_module_casting_to_bf16`\n # has been called in order to properly call autocast if needed.\n self._peft_has_been_casted_to_bf16 = False\n\n if not is_peft_available() and peft_config is not None:\n raise ValueError(\n \"PEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT models\"\n )\n elif is_peft_available() and peft_config is not None:\n if isinstance(model, PeftModel):\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first \"\n \"merge and unload the existing adapter, save the resulting base model, and then pass that base \"\n \"model along with the new `peft_config` to the trainer.\"\n )\n\n if getattr(model, \"is_loaded_in_8bit\", False) or getattr(model, \"is_loaded_in_4bit\", False):\n _support_gc_kwargs = hasattr(\n args, \"gradient_checkpointing_kwargs\"\n ) and \"gradient_checkpointing_kwargs\" in list(\n inspect.signature(prepare_model_for_kbit_training).parameters\n )\n\n prepare_model_kwargs = {\"use_gradient_checkpointing\": args.gradient_checkpointing}\n\n if _support_gc_kwargs:\n prepare_model_kwargs[\"gradient_checkpointing_kwargs\"] = args.gradient_checkpointing_kwargs\n\n model = prepare_model_for_kbit_training(model, **prepare_model_kwargs)\n elif args.gradient_checkpointing:\n # For backward compatibility with older versions of transformers\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n # get peft model with the given config\n model = get_peft_model(model, peft_config)\n if args.bf16 and getattr(model, \"is_loaded_in_4bit\", False):\n peft_module_casting_to_bf16(model)\n # If args.bf16 we need to explicitly call `generate` with torch amp autocast context manager\n self._peft_has_been_casted_to_bf16 = True\n\n # For models that use gradient_checkpointing, we need to attach a hook that enables input\n # to explicitly have `requires_grad=True`, otherwise training will either silently\n # fail or completely fail.\n elif args.gradient_checkpointing:\n # For backward compatibility with older versions of transformers\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n if args.generate_during_eval and not (is_wandb_available() or is_comet_available()):\n raise ValueError(\n \"`generate_during_eval=True` requires Weights and Biases or Comet to be installed.\"\n \" Please install `wandb` or `comet-ml` to resolve.\"\n )\n\n if model is not None:\n self.is_encoder_decoder = model.config.is_encoder_decoder\n elif args.is_encoder_decoder is None:\n raise ValueError(\"When no model is provided, you need to pass the parameter is_encoder_decoder.\")\n else:\n self.is_encoder_decoder = args.is_encoder_decoder\n\n if self.is_encoder_decoder:\n self.decoder_start_token_id = model.config.decoder_start_token_id\n self.pad_token_id = model.config.pad_token_id\n\n if processing_class is None:\n raise ValueError(\"processing_class must be specified to tokenize a CPO dataset.\")\n if args.max_length is None:\n logger.warning(\n \"`max_length` is not set in the CPOConfig's init\"\n \" it will default to `512` by default, but you should do it yourself in the future.\",\n )\n max_length = 512\n else:\n max_length = args.max_length\n\n if args.max_completion_length is None and self.is_encoder_decoder:\n logger.warning(\n \"When using an encoder decoder architecture, you should set `max_completion_length` in the CPOConfig's init\"\n \" it will default to `128` by default, but you should do it yourself in the future.\",\n )\n max_completion_length = 128\n else:\n max_completion_length = args.max_completion_length\n\n if data_collator is None:\n data_collator = DPODataCollatorWithPadding(\n pad_token_id=processing_class.pad_token_id,\n is_encoder_decoder=self.is_encoder_decoder,\n )\n\n if args.remove_unused_columns:\n args.remove_unused_columns = False\n # warn users\n logger.warning(\n \"When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments\"\n \" we have set it for you, but you should do it yourself in the future.\",\n )\n\n self.use_dpo_data_collator = True\n else:\n self.use_dpo_data_collator = False\n\n # Disable dropout in the model\n if args.disable_dropout:\n disable_dropout_in_model(model)\n\n self.max_length = max_length\n self.generate_during_eval = args.generate_during_eval\n self.truncation_mode = args.truncation_mode\n self.max_completion_length = max_completion_length\n self.processing_class = processing_class\n\n if processing_class.pad_token is None:\n processing_class.pad_token = processing_class.eos_token\n self.pad_token_id = processing_class.pad_token_id\n\n if args.loss_type in [\"hinge\", \"ipo\"] and args.label_smoothing > 0:\n logger.warning(\n f\"You are using the {args.loss_type} loss type that does not support label smoothing. The \"\n \"`label_smoothing` parameter will be ignored. Set `label_smoothing` to `0.0` to remove this warning.\",\n )\n if args.loss_type == \"kto_pair\":\n raise ValueError(\"Support for kto_pair has been removed in CPOTrainer. Please use KTOTrainer.\")\n\n self.beta = args.beta\n self.label_smoothing = args.label_smoothing\n self.loss_type = args.loss_type\n self.cpo_alpha = args.cpo_alpha\n self.aux_loss_enabled = getattr(model.config, \"output_router_logits\", False)\n self.aux_loss_coef = getattr(model.config, \"router_aux_loss_coef\", 0.0)\n if self.aux_loss_enabled and self.aux_loss_coef == 0.0:\n logger.warning(\n \"You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to \"\n \"`0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value \"\n \"greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary \"\n \"loss.\",\n )\n\n if args.loss_type == \"simpo\":\n self.simpo_gamma = args.simpo_gamma\n\n # AlphaPO parameter for reward shaping\n self.alpha = args.alpha\n\n self._stored_metrics = defaultdict(lambda: defaultdict(list))\n\n # Compute that only on the main process for faster data processing.\n # see: https://github.com/huggingface/trl/pull/1255\n with PartialState().main_process_first():\n # Extract the prompt if needed, and apply the chat template if needed\n train_dataset = train_dataset.map(maybe_extract_prompt, num_proc=args.dataset_num_proc)\n train_dataset = train_dataset.map(\n maybe_apply_chat_template, fn_kwargs={\"tokenizer\": processing_class}, num_proc=args.dataset_num_proc\n )\n if eval_dataset is not None:\n eval_dataset = eval_dataset.map(maybe_extract_prompt, num_proc=args.dataset_num_proc)\n eval_dataset = eval_dataset.map(\n maybe_apply_chat_template,\n fn_kwargs={\"tokenizer\": processing_class},\n num_proc=args.dataset_num_proc,\n )\n\n # tokenize the dataset\n train_dataset = train_dataset.map(self.tokenize_row, num_proc=args.dataset_num_proc)\n if eval_dataset is not None:\n eval_dataset = eval_dataset.map(self.tokenize_row, num_proc=args.dataset_num_proc)\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n model_init=model_init,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n # Gradient accumulation requires scaled loss. Normally, loss scaling in the parent class depends on whether the\n # model accepts loss-related kwargs. Since we compute our own loss, this check is irrelevant. We set\n # self.model_accepts_loss_kwargs to False to enable scaling.\n self.model_accepts_loss_kwargs = False\n\n # Add tags for models that have been loaded with the correct transformers version\n if hasattr(self.model, \"add_model_tags\"):\n self.model.add_model_tags(self._tag_names)\n\n if not hasattr(self, \"accelerator\"):\n raise AttributeError(\n \"Your `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.\"\n )\n\n def build_tokenized_answer(self, prompt, answer):\n \"\"\"\n Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a +\n b)[len(enc(a)):]`. Reference:\n https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257\n \"\"\"\n\n full_tokenized = self.processing_class(prompt + answer, add_special_tokens=False)\n prompt_input_ids = self.processing_class(prompt, add_special_tokens=False)[\"input_ids\"]\n\n answer_input_ids = full_tokenized[\"input_ids\"][len(prompt_input_ids) :]\n answer_attention_mask = full_tokenized[\"attention_mask\"][len(prompt_input_ids) :]\n\n # Concat tokens to form `enc(a) + enc(a + b)[len(enc(a)):]`\n full_concat_input_ids = np.concatenate([prompt_input_ids, answer_input_ids])\n\n # Prepare input tokens for token by token comparison\n full_input_ids = np.array(full_tokenized[\"input_ids\"])\n\n if len(full_input_ids) != len(full_concat_input_ids):\n raise ValueError(\"Prompt input ids and answer input ids should have the same length.\")\n\n # On some tokenizers, like Llama-2 tokenizer, there are occasions where tokens\n # can be merged together when tokenizing prompt+answer. This could result\n # on the last token from the prompt being different when tokenized on its own\n # vs when done as prompt+answer.\n response_token_ids_start_idx = len(prompt_input_ids)\n\n # If tokenized prompt is different than both prompt+answer, then it means the\n # last token has changed due to merging.\n if prompt_input_ids != full_tokenized[\"input_ids\"][:response_token_ids_start_idx]:\n response_token_ids_start_idx -= 1\n\n prompt_input_ids = full_tokenized[\"input_ids\"][:response_token_ids_start_idx]\n prompt_attention_mask = full_tokenized[\"attention_mask\"][:response_token_ids_start_idx]\n\n if len(prompt_input_ids) != len(prompt_attention_mask):\n raise ValueError(\"Prompt input ids and attention mask should have the same length.\")\n\n answer_input_ids = full_tokenized[\"input_ids\"][response_token_ids_start_idx:]\n answer_attention_mask = full_tokenized[\"attention_mask\"][response_token_ids_start_idx:]\n\n return dict(\n prompt_input_ids=prompt_input_ids,\n prompt_attention_mask=prompt_attention_mask,\n input_ids=answer_input_ids,\n attention_mask=answer_attention_mask,\n )\n\n def tokenize_row(self, feature, model: PreTrainedModel | nn.Module | None = None) -> dict:\n \"\"\"Tokenize a single row from a CPO specific dataset.\n\n At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +\n chosen or prompt + rejected responses is/are too long. First we truncate the prompt; if we're still too long,\n we truncate the chosen/rejected.\n\n We also create the labels for the chosen/rejected responses, which are of length equal to the sum of the length\n of the prompt and the chosen/rejected response, with `-100` for the prompt tokens.\n \"\"\"\n batch = {}\n prompt = feature[\"prompt\"]\n chosen = feature[\"chosen\"]\n rejected = feature[\"rejected\"]\n\n if not self.is_encoder_decoder:\n # Check issues below for more details\n # 1. https://github.com/huggingface/trl/issues/907\n # 2. https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257\n # 3. https://github.com/LianjiaTech/BELLE/issues/337\n\n if not isinstance(prompt, str):\n raise ValueError(f\"prompt should be an str but got {type(prompt)}\")\n prompt_tokens = self.processing_class(prompt, add_special_tokens=False)\n prompt_tokens = {f\"prompt_{k}\": v for k, v in prompt_tokens.items()}\n\n if not isinstance(chosen, str):\n raise ValueError(f\"chosen should be an str but got {type(chosen)}\")\n chosen_tokens = self.build_tokenized_answer(prompt, chosen)\n\n if not isinstance(rejected, str):\n raise ValueError(f\"rejected should be an str but got {type(rejected)}\")\n rejected_tokens = self.build_tokenized_answer(prompt, rejected)\n\n # Last prompt token might get merged by tokenizer and\n # it should not be included for generation if that happens\n prompt_len_input_ids = len(prompt_tokens[\"prompt_input_ids\"])\n\n chosen_prompt_len_input_ids = len(chosen_tokens[\"prompt_input_ids\"])\n rejected_prompt_len_input_ids = len(rejected_tokens[\"prompt_input_ids\"])\n prompt_len_input_ids = min(chosen_prompt_len_input_ids, rejected_prompt_len_input_ids)\n\n for k, v in prompt_tokens.items():\n prompt_tokens[k] = v[:prompt_len_input_ids]\n\n # Make sure prompts only have one different token at most an\n # and length only differs by 1 at most\n num_diff_tokens = sum(\n a != b\n for a, b in zip(chosen_tokens[\"prompt_input_ids\"], rejected_tokens[\"prompt_input_ids\"], strict=False)\n )\n num_diff_len = abs(chosen_prompt_len_input_ids - rejected_prompt_len_input_ids)\n if num_diff_tokens > 1 or num_diff_len > 1:\n raise ValueError(\n \"Chosen and rejected prompt_input_ids might only differ on the \"\n \"last token due to tokenizer merge ops.\"\n )\n\n # add BOS token to head of prompt. Avoid adding if it's already there\n prompt_tokens, chosen_tokens, rejected_tokens = add_bos_token_if_needed(\n self.processing_class.bos_token_id,\n prompt_len_input_ids,\n prompt_tokens,\n chosen_prompt_len_input_ids,\n chosen_tokens,\n rejected_prompt_len_input_ids,\n rejected_tokens,\n )\n\n # add EOS token to end of answer. Avoid adding if it's already there\n chosen_tokens, rejected_tokens = add_eos_token_if_needed(\n self.processing_class.eos_token_id, chosen_tokens, rejected_tokens\n )\n\n longer_response_length = max(len(chosen_tokens[\"input_ids\"]), len(rejected_tokens[\"input_ids\"]))\n\n # if combined sequence is too long, truncate the response\n for answer_tokens in [chosen_tokens, rejected_tokens]:\n if len(answer_tokens[\"prompt_input_ids\"]) + longer_response_length > self.max_length:\n for k in [\"input_ids\", \"attention_mask\"]:\n answer_tokens[k] = answer_tokens[k][: self.max_length - longer_response_length]\n\n # Create labels\n chosen_sequence_tokens = {\n k: chosen_tokens[f\"prompt_{k}\"] + chosen_tokens[k] for k in [\"input_ids\", \"attention_mask\"]\n }\n rejected_sequence_tokens = {\n k: rejected_tokens[f\"prompt_{k}\"] + rejected_tokens[k] for k in [\"input_ids\", \"attention_mask\"]\n }\n chosen_sequence_tokens[\"labels\"] = chosen_sequence_tokens[\"input_ids\"][:]\n chosen_sequence_tokens[\"labels\"][: len(chosen_tokens[\"prompt_input_ids\"])] = [-100] * len(\n chosen_tokens[\"prompt_input_ids\"]\n )\n rejected_sequence_tokens[\"labels\"] = rejected_sequence_tokens[\"input_ids\"][:]\n rejected_sequence_tokens[\"labels\"][: len(rejected_tokens[\"prompt_input_ids\"])] = [-100] * len(\n rejected_tokens[\"prompt_input_ids\"]\n )\n\n for k, toks in {\n \"chosen_\": chosen_sequence_tokens,\n \"rejected_\": rejected_sequence_tokens,\n \"\": prompt_tokens,\n }.items():\n for type_key, tokens in toks.items():\n if type_key == \"token_type_ids\":\n continue\n batch[f\"{k}{type_key}\"] = tokens\n\n else:\n chosen_tokens = self.processing_class(\n chosen, truncation=True, max_length=self.max_completion_length, add_special_tokens=True\n )\n rejected_tokens = self.processing_class(\n rejected, truncation=True, max_length=self.max_completion_length, add_special_tokens=True\n )\n prompt_tokens = self.processing_class(prompt, add_special_tokens=True)\n\n batch[\"chosen_labels\"] = chosen_tokens[\"input_ids\"]\n batch[\"rejected_labels\"] = rejected_tokens[\"input_ids\"]\n batch[\"prompt_input_ids\"] = prompt_tokens[\"input_ids\"]\n batch[\"prompt_attention_mask\"] = prompt_tokens[\"attention_mask\"]\n\n if model is not None and hasattr(model, \"prepare_decoder_input_ids_from_labels\"):\n batch[\"rejected_decoder_input_ids\"] = model.prepare_decoder_input_ids_from_labels(\n labels=torch.tensor(batch[\"rejected_labels\"])\n )\n batch[\"chosen_decoder_input_ids\"] = model.prepare_decoder_input_ids_from_labels(\n labels=torch.tensor(batch[\"chosen_labels\"])\n )\n\n return batch\n\n @staticmethod\n def concatenated_inputs(\n batch: dict[str, list | torch.LongTensor],\n is_encoder_decoder: bool = False,\n padding_value: int = 0,\n device: torch.device | None = None,\n ) -> dict[str, torch.LongTensor]:\n \"\"\"Concatenate the chosen and rejected inputs into a single tensor.\n\n Args:\n batch:\n A batch of data. Must contain the keys 'chosen_input_ids' and 'rejected_input_ids', which are tensors\n of shape (batch_size, sequence_length).\n is_encoder_decoder:\n Whether the model is an encoder-decoder model.\n padding_value:\n The padding value to use for the concatenated inputs_ids.\n device:\n The device for the concatenated inputs.\n\n Returns:\n A dictionary containing the concatenated inputs under the key 'concatenated_input_ids'.\n \"\"\"\n concatenated_batch = {}\n\n if is_encoder_decoder:\n max_length = max(batch[\"chosen_labels\"].shape[1], batch[\"rejected_labels\"].shape[1])\n else:\n max_length = max(batch[\"chosen_input_ids\"].shape[1], batch[\"rejected_input_ids\"].shape[1])\n\n for k in batch:\n if k.startswith(\"chosen\") and isinstance(batch[k], torch.Tensor):\n if \"labels\" in k or is_encoder_decoder:\n pad_value = -100\n elif k.endswith(\"_input_ids\"):\n pad_value = padding_value\n elif k.endswith(\"_attention_mask\"):\n pad_value = 0\n concatenated_key = k.replace(\"chosen\", \"concatenated\")\n concatenated_batch[concatenated_key] = pad_to_length(batch[k], max_length, pad_value=pad_value)\n for k in batch:\n if k.startswith(\"rejected\") and isinstance(batch[k], torch.Tensor):\n if \"labels\" in k or is_encoder_decoder:\n pad_value = -100\n elif k.endswith(\"_input_ids\"):\n pad_value = padding_value\n elif k.endswith(\"_attention_mask\"):\n pad_value = 0\n concatenated_key = k.replace(\"rejected\", \"concatenated\")\n concatenated_batch[concatenated_key] = torch.cat(\n (\n concatenated_batch[concatenated_key],\n pad_to_length(batch[k], max_length, pad_value=pad_value),\n ),\n dim=0,\n ).to(device=device)\n\n if is_encoder_decoder:\n concatenated_batch[\"concatenated_input_ids\"] = batch[\"prompt_input_ids\"].repeat(2, 1).to(device=device)\n concatenated_batch[\"concatenated_attention_mask\"] = (\n batch[\"prompt_attention_mask\"].repeat(2, 1).to(device=device)\n )\n\n return concatenated_batch\n\n def cpo_loss(\n self,\n policy_chosen_logps: torch.FloatTensor,\n policy_rejected_logps: torch.FloatTensor,\n ) -> tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n \"\"\"Compute the CPO loss for a batch of policy and reference model log probabilities.\n\n Args:\n policy_chosen_logps:\n Log probabilities of the policy model for the chosen responses. Shape: (batch_size,)\n policy_rejected_logps:\n Log probabilities of the policy model for the rejected responses. Shape: (batch_size,)\n\n Returns:\n A tuple of three tensors: (losses, chosen_rewards, rejected_rewards). The losses tensor contains the CPO\n loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for\n the chosen and rejected responses, respectively.\n \"\"\"\n # Apply AlphaPO reward transformation if alpha != 0\n if self.alpha != 0.0:\n # Compute probabilities\n chosen_probs = torch.exp(policy_chosen_logps)\n rejected_probs = torch.exp(policy_rejected_logps)\n\n # Apply AlphaPO transformation: r = (1 - p^(-alpha)) / alpha\n policy_chosen_rewards = (1 - chosen_probs.pow(-self.alpha)) / self.alpha\n policy_rejected_rewards = (1 - rejected_probs.pow(-self.alpha)) / self.alpha\n\n logits = (policy_chosen_rewards - policy_rejected_rewards).to(self.accelerator.device)\n else:\n # Standard log probability rewards when alpha = 0\n logits = (policy_chosen_logps - policy_rejected_logps).to(self.accelerator.device)\n\n # The beta is a temperature parameter for the CPO loss, typically something in the range of 0.1 to 0.5.\n # We ignore the reference model as beta -> 0. The label_smoothing parameter encodes our uncertainty about the labels and\n # calculates a conservative CPO loss.\n\n if self.loss_type == \"simpo\":\n gamma_logratios = self.simpo_gamma / self.beta\n logits = logits - gamma_logratios\n # This reduces to Equation 3 from the CPO paper when label_smoothing -> 0.\n losses = (\n -F.logsigmoid(self.beta * logits) * (1 - self.label_smoothing)\n - F.logsigmoid(-self.beta * logits) * self.label_smoothing\n )\n elif self.loss_type == \"sigmoid\":\n # This reduces to Equation 3 from the CPO paper when label_smoothing -> 0.\n losses = (\n -F.logsigmoid(self.beta * logits) * (1 - self.label_smoothing)\n - F.logsigmoid(-self.beta * logits) * self.label_smoothing\n )\n elif self.loss_type == \"hinge\":\n losses = torch.relu(1 - self.beta * logits)\n elif self.loss_type == \"ipo\":\n # eqn (17) of the paper where beta is the regularization parameter for the IPO loss, denoted by tau in the paper.\n losses = (logits - 1 / (2 * self.beta)) ** 2\n else:\n raise ValueError(\n f\"Unknown loss type: {self.loss_type}. Should be one of ['sigmoid', 'hinge', 'ipo', 'simpo']\"\n )\n\n # Calculate rewards for logging\n if self.alpha != 0.0:\n # When using AlphaPO transformation, use the transformed rewards\n chosen_rewards = self.beta * policy_chosen_rewards.to(self.accelerator.device).detach()\n rejected_rewards = self.beta * policy_rejected_rewards.to(self.accelerator.device).detach()\n else:\n # Standard log probability rewards\n chosen_rewards = self.beta * (policy_chosen_logps.to(self.accelerator.device)).detach()\n rejected_rewards = self.beta * (policy_rejected_logps.to(self.accelerator.device)).detach()\n\n return losses, chosen_rewards, rejected_rewards\n\n @staticmethod\n def get_batch_logps(\n logits: torch.FloatTensor,\n labels: torch.LongTensor,\n average_log_prob: bool = False,\n is_encoder_decoder: bool = False,\n ) -> torch.FloatTensor:\n \"\"\"Compute the log probabilities of the given labels under the given logits.\n\n Args:\n logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)\n labels:\n Labels for which to compute the log probabilities. Label tokens with a value of `-100` are ignored.\n Shape: (batch_size, sequence_length)\n average_log_prob:\n If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the\n log probabilities of the (non-masked) tokens.\n is_encoder_decoder: Whether the model is an encoder-decoder model.\n\n Returns:\n A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the\n given logits.\n \"\"\"\n if logits.shape[:-1] != labels.shape:\n raise ValueError(\"Logits (batch and sequence length dim) and labels must have the same shape.\")\n\n if not is_encoder_decoder:\n labels = labels[:, 1:].clone()\n logits = logits[:, :-1, :]\n loss_mask = labels != -100\n\n # dummy token; we'll ignore the losses on these tokens later\n labels[labels == -100] = 0\n\n per_token_logps = selective_log_softmax(logits, labels)\n\n if average_log_prob:\n return (per_token_logps * loss_mask).sum(-1) / loss_mask.sum(-1)\n else:\n return (per_token_logps * loss_mask).sum(-1)\n\n def concatenated_forward(\n self, model: nn.Module, batch: dict[str, list | torch.LongTensor]\n ) -> tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n \"\"\"Run the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.\n\n We do this to avoid doing two forward passes, because it's faster for FSDP.\n \"\"\"\n concatenated_batch = self.concatenated_inputs(\n batch,\n is_encoder_decoder=self.is_encoder_decoder,\n padding_value=self.pad_token_id,\n device=self.accelerator.device,\n )\n len_chosen = batch[\"chosen_labels\"].shape[0]\n\n model_kwargs = (\n {\n \"decoder_input_ids\": self._shift_right(concatenated_batch[\"concatenated_labels\"]),\n }\n if self.is_encoder_decoder\n else {}\n )\n\n if self.aux_loss_enabled:\n model_kwargs[\"output_router_logits\"] = True\n\n outputs = model(\n concatenated_batch[\"concatenated_input_ids\"],\n attention_mask=concatenated_batch[\"concatenated_attention_mask\"],\n use_cache=False,\n **model_kwargs,\n )\n all_logits = outputs.logits\n\n def cross_entropy_loss(logits, labels):\n if not self.is_encoder_decoder:\n # Shift so that tokens < n predict n\n logits = logits[..., :-1, :].contiguous()\n labels = labels[..., 1:].contiguous()\n # Flatten the tokens\n loss_fct = nn.CrossEntropyLoss()\n logits = logits.view(-1, logits.shape[-1])\n labels = labels.view(-1)\n # Enable model parallelism\n labels = labels.to(logits.device)\n loss = loss_fct(logits, labels)\n return loss\n\n labels = concatenated_batch[\"concatenated_labels\"].clone()\n\n if self.cpo_alpha == 0:\n nll_loss = torch.tensor(0.0).to(self.accelerator.device)\n else:\n nll_loss = cross_entropy_loss(all_logits[:len_chosen], labels[:len_chosen])\n\n all_logps = self.get_batch_logps(\n all_logits,\n concatenated_batch[\"concatenated_labels\"],\n average_log_prob=self.loss_type in [\"ipo\", \"simpo\"],\n is_encoder_decoder=self.is_encoder_decoder,\n )\n\n chosen_logps = all_logps[:len_chosen]\n rejected_logps = all_logps[len_chosen:]\n\n chosen_logits = all_logits[:len_chosen]\n rejected_logits = all_logits[len_chosen:]\n\n if self.aux_loss_enabled:\n return (chosen_logps, rejected_logps, chosen_logits, rejected_logits, nll_loss, outputs.aux_loss)\n\n return (chosen_logps, rejected_logps, chosen_logits, rejected_logits, nll_loss)\n\n def get_batch_loss_metrics(\n self,\n model,\n batch: dict[str, list | torch.LongTensor],\n train_eval: Literal[\"train\", \"eval\"] = \"train\",\n ):\n \"\"\"Compute the CPO loss and other metrics for the given batch of inputs for train or test.\"\"\"\n metrics = {}\n\n forward_output = self.concatenated_forward(model, batch)\n (\n policy_chosen_logps,\n policy_rejected_logps,\n policy_chosen_logits,\n policy_rejected_logits,\n policy_nll_loss,\n ) = forward_output[:5]\n if self.aux_loss_enabled:\n aux_loss = forward_output[5]\n\n losses, chosen_rewards, rejected_rewards = self.cpo_loss(\n policy_chosen_logps,\n policy_rejected_logps,\n )\n\n loss = losses.mean() + self.cpo_alpha * policy_nll_loss\n reward_accuracies = (chosen_rewards > rejected_rewards).float()\n\n prefix = \"eval_\" if train_eval == \"eval\" else \"\"\n metrics[f\"{prefix}rewards/chosen\"] = self.accelerator.gather_for_metrics(chosen_rewards).mean().item()\n metrics[f\"{prefix}rewards/rejected\"] = self.accelerator.gather_for_metrics(rejected_rewards).mean().item()\n metrics[f\"{prefix}rewards/accuracies\"] = self.accelerator.gather_for_metrics(reward_accuracies).mean().item()\n metrics[f\"{prefix}rewards/margins\"] = (\n self.accelerator.gather_for_metrics(chosen_rewards - rejected_rewards).mean().item()\n )\n metrics[f\"{prefix}logps/rejected\"] = (\n self.accelerator.gather_for_metrics(policy_rejected_logps).detach().mean().item()\n )\n metrics[f\"{prefix}logps/chosen\"] = (\n self.accelerator.gather_for_metrics(policy_chosen_logps).detach().mean().item()\n )\n metrics[f\"{prefix}logits/rejected\"] = (\n self.accelerator.gather_for_metrics(policy_rejected_logits.detach().mean()).mean().item()\n )\n metrics[f\"{prefix}logits/chosen\"] = (\n self.accelerator.gather_for_metrics(policy_chosen_logits.detach().mean()).mean().item()\n )\n metrics[f\"{prefix}nll_loss\"] = self.accelerator.gather_for_metrics(policy_nll_loss).detach().mean().item()\n\n if self.aux_loss_enabled:\n loss += self.aux_loss_coef * aux_loss\n\n return loss, metrics\n\n def compute_loss(\n self,\n model: PreTrainedModel | nn.Module,\n inputs: dict[str, torch.Tensor | Any],\n return_outputs=False,\n num_items_in_batch=None,\n ) -> torch.Tensor | tuple[torch.Tensor, dict[str, torch.Tensor]]:\n compute_loss_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n\n with compute_loss_context_manager:\n loss, metrics = self.get_batch_loss_metrics(model, inputs, train_eval=\"train\")\n\n # force log the metrics\n self.store_metrics(metrics, train_eval=\"train\")\n\n if return_outputs:\n return (loss, metrics)\n return loss\n\n def generate_from_model(self, model, batch: dict[str, torch.LongTensor]) -> str:\n \"\"\"Generate samples from the model and reference model for the given batch of inputs.\"\"\"\n\n # If one uses `generate_during_eval` with peft + bf16, we need to explicitly call generate with\n # the torch amp context manager as some hidden states are silently casted to full precision.\n generate_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n\n with generate_context_manager:\n policy_output = model.generate(\n input_ids=batch[\"prompt_input_ids\"],\n attention_mask=batch[\"prompt_attention_mask\"],\n max_length=self.max_length,\n do_sample=True,\n pad_token_id=self.processing_class.pad_token_id,\n )\n\n policy_output = pad_to_length(policy_output, self.max_length, self.processing_class.pad_token_id)\n policy_output_decoded = self.processing_class.batch_decode(policy_output, skip_special_tokens=True)\n\n return policy_output_decoded\n\n def prediction_step(\n self,\n model: PreTrainedModel | nn.Module,\n inputs: dict[str, torch.Tensor | Any],\n prediction_loss_only: bool,\n ignore_keys: list[str] | None = None,\n ):\n if ignore_keys is None:\n if hasattr(model, \"config\"):\n ignore_keys = getattr(model.config, \"keys_to_ignore_at_inference\", [])\n else:\n ignore_keys = []\n\n prediction_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n\n with torch.no_grad(), prediction_context_manager:\n loss, metrics = self.get_batch_loss_metrics(model, inputs, train_eval=\"eval\")\n\n # force log the metrics\n self.store_metrics(metrics, train_eval=\"eval\")\n\n if prediction_loss_only:\n return (loss.detach(), None, None)\n\n # logits for the chosen and rejected samples from model\n logits_dict = {\n \"eval_logits/chosen\": metrics[\"eval_logits/chosen\"],\n \"eval_logits/rejected\": metrics[\"eval_logits/rejected\"],\n }\n logits = [v for k, v in logits_dict.items() if k not in ignore_keys]\n logits = torch.tensor(logits, device=self.accelerator.device)\n labels = torch.zeros(logits.shape[0], device=self.accelerator.device)\n\n return (loss.detach(), logits, labels)\n\n def store_metrics(self, metrics: dict[str, float], train_eval: Literal[\"train\", \"eval\"] = \"train\") -> None:\n for key, value in metrics.items():\n self._stored_metrics[train_eval][key].append(value)\n\n def evaluation_loop(\n self,\n dataloader: DataLoader,\n description: str,\n prediction_loss_only: bool | None = None,\n ignore_keys: list[str] | None = None,\n metric_key_prefix: str = \"eval\",\n ) -> EvalLoopOutput:\n \"\"\"\n Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by\n `Trainer.evaluate()` and `Trainer.predict()`.\n\n Works both with or without labels.\n \"\"\"\n\n # Sample and save to game log if requested (for one batch to save time)\n if self.generate_during_eval:\n # Generate random indices within the range of the total number of samples\n num_samples = len(dataloader.dataset)\n random_indices = random.sample(range(num_samples), k=self.args.eval_batch_size)\n\n # Use dataloader.dataset.select to get the random batch without iterating over the DataLoader\n random_batch_dataset = dataloader.dataset.select(random_indices)\n random_batch = self.data_collator(random_batch_dataset)\n random_batch = self._prepare_inputs(random_batch)\n\n policy_output_decoded = self.generate_from_model(self.model, random_batch)\n\n table = pd.DataFrame(\n columns=[\"Prompt\", \"Policy\"],\n data=[\n [prompt, pol[len(prompt) :]]\n for prompt, pol in zip(random_batch[\"prompt\"], policy_output_decoded, strict=True)\n ],\n )\n if \"wandb\" in self.args.report_to:\n wandb.log({\"game_log\": wandb.Table(data=table)})\n\n if \"comet_ml\" in self.args.report_to:\n log_table_to_comet_experiment(\n name=\"game_log.csv\",\n table=table,\n )\n\n # Base evaluation\n initial_output = super().evaluation_loop(\n dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix\n )\n\n return initial_output\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n \"\"\"\n Log `logs` on the various objects watching training, including stored metrics.\n\n Args:\n logs (`dict[str, float]`):\n The values to log.\n start_time (`float`, *optional*):\n Start time of the training.\n \"\"\"\n # logs either has 'loss' or 'eval_loss'\n train_eval = \"train\" if \"loss\" in logs else \"eval\"\n # Add averaged stored metrics to logs\n for key, metrics in self._stored_metrics[train_eval].items():\n logs[key] = torch.tensor(metrics).mean().item()\n del self._stored_metrics[train_eval]\n return super().log(logs, start_time)\n\n def _shift_right(self, input_ids):\n if self.decoder_start_token_id is None:\n raise ValueError(\n \"model.config.decoder_start_token_id has to be defined. It is usually set to the pad_token_id.\"\n )\n\n # shift inputs to the right\n if is_torch_fx_proxy(input_ids):\n # Item assignment is not supported natively for proxies.\n shifted_input_ids = torch.full(input_ids.shape[:-1] + (1,), self.decoder_start_token_id)\n shifted_input_ids = torch.cat([shifted_input_ids, input_ids[..., :-1]], dim=-1)\n else:\n shifted_input_ids = input_ids.new_zeros(input_ids.shape)\n shifted_input_ids[..., 1:] = input_ids[..., :-1].clone()\n shifted_input_ids[..., 0] = self.decoder_start_token_id\n\n if self.pad_token_id is None:\n raise ValueError(\"model.config.pad_token_id has to be defined.\")\n # replace possible -100 values in labels by `pad_token_id`\n shifted_input_ids.masked_fill_(shifted_input_ids == -100, self.pad_token_id)\n\n return shifted_input_ids\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/experimental/dppo/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\nfrom .dppo_config import DPPOConfig\nfrom .dppo_trainer import DPPOTrainer\n" }, { "path": "trl/experimental/dppo/dppo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Literal\n\nfrom ...trainer.grpo_config import GRPOConfig\n\n\n@dataclass\nclass DPPOConfig(GRPOConfig):\n \"\"\"\n Configuration class for DPPOTrainer.\n\n DPPO (Divergence Proximal Policy Optimization) replaces PPO/GRPO's heuristic ratio-clipping with a principled\n trust region based on direct policy divergence estimates.\n\n Paper: \"Rethinking the Trust Region in LLM Reinforcement Learning\" (arXiv:2602.04879)\n\n Args:\n divergence_type (`Literal[\"binary_tv\", \"binary_kl\", \"topk_tv\", \"topk_kl\"]`, *optional*, defaults to `\"binary_tv\"`):\n Divergence approximation used for the trust-region mask. Binary variants use only per-token log-probs;\n top-K variants require storing top-K token IDs and log-probs during rollout generation plus full logits\n during training.\n\n divergence_topk (`int`, *optional*, defaults to `20`):\n K for top-K divergence approximations. Only used when `divergence_type` is `\"topk_tv\"` or `\"topk_kl\"`.\n\n clip_ratio_c (`float`, *optional*, defaults to `20.0`):\n Upper bound on the importance-sampling ratio for stability. The IS ratio is clamped to [0, clip_ratio_c].\n\n epsilon (`float`, inherited from GRPOConfig, default overridden to `0.15`):\n Divergence threshold ฮด_low. Tokens whose divergence exceeds this when the policy moves in the\n advantage-decreasing direction are masked. The paper recommends 0.15 for TV divergence\n and 0.05 for KL divergence.\n\n epsilon_high (`float`, inherited from GRPOConfig, default overridden to `0.15`):\n Divergence threshold ฮด_high. Tokens whose divergence exceeds this when the policy moves in the\n advantage-increasing direction are masked. The paper recommends 0.15 for TV divergence\n and 0.05 for KL divergence.\n \"\"\"\n\n divergence_type: Literal[\"binary_tv\", \"binary_kl\", \"topk_tv\", \"topk_kl\"] = field(\n default=\"binary_tv\",\n metadata={\n \"help\": \"Divergence approximation used for the trust-region mask. Binary variants use only per-token \"\n \"log-probs; top-K variants require storing top-K token IDs and log-probs during rollout generation plus \"\n \"full logits during training.\"\n },\n )\n divergence_topk: int = field(\n default=20,\n metadata={\n \"help\": \"K for top-K divergence approximations. Only used when `divergence_type` is `'topk_tv'` or \"\n \"`'topk_kl'`.\"\n },\n )\n clip_ratio_c: float = field(\n default=20.0,\n metadata={\n \"help\": \"Upper bound on the importance-sampling ratio for stability. The IS ratio is clamped to \"\n \"[0, clip_ratio_c].\"\n },\n )\n epsilon: float = field(\n default=0.15,\n metadata={\n \"help\": \"Divergence threshold ฮด_low. Tokens whose divergence exceeds this when the policy moves in the \"\n \"advantage-decreasing direction are masked. The paper recommends 0.15 for TV divergence and 0.05 for KL \"\n \"divergence.\"\n },\n )\n epsilon_high: float = field(\n default=0.15,\n metadata={\n \"help\": \"Divergence threshold ฮด_high. Tokens whose divergence exceeds this when the policy moves in the \"\n \"advantage-increasing direction are masked. The paper recommends 0.15 for TV divergence and 0.05 for KL \"\n \"divergence.\"\n },\n )\n\n def __post_init__(self):\n super().__post_init__()\n\n if self.divergence_type not in (\"binary_tv\", \"binary_kl\", \"topk_tv\", \"topk_kl\"):\n raise ValueError(\n f\"divergence_type must be one of 'binary_tv', 'binary_kl', 'topk_tv', 'topk_kl', \"\n f\"got {self.divergence_type!r}\"\n )\n\n if self.divergence_topk < 1:\n raise ValueError(f\"divergence_topk must be >= 1, got {self.divergence_topk}\")\n\n if self.clip_ratio_c <= 0:\n raise ValueError(f\"clip_ratio_c must be > 0, got {self.clip_ratio_c}\")\n\n if self.loss_type != \"dapo\":\n raise ValueError(f\"loss_type {self.loss_type} is not supported for DPPO\")\n\n if self.top_entropy_quantile != 1.0:\n raise ValueError(\"top_entropy_quantile is not supported for DPPO\")\n\n if self.off_policy_mask_threshold is not None:\n raise ValueError(\"off_policy_mask_threshold is not supported for DPPO\")\n\n if self.use_transformers_paged:\n raise ValueError(\n \"DPPO requires sampled token logprobs from the generation backend. \"\n \"Transformers paged (`use_transformers_paged=True`) does not support logprob extraction.\"\n )\n" }, { "path": "trl/experimental/dppo/dppo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport asyncio\nimport copy\nimport math\nimport textwrap\nfrom collections.abc import Callable\nfrom contextlib import nullcontext\nfrom copy import copy as shallow_copy\nfrom typing import Any\n\nimport numpy as np\nimport torch\nimport transformers\nfrom accelerate.utils import gather_object\nfrom datasets import Dataset, IterableDataset\nfrom packaging.version import Version\nfrom torch.distributed.fsdp import FullyShardedDataParallel as FSDP\nfrom transformers import (\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n Trainer,\n TrainerCallback,\n)\nfrom transformers.utils import is_peft_available\n\nfrom ...chat_template_utils import parse_response\nfrom ...data_utils import (\n apply_chat_template,\n is_conversational,\n prepare_multimodal_messages,\n)\nfrom ...extras.profiling import profiling_context, profiling_decorator\nfrom ...models import unwrap_model_for_generation\nfrom ...models.utils import disable_gradient_checkpointing\nfrom ...trainer.grpo_trainer import EnvironmentFactory, GRPOTrainer, RewardFunc, RolloutFunc\nfrom ...trainer.utils import (\n entropy_from_logits,\n nanstd,\n pad,\n selective_log_softmax,\n use_adapter,\n)\nfrom .dppo_config import DPPOConfig\n\n\nif is_peft_available():\n from peft import PeftConfig, PeftModel\n\nSAFETY_CLAMP_MAX = 20\n\n\ndef _strip_padding(tensor: torch.Tensor, mask: torch.Tensor) -> list[list]:\n \"\"\"Remove padding from a batched tensor using a mask, returning a ragged list-of-lists.\"\"\"\n return [row[m].tolist() for row, m in zip(tensor, mask.bool(), strict=True)]\n\n\nclass DPPOTrainer(GRPOTrainer):\n \"\"\"\n Trainer for Divergence Proximal Policy Optimization (DPPO).\n\n DPPO replaces PPO/GRPO's heuristic ratio-clipping with a principled trust region based on direct policy\n divergence estimates. PPO-style clipping masks tokens based on probability ratio ฯ€/ฮผ, which over-penalizes\n low-probability tokens and under-penalizes high-probability tokens. In contrast, DPPO masks based on\n direct approximation of policy divergence (e.g TV or KL) ensuring updates stay within a theoretically\n grounded trust region.\n\n\n Four divergence approximations are supported:\n - `binary_tv`: Absolute probability difference |ฯ€(a) - ฮผ(a)| (simplest)\n - `binary_kl`: Bernoulli KL divergence between old and new token probabilities\n - `topk_tv`: Total variation over the top-K tokens of the distribution\n - `topk_kl`: KL divergence over the top-K tokens of the distribution\n\n Args:\n model (`str` or [`~transformers.PreTrainedModel`] or [`~peft.PeftModel`]):\n Model to be trained. Can be either:\n\n - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using `.from_pretrained` (where `` is derived from the model\n config) with the keyword arguments in `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported.\n - A [`~peft.PeftModel`] object. Only causal language models are supported.\n reward_funcs (`RewardFunc | list[RewardFunc]`):\n Reward functions to be used for computing the rewards. To compute the rewards, we call all the reward\n functions with the prompts and completions and sum the rewards. Can be either:\n\n - A single reward function, such as:\n - A string: The *model ID* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using [`~transformers.AutoModelForSequenceClassification.from_pretrained`] with `num_labels=1` and the\n keyword arguments in `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object: Only sequence classification models are supported.\n - A custom reward function: The function is provided with the prompts and the generated completions,\n plus any additional columns in the dataset. It should return a list of rewards. Custom reward\n functions can be either synchronous or asynchronous and can also return `None` when the reward is\n not applicable to those samples. This is useful for multi-task training where different reward\n functions apply to different types of samples. When a reward function returns `None` for a sample,\n that reward function is excluded from the reward calculation for that sample. For more details, see\n [Using a custom reward\n function](#using-a-custom-reward-function).\n\n The trainer's state is also passed to the reward function. The trainer's state is an instance of\n [`~transformers.TrainerState`] and can be accessed by accessing the `trainer_state` argument to the\n reward function's signature.\n - A list of reward functions, where each item can independently be any of the above types. Mixing different\n types within the list (e.g., a string model ID and a custom reward function) is allowed.\n args ([`DPPOConfig`], *optional*):\n Configuration for this trainer. If `None`, a default configuration is used.\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n Dataset to use for training. It must include a column `\"prompt\"`. Any additional columns in the dataset is\n ignored. The format of the samples can be either:\n\n - [Standard](dataset_formats#standard): Each sample contains plain text.\n - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role\n and content).\n eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Dataset | IterableDataset]`):\n Dataset to use for evaluation. It must meet the same requirements as `train_dataset`.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. The padding side must be set to \"left\". If `None`, the\n processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A\n padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token,\n `tokenizer.eos_token` will be used as the default.\n reward_processing_classes ([`~transformers.PreTrainedTokenizerBase`] or `list[PreTrainedTokenizerBase]`, *optional*):\n Processing classes corresponding to the reward functions specified in `reward_funcs`. Can be either:\n\n - A single processing class: Used when `reward_funcs` contains only one reward function.\n - A list of processing classes: Must match the order and length of the reward functions in `reward_funcs`.\n If set to `None`, or if an element of the list corresponding to a [`~transformers.PreTrainedModel`] is\n `None`, the tokenizer for the model is automatically loaded using\n [`~transformers.AutoTokenizer.from_pretrained`]. For elements in `reward_funcs` that are custom reward\n functions (not [`~transformers.PreTrainedModel`]), the corresponding entries in `reward_processing_classes`\n are ignored.\n callbacks (list of [`~transformers.TrainerCallback`], *optional*):\n List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed\n in [here](https://huggingface.co/docs/transformers/main_classes/callback).\n\n If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`]\n method.\n optimizers (`tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None]`, *optional*, defaults to `(None, None)`):\n A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your\n model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`.\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration used to wrap the model. If `None`, the model is not wrapped.\n tools (list of `Callable`, *optional*):\n A list of callable tool functions (sync or async) that the model can invoke during generation. Each tool\n should be a standard Python function with properly type-hinted arguments and return values, and a\n Google-style docstring describing its purpose, arguments, and return value. For more details, see:\n https://huggingface.co/docs/transformers/en/chat_extras#passing-tools. The model uses the function's name,\n type hints, and docstring to determine how to call it. Ensure that the model's chat template supports tool\n use and that it has been fine-tuned for tool calling.\n rollout_func (`RolloutFunc`, *optional*):\n Function to use for generating completions. It receives the list of prompts allocated to the current\n process and the trainer instance. It must return a dict with `\"prompt_ids\"`, `\"completion_ids\"`, and\n `\"logprobs\"` fields. Any other fields are forwarded to the reward functions. This feature is experimental\n and may change or be removed at any time without prior notice.\n \"\"\"\n\n _tag_names = [\"trl\", \"dppo\"]\n _name = \"DPPO\"\n _paper = {\n \"title\": \"Rethinking the Trust Region in LLM Reinforcement Learning\",\n \"id\": \"2602.04879\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{qi2026rethinking,\n title = {{Rethinking the Trust Region in LLM Reinforcement Learning}},\n author = {Qi, Penghui and Zhou, Xiangxin and Liu, Zichen and Pang, Tianyu and Du, Chao and Lin, Min and Lee, Wee Sun},\n journal = {arXiv preprint arXiv:2602.04879},\n year = {2026}\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: \"str | PreTrainedModel | PeftModel\",\n reward_funcs: RewardFunc | list[RewardFunc],\n args: DPPOConfig | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | IterableDataset | dict[str, Dataset | IterableDataset] | None = None,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin | None = None,\n reward_processing_classes: PreTrainedTokenizerBase | list[PreTrainedTokenizerBase] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None] = (None, None),\n peft_config: \"PeftConfig | None\" = None,\n tools: list[Callable] | None = None,\n rollout_func: RolloutFunc | None = None,\n environment_factory: EnvironmentFactory | None = None,\n ):\n if args is None:\n model_name = model if isinstance(model, str) else model.config._name_or_path\n model_name = model_name.split(\"/\")[-1]\n args = DPPOConfig(f\"{model_name}-DPPO\")\n\n self.divergence_type = args.divergence_type\n self.divergence_topk = args.divergence_topk\n self.clip_ratio_c = args.clip_ratio_c\n\n super().__init__(\n model=model,\n reward_funcs=reward_funcs,\n args=args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n reward_processing_classes=reward_processing_classes,\n callbacks=callbacks,\n optimizers=optimizers,\n peft_config=peft_config,\n tools=tools,\n rollout_func=rollout_func,\n environment_factory=environment_factory,\n )\n\n if self.divergence_type in [\"topk_tv\", \"topk_kl\"] and self.use_vllm:\n self.vllm_generation.logprobs = self.divergence_topk\n\n def _tokenize_prompts(self, prompts: list):\n \"\"\"Tokenize prompts and extract images/multimodal fields for generation.\"\"\"\n if is_conversational({\"prompt\": prompts[0]}):\n images = []\n has_images = False\n for prompt in prompts:\n prompt_images = []\n for message in prompt:\n if isinstance(message[\"content\"], list):\n for part in message[\"content\"]:\n if part[\"type\"] == \"image\":\n prompt_images.append(part[\"image\"])\n has_images = True\n images.append(prompt_images if prompt_images else None)\n images = images if has_images else None\n\n # We pass padding=True to work around a bug introduced in transformers 5.2.0 in some processors\n # (e.g. Qwen2.5-VL) that crash on batched unpadded input. We then unpad input_ids using attention_mask.\n # See: https://github.com/huggingface/transformers/issues/44514\n tokenized = self.processing_class.apply_chat_template(\n conversation=prompts,\n tools=self.tools,\n chat_template=self.chat_template,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n padding=True,\n **self.chat_template_kwargs,\n )\n prompt_ids = [\n [tok for tok, mask in zip(ids, attention_mask, strict=True) if mask]\n for ids, attention_mask in zip(tokenized[\"input_ids\"], tokenized[\"attention_mask\"], strict=True)\n ]\n multimodal_fields = {k: v for k, v in tokenized.items() if k not in (\"input_ids\", \"attention_mask\")}\n else:\n prompt_ids = self.processing_class(text=prompts)[\"input_ids\"]\n images = None\n multimodal_fields = {}\n return prompt_ids, images, multimodal_fields\n\n def _generate_single_turn(self, prompt_ids, images, multimodal_fields):\n \"\"\"Generate completions, always extracting sampled token logprobs.\n\n Returns:\n 5-tuple of (prompt_ids, completion_ids, logprobs, topk_logprobs, topk_token_ids).\n topk_logprobs and topk_token_ids are None when divergence_type is not topk.\n \"\"\"\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n needs_topk = self.divergence_type in [\"topk_tv\", \"topk_kl\"]\n K = self.divergence_topk\n\n if self.use_vllm:\n if self.state.global_step != self._last_loaded_step:\n with profiling_context(self, \"sync_weights\"):\n self.vllm_generation.sync_weights()\n self._last_loaded_step = self.state.global_step\n\n num_generations = self.num_generations if mode == \"train\" else self.num_generations_eval\n prompt_ids, completion_ids, logprobs, logprob_token_ids = self.vllm_generation.generate(\n prompts=prompt_ids,\n images=images,\n num_generations=num_generations,\n profiler=profiling_context(self, \"vLLM.generate\"),\n )\n\n if needs_topk:\n # vLLM returns up to K+1 entries sorted by rank (most probable first).\n # The sampled token is always included but may be at any position.\n # Per the paper, A'_t = TopK(ฮผ, K) โˆช {a_t}. We keep exactly K slots: if the\n # sampled token a_t is not in the top-K, it replaces the K-th ranked entry.\n topk_logprobs = []\n topk_token_ids = []\n sampled_logprobs = []\n for seq_lps, seq_tids, seq_cids in zip(logprobs, logprob_token_ids, completion_ids, strict=True):\n seq_topk_lps, seq_topk_tids, seq_sampled = [], [], []\n for step_lps, step_tids, sampled_tid in zip(seq_lps, seq_tids, seq_cids, strict=True):\n idx = step_tids.index(sampled_tid)\n seq_sampled.append(step_lps[idx])\n # Take top-K entries, then ensure sampled token is present\n tk_lps = step_lps[:K]\n tk_tids = step_tids[:K]\n if sampled_tid not in tk_tids:\n tk_lps[-1] = step_lps[idx]\n tk_tids[-1] = sampled_tid\n seq_topk_lps.append(tk_lps)\n seq_topk_tids.append(tk_tids)\n topk_logprobs.append(seq_topk_lps)\n topk_token_ids.append(seq_topk_tids)\n sampled_logprobs.append(seq_sampled)\n else:\n sampled_logprobs = [[step_lps[0] for step_lps in seq_lps] for seq_lps in logprobs]\n topk_logprobs = None\n topk_token_ids = None\n\n return prompt_ids, completion_ids, sampled_logprobs, topk_logprobs, topk_token_ids\n else:\n prompt_tensors = [torch.tensor(ids) for ids in prompt_ids]\n padded_ids = pad(prompt_tensors, padding_value=self.pad_token_id, padding_side=\"left\")\n attention_mask = pad([torch.ones_like(t) for t in prompt_tensors], padding_value=0, padding_side=\"left\")\n generate_inputs = {\"input_ids\": padded_ids, \"attention_mask\": attention_mask}\n for key, value in multimodal_fields.items():\n if isinstance(value, torch.Tensor):\n generate_inputs[key] = value\n elif isinstance(value, list) and value and isinstance(value[0], list):\n generate_inputs[key] = pad([torch.tensor(x) for x in value], padding_value=0, padding_side=\"left\")\n else:\n generate_inputs[key] = torch.tensor(np.array(value))\n generate_inputs = Trainer._prepare_inputs(self, generate_inputs)\n\n gen_config = shallow_copy(self.generation_config)\n gen_config.output_logits = True\n gen_config.return_dict_in_generate = True\n\n with (\n profiling_context(self, \"transformers.generate\"),\n unwrap_model_for_generation(\n self.model_wrapped,\n self.accelerator,\n gather_deepspeed3_params=self.args.ds3_gather_for_generation,\n generation_kwargs=self.generation_kwargs,\n ) as unwrapped_model,\n torch.no_grad(),\n FSDP.summon_full_params(self.model_wrapped, recurse=False) if self.is_fsdp_enabled else nullcontext(),\n ):\n gen_output = unwrapped_model.generate(\n **generate_inputs, generation_config=gen_config, disable_compile=True\n )\n\n prompt_ids_tensor, prompt_mask = generate_inputs[\"input_ids\"], generate_inputs[\"attention_mask\"]\n prompt_length = prompt_ids_tensor.size(1)\n completion_ids = gen_output.sequences[:, prompt_length:]\n\n sampled_chunks = []\n topk_logps_chunks = [] if needs_topk else None\n topk_ids_chunks = [] if needs_topk else None\n\n for t, logits_t in enumerate(gen_output.logits):\n # logits_t: (B, V)\n logits_t = logits_t / self.temperature\n\n # exact sampled-token logprob without allocating (B, V) log_softmax output\n logZ_t = torch.logsumexp(logits_t, dim=-1, keepdim=True)\n sampled_ids_t = completion_ids[:, t : t + 1]\n sampled_lp_t = logits_t.gather(-1, sampled_ids_t) - logZ_t\n sampled_chunks.append(sampled_lp_t.cpu())\n\n if needs_topk:\n topk_logits_t, topk_ids_t = torch.topk(logits_t, k=K, dim=-1) # (B, K), (B, K)\n topk_lp_t = topk_logits_t - logZ_t\n\n # Ensure sampled token is included in A'_t = TopK โˆช {a_t}\n missing = ~(topk_ids_t == sampled_ids_t).any(dim=-1)\n if missing.any():\n topk_ids_t = topk_ids_t.clone()\n topk_lp_t = topk_lp_t.clone()\n topk_ids_t[missing, -1] = sampled_ids_t[missing, 0]\n topk_lp_t[missing, -1] = sampled_lp_t[missing, 0]\n\n topk_ids_chunks.append(topk_ids_t.cpu())\n topk_logps_chunks.append(topk_lp_t.cpu())\n\n # Mask everything after the first EOS token\n is_eos = completion_ids == self.eos_token_id\n has_eos = is_eos.any(dim=1)\n eos_idx = torch.full((is_eos.size(0),), is_eos.size(1), dtype=torch.long, device=device)\n eos_idx[has_eos] = is_eos.int().argmax(dim=1)[has_eos]\n sequence_indices = torch.arange(is_eos.size(1), device=device).expand(is_eos.size(0), -1)\n completion_mask = (sequence_indices <= eos_idx.unsqueeze(1)).int()\n prompt_mask_cpu = prompt_mask.bool().cpu()\n completion_mask_cpu = completion_mask.bool().cpu()\n\n prompt_ids_out = _strip_padding(prompt_ids_tensor.cpu(), prompt_mask_cpu)\n completion_ids_out = _strip_padding(completion_ids.cpu(), completion_mask_cpu)\n logprobs_out = _strip_padding(torch.cat(sampled_chunks, dim=1), completion_mask_cpu)\n if needs_topk:\n topk_logprobs = _strip_padding(torch.stack(topk_logps_chunks, dim=1), completion_mask_cpu)\n topk_token_ids = _strip_padding(torch.stack(topk_ids_chunks, dim=1), completion_mask_cpu)\n else:\n topk_logprobs = None\n topk_token_ids = None\n\n return prompt_ids_out, completion_ids_out, logprobs_out, topk_logprobs, topk_token_ids\n\n def _tool_call_loop(\n self, prompts, prompt_ids, completion_ids, completions, logprobs, topk_logprobs, topk_token_ids\n ):\n \"\"\"Tool execution loop that also threads top-K logprob data alongside logprobs.\n\n Mirrors GRPOTrainer._tool_call_loop but additionally concatenates topk_logprobs and topk_token_ids\n the same way logprobs is concatenated: real data for model-generated tokens, zero-padding for\n tool-result tokens. When topk data is None (binary divergence), behaves identically to the parent.\n \"\"\"\n K = self.divergence_topk\n has_topk = topk_logprobs is not None\n\n tool_calls = [completion[0].get(\"tool_calls\") for completion in completions]\n idxs_with_tool = [idx for idx, tool_call in enumerate(tool_calls) if tool_call]\n tool_calls = [tool_calls[idx] for idx in idxs_with_tool]\n tool_mask = [[1] * len(ids) for ids in completion_ids]\n tool_call_count = 0\n tool_failure_count = 0\n iteration_num = 0\n while idxs_with_tool and iteration_num < self.max_tool_calling_iterations:\n prompt_completion_tools = [prompts[i] for i in idxs_with_tool]\n\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n tool_call_list = tool_calls[idx]\n prompt_completion_tool = prompt_completion_tools[idx]\n sync_tool_dict = self._sync_tool_dicts[idx_with_tool]\n async_tool_dict = self._async_tool_dicts[idx_with_tool]\n prompt_completion_tool.append(completions[idx_with_tool][-1])\n async_coros = []\n tool_call_results = []\n for tool_call in tool_call_list:\n tool_call_count += 1\n if tool_call[\"type\"] == \"function\":\n function = tool_call[\"function\"]\n name = function[\"name\"]\n try:\n if name in sync_tool_dict:\n tool_call_results.append((name, sync_tool_dict[name](**function[\"arguments\"])))\n elif name in async_tool_dict:\n async_coros.append((name, async_tool_dict[name](**function[\"arguments\"])))\n else:\n raise ValueError(f\"Tool {name} not found.\")\n except Exception as err:\n tool_failure_count += 1\n tool_call_results.append((name, {\"error\": str(err)}))\n else:\n tool_failure_count += 1\n name = tool_call.get(\"name\", \"unknown\")\n tool_call_results.append((name, {\"error\": f\"Unsupported tool call type: {tool_call['type']}\"}))\n\n if async_coros:\n\n async def _run_async_tools(async_coros):\n coros = [coro for _, coro in async_coros]\n results = await asyncio.gather(*coros, return_exceptions=True)\n return [(name, result) for (name, _), result in zip(async_coros, results, strict=False)]\n\n async_results = asyncio.run_coroutine_threadsafe(\n _run_async_tools(async_coros), self.async_loop\n ).result()\n\n for name, result in async_results:\n if isinstance(result, Exception):\n tool_failure_count += 1\n tool_call_results.append((name, {\"error\": str(result)}))\n else:\n tool_call_results.append((name, result))\n\n for name, result in tool_call_results:\n tool_message = {\"role\": \"tool\", \"name\": name, \"content\": str(result)}\n prompt_completion_tool.append(tool_message)\n completions[idx_with_tool].append(tool_message)\n\n # Tokenize and filter samples whose length exceeds max allowed length\n pct_ids = self.processing_class.apply_chat_template(\n prompt_completion_tools,\n tools=self.tools,\n chat_template=self.chat_template,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=False,\n **self.chat_template_kwargs,\n )\n if self.use_vllm and self.vllm_mode == \"colocate\":\n max_model_len = self.vllm_generation.llm.llm_engine.model_config.max_model_len\n elif not self.use_vllm:\n max_model_len = self.model.config.max_position_embeddings\n else:\n raise NotImplementedError(\n f\"Unsupported mode detected: use_vllm={self.use_vllm}, vllm_mode={self.vllm_mode}\"\n )\n overlong = [len(pct) >= max_model_len for pct in pct_ids]\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n if overlong[idx]:\n prompt_length = len(prompt_ids[idx_with_tool])\n ct = pct_ids[idx][prompt_length : prompt_length + self.max_completion_length]\n completion_ids[idx_with_tool] = ct\n tool_mask[idx_with_tool] += [1] * (len(ct) - len(tool_mask[idx_with_tool]))\n if logprobs is not None:\n logprobs[idx_with_tool] += [0.0] * (len(ct) - len(logprobs[idx_with_tool]))\n if has_topk:\n topk_logprobs[idx_with_tool] += [[0.0] * K] * (len(ct) - len(topk_logprobs[idx_with_tool]))\n topk_token_ids[idx_with_tool] += [[0] * K] * (len(ct) - len(topk_token_ids[idx_with_tool]))\n\n idxs_with_tool = [idx for idx, o in zip(idxs_with_tool, overlong, strict=True) if not o]\n prompt_completion_tools = [pct for pct, o in zip(prompt_completion_tools, overlong, strict=True) if not o]\n if not idxs_with_tool:\n break\n\n # Generate new completions after tool execution\n pct_prompt_ids, pct_images, pct_multimodal_fields = self._tokenize_prompts(prompt_completion_tools)\n (\n prompt_completion_tool_ids,\n post_tool_ids,\n post_tool_logprobs,\n post_tool_topk_logprobs,\n post_tool_topk_token_ids,\n ) = self._generate_single_turn(pct_prompt_ids, pct_images, pct_multimodal_fields)\n\n # Sanity check: chat template must be prefix-preserving\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n pct = prompt_completion_tool_ids[idx]\n if prompt_ids[idx_with_tool] != pct[: len(prompt_ids[idx_with_tool])]:\n raise ValueError(\n \"The chat template is not prefix-preserving. Please update it to use a prefix-preserving \"\n \"format.\"\n )\n\n # Truncate so that pct[len(prompt_ids[idx]):] + post_tool does not exceed max_completion_length\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n prompt_len = len(prompt_ids[idx_with_tool])\n completion_tool_ids = prompt_completion_tool_ids[idx][prompt_len:]\n excess_length = len(completion_tool_ids) + len(post_tool_ids[idx]) - self.max_completion_length\n if excess_length > 0:\n post_tool_ids[idx] = post_tool_ids[idx][:-excess_length]\n if logprobs is not None:\n post_tool_logprobs[idx] = post_tool_logprobs[idx][:-excess_length]\n if has_topk and post_tool_topk_logprobs is not None:\n post_tool_topk_logprobs[idx] = post_tool_topk_logprobs[idx][:-excess_length]\n post_tool_topk_token_ids[idx] = post_tool_topk_token_ids[idx][:-excess_length]\n excess_length = len(completion_tool_ids) + len(post_tool_ids[idx]) - self.max_completion_length\n if excess_length > 0:\n prompt_completion_tool_ids[idx] = prompt_completion_tool_ids[idx][:-excess_length]\n\n # Update tool_mask and logprobs: tool result tokens get 0/0.0, post-tool model tokens get 1/real values\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n prompt_completion_tool_length = len(prompt_completion_tool_ids[idx])\n prompt_length = len(prompt_ids[idx_with_tool])\n completion_length = len(completion_ids[idx_with_tool])\n post_tool_length = len(post_tool_ids[idx])\n tool_length = prompt_completion_tool_length - prompt_length - completion_length\n tool_mask[idx_with_tool] += [0] * tool_length + [1] * post_tool_length\n if logprobs is not None:\n logprobs[idx_with_tool] += [0.0] * tool_length + post_tool_logprobs[idx]\n if has_topk:\n topk_pad = [[0.0] * K] * tool_length\n tid_pad = [[0] * K] * tool_length\n post_topk_lp = post_tool_topk_logprobs[idx] if post_tool_topk_logprobs is not None else []\n post_topk_tid = post_tool_topk_token_ids[idx] if post_tool_topk_token_ids is not None else []\n topk_logprobs[idx_with_tool] += topk_pad + post_topk_lp\n topk_token_ids[idx_with_tool] += tid_pad + post_topk_tid\n\n # Update completion_ids with the new completions (after tool execution)\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n prompt_length = len(prompt_ids[idx_with_tool])\n pct = prompt_completion_tool_ids[idx]\n completion_ids[idx_with_tool] = pct[prompt_length:] + post_tool_ids[idx]\n\n # Decode post-tool completions\n post_tool_completions = [\n parse_response(self.processing_class, ids) if ids else {} for ids in post_tool_ids\n ]\n\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n if post_tool_completions[idx]:\n completions[idx_with_tool].append(post_tool_completions[idx])\n\n # Check for further tool calls\n tool_calls = [completion.get(\"tool_calls\") for completion in post_tool_completions]\n idxs_with_tool = [idx for idx, tool_call in zip(idxs_with_tool, tool_calls, strict=True) if tool_call]\n tool_calls = [tool_call for tool_call in tool_calls if tool_call]\n iteration_num += 1\n\n return (\n tool_mask,\n completions,\n completion_ids,\n logprobs,\n topk_logprobs,\n topk_token_ids,\n tool_call_count,\n tool_failure_count,\n )\n\n def _generate(self, prompts: list):\n \"\"\"Generate completions, handling tool calls, and thread top-K logprob data through the full pipeline.\n\n Returns:\n 9-tuple of (prompt_ids, completion_ids, tool_mask, completions, total_completion_tokens,\n logprobs, topk_logprobs, topk_token_ids, extra_fields).\n \"\"\"\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n needs_topk = self.divergence_type in [\"topk_tv\", \"topk_kl\"]\n\n # Copy the prompts to avoid modifying the original list\n prompts = copy.deepcopy(prompts)\n\n if self.rollout_func is not None:\n # Keep vLLM weights in sync for custom rollouts that rely on vLLM utilities.\n if self.use_vllm and self.state.global_step != self._last_loaded_step:\n with profiling_context(self, \"sync_weights\"):\n self.vllm_generation.sync_weights()\n self._last_loaded_step = self.state.global_step\n\n # Pass prompts to rollout_func preserving structured messages.\n # Chat templating must happen inside rollout_func, at the backend boundary, so that\n # multimodal content (images, typed content blocks) is not lost before rollout logic runs.\n output = self.rollout_func(prompts, self)\n required_keys = {\"prompt_ids\", \"completion_ids\", \"logprobs\"}\n missing_keys = required_keys - output.keys()\n if missing_keys:\n missing_keys_list = sorted(missing_keys)\n raise ValueError(f\"rollout_func must return keys {missing_keys_list} in its output dict.\")\n extra_fields = {k: v for k, v in output.items() if k not in required_keys}\n prompt_ids = output[\"prompt_ids\"]\n completion_ids = output[\"completion_ids\"]\n logprobs = output[\"logprobs\"]\n topk_logprobs = extra_fields.pop(\"topk_logprobs\", None)\n topk_token_ids = extra_fields.pop(\"topk_token_ids\", None)\n if needs_topk and (topk_logprobs is None or topk_token_ids is None):\n raise ValueError(\n \"rollout_func must return keys ['topk_logprobs', 'topk_token_ids'] when divergence_type is \"\n f\"{self.divergence_type!r}.\"\n )\n else:\n prompt_ids, images, multimodal_fields = self._tokenize_prompts(prompts)\n prompt_ids, completion_ids, logprobs, topk_logprobs, topk_token_ids = self._generate_single_turn(\n prompt_ids, images, multimodal_fields\n )\n extra_fields = {}\n\n # Decode completions. It's important to use `parse_response` when possible, because it handles tool calls.\n if is_conversational({\"prompt\": prompts[0]}):\n if (\n Version(transformers.__version__) >= Version(\"5.0.0\") # parse_response added in v5\n and isinstance(self.processing_class, PreTrainedTokenizerBase) # doesn't work with processors\n and hasattr(self.processing_class, \"response_schema\") # attribute not set by default for now\n and self.processing_class.response_schema is not None # only works if the tokenizer has a schema\n ):\n completions = [[parse_response(self.processing_class, ids)] for ids in completion_ids]\n else:\n contents = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n completions = [[{\"role\": \"assistant\", \"content\": content}] for content in contents]\n else:\n completions = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n\n # Extract tool calls from the completions and (possibly) execute them\n if self.tools:\n (\n tool_mask,\n completions,\n completion_ids,\n logprobs,\n topk_logprobs,\n topk_token_ids,\n tool_call_count,\n tool_failure_count,\n ) = self._tool_call_loop(\n prompts, prompt_ids, completion_ids, completions, logprobs, topk_logprobs, topk_token_ids\n )\n else:\n tool_mask = extra_fields.pop(\"env_mask\", None)\n\n # Get completion length per sequence, used for logging\n prompt_lengths = torch.tensor([len(ids) for ids in prompt_ids], device=device)\n if tool_mask is not None:\n completion_lengths = torch.tensor([sum(mask) for mask in tool_mask], device=device)\n else:\n completion_lengths = torch.tensor([len(ids) for ids in completion_ids], device=device)\n agg_prompt_lengths = self.accelerator.gather(prompt_lengths)\n agg_completion_lengths = self.accelerator.gather(completion_lengths)\n total_prompt_tokens = agg_prompt_lengths.sum()\n total_completion_tokens = agg_completion_lengths.sum()\n\n if mode == \"train\":\n self.state.num_input_tokens_seen += (total_prompt_tokens + total_completion_tokens).item()\n self._metrics[mode][\"num_tokens\"] = [self.state.num_input_tokens_seen]\n\n self._metrics[mode][\"completions/mean_length\"].append(agg_completion_lengths.float().mean().item())\n self._metrics[mode][\"completions/min_length\"].append(agg_completion_lengths.float().min().item())\n self._metrics[mode][\"completions/max_length\"].append(agg_completion_lengths.float().max().item())\n\n eos_and_pad = [self.eos_token_id, self.pad_token_id]\n is_truncated = torch.tensor([ids[-1] not in eos_and_pad for ids in completion_ids], device=device)\n agg_is_truncated = self.accelerator.gather(is_truncated)\n self._metrics[mode][\"completions/clipped_ratio\"].append(agg_is_truncated.float().mean().item())\n term_completion_lengths = agg_completion_lengths[~agg_is_truncated]\n if len(term_completion_lengths) == 0:\n term_completion_lengths = torch.zeros(1, device=device)\n self._metrics[mode][\"completions/mean_terminated_length\"].append(term_completion_lengths.float().mean().item())\n self._metrics[mode][\"completions/min_terminated_length\"].append(term_completion_lengths.float().min().item())\n self._metrics[mode][\"completions/max_terminated_length\"].append(term_completion_lengths.float().max().item())\n\n if self.tools:\n agg_tool_call_count = self.accelerator.gather(torch.tensor(tool_call_count, device=device)).sum()\n tool_call_frequency = (agg_tool_call_count / len(agg_prompt_lengths)).item()\n self._metrics[mode][\"tools/call_frequency\"].append(tool_call_frequency)\n agg_tool_failure_count = self.accelerator.gather(torch.tensor(tool_failure_count, device=device)).sum()\n failure_frequency = (\n (agg_tool_failure_count / agg_tool_call_count).item() if agg_tool_call_count > 0 else 0.0\n )\n self._metrics[mode][\"tools/failure_frequency\"].append(failure_frequency)\n\n return (\n prompt_ids,\n completion_ids,\n tool_mask,\n completions,\n total_completion_tokens,\n logprobs,\n topk_logprobs,\n topk_token_ids,\n extra_fields,\n )\n\n @profiling_decorator\n def _get_per_token_logps_with_topk(\n self,\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n topk_token_ids,\n batch_size=None,\n compute_entropy=False,\n pixel_values=None,\n image_grid_thw=None,\n num_images=None,\n pixel_attention_mask=None,\n image_sizes=None,\n token_type_ids=None,\n mm_token_type_ids=None,\n ) -> tuple[torch.Tensor, torch.Tensor | None, torch.Tensor]:\n \"\"\"Compute per-token log-probs, (optionally) entropies, and top-K log-probs in one forward pass.\n\n Evaluates the current policy's log-probs at the rollout's top-K token IDs from the same\n forward pass used for per_token_logps, avoiding an extra model call.\n\n Args:\n topk_token_ids: Rollout policy's top-K token IDs, shape (B, T, K). The current policy's\n log-probs are evaluated at these positions.\n\n Returns:\n Tuple of (per_token_logps, entropies, current_topk_logps).\n \"\"\"\n batch_size = batch_size or input_ids.size(0)\n all_logps = []\n all_entropies = []\n all_topk_logps = []\n\n for start in range(0, input_ids.size(0), batch_size):\n end = start + batch_size\n input_ids_batch = input_ids[start:end]\n attention_mask_batch = attention_mask[start:end]\n\n model_inputs = {\"input_ids\": input_ids_batch, \"attention_mask\": attention_mask_batch}\n if image_grid_thw is not None and pixel_values is not None:\n rows_per_image = image_grid_thw.prod(dim=-1)\n rows_per_sample = torch.split(rows_per_image, num_images)\n rows_per_sample = torch.stack([s.sum() for s in rows_per_sample])\n cum_rows = torch.cat([torch.tensor([0], device=rows_per_sample.device), rows_per_sample.cumsum(0)])\n row_start, row_end = cum_rows[start].item(), cum_rows[end].item()\n model_inputs[\"pixel_values\"] = pixel_values[row_start:row_end]\n cum_imgs = torch.tensor([0] + num_images).cumsum(0)\n img_start, img_end = cum_imgs[start], cum_imgs[end]\n model_inputs[\"image_grid_thw\"] = image_grid_thw[img_start:img_end]\n elif pixel_values is not None:\n model_inputs[\"pixel_values\"] = pixel_values[start:end]\n if pixel_attention_mask is not None:\n model_inputs[\"pixel_attention_mask\"] = pixel_attention_mask[start:end]\n if image_sizes is not None:\n model_inputs[\"image_sizes\"] = image_sizes[start:end]\n if token_type_ids is not None:\n model_inputs[\"token_type_ids\"] = token_type_ids[start:end]\n if mm_token_type_ids is not None:\n model_inputs[\"mm_token_type_ids\"] = mm_token_type_ids[start:end]\n\n if \"logits_to_keep\" in self.model_kwarg_keys:\n model_inputs[\"logits_to_keep\"] = logits_to_keep + 1\n\n model_inputs[\"use_cache\"] = False\n\n logits = model(**model_inputs).logits\n logits = logits[:, :-1, :]\n logits = logits[:, -logits_to_keep:, :]\n logits = logits / self.temperature\n\n completion_ids = input_ids_batch[:, -logits_to_keep:]\n logps = selective_log_softmax(logits, completion_ids)\n all_logps.append(logps)\n\n if compute_entropy:\n with torch.no_grad():\n entropies = entropy_from_logits(logits)\n all_entropies.append(entropies)\n\n with torch.no_grad():\n topk_logps = selective_log_softmax(logits, topk_token_ids[start:end])\n all_topk_logps.append(topk_logps)\n\n logps = torch.cat(all_logps, dim=0)\n entropies = torch.cat(all_entropies, dim=0) if compute_entropy else None\n topk_logps = torch.cat(all_topk_logps, dim=0)\n return logps, entropies, topk_logps\n\n def _generate_and_score_completions(\n self, inputs: list[dict[str, torch.Tensor | Any]]\n ) -> dict[str, torch.Tensor | Any]:\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n\n prompts = [x[\"prompt\"] for x in inputs]\n\n if self.environments:\n for prompt, environment, reset_kwargs in zip(prompts, self.environments, inputs, strict=True):\n observation = environment.reset(**reset_kwargs)\n if observation is None:\n continue\n prompt[-1][\"content\"] += observation\n\n if \"images\" in inputs[0]:\n images = [example.get(\"images\") for example in inputs]\n elif \"image\" in inputs[0]:\n images = [[example.get(\"image\")] if example.get(\"image\") is not None else None for example in inputs]\n else:\n images = None\n # Transformers requires at least one image in the batch, otherwise it throws an error\n if images is not None and all(img_list == [] for img_list in images):\n images = None\n\n # If the prompts are conversational and the inputs contain images, we need to convert the prompts from\n # [{\"role\": \"user\", \"content\": \"What color is the sky?\"}] to\n # [{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": }, {\"type\": \"text\", \"text\": \"What color is the sky?\"}]}]\n if images is not None:\n if not is_conversational(inputs[0]):\n raise ValueError(\n \"Multimodal training requires conversational prompts. It looks like the dataset contains \"\n \"non-conversational inputs, likely because a chat template was applied before passing the dataset \"\n \"to the trainer. Please provide the raw conversational prompts and let the trainer apply the chat \"\n \"template internally.\"\n )\n prompts = [\n prepare_multimodal_messages(prompt, image_list)\n for prompt, image_list in zip(prompts, images, strict=True)\n ]\n\n (\n prompt_ids_list,\n completion_ids_list,\n tool_mask_list,\n completions,\n num_items_in_batch,\n sampling_per_token_logps_list,\n topk_logprobs_list,\n topk_token_ids_list,\n extra_fields,\n ) = self._generate(prompts)\n\n # Convert lists of token IDs to padded tensors\n prompt_ids = [torch.tensor(ids) for ids in prompt_ids_list]\n prompt_mask = [torch.ones_like(ids, dtype=torch.long) for ids in prompt_ids]\n prompt_ids = pad(\n prompt_ids,\n padding_value=self.pad_token_id,\n padding_side=\"left\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n prompt_mask = pad(\n prompt_mask, padding_value=0, padding_side=\"left\", pad_to_multiple_of=self.pad_to_multiple_of\n ).to(device=device)\n completion_ids = [torch.tensor(ids) for ids in completion_ids_list]\n completion_mask = [torch.ones_like(ids, dtype=torch.long) for ids in completion_ids]\n completion_ids = pad(\n completion_ids,\n padding_value=self.pad_token_id,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n completion_mask = pad(\n completion_mask, padding_value=0, padding_side=\"right\", pad_to_multiple_of=self.pad_to_multiple_of\n ).to(device=device)\n sampling_per_token_logps = [torch.tensor(logps) for logps in sampling_per_token_logps_list]\n sampling_per_token_logps = pad(\n sampling_per_token_logps,\n padding_value=0.0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n if tool_mask_list is not None:\n tool_mask = [torch.tensor(mask) for mask in tool_mask_list]\n tool_mask = pad(\n tool_mask, padding_value=1, padding_side=\"right\", pad_to_multiple_of=self.pad_to_multiple_of\n ).to(device=device)\n else:\n tool_mask = None\n if topk_logprobs_list is not None:\n sampling_topk_logps = [torch.tensor(lp) for lp in topk_logprobs_list]\n sampling_topk_logps = pad(\n sampling_topk_logps,\n padding_value=0.0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n sampling_topk_token_ids = [torch.tensor(tid, dtype=torch.long) for tid in topk_token_ids_list]\n sampling_topk_token_ids = pad(\n sampling_topk_token_ids,\n padding_value=0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n else:\n sampling_topk_logps = None\n sampling_topk_token_ids = None\n\n # If mask_truncated_completions is enabled, zero out truncated completions for attention and loss masking\n if self.mask_truncated_completions:\n eos_and_pad = [self.eos_token_id, self.pad_token_id]\n is_truncated = torch.tensor([ids[-1] not in eos_and_pad for ids in completion_ids_list], device=device)\n # Mask completion_mask for attention masking\n completion_mask = completion_mask * (~is_truncated).unsqueeze(1).int()\n # Also mask tool_mask for consistency in multi-turn training\n if tool_mask is not None:\n tool_mask = tool_mask * (~is_truncated).unsqueeze(1).int()\n\n # Concatenate prompt_mask with completion_mask for logit computation\n prompt_completion_ids = torch.cat([prompt_ids, completion_ids], dim=1) # (B, P+C)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1) # (B, P+C)\n\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n batch_size = self.args.per_device_train_batch_size if mode == \"train\" else self.args.per_device_eval_batch_size\n\n num_images = [len(img_list) for img_list in images] if images is not None else None\n\n # Get forward_kwargs for models with multimodal inputs\n if images is not None:\n prompts_text = [\n apply_chat_template(\n {\"prompt\": prompt}, self.processing_class, tools=self.tools, **self.chat_template_kwargs\n )[\"prompt\"]\n for prompt in prompts\n ]\n prompt_inputs = self.processing_class(images=images, text=prompts_text, padding=True, return_tensors=\"pt\")\n prompt_inputs = Trainer._prepare_inputs(self, prompt_inputs)\n forward_kwargs = {k: v for k, v in prompt_inputs.items() if k not in [\"input_ids\", \"attention_mask\"]}\n else:\n forward_kwargs = {}\n\n # If token_type_ids are used, extend them with zeros for the completion part\n if \"token_type_ids\" in forward_kwargs:\n token_type_ids = forward_kwargs[\"token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n padding_size = prompt_ids.size(1) - token_type_ids.size(1)\n if padding_size > 0:\n token_type_ids = torch.cat(\n [token_type_ids.new_zeros((token_type_ids.size(0), padding_size)), token_type_ids], dim=1\n )\n forward_kwargs[\"token_type_ids\"] = torch.cat(\n [token_type_ids, token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n # If mm_token_type_ids are used, extend them with zeros for the completion part\n if \"mm_token_type_ids\" in forward_kwargs:\n mm_token_type_ids = forward_kwargs[\"mm_token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n padding_size = prompt_ids.size(1) - mm_token_type_ids.size(1)\n if padding_size > 0:\n mm_token_type_ids = torch.cat(\n [mm_token_type_ids.new_zeros((mm_token_type_ids.size(0), padding_size)), mm_token_type_ids],\n dim=1,\n )\n forward_kwargs[\"mm_token_type_ids\"] = torch.cat(\n [mm_token_type_ids, mm_token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n\n # When gradient checkpointing is enabled with use_reentrant=True (non default), calling the model inside a\n # torch.no_grad() block triggers a harmless PyTorch warning (\"None of the inputs have requires_grad=True\").\n # Temporarily disable checkpointing to avoid this warning during inference.\n with torch.no_grad(), disable_gradient_checkpointing(self.model, self.args.gradient_checkpointing_kwargs):\n # Compute the per-token log probabilities for the reference model\n if self.beta != 0.0:\n if self.ref_model is not None:\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.ref_model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n # When training a PEFT adapter, how we obtain the reference depends on the setup:\n # - New adapter: disabling adapters yields the base model.\n # - Re-training an existing adapter: an initial copy is loaded under the name \"ref\".\n model = self.accelerator.unwrap_model(self.model)\n with use_adapter(model, adapter_name=\"ref\" if \"ref\" in model.peft_config else None):\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n ref_per_token_logps = None\n\n # Decode\n prompts_text = self.processing_class.batch_decode(prompt_ids, skip_special_tokens=True)\n completions_text = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n\n # Merge extra_fields from rollout_func into inputs for reward functions\n if extra_fields:\n for i, inp in enumerate(inputs):\n for key, values in extra_fields.items():\n if isinstance(values, list) and i < len(values):\n inp[key] = values[i]\n elif not isinstance(values, list):\n inp[key] = values\n\n # Calculate rewards for each reward function. rewards_per_func aggregates rewards across all processes. This is\n # important because rewards will be normalized per group, and completions are distributed. We will later slice\n # rewards_per_func to extract each process's subset.\n rewards_per_func = self._calculate_rewards(inputs, prompts, completions, completion_ids_list)\n num_generations = self.num_generations if mode == \"train\" else self.num_generations_eval\n\n if self.multi_objective_aggregation == \"sum_then_normalize\":\n # Apply weights to each reward function's output and sum\n rewards = (rewards_per_func * self.reward_weights.to(device).unsqueeze(0)).nansum(dim=1)\n mean_grouped_rewards = rewards.view(-1, num_generations).mean(dim=1)\n mean_grouped_rewards = mean_grouped_rewards.repeat_interleave(num_generations, dim=0)\n if self.scale_rewards in [\"group\", \"none\"]:\n # If self.scale_rewards = \"none\", we'll only use std_rewards to check for zero std for logging\n if num_generations > 1:\n std_rewards = rewards.view(-1, num_generations).std(dim=1)\n std_rewards = std_rewards.repeat_interleave(num_generations, dim=0)\n else: # doesn't occur during training, but could occur in eval when num_generations_eval=1\n std_rewards = torch.zeros_like(rewards)\n elif self.scale_rewards == \"batch\":\n # Compute global std\n if rewards.numel() > 1:\n std_rewards = rewards.std().expand_as(rewards)\n else: # doesn't occur during training, but could occur in eval when num_generations_eval=batch_size=1\n std_rewards = torch.zeros_like(rewards)\n else:\n raise ValueError(\n f\"Invalid value for scale_rewards: {self.scale_rewards}. Must be one of 'batch', 'group', or 'none'.\"\n )\n\n advantages = rewards - mean_grouped_rewards\n if self.scale_rewards != \"none\":\n advantages = advantages / (std_rewards + 1e-4)\n is_std_zero = torch.isclose(std_rewards, torch.zeros_like(std_rewards)) # for logging\n\n elif self.multi_objective_aggregation == \"normalize_then_sum\":\n grouped = rewards_per_func.view(-1, num_generations, len(self.reward_funcs))\n mean_k = torch.nanmean(grouped, dim=1, keepdim=True)\n std_k = nanstd(grouped, dim=1, keepdim=True) if num_generations > 1 else torch.zeros_like(mean_k)\n reward_k = (grouped - mean_k) / (std_k + 1e-4)\n reward_k = reward_k.view(-1, len(self.reward_funcs))\n rewards = (reward_k * self.reward_weights.to(device).unsqueeze(0)).nansum(dim=1)\n std_rewards = rewards.std().expand_as(rewards) if rewards.numel() > 1 else torch.zeros_like(rewards)\n advantages = (rewards - rewards.mean()) / (std_rewards + 1e-4)\n is_std_zero = torch.isclose(std_rewards, torch.zeros_like(std_rewards)) # for logging\n\n else:\n raise ValueError(\n f\"Invalid multi_objective_aggregation: {self.multi_objective_aggregation}. Must be \"\n \"'sum_then_normalize' or 'normalize_then_sum'.\"\n )\n\n # Slice to keep only the local part of the data\n process_slice = slice(\n self.accelerator.process_index * len(prompts),\n (self.accelerator.process_index + 1) * len(prompts),\n )\n all_process_advantages = advantages.clone() # keep the aggregated advantages for logging\n advantages = advantages[process_slice]\n\n # Calculate mean reward per function, but only for samples where the function was applied (non-NaN values)\n for i, reward_func_name in enumerate(self.reward_func_names):\n mean_rewards = torch.nanmean(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/mean\"].append(mean_rewards)\n std_func_rewards = nanstd(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/std\"].append(std_func_rewards)\n rewards = rewards_per_func.nansum(dim=1)\n self._metrics[mode][\"reward\"].append(rewards.mean().item())\n self._metrics[mode][\"reward_std\"].append(rewards.std().item())\n self._metrics[mode][\"frac_reward_zero_std\"].append(is_std_zero.float().mean().item())\n\n # Log prompt and completion texts\n self._logs[\"prompt\"].extend(gather_object(prompts_text))\n self._logs[\"completion\"].extend(gather_object(completions_text))\n for i, name in enumerate(self.reward_func_names):\n self._logs[\"rewards\"][name].extend(rewards_per_func[:, i].tolist())\n self._logs[\"advantages\"].extend(all_process_advantages.tolist())\n\n # Flush user-logged extra columns (from log_extra), gathering across processes.\n # Keys must be sorted so that all ranks call gather_object in the same order, otherwise values\n # get mis-attributed across columns (dict insertion order may differ between processes).\n for column in sorted(self._pending_extra_logs):\n self._logs[\"extra\"][column].extend(gather_object(self._pending_extra_logs[column]))\n self._pending_extra_logs.clear()\n\n # Flush user-logged metrics (from log_metric), averaging across processes.\n # Keys must be sorted so that all ranks call accelerator.gather in the same order, otherwise values\n # get mis-attributed across metrics (dict insertion order may differ between processes).\n for name in sorted(self._pending_metrics):\n values = self._pending_metrics[name]\n local_mean = sum(values) / len(values)\n global_mean = self.accelerator.gather(torch.tensor(local_mean, device=device)).mean().item()\n self._metrics[mode][name].append(global_mean)\n self._pending_metrics.clear()\n\n if images is not None:\n self._logs[\"images\"].extend(gather_object(images))\n\n output = {\n \"prompt_ids\": prompt_ids,\n \"prompt_mask\": prompt_mask,\n \"completion_ids\": completion_ids,\n \"completion_mask\": completion_mask,\n \"advantages\": advantages,\n \"num_items_in_batch\": num_items_in_batch,\n \"sampling_per_token_logps\": sampling_per_token_logps,\n }\n if ref_per_token_logps is not None:\n output[\"ref_per_token_logps\"] = ref_per_token_logps\n if \"pixel_values\" in forward_kwargs:\n output[\"pixel_values\"] = forward_kwargs[\"pixel_values\"]\n if \"image_grid_thw\" in forward_kwargs:\n output[\"image_grid_thw\"] = forward_kwargs[\"image_grid_thw\"]\n if \"pixel_attention_mask\" in forward_kwargs:\n output[\"pixel_attention_mask\"] = forward_kwargs[\"pixel_attention_mask\"]\n if \"image_sizes\" in forward_kwargs:\n output[\"image_sizes\"] = forward_kwargs[\"image_sizes\"]\n if \"token_type_ids\" in forward_kwargs:\n output[\"token_type_ids\"] = forward_kwargs[\"token_type_ids\"]\n if \"mm_token_type_ids\" in forward_kwargs:\n output[\"mm_token_type_ids\"] = forward_kwargs[\"mm_token_type_ids\"]\n if images is not None:\n output[\"num_images\"] = num_images\n if tool_mask is not None:\n output[\"tool_mask\"] = tool_mask\n if sampling_topk_logps is not None:\n output[\"sampling_topk_logps\"] = sampling_topk_logps\n if sampling_topk_token_ids is not None:\n output[\"sampling_topk_token_ids\"] = sampling_topk_token_ids\n return output\n\n @torch.no_grad()\n def _compute_divergence_mask(\n self,\n per_token_logps,\n sampling_per_token_logps,\n advantages,\n completion_mask,\n current_topk_logps=None,\n sampling_topk_logps=None,\n ):\n \"\"\"\n Compute a per-token trust-region mask based on the configured divergence type. Tokens where the policy has\n diverged too far from the sampling distribution (in a direction that would increase the loss) are masked out.\n\n Args:\n per_token_logps (`torch.Tensor`):\n Log-probabilities of the current policy at the sampled tokens, shape `(B, T)`.\n sampling_per_token_logps (`torch.Tensor`):\n Log-probabilities of the sampling (rollout) policy at the sampled tokens, shape `(B, T)`.\n advantages (`torch.Tensor`):\n Per-token or per-sequence advantage estimates, broadcastable to `(B, T)`.\n completion_mask (`torch.Tensor`):\n Binary mask of shape `(B, T)` where `1` indicates valid completion tokens and `0` padding.\n current_topk_logps (`torch.Tensor` or `None`):\n Log-probabilities of the current policy at the rollout's top-K token IDs, shape `(B, T, K)`.\n Required when `divergence_type` is `\"topk_tv\"` or `\"topk_kl\"`.\n sampling_topk_logps (`torch.Tensor` or `None`):\n Log-probabilities of the sampling policy at the rollout's top-K token IDs, shape `(B, T, K)`.\n Required when `divergence_type` is `\"topk_tv\"` or `\"topk_kl\"`.\n\n Returns:\n `torch.Tensor`:\n Float mask of shape `(B, T)` where `1.0` indicates tokens to keep and `0.0` tokens to mask out.\n \"\"\"\n prob = torch.exp(per_token_logps)\n sampling_prob = torch.exp(sampling_per_token_logps)\n\n delta_low = self.epsilon_low\n delta_high = self.epsilon_high\n\n if self.divergence_type == \"binary_tv\":\n # TV = |ฯ€ - ฮผ|\n divergence = (prob - sampling_prob).abs()\n # Mask tokens where divergence > threshold AND policy moves away from trust region\n invalid_pos = (divergence > delta_high) & (prob > sampling_prob)\n invalid_neg = (divergence > delta_low) & (prob < sampling_prob)\n mask = torch.where(advantages > 0, ~invalid_pos, ~invalid_neg)\n\n elif self.divergence_type == \"binary_kl\":\n # Bernoulli KL: D = ฮผ log(ฮผ/ฯ€) + (1-ฮผ) log((1-ฮผ)/(1-ฯ€))\n kl = sampling_prob * (sampling_per_token_logps - per_token_logps) + (1 - sampling_prob) * (\n torch.log1p(-sampling_prob.clamp(max=1 - 1e-7)) - torch.log1p(-prob.clamp(max=1 - 1e-7))\n )\n\n invalid_pos = (kl > delta_high) & (prob > sampling_prob)\n invalid_neg = (kl > delta_low) & (prob < sampling_prob)\n mask = torch.where(advantages > 0, ~invalid_pos, ~invalid_neg)\n\n elif self.divergence_type in (\"topk_tv\", \"topk_kl\"):\n current_topk_probs = torch.exp(current_topk_logps.float())\n rollout_topk_probs = torch.exp(sampling_topk_logps.float())\n\n # Aggregate remaining probability mass outside top-K into a single rest bucket.\n rollout_rest = (1.0 - rollout_topk_probs.sum(dim=-1)).clamp(min=1e-12)\n current_rest = (1.0 - current_topk_probs.sum(dim=-1)).clamp(min=1e-12)\n\n if self.divergence_type == \"topk_tv\":\n topk_tv = (current_topk_probs - rollout_topk_probs).abs().sum(dim=-1)\n rest_tv = (current_rest - rollout_rest).abs()\n divergence = (topk_tv + rest_tv) / 2.0\n else:\n topk_kl = (rollout_topk_probs * (sampling_topk_logps - current_topk_logps)).sum(dim=-1)\n rest_kl = rollout_rest * (rollout_rest.log() - current_rest.log())\n divergence = topk_kl + rest_kl\n\n invalid_pos = (divergence > delta_high) & (prob > sampling_prob)\n invalid_neg = (divergence > delta_low) & (prob < sampling_prob)\n mask = torch.where(advantages > 0, ~invalid_pos, ~invalid_neg)\n\n else:\n raise ValueError(f\"Unknown divergence_type: {self.divergence_type}\")\n\n return mask.float() * completion_mask\n\n def _compute_loss(self, model, inputs):\n # Compute per-token log probabilities for the model\n prompt_ids, prompt_mask = inputs[\"prompt_ids\"], inputs[\"prompt_mask\"]\n completion_ids, completion_mask = inputs[\"completion_ids\"], inputs[\"completion_mask\"]\n input_ids = torch.cat([prompt_ids, completion_ids], dim=1)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1)\n logits_to_keep = completion_ids.size(1)\n mask = completion_mask if \"tool_mask\" not in inputs else completion_mask * inputs[\"tool_mask\"]\n\n forward_kwargs = {\n \"pixel_values\": inputs.get(\"pixel_values\"),\n \"image_grid_thw\": inputs.get(\"image_grid_thw\"),\n \"num_images\": inputs.get(\"num_images\"),\n \"pixel_attention_mask\": inputs.get(\"pixel_attention_mask\"),\n \"image_sizes\": inputs.get(\"image_sizes\"),\n \"token_type_ids\": inputs.get(\"token_type_ids\"),\n \"mm_token_type_ids\": inputs.get(\"mm_token_type_ids\"),\n }\n\n sampling_topk_token_ids = inputs.get(\"sampling_topk_token_ids\")\n if self.divergence_type.startswith(\"topk_\") and sampling_topk_token_ids is not None:\n per_token_logps, entropies, current_topk_logps = self._get_per_token_logps_with_topk(\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n topk_token_ids=sampling_topk_token_ids,\n compute_entropy=True,\n **forward_kwargs,\n )\n else:\n per_token_logps, entropies = self._get_per_token_logps_and_entropies(\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n compute_entropy=True,\n **forward_kwargs,\n )\n current_topk_logps = None\n\n sampling_per_token_logps = inputs[\"sampling_per_token_logps\"]\n sampling_topk_logps = inputs.get(\"sampling_topk_logps\")\n\n advantages = inputs[\"advantages\"]\n if advantages.dim() == 1:\n advantages = advantages.unsqueeze(1)\n\n # DPPO: compute IS ratio (clamped, detached) and divergence mask\n log_ratio = per_token_logps - sampling_per_token_logps\n ratio = torch.exp(log_ratio.clamp(max=math.log(self.clip_ratio_c))).detach()\n divergence_mask = self._compute_divergence_mask(\n per_token_logps,\n sampling_per_token_logps,\n advantages,\n mask,\n current_topk_logps=current_topk_logps,\n sampling_topk_logps=sampling_topk_logps,\n )\n\n # DPPO loss: -advantages * ratio * mask * log_prob\n per_token_loss = -advantages * ratio * divergence_mask * per_token_logps\n\n # KL divergence with reference model\n if self.beta != 0.0:\n ref_per_token_logps = inputs[\"ref_per_token_logps\"]\n per_token_kl = (\n torch.exp(ref_per_token_logps - per_token_logps) - (ref_per_token_logps - per_token_logps) - 1\n )\n per_token_loss = per_token_loss + self.beta * per_token_kl\n\n mode = \"train\" if self.model.training else \"eval\"\n normalizer = inputs[\"num_items_in_batch\"] / self.accelerator.num_processes\n loss = (per_token_loss * mask).sum() / normalizer\n\n # Log metrics\n completion_token_count = mask.sum().clamp(min=1.0)\n\n def masked_batch_mean(x):\n if x.shape[1] == 1:\n return x.mean()\n return (x * mask).sum() / completion_token_count\n\n if self.beta != 0.0:\n mean_kl = masked_batch_mean(per_token_kl)\n self._metrics[mode][\"kl\"].append(self.accelerator.gather(mean_kl).nanmean().item())\n\n mean_entropy = masked_batch_mean(entropies)\n self._metrics[mode][\"entropy\"].append(self.accelerator.gather(mean_entropy).nanmean().item())\n\n prob_diff = (torch.exp(per_token_logps) - torch.exp(sampling_per_token_logps)).abs()\n self._metrics[mode][\"prob_diff/mean\"].append(\n self.accelerator.gather(masked_batch_mean(prob_diff)).nanmean().item()\n )\n per_seq_max = prob_diff.masked_fill(mask == 0, float(\"-inf\")).max(dim=1).values\n per_seq_min = prob_diff.masked_fill(mask == 0, float(\"inf\")).min(dim=1).values\n self._metrics[mode][\"prob_diff/max\"].append(self.accelerator.gather(per_seq_max).max().item())\n self._metrics[mode][\"prob_diff/min\"].append(self.accelerator.gather(per_seq_min).min().item())\n\n self._metrics[mode][\"advantages/mean\"].append(advantages.mean().item())\n self._metrics[mode][\"advantages/std\"].append(advantages.std().item())\n\n # Log divergence mask statistics (analogous to clip_ratio in GRPO)\n is_masked = (divergence_mask == 0) & (mask > 0)\n is_masked_pos = is_masked & (advantages > 0)\n is_masked_neg = is_masked & (advantages < 0)\n\n mask_ratio_pos = masked_batch_mean(is_masked_pos.float())\n mask_ratio_neg = masked_batch_mean(is_masked_neg.float())\n mask_ratio = masked_batch_mean(is_masked.float())\n\n gathered_mask_ratio_neg = self.accelerator.gather(mask_ratio_neg)\n self._metrics[mode][\"mask_ratio/negative_adv_mean\"].append(gathered_mask_ratio_neg.nanmean().item())\n gathered_mask_ratio_pos = self.accelerator.gather(mask_ratio_pos)\n self._metrics[mode][\"mask_ratio/positive_adv_mean\"].append(gathered_mask_ratio_pos.nanmean().item())\n gathered_mask_ratio = self.accelerator.gather(mask_ratio)\n self._metrics[mode][\"mask_ratio/overall_mean\"].append(gathered_mask_ratio.nanmean().item())\n\n return loss\n" }, { "path": "trl/experimental/gfpo/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .gfpo_config import GFPOConfig\nfrom .gfpo_trainer import GFPOTrainer\n" }, { "path": "trl/experimental/gfpo/gfpo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom ...trainer.grpo_config import GRPOConfig as _GRPOConfig\n\n\n@dataclass\nclass GFPOConfig(_GRPOConfig):\n num_remains_in_group: int | None = field(\n default=None,\n metadata={\n \"help\": \"number inputs remains after group filter function, `'num_remains_in_group'` must be >=2 if given.\"\n },\n )\n\n def __post_init__(self):\n super().__post_init__()\n\n if self.num_remains_in_group is not None and self.num_remains_in_group >= self.num_generations:\n raise ValueError(\n f\"Number remains in Group {self.num_remains_in_group} must be less than num_generations : {self.num_generations}.\"\n )\n" }, { "path": "trl/experimental/gfpo/gfpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport logging\nfrom collections.abc import Callable\nfrom typing import Any\n\nimport torch\nfrom accelerate.utils import gather_object\n\nfrom ...data_utils import apply_chat_template, is_conversational, prepare_multimodal_messages\nfrom ...models.utils import disable_gradient_checkpointing\nfrom ...trainer.grpo_trainer import GRPOTrainer as _GRPOTrainer\nfrom ...trainer.utils import nanmax, nanmin, nanstd, pad\n\n\nlogger = logging.getLogger(__name__)\n\nGroupFilterFunc = Callable[[list[list[Any]], list[list[Any]]], list[list[float]]]\n\n\nclass GFPOTrainer(_GRPOTrainer):\n def __init__(\n self,\n model,\n reward_funcs,\n args=None,\n train_dataset=None,\n eval_dataset=None,\n processing_class=None,\n reward_processing_classes=None,\n group_filter_func=None,\n callbacks=None,\n optimizers=(None, None),\n peft_config=None,\n ):\n super().__init__(\n model=model,\n reward_funcs=reward_funcs,\n args=args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n reward_processing_classes=reward_processing_classes,\n callbacks=callbacks,\n optimizers=optimizers,\n peft_config=peft_config,\n )\n self.group_filter_func = group_filter_func\n self.num_remains_in_group = args.num_remains_in_group\n if self.group_filter_func is None and self.num_remains_in_group is not None:\n raise ValueError(\n f\"Group filter function must not be None when num_remains_in_group ({self.num_remains_in_group}) is given.\"\n )\n if self.group_filter_func is not None and self.num_remains_in_group is None:\n logger.warning(\"Group filter function is not activated since num_remains_in_group is not set\")\n\n def _generate_and_score_completions(self, inputs):\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n\n prompts = [x[\"prompt\"] for x in inputs]\n\n if \"images\" in inputs[0]:\n images = [example.get(\"images\") for example in inputs]\n elif \"image\" in inputs[0]:\n images = [[example.get(\"image\")] if example.get(\"image\") is not None else None for example in inputs]\n else:\n images = None\n # Transformers requires at least one image in the batch, otherwise it throws an error\n if images is not None and all(img_list == [] for img_list in images):\n images = None\n\n # If the prompts are conversational and the inputs contain images, we need to convert the prompts from\n # [{\"role\": \"user\", \"content\": \"What color is the sky?\"}] to\n # [{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": }, {\"type\": \"text\", \"text\": \"What color is the sky?\"}]}]\n if images is not None:\n if not is_conversational(inputs[0]):\n raise ValueError(\n \"Multimodal training requires conversational prompts. It looks like the dataset contains \"\n \"non-conversational inputs, likely because a chat template was applied before passing the dataset \"\n \"to the trainer. Please provide the raw conversational prompts and let the trainer apply the chat \"\n \"template internally.\"\n )\n prompts = [\n prepare_multimodal_messages(prompt, image_list)\n for prompt, image_list in zip(prompts, images, strict=True)\n ]\n\n prompt_ids_list, completion_ids_list, num_items_in_batch, sampling_per_token_logps_list, extra_fields = (\n self._generate(prompts)\n )\n\n # Convert lists of token IDs to padded tensors\n prompt_ids = [torch.tensor(ids) for ids in prompt_ids_list]\n prompt_mask = [torch.ones_like(ids, dtype=torch.long) for ids in prompt_ids]\n prompt_ids = pad(\n prompt_ids,\n padding_value=self.pad_token_id,\n padding_side=\"left\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n prompt_mask = pad(\n prompt_mask,\n padding_value=0,\n padding_side=\"left\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n completion_ids = [torch.tensor(ids) for ids in completion_ids_list]\n completion_mask = [torch.ones_like(ids, dtype=torch.long) for ids in completion_ids]\n completion_ids = pad(\n completion_ids,\n padding_value=self.pad_token_id,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n completion_mask = pad(\n completion_mask,\n padding_value=0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n if sampling_per_token_logps_list is not None:\n sampling_per_token_logps = [torch.tensor(logps) for logps in sampling_per_token_logps_list]\n sampling_per_token_logps = pad(\n sampling_per_token_logps,\n padding_value=0.0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n else:\n sampling_per_token_logps = None\n\n # If mask_truncated_completions is enabled, zero out truncated completions in completion_mask\n if self.mask_truncated_completions:\n eos_and_pad = [self.eos_token_id, self.pad_token_id]\n is_truncated = torch.tensor([ids[-1] not in eos_and_pad for ids in completion_ids_list], device=device)\n completion_mask = completion_mask * (~is_truncated).unsqueeze(1).int()\n\n # Concatenate prompt_mask with completion_mask for logit computation\n prompt_completion_ids = torch.cat([prompt_ids, completion_ids], dim=1) # (B, P+C)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1) # (B, P+C)\n\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n batch_size = self.args.per_device_train_batch_size if mode == \"train\" else self.args.per_device_eval_batch_size\n\n num_images = [len(img_list) for img_list in images] if images is not None else None\n\n # Get forward_kwargs for models with multimodal inputs\n if images is not None:\n prompts_text = [\n apply_chat_template({\"prompt\": prompt}, self.processing_class)[\"prompt\"] for prompt in prompts\n ]\n prompt_inputs = self.processing_class(images=images, text=prompts_text, padding=True, return_tensors=\"pt\")\n prompt_inputs = super()._prepare_inputs(prompt_inputs)\n forward_kwargs = {k: v for k, v in prompt_inputs.items() if k not in [\"input_ids\", \"attention_mask\"]}\n else:\n forward_kwargs = {}\n\n # If token_type_ids are used, extend them with zeros for the completion part\n if \"token_type_ids\" in forward_kwargs:\n token_type_ids = forward_kwargs[\"token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n # Needed only with pad_to_multiple_of: otherwise prompt_ids and token_type_ids must have equal len\n padding_size = prompt_ids.size(1) - token_type_ids.size(1)\n if padding_size > 0:\n token_type_ids = torch.cat(\n [token_type_ids.new_zeros((token_type_ids.size(0), padding_size)), token_type_ids], dim=1\n )\n forward_kwargs[\"token_type_ids\"] = torch.cat(\n [token_type_ids, token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n # If mm_token_type_ids are used, extend them with zeros for the completion part\n if \"mm_token_type_ids\" in forward_kwargs:\n mm_token_type_ids = forward_kwargs[\"mm_token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n # Needed only with pad_to_multiple_of: otherwise prompt_ids and mm_token_type_ids must have equal len\n padding_size = prompt_ids.size(1) - mm_token_type_ids.size(1)\n if padding_size > 0:\n mm_token_type_ids = torch.cat(\n [mm_token_type_ids.new_zeros((mm_token_type_ids.size(0), padding_size)), mm_token_type_ids],\n dim=1,\n )\n forward_kwargs[\"mm_token_type_ids\"] = torch.cat(\n [mm_token_type_ids, mm_token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n\n # When gradient checkpointing is enabled with use_reentrant=True (non default), calling the model inside a\n # torch.no_grad() block triggers a harmless PyTorch warning (\"None of the inputs have requires_grad=True\").\n # Temporarily disable checkpointing to avoid this warning during inference.\n with torch.no_grad(), disable_gradient_checkpointing(self.model, self.args.gradient_checkpointing_kwargs):\n # If the generation and optimization steps are misalignedโ€”i.e., if generation does not occur at the end of\n # a full optimizer step (when gradient_accumulation_steps is not a multiple of generate_every)โ€”then the\n # samples may come from an earlier version of the model. In that case, we need to track old_per_token_logps\n # for importance sampling. If the steps are aligned, importance sampling isn't necessary and we set\n # old_per_token_logps to None.\n # When using vLLM, we always compute old_per_token_logps for importance sampling, it was shown that the\n # distribution mismatch between vLLM and the training model can be large and harm the training.\n generate_every = self.args.steps_per_generation * self.num_iterations # generation frequency\n if self.args.gradient_accumulation_steps % generate_every != 0 or (\n self.use_vllm and self.vllm_importance_sampling_correction\n ):\n old_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n old_per_token_logps = None\n\n # Compute the importance sampling ratio when using vLLM, to correct for potential distribution mismatch\n if self.use_vllm and self.vllm_importance_sampling_correction:\n importance_sampling_ratio = torch.exp(old_per_token_logps - sampling_per_token_logps)\n importance_sampling_ratio = torch.clamp(\n importance_sampling_ratio, max=self.vllm_importance_sampling_cap\n )\n\n # Compute the per-token log probabilities for the reference model\n if self.beta != 0.0:\n if self.ref_model is not None:\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.ref_model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n with self.accelerator.unwrap_model(self.model).disable_adapter():\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n ref_per_token_logps = None\n\n # Decode\n prompts_text = self.processing_class.batch_decode(prompt_ids, skip_special_tokens=True)\n completions_text = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n if is_conversational(inputs[0]):\n completions = []\n for prompt, completion in zip(prompts, completions_text, strict=True):\n bootstrap = prompt.pop()[\"content\"] if prompt[-1][\"role\"] == \"assistant\" else \"\"\n if isinstance(bootstrap, list): # for VLM, the format might be [{\"type\": \"text\", \"text\": \"...\"}]\n assert len(bootstrap) == 1 and bootstrap[0][\"type\"] == \"text\"\n bootstrap = bootstrap[0][\"text\"]\n completions.append([{\"role\": \"assistant\", \"content\": bootstrap + completion}])\n else:\n completions = completions_text\n\n # Calculate rewards for each reward function. rewards_per_func aggregates rewards across all processes. This is\n # important because rewards will be normalized per group, and completions are distributed. We will later slice\n # rewards_per_func to extract each process's subset.\n rewards_per_func = self._calculate_rewards(inputs, prompts, completions, completion_ids_list)\n\n # Apply weights to each reward function's output and sum\n rewards = (rewards_per_func * self.reward_weights.to(device).unsqueeze(0)).nansum(dim=1)\n\n num_in_group = self.num_generations\n num_inputs_in_device = len(prompts)\n\n if self.num_remains_in_group is not None and mode == \"train\":\n num_in_group = self.num_remains_in_group\n\n all_completions = gather_object(completions)\n\n group_filter_scores = self.group_filter_func(\n group_completions=[\n all_completions[i : i + 1 * self.num_generations]\n for i in range(len(all_completions) // self.num_generations)\n ],\n group_rewards=rewards.view(-1, self.num_generations).tolist(),\n )\n group_filter_scores = torch.tensor(group_filter_scores, device=device)\n\n _, group_local_indices = torch.topk(group_filter_scores, self.num_remains_in_group, dim=-1)\n group_row_offsets = torch.arange(0, len(all_completions), self.num_generations, device=device).unsqueeze(1)\n group_global_indices = group_row_offsets + group_local_indices\n group_global_indices = group_global_indices.flatten()\n\n rewards = rewards[group_global_indices].contiguous()\n rewards_per_func = rewards_per_func[group_global_indices, :].contiguous()\n\n num_inputs_in_device = int(len(prompts) / self.num_generations * self.num_remains_in_group)\n\n # Compute grouped-wise rewards\n mean_grouped_rewards = rewards.view(-1, num_in_group).mean(dim=1)\n\n # Normalize the rewards to compute the advantages\n mean_grouped_rewards = mean_grouped_rewards.repeat_interleave(num_in_group, dim=0)\n advantages = rewards - mean_grouped_rewards\n\n if self.scale_rewards in [\"group\", \"none\"]:\n # If self.scale_rewards = \"none\", we'll still log group level std\n std_rewards = rewards.view(-1, num_in_group).std(dim=1)\n std_rewards = std_rewards.repeat_interleave(num_in_group, dim=0)\n elif self.scale_rewards == \"batch\":\n # Compute global std\n std_rewards = rewards.std().expand_as(rewards)\n else:\n raise ValueError(\n f\"Invalid value for scale_rewards: {self.scale_rewards}. Must be one of 'batch', 'group', or 'none'.\"\n )\n\n is_std_zero = torch.isclose(std_rewards, torch.zeros_like(std_rewards))\n if self.scale_rewards != \"none\":\n advantages = advantages / (std_rewards + 1e-4)\n\n # Slice to keep only the local part of the data\n process_slice = slice(\n self.accelerator.process_index * num_inputs_in_device,\n (self.accelerator.process_index + 1) * num_inputs_in_device,\n )\n all_process_advantages = advantages.clone() # keep the aggregated advantages for logging\n advantages = advantages[process_slice]\n\n if self.num_remains_in_group is not None and mode == \"train\":\n local_input_indices_to_keep = group_global_indices[process_slice] - self.accelerator.process_index * len(\n prompts\n ) # step is length of prompts\n\n prompt_ids = prompt_ids[local_input_indices_to_keep].contiguous()\n prompt_mask = prompt_mask[local_input_indices_to_keep].contiguous()\n completion_ids = completion_ids[local_input_indices_to_keep].contiguous()\n completion_mask = completion_mask[local_input_indices_to_keep].contiguous()\n attention_mask = attention_mask[local_input_indices_to_keep].contiguous()\n completion_lengths = completion_mask.sum(1)\n agg_completion_lengths = self.accelerator.gather(completion_lengths)\n num_items_in_batch = agg_completion_lengths.sum()\n\n if sampling_per_token_logps is not None:\n sampling_per_token_logps = sampling_per_token_logps[local_input_indices_to_keep].contiguous()\n if old_per_token_logps is not None:\n old_per_token_logps = old_per_token_logps[local_input_indices_to_keep].contiguous()\n if ref_per_token_logps is not None:\n ref_per_token_logps = ref_per_token_logps[local_input_indices_to_keep].contiguous()\n if self.use_vllm and self.vllm_importance_sampling_correction:\n importance_sampling_ratio = importance_sampling_ratio[local_input_indices_to_keep].contiguous()\n\n # Calculate mean reward per function, but only for samples where the function was applied (non-NaN values)\n for i, reward_func_name in enumerate(self.reward_func_names):\n mean_rewards = torch.nanmean(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/mean\"].append(mean_rewards)\n std_func_rewards = nanstd(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/std\"].append(std_func_rewards)\n self._metrics[mode][\"reward\"].append(mean_grouped_rewards.mean().item())\n self._metrics[mode][\"reward_std\"].append(std_rewards.mean().item())\n self._metrics[mode][\"frac_reward_zero_std\"].append(is_std_zero.float().mean().item())\n\n # Log prompt and completion texts\n all_prompts_text = gather_object(prompts_text)\n all_completions_text = gather_object(completions_text)\n all_images = gather_object(images) if images is not None else None\n if self.num_remains_in_group is not None and mode == \"train\":\n group_global_indices_list = group_global_indices.tolist()\n all_prompts_text = [all_prompts_text[i] for i in group_global_indices_list]\n all_completions_text = [all_completions_text[i] for i in group_global_indices_list]\n if images is not None:\n all_images = [all_images[i] for i in group_global_indices_list]\n\n self._logs[\"prompt\"].extend(all_prompts_text)\n self._logs[\"completion\"].extend(all_completions_text)\n for i, name in enumerate(self.reward_func_names):\n self._logs[\"rewards\"][name].extend(rewards_per_func[:, i].tolist())\n self._logs[\"advantages\"].extend(all_process_advantages.tolist())\n\n if images is not None:\n self._logs[\"images\"].extend(all_images)\n\n if self.use_vllm and self.vllm_importance_sampling_correction:\n delta = torch.abs(old_per_token_logps - sampling_per_token_logps)\n delta = delta[completion_mask.bool()]\n mean_delta = torch.mean(delta) if delta.numel() > 0 else torch.tensor(0.0, device=device)\n max_delta = torch.max(delta) if delta.numel() > 0 else torch.tensor(0.0, device=device)\n self._metrics[mode][\"sampling/sampling_logp_difference/mean\"].append(\n self.accelerator.gather(mean_delta).mean().item()\n )\n self._metrics[mode][\"sampling/sampling_logp_difference/max\"].append(\n self.accelerator.gather(max_delta).max().item()\n )\n\n flat_is_ratio = importance_sampling_ratio[completion_mask.bool()]\n min_importance_sampling_ratio = (\n torch.min(flat_is_ratio) if flat_is_ratio.numel() > 0 else torch.tensor(0.0, device=device)\n )\n mean_importance_sampling_ratio = (\n torch.mean(flat_is_ratio) if flat_is_ratio.numel() > 0 else torch.tensor(0.0, device=device)\n )\n max_importance_sampling_ratio = (\n torch.max(flat_is_ratio) if flat_is_ratio.numel() > 0 else torch.tensor(0.0, device=device)\n )\n self._metrics[mode][\"sampling/importance_sampling_ratio/min\"].append(\n nanmin(self.accelerator.gather(min_importance_sampling_ratio)).item()\n )\n self._metrics[mode][\"sampling/importance_sampling_ratio/mean\"].append(\n self.accelerator.gather(mean_importance_sampling_ratio).nanmean().item()\n )\n self._metrics[mode][\"sampling/importance_sampling_ratio/max\"].append(\n nanmax(self.accelerator.gather(max_importance_sampling_ratio)).item()\n )\n\n output = {\n \"prompt_ids\": prompt_ids,\n \"prompt_mask\": prompt_mask,\n \"completion_ids\": completion_ids,\n \"completion_mask\": completion_mask,\n \"advantages\": advantages,\n \"num_items_in_batch\": num_items_in_batch,\n }\n if old_per_token_logps is not None:\n output[\"old_per_token_logps\"] = old_per_token_logps\n if self.use_vllm and self.vllm_importance_sampling_correction:\n output[\"importance_sampling_ratio\"] = importance_sampling_ratio\n if ref_per_token_logps is not None:\n output[\"ref_per_token_logps\"] = ref_per_token_logps\n if \"pixel_values\" in forward_kwargs:\n output[\"pixel_values\"] = forward_kwargs[\"pixel_values\"]\n if \"image_grid_thw\" in forward_kwargs:\n output[\"image_grid_thw\"] = forward_kwargs[\"image_grid_thw\"]\n if \"pixel_attention_mask\" in forward_kwargs:\n output[\"pixel_attention_mask\"] = forward_kwargs[\"pixel_attention_mask\"]\n if \"image_sizes\" in forward_kwargs:\n output[\"image_sizes\"] = forward_kwargs[\"image_sizes\"]\n if \"token_type_ids\" in forward_kwargs:\n output[\"token_type_ids\"] = forward_kwargs[\"token_type_ids\"]\n if images is not None:\n output[\"num_images\"] = num_images\n return output\n" }, { "path": "trl/experimental/gkd/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .gkd_config import GKDConfig\nfrom .gkd_trainer import GKDTrainer\n\n\n__all__ = [\"GKDConfig\", \"GKDTrainer\"]\n" }, { "path": "trl/experimental/gkd/gkd_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom ...trainer.sft_config import SFTConfig\n\n\n@dataclass\nclass GKDConfig(SFTConfig):\n \"\"\"\n Configuration class for [`experimental.gkd.GKDTrainer`].\n\n This class includes only the parameters that are specific to GKD training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] and [`SFTConfig`] documentation.\n\n Args:\n temperature (`float`, *optional*, defaults to `0.9`):\n Temperature for sampling. The higher the temperature, the more random the completions.\n lmbda (`float`, *optional*, defaults to `0.5`):\n Lambda parameter that controls the student data fraction (i.e., the proportion of on-policy\n student-generated outputs).\n beta (`float`, *optional*, defaults to `0.5`):\n Interpolation coefficient between `0.0` and `1.0` of the Generalized Jensen-Shannon Divergence loss. When\n beta is `0.0`, the loss is the KL divergence. When beta is `1.0`, the loss is the Inverse KL Divergence.\n max_new_tokens (`int`, *optional*, defaults to `128`):\n Maximum number of tokens to generate per completion.\n teacher_model_name_or_path (`str`, *optional*):\n Model name or path of the teacher model. If `None`, the teacher model will be the same as the model being\n trained.\n teacher_model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the teacher model\n from a string.\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model.\n seq_kd (`bool`, *optional*, defaults to `False`):\n Seq_kd parameter that controls whether to perform Sequence-Level KD (can be viewed as supervised FT on\n teacher-generated output).\n \"\"\"\n\n _VALID_DICT_FIELDS = SFTConfig._VALID_DICT_FIELDS + [\"teacher_model_init_kwargs\"]\n\n temperature: float = field(\n default=0.9,\n metadata={\"help\": \"Temperature for sampling. The higher the temperature, the more random the completions.\"},\n )\n lmbda: float = field(\n default=0.5,\n metadata={\n \"help\": \"Lambda parameter that controls the student data fraction (i.e., the proportion of on-policy \"\n \"student-generated outputs).\"\n },\n )\n beta: float = field(\n default=0.5,\n metadata={\n \"help\": \"Interpolation coefficient between `0.0` and `1.0` of the Generalized Jensen-Shannon Divergence \"\n \"loss. When beta is `0.0`, the loss is the KL divergence. When beta is `1.0`, the loss is the Inverse KL \"\n \"Divergence.\"\n },\n )\n max_new_tokens: int = field(\n default=128,\n metadata={\"help\": \"Maximum number of tokens to generate per completion.\"},\n )\n teacher_model_name_or_path: str | None = field(\n default=None,\n metadata={\n \"help\": \"Model name or path of the teacher model. If `None`, the teacher model will be the same as the \"\n \"model being trained.\"\n },\n )\n teacher_model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the \"\n \"teacher model from a string.\"\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropouts in `model`.\"},\n )\n seq_kd: bool = field(\n default=False,\n metadata={\n \"help\": \"Seq_kd parameter that controls whether to perform Sequence-Level KD (can be viewed as supervised \"\n \"FT on teacher-generated output).\"\n },\n )\n\n def __post_init__(self):\n super().__post_init__()\n # check lmbda and beta are in the range [0, 1]\n if self.lmbda < 0.0 or self.lmbda > 1.0:\n raise ValueError(\"lmbda must be in the range [0.0, 1.0].\")\n if self.beta < 0.0 or self.beta > 1.0:\n raise ValueError(\"beta must be in the range [0.0, 1.0].\")\n" }, { "path": "trl/experimental/gkd/gkd_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport random\nimport textwrap\nfrom collections.abc import Callable\nfrom typing import Any\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom datasets import Dataset\nfrom transformers import (\n AutoModelForCausalLM,\n BaseImageProcessor,\n DataCollator,\n FeatureExtractionMixin,\n GenerationConfig,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n)\nfrom transformers.trainer_utils import EvalPrediction\nfrom transformers.utils import is_liger_kernel_available, is_peft_available\n\nfrom ...models import prepare_deepspeed\nfrom ...models.utils import unwrap_model_for_generation\nfrom ...trainer.sft_trainer import SFTTrainer\nfrom ...trainer.utils import disable_dropout_in_model\nfrom ..utils import DataCollatorForChatML, empty_cache\nfrom .gkd_config import GKDConfig\n\n\nif is_peft_available():\n from peft import PeftConfig\n\nif is_liger_kernel_available():\n from liger_kernel.chunked_loss import LigerFusedLinearJSDLoss\n\n\nclass GKDTrainer(SFTTrainer):\n \"\"\"Trainer for Generalized Knowledge Distillation (GKD) of language models.\n\n For details on GKD, see the paper: [On-Policy Distillation of Language Models: Learning from Self-Generated\n Mistakes](https://huggingface.co/papers/2306.13649).\n\n Args:\n model ([`~transformers.PreTrainedModel`] or `torch.nn.Module` or `str`, *optional*):\n Model to be trained, or the string identifier of the model to be instantiated from a pretrained model.\n teacher_model ([`~transformers.PreTrainedModel`] or `torch.nn.Module` or `str`, *optional*):\n Teacher model for knowledge distillation, or the string identifier of the model to be instantiated from a\n pretrained model.\n args ([`experimental.gkd.GKDConfig`], *optional*):\n Training arguments.\n data_collator ([`~transformers.DataCollator`], *optional*):\n Data collator to batch samples from the dataset. It defaults to a\n [`experimental.utils.DataCollatorForChatML`] using the `processing_class`.\n train_dataset ([`~datasets.Dataset`], *optional*):\n Dataset for training.\n eval_dataset ([`~datasets.Dataset`] or `dict` of [`~datasets.Dataset`], *optional*):\n Dataset for evaluation.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*):\n Class to process the data.\n compute_metrics (`Callable`, *optional*):\n Function to compute metrics at evaluation. Must take in an [`~transformers.EvalPrediction`] and return a\n dictionary string to float.\n callbacks (`list` of [`~transformers.TrainerCallback`], *optional*):\n Callbacks to use during training.\n optimizers (`tuple` of `torch.optim.Optimizer` and `torch.optim.lr_scheduler.LambdaLR`, *optional*, defaults to `(None, None)`):\n Tuple containing the optimizer and the learning rate scheduler to use for training.\n preprocess_logits_for_metrics (`Callable`, *optional*):\n Function to preprocess the logits before computing the metrics. Must take in the `logits` and `labels` and\n return the logits to be used for metrics computation.\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration to use PEFT for training. If `None`, PEFT is not used. If provided, the `model` will be\n wrapped with the specified PEFT adapter.\n formatting_func (`Callable`, *optional*):\n Function to format the dataset. Must take in an example and return an example.\n \"\"\"\n\n _tag_names = [\"trl\", \"gkd\"]\n _name = \"GKD\"\n _paper = {\n \"title\": \"On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes\",\n \"id\": \"2306.13649\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @inproceedings{agarwal2024on-policy,\n title = {{On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes}},\n author = {Rishabh Agarwal and Nino Vieillard and Yongchao Zhou and Piotr Stanczyk and Sabela Ramos Garea and Matthieu Geist and Olivier Bachem},\n year = 2024,\n booktitle = {The Twelfth International Conference on Learning Representations, {ICLR} 2024, Vienna, Austria, May 7-11, 2024},\n publisher = {OpenReview.net},\n url = {https://openreview.net/forum?id=3zKtaqxLhW},\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module | str | None = None,\n teacher_model: PreTrainedModel | nn.Module | str = None,\n args: GKDConfig | None = None,\n data_collator: DataCollator | None = None, # type: ignore\n train_dataset: Dataset | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n processing_class: PreTrainedTokenizerBase\n | BaseImageProcessor\n | FeatureExtractionMixin\n | ProcessorMixin\n | None = None,\n compute_metrics: Callable[[EvalPrediction], dict] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n peft_config: \"PeftConfig | None\" = None,\n formatting_func: Callable | None = None,\n ):\n # Ensure Trainer does not drop non-signature columns used by the collator (e.g., \"prompts\")\n args.remove_unused_columns = False\n # Respect a user-provided data_collator; otherwise, provide a ChatML collator that\n if data_collator is None:\n data_collator = DataCollatorForChatML(tokenizer=processing_class, max_length=args.max_length)\n\n # Ensure SFTTrainer does not pre-process the dataset when using a ChatML collator,\n # so that raw conversational fields (e.g., \"messages\") remain available to the collator.\n if args.dataset_kwargs is None:\n args.dataset_kwargs = {\"skip_prepare_dataset\": True}\n else:\n args.dataset_kwargs[\"skip_prepare_dataset\"] = True\n\n # Liger fused GKD loss (JSD)\n self.use_liger_gkd_loss = False\n if args.use_liger_kernel:\n self.liger_jsd_loss = LigerFusedLinearJSDLoss(\n beta=args.beta,\n ignore_index=-100,\n temperature=args.temperature,\n compiled=False,\n )\n self.use_liger_gkd_loss = True\n\n super().__init__(\n model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n peft_config=peft_config,\n formatting_func=formatting_func,\n )\n\n if args.teacher_model_init_kwargs is None:\n teacher_model_init_kwargs = {}\n elif not isinstance(teacher_model, str):\n raise ValueError(\n \"You passed teacher_model_init_kwargs to the GKDConfig, but your teacher_model is already instantiated.\"\n )\n else:\n teacher_model_init_kwargs = args.teacher_model_init_kwargs\n teacher_model_init_kwargs[\"dtype\"] = (\n teacher_model_init_kwargs[\"dtype\"]\n if teacher_model_init_kwargs[\"dtype\"] in [\"auto\", None]\n else getattr(torch, teacher_model_init_kwargs[\"dtype\"])\n )\n\n if isinstance(teacher_model, str):\n teacher_model = AutoModelForCausalLM.from_pretrained(teacher_model, **teacher_model_init_kwargs)\n\n # Disable dropout in the model\n if args.disable_dropout:\n disable_dropout_in_model(self.model)\n\n if self.is_deepspeed_enabled:\n self.teacher_model = prepare_deepspeed(teacher_model, self.accelerator)\n else:\n self.teacher_model = self.accelerator.prepare_model(teacher_model, evaluation_mode=True)\n\n self.lmbda = args.lmbda\n self.beta = args.beta\n self.temperature = args.temperature\n self.seq_kd = args.seq_kd\n\n generation_kwargs = {\n \"max_new_tokens\": args.max_new_tokens,\n \"temperature\": args.temperature,\n \"do_sample\": True,\n \"top_k\": 0,\n \"use_cache\": False if args.gradient_checkpointing else True,\n \"pad_token_id\": self.processing_class.pad_token_id,\n }\n self.generation_config = GenerationConfig(**generation_kwargs)\n # Keep training-specific generation kwargs to overwrite model's original generation config\n self.generation_kwargs = generation_kwargs\n # Set custom EOS tokens if they are specified by the model's generation\n # config. This is important for models with the Llama 3 chat template,\n # which use special tokens <|eot_id|> and <|eom_id|> to mark the end of\n # turns or messages.\n if (\n hasattr(self.model.generation_config, \"eos_token_id\")\n and self.model.generation_config.eos_token_id is not None\n ):\n self.generation_config.eos_token_id = self.model.generation_config.eos_token_id\n\n @staticmethod\n def generalized_jsd_loss(\n student_logits, teacher_logits, labels=None, beta=0.5, temperature=1.0, reduction=\"batchmean\"\n ):\n \"\"\"\n Compute the generalized Jensen-Shannon Divergence loss for knowledge distillation using F.kl_div. See Eq. (1)\n of https://huggingface.co/papers/2306.13649 for the definition.\n\n Args:\n student_logits:\n Tensor of shape (batch_size, sequence_length, vocab_size)\n teacher_logits:\n Tensor of shape (batch_size, sequence_length, vocab_size)\n labels:\n Tensor of shape (batch_size, sequence_length) with -100 for padding tokens to ignore when computing\n loss\n beta:\n Interpolation coefficient between 0 and 1 (default: 0.5)\n temperature:\n Softmax temperature (default: 1.0)\n reduction:\n Specifies the reduction to apply to the output (default: 'batchmean')\n\n Returns:\n loss: Scalar tensor with the generalized JSD loss\n \"\"\"\n\n # Apply temperature scaling\n student_logits = student_logits / temperature\n teacher_logits = teacher_logits / temperature\n\n # Compute log probabilities for student and probabilities for teacher\n student_log_probs = F.log_softmax(student_logits, dim=-1)\n teacher_log_probs = F.log_softmax(teacher_logits, dim=-1)\n\n if beta == 0:\n jsd = F.kl_div(student_log_probs, teacher_log_probs, reduction=\"none\", log_target=True)\n elif beta == 1:\n jsd = F.kl_div(teacher_log_probs, student_log_probs, reduction=\"none\", log_target=True)\n else:\n # Compute the log of the mixture distribution\n # log(a + b) = log(exp(log(a)) + exp(log(b))) -> for mixture\n beta = torch.tensor(beta, dtype=student_log_probs.dtype, device=student_log_probs.device)\n mixture_log_probs = torch.logsumexp(\n torch.stack([student_log_probs + torch.log1p(-beta), teacher_log_probs + torch.log(beta)]),\n dim=0,\n )\n\n # Compute KL divergences using F.kl_div\n # PyTorch differs from the standard mathematical definition, so the order of the probability distributions is swapped compared to that defined in the paper.\n kl_teacher = F.kl_div(mixture_log_probs, teacher_log_probs, reduction=\"none\", log_target=True)\n kl_student = F.kl_div(mixture_log_probs, student_log_probs, reduction=\"none\", log_target=True)\n\n # Compute the Generalized Jensen-Shannon Divergence\n jsd = beta * kl_teacher + (1 - beta) * kl_student\n\n # Masking\n if labels is not None:\n mask = labels != -100\n jsd = jsd[mask]\n\n # Apply reduction\n if reduction == \"batchmean\":\n return jsd.sum() / mask.sum() if labels is not None else jsd.sum() / jsd.size(0)\n elif reduction == \"sum\":\n return jsd.sum()\n elif reduction == \"mean\":\n return jsd.mean()\n else:\n return jsd\n\n def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):\n if self.use_liger_gkd_loss:\n # Forward only through the base models (avoid lm_head to save memory)\n unwrapped_student = self.accelerator.unwrap_model(model)\n if hasattr(unwrapped_student, \"get_decoder\") and unwrapped_student.get_decoder() is not None:\n base_student = unwrapped_student.get_decoder()\n else:\n base_student = getattr(\n unwrapped_student, getattr(unwrapped_student, \"base_model_prefix\", \"model\"), unwrapped_student\n )\n\n student_outputs = base_student(\n input_ids=inputs[\"input_ids\"],\n attention_mask=inputs[\"attention_mask\"],\n use_cache=False,\n )\n\n self.teacher_model.eval()\n unwrapped_teacher = self.accelerator.unwrap_model(self.teacher_model)\n if hasattr(unwrapped_teacher, \"get_decoder\") and unwrapped_teacher.get_decoder() is not None:\n base_teacher = unwrapped_teacher.get_decoder()\n else:\n base_teacher = getattr(\n unwrapped_teacher, getattr(unwrapped_teacher, \"base_model_prefix\", \"model\"), unwrapped_teacher\n )\n with torch.no_grad():\n teacher_outputs = base_teacher(\n input_ids=inputs[\"input_ids\"],\n attention_mask=inputs[\"attention_mask\"],\n use_cache=False,\n )\n\n # hidden states (shifted)\n student_hidden = student_outputs.last_hidden_state[:, :-1]\n teacher_hidden = teacher_outputs.last_hidden_state[:, :-1]\n\n # Release full outputs to free memory\n del student_outputs, teacher_outputs\n\n # labels mask and labels (shifted)\n labels_mask = inputs[\"labels\"] != -100\n masked_input_ids = torch.where(\n labels_mask, inputs[\"input_ids\"], torch.full_like(inputs[\"input_ids\"], -100)\n )\n true_labels = masked_input_ids[:, 1:].contiguous()\n\n # Release intermediate tensors\n del labels_mask, masked_input_ids\n\n # heads\n student_head = unwrapped_student.get_output_embeddings()\n teacher_head = unwrapped_teacher.get_output_embeddings()\n\n # liger fused jsd loss\n loss = self.liger_jsd_loss(\n student_input=student_hidden,\n student_weight=student_head.weight,\n teacher_input=teacher_hidden,\n teacher_weight=teacher_head.weight,\n true_labels=true_labels,\n student_bias=getattr(student_head, \"bias\", None),\n teacher_bias=getattr(teacher_head, \"bias\", None),\n )\n\n # Release hidden states after loss computation\n del student_hidden, teacher_hidden, true_labels\n else:\n # compute student output\n student_outputs = model(\n input_ids=inputs[\"input_ids\"],\n attention_mask=inputs[\"attention_mask\"],\n )\n\n # compute teacher output in eval mode\n self.teacher_model.eval()\n with torch.no_grad():\n teacher_outputs = self.teacher_model(\n input_ids=inputs[\"input_ids\"],\n attention_mask=inputs[\"attention_mask\"],\n )\n\n # slice the logits for the generated tokens using the inputs[\"prompts\"] lengths\n prompt_lengths = inputs[\"prompts\"].shape[1]\n shifted_student_logits = student_outputs.logits[:, prompt_lengths - 1 : -1, :]\n shifted_teacher_logits = teacher_outputs.logits[:, prompt_lengths - 1 : -1, :]\n shifted_labels = inputs[\"labels\"][:, prompt_lengths:]\n\n # compute loss\n loss = self.generalized_jsd_loss(\n student_logits=shifted_student_logits,\n teacher_logits=shifted_teacher_logits,\n labels=shifted_labels,\n beta=self.beta,\n )\n\n # empty cache\n empty_cache()\n\n # Return loss\n return (loss, student_outputs) if return_outputs else loss\n\n @staticmethod\n def generate_on_policy_outputs(model, inputs, generation_config, pad_token_id=None):\n # Generate output with respect to the prompt-only\n generated_outputs = model.generate(\n input_ids=inputs[\"prompts\"],\n attention_mask=inputs.get(\"prompt_attention_mask\", None),\n generation_config=generation_config,\n return_dict_in_generate=True,\n )\n\n # Get the generated token IDs\n generated_tokens = generated_outputs.sequences\n # Calculate new attention mask\n new_attention_mask = torch.ones_like(generated_tokens)\n new_labels = generated_tokens.clone()\n\n # If there's pad_token_id, set attention mask to 0 for padding tokens\n if pad_token_id is not None:\n new_labels[new_labels == pad_token_id] = -100\n new_attention_mask[generated_tokens == pad_token_id] = 0\n\n return generated_tokens, new_attention_mask, new_labels\n\n def training_step(\n self, model: nn.Module, inputs: dict[str, torch.Tensor | Any], num_items_in_batch: int | None = None\n ) -> torch.Tensor:\n \"\"\"\n Perform a training step for the Generalized Knowledge Distillation (GKD) model.\n\n This method implements the on-policy learning approach described in the GKD paper. With probability\n `self.lmbda`, it generates new responses using the student model, which are then used for training instead of\n the original inputs.\n \"\"\"\n if self.seq_kd:\n with (\n unwrap_model_for_generation(\n self.teacher_model,\n self.accelerator,\n generation_kwargs=self.generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as unwrapped_model\n ):\n new_input_ids, new_attention_mask, new_labels = self.generate_on_policy_outputs(\n unwrapped_model, inputs, self.generation_config, self.processing_class.pad_token_id\n )\n inputs[\"input_ids\"] = new_input_ids\n inputs[\"attention_mask\"] = new_attention_mask\n inputs[\"labels\"] = new_labels\n if random.random() <= self.lmbda:\n with (\n unwrap_model_for_generation(\n model,\n self.accelerator,\n generation_kwargs=self.generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as unwrapped_model\n ):\n new_input_ids, new_attention_mask, new_labels = self.generate_on_policy_outputs(\n unwrapped_model, inputs, self.generation_config, self.processing_class.pad_token_id\n )\n inputs[\"input_ids\"] = new_input_ids\n inputs[\"attention_mask\"] = new_attention_mask\n inputs[\"labels\"] = new_labels\n\n loss = super().training_step(model, inputs, num_items_in_batch)\n return loss\n" }, { "path": "trl/experimental/gold/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .gold_config import GOLDConfig\nfrom .gold_trainer import GOLDTrainer\n\n\n__all__ = [\"GOLDConfig\", \"GOLDTrainer\"]\n" }, { "path": "trl/experimental/gold/gold.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl @ git+https://github.com/huggingface/trl.git\",\n# \"peft\",\n# \"trackio\",\n# ]\n# ///\n\n# docstyle-ignore\n\"\"\"\n# Full training:\npython trl/experimental/gold/gold.py \\\n --model_name_or_path meta-llama/Llama-3.2-1B-Instruct \\\n --teacher_model_name_or_path Qwen/Qwen2-1.5B-Instruct \\\n --dataset_name trl-lib/chatbot_arena_completions \\\n --learning_rate 2e-5 \\\n --per_device_train_batch_size 4 \\\n --gradient_accumulation_steps 8 \\\n --output_dir gold-model \\\n --num_train_epochs 1 \\\n --push_to_hub\n\n# LoRA:\npython trl/experimental/gold/gold.py \\\n --model_name_or_path meta-llama/Llama-3.2-1B-Instruct \\\n --teacher_model_name_or_path Qwen/Qwen2-1.5B-Instruct \\\n --dataset_name trl-lib/chatbot_arena_completions \\\n --learning_rate 2e-4 \\\n --per_device_train_batch_size 4 \\\n --gradient_accumulation_steps 8 \\\n --output_dir gold-model \\\n --num_train_epochs 1 \\\n --push_to_hub \\\n --use_peft \\\n --lora_r 64 \\\n --lora_alpha 16\n\"\"\"\n\nimport logging\n\nfrom datasets import load_dataset\nfrom transformers import AutoTokenizer, GenerationConfig\n\nfrom trl import (\n LogCompletionsCallback,\n ModelConfig,\n ScriptArguments,\n TrlParser,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n)\nfrom trl.experimental.gold.gold_config import GOLDConfig\nfrom trl.experimental.gold.gold_trainer import GOLDTrainer\n\n\nlogger = logging.getLogger(__name__)\n\n\nif __name__ == \"__main__\":\n parser = TrlParser((ScriptArguments, GOLDConfig, ModelConfig))\n script_args, training_args, model_args = parser.parse_args_and_config()\n\n ################\n # Model & Tokenizer\n ################\n quantization_config = get_quantization_config(model_args)\n model_kwargs = dict(\n revision=model_args.model_revision,\n trust_remote_code=model_args.trust_remote_code,\n attn_implementation=model_args.attn_implementation,\n torch_dtype=model_args.dtype,\n use_cache=False if training_args.gradient_checkpointing else True,\n device_map=get_kbit_device_map() if quantization_config is not None else None,\n quantization_config=quantization_config,\n )\n training_args.model_init_kwargs = model_kwargs\n\n if training_args.teacher_tokenizer_name_or_path is None and training_args.use_uld_loss:\n training_args.teacher_tokenizer_name_or_path = training_args.teacher_model_name_or_path\n teacher_model_kwargs = dict(\n revision=training_args.teacher_model_revision,\n trust_remote_code=model_args.trust_remote_code,\n attn_implementation=model_args.attn_implementation,\n torch_dtype=model_args.dtype,\n use_cache=True,\n device_map=get_kbit_device_map() if quantization_config is not None else None,\n quantization_config=quantization_config,\n )\n if training_args.teacher_model_init_kwargs is not None:\n teacher_model_kwargs.update(training_args.teacher_model_init_kwargs)\n training_args.teacher_model_init_kwargs = teacher_model_kwargs\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path,\n revision=model_args.model_revision,\n trust_remote_code=model_args.trust_remote_code,\n )\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n ################\n # Dataset\n ################\n dataset = load_dataset(script_args.dataset_name, name=script_args.dataset_config)\n\n ################\n # Training\n ################\n eval_dataset = None\n if training_args.eval_strategy != \"no\":\n if script_args.dataset_test_split in dataset:\n eval_dataset = dataset[script_args.dataset_test_split]\n elif \"validation\" in dataset:\n eval_dataset = dataset[\"validation\"]\n elif \"dev\" in dataset:\n eval_dataset = dataset[\"dev\"]\n\n trainer = GOLDTrainer(\n model=model_args.model_name_or_path,\n teacher_model=training_args.teacher_model_name_or_path,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=eval_dataset,\n processing_class=tokenizer,\n peft_config=get_peft_config(model_args),\n )\n\n if training_args.eval_strategy != \"no\":\n generation_config = GenerationConfig(\n max_new_tokens=training_args.max_completion_length, do_sample=True, temperature=training_args.temperature\n )\n completions_callback = LogCompletionsCallback(trainer, generation_config, num_prompts=8)\n trainer.add_callback(completions_callback)\n\n trainer.train()\n\n # Save and push to hub\n trainer.save_model(training_args.output_dir)\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n" }, { "path": "trl/experimental/gold/gold_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport warnings\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom ...trainer.sft_config import SFTConfig\n\n\n@dataclass\nclass GOLDConfig(SFTConfig):\n r\"\"\"\n Configuration class for [`GOLDTrainer`].\n\n This class includes only the parameters that are specific to GOLD training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] and [`SFTConfig`] documentation.\n\n Args:\n temperature (`float`, *optional*, defaults to `0.9`):\n Temperature for sampling. The higher the temperature, the more random the completions.\n lmbda (`float`, *optional*, defaults to `0.5`):\n Lambda parameter that controls the student data fraction (i.e., the proportion of on-policy\n student-generated outputs).\n beta (`float`, *optional*, defaults to `0.5`):\n Interpolation coefficient between `0.0` and `1.0` of the Generalized Jensen-Shannon Divergence loss. When\n beta is `0.0`, the loss is the KL divergence. When beta is `1.0`, the loss is the Inverse KL Divergence.\n max_completion_length (`int`, *optional*, defaults to `128`):\n Maximum number of tokens to generate per completion.\n teacher_model_name_or_path (`str`, *optional*):\n Model name or path of the teacher model. If `None`, the teacher model will be the same as the model being\n trained.\n teacher_model_revision (`str` or `None`, *optional*, defaults to `None`):\n Model revision of the teacher model (e.g., branch name, tag, or commit hash). If `None`, the default\n revision is used.\n teacher_model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the teacher model\n from a string.\n teacher_tokenizer_name_or_path (`str`, *optional*):\n Tokenizer name or path for the teacher model. If None when using ULD loss, will use the same tokenizer as\n the student model (not recommended for cross-tokenizer distillation).\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model.\n seq_kd (`bool`, *optional*, defaults to `False`):\n Seq_kd parameter that controls whether to perform Sequence-Level KD (can be viewed as supervised FT on\n teacher-generated output).\n num_generations (`int`, *optional*, defaults to `1`):\n Number of generations per prompt. Each prompt is repeated this many times in the generation batch.\n generation_batch_size (`int` or `None`, *optional*, defaults to `None`):\n Number of unique prompts per worker per optimizer step. If `None`, it is computed from\n `(per_device_train_batch_size * gradient_accumulation_steps) // num_generations`.\n use_uld_loss (`bool`, *optional*, defaults to `False`):\n Whether to use Universal Logit Distillation (ULD) loss instead of Generalized Jensen-Shannon Divergence\n loss.\n uld_crossentropy_weight (`float`, *optional*, defaults to `0.0`):\n Weight for the cross-entropy loss component in ULD loss. If 0, only ULD distillation loss is used.\n uld_distillation_weight (`float`, *optional*, defaults to `1.0`):\n Weight for the distillation loss component in ULD loss.\n uld_student_temperature (`float`, *optional*, defaults to `1.0`):\n Temperature for student logits in ULD loss computation.\n uld_teacher_temperature (`float`, *optional*, defaults to `1.0`):\n Temperature for teacher logits in ULD loss computation.\n uld_skip_student_eos (`bool`, *optional*, defaults to `True`):\n Whether to skip EOS token for student in ULD loss computation.\n uld_skip_teacher_eos (`bool`, *optional*, defaults to `True`):\n Whether to skip EOS token for teacher in ULD loss computation.\n use_vllm (`bool`, *optional*, defaults to `False`):\n Whether to use vLLM for generating completions from the student model. Requires `vllm` to be installed.\n vllm_mode (`str`, *optional*, defaults to `\"colocate\"`):\n Mode for student vLLM integration. Either `\"server\"` (connect to a running TRL vLLM server) or `\"colocate\"`\n (run vLLM in the same process).\n vllm_server_host (`str`, *optional*, defaults to `\"0.0.0.0\"`):\n Host of the vLLM server for the student model (if `vllm_mode=\"server\"`).\n vllm_server_port (`int`, *optional*, defaults to `8001`):\n Port of the vLLM server for the student model (if `vllm_mode=\"server\"`).\n vllm_server_timeout (`float`, *optional*, defaults to `240.0`):\n Timeout for connecting to the student vLLM server (if `vllm_mode=\"server\"`).\n vllm_gpu_memory_utilization (`float`, *optional*, defaults to `0.9`):\n GPU memory utilization for the colocated student vLLM engine (if `vllm_mode=\"colocate\"`). It is recommended\n to set this to a low value if the student and teacher models share the same GPU.\n vllm_tensor_parallel_size (`int`, *optional*, defaults to `1`):\n Tensor parallel size for the colocated student vLLM engine (if `vllm_mode=\"colocate\"`).\n vllm_structured_outputs_regex (`str`, *optional*):\n Regex for vLLM structured outputs for the student model.\n vllm_sync_frequency (`int`, *optional*, defaults to `1`):\n Frequency (in training steps) to synchronize student model weights to vLLM engine. Set to 1 to sync after\n every step.\n vllm_enable_sleep_mode (`bool`, *optional*, defaults to `False`):\n Enable vLLM sleep mode to offload student weights/cache during the optimizer step. Keeps GPU memory usage\n low, but waking the engine adds hostโ€“device transfer latency.\n \"\"\"\n\n _VALID_DICT_FIELDS = SFTConfig._VALID_DICT_FIELDS + [\"teacher_model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-7,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n # GOLD-specific parameters\n temperature: float = field(\n default=0.9,\n metadata={\"help\": \"Temperature for sampling. The higher the temperature, the more random the completions.\"},\n )\n top_p: float = field(\n default=0.95,\n metadata={\n \"help\": \"If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to \"\n \"`top_p` or higher are kept for generation.\"\n },\n )\n top_k: int = field(\n default=0,\n metadata={\n \"help\": \"Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, \"\n \"top-k-filtering is disabled and all tokens are considered.\"\n },\n )\n lmbda: float = field(\n default=0.5,\n metadata={\n \"help\": \"Lambda parameter that controls the student data fraction (i.e., the proportion of on-policy \"\n \"student-generated outputs).\"\n },\n )\n beta: float = field(\n default=0.5,\n metadata={\n \"help\": \"Interpolation coefficient between `0.0` and `1.0` of the Generalized Jensen-Shannon Divergence \"\n \"loss. When beta is `0.0`, the loss is the KL divergence. When beta is `1.0`, the loss is the Inverse KL \"\n \"Divergence.\"\n },\n )\n max_completion_length: int = field(\n default=128,\n metadata={\"help\": \"Maximum number of tokens to generate per completion.\"},\n )\n teacher_model_name_or_path: str | None = field(\n default=None,\n metadata={\n \"help\": \"Model name or path of the teacher model. If `None`, the teacher model will be the same as the \"\n \"model being trained.\"\n },\n )\n teacher_model_revision: str | None = field(\n default=None,\n metadata={\n \"help\": \"Model revision of the teacher model (e.g., branch name, tag, or commit hash). If `None`, the \"\n \"default revision is used.\"\n },\n )\n teacher_model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the \"\n \"teacher model from a string.\"\n },\n )\n teacher_tokenizer_name_or_path: str | None = field(\n default=None,\n metadata={\n \"help\": \"Tokenizer name or path for the teacher model. If None when using ULD loss, will use the same \"\n \"tokenizer as the student model (not recommended for cross-tokenizer distillation).\"\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropouts in `model`.\"},\n )\n seq_kd: bool = field(\n default=False,\n metadata={\n \"help\": \"Seq_kd parameter that controls whether to perform Sequence-Level KD (can be viewed as supervised \"\n \"FT on teacher-generated output).\"\n },\n )\n num_generations: int = field(\n default=1,\n metadata={\n \"help\": \"Number of generations per prompt. Increasing this will decrease the number of unique prompts per optimization step.\"\n },\n )\n generation_batch_size: int | None = field(\n default=None,\n metadata={\n \"help\": \"Number of unique prompts per worker per optimizer step. \"\n \"If None, computed from (per_device_train_batch_size * gradient_accumulation_steps) // num_generations.\"\n },\n )\n\n # ULD Loss parameters\n use_uld_loss: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use Universal Logit Distillation (ULD) loss instead of Generalized Jensen-Shannon Divergence loss.\"\n },\n )\n use_extended_uld: bool = field(\n default=True,\n metadata={\n \"help\": (\n \"Whether to enable extended ULD alignment that uses tokenizers to align and merge token \"\n \"probabilities across student and teacher tokenizations. When True, the trainer will compute \"\n \"token mappings and merge probabilities for split tokens; when False, ULD will use simple \"\n \"positional truncation like in the original ULD paper.\"\n )\n },\n )\n uld_use_hybrid_loss: bool = field(\n default=False,\n metadata={\n \"help\": (\n \"Whether to use a hybrid loss that combines ULD loss and JSD loss. When True, the final loss is a \"\n \"a combination of JSD for known token mappings and ULD for unknown token mappings.\"\n )\n },\n )\n uld_hybrid_matched_weight: float | None = field(\n default=None,\n metadata={\n \"help\": (\n \"Weight for the matched token loss component when using hybrid ULD + JSD loss. This weight scales \"\n \"the JSD loss computed over tokens that have a direct mapping between student and teacher \"\n \"tokenizations. If None, uses adaptive weighting based on vocabulary overlap. Must be set together \"\n \"with uld_hybrid_unmatched_weight (both None or both float).\"\n )\n },\n )\n uld_hybrid_unmatched_weight: float | None = field(\n default=None,\n metadata={\n \"help\": (\n \"Weight for the unmatched token loss component when using hybrid ULD + JSD loss. This weight scales \"\n \"the ULD loss computed over tokens that do not have a direct mapping between student and teacher \"\n \"tokenizations. If None, uses adaptive weighting based on vocabulary overlap. Must be set together \"\n \"with uld_hybrid_matched_weight (both None or both float).\"\n )\n },\n )\n uld_crossentropy_weight: float = field(\n default=0.0,\n metadata={\"help\": \"Weight for the cross-entropy loss component in ULD loss.\"},\n )\n uld_distillation_weight: float = field(\n default=1.0,\n metadata={\"help\": \"Weight for the distillation loss component in ULD loss.\"},\n )\n uld_student_temperature: float = field(\n default=1.0,\n metadata={\"help\": \"Temperature for student logits in ULD loss computation.\"},\n )\n uld_teacher_temperature: float = field(\n default=1.0,\n metadata={\"help\": \"Temperature for teacher logits in ULD loss computation.\"},\n )\n\n uld_skip_student_eos: bool = field(\n default=True,\n metadata={\"help\": \"Whether to skip EOS token for student in ULD loss computation.\"},\n )\n uld_skip_teacher_eos: bool = field(\n default=True,\n metadata={\"help\": \"Whether to skip EOS token for teacher in ULD loss computation.\"},\n )\n\n # transformers paged attention\n use_transformers_paged: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use the `transformers` paged implementation for generation. If set to `True`, the \"\n \"`transformers` paged implementation will be used for generation instead of the default padded \"\n \"implementation.\"\n },\n )\n\n # vLLM parameters\n use_vllm: bool = field(\n default=False,\n metadata={\"help\": \"Whether to use vLLM for generating completions. Requires `vllm` to be installed.\"},\n )\n vllm_mode: str = field(\n default=\"colocate\",\n metadata={\n \"help\": 'Mode for vLLM integration. Either \"server\" (connect to a running TRL vLLM server) or \"colocate\" (run vLLM in the same process).'\n },\n )\n vllm_server_host: str = field(\n default=\"0.0.0.0\",\n metadata={\"help\": 'Host of the vLLM server when `vllm_mode=\"server\"`.'},\n )\n vllm_server_port: int = field(\n default=8001,\n metadata={\"help\": 'Port of the vLLM server when `vllm_mode=\"server\"`.'},\n )\n vllm_server_timeout: float = field(\n default=240.0,\n metadata={\"help\": 'Timeout (in seconds) for connecting to the vLLM server when `vllm_mode=\"server\"`.'},\n )\n vllm_gpu_memory_utilization: float = field(\n default=0.9,\n metadata={\n \"help\": 'GPU memory utilization for the colocated vLLM engine when `vllm_mode=\"colocate\"`. Lower values reduce contention when sharing a device with the student/teacher models.'\n },\n )\n vllm_tensor_parallel_size: int = field(\n default=1,\n metadata={\"help\": 'Tensor parallel size for the colocated vLLM engine when `vllm_mode=\"colocate\"`.'},\n )\n vllm_structured_outputs_regex: str | None = field(\n default=None,\n metadata={\"help\": \"Regex pattern used for vLLM structured outputs (optional).\"},\n )\n vllm_sync_frequency: int = field(\n default=1,\n metadata={\n \"help\": \"Frequency (in training steps) to synchronize model weights to the vLLM engine. Set to 1 to sync after every step.\"\n },\n )\n vllm_enable_sleep_mode: bool = field(\n default=False,\n metadata={\n \"help\": \"Enable vLLM sleep mode to offload student weights/cache during the optimizer step. Keeps GPU \"\n \"memory usage low, but waking the engine adds hostโ€“device transfer latency.\"\n },\n )\n # Parameters that control the logging\n log_completions: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is \"\n \"installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`.\"\n },\n )\n log_completions_steps: int = field(\n default=100,\n metadata={\n \"help\": \"Number of steps between logging (prompt, completion) pairs. Only used if `log_completions` is \"\n \"set to `True`.\"\n },\n )\n num_completions_to_print: int | None = field(\n default=None,\n metadata={\"help\": \"Number of completions to print with `rich`. If `None`, all completions are logged.\"},\n )\n wandb_entity: str | None = field(\n default=None,\n metadata={\"help\": (\"The entity to store runs under.\")},\n )\n wandb_project: str | None = field(\n default=None,\n metadata={\"help\": (\"The project to store runs under.\")},\n )\n wandb_run_group: str | None = field(\n default=None,\n metadata={\"help\": (\"The group to store runs under.\")},\n )\n wandb_log_unique_prompts: bool = field(\n default=True,\n metadata={\n \"help\": (\"Whether to log the unique prompts to wandb. This will create a new run for each unique prompt.\")\n },\n )\n callbacks: list[str] = field(\n default_factory=lambda: [],\n metadata={\"help\": \"The callbacks to run during training.\"},\n )\n hub_model_revision: str | None = field(\n default=\"main\", metadata={\"help\": \"The Hub model branch to push the model to.\"}\n )\n overwrite_hub_revision: bool = field(default=False, metadata={\"help\": \"Whether to overwrite the Hub revision.\"})\n push_to_hub_revision: bool = field(default=False, metadata={\"help\": \"Whether to push to a Hub revision/branch.\"})\n\n def __post_init__(self):\n super().__post_init__()\n # check lmbda and beta are in the range [0, 1]\n if self.lmbda < 0.0 or self.lmbda > 1.0:\n raise ValueError(\"lmbda must be in the range [0.0, 1.0].\")\n if self.beta < 0.0 or self.beta > 1.0:\n raise ValueError(\"beta must be in the range [0.0, 1.0].\")\n\n # Validate that max_length is sufficient for max_completion_length\n if self.max_length is not None and self.max_completion_length >= self.max_length:\n raise ValueError(\n f\"max_completion_length ({self.max_completion_length}) must be smaller than max_length ({self.max_length}) \"\n f\"to leave room for the prompt. Consider increasing max_length or reducing max_completion_length.\"\n )\n\n if self.num_generations < 1:\n raise ValueError(f\"num_generations must be at least 1, got {self.num_generations}.\")\n local_sequence_batch_size = self.per_device_train_batch_size * self.gradient_accumulation_steps\n if self.generation_batch_size is None:\n self.generation_batch_size = local_sequence_batch_size // self.num_generations\n if self.generation_batch_size < 1:\n raise ValueError(\n f\"generation_batch_size must be at least 1. Got generation_batch_size={self.generation_batch_size}.\"\n )\n if self.generation_batch_size * self.num_generations != local_sequence_batch_size:\n raise ValueError(\n \"generation_batch_size and num_generations must exactly partition the local optimizer-step batch. \"\n \"Expected generation_batch_size * num_generations == per_device_train_batch_size * \"\n f\"gradient_accumulation_steps, got {self.generation_batch_size} * {self.num_generations} != \"\n f\"{self.per_device_train_batch_size} * {self.gradient_accumulation_steps}.\"\n )\n if self.num_generations > 1 and self.lmbda < 1.0:\n warnings.warn(\n f\"num_generations={self.num_generations} with lmbda={self.lmbda} means off-policy batches include \"\n f\"{self.num_generations} copies of each sample; consider lmbda=1.0 when num_generations > 1.\",\n UserWarning,\n stacklevel=2,\n )\n\n # Validate ULD parameters\n if self.use_uld_loss:\n if self.uld_crossentropy_weight < 0.0:\n raise ValueError(\"uld_crossentropy_weight must be non-negative.\")\n if self.uld_distillation_weight < 0.0:\n raise ValueError(\"uld_distillation_weight must be non-negative.\")\n if self.uld_student_temperature <= 0.0:\n raise ValueError(\"uld_student_temperature must be positive.\")\n if self.uld_teacher_temperature <= 0.0:\n raise ValueError(\"uld_teacher_temperature must be positive.\")\n\n # Validate hybrid loss weights - both must be None or both must be set\n if self.uld_use_hybrid_loss:\n if (self.uld_hybrid_matched_weight is None) != (self.uld_hybrid_unmatched_weight is None):\n raise ValueError(\n \"uld_hybrid_matched_weight and uld_hybrid_unmatched_weight must both be None (for adaptive \"\n \"weighting) or both be set to numeric values. Got uld_hybrid_matched_weight=\"\n f\"{self.uld_hybrid_matched_weight} and uld_hybrid_unmatched_weight=\"\n f\"{self.uld_hybrid_unmatched_weight}.\"\n )\n if self.uld_hybrid_matched_weight is not None:\n if self.uld_hybrid_matched_weight < 0.0:\n raise ValueError(\"uld_hybrid_matched_weight must be non-negative.\")\n if self.uld_hybrid_unmatched_weight < 0.0:\n raise ValueError(\"uld_hybrid_unmatched_weight must be non-negative.\")\n" }, { "path": "trl/experimental/gold/gold_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport os\nimport random\nimport textwrap\nimport warnings\nfrom collections import defaultdict, deque\nfrom collections.abc import Callable\nfrom contextlib import nullcontext\nfrom functools import partial\nfrom typing import Any, Optional\n\nimport torch\nimport torch.distributed as dist\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom accelerate import PartialState\nfrom accelerate.utils import DistributedType, broadcast_object_list, gather_object, is_peft_model\nfrom datasets import Dataset, IterableDataset\nfrom torch.distributed.fsdp import FullyShardedDataParallel as FSDP\nfrom torch.utils.data import DataLoader\nfrom transformers import AutoTokenizer, TrainerCallback, TrainerControl, TrainerState, is_bitsandbytes_available\nfrom transformers.data.data_collator import DataCollator\nfrom transformers.feature_extraction_utils import FeatureExtractionMixin\nfrom transformers.generation.configuration_utils import GenerationConfig\nfrom transformers.image_processing_utils import BaseImageProcessor\nfrom transformers.integrations.integration_utils import is_wandb_available\nfrom transformers.modeling_utils import PreTrainedModel\nfrom transformers.processing_utils import ProcessorMixin\nfrom transformers.tokenization_utils_base import PreTrainedTokenizerBase\nfrom transformers.trainer_utils import EvalPrediction, seed_worker\nfrom transformers.utils import (\n is_datasets_available,\n is_flash_attn_2_available,\n is_liger_kernel_available,\n is_peft_available,\n is_rich_available,\n)\n\nfrom ...data_utils import is_conversational, maybe_convert_to_chatml, pack_dataset, truncate_dataset\nfrom ...extras.profiling import profiling_decorator\nfrom ...generation.vllm_client import VLLMClient\nfrom ...import_utils import is_vllm_available\nfrom ...models import prepare_deepspeed\nfrom ...models.utils import unwrap_model_for_generation\nfrom ...trainer.sft_trainer import SFTTrainer\nfrom ...trainer.utils import (\n RepeatSampler,\n create_model_from_path,\n disable_dropout_in_model,\n ensure_master_addr_port,\n pad,\n split_tensor_dict,\n)\nfrom ..utils import DataCollatorForChatML, empty_cache\nfrom .gold_config import GOLDConfig\n\n\nif is_peft_available():\n from peft import PeftConfig\n\nif is_wandb_available():\n import wandb\n\nif is_vllm_available():\n from vllm import LLM, SamplingParams\n from vllm.sampling_params import StructuredOutputsParams\n\nif is_liger_kernel_available():\n from liger_kernel.chunked_loss import LigerFusedLinearJSDLoss\n\nif is_rich_available():\n from rich.console import Console\n from rich.panel import Panel\n from rich.table import Table\n from rich.text import Text\n\nif is_bitsandbytes_available():\n import bitsandbytes as bnb\n\n\ndef print_prompt_completions_sample_uld(\n prompts: list[str],\n completions: list[str],\n step: int,\n num_samples: int = None,\n) -> None:\n \"\"\"\n Print out a sample of model completions to the console with multiple reward metrics.\n\n This function creates a nicely formatted table showing prompt-completion pairs, useful for monitoring model outputs\n during training. It requires the `rich` library to be installed.\n\n Args:\n prompts (`list[str]`):\n List of prompts.\n completions (`list[str]`):\n List of completions corresponding to the prompts.\n rewards (`dict[str, list[float]]`):\n Dictionary where keys are reward names and values are lists of rewards.\n advantages (`list[float]`):\n List of advantages corresponding to the prompts and completions.\n step (`int`):\n Current training step number, used in the output title.\n num_samples (`int` or `None`, *optional*, defaults to `None`):\n Number of random samples to display. If `None` (default), all items will be displayed.\n\n Example:\n ```python\n >>> from trl.trainer.utils import print_prompt_completions_sample\n\n >>> prompts = [\"The sky is\", \"The sun is\"]\n >>> completions = [\" blue.\", \" in the sky.\"]\n >>> rewards = {\"Correctness\": [0.123, 0.456], \"Format\": [0.789, 0.101]}\n >>> advantages = [0.987, 0.654]\n >>> print_prompt_completions_sample(prompts, completions, rewards, advantages, 42)\n โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Step 42 โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ\n โ”‚ โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”‚\n โ”‚ โ”ƒ Prompt โ”ƒ Completion โ”ƒ Correctness โ”ƒ Format โ”ƒ Advantage โ”ƒ โ”‚\n โ”‚ โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ โ”‚\n โ”‚ โ”‚ The sky is โ”‚ blue. โ”‚ 0.12 โ”‚ 0.79 โ”‚ 0.99 โ”‚ โ”‚\n โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚\n โ”‚ โ”‚ The sun is โ”‚ in the sky. โ”‚ 0.46 โ”‚ 0.10 โ”‚ 0.65 โ”‚ โ”‚\n โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚\n โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ\n ```\n \"\"\"\n if not is_rich_available():\n raise ImportError(\n \"The function `print_prompt_completions_sample` requires the `rich` library. Please install it with \"\n \"`pip install rich`.\"\n )\n console = Console()\n table = Table(show_header=True, header_style=\"bold white\", expand=True)\n\n # Add columns\n table.add_column(\"Prompt\", style=\"bright_yellow\")\n table.add_column(\"Completion\", style=\"bright_green\")\n\n # Some basic input validation\n if num_samples is not None:\n if num_samples >= len(prompts):\n num_samples = None\n elif num_samples <= 0:\n return\n\n # Subsample data if num_samples is specified\n if num_samples is not None:\n indices = random.sample(range(len(prompts)), num_samples)\n prompts = [prompts[i] for i in indices]\n completions = [completions[i] for i in indices]\n\n for i in range(len(prompts)):\n table.add_row(Text(prompts[i]), Text(completions[i]))\n table.add_section() # Adds a separator between rows\n\n panel = Panel(table, expand=False, title=f\"Step {step}\", border_style=\"bold white\")\n console.print(panel)\n\n\ndef build_teacher_inputs_from_texts(\n tokenizer: PreTrainedTokenizerBase,\n prompt_texts: list[str],\n completion_texts: list[str],\n) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, int]:\n \"\"\"Tokenize teacher prompts/completions and produce tensors ready for GOLD loss.\"\"\"\n\n pad_token_id = tokenizer.pad_token_id\n eos_token_id = tokenizer.eos_token_id\n\n prompt_token_ids = tokenizer(prompt_texts, add_special_tokens=True)[\"input_ids\"]\n completion_token_ids = tokenizer(completion_texts, add_special_tokens=False)[\"input_ids\"]\n\n sequences: list[torch.Tensor] = []\n attention_masks: list[torch.Tensor] = []\n labels_list: list[torch.Tensor] = []\n prompt_lengths: list[int] = []\n\n for prompt_ids, completion_ids in zip(prompt_token_ids, completion_token_ids, strict=True):\n # Remove trailing EOS from prompt so completions can extend cleanly\n if eos_token_id is not None and prompt_ids and prompt_ids[-1] == eos_token_id:\n prompt_ids = prompt_ids[:-1]\n\n prompt_lengths.append(len(prompt_ids))\n sequence = list(prompt_ids)\n sequence.extend(completion_ids)\n if eos_token_id is not None:\n sequence.append(eos_token_id)\n\n seq_tensor = torch.tensor(sequence, dtype=torch.long)\n sequences.append(seq_tensor)\n attention_masks.append(torch.ones_like(seq_tensor))\n\n labels = seq_tensor.clone()\n labels[: len(prompt_ids)] = -100\n if pad_token_id is not None:\n labels[labels == pad_token_id] = -100\n labels_list.append(labels)\n\n teacher_input_ids = pad(\n sequences,\n padding_side=\"right\",\n padding_value=pad_token_id if pad_token_id is not None else 0,\n )\n teacher_attention_mask = pad(attention_masks, padding_side=\"right\", padding_value=0).bool()\n teacher_labels = pad(labels_list, padding_side=\"right\", padding_value=-100)\n\n if eos_token_id is not None:\n for row in range(teacher_attention_mask.size(0)):\n valid = (\n teacher_input_ids[row] != pad_token_id\n if pad_token_id is not None\n else teacher_attention_mask[row].bool()\n )\n if valid.any():\n last_idx = valid.nonzero(as_tuple=True)[0][-1]\n teacher_attention_mask[row, last_idx + 1 :] = False\n\n teacher_prompt_length = max(prompt_lengths) if prompt_lengths else 0\n\n return teacher_input_ids, teacher_labels, teacher_attention_mask, teacher_prompt_length\n\n\nclass ULDLoss(nn.Module):\n \"\"\"\n Universal Logit Distillation Loss.\n \"\"\"\n\n def __init__(self, config: GOLDConfig, student_tokenizer=None, teacher_tokenizer=None, device=None):\n super().__init__()\n self.device = device\n self.crossentropy_weight = config.uld_crossentropy_weight\n self.distillation_weight = config.uld_distillation_weight\n self.student_temperature = config.uld_student_temperature\n self.teacher_temperature = config.uld_teacher_temperature\n self.skip_student_eos = config.uld_skip_student_eos\n self.skip_teacher_eos = config.uld_skip_teacher_eos\n self.use_extended_uld = config.use_extended_uld\n self.ignore_index = -100\n\n # Add tokenizers for enhanced alignment\n self.student_tokenizer = student_tokenizer\n self.teacher_tokenizer = teacher_tokenizer\n\n # Hybrid ULD configuration\n self.use_hybrid_loss = getattr(config, \"uld_use_hybrid_loss\", False)\n self.hybrid_matched_weight = getattr(config, \"uld_hybrid_matched_weight\", None)\n self.hybrid_unmatched_weight = getattr(config, \"uld_hybrid_unmatched_weight\", None)\n self.beta = getattr(config, \"beta\", 1.0) # For JSD loss in hybrid matched tokens\n\n # Initialize vocabulary mapping for hybrid loss\n self._vocab_mapping = None\n self._teacher_matched_ids = None\n self._student_matched_ids = None\n if self.use_hybrid_loss and student_tokenizer is not None and teacher_tokenizer is not None:\n self._initialize_vocabulary_mapping()\n\n def __call__(\n self, student_logits, teacher_logits, student_labels, teacher_labels, student_input_ids, teacher_input_ids\n ):\n \"\"\"\n Compute ULD loss with GKD trainer interface.\n\n Args:\n student_logits: Student model logits [batch_size, seq_len, vocab_size]\n teacher_logits: Teacher model logits [batch_size, seq_len, vocab_size]\n student_labels: Student target labels [batch_size, seq_len]\n teacher_labels: Teacher target labels [batch_size, seq_len]\n student_input_ids: Student input token IDs [batch_size, seq_len]\n teacher_input_ids: Teacher input token IDs [batch_size, seq_len]\n\n Returns:\n Total loss (cross-entropy + distillation)\n \"\"\"\n # Compute cross-entropy loss for student\n if self.crossentropy_weight > 0:\n shift_logits = student_logits[..., :-1, :].contiguous()\n shift_labels = student_labels[..., 1:].contiguous()\n loss_fct = nn.CrossEntropyLoss(ignore_index=self.ignore_index)\n crossentropy_loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))\n crossentropy_loss = self.crossentropy_weight * crossentropy_loss\n else:\n crossentropy_loss = 0.0\n\n # Compute distillation loss using ULD approximation\n distillation_loss = self._compute_distillation_loss(\n student_logits, teacher_logits, student_labels, teacher_labels, student_input_ids, teacher_input_ids\n )\n\n return crossentropy_loss + distillation_loss\n\n def _initialize_vocabulary_mapping(self):\n \"\"\"Initialize vocabulary mapping for hybrid ULD loss.\"\"\"\n # Computing vocabulary mapping for hybrid ULD\n\n student_vocab = self.student_tokenizer.get_vocab()\n teacher_vocab = self.teacher_tokenizer.get_vocab()\n\n # Create reverse mapping for student\n student_token_to_id = dict(student_vocab.items())\n\n vocab_mapping = {}\n teacher_matched_ids = set()\n student_matched_ids = set()\n\n for token_str, teacher_id in teacher_vocab.items():\n if token_str in student_token_to_id:\n student_id = student_token_to_id[token_str]\n vocab_mapping[teacher_id] = student_id\n teacher_matched_ids.add(teacher_id)\n student_matched_ids.add(student_id)\n\n self._vocab_mapping = vocab_mapping\n self._teacher_matched_ids = teacher_matched_ids\n self._student_matched_ids = student_matched_ids\n\n max_matched_teacher_id = max(self._vocab_mapping.keys())\n self.mapping_tensor = torch.full((max_matched_teacher_id + 1,), -1, dtype=torch.long) # -1 for unmapped ids\n for k, v in self._vocab_mapping.items():\n self.mapping_tensor[k] = v\n if self.device is not None:\n self.mapping_tensor = self.mapping_tensor.to(self.device)\n\n def _compute_distillation_loss(\n self, student_logits, teacher_logits, student_labels, teacher_labels, student_input_ids, teacher_input_ids\n ):\n \"\"\"\n Compute the Universal Logit Distillation loss with token mapping.\n\n This version uses actual input_ids for accurate token mapping and multiplies probabilities for split tokens.\n Both student_input_ids and teacher_input_ids are required for optimal alignment.\n \"\"\"\n # Get answer regions (same as original)\n student_answer_index, student_answer_size = self._get_start_and_size_answers(student_labels)\n teacher_answer_index, teacher_answer_size = self._get_start_and_size_answers(teacher_labels)\n\n if self.skip_student_eos:\n student_answer_size = [size - 1 for size in student_answer_size]\n if self.skip_teacher_eos:\n teacher_answer_size = [size - 1 for size in teacher_answer_size]\n\n # Handle edge case where all answer sizes are 0\n if (\n not student_answer_size\n or not teacher_answer_size\n or max(max(student_answer_size), max(teacher_answer_size)) <= 0\n ):\n return torch.zeros(1, device=student_logits.device, requires_grad=True) * student_logits.sum() * 1e-8\n\n batch_size = student_logits.size(0)\n distillation_losses = []\n\n for i in range(batch_size):\n # Get answer regions for this batch item\n student_start = student_answer_index[i]\n student_size = student_answer_size[i]\n teacher_start = teacher_answer_index[i]\n teacher_size = teacher_answer_size[i]\n\n if student_size <= 0 or teacher_size <= 0:\n loss_i = student_logits[i].sum() * 0.0\n distillation_losses.append(loss_i)\n continue\n\n # Extract answer logits\n student_answer_logits = student_logits[i, student_start : student_start + student_size]\n teacher_answer_logits = teacher_logits[i, teacher_start : teacher_start + teacher_size]\n\n # Convert to probabilities\n student_probs = F.softmax(student_answer_logits / self.student_temperature, dim=-1)\n teacher_probs = F.softmax(teacher_answer_logits / self.teacher_temperature, dim=-1)\n\n # Get token IDs for mapping (always use actual input_ids)\n student_token_ids = student_input_ids[i, student_start : student_start + student_size].tolist()\n teacher_token_ids = teacher_input_ids[i, teacher_start : teacher_start + teacher_size].tolist()\n\n if self.use_extended_uld:\n # Build alignment groups directly from token ids using greedy text matching\n student_alignment_groups, teacher_alignment_groups = self._build_alignment_groups_from_ids(\n student_token_ids, teacher_token_ids\n )\n\n # Merge student probabilities using student alignment groups\n # Pass student_token_ids to enable corrected conditional probability merging\n student_aligned = self._merge_probabilities_with_alignment_groups(\n student_probs, student_alignment_groups, student_token_ids\n )\n\n # Merge teacher probabilities using teacher alignment groups\n # Pass teacher_token_ids to enable corrected conditional probability merging\n teacher_aligned = self._merge_probabilities_with_alignment_groups(\n teacher_probs, teacher_alignment_groups, teacher_token_ids\n )\n else:\n min_length = min(len(student_token_ids), len(teacher_token_ids))\n student_aligned = student_probs[:min_length, :]\n teacher_aligned = teacher_probs[:min_length, :]\n\n # Apply ULD loss computation\n if self.use_hybrid_loss and self._vocab_mapping is not None:\n # Use hybrid approach: direct comparison for matched tokens, sorting for unmatched\n aligned_loss = self._compute_hybrid_uld_loss(student_aligned, teacher_aligned)\n else:\n # Original approach: sort all probabilities\n student_sorted = student_aligned.sort(dim=-1, descending=True).values\n teacher_sorted = teacher_aligned.sort(dim=-1, descending=True).values\n\n # Pad vocabularies to same size\n student_vocab_size = student_sorted.size(-1)\n teacher_vocab_size = teacher_sorted.size(-1)\n max_vocab_size = max(student_vocab_size, teacher_vocab_size)\n\n if student_vocab_size < max_vocab_size:\n student_sorted = F.pad(student_sorted, (0, max_vocab_size - student_vocab_size))\n if teacher_vocab_size < max_vocab_size:\n teacher_sorted = F.pad(teacher_sorted, (0, max_vocab_size - teacher_vocab_size))\n\n # Compute L1 distance (ULD approach)\n aligned_loss = F.l1_loss(student_sorted, teacher_sorted, reduction=\"sum\")\n aligned_loss /= student_aligned.size(0) # Normalize by sequence length\n distillation_losses.append(aligned_loss)\n\n distillation_loss = torch.stack(distillation_losses).mean()\n return self.distillation_weight * distillation_loss\n\n def _build_alignment_groups_from_ids(self, student_token_ids, teacher_token_ids):\n \"\"\"\n Build alignment groups using a greedy substring-equality algorithm on decoded token pieces.\n\n Args:\n student_token_ids: List[int]\n teacher_token_ids: List[int]\n\n Returns:\n Tuple[List[List[int]], List[List[int]]]: student and teacher alignment groups\n \"\"\"\n\n def to_canonical_pieces(tok, ids):\n pieces = []\n prev = \"\"\n for k in range(len(ids)):\n # IMPORTANT: Do NOT skip special tokens - we need to align them too\n cur = tok.decode(ids[: k + 1], skip_special_tokens=False, clean_up_tokenization_spaces=False)\n # Extract the incremental addition (may include spaces/ZWJ/etc.)\n pieces.append(cur[len(prev) :])\n prev = cur\n return pieces\n\n s_pieces = to_canonical_pieces(self.student_tokenizer, student_token_ids)\n t_pieces = to_canonical_pieces(self.teacher_tokenizer, teacher_token_ids)\n\n i = j = 0\n s_buf = t_buf = \"\"\n s_group = []\n t_group = []\n s_groups = []\n t_groups = []\n\n def flush():\n if s_group and t_group:\n s_groups.append(s_group.copy())\n t_groups.append(t_group.copy())\n\n # Greedily accumulate pieces until substrings match, then flush\n while i < len(s_pieces) or j < len(t_pieces):\n if s_buf == t_buf and s_buf != \"\":\n flush()\n s_buf = t_buf = \"\"\n s_group = []\n t_group = []\n continue\n\n if s_buf == \"\" and i < len(s_pieces):\n s_buf += s_pieces[i]\n s_group.append(i)\n i += 1\n continue\n if t_buf == \"\" and j < len(t_pieces):\n t_buf += t_pieces[j]\n t_group.append(j)\n j += 1\n continue\n\n if len(s_buf) <= len(t_buf):\n if i < len(s_pieces):\n s_buf += s_pieces[i]\n s_group.append(i)\n i += 1\n elif j < len(t_pieces):\n t_buf += t_pieces[j]\n t_group.append(j)\n j += 1\n else:\n if j < len(t_pieces):\n t_buf += t_pieces[j]\n t_group.append(j)\n j += 1\n elif i < len(s_pieces):\n s_buf += s_pieces[i]\n s_group.append(i)\n i += 1\n\n # Flush any remainder if both sides accumulated something\n if s_buf == t_buf and s_group and t_group:\n flush()\n elif s_group or t_group:\n # Handle remaining unmatched tokens by forcing a flush\n # This ensures both sides have the same number of alignment groups\n if s_group or t_group:\n # Ensure both groups have content (even if empty list)\n if not s_group:\n s_group = []\n if not t_group:\n t_group = []\n # Force flush even if buffers don't match\n if s_group or t_group:\n s_groups.append(s_group.copy() if s_group else [])\n t_groups.append(t_group.copy() if t_group else [])\n\n return s_groups, t_groups\n\n def _merge_probabilities_with_alignment_groups(self, probs, alignment_groups, token_ids=None):\n \"\"\"\n Merge probabilities based on alignment groups with corrected conditional probability handling.\n\n For a group merging tokens at positions [i, i+1, ..., i+k], we compute:\n P_merged(y | x) = P(y | x) ร— P(token_{i+1} | token_i, x) ร— ... ร— P(token_{i+k} | ..., x)\n\n Where:\n - P(y | x) is the marginal probability distribution over all vocabulary tokens at position i\n - token_{i+1}, ..., token_{i+k} are the ACTUAL tokens that were generated\n - The conditional probabilities P(token_j | ..., x) are extracted as SCALARS\n - y ranges over all vocabulary tokens at position i\n\n This ensures the probability of the actual generated sequence is correct (by the chain rule), while introducing\n a known bias for counterfactual tokens (since we don't have P(token_{i+k} | y, x) for y != token_i). The merged\n distribution is unnormalized but preserves correct relative probabilities.\n\n Args:\n probs: Probability tensor [seq_len, vocab_size]\n alignment_groups: List of alignment groups (each group is a list of positions to merge)\n token_ids: Actual token IDs that were generated [seq_len]. REQUIRED when any group has\n len(group) > 1. If None when multi-token groups exist, raises ValueError.\n\n Returns:\n Merged probability tensor [num_groups, vocab_size]\n\n Raises:\n ValueError: If token_ids is None when merging multi-token groups\n \"\"\"\n if not alignment_groups:\n return probs\n\n # Create aligned tensor\n vocab_size = probs.size(-1)\n target_len = len(alignment_groups)\n aligned_probs = torch.zeros(target_len, vocab_size, device=probs.device, dtype=probs.dtype)\n eps = 1e-8\n\n # Process each alignment group\n for group_idx, group in enumerate(alignment_groups):\n # Handle probability merging\n if len(group) > 1:\n # Multiple tokens map to this group - merge using corrected conditional probability approach\n if token_ids is None:\n raise ValueError(\n \"token_ids must be provided when merging multi-token groups. \"\n \"This is required for mathematically correct probability merging.\"\n )\n\n # Start with the marginal distribution at the first position\n first_pos = group[0]\n marginal_probs = probs[first_pos] # P(y | xโ‚€) for all y\n\n # For each subsequent token in the group, extract the SCALAR conditional probability\n # of the actual token that was generated, and multiply\n conditional_prob_product = 1.0\n for idx in group[1:]:\n # Get the actual token ID that was generated at this position\n actual_token_id = token_ids[idx]\n # Extract its probability (scalar)\n token_prob = probs[idx, actual_token_id].clamp_min(eps)\n conditional_prob_product *= token_prob\n\n # Merge: multiply the scalar conditional prob product with the entire marginal distribution\n # This gives: P(y | x_0) ร— P(token_1 | token_0, x) ร— ... ร— P(token_k | ..., x)\n # Note: This is unnormalized, but preserves the correct joint probability for the actual sequence\n merged_probs = marginal_probs * conditional_prob_product\n aligned_probs[group_idx] = merged_probs\n\n elif len(group) == 1:\n aligned_probs[group_idx] = probs[group[0]]\n else:\n # No tokens map to this group\n aligned_probs[group_idx] = torch.zeros_like(probs[0])\n\n return aligned_probs\n\n def _compute_hybrid_uld_loss(self, student_aligned, teacher_aligned):\n \"\"\"\n Compute hybrid ULD loss on aligned probability distributions. This method:\n 1. Directly compares probabilities for tokens with matching vocabulary entries\n 2. Uses sorting approach only for tokens with different vocabulary entries\n\n Args:\n student_aligned: Aligned student probabilities [seq_len, student_vocab_size]\n teacher_aligned: Aligned teacher probabilities [seq_len, teacher_vocab_size]\n Returns:\n Combined hybrid loss\n \"\"\"\n device = student_aligned.device\n # seq_len = student_aligned.size(0) # Unused variable\n student_vocab_size = student_aligned.size(-1)\n teacher_vocab_size = teacher_aligned.size(-1)\n\n # Convert sets to sorted tensors for indexing\n if self._teacher_matched_ids:\n teacher_matched_indices = torch.tensor(sorted(self._teacher_matched_ids), dtype=torch.long, device=device)\n student_matched_indices = self.mapping_tensor[teacher_matched_indices]\n else:\n teacher_matched_indices = torch.tensor([], dtype=torch.long, device=device)\n student_matched_indices = torch.tensor([], dtype=torch.long, device=device)\n\n # Create masks for unmatched tokens\n teacher_matched_mask = torch.zeros(teacher_vocab_size, dtype=torch.bool, device=device)\n student_matched_mask = torch.zeros(student_vocab_size, dtype=torch.bool, device=device)\n\n if len(teacher_matched_indices) > 0:\n teacher_matched_mask[teacher_matched_indices] = True\n student_matched_mask[student_matched_indices] = True\n\n # 1. JSD loss for matched vocabulary tokens (direct semantic correspondence)\n matched_loss = torch.tensor(0.0, device=device)\n matched_token_count = 0\n if len(teacher_matched_indices) > 0:\n # Extract probabilities for matched tokens\n teacher_matched_probs = teacher_aligned[:, teacher_matched_indices] # [seq_len, num_matched]\n student_matched_probs = student_aligned[:, student_matched_indices] # [seq_len, num_matched]\n matched_token_count = teacher_matched_probs.size(-1)\n\n # Use JSD loss for semantically aligned tokens\n # Convert probabilities back to logits for JSD computation\n\n # Apply generalized JSD loss to matched tokens\n matched_loss = self._compute_jsd_loss_for_matched_tokens(student_matched_probs, teacher_matched_probs)\n\n # 2. Sorted comparison loss for unmatched vocabulary tokens\n teacher_unmatched_mask = ~teacher_matched_mask\n student_unmatched_mask = ~student_matched_mask\n\n teacher_unmatched_probs = teacher_aligned[:, teacher_unmatched_mask] # [seq_len, num_teacher_unmatched]\n student_unmatched_probs = student_aligned[:, student_unmatched_mask] # [seq_len, num_student_unmatched]\n\n unmatched_loss = torch.tensor(0.0, device=device)\n if teacher_unmatched_probs.size(-1) > 0 and student_unmatched_probs.size(-1) > 0:\n # Sort unmatched probabilities\n teacher_unmatched_sorted = teacher_unmatched_probs.sort(dim=-1, descending=True).values\n student_unmatched_sorted = student_unmatched_probs.sort(dim=-1, descending=True).values\n\n # Pad to same size if needed\n teacher_unmatched_size = teacher_unmatched_sorted.size(-1)\n student_unmatched_size = student_unmatched_sorted.size(-1)\n max_unmatched_size = max(teacher_unmatched_size, student_unmatched_size)\n\n if teacher_unmatched_size < max_unmatched_size:\n teacher_unmatched_sorted = F.pad(\n teacher_unmatched_sorted, (0, max_unmatched_size - teacher_unmatched_size)\n )\n if student_unmatched_size < max_unmatched_size:\n student_unmatched_sorted = F.pad(\n student_unmatched_sorted, (0, max_unmatched_size - student_unmatched_size)\n )\n\n # L1 loss on sorted unmatched tokens\n unmatched_loss = F.l1_loss(student_unmatched_sorted, teacher_unmatched_sorted, reduction=\"sum\")\n unmatched_loss /= student_aligned.size(0) # Normalize by sequence length\n\n # 3. Combine losses with weights\n if self.hybrid_matched_weight is None:\n # Use adaptive weighting based on vocabulary overlap\n hybrid_matched_weight = matched_token_count / max(1, teacher_vocab_size)\n hybrid_unmatched_weight = 1.0 - hybrid_matched_weight\n else:\n # Use fixed weights provided in config\n hybrid_matched_weight = self.hybrid_matched_weight\n hybrid_unmatched_weight = self.hybrid_unmatched_weight\n\n total_loss = hybrid_matched_weight * matched_loss + hybrid_unmatched_weight * unmatched_loss\n\n # Store matched/unmatched components for logging\n self.last_matched_loss = matched_loss\n self.last_unmatched_loss = unmatched_loss\n\n return total_loss\n\n def _compute_jsd_loss_for_matched_tokens(self, student_logits, teacher_logits):\n \"\"\"\n Compute JSD loss for matched vocabulary tokens.\n\n Args:\n student_logits: Student logits for matched tokens [seq_len, num_matched]\n teacher_logits: Teacher logits for matched tokens [seq_len, num_matched]\n Returns:\n JSD loss for matched tokens\n \"\"\"\n # Reshape to [batch_size * seq_len, vocab_size] format expected by generalized_jsd_loss\n batch_seq_len, num_matched = student_logits.shape\n\n student_logits_reshaped = student_logits.view(-1, num_matched)\n teacher_logits_reshaped = teacher_logits.view(-1, num_matched)\n\n # Use the GOLD generalized JSD loss implementation that accepts probability inputs\n jsd_loss = GOLDTrainer.generalized_jsd_loss(\n student_logits_reshaped,\n teacher_logits_reshaped,\n labels=None, # No masking needed for matched tokens\n beta=self.beta, # Standard JSD beta\n temperature=1.0, # Already applied in main computation\n reduction=\"batchmean\",\n logits_are_probs=True,\n )\n\n return jsd_loss\n\n def _get_start_and_size_answers(self, answer_tensors):\n answers_index = []\n answers_size = []\n\n for answer in answer_tensors:\n answer_mask = answer.ne(self.ignore_index)\n if not answer_mask.any():\n answers_index.append(0)\n answers_size.append(0)\n continue\n\n valid_indices = answer_mask.nonzero(as_tuple=True)[0]\n answers_index.append(int(valid_indices[0].item()))\n answers_size.append(int(answer_mask.sum().item()))\n return answers_index, answers_size\n\n\nclass GOLDVLLMSyncCallback(TrainerCallback):\n \"\"\"Sync the model weights to vLLM after training steps when it's safe to do so.\"\"\"\n\n def __init__(self, trainer):\n self.trainer = trainer\n\n def on_step_end(self, args, state: TrainerState, control: TrainerControl, **kwargs):\n \"\"\"Sync weights after training step when DeepSpeed is stable.\"\"\"\n if (\n self.trainer.use_vllm\n and state.global_step != self.trainer._last_vllm_sync_step\n and state.global_step % self.trainer.vllm_sync_frequency == 0\n ):\n # Check if this is a step where gradients are synchronized\n # This happens at the end of gradient accumulation cycles\n if hasattr(self.trainer.accelerator, \"sync_gradients\") and self.trainer.accelerator.sync_gradients:\n self.trainer._move_model_to_vllm()\n self.trainer._last_vllm_sync_step = state.global_step\n\n\nclass GOLDTrainer(SFTTrainer):\n _tag_names = [\"trl\", \"gold\"]\n _name = \"GOLD\"\n _paper = {\n \"title\": \"Unlocking On-Policy Distillation for Any Model Family\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @misc{patino2025unlocking,\n title = {{Unlocking On-Policy Distillation for Any Model Family}},\n author = {Carlos Miguel Patiรฑo and Kashif Rasul and Quentin Gallouรฉdec and Ben Burtenshaw and Sergio Paniego and Vaibhav Srivastav and Thibaud Frere and Ed Beeching and Lewis Tunstall and Leandro von Werra and Thomas Wolf},\n year = 2025,\n url = {https://huggingface.co/spaces/HuggingFaceH4/general-on-policy-logit-distillation},\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module | str | None = None,\n teacher_model: PreTrainedModel | nn.Module | str = None,\n args: GOLDConfig | None = None,\n data_collator: DataCollator | None = None, # type: ignore\n train_dataset: Dataset | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n processing_class: PreTrainedTokenizerBase\n | BaseImageProcessor\n | FeatureExtractionMixin\n | ProcessorMixin\n | None = None,\n compute_metrics: Callable[[EvalPrediction], dict] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n peft_config: Optional[\"PeftConfig\"] = None,\n ):\n self.model_name_or_path = model if isinstance(model, str) else model.config._name_or_path\n self.model_revision = (args.model_init_kwargs or {}).get(\"revision\")\n\n # Respect a user-provided data_collator; otherwise, provide a ChatML collator that\n if data_collator is None:\n data_collator = DataCollatorForChatML(tokenizer=processing_class, max_length=args.max_length)\n\n # Liger fused GKD loss (JSD)\n self.use_liger_gkd_loss = False\n if args.use_liger_kernel:\n self.liger_jsd_loss = LigerFusedLinearJSDLoss(\n beta=args.beta,\n ignore_index=-100,\n temperature=args.temperature,\n compiled=False,\n weight_hard_loss=0.0,\n weight_soft_loss=1.0,\n )\n self.use_liger_gkd_loss = True\n\n if args.teacher_model_init_kwargs is None:\n teacher_model_init_kwargs = {}\n elif not isinstance(teacher_model, str):\n raise ValueError(\n \"You passed teacher_model_init_kwargs to the GOLDConfig, but your teacher_model is already instantiated.\"\n )\n else:\n teacher_model_init_kwargs = args.teacher_model_init_kwargs\n teacher_model_init_kwargs[\"torch_dtype\"] = (\n teacher_model_init_kwargs[\"torch_dtype\"]\n if teacher_model_init_kwargs[\"torch_dtype\"] in [\"auto\", None]\n else getattr(torch, teacher_model_init_kwargs[\"torch_dtype\"])\n )\n\n if args.use_uld_loss and args.teacher_tokenizer_name_or_path is None:\n if isinstance(teacher_model, str):\n args.teacher_tokenizer_name_or_path = teacher_model\n else:\n raise ValueError(\n \"`teacher_tokenizer_name_or_path` must be set when using ULD loss with a pre-instantiated teacher model.\"\n )\n\n if isinstance(teacher_model, str):\n init_kwargs = dict(teacher_model_init_kwargs)\n if args.teacher_model_revision is not None:\n init_kwargs.setdefault(\"revision\", args.teacher_model_revision)\n if \"torch_dtype\" in init_kwargs and \"dtype\" not in init_kwargs:\n init_kwargs[\"dtype\"] = init_kwargs.pop(\"torch_dtype\")\n teacher_model = create_model_from_path(teacher_model, **init_kwargs)\n self.use_uld_loss = args.use_uld_loss\n self.teacher_tokenizer = None\n if args.use_uld_loss and args.teacher_tokenizer_name_or_path is not None:\n self.teacher_tokenizer = AutoTokenizer.from_pretrained(args.teacher_tokenizer_name_or_path)\n if not hasattr(self.teacher_tokenizer, \"pad_token\") or self.teacher_tokenizer.pad_token is None:\n self.teacher_tokenizer.pad_token = self.teacher_tokenizer.eos_token\n\n # Hybrid ULD loss configuration is handled in ULDLoss class\n\n super().__init__(\n model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n peft_config=peft_config,\n )\n\n if args.disable_dropout:\n disable_dropout_in_model(self.model)\n if not args.use_uld_loss:\n teacher_model.resize_token_embeddings(self.model.config.vocab_size)\n\n if self.is_deepspeed_enabled:\n self.teacher_model = prepare_deepspeed(teacher_model, self.accelerator)\n else:\n self.teacher_model = self.accelerator.prepare_model(teacher_model, evaluation_mode=True)\n\n self.lmbda = args.lmbda\n self.beta = args.beta\n self.temperature = args.temperature\n self.top_p = args.top_p\n self.seq_kd = args.seq_kd\n self.num_generations = args.num_generations\n\n # Track per-step loss statistics for on/off-policy batches (used in logging)\n self._on_policy_loss_total = 0.0\n self._off_policy_loss_total = 0.0\n self._on_policy_step_equiv = 0.0\n self._off_policy_step_equiv = 0.0\n\n # Buffering for rollouts across gradient accumulation steps\n self._buffered_inputs = None\n self._buffered_on_policy = None\n self._buffered_text_logs = None\n self._step = 0\n\n # Hybrid ULD matched/unmatched accumulators (logged every step when ULD hybrid is used)\n self._matched_sum = 0.0\n self._unmatched_sum = 0.0\n self._matched_step_eq = 0.0\n self._unmatched_step_eq = 0.0\n\n self.use_transformers_paged = args.use_transformers_paged or False\n\n self.uld_loss_fn = None\n if self.use_uld_loss:\n self.uld_loss_fn = ULDLoss(\n config=args,\n student_tokenizer=processing_class,\n teacher_tokenizer=self.teacher_tokenizer,\n device=self.accelerator.device,\n )\n\n generation_kwargs = {\n \"max_new_tokens\": args.max_completion_length,\n \"temperature\": args.temperature,\n \"top_p\": args.top_p,\n \"do_sample\": True,\n \"top_k\": args.top_k,\n \"pad_token_id\": self.processing_class.pad_token_id,\n }\n self.generation_config = GenerationConfig(**generation_kwargs)\n # Keep training-specific generation kwargs to overwrite model's original generation config\n self.generation_kwargs = generation_kwargs\n if (\n hasattr(self.model.generation_config, \"eos_token_id\")\n and self.model.generation_config.eos_token_id is not None\n ):\n self.generation_config.eos_token_id = self.model.generation_config.eos_token_id\n\n # Initialize the metrics\n self._metrics = {\"train\": defaultdict(list), \"eval\": defaultdict(list)}\n self._total_train_tokens = 0\n self.log_completions = args.log_completions\n self.log_completion_steps = args.log_completions_steps\n self.wandb_log_unique_prompts = args.wandb_log_unique_prompts\n self.num_completions_to_print = args.num_completions_to_print\n # maxlen is set to the total number of forward passes per step. This value of `maxlen` ensures we log only the\n # final optimization step.\n maxlen = self.accelerator.num_processes * args.per_device_train_batch_size * args.gradient_accumulation_steps\n self._textual_logs = {\n \"prompt\": deque(maxlen=maxlen),\n \"completion\": deque(maxlen=maxlen),\n \"rewards\": defaultdict(lambda: deque(maxlen=maxlen)),\n \"advantages\": deque(maxlen=maxlen),\n }\n\n self.use_vllm = args.use_vllm\n if self.use_vllm:\n if not is_vllm_available():\n raise ImportError(\n \"vLLM is not available and use_vllm is set to True. Please install vLLM with \"\n \"`pip install vllm` to use it.\"\n )\n self.vllm_mode = args.vllm_mode\n self.vllm_tensor_parallel_size = args.vllm_tensor_parallel_size\n self.vllm_gpu_memory_utilization = args.vllm_gpu_memory_utilization\n self.vllm_enable_sleep_mode = args.vllm_enable_sleep_mode\n if self.vllm_mode == \"server\":\n if self.accelerator.is_main_process:\n self.vllm_client = VLLMClient(\n host=args.vllm_server_host,\n server_port=args.vllm_server_port,\n connection_timeout=args.vllm_server_timeout,\n )\n self.vllm_client.init_communicator()\n elif self.vllm_mode == \"colocate\":\n student_model_name_or_path = self.model_name_or_path\n\n # Make sure tensor_parallel_size divides world size evenly\n if not self.accelerator.num_processes % self.vllm_tensor_parallel_size == 0:\n raise ValueError(\n f\"vllm_tensor_parallel_size ({self.vllm_tensor_parallel_size}) must divide world size \"\n f\"({self.accelerator.num_processes}) evenly.\"\n )\n\n if self.vllm_tensor_parallel_size > 1:\n # Create subgroups of ranks for TP\n self.vllm_tp_group, _ = torch.distributed.new_subgroups_by_enumeration(\n [\n list(\n range(\n i * self.vllm_tensor_parallel_size,\n (i + 1) * self.vllm_tensor_parallel_size,\n )\n )\n for i in range(self.accelerator.num_processes // self.vllm_tensor_parallel_size)\n ]\n )\n\n # vLLM requires the environment variables to be set for distributed training.\n os.environ[\"RANK\"] = str(self.accelerator.process_index)\n os.environ[\"LOCAL_RANK\"] = str(self.accelerator.local_process_index)\n os.environ[\"WORLD_SIZE\"] = str(self.accelerator.num_processes)\n ensure_master_addr_port()\n\n vllm_quantization = None\n if is_bitsandbytes_available():\n for _, module in model.named_modules():\n if isinstance(module, bnb.nn.Linear4bit):\n vllm_quantization = \"bitsandbytes\"\n break\n elif isinstance(module, bnb.nn.Linear8bitLt):\n raise ValueError(\"vLLM does not support in-flight 8-bit quantization.\")\n\n self.vllm_engine = LLM(\n model=student_model_name_or_path,\n revision=self.model_revision,\n tensor_parallel_size=self.vllm_tensor_parallel_size,\n gpu_memory_utilization=self.vllm_gpu_memory_utilization,\n max_num_seqs=self.args.per_device_train_batch_size * self.args.gradient_accumulation_steps,\n max_model_len=args.max_length,\n distributed_executor_backend=\"external_launcher\",\n # Feed identical seed for tp groups to ensure sampling results are the same across workers\n seed=self.accelerator.process_index // self.vllm_tensor_parallel_size,\n enable_sleep_mode=self.vllm_enable_sleep_mode,\n quantization=vllm_quantization,\n )\n\n if self.vllm_enable_sleep_mode:\n self.vllm_engine.sleep(level=2)\n\n # When using vLLM, the main process is responsible for loading the model weights. This can cause process\n # desynchronization and seems to lead to DeepSpeed hanging during initialization. To prevent this, we\n # synchronize all processes after vLLM has been fully initialized.\n self.accelerator.wait_for_everyone()\n else:\n raise ValueError(f\"Unknown vllm_mode: {self.vllm_mode}\")\n self.vllm_structured_outputs_regex = args.vllm_structured_outputs_regex\n self.vllm_sync_frequency = args.vllm_sync_frequency\n self._last_vllm_sync_step = -1\n\n self.add_callback(GOLDVLLMSyncCallback(self))\n\n def _set_signature_columns_if_needed(self):\n super()._set_signature_columns_if_needed()\n required_columns = [\n \"prompts\",\n \"prompt_attention_mask\",\n \"messages\",\n \"chat_template_kwargs\",\n \"tools\",\n \"original_prompt_text\",\n \"original_completion_text\",\n ]\n if self._signature_columns is None:\n self._signature_columns = required_columns\n else:\n for column in required_columns:\n if column not in self._signature_columns:\n self._signature_columns.append(column)\n\n def _get_train_sampler(self, dataset=None):\n if dataset is None:\n dataset = self.train_dataset\n return RepeatSampler(\n data_source=dataset,\n mini_repeat_count=self.num_generations,\n batch_size=self.args.generation_batch_size * self.accelerator.num_processes,\n repeat_count=self.args.gradient_accumulation_steps,\n shuffle=True,\n seed=self.args.seed,\n )\n\n def get_train_dataloader(self):\n \"\"\"\n Override Trainer.get_train_dataloader to load one generation batch per optimizer window.\n\n The dataloader yields local batches of size `per_device_train_batch_size * gradient_accumulation_steps`. The\n `RepeatSampler` (with `repeat_count=gradient_accumulation_steps`) ensures each generation batch is sampled\n `gradient_accumulation_steps` times so Trainer's loop iterates the correct number of times. Only the first\n batch in each window triggers `_fill_buffer`; the rest are ignored by `_prepare_inputs`.\n \"\"\"\n if self.train_dataset is None:\n raise ValueError(\"Trainer: training requires a train_dataset.\")\n\n train_dataset = self.train_dataset\n data_collator = self.data_collator\n if is_datasets_available() and isinstance(train_dataset, Dataset):\n train_dataset = self._remove_unused_columns(train_dataset, description=\"training\")\n else:\n data_collator = self._get_collator_with_removed_columns(data_collator, description=\"training\")\n\n dataloader_params = {\n \"batch_size\": self._train_batch_size * self.args.gradient_accumulation_steps,\n \"collate_fn\": data_collator,\n \"num_workers\": self.args.dataloader_num_workers,\n \"pin_memory\": self.args.dataloader_pin_memory,\n \"persistent_workers\": self.args.dataloader_persistent_workers,\n }\n\n if not isinstance(train_dataset, torch.utils.data.IterableDataset):\n dataloader_params[\"sampler\"] = self._get_train_sampler()\n dataloader_params[\"drop_last\"] = self.args.dataloader_drop_last\n dataloader_params[\"worker_init_fn\"] = partial(\n seed_worker,\n num_workers=self.args.dataloader_num_workers,\n rank=self.args.process_index,\n )\n if self.args.dataloader_num_workers > 0:\n dataloader_params[\"prefetch_factor\"] = self.args.dataloader_prefetch_factor\n\n return self.accelerator.prepare(DataLoader(train_dataset, **dataloader_params))\n\n @profiling_decorator\n def _prepare_inputs(self, generation_batch: dict[str, torch.Tensor | Any]) -> dict[str, torch.Tensor | Any]:\n if not self.model.training:\n return generation_batch\n\n buffer_steps = self.args.gradient_accumulation_steps\n if self._step % buffer_steps == 0 or self._buffered_inputs is None:\n self._fill_buffer(generation_batch, buffer_steps)\n\n slice_idx = self._step % buffer_steps\n inputs = self._buffered_inputs[slice_idx]\n self._step += 1\n return inputs\n\n def _decode_completion_texts_from_labels(self, slice_inputs: dict[str, torch.Tensor | Any]) -> list[str] | None:\n \"\"\"Decode completion text from labels when raw text is absent.\"\"\"\n labels = slice_inputs.get(\"labels\")\n if labels is None or not isinstance(labels, torch.Tensor):\n return None\n\n labels_cpu = labels.detach().cpu()\n decoded_completion_tokens: list[list[int]] = []\n for row in labels_cpu:\n token_ids = row[row != -100].tolist()\n if self.processing_class.pad_token_id is not None:\n token_ids = [tok for tok in token_ids if tok != self.processing_class.pad_token_id]\n decoded_completion_tokens.append(token_ids)\n\n return self.processing_class.batch_decode(\n decoded_completion_tokens,\n skip_special_tokens=False,\n clean_up_tokenization_spaces=False,\n )\n\n def _ensure_original_text_fields(\n self, slice_inputs: dict[str, torch.Tensor | Any]\n ) -> dict[str, torch.Tensor | Any]:\n \"\"\"Populate original prompt/completion text fields when missing.\"\"\"\n if \"original_prompt_text\" in slice_inputs and \"original_completion_text\" in slice_inputs:\n return slice_inputs\n\n prompts = slice_inputs.get(\"prompts\")\n if prompts is None or not isinstance(prompts, torch.Tensor):\n return slice_inputs\n\n prompt_texts = self.processing_class.batch_decode(\n prompts,\n skip_special_tokens=False,\n clean_up_tokenization_spaces=False,\n )\n completion_texts = self._decode_completion_texts_from_labels(slice_inputs)\n if completion_texts is None:\n return slice_inputs\n\n updated_slice = dict(slice_inputs)\n updated_slice[\"original_prompt_text\"] = prompt_texts\n updated_slice[\"original_completion_text\"] = completion_texts\n return updated_slice\n\n @staticmethod\n def _build_sequence_batch(\n new_input_ids: torch.Tensor, prompt_lengths: torch.Tensor, pad_token_id: int | None\n ) -> tuple[torch.Tensor, torch.Tensor]:\n \"\"\"Build attention mask and labels from full sequences and prompt lengths.\"\"\"\n prompt_lengths = prompt_lengths.to(device=new_input_ids.device, dtype=torch.long)\n positions = torch.arange(new_input_ids.shape[1], device=new_input_ids.device).unsqueeze(0)\n completion_mask = positions >= prompt_lengths.unsqueeze(1)\n\n new_attention_mask = torch.ones_like(new_input_ids)\n if pad_token_id is not None:\n new_attention_mask[new_input_ids == pad_token_id] = 0\n\n new_labels = torch.full_like(new_input_ids, -100)\n new_labels[completion_mask] = new_input_ids[completion_mask]\n if pad_token_id is not None:\n new_labels[new_input_ids == pad_token_id] = -100\n\n return new_attention_mask, new_labels\n\n @profiling_decorator\n def _fill_buffer(self, generation_batch: dict[str, torch.Tensor | Any], buffer_steps: int):\n slices = split_tensor_dict(generation_batch, buffer_steps)\n\n if self.accelerator.is_main_process:\n on_policy_flags = [random.random() <= self.lmbda for _ in range(buffer_steps)]\n else:\n on_policy_flags = [False] * buffer_steps\n\n on_policy_flags = broadcast_object_list(on_policy_flags, from_process=0)\n on_policy_indices = [i for i, flag in enumerate(on_policy_flags) if flag]\n\n self._buffered_inputs = [None] * buffer_steps\n self._buffered_on_policy = on_policy_flags\n self._buffered_text_logs = [None] * buffer_steps\n\n for i, flag in enumerate(on_policy_flags):\n if not flag:\n slice_inputs = slices[i]\n\n if self.use_uld_loss and self.teacher_tokenizer is not None:\n slice_inputs = self._ensure_original_text_fields(slice_inputs)\n if \"original_prompt_text\" not in slice_inputs or \"original_completion_text\" not in slice_inputs:\n raise ValueError(\n \"Off-policy batch missing 'original_prompt_text' or 'original_completion_text' fields. \"\n \"When using ULD loss with cross-tokenizer alignment, datasets must be prepared with \"\n \"_prepare_dataset_with_original_text(). Ensure your dataset includes these fields.\"\n )\n\n self._buffered_inputs[i] = slice_inputs\n\n if on_policy_indices:\n self._generate_on_policy_for_slices(slices, on_policy_indices)\n\n @profiling_decorator\n def _generate_on_policy_for_slices(\n self, slices: list[dict[str, torch.Tensor | Any]], on_policy_indices: list[int]\n ):\n local_prompts = []\n local_slice_indices = []\n for slice_idx in on_policy_indices:\n slice_inputs = slices[slice_idx]\n for prompt in slice_inputs[\"prompts\"]:\n local_prompts.append(prompt)\n local_slice_indices.append(slice_idx)\n\n prompts_text_for_vllm = self.processing_class.batch_decode(\n torch.stack(local_prompts) if local_prompts else torch.empty(0, dtype=torch.long),\n skip_special_tokens=True,\n )\n if self.processing_class.pad_token:\n prompts_text_for_vllm = [p.replace(self.processing_class.pad_token, \"\") for p in prompts_text_for_vllm]\n\n prompts_text_with_special = self.processing_class.batch_decode(\n torch.stack(local_prompts) if local_prompts else torch.empty(0, dtype=torch.long),\n skip_special_tokens=False,\n )\n\n if self.use_vllm:\n self._wake_vllm_if_needed()\n\n max_completion_length = self.generation_config.max_new_tokens\n temperature = self.generation_config.temperature\n top_k = (\n self.generation_config.top_k if self.generation_config.top_k and self.generation_config.top_k > 0 else -1\n )\n top_p = self.args.top_p if hasattr(self.args, \"top_p\") else 1.0\n repetition_penalty = self.args.repetition_penalty if hasattr(self.args, \"repetition_penalty\") else 1.0\n min_p = self.args.min_p if hasattr(self.args, \"min_p\") else 0.0\n\n if self.use_vllm and self.vllm_mode == \"server\":\n completion_ids = self._generate_vllm_server_global(\n prompts_text_for_vllm,\n max_completion_length,\n temperature,\n top_k,\n top_p,\n repetition_penalty,\n min_p,\n n=self.num_generations,\n )\n elif self.use_vllm and self.vllm_mode == \"colocate\":\n completion_ids = self._generate_vllm_colocate(\n prompts_text_for_vllm,\n max_completion_length,\n temperature,\n top_k,\n top_p,\n repetition_penalty,\n min_p,\n n=self.num_generations,\n )\n else:\n self._generate_non_vllm_for_slices(slices, on_policy_indices)\n return\n\n self._process_completions_to_buffer(\n slices,\n on_policy_indices,\n local_slice_indices,\n completion_ids,\n prompts_text_for_vllm,\n prompts_text_with_special,\n max_completion_length,\n )\n\n @staticmethod\n def _deduplicate_prompts(\n prompts: list[str], num_generations: int\n ) -> tuple[list[str], list[tuple[int, int]]] | None:\n \"\"\"Deduplicate prompts and build a completion remapping.\"\"\"\n seen: dict[str, list[int]] = {}\n unique_prompts: list[str] = []\n dedup_mapping: list[tuple[int, int]] = []\n\n for prompt in prompts:\n if prompt not in seen:\n seen[prompt] = [len(unique_prompts), 0]\n unique_prompts.append(prompt)\n entry = seen[prompt]\n if entry[1] >= num_generations:\n return None\n dedup_mapping.append((entry[0], entry[1]))\n entry[1] += 1\n\n return unique_prompts, dedup_mapping\n\n def _generate_vllm_server_global(\n self,\n prompts_text: list[str],\n max_tokens: int,\n temperature: float,\n top_k: int,\n top_p: float,\n repetition_penalty: float,\n min_p: float,\n n: int = 1,\n ) -> list:\n all_prompts_text = gather_object(prompts_text)\n local_count = len(prompts_text)\n\n if self.accelerator.is_main_process:\n if all_prompts_text:\n dedup_mapping = None\n if n > 1:\n dedup_result = self._deduplicate_prompts(all_prompts_text, n)\n if dedup_result is not None:\n gen_prompts, dedup_mapping = dedup_result\n gen_n = n\n else:\n gen_prompts = all_prompts_text\n gen_n = 1\n else:\n gen_prompts = all_prompts_text\n gen_n = 1\n\n completion_ids = self.vllm_client.generate(\n prompts=gen_prompts,\n n=gen_n,\n repetition_penalty=repetition_penalty,\n temperature=temperature,\n top_p=top_p,\n top_k=top_k,\n min_p=min_p,\n max_tokens=max_tokens,\n structured_outputs_regex=self.vllm_structured_outputs_regex,\n )[\"completion_ids\"]\n\n if dedup_mapping is not None:\n completion_ids = [completion_ids[uid * gen_n + gid] for uid, gid in dedup_mapping]\n else:\n completion_ids = []\n else:\n completion_ids = [None] * len(all_prompts_text) if all_prompts_text else []\n\n completion_ids = broadcast_object_list(completion_ids, from_process=0)\n process_slice = slice(\n self.accelerator.process_index * local_count,\n (self.accelerator.process_index + 1) * local_count,\n )\n return completion_ids[process_slice]\n\n def _generate_vllm_colocate(\n self,\n prompts_text: list[str],\n max_tokens: int,\n temperature: float,\n top_k: int,\n top_p: float,\n repetition_penalty: float,\n min_p: float,\n n: int = 1,\n ) -> list:\n if self.vllm_structured_outputs_regex:\n structured_outputs = StructuredOutputsParams(backend=\"outlines\", regex=self.vllm_structured_outputs_regex)\n else:\n structured_outputs = None\n\n if hasattr(self, \"vllm_tp_group\") and self.vllm_tensor_parallel_size > 1:\n orig_size = len(prompts_text)\n gathered_prompts = [None for _ in range(self.vllm_tensor_parallel_size)]\n torch.distributed.all_gather_object(gathered_prompts, prompts_text, group=self.vllm_tp_group)\n all_prompts_text = [p for sublist in gathered_prompts for p in sublist]\n else:\n all_prompts_text = prompts_text\n\n dedup_mapping = None\n if n > 1 and all_prompts_text:\n dedup_result = self._deduplicate_prompts(all_prompts_text, n)\n if dedup_result is not None:\n gen_prompts, dedup_mapping = dedup_result\n gen_n = n\n else:\n gen_prompts = all_prompts_text\n gen_n = 1\n else:\n gen_prompts = all_prompts_text\n gen_n = 1\n\n sampling_params = SamplingParams(\n n=gen_n,\n repetition_penalty=repetition_penalty,\n temperature=temperature,\n top_p=top_p,\n top_k=top_k,\n min_p=min_p,\n max_tokens=max_tokens,\n structured_outputs=structured_outputs,\n )\n\n if gen_prompts:\n all_outputs = self.vllm_engine.generate(gen_prompts, sampling_params=sampling_params, use_tqdm=False)\n completion_ids = [output.token_ids for outputs in all_outputs for output in outputs.outputs]\n else:\n completion_ids = []\n\n if dedup_mapping is not None:\n completion_ids = [completion_ids[uid * gen_n + gid] for uid, gid in dedup_mapping]\n\n if hasattr(self, \"vllm_tp_group\") and self.vllm_tensor_parallel_size > 1:\n local_rank_in_group = torch.distributed.get_rank(group=self.vllm_tp_group)\n tp_slice = slice(local_rank_in_group * orig_size, (local_rank_in_group + 1) * orig_size)\n completion_ids = completion_ids[tp_slice]\n\n if self.vllm_enable_sleep_mode:\n self.vllm_engine.sleep(level=2)\n\n return completion_ids\n\n def _generate_non_vllm_for_slices(self, slices: list[dict[str, torch.Tensor | Any]], on_policy_indices: list[int]):\n \"\"\"Fallback generation without vLLM (uses model.generate per slice).\"\"\"\n with unwrap_model_for_generation(\n self.model,\n self.accelerator,\n generation_kwargs=self.generation_kwargs,\n ) as unwrapped_model:\n for slice_idx in on_policy_indices:\n slice_inputs = slices[slice_idx]\n result = self.generate_on_policy_outputs(\n unwrapped_model,\n slice_inputs,\n self.generation_config,\n self.processing_class.pad_token_id,\n )\n new_input_ids, new_attention_mask, new_labels, prompt_texts, completion_texts = result\n\n updated_slice = dict(slice_inputs)\n updated_slice[\"input_ids\"] = new_input_ids\n updated_slice[\"attention_mask\"] = new_attention_mask\n updated_slice[\"labels\"] = new_labels\n updated_slice[\"original_prompt_text\"] = prompt_texts\n updated_slice[\"original_completion_text\"] = completion_texts\n\n self._buffered_inputs[slice_idx] = updated_slice\n self._buffered_text_logs[slice_idx] = (prompt_texts, completion_texts)\n\n def _process_completions_to_buffer(\n self,\n slices: list[dict[str, torch.Tensor | Any]],\n on_policy_indices: list[int],\n local_slice_indices: list[int],\n completion_ids: list,\n prompts_text: list[str],\n prompts_text_with_special: list[str],\n max_completion_length: int,\n ):\n \"\"\"\n Process vLLM completions and update buffered inputs for on-policy slices.\n \"\"\"\n device = self.accelerator.device\n pad_token_id = self.processing_class.pad_token_id if self.processing_class.pad_token_id is not None else 0\n\n slice_completions = {idx: [] for idx in on_policy_indices}\n slice_prompts = {idx: [] for idx in on_policy_indices}\n slice_prompts_special = {idx: [] for idx in on_policy_indices}\n\n for i, slice_idx in enumerate(local_slice_indices):\n slice_completions[slice_idx].append(completion_ids[i])\n slice_prompts[slice_idx].append(prompts_text[i])\n slice_prompts_special[slice_idx].append(prompts_text_with_special[i])\n\n for slice_idx in on_policy_indices:\n slice_inputs = slices[slice_idx]\n completion_ids_for_slice = slice_completions[slice_idx]\n prompt_txts = slice_prompts[slice_idx]\n prompt_txts_with_special = slice_prompts_special[slice_idx]\n\n prompt_max_length = max(1, self.args.max_length - max_completion_length) if self.args.max_length else None\n prompt_tokenized = self.processing_class(\n prompt_txts,\n return_tensors=\"pt\",\n padding=\"longest\",\n padding_side=\"left\",\n truncation=True if prompt_max_length else False,\n max_length=prompt_max_length,\n add_special_tokens=False,\n ).to(device)\n prompt_ids = prompt_tokenized.input_ids\n\n completion_ids_tensors = [torch.tensor(ids, device=device) for ids in completion_ids_for_slice]\n completion_ids_for_text: list[list[int]] = []\n padded_completion_ids_list = []\n for completion_tensor in completion_ids_tensors:\n if len(completion_tensor) > max_completion_length:\n truncated_completion_tensor = completion_tensor[:max_completion_length]\n padded_completion_ids_list.append(truncated_completion_tensor)\n completion_ids_for_text.append(truncated_completion_tensor.tolist())\n elif len(completion_tensor) < max_completion_length:\n padding_needed = max_completion_length - len(completion_tensor)\n padded_tensor = torch.cat(\n [\n completion_tensor,\n torch.full(\n (padding_needed,),\n pad_token_id,\n device=device,\n dtype=completion_tensor.dtype,\n ),\n ]\n )\n padded_completion_ids_list.append(padded_tensor)\n completion_ids_for_text.append(completion_tensor.tolist())\n else:\n padded_completion_ids_list.append(completion_tensor)\n completion_ids_for_text.append(completion_tensor.tolist())\n\n completion_ids_padded = torch.stack(padded_completion_ids_list)\n\n new_input_ids = torch.cat([prompt_ids, completion_ids_padded], dim=1)\n prompt_lengths = torch.full((prompt_ids.shape[0],), prompt_ids.shape[1], device=device)\n new_attention_mask, new_labels = self._build_sequence_batch(new_input_ids, prompt_lengths, pad_token_id)\n\n completion_texts = self.processing_class.batch_decode(\n completion_ids_for_text,\n skip_special_tokens=False,\n clean_up_tokenization_spaces=False,\n )\n\n updated_slice = dict(slice_inputs)\n updated_slice[\"input_ids\"] = new_input_ids\n updated_slice[\"attention_mask\"] = new_attention_mask\n updated_slice[\"labels\"] = new_labels\n updated_slice[\"original_prompt_text\"] = prompt_txts_with_special\n updated_slice[\"original_completion_text\"] = completion_texts\n\n self._buffered_inputs[slice_idx] = updated_slice\n self._buffered_text_logs[slice_idx] = (prompt_txts, completion_texts)\n\n def _prepare_dataset(\n self,\n dataset: Dataset | IterableDataset,\n processing_class: PreTrainedTokenizerBase | BaseImageProcessor | FeatureExtractionMixin | ProcessorMixin,\n args,\n packing: bool,\n formatting_func: Callable[[dict], str] | None,\n dataset_name: str,\n ) -> Dataset | IterableDataset:\n \"\"\"Preserve original text fields for ULD when needed.\"\"\"\n column_names = list(next(iter(dataset)).keys())\n is_processed = \"input_ids\" in column_names\n\n if not is_processed or (self.use_uld_loss and self.teacher_tokenizer is not None):\n return self._prepare_dataset_with_original_text(\n dataset, processing_class, args, packing, formatting_func, dataset_name\n )\n\n return super()._prepare_dataset(dataset, processing_class, args, packing, formatting_func, dataset_name)\n\n def _prepare_dataset_with_original_text(\n self,\n dataset: Dataset | IterableDataset,\n processing_class: PreTrainedTokenizerBase | BaseImageProcessor | FeatureExtractionMixin | ProcessorMixin,\n args,\n packing: bool,\n formatting_func: Callable[[dict], str] | None,\n dataset_name: str,\n ) -> Dataset | IterableDataset:\n \"\"\"\n Prepare dataset while preserving original text for cross-tokenizer distillation.\n \"\"\"\n # Build the kwargs for the `map` function\n map_kwargs = {}\n if isinstance(dataset, Dataset): # IterableDataset does not support num_proc\n map_kwargs[\"num_proc\"] = args.dataset_num_proc\n\n with PartialState().main_process_first():\n # Apply the formatting function if any\n if formatting_func is not None:\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Applying formatting function to {dataset_name} dataset\"\n\n def _func(example):\n return {\"text\": formatting_func(example)}\n\n dataset = dataset.map(_func, batched=False, **map_kwargs)\n\n # Convert the dataset to ChatML if needed\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Converting {dataset_name} dataset to ChatML\"\n column_names = next(iter(dataset)).keys()\n dataset = dataset.map(\n maybe_convert_to_chatml,\n remove_columns=\"conversations\" if \"conversations\" in column_names else None,\n **map_kwargs,\n )\n\n # Apply the chat template if needed and preserve original text\n first_example = next(iter(dataset))\n if not is_conversational(first_example):\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Adding EOS to {dataset_name} dataset\"\n\n def add_eos(example, eos_token):\n if \"text\" in example and not example[\"text\"].endswith(eos_token): # language modeling case\n example[\"text\"] = example[\"text\"] + eos_token\n elif \"completion\" in example and not example[\"completion\"].endswith(eos_token):\n example[\"completion\"] = example[\"completion\"] + eos_token\n return example\n\n dataset = dataset.map(\n add_eos,\n fn_kwargs={\"eos_token\": processing_class.eos_token},\n remove_columns=\"messages\" if \"messages\" in column_names else None, # renamed to \"text\"\n **map_kwargs,\n )\n\n # Tokenize the dataset while preserving original text\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Tokenizing {dataset_name} dataset (preserving original text)\"\n\n def tokenize_with_original_text(example, processing_class, dataset_text_field, assistant_only_loss):\n \"\"\"Modified tokenization function that preserves original text.\"\"\"\n result = {}\n\n if \"prompt\" in example: # prompt-completion case\n # Store original text\n result[\"original_prompt_text\"] = example[\"prompt\"]\n result[\"original_completion_text\"] = example[\"completion\"]\n\n if is_conversational(example):\n prompt_ids = processing_class.apply_chat_template(\n example[\"prompt\"], return_dict=False, **example.get(\"chat_template_kwargs\", {})\n )\n prompt_completion_ids = processing_class.apply_chat_template(\n example[\"prompt\"] + example[\"completion\"],\n return_dict=False,\n **example.get(\"chat_template_kwargs\", {}),\n )\n else:\n prompt_ids = processing_class(text=example[\"prompt\"]).input_ids\n prompt_completion_ids = processing_class(\n text=example[\"prompt\"] + example[\"completion\"]\n ).input_ids\n\n # Check if the tokenized prompt starts with the tokenized prompt+completion\n if not prompt_completion_ids[: len(prompt_ids)] == prompt_ids:\n warnings.warn(\n \"Mismatch between tokenized prompt and the start of tokenized prompt+completion. \"\n \"This may be due to unexpected tokenizer behavior, whitespace issues, or special \"\n \"token handling. Verify that the tokenizer is processing text consistently.\",\n stacklevel=2,\n )\n\n # Create a completion mask\n completion_mask = [0] * len(prompt_ids) + [1] * (len(prompt_completion_ids) - len(prompt_ids))\n result.update(\n {\n \"input_ids\": prompt_completion_ids,\n \"completion_mask\": completion_mask,\n \"attention_mask\": [1] * len(prompt_completion_ids), # Add attention mask\n }\n )\n\n else: # language modeling or conversational case\n if is_conversational(example):\n # For conversational data (ChatML), extract prompt and completion properly\n messages = example[\"messages\"]\n\n # Extract user and assistant messages separately\n user_messages = [msg for msg in messages if msg[\"role\"] != \"assistant\"]\n assistant_messages = [msg for msg in messages if msg[\"role\"] == \"assistant\"]\n\n if user_messages and assistant_messages:\n # Apply chat template to get the prompt (everything up to assistant)\n prompt_text = processing_class.apply_chat_template(\n user_messages,\n add_generation_prompt=True, # add assistant prompt\n tokenize=False,\n **example.get(\"chat_template_kwargs\", {}),\n )\n\n # Get the full conversation with assistant response\n full_text = processing_class.apply_chat_template(\n messages,\n add_generation_prompt=False,\n tokenize=False,\n **example.get(\"chat_template_kwargs\", {}),\n )\n\n # Extract completion as everything after the prompt\n # This ensures we capture any extra tokens (like tags) that the template adds\n if full_text.startswith(prompt_text):\n completion_text = full_text[len(prompt_text) :]\n else:\n # Fallback: use assistant content + EOS\n assistant_content = assistant_messages[0][\"content\"]\n completion_text = (\n assistant_content + processing_class.eos_token\n if hasattr(processing_class, \"eos_token\")\n else assistant_content\n )\n\n # Store original text for cross-tokenizer distillation\n result[\"original_prompt_text\"] = prompt_text\n result[\"original_completion_text\"] = completion_text\n else:\n # Fallback: use empty prompt and full text as completion\n full_text = processing_class.apply_chat_template(\n messages, tokenize=False, **example.get(\"chat_template_kwargs\", {})\n )\n result[\"original_prompt_text\"] = \"\"\n result[\"original_completion_text\"] = full_text\n\n # Process the conversation normally\n processed = processing_class.apply_chat_template(\n example[\"messages\"],\n return_dict=True,\n return_assistant_tokens_mask=assistant_only_loss,\n **example.get(\"chat_template_kwargs\", {}),\n )\n if \"assistant_masks\" in processed and 1 not in processed[\"assistant_masks\"]:\n raise RuntimeError(\n \"You're using `assistant_only_loss=True`, but at least one example has no \"\n \"assistant tokens. This usually means the tokenizer's chat template doesn't \"\n \"generate assistant masks โ€” it may be missing the `{% generation %}` tag. Please \"\n \"check the template and ensure it's correctly configured to support assistant \"\n \"masking.\"\n )\n result.update({k: processed[k] for k in (\"input_ids\", \"assistant_masks\") if k in processed})\n # Add attention_mask if not already present\n if \"attention_mask\" not in result:\n result[\"attention_mask\"] = [1] * len(result[\"input_ids\"])\n else:\n # For regular language modeling, store the full text as completion and empty prompt\n result[\"original_prompt_text\"] = \"\"\n result[\"original_completion_text\"] = example.get(dataset_text_field, example.get(\"text\", \"\"))\n\n tokenized = processing_class(text=example[dataset_text_field])\n result.update(\n {\n \"input_ids\": tokenized.input_ids,\n \"attention_mask\": getattr(tokenized, \"attention_mask\", [1] * len(tokenized.input_ids)),\n }\n )\n\n return result\n\n dataset = dataset.map(\n tokenize_with_original_text,\n fn_kwargs={\n \"processing_class\": processing_class,\n \"dataset_text_field\": args.dataset_text_field,\n \"assistant_only_loss\": args.assistant_only_loss,\n },\n **map_kwargs,\n )\n\n # Pack or truncate\n if packing:\n if args.max_length is None:\n raise ValueError(\"When packing is enabled, `max_length` can't be `None`.\")\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Packing {dataset_name} dataset\"\n\n columns_to_keep = [\"input_ids\", \"original_prompt_text\", \"original_completion_text\"]\n existing_columns = set(dataset.column_names)\n columns_to_select = [col for col in columns_to_keep if col in existing_columns]\n\n dataset = dataset.select_columns(columns_to_select)\n dataset = pack_dataset(dataset, args.max_length, args.packing_strategy, map_kwargs)\n elif args.max_length is not None:\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Truncating {dataset_name} dataset\"\n dataset = truncate_dataset(dataset, args.max_length, map_kwargs=map_kwargs)\n\n if args.use_liger_kernel:\n required_columns = {\n \"input_ids\",\n \"attention_mask\",\n \"position_ids\",\n \"completion_mask\",\n \"messages\",\n \"assistant_masks\",\n \"original_prompt_text\",\n \"original_completion_text\",\n }\n dataset = dataset.select_columns(required_columns.intersection(dataset.column_names))\n\n return dataset\n\n @staticmethod\n def generalized_jsd_loss(\n student_logits,\n teacher_logits,\n labels=None,\n beta=0.5,\n temperature=1.0,\n reduction=\"batchmean\",\n logits_are_probs=False,\n ):\n \"\"\"\n Compute the generalized Jensen-Shannon Divergence loss for knowledge distillation using F.kl_div. See Eq. (1)\n of https://huggingface.co/papers/2306.13649 for the definition.\n\n Args:\n student_logits:\n Tensor of shape (batch_size, sequence_length, vocab_size)\n teacher_logits:\n Tensor of shape (batch_size, sequence_length, vocab_size)\n labels:\n Tensor of shape (batch_size, sequence_length) with -100 for padding tokens to ignore when computing\n loss\n beta:\n Interpolation coefficient between 0 and 1 (default: 0.5)\n temperature:\n Softmax temperature (default: 1.0)\n reduction:\n Specifies the reduction to apply to the output (default: 'batchmean')\n\n Returns:\n loss: Scalar tensor with the generalized JSD loss\n \"\"\"\n\n if logits_are_probs:\n student_log_probs = torch.log(student_logits.clamp_min(1e-8))\n teacher_log_probs = torch.log(teacher_logits.clamp_min(1e-8))\n else:\n # Apply temperature scaling to logits before computing probabilities\n student_logits = student_logits / temperature\n teacher_logits = teacher_logits / temperature\n # Compute log probabilities for student and probabilities for teacher\n student_log_probs = F.log_softmax(student_logits, dim=-1)\n teacher_log_probs = F.log_softmax(teacher_logits, dim=-1)\n\n if beta == 0:\n jsd = F.kl_div(student_log_probs, teacher_log_probs, reduction=\"none\", log_target=True)\n elif beta == 1:\n jsd = F.kl_div(teacher_log_probs, student_log_probs, reduction=\"none\", log_target=True)\n else:\n # Compute the log of the mixture distribution\n # log(a + b) = log(exp(log(a)) + exp(log(b))) -> for mixture\n beta = torch.tensor(beta, dtype=student_log_probs.dtype, device=student_log_probs.device)\n mixture_log_probs = torch.logsumexp(\n torch.stack([student_log_probs + torch.log1p(-beta), teacher_log_probs + torch.log(beta)]),\n dim=0,\n )\n\n # Compute KL divergences using F.kl_div\n # PyTorch differs from the standard mathematical definition, so the order of the probability distributions is swapped compared to that defined in the paper.\n kl_teacher = F.kl_div(mixture_log_probs, teacher_log_probs, reduction=\"none\", log_target=True)\n kl_student = F.kl_div(mixture_log_probs, student_log_probs, reduction=\"none\", log_target=True)\n\n # Compute the Generalized Jensen-Shannon Divergence\n jsd = beta * kl_teacher + (1 - beta) * kl_student\n\n # Masking\n if labels is not None:\n mask = labels != -100\n jsd = jsd[mask]\n\n # Apply reduction\n if reduction == \"batchmean\":\n return jsd.sum() / mask.sum() if labels is not None else jsd.sum() / jsd.size(0)\n elif reduction == \"sum\":\n return jsd.sum()\n elif reduction == \"mean\":\n return jsd.mean()\n else:\n return jsd\n\n def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):\n if self.use_uld_loss and self.teacher_tokenizer is not None:\n if \"original_prompt_text\" in inputs and \"original_completion_text\" in inputs:\n prompt_texts = inputs[\"original_prompt_text\"]\n completion_texts = inputs[\"original_completion_text\"]\n full_texts = [p + c for p, c in zip(prompt_texts, completion_texts, strict=True)]\n else:\n # Fallback: decode student input_ids (current approach)\n # WARNING: This may not work perfectly for cross-tokenizer distillation\n full_sequences = inputs[\"input_ids\"]\n full_texts = self.processing_class.batch_decode(full_sequences, skip_special_tokens=False)\n\n # Try to split prompt/completion using original prompt length\n prompt_lengths = inputs[\"prompts\"].shape[1]\n prompt_texts = self.processing_class.batch_decode(inputs[\"prompts\"], skip_special_tokens=False)\n completion_texts = [\n full.replace(prompt, \"\", 1) for full, prompt in zip(full_texts, prompt_texts, strict=True)\n ]\n\n (\n teacher_input_ids,\n teacher_labels,\n teacher_attention_mask,\n teacher_prompt_length,\n ) = build_teacher_inputs_from_texts(\n self.teacher_tokenizer,\n prompt_texts,\n completion_texts,\n )\n\n teacher_input_ids = teacher_input_ids.to(self.accelerator.device)\n teacher_labels = teacher_labels.to(self.accelerator.device)\n teacher_attention_mask = teacher_attention_mask.to(self.accelerator.device)\n\n outputs_student = model(\n input_ids=inputs[\"input_ids\"],\n attention_mask=inputs[\"attention_mask\"],\n use_cache=False,\n )\n\n self.teacher_model.eval()\n with torch.no_grad():\n outputs_teacher = self.teacher_model(\n input_ids=teacher_input_ids,\n attention_mask=teacher_attention_mask,\n )\n\n # These are not used for ULD loss but are needed if JSD loss were to be used in this branch\n student_prompt_length = inputs[\"prompts\"].shape[1]\n shifted_student_logits = outputs_student.logits[:, student_prompt_length - 1 : -1, :]\n shifted_teacher_logits = outputs_teacher.logits[:, teacher_prompt_length - 1 : -1, :]\n shifted_labels = inputs[\"labels\"][:, student_prompt_length:]\n else:\n if self.use_liger_gkd_loss:\n # Forward only through the base models (avoid lm_head to save memory)\n unwrapped_student = self.accelerator.unwrap_model(model)\n if hasattr(unwrapped_student, \"get_decoder\") and unwrapped_student.get_decoder() is not None:\n base_student = unwrapped_student.get_decoder()\n else:\n base_student = getattr(\n unwrapped_student, getattr(unwrapped_student, \"base_model_prefix\", \"model\"), unwrapped_student\n )\n\n student_outputs = base_student(\n input_ids=inputs[\"input_ids\"],\n attention_mask=inputs[\"attention_mask\"],\n use_cache=False,\n )\n\n self.teacher_model.eval()\n unwrapped_teacher = self.accelerator.unwrap_model(self.teacher_model)\n if hasattr(unwrapped_teacher, \"get_decoder\") and unwrapped_teacher.get_decoder() is not None:\n base_teacher = unwrapped_teacher.get_decoder()\n else:\n base_teacher = getattr(\n unwrapped_teacher, getattr(unwrapped_teacher, \"base_model_prefix\", \"model\"), unwrapped_teacher\n )\n with torch.no_grad():\n teacher_outputs = base_teacher(\n input_ids=inputs[\"input_ids\"],\n attention_mask=inputs[\"attention_mask\"],\n use_cache=False,\n )\n\n student_hidden = student_outputs.last_hidden_state[:, :-1]\n teacher_hidden = teacher_outputs.last_hidden_state[:, :-1]\n\n del student_outputs, teacher_outputs\n\n student_hidden = student_hidden.reshape(-1, student_hidden.shape[-1])\n teacher_hidden = teacher_hidden.reshape(-1, teacher_hidden.shape[-1])\n\n labels_mask = inputs[\"labels\"] != -100\n masked_input_ids = torch.where(\n labels_mask, inputs[\"input_ids\"], torch.full_like(inputs[\"input_ids\"], -100)\n )\n true_labels = masked_input_ids[:, 1:].contiguous().reshape(-1)\n\n student_head = unwrapped_student.get_output_embeddings()\n teacher_head = unwrapped_teacher.get_output_embeddings()\n\n loss = self.liger_jsd_loss(\n student_input=student_hidden,\n student_weight=student_head.weight,\n teacher_input=teacher_hidden,\n teacher_weight=teacher_head.weight,\n true_labels=true_labels,\n student_bias=getattr(student_head, \"bias\", None),\n teacher_bias=getattr(teacher_head, \"bias\", None),\n )\n\n del student_hidden, teacher_hidden, true_labels\n else:\n outputs_student = model(\n input_ids=inputs[\"input_ids\"],\n attention_mask=inputs[\"attention_mask\"],\n )\n\n self.teacher_model.eval()\n with torch.no_grad():\n outputs_teacher = self.teacher_model(\n input_ids=inputs[\"input_ids\"],\n attention_mask=inputs[\"attention_mask\"],\n )\n\n prompt_lengths = inputs[\"prompts\"].shape[1]\n shifted_student_logits = outputs_student.logits[:, prompt_lengths - 1 : -1, :]\n shifted_teacher_logits = outputs_teacher.logits[:, prompt_lengths - 1 : -1, :]\n shifted_labels = inputs[\"labels\"][:, prompt_lengths:]\n loss = self.generalized_jsd_loss(\n student_logits=shifted_student_logits,\n teacher_logits=shifted_teacher_logits,\n labels=shifted_labels,\n beta=self.beta,\n temperature=self.temperature,\n )\n\n if self.use_uld_loss:\n student_input_ids = inputs[\"input_ids\"]\n\n teacher_labels_for_loss = teacher_labels if \"teacher_labels\" in locals() else inputs[\"labels\"]\n teacher_input_ids_for_loss = teacher_input_ids if \"teacher_input_ids\" in locals() else inputs[\"input_ids\"]\n\n student_labels = inputs[\"labels\"].clone()\n if hasattr(self.processing_class, \"pad_token_id\") and self.processing_class.pad_token_id is not None:\n student_labels[student_labels == self.processing_class.pad_token_id] = -100\n\n if (\n hasattr(self, \"teacher_tokenizer\")\n and hasattr(self.teacher_tokenizer, \"pad_token_id\")\n and self.teacher_tokenizer.pad_token_id is not None\n ):\n teacher_labels[teacher_labels == self.teacher_tokenizer.pad_token_id] = -100\n\n loss = self.uld_loss_fn(\n student_logits=outputs_student.logits,\n teacher_logits=outputs_teacher.logits,\n student_labels=student_labels,\n teacher_labels=teacher_labels_for_loss,\n student_input_ids=student_input_ids,\n teacher_input_ids=teacher_input_ids_for_loss,\n )\n\n if hasattr(self.uld_loss_fn, \"last_matched_loss\") and hasattr(self.uld_loss_fn, \"last_unmatched_loss\"):\n ga = max(1, int(self.args.gradient_accumulation_steps))\n step_eq = 1.0 / ga\n matched_val = (\n self.uld_loss_fn.last_matched_loss.item()\n if self.uld_loss_fn.last_matched_loss is not None\n else 0.0\n )\n unmatched_val = (\n self.uld_loss_fn.last_unmatched_loss.item()\n if self.uld_loss_fn.last_unmatched_loss is not None\n else 0.0\n )\n\n self._matched_sum += matched_val\n self._unmatched_sum += unmatched_val\n self._matched_step_eq += step_eq\n self._unmatched_step_eq += step_eq\n\n empty_cache()\n\n return (loss, outputs_student) if return_outputs else loss\n\n def generate_on_policy_outputs(self, model, inputs, generation_config, pad_token_id=None):\n # Generate output with respect to the prompt only\n if self.use_transformers_paged:\n previous_attn = self.model.config._attn_implementation\n if is_flash_attn_2_available():\n model.config._attn_implementation = \"paged_attention\"\n else:\n model.config._attn_implementation = \"sdpa_paged\"\n prompt_mask = inputs.get(\"prompt_attention_mask\")\n prompts_tensor = inputs[\"prompts\"]\n if prompt_mask is not None:\n prompt_sequences = [\n row[mask.bool()].detach().cpu().tolist()\n for row, mask in zip(prompts_tensor, prompt_mask, strict=True)\n ]\n else:\n prompt_sequences = [row.detach().cpu().tolist() for row in prompts_tensor]\n generated_outputs = model.generate_batch(prompt_sequences, generation_config=generation_config)\n model.config._attn_implementation = previous_attn\n\n completion_ids = [output.generated_tokens for output in generated_outputs.values()]\n generated_tokens = torch.stack([torch.tensor(ids, device=model.device) for ids in completion_ids])\n else:\n generated_outputs = model.generate(\n input_ids=inputs[\"prompts\"],\n attention_mask=inputs.get(\"prompt_attention_mask\", None),\n generation_config=generation_config,\n return_dict_in_generate=True,\n )\n # Get the generated token IDs\n generated_tokens = generated_outputs.sequences\n\n batch_size = generated_tokens.size(0)\n device = generated_tokens.device\n\n prompt_mask = inputs.get(\"prompt_attention_mask\")\n pad_token_id = pad_token_id if pad_token_id is not None else self.processing_class.pad_token_id\n\n if self.use_transformers_paged:\n # generate_batch() returns completion-only tokens, so the entire tensor is completion.\n prompt_lengths = torch.zeros(batch_size, dtype=torch.long, device=device)\n else:\n # model.generate() returns full sequences (prompt + completion), so completions start\n # after the full padded prompt width.\n prompt_lengths = torch.full(\n (batch_size,),\n inputs[\"prompts\"].shape[1],\n dtype=torch.long,\n device=device,\n )\n\n new_input_ids = generated_tokens\n new_attention_mask, new_labels = self._build_sequence_batch(new_input_ids, prompt_lengths, pad_token_id)\n\n prompt_texts = []\n completion_texts = []\n for idx in range(batch_size):\n length = int(prompt_lengths[idx].item())\n prompt_tokens = inputs[\"prompts\"][idx]\n if prompt_mask is not None:\n prompt_tokens = prompt_tokens[prompt_mask[idx].bool()]\n elif pad_token_id is not None:\n prompt_tokens = prompt_tokens[prompt_tokens != pad_token_id]\n prompt_texts.append(\n self.processing_class.decode(\n prompt_tokens.tolist(),\n skip_special_tokens=False,\n clean_up_tokenization_spaces=False,\n )\n )\n completion_tokens = new_input_ids[idx, length:]\n completion_texts.append(\n self.processing_class.decode(\n completion_tokens.tolist(),\n skip_special_tokens=False,\n clean_up_tokenization_spaces=False,\n )\n )\n\n return new_input_ids, new_attention_mask, new_labels, prompt_texts, completion_texts\n\n def _sync_fsdp_params_to_vllm(self, module: nn.Module, prefix: str = \"\", visited=None):\n \"\"\"Memory-efficient post-order traversal of FSDP modules to extract full parameters and sync with student vLLM.\"\"\"\n if visited is None:\n visited = set()\n\n for child_name, child_module in module.named_children():\n child_prefix = f\"{prefix}.{child_name}\" if prefix else child_name\n # recurse into the child\n self._sync_fsdp_params_to_vllm(child_module, prefix=child_prefix, visited=visited)\n\n if isinstance(module, FSDP):\n with FSDP.summon_full_params(module, recurse=False, writeback=False):\n for param_name, param in module.named_parameters():\n full_name = f\"{prefix}.{param_name}\" if prefix else param_name\n for extra in (\"_fsdp_wrapped_module.\", \"_checkpoint_wrapped_module.\"):\n full_name = full_name.replace(extra, \"\")\n\n if full_name in visited:\n continue # skip FSDP subtrees already traversed\n visited.add(full_name)\n\n if self.vllm_mode == \"server\" and self.accelerator.is_main_process:\n self.vllm_client.update_named_param(full_name, param.data)\n elif self.vllm_mode == \"colocate\":\n llm_model = self.vllm_engine.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(full_name, param.data)])\n\n def _move_model_to_vllm(self):\n \"\"\"Synchronize student model weights to vLLM engine.\"\"\"\n # For DeepSpeed ZeRO-3 and FSDP, we need to gather all parameters before operations\n deepspeed_plugin = self.accelerator.state.deepspeed_plugin\n zero_stage_3 = deepspeed_plugin is not None and deepspeed_plugin.zero_stage == 3\n if zero_stage_3:\n import deepspeed\n\n gather_if_zero3 = deepspeed.zero.GatheredParameters\n else:\n gather_if_zero3 = nullcontext\n\n if self.vllm_mode == \"colocate\" and self.vllm_enable_sleep_mode:\n empty_cache()\n self.vllm_engine.wake_up(tags=[\"weights\"])\n # Work around for https://github.com/vllm-project/vllm/issues/29341\n self.vllm_engine.collective_rpc(\"reload_weights\")\n\n if is_peft_model(self.model):\n # With PEFT and FSDP/DeepSpeed ZeRO Stage 3, we must gather the full model at once before merging, as\n # merging adapters in a sharded manner is not supported.\n with gather_if_zero3(list(self.model.parameters())):\n self.model.merge_adapter()\n\n # Update vLLM weights while parameters are gathered\n if self.is_fsdp_enabled: # note if using FSDP, gather_if_zero3 is nullcontext\n # Update vLLM weights while parameters are gathered\n # For PEFT with FSDP we need to use the memory efficient post-order traversal\n self._sync_fsdp_params_to_vllm(self.model)\n else:\n # DeepSpeed ZeRO-3 with PEFT\n for name, param in self.model.named_parameters():\n # When using PEFT, we need to recover the original parameter name and discard some parameters\n name = name.removeprefix(\"base_model.model.\").replace(\".base_layer\", \"\")\n if self.model.prefix in name:\n continue\n # When module to save, remove its prefix and discard the original module\n if \"original_module\" in name:\n continue\n name = name.replace(\"modules_to_save.default.\", \"\")\n\n if self.vllm_mode == \"server\" and self.accelerator.is_main_process:\n self.vllm_client.update_named_param(name, param.data)\n elif self.vllm_mode == \"colocate\":\n llm_model = self.vllm_engine.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(name, param.data)])\n # Unmerge adapters while parameters are still gathered\n self.model.unmerge_adapter()\n # Parameters will automatically be repartitioned when exiting the context\n else:\n # For non-PEFT models, simply gather (if needed) and update each parameter individually.\n if self.is_fsdp_enabled:\n # use memory-efficient post-order traversal for FSDP\n self._sync_fsdp_params_to_vllm(self.model)\n else:\n # For DeepSpeed ZeRO-3, gather each parameter individually like GRPO trainer\n for name, param in self.model.named_parameters():\n with gather_if_zero3([param]):\n if self.vllm_mode == \"server\" and self.accelerator.is_main_process:\n self.vllm_client.update_named_param(name, param.data)\n elif self.vllm_mode == \"colocate\":\n llm_model = self.vllm_engine.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(name, param.data)])\n\n # Reset cache on vLLM\n if self.vllm_mode == \"server\" and self.accelerator.is_main_process:\n self.vllm_client.reset_prefix_cache()\n elif self.vllm_mode == \"colocate\":\n self.vllm_engine.reset_prefix_cache()\n\n def _wake_vllm_if_needed(self):\n if self.vllm_mode == \"colocate\" and self.vllm_enable_sleep_mode:\n empty_cache()\n self.vllm_engine.wake_up(tags=[\"kv_cache\"])\n\n def _get_liger_zero3_lm_head_gather_ctx(self, model: nn.Module):\n if not self.use_liger_gkd_loss:\n return nullcontext()\n\n deepspeed_plugin = self.accelerator.state.deepspeed_plugin\n if deepspeed_plugin is None or deepspeed_plugin.zero_stage != 3:\n return nullcontext()\n\n import deepspeed\n\n unwrapped_student = self.accelerator.unwrap_model(model)\n unwrapped_teacher = self.accelerator.unwrap_model(self.teacher_model)\n student_head = unwrapped_student.get_output_embeddings()\n teacher_head = unwrapped_teacher.get_output_embeddings()\n params = [student_head.weight, teacher_head.weight]\n if student_head.bias is not None:\n params.append(student_head.bias)\n if teacher_head.bias is not None:\n params.append(teacher_head.bias)\n return deepspeed.zero.GatheredParameters(params, modifier_rank=None)\n\n @profiling_decorator\n def training_step(\n self, model: nn.Module, inputs: dict[str, torch.Tensor | Any], num_items_in_batch: int | None = None\n ) -> torch.Tensor:\n \"\"\"\n Perform a training step for the General Online Logit Distillation (GOLD) model.\n\n This method implements the on-policy learning approach described in the GOLD blog post. With probability\n `self.lmbda`, it generates new responses using the student model, which are then used for training instead of\n the offline original inputs.\n \"\"\"\n buffer_steps = self.args.gradient_accumulation_steps\n\n # Keep lm_head gathered across forward+backward for Liger + ZeRO-3.\n with self._get_liger_zero3_lm_head_gather_ctx(model):\n loss = super().training_step(model, inputs, num_items_in_batch)\n\n slice_idx = (self._step - 1) % buffer_steps\n\n on_policy = False\n if self._buffered_on_policy is not None and slice_idx < len(self._buffered_on_policy):\n on_policy = self._buffered_on_policy[slice_idx]\n\n if on_policy and self._buffered_text_logs is not None and self._buffered_text_logs[slice_idx] is not None:\n prompt_texts, completion_texts = self._buffered_text_logs[slice_idx]\n self._textual_logs[\"prompt\"].extend(gather_object(prompt_texts))\n self._textual_logs[\"completion\"].extend(gather_object(completion_texts))\n\n loss_scalar = float(loss.detach())\n step_equiv = 1.0 / self.args.gradient_accumulation_steps\n\n if on_policy:\n self._on_policy_loss_total += loss_scalar\n self._on_policy_step_equiv += step_equiv\n else:\n self._off_policy_loss_total += loss_scalar\n self._off_policy_step_equiv += step_equiv\n return loss\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n mode = \"train\" if self.model.training else \"eval\"\n metrics = {key: sum(val) / len(val) for key, val in self._metrics[mode].items()} # average the metrics\n\n if mode == \"train\":\n device = self.accelerator.device if hasattr(self.accelerator, \"device\") else torch.device(\"cpu\")\n vec = torch.tensor(\n [\n self._on_policy_loss_total,\n self._off_policy_loss_total,\n self._on_policy_step_equiv,\n self._off_policy_step_equiv,\n self._matched_sum,\n self._unmatched_sum,\n self._matched_step_eq,\n self._unmatched_step_eq,\n ],\n dtype=torch.float64,\n device=device,\n )\n\n if (\n getattr(self.accelerator, \"distributed_type\", DistributedType.NO) != DistributedType.NO\n and dist.is_available()\n and dist.is_initialized()\n ):\n dist.all_reduce(vec, op=dist.ReduceOp.SUM)\n\n (\n on_sum,\n off_sum,\n on_eq,\n off_eq,\n matched_sum,\n unmatched_sum,\n matched_eq,\n unmatched_eq,\n ) = vec.tolist()\n\n if on_eq > 0:\n logs[\"on_policy_loss\"] = round(on_sum / on_eq, 4)\n if off_eq > 0:\n logs[\"off_policy_loss\"] = round(off_sum / off_eq, 4)\n\n if matched_eq > 0:\n logs[\"matched_loss\"] = round(matched_sum / matched_eq, 4)\n if unmatched_eq > 0:\n logs[\"unmatched_loss\"] = round(unmatched_sum / unmatched_eq, 4)\n\n self._on_policy_loss_total = self._off_policy_loss_total = 0.0\n self._on_policy_step_equiv = self._off_policy_step_equiv = 0.0\n self._matched_sum = self._unmatched_sum = 0.0\n self._matched_step_eq = self._unmatched_step_eq = 0.0\n\n # This method can be called both in training and evaluation. When called in evaluation, the keys in `logs`\n # start with \"eval_\". We need to add the prefix \"eval_\" to the keys in `metrics` to match the format.\n if mode == \"eval\":\n metrics = {f\"eval_{key}\": val for key, val in metrics.items()}\n\n logs = {**logs, **metrics}\n super().log(logs, start_time)\n self._metrics[mode].clear()\n\n if (\n self.accelerator.is_main_process\n and self.log_completions\n and ((self.state.global_step % self.log_completion_steps) == 0)\n ):\n if is_rich_available():\n print_prompt_completions_sample_uld(\n self._textual_logs[\"prompt\"],\n self._textual_logs[\"completion\"],\n self.state.global_step,\n self.num_completions_to_print,\n )\n\n if self.args.report_to and \"wandb\" in self.args.report_to and wandb.run is not None:\n import pandas as pd\n\n table = {\n \"step\": [str(self.state.global_step)] * len(self._textual_logs[\"prompt\"]),\n \"prompt\": self._textual_logs[\"prompt\"],\n \"completion\": self._textual_logs[\"completion\"],\n }\n df = pd.DataFrame(table)\n if self.wandb_log_unique_prompts:\n df = df.drop_duplicates(subset=[\"prompt\"])\n if self.num_completions_to_print and len(df) > 0:\n df = df.sample(n=self.num_completions_to_print, random_state=42)\n wandb.log({\"completions\": wandb.Table(dataframe=df)})\n" }, { "path": "trl/experimental/grpo_with_replay_buffer/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .grpo_with_replay_buffer_config import GRPOWithReplayBufferConfig\nfrom .grpo_with_replay_buffer_trainer import GRPOWithReplayBufferTrainer, ReplayBuffer\n" }, { "path": "trl/experimental/grpo_with_replay_buffer/grpo_with_replay_buffer_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom ...trainer.grpo_config import GRPOConfig\n\n\n@dataclass\nclass GRPOWithReplayBufferConfig(GRPOConfig):\n \"\"\"\n New Parameters:\n replay_buffer_size (`int`, *optional*, defaults to `0`):\n A cache that stores the rollouts with the highest advantage scores and variance per group. If a new\n group has 0 variance, it is replaced with a group sampled from the replay buffer.\n \"\"\"\n\n replay_buffer_size: int = field(\n default=64,\n metadata={\n \"help\": \"A cache that stores the rollouts with the highest advantage scores and variance per group. If a new group has 0 variance, it is replaced with a group sampled from the replay buffer.\"\n },\n )\n" }, { "path": "trl/experimental/grpo_with_replay_buffer/grpo_with_replay_buffer_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport heapq\nfrom typing import Any\n\nimport torch\nfrom accelerate.utils import gather_object\n\nfrom ...data_utils import apply_chat_template, is_conversational, prepare_multimodal_messages\nfrom ...models.utils import disable_gradient_checkpointing\nfrom ...trainer.grpo_trainer import GRPOTrainer\nfrom ...trainer.utils import nanmax, nanmin, nanstd, pad\nfrom .grpo_with_replay_buffer_config import GRPOWithReplayBufferConfig\n\n\nclass ReplayBuffer:\n \"\"\"\n A simple replay buffer to store and sample previously seen rollouts.\n \"\"\"\n\n def __init__(self, max_size: int):\n self.max_size = max_size\n self.heap = [] # Min-heap of (score, data) tuples\n\n def add(self, scores: list[float], data: list[dict]):\n for score, datum in zip(scores, data, strict=True):\n if len(self.heap) < self.max_size:\n heapq.heappush(self.heap, (score, datum))\n else:\n # Only add if score is better than worst (minimum) item\n if score > self.heap[0][0]:\n heapq.heapreplace(self.heap, (score, datum))\n\n def sample(self, num_samples: int) -> list[dict[str, torch.Tensor]]:\n if not self.heap:\n return None\n\n # Sample by normalized scores\n scores = torch.tensor([item[0] for item in self.heap], dtype=torch.float32)\n probabilities = scores / scores.sum()\n replacement = False\n if num_samples > len(self.heap):\n replacement = True\n chosen_indices = torch.multinomial(probabilities, num_samples, replacement=replacement).tolist()\n return [self.heap[i][1] for i in chosen_indices]\n\n\nclass GRPOWithReplayBufferTrainer(GRPOTrainer):\n def __init__(self, args: GRPOWithReplayBufferConfig | None = None, **kwargs):\n super().__init__(args=args, **kwargs)\n self.replay_buffer = ReplayBuffer(args.replay_buffer_size) if args.replay_buffer_size > 0 else None\n\n def _generate_and_score_completions(\n self, inputs: list[dict[str, torch.Tensor | Any]]\n ) -> dict[str, torch.Tensor | Any]:\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n\n prompts = [x[\"prompt\"] for x in inputs]\n\n if \"images\" in inputs[0]:\n images = [example.get(\"images\") for example in inputs]\n elif \"image\" in inputs[0]:\n images = [[example.get(\"image\")] if example.get(\"image\") is not None else None for example in inputs]\n else:\n images = None\n # Transformers requires at least one image in the batch, otherwise it throws an error\n if images is not None and all(img_list == [] for img_list in images):\n images = None\n\n # If the prompts are conversational and the inputs contain images, we need to convert the prompts from\n # [{\"role\": \"user\", \"content\": \"What color is the sky?\"}] to\n # [{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": }, {\"type\": \"text\", \"text\": \"What color is the sky?\"}]}]\n if images is not None:\n if not is_conversational(inputs[0]):\n raise ValueError(\n \"Multimodal training requires conversational prompts. It looks like the dataset contains \"\n \"non-conversational inputs, likely because a chat template was applied before passing the dataset \"\n \"to the trainer. Please provide the raw conversational prompts and let the trainer apply the chat \"\n \"template internally.\"\n )\n prompts = [\n prepare_multimodal_messages(prompt, image_list)\n for prompt, image_list in zip(prompts, images, strict=True)\n ]\n\n (\n prompt_ids_list,\n completion_ids_list,\n tool_mask_list,\n completions,\n num_items_in_batch,\n sampling_per_token_logps_list,\n extra_fields,\n ) = self._generate(prompts)\n\n # Convert lists of token IDs to padded tensors\n prompt_ids = [torch.tensor(ids) for ids in prompt_ids_list]\n prompt_mask = [torch.ones_like(ids, dtype=torch.long) for ids in prompt_ids]\n prompt_ids = pad(\n prompt_ids,\n padding_value=self.pad_token_id,\n padding_side=\"left\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n prompt_mask = pad(\n prompt_mask,\n padding_value=0,\n padding_side=\"left\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n completion_ids = [torch.tensor(ids) for ids in completion_ids_list]\n completion_mask = [torch.ones_like(ids, dtype=torch.long) for ids in completion_ids]\n completion_ids = pad(\n completion_ids,\n padding_value=self.pad_token_id,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n completion_mask = pad(\n completion_mask,\n padding_value=0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n if sampling_per_token_logps_list is not None:\n sampling_per_token_logps = [torch.tensor(logps) for logps in sampling_per_token_logps_list]\n sampling_per_token_logps = pad(\n sampling_per_token_logps,\n padding_value=0.0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n else:\n sampling_per_token_logps = None\n if self.tools:\n tool_mask = [torch.tensor(mask) for mask in tool_mask_list]\n tool_mask = pad(\n tool_mask,\n padding_value=1,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device) # 0 for tool result tokens, 1 elsewhere\n\n # If mask_truncated_completions is enabled, zero out truncated completions in completion_mask\n if self.mask_truncated_completions:\n eos_and_pad = [self.eos_token_id, self.pad_token_id]\n is_truncated = torch.tensor([ids[-1] not in eos_and_pad for ids in completion_ids_list], device=device)\n completion_mask = completion_mask * (~is_truncated).unsqueeze(1).int()\n\n # Concatenate prompt_mask with completion_mask for logit computation\n prompt_completion_ids = torch.cat([prompt_ids, completion_ids], dim=1) # (B, P+C)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1) # (B, P+C)\n\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n batch_size = self.args.per_device_train_batch_size if mode == \"train\" else self.args.per_device_eval_batch_size\n\n num_images = [len(img_list) for img_list in images] if images is not None else None\n\n # Get forward_kwargs for models with multimodal inputs\n if images is not None:\n prompts_text = [\n apply_chat_template(\n {\"prompt\": prompt}, self.processing_class, tools=self.tools, **self.chat_template_kwargs\n )[\"prompt\"]\n for prompt in prompts\n ]\n prompt_inputs = self.processing_class(images=images, text=prompts_text, padding=True, return_tensors=\"pt\")\n prompt_inputs = super()._prepare_inputs(prompt_inputs)\n forward_kwargs = {k: v for k, v in prompt_inputs.items() if k not in [\"input_ids\", \"attention_mask\"]}\n else:\n forward_kwargs = {}\n\n # If token_type_ids are used, extend them with zeros for the completion part\n if \"token_type_ids\" in forward_kwargs:\n token_type_ids = forward_kwargs[\"token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n # Needed only with pad_to_multiple_of: otherwise prompt_ids and token_type_ids must have equal len\n padding_size = prompt_ids.size(1) - token_type_ids.size(1)\n if padding_size > 0:\n token_type_ids = torch.cat(\n [token_type_ids.new_zeros((token_type_ids.size(0), padding_size)), token_type_ids], dim=1\n )\n forward_kwargs[\"token_type_ids\"] = torch.cat(\n [token_type_ids, token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n # If mm_token_type_ids are used, extend them with zeros for the completion part\n if \"mm_token_type_ids\" in forward_kwargs:\n mm_token_type_ids = forward_kwargs[\"mm_token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n # Needed only with pad_to_multiple_of: otherwise prompt_ids and mm_token_type_ids must have equal len\n padding_size = prompt_ids.size(1) - mm_token_type_ids.size(1)\n if padding_size > 0:\n mm_token_type_ids = torch.cat(\n [mm_token_type_ids.new_zeros((mm_token_type_ids.size(0), padding_size)), mm_token_type_ids],\n dim=1,\n )\n forward_kwargs[\"mm_token_type_ids\"] = torch.cat(\n [mm_token_type_ids, mm_token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n\n # When gradient checkpointing is enabled with use_reentrant=True (non default), calling the model inside a\n # torch.no_grad() block triggers a harmless PyTorch warning (\"None of the inputs have requires_grad=True\").\n # Temporarily disable checkpointing to avoid this warning during inference.\n with torch.no_grad(), disable_gradient_checkpointing(self.model, self.args.gradient_checkpointing_kwargs):\n # If the generation and optimization steps are misalignedโ€”i.e., if generation does not occur at the end of\n # a full optimizer step (when gradient_accumulation_steps is not a multiple of generate_every)โ€”then the\n # samples may come from an earlier version of the model. In that case, we need to track old_per_token_logps\n # for importance sampling. If the steps are aligned, importance sampling isn't necessary and we set\n # old_per_token_logps to None.\n # When using vLLM, we always compute old_per_token_logps for importance sampling, it was shown that the\n # distribution mismatch between vLLM and the training model can be large and harm the training.\n generate_every = self.args.steps_per_generation * self.num_iterations # generation frequency\n if self.args.gradient_accumulation_steps % generate_every != 0 or (\n self.use_vllm and self.vllm_importance_sampling_correction\n ):\n old_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n old_per_token_logps = None\n\n # Compute the importance sampling ratio when using vLLM, to correct for potential distribution mismatch\n if self.use_vllm and self.vllm_importance_sampling_correction:\n importance_sampling_ratio = torch.exp(old_per_token_logps - sampling_per_token_logps)\n importance_sampling_ratio = torch.clamp(\n importance_sampling_ratio, max=self.vllm_importance_sampling_cap\n )\n\n # Compute the per-token log probabilities for the reference model\n if self.beta != 0.0:\n if self.ref_model is not None:\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.ref_model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n with self.accelerator.unwrap_model(self.model).disable_adapter():\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n ref_per_token_logps = None\n\n # Decode\n prompts_text = self.processing_class.batch_decode(prompt_ids, skip_special_tokens=True)\n completions_text = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n\n # Merge extra_fields from rollout_func into inputs for reward functions\n if extra_fields:\n for i, inp in enumerate(inputs):\n for key, values in extra_fields.items():\n if isinstance(values, list) and i < len(values):\n inp[key] = values[i]\n elif not isinstance(values, list):\n inp[key] = values\n\n # Calculate rewards for each reward function. rewards_per_func aggregates rewards across all processes. This is\n # important because rewards will be normalized per group, and completions are distributed. We will later slice\n # rewards_per_func to extract each process's subset.\n rewards_per_func = self._calculate_rewards(inputs, prompts, completions, completion_ids_list)\n\n # Apply weights to each reward function's output and sum\n rewards = (rewards_per_func * self.reward_weights.to(device).unsqueeze(0)).nansum(dim=1)\n\n # Compute grouped-wise rewards\n mean_grouped_rewards = rewards.view(-1, self.num_generations).mean(dim=1)\n\n # Normalize the rewards to compute the advantages\n mean_grouped_rewards = mean_grouped_rewards.repeat_interleave(self.num_generations, dim=0)\n advantages = rewards - mean_grouped_rewards\n\n grouped_std_rewards = rewards.view(-1, self.num_generations).std(dim=1)\n grouped_std_rewards = grouped_std_rewards.repeat_interleave(self.num_generations, dim=0)\n\n if self.scale_rewards in [\"group\", \"none\"]:\n # If self.scale_rewards = \"none\", we'll still log group level std\n std_rewards = grouped_std_rewards.clone()\n elif self.scale_rewards == \"batch\":\n # Compute global std\n std_rewards = rewards.std().expand_as(rewards)\n else:\n raise ValueError(\n f\"Invalid value for scale_rewards: {self.scale_rewards}. Must be one of 'batch', 'group', or 'none'.\"\n )\n\n is_std_zero = torch.isclose(std_rewards, torch.zeros_like(std_rewards))\n if self.scale_rewards != \"none\":\n advantages = advantages / (std_rewards + 1e-4)\n\n # Slice to keep only the local part of the data\n process_slice = slice(\n self.accelerator.process_index * len(prompts),\n (self.accelerator.process_index + 1) * len(prompts),\n )\n all_process_advantages = advantages.clone() # keep the aggregated advantages for logging\n advantages = advantages[process_slice]\n grouped_std_rewards = grouped_std_rewards[process_slice]\n\n # Calculate mean reward per function, but only for samples where the function was applied (non-NaN values)\n for i, reward_func_name in enumerate(self.reward_func_names):\n mean_rewards = torch.nanmean(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/mean\"].append(mean_rewards)\n std_func_rewards = nanstd(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/std\"].append(std_func_rewards)\n self._metrics[mode][\"reward\"].append(mean_grouped_rewards.mean().item())\n self._metrics[mode][\"reward_std\"].append(std_rewards.mean().item())\n self._metrics[mode][\"frac_reward_zero_std\"].append(is_std_zero.float().mean().item())\n\n # Log prompt and completion texts\n self._logs[\"prompt\"].extend(gather_object(prompts_text))\n self._logs[\"completion\"].extend(gather_object(completions_text))\n for i, name in enumerate(self.reward_func_names):\n self._logs[\"rewards\"][name].extend(rewards_per_func[:, i].tolist())\n self._logs[\"advantages\"].extend(all_process_advantages.tolist())\n\n if images is not None:\n self._logs[\"images\"].extend(gather_object(images))\n\n if self.use_vllm and self.vllm_importance_sampling_correction:\n delta = torch.abs(old_per_token_logps - sampling_per_token_logps)\n mask = completion_mask.bool() if not self.tools else (completion_mask * tool_mask).bool()\n delta = delta[mask]\n mean_delta = torch.mean(delta) if delta.numel() > 0 else torch.tensor(0.0, device=device)\n max_delta = torch.max(delta) if delta.numel() > 0 else torch.tensor(0.0, device=device)\n self._metrics[mode][\"sampling/sampling_logp_difference/mean\"].append(\n self.accelerator.gather(mean_delta).mean().item()\n )\n self._metrics[mode][\"sampling/sampling_logp_difference/max\"].append(\n self.accelerator.gather(max_delta).max().item()\n )\n\n flat_is_ratio = importance_sampling_ratio[mask]\n min_importance_sampling_ratio = (\n torch.min(flat_is_ratio) if flat_is_ratio.numel() > 0 else torch.tensor(0.0, device=device)\n )\n mean_importance_sampling_ratio = (\n torch.mean(flat_is_ratio) if flat_is_ratio.numel() > 0 else torch.tensor(0.0, device=device)\n )\n max_importance_sampling_ratio = (\n torch.max(flat_is_ratio) if flat_is_ratio.numel() > 0 else torch.tensor(0.0, device=device)\n )\n self._metrics[mode][\"sampling/importance_sampling_ratio/min\"].append(\n nanmin(self.accelerator.gather(min_importance_sampling_ratio)).item()\n )\n self._metrics[mode][\"sampling/importance_sampling_ratio/mean\"].append(\n self.accelerator.gather(mean_importance_sampling_ratio).nanmean().item()\n )\n self._metrics[mode][\"sampling/importance_sampling_ratio/max\"].append(\n nanmax(self.accelerator.gather(max_importance_sampling_ratio)).item()\n )\n outputs_after_sampling_buffer = self.update_with_replay_buffer(\n advantages,\n grouped_std_rewards,\n prompt_ids,\n prompt_mask,\n completion_ids,\n completion_mask,\n forward_kwargs,\n num_items_in_batch,\n old_per_token_logps,\n ref_per_token_logps,\n importance_sampling_ratio if self.use_vllm and self.vllm_importance_sampling_correction else None,\n )\n if outputs_after_sampling_buffer is not None:\n return outputs_after_sampling_buffer\n else:\n output = {\n \"prompt_ids\": prompt_ids,\n \"prompt_mask\": prompt_mask,\n \"completion_ids\": completion_ids,\n \"completion_mask\": completion_mask,\n \"advantages\": advantages,\n \"num_items_in_batch\": num_items_in_batch,\n }\n if old_per_token_logps is not None:\n output[\"old_per_token_logps\"] = old_per_token_logps\n if self.use_vllm and self.vllm_importance_sampling_correction:\n output[\"importance_sampling_ratio\"] = importance_sampling_ratio\n if ref_per_token_logps is not None:\n output[\"ref_per_token_logps\"] = ref_per_token_logps\n if \"pixel_values\" in forward_kwargs:\n output[\"pixel_values\"] = forward_kwargs[\"pixel_values\"]\n if \"image_grid_thw\" in forward_kwargs:\n output[\"image_grid_thw\"] = forward_kwargs[\"image_grid_thw\"]\n if \"pixel_attention_mask\" in forward_kwargs:\n output[\"pixel_attention_mask\"] = forward_kwargs[\"pixel_attention_mask\"]\n if \"image_sizes\" in forward_kwargs:\n output[\"image_sizes\"] = forward_kwargs[\"image_sizes\"]\n if \"token_type_ids\" in forward_kwargs:\n output[\"token_type_ids\"] = forward_kwargs[\"token_type_ids\"]\n if images is not None:\n output[\"num_images\"] = num_images\n if self.tools:\n output[\"tool_mask\"] = tool_mask\n return output\n\n def slice_group_data(\n self, data: torch.Tensor, mask: torch.Tensor, group_idx: int\n ) -> tuple[torch.Tensor, torch.Tensor]:\n \"\"\"\n Slices the input data and mask tensors for a specific group index. Also trims the sequence length to the\n maximum length in the group based on the mask.\n\n Args:\n data: Tensor of shape (num_groups * num_generations, seq_length)\n mask: Tensor of shape (num_groups * num_generations, seq_length)\n group_idx: Index of the group to slice\n Returns:\n Tuple of (sliced_data, sliced_mask) for the specified group, with sequence length trimmed to the maximum\n length in the group.\n \"\"\"\n start_idx = group_idx * self.num_generations\n end_idx = (group_idx + 1) * self.num_generations\n group_data = data[start_idx:end_idx]\n group_mask = mask[start_idx:end_idx]\n group_max_len = group_mask.sum(dim=1).max().item()\n return group_data[:, :group_max_len], group_mask[:, :group_max_len]\n\n def update_replay_buffer(\n self,\n groups_with_variance: torch.Tensor,\n group_advantages: torch.Tensor,\n group_std_rewards: torch.Tensor,\n prompt_ids: torch.Tensor,\n prompt_mask: torch.Tensor,\n completion_ids: torch.Tensor,\n completion_mask: torch.Tensor,\n forward_kwargs: dict,\n optional_vision_fields: list[str] = None,\n old_per_token_logps: torch.Tensor | None = None,\n ref_per_token_logps: torch.Tensor | None = None,\n importance_sampling_ratio: float | None = None,\n ) -> None:\n \"\"\"\n Update the replay buffer with groups that have reward variance (std > 0).\n\n Args:\n groups_with_variance: Boolean tensor indicating which groups have reward variance\n group_advantages: Tensor of shape (num_groups, num_generations) containing advantage values\n std_rewards: Tensor of shape (num_groups, num_generations) containing std of rewards per group\n prompt_ids: Tensor containing prompt token IDs\n prompt_mask: Tensor containing prompt attention masks\n completion_ids: Tensor containing completion token IDs\n completion_mask: Tensor containing completion attention masks\n forward_kwargs: Dictionary containing additional prompt inputs (vision data, etc.)\n optional_vision_fields: List of optional vision-related fields to include if present in forward_kwargs\n old_per_token_logps: Optional tensor of old per-token log probabilities\n ref_per_token_logps: Optional tensor of reference per-token log probabilities\n importance_sampling_ratio: Optional importance sampling correction ratio\n \"\"\"\n # Prepare buffered outputs for groups with variance\n buffered_outputs = []\n for _, group_idx in enumerate(groups_with_variance.nonzero(as_tuple=True)[0].unique().tolist()):\n group_prompt_ids, group_prompt_mask = self.slice_group_data(prompt_ids, prompt_mask, group_idx)\n group_completion_ids, group_completion_mask = self.slice_group_data(\n completion_ids, completion_mask, group_idx\n )\n\n # Store unpadded data in the buffer\n buffered_output = {\n \"prompt_ids\": group_prompt_ids,\n \"completion_ids\": group_completion_ids,\n \"advantages\": group_advantages[group_idx].tolist(),\n \"prompt_mask\": group_prompt_mask,\n \"completion_mask\": group_completion_mask,\n }\n\n # Add optional fields if they exist\n optional_fields = {\n \"old_per_token_logps\": old_per_token_logps if old_per_token_logps is not None else None,\n \"ref_per_token_logps\": ref_per_token_logps if ref_per_token_logps is not None else None,\n }\n\n for field_name, field_data in optional_fields.items():\n if field_data is not None:\n buffered_output[field_name] = self.slice_group_data(field_data, completion_mask, group_idx)[0]\n\n # Add importance sampling if needed\n if self.use_vllm and self.vllm_importance_sampling_correction:\n buffered_output[\"importance_sampling_ratio\"] = importance_sampling_ratio\n\n if optional_vision_fields:\n # Add vision-related fields if they exist\n for field_name in optional_vision_fields:\n if field_name in forward_kwargs:\n buffered_output[field_name] = self.slice_group_data(\n forward_kwargs[field_name], prompt_mask, group_idx\n )[0]\n\n buffered_outputs.append(buffered_output)\n\n if groups_with_variance.any():\n # Calculate replay buffer scores for groups with variance\n replay_buffer_scores = (group_advantages.abs() * group_std_rewards).sum(dim=-1)[groups_with_variance]\n # Add all groups to replay buffer at once (batch operation)\n self.replay_buffer.add(replay_buffer_scores.tolist(), buffered_outputs)\n\n def sample_from_replay_buffer(\n self, num_samples: int, optional_vision_fields: list[str] = None, optional_tensor_fields: list[str] = None\n ) -> list[dict]:\n \"\"\"\n Sample groups from the replay buffer.\n\n Args:\n num_samples: Number of samples to draw from the replay buffer\n optional_vision_fields: List of optional vision-related fields to include if present in sampled data\n optional_tensor_fields: List of optional tensor fields to include if present in sampled data\n Returns:\n List of sampled data dictionaries from the replay buffer\n \"\"\"\n sampled = self.replay_buffer.sample(num_samples=num_samples)\n\n # Extract and concatenate sampled data\n sampled_data = {\n \"prompt_ids\": [],\n \"prompt_mask\": [],\n \"completion_ids\": [],\n \"completion_mask\": [],\n \"advantages\": [],\n }\n\n all_optional_fields = (optional_tensor_fields or []) + (optional_vision_fields or [])\n # Initialize containers for optional fields if they exist in sampled data\n for field in all_optional_fields:\n if sampled and field in sampled[0]:\n sampled_data[field] = []\n\n # Extract data from each sampled item\n for item in sampled:\n # Handle core fields\n for key in [\"prompt_ids\", \"prompt_mask\", \"completion_ids\", \"completion_mask\"]:\n sampled_data[key].append(item[key])\n\n # Handle advantages (list, not tensor)\n sampled_data[\"advantages\"].append(item[\"advantages\"])\n\n # Handle optional fields\n for field in all_optional_fields:\n if field in item:\n sampled_data[field].append(item[field])\n\n return sampled_data\n\n def update_with_replay_buffer(\n self,\n group_advantages: torch.Tensor,\n group_std_rewards: torch.Tensor,\n prompt_ids: torch.Tensor,\n prompt_mask: torch.Tensor,\n completion_ids: torch.Tensor,\n completion_mask: torch.Tensor,\n forward_kwargs: dict,\n num_items_in_batch: int,\n old_per_token_logps: torch.Tensor | None = None,\n ref_per_token_logps: torch.Tensor | None = None,\n importance_sampling_ratio: float | None = None,\n ) -> None:\n \"\"\"\n Update current batch data with samples from replay buffer.\n\n Groups with reward variance (std > 0) are added to the replay buffer and then replaced with samples from the\n buffer to improve training stability.\n\n Args:\n group_advantages: Tensor of shape (num_groups, num_generations) containing advantage values\n std_rewards: Tensor of shape (num_groups, num_generations) containing std of rewards per group\n prompt_ids: Tensor containing prompt token IDs\n prompt_mask: Tensor containing prompt attention masks\n completion_ids: Tensor containing completion token IDs\n completion_mask: Tensor containing completion attention masks\n forward_kwargs: Dictionary containing additional prompt inputs (vision data, etc.)\n num_items_in_batch: Number of items in the current batch\n old_per_token_logps: Optional tensor of old per-token log probabilities\n ref_per_token_logps: Optional tensor of reference per-token log probabilities\n importance_sampling_ratio: Optional importance sampling correction ratio\n \"\"\"\n if self.replay_buffer.max_size <= 0:\n return\n\n # Groups to consider for adding to the replay buffer\n groups_with_variance = group_std_rewards.max(dim=0).values > 0\n # Groups to replace from the replay buffer\n groups_without_variance = ~groups_with_variance\n\n # Track which optional fields are present in sampled data\n optional_tensor_fields = [\"old_per_token_logps\", \"ref_per_token_logps\"]\n vision_fields = [\"pixel_values\", \"image_grid_thw\", \"pixel_attention_mask\", \"image_sizes\"]\n\n self.update_replay_buffer(\n groups_with_variance,\n group_advantages,\n group_std_rewards,\n prompt_ids,\n prompt_mask,\n completion_ids,\n completion_mask,\n forward_kwargs,\n vision_fields,\n old_per_token_logps,\n ref_per_token_logps,\n importance_sampling_ratio,\n )\n\n # Sample from replay buffer to replace groups with variance\n num_groups_to_replace = groups_without_variance.sum().item()\n if not num_groups_to_replace:\n return\n\n sampled_data = self.sample_from_replay_buffer(\n num_samples=num_groups_to_replace,\n optional_vision_fields=vision_fields,\n optional_tensor_fields=optional_tensor_fields,\n )\n\n # Pad sampled data if they are shorter than the current batch sequences\n # Or pad the current batch if sampled are longer\n current_batch_prompt_seq_len = prompt_ids.size(1)\n current_batch_completion_seq_len = completion_ids.size(1)\n\n groups_to_replace_idxs = groups_with_variance.logical_not().nonzero(as_tuple=True)[0].unique().tolist()\n\n # Determine target (max) sequence lengths once\n sampled_prompt_lengths = [t.size(1) for t in sampled_data[\"prompt_ids\"]]\n sampled_completion_lengths = [t.size(1) for t in sampled_data[\"completion_ids\"]]\n target_prompt_len = max([current_batch_prompt_seq_len] + sampled_prompt_lengths)\n target_completion_len = max([current_batch_completion_seq_len] + sampled_completion_lengths)\n\n # If any sampled prompt is longer, pad the whole batch prompt tensors once (left padding)\n if target_prompt_len > current_batch_prompt_seq_len:\n prompt_ids = pad(\n list(prompt_ids.unbind(0)),\n padding_value=self.pad_token_id,\n pad_to_multiple_of=target_prompt_len,\n padding_side=\"left\",\n )\n prompt_mask = pad(\n list(prompt_mask.unbind(0)), padding_value=0, pad_to_multiple_of=target_prompt_len, padding_side=\"left\"\n )\n # If any sampled completion is longer, pad the whole batch completion tensors once (right padding)\n if target_completion_len > current_batch_completion_seq_len:\n completion_ids = pad(\n list(completion_ids.unbind(0)),\n padding_value=self.pad_token_id,\n pad_to_multiple_of=target_completion_len,\n padding_side=\"right\",\n )\n completion_mask = pad(\n list(completion_mask.unbind(0)),\n padding_value=0,\n pad_to_multiple_of=target_completion_len,\n padding_side=\"right\",\n )\n if old_per_token_logps is not None:\n old_per_token_logps = pad(\n list(old_per_token_logps.unbind(0)),\n padding_value=0.0,\n pad_to_multiple_of=target_completion_len,\n padding_side=\"right\",\n )\n if ref_per_token_logps is not None:\n ref_per_token_logps = pad(\n list(ref_per_token_logps.unbind(0)),\n padding_value=0.0,\n pad_to_multiple_of=target_completion_len,\n padding_side=\"right\",\n )\n\n # Replace per-group data, padding only sampled groups that are shorter than the target\n for i, group_idx in enumerate(groups_to_replace_idxs):\n start_idx = group_idx * self.num_generations\n end_idx = (group_idx + 1) * self.num_generations\n idx_range = slice(start_idx, end_idx)\n\n # Pad sampled prompt to target length if needed\n if sampled_data[\"prompt_ids\"][i].size(1) < target_prompt_len:\n sampled_data[\"prompt_ids\"][i] = pad(\n sampled_data[\"prompt_ids\"][i],\n padding_value=self.pad_token_id,\n pad_to_multiple_of=target_prompt_len,\n padding_side=\"left\",\n )\n sampled_data[\"prompt_mask\"][i] = pad(\n sampled_data[\"prompt_mask\"][i],\n padding_value=0,\n pad_to_multiple_of=target_prompt_len,\n padding_side=\"left\",\n )\n\n # Pad sampled completion to target length if needed\n if sampled_data[\"completion_ids\"][i].size(1) < target_completion_len:\n sampled_data[\"completion_ids\"][i] = pad(\n sampled_data[\"completion_ids\"][i],\n padding_value=self.pad_token_id,\n pad_to_multiple_of=target_completion_len,\n padding_side=\"right\",\n )\n sampled_data[\"completion_mask\"][i] = pad(\n sampled_data[\"completion_mask\"][i],\n padding_value=0,\n pad_to_multiple_of=target_completion_len,\n padding_side=\"right\",\n )\n if \"old_per_token_logps\" in sampled_data:\n sampled_data[\"old_per_token_logps\"][i] = pad(\n sampled_data[\"old_per_token_logps\"][i],\n padding_value=0.0,\n pad_to_multiple_of=target_completion_len,\n padding_side=\"right\",\n )\n if \"ref_per_token_logps\" in sampled_data:\n sampled_data[\"ref_per_token_logps\"][i] = pad(\n sampled_data[\"ref_per_token_logps\"][i],\n padding_value=0.0,\n pad_to_multiple_of=target_completion_len,\n padding_side=\"right\",\n )\n\n # Assign (replace) group slice\n prompt_ids[idx_range] = sampled_data[\"prompt_ids\"][i]\n prompt_mask[idx_range] = sampled_data[\"prompt_mask\"][i]\n completion_ids[idx_range] = sampled_data[\"completion_ids\"][i]\n completion_mask[idx_range] = sampled_data[\"completion_mask\"][i]\n group_advantages[group_idx] = sampled_data[\"advantages\"][i]\n\n if \"old_per_token_logps\" in sampled_data:\n old_per_token_logps[idx_range] = sampled_data[\"old_per_token_logps\"][i]\n if \"ref_per_token_logps\" in sampled_data:\n ref_per_token_logps[idx_range] = sampled_data[\"ref_per_token_logps\"][i]\n\n for field in vision_fields:\n if field in sampled_data and field in forward_kwargs:\n forward_kwargs[field][idx_range] = sampled_data[field][i]\n\n # Prepare final outputs after sampling and replacement\n outputs_after_sampling_buffer = {\n \"prompt_ids\": prompt_ids,\n \"prompt_mask\": prompt_mask,\n \"completion_ids\": completion_ids,\n \"completion_mask\": completion_mask,\n \"advantages\": group_advantages,\n }\n\n # Replace optional tensor fields if they exist\n for field in optional_tensor_fields:\n if field in sampled_data:\n outputs_after_sampling_buffer[field] = (\n old_per_token_logps if field == \"old_per_token_logps\" else ref_per_token_logps\n )\n\n # Replace vision fields if they exist\n for field in vision_fields:\n if field in sampled_data and field in forward_kwargs:\n outputs_after_sampling_buffer[field] = forward_kwargs[field]\n\n outputs_after_sampling_buffer[\"num_items_in_batch\"] = num_items_in_batch\n if self.use_vllm and self.vllm_importance_sampling_correction:\n outputs_after_sampling_buffer[\"importance_sampling_ratio\"] = importance_sampling_ratio\n\n return outputs_after_sampling_buffer\n" }, { "path": "trl/experimental/gspo_token/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .grpo_trainer import GRPOTrainer\n" }, { "path": "trl/experimental/gspo_token/grpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport torch\n\nfrom ...trainer.grpo_trainer import GRPOTrainer as _GRPOTrainer\nfrom ...trainer.utils import nanmax, nanmin\n\n\nclass GRPOTrainer(_GRPOTrainer):\n def _compute_loss(self, model, inputs):\n # Compute the per-token log probabilities for the model\n prompt_ids, prompt_mask = inputs[\"prompt_ids\"], inputs[\"prompt_mask\"]\n completion_ids, completion_mask = inputs[\"completion_ids\"], inputs[\"completion_mask\"]\n input_ids = torch.cat([prompt_ids, completion_ids], dim=1)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1)\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n\n # Compute the per_token_logps and the entropy at each position in the completion\n per_token_logps, entropies = self._get_per_token_logps_and_entropies(\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n compute_entropy=True,\n pixel_values=inputs.get(\"pixel_values\"),\n image_grid_thw=inputs.get(\"image_grid_thw\"),\n num_images=inputs.get(\"num_images\"),\n pixel_attention_mask=inputs.get(\"pixel_attention_mask\"),\n image_sizes=inputs.get(\"image_sizes\"),\n token_type_ids=inputs.get(\"token_type_ids\"),\n )\n\n if self.top_entropy_quantile < 1.0:\n entropy_mask = self.get_high_entropy_mask(entropies, completion_mask, 1 - self.top_entropy_quantile)\n else:\n entropy_mask = None\n\n # Compute the KL divergence between the model and the reference model\n if self.beta != 0.0:\n ref_per_token_logps = inputs[\"ref_per_token_logps\"]\n per_token_kl = (\n torch.exp(ref_per_token_logps - per_token_logps) - (ref_per_token_logps - per_token_logps) - 1\n )\n\n # Compute the loss\n advantages = inputs[\"advantages\"]\n # When num_iterations == 1 and steps_per_generation <= gradient_accumulation_steps,\n # old_per_token_logps == per_token_logps. In this case we can skip its computation\n # (see _generate_and_score_completions) and instead use per_token_logps.detach().\n # The exception is when using vLLM, where we always compute old_per_token_logps\n # for importance sampling\n old_per_token_logps = inputs.get(\"old_per_token_logps\")\n old_per_token_logps = per_token_logps.detach() if old_per_token_logps is None else old_per_token_logps\n\n log_ratio = per_token_logps - old_per_token_logps\n if self.importance_sampling_level == \"token\":\n log_importance_weights = log_ratio\n elif self.importance_sampling_level == \"sequence\":\n log_importance_weights = (log_ratio * completion_mask).sum(-1) / completion_mask.sum(-1).clamp(min=1.0)\n log_importance_weights = log_importance_weights.unsqueeze(-1)\n elif self.importance_sampling_level == \"sequence_token\":\n # GSPO-token: sg[si(ฮธ)] * ฯ€ฮธ(yi,t)/sg[ฯ€ฮธ(yi,t)]\n seq_level_log_weight = (log_ratio * completion_mask).sum(-1) / completion_mask.sum(-1).clamp(min=1.0)\n seq_level_log_weight = seq_level_log_weight.detach().unsqueeze(-1) # Stop gradient\n log_importance_weights = per_token_logps - per_token_logps.detach() + seq_level_log_weight\n else:\n raise ValueError(\n f\"Unknown importance sampling level: {self.importance_sampling_level}. Possible values are 'token' \"\n \"and 'sequence'.\"\n )\n # From here, log_importance_weights (and all subsequent tensors, coef_1, coef_2, etc.) shape depends on\n # importance_sampling_level: \"token\" level: (B, T); \"sequence\" level: (B, 1)\n\n coef_1 = torch.exp(log_importance_weights)\n coef_2 = torch.clamp(coef_1, 1 - self.epsilon_low, 1 + self.epsilon_high)\n\n # Two-sided clipping\n if self.args.delta is not None:\n coef_1 = torch.clamp(coef_1, max=self.args.delta)\n\n per_token_loss1 = coef_1 * advantages.unsqueeze(1)\n per_token_loss2 = coef_2 * advantages.unsqueeze(1)\n per_token_loss = -torch.min(per_token_loss1, per_token_loss2)\n if entropy_mask is not None:\n per_token_loss = per_token_loss * entropy_mask\n\n if self.use_vllm and self.vllm_importance_sampling_correction:\n per_token_loss = per_token_loss * inputs[\"importance_sampling_ratio\"]\n\n if self.beta != 0.0:\n per_token_loss = per_token_loss + self.beta * per_token_kl\n\n mode = \"train\" if self.model.training else \"eval\"\n if self.loss_type == \"grpo\":\n loss = ((per_token_loss * completion_mask).sum(-1) / completion_mask.sum(-1).clamp(min=1.0)).mean()\n normalizer = self.current_gradient_accumulation_steps if mode == \"train\" else 1.0 # no accum in eval\n loss = loss / normalizer\n elif self.loss_type == \"bnpo\":\n loss = (per_token_loss * completion_mask).sum() / completion_mask.sum().clamp(min=1.0)\n normalizer = self.current_gradient_accumulation_steps if mode == \"train\" else 1.0 # no accum in eval\n loss = loss / normalizer\n elif self.loss_type == \"dr_grpo\":\n loss = (per_token_loss * completion_mask).sum() / (per_token_loss.size(0) * self.max_completion_length)\n normalizer = self.current_gradient_accumulation_steps if mode == \"train\" else 1.0 # no accum in eval\n loss = loss / normalizer\n elif self.loss_type == \"dapo\":\n normalizer = inputs[\"num_items_in_batch\"] / self.accelerator.num_processes\n loss = (per_token_loss * completion_mask).sum() / normalizer\n else:\n raise ValueError(f\"Unknown loss type: {self.loss_type}\")\n\n # Log the metrics\n completion_token_count = completion_mask.sum().clamp(min=1.0)\n\n def masked_batch_mean(x):\n if x.shape[1] == 1: # when importance_sampling_level == \"sequence\"\n return x.mean()\n else:\n return (x * completion_mask).sum() / completion_token_count\n\n if self.beta != 0.0:\n mean_kl = masked_batch_mean(per_token_kl)\n self._metrics[mode][\"kl\"].append(self.accelerator.gather(mean_kl).nanmean().item())\n\n mean_entropy = masked_batch_mean(entropies)\n self._metrics[mode][\"entropy\"].append(self.accelerator.gather(mean_entropy).nanmean().item())\n\n # Compute the clipped probability ratios\n is_low_clipped = (coef_1 < 1 - self.epsilon_low) & (advantages.unsqueeze(1) < 0)\n is_high_clipped = (coef_1 > 1 + self.epsilon_high) & (advantages.unsqueeze(1) > 0)\n is_region_clipped = is_low_clipped | is_high_clipped\n\n low_clip = masked_batch_mean(is_low_clipped.float())\n high_clip = masked_batch_mean(is_high_clipped.float())\n clip_ratio = masked_batch_mean(is_region_clipped.float())\n\n gathered_low_clip = self.accelerator.gather(low_clip)\n self._metrics[mode][\"clip_ratio/low_mean\"].append(gathered_low_clip.nanmean().item())\n self._metrics[mode][\"clip_ratio/low_min\"].append(nanmin(gathered_low_clip).item())\n gathered_high_clip = self.accelerator.gather(high_clip)\n self._metrics[mode][\"clip_ratio/high_mean\"].append(gathered_high_clip.nanmean().item())\n self._metrics[mode][\"clip_ratio/high_max\"].append(nanmax(gathered_high_clip).item())\n gathered_clip_ratio = self.accelerator.gather(clip_ratio)\n self._metrics[mode][\"clip_ratio/region_mean\"].append(gathered_clip_ratio.nanmean().item())\n return loss\n" }, { "path": "trl/experimental/judges/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .judges import (\n AllTrueJudge,\n BaseBinaryJudge,\n BaseJudge,\n BasePairwiseJudge,\n BaseRankJudge,\n HfPairwiseJudge,\n OpenAIPairwiseJudge,\n PairRMJudge,\n)\n\n\n__all__ = [\n \"AllTrueJudge\",\n \"BaseBinaryJudge\",\n \"BaseJudge\",\n \"BasePairwiseJudge\",\n \"BaseRankJudge\",\n \"HfPairwiseJudge\",\n \"OpenAIPairwiseJudge\",\n \"PairRMJudge\",\n]\n" }, { "path": "trl/experimental/judges/judges.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport concurrent.futures\nimport logging\nfrom abc import ABC, abstractmethod\n\nimport numpy as np\nfrom accelerate import Accelerator\nfrom huggingface_hub import InferenceClient\nfrom packaging.version import Version\nfrom transformers.utils import is_openai_available\n\nfrom ...import_utils import is_llm_blender_available\n\n\nDEFAULT_PAIRWISE_SYSTEM_PROMPT = '''I require a leaderboard for various large language models. I'll provide you with prompts given to these models and their corresponding outputs. Your task is to assess these responses, and select the model that produces the best output from a human perspective.\n\n## Instruction\n\n{{\n \"instruction\": \"\"\"{prompt}\"\"\",\n}}\n\n## Model Outputs\n\nHere are the unordered outputs from the models. Each output is associated with a specific model, identified by a unique model identifier.\n\n{{\n {{\n \"model_identifier\": \"0\",\n \"output\": \"\"\"{response0}\"\"\"\n }},\n {{\n \"model_identifier\": \"1\",\n \"output\": \"\"\"{response1}\"\"\"\n }}\n}}\n\n## Task\n\nEvaluate the models on the basis of the quality and relevance of their results, and select the model that generated the best result. Reply with the identifier of the best model. Our evaluation will only take into account the first character of your answer, so make sure it contains only one of the identifiers and nothing else (no quotation marks, no spaces, no new lines, ...).\n'''\n\n\ndef _ensure_llm_blender_importable() -> None:\n \"\"\"\n Pre-import shim to work around a known `llm-blender` issue.\n\n As of `llm-blender` v0.0.2 (see upstream issue: https://github.com/yuchenlin/LLM-Blender/issues/33), importing\n `llm_blender` may fail on `transformers` >= 5.0.0 because it unconditionally accesses\n `transformers.utils.hub.TRANSFORMERS_CACHE`.\n\n We set this attribute to a dummy value before importing `llm_blender` so that the import succeeds. This helper is\n intentionally a no-op on older `transformers` versions.\n\n This shim can be removed once the upstream issue is fixed and the minimum required `llm-blender` version includes\n that fix.\n \"\"\"\n import transformers.utils.hub\n\n if Version(transformers.__version__) >= Version(\"5.0.0\"):\n transformers.utils.hub.TRANSFORMERS_CACHE = None # unused; just needs to exist\n\n\nclass BaseJudge(ABC):\n \"\"\"\n Base class for judges. The subclasses of this class should implement the `judge` method.\n \"\"\"\n\n @abstractmethod\n def judge(self, prompts: list[str], completions: list[str], shuffle_order: bool = True) -> list:\n raise NotImplementedError(\"Judge subclasses must implement the `judge` method.\")\n\n\nclass BaseRankJudge(ABC):\n \"\"\"\n Base class for LLM ranking judges.\n\n **Example**:\n ```python\n class MyRankJudge(BaseRankJudge):\n def judge(self, prompts, completions, shuffle_order=True):\n return ... # Your ranking logic here\n\n\n judge = MyRankJudge()\n judge.judge(\n prompts=[\"The capital of France is\", \"The capital of Germany is\"],\n completions=[[\" Paris\", \" Marseille\", \"Lyon\"], [\" Munich\", \" Berlin\"]],\n ) # [[0, 1, 2], [1, 0]]\n ```\n \"\"\"\n\n @abstractmethod\n def judge(self, prompts: list[str], completions: list[list[str]], shuffle_order: bool = True) -> list[list[int]]:\n \"\"\"\n Judge the completion for the given prompts and return the ranks of each completion.\n\n Args:\n prompts (`list[str]`):\n List of prompts.\n completions (`list[list[str]]`):\n List of completions list, where each element is a list of completions for the corresponding prompt.\n shuffle_order (`bool`, *optional*, defaults to `True`):\n Whether to shuffle the order of the completions to avoid positional bias.\n\n Returns:\n `list[list[int]]`:\n List of lists of idxs, where each list contains the ranks of the completions for the corresponding\n prompt. E.g., `[1, 2, 0]` means that the second completion (`idx=1`) is the best, followed by the\n third, and then the first.\n \"\"\"\n raise NotImplementedError(\"Judge subclasses must implement the `judge` method.\")\n\n\nclass BasePairwiseJudge(BaseJudge):\n \"\"\"\n Base class for pairwise judges.\n \"\"\"\n\n @abstractmethod\n def judge(self, prompts: list[str], completions: list[list[str]], shuffle_order: bool = True) -> list[int]:\n \"\"\"\n Judge the completion pairs for the given prompts.\n\n Args:\n prompts (`list[str]`):\n List of prompts.\n completions (`list[list[str]]`):\n List of completions pairs, where each element is a pair of completions for the corresponding prompt.\n shuffle_order (`bool`, *optional*, defaults to `True`):\n Whether to shuffle the order of the completions to avoid positional bias.\n\n Returns:\n `list[int]`:\n List of idxs, where each idx is the rank of the best completion for the corresponding prompt. E.g., `1`\n means that the second completion (`idx=1`) is the best.\n\n Note:\n If the judge returns `-1` for any prompt, it indicates that the inner process used to compute the\n preference has failed. For instance, this could occur if the underlying language model returned an invalid\n answer. In such cases, the caller should handle these invalid indices appropriately, possibly by\n implementing fallback logic or error handling.\n \"\"\"\n raise NotImplementedError(\"Judge subclasses must implement the `judge` method.\")\n\n\nclass BaseBinaryJudge(BaseJudge):\n \"\"\"\n Base class for binary judges.\n \"\"\"\n\n @abstractmethod\n def judge(\n self,\n prompts: list[str],\n completions: list[str],\n gold_completions: list[str] | None = None,\n shuffle_order: bool = True,\n ) -> list[int]:\n \"\"\"\n Judge the completion for a given prompt. Used to assess if a completion satisfies a constraint.\n\n This base class should be used to implement binary evaluations as done in section 4.1.4 of the [CGPO\n paper](https://huggingface.co/papers/2409.20370). It is relevant for assessing whether a prompt-completion pair\n satisfies a specific constraint.\n\n Args:\n prompts (`list[str]`): List of prompts.\n completions (`list[str]`): List of completions.\n gold_completions (`list[str]`, `optional`): List of gold completions if it exists.\n shuffle_order (`bool`): Whether to shuffle the order of the completions to avoid positional bias.\n\n Returns:\n list[int]: A list of binary labels:\n - 1 indicates that the completion satisfies the evaluated constraint.\n - 0 indicates that the completion does not satisfy the evaluated constraint.\n\n Note:\n If the judge returns -1 for any prompt, it indicates that the inner process used to compute the preference\n has failed. For instance, this could occur if the underlying language model or rule based constraint\n returned an invalid answer. In such cases, the caller should handle these invalid indices appropriately,\n possibly by implementing fallback logic or error handling.\n \"\"\"\n raise NotImplementedError(\"Judge subclasses must implement the `judge` method.\")\n\n\nclass PairRMJudge(BasePairwiseJudge):\n # docstyle-ignore\n \"\"\"\n LLM judge based on the PairRM model from AllenAI.\n\n This judge uses the PairRM model to rank pairs of completions for given prompts. It's designed for pairwise\n comparison of language model outputs. The PairRM model is loaded using the llm-blender library and runs on the\n default Accelerator device.\n\n **Attributes**:\n\n blender (`llm_blender.Blender`):\n An instance of the Blender class from llm-blender.\n\n **Example**:\n ```python\n >>> pairrm_judge = PairRMJudge()\n >>> prompts = [\"Translate 'hello' to French\", \"What's the capital of Japan?\"]\n >>> completions = [[\"Bonjour\", \"Salut\"], [\"Kyoto\", \"Tokyo\"]]\n >>> results = pairrm_judge.judge(prompts, completions)\n >>> print(results) # [0, 1] (indicating the first completion is preferred for the first prompt and the second)\n ```\n\n > [!TIP]\n > This class requires the llm-blender library to be installed. Install it with: `pip install llm-blender`.\n \"\"\"\n\n def __init__(self):\n if not is_llm_blender_available():\n raise ValueError(\"llm-blender is not installed. Please install it with `pip install llm-blender`.\")\n import transformers\n\n if Version(transformers.__version__) >= Version(\"5.0.0\"):\n raise RuntimeError(\n \"llm-blender currently supports transformers < 5.0.0. Please install a compatible version: `pip install 'transformers<5.0.0'`. Check the issue tracker for updates: https://github.com/huggingface/trl/issues/4918\"\n )\n _ensure_llm_blender_importable()\n import llm_blender\n\n self.blender = llm_blender.Blender()\n self.blender.loadranker(\"llm-blender/PairRM\", device=Accelerator().device)\n\n def judge(\n self,\n prompts: list[str],\n completions: list[list[str]],\n shuffle_order: bool = True,\n return_scores: bool = False,\n temperature: float = 1.0,\n ) -> list[int | float]:\n \"\"\"\n Judge the completion pairs for the given prompts using the PairRM model.\n\n Args:\n prompts (`list[str]`):\n List of prompts to judge.\n completions (`list[list[str]]`):\n List of completion pairs for each prompt.\n shuffle_order (`bool`, *optional*, defaults to `True`):\n Whether to shuffle the order of the completions to avoid positional bias.\n return_scores (`bool`, *optional*, defaults to `False`):\n If `True`, return probability scores of the first completion instead of ranks (i.e. a *soft-judge*).\n temperature (`float`, *optional*, defaults to `1.0`):\n Temperature for scaling logits if `return_scores` is True.\n\n Returns:\n `list[int | float]`:\n If `return_scores` is `False`, returns a list of ranks (`0` or `1`) for each prompt, indicating which\n completion is preferred. If `return_scores` is `True`, returns softmax probabilities for the first\n completion.\n\n Raises:\n `ValueError`:\n If the number of completions per prompt is not exactly 2.\n\n Note:\n Unlike llm-blender, ranks are 0-indexed (`0` means the first completion is preferred).\n \"\"\"\n\n if len(completions[0]) != 2:\n raise ValueError(\"PairRM judge requires exactly 2 completions per prompt.\")\n\n # Shuffle the order of the completions to avoid positional bias\n if shuffle_order:\n flip_mask = np.random.choice([True, False], size=len(prompts))\n completions = [pair[::-1] if flip else pair for flip, pair in zip(flip_mask, completions, strict=True)]\n\n # Rank the completions\n ranks = self.blender.rank(prompts, completions, return_scores=return_scores, disable_tqdm=True)\n if not return_scores:\n ranks -= 1 # PairRM rank is 1-indexed, so we subtract 1 to make it 0-indexed\n else:\n # scale the logits by temperature\n ranks /= temperature\n\n # Flip back the ranks or scores to the original order if needed\n if shuffle_order:\n ranks[flip_mask] = ranks[flip_mask][:, ::-1]\n\n # Return the ranks or score probability\n if return_scores:\n logit_max = np.amax(ranks, axis=-1, keepdims=True)\n exp_logit_shifted = np.exp(ranks - logit_max)\n probs = exp_logit_shifted / np.sum(exp_logit_shifted, axis=-1, keepdims=True)\n return probs[:, 0].tolist()\n else:\n return ranks[:, 0].tolist()\n\n\nclass HfPairwiseJudge(BasePairwiseJudge):\n \"\"\"\n Pairwise judge based on the Hugging Face API with chat completion.\n\n This judge is relevant for assessing the quality chat models, where the completion is a response to a given prompt.\n\n Args:\n model (`str`, *optional*, defaults to `\"meta-llama/Meta-Llama-3-70B-Instruct\"`):\n Model to use for the judge.\n token (`str`, *optional*):\n Hugging Face API token to use for the [`huggingface_hub.InferenceClient`].\n system_prompt (`str`, *optional*):\n The system prompt to be used for the judge. If not provided, a default prompt is used. Note that the system\n prompt should contain the following placeholders: `{prompt}`, `{response0}`, and `{response1}`. Also, the\n inference is called with `max_tokens=1`, consequently the system prompt should ask for a single token\n response.\n \"\"\"\n\n def __init__(\n self,\n model=\"meta-llama/Meta-Llama-3-70B-Instruct\",\n token: str | None = None,\n system_prompt: str | None = None,\n ):\n self.client = InferenceClient(model=model, token=token)\n self.system_prompt = system_prompt or DEFAULT_PAIRWISE_SYSTEM_PROMPT\n\n def judge(self, prompts: list[str], completions: list[list[str]], shuffle_order: bool = True) -> list[int]:\n # Shuffle the order of the completions to avoid positional bias\n if shuffle_order:\n flip_mask = np.random.choice([True, False], size=len(prompts))\n completions = [pair[::-1] if flip else pair for flip, pair in zip(flip_mask, completions, strict=True)]\n\n # Define a function to get the rank for a single prompt, will be called concurrently\n def get_rank(prompt, candidates):\n content = self.system_prompt.format(prompt=prompt, response0=candidates[0], response1=candidates[1])\n completion = self.client.chat_completion(messages=[{\"role\": \"user\", \"content\": content}], max_tokens=1)\n response = completion.choices[0].message.content\n if response in [\"0\", \"1\"]:\n return int(response)\n else:\n logging.debug(f\"Invalid response from the judge model: '{response}'. Returning -1.\")\n return -1\n\n # Call the completions concurrently\n with concurrent.futures.ThreadPoolExecutor() as executor:\n ranks = list(executor.map(get_rank, prompts, completions))\n\n # Flip back the ranks to the original order if needed\n if shuffle_order:\n ranks = [ranks[i] if not flip else 1 - ranks[i] for i, flip in enumerate(flip_mask)]\n\n # Return the ranks\n return ranks\n\n\nclass OpenAIPairwiseJudge(BasePairwiseJudge):\n \"\"\"\n Judge based on the OpenAI API.\n\n This judge is relevant for assessing the quality chat models, where the completion is a response to a given prompt.\n\n Args:\n model (`str`, *optional*, defaults to `\"gpt-4-turbo-preview\"`):\n Model to use for the judge.\n system_prompt (`str`, *optional*):\n System prompt to be used for the judge. If not provided, a default prompt is used. Note that the system\n prompt should contain the following placeholders: `{prompt}`, `{response0}`, and `{response1}`. Also, the\n inference is called with `max_tokens=1`, consequently the system prompt should ask for a single token\n response.\n max_requests (`int` or `None`, *optional*, defaults to `1000`):\n Maximum number of requests to make to the OpenAI API. If set to `None`, there is no limit.\n \"\"\"\n\n def __init__(\n self, model=\"gpt-4-turbo-preview\", system_prompt: str | None = None, max_requests: int | None = 1_000\n ):\n if not is_openai_available():\n raise ValueError(\"OpenAI client is not installed. Please install it with 'pip install openai'.\")\n from openai import OpenAI\n\n self.client = OpenAI()\n self.model = model\n self.system_prompt = system_prompt or DEFAULT_PAIRWISE_SYSTEM_PROMPT\n self.max_requests = max_requests\n self.num_requests = 0\n self._warned = False\n\n def judge(self, prompts: list[str], completions: list[list[str]], shuffle_order: bool = True) -> list[int]:\n # Check if the limit of requests is reached, if so, use random choice instead\n if self.max_requests is not None and self.num_requests >= self.max_requests:\n if not self._warned: # Print the warning only once\n logging.warning(\n f\"Reached the maximum number of requests ({self.max_requests}). From now on, returning -1 instead. \"\n \" To increase the limit, set `max_requests` to a higher value, or to `None` for no limit.\"\n )\n self._warned = True\n return [-1] * len(prompts)\n\n # Shuffle the order of the completions to avoid positional bias\n if shuffle_order:\n flip_mask = np.random.choice([True, False], size=len(prompts))\n completions = [pair[::-1] if flip else pair for flip, pair in zip(flip_mask, completions, strict=True)]\n\n # Define a function to get the rank for a single prompt, will be called concurrently\n def get_rank(prompt, candidates):\n content = self.system_prompt.format(prompt=prompt, response0=candidates[0], response1=candidates[1])\n messages = [{\"role\": \"user\", \"content\": content}]\n completion = self.client.chat.completions.create(model=self.model, messages=messages, max_tokens=1)\n response = completion.choices[0].message.content\n if response in [\"0\", \"1\"]:\n return int(response)\n else:\n logging.debug(f\"Invalid response from the judge model: '{response}'. Returning -1.\")\n return -1\n\n # Call the completions concurrently\n with concurrent.futures.ThreadPoolExecutor() as executor:\n ranks = list(executor.map(get_rank, prompts, completions))\n\n # Flip back the ranks to the original order if needed\n if shuffle_order:\n ranks = [ranks[i] if not flip else 1 - ranks[i] for i, flip in enumerate(flip_mask)]\n\n # Update the number of requests\n self.num_requests += len(prompts)\n\n # Return the ranks\n return ranks\n\n\nclass AllTrueJudge(BaseBinaryJudge):\n \"\"\"\n Unify the decision of multiple [`experimental.judges.BaseBinaryJudge`] instances.\n\n Returns `1` only if all inner binary judges return `1`. If any judge returns `0`, it returns `0`. If any judge\n returns `-1`, indicating a failure in its process, this judge will also return `-1`.\n\n Implements the Mixture of Judges as described in the [CGPO paper](https://huggingface.co/papers/2409.20370).\n\n Args:\n judges (`list` of [`experimental.judges.BaseBinaryJudge`]):\n A list of [`experimental.judges.BaseBinaryJudge`] instances whose decisions will be unified.\n \"\"\"\n\n def __init__(self, judges: list[BaseBinaryJudge]):\n self.judges = judges\n\n def judge(\n self,\n prompts: list[str],\n completions: list[str],\n gold_completions: list[str] | None = None,\n shuffle_order: bool = True,\n ) -> list[int]:\n all_binary_judgments = [\n judge.judge(prompts, completions, gold_completions, shuffle_order) for judge in self.judges\n ]\n output = []\n for binary_judgments in zip(*all_binary_judgments, strict=True):\n # Check that all values are in {0, 1, -1}\n if any(binary_judgment not in {0, 1, -1} for binary_judgment in binary_judgments):\n raise ValueError(\n f\"Invalid binary judgment: {binary_judgments}, expected list of values in {{0, 1, -1}}.\"\n )\n\n # Unify the decision\n if -1 in binary_judgments:\n output.append(-1)\n elif all(binary_judgment == 1 for binary_judgment in binary_judgments):\n output.append(1)\n else:\n output.append(0)\n return output\n" }, { "path": "trl/experimental/kto/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .kto_config import KTOConfig\nfrom .kto_trainer import KTOTrainer\n\n\n__all__ = [\"KTOConfig\", \"KTOTrainer\"]\n" }, { "path": "trl/experimental/kto/kto_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom ...trainer.base_config import _BaseConfig\n\n\n@dataclass\nclass KTOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`experimental.kto.KTOTrainer`].\n\n This class includes only the parameters that are specific to KTO training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n max_length (`int` or `None`, *optional*, defaults to `1024`):\n Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want\n to use the default data collator.\n beta (`float`, *optional*, defaults to `0.1`):\n Parameter controlling the deviation from the reference model. Higher ฮฒ means less deviation from the\n reference model.\n loss_type (`str`, *optional*, defaults to `\"kto\"`):\n Type of loss to use. Possible values are:\n\n - `\"kto\"`: KTO loss from the [KTO](https://huggingface.co/papers/2402.01306) paper.\n - `\"apo_zero_unpaired\"`: Unpaired variant of APO-zero loss from the\n [APO](https://huggingface.co/papers/2408.06266) paper.\n\n desirable_weight (`float`, *optional*, defaults to `1.0`):\n Desirable losses are weighed by this factor to counter unequal number of desirable and undesirable paris.\n undesirable_weight (`float`, *optional*, defaults to `1.0`):\n Undesirable losses are weighed by this factor to counter unequal number of desirable and undesirable pairs.\n generate_during_eval (`bool`, *optional*, defaults to `False`):\n If `True`, generates and logs completions from both the model and the reference model to W&B or Comet\n during evaluation.\n precompute_ref_log_probs (`bool`, *optional*, defaults to `False`):\n Whether to precompute reference model log probabilities for training and evaluation datasets. This is\n useful when training without the reference model to reduce the total GPU memory needed.\n model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a\n string.\n dataset_num_proc: (`int`, *optional*):\n Number of processes to use for processing the dataset.\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model and reference model.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `1e-6` instead of `5e-5`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-6,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n max_length: int | None = field(\n default=1024,\n metadata={\"help\": \"Maximum length of the sequences (prompt + completion) in the batch.\"},\n )\n beta: float = field(\n default=0.1,\n metadata={\n \"help\": \"Parameter controlling the deviation from the reference model. Higher ฮฒ means less deviation from \"\n \"the reference model.\"\n },\n )\n loss_type: str = field(\n default=\"kto\",\n metadata={\n \"help\": \"Type of loss to use.\",\n \"choices\": [\"kto\", \"apo_zero_unpaired\"],\n },\n )\n desirable_weight: float = field(\n default=1.0,\n metadata={\n \"help\": \"Desirable losses are weighed by this factor to counter unequal number of desirable and \"\n \"undesirable pairs.\",\n },\n )\n undesirable_weight: float = field(\n default=1.0,\n metadata={\n \"help\": \"Undesirable losses are weighed by this factor to counter unequal number of desirable and \"\n \"undesirable pairs.\",\n },\n )\n generate_during_eval: bool = field(\n default=False,\n metadata={\n \"help\": \"If `True`, generates and logs completions from both the model and the reference model to W&B \"\n \"during evaluation.\"\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropout in the model.\"},\n )\n precompute_ref_log_probs: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to precompute reference model log probabilities for training and evaluation datasets. \"\n \"This is useful when training without the reference model to reduce the total GPU memory needed.\"\n },\n )\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model \"\n \"from a string.\"\n },\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes to use for processing the dataset.\"},\n )\n\n def __post_init__(self):\n self.bf16 = not (self.fp16) if self.bf16 is None else self.bf16\n\n super().__post_init__()\n" }, { "path": "trl/experimental/kto/kto_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport inspect\nimport random\nimport textwrap\nfrom collections import defaultdict\nfrom collections.abc import Callable\nfrom contextlib import contextmanager, nullcontext\nfrom operator import itemgetter\nfrom pathlib import Path\nfrom typing import TYPE_CHECKING, Any, Literal\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport transformers\nfrom accelerate import PartialState, logging\nfrom accelerate.utils import tqdm\nfrom datasets import Dataset, concatenate_datasets\nfrom packaging.version import Version\nfrom torch import autocast\nfrom torch.utils.data import DataLoader, SequentialSampler\nfrom transformers import (\n BaseImageProcessor,\n DataCollator,\n FeatureExtractionMixin,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n TrainingArguments,\n is_comet_available,\n is_wandb_available,\n)\nfrom transformers.trainer_utils import EvalLoopOutput, has_length\nfrom transformers.utils import is_peft_available\n\nfrom ...data_utils import maybe_apply_chat_template, maybe_extract_prompt, maybe_unpair_preference_dataset\nfrom ...import_utils import is_liger_kernel_available\nfrom ...models.utils import prepare_deepspeed\nfrom ...trainer.base_trainer import _BaseTrainer\nfrom ...trainer.utils import (\n create_model_from_path,\n disable_dropout_in_model,\n log_table_to_comet_experiment,\n selective_log_softmax,\n)\nfrom ..utils import DPODataCollatorWithPadding, create_reference_model, pad_to_length, peft_module_casting_to_bf16\nfrom .kto_config import KTOConfig\n\n\nif is_liger_kernel_available():\n from liger_kernel.chunked_loss import LigerFusedLinearKTOLoss\n\nif is_peft_available():\n from peft import PeftModel, get_peft_model, prepare_model_for_kbit_training\n\nif is_wandb_available():\n import wandb\n\n\nif TYPE_CHECKING:\n from transformers import PreTrainedModel, PreTrainedTokenizer\n\n\nlogger = logging.get_logger(__name__)\n\nRUNNING_NAME = \"running.pt\"\n\n\ndef _get_kl_dataset(batch: dict[str, list[Any]]) -> dict[str, list[Any]]:\n \"\"\"\n Creates mismatched pairs of prompts and completions for the KL dataset by adding a +1 offset to the order of\n completions. For best results, the mismatched outputs y' used to estimate the KL term for a batch should be the\n same set as the matched outputs y used to estimate the rewards in that batch, just paired with different x.\n \"\"\"\n batch[\"answer_input_ids\"] = [batch[\"answer_input_ids\"][-1]] + batch[\"answer_input_ids\"][:-1]\n batch[\"answer_attention_mask\"] = [batch[\"answer_attention_mask\"][-1]] + batch[\"answer_attention_mask\"][:-1]\n return batch\n\n\ndef _tokenize(\n batch: dict[str, list[Any]],\n tokenizer: \"PreTrainedTokenizer\",\n) -> dict[str, list[Any]]:\n \"\"\"Tokenize a batch from a KTO specific dataset.\"\"\"\n prompt_tokenized = tokenizer(batch[\"prompt\"], add_special_tokens=False)\n prompt_input_ids = prompt_tokenized[\"input_ids\"]\n prompt_attention_mask = prompt_tokenized[\"attention_mask\"]\n prompt_and_completion = [\n prompt + completion for prompt, completion in zip(batch[\"prompt\"], batch[\"completion\"], strict=True)\n ]\n full_tokenized = tokenizer(prompt_and_completion, add_special_tokens=False)\n full_input_ids = full_tokenized[\"input_ids\"]\n full_attention_mask = full_tokenized[\"attention_mask\"]\n\n answer_input_ids = [f[len(p) :] for f, p in zip(full_input_ids, prompt_input_ids, strict=True)]\n answer_attention_mask = [f[len(p) :] for f, p in zip(full_attention_mask, prompt_attention_mask, strict=True)]\n\n # Concat tokens to form `enc(a) + enc(a + b)[len(enc(a)):]`\n full_concat_input_ids = [np.concatenate([p, a]) for p, a in zip(prompt_input_ids, answer_input_ids, strict=True)]\n # Prepare input tokens for token by token comparison\n full_input_ids = [np.array(f) for f in full_input_ids]\n for full, concat in zip(full_input_ids, full_concat_input_ids, strict=True):\n if len(full) != len(concat):\n raise ValueError(\n \"The elements in 'full_input_ids' and 'full_concat_input_ids' must have the same pairwise length.\"\n )\n\n # On some tokenizers, like Llama-2 tokenizer, there are occasions where tokens\n # can be merged together when tokenizing prompt+answer. This could result\n # on the last token from the prompt being different when tokenized on its own\n # vs when done as prompt+answer.\n response_token_ids_start_idx = [len(p) for p in prompt_input_ids]\n\n # If tokenized prompt is different than both prompt+answer, then it means the\n # last token has changed due to merging.\n for idx, (p, f, r) in enumerate(zip(prompt_input_ids, full_input_ids, response_token_ids_start_idx, strict=True)):\n if not np.array_equal(p, f[:r]):\n response_token_ids_start_idx[idx] -= 1\n\n prompt_input_ids = [f[:r] for f, r in zip(full_input_ids, response_token_ids_start_idx, strict=True)]\n prompt_attention_mask = [f[:r] for f, r in zip(full_attention_mask, response_token_ids_start_idx, strict=True)]\n\n for p, m in zip(prompt_input_ids, prompt_attention_mask, strict=True):\n if len(p) != len(m):\n raise ValueError(\"Prompt input ids and attention mask should have the same length.\")\n\n answer_input_ids = [f[r:] for f, r in zip(full_input_ids, response_token_ids_start_idx, strict=True)]\n answer_attention_mask = [f[r:] for f, r in zip(full_attention_mask, response_token_ids_start_idx, strict=True)]\n\n output = dict(\n prompt_input_ids=prompt_input_ids,\n prompt_attention_mask=prompt_attention_mask,\n answer_input_ids=answer_input_ids,\n answer_attention_mask=answer_attention_mask,\n )\n\n return output\n\n\ndef _process_tokens(example: dict[str, Any], model: \"PreTrainedModel\" = None, **kwargs) -> dict:\n \"\"\"Process tokens of a KTO specific dataset.\n\n At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +\n completion responses is/are too long. We truncate from the end (completion) to fit within max_length.\n\n We also create the labels for the completion responses, which are of length equal to the sum of the length of the\n prompt and the completion response, with `-100` for the prompt tokens.\n \"\"\"\n prompt = example[\"prompt\"]\n completion = example[\"completion\"]\n\n batch = {\n f\"{kwargs['prefix']}prompt\": prompt,\n f\"{kwargs['prefix']}completion\": completion,\n f\"{kwargs['prefix']}label\": example[\"label\"],\n }\n\n # Check issues below for more details\n # 1. https://github.com/huggingface/trl/issues/907\n # 2. https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257\n # 3. https://github.com/LianjiaTech/BELLE/issues/337\n\n if not isinstance(prompt, str):\n raise ValueError(f\"prompt should be an str but got {type(prompt)}\")\n\n if not isinstance(completion, str):\n raise ValueError(f\"completion should be an str but got {type(completion)}\")\n\n # keys of format prompt_* refers to just the prompt and answer_* refers to just the answer\n all_tokens = {\n \"prompt_input_ids\": example[\"prompt_input_ids\"],\n \"prompt_attention_mask\": example[\"prompt_attention_mask\"],\n \"answer_input_ids\": example[\"answer_input_ids\"],\n \"answer_attention_mask\": example[\"answer_attention_mask\"],\n }\n\n # calculate max length by checking if BOS/EOS is already there\n max_length = kwargs[\"max_length\"]\n bos_token_id = kwargs[\"tokenizer\"].bos_token_id\n eos_token_id = kwargs[\"tokenizer\"].eos_token_id\n if len(all_tokens[\"prompt_input_ids\"]) > 0 and bos_token_id != all_tokens[\"prompt_input_ids\"][0]:\n max_length -= 1\n if len(all_tokens[\"answer_input_ids\"]) > 0 and eos_token_id != all_tokens[\"answer_input_ids\"][-1]:\n max_length -= 1\n\n # if combined sequence is too long, truncate the completion (answer) from the end\n prompt_length = len(all_tokens[\"prompt_input_ids\"])\n completion_length = len(all_tokens[\"answer_input_ids\"])\n if prompt_length + completion_length > max_length:\n max_completion_length = max_length - prompt_length\n for k in [\"answer_input_ids\", \"answer_attention_mask\"]:\n all_tokens[k] = all_tokens[k][:max_completion_length]\n\n # all input_ids and attention mask as is. We then check if we need to add BOS/EOS tokens\n batch[f\"{kwargs['prefix']}prompt_input_ids\"] = all_tokens[\"prompt_input_ids\"]\n batch[f\"{kwargs['prefix']}prompt_attention_mask\"] = all_tokens[\"prompt_attention_mask\"]\n batch[f\"{kwargs['prefix']}completion_input_ids\"] = all_tokens[\"prompt_input_ids\"] + all_tokens[\"answer_input_ids\"]\n batch[f\"{kwargs['prefix']}completion_attention_mask\"] = (\n all_tokens[\"prompt_attention_mask\"] + all_tokens[\"answer_attention_mask\"]\n )\n\n # add BOS, which affects both prompt and the full completion\n if bos_token_id is not None:\n if len(all_tokens[\"prompt_input_ids\"]) == 0 or bos_token_id != all_tokens[\"prompt_input_ids\"][0]:\n batch[f\"{kwargs['prefix']}prompt_input_ids\"] = [bos_token_id] + batch[\n f\"{kwargs['prefix']}prompt_input_ids\"\n ]\n batch[f\"{kwargs['prefix']}prompt_attention_mask\"] = [1] + batch[f\"{kwargs['prefix']}prompt_attention_mask\"]\n batch[f\"{kwargs['prefix']}completion_input_ids\"] = [bos_token_id] + batch[\n f\"{kwargs['prefix']}completion_input_ids\"\n ]\n batch[f\"{kwargs['prefix']}completion_attention_mask\"] = [1] + batch[\n f\"{kwargs['prefix']}completion_attention_mask\"\n ]\n # add EOS, which affects only the full completion\n if len(all_tokens[\"answer_input_ids\"]) == 0 or eos_token_id != all_tokens[\"answer_input_ids\"][-1]:\n batch[f\"{kwargs['prefix']}completion_input_ids\"] = batch[f\"{kwargs['prefix']}completion_input_ids\"] + [\n eos_token_id\n ]\n batch[f\"{kwargs['prefix']}completion_attention_mask\"] = batch[\n f\"{kwargs['prefix']}completion_attention_mask\"\n ] + [1]\n\n batch[f\"{kwargs['prefix']}completion_labels\"] = batch[f\"{kwargs['prefix']}completion_input_ids\"][:]\n batch[f\"{kwargs['prefix']}completion_labels\"][: len(batch[f\"{kwargs['prefix']}prompt_input_ids\"])] = [-100] * len(\n batch[f\"{kwargs['prefix']}prompt_input_ids\"]\n )\n\n return batch\n\n\nclass KTOTrainer(_BaseTrainer):\n r\"\"\"\n Initialize KTOTrainer.\n\n Args:\n model ([`~transformers.PreTrainedModel`]):\n The model to train, preferably an [`~transformers.AutoModelForSequenceClassification`].\n ref_model ([`~transformers.PreTrainedModel`]):\n Hugging Face transformer model with a casual language modelling head. Used for implicit reward computation\n and loss. If no reference model is provided, the trainer will create a reference model with the same\n architecture as the model to be optimized.\n args ([`experimental.kto.KTOConfig`]):\n The arguments to use for training.\n train_dataset ([`~datasets.Dataset`]):\n The dataset to use for training.\n eval_dataset ([`~datasets.Dataset`]):\n The dataset to use for evaluation.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. If provided, will be used to automatically process the inputs\n for the model, and it will be saved along the model to make it easier to rerun an interrupted training or\n reuse the fine-tuned model.\n data_collator ([`~transformers.DataCollator`], *optional*):\n The data collator to use for training. If None is specified, the default data collator\n ([`experimental.utils.DPODataCollatorWithPadding`]) will be used which will pad the sequences to the\n maximum length of the sequences in the batch, given a dataset of paired sequences.\n model_init (`Callable[[], transformers.PreTrainedModel]`):\n The model initializer to use for training. If None is specified, the default model initializer will be\n used.\n callbacks (`list[transformers.TrainerCallback]`):\n The callbacks to use for training.\n optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):\n The optimizer and scheduler to use for training.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):\n The function to use to preprocess the logits before computing the metrics.\n peft_config (`dict`, defaults to `None`):\n The PEFT configuration to use for training. If you pass a PEFT configuration, the model will be wrapped in\n a PEFT model.\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to\n metric values.\n model_adapter_name (`str`, defaults to `None`):\n Name of the train target PEFT adapter, when using LoRA with multiple adapters.\n ref_adapter_name (`str`, defaults to `None`):\n Name of the reference PEFT adapter, when using LoRA with multiple adapters.\n \"\"\"\n\n _tag_names = [\"trl\", \"kto\"]\n _name = \"KTO\"\n _paper = {\n \"title\": \"KTO: Model Alignment as Prospect Theoretic Optimization\",\n \"id\": \"2402.01306\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{ethayarajh2024kto,\n title = {{KTO: Model Alignment as Prospect Theoretic Optimization}},\n author = {Kawin Ethayarajh and Winnie Xu and Niklas Muennighoff and Dan Jurafsky and Douwe Kiela},\n year = 2024,\n eprint = {arXiv:2402.01306},\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module | str = None,\n ref_model: PreTrainedModel | nn.Module | str | None = None,\n args: KTOConfig = None,\n train_dataset: Dataset | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n processing_class: PreTrainedTokenizerBase\n | BaseImageProcessor\n | FeatureExtractionMixin\n | ProcessorMixin\n | None = None,\n data_collator: DataCollator | None = None,\n model_init: Callable[[], PreTrainedModel] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n peft_config: dict | None = None,\n compute_metrics: Callable[[EvalLoopOutput], dict] | None = None,\n model_adapter_name: str | None = None,\n ref_adapter_name: str | None = None,\n ):\n if type(args) is TrainingArguments:\n raise ValueError(\"Please use `KTOConfig` instead TrainingArguments.\")\n\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n\n if not isinstance(model, str) and ref_model is model:\n raise ValueError(\n \"`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the \"\n \"same as `model`, you must mass a copy of it, or `None` if you use peft.\"\n )\n\n # Model initialization\n if isinstance(model, str):\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n model = create_model_from_path(model, **model_init_kwargs)\n else:\n if args.model_init_kwargs is not None:\n logger.warning(\n \"You passed `model_init_kwargs` to the KTOConfig, but your model is already instantiated. \"\n \"The `model_init_kwargs` will be ignored.\"\n )\n\n # Reference model initialization\n if isinstance(ref_model, str):\n ref_model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n ref_model_init_kwargs[\"device_map\"] = None\n ref_model = create_model_from_path(ref_model, **ref_model_init_kwargs)\n\n # Initialize this variable to False. This helps tracking the case when `peft_module_casting_to_bf16`\n # has been called in order to properly call autocast if needed.\n self._peft_has_been_casted_to_bf16 = False\n\n if not is_peft_available() and peft_config is not None:\n raise ValueError(\n \"PEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it with `pip install peft` to use the PEFT models\"\n )\n elif is_peft_available() and peft_config is not None:\n if isinstance(model, PeftModel):\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first \"\n \"merge and unload the existing adapter, save the resulting base model, and then pass that base \"\n \"model along with the new `peft_config` to the trainer.\"\n )\n\n if getattr(model, \"is_loaded_in_8bit\", False) or getattr(model, \"is_loaded_in_4bit\", False):\n _support_gc_kwargs = hasattr(\n args, \"gradient_checkpointing_kwargs\"\n ) and \"gradient_checkpointing_kwargs\" in list(\n inspect.signature(prepare_model_for_kbit_training).parameters\n )\n\n prepare_model_kwargs = {\"use_gradient_checkpointing\": args.gradient_checkpointing}\n\n if _support_gc_kwargs:\n prepare_model_kwargs[\"gradient_checkpointing_kwargs\"] = args.gradient_checkpointing_kwargs\n\n model = prepare_model_for_kbit_training(model, **prepare_model_kwargs)\n elif args.gradient_checkpointing:\n # For backward compatibility with older versions of transformers\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n # get peft model with the given config\n model = get_peft_model(model, peft_config)\n if args.bf16 and getattr(model, \"is_loaded_in_4bit\", False):\n peft_module_casting_to_bf16(model)\n # If args.bf16 we need to explicitly call `generate` with torch amp autocast context manager\n self._peft_has_been_casted_to_bf16 = True\n\n # For models that use gradient_checkpointing, we need to attach a hook that enables input\n # to explicitly have `requires_grad=True`, otherwise training will either silently\n # fail or completely fail.\n elif args.gradient_checkpointing:\n # For backward compatibility with older versions of transformers\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n if args.generate_during_eval and not (is_wandb_available() or is_comet_available()):\n raise ValueError(\n \"`generate_during_eval=True` requires Weights and Biases or Comet to be installed.\"\n \" Please install `wandb` or `comet-ml` to resolve.\"\n )\n\n # KTO only supports causal language models, not encoder-decoder models\n if model is not None and hasattr(model.config, \"is_encoder_decoder\") and model.config.is_encoder_decoder:\n raise ValueError(\n \"KTO only supports causal language models. Encoder-decoder models are not supported. \"\n \"Please use a causal LM (e.g., GPT, Llama, Mistral) instead of an encoder-decoder model (e.g., T5, BART).\"\n )\n\n self.is_peft_model = is_peft_available() and isinstance(model, PeftModel)\n self.model_adapter_name = model_adapter_name\n self.ref_adapter_name = ref_adapter_name\n\n if ref_model:\n self.ref_model = ref_model\n elif self.is_peft_model or args.precompute_ref_log_probs:\n # The `model` with adapters turned off will be used as the reference model\n self.ref_model = None\n else:\n self.ref_model = create_reference_model(model)\n\n if processing_class is None:\n raise ValueError(\n \"max_length or a processing_class must be specified when using the default DPODataCollatorWithPadding\"\n )\n if args.max_length is None:\n logger.warning(\n \"When using DPODataCollatorWithPadding, you should set `max_length` in the KTOTrainer's init\"\n \" it will be set to `512` by default, but you should do it yourself in the future.\",\n )\n max_length = 512\n if args.max_length is not None:\n max_length = args.max_length\n\n if data_collator is None:\n data_collator = DPODataCollatorWithPadding(\n pad_token_id=processing_class.pad_token_id,\n )\n\n if args.remove_unused_columns:\n args.remove_unused_columns = False\n # warn users\n logger.warning(\n \"When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your KTOConfig\"\n \" we have set it for you, but you should do it yourself in the future.\",\n )\n\n self.use_dpo_data_collator = True\n else:\n self.use_dpo_data_collator = False\n\n # Disable dropout in the model and reference model\n if args.disable_dropout:\n disable_dropout_in_model(model)\n if self.ref_model is not None:\n disable_dropout_in_model(self.ref_model)\n\n self.loss_type = args.loss_type\n self.max_length = max_length\n self.generate_during_eval = args.generate_during_eval\n self.processing_class = processing_class\n self.precompute_ref_log_probs = args.precompute_ref_log_probs\n\n # Not all losses require a KL calculation\n self.calculate_KL = True\n if self.loss_type in [\"apo_zero_unpaired\"]:\n self.calculate_KL = False\n\n # Since ref_logs are precomputed on the first call to get_train/eval_dataloader\n # keep track of first called to avoid computation of future calls\n self._precomputed_train_ref_log_probs = False\n self._precomputed_eval_ref_log_probs = False\n\n # metric\n self._stored_metrics = defaultdict(lambda: defaultdict(list))\n\n # KTO parameter\n self.beta = args.beta\n self.desirable_weight = args.desirable_weight\n self.undesirable_weight = args.undesirable_weight\n self.aux_loss_enabled = getattr(model.config, \"output_router_logits\", False)\n self.aux_loss_coef = getattr(model.config, \"router_aux_loss_coef\", 0.0)\n if self.aux_loss_enabled and self.aux_loss_coef == 0.0:\n logger.warning(\n \"You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to \"\n \"`0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value \"\n \"greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary \"\n \"loss.\",\n )\n\n # Compute that only on the main process for faster data processing.\n # see: https://github.com/huggingface/trl/pull/1255\n with PartialState().main_process_first():\n # Extract the prompt if needed\n train_dataset = train_dataset.map(\n maybe_extract_prompt, num_proc=args.dataset_num_proc, desc=\"Extracting prompt from train dataset\"\n )\n # Unpair the dataset if needed\n train_dataset = maybe_unpair_preference_dataset(\n train_dataset, args.dataset_num_proc, desc=\"Unpairing train dataset\"\n )\n # Apply the chat template if needed\n train_dataset = train_dataset.map(\n maybe_apply_chat_template,\n fn_kwargs={\"tokenizer\": processing_class},\n num_proc=args.dataset_num_proc,\n desc=\"Applying chat template to train dataset\",\n )\n if eval_dataset is not None:\n eval_dataset = eval_dataset.map(\n maybe_extract_prompt, num_proc=args.dataset_num_proc, desc=\"Extracting prompt from eval dataset\"\n )\n eval_dataset = maybe_unpair_preference_dataset(\n eval_dataset, args.dataset_num_proc, desc=\"Unpairing eval dataset\"\n )\n eval_dataset = eval_dataset.map(\n maybe_apply_chat_template,\n fn_kwargs={\"tokenizer\": processing_class},\n num_proc=args.dataset_num_proc,\n desc=\"Applying chat template to eval dataset\",\n )\n\n # Tokenize and prepare the training datasets\n train_dataset = train_dataset.map(\n _tokenize,\n batched=True,\n fn_kwargs={\"tokenizer\": self.processing_class},\n num_proc=args.dataset_num_proc,\n desc=\"Tokenizing train dataset\",\n )\n\n fn_kwargs = {\n \"prefix\": \"\",\n \"tokenizer\": self.processing_class,\n \"max_length\": self.max_length,\n }\n\n train_dataset = train_dataset.map(\n _process_tokens,\n fn_kwargs=fn_kwargs,\n num_proc=args.dataset_num_proc,\n desc=\"Processing tokenized train dataset\",\n )\n\n # Tokenize and prepare the eval datasets\n if eval_dataset is not None:\n eval_dataset = eval_dataset.map(\n _tokenize,\n fn_kwargs={\"tokenizer\": self.processing_class},\n batched=True,\n num_proc=args.dataset_num_proc,\n desc=\"Tokenizing eval dataset\",\n )\n\n eval_dataset = eval_dataset.map(\n _process_tokens,\n fn_kwargs=fn_kwargs,\n num_proc=args.dataset_num_proc,\n desc=\"Processing tokenized eval dataset\",\n )\n\n # Get KL datasets if needed\n if self.calculate_KL:\n if args.per_device_train_batch_size <= 1:\n raise ValueError(\n \"Actual (not effective) batch size must be > 1. KTO will not work properly because the KL term will be equivalent to the implied reward.\"\n )\n\n # create pairs for estimating the KL term by flipping the matched pairs in each batch of size total_batch_size\n # i.e., (x_1, y_1), ..., (x_n, y_n) --> (x_1, y_n), ..., (x_n, y_1) = (x'_1, y'_1), ..., (x'_n, y'_n)\n train_kl_dataset = train_dataset.map(\n _get_kl_dataset,\n batched=True,\n batch_size=args.per_device_train_batch_size,\n num_proc=args.dataset_num_proc,\n desc=\"Extracting KL train dataset\",\n )\n\n fn_kwargs[\"prefix\"] = \"KL_\"\n train_kl_dataset = train_kl_dataset.map(\n _process_tokens,\n fn_kwargs=fn_kwargs,\n num_proc=args.dataset_num_proc,\n remove_columns=[c for c in train_kl_dataset.column_names if c in train_dataset.column_names],\n desc=\"Processing tokenized train KL dataset\",\n )\n\n # merge the datasets\n train_dataset = concatenate_datasets([train_dataset, train_kl_dataset], axis=1)\n\n if eval_dataset is not None:\n # Get KL dataset\n eval_kl_dataset = eval_dataset.map(\n _get_kl_dataset,\n batched=True,\n batch_size=args.per_device_train_batch_size,\n num_proc=args.dataset_num_proc,\n desc=\"Extracting eval KL dataset\",\n )\n\n eval_kl_dataset = eval_kl_dataset.map(\n _process_tokens,\n fn_kwargs=fn_kwargs,\n num_proc=args.dataset_num_proc,\n remove_columns=[c for c in eval_kl_dataset.column_names if c in eval_dataset.column_names],\n desc=\"Processing tokenized eval KL dataset\",\n )\n\n # merge the datasets\n eval_dataset = concatenate_datasets([eval_dataset, eval_kl_dataset], axis=1)\n\n # calculate dataset desirability balance\n num_desirable = max(sum(train_dataset[\"label\"]), 1)\n num_undesirable = max(len(train_dataset[\"label\"]) - num_desirable, 1) # \"label\" is binary\n\n if num_desirable != num_undesirable:\n # The lower and upper bounds come from Eq. (8) of https://huggingface.co/papers/2402.01306\n des_weight_lower_bound = round((num_undesirable * self.undesirable_weight / num_desirable) * 1, 2)\n des_weight_upper_bound = round((num_undesirable * self.undesirable_weight / num_desirable) * 1.33, 2)\n und_weight_lower_bound = round((num_desirable * self.desirable_weight / num_undesirable) / 1.33, 2)\n und_weight_upper_bound = round((num_desirable * self.desirable_weight / num_undesirable) / 1, 2)\n\n des_weight_in_range = des_weight_lower_bound <= self.desirable_weight <= des_weight_upper_bound\n und_weight_in_range = und_weight_lower_bound <= self.undesirable_weight <= und_weight_upper_bound\n\n if not (des_weight_in_range or und_weight_in_range):\n logger.warning(\n \"You have different amounts of desirable/positive and undesirable/negative examples but the \"\n \"weights on the desirable and undesirable losses don't seem to be in an ideal range. Based \"\n f\"on your data, we recommend EITHER \"\n f\"desirable_weight in [{des_weight_lower_bound}, {des_weight_upper_bound}] or \"\n f\"undesirable_weight in [{und_weight_lower_bound}, {und_weight_upper_bound}] (but NOT BOTH). \"\n \"See the documentation on how to optimally set these weights.\",\n )\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n model_init=model_init,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n # Gradient accumulation requires scaled loss. Normally, loss scaling in the parent class depends on whether the\n # model accepts loss-related kwargs. Since we compute our own loss, this check is irrelevant. We set\n # self.model_accepts_loss_kwargs to False to enable scaling.\n self.model_accepts_loss_kwargs = False\n\n # Add tags for models that have been loaded with the correct transformers version\n if hasattr(self.model, \"add_model_tags\"):\n self.model.add_model_tags(self._tag_names)\n\n if not hasattr(self, \"accelerator\"):\n raise AttributeError(\n \"Your `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.\"\n )\n\n # Deepspeed Zero-3 does not support precompute_ref_log_probs\n if self.is_deepspeed_enabled:\n if self.accelerator.state.deepspeed_plugin.zero_stage == 3 and self.precompute_ref_log_probs:\n raise ValueError(\n \"You cannot use `precompute_ref_log_probs=True` with Deepspeed ZeRO-3. Please set `precompute_ref_log_probs=False`.\"\n )\n\n if self.ref_model is None:\n if not (self.is_peft_model or self.precompute_ref_log_probs):\n raise ValueError(\n \"No reference model and model is not a Peft model. Try setting `precompute_ref_log_probs=True`\"\n )\n else:\n if self.is_deepspeed_enabled:\n self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)\n else:\n self.ref_model = self.accelerator.prepare_model(self.ref_model, evaluation_mode=True)\n\n # Import Liger kernel if enabled\n if self.args.use_liger_kernel:\n if not is_liger_kernel_available():\n raise ImportError(\n \"You set `use_liger_kernel=True` but the liger kernel is not available. \"\n \"Please install liger-kernel first: `pip install liger-kernel`\"\n )\n if self.loss_type in [\"apo_zero_unpaired\"]:\n raise ValueError(\n \"You cannot set `loss_type='apo_zero_unpaired'` with liger-kernel.\"\n \"Only KTO loss is supported with liger-kernel.\"\n )\n if self.precompute_ref_log_probs:\n raise ValueError(\n \"You cannot use `precompute_ref_log_probs=True` with liger kernel. Please set \"\n \"`precompute_ref_log_probs=False`.\"\n )\n if self.is_peft_model or self.ref_adapter_name is not None:\n raise ValueError(\n \"You cannot use `use_liger_kernel=True` with Peft models. Please set `use_liger_kernel=False`.\"\n )\n self.kto_loss_fn = LigerFusedLinearKTOLoss(beta=self.beta, use_ref_model=(self.ref_model is not None))\n\n @contextmanager\n def null_ref_context(self):\n \"\"\"Context manager for handling null reference model (that is, peft adapter manipulation).\"\"\"\n with (\n self.accelerator.unwrap_model(self.model).disable_adapter()\n if self.is_peft_model and not self.ref_adapter_name\n else nullcontext()\n ):\n if self.ref_adapter_name:\n self.model.set_adapter(self.ref_adapter_name)\n yield\n if self.ref_adapter_name:\n self.model.set_adapter(self.model_adapter_name or \"default\")\n\n def get_train_dataloader(self) -> DataLoader:\n \"\"\"\n Returns the training [`~torch.utils.data.DataLoader`].\n\n Subclass of transformers.src.transformers.trainer.get_train_dataloader to precompute `ref_log_probs`.\n \"\"\"\n\n if self.precompute_ref_log_probs and not self._precomputed_train_ref_log_probs:\n dataloader_params = {\n \"batch_size\": self.args.per_device_train_batch_size,\n \"collate_fn\": self.data_collator,\n \"num_workers\": self.args.dataloader_num_workers,\n \"pin_memory\": self.args.dataloader_pin_memory,\n \"shuffle\": False,\n }\n\n # prepare dataloader\n data_loader = self.accelerator.prepare(DataLoader(self.train_dataset, **dataloader_params))\n reference_completion_logps = []\n reference_KL_logps = []\n\n for padded_batch in tqdm(iterable=data_loader, desc=\"Train dataset reference log probs\"):\n reference_completion_logp, reference_KL_logp = self.compute_reference_log_probs(padded_batch)\n\n reference_completion_logp = self.accelerator.gather_for_metrics(reference_completion_logp)\n reference_completion_logps.append(reference_completion_logp.cpu())\n\n if self.calculate_KL:\n reference_KL_logp = self.accelerator.gather_for_metrics(reference_KL_logp)\n reference_KL_logps.append(reference_KL_logp.cpu())\n\n self.train_dataset = self.train_dataset.add_column(\n name=\"reference_logps\", column=torch.cat(reference_completion_logps).float().numpy()\n )\n\n if self.calculate_KL:\n self.train_dataset = self.train_dataset.add_column(\n name=\"reference_KL_logps\", column=torch.cat(reference_KL_logps).float().numpy()\n )\n\n self._precomputed_train_ref_log_probs = True\n\n return super().get_train_dataloader()\n\n def get_eval_dataloader(self, eval_dataset: Dataset | None = None) -> DataLoader:\n \"\"\"\n Returns the evaluation [`~torch.utils.data.DataLoader`].\n\n Subclass of transformers.src.transformers.trainer.get_eval_dataloader to precompute `ref_log_probs`.\n\n Args:\n eval_dataset (`torch.utils.data.Dataset`, *optional*):\n If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted\n by the `model.forward()` method are automatically removed. It must implement `__len__`.\n \"\"\"\n if eval_dataset is None and self.eval_dataset is None:\n raise ValueError(\"Trainer: evaluation requires an eval_dataset.\")\n eval_dataset = eval_dataset if eval_dataset is not None else self.eval_dataset\n\n if self.precompute_ref_log_probs and not self._precomputed_eval_ref_log_probs:\n dataloader_params = {\n \"batch_size\": self.args.per_device_eval_batch_size,\n \"collate_fn\": self.data_collator,\n \"num_workers\": self.args.dataloader_num_workers,\n \"pin_memory\": self.args.dataloader_pin_memory,\n \"shuffle\": False,\n }\n\n # prepare dataloader\n data_loader = self.accelerator.prepare(DataLoader(eval_dataset, **dataloader_params))\n\n reference_completion_logps = []\n reference_KL_logps = []\n\n for padded_batch in tqdm(iterable=data_loader, desc=\"Eval dataset reference log probs\"):\n reference_completion_logp, reference_KL_logp = self.compute_reference_log_probs(padded_batch)\n\n reference_completion_logp = self.accelerator.gather_for_metrics(reference_completion_logp)\n reference_completion_logps.append(reference_completion_logp.cpu())\n\n if self.calculate_KL:\n reference_KL_logp = self.accelerator.gather_for_metrics(reference_KL_logp)\n reference_KL_logps.append(reference_KL_logp.cpu())\n\n eval_dataset = eval_dataset.add_column(\n name=\"reference_logps\", column=torch.cat(reference_completion_logps).float().numpy()\n )\n if self.calculate_KL:\n eval_dataset = eval_dataset.add_column(\n name=\"reference_KL_logps\", column=torch.cat(reference_KL_logps).float().numpy()\n )\n\n # Save calculated reference_chosen_logps and reference_rejected_logps to the eval_dataset for subsequent runs\n if self.eval_dataset is not None:\n self.eval_dataset = eval_dataset\n self._precomputed_eval_ref_log_probs = True\n\n return super().get_eval_dataloader(eval_dataset=eval_dataset)\n\n def compute_reference_log_probs(self, padded_batch: dict) -> dict:\n \"\"\"Computes log probabilities of the reference model for a single padded batch of a KTO specific dataset.\"\"\"\n with torch.no_grad():\n if self.ref_model is None:\n with self.null_ref_context():\n completion_logits = self.model(\n padded_batch[\"completion_input_ids\"],\n attention_mask=padded_batch[\"completion_attention_mask\"],\n ).logits\n\n if self.calculate_KL:\n KL_logits = self.model(\n padded_batch[\"KL_completion_input_ids\"],\n attention_mask=padded_batch[\"KL_completion_attention_mask\"],\n ).logits\n else:\n completion_logits = self.ref_model(\n padded_batch[\"completion_input_ids\"], attention_mask=padded_batch[\"completion_attention_mask\"]\n ).logits\n\n if self.calculate_KL:\n KL_logits = self.ref_model(\n padded_batch[\"KL_completion_input_ids\"],\n attention_mask=padded_batch[\"KL_completion_attention_mask\"],\n ).logits\n\n completion_logps = self.get_batch_logps(\n completion_logits,\n padded_batch[\"completion_labels\"],\n average_log_prob=False,\n )\n\n if self.calculate_KL:\n KL_logps = self.get_batch_logps(\n KL_logits,\n padded_batch[\"KL_completion_labels\"],\n average_log_prob=False,\n )\n else:\n KL_logps = None\n\n return completion_logps, KL_logps\n\n @staticmethod\n def get_batch_logps(\n logits: torch.FloatTensor,\n labels: torch.LongTensor,\n average_log_prob: bool = False,\n ) -> torch.FloatTensor:\n \"\"\"Compute the log probabilities of the given labels under the given logits.\n\n Args:\n logits:\n Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)\n labels:\n Labels for which to compute the log probabilities. Label tokens with a value of `-100` are ignored.\n Shape: (batch_size, sequence_length)\n average_log_prob:\n If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the\n log probabilities of the (non-masked) tokens.\n\n Returns:\n A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the\n given logits.\n \"\"\"\n if logits.shape[:-1] != labels.shape:\n raise ValueError(\"Logits (batch and sequence length dim) and labels must have the same shape.\")\n\n # For causal LM, shift labels and logits by one position\n labels = labels[:, 1:].clone()\n logits = logits[:, :-1, :]\n\n loss_mask = labels != -100\n\n # dummy token; we'll ignore the losses on these tokens later\n labels[labels == -100] = 0\n\n per_token_logps = selective_log_softmax(logits, labels)\n\n if average_log_prob:\n return (per_token_logps * loss_mask).sum(-1) / loss_mask.sum(-1)\n else:\n return (per_token_logps * loss_mask).sum(-1)\n\n def forward(\n self, model: nn.Module, batch: dict[str, list | torch.LongTensor]\n ) -> tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n KL_logps = self._compute_kl_logps(model, batch)\n\n model_kwargs = {}\n if self.aux_loss_enabled:\n model_kwargs[\"output_router_logits\"] = True\n\n outputs = model(\n batch[\"completion_input_ids\"],\n attention_mask=batch[\"completion_attention_mask\"],\n **model_kwargs,\n )\n completion_logits = outputs.logits\n\n completion_logps = self.get_batch_logps(\n completion_logits,\n batch[\"completion_labels\"],\n average_log_prob=False,\n )\n\n if completion_logps.shape[0] != len(batch[\"label\"]):\n raise ValueError(\n \"There is a mismatch between the number of examples in this batch and the number of \"\n \"examples for which an output sequence was predicted.\"\n )\n\n # Use torch.nonzero for efficient tensor index selection\n device = completion_logits.device\n labels = torch.as_tensor(batch[\"label\"], dtype=torch.bool, device=device)\n chosen_idx = torch.nonzero(labels, as_tuple=False).view(-1)\n rejected_idx = torch.nonzero(~labels, as_tuple=False).view(-1)\n\n # Use index_select for efficient CUDA operations\n chosen_logps = completion_logps.index_select(0, chosen_idx)\n rejected_logps = completion_logps.index_select(0, rejected_idx)\n\n chosen_logits = completion_logits.index_select(0, chosen_idx)\n rejected_logits = completion_logits.index_select(0, rejected_idx)\n\n if self.aux_loss_enabled:\n return (chosen_logps, rejected_logps, chosen_logits, rejected_logits, KL_logps, outputs.aux_loss)\n else:\n return (chosen_logps, rejected_logps, chosen_logits, rejected_logits, KL_logps)\n\n def kto_loss(\n self,\n policy_chosen_logps: torch.FloatTensor,\n policy_rejected_logps: torch.FloatTensor,\n policy_KL_logps: torch.FloatTensor,\n reference_chosen_logps: torch.FloatTensor,\n reference_rejected_logps: torch.FloatTensor,\n reference_KL_logps: torch.FloatTensor,\n ) -> tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n \"\"\"Compute the KTO loss for a batch of policy and reference model log probabilities.\n\n Args:\n policy_chosen_logps:\n Log probabilities of the policy model for the chosen responses. Shape: (num(chosen) in batch_size,)\n policy_rejected_logps:\n Log probabilities of the policy model for the rejected responses. Shape: (num(rejected) in batch_size,)\n policy_KL_logps: Log probabilities of the policy model for the KL responses. Shape: (batch_size,)\n reference_chosen_logps:\n Log probabilities of the reference model for the chosen responses. Shape: (num(chosen) in batch_size,)\n reference_rejected_logps:\n Log probabilities of the reference model for the rejected responses. Shape: (num(rejected) in\n batch_size,)\n reference_KL_logps: Log probabilities of the reference model for the KL responses. Shape: (batch_size,)\n\n Returns:\n A tuple of four tensors: (losses, chosen_rewards, rejected_rewards, KL). The losses tensor contains the KTO\n loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for\n the chosen and rejected responses, respectively. The KL tensor contains the detached KL divergence estimate\n between the policy and reference models.\n \"\"\"\n if self.calculate_KL:\n kl = (policy_KL_logps - reference_KL_logps).mean().detach()\n kl = self.accelerator.gather_for_metrics(kl).mean().clamp(min=0)\n else:\n kl = torch.zeros(1).to(policy_chosen_logps.device)\n\n # Chosen losses\n if policy_chosen_logps.shape[0] != 0 or reference_chosen_logps.shape[0] != 0:\n chosen_logratios = policy_chosen_logps - reference_chosen_logps\n\n if self.loss_type == \"kto\":\n # Eqn (7) of the KTO paper (https://huggingface.co/papers/2402.01306)\n chosen_losses = 1 - F.sigmoid(self.beta * (chosen_logratios - kl))\n elif self.loss_type == \"apo_zero_unpaired\":\n # Unpaired variant of Eqn (7) of the APO paper (https://huggingface.co/papers/2408.06266)\n # Use this loss when you believe the chosen outputs are better than your model's default output\n chosen_losses = 1 - F.sigmoid(self.beta * chosen_logratios)\n\n chosen_rewards = self.beta * chosen_logratios.detach()\n\n else:\n # lists can't be empty -- if they are, then accelerate.gather will hang\n chosen_losses = torch.Tensor([]).to(self.accelerator.device)\n chosen_rewards = torch.Tensor([]).to(self.accelerator.device)\n\n # Rejected losses\n if policy_rejected_logps.shape[0] != 0 or reference_rejected_logps.shape[0] != 0:\n rejected_logratios = policy_rejected_logps - reference_rejected_logps\n\n if self.loss_type == \"kto\":\n rejected_losses = 1 - F.sigmoid(self.beta * (kl - rejected_logratios))\n elif self.loss_type == \"apo_zero_unpaired\":\n rejected_losses = F.sigmoid(self.beta * rejected_logratios)\n\n rejected_rewards = self.beta * rejected_logratios.detach()\n else:\n # lists can't be empty -- if they are, then accelerate.gather will hang\n rejected_losses = torch.Tensor([]).to(self.accelerator.device)\n rejected_rewards = torch.Tensor([]).to(self.accelerator.device)\n\n losses = torch.cat(\n (self.desirable_weight * chosen_losses, self.undesirable_weight * rejected_losses),\n 0,\n )\n\n return losses, chosen_rewards, rejected_rewards, kl\n\n def _compute_kl_logps(self, model, batch):\n \"\"\"Compute KL log probabilities for a given batch.\"\"\"\n KL_logps = None\n if self.calculate_KL:\n KL_model_kwargs = {\n \"input_ids\": batch[\"KL_completion_input_ids\"],\n \"attention_mask\": batch[\"KL_completion_attention_mask\"],\n }\n\n with torch.no_grad():\n KL_logits = model(**KL_model_kwargs).logits\n\n KL_logps = self.get_batch_logps(\n KL_logits,\n batch[\"KL_completion_labels\"],\n average_log_prob=False,\n )\n return KL_logps\n\n def _compute_loss_liger(self, model, batch):\n \"\"\"\n Compute the KTO loss using the Liger-Kernel's LigerFusedLinearKTOLoss.\n\n Args:\n model:\n The policy model used for generating log probabilities and outputs. It could be an encoder-decoder\n model or a regular language model.\n batch: A dictionary containing the input data and labels for the batch.\n\n Returns:\n A dictionary containing the following keys:\n - \"loss\": The computed KTO loss for the batch.\n - \"chosen_logits_sum\": Sum of the logits for the chosen responses from the policy model.\n - \"rejected_logits_sum\": Sum of the logits for the rejected responses from the policy model.\n - \"chosen_logps\": Log probabilities of the chosen responses from the policy model.\n - \"rejected_logps\": Log probabilities of the rejected responses from the policy model.\n - \"chosen_rewards\": Rewards for the chosen responses.\n - \"rejected_rewards\": Rewards for the rejected responses.\n - \"kl\": The KL divergence between the policy and reference models (detached).\n\n If auxiliary loss is enabled, the dictionary will also include:\n - \"aux_loss\": The auxiliary loss from the model outputs.\n \"\"\"\n policy_KL_logps = self._compute_kl_logps(model, batch)\n reference_KL_logps = self._compute_kl_logps(self.ref_model, batch)\n if self.calculate_KL:\n kl = (policy_KL_logps - reference_KL_logps).mean().detach()\n kl = self.accelerator.gather_for_metrics(kl).mean().clamp(min=0)\n else:\n kl = torch.zeros(1).to(self.accelerator.device)\n\n model_kwargs = {}\n if self.aux_loss_enabled:\n model_kwargs[\"output_router_logits\"] = True\n\n # skip the lm head and get the last hidden state\n base_model = model.get_decoder()\n outputs = base_model(\n batch[\"completion_input_ids\"],\n attention_mask=batch[\"completion_attention_mask\"],\n use_cache=False,\n **model_kwargs,\n )\n\n # reference model\n ref_base_model = self.ref_model.get_decoder()\n ref_outputs = ref_base_model(\n batch[\"completion_input_ids\"],\n attention_mask=batch[\"completion_attention_mask\"],\n use_cache=False,\n **model_kwargs,\n )\n lm_head = model.get_output_embeddings()\n ref_lm_head = self.ref_model.get_output_embeddings()\n\n (\n loss,\n (\n chosen_logps_sum,\n rejected_logps_sum,\n chosen_logits_sum,\n rejected_logits_sum,\n chosen_rewards_sum,\n rejected_rewards_sum,\n ),\n ) = self.kto_loss_fn(\n _input=outputs.last_hidden_state[:, :-1],\n lin_weight=lm_head.weight,\n target=batch[\"completion_labels\"][:, 1:],\n bias=lm_head.bias if hasattr(lm_head, \"bias\") else None,\n preference_labels=torch.tensor(batch[\"label\"], dtype=torch.bool).to(self.accelerator.device),\n ref_input=ref_outputs.last_hidden_state[:, :-1],\n ref_weight=ref_lm_head.weight,\n ref_bias=ref_lm_head.bias if hasattr(lm_head, \"bias\") else None,\n kl=kl,\n )\n\n output = {\n \"loss\": loss,\n \"chosen_logits_sum\": chosen_logits_sum,\n \"rejected_logits_sum\": rejected_logits_sum,\n \"chosen_logps_sum\": chosen_logps_sum,\n \"rejected_logps_sum\": rejected_logps_sum,\n \"chosen_rewards_sum\": chosen_rewards_sum,\n \"rejected_rewards_sum\": rejected_rewards_sum,\n \"kl\": kl,\n }\n if self.aux_loss_enabled:\n output[\"aux_loss\"] = outputs.aux_loss\n\n return output\n\n def get_batch_loss_metrics(\n self,\n model,\n batch: dict[str, list | torch.LongTensor],\n ):\n \"\"\"Compute the KTO loss and other metrics for the given batch of inputs for train or test.\"\"\"\n metrics = {}\n batch = {k: (v.to(self.accelerator.device) if isinstance(v, torch.Tensor) else v) for k, v in batch.items()}\n\n labels = torch.tensor(batch[\"label\"])\n num_chosen = labels.sum().to(self.accelerator.device)\n num_rejected = (len(labels) - num_chosen).to(self.accelerator.device)\n\n if self.args.use_liger_kernel:\n model_output = self._compute_loss_liger(model, batch)\n losses = model_output[\"loss\"]\n policy_chosen_logits = model_output[\"chosen_logits_sum\"]\n policy_rejected_logits = model_output[\"rejected_logits_sum\"]\n policy_chosen_logps = model_output[\"chosen_logps_sum\"]\n policy_rejected_logps = model_output[\"rejected_logps_sum\"]\n chosen_rewards = model_output[\"chosen_rewards_sum\"]\n rejected_rewards = model_output[\"rejected_rewards_sum\"]\n kl = model_output[\"kl\"]\n if self.aux_loss_enabled:\n aux_loss = model_output[\"aux_loss\"]\n else:\n forward_output = self.forward(model, batch)\n (\n policy_chosen_logps,\n policy_rejected_logps,\n policy_chosen_logits,\n policy_rejected_logits,\n policy_KL_logps,\n ) = forward_output[:5]\n if self.aux_loss_enabled:\n aux_loss = forward_output[5]\n\n # if reference_logps in batch use them, otherwise use the reference model\n if \"reference_logps\" in batch:\n # Convert Python lists to tensor indices for efficient CUDA operations\n device = batch[\"reference_logps\"].device\n labels = torch.as_tensor(batch[\"label\"], dtype=torch.bool, device=device)\n chosen_idx = torch.nonzero(labels, as_tuple=False).view(-1)\n rejected_idx = torch.nonzero(~labels, as_tuple=False).view(-1)\n\n # Use index_select for efficient CUDA operations\n reference_chosen_logps = batch[\"reference_logps\"].index_select(0, chosen_idx)\n reference_rejected_logps = batch[\"reference_logps\"].index_select(0, rejected_idx)\n if self.calculate_KL:\n reference_KL_logps = batch[\"reference_KL_logps\"]\n else:\n reference_KL_logps = None\n else:\n with torch.no_grad():\n if self.ref_model is None:\n with self.null_ref_context():\n (\n reference_chosen_logps,\n reference_rejected_logps,\n _,\n _,\n reference_KL_logps,\n ) = self.forward(self.model, batch)[:5]\n else:\n (\n reference_chosen_logps,\n reference_rejected_logps,\n _,\n _,\n reference_KL_logps,\n ) = self.forward(self.ref_model, batch)[:5]\n\n losses, chosen_rewards, rejected_rewards, kl = self.kto_loss(\n policy_chosen_logps,\n policy_rejected_logps,\n policy_KL_logps,\n reference_chosen_logps,\n reference_rejected_logps,\n reference_KL_logps,\n )\n\n metrics[\"kl\"] = kl.item()\n\n all_num_chosen = self.accelerator.gather_for_metrics(num_chosen).sum().item()\n all_num_rejected = self.accelerator.gather_for_metrics(num_rejected).sum().item()\n\n if all_num_chosen > 0:\n metrics[\"rewards/chosen_sum\"] = (\n self.accelerator.gather_for_metrics(chosen_rewards.nansum()).nansum().item()\n )\n metrics[\"logps/chosen_sum\"] = (\n self.accelerator.gather_for_metrics(policy_chosen_logps.nansum()).nansum().item()\n )\n metrics[\"logits/chosen_sum\"] = (\n self.accelerator.gather_for_metrics(policy_chosen_logits.nansum()).nansum().item()\n )\n metrics[\"count/chosen\"] = all_num_chosen\n\n if all_num_rejected > 0:\n metrics[\"rewards/rejected_sum\"] = (\n self.accelerator.gather_for_metrics(rejected_rewards.nansum()).nansum().item()\n )\n metrics[\"logps/rejected_sum\"] = (\n self.accelerator.gather_for_metrics(policy_rejected_logps.nansum()).nansum().item()\n )\n metrics[\"logits/rejected_sum\"] = (\n self.accelerator.gather_for_metrics(policy_rejected_logits.nansum()).nansum().item()\n )\n metrics[\"count/rejected\"] = all_num_rejected\n\n loss = losses.nanmean()\n if self.aux_loss_enabled:\n loss += self.aux_loss_coef * aux_loss\n\n return loss, metrics\n\n def compute_loss(\n self,\n model: PreTrainedModel | nn.Module,\n inputs: dict[str, torch.Tensor | Any],\n return_outputs=False,\n num_items_in_batch=None,\n ) -> torch.Tensor | tuple[torch.Tensor, dict[str, torch.Tensor]]:\n compute_loss_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n\n with compute_loss_context_manager:\n loss, metrics = self.get_batch_loss_metrics(model, inputs)\n\n # Make sure to move the loss to the device the original accumulating loss is at back in the `Trainer` class:\n loss = loss.to(self.args.device)\n # force log the metrics\n if self.accelerator.is_main_process:\n self.store_metrics(metrics, train_eval=\"train\")\n\n if return_outputs:\n return (loss, metrics)\n return loss\n\n def store_metrics(self, metrics: dict[str, float], train_eval: Literal[\"train\", \"eval\"] = \"train\") -> None:\n for key, value in metrics.items():\n self._stored_metrics[train_eval][key].append(value)\n\n def _get_train_sampler(self, dataset: Dataset | None = None) -> torch.utils.data.Sampler | None:\n if dataset is None:\n dataset = self.train_dataset\n if dataset is None or not has_length(dataset):\n return None\n return SequentialSampler(dataset)\n\n def generate_from_model_and_ref(self, model, batch: dict[str, torch.LongTensor]) -> tuple[str, str]:\n \"\"\"Generate samples from the model and reference model for the given batch of inputs.\"\"\"\n\n # If one uses `generate_during_eval` with peft + bf16, we need to explicitly call generate with\n # the torch amp context manager as some hidden states are silently casted to full precision.\n generate_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n\n with generate_context_manager:\n policy_output = model.generate(\n input_ids=batch[\"prompt_input_ids\"],\n attention_mask=batch[\"prompt_attention_mask\"],\n max_length=self.max_length,\n do_sample=True,\n pad_token_id=self.processing_class.pad_token_id,\n )\n\n # if reference_output in batch use that otherwise use the reference model\n if \"reference_output\" in batch:\n reference_output = batch[\"reference_output\"]\n else:\n if self.ref_model is None:\n with self.null_ref_context():\n reference_output = self.model.generate(\n input_ids=batch[\"prompt_input_ids\"],\n attention_mask=batch[\"prompt_attention_mask\"],\n max_length=self.max_length,\n do_sample=True,\n pad_token_id=self.processing_class.pad_token_id,\n )\n else:\n reference_output = self.ref_model.generate(\n input_ids=batch[\"prompt_input_ids\"],\n attention_mask=batch[\"prompt_attention_mask\"],\n max_length=self.max_length,\n do_sample=True,\n pad_token_id=self.processing_class.pad_token_id,\n )\n\n policy_output = pad_to_length(policy_output, self.max_length, self.processing_class.pad_token_id)\n policy_output_decoded = self.processing_class.batch_decode(policy_output, skip_special_tokens=True)\n\n reference_output = pad_to_length(reference_output, self.max_length, self.processing_class.pad_token_id)\n reference_output_decoded = self.processing_class.batch_decode(reference_output, skip_special_tokens=True)\n\n return policy_output_decoded, reference_output_decoded\n\n def prediction_step(\n self,\n model: PreTrainedModel | nn.Module,\n inputs: dict[str, torch.Tensor | Any],\n prediction_loss_only: bool,\n ignore_keys: list[str] | None = None,\n ):\n if ignore_keys is None:\n if hasattr(model, \"config\"):\n ignore_keys = getattr(model.config, \"keys_to_ignore_at_inference\", [])\n else:\n ignore_keys = []\n\n prediction_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n with torch.no_grad(), prediction_context_manager:\n loss, metrics = self.get_batch_loss_metrics(model, inputs)\n\n # force log the metrics\n if self.accelerator.is_main_process:\n self.store_metrics(metrics, train_eval=\"eval\")\n\n if prediction_loss_only:\n return (loss.detach(), None, None)\n\n # logits for the chosen and rejected samples from model\n logits_dict = {}\n if \"logits/chosen_sum\" in metrics:\n logits_dict[\"eval_logits/chosen\"] = metrics[\"logits/chosen_sum\"]\n if \"logits/rejected_sum\" in metrics:\n logits_dict[\"eval_logits/rejected\"] = metrics[\"logits/rejected_sum\"]\n logits = [v for k, v in logits_dict.items() if k not in ignore_keys]\n logits = torch.tensor(logits, device=self.accelerator.device)\n labels = torch.zeros(logits.shape[0], device=self.accelerator.device)\n\n return (loss.detach(), logits, labels)\n\n def evaluation_loop(\n self,\n dataloader: DataLoader,\n description: str,\n prediction_loss_only: bool | None = None,\n ignore_keys: list[str] | None = None,\n metric_key_prefix: str = \"eval\",\n ) -> EvalLoopOutput:\n \"\"\"\n Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by\n `Trainer.evaluate()` and `Trainer.predict()`.\n\n Works both with or without labels.\n \"\"\"\n\n # Sample and save to game log if requested (for one batch to save time)\n if self.generate_during_eval:\n # Generate random indices within the range of the total number of samples\n num_samples = len(dataloader.dataset)\n random_indices = random.sample(range(num_samples), k=self.args.eval_batch_size)\n\n # Use dataloader.dataset.select to get the random batch without iterating over the DataLoader\n random_batch_dataset = dataloader.dataset.select(random_indices)\n random_batch = self.data_collator(random_batch_dataset)\n random_batch = self._prepare_inputs(random_batch)\n\n target_labels = torch.tensor(random_batch[\"label\"], dtype=torch.bool, device=self.accelerator.device)\n target_indices = torch.where(~target_labels)[0]\n target_batch = {\n \"prompt_input_ids\": random_batch[\"prompt_input_ids\"][target_indices],\n \"prompt_attention_mask\": random_batch[\"prompt_attention_mask\"][target_indices],\n \"prompt\": itemgetter(*target_indices)(random_batch[\"prompt\"]),\n }\n policy_output_decoded, ref_output_decoded = self.generate_from_model_and_ref(self.model, target_batch)\n\n table = pd.DataFrame(\n columns=[\"Prompt\", \"Policy\", \"Ref Model\"],\n data=[\n [prompt, pol[len(prompt) :], ref[len(prompt) :]]\n for prompt, pol, ref in zip(\n target_batch[\"prompt\"], policy_output_decoded, ref_output_decoded, strict=True\n )\n ],\n )\n if \"wandb\" in self.args.report_to:\n wandb.log({\"game_log\": wandb.Table(data=table)})\n\n if \"comet_ml\" in self.args.report_to:\n log_table_to_comet_experiment(\n name=\"game_log.csv\",\n table=table,\n )\n\n # Base evaluation\n initial_output = super().evaluation_loop(\n dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix\n )\n\n return initial_output\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n \"\"\"\n Log `logs` on the various objects watching training, including stored metrics.\n\n Args:\n logs (`dict[str, float]`):\n The values to log.\n start_time (`float`, *optional*):\n Start time of the training.\n \"\"\"\n # logs either has 'loss' or 'eval_loss'\n train_eval = \"train\" if \"loss\" in logs else \"eval\"\n # train metrics should have no prefix, eval should have 'eval_'\n prefix = \"eval_\" if train_eval == \"eval\" else \"\"\n # accumulate average metrics from sums and lengths\n for split in [\"chosen\", \"rejected\"]:\n if f\"count/{split}\" in self._stored_metrics[train_eval]:\n count_sum = torch.Tensor(self._stored_metrics[train_eval][f\"count/{split}\"]).sum().item()\n for metric in [\"rewards\", \"logps\", \"logits\"]:\n logs[f\"{prefix}{metric}/{split}\"] = (\n torch.Tensor(self._stored_metrics[train_eval][f\"{metric}/{split}_sum\"]).sum().item()\n / count_sum\n )\n # delete obsolete metric\n del self._stored_metrics[train_eval][f\"{metric}/{split}_sum\"]\n del self._stored_metrics[train_eval][f\"count/{split}\"]\n # calculate reward margin\n if f\"{prefix}rewards/chosen\" in logs and f\"{prefix}rewards/rejected\" in logs:\n logs[f\"{prefix}rewards/margins\"] = logs[f\"{prefix}rewards/chosen\"] - logs[f\"{prefix}rewards/rejected\"]\n # Add averaged stored metrics to logs\n for key, metrics in self._stored_metrics[train_eval].items():\n logs[f\"{prefix}{key}\"] = torch.Tensor(metrics).mean().item()\n del self._stored_metrics[train_eval]\n return super().log(logs, start_time)\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/experimental/merge_model_callback.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport logging\nimport os\n\nimport torch\nfrom huggingface_hub import HfApi\nfrom transformers import TrainerCallback\n\nfrom ..import_utils import is_mergekit_available\nfrom ..trainer.utils import get_config_model_id\n\n\nif is_mergekit_available():\n from mergekit.config import MergeConfiguration\n from mergekit.merge import MergeOptions, run_merge\n\n\n# Logger for module-level logging\nlogger = logging.getLogger(__name__)\n\n\ndef upload_model_to_hf(folder_path: str, repo_id: str):\n api = HfApi()\n # Create the repository if it doesn't exist\n repo = api.create_repo(repo_id, repo_type=\"model\")\n\n # Upload the folder to the specified repository\n api.upload_folder(\n folder_path=folder_path,\n repo_id=repo.repo_id,\n repo_type=repo.repo_type,\n )\n\n\nclass MergeConfig:\n r\"\"\"\n Configuration class for merging two models using `mergekit`.\n\n This class provides a structured way to configure and generate merge configurations for various merge methods, such\n as `linear`, `ties`, `dare_ties`, and `slerp`.\n\n Args:\n method (`str`, *optional*, defaults to `\"linear\"`):\n Merge method to use. Supported methods include:\n\n - `\"linear\"`: Linearly combines two models with specified weights.\n - `\"ties\"`: Combines two models using the TIES method with density parameters.\n - `\"dare_ties\"`: A variant of TIES for domain adaptation.\n - `\"slerp\"`: Combines models using spherical linear interpolation.\n\n Note:\n\n For more details about the merge methods and how they are implemented, see the [MergeKit GitHub\n repository](https://github.com/arcee-ai/mergekit?tab=readme-ov-file#merge-methods).\n\n Attributes:\n method (`str`): The merge method to use.\n policy_model_path (`str` or `None`): Path to the policy model.\n target_model_path (`str` or `None`): Path to the target model.\n policy_model_weight (`float`): Weight for the policy model (for `linear` and `ties` methods).\n target_model_weight (`float`): Weight for the target model (for `linear` and `ties` methods).\n policy_model_density (`list[float]`): Density parameters for the policy model (for `ties` and `dare_ties`).\n target_model_density (`list[float]`): Density parameters for the target model (for `ties` and `dare_ties`).\n normalize (`float` or `None`): Normalization factor for the TIES method.\n t_values (`float` or `None`): Interpolation factor for the SLERP method.\n dtype (`str`): Data type to use for merging, e.g., `\"float16\"`.\n \"\"\"\n\n def __init__(self, method: str = \"linear\"):\n if not is_mergekit_available():\n raise ImportError(\"MergeConfig requires the `mergekit` extra. To install, run `pip install mergekit`.\")\n self.method = method\n self.policy_model_path = None\n self.target_model_path = None\n\n # Initialize relevant parameters based on the method\n if method == \"linear\":\n self.policy_model_weight = 0.5\n self.target_model_weight = 0.5\n self.dtype = \"float16\"\n elif method == \"ties\":\n self.policy_model_weight = 1.0\n self.policy_model_density = [1.0, 0.7, 0.1]\n self.target_model_weight = 1.0\n self.target_model_density = [1.0]\n self.normalize = 1.0\n self.dtype = \"float16\"\n elif method == \"dare_ties\":\n self.policy_model_weight = 1.0\n self.policy_model_density = [1.0, 0.7, 0.1]\n self.target_model_weight = 1.0\n self.target_model_density = [1.0]\n self.normalize = 1.0\n self.dtype = \"float16\"\n elif method == \"slerp\":\n self.t_values = 0.5\n self.dtype = \"float16\"\n else:\n raise ValueError(f\"Unsupported merge method: {method}\")\n\n def create_merge_config_linear(self) -> \"MergeConfiguration\":\n \"\"\"\n Creates a merge configuration for a linear merge of two models with specified weights.\n \"\"\"\n # Create the merge configuration dictionary\n merge_config_dict = {\n \"dtype\": self.dtype,\n \"merge_method\": \"linear\",\n \"models\": [\n {\"model\": self.policy_model_path, \"parameters\": {\"weight\": self.policy_model_weight}},\n {\"model\": self.target_model_path, \"parameters\": {\"weight\": self.target_model_weight}},\n ],\n }\n\n # Create the MergeConfiguration from the dictionary\n merge_config = MergeConfiguration.model_validate(merge_config_dict)\n\n return merge_config\n\n def create_merge_config_ties(self) -> \"MergeConfiguration\":\n \"\"\"\n Creates a merge configuration for a TIES merge of two models, with specified weights and densities.\n \"\"\"\n # Create the TIES merge configuration dictionary\n merge_config_dict = {\n \"merge_method\": \"ties\",\n \"slices\": None, # Optional slices if needed\n \"models\": [\n {\n \"model\": {\n \"model\": {\"path\": self.target_model_path, \"revision\": None},\n \"lora\": None,\n \"override_architecture\": None,\n },\n \"parameters\": {\"density\": self.target_model_density, \"weight\": self.target_model_weight},\n },\n {\n \"model\": {\n \"model\": {\"path\": self.policy_model_path, \"revision\": None},\n \"lora\": None,\n \"override_architecture\": None,\n },\n \"parameters\": {\"density\": self.policy_model_density, \"weight\": self.policy_model_weight},\n },\n ],\n \"parameters\": {\"normalize\": self.normalize},\n \"base_model\": {\n \"model\": {\"path\": self.policy_model_path, \"revision\": None},\n \"lora\": None,\n \"override_architecture\": None,\n },\n \"dtype\": self.dtype,\n \"tokenizer_source\": None,\n \"tokenizer\": None,\n \"chat_template\": None,\n \"out_dtype\": None,\n }\n\n # Create the MergeConfiguration from the dictionary\n merge_config = MergeConfiguration.model_validate(merge_config_dict)\n\n return merge_config\n\n def create_merge_config_dare_ties(self) -> \"MergeConfiguration\":\n \"\"\"\n Creates a merge configuration for a DARE TIES merge of two models, with specified weights and densities.\n \"\"\"\n # Create the DARE TIES merge configuration dictionary\n merge_config_dict = {\n \"merge_method\": \"dare_ties\",\n \"slices\": None, # Optional slices if needed\n \"models\": [\n {\n \"model\": {\n \"model\": {\"path\": self.target_model_path, \"revision\": None},\n \"lora\": None,\n \"override_architecture\": None,\n },\n \"parameters\": {\"density\": self.target_model_density, \"weight\": self.target_model_weight},\n },\n {\n \"model\": {\n \"model\": {\"path\": self.policy_model_path, \"revision\": None},\n \"lora\": None,\n \"override_architecture\": None,\n },\n \"parameters\": {\"density\": self.policy_model_density, \"weight\": self.policy_model_weight},\n },\n ],\n \"parameters\": {\"normalize\": self.normalize},\n \"base_model\": {\n \"model\": {\"path\": self.policy_model_path, \"revision\": None},\n \"lora\": None,\n \"override_architecture\": None,\n },\n \"dtype\": self.dtype,\n \"tokenizer_source\": None,\n \"tokenizer\": None,\n \"chat_template\": None,\n \"out_dtype\": None,\n }\n\n # Create the MergeConfiguration from the dictionary\n merge_config = MergeConfiguration.model_validate(merge_config_dict)\n\n return merge_config\n\n def create_merge_config_slerp(self) -> \"MergeConfiguration\":\n \"\"\"\n Creates a merge configuration for a SLERP merge of a model with a base model.\n \"\"\"\n\n # Create the SLERP merge configuration dictionary\n merge_config_dict = {\n \"merge_method\": \"slerp\",\n \"slices\": None, # Optional slices if needed\n \"models\": [\n {\n \"model\": {\n \"model\": {\"path\": self.target_model_path, \"revision\": None},\n \"lora\": None,\n \"override_architecture\": None,\n },\n \"parameters\": None, # No specific parameters for SLERP model\n }\n ],\n \"parameters\": {\n \"t\": self.t_values # Set the t values for SLERP\n },\n \"base_model\": {\n \"model\": {\"path\": self.policy_model_path, \"revision\": None},\n \"lora\": None,\n \"override_architecture\": None,\n },\n \"dtype\": self.dtype,\n \"tokenizer_source\": None,\n \"tokenizer\": None,\n \"chat_template\": None,\n \"out_dtype\": None,\n }\n\n # Create the MergeConfiguration from the dictionary\n merge_config = MergeConfiguration.model_validate(merge_config_dict)\n\n return merge_config\n\n def create(self) -> \"MergeConfiguration\":\n if self.method == \"linear\":\n return self.create_merge_config_linear()\n elif self.method == \"ties\":\n return self.create_merge_config_ties()\n elif self.method == \"dare_ties\":\n return self.create_merge_config_dare_ties()\n elif self.method == \"slerp\":\n return self.create_merge_config_slerp()\n\n\ndef merge_models(config: \"MergeConfiguration\", out_path: str):\n \"\"\"\n Merge two models using mergekit\n\n Args:\n config (`MergeConfiguration`): The merge configuration.\n out_path (`str`): The output path for the merged model.\n \"\"\"\n if not is_mergekit_available():\n raise ImportError(\"merge_models requires the `mergekit` extra. To install, run `pip install mergekit`.\")\n run_merge(\n config,\n out_path=out_path,\n options=MergeOptions(\n device=\"auto\",\n cuda=torch.cuda.is_available(),\n copy_tokenizer=True,\n lazy_unpickle=False,\n low_cpu_memory=False,\n ),\n )\n\n\nclass MergeModelCallback(TrainerCallback):\n r\"\"\"\n A [`~transformers.TrainerCallback`] that merges the policy model (the model being trained) with another model based\n on a merge configuration.\n\n Args:\n merge_config ([`experimental.merge_model_callback.MergeConfig`], *optional*):\n Configuration used for the merging process. If not provided, the default\n [`~experimental.merge_model_callback.MergeConfig`] is used.\n merge_at_every_checkpoint (`bool`, *optional*, defaults to `False`):\n Whether to merge the model at every checkpoint.\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the merged model to the Hub after merging.\n\n Example:\n\n ```python\n from trl.experimental.merge_model_callback import MergeConfig, MergeModelCallback\n\n config = MergeConfig()\n merge_callback = MergeModelCallback(config)\n trainer = DPOTrainer(..., callbacks=[merge_callback])\n ```\n \"\"\"\n\n def __init__(\n self,\n merge_config: \"MergeConfig | None\" = None,\n merge_at_every_checkpoint: bool = False,\n push_to_hub: bool = False,\n ):\n if not is_mergekit_available():\n raise ImportError(\n \"MergeModelCallback requires the `mergekit` extra. To install, run `pip install mergekit`.\"\n )\n self.merge_config = merge_config or MergeConfig()\n self.merge_at_every_checkpoint = merge_at_every_checkpoint\n self.push_to_hub = push_to_hub\n\n def _merge_and_maybe_push(self, output_dir, global_step, model):\n checkpoint_path = os.path.join(output_dir, f\"checkpoint-{global_step}\")\n self.merge_config.policy_model_path = checkpoint_path\n if self.merge_config.target_model_path is None:\n self.merge_config.target_model_path = get_config_model_id(model.config)\n merge_path = os.path.join(checkpoint_path, \"merged\")\n\n merge_models(self.merge_config.create(), merge_path)\n\n if self.push_to_hub:\n repo_name = f\"{output_dir}_checkpoint-{global_step}_merged\"\n upload_model_to_hf(merge_path, repo_name)\n\n def on_save(self, args, state, control, model=None, **kwargs):\n if self.merge_at_every_checkpoint:\n self._merge_and_maybe_push(args.output_dir, state.global_step, model)\n\n def on_train_end(self, args, state, control, model=None, **kwargs):\n if not self.merge_at_every_checkpoint:\n self._merge_and_maybe_push(args.output_dir, state.global_step, model)\n" }, { "path": "trl/experimental/minillm/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .minillm_config import MiniLLMConfig\nfrom .minillm_trainer import MiniLLMTrainer\n\n\n__all__ = [\"MiniLLMConfig\", \"MiniLLMTrainer\"]\n" }, { "path": "trl/experimental/minillm/minillm_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom transformers import TrainingArguments\n\nfrom ...trainer.grpo_config import GRPOConfig\n\n\n@dataclass\nclass MiniLLMConfig(GRPOConfig):\n \"\"\"\n Configuration class for [`MiniLLMTrainer`].\n\n This class includes only the parameters that are specific to MiniLLM training. For a full list of training\n arguments, please refer to the [`~transformers.TrainingArguments`] and [`GRPOConfig`] documentation.\n\n Args:\n teacher_model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the teacher model\n from a string.\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model.\n rkl_advantage (`bool`, *optional*, defaults to `True`):\n Whether to add the reverse KL advantage to the reward advantage.\n single_step_decomposition (`bool`, *optional*, defaults to `True`):\n Whether to use single-step decomposition for the KL divergence computation.\n kd_temperature (`float`, *optional*, defaults to `1.0`):\n Temperature for knowledge distillation. Higher temperatures produce softer probability distributions over\n classes.\n gamma (`float`, *optional*, defaults to `0.0`):\n Discount factor for future rewards in reinforcement learning.\n length_normalization (`bool`, *optional*, defaults to `True`):\n Whether to apply length normalization to the rewards.\n \"\"\"\n\n _VALID_DICT_FIELDS = GRPOConfig._VALID_DICT_FIELDS + [\"teacher_model_init_kwargs\"]\n\n teacher_model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the \"\n \"teacher model from a string.\"\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropouts in `model`.\"},\n )\n rkl_advantage: bool = field(\n default=True,\n metadata={\"help\": \"Whether to add the reverse KL advantage to the reward advantage.\"},\n )\n single_step_decomposition: bool = field(\n default=True,\n metadata={\"help\": \"Whether to use single-step decomposition for the KL divergence computation.\"},\n )\n kd_temperature: float = field(\n default=1.0,\n metadata={\n \"help\": \"Temperature for knowledge distillation. Higher temperatures produce softer probability \"\n \"distributions over classes.\"\n },\n )\n gamma: float = field(\n default=0.0,\n metadata={\"help\": \"Discount factor for future rewards in reinforcement learning.\"},\n )\n length_normalization: bool = field(\n default=True,\n metadata={\"help\": \"Whether to apply length normalization to the rewards.\"},\n )\n\n def __post_init__(self):\n # We do not use the post_init of GRPOConfig because:\n # 1. num_generations can be < 2 in MiniLLMConfig. Scale_rewards must be set to \"none\" to avoid nan.\n self.bf16 = not (self.fp16) if self.bf16 is None else self.bf16\n\n TrainingArguments.__post_init__(self)\n\n self.scale_rewards = {True: \"group\", False: \"none\"}.get(self.scale_rewards, self.scale_rewards)\n if self.num_generations == 1:\n self.scale_rewards = \"none\"\n\n num_processes = self.world_size\n # The current default effective batch size\n if self.generation_batch_size is None and self.steps_per_generation is None:\n self.steps_per_generation = self.gradient_accumulation_steps\n self.generation_batch_size = self.per_device_train_batch_size * num_processes * self.steps_per_generation\n elif self.generation_batch_size is not None and self.steps_per_generation is None:\n # Just ensure the value is divisible by the global batch size\n if self.generation_batch_size % (self.per_device_train_batch_size * num_processes) != 0:\n raise ValueError(\n f\"generation_batch_size ({self.generation_batch_size}) must be divisible by the global batch size \"\n f\"({self.per_device_train_batch_size * num_processes}).\"\n )\n self.steps_per_generation = self.generation_batch_size // (\n self.per_device_train_batch_size * num_processes\n )\n elif self.generation_batch_size is None and self.steps_per_generation is not None:\n self.generation_batch_size = self.per_device_train_batch_size * num_processes * self.steps_per_generation\n else:\n raise ValueError(\n \"'generation_batch_size' and 'steps_per_generation' can not be both configured at the same time\"\n )\n\n if self.do_eval and self.eval_strategy != \"no\":\n # Determine the number of generations to use for evaluation\n num_generations = self.num_generations_eval or self.num_generations\n\n # Just ensure the value is divisible by the global batch size\n if (self.per_device_eval_batch_size * num_processes) % num_generations != 0:\n raise ValueError(\n f\"The global eval batch size ({self.per_device_eval_batch_size} * {num_processes}) must be \"\n f\"divisible by the number of generations used for evaluation ({num_generations}).\"\n )\n\n # The generation batch must contain full prompt groups (no partials), so it must be divisible by\n # num_generations.\n if self.generation_batch_size % self.num_generations != 0:\n raise ValueError(\n f\"generation_batch_size ({self.generation_batch_size}) must be divisible by num_generations \"\n f\"({self.num_generations}).\"\n )\n\n if self.delta is not None and self.use_liger_kernel:\n raise ValueError(\"Liger kernel does not support two-sided GRPO loss yet.\")\n" }, { "path": "trl/experimental/minillm/minillm_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport textwrap\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport transformers\nfrom datasets import Dataset, IterableDataset\nfrom packaging.version import Version\nfrom transformers import (\n AutoModelForCausalLM,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n)\nfrom transformers.utils import is_peft_available\n\nfrom ...models import prepare_deepspeed\nfrom ...trainer.grpo_trainer import GRPOTrainer, RewardFunc, RolloutFunc\nfrom ...trainer.utils import disable_dropout_in_model, get_config_model_id\nfrom ..utils import empty_cache\nfrom .minillm_config import MiniLLMConfig\n\n\nif is_peft_available():\n from peft import PeftConfig\n\n\ndef dummy_reward_func(completions: list, **kwargs):\n # placeholder reward function when no reward function is provided\n return [1.0 for _ in completions]\n\n\nclass MiniLLMTrainer(GRPOTrainer):\n \"\"\"\n Trainer for the Knowledge Distillation of Language Models (MiniLLM) method. This algorithm was initially proposed\n in the paper [Knowledge Distillation of Large Language Models](https://huggingface.co/papers/2306.08543).\n\n Example:\n\n ```python\n from datasets import load_dataset\n from trl.experimental.minillm import MiniLLMTrainer\n\n dataset = load_dataset(\"trl-lib/tldr\", split=\"train\")\n\n trainer = MiniLLMTrainer(\n model=\"Qwen/Qwen3-0.6B\",\n teacher_model=\"Qwen/Qwen3-1.7B\",\n train_dataset=dataset,\n )\n trainer.train()\n ```\n\n Args:\n model (`str | PreTrainedModel`):\n Model to be trained. Can be either:\n\n - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using [`~transformers.AutoModelForCausalLM.from_pretrained`] with the keyword arguments in\n `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported.\n teacher_model (`PreTrainedModel | nn.Module | str`):\n Teacher model used for knowledge distillation. Instantiated similarly to `model`.\n reward_funcs (`RewardFunc | list[RewardFunc]`, *optional*):\n Reward functions to be used for computing the rewards. To compute the rewards, we call all the reward\n functions with the prompts and completions and sum the rewards. Can be either:\n\n - A single reward function, such as:\n - A string: The *model ID* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using [`~transformers.AutoModelForSequenceClassification.from_pretrained`] with `num_labels=1` and the\n keyword arguments in `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object: Only sequence classification models are supported.\n - A custom reward function: The function is provided with the prompts and the generated completions,\n plus any additional columns in the dataset. It should return a list of rewards. Custom reward\n functions can also return `None` when the reward is not applicable to those samples. This is useful\n for multi-task training where different reward functions apply to different types of samples. When a\n reward function returns `None` for a sample, that reward function is excluded from the reward\n calculation for that sample. For more details, see [Using a custom reward\n function](#using-a-custom-reward-function).\n\n The trainer's state is also passed to the reward function. The trainer's state is an instance of\n [`~transformers.TrainerState`] and can be accessed by accessing the `trainer_state` argument to the\n reward function's signature.\n - A list of reward functions, where each item can independently be any of the above types. Mixing different\n types within the list (e.g., a string model ID and a custom reward function) is allowed.\n args ([`experimental.minillm.MiniLLMConfig`], *optional*):\n Configuration for this trainer. If `None`, a default configuration is used.\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n Dataset to use for training. It must include a column `\"prompt\"`. Any additional columns in the dataset is\n ignored. The format of the samples can be either:\n\n - [Standard](dataset_formats#standard): Each sample contains plain text.\n - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role\n and content).\n eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Dataset | IterableDataset]`):\n Dataset to use for evaluation. It must meet the same requirements as `train_dataset`.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. The padding side must be set to \"left\". If `None`, the\n processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A\n padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token,\n `tokenizer.eos_token` will be used as the default.\n reward_processing_classes ([`~transformers.PreTrainedTokenizerBase`] or `list[PreTrainedTokenizerBase]`, *optional*):\n Processing classes corresponding to the reward functions specified in `reward_funcs`. Can be either:\n\n - A single processing class: Used when `reward_funcs` contains only one reward function.\n - A list of processing classes: Must match the order and length of the reward functions in `reward_funcs`.\n If set to `None`, or if an element of the list corresponding to a [`~transformers.PreTrainedModel`] is\n `None`, the tokenizer for the model is automatically loaded using\n [`~transformers.AutoTokenizer.from_pretrained`]. For elements in `reward_funcs` that are custom reward\n functions (not [`~transformers.PreTrainedModel`]), the corresponding entries in `reward_processing_classes`\n are ignored.\n callbacks (list of [`~transformers.TrainerCallback`], *optional*):\n List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed\n in [here](https://huggingface.co/docs/transformers/main_classes/callback).\n\n If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`]\n method.\n optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`, *optional*, defaults to `(None, None)`):\n A tuple containing the optimizer and the scheduler to use. Will default to an instance of [`AdamW`] on your\n model and a scheduler given by [`get_linear_schedule_with_warmup`] controlled by `args`.\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration used to wrap the model. If `None`, the model is not wrapped.\n rollout_func (`RolloutFunc`, *optional*):\n Function to use for generating completions. It must take prompts, args, and processing_class as parameters\n and return a dict with `\"prompt_ids\"`, `\"completion_ids\"`, and `\"logprobs\"` fields. Any other fields that\n are forwarded to the reward functions. This feature is experimental and may change or be removed at any\n time without prior notice.\n \"\"\"\n\n _tag_names = [\"trl\", \"minillm\"]\n _name = \"MiniLLM\"\n _paper = {\n \"title\": \"MiniLLM: Knowledge Distillation of Large Language Models\",\n \"id\": \"2306.08543\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @inproceedings{\n gu2024minillm,\n title={{MiniLLM: Knowledge Distillation of Large Language Models}},\n author={Yuxian Gu and Li Dong and Furu Wei and Minlie Huang},\n booktitle={The Twelfth International Conference on Learning Representations},\n year={2024},\n url={https://openreview.net/forum?id=5h0qf7IBZZ}\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: str | PreTrainedModel,\n teacher_model: PreTrainedModel | nn.Module | str,\n reward_funcs: RewardFunc | list[RewardFunc] | None = None,\n args: MiniLLMConfig | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | IterableDataset | dict[str, Dataset | IterableDataset] | None = None,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin | None = None,\n reward_processing_classes: PreTrainedTokenizerBase | list[PreTrainedTokenizerBase] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None] = (None, None),\n peft_config: \"PeftConfig | None\" = None,\n rollout_func: RolloutFunc | None = None,\n ):\n if reward_funcs is None:\n reward_funcs = [dummy_reward_func]\n\n # Args\n if args is None:\n model_name = model if isinstance(model, str) else get_config_model_id(model.config)\n model_name = model_name.split(\"/\")[-1]\n args = MiniLLMConfig(f\"{model_name}-MiniLLM\")\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model,\n reward_funcs,\n args=args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n reward_processing_classes=reward_processing_classes,\n callbacks=callbacks,\n optimizers=optimizers,\n peft_config=peft_config,\n rollout_func=rollout_func,\n )\n\n if args.teacher_model_init_kwargs is None:\n teacher_model_init_kwargs = {}\n elif not isinstance(teacher_model, str):\n raise ValueError(\n \"You passed teacher_model_init_kwargs to the MiniLLMConfig, but your teacher_model is already instantiated.\"\n )\n else:\n teacher_model_init_kwargs = args.teacher_model_init_kwargs\n teacher_model_init_kwargs[\"dtype\"] = (\n teacher_model_init_kwargs[\"dtype\"]\n if teacher_model_init_kwargs[\"dtype\"] in [\"auto\", None]\n else getattr(torch, teacher_model_init_kwargs[\"dtype\"])\n )\n\n if isinstance(teacher_model, str):\n teacher_model = AutoModelForCausalLM.from_pretrained(teacher_model, **teacher_model_init_kwargs)\n\n # Disable dropout in the model\n if args.disable_dropout:\n disable_dropout_in_model(self.model)\n\n if self.is_deepspeed_enabled:\n self.teacher_model = prepare_deepspeed(teacher_model, self.accelerator)\n else:\n self.teacher_model = self.accelerator.prepare_model(teacher_model, evaluation_mode=True)\n\n self.temperature = args.temperature\n self.kd_temperature = args.kd_temperature\n self.single_step_decomposition = args.single_step_decomposition\n self.rkl_advantage = args.rkl_advantage\n self.gamma = args.gamma\n self.length_normalization = args.length_normalization\n\n def _single_step_decomposition_loss(\n self,\n student_log_probs: torch.Tensor,\n teacher_log_probs: torch.Tensor,\n mask: torch.Tensor | None = None,\n reduction: str = \"batchmean\",\n ):\n \"\"\"\n Compute the MiniLLM loss for knowledge distillation using F.kl_div. See Eq. (1) of\n https://huggingface.co/papers/2306.08543 for the definition.\n\n Args:\n student_logits:\n Tensor of shape (batch_size, sequence_length, vocab_size)\n teacher_logits:\n Tensor of shape (batch_size, sequence_length, vocab_size)\n labels:\n Tensor of shape (batch_size, sequence_length) with -100 for padding tokens to ignore when computing\n loss\n beta:\n Interpolation coefficient between 0 and 1 (default: 0.5)\n temperature:\n Softmax temperature (default: 1.0)\n reduction:\n Specifies the reduction to apply to the output (default: 'batchmean')\n\n Returns:\n loss: Scalar tensor with the generalized JSD loss\n \"\"\"\n reg_loss = F.kl_div(\n teacher_log_probs, student_log_probs, reduction=\"none\", log_target=True\n ) # (batch_size, sequence_length)\n\n # Masking\n if mask is not None:\n reg_loss = reg_loss[mask]\n\n # Apply reduction\n if reduction == \"batchmean\":\n return reg_loss.sum() / mask.sum() if mask is not None else reg_loss.sum() / reg_loss.size(0)\n elif reduction == \"sum\":\n return reg_loss.sum()\n elif reduction == \"mean\":\n return reg_loss.mean()\n else:\n return reg_loss\n\n def _compute_advantage(\n self,\n student_log_probs_on_labels: torch.Tensor,\n teacher_log_probs_on_labels: torch.Tensor,\n mask: torch.Tensor | None = None,\n ) -> torch.Tensor:\n r\"\"\"Compute the advantage for Reverse KL Divergence.\n\n Mostly following [this\n implementation](https://github.com/microsoft/LMOps/blob/e210d2c026b9958617887762400778ace81172e6/minillm/minillm/losses.py#L37-L49).\n\n $$ \\text{rewards}_t = \\text{teacher\\_log\\_probs\\_on\\_labels}_t - \\text{student\\_log\\_probs\\_on\\_labels}_t $$\n\n If length normalization is enabled:\n\n $$ \\text{lengths}_t = \\sum_{i=t}^{T} \\gamma^{i-t} $$\n\n $$ \\text{advantages}_t = \\frac{\\sum_{i=t}^{T} \\gamma^{i-t} R_i}{\\text{lengths}_t} $$\n\n Otherwise:\n\n $$ \\text{advantages}_t = \\sum_{i=t}^{T} \\gamma^{i-t} R_i $$\n\n Args:\n student_log_probs_on_labels: Log probabilities of the student model on the labels.\n Shape: (batch_size, sequence_length)\n teacher_log_probs_on_labels: Log probabilities of the teacher model on the labels.\n Shape: (batch_size, sequence_length)\n mask: Optional mask to apply to the log probabilities. Shape: (batch_size, sequence_length)\n Returns:\n advantage: Computed advantage. Shape: (batch_size, sequence_length)\n \"\"\"\n response_length = student_log_probs_on_labels.size(1)\n if mask is None:\n mask = torch.ones_like(student_log_probs_on_labels)\n mask = mask.float()\n student_log_probs_on_labels = student_log_probs_on_labels * mask\n teacher_log_probs_on_labels = teacher_log_probs_on_labels * mask\n\n rewards = teacher_log_probs_on_labels - student_log_probs_on_labels # (batch_size, sequence_length)\n\n if self.gamma > 0.0:\n gamma_pow = torch.pow(self.gamma, torch.arange(response_length, device=rewards.device))\n\n advantages = rewards * gamma_pow\n advantages = advantages.flip(1).cumsum(dim=1).flip(1)\n\n if self.length_normalization:\n mask = torch.where(mask < 0.5, 1e-4, mask)\n lengths = mask * gamma_pow\n lengths = lengths.flip(1).cumsum(dim=1).flip(1)\n advantages = advantages / lengths\n else:\n advantages = rewards\n\n return advantages\n\n def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):\n input_ids = torch.cat([inputs[\"prompt_ids\"], inputs[\"completion_ids\"]], dim=1)\n attention_mask = torch.cat([inputs[\"prompt_mask\"], inputs[\"completion_mask\"]], dim=1)\n labels = input_ids.clone()\n labels[attention_mask == 0] = -100\n\n # Compute student output\n student_outputs = model(input_ids=input_ids, attention_mask=attention_mask, use_cache=False)\n\n # Compute teacher output in eval mode\n self.teacher_model.eval()\n with torch.no_grad():\n teacher_outputs = self.teacher_model(input_ids=input_ids, attention_mask=attention_mask, use_cache=False)\n\n # Slice the logits for the generated tokens using the inputs[\"prompts\"] lengths\n prompt_lengths = inputs[\"prompt_ids\"].shape[1]\n student_logits = student_outputs.logits[:, prompt_lengths - 1 : -1, :]\n teacher_logits = teacher_outputs.logits[:, prompt_lengths - 1 : -1, :]\n shifted_labels = input_ids[:, prompt_lengths:]\n\n # Apply temperature scaling\n student_logits = student_logits / self.kd_temperature\n teacher_logits = teacher_logits / self.kd_temperature\n\n # Compute log probabilities for student and probabilities for teacher\n student_log_probs = F.log_softmax(student_logits, dim=-1)\n teacher_log_probs = F.log_softmax(teacher_logits, dim=-1)\n\n student_log_probs_on_labels = torch.gather(\n student_log_probs, dim=-1, index=shifted_labels.unsqueeze(-1)\n ).squeeze(-1)\n teacher_log_probs_on_labels = torch.gather(\n teacher_log_probs, dim=-1, index=shifted_labels.unsqueeze(-1)\n ).squeeze(-1)\n\n mask = shifted_labels != -100\n\n if self.rkl_advantage:\n reverse_kl_advantage = self._compute_advantage(\n student_log_probs_on_labels=student_log_probs_on_labels,\n teacher_log_probs_on_labels=teacher_log_probs_on_labels,\n mask=mask,\n )\n\n inputs[\"advantages\"] = inputs[\"advantages\"].unsqueeze(1) + reverse_kl_advantage\n\n # Compute GRPO loss on verifiable reward\n loss = self._compute_loss(model, inputs)\n\n # Compute loss\n if self.single_step_decomposition:\n single_step_decomposition_loss = self._single_step_decomposition_loss(\n student_log_probs=student_log_probs,\n teacher_log_probs=teacher_log_probs,\n mask=mask,\n )\n\n loss += single_step_decomposition_loss\n\n # Empty cache\n empty_cache()\n\n # Return loss\n return (loss, student_outputs) if return_outputs else loss\n" }, { "path": "trl/experimental/nash_md/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .nash_md_config import NashMDConfig\nfrom .nash_md_trainer import NashMDTrainer\n\n\n__all__ = [\"NashMDConfig\", \"NashMDTrainer\"]\n" }, { "path": "trl/experimental/nash_md/nash_md_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom ..online_dpo import OnlineDPOConfig\n\n\n@dataclass\nclass NashMDConfig(OnlineDPOConfig):\n r\"\"\"\n Configuration class for the [`experimental.nash_md.NashMDTrainer`].\n\n Subclass of [`experimental.online_dpo.OnlineDPOConfig`] we can use all its arguments and add the following:\n\n Parameters:\n mixture_coef (`float` or `list[float]`, *optional*, defaults to `0.5`):\n Logit mixture coefficient for the model and reference model. If a list of floats is provided then the\n mixture coefficient is selected for each new epoch and the last coefficient is used for the rest of the\n epochs.\n \"\"\"\n\n mixture_coef: list[float] = field(\n default_factory=lambda: [0.5],\n metadata={\n \"help\": \"Logit mixture coefficient for the model and reference model. If a list of floats is provided \"\n \"then the mixture coefficient is selected for each new epoch and the last coefficient is used for the \"\n \"rest of the epochs.\"\n },\n )\n\n def __post_init__(self):\n super().__post_init__()\n if hasattr(self.mixture_coef, \"__len__\") and len(self.mixture_coef) == 1:\n self.mixture_coef = self.mixture_coef[0]\n" }, { "path": "trl/experimental/nash_md/nash_md_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport textwrap\nfrom collections.abc import Callable\nfrom typing import Any\n\nimport jinja2\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom datasets import Dataset, IterableDataset\nfrom transformers import (\n BaseImageProcessor,\n FeatureExtractionMixin,\n GenerationMixin,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n)\nfrom transformers.trainer_utils import EvalPrediction\nfrom transformers.training_args import OptimizerNames\nfrom transformers.utils import is_peft_available\n\nfrom ...data_utils import is_conversational, maybe_apply_chat_template\nfrom ...models.utils import unwrap_model_for_generation\nfrom ...trainer.utils import selective_log_softmax\nfrom ..judges import BasePairwiseJudge\nfrom ..online_dpo import OnlineDPOTrainer\nfrom ..utils import SIMPLE_CHAT_TEMPLATE, empty_cache, get_reward, truncate_right\nfrom .nash_md_config import NashMDConfig\n\n\nif is_peft_available():\n from peft import PeftModel\n\n\nclass GeometricMixtureWrapper(GenerationMixin):\n \"\"\"\n Geometric Mixture generation wrapper that samples from the logits of two model's geometric mixture.\n\n Args:\n model ([`~transformers.PreTrainedModel`]): The model to be wrapped.\n ref_model ([`~transformers.PreTrainedModel`]): The reference model.\n generation_config ([`~transformers.GenerationConfig`]): The generation config.\n mixture_coef (`float`, *optional* - default: 0.5): The mixture coefficient.\n \"\"\"\n\n main_input_name = \"input_ids\"\n _supports_cache_class = False\n _supports_static_cache = False\n _is_stateful = False\n\n def __init__(self, model, ref_model, generation_config, mixture_coef=0.5, device=None):\n super().__init__()\n\n self.model = model\n self.config = model.config\n self.ref_model = ref_model\n self.generation_config = generation_config\n self.mixture_coef = mixture_coef\n self.device = device\n if hasattr(self.model, \"_is_stateful\"):\n self._is_stateful = self.model._is_stateful\n\n def __call__(self, *args, **kwargs):\n return self.forward(*args, **kwargs)\n\n @torch.inference_mode()\n def forward(self, *args, **kwargs):\n model_outputs = self.model(*args, **kwargs)\n model_logits = model_outputs.logits\n ref_model_logits = self.ref_model(*args, **kwargs).logits\n\n model_outputs.logits = torch.nn.functional.log_softmax(\n self.mixture_coef * ref_model_logits + (1 - self.mixture_coef) * model_logits, dim=-1\n )\n\n return model_outputs\n\n def prepare_inputs_for_generation(self, *args, **kwargs):\n # turn off cache in the generation config\n kwargs[\"use_cache\"] = False\n model_inputs = self.model.prepare_inputs_for_generation(*args, **kwargs)\n _ = self.ref_model.prepare_inputs_for_generation(*args, **kwargs)\n\n return model_inputs\n\n def _validate_model_class(self):\n self.model._validate_model_class()\n\n def _validate_model_kwargs(self, model_kwargs):\n return self.model._validate_model_kwargs(model_kwargs)\n\n\nclass NashMDTrainer(OnlineDPOTrainer):\n \"\"\"\n Trainer for the Nash-MD method.\n\n It is implemented as a subclass of [`experimental.online_dpo.OnlineDPOTrainer`].\n\n Args:\n model ([`~transformers.PreTrainedModel`]):\n The model to train, preferably an `AutoModelForCausalLM`.\n ref_model ([`~transformers.PreTrainedModel`]):\n Hugging Face transformer model with a casual language modelling head. Used for implicit reward computation\n and loss. If no reference model is provided, the trainer will create a reference model with the same\n architecture as the model to be optimized.\n reward_funcs ([`~transformers.PreTrainedModel`]):\n The reward model to score completions with, preferably an\n [`~transformers.AutoModelForSequenceClassification`].\n judge ([`experimental.judges.BasePairwiseJudge`]):\n The judge to use for pairwise comparison of model completions.\n args ([`experimental.nash_md.NashMDConfig`]):\n The NashMD config arguments to use for training.\n data_collator ([`~transformers.DataCollator`]):\n The data collator to use for training. If None is specified, the default data collator\n ([`experimental.utils.DPODataCollatorWithPadding`]) will be used which will pad the sequences to the\n maximum length of the sequences in the batch, given a dataset of paired sequences.\n train_dataset ([`~datasets.Dataset`]):\n The dataset to use for training.\n eval_dataset ([`~datasets.Dataset`]):\n The dataset to use for evaluation.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. If provided, will be used to automatically process the inputs\n for the model, and it will be saved along the model to make it easier to rerun an interrupted training or\n reuse the fine-tuned model.\n peft_config (`dict`):\n The peft config to use for training.\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to\n metric values.\n callbacks (`list[transformers.TrainerCallback]`):\n The callbacks to use for training.\n optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):\n The optimizer and scheduler to use for training.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):\n The function to use to preprocess the logits before computing the metrics.\n \"\"\"\n\n _tag_names = [\"trl\", \"nash-md\"]\n _name = \"Nash-MD\"\n _paper = {\n \"title\": \"Nash Learning from Human Feedback\",\n \"id\": \"2312.00886\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @inproceedings{munos2024nash,\n title = {{Nash Learning from Human Feedback}},\n author = {R{\\'{e}}mi Munos and Michal Valko and Daniele Calandriello and Mohammad Gheshlaghi Azar and Mark Rowland and Zhaohan Daniel Guo and Yunhao Tang and Matthieu Geist and Thomas Mesnard and C{\\\\^{o}}me Fiegel and Andrea Michi and Marco Selvi and Sertan Girgin and Nikola Momchev and Olivier Bachem and Daniel J. Mankowitz and Doina Precup and Bilal Piot},\n year = 2024,\n booktitle = {Forty-first International Conference on Machine Learning, {ICML} 2024, Vienna, Austria, July 21-27, 2024},\n publisher = {OpenReview.net},\n url = {https://openreview.net/forum?id=Y5AmNYiyCQ}\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module = None,\n ref_model: PreTrainedModel | nn.Module = None,\n reward_funcs: PreTrainedModel | nn.Module | None = None,\n judge: BasePairwiseJudge | None = None,\n args: NashMDConfig | None = None,\n data_collator: Callable | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n processing_class: PreTrainedTokenizerBase\n | BaseImageProcessor\n | FeatureExtractionMixin\n | ProcessorMixin\n | None = None,\n peft_config: dict | None = None,\n compute_metrics: Callable[[EvalPrediction], dict] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n ) -> None:\n super().__init__(\n model=model,\n ref_model=ref_model,\n reward_funcs=reward_funcs,\n judge=judge,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n reward_processing_classes=processing_class,\n peft_config=peft_config,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n self._mixture_coef = self.args.mixture_coef\n\n # Overwrite the stats dictionary to include NashMD specific statistics\n self.stats = {\n # Remove \"non_score_reward\", \"rlhf_reward\", \"scores_margin\"\n # Add \"mixture_coef\"\n \"loss/kl\": [],\n \"objective/entropy\": [],\n \"loss/score\": [],\n \"rewards/probabilities\": [],\n \"rewards/accuracies\": [],\n \"rewards/margins\": [],\n \"logps/chosen\": [],\n \"logps/rejected\": [],\n \"val/model_contain_eos_token\": [],\n \"val/ref_contain_eos_token\": [],\n \"beta\": [],\n \"mixture_coef\": [],\n }\n if self.reward_funcs is not None:\n if len(self.reward_funcs) != 1:\n raise ValueError(\"NashMDTrainer only supports one reward function/model.\")\n self.reward_funcs = self.reward_funcs[0]\n self.stats[\"rewards/chosen\"] = []\n self.stats[\"rewards/rejected\"] = []\n\n @property\n def mixture_coef(self):\n if isinstance(self._mixture_coef, list):\n epoch = self.state.epoch\n return self._mixture_coef[epoch] if epoch < len(self._mixture_coef) else self._mixture_coef[-1]\n else:\n return self._mixture_coef\n\n def _generate_completions(self, model, prompts):\n # Generate completions from the policy model.\n with (\n unwrap_model_for_generation(\n model,\n self.accelerator,\n generation_kwargs=self.generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as unwrapped_policy_for_gen_ctx\n ):\n model_output = unwrapped_policy_for_gen_ctx.generate(\n input_ids=prompts[\"input_ids\"],\n attention_mask=prompts[\"attention_mask\"],\n generation_config=self.generation_config,\n )\n\n # Get the DDP/FSDP unwrapped version of the main model.\n # This will be the policy model for GeometricMixtureWrapper (PEFT adapters active if PEFT is used).\n policy_model_for_gmw = self.accelerator.unwrap_model(model)\n\n # Determine the correct reference model for GeometricMixtureWrapper.\n # This also needs to be DDP/FSDP unwrapped.\n ref_model_for_gmw: torch.nn.Module\n if self.ref_model is None:\n # No explicit ref_model is provided.\n # Use the base of the main `model` if it's a PEFT model.\n # policy_model_for_gmw is already DDP-unwrapped.\n if is_peft_available() and isinstance(policy_model_for_gmw, PeftModel):\n ref_model_for_gmw = policy_model_for_gmw.get_base_model()\n else:\n # Not a PEFT model (or PEFT not available), or already a base model.\n # Use the DDP-unwrapped policy model itself as the reference.\n ref_model_for_gmw = policy_model_for_gmw\n else:\n # An explicit ref_model is provided. Unwrap it for DDP/FSDP.\n ref_model_for_gmw = self.accelerator.unwrap_model(self.ref_model)\n\n # Both models given to GeometricMixtureWrapper (policy_model_for_gmw and ref_model_for_gmw) are DDP-unwrapped.\n with torch.no_grad(): # Ensure no_grad context for mixture model generation\n mixture_model = GeometricMixtureWrapper(\n model=policy_model_for_gmw,\n ref_model=ref_model_for_gmw,\n generation_config=self.generation_config,\n mixture_coef=self.mixture_coef,\n device=self.accelerator.device,\n )\n\n # TODO: use self._override_model_generation_config for both models?\n mixture_output = mixture_model.generate(\n input_ids=prompts[\"input_ids\"],\n attention_mask=prompts[\"attention_mask\"],\n generation_config=self.generation_config,\n )\n\n return model_output, mixture_output\n\n def _process_completions(self, model_output, mixture_output, prompts):\n context_length = prompts[\"input_ids\"].shape[1]\n\n # Process model completions\n model_completion_ids = model_output[:, context_length:]\n model_completion_ids, model_completion_mask = truncate_right(\n model_completion_ids, self.processing_class.eos_token_id, self.processing_class.pad_token_id\n )\n model_data = {\n \"input_ids\": torch.cat((prompts[\"input_ids\"], model_completion_ids), dim=1),\n \"attention_mask\": torch.cat((prompts[\"attention_mask\"], model_completion_mask), dim=1),\n \"raw\": prompts[\"raw\"],\n }\n\n # Process reference model completions\n mixture_completion_ids = mixture_output[:, context_length:]\n mixture_completion_ids, mixture_completion_mask = truncate_right(\n mixture_completion_ids, self.processing_class.eos_token_id, self.processing_class.pad_token_id\n )\n mixture_data = {\n \"input_ids\": torch.cat((prompts[\"input_ids\"], mixture_completion_ids), dim=1),\n \"attention_mask\": torch.cat((prompts[\"attention_mask\"], mixture_completion_mask), dim=1),\n \"raw\": prompts[\"raw\"],\n }\n\n return model_data, mixture_data\n\n def _compute_rewards(self, model_data, mixture_data, context_length):\n with torch.no_grad():\n _, model_scores, _ = get_reward(\n self.reward_funcs, model_data[\"input_ids\"], self.processing_class.pad_token_id, context_length\n )\n _, mixture_scores, _ = get_reward(\n self.reward_funcs, mixture_data[\"input_ids\"], self.processing_class.pad_token_id, context_length\n )\n\n # Apply EOS penalty if needed\n if self.args.missing_eos_penalty is not None:\n model_contain_eos = torch.any(model_data[\"input_ids\"] == self.processing_class.eos_token_id, dim=-1)\n mixture_contain_eos = torch.any(mixture_data[\"input_ids\"] == self.processing_class.eos_token_id, dim=-1)\n model_scores[~model_contain_eos] -= self.args.missing_eos_penalty\n mixture_scores[~mixture_contain_eos] -= self.args.missing_eos_penalty\n\n return model_scores, mixture_scores\n\n def _compute_judge(self, model_data, mixture_data, context_length):\n prompts = model_data[\"raw\"]\n model_data_completions = self.processing_class.batch_decode(\n model_data[\"input_ids\"][:, context_length:], skip_special_tokens=True\n )\n model_data_completions = [completion.strip() for completion in model_data_completions]\n\n mixture_data_completions = self.processing_class.batch_decode(\n mixture_data[\"input_ids\"][:, context_length:], skip_special_tokens=True\n )\n mixture_data_completions = [completion.strip() for completion in mixture_data_completions]\n if is_conversational({\"prompt\": prompts[0]}):\n model_data_completions = [\n [{\"role\": \"assistant\", \"content\": completion}] for completion in model_data_completions\n ]\n environment = jinja2.Environment()\n template = environment.from_string(SIMPLE_CHAT_TEMPLATE)\n prompts = [template.render(messages=message) for message in prompts]\n model_data_completions = [template.render(messages=completion) for completion in model_data_completions]\n\n mixture_data_completions = [\n [{\"role\": \"assistant\", \"content\": completion}] for completion in mixture_data_completions\n ]\n mixture_data_completions = [\n template.render(messages=completion) for completion in mixture_data_completions\n ]\n\n probability = self.judge.judge(\n prompts,\n list(zip(model_data_completions, mixture_data_completions, strict=True)),\n return_scores=True,\n )\n return torch.tensor(probability, device=model_data[\"input_ids\"].device)\n\n def _compute_logprobs(self, model, model_data, context_length):\n def compute_logprobs_for_data(m, data):\n output = m(data[\"input_ids\"], attention_mask=data[\"attention_mask\"])\n logits = output.logits[:, context_length - 1 : -1]\n token_logprobs = selective_log_softmax(logits, data[\"input_ids\"][:, context_length:])\n return token_logprobs\n\n # Compute logprobs for model completions under the model\n model_logprobs_model_data = compute_logprobs_for_data(model, model_data)\n\n # Compute logprobs of model completions under the reference model\n with torch.no_grad():\n if self.ref_model is None:\n with model.disable_adapter():\n ref_logprobs_model_data = compute_logprobs_for_data(model, model_data)\n else:\n ref_logprobs_model_data = compute_logprobs_for_data(self.ref_model, model_data)\n\n # Mask padding tokens\n model_padding_mask = model_data[\"attention_mask\"][:, context_length:] == 0\n model_logprobs_model_data = model_logprobs_model_data.masked_fill(model_padding_mask, 0.0)\n ref_logprobs_model_data = ref_logprobs_model_data.masked_fill(model_padding_mask, 0.0)\n\n return (model_logprobs_model_data, ref_logprobs_model_data)\n\n def _compute_losses(\n self,\n model_logprobs_model_data,\n ref_logprobs_model_data,\n probability,\n ):\n # reinforce score where 0.5 is a control variate\n score = (probability - 0.5) * model_logprobs_model_data.sum(1)\n\n # kl divergence via reinforce\n with torch.no_grad():\n log_ratio = model_logprobs_model_data - ref_logprobs_model_data\n kl_div_log = log_ratio.sum(1)\n kl_div_loss = (log_ratio * model_logprobs_model_data).sum(1)\n\n # final loss\n loss = self.beta * kl_div_loss - score\n\n return loss.mean(), score, kl_div_log\n\n def _log_statistics(\n self,\n model_data,\n mixture_data,\n model_logprobs_model_data,\n ref_logprobs_model_data,\n probability,\n score,\n kl_div,\n context_length,\n model_scores=None,\n mixture_scores=None,\n ):\n # Helper function to gather and compute mean\n def gather_mean(tensor):\n return self.accelerator.gather_for_metrics(tensor).mean().item()\n\n # Log score\n self.stats[\"loss/score\"].append(gather_mean(score))\n # Log KL divergence\n self.stats[\"loss/kl\"].append(gather_mean(kl_div))\n\n # Log logprobs\n model_logprobs_model_data_sum = model_logprobs_model_data.sum(1)\n ref_logprobs_model_data_sum = ref_logprobs_model_data.sum(1)\n\n self.stats[\"logps/chosen\"].append(gather_mean(model_logprobs_model_data_sum))\n self.stats[\"logps/rejected\"].append(gather_mean(ref_logprobs_model_data_sum))\n\n # Log rewards\n if self.reward_funcs is not None:\n self.stats[\"rewards/chosen\"].append(gather_mean(model_scores))\n self.stats[\"rewards/rejected\"].append(gather_mean(mixture_scores))\n\n # Log probabilities\n self.stats[\"rewards/probabilities\"].append(gather_mean(probability))\n\n # Calculate entropy for model data\n entropy_model_data = -model_logprobs_model_data.sum(1)\n self.stats[\"objective/entropy\"].append(gather_mean(entropy_model_data))\n\n # Calculate margins\n margin = model_logprobs_model_data_sum - ref_logprobs_model_data_sum\n self.stats[\"rewards/margins\"].append(gather_mean(margin))\n\n # Calculate accuracy\n accuracy = (margin > 0).float()\n self.stats[\"rewards/accuracies\"].append(gather_mean(accuracy))\n\n # Log EOS token statistics\n model_eos = (model_data[\"input_ids\"][:, context_length:] == self.processing_class.eos_token_id).any(dim=1)\n mixture_eos = (mixture_data[\"input_ids\"][:, context_length:] == self.processing_class.eos_token_id).any(dim=1)\n self.stats[\"val/model_contain_eos_token\"].append(gather_mean(model_eos.float()))\n self.stats[\"val/ref_contain_eos_token\"].append(gather_mean(mixture_eos.float()))\n\n # Log beta and mixture coef\n self.stats[\"beta\"].append(self.beta)\n self.stats[\"mixture_coef\"].append(self.mixture_coef)\n\n def training_step(\n self, model: nn.Module, inputs: dict[str, torch.Tensor | Any], num_items_in_batch: int | None = None\n ) -> torch.Tensor:\n model.train()\n\n # Apply chat template and tokenize the input\n batch_size = len(next(iter(inputs.values())))\n prompts = inputs[\"prompt\"]\n inputs = [{k: v[i] for k, v in inputs.items()} for i in range(batch_size)]\n inputs = [maybe_apply_chat_template(x, self.processing_class) for x in inputs]\n inputs = [self.tokenize_row(x, self.model.config.is_encoder_decoder, self.processing_class) for x in inputs]\n inputs = self.data_collator(inputs)\n\n # need the prompt_ only\n inputs = self._prepare_inputs(inputs)\n context_length = inputs[\"prompt_input_ids\"].shape[1]\n prompts = {\n \"input_ids\": inputs[\"prompt_input_ids\"],\n \"attention_mask\": inputs[\"prompt_attention_mask\"],\n \"raw\": prompts,\n }\n del inputs\n\n # Sample completions from both the model and the reference model\n model_output, mixture_output = self._generate_completions(model, prompts)\n\n # Process model completions\n model_data, mixture_data = self._process_completions(model_output, mixture_output, prompts)\n\n # Compute rewards\n if self.reward_funcs is not None:\n model_scores, mixture_scores = self._compute_rewards(model_data, mixture_data, context_length)\n # probability of the model data vs the mixture data\n probability = F.sigmoid(model_scores - mixture_scores)\n else:\n model_scores, mixture_scores = None, None\n probability = self._compute_judge(model_data, mixture_data, context_length)\n\n # Compute logprobs\n model_logprobs_model_data, ref_logprobs_model_data = self._compute_logprobs(model, model_data, context_length)\n\n # Compute loss\n loss, score, kl_div = self._compute_losses(model_logprobs_model_data, ref_logprobs_model_data, probability)\n\n # Log everything\n self._log_statistics(\n model_data,\n mixture_data,\n model_logprobs_model_data.detach(),\n ref_logprobs_model_data,\n probability,\n score.detach(),\n kl_div.detach(),\n context_length,\n model_scores,\n mixture_scores,\n )\n\n if (\n self.args.torch_empty_cache_steps is not None\n and self.state.global_step % self.args.torch_empty_cache_steps == 0\n ):\n empty_cache()\n\n kwargs = {}\n # For LOMO optimizers you need to explicitly use the learning rate\n if self.args.optim in [OptimizerNames.LOMO, OptimizerNames.ADALOMO]:\n kwargs[\"learning_rate\"] = self._get_learning_rate()\n\n if self.args.n_gpu > 1:\n loss = loss.mean() # mean() to average on multi-gpu parallel training\n\n self.accelerator.backward(loss, **kwargs)\n\n return loss.detach() / self.args.gradient_accumulation_steps\n" }, { "path": "trl/experimental/online_dpo/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .online_dpo_config import OnlineDPOConfig\nfrom .online_dpo_trainer import OnlineDPOTrainer\n\n\n__all__ = [\"OnlineDPOConfig\", \"OnlineDPOTrainer\"]\n" }, { "path": "trl/experimental/online_dpo/online_dpo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport warnings\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom ...trainer.base_config import _BaseConfig\n\n\n@dataclass\nclass OnlineDPOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`experimental.online_dpo.OnlineDPOTrainer`].\n\n This class includes only the parameters that are specific to Online DPO training. For a full list of training\n arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this\n class may differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n reward_model_path (`str`, *optional*):\n Path to the reward model. Either `judge` or `reward_model_path` must be set, but not both.\n judge (`str`, *optional*):\n Name of the judge to use. Either `judge` or `reward_model_path` must be set, but not both.\n max_new_tokens (`int`, *optional*, defaults to `64`):\n Maximum number of tokens to generate per completion.\n max_length (`int`, *optional*, defaults to `256`):\n Maximum total length of the sequence (prompt + completion) used to compute log probabilities. If the\n sequence exceeds this limit, the leftmost tokens will be truncated to preserve as much of the completion as\n possible.\n temperature (`float`, *optional*, defaults to `0.9`):\n Temperature for sampling. The higher the temperature, the more random the completions.\n missing_eos_penalty (`float`, *optional*):\n Penalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to\n generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive\n value. This parameter only works when using `reward_funcs` and not when using `judge`.\n beta (`float` or `list[float]`, *optional*, defaults to `0.1`):\n Parameter controlling the deviation from the reference model. Higher ฮฒ means less deviation from the\n reference model. For the IPO loss (`loss_type=\"ipo\"`), ฮฒ is the regularization parameter denoted by ฯ„ in\n the [paper](https://huggingface.co/papers/2310.12036). If a list of floats is provided then the ฮฒ is\n selected for each new epoch and the last ฮฒ is used for the rest of the epochs.\n loss_type (`str`, *optional*, defaults to `\"sigmoid\"`):\n Type of loss to use. Possible values are:\n\n - `\"sigmoid\"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.\n - `\"ipo\"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model and reference model.\n\n > Parameters that control generation\n\n top_p (`float`, *optional*, defaults to `1.0`):\n Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to\n `1.0` to consider all tokens.\n top_k (`int`, *optional*, defaults to `0`):\n Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, top-k-filtering is\n disabled and all tokens are considered.\n min_p (`float`, *optional*):\n Minimum token probability, which will be scaled by the probability of the most likely token. It must be a\n value between `0.0` and `1.0`. Typical values are in the `0.01-0.2` range.\n repetition_penalty (`float`, *optional*, defaults to `1.0`):\n Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far.\n Values > `1.0` encourage the model to use new tokens, while values < `1.0` encourage the model to repeat\n tokens.\n use_transformers_paged (`bool`, *optional*, defaults to `False`):\n Whether to use the `transformers` paged implementation for generation. If set to `True`, the `transformers`\n paged implementation will be used for generation instead of the default padded implementation. This\n parameter is only effective when `use_vllm` is set to `False`.\n cache_implementation (`str`, *optional*):\n Implementation of the cache method for faster generation when `use_vllm` is set to `False`.\n generation_kwargs (`dict[str, Any]`, *optional*):\n Additional keyword arguments to pass to [`~transformers.GenerationConfig`] (if using transformers) or\n `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the\n generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that conflict\n with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.\n\n > Parameters that control generation acceleration powered by vLLM\n\n use_vllm (`bool`, *optional*, defaults to `False`):\n Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation\n instead of the default model.generate(). Requires `vllm` to be installed.\n vllm_model_impl (`str`, *optional*, defaults to `\"vllm\"`):\n Model implementation to use for vLLM. Must be one of `\"transformers\"` or `\"vllm\"`. `\"transformers\"`: Use\n the `transformers` backend for model implementation. `\"vllm\"`: Use the `vllm` library for model\n implementation.\n vllm_mode (`str`, *optional*, defaults to `\"colocate\"`):\n Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `\"server\"` or\n `\"colocate\"`.\n\n - `\"server\"`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM\n server is running (start with `trl vllm-serve`).\n - `\"colocate\"`: vLLM will run in the same process and share the training GPUs. This avoids the need for a\n separate server but may cause resource contention with training.\n vllm_structured_outputs_regex (`str`, *optional*):\n Regex for vLLM structured outputs. If `None` (default), structured outputs is disabled.\n\n > Parameters that control the vLLM server (only used when `vllm_mode` is `\"server\"`)\n\n vllm_server_base_url (`str`, *optional*):\n Base URL for the vLLM server (e.g., `\"http://localhost:8000\"`). If provided, `vllm_server_host` and\n `vllm_server_port` are ignored.\n vllm_server_host (`str`, *optional*, defaults to `\"0.0.0.0\"`):\n Host of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided.\n vllm_server_port (`int`, *optional*, defaults to `8000`):\n Port of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided.\n vllm_server_timeout (`float`, *optional*, defaults to `240.0`):\n Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the\n timeout, a `ConnectionError` is raised.\n vllm_group_port (`int`, *optional*, defaults to `51216`):\n Port number for the weight update group. This is used to communicate with the vLLM server. Unless the port\n is occupied, there is no need to change it.\n\n > Parameters that control colocated vLLM execution (only used when `vllm_mode` is `\"colocate\"`)\n\n vllm_gpu_memory_utilization (`float`, *optional*, defaults to `0.55`):\n Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to\n `\"colocate\"`. If you are using `vllm_mode=\"server\"`, this parameter must be passed separately when\n launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.\n vllm_tensor_parallel_size (`int`, *optional*, defaults to `1`):\n Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to\n `\"colocate\"`. If you are using `vllm_mode=\"server\"`, this parameter must be passed separately when\n launching the vLLM server via the `--vllm_tensor_parallel_size` flag.\n vllm_enable_sleep_mode (`bool`, *optional*, defaults to `False`):\n Enable vLLM sleep mode to offload weights/cache during the optimizer step. Keeps GPU memory usage low, but\n waking the engine adds hostโ€“device transfer latency.\n\n > Other parameters\n\n ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):\n This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,\n improving generation speed. However, disabling this option allows training models that exceed the VRAM\n capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible\n with vLLM generation.\n model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a\n string.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `5e-7` instead of `5e-5`.\n > - `remove_unused_columns`: Defaults to `False` instead of `True`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=5e-7,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n remove_unused_columns: bool = field(\n default=False,\n metadata={\"help\": \"Whether or not to automatically remove the columns unused by the model forward method.\"},\n )\n\n reward_model_path: str | None = field(\n default=None,\n metadata={\n \"help\": \"Path to the reward model. Either `judge` or `reward_model_path` must be set, but not both.\"\n },\n )\n judge: str | None = field(\n default=None,\n metadata={\n \"help\": \"Name of the judge to use. Either `judge` or `reward_model_path` must be set, but not both.\"\n },\n )\n max_new_tokens: int = field(\n default=64,\n metadata={\"help\": \"Maximum number of tokens to generate per completion.\"},\n )\n max_length: int = field(\n default=512,\n metadata={\n \"help\": \"Maximum total length of the sequence (prompt + completion) used to compute log probabilities. If \"\n \"the sequence exceeds this limit, the leftmost tokens will be truncated to preserve as much of the \"\n \"completion as possible.\"\n },\n )\n temperature: float = field(\n default=0.9,\n metadata={\"help\": \"Temperature for sampling. The higher the temperature, the more random the completions.\"},\n )\n top_p: float = field(\n default=1.0,\n metadata={\n \"help\": \"Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. \"\n \"Set to 1.0 to consider all tokens.\"\n },\n )\n top_k: int = field(\n default=0,\n metadata={\n \"help\": \"Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, \"\n \"top-k-filtering is disabled and all tokens are considered.\"\n },\n )\n min_p: float | None = field(\n default=None,\n metadata={\n \"help\": \"Minimum token probability, which will be scaled by the probability of the most likely token. It \"\n \"must be a value between 0.0 and 1.0. Typical values are in the 0.01-0.2 range.\"\n },\n )\n repetition_penalty: float = field(\n default=1.0,\n metadata={\n \"help\": \"Float that penalizes new tokens based on whether they appear in the prompt and the generated \"\n \"text so far. Values > 1.0 encourage the model to use new tokens, while values < 1.0 encourage the model \"\n \"to repeat tokens.\"\n },\n )\n generation_kwargs: dict | None = field(\n default=None,\n metadata={\n \"help\": \"Additional keyword arguments to pass to `GenerationConfig` (if using transformers) or \"\n \"`SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the \"\n \"generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that \"\n \"conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.\"\n },\n )\n use_transformers_paged: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use the `transformers` paged implementation for generation. If set to `True`, the \"\n \"`transformers` paged implementation will be used for generation instead of the default padded \"\n \"implementation. This parameter is only effective when `use_vllm` is set to `False`.\"\n },\n )\n cache_implementation: str | None = field(\n default=None,\n metadata={\"help\": \"Implementation of the cache method for faster generation when use_vllm is set to False.\"},\n )\n missing_eos_penalty: float | None = field(\n default=None,\n metadata={\n \"help\": \"Penalty applied to the score when the model fails to generate an EOS token. This is useful to \"\n \"encourage to generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be \"\n \"a positive value.\"\n },\n )\n beta: list[float] = field(\n default_factory=lambda: [0.1],\n metadata={\n \"help\": \"Parameter controlling the deviation from the reference model. Higher ฮฒ means less deviation from \"\n \"the reference model. For the IPO loss (`loss_type='ipo'`), ฮฒ is the regularization parameter denoted by \"\n \"ฯ„ in the [paper](https://huggingface.co/papers/2310.12036). If a list of floats is provided then the ฮฒ \"\n \"is selected for each new epoch and the last ฮฒ is used for the rest of the epochs.\"\n },\n )\n loss_type: str = field(\n default=\"sigmoid\",\n metadata={\n \"help\": \"Type of loss to use.\",\n \"choices\": [\"sigmoid\", \"ipo\"],\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropout in the model.\"},\n )\n use_vllm: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use vLLM for generating completions. Requires vLLM to be installed \"\n \"(`pip install trl[vllm]`).\"\n },\n )\n vllm_model_impl: str = field(\n default=\"vllm\",\n metadata={\n \"help\": \"Model implementation to use for vLLM. Must be one of `transformers` or `vllm`. `transformers`: \"\n \"Use the `transformers` backend for model implementation. `vllm`: Use the `vllm` library for \"\n \"model implementation.\"\n },\n )\n vllm_structured_outputs_regex: str | None = field(\n default=None,\n metadata={\"help\": \"Regex for vLLM structured outputs. If `None` (default), structured outputs is disabled.\"},\n )\n vllm_gpu_memory_utilization: float | None = field(\n default=0.55,\n metadata={\n \"help\": \"Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set \"\n \"to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when \"\n \"launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.\",\n },\n )\n vllm_mode: str = field(\n default=\"colocate\",\n metadata={\n \"help\": \"Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `'server'` or \"\n \"`'colocate'`. `'server'`: The trainer will send generation requests to a separate vLLM server. Make sure \"\n \"a TRL vLLM server is running (start with `trl vllm-serve`). `'colocate'`: vLLM will run in the same \"\n \"process and share the training GPUs. This avoids the need for a separate server but may cause resource \"\n \"contention with training.\",\n },\n )\n vllm_server_base_url: str | None = field(\n default=None,\n metadata={\n \"help\": \"Base URL for the vLLM server (e.g., 'http://localhost:8000'). If provided, `vllm_server_host` \"\n \"and `vllm_server_port` are ignored.\",\n },\n )\n vllm_server_host: str = field(\n default=\"0.0.0.0\",\n metadata={\"help\": \"Host of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.\"},\n )\n vllm_server_port: int = field(\n default=8000,\n metadata={\"help\": \"Port of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.\"},\n )\n vllm_server_timeout: float = field(\n default=240.0,\n metadata={\n \"help\": \"Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up \"\n \"after the timeout, a `ConnectionError` is raised.\",\n },\n )\n vllm_group_port: int = field(\n default=51216,\n metadata={\n \"help\": \"Port number for the weight update group. This is used to communicate with the vLLM server. \"\n \"Unless the port is occupied, there is no need to change it.\",\n },\n )\n vllm_tensor_parallel_size: int = field(\n default=1,\n metadata={\n \"help\": \"Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set \"\n \"to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when \"\n \"launching the vLLM server via the `--vllm_tensor_parallel_size` flag.\",\n },\n )\n vllm_enable_sleep_mode: bool = field(\n default=False,\n metadata={\n \"help\": \"Enable vLLM sleep mode to offload weights/cache during the optimizer step. Keeps GPU memory \"\n \"usage low, but waking the engine adds hostโ€“device transfer latency.\"\n },\n )\n ds3_gather_for_generation: bool = field(\n default=True,\n metadata={\n \"help\": \"This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for \"\n \"generation, improving generation speed. However, disabling this option allows training models that \"\n \"exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option \"\n \"is not compatible with vLLM generation.\"\n },\n )\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model \"\n \"from a string.\"\n },\n )\n reward_weights: list[float] | None = field(\n default=None,\n metadata={\n \"help\": \"Weights for combining multiple reward functions. Must match the number of reward functions. \"\n \"If None, all reward functions are equally weighted.\"\n },\n )\n\n def __post_init__(self):\n super().__post_init__()\n\n if hasattr(self.beta, \"__len__\") and len(self.beta) == 1:\n self.beta = self.beta[0]\n\n if self.max_new_tokens >= self.max_length:\n warnings.warn(\n f\"The configuration has `max_new_tokens` ({self.max_new_tokens}) >= `max_length` ({self.max_length}). \"\n \"This will cause prompts to be truncated or completely removed in the forward pass. \"\n \"To preserve prompts, ensure e.g. `max_length > max_new_tokens + 512`. \",\n stacklevel=3,\n )\n" }, { "path": "trl/experimental/online_dpo/online_dpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport os\nimport re\nimport textwrap\nfrom collections.abc import Callable\nfrom contextlib import nullcontext\nfrom pathlib import Path\nfrom typing import Any\n\nimport jinja2\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.utils.data\nimport transformers\nfrom accelerate import logging\nfrom accelerate.utils import broadcast_object_list, gather_object, is_peft_model\nfrom datasets import Dataset\nfrom packaging.version import Version\nfrom torch.distributed.fsdp import FullyShardedDataParallel as FSDP\nfrom torch.utils.data import IterableDataset\nfrom transformers import (\n AutoModelForCausalLM,\n AutoModelForSequenceClassification,\n AutoTokenizer,\n DataCollator,\n GenerationConfig,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n is_bitsandbytes_available,\n)\nfrom transformers.models.auto.modeling_auto import MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES\nfrom transformers.trainer_utils import EvalPrediction\nfrom transformers.training_args import OptimizerNames\nfrom transformers.utils import is_flash_attn_2_available, is_peft_available, is_sagemaker_mp_enabled\n\nfrom ...data_utils import apply_chat_template, is_conversational, maybe_apply_chat_template\nfrom ...extras.profiling import profiling_context\nfrom ...generation.vllm_client import VLLMClient\nfrom ...import_utils import is_vllm_available\nfrom ...models.utils import prepare_deepspeed, prepare_fsdp, unwrap_model_for_generation\nfrom ...trainer.base_trainer import _BaseTrainer\nfrom ...trainer.utils import disable_dropout_in_model, ensure_master_addr_port, get_config_model_id, pad\nfrom ..judges import BasePairwiseJudge\nfrom ..utils import (\n SIMPLE_CHAT_TEMPLATE,\n DPODataCollatorWithPadding,\n create_reference_model,\n empty_cache,\n prepare_peft_model,\n truncate_right,\n)\nfrom .online_dpo_config import OnlineDPOConfig\n\n\nif is_peft_available():\n from peft import PeftConfig, PeftModel\n\n\nif is_sagemaker_mp_enabled():\n from smdistributed.modelparallel import __version__ as SMP_VERSION\n\n IS_SAGEMAKER_MP_POST_1_10 = Version(SMP_VERSION) >= Version(\"1.10\")\n\nelse:\n IS_SAGEMAKER_MP_POST_1_10 = False\n\n\nif Version(transformers.__version__) >= Version(\"5.2.0\"):\n from transformers.trainer_pt_utils import nested_gather\n\n\nif is_vllm_available():\n from vllm import LLM, SamplingParams\n from vllm.sampling_params import StructuredOutputsParams\n\nif is_bitsandbytes_available():\n import bitsandbytes as bnb\n\nlogger = logging.get_logger(__name__)\n\n# A reward function can be a string, interpreted as a model ID and loaded as a pretrained model, a pretrained model, or\n# a callable that returns a list of floats (the rewards). The callable receives prompts, completions, and additional\n# arguments from the trainer (refer to the trainer's source for details). To ensure forward compatibility, it should\n# accept **kwargs.\nRewardFunc = str | PreTrainedModel | Callable[..., list[float | None]]\n\n\nclass OnlineDPOTrainer(_BaseTrainer):\n r\"\"\"\n Initialize OnlineDPOTrainer.\n\n Args:\n model (`str | nn.Module | PreTrainedModel`):\n Model to be trained. Can be either:\n\n - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using [`~transformers.AutoModelForCausalLM.from_pretrained`] with the keyword arguments in\n `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported.\n ref_model ([`~transformers.PreTrainedModel`] or `torch.nn.Module` or `None`):\n The reference model to use for training. If None is specified, the reference model will be created from the\n model.\n judge ([`experimental.judges.BasePairwiseJudge`]):\n The judge to use for pairwise comparison of model completions.\n reward_funcs (`RewardFunc | list[RewardFunc]`, *optional*):\n Reward functions to be used for computing the rewards. To compute the rewards, we call all the reward\n functions with the prompts and completions and sum the rewards. Can be either:\n\n - A single reward function: Can be a string (path to model), a [`~transformers.PreTrainedModel`], or a\n custom callable function.\n - A list of reward functions: Must all be of compatible types.\n\n Note: Only one of `judge`, or `reward_funcs` should be provided.\n args ([`experimental.online_dpo.OnlineDPOConfig`]):\n The online DPO config arguments to use for training.\n data_collator ([`~transformers.DataCollator`]):\n The data collator to use for training. If None is specified, the default data collator\n ([`experimental.utils.DPODataCollatorWithPadding`]) will be used which will pad the sequences to the\n maximum length of the sequences in the batch, given a dataset of paired sequences.\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n The dataset to use for training.\n eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Dataset | IterableDataset]`):\n The dataset to use for evaluation.\n processing_class ([`~transformers.PreTrainedTokenizerBase`] or [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. If provided, will be used to automatically process the inputs\n for the model, and it will be saved along the model to make it easier to rerun an interrupted training or\n reuse the fine-tuned model.\n reward_processing_classes ([`~transformers.PreTrainedTokenizerBase`] or `list[PreTrainedTokenizerBase]`, *optional*):\n Processing classes corresponding to the reward functions specified in `reward_funcs`. Can be either:\n\n - A single processing class: Used when `reward_funcs` contains only one reward function.\n - A list of processing classes: Must match the order and length of the reward functions in `reward_funcs`.\n\n If set to `None`, the tokenizer for each model-based reward function is automatically loaded using\n [`~transformers.AutoTokenizer.from_pretrained`].\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration used to wrap the model. If `None`, the model is not wrapped.\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to\n metric values.\n callbacks (`list[transformers.TrainerCallback]`):\n The callbacks to use for training.\n optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):\n The optimizer and scheduler to use for training.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):\n The function to use to preprocess the logits before computing the metrics.\n \"\"\"\n\n _tag_names = [\"trl\", \"online-dpo\"]\n _name = \"Online DPO\"\n _paper = {\n \"title\": \"Direct Language Model Alignment from Online AI Feedback\",\n \"id\": \"2402.04792\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{guo2024direct,\n title = {{Direct Language Model Alignment from Online AI Feedback}},\n author = {Shangmin Guo and Biao Zhang and Tianlin Liu and Tianqi Liu and Misha Khalman and Felipe Llinares and Alexandre Ram{\\'{e}} and Thomas Mesnard and Yao Zhao and Bilal Piot and Johan Ferret and Mathieu Blondel},\n year = 2024,\n eprint = {arXiv:2402.04792}\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module | str,\n ref_model: PreTrainedModel | nn.Module | None = None,\n reward_funcs: RewardFunc | list[RewardFunc] | None = None,\n judge: BasePairwiseJudge | None = None,\n args: OnlineDPOConfig | None = None,\n data_collator: DataCollator | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | IterableDataset | dict[str, Dataset | IterableDataset] | None = None,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin | None = None,\n reward_processing_classes: PreTrainedTokenizerBase | list[PreTrainedTokenizerBase] | None = None,\n peft_config: \"PeftConfig | None\" = None,\n compute_metrics: Callable[[EvalPrediction], dict] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n ) -> None:\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n\n if ref_model is model:\n raise ValueError(\n \"`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the \"\n \"same as `model`, either omit the `ref_model` argument or pass `None`.\"\n )\n\n self.ref_model = ref_model\n\n # Validate reward configuration - must have exactly one of: judge, or reward_funcs\n reward_configs = sum(x is not None for x in [judge, reward_funcs])\n if reward_configs == 0:\n raise ValueError(\"One of `judge` or `reward_funcs` must be provided.\")\n elif reward_configs > 1:\n if judge is not None:\n logger.warning(\n \"Both `judge` and `reward_funcs` are provided. Using `judge` and ignoring `reward_funcs`.\",\n UserWarning,\n )\n reward_funcs = None\n self.judge = judge\n\n # Handle reward_funcs\n if reward_funcs is not None:\n if not isinstance(reward_funcs, list):\n reward_funcs = [reward_funcs]\n self.reward_func_names = []\n\n # Process reward functions (convert strings to models, collect names)\n model_init_kwargs = args.model_init_kwargs or {}\n for i, reward_func in enumerate(reward_funcs):\n if isinstance(reward_func, str):\n # Load model from string path\n reward_funcs[i] = AutoModelForSequenceClassification.from_pretrained(\n reward_func, num_labels=1, **model_init_kwargs\n )\n if isinstance(reward_funcs[i], nn.Module):\n self.reward_func_names.append(get_config_model_id(reward_funcs[i].config).split(\"/\")[-1])\n else:\n self.reward_func_names.append(reward_funcs[i].__name__)\n self.reward_funcs = reward_funcs\n\n # Handle reward processing classes for reward_funcs\n if reward_processing_classes is None:\n reward_processing_classes = [None] * len(reward_funcs)\n elif not isinstance(reward_processing_classes, list):\n reward_processing_classes = [reward_processing_classes]\n else:\n if len(reward_processing_classes) != len(reward_funcs):\n raise ValueError(\n \"The number of reward processing classes must match the number of reward functions.\"\n )\n\n self.reward_processing_classes = []\n for reward_processing_class_i, reward_func in zip(reward_processing_classes, reward_funcs, strict=True):\n if isinstance(reward_func, PreTrainedModel):\n if reward_processing_class_i is None:\n reward_processing_class_i = AutoTokenizer.from_pretrained(reward_func.config._name_or_path)\n if reward_processing_class_i.pad_token_id is None:\n reward_processing_class_i.pad_token = reward_processing_class_i.eos_token\n # Set pad token ID on reward model config\n reward_func.config.pad_token_id = reward_processing_class_i.pad_token_id\n self.reward_processing_classes.append(reward_processing_class_i)\n else:\n self.reward_funcs = None\n self.reward_func_names = []\n self.reward_processing_classes = []\n\n # Handle reward_weights\n if reward_funcs is not None:\n if args.reward_weights is not None:\n if len(args.reward_weights) != len(self.reward_funcs):\n raise ValueError(\n f\"Number of reward weights ({len(args.reward_weights)}) must match number of reward \"\n f\"functions ({len(self.reward_funcs)})\"\n )\n self.reward_weights = torch.tensor(args.reward_weights, dtype=torch.float32)\n else:\n self.reward_weights = torch.ones(len(self.reward_funcs), dtype=torch.float32)\n else:\n self.reward_weights = None\n\n if args.missing_eos_penalty is not None and reward_funcs is None and judge is None:\n raise ValueError(\"`missing_eos_penalty` is only supported when `reward_funcs` is provided.\")\n\n if args is None:\n raise ValueError(\"`args` must be provided.\")\n\n # Check that the processing_class is provided\n if processing_class is None:\n raise ValueError(\"`processing_class` must be provided.\")\n\n model_init_kwargs = args.model_init_kwargs or {}\n if isinstance(model, str):\n model_id = model\n\n # Handle dtype in model_init_kwargs\n dtype = model_init_kwargs.get(\"dtype\", \"auto\")\n if isinstance(dtype, torch.dtype) or dtype == \"auto\" or dtype is None:\n pass\n elif isinstance(dtype, str):\n dtype = getattr(torch, dtype)\n model_init_kwargs[\"dtype\"] = dtype\n else:\n raise ValueError(\n \"Invalid `dtype` passed to `OnlineDPOConfig`. Expected either 'auto' or a string \"\n f\"representing a `torch.dtype` (e.g., 'float32'), but got {dtype}.\"\n )\n model_init_kwargs[\"device_map\"] = model_init_kwargs.get(\"device_map\", \"auto\")\n\n model = AutoModelForCausalLM.from_pretrained(model_id, **model_init_kwargs)\n else:\n if args.model_init_kwargs is not None:\n raise ValueError(\n \"You passed `model_init_kwargs` to the `OnlineDPOConfig`, but your model is already instantiated. \"\n \"This argument can only be used when the `model` argument is a string.\"\n )\n self.is_encoder_decoder = model.config.is_encoder_decoder\n self.is_vision_model = model.config.model_type in MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES.keys()\n\n if peft_config is not None or (is_peft_available() and isinstance(model, PeftModel)):\n model = prepare_peft_model(model, peft_config, args)\n\n # Enable gradient checkpointing if requested\n if args.gradient_checkpointing:\n model = self._enable_gradient_checkpointing(model, args)\n\n # Disable dropout in the model and reference model\n if args.disable_dropout:\n disable_dropout_in_model(model)\n if self.ref_model is not None:\n disable_dropout_in_model(self.ref_model)\n\n # Handle the ref_model\n # Usually, the user wants the ref model to be the initial version of the model. When using PEFT, it's easy to\n # get the ref model, as it's just the model with a disabled adapter. When not using PEFT, we need to create\n # the ref model from the model by copying it and disable the gradients and set it in evaluation mode.\n if ref_model is None: # No ref model provided, the most common case\n if peft_config is None:\n self.ref_model = create_reference_model(model) # copy, disable gradients, set eval mode\n else:\n self.ref_model = None # we don't need a ref model here, we can just disable the adapter.\n else: # rare case, the user provided a ref model\n self.ref_model = ref_model\n self.ref_model.eval()\n\n # Disable the gradient and set the reward model in eval mode\n if reward_funcs is not None:\n for reward_func in reward_funcs:\n if isinstance(reward_func, PreTrainedModel):\n reward_func.eval()\n\n self.max_length = args.max_length\n\n self.stats = {\n \"objective/kl\": [],\n \"objective/entropy\": [],\n \"objective/non_score_reward\": [],\n \"rewards/chosen\": [],\n \"rewards/rejected\": [],\n \"rewards/accuracies\": [],\n \"rewards/margins\": [],\n \"logps/chosen\": [],\n \"logps/rejected\": [],\n \"val/contain_eos_token\": [],\n \"beta\": [],\n }\n if self.reward_funcs is not None:\n self.stats[\"objective/rlhf_reward\"] = []\n self.stats[\"objective/scores_margin\"] = []\n self.stats[\"objective/scores\"] = []\n\n # Store generation parameters for later use\n self.use_vllm = args.use_vllm\n self.num_generations = 2 # Generate 2 completions per prompt for Online DPO\n self.temperature = args.temperature\n self.top_p = args.top_p\n self.top_k = args.top_k\n self.min_p = args.min_p\n self.repetition_penalty = args.repetition_penalty\n self.use_transformers_paged = args.use_transformers_paged\n self.vllm_mode = args.vllm_mode if args.use_vllm else None\n self.vllm_gpu_memory_utilization = args.vllm_gpu_memory_utilization\n self.vllm_tensor_parallel_size = args.vllm_tensor_parallel_size\n self.vllm_model_impl = args.vllm_model_impl\n\n # Handle pad token for processors or tokenizers\n if isinstance(processing_class, ProcessorMixin):\n tokenizer = processing_class.tokenizer\n elif isinstance(processing_class, PreTrainedTokenizerBase):\n tokenizer = processing_class\n else:\n raise TypeError(\"The `processing_class` must be either a `PreTrainedTokenizerBase` or a `ProcessorMixin`\")\n\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n self.pad_token = tokenizer.pad_token\n self.pad_token_id = tokenizer.pad_token_id\n self.eos_token_id = tokenizer.eos_token_id\n\n # Vision tokens for VLM support\n self.image_token_id = getattr(processing_class, \"image_token_id\", None)\n self.vision_start_token_id = getattr(processing_class, \"vision_start_token_id\", None)\n self.vision_end_token_id = getattr(processing_class, \"vision_end_token_id\", None)\n # Get the image token string for token collapsing\n self.image_token = None\n if self.image_token_id is not None:\n self.image_token = tokenizer.decode([self.image_token_id])\n\n # Define the collator if not provided\n if data_collator is None:\n data_collator = DPODataCollatorWithPadding(pad_token_id=self.pad_token_id)\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n # Add tags for models that have been loaded with the correct transformers version\n if hasattr(self.model, \"add_model_tags\"):\n self.model.add_model_tags(self._tag_names)\n\n self._beta = args.beta\n\n # Set up generation configuration and vLLM after super().__init__\n if self.use_vllm:\n if not is_vllm_available():\n raise ImportError(\n \"vLLM is not available and `use_vllm` is set to True. Please install vLLM with \"\n \"`pip install trl[vllm]` to use it.\"\n )\n\n if self.vllm_mode == \"server\":\n if self.accelerator.is_main_process:\n if args.vllm_server_base_url is not None:\n base_url = args.vllm_server_base_url\n else:\n base_url = f\"http://{args.vllm_server_host}:{args.vllm_server_port}\"\n self.vllm_client = VLLMClient(\n base_url=base_url, group_port=args.vllm_group_port, connection_timeout=args.vllm_server_timeout\n )\n\n # Determine device type (supports cuda, xpu, etc.)\n accelerator_type = torch.accelerator.current_accelerator().type\n current_device = getattr(torch, accelerator_type).current_device()\n self.vllm_client.init_communicator(device=current_device)\n else:\n self.vllm_client = None\n elif self.vllm_mode == \"colocate\":\n # vLLM dynamically adjusts the size of the key-value cache based on available GPU memory at instantiation.\n # A larger cache size improves speed, so we would expect gpu_memory_utilization=1.\n # However, at this stage, the optimizer's weights are not yet loaded onto the GPU; they will be loaded\n # after the first optimizer step and remain in GPU memory throughout training. So we must reserve enough\n # space for them.\n # Configure vLLM parameters\n vllm_quantization = None\n if is_bitsandbytes_available():\n for _, module in model.named_modules():\n if isinstance(module, bnb.nn.Linear4bit):\n vllm_quantization = \"bitsandbytes\"\n break\n elif isinstance(module, bnb.nn.Linear8bitLt):\n raise ValueError(\"vLLM does not support in-flight 8-bit quantization.\")\n vllm_kwargs = {\n \"model\": model.name_or_path,\n \"tensor_parallel_size\": self.vllm_tensor_parallel_size,\n \"gpu_memory_utilization\": self.vllm_gpu_memory_utilization,\n \"model_impl\": self.vllm_model_impl,\n \"max_num_seqs\": self.args.per_device_train_batch_size * self.vllm_tensor_parallel_size,\n \"max_model_len\": args.max_length + args.max_new_tokens, # max_length includes prompt + completion\n \"distributed_executor_backend\": \"external_launcher\",\n # Feed identical seed for tp groups to ensure sampling results are the same across workers\n \"seed\": self.accelerator.process_index // self.vllm_tensor_parallel_size,\n # Latest vLLM v1 memory profiler is misled by the high default value (i.e., 32768)\n \"max_num_batched_tokens\": 4096,\n \"enable_sleep_mode\": self.args.vllm_enable_sleep_mode,\n \"quantization\": vllm_quantization,\n }\n\n # vLLM requires the environment variables to be set for distributed training.\n os.environ[\"RANK\"] = str(self.accelerator.process_index)\n os.environ[\"LOCAL_RANK\"] = str(self.accelerator.local_process_index)\n os.environ[\"WORLD_SIZE\"] = str(self.accelerator.num_processes)\n # Ensure distributed rendezvous variables are set without colliding across concurrent runs\n ensure_master_addr_port()\n\n self.llm = LLM(**vllm_kwargs)\n if self.args.vllm_enable_sleep_mode:\n self.llm.sleep(level=2)\n else:\n raise ValueError(f\"vllm_mode must be either 'server' or 'colocate', got '{self.vllm_mode}'.\")\n # vLLM specific sampling arguments\n self.structured_outputs_regex = args.vllm_structured_outputs_regex\n self._last_loaded_step = -1 # tag to avoid useless loading during grad accumulation\n\n # Set up vLLM generation config\n generation_params = {\n \"n\": 2, # 2 generations per prompt for Online DPO\n \"repetition_penalty\": self.repetition_penalty,\n \"temperature\": self.temperature,\n \"top_p\": self.top_p,\n \"top_k\": self.top_k,\n \"min_p\": 0.0 if self.min_p is None else self.min_p,\n \"max_tokens\": args.max_new_tokens,\n \"detokenize\": False, # to avoid vllm to decode (we don't need it)\n }\n if args.generation_kwargs is not None:\n generation_params.update(args.generation_kwargs)\n if self.structured_outputs_regex is not None:\n if generation_params.get(\"structured_outputs\") is not None:\n logger.warning(\n \"Both `vllm_structured_outputs_regex` and `generation_kwargs['structured_outputs']` are set; \"\n \"`vllm_structured_outputs_regex` takes precedence.\"\n )\n generation_params[\"structured_outputs\"] = StructuredOutputsParams(regex=self.structured_outputs_regex)\n elif isinstance(generation_params.get(\"structured_outputs\"), dict):\n structured_outputs_dict = generation_params.get(\"structured_outputs\")\n generation_params[\"structured_outputs\"] = StructuredOutputsParams(**structured_outputs_dict)\n self.generation_config = SamplingParams(**generation_params)\n\n # When using vLLM, the main process is responsible for loading the model weights. This can cause process\n # desynchronization and seems to lead to DeepSpeed hanging during initialization. To prevent this, we\n # synchronize all processes after vLLM has been fully initialized.\n self.accelerator.wait_for_everyone()\n else:\n # Set up transformers generation config\n generation_kwargs = {\n \"max_new_tokens\": args.max_new_tokens,\n \"do_sample\": True,\n \"pad_token_id\": self.pad_token_id,\n \"bos_token_id\": tokenizer.bos_token_id,\n \"eos_token_id\": self.eos_token_id,\n \"temperature\": self.temperature,\n \"top_k\": self.top_k,\n \"top_p\": self.top_p,\n \"repetition_penalty\": self.repetition_penalty,\n \"use_cache\": True if not self.args.gradient_checkpointing else False,\n }\n # Add min_p if supported\n if self.min_p is not None:\n generation_kwargs[\"min_p\"] = self.min_p\n if args.generation_kwargs is not None:\n generation_kwargs.update(args.generation_kwargs)\n # Remove None values\n generation_kwargs = {k: v for k, v in generation_kwargs.items() if v is not None}\n self.generation_config = GenerationConfig(**generation_kwargs)\n # Keep training-specific generation kwargs to overwrite model's original generation config\n self.generation_kwargs = generation_kwargs\n\n if self.ref_model is not None:\n if self.is_deepspeed_enabled:\n self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)\n elif self.is_fsdp_enabled:\n self.ref_model = prepare_fsdp(self.ref_model, self.accelerator)\n else:\n self.ref_model = self.accelerator.prepare_model(self.ref_model, evaluation_mode=True)\n if self.reward_funcs is not None:\n for i, reward_func in enumerate(self.reward_funcs):\n if isinstance(reward_func, PreTrainedModel):\n if self.is_deepspeed_enabled:\n self.reward_funcs[i] = prepare_deepspeed(reward_func, self.accelerator)\n else:\n # set device placement to True to make `prepare_model` move `reward_func` to device when using fsdp\n self.reward_funcs[i] = self.accelerator.prepare_model(\n reward_func, evaluation_mode=True, device_placement=True\n )\n\n @property\n def beta(self):\n if isinstance(self._beta, list):\n epoch = self.state.epoch\n return self._beta[epoch] if epoch < len(self._beta) else self._beta[-1]\n else:\n return self._beta\n\n @staticmethod\n def tokenize_row(feature, is_encoder_decoder: bool, tokenizer: PreTrainedTokenizerBase) -> dict[str, Any]:\n \"\"\"Tokenize a single row from a DPO specific dataset.\"\"\"\n if not is_encoder_decoder:\n batch = tokenizer(feature[\"prompt\"], add_special_tokens=False)\n # Add BOS token to head of prompt. Avoid adding if it's already there\n if tokenizer.bos_token_id is not None:\n prompt_len_input_ids = len(batch[\"input_ids\"])\n if prompt_len_input_ids == 0 or tokenizer.bos_token_id != batch[\"input_ids\"][0]:\n batch[\"input_ids\"] = [tokenizer.bos_token_id] + batch[\"input_ids\"]\n batch[\"attention_mask\"] = [1] + batch[\"attention_mask\"]\n else:\n batch = tokenizer(feature[\"prompt\"], add_special_tokens=True)\n batch = {f\"prompt_{key}\": value for key, value in batch.items()}\n return batch\n\n def _enable_gradient_checkpointing(self, model: PreTrainedModel, args: OnlineDPOConfig) -> PreTrainedModel:\n \"\"\"Enables gradient checkpointing for the model.\"\"\"\n # Ensure use_cache is disabled\n model.config.use_cache = False\n\n # Enable gradient checkpointing on the base model for PEFT\n if is_peft_model(model):\n model.base_model.gradient_checkpointing_enable()\n # Enable gradient checkpointing for non-PEFT models\n else:\n model.gradient_checkpointing_enable()\n\n model.enable_input_require_grads()\n return model\n\n def _generate_vllm(self, prompts, images=None):\n eos_token_id = self.eos_token_id\n pad_token_id = self.pad_token_id\n\n # Generate completion_ids and prompt_ids based on mode\n if self.vllm_mode == \"server\":\n completion_ids, prompt_ids = self._generate_vllm_server(prompts, images)\n elif self.vllm_mode == \"colocate\":\n completion_ids, prompt_ids = self._generate_vllm_colocate(prompts, images)\n\n # Shared padding, masking, and tensor conversion logic\n max_prompt_length = max(len(ids) for ids in prompt_ids)\n prompt_mask = [[0] * (max_prompt_length - len(ids)) + [1] * len(ids) for ids in prompt_ids]\n prompt_ids = [[pad_token_id] * (max_prompt_length - len(ids)) + ids for ids in prompt_ids]\n max_tokens = self.generation_config.max_tokens\n completion_mask = [[1] * len(ids) + [0] * (max_tokens - len(ids)) for ids in completion_ids]\n completion_ids = [\n ids + [eos_token_id] if ids[-1] != eos_token_id and len(ids) < max_tokens else ids\n for ids in completion_ids\n ]\n completion_ids = [ids + [pad_token_id] * (max_tokens - len(ids)) for ids in completion_ids]\n\n # Convert to tensors\n prompt_ids = torch.tensor(prompt_ids, device=self.accelerator.device)\n prompt_mask = torch.tensor(prompt_mask, device=self.accelerator.device)\n completion_ids = torch.tensor(completion_ids, device=self.accelerator.device)\n completion_mask = torch.tensor(completion_mask, device=self.accelerator.device)\n\n return prompt_ids, prompt_mask, completion_ids, completion_mask\n\n def _generate_vllm_server(self, prompts, images=None):\n \"\"\"Generate completions using vLLM server mode\"\"\"\n has_images = images is not None\n\n # Update vLLM server weights if needed\n if hasattr(self, \"_last_loaded_step\") and self.state.global_step != self._last_loaded_step:\n self._move_model_to_vllm()\n self._last_loaded_step = self.state.global_step\n elif not hasattr(self, \"_last_loaded_step\"):\n self._move_model_to_vllm()\n self._last_loaded_step = self.state.global_step\n\n # Apply chat template if conversational\n if is_conversational({\"prompt\": prompts[0]}):\n prompts_text = [apply_chat_template({\"prompt\": p}, self.processing_class)[\"prompt\"] for p in prompts]\n else:\n prompts_text = prompts\n # Gather all prompts to main process\n all_prompts = gather_object(prompts_text)\n if has_images:\n all_images = gather_object(images)\n\n if self.accelerator.is_main_process:\n # Since 'prompts' contains 'num_generations' duplicates, we first take unique prompts, and generate\n # num_generations outputs for each one. This is faster than generating outputs for each duplicate\n # prompt individually.\n ordered_set_of_prompts = all_prompts[:: self.num_generations]\n if has_images:\n ordered_set_of_images = [\n [img] if img is not None else None for img in all_images[:: self.num_generations]\n ]\n else:\n ordered_set_of_images = None\n completion_ids = self.vllm_client.generate(\n prompts=ordered_set_of_prompts,\n images=ordered_set_of_images,\n n=self.num_generations,\n repetition_penalty=self.repetition_penalty,\n temperature=self.temperature,\n top_p=self.top_p,\n top_k=-1 if self.top_k is None else self.top_k,\n min_p=0.0 if self.min_p is None else self.min_p,\n max_tokens=self.generation_config.max_tokens,\n structured_outputs_regex=self.structured_outputs_regex\n if hasattr(self, \"structured_outputs_regex\")\n else None,\n generation_kwargs=self.args.generation_kwargs,\n )[\"completion_ids\"]\n # Flatten: each prompt generates 2 completions\n completion_ids = [[comp_id] for prompt_completions in completion_ids for comp_id in prompt_completions]\n else:\n completion_ids = [None] * (len(all_prompts) * 2)\n\n # Broadcast completions to all processes\n completion_ids = broadcast_object_list(completion_ids, from_process=0)\n\n # Each process takes its slice\n process_slice = slice(\n self.accelerator.process_index * len(prompts) * 2,\n (self.accelerator.process_index + 1) * len(prompts) * 2,\n )\n completion_ids = completion_ids[process_slice]\n\n # Create prompt_ids by tokenizing locally\n prompt_inputs = self.processing_class(\n text=prompts_text,\n return_tensors=\"pt\",\n padding=True,\n padding_side=\"left\",\n add_special_tokens=False,\n )\n prompt_ids = []\n for prompt_tokens in prompt_inputs[\"input_ids\"]:\n prompt_ids.extend([prompt_tokens.tolist(), prompt_tokens.tolist()]) # 2 copies for 2 completions\n return completion_ids, prompt_ids\n\n def _generate_vllm_colocate(self, prompts, images=None):\n \"\"\"Generate completions using vLLM colocate mode\"\"\"\n if self.args.vllm_enable_sleep_mode:\n # wake up colocated vLLM instances if needed\n torch.cuda.empty_cache() # required to avoid OOM in some cases\n self.llm.wake_up(tags=[\"weights\"])\n\n # Update model weights if needed - only after gradient accumulation completes\n if self.state.global_step != self._last_loaded_step:\n self._move_model_to_vllm()\n self._last_loaded_step = self.state.global_step\n\n # Apply chat template if conversational\n if is_conversational({\"prompt\": prompts[0]}):\n prompts_text = [apply_chat_template({\"prompt\": p}, self.processing_class)[\"prompt\"] for p in prompts]\n else:\n prompts_text = prompts\n\n # Prepare vLLM inputs with images if available\n if images is not None:\n vllm_inputs = []\n for prompt, image in zip(prompts_text, images, strict=True):\n if image is not None:\n vllm_inputs.append({\"prompt\": prompt, \"multi_modal_data\": {\"image\": image}})\n else:\n vllm_inputs.append(prompt)\n else:\n vllm_inputs = prompts_text\n\n if self.args.vllm_enable_sleep_mode:\n self.llm.wake_up(tags=[\"kv_cache\"])\n\n outputs = self.llm.generate(vllm_inputs, self.generation_config, use_tqdm=False)\n\n completion_ids = [list(output.outputs[i].token_ids) for i in range(2) for output in outputs]\n prompt_ids = [list(output.prompt_token_ids) for _ in range(2) for output in outputs]\n if self.args.vllm_enable_sleep_mode:\n self.llm.sleep(level=2)\n\n return completion_ids, prompt_ids\n\n def _sync_fsdp2_params_to_vllm(self, module: nn.Module):\n # For FSDP2, module.state_dict() already covers all parameters, so no need for recursion\n for name, param in module.state_dict().items():\n # When using PEFT, we need to recover the original parameter name\n name = name.removeprefix(\"base_model.model.\").replace(\".base_layer\", \"\")\n # Skip PEFT layers: they donโ€™t exist in vLLM, and they are merged already.\n if is_peft_model(module) and module.prefix in name:\n continue\n # When module to save, remove its prefix and discard the original module\n if \"original_module\" in name:\n continue\n name = self._fix_param_name_to_vllm(name, extra_prefixes=[\"modules_to_save.default.\"])\n\n if param.is_cpu:\n param = param.to(torch.device(\"cuda\"))\n param = param.full_tensor()\n\n if self.vllm_mode == \"server\" and self.accelerator.is_main_process:\n self.vllm_client.update_named_param(name, param)\n elif self.vllm_mode == \"colocate\":\n llm_model = self.llm.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(name, param)])\n\n def _move_model_to_vllm(self):\n # For DeepSpeed ZeRO-3 and FSDP, we need to gather all parameters before operations\n deepspeed_plugin = self.accelerator.state.deepspeed_plugin\n zero_stage_3 = deepspeed_plugin is not None and deepspeed_plugin.zero_stage == 3\n if zero_stage_3:\n import deepspeed\n\n gather_if_zero3 = deepspeed.zero.GatheredParameters\n else:\n gather_if_zero3 = nullcontext\n\n if is_peft_model(self.model):\n # With PEFT and FSDP/DeepSpeed ZeRO Stage 3, we must gather the full model at once before merging, as\n # merging adapters in a sharded manner is not supported.\n # TODO: does this work with FSDP?\n with gather_if_zero3(list(self.model.parameters())):\n self.model.merge_adapter()\n\n # Update vLLM weights while parameters are gathered\n if self.is_fsdp_enabled: # note if using FSDP, gather_if_zero3 is nullcontext\n # Update vLLM weights while parameters are gathered\n # For PEFT with FSDP we need to use the memory efficient post-order traversal\n fsdp_plugin = getattr(self.accelerator.state, \"fsdp_plugin\", None)\n fsdp_version = getattr(fsdp_plugin, \"fsdp_version\", 1) if fsdp_plugin else 1\n if fsdp_version == 1:\n self._sync_fsdp1_params_to_vllm(\n self.model\n ) # use memory-efficient post-order traversal for FSDP\n elif fsdp_version == 2:\n self._sync_fsdp2_params_to_vllm(self.model)\n else:\n # DeepSpeed ZeRO-3 with PEFT\n for name, param in self.model.named_parameters():\n # When using PEFT, we need to recover the original parameter name\n name = name.removeprefix(\"base_model.model.\").replace(\".base_layer\", \"\")\n # Skip PEFT layers: they donโ€™t exist in vLLM, and they are merged already.\n if self.model.prefix in name:\n continue\n # When module to save, remove its prefix and discard the original module\n if \"original_module\" in name:\n continue\n name = self._fix_param_name_to_vllm(name, extra_prefixes=[\"modules_to_save.default.\"])\n\n if self.vllm_mode == \"server\" and self.accelerator.is_main_process:\n self.vllm_client.update_named_param(name, param.data)\n elif self.vllm_mode == \"colocate\":\n llm_model = self.llm.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(name, param.data)])\n # Unmerge adapters while parameters are still gathered\n self.model.unmerge_adapter()\n # Parameters will automatically be repartitioned when exiting the context\n else:\n # For non-PEFT models, simply gather (if needed) and update each parameter individually.\n if self.is_fsdp_enabled:\n fsdp_plugin = getattr(self.accelerator.state, \"fsdp_plugin\", None)\n fsdp_version = getattr(fsdp_plugin, \"fsdp_version\", 1) if fsdp_plugin else 1\n if fsdp_version == 1:\n self._sync_fsdp1_params_to_vllm(self.model) # use memory-efficient post-order traversal for FSDP\n elif fsdp_version == 2:\n self._sync_fsdp2_params_to_vllm(self.model)\n else:\n for name, param in self.model.named_parameters():\n name = self._fix_param_name_to_vllm(name)\n with gather_if_zero3([param]):\n if self.vllm_mode == \"server\" and self.accelerator.is_main_process:\n self.vllm_client.update_named_param(name, param.data)\n elif self.vllm_mode == \"colocate\":\n llm_model = self.llm.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(name, param.data)])\n\n # Reset cache on vLLM\n if self.vllm_mode == \"server\" and self.accelerator.is_main_process:\n self.vllm_client.reset_prefix_cache()\n elif self.vllm_mode == \"colocate\":\n self.llm.reset_prefix_cache()\n\n def _sync_fsdp1_params_to_vllm(self, module: nn.Module, prefix: str = \"\", visited=None):\n \"\"\"Memory-efficient post-order traversal of FSDP modules to extract full parameters and sync with vLLM.\"\"\"\n # For FSDP1, we need to recurse into children and also use summon_full_params\n if visited is None:\n visited = set()\n for child_name, child_module in module.named_children():\n child_prefix = f\"{prefix}.{child_name}\" if prefix else child_name\n self._sync_fsdp1_params_to_vllm(\n child_module, prefix=child_prefix, visited=visited\n ) # recurse into the child\n\n if isinstance(module, FSDP):\n with FSDP.summon_full_params(module, recurse=False, writeback=False):\n for param_name, param in module.named_parameters():\n full_name = f\"{prefix}.{param_name}\" if prefix else param_name\n full_name = self._fix_param_name_to_vllm(full_name, extra_prefixes=[\"_fsdp_wrapped_module.\"])\n\n if full_name in visited:\n continue # skip FSDP subtrees already traversed\n visited.add(full_name)\n\n if self.vllm_mode == \"server\" and self.accelerator.is_main_process:\n self.vllm_client.update_named_param(full_name, param.data)\n elif self.vllm_mode == \"colocate\":\n llm_model = self.llm.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(full_name, param.data)])\n\n def _fix_param_name_to_vllm(self, name, extra_prefixes: list[str] | None = None):\n \"\"\"Clean parameter names for vLLM compatibility\"\"\"\n extra_prefixes = extra_prefixes or []\n prefixes = [\"_checkpoint_wrapped_module.\"] + extra_prefixes\n for prefix in prefixes:\n name = name.replace(prefix, \"\")\n return name\n\n def process_vision_row(\n self, features: dict[str, list | torch.Tensor], processing_class=None\n ) -> dict[str, list[int]]:\n \"\"\"\n Process a vision row for VLM models (adapted from DPO trainer)\n \"\"\"\n processor = processing_class or self.processing_class\n processed_features = processor(images=[features[\"image\"]], text=features[\"prompt\"], add_special_tokens=False)\n\n prompt_input_ids = processed_features[\"input_ids\"][0]\n\n # Create the output dict with required fields\n output = {\n \"prompt_input_ids\": prompt_input_ids,\n \"prompt_attention_mask\": processed_features[\"attention_mask\"][0],\n }\n\n # Add vision-specific fields\n if \"pixel_values\" in processed_features:\n output[\"pixel_values\"] = processed_features[\"pixel_values\"][0]\n if \"pixel_attention_mask\" in processed_features:\n output[\"pixel_attention_mask\"] = processed_features[\"pixel_attention_mask\"][0]\n if \"image_sizes\" in processed_features:\n output[\"image_sizes\"] = processed_features[\"image_sizes\"][0]\n\n return output\n\n def _generate(self, model, prompts, images=None):\n \"\"\"Generate completions using the model\"\"\"\n device = next(model.parameters()).device\n eos_token_id = self.eos_token_id\n pad_token_id = self.pad_token_id\n\n # Apply chat template and tokenize the input\n inputs = [{\"prompt\": prompt} for prompt in prompts]\n\n # Add images if provided (VLM support)\n if images is not None:\n for i, image in enumerate(images):\n inputs[i][\"image\"] = image\n\n # Apply chat template to get text prompts\n prompts_text = [maybe_apply_chat_template(x, self.processing_class)[\"prompt\"] for x in inputs]\n\n # Handle image token collapsing/removal\n # The chat template sometimes inserts a single image token into the prompt text. However, when this text is\n # later tokenized, the single image token string is expanded into multiple image token IDs, depending on the\n # image size. We need to handle this properly.\n if self.image_token is not None and images is not None:\n escaped_img_token = re.escape(self.image_token)\n # Search for the image token in the chat template\n if hasattr(self.processing_class, \"chat_template\") and self.processing_class.chat_template:\n if re.search(escaped_img_token, self.processing_class.chat_template):\n # Collapse repeated image tokens back into a single token\n prompts_text = [\n re.sub(rf\"({escaped_img_token})+\", self.image_token, text) for text in prompts_text\n ]\n else:\n # If the chat template doesn't use the image token, remove all instances\n if self.vision_end_token_id is not None:\n escaped_eoi_token = re.escape(\n self.processing_class.tokenizer.decode([self.vision_end_token_id])\n )\n prompts_text = [\n re.sub(rf\"({escaped_img_token})+{escaped_eoi_token}\", \"\", text) for text in prompts_text\n ]\n else:\n # If vision_end_token_id is None, just remove the image tokens\n prompts_text = [re.sub(rf\"({escaped_img_token})+\", \"\", text) for text in prompts_text]\n\n # Prepare kwargs for processing class\n kwargs = {}\n if images is not None:\n kwargs = {\"images\": [[img] for img in images]}\n\n # Process inputs using the processing class (handles both VLM and LLM)\n prompt_inputs = self.processing_class(\n text=prompts_text,\n return_tensors=\"pt\",\n padding=True,\n padding_side=\"left\",\n add_special_tokens=False,\n **kwargs,\n )\n\n prompt_inputs = {k: v.to(device) for k, v in prompt_inputs.items()}\n # Convert vision inputs to model's dtype for proper computation\n if \"pixel_values\" in prompt_inputs:\n # Handle DataParallel wrapped models\n model_dtype = getattr(model, \"dtype\", None)\n if model_dtype is None and hasattr(model, \"module\"):\n model_dtype = model.module.dtype\n if model_dtype is not None:\n prompt_inputs[\"pixel_values\"] = prompt_inputs[\"pixel_values\"].to(model_dtype)\n\n # Sample 2 completions per prompt of size `max_new_tokens` from the model\n prompt_ids = prompt_inputs[\"input_ids\"].repeat(2, 1)\n prompt_mask = prompt_inputs[\"attention_mask\"].repeat(2, 1)\n\n # Prepare vision inputs if available\n vision_generation_kwargs = {}\n if self.is_vision_model and images is not None:\n if \"pixel_values\" in prompt_inputs:\n vision_generation_kwargs[\"pixel_values\"] = prompt_inputs[\"pixel_values\"].repeat(2, 1, 1, 1)\n if \"pixel_attention_mask\" in prompt_inputs:\n vision_generation_kwargs[\"pixel_attention_mask\"] = prompt_inputs[\"pixel_attention_mask\"].repeat(2, 1)\n if \"image_sizes\" in prompt_inputs:\n vision_generation_kwargs[\"image_sizes\"] = prompt_inputs[\"image_sizes\"].repeat(2, 1)\n if \"image_grid_thw\" in prompt_inputs:\n vision_generation_kwargs[\"image_grid_thw\"] = prompt_inputs[\"image_grid_thw\"].repeat(2, 1)\n\n if self.use_transformers_paged:\n previous_attn = self.model_wrapped.config._attn_implementation\n\n if Version(transformers.__version__).release >= Version(\"5.0.0\").release:\n new_attn = \"paged|flash_attention_2\" if is_flash_attn_2_available() else \"paged|sdpa\"\n else:\n new_attn = \"paged_attention\" if is_flash_attn_2_available() else \"sdpa_paged\"\n self.model_wrapped.config._attn_implementation = new_attn\n with (\n profiling_context(self, \"transformers.generate_batch\"),\n unwrap_model_for_generation(\n model, self.accelerator, gather_deepspeed3_params=self.args.ds3_gather_for_generation\n ) as unwrapped_model,\n torch.no_grad(),\n FSDP.summon_full_params(self.model_wrapped, recurse=False) if self.is_fsdp_enabled else nullcontext(),\n ):\n # Cast to the appropriate dtype based on training configuration\n if self.args.bf16:\n unwrapped_model.to(torch.bfloat16)\n elif self.args.fp16:\n unwrapped_model.to(torch.float16)\n with torch.inference_mode():\n all_outputs = unwrapped_model.generate_batch(\n prompt_ids.tolist(),\n generation_config=self.generation_config,\n progress_bar=False,\n )\n unwrapped_model.train() # restore training mode, as generate_batch forces eval mode\n completion_ids = [output.generated_tokens for output in all_outputs.values()]\n completion_ids = [torch.tensor(ids, device=device) for ids in completion_ids]\n completion_ids = pad(completion_ids, padding_value=self.pad_token_id, padding_side=\"right\")\n prompt_completion_ids = torch.cat([prompt_ids, completion_ids], dim=1)\n # Restore the original attention implementation, training mode\n self.model_wrapped.config._attn_implementation = previous_attn\n\n # Extract completion_ids and create completion_mask\n prompt_length = prompt_ids.size(1)\n completion_ids = prompt_completion_ids[:, prompt_length:]\n completion_ids, completion_mask = truncate_right(completion_ids, eos_token_id, pad_token_id)\n\n return prompt_ids, prompt_mask, completion_ids, completion_mask\n else:\n # Regular generation path\n with (\n profiling_context(self, \"transformers.generate\"),\n unwrap_model_for_generation(\n model,\n self.accelerator,\n gather_deepspeed3_params=self.args.ds3_gather_for_generation,\n generation_kwargs=self.generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as unwrapped_model,\n torch.no_grad(),\n FSDP.summon_full_params(self.model_wrapped, recurse=False) if self.is_fsdp_enabled else nullcontext(),\n ):\n # Setup cache implementation if specified\n if self.args.cache_implementation is not None:\n unwrapped_model.generation_config.cache_implementation = self.args.cache_implementation\n\n # Standard generation\n output = unwrapped_model.generate(\n input_ids=prompt_ids,\n attention_mask=prompt_mask,\n generation_config=self.generation_config,\n **vision_generation_kwargs,\n )\n\n completion_ids = output[:, prompt_ids.size(1) :]\n completion_ids, completion_mask = truncate_right(completion_ids, eos_token_id, pad_token_id)\n\n return prompt_ids, prompt_mask, completion_ids, completion_mask\n\n def _calculate_rewards_from_functions(self, prompts, completions, completion_ids_list, **reward_kwargs):\n \"\"\"\n Calculate rewards using reward functions\n \"\"\"\n device = self.accelerator.device\n rewards_per_func = torch.zeros(len(prompts), len(self.reward_funcs), device=device)\n\n # Add trainer state to reward kwargs for dynamic reward shaping\n reward_kwargs[\"trainer_state\"] = self.state\n\n for i, (reward_func, reward_processing_class) in enumerate(\n zip(self.reward_funcs, self.reward_processing_classes, strict=True)\n ):\n if isinstance(reward_func, nn.Module): # Model-based reward function\n # Handle conversational vs text input\n if is_conversational({\"prompt\": prompts[0]}):\n messages = [{\"messages\": p + c} for p, c in zip(prompts, completions, strict=True)]\n texts = [apply_chat_template(x, reward_processing_class)[\"text\"] for x in messages]\n else:\n texts = [p + c for p, c in zip(prompts, completions, strict=True)]\n\n # Tokenize and get reward scores\n reward_inputs = reward_processing_class(\n text=texts, return_tensors=\"pt\", padding=True, padding_side=\"right\", add_special_tokens=False\n )\n reward_inputs = {k: v.to(device) for k, v in reward_inputs.items()}\n\n with torch.inference_mode():\n rewards_per_func[:, i] = reward_func(**reward_inputs).logits[:, 0] # Shape (B*G,)\n else:\n # Custom reward function\n output_reward_func = reward_func(\n prompts=prompts, completions=completions, completion_ids=completion_ids_list, **reward_kwargs\n )\n # Convert None values to NaN\n output_reward_func = [reward if reward is not None else torch.nan for reward in output_reward_func]\n rewards_per_func[:, i] = torch.tensor(output_reward_func, dtype=torch.float32, device=device)\n\n # Weight and sum across all reward functions\n if self.reward_weights is not None:\n total_rewards = (rewards_per_func * self.reward_weights.to(device).unsqueeze(0)).nansum(dim=1)\n else:\n total_rewards = rewards_per_func.nansum(dim=1)\n\n return total_rewards\n\n def _forward(self, model, prompt_ids, prompt_mask, completion_ids, completion_mask, vision_inputs=None):\n # Get the number of tokens to truncate from prompt\n num_tokens_to_truncate = max(prompt_ids.size(1) + completion_ids.size(1) - self.max_length, 0)\n\n # Truncate left to avoid oom\n prompt_ids = prompt_ids[:, num_tokens_to_truncate:]\n prompt_mask = prompt_mask[:, num_tokens_to_truncate:]\n\n # Concat the prompt and completion\n prompt_completion_ids = torch.cat((prompt_ids, completion_ids), dim=1)\n prompt_completion_mask = torch.cat((prompt_mask, completion_mask), dim=1)\n\n # Prepare model kwargs with vision inputs if available\n model_kwargs = {\"attention_mask\": prompt_completion_mask}\n if vision_inputs is not None:\n if \"pixel_values\" in vision_inputs:\n model_kwargs[\"pixel_values\"] = vision_inputs[\"pixel_values\"]\n if \"pixel_attention_mask\" in vision_inputs:\n model_kwargs[\"pixel_attention_mask\"] = vision_inputs[\"pixel_attention_mask\"]\n if \"image_sizes\" in vision_inputs:\n model_kwargs[\"image_sizes\"] = vision_inputs[\"image_sizes\"]\n if \"image_grid_thw\" in vision_inputs:\n model_kwargs[\"image_grid_thw\"] = vision_inputs[\"image_grid_thw\"]\n\n # Get the logprobs of the completions from the model\n output = model(prompt_completion_ids, **model_kwargs)\n\n # There is 1 offset, because the model predicts the next token\n prompt_len = prompt_ids.size(1)\n start_idx = prompt_len - 1 if prompt_len > 0 else 0\n # Only slice off the last logit when we have a prompt, otherwise we need all logits\n end_idx = -1 if prompt_len > 0 else None\n logits = output.logits[:, start_idx:end_idx]\n\n # Take the completion tokens logprob\n logprobs = torch.take_along_dim(logits.log_softmax(dim=-1), completion_ids.unsqueeze(-1), dim=2).squeeze(-1)\n return logprobs\n\n def training_step(\n self, model: nn.Module, inputs: dict[str, torch.Tensor | Any], num_items_in_batch: int | None = None\n ) -> torch.Tensor:\n model.train()\n\n prompts = inputs[\"prompt\"]\n batch_size = len(prompts)\n\n # Handle images for VLM support\n has_images = \"image\" in inputs\n images = None\n if has_images:\n images = inputs[\"image\"]\n # Convert conversational prompts to include image tokens\n for prompt in prompts:\n if isinstance(prompt, list):\n for message in prompt:\n if not isinstance(message, dict):\n continue\n content = message.get(\"content\")\n role = message.get(\"role\")\n if isinstance(content, str):\n if role == \"user\":\n message[\"content\"] = [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": content}]\n elif role == \"system\":\n message[\"content\"] = [{\"type\": \"text\", \"text\": content}]\n\n if self.args.use_vllm:\n prompt_ids, prompt_mask, completion_ids, completion_mask = self._generate_vllm(prompts, images)\n else:\n prompt_ids, prompt_mask, completion_ids, completion_mask = self._generate(model, prompts, images)\n\n contain_eos_token = torch.any(completion_ids == self.eos_token_id, dim=-1)\n\n # Extract vision inputs if available for VLM support\n vision_inputs = None\n if has_images and self.is_vision_model and not self.args.use_vllm:\n # For vision models with transformers generation, we need to prepare vision inputs\n # Process the images to get vision inputs that can be passed through the forward pass\n vision_inputs = {}\n kwargs = {\"images\": [[img] for img in images]}\n processed = self.processing_class(\n text=[\"\"] * len(images), # Dummy text for vision processing\n return_tensors=\"pt\",\n **kwargs,\n )\n # Handle DataParallel wrapped models\n model_device = getattr(model, \"device\", None)\n model_dtype = getattr(model, \"dtype\", None)\n if model_device is None and hasattr(model, \"module\"):\n model_device = model.module.device\n model_dtype = model.module.dtype\n # Move vision tensors to device and convert to model dtype\n # Need to duplicate for 2 completions per prompt\n if \"pixel_values\" in processed:\n vision_inputs[\"pixel_values\"] = (\n processed[\"pixel_values\"].to(model_device, dtype=model_dtype).repeat(2, 1, 1, 1)\n )\n if \"pixel_attention_mask\" in processed:\n vision_inputs[\"pixel_attention_mask\"] = processed[\"pixel_attention_mask\"].to(model_device).repeat(2, 1)\n if \"image_sizes\" in processed:\n vision_inputs[\"image_sizes\"] = processed[\"image_sizes\"].to(model_device).repeat(2, 1)\n if \"image_grid_thw\" in processed:\n vision_inputs[\"image_grid_thw\"] = processed[\"image_grid_thw\"].to(model_device).repeat(2, 1)\n\n logprobs = self._forward(model, prompt_ids, prompt_mask, completion_ids, completion_mask, vision_inputs)\n with torch.no_grad():\n if self.ref_model is not None:\n ref_logprobs = self._forward(\n self.ref_model, prompt_ids, prompt_mask, completion_ids, completion_mask, vision_inputs\n )\n else: # peft case: we just need to disable the adapter\n with self.model.disable_adapter():\n ref_logprobs = self._forward(\n self.model, prompt_ids, prompt_mask, completion_ids, completion_mask, vision_inputs\n )\n\n # Decode the completions, and format them if the input is conversational\n device = logprobs.device\n completions = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n if is_conversational({\"prompt\": prompts[0]}):\n completions = [[{\"role\": \"assistant\", \"content\": completion}] for completion in completions]\n\n # Get the reward from reward functions or judge\n if self.reward_funcs is not None:\n # First create completion_ids_list for custom reward functions\n completion_ids_list = [completion_ids[i].tolist() for i in range(completion_ids.shape[0])]\n\n # Extract additional fields from inputs for reward functions\n reward_kwargs = {}\n keys = [key for key in inputs if key not in [\"prompt\"]]\n for key in keys:\n if isinstance(inputs[key], (list, tuple)):\n # Repeat input fields to match number of completions (2 per prompt)\n reward_kwargs[key] = inputs[key] * 2\n else:\n reward_kwargs[key] = inputs[key]\n\n # Calculate rewards using reward functions\n rewards = self._calculate_rewards_from_functions(\n prompts=2 * prompts, completions=completions, completion_ids_list=completion_ids_list, **reward_kwargs\n )\n\n # Apply missing EOS penalty if configured\n if self.args.missing_eos_penalty is not None:\n rewards[~contain_eos_token] -= self.args.missing_eos_penalty\n\n # Split rewards into chosen/rejected pairs\n first_half, second_half = rewards.split(batch_size)\n mask = first_half >= second_half\n elif self.judge is not None:\n # Once formatted, conversational data may contain special tokens (such as <|im_start|>) that are not\n # directly understandable by the judge and could alter its judgment. To avoid this and make the judge\n # independent of the model's chat template, we use the raw conversation data, and apply our own chat\n # template to it.\n if is_conversational({\"prompt\": prompts[0]}):\n environment = jinja2.Environment()\n template = environment.from_string(SIMPLE_CHAT_TEMPLATE)\n prompts = [template.render(messages=prompt) for prompt in prompts]\n completions = [template.render(messages=completion) for completion in completions]\n\n ranks_of_first_completion = self.judge.judge(\n prompts, list(zip(completions[:batch_size], completions[batch_size:], strict=True))\n )\n\n # convert ranks to a True/False mask:\n # when rank == 0, it means the first completion is the best\n # when rank == 1, it means the second completion is the best\n mask = torch.tensor([rank == 0 for rank in ranks_of_first_completion], device=device)\n\n batch_range = torch.arange(batch_size, device=device)\n chosen_indices = batch_range + (~mask * batch_size)\n rejected_indices = batch_range + (mask * batch_size)\n\n # Build tensor so that the first half is the chosen examples and the second half the rejected examples\n cr_indices = torch.cat((chosen_indices, rejected_indices), dim=0) # cr = chosen and rejected\n cr_logprobs = logprobs[cr_indices]\n cr_ref_logprobs = ref_logprobs[cr_indices]\n\n # mask out the padding tokens\n padding_mask = ~completion_mask.bool()\n cr_padding_mask = padding_mask[cr_indices]\n\n cr_logprobs_sum = (cr_logprobs * ~cr_padding_mask).sum(1)\n cr_ref_logprobs_sum = (cr_ref_logprobs * ~cr_padding_mask).sum(1)\n\n # Split the chosen and rejected examples\n chosen_logprobs_sum, rejected_logprobs_sum = torch.split(cr_logprobs_sum, batch_size)\n chosen_ref_logprobs_sum, rejected_ref_logprobs_sum = torch.split(cr_ref_logprobs_sum, batch_size)\n pi_logratios = chosen_logprobs_sum - rejected_logprobs_sum\n ref_logratios = chosen_ref_logprobs_sum - rejected_ref_logprobs_sum\n\n logits = pi_logratios - ref_logratios\n\n if self.args.loss_type == \"sigmoid\":\n losses = -F.logsigmoid(self.beta * logits)\n elif self.args.loss_type == \"ipo\":\n losses = (logits - 1 / (2 * self.beta)) ** 2\n else:\n raise NotImplementedError(f\"invalid loss type {self.args.loss_type}\")\n\n loss = losses.mean()\n\n # Log everything\n if self.reward_funcs is not None:\n # When using reward_funcs, we have rewards instead of scores\n scores_margin = rewards[chosen_indices] - rewards[rejected_indices]\n self.stats[\"objective/scores_margin\"].append(\n self.accelerator.gather_for_metrics(scores_margin.mean()).mean().item()\n )\n self.stats[\"objective/scores\"].append(self.accelerator.gather_for_metrics(rewards.mean()).mean().item())\n self.stats[\"val/contain_eos_token\"].append(contain_eos_token.float().mean().item())\n self.stats[\"logps/chosen\"].append(self.accelerator.gather_for_metrics(chosen_logprobs_sum).mean().item())\n self.stats[\"logps/rejected\"].append(self.accelerator.gather_for_metrics(rejected_logprobs_sum).mean().item())\n\n kl = logprobs - ref_logprobs\n mean_kl = kl.sum(1).mean()\n self.stats[\"objective/kl\"].append(self.accelerator.gather_for_metrics(mean_kl).mean().item())\n non_score_reward = (-self.beta * kl).sum(1)\n mean_non_score_reward = non_score_reward.mean()\n self.stats[\"objective/non_score_reward\"].append(\n self.accelerator.gather_for_metrics(mean_non_score_reward).mean().item()\n )\n if self.reward_funcs is not None:\n # Calculate RLHF reward by combining rewards with non_score_reward\n rlhf_reward = rewards + non_score_reward\n self.stats[\"objective/rlhf_reward\"].append(self.accelerator.gather_for_metrics(rlhf_reward).mean().item())\n\n mean_entropy = -logprobs.sum(1).mean()\n self.stats[\"objective/entropy\"].append(self.accelerator.gather_for_metrics(mean_entropy).mean().item())\n chosen_rewards = self.beta * (chosen_logprobs_sum - chosen_ref_logprobs_sum)\n gathered_chosen_rewards = self.accelerator.gather_for_metrics(chosen_rewards)\n self.stats[\"rewards/chosen\"].append(gathered_chosen_rewards.mean().item())\n rejected_rewards = self.beta * (rejected_logprobs_sum - rejected_ref_logprobs_sum)\n gathered_rejected_rewards = self.accelerator.gather_for_metrics(rejected_rewards)\n self.stats[\"rewards/rejected\"].append(gathered_rejected_rewards.mean().item())\n margin = gathered_chosen_rewards - gathered_rejected_rewards\n self.stats[\"rewards/margins\"].append(margin.mean().item())\n accuracy = margin > 0\n self.stats[\"rewards/accuracies\"].append(accuracy.float().mean().item())\n self.stats[\"beta\"].append(self.beta)\n\n if (\n self.args.torch_empty_cache_steps is not None\n and self.state.global_step % self.args.torch_empty_cache_steps == 0\n ):\n empty_cache()\n\n kwargs = {}\n\n # For LOMO optimizers you need to explicitly use the learning rate\n if self.args.optim in [OptimizerNames.LOMO, OptimizerNames.ADALOMO]:\n kwargs[\"learning_rate\"] = self._get_learning_rate()\n\n if self.args.n_gpu > 1:\n loss = loss.mean() # mean() to average on multi-gpu parallel training\n\n self.accelerator.backward(loss, **kwargs)\n\n return loss.detach() / self.args.gradient_accumulation_steps\n\n # Same as Trainer._maybe_log_save_evaluate but log our metrics\n def _maybe_log_save_evaluate(\n self, tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval, start_time, learning_rate=None\n ):\n if self.control.should_log and self.state.global_step > self._globalstep_last_logged:\n logs: dict[str, float] = {}\n\n # all_gather + mean() to get average loss over all processes\n if Version(transformers.__version__) >= Version(\"5.2.0\"):\n tr_loss_scalar = nested_gather(tr_loss, self.args.parallel_mode).mean().item()\n else:\n tr_loss_scalar = self._nested_gather(tr_loss).mean().item()\n\n # reset tr_loss to zero\n tr_loss -= tr_loss\n\n logs[\"loss\"] = round(tr_loss_scalar / (self.state.global_step - self._globalstep_last_logged), 4)\n if grad_norm is not None:\n logs[\"grad_norm\"] = grad_norm.detach().item() if isinstance(grad_norm, torch.Tensor) else grad_norm\n if learning_rate is not None:\n logs[\"learning_rate\"] = learning_rate\n else:\n logs[\"learning_rate\"] = self._get_learning_rate()\n\n # Add our metrics\n for key, val in self.stats.items():\n logs[key] = sum(val) / len(val)\n self.stats = {key: [] for key in self.stats} # reset stats\n\n self._total_loss_scalar += tr_loss_scalar\n self._globalstep_last_logged = self.state.global_step\n self.store_flos()\n self.log(logs, start_time)\n\n metrics = None\n if self.control.should_evaluate:\n metrics = self._evaluate(trial, ignore_keys_for_eval)\n is_new_best_metric = self._determine_best_metric(metrics=metrics, trial=trial)\n\n if self.args.save_strategy == \"best\":\n self.control.should_save = is_new_best_metric\n\n if self.control.should_save:\n self._save_checkpoint(model, trial)\n self.control = self.callback_handler.on_save(self.args, self.state, self.control)\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/experimental/openenv/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .utils import generate_rollout_completions\n\n\n__all__ = [\"generate_rollout_completions\"]\n" }, { "path": "trl/experimental/openenv/utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom typing import Any\n\nimport torch\n\nfrom ...data_utils import is_conversational\nfrom ...extras.profiling import profiling_context\nfrom ...import_utils import is_vllm_available\n\n\nif is_vllm_available():\n from vllm import SamplingParams\n from vllm.sampling_params import StructuredOutputsParams\n\n\ndef _build_base_generation_kwargs(\n trainer,\n overrides: dict[str, Any] | None = None,\n) -> dict[str, Any]:\n \"\"\"Build base generation kwargs common to both colocate and server modes.\"\"\"\n generation_kwargs: dict[str, Any] = {\n \"n\": 1,\n \"temperature\": trainer.temperature,\n \"top_k\": trainer.top_k,\n \"min_p\": 0.0 if trainer.min_p is None else trainer.min_p,\n \"max_tokens\": trainer.max_completion_length,\n }\n if trainer.repetition_penalty is not None:\n generation_kwargs[\"repetition_penalty\"] = trainer.repetition_penalty\n if trainer.top_p is not None:\n generation_kwargs[\"top_p\"] = trainer.top_p\n\n if trainer.args.generation_kwargs is not None:\n generation_kwargs.update(trainer.args.generation_kwargs)\n\n if overrides is not None:\n generation_kwargs.update(overrides)\n\n generation_kwargs = {key: value for key, value in generation_kwargs.items() if value is not None}\n\n if generation_kwargs.get(\"n\", 1) != 1:\n raise ValueError(\"generate_rollout_completions expects n=1.\")\n\n return generation_kwargs\n\n\ndef _build_colocate_sampling_params(\n trainer,\n overrides: dict[str, Any] | None = None,\n *,\n logprobs: bool = True,\n) -> \"SamplingParams\":\n \"\"\"Build SamplingParams for colocate mode.\"\"\"\n generation_kwargs = _build_base_generation_kwargs(trainer, overrides)\n\n # Add colocate-specific parameters\n if trainer.vllm_generation.structured_outputs_regex:\n generation_kwargs[\"structured_outputs\"] = StructuredOutputsParams(\n regex=trainer.vllm_generation.structured_outputs_regex\n )\n if logprobs:\n generation_kwargs[\"logprobs\"] = 0\n\n return SamplingParams(**generation_kwargs)\n\n\ndef _build_server_generation_kwargs(\n trainer,\n overrides: dict[str, Any] | None = None,\n) -> dict[str, Any]:\n \"\"\"Build generation kwargs for server mode.\"\"\"\n return _build_base_generation_kwargs(trainer, overrides)\n\n\ndef generate_rollout_completions(\n trainer,\n prompts: list[str],\n *,\n generation_overrides: dict[str, Any] | None = None,\n as_chat: bool | None = None,\n) -> list[dict[str, Any]]:\n \"\"\"\n Generate completions for custom rollouts when vLLM is running in colocate or server mode.\n\n Returns one result per prompt, containing prompt and completion token ids along with per-token log probabilities\n and the generated text.\n \"\"\"\n\n if not prompts:\n return []\n\n if not trainer.use_vllm:\n raise RuntimeError(\"Custom rollouts require vLLM to call generate_rollout_completions.\")\n\n if trainer.vllm_mode == \"server\":\n return _generate_rollout_completions_server(trainer, prompts, generation_overrides, as_chat)\n elif trainer.vllm_mode == \"colocate\":\n return _generate_rollout_completions_colocate(trainer, prompts, generation_overrides, as_chat)\n else:\n raise ValueError(f\"vllm_mode must be 'server' or 'colocate', got '{trainer.vllm_mode}'\")\n\n\ndef _generate_rollout_completions_server(\n trainer,\n prompts: list[str],\n generation_overrides: dict[str, Any] | None = None,\n as_chat: bool | None = None,\n) -> list[dict[str, Any]]:\n \"\"\"Generate completions using vLLM server mode.\"\"\"\n generation_kwargs = _build_server_generation_kwargs(trainer, generation_overrides)\n\n if as_chat is None:\n as_chat = prompts and is_conversational({\"prompt\": prompts[0]})\n\n with profiling_context(trainer, \"vLLM.generate_rollout_server\"):\n if as_chat:\n # Prompts are raw message dicts; use .chat() so the vLLM server applies the chat template\n output = trainer.vllm_generation.vllm_client.chat(\n messages=prompts,\n **generation_kwargs,\n chat_template_kwargs=trainer.chat_template_kwargs,\n tools=trainer.tools or None,\n chat_template=trainer.chat_template,\n )\n else:\n output = trainer.vllm_generation.vllm_client.generate(prompts=prompts, **generation_kwargs)\n\n # Format results to match colocate output format\n results: list[dict[str, Any]] = []\n for i in range(len(prompts)):\n results.append(\n {\n \"prompt_ids\": output[\"prompt_ids\"][i],\n \"completion_ids\": list(output[\"completion_ids\"][i]),\n \"logprobs\": list(output[\"logprobs\"][i]),\n \"text\": trainer.processing_class.decode(output[\"completion_ids\"][i], skip_special_tokens=True),\n }\n )\n\n return results\n\n\ndef _generate_rollout_completions_colocate(\n trainer,\n prompts: list[str],\n generation_overrides: dict[str, Any] | None = None,\n as_chat: bool | None = None,\n) -> list[dict[str, Any]]:\n \"\"\"Generate completions using vLLM colocate mode.\"\"\"\n sampling_params = _build_colocate_sampling_params(trainer, generation_overrides)\n prompts_for_generation = prompts\n original_size = len(prompts)\n\n if trainer.vllm_tensor_parallel_size > 1:\n gathered_prompts = [None for _ in range(trainer.vllm_tensor_parallel_size)]\n torch.distributed.all_gather_object(gathered_prompts, prompts, group=trainer.vllm_generation.tp_group)\n prompts_for_generation = [prompt for group_prompts in gathered_prompts for prompt in group_prompts]\n\n if as_chat is None:\n as_chat = prompts_for_generation and is_conversational({\"prompt\": prompts_for_generation[0]})\n\n if trainer.args.vllm_enable_sleep_mode:\n trainer.vllm_generation.llm.wake_up(tags=[\"kv_cache\"])\n # Work around for https://github.com/vllm-project/vllm/issues/29341\n trainer.vllm_generation.llm.collective_rpc(\"reload_weights\")\n\n with profiling_context(trainer, \"vLLM.generate_rollout\"):\n if as_chat:\n vllm_outputs = trainer.vllm_generation.llm.chat(\n prompts_for_generation, sampling_params=sampling_params, use_tqdm=False\n )\n else:\n vllm_outputs = trainer.vllm_generation.llm.generate(\n prompts_for_generation, sampling_params=sampling_params, use_tqdm=False\n )\n\n results: list[dict[str, Any]] = []\n for request in vllm_outputs:\n if not request.outputs:\n results.append({\"prompt_ids\": request.prompt_token_ids, \"completion_ids\": [], \"logprobs\": [], \"text\": \"\"})\n continue\n sequence = request.outputs[0]\n logprobs = [next(iter(token_logprob.values())).logprob for token_logprob in sequence.logprobs]\n results.append(\n {\n \"prompt_ids\": request.prompt_token_ids,\n \"completion_ids\": sequence.token_ids,\n \"logprobs\": logprobs,\n \"text\": sequence.text,\n }\n )\n\n if trainer.vllm_tensor_parallel_size > 1:\n local_rank_in_group = torch.distributed.get_rank(group=trainer.vllm_generation.tp_group)\n tp_slice = slice(local_rank_in_group * original_size, (local_rank_in_group + 1) * original_size)\n results = results[tp_slice]\n\n if trainer.args.vllm_enable_sleep_mode:\n trainer.vllm_generation.llm.sleep(level=2)\n\n return results\n" }, { "path": "trl/experimental/orpo/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .orpo_config import ORPOConfig\nfrom .orpo_trainer import ORPOTrainer\n\n\n__all__ = [\"ORPOConfig\", \"ORPOTrainer\"]\n" }, { "path": "trl/experimental/orpo/orpo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom ...trainer.base_config import _BaseConfig\n\n\n@dataclass\nclass ORPOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`experimental.orpo.ORPOTrainer`].\n\n This class includes only the parameters that are specific to ORPO training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n max_length (`int` or `None`, *optional*, defaults to `1024`):\n Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want\n to use the default data collator.\n max_completion_length (`int`, *optional*):\n Maximum length of the completion. This argument is required if you want to use the default data collator\n and your model is an encoder-decoder.\n beta (`float`, *optional*, defaults to `0.1`):\n Parameter controlling the relative ratio loss weight in the ORPO loss. In the\n [paper](https://huggingface.co/papers/2403.07691), it is denoted by ฮป. In the\n [code](https://github.com/xfactlab/orpo), it is denoted by `alpha`.\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model.\n padding_value (`int`, *optional*):\n Padding value to use. If `None`, the padding value of the tokenizer is used.\n truncation_mode (`str`, *optional*, defaults to `\"keep_end\"`):\n Truncation mode to use when the prompt is too long. Possible values are `\"keep_end\"` or `\"keep_start\"`.\n This argument is required if you want to use the default data collator.\n generate_during_eval (`bool`, *optional*, defaults to `False`):\n If `True`, generates and logs completions from the model to W&B or Comet during evaluation.\n is_encoder_decoder (`bool`, *optional*):\n When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,\n you need to specify if the model returned by the callable is an encoder-decoder model.\n model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a\n string.\n dataset_num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `1e-6` instead of `5e-5`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-6,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n max_length: int | None = field(\n default=1024,\n metadata={\"help\": \"Maximum length of the sequences (prompt + completion) in the batch.\"},\n )\n max_completion_length: int | None = field(\n default=None,\n metadata={\n \"help\": \"Maximum length of the completion. This argument is required if you want to use the default data \"\n \"collator and your model is an encoder-decoder.\"\n },\n )\n beta: float = field(\n default=0.1,\n metadata={\n \"help\": \"Parameter controlling the relative ratio loss weight in the ORPO loss. In the paper, it is \"\n \"denoted by ฮป.\"\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropout in the model.\"},\n )\n padding_value: int | None = field(\n default=None,\n metadata={\"help\": \"Padding value to use. If `None`, the padding value of the tokenizer is used.\"},\n )\n truncation_mode: str = field(\n default=\"keep_end\",\n metadata={\n \"help\": \"Truncation mode to use when the prompt is too long.\",\n \"choices\": [\"keep_end\", \"keep_start\"],\n },\n )\n generate_during_eval: bool = field(\n default=False,\n metadata={\"help\": \"If `True`, generates and logs completions from the model to W&B during evaluation.\"},\n )\n is_encoder_decoder: bool | None = field(\n default=None,\n metadata={\n \"help\": \"When using the `model_init` argument (callable) to instantiate the model instead of the `model` \"\n \"argument, you need to specify if the model returned by the callable is an encoder-decoder model.\"\n },\n )\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model \"\n \"from a string.\"\n },\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes to use for processing the dataset.\"},\n )\n" }, { "path": "trl/experimental/orpo/orpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport inspect\nimport random\nimport textwrap\nfrom collections import defaultdict\nfrom collections.abc import Callable\nfrom contextlib import nullcontext\nfrom pathlib import Path\nfrom typing import Any, Literal\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport transformers\nfrom accelerate import PartialState, logging\nfrom datasets import Dataset\nfrom packaging.version import Version\nfrom torch import autocast\nfrom torch.utils.data import DataLoader\nfrom transformers import (\n AutoModelForCausalLM,\n BaseImageProcessor,\n DataCollator,\n FeatureExtractionMixin,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n is_comet_available,\n is_torch_xla_available,\n is_wandb_available,\n)\nfrom transformers.trainer_utils import EvalLoopOutput\nfrom transformers.utils import is_peft_available, is_torch_fx_proxy\n\nfrom ...data_utils import maybe_apply_chat_template, maybe_extract_prompt\nfrom ...trainer.base_trainer import _BaseTrainer\nfrom ...trainer.utils import disable_dropout_in_model, log_table_to_comet_experiment, selective_log_softmax\nfrom ..utils import (\n DPODataCollatorWithPadding,\n add_bos_token_if_needed,\n add_eos_token_if_needed,\n pad_to_length,\n peft_module_casting_to_bf16,\n)\nfrom .orpo_config import ORPOConfig\n\n\nif is_peft_available():\n from peft import PeftModel, get_peft_model, prepare_model_for_kbit_training\n\n\nif is_wandb_available():\n import wandb\n\nif is_torch_xla_available():\n import torch_xla.core.xla_model as xm\n\n\nlogger = logging.get_logger(__name__)\n\n\ndef log1mexp(x: torch.FloatTensor) -> torch.FloatTensor:\n \"\"\"Numerically stable computation of log(1-exp(x)).\"\"\"\n # branch at -ln 2 ~ -0.693 to avoid cancellation\n t = -0.6931471805599453\n return torch.where(x < t, torch.log1p(-torch.exp(x)), torch.log(-torch.expm1(x)))\n\n\nclass ORPOTrainer(_BaseTrainer):\n r\"\"\"\n Initialize ORPOTrainer.\n\n Args:\n model ([`~transformers.PreTrainedModel`]):\n The model to train, preferably an [`~transformers.AutoModelForSequenceClassification`].\n args ([`experimental.orpo.ORPOConfig`]):\n The ORPO config arguments to use for training.\n data_collator ([`~transformers.DataCollator`]):\n The data collator to use for training. If None is specified, the default data collator\n ([`experimental.utils.DPODataCollatorWithPadding`]) will be used which will pad the sequences to the\n maximum length of the sequences in the batch, given a dataset of paired sequences.\n train_dataset ([`~datasets.Dataset`]):\n The dataset to use for training.\n eval_dataset ([`~datasets.Dataset`]):\n The dataset to use for evaluation.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. If provided, will be used to automatically process the inputs\n for the model, and it will be saved along the model to make it easier to rerun an interrupted training or\n reuse the fine-tuned model.\n model_init (`Callable[[], transformers.PreTrainedModel]`):\n The model initializer to use for training. If None is specified, the default model initializer will be\n used.\n callbacks (`list[transformers.TrainerCallback]`):\n The callbacks to use for training.\n optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):\n The optimizer and scheduler to use for training.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):\n The function to use to preprocess the logits before computing the metrics.\n peft_config (`dict`, defaults to `None`):\n The PEFT configuration to use for training. If you pass a PEFT configuration, the model will be wrapped in\n a PEFT model.\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to\n metric values.\n \"\"\"\n\n _tag_names = [\"trl\", \"orpo\"]\n _name = \"ORPO\"\n _paper = {\n \"title\": \"ORPO: Monolithic Preference Optimization without Reference Model\",\n \"id\": \"2403.07691\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{hong2024orpo,\n title = {{ORPO: Monolithic Preference Optimization without Reference Model}},\n author = {Jiwoo Hong and Noah Lee and James Thorne},\n year = 2024,\n eprint = {arXiv:2403.07691}\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module | str | None = None,\n args: ORPOConfig | None = None,\n data_collator: DataCollator | None = None,\n train_dataset: Dataset | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n processing_class: PreTrainedTokenizerBase\n | BaseImageProcessor\n | FeatureExtractionMixin\n | ProcessorMixin\n | None = None,\n model_init: Callable[[], PreTrainedModel] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n peft_config: dict | None = None,\n compute_metrics: Callable[[EvalLoopOutput], dict] | None = None,\n ):\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n\n if args.model_init_kwargs is None:\n model_init_kwargs = {}\n elif not isinstance(model, str):\n raise ValueError(\"You passed model_kwargs to the ORPOTrainer. But your model is already instantiated.\")\n else:\n model_init_kwargs = args.model_init_kwargs\n dtype = model_init_kwargs.get(\"dtype\", \"auto\")\n if dtype is not None:\n # Convert to `torch.dtype` if an str is passed\n if isinstance(dtype, str) and dtype != \"auto\":\n dtype = getattr(torch, dtype)\n if dtype != \"auto\" and not isinstance(dtype, torch.dtype):\n raise ValueError(\n f\"Invalid `dtype` passed to the ORPOConfig. Expected a string with either `torch.dtype` or 'auto', but got {dtype}.\"\n )\n model_init_kwargs[\"dtype\"] = dtype\n model_init_kwargs[\"device_map\"] = model_init_kwargs.get(\"device_map\", \"auto\")\n\n if isinstance(model, str):\n model = AutoModelForCausalLM.from_pretrained(model, **model_init_kwargs)\n\n # Initialize this variable to False. This helps tracking the case when `peft_module_casting_to_bf16`\n # has been called in order to properly call autocast if needed.\n self._peft_has_been_casted_to_bf16 = False\n\n if not is_peft_available() and peft_config is not None:\n raise ValueError(\n \"PEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT models\"\n )\n elif is_peft_available() and peft_config is not None:\n if isinstance(model, PeftModel):\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first \"\n \"merge and unload the existing adapter, save the resulting base model, and then pass that base \"\n \"model along with the new `peft_config` to the trainer.\"\n )\n\n if getattr(model, \"is_loaded_in_8bit\", False) or getattr(model, \"is_loaded_in_4bit\", False):\n _support_gc_kwargs = hasattr(\n args, \"gradient_checkpointing_kwargs\"\n ) and \"gradient_checkpointing_kwargs\" in list(\n inspect.signature(prepare_model_for_kbit_training).parameters\n )\n\n prepare_model_kwargs = {\"use_gradient_checkpointing\": args.gradient_checkpointing}\n\n if _support_gc_kwargs:\n prepare_model_kwargs[\"gradient_checkpointing_kwargs\"] = args.gradient_checkpointing_kwargs\n\n model = prepare_model_for_kbit_training(model, **prepare_model_kwargs)\n elif args.gradient_checkpointing:\n # For backward compatibility with older versions of transformers\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n # get peft model with the given config\n model = get_peft_model(model, peft_config)\n if args.bf16 and getattr(model, \"is_loaded_in_4bit\", False):\n peft_module_casting_to_bf16(model)\n # If args.bf16 we need to explicitly call `generate` with torch amp autocast context manager\n self._peft_has_been_casted_to_bf16 = True\n\n # For models that use gradient_checkpointing, we need to attach a hook that enables input\n # to explicitly have `requires_grad=True`, otherwise training will either silently\n # fail or completely fail.\n elif args.gradient_checkpointing:\n # For backward compatibility with older versions of transformers\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n if args.generate_during_eval and not (is_wandb_available() or is_comet_available()):\n raise ValueError(\n \"`generate_during_eval=True` requires Weights and Biases or Comet to be installed.\"\n \" Please install `wandb` or `comet-ml` to resolve.\"\n )\n\n if model is not None:\n self.is_encoder_decoder = model.config.is_encoder_decoder\n elif args.is_encoder_decoder is None:\n raise ValueError(\"When no model is provided, you need to pass the parameter is_encoder_decoder.\")\n else:\n self.is_encoder_decoder = args.is_encoder_decoder\n\n if self.is_encoder_decoder:\n self.decoder_start_token_id = model.config.decoder_start_token_id\n self.pad_token_id = model.config.pad_token_id\n\n if processing_class is None:\n raise ValueError(\"processing_class must be specified to tokenize a ORPO dataset.\")\n if args.max_length is None:\n logger.warning(\n \"`max_length` is not set in the ORPOConfig's init\"\n \" it will default to `512` by default, but you should do it yourself in the future.\",\n )\n max_length = 512\n else:\n max_length = args.max_length\n\n if args.max_completion_length is None and self.is_encoder_decoder:\n logger.warning(\n \"When using an encoder decoder architecture, you should set `max_completion_length` in the ORPOConfig's init\"\n \" it will default to `128` by default, but you should do it yourself in the future.\",\n )\n self.max_completion_length = 128\n else:\n self.max_completion_length = args.max_completion_length\n\n if data_collator is None:\n data_collator = DPODataCollatorWithPadding(\n pad_token_id=processing_class.pad_token_id,\n is_encoder_decoder=self.is_encoder_decoder,\n )\n\n if args.remove_unused_columns:\n args.remove_unused_columns = False\n # warn users\n logger.warning(\n \"When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments\"\n \" we have set it for you, but you should do it yourself in the future.\",\n )\n\n self.use_dpo_data_collator = True\n else:\n self.use_dpo_data_collator = False\n\n # Disable dropout in the model and reference model\n if args.disable_dropout:\n disable_dropout_in_model(model)\n\n self.max_length = max_length\n self.generate_during_eval = args.generate_during_eval\n self.padding_value = args.padding_value if args.padding_value is not None else processing_class.pad_token_id\n self.truncation_mode = args.truncation_mode\n self.processing_class = processing_class\n\n self.beta = args.beta\n self.aux_loss_enabled = getattr(model.config, \"output_router_logits\", False)\n self.aux_loss_coef = getattr(model.config, \"router_aux_loss_coef\", 0.0)\n if self.aux_loss_enabled and self.aux_loss_coef == 0.0:\n logger.warning(\n \"You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to \"\n \"`0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value \"\n \"greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary \"\n \"loss.\",\n )\n\n self._stored_metrics = defaultdict(lambda: defaultdict(list))\n\n # Compute that only on the main process for faster data processing.\n # see: https://github.com/huggingface/trl/pull/1255\n with PartialState().main_process_first():\n # Extract the prompt if needed, and apply the chat template if needed\n train_dataset = train_dataset.map(maybe_extract_prompt, num_proc=args.dataset_num_proc)\n train_dataset = train_dataset.map(\n maybe_apply_chat_template, fn_kwargs={\"tokenizer\": processing_class}, num_proc=args.dataset_num_proc\n )\n train_dataset = train_dataset.map(self.tokenize_row, num_proc=args.dataset_num_proc)\n if eval_dataset is not None:\n eval_dataset = eval_dataset.map(maybe_extract_prompt, num_proc=args.dataset_num_proc)\n eval_dataset = eval_dataset.map(\n maybe_apply_chat_template,\n fn_kwargs={\"tokenizer\": processing_class},\n num_proc=args.dataset_num_proc,\n )\n eval_dataset = eval_dataset.map(self.tokenize_row, num_proc=args.dataset_num_proc)\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n model_init=model_init,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n # Gradient accumulation requires scaled loss. Normally, loss scaling in the parent class depends on whether the\n # model accepts loss-related kwargs. Since we compute our own loss, this check is irrelevant. We set\n # self.model_accepts_loss_kwargs to False to enable scaling.\n self.model_accepts_loss_kwargs = False\n\n # Add tags for models that have been loaded with the correct transformers version\n if hasattr(self.model, \"add_model_tags\"):\n self.model.add_model_tags(self._tag_names)\n\n if not hasattr(self, \"accelerator\"):\n raise AttributeError(\n \"Your `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.\"\n )\n\n def build_tokenized_answer(self, prompt, answer):\n \"\"\"\n Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a +\n b)[len(enc(a)):]`. Reference:\n https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257\n \"\"\"\n\n full_tokenized = self.processing_class(prompt + answer, add_special_tokens=False)\n prompt_input_ids = self.processing_class(prompt, add_special_tokens=False)[\"input_ids\"]\n\n answer_input_ids = full_tokenized[\"input_ids\"][len(prompt_input_ids) :]\n answer_attention_mask = full_tokenized[\"attention_mask\"][len(prompt_input_ids) :]\n\n # Concat tokens to form `enc(a) + enc(a + b)[len(enc(a)):]`\n full_concat_input_ids = np.concatenate([prompt_input_ids, answer_input_ids])\n\n # Prepare input tokens for token by token comparison\n full_input_ids = np.array(full_tokenized[\"input_ids\"])\n\n if len(full_input_ids) != len(full_concat_input_ids):\n raise ValueError(\"Prompt input ids and answer input ids should have the same length.\")\n\n # On some tokenizers, like Llama-2 tokenizer, there are occasions where tokens\n # can be merged together when tokenizing prompt+answer. This could result\n # on the last token from the prompt being different when tokenized on its own\n # vs when done as prompt+answer.\n response_token_ids_start_idx = len(prompt_input_ids)\n\n # If tokenized prompt is different than both prompt+answer, then it means the\n # last token has changed due to merging.\n if prompt_input_ids != full_tokenized[\"input_ids\"][:response_token_ids_start_idx]:\n response_token_ids_start_idx -= 1\n\n prompt_input_ids = full_tokenized[\"input_ids\"][:response_token_ids_start_idx]\n prompt_attention_mask = full_tokenized[\"attention_mask\"][:response_token_ids_start_idx]\n\n if len(prompt_input_ids) != len(prompt_attention_mask):\n raise ValueError(\"Prompt input ids and attention mask should have the same length.\")\n\n answer_input_ids = full_tokenized[\"input_ids\"][response_token_ids_start_idx:]\n answer_attention_mask = full_tokenized[\"attention_mask\"][response_token_ids_start_idx:]\n\n return dict(\n prompt_input_ids=prompt_input_ids,\n prompt_attention_mask=prompt_attention_mask,\n input_ids=answer_input_ids,\n attention_mask=answer_attention_mask,\n )\n\n def tokenize_row(self, feature, model: PreTrainedModel | nn.Module | None = None) -> dict:\n \"\"\"Tokenize a single row from a ORPO specific dataset.\n\n At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +\n chosen or prompt + rejected responses is/are too long. First we truncate the prompt; if we're still too long,\n we truncate the chosen/rejected.\n\n We also create the labels for the chosen/rejected responses, which are of length equal to the sum of the length\n of the prompt and the chosen/rejected response, with `-100` for the prompt tokens.\n \"\"\"\n batch = {}\n prompt = feature[\"prompt\"]\n chosen = feature[\"chosen\"]\n rejected = feature[\"rejected\"]\n\n if not self.is_encoder_decoder:\n # Check issues below for more details\n # 1. https://github.com/huggingface/trl/issues/907\n # 2. https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257\n # 3. https://github.com/LianjiaTech/BELLE/issues/337\n\n if not isinstance(prompt, str):\n raise ValueError(f\"prompt should be an str but got {type(prompt)}\")\n prompt_tokens = self.processing_class(prompt, add_special_tokens=False)\n prompt_tokens = {f\"prompt_{k}\": v for k, v in prompt_tokens.items()}\n\n if not isinstance(chosen, str):\n raise ValueError(f\"chosen should be an str but got {type(chosen)}\")\n chosen_tokens = self.build_tokenized_answer(prompt, chosen)\n\n if not isinstance(rejected, str):\n raise ValueError(f\"rejected should be an str but got {type(rejected)}\")\n rejected_tokens = self.build_tokenized_answer(prompt, rejected)\n\n # Last prompt token might get merged by tokenizer and\n # it should not be included for generation if that happens\n prompt_len_input_ids = len(prompt_tokens[\"prompt_input_ids\"])\n\n chosen_prompt_len_input_ids = len(chosen_tokens[\"prompt_input_ids\"])\n rejected_prompt_len_input_ids = len(rejected_tokens[\"prompt_input_ids\"])\n prompt_len_input_ids = min(chosen_prompt_len_input_ids, rejected_prompt_len_input_ids)\n\n for k, v in prompt_tokens.items():\n prompt_tokens[k] = v[:prompt_len_input_ids]\n\n # Make sure prompts only have one different token at most an\n # and length only differs by 1 at most\n num_diff_tokens = sum(\n a != b\n for a, b in zip(chosen_tokens[\"prompt_input_ids\"], rejected_tokens[\"prompt_input_ids\"], strict=False)\n )\n num_diff_len = abs(chosen_prompt_len_input_ids - rejected_prompt_len_input_ids)\n if num_diff_tokens > 1 or num_diff_len > 1:\n raise ValueError(\n \"Chosen and rejected prompt_input_ids might only differ on the \"\n \"last token due to tokenizer merge ops.\"\n )\n\n # add BOS token to head of prompt. Avoid adding if it's already there\n prompt_tokens, chosen_tokens, rejected_tokens = add_bos_token_if_needed(\n self.processing_class.bos_token_id,\n prompt_len_input_ids,\n prompt_tokens,\n chosen_prompt_len_input_ids,\n chosen_tokens,\n rejected_prompt_len_input_ids,\n rejected_tokens,\n )\n\n # add EOS token to end of answer. Avoid adding if it's already there\n chosen_tokens, rejected_tokens = add_eos_token_if_needed(\n self.processing_class.eos_token_id, chosen_tokens, rejected_tokens\n )\n\n longer_response_length = max(len(chosen_tokens[\"input_ids\"]), len(rejected_tokens[\"input_ids\"]))\n\n # if combined sequence is too long, truncate the response\n for answer_tokens in [chosen_tokens, rejected_tokens]:\n if len(answer_tokens[\"prompt_input_ids\"]) + longer_response_length > self.max_length:\n for k in [\"input_ids\", \"attention_mask\"]:\n answer_tokens[k] = answer_tokens[k][: self.max_length - longer_response_length]\n\n # Create labels\n chosen_sequence_tokens = {\n k: chosen_tokens[f\"prompt_{k}\"] + chosen_tokens[k] for k in [\"input_ids\", \"attention_mask\"]\n }\n rejected_sequence_tokens = {\n k: rejected_tokens[f\"prompt_{k}\"] + rejected_tokens[k] for k in [\"input_ids\", \"attention_mask\"]\n }\n chosen_sequence_tokens[\"labels\"] = chosen_sequence_tokens[\"input_ids\"][:]\n chosen_sequence_tokens[\"labels\"][: len(chosen_tokens[\"prompt_input_ids\"])] = [-100] * len(\n chosen_tokens[\"prompt_input_ids\"]\n )\n rejected_sequence_tokens[\"labels\"] = rejected_sequence_tokens[\"input_ids\"][:]\n rejected_sequence_tokens[\"labels\"][: len(rejected_tokens[\"prompt_input_ids\"])] = [-100] * len(\n rejected_tokens[\"prompt_input_ids\"]\n )\n\n for k, toks in {\n \"chosen_\": chosen_sequence_tokens,\n \"rejected_\": rejected_sequence_tokens,\n \"\": prompt_tokens,\n }.items():\n for type_key, tokens in toks.items():\n if type_key == \"token_type_ids\":\n continue\n batch[f\"{k}{type_key}\"] = tokens\n\n else:\n chosen_tokens = self.processing_class(\n chosen, truncation=True, max_length=self.max_completion_length, add_special_tokens=True\n )\n rejected_tokens = self.processing_class(\n rejected, truncation=True, max_length=self.max_completion_length, add_special_tokens=True\n )\n prompt_tokens = self.processing_class(prompt, add_special_tokens=True)\n\n batch[\"chosen_labels\"] = chosen_tokens[\"input_ids\"]\n batch[\"rejected_labels\"] = rejected_tokens[\"input_ids\"]\n batch[\"prompt_input_ids\"] = prompt_tokens[\"input_ids\"]\n batch[\"prompt_attention_mask\"] = prompt_tokens[\"attention_mask\"]\n\n if model is not None and hasattr(model, \"prepare_decoder_input_ids_from_labels\"):\n batch[\"rejected_decoder_input_ids\"] = model.prepare_decoder_input_ids_from_labels(\n labels=torch.tensor(batch[\"rejected_labels\"])\n )\n batch[\"chosen_decoder_input_ids\"] = model.prepare_decoder_input_ids_from_labels(\n labels=torch.tensor(batch[\"chosen_labels\"])\n )\n\n if is_torch_xla_available():\n # Pad the sequences to global max_length to avoid TorchXLA recompilation\n for k in batch:\n if \"labels\" in k or self.is_encoder_decoder:\n pad_value = -100\n elif k.endswith(\"_input_ids\"):\n pad_value = self.padding_value\n elif k.endswith(\"_attention_mask\"):\n pad_value = 0\n batch[k] = batch[k] + [pad_value] * (self.max_length - len(batch[k]))\n return batch\n\n @staticmethod\n def concatenated_inputs(\n batch: dict[str, list | torch.LongTensor],\n is_encoder_decoder: bool = False,\n padding_value: int = 0,\n device: torch.device | None = None,\n ) -> dict[str, torch.LongTensor]:\n \"\"\"Concatenate the chosen and rejected inputs into a single tensor.\n\n Args:\n batch:\n A batch of data. Must contain the keys 'chosen_input_ids' and 'rejected_input_ids', which are tensors\n of shape (batch_size, sequence_length).\n is_encoder_decoder:\n Whether the model is an encoder-decoder model.\n padding_value:\n The padding value to use for the concatenated inputs_ids.\n device:\n The device for the concatenated inputs.\n\n Returns:\n A dictionary containing the concatenated inputs under the key 'concatenated_input_ids'.\n \"\"\"\n concatenated_batch = {}\n\n if is_encoder_decoder:\n max_length = max(batch[\"chosen_labels\"].shape[1], batch[\"rejected_labels\"].shape[1])\n else:\n max_length = max(batch[\"chosen_input_ids\"].shape[1], batch[\"rejected_input_ids\"].shape[1])\n\n for k in batch:\n if k.startswith(\"chosen\") and isinstance(batch[k], torch.Tensor):\n if \"labels\" in k or is_encoder_decoder:\n pad_value = -100\n elif k.endswith(\"_input_ids\"):\n pad_value = padding_value\n elif k.endswith(\"_attention_mask\"):\n pad_value = 0\n concatenated_key = k.replace(\"chosen\", \"concatenated\")\n concatenated_batch[concatenated_key] = pad_to_length(batch[k], max_length, pad_value=pad_value)\n for k in batch:\n if k.startswith(\"rejected\") and isinstance(batch[k], torch.Tensor):\n if \"labels\" in k or is_encoder_decoder:\n pad_value = -100\n elif k.endswith(\"_input_ids\"):\n pad_value = padding_value\n elif k.endswith(\"_attention_mask\"):\n pad_value = 0\n concatenated_key = k.replace(\"rejected\", \"concatenated\")\n concatenated_batch[concatenated_key] = torch.cat(\n (\n concatenated_batch[concatenated_key],\n pad_to_length(batch[k], max_length, pad_value=pad_value),\n ),\n dim=0,\n ).to(device=device)\n\n if is_encoder_decoder:\n concatenated_batch[\"concatenated_input_ids\"] = batch[\"prompt_input_ids\"].repeat(2, 1).to(device=device)\n concatenated_batch[\"concatenated_attention_mask\"] = (\n batch[\"prompt_attention_mask\"].repeat(2, 1).to(device=device)\n )\n\n return concatenated_batch\n\n def odds_ratio_loss(\n self,\n policy_chosen_logps: torch.FloatTensor,\n policy_rejected_logps: torch.FloatTensor,\n ) -> tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n \"\"\"Compute ORPO's odds ratio (OR) loss for a batch of policy and reference model log probabilities.\n\n Args:\n policy_chosen_logps:\n Log probabilities of the policy model for the chosen responses. Shape: (batch_size,)\n policy_rejected_logps:\n Log probabilities of the policy model for the rejected responses. Shape: (batch_size,)\n\n Returns:\n A tuple of three tensors: (losses, chosen_rewards, rejected_rewards). The losses tensor contains the ORPO\n loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for\n the chosen and rejected responses, respectively. The log odds ratio of the chosen responses over the\n rejected responses ratio for logging purposes. The `log(sigmoid(log_odds_chosen))` for logging purposes.\n \"\"\"\n\n # Derived from Eqs. (4) and (7) from https://huggingface.co/papers/2403.07691 by using log identities and exp(log(P(y|x)) = P(y|x)\n policy_chosen_logps = policy_chosen_logps.float()\n policy_rejected_logps = policy_rejected_logps.float()\n log_odds = (policy_chosen_logps - policy_rejected_logps) - (\n log1mexp(policy_chosen_logps) - log1mexp(policy_rejected_logps)\n )\n ratio = F.logsigmoid(log_odds)\n losses = self.beta * ratio\n\n chosen_rewards = self.beta * (policy_chosen_logps.to(self.accelerator.device)).detach()\n rejected_rewards = self.beta * (policy_rejected_logps.to(self.accelerator.device)).detach()\n\n return losses, chosen_rewards, rejected_rewards, torch.mean(ratio), torch.mean(log_odds)\n\n @staticmethod\n def get_batch_logps(\n logits: torch.FloatTensor,\n labels: torch.LongTensor,\n average_log_prob: bool = False,\n is_encoder_decoder: bool = False,\n ) -> torch.FloatTensor:\n \"\"\"Compute the log probabilities of the given labels under the given logits.\n\n Args:\n logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)\n labels:\n Labels for which to compute the log probabilities. Label tokens with a value of `-100` are ignored.\n Shape: (batch_size, sequence_length)\n average_log_prob:\n If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the\n log probabilities of the (non-masked) tokens.\n is_encoder_decoder: Whether the model is an encoder-decoder model.\n\n Returns:\n A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the\n given logits.\n \"\"\"\n if logits.shape[:-1] != labels.shape:\n raise ValueError(\"Logits (batch and sequence length dim) and labels must have the same shape.\")\n\n if not is_encoder_decoder:\n labels = labels[:, 1:].clone()\n logits = logits[:, :-1, :]\n loss_mask = labels != -100\n\n # dummy token; we'll ignore the losses on these tokens later\n labels = torch.where(labels == -100, 0, labels)\n\n per_token_logps = selective_log_softmax(logits, labels)\n\n if average_log_prob:\n return (per_token_logps * loss_mask).sum(-1) / loss_mask.sum(-1)\n else:\n return (per_token_logps * loss_mask).sum(-1)\n\n def concatenated_forward(\n self, model: nn.Module, batch: dict[str, list | torch.LongTensor]\n ) -> tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n \"\"\"Run the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.\n\n We do this to avoid doing two forward passes, because it's faster for FSDP.\n \"\"\"\n concatenated_batch = self.concatenated_inputs(\n batch,\n is_encoder_decoder=self.is_encoder_decoder,\n padding_value=self.padding_value,\n device=self.accelerator.device,\n )\n len_chosen = batch[\"chosen_labels\"].shape[0]\n\n model_kwargs = (\n {\n \"decoder_input_ids\": self._shift_right(concatenated_batch[\"concatenated_labels\"]),\n }\n if self.is_encoder_decoder\n else {}\n )\n\n if self.aux_loss_enabled:\n model_kwargs[\"output_router_logits\"] = True\n\n outputs = model(\n concatenated_batch[\"concatenated_input_ids\"],\n attention_mask=concatenated_batch[\"concatenated_attention_mask\"],\n use_cache=False,\n **model_kwargs,\n )\n all_logits = outputs.logits\n\n def cross_entropy_loss(logits, labels):\n if not self.is_encoder_decoder:\n # Shift so that tokens < n predict n\n logits = logits[..., :-1, :].contiguous()\n labels = labels[..., 1:].contiguous()\n # Flatten the tokens\n loss_fct = nn.CrossEntropyLoss()\n logits = logits.view(-1, logits.shape[-1])\n labels = labels.view(-1)\n # Enable model parallelism\n labels = labels.to(logits.device)\n loss = loss_fct(logits, labels)\n return loss\n\n if self.is_encoder_decoder:\n labels = concatenated_batch[\"concatenated_labels\"].clone()\n else:\n labels = concatenated_batch[\"concatenated_input_ids\"].clone()\n attention_mask = concatenated_batch[\"concatenated_attention_mask\"]\n labels = torch.where(attention_mask == 1, labels, -100)\n # orpo chosen nll loss is computed over the full prompt and response\n chosen_nll_loss = cross_entropy_loss(all_logits[:len_chosen], labels[:len_chosen])\n\n all_logps = self.get_batch_logps(\n all_logits,\n concatenated_batch[\"concatenated_labels\"],\n average_log_prob=True,\n is_encoder_decoder=self.is_encoder_decoder,\n )\n\n chosen_logps = all_logps[:len_chosen]\n rejected_logps = all_logps[len_chosen:]\n\n if not self.is_encoder_decoder:\n chosen_logits = all_logits[:len_chosen, :-1, :]\n rejected_logits = all_logits[len_chosen:, :-1, :]\n else:\n chosen_logits = all_logits[:len_chosen]\n rejected_logits = all_logits[len_chosen:]\n\n if self.aux_loss_enabled:\n return (chosen_logps, rejected_logps, chosen_logits, rejected_logits, chosen_nll_loss, outputs.aux_loss)\n\n return (chosen_logps, rejected_logps, chosen_logits, rejected_logits, chosen_nll_loss)\n\n def get_batch_loss_metrics(\n self,\n model,\n batch: dict[str, list | torch.LongTensor],\n train_eval: Literal[\"train\", \"eval\"] = \"train\",\n ):\n \"\"\"Compute the ORPO loss and other metrics for the given batch of inputs for train or test.\"\"\"\n metrics = {}\n\n forward_output = self.concatenated_forward(model, batch)\n (\n policy_chosen_logps,\n policy_rejected_logps,\n policy_chosen_logits,\n policy_rejected_logits,\n policy_nll_loss,\n ) = forward_output[:5]\n if self.aux_loss_enabled:\n aux_loss = forward_output[5]\n\n losses, chosen_rewards, rejected_rewards, log_odds_ratio, log_odds_chosen = self.odds_ratio_loss(\n policy_chosen_logps, policy_rejected_logps\n )\n # full ORPO loss\n loss = policy_nll_loss - losses.mean()\n\n reward_accuracies = (chosen_rewards > rejected_rewards).float()\n\n prefix = \"eval_\" if train_eval == \"eval\" else \"\"\n metrics[f\"{prefix}rewards/chosen\"] = self.accelerator.gather_for_metrics(chosen_rewards).mean()\n metrics[f\"{prefix}rewards/rejected\"] = self.accelerator.gather_for_metrics(rejected_rewards).mean()\n metrics[f\"{prefix}rewards/accuracies\"] = self.accelerator.gather_for_metrics(reward_accuracies).mean()\n metrics[f\"{prefix}rewards/margins\"] = self.accelerator.gather_for_metrics(\n chosen_rewards - rejected_rewards\n ).mean()\n metrics[f\"{prefix}logps/rejected\"] = self.accelerator.gather_for_metrics(policy_rejected_logps).detach().mean()\n metrics[f\"{prefix}logps/chosen\"] = self.accelerator.gather_for_metrics(policy_chosen_logps).detach().mean()\n metrics[f\"{prefix}logits/rejected\"] = self.accelerator.gather_for_metrics(\n policy_rejected_logits.detach().mean()\n ).mean()\n metrics[f\"{prefix}logits/chosen\"] = self.accelerator.gather_for_metrics(\n policy_chosen_logits.detach().mean()\n ).mean()\n metrics[f\"{prefix}nll_loss\"] = self.accelerator.gather_for_metrics(policy_nll_loss).detach().mean()\n metrics[f\"{prefix}log_odds_ratio\"] = self.accelerator.gather_for_metrics(log_odds_ratio).detach().mean()\n metrics[f\"{prefix}log_odds_chosen\"] = self.accelerator.gather_for_metrics(log_odds_chosen).detach().mean()\n if is_torch_xla_available():\n xm.mark_step() # needed because .item() calls\n for k, v in metrics.items():\n metrics[k] = v.item()\n if self.aux_loss_enabled:\n loss += self.aux_loss_coef * aux_loss\n\n return loss, metrics\n\n def compute_loss(\n self,\n model: PreTrainedModel | nn.Module,\n inputs: dict[str, torch.Tensor | Any],\n return_outputs=False,\n num_items_in_batch=None,\n ) -> torch.Tensor | tuple[torch.Tensor, dict[str, torch.Tensor]]:\n compute_loss_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n\n with compute_loss_context_manager:\n loss, metrics = self.get_batch_loss_metrics(model, inputs, train_eval=\"train\")\n\n # Make sure to move the loss to the device the original accumulating loss is at back in the `Trainer` class:\n loss = loss.to(self.args.device)\n\n # force log the metrics\n self.store_metrics(metrics, train_eval=\"train\")\n\n if return_outputs:\n return (loss, metrics)\n return loss\n\n def generate_from_model(self, model, batch: dict[str, torch.LongTensor]) -> str:\n \"\"\"Generate samples from the model and reference model for the given batch of inputs.\"\"\"\n\n # If one uses `generate_during_eval` with peft + bf16, we need to explicitly call generate with\n # the torch amp context manager as some hidden states are silently casted to full precision.\n generate_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n\n with generate_context_manager:\n policy_output = model.generate(\n input_ids=batch[\"prompt_input_ids\"],\n attention_mask=batch[\"prompt_attention_mask\"],\n max_length=self.max_length,\n do_sample=True,\n pad_token_id=self.processing_class.pad_token_id,\n )\n\n policy_output = pad_to_length(policy_output, self.max_length, self.processing_class.pad_token_id)\n policy_output_decoded = self.processing_class.batch_decode(policy_output, skip_special_tokens=True)\n\n return policy_output_decoded\n\n def prediction_step(\n self,\n model: PreTrainedModel | nn.Module,\n inputs: dict[str, torch.Tensor | Any],\n prediction_loss_only: bool,\n ignore_keys: list[str] | None = None,\n ):\n if not self.use_dpo_data_collator:\n logger.warning(\n \"prediction_step is only implemented for DPODataCollatorWithPadding, and you passed a datacollator that is different than \"\n \"DPODataCollatorWithPadding - you might see unexpected behavior. Alternatively, you can implement your own prediction_step method if you are using a custom data collator\"\n )\n if ignore_keys is None:\n if hasattr(model, \"config\"):\n ignore_keys = getattr(model.config, \"keys_to_ignore_at_inference\", [])\n else:\n ignore_keys = []\n\n prediction_context_manager = (\n autocast(self.accelerator.device.type) if self._peft_has_been_casted_to_bf16 else nullcontext()\n )\n\n with torch.no_grad(), prediction_context_manager:\n loss, metrics = self.get_batch_loss_metrics(model, inputs, train_eval=\"eval\")\n\n # force log the metrics\n self.store_metrics(metrics, train_eval=\"eval\")\n\n if prediction_loss_only:\n return (loss.detach(), None, None)\n\n # logits for the chosen and rejected samples from model\n logits_dict = {\n \"eval_logits/chosen\": metrics[\"eval_logits/chosen\"],\n \"eval_logits/rejected\": metrics[\"eval_logits/rejected\"],\n }\n logits = [v for k, v in logits_dict.items() if k not in ignore_keys]\n logits = torch.tensor(logits, device=self.accelerator.device)\n labels = torch.zeros(logits.shape[0], device=self.accelerator.device)\n\n return (loss.detach(), logits, labels)\n\n def store_metrics(self, metrics: dict[str, float], train_eval: Literal[\"train\", \"eval\"] = \"train\") -> None:\n for key, value in metrics.items():\n self._stored_metrics[train_eval][key].append(value)\n\n def evaluation_loop(\n self,\n dataloader: DataLoader,\n description: str,\n prediction_loss_only: bool | None = None,\n ignore_keys: list[str] | None = None,\n metric_key_prefix: str = \"eval\",\n ) -> EvalLoopOutput:\n \"\"\"\n Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by\n `Trainer.evaluate()` and `Trainer.predict()`.\n\n Works both with or without labels.\n \"\"\"\n\n # Sample and save to game log if requested (for one batch to save time)\n if self.generate_during_eval:\n # Generate random indices within the range of the total number of samples\n num_samples = len(dataloader.dataset)\n random_indices = random.sample(range(num_samples), k=self.args.eval_batch_size)\n\n # Use dataloader.dataset.select to get the random batch without iterating over the DataLoader\n random_batch_dataset = dataloader.dataset.select(random_indices)\n random_batch = self.data_collator(random_batch_dataset)\n random_batch = self._prepare_inputs(random_batch)\n\n policy_output_decoded = self.generate_from_model(self.model, random_batch)\n\n table = pd.DataFrame(\n columns=[\"Prompt\", \"Policy\"],\n data=[\n [prompt, pol[len(prompt) :]]\n for prompt, pol in zip(random_batch[\"prompt\"], policy_output_decoded, strict=True)\n ],\n )\n if \"wandb\" in self.args.report_to:\n wandb.log({\"game_log\": wandb.Table(data=table)})\n\n if \"comet_ml\" in self.args.report_to:\n log_table_to_comet_experiment(\n name=\"game_log.csv\",\n table=table,\n )\n\n # Base evaluation\n initial_output = super().evaluation_loop(\n dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix\n )\n\n return initial_output\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n \"\"\"\n Log `logs` on the various objects watching training, including stored metrics.\n\n Args:\n logs (`dict[str, float]`):\n The values to log.\n start_time (`float`, *optional*):\n Start time of the training.\n \"\"\"\n # logs either has 'loss' or 'eval_loss'\n train_eval = \"train\" if \"loss\" in logs else \"eval\"\n # Add averaged stored metrics to logs\n for key, metrics in self._stored_metrics[train_eval].items():\n logs[key] = torch.tensor(metrics).mean().item()\n del self._stored_metrics[train_eval]\n return super().log(logs, start_time)\n\n def _shift_right(self, input_ids):\n if self.decoder_start_token_id is None:\n raise ValueError(\n \"model.config.decoder_start_token_id has to be defined. It is usually set to the pad_token_id.\"\n )\n\n # shift inputs to the right\n if is_torch_fx_proxy(input_ids):\n # Item assignment is not supported natively for proxies.\n shifted_input_ids = torch.full(input_ids.shape[:-1] + (1,), self.decoder_start_token_id)\n shifted_input_ids = torch.cat([shifted_input_ids, input_ids[..., :-1]], dim=-1)\n else:\n shifted_input_ids = input_ids.new_zeros(input_ids.shape)\n shifted_input_ids[..., 1:] = input_ids[..., :-1].clone()\n shifted_input_ids[..., 0] = self.decoder_start_token_id\n\n if self.pad_token_id is None:\n raise ValueError(\"model.config.pad_token_id has to be defined.\")\n # replace possible -100 values in labels by `pad_token_id`\n shifted_input_ids.masked_fill_(shifted_input_ids == -100, self.pad_token_id)\n\n return shifted_input_ids\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/experimental/papo/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\nfrom .papo_config import PAPOConfig\nfrom .papo_trainer import PAPOTrainer\n" }, { "path": "trl/experimental/papo/papo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass\nfrom typing import Literal\n\nfrom ...trainer.grpo_config import GRPOConfig\n\n\n@dataclass\nclass PAPOConfig(GRPOConfig):\n \"\"\"\n Configuration class for PAPOTrainer.\n\n PAPO (Perception-Aware Policy Optimization) extends GRPO/DAPO for multimodal reasoning by adding an implicit\n perception loss and double entropy regularization.\n\n Args:\n perception_loss_weight (`float`, *optional*, defaults to `0.1`):\n gamma Weight coefficient for the perception loss term. This encourages the model to be sensitive to visual\n changes.\n\n mask_ratio (`float`, *optional*, defaults to `0.3`):\n Ratio of the image to mask when computing perception loss.\n\n mask_type (`Literal[\"random\", \"patch\", \"grid\"]`, *optional*, defaults to `\"random\"`):\n Type of masking strategy to use.\n\n der_loss_weight1 (`float`, *optional*, defaults to `0.03`):\n eta1 Weight coefficient for the Double Entropy Regularization (DER) term. This term encourages confident\n predictions with original images (low entropy) and uncertain predictions with masked images (high entropy).\n\n der_loss_weight2 (`float`, *optional*, defaults to `0.03`):\n eta2 Weight coefficient for the Double Entropy Regularization (DER) term. This term encourages confident\n predictions with original images (low entropy) and uncertain predictions with masked images (high entropy).\n\n loss_type (`Literal[\"grpo\", \"dapo\"]`, inherited from GRPOConfig):\n Base loss type to use. Set to \"grpo\" for PAPO-G or \"dapo\" for PAPO-D.\n \"\"\"\n\n perception_loss_weight: float = 0.1\n mask_ratio: float = 0.3\n mask_type: Literal[\"random\", \"patch\", \"grid\"] = \"random\"\n\n # Added for Double Entropy Regularization\n der_loss_weight1: float = 0.03\n der_loss_weight2: float = 0.03\n\n def __post_init__(self):\n super().__post_init__()\n\n # Validation\n if not 0.0 <= self.mask_ratio <= 1.0:\n raise ValueError(f\"mask_ratio must be between 0 and 1, got {self.mask_ratio}\")\n\n if self.der_loss_weight1 < 0 or self.der_loss_weight2 < 0:\n raise ValueError(\n f\"der_loss_weight1 and der_loss_weight2 must be non-negative, got {self.der_loss_weight1} and {self.der_loss_weight2}\"\n )\n\n if self.mask_type not in [\"random\", \"patch\", \"grid\"]:\n raise ValueError(f\"mask_type must be one of ['random', 'patch', 'grid'], got {self.mask_type}\")\n" }, { "path": "trl/experimental/papo/papo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport random\nimport textwrap\n\nimport torch\nfrom datasets import Dataset, IterableDataset\nfrom transformers import PreTrainedModel, PreTrainedTokenizerBase, ProcessorMixin\n\nfrom ...trainer.grpo_trainer import GRPOTrainer, RewardFunc\nfrom ...trainer.utils import nanmax, nanmin\nfrom .papo_config import PAPOConfig\n\n\nclass PAPOTrainer(GRPOTrainer):\n \"\"\"\n Trainer for Perception-Aware Policy Optimization (PAPO).\n\n PAPO extends GRPO/DAPO for multimodal reasoning by adding an implicit perception loss that encourages the model to\n better utilize visual information. The key innovation is computing KL divergence between model outputs on original\n vs. corrupted (masked) images.\n\n Two variants are supported:\n - PAPO-G: PAPO + GRPO (use loss_type=\"grpo\")\n - PAPO-D: PAPO + DAPO (use loss_type=\"dapo\")\n\n Example:\n\n ```python\n from datasets import load_dataset\n from trl.experimental.papo import PAPOTrainer, PAPOConfig\n\n dataset = load_dataset(\"your-vlm-dataset\", split=\"train\")\n\n\n def reward_func(completions, **kwargs):\n # Your reward function for multimodal reasoning\n return [compute_reward(c) for c in completions]\n\n\n # PAPO-G\n config = PAPOConfig(\n loss_type=\"grpo\", # Use GRPO as base\n perception_loss_weight=0.1,\n mask_ratio=0.3,\n )\n\n # PAPO-G\n config = PAPOConfig(\n loss_type=\"dapo\", # Use DAPO as base\n perception_loss_weight=0.1,\n mask_ratio=0.3,\n )\n\n trainer = PAPOTrainer(\n model=\"Qwen/Qwen2-VL-2B-Instruct\",\n reward_funcs=reward_func,\n args=config,\n train_dataset=dataset,\n )\n\n trainer.train()\n ```\n\n Args:\n model (`Union[str, PreTrainedModel]`):\n Model to be trained (must be a vision-language model).\n reward_funcs (`Union[RewardFunc, list[RewardFunc]]`):\n Reward functions for computing rewards (same as GRPO).\n args ([`PAPOConfig`], *optional*, defaults to `None`):\n Configuration for this trainer. If `None`, a default configuration is used.\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n Dataset to use for training. Must include \"prompt\" and \"image\" columns.\n eval_dataset: Same requirements as train_dataset.\n processing_class: Processing class (tokenizer/processor) for the model.\n reward_processing_classes: Processing classes for reward models.\n callbacks: Training callbacks.\n optimizers: Optimizer and scheduler tuple.\n peft_config: PEFT configuration if using parameter-efficient fine-tuning.\n \"\"\"\n\n _tag_names = [\"trl\", \"papo\"]\n _name = \"PAPO\"\n _paper = {\n \"title\": \"Perception-Aware Policy Optimization for Multimodal Reasoning\",\n \"id\": \"2507.06448\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\n \"\"\"\\\n @misc{wang2025perceptionawarepolicyoptimizationmultimodal,\n title = {{Perception-Aware Policy Optimization for Multimodal Reasoning}},\n author = {Zhenhailong Wang and Xuehang Guo and Sofia Stoica and Haiyang Xu and Hongru Wang and Hyeonjeong Ha and Xiusi Chen and Yangyi Chen and Ming Yan and Fei Huang and Heng Ji},\n year = 2025,\n url = {https://arxiv.org/abs/2507.06448},\n archivePrefix= {arXiv},\n eprint = {2507.06448},\n primaryClass = {cs.CL}\n }\"\"\"\n ),\n }\n\n def __init__(\n self,\n model: str | PreTrainedModel,\n reward_funcs: RewardFunc | list[RewardFunc],\n args: PAPOConfig | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | IterableDataset | dict[str, Dataset | IterableDataset] | None = None,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin | None = None,\n reward_processing_classes: PreTrainedTokenizerBase | list[PreTrainedTokenizerBase] | None = None,\n callbacks=None,\n optimizers=(None, None),\n peft_config=None,\n ):\n # Initialize with default PAPO config if not provided\n if args is None:\n model_name = model if isinstance(model, str) else model.config._name_or_path\n model_name = model_name.split(\"/\")[-1]\n args = PAPOConfig(f\"{model_name}-PAPO\")\n\n # Store PAPO-specific parameters\n self.perception_loss_weight = args.perception_loss_weight\n self.mask_ratio = args.mask_ratio\n self.mask_type = args.mask_type\n self.der_loss_weight1 = args.der_loss_weight1\n self.der_loss_weight2 = args.der_loss_weight2\n\n # Initialize parent GRPO trainer\n super().__init__(\n model=model,\n reward_funcs=reward_funcs,\n args=args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n reward_processing_classes=reward_processing_classes,\n callbacks=callbacks,\n optimizers=optimizers,\n peft_config=peft_config,\n )\n\n def _mask_image(self, pixel_values: torch.Tensor, mask_ratio: float = None) -> torch.Tensor:\n \"\"\"\n Apply masking to image pixel values.\n\n Args:\n pixel_values: Image tensor of shape (B, C, H, W) or (B, N, C, H, W) for multi-image\n mask_ratio: Ratio of image to mask (defaults to self.mask_ratio)\n\n Returns:\n Masked pixel values tensor\n \"\"\"\n if mask_ratio is None:\n mask_ratio = self.mask_ratio\n\n masked_pixel_values = pixel_values.clone()\n\n if self.mask_type == \"random\":\n # Random pixel masking\n mask = torch.rand_like(pixel_values) > mask_ratio\n masked_pixel_values = masked_pixel_values * mask\n\n elif self.mask_type == \"patch\":\n # Patch-based masking (mask contiguous regions)\n B = pixel_values.shape[0]\n if pixel_values.ndim == 4: # (B, C, H, W)\n C, H, W = pixel_values.shape[1:]\n for i in range(B):\n # Calculate patch size to mask\n patch_h = int(H * mask_ratio**0.5)\n patch_w = int(W * mask_ratio**0.5)\n # Random starting position\n start_h = random.randint(0, max(0, H - patch_h))\n start_w = random.randint(0, max(0, W - patch_w))\n # Apply mask\n masked_pixel_values[i, :, start_h : start_h + patch_h, start_w : start_w + patch_w] = 0\n\n elif pixel_values.ndim == 5: # (B, N, C, H, W) for multi-image\n N, C, H, W = pixel_values.shape[1:]\n for i in range(B):\n for n in range(N):\n patch_h = int(H * mask_ratio**0.5)\n patch_w = int(W * mask_ratio**0.5)\n start_h = random.randint(0, max(0, H - patch_h))\n start_w = random.randint(0, max(0, W - patch_w))\n masked_pixel_values[i, n, :, start_h : start_h + patch_h, start_w : start_w + patch_w] = 0\n\n elif self.mask_type == \"grid\":\n # Grid-based masking (mask regular grid cells)\n if pixel_values.ndim == 4:\n C, H, W = pixel_values.shape[1:]\n grid_size = int((1 / mask_ratio) ** 0.5)\n cell_h, cell_w = H // grid_size, W // grid_size\n\n for i in range(grid_size):\n for j in range(grid_size):\n if random.random() < mask_ratio:\n masked_pixel_values[:, :, i * cell_h : (i + 1) * cell_h, j * cell_w : (j + 1) * cell_w] = 0\n\n return masked_pixel_values\n\n def _compute_loss(self, model, inputs):\n # >>> 1. GRPO loss\n # Compute the per-token log probabilities for the model\n prompt_ids, prompt_mask = inputs[\"prompt_ids\"], inputs[\"prompt_mask\"]\n completion_ids, completion_mask = inputs[\"completion_ids\"], inputs[\"completion_mask\"]\n input_ids = torch.cat([prompt_ids, completion_ids], dim=1)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1)\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n\n # Compute the per_token_logps and the entropy at each position in the completion\n per_token_logps, entropies = self._get_per_token_logps_and_entropies(\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n compute_entropy=True,\n pixel_values=inputs.get(\"pixel_values\"),\n image_grid_thw=inputs.get(\"image_grid_thw\"),\n num_images=inputs.get(\"num_images\"),\n pixel_attention_mask=inputs.get(\"pixel_attention_mask\"),\n image_sizes=inputs.get(\"image_sizes\"),\n )\n\n if self.top_entropy_quantile < 1.0:\n entropy_mask = self.get_high_entropy_mask(entropies, completion_mask, 1 - self.top_entropy_quantile)\n else:\n entropy_mask = None\n\n # Compute the KL divergence between the model and the reference model\n if self.beta != 0.0:\n ref_per_token_logps = inputs[\"ref_per_token_logps\"]\n per_token_kl = (\n torch.exp(ref_per_token_logps - per_token_logps) - (ref_per_token_logps - per_token_logps) - 1\n )\n\n # Compute the loss\n advantages = inputs[\"advantages\"]\n # When using num_iterations == 1 and steps_per_generation <= gradient_accumulation_steps\n # old_per_token_logps == per_token_logps, so we can skip it's computation\n # (see _generate_and_score_completions) and use per_token_logps.detach() instead.\n old_per_token_logps = inputs.get(\"old_per_token_logps\")\n old_per_token_logps = per_token_logps.detach() if old_per_token_logps is None else old_per_token_logps\n\n log_ratio = per_token_logps - old_per_token_logps\n if self.importance_sampling_level == \"token\":\n log_importance_weights = log_ratio\n elif self.importance_sampling_level == \"sequence\":\n log_importance_weights = (log_ratio * completion_mask).sum(-1) / completion_mask.sum(-1).clamp(min=1.0)\n log_importance_weights = log_importance_weights.unsqueeze(-1)\n else:\n raise ValueError(\n f\"Unknown importance sampling level: {self.importance_sampling_level}. Possible values are 'token' \"\n \"and 'sequence'.\"\n )\n # From here, log_importance_weights (and all subsequent tensors, coef_1, coef_2, etc.) shape depends on\n # importance_sampling_level: \"token\" level: (B, T); \"sequence\" level: (B, 1)\n\n coef_1 = torch.exp(log_importance_weights)\n coef_2 = torch.clamp(coef_1, 1 - self.epsilon_low, 1 + self.epsilon_high)\n\n # Two-sided clipping\n if self.args.delta is not None:\n coef_1 = torch.clamp(coef_1, max=self.args.delta)\n\n per_token_loss1 = coef_1 * advantages.unsqueeze(1)\n per_token_loss2 = coef_2 * advantages.unsqueeze(1)\n per_token_loss = -torch.min(per_token_loss1, per_token_loss2)\n if entropy_mask is not None:\n per_token_loss = per_token_loss * entropy_mask\n if self.beta != 0.0:\n per_token_loss = per_token_loss + self.beta * per_token_kl\n\n if self.loss_type == \"grpo\":\n loss = ((per_token_loss * completion_mask).sum(-1) / completion_mask.sum(-1).clamp(min=1.0)).mean()\n loss = loss / self.current_gradient_accumulation_steps\n elif self.loss_type == \"dapo\":\n normalizer = inputs[\"num_items_in_batch\"] / self.accelerator.num_processes\n loss = (per_token_loss * completion_mask).sum() / normalizer\n else:\n raise ValueError(f\"Unknown loss type: {self.loss_type}\")\n # >>> 2. Implicit Perception Loss\n inputs[\"pixel_values\"] = self._mask_image(inputs[\"pixel_values\"], self.mask_ratio)\n mask_img_per_token_logps, mask_img_entropies = self._get_per_token_logps_and_entropies(\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n compute_entropy=True,\n pixel_values=inputs.get(\"pixel_values\"),\n image_grid_thw=inputs.get(\"image_grid_thw\"),\n num_images=inputs.get(\"num_images\"),\n pixel_attention_mask=inputs.get(\"pixel_attention_mask\"),\n image_sizes=inputs.get(\"image_sizes\"),\n )\n perception_kl = (\n torch.exp(mask_img_per_token_logps - per_token_logps) - (mask_img_per_token_logps - per_token_logps) - 1\n )\n perception_kl = torch.clamp(perception_kl, min=0.0, max=0.2)\n perception_loss = self.perception_loss_weight * perception_kl\n\n # >>> 3. Double Entropy Loss\n der_loss = self.der_loss_weight1 * entropies + self.der_loss_weight2 * mask_img_entropies\n\n # PAPO Loss\n loss = (loss - perception_loss + der_loss).mean()\n # Log the metrics\n mode = \"train\" if self.model.training else \"eval\"\n\n completion_token_count = completion_mask.sum().clamp(min=1.0)\n\n def masked_batch_mean(x):\n if x.shape[1] == 1: # when importance_sampling_level == \"sequence\"\n return x.mean()\n else:\n return (x * completion_mask).sum() / completion_token_count\n\n if self.beta != 0.0:\n mean_kl = masked_batch_mean(per_token_kl)\n self._metrics[mode][\"kl\"].append(self.accelerator.gather(mean_kl).nanmean().item())\n\n mean_entropy = masked_batch_mean(entropies)\n self._metrics[mode][\"entropy\"].append(self.accelerator.gather(mean_entropy).nanmean().item())\n\n # Compute the clipped probability ratios\n is_low_clipped = (coef_1 < 1 - self.epsilon_low) & (advantages.unsqueeze(1) < 0)\n is_high_clipped = (coef_1 > 1 + self.epsilon_high) & (advantages.unsqueeze(1) > 0)\n is_region_clipped = is_low_clipped | is_high_clipped\n\n low_clip = masked_batch_mean(is_low_clipped.float())\n high_clip = masked_batch_mean(is_high_clipped.float())\n clip_ratio = masked_batch_mean(is_region_clipped.float())\n\n gathered_low_clip = self.accelerator.gather(low_clip)\n self._metrics[mode][\"clip_ratio/low_mean\"].append(gathered_low_clip.nanmean().item())\n self._metrics[mode][\"clip_ratio/low_min\"].append(nanmin(gathered_low_clip).item())\n gathered_high_clip = self.accelerator.gather(high_clip)\n self._metrics[mode][\"clip_ratio/high_mean\"].append(gathered_high_clip.nanmean().item())\n self._metrics[mode][\"clip_ratio/high_max\"].append(nanmax(gathered_high_clip).item())\n gathered_clip_ratio = self.accelerator.gather(clip_ratio)\n self._metrics[mode][\"clip_ratio/region_mean\"].append(gathered_clip_ratio.nanmean().item())\n return loss\n" }, { "path": "trl/experimental/ppo/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .modeling_value_head import (\n AutoModelForCausalLMWithValueHead,\n AutoModelForSeq2SeqLMWithValueHead,\n PreTrainedModelWrapper,\n)\nfrom .ppo_config import PPOConfig\nfrom .ppo_trainer import PPOTrainer\n\n\n__all__ = [\n \"AutoModelForCausalLMWithValueHead\",\n \"AutoModelForSeq2SeqLMWithValueHead\",\n \"PreTrainedModelWrapper\",\n \"PPOConfig\",\n \"PPOTrainer\",\n]\n" }, { "path": "trl/experimental/ppo/modeling_value_head.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport json\nimport logging\nimport os\n\nimport torch\nimport torch.nn as nn\nfrom accelerate import PartialState\nfrom huggingface_hub import hf_hub_download\nfrom huggingface_hub.utils import (\n EntryNotFoundError,\n HFValidationError,\n LocalEntryNotFoundError,\n RepositoryNotFoundError,\n)\nfrom safetensors.torch import load_file as safe_load_file\nfrom transformers import (\n AutoModelForCausalLM,\n AutoModelForSeq2SeqLM,\n PreTrainedModel,\n is_torch_npu_available,\n is_torch_xpu_available,\n)\nfrom transformers.utils import is_peft_available\n\n\nif is_peft_available():\n from peft import (\n PeftConfig,\n PeftModel,\n PeftModelForCausalLM,\n PeftModelForSeq2SeqLM,\n PromptLearningConfig,\n get_peft_model,\n prepare_model_for_kbit_training,\n )\n\n\nclass PreTrainedModelWrapper(nn.Module):\n \"\"\"\n Wrapper for a [`~transformers.PreTrainedModel`] implemented as a standard PyTorch [`torch.nn.Module`].\n\n This class provides a compatibility layer that preserves the key attributes and methods of the original\n [`~transformers.PreTrainedModel`], while exposing a uniform interface consistent with PyTorch modules. It enables\n seamless integration of pretrained Transformer models into custom training, evaluation, or inference workflows.\n\n Attributes:\n pretrained_model ([`~transformers.PreTrainedModel`]):\n The model to be wrapped.\n parent_class ([`~transformers.PreTrainedModel`]):\n The parent class of the model to be wrapped.\n supported_args (`list`):\n The list of arguments that are supported by the wrapper class.\n \"\"\"\n\n transformers_parent_class = None\n supported_args = None\n supported_modules = (\"v_head\",)\n supported_rm_modules = (\"score\",)\n supported_pretrained_model_architectures = (\n (PreTrainedModel)\n if not is_peft_available()\n else (PreTrainedModel, PeftModelForCausalLM, PeftModelForSeq2SeqLM)\n )\n\n def __init__(\n self, pretrained_model=None, score_module=None, supports_rm_adapter=False, rm_adapter_name=None, **kwargs\n ):\n super().__init__()\n self.pretrained_model = pretrained_model\n\n self.config = pretrained_model.config\n self.prepare_inputs_for_generation = pretrained_model.prepare_inputs_for_generation\n self.is_loaded_in_8bit = getattr(pretrained_model, \"is_loaded_in_8bit\", False)\n self.is_loaded_in_4bit = getattr(pretrained_model, \"is_loaded_in_4bit\", False)\n self.is_sequential_parallel = False\n\n if hasattr(pretrained_model, \"gradient_checkpointing_disable\"):\n self.gradient_checkpointing_disable = pretrained_model.gradient_checkpointing_disable\n\n if hasattr(pretrained_model, \"gradient_checkpointing_enable\"):\n self.gradient_checkpointing_enable = pretrained_model.gradient_checkpointing_enable\n\n if hasattr(pretrained_model, \"enable_input_require_grads\"):\n self.enable_input_require_grads = pretrained_model.enable_input_require_grads\n\n self.supports_rm_adapter = supports_rm_adapter\n self.rm_adapter_name = rm_adapter_name\n self.policy_adapter_name = \"default\"\n if score_module is not None:\n self.score = score_module\n\n @classmethod\n def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):\n r\"\"\"\n Instantiates a new model from a pretrained model from `transformers`. The pretrained model is loaded using the\n `from_pretrained` method of the [`~transformers.PreTrainedModel`] class. The arguments that are specific to the\n [`~transformers.PreTrainedModel`] class are passed along this method and filtered out from the `kwargs`\n argument.\n\n Args:\n pretrained_model_name_or_path (`str` or [`~transformers.PreTrainedModel`]):\n The path to the pretrained model or its name.\n *model_args (`list`, *optional*):\n Additional positional arguments passed along to the underlying model's `from_pretrained` method.\n **kwargs (`dict`, *optional*):\n Additional keyword arguments passed along to the underlying model's `from_pretrained` method. We also\n pre-process the kwargs to extract the arguments that are specific to the\n [`~transformers.PreTrainedModel`] class and the arguments that are specific to trl models. The kwargs\n also support `prepare_model_for_kbit_training` arguments from `peft` library.\n \"\"\"\n if kwargs is not None:\n peft_config = kwargs.pop(\"peft_config\", None)\n reward_adapter = kwargs.pop(\"reward_adapter\", None)\n reward_adapter_name = kwargs.pop(\"reward_adapter_name\", \"reward_adapter\")\n is_trainable = kwargs.pop(\"is_trainable\", False)\n trl_model_args, pretrained_kwargs, peft_quantization_kwargs = cls._split_kwargs(kwargs)\n token = pretrained_kwargs.get(\"token\", None)\n else:\n peft_config = None\n is_trainable = False\n trl_model_args = {}\n pretrained_kwargs = {}\n peft_quantization_kwargs = {}\n token = None\n\n if reward_adapter is not None and not isinstance(reward_adapter, str):\n raise ValueError(\n \"The `reward_adapter` argument should be a string representing the name of local path or the Hub id to the Reward Modeling adapter.\"\n )\n\n is_peft_model = False\n\n current_device = cls._get_current_device()\n if isinstance(pretrained_model_name_or_path, str):\n quantization_config = pretrained_kwargs.get(\"quantization_config\", None)\n if quantization_config is not None:\n is_loaded_in_8bit = getattr(quantization_config, \"load_in_8bit\", False)\n is_loaded_in_4bit = getattr(quantization_config, \"load_in_4bit\", False)\n else:\n is_loaded_in_8bit = pretrained_kwargs[\"load_in_8bit\"] if \"load_in_8bit\" in pretrained_kwargs else False\n is_loaded_in_4bit = pretrained_kwargs[\"load_in_4bit\"] if \"load_in_4bit\" in pretrained_kwargs else False\n else:\n is_loaded_in_8bit = getattr(pretrained_model_name_or_path, \"is_loaded_in_8bit\", False)\n is_loaded_in_4bit = getattr(pretrained_model_name_or_path, \"is_loaded_in_4bit\", False)\n\n if (is_loaded_in_8bit or is_loaded_in_4bit) and \"device_map\" not in pretrained_kwargs:\n # warn users\n logging.warning(\n \"The `device_map` argument is not provided. We will override the device_map argument.\"\n \" to set the entire\"\n \" model on the current device. If you want to set the model on multiple devices, please provide\"\n \" a custom `device_map` argument.\"\n )\n pretrained_kwargs[\"device_map\"] = {\"\": current_device}\n\n if is_peft_available() and peft_config is not None and not isinstance(peft_config, PeftConfig):\n raise ValueError(\"The `peft_config` argument should be an instance of `peft.PeftConfig` class.\")\n\n # First, load the pre-trained model using the parent-class\n # either `AutoModelForCausalLM` or `AutoModelForSeq2SeqLM`\n if isinstance(pretrained_model_name_or_path, str):\n if is_peft_available():\n try:\n # If there is a trained peft adapter in the hub, load its config.\n remote_adapter_config = hf_hub_download(\n pretrained_model_name_or_path,\n \"adapter_config.json\",\n token=token,\n )\n except (EntryNotFoundError, LocalEntryNotFoundError, HFValidationError, RepositoryNotFoundError):\n remote_adapter_config = None\n else:\n remote_adapter_config = None\n\n local_adapter_present = os.path.exists(os.path.join(pretrained_model_name_or_path, \"adapter_config.json\"))\n\n if (local_adapter_present or remote_adapter_config is not None) and is_peft_available():\n if peft_config is not None:\n logging.warning(\n \"`peft_config` argument ignored since a peft config file was found in \"\n f\"{pretrained_model_name_or_path}\"\n )\n\n # Load the trained peft adapter config\n if local_adapter_present:\n trained_adapter_config = PeftConfig.from_pretrained(pretrained_model_name_or_path)\n else:\n remote_adapter_dir = os.path.dirname(remote_adapter_config)\n trained_adapter_config = PeftConfig.from_pretrained(remote_adapter_dir)\n\n # Load the pretrained base model\n pretrained_model = cls.transformers_parent_class.from_pretrained(\n trained_adapter_config.base_model_name_or_path, *model_args, **pretrained_kwargs\n )\n\n # Wrap the pretrained model with the trained peft adapter\n pretrained_model = PeftModel.from_pretrained(\n pretrained_model, pretrained_model_name_or_path, is_trainable=is_trainable, token=token\n )\n logging.info(\"Trained peft adapter loaded\")\n else:\n pretrained_model = cls.transformers_parent_class.from_pretrained(\n pretrained_model_name_or_path, *model_args, **pretrained_kwargs\n )\n\n if peft_config is not None:\n # Initialize a new peft adapter with the given config\n if is_loaded_in_8bit or is_loaded_in_4bit:\n pretrained_model = prepare_model_for_kbit_training(\n pretrained_model,\n **peft_quantization_kwargs,\n )\n pretrained_model = get_peft_model(pretrained_model, peft_config)\n logging.info(\"peft adapter initialised\")\n\n elif isinstance(pretrained_model_name_or_path, cls.supported_pretrained_model_architectures):\n pretrained_model = pretrained_model_name_or_path\n\n if peft_config is not None and isinstance(pretrained_model, PreTrainedModel):\n # Initialize a new peft adapter with the given config\n if is_loaded_in_8bit or is_loaded_in_4bit:\n pretrained_model = prepare_model_for_kbit_training(\n pretrained_model,\n **peft_quantization_kwargs,\n )\n pretrained_model = get_peft_model(pretrained_model, peft_config)\n logging.info(\"peft adapter initialised\")\n else:\n raise ValueError(\n \"pretrained_model_name_or_path should be a string or a PreTrainedModel, \"\n f\"but is {type(pretrained_model_name_or_path)}\"\n )\n\n if is_peft_available():\n if isinstance(pretrained_model, PeftModel):\n is_peft_model = True\n # for backward compatibility\n if hasattr(pretrained_model, \"active_peft_config\") and isinstance(\n pretrained_model.active_peft_config, PromptLearningConfig\n ):\n raise ValueError(\"PromptLearningConfig is not supported for PPO training.\")\n\n # Add reward modeling adapter if specified\n if not is_peft_model and reward_adapter is not None:\n raise ValueError(\"reward_adapter can only be used with a PeftModel. \")\n elif is_peft_model and reward_adapter is not None:\n score_module = cls.add_and_load_reward_modeling_adapter(\n pretrained_model, reward_adapter, reward_adapter_name, token=token\n )\n multi_adapter_args = {\n \"score_module\": score_module,\n \"supports_rm_adapter\": True,\n \"rm_adapter_name\": reward_adapter_name,\n }\n else:\n multi_adapter_args = {\"supports_rm_adapter\": False}\n\n # Then, create the full model by instantiating the wrapper class\n model = cls(pretrained_model, **multi_adapter_args, **trl_model_args)\n\n # if resume_training, load the state_dict again - this is ok since the\n # state_dict is removed from the model after loading it.\n is_resuming_training = True\n if isinstance(pretrained_model_name_or_path, str):\n safe_filename = os.path.join(pretrained_model_name_or_path, \"model.safetensors\")\n filename = os.path.join(pretrained_model_name_or_path, \"pytorch_model.bin\")\n\n sharded_index_filename = os.path.join(pretrained_model_name_or_path, \"pytorch_model.bin.index.json\")\n safe_sharded_index_filename = os.path.join(pretrained_model_name_or_path, \"model.safetensors.index.json\")\n is_sharded = False\n use_safe = os.path.exists(safe_filename)\n\n if not (os.path.exists(filename) or os.path.exists(safe_filename)):\n # Try with `pytorch_model.bin`\n filename, files_to_download, is_sharded, is_resuming_training = cls._get_checkpoint_from_hub(\n pretrained_model,\n pretrained_model_name_or_path,\n sharded_index_filename,\n token=token,\n )\n # Try with safetensors\n if filename is None and files_to_download is None:\n safe_filename, files_to_download, is_sharded, is_resuming_training = cls._get_checkpoint_from_hub(\n pretrained_model,\n pretrained_model_name_or_path,\n safe_sharded_index_filename,\n token=token,\n model_name=\"model.safetensors\",\n model_index_name=\"model.safetensors.index.json\",\n )\n use_safe = True\n else:\n use_safe = False\n\n loading_func = safe_load_file if use_safe else torch.load\n load_kwargs = {} if use_safe else {\"map_location\": \"cpu\", \"weights_only\": True}\n\n if is_resuming_training:\n if is_sharded:\n # download each file and add it to the state_dict\n state_dict = {}\n\n for shard_file in files_to_download:\n filename = hf_hub_download(\n pretrained_model_name_or_path,\n shard_file,\n token=token,\n )\n state_dict.update(loading_func(filename, **load_kwargs))\n else:\n state_dict = loading_func(filename if not use_safe else safe_filename, **load_kwargs)\n\n else:\n state_dict = pretrained_model_name_or_path.state_dict()\n\n model.is_peft_model = is_peft_model\n model.current_device = current_device\n\n if is_resuming_training:\n model.post_init(state_dict=state_dict)\n\n return model\n\n @classmethod\n def _get_checkpoint_from_hub(\n cls,\n pretrained_model,\n pretrained_model_name_or_path,\n index_filename,\n token=None,\n model_name=\"pytorch_model.bin\",\n model_index_name=\"pytorch_model.bin.index.json\",\n ):\n files_to_download = None\n filename = None\n is_resuming_training = True\n is_sharded = False\n\n try:\n filename = hf_hub_download(\n pretrained_model_name_or_path,\n model_name,\n token=token,\n )\n # sharded\n except (EntryNotFoundError, LocalEntryNotFoundError, HFValidationError, RepositoryNotFoundError):\n if os.path.exists(index_filename):\n index_file_name = index_filename\n else:\n try:\n index_file_name = hf_hub_download(\n pretrained_model_name_or_path,\n model_index_name,\n token=token,\n )\n except (EntryNotFoundError, LocalEntryNotFoundError, HFValidationError, RepositoryNotFoundError):\n # not continue training, do not have v_head weight\n is_resuming_training = False\n logging.warning(\n f\"A {type(pretrained_model)} model is loaded from '{pretrained_model_name_or_path}', \"\n f\"and no v_head weight is found. This IS expected if you are not resuming PPO training.\"\n )\n # load json\n if is_resuming_training:\n with open(index_file_name) as f:\n index = json.load(f)\n # check filename with `v_head` or any known extra module:\n files_to_download = set()\n for k, v in index[\"weight_map\"].items():\n if any(module in k for module in cls.supported_modules):\n files_to_download.add(v)\n is_sharded = True\n\n return filename, files_to_download, is_sharded, is_resuming_training\n\n @classmethod\n def _get_current_device(cls):\n r\"\"\"\n Get the current device. For GPU & XPU, we return the local process index using the `accelerate.PartialState`\n object to handle corner cases when running scripts in distributed environments.\n\n Returns:\n current_device (`int | str`):\n The current device.\n \"\"\"\n state = PartialState()\n if torch.cuda.is_available() or is_torch_xpu_available():\n return state.local_process_index\n elif is_torch_npu_available():\n return f\"npu:{state.local_process_index}\"\n else:\n return \"cpu\"\n\n @classmethod\n def _split_kwargs(cls, kwargs):\n \"\"\"\n Separate the kwargs from the arguments that we support inside `supported_args` and the ones that we don't.\n \"\"\"\n check_peft_kwargs = False\n\n if is_peft_available():\n from peft import prepare_model_for_kbit_training\n\n check_peft_kwargs = True\n\n supported_kwargs = {}\n unsupported_kwargs = {}\n peft_kwargs = {}\n\n for key, value in kwargs.items():\n if key in cls.supported_args:\n supported_kwargs[key] = value\n else:\n unsupported_kwargs[key] = value\n\n if check_peft_kwargs:\n if key in prepare_model_for_kbit_training.__code__.co_varnames:\n peft_kwargs[key] = value\n if key in unsupported_kwargs:\n unsupported_kwargs.pop(key)\n\n return supported_kwargs, unsupported_kwargs, peft_kwargs\n\n @classmethod\n def add_and_load_reward_modeling_adapter(\n cls, pretrained_model, adapter_model_id, adapter_name=\"reward_model_adapter\", token=None\n ):\n r\"\"\"\n Add and load a reward modeling adapter. This method can only be used if the model is a `PeftModel` and if you\n have initialized the model with the `reward_modeling_adapter_id` argument, pointing to the id of the reward\n modeling adapter. The latest needs also to contain the score head in order to produce the reward.\n \"\"\"\n pretrained_model.load_adapter(adapter_model_id, adapter_name, is_trainable=False)\n pretrained_model.train()\n\n filename = os.path.join(adapter_model_id, \"adapter_model.bin\")\n safe_loading = False\n if not os.path.exists(filename):\n try:\n local_filename = hf_hub_download(\n adapter_model_id,\n \"adapter_model.bin\",\n token=token,\n )\n except Exception:\n filename = os.path.join(adapter_model_id, \"adapter_model.safetensors\")\n safe_loading = True\n if not os.path.exists(filename):\n try:\n local_filename = hf_hub_download(\n adapter_model_id,\n \"adapter_model.safetensors\",\n token=token,\n )\n except Exception as exc:\n raise ValueError(\n \"Could not find adapter model in the Hub, make sure you have the correct adapter model id.\"\n ) from exc\n else:\n local_filename = filename\n else:\n local_filename = filename\n\n loading_func = safe_load_file if safe_loading else torch.load\n load_kwargs = {} if safe_loading else {\"map_location\": \"cpu\", \"weights_only\": True}\n\n adapter_state_dict = loading_func(local_filename, **load_kwargs)\n\n for score_name_candidate in cls.supported_rm_modules:\n if any(score_name_candidate in name for name in adapter_state_dict.keys()):\n score_name = score_name_candidate\n # we have found the correct head name and can break\n break\n\n score_dict = {}\n\n for name, param in adapter_state_dict.items():\n if score_name in name:\n key_name = \".\".join(name.split(\".\")[-1:])\n score_dict[key_name] = param.to(cls._get_current_device())\n\n num_labels, hidden_dim = score_dict[\"weight\"].shape\n has_bias = any(\"bias\" in name for name in adapter_state_dict.keys())\n\n score = nn.Linear(hidden_dim, num_labels, bias=has_bias).to(\n device=cls._get_current_device(),\n dtype=pretrained_model.dtype,\n )\n score.load_state_dict(score_dict)\n for param in score.parameters():\n param.requires_grad = False\n\n return score\n\n def push_to_hub(self, *args, **kwargs):\n r\"\"\"\n Push the pretrained model to the hub. This method is a wrapper around\n [`~transformers.PreTrainedModel.push_to_hub`]. Please refer to the documentation of\n [`~transformers.PreTrainedModel.push_to_hub`] for more information.\n\n Args:\n *args (`list`, *optional*):\n Positional arguments passed along to the underlying model's `push_to_hub` method.\n **kwargs (`dict`, *optional*):\n Keyword arguments passed along to the underlying model's `push_to_hub` method.\n \"\"\"\n raise NotImplementedError\n\n def save_pretrained(self, *args, **kwargs):\n r\"\"\"\n Save the pretrained model to a directory. This method is a wrapper around\n [`~transformers.PreTrainedModel.save_pretrained`]. Please refer to the documentation of\n [`~transformers.PreTrainedModel.save_pretrained`] for more information.\n\n Args:\n *args (`list`, *optional*):\n Positional arguments passed along to the underlying model's `save_pretrained` method.\n **kwargs (`dict`, *optional*):\n Keyword arguments passed along to the underlying model's `save_pretrained` method.\n \"\"\"\n state_dict = kwargs.get(\"state_dict\")\n if state_dict is None:\n state_dict = self.state_dict()\n kwargs[\"state_dict\"] = state_dict\n\n # if it is a peft model only save the `v_head` state_dict and\n # pop the `state_dict` from the kwargs to avoid silent bugs with `peft`\n if self.is_peft_model:\n save_path = args[0]\n save_path = os.path.join(save_path, \"pytorch_model.bin\")\n torch.save(state_dict, save_path)\n _ = kwargs.pop(\"state_dict\", None)\n\n return self.pretrained_model.save_pretrained(*args, **kwargs)\n\n def state_dict(self, *args, **kwargs):\n r\"\"\"\n Return the state_dict of the pretrained model.\n \"\"\"\n raise NotImplementedError\n\n def post_init(self, *args, **kwargs):\n r\"\"\"\n Post initialization method. This method is called after the model is instantiated and loaded from a checkpoint.\n It can be used to perform additional operations such as loading the state_dict.\n \"\"\"\n raise NotImplementedError\n\n def compute_reward_score(self, input_ids, attention_mask=None, **kwargs):\n r\"\"\"\n Computes the reward score for a given input. The method has first to enable the adapter and then compute the\n reward score. After that the model disables the reward modeling adapter and enables the default ppo adapter\n again.\n \"\"\"\n if not self.supports_rm_adapter:\n raise ValueError(\"This model does not support reward modeling adapter.\")\n\n # enable rm adapter\n self.pretrained_model.set_adapter(self.rm_adapter_name)\n self.pretrained_model.eval()\n\n with torch.no_grad():\n base_model_output = self.pretrained_model(\n input_ids=input_ids,\n attention_mask=attention_mask,\n output_hidden_states=True,\n return_dict=True,\n **kwargs,\n )\n\n last_hidden_states = base_model_output.hidden_states[-1]\n scores = self.score(last_hidden_states)\n\n self.pretrained_model.set_adapter(self.policy_adapter_name)\n self.pretrained_model.eval()\n\n return scores\n\n\nclass ValueHead(nn.Module):\n r\"\"\"\n The ValueHead class implements a head for GPT2 that returns a scalar for each output token.\n \"\"\"\n\n def __init__(self, config, **kwargs):\n super().__init__()\n if not hasattr(config, \"summary_dropout_prob\"):\n summary_dropout_prob = kwargs.pop(\"summary_dropout_prob\", 0.1)\n else:\n summary_dropout_prob = config.summary_dropout_prob\n\n self.dropout = nn.Dropout(summary_dropout_prob) if summary_dropout_prob else nn.Identity()\n\n # some models such as OPT have a projection layer before the word embeddings - e.g. OPT-350m\n if hasattr(config, \"hidden_size\"):\n hidden_size = config.hidden_size\n if hasattr(config, \"word_embed_proj_dim\"):\n hidden_size = config.word_embed_proj_dim\n elif hasattr(config, \"is_encoder_decoder\"):\n if config.is_encoder_decoder and hasattr(config, \"decoder\"):\n if hasattr(config.decoder, \"hidden_size\"):\n hidden_size = config.decoder.hidden_size\n\n self.summary = nn.Linear(hidden_size, 1)\n\n self.flatten = nn.Flatten()\n\n def forward(self, hidden_states):\n output = self.dropout(hidden_states)\n\n # For now force upcast in fp32 if needed. Let's keep the\n # output in fp32 for numerical stability.\n if output.dtype != self.summary.weight.dtype:\n output = output.to(self.summary.weight.dtype)\n\n output = self.summary(output)\n return output\n\n\nclass AutoModelForCausalLMWithValueHead(PreTrainedModelWrapper):\n \"\"\"\n An autoregressive model with a value head in addition to the language model head. This class inherits from\n [`experimental.ppo.PreTrainedModelWrapper`] and wraps a [`~transformers.PreTrainedModel`] class. The wrapper class\n supports classic functions such as `from_pretrained`, `push_to_hub` and `generate`. To call a method of the wrapped\n model, simply manipulate the `pretrained_model` attribute of this class.\n\n Class attributes:\n - **transformers_parent_class** ([`~transformers.PreTrainedModel`]) -- The parent class of the wrapped model.\n This\n should be set to `transformers.AutoModelForCausalLM` for this class.\n - **supported_args** (`tuple`) -- A tuple of strings that are used to identify the arguments that are supported\n by the [`ValueHead`] class. Currently, the supported args are:\n - **summary_dropout_prob** (`float`, `optional`, defaults to `None`) -- The dropout probability for the\n [`ValueHead`] class.\n - **v_head_initializer_range** (`float`, `optional`, defaults to `0.2`) -- The initializer range for the\n [`ValueHead`] if a specific initialization strategy is selected.\n - **v_head_init_strategy** (`str`, `optional`, defaults to `None`) -- The initialization strategy for the\n [`ValueHead`]. Currently, the supported strategies are:\n - **`None`** -- Initializes the weights of the [`ValueHead`] with a random distribution. This is the\n default strategy.\n - **\"normal\"** -- Initializes the weights of the [`ValueHead`] with a normal distribution.\n \"\"\"\n\n transformers_parent_class = AutoModelForCausalLM\n supported_args = (\n \"summary_dropout_prob\",\n \"v_head_initializer_range\",\n \"v_head_init_strategy\",\n )\n\n def __init__(self, pretrained_model, **kwargs):\n \"\"\"\n Initializes the model.\n\n Args:\n pretrained_model ([`~transformers.PreTrainedModel`]):\n The model to wrap. It should be a causal language model such as GPT2. or any model mapped inside the\n `AutoModelForCausalLM` class.\n kwargs (`dict`, `optional`):\n Additional keyword arguments, that are passed to the [`ValueHead`] class.\n \"\"\"\n super().__init__(pretrained_model, **kwargs)\n v_head_kwargs, _, _ = self._split_kwargs(kwargs)\n self.v_head = ValueHead(self.pretrained_model.config, **v_head_kwargs)\n self._init_weights(**v_head_kwargs)\n\n def _init_weights(self, **kwargs):\n r\"\"\"\n Initializes the weights of the value head. The default initialization strategy is random. Users can pass a\n different initialization strategy by passing the `v_head_init_strategy` argument when calling\n `.from_pretrained`. Supported strategies are:\n - `normal`: initializes the weights with a normal distribution.\n\n Args:\n **kwargs (`dict`, `optional`):\n Additional keyword arguments, that are passed to the [`ValueHead`] class. These arguments can contain\n the `v_head_init_strategy` argument as well as the `v_head_initializer_range` argument.\n \"\"\"\n initializer_range = kwargs.pop(\"v_head_initializer_range\", 0.2)\n # random init by default\n init_strategy = kwargs.pop(\"v_head_init_strategy\", None)\n if init_strategy is None:\n # do nothing\n pass\n elif init_strategy == \"normal\":\n self.v_head.summary.weight.data.normal_(mean=0.0, std=initializer_range)\n self.v_head.summary.bias.data.zero_()\n\n def forward(\n self,\n input_ids=None,\n past_key_values=None,\n attention_mask=None,\n return_past_key_values=False,\n **kwargs,\n ):\n r\"\"\"\n Applies a forward pass to the wrapped model and returns the logits of the value head.\n\n Args:\n input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):\n Indices of input sequence tokens in the vocabulary.\n past_key_values (`tuple(tuple(torch.FloatTensor))`, `optional`):\n Contains pre-computed hidden-states (key and values in the attention blocks) as computed by the model\n (see `past_key_values` input) to speed up sequential decoding.\n attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, `optional`):\n Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``:\n - 1 for tokens that are **not masked**,\n - 0 for tokens that are **masked**.\n return_past_key_values (bool): A flag indicating if the computed hidden-states should be returned.\n kwargs (`dict`, `optional`):\n Additional keyword arguments, that are passed to the wrapped model.\n \"\"\"\n kwargs[\"output_hidden_states\"] = True # this had already been set in the LORA / PEFT examples\n kwargs[\"past_key_values\"] = past_key_values\n\n if self.is_peft_model and self.pretrained_model.active_peft_config.peft_type == \"PREFIX_TUNING\":\n kwargs.pop(\"past_key_values\")\n\n base_model_output = self.pretrained_model(\n input_ids=input_ids,\n attention_mask=attention_mask,\n **kwargs,\n )\n\n last_hidden_state = base_model_output.hidden_states[-1]\n lm_logits = base_model_output.logits\n loss = base_model_output.loss\n\n if last_hidden_state.device != self.v_head.summary.weight.device:\n last_hidden_state = last_hidden_state.to(self.v_head.summary.weight.device)\n\n value = self.v_head(last_hidden_state).squeeze(-1)\n\n # force upcast in fp32 if logits are in half-precision\n if lm_logits.dtype != torch.float32:\n lm_logits = lm_logits.float()\n\n if return_past_key_values:\n return (lm_logits, loss, value, base_model_output.past_key_values)\n else:\n return (lm_logits, loss, value)\n\n def generate(self, *args, **kwargs):\n r\"\"\"\n A simple wrapper around the `generate` method of the wrapped model. Please refer to the\n [`generate`](https://huggingface.co/docs/transformers/internal/generation_utils) method of the wrapped model\n for more information about the supported arguments.\n\n Args:\n *args (`list`, *optional*):\n Positional arguments passed to the `generate` method of the wrapped model.\n **kwargs (`dict`, *optional*):\n Keyword arguments passed to the `generate` method of the wrapped model.\n \"\"\"\n return self.pretrained_model.generate(*args, **kwargs)\n\n def state_dict(self, *args, **kwargs):\n r\"\"\"\n Returns the state dictionary of the model. We add the state dictionary of the value head to the state\n dictionary of the wrapped model by prepending the key with `v_head.`.\n \"\"\"\n if not self.is_peft_model:\n pretrained_model_state_dict = self.pretrained_model.state_dict(*args, **kwargs)\n else:\n # if it is a peft model, only save the v_head\n pretrained_model_state_dict = {}\n\n v_head_state_dict = self.v_head.state_dict(*args, **kwargs)\n for k, v in v_head_state_dict.items():\n pretrained_model_state_dict[f\"v_head.{k}\"] = v\n return pretrained_model_state_dict\n\n def push_to_hub(self, *args, **kwargs):\n self.pretrained_model.v_head = self.v_head\n\n return self.pretrained_model.push_to_hub(*args, **kwargs)\n\n def post_init(self, state_dict):\n r\"\"\"\n We add the state dictionary of the value head to the state dictionary of the wrapped model by prepending the\n key with `v_head.`. This function removes the `v_head.` prefix from the keys of the value head state\n dictionary.\n \"\"\"\n for k in list(state_dict.keys()):\n if \"v_head.\" in k:\n state_dict[k.replace(\"v_head.\", \"\")] = state_dict.pop(k)\n self.v_head.load_state_dict(state_dict, strict=False)\n del state_dict\n\n if hasattr(self.pretrained_model, \"hf_device_map\"):\n if (\n \"cpu\" in self.pretrained_model.hf_device_map.values()\n or \"disk\" in self.pretrained_model.hf_device_map.values()\n ):\n raise ValueError(\n \"The model is offloaded on CPU or disk - CPU & disk offloading is not supported for ValueHead models.\"\n )\n\n first_device = list(set(self.pretrained_model.hf_device_map.values()))[0]\n if isinstance(first_device, int):\n if is_torch_npu_available():\n first_device = f\"npu:{first_device}\"\n elif is_torch_xpu_available():\n first_device = f\"xpu:{first_device}\"\n else:\n first_device = f\"cuda:{first_device}\"\n self.v_head = self.v_head.to(first_device)\n\n def set_device_hook(module, input, outputs):\n new_output = ()\n for output in outputs:\n if isinstance(output, torch.Tensor):\n new_output += (output.to(first_device),)\n else:\n new_output += (output,)\n return new_output\n\n self.register_forward_hook(set_device_hook)\n\n self.is_sequential_parallel = True\n\n\nclass AutoModelForSeq2SeqLMWithValueHead(PreTrainedModelWrapper):\n \"\"\"\n A seq2seq model with a value head in addition to the language model head. This class inherits from\n [`experimental.ppo.PreTrainedModelWrapper`] and wraps a [`~transformers.PreTrainedModel`] class. The wrapper class\n supports classic functions such as `from_pretrained` and `push_to_hub` and also provides some additional\n functionalities such as `generate`.\n\n Args:\n pretrained_model ([`~transformers.PreTrainedModel`]):\n The model to wrap. It should be a causal language model such as GPT2. or any model mapped inside the\n [`~transformers.AutoModelForSeq2SeqLM`] class.\n kwargs:\n Additional keyword arguments passed along to the [`ValueHead`] class.\n \"\"\"\n\n transformers_parent_class = AutoModelForSeq2SeqLM\n lm_head_namings = [\"lm_head\", \"embed_out\", \"output_projection\"]\n supported_args = (\n \"summary_dropout_prob\",\n \"v_head_initializer_range\",\n \"v_head_init_strategy\",\n )\n\n def __init__(self, pretrained_model, **kwargs):\n super().__init__(pretrained_model, **kwargs)\n v_head_kwargs, _, _ = self._split_kwargs(kwargs)\n self.is_encoder_decoder = True\n\n if not self._has_lm_head():\n raise ValueError(\"The model does not have a language model head, please use a model that has one.\")\n\n self.v_head = ValueHead(self.pretrained_model.config, **v_head_kwargs)\n\n self._init_weights(**v_head_kwargs)\n\n def _has_lm_head(self):\n # check module names of all modules inside `pretrained_model` to find the language model head\n for name, _module in self.pretrained_model.named_modules():\n if any(attribute in name for attribute in self.lm_head_namings):\n return True\n return False\n\n def post_init(self, state_dict):\n r\"\"\"\n We add the state dictionary of the value head to the state dictionary of the wrapped model by prepending the\n key with `v_head.`. This function removes the `v_head.` prefix from the keys of the value head state\n dictionary.\n \"\"\"\n for k in list(state_dict.keys()):\n if \"v_head.\" in k:\n state_dict[k.replace(\"v_head.\", \"\")] = state_dict.pop(k)\n self.v_head.load_state_dict(state_dict, strict=False)\n del state_dict\n\n if hasattr(self.pretrained_model, \"hf_device_map\"):\n if (\n \"cpu\" in self.pretrained_model.hf_device_map.values()\n or \"disk\" in self.pretrained_model.hf_device_map.values()\n ):\n raise ValueError(\n \"The model is offloaded on CPU or disk - CPU & disk offloading is not supported for ValueHead models.\"\n )\n\n # get the lm_head device\n for name, module in self.pretrained_model.named_modules():\n if any(attribute in name for attribute in self.lm_head_namings):\n lm_head_device = module.weight.device\n break\n\n # put v_head on the same device as the lm_head to avoid issues\n self.v_head = self.v_head.to(lm_head_device)\n\n def set_device_hook(module, input, outputs):\n r\"\"\"\n A hook that sets the device of the output of the model to the device of the first parameter of the\n model.\n\n Args:\n module (`nn.Module`):\n The module to which the hook is attached.\n input (`tuple`):\n The input to the module.\n outputs (`tuple`):\n The output of the module.\n \"\"\"\n new_output = ()\n for output in outputs:\n if isinstance(output, torch.Tensor):\n new_output += (output.to(lm_head_device),)\n else:\n new_output += (output,)\n return new_output\n\n self.register_forward_hook(set_device_hook)\n self.is_sequential_parallel = True\n\n def state_dict(self, *args, **kwargs):\n r\"\"\"\n Returns the state dictionary of the model. We add the state dictionary of the value head to the state\n dictionary of the wrapped model by prepending the key with `v_head.`.\n \"\"\"\n if not self.is_peft_model:\n pretrained_model_state_dict = self.pretrained_model.state_dict(*args, **kwargs)\n else:\n # if it is a peft model, only save the v_head\n pretrained_model_state_dict = {}\n\n v_head_state_dict = self.v_head.state_dict(*args, **kwargs)\n for k, v in v_head_state_dict.items():\n pretrained_model_state_dict[f\"v_head.{k}\"] = v\n return pretrained_model_state_dict\n\n def push_to_hub(self, *args, **kwargs):\n self.pretrained_model.v_head = self.v_head\n\n return self.pretrained_model.push_to_hub(*args, **kwargs)\n\n def _init_weights(self, **kwargs):\n r\"\"\"\n We initialize the weights of the value head.\n \"\"\"\n initializer_range = kwargs.pop(\"v_head_initializer_range\", 0.2)\n # random init by default\n init_strategy = kwargs.pop(\"v_head_init_strategy\", None)\n if init_strategy is None:\n # do nothing\n pass\n elif init_strategy == \"normal\":\n self.v_head.summary.weight.data.normal_(mean=0.0, std=initializer_range)\n self.v_head.summary.bias.data.zero_()\n\n def forward(\n self,\n input_ids=None,\n past_key_values=None,\n attention_mask=None,\n return_past_key_values=False,\n **kwargs,\n ):\n kwargs[\"past_key_values\"] = past_key_values\n if self.is_peft_model and self.pretrained_model.active_peft_config.peft_type == \"PREFIX_TUNING\":\n kwargs.pop(\"past_key_values\")\n\n base_model_output = self.pretrained_model(\n input_ids=input_ids,\n attention_mask=attention_mask,\n output_hidden_states=True, # We force the model to output hidden states\n **kwargs,\n )\n\n last_hidden_state = base_model_output.decoder_hidden_states[-1]\n lm_logits = base_model_output.logits\n loss = base_model_output.loss\n\n value = self.v_head(last_hidden_state).squeeze(-1)\n\n # force upcast in fp32 if logits are in half-precision\n if lm_logits.dtype != torch.float32:\n lm_logits = lm_logits.float()\n\n if return_past_key_values:\n return (lm_logits, loss, value, base_model_output.past_key_values)\n else:\n return (lm_logits, loss, value)\n\n def generate(self, *args, **kwargs):\n r\"\"\"\n We call `generate` on the wrapped model.\n \"\"\"\n return self.pretrained_model.generate(*args, **kwargs)\n" }, { "path": "trl/experimental/ppo/ppo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Literal\n\nfrom ...trainer.base_config import _BaseConfig\n\n\n@dataclass\nclass PPOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`experimental.ppo.PPOTrainer`].\n\n This class includes only the parameters that are specific to PPO training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n dataset_num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n num_mini_batches (`int`, *optional*, defaults to `1`):\n Number of minibatches to split a batch into.\n total_episodes (`int`, *optional*):\n Total number of episodes in the dataset.\n local_rollout_forward_batch_size (`int`, *optional*, defaults to `64`):\n Per rank no grad forward pass in the rollout phase.\n num_sample_generations (`int`, *optional*, defaults to `10`):\n Number of debugging samples generations (i.e., `generate_completions` calls) throughout training.\n response_length (`int`, *optional*, defaults to `53`):\n Length of the response.\n stop_token (`str`, *optional*):\n Specifies the stop token to use for text generation. This parameter is mutually exclusive with\n `stop_token_id`.\n\n - `None`: No stop token is applied, unless `stop_token_id` is specified.\n - `'eos'`: Uses the tokenizer's `eos_token`.\n\n stop_token_id (`int`, *optional*):\n Specifies the ID of the stop token to use for text generation. If `None`, no stop token ID is applied,\n unless `stop_token` is specified. This parameter is mutually exclusive with `stop_token`.\n temperature (`float`, *optional*, defaults to `0.7`):\n Sampling temperature.\n missing_eos_penalty (`float`, *optional*):\n Penalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to\n generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive\n value.\n sft_model_path (`str`, *optional*, defaults to `\"EleutherAI/pythia-160m\"`):\n Path to the SFT model.\n world_size (`int`, *optional*):\n Number of processes (GPUs) to use for the training.\n num_total_batches (`int`, *optional*):\n Number of total batches to train.\n micro_batch_size (`int`, *optional*):\n Micro batch size across devices (HF's `per_device_train_batch_size` * `world_size`).\n local_batch_size (`int`, *optional*):\n Batch size per GPU (HF's `per_device_train_batch_size` * `gradient_accumulation_steps`).\n batch_size (`int`, *optional*):\n Batch size across devices (HF's `per_device_train_batch_size` * `world_size` *\n `gradient_accumulation_steps`).\n local_mini_batch_size (`int`, *optional*):\n Mini batch size per GPU.\n mini_batch_size (`int`, *optional*):\n Mini batch size across GPUs.\n push_to_hub (`bool`, *optional*, defaults to `False`):\n Whether to push the model to the Hub after training.\n reward_model_path (`str`, *optional*, defaults to `\"EleutherAI/pythia-160m\"`):\n Path to the reward model.\n model_adapter_name (`str`, *optional*):\n Name of the train target PEFT adapter, when using LoRA with multiple adapters.\n ref_adapter_name (`str`, *optional*):\n Name of the reference PEFT adapter, when using LoRA with multiple adapters.\n num_ppo_epochs (`int`, *optional*, defaults to `4`):\n Number of epochs to train.\n whiten_rewards (`bool`, *optional*, defaults to `False`):\n Whether to whiten the rewards.\n kl_coef (`float`, *optional*, defaults to `0.05`):\n KL coefficient.\n kl_estimator (`Literal[\"k1\", \"k3\"]`, *optional*, defaults to `\"k1\"`):\n Which estimator for KL-Divergence to use from [Approximating KL\n Divergence](http://joschu.net/blog/kl-approx.html). Defaults to \"k1\", a straightforward, unbiased\n estimator. Can be set to \"k3\", an unbiased estimator with lower variance which \"appears to be a strictly\n better estimator\". Cannot be set to \"k2\", as it is used for logging purposes.\n cliprange (`float`, *optional*, defaults to `0.2`):\n Clip range.\n vf_coef (`float`, *optional*, defaults to `0.1`):\n Value function coefficient.\n cliprange_value (`float`, *optional*, defaults to `0.2`):\n Clip range for the value function.\n gamma (`float`, *optional*, defaults to `1.0`):\n Discount factor.\n lam (`float`, *optional*, defaults to `0.95`):\n Lambda value for GAE.\n ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):\n This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,\n improving generation speed. However, disabling this option allows training models that exceed the VRAM\n capacity of a single GPU, albeit at the cost of slower generation.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `3e-6` instead of `5e-5`.\n \"\"\"\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=3e-6,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes to use for processing the dataset.\"},\n )\n num_mini_batches: int = field(\n default=1,\n metadata={\"help\": \"Number of minibatches to split a batch into.\"},\n )\n total_episodes: int | None = field(\n default=None,\n metadata={\"help\": \"Total number of episodes in the dataset.\"},\n )\n local_rollout_forward_batch_size: int = field(\n default=64,\n metadata={\"help\": \"Per rank no grad forward pass in the rollout phase.\"},\n )\n num_sample_generations: int = field(\n default=10,\n metadata={\n \"help\": \"Number of debugging samples generations (i.e., `generate_completions` calls) throughout training.\"\n },\n )\n response_length: int = field(\n default=53,\n metadata={\"help\": \"Length of the response.\"},\n )\n stop_token: Literal[\"eos\"] | None = field(\n default=None,\n metadata={\n \"help\": \"Specifies the stop token to use for text generation. This parameter is mutually exclusive with \"\n \"`stop_token_id`.\"\n },\n )\n stop_token_id: int | None = field(\n default=None,\n metadata={\n \"help\": \"Specifies the ID of the stop token to use for text generation. If `None`, no stop token ID is \"\n \"applied, unless `stop_token` is specified. This parameter is mutually exclusive with `stop_token`.\"\n },\n )\n temperature: float = field(\n default=0.7,\n metadata={\"help\": \"Sampling temperature.\"},\n )\n missing_eos_penalty: float | None = field(\n default=None,\n metadata={\n \"help\": \"Penalty applied to the score when the model fails to generate an EOS token. This is useful to \"\n \"encourage to generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be \"\n \"a positive value.\"\n },\n )\n sft_model_path: str = field(\n default=\"EleutherAI/pythia-160m\",\n metadata={\"help\": \"Path to the SFT model.\"},\n )\n world_size: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes (GPUs) to use for the training.\"},\n )\n num_total_batches: int | None = field(\n default=None,\n metadata={\"help\": \"Number of total batches to train.\"},\n )\n micro_batch_size: int | None = field(\n default=None,\n metadata={\"help\": \"Micro batch size across devices (HF's `per_device_train_batch_size` * `world_size`).\"},\n )\n local_batch_size: int | None = field(\n default=None,\n metadata={\"help\": \"Batch size per GPU (HF's `per_device_train_batch_size` * `gradient_accumulation_steps`).\"},\n )\n batch_size: int | None = field(\n default=None,\n metadata={\n \"help\": \"Batch size across devices (HF's `per_device_train_batch_size` * `world_size` * \"\n \"`gradient_accumulation_steps`).\"\n },\n )\n local_mini_batch_size: int | None = field(\n default=None,\n metadata={\"help\": \"Mini batch size per GPU.\"},\n )\n mini_batch_size: int | None = field(\n default=None,\n metadata={\"help\": \"Mini batch size across GPUs.\"},\n )\n push_to_hub: bool = field(\n default=False,\n metadata={\"help\": \"Whether to push the model to the Hub after training.\"},\n )\n reward_model_path: str = field(\n default=\"EleutherAI/pythia-160m\",\n metadata={\"help\": \"Path to the reward model.\"},\n )\n model_adapter_name: str | None = field(\n default=None,\n metadata={\"help\": \"Name of the train target PEFT adapter, when using LoRA with multiple adapters.\"},\n )\n ref_adapter_name: str | None = field(\n default=None,\n metadata={\"help\": \"Name of the reference PEFT adapter, when using LoRA with multiple adapters.\"},\n )\n num_ppo_epochs: int = field(\n default=4,\n metadata={\"help\": \"Number of epochs to train.\"},\n )\n whiten_rewards: bool = field(\n default=False,\n metadata={\"help\": \"Whether to whiten the rewards.\"},\n )\n kl_coef: float = field(\n default=0.05,\n metadata={\"help\": \"KL coefficient.\"},\n )\n kl_estimator: Literal[\"k1\", \"k3\"] = field(\n default=\"k1\",\n metadata={\n \"help\": \"Which estimator for KL-Divergence to use from Approximating KL Divergence \"\n \"(http://joschu.net/blog/kl-approx.html). Defaults to 'k1', a straightforward, unbiased estimator. Can be \"\n \"set to 'k3', an unbiased estimator with lower variance which 'appears to be a strictly better \"\n \"estimator'. Cannot be set to 'k2', as it is used for logging purposes.\"\n },\n )\n cliprange: float = field(\n default=0.2,\n metadata={\"help\": \"Clip range.\"},\n )\n vf_coef: float = field(\n default=0.1,\n metadata={\"help\": \"Value function coefficient.\"},\n )\n cliprange_value: float = field(\n default=0.2,\n metadata={\"help\": \"Clip range for the value function.\"},\n )\n gamma: float = field(\n default=1.0,\n metadata={\"help\": \"Discount factor.\"},\n )\n lam: float = field(\n default=0.95,\n metadata={\"help\": \"Lambda value for GAE.\"},\n )\n ds3_gather_for_generation: bool = field(\n default=True,\n metadata={\n \"help\": \"This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for \"\n \"generation, improving generation speed. However, disabling this option allows training models that \"\n \"exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation.\"\n },\n )\n" }, { "path": "trl/experimental/ppo/ppo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport gc\nimport math\nimport os\nimport textwrap\nimport time\nfrom collections import defaultdict\nfrom contextlib import contextmanager, nullcontext\nfrom dataclasses import dataclass\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport transformers\nfrom accelerate import Accelerator, logging\nfrom accelerate.utils import gather_object\nfrom datasets import Dataset\nfrom packaging.version import Version\nfrom torch.utils.data import DataLoader\nfrom transformers import (\n BaseImageProcessor,\n DataCollatorWithPadding,\n FeatureExtractionMixin,\n GenerationConfig,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n TrainerControl,\n TrainerState,\n)\nfrom transformers.integrations import get_reporting_integration_callbacks\nfrom transformers.trainer import DEFAULT_CALLBACKS, DEFAULT_PROGRESS_CALLBACK\nfrom transformers.trainer_callback import CallbackHandler, ExportableState, PrinterCallback\nfrom transformers.utils import ModelOutput, is_peft_available, is_rich_available\n\nfrom ...models.utils import unwrap_model_for_generation\nfrom ...trainer.base_trainer import _BaseTrainer\nfrom ...trainer.utils import (\n disable_dropout_in_model,\n log_table_to_comet_experiment,\n pad,\n prepare_deepspeed,\n selective_log_softmax,\n)\nfrom ..utils import (\n create_reference_model,\n empty_cache,\n first_true_indices,\n get_reward,\n peft_module_casting_to_bf16,\n)\nfrom .ppo_config import PPOConfig\n\n\nif is_rich_available():\n from rich.console import Console\n from rich.table import Table\n\n\nlogger = logging.get_logger(__name__)\n\nif is_peft_available():\n from peft import PeftConfig, PeftModel, get_peft_model\n\n\nINVALID_LOGPROB = 1.0\n\n\ndef generate(\n lm_backbone: torch.nn.Module, queries: torch.Tensor, pad_token_id: int, generation_config: GenerationConfig\n) -> tuple[torch.Tensor, torch.Tensor]:\n \"\"\"\n Generates sequences from the language model backbone in a way that does not affect padding tokens.\n\n Args:\n lm_backbone (`torch.nn.Module`):\n The language model backbone used for generation.\n queries (`torch.Tensor`):\n The tensor containing the input queries.\n pad_token_id (`int`):\n The token ID representing the pad token.\n generation_config ([`~transformers.GenerationConfig`]):\n The configuration for the generation process.\n\n Returns:\n tuple:\n - `generated_sequences` (`torch.Tensor`):\n The concatenated tensor of input queries and generated sequences.\n - `logits` (`torch.Tensor`):\n The logits output from the generation process.\n \"\"\"\n context_length = queries.shape[1]\n attention_mask = queries != pad_token_id\n input_ids = torch.masked_fill(queries, ~attention_mask, 0)\n output = lm_backbone.generate(\n input_ids=input_ids,\n attention_mask=attention_mask,\n # position_ids=attention_mask.cumsum(1) - attention_mask.long(), # not needed: already adjusted in generations\n # https://github.com/huggingface/transformers/blob/ac33aeeeee2a7a89b89c93c2962e6feb90daef0a/src/transformers/models/gpt2/modeling_gpt2.py#L1227-L1250\n generation_config=generation_config,\n return_dict_in_generate=True,\n output_scores=True,\n )\n logits = torch.stack(output.scores, 1)\n return torch.cat((queries, output.sequences[:, context_length:]), dim=1), logits\n\n\n@torch.no_grad()\ndef batch_generation(\n model: torch.nn.Module,\n queries: torch.Tensor,\n local_rollout_forward_batch_size: int,\n pad_token_id: int,\n generation_config: GenerationConfig,\n):\n query_responses = []\n logitss = []\n batch_size = queries.shape[0]\n for i in range(0, batch_size, local_rollout_forward_batch_size):\n query = queries[i : i + local_rollout_forward_batch_size]\n query_response, logits = generate(\n model,\n query,\n pad_token_id,\n generation_config,\n )\n query_responses.append(query_response)\n logitss.append(logits)\n\n # padding tensors\n padded_query_responses = pad(query_responses, padding_value=pad_token_id, padding_side=\"right\")\n padded_logitss = pad(logitss, padding_value=0, padding_side=\"right\")\n\n # reshaping\n padded_query_responses = padded_query_responses.view(-1, padded_query_responses.shape[-1])[:batch_size]\n padded_logitss = padded_logitss.view(-1, *padded_logitss.shape[2:])[:batch_size]\n\n return padded_query_responses, padded_logitss\n\n\ndef exact_div(a, b, custom_error_message=\"\"):\n q = a // b\n if a != q * b:\n raise ValueError(f\"{custom_error_message}, inexact division: {a} / {b} = {a / b}\")\n return q\n\n\ndef print_rich_table(df: pd.DataFrame) -> None:\n if not is_rich_available():\n raise ImportError(\n \"The function `print_rich_table` requires the `rich` library. Please install it with `pip install rich`.\"\n )\n console = Console()\n table = Table(show_lines=True)\n for column in df.columns:\n table.add_column(column)\n for _, row in df.iterrows():\n table.add_row(*row.astype(str).tolist())\n console.print(table)\n\n\ndef truncate_response(stop_token_id: int, pad_token_id: int, responses: torch.Tensor) -> torch.Tensor:\n \"\"\"\n Truncates the responses at the first occurrence of the stop token, filling the rest with pad tokens.\n\n Args:\n stop_token_id (`int`):\n The token ID representing the stop token where truncation occurs.\n pad_token_id (`int`):\n The token ID representing the pad token used to fill the truncated responses.\n responses (`torch.Tensor`):\n The tensor containing the responses to be truncated.\n\n Returns:\n `torch.Tensor`:\n The truncated responses tensor with pad tokens filled after the stop token.\n \"\"\"\n trunc_idxs = first_true_indices(responses == stop_token_id).unsqueeze(-1)\n new_size = [1] * (len(responses.size()) - 1) + [responses.shape[1]]\n idxs = torch.arange(responses.shape[1], device=responses.device).view(*new_size)\n postprocessed_responses = torch.masked_fill(responses, idxs > trunc_idxs, pad_token_id)\n return postprocessed_responses\n\n\ndef forward(\n model: torch.nn.Module,\n query_responses: torch.Tensor,\n pad_token_id: int,\n) -> ModelOutput:\n \"\"\"\n Performs a forward pass through the model with the given query responses and pad token ID.\n\n Args:\n model (`torch.nn.Module`):\n The model to perform the forward pass.\n query_responses (`torch.Tensor`):\n The tensor containing the query responses.\n pad_token_id (`int`):\n The token ID representing the pad token.\n\n Returns:\n `ModelOutput`:\n The output of the model, including hidden states.\n \"\"\"\n attention_mask = query_responses != pad_token_id\n position_ids = attention_mask.cumsum(1) - attention_mask.long()\n input_ids = torch.masked_fill(query_responses, ~attention_mask, 0)\n return model(\n input_ids=input_ids,\n attention_mask=attention_mask,\n position_ids=position_ids,\n return_dict=True,\n output_hidden_states=True,\n )\n\n\n@dataclass\nclass OnlineTrainerState(TrainerState):\n \"\"\"\n Training state for online/on-policy trainers.\n\n Extends [`~transformers.TrainerState`] with an `episode` counter to track the current rollout/episode.\n\n Args:\n episode (`int`, defaults to 0): Zero-based episode index.\n \"\"\"\n\n episode: int = 0\n\n\ndef masked_mean(values: torch.Tensor, mask: torch.Tensor, axis: bool | None = None) -> torch.Tensor:\n \"\"\"Compute mean of tensor with a masked values.\"\"\"\n if axis is not None:\n return (values * mask).sum(axis=axis) / mask.sum(axis=axis)\n else:\n return (values * mask).sum() / mask.sum()\n\n\ndef masked_var(values: torch.Tensor, mask: torch.Tensor, unbiased: bool = True) -> torch.Tensor:\n \"\"\"Compute variance of tensor with masked values.\"\"\"\n mean = masked_mean(values, mask)\n centered_values = values - mean\n variance = masked_mean(centered_values**2, mask)\n if unbiased:\n mask_sum = mask.sum()\n if mask_sum == 0:\n raise ValueError(\n \"The sum of the mask is zero, which can happen when `mini_batch_size=1`;\"\n \"try increase the `mini_batch_size` or `gradient_accumulation_steps`\"\n )\n # note that if mask_sum == 1, then there is a division by zero issue\n # to avoid it you just need to use a larger minibatch_size\n bessel_correction = mask_sum / (mask_sum - 1)\n variance = variance * bessel_correction\n return variance\n\n\ndef masked_whiten(values: torch.Tensor, mask: torch.Tensor, shift_mean: bool = True) -> torch.Tensor:\n \"\"\"Whiten values with masked values.\"\"\"\n mean, var = masked_mean(values, mask), masked_var(values, mask)\n whitened = (values - mean) * torch.rsqrt(var + 1e-8)\n if not shift_mean:\n whitened += mean\n return whitened\n\n\n# taken from https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f2b71b16919addede0341d2bef70825d/ppo/ppo_trainer.py#L29\n# we did this we can do a single `model = accelerator.prepare(model)`\nclass PolicyAndValueWrapper(nn.Module):\n def __init__(self, policy, value_model) -> None:\n super().__init__()\n self.policy = policy\n self.value_model = value_model\n self.critic_backbone = getattr(value_model, value_model.base_model_prefix)\n self.is_gradient_checkpointing = policy.is_gradient_checkpointing\n\n def gradient_checkpointing_enable(self, **kwargs):\n self.policy.gradient_checkpointing_enable(**kwargs)\n self.is_gradient_checkpointing = True\n\n def gradient_checkpointing_disable(self):\n self.policy.gradient_checkpointing_disable()\n self.is_gradient_checkpointing = False\n\n def forward(self, **kwargs):\n output = self.critic_backbone(**kwargs)\n logits = self.value_model.score(output.hidden_states[-1])\n return self.policy(**kwargs), logits\n\n\nclass PPOTrainer(_BaseTrainer):\n \"\"\"Trainer for Proximal Policy Optimization (PPO).\n\n For details on PPO, see the paper: [Proximal Policy Optimization\n Algorithms](https://huggingface.co/papers/1707.06347).\n\n Args:\n args ([`experimental.ppo.PPOConfig`]):\n Training arguments.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`]):\n Class to process the data.\n model (`torch.nn.Module`):\n Model to be trained. This is the policy model.\n ref_model (`torch.nn.Module`, *optional*):\n Reference model used to compute the KL divergence. If `None`, a copy of the policy model is created.\n reward_model (`torch.nn.Module`):\n Reward model used to compute the rewards.\n train_dataset ([`~datasets.Dataset`]):\n Dataset for training.\n value_model (`torch.nn.Module`):\n Value model used to predict the value of a state.\n data_collator ([`~transformers.DataCollatorWithPadding`], *optional*):\n Data collator to batch and pad samples from the dataset. If `None`, a default data collator is created\n using the `processing_class`.\n eval_dataset ([`~datasets.Dataset`] or `dict` of [`~datasets.Dataset`], *optional*):\n Dataset for evaluation.\n optimizers (`tuple` of `torch.optim.Optimizer` and `torch.optim.lr_scheduler.LambdaLR`, *optional*, defaults to `(None, None)`):\n Tuple containing the optimizer and the learning rate scheduler to use for training. If `None`, the\n optimizer and the learning rate scheduler are created using the\n [`~transformers.Trainer.create_optimizer_and_scheduler`] method.\n callbacks (`list` of [`~transformers.TrainerCallback`], *optional*):\n Callbacks to use during training.\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration to use PEFT for training. If `None`, PEFT is not used. If provided, the policy `model`\n will be wrapped with the specified PEFT adapter.\n \"\"\"\n\n _tag_names = [\"trl\", \"ppo\"]\n _name = \"PPO\"\n _paper = {\n \"title\": \"Fine-Tuning Language Models from Human Preferences\",\n \"id\": \"1909.08593\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{mziegler2019fine-tuning,\n title = {{Fine-Tuning Language Models from Human Preferences}},\n author = {Daniel M. Ziegler and Nisan Stiennon and Jeffrey Wu and Tom B. Brown and Alec Radford and Dario Amodei and Paul F. Christiano and Geoffrey Irving},\n year = 2019,\n eprint = {arXiv:1909.08593}\n }\"\"\"),\n }\n\n def __init__(\n self,\n args: PPOConfig,\n processing_class: PreTrainedTokenizerBase | BaseImageProcessor | FeatureExtractionMixin | ProcessorMixin,\n model: nn.Module,\n ref_model: nn.Module | None,\n reward_model: nn.Module,\n train_dataset: Dataset,\n value_model: nn.Module,\n data_collator: DataCollatorWithPadding | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n # less commonly used\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n callbacks: list[TrainerCallback] | None = None,\n peft_config: \"PeftConfig | None\" = None,\n ) -> None:\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n\n if ref_model is model:\n raise ValueError(\n \"`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the \"\n \"same as `model`, you must make a copy of it, or `None` if you use peft.\"\n )\n\n self.args = args\n self.processing_class = processing_class\n self.policy_model = model\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n # Define the collator if not provided\n if data_collator is None:\n data_collator = DataCollatorWithPadding(self.processing_class)\n\n # Handle stop token settings: update policy model's generation_config to use provided stop token\n if args.stop_token and args.stop_token_id:\n raise ValueError(\"You cannot set both `stop_token` and `stop_token_id`.\")\n elif args.stop_token:\n if args.stop_token == \"eos\":\n self.policy_model.generation_config.eos_token_id = self.stop_token_id = processing_class.eos_token_id\n else:\n raise ValueError(\n f\"Unknown `stop_token` {args.stop_token}. Allowed values are: `'eos'` and `None` (no stop token).\"\n )\n else:\n self.policy_model.generation_config.eos_token_id = self.stop_token_id = args.stop_token_id # None or int\n\n # Check that the kl estimator is valid\n if self.args.kl_estimator not in {\"k1\", \"k3\"}:\n raise ValueError(\n \"kl_estimator must be either 'k1' (straightforward, unbiased) or 'k3' (lower variance, unbiased, \"\n \"appears to be a strictly better estimator). See \"\n \"[Approximating KL Divergence](http://joschu.net/blog/kl-approx.html) for details.\"\n )\n\n # peft support\n if not is_peft_available() and peft_config is not None:\n raise ImportError(\n \"PEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT models\"\n )\n elif is_peft_available() and peft_config is not None:\n if isinstance(self.policy_model, PeftModel):\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first \"\n \"merge and unload the existing adapter, save the resulting base model, and then pass that base \"\n \"model along with the new `peft_config` to the trainer.\"\n )\n\n # get peft model with the given config\n self.policy_model = get_peft_model(self.policy_model, peft_config)\n if args.bf16 and getattr(self.policy_model, \"is_loaded_in_4bit\", False):\n peft_module_casting_to_bf16(self.policy_model)\n\n self.is_peft_model = is_peft_available() and isinstance(self.policy_model, PeftModel)\n self.model_adapter_name = args.model_adapter_name\n self.ref_adapter_name = args.ref_adapter_name\n\n if ref_model:\n self.ref_model = ref_model\n elif self.is_peft_model:\n self.ref_model = None\n else:\n self.ref_model = create_reference_model(self.policy_model)\n\n self.reward_model = reward_model\n self.train_dataset = train_dataset\n self.train_dataset_len = len(train_dataset)\n self.value_model = value_model\n self.data_collator = data_collator\n self.eval_dataset = eval_dataset\n self.optimizer, self.lr_scheduler = optimizers\n self.optimizer_cls_and_kwargs = None # needed for transformers >= 4.47\n\n #########\n # calculate various batch sizes\n #########\n if args.total_episodes is None: # allow the users to define episodes in terms of epochs.\n args.total_episodes = int(args.num_train_epochs * self.train_dataset_len)\n accelerator = Accelerator(gradient_accumulation_steps=args.gradient_accumulation_steps)\n self.accelerator = accelerator\n args.world_size = accelerator.num_processes\n args.local_batch_size = args.per_device_train_batch_size * args.gradient_accumulation_steps\n args.micro_batch_size = int(args.per_device_train_batch_size * args.world_size)\n args.batch_size = int(args.local_batch_size * args.world_size)\n args.mini_batch_size = exact_div(\n args.batch_size, args.num_mini_batches, \"`batch_size` must be a multiple of `num_mini_batches`\"\n )\n args.local_mini_batch_size = exact_div(\n args.local_batch_size, args.num_mini_batches, \"`local_batch_size` must be a multiple of `num_mini_batches`\"\n )\n if args.whiten_rewards:\n assert args.local_mini_batch_size >= 8, (\n f\"Per-rank minibatch size {args.local_mini_batch_size} is insufficient for whitening\"\n )\n # `per_rank_rollout_batch_size` is our `args.local_batch_size`\n # `per_rank_minibatch_size` is our `args.local_mini_batch_size`\n args.num_total_batches = math.ceil(\n args.total_episodes / args.batch_size\n ) # we may train for more than `total_episodes`\n self.local_seed = args.seed + accelerator.process_index * 100003 # Prime\n if args.num_sample_generations > 0:\n self.sample_generations_freq = max(1, args.num_total_batches // args.num_sample_generations)\n self.local_dataloader_batch_size = args.local_batch_size\n\n #########\n # setup model, optimizer, and others\n #########\n for module in [self.policy_model, self.ref_model, self.value_model, self.reward_model]:\n if module is not None:\n disable_dropout_in_model(module)\n self.model = PolicyAndValueWrapper(self.policy_model, self.value_model)\n self.model.config = self.policy_model.config # needed for pushing to hub\n self.create_optimizer_and_scheduler(\n num_training_steps=args.num_total_batches\n ) # note that we are calling `self.lr_scheduler.step()` manually only at the batch level\n\n #########\n # trainer specifics\n #########\n default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to)\n self.callbacks = default_callbacks if callbacks is None else default_callbacks + callbacks\n self.callback_handler = CallbackHandler(\n self.callbacks, self.model, self.processing_class, self.optimizer, self.lr_scheduler\n )\n self.add_callback(PrinterCallback if self.args.disable_tqdm else DEFAULT_PROGRESS_CALLBACK)\n self.control = TrainerControl()\n self.state = OnlineTrainerState(\n is_local_process_zero=self.is_local_process_zero(),\n is_world_process_zero=self.is_world_process_zero(),\n stateful_callbacks=[\n cb for cb in self.callback_handler.callbacks + [self.control] if isinstance(cb, ExportableState)\n ],\n )\n self.current_flos = 0\n self.hp_search_backend = None\n self.is_deepspeed_enabled = getattr(self.accelerator.state, \"deepspeed_plugin\", None) is not None\n self.is_fsdp_enabled = getattr(self.accelerator.state, \"fsdp_plugin\", None) is not None\n # Create distant repo and output directory if needed\n self.hub_model_id = None\n if self.args.push_to_hub:\n self.init_hf_repo()\n if self.args.should_save:\n os.makedirs(self.args.output_dir, exist_ok=True)\n\n # Add tags for models that have been loaded with the correct transformers version\n if hasattr(self.model, \"add_model_tags\"):\n self.model.add_model_tags(self._tag_names)\n\n #########\n # setup dataloader\n #########\n self.dataloader = DataLoader(\n self.train_dataset,\n batch_size=self.local_dataloader_batch_size,\n shuffle=True,\n collate_fn=self.data_collator,\n drop_last=True, # needed; otherwise the last batch will be of ragged shape\n )\n # sync random states for DataLoader(shuffle=True) before `accelerator.prepare`\n # see https://gist.github.com/vwxyzjn/2581bff1e48e185e0b85b6dfe1def79c\n torch.manual_seed(args.seed)\n self.model, self.optimizer, self.dataloader = accelerator.prepare(self.model, self.optimizer, self.dataloader)\n torch.manual_seed(self.local_seed) # reset the local seed again\n\n self.eval_dataloader = DataLoader(\n self.eval_dataset,\n batch_size=args.per_device_eval_batch_size,\n collate_fn=self.data_collator,\n drop_last=True,\n ) # no need to shuffle eval dataset\n self.eval_dataloader = accelerator.prepare(self.eval_dataloader)\n\n if self.is_deepspeed_enabled:\n self.reward_model = prepare_deepspeed(\n self.reward_model, args.per_device_train_batch_size, args.fp16, args.bf16\n )\n\n if self.ref_model is None:\n if not self.is_peft_model:\n raise ValueError(\"No reference model and model is not a Peft model.\")\n else:\n self.ref_model = prepare_deepspeed(\n self.ref_model, args.per_device_train_batch_size, args.fp16, args.bf16\n )\n else:\n if self.ref_model is None:\n if not self.is_peft_model:\n raise ValueError(\"No reference model and model is not a Peft model.\")\n else:\n self.ref_model = self.ref_model.to(self.accelerator.device)\n self.reward_model = self.reward_model.to(self.accelerator.device)\n\n def get_train_dataloader(self) -> DataLoader:\n return self.dataloader\n\n def get_eval_dataloader(self) -> DataLoader:\n return self.eval_dataloader\n\n @contextmanager\n def null_ref_context(self):\n \"\"\"Context manager for handling null reference model (that is, peft adapter manipulation).\"\"\"\n with (\n self.accelerator.unwrap_model(self.model.policy).disable_adapter()\n if self.is_peft_model and not self.ref_adapter_name\n else nullcontext()\n ):\n if self.ref_adapter_name:\n self.model.policy.set_adapter(self.ref_adapter_name)\n yield\n if self.ref_adapter_name:\n self.model.policy.set_adapter(self.model_adapter_name or \"default\")\n\n def save_model(self, output_dir: str | None = None, _internal_call: bool = False):\n backup_model = self.model\n if hasattr(self.model, \"policy\"):\n self.model = self.model.policy # save only the policy for inference\n if self.is_deepspeed_enabled:\n backup_deepspeed = self.deepspeed\n self.deepspeed = self.model\n\n super().save_model(output_dir, _internal_call)\n\n self.model = backup_model\n if self.is_deepspeed_enabled:\n self.deepspeed = backup_deepspeed\n\n def train(self):\n args = self.args\n accelerator = self.accelerator\n optimizer = self.optimizer\n model = self.model\n ref_policy = self.ref_model\n reward_model = self.reward_model\n processing_class = self.processing_class\n dataloader = self.dataloader\n device = accelerator.device\n\n def repeat_generator():\n while True:\n yield from dataloader\n\n iter_dataloader = iter(repeat_generator())\n generation_kwargs = {\n \"max_new_tokens\": args.response_length,\n \"temperature\": (args.temperature + 1e-7),\n \"top_k\": 0.0,\n \"top_p\": 1.0,\n \"do_sample\": True,\n }\n generation_config = GenerationConfig(**generation_kwargs)\n\n accelerator.print(\"===training policy===\")\n start_time = time.time()\n stats_shape = (args.num_ppo_epochs, args.num_mini_batches, args.gradient_accumulation_steps)\n approxkl_stats = torch.zeros(stats_shape, device=device)\n pg_clipfrac_stats = torch.zeros(stats_shape, device=device)\n pg_loss_stats = torch.zeros(stats_shape, device=device)\n vf_loss_stats = torch.zeros(stats_shape, device=device)\n vf_clipfrac_stats = torch.zeros(stats_shape, device=device)\n entropy_stats = torch.zeros(stats_shape, device=device)\n ratio_stats = torch.zeros(stats_shape, device=device)\n model.train()\n\n # trainer state initialization\n self.state.global_step = 0\n self.state.episode = 0\n self.state.max_steps = args.num_total_batches\n self.state.num_train_epochs = args.total_episodes / self.train_dataset_len\n # Compute absolute values for logging, eval, and save if given as ratio\n if args.logging_steps is not None:\n if args.logging_steps < 1:\n self.state.logging_steps = math.ceil(self.state.max_steps * args.logging_steps)\n else:\n self.state.logging_steps = args.logging_steps\n if args.eval_steps is not None:\n if args.eval_steps < 1:\n self.state.eval_steps = math.ceil(self.state.max_steps * args.eval_steps)\n else:\n self.state.eval_steps = args.eval_steps\n if args.save_steps is not None:\n if args.save_steps < 1:\n self.state.save_steps = math.ceil(self.state.max_steps * args.save_steps)\n else:\n self.state.save_steps = args.save_steps\n self.control = self.callback_handler.on_train_begin(args, self.state, self.control)\n\n # backward compatibility\n if self.is_deepspeed_enabled:\n self.deepspeed = self.model\n self.model_wrapped = self.model\n\n for update in range(1, args.num_total_batches + 1):\n self.state.episode += 1 * args.batch_size\n data = next(iter_dataloader)\n with torch.no_grad():\n queries = data[\"input_ids\"].to(device)\n context_length = queries.shape[1]\n responses = []\n postprocessed_responses = []\n logprobs = []\n ref_logprobs = []\n scores = []\n sequence_lengths = []\n values = []\n with (\n unwrap_model_for_generation(\n self.model,\n self.accelerator,\n gather_deepspeed3_params=self.args.ds3_gather_for_generation,\n generation_kwargs=generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as unwrapped_model\n ):\n query_responses, logitss = batch_generation(\n unwrapped_model.policy,\n queries,\n args.local_rollout_forward_batch_size,\n processing_class.pad_token_id,\n generation_config,\n )\n\n for i in range(0, queries.shape[0], args.local_rollout_forward_batch_size):\n query = queries[i : i + args.local_rollout_forward_batch_size]\n query_response = query_responses[i : i + args.local_rollout_forward_batch_size]\n response = query_response[:, context_length:]\n logits = logitss[i : i + args.local_rollout_forward_batch_size]\n logprob = selective_log_softmax(logits, response)\n del logits\n empty_cache()\n\n if ref_policy is None:\n with self.null_ref_context():\n ref_output = forward(model.policy, query_response, processing_class.pad_token_id)\n else:\n ref_output = forward(ref_policy, query_response, processing_class.pad_token_id)\n ref_logits = ref_output.logits[:, context_length - 1 : -1]\n ref_logits /= args.temperature + 1e-7\n ref_logprob = selective_log_softmax(ref_logits, response)\n del ref_output, ref_logits\n empty_cache()\n\n # Response Processing 1. truncate response after the first occurrence of `stop_token_id`\n postprocessed_response = response\n if self.stop_token_id is not None: # handle the edge case when stop_token_id exists but is 0\n postprocessed_response = truncate_response(\n self.stop_token_id, processing_class.pad_token_id, response\n )\n\n # Response Processing 2. run reward model on the truncated responses\n postprocessed_query_response = torch.cat((query, postprocessed_response), 1)\n sequence_length = first_true_indices(postprocessed_response == processing_class.pad_token_id) - 1\n unwrapped_value_model = accelerator.unwrap_model(model).value_model\n full_value, _, _ = get_reward(\n unwrapped_value_model, query_response, processing_class.pad_token_id, context_length\n )\n value = full_value[:, context_length - 1 : -1].squeeze(-1)\n _, score, _ = get_reward(\n reward_model, postprocessed_query_response, processing_class.pad_token_id, context_length\n )\n\n responses.append(response)\n postprocessed_responses.append(postprocessed_response)\n logprobs.append(logprob)\n ref_logprobs.append(ref_logprob)\n sequence_lengths.append(sequence_length)\n scores.append(score)\n values.append(value)\n responses = torch.cat(responses, 0)\n postprocessed_responses = torch.cat(postprocessed_responses, 0)\n logprobs = torch.cat(logprobs, 0)\n ref_logprobs = torch.cat(ref_logprobs, 0)\n sequence_lengths = torch.cat(sequence_lengths, 0)\n scores = torch.cat(scores, 0)\n values = torch.cat(values, 0)\n del (logprob, ref_logprob, full_value, value, score, unwrapped_model)\n empty_cache()\n gc.collect()\n\n # Response Processing 3. Filter completion. Ensure that the sample contains stop_token_id\n # Completions not passing that filter will receive a lower score.\n contain_eos_token = torch.any(postprocessed_responses == self.processing_class.eos_token_id, dim=-1)\n if self.args.missing_eos_penalty is not None:\n scores[~contain_eos_token] -= self.args.missing_eos_penalty\n # accelerator.print(f\"{scores=}, {(contain_eos_token.sum() / len(contain_eos_token))=}\")\n\n # be very careful with `padding_mask_p1`; see https://excalidraw.com/#json=LWnzG4w2k5DjF_EOL_xPt,e2w3a-hFJ_gX5vOfeyXGTw\n response_idxs = torch.arange(responses.shape[1], device=responses.device).repeat(responses.shape[0], 1)\n padding_mask = response_idxs > sequence_lengths.unsqueeze(1)\n logprobs = torch.masked_fill(logprobs, padding_mask, INVALID_LOGPROB)\n ref_logprobs = torch.masked_fill(ref_logprobs, padding_mask, INVALID_LOGPROB)\n sequence_lengths_p1 = sequence_lengths + 1\n padding_mask_p1 = response_idxs > (sequence_lengths_p1.unsqueeze(1))\n values = torch.masked_fill(values, padding_mask_p1, 0)\n\n # 4. compute rewards\n # Formula used by http://joschu.net/blog/kl-approx.html for the k1 and k3 estimators\n logr = ref_logprobs - logprobs\n kl = -logr if args.kl_estimator == \"k1\" else (logr.exp() - 1) - logr # Else statement is k3\n non_score_reward = -args.kl_coef * kl\n rewards = non_score_reward.clone()\n actual_start = torch.arange(rewards.size(0), device=rewards.device)\n actual_end = torch.where(sequence_lengths_p1 < rewards.size(1), sequence_lengths_p1, sequence_lengths)\n rewards[actual_start, actual_end] += scores\n\n # 5. whiten rewards\n if args.whiten_rewards:\n rewards = masked_whiten(rewards, mask=~padding_mask_p1, shift_mean=False)\n rewards = torch.masked_fill(rewards, padding_mask_p1, 0)\n\n # 6. compute advantages and returns\n lastgaelam = 0\n advantages_reversed = []\n gen_length = responses.shape[1]\n for t in reversed(range(gen_length)):\n nextvalues = values[:, t + 1] if t < gen_length - 1 else 0.0\n delta = rewards[:, t] + args.gamma * nextvalues - values[:, t]\n lastgaelam = delta + args.gamma * args.lam * lastgaelam\n advantages_reversed.append(lastgaelam)\n advantages = torch.stack(advantages_reversed[::-1], axis=1)\n returns = advantages + values\n advantages = masked_whiten(advantages, ~padding_mask)\n advantages = torch.masked_fill(advantages, padding_mask, 0)\n empty_cache()\n\n # Do multiple epochs of PPO training, with a fresh random shuffle in each epoch\n for ppo_epoch_idx in range(args.num_ppo_epochs):\n b_inds = np.random.permutation(args.local_batch_size)\n minibatch_idx = 0\n for mini_batch_start in range(0, args.local_batch_size, args.local_mini_batch_size):\n mini_batch_end = mini_batch_start + args.local_mini_batch_size\n mini_batch_inds = b_inds[mini_batch_start:mini_batch_end]\n gradient_accumulation_idx = 0\n for micro_batch_start in range(0, args.local_mini_batch_size, args.per_device_train_batch_size):\n with accelerator.accumulate(model):\n micro_batch_end = micro_batch_start + args.per_device_train_batch_size\n micro_batch_inds = mini_batch_inds[micro_batch_start:micro_batch_end]\n mb_advantage = advantages[micro_batch_inds]\n mb_responses = responses[micro_batch_inds]\n mb_query_responses = query_responses[micro_batch_inds]\n mb_logprobs = logprobs[micro_batch_inds]\n mb_return = returns[micro_batch_inds]\n mb_values = values[micro_batch_inds]\n\n output, vpred_temp = forward(model, mb_query_responses, processing_class.pad_token_id)\n logits = output.logits[:, context_length - 1 : -1]\n logits /= args.temperature + 1e-7\n new_logprobs = selective_log_softmax(logits, mb_responses)\n new_logprobs = torch.masked_fill(\n new_logprobs, padding_mask[micro_batch_inds], INVALID_LOGPROB\n )\n vpred = vpred_temp[:, context_length - 1 : -1].squeeze(-1)\n vpred = torch.masked_fill(vpred, padding_mask_p1[micro_batch_inds], 0)\n vpredclipped = torch.clamp(\n vpred,\n mb_values - args.cliprange_value,\n mb_values + args.cliprange_value,\n )\n vf_losses1 = torch.square(vpred - mb_return)\n vf_losses2 = torch.square(vpredclipped - mb_return)\n vf_loss_max = torch.max(vf_losses1, vf_losses2)\n vf_loss = 0.5 * masked_mean(vf_loss_max, ~padding_mask_p1[micro_batch_inds])\n vf_clipfrac = masked_mean(\n (vf_losses2 > vf_losses1).float(), ~padding_mask_p1[micro_batch_inds]\n )\n logprobs_diff = new_logprobs - mb_logprobs\n ratio = torch.exp(logprobs_diff)\n pg_losses = -mb_advantage * ratio\n pg_losses2 = -mb_advantage * torch.clamp(ratio, 1.0 - args.cliprange, 1.0 + args.cliprange)\n pg_loss_max = torch.max(pg_losses, pg_losses2)\n pg_loss = masked_mean(pg_loss_max, ~padding_mask[micro_batch_inds])\n loss = pg_loss + args.vf_coef * vf_loss\n accelerator.backward(loss)\n optimizer.step()\n optimizer.zero_grad()\n with torch.no_grad():\n pg_clipfrac = masked_mean(\n (pg_losses2 > pg_losses).float(), ~padding_mask[micro_batch_inds]\n )\n prob_dist = torch.nn.functional.softmax(logits, dim=-1)\n entropy = torch.logsumexp(logits, dim=-1) - torch.sum(prob_dist * logits, dim=-1)\n approxkl = 0.5 * (logprobs_diff**2).mean()\n approxkl_stats[ppo_epoch_idx, minibatch_idx, gradient_accumulation_idx] = approxkl\n pg_clipfrac_stats[ppo_epoch_idx, minibatch_idx, gradient_accumulation_idx] = (\n pg_clipfrac\n )\n pg_loss_stats[ppo_epoch_idx, minibatch_idx, gradient_accumulation_idx] = pg_loss\n vf_loss_stats[ppo_epoch_idx, minibatch_idx, gradient_accumulation_idx] = vf_loss\n vf_clipfrac_stats[ppo_epoch_idx, minibatch_idx, gradient_accumulation_idx] = (\n vf_clipfrac\n )\n entropy_stats[ppo_epoch_idx, minibatch_idx, gradient_accumulation_idx] = entropy.mean()\n ratio_stats[ppo_epoch_idx, minibatch_idx, gradient_accumulation_idx] = ratio.mean()\n gradient_accumulation_idx += 1\n minibatch_idx += 1\n # del everything and empty cache\n # fmt: off\n del (\n output, vpred_temp, logits, new_logprobs, vpred, vpredclipped,\n vf_losses1, vf_losses2, vf_loss, vf_clipfrac, logprobs_diff, ratio, pg_losses, pg_losses2, pg_loss_max,\n pg_loss, loss, pg_clipfrac, prob_dist, entropy, approxkl, mb_return,\n mb_advantage, mb_values, mb_responses, mb_query_responses, mb_logprobs,\n )\n # fmt: on\n empty_cache()\n with torch.no_grad():\n mean_kl = kl.sum(1).mean()\n mean_entropy = (-logprobs).sum(1).mean()\n mean_non_score_reward = non_score_reward.sum(1).mean()\n rlhf_reward = mean_non_score_reward + scores.mean()\n eps = int(self.state.episode / (time.time() - start_time))\n metrics = {}\n metrics[\"eps\"] = eps\n metrics[\"objective/kl\"] = self.accelerator.gather_for_metrics(mean_kl).mean().item()\n metrics[\"objective/entropy\"] = self.accelerator.gather_for_metrics(mean_entropy).mean().item()\n metrics[\"objective/non_score_reward\"] = (\n self.accelerator.gather_for_metrics(mean_non_score_reward).mean().item()\n )\n metrics[\"objective/rlhf_reward\"] = self.accelerator.gather_for_metrics(rlhf_reward).mean().item()\n metrics[\"objective/scores\"] = self.accelerator.gather_for_metrics(scores.mean()).mean().item()\n metrics[\"policy/approxkl_avg\"] = self.accelerator.gather_for_metrics(approxkl_stats).mean().item()\n metrics[\"policy/clipfrac_avg\"] = self.accelerator.gather_for_metrics(pg_clipfrac_stats).mean().item()\n metrics[\"loss/policy_avg\"] = self.accelerator.gather_for_metrics(pg_loss_stats).mean().item()\n metrics[\"loss/value_avg\"] = self.accelerator.gather_for_metrics(vf_loss_stats).mean().item()\n metrics[\"val/clipfrac_avg\"] = self.accelerator.gather_for_metrics(vf_clipfrac_stats).mean().item()\n metrics[\"policy/entropy_avg\"] = self.accelerator.gather_for_metrics(entropy_stats).mean().item()\n metrics[\"val/ratio\"] = self.accelerator.gather_for_metrics(ratio_stats).mean().item()\n metrics[\"val/ratio_var\"] = self.accelerator.gather_for_metrics(ratio_stats).var().item()\n metrics[\"val/num_eos_tokens\"] = (responses == processing_class.eos_token_id).sum().item()\n metrics[\"lr\"] = self.lr_scheduler.get_last_lr()[0]\n metrics[\"episode\"] = self.state.episode\n self.state.epoch = self.state.episode / self.train_dataset_len # used by self.log\n self.state.global_step += 1\n self.log(metrics)\n\n self.lr_scheduler.step()\n self.control = self.callback_handler.on_step_end(args, self.state, self.control)\n if self.control.should_save:\n self._save_checkpoint(model, trial=None)\n self.control = self.callback_handler.on_save(self.args, self.state, self.control)\n del kl, mean_kl, mean_entropy, mean_non_score_reward, scores, metrics, non_score_reward\n empty_cache()\n gc.collect()\n\n if args.num_sample_generations > 0 and (update - 1) % self.sample_generations_freq == 0:\n self.generate_completions(sampling=True)\n empty_cache()\n del (\n query_responses,\n responses,\n postprocessed_responses,\n logprobs,\n ref_logprobs,\n values,\n sequence_lengths,\n contain_eos_token,\n sequence_lengths_p1,\n response_idxs,\n padding_mask,\n padding_mask_p1,\n rewards,\n actual_start,\n actual_end,\n advantages,\n returns,\n )\n empty_cache()\n\n # HF trainer specifics\n self.control = self.callback_handler.on_train_end(args, self.state, self.control)\n if self.control.should_save:\n self._save_checkpoint(model, trial=None)\n self.control = self.callback_handler.on_save(self.args, self.state, self.control)\n\n def generate_completions(self, sampling: bool = False):\n if self.eval_dataset is None:\n return # no eval set to sample from (pass eval_dataset and eval_strategy != \"no\" for sample generations)\n args = self.args\n processing_class = self.processing_class\n generation_kwargs = {\n \"max_new_tokens\": args.response_length,\n \"temperature\": (0.01 + 1e-7),\n \"top_k\": 0.0,\n \"top_p\": 1.0,\n \"do_sample\": True,\n }\n generation_config = GenerationConfig(**generation_kwargs)\n\n table = defaultdict(list)\n with (\n unwrap_model_for_generation(\n self.model,\n self.accelerator,\n gather_deepspeed3_params=self.args.ds3_gather_for_generation,\n generation_kwargs=generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as unwrapped_model\n ):\n for batch in self.eval_dataloader:\n query = batch[\"input_ids\"]\n with torch.no_grad():\n context_length = query.shape[1]\n query_response, _ = batch_generation(\n unwrapped_model.policy,\n query,\n query.shape[0],\n processing_class.pad_token_id,\n generation_config,\n )\n response = query_response[:, context_length:]\n postprocessed_response = response\n if self.stop_token_id is not None: # handle the edge case when stop_token_id exists but is 0\n postprocessed_response = truncate_response(\n self.stop_token_id, processing_class.pad_token_id, response\n )\n table[\"query\"].extend(\n gather_object(processing_class.batch_decode(query, skip_special_tokens=True))\n )\n table[\"model response\"].extend(\n gather_object(processing_class.batch_decode(postprocessed_response))\n )\n\n postprocessed_query_response = torch.cat((query, postprocessed_response), 1)\n _, score, _ = get_reward(\n self.reward_model, postprocessed_query_response, processing_class.pad_token_id, context_length\n )\n table[\"score\"].extend(self.accelerator.gather_for_metrics(score).float().cpu().numpy())\n\n if sampling:\n break\n df = pd.DataFrame(table)\n\n if self.accelerator.is_main_process:\n if is_rich_available():\n print_rich_table(df.iloc[0 : 0 + 5])\n if \"wandb\" in args.report_to:\n import wandb\n\n if wandb.run is not None:\n wandb.log({\"completions\": wandb.Table(dataframe=df)})\n\n if \"comet_ml\" in args.report_to:\n log_table_to_comet_experiment(\n name=\"completions.csv\",\n table=df,\n )\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/experimental/prm/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .prm_config import PRMConfig\nfrom .prm_trainer import PRMTrainer\n\n\n__all__ = [\"PRMConfig\", \"PRMTrainer\"]\n" }, { "path": "trl/experimental/prm/prm_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom ...trainer.base_config import _BaseConfig\n\n\n@dataclass\nclass PRMConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`experimental.prm.PRMTrainer`].\n\n This class includes only the parameters that are specific to PRM training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n max_length (`int` or `None`, *optional*, defaults to `1024`):\n Maximum length of the sequences (prompt + completion) used for truncation.\n max_completion_length (`int`, *optional*):\n Maximum length of the completion used for truncation. The completion is the concatenation of the steps.\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model.\n step_separator (`str`, *optional*, defaults to `\"\\n\"`):\n Separator used to separate each step of the reasoning process.\n train_on_last_step_only (`bool`, *optional*, defaults to `False`):\n Whether to train only on the last step.\n dataset_num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `1e-5` instead of `5e-5`.\n \"\"\"\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-5,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n max_length: int | None = field(\n default=1024,\n metadata={\"help\": \"Maximum length of the sequences (prompt + completion) used for truncation.\"},\n )\n max_completion_length: int | None = field(\n default=None,\n metadata={\n \"help\": \"Maximum length of the completion used for truncation. The completion is the concatenation of the \"\n \"steps.\"\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropout in the model and reference model.\"},\n )\n step_separator: str = field(\n default=\"\\n\",\n metadata={\"help\": \"Separator used to separate each step of the reasoning process.\"},\n )\n train_on_last_step_only: bool = field(\n default=False,\n metadata={\"help\": \"Whether to train only on the last step.\"},\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes to use for processing the dataset.\"},\n )\n" }, { "path": "trl/experimental/prm/prm_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport textwrap\nfrom collections.abc import Callable\nfrom itertools import chain\nfrom pathlib import Path\n\nimport numpy as np\nimport torch\nimport torch.nn as nn\nimport transformers\nfrom accelerate import PartialState, logging\nfrom datasets import Dataset, features\nfrom packaging.version import Version\nfrom transformers import (\n BaseImageProcessor,\n DataCollator,\n DataCollatorForTokenClassification,\n FeatureExtractionMixin,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n)\nfrom transformers.trainer_utils import EvalPrediction\nfrom transformers.utils import is_peft_available\n\nfrom ...trainer.base_trainer import _BaseTrainer\nfrom ...trainer.utils import disable_dropout_in_model\nfrom ..utils import prepare_peft_model\nfrom .prm_config import PRMConfig\n\n\nif is_peft_available():\n from peft import PeftModel\n\nlogger = logging.get_logger(__name__)\n\n\ndef compute_accuracy(eval_pred: EvalPrediction) -> dict[str, float]:\n predictions, labels = eval_pred\n if predictions.ndim == 3:\n # Token classification task. Shapes are (batch_size, seq_len, num_labels) and (batch_size, seq_len)\n # Used to compute the accuracy in the prm_trainer.\n predictions = np.argmax(predictions, axis=2)\n\n # Flatten the predictions and labels to remove the ignored tokens.\n predictions = np.array(\n [\n p\n for prediction, label in zip(predictions, labels, strict=True)\n for (p, lbl) in zip(prediction, label, strict=True)\n if lbl != -100\n ]\n )\n labels = np.array([lbl for label in labels for lbl in label if lbl != -100])\n\n else:\n # Here, predictions is rewards_chosen and rewards_rejected. Shapes are (batch_size, 2) and (batch_size,)\n # We want to see how much of the time rewards_chosen > rewards_rejected.\n equal_mask = predictions[:, 0] == predictions[:, 1]\n equal_predictions_count = int(equal_mask.sum())\n\n if equal_predictions_count > 0:\n # Before using the logger, the accelerate state must be initialized. It'susually the case when using this\n # function inside a Trainer, but it may not be the case otherwise, in particular when unit testing.\n PartialState()\n\n logger.warning(\n f\"There are {equal_predictions_count} out of {len(predictions[:, 0])} instances where the predictions \"\n \"for both options are equal. These instances are ignored in the accuracy computation.\",\n )\n\n # Filter out equal predictions\n predictions = predictions[~equal_mask]\n labels = labels[~equal_mask]\n\n # Use the remaining predictions for accuracy calculation\n predictions = np.argmax(predictions, axis=1)\n\n accuracy = np.array(predictions == labels, dtype=float).mean().item()\n return {\"accuracy\": accuracy}\n\n\nclass PRMTrainer(_BaseTrainer):\n \"\"\"\n Initialize PRMTrainer.\n\n Args:\n model ([`~transformers.PreTrainedModel`]):\n The model to train, preferably an `AutoModelForTokenClassification`.\n args ([`experimental.prm.PRMConfig`]):\n The arguments to use for training.\n data_collator ([`~transformers.DataCollator`]):\n The data collator to use for training. If None is specified, the default data collator\n ([`~transformers.DataCollatorForTokenClassification`]) will be used which will pad the sequences to the\n maximum length of the sequences in the batch, given a dataset of paired sequences.\n train_dataset ([`~datasets.Dataset`]):\n The dataset to use for training.\n eval_dataset ([`~datasets.Dataset`]):\n The dataset to use for evaluation.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. If provided, will be used to automatically process the inputs\n for the model, and it will be saved along the model to make it easier to rerun an interrupted training or\n reuse the fine-tuned model.\n model_init (`Callable[[], transformers.PreTrainedModel]`):\n The model initializer to use for training. If None is specified, the default model initializer will be\n used.\n compute_metrics (`Callable[[transformers.EvalPrediction], dict]`, *optional* defaults to `compute_accuracy`):\n The metrics to use for evaluation. If no metrics are specified, the default metric (`compute_accuracy`)\n will be used.\n callbacks (`list[transformers.TrainerCallback]`):\n The callbacks to use for training.\n optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):\n The optimizer and scheduler to use for training.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):\n The function to use to preprocess the logits before computing the metrics.\n peft_config (`dict`, defaults to `None`):\n The PEFT configuration to use for training. If you pass a PEFT configuration, the model will be wrapped in\n a PEFT model.\n \"\"\"\n\n _tag_names = [\"trl\", \"prm\"]\n _name = \"PRM\"\n _paper = {\n \"title\": \"Solving math word problems with process-and outcome-based feedback\",\n \"id\": \"2211.14275\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{uesato2022solving,\n title = {{Solving Math Word Problems With Process- and Outcome-Based Feedback}},\n author = {Uesato, Jonathan and Kushman, Nate and Kumar, Ramana and Song, Francis and Siegel, Noah and Wang, Lisa and Creswell, Antonia and Irving, Geoffrey and Higgins, Irina},\n year = 2022,\n journal = {arXiv preprint arXiv:2211.14275}\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module | None = None,\n args: PRMConfig | None = None,\n data_collator: DataCollator | None = None,\n train_dataset: Dataset | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n processing_class: PreTrainedTokenizerBase\n | BaseImageProcessor\n | FeatureExtractionMixin\n | ProcessorMixin\n | None = None,\n model_init: Callable[[], PreTrainedModel] | None = None,\n compute_metrics: Callable[[EvalPrediction], dict] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (\n None,\n None,\n ),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n peft_config: dict | None = None,\n ):\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n\n if peft_config is not None or (is_peft_available() and isinstance(model, PeftModel)):\n model = prepare_peft_model(model, peft_config, args)\n\n # Disable dropout in the model\n if args.disable_dropout:\n disable_dropout_in_model(model)\n\n if compute_metrics is None:\n compute_metrics = compute_accuracy\n\n if data_collator is None:\n if processing_class is None:\n raise ValueError(\n \"A processing_class must be specified when using the default DataCollatorForTokenClassification\"\n )\n data_collator = DataCollatorForTokenClassification(processing_class)\n\n if \"input_ids\" not in train_dataset.column_names:\n with PartialState().main_process_first():\n fn_kwargs = {\n \"tokenizer\": processing_class,\n \"step_separator\": args.step_separator,\n \"max_length\": args.max_length,\n \"max_completion_length\": args.max_completion_length,\n \"train_on_last_step_only\": args.train_on_last_step_only,\n }\n train_fn_kwargs = {**fn_kwargs, \"is_eval\": False}\n train_dataset = train_dataset.map(\n self.tokenize_row,\n fn_kwargs=train_fn_kwargs,\n num_proc=args.dataset_num_proc,\n remove_columns=train_dataset.features,\n desc=\"Tokenizing train dataset\",\n features=features.Features( # needed to avoid map to cast labels to bool\n {\n \"labels\": features.Sequence(features.Value(\"int64\")),\n \"input_ids\": features.Sequence(features.Value(\"int64\")),\n }\n ),\n )\n\n eval_fn_kwargs = {**fn_kwargs, \"is_eval\": True}\n if eval_dataset is not None:\n eval_dataset = eval_dataset.map(\n self.tokenize_row,\n fn_kwargs=eval_fn_kwargs,\n num_proc=args.dataset_num_proc,\n remove_columns=eval_dataset.features,\n desc=\"Tokenizing eval dataset\",\n features=features.Features( # needed to avoid map to cast labels to bool\n {\n \"labels\": features.Sequence(features.Value(\"int64\")),\n \"input_ids\": features.Sequence(features.Value(\"int64\")),\n }\n ),\n )\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n model_init=model_init,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n # Add tags for models that have been loaded with the correct transformers version\n if hasattr(self.model, \"add_model_tags\"):\n self.model.add_model_tags(self._tag_names)\n\n @staticmethod\n def tokenize_row(\n features,\n tokenizer,\n step_separator,\n max_length,\n max_completion_length,\n train_on_last_step_only,\n is_eval,\n ):\n r\"\"\"\n Tokenize a row of the dataset.\n\n Args:\n features (`dict[str, str]`):\n Row of the dataset, should contain the keys `\"prompt\"`, `\"completions\"`, and `\"labels\"`.\n tokenizer ([`~transformers.PreTrainedTokenizerBase`]):\n Tokenizer used to process the data.\n step_separator (`str`):\n Separator between steps in the completion.\n max_length (`int` or `None`):\n Maximum length of the sequences (prompt + completion). If `None`, the sequences are not truncated.\n max_completion_length (`int` or `None`):\n Maximum length of the completion sequences. If `None`, the completion sequences are not truncated.\n train_on_last_step_only (`bool`):\n Whether to train only on the last step. If `True`, the labels are `-100` for all tokens except the last\n token of the completion.\n is_eval (`bool`):\n Whether the function is used to tokenize samples from a training or an evaluation dataset. Used only if\n `train_on_last_step_only` is set to `True`.\n\n Returns:\n `dict[str, list[int]]`:\n Tokenized sequences with the keys `\"input_ids\"`, and `\"labels\".\n\n Example:\n ```python\n >>> from transformers import AutoTokenizer\n\n >>> tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-0.5B\")\n >>> features = {\n ... \"prompt\": \"Which number is larger, 9.8 or 9.11?\",\n ... \"completions\": [\"11 is greater than 8.\", \"Hence, 9.11 > 9.8.\"],\n ... \"labels\": [True, False],\n ... }\n >>> PRMTrainer.tokenize_row(\n ... features, tokenizer, \"\\n\", max_completion_length=None, train_on_last_step_only=False, is_eval=False\n ... )\n {'input_ids': [23085, 1372, 374, 8131, 11, 220, 24, 13, 23, 476, 220, 24, 13, 16, 16, 30, 16, 16, 374, 7046, 1091, 220, 23, 13, 198, 39, 763, 11, 220, 24, 13, 16, 16, 861, 220, 24, 13, 23, 13, 198],\n 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, 1, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 0]}\n ```\n \"\"\"\n # Tokenize the prompt and completions\n prompt_ids = tokenizer(features[\"prompt\"], add_special_tokens=False)[\"input_ids\"]\n completions_ids = [\n tokenizer(completion, add_special_tokens=False)[\"input_ids\"] for completion in features[\"completions\"]\n ]\n if train_on_last_step_only and not is_eval:\n labels = [-100] * (len(features[\"labels\"]) - 1) + [int(features[\"labels\"][-1])]\n else:\n labels = [int(label) for label in features[\"labels\"]]\n\n # Get the ID of the separator token and add it to the completions\n separator_ids = tokenizer.encode(step_separator, add_special_tokens=False)\n completions_ids = [completion + separator_ids for completion in completions_ids]\n\n # Create the label\n labels = [\n [-100] * (len(completion) - 1) + [label] for completion, label in zip(completions_ids, labels, strict=True)\n ]\n\n # Join the completions and labels steps\n completion_ids = list(chain(*completions_ids))\n labels = list(chain(*labels))\n\n if tokenizer.bos_token_id is not None:\n prompt_ids = [tokenizer.bos_token_id] + prompt_ids\n\n # Truncate completion sequences\n if max_completion_length is not None:\n completion_ids = completion_ids[:max_completion_length]\n labels = labels[:max_completion_length]\n\n input_ids = prompt_ids + completion_ids\n labels = [-100] * len(prompt_ids) + labels\n\n if max_length is not None:\n input_ids = input_ids[:max_length]\n labels = labels[:max_length]\n\n return {\"input_ids\": input_ids, \"labels\": labels}\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/experimental/utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# This file contains utility classes and functions that are used across more than one experimental trainer or feature.\n\nimport inspect\nimport logging\nfrom copy import deepcopy\nfrom dataclasses import dataclass\nfrom typing import Any\n\nimport torch\nfrom accelerate.utils import is_peft_model\nfrom packaging.version import Version\nfrom torch import nn\nfrom torch.nn.utils.rnn import pad_sequence\nfrom transformers import PreTrainedModel, PreTrainedTokenizerBase, TrainingArguments\nfrom transformers.integrations.deepspeed import is_deepspeed_zero3_enabled\nfrom transformers.utils import (\n is_peft_available,\n is_torch_mlu_available,\n is_torch_npu_available,\n is_torch_xpu_available,\n)\n\nfrom ..trainer.utils import pad\n\n\nif is_peft_available():\n import peft\n from peft import PeftConfig, PeftModel, get_peft_model\n\n\n@dataclass\nclass DPODataCollatorWithPadding:\n r\"\"\"\n DPO DataCollator class that pads the tokenized inputs to the maximum length of the batch.\n\n Args:\n pad_token_id (`int` defaults to 0):\n The tokenizer's pad_token_id.\n is_encoder_decoder (`bool` or `None`, `optional`, defaults to `None`):\n Whether you model has an encoder_decoder architecture.\n \"\"\"\n\n pad_token_id: int = 0\n is_encoder_decoder: bool | None = False\n\n def __call__(self, features: list[dict[str, Any]]) -> dict[str, Any]:\n # first, pad everything to the same length\n padded_batch = {}\n for k in features[0].keys():\n if k.endswith((\"_input_ids\", \"_attention_mask\", \"_labels\", \"_pixel_values\")):\n if self.is_encoder_decoder:\n to_pad = [torch.LongTensor(ex[k]) for ex in features]\n\n if (k.startswith(\"prompt\")) and (k.endswith(\"input_ids\")):\n if self.pad_token_id is None:\n raise ValueError(\n \"Padding is enabled, but the tokenizer is not configured with a padding token.\"\n \" Explicitly set `tokenizer.pad_token` (e.g. `tokenizer.pad_token = tokenizer.eos_token`)\"\n \" before calling the trainer.\"\n )\n padding_value = self.pad_token_id\n elif k.endswith(\"_attention_mask\"):\n padding_value = 0\n elif k.startswith((\"chosen\", \"rejected\", \"completion\")) or (\"decoder\" in k):\n padding_value = -100\n else:\n raise ValueError(f\"Unexpected key in batch '{k}'\")\n padded_batch[k] = pad_sequence(to_pad, batch_first=True, padding_value=padding_value)\n else:\n # Set padding value based on the key\n if k.endswith(\"_input_ids\"):\n if self.pad_token_id is None:\n raise ValueError(\n \"Padding is enabled, but the tokenizer is not configured with a padding token.\"\n \" Explicitly set `tokenizer.pad_token` (e.g. `tokenizer.pad_token = tokenizer.eos_token`)\"\n \" before calling the trainer.\"\n )\n padding_value = self.pad_token_id\n elif k.endswith(\"_labels\"):\n padding_value = -100\n elif k.endswith(\"_attention_mask\"):\n padding_value = 0\n elif k.endswith(\"_pixel_values\"):\n padding_value = 0 # TODO: check if this is correct\n else:\n raise ValueError(f\"Unexpected key in batch '{k}'\")\n\n # Set padding side based on the key\n if k in [\"prompt_input_ids\", \"prompt_attention_mask\"]:\n padding_side = \"left\"\n else:\n padding_side = \"right\"\n\n # Set the dtype\n if k.endswith(\"_pixel_values\"):\n dtype = torch.float32 # will be downcasted if necessary by the Trainer\n else:\n dtype = torch.int64\n\n # Convert to tensor and pad\n to_pad = [torch.tensor(ex[k], dtype=dtype) for ex in features]\n padded_batch[k] = pad(to_pad, padding_value=padding_value, padding_side=padding_side)\n elif k.endswith(\"_logps\"):\n # the cached reference model logprobs\n padded_batch[k] = torch.tensor([ex[k] for ex in features])\n else:\n padded_batch[k] = [ex[k] for ex in features]\n\n return padded_batch\n\n\n@dataclass\nclass DataCollatorForChatML:\n \"\"\"\n Data collator for ChatML format datasets.\n \"\"\"\n\n tokenizer: PreTrainedTokenizerBase\n ignore_index: int = -100\n max_length: int = None\n prompt_key: str = \"prompt\"\n messages_key: str = \"messages\"\n\n def __post_init__(self):\n if self.tokenizer.pad_token_id is None:\n raise ValueError(\"The tokenizer does not have a pad token. Please set `pad_token_id` in the tokenizer.\")\n if self.max_length is None:\n # set a sensible default\n self.max_length = min(self.tokenizer.model_max_length, 1024)\n\n def __call__(self, examples: list[dict[str, Any]]) -> dict[str, torch.Tensor]:\n input_ids = []\n attention_mask = []\n prompts_input_ids = []\n prompt_attention_mask = []\n labels = []\n\n for example in examples:\n formatted_prompt = example.get(self.prompt_key, None)\n if formatted_prompt is None:\n prompt = example[self.messages_key][:-1]\n formatted_prompt = self.tokenizer.apply_chat_template(\n prompt, add_generation_prompt=True, tokenize=False\n )\n\n if \"input_ids\" not in example:\n message = example[self.messages_key]\n formatted_message = self.tokenizer.apply_chat_template(\n message, add_generation_prompt=False, tokenize=False\n )\n\n tokenized_message = self.tokenizer(\n formatted_message,\n truncation=False,\n padding=False,\n return_tensors=None,\n add_special_tokens=False,\n return_offsets_mapping=True,\n )\n message_input_ids_full = tokenized_message[\"input_ids\"]\n offsets = tokenized_message.get(\"offset_mapping\")\n\n if offsets is not None:\n prompt_char_len = len(formatted_prompt)\n completion_start_idx_full = next(\n (idx for idx, (start, _) in enumerate(offsets) if start >= prompt_char_len),\n len(message_input_ids_full),\n )\n else:\n tokenized_prompt_full = self.tokenizer(\n formatted_prompt,\n truncation=False,\n padding=False,\n return_tensors=None,\n add_special_tokens=False,\n )\n completion_start_idx_full = len(tokenized_prompt_full[\"input_ids\"])\n\n prompt_tokens_full = message_input_ids_full[:completion_start_idx_full]\n completion_input_ids_full = message_input_ids_full[completion_start_idx_full:]\n\n if self.max_length is not None and len(message_input_ids_full) > self.max_length:\n completion_ids = completion_input_ids_full\n if len(completion_ids) >= self.max_length:\n completion_ids = completion_ids[-self.max_length :]\n prompt_ids = []\n else:\n max_prompt_tokens = self.max_length - len(completion_ids)\n prompt_ids = prompt_tokens_full[-max_prompt_tokens:] if max_prompt_tokens > 0 else []\n message_input_ids = prompt_ids + completion_ids\n else:\n message_input_ids = message_input_ids_full\n prompt_ids = prompt_tokens_full\n\n input_ids.append(message_input_ids)\n attention_mask.append([1] * len(message_input_ids))\n current_prompt_ids = prompt_ids\n else:\n message_input_ids = example[\"input_ids\"]\n input_ids.append(message_input_ids)\n if \"attention_mask\" in example:\n attention_mask.append(example[\"attention_mask\"])\n else:\n attention_mask.append([1] * len(message_input_ids))\n\n tokenized_prompt = self.tokenizer(\n formatted_prompt,\n truncation=True,\n max_length=len(message_input_ids),\n padding=False,\n return_tensors=None,\n add_special_tokens=False,\n )\n current_prompt_ids = tokenized_prompt[\"input_ids\"]\n\n prompts_input_ids.append(current_prompt_ids)\n prompt_attention_mask.append([1] * len(current_prompt_ids))\n\n label = [self.ignore_index] * len(input_ids[-1])\n completion_start_idx = len(current_prompt_ids)\n label[completion_start_idx:] = input_ids[-1][completion_start_idx:]\n labels.append(label)\n\n # convert to list of tensors and pad\n input_ids = [torch.tensor(ids, dtype=torch.long) for ids in input_ids]\n attention_mask = [torch.tensor(mask, dtype=torch.long) for mask in attention_mask]\n labels = [torch.tensor(label, dtype=torch.long) for label in labels]\n input_ids = pad(input_ids, padding_side=\"left\", padding_value=self.tokenizer.pad_token_id)\n attention_mask = pad(attention_mask, padding_side=\"left\", padding_value=0)\n labels = pad(labels, padding_side=\"left\", padding_value=self.ignore_index)\n\n prompts_input_ids = [torch.tensor(ids, dtype=torch.long) for ids in prompts_input_ids]\n prompt_attention_mask = [torch.tensor(mask, dtype=torch.long) for mask in prompt_attention_mask]\n prompts_input_ids = pad(prompts_input_ids, padding_side=\"left\", padding_value=self.tokenizer.pad_token_id)\n prompt_attention_mask = pad(prompt_attention_mask, padding_side=\"left\", padding_value=0)\n\n return {\n \"input_ids\": input_ids,\n \"attention_mask\": attention_mask,\n \"labels\": labels,\n \"prompts\": prompts_input_ids,\n \"prompt_attention_mask\": prompt_attention_mask,\n }\n\n\ndef truncate_right(\n input_ids: torch.Tensor, stop_token_id: int, pad_token_id: int\n) -> tuple[torch.Tensor, torch.Tensor]:\n \"\"\"\n Truncates the input tensor from the right side after the first occurrence of the stop token.\n\n Args:\n input_ids (`torch.Tensor`):\n The tensor containing the responses to be truncated\n stop_token_id (`int`):\n The token ID representing the stop token where truncation occurs\n pad_token_id (`int`):\n The token ID representing the pad token used to fill the truncated responses\n\n Returns:\n tuple:\n - `output_ids` (`torch.Tensor`):\n The truncated responses tensor with pad tokens filled after the stop token\n - `mask` (`torch.Tensor`):\n The mask tensor to indicate the padding tokens\n \"\"\"\n trunc_idxs = first_true_indices(input_ids == stop_token_id).unsqueeze(-1)\n new_size = [1] * (len(input_ids.size()) - 1) + [input_ids.shape[1]]\n idxs = torch.arange(input_ids.shape[1], device=input_ids.device).view(*new_size)\n output_ids = torch.masked_fill(input_ids, idxs > trunc_idxs, pad_token_id)\n mask = torch.masked_fill(torch.ones_like(input_ids), idxs > trunc_idxs, 0)\n return output_ids, mask\n\n\nSIMPLE_CHAT_TEMPLATE = \"{% for message in messages %}{{message['role'].capitalize() + ': ' + message['content'] + '\\n\\n'}}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}\"\n\n\ndef add_bos_token_if_needed(\n bos_token_id: int | None,\n prompt_len_input_ids: int,\n prompt_tokens: dict[str, list[int]],\n chosen_prompt_len_input_ids: int,\n chosen_tokens: dict[str, list[int]],\n rejected_prompt_len_input_ids: int,\n rejected_tokens: dict[str, list[int]],\n):\n if bos_token_id is not None:\n if prompt_len_input_ids == 0 or bos_token_id != prompt_tokens[\"prompt_input_ids\"][0]:\n prompt_tokens[\"prompt_input_ids\"] = [bos_token_id] + prompt_tokens[\"prompt_input_ids\"]\n prompt_tokens[\"prompt_attention_mask\"] = [1] + prompt_tokens[\"prompt_attention_mask\"]\n if chosen_prompt_len_input_ids == 0 or bos_token_id != chosen_tokens[\"prompt_input_ids\"][0]:\n chosen_tokens[\"prompt_input_ids\"] = [bos_token_id] + chosen_tokens[\"prompt_input_ids\"]\n chosen_tokens[\"prompt_attention_mask\"] = [1] + chosen_tokens[\"prompt_attention_mask\"]\n if rejected_prompt_len_input_ids == 0 or bos_token_id != rejected_tokens[\"prompt_input_ids\"][0]:\n rejected_tokens[\"prompt_input_ids\"] = [bos_token_id] + rejected_tokens[\"prompt_input_ids\"]\n rejected_tokens[\"prompt_attention_mask\"] = [1] + rejected_tokens[\"prompt_attention_mask\"]\n return prompt_tokens, chosen_tokens, rejected_tokens\n\n\ndef add_eos_token_if_needed(\n eos_token_id: int, chosen_tokens: dict[str, list[int]], rejected_tokens: dict[str, list[int]]\n):\n if len(chosen_tokens[\"input_ids\"]) == 0 or eos_token_id != chosen_tokens[\"input_ids\"][-1]:\n chosen_tokens[\"input_ids\"].append(eos_token_id)\n chosen_tokens[\"attention_mask\"].append(1)\n if len(rejected_tokens[\"input_ids\"]) == 0 or eos_token_id != rejected_tokens[\"input_ids\"][-1]:\n rejected_tokens[\"input_ids\"].append(eos_token_id)\n rejected_tokens[\"attention_mask\"].append(1)\n return chosen_tokens, rejected_tokens\n\n\ndef first_true_indices(bools: torch.Tensor, dtype=torch.long) -> torch.Tensor:\n \"\"\"\n Takes an N-dimensional bool tensor and returns an (N-1)-dimensional tensor of integers giving the position of the\n first True in each \"row\".\n\n Returns the length of the rows (bools.size(-1)) if no element is True in a given row.\n\n Args:\n bools (`torch.Tensor`):\n An N-dimensional boolean tensor.\n dtype (`torch.dtype`, optional):\n The desired data type of the output tensor. Defaults to `torch.long`.\n\n Returns:\n `torch.Tensor`:\n An (N-1)-dimensional tensor of integers indicating the position of the first True in each row. If no True\n value is found in a row, returns the length of the row.\n \"\"\"\n row_len = bools.size(-1)\n zero_or_index = row_len * (~bools).type(dtype) + torch.arange(row_len, dtype=dtype, device=bools.device)\n return torch.min(zero_or_index, dim=-1).values\n\n\ndef get_reward(\n model: torch.nn.Module, query_responses: torch.Tensor, pad_token_id: int, context_length: int\n) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:\n \"\"\"\n Computes the reward logits and the rewards for a given model and query responses.\n\n Args:\n model (`torch.nn.Module`):\n The model used to compute the reward logits.\n query_responses (`torch.Tensor`):\n The tensor containing the query responses.\n pad_token_id (`int`):\n The token ID representing the pad token.\n context_length (`int`):\n The length of the context in the query responses.\n\n Returns:\n tuple:\n - `reward_logits` (`torch.Tensor`):\n The logits for the reward model.\n - `final_rewards` (`torch.Tensor`):\n The final rewards for each query response.\n - `sequence_lengths` (`torch.Tensor`):\n The lengths of the sequences in the query responses.\n \"\"\"\n attention_mask = query_responses != pad_token_id\n position_ids = attention_mask.cumsum(1) - attention_mask.long() # exclusive cumsum\n lm_backbone = getattr(model, model.base_model_prefix)\n input_ids = torch.masked_fill(query_responses, ~attention_mask, 0)\n output = lm_backbone(\n input_ids=input_ids,\n attention_mask=attention_mask,\n position_ids=position_ids,\n return_dict=True,\n output_hidden_states=True,\n use_cache=False, # otherwise mistral-based RM would error out\n )\n reward_logits = model.score(output.hidden_states[-1])\n sequence_lengths = first_true_indices(query_responses[:, context_length:] == pad_token_id) - 1 + context_length\n # https://github.com/huggingface/transformers/blob/dc68a39c8111217683bf49a4912d0c9018bab33d/src/transformers/models/gpt2/modeling_gpt2.py#L1454\n return (\n reward_logits,\n reward_logits[\n torch.arange(reward_logits.size(0), device=reward_logits.device),\n sequence_lengths,\n ].squeeze(-1),\n sequence_lengths,\n )\n\n\ndef prepare_model_for_kbit_training(model, use_gradient_checkpointing=True, gradient_checkpointing_kwargs=None):\n r\"\"\"\n Prepare a k-bit quantized transformers model for training (PEFT/QLoRA).\n \"\"\"\n loaded_in_kbit = getattr(model, \"is_loaded_in_8bit\", False) or getattr(model, \"is_loaded_in_4bit\", False)\n quant_methods = [\"gptq\", \"aqlm\", \"eetq\", \"torchao\", \"hqq\"]\n is_quantized = getattr(model, \"quantization_method\", None) in quant_methods or getattr(\n model, \"hqq_quantized\", False\n )\n\n if gradient_checkpointing_kwargs is None:\n gradient_checkpointing_kwargs = {}\n\n for _, param in model.named_parameters():\n # freeze all parameters\n param.requires_grad = False\n\n # Enable gradient checkpointing if needed\n if (loaded_in_kbit or is_quantized) and use_gradient_checkpointing:\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n # backward-compatible hook\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n supports_gc_kwargs = \"gradient_checkpointing_kwargs\" in list(\n inspect.signature(model.gradient_checkpointing_enable).parameters\n )\n gc_kwargs = {\"gradient_checkpointing_kwargs\": gradient_checkpointing_kwargs} if supports_gc_kwargs else {}\n model.gradient_checkpointing_enable(**gc_kwargs)\n\n return model\n\n\ndef enable_gradient_checkpointing(\n model: PreTrainedModel, gradient_checkpointing_kwargs: dict | None\n) -> PreTrainedModel:\n \"\"\"Enables gradient checkpointing for the model.\"\"\"\n # Enable gradient checkpointing on the base model for PEFT\n if is_peft_model(model):\n model.base_model.gradient_checkpointing_enable()\n # Enable gradient checkpointing for non-PEFT models\n else:\n model.gradient_checkpointing_enable()\n\n gradient_checkpointing_kwargs = gradient_checkpointing_kwargs or {}\n use_reentrant = (\n \"use_reentrant\" not in gradient_checkpointing_kwargs or gradient_checkpointing_kwargs[\"use_reentrant\"]\n )\n\n if use_reentrant:\n if hasattr(model, \"enable_input_require_grads\"):\n model.enable_input_require_grads()\n else:\n\n def make_inputs_require_grad(module, input, output):\n output.requires_grad_(True)\n\n model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)\n\n return model\n\n\ndef prepare_peft_model(\n model: PreTrainedModel, peft_config: \"PeftConfig | None\", args: TrainingArguments\n) -> PreTrainedModel:\n \"\"\"Prepares a model for PEFT training.\"\"\"\n if not is_peft_available():\n raise ImportError(\"PEFT is required to use a peft model. Run `pip install peft`.\")\n\n if isinstance(model, PeftModel) and peft_config is not None:\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first merge and \"\n \"unload the existing adapter, save the resulting base model, and then pass that base model along with the \"\n \"new `peft_config` to the trainer.\"\n )\n\n # Handle quantized models (QLoRA)\n is_qlora = getattr(model, \"is_loaded_in_4bit\", False) or getattr(model, \"is_loaded_in_8bit\", False)\n\n is_sharded_qlora = False\n if getattr(model, \"is_loaded_in_4bit\", False):\n # Check if model is sharded (FSDP/DS-Zero3)\n for _, param in model.named_parameters():\n if param.__class__.__name__ == \"Params4bit\":\n is_sharded_qlora = param.data.device.type in {\"cpu\", \"meta\"}\n break\n\n # Prepare model for kbit training if needed\n if is_qlora and not is_sharded_qlora and not isinstance(model, PeftModel):\n model = prepare_model_for_kbit_training(\n model,\n use_gradient_checkpointing=args.gradient_checkpointing,\n gradient_checkpointing_kwargs=args.gradient_checkpointing_kwargs or {},\n )\n # Disable gradient checkpointing as it's handled by prepare_model_for_kbit_training\n args.gradient_checkpointing = False\n elif args.gradient_checkpointing:\n model = enable_gradient_checkpointing(model, args.gradient_checkpointing_kwargs)\n\n # Create PEFT model\n if peft_config is not None:\n if (\n Version(peft.__version__) >= Version(\"0.12\") # autocast_adapter_dtype introduced in 0.12\n and getattr(model, \"is_loaded_in_4bit\", False)\n and is_sharded_qlora\n ):\n model = get_peft_model(model, peft_config, autocast_adapter_dtype=False)\n else:\n model = get_peft_model(model, peft_config)\n\n # Handle bf16 casting for 4-bit models\n if args.bf16 and getattr(model, \"is_loaded_in_4bit\", False) and not is_sharded_qlora:\n peft_module_casting_to_bf16(model)\n\n return model\n\n\ndef pad_to_length(tensor: torch.Tensor, length: int, pad_value: int | float, dim: int = -1) -> torch.Tensor:\n if tensor.size(dim) >= length:\n return tensor\n else:\n pad_size = list(tensor.shape)\n pad_size[dim] = length - tensor.size(dim)\n return torch.cat(\n [\n tensor,\n pad_value * torch.ones(*pad_size, dtype=tensor.dtype, device=tensor.device),\n ],\n dim=dim,\n )\n\n\ndef empty_cache() -> None:\n \"\"\"Empties the cache of the available torch device.\n\n This function checks for the availability of different torch devices (XPU, MLU, NPU, CUDA) and empties the cache of\n the first available device it finds.\n\n If none of the specific devices are available, it defaults to emptying the CUDA cache.\n \"\"\"\n if is_torch_xpu_available():\n torch.xpu.empty_cache()\n elif is_torch_mlu_available():\n torch.mlu.empty_cache()\n elif is_torch_npu_available():\n torch.npu.empty_cache()\n else:\n torch.cuda.empty_cache()\n\n\ndef peft_module_casting_to_bf16(model):\n for name, module in model.named_modules():\n if isinstance(module, torch.nn.LayerNorm) or \"norm\" in name:\n module = module.to(torch.float32)\n elif any(x in name for x in [\"lm_head\", \"embed_tokens\", \"wte\", \"wpe\"]):\n if hasattr(module, \"weight\"):\n if module.weight.dtype == torch.float32:\n module = module.to(torch.bfloat16)\n\n\nLAYER_PATTERNS = [\n \"transformer.h.{layer}\",\n \"model.decoder.layers.{layer}\",\n \"gpt_neox.layers.{layer}\",\n \"model.layers.{layer}\",\n]\n\n\ndef create_reference_model(\n model: nn.Module, num_shared_layers: int | None = None, pattern: str | None = None\n) -> nn.Module:\n \"\"\"\n Creates a static reference copy of a model. Note that model will be in `.eval()` mode.\n\n Args:\n model ([`nn.Module`]): The model to be copied.\n num_shared_layers (`int`, *optional*):\n The number of initial layers that are shared between both models and kept frozen.\n pattern (`str`, *optional*): The shared layers are selected with a string pattern\n (e.g. \"transformer.h.{layer}\" for GPT2) and if a custom pattern is necessary it can be passed here.\n\n Returns:\n [`nn.Module`]\n \"\"\"\n if is_deepspeed_zero3_enabled():\n raise ValueError(\n \"DeepSpeed ZeRO-3 is enabled and is not compatible with `create_reference_model()`. Please instantiate your reference model directly with `AutoModelForCausalLM.from_pretrained()`.\"\n )\n\n parameter_names = [n for n, _ in model.named_parameters()]\n ref_model = deepcopy(model)\n\n # if no layers are shared, return copy of model\n if num_shared_layers is None:\n for param_name in parameter_names:\n param = ref_model.get_parameter(param_name)\n param.requires_grad = False\n return ref_model.eval()\n\n # identify layer name pattern\n if pattern is not None:\n pattern = pattern.format(layer=num_shared_layers)\n else:\n for pattern_candidate in LAYER_PATTERNS:\n pattern_candidate = pattern_candidate.format(layer=num_shared_layers)\n if any(pattern_candidate in name for name in parameter_names):\n pattern = pattern_candidate\n break\n\n if pattern is None:\n raise ValueError(\"Layer pattern could not be matched.\")\n\n # divide parameters in shared and unshared parameter lists\n shared_param_list = []\n unshared_param_list = []\n\n shared_parameter = True\n for name, _param in model.named_parameters():\n if pattern in name:\n shared_parameter = False\n if shared_parameter:\n shared_param_list.append(name)\n else:\n unshared_param_list.append(name)\n\n # create reference of the original parameter if they are shared\n for param_name in shared_param_list:\n param = model.get_parameter(param_name)\n param.requires_grad = False\n\n _ref_param = ref_model.get_parameter(param_name)\n\n # for all other parameters just make sure they don't use gradients\n for param_name in unshared_param_list:\n param = ref_model.get_parameter(param_name)\n param.requires_grad = False\n\n if pattern is not None and len(unshared_param_list) == 0:\n logging.warning(\"Pattern passed or found, but no layers matched in the model. Check for a typo.\")\n\n return ref_model.eval()\n" }, { "path": "trl/experimental/winrate_callback.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport logging\n\nimport pandas as pd\nfrom accelerate import Accelerator\nfrom accelerate.utils import gather_object, is_wandb_available\nfrom transformers import (\n GenerationConfig,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n Trainer,\n TrainerCallback,\n TrainerControl,\n TrainerState,\n TrainingArguments,\n)\n\nfrom ..models.utils import unwrap_model_for_generation\nfrom ..trainer.utils import log_table_to_comet_experiment\n\n\nif is_wandb_available():\n import wandb\n\n# Logger for module-level logging\nlogger = logging.getLogger(__name__)\n\n\ndef _generate_completions(\n prompts: list[str],\n model: PreTrainedModel,\n tokenizer: PreTrainedTokenizerBase,\n accelerator: Accelerator,\n generation_config: GenerationConfig | None,\n batch_size: int = 1,\n) -> list[str]:\n \"\"\"\n Generates completions for a list of pre-formatted prompts from the given model.\n\n Args:\n prompts (list[str]): A list of input prompts for which completions are to be generated.\n model (PreTrainedModel): The pre-trained model to be used for generation.\n tokenizer (PreTrainedTokenizerBase): The tokenizer to be used for encoding and decoding.\n accelerator (Accelerator): The accelerator to be used for model execution.\n generation_config (GenerationConfig): Configuration for text generation.\n batch_size (int, optional): The number of prompts to process in each batch. Default is 1.\n\n Returns:\n list[str]: A list of generated text completions corresponding to the input prompts.\n \"\"\"\n completions = []\n # TODO: Override model.generation_config with generation_kwargs\n with unwrap_model_for_generation(model, accelerator) as unwrapped_model:\n for idx in range(0, len(prompts), batch_size):\n batch = prompts[idx : idx + batch_size]\n tokenized_batch = tokenizer(batch, return_tensors=\"pt\", padding=True, truncation=True).to(model.device)\n generations = unwrapped_model.generate(\n **tokenized_batch,\n generation_config=generation_config,\n )\n for prompt, generation in zip(tokenized_batch.input_ids, generations, strict=True):\n # Remove prompt from generation\n generation = generation[len(prompt) :]\n completion = tokenizer.decode(generation, skip_special_tokens=True)\n completions.append(completion)\n return completions\n\n\ndef _win_rate_completions_df(\n state: TrainerState, prompts: list[str], completions: list[str], winner_indices: list[str]\n) -> pd.DataFrame:\n global_step = [str(state.global_step)] * len(prompts)\n data = list(zip(global_step, prompts, completions, winner_indices, strict=True))\n # Split completions from reference model and policy\n split_data = [(item[0], item[1], item[2][0], item[2][1], item[3]) for item in data]\n return pd.DataFrame(split_data, columns=[\"step\", \"prompt\", \"reference_model\", \"policy\", \"winner_index\"])\n\n\nclass WinRateCallback(TrainerCallback):\n \"\"\"\n A [`~transformers.TrainerCallback`] that computes the win rate of a model based on a reference.\n\n It generates completions using prompts from the evaluation dataset and compares the trained model's outputs against\n a reference. The reference is either the initial version of the model (before training) or the reference model, if\n available in the trainer. During each evaluation step, a judge determines how often the trained model's completions\n win against the reference using a judge. The win rate is then logged in the trainer's logs under the key\n `\"eval_win_rate\"`.\n\n Usage:\n ```python\n from trl import DPOTrainer\n from trl.experimental.judges import PairRMJudge\n from trl.experimental.winrate_callback import WinRateCallback\n\n trainer = DPOTrainer(...)\n judge = PairRMJudge()\n win_rate_callback = WinRateCallback(judge=judge, trainer=trainer)\n trainer.add_callback(win_rate_callback)\n ```\n\n Args:\n judge ([`experimental.judges.BasePairwiseJudge`]):\n The judge to use for comparing completions.\n trainer (`Trainer`):\n Trainer to which the callback will be attached. The trainer's evaluation dataset must include a `\"prompt\"`\n column containing the prompts for generating completions. If the `Trainer` has a reference model (via the\n `ref_model` attribute), it will use this reference model for generating the reference completions;\n otherwise, it defaults to using the initial model.\n generation_config ([`~transformers.GenerationConfig`], *optional*):\n The generation config to use for generating completions.\n num_prompts (`int`, *optional*):\n The number of prompts to generate completions for. If not provided, defaults to the number of examples in\n the evaluation dataset.\n shuffle_order (`bool`, *optional*, defaults to `True`):\n Whether to shuffle the order of the completions before judging.\n use_soft_judge (`bool`, *optional*, defaults to `False`):\n Whether to use a soft judge that returns a win probability between 0 and 1 for the first completion vs the\n second.\n \"\"\"\n\n def __init__(\n self,\n judge,\n trainer: Trainer,\n generation_config: GenerationConfig | None = None,\n num_prompts: int | None = None,\n shuffle_order: bool = True,\n use_soft_judge: bool = False,\n ):\n self.judge = judge\n self.trainer = trainer\n self.shuffle_order = shuffle_order\n self.generation_config = generation_config\n self.ref_completions = []\n self.use_soft_judge = use_soft_judge\n\n if self.trainer.eval_dataset is None:\n raise ValueError(\"Trainer must have an evaluation dataset to use the WinRateCallback.\")\n else:\n self.eval_dataset = self.trainer.eval_dataset\n\n if num_prompts is not None:\n self.eval_dataset = self.eval_dataset.select(range(num_prompts))\n\n def on_train_begin(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):\n # When the trainer is initialized, we generate completions for the reference model.\n tokenizer = kwargs[\"processing_class\"]\n tokenizer.padding_side = \"left\"\n accelerator = self.trainer.accelerator\n # Use the reference model if available, otherwise use the initial model\n model = getattr(self.trainer, \"ref_model\", None)\n # At this point, there are two cases where `ref_model` is None:\n # 1. The method doesn't require a reference model.\n # 2. The method uses a reference model, but `ref_model` is set to None.\n # This occurs when using PEFT, where the reference model can be obtained by simply disabling the model's adapter.\n # In theory, we should disable the adapter here, but since it's zero-initialized at the start of training,\n # the model behaves identically with or without the adapter.\n # Therefore, there's no need to explicitly disable it at this point.\n if model is None:\n model = self.trainer.model_wrapped\n with accelerator.split_between_processes(self.eval_dataset[\"prompt\"]) as prompts:\n self.ref_completions = _generate_completions(\n prompts,\n model=model,\n tokenizer=tokenizer,\n accelerator=accelerator,\n generation_config=self.generation_config,\n batch_size=args.per_device_eval_batch_size,\n )\n # Compute initial win rate as a reference point\n completions = list(zip(self.ref_completions, self.ref_completions, strict=True))\n if self.use_soft_judge:\n ref_win_probs = self.judge.judge(prompts, completions, self.shuffle_order, return_scores=True)\n winner_indices = [0 if score > 0.5 else 1 for score in ref_win_probs]\n ref_win_probs = gather_object(ref_win_probs)\n else:\n winner_indices = self.judge.judge(prompts, completions, self.shuffle_order)\n prompts = gather_object(prompts)\n completions = gather_object(completions)\n winner_indices = gather_object(winner_indices)\n\n # Logging\n if self.trainer.accelerator.is_main_process:\n win_rate = sum(winner_idx == 1 for winner_idx in winner_indices) / len(winner_indices)\n if self.use_soft_judge:\n avg_win_prob = 1.0 - sum(ref_win_probs) / len(ref_win_probs)\n self.trainer.log({\"eval_avg_win_prob\": avg_win_prob, \"eval_win_rate\": win_rate})\n else:\n self.trainer.log({\"eval_win_rate\": win_rate})\n\n if \"wandb\" in args.report_to:\n if wandb.run is not None:\n df = _win_rate_completions_df(\n state=state,\n prompts=prompts,\n completions=completions,\n winner_indices=winner_indices,\n )\n wandb.log({\"win_rate_completions\": wandb.Table(dataframe=df)})\n\n if \"comet_ml\" in args.report_to:\n df = _win_rate_completions_df(\n state=state,\n prompts=prompts,\n completions=completions,\n winner_indices=winner_indices,\n )\n log_table_to_comet_experiment(\n name=\"win_rate_completions.csv\",\n table=df,\n )\n\n def on_evaluate(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):\n # At every evaluation step, we generate completions for the model and compare them with the reference\n # completions that have been generated at the beginning of training. We then compute the win rate and log it to\n # the trainer.\n tokenizer = kwargs[\"processing_class\"]\n tokenizer.padding_side = \"left\"\n accelerator = self.trainer.accelerator\n model = self.trainer.model_wrapped\n with accelerator.split_between_processes(self.eval_dataset[\"prompt\"]) as prompts:\n completions = _generate_completions(\n prompts,\n model=model,\n tokenizer=tokenizer,\n accelerator=accelerator,\n generation_config=self.generation_config,\n batch_size=args.per_device_eval_batch_size,\n )\n\n completions = list(zip(self.ref_completions, completions, strict=True))\n\n if self.use_soft_judge:\n ref_win_probs = self.judge.judge(prompts, completions, self.shuffle_order, return_scores=True)\n winner_indices = [0 if score > 0.5 else 1 for score in ref_win_probs]\n ref_win_probs = gather_object(ref_win_probs)\n else:\n winner_indices = self.judge.judge(prompts, completions, self.shuffle_order)\n prompts = gather_object(prompts)\n completions = gather_object(completions)\n winner_indices = gather_object(winner_indices)\n\n # Logging\n if self.trainer.accelerator.is_main_process:\n win_rate = sum(winner_idx == 1 for winner_idx in winner_indices) / len(winner_indices)\n if self.use_soft_judge:\n avg_win_prob = 1.0 - sum(ref_win_probs) / len(ref_win_probs)\n self.trainer.log({\"eval_avg_win_prob\": avg_win_prob, \"eval_win_rate\": win_rate})\n else:\n self.trainer.log({\"eval_win_rate\": win_rate})\n\n if \"wandb\" in args.report_to:\n if wandb.run is not None:\n df = _win_rate_completions_df(\n state=state,\n prompts=prompts,\n completions=completions,\n winner_indices=winner_indices,\n )\n wandb.log({\"win_rate_completions\": wandb.Table(dataframe=df)})\n\n if \"comet_ml\" in args.report_to:\n df = _win_rate_completions_df(\n state=state,\n prompts=prompts,\n completions=completions,\n winner_indices=winner_indices,\n )\n log_table_to_comet_experiment(\n name=\"win_rate_completions.csv\",\n table=df,\n )\n" }, { "path": "trl/experimental/xpo/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .xpo_config import XPOConfig\nfrom .xpo_trainer import XPOTrainer\n\n\n__all__ = [\"XPOConfig\", \"XPOTrainer\"]\n" }, { "path": "trl/experimental/xpo/xpo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom ..online_dpo import OnlineDPOConfig\n\n\n@dataclass\nclass XPOConfig(OnlineDPOConfig):\n r\"\"\"\n Configuration class for the [`experimental.xpo.XPOTrainer`].\n\n Subclass of [`experimental.online_dpo.OnlineDPOConfig`] we can use all its arguments and add the following:\n\n Parameters:\n alpha (`float` or `list[float]`, *optional*, defaults to `1e-5`):\n Weight of the XPO loss term. If a list of floats is provided then the alpha is selected for each new epoch\n and the last alpha is used for the rest of the epochs.\n \"\"\"\n\n alpha: list[float] = field(\n default_factory=lambda: [1e-5],\n metadata={\n \"help\": \"Weight of the XPO loss term. If a list of floats is provided then the alpha is selected for each \"\n \"new epoch and the last alpha is used for the rest of the epochs.\"\n },\n )\n\n def __post_init__(self):\n super().__post_init__()\n if hasattr(self.alpha, \"__len__\") and len(self.alpha) == 1:\n self.alpha = self.alpha[0]\n" }, { "path": "trl/experimental/xpo/xpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport textwrap\nfrom collections.abc import Callable\nfrom typing import Any\n\nimport jinja2\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom datasets import Dataset, IterableDataset\nfrom transformers import (\n BaseImageProcessor,\n FeatureExtractionMixin,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n)\nfrom transformers.trainer_utils import EvalPrediction\nfrom transformers.training_args import OptimizerNames\nfrom transformers.utils import is_peft_available\n\nfrom ...data_utils import is_conversational, maybe_apply_chat_template\nfrom ...models.utils import unwrap_model_for_generation\nfrom ...trainer.utils import selective_log_softmax\nfrom ..judges import BasePairwiseJudge\nfrom ..online_dpo import OnlineDPOTrainer\nfrom ..utils import SIMPLE_CHAT_TEMPLATE, empty_cache, get_reward, truncate_right\nfrom .xpo_config import XPOConfig\n\n\nif is_peft_available():\n from peft import PeftModel\n\n\nclass XPOTrainer(OnlineDPOTrainer):\n \"\"\"\n Trainer for Exploratory Preference Optimization (XPO).\n\n It is implemented as a subclass of [`experimental.online_dpo.OnlineDPOTrainer`].\n\n Args:\n model ([`~transformers.PreTrainedModel`]):\n The model to train, preferably an `AutoModelForCausalLM`.\n ref_model ([`~transformers.PreTrainedModel`]):\n Hugging Face transformer model with a casual language modelling head. Used for implicit reward computation\n and loss. If no reference model is provided, the trainer will create a reference model with the same\n architecture as the model to be optimized.\n reward_funcs ([`~transformers.PreTrainedModel`]):\n The reward model to score completions with, preferably an\n [`~transformers.AutoModelForSequenceClassification`].\n judge ([`experimental.judges.BasePairwiseJudge`]):\n The judge to use for pairwise comparison of model completions.\n args ([`experimental.xpo.XPOConfig`]):\n The XPO config arguments to use for training.\n data_collator ([`~transformers.DataCollator`]):\n The data collator to use for training. If None is specified, the default data collator\n ([`experimental.utils.DPODataCollatorWithPadding`]) will be used which will pad the sequences to the\n maximum length of the sequences in the batch, given a dataset of paired sequences.\n train_dataset ([`~datasets.Dataset`]):\n The dataset to use for training.\n eval_dataset ([`~datasets.Dataset`]):\n The dataset to use for evaluation.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. If provided, will be used to automatically process the inputs\n for the model, and it will be saved along the model to make it easier to rerun an interrupted training or\n reuse the fine-tuned model.\n peft_config (`dict`):\n The peft config to use for training.\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to\n metric values.\n callbacks (`list[transformers.TrainerCallback]`):\n The callbacks to use for training.\n optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):\n The optimizer and scheduler to use for training.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):\n The function to use to preprocess the logits before computing the metrics.\n \"\"\"\n\n _tag_names = [\"trl\", \"xpo\"]\n _name = \"XPO\"\n _paper = {\n \"title\": \"Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF\",\n \"id\": \"2405.21046\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{jung2024binary,\n title = {{Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF}},\n author = {Tengyang Xie and Dylan J. Foster and Akshay Krishnamurthy and Corby Rosset and Ahmed Awadallah and Alexander Rakhlin},\n year = 2024,\n eprint = {arXiv:2405.21046}\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: PreTrainedModel | nn.Module = None,\n ref_model: PreTrainedModel | nn.Module = None,\n reward_funcs: nn.Module | None = None,\n judge: BasePairwiseJudge | None = None,\n args: XPOConfig | None = None,\n data_collator: Callable | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | dict[str, Dataset] | None = None,\n processing_class: PreTrainedTokenizerBase\n | BaseImageProcessor\n | FeatureExtractionMixin\n | ProcessorMixin\n | None = None,\n reward_processing_classes: PreTrainedTokenizerBase | list[PreTrainedTokenizerBase] | None = None,\n peft_config: dict | None = None,\n compute_metrics: Callable[[EvalPrediction], dict] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None),\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n ) -> None:\n super().__init__(\n model=model,\n ref_model=ref_model,\n judge=judge,\n reward_funcs=reward_funcs,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n reward_processing_classes=reward_processing_classes,\n peft_config=peft_config,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n self._alpha = self.args.alpha\n\n # Overwrite the stats dictionary to include XPO specific statistics\n self.stats = {\n # Remove \"non_score_reward\", \"rlhf_reward\", \"scores\"\n # Add \"loss/dpo\", \"loss/xpo\"\n \"loss/dpo\": [],\n \"loss/xpo\": [],\n \"objective/kl\": [],\n \"objective/entropy\": [],\n \"rewards/chosen\": [],\n \"rewards/rejected\": [],\n \"rewards/accuracies\": [],\n \"rewards/margins\": [],\n \"logps/chosen\": [],\n \"logps/rejected\": [],\n # Replace \"contain_eos_token\" by \"model_contain_eos_token\" and \"ref_contain_eos_token\"\n \"val/model_contain_eos_token\": [],\n \"val/ref_contain_eos_token\": [],\n \"alpha\": [],\n \"beta\": [],\n }\n if self.reward_funcs is not None:\n if len(self.reward_funcs) != 1:\n raise ValueError(\"XPOTrainer only supports one reward function/model.\")\n self.reward_funcs = self.reward_funcs[0]\n self.stats[\"objective/model_scores\"] = []\n self.stats[\"objective/ref_scores\"] = []\n self.stats[\"objective/scores_margin\"] = []\n\n @property\n def alpha(self):\n if isinstance(self._alpha, list):\n epoch = self.state.epoch\n return self._alpha[epoch] if epoch < len(self._alpha) else self._alpha[-1]\n else:\n return self._alpha\n\n def _generate_completions(self, prompts, model):\n with (\n unwrap_model_for_generation(\n model,\n self.accelerator,\n generation_kwargs=self.generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as unwrapped_policy_model_for_gen,\n ):\n model_output = unwrapped_policy_model_for_gen.generate(\n input_ids=prompts[\"input_ids\"],\n attention_mask=prompts[\"attention_mask\"],\n generation_config=self.generation_config,\n )\n\n actual_model_for_ref_generation: torch.nn.Module\n if self.ref_model is None:\n unwrapped_main_model_for_ref_logic = self.accelerator.unwrap_model(model)\n\n if is_peft_available() and isinstance(unwrapped_main_model_for_ref_logic, PeftModel):\n actual_model_for_ref_generation = unwrapped_main_model_for_ref_logic.get_base_model()\n else:\n actual_model_for_ref_generation = unwrapped_main_model_for_ref_logic\n else:\n actual_model_for_ref_generation = self.accelerator.unwrap_model(self.ref_model)\n\n with (\n unwrap_model_for_generation(\n actual_model_for_ref_generation,\n self.accelerator,\n generation_kwargs=self.generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as final_ref_model_for_gen,\n ):\n ref_output = final_ref_model_for_gen.generate(\n input_ids=prompts[\"input_ids\"],\n attention_mask=prompts[\"attention_mask\"],\n generation_config=self.generation_config,\n )\n\n return model_output, ref_output\n\n def _process_completions(self, model_output, ref_output, prompts):\n context_length = prompts[\"input_ids\"].shape[1]\n\n # Process model completions\n model_completion_ids = model_output[:, context_length:]\n model_completion_ids, model_completion_mask = truncate_right(\n model_completion_ids, self.processing_class.eos_token_id, self.processing_class.pad_token_id\n )\n model_data = {\n \"input_ids\": torch.cat((prompts[\"input_ids\"], model_completion_ids), dim=1),\n \"attention_mask\": torch.cat((prompts[\"attention_mask\"], model_completion_mask), dim=1),\n \"raw\": prompts[\"raw\"],\n }\n\n # Process reference model completions\n ref_completion_ids = ref_output[:, context_length:]\n ref_completion_ids, ref_completion_mask = truncate_right(\n ref_completion_ids, self.processing_class.eos_token_id, self.processing_class.pad_token_id\n )\n ref_data = {\n \"input_ids\": torch.cat((prompts[\"input_ids\"], ref_completion_ids), dim=1),\n \"attention_mask\": torch.cat((prompts[\"attention_mask\"], ref_completion_mask), dim=1),\n \"raw\": prompts[\"raw\"],\n }\n\n return model_data, ref_data\n\n def _compute_rewards(self, model_data, ref_data, context_length):\n with torch.no_grad():\n _, model_scores, _ = get_reward(\n self.reward_funcs, model_data[\"input_ids\"], self.processing_class.pad_token_id, context_length\n )\n _, ref_scores, _ = get_reward(\n self.reward_funcs, ref_data[\"input_ids\"], self.processing_class.pad_token_id, context_length\n )\n\n # Apply EOS penalty if needed\n if self.args.missing_eos_penalty is not None:\n model_contain_eos = torch.any(model_data[\"input_ids\"] == self.processing_class.eos_token_id, dim=-1)\n ref_contain_eos = torch.any(ref_data[\"input_ids\"] == self.processing_class.eos_token_id, dim=-1)\n model_scores[~model_contain_eos] -= self.args.missing_eos_penalty\n ref_scores[~ref_contain_eos] -= self.args.missing_eos_penalty\n\n return model_scores, ref_scores\n\n def _compute_judge(self, model_data, ref_data, context_length):\n prompts = model_data[\"raw\"]\n model_data_completions = self.processing_class.batch_decode(\n model_data[\"input_ids\"][:, context_length:], skip_special_tokens=True\n )\n model_data_completions = [completion.strip() for completion in model_data_completions]\n\n ref_data_completions = self.processing_class.batch_decode(\n ref_data[\"input_ids\"][:, context_length:], skip_special_tokens=True\n )\n ref_data_completions = [completion.strip() for completion in ref_data_completions]\n\n if is_conversational({\"prompt\": prompts[0]}):\n model_data_completions = [\n [{\"role\": \"assistant\", \"content\": completion}] for completion in model_data_completions\n ]\n environment = jinja2.Environment()\n template = environment.from_string(SIMPLE_CHAT_TEMPLATE)\n prompts = [template.render(messages=message) for message in prompts]\n model_data_completions = [template.render(messages=completion) for completion in model_data_completions]\n\n ref_data_completions = [\n [{\"role\": \"assistant\", \"content\": completion}] for completion in ref_data_completions\n ]\n ref_data_completions = [template.render(messages=completion) for completion in ref_data_completions]\n\n ranks_of_first_completion = self.judge.judge(\n prompts,\n list(zip(model_data_completions, ref_data_completions, strict=True)),\n )\n # convert ranks to a True/False mask:\n # when rank == 0, it means the first completion is the best\n # when rank == 1, it means the second completion is the best\n return torch.tensor([rank == 0 for rank in ranks_of_first_completion], device=model_data[\"input_ids\"].device)\n\n def _compute_logprobs(self, model, model_data, ref_data, context_length):\n def compute_logprobs_for_data(m, data):\n output = m(data[\"input_ids\"], attention_mask=data[\"attention_mask\"])\n logits = output.logits[:, context_length - 1 : -1]\n token_logprobs = selective_log_softmax(logits, data[\"input_ids\"][:, context_length:])\n return token_logprobs\n\n # Compute logprobs for model completions\n model_logprobs_model_data = compute_logprobs_for_data(model, model_data)\n # Compute logprobs for model on reference completions (for XPO loss)\n model_logprobs_ref_data = compute_logprobs_for_data(model, ref_data)\n\n # Compute logprobs for reference model completions\n with torch.no_grad():\n if self.ref_model is None:\n with model.disable_adapter():\n ref_logprobs_model_data = compute_logprobs_for_data(model, model_data)\n ref_logprobs_ref_data = compute_logprobs_for_data(model, ref_data)\n else:\n ref_logprobs_model_data = compute_logprobs_for_data(self.ref_model, model_data)\n ref_logprobs_ref_data = compute_logprobs_for_data(self.ref_model, ref_data)\n\n # Mask padding tokens\n model_padding_mask = model_data[\"attention_mask\"][:, context_length:] == 0\n ref_padding_mask = ref_data[\"attention_mask\"][:, context_length:] == 0\n model_logprobs_model_data = model_logprobs_model_data.masked_fill(model_padding_mask, 0.0)\n model_logprobs_ref_data = model_logprobs_ref_data.masked_fill(ref_padding_mask, 0.0)\n ref_logprobs_ref_data = ref_logprobs_ref_data.masked_fill(ref_padding_mask, 0.0)\n ref_logprobs_model_data = ref_logprobs_model_data.masked_fill(model_padding_mask, 0.0)\n\n return model_logprobs_model_data, model_logprobs_ref_data, ref_logprobs_ref_data, ref_logprobs_model_data\n\n def _compute_losses(\n self,\n model_logprobs_model_data,\n model_logprobs_ref_data,\n ref_logprobs_ref_data,\n ref_logprobs_model_data,\n chosen_mask,\n ):\n # Compute log probs\n model_logprobs_model_data_sum = model_logprobs_model_data.sum(1)\n model_logprobs_ref_data_sum = model_logprobs_ref_data.sum(1)\n ref_logprobs_ref_data_sum = ref_logprobs_ref_data.sum(1)\n ref_logprobs_model_data_sum = ref_logprobs_model_data.sum(1)\n\n chosen_model_logprobs = torch.where(chosen_mask, model_logprobs_model_data_sum, model_logprobs_ref_data_sum)\n chosen_ref_logprobs = torch.where(chosen_mask, ref_logprobs_model_data_sum, ref_logprobs_ref_data_sum)\n chosen_log_ratios = chosen_model_logprobs - chosen_ref_logprobs\n\n rejected_model_logprobs = torch.where(~chosen_mask, model_logprobs_model_data_sum, model_logprobs_ref_data_sum)\n rejected_ref_logprobs = torch.where(~chosen_mask, ref_logprobs_model_data_sum, ref_logprobs_ref_data_sum)\n rejected_log_ratios = rejected_model_logprobs - rejected_ref_logprobs\n\n # Compute logits as the difference between chosen and rejected log ratios\n logits = chosen_log_ratios - rejected_log_ratios\n\n if self.args.loss_type == \"sigmoid\":\n dpo_losses = -F.logsigmoid(self.beta * logits)\n elif self.args.loss_type == \"ipo\":\n dpo_losses = (logits - 1 / (2 * self.beta)) ** 2\n else:\n raise NotImplementedError(f\"invalid loss type {self.args.loss_type}\")\n\n # Compute XPO specific loss\n xpo_losses = self.alpha * model_logprobs_ref_data_sum\n\n # Total loss\n loss = (dpo_losses + xpo_losses).mean()\n\n return loss, dpo_losses, xpo_losses\n\n def _log_statistics(\n self,\n model_data,\n ref_data,\n model_logprobs_model_data,\n model_logprobs_ref_data,\n ref_logprobs_ref_data,\n ref_logprobs_model_data,\n chosen_mask,\n dpo_losses,\n xpo_losses,\n context_length,\n model_scores=None,\n ref_scores=None,\n ):\n # Helper function to gather and compute mean\n def gather_mean(tensor):\n return self.accelerator.gather_for_metrics(tensor).mean().item()\n\n # Log losses\n self.stats[\"loss/dpo\"].append(gather_mean(dpo_losses))\n self.stats[\"loss/xpo\"].append(gather_mean(xpo_losses))\n\n # Log scores\n if self.reward_funcs is not None:\n self.stats[\"objective/model_scores\"].append(gather_mean(model_scores))\n self.stats[\"objective/ref_scores\"].append(gather_mean(ref_scores))\n self.stats[\"objective/scores_margin\"].append(gather_mean(model_scores - ref_scores))\n\n # Log logprobs\n model_logprobs_model_data_sum = model_logprobs_model_data.sum(1)\n model_logprobs_ref_data_sum = model_logprobs_ref_data.sum(1)\n ref_logprobs_ref_data_sum = ref_logprobs_ref_data.sum(1)\n ref_logprobs_model_data_sum = ref_logprobs_model_data.sum(1)\n\n chosen_model_logprobs = torch.where(chosen_mask, model_logprobs_model_data_sum, model_logprobs_ref_data_sum)\n chosen_ref_logprobs = torch.where(chosen_mask, ref_logprobs_model_data_sum, ref_logprobs_ref_data_sum)\n chosen_log_ratios = chosen_model_logprobs - chosen_ref_logprobs\n\n rejected_model_logprobs = torch.where(~chosen_mask, model_logprobs_model_data_sum, model_logprobs_ref_data_sum)\n rejected_ref_logprobs = torch.where(~chosen_mask, ref_logprobs_model_data_sum, ref_logprobs_ref_data_sum)\n rejected_log_ratios = rejected_model_logprobs - rejected_ref_logprobs\n\n self.stats[\"logps/chosen\"].append(gather_mean(chosen_model_logprobs.mean() + chosen_ref_logprobs.mean()))\n self.stats[\"logps/rejected\"].append(gather_mean(rejected_model_logprobs.mean() + rejected_ref_logprobs.mean()))\n\n # Log rewards\n # Compute various statistics\n chosen_rewards = chosen_log_ratios * self.beta\n rejected_rewards = rejected_log_ratios * self.beta\n self.stats[\"rewards/chosen\"].append(gather_mean(chosen_rewards.mean()))\n self.stats[\"rewards/rejected\"].append(gather_mean(rejected_rewards.mean()))\n\n # Calculate KL divergence for model and ref data\n kl_model_data = model_logprobs_model_data - ref_logprobs_model_data\n kl_ref_data = model_logprobs_ref_data - ref_logprobs_ref_data\n mean_kl = (kl_model_data.sum(1) + kl_ref_data.sum(1)).mean() / 2\n self.stats[\"objective/kl\"].append(gather_mean(mean_kl))\n\n # Calculate entropy for model and ref data\n entropy_model_data = -model_logprobs_model_data.sum(1)\n entropy_ref_data = -model_logprobs_ref_data.sum(1)\n mean_entropy = (entropy_model_data.mean() + entropy_ref_data.mean()) / 2\n self.stats[\"objective/entropy\"].append(gather_mean(mean_entropy))\n\n # Calculate margins\n margin = chosen_rewards - rejected_rewards\n self.stats[\"rewards/margins\"].append(gather_mean(margin.mean()))\n\n # Calculate accuracy\n accuracy = (margin > 0).float()\n self.stats[\"rewards/accuracies\"].append(gather_mean(accuracy.mean()))\n\n # Log EOS token statistics\n model_eos = (model_data[\"input_ids\"][:, context_length:] == self.processing_class.eos_token_id).any(dim=1)\n ref_eos = (ref_data[\"input_ids\"][:, context_length:] == self.processing_class.eos_token_id).any(dim=1)\n self.stats[\"val/model_contain_eos_token\"].append(gather_mean(model_eos.float()))\n self.stats[\"val/ref_contain_eos_token\"].append(gather_mean(ref_eos.float()))\n\n # Log alpha and beta\n self.stats[\"alpha\"].append(self.alpha)\n self.stats[\"beta\"].append(self.beta)\n\n def training_step(\n self, model: nn.Module, inputs: dict[str, torch.Tensor | Any], num_items_in_batch: int | None = None\n ) -> torch.Tensor:\n model.train()\n\n # Apply chat template and tokenize the input\n batch_size = len(next(iter(inputs.values())))\n prompts = inputs[\"prompt\"]\n inputs = [{k: v[i] for k, v in inputs.items()} for i in range(batch_size)]\n inputs = [maybe_apply_chat_template(x, self.processing_class) for x in inputs]\n inputs = [self.tokenize_row(x, self.model.config.is_encoder_decoder, self.processing_class) for x in inputs]\n inputs = self.data_collator(inputs)\n\n # need the prompt_ only\n inputs = self._prepare_inputs(inputs)\n context_length = inputs[\"prompt_input_ids\"].shape[1]\n prompts = {\n \"input_ids\": inputs[\"prompt_input_ids\"],\n \"attention_mask\": inputs[\"prompt_attention_mask\"],\n \"raw\": prompts,\n }\n del inputs\n\n # Sample completions from both the model and the reference model\n model_output, ref_output = self._generate_completions(prompts, model)\n\n # Process model completions\n model_data, ref_data = self._process_completions(model_output, ref_output, prompts)\n\n # Compute rewards\n if self.reward_funcs is not None:\n model_scores, ref_scores = self._compute_rewards(model_data, ref_data, context_length)\n chosen_mask = model_scores >= ref_scores\n else:\n model_scores, ref_scores = None, None\n chosen_mask = self._compute_judge(model_data, ref_data, context_length)\n\n # Compute logprobs\n model_logprobs_model_data, model_logprobs_ref_data, ref_logprobs_ref_data, ref_logprobs_model_data = (\n self._compute_logprobs(model, model_data, ref_data, context_length)\n )\n\n # Compute loss\n loss, dpo_losses, xpo_losses = self._compute_losses(\n model_logprobs_model_data,\n model_logprobs_ref_data,\n ref_logprobs_ref_data,\n ref_logprobs_model_data,\n chosen_mask,\n )\n\n # Log everything\n self._log_statistics(\n model_data,\n ref_data,\n model_logprobs_model_data.detach(),\n model_logprobs_ref_data.detach(),\n ref_logprobs_ref_data,\n ref_logprobs_model_data,\n chosen_mask,\n dpo_losses.detach(),\n xpo_losses.detach(),\n context_length,\n model_scores,\n ref_scores,\n )\n\n if (\n self.args.torch_empty_cache_steps is not None\n and self.state.global_step % self.args.torch_empty_cache_steps == 0\n ):\n empty_cache()\n\n kwargs = {}\n # For LOMO optimizers you need to explicitly use the learning rate\n if self.args.optim in [OptimizerNames.LOMO, OptimizerNames.ADALOMO]:\n kwargs[\"learning_rate\"] = self._get_learning_rate()\n\n if self.args.n_gpu > 1:\n loss = loss.mean() # mean() to average on multi-gpu parallel training\n\n self.accelerator.backward(loss, **kwargs)\n\n return loss.detach() / self.args.gradient_accumulation_steps\n" }, { "path": "trl/extras/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n" }, { "path": "trl/extras/dataset_formatting.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\nimport datasets\nfrom datasets import Value\nfrom packaging.version import Version\n\n\nif Version(datasets.__version__) >= Version(\"4.0.0\"):\n from datasets import List\n\n FORMAT_MAPPING = {\n \"chatml\": List({\"content\": Value(dtype=\"string\", id=None), \"role\": Value(dtype=\"string\", id=None)}),\n \"instruction\": {\"completion\": Value(dtype=\"string\", id=None), \"prompt\": Value(dtype=\"string\", id=None)},\n }\nelse:\n FORMAT_MAPPING = {\n \"chatml\": [{\"content\": Value(dtype=\"string\", id=None), \"role\": Value(dtype=\"string\", id=None)}],\n \"instruction\": {\"completion\": Value(dtype=\"string\", id=None), \"prompt\": Value(dtype=\"string\", id=None)},\n }\n" }, { "path": "trl/extras/profiling.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport functools\nimport time\nfrom collections.abc import Callable\n\nfrom transformers import Trainer\nfrom transformers.integrations import is_mlflow_available, is_wandb_available\n\n\nif is_wandb_available():\n import wandb\n\nif is_mlflow_available():\n import mlflow\n\n\nclass ProfilingContext:\n \"\"\"\n Context manager for profiling code blocks with configurable logging.\n\n This class handles timing of code execution and logging metrics to various backends (Weights & Biases, MLflow)\n without being coupled to the Trainer class.\n\n Args:\n name (`str`):\n Name of the profiling context. Used in the metric name.\n report_to (`list` of `str`):\n List of integrations to report metrics to (e.g., [\"wandb\", \"mlflow\"]).\n is_main_process (`bool`, *optional*, defaults to `True`):\n Whether this is the main process in distributed training. Metrics are only logged from the main process.\n step (`int` or `None`, *optional*):\n Training step to associate with the logged metrics.\n metric_prefix (`str`, *optional*, defaults to `\"profiling/Time taken\"`):\n Prefix for the metric name in logs.\n\n Example:\n ```python\n # Direct usage\n from trl.extras.profiling import ProfilingContext\n\n with ProfilingContext(\n name=\"MyClass.expensive_operation\",\n report_to=[\"wandb\"],\n is_main_process=True,\n step=100,\n ):\n # Code to profile\n result = expensive_computation()\n\n # With Trainer (backwards compatible via profiling_context function)\n from transformers import Trainer\n from trl.extras.profiling import profiling_context\n\n\n class MyTrainer(Trainer):\n def some_method(self):\n with profiling_context(self, \"matrix_multiplication\"):\n result = matrix_multiply()\n ```\n \"\"\"\n\n def __init__(\n self,\n name: str,\n report_to: list[str],\n is_main_process: bool = True,\n step: int | None = None,\n metric_prefix: str = \"profiling/Time taken\",\n ):\n self.name = name\n self.report_to = report_to\n self.is_main_process = is_main_process\n self.step = step\n self.metric_prefix = metric_prefix\n self._start_time = None\n\n def __enter__(self):\n \"\"\"Start timing when entering the context.\"\"\"\n self._start_time = time.perf_counter()\n return self\n\n def __exit__(self, exc_type, exc_val, exc_tb):\n \"\"\"Stop timing and log metrics when exiting the context.\"\"\"\n if self._start_time is not None:\n duration = time.perf_counter() - self._start_time\n self._log_metrics(duration)\n return False\n\n def _log_metrics(self, duration: float) -> None:\n \"\"\"\n Log profiling metrics to configured backends.\n\n Args:\n duration (`float`):\n Execution time in seconds.\n \"\"\"\n if not self.is_main_process:\n return\n\n metric_name = f\"{self.metric_prefix}: {self.name}\"\n metrics = {metric_name: duration}\n\n # Log to Weights & Biases if configured\n if \"wandb\" in self.report_to and is_wandb_available() and wandb.run is not None:\n wandb.log(metrics)\n\n # Log to MLflow if configured\n if \"mlflow\" in self.report_to and is_mlflow_available() and mlflow.active_run() is not None:\n mlflow.log_metrics(metrics, step=self.step)\n\n\ndef profiling_context(trainer: Trainer, name: str) -> ProfilingContext:\n \"\"\"\n Factory function to create a ProfilingContext from a Trainer instance.\n\n This function maintains backwards compatibility with existing code while using the decoupled ProfilingContext class\n internally.\n\n Args:\n trainer (`~transformers.Trainer`):\n Trainer object containing configuration for logging.\n name (`str`):\n Name of the block to be profiled. Will be prefixed with the trainer class name.\n\n Returns:\n `ProfilingContext`: A configured profiling context manager.\n\n Example:\n ```python\n from transformers import Trainer\n from trl.extras.profiling import profiling_context\n\n\n class MyTrainer(Trainer):\n def some_method(self):\n A = np.random.rand(1000, 1000)\n B = np.random.rand(1000, 1000)\n with profiling_context(self, \"matrix_multiplication\"):\n # Code to profile: simulate a computationally expensive operation\n result = A @ B # Matrix multiplication\n ```\n \"\"\"\n context_name = f\"{trainer.__class__.__name__}.{name}\"\n step = trainer.state.global_step\n\n return ProfilingContext(\n name=context_name,\n report_to=trainer.args.report_to,\n is_main_process=trainer.accelerator.is_main_process,\n step=step,\n )\n\n\ndef profiling_decorator(func: Callable) -> Callable:\n \"\"\"\n Decorator to profile a function and log execution time using [`extras.profiling.profiling_context`].\n\n This decorator works with methods that have access to a trainer instance (typically as `self`). For non-Trainer\n objects that have an `accelerator` attribute, it will use that for logging configuration.\n\n Args:\n func (`Callable`):\n Function to be profiled.\n\n Returns:\n `Callable`: Wrapped function that profiles execution time.\n\n Example:\n ```python\n from transformers import Trainer\n from trl.extras.profiling import profiling_decorator\n\n\n class MyTrainer(Trainer):\n @profiling_decorator\n def some_method(self):\n A = np.random.rand(1000, 1000)\n B = np.random.rand(1000, 1000)\n # Code to profile: simulate a computationally expensive operation\n result = A @ B\n ```\n \"\"\"\n\n @functools.wraps(func)\n def wrapper(self, *args, **kwargs):\n # Check if self is a Trainer-like object with required attributes\n if hasattr(self, \"state\") and hasattr(self, \"args\"):\n with profiling_context(self, func.__name__):\n return func(self, *args, **kwargs)\n # For non-Trainer objects (e.g., VLLMGeneration), use ProfilingContext directly\n elif hasattr(self, \"accelerator\"):\n context_name = f\"{self.__class__.__name__}.{func.__name__}\"\n with ProfilingContext(\n name=context_name,\n report_to=[], # No reporting for non-Trainer objects without args\n is_main_process=self.accelerator.is_main_process,\n step=None,\n ):\n return func(self, *args, **kwargs)\n else:\n # No profiling available, just run the function\n return func(self, *args, **kwargs)\n\n return wrapper\n" }, { "path": "trl/generation/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\"\"\"Generation backends for TRL trainers.\"\"\"\n\nfrom ..import_utils import is_vllm_available\n\n\n__all__ = []\n\nif is_vllm_available():\n from .vllm_generation import VLLMGeneration\n\n __all__.append(\"VLLMGeneration\")\n" }, { "path": "trl/generation/vllm_client.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport atexit\nimport base64\nimport copy\nimport logging\nimport socket\nimport time\nfrom io import BytesIO\nfrom urllib.parse import urlparse\n\nimport torch\nimport torch.distributed.distributed_c10d as c10d\nfrom requests.adapters import HTTPAdapter\nfrom torch import nn\nfrom transformers import is_torch_xpu_available\nfrom transformers.utils import get_json_schema\nfrom urllib3.util.retry import Retry\n\nfrom ..import_utils import is_requests_available, is_vllm_ascend_available, is_vllm_available\n\n\nif is_requests_available():\n import requests\n from requests import ConnectionError\n\n\nif is_vllm_available():\n from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator\n from vllm.distributed.utils import StatelessProcessGroup\n\n if is_vllm_ascend_available():\n from vllm_ascend.distributed.device_communicators.pyhccl import PyHcclCommunicator as PyNcclCommunicator\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef pil_to_base64(image):\n buffer = BytesIO()\n image.save(buffer, format=\"PNG\")\n img_bytes = buffer.getvalue()\n return base64.b64encode(img_bytes).decode(\"utf-8\")\n\n\nclass VLLMClient:\n \"\"\"\n A client class to interact with a vLLM server.\n\n This class provides methods to generate completions, initialize and manage weight update groups, and update model\n weights in a distributed setting. Before using it, start the vLLM server with `trl vllm-serve`.\n\n Args:\n base_url (`str`, *optional*):\n Base URL for the vLLM server (e.g., `\"http://localhost:8000\"`). If provided, `host` and `server_port` are\n ignored.\n host (`str`, *optional*, defaults to `\"0.0.0.0\"`):\n IP address of the vLLM server. Ignored if `base_url` is provided.\n server_port (`int`, *optional*, defaults to `8000`):\n Port number of the vLLM server. Ignored if `base_url` is provided.\n group_port (`int`, *optional*, defaults to `51216`):\n Port number for the weight update group.\n connection_timeout (`float`, *optional*, defaults to `0.0`):\n Total timeout duration in seconds to wait for the server to be up. If the server is not up after the\n timeout, a `ConnectionError` is raised.\n\n Examples:\n Run the vLLM server with the model `Qwen/Qwen2.5-7B`:\n\n ```\n $ trl vllm-serve --model Qwen/Qwen2.5-7B\n ...\n INFO: Application startup complete.\n INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)\n ```\n\n Use the client to generate completions and update model weights:\n\n ```python\n >>> from trl.generation.vllm_client import VLLMClient\n\n >>> client = VLLMClient()\n >>> client.generate([\"Hello, AI!\", \"Tell me a joke\"])\n {'prompt_ids': [[9707, 11, 15235, 0],\n [40451, 752, 264, 21646]],\n 'completion_ids': [[2980, 498, 1492, 752, 448, 264, 13027, 8645, 30, 358, 2776, 4460, 311, 3270, 264, 2025],\n [911, 98072, 2142, 624, 45, 51426, 2142, 374, 279, 16396, 429, 4302, 702, 36988, 7290, 476]],\n 'logprobs': [[[-1.6612], [-0.0081], [-1.5189], [-0.0123], [-1.2045], [-0.6227], [-2.9791], [-2.8387], [-0.1267], [-0.0366], [-2.6528], [-0.3197], [-0.0001], [-1.8174], [-0.0251], [-1.473]],\n [[-0.018], [-10.7331], [-0.1605], [-0.891], [-3.7945], [-0.0127], [-0.3073], [-1.1648], [-1.8025], [-0.409], [-0.0256], [-1.6127], [-2.2935], [-4.1785], [-0.6531], [-0.2629]]],\n 'logprob_token_ids': [[[2980], [498], [1492], [752], [448], [264], [13027], [8645], [30], [358], [2776], [4460], [311], [3270], [264], [2025]],\n [[911], [98072], [2142], [624], [45], [51426], [2142], [374], [279], [16396], [429], [4302], [702], [36988], [7290], [476]]]}\n\n >>> from transformers import AutoModelForCausalLM\n\n >>> model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2.5-7B\", device_map=\"cuda\")\n >>> client.init_communicator(device=\"cuda\")\n >>> client.update_model_params(model)\n ```\n\n There are several ways to initialize the client:\n\n ```python\n VLLMClient(base_url=\"http://localhost:8000\")\n VLLMClient(base_url=\"http://192.168.1.100:8000\")\n VLLMClient(host=\"localhost\", server_port=8000)\n VLLMClient(host=\"192.168.1.100\", server_port=8000)\n ```\n \"\"\"\n\n def __init__(\n self,\n base_url: str | None = None,\n host: str = \"0.0.0.0\",\n server_port: int = 8000,\n group_port: int = 51216,\n connection_timeout: float = 0.0,\n ):\n if not is_requests_available():\n raise ImportError(\"requests is not installed. Please install it with `pip install requests`.\")\n if not is_vllm_available():\n raise ImportError(\"vLLM is not installed. Please install it with `pip install trl[vllm]`.\")\n\n self.session = requests.Session()\n\n # Configure retries for HTTP requests made through this session.\n # This is not strictly required for correctness, but it helps make training more robust to rare, transient\n # failures (network hiccups, temporary 5xx errors, overloaded servers). Without this, such failures could cause\n # an otherwise healthy training run to fail.\n retry_strategy = Retry(\n total=5, # global cap on the total number of retries across all failure types\n connect=5, # retry connection-level failures (DNS issues, refused connections, etc)\n read=5, # retry failures while reading the response after the connection was successfully established\n status=3, # retry a limited number of times when we receive certain HTTP error responses from the server\n status_forcelist=[500, 502, 503], # only retry on server-side errors that are usually temporary\n backoff_factor=2, # exponential backoff between retries (2s, 4s, 8s, ...)\n allowed_methods=[\"POST\", \"GET\"], # allow POST as well, even though we're not sure it's safe here\n )\n\n adapter = HTTPAdapter(max_retries=retry_strategy)\n self.session.mount(\"http://\", adapter)\n self.session.mount(\"https://\", adapter)\n\n if base_url is not None:\n # Parse the base_url to extract host and port\n parsed_url = urlparse(base_url)\n self.host = socket.gethostbyname(parsed_url.hostname)\n scheme = parsed_url.scheme or \"http\"\n self.base_url = f\"{scheme}://{parsed_url.netloc}{parsed_url.path}\"\n else:\n self.host = host\n self.server_port = server_port\n self.base_url = f\"http://{self.host}:{self.server_port}\"\n self.group_port = group_port\n self.check_server(connection_timeout) # check server and fail after timeout\n\n def check_server(self, total_timeout: float = 0.0, retry_interval: float = 2.0):\n \"\"\"\n Check server availability with retries on failure, within a total timeout duration. If the server is not up\n after the total timeout duration, raise a `ConnectionError`.\n\n Args:\n retry_interval (`float`, *optional*, defaults to `2.0`):\n Interval in seconds between retries.\n total_timeout (`float`, *optional*, defaults to `0.0`):\n Total timeout duration in seconds.\n \"\"\"\n url = f\"{self.base_url}/health/\"\n start_time = time.time() # Record the start time\n\n while True:\n try:\n response = requests.get(url)\n except requests.exceptions.RequestException as exc:\n # Check if the total timeout duration has passed\n elapsed_time = time.time() - start_time\n if elapsed_time >= total_timeout:\n raise ConnectionError(\n f\"The vLLM server can't be reached at {self.base_url} after {total_timeout} seconds. Make \"\n \"sure the server is running by running `trl vllm-serve`.\"\n ) from exc\n else:\n if response.status_code == 200:\n if \"X-Forwarded-For\" in response.headers:\n self.host = response.headers[\"X-Forwarded-For\"]\n logger.info(\"Server is up!\")\n return None\n\n # Retry logic: wait before trying again\n logger.info(f\"Server is not up yet. Retrying in {retry_interval} seconds...\")\n time.sleep(retry_interval)\n\n def generate(\n self,\n prompts: list[str] | list[list[int]],\n images: list | None = None,\n n: int = 1,\n repetition_penalty: float = 1.0,\n temperature: float = 1.0,\n top_p: float = 1.0,\n top_k: int = 0,\n min_p: float = 0.0,\n max_tokens: int = 16,\n logprobs: int | None = 0,\n structured_outputs_regex: str | None = None,\n generation_kwargs: dict | None = None,\n ) -> dict[str, list[list[int]]]:\n \"\"\"\n Generates model completions for the provided prompts.\n\n Args:\n prompts (`list[str]` or `list[list[int]]`):\n List of text prompts or list of token ID lists for which the model will generate completions.\n images (`list[list[PIL.Image] | None]`, *optional*):\n List of image lists for VLM support. Each element is a list of PIL images for the corresponding prompt,\n or `None` if no images for that prompt.\n n (`int`, *optional*, defaults to `1`):\n Number of completions to generate for each prompt.\n repetition_penalty (`float`, *optional*, defaults to `1.0`):\n Parameter for repetition penalty. 1.0 means no penalty.\n temperature (`float`, *optional*, defaults to `1.0`):\n Temperature parameter for sampling. Higher values increase diversity.\n top_p (`float`, *optional*, defaults to `1.0`):\n Top-p sampling parameter.`1.0` means no truncation.\n top_k (`int`, *optional*, defaults to `0`):\n Top-k sampling parameter. `0` means no truncation.\n min_p (`float`, *optional*, defaults to `0.0`):\n Minimum probability for sampling.\n max_tokens (`int`, *optional*, defaults to `16`):\n Maximum number of tokens to generate for each prompt.\n logprobs (`int` or `None`, *optional*, defaults to `0`):\n Number of top logprobs to return per token. When 0, only the sampled token's logprob is returned. When\n N>0, returns up to N+1 logprobs sorted by descending probability, because vLLM always includes the\n sampled token's logprob (which may fall outside the top-N).\n structured_outputs_regex (`str`, *optional*):\n Regular expression to guide the decoding process.\n generation_kwargs (`dict`, *optional*):\n Additional generation parameters to pass to the vLLM `SamplingParams`. This can include parameters like\n `seed`, `frequency_penalty`, etc. If it contains keys that conflict with the other parameters, they\n will override them.\n\n Returns:\n `dict` with keys:\n - `prompt_ids` (`list[list[int]]`):\n List of lists of token IDs representing the tokenized input prompts.\n - `completion_ids` (`list[list[int]]`):\n List of lists of token IDs representing the model-generated completions for each prompt.\n - `logprobs` (`list[list[list[float]]]`):\n Per-token logprobs of shape (num_sequences, seq_len, num_logprobs), sorted by descending\n probability.\n - `logprob_token_ids` (`list[list[list[int]]]`):\n Token IDs corresponding to each logprob, same shape as `logprobs`.\n \"\"\"\n url = f\"{self.base_url}/generate/\"\n\n # Convert PIL images to base64 strings. Each element is a list of images for the corresponding prompt,\n # or None if no images for that prompt.\n if images:\n images = [\n [pil_to_base64(img) for img in img_list] if img_list is not None else None for img_list in images\n ]\n\n response = self.session.post(\n url,\n json={\n \"prompts\": prompts,\n \"images\": images,\n \"n\": n,\n \"repetition_penalty\": repetition_penalty,\n \"temperature\": temperature,\n \"top_p\": top_p,\n \"top_k\": top_k,\n \"min_p\": min_p,\n \"max_tokens\": max_tokens,\n \"logprobs\": logprobs,\n \"structured_outputs_regex\": structured_outputs_regex,\n \"generation_kwargs\": generation_kwargs or {},\n },\n )\n if response.status_code == 200:\n json_response = response.json()\n return {\n \"prompt_ids\": json_response[\"prompt_ids\"],\n \"completion_ids\": json_response[\"completion_ids\"],\n \"logprobs\": json_response[\"logprobs\"],\n \"logprob_token_ids\": json_response[\"logprob_token_ids\"],\n }\n else:\n raise Exception(f\"Request failed: {response.status_code}, {response.text}\")\n\n def chat(\n self,\n messages: list[list[dict]],\n n: int = 1,\n repetition_penalty: float = 1.0,\n temperature: float = 1.0,\n top_p: float = 1.0,\n top_k: int = 0,\n min_p: float = 0.0,\n max_tokens: int = 16,\n logprobs: int | None = 0,\n structured_outputs_regex: str | None = None,\n generation_kwargs: dict | None = None,\n chat_template_kwargs: dict | None = None,\n tools: list | None = None,\n chat_template: str | None = None,\n ) -> dict[str, list[list[int]]]:\n \"\"\"\n Generates model completions for the provided chat messages.\n\n Args:\n messages (`list[list[dict]]`):\n List of message lists for which the model will generate completions. Each message is a dictionary with\n keys like \"role\" and \"content\".\n n (`int`, *optional*, defaults to `1`):\n Number of completions to generate for each message list.\n repetition_penalty (`float`, *optional*, defaults to `1.0`):\n Parameter for repetition penalty. 1.0 means no penalty.\n temperature (`float`, *optional*, defaults to `1.0`):\n Temperature parameter for sampling. Higher values increase diversity.\n top_p (`float`, *optional*, defaults to `1.0`):\n Top-p sampling parameter.`1.0` means no truncation.\n top_k (`int`, *optional*, defaults to `0`):\n Top-k sampling parameter. `0` means no truncation.\n min_p (`float`, *optional*, defaults to `0.0`):\n Minimum probability for sampling.\n max_tokens (`int`, *optional*, defaults to `16`):\n Maximum number of tokens to generate for each message list.\n logprobs (`int` or `None`, *optional*, defaults to `0`):\n Number of top logprobs to return per token. When 0, only the sampled token's logprob is returned. When\n N>0, returns up to N+1 logprobs sorted by descending probability, because vLLM always includes the\n sampled token's logprob (which may fall outside the top-N).\n structured_outputs_regex (`str`, *optional*):\n Regular expression to guide the decoding process.\n generation_kwargs (`dict`, *optional*):\n Additional generation parameters to pass to the vLLM `SamplingParams`. This can include parameters like\n `seed`, `frequency_penalty`, etc. If it contains keys that conflict with the other parameters, they\n will override them.\n chat_template_kwargs (`dict`, *optional*):\n Additional keyword arguments to customize the chat template used by the model.\n tools (`list[dict | Callable]`, *optional*):\n List of tool functions available for tool calling during chat generation.\n chat_template (`str`, *optional*):\n Template to use for structuring the chat. If not provided, the model's default chat template will be\n used.\n\n Returns:\n `dict` with keys:\n - `prompt_ids` (`list[list[int]]`):\n List of lists of token IDs representing the tokenized input messages.\n - `completion_ids` (`list[list[int]]`):\n List of lists of token IDs representing the model-generated completions for each message list.\n - `logprobs` (`list[list[list[float]]]`):\n Per-token logprobs of shape (num_sequences, seq_len, num_logprobs), sorted by descending\n probability.\n - `logprob_token_ids` (`list[list[list[int]]]`):\n Token IDs corresponding to each logprob, same shape as `logprobs`.\n \"\"\"\n if chat_template is not None:\n raise NotImplementedError(\"Custom chat templates are not yet implemented in VLLMClient.chat().\")\n\n url = f\"{self.base_url}/chat/\"\n\n # Convert PIL images to base64 strings\n messages = copy.deepcopy(messages) # avoid modifying the original messages\n for message_list in messages:\n for message in message_list:\n if isinstance(message[\"content\"], list):\n for part in message[\"content\"]:\n if part[\"type\"] == \"image_pil\":\n part[\"image_pil\"] = pil_to_base64(part[\"image_pil\"])\n\n if isinstance(tools, list) and len(tools) > 0:\n tools = [get_json_schema(tool) if callable(tool) else tool for tool in tools]\n\n response = self.session.post(\n url,\n json={\n \"messages\": messages,\n \"n\": n,\n \"repetition_penalty\": repetition_penalty,\n \"temperature\": temperature,\n \"top_p\": top_p,\n \"top_k\": top_k,\n \"min_p\": min_p,\n \"max_tokens\": max_tokens,\n \"logprobs\": logprobs,\n \"structured_outputs_regex\": structured_outputs_regex,\n \"generation_kwargs\": generation_kwargs or {},\n \"chat_template_kwargs\": chat_template_kwargs or {},\n \"tools\": tools,\n },\n )\n if response.status_code == 200:\n json_response = response.json()\n return {\n \"prompt_ids\": json_response[\"prompt_ids\"],\n \"completion_ids\": json_response[\"completion_ids\"],\n \"logprobs\": json_response[\"logprobs\"],\n \"logprob_token_ids\": json_response[\"logprob_token_ids\"],\n }\n else:\n raise Exception(f\"Request failed: {response.status_code}, {response.text}\")\n\n def init_communicator(self, device: torch.device | str | int = 0):\n \"\"\"\n Initializes the weight update group in a distributed setup for model synchronization.\n\n Args:\n device (`torch.device`, `str`, or `int`, *optional*, defaults to `0`):\n Device of trainer main process. It's the device that will be used for the weights synchronization. Can\n be a `torch.device` object, a string like `'cuda:0'`, or an integer device index.\n \"\"\"\n # Get the world size from the server\n url = f\"{self.base_url}/get_world_size/\"\n response = requests.get(url)\n if response.status_code == 200:\n vllm_world_size = response.json()[\"world_size\"]\n else:\n raise Exception(f\"Request failed: {response.status_code}, {response.text}\")\n\n world_size = vllm_world_size + 1 # add the client to the world\n self.rank = vllm_world_size # the client's rank is the last process\n\n # Initialize weight update group\n url = f\"{self.base_url}/init_communicator/\"\n # Will simplify it after torch xpu 2.9 support get uuid.\n if is_torch_xpu_available():\n if hasattr(torch.xpu.get_device_properties(device), \"uuid\"):\n client_device_uuid = str(torch.xpu.get_device_properties(device).uuid)\n else:\n client_device_uuid = \"42\"\n else:\n client_device_uuid = str(torch.cuda.get_device_properties(device).uuid)\n\n # Set the weight update group's host to \"0.0.0.0\" so that\n # clients from different IPs can send updated weights\n response = self.session.post(\n url,\n json={\n \"host\": \"0.0.0.0\",\n \"port\": self.group_port,\n \"world_size\": world_size,\n \"client_device_uuid\": client_device_uuid,\n },\n )\n if response.status_code != 200:\n raise Exception(f\"Request failed: {response.status_code}, {response.text}\")\n\n # Brief delay to allow server initialization. While not strictly required (client socket will retry on\n # connection failure), this prevents log warnings like:\n # [W416 23:24:57.460001114 socket.cpp:204] [c10d] The hostname of the client socket cannot be retrieved. err=-3\n time.sleep(0.1)\n\n # Set up the communication group for weight broadcasting\n if is_torch_xpu_available():\n store = torch.distributed.TCPStore(\n host_name=self.host, port=self.group_port, world_size=world_size, is_master=(self.rank == 0)\n )\n prefixed_store = c10d.PrefixStore(\"client2server\", store)\n xccl_options = c10d.ProcessGroupXCCL.Options()\n pg = c10d.ProcessGroupXCCL(\n store=prefixed_store,\n rank=self.rank,\n size=world_size,\n options=xccl_options,\n )\n self.communicator = pg\n else:\n pg = StatelessProcessGroup.create(\n host=self.host, port=self.group_port, rank=self.rank, world_size=world_size\n )\n self.communicator = PyNcclCommunicator(pg, device=device)\n\n # When the client object is deleted, close the weight update group\n atexit.register(self.close_communicator)\n\n def update_named_param(self, name: str, weights: torch.Tensor):\n \"\"\"\n Updates a specific named parameter in the model and broadcasts it to other processes.\n\n Args:\n name (`str`):\n Name of the layer whose weights are being updated.\n weights (`torch.Tensor`):\n Tensor containing the updated weights.\n \"\"\"\n dtype, shape = str(weights.dtype), tuple(weights.shape)\n url = f\"{self.base_url}/update_named_param/\"\n response = self.session.post(url, json={\"name\": name, \"dtype\": dtype, \"shape\": shape})\n if response.status_code != 200:\n raise Exception(f\"Request failed: {response.status_code}, {response.text}\")\n\n if is_torch_xpu_available():\n # Use XCCL to broadcast the updated weights from the client (src) to all workers.\n self.communicator.broadcast(weights, root=self.rank)\n self.communicator.barrier()\n else:\n # Use NCCL to broadcast the updated weights from the client (src) to all workers.\n self.communicator.broadcast(weights, src=self.rank)\n self.communicator.group.barrier()\n\n def update_model_params(self, model: nn.Module):\n \"\"\"\n Updates all parameters of the given model by calling `update_named_param` for each parameter in the model.\n\n Args:\n model (`nn.Module`):\n Model whose parameters (weights/biases) are to be updated.\n \"\"\"\n for name, param in model.named_parameters():\n # Update each parameter individually\n self.update_named_param(name, param.data)\n\n def reset_prefix_cache(self):\n \"\"\"\n Resets the prefix cache for the model.\n \"\"\"\n url = f\"{self.base_url}/reset_prefix_cache/\"\n response = self.session.post(url)\n if response.status_code != 200:\n raise Exception(f\"Request failed: {response.status_code}, {response.text}\")\n\n def close_communicator(self):\n \"\"\"\n Closes the weight update group and cleans up the communication group.\n \"\"\"\n url = f\"{self.base_url}/close_communicator/\"\n\n try:\n response = self.session.post(url)\n except ConnectionError:\n # The server might be already down, so we don't need to close the communicator\n pass\n else:\n if response.status_code != 200:\n raise Exception(f\"Request failed: {response.status_code}, {response.text}\")\n\n if self.communicator is not None:\n self.communicator = None\n\n\n# Example usage\nif __name__ == \"__main__\":\n from vllm import SamplingParams\n\n device = \"xpu\" if is_torch_xpu_available() else \"cuda\"\n client = VLLMClient()\n client.init_communicator(device=device)\n\n # Generate completions\n responses = client.generate([\"Hello, AI!\", \"Tell me a joke\"], n=4, max_tokens=32, sampling_params=SamplingParams())\n print(\"Responses:\", responses) # noqa\n\n # Update model weights\n from transformers import AutoModelForCausalLM\n\n model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2.5-7B\").to(device)\n client.update_model_params(model)\n" }, { "path": "trl/generation/vllm_generation.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\"\"\"vLLM-based generation backend for TRL trainers.\"\"\"\n\nimport logging\nimport math\nimport os\nfrom contextlib import nullcontext\nfrom typing import TYPE_CHECKING\n\nimport torch\nfrom accelerate.utils import broadcast_object_list, gather_object, is_peft_model\nfrom packaging.version import Version\nfrom torch import nn\nfrom torch.distributed.fsdp import FullyShardedDataParallel as FSDP\nfrom transformers import PreTrainedModel, PreTrainedTokenizerBase, ProcessorMixin, is_bitsandbytes_available\nfrom transformers.utils import is_torch_mlu_available, is_torch_npu_available, is_torch_xpu_available\n\nfrom ..extras.profiling import ProfilingContext\nfrom ..import_utils import is_vllm_available\nfrom ..trainer.utils import ensure_master_addr_port\nfrom .vllm_client import VLLMClient\n\n\nif is_vllm_available():\n from vllm import LLM, RequestOutput, SamplingParams\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef empty_cache() -> None:\n \"\"\"Empties the cache of the available torch device.\n\n This function checks for the availability of different torch devices (XPU, MLU, NPU, CUDA) and empties the cache of\n the first available device it finds.\n\n If none of the specific devices are available, it defaults to emptying the CUDA cache.\n \"\"\"\n if is_torch_xpu_available():\n torch.xpu.empty_cache()\n elif is_torch_mlu_available():\n torch.mlu.empty_cache()\n elif is_torch_npu_available():\n torch.npu.empty_cache()\n else:\n torch.cuda.empty_cache()\n\n\ndef extract_logprobs(all_outputs: list[\"RequestOutput\"]):\n \"\"\"\n Extract logprobs and token IDs from vLLM generation outputs.\n\n Returns logprobs and token IDs sorted by rank (most probable first). Each returned list has shape (num_sequences,\n seq_len, num_logprobs), where num_logprobs is determined by the `logprobs` parameter passed to vLLM (1 when\n `logprobs=0`, up to N+1 when `logprobs=N`). NaN logprob values are replaced with `None`.\n\n Args:\n all_outputs (list of `RequestOutput`):\n List of vLLM `RequestOutput` objects from generation.\n\n Returns:\n Tuple of (logprobs, logprob_token_ids), each of shape (num_sequences, seq_len, num_logprobs).\n \"\"\"\n all_logprobs = []\n all_token_ids = []\n for outputs in all_outputs:\n for output in outputs.outputs:\n if output.logprobs is None:\n return None, None\n seq_logprobs = []\n seq_token_ids = []\n for lp in output.logprobs:\n sorted_items = sorted(lp.items(), key=lambda x: x[1].rank)\n seq_token_ids.append([token_id for token_id, _ in sorted_items])\n seq_logprobs.append([None if math.isnan(item.logprob) else item.logprob for _, item in sorted_items])\n all_logprobs.append(seq_logprobs)\n all_token_ids.append(seq_token_ids)\n return all_logprobs, all_token_ids\n\n\nif TYPE_CHECKING:\n from accelerate import Accelerator\n from peft import PeftModel\n\n\nif is_bitsandbytes_available():\n import bitsandbytes as bnb\n\n\nclass VLLMGeneration:\n \"\"\"Handles vLLM-based generation for trainers.\n\n Extracts all vLLM-specific logic (initialization, generation, weight sync) from trainers into a separate, testable\n class.\n\n Args:\n model ([`~transformers.PreTrainedModel`] or [`~peft.PeftModel`]):\n Model to use for generation.\n accelerator ([`~accelerate.Accelerator`]):\n Accelerator for distributed training.\n is_fsdp_enabled (`bool`):\n Whether FSDP is enabled.\n processing_class ([`~transformers.PreTrainedTokenizerBase`] or [`~transformers.ProcessorMixin`]):\n Tokenizer or processor for the model.\n\n > Parameters for vLLM:\n\n mode (`str`, *optional*, defaults to `\"colocate\"`):\n vLLM mode. Must be one of `\"colocate\"` or `\"server\"`.\n\n - `\"colocate\"`: vLLM will run in the same process and share the training GPUs. This avoids the need for a\n separate server but may cause resource contention with training.\n - `\"server\"`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM\n server is running (start with `trl vllm-serve`).\n\n structured_outputs_regex (`str`, *optional*):\n Regex for vLLM structured outputs. If `None` (default), structured outputs is disabled.\n\n > Parameters for \"server\" vLLM mode:\n\n server_base_url (`str`, *optional*):\n Base URL for the vLLM server (e.g., `\"http://localhost:8000\"`). If provided, `server_host` and\n `server_port` are ignored.\n server_host (`str`, *optional*, defaults to `\"0.0.0.0\"`):\n Host of the vLLM server to connect to. Ignored if `server_base_url` is provided.\n server_port (`int`, *optional*, defaults to `8000`):\n Port of the vLLM server to connect to. Ignored if `server_base_url` is provided.\n server_timeout (`float`, *optional*, defaults to `240.0`):\n Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the\n timeout, a `ConnectionError` is raised.\n group_port (`int`, *optional*, defaults to `51216`):\n Port number for the weight update group. This is used to communicate with the vLLM server. Unless the port\n is occupied, there is no need to change it.\n\n > Parameters for \"colocate\" vLLM mode:\n\n tensor_parallel_size (`int`, *optional*, defaults to `1`):\n The number of GPUs to use for distributed execution with tensor parallelism. This setting only applies when\n `mode` is set to `\"colocate\"`. If you are using `mode=\"server\"`, this parameter must be passed separately\n when launching the vLLM server via the `--vllm_tensor_parallel_size` flag.\n gpu_memory_utilization (`float`, *optional*, defaults to `0.9`):\n Ratio (between 0 and 1) of GPU memory to reserve for the model weights, activations, and KV cache. Higher\n values will increase the KV cache size and thus improve the model's throughput. However, if the value is\n too high, it may cause out-of- memory (OOM) errors. This setting only applies when `mode` is set to\n `\"colocate\"`. If you are using `mode=\"server\"`, this parameter must be passed separately when launching the\n vLLM server via the `--vllm_gpu_memory_utilization` flag.\n max_model_length (`int`, *optional*):\n Model context length (prompt and completion). Set it to at least the maximum prompt length in the dataset\n plus `max_completion_length`; if omitted, it is inferred from the model config.\n max_num_seqs (`int`, *optional*):\n Maximum number of sequences to process in parallel, effectively capping the batch size.\n enable_sleep_mode (`bool`, *optional*, defaults to `False`):\n Whether to enable sleep mode for the engine to offload weights/cache during the optimizer step. Keeps GPU\n memory usage low, but waking the engine adds hostโ€“device transfer latency.\n model_impl (`str`, *optional*, defaults to `\"auto\"`):\n Model implementation to use for vLLM.\n - \"auto\" will try to use the vLLM implementation, if it exists, and fall back to the Transformers\n implementation if no vLLM implementation is available.\n - \"vllm\" will use the vLLM model implementation.\n - \"transformers\" will use the Transformers model implementation.\n - \"terratorch\" will use the TerraTorch model implementation.\n\n > Parameters for generation:\n\n repetition_penalty (`float`, *optional*, defaults to `1.0`):\n Parameter for repetition penalty. It penalizes new tokens based on whether they appear in the prompt and\n the generated text so far. Values > 1 encourage the model to use new tokens, while values < 1 encourage the\n model to repeat tokens. Default `1.0` means no penalty.\n temperature(`float`, *optional*, defaults to `1.0`):\n Sampling temperature. It controls the randomness of the sampling. Lower values make the model more\n deterministic, while higher values make the model more random and increase diversity.\n top_p: (`float`, *optional*, defaults to `1.0`):\n Top-p sampling parameter. It controls the cumulative probability of the top tokens to consider. Defaults to\n `1.0` to consider all tokens.\n top_k (`int`, *optional*, defaults to `0`):\n Top-k sampling parameter. It controls the number of top tokens to consider. Defaults to `0` to consider all\n tokens.\n min_p (`float`, *optional*, defaults to `0.0`):\n Min-p sampling parameter. It represents the minimum probability for a token to be considered, relative to\n the probability of the most likely token. Default `0.0` means min-p is disabled.\n max_completion_length (`int`, *optional*, defaults to `16`):\n Maximum number of tokens to generate for each prompt.\n logprobs (`int` or `None`, *optional*, defaults to `0`):\n Number of top logprobs to return per token. When 0 (default), only the sampled token's logprob is returned\n (inner dimension = 1). When N>0, returns up to N+1 logprobs sorted by descending probability, because vLLM\n always includes the sampled token's logprob alongside the top-N (the sampled token may or may not already\n be in the top-N).\n generation_kwargs (`dict`, *optional*):\n Additional generation parameters to pass to the vLLM `SamplingParams`. This can include parameters like\n `seed`, `frequency_penalty`, etc. If it contains keys that conflict with the other parameters, they will\n override them.\n\n > Parameters for chat/tools:\n\n chat_template (`str`, *optional*):\n Template to use for structuring the chat. If not provided, the model's default chat template will be used.\n chat_template_kwargs (`dict`, *optional*):\n Additional keyword arguments to customize the chat template used by the model.\n tools (`list`, *optional*):\n Tools available for tool calling during chat generation.\n \"\"\"\n\n def __init__(\n self,\n model: \"PreTrainedModel | PeftModel\",\n accelerator: \"Accelerator\",\n is_fsdp_enabled: bool,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin,\n # vLLM configuration\n mode: str = \"colocate\",\n structured_outputs_regex: str | None = None,\n # Server mode configuration\n server_base_url: str | None = None,\n server_host: str = \"0.0.0.0\",\n server_port: int = 8000,\n server_timeout: float = 240.0,\n group_port: int = 51216,\n # Colocate mode configuration\n tensor_parallel_size: int = 1,\n gpu_memory_utilization: float = 0.9,\n max_model_length: int | None = None,\n max_num_seqs: int | None = None,\n enable_sleep_mode: bool = False,\n model_impl: str = \"auto\",\n # Generation configuration\n repetition_penalty: float = 1.0,\n temperature: float = 1.0,\n top_p: float = 1.0,\n top_k: int = 0,\n min_p: float = 0.0,\n max_completion_length: int = 16,\n logprobs: int | None = 0,\n generation_kwargs: dict | None = None,\n ):\n self.model = model\n self.accelerator = accelerator\n self.is_fsdp_enabled = is_fsdp_enabled\n self.processing_class = processing_class\n\n # vLLM configuration\n self.mode = mode\n self.structured_outputs_regex = structured_outputs_regex\n\n # Server mode configuration\n self.server_base_url = server_base_url\n self.server_host = server_host\n self.server_port = server_port\n self.group_port = group_port\n self.server_timeout = server_timeout\n\n # Colocate mode configuration\n self.tensor_parallel_size = tensor_parallel_size\n self.gpu_memory_utilization = gpu_memory_utilization\n self.max_model_length = max_model_length\n self.max_num_seqs = max_num_seqs\n self.enable_sleep_mode = enable_sleep_mode\n self.model_impl = model_impl\n\n # Generation configuration\n self.repetition_penalty = repetition_penalty\n self.temperature = temperature\n self.top_p = top_p\n self.top_k = top_k\n self.min_p = min_p\n self.max_completion_length = max_completion_length\n self.logprobs = logprobs\n self.generation_kwargs = generation_kwargs or {}\n\n self._init_vllm()\n\n def _init_vllm(self):\n \"\"\"Initialize vLLM in server or colocate mode.\"\"\"\n model = self.model\n accelerator = self.accelerator\n\n if not is_vllm_available():\n raise ImportError(\n \"vLLM is not available and `use_vllm` is set to True. Please install vLLM with \"\n \"`pip install trl[vllm]` to use it.\"\n )\n\n if self.mode == \"server\":\n if accelerator.is_main_process:\n if self.server_base_url is not None:\n base_url = self.server_base_url\n else:\n base_url = f\"http://{self.server_host}:{self.server_port}\"\n self.vllm_client = VLLMClient(\n base_url=base_url, group_port=self.group_port, connection_timeout=self.server_timeout\n )\n self.vllm_client.init_communicator(device=torch.cuda.current_device())\n\n elif self.mode == \"colocate\":\n # Make sure tensor_parallel_size group size evenly divides the world size - each group should have\n # the same number of ranks\n if not accelerator.num_processes % self.tensor_parallel_size == 0:\n raise ValueError(\n f\"tensor_parallel_size ({self.tensor_parallel_size}) must divide world size \"\n f\"({accelerator.num_processes}) evenly.\"\n )\n\n if self.tensor_parallel_size > 1:\n # Create subgroups of ranks for TP, each group with `tensor_parallel_size` ranks.\n # For example, if world_size=8 and tensor_parallel_size=2 โ†’ groups: [0,1], [2,3], [4,5], [6,7]\n self.tp_group, _ = torch.distributed.new_subgroups_by_enumeration(\n [\n list(range(i * self.tensor_parallel_size, (i + 1) * self.tensor_parallel_size))\n for i in range(accelerator.num_processes // self.tensor_parallel_size)\n ]\n )\n\n # vLLM requires the environment variables to be set for distributed training.\n os.environ[\"RANK\"] = str(accelerator.process_index)\n os.environ[\"LOCAL_RANK\"] = str(accelerator.local_process_index)\n os.environ[\"WORLD_SIZE\"] = str(accelerator.num_processes)\n # Ensure distributed rendezvous variables are set without colliding across concurrent runs\n ensure_master_addr_port()\n\n quantization = None\n if is_bitsandbytes_available():\n for _, module in model.named_modules():\n if isinstance(module, bnb.nn.Linear4bit):\n quantization = \"bitsandbytes\"\n break\n elif isinstance(module, bnb.nn.Linear8bitLt):\n raise ValueError(\"vLLM does not support in-flight 8-bit quantization.\")\n\n # Build LLM initialization kwargs\n self.llm = LLM(\n model=model.name_or_path,\n tensor_parallel_size=self.tensor_parallel_size,\n gpu_memory_utilization=self.gpu_memory_utilization,\n max_model_len=self.max_model_length,\n max_num_seqs=self.max_num_seqs,\n enable_sleep_mode=self.enable_sleep_mode,\n model_impl=self.model_impl,\n distributed_executor_backend=\"external_launcher\",\n # Feed identical seed for tp groups to ensure sampling results are the same across workers\n seed=accelerator.process_index // self.tensor_parallel_size,\n # Latest vLLM v1 memory profiler is misled by the high default value (i.e., 32768) - thinking there's not enough memory\n max_num_batched_tokens=4096,\n # Important so temperature scaling/logit tweaking affects the TIS log probs\n logprobs_mode=\"processed_logprobs\",\n quantization=quantization,\n )\n if self.enable_sleep_mode:\n self.llm.sleep(level=2)\n else:\n raise ValueError(f\"vllm_mode must be either 'server' or 'colocate', got '{self.mode}'.\")\n\n # When using vLLM, the main process is responsible for loading the model weights. This can cause process\n # desynchronization and seems to lead to DeepSpeed hanging during initialization. To prevent this, we\n # synchronize all processes after vLLM has been fully initialized.\n accelerator.wait_for_everyone()\n\n def _fix_param_name_to_vllm(self, name: str, extra_prefixes: list[str] | None = None) -> str:\n \"\"\"Fix parameter name for vLLM compatibility.\"\"\"\n extra_prefixes = extra_prefixes or []\n prefixes = [\"_checkpoint_wrapped_module.\"] + extra_prefixes\n for prefix in prefixes:\n name = name.replace(prefix, \"\")\n return name\n\n def _sync_fsdp1_params_to_vllm(self, module: nn.Module, prefix: str = \"\", visited: set[str] | None = None):\n \"\"\"Memory-efficient post-order traversal of FSDP modules to extract full parameters and sync with vLLM.\"\"\"\n # For FSDP1, we need to recurse into children and also use summon_full_params\n accelerator = self.accelerator\n\n if visited is None:\n visited = set()\n for child_name, child_module in module.named_children():\n child_prefix = f\"{prefix}.{child_name}\" if prefix else child_name\n self._sync_fsdp1_params_to_vllm(\n child_module, prefix=child_prefix, visited=visited\n ) # recurse into the child\n\n if isinstance(module, FSDP):\n with FSDP.summon_full_params(module, recurse=False, writeback=False):\n for param_name, param in module.named_parameters():\n full_name = f\"{prefix}.{param_name}\" if prefix else param_name\n full_name = self._fix_param_name_to_vllm(full_name, extra_prefixes=[\"_fsdp_wrapped_module.\"])\n\n if full_name in visited:\n continue # skip FSDP subtrees already traversed\n visited.add(full_name)\n\n if self.mode == \"server\" and accelerator.is_main_process:\n self.vllm_client.update_named_param(full_name, param.data)\n elif self.mode == \"colocate\":\n llm_model = self.llm.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(full_name, param.data)])\n\n def _sync_fsdp2_params_to_vllm(self, module: nn.Module):\n \"\"\"FSDP2-specific parameter synchronization.\"\"\"\n accelerator = self.accelerator\n\n # For FSDP2, module.state_dict() already covers all parameters, so no need for recursion\n for name, param in module.state_dict().items():\n # When using PEFT, we need to recover the original parameter name\n name = name.removeprefix(\"base_model.model.\").replace(\".base_layer\", \"\")\n # Skip PEFT layers: they don't exist in vLLM, and they are merged already.\n if is_peft_model(module) and module.prefix in name:\n continue\n # When module to save, remove its prefix and discard the original module\n if \"original_module\" in name:\n continue\n name = self._fix_param_name_to_vllm(name, extra_prefixes=[\"modules_to_save.default.\"])\n\n if param.is_cpu:\n param = param.to(torch.device(\"cuda\"))\n param = param.full_tensor()\n\n if self.mode == \"server\" and accelerator.is_main_process:\n self.vllm_client.update_named_param(name, param)\n elif self.mode == \"colocate\":\n llm_model = self.llm.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(name, param)])\n\n def sync_weights(self):\n \"\"\"Synchronize model weights to vLLM.\n\n Handles FSDP, DeepSpeed, PEFT weight synchronization.\n \"\"\"\n # Wake up vLLM weights before loading to ensure device memory is mapped. Without this, load_weights() writes to\n # freed/unmapped memory when sleep mode is active, which crashes on backends with strict physical memory\n # management (e.g., Ascend NPU). See https://github.com/huggingface/trl/issues/5142\n if self.mode == \"colocate\" and self.enable_sleep_mode:\n empty_cache() # required to avoid OOM in some cases\n self.llm.wake_up(tags=[\"weights\"])\n\n model = self.model\n accelerator = self.accelerator\n is_fsdp_enabled = self.is_fsdp_enabled\n\n # For DeepSpeed ZeRO-3 and FSDP, we need to gather all parameters before operations\n deepspeed_plugin = accelerator.state.deepspeed_plugin\n zero_stage_3 = deepspeed_plugin is not None and deepspeed_plugin.zero_stage == 3\n if zero_stage_3:\n import deepspeed\n\n gather_if_zero3 = deepspeed.zero.GatheredParameters\n else:\n gather_if_zero3 = nullcontext\n\n if is_peft_model(model):\n # With PEFT and FSDP/DeepSpeed ZeRO Stage 3, we must gather the full model at once before merging, as\n # merging adapters in a sharded manner is not supported.\n # TODO: does this work with FSDP?\n with gather_if_zero3(list(model.parameters())):\n model.merge_adapter()\n\n # Update vLLM weights while parameters are gathered\n if is_fsdp_enabled: # note if using FSDP, gather_if_zero3 is nullcontext\n # Update vLLM weights while parameters are gathered\n # For PEFT with FSDP we need to use the memory efficient post-order traversal\n fsdp_plugin = getattr(accelerator.state, \"fsdp_plugin\", None)\n fsdp_version = getattr(fsdp_plugin, \"fsdp_version\", 1) if fsdp_plugin else 1\n if fsdp_version == 1:\n self._sync_fsdp1_params_to_vllm(model) # use memory-efficient post-order traversal for FSDP\n elif fsdp_version == 2:\n self._sync_fsdp2_params_to_vllm(model)\n else:\n # DeepSpeed ZeRO-3 with PEFT\n for name, param in model.named_parameters():\n # When using PEFT, we need to recover the original parameter name\n name = name.removeprefix(\"base_model.model.\").replace(\".base_layer\", \"\")\n # Skip PEFT layers: they don't exist in vLLM, and they are merged already.\n if model.prefix in name:\n continue\n # When module to save, remove its prefix and discard the original module\n if \"original_module\" in name:\n continue\n name = self._fix_param_name_to_vllm(name, extra_prefixes=[\"modules_to_save.default.\"])\n\n if self.mode == \"server\" and accelerator.is_main_process:\n self.vllm_client.update_named_param(name, param.data)\n elif self.mode == \"colocate\":\n llm_model = self.llm.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(name, param.data)])\n # Unmerge adapters while parameters are still gathered\n model.unmerge_adapter()\n # Parameters will automatically be repartitioned when exiting the context\n else:\n # For non-PEFT models, simply gather (if needed) and update each parameter individually.\n if is_fsdp_enabled:\n fsdp_plugin = getattr(accelerator.state, \"fsdp_plugin\", None)\n fsdp_version = getattr(fsdp_plugin, \"fsdp_version\", 1) if fsdp_plugin else 1\n if fsdp_version == 1:\n self._sync_fsdp1_params_to_vllm(model) # use memory-efficient post-order traversal for FSDP\n elif fsdp_version == 2:\n self._sync_fsdp2_params_to_vllm(model)\n else:\n for name, param in model.named_parameters():\n name = self._fix_param_name_to_vllm(name)\n with gather_if_zero3([param]):\n if self.mode == \"server\" and accelerator.is_main_process:\n self.vllm_client.update_named_param(name, param.data)\n elif self.mode == \"colocate\":\n llm_model = self.llm.llm_engine.model_executor.driver_worker.model_runner.model\n llm_model.load_weights([(name, param.data)])\n\n # Reset cache on vLLM\n if self.mode == \"server\" and accelerator.is_main_process:\n self.vllm_client.reset_prefix_cache()\n elif self.mode == \"colocate\":\n self.llm.reset_prefix_cache()\n\n def generate(\n self,\n prompts: list[list[int]],\n images: list[list | None] | None,\n num_generations: int,\n profiler: ProfilingContext | None = None,\n ) -> tuple:\n \"\"\"Generate completions using vLLM.\n\n Args:\n prompts: List of token ID lists, one per prompt (already tokenized).\n images: Optional list of image lists for VLM support. Each element is a list of PIL images for the\n corresponding prompt, or `None` if no images for that prompt. `None` if no images at all.\n num_generations: Number of generations per prompt.\n profiler: Optional profiler for performance tracking.\n\n Returns:\n Tuple of (prompt_ids, completion_ids, logprobs, logprob_token_ids).\n\n - `prompt_ids`: `list[list[int]]` of shape `(batch_size, prompt_len)`.\n - `completion_ids`: `list[list[int]]` of shape `(batch_size, completion_len)`.\n - `logprobs`: `list[list[list[float | None]]]` of shape `(batch_size, completion_len, num_logprobs)`.\n - `logprob_token_ids`: `list[list[list[int]]]` of shape `(batch_size, completion_len, num_logprobs)`.\n\n `num_logprobs` is 1 when `logprobs=0`, or up to N+1 when `logprobs=N` (the sampled token is always included\n and may fall outside the top-N).\n \"\"\"\n import vllm\n\n if Version(vllm.__version__) <= Version(\"0.10.2\"):\n from vllm.sampling_params import GuidedDecodingParams as StructuredOutputsParams\n\n structured_outputs_key = \"guided_decoding\"\n else:\n from vllm.sampling_params import StructuredOutputsParams\n\n structured_outputs_key = \"structured_outputs\"\n\n profiler = profiler or nullcontext()\n accelerator = self.accelerator\n temperature = self.temperature\n top_p = self.top_p\n top_k = self.top_k\n min_p = self.min_p\n repetition_penalty = self.repetition_penalty\n max_completion_length = self.max_completion_length\n\n # Wake up colocated vLLM weights if needed (idempotent if already awake from sync_weights)\n if self.mode == \"colocate\" and self.enable_sleep_mode:\n empty_cache() # required to avoid OOM in some cases\n self.llm.wake_up(tags=[\"weights\"])\n # Work around for https://github.com/vllm-project/vllm/issues/29341\n try:\n self.llm.collective_rpc(\"reload_weights\")\n except NotImplementedError:\n # Non-CUDA vLLM backends (e.g., vllm-ascend's NPUWorkerV1), don't implement reload_weights\n pass\n\n # Generate completions using vLLM: gather all prompts and use them in a single call in the main process\n if self.mode == \"server\":\n all_prompts = gather_object(prompts)\n # Always gather images (even when None) to avoid deadlock: images may be None on some ranks\n # and non-None on others in mixed datasets, and gather_object is a collective operation.\n all_images = gather_object(images if images is not None else [None] * len(prompts))\n if all(img is None for img in all_images):\n all_images = None\n\n if accelerator.is_main_process:\n # Since 'prompts' contains 'num_generations' duplicates, we first take unique prompts, and\n # generate num_generations outputs for each one. This is faster than generating outputs for each\n # duplicate prompt individually.\n ordered_set_of_prompt_ids = all_prompts[::num_generations]\n ordered_set_of_images = all_images[::num_generations] if all_images is not None else None\n\n sampling_params = {\n \"n\": num_generations,\n \"repetition_penalty\": repetition_penalty,\n \"temperature\": temperature,\n \"top_p\": top_p,\n \"top_k\": top_k,\n \"min_p\": 0.0 if min_p is None else min_p,\n \"max_tokens\": max_completion_length,\n \"logprobs\": self.logprobs,\n \"structured_outputs_regex\": self.structured_outputs_regex,\n \"generation_kwargs\": self.generation_kwargs,\n }\n with profiler:\n output = self.vllm_client.generate(\n prompts=ordered_set_of_prompt_ids,\n images=ordered_set_of_images,\n **sampling_params,\n )\n payload = (\n output[\"prompt_ids\"],\n output[\"completion_ids\"],\n output[\"logprobs\"],\n output.get(\"logprob_token_ids\"),\n )\n else:\n payload = None\n\n # Broadcast the completions from the main process to all processes, ensuring each process receives its corresponding slice.\n obj_list = [payload]\n broadcast_object_list(obj_list, from_process=0)\n all_prompt_ids, all_completion_ids, all_logprobs, all_logprob_token_ids = obj_list[0]\n\n # vllm_client.generate(n=num_generations) returns num_generations completions per prompt.\n # Duplicate prompt_ids to align with per-completion entries.\n all_prompt_ids = [ids for ids in all_prompt_ids for _ in range(num_generations)]\n\n process_slice = slice(\n accelerator.process_index * len(prompts),\n (accelerator.process_index + 1) * len(prompts),\n )\n prompt_ids = all_prompt_ids[process_slice]\n completion_ids = all_completion_ids[process_slice]\n logprobs = all_logprobs[process_slice] if all_logprobs is not None else None\n logprob_token_ids = all_logprob_token_ids[process_slice] if all_logprob_token_ids is not None else None\n\n # Generate completions using colocated vLLM instances: each device holds vLLM copy and work on their own batch of prompts\n elif self.mode == \"colocate\":\n generation_kwargs = {\n \"n\": 1, # vLLM on each GPU generates only 1 in colocate mode\n \"repetition_penalty\": repetition_penalty,\n \"temperature\": temperature,\n \"top_p\": top_p,\n \"top_k\": top_k,\n \"min_p\": 0.0 if min_p is None else min_p,\n \"max_tokens\": max_completion_length,\n \"logprobs\": self.logprobs,\n }\n generation_kwargs.update(self.generation_kwargs)\n\n if self.structured_outputs_regex is not None:\n if generation_kwargs.get(structured_outputs_key) is not None:\n logger.warning(\n f\"Both `structured_outputs_regex` and `generation_kwargs['{structured_outputs_key}']` are set; \"\n \"`structured_outputs_regex` takes precedence.\"\n )\n generation_kwargs[structured_outputs_key] = StructuredOutputsParams(\n regex=self.structured_outputs_regex\n )\n elif isinstance(structured_outputs_kwargs := generation_kwargs.get(structured_outputs_key), dict):\n generation_kwargs[structured_outputs_key] = StructuredOutputsParams(**structured_outputs_kwargs)\n sampling_params = SamplingParams(**generation_kwargs)\n\n if self.tensor_parallel_size > 1:\n # Gather prompts from all ranks in the TP group and flatten.\n # Each rank starts with its own prompts; after gathering, all ranks see the full group set.\n orig_size = len(prompts)\n gathered_prompts = [None for _ in range(self.tensor_parallel_size)]\n torch.distributed.all_gather_object(gathered_prompts, prompts, group=self.tp_group)\n all_prompts = [p for sublist in gathered_prompts for p in sublist]\n # Always gather images (even when None) to avoid deadlock: images may be None on some\n # ranks and non-None on others in mixed datasets, and all_gather_object is collective.\n local_images = images if images is not None else [None] * len(prompts)\n gathered_images = [None for _ in range(self.tensor_parallel_size)]\n torch.distributed.all_gather_object(gathered_images, local_images, group=self.tp_group)\n all_images = [img for sublist in gathered_images for img in sublist]\n if all(img is None for img in all_images):\n all_images = None\n else:\n all_prompts = prompts\n all_images = images\n\n if self.enable_sleep_mode:\n self.llm.wake_up(tags=[\"kv_cache\"])\n\n # Build vLLM-compatible prompt inputs with token IDs and optional multi-modal data\n vllm_prompts = []\n if all_images is not None:\n for ids, img_list in zip(all_prompts, all_images, strict=True):\n row = {\"prompt_token_ids\": ids}\n if img_list is not None:\n row[\"multi_modal_data\"] = {\"image\": img_list if len(img_list) > 1 else img_list[0]}\n vllm_prompts.append(row)\n else:\n vllm_prompts = [{\"prompt_token_ids\": ids} for ids in all_prompts]\n\n with profiler:\n all_outputs = self.llm.generate(vllm_prompts, sampling_params=sampling_params, use_tqdm=False)\n\n all_prompt_ids = [output.prompt_token_ids for output in all_outputs]\n all_completion_ids = [output.token_ids for outputs in all_outputs for output in outputs.outputs]\n all_logprobs, all_logprob_token_ids = extract_logprobs(all_outputs)\n\n if self.tensor_parallel_size > 1:\n # Slice completions for this rank within its TP group.\n # Each rank generates all outputs โ€” we keep only our share.\n local_rank_in_group = torch.distributed.get_rank(group=self.tp_group)\n tp_slice = slice(local_rank_in_group * orig_size, (local_rank_in_group + 1) * orig_size)\n prompt_ids = all_prompt_ids[tp_slice]\n completion_ids = all_completion_ids[tp_slice]\n logprobs = all_logprobs[tp_slice] if all_logprobs is not None else None\n logprob_token_ids = all_logprob_token_ids[tp_slice] if all_logprob_token_ids is not None else None\n else:\n prompt_ids = all_prompt_ids\n completion_ids = all_completion_ids\n logprobs = all_logprobs\n logprob_token_ids = all_logprob_token_ids\n\n if self.enable_sleep_mode:\n self.llm.sleep(level=2)\n\n return prompt_ids, completion_ids, logprobs, logprob_token_ids\n" }, { "path": "trl/import_utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport importlib\nimport importlib.metadata\nimport importlib.util\nimport warnings\nfrom contextlib import contextmanager\n\nfrom packaging.version import Version\n\n\nLIGER_KERNEL_MIN_VERSION = \"0.7.0\"\nPACKAGE_DISTRIBUTION_MAPPING = importlib.metadata.packages_distributions()\n\n\n# From transformers: https://github.com/huggingface/transformers/blob/556312cd45a5e619c41b0f8adf680eab0d334324/src/transformers/utils/import_utils.py#L48-L77\ndef _is_package_available(pkg_name: str, return_version: bool = False) -> tuple[bool, str] | bool:\n \"\"\"Check if `pkg_name` exist, and optionally try to get its version\"\"\"\n spec = importlib.util.find_spec(pkg_name)\n package_exists = spec is not None\n package_version = \"N/A\"\n if package_exists and return_version:\n try:\n # importlib.metadata works with the distribution package, which may be different from the import\n # name (e.g. `PIL` is the import name, but `pillow` is the distribution name)\n distributions = PACKAGE_DISTRIBUTION_MAPPING[pkg_name]\n # Per PEP 503, underscores and hyphens are equivalent in package names.\n # Prefer the distribution that matches the (normalized) package name.\n normalized_pkg_name = pkg_name.replace(\"_\", \"-\")\n if normalized_pkg_name in distributions:\n distribution_name = normalized_pkg_name\n elif pkg_name in distributions:\n distribution_name = pkg_name\n else:\n distribution_name = distributions[0]\n package_version = importlib.metadata.version(distribution_name)\n except (importlib.metadata.PackageNotFoundError, KeyError):\n # If we cannot find the metadata (because of editable install for example), try to import directly.\n # Note that this branch will almost never be run, so we do not import packages for nothing here\n package = importlib.import_module(pkg_name)\n package_version = getattr(package, \"__version__\", \"N/A\")\n if return_version:\n return package_exists, package_version\n else:\n return package_exists\n\n\ndef is_deepspeed_available() -> bool:\n return _is_package_available(\"deepspeed\")\n\n\ndef is_fastapi_available() -> bool:\n return _is_package_available(\"fastapi\")\n\n\ndef is_jmespath_available() -> bool:\n return _is_package_available(\"jmespath\")\n\n\ndef is_joblib_available() -> bool:\n return _is_package_available(\"joblib\")\n\n\ndef is_liger_kernel_available(min_version: str = LIGER_KERNEL_MIN_VERSION) -> bool:\n _liger_kernel_available, _liger_kernel_version = _is_package_available(\"liger_kernel\", return_version=True)\n return _liger_kernel_available and Version(_liger_kernel_version) >= Version(min_version)\n\n\ndef is_llm_blender_available() -> bool:\n return _is_package_available(\"llm_blender\")\n\n\ndef is_math_verify_available() -> bool:\n return _is_package_available(\"math_verify\")\n\n\ndef is_mergekit_available() -> bool:\n return _is_package_available(\"mergekit\")\n\n\ndef is_pydantic_available() -> bool:\n return _is_package_available(\"pydantic\")\n\n\ndef is_requests_available() -> bool:\n return _is_package_available(\"requests\")\n\n\ndef is_unsloth_available() -> bool:\n return _is_package_available(\"unsloth\")\n\n\ndef is_uvicorn_available() -> bool:\n return _is_package_available(\"uvicorn\")\n\n\ndef is_vllm_available(min_version: str | None = None) -> bool:\n _vllm_available, _vllm_version = _is_package_available(\"vllm\", return_version=True)\n if _vllm_available:\n if not (Version(\"0.10.2\") <= Version(_vllm_version) <= Version(\"0.17.1\")):\n warnings.warn(\n f\"TRL currently supports vLLM versions from 0.10.2 to 0.17.1. You have version {_vllm_version} \"\n \"installed. We recommend installing a supported version to avoid compatibility issues.\",\n stacklevel=2,\n )\n if min_version is not None and Version(_vllm_version) < Version(min_version):\n return False\n return _vllm_available\n\n\ndef is_vllm_ascend_available() -> bool:\n return _is_package_available(\"vllm_ascend\")\n\n\ndef is_weave_available() -> bool:\n return _is_package_available(\"weave\")\n\n\nclass TRLExperimentalWarning(UserWarning):\n \"\"\"Warning for using the 'trl.experimental' submodule.\"\"\"\n\n pass\n\n\n@contextmanager\ndef suppress_warning(category):\n with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\", category=category)\n yield\n\n\ndef suppress_experimental_warning():\n return suppress_warning(TRLExperimentalWarning)\n" }, { "path": "trl/models/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom typing import TYPE_CHECKING\n\nfrom .._lazy_module import _LazyModule\n\n\n_import_structure = {\n \"activation_offloading\": [\"get_act_offloading_ctx_manager\"],\n \"utils\": [\"create_reference_model\", \"prepare_deepspeed\", \"prepare_fsdp\", \"unwrap_model_for_generation\"],\n}\n\n\nif TYPE_CHECKING:\n from .activation_offloading import get_act_offloading_ctx_manager\n from .utils import create_reference_model, prepare_deepspeed, prepare_fsdp, unwrap_model_for_generation\nelse:\n import sys\n\n sys.modules[__name__] = _LazyModule(__name__, globals()[\"__file__\"], _import_structure, module_spec=__spec__)\n" }, { "path": "trl/models/activation_offloading.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Copyright (c) Meta Platforms, Inc. and affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the BSD-style license found in the\n# LICENSE file in the root directory of https://github.com/pytorch/torchtune.\n\n\nimport psutil\nimport torch\nfrom accelerate import logging\nfrom accelerate.utils.versions import is_torch_version\nfrom torch import nn\nfrom torch.autograd.graph import saved_tensors_hooks\nfrom transformers import is_torch_npu_available\n\n\nif is_torch_npu_available():\n import torch_npu # noqa: F401\n\n# Import DTensor for FSDP v2 support with version-aware import path\nDTensor = None\nif torch.distributed.is_available():\n try:\n if is_torch_version(\">=\", \"2.5.0\"):\n from torch.distributed.tensor import DTensor\n else:\n # from torch 2.0.0 (oldest supported accelerate torch version), DTensor is in torch.distributed._tensor\n from torch.distributed._tensor import DTensor\n except (ImportError, AttributeError):\n DTensor = None\n\nlogger = logging.get_logger(__name__)\n\n\ndef _get_unique_tensor_key(tensor: torch.Tensor) -> tuple:\n \"\"\"\n Get a unique key for a tensor based on its storage pointer and dtype. This allows deduplication of tensors that\n share the same underlying storage. From:\n https://github.com/volcengine/verl/blob/main/verl/utils/activation_offload.py\n\n Args:\n tensor: The tensor to get the key for\n\n Returns:\n A tuple of (storage_pointer, dtype) that uniquely identifies the tensor's storage\n \"\"\"\n # Handle special tensor types - primarily for FSDP v2 DTensor\n actual_tensor = tensor\n\n # For DTensor (FSDP v2), extract the local tensor\n if DTensor is not None and isinstance(tensor, DTensor) and hasattr(tensor, \"_local_tensor\"):\n actual_tensor = tensor._local_tensor\n\n # Try to get storage pointer, but fall back to tensor id if not accessible\n try:\n storage_ptr = actual_tensor.untyped_storage().data_ptr() + actual_tensor.storage_offset()\n except (RuntimeError, AttributeError):\n # For tensors with invalid storage, use tensor id\n # This won't enable deduplication for these tensors, but allows offloading to work\n storage_ptr = id(actual_tensor)\n\n return (storage_ptr, actual_tensor.dtype)\n\n\nclass OffloadActivations(saved_tensors_hooks):\n \"\"\"\n Context manager under which activation tensors created in the forward pass will be offloaded.\n\n Enable the memory efficiency technique of activation offloading, where activations bigger than `min_offload_size`\n bytes will be offloaded to CPU in the forward and brought back in the backward. This is in contrast to maintaining\n the activation on GPU VRAM throughout the program.\n\n This manager contains the option of using one additional CUDA stream to handle the communication between CUDA and\n CPU, which is intended to overlap with the default computation stream to improve runtime. We designed\n synchronization with a few heuristics for optimizing the tradeoff between runtime vs memory usage.\n\n Args:\n use_pin_memory (`bool`, *optional*, defaults to `True`):\n Whether to offloaded Tensor will be placed in pinned memory on the CPU. Pinned memory allows the Tensor to\n be moved back onto GPU more quickly but is a limited resource.\n use_streams (`bool`, *optional*, defaults to `True`):\n Whether to use streams for performance optimization where the communications get overlapped with the\n computation. Requires a torch build after torch-2.5.0.\n min_offload_size (`int`, *optional*, defaults to `1024`):\n Minimum number of bytes a Tensor must be in order to qualify for offloading. If the tensor is too small, we\n do not want to waste bandwidth and resources moving it to CPU and back.\n max_fwd_stash_size (`int`, *optional*, defaults to `5`):\n Maximum size of the forward stash, or the maximum number of consecutive activations to keep alive during\n the forward pass. This number must be at least 1. Keeping alive more activations will potentially allow\n more overlap between the communication and compute streams at the cost of increasing memory usage. Keeping\n alive fewer activations will conserve memory, but may cause poor overlap between the streams, increasing\n runtime.\n\n Raises:\n ValueError: if `max_fwd_stash_size` is not at least `1`.\n\n Example:\n ```python\n >>> with OffloadActivations():\n ... outputs = model(inputs, labels=labels)\n >>> loss = outputs.loss\n >>> loss.backward()\n ```\n \"\"\"\n\n def __init__(\n self,\n use_pin_memory: bool = True,\n use_streams: bool = True,\n min_offload_size: int = 1024,\n max_fwd_stash_size: int = 5,\n ) -> None:\n self.use_streams = use_streams\n\n self.min_tensor_size_bytes = min_offload_size # we don't want to bother with small tensors\n self.tracker = {} # tensor_id => (new_tensor, if_modified) ---> track what saved/offloaded tensors are where\n self.tensor_id = 0\n self.is_first_forward_call = True\n self.is_first_backward_call = True\n self.is_first_forward_pass = True\n\n # Storage deduplication: maps storage key to tensor_id to avoid offloading same storage multiple times\n self.storage_to_tensor_id = {}\n\n # Parameter filtering: track parameter storage pointers to skip them during offloading\n self.param_storages = set()\n\n # Managing cpu memory\n self.use_pin_memory = use_pin_memory\n self.virtual_memory_safe_pct = 60 # we should not exceed this percentage of memory\n\n self.accelerator_type = (\n torch.accelerator.current_accelerator().type if hasattr(torch, \"accelerator\") else \"cuda\"\n )\n # NOTE: xpu doesn't have `default_stream` API, use `current_stream` instead\n if self.accelerator_type == \"xpu\": # comp stream\n self.s0 = torch.xpu.current_stream()\n elif is_torch_npu_available() and self.accelerator_type == \"npu\":\n self.s0 = torch.npu.current_stream()\n else:\n self.s0 = torch.cuda.default_stream()\n\n # For streaming\n if self.use_streams:\n if self.accelerator_type == \"xpu\": # comms stream\n self.s1 = torch.xpu.Stream()\n elif self.accelerator_type == \"npu\":\n self.s1 = torch.npu.Stream()\n else:\n self.s1 = torch.cuda.Stream()\n self.fwd_stash = {} # tensor_id => (activation, ev1)\n if max_fwd_stash_size < 1:\n raise ValueError(f\"max_fwd_stash_size should be at least 1 but is {max_fwd_stash_size}\")\n self.max_fwd_stash_size = max_fwd_stash_size\n self.bwd_tensor_stash = {} # tensor_id => activation\n self.bwd_ev_stash = {} # tensor_id => ev0\n self.curr_graph_id = None\n self.curr_autograd_node = None\n\n # -------- platform util functions -------- #\n def verify_sufficient_virtual_memory():\n curr_pct = get_cpu_ram_pct()\n if curr_pct > self.virtual_memory_safe_pct:\n logger.warning(f\"{curr_pct=}% > {self.virtual_memory_safe_pct=}% of virtual memory used\")\n\n def get_cpu_ram_pct() -> float:\n # get the percentage of memory used by the system\n return psutil.virtual_memory().percent\n\n def get_tensor_id() -> int:\n # create a unique id for each tensor we are managing\n self.tensor_id += 1\n return self.tensor_id\n\n def get_num_bytes_tensor(x: torch.Tensor) -> int:\n # get the number of bytes in a tensor, for memory management purposes\n return x.element_size() * x.nelement() # x.element_size() * x._base_storage().nbytes()\n\n # -------- core pack / unpack work -------- #\n def pack_tensor(activation: torch.Tensor) -> int:\n # activations are passed in during forward pass - from here we take over and return a unique id\n if self.is_first_forward_call:\n if len(self.tracker) != 0:\n raise ValueError(\"Backward pass should have cleared tracker of all tensors\")\n\n # set training phase trackers\n self.is_first_forward_call = False\n self.is_first_backward_call = True\n # Reset deduplication map for new forward pass\n self.storage_to_tensor_id = {}\n\n # query for basic tensor info\n num_bytes = get_num_bytes_tensor(activation)\n tensor_id = get_tensor_id()\n\n # Check for tensor deduplication using storage pointer\n # If this storage is already being tracked, we still create a new tensor_id\n # but don't offload again (just keep the tensor in GPU)\n storage_key = _get_unique_tensor_key(activation)\n if storage_key in self.storage_to_tensor_id:\n # Storage already offloaded - don't offload again, just track the reference\n self.tracker[tensor_id] = (activation, False, None, None, None) # Keep on GPU, don't offload\n return tensor_id\n\n # Check if tensor is on CPU (skip offloading)\n if activation.device.type not in [\"cuda\", \"xpu\", \"npu\"]:\n self.tracker[tensor_id] = (activation, False, None, None, None)\n return tensor_id\n\n # Check if tensor is too small\n if num_bytes < self.min_tensor_size_bytes:\n self.tracker[tensor_id] = (activation, False, None, None, None)\n return tensor_id\n\n # Check if tensor is a parameter or buffer\n if isinstance(activation, torch.nn.Parameter) or (\n hasattr(torch.nn, \"Buffer\") and isinstance(activation, torch.nn.Buffer)\n ):\n self.tracker[tensor_id] = (activation, False, None, None, None)\n return tensor_id\n\n # Check if tensor is an FP8 tensor (TorchAO) - skip offloading as they're already compressed\n tensor_class_name = type(activation).__name__\n if tensor_class_name in [\"Float8TrainingTensor\", \"ScaledMMConfig\", \"LinearMMConfig\"]:\n self.tracker[tensor_id] = (activation, False, None, None, None)\n return tensor_id\n\n # Check if tensor storage is a model parameter (for FSDP compatibility)\n try:\n # Extract actual tensor for DTensor\n check_tensor = activation\n if DTensor is not None and isinstance(activation, DTensor) and hasattr(activation, \"_local_tensor\"):\n check_tensor = activation._local_tensor\n\n if check_tensor.untyped_storage().data_ptr() in self.param_storages:\n self.tracker[tensor_id] = (activation, False, None, None, None)\n return tensor_id\n except (RuntimeError, AttributeError):\n # If we can't get data_ptr, skip this check\n pass\n\n # Tensor qualifies for offloading\n if self.use_streams:\n # First, sync back and dereference previously offloaded tensors\n # as the offloading should be done sufficiently long ago.\n for id in list(self.fwd_stash.keys()):\n if id <= tensor_id - self.max_fwd_stash_size:\n _, ev = self.fwd_stash[id]\n self.s0.wait_event(ev)\n del self.fwd_stash[id]\n else:\n break\n\n # Sync in, offload, and add an event to sync back later\n self.s1.wait_stream(self.s0)\n\n stream = self.s1 if self.use_streams else self.s0\n if self.accelerator_type == \"xpu\":\n stream_ctx = torch.xpu.stream(stream)\n elif self.accelerator_type == \"npu\":\n stream_ctx = torch.npu.stream(stream)\n else:\n stream_ctx = torch.cuda.stream(stream)\n with stream_ctx:\n # Save original stride and shape information\n original_stride = activation.stride()\n original_storage_offset = activation.storage_offset()\n original_shape = activation.size()\n\n # Check if tensor has broadcast dimensions (stride == 0)\n # If so, copy the underlying storage directly instead of materializing the broadcast\n has_broadcast = 0 in original_stride\n\n if has_broadcast:\n # Copy only the actual underlying storage, not the materialized broadcast\n # Create CPU tensor with same storage size as original\n storage_size = activation.untyped_storage().size()\n cpu_storage = torch.empty(\n storage_size // activation.element_size(),\n dtype=activation.dtype,\n pin_memory=self.use_pin_memory,\n device=\"cpu\",\n )\n # Copy the raw storage\n cpu_storage_view = torch.as_strided(\n activation, size=(storage_size // activation.element_size(),), stride=(1,), storage_offset=0\n )\n cpu_storage.copy_(cpu_storage_view, non_blocking=True)\n cpu_tensor = cpu_storage\n else:\n # No broadcast - use normal contiguous copy\n cpu_tensor = torch.empty_like(activation, pin_memory=self.use_pin_memory, device=\"cpu\")\n cpu_tensor.copy_(activation, non_blocking=True)\n\n # Store CPU tensor along with stride information\n self.tracker[tensor_id] = (\n cpu_tensor,\n True, # True = (in future) modified\n original_stride, # Save original GPU stride\n original_storage_offset, # Save original storage offset\n original_shape, # Save original shape for broadcast restoration\n )\n\n if self.use_streams:\n event = self.s1.record_event()\n\n # Stash to keep activation alive til s1 is done\n self.fwd_stash[tensor_id] = (activation, event)\n\n # Track this storage for deduplication\n self.storage_to_tensor_id[storage_key] = tensor_id\n\n return tensor_id\n\n def unpack_tensor_single_stream(unpack_tensor_id: int) -> torch.Tensor:\n # backward pass - we are called with the tensor_id, which\n # we will use to retrieve the saved/offloaded tensor\n if self.is_first_backward_call:\n if self.is_first_forward_pass:\n self.is_first_forward_pass = False\n if self.use_pin_memory:\n verify_sufficient_virtual_memory()\n\n self.is_first_backward_call = False\n\n if unpack_tensor_id not in self.tracker:\n raise ValueError(f\"Untracked tensor with id {unpack_tensor_id}\")\n\n (\n maybe_accelerator_tensor,\n modified,\n original_stride,\n original_storage_offset,\n original_shape,\n ) = self.tracker[unpack_tensor_id]\n\n if modified:\n # Restore tensor to GPU\n accelerator_tensor = maybe_accelerator_tensor.to(self.accelerator_type, non_blocking=True)\n # Restore original stride if we saved it (handles both broadcast and non-broadcast cases)\n if original_stride is not None:\n accelerator_tensor = torch.as_strided(\n accelerator_tensor,\n size=original_shape,\n stride=original_stride,\n storage_offset=original_storage_offset,\n )\n maybe_accelerator_tensor = accelerator_tensor\n\n # clear tensor from tracking\n del self.tracker[unpack_tensor_id]\n # Only set is_first_forward_call to True when all tensors have been unpacked\n if len(self.tracker) == 0:\n self.is_first_forward_call = True\n return maybe_accelerator_tensor\n\n def unpack_tensor_with_streams(unpack_tensor_id: int) -> torch.Tensor:\n # backward pass - we are called with the tensor_id, which\n # we will use to retrieve the saved/offloaded tensor\n if self.is_first_backward_call:\n self.curr_graph_id = torch._C._current_graph_task_id()\n\n def wait_and_del_remaining_references() -> None:\n for id in list(self.bwd_tensor_stash.keys()):\n if id in self.bwd_ev_stash:\n event = self.bwd_ev_stash[id]\n self.s1.wait_event(event)\n del self.bwd_tensor_stash[id]\n\n # Register a callback to the end of autograd to clean everything up\n torch.autograd.variable.Variable._execution_engine.queue_callback(wait_and_del_remaining_references)\n\n if self.is_first_forward_pass:\n self.is_first_forward_pass = False\n if self.use_pin_memory:\n verify_sufficient_virtual_memory()\n\n self.is_first_backward_call = False\n\n if unpack_tensor_id not in self.tracker:\n raise ValueError(f\"untracked tensor with id {unpack_tensor_id}\")\n\n (\n maybe_accelerator_tensor,\n modified,\n original_stride,\n original_storage_offset,\n original_shape,\n ) = self.tracker[unpack_tensor_id]\n\n if modified:\n # Get data on the current autograd node\n graph_id = torch._C._current_graph_task_id()\n node = torch._C._current_autograd_node()\n prev_node_ids = []\n\n # If we're on a new node, mark prev node's tensors to be freed later\n if graph_id == self.curr_graph_id and self.curr_autograd_node != node:\n self.curr_autograd_node = node\n prev_node_ids = list(self.bwd_tensor_stash.keys())\n\n brought_back_from_cpu = True\n if unpack_tensor_id in self.fwd_stash:\n maybe_accelerator_tensor = self.fwd_stash[unpack_tensor_id][0]\n brought_back_from_cpu = False\n else:\n # Kick off the process to bring tensors back\n if self.accelerator_type == \"xpu\":\n stream_ctx = torch.xpu.stream(self.s1)\n elif self.accelerator_type == \"npu\":\n stream_ctx = torch.npu.stream(self.s1)\n else:\n stream_ctx = torch.cuda.stream(self.s1)\n with stream_ctx:\n # Restore tensor to GPU\n accelerator_tensor = maybe_accelerator_tensor.to(self.accelerator_type, non_blocking=True)\n # Restore original stride if we saved it (handles both broadcast and non-broadcast cases)\n if original_stride is not None:\n accelerator_tensor = torch.as_strided(\n accelerator_tensor,\n size=original_shape,\n stride=original_stride,\n storage_offset=original_storage_offset,\n )\n maybe_accelerator_tensor = accelerator_tensor\n\n # Tell comp stream to wait for the info to be loaded before executing\n self.s0.wait_stream(self.s1)\n\n # Stash the tensor to keep memory alive until compute stream is complete\n self.bwd_tensor_stash[unpack_tensor_id] = maybe_accelerator_tensor\n\n # Note: [Track views of the unpacked]\n # Why do we get the use count of the unpacked tensor here? We want an\n # initial count to compare to later, during the post-hook of the\n # backward node, when we need to decide whether we're allowed to free\n # the tensor yet. In what obscure cases must we delay freeing the\n # tensor (and thus call record_stream)?\n # 1. Any of the outputs of the backward node is a view of the unpacked\n # tensor.\n # 2. In the case that this unpacked tensor will be used in a\n # checkpointed region, if one of the recomputed saved tensors ends\n # up as a view of the unpacked tensor.\n # 3. The user abuses the system somehow and manually relies on the\n # unpacked tensor to exist after the backward node has executed.\n if self.accelerator_type == \"npu\":\n storage_refcount = torch_npu._C._storage_Use_Count(\n maybe_accelerator_tensor.untyped_storage()._cdata\n )\n else:\n storage_refcount = torch._C._storage_Use_Count(\n maybe_accelerator_tensor.untyped_storage()._cdata\n )\n\n def hook(outputs, inputs):\n # create events for the current node inputs/outputs if they were streamed in\n if brought_back_from_cpu:\n # See Note: [Track views of the unpacked]\n # IF any of the outputs is a view of the tensor, OR if a view of\n # the tensor has been saved as a part of checkpoint's recompute\n # process, OR the user has abusedly incurred a reference on the\n # unpacked tensor, THEN the tensor might be used later and we\n # cannot presume to delete it after only the current node is\n # done! So we use our frenemy, record_stream, to ensure the\n # Tensor stays unmessed with until it's done getting used in the\n # compute stream (s0 here). Note that the con here is we introduce\n # non-deterministic (thus higher) memory usage, but this case\n # should not happen often.\n # Check if tensor still exists (might have been cleaned up by a previous node)\n if unpack_tensor_id in self.bwd_tensor_stash:\n unpacked_tensor = self.bwd_tensor_stash[unpack_tensor_id]\n if self.accelerator_type == \"npu\":\n storage_count = torch_npu._C._storage_Use_Count(\n unpacked_tensor.untyped_storage()._cdata\n )\n else:\n storage_count = torch._C._storage_Use_Count(unpacked_tensor.untyped_storage()._cdata)\n if storage_count > storage_refcount:\n unpacked_tensor.record_stream(self.s0)\n del self.bwd_tensor_stash[unpack_tensor_id]\n else:\n event = self.s0.record_event()\n self.bwd_ev_stash[unpack_tensor_id] = event\n\n # if there are still things in the fwd_stash, get rid of them as we're in bwd now\n for id in list(self.fwd_stash.keys()):\n _, ev = self.fwd_stash[id]\n self.s0.wait_event(ev)\n del self.fwd_stash[id]\n\n # wait on prev node's events and del those\n for id in prev_node_ids:\n # Only wait on events that exist (some tensors may have used record_stream instead)\n if id in self.bwd_ev_stash:\n event = self.bwd_ev_stash[id]\n self.s1.wait_event(event)\n del self.bwd_ev_stash[id]\n if id in self.bwd_tensor_stash:\n del self.bwd_tensor_stash[id]\n\n return outputs\n\n node.register_hook(hook)\n\n # clear tensor from tracking\n del self.tracker[unpack_tensor_id]\n # Only set is_first_forward_call to True when all tensors have been unpacked\n if len(self.tracker) == 0:\n self.is_first_forward_call = True\n return maybe_accelerator_tensor\n\n unpack_tensor = unpack_tensor_with_streams if self.use_streams else unpack_tensor_single_stream\n super().__init__(pack_tensor, unpack_tensor)\n\n def update_model_params(self, model: nn.Module):\n \"\"\"\n Update the set of parameter storage pointers from the model. This allows filtering out model parameters during\n offloading, which is especially important for FSDP models where parameters may not be detected by isinstance\n checks.\n\n For FSDP v2, this method handles DTensor parameters which may be sharded across ranks and not have valid local\n storage on all ranks. We extract the local tensor from DTensors using _local_tensor when available.\n\n Args:\n model: The model whose parameters should be tracked\n \"\"\"\n param_storages = set()\n\n for p in model.parameters():\n # For FSDP v2: extract local tensor from DTensor\n actual_tensor = p\n if DTensor is not None and isinstance(p, DTensor) and hasattr(p, \"_local_tensor\"):\n actual_tensor = p._local_tensor\n\n # Try to get storage pointer\n try:\n storage_ptr = actual_tensor.untyped_storage().data_ptr()\n if storage_ptr != 0:\n param_storages.add(storage_ptr)\n except RuntimeError:\n # Parameter doesn't have accessible storage (e.g., FSDP v2 sharded without local shard, FP8 parameters)\n # These will be caught by other checks (isinstance for Parameter, class name for FP8)\n continue\n\n self.param_storages = param_storages\n\n\nclass NoOpManager(saved_tensors_hooks):\n \"\"\"\n A `saved_tensors_hook` manager used to disable any other `saved_tensors_hook` manager applied before. This relies\n on the behavior that only the most recently registered `saved_tensors_hook` will run.\n\n One example usage is to opt a local region of code out of activations offloading, which is usually applied globally\n to best track state.\n \"\"\"\n\n def __init__(self) -> None:\n def noop(tensor):\n return tensor\n\n super().__init__(noop, noop)\n\n\ndef get_act_offloading_ctx_manager(\n model: nn.Module,\n use_pin_memory: bool = True,\n use_streams: bool = True,\n min_offload_size: int = 1024,\n max_fwd_stash_size: int = 5,\n warn_if_no_head: bool = True,\n) -> OffloadActivations:\n \"\"\"\n Returns the activation offloading context manager for the model. All but the last output Linear in every step will\n be offloaded.\n\n If activation offloading is enabled, we return the OffloadActivations context manager. If activation offloading is\n disabled, we return a NoOpManager context manager.\n\n Args:\n model (`nn.Module`):\n Model to wrap with the activation offloading context manager.\n use_pin_memory (`bool`, *optional*, defaults to `True`):\n Whether to offloaded Tensor will be placed in pinned memory on the CPU. Pinned memory allows the Tensor to\n be moved back onto GPU more quickly but is a limited resource.\n use_streams (`bool`, *optional*, defaults to `True`):\n Whether to use streams for performance optimization where the communications get overlapped with the\n computation. Requires a torch build after torch-2.5.0.\n min_offload_size (`int`, *optional*, defaults to `1024`):\n Minimum number of bytes a Tensor must be in order to qualify for offloading. If the tensor is too small, we\n do not want to waste bandwidth and resources moving it to CPU and back.\n max_fwd_stash_size (`int`, *optional*, defaults to `5`):\n Maximum size of the forward stash, or the maximum number of consecutive activations to keep alive during\n the forward pass. This number must be at least 1. Keeping alive more activations will potentially allow\n more overlap between the communication and compute streams at the cost of increasing memory usage. Keeping\n alive fewer activations will conserve memory, but may cause poor overlap between the streams, increasing\n runtime.\n warn_if_no_head (`bool`, *optional*, defaults to `True`):\n Whether to warn if no output head is detected. If set to `False`, no warning will be raised if no output\n head is detected.\n\n Returns:\n `contextlib.ContextDecorator`:\n Activation offloading context manager for the model.\n \"\"\"\n activations_handling_ctx = OffloadActivations(\n use_pin_memory=use_pin_memory,\n use_streams=use_streams,\n min_offload_size=min_offload_size,\n max_fwd_stash_size=max_fwd_stash_size,\n )\n\n # Update parameter storages to filter them during offloading (important for FSDP)\n activations_handling_ctx.update_model_params(model)\n\n # Below is our hack to disable offloading the last output Linear in every\n # step, as the cost for offloading the activation and then soon after bringing\n # it back is expensive.\n output_head_detected = False\n noop_ctx = NoOpManager()\n\n # Try to get the actual model if it's wrapped\n unwrapped_model = model\n if hasattr(unwrapped_model, \"module\"):\n unwrapped_model = unwrapped_model.module\n # check for PEFT models\n if hasattr(unwrapped_model, \"base_model\") and hasattr(unwrapped_model, \"peft_config\"):\n unwrapped_model = unwrapped_model.base_model\n\n # Check for different types of output heads\n if hasattr(unwrapped_model, \"output\"):\n if isinstance(unwrapped_model.output, nn.Module):\n unwrapped_model.output.register_forward_pre_hook(lambda *args: noop_ctx.__enter__())\n unwrapped_model.output.register_forward_hook(lambda *args: noop_ctx.__exit__(), always_call=True)\n output_head_detected = True\n elif hasattr(unwrapped_model.output, \"linear\") and isinstance(unwrapped_model.output.linear, nn.Module):\n unwrapped_model.output.linear.register_forward_pre_hook(lambda *args: noop_ctx.__enter__())\n unwrapped_model.output.linear.register_forward_hook(lambda *args: noop_ctx.__exit__(), always_call=True)\n output_head_detected = True\n\n # Check for HuggingFace model output heads\n elif hasattr(unwrapped_model, \"lm_head\"):\n unwrapped_model.lm_head.register_forward_pre_hook(lambda *args: noop_ctx.__enter__())\n unwrapped_model.lm_head.register_forward_hook(lambda *args: noop_ctx.__exit__(), always_call=True)\n output_head_detected = True\n\n # Check for decoder-based models\n elif hasattr(unwrapped_model, \"decoder\"):\n decoder = unwrapped_model.decoder\n if hasattr(decoder, \"output\"):\n decoder.output.register_forward_pre_hook(lambda *args: noop_ctx.__enter__())\n decoder.output.register_forward_hook(lambda *args: noop_ctx.__exit__(), always_call=True)\n output_head_detected = True\n # Some models have lm_head in the decoder\n elif hasattr(decoder, \"lm_head\"):\n decoder.lm_head.register_forward_pre_hook(lambda *args: noop_ctx.__enter__())\n decoder.lm_head.register_forward_hook(lambda *args: noop_ctx.__exit__(), always_call=True)\n output_head_detected = True\n\n # Check for transformer models with final layer norm\n elif hasattr(unwrapped_model, \"final_layer_norm\") or hasattr(unwrapped_model, \"ln_f\"):\n final_norm = getattr(unwrapped_model, \"final_layer_norm\", None) or unwrapped_model.ln_f\n final_norm.register_forward_pre_hook(lambda *args: noop_ctx.__enter__())\n final_norm.register_forward_hook(lambda *args: noop_ctx.__exit__(), always_call=True)\n output_head_detected = True\n\n # Check for models with head module\n elif hasattr(unwrapped_model, \"head\") and isinstance(unwrapped_model.head, nn.Module):\n unwrapped_model.head.register_forward_pre_hook(lambda *args: noop_ctx.__enter__())\n unwrapped_model.head.register_forward_hook(lambda *args: noop_ctx.__exit__(), always_call=True)\n output_head_detected = True\n\n if not output_head_detected and warn_if_no_head:\n logger.warning(\n \"During activation offloading, no output head was detected. If your model has an output head, it will be \"\n \"offloaded. This usually greatly slows training, given the large vocabulary size. To change this \"\n \"behavior, set your output head as model.output and make it an nn.Module. You can disable this warning by \"\n \"passing `warn_if_no_head=False`.\"\n )\n\n # Disable offloading for any Liger modules\n for name, module in unwrapped_model.named_modules():\n if \"liger\" in name.lower():\n module.register_forward_pre_hook(lambda *args: noop_ctx.__enter__())\n module.register_forward_hook(lambda *args: noop_ctx.__exit__(), always_call=True)\n\n return activations_handling_ctx\n" }, { "path": "trl/models/utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport itertools\nimport warnings\nfrom collections.abc import Callable\nfrom contextlib import contextmanager\nfrom copy import deepcopy\nfrom typing import TYPE_CHECKING, Any\n\nimport accelerate\nimport torch.nn as nn\nimport transformers\nfrom accelerate import Accelerator\nfrom packaging.version import Version\nfrom torch.distributed.fsdp import FSDPModule\nfrom torch.distributed.fsdp.fully_sharded_data_parallel import FullyShardedDataParallel as FSDP\nfrom transformers import GenerationConfig, PreTrainedModel\n\nfrom ..import_utils import suppress_experimental_warning\n\n\nwith suppress_experimental_warning():\n from ..experimental.utils import create_reference_model as _create_reference_model\n\n\nif Version(accelerate.__version__) >= Version(\"1.11.0\"):\n from accelerate.utils.fsdp_utils import get_parameters_from_modules\n\nif TYPE_CHECKING:\n from deepspeed.runtime.engine import DeepSpeedEngine\n from torch.nn import Module\n from torch.nn.parallel.distributed import DistributedDataParallel\n\n\ndef remove_hooks(model: \"DeepSpeedEngine\") -> None:\n \"\"\"Removes the optimizer hooks from a DeepSpeed ZeRO-3 model.\"\"\"\n if not hasattr(model, \"optimizer\"): # before the first training step, the model has no optimizer\n return\n if model.optimizer is not None and hasattr(model.optimizer, \"parameter_offload\"):\n optimizer_offload = model.optimizer.parameter_offload\n elif model.optimizer is not None:\n optimizer_offload = model.optimizer\n else:\n raise RuntimeError(\"The model optimizer is None, which is not yet supported.\")\n\n for param in iter_params(optimizer_offload.module, recurse=True):\n param.ds_active_sub_modules.clear()\n\n for hook in optimizer_offload.forward_hooks:\n hook.remove()\n for hook in optimizer_offload.backward_hooks:\n hook.remove()\n\n optimizer_offload.forward_hooks = []\n optimizer_offload.backward_hooks = []\n\n\ndef get_all_parameters(sub_module, recurse=False):\n return itertools.chain(sub_module.named_parameters(recurse=recurse), sub_module.ds_external_parameters())\n\n\ndef iter_params(module, recurse=False):\n return [param for _, param in get_all_parameters(module, recurse)]\n\n\ndef add_hooks(model: \"DeepSpeedEngine\") -> None:\n \"\"\"Adds the optimizer hooks from a DeepSpeed ZeRO-3 model.\"\"\"\n import deepspeed\n\n if not hasattr(model, \"optimizer\"): # before the first training step, the model has no optimizer\n return\n if model.optimizer is not None and hasattr(model.optimizer, \"parameter_offload\"):\n optimizer_offload = model.optimizer.parameter_offload\n elif model.optimizer is not None:\n optimizer_offload = model.optimizer\n else:\n raise RuntimeError(\"The model optimizer is None, which is not yet supported.\")\n if Version(deepspeed.__version__) >= Version(\"0.16.4\"):\n # Account for renaming in https://github.com/deepspeedai/DeepSpeed/pull/6847\n optimizer_offload._register_deepspeed_module(optimizer_offload.module)\n else:\n optimizer_offload._register_hooks_recursively(optimizer_offload.module)\n\n\n@contextmanager\ndef _unwrap_model_for_generation(\n model: \"DistributedDataParallel | DeepSpeedEngine\",\n accelerator: \"Accelerator\",\n gather_deepspeed3_params: bool = True,\n):\n \"\"\"\n Context manager to unwrap distributed or accelerated models for generation tasks.\n\n Args:\n model (`DistributedDataParallel | DeepSpeedEngine`):\n Model to be unwrapped.\n accelerator ([`~accelerate.Accelerator`]):\n Accelerator instance managing the model.\n gather_deepspeed3_params (`bool`, *optional*, defaults to `True`):\n Whether to gather weights for DeepSpeed ZeRO Stage 3 models. If `False`, skips parameter gathering, which\n can be more memory-efficient but may lead to slower generation times.\n\n Yields:\n Unwrapped model.\n\n Example:\n ```python\n with _unwrap_model_for_generation(model, accelerator) as unwrapped_model:\n generated_outputs = unwrapped_model.generate(input_ids)\n ```\n \"\"\"\n unwrapped_model = accelerator.unwrap_model(model)\n is_gradient_checkpointing = unwrapped_model.is_gradient_checkpointing\n if is_gradient_checkpointing:\n unwrapped_model.gradient_checkpointing_disable()\n if accelerator.state.deepspeed_plugin is not None and accelerator.state.deepspeed_plugin.zero_stage == 3:\n if not gather_deepspeed3_params:\n yield accelerator.unwrap_model(model)\n else:\n import deepspeed\n\n with deepspeed.zero.GatheredParameters(model.parameters()):\n remove_hooks(model)\n yield accelerator.unwrap_model(model)\n add_hooks(model)\n else:\n yield unwrapped_model\n if is_gradient_checkpointing:\n unwrapped_model.gradient_checkpointing_enable()\n\n\n@contextmanager\ndef _override_model_generation_config(model, generation_kwargs=None):\n \"\"\"\n Context manager to temporarily override a model's generation_config with training config.\n\n This works around transformers' config merging logic that would otherwise overwrite values matching global defaults\n with model-specific values (see upstream issue transformers#42762; fixed in transformers v5 by PR\n `transformers#42702`).\n\n By temporarily setting the model's generation_config to match the passed generation_config, we avoid the conflict.\n\n The model's original generation_config is preserved outside this context, ensuring that saved/pushed models retain\n their intended inference behavior.\n\n Args:\n model: The model (typically unwrapped_model) whose generation_config to temporarily override.\n generation_kwargs (dict): Generation kwargs to be used to override model's generation config.\n \"\"\"\n if (\n # Issue fixed in transformers v5 by PR transformers#42702\n Version(transformers.__version__) >= Version(\"5.0.0\")\n or generation_kwargs is None\n or not hasattr(model, \"generation_config\")\n ):\n yield model\n return\n # If it is a PEFT model, override the underlying base model\n if hasattr(model, \"get_base_model\"):\n model = model.get_base_model()\n # Keep original model generation_config\n original_config = model.generation_config\n # Create training-specific generation config from the model's original generation config\n # Then overwrite it with the training-specific generation kwargs\n generation_config = GenerationConfig.from_dict(model.generation_config.to_dict())\n generation_config.update(**generation_kwargs)\n model.generation_config = generation_config\n try:\n yield\n finally:\n model.generation_config = original_config\n\n\n@contextmanager\ndef unwrap_model_for_generation(\n model: \"DistributedDataParallel | DeepSpeedEngine\",\n accelerator: \"Accelerator\",\n gather_deepspeed3_params: bool = True,\n generation_kwargs: dict | None = None,\n):\n \"\"\"\n Context manager to unwrap distributed or accelerated models for generation tasks.\n\n This function unwraps distributed models (FSDP, DeepSpeed) and optionally overrides the model's generation_config\n temporarily during generation. This is useful for applying training-specific generation parameters without\n permanently modifying the model's original generation_config.\n\n Args:\n model (`DistributedDataParallel | DeepSpeedEngine`):\n Model to be unwrapped.\n accelerator ([`~accelerate.Accelerator`]):\n Accelerator instance managing the model.\n gather_deepspeed3_params (`bool`, *optional*, defaults to `True`):\n Whether to gather weights for DeepSpeed ZeRO Stage 3 models. If `False`, skips parameter gathering, which\n can be more memory-efficient but may lead to slower generation times.\n generation_kwargs (dict, *optional*):\n If provided, temporarily overrides the model's generation_config during generation. The original config is\n automatically restored when exiting the context. This is useful for using different generation parameters\n during training vs. inference.\n\n Yields:\n Unwrapped model with optionally overridden generation_config.\n \"\"\"\n with (\n _unwrap_model_for_generation(\n model, accelerator, gather_deepspeed3_params=gather_deepspeed3_params\n ) as unwrapped_model,\n _override_model_generation_config(unwrapped_model, generation_kwargs=generation_kwargs),\n ):\n yield unwrapped_model\n\n\ndef prepare_deepspeed(model: \"Module\", accelerator: \"Accelerator\"):\n \"\"\"Prepares the model for DeepSpeed inference or evaluation by initializing it with the appropriate configuration.\n\n Adapted from accelerate:\n https://github.com/huggingface/accelerate/blob/739b135f8367becb67ffaada12fe76e3aa60fefd/src/accelerate/accelerator.py#L1473\n \"\"\"\n import deepspeed # local import (instead of top-level) to avoid DS init interfering with other backends (like vllm): https://github.com/deepspeedai/DeepSpeed/issues/7252\n\n deepspeed_plugin = accelerator.state.deepspeed_plugin\n config_kwargs = deepcopy(deepspeed_plugin.deepspeed_config)\n stage = config_kwargs[\"zero_optimization\"][\"stage\"]\n\n if model is not None:\n hidden_size = (\n max(model.config.hidden_sizes)\n if getattr(model.config, \"hidden_sizes\", None)\n else getattr(model.config, \"hidden_size\", None)\n )\n if hidden_size is not None and stage == 3:\n # Note that `stage3_prefetch_bucket_size` can produce DeepSpeed messages like: `Invalidate trace cache\n # @ step 0: expected module 1, but got module 0`\n # This is expected and is not an error, see: https://github.com/microsoft/DeepSpeed/discussions/4081\n config_kwargs.update(\n {\n \"zero_optimization.reduce_bucket_size\": hidden_size * hidden_size,\n \"zero_optimization.stage3_param_persistence_threshold\": 10 * hidden_size,\n \"zero_optimization.stage3_prefetch_bucket_size\": 0.9 * hidden_size * hidden_size,\n }\n )\n\n # If ZeRO-3 is used, we shard both the active and reference model.\n # Otherwise, we assume the reference model fits in memory and is initialized on each device with ZeRO\n # disabled (stage 0)\n if stage != 3:\n config_kwargs[\"zero_optimization\"][\"stage\"] = 0\n model, *_ = deepspeed.initialize(model=model, config=config_kwargs)\n model.eval()\n return model\n\n\ndef prepare_fsdp(model, accelerator: Accelerator) -> FSDP | FSDPModule:\n # Check if the model is already a FSDP model due to `Manual Wrapping` and if so, don't wrap it again\n if not isinstance(model, (FSDP, FSDPModule)):\n fsdp_plugin = accelerator.state.fsdp_plugin\n if fsdp_plugin.fsdp_version == 1:\n accelerator.state.fsdp_plugin.set_auto_wrap_policy(model)\n kwargs = {\n \"sharding_strategy\": fsdp_plugin.sharding_strategy or fsdp_plugin.reshard_after_forward,\n \"cpu_offload\": fsdp_plugin.cpu_offload,\n \"auto_wrap_policy\": fsdp_plugin.auto_wrap_policy,\n \"mixed_precision\": fsdp_plugin.mixed_precision_policy,\n \"sync_module_states\": fsdp_plugin.sync_module_states,\n \"backward_prefetch\": fsdp_plugin.backward_prefetch,\n \"forward_prefetch\": fsdp_plugin.forward_prefetch,\n \"use_orig_params\": fsdp_plugin.use_orig_params,\n \"param_init_fn\": fsdp_plugin.param_init_fn,\n \"ignored_modules\": fsdp_plugin.ignored_modules,\n \"limit_all_gathers\": fsdp_plugin.limit_all_gathers,\n \"device_id\": accelerator.device,\n }\n model = FSDP(model, **kwargs)\n elif fsdp_plugin.fsdp_version == 2:\n from torch.distributed.fsdp import MixedPrecisionPolicy, fully_shard\n\n mesh = getattr(accelerator, \"torch_device_mesh\", None)\n if Version(accelerate.__version__) >= Version(\"1.11.0\"):\n ignored_params = get_parameters_from_modules(fsdp_plugin.ignored_modules, model, accelerator.device)\n else:\n warnings.warn(\n \"FSDP version 2 is being used with accelerate version < 1.11.0, which may lead to incorrect \"\n \"handling of ignored modules. Please upgrade accelerate to v1.11.0 or later for proper support.\"\n )\n ignored_params = None\n fully_shard(\n model,\n reshard_after_forward=fsdp_plugin.reshard_after_forward,\n offload_policy=fsdp_plugin.cpu_offload,\n # `fully_shard` doesn't accept `None` in case of `MixedPrecisionPolicy`\n mp_policy=fsdp_plugin.mixed_precision_policy or MixedPrecisionPolicy(),\n mesh=mesh[tuple(accelerator.parallelism_config.fsdp_dim_names)] if mesh is not None else None,\n ignored_params=ignored_params,\n )\n else:\n raise ValueError(f\"FSDP version {fsdp_plugin.fsdp_version} is not supported.\")\n model.eval()\n return model\n\n\nclass _ForwardRedirection:\n \"\"\"Implements the `forward-redirection`.\n\n Taken from Pytorch-lightning:\n https://github.com/Lightning-AI/pytorch-lightning/blob/02311d03fb982560246eead7c08104481fac9579/src/lightning/pytorch/strategies/strategy.py#L602\n\n A method call to a wrapped module gets rerouted through the wrapper's `forward` method instead.\n\n \"\"\"\n\n def __call__(\n self, wrapper_module: nn.Module, original_module: nn.Module, method: Callable, *args: Any, **kwargs: Any\n ):\n \"\"\"Reroutes a method call through the `wrapper_module`'s `forward` method.\n\n Args:\n wrapper_module: The module that has `original_module` wrapped.\n original_module: The module that was wrapped inside `wrapper_module`.\n method: The method that should be called on the `original_module` after inputs get\n redirected through the `wrapper_module`'s `forward` method.\n *args: The positional arguments to the `method`. They will get passed to a patched\n `forward` method instead.\n **kwargs: The keyword arguments to the `method`. They will get passed to a patched\n `forward` method instead.\n\n \"\"\"\n original_forward = original_module.forward\n\n def wrapped_forward(*_args: Any, **_kwargs: Any) -> Any:\n # Unpatch ourselves immediately before calling the method `method_name`\n # because itself may want to call the real `forward`\n original_module.forward = original_forward # type: ignore[method-assign]\n # Call the actual method e.g. `.training_step(...)`\n out = method(*_args, **_kwargs)\n self.on_after_inner_forward(wrapper_module, original_module)\n return out\n\n # Patch the original_module's forward so we can redirect the arguments back to the real method\n original_module.forward = wrapped_forward # type: ignore[method-assign]\n\n wrapper_output = wrapper_module(*args, **kwargs)\n self.on_after_outer_forward(wrapper_module, original_module)\n return wrapper_output\n\n def on_after_inner_forward(self, wrapper_module: nn.Module, original_module: nn.Module) -> None:\n pass\n\n def on_after_outer_forward(self, wrapper_module: nn.Module, original_module: nn.Module) -> None:\n pass\n\n\n@contextmanager\ndef disable_gradient_checkpointing(model: PreTrainedModel, gradient_checkpointing_kwargs: dict | None = None):\n \"\"\"\n Temporarily disable gradient checkpointing, restoring the previous state afterward.\n\n Args:\n model (`PreTrainedModel`):\n Model for which to temporarily disable gradient checkpointing.\n gradient_checkpointing_kwargs (`dict` or `None`, *optional*):\n Additional kwargs for gradient checkpointing enabling.\n \"\"\"\n was_enabled = model.is_gradient_checkpointing\n if was_enabled:\n model.gradient_checkpointing_disable()\n try:\n yield\n finally:\n if was_enabled:\n model.gradient_checkpointing_enable(gradient_checkpointing_kwargs)\n\n\ndef create_reference_model(\n model: nn.Module, num_shared_layers: int | None = None, pattern: str | None = None\n) -> nn.Module:\n warnings.warn(\n \"The `create_reference_model` function is now located in `trl.experimental.utils`. Please update your \"\n \"imports to `from trl.experimental.utils import create_reference_model`. This import path will be removed in \"\n \"TRL 1.0.0.\",\n FutureWarning,\n stacklevel=2,\n )\n return _create_reference_model(model, num_shared_layers=num_shared_layers, pattern=pattern)\n" }, { "path": "trl/py.typed", "content": "" }, { "path": "trl/rewards/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport sys\nfrom typing import TYPE_CHECKING\n\nfrom .._lazy_module import _LazyModule\n\n\n_import_structure = {\n \"accuracy_rewards\": [\"accuracy_reward\", \"reasoning_accuracy_reward\"],\n \"format_rewards\": [\"think_format_reward\"],\n \"other_rewards\": [\"get_soft_overlong_punishment\"],\n}\n\n\nif TYPE_CHECKING:\n from .accuracy_rewards import accuracy_reward, reasoning_accuracy_reward\n from .format_rewards import think_format_reward\n from .other_rewards import get_soft_overlong_punishment\n\n\nelse:\n sys.modules[__name__] = _LazyModule(__name__, __file__, _import_structure, module_spec=__spec__)\n" }, { "path": "trl/rewards/accuracy_rewards.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport logging\nimport threading\n\nfrom ..import_utils import is_math_verify_available\n\n\nif is_math_verify_available():\n from latex2sympy2_extended import NormalizationConfig\n from math_verify import LatexExtractionConfig, parse, verify\n\n\ndef accuracy_reward(completions: list[list[dict[str, str]]], solution: list[str], **kwargs) -> list[float | None]:\n r\"\"\"\n Reward function that checks if the completion matches the ground truth.\n - If both gold and prediction are parseable โ†’ use math verification.\n - If gold is not parseable โ†’ return `None` to skip the example.\n\n Args:\n completions (`list[list[dict[str, str]]]`):\n List of completions to be evaluated. Each completion must be a list of one message, i.e. a dictionary\n containing the key `\"content\"` with the value being the text of the completion.\n solution: (`list[str]`):\n List of the raw-text solutions to the questions/problems/prompts.\n **kwargs:\n Additional keyword arguments. This function does not use them, but they are required in the function\n signature to ensure compatibility with trainers like [`GRPOTrainer`].\n Example:\n ```python\n >>> from trl.rewards import accuracy_reward\n\n >>> solutions = [r\"\\frac{1}{3}\", r\"\\frac{1}{3}\"]\n >>> completions = [\n ... [{\"role\": \"assistant\", \"content\": r\"My answer is \\boxed{\\frac{1}{3}}\"}],\n ... [{\"role\": \"assistant\", \"content\": r\"My answer is \\boxed{\\frac{1}{2}}\"}],\n ... ]\n >>> accuracy_reward(completions, solutions)\n [1.0, 0.0]\n ```\n \"\"\"\n if not is_math_verify_available():\n raise ImportError(\"Please install the `math_verify` package to use accuracy_reward\")\n\n contents = [completion[0][\"content\"] for completion in completions]\n rewards = []\n\n # math_verify uses signal.alarm() for timeouts, which only works in the main thread.\n # Disable timeouts when running in a non-main thread to avoid ValueError.\n is_main_thread = threading.current_thread() is threading.main_thread()\n parsing_timeout = None if not is_main_thread else 10\n verify_timeout = None if not is_main_thread else 5\n\n # Suppress the \"Timeout is disabled\" warnings from math_verify when we intentionally disable timeouts\n if not is_main_thread:\n logging.getLogger(\"math_verify.parser\").setLevel(logging.ERROR)\n logging.getLogger(\"math_verify.grader\").setLevel(logging.ERROR)\n\n for content, sol in zip(contents, solution, strict=True):\n gold_parsed = parse(sol, parsing_timeout=parsing_timeout)\n if len(gold_parsed) != 0:\n # We require the answer to be provided in correct latex (no malformed operators)\n answer_parsed = parse(\n content,\n extraction_config=[\n LatexExtractionConfig(\n normalization_config=NormalizationConfig(units=True),\n # Ensures that boxed is tried first\n boxed_match_priority=0,\n try_extract_without_anchor=False,\n )\n ],\n extraction_mode=\"first_match\",\n parsing_timeout=parsing_timeout,\n )\n reward = float(verify(gold_parsed, answer_parsed, timeout_seconds=verify_timeout))\n else:\n # If the gold solution cannot be parsed, we assign `None` to skip this example\n reward = None\n rewards.append(reward)\n\n return rewards\n\n\ndef reasoning_accuracy_reward(\n completions: list[list[dict[str, str]]],\n solution: list[str],\n reasoning_delimiters: list[str] | None = None,\n **kwargs,\n) -> list[float | None]:\n r\"\"\"\n Reward function that removes the reasoning content and checks if the final answer matches the ground truth.\n - If both gold and prediction are parseable โ†’ use math verification.\n - If gold is not parseable โ†’ return `None` to skip the example.\n\n Args:\n completions (`list[list[dict[str, str]]]`):\n List of completions to be evaluated. Each completion must be a list of one message, i.e. a dictionary\n containing the key `\"content\"` with the value being the text of the completion.\n solution: (`list[str]`):\n List of the raw-text solutions to the questions/problems/prompts.\n reasoning_delimiters (`list[str]]`, *optional*):\n List of strings indicating where the reasoning content ends. The final answer is assumed to be after the\n last occurrence of any of these delimiters. If `None`, defaults to `[\"\"]`.\n **kwargs:\n Additional keyword arguments. This function does not use them, but they are required in the function\n signature to ensure compatibility with trainers like [`GRPOTrainer`].\n Example:\n ```python\n >>> from trl.rewards import reasoning_accuracy_reward\n\n >>> reasoning_delimiters = [\"\"]\n >>> solutions = [r\"\\frac{1}{3}\", r\"\\frac{1}{3}\", r\"\\frac{1}{3}\"]\n >>> completions = [\n ... [\n ... {\n ... \"role\": \"assistant\",\n ... \"content\": r\" Reasoning content The final answer is \\boxed{\\frac{1}{3}}\",\n ... }\n ... ],\n ... [\n ... {\n ... \"role\": \"assistant\",\n ... \"content\": r\" Reasoning content The final answer is \\boxed{\\frac{1}{2}}\",\n ... }\n ... ],\n ... [\n ... {\n ... \"role\": \"assistant\",\n ... \"content\": r\" Reasoning content with partial answers \\boxed{\\frac{1}{3}} but no final answer\",\n ... }\n ... ],\n ... ]\n >>> reasoning_accuracy_reward(completions, solutions, reasoning_delimiters=reasoning_delimiters)\n [1.0, 0.0, 0.0]\n ```\n \"\"\"\n if not is_math_verify_available():\n raise ImportError(\"Please install the `math_verify` package to use reasoning_accuracy_reward\")\n\n if reasoning_delimiters is None:\n # Use sensible defaults for majority of reasoning models\n reasoning_delimiters = [\"\"]\n\n rewards = []\n contents = [completion[0][\"content\"] for completion in completions]\n\n # math_verify uses signal.alarm() for timeouts, which only works in the main thread.\n # Disable timeouts when running in a non-main thread to avoid ValueError.\n is_main_thread = threading.current_thread() is threading.main_thread()\n parsing_timeout = None if not is_main_thread else 10\n verify_timeout = None if not is_main_thread else 5\n\n # Suppress the \"Timeout is disabled\" warnings from math_verify when we intentionally disable timeouts\n if not is_main_thread:\n logging.getLogger(\"math_verify.parser\").setLevel(logging.ERROR)\n logging.getLogger(\"math_verify.grader\").setLevel(logging.ERROR)\n\n for content, sol in zip(contents, solution, strict=True):\n # Split final answer from reasoning content\n is_reasoning_complete = False\n for delim in reasoning_delimiters:\n if delim in content:\n content = content.split(delim)[-1]\n is_reasoning_complete = True\n break\n if not is_reasoning_complete:\n # We assign zero reward instead of `None` to penalize incomplete reasoning\n rewards.append(0.0)\n continue\n\n gold_parsed = parse(sol, parsing_timeout=parsing_timeout)\n if len(gold_parsed) != 0:\n # We require the answer to be provided in correct latex (no malformed operators)\n answer_parsed = parse(\n content,\n extraction_config=[\n LatexExtractionConfig(\n boxed_match_priority=0,\n normalization_config=NormalizationConfig(\n units=True,\n ),\n try_extract_without_anchor=False,\n )\n ],\n extraction_mode=\"first_match\",\n parsing_timeout=parsing_timeout,\n )\n reward = float(verify(gold_parsed, answer_parsed, timeout_seconds=verify_timeout))\n else:\n # If the gold solution cannot be parsed, we assign `None` to skip this example\n reward = None\n rewards.append(reward)\n\n return rewards\n" }, { "path": "trl/rewards/format_rewards.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport re\n\n\ndef think_format_reward(completions: list[list[dict[str, str]]], **kwargs) -> list[float]:\n r\"\"\"\n Reward function that checks if the reasoning process is enclosed within `\"\"` and `\"\"` tags. The\n function returns a reward of 1.0 if the format is correct, otherwise 0.0.\n\n Args:\n completions (`list[list[dict[str, str]]]`):\n List of completions to be evaluated. Each completion must be a list of one message, i.e. a dictionary\n containing the key `\"content\"` with the value being the text of the completion.\n **kwargs:\n Additional keyword arguments. This function does not use them, but they are required in the function\n signature to ensure compatibility with trainers like [`GRPOTrainer`].\n\n Returns:\n `list[float]`:\n A list of rewards, where each reward is 1.0 if the completion matches the expected format, otherwise 0.0.\n\n Example:\n ```python\n >>> from trl.rewards import think_format_reward\n\n >>> completions = [\n ... [{\"content\": \"\\nThis is my reasoning.\\n\\nThis is my answer.\"}],\n ... [{\"content\": \"\\nThis is my reasoning.\\nThis is my answer.\"}],\n ... ]\n >>> think_format_reward(completions)\n [1.0, 0.0]\n ```\n \"\"\"\n pattern = r\"^(?!.*)(.*?).*$\"\n completion_contents = [completion[0][\"content\"] for completion in completions]\n matches = [re.match(pattern, content, re.DOTALL | re.MULTILINE) for content in completion_contents]\n return [1.0 if match else 0.0 for match in matches]\n" }, { "path": "trl/rewards/other_rewards.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom collections.abc import Callable\n\n\ndef get_soft_overlong_punishment(max_completion_len: int, soft_punish_cache: int) -> Callable:\n # docstyle-ignore\n r\"\"\"\n Reward function that penalizes overlong completions. It is used to penalize overlong completions, but not to reward\n shorter completions. Reference: Eq. (13) from the DAPO paper (https://huggingface.co/papers/2503.14476)\n\n $$\n R_{\\text{length}}(y) = \\begin{cases}\n 0, & |y| \\le L_{\\max} - L_{\\text{cache}} \\\\\n \\dfrac{(L_{\\max} - L_{\\text{cache}}) - |y|}{L_{\\text{cache}}}, & L_{\\max} - L_{\\text{cache}} < |y| \\le L_{\\max} \\\\\n -1, & L_{\\max} < |y|\n \\end{cases}\n $$\n\n Args:\n max_completion_len (`int`):\n Maximum length of the completion, \\( L_{\\max} \\).\n soft_punish_cache (`int`):\n Minimum length of the completion, \\( L_{\\text{cache}} \\). If set to `0`, no minimum length is applied.\n\n Example:\n ```python\n from trl.rewards import get_soft_overlong_punishment\n\n soft_overlong_punishment = get_soft_overlong_punishment(max_completion_len=100, soft_punish_cache=20)\n completion_ids = [[1] * 90] # simulating a completion with 90 tokens. 90 is between 80 and 100.\n rewards = soft_overlong_punishment(completion_ids)\n print(rewards) # [-0.5]\n ```\n \"\"\"\n\n def soft_overlong_punishment_reward(completion_ids: list[list[int]], **kwargs) -> list[float]:\n \"\"\"Reward function that penalizes overlong completions.\"\"\"\n rewards = []\n for ids in completion_ids:\n completion_length = len(ids)\n if completion_length <= max_completion_len - soft_punish_cache:\n rewards.append(0.0)\n elif max_completion_len - soft_punish_cache < completion_length <= max_completion_len:\n rewards.append((max_completion_len - soft_punish_cache - completion_length) / soft_punish_cache)\n else:\n rewards.append(-1.0)\n return rewards\n\n return soft_overlong_punishment_reward\n" }, { "path": "trl/scripts/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom typing import TYPE_CHECKING\n\nfrom .._lazy_module import _LazyModule\n\n\n_import_structure = {\n \"utils\": [\"DatasetMixtureConfig\", \"ScriptArguments\", \"TrlParser\", \"get_dataset\", \"init_zero_verbose\"],\n}\n\nif TYPE_CHECKING:\n from .utils import DatasetMixtureConfig, ScriptArguments, TrlParser, get_dataset, init_zero_verbose\nelse:\n import sys\n\n sys.modules[__name__] = _LazyModule(__name__, globals()[\"__file__\"], _import_structure, module_spec=__spec__)\n" }, { "path": "trl/scripts/_hf_argparser.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\n# Copied from: https://github.com/huggingface/transformers/blob/3a275d3581c0ecf962f7412aa764c2047331fd6b/src/transformers/hf_argparser.py\n# This avoids an upstream latency issue: https://github.com/huggingface/transformers/issues/44273\n# - Moved yaml import inside function\n\n\nimport dataclasses\nimport json\nimport os\nimport sys\nimport types\nfrom argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, ArgumentTypeError\nfrom collections.abc import Callable, Iterable\nfrom copy import copy\nfrom enum import Enum\nfrom inspect import isclass\nfrom pathlib import Path\nfrom typing import Any, Literal, NewType, Union, get_type_hints\n\n\nDataClass = NewType(\"DataClass\", Any)\nDataClassType = NewType(\"DataClassType\", Any)\n\n\n# From https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse\ndef string_to_bool(v):\n if isinstance(v, bool):\n return v\n if v.lower() in (\"yes\", \"true\", \"t\", \"y\", \"1\"):\n return True\n elif v.lower() in (\"no\", \"false\", \"f\", \"n\", \"0\"):\n return False\n else:\n raise ArgumentTypeError(\n f\"Truthy value expected: got {v} but expected one of yes/no, true/false, t/f, y/n, 1/0 (case insensitive).\"\n )\n\n\ndef make_choice_type_function(choices: list) -> Callable[[str], Any]:\n \"\"\"\n Creates a mapping function from each choices string representation to the actual value. Used to support multiple\n value types for a single argument.\n\n Args:\n choices (list): List of choices.\n\n Returns:\n Callable[[str], Any]: Mapping function from string representation to actual value for each choice.\n \"\"\"\n str_to_choice = {str(choice): choice for choice in choices}\n return lambda arg: str_to_choice.get(arg, arg)\n\n\ndef HfArg(\n *,\n aliases: str | list[str] | None = None,\n help: str | None = None,\n default: Any = dataclasses.MISSING,\n default_factory: Callable[[], Any] = dataclasses.MISSING,\n metadata: dict | None = None,\n **kwargs,\n) -> dataclasses.Field:\n \"\"\"Argument helper enabling a concise syntax to create dataclass fields for parsing with `HfArgumentParser`.\n\n Example comparing the use of `HfArg` and `dataclasses.field`:\n ```\n @dataclass\n class Args:\n regular_arg: str = dataclasses.field(default=\"Huggingface\", metadata={\"aliases\": [\"--example\", \"-e\"], \"help\": \"This syntax could be better!\"})\n hf_arg: str = HfArg(default=\"Huggingface\", aliases=[\"--example\", \"-e\"], help=\"What a nice syntax!\")\n ```\n\n Args:\n aliases (Union[str, list[str]], optional):\n Single string or list of strings of aliases to pass on to argparse, e.g. `aliases=[\"--example\", \"-e\"]`.\n Defaults to None.\n help (str, optional): Help string to pass on to argparse that can be displayed with --help. Defaults to None.\n default (Any, optional):\n Default value for the argument. If not default or default_factory is specified, the argument is required.\n Defaults to dataclasses.MISSING.\n default_factory (Callable[[], Any], optional):\n The default_factory is a 0-argument function called to initialize a field's value. It is useful to provide\n default values for mutable types, e.g. lists: `default_factory=list`. Mutually exclusive with `default=`.\n Defaults to dataclasses.MISSING.\n metadata (dict, optional): Further metadata to pass on to `dataclasses.field`. Defaults to None.\n\n Returns:\n Field: A `dataclasses.Field` with the desired properties.\n \"\"\"\n if metadata is None:\n # Important, don't use as default param in function signature because dict is mutable and shared across function calls\n metadata = {}\n if aliases is not None:\n metadata[\"aliases\"] = aliases\n if help is not None:\n metadata[\"help\"] = help\n\n return dataclasses.field(metadata=metadata, default=default, default_factory=default_factory, **kwargs)\n\n\nclass HfArgumentParser(ArgumentParser):\n \"\"\"\n This subclass of `argparse.ArgumentParser` uses type hints on dataclasses to generate arguments.\n\n The class is designed to play well with the native argparse. In particular, you can add more (non-dataclass backed)\n arguments to the parser after initialization and you'll get the output back after parsing as an additional\n namespace. Optional: To create sub argument groups use the `_argument_group_name` attribute in the dataclass.\n\n Args:\n dataclass_types (`DataClassType` or `Iterable[DataClassType]`, *optional*):\n Dataclass type, or list of dataclass types for which we will \"fill\" instances with the parsed args.\n kwargs (`dict[str, Any]`, *optional*):\n Passed to `argparse.ArgumentParser()` in the regular way.\n \"\"\"\n\n dataclass_types: Iterable[DataClassType]\n\n def __init__(self, dataclass_types: DataClassType | Iterable[DataClassType] | None = None, **kwargs):\n # Make sure dataclass_types is an iterable\n if dataclass_types is None:\n dataclass_types = []\n elif not isinstance(dataclass_types, Iterable):\n dataclass_types = [dataclass_types]\n\n # To make the default appear when using --help\n if \"formatter_class\" not in kwargs:\n kwargs[\"formatter_class\"] = ArgumentDefaultsHelpFormatter\n super().__init__(**kwargs)\n if dataclasses.is_dataclass(dataclass_types):\n dataclass_types = [dataclass_types]\n self.dataclass_types = list(dataclass_types)\n for dtype in self.dataclass_types:\n self._add_dataclass_arguments(dtype)\n\n @staticmethod\n def _parse_dataclass_field(parser: ArgumentParser, field: dataclasses.Field):\n # Long-option strings are conventionlly separated by hyphens rather\n # than underscores, e.g., \"--long-format\" rather than \"--long_format\".\n # Argparse converts hyphens to underscores so that the destination\n # string is a valid attribute name. Hf_argparser should do the same.\n long_options = [f\"--{field.name}\"]\n if \"_\" in field.name:\n long_options.append(f\"--{field.name.replace('_', '-')}\")\n\n kwargs = field.metadata.copy()\n # field.metadata is not used at all by Data Classes,\n # it is provided as a third-party extension mechanism.\n if isinstance(field.type, str):\n raise RuntimeError(\n \"Unresolved type detected, which should have been done with the help of \"\n \"`typing.get_type_hints` method by default\"\n )\n\n aliases = kwargs.pop(\"aliases\", [])\n if isinstance(aliases, str):\n aliases = [aliases]\n\n origin_type = getattr(field.type, \"__origin__\", field.type)\n if origin_type is Union or (hasattr(types, \"UnionType\") and isinstance(origin_type, types.UnionType)):\n if str not in field.type.__args__ and (\n len(field.type.__args__) != 2 or type(None) not in field.type.__args__\n ):\n raise ValueError(\n \"Only `Union[X, NoneType]` (i.e., `Optional[X]`) is allowed for `Union` because\"\n \" the argument parser only supports one type per argument.\"\n f\" Problem encountered in field '{field.name}'.\"\n )\n if type(None) not in field.type.__args__:\n # filter `str` in Union\n field.type = field.type.__args__[0] if field.type.__args__[1] is str else field.type.__args__[1]\n origin_type = getattr(field.type, \"__origin__\", field.type)\n elif bool not in field.type.__args__:\n # filter `NoneType` in Union (except for `Union[bool, NoneType]`)\n field.type = (\n field.type.__args__[0] if isinstance(None, field.type.__args__[1]) else field.type.__args__[1]\n )\n origin_type = getattr(field.type, \"__origin__\", field.type)\n\n # A variable to store kwargs for a boolean field, if needed\n # so that we can init a `no_*` complement argument (see below)\n bool_kwargs = {}\n if origin_type is Literal or (isinstance(field.type, type) and issubclass(field.type, Enum)):\n if origin_type is Literal:\n kwargs[\"choices\"] = field.type.__args__\n else:\n kwargs[\"choices\"] = [x.value for x in field.type]\n\n kwargs[\"type\"] = make_choice_type_function(kwargs[\"choices\"])\n\n if field.default is not dataclasses.MISSING:\n kwargs[\"default\"] = field.default\n else:\n kwargs[\"required\"] = True\n elif field.type is bool or field.type == bool | None:\n # Copy the correct kwargs to use to instantiate a `no_*` complement argument below.\n # We do not initialize it here because the `no_*` alternative must be instantiated after the real argument\n bool_kwargs = copy(kwargs)\n\n # Hack because type=bool in argparse does not behave as we want.\n kwargs[\"type\"] = string_to_bool\n if field.type is bool or (field.default is not None and field.default is not dataclasses.MISSING):\n # Default value is False if we have no default when of type bool.\n default = False if field.default is dataclasses.MISSING else field.default\n # This is the value that will get picked if we don't include --{field.name} in any way\n kwargs[\"default\"] = default\n # This tells argparse we accept 0 or 1 value after --{field.name}\n kwargs[\"nargs\"] = \"?\"\n # This is the value that will get picked if we do --{field.name} (without value)\n kwargs[\"const\"] = True\n elif isclass(origin_type) and issubclass(origin_type, list):\n kwargs[\"type\"] = field.type.__args__[0]\n kwargs[\"nargs\"] = \"+\"\n if field.default_factory is not dataclasses.MISSING:\n kwargs[\"default\"] = field.default_factory()\n elif field.default is dataclasses.MISSING:\n kwargs[\"required\"] = True\n else:\n kwargs[\"type\"] = field.type\n if field.default is not dataclasses.MISSING:\n kwargs[\"default\"] = field.default\n elif field.default_factory is not dataclasses.MISSING:\n kwargs[\"default\"] = field.default_factory()\n else:\n kwargs[\"required\"] = True\n parser.add_argument(*long_options, *aliases, **kwargs)\n\n # Add a complement `no_*` argument for a boolean field AFTER the initial field has already been added.\n # Order is important for arguments with the same destination!\n # We use a copy of earlier kwargs because the original kwargs have changed a lot before reaching down\n # here and we do not need those changes/additional keys.\n if field.default is True and (field.type is bool or field.type == bool | None):\n bool_kwargs[\"default\"] = False\n parser.add_argument(\n f\"--no_{field.name}\",\n f\"--no-{field.name.replace('_', '-')}\",\n action=\"store_false\",\n dest=field.name,\n **bool_kwargs,\n )\n\n def _add_dataclass_arguments(self, dtype: DataClassType):\n if hasattr(dtype, \"_argument_group_name\"):\n parser = self.add_argument_group(dtype._argument_group_name)\n else:\n parser = self\n\n try:\n type_hints: dict[str, type] = get_type_hints(dtype)\n except NameError:\n raise RuntimeError(\n f\"Type resolution failed for {dtype}. Try declaring the class in global scope or \"\n \"removing line of `from __future__ import annotations` which opts in Postponed \"\n \"Evaluation of Annotations (PEP 563)\"\n ) from None\n\n for field in dataclasses.fields(dtype):\n if not field.init:\n continue\n field.type = type_hints[field.name]\n self._parse_dataclass_field(parser, field)\n\n def parse_args_into_dataclasses(\n self,\n args=None,\n return_remaining_strings=False,\n look_for_args_file=True,\n args_filename=None,\n args_file_flag=None,\n ) -> tuple[DataClass, ...]:\n \"\"\"\n Parse command-line args into instances of the specified dataclass types.\n\n This relies on argparse's `ArgumentParser.parse_known_args`. See the doc at:\n docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_args\n\n Args:\n args:\n List of strings to parse. The default is taken from sys.argv. (same as argparse.ArgumentParser)\n return_remaining_strings:\n If true, also return a list of remaining argument strings.\n look_for_args_file:\n If true, will look for a \".args\" file with the same base name as the entry point script for this\n process, and will append its potential content to the command line args.\n args_filename:\n If not None, will uses this file instead of the \".args\" file specified in the previous argument.\n args_file_flag:\n If not None, will look for a file in the command-line args specified with this flag. The flag can be\n specified multiple times and precedence is determined by the order (last one wins).\n\n Returns:\n Tuple consisting of:\n\n - the dataclass instances in the same order as they were passed to the initializer.abspath\n - if applicable, an additional namespace for more (non-dataclass backed) arguments added to the parser\n after initialization.\n - The potential list of remaining argument strings. (same as argparse.ArgumentParser.parse_known_args)\n \"\"\"\n\n if args_file_flag or args_filename or (look_for_args_file and len(sys.argv)):\n args_files = []\n\n if args_filename:\n args_files.append(Path(args_filename))\n elif look_for_args_file and len(sys.argv):\n args_files.append(Path(sys.argv[0]).with_suffix(\".args\"))\n\n # args files specified via command line flag should overwrite default args files so we add them last\n if args_file_flag:\n # Create special parser just to extract the args_file_flag values\n args_file_parser = ArgumentParser()\n args_file_parser.add_argument(args_file_flag, type=str, action=\"append\")\n\n # Use only remaining args for further parsing (remove the args_file_flag)\n cfg, args = args_file_parser.parse_known_args(args=args)\n cmd_args_file_paths = vars(cfg).get(args_file_flag.lstrip(\"-\"), None)\n\n if cmd_args_file_paths:\n args_files.extend([Path(p) for p in cmd_args_file_paths])\n\n file_args = []\n for args_file in args_files:\n if args_file.exists():\n file_args += args_file.read_text().split()\n\n # in case of duplicate arguments the last one has precedence\n # args specified via the command line should overwrite args from files, so we add them last\n args = file_args + args if args is not None else file_args + sys.argv[1:]\n namespace, remaining_args = self.parse_known_args(args=args)\n outputs = []\n for dtype in self.dataclass_types:\n keys = {f.name for f in dataclasses.fields(dtype) if f.init}\n inputs = {k: v for k, v in vars(namespace).items() if k in keys}\n for k in keys:\n delattr(namespace, k)\n obj = dtype(**inputs)\n outputs.append(obj)\n if len(namespace.__dict__) > 0:\n # additional namespace.\n outputs.append(namespace)\n if return_remaining_strings:\n return (*outputs, remaining_args)\n else:\n if remaining_args:\n raise ValueError(f\"Some specified arguments are not used by the HfArgumentParser: {remaining_args}\")\n\n return (*outputs,)\n\n def parse_dict(self, args: dict[str, Any], allow_extra_keys: bool = False) -> tuple[DataClass, ...]:\n \"\"\"\n Alternative helper method that does not use `argparse` at all, instead uses a dict and populating the dataclass\n types.\n\n Args:\n args (`dict`):\n dict containing config values\n allow_extra_keys (`bool`, *optional*, defaults to `False`):\n Defaults to False. If False, will raise an exception if the dict contains keys that are not parsed.\n\n Returns:\n Tuple consisting of:\n\n - the dataclass instances in the same order as they were passed to the initializer.\n \"\"\"\n unused_keys = set(args.keys())\n outputs = []\n for dtype in self.dataclass_types:\n keys = {f.name for f in dataclasses.fields(dtype) if f.init}\n inputs = {k: v for k, v in args.items() if k in keys}\n unused_keys.difference_update(inputs.keys())\n obj = dtype(**inputs)\n outputs.append(obj)\n if not allow_extra_keys and unused_keys:\n raise ValueError(f\"Some keys are not used by the HfArgumentParser: {sorted(unused_keys)}\")\n return tuple(outputs)\n\n def parse_json_file(self, json_file: str | os.PathLike, allow_extra_keys: bool = False) -> tuple[DataClass, ...]:\n \"\"\"\n Alternative helper method that does not use `argparse` at all, instead loading a json file and populating the\n dataclass types.\n\n Args:\n json_file (`str` or `os.PathLike`):\n File name of the json file to parse\n allow_extra_keys (`bool`, *optional*, defaults to `False`):\n Defaults to False. If False, will raise an exception if the json file contains keys that are not\n parsed.\n\n Returns:\n Tuple consisting of:\n\n - the dataclass instances in the same order as they were passed to the initializer.\n \"\"\"\n with open(Path(json_file), encoding=\"utf-8\") as open_json_file:\n data = json.loads(open_json_file.read())\n outputs = self.parse_dict(data, allow_extra_keys=allow_extra_keys)\n return tuple(outputs)\n\n def parse_yaml_file(self, yaml_file: str | os.PathLike, allow_extra_keys: bool = False) -> tuple[DataClass, ...]:\n \"\"\"\n Alternative helper method that does not use `argparse` at all, instead loading a yaml file and populating the\n dataclass types.\n\n Args:\n yaml_file (`str` or `os.PathLike`):\n File name of the yaml file to parse\n allow_extra_keys (`bool`, *optional*, defaults to `False`):\n Defaults to False. If False, will raise an exception if the json file contains keys that are not\n parsed.\n\n Returns:\n Tuple consisting of:\n\n - the dataclass instances in the same order as they were passed to the initializer.\n \"\"\"\n import yaml\n\n outputs = self.parse_dict(yaml.safe_load(Path(yaml_file).read_text()), allow_extra_keys=allow_extra_keys)\n return tuple(outputs)\n" }, { "path": "trl/scripts/dpo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\n# Full training\n```bash\npython trl/scripts/dpo.py \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --learning_rate 5.0e-7 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 2 \\\n --max_steps 1000 \\\n --gradient_accumulation_steps 8 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --output_dir Qwen2-0.5B-DPO \\\n --no_remove_unused_columns\n```\n\n# LoRA:\n```bash\npython trl/scripts/dpo.py \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --learning_rate 5.0e-6 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 2 \\\n --max_steps 1000 \\\n --gradient_accumulation_steps 8 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --output_dir Qwen2-0.5B-DPO \\\n --no_remove_unused_columns \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16\n```\n\"\"\"\n\nimport argparse\nimport os\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef main(script_args, training_args, model_args, dataset_args):\n import torch\n from accelerate import logging\n from datasets import load_dataset\n from transformers import AutoModelForCausalLM\n\n from trl import DPOTrainer, get_dataset, get_kbit_device_map, get_peft_config, get_quantization_config\n\n logger = logging.get_logger(__name__)\n\n ################\n # Model\n ###################\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code, **model_kwargs\n )\n peft_config = get_peft_config(model_args)\n if script_args.ignore_bias_buffers:\n # torch distributed hack\n model._ddp_params_and_buffers_to_ignore = [\n name for name, buffer in model.named_buffers() if buffer.dtype == torch.bool\n ]\n\n # Load the dataset\n if dataset_args.datasets and script_args.dataset_name:\n logger.warning(\n \"Both `datasets` and `dataset_name` are provided. The `datasets` argument will be used to load the \"\n \"dataset and `dataset_name` will be ignored.\"\n )\n dataset = get_dataset(dataset_args)\n elif dataset_args.datasets and not script_args.dataset_name:\n dataset = get_dataset(dataset_args)\n elif not dataset_args.datasets and script_args.dataset_name:\n dataset = load_dataset(\n script_args.dataset_name, name=script_args.dataset_config, streaming=script_args.dataset_streaming\n )\n else:\n raise ValueError(\"Either `datasets` or `dataset_name` must be provided.\")\n\n # Initialize the DPO trainer\n trainer = DPOTrainer(\n model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=peft_config,\n )\n\n # Train the model\n trainer.train()\n\n # Log training complete\n trainer.accelerator.print(\"โœ… Training completed.\")\n\n if training_args.eval_strategy != \"no\":\n metrics = trainer.evaluate()\n trainer.log_metrics(\"eval\", metrics)\n trainer.save_metrics(\"eval\", metrics)\n\n # Save and push to Hub\n trainer.save_model(training_args.output_dir)\n trainer.accelerator.print(f\"๐Ÿ’พ Model saved to {training_args.output_dir}.\")\n\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n trainer.accelerator.print(f\"๐Ÿค— Model pushed to the Hub in https://huggingface.co/{trainer.hub_model_id}.\")\n\n\ndef make_parser(subparsers: argparse._SubParsersAction | None = None, prog: str | None = None):\n from trl import DatasetMixtureConfig, DPOConfig, ModelConfig, ScriptArguments, TrlParser\n\n dataclass_types = (ScriptArguments, DPOConfig, ModelConfig, DatasetMixtureConfig)\n if subparsers is not None:\n parser = subparsers.add_parser(\"dpo\", help=\"Run the DPO training script\", dataclass_types=dataclass_types)\n else:\n parser = TrlParser(dataclass_types, prog=prog)\n return parser\n\n\nif __name__ == \"__main__\":\n parser = make_parser()\n script_args, training_args, model_args, dataset_args = parser.parse_args_and_config(fail_with_unknown_args=False)\n main(script_args, training_args, model_args, dataset_args)\n" }, { "path": "trl/scripts/env.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# ]\n# ///\n\nimport os\nimport platform\nfrom importlib.metadata import version\n\n\ndef print_env():\n import torch\n from accelerate.commands.config import default_config_file, load_config_from_file\n from transformers import is_bitsandbytes_available\n from transformers.utils import is_openai_available, is_peft_available\n\n from trl import __version__\n from trl.import_utils import (\n is_deepspeed_available,\n is_liger_kernel_available,\n is_llm_blender_available,\n is_vllm_available,\n )\n from trl.scripts.utils import get_git_commit_hash\n\n devices = None\n if torch.cuda.is_available():\n devices = [torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())]\n elif torch.backends.mps.is_available():\n devices = [\"MPS\"]\n elif torch.xpu.is_available():\n devices = [torch.xpu.get_device_name(i) for i in range(torch.xpu.device_count())]\n\n accelerate_config = accelerate_config_str = \"not found\"\n\n # Get the default from the config file.\n if os.path.isfile(default_config_file):\n accelerate_config = load_config_from_file(default_config_file).to_dict()\n\n accelerate_config_str = (\n \"\\n\" + \"\\n\".join([f\" - {prop}: {val}\" for prop, val in accelerate_config.items()])\n if isinstance(accelerate_config, dict)\n else accelerate_config\n )\n\n commit_hash = get_git_commit_hash(\"trl\")\n\n info = {\n \"Platform\": platform.platform(),\n \"Python version\": platform.python_version(),\n \"TRL version\": f\"{__version__}+{commit_hash[:7]}\" if commit_hash else __version__,\n \"PyTorch version\": version(\"torch\"),\n \"accelerator(s)\": \", \".join(devices) if devices is not None else \"cpu\",\n \"Transformers version\": version(\"transformers\"),\n \"Accelerate version\": version(\"accelerate\"),\n \"Accelerate config\": accelerate_config_str,\n \"Datasets version\": version(\"datasets\"),\n \"HF Hub version\": version(\"huggingface_hub\"),\n \"bitsandbytes version\": version(\"bitsandbytes\") if is_bitsandbytes_available() else \"not installed\",\n \"DeepSpeed version\": version(\"deepspeed\") if is_deepspeed_available() else \"not installed\",\n \"Liger-Kernel version\": version(\"liger_kernel\") if is_liger_kernel_available() else \"not installed\",\n \"LLM-Blender version\": version(\"llm_blender\") if is_llm_blender_available() else \"not installed\",\n \"OpenAI version\": version(\"openai\") if is_openai_available() else \"not installed\",\n \"PEFT version\": version(\"peft\") if is_peft_available() else \"not installed\",\n \"vLLM version\": version(\"vllm\") if is_vllm_available() else \"not installed\",\n }\n\n info_str = \"\\n\".join([f\"- {prop}: {val}\" for prop, val in info.items()])\n print(f\"\\nCopy-paste the following information when reporting an issue:\\n\\n{info_str}\\n\") # noqa\n\n\nif __name__ == \"__main__\":\n print_env()\n" }, { "path": "trl/scripts/grpo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\nimport argparse\nimport importlib\nimport os\nimport sys\nfrom dataclasses import dataclass, field\n\nfrom trl import ScriptArguments\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\n@dataclass\nclass GRPOScriptArguments(ScriptArguments):\n \"\"\"\n Script arguments for the GRPO training script.\n\n Args:\n reward_model_name_or_path (`str`, *optional*):\n Reward model id of a pretrained model hosted inside a model repo on huggingface.co or local path to a\n directory containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`].\n reward_funcs (`list[str]`, *optional*):\n Reward functions to use. Supported values are:\n - `\"accuracy_reward\"`\n - `\"reasoning_accuracy_reward\"`\n - `\"think_format_reward\"`\n - `\"get_soft_overlong_punishment\"` (used value are `max_completion_len=1280`, `soft_punish_cache=256`)\n - any dotted import path \" (e.g., `'my_lib.rewards.custom_reward'`).\n \"\"\"\n\n reward_model_name_or_path: str | None = field(\n default=None,\n metadata={\n \"help\": \"Reward model id of a pretrained model hosted inside a model repo on huggingface.co or \"\n \"local path to a directory containing model weights saved using `PreTrainedModel.save_pretrained`.\"\n },\n )\n reward_funcs: list[str] | None = field(\n default=None,\n metadata={\n \"help\": \"Reward functions to use. Supported values are: `accuracy_reward`, `reasoning_accuracy_reward`, `think_format_reward`, \"\n \"`get_soft_overlong_punishment` (used values are `max_completion_len=1280`, `soft_punish_cache=256`), or \"\n \"any dotted import path (e.g., `'my_lib.rewards.custom_reward'`).\"\n },\n )\n\n\ndef main(script_args, training_args, model_args, dataset_args):\n import torch\n from accelerate import logging\n from datasets import load_dataset\n\n from trl import GRPOTrainer, get_dataset, get_kbit_device_map, get_peft_config, get_quantization_config\n from trl.rewards import (\n accuracy_reward,\n get_soft_overlong_punishment,\n reasoning_accuracy_reward,\n think_format_reward,\n )\n\n logger = logging.get_logger(__name__)\n\n reward_funcs_registry = {\n \"accuracy_reward\": accuracy_reward,\n \"reasoning_accuracy_reward\": reasoning_accuracy_reward,\n \"think_format_reward\": think_format_reward,\n \"get_soft_overlong_punishment\": get_soft_overlong_punishment(max_completion_len=1280, soft_punish_cache=256),\n }\n\n # Get the reward models and functions\n reward_funcs = []\n if script_args.reward_model_name_or_path:\n reward_funcs.append(script_args.reward_model_name_or_path)\n\n if script_args.reward_funcs:\n for func_name in script_args.reward_funcs:\n if func_name in reward_funcs_registry:\n reward_funcs.append(reward_funcs_registry[func_name])\n elif \".\" in func_name:\n module_path, func_name = func_name.rsplit(\".\", 1)\n sys.path.insert(0, os.getcwd())\n module = importlib.import_module(module_path)\n reward_func = getattr(module, func_name)\n reward_funcs.append(reward_func)\n else:\n raise ValueError(\n f\"Could not load reward function '{func_name}'. Expected one of \"\n f\"{list(reward_funcs_registry.keys())} or a valid import path.\"\n )\n dtype = model_args.dtype if model_args.dtype in [\"auto\", None] else getattr(torch, model_args.dtype)\n\n model_kwargs = dict(\n revision=model_args.model_revision,\n attn_implementation=model_args.attn_implementation,\n dtype=dtype,\n )\n quantization_config = get_quantization_config(model_args)\n\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n training_args.model_init_kwargs = model_kwargs\n\n # Load the dataset\n if dataset_args.datasets and script_args.dataset_name:\n logger.warning(\n \"Both `datasets` and `dataset_name` are provided. The `datasets` argument will be used to load the \"\n \"dataset and `dataset_name` will be ignored.\"\n )\n dataset = get_dataset(dataset_args)\n elif dataset_args.datasets and not script_args.dataset_name:\n dataset = get_dataset(dataset_args)\n elif not dataset_args.datasets and script_args.dataset_name:\n dataset = load_dataset(\n script_args.dataset_name, name=script_args.dataset_config, streaming=script_args.dataset_streaming\n )\n else:\n raise ValueError(\"Either `datasets` or `dataset_name` must be provided.\")\n\n # Initialize the GRPO trainer\n trainer = GRPOTrainer(\n model=model_args.model_name_or_path,\n reward_funcs=reward_funcs,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=get_peft_config(model_args),\n )\n\n # Train the model\n trainer.train()\n\n # Log training complete\n trainer.accelerator.print(\"โœ… Training completed.\")\n\n # Save and push to Hub\n trainer.save_model(training_args.output_dir)\n trainer.accelerator.print(f\"๐Ÿ’พ Model saved to {training_args.output_dir}.\")\n\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n trainer.accelerator.print(f\"๐Ÿค— Model pushed to the Hub in https://huggingface.co/{trainer.hub_model_id}.\")\n\n\ndef make_parser(subparsers: argparse._SubParsersAction | None = None, prog: str | None = None):\n from trl import DatasetMixtureConfig, GRPOConfig, ModelConfig, TrlParser\n\n dataclass_types = (GRPOScriptArguments, GRPOConfig, ModelConfig, DatasetMixtureConfig)\n if subparsers is not None:\n parser = subparsers.add_parser(\"grpo\", help=\"Run the GRPO training script\", dataclass_types=dataclass_types)\n else:\n parser = TrlParser(dataclass_types, prog=prog)\n return parser\n\n\nif __name__ == \"__main__\":\n parser = make_parser()\n script_args, training_args, model_args, dataset_args = parser.parse_args_and_config(fail_with_unknown_args=False)\n main(script_args, training_args, model_args, dataset_args)\n" }, { "path": "trl/scripts/kto.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\nRun the KTO training script with the commands below. In general, the optimal configuration for KTO will be similar to\nthat of DPO.\n\n# Full training:\n```bash\npython trl/scripts/kto.py \\\n --dataset_name trl-lib/kto-mix-14k \\\n --model_name_or_path=trl-lib/qwen1.5-1.8b-sft \\\n --per_device_train_batch_size 16 \\\n --num_train_epochs 1 \\\n --learning_rate 5e-7 \\\n --lr_scheduler_type=cosine \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 500 \\\n --output_dir=kto-aligned-model \\\n --warmup_steps 0.1 \\\n --logging_first_step\n```\n\n# QLoRA:\n```bash\n# QLoRA:\npython trl/scripts/kto.py \\\n --dataset_name trl-lib/kto-mix-14k \\\n --model_name_or_path=trl-lib/qwen1.5-1.8b-sft \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 5e-7 \\\n --lr_scheduler_type=cosine \\\n --gradient_accumulation_steps 1 \\\n --eval_steps 500 \\\n --output_dir=kto-aligned-model-lora \\\n --warmup_steps 0.1 \\\n --logging_first_step \\\n --use_peft \\\n --load_in_4bit \\\n --lora_target_modules=all-linear \\\n --lora_r=16 \\\n --lora_alpha=16\n```\n\"\"\"\n\nimport argparse\nimport os\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef main(script_args, training_args, model_args, dataset_args):\n from accelerate import logging\n from datasets import load_dataset\n from transformers import AutoModelForCausalLM, AutoTokenizer\n\n from trl import get_dataset, get_peft_config\n from trl.experimental.kto import KTOTrainer\n\n logger = logging.get_logger(__name__)\n\n # Load a pretrained model\n model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n ref_model = AutoModelForCausalLM.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n\n tokenizer = AutoTokenizer.from_pretrained(\n model_args.model_name_or_path, trust_remote_code=model_args.trust_remote_code\n )\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n # Load the dataset\n if dataset_args.datasets and script_args.dataset_name:\n logger.warning(\n \"Both `datasets` and `dataset_name` are provided. The `datasets` argument will be used to load the \"\n \"dataset and `dataset_name` will be ignored.\"\n )\n dataset = get_dataset(dataset_args)\n elif dataset_args.datasets and not script_args.dataset_name:\n dataset = get_dataset(dataset_args)\n elif not dataset_args.datasets and script_args.dataset_name:\n dataset = load_dataset(\n script_args.dataset_name, name=script_args.dataset_config, streaming=script_args.dataset_streaming\n )\n else:\n raise ValueError(\"Either `datasets` or `dataset_name` must be provided.\")\n\n # Initialize the KTO trainer\n trainer = KTOTrainer(\n model,\n ref_model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n processing_class=tokenizer,\n peft_config=get_peft_config(model_args),\n )\n\n # Train the model\n trainer.train()\n\n # Log training complete\n trainer.accelerator.print(\"โœ… Training completed.\")\n\n # Save and push to Hub\n trainer.save_model(training_args.output_dir)\n trainer.accelerator.print(f\"๐Ÿ’พ Model saved to {training_args.output_dir}.\")\n\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n trainer.accelerator.print(f\"๐Ÿค— Model pushed to the Hub in https://huggingface.co/{trainer.hub_model_id}.\")\n\n\ndef make_parser(subparsers: argparse._SubParsersAction | None = None, prog: str | None = None):\n from trl import DatasetMixtureConfig, ModelConfig, ScriptArguments, TrlParser\n from trl.experimental.kto import KTOConfig\n\n dataclass_types = (ScriptArguments, KTOConfig, ModelConfig, DatasetMixtureConfig)\n if subparsers is not None:\n parser = subparsers.add_parser(\"kto\", help=\"Run the KTO training script\", dataclass_types=dataclass_types)\n else:\n parser = TrlParser(dataclass_types, prog=prog)\n return parser\n\n\nif __name__ == \"__main__\":\n parser = make_parser()\n script_args, training_args, model_args, dataset_args = parser.parse_args_and_config(fail_with_unknown_args=False)\n main(script_args, training_args, model_args, dataset_args)\n" }, { "path": "trl/scripts/reward.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\nimport argparse\nimport os\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef main(script_args, training_args, model_args, dataset_args):\n from accelerate import logging\n from datasets import load_dataset\n\n from trl import RewardTrainer, get_dataset, get_peft_config\n\n logger = logging.get_logger(__name__)\n\n # Load the dataset\n if dataset_args.datasets and script_args.dataset_name:\n logger.warning(\n \"Both `datasets` and `dataset_name` are provided. The `datasets` argument will be used to load the \"\n \"dataset and `dataset_name` will be ignored.\"\n )\n dataset = get_dataset(dataset_args)\n elif dataset_args.datasets and not script_args.dataset_name:\n dataset = get_dataset(dataset_args)\n elif not dataset_args.datasets and script_args.dataset_name:\n dataset = load_dataset(\n script_args.dataset_name, name=script_args.dataset_config, streaming=script_args.dataset_streaming\n )\n else:\n raise ValueError(\"Either `datasets` or `dataset_name` must be provided.\")\n\n # Initialize the RewardTrainer\n trainer = RewardTrainer(\n model=model_args.model_name_or_path,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=get_peft_config(model_args),\n )\n\n # Train the model\n trainer.train()\n\n # Log training complete\n trainer.accelerator.print(\"โœ… Training completed.\")\n\n # Save and push to Hub\n trainer.save_model(training_args.output_dir)\n trainer.accelerator.print(f\"๐Ÿ’พ Model saved to {training_args.output_dir}.\")\n\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n trainer.accelerator.print(f\"๐Ÿค— Model pushed to the Hub in https://huggingface.co/{trainer.hub_model_id}.\")\n\n\ndef make_parser(subparsers: argparse._SubParsersAction | None = None, prog: str | None = None):\n from trl import DatasetMixtureConfig, ModelConfig, RewardConfig, ScriptArguments, TrlParser\n\n dataclass_types = (ScriptArguments, RewardConfig, ModelConfig, DatasetMixtureConfig)\n if subparsers is not None:\n parser = subparsers.add_parser(\n \"reward\", help=\"Run the reward training script\", dataclass_types=dataclass_types\n )\n else:\n parser = TrlParser(dataclass_types, prog=prog)\n return parser\n\n\nif __name__ == \"__main__\":\n parser = make_parser()\n script_args, training_args, model_args, dataset_args = parser.parse_args_and_config(fail_with_unknown_args=False)\n main(script_args, training_args, model_args, dataset_args)\n" }, { "path": "trl/scripts/rloo.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\nimport argparse\nimport importlib\nimport os\nimport sys\nfrom dataclasses import dataclass, field\n\nfrom trl import ScriptArguments\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\n@dataclass\nclass RLOOScriptArguments(ScriptArguments):\n \"\"\"\n Script arguments for the RLOO training script.\n\n Args:\n reward_model_name_or_path (`str`, *optional*):\n Reward model id of a pretrained model hosted inside a model repo on huggingface.co or local path to a\n directory containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`].\n reward_funcs (`list[str]`, *optional*):\n Reward functions to use. Supported values are:\n - `\"accuracy_reward\"`\n - `\"reasoning_accuracy_reward\"`\n - `\"think_format_reward\"`\n - `\"get_soft_overlong_punishment\"` (used value are `max_completion_len=1280`, `soft_punish_cache=256`)\n - any dotted import path \" (e.g., `'my_lib.rewards.custom_reward'`).\n \"\"\"\n\n reward_model_name_or_path: str | None = field(\n default=None,\n metadata={\n \"help\": \"Reward model id of a pretrained model hosted inside a model repo on huggingface.co or \"\n \"local path to a directory containing model weights saved using `PreTrainedModel.save_pretrained`.\"\n },\n )\n reward_funcs: list[str] | None = field(\n default=None,\n metadata={\n \"help\": \"Reward functions to use. Supported values are: `accuracy_reward`, `reasoning_accuracy_reward`, `think_format_reward`, \"\n \"`get_soft_overlong_punishment` (used values are `max_completion_len=1280`, `soft_punish_cache=256`), or \"\n \"any dotted import path (e.g., `'my_lib.rewards.custom_reward'`).\"\n },\n )\n\n\ndef main(script_args, training_args, model_args, dataset_args):\n from accelerate import logging\n from datasets import load_dataset\n\n from trl import RLOOTrainer, get_dataset, get_peft_config\n from trl.rewards import (\n accuracy_reward,\n get_soft_overlong_punishment,\n reasoning_accuracy_reward,\n think_format_reward,\n )\n\n logger = logging.get_logger(__name__)\n\n reward_funcs_registry = {\n \"accuracy_reward\": accuracy_reward,\n \"reasoning_accuracy_reward\": reasoning_accuracy_reward,\n \"think_format_reward\": think_format_reward,\n \"get_soft_overlong_punishment\": get_soft_overlong_punishment(max_completion_len=1280, soft_punish_cache=256),\n }\n\n # Get the reward models and functions\n reward_funcs = []\n if script_args.reward_model_name_or_path:\n reward_funcs.append(script_args.reward_model_name_or_path)\n\n if script_args.reward_funcs:\n for func_name in script_args.reward_funcs:\n if func_name in reward_funcs_registry:\n reward_funcs.append(reward_funcs_registry[func_name])\n elif \".\" in func_name:\n module_path, func_name = func_name.rsplit(\".\", 1)\n sys.path.insert(0, os.getcwd())\n module = importlib.import_module(module_path)\n reward_func = getattr(module, func_name)\n reward_funcs.append(reward_func)\n else:\n raise ValueError(\n f\"Could not load reward function '{func_name}'. Expected one of \"\n f\"{list(reward_funcs_registry.keys())} or a valid import path.\"\n )\n\n # Load the dataset\n if dataset_args.datasets and script_args.dataset_name:\n logger.warning(\n \"Both `datasets` and `dataset_name` are provided. The `datasets` argument will be used to load the \"\n \"dataset and `dataset_name` will be ignored.\"\n )\n dataset = get_dataset(dataset_args)\n elif dataset_args.datasets and not script_args.dataset_name:\n dataset = get_dataset(dataset_args)\n elif not dataset_args.datasets and script_args.dataset_name:\n dataset = load_dataset(\n script_args.dataset_name, name=script_args.dataset_config, streaming=script_args.dataset_streaming\n )\n else:\n raise ValueError(\"Either `datasets` or `dataset_name` must be provided.\")\n\n # Initialize the RLOO trainer\n trainer = RLOOTrainer(\n model=model_args.model_name_or_path,\n reward_funcs=reward_funcs,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=get_peft_config(model_args),\n )\n\n # Train the model\n trainer.train()\n\n # Log training complete\n trainer.accelerator.print(\"โœ… Training completed.\")\n\n # Save and push to Hub\n trainer.save_model(training_args.output_dir)\n trainer.accelerator.print(f\"๐Ÿ’พ Model saved to {training_args.output_dir}.\")\n\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n trainer.accelerator.print(f\"๐Ÿค— Model pushed to the Hub in https://huggingface.co/{trainer.hub_model_id}.\")\n\n\ndef make_parser(subparsers: argparse._SubParsersAction | None = None, prog: str | None = None):\n from trl import DatasetMixtureConfig, ModelConfig, RLOOConfig, TrlParser\n\n dataclass_types = (RLOOScriptArguments, RLOOConfig, ModelConfig, DatasetMixtureConfig)\n if subparsers is not None:\n parser = subparsers.add_parser(\"rloo\", help=\"Run the RLOO training script\", dataclass_types=dataclass_types)\n else:\n parser = TrlParser(dataclass_types, prog=prog)\n return parser\n\n\nif __name__ == \"__main__\":\n parser = make_parser()\n script_args, training_args, model_args, dataset_args = parser.parse_args_and_config(fail_with_unknown_args=False)\n main(script_args, training_args, model_args, dataset_args)\n" }, { "path": "trl/scripts/sft.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# /// script\n# dependencies = [\n# \"trl\",\n# \"peft\",\n# \"trackio\",\n# \"kernels\",\n# ]\n# ///\n\n\"\"\"\n# Full training\n```\npython trl/scripts/sft.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --learning_rate 2.0e-5 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --eos_token '<|im_end|>' \\\n --eval_strategy steps \\\n --eval_steps 100 \\\n --output_dir Qwen2-0.5B-SFT \\\n --push_to_hub\n```\n\n# LoRA\n```\npython trl/scripts/sft.py \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --learning_rate 2.0e-4 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --eos_token '<|im_end|>' \\\n --eval_strategy steps \\\n --eval_steps 100 \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --output_dir Qwen2-0.5B-SFT \\\n --push_to_hub\n```\n\"\"\"\n\nimport argparse\nimport os\n\n\n# Enable logging in a Hugging Face Space\nos.environ.setdefault(\"TRACKIO_SPACE_ID\", \"trl-trackio\")\n\n\ndef main(script_args, training_args, model_args, dataset_args):\n from accelerate import logging\n from datasets import load_dataset\n from transformers import AutoConfig, AutoModelForCausalLM\n from transformers.models.auto.modeling_auto import MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES\n\n from trl import SFTTrainer, get_dataset, get_kbit_device_map, get_peft_config, get_quantization_config\n\n logger = logging.get_logger(__name__)\n\n ################\n # Model init kwargs\n ################\n model_kwargs = dict(\n revision=model_args.model_revision,\n trust_remote_code=model_args.trust_remote_code,\n attn_implementation=model_args.attn_implementation,\n dtype=model_args.dtype,\n )\n quantization_config = get_quantization_config(model_args)\n if quantization_config is not None:\n # Passing None would not be treated the same as omitting the argument, so we include it only when valid.\n model_kwargs[\"device_map\"] = get_kbit_device_map()\n model_kwargs[\"quantization_config\"] = quantization_config\n\n # Create model\n config = AutoConfig.from_pretrained(model_args.model_name_or_path)\n valid_image_text_architectures = MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES.values()\n\n if config.architectures and any(arch in valid_image_text_architectures for arch in config.architectures):\n from transformers import AutoModelForImageTextToText\n\n model = AutoModelForImageTextToText.from_pretrained(model_args.model_name_or_path, **model_kwargs)\n else:\n model = AutoModelForCausalLM.from_pretrained(model_args.model_name_or_path, **model_kwargs)\n\n # Load the dataset\n if dataset_args.datasets and script_args.dataset_name:\n logger.warning(\n \"Both `datasets` and `dataset_name` are provided. The `datasets` argument will be used to load the \"\n \"dataset and `dataset_name` will be ignored.\"\n )\n dataset = get_dataset(dataset_args)\n elif dataset_args.datasets and not script_args.dataset_name:\n dataset = get_dataset(dataset_args)\n elif not dataset_args.datasets and script_args.dataset_name:\n dataset = load_dataset(\n script_args.dataset_name, name=script_args.dataset_config, streaming=script_args.dataset_streaming\n )\n else:\n raise ValueError(\"Either `datasets` or `dataset_name` must be provided.\")\n\n # Initialize the SFT trainer\n trainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset[script_args.dataset_train_split],\n eval_dataset=dataset[script_args.dataset_test_split] if training_args.eval_strategy != \"no\" else None,\n peft_config=get_peft_config(model_args),\n )\n\n # Train the model\n trainer.train()\n\n # Log training complete\n trainer.accelerator.print(\"โœ… Training completed.\")\n\n # Save and push to Hub\n trainer.save_model(training_args.output_dir)\n trainer.accelerator.print(f\"๐Ÿ’พ Model saved to {training_args.output_dir}.\")\n\n if training_args.push_to_hub:\n trainer.push_to_hub(dataset_name=script_args.dataset_name)\n trainer.accelerator.print(f\"๐Ÿค— Model pushed to the Hub in https://huggingface.co/{trainer.hub_model_id}.\")\n\n\ndef make_parser(subparsers: argparse._SubParsersAction | None = None, prog: str | None = None):\n from trl import DatasetMixtureConfig, ModelConfig, ScriptArguments, SFTConfig, TrlParser\n\n dataclass_types = (ScriptArguments, SFTConfig, ModelConfig, DatasetMixtureConfig)\n if subparsers is not None:\n parser = subparsers.add_parser(\"sft\", help=\"Run the SFT training script\", dataclass_types=dataclass_types)\n else:\n parser = TrlParser(dataclass_types, prog=prog)\n return parser\n\n\nif __name__ == \"__main__\":\n parser = make_parser()\n script_args, training_args, model_args, dataset_args = parser.parse_args_and_config(fail_with_unknown_args=False)\n main(script_args, training_args, model_args, dataset_args)\n" }, { "path": "trl/scripts/utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport argparse\nimport importlib\nimport inspect\nimport logging\nimport os\nimport subprocess\nimport sys\nfrom collections.abc import Iterable\nfrom dataclasses import dataclass, field\nfrom typing import TYPE_CHECKING\n\n# Temporarily import from the local module instead of transformers to avoid an upstream latency issue\n# See: https://github.com/huggingface/transformers/issues/44273\n# This workaround can be reverted once the fix is included in the minimum required transformers version\nfrom trl.scripts._hf_argparser import DataClass, DataClassType, HfArgumentParser\n\n\nif TYPE_CHECKING:\n from datasets import DatasetDict\n\n\nlogger = logging.getLogger(__name__)\n\n\n@dataclass\nclass DatasetConfig:\n \"\"\"\n Configuration for a dataset.\n\n This class matches the signature of [`~datasets.load_dataset`] and the arguments are used directly in the\n [`~datasets.load_dataset`] function. You can refer to the [`~datasets.load_dataset`] documentation for more\n details.\n\n Parameters:\n path (`str`):\n Path or name of the dataset.\n name (`str`, *optional*):\n Defining the name of the dataset configuration.\n data_dir (`str`, *optional*):\n Defining the `data_dir` of the dataset configuration. If specified for the generic builders(csv, text etc.)\n or the Hub datasets and `data_files` is `None`, the behavior is equal to passing `os.path.join(data_dir,\n **)` as `data_files` to reference all the files in a directory.\n data_files (`str` or `Sequence` or `Mapping`, *optional*):\n Path(s) to source data file(s).\n split (`str`, *optional*, defaults to `\"train\"`):\n Which split of the data to load.\n columns (`list[str]`, *optional*):\n List of column names to select from the dataset. If `None`, all columns are selected.\n \"\"\"\n\n path: str\n name: str | None = None\n data_dir: str | None = None\n data_files: str | list[str] | dict[str, str] | None = None\n split: str = \"train\"\n columns: list[str] | None = None\n\n\n@dataclass\nclass DatasetMixtureConfig:\n \"\"\"\n Configuration class for a mixture of datasets.\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n datasets (`list[DatasetConfig]`):\n List of dataset configurations to include in the mixture.\n streaming (`bool`, *optional*, defaults to `False`):\n Whether to stream the datasets. If `True`, the datasets will be loaded in streaming mode.\n test_split_size (`float`, *optional*):\n Size of the test split. Refer to the `test_size` parameter in the [`~datasets.train_test_split`] function\n for more details. If `None`, the dataset will not be split into train and test sets.\n\n Usage:\n When using the CLI, you can add the following section to your YAML config file:\n\n ```yaml\n datasets:\n - path: ...\n name: ...\n data_dir: ...\n data_files: ...\n split: ...\n columns: ...\n - path: ...\n name: ...\n data_dir: ...\n data_files: ...\n split: ...\n columns: ...\n streaming: ...\n test_split_size: ...\n ```\n \"\"\"\n\n datasets: list[DatasetConfig] = field(\n default_factory=list,\n metadata={\"help\": \"List of dataset configurations to include in the mixture.\"},\n )\n streaming: bool = field(\n default=False,\n metadata={\"help\": \"Whether to stream the datasets. If True, the datasets will be loaded in streaming mode.\"},\n )\n test_split_size: float | None = field(\n default=None,\n metadata={\n \"help\": \"Size of the test split. Refer to the `test_size` parameter in the `datasets.train_test_split` \"\n \"function for more details. If None, the dataset will not be split into train and test sets.\"\n },\n )\n\n def __post_init__(self):\n # Convert any dataset dicts (from CLI/config parsing) into DatasetConfig objects\n for idx, dataset in enumerate(self.datasets):\n if isinstance(dataset, dict):\n # If it's a dict, convert it to DatasetConfig\n self.datasets[idx] = DatasetConfig(**dataset)\n\n\n@dataclass\nclass ScriptArguments:\n \"\"\"\n Arguments common to all scripts.\n\n Args:\n dataset_name (`str`,, *optional*):\n Path or name of the dataset to load. If `datasets` is provided, this will be ignored.\n dataset_config (`str`, *optional*):\n Dataset configuration name. Corresponds to the `name` argument of the [`~datasets.load_dataset`] function.\n If `datasets` is provided, this will be ignored.\n dataset_train_split (`str`, *optional*, defaults to `\"train\"`):\n Dataset split to use for training. If `datasets` is provided, this will be ignored.\n dataset_test_split (`str`, *optional*, defaults to `\"test\"`):\n Dataset split to use for evaluation. If `datasets` is provided, this will be ignored.\n dataset_streaming (`bool`, *optional*, defaults to `False`):\n Whether to stream the dataset. If True, the dataset will be loaded in streaming mode. If `datasets` is\n provided, this will be ignored.\n ignore_bias_buffers (`bool`, *optional*, defaults to `False`):\n Debug argument for distributed training. Fix for DDP issues with LM bias/mask buffers - invalid scalar\n type, inplace operation. See\n https://github.com/huggingface/transformers/issues/22482#issuecomment-1595790992.\n \"\"\"\n\n dataset_name: str | None = field(\n default=None,\n metadata={\"help\": \"Path or name of the dataset to load. If `datasets` is provided, this will be ignored.\"},\n )\n dataset_config: str | None = field(\n default=None,\n metadata={\n \"help\": \"Dataset configuration name. Corresponds to the `name` argument of the `datasets.load_dataset` \"\n \"function. If `datasets` is provided, this will be ignored.\"\n },\n )\n dataset_train_split: str = field(\n default=\"train\",\n metadata={\"help\": \"Dataset split to use for training. If `datasets` is provided, this will be ignored.\"},\n )\n dataset_test_split: str = field(\n default=\"test\",\n metadata={\"help\": \"Dataset split to use for evaluation. If `datasets` is provided, this will be ignored.\"},\n )\n dataset_streaming: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to stream the dataset. If True, the dataset will be loaded in streaming mode. If \"\n \"`datasets` is provided, this will be ignored.\"\n },\n )\n ignore_bias_buffers: bool = field(\n default=False,\n metadata={\n \"help\": \"Debug argument for distributed training. Fix for DDP issues with LM bias/mask buffers - invalid \"\n \"scalar type, inplace operation. See \"\n \"https://github.com/huggingface/transformers/issues/22482#issuecomment-1595790992.\"\n },\n )\n\n\ndef init_zero_verbose():\n \"\"\"\n Perform zero verbose init - use this method on top of the CLI modules to make logging and warning output cleaner.\n Uses Rich if available, falls back otherwise.\n \"\"\"\n import logging\n import warnings\n\n from transformers.utils import is_rich_available\n\n FORMAT = \"%(message)s\"\n\n if is_rich_available():\n from rich.logging import RichHandler\n\n handler = RichHandler()\n else:\n handler = logging.StreamHandler()\n\n logging.basicConfig(format=FORMAT, datefmt=\"[%X]\", handlers=[handler], level=logging.ERROR)\n\n # Custom warning handler to redirect warnings to the logging system\n def warning_handler(message, category, filename, lineno, file=None, line=None):\n logging.warning(f\"{filename}:{lineno}: {category.__name__}: {message}\")\n\n # Add the custom warning handler - we need to do that before importing anything to make sure the loggers work well\n warnings.showwarning = warning_handler\n\n\nclass TrlParser(HfArgumentParser):\n \"\"\"\n A subclass of [`transformers.HfArgumentParser`] designed for parsing command-line arguments with dataclass-backed\n configurations, while also supporting configuration file loading and environment variable management.\n\n Args:\n dataclass_types (`DataClassType | Iterable[DataClassType]`, *optional*):\n Dataclass types to use for argument parsing.\n **kwargs:\n Additional keyword arguments passed to the [`transformers.HfArgumentParser`] constructor.\n\n Examples:\n\n ```yaml\n # config.yaml\n env:\n VAR1: value1\n arg1: 23\n ```\n\n ```python\n # main.py\n import os\n from dataclasses import dataclass\n from trl import TrlParser\n\n\n @dataclass\n class MyArguments:\n arg1: int\n arg2: str = \"alpha\"\n\n\n parser = TrlParser(dataclass_types=[MyArguments])\n training_args = parser.parse_args_and_config()\n\n print(training_args, os.environ.get(\"VAR1\"))\n ```\n\n ```bash\n $ python main.py --config config.yaml\n (MyArguments(arg1=23, arg2='alpha'),) value1\n\n $ python main.py --arg1 5 --arg2 beta\n (MyArguments(arg1=5, arg2='beta'),) None\n ```\n \"\"\"\n\n def __init__(\n self,\n dataclass_types: DataClassType | Iterable[DataClassType] | None = None,\n **kwargs,\n ):\n # Make sure dataclass_types is an iterable\n if dataclass_types is None:\n dataclass_types = []\n elif not isinstance(dataclass_types, Iterable):\n dataclass_types = [dataclass_types]\n\n # Check that none of the dataclasses have the \"config\" field\n for dataclass_type in dataclass_types:\n if \"config\" in dataclass_type.__dataclass_fields__:\n raise ValueError(\n f\"Dataclass {dataclass_type.__name__} has a field named 'config'. This field is reserved for the \"\n f\"config file path and should not be used in the dataclass.\"\n )\n\n super().__init__(dataclass_types=dataclass_types, **kwargs)\n\n def parse_args_and_config(\n self,\n args: Iterable[str] | None = None,\n return_remaining_strings: bool = False,\n fail_with_unknown_args: bool = True,\n separate_remaining_strings: bool = False,\n ) -> tuple[DataClass, ...]:\n \"\"\"\n Parse command-line args and config file into instances of the specified dataclass types.\n\n This method wraps [`transformers.HfArgumentParser.parse_args_into_dataclasses`] and also parses the config file\n specified with the `--config` flag. The config file (in YAML format) provides argument values that replace the\n default values in the dataclasses. Command line arguments can override values set by the config file. The\n method also sets any environment variables specified in the `env` field of the config file.\n \"\"\"\n import yaml\n\n args = list(args) if args is not None else sys.argv[1:]\n if \"--config\" in args:\n # Get the config file path from\n config_index = args.index(\"--config\")\n args.pop(config_index) # remove the --config flag\n config_path = args.pop(config_index) # get the path to the config file\n with open(config_path) as yaml_file:\n config = yaml.safe_load(yaml_file)\n\n # Set the environment variables specified in the config file\n if \"env\" in config:\n env_vars = config.pop(\"env\", {})\n if not isinstance(env_vars, dict):\n raise ValueError(\"`env` field should be a dict in the YAML file.\")\n for key, value in env_vars.items():\n os.environ[key] = str(value)\n\n # Set the defaults from the config values\n config_remaining_strings = self.set_defaults_with_config(**config)\n else:\n config_remaining_strings = []\n\n # Parse the arguments from the command line\n output = self.parse_args_into_dataclasses(args=args, return_remaining_strings=return_remaining_strings)\n\n # Merge remaining strings from the config file with the remaining strings from the command line\n if return_remaining_strings:\n args_remaining_strings = output[-1]\n if separate_remaining_strings:\n return output[:-1] + (config_remaining_strings, args_remaining_strings)\n return output[:-1] + (config_remaining_strings + args_remaining_strings,)\n elif fail_with_unknown_args and config_remaining_strings:\n raise ValueError(\n f\"Unknown arguments from config file: {config_remaining_strings}. Please remove them, add them to the \"\n \"dataclass, or set `fail_with_unknown_args=False`.\"\n )\n else:\n return output\n\n def set_defaults_with_config(self, **kwargs) -> list[str]:\n \"\"\"\n Overrides the parser's default values with those provided via keyword arguments, including for subparsers.\n\n Any argument with an updated default will also be marked as not required if it was previously required.\n\n Returns a list of strings that were not consumed by the parser.\n \"\"\"\n\n def apply_defaults(parser, kw):\n used_keys = set()\n for action in parser._actions:\n # Handle subparsers recursively\n if isinstance(action, argparse._SubParsersAction):\n for subparser in action.choices.values():\n used_keys.update(apply_defaults(subparser, kw))\n elif action.dest in kw:\n action.default = kw[action.dest]\n action.required = False\n used_keys.add(action.dest)\n return used_keys\n\n used_keys = apply_defaults(self, kwargs)\n # Remaining args not consumed by the parser\n remaining = [\n item for key, value in kwargs.items() if key not in used_keys for item in (f\"--{key}\", str(value))\n ]\n return remaining\n\n\ndef get_git_commit_hash(package_name):\n try:\n # Import the package to locate its path\n package = importlib.import_module(package_name)\n # Get the path to the package using inspect\n package_path = os.path.dirname(inspect.getfile(package))\n\n # Navigate up to the Git repository root if the package is inside a subdirectory\n git_repo_path = os.path.abspath(os.path.join(package_path, \"..\"))\n git_dir = os.path.join(git_repo_path, \".git\")\n\n if os.path.isdir(git_dir):\n # Run the git command to get the current commit hash\n commit_hash = (\n subprocess.check_output([\"git\", \"rev-parse\", \"HEAD\"], cwd=git_repo_path).strip().decode(\"utf-8\")\n )\n return commit_hash\n else:\n return None\n except Exception as e:\n return f\"Error: {str(e)}\"\n\n\ndef get_dataset(mixture_config: DatasetMixtureConfig) -> \"DatasetDict\":\n \"\"\"\n Load a mixture of datasets based on the configuration.\n\n Args:\n mixture_config ([`DatasetMixtureConfig`]):\n Script arguments containing dataset configuration.\n\n Returns:\n [`~datasets.DatasetDict`]:\n Combined dataset(s) from the mixture configuration, with optional train/test split if `test_split_size` is\n set.\n\n Example:\n ```python\n from trl import DatasetMixtureConfig, get_dataset\n from trl.scripts.utils import DatasetConfig\n\n mixture_config = DatasetMixtureConfig(datasets=[DatasetConfig(path=\"trl-lib/tldr\")])\n dataset = get_dataset(mixture_config)\n print(dataset)\n ```\n\n ```\n DatasetDict({\n train: Dataset({\n features: ['prompt', 'completion'],\n num_rows: 116722\n })\n })\n ```\n \"\"\"\n import datasets\n\n logger.info(f\"Creating dataset mixture with {len(mixture_config.datasets)} datasets\")\n datasets_list = []\n for dataset_config in mixture_config.datasets:\n logger.info(f\"Loading dataset for mixture: {dataset_config.path} (config name: {dataset_config.name})\")\n dataset = datasets.load_dataset(\n path=dataset_config.path,\n name=dataset_config.name,\n data_dir=dataset_config.data_dir,\n data_files=dataset_config.data_files,\n split=dataset_config.split,\n streaming=mixture_config.streaming,\n )\n if dataset_config.columns is not None:\n dataset = dataset.select_columns(dataset_config.columns)\n datasets_list.append(dataset)\n\n if datasets_list:\n combined_dataset = datasets.concatenate_datasets(datasets_list)\n if isinstance(combined_dataset, datasets.Dataset): # IterableDataset does not have a length\n logger.info(f\"Created dataset mixture with {len(combined_dataset)} examples\")\n\n if mixture_config.test_split_size is not None:\n logger.info(f\"Splitting dataset into train and test sets with test size: {mixture_config.test_split_size}\")\n combined_dataset = combined_dataset.train_test_split(test_size=mixture_config.test_split_size)\n return combined_dataset\n else:\n return datasets.DatasetDict({\"train\": combined_dataset})\n else:\n raise ValueError(\"No datasets were loaded from the mixture configuration\")\n" }, { "path": "trl/scripts/vllm_serve.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport argparse\nimport base64\nimport logging\nimport os\nfrom collections.abc import Sequence\nfrom contextlib import asynccontextmanager\nfrom dataclasses import dataclass, field\nfrom io import BytesIO\nfrom itertools import chain\nfrom multiprocessing import Pipe, Process\nfrom multiprocessing.connection import Connection\n\n\n# We use CUDA with multiprocessing, so we must use the 'spawn' start method. Otherwise, we will get the following\n# error: RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use\n# the 'spawn' start method\nos.environ[\"VLLM_WORKER_MULTIPROC_METHOD\"] = \"spawn\"\n\n\nclass WeightSyncWorkerExtension:\n \"\"\"\n A vLLM worker extension that enables weight synchronization between a client and multiple server workers.\n\n This worker uses a `StatelessProcessGroup` to establish communication and a `PyNcclCommunicator` or\n `ProcessGroupXCCL` to handle efficient GPU-based communication using NCCL. The primary purpose of this class is to\n receive updated model weights from a client process and distribute them to all worker processes participating in\n model inference.\n \"\"\"\n\n # The following attributes are initialized when `init_communicator` method is called.\n communicator = None # Communicator for weight updates\n client_rank = None # Source rank for broadcasting updated weights\n\n def init_communicator(self, host: str, port: int, world_size: int, client_device_uuid: str) -> None:\n \"\"\"\n Initializes the weight update communicator using a stateless process group.\n\n This method creates a `StatelessProcessGroup` that allows external training processes to communicate with vLLM\n workers without interfering with the global torch distributed group.\n\n Args:\n host (`str`):\n Hostname or IP address of the master node.\n port (`int`):\n Port number to be used for communication.\n world_size (`int`):\n Total number of participating processes in the update group.\n client_device_uuid (`str`):\n UUID of the device of client main process. Used to assert that devices are different from vllm workers\n devices.\n \"\"\"\n import torch\n import torch.distributed.distributed_c10d as c10d\n from transformers import is_torch_xpu_available\n from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator\n from vllm.distributed.parallel_state import get_world_group\n from vllm.distributed.utils import StatelessProcessGroup\n\n from trl.import_utils import is_vllm_ascend_available\n\n if is_vllm_ascend_available():\n from vllm_ascend.distributed.device_communicators.pyhccl import PyHcclCommunicator as PyNcclCommunicator\n\n if self.communicator is not None:\n raise RuntimeError(\"Weight update group already initialized. Call close_communicator first.\")\n\n # TODO: will remove after torch xpu 2.9 support uuid in get_device_properties\n if torch.cuda.is_available() or (\n is_torch_xpu_available() and hasattr(torch.xpu.get_device_properties(self.device), \"uuid\")\n ):\n accelerator_module = torch.xpu if is_torch_xpu_available() else torch.cuda\n if client_device_uuid == str(accelerator_module.get_device_properties(self.device).uuid):\n raise RuntimeError(\n f\"Attempting to use the same CUDA device (UUID: {client_device_uuid}) for multiple distinct \"\n \"roles/ranks within the same communicator. This setup is unsupported and will likely lead to program \"\n \"hangs or incorrect behavior. Ensure that trainer is using different devices than vLLM server.\"\n )\n # Get the rank of the current worker in the global world group.\n rank = get_world_group().rank\n\n if is_torch_xpu_available():\n store = torch.distributed.TCPStore(host_name=host, port=port, world_size=world_size, is_master=(rank == 0))\n prefixed_store = c10d.PrefixStore(\"client2server\", store)\n xccl_options = c10d.ProcessGroupXCCL.Options()\n pg = c10d.ProcessGroupXCCL(\n store=prefixed_store,\n rank=rank,\n size=world_size,\n options=xccl_options,\n )\n self.communicator = pg\n else:\n # Create a stateless process group to manage communication between training processes and vLLM workers.\n # Initialize the NCCL-based communicator for weight synchronization.\n pg = StatelessProcessGroup.create(host=host, port=port, rank=rank, world_size=world_size)\n self.communicator = PyNcclCommunicator(pg, device=self.device)\n\n # The client process that sends updated weights has the highest rank (world_size - 1).\n self.client_rank = world_size - 1\n\n def update_named_param(self, name: str, dtype: str, shape: Sequence[int]) -> None:\n \"\"\"\n Receives updated weights from the client process and updates the named parameter in the model.\n\n Args:\n name (`str`):\n Name of the weight tensor being updated.\n dtype (`str`):\n Data type of the weight tensor as a string (e.g., `\"torch.float32\"`).\n shape (`Sequence[int]`):\n Shape of the weight tensor.\n \"\"\"\n import torch\n from transformers import is_torch_xpu_available\n\n if self.communicator is None:\n raise RuntimeError(\"Communicator not initialized. Call `init_communicator` first.\")\n\n dtype = getattr(torch, dtype.split(\".\")[-1])\n # Allocate memory for the incoming weight tensor on the correct device.\n weight = torch.empty(shape, dtype=dtype, device=self.device)\n\n if is_torch_xpu_available():\n # Use XCCL to broadcast the updated weights from the client (src) to all workers.\n self.communicator.broadcast(weight, root=self.client_rank)\n self.communicator.barrier()\n else:\n # Use NCCL to broadcast the updated weights from the client (src) to all workers.\n self.communicator.broadcast(weight, src=self.client_rank)\n self.communicator.group.barrier()\n\n # Load the received weights into the model.\n self.model_runner.model.load_weights(weights=[(name, weight)])\n\n def close_communicator(self) -> None:\n \"\"\"\n Closes the communicator when weight synchronization is no longer needed.\n\n This method deletes the NCCL communicator to release associated resources.\n \"\"\"\n\n if self.communicator is not None:\n del self.communicator\n self.communicator = None # Ensure attribute is reset to None\n self.client_rank = None # Ensure attribute is reset to None\n\n\n@dataclass\nclass ScriptArguments:\n r\"\"\"\n Arguments for the script.\n\n Args:\n model (`str`):\n Model name or path to load the model from.\n revision (`str`, *optional*):\n Revision to use for the model. If not specified, the default branch will be used.\n tensor_parallel_size (`int`, *optional*, defaults to `1`):\n Number of tensor parallel workers to use.\n data_parallel_size (`int`, *optional*, defaults to `1`):\n Number of data parallel workers to use. For dense models, keep this at 1. Starting from vLLM `0.14.0`,\n setting this above `1` for dense models is no longer supported/useful and will error out (see vLLM PR\n #30739).\n host (`str`, *optional*, defaults to `\"0.0.0.0\"`):\n Host address to run the server on.\n port (`int`, *optional*, defaults to `8000`):\n Port to run the server on.\n gpu_memory_utilization (`float`, *optional*, defaults to `0.9`):\n Ratio (between 0 and 1) of GPU memory to reserve for the model weights, activations, and KV cache on the\n device dedicated to generation powered by vLLM. Higher values will increase the KV cache size and thus\n improve the model's throughput. However, if the value is too high, it may cause out-of-memory (OOM) errors\n during initialization.\n dtype (`str`, *optional*, defaults to `\"auto\"`):\n Data type to use for vLLM generation. If set to `\"auto\"`, the data type will be automatically determined\n based on the model configuration. Find the supported values in the vLLM documentation.\n max_model_len (`int`, *optional*):\n If set, the `max_model_len` to use for vLLM. This can be useful when running with reduced\n `vllm_gpu_memory_utilization`, leading to a reduced KV cache size. If not set, vLLM will use the model\n context size, which might be much larger than the KV cache, leading to inefficiencies.\n enable_prefix_caching (`bool`, *optional*):\n Whether to enable prefix caching in vLLM. If set to `True`, ensure that the model and the hardware support\n this feature.\n enforce_eager (`bool`, *optional*, defaults to `False`):\n Whether to enforce eager execution. If set to `True`, we will disable CUDA graph and always execute the\n model in eager mode. If `False` (default behavior), we will use CUDA graph and eager execution in hybrid.\n vllm_model_impl (`str`, *optional*, defaults to `\"vllm\"`):\n Model implementation to use for vLLM. Must be one of `\"transformers\"` or `\"vllm\"`. `\"transformers\"`: Use\n the `transformers` backend for model implementation. `\"vllm\"`: Use the `vllm` library for model\n implementation.\n kv_cache_dtype (`str`, *optional*, defaults to `\"auto\"`):\n Data type to use for KV cache. If set to `\"auto\"`, the dtype will default to the model data type.\n trust_remote_code (`bool`, *optional*, defaults to `False`):\n Whether to trust remote code when loading models. Set to `True` to allow executing code from model\n repositories. This is required for some custom models but introduces security risks.\n log_level (`str`, *optional*, defaults to `\"info\"`):\n Log level for uvicorn. Possible choices: `\"critical\"`, `\"error\"`, `\"warning\"`, `\"info\"`, `\"debug\"`,\n `\"trace\"`.\n \"\"\"\n\n model: str = field(\n metadata={\"help\": \"Model name or path to load the model from.\"},\n )\n revision: str | None = field(\n default=None,\n metadata={\"help\": \"Revision to use for the model. If not specified, the default branch will be used.\"},\n )\n tensor_parallel_size: int = field(\n default=1,\n metadata={\"help\": \"Number of tensor parallel workers to use.\"},\n )\n data_parallel_size: int = field(\n default=1,\n metadata={\n \"help\": \"Number of data parallel workers to use. For dense models, keep this at 1. Starting from vLLM \"\n \"`0.14.0`, setting this above `1` for dense models is no longer supported/useful and will error out (see \"\n \"vLLM PR #30739).\"\n },\n )\n host: str = field(\n default=\"0.0.0.0\",\n metadata={\"help\": \"Host address to run the server on.\"},\n )\n port: int = field(\n default=8000,\n metadata={\"help\": \"Port to run the server on.\"},\n )\n gpu_memory_utilization: float = field(\n default=0.9,\n metadata={\n \"help\": \"Ratio (between 0 and 1) of GPU memory to reserve for the model weights, activations, and KV \"\n \"cache on the device dedicated to generation powered by vLLM. Higher values will increase the KV cache \"\n \"size and thus improve the model's throughput. However, if the value is too high, it may cause \"\n \"out-of-memory (OOM) errors during initialization.\"\n },\n )\n dtype: str = field(\n default=\"auto\",\n metadata={\n \"help\": \"Data type to use for vLLM generation. If set to 'auto', the data type will be automatically \"\n \"determined based on the model configuration. Find the supported values in the vLLM documentation.\"\n },\n )\n max_model_len: int | None = field(\n default=None,\n metadata={\n \"help\": \"If set, the `max_model_len` to use for vLLM. This can be useful when running with reduced \"\n \"`vllm_gpu_memory_utilization`, leading to a reduced KV cache size. If not set, vLLM will use the model \"\n \"context size, which might be much larger than the KV cache, leading to inefficiencies.\"\n },\n )\n enable_prefix_caching: bool | None = field(\n default=None,\n metadata={\n \"help\": \"Whether to enable prefix caching in vLLM. If set to `True`, ensure that the model and the \"\n \"hardware support this feature.\"\n },\n )\n enforce_eager: bool | None = field(\n default=False,\n metadata={\n \"help\": \"Whether to enforce eager execution. If set to `True`, we will disable CUDA graph and always \"\n \"execute the model in eager mode. If `False` (default behavior), we will use CUDA graph and eager \"\n \"execution in hybrid.\"\n },\n )\n kv_cache_dtype: str = field(\n default=\"auto\",\n metadata={\n \"help\": \"Data type to use for KV cache. If set to 'auto', the dtype will default to the model data type.\"\n },\n )\n trust_remote_code: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to trust remote code when loading models. Set to True to allow executing code from model \"\n \"repositories. This is required for some custom models but introduces security risks.\"\n },\n )\n log_level: str = field(\n default=\"info\",\n metadata={\n \"help\": \"Log level for uvicorn. Possible choices: 'critical', 'error', 'warning', 'info', 'debug', \"\n \"'trace'.\"\n },\n )\n vllm_model_impl: str = field(\n default=\"vllm\",\n metadata={\n \"help\": \"Model implementation to use for vLLM. Must be one of `transformers` or `vllm`. `transformers`: \"\n \"Use the `transformers` backend for model implementation. `vllm`: Use the `vllm` library for \"\n \"model implementation.\"\n },\n )\n\n\ndef llm_worker(\n script_args: ScriptArguments, data_parallel_rank: int, master_port: int, connection: Connection\n) -> None:\n from vllm import LLM\n\n # Set required environment variables for DP to work with vLLM\n os.environ[\"VLLM_DP_RANK\"] = str(data_parallel_rank)\n os.environ[\"VLLM_DP_RANK_LOCAL\"] = str(data_parallel_rank)\n os.environ[\"VLLM_DP_SIZE\"] = str(script_args.data_parallel_size)\n os.environ[\"VLLM_DP_MASTER_PORT\"] = str(master_port)\n\n llm = LLM(\n model=script_args.model,\n revision=script_args.revision,\n tensor_parallel_size=script_args.tensor_parallel_size,\n gpu_memory_utilization=script_args.gpu_memory_utilization,\n enforce_eager=script_args.enforce_eager,\n dtype=script_args.dtype,\n # Automatic Prefix Caching caches the KV cache of existing queries, so that a new query can\n # directly reuse the KV cache if it shares the same prefix with one of the existing queries.\n # This is particularly useful here because we generate completions from the same prompts.\n enable_prefix_caching=script_args.enable_prefix_caching,\n kv_cache_dtype=script_args.kv_cache_dtype,\n max_model_len=script_args.max_model_len,\n worker_extension_cls=\"trl.scripts.vllm_serve.WeightSyncWorkerExtension\",\n trust_remote_code=script_args.trust_remote_code,\n model_impl=script_args.vllm_model_impl,\n # Important so temperature scaling/logit tweaking affects the TIS log probs\n logprobs_mode=\"processed_logprobs\",\n )\n\n # Send ready signal to parent process\n connection.send({\"status\": \"ready\"})\n\n while True:\n # Wait for commands from the parent process\n try:\n command = connection.recv()\n except KeyboardInterrupt:\n llm.collective_rpc(method=\"close_communicator\")\n break\n\n # Handle commands\n if command[\"type\"] in [\"call\", \"fire_and_forget\"]:\n method_name = command[\"method\"]\n args, kwargs = command.get(\"args\", ()), command.get(\"kwargs\", {})\n method = getattr(llm, method_name)\n result = method(*args, **kwargs)\n if command[\"type\"] == \"call\":\n connection.send(result)\n elif command[\"type\"] == \"shutdown\":\n break\n\n\ndef chunk_list(lst: list, n: int) -> list[list]:\n \"\"\"\n Split list `lst` into `n` evenly distributed sublists.\n\n Example:\n ```python\n >>> chunk_list([1, 2, 3, 4, 5, 6], 2)\n [[1, 2, 3], [4, 5, 6]]\n\n >>> chunk_list([1, 2, 3, 4, 5, 6], 4)\n [[1, 2], [3, 4], [5], [6]]\n\n >>> chunk_list([1, 2, 3, 4, 5, 6], 8)\n [[1], [2], [3], [4], [5], [6], [], []]\n ```\n \"\"\"\n k, r = divmod(len(lst), n)\n return [lst[i * k + min(i, r) : (i + 1) * k + min(i + 1, r)] for i in range(n)]\n\n\ndef main(script_args: ScriptArguments):\n from packaging.version import Version\n from transformers import is_vision_available\n\n from trl.generation.vllm_generation import extract_logprobs\n from trl.import_utils import (\n is_fastapi_available,\n is_pydantic_available,\n is_uvicorn_available,\n is_vllm_available,\n )\n\n if not is_fastapi_available():\n raise ImportError(\n \"FastAPI is required to run the vLLM serve script. Please install it using `pip install fastapi`.\"\n )\n\n if not is_pydantic_available():\n raise ImportError(\n \"Pydantic is required to run the vLLM serve script. Please install it using `pip install pydantic`.\"\n )\n\n if not is_uvicorn_available():\n raise ImportError(\n \"Uvicorn is required to run the vLLM serve script. Please install it using `pip install uvicorn`.\"\n )\n\n if not is_vllm_available():\n raise ImportError(\"vLLM is required to run the vLLM serve script. Please install it using `pip install vllm`.\")\n\n import uvicorn\n import vllm\n from fastapi import FastAPI\n from pydantic import BaseModel\n from vllm import SamplingParams\n\n if Version(vllm.__version__) <= Version(\"0.11.0\"):\n from vllm.utils import get_open_port\n else:\n from vllm.utils.network_utils import get_open_port\n\n if is_vision_available():\n from PIL import Image\n\n logger = logging.getLogger(__name__)\n\n # Spawn dp workers, and setup pipes for communication\n master_port = get_open_port()\n connections = []\n processes = []\n for data_parallel_rank in range(script_args.data_parallel_size):\n parent_connection, child_connection = Pipe()\n process = Process(target=llm_worker, args=(script_args, data_parallel_rank, master_port, child_connection))\n process.start()\n connections.append(parent_connection)\n processes.append(process)\n\n @asynccontextmanager\n async def lifespan(app: FastAPI):\n # Wait for all workers to send \"ready\"\n ready_connections = set()\n while len(ready_connections) < script_args.data_parallel_size:\n for connection in connections:\n msg = connection.recv()\n if isinstance(msg, dict) and msg.get(\"status\") == \"ready\":\n ready_connections.add(connection)\n\n yield\n\n # Wait for processes to terminate\n for process in processes:\n process.join(timeout=10) # Wait for 10 seconds for the process to terminate\n if process.is_alive():\n logger.warning(f\"Process {process} is still alive after 10 seconds, attempting to terminate...\")\n process.terminate()\n process.join() # ensure process termination after calling terminate()\n\n app = FastAPI(lifespan=lifespan)\n\n # Define the endpoints for the model server\n @app.get(\"/health/\")\n async def health():\n \"\"\"\n Health check endpoint to verify that the server is running.\n \"\"\"\n return {\"status\": \"ok\"}\n\n @app.get(\"/get_world_size/\")\n async def get_world_size():\n \"\"\"\n Retrieves the world size of the LLM engine, which is `tensor_parallel_size * data_parallel_size`.\n\n Returns:\n `dict`:\n A dictionary containing the world size.\n\n Example response:\n ```json\n {\"world_size\": 8}\n ```\n \"\"\"\n return {\"world_size\": script_args.tensor_parallel_size * script_args.data_parallel_size}\n\n class GenerateRequest(BaseModel):\n prompts: list[str] | list[list[int]]\n images: list[list[str] | None] | None = None\n n: int = 1\n repetition_penalty: float = 1.0\n temperature: float = 1.0\n top_p: float = 1.0\n top_k: int = -1\n min_p: float = 0.0\n max_tokens: int = 16\n logprobs: int | None = 0\n structured_outputs_regex: str | None = None\n generation_kwargs: dict = field(default_factory=dict)\n\n class GenerateResponse(BaseModel):\n prompt_ids: list[list[int]]\n completion_ids: list[list[int]]\n logprobs: list[list[list[float | None]]] | None\n logprob_token_ids: list[list[list[int]]] | None\n\n @app.post(\"/generate/\", response_model=GenerateResponse)\n async def generate(request: GenerateRequest):\n \"\"\"\n Generates completions for the provided prompts.\n\n Args:\n request (`GenerateRequest`):\n - `prompts` (list of `str` or list of list of `int`): A list of prompts. It accepts either text strings\n or pre-tokenized token ID lists. When text strings are provided, `images` can optionally be included.\n - `images` (list of list of `str` or `None`, *optional*): A list of image lists. Each element is a list\n of base64-encoded images for the corresponding prompt, or `None` if no images for that prompt.\n - `n` (`int`, *optional*, defaults to `1`): Number of completions to generate for each prompt.\n - `repetition_penalty` (`float`, *optional*, defaults to `1.0`): Repetition penalty to apply during\n generation.\n - `temperature` (`float`, *optional*, defaults to `1.0`): Temperature for sampling. Higher values lead\n to more random outputs.\n - `top_p` (`float`, *optional*, defaults to `1.0`): Top-p (nucleus) sampling parameter. It controls the\n diversity of the generated text.\n - `top_k` (`int`, *optional*, defaults to `-1`): Top-k sampling parameter. If set to `-1`, it disables\n top-k sampling.\n - `min_p` (`float`, *optional*, defaults to `0.0`): Minimum probability threshold for sampling.\n - `max_tokens` (`int`, *optional*, defaults to `16`): Maximum number of tokens to generate for each\n completion.\n - `logprobs` (`int`, *optional*, defaults to `0`): Number of top logprobs to return per token. When 0,\n only the sampled token's logprob is returned. When N>0, returns up to N+1 logprobs sorted by\n descending probability, because vLLM always includes the sampled token's logprob (which may fall\n outside the top-N).\n - `structured_outputs_regex` (`str`, *optional*): A regex pattern for structured outputs. If provided,\n the model will only generate tokens that match this regex pattern.\n - `generation_kwargs` (`dict`, *optional*): Additional generation parameters to pass to the vLLM\n `SamplingParams`. This can include parameters like `seed`, `frequency_penalty`, etc. If it contains\n keys that conflict with the other parameters, they will override them.\n\n Returns:\n `GenerateResponse`:\n - `prompt_ids` (list of list of `int`): A list of lists of token IDs for each input prompt.\n - `completion_ids` (list of list of `int`): A list of lists of token IDs for each generated completion.\n - `logprobs` (list of list of list of `float`): Per-token logprobs of shape (num_sequences, seq_len,\n num_logprobs), sorted by descending probability.\n - `logprob_token_ids` (list of list of list of `int`): Token IDs corresponding to each logprob, same\n shape as `logprobs`.\n\n Example request (text prompts):\n ```json\n {\"prompts\": [\"Hello world\", \"What is AI?\"]}\n ```\n\n Example request (token IDs):\n ```json\n {\"prompts\": [[101, 102], [201, 202]]}\n ```\n\n Example response:\n ```json\n {\n \"prompt_ids\": [[101, 102], [201, 202]],\n \"completion_ids\": [[103, 104, 105], [203, 204, 205]],\n \"logprobs\": [[[-0.1], [-0.2], [-0.3]], [[-0.4], [-0.5], [-0.6]]],\n \"logprob_token_ids\": [[[103], [104], [105]], [[203], [204], [205]]]\n }\n ```\n \"\"\"\n # Build vLLM-compatible prompt inputs\n is_token_ids = request.prompts and isinstance(request.prompts[0], list)\n request.images = request.images or [None] * len(request.prompts)\n\n prompts = []\n for prompt, image_list in zip(request.prompts, request.images, strict=True):\n row = {\"prompt_token_ids\": prompt} if is_token_ids else {\"prompt\": prompt}\n if image_list is not None:\n row[\"multi_modal_data\"] = {\"image\": [Image.open(BytesIO(base64.b64decode(img))) for img in image_list]}\n prompts.append(row)\n\n generation_kwargs = {\n \"n\": request.n,\n \"repetition_penalty\": request.repetition_penalty,\n \"temperature\": request.temperature,\n \"top_p\": request.top_p,\n \"top_k\": request.top_k,\n \"min_p\": request.min_p,\n \"max_tokens\": request.max_tokens,\n \"logprobs\": request.logprobs,\n }\n generation_kwargs.update(request.generation_kwargs)\n\n # Structured outputs, if enabled\n if Version(vllm.__version__) <= Version(\"0.10.2\"):\n from vllm.sampling_params import GuidedDecodingParams as StructuredOutputsParams\n\n structured_outputs_key = \"guided_decoding\"\n else:\n from vllm.sampling_params import StructuredOutputsParams\n\n structured_outputs_key = \"structured_outputs\"\n if request.structured_outputs_regex is not None:\n if generation_kwargs.get(structured_outputs_key) is not None:\n logger.warning(\n f\"Both `structured_outputs_regex` and `generation_kwargs['{structured_outputs_key}']` are set; \"\n \"`structured_outputs_regex` takes precedence.\"\n )\n generation_kwargs[structured_outputs_key] = StructuredOutputsParams(regex=request.structured_outputs_regex)\n elif isinstance(structured_outputs_kwargs := generation_kwargs.get(structured_outputs_key), dict):\n generation_kwargs[structured_outputs_key] = StructuredOutputsParams(**structured_outputs_kwargs)\n sampling_params = SamplingParams(**generation_kwargs)\n\n # Evenly distribute prompts across DP ranks\n chunked_prompts = chunk_list(prompts, script_args.data_parallel_size)\n\n # Send the prompts to each worker\n for connection, prompts in zip(connections, chunked_prompts, strict=True):\n # When the number of prompts is less than data_parallel_size, some workers will receive empty prompts.\n # However, vLLM requires that we always send at least one prompt. So we send a placeholder prompt to comply\n # with vLLM's requirement, and we later ignore the result.\n if not prompts:\n prompts = [\"\"]\n kwargs = {\"prompts\": prompts, \"sampling_params\": sampling_params}\n connection.send({\"type\": \"call\", \"method\": \"generate\", \"kwargs\": kwargs})\n\n # Receive results\n all_outputs = [connection.recv() for connection in connections]\n\n # Handle empty prompts (see above)\n all_outputs = [output for output, prompts in zip(all_outputs, chunked_prompts, strict=True) if prompts]\n\n # Flatten and combine all results\n all_outputs = list(chain.from_iterable(all_outputs)) # from list of list to single list\n prompt_ids = [output.prompt_token_ids for output in all_outputs]\n completion_ids = [list(output.token_ids) for outputs in all_outputs for output in outputs.outputs]\n logprobs, logprob_token_ids = extract_logprobs(all_outputs)\n\n return {\n \"prompt_ids\": prompt_ids,\n \"completion_ids\": completion_ids,\n \"logprobs\": logprobs,\n \"logprob_token_ids\": logprob_token_ids,\n }\n\n class ChatRequest(BaseModel):\n messages: list[list[dict]]\n n: int = 1\n repetition_penalty: float = 1.0\n temperature: float = 1.0\n top_p: float = 1.0\n top_k: int = -1\n min_p: float = 0.0\n max_tokens: int = 16\n logprobs: int | None = 0\n structured_outputs_regex: str | None = None\n generation_kwargs: dict = field(default_factory=dict)\n chat_template_kwargs: dict = field(default_factory=dict)\n tools: list | None = None\n\n class ChatResponse(BaseModel):\n prompt_ids: list[list[int]]\n completion_ids: list[list[int]]\n logprobs: list[list[list[float | None]]] | None\n logprob_token_ids: list[list[list[int]]] | None\n\n @app.post(\"/chat/\", response_model=ChatResponse)\n async def chat(request: ChatRequest):\n \"\"\"\n Generates completions for the provided chat messages.\n\n Args:\n request (`ChatRequest`):\n - `messages` (list of `dict`): A list of messages (dicts with \"role\" and \"content\" keys) for the model\n to generate completions.\n - `n` (`int`, *optional*, defaults to `1`): Number of completions to generate for each prompt.\n - `repetition_penalty` (`float`, *optional*, defaults to `1.0`): Repetition penalty to apply during\n generation.\n - `temperature` (`float`, *optional*, defaults to `1.0`): Temperature for sampling. Higher values lead\n to more random outputs.\n - `top_p` (`float`, *optional*, defaults to `1.0`): Top-p (nucleus) sampling parameter. It controls the\n diversity of the generated text.\n - `top_k` (`int`, *optional*, defaults to `-1`): Top-k sampling parameter. If set to `-1`, it disables\n top-k sampling.\n - `min_p` (`float`, *optional*, defaults to `0.0`): Minimum probability threshold for sampling.\n - `max_tokens` (`int`, *optional*, defaults to `16`): Maximum number of tokens to generate for each\n completion.\n - `logprobs` (`int`, *optional*, defaults to `0`): Number of top logprobs to return per token. When 0,\n only the sampled token's logprob is returned. When N>0, returns up to N+1 logprobs sorted by\n descending probability, because vLLM always includes the sampled token's logprob (which may fall\n outside the top-N).\n - `structured_outputs_regex` (`str`, *optional*): A regex pattern for structured outputs. If provided,\n the model will only generate tokens that match this regex pattern.\n - `generation_kwargs` (`dict`, *optional*): Additional generation parameters to pass to the vLLM\n `SamplingParams`. This can include parameters like `seed`, `frequency_penalty`, etc. If it contains\n keys that conflict with the other parameters, they will override them.\n - `chat_template_kwargs` (`dict`, *optional*): Additional keyword arguments to pass to the chat\n template.\n\n Returns:\n `ChatResponse`:\n - `prompt_ids` (list of list of `int`): A list of lists of token IDs for each input prompt.\n - `completion_ids` (list of list of `int`): A list of lists of token IDs for each generated completion.\n - `logprobs` (list of list of list of `float`): Per-token logprobs of shape (num_sequences, seq_len,\n num_logprobs), sorted by descending probability.\n - `logprob_token_ids` (list of list of list of `int`): Token IDs corresponding to each logprob, same\n shape as `logprobs`.\n\n Example request:\n ```bash\n curl -X POST 'http://0.0.0.0:8000/chat/' \\\n -H 'Content-Type: application/json' \\\n -d '{\"messages\": [[{ \"role\": \"user\", \"content\": \"Hello!\" }]]}'\n ```\n\n Example response:\n ```json\n {\n \"prompt_ids\": [[151644, 872, 198, 9707, 0, 151645, 198, 151644, 77091, 198]],\n \"completion_ids\": [[151667, 198, 32313, 11, 279]],\n \"logprobs\": [[[-0.0003], [-3.58e-07], [-0.0902], [-6.39e-05], [-0.0387]]],\n \"logprob_token_ids\": [[[151667], [198], [32313], [11], [279]]]\n }\n ```\n \"\"\"\n # Convert PIL images to base64 strings\n for message_list in request.messages:\n for message in message_list:\n if isinstance(message[\"content\"], list):\n for part in message[\"content\"]:\n if part[\"type\"] == \"image_pil\":\n part[\"image_pil\"] = Image.open(BytesIO(base64.b64decode(part[\"image_pil\"])))\n\n generation_kwargs = {\n \"n\": request.n,\n \"repetition_penalty\": request.repetition_penalty,\n \"temperature\": request.temperature,\n \"top_p\": request.top_p,\n \"top_k\": request.top_k,\n \"min_p\": request.min_p,\n \"max_tokens\": request.max_tokens,\n \"logprobs\": request.logprobs,\n }\n generation_kwargs.update(request.generation_kwargs)\n\n # Structured outputs, if enabled\n if Version(vllm.__version__) <= Version(\"0.10.2\"):\n from vllm.sampling_params import GuidedDecodingParams as StructuredOutputsParams\n\n structured_outputs_key = \"guided_decoding\"\n else:\n from vllm.sampling_params import StructuredOutputsParams\n\n structured_outputs_key = \"structured_outputs\"\n if request.structured_outputs_regex is not None:\n if generation_kwargs.get(structured_outputs_key) is not None:\n logger.warning(\n f\"Both `structured_outputs_regex` and `generation_kwargs['{structured_outputs_key}']` are set; \"\n \"`structured_outputs_regex` takes precedence.\"\n )\n generation_kwargs[structured_outputs_key] = StructuredOutputsParams(regex=request.structured_outputs_regex)\n elif isinstance(structured_outputs_kwargs := generation_kwargs.get(structured_outputs_key), dict):\n generation_kwargs[structured_outputs_key] = StructuredOutputsParams(**structured_outputs_kwargs)\n sampling_params = SamplingParams(**generation_kwargs)\n\n # Evenly distribute prompts across DP ranks\n chunked_messages = chunk_list(request.messages, script_args.data_parallel_size)\n\n # Send the messages to each worker\n for connection, messages in zip(connections, chunked_messages, strict=True):\n # When the number of messages is less than data_parallel_size, some workers will receive empty messages.\n # However, vLLM requires that we always send at least one prompt. So we send a placeholder prompt to comply\n # with vLLM's requirement, and we later ignore the result.\n if not messages:\n messages = [[{\"role\": \"user\", \"content\": \"\"}]]\n kwargs = {\n \"messages\": messages,\n \"sampling_params\": sampling_params,\n \"chat_template_kwargs\": request.chat_template_kwargs,\n \"tools\": request.tools,\n }\n connection.send({\"type\": \"call\", \"method\": \"chat\", \"kwargs\": kwargs})\n\n # Receive results\n all_outputs = [connection.recv() for connection in connections]\n\n # Handle empty prompts (see above)\n all_outputs = [output for output, prompts in zip(all_outputs, chunked_messages, strict=True) if prompts]\n\n # Flatten and combine all results\n all_outputs = list(chain.from_iterable(all_outputs)) # from list of list to single list\n prompt_ids = [output.prompt_token_ids for output in all_outputs]\n completion_ids = [list(output.token_ids) for outputs in all_outputs for output in outputs.outputs]\n logprobs, logprob_token_ids = extract_logprobs(all_outputs)\n\n return {\n \"prompt_ids\": prompt_ids,\n \"completion_ids\": completion_ids,\n \"logprobs\": logprobs,\n \"logprob_token_ids\": logprob_token_ids,\n }\n\n class InitCommunicatorRequest(BaseModel):\n host: str\n port: int\n world_size: int\n client_device_uuid: str\n\n @app.post(\"/init_communicator/\")\n async def init_communicator(request: InitCommunicatorRequest):\n \"\"\"\n Initializes the communicator for synchronizing model weights between a client and multiple server workers.\n\n Args:\n request (`InitCommunicatorRequest`):\n - `host` (`str`): Hostname or IP address of the master node.\n - `port` (`int`): Port number to be used for communication.\n - `world_size` (`int`): Total number of participating processes in the group.\n - `client_device_uuid` (`str`): UUID of the device of client main process. Used to assert that devices\n are different from vLLM workers devices.\n \"\"\"\n world_size = script_args.tensor_parallel_size * script_args.data_parallel_size + 1\n\n # The function init_communicator is called this way: init_communicator(host, port, world_size)\n # So with collective_rpc we need to call it this way:\n # llm.collective_rpc(method=\"init_communicator\", args=(host, port, world_size))\n kwargs = {\n \"method\": \"init_communicator\",\n \"args\": (request.host, request.port, world_size, request.client_device_uuid),\n }\n for connection in connections:\n connection.send({\"type\": \"fire_and_forget\", \"method\": \"collective_rpc\", \"kwargs\": kwargs})\n\n return {\"message\": \"Request received, initializing communicator\"}\n\n class UpdateWeightsRequest(BaseModel):\n name: str\n dtype: str\n shape: list[int]\n\n @app.post(\"/update_named_param/\")\n async def update_named_param(request: UpdateWeightsRequest):\n \"\"\"\n Updates the model weights with the provided tensor.\n\n Once this endpoint is called, the client process should broadcast the updated weights to all server workers.\n\n Args:\n request (`UpdateWeightsRequest`):\n - `name` (`str`): Name of the weight tensor being updated.\n - `dtype` (`str`): Data type of the weight tensor (e.g., `\"torch.float32\"`).\n - `shape` (list of `int`): Shape of the weight\n\n \"\"\"\n # The function update_named_param is called this way: update_named_param(\"name\", \"torch.float32\", (10, 10))\n # So with collective_rpc we need to call it this way:\n # llm.collective_rpc(\"update_named_param\", args=(\"name\", \"torch.float32\", (10, 10)))\n kwargs = {\"method\": \"update_named_param\", \"args\": (request.name, request.dtype, tuple(request.shape))}\n for connection in connections:\n connection.send({\"type\": \"fire_and_forget\", \"method\": \"collective_rpc\", \"kwargs\": kwargs})\n\n return {\"message\": \"Request received, updating named parameter\"}\n\n @app.post(\"/reset_prefix_cache/\")\n async def reset_prefix_cache():\n \"\"\"\n Resets the prefix cache for the model.\n \"\"\"\n for connection in connections:\n connection.send({\"type\": \"call\", \"method\": \"reset_prefix_cache\"})\n # Wait for and collect all results\n all_outputs = [connection.recv() for connection in connections]\n success = all(output for output in all_outputs)\n return {\"message\": \"Request received, resetting prefix cache status: \" + str(success)}\n\n @app.post(\"/close_communicator/\")\n async def close_communicator():\n \"\"\"\n Closes the weight update group and cleans up associated resources.\n \"\"\"\n kwargs = {\"method\": \"close_communicator\"}\n for connection in connections:\n connection.send({\"type\": \"fire_and_forget\", \"method\": \"collective_rpc\", \"kwargs\": kwargs})\n return {\"message\": \"Request received, closing communicator\"}\n\n # Start the server\n uvicorn.run(app, host=script_args.host, port=script_args.port, log_level=script_args.log_level)\n\n\ndef make_parser(subparsers: argparse._SubParsersAction | None = None, prog: str | None = None):\n from trl import TrlParser\n\n if subparsers is not None:\n parser = subparsers.add_parser(\"vllm-serve\", help=\"Run the vLLM serve script\", dataclass_types=ScriptArguments)\n else:\n parser = TrlParser(ScriptArguments, prog=prog)\n return parser\n\n\nif __name__ == \"__main__\":\n parser = make_parser()\n (script_args,) = parser.parse_args_and_config()\n main(script_args)\n" }, { "path": "trl/skills/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom .skills import (\n install_skill,\n list_agent_names,\n list_skills,\n resolve_target_path,\n uninstall_skill,\n)\n" }, { "path": "trl/skills/cli.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\"\"\"\nCLI commands for TRL skills installation and management.\n\nThis module provides command-line interface for installing TRL skills to various AI agent directories.\n\"\"\"\n\nimport argparse\n\nfrom .skills import install_skill, list_agent_names, list_skills, resolve_target_path, uninstall_skill\n\n\ndef add_skills_subcommands(subparsers: argparse._SubParsersAction) -> None:\n \"\"\"\n Add skills subcommands to the parser.\n\n This creates nested subcommands under 'trl skills' for managing skill installations.\n\n Args:\n subparsers: Subparsers from 'trl skills' command\n \"\"\"\n # Parent parser for common target options\n target_parser = argparse.ArgumentParser(add_help=False)\n target_parser.add_argument(\n \"--target\",\n required=True,\n help=f\"Installation target: agent name ({', '.join(list_agent_names())}) or directory path\",\n )\n target_parser.add_argument(\n \"--scope\",\n choices=[\"project\", \"global\"],\n default=\"project\",\n help=\"Scope when using --target with agent name: project (./agent/skills/) or global (user-level like ~/.agent/skills/)\",\n )\n\n # trl skills list (no target required - lists TRL's built-in skills by default)\n list_parser = subparsers.add_parser(\n \"list\",\n help=\"List available TRL skills or installed skills in a target\",\n description=\"Show TRL skills available for installation, or if --target is specified, show installed skills\",\n )\n list_parser.add_argument(\n \"--target\",\n help=\"Optional: show installed skills in target (agent name or directory path)\",\n )\n list_parser.add_argument(\n \"--scope\",\n choices=[\"project\", \"global\"],\n default=\"project\",\n help=\"Scope when using --target with agent name: project (./agent/skills/) or global (user-level like ~/.agent/skills/)\",\n )\n list_parser.set_defaults(func=cmd_list)\n\n # trl skills install\n install_parser = subparsers.add_parser(\n \"install\",\n parents=[target_parser],\n help=\"Install skill\",\n description=\"Install TRL skill to to target\",\n )\n install_parser.add_argument(\"skill\", nargs=\"?\", help=\"Skill name to install (omit to use --all)\")\n install_parser.add_argument(\"--all\", action=\"store_true\", help=\"Install all available TRL skills\")\n install_parser.add_argument(\"--force\", action=\"store_true\", help=\"Overwrite if skill already exists\")\n install_parser.set_defaults(func=cmd_install)\n\n # trl skills uninstall\n uninstall_parser = subparsers.add_parser(\n \"uninstall\",\n parents=[target_parser],\n help=\"Uninstall skill from target\",\n description=\"Remove a TRL skill from an AI agent's skills directory\",\n )\n uninstall_parser.add_argument(\"skill\", help=\"Skill name to uninstall\")\n uninstall_parser.set_defaults(func=cmd_uninstall)\n\n\ndef cmd_install(args):\n \"\"\"Handle 'trl skills install' command.\"\"\"\n # Validate arguments\n if not args.skill and not args.all:\n print(\"Error: Either provide a skill name or use --all to install all skills\")\n print(\"Usage: trl skills install --target \")\n print(\" or: trl skills install --all --target \")\n return 1\n\n if args.skill and args.all:\n print(\"Error: Cannot specify both a skill name and --all\")\n return 1\n\n # Determine skills to install\n if args.all:\n skills_to_install = list_skills()\n if not skills_to_install:\n print(\"No skills available to install\")\n return 1\n print(f\"Installing {len(skills_to_install)} skills to {args.target}\")\n else:\n skills_to_install = [args.skill]\n\n # Install each skill\n success_count = 0\n for skill_name in skills_to_install:\n try:\n print(f\"Installing '{skill_name}'...\", end=\" \")\n install_skill(\n skill_name=skill_name,\n target=args.target,\n scope=args.scope,\n force=args.force,\n )\n print(\"โœ“\")\n success_count += 1\n\n except FileExistsError as e:\n print(\"โœ—\")\n print(f\" Error: {e}\")\n if not args.force:\n print(\" Use --force to overwrite\")\n except (FileNotFoundError, ValueError) as e:\n print(\"โœ—\")\n print(f\" Error: {e}\")\n\n # Summary\n print(f\"\\n{success_count}/{len(skills_to_install)} skills installed successfully\")\n\n if success_count > 0:\n target_path = resolve_target_path(args.target, args.scope)\n print(f\"\\nSkills are now available at: {target_path}\")\n print(\"You may need to restart your AI agent to use the new skills.\")\n\n return 0 if success_count == len(skills_to_install) else 1\n\n\ndef cmd_uninstall(args):\n \"\"\"Handle 'trl skills uninstall' command.\"\"\"\n try:\n print(f\"Uninstalling '{args.skill}' from {args.target}...\", end=\" \")\n uninstall_skill(args.skill, target=args.target, scope=args.scope)\n print(\"โœ“\")\n print(f\"\\nSkill '{args.skill}' has been removed\")\n return 0\n\n except (FileNotFoundError, PermissionError, ValueError) as e:\n print(\"โœ—\")\n print(f\"Error: {e}\")\n return 1\n\n\ndef cmd_list(args):\n \"\"\"Handle 'trl skills list' command.\"\"\"\n try:\n # List skills - if no target specified, list TRL's built-in skills\n if args.target:\n skills = list_skills(target=args.target, scope=args.scope)\n location = args.target\n else:\n skills = list_skills()\n location = \"TRL (available for installation)\"\n\n if not skills:\n if args.target:\n print(f\"No skills installed in {args.target}\")\n else:\n print(\"No TRL skills available\")\n return 0\n\n print(f\"\\nSkills in {location}:\\n\")\n\n for skill in skills:\n print(f\" {skill}\")\n\n print(f\"\\nTotal: {len(skills)} skill(s)\")\n\n if not args.target:\n print(\"\\nUse 'trl skills install --target ' to install a skill\")\n\n return 0\n\n except ValueError as e:\n print(f\"Error: {e}\")\n return 1\n\n\n__all__ = [\n \"add_skills_subcommands\",\n]\n" }, { "path": "trl/skills/skills.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n\"\"\"\nAgent Skills.\n\nThis module:\n- provides utilities for discovering and accessing TRL skills that can be used by AI agents to learn how to use the TRL\n CLI\n- handles installation, uninstallation, and management of TRL skills\n- defines where different AI agents and coding tools look for skills, enabling easy installation of TRL skills to the\n appropriate directories\n\nAgent Skills are folders of instructions, scripts, and resources that agents can discover and use to perform tasks more\naccurately and efficiently. Learn more at https://agentskills.io\n\"\"\"\n\nimport importlib.resources as resources\nimport shutil\nfrom pathlib import Path\n\n\nAGENT_PATHS = {\n \"claude\": {\n \"global\": Path(\"~/.claude/skills\"),\n \"project\": Path(\"./.claude/skills\"),\n },\n \"codex\": {\n \"global\": Path(\"~/.codex/skills\"),\n \"project\": Path(\"./.codex/skills\"),\n },\n \"opencode\": {\n \"global\": Path(\"~/.config/opencode/skills\"),\n \"project\": Path(\".opencode/skills\"),\n },\n}\n\n\ndef list_agent_names() -> list[str]:\n \"\"\"\n List available predefined agent names.\n\n Returns:\n `list[str]`: Sorted list of agent names (e.g., ['claude', 'codex', 'opencode']).\n \"\"\"\n return sorted(AGENT_PATHS.keys())\n\n\ndef _get_trl_skills_dir() -> Path:\n \"\"\"\n Get the path to the TRL skills directory.\n\n This is the directory inside the TRL package containing skills that can be installed to AI agent directories.\n\n Returns:\n `Path`: TRL skills directory.\n \"\"\"\n return Path(str(resources.files(\"trl.skills\")))\n\n\ndef resolve_target_path(target: str | Path, scope: str = \"project\") -> Path:\n \"\"\"\n Resolve target to a concrete directory path.\n\n Converts semantic agent names (e.g., 'claude') with scope to actual filesystem paths, or normalizes provided paths.\n\n Args:\n target (`str | Path`): Agent name (e.g., 'claude', 'codex') or directory path.\n scope (`str`, defaults to `\"project\"`):\n Scope for agent names: 'global' (user-level like ~/.agent/skills/) or 'project' (./agent/skills/).\n\n Returns:\n `Path`: Resolved absolute path.\n\n Raises:\n `ValueError`: If `scope` is invalid for a predefined agent target.\n\n Example:\n ```python\n from trl.skills import resolve_target_path\n\n # Resolve agent name with scope\n path = resolve_target_path(\"claude\", \"global\")\n print(path) # /home/user/.claude/skills\n\n # Resolve custom path\n path = resolve_target_path(\"/custom/skills\")\n print(path) # /custom/skills\n ```\n \"\"\"\n if isinstance(target, Path):\n return target.expanduser().resolve()\n\n # Check if it's a predefined agent\n if target in AGENT_PATHS:\n if scope not in AGENT_PATHS[target]:\n valid_scopes = \", \".join(sorted(AGENT_PATHS[target]))\n raise ValueError(f\"Invalid scope '{scope}' for agent '{target}'. Expected one of: {valid_scopes}\")\n agent_path = AGENT_PATHS[target][scope]\n return agent_path.expanduser().resolve()\n\n # Treat as custom path string\n return Path(target).expanduser().resolve()\n\n\ndef _list_skills_in_dir(skills_dir: Path) -> list[str]:\n \"\"\"\n List skills in directory.\n\n A skill is a directory containing a SKILL.md file.\n\n Args:\n skills_dir (`Path`): Skills directory to scan.\n\n Returns:\n `list[str]`: Skill names (directory names containing SKILL.md).\n \"\"\"\n if not skills_dir.exists():\n return []\n skills = []\n for item in skills_dir.iterdir():\n if item.is_dir() and (item / \"SKILL.md\").exists():\n skills.append(item.name)\n return sorted(skills)\n\n\ndef list_skills(target: str | Path | None = None, scope: str = \"project\") -> list[str]:\n \"\"\"\n List skills.\n\n A skill is a directory containing a SKILL.md file.\n\n Args:\n target (`str | Path`, *optional*):\n Agent name (e.g., 'claude'), directory path, or `None` for TRL's built-in skills.\n scope (`str`, defaults to `\"project\"`):\n For agent names: 'global' (user-level) or 'project' (current directory).\n\n Returns:\n `list[str]`: Skill names (directory names containing SKILL.md).\n\n Example:\n ```python\n from trl.skills import list_skills\n\n # List TRL's built-in skills\n skills = list_skills()\n print(skills) # ['trl-training']\n\n # List skills installed for Claude globally\n installed = list_skills(target=\"claude\", scope=\"global\")\n print(installed) # ['trl-training', 'custom-skill']\n\n # List skills in custom directory\n custom = list_skills(target=\"/path/to/skills\")\n print(custom) # [...]\n ```\n \"\"\"\n if target is None:\n # List TRL's built-in skills\n return _list_skills_in_dir(_get_trl_skills_dir())\n\n target_dir = resolve_target_path(target, scope)\n return _list_skills_in_dir(target_dir)\n\n\ndef _install_skill_to_dir(\n skill_name: str,\n target_dir: Path,\n source_dir: Path,\n force: bool = False,\n) -> bool:\n \"\"\"\n Install a skill to target directory.\n\n Args:\n skill_name (`str`): Name of skill to install.\n target_dir (`Path`): Target installation directory.\n source_dir (`Path`): Source directory containing skills.\n force (`bool`, defaults to `False`): Whether to overwrite if exists.\n\n Returns:\n `bool`: True if installed successfully.\n\n Raises:\n - `FileNotFoundError`: If skill doesn't exist in source_dir.\n - `FileExistsError`: If skill already installed and not force.\n - `PermissionError`: If no permission to write to target_dir.\n - `ValueError`: If source_dir entry exists but is not a directory.\n - `OSError`: If copying the skill fails.\n \"\"\"\n source_skill = source_dir / skill_name\n\n # Check if source skill exists\n if not source_skill.exists():\n available = \", \".join(list_skills(target=source_dir))\n source_msg = f\"source directory {source_dir}\"\n if available:\n raise FileNotFoundError(f\"Skill '{skill_name}' not found in {source_msg}. Available skills: {available}\")\n raise FileNotFoundError(f\"Skill '{skill_name}' not found in {source_msg}\")\n\n if not source_skill.is_dir():\n raise ValueError(f\"Skill '{skill_name}' is not a directory\")\n\n target_skill = target_dir / skill_name\n\n # Check if already exists\n if target_skill.exists() and not force:\n raise FileExistsError(f\"Skill '{skill_name}' already installed at {target_skill}. Use --force to overwrite.\")\n\n # Create target directory\n try:\n target_dir.mkdir(parents=True, exist_ok=True)\n except PermissionError as e:\n raise PermissionError(f\"Cannot create directory {target_dir}: {e}\") from e\n\n # Remove existing if force\n if target_skill.exists() and force:\n if target_skill.is_symlink():\n target_skill.unlink()\n else:\n shutil.rmtree(target_skill)\n\n # Install\n try:\n shutil.copytree(source_skill, target_skill)\n except OSError as e:\n raise OSError(f\"Failed to install skill: {e}\") from e\n\n return True\n\n\ndef install_skill(\n skill_name: str,\n target: str | Path,\n scope: str = \"project\",\n source: str | Path | None = None,\n force: bool = False,\n) -> bool:\n \"\"\"\n Install a skill.\n\n Args:\n skill_name (`str`): Name of skill to install.\n target (`str | Path`): Agent name (e.g., 'claude', 'codex') or directory path.\n scope (`str`, defaults to `\"project\"`):\n Scope for agent names: 'global' (user-level) or 'project' (current directory).\n source (`str | Path`, *optional*):\n Source directory containing skills. If `None`, defaults to TRL skills directory.\n force (`bool`, defaults to `False`): Whether to overwrite if skill already exists.\n\n Returns:\n `bool`: True if installed successfully.\n\n Raises:\n - `FileNotFoundError`: If skill doesn't exist in source.\n - `FileExistsError`: If skill already installed and not force.\n - `PermissionError`: If no permission to write to target.\n - `ValueError`:\n - If `scope` is invalid for a predefined agent target.\n - If `source` entry exists but is not a directory.\n - `OSError`: If copying the skill fails.\n\n Example:\n ```python\n from trl.skills import install_skill\n\n # Install to Claude's global skills directory\n install_skill(\"trl-training\", target=\"claude\", scope=\"global\")\n\n # Install to custom directory\n install_skill(\"trl-training\", target=\"/path/to/skills\")\n\n # Overwrite existing installation\n install_skill(\"trl-training\", target=\"claude\", force=True)\n ```\n \"\"\"\n target_dir = resolve_target_path(target, scope)\n source_dir = Path(source).expanduser().resolve() if source else _get_trl_skills_dir()\n return _install_skill_to_dir(skill_name, target_dir, source_dir, force)\n\n\ndef _uninstall_skill_from_dir(skill_name: str, target_dir: Path) -> bool:\n \"\"\"\n Uninstall a skill from target directory.\n\n Args:\n skill_name (`str`): Name of skill to uninstall.\n target_dir (`Path`): Directory skill is installed in.\n\n Returns:\n `bool`: True if uninstalled successfully.\n\n Raises:\n - `FileNotFoundError`: If skill not installed.\n - `PermissionError`: If no permission to remove.\n - `OSError`: If removing the skill fails for another filesystem reason.\n \"\"\"\n target_skill = target_dir / skill_name\n\n if not target_skill.exists():\n raise FileNotFoundError(f\"Skill '{skill_name}' not installed at {target_dir}\")\n\n # Remove symlink or directory\n try:\n shutil.rmtree(target_skill)\n except PermissionError as e:\n raise PermissionError(f\"Cannot remove skill: {e}\") from e\n except OSError as e:\n raise OSError(f\"Failed to remove skill: {e}\") from e\n\n return True\n\n\ndef uninstall_skill(skill_name: str, target: str | Path, scope: str = \"project\") -> bool:\n \"\"\"\n Uninstall a skill.\n\n Args:\n skill_name (`str`): Name of skill to uninstall.\n target (`str | Path`): Agent name (e.g., 'claude', 'codex') or directory path.\n scope (`str`, defaults to `\"project\"`):\n Scope for agent names: 'global' (user-level) or 'project' (current directory).\n\n Returns:\n `bool`: True if uninstalled successfully.\n\n Raises:\n - `FileNotFoundError`: If skill not installed.\n - `PermissionError`: If no permission to remove.\n - `OSError`: If removing the skill fails for another filesystem reason.\n - `ValueError`: If `scope` is invalid for a predefined agent target.\n\n Example:\n ```python\n from trl.skills import uninstall_skill\n\n # Uninstall from Claude's global directory\n uninstall_skill(\"trl-training\", target=\"claude\", scope=\"global\")\n\n # Uninstall from custom directory\n uninstall_skill(\"trl-training\", target=\"/path/to/skills\")\n ```\n \"\"\"\n target_dir = resolve_target_path(target, scope)\n return _uninstall_skill_from_dir(skill_name, target_dir)\n" }, { "path": "trl/skills/trl-training/SKILL.md", "content": "---\nname: trl-training\ndescription: Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.\nlicense: Apache-2.0\nmetadata:\n version: \"1.0.0\"\n author: huggingface\n commands:\n - trl sft\n - trl dpo\n - trl grpo\n - trl kto\n - trl rloo\n - trl reward\n categories:\n - machine-learning\n - llm-training\n - reinforcement-learning\n tags:\n - rlhf\n - supervised-fine-tuning\n - dpo\n - grpo\n - huggingface\n - transformers\n documentation: https://huggingface.co/docs/trl/en/clis\n---\n\n# TRL Training Skill\n\nYou are an expert at using the TRL (Transformers Reinforcement Learning) library to train and fine-tune large language models.\n\n## Overview\n\nTRL provides CLI commands for post-training foundation models using state-of-the-art techniques:\n\n- **SFT** (Supervised Fine-Tuning): Fine-tune models on instruction-following or conversational datasets\n- **DPO** (Direct Preference Optimization): Align models using preference data\n- **GRPO** (Group Relative Policy Optimization): Train models by ranking multiple sampled outputs relative to each other and optimizing based on their comparative rewards.\n- **RLOO** (Reinforce Leave One Out): Online RL training with generation-based rewards\n- **Reward Model Training**: Train reward models for RLHF\n\nTRL is built on top of Hugging Face Transformers and Accelerate, providing seamless integration with the Hugging Face ecosystem.\n\n## Core Commands\n\n### trl sft - Supervised Fine-Tuning\n\nFine-tune language models on instruction-following or conversational datasets.\n\n**Full training:**\n\n```bash\ntrl sft \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --learning_rate 2.0e-5 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --eos_token '<|im_end|>' \\\n --eval_strategy steps \\\n --eval_steps 100 \\\n --output_dir Qwen2-0.5B-SFT \\\n --push_to_hub\n```\n\n**Train with LoRA adapters:**\n\n```bash\ntrl sft \\\n --model_name_or_path Qwen/Qwen2-0.5B \\\n --dataset_name trl-lib/Capybara \\\n --learning_rate 2.0e-4 \\\n --num_train_epochs 1 \\\n --packing \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 8 \\\n --eos_token '<|im_end|>' \\\n --eval_strategy steps \\\n --eval_steps 100 \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16 \\\n --output_dir Qwen2-0.5B-SFT \\\n --push_to_hub\n```\n\n### trl dpo - Direct Preference Optimization\n\nAlign models using preference data (chosen/rejected pairs).\n\n**Full training:**\n\n```bash\ntrl dpo \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --learning_rate 5.0e-7 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 2 \\\n --max_steps 1000 \\\n --gradient_accumulation_steps 8 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --output_dir Qwen2-0.5B-DPO \\\n --no_remove_unused_columns\n```\n\n**Train with LoRA adapters:**\n\n```bash\ntrl dpo \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --learning_rate 5.0e-6 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 2 \\\n --max_steps 1000 \\\n --gradient_accumulation_steps 8 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --output_dir Qwen2-0.5B-DPO \\\n --no_remove_unused_columns \\\n --use_peft \\\n --lora_r 32 \\\n --lora_alpha 16\n```\n\n### trl grpo - Group Relative Policy Optimization\n\nTrain models using reward functions or LLM-as-a-judge for evaluating generations and providing rewards.\n\n**Basic usage:**\n\n```bash\ntrl grpo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/gsm8k \\\n --reward_funcs accuracy_reward \\\n --output_dir Qwen2-0.5B-GRPO \\\n --push_to_hub\n```\n\n### trl rloo - Reinforce Leave One Out\n\nOnline RL training where the model generates text and receives rewards based on custom criteria.\n\n**Basic usage:**\n\n```bash\ntrl rloo \\\n --model_name_or_path Qwen/Qwen2.5-0.5B \\\n --dataset_name trl-lib/tldr \\\n --reward_model_name_or_path sentiment-analysis:nlptown/bert-base-multilingual-uncased-sentiment \\\n --output_dir Qwen2-0.5B-RLOO \\\n --push_to_hub\n```\n\n### trl reward - Reward Model Training\n\nTrain a reward model to score text quality for RLHF.\n\n**Full training:**\n\n```bash\ntrl reward \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --output_dir Qwen2-0.5B-Reward \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-5 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --max_length 2048\n```\n\n**Train with LoRA adapters:**\n\n```bash\ntrl reward \\\n --model_name_or_path Qwen/Qwen2-0.5B-Instruct \\\n --dataset_name trl-lib/ultrafeedback_binarized \\\n --output_dir Qwen2-0.5B-Reward-LoRA \\\n --per_device_train_batch_size 8 \\\n --num_train_epochs 1 \\\n --learning_rate 1.0e-4 \\\n --eval_strategy steps \\\n --eval_steps 50 \\\n --max_length 2048 \\\n --use_peft \\\n --lora_task_type SEQ_CLS \\\n --lora_r 32 \\\n --lora_alpha 16\n```\n\n## Configuration Files\n\nTRL supports YAML configuration files for reproducible training. All CLI arguments can be specified in a config file.\n\n**Example config (sft_config.yaml):**\n\n```yaml\nmodel_name_or_path: Qwen/Qwen2.5-0.5B\ndataset_name: trl-lib/Capybara\nlearning_rate: 2.0e-5\nnum_train_epochs: 1\nper_device_train_batch_size: 8\ngradient_accumulation_steps: 2\noutput_dir: ./sft_output\nuse_peft: true\nlora_r: 16\nlora_alpha: 16\nreport_to: trackio\n```\n\n**Launch with config:**\n\n```bash\ntrl sft --config sft_config.yaml\n```\n\n**Override config values:**\n\n```bash\ntrl sft --config sft_config.yaml --learning_rate 1.0e-5\n```\n\n## Distributed Training\n\nTRL integrates with Accelerate for multi-GPU and multi-node training.\n\n**Multi-GPU training:**\n\n```bash\ntrl sft \\\n --config sft_config.yaml \\\n --num_processes 4\n```\n\n**Use predefined Accelerate configs:**\n\nTRL provides predefined configs: `single_gpu`, `multi_gpu`, `fsdp1`, `fsdp2`, `zero1`, `zero2`, `zero3`\n\n```bash\ntrl sft \\\n --config sft_config.yaml \\\n --accelerate_config zero2\n```\n\n**Custom Accelerate config:**\n\n```bash\n# Generate custom config\naccelerate config\n\n# Use custom config\ntrl sft --config sft_config.yaml --config_file ~/.cache/huggingface/accelerate/default_config.yaml\n```\n\n**Fully Sharded Data Parallel (FSDP):**\n\n```bash\ntrl sft --config sft_config.yaml --accelerate_config fsdp2\n```\n\n**DeepSpeed ZeRO:**\n\n```bash\ntrl sft --config sft_config.yaml --accelerate_config zero3\n```\n\n## Troubleshooting\n\n### CUDA Out of Memory\n\n- Reduce `--per_device_train_batch_size` and increase `--gradient_accumulation_steps`\n- Enable `--use_peft` for LoRA training\n- Use `--gradient_checkpointing` to save memory\n- Try smaller model or longer sequence truncation\n\n### Dataset Loading Issues\n\n- Verify dataset exists: check Hugging Face Hub or local path\n- Check dataset format matches expected columns\n- Use `--dataset_config` for multi-config datasets\n- Inspect dataset: `from datasets import load_dataset; ds = load_dataset(name)`\n\n### Model Loading Issues\n\n- Verify model exists on Hugging Face Hub\n- Check if gated model requires authentication: `hf auth login`\n- For local models, provide absolute path\n- Ensure sufficient disk space and memory\n\n### Slow Training\n\n- Enable dataset `--packing` for short sequences\n- Use larger `--per_device_train_batch_size` if memory allows\n- Enable `--tf32` for faster computation on Ampere GPUs\n- Use `--bf16` on supported hardware\n- Consider multi-GPU training with `--num_processes`\n\n### Generation Issues (GRPO/RLOO)\n\n- Check prompt format in dataset\n- Adjust `--temperature` and `--top_p` for generation\n- Verify the reward function (for GRPO/RLOO)\n\n## Additional Resources\n\n- **Documentation**: https://huggingface.co/docs/trl\n- **GitHub**: https://github.com/huggingface/trl\n- **Examples**: https://github.com/huggingface/trl/tree/main/examples\n\n## Best Practices\n\n1. **Start with SFT**: Always fine-tune base models with SFT before preference alignment\n2. **Use LoRA for efficiency**: Enable `--use_peft` for faster training and lower memory\n3. **Monitor training**: Use `--report_to trackio` (or `--report_to wandb` or `--report_to tensorboard`) for tracking\n4. **Save checkpoints**: TRL automatically saves checkpoints in `--output_dir`\n5. **Test on small datasets first**: Verify pipeline works before full training\n6. **Use configuration files**: Create YAML configs for reproducibility\n7. **Leverage Accelerate**: Use multi-GPU training for faster iteration\n\nWhen helping users with TRL:\n- Always check which training method is appropriate for their use case\n- Verify dataset format matches the expected schema\n- Recommend starting with smaller models for testing\n- Suggest LoRA for resource-constrained environments\n- Point to specific documentation sections for advanced features\n" }, { "path": "trl/templates/completions_dataset_card.md", "content": "---\n{{ card_data }}\n---\n\n# TRL Completion logs\n\nThis dataset contains the completions generated during training using `trl`.\n\n{% if hub_model_id %}\nFind the trained model at https://huggingface.co/{{ hub_model_id }}.\n\n{% endif %}\nThe completions are stored in parquet files, and each file contains the completions for a single step of training (depending on the `logging_steps` argument).\n\nEach file contains the following columns:\n\n- `step`: the step of training\n- `prompt`: the prompt used to generate the completion\n- `completion`: the completion generated by the model\n- ``: the reward(s) assigned to the completion by the reward function(s) used during training\n- `advantage`: the computed advantage for the completion\n\nHaving this data stored as a simple parquet file makes it easy to load and analyze using the Datasets Viewer, Polars, Pandas, etc.\n\nYou can load the dataset using the `datasets` library:\n\n```python\nimport datasets\n\ndataset = datasets.load_dataset(\"{{ repo_id }}\")\n```\n\nYou can also load the dataset using Polars:\n\n```python\nimport polars as pl\n\n# Login using e.g. `huggingface-cli login` to access this dataset if it's private\ndf = pl.read_parquet(f\"hf://datasets/{{ repo_id }}/*.parquet\")\n```\n" }, { "path": "trl/templates/lm_model_card.md", "content": "---\n{{ card_data }}\n---\n\n# Model Card for {{ model_name }}\n\nThis model is a fine-tuned version of [{{ base_model }}](https://huggingface.co/{{ base_model }}){% if dataset_name %} on the [{{ dataset_name }}](https://huggingface.co/datasets/{{ dataset_name }}) dataset{% endif %}.\nIt has been trained using [TRL](https://github.com/huggingface/trl).\n\n## Quick start\n\n```python\nfrom transformers import pipeline\n\nquestion = \"If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?\"\ngenerator = pipeline(\"text-generation\", model=\"{{ hub_model_id }}\", device=\"cuda\")\noutput = generator([{\"role\": \"user\", \"content\": question}], max_new_tokens=128, return_full_text=False)[0]\nprint(output[\"generated_text\"])\n```\n\n## Training procedure\n\n{% if wandb_url %}[\"Visualize]({{ wandb_url }}){% endif %} \n{% if trackio_url %}[\"Visualize]({{ trackio_url }}){% endif %}\n{% if comet_url %}[\"Visualize]({{ comet_url }}){% endif %}\n\nThis model was trained with {{ trainer_name }}{% if paper_id %}, a method introduced in [{{ paper_title }}](https://huggingface.co/papers/{{ paper_id }}){% endif %}.\n\n### Framework versions\n\n- TRL: {{ trl_version }}\n- Transformers: {{ transformers_version }}\n- Pytorch: {{ pytorch_version }}\n- Datasets: {{ datasets_version }}\n- Tokenizers: {{ tokenizers_version }}\n\n## Citations\n\n{% if trainer_citation %}Cite {{ trainer_name }} as:\n\n```bibtex\n{{ trainer_citation }}\n```{% endif %}\n\nCite TRL as:\n \n```bibtex\n{% raw %}@software{vonwerra2020trl,\n title = {{TRL: Transformers Reinforcement Learning}},\n author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouรฉdec, Quentin},\n license = {Apache-2.0},\n url = {https://github.com/huggingface/trl},\n year = {2020}\n}{% endraw %}\n```\n" }, { "path": "trl/templates/rm_model_card.md", "content": "---\n{{ card_data }}\n---\n\n# Model Card for {{ model_name }}\n\nThis model is a fine-tuned version of [{{ base_model }}](https://huggingface.co/{{ base_model }}){% if dataset_name %} on the [{{ dataset_name }}](https://huggingface.co/datasets/{{ dataset_name }}) dataset{% endif %}.\nIt has been trained using [TRL](https://github.com/huggingface/trl).\n\n## Quick start\n\n```python\nfrom transformers import pipeline\n\ntext = \"The capital of France is Paris.\"\nrewarder = pipeline(model=\"{{ hub_model_id }}\", device=\"cuda\")\noutput = rewarder(text)[0]\nprint(output[\"score\"])\n```\n\n## Training procedure\n\n{% if wandb_url %}[\"Visualize]({{ wandb_url }}){% endif %}\n{% if trackio_url %}[\"Visualize]({{ trackio_url }}){% endif %}\n{% if comet_url %}[\"Visualize]({{ comet_url }}){% endif %}\n\nThis model was trained with {{ trainer_name }}{% if paper_id %}, a method introduced in [{{ paper_title }}](https://huggingface.co/papers/{{ paper_id }}){% endif %}.\n\n### Framework versions\n\n- TRL: {{ trl_version }}\n- Transformers: {{ transformers_version }}\n- Pytorch: {{ pytorch_version }}\n- Datasets: {{ datasets_version }}\n- Tokenizers: {{ tokenizers_version }}\n\n## Citations\n\n{% if trainer_citation %}Cite {{ trainer_name }} as:\n\n```bibtex\n{{ trainer_citation }}\n```{% endif %}\n\nCite TRL as:\n \n```bibtex\n{% raw %}@software{vonwerra2020trl,\n title = {{TRL: Transformers Reinforcement Learning}},\n author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouรฉdec, Quentin},\n license = {Apache-2.0},\n url = {https://github.com/huggingface/trl},\n year = {2020}\n}{% endraw %}\n```\n" }, { "path": "trl/trainer/__init__.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom typing import TYPE_CHECKING\n\nfrom .._lazy_module import _LazyModule\n\n\n_import_structure = {\n \"callbacks\": [\n \"BEMACallback\",\n \"LogCompletionsCallback\",\n \"RichProgressCallback\",\n \"SyncRefModelCallback\",\n \"WeaveCallback\",\n ],\n \"dpo_config\": [\"DPOConfig\"],\n \"dpo_trainer\": [\"DPOTrainer\"],\n \"grpo_config\": [\"GRPOConfig\"],\n \"grpo_trainer\": [\"GRPOTrainer\"],\n \"kto_config\": [\"KTOConfig\"],\n \"kto_trainer\": [\"KTOTrainer\"],\n \"model_config\": [\"ModelConfig\"],\n \"reward_config\": [\"RewardConfig\"],\n \"reward_trainer\": [\"RewardTrainer\"],\n \"rloo_config\": [\"RLOOConfig\"],\n \"rloo_trainer\": [\"RLOOTrainer\"],\n \"sft_config\": [\"SFTConfig\"],\n \"sft_trainer\": [\"SFTTrainer\"],\n \"utils\": [\n \"disable_dropout_in_model\",\n \"ensure_master_addr_port\",\n \"get_kbit_device_map\",\n \"get_peft_config\",\n \"get_quantization_config\",\n ],\n}\n\nif TYPE_CHECKING:\n from .callbacks import (\n BEMACallback,\n LogCompletionsCallback,\n RichProgressCallback,\n SyncRefModelCallback,\n WeaveCallback,\n )\n from .dpo_config import DPOConfig\n from .dpo_trainer import DPOTrainer\n from .grpo_config import GRPOConfig\n from .grpo_trainer import GRPOTrainer\n from .kto_config import KTOConfig\n from .kto_trainer import KTOTrainer\n from .model_config import ModelConfig\n from .reward_config import RewardConfig\n from .reward_trainer import RewardTrainer\n from .rloo_config import RLOOConfig\n from .rloo_trainer import RLOOTrainer\n from .sft_config import SFTConfig\n from .sft_trainer import SFTTrainer\n from .utils import (\n disable_dropout_in_model,\n ensure_master_addr_port,\n get_kbit_device_map,\n get_peft_config,\n get_quantization_config,\n )\nelse:\n import sys\n\n sys.modules[__name__] = _LazyModule(__name__, globals()[\"__file__\"], _import_structure, module_spec=__spec__)\n" }, { "path": "trl/trainer/base_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\nfrom transformers import TrainingArguments\n\n\n@dataclass\nclass _BaseConfig(TrainingArguments):\n \"\"\"\n Base configuration class for all TRL trainer configurations.\n\n Subclasses [`~transformers.TrainingArguments`] and overrides fields that are common across TRL trainers or that\n contain unescaped \"%\" characters which would cause argparse to raise a `TypeError` when rendering `--help` output.\n\n Parameters:\n logging_steps (`int` or `float`, *optional*, defaults to `10`):\n Number of update steps between two logs if `logging_strategy=\"steps\"`. Should be an integer or a float in\n range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps.\n gradient_checkpointing (`bool`, *optional*, defaults to `True`):\n Whether to enable gradient checkpointing to trade compute for memory. Reduces memory usage by clearing\n activations during forward pass and recomputing them during backward pass. Enables training larger models\n or batch sizes at the cost of ~20% slower training.\n bf16 (`bool`, *optional*):\n Whether to use bfloat16 (BF16) mixed precision instead of 32-bit. Generally preferred over FP16 due to\n better numerical stability and no loss scaling required. Requires Ampere or higher NVIDIA architecture or\n Intel XPU or using CPU (use_cpu) or Ascend NPU. If not set, it defaults to `True` if `fp16` is not set.\n lr_scheduler_kwargs (`dict` or `str`, *optional*):\n Additional parameters for the lr_scheduler, such as `{'num_cycles': 1}` for cosine with hard restarts. See\n the documentation of each scheduler for possible values.\n use_liger_kernel (`bool`, *optional*, defaults to `False`):\n Enable [Liger Kernel](https://github.com/linkedin/Liger-Kernel) optimizations. Increases multi-GPU\n throughput by ~20% and reduces memory usage by ~60%. Works with Flash Attention, FSDP, and DeepSpeed.\n Currently, supports Llama, Mistral, Mixtral, and Gemma models.\n torch_empty_cache_steps (`int`, *optional*):\n Number of steps to wait before calling `torch..empty_cache()`. If left unset or set to None, cache\n will not be emptied. This can help avoid CUDA out-of-memory errors by lowering peak VRAM usage at a cost of\n about [10% slower performance](https://github.com/huggingface/transformers/issues/31372).\n \"\"\"\n\n # Override fields from TrainingArguments to set defaults preferred by all TRL trainers.\n logging_steps: float = field(\n default=10,\n metadata={\n \"help\": \"Log every X updates steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, \"\n \"will be interpreted as ratio of total training steps.\"\n },\n )\n gradient_checkpointing: bool = field(\n default=True,\n metadata={\n \"help\": \"Enable gradient checkpointing to trade compute for memory. Reduces memory at the cost of ~20%% slower training.\"\n },\n )\n bf16: bool | None = field(\n default=None,\n metadata={\n \"help\": \"Whether to use bf16 (mixed) precision instead of 32-bit. Requires Ampere or higher NVIDIA \"\n \"architecture or Intel XPU or using CPU (use_cpu) or Ascend NPU. If not set, it defaults to `True` if \"\n \"`fp16` is not set.\"\n },\n )\n # Transformers 4.57.0 introduced a bug that caused the dtype of `lr_scheduler_kwargs` to be unparsable. This issue\n # was fixed in https://github.com/huggingface/transformers/pull/41322 and released in 4.57.5. We add a temporary\n # workaround here, which can be removed once we drop support for versions older than 4.57.5.\n lr_scheduler_kwargs: dict | str | None = field(\n default=None,\n metadata={\n \"help\": \"Additional parameters for the lr_scheduler, such as {'num_cycles': 1} for cosine with hard \"\n \"restarts. See the documentation of each scheduler for possible values.\"\n },\n )\n\n # Override fields from TrainingArguments whose help strings contain unescaped \"%\" characters.\n # argparse interprets \"%\" as a format specifier, raising TypeError when rendering --help output.\n # Fixed upstream in transformers v5.3.0, but overridden here to support older versions.\n # - Introduced in v5.2.0; fixed in v5.3.0\n use_liger_kernel: bool = field(\n default=False,\n metadata={\n \"help\": \"Enable Liger Kernel optimizations. Increases throughput by ~20%% and reduces memory by ~60%%.\"\n },\n )\n # - Introduced in v4.54.1; fixed in v5.3.0\n torch_empty_cache_steps: int | None = field(\n default=None,\n metadata={\n \"help\": \"Number of steps to wait before calling `torch..empty_cache()`. Helps avoid CUDA OOM at a cost of ~10%% slower performance. If None, cache will not be emptied.\"\n },\n )\n\n def __post_init__(self):\n self.bf16 = not (self.fp16) if self.bf16 is None else self.bf16\n\n super().__post_init__()\n" }, { "path": "trl/trainer/base_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport os\n\nfrom transformers import Trainer, is_wandb_available\n\nfrom .utils import generate_model_card, get_comet_experiment_url, get_config_model_id, get_trackio_space_url\n\n\nif is_wandb_available():\n import wandb\n\n\nclass _BaseTrainer(Trainer):\n _tag_names = []\n _name = \"Base\"\n _paper = {}\n _template_file = None\n\n def create_model_card(\n self,\n model_name: str | None = None,\n dataset_name: str | None = None,\n tags: str | list[str] | None = None,\n ):\n \"\"\"\n Creates a draft of a model card using the information available to the `Trainer`.\n\n Args:\n model_name (`str`, *optional*):\n Name of the model.\n dataset_name (`str`, *optional*):\n Name of the dataset used for training.\n tags (`str`, `list[str]`, *optional*):\n Tags to be associated with the model card.\n \"\"\"\n if not self.is_world_process_zero():\n return\n\n model_name_or_path = get_config_model_id(self.model.config)\n if model_name_or_path and not os.path.isdir(model_name_or_path):\n base_model = model_name_or_path\n else:\n base_model = None\n\n # Normalize tags\n if tags is None:\n tags = set()\n elif isinstance(tags, str):\n tags = {tags}\n else:\n tags = set(tags)\n if hasattr(self.model.config, \"unsloth_version\"):\n tags.add(\"unsloth\")\n if \"JOB_ID\" in os.environ:\n tags.add(\"hf_jobs\")\n tags.update(self._tag_names)\n\n trackio_url = get_trackio_space_url()\n # Pop existing Trackio tag and re-add the one with the proper url parameters\n if trackio_url is not None:\n for tag in list(tags):\n if tag.startswith(\"trackio:\"):\n tags.remove(tag)\n tags.add(f\"trackio:{trackio_url}\")\n\n tags = list(tags)\n\n model_card = generate_model_card(\n base_model=base_model,\n model_name=model_name,\n hub_model_id=self.hub_model_id,\n dataset_name=dataset_name,\n tags=tags,\n wandb_url=wandb.run.url if is_wandb_available() and wandb.run is not None else None,\n trackio_url=trackio_url,\n comet_url=get_comet_experiment_url(),\n trainer_name=self._name,\n trainer_citation=self._paper.get(\"citation\"),\n template_file=self._template_file,\n paper_title=self._paper.get(\"title\"),\n paper_id=self._paper.get(\"id\"),\n )\n model_card.save(os.path.join(self.args.output_dir, \"README.md\"))\n" }, { "path": "trl/trainer/callbacks.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport logging\n\nimport pandas as pd\nimport torch\nfrom accelerate import Accelerator\nfrom accelerate.state import AcceleratorState\nfrom accelerate.utils import gather_object, is_wandb_available\nfrom transformers import (\n GenerationConfig,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n Trainer,\n TrainerCallback,\n TrainerControl,\n TrainerState,\n TrainingArguments,\n)\nfrom transformers.trainer_utils import has_length\nfrom transformers.utils import is_rich_available\n\nfrom ..data_utils import maybe_apply_chat_template\nfrom ..import_utils import is_weave_available\nfrom ..models.utils import unwrap_model_for_generation\nfrom .utils import log_table_to_comet_experiment\n\n\nif is_rich_available():\n from rich.columns import Columns\n from rich.console import Console, Group\n from rich.live import Live\n from rich.panel import Panel\n from rich.progress import Progress\n from rich.table import Table\n\nif is_wandb_available():\n import wandb\n\nif is_weave_available():\n import weave\n from weave import EvaluationLogger\n from weave.trace.context import weave_client_context\n\n\n# Logger for module-level logging\nlogger = logging.getLogger(__name__)\n\n\ndef _generate_completions(\n prompts: list[str],\n model: PreTrainedModel,\n tokenizer: PreTrainedTokenizerBase,\n accelerator: Accelerator,\n generation_config: GenerationConfig | None,\n batch_size: int = 1,\n) -> list[str]:\n \"\"\"\n Generates completions for a list of pre-formatted prompts from the given model.\n\n Args:\n prompts (list[str]): A list of input prompts for which completions are to be generated.\n model (PreTrainedModel): The pre-trained model to be used for generation.\n tokenizer (PreTrainedTokenizerBase): The tokenizer to be used for encoding and decoding.\n accelerator (Accelerator): The accelerator to be used for model execution.\n generation_config (GenerationConfig): Configuration for text generation.\n batch_size (int, optional): The number of prompts to process in each batch. Default is 1.\n\n Returns:\n list[str]: A list of generated text completions corresponding to the input prompts.\n \"\"\"\n completions = []\n # TODO: Override model.generation_config with generation_kwargs\n with unwrap_model_for_generation(model, accelerator) as unwrapped_model:\n for idx in range(0, len(prompts), batch_size):\n batch = prompts[idx : idx + batch_size]\n tokenized_batch = tokenizer(batch, return_tensors=\"pt\", padding=True, truncation=True).to(model.device)\n generations = unwrapped_model.generate(\n **tokenized_batch,\n generation_config=generation_config,\n )\n for prompt, generation in zip(tokenized_batch.input_ids, generations, strict=True):\n # Remove prompt from generation\n generation = generation[len(prompt) :]\n completion = tokenizer.decode(generation, skip_special_tokens=True)\n completions.append(completion)\n return completions\n\n\nclass SyncRefModelCallback(TrainerCallback):\n \"\"\"\n Callback to synchronize the model with a reference model.\n \"\"\"\n\n def __init__(\n self,\n ref_model: PreTrainedModel | torch.nn.Module,\n accelerator: Accelerator | None,\n ):\n self.accelerator = accelerator\n self.ref_model = ref_model\n\n @staticmethod\n def _sync_target_model(model, target_model, alpha):\n for target_param, copy_param in zip(target_model.parameters(), model.parameters(), strict=True):\n target_param.data.mul_(1.0 - alpha).add_(copy_param.data, alpha=alpha)\n\n @staticmethod\n def sync_target_model(model, target_model, alpha):\n deepspeed_plugin = AcceleratorState().deepspeed_plugin\n if deepspeed_plugin is not None and deepspeed_plugin.zero_stage == 3:\n import deepspeed\n\n with deepspeed.zero.GatheredParameters(\n list(model.parameters()) + list(target_model.parameters()), modifier_rank=0\n ):\n if deepspeed.comm.get_rank() == 0:\n SyncRefModelCallback._sync_target_model(model, target_model, alpha)\n else:\n SyncRefModelCallback._sync_target_model(model, target_model, alpha)\n\n def on_step_end(self, args, state, control, **kwargs):\n model: PreTrainedModel = kwargs[\"model\"]\n\n if self.ref_model is not None and state.global_step % args.ref_model_sync_steps == 0:\n if self.accelerator:\n model = self.accelerator.unwrap_model(model)\n self.sync_target_model(model, self.ref_model, args.ref_model_mixup_alpha)\n\n\nclass RichProgressCallback(TrainerCallback):\n \"\"\"\n A [`TrainerCallback`] that displays the progress of training or evaluation using Rich.\n \"\"\"\n\n def __init__(self):\n if not is_rich_available():\n raise ImportError(\"RichProgressCallback requires the `rich` extra. To install, run `pip install rich`.\")\n\n self.training_bar = None\n self.evaluation_bar = None\n self.training_task = None\n self.evaluation_task = None\n self.rich_group = None\n self.rich_console = None\n self.training_status = None\n self.current_step = None\n\n def on_train_begin(self, args, state, control, **kwargs):\n if not state.is_world_process_zero:\n return\n\n self.training_bar = Progress()\n self.evaluation_bar = Progress()\n self.rich_console = Console()\n self.training_status = self.rich_console.status(\"Nothing to log yet ...\")\n self.rich_group = Live(Panel(Group(self.training_bar, self.evaluation_bar, self.training_status)))\n self.rich_group.start()\n self.training_task = self.training_bar.add_task(\"[blue]Training \", total=state.max_steps)\n self.current_step = 0\n\n def on_step_end(self, args, state, control, **kwargs):\n if not state.is_world_process_zero:\n return\n\n self.training_bar.update(self.training_task, advance=state.global_step - self.current_step, update=True)\n self.current_step = state.global_step\n\n def on_prediction_step(self, args, state, control, eval_dataloader=None, **kwargs):\n if not state.is_world_process_zero:\n return\n\n if has_length(eval_dataloader):\n if self.evaluation_task is None:\n self.evaluation_task = self.evaluation_bar.add_task(\"[blue]Evaluation\", total=len(eval_dataloader))\n self.evaluation_bar.update(self.evaluation_task, advance=1, update=True)\n\n def on_evaluate(self, args, state, control, **kwargs):\n if not state.is_world_process_zero:\n return\n\n if self.evaluation_task is not None:\n self.evaluation_bar.remove_task(self.evaluation_task)\n self.evaluation_task = None\n\n def on_predict(self, args, state, control, **kwargs):\n if not state.is_world_process_zero:\n return\n\n if self.evaluation_task is not None:\n self.evaluation_bar.remove_task(self.evaluation_task)\n self.evaluation_task = None\n\n def on_log(self, args, state, control, logs=None, **kwargs):\n if not (state.is_world_process_zero and self.training_bar):\n return\n\n # Group keys by top-level prefix\n grouped_logs = {}\n for key, value in logs.items():\n parts = key.split(\"/\")\n group = parts[0] if len(parts) > 1 else None\n subkey = \"/\".join(parts[1:]) if len(parts) > 1 else key\n grouped_logs.setdefault(group, {})[subkey] = value\n\n # Create a table per group\n tables = []\n for group_name, metrics in grouped_logs.items():\n table = Table(\n title=f\"[bold blue]{group_name}[/]\" if group_name else None, header_style=\"bold magenta\", box=None\n )\n table.add_column(\"Metric\", justify=\"left\", no_wrap=True)\n table.add_column(\"Value\", justify=\"right\")\n\n for metric, val in metrics.items():\n formatted = f\"{val:.3f}\" if isinstance(val, (float, int)) else str(val)\n table.add_row(metric, formatted)\n\n tables.append(Panel(table, border_style=\"cyan\", padding=(0, 1)))\n\n # Arrange tables in columns using Columns\n column_layout = Columns(tables, equal=False, expand=True)\n self.training_status.update(\n Panel(column_layout, title=f\"[bold green]Step {state.global_step}[/bold green]\", border_style=\"green\")\n )\n\n def on_train_end(self, args, state, control, **kwargs):\n if not state.is_world_process_zero:\n return\n\n self.rich_group.stop()\n self.training_bar = None\n self.evaluation_bar = None\n self.training_task = None\n self.evaluation_task = None\n self.rich_group = None\n self.rich_console = None\n self.training_status = None\n self.current_step = None\n\n\nclass LogCompletionsCallback(TrainerCallback):\n r\"\"\"\n A [`~transformers.TrainerCallback`] that logs completions to Weights & Biases and/or Comet.\n\n Usage:\n ```python\n trainer = DPOTrainer(...)\n completions_callback = LogCompletionsCallback(trainer=trainer)\n trainer.add_callback(completions_callback)\n ```\n\n Args:\n trainer (`Trainer`):\n Trainer to which the callback will be attached. The trainer's evaluation dataset must include a `\"prompt\"`\n column containing the prompts for generating completions.\n generation_config ([`~transformers.GenerationConfig`], *optional*):\n The generation config to use for generating completions.\n num_prompts (`int`, *optional*):\n The number of prompts to generate completions for. If not provided, defaults to the number of examples in\n the evaluation dataset.\n freq (`int`, *optional*):\n The frequency at which to log completions. If not provided, defaults to the trainer's `eval_steps`.\n \"\"\"\n\n def __init__(\n self,\n trainer: Trainer,\n generation_config: GenerationConfig | None = None,\n num_prompts: int | None = None,\n freq: int | None = None,\n ):\n self.trainer = trainer\n self.generation_config = generation_config\n self.freq = freq\n self.table = []\n self._last_logged_step = -1\n\n if self.trainer.eval_dataset is None:\n raise ValueError(\"Trainer must have an evaluation dataset to use the LogCompletionsCallback.\")\n else:\n self.eval_dataset = self.trainer.eval_dataset\n\n if num_prompts is not None:\n self.eval_dataset = self.eval_dataset.select(range(num_prompts))\n\n def on_step_end(self, args, state, control, **kwargs):\n # Only log once per step (this method may be called multiple times)\n if state.global_step == self._last_logged_step:\n return\n\n # Only log every `freq` steps (if no `freq` is provided, log every `eval_steps` steps)\n freq = self.freq or state.eval_steps\n if state.global_step % freq != 0:\n return\n\n tokenizer = kwargs[\"processing_class\"]\n tokenizer.padding_side = \"left\"\n accelerator = self.trainer.accelerator\n model = self.trainer.model_wrapped\n with accelerator.split_between_processes(self.eval_dataset[\"prompt\"]) as prompts:\n prompts = [maybe_apply_chat_template({\"prompt\": prompt}, tokenizer)[\"prompt\"] for prompt in prompts]\n completions = _generate_completions(\n prompts,\n model=model,\n tokenizer=tokenizer,\n accelerator=accelerator,\n generation_config=self.generation_config,\n batch_size=args.per_device_eval_batch_size,\n )\n completions = gather_object(completions)\n prompts = gather_object(prompts)\n\n # Build the data to log\n if self.trainer.accelerator.is_main_process:\n global_step = [str(state.global_step)] * len(prompts)\n data = list(zip(global_step, prompts, completions, strict=True))\n self.table.extend(data)\n table = pd.DataFrame(columns=[\"step\", \"prompt\", \"completion\"], data=self.table)\n\n if \"wandb\" in args.report_to:\n wandb.log({\"completions\": table})\n\n if \"comet_ml\" in args.report_to:\n log_table_to_comet_experiment(\n name=\"completions.csv\",\n table=table,\n )\n\n # Save the last logged step, so we don't log the same completions multiple times\n self._last_logged_step = state.global_step\n\n\nclass WeaveCallback(TrainerCallback):\n r\"\"\"\n A [`~transformers.TrainerCallback`] that logs traces and evaluations to W&B Weave. The callback uses\n https://weave-docs.wandb.ai/guides/evaluation/evaluation_logger/ to log traces and evaluations at each evaluation\n step.\n\n Supports two modes based on the `scorers` parameter:\n - **Tracing Mode** (when scorers=None): Logs predictions for data exploration and analysis\n - **Evaluation Mode** (when scorers provided): Logs predictions with scoring and summary metrics\n\n Both modes use Weave's EvaluationLogger for structured, consistent data logging.\n\n The callback logs data during evaluation phases (`on_evaluate`) rather than training steps, making it more\n efficient and semantically correct. It gracefully handles missing weave installation by logging warnings and\n skipping weave-specific functionality. It also checks for existing weave clients before initializing new ones.\n\n Usage:\n ```python\n # Tracing mode (just log predictions)\n trainer = DPOTrainer(...)\n weave_callback = WeaveTraceCallback(trainer=trainer) # project_name optional\n trainer.add_callback(weave_callback)\n\n # Or specify a project name\n weave_callback = WeaveTraceCallback(trainer=trainer, project_name=\"my-llm-training\")\n trainer.add_callback(weave_callback)\n\n\n # Evaluation mode (log predictions + scores + summary)\n def accuracy_scorer(prompt: str, completion: str) -> float:\n # Your scoring logic here (metadata available via eval_attributes)\n return score\n\n\n weave_callback = WeaveTraceCallback(\n trainer=trainer,\n project_name=\"my-llm-training\", # optional and needed only if weave client is not initialized\n scorers={\"accuracy\": accuracy_scorer},\n )\n trainer.add_callback(weave_callback)\n ```\n\n Args:\n trainer (`Trainer`):\n Trainer to which the callback will be attached. The trainer's evaluation dataset must include a `\"prompt\"`\n column containing the prompts for generating completions.\n project_name (`str`, *optional*):\n Name of the Weave project where data will be logged. If not provided, will try to use existing weave client\n or fall back to the active wandb run's project name. Raises an error if none of these are available.\n scorers (`dict[str, Callable]`, *optional*):\n Dictionary mapping scorer names to scorer functions. If `None`, operates in tracing mode (predictions\n only). If provided, operates in evaluation mode (predictions + scores + summary). Scorer functions should\n have signature: `scorer(prompt: str, completion: str) -> float | int`\n generation_config ([`~transformers.GenerationConfig`], *optional*):\n Generation config to use for generating completions.\n num_prompts (`int` or `None`, *optional*):\n Number of prompts to generate completions for. If not provided, defaults to the number of examples in the\n evaluation dataset.\n dataset_name (`str`, *optional*, defaults to `\"eval_dataset\"`):\n Name for the dataset metadata in Weave.\n model_name (`str`, *optional*):\n Name for the model metadata in Weave. If not provided, attempts to extract from model config.\n \"\"\"\n\n def __init__(\n self,\n trainer: Trainer,\n project_name: str | None = None,\n scorers: dict[str, callable] | None = None,\n generation_config: GenerationConfig | None = None,\n num_prompts: int | None = None,\n dataset_name: str = \"eval_dataset\",\n model_name: str | None = None,\n ):\n self.trainer = trainer\n self.project_name = project_name\n self.scorers = scorers or {}\n self.generation_config = generation_config\n self.dataset_name = dataset_name\n self.model_name = model_name\n self._last_logged_step = -1\n self._weave_initialized = False\n self._eval_logger = None\n\n if self.trainer.eval_dataset is None:\n raise ValueError(\"Trainer must have an evaluation dataset to use the WeaveCallback.\")\n else:\n self.eval_dataset = self.trainer.eval_dataset\n\n if num_prompts is not None:\n self.eval_dataset = self.eval_dataset.select(range(num_prompts))\n\n def _initialize_weave(self):\n \"\"\"Initialize Weave and EvaluationLogger if not already initialized.\"\"\"\n if not self._weave_initialized:\n if not is_weave_available():\n logger.warning(\"Weave is not available. Please install weave to enable logging: `pip install weave`\")\n return\n\n if wc := weave_client_context.get_weave_client():\n self._weave_client = wc\n else:\n if self.project_name is None:\n if is_wandb_available():\n if wandb.run is not None:\n self.project_name = wandb.run.entity + \"/\" + wandb.run.project\n logger.info(f\"Using project name from active wandb run: {self.project_name}\")\n\n if self.project_name is None:\n raise ValueError(\n \"No existing Weave client found and no project_name provided. \"\n \"Please either initialize weave with `weave.init('project-name')`, \"\n \"provide a project_name to the `WeaveTraceCallback`, \"\n \"or ensure an active wandb run exists.\"\n )\n\n self._weave_client = weave.init(self.project_name)\n logger.info(f\"Initialized Weave with project: {self.project_name}\")\n\n if self.model_name is None:\n self.model_name = getattr(self.trainer.model_wrapped.config, \"_name_or_path\", \"unknown_model\")\n\n self._EvaluationLogger = EvaluationLogger\n\n self._weave_initialized = True\n\n @property\n def is_evaluation_mode(self) -> bool:\n \"\"\"True if scorers are provided (evaluation mode), False for tracing mode.\"\"\"\n return bool(self.scorers)\n\n def on_train_begin(self, args, state, control, **kwargs):\n \"\"\"Initialize Weave when training begins.\"\"\"\n self._initialize_weave()\n\n def on_evaluate(self, args, state, control, **kwargs):\n if state.global_step == self._last_logged_step:\n return\n\n self._initialize_weave()\n\n if not self._weave_initialized:\n logger.debug(\"Weave not initialized, skipping logging\")\n return\n\n tokenizer = kwargs[\"processing_class\"]\n tokenizer.padding_side = \"left\"\n accelerator = self.trainer.accelerator\n model = self.trainer.model_wrapped\n\n with accelerator.split_between_processes(self.eval_dataset[\"prompt\"]) as prompts:\n prompts = [maybe_apply_chat_template({\"prompt\": prompt}, tokenizer)[\"prompt\"] for prompt in prompts]\n\n completions = _generate_completions(\n prompts=prompts,\n model=model,\n tokenizer=tokenizer,\n accelerator=accelerator,\n generation_config=self.generation_config,\n batch_size=args.per_device_eval_batch_size,\n )\n\n all_prompts = gather_object(prompts)\n all_completions = gather_object(completions)\n\n if self.trainer.accelerator.is_main_process:\n eval_attributes = {\n \"training_step\": state.global_step,\n \"model_name\": self.model_name,\n \"generation_config\": (self.generation_config.to_dict() if self.generation_config else None),\n }\n\n eval_logger = self._EvaluationLogger(\n model=self.model_name,\n dataset=self.dataset_name,\n eval_attributes=eval_attributes,\n )\n\n successful_predictions = 0\n total_score_values = {} # For summary statistics\n\n for prompt, completion in zip(all_prompts, all_completions, strict=True):\n try:\n pred_logger = eval_logger.log_prediction(inputs={\"prompt\": prompt}, output=completion)\n\n if self.is_evaluation_mode:\n for scorer_name, scorer_func in self.scorers.items():\n try:\n score = scorer_func(prompt, completion)\n pred_logger.log_score(scorer=scorer_name, score=score)\n\n if scorer_name not in total_score_values:\n total_score_values[scorer_name] = []\n total_score_values[scorer_name].append(score)\n\n except Exception as scorer_e:\n logger.warning(f\"Failed to apply scorer '{scorer_name}': {scorer_e}\")\n\n pred_logger.finish()\n successful_predictions += 1\n\n except Exception as pred_e:\n logger.warning(f\"Failed to log prediction for prompt: {pred_e}\")\n # Continue with other predictions even if one fails\n\n if self.is_evaluation_mode and total_score_values:\n try:\n summary_stats = {\n \"total_predictions\": len(all_prompts),\n \"successful_predictions\": successful_predictions,\n }\n\n for scorer_name, scores in total_score_values.items():\n if scores: # Only if we have valid scores\n summary_stats[f\"avg_{scorer_name}\"] = sum(scores) / len(scores)\n\n eval_logger.log_summary(summary_stats)\n\n except Exception as summary_e:\n logger.warning(f\"Failed to log summary: {summary_e}\")\n else:\n try:\n eval_logger.finish()\n except Exception as finish_e:\n logger.warning(f\"Failed to finish evaluation logger: {finish_e}\")\n\n self._last_logged_step = state.global_step\n\n\nclass BEMACallback(TrainerCallback):\n # docstyle-ignore\n r\"\"\"\n A [`~transformers.TrainerCallback`] that implements [BEMA](https://huggingface.co/papers/2508.00180)\n (Bias-Corrected Exponential Moving Average) by [Adam Block](https://huggingface.co/abblock) and [Cyril\n Zhang](https://huggingface.co/cyrilzhang). Code from https://github.com/abblock/bema under MIT license.\n\n BEMA computes model weights that scale like:\n\n $$\n \\theta_t' = \\alpha_t \\cdot (\\theta_t - \\theta_0) + \\text{EMA}_t\n $$\n\n where \\\\( \\theta_t \\\\) is the current model weights, \\\\( \\theta_0 \\\\) is a snapshot of the model weights at the\n first `update_after` step, \\\\( \\text{EMA}_t \\\\) is the exponential moving average of the model weights, and\n \\\\( \\alpha_t \\\\) is a scaling factor that decays with the number of steps \\\\( t \\\\) as\n\n $$\n \\alpha_t = (\\rho + \\gamma \\cdot t)^{-\\eta}.\n $$\n\n The EMA is computed as:\n\n $$\n \\text{EMA}_t = (1 - \\beta_t) \\cdot \\text{EMA}_{t-1} + \\beta_t \\cdot \\theta_t\n $$\n\n where \\\\( \\beta_t \\\\) is a decay factor that decays with the number of steps \\\\( t \\\\) as\n\n $$\n \\beta_t = (\\rho + \\gamma \\cdot t)^{-\\kappa}.\n $$\n\n Args:\n update_freq (`int`, *optional*, defaults to `400`):\n Update the BEMA weights every X steps. Denoted this as \\\\( \\phi \\\\) in the paper.\n ema_power (`float`, *optional*, defaults to `0.5`):\n Power for the EMA decay factor. Denoted \\\\( \\kappa \\\\) in the paper. To disable EMA, set this to `0.0`.\n bias_power (`float`, *optional*, defaults to `0.2`):\n Power for the BEMA scaling factor. Denoted \\\\( \\eta \\\\) in the paper. To disable BEMA, set this to `0.0`.\n lag (`int`, *optional*, defaults to `10`):\n Initial offset in the weight decay schedule that controls early-stage smoothness by acting as a virtual\n starting age for the updates. Denoted as \\\\( \\rho \\\\) in the paper.\n update_after (`int`, *optional*, defaults to `0`):\n Burn-in time before starting to update the BEMA weights. Denoted \\\\( \\tau \\\\) in the paper.\n multiplier (`float`, *optional*, defaults to `1.0`):\n Initial value for the EMA decay factor. Denoted as \\\\( \\gamma \\\\) in the paper.\n min_ema_multiplier (`float`, *optional*, defaults to `0.0`):\n Minimum value for the EMA decay factor.\n device (`str`, *optional*, defaults to `\"cpu\"`):\n Device to use for the BEMA buffers, e.g. `\"cpu\"` or `\"cuda\"`. Note that in most cases, this device SHOULD\n BE DIFFERENT from the device used for training in order to avoid OOM.\n\n Example:\n\n ```python\n from trl import BEMACallback\n\n trainer = Trainer(..., callbacks=[BEMACallback()])\n ```\n \"\"\"\n\n def __init__(\n self,\n update_freq: int = 400,\n ema_power: float = 0.5,\n bias_power: float = 0.2,\n lag: int = 10,\n update_after: int = 0,\n multiplier: float = 1.0,\n min_ema_multiplier: float = 0.0,\n device: str = \"cpu\",\n ):\n # User-provided hyperparams\n self.update_freq = update_freq\n self.ema_power = ema_power\n self.bias_power = bias_power\n self.lag = lag\n self.update_after = update_after\n self.multiplier = multiplier\n self.min_ema_multiplier = min_ema_multiplier\n self.device = device\n\n # Internal state\n self.param_names = [] # references to training model param names\n self.thetat_params = [] # references to training model params\n self.theta0_params = [] # ฮธโ‚€ buffers (on self.device)\n self.ema_params = [] # EMA buffers (on self.device)\n self.running_model = None # a copy of the model to run BEMA on\n\n @staticmethod\n def _unwrap_model(model):\n \"\"\"\n Helper function to unwrap model from various wrappers including DataParallel, DistributedDataParallel,\n DeepSpeed, and FSDP.\n \"\"\"\n # Handle DeepSpeed\n if hasattr(model, \"module\") and hasattr(model, \"engine\"):\n # DeepSpeed engine\n return model.module\n\n # Handle FSDP\n if hasattr(model, \"_fsdp_wrapped_module\"):\n # FSDP wrapped model\n return model._fsdp_wrapped_module\n\n # Handle DataParallel/DistributedDataParallel\n if hasattr(model, \"module\"):\n return model.module\n\n return model\n\n @torch.no_grad()\n def on_train_begin(\n self, args: TrainingArguments, state: TrainerState, control: TrainerControl, model: PreTrainedModel, **kwargs\n ):\n model = self._unwrap_model(model)\n\n # Create a new instance and load state_dict\n self.running_model = type(model)(model.config).to(self.device)\n self.running_model.load_state_dict(model.state_dict())\n\n # Cache trainable parameters once in a fixed order\n for name, param in model.named_parameters():\n if not param.requires_grad:\n continue\n self.param_names.append(name)\n self.thetat_params.append(param)\n\n # Clone ฮธโ‚€ and EMA on the same device as model\n theta0 = param.detach().clone().to(self.device)\n self.theta0_params.append(theta0)\n self.ema_params.append(theta0.clone()) # initialize EMA with ฮธโ‚€\n\n def _ema_beta(self, step: int) -> float:\n \"\"\"Compute the EMA decay factor ฮฒโ‚œ = (ฯ + ฮณยทt)โปแตแตƒแต–แต–แตƒ.\"\"\"\n beta = (self.lag + self.multiplier * step) ** (-self.ema_power)\n return max(beta, self.min_ema_multiplier)\n\n def _bema_alpha(self, step: int) -> float:\n \"\"\"Compute the BEMA scaling factor ฮฑโ‚œ = (ฯ + ฮณยทt)โปแต‰แต—แตƒ.\"\"\"\n return (self.lag + self.multiplier * step) ** (-self.bias_power)\n\n def _update_bema_weights(self, step: int):\n beta = self._ema_beta(step)\n alpha = self._bema_alpha(step)\n\n # Compute EMA + BEMA in-place and write directly to running_model\n for thetat, theta0, ema, run_param in zip(\n self.thetat_params, self.theta0_params, self.ema_params, self.running_model.parameters(), strict=True\n ):\n thetat = thetat.detach().to(self.device)\n ema.mul_(1 - beta).add_(thetat, alpha=beta) # EMA update: ema = (1 - beta) * ema + beta * ฮธโ‚œ\n run_param.copy_(ema + alpha * (thetat - theta0)) # BEMA update: run_param = ema + alpha * (ฮธโ‚œ - ฮธโ‚€)\n\n @torch.no_grad()\n def on_step_end(\n self, args: TrainingArguments, state: TrainerState, control: TrainerControl, model: PreTrainedModel, **kwargs\n ):\n step = state.global_step\n\n # If we haven't reached the update_after step, skip the BEMA update\n if step < self.update_after:\n return\n\n # Snapshot ฮธโ‚€ and EMA at first update\n if step == self.update_after:\n for thetat_param, theta0_param, ema_param in zip(\n self.thetat_params, self.theta0_params, self.ema_params, strict=True\n ):\n theta0_param.copy_(thetat_param)\n ema_param.copy_(thetat_param)\n\n # Update BEMA weights every `update_freq` steps\n elif (step - self.update_after) % self.update_freq == 0:\n self._update_bema_weights(step)\n logger.info(f\"Updated BEMA weights at step {step}\")\n\n @torch.no_grad()\n def on_train_end(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):\n if state.is_world_process_zero:\n save_directory = f\"{args.output_dir}/bema\"\n self.running_model.save_pretrained(save_directory)\n logger.info(f\"Saved BEMA model to {save_directory}\")\n" }, { "path": "trl/trainer/dpo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom .base_config import _BaseConfig\n\n\n@dataclass\nclass DPOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`DPOTrainer`].\n\n This class includes only the parameters that are specific to DPO training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n > Parameters that control the model\n\n model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model`\n argument of the [`DPOTrainer`] is provided as a string.\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model and reference model.\n\n > Parameters that control the data preprocessing\n\n dataset_num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n pad_token (`str`, *optional*):\n Token used for padding. If `None`, it defaults to `processing_class.pad_token`, or if that is also `None`,\n it falls back to `processing_class.eos_token`.\n max_length (`int` or `None`, *optional*, defaults to `1024`):\n Maximum length of the tokenized sequence. Sequences longer than `max_length` are truncated from the left or\n right depending on the `truncation_mode`. If `None`, no truncation is applied.\n truncation_mode (`str`, *optional*, defaults to `\"keep_start\"`):\n Truncation mode to use when the sequence exceeds `max_length`. Possible values are `\"keep_end\"` and\n `\"keep_start\"`.\n padding_free (`bool`, *optional*, defaults to `False`):\n Whether to perform forward passes without padding by flattening all sequences in the batch into a single\n continuous sequence. This reduces memory usage by eliminating padding overhead. Currently, this is only\n supported with the FlashAttention 2 or 3, which can efficiently handle the flattened batch structure.\n pad_to_multiple_of (`int`, *optional*):\n If set, the sequences will be padded to a multiple of this value.\n precompute_ref_log_probs (`bool`, *optional*, defaults to `False`):\n Whether to precompute the reference model log probabilities for the entire training dataset before\n training. This allows to save memory during training, as the reference model does not need to be kept in\n memory.\n precompute_ref_batch_size (`int`, *optional*):\n Batch size to use when precomputing reference model log probabilities. This can be set higher than the\n training batch size to speed up preprocessing. If `None`, defaults to `per_device_train_batch_size` for\n training and `per_device_eval_batch_size` for evaluation.\n\n > Parameters that control the training\n\n loss_type (`list[str]`, *optional*, defaults to `[\"sigmoid\"]`):\n Type of loss to use. Possible values are: `'sigmoid'`, `'hinge'`, `'ipo'`, `'exo_pair'`, `'nca_pair'`,\n `'robust'`, `'bco_pair'`, `'sppo_hard'`, `'aot'`, `'aot_unpaired'`, `'apo_zero'`, `'apo_down'`,\n `'discopop'`, `'sft'`. If multiple loss types are provided, they will be combined using the weights\n specified in `loss_weights`.\n loss_weights (`list[float]`, *optional*):\n List of loss weights for multi-loss combinations. Used when combining multiple loss types. Example: `[0.8,\n 0.2, 1.0]` for MPO. If not provided, defaults to equal weights (`1.0`) for all loss types.\n ld_alpha (`float`, *optional*):\n ฮฑ parameter from the LD-DPO paper, which controls the weighting of the verbose token log-probabilities in\n responses. If `None`, no weighting is applied to the verbose part, and the loss is equivalent to the\n standard DPO loss. Must be in [0.0, 1.0]: `ld_alpha=1.0` applies no weighting, and `ld_alpha=0.0` masks\n tokens beyond shared lengths.\n f_divergence_type (`str`, *optional*, defaults to `\"reverse_kl\"`):\n f-divergence regularizer between policy and reference (f-DPO paper). Possible values are: `reverse_kl`\n (default), `forward_kl`, `js_divergence`, `alpha_divergence`.\n f_alpha_divergence_coef (`float`, *optional*, defaults to `0.5`):\n ฮฑ coefficient for the ฮฑ-divergence u^-ฮฑ regularizer, used only when `f_divergence_type='alpha_divergence'`.\n label_smoothing (`float`, *optional*, defaults to `0.0`):\n Label smoothing parameter used in Robust DPO and EXO. In Robust DPO, it is interpreted as the probability\n that a preference label is flipped and must lie in [0.0, 0.5); a typical value recommended by the Robust\n DPO paper is 0.1. In EXO, it corresponds to the ฮต label smoothing parameter, for which the paper recommends\n a typical value of 1e-3.\n beta (`float`, *optional*, defaults to `0.1`):\n Parameter controlling the deviation from the reference model. Higher ฮฒ means less deviation from the\n reference model. For the IPO loss (`loss_type='ipo'`), this value is the regularization parameter denoted\n by ฯ„ in the [paper](https://huggingface.co/papers/2310.12036).\n use_weighting (`bool`, *optional*, defaults to `False`):\n Whether to apply WPO-style weighting (https://huggingface.co/papers/2406.11827) to preference pairs using\n the policy's length-normalized sequence probabilities.\n discopop_tau (`float`, *optional*, defaults to `0.05`):\n ฯ„/temperature parameter from the DiscoPOP paper, which controls the shape of the log-ratio modulated loss\n when using `loss_type='discopop'`. The paper recommends the default value `discopop_tau=0.05`.\n activation_offloading (`bool`, *optional*, defaults to `False`):\n Whether to offload the activations to the CPU.\n sync_ref_model (`bool`, *optional*, defaults to `False`):\n Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using\n the `ref_model_mixup_alpha` parameter. This synchronization originates from the\n [TR-DPO](https://huggingface.co/papers/2404.09656) paper. `sync_ref_model=True` is not yet compatible with\n PEFT or `precompute_ref_log_probs=True`.\n ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`):\n ฮฑ parameter from the TR-DPO paper, which controls the mix between the current policy and the previous\n reference policy during updates. The reference policy is updated according to the equation: `ฯ€_ref = ฮฑ *\n ฯ€_ฮธ + (1 - ฮฑ) * ฯ€_ref_prev`. To use this parameter, you must set `sync_ref_model=True`.\n ref_model_sync_steps (`int`, *optional*, defaults to `512`):\n ฯ„ parameter from the TR-DPO paper, which determines how frequently the current policy is synchronized with\n the reference policy. To use this parameter, you must set `sync_ref_model=True`.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `1e-6` instead of `5e-5`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-6,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n # Parameters that control the model\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `model` argument of \"\n \"the `DPOTrainer` is provided as a string.\"\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropout in the model and reference model.\"},\n )\n\n # Parameters that control the data preprocessing\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes to use for processing the dataset.\"},\n )\n pad_token: str | None = field(\n default=None,\n metadata={\n \"help\": \"Token used for padding. If `None`, it defaults to `processing_class.pad_token`, or if that \"\n \"is also `None`, it falls back to `processing_class.eos_token`.\"\n },\n )\n max_length: int | None = field(\n default=1024,\n metadata={\n \"help\": \"Maximum length of the tokenized sequence. Sequences longer than `max_length` are truncated from \"\n \"the left or right depending on the `truncation_mode`. If `None`, no truncation is applied.\"\n },\n )\n truncation_mode: str = field(\n default=\"keep_start\",\n metadata={\n \"help\": \"Truncation mode to use when the sequence exceeds `max_length`. Possible values are `'keep_end'` \"\n \"and `'keep_start'`.\",\n \"choices\": [\"keep_end\", \"keep_start\"],\n },\n )\n padding_free: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to perform forward passes without padding by flattening all sequences in the batch into \"\n \"a single continuous sequence. This reduces memory usage by eliminating padding overhead. Currently, this \"\n \"is only supported with the FlashAttention 2 or 3, which can efficiently handle the flattened batch \"\n \"structure.\"\n },\n )\n pad_to_multiple_of: int | None = field(\n default=None,\n metadata={\"help\": \"If set, the sequences will be padded to a multiple of this value.\"},\n )\n precompute_ref_log_probs: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to precompute the reference model log probabilities for the entire training dataset \"\n \"before training. This allows to save memory during training, as the reference model does not need to be \"\n \"kept in memory.\"\n },\n )\n precompute_ref_batch_size: int | None = field(\n default=None,\n metadata={\n \"help\": \"Batch size to use when precomputing reference model log probabilities. This can be set higher \"\n \"than the training batch size to speed up preprocessing. If `None`, defaults to \"\n \"`per_device_train_batch_size` for training and `per_device_eval_batch_size` for evaluation.\"\n },\n )\n\n # Parameters that control the training\n loss_type: list[str] = field(\n default_factory=lambda: [\"sigmoid\"],\n metadata={\n \"help\": \"Type of loss to use. Possible values are: `'sigmoid'`, `'hinge'`, `'ipo'`, `'exo_pair'`, \"\n \"`'nca_pair'`, `'robust'`, `'bco_pair'`, `'sppo_hard'`, `'aot'`, `'aot_unpaired'`, `'apo_zero'`, \"\n \"`'apo_down'`, `'discopop'`, `'sft'`. If multiple loss types are provided, they will be combined using \"\n \"the weights specified in `loss_weights`.\",\n },\n )\n loss_weights: list[float] | None = field(\n default=None,\n metadata={\n \"help\": \"List of loss weights for multi-loss combinations. Used when combining multiple loss types. \"\n \"Example: `[0.8, 0.2, 1.0]` for MPO. If not provided, defaults to equal weights (`1.0`) for all loss \"\n \"types.\"\n },\n )\n ld_alpha: float | None = field(\n default=None,\n metadata={\n \"help\": \"ฮฑ parameter from the LD-DPO paper, which controls the weighting of the verbose token \"\n \"log-probabilities in responses. If `None`, no weighting is applied to the verbose part, and the loss is \"\n \"equivalent to the standard DPO loss. Must be in [0.0, 1.0]: `ld_alpha=1.0` applies no weighting, and \"\n \"`ld_alpha=0.0` masks tokens beyond shared lengths.\",\n },\n )\n f_divergence_type: str = field(\n default=\"reverse_kl\",\n metadata={\n \"help\": \"f-divergence regularizer between policy and reference (f-DPO paper). Possible values are: \"\n \"`reverse_kl` (default), `forward_kl`, `js_divergence`, `alpha_divergence`.\",\n },\n )\n f_alpha_divergence_coef: float = field(\n default=0.5,\n metadata={\n \"help\": \"ฮฑ coefficient for the ฮฑ-divergence u^-ฮฑ regularizer, used only when \"\n \"`f_divergence_type='alpha_divergence'`.\"\n },\n )\n label_smoothing: float = field(\n default=0.0,\n metadata={\n \"help\": \"Label smoothing parameter used in Robust DPO and EXO. In Robust DPO, it is interpreted as the \"\n \"probability that a preference label is flipped and must lie in [0.0, 0.5); a typical value recommended \"\n \"by the Robust DPO paper is 0.1. In EXO, it corresponds to the ฮต label smoothing parameter, for which the \"\n \"paper recommends a typical value of 1e-3.\"\n },\n )\n beta: float = field(\n default=0.1,\n metadata={\n \"help\": \"Parameter controlling the deviation from the reference model. Higher ฮฒ means less deviation from \"\n \"the reference model. For the IPO loss (`loss_type='ipo'`), this value is the regularization parameter \"\n \"denoted by ฯ„ in the [paper](https://huggingface.co/papers/2310.12036).\"\n },\n )\n use_weighting: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to apply WPO-style weighting (https://huggingface.co/papers/2406.11827) to preference \"\n \"pairs using the policy's length-normalized sequence probabilities.\"\n },\n )\n discopop_tau: float = field(\n default=0.05,\n metadata={\n \"help\": \"ฯ„/temperature parameter from the DiscoPOP paper, which controls the shape of the log-ratio \"\n \"modulated loss when using `loss_type='discopop'`. The paper recommends the default value \"\n \"`discopop_tau=0.05`.\"\n },\n )\n activation_offloading: bool = field(\n default=False,\n metadata={\"help\": \"Whether to offload the activations to the CPU.\"},\n )\n sync_ref_model: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to synchronize the reference model with the active model every `ref_model_sync_steps` \"\n \"steps, using the `ref_model_mixup_alpha` parameter. This synchronization originates from the \"\n \"[TR-DPO](https://huggingface.co/papers/2404.09656) paper. `sync_ref_model=True` is not yet compatible \"\n \"with PEFT or `precompute_ref_log_probs=True`.\"\n },\n )\n ref_model_mixup_alpha: float = field(\n default=0.6,\n metadata={\n \"help\": \"ฮฑ parameter from the TR-DPO paper, which controls the mix between the current policy and the \"\n \"previous reference policy during updates. The reference policy is updated according to the equation: \"\n \"`ฯ€_ref = ฮฑ * ฯ€_ฮธ + (1 - ฮฑ) * ฯ€_ref_prev`. To use this parameter, you must set `sync_ref_model=True`.\"\n },\n )\n ref_model_sync_steps: int = field(\n default=512,\n metadata={\n \"help\": \"ฯ„ parameter from the TR-DPO paper, which determines how frequently the current policy is \"\n \"synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`.\"\n },\n )\n\n def __post_init__(self):\n if isinstance(self.loss_type, str):\n self.loss_type = [self.loss_type]\n if self.loss_weights is not None and len(self.loss_weights) != len(self.loss_type):\n raise ValueError(\n \"`loss_weights` must have the same length as `loss_type` when combining multiple losses. \"\n f\"Got {len(self.loss_weights)} weights for {len(self.loss_type)} loss types.\"\n )\n\n super().__post_init__()\n" }, { "path": "trl/trainer/dpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport contextlib\nimport json\nimport os\nimport textwrap\nfrom collections import defaultdict\nfrom collections.abc import Callable\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Any\n\nimport numpy as np\nimport torch\nimport torch.nn.functional as F\nimport transformers\nfrom accelerate import PartialState\nfrom accelerate.logging import get_logger\nfrom accelerate.utils import is_peft_model, tqdm\nfrom datasets import Dataset, IterableDataset, IterableDatasetDict\nfrom datasets.fingerprint import Hasher\nfrom packaging.version import Version\nfrom torch.utils.data import DataLoader\nfrom transformers import (\n AutoProcessor,\n DataCollator,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n)\nfrom transformers.data.data_collator import DataCollatorMixin\nfrom transformers.trainer_callback import TrainerCallback\nfrom transformers.trainer_utils import EvalPrediction\nfrom transformers.utils import is_liger_kernel_available, is_peft_available\n\nfrom ..data_utils import apply_chat_template, extract_prompt, is_conversational, prepare_multimodal_messages\nfrom ..models import get_act_offloading_ctx_manager, prepare_deepspeed, prepare_fsdp\nfrom ..models.utils import disable_gradient_checkpointing\nfrom .base_trainer import _BaseTrainer\nfrom .callbacks import SyncRefModelCallback\nfrom .dpo_config import DPOConfig\nfrom .utils import (\n create_model_from_path,\n disable_dropout_in_model,\n entropy_from_logits,\n flush_left,\n flush_right,\n get_config_model_id,\n hash_module,\n pad,\n remove_none_values,\n selective_log_softmax,\n use_adapter,\n)\n\n\nif is_peft_available():\n from peft import PeftConfig, PeftModel, get_peft_model\n\n\nif is_liger_kernel_available():\n from liger_kernel.chunked_loss import LigerFusedLinearDPOLoss\n\n\nlogger = get_logger(__name__)\n\n\nFLASH_ATTENTION_VARIANTS = {\n \"flash_attention_2\",\n \"flash_attention_3\",\n \"kernels-community/flash-attn2\",\n \"kernels-community/flash-attn3\",\n \"kernels-community/vllm-flash-attn3\",\n}\n\n\ndef get_dataset_column_names(dataset: Dataset | IterableDataset) -> list[str]:\n return list(next(iter(dataset)).keys()) if dataset.column_names is None else dataset.column_names\n\n\n@dataclass\nclass DataCollatorForPreference(DataCollatorMixin):\n \"\"\"\n Data collator used for preference data. Inputs are dynamically padded to the maximum length of a batch.\n\n This collator expects each example in the input list to be a dictionary containing the keys `\"prompt_ids\"`,\n `\"chosen_ids\"` and `\"rejected_ids\"`. The collator returns a dictionary containing the following keys:\n - `\"input_ids\"`: Tensor of input IDs, padded to the maximum length of the batch. The first half of the batch\n corresponds to the `\"chosen_ids\"` and the second half to the `\"rejected_ids\"`.\n - `\"attention_mask\"`: Tensor of attention mask, padded to the maximum length of the batch.\n - `\"completion_mask\"`: Tensor indicating the positions of the completion tokens, padded to the maximum length of\n the batch.\n\n Optionally, the examples can contain a `\"ref_chosen_logps\"` and `\"ref_rejected_logps\"` keys, in which case the\n returned dictionary will also contain these keys with the corresponding tensors.\n\n Args:\n pad_token_id (`int`):\n Token ID to use for padding.\n max_length (`int`, *optional*):\n Maximum length of the sequences after concatenation. Sequences longer than `max_length` are truncated\n before padding, which avoids allocating oversized tensors for batches containing very long sequences.\n truncation_mode (`str`, *optional*, defaults to `\"keep_start\"`):\n Truncation mode when a concatenated sequence exceeds `max_length`. Possible values are `\"keep_end\"` and\n `\"keep_start\"`.\n pad_to_multiple_of (`int`, *optional*):\n If set, the sequences will be padded to a multiple of this value.\n return_tensors (`str`, *optional*, defaults to `\"pt\"`):\n Type of Tensor to return. Only `\"pt\"` is currently supported.\n\n Examples:\n ```python\n >>> from trl.trainer.dpo_trainer import DataCollatorForPreference\n\n >>> collator = DataCollatorForPreference(pad_token_id=0)\n >>> examples = [\n ... {\"prompt_ids\": [1, 2, 3], \"chosen_ids\": [4, 5], \"rejected_ids\": [6]},\n ... {\"prompt_ids\": [7, 8], \"chosen_ids\": [9], \"rejected_ids\": [10, 11]},\n ... ]\n >>> collator(examples)\n {'input_ids': tensor([[ 1, 2, 3, 4, 5],\n [ 7, 8, 9, 0, 0],\n [ 1, 2, 3, 6, 0],\n [ 7, 8, 10, 11, 0]]),\n 'attention_mask': tensor([[1, 1, 1, 1, 1],\n [1, 1, 1, 0, 0],\n [1, 1, 1, 1, 0],\n [1, 1, 1, 1, 0]]),\n 'completion_mask': tensor([[0, 0, 0, 1, 1],\n [0, 0, 1, 0, 0],\n [0, 0, 0, 1, 0],\n [0, 0, 1, 1, 0]])}\n ```\n \"\"\"\n\n pad_token_id: int\n max_length: int | None = None\n truncation_mode: str = \"keep_start\"\n pad_to_multiple_of: int | None = None\n return_tensors: str = \"pt\"\n\n def torch_call(self, examples: list[dict[str, Any]]) -> dict[str, Any]:\n prompt_chosen_ids = [example[\"prompt_ids\"] + example[\"chosen_ids\"] for example in examples]\n prompt_rejected_ids = [example[\"prompt_ids\"] + example[\"rejected_ids\"] for example in examples]\n chosen_mask = [[0] * len(example[\"prompt_ids\"]) + [1] * len(example[\"chosen_ids\"]) for example in examples]\n rejected_mask = [[0] * len(example[\"prompt_ids\"]) + [1] * len(example[\"rejected_ids\"]) for example in examples]\n\n if self.max_length is not None:\n if self.truncation_mode == \"keep_start\":\n prompt_chosen_ids = [ids[: self.max_length] for ids in prompt_chosen_ids]\n prompt_rejected_ids = [ids[: self.max_length] for ids in prompt_rejected_ids]\n chosen_mask = [m[: self.max_length] for m in chosen_mask]\n rejected_mask = [m[: self.max_length] for m in rejected_mask]\n elif self.truncation_mode == \"keep_end\":\n prompt_chosen_ids = [ids[-self.max_length :] for ids in prompt_chosen_ids]\n prompt_rejected_ids = [ids[-self.max_length :] for ids in prompt_rejected_ids]\n chosen_mask = [m[-self.max_length :] for m in chosen_mask]\n rejected_mask = [m[-self.max_length :] for m in rejected_mask]\n\n chosen_attention_mask = [[1] * len(ids) for ids in prompt_chosen_ids]\n rejected_attention_mask = [[1] * len(ids) for ids in prompt_rejected_ids]\n input_ids = prompt_chosen_ids + prompt_rejected_ids\n attention_mask = chosen_attention_mask + rejected_attention_mask\n completion_mask = chosen_mask + rejected_mask\n\n # Convert to tensor\n input_ids = [torch.tensor(ids) for ids in input_ids]\n attention_mask = [torch.tensor(m, dtype=torch.long) for m in attention_mask]\n completion_mask = [torch.tensor(m, dtype=torch.long) for m in completion_mask]\n if \"ref_chosen_logps\" in examples[0]:\n ref_chosen_logps = torch.tensor([example[\"ref_chosen_logps\"] for example in examples])\n if \"ref_rejected_logps\" in examples[0]:\n ref_rejected_logps = torch.tensor([example[\"ref_rejected_logps\"] for example in examples])\n\n # Pad\n output = {}\n output[\"input_ids\"] = pad(\n input_ids,\n padding_value=self.pad_token_id,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n )\n output[\"attention_mask\"] = pad(\n attention_mask,\n padding_value=0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n )\n output[\"completion_mask\"] = pad(\n completion_mask,\n padding_value=0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n )\n if \"ref_chosen_logps\" in examples[0]:\n output[\"ref_chosen_logps\"] = ref_chosen_logps\n if \"ref_rejected_logps\" in examples[0]:\n output[\"ref_rejected_logps\"] = ref_rejected_logps\n return output\n\n\n@dataclass\nclass DataCollatorForVisionPreference(DataCollatorMixin):\n \"\"\"\n Data collator for vision-preference tasks.\n\n Unlike text-only datasets, where the collator typically receives pre-tokenized inputs ready for batching,\n vision-language data processing involves converting images into pixel values. This conversion is disk-intensive,\n making upfront preprocessing of the entire dataset impractical. Therefore, this collator performs tokenization and\n image processing on-the-fly to efficiently prepare batches.\n\n Each input example should be a dictionary containing at least:\n - An `\"images\"` key holding a list of images, or an `\"image\"` key holding a single image.\n - Keys `\"prompt\"` `\"chosen\"` and `\"rejected\"` for the prompt and preference responses.\n\n The collator outputs a dictionary including:\n - `\"input_ids\"`: Tensor of token IDs.\n - `\"attention_mask\"`: Tensor indicating attention mask.\n - `\"completion_mask\"`: Tensor indicating which tokens correspond to completions.\n - `\"pixel_values\"`: Tensor representing image pixel values.\n\n Additional keys may be present depending on the processor, such as `\"image_grid_thw\"`.\n\n Args:\n processor ([`~transformers.ProcessorMixin`]):\n The processor used to tokenize text and process images. It must be a subclass of\n [`~transformers.ProcessorMixin`] and include a `tokenizer` with a defined `pad_token_id`.\n max_length (`int`, *optional*):\n Maximum sequence length. Sequences longer than `max_length` are truncated before padding, which avoids\n allocating oversized tensors for batches containing very long sequences. Only `\"keep_start\"` truncation\n applies to vision datasets; `\"keep_end\"` is rejected upstream.\n pad_to_multiple_of (`int` or `None`, optional, defaults to `None`):\n If set, the sequences will be padded to a multiple of this value.\n return_tensors (`str`, optional, defaults to `\"pt\"`):\n The tensor type to return. Currently, only `\"pt\"` (PyTorch tensors) is supported.\n\n Example:\n ```python\n >>> from trl.trainer.dpo_trainer import DataCollatorForVisionPreference\n >>> from transformers import AutoProcessor\n\n >>> processor = AutoProcessor.from_pretrained(\"Qwen/Qwen2.5-VL-7B-Instruct\")\n >>> collator = DataCollatorForVisionPreference(processor)\n >>> examples = [\n ... {\n ... \"images\": [Image.open(\"image_0.png\")],\n ... \"prompt\": [{\"role\": \"user\", \"content\": \"What is this?\"}],\n ... \"chosen\": [{\"role\": \"assistant\", \"content\": \"This is a cat.\"}],\n ... \"rejected\": [{\"role\": \"assistant\", \"content\": \"This is a dog.\"}],\n ... },\n ... {\n ... \"images\": [Image.open(\"image_1.png\")],\n ... \"prompt\": [{\"role\": \"user\", \"content\": \"Describe this image.\"}],\n ... \"chosen\": [{\"role\": \"assistant\", \"content\": \"A beautiful landscape.\"}],\n ... \"rejected\": [{\"role\": \"assistant\", \"content\": \"An urban cityscape.\"}],\n ... },\n ... ]\n >>> collator(examples)\n {'input_ids': tensor([[151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 3838, 374, 419, 30, 151645, 198, 151644, 77091, 198, 1986, 374, 264, 8251, 13, 151645, 198],\n [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 74785, 419, 2168, 13, 151645, 198, 151644, 77091, 198, 32, 6233, 18414, 13, 151645, 198, 151643],\n [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 3838, 374, 419, 30, 151645, 198, 151644, 77091, 198, 1986, 374, 264, 5562, 13, 151645, 198],\n [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 74785, 419, 2168, 13, 151645, 198, 151644, 77091, 198, 2082, 15662, 3283, 57518, 13, 151645, 198]]),\n 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]),\n 'pixel_values': tensor([[-1.3251, 0.1347, -0.4784, ..., 0.4537, -0.0156, 1.2358],\n [ 0.5727, 0.4997, -0.9164, ..., -0.5701, 0.7950, -0.7123],\n [-0.0550, -0.8288, 1.0690, ..., -0.1293, -0.1151, 1.6055],\n ...,\n [ 0.2953, 0.5581, 0.1785, ..., -0.7123, -0.7977, 0.1693],\n [-0.7558, 1.0398, 1.3464, ..., -0.5417, -0.5417, 0.4395],\n [ 0.8063, 0.6895, 0.4267, ..., -0.4422, 1.3354, 0.1266]]),\n 'image_grid_thw': tensor([[1, 4, 4],\n [1, 4, 4],\n [1, 4, 4],\n [1, 4, 4]]),\n 'completion_mask': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0],\n [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]])}\n ```\n \"\"\"\n\n processor: ProcessorMixin\n max_length: int | None = None\n pad_to_multiple_of: int | None = None\n return_tensors: str = \"pt\"\n\n def torch_call(self, examples: list[dict[str, Any]]) -> dict[str, Any]:\n if self.pad_to_multiple_of is not None:\n raise NotImplementedError(\n \"Padding to a multiple of a value is not yet implemented for vision-language modeling and \"\n \"prompt-completion data.\"\n )\n if \"image\" in examples[0]:\n for example in examples:\n example[\"images\"] = [example.pop(\"image\")]\n images = [example[\"images\"] for example in examples] * 2 # repeat for chosen and rejected\n # Transformers requires at least one image in the batch, otherwise it throws an error\n if all(img_list == [] for img_list in images):\n images = None\n if is_conversational(examples[0]): # conversational case\n for example in examples:\n example[\"prompt\"] = prepare_multimodal_messages(example[\"prompt\"], images=example[\"images\"])\n example[\"chosen\"] = prepare_multimodal_messages(example[\"chosen\"], images=[])\n example[\"rejected\"] = prepare_multimodal_messages(example[\"rejected\"], images=[])\n examples = [apply_chat_template(example, self.processor) for example in examples]\n\n prompts = [example[\"prompt\"] for example in examples] * 2 # repeat for chosen and rejected\n chosens = [example[\"chosen\"] for example in examples]\n rejecteds = [example[\"rejected\"] for example in examples]\n\n processed_prompts = self.processor(\n images=images,\n text=prompts,\n padding=True,\n padding_side=\"left\",\n return_tensors=self.return_tensors,\n add_special_tokens=False, # to avoid adding the BOS, twice see https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior#7-chat-template-and-tokenization-dont-compose-due-to-special-tokens\n )\n processed_chosens = self.processor(\n text=chosens,\n padding=True,\n padding_side=\"right\",\n return_tensors=self.return_tensors,\n add_special_tokens=False, # to avoid adding the BOS, twice see https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior#7-chat-template-and-tokenization-dont-compose-due-to-special-tokens\n )\n processed_rejecteds = self.processor(\n text=rejecteds,\n padding=True,\n padding_side=\"right\",\n return_tensors=self.return_tensors,\n add_special_tokens=False, # to avoid adding the BOS, twice see https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior#7-chat-template-and-tokenization-dont-compose-due-to-special-tokens\n )\n\n # Concatenate prompts and completions\n prompt_ids, prompt_mask = processed_prompts[\"input_ids\"], processed_prompts[\"attention_mask\"]\n chosen_ids, chosen_mask = processed_chosens[\"input_ids\"], processed_chosens[\"attention_mask\"]\n rejected_ids, rejected_mask = processed_rejecteds[\"input_ids\"], processed_rejecteds[\"attention_mask\"]\n pad_token_id = self.processor.tokenizer.pad_token_id or self.processor.tokenizer.eos_token_id\n completion_ids = torch.cat(tuple(pad([chosen_ids, rejected_ids], padding_value=pad_token_id)))\n completion_mask = torch.cat(tuple(pad([chosen_mask, rejected_mask], padding_value=0)))\n input_ids = torch.cat((prompt_ids, completion_ids), dim=1)\n attention_mask = torch.cat((prompt_mask, completion_mask), dim=1)\n completion_mask = torch.cat((torch.zeros_like(prompt_mask), completion_mask), dim=1)\n if \"token_type_ids\" in processed_prompts: # special case for Gemma\n prompt_token_type_ids = processed_prompts[\"token_type_ids\"]\n chosen_type_ids = processed_chosens[\"token_type_ids\"]\n rejected_type_ids = processed_rejecteds[\"token_type_ids\"]\n completion_token_type_ids = torch.cat(tuple(pad([chosen_type_ids, rejected_type_ids], padding_value=0)))\n token_type_ids = torch.cat((prompt_token_type_ids, completion_token_type_ids), dim=1)\n if \"mm_token_type_ids\" in processed_prompts: # special case for Qwen2.5-VL\n prompt_mm_token_type_ids = processed_prompts[\"mm_token_type_ids\"]\n mm_token_type_ids = torch.cat((prompt_mm_token_type_ids, torch.zeros_like(completion_ids)), dim=1)\n\n # Flush left to reduce padding\n if \"token_type_ids\" in processed_prompts and \"mm_token_type_ids\" in processed_prompts:\n attention_mask, input_ids, completion_mask, token_type_ids, mm_token_type_ids = flush_left(\n attention_mask, input_ids, completion_mask, token_type_ids, mm_token_type_ids\n )\n elif \"token_type_ids\" in processed_prompts:\n attention_mask, input_ids, completion_mask, token_type_ids = flush_left(\n attention_mask, input_ids, completion_mask, token_type_ids\n )\n elif \"mm_token_type_ids\" in processed_prompts:\n attention_mask, input_ids, completion_mask, mm_token_type_ids = flush_left(\n attention_mask, input_ids, completion_mask, mm_token_type_ids\n )\n else:\n attention_mask, input_ids, completion_mask = flush_left(attention_mask, input_ids, completion_mask)\n\n if self.max_length is not None:\n input_ids = input_ids[:, : self.max_length]\n attention_mask = attention_mask[:, : self.max_length]\n completion_mask = completion_mask[:, : self.max_length]\n if \"token_type_ids\" in processed_prompts:\n token_type_ids = token_type_ids[:, : self.max_length]\n if \"mm_token_type_ids\" in processed_prompts:\n mm_token_type_ids = mm_token_type_ids[:, : self.max_length]\n\n # Build the output dictionary\n output = processed_prompts # we take processed_prompts because it contains the images\n output[\"input_ids\"] = input_ids\n output[\"attention_mask\"] = attention_mask\n output[\"completion_mask\"] = completion_mask\n if \"token_type_ids\" in processed_prompts:\n output[\"token_type_ids\"] = token_type_ids\n if \"mm_token_type_ids\" in processed_prompts:\n output[\"mm_token_type_ids\"] = mm_token_type_ids\n return output\n\n\nclass DPOTrainer(_BaseTrainer):\n \"\"\"\n Trainer for Direct Preference Optimization (DPO) method. This algorithm was initially proposed in the paper [Direct\n Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).\n This class is a wrapper around the [`~transformers.Trainer`] class and inherits all of its attributes and methods.\n\n Example:\n\n ```python\n from trl import DPOTrainer\n from datasets import load_dataset\n\n dataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\n trainer = DPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n train_dataset=dataset,\n )\n trainer.train()\n ```\n\n Args:\n model (`str` or [`~transformers.PreTrainedModel`] or [`~peft.PeftModel`]):\n Model to be trained. Can be either:\n\n - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using `.from_pretrained` (where `` is derived from the model\n config) with the keyword arguments in `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported.\n - A [`~peft.PeftModel`] object. Only causal language models are supported.\n ref_model (`PreTrainedModel`, *optional*):\n Reference model used to compute the reference log probabilities.\n\n - If provided, this model is used directly as the reference policy.\n - If `None`, the trainer will automatically use the initial policy corresponding to `model`, i.e. the model\n state before DPO training starts.\n args ([`DPOConfig`], *optional*):\n Configuration for this trainer. If `None`, a default configuration is used.\n data_collator ([`~transformers.DataCollator`], *optional*):\n Function to use to form a batch from a list of elements of the processed `train_dataset` or `eval_dataset`.\n Will default to [`~trainer.dpo_trainer.DataCollatorForPreference`] if the model is a language model and\n [`~trainer.dpo_trainer.DataCollatorForVisionPreference`] if the model is a vision-language model.\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n Dataset to use for training. This trainer supports both [language modeling](#language-modeling) type and\n [prompt-completion](#prompt-completion) type. The format of the samples can be either:\n\n - [Standard](dataset_formats#standard): Each sample contains plain text.\n - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role\n and content).\n eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Dataset | IterableDataset]`):\n Dataset to use for evaluation. It must meet the same requirements as `train_dataset`.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. The padding side must be set to \"left\". If `None`, the\n processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A\n padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token,\n `tokenizer.eos_token` will be used as the default.\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function that will be used to compute metrics at evaluation. Must take a\n [`~transformers.EvalPrediction`] and return a dictionary string to metric values. When passing\n [`SFTConfig`] with `batch_eval_metrics` set to `True`, your `compute_metrics` function must take a boolean\n `compute_result` argument. This will be triggered after the last eval batch to signal that the function\n needs to calculate and return the global summary statistics rather than accumulating the batch-level\n statistics.\n callbacks (list of [`~transformers.TrainerCallback`], *optional*):\n List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed\n in [here](https://huggingface.co/docs/transformers/main_classes/callback).\n\n If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`]\n method.\n optimizers (`tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None]`, *optional*, defaults to `(None, None)`):\n A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your\n model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`.\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration used to wrap the model. If `None`, the model is not wrapped.\n \"\"\"\n\n _tag_names = [\"trl\", \"dpo\"]\n _name = \"DPO\"\n _paper = {\n \"title\": \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\",\n \"id\": \"2305.18290\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @inproceedings{rafailov2023direct,\n title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},\n author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},\n year = 2023,\n booktitle = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},\n url = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},\n editor = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: \"str | PreTrainedModel | PeftModel\",\n ref_model: PreTrainedModel | None = None,\n args: DPOConfig | None = None,\n data_collator: DataCollator | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | IterableDataset | dict[str, Dataset | IterableDataset] | None = None,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin | None = None,\n compute_metrics: Callable[[EvalPrediction], dict] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None] = (None, None),\n peft_config: \"PeftConfig | None\" = None,\n ):\n # Args\n if args is None:\n model_name = model if isinstance(model, str) else get_config_model_id(model.config)\n model_name = model_name.split(\"/\")[-1]\n args = DPOConfig(f\"{model_name}-DPO\")\n\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n elif isinstance(train_dataset, IterableDataset):\n # IterableDataset requires dispatch_batches=False because Accelerate's dispatch mode may try to concatenate\n # batches from multiple processes, leading to mismatch errors.\n if args.accelerator_config.dispatch_batches is True:\n logger.warning(\n \"You are using an `IterableDataset` for training with `dispatch_batches=True`. `dispatch_batches` \"\n \"is forced to `False` when using an `IterableDataset`. To remove this warning, unset \"\n \"`dispatch_batches` in `DPOConfig` or set it to `False`.\"\n )\n args.accelerator_config.dispatch_batches = False\n\n # Model\n if isinstance(model, str):\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n model = create_model_from_path(model, **model_init_kwargs)\n else:\n if args.model_init_kwargs is not None:\n logger.warning(\n \"You passed `model_init_kwargs` to the `DPOConfig`, but your model is already instantiated. \"\n \"The `model_init_kwargs` will be ignored.\"\n )\n if ref_model is model:\n raise ValueError(\n \"`model` and `ref_model` cannot be the same object. In most cases you should omit `ref_model` and \"\n \"we'll initialize it to a copy of `model` for you.\"\n )\n\n # Processing class\n if processing_class is None:\n processing_class = AutoProcessor.from_pretrained(get_config_model_id(model.config))\n\n # Handle pad token for processors or tokenizers\n if isinstance(processing_class, ProcessorMixin):\n tokenizer = processing_class.tokenizer\n self._is_vlm = True\n elif isinstance(processing_class, PreTrainedTokenizerBase):\n tokenizer = processing_class\n self._is_vlm = False\n else:\n raise TypeError(\"The `processing_class` must be either a `PreTrainedTokenizerBase` or a `ProcessorMixin`\")\n\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n self.pad_token = tokenizer.pad_token\n self.pad_token_id = tokenizer.pad_token_id\n self.eos_token_id = tokenizer.eos_token_id\n\n if is_peft_available() and is_peft_model(model) and peft_config is not None:\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first merge \"\n \"and unload the existing adapter, save the resulting base model, and then pass that base model along \"\n \"with the new `peft_config` to the trainer.\"\n )\n if is_peft_available() and is_peft_model(model) and ref_model is None:\n # If the model is a PEFT model with a pretrained adapter, we need to create a \"ref\" adapter that is a copy\n # of the \"default\" adapter, so that we can use it as the reference model during DPO training.\n model.add_adapter(\"ref\", model.peft_config[\"default\"])\n for name, param in model.named_parameters():\n if \".default.\" in name:\n ref_name = name.replace(\".default.\", \".ref.\")\n ref_param = model.get_parameter(ref_name)\n ref_param.data.copy_(param.data)\n\n # Create PEFT model\n if peft_config is not None:\n model = get_peft_model(model, peft_config)\n\n # When using gradient checkpointing with PEFT, we need to enable input gradients. transformers.Trainer normally\n # handles this, but a bug currently prevents it; see https://github.com/huggingface/transformers/issues/42489\n if is_peft_available() and isinstance(model, PeftModel) and args.gradient_checkpointing:\n model.enable_input_require_grads()\n\n # When using QLoRA, the PEFT adapter weights are converted to bf16 to follow the recommendations from the\n # original paper (see https://huggingface.co/papers/2305.14314, paragraph 3). Normally, this can be done by\n # passing `autocast_adapter_dtype=False` to `get_peft_model`, but this option is not yet supported for\n # quantized models. See: https://github.com/huggingface/peft/issues/2889\n # Non-quantized models do not have the `is_loaded_in_{8,4}bit` attributes, whereas quantized models do\n if getattr(model, \"is_loaded_in_4bit\", False) or getattr(model, \"is_loaded_in_8bit\", False):\n for param in model.parameters():\n if param.requires_grad:\n param.data = param.data.to(torch.bfloat16)\n\n # Data collator\n self.padding_free = args.padding_free\n if self.padding_free:\n logger.warning(\n \"`padding_free=True` is temporarily unavailable after a refactor and is currently disabled. Falling \"\n \"back to standard padding (`padding_free=False`). This feature is planned to return in a future \"\n \"update; for now, please set `padding_free=False` explicitly.\"\n )\n self.padding_free = False\n dataset_sample = next(iter(train_dataset))\n self._is_vision_dataset = \"image\" in dataset_sample or \"images\" in dataset_sample\n if self._is_vision_dataset and not self._is_vlm:\n raise ValueError(\n \"The dataset appears to be vision-related (contains 'image' or 'images' keys), but the provided \"\n \"model does not seem to be a vision-language model. Please check your model and dataset.\"\n )\n if self._is_vision_dataset and args.max_length is not None and args.truncation_mode == \"keep_end\":\n raise ValueError(\n \"truncation_mode='keep_end' is not supported for vision-language models. Image tokens reside \"\n \"inside the prompt portion of the sequence; depending on the example, keep_end may silently \"\n \"drop them, causing pixel_values to be forwarded to the model with no corresponding visual \"\n \"tokens in input_ids. Use truncation_mode='keep_start' (the default) or set max_length=None.\"\n )\n if data_collator is None and not self._is_vision_dataset:\n # Get the pad token: if not provided, use the one from the processing class or the eos token\n # if the processing class does not have a pad token.\n pad_token = args.pad_token or tokenizer.pad_token or tokenizer.eos_token\n pad_token_id = tokenizer.convert_tokens_to_ids(pad_token)\n if pad_token_id is None:\n raise ValueError(\n f\"The specified `pad_token` ('{pad_token}') is not found in the vocabulary of the given \"\n f\"`processing_class` ({processing_class.__class__.__name__}). Ensure that the `pad_token` exists \"\n \"in the vocabulary before using it as a padding token.\"\n )\n data_collator = DataCollatorForPreference(\n pad_token_id=pad_token_id,\n max_length=args.max_length,\n truncation_mode=args.truncation_mode,\n pad_to_multiple_of=args.pad_to_multiple_of,\n )\n elif data_collator is None and self._is_vision_dataset:\n data_collator = DataCollatorForVisionPreference(\n processor=processing_class,\n max_length=args.max_length,\n pad_to_multiple_of=args.pad_to_multiple_of,\n )\n\n # Training arguments\n self.beta = args.beta\n self.precompute_ref_logps = args.precompute_ref_log_probs\n self.loss_types = args.loss_type # args.loss_type is already a list\n self.loss_weights = args.loss_weights or [1.0] * len(self.loss_types)\n self.ld_alpha = args.ld_alpha\n self.f_divergence_type = args.f_divergence_type\n self.f_alpha_divergence_coef = args.f_alpha_divergence_coef\n self.label_smoothing = args.label_smoothing\n self.use_weighting = args.use_weighting\n if self.use_weighting and any(loss_type in {\"aot\", \"aot_unpaired\"} for loss_type in self.loss_types):\n raise NotImplementedError(\n \"WPO-style weighting is not implemented for 'aot' or 'aot_unpaired' because those losses sort \"\n \"samples, which would misalign per-pair weights.\"\n )\n if \"robust\" in self.loss_types and not (0.0 <= self.label_smoothing < 0.5):\n logger.warning(\n \"The `label_smoothing` parameter should lie in [0.0, 0.5) for the 'robust' loss. You provided \"\n f\"{self.label_smoothing}.\"\n )\n if \"exo_pair\" in self.loss_types and self.label_smoothing == 0.0:\n raise ValueError(\n \"Label smoothing must be greater than 0.0 when using 'exo_pair' loss. The EXO paper recommends a \"\n \"value of 1e-3.\"\n )\n self.use_liger_kernel = args.use_liger_kernel\n if args.use_liger_kernel:\n if not is_liger_kernel_available():\n raise ImportError(\n \"You set `use_liger_kernel=True` but the liger kernel is not available. \"\n \"Please install liger-kernel first: `pip install liger-kernel`\"\n )\n if len(self.loss_types) != 1:\n raise NotImplementedError(\n \"Multiple loss types are not yet supported when using Liger kernel. If you need this feature, \"\n \"please open a feature request at https://github.com/huggingface/trl/issues.\"\n )\n self.liger_loss_fn = LigerFusedLinearDPOLoss(beta=args.beta, loss_type=self.loss_types[0])\n if compute_metrics is not None:\n raise ValueError(\n \"compute_metrics is not supported with the Liger kernel. compute_metrics requires to be able to \"\n \"recover the logits from the forward pass, but Liger kernel does not materialize logits.\"\n )\n if self.precompute_ref_logps:\n raise ValueError(\n \"Liger DPO loss does not support precomputing reference log probabilities. Either disable \"\n \"`precompute_ref_log_probs` or set `use_liger_kernel` to False.\"\n )\n\n # Dataset\n # Skip dataset preparation if it's a VLM, where preprocessing (e.g., image-to-pixel conversion) is too costly\n # and done on the fly instead.\n skip_prepare_dataset = self._is_vision_dataset\n if not skip_prepare_dataset:\n train_dataset = self._prepare_dataset(train_dataset, processing_class, args, \"train\")\n if eval_dataset is not None:\n if isinstance(eval_dataset, dict):\n eval_dataset = {\n key: self._prepare_dataset(dataset, processing_class, args, key)\n for key, dataset in eval_dataset.items()\n }\n else:\n eval_dataset = self._prepare_dataset(eval_dataset, processing_class, args, \"eval\")\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n )\n\n # Initialize activation offloading context\n if self.args.activation_offloading:\n self.maybe_activation_offload_context = get_act_offloading_ctx_manager(model=self.model)\n else:\n self.maybe_activation_offload_context = contextlib.nullcontext()\n\n # Reference model\n if ref_model is None:\n if is_peft_model(self.model):\n # If PEFT is used, the reference model is not needed since the adapter can be disabled to revert to the\n # initial model.\n self.ref_model = None\n else:\n ref_model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if self.args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n ref_model_init_kwargs[\"device_map\"] = None\n ref_model_path = get_config_model_id(self.model.config)\n self.ref_model = create_model_from_path(ref_model_path, **ref_model_init_kwargs)\n else:\n self.ref_model = ref_model\n\n # Disable dropout in the models\n if args.disable_dropout:\n disable_dropout_in_model(model)\n if self.ref_model is not None:\n disable_dropout_in_model(self.ref_model)\n\n # Initialize the metrics\n self._metrics = {\"train\": defaultdict(list), \"eval\": defaultdict(list)}\n self._total_train_tokens = 0\n\n # Add tags to the model\n self.model.add_model_tags(self._tag_names)\n\n if self.ref_model is not None:\n if self.is_deepspeed_enabled:\n self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)\n elif self.is_fsdp_enabled:\n self.ref_model = prepare_fsdp(self.ref_model, self.accelerator)\n else:\n self.ref_model = self.accelerator.prepare_model(self.ref_model, evaluation_mode=True)\n\n if args.sync_ref_model:\n if self.ref_model is None:\n raise NotImplementedError(\n \"You passed `sync_ref_model=True` while using a PEFT model, which is currently not supported. \"\n \"With PEFT, DPOTrainer does not keep a separate reference model in memory; instead, it recovers \"\n \"reference behavior by temporarily disabling the adapter. As a result, there is no standalone \"\n \"`ref_model` instance to synchronize. Use `sync_ref_model=False`, or opt for full fine-tuning if \"\n \"you need a synced reference model. If you need `sync_ref_model` to work with PEFT, please open a \"\n \"feature request at https://github.com/huggingface/trl/issues.\"\n )\n if args.precompute_ref_log_probs:\n raise ValueError(\n \"You cannot use `sync_ref_model=True` together with `precompute_ref_log_probs=True`. \"\n \"`precompute_ref_log_probs=True` assumes a fixed reference model, but with `sync_ref_model=True` \"\n \"the reference model is periodically updated during training, making any precomputed reference \"\n \"log-probs stale. Set `precompute_ref_log_probs=False` or disable `sync_ref_model`.\"\n )\n self.add_callback(SyncRefModelCallback(ref_model=self.ref_model, accelerator=self.accelerator))\n\n if args.precompute_ref_log_probs:\n if isinstance(self.train_dataset, IterableDataset) or isinstance(\n self.eval_dataset, (IterableDataset, IterableDatasetDict)\n ):\n raise ValueError(\n \"`precompute_ref_log_probs=True` is not supported with IterableDataset. Please use a map-style \"\n \"Dataset or set `precompute_ref_log_probs=False`.\"\n )\n\n batch_size = self.args.precompute_ref_batch_size or self.args.per_device_train_batch_size\n self.train_dataset = self._precompute_ref_logps(self.train_dataset, \"train\", batch_size)\n if self.eval_dataset is not None:\n batch_size = self.args.precompute_ref_batch_size or self.args.per_device_eval_batch_size\n if isinstance(self.eval_dataset, dict):\n self.eval_dataset = {\n name: self._precompute_ref_logps(dataset, name, batch_size)\n for name, dataset in self.eval_dataset.items()\n }\n else:\n self.eval_dataset = self._precompute_ref_logps(self.eval_dataset, \"eval\", batch_size)\n\n def _prepare_dataset(\n self,\n dataset: Dataset | IterableDataset,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin,\n args: DPOConfig,\n dataset_name: str,\n ) -> Dataset | IterableDataset:\n # Tabular backends like Arrow/Parquet insert `None` for mismatched keys in nested structures. Clean them from\n # sampled data.\n if isinstance(dataset, Dataset): # IterableDataset does not support `with_transform`\n dataset = dataset.with_transform(remove_none_values)\n\n # Build the kwargs for the `map` function\n map_kwargs = {}\n if isinstance(dataset, Dataset): # IterableDataset does not support num_proc\n map_kwargs[\"num_proc\"] = args.dataset_num_proc\n\n with PartialState().main_process_first():\n # Extract the prompt if needed\n first_example = next(iter(dataset))\n if \"prompt\" not in first_example:\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Extracting prompt from {dataset_name} dataset\"\n dataset = dataset.map(extract_prompt, **map_kwargs)\n\n # Apply the chat template if needed\n first_example = next(iter(dataset))\n if not is_conversational(first_example):\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Adding EOS to {dataset_name} dataset\"\n\n def add_eos(example, eos_token):\n if not example[\"chosen\"].endswith(eos_token):\n example[\"chosen\"] = example[\"chosen\"] + eos_token\n if not example[\"rejected\"].endswith(eos_token):\n example[\"rejected\"] = example[\"rejected\"] + eos_token\n return example\n\n eos_token = processing_class.tokenizer.eos_token if self._is_vlm else processing_class.eos_token\n dataset = dataset.map(add_eos, fn_kwargs={\"eos_token\": eos_token}, **map_kwargs)\n\n # Tokenize the dataset\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Tokenizing {dataset_name} dataset\"\n\n def tokenize_fn(example, processing_class):\n tools = example.get(\"tools\")\n tools = json.loads(tools) if isinstance(tools, str) else tools\n output = {}\n if is_conversational(example):\n if self._is_vlm:\n prompt = prepare_multimodal_messages(example[\"prompt\"], images=[])\n chosen = prepare_multimodal_messages(example[\"chosen\"], images=[])\n rejected = prepare_multimodal_messages(example[\"rejected\"], images=[])\n else:\n prompt = example[\"prompt\"]\n chosen = example[\"chosen\"]\n rejected = example[\"rejected\"]\n prompt_ids = processing_class.apply_chat_template(\n prompt,\n tools=tools,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=False,\n **example.get(\"chat_template_kwargs\", {}),\n )\n prompt_chosen_processed = processing_class.apply_chat_template(\n prompt + chosen,\n tools=tools,\n tokenize=True,\n return_dict=True,\n **example.get(\"chat_template_kwargs\", {}),\n )\n prompt_rejected_processed = processing_class.apply_chat_template(\n prompt + rejected,\n tools=tools,\n tokenize=True,\n return_dict=True,\n **example.get(\"chat_template_kwargs\", {}),\n )\n # Fix transformers inconsistency: for VLMs, apply_chat_template returns lists of lists\n # even for single examples, while for LLMs it returns lists of ints.\n prompt_ids = prompt_ids[0] if isinstance(prompt_ids[0], list) else prompt_ids\n prompt_chosen_processed = {\n k: v[0] if isinstance(v[0], list) else v for k, v in prompt_chosen_processed.items()\n }\n prompt_rejected_processed = {\n k: v[0] if isinstance(v[0], list) else v for k, v in prompt_rejected_processed.items()\n }\n prompt_chosen_ids = prompt_chosen_processed[\"input_ids\"]\n prompt_rejected_ids = prompt_rejected_processed[\"input_ids\"]\n else:\n prompt_ids = processing_class(text=example[\"prompt\"])[\"input_ids\"]\n prompt_chosen_ids = processing_class(text=example[\"prompt\"] + example[\"chosen\"])[\"input_ids\"]\n prompt_rejected_ids = processing_class(text=example[\"prompt\"] + example[\"rejected\"])[\"input_ids\"]\n # Fix transformers inconsistency: for VLMs, processing_class returns lists of lists\n # even for single examples, while for LLMs it returns lists of ints.\n prompt_ids = prompt_ids[0] if isinstance(prompt_ids[0], list) else prompt_ids\n prompt_chosen_ids = (\n prompt_chosen_ids[0] if isinstance(prompt_chosen_ids[0], list) else prompt_chosen_ids\n )\n prompt_rejected_ids = (\n prompt_rejected_ids[0] if isinstance(prompt_rejected_ids[0], list) else prompt_rejected_ids\n )\n\n # Check if the tokenized prompt starts with the tokenized prompt+completion\n if not prompt_chosen_ids[: len(prompt_ids)] == prompt_ids:\n logger.warning(\n \"Mismatch between tokenized prompt and the start of tokenized prompt+chosen. \"\n \"This may be due to unexpected tokenizer behavior, whitespace issues, or special \"\n \"token handling. Verify that the tokenizer is processing text consistently.\"\n )\n if not prompt_rejected_ids[: len(prompt_ids)] == prompt_ids:\n logger.warning(\n \"Mismatch between tokenized prompt and the start of tokenized prompt+rejected. \"\n \"This may be due to unexpected tokenizer behavior, whitespace issues, or special \"\n \"token handling. Verify that the tokenizer is processing text consistently.\"\n )\n\n output[\"prompt_ids\"] = prompt_ids\n output[\"chosen_ids\"] = prompt_chosen_ids[len(prompt_ids) :]\n output[\"rejected_ids\"] = prompt_rejected_ids[len(prompt_ids) :]\n return output\n\n dataset = dataset.map(tokenize_fn, fn_kwargs={\"processing_class\": processing_class}, **map_kwargs)\n\n return dataset\n\n def _set_signature_columns_if_needed(self):\n # If `self.args.remove_unused_columns` is True, non-signature columns are removed.\n # By default, this method sets `self._signature_columns` to the model's expected inputs (usually, \"input_ids\"\n # and \"attention_mask\").\n if self._signature_columns is None:\n if self._is_vision_dataset:\n self._signature_columns = [\n \"prompt\",\n \"chosen\",\n \"rejected\",\n \"image\",\n \"images\",\n \"tools\",\n \"chat_template_kwargs\",\n ]\n else:\n self._signature_columns = [\n \"prompt_ids\",\n \"chosen_ids\",\n \"rejected_ids\",\n \"ref_chosen_logps\",\n \"ref_rejected_logps\",\n ]\n\n def _precompute_ref_logps(self, dataset: Dataset, name: str, batch_size: int) -> Dataset:\n model_hash = hash_module(self.ref_model or self.model)\n fingerprint = Hasher.hash((dataset._fingerprint, model_hash))\n cache_file = dataset._get_cache_file_path(fingerprint).removesuffix(\".arrow\") + \".npz\"\n if os.path.exists(cache_file):\n loaded = np.load(cache_file)\n ref_chosen_logps = loaded[\"ref_chosen_logps\"]\n ref_rejected_logps = loaded[\"ref_rejected_logps\"]\n else:\n dataloader = DataLoader(\n dataset,\n batch_size=batch_size,\n collate_fn=self.data_collator,\n num_workers=self.args.dataloader_num_workers,\n pin_memory=self.args.dataloader_pin_memory,\n shuffle=False,\n )\n data_loader = self.accelerator.prepare(dataloader)\n ref_chosen_logps = []\n ref_rejected_logps = []\n for padded_batch in tqdm(iterable=data_loader, desc=f\"Computing reference log probs for {name} dataset\"):\n ref_chosen_logp, ref_rejected_logp = self.compute_ref_log_probs(padded_batch)\n ref_chosen_logp, ref_rejected_logp = self.accelerator.gather_for_metrics(\n (ref_chosen_logp, ref_rejected_logp)\n )\n ref_chosen_logps.append(ref_chosen_logp.cpu())\n ref_rejected_logps.append(ref_rejected_logp.cpu())\n\n # Save the reference log probabilities to cache. We need .float() because bf16 is not supported by numpy\n ref_chosen_logps = torch.cat(ref_chosen_logps).float().numpy()\n ref_rejected_logps = torch.cat(ref_rejected_logps).float().numpy()\n if self.accelerator.is_main_process:\n np.savez_compressed(\n cache_file, ref_chosen_logps=ref_chosen_logps, ref_rejected_logps=ref_rejected_logps\n )\n self.accelerator.wait_for_everyone()\n\n dataset = dataset.add_column(name=\"ref_chosen_logps\", column=ref_chosen_logps)\n dataset = dataset.add_column(name=\"ref_rejected_logps\", column=ref_rejected_logps, new_fingerprint=fingerprint)\n\n return dataset\n\n def _truncate_inputs(\n self,\n input_ids: torch.Tensor,\n attention_mask: torch.Tensor,\n completion_mask: torch.Tensor,\n *extra: torch.Tensor,\n ) -> tuple[torch.Tensor, ...]:\n if self.args.max_length is None:\n return input_ids, attention_mask, completion_mask, *extra\n\n if self.args.truncation_mode == \"keep_start\":\n input_ids = input_ids[:, : self.args.max_length]\n attention_mask = attention_mask[:, : self.args.max_length]\n completion_mask = completion_mask[:, : self.args.max_length]\n extra = tuple(t[:, : self.args.max_length] for t in extra)\n elif self.args.truncation_mode == \"keep_end\":\n attention_mask, input_ids, completion_mask, *extra = flush_right(\n attention_mask, input_ids, completion_mask, *extra\n )\n input_ids = input_ids[:, -self.args.max_length :]\n attention_mask = attention_mask[:, -self.args.max_length :]\n completion_mask = completion_mask[:, -self.args.max_length :]\n extra = tuple(t[:, -self.args.max_length :] for t in extra)\n attention_mask, input_ids, completion_mask, *extra = flush_left(\n attention_mask, input_ids, completion_mask, *extra\n )\n extra = tuple(extra)\n else:\n raise ValueError(\n f\"Unsupported truncation mode: {self.args.truncation_mode}, expected 'keep_start' or 'keep_end'\"\n )\n\n return input_ids, attention_mask, completion_mask, *extra\n\n def compute_ref_log_probs(self, inputs):\n \"\"\"Computes reference log probabilities for a single padded batch.\"\"\"\n device = self.accelerator.device\n\n input_ids = inputs[\"input_ids\"]\n attention_mask = inputs[\"attention_mask\"]\n completion_mask = inputs[\"completion_mask\"]\n # token_type_ids and mm_token_type_ids are sequence-length-aligned: truncate to match input_ids\n extra_keys = [k for k in (\"token_type_ids\", \"mm_token_type_ids\") if k in inputs]\n input_ids, attention_mask, completion_mask, *extra = self._truncate_inputs(\n input_ids, attention_mask, completion_mask, *[inputs[k] for k in extra_keys]\n )\n\n shift_labels = input_ids[..., 1:].contiguous()\n shift_completion_mask = completion_mask[..., 1:].contiguous()\n\n model_kwargs = {\"input_ids\": input_ids, \"attention_mask\": attention_mask, \"use_cache\": False}\n for key, val in zip(extra_keys, extra, strict=False):\n model_kwargs[key] = val\n for key in (\"pixel_values\", \"pixel_attention_mask\", \"image_grid_thw\", \"image_sizes\"):\n if key in inputs:\n model_kwargs[key] = inputs[key]\n\n with torch.no_grad(), disable_gradient_checkpointing(self.model, self.args.gradient_checkpointing_kwargs):\n if is_peft_model(self.model) and self.ref_model is None:\n model = self.accelerator.unwrap_model(self.model)\n with use_adapter(model, adapter_name=\"ref\" if \"ref\" in model.peft_config else None):\n ref_outputs = self.model(**model_kwargs)\n else:\n ref_outputs = self.ref_model(**model_kwargs)\n\n ref_shift_logits = ref_outputs.logits[..., :-1, :].contiguous()\n ref_per_token_logps = selective_log_softmax(ref_shift_logits, shift_labels)\n ref_per_token_logps[shift_completion_mask == 0] = 0.0\n\n if self.ld_alpha is None:\n ref_logps = ref_per_token_logps.sum(dim=1)\n else:\n comp_pos = shift_completion_mask.cumsum(dim=1)\n comp_lens = shift_completion_mask.sum(dim=1).long()\n chosen_lens, rejected_lens = comp_lens.chunk(2, dim=0)\n shared_lens = torch.minimum(chosen_lens, rejected_lens)\n shared_lens = torch.cat([shared_lens, shared_lens], dim=0).to(device)\n shared_mask = (comp_pos > 0) & (comp_pos <= shared_lens.unsqueeze(1))\n tail_mask = comp_pos > shared_lens.unsqueeze(1)\n shared_logps = (ref_per_token_logps * shared_mask).sum(dim=1)\n tail_logps = (ref_per_token_logps * tail_mask).sum(dim=1)\n ref_logps = shared_logps + self.ld_alpha * tail_logps\n\n ref_chosen_logps, ref_rejected_logps = ref_logps.chunk(2, dim=0)\n return ref_chosen_logps, ref_rejected_logps\n\n def _compute_loss_liger(self, model, inputs, return_outputs):\n if return_outputs:\n raise RuntimeError(\n \"return_outputs=True is not supported with the Liger DPO loss. The Liger loss computes the loss \"\n \"without materializing logits, so outputs cannot be returned.\"\n )\n\n mode = \"train\" if self.model.training else \"eval\"\n\n input_ids = inputs[\"input_ids\"]\n attention_mask = inputs[\"attention_mask\"]\n completion_mask = inputs[\"completion_mask\"]\n input_ids, attention_mask, completion_mask = self._truncate_inputs(input_ids, attention_mask, completion_mask)\n\n decoder = model.get_decoder()\n outputs = decoder(input_ids, attention_mask=attention_mask, use_cache=False)\n hidden_states = outputs.last_hidden_state[:, :-1].contiguous()\n lm_head = model.get_output_embeddings()\n weight = lm_head.weight\n bias = lm_head.bias\n\n if is_peft_model(model):\n raise NotImplementedError(\"Liger DPO loss is not implemented for PEFT models.\")\n else:\n with torch.no_grad(), disable_gradient_checkpointing(self.model, self.args.gradient_checkpointing_kwargs):\n ref_decoder = self.ref_model.get_decoder()\n ref_outputs = ref_decoder(input_ids, attention_mask=attention_mask, use_cache=False)\n ref_lm_head = self.ref_model.get_output_embeddings()\n ref_hidden_states = ref_outputs.last_hidden_state[:, :-1].contiguous()\n ref_weight = ref_lm_head.weight\n ref_bias = ref_lm_head.bias\n\n shift_completion_mask = completion_mask[:, 1:].contiguous()\n labels = input_ids[:, 1:].clone()\n labels[shift_completion_mask == 0] = -100\n\n loss, metrics = self.liger_loss_fn(\n weight, hidden_states, labels, bias, ref_hidden_states, ref_weight, ref_bias\n )\n\n (\n chosen_logps,\n rejected_logps,\n chosen_logits_mean,\n rejected_logits_mean,\n nll_loss,\n chosen_rewards,\n rejected_rewards,\n ) = metrics\n\n if mode == \"train\":\n num_tokens_in_batch = self.accelerator.gather_for_metrics(inputs[\"attention_mask\"].sum()).sum().item()\n self._total_train_tokens += num_tokens_in_batch\n self._metrics[mode][\"num_tokens\"] = [self._total_train_tokens]\n\n avg_chosen_logits = self.accelerator.gather_for_metrics(chosen_logits_mean).mean().item()\n avg_rejected_logits = self.accelerator.gather_for_metrics(rejected_logits_mean).mean().item()\n self._metrics[mode][\"logits/chosen\"].append(avg_chosen_logits)\n self._metrics[mode][\"logits/rejected\"].append(avg_rejected_logits)\n\n agg_chosen_rewards = self.accelerator.gather(chosen_rewards)\n agg_rejected_rewards = self.accelerator.gather(rejected_rewards)\n self._metrics[mode][\"rewards/chosen\"].append(agg_chosen_rewards.mean().item())\n self._metrics[mode][\"rewards/rejected\"].append(agg_rejected_rewards.mean().item())\n\n reward_accuracies = (chosen_rewards > rejected_rewards).float()\n agg_reward_accuracies = self.accelerator.gather(reward_accuracies)\n self._metrics[mode][\"rewards/accuracies\"].append(agg_reward_accuracies.mean().item())\n\n margins = chosen_rewards - rejected_rewards\n agg_margins = self.accelerator.gather(margins)\n self._metrics[mode][\"rewards/margins\"].append(agg_margins.mean().item())\n\n self._metrics[mode][\"logps/chosen\"].append(self.accelerator.gather(chosen_logps).mean().item())\n self._metrics[mode][\"logps/rejected\"].append(self.accelerator.gather(rejected_logps).mean().item())\n\n return loss\n\n def _compute_loss(self, model, inputs, return_outputs):\n mode = \"train\" if self.model.training else \"eval\"\n device = self.accelerator.device\n\n input_ids = inputs[\"input_ids\"]\n attention_mask = inputs[\"attention_mask\"]\n completion_mask = inputs[\"completion_mask\"]\n # token_type_ids and mm_token_type_ids are sequence-length-aligned: truncate to match input_ids\n extra_keys = [k for k in (\"token_type_ids\", \"mm_token_type_ids\") if k in inputs]\n input_ids, attention_mask, completion_mask, *extra = self._truncate_inputs(\n input_ids, attention_mask, completion_mask, *[inputs[k] for k in extra_keys]\n )\n\n model_kwargs = {\"input_ids\": input_ids, \"attention_mask\": attention_mask, \"use_cache\": False}\n for key, val in zip(extra_keys, extra, strict=False):\n model_kwargs[key] = val\n for key in (\"pixel_values\", \"pixel_attention_mask\", \"image_grid_thw\", \"image_sizes\"):\n if key in inputs:\n model_kwargs[key] = inputs[key]\n\n outputs = model(**model_kwargs)\n shift_logits = outputs.logits[..., :-1, :].contiguous()\n shift_labels = input_ids[..., 1:].contiguous()\n shift_completion_mask = completion_mask[..., 1:].contiguous()\n per_token_logps = selective_log_softmax(shift_logits, shift_labels)\n per_token_logps[shift_completion_mask == 0] = 0.0 # mask out non-completion tokens\n if self.ld_alpha is None:\n logps = per_token_logps.sum(dim=1) # sum over sequence length\n else:\n comp_pos = shift_completion_mask.cumsum(dim=1)\n comp_lens = shift_completion_mask.sum(dim=1).long()\n chosen_lens, rejected_lens = comp_lens.chunk(2, dim=0)\n shared_lens = torch.minimum(chosen_lens, rejected_lens)\n shared_lens = torch.cat([shared_lens, shared_lens], dim=0).to(device)\n shared_mask = (comp_pos > 0) & (comp_pos <= shared_lens.unsqueeze(1)) # shared: 1 <= pos <= shared_len\n tail_mask = comp_pos > shared_lens.unsqueeze(1) # tail: pos > shared_len\n shared_logps = (per_token_logps * shared_mask).sum(dim=1)\n tail_logps = (per_token_logps * tail_mask).sum(dim=1)\n logps = shared_logps + self.ld_alpha * tail_logps\n chosen_logps, rejected_logps = logps.chunk(2, dim=0) # batch is [chosen, rejected]\n\n if self.precompute_ref_logps:\n ref_chosen_logps, ref_rejected_logps = inputs[\"ref_chosen_logps\"], inputs[\"ref_rejected_logps\"]\n else:\n # When gradient checkpointing is enabled with use_reentrant=True (default), calling the model inside a\n # torch.no_grad() block triggers a harmless PyTorch warning (\"None of the inputs have requires_grad=True\").\n # Temporarily disable checkpointing to avoid this warning during inference.\n with torch.no_grad(), disable_gradient_checkpointing(self.model, self.args.gradient_checkpointing_kwargs):\n if is_peft_model(model) and self.ref_model is None:\n # When training a PEFT adapter, how we obtain the reference depends on the setup:\n # - New adapter: disabling adapters yields the base model.\n # - Re-training an existing adapter: an initial copy is loaded under the name \"ref\".\n model = self.accelerator.unwrap_model(model)\n with use_adapter(model, adapter_name=\"ref\" if \"ref\" in model.peft_config else None):\n ref_outputs = self.model(**model_kwargs)\n else:\n ref_outputs = self.ref_model(**model_kwargs)\n\n ref_shift_logits = ref_outputs.logits[..., :-1, :].contiguous()\n ref_per_token_logps = selective_log_softmax(ref_shift_logits, shift_labels)\n ref_per_token_logps[shift_completion_mask == 0] = 0.0 # mask out non-completion tokens\n if self.ld_alpha is None:\n ref_logps = ref_per_token_logps.sum(dim=1) # sum over sequence length\n else:\n # reuse comp_pos/shared_mask/tail_mask computed above (they depend only on completion_mask)\n ref_shared_logps = (ref_per_token_logps * shared_mask).sum(dim=1)\n ref_tail_logps = (ref_per_token_logps * tail_mask).sum(dim=1)\n ref_logps = ref_shared_logps + self.ld_alpha * ref_tail_logps\n ref_chosen_logps, ref_rejected_logps = ref_logps.chunk(2, dim=0) # batch is [chosen, rejected]\n\n # Get the log ratios for the chosen and rejected responses\n chosen_logratios = chosen_logps - ref_chosen_logps\n rejected_logratios = rejected_logps - ref_rejected_logps\n\n if self.f_divergence_type == \"reverse_kl\": # standard DPO\n chosen_scores = chosen_logratios\n rejected_scores = rejected_logratios\n elif self.f_divergence_type == \"forward_kl\":\n # f'(t) = 1 - 1/t -> drop constant -> -exp(-logratio)\n chosen_scores = -torch.exp(-chosen_logratios)\n rejected_scores = -torch.exp(-rejected_logratios)\n elif self.f_divergence_type == \"js_divergence\":\n # f'(t) = log(2t/(t+1)) -> drop log 2\n chosen_scores = F.logsigmoid(chosen_logratios)\n rejected_scores = F.logsigmoid(rejected_logratios)\n elif self.f_divergence_type == \"alpha_divergence\":\n # alpha-divergence: f'(t) = (t^(ฮฑ-1) - 1)/(ฮฑ-1)\n if abs(self.f_alpha_divergence_coef - 1.0) < 1e-6: # limit case f'(t) -> log(t), fall back to reverse_kl\n chosen_scores = chosen_logratios\n rejected_scores = rejected_logratios\n else:\n coef = 1.0 / (self.f_alpha_divergence_coef - 1.0)\n t_chosen = (self.f_alpha_divergence_coef - 1.0) * chosen_logratios\n t_rejected = (self.f_alpha_divergence_coef - 1.0) * rejected_logratios\n dtype = t_chosen.dtype\n # Clamp max so exp(.) stays representable after casting back\n clamp_max = {torch.float16: 11.0, torch.bfloat16: 80.0, torch.float32: 80.0}[dtype]\n t_chosen_float = torch.clamp(t_chosen.float(), max=clamp_max)\n t_rejected_float = torch.clamp(t_rejected.float(), max=clamp_max)\n chosen_scores = torch.exp(t_chosen_float).to(dtype) * coef\n rejected_scores = torch.exp(t_rejected_float).to(dtype) * coef\n else:\n raise ValueError(f\"Unknown f_divergence_type: {self.f_divergence_type}\")\n\n delta_score = chosen_scores - rejected_scores\n\n loss = 0.0\n for loss_type, loss_weight in zip(self.loss_types, self.loss_weights, strict=True):\n if loss_type == \"sigmoid\":\n per_sequence_loss = -F.logsigmoid(self.beta * delta_score)\n\n elif loss_type == \"hinge\":\n per_sequence_loss = torch.relu(1 - self.beta * delta_score)\n\n elif loss_type == \"ipo\":\n # IPO uses sequence-level log-prob differences; in code these are token-summed over the completion,\n # which makes the squared loss scale with completion length. We therefore normalize by the number of\n # completion tokens (average per token) to make ฮฒ/loss comparable across variable lengths. This length\n # normalization is not explicitly discussed in the IPO paper; we confirmed this choice with the IPO\n # authors, and the results reported in the paper correspond to this normalized form.\n chosen_mask, rejected_mask = completion_mask.chunk(2, dim=0)\n chosen_avg_score = chosen_scores / chosen_mask.sum(dim=1).clamp(min=1.0)\n rejected_avg_score = rejected_scores / rejected_mask.sum(dim=1).clamp(min=1.0)\n ipo_delta = chosen_avg_score - rejected_avg_score\n # (Eq. 17) of the paper where beta is the regularization parameter for the IPO loss, denoted by ฯ„.\n per_sequence_loss = (ipo_delta - 1 / (2 * self.beta)) ** 2\n\n elif loss_type == \"exo_pair\":\n # Implements EXO-pref from the paper https://huggingface.co/papers/2402.00856, (Eq. 16)\n # Minimize KL(p_fฮธ || p_rh) for K=2; p_fฮธ = softmax(ฮฒฯ€ * (log ฯ€ฮธ โˆ’ log ฯ€_ref)) over {chosen, rejected}\n # p_rh = [(1โˆ’ฮต), ฮต]; expanded KL gives the weighted logsigmoid form below\n epsilon = torch.tensor(self.label_smoothing, device=device)\n qw = torch.sigmoid(self.beta * delta_score)\n log_qw = F.logsigmoid(self.beta * delta_score)\n log_pw = torch.log1p(-epsilon)\n ql = torch.sigmoid(-self.beta * delta_score)\n log_ql = F.logsigmoid(-self.beta * delta_score)\n log_pl = torch.log(epsilon)\n per_sequence_loss = qw * (log_qw - log_pw) + ql * (log_ql - log_pl)\n\n elif loss_type == \"nca_pair\":\n chosen_rewards = self.beta * chosen_scores\n rejected_rewards = self.beta * rejected_scores\n per_sequence_loss = (\n -F.logsigmoid(chosen_rewards)\n - 0.5 * F.logsigmoid(-chosen_rewards)\n - 0.5 * F.logsigmoid(-rejected_rewards)\n )\n\n elif loss_type == \"robust\":\n clean_loss_term = -(1 - self.label_smoothing) * F.logsigmoid(self.beta * delta_score)\n flipped_loss_term = -self.label_smoothing * F.logsigmoid(-self.beta * delta_score)\n per_sequence_loss = (clean_loss_term - flipped_loss_term) / (1 - 2 * self.label_smoothing)\n\n elif loss_type == \"bco_pair\":\n chosen_rewards = self.beta * chosen_scores\n rejected_rewards = self.beta * rejected_scores\n per_sequence_loss = -F.logsigmoid(chosen_rewards) - F.logsigmoid(-rejected_rewards)\n\n elif loss_type == \"sppo_hard\":\n # In the paper (https://huggingface.co/papers/2405.00675), SPPO employs a soft probability approach,\n # estimated using the PairRM score. The probability calculation is conducted outside of the trainer\n # class. The version described here is the hard probability version, where P in Equation (4.7) of\n # Algorithm 1 is set to 1 for the winner and 0 for the loser.\n winner_margin_error = (chosen_scores - 0.5 / self.beta) ** 2\n loser_margin_error = (rejected_scores + 0.5 / self.beta) ** 2\n per_sequence_loss = winner_margin_error + loser_margin_error\n\n elif loss_type == \"aot\":\n logratios = chosen_logps - rejected_logps\n ref_logratios = ref_chosen_logps - ref_rejected_logps\n logratios_sorted, _ = torch.sort(logratios, dim=0)\n ref_logratios_sorted, _ = torch.sort(ref_logratios, dim=0)\n delta = logratios_sorted - ref_logratios_sorted\n per_sequence_loss = (\n -F.logsigmoid(self.beta * delta) * (1 - self.label_smoothing)\n - F.logsigmoid(-self.beta * delta) * self.label_smoothing\n )\n\n elif loss_type == \"aot_unpaired\":\n chosen_logratios_sorted, _ = torch.sort(chosen_logratios, dim=0)\n rejected_logratios_sorted, _ = torch.sort(rejected_logratios, dim=0)\n delta = chosen_logratios_sorted - rejected_logratios_sorted\n per_sequence_loss = (\n -F.logsigmoid(self.beta * delta) * (1 - self.label_smoothing)\n - F.logsigmoid(-self.beta * delta) * self.label_smoothing\n )\n\n elif loss_type == \"apo_zero\":\n # Eqn (7) of the APO paper (https://huggingface.co/papers/2408.06266)\n # Use this loss when you believe the chosen outputs are better than your model's default output\n # Increase chosen likelihood and decrease rejected likelihood\n losses_chosen = 1 - torch.sigmoid(self.beta * chosen_logratios)\n losses_rejected = torch.sigmoid(self.beta * rejected_logratios)\n per_sequence_loss = losses_chosen + losses_rejected\n\n elif loss_type == \"apo_down\":\n # Eqn (8) of the APO paper (https://huggingface.co/papers/2408.06266)\n # Use this loss when you believe the chosen outputs are worse than your model's default output.\n # Decrease chosen likelihood and decrease rejected likelihood more\n losses_chosen = torch.sigmoid(self.beta * chosen_logratios)\n losses_rejected = 1 - torch.sigmoid(self.beta * delta_score)\n per_sequence_loss = losses_chosen + losses_rejected\n\n elif loss_type == \"discopop\":\n # Eqn (5) of the DiscoPOP paper (https://huggingface.co/papers/2406.08414)\n logits = delta_score * self.beta\n # Modulate the mixing coefficient based on the log ratio magnitudes\n log_ratio_modulation = torch.sigmoid(logits / self.args.discopop_tau)\n logistic_component = -F.logsigmoid(logits)\n exp_component = torch.exp(-logits)\n # Blend between logistic and exponential component based on log ratio modulation\n per_sequence_loss = (\n logistic_component * (1 - log_ratio_modulation) + exp_component * log_ratio_modulation\n )\n\n elif loss_type == \"sft\":\n chosen_logits, _ = shift_logits.chunk(2, dim=0)\n chosen_labels, _ = shift_labels.chunk(2, dim=0)\n chosen_mask, _ = shift_completion_mask.chunk(2, dim=0)\n batch_loss = F.cross_entropy(chosen_logits[chosen_mask.bool()], chosen_labels[chosen_mask.bool()])\n # Implementation convenience: expand the scalar SFT loss to a per-sequence tensor so it matches the\n # shape of other losses; only the mean is used, so this is a no-op numerically.\n per_sequence_loss = batch_loss.expand(chosen_logits.size(0))\n\n else:\n raise ValueError(\n f\"Unknown loss type: {loss_type}. Should be one of ['sigmoid', 'hinge', 'ipo', 'exo_pair', \"\n \"'nca_pair', 'robust', 'bco_pair', 'sppo_hard', 'aot', 'aot_unpaired', 'apo_zero', 'apo_down', \"\n \"'discopop', 'sft']\"\n )\n\n if self.use_weighting:\n # Eq (2) of the WPO paper: https://huggingface.co/papers/2406.11827\n completion_lengths = shift_completion_mask.sum(dim=1).clamp_min(1)\n with torch.no_grad():\n lse1 = torch.logsumexp(shift_logits, dim=-1)\n lse2 = torch.logsumexp(2.0 * shift_logits, dim=-1)\n log_denom = lse2 - 2.0 * lse1\n aligned_logps = (per_token_logps - log_denom) * shift_completion_mask\n mean_logps = aligned_logps.sum(dim=1) / completion_lengths\n weights = torch.exp(mean_logps)\n chosen_weights, rejected_weights = weights.chunk(2, dim=0)\n per_sequence_loss *= chosen_weights * rejected_weights\n\n loss += per_sequence_loss.mean() * loss_weight\n\n # Log the metrics\n # Entropy\n per_token_entropy = entropy_from_logits(shift_logits.detach())\n entropy = per_token_entropy[shift_completion_mask.bool()].mean()\n entropy = self.accelerator.gather_for_metrics(entropy).mean().item()\n self._metrics[mode][\"entropy\"].append(entropy)\n\n # Number of tokens\n if mode == \"train\":\n num_tokens_in_batch = self.accelerator.gather_for_metrics(inputs[\"attention_mask\"].sum()).sum().item()\n self._total_train_tokens += num_tokens_in_batch\n self._metrics[mode][\"num_tokens\"] = [self._total_train_tokens]\n\n # Average logits for chosen and rejected completions\n chosen_logits, rejected_logits = shift_logits.detach().chunk(2, dim=0)\n chosen_mask, rejected_mask = shift_completion_mask.chunk(2, dim=0)\n total_chosen_logits = chosen_logits[chosen_mask.bool()].mean(-1).sum()\n total_chosen_tokens = chosen_mask.sum()\n total_rejected_logits = rejected_logits[rejected_mask.bool()].mean(-1).sum()\n total_rejected_tokens = rejected_mask.sum()\n total_chosen_logits = self.accelerator.gather_for_metrics(total_chosen_logits).sum().item()\n total_chosen_tokens = self.accelerator.gather_for_metrics(total_chosen_tokens).sum().item()\n total_rejected_logits = self.accelerator.gather_for_metrics(total_rejected_logits).sum().item()\n total_rejected_tokens = self.accelerator.gather_for_metrics(total_rejected_tokens).sum().item()\n avg_chosen_logits = total_chosen_logits / total_chosen_tokens if total_chosen_tokens > 0 else 0.0\n avg_rejected_logits = total_rejected_logits / total_rejected_tokens if total_rejected_tokens > 0 else 0.0\n self._metrics[mode][\"logits/chosen\"].append(avg_chosen_logits)\n self._metrics[mode][\"logits/rejected\"].append(avg_rejected_logits)\n\n # Token accuracy for the chosen completions\n predictions = chosen_logits.argmax(dim=-1)\n chosen_mask = shift_completion_mask[: len(shift_completion_mask) // 2].bool()\n chosen_labels = shift_labels[: len(shift_labels) // 2]\n correct_predictions = (predictions == chosen_labels) & chosen_mask\n total_tokens = chosen_mask.sum()\n correct_tokens = correct_predictions.sum()\n correct_tokens = self.accelerator.gather_for_metrics(correct_tokens)\n total_tokens = self.accelerator.gather_for_metrics(total_tokens)\n total_sum = total_tokens.sum()\n accuracy = (correct_tokens.sum() / total_sum).item() if total_sum > 0 else 0.0\n self._metrics[mode][\"mean_token_accuracy\"].append(accuracy)\n\n # Rewards for chosen and rejected completions\n chosen_rewards = self.beta * chosen_logratios.detach()\n rejected_rewards = self.beta * rejected_logratios.detach()\n agg_chosen_rewards = self.accelerator.gather(chosen_rewards)\n agg_rejected_rewards = self.accelerator.gather(rejected_rewards)\n self._metrics[mode][\"rewards/chosen\"].append(agg_chosen_rewards.mean().item())\n self._metrics[mode][\"rewards/rejected\"].append(agg_rejected_rewards.mean().item())\n\n # Reward accuracy\n reward_accuracies = (chosen_rewards > rejected_rewards).float()\n agg_reward_accuracies = self.accelerator.gather(reward_accuracies)\n self._metrics[mode][\"rewards/accuracies\"].append(agg_reward_accuracies.mean().item())\n\n # Reward margins\n margins = chosen_rewards - rejected_rewards\n agg_margins = self.accelerator.gather(margins)\n self._metrics[mode][\"rewards/margins\"].append(agg_margins.mean().item())\n\n # Average log probabilities for chosen and rejected completions\n self._metrics[mode][\"logps/chosen\"].append(self.accelerator.gather(chosen_logps).mean().item())\n self._metrics[mode][\"logps/rejected\"].append(self.accelerator.gather(rejected_logps).mean().item())\n\n return (loss, outputs) if return_outputs else loss\n\n def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):\n if self.use_liger_kernel:\n return self._compute_loss_liger(model, inputs, return_outputs)\n else:\n return self._compute_loss(model, inputs, return_outputs)\n\n # Override training step to add activation offloading context.\n def training_step(self, *args, **kwargs):\n with self.maybe_activation_offload_context:\n return super().training_step(*args, **kwargs)\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n mode = \"train\" if self.model.training else \"eval\"\n metrics = {key: sum(val) / len(val) for key, val in self._metrics[mode].items()} # average the metrics\n\n # This method can be called both in training and evaluation. When called in evaluation, the keys in `logs`\n # start with \"eval_\". We need to add the prefix \"eval_\" to the keys in `metrics` to match the format.\n if mode == \"eval\":\n metrics = {f\"eval_{key}\": val for key, val in metrics.items()}\n\n logs = {**logs, **metrics}\n super().log(logs, start_time)\n self._metrics[mode].clear()\n\n # During eval, Trainer calls prediction_step. If no labels are present in the inputs, it only runs forward and\n # returns logits. We override prediction_step to force compute_loss, because this trainer doesn't involve labels.\n def prediction_step(self, model, inputs, prediction_loss_only, ignore_keys: list[str] | None = None):\n inputs = self._prepare_inputs(inputs)\n with torch.no_grad(), self.compute_loss_context_manager():\n if prediction_loss_only:\n loss = self.compute_loss(model, inputs, return_outputs=False) # logits aren't materialized with liger\n logits, labels = None, None\n else:\n loss, outputs = self.compute_loss(model, inputs, return_outputs=True)\n logits, labels = outputs.logits, inputs[\"input_ids\"]\n return loss, logits, labels\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/trainer/grpo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom .base_config import _BaseConfig\n\n\n@dataclass\nclass GRPOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`GRPOTrainer`].\n\n This class includes only the parameters that are specific to GRPO training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n > Parameters that control the model and reference model\n\n model_init_kwargs (`str`, `dict[str, Any]`, *optional*):\n Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model`\n argument of the [`GRPOTrainer`] is provided as a string.\n disable_dropout (`bool`, *optional*, defaults to `False`):\n Whether to disable dropout in the model. This is useful for training with a reference model, as it prevents\n the model from generating different logprobs for the same input.\n cast_lm_head_to_fp32 (`bool`, *optional*, defaults to `False`):\n Whether to cast the language modeling head of the policy and reference models to float32. As recommended by\n the [ScaleRL](https://huggingface.co/papers/2510.13786) recipe. This flag is only supported when the model\n has untied word embedding and language modeling head layers i.e. `tie_word_embeddings` in the model config\n is False.\n\n > Parameters that control the data preprocessing\n\n remove_unused_columns (`bool`, *optional*, defaults to `False`):\n Whether to only keep the column `\"prompt\"` in the dataset. If you use a custom reward function that\n requires any column other than `\"prompts\"` and `\"completions\"`, you should keep this to `False`.\n num_generations (`int`, *optional*, defaults to `8`):\n Number of generations per prompt to sample. The effective batch size (num_processes * per_device_batch_size\n * gradient_accumulation_steps) must be evenly divisible by this value.\n num_generations_eval (`int` or `None`, *optional*):\n Number of generations to sample during evaluation. This allows using fewer generations during evaluation to\n save computation. If `None`, uses the value of `num_generations`.\n max_completion_length (`int` or `None`, *optional*, defaults to `256`):\n Maximum length of the generated completion.\n ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):\n This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,\n improving generation speed. However, disabling this option allows training models that exceed the VRAM\n capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible\n with vLLM generation.\n shuffle_dataset (`bool`, *optional*, defaults to `True`):\n Whether to shuffle the training dataset.\n pad_to_multiple_of (`int`, *optional*):\n If set, the prompts ids and completions ids will be padded to a multiple of this value.\n\n > Parameters that control generation\n\n generation_batch_size: (`int`, *optional*):\n Batch size to use for generation. If `None`, it defaults to the effective training batch size:\n `per_device_train_batch_size * num_processes * steps_per_generation`. In other words, there is one\n generation batch processed per optimization step. Mutually exclusive with `steps_per_generation`.\n steps_per_generation: (`int`, *optional*):\n Number of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`. Mutually exclusive\n with `generation_batch_size`.\n temperature (`float`, defaults to `1.0`):\n Temperature for sampling. The higher the temperature, the more random the completions.\n top_p (`float`, *optional*, defaults to `1.0`):\n Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to\n `1.0` to consider all tokens.\n top_k (`int`, *optional*, defaults to `0`):\n Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, top-k-filtering is\n disabled and all tokens are considered.\n min_p (`float`, *optional*):\n Minimum token probability, which will be scaled by the probability of the most likely token. It must be a\n value between `0.0` and `1.0`. Typical values are in the `0.01-0.2` range.\n generation_kwargs (`dict[str, Any]`, *optional*):\n Additional keyword arguments to pass to [`~transformers.GenerationConfig`] (if using transformers) or\n `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the\n generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that conflict\n with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.\n chat_template_kwargs (`dict[str, Any]`, *optional*):\n Additional keyword arguments to pass to the `apply_chat_template` function when generating completions.\n repetition_penalty (`float`, *optional*, defaults to `1.0`):\n Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far.\n Values > `1.0` encourage the model to use new tokens, while values < `1.0` encourage the model to repeat\n tokens.\n use_transformers_paged (`bool`, *optional*, defaults to `False`):\n Whether to use the `transformers` paged implementation for generation. If set to `True`, the `transformers`\n paged implementation will be used for generation instead of the default padded implementation. This\n parameter is only effective when `use_vllm` is set to `False`.\n cache_implementation (`str`, *optional*):\n Implementation of the cache method for faster generation when `use_vllm` is set to `False`.\n\n > Parameters that control generation acceleration powered by vLLM\n\n use_vllm (`bool`, *optional*, defaults to `False`):\n Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation\n instead of the default model.generate(). Requires `vllm` to be installed.\n vllm_mode (`str`, *optional*, defaults to `\"colocate\"`):\n Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `\"server\"` or\n `\"colocate\"`.\n\n - `\"server\"`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM\n server is running (start with `trl vllm-serve`).\n - `\"colocate\"`: vLLM will run in the same process and share the training GPUs. This avoids the need for a\n separate server but may cause resource contention with training.\n vllm_model_impl (`str`, *optional*, defaults to `\"vllm\"`):\n Model implementation to use for vLLM. Must be one of `\"transformers\"` or `\"vllm\"`. `\"transformers\"`: Use\n the `transformers` backend for model implementation. `\"vllm\"`: Use the `vllm` library for model\n implementation.\n vllm_structured_outputs_regex (`str`, *optional*):\n Regex for vLLM structured outputs. If `None` (default), structured outputs is disabled.\n\n > Parameters that control the vLLM server (only used when `vllm_mode` is `\"server\"`)\n\n vllm_server_base_url (`str`, *optional*):\n Base URL for the vLLM server (e.g., `\"http://localhost:8000\"`). If provided, `vllm_server_host` and\n `vllm_server_port` are ignored.\n vllm_server_host (`str`, *optional*, defaults to `\"0.0.0.0\"`):\n Host of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided.\n vllm_server_port (`int`, *optional*, defaults to `8000`):\n Port of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided.\n vllm_server_timeout (`float`, *optional*, defaults to `240.0`):\n Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the\n timeout, a `ConnectionError` is raised.\n vllm_group_port (`int`, *optional*, defaults to `51216`):\n Port number for the weight update group. This is used to communicate with the vLLM server. Unless the port\n is occupied, there is no need to change it.\n\n > Parameters that control colocated vLLM execution (only used when `vllm_mode` is `\"colocate\"`)\n\n vllm_gpu_memory_utilization (`float`, *optional*, defaults to `0.3`):\n Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to\n `\"colocate\"`. If you are using `vllm_mode=\"server\"`, this parameter must be passed separately when\n launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.\n vllm_max_model_length (`int`, *optional*):\n Context window for vLLM. Set it to at least the maximum prompt length in the dataset plus\n `max_completion_length`; if omitted, it is inferred from the model config.\n vllm_tensor_parallel_size (`int`, *optional*, defaults to `1`):\n Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to\n `\"colocate\"`. If you are using `vllm_mode=\"server\"`, this parameter must be passed separately when\n launching the vLLM server via the `--vllm_tensor_parallel_size` flag.\n vllm_enable_sleep_mode (`bool`, *optional*, defaults to `False`):\n Enable vLLM sleep mode to offload weights/cache during the optimizer step. Keeps GPU memory usage low, but\n waking the engine adds hostโ€“device transfer latency.\n\n > Parameters that control the training\n\n beta (`float`, *optional*, defaults to `0.0`):\n KL coefficient. If `0.0` (default), the reference model is not loaded, reducing memory usage and improving\n training speed. [DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement\n learning](https://huggingface.co/papers/2501.12948) use a value of `0.001`.\n num_iterations (`int`, *optional*, defaults to `1`):\n Number of iterations per batch (denoted as ฮผ in the algorithm).\n epsilon (`float`, *optional*, defaults to `0.2`):\n Epsilon value for clipping.\n delta (`float`, *optional*):\n Enables the upper clipping bound in two-sided GRPO loss when set to a float. If `None` (default), standard\n GRPO clipping is used. Recommended to be greater than `1 + ฮต` when enabled. This method is introduced in\n the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291).\n epsilon_high (`float`, *optional*):\n Upper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound\n specified in argument `epsilon`. Paper [DAPO](https://huggingface.co/papers/2503.14476) recommends `0.28`.\n When used with `loss_type='cispo'`, this corresponds to the ฮต_max param specified in the [ScaleRL\n paper](https://huggingface.co/papers/2510.13786) and the recommended value is `5.0`.\n sapo_temperature_neg (`float`, *optional*, defaults to `1.05`):\n Temperature for tokens with non-positive advantage scores used in the `sapo` loss function. This parameter\n is introduced in the [Soft Adaptive Policy Optimization paper](https://huggingface.co/papers/2511.20347).\n sapo_temperature_pos (`float`, *optional*, defaults to `1.0`):\n Temperature for tokens with positive advantage scores used in the `sapo` loss function. This parameter is\n introduced in the [Soft Adaptive Policy Optimization paper](https://huggingface.co/papers/2511.20347).\n vespo_k_pos (`float`, *optional*, defaults to `2.0`):\n k parameter for positive advantages, it is the power exponent in the VESPO loss. Controls how aggressively\n we down-weight samples with low importance weights (when the importance sampling ratio < 1).\n vespo_lambda_pos (`float`, *optional*, defaults to `3.0`):\n lambda parameter for positive advantages, it is the decay factor in the VESPO loss. Controls how\n aggressively we down-weight samples with high importance weights (when the importance sampling ratio > 1).\n vespo_k_neg (`float`, *optional*, defaults to `3.0`):\n k parameter for negative advantages, it is the power exponent in the VESPO loss. Controls how aggressively\n we down-weight samples with low importance weights (when the importance sampling ratio < 1).\n vespo_lambda_neg (`float`, *optional*, defaults to `2.0`):\n lambda parameter for negative advantages, it is the exponential decay factor in the VESPO loss. Controls\n how aggressively we down-weight samples with high importance weights (when the importance sampling ratio >\n 1).\n importance_sampling_level (`str`, *optional*, defaults to `\"token\"`):\n Controls whether importance sampling ratios are computed at the `\"token\"` or `\"sequence\"` level. `\"token\"`\n keeps the raw per-token log-probability ratios (one weight per token). `\"sequence\"` averages the\n log-probability ratios across valid tokens to produce a single ratio per sequence. The [GSPO\n paper](https://huggingface.co/papers/2507.18071) shows that sequence-level sampling often yields more\n stable training and better alignment with sequence-level rewards.\n reward_weights (`list[float]`, *optional*):\n Weights for each reward function. Must match the number of reward functions. If `None`, all rewards are\n weighted equally with weight `1.0`.\n multi_objective_aggregation (`str`, *optional*, defaults to `\"sum_then_normalize\"`):\n Method to aggregate multiple reward functions. Supported values are:\n\n - `\"sum_then_normalize\"` (default): First sums the weighted rewards from each reward function, then applies\n reward scaling/normalization as specified by `scale_rewards` (see `scale_rewards` for details).\n - `\"normalize_then_sum\"`: First normalizes/scales each reward function across generations (within each\n group), then sums the normalized rewards using the specified weights. The aggregated reward is then\n normalized at the batch level when forming advantages. This is the suggested approach from the paper\n [GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL\n Optimization](https://huggingface.co/papers/2601.05242).\n scale_rewards (`str` or `bool`, *optional*, defaults to `\"group\"`):\n Specifies the scaling strategy for rewards. Supported values are:\n\n - `True` or `\"group\"` (default): rewards are scaled by the standard deviation within each group, ensuring\n unit variance within a group.\n - `\"batch\"`: rewards are scaled by the standard deviation across the entire batch, as recommended in the\n [PPO Lite paper](https://huggingface.co/papers/2508.08221).\n - `False` or `\"none\"`: no scaling is applied. The [Dr. GRPO\n paper](https://huggingface.co/papers/2503.20783) recommends not scaling rewards, as scaling by the\n standard deviation introduces a question-level difficulty bias.\n loss_type (`str`, *optional*, defaults to `\"dapo\"`):\n Specifies the loss formulation to use. Supported values are:\n\n - `\"grpo\"`: Aggregates token-level losses by normalizing over sequence length. Not recommended due to\n length biasโ€”this approach tends to prefer shorter completions with positive advantages and longer ones\n with negative advantages.\n - `\"dr_grpo\"`: Aggregates token-level losses by normalizing with a global constant. This method was\n introduced in the [Dr. GRPO paper](https://huggingface.co/papers/2503.20783) to eliminate length bias.\n The value of the constant corresponds to `max_completion_length`.\n - `\"dapo\"` (default): Aggregates token-level losses by normalizing with the number of active token in the\n global accumulated batch. This method was introduced in the [DAPO\n paper](https://huggingface.co/papers/2503.14476) to eliminate length bias.\n - `\"bnpo\"`: Aggregates token-level losses by normalizing with the number of active token in the local\n batch. Note that normalization is performed over the local batch only, so results may slightly vary\n depending on the local batch size, despite a constant effective batch size. When using\n `per_device_train_batch_size==1`, the loss is equivalent to the GRPO loss.\n - `\"cispo\"`: Clips the importance sampling weights instead of the advantage scaled importance weights. The\n clipped weights are then multiplied with the advantages and policy model's log probs. Individual token\n losses are aggregated by normalizing with the number of active tokens in the global accumulated batch.\n This method was introduced in the [MiniMax-M1 paper](https://huggingface.co/papers/2506.13585).\n - `\"sapo\"`: Soft Adaptive Policy Optimization loss, as introduced in the [Soft Adaptive Policy Optimization\n paper](https://huggingface.co/papers/2511.20347). Replaces hard clipping with a smooth,\n temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful\n learning signals.\n - `\"luspo\"`: Length-Unbiased Sequence Policy Optimization loss. A sequence-level loss that scales each\n sequence's loss by its length. This is a modification of GSPO and requires\n `importance_sampling_level=\"sequence\"`. Introduced in the [LUSPO\n paper](https://huggingface.co/papers/2602.05261).\n - `\"vespo\"`: Variational Sequence-Level Soft Policy Optimization. Replaces hard clipping with a smooth,\n asymmetric Gamma weighting function applied directly to sequence-level importance weights. Introduced in\n the [VESPO paper](https://huggingface.co/papers/2602.10693).\n mask_truncated_completions (`bool`, *optional*, defaults to `False`):\n When enabled, truncated completions are excluded from the loss calculation, preventing them from being\n incorrectly penalized and introducing noise during training. According to the\n [DAPO](https://huggingface.co/papers/2503.14476) paper, this is a good practice for training stability.\n sync_ref_model (`bool`, *optional*, defaults to `False`):\n Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using\n the `ref_model_mixup_alpha` parameter. This synchronization originates from the\n [TR-DPO](https://huggingface.co/papers/2404.09656) paper.\n ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`):\n ฮฑ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix\n between the current policy and the previous reference policy during updates. The reference policy is\n updated according to the equation: `ฯ€_ref = ฮฑ * ฯ€_ฮธ + (1 - ฮฑ) * ฯ€_ref_prev`. To use this parameter, you\n must set `sync_ref_model=True`.\n ref_model_sync_steps (`int`, *optional*, defaults to `512`):\n ฯ„ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how\n frequently the current policy is synchronized with the reference policy. To use this parameter, you must\n set `sync_ref_model=True`.\n top_entropy_quantile (`float`, *optional*, defaults to `1.0`):\n ฯ parameter from [Beyond the 80/20 Rule](https://huggingface.co/papers/2506.01939). Keeps in the policy\n loss term only the top-ฯ quantile of tokens by entropy of the probability distribution at each sequence\n position, improving results. Range: `[0.0-1.0]`. A value of `0.0` masks all but the highest entropy token;\n `1.0` keeps all tokens. The paper recommends a value of `0.2`. If used with\n `mask_truncated_completions=True`, only tokens from non-truncated completions are considered.\n max_tool_calling_iterations (`int`, *optional*):\n Maximum number of tool-calling turns when training an agent. If `None`, there is no limit and generation\n stops when the model generates a response turn with no tool calls or when the total response length reaches\n `max_model_length`.\n vllm_importance_sampling_correction (`bool`, *optional*, defaults to `True`):\n Whether to apply Importance Sampling (IS) to correct for the mismatch between vLLM completion logprobs and\n recomputed training logprobs. If set to `False`, no IS is applied regardless of\n `vllm_importance_sampling_mode`. When `True`, the selected mode determines how the IS ratios are computed\n and constrained.\n vllm_importance_sampling_mode (`str`, *optional*, defaults to `\"sequence_mask\"`):\n Specifies how Importance Sampling is performed when `vllm_importance_sampling_correction=True`. Possible\n values are:\n\n - `\"token_truncate\"`: Token-level truncated IS (default). Per-token ratios are clipped from above at C.\n - `\"token_mask\"`: Token-level masked IS. Per-token ratios above C are set to zero.\n - `\"sequence_truncate\"`: Sequence-level truncated IS. A single sequence ratio is clipped from above at\n C and applied to all tokens in the sequence.\n - `\"sequence_mask\"`: Sequence-level masked IS. Sequences with ratios above C are masked out.\n vllm_importance_sampling_cap (`float`, *optional*, defaults to `3.0`):\n Importance sampling cap C used by `vllm_importance_sampling_mode`. For `*_truncate` modes, importance\n ratios are clipped from above at C. For `*_mask` modes, ratios larger than C are set to zero.\n off_policy_mask_threshold (`float`, *optional*):\n Threshold for off-policy sequence masking. If `None`, off-policy sequence masking is disabled. When set,\n sequences with negative advantages and high KL divergence are masked out to stabilize training. This\n parameter corresponds to the `delta` threshold in Equation 9 of the [DeepSeek-V3.2\n paper](https://huggingface.co/papers/2512.02556). It expects a positive value (e.g., 0.5).\n use_bias_correction_kl (`bool`, *optional*, defaults to `False`):\n Whether to use the unbiased KL divergence estimator with importance sampling correction. This corrects the\n KL divergence estimate by multiplying it with the importance sampling ratio. This is described in the\n [DeepSeek-V3.2 paper](https://huggingface.co/papers/2512.02556).\n\n > Parameters that control the logging\n\n log_completions (`bool`, *optional*, defaults to `False`):\n Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed,\n it prints the sample. If `wandb` and/or `trackio` logging is enabled, it logs it to `wandb` and/or\n `trackio`.\n num_completions_to_print (`int`, *optional*):\n Number of completions to print with `rich`. If `None`, all completions are logged.\n log_unique_prompts (`bool`, *optional*, defaults to `False`):\n Whether to log unique prompts. If `True`, only unique prompts are logged. If `False`, all prompts are\n logged.\n log_completions_hub_repo (`str`, *optional*):\n Hugging Face Hub repository to save the completions. Should be a complete repository name like\n `'username/reponame'` or `'orgname/reponame'`, or just `'reponame'` in which case the repository will be\n created in the currently-logged-in Hugging Face user's namespace. Note that this repository will be public\n unless you set `hub_private_repo=True` or your organization's default is to create private repositories.\"\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `1e-6` instead of `5e-5`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-6,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n # Parameters that control the model and reference model\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments for `transformers.AutoModelForCausalLM.from_pretrained`, used when the `model` \"\n \"argument of the `GRPOTrainer` is provided as a string.\"\n },\n )\n disable_dropout: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to disable dropout in the model. This is useful for training with a reference model, as \"\n \"it prevents the model from generating different logprobs for the same input.\"\n },\n )\n cast_lm_head_to_fp32: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to cast the language modeling head of the policy and reference, models to float32.\"\n \"As recommended by the [ScaleRL](https://huggingface.co/papers/2510.13786) recipe. This flag is only \"\n \"supported when the model has untied word embedding and language modeling head layers i.e. \"\n \"`tie_word_embeddings` in the model config is False.\"\n },\n )\n\n # Parameters that control the data preprocessing\n # The default value remove_unused_columns is overwritten from the parent class, because in GRPO we usually rely on\n # additional columns to compute the reward\n remove_unused_columns: bool | None = field(\n default=False,\n metadata={\n \"help\": \"Whether to only keep the column 'prompt' in the dataset. If you use a custom reward function \"\n \"that requires any column other than 'prompts' and 'completions', you should keep this to `False`.\"\n },\n )\n num_generations: int | None = field(\n default=8,\n metadata={\n \"help\": \"Number of generations to sample. The effective batch size (num_processes * per_device_batch_size \"\n \"* gradient_accumulation_steps) must be evenly divisible by this value.\"\n },\n )\n num_generations_eval: int | None = field(\n default=None,\n metadata={\n \"help\": \"Number of generations to sample during evaluation. This allows using fewer generations during \"\n \"evaluation to save computation. If `None`, uses the value of `num_generations`.\"\n },\n )\n max_completion_length: int | None = field(\n default=256,\n metadata={\"help\": \"Maximum length of the generated completion.\"},\n )\n ds3_gather_for_generation: bool = field(\n default=True,\n metadata={\n \"help\": \"This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for \"\n \"generation, improving generation speed. However, disabling this option allows training models that \"\n \"exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option \"\n \"is not compatible with vLLM generation.\"\n },\n )\n shuffle_dataset: bool | None = field(\n default=True,\n metadata={\"help\": \"Whether to shuffle the training dataset.\"},\n )\n pad_to_multiple_of: int | None = field(\n default=None,\n metadata={\"help\": \"If set, the prompts ids and completions ids will be padded to a multiple of this value.\"},\n )\n\n # Parameters that control generation\n generation_batch_size: int | None = field(\n default=None,\n metadata={\n \"help\": \"Batch size to use for generation. If `None`, it defaults to the effective training batch size: \"\n \"`per_device_train_batch_size * num_processes * steps_per_generation`.\"\n },\n )\n steps_per_generation: int | None = field(\n default=None,\n metadata={\"help\": \"Number of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`.\"},\n )\n temperature: float = field(\n default=1.0,\n metadata={\"help\": \"Temperature for sampling. The higher the temperature, the more random the completions.\"},\n )\n top_p: float = field(\n default=1.0,\n metadata={\n \"help\": \"Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. \"\n \"Set to 1.0 to consider all tokens.\"\n },\n )\n top_k: int = field(\n default=0,\n metadata={\n \"help\": \"Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, \"\n \"top-k-filtering is disabled and all tokens are considered.\"\n },\n )\n min_p: float | None = field(\n default=None,\n metadata={\n \"help\": \"Minimum token probability, which will be scaled by the probability of the most likely token. It \"\n \"must be a value between 0.0 and 1.0. Typical values are in the 0.01-0.2 range.\"\n },\n )\n generation_kwargs: dict | None = field(\n default=None,\n metadata={\n \"help\": \"Additional keyword arguments to pass to `GenerationConfig` (if using transformers) or \"\n \"`SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the \"\n \"generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that \"\n \"conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.\"\n },\n )\n chat_template_kwargs: dict | None = field(\n default=None,\n metadata={\n \"help\": \"Additional keyword arguments to pass to the `apply_chat_template` function when generating \"\n \"completions.\"\n },\n )\n repetition_penalty: float = field(\n default=1.0,\n metadata={\n \"help\": \"Float that penalizes new tokens based on whether they appear in the prompt and the generated \"\n \"text so far. Values > 1.0 encourage the model to use new tokens, while values < 1.0 encourage the model \"\n \"to repeat tokens.\"\n },\n )\n use_transformers_paged: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use the `transformers` paged implementation for generation. If set to `True`, the \"\n \"`transformers` paged implementation will be used for generation instead of the default padded \"\n \"implementation. This parameter is only effective when `use_vllm` is set to `False`.\"\n },\n )\n cache_implementation: str | None = field(\n default=None,\n metadata={\"help\": \"Implementation of the cache method for faster generation when use_vllm is set to False.\"},\n )\n\n # Parameters that control generation acceleration powered by vLLM\n use_vllm: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for \"\n \"generation instead of the default model.generate(). Requires `vllm` to be installed.\"\n },\n )\n vllm_mode: str = field(\n default=\"colocate\",\n metadata={\n \"help\": \"Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `'server'` or \"\n \"`'colocate'`. `'server'`: The trainer will send generation requests to a separate vLLM server. Make sure \"\n \"a TRL vLLM server is running (start with `trl vllm-serve`). `'colocate'`: vLLM will run in the same \"\n \"process and share the training GPUs. This avoids the need for a separate server but may cause resource \"\n \"contention with training.\"\n },\n )\n vllm_model_impl: str = field(\n default=\"vllm\",\n metadata={\n \"help\": \"Model implementation to use for vLLM. Must be one of `transformers` or `vllm`. `transformers`: \"\n \"Use the `transformers` backend for model implementation. `vllm`: Use the `vllm` library for \"\n \"model implementation.\"\n },\n )\n vllm_enable_sleep_mode: bool = field(\n default=False,\n metadata={\n \"help\": \"Enable vLLM sleep mode to offload weights/cache during the optimizer step. Keeps GPU memory \"\n \"usage low, but waking the engine adds hostโ€“device transfer latency.\"\n },\n )\n vllm_structured_outputs_regex: str | None = field(\n default=None,\n metadata={\"help\": \"Regex for vLLM structured outputs. If `None` (default), structured outputs is disabled.\"},\n )\n\n # Parameters that control the vLLM server (only used when `vllm_mode` is `\"server\"`)\n vllm_server_base_url: str | None = field(\n default=None,\n metadata={\n \"help\": \"Base URL for the vLLM server (e.g., 'http://localhost:8000'). If provided, `vllm_server_host` \"\n \"and `vllm_server_port` are ignored.\"\n },\n )\n vllm_server_host: str = field(\n default=\"0.0.0.0\",\n metadata={\"help\": \"Host of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.\"},\n )\n vllm_server_port: int = field(\n default=8000,\n metadata={\"help\": \"Port of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.\"},\n )\n vllm_server_timeout: float = field(\n default=240.0,\n metadata={\n \"help\": \"Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up \"\n \"after the timeout, a `ConnectionError` is raised.\"\n },\n )\n vllm_group_port: int = field(\n default=51216,\n metadata={\n \"help\": \"Port number for the weight update group. This is used to communicate with the vLLM server. \"\n \"Unless the port is occupied, there is no need to change it.\",\n },\n )\n\n # Parameters that control colocated vLLM execution (only used when `vllm_mode` is `\"colocate\"`)\n vllm_gpu_memory_utilization: float = field(\n default=0.3,\n metadata={\n \"help\": \"Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set \"\n \"to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when \"\n \"launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.\"\n },\n )\n vllm_max_model_length: int | None = field(\n default=None,\n metadata={\n \"help\": \"Context window for vLLM. Set it to at least the maximum prompt length in the dataset plus \"\n \"`max_completion_length`; if omitted, it is inferred from the model config.\"\n },\n )\n vllm_tensor_parallel_size: int = field(\n default=1,\n metadata={\n \"help\": \"Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set \"\n \"to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when \"\n \"launching the vLLM server via the `--vllm_tensor_parallel_size` flag.\"\n },\n )\n\n # Parameters that control the training\n beta: float = field(\n default=0.0,\n metadata={\n \"help\": \"KL coefficient. If `0.0` (default), the reference model is not loaded, reducing memory usage and \"\n \"improving training speed. [DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement \"\n \"learning](https://huggingface.co/papers/2501.12948) use a value of `0.001`.\"\n },\n )\n num_iterations: int = field(\n default=1,\n metadata={\"help\": \"Number of iterations per batch (denoted as ฮผ in the algorithm).\"},\n )\n epsilon: float = field(\n default=0.2,\n metadata={\"help\": \"Epsilon value for clipping.\"},\n )\n delta: float | None = field(\n default=None,\n metadata={\n \"help\": \"Enables the upper clipping bound in two-sided GRPO loss when set to a float. If `None` \"\n \"(default), standard GRPO clipping is used. Recommended to be greater than `1 + ฮต` when enabled. This \"\n \"method is introduced in the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291).\"\n },\n )\n epsilon_high: float | None = field(\n default=None,\n metadata={\n \"help\": \"Upper-bound epsilon value for clipping. If not specified, it defaults to the same value as the \"\n \"lower-bound specified in argument `epsilon`. Paper DAPO recommends `0.28`. \"\n \"When used with `loss_type='cispo'`, this corresponds to the ฮต_max param specified in the\"\n \"[ScaleRL paper]https://huggingface.co/papers/2510.13786) and the recommended value is `5.0`.\"\n },\n )\n sapo_temperature_neg: float = field(\n default=1.05,\n metadata={\n \"help\": \"Temperature for tokens with non-positive advantage scores used in the `sapo` loss function. \"\n \"This parameter is introduced in the [Soft Adaptive Policy Optimization \"\n \"paper](https://huggingface.co/papers/2511.20347).\"\n },\n )\n sapo_temperature_pos: float = field(\n default=1.0,\n metadata={\n \"help\": \"Temperature for tokens with positive advantage scores used in the `sapo` loss function. \"\n \"This parameter is introduced in the [Soft Adaptive Policy Optimization \"\n \"paper](https://huggingface.co/papers/2511.20347).\"\n },\n )\n vespo_k_pos: float = field(\n default=2.0,\n metadata={\n \"help\": \"k parameter for positive advantages, it is the power exponent in the VESPO loss. Controls how \"\n \"aggressively we down-weight samples with low importance weights (when the importance sampling ratio < 1).\"\n },\n )\n vespo_lambda_pos: float = field(\n default=3.0,\n metadata={\n \"help\": \"lambda parameter for positive advantages, it is the decay factor in the VESPO loss. Controls \"\n \"how aggressively we down-weight samples with high importance weights (when the importance sampling ratio \"\n \"> 1).\"\n },\n )\n vespo_k_neg: float = field(\n default=3.0,\n metadata={\n \"help\": \"k parameter for negative advantages, it is the power exponent in the VESPO loss. Controls how \"\n \"aggressively we down-weight samples with low importance weights (when the importance sampling ratio < 1).\"\n },\n )\n vespo_lambda_neg: float = field(\n default=2.0,\n metadata={\n \"help\": \"lambda parameter for negative advantages, it is the exponential decay factor in the VESPO loss. \"\n \"Controls how aggressively we down-weight samples with high importance weights (when the importance \"\n \"sampling ratio > 1).\"\n },\n )\n importance_sampling_level: str = field(\n default=\"token\",\n metadata={\n \"help\": \"Controls whether importance sampling ratios are computed at the `'token'` or `'sequence'` level. \"\n \"`'token'` keeps the raw per-token log-probability ratios (one weight per token). `'sequence'` averages \"\n \"the log-probability ratios across valid tokens to produce a single ratio per sequence. The GSPO paper \"\n \"shows that sequence-level sampling often yields more stable training and better alignment with \"\n \"sequence-level rewards.\"\n },\n )\n reward_weights: list[float] | None = field(\n default=None,\n metadata={\n \"help\": \"Weights for each reward function. Must match the number of reward functions. If `None`, all \"\n \"rewards are weighted equally with weight `1.0`.\"\n },\n )\n multi_objective_aggregation: str = field(\n default=\"sum_then_normalize\",\n metadata={\n \"help\": \"Method to aggregate multiple reward functions. Supported values are: \"\n \"`'sum_then_normalize'` (default): First sums the weighted rewards from each reward function, then \"\n \"applies reward scaling/normalization as specified by `scale_rewards` (see `scale_rewards` for details). \"\n \"`'normalize_then_sum'`: First normalizes/scales each reward function across generations (within each \"\n \"group), then sums the normalized rewards using the specified weights. The aggregated reward is then \"\n \"normalized at the batch level when forming advantages. This is the suggested approach from the paper \"\n \"GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization.\"\n },\n )\n scale_rewards: str = field(\n default=\"group\",\n metadata={\n \"help\": \"Specifies the scaling strategy for rewards. Supported values are: \"\n \"`True` or `group'` (default): rewards are scaled by the standard deviation within each group, ensuring \"\n \"unit variance within a group. \"\n \"`'batch'`: rewards are scaled by the standard deviation across the entire batch, as recommended in the \"\n \"PPO Lite paper. \"\n \"`False` or `'none'`: no scaling is applied. The Dr. GRPO paper recommends not scaling rewards, as \"\n \"scaling by the standard deviation introduces a question-level difficulty bias.\"\n },\n )\n loss_type: str = field(\n default=\"dapo\",\n metadata={\n \"help\": \"Specifies the loss formulation to use. Supported values are 'grpo', 'dapo', 'bnpo', and \"\n \"'dr_grpo'. \"\n \"'grpo': Aggregates token-level losses by normalizing over sequence length. Not recommended due to length \"\n \"biasโ€”this approach tends to prefer shorter completions with positive advantages and longer ones with \"\n \"negative advantages. \"\n \"'dapo' (default): Aggregates token-level losses by normalizing with the number of active token in the \"\n \"global accumulated batch. This method was introduced in the DAPO paper to eliminate length bias. \"\n \"'dr_grpo': Aggregates token-level losses by normalizing with a global constant. This method was \"\n \"introduced in the Dr. GRPO paper to eliminate length bias. The value of the constant corresponds to \"\n \"`max_completion_length`. \"\n \"'bnpo': Aggregates token-level losses by normalizing with the number of active token in the local batch. \"\n \"Note that normalization is performed over the local batch only, so results may slightly vary depending \"\n \"on the local batch size, despite a constant effective batch size. When using \"\n \"`per_device_train_batch_size==1`, the loss is equivalent to the GRPO loss.\"\n \"'cispo': Clips the importance sampling weights instead of the advantage scaled importance weights. \"\n \"The clipped weights are then multiplied with the advantages and policy model's log probs. \"\n \"Individual token losses are aggregated by normalizing with the number of active tokens in \"\n \"the global accumulated batch. This method was introduced in the \"\n \"[MiniMax-M1 paper](https://huggingface.co/papers/2506.13585). \"\n \"'sapo': Soft Adaptive Policy Optimization loss, as introduced in the \"\n \"[Soft Adaptive Policy Optimization paper](https://huggingface.co/papers/2511.20347). \"\n \"Replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates \"\n \"off-policy updates while preserving useful learning signals.\"\n \"'luspo': Length-Unbiased Sequence Policy Optimization loss. A sequence-level loss that scales each \"\n \"sequence's loss by its length. This is a modification of GSPO and requires \"\n \"`importance_sampling_level='sequence'`. Introduced in the [LUSPO \"\n \"paper](https://huggingface.co/papers/2602.05261).\"\n \"'vespo': Variational Sequence-Level Soft Policy Optimization. Replaces hard clipping with a smooth, \"\n \"asymmetric Gamma weighting function applied directly to sequence-level importance weights. Introduced in \"\n \"the [VESPO paper](https://huggingface.co/papers/2602.10693).\"\n },\n )\n mask_truncated_completions: bool = field(\n default=False,\n metadata={\n \"help\": \"When enabled, truncated completions are excluded from the loss calculation, preventing them from \"\n \"being incorrectly penalized and introducing noise during training. According to the DAPO paper, this is \"\n \"a good practice for training stability.\"\n },\n )\n sync_ref_model: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to synchronize the reference model with the active model every `ref_model_sync_steps` \"\n \"steps, using the `ref_model_mixup_alpha` parameter.\"\n },\n )\n ref_model_mixup_alpha: float = field(\n default=0.6,\n metadata={\n \"help\": \"ฮฑ parameter from the TR-DPO paper, which controls the mix between the current policy and the \"\n \"previous reference policy during updates. The reference policy is updated according to the equation: \"\n \"`ฯ€_ref = ฮฑ * ฯ€_ฮธ + (1 - ฮฑ) * ฯ€_ref_prev`. To use this parameter, you must set `sync_ref_model=True`.\"\n },\n )\n ref_model_sync_steps: int = field(\n default=512,\n metadata={\n \"help\": \"ฯ„ parameter from the TR-DPO paper, which determines how frequently the current policy is \"\n \"synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`.\"\n },\n )\n top_entropy_quantile: float = field(\n default=1.0,\n metadata={\n \"help\": \"ฯ parameter from Beyond the 80/20 Rule. Keeps in the policy loss term only the top-ฯ quantile of \"\n \"tokens by entropy of the probability distribution at each sequence position, improving results. Range: \"\n \"[0.0-1.0]. A value of `0.0` masks all but the highest entropy token; `1.0` keeps all tokens. The paper \"\n \"recommends a value of `0.2`. If used with `mask_truncated_completions=True`, only tokens from \"\n \"non-truncated completions are considered.\"\n },\n )\n max_tool_calling_iterations: int | None = field(\n default=None,\n metadata={\n \"help\": \"Maximum number of tool-calling turns when training an agent. If `None`, there is no limit and \"\n \"generation stops when the model generates a response turn with no tool calls or when the total \"\n \"response length reaches `max_model_length`.\"\n },\n )\n vllm_importance_sampling_correction: bool = field(\n default=True,\n metadata={\n \"help\": \"Whether to apply Importance Sampling (IS) to correct for the mismatch between vLLM \"\n \"completion logprobs and recomputed training logprobs. If set to `False`, no IS is applied \"\n \"regardless of `vllm_importance_sampling_mode`. When `True`, the selected mode determines how \"\n \"IS ratios are computed and constrained.\"\n },\n )\n vllm_importance_sampling_mode: str = field(\n default=\"sequence_mask\",\n metadata={\n \"help\": \"Specifies how Importance Sampling (IS) is performed when \"\n \"vllm_importance_sampling_correction=True. Modes are defined along two orthogonal \"\n \"dimensions: (1) constraint, which determines how to handle ratios above \"\n \"vllm_importance_sampling_cap (C)โ€”either truncation (clip from above, ฯ โ† min(ฯ, C)) or \"\n \"masking (set ratios above C to zero); and (2) granularity, which determines whether \"\n \"ratios are computed per token or as a single sequence-level ratio applied to all tokens. \"\n \"Supported options are: 'token_truncate', 'token_mask', 'sequence_truncate', and \"\n \"'sequence_mask'.\"\n },\n )\n vllm_importance_sampling_cap: float = field(\n default=3.0,\n metadata={\n \"help\": \"Importance sampling cap C used by `vllm_importance_sampling_mode`. For '*_truncate' modes, \"\n \"ratios are clipped from above at C. For '*_mask' modes, ratios larger than C are set to zero.\"\n },\n )\n off_policy_mask_threshold: float | None = field(\n default=None,\n metadata={\n \"help\": \"Threshold for off-policy sequence masking. If `None`, off-policy sequence masking is disabled. \"\n \"When set, sequences with negative advantages and high KL divergence are masked out to stabilize \"\n \"training. This parameter corresponds to the `delta` threshold in Equation 9 of the [DeepSeek-V3.2 \"\n \"paper](https://huggingface.co/papers/2512.02556). It expects a positive value (e.g., 0.5).\"\n },\n )\n use_bias_correction_kl: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use the unbiased KL divergence estimator with importance sampling correction. This \"\n \"corrects the KL divergence estimate by multiplying it with the importance sampling ratio. \"\n \"This is described in the [DeepSeek-V3.2 paper](https://huggingface.co/papers/2512.02556).\"\n },\n )\n\n # Parameters that control the logging\n log_completions: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is \"\n \"installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`.\"\n },\n )\n num_completions_to_print: int | None = field(\n default=None,\n metadata={\"help\": \"Number of completions to print with `rich`. If `None`, all completions are logged.\"},\n )\n log_unique_prompts: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to log unique prompts. If `True`, only unique prompts are logged. If `False`, all \"\n \"prompts are logged.\"\n },\n )\n log_completions_hub_repo: str | None = field(\n default=None,\n metadata={\n \"help\": \"Hugging Face Hub repository to save the completions. Should be a complete repository name like \"\n \"`'username/reponame'` or `'orgname/reponame'`, or just `'reponame'` in which case the repository will \"\n \"be created in the currently-logged-in Hugging Face user's namespace. Note that this repository will be \"\n \"public unless you set `hub_private_repo=True` or your organization's default is to create private \"\n \"repositories.\"\n },\n )\n\n def __post_init__(self):\n super().__post_init__()\n\n self.scale_rewards = {True: \"group\", False: \"none\"}.get(self.scale_rewards, self.scale_rewards)\n\n if self.log_completions_hub_repo is not None and not self.log_completions:\n raise ValueError(\n \"log_completions_hub_repo is set, but log_completions is False. Enable log_completions to upload \"\n \"completions to the Hub, or unset log_completions_hub_repo.\"\n )\n\n num_processes = self.world_size\n # The current default effective batch size\n if self.generation_batch_size is None and self.steps_per_generation is None:\n self.steps_per_generation = self.gradient_accumulation_steps\n self.generation_batch_size = self.per_device_train_batch_size * num_processes * self.steps_per_generation\n elif self.generation_batch_size is not None and self.steps_per_generation is None:\n # Just ensure the value is divisible by the global batch size\n if self.generation_batch_size % (self.per_device_train_batch_size * num_processes) != 0:\n raise ValueError(\n f\"generation_batch_size ({self.generation_batch_size}) must be divisible by the global batch size \"\n f\"({self.per_device_train_batch_size * num_processes}).\"\n )\n self.steps_per_generation = self.generation_batch_size // (\n self.per_device_train_batch_size * num_processes\n )\n elif self.generation_batch_size is None and self.steps_per_generation is not None:\n self.generation_batch_size = self.per_device_train_batch_size * num_processes * self.steps_per_generation\n else:\n raise ValueError(\n \"'generation_batch_size' and 'steps_per_generation' can not be both configured at the same time\"\n )\n\n if self.do_eval and self.eval_strategy != \"no\":\n # Determine the number of generations to use for evaluation\n num_generations = self.num_generations_eval or self.num_generations\n\n # Just ensure the value is divisible by the global batch size\n if (self.per_device_eval_batch_size * num_processes) % num_generations != 0:\n raise ValueError(\n f\"The global eval batch size ({self.per_device_eval_batch_size} * {num_processes}) must be \"\n f\"divisible by the number of generations used for evaluation ({num_generations}).\"\n )\n\n # The generation batch must contain full prompt groups (no partials), so it must be divisible by\n # num_generations.\n if self.generation_batch_size % self.num_generations != 0:\n raise ValueError(\n f\"generation_batch_size ({self.generation_batch_size}) must be divisible by num_generations \"\n f\"({self.num_generations}).\"\n )\n\n if self.num_generations < 2:\n raise ValueError(\n \"GRPO requires at least 2 generations per prompt to calculate the advantages. You provided \"\n f\"{self.num_generations}, which is less than the minimum required.\"\n )\n\n if self.delta is not None and self.use_liger_kernel:\n raise ValueError(\"Liger kernel does not support two-sided GRPO loss yet.\")\n" }, { "path": "trl/trainer/grpo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport asyncio\nimport atexit\nimport copy\nimport importlib.resources as pkg_resources\nimport inspect\nimport math\nimport os\nimport sys\nimport textwrap\nimport time\nimport warnings\nfrom collections import defaultdict, deque\nfrom collections.abc import Callable\nfrom contextlib import nullcontext\nfrom pathlib import Path\nfrom typing import Any, Protocol\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.utils.data\nimport transformers\nfrom accelerate.logging import get_logger\nfrom accelerate.utils import gather, gather_object, is_peft_model, set_seed\nfrom datasets import Dataset, IterableDataset\nfrom huggingface_hub import CommitScheduler, DatasetCard, DatasetCardData, create_repo\nfrom packaging.version import Version\nfrom torch import nn\nfrom torch.distributed.fsdp import FullyShardedDataParallel as FSDP\nfrom torch.utils.data import Sampler\nfrom transformers import (\n AutoModelForSequenceClassification,\n AutoProcessor,\n AutoTokenizer,\n GenerationConfig,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n is_trackio_available,\n is_wandb_available,\n)\nfrom transformers.utils import is_peft_available, is_rich_available\n\nfrom ..chat_template_utils import add_response_schema, get_training_chat_template, parse_response\nfrom ..data_utils import (\n apply_chat_template,\n is_conversational,\n prepare_multimodal_messages,\n)\nfrom ..extras.profiling import profiling_context, profiling_decorator\nfrom ..generation.vllm_generation import VLLMGeneration\nfrom ..import_utils import is_jmespath_available, is_liger_kernel_available\nfrom ..models import prepare_deepspeed, prepare_fsdp, unwrap_model_for_generation\nfrom ..models.utils import _ForwardRedirection, disable_gradient_checkpointing\nfrom .base_trainer import _BaseTrainer\nfrom .callbacks import SyncRefModelCallback\nfrom .grpo_config import GRPOConfig\nfrom .utils import (\n RepeatSampler,\n create_model_from_path,\n disable_dropout_in_model,\n entropy_from_logits,\n get_config_model_id,\n identity,\n nanmax,\n nanmin,\n nanstd,\n pad,\n print_prompt_completions_sample,\n selective_log_softmax,\n shuffle_sequence_dict,\n shutdown_event_loop_in_daemon,\n split_pixel_values_by_grid,\n split_tensor_dict,\n start_event_loop_in_daemon,\n unsplit_pixel_values_by_grid,\n use_adapter,\n)\n\n\nif is_peft_available():\n from peft import PeftConfig, PeftModel, get_peft_model\n\nif is_liger_kernel_available():\n from liger_kernel.chunked_loss import LigerFusedLinearGRPOLoss\n\n\nif is_wandb_available():\n import wandb\n\nif is_trackio_available():\n import trackio\n\nlogger = get_logger(__name__)\n\n# A reward function can be a string, interpreted as a model ID and loaded as a pretrained model, a pretrained model, or\n# a callable that returns a list of floats (the rewards). The callable receives prompts, completions, and additional\n# arguments from the trainer (refer to the trainer's source for details). To ensure forward compatibility, it should\n# accept **kwargs.\nRewardFunc = str | PreTrainedModel | Callable[..., list[float | None]]\n\n# What we call a rollout function is a callable that takes prompts (list) and the trainer instance as parameters and\n# returns a dict of generation results. Those results must include \"prompt_ids\", \"completion_ids\", and \"logprobs\"\n# fields. Any extra fields (per-completion) are forwarded to the reward functions.\nRolloutFunc = Callable[[list[str], \"GRPOTrainer\"], dict[str, Any]]\n\n\nclass _SupportsReset(Protocol):\n def reset(self, **kwargs) -> str | None: ...\n\n\nEnvironmentFactory = Callable[[], _SupportsReset]\n\n\nclass GRPOTrainer(_BaseTrainer):\n \"\"\"\n Trainer for the Group Relative Policy Optimization (GRPO) method. This algorithm was initially proposed in the\n paper [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language\n Models](https://huggingface.co/papers/2402.03300).\n\n Example:\n\n ```python\n from trl import GRPOTrainer\n from trl.rewards import accuracy_reward\n from datasets import load_dataset\n\n dataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\n trainer = GRPOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n )\n trainer.train()\n ```\n\n Args:\n model (`str` or [`~transformers.PreTrainedModel`] or [`~peft.PeftModel`]):\n Model to be trained. Can be either:\n\n - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using `.from_pretrained` (where `` is derived from the model\n config) with the keyword arguments in `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported.\n - A [`~peft.PeftModel`] object. Only causal language models are supported.\n reward_funcs (`RewardFunc | list[RewardFunc]`):\n Reward functions to be used for computing the rewards. To compute the rewards, we call all the reward\n functions with the prompts and completions and sum the rewards. Can be either:\n\n - A single reward function, such as:\n - A string: The *model ID* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using [`~transformers.AutoModelForSequenceClassification.from_pretrained`] with `num_labels=1` and the\n keyword arguments in `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object: Only sequence classification models are supported.\n - A custom reward function: The function is provided with the prompts and the generated completions,\n plus any additional columns in the dataset. It should return a list of rewards. Custom reward\n functions can be either synchronous or asynchronous and can also return `None` when the reward is\n not applicable to those samples. This is useful for multi-task training where different reward\n functions apply to different types of samples. When a reward function returns `None` for a sample,\n that reward function is excluded from the reward calculation for that sample. For more details, see\n [Using a custom reward\n function](#using-a-custom-reward-function).\n\n The trainer's state is also passed to the reward function. The trainer's state is an instance of\n [`~transformers.TrainerState`] and can be accessed by accessing the `trainer_state` argument to the\n reward function's signature.\n - A list of reward functions, where each item can independently be any of the above types. Mixing different\n types within the list (e.g., a string model ID and a custom reward function) is allowed.\n args ([`GRPOConfig`], *optional*):\n Configuration for this trainer. If `None`, a default configuration is used.\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n Dataset to use for training. It must include a column `\"prompt\"`. Any additional columns in the dataset is\n ignored. The format of the samples can be either:\n\n - [Standard](dataset_formats#standard): Each sample contains plain text.\n - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role\n and content).\n eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Dataset | IterableDataset]`):\n Dataset to use for evaluation. It must meet the same requirements as `train_dataset`.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. The padding side must be set to \"left\". If `None`, the\n processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A\n padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token,\n `tokenizer.eos_token` will be used as the default.\n reward_processing_classes ([`~transformers.PreTrainedTokenizerBase`] or `list[PreTrainedTokenizerBase]`, *optional*):\n Processing classes corresponding to the reward functions specified in `reward_funcs`. Can be either:\n\n - A single processing class: Used when `reward_funcs` contains only one reward function.\n - A list of processing classes: Must match the order and length of the reward functions in `reward_funcs`.\n If set to `None`, or if an element of the list corresponding to a [`~transformers.PreTrainedModel`] is\n `None`, the tokenizer for the model is automatically loaded using\n [`~transformers.AutoTokenizer.from_pretrained`]. For elements in `reward_funcs` that are custom reward\n functions (not [`~transformers.PreTrainedModel`]), the corresponding entries in `reward_processing_classes`\n are ignored.\n callbacks (list of [`~transformers.TrainerCallback`], *optional*):\n List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed\n in [here](https://huggingface.co/docs/transformers/main_classes/callback).\n\n If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`]\n method.\n optimizers (`tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None]`, *optional*, defaults to `(None, None)`):\n A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your\n model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`.\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration used to wrap the model. If `None`, the model is not wrapped.\n tools (list of `Callable`, *optional*):\n A list of callable tool functions (sync or async) that the model can invoke during generation. Each tool\n should be a standard Python function with properly type-hinted arguments and return values, and a\n Google-style docstring describing its purpose, arguments, and return value. For more details, see:\n https://huggingface.co/docs/transformers/en/chat_extras#passing-tools. The model uses the function's name,\n type hints, and docstring to determine how to call it. Ensure that the model's chat template supports tool\n use and that it has been fine-tuned for tool calling.\n rollout_func (`RolloutFunc`, *optional*):\n Function to use for generating completions. It receives the list of prompts allocated to the current\n process and the trainer instance. It must return a dict with `\"prompt_ids\"`, `\"completion_ids\"`, and\n `\"logprobs\"` fields, and can optionally return `\"logprob_token_ids\"` (same shape as `\"logprobs\"`). Any\n other fields are forwarded to the reward functions. The function receives the raw per-process prompt slice\n with no duplication; it is responsible for returning the correct number of completions per prompt (see\n `num_generations` / `num_generations_eval` on the trainer). This feature is experimental and may change or\n be removed at any time without prior notice.\n environment_factory (`EnvironmentFactory`, *optional*):\n A callable that creates and returns an environment instance. The environment class should define methods\n that can be invoked as tools during generation. Each method should comply with the same requirements as the\n `tools` described above. If `environment_factory` is provided, an instance of the environment is created\n for each generation in the batch, allowing for parallel and independent interactions. The environment must\n also implement a callable `reset` method that can be used to reset state between generations. The `reset`\n method should return either `None` or a string: when it returns a string, that string is appended to the\n last user message before generation. This feature is experimental and may change or be removed at any time\n without prior notice.\n \"\"\"\n\n _tag_names = [\"trl\", \"grpo\"]\n _name = \"GRPO\"\n _paper = {\n \"title\": \"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\",\n \"id\": \"2402.03300\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @article{shao2024deepseekmath,\n title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},\n author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},\n year = 2024,\n eprint = {arXiv:2402.03300},\n }\n \"\"\"),\n }\n\n def __init__(\n self,\n model: \"str | PreTrainedModel | PeftModel\",\n reward_funcs: RewardFunc | list[RewardFunc],\n args: GRPOConfig | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | IterableDataset | dict[str, Dataset | IterableDataset] | None = None,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin | None = None,\n reward_processing_classes: PreTrainedTokenizerBase | list[PreTrainedTokenizerBase] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None] = (None, None),\n peft_config: \"PeftConfig | None\" = None,\n tools: list[Callable] | None = None,\n rollout_func: RolloutFunc | None = None,\n environment_factory: EnvironmentFactory | None = None,\n ):\n # Args\n if args is None:\n model_name = model if isinstance(model, str) else get_config_model_id(model.config)\n model_name = model_name.split(\"/\")[-1]\n args = GRPOConfig(f\"{model_name}-GRPO\")\n\n # Model\n if isinstance(model, str):\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n model = create_model_from_path(model, **model_init_kwargs)\n else:\n if args.model_init_kwargs is not None:\n logger.warning(\n \"You passed `model_init_kwargs` to the `GRPOConfig`, but your model is already instantiated. \"\n \"The `model_init_kwargs` will be ignored.\"\n )\n\n # Some models (SmolVLM/Idefics3) don't support `logits_to_keep` argument and error out if we pass it\n # Inspect the forward method before we wrap the model with PEFT\n self.model_kwarg_keys = (\n inspect.signature(model.forward).parameters.keys()\n if not hasattr(model, \"get_base_model\")\n else inspect.signature(model.get_base_model().forward).parameters.keys()\n )\n\n # Processing class\n if processing_class is None:\n processing_class = AutoProcessor.from_pretrained(\n get_config_model_id(model.config), truncation_side=\"left\", padding_side=\"left\"\n )\n\n # Handle pad token for processors or tokenizers\n if isinstance(processing_class, ProcessorMixin):\n tokenizer = processing_class.tokenizer\n elif isinstance(processing_class, PreTrainedTokenizerBase):\n tokenizer = processing_class\n else:\n raise TypeError(\"The `processing_class` must be either a `PreTrainedTokenizerBase` or a `ProcessorMixin`\")\n\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n self.pad_token = tokenizer.pad_token\n self.pad_token_id = tokenizer.pad_token_id\n self.eos_token_id = tokenizer.eos_token_id\n\n if is_peft_available() and is_peft_model(model) and peft_config is not None:\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first merge \"\n \"and unload the existing adapter, save the resulting base model, and then pass that base model along \"\n \"with the new `peft_config` to the trainer.\"\n )\n\n if is_peft_available() and is_peft_model(model) and args.beta != 0.0:\n # If the model is a PEFT model with a pretrained adapter, we need to create a \"ref\" adapter that is a copy\n # of the \"default\" adapter, so that we can use it as the reference model during GRPO training.\n model.add_adapter(\"ref\", model.peft_config[\"default\"])\n for name, param in model.named_parameters():\n if \".default.\" in name:\n ref_name = name.replace(\".default.\", \".ref.\")\n ref_param = model.get_parameter(ref_name)\n ref_param.data.copy_(param.data)\n\n # Create PEFT model\n if peft_config is not None:\n model = get_peft_model(model, peft_config)\n\n # When using gradient checkpointing with PEFT, we need to enable input gradients. transformers.Trainer normally\n # handles this, but a bug currently prevents it; see https://github.com/huggingface/transformers/issues/42489\n if is_peft_available() and is_peft_model(model) and args.gradient_checkpointing:\n model.enable_input_require_grads()\n\n # When using QLoRA, the PEFT adapter weights are converted to bf16 to follow the recommendations from the\n # original paper (see https://huggingface.co/papers/2305.14314, paragraph 3). Normally, this can be done by\n # passing `autocast_adapter_dtype=False` to `get_peft_model`, but this option is not yet supported for\n # quantized models. See: https://github.com/huggingface/peft/issues/2889\n # Non-quantized models do not have the `is_loaded_in_{8,4}bit` attributes, whereas quantized models do\n if getattr(model, \"is_loaded_in_4bit\", False) or getattr(model, \"is_loaded_in_8bit\", False):\n for param in model.parameters():\n if param.requires_grad:\n param.data = param.data.to(torch.bfloat16)\n\n # Reward functions\n if not isinstance(reward_funcs, list):\n reward_funcs = [reward_funcs]\n self.reward_func_names = []\n for i, reward_func in enumerate(reward_funcs):\n if isinstance(reward_func, str):\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n reward_funcs[i] = AutoModelForSequenceClassification.from_pretrained(\n reward_func, num_labels=1, **model_init_kwargs\n )\n if isinstance(reward_funcs[i], nn.Module): # Use Module over PretrainedModel for compat w/ compiled models\n self.reward_func_names.append(get_config_model_id(reward_funcs[i].config).split(\"/\")[-1])\n else:\n self.reward_func_names.append(reward_funcs[i].__name__)\n self.reward_funcs = reward_funcs\n\n # Reward weights\n if args.reward_weights is not None:\n if len(args.reward_weights) != len(reward_funcs):\n raise ValueError(\n f\"Number of reward weights ({len(args.reward_weights)}) must match number of reward \"\n f\"functions ({len(reward_funcs)})\"\n )\n self.reward_weights = torch.tensor(args.reward_weights, dtype=torch.float32)\n else:\n self.reward_weights = torch.ones(len(reward_funcs), dtype=torch.float32)\n\n # Reward processing class\n if reward_processing_classes is None:\n reward_processing_classes = [None] * len(reward_funcs)\n elif not isinstance(reward_processing_classes, list):\n reward_processing_classes = [reward_processing_classes]\n if len(reward_processing_classes) != len(reward_funcs):\n raise ValueError(\n f\"The number of reward processing classes ({len(reward_processing_classes)}) must match the number of \"\n f\"reward functions ({len(reward_funcs)}).\"\n )\n\n for i, (reward_processing_class, reward_func) in enumerate(\n zip(reward_processing_classes, reward_funcs, strict=True)\n ):\n if isinstance(reward_func, PreTrainedModel):\n if reward_processing_class is None:\n reward_processing_class = AutoTokenizer.from_pretrained(get_config_model_id(reward_func.config))\n if reward_processing_class.pad_token_id is None:\n reward_processing_class.pad_token = reward_processing_class.eos_token\n # The reward model computes the reward for the latest non-padded token in the input sequence.\n # So it's important to set the pad token ID to the padding token ID of the processing class.\n reward_func.config.pad_token_id = reward_processing_class.pad_token_id\n reward_processing_classes[i] = reward_processing_class\n\n self.reward_processing_classes = reward_processing_classes\n\n # Rollout function\n if rollout_func is not None and os.environ.get(\"TRL_EXPERIMENTAL_SILENCE\", \"0\") != \"1\":\n warnings.warn(\n \"You are using 'rollout_func', which is an experimental feature. This API may change or be removed at \"\n \"any time without prior notice. Silence this warning by setting environment variable \"\n \"TRL_EXPERIMENTAL_SILENCE=1.\",\n UserWarning,\n stacklevel=2,\n )\n self.rollout_func = rollout_func\n if environment_factory is not None and os.environ.get(\"TRL_EXPERIMENTAL_SILENCE\", \"0\") != \"1\":\n warnings.warn(\n \"You are using 'environment_factory', which is an experimental feature. This API may change or be \"\n \"removed at any time without prior notice. Silence this warning by setting environment variable \"\n \"TRL_EXPERIMENTAL_SILENCE=1.\",\n UserWarning,\n stacklevel=2,\n )\n\n # Tools\n if tools:\n if not Version(transformers.__version__) >= Version(\"5.0.0\"):\n raise ImportError(\n \"Using tools with GRPOTrainer requires transformers version 5.0.0 or higher. Please upgrade \"\n \"transformers with `pip install --upgrade transformers` to use this feature.\"\n )\n if environment_factory:\n if not Version(transformers.__version__) >= Version(\"5.2.0\"):\n raise ImportError(\n \"Using `environment_factory` with GRPOTrainer requires transformers version 5.2.0 or higher. \"\n \"Please install transformers from the main branch with `pip install \"\n \"git+https://github.com/huggingface/transformers.git@main` to use this feature.\"\n )\n if tools or environment_factory:\n if not is_jmespath_available():\n raise ImportError(\n \"Using tools with GRPOTrainer requires the jmespath library for response parsing. Please install \"\n \"it with `pip install jmespath` to use this feature.\"\n )\n\n # Create the environments and extract their methods to be used as tools. We create one environment per rollout\n generation_batch_size = args.per_device_train_batch_size * args.steps_per_generation\n if environment_factory is not None:\n self.environments = [environment_factory() for _ in range(generation_batch_size)]\n environment_methods = [[] for _ in range(generation_batch_size)]\n for i, environment in enumerate(self.environments):\n has_reset = False\n for name, member in inspect.getmembers(environment, predicate=inspect.ismethod):\n if name == \"reset\":\n has_reset = True\n elif not name.startswith(\"_\"):\n environment_methods[i].append(member)\n if not has_reset:\n raise ValueError(\n \"Each environment instance returned by `environment_factory` must define a callable `reset` \"\n )\n else:\n self.environments = None\n\n tools = tools or []\n self._sync_tool_dicts = [{} for _ in range(generation_batch_size)]\n self._async_tool_dicts = [{} for _ in range(generation_batch_size)]\n for i in range(generation_batch_size):\n for tool in tools + (environment_methods[i] if self.environments is not None else []):\n if inspect.iscoroutinefunction(tool):\n self._async_tool_dicts[i][tool.__name__] = tool\n else:\n self._sync_tool_dicts[i][tool.__name__] = tool\n\n self.tools = tools + (environment_methods[0] if self.environments is not None else [])\n\n # Check for async functions to start an event loop on a daemon thread\n self._has_async_funcs = any(inspect.iscoroutinefunction(func) for func in self.reward_funcs + self.tools)\n\n if self._has_async_funcs:\n self.async_loop_thread, self.async_loop, self.async_loop_ready_event = start_event_loop_in_daemon(\n name=\"GRPOTrainer-AsyncLoop\"\n )\n # wait until the event loop is running in the daemon thread\n self.async_loop_ready_event.wait()\n atexit.register(shutdown_event_loop_in_daemon, self.async_loop_thread, self.async_loop)\n\n # At the time of initial implementation, most tokenizers do not have built-in support for response schemas.\n # While waiting for broader adoption, we provide this utility function to manually set the response schema for\n # known chat templates.\n # We need `getattr`` until the base class sets a default None value for response_schema\n if self.tools and not getattr(processing_class, \"response_schema\", None):\n processing_class = add_response_schema(processing_class)\n # In multi-turn training, the chat template *must* be prefix-preserving. If the tokenizer's original template\n # isn't, we replace it at initialization with a training-safe, prefix-preserving template.\n if self.tools:\n self.chat_template = get_training_chat_template(processing_class)\n else:\n self.chat_template = None\n\n # Training arguments\n self.max_completion_length = args.max_completion_length # = |o_i| in the GRPO paper\n self.num_generations = args.num_generations # = G in the GRPO paper\n self.max_tool_calling_iterations = args.max_tool_calling_iterations or sys.maxsize\n self.num_generations_eval = args.num_generations_eval or self.num_generations\n self.chat_template_kwargs = args.chat_template_kwargs or {}\n self.temperature = args.temperature\n self.top_p = args.top_p\n self.top_k = args.top_k\n self.min_p = args.min_p\n self.repetition_penalty = args.repetition_penalty\n self.use_transformers_paged = args.use_transformers_paged\n self.pad_to_multiple_of = args.pad_to_multiple_of\n self.use_vllm = args.use_vllm\n self.vllm_mode = args.vllm_mode\n self.vllm_gpu_memory_utilization = args.vllm_gpu_memory_utilization # only applies to colocation mode\n self.vllm_tensor_parallel_size = args.vllm_tensor_parallel_size # only applies to colocation mode\n self.vllm_importance_sampling_correction = args.vllm_importance_sampling_correction\n self.vllm_importance_sampling_mode = args.vllm_importance_sampling_mode\n self.vllm_importance_sampling_cap = args.vllm_importance_sampling_cap\n self.use_liger_kernel = args.use_liger_kernel\n self.loss_type = args.loss_type\n self.multi_objective_aggregation = args.multi_objective_aggregation\n self.scale_rewards = args.scale_rewards\n self.importance_sampling_level = args.importance_sampling_level\n self.off_policy_mask_threshold = args.off_policy_mask_threshold\n if self.use_liger_kernel and self.off_policy_mask_threshold is not None:\n raise ValueError(\"Liger kernel does not support off-policy sequence masking yet.\")\n self.mask_truncated_completions = args.mask_truncated_completions\n self.top_entropy_quantile = args.top_entropy_quantile\n if self.use_liger_kernel and self.top_entropy_quantile < 1.0:\n raise NotImplementedError(\n \"Liger Kernels don't currently support masking token positions based on entropy.\"\n )\n if self.use_liger_kernel and self.importance_sampling_level not in (\"token\", \"sequence\"):\n raise ValueError(\n f\"Unknown importance sampling level: {self.importance_sampling_level}. \"\n \"Possible values are 'token' and 'sequence'.\"\n )\n\n # Datasets\n self.shuffle_dataset = args.shuffle_dataset\n\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n elif (\n isinstance(train_dataset, IterableDataset)\n or isinstance(eval_dataset, IterableDataset)\n or (\n isinstance(eval_dataset, dict) and any(isinstance(ds, IterableDataset) for ds in eval_dataset.values())\n )\n ):\n # See https://github.com/huggingface/trl/issues/3213\n raise NotImplementedError(\n \"Iterable datasets are not yet supported in GRPOTrainer. Please use a standard dataset instead.\"\n )\n\n if args.loss_type == \"luspo\" and args.importance_sampling_level != \"sequence\":\n logger.warning(\n \"When using `'luspo'` loss, `importance_sampling_level` should be set to `'sequence'` to mirror the \"\n \"paper's setup.\"\n )\n\n if args.loss_type == \"vespo\" and args.importance_sampling_level != \"token\":\n logger.warning(\n \"VESPO computes sequence-level importance weights internally. `importance_sampling_level` should be \"\n \"set to `'token'` (the default).\"\n )\n\n if self.loss_type == \"vespo\" and self.use_vllm and self.vllm_importance_sampling_correction:\n if self.vllm_importance_sampling_mode not in [\"token_truncate\", \"token_mask\"]:\n raise ValueError(\n f\"VESPO loss requires `vllm_importance_sampling_mode` to be either 'token_truncate' or \"\n f\"'token_mask'. Got: {self.vllm_importance_sampling_mode}.\"\n )\n\n # Multi-step\n self.num_iterations = args.num_iterations # = ๐œ‡ in the GRPO paper\n self.epsilon_low = args.epsilon\n self.epsilon_high = args.epsilon_high if args.epsilon_high is not None else args.epsilon\n # Tracks the number of iterations (forward + backward passes), including those within a grad accum cycle\n self._step = 0\n # Buffer the batch to reuse generated outputs across multiple updates. For more details, see\n # `_get_train_sampler` and `_prepare_inputs`.\n self._buffered_inputs = None\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=identity, # No data collation is needed in GRPO\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n callbacks=callbacks,\n optimizers=optimizers,\n # In Trainer, `training_step` scales the loss by `gradient_accumulation_steps` only if `compute_loss_func`\n # is None. For DAPO, loss scaling instead depends on the total number of completions tokens across the\n # global accumulated batch. To control scaling ourselves, we must disable Trainerโ€™s built-in scaling. The\n # simplest (though a bit hacky) way is to set `compute_loss_func` to any non-None value, which bypasses\n # that behavior without rewriting `training_step`.\n compute_loss_func=\"non-None value to disable scaling\",\n )\n\n # Reference model\n self.beta = args.beta\n if self.beta == 0.0:\n # If beta is 0.0, the reference model is not needed\n self.ref_model = None\n elif is_peft_model(model):\n # If PEFT is used, the reference model is not needed since the adapter can be disabled\n # to revert to the initial model.\n self.ref_model = None\n else:\n # For deepspeed, fsdp or non-distributed models, create a reference model from scratch\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if self.args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n self.ref_model = create_model_from_path(get_config_model_id(self.model.config), **model_init_kwargs)\n\n # Disable dropout in the models\n if args.disable_dropout:\n disable_dropout_in_model(model)\n if self.ref_model is not None:\n disable_dropout_in_model(self.ref_model)\n\n # Cast LM Head To FP32\n if args.cast_lm_head_to_fp32:\n\n def _cast_lm_head_to_fp32(target_model: PreTrainedModel):\n \"\"\"Cast lm_head to fp32 while preserving embedding output dtype if tied.\"\"\"\n\n def cast_inputs_to_fp32(module, inputs):\n # Preserve other positional args and kwargs untouched\n if not inputs:\n return inputs\n return (inputs[0].to(torch.float32),) + inputs[1:]\n\n original_dtype_local = target_model.lm_head.weight.dtype\n target_model.lm_head = target_model.lm_head.float()\n target_model.lm_head.register_forward_pre_hook(cast_inputs_to_fp32)\n\n if target_model.config.tie_word_embeddings:\n\n def cast_outputs_to_original_dtype(module, args, output):\n return output.to(original_dtype_local)\n\n # Only cast activations; weights are now fp32 (intentional for numerical stability of logits)\n target_model.model.embed_tokens.register_forward_hook(cast_outputs_to_original_dtype)\n\n _cast_lm_head_to_fp32(model)\n if self.ref_model is not None:\n _cast_lm_head_to_fp32(self.ref_model)\n\n # Liger loss\n if self.use_liger_kernel:\n if not is_liger_kernel_available():\n raise ImportError(\n \"Liger is required to use `use_liger_kernel` as the GRPO loss. Run `pip install liger-kernel`.\"\n )\n # redirect the model.module forward to the model forward to ensure pre-forward hooks are called\n self._forward_redirection = _ForwardRedirection()\n\n self.liger_grpo_loss = LigerFusedLinearGRPOLoss(\n beta=self.beta,\n epsilon_low=self.epsilon_low,\n epsilon_high=self.epsilon_high,\n temperature=self.temperature,\n use_ref_model=self.beta != 0.0,\n loss_type=self.loss_type,\n max_completion_length=self.max_completion_length,\n importance_sampling_level=self.importance_sampling_level,\n )\n\n # Initialize the metrics\n self._metrics = {\"train\": defaultdict(list), \"eval\": defaultdict(list)}\n self._total_train_tokens = 0\n self._current_train_step_time = 0.0\n self.log_completions = args.log_completions\n self.log_unique_prompts = args.log_unique_prompts\n self.num_completions_to_print = args.num_completions_to_print\n # Keep logs sized to the generation batch to record only outputs from the latest model update.\n self._logs = {\n \"images\": deque(maxlen=args.generation_batch_size),\n \"prompt\": deque(maxlen=args.generation_batch_size),\n \"completion\": deque(maxlen=args.generation_batch_size),\n \"rewards\": defaultdict(lambda: deque(maxlen=args.generation_batch_size)),\n \"advantages\": deque(maxlen=args.generation_batch_size),\n \"extra\": defaultdict(lambda: deque(maxlen=args.generation_batch_size)),\n }\n # Buffers for user-logged data from reward functions, flushed after gathering\n self._pending_extra_logs = defaultdict(list)\n self._pending_metrics = defaultdict(list)\n\n # Ensure each process receives a unique seed to prevent duplicate completions when generating with\n # transformers if num_generations exceeds per_device_train_batch_size. We could skip it if we use vLLM, but\n # it's safer to set it in all cases.\n set_seed(args.seed, device_specific=True)\n\n if self.use_vllm:\n # Initialize vLLM generation backend\n self.vllm_generation = VLLMGeneration(\n model=self.model,\n accelerator=self.accelerator,\n is_fsdp_enabled=self.is_fsdp_enabled,\n processing_class=self.processing_class,\n # vLLM configuration\n mode=args.vllm_mode,\n structured_outputs_regex=args.vllm_structured_outputs_regex,\n # Server mode configuration\n server_base_url=args.vllm_server_base_url,\n server_host=args.vllm_server_host,\n server_port=args.vllm_server_port,\n group_port=args.vllm_group_port,\n server_timeout=args.vllm_server_timeout,\n # Colocate mode configuration\n tensor_parallel_size=args.vllm_tensor_parallel_size,\n gpu_memory_utilization=args.vllm_gpu_memory_utilization,\n max_model_length=args.vllm_max_model_length,\n max_num_seqs=args.per_device_train_batch_size\n * args.vllm_tensor_parallel_size\n * args.steps_per_generation,\n enable_sleep_mode=args.vllm_enable_sleep_mode,\n model_impl=args.vllm_model_impl,\n # Generation configuration\n repetition_penalty=self.repetition_penalty,\n temperature=self.temperature,\n top_p=self.top_p,\n top_k=self.top_k,\n min_p=self.min_p,\n max_completion_length=self.max_completion_length,\n logprobs=0, # we only need the generated token logprobs for the importance sampling correction\n generation_kwargs=args.generation_kwargs,\n )\n self._last_loaded_step = -1 # tag to avoid useless loading during grad accumulation\n else:\n generation_kwargs = {\n \"max_new_tokens\": self.max_completion_length,\n \"do_sample\": True,\n \"pad_token_id\": tokenizer.pad_token_id,\n \"bos_token_id\": tokenizer.bos_token_id,\n \"eos_token_id\": tokenizer.eos_token_id,\n \"temperature\": self.temperature,\n \"top_p\": self.top_p,\n \"top_k\": self.top_k,\n \"min_p\": self.min_p,\n \"repetition_penalty\": self.repetition_penalty,\n \"cache_implementation\": args.cache_implementation,\n }\n if args.generation_kwargs is not None:\n generation_kwargs.update(args.generation_kwargs)\n self.generation_config = GenerationConfig(**generation_kwargs)\n # Keep training-specific generation kwargs to overwrite model's original generation config\n self.generation_kwargs = generation_kwargs\n\n # Gradient accumulation requires scaled loss. Normally, loss scaling in the parent class depends on whether the\n # model accepts loss-related kwargs. Since we compute our own loss, this check is irrelevant. We set\n # self.model_accepts_loss_kwargs to False to enable scaling.\n self.model_accepts_loss_kwargs = False\n\n # Add tags to the model\n self.model.add_model_tags(self._tag_names)\n\n if self.ref_model is not None:\n if self.is_deepspeed_enabled:\n self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)\n elif self.is_fsdp_enabled:\n self.ref_model = prepare_fsdp(self.ref_model, self.accelerator)\n else:\n self.ref_model = self.accelerator.prepare_model(self.ref_model, evaluation_mode=True)\n\n if args.sync_ref_model:\n if self.beta == 0.0:\n raise ValueError(\n \"You passed `sync_ref_model=True` while `beta=0.0`, which means the reference model is not used \"\n \"during training. Consequently, GRPOTrainer does not create a `ref_model` instance, and there is \"\n \"nothing to synchronize. Please set `sync_ref_model=False`, or set `beta` to a non-zero value.\"\n )\n if is_peft_model(model):\n raise NotImplementedError(\n \"You passed `sync_ref_model=True` while using a PEFT model, which is currently not supported. \"\n \"With PEFT, GRPOTrainer does not keep a separate reference model in memory; instead, it recovers \"\n \"reference behavior by temporarily disabling the adapter. As a result, there is no standalone \"\n \"`ref_model` instance to synchronize. Use `sync_ref_model=False`, or opt for full fine-tuning if \"\n \"you need a synced reference model. If you need `sync_ref_model` to work with PEFT, please open a \"\n \"feature request at https://github.com/huggingface/trl/issues.\"\n )\n self.add_callback(SyncRefModelCallback(ref_model=self.ref_model, accelerator=self.accelerator))\n\n for i, reward_func in enumerate(self.reward_funcs):\n if isinstance(reward_func, PreTrainedModel):\n if self.is_deepspeed_enabled:\n self.reward_funcs[i] = prepare_deepspeed(reward_func, self.accelerator)\n else:\n # set device placement to True to make `prepare_model` move `reward_func` to device when using fsdp\n self.reward_funcs[i] = self.accelerator.prepare_model(\n reward_func, evaluation_mode=True, device_placement=True\n )\n\n if self.accelerator.is_main_process and self.log_completions:\n os.makedirs(os.path.join(self.args.output_dir, \"completions\"), exist_ok=True)\n if self.args.log_completions_hub_repo is not None:\n repo_id = self.args.log_completions_hub_repo\n create_repo(repo_id, private=self.args.hub_private_repo, repo_type=\"dataset\", exist_ok=True)\n template_path = pkg_resources.files(\"trl\").joinpath(\"templates/completions_dataset_card.md\")\n card_data = DatasetCardData(\n pretty_name=\"TRL Completion logs\",\n tags=[\"trl\", \"trl-logs\", \"completions\"],\n )\n card = DatasetCard.from_template(\n card_data=card_data,\n template_path=str(template_path),\n repo_id=repo_id,\n hub_model_id=self.args.hub_model_id,\n )\n card.push_to_hub(repo_id)\n self.commit_scheduler = CommitScheduler(\n repo_id=repo_id,\n repo_type=\"dataset\",\n folder_path=f\"{self.args.output_dir}/completions\",\n every=2, # minutes\n allow_patterns=[\"*.parquet\"],\n )\n\n def _set_signature_columns_if_needed(self):\n # If `self.args.remove_unused_columns` is True, non-signature columns are removed.\n # By default, this method sets `self._signature_columns` to the model's expected inputs (usually, \"input_ids\"\n # and \"attention_mask\"). In GRPOTrainer, we preprocess data, so using the model's signature columns doesn't\n # work. Instead, we set them to the columns expected by the `training_step` method, hence the override.\n if self._signature_columns is None:\n self._signature_columns = [\"prompt\", \"image\", \"images\"]\n\n # This method overrides `Trainer.get_train_dataloader` to support our custom batching strategy.\n # Instead of returning a standard per-step batch (i.e., `per_device_batch_size), our dataloader loads an\n # *generation* batch (i.e., `per_device_batch_size ร— steps_per_generation`). This allows us to generate completions\n # once every steps_per_generation stepโ€”rather than once per accumulation stepโ€”which is significantly more\n # efficient. The only change from the original implementation is multiplying the batch size by\n # `steps_per_generation`. Thus, `_prepare_inputs` is called with this *generation* batch, and it handles the\n # splitting internally.\n # Maintenance note: This method is a copy-paste of the original `Trainer.get_train_dataloader` with only one line\n # modification.\n def get_train_dataloader(self):\n return self._get_dataloader(\n dataset=self.train_dataset,\n description=\"Training\",\n batch_size=self._train_batch_size * self.args.steps_per_generation, # < this is the change\n sampler_fn=self._get_train_sampler,\n is_training=True,\n )\n\n def _get_train_sampler(self, dataset: Dataset | None = None) -> Sampler:\n # Returns a sampler that\n # 1. ensures each prompt is repeated across multiple processes. This guarantees that identical prompts are\n # distributed to different GPUs, allowing rewards to be computed and normalized correctly within each prompt\n # group. Using the same seed across processes ensures consistent prompt assignment, preventing discrepancies\n # in group formation.\n # 2. repeats the batch multiple times to allow reusing generations across multiple updates. Refer to\n # _prepare_inputs to see how the generations are stored and reused.\n\n # In the following figure, the values are the prompt indices. The first row shows the first sampled batch, the\n # second row shows the second sampled batch, and so on.\n #\n # | GPU 0 | GPU 1 |\n #\n # global_step step <-โ”€โ”€โ”€> num_generations=2\n # <-โ”€โ”€โ”€โ”€โ”€โ”€โ”€> per_device_train_batch_size=3\n # grad_accum โ–ฒ โ–ฒ 0 0 0 0 1 1 2 2 <- Generate for the first `steps_per_generation` (prompts 0 to 11); store the completions; use the first slice to compute the loss\n # =2 โ–ผ | 0 1 3 3 4 4 5 5 <- Take the stored generations and use the second slice to compute the loss\n # |\n # | 1 2 6 6 7 7 8 8 <- Take the stored generations and use the third slice to compute the loss\n # steps_per_gen=4 โ–ผ 1 3 9 9 10 10 11 11 <- Take the stored generations and use the fourth slice to compute the loss\n #\n # 2 4 12 12 13 13 14 14 <- Generate for the second `steps_per_generation` (prompts 12 to 23); store the completions; use the first slice to compute the loss\n # 2 5 15 15 16 16 17 17 <- Take the stored generations and use the second slice to compute the loss\n # ...\n if dataset is None:\n dataset = self.train_dataset\n return RepeatSampler(\n data_source=dataset,\n mini_repeat_count=self.num_generations,\n batch_size=self.args.generation_batch_size // self.num_generations,\n repeat_count=self.num_iterations * self.args.steps_per_generation,\n shuffle=self.shuffle_dataset,\n seed=self.args.seed,\n )\n\n def _get_eval_sampler(self, eval_dataset) -> Sampler:\n # See _get_train_sampler for an explanation of the sampler.\n return RepeatSampler(\n data_source=eval_dataset,\n mini_repeat_count=self.num_generations_eval,\n seed=self.args.seed,\n )\n\n @profiling_decorator\n def _get_last_hidden_state(\n self,\n unwrapped_model,\n input_ids,\n attention_mask,\n logits_to_keep,\n pixel_values=None,\n image_grid_thw=None,\n pixel_attention_mask=None,\n image_sizes=None,\n ):\n if is_peft_model(unwrapped_model):\n unwrapped_model = unwrapped_model.base_model.model\n\n # Build model inputs - check if the model supports logits_to_keep (some models and VLMs don't)\n model_inputs = {\"input_ids\": input_ids, \"attention_mask\": attention_mask}\n\n # For Qwen models:\n if image_grid_thw is not None and pixel_values is not None:\n model_inputs[\"image_grid_thw\"] = image_grid_thw\n # For Gemma, SmolVLM2, LLaVa-Next etc.:\n if pixel_values is not None:\n model_inputs[\"pixel_values\"] = pixel_values\n # For SmolVLM2\n if pixel_attention_mask is not None:\n model_inputs[\"pixel_attention_mask\"] = pixel_attention_mask\n # For LLaVa-Next\n if image_sizes is not None:\n model_inputs[\"image_sizes\"] = image_sizes\n\n # Only add logits_to_keep if the model supports it\n if \"logits_to_keep\" in self.model_kwarg_keys:\n # We add 1 to `logits_to_keep` because the last logits of the sequence is later excluded\n model_inputs[\"logits_to_keep\"] = logits_to_keep + 1\n\n model_inputs[\"use_cache\"] = False # only used in generation; set False to suppress warnings\n\n last_hidden_state = unwrapped_model.model(**model_inputs).last_hidden_state\n # Exclude the last value: it corresponds to the next token pred\n last_hidden_state = last_hidden_state[:, :-1, :] # (B, L-1, H)\n # Only keep the last logits_to_keep. For model that support logits_to_keep, this is a no-op.\n last_hidden_state = last_hidden_state[:, -logits_to_keep:, :] # (B, logits_to_keep, H)\n return last_hidden_state\n\n def get_high_entropy_mask(self, entropies: torch.Tensor, mask: torch.Tensor, threshold: float) -> torch.Tensor:\n \"\"\"\n Returns a binary mask identifying tokens whose entropy exceeds a given quantile threshold.\n\n Args:\n entropies (`torch.Tensor`):\n Tensor of shape (batch_size, seq_len) with per-token entropy values.\n mask (`torch.Tensor`):\n Binary mask of the same shape as `entropies`, where `1` indicates valid tokens and `0` padding.\n threshold (`float`):\n Quantile threshold between `0.0` and `1.0` to select high-entropy tokens.\n\n Returns:\n `torch.Tensor`:\n Boolean mask of shape (batch_size, seq_len), where `True` indicates tokens with entropy >= threshold\n and `False` otherwise.\n \"\"\"\n local = entropies[mask.bool()].float()\n\n # Use a negative pad_value as a sentinel because entropy values are always >= 0.\n # This guarantees that the sentinel cannot collide with any real entropy value.\n pad_value = -1e9\n\n # Pad across processes so that every rank has the same tensor length\n padded = self.accelerator.pad_across_processes(local, dim=0, pad_index=pad_value)\n gathered = self.accelerator.gather(padded)\n\n # Drop sentinel values (safe because no entropy can be negative)\n gathered = gathered[gathered != pad_value]\n\n if gathered.numel() == 0:\n return torch.zeros_like(entropies, dtype=torch.bool)\n\n entropy_threshold = torch.quantile(gathered, threshold)\n masked_entropies = entropies * mask.float()\n entropy_mask = masked_entropies >= entropy_threshold\n return entropy_mask & mask.bool() # ensure padding tokens are always masked out\n\n @profiling_decorator\n def _get_per_token_logps_and_entropies(\n self,\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n batch_size=None,\n compute_entropy=False,\n pixel_values=None,\n image_grid_thw=None,\n num_images=None,\n pixel_attention_mask=None,\n image_sizes=None,\n token_type_ids=None,\n mm_token_type_ids=None,\n ) -> dict[str, torch.Tensor | None]:\n \"\"\"Compute log-probs and (optionally) entropies for each token.\"\"\"\n batch_size = batch_size or input_ids.size(0) # Chunk inputs into smaller batches to reduce memory peak\n all_logps = []\n all_entropies = []\n for start in range(0, input_ids.size(0), batch_size):\n input_ids_batch = input_ids[start : start + batch_size]\n attention_mask_batch = attention_mask[start : start + batch_size]\n\n # Build model inputs - check if the model supports logits_to_keep (some models and VLMs don't)\n model_inputs = {\"input_ids\": input_ids_batch, \"attention_mask\": attention_mask_batch}\n if image_grid_thw is not None and pixel_values is not None:\n rows_per_image = image_grid_thw.prod(dim=-1)\n rows_per_sample = torch.split(rows_per_image, num_images)\n rows_per_sample = torch.stack([s.sum() for s in rows_per_sample])\n cum_rows = torch.cat([torch.tensor([0], device=rows_per_sample.device), rows_per_sample.cumsum(0)])\n row_start, row_end = cum_rows[start].item(), cum_rows[start + batch_size].item()\n model_inputs[\"pixel_values\"] = pixel_values[row_start:row_end]\n cum_imgs = torch.tensor([0] + num_images).cumsum(0)\n img_start, img_end = cum_imgs[start], cum_imgs[start + batch_size]\n model_inputs[\"image_grid_thw\"] = image_grid_thw[img_start:img_end]\n elif pixel_values is not None:\n model_inputs[\"pixel_values\"] = pixel_values[start : start + batch_size]\n if pixel_attention_mask is not None:\n model_inputs[\"pixel_attention_mask\"] = pixel_attention_mask[start : start + batch_size]\n if image_sizes is not None:\n model_inputs[\"image_sizes\"] = image_sizes[start : start + batch_size]\n if token_type_ids is not None:\n model_inputs[\"token_type_ids\"] = token_type_ids[start : start + batch_size]\n if mm_token_type_ids is not None:\n model_inputs[\"mm_token_type_ids\"] = mm_token_type_ids[start : start + batch_size]\n\n # Only add logits_to_keep if the model supports it\n if \"logits_to_keep\" in self.model_kwarg_keys:\n # We add 1 to `logits_to_keep` because the last logits of the sequence is later excluded\n model_inputs[\"logits_to_keep\"] = logits_to_keep + 1\n\n model_inputs[\"use_cache\"] = False # only used in generation; set False to suppress warnings\n\n logits = model(**model_inputs).logits\n # Exclude the last value: it corresponds to the next token pred\n logits = logits[:, :-1, :] # (B, L-1, H)\n # Only keep the last logits_to_keep. For model that support logits_to_keep, this is a no-op.\n logits = logits[:, -logits_to_keep:, :] # (B, logits_to_keep, H)\n # Divide logits by sampling temperature.\n # See https://huggingface.co/blog/the_n_implementation_details_of_rlhf_with_ppo#policy-training-implementation-details\n logits.div_(self.temperature)\n completion_ids = input_ids_batch[:, -logits_to_keep:]\n logps = selective_log_softmax(logits, completion_ids) # compute logprobs\n all_logps.append(logps)\n\n if compute_entropy:\n with torch.no_grad():\n entropies = entropy_from_logits(logits)\n all_entropies.append(entropies)\n\n logps = torch.cat(all_logps, dim=0)\n entropies = torch.cat(all_entropies, dim=0) if compute_entropy else None\n return logps, entropies\n\n def training_step(self, model, inputs, num_items_in_batch):\n time_before = time.perf_counter()\n output = super().training_step(model, inputs, num_items_in_batch)\n self._step += 1\n time_after = time.perf_counter()\n self._current_train_step_time += time_after - time_before\n if self._step % self.current_gradient_accumulation_steps == 0:\n self._metrics[\"train\"][\"step_time\"].append(self._current_train_step_time)\n self._current_train_step_time = 0.0\n return output\n\n @profiling_decorator\n def _prepare_inputs(self, generation_batch: dict[str, torch.Tensor | Any]) -> dict[str, torch.Tensor | Any]:\n # Prepares inputs for model training/evaluation by managing completion generation and batch handling.\n # During training:\n # - Receives the local generation batch (Per-GPU batch size ร— steps per generation)\n # from the modified training dataloader instead of the standard local batch\n # - Generates completions once for the entire generation batch and splits it into batches of size\n # `per_device_train_batch_size`\n # - Buffers these completions and returns the appropriate slice for the current accumulation step\n # - Optimizes by regenerating completions only periodically (every steps_per_generation * num_iterations)\n # During evaluation:\n # - The input is treated as a standard local batch (no accumulation, no multiple iterations)\n # - Completions are generated for each batch without buffering or reuse\n # Returns a single local batch in both cases.\n\n mode = \"train\" if self.model.training else \"eval\"\n if mode == \"train\":\n generate_every = self.args.steps_per_generation * self.num_iterations\n if self._step % generate_every == 0 or self._buffered_inputs is None:\n # self._buffered_inputs=None can occur when resuming from a checkpoint\n generation_batch = self._generate_and_score_completions(generation_batch)\n generation_batch = split_pixel_values_by_grid(generation_batch)\n generation_batch = shuffle_sequence_dict(generation_batch)\n generation_batches = split_tensor_dict(generation_batch, self.args.steps_per_generation)\n self._buffered_inputs = [unsplit_pixel_values_by_grid(batch) for batch in generation_batches]\n inputs = self._buffered_inputs[self._step % self.args.steps_per_generation]\n else:\n # In evaluation, there is neither batch grouping for generation, nor multiple iterations, hence\n # local generation batch == local eval batch\n inputs = self._generate_and_score_completions(generation_batch)\n return inputs\n\n def _log_completion_extra(self, column: str, values: list):\n \"\"\"\n Log extra columns to the completions table. Called from reward functions via the `log_extra` kwarg.\n\n Args:\n column (`str`):\n Name of the column to add.\n values (`list`):\n Values for the column, one per sample in the batch.\n \"\"\"\n self._pending_extra_logs[column].extend(values)\n\n def _log_metric(self, name: str, value: float):\n \"\"\"\n Log a scalar metric from a reward function. Called via the `log_metric` kwarg. Values are averaged over each\n logging step and reported alongside built-in metrics like `kl` and `entropy`.\n\n Args:\n name (`str`):\n Name of the metric.\n value (`float`):\n Scalar value for this batch.\n \"\"\"\n self._pending_metrics[name].append(value)\n\n @profiling_decorator\n def _calculate_rewards(self, inputs, prompts, completions, completion_ids_list):\n device = self.accelerator.device\n rewards_per_func = torch.zeros(len(prompts), len(self.reward_funcs), device=device)\n\n # Repeat all input columns (but \"prompt\", \"completion\", and \"completion_ids\") to match the num of generations\n keys = [key for key in inputs[0] if key not in [\"prompt\", \"completion\", \"completion_ids\"]]\n reward_kwargs = {key: [example[key] for example in inputs] for key in keys}\n\n # This allows for dynamic reward shaping based on training progress.\n reward_kwargs[\"trainer_state\"] = self.state\n\n # Allow reward functions to log extra columns to the completions table.\n reward_kwargs[\"log_extra\"] = self._log_completion_extra\n\n # Allow reward functions to log additional scalar metrics.\n reward_kwargs[\"log_metric\"] = self._log_metric\n\n async_funcs_info = [] # async custom functions for asyncio.gather\n\n for i, (reward_func, reward_processing_class, reward_func_name) in enumerate(\n zip(self.reward_funcs, self.reward_processing_classes, self.reward_func_names, strict=True)\n ):\n if isinstance(reward_func, nn.Module): # Module (no PretrainedModel) for compat with compiled models\n with profiling_context(self, reward_func_name):\n if is_conversational(inputs[0]):\n messages = [{\"messages\": p + c} for p, c in zip(prompts, completions, strict=True)]\n texts = [\n apply_chat_template(x, reward_processing_class, **self.chat_template_kwargs)[\"text\"]\n for x in messages\n ]\n else:\n texts = [p + c for p, c in zip(prompts, completions, strict=True)]\n reward_inputs = reward_processing_class(\n text=texts, return_tensors=\"pt\", padding=True, padding_side=\"right\", add_special_tokens=False\n )\n reward_inputs = super()._prepare_inputs(reward_inputs)\n with torch.inference_mode():\n rewards_per_func[:, i] = reward_func(**reward_inputs).logits[:, 0] # Shape (B*G,)\n elif inspect.iscoroutinefunction(reward_func): # Separate async reward funcs to run them in parallel later\n async_funcs_info.append((i, reward_func, reward_func_name))\n else:\n # Run synchronous reward function\n with profiling_context(self, reward_func_name):\n if self.environments is not None:\n reward_kwargs[\"environments\"] = self.environments\n output_reward_func = reward_func(\n prompts=prompts, completions=completions, completion_ids=completion_ids_list, **reward_kwargs\n )\n # Convert None values to NaN\n output_reward_func = [reward if reward is not None else torch.nan for reward in output_reward_func]\n rewards_per_func[:, i] = torch.tensor(output_reward_func, dtype=torch.float32, device=device)\n\n # Execute async custom functions in parallel using asyncio.gather\n if async_funcs_info:\n\n async def _invoke_async(index, func, func_name):\n with profiling_context(self, func_name):\n output = await func(\n prompts=prompts, completions=completions, completion_ids=completion_ids_list, **reward_kwargs\n )\n output = [r if r is not None else torch.nan for r in output]\n return index, output\n\n async def _run_async_funcs():\n coros = [_invoke_async(i, func, func_name) for (i, func, func_name) in async_funcs_info]\n return await asyncio.gather(*coros)\n\n async_results = asyncio.run_coroutine_threadsafe(_run_async_funcs(), self.async_loop).result()\n for idx, output_reward_func in async_results:\n rewards_per_func[:, idx] = torch.tensor(output_reward_func, dtype=torch.float32, device=device)\n\n # If all reward functions return None for a given row, issue a detailed warning\n if torch.isnan(rewards_per_func).all(dim=1).any():\n nan_row_idx = torch.isnan(rewards_per_func).all(dim=1).nonzero(as_tuple=True)[0][0]\n row_reward_kwargs = {\n key: value[nan_row_idx]\n for key, value in reward_kwargs.items()\n if key not in (\"trainer_state\", \"log_extra\", \"log_metric\")\n }\n row_reward_kwargs[\"prompt\"] = prompts[nan_row_idx]\n row_reward_kwargs[\"completion\"] = completions[nan_row_idx]\n logger.warning(\n f\"All reward functions returned None for the following kwargs:\\n{row_reward_kwargs}\\n\"\n \"Please ensure that at least one reward function returns a valid reward.\"\n )\n\n # Gather the reward per function: this part is crucial, because the rewards are normalized per group and the\n # completions may be distributed across processes\n rewards_per_func = gather(rewards_per_func)\n return rewards_per_func\n\n def _tokenize_prompts(self, prompts: list):\n \"\"\"Tokenize prompts and extract images/multimodal fields for generation.\"\"\"\n if is_conversational({\"prompt\": prompts[0]}):\n # Extract images from messages for VLM support\n images = []\n has_images = False\n for prompt in prompts:\n prompt_images = []\n for message in prompt:\n if isinstance(message[\"content\"], list):\n for part in message[\"content\"]:\n if part[\"type\"] == \"image\":\n prompt_images.append(part[\"image\"])\n has_images = True\n images.append(prompt_images if prompt_images else None)\n images = images if has_images else None\n\n # We pass padding=True to work around a bug introduced in transformers 5.2.0 in some processors\n # (e.g. Qwen2.5-VL) that crash on batched unpadded input. We then unpad input_ids using attention_mask.\n # See: https://github.com/huggingface/transformers/issues/44514\n tokenized = self.processing_class.apply_chat_template(\n conversation=prompts,\n tools=self.tools,\n chat_template=self.chat_template,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n padding=True,\n **self.chat_template_kwargs,\n )\n # Unpad input_ids: remove padding tokens using attention_mask to get per-sequence lists\n prompt_ids = [\n [tok for tok, m in zip(ids, mask, strict=True) if m]\n for ids, mask in zip(tokenized[\"input_ids\"], tokenized[\"attention_mask\"], strict=True)\n ]\n # For VLMs, the processor returns extra multimodal fields (pixel_values, image_grid_thw, etc.)\n multimodal_fields = {k: v for k, v in tokenized.items() if k not in (\"input_ids\", \"attention_mask\")}\n else:\n prompt_ids = self.processing_class(text=prompts)[\"input_ids\"]\n images = None\n multimodal_fields = {}\n return prompt_ids, images, multimodal_fields\n\n def _generate_single_turn(self, prompt_ids, images, multimodal_fields):\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n\n # Generate completions using either vLLM or regular generation\n if self.use_vllm:\n # Sync weights if training step changed\n if self.state.global_step != self._last_loaded_step:\n with profiling_context(self, \"sync_weights\"):\n self.vllm_generation.sync_weights()\n self._last_loaded_step = self.state.global_step\n\n # Generate using vLLM with raw token IDs\n num_generations = self.num_generations if mode == \"train\" else self.num_generations_eval\n _, completion_ids, logprobs, _ = self.vllm_generation.generate(\n prompts=prompt_ids,\n images=images,\n num_generations=num_generations,\n profiler=profiling_context(self, \"vLLM.generate\"),\n )\n # vLLM returns per-token top-k logprobs; keep only the top-1 (sampled token) logprob\n logprobs = [[lp[0] for lp in seq] for seq in logprobs]\n\n elif self.use_transformers_paged:\n with (\n profiling_context(self, \"transformers.generate_batch\"),\n unwrap_model_for_generation(\n self.model_wrapped, self.accelerator, gather_deepspeed3_params=self.args.ds3_gather_for_generation\n ) as unwrapped_model,\n torch.no_grad(),\n FSDP.summon_full_params(self.model_wrapped, recurse=False) if self.is_fsdp_enabled else nullcontext(),\n ):\n # Cast to the appropriate dtype based on training configuration\n if self.args.bf16:\n unwrapped_model.to(torch.bfloat16)\n elif self.args.fp16:\n unwrapped_model.to(torch.float16)\n if self.args.cast_lm_head_to_fp32:\n unwrapped_model.lm_head.to(torch.float32)\n with torch.inference_mode():\n # Continuous batching API expects 'inputs' arg only\n all_outputs = unwrapped_model.generate_batch(\n prompt_ids, generation_config=self.generation_config, progress_bar=False\n )\n unwrapped_model.train() # restore training mode, as generate_batch forces eval mode\n completion_ids = [output.generated_tokens for output in all_outputs.values()]\n logprobs = None # not used in this case\n\n else:\n # Regular generation path: left-pad token IDs into tensors\n prompt_tensors = [torch.tensor(ids) for ids in prompt_ids]\n padded_ids = pad(prompt_tensors, padding_value=self.pad_token_id, padding_side=\"left\")\n attention_mask = pad([torch.ones_like(t) for t in prompt_tensors], padding_value=0, padding_side=\"left\")\n generate_inputs = {\"input_ids\": padded_ids, \"attention_mask\": attention_mask}\n # For VLMs, include multimodal fields as tensors (pixel_values, image_grid_thw, etc.)\n for k, v in multimodal_fields.items():\n if isinstance(v, torch.Tensor):\n generate_inputs[k] = v\n elif isinstance(v, list) and v and isinstance(v[0], list):\n # Per-token field (e.g., token_type_ids): left-pad like input_ids\n generate_inputs[k] = pad([torch.tensor(x) for x in v], padding_value=0, padding_side=\"left\")\n else:\n generate_inputs[k] = torch.tensor(np.array(v))\n generate_inputs = super()._prepare_inputs(generate_inputs)\n\n with (\n profiling_context(self, \"transformers.generate\"),\n unwrap_model_for_generation(\n self.model_wrapped,\n self.accelerator,\n gather_deepspeed3_params=self.args.ds3_gather_for_generation,\n generation_kwargs=self.generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as unwrapped_model,\n torch.no_grad(),\n FSDP.summon_full_params(self.model_wrapped, recurse=False) if self.is_fsdp_enabled else nullcontext(),\n ):\n prompt_completion_ids = unwrapped_model.generate(\n **generate_inputs, generation_config=self.generation_config, disable_compile=True\n )\n # Compute prompt length and extract completion ids\n prompt_length = generate_inputs[\"input_ids\"].size(1)\n completion_ids = prompt_completion_ids[:, prompt_length:]\n\n # Mask everything after the first EOS token\n is_eos = completion_ids == self.eos_token_id\n eos_idx = torch.full((is_eos.size(0),), is_eos.size(1), dtype=torch.long, device=device)\n eos_idx[is_eos.any(dim=1)] = is_eos.int().argmax(dim=1)[is_eos.any(dim=1)]\n sequence_indices = torch.arange(is_eos.size(1), device=device).expand(is_eos.size(0), -1)\n completion_mask = (sequence_indices <= eos_idx.unsqueeze(1)).int()\n completion_ids = [\n c[m].tolist() for c, m in zip(completion_ids.cpu(), completion_mask.bool().cpu(), strict=True)\n ]\n logprobs = None # not used in this case\n\n return completion_ids, logprobs\n\n def _get_tool_suffix_ids(self, tool_messages):\n \"\"\"Get token IDs for tool result formatting by using a minimal dummy conversation.\"\"\"\n dummy_messages = [{\"role\": \"user\", \"content\": \"dummy\"}, {\"role\": \"assistant\", \"content\": \"dummy\"}]\n prefix_ids = self.processing_class.apply_chat_template(\n dummy_messages,\n add_generation_prompt=False,\n chat_template=self.chat_template,\n return_dict=False,\n **self.chat_template_kwargs,\n )\n full_ids = self.processing_class.apply_chat_template(\n dummy_messages + tool_messages,\n add_generation_prompt=True,\n chat_template=self.chat_template,\n return_dict=False,\n **self.chat_template_kwargs,\n )\n if not full_ids[: len(prefix_ids)] == prefix_ids:\n raise ValueError(\"Unexpected tokenization: the prefix IDs are not a prefix of the full IDs.\")\n return full_ids[len(prefix_ids) :]\n\n def _tool_call_loop(self, prompts, prompt_ids, completion_ids, completions, logprobs, images, multimodal_fields):\n # Tool execution loop: execute tools, then regenerate completions with tool results appended to the prompt\n tool_calls = [completion[0].get(\"tool_calls\") for completion in completions]\n idxs_with_tool = [idx for idx, tool_call in enumerate(tool_calls) if tool_call]\n tool_calls = [tool_calls[idx] for idx in idxs_with_tool]\n tool_mask = [[1] * len(ids) for ids in completion_ids] # 0 for tool result tokens, 1 elsewhere\n tool_call_count = 0\n tool_failure_count = 0\n iteration_num = 0\n while idxs_with_tool and iteration_num < self.max_tool_calling_iterations:\n prompt_completion_tools = [prompts[i] for i in idxs_with_tool] # select only prompts that need tool calls\n\n # Call the tools, and build the new prompt for generation\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n tool_call_list = tool_calls[idx]\n prompt_completion_tool = prompt_completion_tools[idx]\n sync_tool_dict = self._sync_tool_dicts[idx_with_tool]\n async_tool_dict = self._async_tool_dicts[idx_with_tool]\n # Append the last assistant message (which triggered tool_calls) to the prompt\n prompt_completion_tool.append(completions[idx_with_tool][-1])\n async_coros = []\n tool_call_results = []\n for tool_call in tool_call_list:\n tool_call_count += 1\n if tool_call[\"type\"] == \"function\":\n function = tool_call[\"function\"]\n name = function[\"name\"]\n try:\n if name in sync_tool_dict:\n tool_call_results.append((name, sync_tool_dict[name](**function[\"arguments\"])))\n elif name in async_tool_dict:\n async_coros.append((name, async_tool_dict[name](**function[\"arguments\"])))\n else:\n raise ValueError(f\"Tool {name} not found.\")\n except Exception as e:\n tool_failure_count += 1\n result = {\"error\": str(e)}\n tool_call_results.append((name, result))\n else:\n tool_failure_count += 1\n name = tool_call.get(\"name\", \"unknown\")\n tool_call_results.append((name, {\"error\": f\"Unsupported tool call type: {tool_call['type']}\"}))\n\n if async_coros:\n\n async def _run_async_tools(async_coros):\n coros = [coro for _, coro in async_coros]\n results = await asyncio.gather(*coros, return_exceptions=True)\n return [(name, result) for (name, _), result in zip(async_coros, results, strict=False)]\n\n async_results = asyncio.run_coroutine_threadsafe(\n _run_async_tools(async_coros), self.async_loop\n ).result()\n\n for name, result in async_results:\n if isinstance(result, Exception):\n tool_failure_count += 1\n tool_call_results.append((name, {\"error\": str(result)}))\n else:\n tool_call_results.append((name, result))\n\n for name, result in tool_call_results:\n tool_message = {\"role\": \"tool\", \"name\": name, \"content\": str(result)}\n prompt_completion_tool.append(tool_message)\n completions[idx_with_tool].append(tool_message)\n\n # Build token IDs by concatenation: prompt + completion + tool_suffix.\n prompt_completion_tool_ids = []\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n # Extract trailing tool messages from completions\n tool_messages = []\n for message in reversed(completions[idx_with_tool]):\n if message[\"role\"] == \"tool\":\n tool_messages.insert(0, message)\n else:\n break\n suffix_ids = self._get_tool_suffix_ids(tool_messages)\n prompt_completion_tool_ids.append(\n prompt_ids[idx_with_tool] + completion_ids[idx_with_tool] + suffix_ids\n )\n\n # Filter samples whose length exceeds max allowed length. This is important, because both\n # vLLM and transformers will error out if the input is longer than the model's max length.\n if self.use_vllm and self.vllm_mode == \"colocate\":\n max_model_len = self.vllm_generation.llm.llm_engine.model_config.max_model_len\n elif not self.use_vllm:\n max_model_len = self.model.config.max_position_embeddings\n else:\n raise NotImplementedError(\n f\"Unsupported mode detected: use_vllm={self.use_vllm}, vllm_mode={self.vllm_mode}\"\n )\n overlong = [len(pct) >= max_model_len for pct in prompt_completion_tool_ids]\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n if overlong[idx]:\n prompt_length = len(prompt_ids[idx_with_tool])\n ct = prompt_completion_tool_ids[idx][prompt_length : prompt_length + self.max_completion_length]\n completion_ids[idx_with_tool] = ct\n tool_mask[idx_with_tool] += [1] * (len(ct) - len(tool_mask[idx_with_tool]))\n if logprobs is not None:\n logprobs[idx_with_tool] += [0.0] * (len(ct) - len(logprobs[idx_with_tool]))\n # Keep only non-overlong items for further processing\n idxs_with_tool = [idx for idx, o in zip(idxs_with_tool, overlong, strict=True) if not o]\n prompt_completion_tools = [pct for pct, o in zip(prompt_completion_tools, overlong, strict=True) if not o]\n prompt_completion_tool_ids = [\n pct for pct, o in zip(prompt_completion_tool_ids, overlong, strict=True) if not o\n ]\n if not idxs_with_tool:\n break # all overlong, exit tool loop\n\n # Filter images and multimodal fields to match the current subset (index into full batch)\n loop_images = [images[i] for i in idxs_with_tool] if images else None\n loop_multimodal_fields = (\n {k: [v[i] for i in idxs_with_tool] for k, v in multimodal_fields.items()} if multimodal_fields else {}\n )\n\n # Generate new completions after tool execution (using concatenated IDs, no re-tokenization)\n post_tool_ids, post_tool_logprobs = self._generate_single_turn(\n prompt_completion_tool_ids, loop_images, loop_multimodal_fields\n )\n\n # Truncate so that pct[len(prompt_ids[idx]) :] + post_tool does not exceed max_completion_length\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n prompt_len = len(prompt_ids[idx_with_tool])\n completion_tool_ids = prompt_completion_tool_ids[idx][prompt_len:]\n excess_length = len(completion_tool_ids) + len(post_tool_ids[idx]) - self.max_completion_length\n if excess_length > 0:\n # If exceeding max length, truncate post_tool_ids\n post_tool_ids[idx] = post_tool_ids[idx][:-excess_length]\n if logprobs is not None:\n post_tool_logprobs[idx] = post_tool_logprobs[idx][:-excess_length]\n excess_length = len(completion_tool_ids) + len(post_tool_ids[idx]) - self.max_completion_length\n if excess_length > 0:\n # If still exceeding max length, truncate completion_tool_ids as well\n prompt_completion_tool_ids[idx] = prompt_completion_tool_ids[idx][:-excess_length]\n\n # Update tool_mask: the tool result should be 0 and the post-tool 1\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n prompt_completion_tool_length = len(prompt_completion_tool_ids[idx])\n prompt_length = len(prompt_ids[idx_with_tool])\n completion_length = len(completion_ids[idx_with_tool])\n post_tool_length = len(post_tool_ids[idx])\n tool_length = prompt_completion_tool_length - prompt_length - completion_length\n tool_mask[idx_with_tool] += [0] * tool_length + [1] * post_tool_length\n if logprobs is not None:\n logprobs[idx_with_tool] += [0.0] * tool_length + post_tool_logprobs[idx]\n\n # Update completion_ids with the new completions (after tool execution)\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n prompt_length = len(prompt_ids[idx_with_tool])\n pct = prompt_completion_tool_ids[idx] # = prompt-completion-tool\n completion_ids[idx_with_tool] = pct[prompt_length:] + post_tool_ids[idx]\n\n # Decode post-tool completions\n post_tool_completions = [\n parse_response(self.processing_class, ids) if ids else {} for ids in post_tool_ids\n ]\n\n # Add post-tool completions to the existing completions\n for idx in range(len(idxs_with_tool)):\n idx_with_tool = idxs_with_tool[idx]\n if post_tool_completions[idx]: # {} if post-tool completions completely truncated\n completions[idx_with_tool].append(post_tool_completions[idx])\n\n # Check for further tool calls\n tool_calls = [completion.get(\"tool_calls\") for completion in post_tool_completions]\n idxs_with_tool = [idx for idx, tool_call in zip(idxs_with_tool, tool_calls, strict=True) if tool_call]\n tool_calls = [tool_call for tool_call in tool_calls if tool_call]\n iteration_num += 1\n return tool_mask, completions, completion_ids, logprobs, tool_call_count, tool_failure_count\n\n def _generate(self, prompts: list):\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n\n # Copy the prompts to avoid modifying the original list\n prompts = copy.deepcopy(prompts)\n\n if self.rollout_func is not None:\n # Keep vLLM weights in sync for custom rollouts that rely on vLLM utilities.\n if self.use_vllm and self.state.global_step != self._last_loaded_step:\n with profiling_context(self, \"sync_weights\"):\n self.vllm_generation.sync_weights()\n self._last_loaded_step = self.state.global_step\n\n # Pass prompts to rollout_func preserving structured messages.\n # Chat templating must happen inside rollout_func, at the backend boundary, so that\n # multimodal content (images, typed content blocks) is not lost before rollout logic runs.\n output = self.rollout_func(prompts, self)\n required_keys = {\"prompt_ids\", \"completion_ids\", \"logprobs\"}\n missing_keys = required_keys - output.keys()\n if missing_keys:\n missing_keys_list = sorted(missing_keys)\n raise ValueError(f\"rollout_func must return keys {missing_keys_list} in its output dict.\")\n extra_fields = {k: v for k, v in output.items() if k not in required_keys}\n prompt_ids, completion_ids, logprobs = output[\"prompt_ids\"], output[\"completion_ids\"], output[\"logprobs\"]\n else:\n prompt_ids, images, multimodal_fields = self._tokenize_prompts(prompts)\n completion_ids, logprobs = self._generate_single_turn(prompt_ids, images, multimodal_fields)\n extra_fields = {}\n\n # Decode completions. It's important to use `parse_response` when possible, because it handles tool calls.\n if is_conversational({\"prompt\": prompts[0]}):\n if (\n Version(transformers.__version__) >= Version(\"5.0.0\") # parse_response added in v5\n and isinstance(self.processing_class, PreTrainedTokenizerBase) # doesn't work with processors\n and hasattr(self.processing_class, \"response_schema\") # attribute not set by default for now\n and self.processing_class.response_schema is not None # only works if the tokenizer has a schema\n ):\n completions = [[parse_response(self.processing_class, ids)] for ids in completion_ids]\n else:\n contents = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n completions = [[{\"role\": \"assistant\", \"content\": content}] for content in contents]\n else:\n completions = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n\n # Extract tool calls from the completions and (possibly) execute them\n if self.tools:\n (\n tool_mask,\n completions,\n completion_ids,\n logprobs,\n tool_call_count,\n tool_failure_count,\n ) = self._tool_call_loop(\n prompts, prompt_ids, completion_ids, completions, logprobs, images, multimodal_fields\n )\n else:\n # Support custom env_mask from rollout_func (e.g., for environment feedback masking)\n # Internally treated as tool_mask - marks model tokens (1) vs external tokens (0)\n tool_mask = extra_fields.pop(\"env_mask\", None)\n\n # Get completion length per sequence, used for logging\n prompt_lengths = torch.tensor([len(ids) for ids in prompt_ids], device=device)\n if tool_mask is not None: # count only model-generated tokens (tool_mask=1)\n completion_lengths = torch.tensor([sum(mask) for mask in tool_mask], device=device)\n else:\n completion_lengths = torch.tensor([len(ids) for ids in completion_ids], device=device)\n agg_prompt_lengths = self.accelerator.gather(prompt_lengths)\n agg_completion_lengths = self.accelerator.gather(completion_lengths)\n total_prompt_tokens = agg_prompt_lengths.sum()\n total_completion_tokens = agg_completion_lengths.sum() # = num_items_in_batch, required for the DAPO loss\n\n # Log the metrics\n if mode == \"train\":\n self.state.num_input_tokens_seen += (total_prompt_tokens + total_completion_tokens).item()\n self._metrics[mode][\"num_tokens\"] = [self.state.num_input_tokens_seen]\n\n # Log completion lengths, mean, min, max\n self._metrics[mode][\"completions/mean_length\"].append(agg_completion_lengths.float().mean().item())\n self._metrics[mode][\"completions/min_length\"].append(agg_completion_lengths.float().min().item())\n self._metrics[mode][\"completions/max_length\"].append(agg_completion_lengths.float().max().item())\n\n # Identify sequences that terminated with EOS and log their lengths\n eos_and_pad = [self.eos_token_id, self.pad_token_id]\n is_truncated = torch.tensor([ids[-1] not in eos_and_pad for ids in completion_ids], device=device)\n agg_is_truncated = self.accelerator.gather(is_truncated)\n self._metrics[mode][\"completions/clipped_ratio\"].append(agg_is_truncated.float().mean().item())\n term_completion_lengths = agg_completion_lengths[~agg_is_truncated]\n if len(term_completion_lengths) == 0: # edge case where no terminated sequences are found\n term_completion_lengths = torch.zeros(1, device=device)\n self._metrics[mode][\"completions/mean_terminated_length\"].append(term_completion_lengths.float().mean().item())\n self._metrics[mode][\"completions/min_terminated_length\"].append(term_completion_lengths.float().min().item())\n self._metrics[mode][\"completions/max_terminated_length\"].append(term_completion_lengths.float().max().item())\n\n if self.tools:\n agg_tool_call_count = self.accelerator.gather(torch.tensor(tool_call_count, device=device)).sum()\n tool_call_frequency = (agg_tool_call_count / len(agg_prompt_lengths)).item()\n self._metrics[mode][\"tools/call_frequency\"].append(tool_call_frequency)\n agg_tool_failure_count = self.accelerator.gather(torch.tensor(tool_failure_count, device=device)).sum()\n failure_frequency = (\n (agg_tool_failure_count / agg_tool_call_count).item() if agg_tool_call_count > 0 else 0.0\n )\n self._metrics[mode][\"tools/failure_frequency\"].append(failure_frequency)\n\n return (\n prompt_ids,\n completion_ids,\n tool_mask,\n completions,\n total_completion_tokens,\n logprobs,\n extra_fields,\n )\n\n def _generate_and_score_completions(\n self, inputs: list[dict[str, torch.Tensor | Any]]\n ) -> dict[str, torch.Tensor | Any]:\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n\n prompts = [x[\"prompt\"] for x in inputs]\n\n if self.environments:\n for prompt, environment, reset_kwargs in zip(prompts, self.environments, inputs, strict=True):\n observation = environment.reset(**reset_kwargs)\n if observation is None:\n continue\n prompt[-1][\"content\"] += observation\n\n if \"images\" in inputs[0]:\n images = [example.get(\"images\") for example in inputs]\n elif \"image\" in inputs[0]:\n images = [[example.get(\"image\")] if example.get(\"image\") is not None else None for example in inputs]\n else:\n images = None\n # Transformers requires at least one image in the batch, otherwise it throws an error\n if images is not None and all(img_list == [] for img_list in images):\n images = None\n\n # If the prompts are conversational and the inputs contain images, we need to convert the prompts from\n # [{\"role\": \"user\", \"content\": \"What color is the sky?\"}] to\n # [{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": }, {\"type\": \"text\", \"text\": \"What color is the sky?\"}]}]\n if images is not None:\n if not is_conversational(inputs[0]):\n raise ValueError(\n \"Multimodal training requires conversational prompts. It looks like the dataset contains \"\n \"non-conversational inputs, likely because a chat template was applied before passing the dataset \"\n \"to the trainer. Please provide the raw conversational prompts and let the trainer apply the chat \"\n \"template internally.\"\n )\n prompts = [\n prepare_multimodal_messages(prompt, image_list)\n for prompt, image_list in zip(prompts, images, strict=True)\n ]\n\n (\n prompt_ids_list,\n completion_ids_list,\n tool_mask_list,\n completions,\n num_items_in_batch,\n sampling_per_token_logps_list,\n extra_fields,\n ) = self._generate(prompts)\n\n # Convert lists of token IDs to padded tensors\n prompt_ids = [torch.tensor(ids) for ids in prompt_ids_list]\n prompt_mask = [torch.ones_like(ids, dtype=torch.long) for ids in prompt_ids]\n prompt_ids = pad(\n prompt_ids,\n padding_value=self.pad_token_id,\n padding_side=\"left\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n prompt_mask = pad(\n prompt_mask, padding_value=0, padding_side=\"left\", pad_to_multiple_of=self.pad_to_multiple_of\n ).to(device=device)\n completion_ids = [torch.tensor(ids) for ids in completion_ids_list]\n completion_mask = [torch.ones_like(ids, dtype=torch.long) for ids in completion_ids]\n completion_ids = pad(\n completion_ids,\n padding_value=self.pad_token_id,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n completion_mask = pad(\n completion_mask, padding_value=0, padding_side=\"right\", pad_to_multiple_of=self.pad_to_multiple_of\n ).to(device=device)\n if sampling_per_token_logps_list is not None:\n sampling_per_token_logps = [torch.tensor(logps) for logps in sampling_per_token_logps_list]\n sampling_per_token_logps = pad(\n sampling_per_token_logps,\n padding_value=0.0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n else:\n sampling_per_token_logps = None\n if tool_mask_list is not None:\n tool_mask = [torch.tensor(mask) for mask in tool_mask_list]\n tool_mask = pad(\n tool_mask, padding_value=1, padding_side=\"right\", pad_to_multiple_of=self.pad_to_multiple_of\n ).to(device=device)\n else:\n tool_mask = None\n\n # If mask_truncated_completions is enabled, zero out truncated completions for attention and loss masking\n if self.mask_truncated_completions:\n eos_and_pad = [self.eos_token_id, self.pad_token_id]\n is_truncated = torch.tensor([ids[-1] not in eos_and_pad for ids in completion_ids_list], device=device)\n # Mask completion_mask for attention masking\n completion_mask = completion_mask * (~is_truncated).unsqueeze(1).int()\n # Also mask tool_mask for consistency in multi-turn training\n if tool_mask is not None:\n tool_mask = tool_mask * (~is_truncated).unsqueeze(1).int()\n\n # Concatenate prompt_mask with completion_mask for logit computation\n prompt_completion_ids = torch.cat([prompt_ids, completion_ids], dim=1) # (B, P+C)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1) # (B, P+C)\n\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n batch_size = self.args.per_device_train_batch_size if mode == \"train\" else self.args.per_device_eval_batch_size\n\n num_images = [len(img_list) for img_list in images] if images is not None else None\n\n # Get forward_kwargs for models with multimodal inputs\n if images is not None:\n prompts_text = [\n apply_chat_template(\n {\"prompt\": prompt}, self.processing_class, tools=self.tools, **self.chat_template_kwargs\n )[\"prompt\"]\n for prompt in prompts\n ]\n prompt_inputs = self.processing_class(images=images, text=prompts_text, padding=True, return_tensors=\"pt\")\n prompt_inputs = super()._prepare_inputs(prompt_inputs)\n forward_kwargs = {k: v for k, v in prompt_inputs.items() if k not in [\"input_ids\", \"attention_mask\"]}\n else:\n forward_kwargs = {}\n\n # If token_type_ids are used, extend them with zeros for the completion part\n if \"token_type_ids\" in forward_kwargs:\n token_type_ids = forward_kwargs[\"token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n # Needed only with pad_to_multiple_of: otherwise prompt_ids and token_type_ids must have equal len\n padding_size = prompt_ids.size(1) - token_type_ids.size(1)\n if padding_size > 0:\n token_type_ids = torch.cat(\n [token_type_ids.new_zeros((token_type_ids.size(0), padding_size)), token_type_ids], dim=1\n )\n forward_kwargs[\"token_type_ids\"] = torch.cat(\n [token_type_ids, token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n # If mm_token_type_ids are used, extend them with zeros for the completion part\n if \"mm_token_type_ids\" in forward_kwargs:\n mm_token_type_ids = forward_kwargs[\"mm_token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n # Needed only with pad_to_multiple_of: otherwise prompt_ids and mm_token_type_ids must have equal len\n padding_size = prompt_ids.size(1) - mm_token_type_ids.size(1)\n if padding_size > 0:\n mm_token_type_ids = torch.cat(\n [mm_token_type_ids.new_zeros((mm_token_type_ids.size(0), padding_size)), mm_token_type_ids],\n dim=1,\n )\n forward_kwargs[\"mm_token_type_ids\"] = torch.cat(\n [mm_token_type_ids, mm_token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n\n # When gradient checkpointing is enabled with use_reentrant=True (non default), calling the model inside a\n # torch.no_grad() block triggers a harmless PyTorch warning (\"None of the inputs have requires_grad=True\").\n # Temporarily disable checkpointing to avoid this warning during inference.\n with torch.no_grad(), disable_gradient_checkpointing(self.model, self.args.gradient_checkpointing_kwargs):\n # If the generation and optimization steps are misalignedโ€”i.e., if generation does not occur at the end of\n # a full optimizer step (when gradient_accumulation_steps is not a multiple of generate_every)โ€”then the\n # samples may come from an earlier version of the model. In that case, we need to track old_per_token_logps\n # for importance sampling. If the steps are aligned, importance sampling isn't necessary and we set\n # old_per_token_logps to None.\n # When using vLLM, we always compute old_per_token_logps for importance sampling, it was shown that the\n # distribution mismatch between vLLM and the training model can be large and harm the training.\n generate_every = self.args.steps_per_generation * self.num_iterations # generation frequency\n if self.args.gradient_accumulation_steps % generate_every != 0 or (\n self.use_vllm and self.vllm_importance_sampling_correction\n ):\n old_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n old_per_token_logps = None\n\n # Compute the importance sampling ratio when using vLLM, to correct for potential distribution mismatch\n if self.use_vllm and self.vllm_importance_sampling_correction:\n mask = completion_mask if tool_mask is None else completion_mask * tool_mask\n per_token_logps_diff = (old_per_token_logps - sampling_per_token_logps) * mask\n\n sequence_level_is = self.vllm_importance_sampling_mode in [\"sequence_mask\", \"sequence_truncate\"]\n if sequence_level_is:\n per_sequence_logps_diff = per_token_logps_diff.sum(dim=-1, keepdim=True)\n logps_diff = per_sequence_logps_diff\n else:\n logps_diff = per_token_logps_diff\n\n vllm_importance_sampling_ratio = torch.exp(logps_diff)\n\n # vllm_importance_sampling_ratio.shape:\n # token_* modes: (B, T) (per-token ratio)\n # sequence_* modes: (B, 1) (per-sequence ratio)\n\n if self.vllm_importance_sampling_mode in [\"sequence_truncate\", \"token_truncate\"]:\n vllm_importance_sampling_ratio = torch.clamp(\n vllm_importance_sampling_ratio, max=self.vllm_importance_sampling_cap\n )\n elif self.vllm_importance_sampling_mode in [\"sequence_mask\", \"token_mask\"]:\n vllm_importance_sampling_ratio = vllm_importance_sampling_ratio.masked_fill(\n vllm_importance_sampling_ratio > self.vllm_importance_sampling_cap, value=0.0\n )\n else:\n raise ValueError(\n f\"Unknown vLLM importance sampling level: {self.vllm_importance_sampling_mode}. Possible values are 'token_truncate', 'token_mask', 'sequence_truncate', and 'sequence_mask'.\"\n )\n\n # Compute the per-token log probabilities for the reference model\n if self.beta != 0.0:\n if self.ref_model is not None:\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.ref_model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n # When training a PEFT adapter, how we obtain the reference depends on the setup:\n # - New adapter: disabling adapters yields the base model.\n # - Re-training an existing adapter: an initial copy is loaded under the name \"ref\".\n model = self.accelerator.unwrap_model(self.model)\n with use_adapter(model, adapter_name=\"ref\" if \"ref\" in model.peft_config else None):\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n ref_per_token_logps = None\n\n # Decode\n prompts_text = self.processing_class.batch_decode(prompt_ids, skip_special_tokens=True)\n completions_text = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n\n # Merge extra_fields from rollout_func into inputs for reward functions\n if extra_fields:\n for i, inp in enumerate(inputs):\n for key, values in extra_fields.items():\n if isinstance(values, list) and i < len(values):\n inp[key] = values[i]\n elif not isinstance(values, list):\n inp[key] = values\n\n # Calculate rewards for each reward function. rewards_per_func aggregates rewards across all processes. This is\n # important because rewards will be normalized per group, and completions are distributed. We will later slice\n # rewards_per_func to extract each process's subset.\n rewards_per_func = self._calculate_rewards(inputs, prompts, completions, completion_ids_list)\n num_generations = self.num_generations if mode == \"train\" else self.num_generations_eval\n\n if self.multi_objective_aggregation == \"sum_then_normalize\":\n # Apply weights to each reward function's output and sum\n rewards = (rewards_per_func * self.reward_weights.to(device).unsqueeze(0)).nansum(dim=1)\n mean_grouped_rewards = rewards.view(-1, num_generations).mean(dim=1)\n mean_grouped_rewards = mean_grouped_rewards.repeat_interleave(num_generations, dim=0)\n if self.scale_rewards in [\"group\", \"none\"]:\n # If self.scale_rewards = \"none\", we'll only use std_rewards to check for zero std for logging\n if num_generations > 1:\n std_rewards = rewards.view(-1, num_generations).std(dim=1)\n std_rewards = std_rewards.repeat_interleave(num_generations, dim=0)\n else: # doesn't occur during training, but could occur in eval when num_generations_eval=1\n std_rewards = torch.zeros_like(rewards)\n elif self.scale_rewards == \"batch\":\n # Compute global std\n if rewards.numel() > 1:\n std_rewards = rewards.std().expand_as(rewards)\n else: # doesn't occur during training, but could occur in eval when num_generations_eval=batch_size=1\n std_rewards = torch.zeros_like(rewards)\n else:\n raise ValueError(\n f\"Invalid value for scale_rewards: {self.scale_rewards}. Must be one of 'batch', 'group', or 'none'.\"\n )\n\n advantages = rewards - mean_grouped_rewards\n if self.scale_rewards != \"none\":\n advantages = advantages / (std_rewards + 1e-4)\n is_std_zero = torch.isclose(std_rewards, torch.zeros_like(std_rewards)) # for logging\n\n elif self.multi_objective_aggregation == \"normalize_then_sum\":\n grouped = rewards_per_func.view(-1, num_generations, len(self.reward_funcs))\n mean_k = torch.nanmean(grouped, dim=1, keepdim=True)\n std_k = nanstd(grouped, dim=1, keepdim=True) if num_generations > 1 else torch.zeros_like(mean_k)\n reward_k = (grouped - mean_k) / (std_k + 1e-4)\n reward_k = reward_k.view(-1, len(self.reward_funcs))\n rewards = (reward_k * self.reward_weights.to(device).unsqueeze(0)).nansum(dim=1)\n std_rewards = rewards.std().expand_as(rewards) if rewards.numel() > 1 else torch.zeros_like(rewards)\n advantages = (rewards - rewards.mean()) / (std_rewards + 1e-4)\n is_std_zero = torch.isclose(std_rewards, torch.zeros_like(std_rewards)) # for logging\n\n else:\n raise ValueError(\n f\"Invalid multi_objective_aggregation: {self.multi_objective_aggregation}. Must be \"\n \"'sum_then_normalize' or 'normalize_then_sum'.\"\n )\n\n # Slice to keep only the local part of the data\n process_slice = slice(\n self.accelerator.process_index * len(prompts),\n (self.accelerator.process_index + 1) * len(prompts),\n )\n all_process_advantages = advantages.clone() # keep the aggregated advantages for logging\n advantages = advantages[process_slice]\n\n # Calculate mean reward per function, but only for samples where the function was applied (non-NaN values)\n for i, reward_func_name in enumerate(self.reward_func_names):\n mean_rewards = torch.nanmean(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/mean\"].append(mean_rewards)\n std_func_rewards = nanstd(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/std\"].append(std_func_rewards)\n rewards = rewards_per_func.nansum(dim=1)\n self._metrics[mode][\"reward\"].append(rewards.mean().item())\n self._metrics[mode][\"reward_std\"].append(rewards.std().item())\n self._metrics[mode][\"frac_reward_zero_std\"].append(is_std_zero.float().mean().item())\n\n # Log prompt and completion texts\n self._logs[\"prompt\"].extend(gather_object(prompts_text))\n self._logs[\"completion\"].extend(gather_object(completions_text))\n for i, name in enumerate(self.reward_func_names):\n self._logs[\"rewards\"][name].extend(rewards_per_func[:, i].tolist())\n self._logs[\"advantages\"].extend(all_process_advantages.tolist())\n\n # Flush user-logged extra columns (from log_extra), gathering across processes.\n # Keys must be sorted so that all ranks call gather_object in the same order, otherwise values\n # get mis-attributed across columns (dict insertion order may differ between processes).\n for column in sorted(self._pending_extra_logs):\n self._logs[\"extra\"][column].extend(gather_object(self._pending_extra_logs[column]))\n self._pending_extra_logs.clear()\n\n # Flush user-logged metrics (from log_metric), averaging across processes.\n # Keys must be sorted so that all ranks call accelerator.gather in the same order, otherwise values\n # get mis-attributed across metrics (dict insertion order may differ between processes).\n for name in sorted(self._pending_metrics):\n values = self._pending_metrics[name]\n local_mean = sum(values) / len(values)\n global_mean = self.accelerator.gather(torch.tensor(local_mean, device=device)).mean().item()\n self._metrics[mode][name].append(global_mean)\n self._pending_metrics.clear()\n\n if images is not None:\n self._logs[\"images\"].extend(gather_object(images))\n\n if self.use_vllm and self.vllm_importance_sampling_correction:\n delta = torch.abs(old_per_token_logps - sampling_per_token_logps)\n mask = completion_mask.bool() if tool_mask is None else (completion_mask * tool_mask).bool()\n delta = delta[mask]\n mean_delta = torch.mean(delta) if delta.numel() > 0 else torch.tensor(0.0, device=device)\n max_delta = torch.max(delta) if delta.numel() > 0 else torch.tensor(0.0, device=device)\n self._metrics[mode][\"sampling/sampling_logp_difference/mean\"].append(\n self.accelerator.gather(mean_delta).mean().item()\n )\n self._metrics[mode][\"sampling/sampling_logp_difference/max\"].append(\n self.accelerator.gather(max_delta).max().item()\n )\n if sequence_level_is:\n flat_is_ratio = vllm_importance_sampling_ratio.flatten()\n else:\n flat_is_ratio = vllm_importance_sampling_ratio[mask]\n\n min_importance_sampling_ratio = (\n torch.min(flat_is_ratio) if flat_is_ratio.numel() > 0 else torch.tensor(0.0, device=device)\n )\n mean_importance_sampling_ratio = (\n torch.mean(flat_is_ratio) if flat_is_ratio.numel() > 0 else torch.tensor(0.0, device=device)\n )\n max_importance_sampling_ratio = (\n torch.max(flat_is_ratio) if flat_is_ratio.numel() > 0 else torch.tensor(0.0, device=device)\n )\n self._metrics[mode][\"sampling/importance_sampling_ratio/min\"].append(\n nanmin(self.accelerator.gather(min_importance_sampling_ratio)).item()\n )\n self._metrics[mode][\"sampling/importance_sampling_ratio/mean\"].append(\n self.accelerator.gather(mean_importance_sampling_ratio).nanmean().item()\n )\n self._metrics[mode][\"sampling/importance_sampling_ratio/max\"].append(\n nanmax(self.accelerator.gather(max_importance_sampling_ratio)).item()\n )\n\n output = {\n \"prompt_ids\": prompt_ids,\n \"prompt_mask\": prompt_mask,\n \"completion_ids\": completion_ids,\n \"completion_mask\": completion_mask,\n \"advantages\": advantages,\n \"num_items_in_batch\": num_items_in_batch,\n }\n if old_per_token_logps is not None:\n output[\"old_per_token_logps\"] = old_per_token_logps\n if self.use_vllm and self.vllm_importance_sampling_correction:\n output[\"importance_sampling_ratio\"] = vllm_importance_sampling_ratio\n if sampling_per_token_logps is not None:\n output[\"sampling_per_token_logps\"] = sampling_per_token_logps\n if ref_per_token_logps is not None:\n output[\"ref_per_token_logps\"] = ref_per_token_logps\n if \"pixel_values\" in forward_kwargs:\n output[\"pixel_values\"] = forward_kwargs[\"pixel_values\"]\n if \"image_grid_thw\" in forward_kwargs:\n output[\"image_grid_thw\"] = forward_kwargs[\"image_grid_thw\"]\n if \"pixel_attention_mask\" in forward_kwargs:\n output[\"pixel_attention_mask\"] = forward_kwargs[\"pixel_attention_mask\"]\n if \"image_sizes\" in forward_kwargs:\n output[\"image_sizes\"] = forward_kwargs[\"image_sizes\"]\n if \"token_type_ids\" in forward_kwargs:\n output[\"token_type_ids\"] = forward_kwargs[\"token_type_ids\"]\n if \"mm_token_type_ids\" in forward_kwargs:\n output[\"mm_token_type_ids\"] = forward_kwargs[\"mm_token_type_ids\"]\n if images is not None:\n output[\"num_images\"] = num_images\n if tool_mask is not None:\n output[\"tool_mask\"] = tool_mask\n return output\n\n def compute_liger_loss(self, unwrapped_model, inputs):\n # Compute the per-token log probabilities for the model\n prompt_ids, prompt_mask = inputs[\"prompt_ids\"], inputs[\"prompt_mask\"]\n completion_ids, completion_mask = inputs[\"completion_ids\"], inputs[\"completion_mask\"]\n input_ids = torch.cat([prompt_ids, completion_ids], dim=1)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1)\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n\n # Get the last hidden state of the model\n last_hidden_state = self._get_last_hidden_state(\n unwrapped_model,\n input_ids,\n attention_mask,\n logits_to_keep,\n inputs.get(\"pixel_values\"),\n inputs.get(\"image_grid_thw\"),\n inputs.get(\"pixel_attention_mask\"),\n inputs.get(\"image_sizes\"),\n )\n\n # Apply tool_mask (from env_mask) for loss computation in multi-turn training scenarios\n loss_mask = completion_mask if \"tool_mask\" not in inputs else completion_mask * inputs[\"tool_mask\"]\n # Compute loss and metrics using liger grpo loss\n loss, metrics = self.liger_grpo_loss(\n _input=last_hidden_state,\n lin_weight=unwrapped_model.lm_head.weight,\n selected_token_ids=completion_ids,\n # The attention_mask parameter in liger loss is actually used as a loss mask (not model attention)\n attention_mask=loss_mask,\n advantages=inputs[\"advantages\"],\n bias=unwrapped_model.lm_head.bias,\n old_per_token_logps=inputs.get(\"old_per_token_logps\"),\n ref_per_token_logps=inputs.get(\"ref_per_token_logps\"),\n vllm_is_ratio=inputs.get(\"importance_sampling_ratio\"),\n )\n # Extract metrics from the liger_grpo_loss output\n # KL divergence is the first metric when beta is non-zero\n mean_kl = metrics[0] if self.beta != 0.0 else None\n clip_ratio = metrics[-1]\n\n mode = \"train\" if self.model.training else \"eval\"\n if self.beta != 0.0:\n self._metrics[mode][\"kl\"].append(self.accelerator.gather(mean_kl).mean().item())\n self._metrics[mode][\"clip_ratio\"].append(self.accelerator.gather(clip_ratio).mean().item())\n normalizer = self.current_gradient_accumulation_steps if mode == \"train\" else 1.0 # no accum in eval\n return loss / normalizer\n\n @profiling_decorator\n def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):\n if return_outputs:\n raise ValueError(\"The GRPOTrainer does not support returning outputs\")\n if self.use_liger_kernel:\n # Compute the loss using the liger grpo loss\n unwrapped_model = self.accelerator.unwrap_model(model)\n return self._forward_redirection(model, unwrapped_model, self.compute_liger_loss, unwrapped_model, inputs)\n else:\n return self._compute_loss(model, inputs)\n\n @staticmethod\n def get_off_policy_mask(\n advantages: torch.Tensor,\n per_token_logps: torch.Tensor,\n sampling_per_token_logps: torch.Tensor,\n mask: torch.Tensor,\n off_policy_threshold: float,\n ) -> torch.Tensor:\n \"\"\"\n Computes the Off-Policy Sequence Mask from DeepSeek-V3.2 paper. Returns a (B, 1) tensor where 1.0 indicates\n \"Keep\" and 0.0 indicates \"Drop\".\n \"\"\"\n # forward KL div: log(pi_old) - log(pi_theta)\n kl_div = sampling_per_token_logps - per_token_logps.detach()\n # Sequence-level Mean KL (ignoring prompt+padding)\n seq_kl_sum = (kl_div * mask).sum(dim=1, keepdim=True)\n avg_seq_kl = seq_kl_sum / mask.sum(dim=1, keepdim=True).clamp(min=1.0)\n # Keep if (Advantage >= 0) OR (KL <= delta)\n is_pos_adv = advantages >= 0\n is_low_kl = avg_seq_kl <= off_policy_threshold\n return (is_pos_adv | is_low_kl).to(dtype=mask.dtype) # (B, 1)\n\n @staticmethod\n @torch.no_grad()\n def get_gamma_weights(\n advantages: torch.Tensor,\n log_ratio_per_token: torch.Tensor,\n mask: torch.Tensor,\n importance_sampling_ratio: torch.Tensor | None, # (B, T)\n k_pos: float = 2.0,\n lambda_pos: float = 3.0,\n k_neg: float = 3.0,\n lambda_neg: float = 2.0,\n ) -> torch.Tensor:\n \"\"\"\n Computes the Gamma weights for the VESPO loss. For reference:\n ฯ†(w) = e^ฮป ร— w^k ร— e^{-ฮปw} is the gamma weighting (normalized so ฯ†(1)=1)\n with w = sequence-level importance sampling ratio\n note: we will compute ฯ†(w) in log space\n\n ฯ†(w) is detached via @torch.no_grad(), only acts as gradient scaling coefficient\n\n VESPO loss = -ฯ†(w) ร— A ร— log_prob, gradient naturally gives ฯ†(w) ร— A ร— โˆ‡log ฯ€\n \"\"\"\n # reducing clamp range directly to log(1e-8) ~ -18.42, to avoid recomputing log_w=log(w.clamp(min=1e-8)) later\n # This is solely for matching truthfully the original implementation, otherwise keeping -20 could be fine.\n lower_clamp = math.log(1e-8)\n\n # Sequence-level log ratio ฮฃ log(ฯ€_ฮธ/ฯ€_old) (not a mean like for `log_importance_weights`)\n log_ratio_clamped = torch.clamp(log_ratio_per_token, -20.0, 20.0)\n seq_log_ratio = torch.sum(log_ratio_clamped * mask, dim=-1, keepdim=True) # (B, 1)\n\n # Apply token-level TIS or MIS correction (in log space)\n if importance_sampling_ratio is not None:\n log_is_ratio = torch.clamp(torch.log(importance_sampling_ratio), lower_clamp, 20.0)\n # log(w) = log(ฯ€_ฮธ/ฯ€_old) + log(ฯ€_old/ฯ€_sampler)\n seq_log_ratio += torch.sum(log_is_ratio, dim=-1, keepdim=True)\n\n log_w_seq = torch.clamp(seq_log_ratio, lower_clamp, 20.0)\n w_seq = torch.exp(log_w_seq)\n\n # compute k and lambda based on advantage sign\n is_nonneg_adv = advantages >= 0\n k_seq = torch.where(is_nonneg_adv, k_pos, k_neg)\n lambda_seq = torch.where(is_nonneg_adv, lambda_pos, lambda_neg).clamp(min=1e-4)\n\n # log(ฯ†(w)) = ฮป + k ร— log(w) - ฮป ร— w\n log_phi = lambda_seq + k_seq * log_w_seq - lambda_seq * w_seq\n phi_seq = torch.exp(log_phi).nan_to_num(nan=0.0, posinf=0.0, neginf=0.0)\n\n return phi_seq # (B, 1)\n\n def _compute_loss(self, model, inputs):\n # Compute the per-token log probabilities for the model\n prompt_ids, prompt_mask = inputs[\"prompt_ids\"], inputs[\"prompt_mask\"]\n completion_ids, completion_mask = inputs[\"completion_ids\"], inputs[\"completion_mask\"]\n input_ids = torch.cat([prompt_ids, completion_ids], dim=1)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1)\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n mask = completion_mask if \"tool_mask\" not in inputs else completion_mask * inputs[\"tool_mask\"]\n\n # Compute the per_token_logps and the entropy at each position in the completion\n per_token_logps, entropies = self._get_per_token_logps_and_entropies(\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n compute_entropy=True,\n pixel_values=inputs.get(\"pixel_values\"),\n image_grid_thw=inputs.get(\"image_grid_thw\"),\n num_images=inputs.get(\"num_images\"),\n pixel_attention_mask=inputs.get(\"pixel_attention_mask\"),\n image_sizes=inputs.get(\"image_sizes\"),\n token_type_ids=inputs.get(\"token_type_ids\"),\n mm_token_type_ids=inputs.get(\"mm_token_type_ids\"),\n )\n\n if self.top_entropy_quantile < 1.0:\n entropy_mask = self.get_high_entropy_mask(entropies, mask, 1 - self.top_entropy_quantile)\n else:\n entropy_mask = None\n\n # Compute the loss\n advantages = inputs[\"advantages\"]\n # In the base GRPO implementation, advantages are expected to have shape (B,). To support subclasses that\n # provide advantages with shape (B, T) (e.g., MiniLLM), we *conditionally* unsqueeze the tensor.\n if advantages.dim() == 1:\n advantages = advantages.unsqueeze(1)\n # When num_iterations == 1 and steps_per_generation <= gradient_accumulation_steps,\n # old_per_token_logps == per_token_logps. In this case we can skip its computation\n # (see _generate_and_score_completions) and instead use per_token_logps.detach().\n # The exception is when using vLLM, where we always compute old_per_token_logps\n # for importance sampling\n old_per_token_logps = inputs.get(\"old_per_token_logps\")\n old_per_token_logps = per_token_logps.detach() if old_per_token_logps is None else old_per_token_logps\n\n if self.off_policy_mask_threshold is not None:\n # OPSM should use inference-time logprobs to detect both sources of off-policyness:\n # 1. Drift from gradient updates (always present)\n # 2. Drift from training-inference mismatch (when using vLLM)\n # When using vLLM, prioritize sampling_per_token_logps, otherwise use old_per_token_logps\n sampling_per_token_logps = inputs.get(\"sampling_per_token_logps\", old_per_token_logps)\n\n off_policy_mask = self.get_off_policy_mask(\n advantages=advantages,\n per_token_logps=per_token_logps,\n sampling_per_token_logps=sampling_per_token_logps,\n mask=mask,\n off_policy_threshold=self.off_policy_mask_threshold,\n )\n\n log_ratio = per_token_logps - old_per_token_logps\n if self.importance_sampling_level == \"token\":\n log_importance_weights = log_ratio\n elif self.importance_sampling_level == \"sequence\":\n log_importance_weights = (log_ratio * mask).sum(-1) / mask.sum(-1).clamp(min=1.0)\n log_importance_weights = log_importance_weights.unsqueeze(-1)\n else:\n raise ValueError(\n f\"Unknown importance sampling level: {self.importance_sampling_level}. Possible values are 'token' \"\n \"and 'sequence'.\"\n )\n\n coef_1 = torch.exp(log_importance_weights)\n\n # Compute the KL divergence between the model and the reference model\n if self.beta != 0.0:\n ref_per_token_logps = inputs[\"ref_per_token_logps\"]\n per_token_kl = (\n torch.exp(ref_per_token_logps - per_token_logps) - (ref_per_token_logps - per_token_logps) - 1\n )\n # Importance sampling correction for the KL divergence\n if self.args.use_bias_correction_kl:\n per_token_kl = per_token_kl * coef_1\n\n # From here, log_importance_weights (and all subsequent tensors, coef_1, coef_2, etc.) shape depends on\n # importance_sampling_level: \"token\" level: (B, T); \"sequence\" level: (B, 1)\n if self.loss_type == \"cispo\":\n clamped_ratios = torch.clamp(coef_1, max=self.epsilon_high).detach()\n per_token_loss = -clamped_ratios * advantages * per_token_logps\n elif self.loss_type in [\"grpo\", \"bnpo\", \"dr_grpo\", \"dapo\", \"luspo\"]:\n coef_2 = torch.clamp(coef_1, 1 - self.epsilon_low, 1 + self.epsilon_high)\n # Two-sided clipping\n if self.args.delta is not None:\n coef_1 = torch.clamp(coef_1, max=self.args.delta)\n\n per_token_loss1 = coef_1 * advantages\n per_token_loss2 = coef_2 * advantages\n per_token_loss = -torch.min(per_token_loss1, per_token_loss2)\n elif self.loss_type == \"sapo\":\n temperatures = torch.where(advantages > 0, self.args.sapo_temperature_pos, self.args.sapo_temperature_neg)\n soft_coef_1 = torch.sigmoid(temperatures * (coef_1 - 1)) * 4 / temperatures\n per_token_loss = -soft_coef_1 * advantages\n elif self.loss_type == \"vespo\":\n phi_seq = self.get_gamma_weights(\n advantages=advantages,\n log_ratio_per_token=log_ratio,\n mask=mask,\n importance_sampling_ratio=inputs.get(\"importance_sampling_ratio\"),\n k_pos=self.args.vespo_k_pos,\n lambda_pos=self.args.vespo_lambda_pos,\n k_neg=self.args.vespo_k_neg,\n lambda_neg=self.args.vespo_lambda_neg,\n )\n per_token_loss = -phi_seq * advantages * per_token_logps\n else:\n raise ValueError(f\"Unknown loss type: {self.loss_type}\")\n\n if self.off_policy_mask_threshold is not None:\n per_token_loss = per_token_loss * off_policy_mask\n\n if entropy_mask is not None:\n per_token_loss = per_token_loss * entropy_mask\n\n if self.use_vllm and self.vllm_importance_sampling_correction and self.loss_type != \"vespo\":\n per_token_loss = per_token_loss * inputs[\"importance_sampling_ratio\"]\n\n if self.beta != 0.0:\n per_token_loss = per_token_loss + self.beta * per_token_kl\n\n mode = \"train\" if self.model.training else \"eval\"\n if self.loss_type in [\"grpo\", \"sapo\"]:\n loss = ((per_token_loss * mask).sum(-1) / mask.sum(-1).clamp(min=1.0)).mean()\n normalizer = self.current_gradient_accumulation_steps if mode == \"train\" else 1.0 # no accum in eval\n loss = loss / normalizer\n elif self.loss_type == \"bnpo\":\n loss = (per_token_loss * mask).sum() / mask.sum().clamp(min=1.0)\n normalizer = self.current_gradient_accumulation_steps if mode == \"train\" else 1.0 # no accum in eval\n loss = loss / normalizer\n elif self.loss_type == \"dr_grpo\":\n loss = (per_token_loss * mask).sum() / (per_token_loss.size(0) * self.max_completion_length)\n normalizer = self.current_gradient_accumulation_steps if mode == \"train\" else 1.0 # no accum in eval\n loss = loss / normalizer\n elif self.loss_type in [\"cispo\", \"dapo\", \"vespo\"]:\n normalizer = inputs[\"num_items_in_batch\"] / self.accelerator.num_processes\n loss = (per_token_loss * mask).sum() / normalizer\n elif self.loss_type == \"luspo\":\n # Unless importance_sampling_level=\"token\" (not recommended here), per_token_loss is expected to be (B, 1)\n loss = (per_token_loss * mask.sum(1, keepdim=True)).mean()\n normalizer = self.current_gradient_accumulation_steps if mode == \"train\" else 1.0\n loss = loss / normalizer\n else:\n raise ValueError(f\"Unknown loss type: {self.loss_type}\")\n\n # Log the metrics\n completion_token_count = mask.sum().clamp(min=1.0)\n\n def masked_batch_mean(x):\n if x.shape[1] == 1: # when importance_sampling_level == \"sequence\"\n return x.mean()\n else:\n return (x * mask).sum() / completion_token_count\n\n if self.beta != 0.0:\n mean_kl = masked_batch_mean(per_token_kl)\n self._metrics[mode][\"kl\"].append(self.accelerator.gather(mean_kl).nanmean().item())\n\n mean_entropy = masked_batch_mean(entropies)\n self._metrics[mode][\"entropy\"].append(self.accelerator.gather(mean_entropy).nanmean().item())\n\n if self.loss_type in [\"grpo\", \"bnpo\", \"dr_grpo\", \"dapo\", \"luspo\"]:\n # Compute the clipped probability ratios\n is_low_clipped = (coef_1 < 1 - self.epsilon_low) & (advantages < 0)\n is_high_clipped = (coef_1 > 1 + self.epsilon_high) & (advantages > 0)\n is_region_clipped = is_low_clipped | is_high_clipped\n\n low_clip = masked_batch_mean(is_low_clipped.float())\n high_clip = masked_batch_mean(is_high_clipped.float())\n clip_ratio = masked_batch_mean(is_region_clipped.float())\n\n gathered_low_clip = self.accelerator.gather(low_clip)\n self._metrics[mode][\"clip_ratio/low_mean\"].append(gathered_low_clip.nanmean().item())\n self._metrics[mode][\"clip_ratio/low_min\"].append(nanmin(gathered_low_clip).item())\n gathered_high_clip = self.accelerator.gather(high_clip)\n self._metrics[mode][\"clip_ratio/high_mean\"].append(gathered_high_clip.nanmean().item())\n self._metrics[mode][\"clip_ratio/high_max\"].append(nanmax(gathered_high_clip).item())\n gathered_clip_ratio = self.accelerator.gather(clip_ratio)\n self._metrics[mode][\"clip_ratio/region_mean\"].append(gathered_clip_ratio.nanmean().item())\n elif self.loss_type == \"cispo\":\n is_cispo_clipped = (coef_1 > self.epsilon_high) & (advantages > 0)\n cispo_clip_ratio = masked_batch_mean(is_cispo_clipped.float())\n gathered_cispo_clip_ratio = self.accelerator.gather(cispo_clip_ratio)\n self._metrics[mode][\"cispo_clip_ratio\"].append(gathered_cispo_clip_ratio.nanmean().item())\n elif self.loss_type == \"vespo\":\n gathered_phi_seq = self.accelerator.gather(phi_seq)\n self._metrics[mode][\"vespo/phi_seq_mean\"].append(gathered_phi_seq.nanmean().item())\n\n return loss\n\n # During eval, Trainer calls prediction_step. If no labels are present in the inputs, it only runs forward and\n # returns logits. We override prediction_step to force compute_loss, because this trainer doesn't involve labels.\n def prediction_step(self, model, inputs, prediction_loss_only, ignore_keys: list[str] | None = None):\n inputs = self._prepare_inputs(inputs)\n with torch.no_grad():\n with self.compute_loss_context_manager():\n loss = self.compute_loss(model, inputs)\n loss = loss.mean().detach()\n return loss, None, None\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n mode = \"train\" if self.model.training else \"eval\"\n metrics = {key: sum(val) / len(val) for key, val in self._metrics[mode].items()} # average the metrics\n\n # This method can be called both in training and evaluation. When called in evaluation, the keys in `logs`\n # start with \"eval_\". We need to add the prefix \"eval_\" to the keys in `metrics` to match the format.\n if mode == \"eval\":\n metrics = {f\"eval_{key}\": val for key, val in metrics.items()}\n\n logs = {**logs, **metrics}\n super().log(logs, start_time)\n self._metrics[mode].clear()\n\n if self.accelerator.is_main_process and self.log_completions:\n if is_rich_available():\n print_prompt_completions_sample(\n self._logs[\"prompt\"],\n self._logs[\"completion\"],\n self._logs[\"rewards\"],\n self._logs[\"advantages\"],\n self.state.global_step,\n self.num_completions_to_print,\n )\n\n logging_backends = []\n if self.args.report_to and \"wandb\" in self.args.report_to and wandb.run is not None:\n logging_backends.append(wandb)\n if self.args.report_to and \"trackio\" in self.args.report_to:\n logging_backends.append(trackio)\n\n table = {\n \"step\": [self.state.global_step] * len(self._logs[\"prompt\"]),\n \"prompt\": self._logs[\"prompt\"],\n \"completion\": self._logs[\"completion\"],\n **self._logs[\"rewards\"],\n **self._logs[\"extra\"],\n \"advantage\": self._logs[\"advantages\"],\n }\n\n df_base = pd.DataFrame(table)\n df_base.to_parquet(\n os.path.join(\n self.args.output_dir,\n \"completions\",\n f\"completions_{self.state.global_step:05d}.parquet\",\n )\n )\n\n images_raw = self._logs[\"images\"] or []\n\n for logging_backend in logging_backends:\n if images_raw:\n images = []\n for image_list in self._logs[\"images\"]:\n images.append([logging_backend.Image(image) for image in image_list])\n df = pd.concat(\n [df_base, pd.Series(images, name=\"image\")],\n axis=1,\n copy=False,\n )\n else:\n df = df_base\n\n if self.log_unique_prompts:\n df = df.drop_duplicates(subset=[\"prompt\"])\n\n logging_backend.log({\"completions\": logging_backend.Table(dataframe=df)})\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/trainer/kto_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport warnings\nfrom dataclasses import dataclass\n\nfrom ..import_utils import suppress_experimental_warning\n\n\nwith suppress_experimental_warning():\n from ..experimental.kto import KTOConfig as _KTOConfig\n\n\n@dataclass\nclass KTOConfig(_KTOConfig):\n def __post_init__(self):\n warnings.warn(\n \"The `KTOConfig` is now located in `trl.experimental`. Please update your imports to \"\n \"`from trl.experimental.kto import KTOConfig`. For more information, see \"\n \"https://github.com/huggingface/trl/issues/4223. Promoting KTO to the stable API is a high-priority task. \"\n \"Until then, this current path (`from trl import KTOConfig`) will remain, but API changes may occur.\",\n FutureWarning,\n stacklevel=3,\n )\n super().__post_init__()\n" }, { "path": "trl/trainer/kto_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport warnings\nfrom dataclasses import dataclass\n\nfrom ..import_utils import suppress_experimental_warning\n\n\nwith suppress_experimental_warning():\n from ..experimental.kto import KTOTrainer as _KTOTrainer\n\n\n@dataclass\nclass KTOTrainer(_KTOTrainer):\n def __init__(self, *args, **kwargs):\n warnings.warn(\n \"The `KTOTrainer` is now located in `trl.experimental`. Please update your imports to \"\n \"`from trl.experimental.kto import KTOTrainer`. For more information, see \"\n \"https://github.com/huggingface/trl/issues/4223. Promoting KTO to the stable API is a high-priority task. \"\n \"Until then, this current path (`from trl import KTOTrainer`) will remain, but API changes may occur.\",\n FutureWarning,\n stacklevel=2,\n )\n super().__init__(*args, **kwargs)\n" }, { "path": "trl/trainer/model_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\n\n\n@dataclass\nclass ModelConfig:\n \"\"\"\n Configuration class for the models.\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n model_name_or_path (`str`, *optional*):\n Model checkpoint for weights initialization.\n model_revision (`str`, *optional*, defaults to `\"main\"`):\n Specific model version to use. It can be a branch name, a tag name, or a commit id.\n dtype (`Literal[\"auto\", \"bfloat16\", \"float16\", \"float32\"]`, *optional*, defaults to `\"float32\"`):\n Override the default `torch.dtype` and load the model under this dtype. Possible values are\n\n - `\"bfloat16\"`: `torch.bfloat16`\n - `\"float16\"`: `torch.float16`\n - `\"float32\"`: `torch.float32`\n - `\"auto\"`: Automatically derive the dtype from the model's weights.\n\n trust_remote_code (`bool`, *optional*, defaults to `False`):\n Whether to allow for custom models defined on the Hub in their own modeling files. This option should only\n be set to `True` for repositories you trust and in which you have read the code, as it will execute code\n present on the Hub on your local machine.\n attn_implementation (`str`, *optional*):\n Which attention implementation to use. More information in the [Kernels Hub Integrations\n Guide](kernels_hub).\n use_peft (`bool`, *optional*, defaults to `False`):\n Whether to use PEFT for training.\n lora_r (`int`, *optional*, defaults to `16`):\n LoRA R value.\n lora_alpha (`int`, *optional*, defaults to `32`):\n LoRA alpha.\n lora_dropout (`float`, *optional*, defaults to `0.05`):\n LoRA dropout.\n lora_target_modules (`str | list[str]`, *optional*):\n LoRA target modules.\n lora_target_parameters (`str | list[str]`, *optional*):\n List of target parameters for LoRA.\n lora_modules_to_save (`list[str]`, *optional*):\n Model layers to unfreeze & train.\n lora_task_type (`str`, *optional*, defaults to `\"CAUSAL_LM\"`):\n Task type to pass for LoRA (use `\"SEQ_CLS\"` for reward modeling).\n use_rslora (`bool`, *optional*, defaults to `False`):\n Whether to use Rank-Stabilized LoRA, which sets the adapter scaling factor to `lora_alpha/โˆšr`, instead of\n the original default value of `lora_alpha/r`.\n use_dora (`bool`, *optional*, defaults to `False`):\n Enable [Weight-Decomposed Low-Rank Adaptation (DoRA)](https://huggingface.co/papers/2402.09353). This\n technique decomposes the updates of the weights into two parts, magnitude and direction. Direction is\n handled by normal LoRA, whereas the magnitude is handled by a separate learnable parameter. This can\n improve the performance of LoRA, especially at low ranks. Right now, DoRA only supports linear and Conv2D\n layers. DoRA introduces a bigger overhead than pure LoRA, so it is recommended to merge weights for\n inference.\n load_in_8bit (`bool`, *optional*, defaults to `False`):\n Whether to use 8 bit precision for the base model. Works only with LoRA.\n load_in_4bit (`bool`, *optional*, defaults to `False`):\n Whether to use 4 bit precision for the base model. Works only with LoRA.\n bnb_4bit_quant_type (`str`, *optional*, defaults to `\"nf4\"`):\n Quantization type (`\"fp4\"` or `\"nf4\"`).\n use_bnb_nested_quant (`bool`, *optional*, defaults to `False`):\n Whether to use nested quantization.\n \"\"\"\n\n model_name_or_path: str | None = field(\n default=None,\n metadata={\"help\": \"Model checkpoint for weights initialization.\"},\n )\n model_revision: str = field(\n default=\"main\",\n metadata={\"help\": \"Specific model version to use. It can be a branch name, a tag name, or a commit id.\"},\n )\n dtype: str | None = field(\n default=\"float32\",\n metadata={\n \"help\": \"Override the default `torch.dtype` and load the model under this dtype. It defaults to `'float32'`.\",\n \"choices\": [\"auto\", \"bfloat16\", \"float16\", \"float32\"],\n },\n )\n trust_remote_code: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to allow for custom models defined on the Hub in their own modeling files. This option \"\n \"should only be set to `True` for repositories you trust and in which you have read the code, as it will \"\n \"execute code present on the Hub on your local machine.\"\n },\n )\n attn_implementation: str | None = field(\n default=None,\n metadata={\n \"help\": \"Which attention implementation to use. You can run `--attn_implementation=flash_attention_2`, in \"\n \"which case you must install this manually by running `pip install flash-attn --no-build-isolation`.\"\n },\n )\n use_peft: bool = field(\n default=False,\n metadata={\"help\": \"Whether to use PEFT for training.\"},\n )\n lora_r: int = field(\n default=16,\n metadata={\"help\": \"LoRA R value.\"},\n )\n lora_alpha: int = field(\n default=32,\n metadata={\"help\": \"LoRA alpha.\"},\n )\n lora_dropout: float = field(\n default=0.05,\n metadata={\"help\": \"LoRA dropout.\"},\n )\n lora_target_modules: list[str] | None = field(\n default=None,\n metadata={\"help\": \"LoRA target modules.\"},\n )\n lora_target_parameters: list[str] | None = field(\n default=None,\n metadata={\"help\": \"List of target parameters for LoRA.\"},\n )\n lora_modules_to_save: list[str] | None = field(\n default=None,\n metadata={\"help\": \"Model layers to unfreeze & train.\"},\n )\n lora_task_type: str = field(\n default=\"CAUSAL_LM\",\n metadata={\"help\": \"Task type to pass for LoRA (use 'SEQ_CLS' for reward modeling).\"},\n )\n use_rslora: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use Rank-Stabilized LoRA, which sets the adapter scaling factor to `lora_alpha/โˆšr`, \"\n \"instead of the original default value of `lora_alpha/r`.\"\n },\n )\n use_dora: bool = field(\n default=False,\n metadata={\n \"help\": \"Enable Weight-Decomposed Low-Rank Adaptation (DoRA). This technique decomposes the updates of \"\n \"the weights into two parts, magnitude and direction. Direction is handled by normal LoRA, whereas the \"\n \"magnitude is handled by a separate learnable parameter. This can improve the performance of LoRA, \"\n \"especially at low ranks. Right now, DoRA only supports linear and Conv2D layers. DoRA introduces a \"\n \"bigger overhead than pure LoRA, so it is recommended to merge weights for inference.\"\n },\n )\n load_in_8bit: bool = field(\n default=False,\n metadata={\"help\": \"Whether to use 8 bit precision for the base model. Works only with LoRA.\"},\n )\n load_in_4bit: bool = field(\n default=False,\n metadata={\"help\": \"Whether to use 4 bit precision for the base model. Works only with LoRA.\"},\n )\n bnb_4bit_quant_type: str = field(\n default=\"nf4\",\n metadata={\"help\": \"Quantization type.\", \"choices\": [\"fp4\", \"nf4\"]},\n )\n use_bnb_nested_quant: bool = field(\n default=False,\n metadata={\"help\": \"Whether to use nested quantization.\"},\n )\n bnb_4bit_quant_storage: str | None = field(\n default=None,\n metadata={\"help\": \"Quantization storage dtype\"},\n )\n\n def __post_init__(self):\n if self.load_in_8bit and self.load_in_4bit:\n raise ValueError(\"You can't use 8 bit and 4 bit precision at the same time\")\n\n if hasattr(self.lora_target_modules, \"__len__\") and len(self.lora_target_modules) == 1:\n self.lora_target_modules = self.lora_target_modules[0]\n" }, { "path": "trl/trainer/reward_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom .base_config import _BaseConfig\n\n\n@dataclass\nclass RewardConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`RewardTrainer`].\n\n This class includes only the parameters that are specific to Reward training. For a full list of training\n arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this\n class may differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n > Parameters that control the model\n\n model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model`\n argument of the [`RewardTrainer`] is provided as a string. If you're training a MoE architecture and want\n to include the load balancing/auxiliary loss as a part of the final loss, remember to set\n `output_router_logits=True` in this dictionary.\n chat_template_path (`str`, *optional*):\n If specified, sets the model's chat template. This can either be the path to a tokenizer (local directory\n or Hugging Face Hub model) or a direct path to a Jinja template file. When using a Jinja file, you must\n ensure that any special tokens referenced in the template are added to the tokenizer and that the model's\n embedding layer is resized accordingly.\n disable_dropout (`bool`, *optional*, defaults to `True`):\n Whether to disable dropout in the model.\n\n > Parameters that control the data preprocessing\n\n dataset_num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n eos_token (`str`, *optional*):\n Token used to indicate the end of a turn or sequence. If `None`, it defaults to\n `processing_class.eos_token`.\n pad_token (`str`, *optional*):\n Token used for padding. If `None`, it defaults to `processing_class.pad_token`, or if that is also `None`,\n it falls back to `processing_class.eos_token`.\n max_length (`int` or `None`, *optional*, defaults to `1024`):\n Maximum length of the tokenized sequence. Samples are filtered out if either chosen or rejected sequence\n exceeds this value. If `None`, no filtering is applied.\n pad_to_multiple_of (`int`, *optional*):\n If set, the sequences will be padded to a multiple of this value.\n\n > Parameters that control the training\n\n center_rewards_coefficient (`float`, *optional*):\n Coefficient to incentivize the reward model to output mean-zero rewards (proposed by\n https://huggingface.co/papers/2312.09244, Eq. 2). Recommended value: `0.01`.\n activation_offloading (`bool`, *optional*, defaults to `False`):\n Whether to offload the activations to the CPU.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `1e-4` instead of `5e-5`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-4,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n # Parameters that control the model\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `model` argument of \"\n \"the `RewardTrainer` is provided as a string. If you're training a MoE architecture and want to include \"\n \"the load balancing/auxiliary loss as a part of the final loss, remember to set \"\n \"`output_router_logits=True` in this dictionary.\"\n },\n )\n chat_template_path: str | None = field(\n default=None,\n metadata={\n \"help\": \"If specified, sets the model's chat template. This can either be the path to a tokenizer (local \"\n \"directory or Hugging Face Hub model) or a direct path to a Jinja template file. When using a Jinja file, \"\n \"you must ensure that any special tokens referenced in the template are added to the tokenizer and \"\n \"that the model's embedding layer is resized accordingly.\"\n },\n )\n disable_dropout: bool = field(\n default=True,\n metadata={\"help\": \"Whether to disable dropout in the model.\"},\n )\n\n # Parameters that control the data preprocessing\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes to use for processing the dataset.\"},\n )\n eos_token: str | None = field(\n default=None,\n metadata={\n \"help\": \"Token used to indicate the end of a turn or sequence. If `None`, it defaults to `processing_class.eos_token`.\"\n },\n )\n pad_token: str | None = field(\n default=None,\n metadata={\n \"help\": \"Token used for padding. If `None`, it defaults to `processing_class.pad_token`, or if that \"\n \"is also `None`, it falls back to `processing_class.eos_token`.\"\n },\n )\n max_length: int | None = field(\n default=1024,\n metadata={\n \"help\": \"Maximum length of the tokenized sequence. Sequences longer than `max_length` are truncated from \"\n \"the right. If `None`, no truncation is applied.\"\n },\n )\n pad_to_multiple_of: int | None = field(\n default=None,\n metadata={\"help\": \"If set, the sequences will be padded to a multiple of this value.\"},\n )\n\n # Parameters that control the training\n center_rewards_coefficient: float | None = field(\n default=None,\n metadata={\n \"help\": \"Coefficient to incentivize the reward model to output mean-zero rewards (proposed by \"\n \"https://huggingface.co/papers/2312.09244, Eq. 2). Recommended value: `0.01`.\"\n },\n )\n activation_offloading: bool = field(\n default=False,\n metadata={\"help\": \"Whether to offload the activations to the CPU.\"},\n )\n" }, { "path": "trl/trainer/reward_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport contextlib\nimport json\nimport logging\nimport os\nimport re\nimport warnings\nfrom collections import defaultdict\nfrom collections.abc import Callable\nfrom contextlib import contextmanager\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Any\n\nimport torch\nimport torch.nn as nn\nimport transformers\nfrom accelerate import PartialState\nfrom accelerate.logging import get_logger\nfrom accelerate.utils import is_peft_model\nfrom datasets import Dataset, IterableDataset\nfrom packaging.version import Version\nfrom transformers import (\n AutoModelForSequenceClassification,\n AutoTokenizer,\n DataCollator,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n TrainerCallback,\n set_seed,\n)\nfrom transformers.data.data_collator import DataCollatorMixin\nfrom transformers.modeling_layers import GenericForSequenceClassification\nfrom transformers.trainer_utils import EvalPrediction\nfrom transformers.utils import is_peft_available\n\nfrom ..chat_template_utils import clone_chat_template\nfrom ..data_utils import is_conversational\nfrom ..models import get_act_offloading_ctx_manager\nfrom .base_trainer import _BaseTrainer\nfrom .reward_config import RewardConfig\nfrom .utils import create_model_from_path, disable_dropout_in_model, get_config_model_id, pad, remove_none_values\n\n\nif is_peft_available():\n from peft import PeftConfig, PeftModel, get_peft_model\n\n\nlogger = get_logger(__name__)\n\n\n# Loading a CausalLM checkpoint into AutoModelForSequenceClassification triggers harmless warnings:\n# - MISSING score.weight : the new seq-clf head was not in the checkpoint and is randomly initialized.\n# - UNEXPECTED lm_head.weight: the causal LM head is in the checkpoint but absent from seq-clf (>= 4.57.0 only).\n# Both are expected consequences of intentional cross-architecture loading. We suppress them to avoid\n# confusing users.\n\n\n# Old approach using logging filter (for transformers < 4.57.0)\n# Note: in transformers < 4.57.0, only the MISSING score.weight warning is emitted; lm_head.weight is not reported.\n@contextmanager\ndef _suppress_seqcls_cross_arch_keys(logger: logging.Logger):\n pattern = re.compile(\n r\"^Some weights of \\S+ were not initialized from the model checkpoint at \\S+ and are newly initialized: \"\n r\"\\[.*\\]\\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and \"\n r\"inference\\.$\"\n )\n\n class _Filter(logging.Filter):\n def filter(self, record: logging.LogRecord) -> bool:\n return not pattern.search(record.getMessage())\n\n f = _Filter()\n logger.addFilter(f)\n try:\n yield\n finally:\n logger.removeFilter(f)\n\n\n# New approach using scoped override (for transformers >= 4.57.0)\n@contextmanager\ndef _ignore_seqcls_cross_arch_keys():\n # Scoped override: ignore the expected seq-clf head key (newly added) and the causal LM head\n # key (present in the checkpoint but absent from seq-clf).\n old_missing = getattr(GenericForSequenceClassification, \"_keys_to_ignore_on_load_missing\", None)\n old_unexpected = getattr(GenericForSequenceClassification, \"_keys_to_ignore_on_load_unexpected\", None)\n\n merged_missing = list(old_missing) if old_missing is not None else []\n if r\"^score\\.weight$\" not in merged_missing:\n merged_missing.append(r\"^score\\.weight$\")\n\n merged_unexpected = list(old_unexpected) if old_unexpected is not None else []\n if r\"^lm_head\\.\" not in merged_unexpected:\n merged_unexpected.append(r\"^lm_head\\.\")\n\n GenericForSequenceClassification._keys_to_ignore_on_load_missing = merged_missing\n GenericForSequenceClassification._keys_to_ignore_on_load_unexpected = merged_unexpected\n try:\n yield\n finally:\n GenericForSequenceClassification._keys_to_ignore_on_load_missing = old_missing\n GenericForSequenceClassification._keys_to_ignore_on_load_unexpected = old_unexpected\n\n\n# Version-aware wrapper that chooses the appropriate approach\n@contextmanager\ndef suppress_seqcls_warning():\n # Use the new approach for transformers >= 4.57.0, old approach for earlier versions\n # The old approach is needed for 4.56.2 to avoid meta tensor issues with device_map=None\n if Version(transformers.__version__) >= Version(\"4.57.0\"):\n with _ignore_seqcls_cross_arch_keys():\n yield\n else:\n # Get the transformers logger\n transformers_logger = logging.getLogger(\"transformers.modeling_utils\")\n with _suppress_seqcls_cross_arch_keys(transformers_logger):\n yield\n\n\ndef get_dataset_column_names(dataset: Dataset | IterableDataset) -> list[str]:\n return list(next(iter(dataset)).keys()) if dataset.column_names is None else dataset.column_names\n\n\n@dataclass\nclass DataCollatorForPreference(DataCollatorMixin):\n \"\"\"\n Data collator used for preference data. Inputs are dynamically padded to the maximum length of a batch.\n\n This collator expects each example in the input list to be a dictionary containing the `\"chosen_ids\"` and\n `\"rejected_ids\"` keys. The collator returns a dictionary containing the following keys:\n - `\"input_ids\"`: Tensor of input IDs, padded to the maximum length of the batch. The first half of the batch\n corresponds to the `\"chosen_ids\"` and the second half to the `\"rejected_ids\"`.\n - `\"attention_mask\"`: Tensor of attention mask, padded to the maximum length of the batch.\n\n Optionally, the examples can contain a `\"margin\"` key, in which case the returned dictionary will also contain a\n `\"margin\"` key with a tensor of margins.\n\n Args:\n pad_token_id (`int`):\n Token ID to use for padding.\n pad_to_multiple_of (`int`, *optional*):\n If set, the sequences will be padded to a multiple of this value.\n return_tensors (`str`, *optional*, defaults to `\"pt\"`):\n Type of Tensor to return. Only `\"pt\"` is currently supported.\n\n Examples:\n ```python\n >>> from trl.trainer.reward_trainer import DataCollatorForPreference\n\n >>> collator = DataCollatorForPreference(pad_token_id=0)\n >>> examples = [\n ... {\"chosen_ids\": [1, 2, 3], \"rejected_ids\": [4, 5]},\n ... {\"chosen_ids\": [6, 7], \"rejected_ids\": [8]},\n ... ]\n >>> collator(examples)\n {'input_ids': tensor([[1, 2, 3],\n [6, 7, 0],\n [4, 5, 0],\n [8, 0, 0]]),\n 'attention_mask': tensor([[1, 1, 1],\n [1, 1, 0],\n [1, 1, 0],\n [1, 0, 0]])}\n\n >>> examples = [\n ... {\"chosen_ids\": [1, 2, 3], \"rejected_ids\": [4, 5], \"margin\": 0.5},\n ... {\"chosen_ids\": [6, 7], \"rejected_ids\": [8], \"margin\": 0.0},\n ... ]\n >>> collator(examples)\n {'input_ids': tensor([[1, 2, 3],\n [6, 7, 0],\n [4, 5, 0],\n [8, 0, 0]]),\n 'attention_mask': tensor([[1, 1, 1],\n [1, 1, 0],\n [1, 1, 0],\n [1, 0, 0]]),\n 'margin': tensor([0.5, 0.0])}\n ```\n \"\"\"\n\n pad_token_id: int\n pad_to_multiple_of: int | None = None\n return_tensors: str = \"pt\"\n\n def torch_call(self, examples: list[dict[str, Any]]) -> dict[str, Any]:\n # Convert to tensor\n chosen_ids = [torch.tensor(example[\"chosen_ids\"]) for example in examples]\n rejected_ids = [torch.tensor(example[\"rejected_ids\"]) for example in examples]\n if \"margin\" in examples[0]:\n margins = torch.tensor([example[\"margin\"] for example in examples], dtype=torch.float)\n input_ids = chosen_ids + rejected_ids\n attention_mask = [torch.ones_like(ids) for ids in input_ids]\n\n output = {}\n\n # Pad\n output[\"input_ids\"] = pad(\n input_ids,\n padding_value=self.pad_token_id,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n )\n output[\"attention_mask\"] = pad(\n attention_mask,\n padding_value=0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n )\n if \"margin\" in examples[0]:\n output[\"margin\"] = margins\n return output\n\n\nclass RewardTrainer(_BaseTrainer):\n \"\"\"\n Trainer for Outcome-supervised Reward Models (ORM).\n\n This class is a wrapper around the [`~transformers.Trainer`] class and inherits all of its attributes and methods.\n\n Example:\n\n ```python\n from trl import RewardTrainer\n from datasets import load_dataset\n\n dataset = load_dataset(\"trl-lib/ultrafeedback_binarized\", split=\"train\")\n\n trainer = RewardTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n train_dataset=dataset,\n )\n trainer.train()\n ```\n\n Args:\n model (`str` or [`~transformers.PreTrainedModel`] or [`~peft.PeftModel`]):\n Model to be trained. Can be either:\n\n - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using `AutoModelForSequenceClassification.from_pretrained` with the keyword arguments in\n `args.model_init_kwargs`.\n - A sequence classification [`~transformers.PreTrainedModel`] object.\n - A sequence classification [`~peft.PeftModel`] object.\n args ([`RewardConfig`], *optional*):\n Configuration for this trainer. If `None`, a default configuration is used.\n data_collator ([`~transformers.DataCollator`], *optional*):\n Function to use to form a batch from a list of elements of the processed `train_dataset` or `eval_dataset`.\n Will default to [`~trainer.reward_trainer.DataCollatorForPreference`].\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n Dataset to use for training. This trainer supports [preference](#preference) type (both implicit and\n explicit prompt). The format of the samples can be either:\n\n - [Standard](dataset_formats#standard): Each sample contains plain text.\n - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role\n and content).\n\n The trainer also supports processed datasets (tokenized) as long as they contain `chosen_ids` and\n `rejected_ids` fields.\n eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Dataset | IterableDataset]`):\n Dataset to use for evaluation. It must meet the same requirements as `train_dataset`.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], *optional*):\n Tokenizer used to process the data. If `None`, the tokenizer is loaded from the model's name with\n [`~transformers.AutoTokenizer.from_pretrained`]. A padding token, `processing_class.pad_token`, must be\n set. If the processing class has not set a padding token, `processing_class.eos_token` will be used as the\n default.\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function that will be used to compute metrics at evaluation. Must take a\n [`~transformers.EvalPrediction`] and return a dictionary string to metric values. When passing\n [`RewardConfig`] with `batch_eval_metrics` set to `True`, your `compute_metrics` function must take a\n boolean `compute_result` argument. This will be triggered after the last eval batch to signal that the\n function needs to calculate and return the global summary statistics rather than accumulating the\n batch-level statistics.\n callbacks (list of [`~transformers.TrainerCallback`], *optional*):\n List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed\n in [here](https://huggingface.co/docs/transformers/main_classes/callback).\n\n If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`]\n method.\n optimizers (`tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None]`, *optional*, defaults to `(None, None)`):\n A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your\n model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`.\n optimizer_cls_and_kwargs (`tuple[Type[torch.optim.Optimizer], Dict[str, Any]]`, *optional*):\n A tuple containing the optimizer class and keyword arguments to use. Overrides `optim` and `optim_args` in\n `args`. Incompatible with the `optimizers` argument.\n\n Unlike `optimizers`, this argument avoids the need to place model parameters on the correct devices before\n initializing the Trainer.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`, *optional*):\n A function that preprocess the logits right before caching them at each evaluation step. Must take two\n tensors, the logits and the labels, and return the logits once processed as desired. The modifications made\n by this function will be reflected in the predictions received by `compute_metrics`.\n\n Note that the labels (second parameter) will be `None` if the dataset does not have them.\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration used to wrap the model. If `None`, the model is not wrapped. Note that if the loaded\n model is a causal LM, it's highly recommended to set `modules_to_save=[\"score\"]` in the PEFT configuration\n to ensure that the reward head is properly trained.\n \"\"\"\n\n _tag_names = [\"trl\", \"reward-trainer\"]\n _name = \"Reward\"\n _template_file = \"rm_model_card.md\"\n\n def __init__(\n self,\n model: \"str | PreTrainedModel | PeftModel\",\n args: RewardConfig | None = None,\n data_collator: DataCollator | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | IterableDataset | dict[str, Dataset | IterableDataset] | None = None,\n processing_class: PreTrainedTokenizerBase | None = None,\n compute_metrics: Callable[[EvalPrediction], dict] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None] = (None, None),\n optimizer_cls_and_kwargs: tuple[type[torch.optim.Optimizer], dict[str, Any]] | None = None,\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n peft_config: \"PeftConfig | None\" = None,\n ):\n # Args\n if args is None:\n model_name = model if isinstance(model, str) else get_config_model_id(model.config)\n model_name = model_name.split(\"/\")[-1]\n args = RewardConfig(f\"{model_name}-Reward\")\n\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n elif isinstance(train_dataset, IterableDataset):\n # IterableDataset requires dispatch_batches=False because Accelerate's dispatch mode may try to concatenate\n # batches from multiple processes, leading to mismatch errors.\n if args.accelerator_config.dispatch_batches is True:\n logger.warning(\n \"You are using an `IterableDataset` for training with `dispatch_batches=True`. `dispatch_batches` \"\n \"is forced to `False` when using an `IterableDataset`. To remove this warning, unset \"\n \"`dispatch_batches` in `RewardConfig` or set it to `False`.\"\n )\n args.accelerator_config.dispatch_batches = False\n\n # Model\n # As AutoModelForSequenceClassification.from_pretrained() will add a random head for the model, set_seed must\n # be done before loading the model to ensure reproducibility.\n set_seed(args.seed)\n if isinstance(model, str):\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n model_init_kwargs[\"num_labels\"] = 1 # the only output of the model is the reward score\n with suppress_seqcls_warning():\n model = create_model_from_path(model, AutoModelForSequenceClassification, **model_init_kwargs)\n else:\n if args.model_init_kwargs is not None:\n logger.warning(\n \"You passed `model_init_kwargs` to the `RewardConfig`, but your model is already instantiated. \"\n \"The `model_init_kwargs` will be ignored.\"\n )\n # Validate that the model has num_labels = 1 (required for reward models)\n if getattr(model.config, \"num_labels\", None) != 1:\n raise ValueError(\n f\"The model has `num_labels={model.config.num_labels}`, but reward models require `num_labels=1` \"\n \"to output a single scalar reward per sequence. Please instantiate your model with `num_labels=1` \"\n \"or pass a model name as a string to have it configured automatically.\"\n )\n\n # Processing class\n if processing_class is None:\n processing_class = AutoTokenizer.from_pretrained(get_config_model_id(model.config))\n\n # Handle pad token for processors or tokenizers\n if args.eos_token is not None:\n eos_token = args.eos_token\n eos_token_id = processing_class.convert_tokens_to_ids(eos_token)\n if eos_token_id is None:\n raise ValueError(\n f\"The specified `eos_token` ('{eos_token}') is not found in the vocabulary of the given \"\n f\"`processing_class` ({processing_class.__class__.__name__}). Ensure that the `eos_token` exists \"\n \"in the vocabulary before using it as an EOS token.\"\n )\n processing_class.eos_token_id = eos_token_id\n\n if args.chat_template_path is not None:\n if os.path.isfile(args.chat_template_path) and args.chat_template_path.endswith((\".jinja\", \".j2\")):\n with open(args.chat_template_path, encoding=\"utf-8\") as chat_template_file:\n processing_class.chat_template = chat_template_file.read()\n added_tokens = []\n else:\n model, processing_class, added_tokens = clone_chat_template(\n model, processing_class, args.chat_template_path\n )\n else:\n added_tokens = []\n\n # PEFT configuration and model wrapping\n if peft_config is not None:\n if added_tokens:\n # Ensure that the added tokens are trainable\n if peft_config.trainable_token_indices is None:\n peft_config.trainable_token_indices = {\"embed_tokens\": added_tokens}\n elif \"embed_tokens\" not in peft_config.trainable_token_indices:\n peft_config.trainable_token_indices[\"embed_tokens\"] = added_tokens\n else:\n peft_config.trainable_token_indices[\"embed_tokens\"].extend(added_tokens)\n\n # Ensure that the lm_head is trainable\n if peft_config.modules_to_save is None or \"lm_head\" not in peft_config.modules_to_save:\n logger.warning(\n \"Cloning chat template added new tokens to the tokenizer, but 'lm_head' is not in PEFT's \"\n \"`modules_to_save`. As a result, the model may not learn to generate outputs with these new \"\n \"tokens, leading to degraded generation quality. To fix this, add \"\n \"`modules_to_save=['lm_head']` to your PEFT configuration.\"\n )\n\n if peft_config.modules_to_save is None:\n peft_config.modules_to_save = [\"lm_head\"]\n else:\n peft_config.modules_to_save.append(\"lm_head\")\n\n if is_peft_available() and is_peft_model(model) and peft_config is not None:\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first merge \"\n \"and unload the existing adapter, save the resulting base model, and then pass that base model along \"\n \"with the new `peft_config` to the trainer.\"\n )\n\n # Create PEFT model\n if peft_config is not None:\n model = get_peft_model(model, peft_config)\n\n # When using gradient checkpointing with PEFT, we need to enable input gradients. transformers.Trainer normally\n # handles this, but a bug currently prevents it; see https://github.com/huggingface/transformers/issues/42489\n if is_peft_available() and is_peft_model(model) and args.gradient_checkpointing:\n model.enable_input_require_grads()\n\n # When using QLoRA, the PEFT adapter weights are converted to bf16 to follow the recommendations from the\n # original paper (see https://huggingface.co/papers/2305.14314, paragraph 3). Normally, this can be done by\n # passing `autocast_adapter_dtype=False` to `get_peft_model`, but this option is not yet supported for\n # quantized models. See: https://github.com/huggingface/peft/issues/2889\n # Non-quantized models do not have the `is_loaded_in_{8,4}bit` attributes, whereas quantized models do\n if getattr(model, \"is_loaded_in_4bit\", False) or getattr(model, \"is_loaded_in_8bit\", False):\n for param in model.parameters():\n if param.requires_grad:\n param.data = param.data.to(torch.bfloat16)\n\n # Disable dropout in the model\n if args.disable_dropout:\n disable_dropout_in_model(model)\n\n # Pad token (needed for SequenceClassification models)\n # If not provided, use the one from the processing class or the eos token if the processing class does not have\n # a pad token.\n pad_token = args.pad_token or processing_class.pad_token or processing_class.eos_token\n pad_token_id = processing_class.convert_tokens_to_ids(pad_token)\n if pad_token_id is None:\n raise ValueError(\n f\"The specified `pad_token` ('{pad_token}') is not found in the vocabulary of the given \"\n f\"`processing_class` ({processing_class.__class__.__name__}). Ensure that the `pad_token` exists \"\n \"in the vocabulary before using it as a padding token.\"\n )\n model.config.pad_token_id = pad_token_id\n processing_class.pad_token_id = pad_token_id\n\n # Data collator\n if data_collator is None:\n data_collator = DataCollatorForPreference(\n pad_token_id=pad_token_id,\n pad_to_multiple_of=args.pad_to_multiple_of,\n )\n\n # Dataset\n train_dataset = self._prepare_dataset(train_dataset, processing_class, args, \"train\")\n if eval_dataset is not None:\n if isinstance(eval_dataset, dict):\n eval_dataset = {\n key: self._prepare_dataset(dataset, processing_class, args, key)\n for key, dataset in eval_dataset.items()\n }\n else:\n eval_dataset = self._prepare_dataset(eval_dataset, processing_class, args, \"eval\")\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n optimizer_cls_and_kwargs=optimizer_cls_and_kwargs,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n # During evaluation, Trainer calls compute_loss() only if can_return_loss is True and label_names is empty.\n self.can_return_loss = True\n self.label_names = []\n\n # Initialize activation offloading context\n if self.args.activation_offloading:\n self.maybe_activation_offload_context = get_act_offloading_ctx_manager(model=self.model)\n else:\n self.maybe_activation_offload_context = contextlib.nullcontext()\n\n self.aux_loss_enabled = getattr(model.config, \"output_router_logits\", False)\n\n # Initialize the metrics\n self._metrics = {\"train\": defaultdict(list), \"eval\": defaultdict(list)}\n self._total_train_tokens = 0\n\n # Add tags to the model\n self.model.add_model_tags(self._tag_names)\n\n def _prepare_dataset(\n self,\n dataset: Dataset | IterableDataset,\n processing_class: PreTrainedTokenizerBase,\n args: RewardConfig,\n dataset_name: str,\n ) -> Dataset | IterableDataset:\n # Tabular backends like Arrow/Parquet insert `None` for mismatched keys in nested structures. Clean them from\n # sampled data.\n if isinstance(dataset, Dataset): # IterableDataset does not support `with_transform`\n dataset = dataset.with_transform(remove_none_values)\n\n # If the dataset is already preprocessed (tokenized), skip the processing steps.\n column_names = get_dataset_column_names(dataset)\n is_processed = \"chosen_ids\" in column_names and \"rejected_ids\" in column_names\n has_legacy_processed_columns = \"chosen_input_ids\" in column_names and \"rejected_input_ids\" in column_names\n if has_legacy_processed_columns and not is_processed:\n warnings.warn(\n \"Detected legacy dataset columns `chosen_input_ids`/`rejected_input_ids`; they are deprecated and \"\n \"will not be supported in v1. Please migrate to `chosen_ids`/`rejected_ids`.\",\n FutureWarning,\n stacklevel=2,\n )\n dataset = dataset.rename_column(\"chosen_input_ids\", \"chosen_ids\")\n dataset = dataset.rename_column(\"rejected_input_ids\", \"rejected_ids\")\n is_processed = True\n\n # Build the kwargs for the `map` function\n map_kwargs = {}\n if isinstance(dataset, Dataset): # IterableDataset does not support num_proc\n map_kwargs[\"num_proc\"] = args.dataset_num_proc\n\n with PartialState().main_process_first():\n if not is_processed:\n # Add EOS token to the end of the sequences if needed\n first_example = next(iter(dataset))\n if not is_conversational(first_example):\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Adding EOS to {dataset_name} dataset\"\n\n def add_eos(example, eos_token):\n if not example[\"chosen\"].endswith(eos_token):\n example[\"chosen\"] = example[\"chosen\"] + eos_token\n if \"rejected\" in example and not example[\"rejected\"].endswith(eos_token):\n example[\"rejected\"] = example[\"rejected\"] + eos_token\n return example\n\n dataset = dataset.map(\n add_eos,\n fn_kwargs={\"eos_token\": processing_class.eos_token},\n **map_kwargs,\n )\n\n # Tokenize the dataset\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Tokenizing {dataset_name} dataset\"\n\n def tokenize_fn(example, processing_class):\n tools = example.get(\"tools\")\n tools = json.loads(tools) if isinstance(tools, str) else tools\n if \"prompt\" in example: # explicit prompt case\n example[\"chosen\"] = example[\"prompt\"] + example[\"chosen\"]\n example[\"rejected\"] = example[\"prompt\"] + example[\"rejected\"]\n\n if is_conversational(example):\n chosen_ids = processing_class.apply_chat_template(\n example[\"chosen\"],\n tools=tools,\n return_dict=True,\n **example.get(\"chat_template_kwargs\", {}),\n )[\"input_ids\"]\n rejected_ids = processing_class.apply_chat_template(\n example[\"rejected\"],\n tools=tools,\n return_dict=True,\n **example.get(\"chat_template_kwargs\", {}),\n )[\"input_ids\"]\n output = {\"chosen_ids\": chosen_ids, \"rejected_ids\": rejected_ids}\n else:\n output = {\n \"chosen_ids\": processing_class(text=example[\"chosen\"])[\"input_ids\"],\n \"rejected_ids\": processing_class(text=example[\"rejected\"])[\"input_ids\"],\n }\n return output\n\n dataset = dataset.map(tokenize_fn, fn_kwargs={\"processing_class\": processing_class}, **map_kwargs)\n\n # Filter samples that are longer than `max_length`\n if args.max_length is not None:\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Filtering {dataset_name} >{args.max_length} tokens\"\n dataset = dataset.filter(\n lambda example: len(example[\"chosen_ids\"]) <= args.max_length\n and len(example[\"rejected_ids\"]) <= args.max_length,\n **map_kwargs,\n )\n\n return dataset\n\n def _set_signature_columns_if_needed(self):\n # If `self.args.remove_unused_columns` is True, non-signature columns are removed.\n # By default, this method sets `self._signature_columns` to the model's expected inputs (usually, \"input_ids\"\n # and \"attention_mask\").\n if self._signature_columns is None:\n self._signature_columns = [\"chosen_ids\", \"rejected_ids\", \"margin\"]\n\n def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):\n mode = \"train\" if self.model.training else \"eval\"\n\n # If not set, defaults from model config and may warn since cache isn't compatible with gradient checkpointing\n inputs[\"use_cache\"] = False\n outputs = model(**inputs)\n\n # Split the rewards into chosen and rejected\n rewards_chosen, rewards_rejected = torch.chunk(outputs.logits.squeeze(-1), chunks=2)\n\n # Calculate loss, optionally modulate with margin\n if \"margin\" in inputs:\n loss = -nn.functional.logsigmoid(rewards_chosen - rewards_rejected - inputs[\"margin\"]).mean()\n else:\n loss = -nn.functional.logsigmoid(rewards_chosen - rewards_rejected).mean()\n\n if self.args.center_rewards_coefficient is not None:\n loss += self.args.center_rewards_coefficient * torch.mean((rewards_chosen + rewards_rejected) ** 2)\n\n if mode == \"train\":\n num_tokens_in_batch = self.accelerator.gather_for_metrics(inputs[\"attention_mask\"].sum()).sum().item()\n self._total_train_tokens += num_tokens_in_batch\n self._metrics[mode][\"num_tokens\"] = [self._total_train_tokens]\n\n # Compute min, mean, max, accuracy and margin\n with torch.no_grad():\n all_rewards = self.accelerator.gather(outputs.logits)\n self._metrics[mode][\"min_reward\"].append(all_rewards.min().item())\n self._metrics[mode][\"mean_reward\"].append(all_rewards.mean().item())\n self._metrics[mode][\"max_reward\"].append(all_rewards.max().item())\n\n mean_accuracy = (rewards_chosen > rewards_rejected).float().mean()\n mean_accuracy = self.accelerator.gather_for_metrics(mean_accuracy).mean().item()\n self._metrics[mode][\"accuracy\"].append(mean_accuracy)\n\n mean_margin = (rewards_chosen - rewards_rejected).mean()\n mean_margin = self.accelerator.gather_for_metrics(mean_margin).mean()\n self._metrics[mode][\"margin\"].append(mean_margin.item())\n\n return (loss, outputs) if return_outputs else loss\n\n # Override training step to add activation offloading context.\n def training_step(self, *args, **kwargs):\n with self.maybe_activation_offload_context:\n return super().training_step(*args, **kwargs)\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n mode = \"train\" if self.model.training else \"eval\"\n metrics = {key: sum(val) / len(val) for key, val in self._metrics[mode].items()} # average the metrics\n\n # This method can be called both in training and evaluation. When called in evaluation, the keys in `logs`\n # start with \"eval_\". We need to add the prefix \"eval_\" to the keys in `metrics` to match the format.\n if mode == \"eval\":\n metrics = {f\"eval_{key}\": val for key, val in metrics.items()}\n\n logs = {**logs, **metrics}\n super().log(logs, start_time)\n self._metrics[mode].clear()\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/trainer/rloo_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom .base_config import _BaseConfig\n\n\n@dataclass\nclass RLOOConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`RLOOTrainer`].\n\n This class includes only the parameters that are specific to RLOO training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n > Parameters that control the model and reference model\n\n model_init_kwargs (`str`, `dict[str, Any]`, *optional*):\n Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model`\n argument of the [`RLOOTrainer`] is provided as a string.\n disable_dropout (`bool`, *optional*, defaults to `False`):\n Whether to disable dropout in the model. This is useful for training with a reference model, as it prevents\n the model from generating different logprobs for the same input.\n\n > Parameters that control the data preprocessing\n\n remove_unused_columns (`bool`, *optional*, defaults to `False`):\n Whether to only keep the column `\"prompt\"` in the dataset. If you use a custom reward function that\n requires any column other than `\"prompts\"` and `\"completions\"`, you should keep this to `False`.\n num_generations (`int`, *optional*, defaults to `2`):\n Number of generations per prompt to sample. The effective batch size (num_processes * per_device_batch_size\n * gradient_accumulation_steps) must be evenly divisible by this value.\n num_generations_eval (`int` or `None`, *optional*):\n Number of generations to sample during evaluation. This allows using fewer generations during evaluation to\n save computation. If `None`, uses the value of `num_generations`.\n max_completion_length (`int` or `None`, *optional*, defaults to `256`):\n Maximum length of the generated completion.\n ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):\n This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,\n improving generation speed. However, disabling this option allows training models that exceed the VRAM\n capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible\n with vLLM generation.\n shuffle_dataset (`bool`, *optional*, defaults to `True`):\n Whether to shuffle the training dataset.\n pad_to_multiple_of (`int`, *optional*):\n If set, the prompts ids and completions ids will be padded to a multiple of this value.\n\n > Parameters that control generation\n\n generation_batch_size: (`int`, *optional*):\n Batch size to use for generation. If `None`, it defaults to the effective training batch size:\n `per_device_train_batch_size * num_processes * steps_per_generation`. In other words, there is one\n generation batch processed per optimization step. Mutually exclusive with `steps_per_generation`.\n steps_per_generation: (`int`, *optional*):\n Number of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`. Mutually exclusive\n with `generation_batch_size`.\n temperature (`float`, defaults to `1.0`):\n Temperature for sampling. The higher the temperature, the more random the completions.\n top_p (`float`, *optional*, defaults to `1.0`):\n Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to\n `1.0` to consider all tokens.\n top_k (`int`, *optional*, defaults to `0`):\n Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, top-k-filtering is\n disabled and all tokens are considered.\n min_p (`float`, *optional*):\n Minimum token probability, which will be scaled by the probability of the most likely token. It must be a\n value between `0.0` and `1.0`. Typical values are in the `0.01-0.2` range.\n generation_kwargs (`dict[str, Any]`, *optional*):\n Additional keyword arguments to pass to [`~transformers.GenerationConfig`] (if using transformers) or\n `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the\n generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that conflict\n with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.\n chat_template_kwargs (`dict[str, Any]`, *optional*):\n Additional keyword arguments to pass to the `apply_chat_template` function when generating completions.\n repetition_penalty (`float`, *optional*, defaults to `1.0`):\n Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far.\n Values > `1.0` encourage the model to use new tokens, while values < `1.0` encourage the model to repeat\n tokens.\n use_transformers_paged (`bool`, *optional*, defaults to `False`):\n Whether to use the `transformers` paged implementation for generation. If set to `True`, the `transformers`\n paged implementation will be used for generation instead of the default padded implementation. This\n parameter is only effective when `use_vllm` is set to `False`.\n cache_implementation (`str`, *optional*):\n Implementation of the cache method for faster generation when `use_vllm` is set to `False`.\n\n > Parameters that control generation acceleration powered by vLLM\n\n use_vllm (`bool`, *optional*, defaults to `False`):\n Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation\n instead of the default model.generate(). Requires `vllm` to be installed.\n vllm_mode (`str`, *optional*, defaults to `\"colocate\"`):\n Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `\"server\"` or\n `\"colocate\"`.\n\n - `\"server\"`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM\n server is running (start with `trl vllm-serve`).\n - `\"colocate\"`: vLLM will run in the same process and share the training GPUs. This avoids the need for a\n separate server but may cause resource contention with training.\n vllm_model_impl (`str`, *optional*, defaults to `\"vllm\"`):\n Model implementation to use for vLLM. Must be one of `\"transformers\"` or `\"vllm\"`. `\"transformers\"`: Use\n the `transformers` backend for model implementation. `\"vllm\"`: Use the `vllm` library for model\n implementation.\n vllm_structured_outputs_regex (`str`, *optional*):\n Regex for vLLM structured outputs. If `None` (default), structured outputs is disabled.\n\n > Parameters that control the vLLM server (only used when `vllm_mode` is `\"server\"`)\n\n vllm_server_base_url (`str`, *optional*):\n Base URL for the vLLM server (e.g., `\"http://localhost:8000\"`). If provided, `vllm_server_host` and\n `vllm_server_port` are ignored.\n vllm_server_host (`str`, *optional*, defaults to `\"0.0.0.0\"`):\n Host of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided.\n vllm_server_port (`int`, *optional*, defaults to `8000`):\n Port of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided.\n vllm_server_timeout (`float`, *optional*, defaults to `240.0`):\n Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the\n timeout, a `ConnectionError` is raised.\n vllm_group_port (`int`, *optional*, defaults to `51216`):\n Port number for the weight update group. This is used to communicate with the vLLM server. Unless the port\n is occupied, there is no need to change it.\n\n > Parameters that control colocated vLLM execution (only used when `vllm_mode` is `\"colocate\"`)\n\n vllm_gpu_memory_utilization (`float`, *optional*, defaults to `0.3`):\n Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to\n `\"colocate\"`. If you are using `vllm_mode=\"server\"`, this parameter must be passed separately when\n launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.\n vllm_max_model_length (`int`, *optional*):\n Context window for vLLM. Set it to at least the maximum prompt length in the dataset plus\n `max_completion_length`; if omitted, it is inferred from the model config.\n vllm_tensor_parallel_size (`int`, *optional*, defaults to `1`):\n Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to\n `\"colocate\"`. If you are using `vllm_mode=\"server\"`, this parameter must be passed separately when\n launching the vLLM server via the `--vllm_tensor_parallel_size` flag.\n vllm_enable_sleep_mode (`bool`, *optional*, defaults to `False`):\n Enable vLLM sleep mode to offload weights/cache during the optimizer step. Keeps GPU memory usage low, but\n waking the engine adds hostโ€“device transfer latency.\n\n > Parameters that control the training\n\n beta (`float`, *optional*, defaults to `0.05`):\n KL coefficient. If `0.0`, the reference model is not loaded, reducing memory usage and improving training\n speed.\n num_iterations (`int`, *optional*, defaults to `1`):\n Number of iterations per batch (denoted as ฮผ in the algorithm).\n epsilon (`float`, *optional*, defaults to `0.2`):\n Epsilon value for clipping.\n epsilon_high (`float`, *optional*):\n Upper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound\n specified in argument `epsilon`. Paper [DAPO](https://huggingface.co/papers/2503.14476) recommends `0.28`.\n reward_weights (`list[float]`, *optional*):\n Weights for each reward function. Must match the number of reward functions. If `None`, all rewards are\n weighted equally with weight `1.0`.\n normalize_advantages (`bool`, *optional*, defaults to `False`):\n Whether to normalize advantages. Normalization is done per generation batch to have mean `0.0` and standard\n deviation of `1.0`.\n reward_clip_range (`tuple[float, float]`, *optional*):\n Clip range for rewards as (min, max). If `None`, no clipping is applied.\n mask_truncated_completions (`bool`, *optional*, defaults to `False`):\n When enabled, truncated completions are excluded from the loss calculation, preventing them from being\n incorrectly penalized and introducing noise during training. According to the\n [DAPO](https://huggingface.co/papers/2503.14476) paper, this is a good practice for training stability.\n sync_ref_model (`bool`, *optional*, defaults to `False`):\n Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using\n the `ref_model_mixup_alpha` parameter. This synchronization originates from the\n [TR-DPO](https://huggingface.co/papers/2404.09656) paper.\n ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`):\n ฮฑ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix\n between the current policy and the previous reference policy during updates. The reference policy is\n updated according to the equation: `ฯ€_ref = ฮฑ * ฯ€_ฮธ + (1 - ฮฑ) * ฯ€_ref_prev`. To use this parameter, you\n must set `sync_ref_model=True`.\n ref_model_sync_steps (`int`, *optional*, defaults to `512`):\n ฯ„ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how\n frequently the current policy is synchronized with the reference policy. To use this parameter, you must\n set `sync_ref_model=True`.\n\n > Parameters that control the logging\n\n log_completions (`bool`, *optional*, defaults to `False`):\n Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed,\n it prints the sample. If `wandb` and/or `trackio` logging is enabled, it logs it to `wandb` and/or\n `trackio`.\n num_completions_to_print (`int`, *optional*):\n Number of completions to print with `rich`. If `None`, all completions are logged.\n log_unique_prompts (`bool`, *optional*, defaults to `False`):\n Whether to log unique prompts. If `True`, only unique prompts are logged. If `False`, all prompts are\n logged.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `1e-6` instead of `5e-5`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=1e-6,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n # Parameters that control the model and reference model\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments for `transformers.AutoModelForCausalLM.from_pretrained`, used when the `model` \"\n \"argument of the `RLOOTrainer` is provided as a string.\"\n },\n )\n disable_dropout: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to disable dropout in the model. This is useful for training with a reference model, as \"\n \"it prevents the model from generating different logprobs for the same input.\"\n },\n )\n\n # Parameters that control the data preprocessing\n # The default value remove_unused_columns is overwritten from the parent class, because in RLOO we usually rely on\n # additional columns to compute the reward\n remove_unused_columns: bool | None = field(\n default=False,\n metadata={\n \"help\": \"Whether to only keep the column 'prompt' in the dataset. If you use a custom reward function \"\n \"that requires any column other than 'prompts' and 'completions', you should keep this to `False`.\"\n },\n )\n num_generations: int | None = field(\n default=2,\n metadata={\n \"help\": \"Number of generations to sample. The effective batch size (num_processes * per_device_batch_size \"\n \"* gradient_accumulation_steps) must be evenly divisible by this value.\"\n },\n )\n num_generations_eval: int | None = field(\n default=None,\n metadata={\n \"help\": \"Number of generations to sample during evaluation. This allows using fewer generations during \"\n \"evaluation to save computation. If `None`, uses the value of `num_generations`.\"\n },\n )\n max_completion_length: int | None = field(\n default=256,\n metadata={\"help\": \"Maximum length of the generated completion.\"},\n )\n ds3_gather_for_generation: bool = field(\n default=True,\n metadata={\n \"help\": \"This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for \"\n \"generation, improving generation speed. However, disabling this option allows training models that \"\n \"exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option \"\n \"is not compatible with vLLM generation.\"\n },\n )\n shuffle_dataset: bool | None = field(\n default=True,\n metadata={\"help\": \"Whether to shuffle the training dataset.\"},\n )\n pad_to_multiple_of: int | None = field(\n default=None,\n metadata={\"help\": \"If set, the prompts ids and completions ids will be padded to a multiple of this value.\"},\n )\n\n # Parameters that control generation\n generation_batch_size: int | None = field(\n default=None,\n metadata={\n \"help\": \"Batch size to use for generation. If `None`, it defaults to the effective training batch size: \"\n \"`per_device_train_batch_size * num_processes * steps_per_generation`.\"\n },\n )\n steps_per_generation: int | None = field(\n default=None,\n metadata={\"help\": \"Number of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`.\"},\n )\n temperature: float = field(\n default=1.0,\n metadata={\"help\": \"Temperature for sampling. The higher the temperature, the more random the completions.\"},\n )\n top_p: float = field(\n default=1.0,\n metadata={\n \"help\": \"Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. \"\n \"Set to 1.0 to consider all tokens.\"\n },\n )\n top_k: int = field(\n default=0,\n metadata={\n \"help\": \"Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, \"\n \"top-k-filtering is disabled and all tokens are considered.\"\n },\n )\n min_p: float | None = field(\n default=None,\n metadata={\n \"help\": \"Minimum token probability, which will be scaled by the probability of the most likely token. It \"\n \"must be a value between 0.0 and 1.0. Typical values are in the 0.01-0.2 range.\"\n },\n )\n generation_kwargs: dict | None = field(\n default=None,\n metadata={\n \"help\": \"Additional keyword arguments to pass to `GenerationConfig` (if using transformers) or \"\n \"`SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the \"\n \"generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that \"\n \"conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.\"\n },\n )\n chat_template_kwargs: dict | None = field(\n default=None,\n metadata={\n \"help\": \"Additional keyword arguments to pass to the `apply_chat_template` function when generating \"\n \"completions.\"\n },\n )\n repetition_penalty: float = field(\n default=1.0,\n metadata={\n \"help\": \"Float that penalizes new tokens based on whether they appear in the prompt and the generated \"\n \"text so far. Values > 1.0 encourage the model to use new tokens, while values < 1.0 encourage the model \"\n \"to repeat tokens.\"\n },\n )\n use_transformers_paged: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use the `transformers` paged implementation for generation. If set to `True`, the \"\n \"`transformers` paged implementation will be used for generation instead of the default padded \"\n \"implementation. This parameter is only effective when `use_vllm` is set to `False`.\"\n },\n )\n cache_implementation: str | None = field(\n default=None,\n metadata={\"help\": \"Implementation of the cache method for faster generation when use_vllm is set to False.\"},\n )\n\n # Parameters that control generation acceleration powered by vLLM\n use_vllm: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for \"\n \"generation instead of the default model.generate(). Requires `vllm` to be installed.\"\n },\n )\n vllm_mode: str = field(\n default=\"colocate\",\n metadata={\n \"help\": \"Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `'server'` or \"\n \"`'colocate'`. `'server'`: The trainer will send generation requests to a separate vLLM server. Make sure \"\n \"a TRL vLLM server is running (start with `trl vllm-serve`). `'colocate'`: vLLM will run in the same \"\n \"process and share the training GPUs. This avoids the need for a separate server but may cause resource \"\n \"contention with training.\"\n },\n )\n vllm_model_impl: str = field(\n default=\"vllm\",\n metadata={\n \"help\": \"Model implementation to use for vLLM. Must be one of `transformers` or `vllm`. `transformers`: \"\n \"Use the `transformers` backend for model implementation. `vllm`: Use the `vllm` library for \"\n \"model implementation.\"\n },\n )\n vllm_enable_sleep_mode: bool = field(\n default=False,\n metadata={\n \"help\": \"Enable vLLM sleep mode to offload weights/cache during the optimizer step. Keeps GPU memory \"\n \"usage low, but waking the engine adds hostโ€“device transfer latency.\"\n },\n )\n vllm_structured_outputs_regex: str | None = field(\n default=None,\n metadata={\"help\": \"Regex for vLLM structured outputs. If `None` (default), structured outputs is disabled.\"},\n )\n\n # Parameters that control the vLLM server (only used when `vllm_mode` is `\"server\"`)\n vllm_server_base_url: str | None = field(\n default=None,\n metadata={\n \"help\": \"Base URL for the vLLM server (e.g., 'http://localhost:8000'). If provided, `vllm_server_host` \"\n \"and `vllm_server_port` are ignored.\"\n },\n )\n vllm_server_host: str = field(\n default=\"0.0.0.0\",\n metadata={\"help\": \"Host of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.\"},\n )\n vllm_server_port: int = field(\n default=8000,\n metadata={\"help\": \"Port of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.\"},\n )\n vllm_server_timeout: float = field(\n default=240.0,\n metadata={\n \"help\": \"Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up \"\n \"after the timeout, a `ConnectionError` is raised.\"\n },\n )\n vllm_group_port: int = field(\n default=51216,\n metadata={\n \"help\": \"Port number for the weight update group. This is used to communicate with the vLLM server. \"\n \"Unless the port is occupied, there is no need to change it.\",\n },\n )\n\n # Parameters that control colocated vLLM execution (only used when `vllm_mode` is `\"colocate\"`)\n vllm_gpu_memory_utilization: float = field(\n default=0.3,\n metadata={\n \"help\": \"Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set \"\n \"to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when \"\n \"launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.\"\n },\n )\n vllm_max_model_length: int | None = field(\n default=None,\n metadata={\n \"help\": \"Context window for vLLM. Set it to at least the maximum prompt length in the dataset plus \"\n \"`max_completion_length`; if omitted, it is inferred from the model config.\"\n },\n )\n vllm_tensor_parallel_size: int = field(\n default=1,\n metadata={\n \"help\": \"Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set \"\n \"to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when \"\n \"launching the vLLM server via the `--vllm_tensor_parallel_size` flag.\"\n },\n )\n\n # Parameters that control the training\n beta: float = field(\n default=0.05,\n metadata={\n \"help\": \"KL coefficient. If `0.0`, the reference model is not loaded, reducing memory usage and improving \"\n \"training speed.\"\n },\n )\n num_iterations: int = field(\n default=1,\n metadata={\"help\": \"Number of iterations per batch (denoted as ฮผ in the algorithm).\"},\n )\n epsilon: float = field(\n default=0.2,\n metadata={\"help\": \"Epsilon value for clipping.\"},\n )\n epsilon_high: float | None = field(\n default=None,\n metadata={\n \"help\": \"Upper-bound epsilon value for clipping. If not specified, it defaults to the same value as the \"\n \"lower-bound specified in argument `epsilon`. Paper DAPO recommends `0.28`.\"\n },\n )\n reward_weights: list[float] | None = field(\n default=None,\n metadata={\n \"help\": \"Weights for each reward function. Must match the number of reward functions. If `None`, all \"\n \"rewards are weighted equally with weight `1.0`.\"\n },\n )\n normalize_advantages: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to normalize advantages. Normalization is done per generation batch to have mean `0.0` \"\n \"and standard deviation of `1.0`.\"\n },\n )\n reward_clip_range: tuple[float, float] | None = field(\n default=None,\n metadata={\"help\": \"Clip range for rewards as (min, max). If None, no clipping is applied.\"},\n )\n mask_truncated_completions: bool = field(\n default=False,\n metadata={\n \"help\": \"When enabled, truncated completions are excluded from the loss calculation, preventing them from \"\n \"being incorrectly penalized and introducing noise during training. According to the DAPO paper, this is \"\n \"a good practice for training stability.\"\n },\n )\n sync_ref_model: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to synchronize the reference model with the active model every `ref_model_sync_steps` \"\n \"steps, using the `ref_model_mixup_alpha` parameter.\"\n },\n )\n ref_model_mixup_alpha: float = field(\n default=0.6,\n metadata={\n \"help\": \"ฮฑ parameter from the TR-DPO paper, which controls the mix between the current policy and the \"\n \"previous reference policy during updates. The reference policy is updated according to the equation: \"\n \"`ฯ€_ref = ฮฑ * ฯ€_ฮธ + (1 - ฮฑ) * ฯ€_ref_prev`. To use this parameter, you must set `sync_ref_model=True`.\"\n },\n )\n ref_model_sync_steps: int = field(\n default=512,\n metadata={\n \"help\": \"ฯ„ parameter from the TR-DPO paper, which determines how frequently the current policy is \"\n \"synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`.\"\n },\n )\n\n # Parameters that control the logging\n log_completions: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is \"\n \"installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`.\"\n },\n )\n num_completions_to_print: int | None = field(\n default=None,\n metadata={\"help\": \"Number of completions to print with `rich`. If `None`, all completions are logged.\"},\n )\n log_unique_prompts: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to log unique prompts. If `True`, only unique prompts are logged. If `False`, all \"\n \"prompts are logged.\"\n },\n )\n\n def __post_init__(self):\n super().__post_init__()\n\n num_processes = self.world_size\n # The current default effective batch size\n if self.generation_batch_size is None and self.steps_per_generation is None:\n self.steps_per_generation = self.gradient_accumulation_steps\n self.generation_batch_size = self.per_device_train_batch_size * num_processes * self.steps_per_generation\n elif self.generation_batch_size is not None and self.steps_per_generation is None:\n # Just ensure the value is divisible by the global batch size\n if self.generation_batch_size % (self.per_device_train_batch_size * num_processes) != 0:\n raise ValueError(\n f\"generation_batch_size ({self.generation_batch_size}) must be divisible by the global batch size \"\n f\"({self.per_device_train_batch_size * num_processes}).\"\n )\n self.steps_per_generation = self.generation_batch_size // (\n self.per_device_train_batch_size * num_processes\n )\n elif self.generation_batch_size is None and self.steps_per_generation is not None:\n self.generation_batch_size = self.per_device_train_batch_size * num_processes * self.steps_per_generation\n else:\n raise ValueError(\n \"'generation_batch_size' and 'steps_per_generation' can not be both configured at the same time\"\n )\n\n if self.do_eval and self.eval_strategy != \"no\":\n # Determine the number of generations to use for evaluation\n num_generations = self.num_generations_eval or self.num_generations\n\n # Just ensure the value is divisible by the global batch size\n if (self.per_device_eval_batch_size * num_processes) % num_generations != 0:\n raise ValueError(\n f\"The global eval batch size ({self.per_device_eval_batch_size} * {num_processes}) must be \"\n f\"divisible by the number of generations used for evaluation ({num_generations}).\"\n )\n\n # The generation batch must contain full prompt groups (no partials), so it must be divisible by\n # num_generations.\n if self.generation_batch_size % self.num_generations != 0:\n raise ValueError(\n f\"generation_batch_size ({self.generation_batch_size}) must be divisible by num_generations \"\n f\"({self.num_generations}).\"\n )\n\n if self.num_generations < 2:\n raise ValueError(\n \"RLOO requires at least 2 generations per prompt to calculate the advantages. You provided \"\n f\"{self.num_generations}, which is less than the minimum required.\"\n )\n" }, { "path": "trl/trainer/rloo_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport asyncio\nimport atexit\nimport copy\nimport inspect\nimport textwrap\nimport time\nfrom collections import defaultdict, deque\nfrom collections.abc import Callable\nfrom contextlib import nullcontext\nfrom pathlib import Path\nfrom typing import Any\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.utils.data\nimport transformers\nfrom accelerate.logging import get_logger\nfrom accelerate.utils import gather, gather_object, is_peft_model, set_seed\nfrom datasets import Dataset, IterableDataset\nfrom packaging.version import Version\nfrom torch import nn\nfrom torch.distributed.fsdp import FullyShardedDataParallel as FSDP\nfrom torch.utils.data import Sampler\nfrom transformers import (\n AutoModelForSequenceClassification,\n AutoProcessor,\n AutoTokenizer,\n GenerationConfig,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainerCallback,\n is_trackio_available,\n is_wandb_available,\n)\nfrom transformers.utils import is_peft_available, is_rich_available\n\nfrom ..data_utils import apply_chat_template, is_conversational, prepare_multimodal_messages\nfrom ..extras.profiling import profiling_context, profiling_decorator\nfrom ..generation.vllm_generation import VLLMGeneration\nfrom ..models import prepare_deepspeed, prepare_fsdp, unwrap_model_for_generation\nfrom ..models.utils import disable_gradient_checkpointing\nfrom .base_trainer import _BaseTrainer\nfrom .callbacks import SyncRefModelCallback\nfrom .rloo_config import RLOOConfig\nfrom .utils import (\n RepeatSampler,\n create_model_from_path,\n disable_dropout_in_model,\n entropy_from_logits,\n get_config_model_id,\n identity,\n nanmax,\n nanmin,\n nanstd,\n pad,\n print_prompt_completions_sample,\n selective_log_softmax,\n shuffle_sequence_dict,\n shutdown_event_loop_in_daemon,\n split_pixel_values_by_grid,\n split_tensor_dict,\n start_event_loop_in_daemon,\n unsplit_pixel_values_by_grid,\n use_adapter,\n)\n\n\nif is_peft_available():\n from peft import PeftConfig, PeftModel, get_peft_model\n\n\nif is_wandb_available():\n import wandb\n\nif is_trackio_available():\n import trackio\n\n\nlogger = get_logger(__name__)\n\n# A reward function can be a string, interpreted as a model ID and loaded as a pretrained model, a pretrained model, or\n# a callable that returns a list of floats (the rewards). The callable receives prompts, completions, and additional\n# arguments from the trainer (refer to the trainer's source for details). To ensure forward compatibility, it should\n# accept **kwargs.\nRewardFunc = str | PreTrainedModel | Callable[..., list[float | None]]\n\n\nclass RLOOTrainer(_BaseTrainer):\n \"\"\"\n Trainer for the Reinforce Leave One Out (RLOO) method. This algorithm was initially proposed in the paper [Back to\n Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in\n LLMs](https://huggingface.co/papers/2402.14740).\n\n Example:\n\n ```python\n from trl import RLOOTrainer\n from trl.rewards import accuracy_reward\n from datasets import load_dataset\n\n dataset = load_dataset(\"trl-lib/DeepMath-103K\", split=\"train\")\n\n trainer = RLOOTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n reward_funcs=accuracy_reward,\n train_dataset=dataset,\n )\n trainer.train()\n ```\n\n Args:\n model (`str` or [`~transformers.PreTrainedModel`] or [`~peft.PeftModel`]):\n Model to be trained. Can be either:\n\n - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using `.from_pretrained` (where `` is derived from the model\n config) with the keyword arguments in `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported.\n - A [`~peft.PeftModel`] object. Only causal language models are supported.\n reward_funcs (`RewardFunc | list[RewardFunc]`):\n Reward functions to be used for computing the rewards. To compute the rewards, we call all the reward\n functions with the prompts and completions and sum the rewards. Can be either:\n\n - A single reward function, such as:\n - A string: The *model ID* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using [`~transformers.AutoModelForSequenceClassification.from_pretrained`] with `num_labels=1` and the\n keyword arguments in `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object: Only sequence classification models are supported.\n - A custom reward function: The function is provided with the prompts and the generated completions,\n plus any additional columns in the dataset. It should return a list of rewards. Custom reward\n functions can be either synchronous or asynchronous and can also return `None` when the reward is\n not applicable to those samples. This is useful for multi-task training where different reward\n functions apply to different types of samples. When a reward function returns `None` for a sample,\n that reward function is excluded from the reward calculation for that sample. For more details, see\n [Using a custom reward\n function](#using-a-custom-reward-function).\n\n The trainer's state is also passed to the reward function. The trainer's state is an instance of\n [`~transformers.TrainerState`] and can be accessed by accessing the `trainer_state` argument to the\n reward function's signature.\n - A list of reward functions, where each item can independently be any of the above types. Mixing different\n types within the list (e.g., a string model ID and a custom reward function) is allowed.\n args ([`RLOOConfig`], *optional*):\n Configuration for this trainer. If `None`, a default configuration is used.\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n Dataset to use for training. It must include a column `\"prompt\"`. Any additional columns in the dataset is\n ignored. The format of the samples can be either:\n\n - [Standard](dataset_formats#standard): Each sample contains plain text.\n - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role\n and content).\n eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Dataset | IterableDataset]`):\n Dataset to use for evaluation. It must meet the same requirements as `train_dataset`.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. The padding side must be set to \"left\". If `None`, the\n processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A\n padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token,\n `tokenizer.eos_token` will be used as the default.\n reward_processing_classes ([`~transformers.PreTrainedTokenizerBase`] or `list[PreTrainedTokenizerBase]`, *optional*):\n Processing classes corresponding to the reward functions specified in `reward_funcs`. Can be either:\n\n - A single processing class: Used when `reward_funcs` contains only one reward function.\n - A list of processing classes: Must match the order and length of the reward functions in `reward_funcs`.\n If set to `None`, or if an element of the list corresponding to a [`~transformers.PreTrainedModel`] is\n `None`, the tokenizer for the model is automatically loaded using\n [`~transformers.AutoTokenizer.from_pretrained`]. For elements in `reward_funcs` that are custom reward\n functions (not [`~transformers.PreTrainedModel`]), the corresponding entries in `reward_processing_classes`\n are ignored.\n callbacks (list of [`~transformers.TrainerCallback`], *optional*):\n List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed\n in [here](https://huggingface.co/docs/transformers/main_classes/callback).\n\n If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`]\n method.\n optimizers (`tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None]`, *optional*, defaults to `(None, None)`):\n A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your\n model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`.\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration used to wrap the model. If `None`, the model is not wrapped.\n \"\"\"\n\n _tag_names = [\"trl\", \"rloo\"]\n _name = \"RLOO\"\n _paper = {\n \"title\": \"Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs\",\n \"id\": \"2402.14740\",\n # docstyle-ignore\n \"citation\": textwrap.dedent(\"\"\"\\\n @inproceedings{ahmadian2024back,\n title = {{Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs}},\n author = {Arash Ahmadian and Chris Cremer and Matthias Gall{\\'{e}} and Marzieh Fadaee and Julia Kreutzer and Olivier Pietquin and Ahmet {\\\"{U}}st{\\\"{u}}n and Sara Hooker},\n year = 2024,\n booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), {ACL} 2024, Bangkok, Thailand, August 11-16, 2024},\n pages = {12248--12267},\n publisher = {Association for Computational Linguistics},\n editor = {Lun{-}Wei Ku and Andre Martins and Vivek Srikumar},\n }\"\"\"),\n }\n\n def __init__(\n self,\n model: \"str | PreTrainedModel | PeftModel\",\n reward_funcs: RewardFunc | list[RewardFunc],\n args: RLOOConfig | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | IterableDataset | dict[str, Dataset | IterableDataset] | None = None,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin | None = None,\n reward_processing_classes: PreTrainedTokenizerBase | list[PreTrainedTokenizerBase] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None] = (None, None),\n peft_config: \"PeftConfig | None\" = None,\n ):\n # Args\n if args is None:\n model_name = model if isinstance(model, str) else get_config_model_id(model.config)\n model_name = model_name.split(\"/\")[-1]\n args = RLOOConfig(f\"{model_name}-RLOO\")\n\n # Model\n if isinstance(model, str):\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n model = create_model_from_path(model, **model_init_kwargs)\n else:\n if args.model_init_kwargs is not None:\n logger.warning(\n \"You passed `model_init_kwargs` to the `RLOOConfig`, but your model is already instantiated. \"\n \"The `model_init_kwargs` will be ignored.\"\n )\n\n # Some models (SmolVLM/Idefics3) don't support `logits_to_keep` argument and error out if we pass it\n # Inspect the forward method before we wrap the model with PEFT\n self.model_kwarg_keys = (\n inspect.signature(model.forward).parameters.keys()\n if not hasattr(model, \"get_base_model\")\n else inspect.signature(model.get_base_model().forward).parameters.keys()\n )\n\n # Processing class\n if processing_class is None:\n processing_class = AutoProcessor.from_pretrained(\n get_config_model_id(model.config), truncation_side=\"left\", padding_side=\"left\"\n )\n\n # Handle pad token for processors or tokenizers\n if isinstance(processing_class, ProcessorMixin):\n tokenizer = processing_class.tokenizer\n elif isinstance(processing_class, PreTrainedTokenizerBase):\n tokenizer = processing_class\n else:\n raise TypeError(\"The `processing_class` must be either a `PreTrainedTokenizerBase` or a `ProcessorMixin`\")\n\n if tokenizer.pad_token is None:\n tokenizer.pad_token = tokenizer.eos_token\n\n self.pad_token = tokenizer.pad_token\n self.pad_token_id = tokenizer.pad_token_id\n self.eos_token_id = tokenizer.eos_token_id\n\n if is_peft_available() and is_peft_model(model) and peft_config is not None:\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first merge \"\n \"and unload the existing adapter, save the resulting base model, and then pass that base model along \"\n \"with the new `peft_config` to the trainer.\"\n )\n if is_peft_available() and is_peft_model(model):\n # If the model is a PEFT model with a pretrained adapter, we need to create a \"ref\" adapter that is a copy\n # of the \"default\" adapter, so that we can use it as the reference model during the training.\n model.add_adapter(\"ref\", model.peft_config[\"default\"])\n for name, param in model.named_parameters():\n if \".default.\" in name:\n ref_name = name.replace(\".default.\", \".ref.\")\n ref_param = model.get_parameter(ref_name)\n ref_param.data.copy_(param.data)\n\n # Create PEFT model\n if peft_config is not None:\n model = get_peft_model(model, peft_config)\n\n # When using gradient checkpointing with PEFT, we need to enable input gradients. transformers.Trainer normally\n # handles this, but a bug currently prevents it; see https://github.com/huggingface/transformers/issues/42489\n if is_peft_available() and is_peft_model(model) and args.gradient_checkpointing:\n model.enable_input_require_grads()\n\n # When using QLoRA, the PEFT adapter weights are converted to bf16 to follow the recommendations from the\n # original paper (see https://huggingface.co/papers/2305.14314, paragraph 3). Normally, this can be done by\n # passing `autocast_adapter_dtype=False` to `get_peft_model`, but this option is not yet supported for\n # quantized models. See: https://github.com/huggingface/peft/issues/2889\n # Non-quantized models do not have the `is_loaded_in_{8,4}bit` attributes, whereas quantized models do\n if getattr(model, \"is_loaded_in_4bit\", False) or getattr(model, \"is_loaded_in_8bit\", False):\n for param in model.parameters():\n if param.requires_grad:\n param.data = param.data.to(torch.bfloat16)\n\n # Reward functions\n if not isinstance(reward_funcs, list):\n reward_funcs = [reward_funcs]\n self.reward_func_names = []\n for i, reward_func in enumerate(reward_funcs):\n if isinstance(reward_func, str):\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n reward_funcs[i] = AutoModelForSequenceClassification.from_pretrained(\n reward_func, num_labels=1, **model_init_kwargs\n )\n if isinstance(reward_funcs[i], nn.Module): # Use Module over PretrainedModel for compat w/ compiled models\n self.reward_func_names.append(get_config_model_id(reward_funcs[i].config).split(\"/\")[-1])\n else:\n self.reward_func_names.append(reward_funcs[i].__name__)\n self.reward_funcs = reward_funcs\n\n self._has_async_reward_funcs = any(inspect.iscoroutinefunction(func) for func in self.reward_funcs)\n if self._has_async_reward_funcs:\n self.async_reward_loop_thread, self.async_reward_loop, self.async_reward_loop_ready_event = (\n start_event_loop_in_daemon(name=\"RLOOTrainer-AsyncRewardLoop\")\n )\n # wait until the event loop is running in the daemon thread\n self.async_reward_loop_ready_event.wait()\n atexit.register(shutdown_event_loop_in_daemon, self.async_reward_loop_thread, self.async_reward_loop)\n\n # Reward weights\n if args.reward_weights is not None:\n if len(args.reward_weights) != len(reward_funcs):\n raise ValueError(\n f\"Number of reward weights ({len(args.reward_weights)}) must match number of reward \"\n f\"functions ({len(reward_funcs)})\"\n )\n self.reward_weights = torch.tensor(args.reward_weights, dtype=torch.float32)\n else:\n self.reward_weights = torch.ones(len(reward_funcs), dtype=torch.float32)\n\n # Reward processing class\n if reward_processing_classes is None:\n reward_processing_classes = [None] * len(reward_funcs)\n elif not isinstance(reward_processing_classes, list):\n reward_processing_classes = [reward_processing_classes]\n if len(reward_processing_classes) != len(reward_funcs):\n raise ValueError(\n f\"The number of reward processing classes ({len(reward_processing_classes)}) must match the number of \"\n f\"reward functions ({len(reward_funcs)}).\"\n )\n\n for i, (reward_processing_class, reward_func) in enumerate(\n zip(reward_processing_classes, reward_funcs, strict=True)\n ):\n if isinstance(reward_func, PreTrainedModel):\n if reward_processing_class is None:\n reward_processing_class = AutoTokenizer.from_pretrained(get_config_model_id(reward_func.config))\n if reward_processing_class.pad_token_id is None:\n reward_processing_class.pad_token = reward_processing_class.eos_token\n # The reward model computes the reward for the latest non-padded token in the input sequence.\n # So it's important to set the pad token ID to the padding token ID of the processing class.\n reward_func.config.pad_token_id = reward_processing_class.pad_token_id\n reward_processing_classes[i] = reward_processing_class\n\n self.reward_processing_classes = reward_processing_classes\n\n # Training arguments\n self.max_completion_length = args.max_completion_length\n self.num_generations = args.num_generations\n self.num_generations_eval = args.num_generations_eval or self.num_generations\n self.chat_template_kwargs = args.chat_template_kwargs or {}\n self.temperature = args.temperature\n self.top_p = args.top_p\n self.top_k = args.top_k\n self.min_p = args.min_p\n self.repetition_penalty = args.repetition_penalty\n self.use_transformers_paged = args.use_transformers_paged\n self.pad_to_multiple_of = args.pad_to_multiple_of\n self.use_vllm = args.use_vllm\n self.vllm_mode = args.vllm_mode\n self.vllm_gpu_memory_utilization = args.vllm_gpu_memory_utilization # only applies to colocation mode\n self.vllm_tensor_parallel_size = args.vllm_tensor_parallel_size # only applies to colocation mode\n self.normalize_advantages = args.normalize_advantages\n self.mask_truncated_completions = args.mask_truncated_completions\n self.reward_clip_range = args.reward_clip_range\n\n # Datasets\n self.shuffle_dataset = args.shuffle_dataset\n\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n elif (\n isinstance(train_dataset, IterableDataset)\n or isinstance(eval_dataset, IterableDataset)\n or (\n isinstance(eval_dataset, dict) and any(isinstance(ds, IterableDataset) for ds in eval_dataset.values())\n )\n ):\n # See https://github.com/huggingface/trl/issues/3213\n raise NotImplementedError(\n \"Iterable datasets are not yet supported in RLOOTrainer. Please use a standard dataset instead.\"\n )\n\n # Multi-step\n self.num_iterations = args.num_iterations\n self.epsilon_low = args.epsilon\n self.epsilon_high = args.epsilon_high if args.epsilon_high is not None else args.epsilon\n # Tracks the number of iterations (forward + backward passes), including those within a grad accum cycle\n self._step = 0\n # Buffer the batch to reuse generated outputs across multiple updates. For more details, see\n # `_get_train_sampler` and `_prepare_inputs`.\n self._buffered_inputs = None\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=identity, # No data collation is needed in RLOO\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n callbacks=callbacks,\n optimizers=optimizers,\n )\n\n # Reference model\n self.beta = args.beta\n if self.beta == 0.0:\n # If beta is 0.0, the reference model is not needed\n self.ref_model = None\n elif is_peft_model(model):\n # If PEFT is used, the reference model is not needed since the adapter can be disabled\n # to revert to the initial model.\n self.ref_model = None\n else:\n # For deepspeed, fsdp or non-distributed models, create a reference model from scratch\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if self.args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n self.ref_model = create_model_from_path(get_config_model_id(self.model.config), **model_init_kwargs)\n\n # Disable dropout in the models\n if args.disable_dropout:\n disable_dropout_in_model(model)\n if self.ref_model is not None:\n disable_dropout_in_model(self.ref_model)\n\n # Initialize the metrics\n self._metrics = {\"train\": defaultdict(list), \"eval\": defaultdict(list)}\n self._total_train_tokens = 0\n self._current_train_step_time = 0.0\n self.log_completions = args.log_completions\n self.log_unique_prompts = args.log_unique_prompts\n self.num_completions_to_print = args.num_completions_to_print\n # Keep logs sized to the generation batch to record only outputs from the latest model update.\n self._logs = {\n \"images\": deque(maxlen=args.generation_batch_size),\n \"prompt\": deque(maxlen=args.generation_batch_size),\n \"completion\": deque(maxlen=args.generation_batch_size),\n \"rewards\": defaultdict(lambda: deque(maxlen=args.generation_batch_size)),\n \"advantages\": deque(maxlen=args.generation_batch_size),\n \"extra\": defaultdict(lambda: deque(maxlen=args.generation_batch_size)),\n }\n # Buffers for user-logged data from reward functions, flushed after gathering\n self._pending_extra_logs = defaultdict(list)\n self._pending_metrics = defaultdict(list)\n\n # Ensure each process receives a unique seed to prevent duplicate completions when generating with\n # transformers if num_generations exceeds per_device_train_batch_size. We could skip it if we use vLLM, but\n # it's safer to set it in all cases.\n set_seed(args.seed, device_specific=True)\n\n if self.use_vllm:\n # Initialize vLLM generation backend\n self.vllm_generation = VLLMGeneration(\n model=self.model,\n accelerator=self.accelerator,\n is_fsdp_enabled=self.is_fsdp_enabled,\n processing_class=self.processing_class,\n # vLLM configuration\n mode=args.vllm_mode,\n structured_outputs_regex=args.vllm_structured_outputs_regex,\n # Server mode configuration\n server_base_url=args.vllm_server_base_url,\n server_host=args.vllm_server_host,\n server_port=args.vllm_server_port,\n group_port=args.vllm_group_port,\n server_timeout=args.vllm_server_timeout,\n # Colocate mode configuration\n tensor_parallel_size=args.vllm_tensor_parallel_size,\n gpu_memory_utilization=args.vllm_gpu_memory_utilization,\n max_model_length=args.vllm_max_model_length,\n max_num_seqs=args.per_device_train_batch_size\n * args.vllm_tensor_parallel_size\n * args.steps_per_generation,\n enable_sleep_mode=args.vllm_enable_sleep_mode,\n model_impl=args.vllm_model_impl,\n # Generation configuration\n repetition_penalty=self.repetition_penalty,\n temperature=self.temperature,\n top_p=self.top_p,\n top_k=self.top_k,\n min_p=self.min_p,\n max_completion_length=self.max_completion_length,\n logprobs=None, # we don't need logprobs from vLLM in RLOO\n generation_kwargs=args.generation_kwargs,\n )\n self._last_loaded_step = -1 # tag to avoid useless loading during grad accumulation\n else:\n generation_kwargs = {\n \"max_new_tokens\": self.max_completion_length,\n \"do_sample\": True,\n \"pad_token_id\": tokenizer.pad_token_id,\n \"bos_token_id\": tokenizer.bos_token_id,\n \"eos_token_id\": tokenizer.eos_token_id,\n \"temperature\": self.temperature,\n \"top_p\": self.top_p,\n \"top_k\": self.top_k,\n \"min_p\": self.min_p,\n \"repetition_penalty\": self.repetition_penalty,\n \"cache_implementation\": args.cache_implementation,\n }\n if args.generation_kwargs is not None:\n generation_kwargs.update(args.generation_kwargs)\n self.generation_config = GenerationConfig(**generation_kwargs)\n # Keep training-specific generation kwargs to overwrite model's original generation config\n self.generation_kwargs = generation_kwargs\n\n # Gradient accumulation requires scaled loss. Normally, loss scaling in the parent class depends on whether the\n # model accepts loss-related kwargs. Since we compute our own loss, this check is irrelevant. We set\n # self.model_accepts_loss_kwargs to False to enable scaling.\n self.model_accepts_loss_kwargs = False\n\n # Add tags to the model\n self.model.add_model_tags(self._tag_names)\n\n if self.ref_model is not None:\n if self.is_deepspeed_enabled:\n self.ref_model = prepare_deepspeed(self.ref_model, self.accelerator)\n elif self.is_fsdp_enabled:\n self.ref_model = prepare_fsdp(self.ref_model, self.accelerator)\n else:\n self.ref_model = self.accelerator.prepare_model(self.ref_model, evaluation_mode=True)\n\n if args.sync_ref_model:\n if self.beta == 0.0:\n raise ValueError(\n \"You passed `sync_ref_model=True` while `beta=0.0`, which means the reference model is not used \"\n \"during training. Consequently, RLOOTrainer does not create a `ref_model` instance, and there is \"\n \"nothing to synchronize. Please set `sync_ref_model=False`, or set `beta` to a non-zero value.\"\n )\n if is_peft_model(model):\n raise NotImplementedError(\n \"You passed `sync_ref_model=True` while using a PEFT model, which is currently not supported. \"\n \"With PEFT, RLOOTrainer does not keep a separate reference model in memory; instead, it recovers \"\n \"reference behavior by temporarily disabling the adapter. As a result, there is no standalone \"\n \"`ref_model` instance to synchronize. Use `sync_ref_model=False`, or opt for full fine-tuning if \"\n \"you need a synced reference model. If you need `sync_ref_model` to work with PEFT, please open a \"\n \"feature request at https://github.com/huggingface/trl/issues.\"\n )\n self.add_callback(SyncRefModelCallback(ref_model=self.ref_model, accelerator=self.accelerator))\n\n for i, reward_func in enumerate(self.reward_funcs):\n if isinstance(reward_func, PreTrainedModel):\n if self.is_deepspeed_enabled:\n self.reward_funcs[i] = prepare_deepspeed(reward_func, self.accelerator)\n else:\n # set device placement to True to make `prepare_model` move `reward_func` to device when using fsdp\n self.reward_funcs[i] = self.accelerator.prepare_model(\n reward_func, evaluation_mode=True, device_placement=True\n )\n\n def _set_signature_columns_if_needed(self):\n # If `self.args.remove_unused_columns` is True, non-signature columns are removed.\n # By default, this method sets `self._signature_columns` to the model's expected inputs (usually, \"input_ids\"\n # and \"attention_mask\"). In RLOOTrainer, we preprocess data, so using the model's signature columns doesn't\n # work. Instead, we set them to the columns expected by the `training_step` method, hence the override.\n if self._signature_columns is None:\n self._signature_columns = [\"prompt\", \"image\", \"images\"]\n\n # This method overrides `Trainer.get_train_dataloader` to support our custom batching strategy.\n # Instead of returning a standard per-step batch (i.e., `per_device_batch_size), our dataloader loads an\n # *generation* batch (i.e., `per_device_batch_size ร— steps_per_generation`). This allows us to generate completions\n # once every steps_per_generation stepโ€”rather than once per accumulation stepโ€”which is significantly more\n # efficient. The only change from the original implementation is multiplying the batch size by\n # `steps_per_generation`. Thus, `_prepare_inputs` is called with this *generation* batch, and it handles the\n # splitting internally.\n # Maintenance note: This method is a copy-paste of the original `Trainer.get_train_dataloader` with only one line\n # modification.\n def get_train_dataloader(self):\n return self._get_dataloader(\n dataset=self.train_dataset,\n description=\"Training\",\n batch_size=self._train_batch_size * self.args.steps_per_generation, # < this is the change\n sampler_fn=self._get_train_sampler,\n is_training=True,\n )\n\n def _get_train_sampler(self, dataset: Dataset | None = None) -> Sampler:\n # Returns a sampler that\n # 1. ensures each prompt is repeated across multiple processes. This guarantees that identical prompts are\n # distributed to different GPUs, allowing rewards to be computed and normalized correctly within each prompt\n # group. Using the same seed across processes ensures consistent prompt assignment, preventing discrepancies\n # in group formation.\n # 2. repeats the batch multiple times to allow reusing generations across multiple updates. Refer to\n # _prepare_inputs to see how the generations are stored and reused.\n\n # In the following figure, the values are the prompt indices. The first row shows the first sampled batch, the\n # second row shows the second sampled batch, and so on.\n #\n # | GPU 0 | GPU 1 |\n #\n # global_step step <-โ”€โ”€โ”€> num_generations=2\n # <-โ”€โ”€โ”€โ”€โ”€โ”€โ”€> per_device_train_batch_size=3\n # grad_accum โ–ฒ โ–ฒ 0 0 0 0 1 1 2 2 <- Generate for the first `steps_per_generation` (prompts 0 to 11); store the completions; use the first slice to compute the loss\n # =2 โ–ผ | 0 1 3 3 4 4 5 5 <- Take the stored generations and use the second slice to compute the loss\n # |\n # | 1 2 6 6 7 7 8 8 <- Take the stored generations and use the third slice to compute the loss\n # steps_per_gen=4 โ–ผ 1 3 9 9 10 10 11 11 <- Take the stored generations and use the fourth slice to compute the loss\n #\n # 2 4 12 12 13 13 14 14 <- Generate for the second `steps_per_generation` (prompts 12 to 23); store the completions; use the first slice to compute the loss\n # 2 5 15 15 16 16 17 17 <- Take the stored generations and use the second slice to compute the loss\n # ...\n if dataset is None:\n dataset = self.train_dataset\n return RepeatSampler(\n data_source=dataset,\n mini_repeat_count=self.num_generations,\n batch_size=self.args.generation_batch_size // self.num_generations,\n repeat_count=self.num_iterations * self.args.steps_per_generation,\n shuffle=self.shuffle_dataset,\n seed=self.args.seed,\n )\n\n def _get_eval_sampler(self, eval_dataset) -> Sampler:\n # See _get_train_sampler for an explanation of the sampler.\n return RepeatSampler(\n data_source=eval_dataset,\n mini_repeat_count=self.num_generations_eval,\n seed=self.args.seed,\n )\n\n @profiling_decorator\n def _get_per_token_logps_and_entropies(\n self,\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n batch_size=None,\n compute_entropy=False,\n pixel_values=None,\n image_grid_thw=None,\n num_images=None,\n pixel_attention_mask=None,\n image_sizes=None,\n token_type_ids=None,\n mm_token_type_ids=None,\n ) -> dict[str, torch.Tensor | None]:\n \"\"\"Compute log-probs and (optionally) entropies for each token.\"\"\"\n batch_size = batch_size or input_ids.size(0) # Chunk inputs into smaller batches to reduce memory peak\n all_logps = []\n all_entropies = []\n for start in range(0, input_ids.size(0), batch_size):\n input_ids_batch = input_ids[start : start + batch_size]\n attention_mask_batch = attention_mask[start : start + batch_size]\n\n # Build model inputs - check if the model supports logits_to_keep (some models and VLMs don't)\n model_inputs = {\"input_ids\": input_ids_batch, \"attention_mask\": attention_mask_batch}\n if image_grid_thw is not None and pixel_values is not None:\n rows_per_image = image_grid_thw.prod(dim=-1)\n rows_per_sample = torch.split(rows_per_image, num_images)\n rows_per_sample = torch.stack([s.sum() for s in rows_per_sample])\n cum_rows = torch.cat([torch.tensor([0], device=rows_per_sample.device), rows_per_sample.cumsum(0)])\n row_start, row_end = cum_rows[start].item(), cum_rows[start + batch_size].item()\n model_inputs[\"pixel_values\"] = pixel_values[row_start:row_end]\n cum_imgs = torch.tensor([0] + num_images).cumsum(0)\n img_start, img_end = cum_imgs[start], cum_imgs[start + batch_size]\n model_inputs[\"image_grid_thw\"] = image_grid_thw[img_start:img_end]\n elif pixel_values is not None:\n model_inputs[\"pixel_values\"] = pixel_values[start : start + batch_size]\n if pixel_attention_mask is not None:\n model_inputs[\"pixel_attention_mask\"] = pixel_attention_mask[start : start + batch_size]\n if image_sizes is not None:\n model_inputs[\"image_sizes\"] = image_sizes[start : start + batch_size]\n if token_type_ids is not None:\n model_inputs[\"token_type_ids\"] = token_type_ids[start : start + batch_size]\n if mm_token_type_ids is not None:\n model_inputs[\"mm_token_type_ids\"] = mm_token_type_ids[start : start + batch_size]\n\n # Only add logits_to_keep if the model supports it\n if \"logits_to_keep\" in self.model_kwarg_keys:\n # We add 1 to `logits_to_keep` because the last logits of the sequence is later excluded\n model_inputs[\"logits_to_keep\"] = logits_to_keep + 1\n\n model_inputs[\"use_cache\"] = False # only used in generation; set False to suppress warnings\n\n logits = model(**model_inputs).logits\n # Exclude the last value: it corresponds to the next token pred\n logits = logits[:, :-1, :] # (B, L-1, H)\n # Only keep the last logits_to_keep. For model that support logits_to_keep, this is a no-op.\n logits = logits[:, -logits_to_keep:, :] # (B, logits_to_keep, H)\n # Divide logits by sampling temperature.\n # See https://huggingface.co/blog/the_n_implementation_details_of_rlhf_with_ppo#policy-training-implementation-details\n logits = logits / self.temperature\n completion_ids = input_ids_batch[:, -logits_to_keep:]\n logps = selective_log_softmax(logits, completion_ids) # compute logprobs\n all_logps.append(logps)\n\n if compute_entropy:\n with torch.no_grad():\n entropies = entropy_from_logits(logits)\n all_entropies.append(entropies)\n\n logps = torch.cat(all_logps, dim=0)\n entropies = torch.cat(all_entropies, dim=0) if compute_entropy else None\n return logps, entropies\n\n def training_step(self, model, inputs, num_items_in_batch):\n time_before = time.perf_counter()\n output = super().training_step(model, inputs, num_items_in_batch)\n self._step += 1\n time_after = time.perf_counter()\n self._current_train_step_time += time_after - time_before\n if self._step % self.current_gradient_accumulation_steps == 0:\n self._metrics[\"train\"][\"step_time\"].append(self._current_train_step_time)\n self._current_train_step_time = 0.0\n return output\n\n @profiling_decorator\n def _prepare_inputs(self, generation_batch: dict[str, torch.Tensor | Any]) -> dict[str, torch.Tensor | Any]:\n # Prepares inputs for model training/evaluation by managing completion generation and batch handling.\n # During training:\n # - Receives the local generation batch (Per-GPU batch size ร— steps per generation)\n # from the modified training dataloader instead of the standard local batch\n # - Generates completions once for the entire generation batch and splits it into batches of size\n # `per_device_train_batch_size`\n # - Buffers these completions and returns the appropriate slice for the current accumulation step\n # - Optimizes by regenerating completions only periodically (every steps_per_generation * num_iterations)\n # During evaluation:\n # - The input is treated as a standard local batch (no accumulation, no multiple iterations)\n # - Completions are generated for each batch without buffering or reuse\n # Returns a single local batch in both cases.\n\n mode = \"train\" if self.model.training else \"eval\"\n if mode == \"train\":\n generate_every = self.args.steps_per_generation * self.num_iterations\n if self._step % generate_every == 0 or self._buffered_inputs is None:\n # self._buffered_inputs=None can occur when resuming from a checkpoint\n generation_batch = self._generate_and_score_completions(generation_batch)\n generation_batch = split_pixel_values_by_grid(generation_batch)\n generation_batch = shuffle_sequence_dict(generation_batch)\n generation_batches = split_tensor_dict(generation_batch, self.args.steps_per_generation)\n self._buffered_inputs = [unsplit_pixel_values_by_grid(batch) for batch in generation_batches]\n inputs = self._buffered_inputs[self._step % self.args.steps_per_generation]\n else:\n # In evaluation, there is neither batch grouping for generation, nor multiple iterations, hence\n # local generation batch == local eval batch\n inputs = self._generate_and_score_completions(generation_batch)\n return inputs\n\n def _log_completion_extra(self, column: str, values: list):\n \"\"\"\n Log extra columns to the completions table. Called from reward functions via the `log_extra` kwarg.\n\n Args:\n column (`str`):\n Name of the column to add.\n values (`list`):\n Values for the column, one per sample in the batch.\n \"\"\"\n self._pending_extra_logs[column].extend(values)\n\n def _log_metric(self, name: str, value: float):\n \"\"\"\n Log a scalar metric from a reward function. Called via the `log_metric` kwarg. Values are averaged over each\n logging step and reported alongside built-in metrics like `kl` and `entropy`.\n\n Args:\n name (`str`):\n Name of the metric.\n value (`float`):\n Scalar value for this batch.\n \"\"\"\n self._pending_metrics[name].append(value)\n\n @profiling_decorator\n def _calculate_rewards(self, inputs, prompts, completions, completion_ids_list):\n device = self.accelerator.device\n rewards_per_func = torch.zeros(len(prompts), len(self.reward_funcs), device=device)\n\n # Repeat all input columns (but \"prompt\", \"completion\", and \"completion_ids\") to match the num of generations\n keys = [key for key in inputs[0] if key not in [\"prompt\", \"completion\", \"completion_ids\"]]\n reward_kwargs = {key: [example[key] for example in inputs] for key in keys}\n\n # This allows for dynamic reward shaping based on training progress.\n reward_kwargs[\"trainer_state\"] = self.state\n\n # Allow reward functions to log extra columns to the completions table.\n reward_kwargs[\"log_extra\"] = self._log_completion_extra\n\n # Allow reward functions to log additional scalar metrics.\n reward_kwargs[\"log_metric\"] = self._log_metric\n\n async_funcs_info = [] # async custom functions for asyncio.gather\n\n for i, (reward_func, reward_processing_class, reward_func_name) in enumerate(\n zip(self.reward_funcs, self.reward_processing_classes, self.reward_func_names, strict=True)\n ):\n if isinstance(reward_func, nn.Module): # Module (no PretrainedModel) for compat with compiled models\n with profiling_context(self, reward_func_name):\n if is_conversational(inputs[0]):\n messages = [{\"messages\": p + c} for p, c in zip(prompts, completions, strict=True)]\n texts = [\n apply_chat_template(x, reward_processing_class, **self.chat_template_kwargs)[\"text\"]\n for x in messages\n ]\n else:\n texts = [p + c for p, c in zip(prompts, completions, strict=True)]\n reward_inputs = reward_processing_class(\n text=texts, return_tensors=\"pt\", padding=True, padding_side=\"right\", add_special_tokens=False\n )\n reward_inputs = super()._prepare_inputs(reward_inputs)\n with torch.inference_mode():\n rewards_per_func[:, i] = reward_func(**reward_inputs).logits[:, 0] # Shape (B*G,)\n elif inspect.iscoroutinefunction(reward_func): # Separate async reward funcs to run them in parallel later\n async_funcs_info.append((i, reward_func, reward_func_name))\n else:\n # Run synchronous reward function\n with profiling_context(self, reward_func_name):\n output_reward_func = reward_func(\n prompts=prompts, completions=completions, completion_ids=completion_ids_list, **reward_kwargs\n )\n # Convert None values to NaN\n output_reward_func = [reward if reward is not None else torch.nan for reward in output_reward_func]\n rewards_per_func[:, i] = torch.tensor(output_reward_func, dtype=torch.float32, device=device)\n\n # Execute async custom functions in parallel using asyncio.gather\n if async_funcs_info:\n\n async def _invoke_async_reward(index, func, func_name):\n with profiling_context(self, func_name):\n output = await func(\n prompts=prompts, completions=completions, completion_ids=completion_ids_list, **reward_kwargs\n )\n output = [r if r is not None else torch.nan for r in output]\n return index, output\n\n async def _run_async_funcs():\n coros = [_invoke_async_reward(i, func, func_name) for (i, func, func_name) in async_funcs_info]\n return await asyncio.gather(*coros)\n\n async_results = asyncio.run_coroutine_threadsafe(_run_async_funcs(), self.async_reward_loop).result()\n for idx, output_reward_func in async_results:\n rewards_per_func[:, idx] = torch.tensor(output_reward_func, dtype=torch.float32, device=device)\n\n # If all reward functions return None for a given row, issue a detailed warning\n if torch.isnan(rewards_per_func).all(dim=1).any():\n nan_row_idx = torch.isnan(rewards_per_func).all(dim=1).nonzero(as_tuple=True)[0][0]\n row_reward_kwargs = {\n key: value[nan_row_idx]\n for key, value in reward_kwargs.items()\n if key not in (\"trainer_state\", \"log_extra\", \"log_metric\")\n }\n row_reward_kwargs[\"prompt\"] = prompts[nan_row_idx]\n row_reward_kwargs[\"completion\"] = completions[nan_row_idx]\n logger.warning(\n f\"All reward functions returned None for the following kwargs:\\n{row_reward_kwargs}\\n\"\n \"Please ensure that at least one reward function returns a valid reward.\"\n )\n\n # Gather the reward per function: this part is crucial, because the rewards are normalized per group and the\n # completions may be distributed across processes\n rewards_per_func = gather(rewards_per_func)\n return rewards_per_func\n\n def _tokenize_prompts(self, prompts: list):\n \"\"\"Tokenize prompts and extract images/multimodal fields for generation.\"\"\"\n if is_conversational({\"prompt\": prompts[0]}):\n # Extract images from messages for VLM support\n images = []\n has_images = False\n for prompt in prompts:\n prompt_images = []\n for message in prompt:\n if isinstance(message[\"content\"], list):\n for part in message[\"content\"]:\n if part[\"type\"] == \"image\":\n prompt_images.append(part[\"image\"])\n has_images = True\n images.append(prompt_images if prompt_images else None)\n images = images if has_images else None\n\n # We pass padding=True to work around a bug introduced in transformers 5.2.0 in some processors\n # (e.g. Qwen2.5-VL) that crash on batched unpadded input. We then unpad input_ids using attention_mask.\n # See: https://github.com/huggingface/transformers/issues/44514\n tokenized = self.processing_class.apply_chat_template(\n conversation=prompts,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n padding=True,\n **self.chat_template_kwargs,\n )\n # Unpad input_ids: remove padding tokens using attention_mask to get per-sequence lists\n prompt_ids = [\n [tok for tok, m in zip(ids, mask, strict=True) if m]\n for ids, mask in zip(tokenized[\"input_ids\"], tokenized[\"attention_mask\"], strict=True)\n ]\n # For VLMs, the processor returns extra multimodal fields (pixel_values, image_grid_thw, etc.)\n multimodal_fields = {k: v for k, v in tokenized.items() if k not in (\"input_ids\", \"attention_mask\")}\n else:\n prompt_ids = self.processing_class(text=prompts)[\"input_ids\"]\n images = None\n multimodal_fields = {}\n return prompt_ids, images, multimodal_fields\n\n def _generate_single_turn(self, prompt_ids, images, multimodal_fields):\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n\n # Generate completions using either vLLM or regular generation\n if self.use_vllm:\n # Sync weights if training step changed\n if self.state.global_step != self._last_loaded_step:\n with profiling_context(self, \"sync_weights\"):\n self.vllm_generation.sync_weights()\n self._last_loaded_step = self.state.global_step\n\n # Generate using vLLM (note: RLOO doesn't use logprobs from generation, so we ignore them)\n num_generations = self.num_generations if mode == \"train\" else self.num_generations_eval\n _, completion_ids, _, _ = self.vllm_generation.generate(\n prompts=prompt_ids,\n images=images,\n num_generations=num_generations,\n profiler=profiling_context(self, \"vLLM.generate\"),\n )\n\n elif self.use_transformers_paged:\n with (\n profiling_context(self, \"transformers.generate_batch\"),\n unwrap_model_for_generation(\n self.model_wrapped, self.accelerator, gather_deepspeed3_params=self.args.ds3_gather_for_generation\n ) as unwrapped_model,\n torch.no_grad(),\n FSDP.summon_full_params(self.model_wrapped, recurse=False) if self.is_fsdp_enabled else nullcontext(),\n ):\n # Cast to the appropriate dtype based on training configuration\n if self.args.bf16:\n unwrapped_model.to(torch.bfloat16)\n elif self.args.fp16:\n unwrapped_model.to(torch.float16)\n with torch.inference_mode():\n # Continuous batching API expects 'inputs' arg only\n all_outputs = unwrapped_model.generate_batch(\n prompt_ids, generation_config=self.generation_config, progress_bar=False\n )\n unwrapped_model.train() # restore training mode, as generate_batch forces eval mode\n completion_ids = [output.generated_tokens for output in all_outputs.values()]\n\n else:\n # Regular generation path: left-pad token IDs into tensors\n prompt_tensors = [torch.tensor(ids) for ids in prompt_ids]\n padded_ids = pad(prompt_tensors, padding_value=self.pad_token_id, padding_side=\"left\")\n attention_mask = pad([torch.ones_like(t) for t in prompt_tensors], padding_value=0, padding_side=\"left\")\n generate_inputs = {\"input_ids\": padded_ids, \"attention_mask\": attention_mask}\n # For VLMs, include multimodal fields as tensors (pixel_values, image_grid_thw, etc.)\n for k, v in multimodal_fields.items():\n if isinstance(v, torch.Tensor):\n generate_inputs[k] = v\n elif isinstance(v, list) and v and isinstance(v[0], list):\n # Per-token field (e.g., token_type_ids): left-pad like input_ids\n generate_inputs[k] = pad([torch.tensor(x) for x in v], padding_value=0, padding_side=\"left\")\n else:\n generate_inputs[k] = torch.tensor(np.array(v))\n generate_inputs = super()._prepare_inputs(generate_inputs)\n\n with (\n profiling_context(self, \"transformers.generate\"),\n unwrap_model_for_generation(\n self.model_wrapped,\n self.accelerator,\n gather_deepspeed3_params=self.args.ds3_gather_for_generation,\n generation_kwargs=self.generation_kwargs, # Override model.generation_config with generation_kwargs to fix transformers#42762\n ) as unwrapped_model,\n torch.no_grad(),\n FSDP.summon_full_params(self.model_wrapped, recurse=False) if self.is_fsdp_enabled else nullcontext(),\n ):\n prompt_completion_ids = unwrapped_model.generate(\n **generate_inputs, generation_config=self.generation_config, disable_compile=True\n )\n # Compute prompt length and extract completion ids\n prompt_length = generate_inputs[\"input_ids\"].size(1)\n completion_ids = prompt_completion_ids[:, prompt_length:]\n\n # Mask everything after the first EOS token\n is_eos = completion_ids == self.eos_token_id\n eos_idx = torch.full((is_eos.size(0),), is_eos.size(1), dtype=torch.long, device=device)\n eos_idx[is_eos.any(dim=1)] = is_eos.int().argmax(dim=1)[is_eos.any(dim=1)]\n sequence_indices = torch.arange(is_eos.size(1), device=device).expand(is_eos.size(0), -1)\n completion_mask = (sequence_indices <= eos_idx.unsqueeze(1)).int()\n completion_ids = [\n c[m].tolist() for c, m in zip(completion_ids.cpu(), completion_mask.bool().cpu(), strict=True)\n ]\n\n return completion_ids\n\n def _generate(self, prompts: list):\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n\n # Copy the prompts to avoid modifying the original list\n prompts = copy.deepcopy(prompts)\n\n prompt_ids, images, multimodal_fields = self._tokenize_prompts(prompts)\n completion_ids = self._generate_single_turn(prompt_ids, images, multimodal_fields)\n\n # Decode completions. It's important to use `parse_response` when possible, because it handles tool calls.\n if is_conversational({\"prompt\": prompts[0]}):\n contents = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n completions = [[{\"role\": \"assistant\", \"content\": content}] for content in contents]\n else:\n completions = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n\n # Get completion length per sequence, used for logging\n prompt_lengths = torch.tensor([len(ids) for ids in prompt_ids], device=device)\n completion_lengths = torch.tensor([len(ids) for ids in completion_ids], device=device)\n agg_prompt_lengths = self.accelerator.gather(prompt_lengths)\n agg_completion_lengths = self.accelerator.gather(completion_lengths)\n total_prompt_tokens = agg_prompt_lengths.sum()\n total_completion_tokens = agg_completion_lengths.sum() # = num_items_in_batch, required for the DAPO loss\n\n # Log the metrics\n if mode == \"train\":\n self.state.num_input_tokens_seen += (total_prompt_tokens + total_completion_tokens).item()\n self._metrics[mode][\"num_tokens\"] = [self.state.num_input_tokens_seen]\n\n # Log completion lengths, mean, min, max\n self._metrics[mode][\"completions/mean_length\"].append(agg_completion_lengths.float().mean().item())\n self._metrics[mode][\"completions/min_length\"].append(agg_completion_lengths.float().min().item())\n self._metrics[mode][\"completions/max_length\"].append(agg_completion_lengths.float().max().item())\n\n # Identify sequences that terminated with EOS and log their lengths\n eos_and_pad = [self.eos_token_id, self.pad_token_id]\n is_truncated = torch.tensor([ids[-1] not in eos_and_pad for ids in completion_ids], device=device)\n agg_is_truncated = self.accelerator.gather(is_truncated)\n self._metrics[mode][\"completions/clipped_ratio\"].append(agg_is_truncated.float().mean().item())\n term_completion_lengths = agg_completion_lengths[~agg_is_truncated]\n if len(term_completion_lengths) == 0: # edge case where no terminated sequences are found\n term_completion_lengths = torch.zeros(1, device=device)\n self._metrics[mode][\"completions/mean_terminated_length\"].append(term_completion_lengths.float().mean().item())\n self._metrics[mode][\"completions/min_terminated_length\"].append(term_completion_lengths.float().min().item())\n self._metrics[mode][\"completions/max_terminated_length\"].append(term_completion_lengths.float().max().item())\n\n return prompt_ids, completion_ids, completions\n\n def _generate_and_score_completions(\n self, inputs: list[dict[str, torch.Tensor | Any]]\n ) -> dict[str, torch.Tensor | Any]:\n device = self.accelerator.device\n mode = \"train\" if self.model.training else \"eval\"\n\n prompts = [x[\"prompt\"] for x in inputs]\n\n if \"images\" in inputs[0]:\n images = [example.get(\"images\") for example in inputs]\n elif \"image\" in inputs[0]:\n images = [[example.get(\"image\")] if example.get(\"image\") is not None else None for example in inputs]\n else:\n images = None\n # Transformers requires at least one image in the batch, otherwise it throws an error\n if images is not None and all(img_list == [] for img_list in images):\n images = None\n\n # If the prompts are conversational and the inputs contain images, we need to convert the prompts from\n # [{\"role\": \"user\", \"content\": \"What color is the sky?\"}] to\n # [{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": }, {\"type\": \"text\", \"text\": \"What color is the sky?\"}]}]\n if images is not None:\n if not is_conversational(inputs[0]):\n raise ValueError(\n \"Multimodal training requires conversational prompts. It looks like the dataset contains \"\n \"non-conversational inputs, likely because a chat template was applied before passing the dataset \"\n \"to the trainer. Please provide the raw conversational prompts and let the trainer apply the chat \"\n \"template internally.\"\n )\n prompts = [\n prepare_multimodal_messages(prompt, image_list)\n for prompt, image_list in zip(prompts, images, strict=True)\n ]\n\n prompt_ids_list, completion_ids_list, completions = self._generate(prompts)\n\n # Convert lists of token IDs to padded tensors\n prompt_ids = [torch.tensor(ids) for ids in prompt_ids_list]\n prompt_mask = [torch.ones_like(ids, dtype=torch.long) for ids in prompt_ids]\n prompt_ids = pad(\n prompt_ids,\n padding_value=self.pad_token_id,\n padding_side=\"left\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n prompt_mask = pad(\n prompt_mask,\n padding_value=0,\n padding_side=\"left\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n completion_ids = [torch.tensor(ids) for ids in completion_ids_list]\n completion_mask = [torch.ones_like(ids, dtype=torch.long) for ids in completion_ids]\n completion_ids = pad(\n completion_ids,\n padding_value=self.pad_token_id,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n completion_mask = pad(\n completion_mask,\n padding_value=0,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n ).to(device=device)\n\n # If mask_truncated_completions is enabled, zero out truncated completions in completion_mask\n if self.mask_truncated_completions:\n eos_and_pad = [self.eos_token_id, self.pad_token_id]\n is_truncated = torch.tensor([ids[-1] not in eos_and_pad for ids in completion_ids_list], device=device)\n completion_mask = completion_mask * (~is_truncated).unsqueeze(1).int()\n\n # Concatenate prompt_mask with completion_mask for logit computation\n prompt_completion_ids = torch.cat([prompt_ids, completion_ids], dim=1) # (B, P+C)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1) # (B, P+C)\n\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n batch_size = self.args.per_device_train_batch_size if mode == \"train\" else self.args.per_device_eval_batch_size\n\n num_images = [len(img_list) for img_list in images] if images is not None else None\n\n # Get forward_kwargs for models with multimodal inputs\n if images is not None:\n prompts_text = [\n apply_chat_template({\"prompt\": prompt}, self.processing_class, **self.chat_template_kwargs)[\"prompt\"]\n for prompt in prompts\n ]\n prompt_inputs = self.processing_class(images=images, text=prompts_text, padding=True, return_tensors=\"pt\")\n prompt_inputs = super()._prepare_inputs(prompt_inputs)\n forward_kwargs = {k: v for k, v in prompt_inputs.items() if k not in [\"input_ids\", \"attention_mask\"]}\n else:\n forward_kwargs = {}\n\n # If token_type_ids are used, extend them with zeros for the completion part\n if \"token_type_ids\" in forward_kwargs:\n token_type_ids = forward_kwargs[\"token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n # Needed only with pad_to_multiple_of: otherwise prompt_ids and token_type_ids must have equal len\n padding_size = prompt_ids.size(1) - token_type_ids.size(1)\n if padding_size > 0:\n token_type_ids = torch.cat(\n [token_type_ids.new_zeros((token_type_ids.size(0), padding_size)), token_type_ids], dim=1\n )\n forward_kwargs[\"token_type_ids\"] = torch.cat(\n [token_type_ids, token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n # If mm_token_type_ids are used, extend them with zeros for the completion part\n if \"mm_token_type_ids\" in forward_kwargs:\n mm_token_type_ids = forward_kwargs[\"mm_token_type_ids\"]\n if self.pad_to_multiple_of is not None:\n # Needed only with pad_to_multiple_of: otherwise prompt_ids and mm_token_type_ids must have equal len\n padding_size = prompt_ids.size(1) - mm_token_type_ids.size(1)\n if padding_size > 0:\n mm_token_type_ids = torch.cat(\n [mm_token_type_ids.new_zeros((mm_token_type_ids.size(0), padding_size)), mm_token_type_ids],\n dim=1,\n )\n forward_kwargs[\"mm_token_type_ids\"] = torch.cat(\n [mm_token_type_ids, mm_token_type_ids.new_zeros(completion_ids.shape)], dim=1\n )\n\n # When gradient checkpointing is enabled with use_reentrant=True (non default), calling the model inside a\n # torch.no_grad() block triggers a harmless PyTorch warning (\"None of the inputs have requires_grad=True\").\n # Temporarily disable checkpointing to avoid this warning during inference.\n with torch.no_grad(), disable_gradient_checkpointing(self.model, self.args.gradient_checkpointing_kwargs):\n # Compute the per-token log probabilities for the current model\n old_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n old_logps = (old_per_token_logps * completion_mask).sum(1) # mask out padding and tokens after EOS\n\n # Compute the per-token log probabilities for the reference model\n if self.beta != 0.0:\n if self.ref_model is not None:\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.ref_model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n # When training a PEFT adapter, how we obtain the reference depends on the setup:\n # - New adapter: disabling adapters yields the base model.\n # - Re-training an existing adapter: an initial copy is loaded under the name \"ref\".\n model = self.accelerator.unwrap_model(self.model)\n with use_adapter(model, adapter_name=\"ref\" if \"ref\" in model.peft_config else None):\n ref_per_token_logps, _ = self._get_per_token_logps_and_entropies(\n self.model,\n prompt_completion_ids,\n attention_mask,\n logits_to_keep,\n batch_size=batch_size,\n num_images=num_images,\n **forward_kwargs, # may contain pixel_values, image_grid_thw, pixel_attention_mask and image_sizes\n )\n else:\n ref_per_token_logps = None\n\n # Decode\n prompts_text = self.processing_class.batch_decode(prompt_ids, skip_special_tokens=True)\n completions_text = self.processing_class.batch_decode(completion_ids, skip_special_tokens=True)\n\n # Calculate rewards for each reward function. rewards_per_func aggregates rewards across all processes. This is\n # important because rewards will be normalized per group, and completions are distributed. We will later slice\n # rewards_per_func to extract each process's subset.\n rewards_per_func = self._calculate_rewards(inputs, prompts, completions, completion_ids_list)\n num_generations = self.num_generations if mode == \"train\" else self.num_generations_eval\n\n # Apply weights to each reward function's output and sum\n rewards = (rewards_per_func * self.reward_weights.to(device).unsqueeze(0)).nansum(dim=1)\n\n # Apply reward clipping if specified\n if self.reward_clip_range:\n rewards = rewards.clamp(min=self.reward_clip_range[0], max=self.reward_clip_range[1])\n\n # Include the KL penalty in the reward\n if self.beta != 0.0:\n per_token_kl = old_per_token_logps - ref_per_token_logps\n # Apply sequence-level KL penalty to rewards (sum KL across tokens first, then apply to each sequence)\n kl = (per_token_kl * completion_mask).sum(-1)\n kl = gather(kl) # rewards are gathered, so kl must be too\n rewards = rewards - self.beta * kl\n\n grouped_rewards = rewards.view(-1, num_generations)\n mean_grouped_rewards = grouped_rewards.mean(dim=1)\n if num_generations > 1:\n std_rewards = grouped_rewards.std(dim=1)\n else: # doesn't occur during training, but could occur in eval when num_generations_eval=1\n std_rewards = torch.zeros_like(mean_grouped_rewards)\n\n # RLOO advantages computation\n grouped_sum = grouped_rewards.sum(dim=1, keepdim=True) # (num_prompts, 1)\n if num_generations > 1:\n baselines = (grouped_sum - grouped_rewards) / (num_generations - 1) # (num_prompts, num_generations)\n baselines = baselines.view(-1) # Flatten back to match rewards shape\n advantages = rewards - baselines\n else: # this case doesn't occur during training, but could in eval when num_generations_eval=1\n advantages = torch.zeros_like(rewards)\n\n # Normalize advantages\n if self.normalize_advantages:\n advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-4)\n\n is_std_zero = torch.isclose(std_rewards, torch.zeros_like(std_rewards)) # for logging\n\n # Slice to keep only the local part of the data\n process_slice = slice(\n self.accelerator.process_index * len(prompts),\n (self.accelerator.process_index + 1) * len(prompts),\n )\n all_process_advantages = advantages.clone() # keep the aggregated advantages for logging\n advantages = advantages[process_slice]\n\n # Calculate and log the mean KL divergence between current and reference model\n if self.beta != 0.0:\n mean_kl = (per_token_kl * completion_mask).sum() / completion_mask.sum().clamp(min=1.0)\n self._metrics[mode][\"kl\"].append(self.accelerator.gather(mean_kl).nanmean().item())\n\n # Calculate mean reward per function, but only for samples where the function was applied (non-NaN values)\n for i, reward_func_name in enumerate(self.reward_func_names):\n mean_rewards = torch.nanmean(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/mean\"].append(mean_rewards)\n std_func_rewards = nanstd(rewards_per_func[:, i]).item()\n self._metrics[mode][f\"rewards/{reward_func_name}/std\"].append(std_func_rewards)\n rewards = rewards_per_func.nansum(dim=1)\n self._metrics[mode][\"reward\"].append(rewards.mean().item())\n self._metrics[mode][\"reward_std\"].append(rewards.std().item())\n self._metrics[mode][\"frac_reward_zero_std\"].append(is_std_zero.float().mean().item())\n\n # Log prompt and completion texts\n self._logs[\"prompt\"].extend(gather_object(prompts_text))\n self._logs[\"completion\"].extend(gather_object(completions_text))\n for i, name in enumerate(self.reward_func_names):\n self._logs[\"rewards\"][name].extend(rewards_per_func[:, i].tolist())\n self._logs[\"advantages\"].extend(all_process_advantages.tolist())\n\n # Flush user-logged extra columns (from log_extra), gathering across processes.\n # Keys must be sorted so that all ranks call gather_object in the same order, otherwise values\n # get mis-attributed across columns (dict insertion order may differ between processes).\n for column in sorted(self._pending_extra_logs):\n self._logs[\"extra\"][column].extend(gather_object(self._pending_extra_logs[column]))\n self._pending_extra_logs.clear()\n\n # Flush user-logged metrics (from log_metric), averaging across processes.\n # Keys must be sorted so that all ranks call accelerator.gather in the same order, otherwise values\n # get mis-attributed across metrics (dict insertion order may differ between processes).\n for name in sorted(self._pending_metrics):\n values = self._pending_metrics[name]\n local_mean = sum(values) / len(values)\n global_mean = self.accelerator.gather(torch.tensor(local_mean, device=device)).mean().item()\n self._metrics[mode][name].append(global_mean)\n self._pending_metrics.clear()\n\n if images is not None:\n self._logs[\"images\"].extend(gather_object(images))\n\n output = {\n \"prompt_ids\": prompt_ids,\n \"prompt_mask\": prompt_mask,\n \"completion_ids\": completion_ids,\n \"completion_mask\": completion_mask,\n \"old_logps\": old_logps,\n \"advantages\": advantages,\n }\n if \"pixel_values\" in forward_kwargs:\n output[\"pixel_values\"] = forward_kwargs[\"pixel_values\"]\n if \"image_grid_thw\" in forward_kwargs:\n output[\"image_grid_thw\"] = forward_kwargs[\"image_grid_thw\"]\n if \"pixel_attention_mask\" in forward_kwargs:\n output[\"pixel_attention_mask\"] = forward_kwargs[\"pixel_attention_mask\"]\n if \"image_sizes\" in forward_kwargs:\n output[\"image_sizes\"] = forward_kwargs[\"image_sizes\"]\n if \"token_type_ids\" in forward_kwargs:\n output[\"token_type_ids\"] = forward_kwargs[\"token_type_ids\"]\n if \"mm_token_type_ids\" in forward_kwargs:\n output[\"mm_token_type_ids\"] = forward_kwargs[\"mm_token_type_ids\"]\n if images is not None:\n output[\"num_images\"] = num_images\n return output\n\n @profiling_decorator\n def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):\n if return_outputs:\n raise ValueError(\"The RLOOTrainer does not support returning outputs\")\n return self._compute_loss(model, inputs)\n\n def _compute_loss(self, model, inputs):\n # Compute the per-token log probabilities for the model\n prompt_ids, prompt_mask = inputs[\"prompt_ids\"], inputs[\"prompt_mask\"]\n completion_ids, completion_mask = inputs[\"completion_ids\"], inputs[\"completion_mask\"]\n input_ids = torch.cat([prompt_ids, completion_ids], dim=1)\n attention_mask = torch.cat([prompt_mask, completion_mask], dim=1)\n logits_to_keep = completion_ids.size(1) # we only need to compute the logits for the completion tokens\n\n # Compute the per_token_logps and the entropy at each position in the completion\n per_token_logps, entropies = self._get_per_token_logps_and_entropies(\n model,\n input_ids,\n attention_mask,\n logits_to_keep,\n compute_entropy=True,\n pixel_values=inputs.get(\"pixel_values\"),\n image_grid_thw=inputs.get(\"image_grid_thw\"),\n num_images=inputs.get(\"num_images\"),\n pixel_attention_mask=inputs.get(\"pixel_attention_mask\"),\n image_sizes=inputs.get(\"image_sizes\"),\n token_type_ids=inputs.get(\"token_type_ids\"),\n mm_token_type_ids=inputs.get(\"mm_token_type_ids\"),\n )\n\n logps = (per_token_logps * completion_mask).sum(1) # mask out padding and tokens after EOS\n old_logps = inputs[\"old_logps\"]\n log_ratio = logps - old_logps\n\n # Compute the loss\n advantages = inputs[\"advantages\"]\n coef_1 = torch.exp(log_ratio)\n coef_2 = torch.clamp(coef_1, 1 - self.epsilon_low, 1 + self.epsilon_high)\n per_sequence_loss1 = coef_1 * advantages\n per_sequence_loss2 = coef_2 * advantages\n per_sequence_loss = -torch.min(per_sequence_loss1, per_sequence_loss2)\n loss = per_sequence_loss.mean()\n\n # Log the metrics\n mode = \"train\" if self.model.training else \"eval\"\n\n # Entropy\n mean_entropy = (entropies * completion_mask).sum() / completion_mask.sum().clamp(min=1.0)\n self._metrics[mode][\"entropy\"].append(self.accelerator.gather(mean_entropy).nanmean().item())\n\n # Compute the clipped probability ratios\n is_low_clipped = (coef_1 < 1 - self.epsilon_low) & (advantages < 0)\n is_high_clipped = (coef_1 > 1 + self.epsilon_high) & (advantages > 0)\n is_region_clipped = is_low_clipped | is_high_clipped\n gathered_low_clip = self.accelerator.gather(is_low_clipped.float().mean())\n self._metrics[mode][\"clip_ratio/low_mean\"].append(gathered_low_clip.nanmean().item())\n self._metrics[mode][\"clip_ratio/low_min\"].append(nanmin(gathered_low_clip).item())\n gathered_high_clip = self.accelerator.gather(is_high_clipped.float().mean())\n self._metrics[mode][\"clip_ratio/high_mean\"].append(gathered_high_clip.nanmean().item())\n self._metrics[mode][\"clip_ratio/high_max\"].append(nanmax(gathered_high_clip).item())\n gathered_clip_ratio = self.accelerator.gather(is_region_clipped.float().mean())\n self._metrics[mode][\"clip_ratio/region_mean\"].append(gathered_clip_ratio.nanmean().item())\n return loss\n\n # During eval, Trainer calls prediction_step. If no labels are present in the inputs, it only runs forward and\n # returns logits. We override prediction_step to force compute_loss, because this trainer doesn't involve labels.\n def prediction_step(self, model, inputs, prediction_loss_only, ignore_keys: list[str] | None = None):\n inputs = self._prepare_inputs(inputs)\n with torch.no_grad():\n with self.compute_loss_context_manager():\n loss = self.compute_loss(model, inputs)\n loss = loss.mean().detach()\n return loss, None, None\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n mode = \"train\" if self.model.training else \"eval\"\n metrics = {key: sum(val) / len(val) for key, val in self._metrics[mode].items()} # average the metrics\n\n # This method can be called both in training and evaluation. When called in evaluation, the keys in `logs`\n # start with \"eval_\". We need to add the prefix \"eval_\" to the keys in `metrics` to match the format.\n if mode == \"eval\":\n metrics = {f\"eval_{key}\": val for key, val in metrics.items()}\n\n logs = {**logs, **metrics}\n super().log(logs, start_time)\n self._metrics[mode].clear()\n\n if self.accelerator.is_main_process and self.log_completions:\n if is_rich_available():\n print_prompt_completions_sample(\n self._logs[\"prompt\"],\n self._logs[\"completion\"],\n self._logs[\"rewards\"],\n self._logs[\"advantages\"],\n self.state.global_step,\n self.num_completions_to_print,\n )\n\n logging_backends = []\n if self.args.report_to and \"wandb\" in self.args.report_to and wandb.run is not None:\n logging_backends.append(wandb)\n if self.args.report_to and \"trackio\" in self.args.report_to:\n logging_backends.append(trackio)\n\n table = {\n \"step\": [str(self.state.global_step)] * len(self._logs[\"prompt\"]),\n \"prompt\": self._logs[\"prompt\"],\n \"completion\": self._logs[\"completion\"],\n **self._logs[\"rewards\"],\n **self._logs[\"extra\"],\n \"advantage\": self._logs[\"advantages\"],\n }\n\n df_base = pd.DataFrame(table)\n images_raw = self._logs[\"images\"] or []\n\n for logging_backend in logging_backends:\n if images_raw:\n images = []\n for image_list in self._logs[\"images\"]:\n images.append([logging_backend.Image(image) for image in image_list])\n df = pd.concat(\n [df_base, pd.Series(images, name=\"image\")],\n axis=1,\n copy=False,\n )\n else:\n df = df_base\n\n if self.log_unique_prompts:\n df = df.drop_duplicates(subset=[\"prompt\"])\n\n logging_backend.log({\"completions\": logging_backend.Table(dataframe=df)})\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/trainer/sft_config.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport warnings\nfrom dataclasses import dataclass, field\nfrom typing import Any\n\nfrom .base_config import _BaseConfig\n\n\n@dataclass\nclass SFTConfig(_BaseConfig):\n # docstyle-ignore\n r\"\"\"\n Configuration class for the [`SFTTrainer`].\n\n This class includes only the parameters that are specific to SFT training. For a full list of training arguments,\n please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may\n differ from those in [`~transformers.TrainingArguments`].\n\n Using [`~transformers.HfArgumentParser`] we can turn this class into\n [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the\n command line.\n\n Parameters:\n > Parameters that control the model\n\n model_init_kwargs (`dict[str, Any]`, *optional*):\n Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model`\n argument of the [`SFTTrainer`] is provided as a string. If you're training a MoE architecture and want to\n include the load balancing/auxiliary loss as a part of the final loss, remember to set\n `output_router_logits=True` in this dictionary.\n chat_template_path (`str`, *optional*):\n If specified, sets the model's chat template. This can either be the path to a tokenizer (local directory\n or Hugging Face Hub model) or a direct path to a Jinja template file. When using a Jinja file, you must\n ensure that any special tokens referenced in the template are added to the tokenizer and that the model's\n embedding layer is resized accordingly.\n\n > Parameters that control the data preprocessing\n\n dataset_text_field (`str`, *optional*, defaults to `\"text\"`):\n Name of the column that contains text data in the dataset.\n dataset_kwargs (`dict[str, Any]`, *optional*):\n Dictionary of optional keyword arguments for the dataset preparation. The only supported key is\n `skip_prepare_dataset`. When the model is a VLM, `skip_prepare_dataset` is automatically treated as `True`\n regardless of the provided value, since preprocessing is done on the fly.\n dataset_num_proc (`int`, *optional*):\n Number of processes to use for processing the dataset.\n eos_token (`str`, *optional*):\n Token used to indicate the end of a turn or sequence. If `None`, it defaults to\n `processing_class.eos_token`.\n pad_token (`str`, *optional*):\n Token used for padding. If `None`, it defaults to `processing_class.pad_token`, or if that is also `None`,\n it falls back to `processing_class.eos_token`.\n max_length (`int` or `None`, *optional*, defaults to `1024`):\n Maximum length of the tokenized sequence. Sequences longer than `max_length` are truncated from the left\n or right depending on `truncation_mode`. If `None`, no truncation is applied. When packing is enabled,\n this value sets the sequence length.\n truncation_mode (`str`, *optional*, defaults to `\"keep_start\"`):\n Truncation mode to use when the sequence exceeds `max_length`. Possible values are `\"keep_end\"` and\n `\"keep_start\"`.\n shuffle_dataset (`bool`, *optional*, defaults to `False`):\n Whether to shuffle the dataset.\n packing (`bool`, *optional*, defaults to `False`):\n Whether to group multiple sequences into fixed-length blocks to improve computational efficiency and reduce\n padding. Uses `max_length` to define sequence length.\n packing_strategy (`str`, *optional*, defaults to `\"bfd\"`):\n Strategy for packing sequences. Can be `\"bfd\"` (best-fit decreasing, truncates overflow), `\"bfd_split\"`\n (best-fit decreasing, splits overflow sequences), or `\"wrapped\"` (aggressive, cuts mid-sequence).\n padding_free (`bool`, *optional*, defaults to `False`):\n Whether to perform forward passes without padding by flattening all sequences in the batch into a single\n continuous sequence. This reduces memory usage by eliminating padding overhead. Currently, this is only\n supported with the FlashAttention 2 or 3, which can efficiently handle the flattened batch structure. When\n packing is enabled with strategy `\"bfd\"`, padding-free is enabled, regardless of the value of this\n parameter.\n pad_to_multiple_of (`int`, *optional*):\n If set, the sequences will be padded to a multiple of this value.\n eval_packing (`bool`, *optional*):\n Whether to pack the eval dataset. If `None`, uses the same value as `packing`.\n\n > Parameters that control the training\n\n completion_only_loss (`bool`, *optional*):\n Whether to compute loss only on the completion part of the sequence. If set to `True`, loss is computed\n only on the completion, which is supported only for [prompt-completion](#prompt-completion) datasets. If\n `False`, loss is computed on the entire sequence. If `None` (default), the behavior depends on the dataset:\n loss is computed on the completion for [prompt-completion](#prompt-completion) datasets, and on the full\n sequence for [language modeling](#language-modeling) datasets.\n assistant_only_loss (`bool`, *optional*, defaults to `False`):\n Whether to compute loss only on the assistant part of the sequence. If set to `True`, loss is computed only\n on the assistant responses, which is supported only for [conversational](#conversational) datasets. If\n `False`, loss is computed on the entire sequence.\n loss_type (`str`, *optional*, defaults to `\"nll\"`):\n Type of loss to use. Possible values are `\"nll\"` (negative log-likelihood, default) and `\"dft\"` (Dynamic\n Fine-Tuning, as described in [this paper](https://huggingface.co/papers/2508.05629)).\n activation_offloading (`bool`, *optional*, defaults to `False`):\n Whether to offload the activations to the CPU.\n\n > [!NOTE]\n > These parameters have default values different from [`~transformers.TrainingArguments`]:\n > - `logging_steps`: Defaults to `10` instead of `500`.\n > - `gradient_checkpointing`: Defaults to `True` instead of `False`.\n > - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.\n > - `learning_rate`: Defaults to `2e-5` instead of `5e-5`.\n \"\"\"\n\n _VALID_DICT_FIELDS = _BaseConfig._VALID_DICT_FIELDS + [\"model_init_kwargs\"]\n\n # Parameters whose default values are overridden from TrainingArguments\n learning_rate: float = field(\n default=2e-5,\n metadata={\"help\": \"The initial learning rate for AdamW.\"},\n )\n\n # Parameters that control the model\n model_init_kwargs: dict[str, Any] | str | None = field(\n default=None,\n metadata={\n \"help\": \"Keyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `model` argument of \"\n \"the `SFTTrainer` is provided as a string. If you're training a MoE architecture and want to include the \"\n \"load balancing/auxiliary loss as a part of the final loss, remember to set `output_router_logits=True` \"\n \"in this dictionary.\"\n },\n )\n chat_template_path: str | None = field(\n default=None,\n metadata={\n \"help\": \"If specified, sets the model's chat template. This can either be the path to a tokenizer (local \"\n \"directory or Hugging Face Hub model) or a direct path to a Jinja template file. When using a Jinja file, \"\n \"you must ensure that any special tokens referenced in the template are added to the tokenizer and \"\n \"that the model's embedding layer is resized accordingly.\"\n },\n )\n\n # Parameters that control the data preprocessing\n dataset_text_field: str = field(\n default=\"text\",\n metadata={\"help\": \"Name of the column that contains text data in the dataset.\"},\n )\n dataset_kwargs: dict[str, Any] | None = field(\n default=None,\n metadata={\n \"help\": \"Dictionary of optional keyword arguments for the dataset preparation. The only supported key is \"\n \"`skip_prepare_dataset`. If the model is a VLM, `skip_prepare_dataset` value is ignored. When the model \"\n \"is a VLM, `skip_prepare_dataset` is automatically treated as `True` regardless of the provided value, \"\n \"since preprocessing is done on the fly.\"\n },\n )\n dataset_num_proc: int | None = field(\n default=None,\n metadata={\"help\": \"Number of processes to use for processing the dataset.\"},\n )\n eos_token: str | None = field(\n default=None,\n metadata={\n \"help\": \"Token used to indicate the end of a turn or sequence. If `None`, it defaults to `processing_class.eos_token`.\"\n },\n )\n pad_token: str | None = field(\n default=None,\n metadata={\n \"help\": \"Token used for padding. If `None`, it defaults to `processing_class.pad_token`, or if that \"\n \"is also `None`, it falls back to `processing_class.eos_token`.\"\n },\n )\n max_length: int | None = field(\n default=1024,\n metadata={\n \"help\": \"Maximum length of the tokenized sequence. Sequences longer than `max_length` are truncated from \"\n \"the left or right depending on the `truncation_mode`. If `None`, no truncation is applied. When packing \"\n \"is enabled, this value sets the sequence length.\"\n },\n )\n truncation_mode: str = field(\n default=\"keep_start\",\n metadata={\n \"help\": \"Truncation mode to use when the sequence exceeds `max_length`. Possible values are `'keep_end'` \"\n \"and `'keep_start'`.\",\n \"choices\": [\"keep_end\", \"keep_start\"],\n },\n )\n shuffle_dataset: bool = field(\n default=False,\n metadata={\"help\": \"Whether to shuffle the dataset.\"},\n )\n packing: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to group multiple sequences into fixed-length blocks to improve computational efficiency \"\n \"and reduce padding. Uses `max_length` to define sequence length.\"\n },\n )\n packing_strategy: str = field(\n default=\"bfd\",\n metadata={\n \"help\": \"Strategy for packing sequences. Can be `'bfd'` (best-fit decreasing, truncates overflow), \"\n \"`'bfd_split'` (best-fit decreasing, splits overflow sequences), or `'wrapped'` (aggressive, cuts \"\n \"mid-sequence).\",\n \"choices\": [\"bfd\", \"bfd_split\", \"wrapped\"],\n },\n )\n padding_free: bool = field(\n default=False,\n metadata={\n \"help\": \"Whether to perform forward passes without padding by flattening all sequences in the batch into \"\n \"a single continuous sequence. This reduces memory usage by eliminating padding overhead. Currently, this \"\n \"is only supported with the FlashAttention 2 or 3, which can efficiently handle the flattened batch \"\n \"structure. When packing is enabled with strategy `'bfd'`, padding-free is enabled, regardless of the \"\n \"value of this parameter.\"\n },\n )\n pad_to_multiple_of: int | None = field(\n default=None,\n metadata={\"help\": \"If set, the sequences will be padded to a multiple of this value.\"},\n )\n eval_packing: bool | None = field(\n default=None,\n metadata={\"help\": \"Whether to pack the eval dataset. If `None`, uses the same value as `packing`.\"},\n )\n\n # Parameters that control the training\n completion_only_loss: bool | None = field(\n default=None,\n metadata={\n \"help\": (\n \"Whether to compute loss only on the completion part of the sequence. If set to `True`, loss is \"\n \"computed only on the completion, which is supported only for prompt-completion datasets. If `False`, \"\n \"loss is computed on the entire sequence. If `None` (default), the behavior depends on the dataset: \"\n \"loss is computed on the completion for prompt-completion datasets, and on the full sequence for \"\n \"language modeling datasets.\"\n )\n },\n )\n assistant_only_loss: bool = field(\n default=False,\n metadata={\n \"help\": (\n \"Whether to compute loss only on the assistant part of the sequence. If set to `True`, loss is \"\n \"computed only on the assistant responses, which is supported only for conversational datasets. If `False`, \"\n \"loss is computed on the entire sequence.\"\n )\n },\n )\n loss_type: str = field(\n default=\"nll\",\n metadata={\n \"help\": (\n 'Type of loss to use. Possible values are `\"nll\"` (negative log-likelihood, default) and `\"dft\"` '\n \"(Dynamic Fine-Tuning, as described in https://huggingface.co/papers/2508.05629).\"\n )\n },\n )\n activation_offloading: bool = field(\n default=False,\n metadata={\"help\": \"Whether to offload the activations to the CPU.\"},\n )\n\n def __post_init__(self):\n super().__post_init__()\n\n if self.packing_strategy == \"bfd-requeue\":\n warnings.warn(\n \"The `bfd-requeue` packing strategy has been renamed to `bfd_split`. Please update your configuration accordingly. \"\n \"The `bfd-requeue` strategy is deprecated and will be removed in a future version.\",\n FutureWarning,\n )\n self.packing_strategy = \"bfd_split\"\n" }, { "path": "trl/trainer/sft_trainer.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport contextlib\nimport json\nimport os\nimport warnings\nfrom collections import defaultdict\nfrom collections.abc import Callable\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Any\n\nimport torch\nimport torch.nn as nn\nimport transformers\nfrom accelerate import PartialState\nfrom accelerate.logging import get_logger\nfrom accelerate.utils import is_peft_model\nfrom datasets import Dataset, IterableDataset\nfrom packaging.version import Version\nfrom transformers import (\n AutoProcessor,\n DataCollator,\n PreTrainedModel,\n PreTrainedTokenizerBase,\n ProcessorMixin,\n TrainingArguments,\n)\nfrom transformers.data.data_collator import DataCollatorMixin\nfrom transformers.trainer_callback import TrainerCallback\nfrom transformers.trainer_utils import EvalPrediction\nfrom transformers.utils import is_peft_available\n\nfrom ..chat_template_utils import clone_chat_template\nfrom ..data_utils import (\n apply_chat_template,\n is_conversational,\n is_conversational_from_value,\n maybe_convert_to_chatml,\n pack_dataset,\n prepare_multimodal_messages,\n truncate_dataset,\n)\nfrom ..models import get_act_offloading_ctx_manager\nfrom .base_trainer import _BaseTrainer\nfrom .sft_config import SFTConfig\nfrom .utils import (\n create_model_from_path,\n entropy_from_logits,\n flush_left,\n get_config_model_id,\n pad,\n remove_none_values,\n selective_log_softmax,\n)\n\n\nif is_peft_available():\n from peft import PeftConfig, PeftModel, PeftType, get_peft_model\n\n\nlogger = get_logger(__name__)\n\n\nFLASH_ATTENTION_VARIANTS = {\n \"flash_attention_2\",\n \"flash_attention_3\",\n \"kernels-community/flash-attn2\",\n \"kernels-community/flash-attn3\",\n \"kernels-community/vllm-flash-attn3\",\n}\n\n\ndef get_dataset_column_names(dataset: Dataset | IterableDataset) -> list[str]:\n return list(next(iter(dataset)).keys()) if dataset.column_names is None else dataset.column_names\n\n\n@dataclass\nclass DataCollatorForLanguageModeling(DataCollatorMixin):\n \"\"\"\n Data collator used for language modeling data. Inputs are dynamically padded to the maximum length of a batch.\n\n This collator expects each example in the input list to be a dictionary containing at least the `\"input_ids\"` key.\n If the input contains a `\"completion_mask\"`, it is used to set the labels to `-100` for tokens that are not in the\n completion. If `\"assistant_masks\"` are present, they are used to set the labels to `-100` for tokens that are not\n in the assistant part of the sequence. The collator returns a dictionary containing the following keys:\n - `\"input_ids\"`: Tensor of input IDs, padded to the maximum length of the batch.\n - `\"labels\"`: Tensor of labels, padded to the maximum length of the batch. If `completion_only_loss` is set to\n `True`, tokens that are not in the completion are set to -100. If `assistant_masks` are present, tokens that are\n not in the assistant part of the sequence are set to -100. If `padding_free` is set to `False`, the following key\n is also returned:\n - `\"attention_mask\"`: Tensor of attention masks, padded to the maximum length of the batch.\n If `padding_free` is set to `True`, the following key is also returned:\n - `\"position_ids\"`: Tensor of position IDs, padded to the maximum length of the batch.\n\n Args:\n pad_token_id (`int`):\n Token ID to use for padding.\n completion_only_loss (`bool`, *optional*, defaults to `True`):\n When the input contains a completion mask (`completion_mask`), the labels are set to -100 for the tokens\n that are no in the completion.\n padding_free (`bool`, *optional*, defaults to `False`):\n If set to `True`, the sequences will be flattened into a single sequence, and the position IDs will be\n generated accordingly and returned instead of the attention mask.\n pad_to_multiple_of (`int`, *optional*):\n If set, the sequences will be padded to a multiple of this value.\n return_tensors (`str`, *optional*, defaults to `\"pt\"`):\n Type of Tensor to return. Only `\"pt\"` is currently supported.\n\n Examples:\n ```python\n >>> from trl.trainer.sft_trainer import DataCollatorForLanguageModeling\n\n >>> collator = DataCollatorForLanguageModeling(pad_token_id=0)\n >>> examples = [{\"input_ids\": [1, 2, 3]}, {\"input_ids\": [4, 5]}]\n >>> collator(examples)\n {'input_ids': tensor([[ 1, 2, 3],\n [ 4, 5, 0]]),\n 'attention_mask': tensor([[ 1, 1, 1],\n [ 1, 1, 0]]),\n 'labels': tensor([[ 1, 2, 3],\n [ 4, 5, -100]])}\n\n >>> # With completion mask\n >>> examples = [\n ... {\"input_ids\": [1, 2, 3], \"completion_mask\": [0, 1, 1]},\n ... {\"input_ids\": [4, 5], \"completion_mask\": [0, 1]},\n ... ]\n >>> collator(examples)\n {'input_ids': tensor([[ 1, 2, 3],\n [ 4, 5, 0]]),\n 'attention_mask': tensor([[ 1, 1, 1],\n [ 1, 1, 0]]),\n 'labels': tensor([[-100, 2, 3],\n [-100, 5, -100]])}\n\n >>> # With padding_free\n >>> collator = DataCollatorForLanguageModeling(pad_token_id=0, padding_free=True)\n >>> collator(examples)\n {'input_ids': tensor([[ 1, 2, 3, 4, 5]]),\n 'position_ids': tensor([[0, 1, 2, 0, 1]]),\n 'labels': tensor([[1, 2, 3, 4, 5]])}\n ```\n \"\"\"\n\n pad_token_id: int\n completion_only_loss: bool = True\n padding_free: bool = False\n pad_to_multiple_of: int | None = None\n return_tensors: str = \"pt\"\n\n def torch_call(self, examples: list[dict[str, Any]]) -> dict[str, Any]:\n # Convert to tensor\n input_ids = [torch.tensor(example[\"input_ids\"]) for example in examples]\n if \"labels\" in examples[0]:\n labels = [torch.tensor(example[\"labels\"]) for example in examples]\n else:\n labels = [torch.tensor(example[\"input_ids\"]) for example in examples]\n\n # For padding-free, we should NOT create attention_mask as it causes FlashAttention to ignore position_ids and\n # compute wrong cu_seq_lens from the all-1s mask\n if self.padding_free:\n if \"seq_lengths\" in examples[0]:\n position_ids = self.get_position_ids_from_packed_seq_lengths(\n [example[\"seq_lengths\"] for example in examples]\n )\n else:\n position_ids = [torch.arange(len(ids)) for ids in input_ids]\n else:\n attention_mask = [torch.ones_like(ids) for ids in input_ids]\n if self.completion_only_loss and \"completion_mask\" in examples[0]:\n completion_mask = [torch.tensor(example[\"completion_mask\"]) for example in examples]\n if \"assistant_masks\" in examples[0]:\n assistant_masks = [torch.tensor(example[\"assistant_masks\"]) for example in examples]\n\n # If padding_free, flatten everything into a single sequence\n output = {}\n if self.padding_free:\n input_ids = [torch.cat(input_ids, dim=0)]\n labels = [torch.cat(labels, dim=0)]\n position_ids = [torch.cat(position_ids, dim=0)]\n if self.completion_only_loss and \"completion_mask\" in examples[0]:\n completion_mask = [torch.cat(completion_mask, dim=0)]\n if \"assistant_masks\" in examples[0]:\n assistant_masks = [torch.cat(assistant_masks, dim=0)]\n\n # Pad\n output[\"input_ids\"] = pad(\n input_ids,\n padding_value=self.pad_token_id,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n )\n output[\"labels\"] = pad(\n labels, padding_value=-100, padding_side=\"right\", pad_to_multiple_of=self.pad_to_multiple_of\n )\n if self.padding_free:\n output[\"position_ids\"] = pad(\n position_ids, padding_value=0, padding_side=\"right\", pad_to_multiple_of=self.pad_to_multiple_of\n )\n output[\"labels\"][output[\"position_ids\"] == 0] = -100\n else:\n output[\"attention_mask\"] = pad(\n attention_mask, padding_value=0, padding_side=\"right\", pad_to_multiple_of=self.pad_to_multiple_of\n )\n if self.completion_only_loss and \"completion_mask\" in examples[0]:\n completion_mask = pad(\n completion_mask, padding_value=0, padding_side=\"right\", pad_to_multiple_of=self.pad_to_multiple_of\n )\n output[\"labels\"][completion_mask == 0] = -100 # mask everything that is not in the completion\n if \"assistant_masks\" in examples[0]:\n assistant_masks = pad(\n assistant_masks, padding_value=0, padding_side=\"right\", pad_to_multiple_of=self.pad_to_multiple_of\n )\n output[\"labels\"][assistant_masks == 0] = -100\n return output\n\n @staticmethod\n def get_position_ids_from_packed_seq_lengths(batch_seq_lengths: list[list[int]]) -> list[torch.Tensor]:\n \"\"\"\n Get position IDs for packed sequences.\n\n Args:\n batch_seq_lengths (`list[list[int]]`):\n A list of lists containing the lengths of each individual document in the packed batch.\n\n Return:\n `list[torch.Tensor]`:\n A list of tensors containing the position IDs for each packed sequence.\n \"\"\"\n # Get lengths per row\n example_lengths = [sum(seq_lengths) for seq_lengths in batch_seq_lengths]\n # Flat list of lengths\n batch_seq_lengths = torch.tensor(\n [seq_length for seq_lengths in batch_seq_lengths for seq_length in seq_lengths]\n )\n position_ids = torch.ones(sum(example_lengths), dtype=batch_seq_lengths.dtype)\n position_ids[0] = 0\n # Reset position ids to 0 at the start of each sequence\n position_ids[batch_seq_lengths[:-1].cumsum(0)] = -(batch_seq_lengths[:-1] - 1)\n position_ids = position_ids.cumsum(0)\n # Split back into one tensor per example\n return list(position_ids.split(example_lengths))\n\n\n@dataclass\nclass DataCollatorForVisionLanguageModeling(DataCollatorMixin):\n \"\"\"\n Data collator for vision-language modeling tasks.\n\n Unlike text-only datasets, where the collator typically receives pre-tokenized inputs ready for batching,\n vision-language data processing involves converting images into pixel values. This conversion is disk-intensive,\n making upfront preprocessing of the entire dataset impractical. Therefore, this collator performs tokenization and\n image processing on-the-fly to efficiently prepare batches.\n\n Each input example should be a dictionary containing at least:\n - An `\"images\"` key holding a list of images, or an `\"image\"` key holding a single image.\n - [language modeling](#language-modeling) type: either a `\"messages\"` key for conversational inputs or a `\"text\"`\n key for standard text inputs.\n - [prompt-completion](#prompt-completion) type: keys `\"prompt\"` and `\"completion\"` for the prompt and completion.\n\n The collator outputs a dictionary including:\n - `\"input_ids\"`: Tensor of token IDs.\n - `\"attention_mask\"`: Tensor indicating attention mask.\n - `\"pixel_values\"`: Tensor representing image pixel values.\n - `\"labels\"`: Tensor for training labels.\n\n Additional keys may be present depending on the processor, such as `\"image_grid_thw\"`.\n\n Args:\n processor ([`~transformers.ProcessorMixin`]):\n The processor used to tokenize text and process images. It must be a subclass of\n [`~transformers.ProcessorMixin`] and include a `tokenizer` with a defined `pad_token_id`.\n max_length (`int` or `None`, optional, defaults to `None`):\n Maximum sequence length for input tokens. If `None`, no truncation is applied.\n completion_only_loss (`bool`, *optional*, defaults to `False`):\n Whether to compute loss only on the completion part of the sequence. When `True`, the labels for the prompt\n part are set to -100. It requires the dataset type to be prompt-completion.\n pad_to_multiple_of (`int` or `None`, optional, defaults to `None`):\n If set, the sequences will be padded to a multiple of this value.\n dataset_text_field (`str`, optional, defaults to `\"text\"`):\n Name of the column that contains text data in the dataset. This parameter is only relevant for [standard\n datasets format](dataset_formats#standard).\n return_tensors (`str`, optional, defaults to `\"pt\"`):\n The tensor type to return. Currently, only `\"pt\"` (PyTorch tensors) is supported.\n\n Example:\n ```python\n >>> from trl.trainer.sft_trainer import DataCollatorForVisionLanguageModeling\n >>> from transformers import AutoProcessor\n\n >>> processor = AutoProcessor.from_pretrained(\"Qwen/Qwen2.5-VL-7B-Instruct\")\n >>> collator = DataCollatorForVisionLanguageModeling(processor)\n >>> examples = [\n ... {\"images\": [Image.open(\"image_0.png\")], \"messages\": [{\"role\": \"user\", \"content\": \"What is this?\"}]},\n ... {\"images\": [Image.open(\"image_1.png\")], \"messages\": [{\"role\": \"user\", \"content\": \"Describe this image.\"}]},\n ... ]\n >>> collator(examples)\n {'input_ids': tensor([[151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198,\n 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 3838, 374,\n 419, 30, 151645, 198],\n [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198,\n 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 74785, 419,\n 2168, 13, 151645, 198]]),\n 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]),\n 'pixel_values': tensor([[-0.9893, 0.1785, 1.5362, ..., -0.0582, 0.8661, -0.2431],\n [-0.2302, 0.9522, -1.1061, ..., 0.0555, 1.3354, -0.6412],\n [ 1.2150, 0.9084, 0.7041, ..., 0.2404, -0.8403, -0.5133],\n ...,\n [ 0.6895, 0.2807, 0.2515, ..., -0.2004, -1.2100, 0.0555],\n [ 0.8209, -0.9748, 1.5654, ..., 1.6055, -0.4706, 0.5817],\n [-1.0915, 0.4559, 0.9230, ..., 0.5106, 0.0982, -0.1720]]),\n 'image_grid_thw': tensor([[1, 4, 4],\n [1, 4, 4]]),\n 'labels': tensor([[151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198,\n 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 3838, 374,\n 419, 30, 151645, 198],\n [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198,\n 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 74785, 419,\n 2168, 13, 151645, 198]])}\n ```\n \"\"\"\n\n processor: ProcessorMixin\n max_length: int | None = None\n completion_only_loss: bool = False # default not used in practice; SFTTrainer always passes the relevant value\n pad_to_multiple_of: int | None = None\n dataset_text_field: str = \"text\"\n return_tensors: str = \"pt\"\n\n def torch_call(self, examples: list[dict[str, Any]]) -> dict[str, Any]:\n if \"messages\" in examples[0] or self.dataset_text_field in examples[0]:\n if self.completion_only_loss:\n raise ValueError(\n \"The `completion_only_loss` argument is not supported for language modeling datasets.\"\n )\n return self._collate_language_modeling(examples)\n elif \"prompt\" in examples[0] and \"completion\" in examples[0]:\n return self._collate_prompt_completion(examples)\n else:\n raise KeyError(f\"Unexpected input keys in examples: {list(examples[0].keys())}.\")\n\n def _collate_language_modeling(self, examples: list[dict[str, Any]]) -> dict[str, Any]:\n if \"image\" in examples[0]:\n for example in examples:\n example[\"images\"] = [example.pop(\"image\")]\n images = [example[\"images\"] for example in examples]\n # Transformers requires at least one image in the batch, otherwise it throws an error\n if all(img_list == [] for img_list in images):\n images = None\n\n if \"messages\" in examples[0]: # conversational case\n messages = [prepare_multimodal_messages(example[\"messages\"], example[\"images\"]) for example in examples]\n texts = self.processor.apply_chat_template(messages)\n elif self.dataset_text_field in examples[0]: # standard case\n texts = [example[self.dataset_text_field] for example in examples]\n else:\n raise KeyError(\n \"The input examples must contain either 'messages' for conversational data or 'text' for standard \"\n \"data.\"\n )\n\n output = self.processor(\n images=images,\n text=texts,\n padding=True,\n padding_side=\"right\",\n pad_to_multiple_of=self.pad_to_multiple_of,\n truncation=self.max_length is not None,\n max_length=self.max_length,\n return_tensors=self.return_tensors,\n add_special_tokens=False, # to avoid adding the BOS, twice see https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior#7-chat-template-and-tokenization-dont-compose-due-to-special-tokens\n )\n labels = output[\"input_ids\"].clone()\n labels[output[\"attention_mask\"] == 0] = -100\n # We mask only padding tokens (-100) in the labels. Vision tokens are left unchanged because their handling in\n # loss computation has to be done by the model, and masking them here would be infeasible in practice as vision\n # token definitions vary across architectures.\n output[\"labels\"] = labels\n return output\n\n def _collate_prompt_completion(self, examples: list[dict[str, Any]]) -> dict[str, Any]:\n if self.pad_to_multiple_of is not None:\n raise NotImplementedError(\n \"Padding to a multiple of a value is not yet implemented for vision-language modeling and \"\n \"prompt-completion data.\"\n )\n if \"image\" in examples[0]:\n for example in examples:\n example[\"images\"] = [example.pop(\"image\")]\n images = [example[\"images\"] for example in examples]\n # Transformers requires at least one image in the batch, otherwise it throws an error\n if all(img_list == [] for img_list in images):\n images = None\n if is_conversational(examples[0]): # conversational case\n for example in examples:\n example[\"prompt\"] = prepare_multimodal_messages(example[\"prompt\"], images=example[\"images\"])\n example[\"completion\"] = prepare_multimodal_messages(example[\"completion\"], images=[])\n examples = [apply_chat_template(example, self.processor) for example in examples]\n\n prompts = [example[\"prompt\"] for example in examples]\n completions = [example[\"completion\"] for example in examples]\n\n processed_prompts = self.processor(\n images=images,\n text=prompts,\n padding=True,\n padding_side=\"left\",\n return_tensors=self.return_tensors,\n add_special_tokens=False, # to avoid adding the BOS, twice see https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior#7-chat-template-and-tokenization-dont-compose-due-to-special-tokens\n )\n processed_completions = self.processor(\n text=completions,\n padding=True,\n padding_side=\"right\",\n return_tensors=self.return_tensors,\n add_special_tokens=False, # to avoid adding the BOS, twice see https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior#7-chat-template-and-tokenization-dont-compose-due-to-special-tokens\n )\n\n # Concatenate prompts and completions\n prompt_ids, prompt_mask = processed_prompts[\"input_ids\"], processed_prompts[\"attention_mask\"]\n completion_ids, completion_mask = processed_completions[\"input_ids\"], processed_completions[\"attention_mask\"]\n input_ids = torch.cat((prompt_ids, completion_ids), dim=1)\n attention_mask = torch.cat((prompt_mask, completion_mask), dim=1)\n completion_mask = torch.cat((torch.zeros_like(prompt_mask), completion_mask), dim=1)\n if \"token_type_ids\" in processed_prompts: # special case for Gemma\n prompt_token_type_ids = processed_prompts[\"token_type_ids\"]\n completion_token_type_ids = processed_completions[\"token_type_ids\"]\n token_type_ids = torch.cat((prompt_token_type_ids, completion_token_type_ids), dim=1)\n if \"mm_token_type_ids\" in processed_prompts: # special case for ERNIE-VL\n prompt_mm_token_type_ids = processed_prompts[\"mm_token_type_ids\"]\n completion_mm_token_type_ids = processed_completions.get(\n \"mm_token_type_ids\", torch.zeros_like(completion_ids)\n )\n mm_token_type_ids = torch.cat((prompt_mm_token_type_ids, completion_mm_token_type_ids), dim=1)\n\n # Flush left to reduce padding\n if \"token_type_ids\" in processed_prompts and \"mm_token_type_ids\" in processed_prompts:\n attention_mask, input_ids, completion_mask, token_type_ids, mm_token_type_ids = flush_left(\n attention_mask, input_ids, completion_mask, token_type_ids, mm_token_type_ids\n )\n elif \"token_type_ids\" in processed_prompts:\n attention_mask, input_ids, completion_mask, token_type_ids = flush_left(\n attention_mask, input_ids, completion_mask, token_type_ids\n )\n elif \"mm_token_type_ids\" in processed_prompts:\n attention_mask, input_ids, completion_mask, mm_token_type_ids = flush_left(\n attention_mask, input_ids, completion_mask, mm_token_type_ids\n )\n else:\n attention_mask, input_ids, completion_mask = flush_left(attention_mask, input_ids, completion_mask)\n\n # Truncate if necessary\n if self.max_length is not None:\n input_ids = input_ids[:, : self.max_length]\n attention_mask = attention_mask[:, : self.max_length]\n completion_mask = completion_mask[:, : self.max_length]\n if \"token_type_ids\" in processed_prompts:\n token_type_ids = token_type_ids[:, : self.max_length]\n if \"mm_token_type_ids\" in processed_prompts:\n mm_token_type_ids = mm_token_type_ids[:, : self.max_length]\n\n # Create labels and mask padding tokens\n labels = input_ids.clone()\n labels[attention_mask == 0] = -100\n if self.completion_only_loss:\n labels[completion_mask == 0] = -100\n\n # Build the output dictionary\n output = processed_prompts # we take processed_prompts because it contains the images\n output[\"input_ids\"] = input_ids\n output[\"attention_mask\"] = attention_mask\n output[\"labels\"] = labels\n if \"token_type_ids\" in processed_prompts:\n output[\"token_type_ids\"] = token_type_ids\n if \"mm_token_type_ids\" in processed_prompts:\n output[\"mm_token_type_ids\"] = mm_token_type_ids\n return output\n\n\ndef dft_loss(outputs, labels, num_items_in_batch=None):\n \"\"\"\n DFT loss function, as presented in [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward\n Rectification](https://huggingface.co/papers/2508.05629)\n \"\"\"\n labels = nn.functional.pad(labels, (0, 1), value=-100)\n shift_labels = labels[..., 1:].contiguous()\n loss_mask = shift_labels != -100\n shift_labels[~loss_mask] = 0\n logprobs = selective_log_softmax(outputs.logits, shift_labels)\n per_token_loss = -logprobs.exp().detach() * logprobs\n if num_items_in_batch is None:\n num_items_in_batch = loss_mask.sum()\n loss = (per_token_loss * loss_mask).sum() / num_items_in_batch\n return loss\n\n\nclass SFTTrainer(_BaseTrainer):\n \"\"\"\n Trainer for Supervised Fine-Tuning (SFT) method.\n\n This class is a wrapper around the [`~transformers.Trainer`] class and inherits all of its attributes and methods.\n\n Example:\n\n ```python\n from trl import SFTTrainer\n from datasets import load_dataset\n\n dataset = load_dataset(\"roneneldan/TinyStories\", split=\"train[:1%]\")\n\n trainer = SFTTrainer(\n model=\"Qwen/Qwen2.5-0.5B-Instruct\",\n train_dataset=dataset,\n )\n trainer.train()\n ```\n\n Args:\n model (`str` or [`~transformers.PreTrainedModel`] or [`~peft.PeftModel`]):\n Model to be trained. Can be either:\n\n - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a\n path to a *directory* containing model weights saved using\n [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded\n using `.from_pretrained` (where `` is derived from the model\n config) with the keyword arguments in `args.model_init_kwargs`.\n - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported.\n - A [`~peft.PeftModel`] object. Only causal language models are supported.\n If you're training a model with an MoE architecture and want to include the load balancing/auxiliary loss\n as a part of the final loss, remember to set the `output_router_logits` config of the model to `True`.\n args ([`SFTConfig`], *optional*):\n Configuration for this trainer. If `None`, a default configuration is used.\n data_collator ([`~transformers.DataCollator`], *optional*):\n Function to use to form a batch from a list of elements of the processed `train_dataset` or `eval_dataset`.\n Will default to [`~trainer.sft_trainer.DataCollatorForLanguageModeling`] if the model is a language model\n and [`~trainer.sft_trainer.DataCollatorForVisionLanguageModeling`] if the model is a vision-language model.\n train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]):\n Dataset to use for training. This trainer supports both [language modeling](#language-modeling) type and\n [prompt-completion](#prompt-completion) type. The format of the samples can be either:\n\n - [Standard](dataset_formats#standard): Each sample contains plain text.\n - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role\n and content).\n\n The trainer also supports processed datasets (tokenized) as long as they contain an `input_ids` field.\n eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Dataset | IterableDataset]`):\n Dataset to use for evaluation. It must meet the same requirements as `train_dataset`.\n processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`], *optional*):\n Processing class used to process the data. If `None`, the processing class is loaded from the model's name\n with [`~transformers.AutoProcessor.from_pretrained`]. A padding token, `tokenizer.pad_token`, must be set.\n If the processing class has not set a padding token, `tokenizer.eos_token` will be used as the default.\n compute_loss_func (`Callable`, *optional*):\n A function that accepts the raw model outputs, labels, and the number of items in the entire accumulated\n batch (batch_size * gradient_accumulation_steps) and returns the loss. For example, see the default [loss\n function](https://github.com/huggingface/transformers/blob/052e652d6d53c2b26ffde87e039b723949a53493/src/transformers/trainer.py#L3618)\n used by [`Trainer`].\n compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):\n The function that will be used to compute metrics at evaluation. Must take a\n [`~transformers.EvalPrediction`] and return a dictionary string to metric values. When passing\n [`SFTConfig`] with `batch_eval_metrics` set to `True`, your `compute_metrics` function must take a boolean\n `compute_result` argument. This will be triggered after the last eval batch to signal that the function\n needs to calculate and return the global summary statistics rather than accumulating the batch-level\n statistics.\n callbacks (list of [`~transformers.TrainerCallback`], *optional*):\n List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed\n in [here](https://huggingface.co/docs/transformers/main_classes/callback).\n\n If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`]\n method.\n optimizers (`tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None]`, *optional*, defaults to `(None, None)`):\n A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your\n model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`.\n optimizer_cls_and_kwargs (`tuple[Type[torch.optim.Optimizer], Dict[str, Any]]`, *optional*):\n A tuple containing the optimizer class and keyword arguments to use. Overrides `optim` and `optim_args` in\n `args`. Incompatible with the `optimizers` argument.\n\n Unlike `optimizers`, this argument avoids the need to place model parameters on the correct devices before\n initializing the Trainer.\n preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`, *optional*):\n A function that preprocess the logits right before caching them at each evaluation step. Must take two\n tensors, the logits and the labels, and return the logits once processed as desired. The modifications made\n by this function will be reflected in the predictions received by `compute_metrics`.\n\n Note that the labels (second parameter) will be `None` if the dataset does not have them.\n peft_config ([`~peft.PeftConfig`], *optional*):\n PEFT configuration used to wrap the model. If `None`, the model is not wrapped.\n formatting_func (`Callable`, *optional*):\n Formatting function applied to the dataset before tokenization. Applying the formatting function explicitly\n converts the dataset into a [language modeling](#language-modeling) type.\n \"\"\"\n\n _tag_names = [\"trl\", \"sft\"]\n _name = \"SFT\"\n\n def __init__(\n self,\n model: \"str | PreTrainedModel | PeftModel\",\n args: SFTConfig | TrainingArguments | None = None,\n data_collator: DataCollator | None = None,\n train_dataset: Dataset | IterableDataset | None = None,\n eval_dataset: Dataset | IterableDataset | dict[str, Dataset | IterableDataset] | None = None,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin | None = None,\n compute_loss_func: Callable | None = None,\n compute_metrics: Callable[[EvalPrediction], dict] | None = None,\n callbacks: list[TrainerCallback] | None = None,\n optimizers: tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None] = (None, None),\n optimizer_cls_and_kwargs: tuple[type[torch.optim.Optimizer], dict[str, Any]] | None = None,\n preprocess_logits_for_metrics: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | None = None,\n peft_config: \"PeftConfig | None\" = None,\n formatting_func: Callable[[dict], str] | None = None,\n ):\n # Args\n if args is None:\n model_name = model if isinstance(model, str) else get_config_model_id(model.config)\n model_name = model_name.split(\"/\")[-1]\n args = SFTConfig(f\"{model_name}-SFT\")\n elif isinstance(args, TrainingArguments) and not isinstance(args, SFTConfig):\n dict_args = args.to_dict()\n dict_args[\"hub_token\"] = args.hub_token # to_dict hides the hub_token\n if Version(transformers.__version__) < Version(\"5.0.0\"):\n dict_args.pop(\"push_to_hub_token\")\n args = SFTConfig(**dict_args)\n\n if train_dataset is None:\n raise ValueError(\"`train_dataset` is required\")\n elif isinstance(train_dataset, IterableDataset):\n # IterableDataset requires dispatch_batches=False because Accelerate's dispatch mode may try to concatenate\n # batches from multiple processes, leading to mismatch errors.\n if args.accelerator_config.dispatch_batches is True:\n logger.warning(\n \"You are using an `IterableDataset` for training with `dispatch_batches=True`. `dispatch_batches` \"\n \"is forced to `False` when using an `IterableDataset`. To remove this warning, unset \"\n \"`dispatch_batches` in `SFTConfig` or set it to `False`.\"\n )\n args.accelerator_config.dispatch_batches = False\n\n # Model\n if isinstance(model, str):\n model_init_kwargs = args.model_init_kwargs or {}\n # Distributed training requires device_map=None (\"auto\" fails)\n if args.distributed_state.distributed_type in [\"MULTI_GPU\", \"DEEPSPEED\"]:\n model_init_kwargs[\"device_map\"] = None\n model = create_model_from_path(model, **model_init_kwargs)\n else:\n if args.model_init_kwargs is not None:\n logger.warning(\n \"You passed `model_init_kwargs` to the `SFTConfig`, but your model is already instantiated. \"\n \"The `model_init_kwargs` will be ignored.\"\n )\n\n # Processing class\n if processing_class is None:\n processing_class = AutoProcessor.from_pretrained(get_config_model_id(model.config))\n\n # Handle pad token for processors or tokenizers\n if isinstance(processing_class, ProcessorMixin):\n tokenizer = processing_class.tokenizer\n self._is_vlm = True\n elif isinstance(processing_class, PreTrainedTokenizerBase):\n tokenizer = processing_class\n self._is_vlm = False\n else:\n raise TypeError(\"The `processing_class` must be either a `PreTrainedTokenizerBase` or a `ProcessorMixin`\")\n\n if args.eos_token is not None:\n eos_token = args.eos_token\n eos_token_id = tokenizer.convert_tokens_to_ids(eos_token)\n if eos_token_id is None:\n raise ValueError(\n f\"The specified `eos_token` ('{eos_token}') is not found in the vocabulary of the given \"\n f\"`processing_class` ({processing_class.__class__.__name__}). Ensure that the `eos_token` exists \"\n \"in the vocabulary before using it as an EOS token.\"\n )\n tokenizer.eos_token_id = eos_token_id\n\n if args.chat_template_path is not None:\n if os.path.isfile(args.chat_template_path) and args.chat_template_path.endswith((\".jinja\", \".j2\")):\n with open(args.chat_template_path, encoding=\"utf-8\") as chat_template_file:\n processing_class.chat_template = chat_template_file.read()\n added_tokens = []\n else:\n model, processing_class, added_tokens = clone_chat_template(\n model, processing_class, args.chat_template_path\n )\n else:\n added_tokens = []\n\n # Catch some wrong configurations related to VLMs\n if self._is_vlm and args.packing:\n raise ValueError(\n \"Packing is not supported for vision-language models. Please set `packing=False` in the SFTConfig.\"\n )\n if self._is_vlm and args.padding_free:\n raise ValueError(\n \"Padding-free training is yet not supported for vision-language models. Please set \"\n \"`padding_free=False` in the `SFTConfig`.\"\n )\n if self._is_vlm and args.assistant_only_loss:\n raise ValueError(\n \"Assistant-only loss is not yet supported for vision-language models. Please set \"\n \"`assistant_only_loss=False` in the `SFTConfig`.\"\n )\n if self._is_vlm and args.max_length is not None and args.truncation_mode == \"keep_end\":\n raise ValueError(\n \"truncation_mode='keep_end' is not supported for vision-language models. Image tokens reside \"\n \"inside the prompt portion of the sequence; depending on the example, keep_end may silently \"\n \"drop them, causing pixel_values to be forwarded to the model with no corresponding visual \"\n \"tokens in input_ids. Use truncation_mode='keep_start' (the default) or set max_length=None.\"\n )\n\n # PEFT configuration and model wrapping\n if peft_config is not None:\n if added_tokens:\n # Ensure that the added tokens are trainable\n if peft_config.trainable_token_indices is None:\n peft_config.trainable_token_indices = {\"embed_tokens\": added_tokens}\n elif \"embed_tokens\" not in peft_config.trainable_token_indices:\n peft_config.trainable_token_indices[\"embed_tokens\"] = added_tokens\n else:\n peft_config.trainable_token_indices[\"embed_tokens\"].extend(added_tokens)\n\n # Ensure that the lm_head is trainable\n if peft_config.modules_to_save is None or \"lm_head\" not in peft_config.modules_to_save:\n logger.warning(\n \"Cloning chat template added new tokens to the tokenizer, but 'lm_head' is not in PEFT's \"\n \"`modules_to_save`. As a result, the model may not learn to generate outputs with these new \"\n \"tokens, leading to degraded generation quality. To fix this, add \"\n \"`modules_to_save=['lm_head']` to your PEFT configuration.\"\n )\n\n if peft_config.modules_to_save is None:\n peft_config.modules_to_save = [\"lm_head\"]\n else:\n peft_config.modules_to_save.append(\"lm_head\")\n\n if is_peft_available() and is_peft_model(model) and peft_config is not None:\n raise ValueError(\n \"You passed a `PeftModel` instance together with a `peft_config` to the trainer. Please first merge \"\n \"and unload the existing adapter, save the resulting base model, and then pass that base model along \"\n \"with the new `peft_config` to the trainer.\"\n )\n\n # Create PEFT model\n if peft_config is not None:\n model = get_peft_model(model, peft_config)\n\n # PEFT + DeepSpeed ZeRO-3 requires reentrant checkpointing. For more details, see\n # https://github.com/huggingface/trl/issues/2514#issuecomment-2692152703\n if (\n is_peft_model(model)\n and args.deepspeed_plugin is not None\n and args.deepspeed_plugin.zero_stage == 3\n and args.gradient_checkpointing\n ):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n use_reentrant = args.gradient_checkpointing_kwargs.get(\"use_reentrant\")\n if use_reentrant is False:\n logger.warning(\n \"You are using PEFT with DeepSpeed ZeRO-3 and gradient checkpointing with `use_reentrant=False`. \"\n \"`use_reentrant` is forced to `True` in this configuration to ensure correct training. To remove \"\n \"this warning, unset `use_reentrant` in `gradient_checkpointing_kwargs` or set it to `True`.\"\n )\n args.gradient_checkpointing_kwargs[\"use_reentrant\"] = True\n\n # When using gradient checkpointing with PEFT, we need to enable input gradients. transformers.Trainer normally\n # handles this, but a bug currently prevents it; see https://github.com/huggingface/transformers/issues/42489\n if is_peft_available() and is_peft_model(model) and args.gradient_checkpointing:\n model.enable_input_require_grads()\n\n # When using QLoRA, the PEFT adapter weights are converted to bf16 to follow the recommendations from the\n # original paper (see https://huggingface.co/papers/2305.14314, paragraph 3). Normally, this can be done by\n # passing `autocast_adapter_dtype=False` to `get_peft_model`, but this option is not yet supported for\n # quantized models. See: https://github.com/huggingface/peft/issues/2889\n # Non-quantized models do not have the `is_loaded_in_{8,4}bit` attributes, whereas quantized models do\n if getattr(model, \"is_loaded_in_4bit\", False) or getattr(model, \"is_loaded_in_8bit\", False):\n for param in model.parameters():\n if param.requires_grad:\n param.data = param.data.to(torch.bfloat16)\n\n # In Prompt Tuning a small set of trainable virtual tokens (continuous prompt embeddings) is prepended to the\n # input. We store the number of these tokens so we can account for them correctly when calculating accuracy.\n self.num_virtual_tokens = 0\n if is_peft_available() and is_peft_model(model):\n if model.active_adapter in model.peft_config:\n peft_model_config = model.peft_config[model.active_adapter]\n self.num_virtual_tokens = getattr(peft_model_config, \"num_virtual_tokens\", 0)\n\n # Data collator\n # BFD packing requires padding-free mode; otherwise, the collator outputs padded attention masks, causing\n # FlashAttention to ignore position_ids and recompute them incorrectly from the padded attention mask.\n self.padding_free = args.padding_free or (args.packing and args.packing_strategy in {\"bfd\", \"bfd_split\"})\n use_flash_attention = model.config._attn_implementation in FLASH_ATTENTION_VARIANTS\n if self.padding_free:\n if data_collator is not None:\n raise ValueError(\"Passing a custom data collator is not supported when using padding-free.\")\n if args.packing and args.packing_strategy == \"wrapped\":\n logger.warning(\n \"You are passing `padding_free=True` with the 'wrapped' packing strategy, which is not \"\n \"recommended. Please refer to the documentation to understand why this is not recommended.\"\n )\n if not use_flash_attention:\n logger.warning(\n \"Padding-free training is enabled, but the attention implementation is not set to a supported \"\n \"flash attention variant. Padding-free training flattens batches into a single sequence, and only \"\n \"the following implementations are known to reliably support this: \"\n f\"{', '.join(sorted(FLASH_ATTENTION_VARIANTS))}. Using other implementations may lead to \"\n \"unexpected behavior. To ensure compatibility, set `attn_implementation` in the model \"\n \"configuration to one of these supported options or verify that your attention mechanism can \"\n \"handle flattened sequences.\"\n )\n\n if args.per_device_train_batch_size == 1 and not args.packing:\n logger.warning(\n \"You are using a per_device_train_batch_size of 1 with padding-free training. Using a batch size \"\n \"of 1 annihilate the benefits of padding-free training. Please consider increasing the batch size \"\n \"to at least 2.\"\n )\n\n # Decide whether to use completion-only loss: if not specified, then it is set to True if the dataset format\n # is prompt-completion, and False if the dataset format is language modeling.\n dataset_sample = next(iter(train_dataset))\n if args.completion_only_loss is None:\n self.completion_only_loss = \"prompt\" in dataset_sample and \"completion\" in dataset_sample\n else:\n self.completion_only_loss = args.completion_only_loss\n\n self._is_vision_dataset = \"image\" in dataset_sample or \"images\" in dataset_sample\n if self._is_vision_dataset and not self._is_vlm:\n raise ValueError(\n \"The dataset appears to be vision-related (contains 'image' or 'images' keys), but the provided \"\n \"model does not seem to be a vision-language model. Please check your model and dataset.\"\n )\n\n if data_collator is None and not self._is_vision_dataset:\n # Get the pad token: if not provided, use the one from the processing class or the eos token\n # if the processing class does not have a pad token.\n pad_token = args.pad_token or tokenizer.pad_token or tokenizer.eos_token\n pad_token_id = tokenizer.convert_tokens_to_ids(pad_token)\n if pad_token_id is None:\n raise ValueError(\n f\"The specified `pad_token` ('{pad_token}') is not found in the vocabulary of the given \"\n f\"`processing_class` ({processing_class.__class__.__name__}). Ensure that the `pad_token` exists \"\n \"in the vocabulary before using it as a padding token.\"\n )\n data_collator = DataCollatorForLanguageModeling(\n pad_token_id=pad_token_id,\n completion_only_loss=self.completion_only_loss,\n padding_free=self.padding_free,\n pad_to_multiple_of=args.pad_to_multiple_of,\n )\n elif data_collator is None and self._is_vision_dataset:\n data_collator = DataCollatorForVisionLanguageModeling(\n processor=processing_class,\n max_length=args.max_length,\n completion_only_loss=self.completion_only_loss,\n pad_to_multiple_of=args.pad_to_multiple_of,\n dataset_text_field=args.dataset_text_field,\n )\n\n if args.packing and args.packing_strategy in {\"bfd\", \"bfd_split\"} and not use_flash_attention:\n logger.warning(\n \"You are using packing, but the attention implementation is not set to a supported flash attention \"\n \"variant. Packing gathers multiple samples into a single sequence, and only the following \"\n f\"implementations are known to reliably support this: {', '.join(sorted(FLASH_ATTENTION_VARIANTS))}. \"\n \"Using other implementations may lead to cross-contamination between samples. To avoid this, either \"\n \"disable packing by setting `packing=False`, or set `attn_implementation` in the model configuration \"\n \"to one of these supported options.\"\n )\n if args.assistant_only_loss and not is_conversational(dataset_sample):\n raise ValueError(\n \"You set `assistant_only_loss=True`, but the dataset is not conversational. This option is only \"\n \"supported for conversational datasets.\"\n )\n\n # Dataset\n # Skip dataset preparation if `skip_prepare_dataset=True` in `dataset_kwargs`, or if it's a VLM, where\n # preprocessing (e.g., image-to-pixel conversion) is too costly and done on the fly instead.\n skip_prepare_dataset = (\n args.dataset_kwargs is not None\n and args.dataset_kwargs.get(\"skip_prepare_dataset\", False)\n or self._is_vision_dataset\n )\n if not skip_prepare_dataset:\n if self.completion_only_loss and formatting_func:\n raise ValueError(\n \"A formatting function was provided while `completion_only_loss=True`, which is incompatible. \"\n \"Using a formatter converts the dataset to a language modeling type, conflicting with \"\n \"completion-only loss. To resolve this, apply your formatting function before passing the \"\n \"dataset, or disable `completion_only_loss` in `SFTConfig`.\"\n )\n train_dataset = self._prepare_dataset(\n train_dataset, processing_class, args, args.packing, formatting_func, \"train\"\n )\n if eval_dataset is not None:\n packing = args.packing if args.eval_packing is None else args.eval_packing\n if isinstance(eval_dataset, dict):\n eval_dataset = {\n key: self._prepare_dataset(dataset, processing_class, args, packing, formatting_func, key)\n for key, dataset in eval_dataset.items()\n }\n else:\n eval_dataset = self._prepare_dataset(\n eval_dataset, processing_class, args, packing, formatting_func, \"eval\"\n )\n\n # Loss function\n if not args.use_liger_kernel: # liger supports dft loss by just passing use_token_scaling=True\n if args.loss_type == \"nll\":\n pass # use the default loss\n elif args.loss_type == \"dft\":\n if compute_loss_func is not None:\n raise ValueError(\n \"You passed a `compute_loss_func` together with `loss_type='dft'` to the `SFTTrainer`. \"\n \"When using `loss_type='dft'`, the loss function is internally set to the DFT loss, so \"\n \"passing a `compute_loss_func` is not allowed.\"\n )\n compute_loss_func = dft_loss\n else:\n raise ValueError(f\"Invalid `loss_type` {args.loss_type} passed. Supported values are 'nll' and 'dft'.\")\n\n # Transformers explicitly set use_reentrant=True in the past to silence a PyTorch warning, but the default was\n # never updated once PyTorch switched to recommending use_reentrant=False. Until that change lands upstream\n # (see https://github.com/huggingface/transformers/pull/43203) and is released (most likely in 5.0.0), we\n # default to the recommended non-reentrant behavior here, while preserving any user-provided value.\n if args.gradient_checkpointing and Version(transformers.__version__) < Version(\"5.0.0\"):\n args.gradient_checkpointing_kwargs = args.gradient_checkpointing_kwargs or {}\n args.gradient_checkpointing_kwargs.setdefault(\"use_reentrant\", False)\n\n super().__init__(\n model=model,\n args=args,\n data_collator=data_collator,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n processing_class=processing_class,\n compute_loss_func=compute_loss_func,\n compute_metrics=compute_metrics,\n callbacks=callbacks,\n optimizers=optimizers,\n optimizer_cls_and_kwargs=optimizer_cls_and_kwargs,\n preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n )\n\n # Initialize activation offloading context\n if self.args.activation_offloading:\n self.maybe_activation_offload_context = get_act_offloading_ctx_manager(model=self.model)\n else:\n self.maybe_activation_offload_context = contextlib.nullcontext()\n\n self.aux_loss_enabled = getattr(model.config, \"output_router_logits\", False)\n\n # Initialize the metrics\n self._metrics = {\"train\": defaultdict(list), \"eval\": defaultdict(list)}\n self._total_train_tokens = 0\n\n # Add tags to the model\n self.model.add_model_tags(self._tag_names)\n\n def _prepare_dataset(\n self,\n dataset: Dataset | IterableDataset,\n processing_class: PreTrainedTokenizerBase | ProcessorMixin,\n args: SFTConfig,\n packing: bool,\n formatting_func: Callable[[dict], str] | None,\n dataset_name: str,\n ) -> Dataset | IterableDataset:\n # Tabular backends like Arrow/Parquet insert `None` for mismatched keys in nested structures. Clean them from\n # sampled data.\n if isinstance(dataset, Dataset): # IterableDataset does not support `with_transform`\n dataset = dataset.with_transform(remove_none_values)\n\n # If the dataset is already preprocessed (tokenized), skip the processing steps.\n column_names = get_dataset_column_names(dataset)\n is_processed = \"input_ids\" in column_names\n\n # Build the kwargs for the `map` function\n map_kwargs = {}\n if isinstance(dataset, Dataset): # IterableDataset does not support num_proc\n map_kwargs[\"num_proc\"] = args.dataset_num_proc\n\n with PartialState().main_process_first():\n # Apply the formatting function if any\n if formatting_func is not None and is_processed:\n logger.warning(\n \"You passed a dataset that is already processed (contains an `input_ids` field) together with a \"\n \"formatting function. Therefore `formatting_func` will be ignored. Either remove the \"\n \"`formatting_func` or pass a dataset that is not already processed.\",\n )\n\n if formatting_func is not None and not is_processed:\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Applying formatting function to {dataset_name} dataset\"\n\n def _func(example):\n return {\"text\": formatting_func(example)}\n\n dataset = dataset.map(_func, batched=False, **map_kwargs)\n\n if not is_processed:\n # Convert the dataset to ChatML if needed\n first_example = next(iter(dataset))\n if is_conversational_from_value(first_example):\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Converting {dataset_name} dataset to ChatML\"\n column_names = get_dataset_column_names(dataset)\n dataset = dataset.map(\n maybe_convert_to_chatml,\n remove_columns=\"conversations\" if \"conversations\" in column_names else None,\n **map_kwargs,\n )\n\n # Apply the chat template if needed\n first_example = next(iter(dataset))\n if not is_conversational(first_example):\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Adding EOS to {dataset_name} dataset\"\n\n def add_eos(example, eos_token):\n if \"text\" in example and not example[\"text\"].endswith(eos_token): # language modeling case\n example[\"text\"] = example[\"text\"] + eos_token\n elif \"completion\" in example and not example[\"completion\"].endswith(eos_token):\n example[\"completion\"] = example[\"completion\"] + eos_token\n return example\n\n eos_token = processing_class.tokenizer.eos_token if self._is_vlm else processing_class.eos_token\n dataset = dataset.map(\n add_eos,\n fn_kwargs={\"eos_token\": eos_token},\n remove_columns=\"messages\" if \"messages\" in column_names else None, # renamed to \"text\"\n **map_kwargs,\n )\n\n # Tokenize the dataset\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Tokenizing {dataset_name} dataset\"\n\n def tokenize_fn(example, processing_class, dataset_text_field, assistant_only_loss):\n tools = example.get(\"tools\")\n tools = json.loads(tools) if isinstance(tools, str) else tools\n if \"prompt\" in example: # prompt-completion case\n output = {}\n if is_conversational(example):\n if self._is_vlm:\n prompt = prepare_multimodal_messages(example[\"prompt\"], images=[])\n completion = prepare_multimodal_messages(example[\"completion\"], images=[])\n else:\n prompt = example[\"prompt\"]\n completion = example[\"completion\"]\n prompt_ids = processing_class.apply_chat_template(\n prompt,\n tools=tools,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=False,\n **example.get(\"chat_template_kwargs\", {}),\n )\n # Fix transformers inconsistency: for VLMs, apply_chat_template returns lists of lists\n # even for single examples, while for LLMs it returns lists of ints.\n prompt_ids = prompt_ids[0] if isinstance(prompt_ids[0], list) else prompt_ids\n prompt_completion_processed = processing_class.apply_chat_template(\n prompt + completion,\n tools=tools,\n tokenize=True,\n return_dict=True,\n return_assistant_tokens_mask=assistant_only_loss,\n **example.get(\"chat_template_kwargs\", {}),\n )\n # Fix transformers inconsistency: for VLMs, apply_chat_template returns lists of lists\n # even for single examples, while for LLMs it returns lists of ints.\n prompt_completion_processed = {\n k: v[0] if isinstance(v[0], list) else v\n for k, v in prompt_completion_processed.items()\n }\n prompt_completion_ids = prompt_completion_processed[\"input_ids\"]\n if \"assistant_masks\" in prompt_completion_processed:\n output[\"assistant_masks\"] = prompt_completion_processed[\"assistant_masks\"]\n else:\n prompt_ids = processing_class(text=example[\"prompt\"])[\"input_ids\"]\n prompt_completion_ids = processing_class(text=example[\"prompt\"] + example[\"completion\"])[\n \"input_ids\"\n ]\n # Fix transformers inconsistency: for VLMs, processing_class returns lists of lists\n # even for single examples, while for LLMs it returns lists of ints.\n prompt_ids = prompt_ids[0] if isinstance(prompt_ids[0], list) else prompt_ids\n prompt_completion_ids = (\n prompt_completion_ids[0]\n if isinstance(prompt_completion_ids[0], list)\n else prompt_completion_ids\n )\n\n # Check if the tokenized prompt starts with the tokenized prompt+completion\n if not prompt_completion_ids[: len(prompt_ids)] == prompt_ids:\n logger.warning(\n \"Mismatch between tokenized prompt and the start of tokenized prompt+completion. \"\n \"This may be due to unexpected tokenizer behavior, whitespace issues, or special \"\n \"token handling. Verify that the tokenizer is processing text consistently.\"\n )\n\n # Create completion mask\n completion_mask = [0] * len(prompt_ids) + [1] * (len(prompt_completion_ids) - len(prompt_ids))\n output[\"input_ids\"] = prompt_completion_ids\n output[\"completion_mask\"] = completion_mask\n\n else: # language modeling case\n if is_conversational(example):\n if self._is_vlm:\n messages = prepare_multimodal_messages(example[\"messages\"], images=[])\n else:\n messages = example[\"messages\"]\n processed = processing_class.apply_chat_template(\n messages,\n tools=tools,\n tokenize=True,\n return_dict=True,\n return_assistant_tokens_mask=assistant_only_loss,\n **example.get(\"chat_template_kwargs\", {}),\n )\n # Fix transformers inconsistency: for VLMs, apply_chat_template returns lists of lists\n # even for single examples, while for LLMs it returns lists of ints.\n processed = {k: v[0] if isinstance(v[0], list) else v for k, v in processed.items()}\n output = {k: processed[k] for k in (\"input_ids\", \"assistant_masks\") if k in processed}\n else:\n output = {\"input_ids\": processing_class(text=example[dataset_text_field])[\"input_ids\"]}\n\n if \"assistant_masks\" in output and 1 not in output[\"assistant_masks\"]:\n raise RuntimeError(\n \"You're using `assistant_only_loss=True`, but at least one example has no assistant \"\n \"tokens. This usually means the tokenizer's chat template doesn't generate assistant \"\n \"masks โ€” it may be missing the `{% generation %}` keyword. Please check the template and \"\n \"ensure it's correctly configured to support assistant masking.\"\n )\n return output\n\n dataset = dataset.map(\n tokenize_fn,\n fn_kwargs={\n \"processing_class\": processing_class,\n \"dataset_text_field\": args.dataset_text_field,\n \"assistant_only_loss\": args.assistant_only_loss,\n },\n **map_kwargs,\n )\n\n # Pack or truncate\n if packing:\n if args.max_length is None:\n raise ValueError(\"When packing is enabled, `max_length` can't be `None`.\")\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Packing {dataset_name} dataset\"\n\n columns = [\"input_ids\"]\n if \"completion_mask\" in get_dataset_column_names(dataset):\n columns.append(\"completion_mask\")\n if \"assistant_masks\" in get_dataset_column_names(dataset):\n columns.append(\"assistant_masks\")\n\n dataset = dataset.select_columns(columns)\n\n # Shuffle the dataset before packing. When using wrapped packing, it's important to shuffle before\n # packing as well to avoid correlations between sequences packed together.\n if args.shuffle_dataset:\n dataset = dataset.shuffle(seed=args.seed)\n\n # Packing adds new column \"seq_lengths\" needed for document aware FlashAttention\n dataset = pack_dataset(dataset, args.max_length, args.packing_strategy, map_kwargs)\n elif args.max_length is not None:\n if isinstance(dataset, Dataset): # `IterableDataset.map` does not support `desc`\n map_kwargs[\"desc\"] = f\"Truncating {dataset_name} dataset\"\n dataset = truncate_dataset(\n dataset, args.max_length, truncation_mode=args.truncation_mode, map_kwargs=map_kwargs\n )\n # For Liger kernel, ensure only the essential columns\n if args.use_liger_kernel:\n collator_expected_keys = {\"input_ids\", \"seq_lengths\", \"completion_mask\", \"assistant_masks\"}\n column_names = get_dataset_column_names(dataset)\n dataset = dataset.select_columns(collator_expected_keys.intersection(column_names))\n\n if args.shuffle_dataset:\n dataset = dataset.shuffle(seed=args.seed)\n\n return dataset\n\n def _set_signature_columns_if_needed(self):\n # If `self.args.remove_unused_columns` is True, non-signature columns are removed.\n # By default, this method sets `self._signature_columns` to the model's expected inputs (usually, \"input_ids\"\n # and \"attention_mask\"). When using `train_on_completion_only` we add a \"completion_mask\" column to the\n # dataset. So we need to override the default signature columns to include \"completion_mask\" as well.\n if self._signature_columns is None:\n if self._is_vision_dataset:\n self._signature_columns = [\"messages\", \"prompt\", \"completion\", \"image\", \"images\"]\n else:\n self._signature_columns = [\"input_ids\", \"labels\", \"seq_lengths\", \"completion_mask\", \"assistant_masks\"]\n\n def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):\n mode = \"train\" if self.model.training else \"eval\"\n prediction_loss_only = inputs.pop(\"_prediction_loss_only\", None)\n\n # Set aside labels as it will be dropped by super().compute_loss() if a custom `compute_loss_func` is used.\n # This can be removed when this issue is fixed.\n # When using CP or SP, labels are pre-shifted, we must use shift_labels instead.\n labels = inputs[\"labels\"] if \"shift_labels\" not in inputs else None\n\n # If not set, defaults from model config and may warn since cache isn't compatible with gradient checkpointing\n inputs[\"use_cache\"] = False\n\n # Request token accuracy from Liger kernel and set token scaling if using DFT loss\n if self.args.use_liger_kernel:\n # Avoid materializing full logits during eval unless explicitly needed.\n # By default, liger kernel only skips logits during training (self.training=True).\n # When only loss is needed for eval (no compute_metrics), we can safely skip logits.\n # prediction_step communicates whether logits are expected via `_prediction_loss_only`;\n # this prevents skipping logits during `predict()` where outputs are requested.\n # Keep logits when preprocess_logits_for_metrics is set, even if compute_metrics is None.\n # to prevent massive vRAM spikes from the lm_head projection.\n # See: https://github.com/huggingface/trl/issues/4679\n inputs[\"skip_logits\"] = (\n self.model.training\n or self.args.prediction_loss_only\n or (\n self.compute_metrics is None\n and self.preprocess_logits_for_metrics is None\n and prediction_loss_only is not False\n )\n )\n inputs[\"return_token_accuracy\"] = True\n inputs[\"use_token_scaling\"] = self.args.loss_type == \"dft\"\n\n (loss, outputs) = super().compute_loss(\n model, inputs, return_outputs=True, num_items_in_batch=num_items_in_batch\n )\n\n # Compute entropy\n if not self.args.use_liger_kernel: # liger doesn't return logits\n with torch.no_grad():\n per_token_entropy = entropy_from_logits(outputs.logits)\n # When using Prompt Tuning, skip the virtual tokens in logits before entropy computation, since they\n # do not correspond to actual input tokens.\n if (\n self.num_virtual_tokens > 0\n and model.peft_config[model.active_adapter].peft_type != PeftType.PREFIX_TUNING\n ):\n per_token_entropy = per_token_entropy[:, self.num_virtual_tokens :]\n if \"attention_mask\" in inputs:\n attention_mask = inputs[\"attention_mask\"]\n entropy = torch.sum(per_token_entropy * attention_mask) / attention_mask.sum()\n elif \"position_ids\" in inputs:\n entropy = torch.mean(per_token_entropy)\n else:\n raise ValueError(\"Expected 'attention_mask' or 'position_ids' in inputs.\")\n entropy = self.accelerator.gather_for_metrics(entropy).mean().item()\n self._metrics[mode][\"entropy\"].append(entropy)\n\n if mode == \"train\":\n # When using padding-free, the attention_mask is not present in the inputs, instead we have cu_seq_lens_q,\n # cu_seq_lens_k, and max_length_k, max_length_q and position_ids.\n if \"attention_mask\" in inputs:\n num_tokens_in_batch = self.accelerator.gather_for_metrics(inputs[\"attention_mask\"].sum()).sum().item()\n elif \"position_ids\" in inputs:\n local_num_tokens = torch.tensor(inputs[\"position_ids\"].size(1), device=inputs[\"position_ids\"].device)\n num_tokens_in_batch = self.accelerator.gather_for_metrics(local_num_tokens).sum().item()\n else:\n raise ValueError(\"Expected 'attention_mask' or 'position_ids' in inputs.\")\n self._total_train_tokens += num_tokens_in_batch\n self._metrics[mode][\"num_tokens\"] = [self._total_train_tokens]\n\n if self.args.use_liger_kernel:\n if hasattr(outputs, \"token_accuracy\") and outputs.token_accuracy is not None:\n token_accuracy = self.accelerator.gather_for_metrics(outputs.token_accuracy).mean().item()\n self._metrics[mode][\"mean_token_accuracy\"].append(token_accuracy)\n else:\n warnings.warn(\n \"liger-kernel did not return token_accuracy when requested. The mean_token_accuracy metric will \"\n \"not be logged. This is unexpected; please report it to the liger-kernel repository.\",\n stacklevel=2,\n )\n else:\n # Compute accuracy from logits using argmax (traditional method)\n with torch.no_grad():\n if \"shift_labels\" in inputs:\n # When using CP or SP, labels are pre-shifted. We must use these (and cannot manually shift) because:\n # - The first discarded token from inputs[\"labels\"] actually belongs to process n-1\n # - The last logits require the label from process n+1\n shift_logits = outputs.logits.contiguous()\n shift_labels = inputs[\"shift_labels\"]\n else:\n shift_logits = outputs.logits[..., :-1, :].contiguous()\n shift_labels = labels[..., 1:].contiguous()\n\n # Prompt Tuning and P-Tuning output logits for virtual tokens but Prefix-Tuning does not.\n if (\n self.num_virtual_tokens > 0\n and model.peft_config[model.active_adapter].peft_type != PeftType.PREFIX_TUNING\n ):\n shift_logits = shift_logits[:, self.num_virtual_tokens :, :]\n\n # Get predictions\n predictions = shift_logits.argmax(dim=-1)\n\n # Create mask for non-padding tokens (assuming ignore_index is -100)\n mask = shift_labels != -100\n\n # Calculate accuracy only on non-padding tokens\n correct_predictions = (predictions == shift_labels) & mask\n total_tokens = mask.sum()\n correct_tokens = correct_predictions.sum()\n\n # Gather the correct_tokens and total_tokens across all processes\n correct_tokens = self.accelerator.gather_for_metrics(correct_tokens)\n total_tokens = self.accelerator.gather_for_metrics(total_tokens)\n\n # Compute the mean token accuracy and log it\n total_sum = total_tokens.sum()\n accuracy = (correct_tokens.sum() / total_sum).item() if total_sum > 0 else 0.0\n self._metrics[mode][\"mean_token_accuracy\"].append(accuracy)\n\n # Log auxiliary loss if enabled (applies to both Liger and non-Liger)\n if self.aux_loss_enabled:\n aux_loss = outputs.aux_loss\n aux_loss = self.accelerator.gather_for_metrics(aux_loss).mean().item()\n self._metrics[mode][\"aux_loss\"].append(aux_loss)\n\n return (loss, outputs) if return_outputs else loss\n\n def prediction_step(self, model, inputs, prediction_loss_only, ignore_keys=None):\n # Preserve the eval loop intent so compute_loss can decide whether logits are needed.\n inputs[\"_prediction_loss_only\"] = prediction_loss_only\n return super().prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)\n\n # Override training step to add activation offloading context.\n def training_step(self, *args, **kwargs):\n with self.maybe_activation_offload_context:\n return super().training_step(*args, **kwargs)\n\n def log(self, logs: dict[str, float], start_time: float | None = None) -> None:\n mode = \"train\" if self.model.training else \"eval\"\n metrics = {key: sum(val) / len(val) for key, val in self._metrics[mode].items()} # average the metrics\n\n # This method can be called both in training and evaluation. When called in evaluation, the keys in `logs`\n # start with \"eval_\". We need to add the prefix \"eval_\" to the keys in `metrics` to match the format.\n if mode == \"eval\":\n metrics = {f\"eval_{key}\": val for key, val in metrics.items()}\n\n logs = {**logs, **metrics}\n super().log(logs, start_time)\n self._metrics[mode].clear()\n\n # Ensure the model card is saved along with the checkpoint\n def _save_checkpoint(self, model, trial):\n if self.args.hub_model_id is None:\n model_name = Path(self.args.output_dir).name\n else:\n model_name = self.args.hub_model_id.split(\"/\")[-1]\n self.create_model_card(model_name=model_name)\n super()._save_checkpoint(model, trial)\n" }, { "path": "trl/trainer/utils.py", "content": "# Copyright 2020-2026 The HuggingFace Team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport asyncio\nimport hashlib\nimport importlib.resources as pkg_resources\nimport os\nimport random\nimport socket\nimport threading\nfrom collections.abc import Mapping, Sequence, Sized\nfrom contextlib import contextmanager\nfrom dataclasses import dataclass\nfrom importlib.metadata import version\nfrom itertools import accumulate\nfrom typing import TypeVar\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn.functional as F\nimport transformers\nfrom accelerate import PartialState, logging\nfrom accelerate.state import AcceleratorState\nfrom huggingface_hub import ModelCard, ModelCardData\nfrom torch.utils.data import Sampler\nfrom transformers import (\n AutoConfig,\n BitsAndBytesConfig,\n PretrainedConfig,\n PreTrainedModel,\n is_comet_available,\n is_trackio_available,\n)\nfrom transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast\nfrom transformers.models.auto.auto_factory import _BaseAutoModelClass\nfrom transformers.utils import (\n is_peft_available,\n is_rich_available,\n is_torch_xpu_available,\n)\n\nfrom ..trainer.model_config import ModelConfig\n\n\nif is_rich_available():\n from rich.console import Console\n from rich.panel import Panel\n from rich.table import Table\n from rich.text import Text\n\nif is_comet_available():\n import comet_ml\n\nif is_peft_available():\n from peft import LoraConfig, PeftConfig, PeftModel\n\n\nlogger = logging.get_logger(__name__)\n\n\ndef _is_port_free(port: int, host: str = \"127.0.0.1\") -> bool:\n try:\n with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:\n s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n s.bind((host, port))\n return True\n except OSError:\n return False\n\n\ndef _find_free_port() -> int:\n candidates = (29500, 23456, 12355, 12345)\n for p in candidates:\n if _is_port_free(p):\n return p\n with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:\n s.bind((\"\", 0))\n return s.getsockname()[1]\n\n\ndef ensure_master_addr_port(addr: str | None = None, port: int | None = None) -> None:\n \"\"\"\n Ensure `MASTER_ADDR`/`MASTER_PORT` are set safely.\n\n - Respects existing environment variables.\n - Defaults `MASTER_ADDR` to localhost if unset.\n - Chooses a free TCP port if `MASTER_PORT` is unset to avoid collisions.\n - If `MASTER_PORT` is set to `\"0\"` or `\"auto\"`, it is resolved to a free port.\n \"\"\"\n os.environ[\"MASTER_ADDR\"] = os.environ.get(\"MASTER_ADDR\") or addr or \"localhost\"\n\n env_port = os.environ.get(\"MASTER_PORT\", \"\").strip().lower()\n if port is None and env_port not in {\"\", \"0\", \"auto\"}:\n try:\n port = int(env_port)\n except ValueError:\n pass\n\n os.environ[\"MASTER_PORT\"] = str(_find_free_port() if port in (None, 0) else port)\n\n\ndef pad(\n tensors: list[torch.Tensor],\n padding_value: int = 0,\n padding_side: str = \"right\",\n pad_to_multiple_of: int | None = None,\n) -> torch.Tensor:\n \"\"\"\n Pads a list of tensors to the same shape along the first dimension.\n\n Args:\n tensors (`list[torch.Tensor]`):\n List of input tensors to pad.\n padding_value (`int`):\n Value to use for padding. Default is 0.\n padding_side (`str`):\n Side on which to add padding. Must be 'left' or 'right'. Default is 'right'.\n pad_to_multiple_of (`int`, *optional*):\n If set will pad the sequence to a multiple of the provided value.\n\n Returns:\n `torch.Tensor`:\n A single tensor containing the padded tensors.\n\n Examples:\n ```python\n >>> import torch\n\n >>> pad([torch.tensor([1, 2, 3]), torch.tensor([4, 5])])\n tensor([[1, 2, 3],\n [4, 5, 0]])\n\n >>> pad([torch.tensor([[1, 2], [3, 4]]), torch.tensor([[5, 6]])])\n tensor([[[1, 2],\n [3, 4]],\n [[5, 6],\n [0, 0]]])\n ```\n \"\"\"\n # Determine the maximum shape for each dimension\n output_shape = np.max([t.shape for t in tensors], 0).tolist()\n\n # Apply pad_to_multiple_of to the first (sequence) dimension\n if pad_to_multiple_of is not None:\n remainder = output_shape[0] % pad_to_multiple_of\n if remainder != 0:\n output_shape[0] += pad_to_multiple_of - remainder\n\n # Create an output tensor filled with the padding value\n output = torch.full((len(tensors), *output_shape), padding_value, dtype=tensors[0].dtype, device=tensors[0].device)\n\n for i, t in enumerate(tensors):\n if padding_side == \"left\":\n seq_start = output_shape[0] - t.shape[0]\n elif padding_side == \"right\":\n seq_start = 0\n else:\n raise ValueError(\"padding_side must be 'left' or 'right'\")\n\n # Define the slices\n seq_slice = slice(seq_start, seq_start + t.shape[0])\n slices = (seq_slice,) + tuple(slice(0, s) for s in t.shape[1:])\n output[i][slices] = t\n\n return output\n\n\ndef disable_dropout_in_model(model: torch.nn.Module) -> None:\n for module in model.modules():\n if isinstance(module, torch.nn.Dropout):\n module.p = 0\n\n\ndef get_quantization_config(model_args: ModelConfig) -> BitsAndBytesConfig | None:\n if model_args.load_in_4bit:\n quantization_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_compute_dtype=model_args.dtype, # For consistency with model weights, we use the same value as `dtype`\n bnb_4bit_quant_type=model_args.bnb_4bit_quant_type,\n bnb_4bit_use_double_quant=model_args.use_bnb_nested_quant,\n bnb_4bit_quant_storage=model_args.bnb_4bit_quant_storage,\n )\n elif model_args.load_in_8bit:\n quantization_config = BitsAndBytesConfig(\n load_in_8bit=True,\n )\n else:\n quantization_config = None\n\n return quantization_config\n\n\ndef get_kbit_device_map() -> dict[str, int] | None:\n if torch.cuda.is_available() or is_torch_xpu_available():\n return {\"\": PartialState().local_process_index}\n else:\n return None\n\n\ndef get_peft_config(model_args: ModelConfig) -> \"PeftConfig | None\":\n if model_args.use_peft is False:\n return None\n\n if not is_peft_available():\n raise ValueError(\n \"You need to have PEFT library installed in your environment, make sure to install `peft`. \"\n \"Make sure to run `pip install -U peft`.\"\n )\n\n peft_config = LoraConfig(\n task_type=model_args.lora_task_type,\n r=model_args.lora_r,\n target_modules=model_args.lora_target_modules,\n target_parameters=model_args.lora_target_parameters,\n lora_alpha=model_args.lora_alpha,\n lora_dropout=model_args.lora_dropout,\n bias=\"none\",\n use_rslora=model_args.use_rslora,\n use_dora=model_args.use_dora,\n modules_to_save=model_args.lora_modules_to_save,\n )\n\n return peft_config\n\n\ndef prepare_deepspeed(\n model: torch.nn.Module, per_device_train_batch_size: int, fp16: bool = False, bf16: bool = False\n) -> torch.nn.Module:\n \"\"\"\n Prepares the model for training with DeepSpeed (both for stage 2 and 3), configuring the appropriate settings based\n on the model and batch size.\n\n Args:\n model (`torch.nn.Module`):\n The model to be prepared for DeepSpeed training.\n per_device_train_batch_size (`int`):\n The training batch size per device.\n fp16 (`bool`, defaults to `False`):\n Whether to use FP16 precision.\n bf16 (`bool`, defaults to `False`):\n Whether to use BF16 precision.\n\n Returns:\n `torch.nn.Module`:\n The model initialized and configured with DeepSpeed for training.\n \"\"\"\n import deepspeed\n\n deepspeed_plugin = AcceleratorState().deepspeed_plugin\n config_kwargs = deepspeed_plugin.deepspeed_config\n if config_kwargs[\"zero_optimization\"][\"stage\"] != 3:\n config_kwargs[\"train_micro_batch_size_per_gpu\"] = per_device_train_batch_size\n config_kwargs = {\n \"train_micro_batch_size_per_gpu\": config_kwargs[\"train_micro_batch_size_per_gpu\"],\n \"prescale_gradients\": False,\n \"wall_clock_breakdown\": False,\n }\n if bf16:\n config_kwargs[\"bf16\"] = {\"enabled\": True}\n elif fp16:\n config_kwargs[\"fp16\"] = {\"enabled\": True}\n else:\n if hasattr(model, \"config\"):\n hidden_size = (\n max(model.config.hidden_sizes)\n if getattr(model.config, \"hidden_sizes\", None)\n else getattr(model.config, \"hidden_size\", None)\n )\n if hidden_size is not None and config_kwargs[\"zero_optimization\"][\"stage\"] == 3:\n # Note that `stage3_prefetch_bucket_size` can produce DeepSpeed messages like: `Invalidate trace cache @ step 0: expected module 1, but got module 0`\n # This is expected and is not an error, see: https://github.com/microsoft/DeepSpeed/discussions/4081\n config_kwargs.update(\n {\n \"zero_optimization.reduce_bucket_size\": hidden_size * hidden_size,\n \"zero_optimization.stage3_param_persistence_threshold\": 10 * hidden_size,\n \"zero_optimization.stage3_prefetch_bucket_size\": 0,\n }\n )\n model, *_ = deepspeed.initialize(model=model, config=config_kwargs)\n model.eval()\n return model\n\n\ndef generate_model_card(\n base_model: str | None,\n model_name: str,\n hub_model_id: str,\n dataset_name: str | None,\n tags: list[str],\n wandb_url: str | None,\n trackio_url: str | None,\n trainer_name: str,\n trainer_citation: str | None = None,\n template_file: str | None = None,\n paper_title: str | None = None,\n paper_id: str | None = None,\n comet_url: str | None = None,\n) -> ModelCard:\n \"\"\"\n Generate a [`~huggingface_hub.ModelCard`] from a template.\n\n Args:\n base_model (`str` or `None`):\n Base model name.\n model_name (`str`):\n Model name.\n hub_model_id (`str`):\n Hub model ID as `username/model_id`.\n dataset_name (`str` or `None`):\n Dataset name.\n tags (`list[str]`):\n Tags.\n wandb_url (`str` or `None`):\n Weights & Biases run URL.\n trackio_url (`str` or `None`):\n Trackio Space URL.\n comet_url (`str` or `None`):\n Comet experiment URL.\n trainer_name (`str`):\n Trainer name.\n trainer_citation (`str` or `None`, defaults to `None`):\n Trainer citation as a BibTeX entry.\n template_file (`str` *optional*):\n Template file name located in the `trl/templates` directory. Defaults to `lm_model_card.md`.\n paper_title (`str` or `None`, defaults to `None`):\n Paper title.\n paper_id (`str` or `None`, defaults to `None`):\n ArXiv paper ID as `YYMM.NNNNN`.\n\n Returns:\n [`~huggingface_hub.ModelCard`]:\n A ModelCard object.\n \"\"\"\n card_data = ModelCardData(\n base_model=base_model,\n datasets=dataset_name,\n library_name=\"transformers\",\n licence=\"license\",\n model_name=model_name,\n tags=[\"generated_from_trainer\", *tags],\n )\n template_file = template_file or \"lm_model_card.md\"\n card = ModelCard.from_template(\n card_data,\n template_path=str(pkg_resources.files(\"trl\").joinpath(f\"templates/{template_file}\")),\n base_model=base_model,\n model_name=model_name,\n hub_model_id=hub_model_id,\n dataset_name=dataset_name,\n wandb_url=wandb_url,\n trackio_url=trackio_url,\n comet_url=comet_url,\n trainer_name=trainer_name,\n trainer_citation=trainer_citation,\n paper_title=paper_title,\n paper_id=paper_id,\n trl_version=version(\"trl\"),\n transformers_version=version(\"transformers\"),\n pytorch_version=version(\"torch\"),\n datasets_version=version(\"datasets\"),\n tokenizers_version=version(\"tokenizers\"),\n )\n return card\n\n\ndef get_comet_experiment_url() -> str | None:\n \"\"\"\n If Comet integration is enabled, return the URL of the current Comet experiment; otherwise, return `None`.\n \"\"\"\n if not is_comet_available():\n return None\n\n if comet_ml.get_running_experiment() is not None:\n return comet_ml.get_running_experiment().url\n\n return None\n\n\ndef get_trackio_space_url() -> str | None:\n \"\"\"\n If Trackio integration is enabled, return the URL of the current Trackio Space; otherwise, return `None`.\n \"\"\"\n if not is_trackio_available():\n return None\n\n from trackio import context_vars\n\n run = context_vars.current_run.get()\n if run is None:\n return None\n space_id = run._space_id\n if space_id is None:\n return None\n space_id = space_id.replace(\"/\", \"-\")\n project = run.project\n name = run.name\n return f\"https://{space_id}.hf.space?project={project}&runs={name}&sidebar=collapsed\"\n\n\ndef log_table_to_comet_experiment(name: str, table: pd.DataFrame) -> None:\n \"\"\"\n If Comet integration is enabled logs a table to the Comet experiment if it is currently running.\n\n Args:\n name (`str`):\n Table name.\n table (`pandas.DataFrame`):\n The Pandas DataFrame containing the table to log.\n \"\"\"\n if not is_comet_available():\n raise ModuleNotFoundError(\"The comet-ml is not installed. Please install it first: pip install comet-ml\")\n\n experiment = comet_ml.get_running_experiment()\n if experiment is not None:\n experiment.log_table(tabular_data=table, filename=name)\n\n\ndef flush_left(mask: torch.Tensor, *tensors: torch.Tensor) -> torch.Tensor | tuple[torch.Tensor, ...]:\n \"\"\"\n Shift non-zero elements in the mask and corresponding tensors to the left.\n\n This function operates on a binary mask and any number of additional tensors with the same dimensions as the mask.\n For each row, non-zero values are shifted to the leftmost positions. Then, columns that contain only zeros across\n all rows are truncated from the mask and tensors. Visually, this operation can be represented as follows:\n\n ```\n [[0, 0, x, x, x, x], -> [[x, x, x, x],\n [0, x, x, x, 0, 0]] [x, x, x, 0]]\n ```\n\n Args:\n mask (`torch.Tensor`):\n 2D tensor (binary mask) with shape `(N, M)`.\n *tensors (`torch.Tensor`):\n One or more 2D tensors with the same shape as `mask`. These tensors will be processed alongside `mask`,\n with non-zero values shifted and excess zero columns truncated in the same manner.\n\n Returns:\n `torch.Tensor`:\n Updated binary mask with non-zero values flushed to the left and trailing zero columns removed.\n `*torch.Tensor`\n Updated tensors, processed in the same way as the mask.\n\n Example:\n ```python\n >>> mask = torch.tensor([[0, 0, 1, 1, 1], [0, 1, 1, 0, 0]])\n >>> tensor = torch.tensor([[9, 9, 2, 3, 4], [9, 5, 6, 9, 9]])\n >>> new_mask, new_tensor = flush_left(mask, tensor)\n >>> print(new_mask)\n tensor([[1, 1, 1],\n [1, 1, 0]])\n\n >>> print(new_tensor)\n tensor([[2, 3, 4],\n [5, 6, 0]])\n ```\n \"\"\"\n _, M = mask.shape\n\n # Create copy of mask and tensors\n mask_copy = mask.clone()\n tensors = [t.clone() for t in tensors]\n\n # Shift non-zero values to the left\n first_non_zero = mask_copy.argmax(dim=1)\n pos = torch.arange(M, device=mask_copy.device).unsqueeze(0)\n idx_roll = (pos + first_non_zero.unsqueeze(1)) % M\n mask_roll = mask_copy.gather(1, idx_roll)\n rolled_tensors = [t.gather(1, idx_roll) for t in tensors]\n\n # Truncate trailing columns that are all zeros in mask_roll\n col_sums = mask_roll.sum(dim=0)\n empty_cols = col_sums == 0\n first_empty_col = int(empty_cols.to(torch.int8).argmax()) if empty_cols.any() else M\n flushed_mask = mask_roll[:, :first_empty_col]\n flushed_tensors = [t[:, :first_empty_col] for t in rolled_tensors]\n\n if not flushed_tensors:\n return flushed_mask\n return flushed_mask, *flushed_tensors\n\n\ndef flush_right(mask: torch.Tensor, *tensors: torch.Tensor) -> torch.Tensor | tuple[torch.Tensor, ...]:\n \"\"\"\n Shift non-zero elements in the mask and corresponding tensors to the right. See `flush_left` for details.\n \"\"\"\n _, M = mask.shape\n\n # Create copy of mask and tensors\n mask_copy = mask.clone()\n tensors = [t.clone() for t in tensors]\n\n # Shift non-zero values to the right\n flipped_mask = torch.fliplr(mask_copy)\n first_non_zero = flipped_mask.argmax(dim=1)\n pos = torch.arange(M, device=mask_copy.device).unsqueeze(0)\n idx_roll = (pos - first_non_zero.unsqueeze(1)) % M\n mask_roll = mask_copy.gather(1, idx_roll)\n rolled_tensors = [t.gather(1, idx_roll) for t in tensors]\n\n # Truncate leading columns that are all zeros in mask_roll\n col_sums = mask_roll.sum(dim=0)\n non_empty_cols = col_sums != 0\n first_non_empty_col = int(non_empty_cols.to(torch.int8).argmax()) if non_empty_cols.any() else M\n flushed_mask = mask_roll[:, first_non_empty_col:]\n flushed_tensors = [t[:, first_non_empty_col:] for t in rolled_tensors]\n\n if not flushed_tensors:\n return flushed_mask\n return flushed_mask, *flushed_tensors\n\n\ndef selective_log_softmax(logits, index) -> torch.Tensor:\n \"\"\"\n A memory-efficient implementation of the common `log_softmax -> gather` operation.\n\n This function is equivalent to the following naive implementation:\n ```python\n # for index with shape (...):\n logps = torch.gather(logits.log_softmax(-1), dim=-1, index=index.unsqueeze(-1)).squeeze(-1)\n # for index with shape (..., K):\n logps = torch.gather(logits.log_softmax(-1), dim=-1, index=index)\n ```\n\n Args:\n logits (`torch.Tensor`):\n Logits tensor of shape `(..., num_classes)`.\n index (`torch.Tensor`):\n Index tensor of shape `(..., K)` or `(...)`, specifying the positions to gather from the log-softmax\n output. When the last case is used, `K` log-probabilities are gathered per position (e.g. for top-K)\n\n Returns:\n `torch.Tensor`:\n Gathered log probabilities with the same shape as `index`.\n \"\"\"\n squeeze = index.ndim == logits.ndim - 1\n if squeeze:\n index = index.unsqueeze(-1)\n\n if logits.dtype in [torch.float32, torch.float64]:\n selected_logits = torch.gather(logits, dim=-1, index=index)\n # loop to reduce peak mem consumption\n logsumexp_values = torch.stack([torch.logsumexp(lg, dim=-1) for lg in logits])\n per_token_logps = selected_logits - logsumexp_values.unsqueeze(-1) # log_softmax(x_i) = x_i - logsumexp(x)\n else:\n # logsumexp approach is unstable with bfloat16, fall back to slightly less efficient approach\n per_token_logps = []\n for row_logits, row_labels in zip(logits, index, strict=True): # loop to reduce peak mem consumption\n row_logps = F.log_softmax(row_logits, dim=-1)\n row_per_token_logps = row_logps.gather(dim=-1, index=row_labels)\n per_token_logps.append(row_per_token_logps)\n per_token_logps = torch.stack(per_token_logps)\n\n if squeeze:\n per_token_logps = per_token_logps.squeeze(-1)\n\n return per_token_logps\n\n\ndef entropy_from_logits(logits: torch.Tensor, chunk_size: int = 128) -> torch.Tensor:\n \"\"\"\n Compute the Shannon entropy (in nats) for each row of *logits* in a memory-efficient way.\n\n Instead of materializing the full softmax for all rows at once, the logits are flattened to shape (N, num_classes),\n where N is the product of all leading dimensions. Computation is then performed in chunks of size `chunk_size`\n along this flattened dimension, reducing peak memory usage. The result is reshaped back to match the input's\n leading dimensions.\n\n Args:\n logits (`torch.Tensor`):\n Logits tensor of shape `(..., num_classes)`. Entropy is taken along the last axis; all leading dimensions\n are preserved in the output.\n chunk_size (`int`, *optional*, defaults to `128`):\n Number of rows from the flattened logits to process per iteration. Smaller values reduce memory usage at\n the cost of more iterations.\n\n Returns:\n `torch.Tensor`:\n Entropy values with shape `logits.shape[:-1]`.\n \"\"\"\n original_shape = logits.shape[:-1] # all dims except num_classes\n num_classes = logits.shape[-1]\n\n # Flatten all leading dimensions into one\n flat_logits = logits.reshape(-1, num_classes)\n\n entropies = []\n for chunk in flat_logits.split(chunk_size, dim=0):\n logps = F.log_softmax(chunk, dim=-1)\n chunk_entropy = -(torch.exp(logps) * logps).sum(-1)\n entropies.append(chunk_entropy)\n\n entropies = torch.cat(entropies, dim=0)\n return entropies.reshape(original_shape)\n\n\ndef print_prompt_completions_sample(\n prompts: list,\n completions: list,\n rewards: dict[str, list[float]],\n advantages: list[float],\n step: int,\n num_samples: int = None,\n) -> None:\n \"\"\"\n Print out a sample of model completions to the console with multiple reward metrics.\n\n This function creates a nicely formatted table showing prompt-completion pairs, useful for monitoring model outputs\n during training. It requires the `rich` library to be installed.\n\n Args:\n prompts (`list`):\n List of prompts. Can be either strings or lists of messages.\n completions (`list`):\n List of completions corresponding to the prompts. Can be either strings or lists of messages.\n rewards (`dict[str, list[float]]`):\n Dictionary where keys are reward names and values are lists of rewards.\n advantages (`list[float]`):\n List of advantages corresponding to the prompts and completions.\n step (`int`):\n Current training step number, used in the output title.\n num_samples (`int`, *optional*):\n Number of random samples to display. If `None` (default), all items will be displayed.\n\n Example:\n ```python\n >>> from trl.trainer.utils import print_prompt_completions_sample\n\n >>> prompts = [\"The sky is\", \"The sun is\"]\n >>> completions = [\" blue.\", \" in the sky.\"]\n >>> rewards = {\"Correctness\": [0.123, 0.456], \"Format\": [0.789, 0.101]}\n >>> advantages = [0.987, 0.654]\n >>> print_prompt_completions_sample(prompts, completions, rewards, advantages, 42)\n โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Step 42 โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ\n โ”‚ โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”‚\n โ”‚ โ”ƒ Prompt โ”ƒ Completion โ”ƒ Correctness โ”ƒ Format โ”ƒ Advantage โ”ƒ โ”‚\n โ”‚ โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ โ”‚\n โ”‚ โ”‚ The sky is โ”‚ blue. โ”‚ 0.12 โ”‚ 0.79 โ”‚ 0.99 โ”‚ โ”‚\n โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚\n โ”‚ โ”‚ The sun is โ”‚ in the sky. โ”‚ 0.46 โ”‚ 0.10 โ”‚ 0.65 โ”‚ โ”‚\n โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚\n โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ\n ```\n \"\"\"\n if not is_rich_available():\n raise ImportError(\n \"The function `print_prompt_completions_sample` requires the `rich` library. Please install it with \"\n \"`pip install rich`.\"\n )\n console = Console()\n table = Table(show_header=True, header_style=\"bold white\", expand=True)\n\n # Add columns\n table.add_column(\"Prompt\", style=\"bright_yellow\")\n table.add_column(\"Completion\", style=\"bright_green\")\n for reward_name in rewards.keys():\n table.add_column(reward_name, style=\"bold cyan\", justify=\"right\")\n table.add_column(\"Advantage\", style=\"bold magenta\", justify=\"right\")\n\n def format_entry(entry) -> Text:\n t = Text()\n if isinstance(entry, list) and all(isinstance(m, dict) for m in entry):\n for j, msg in enumerate(entry):\n role = msg.get(\"role\", \"\")\n if \"content\" in msg:\n # Chat message\n t.append(f\"{role.upper()}\\n\", style=\"bold red\")\n t.append(msg[\"content\"])\n elif \"name\" in msg and \"args\" in msg:\n # Tool call\n t.append(f\"{role.upper()}\\n\", style=\"bold red\")\n t.append(f\"{msg['name']}({msg['args']})\")\n else:\n # Fallback\n t.append(str(msg))\n if j < len(entry) - 1:\n t.append(\"\\n\\n\")\n else:\n t.append(str(entry))\n return t\n\n # Some basic input validation\n if num_samples is not None:\n if num_samples >= len(prompts):\n num_samples = None\n elif num_samples <= 0:\n return\n\n # Subsample data if num_samples is specified\n if num_samples is not None:\n indices = random.sample(range(len(prompts)), num_samples)\n prompts = [prompts[i] for i in indices]\n completions = [completions[i] for i in indices]\n rewards = {key: [val[i] for i in indices] for key, val in rewards.items()}\n advantages = [advantages[i] for i in indices]\n\n for i in range(len(prompts)):\n reward_values = [f\"{rewards[key][i]:.2f}\" for key in rewards.keys()] # 2 decimals\n table.add_row(\n format_entry(prompts[i]),\n format_entry(completions[i]),\n *reward_values,\n f\"{advantages[i]:.2f}\",\n )\n table.add_section() # Adds a separator between rows\n\n panel = Panel(table, expand=False, title=f\"Step {step}\", border_style=\"bold white\")\n console.print(panel)\n\n\nclass RepeatSampler(Sampler):\n \"\"\"\n Sampler that repeats the indices of a dataset in a structured manner.\n\n Args:\n data_source (`Sized`):\n Dataset to sample from.\n mini_repeat_count (`int`):\n Number of times to repeat each index per batch.\n batch_size (`int`, *optional*, defaults to `1`):\n Number of unique indices per batch.\n repeat_count (`int`, *optional*, defaults to `1`):\n Number of times to repeat the full sampling process.\n shuffle (`bool`, *optional*, defaults to `True`):\n Whether to shuffle the dataset.\n seed (`int`, *optional*):\n Random seed for reproducibility (only affects this sampler).\n\n Example:\n ```python\n >>> sampler = RepeatSampler([\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\"], mini_repeat_count=2, batch_size=3, repeat_count=4)\n >>> list(sampler)\n [4, 4, 3, 3, 0, 0,\n 4, 4, 3, 3, 0, 0,\n 4, 4, 3, 3, 0, 0,\n 4, 4, 3, 3, 0, 0,\n 1, 1, 2, 2, 6, 6,\n 1, 1, 2, 2, 6, 6,\n 1, 1, 2, 2, 6, 6,\n 1, 1, 2, 2, 6, 6]\n ```\n\n ```txt\n mini_repeat_count = 3\n - - -\n [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, |\n 4, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, |\n 8, 8, 8, 9, 9, 9, 10, 10, 10, 11, 11, 11, |\n repeat_count = 2\n 0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, |\n 4, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, |\n 8, 8, 8, 9, 9, 9, 10, 10, 10, 11, 11, 11, ...] |\n --------- --------- --------- ---------\n --------- --------- --------- ---------\n --------- --------- --------- ---------\n batch_size = 12\n ```\n \"\"\"\n\n def __init__(\n self,\n data_source: Sized,\n mini_repeat_count: int,\n batch_size: int = 1,\n repeat_count: int = 1,\n shuffle: bool = True,\n seed: int | None = None,\n ):\n self.data_source = data_source\n self.mini_repeat_count = mini_repeat_count\n self.batch_size = batch_size\n self.repeat_count = repeat_count\n self.num_samples = len(data_source)\n self.shuffle = shuffle\n self.seed = seed\n\n if shuffle:\n self.generator = torch.Generator() # Create a local random generator\n if seed is not None:\n self.generator.manual_seed(seed)\n\n def __iter__(self):\n if self.shuffle:\n # E.g., [2, 4, 3, 1, 0, 6, 5] (num_samples = 7)\n indexes = torch.randperm(self.num_samples, generator=self.generator).tolist()\n else:\n indexes = list(range(self.num_samples))\n\n # [2, 4, 3, 1, 0, 6, 5]\n # -> [[2, 4, 3], [1, 0, 6], [5]] (batch_size = 3)\n indexes = [indexes[i : i + self.batch_size] for i in range(0, len(indexes), self.batch_size)]\n\n # [[2, 4, 3], [1, 0, 6], [5]]\n # -> [[2, 4, 3], [1, 0, 6]]\n indexes = [chunk for chunk in indexes if len(chunk) == self.batch_size]\n\n for chunk in indexes:\n for _ in range(self.repeat_count):\n for index in chunk:\n for _ in range(self.mini_repeat_count):\n yield index\n\n def __len__(self) -> int:\n return (self.num_samples // self.batch_size) * self.batch_size * self.mini_repeat_count * self.repeat_count\n\n\n# torch.nanstd doesn't exist, so we define it here\ndef nanstd(tensor: torch.Tensor, dim: int | tuple[int, ...] | None = None, keepdim: bool = False) -> torch.Tensor:\n \"\"\"\n Compute the standard deviation of a tensor, ignoring NaNs.\n\n Args:\n tensor (`torch.Tensor`):\n Input tensor.\n dim (`int` or `tuple[int, ...]`, *optional*):\n Dimension(s) to reduce. Defaults to all dimensions.\n keepdim (`bool`, *optional*, defaults to `False`):\n Whether to keep reduced dimensions.\n\n Returns:\n `torch.Tensor`:\n Standard deviation of the tensor, ignoring NaNs.\n \"\"\"\n # Compute variance ignoring NaNs\n mean = torch.nanmean(tensor, dim=dim, keepdim=True)\n variance = torch.nanmean((tensor - mean) ** 2, dim=dim, keepdim=True)\n count = torch.sum(~torch.isnan(tensor), dim=dim, keepdim=True) # count of non-NaN values\n correction = count / (count - 1)\n correction = torch.where(count > 1, correction, torch.full_like(correction, float(\"nan\")))\n variance *= correction # Bessel's correction\n std = torch.sqrt(variance)\n if keepdim:\n return std\n if dim is None:\n return std.squeeze()\n if isinstance(dim, int):\n return std.squeeze(dim)\n dims = [(d if d >= 0 else d + std.ndim) for d in dim]\n for d in sorted(dims, reverse=True):\n std = std.squeeze(d)\n return std\n\n\ndef split_tensor_dict(\n tensor_dict: dict[str, torch.Tensor | None], num_chunks: int\n) -> list[dict[str, torch.Tensor | None]]:\n \"\"\"\n Splits a dictionary of tensors along the first dimension into `num_chunks` equal parts.\n\n Example:\n ```python\n >>> x = torch.arange(12).reshape(6, 2)\n >>> y = torch.arange(6).reshape(6, 1)\n >>> tensor_dict = {\"x\": x, \"y\": y}\n >>> split_tensor_dict(tensor_dict, 3)\n [\n {\"x\": tensor([[0, 1], [2, 3]]), \"y\": tensor([[0], [1]])},\n {\"x\": tensor([[4, 5], [6, 7]]), \"y\": tensor([[2], [3]])},\n {\"x\": tensor([[ 8, 9], [10, 11]]), \"y\": tensor([[4], [5]])}\n ]\n ```\n \"\"\"\n first_tensor = next(tensor for tensor in tensor_dict.values() if tensor is not None)\n chunk_size = first_tensor.shape[0] // num_chunks\n chunks = []\n for i in range(num_chunks):\n chunk_dict = {}\n for key, tensor in tensor_dict.items():\n if tensor is not None and (isinstance(tensor, list) or tensor.ndim > 0):\n chunk_dict[key] = tensor[i * chunk_size : (i + 1) * chunk_size]\n elif tensor is not None and tensor.ndim == 0:\n chunk_dict[key] = tensor\n else:\n chunk_dict[key] = None\n chunks.append(chunk_dict)\n return chunks\n\n\ndef shuffle_sequence_dict(seq_dict: dict[str, Sequence | None]) -> dict[str, Sequence | None]:\n \"\"\"\n Shuffles all sequence-like values in a dictionary along the first dimension in unison.\n\n Example:\n ```python\n >>> x = torch.arange(6).reshape(3, 2)\n >>> y = [\"a\", \"b\", \"c\"]\n >>> seq_dict = {\"x\": x, \"y\": y}\n >>> shuffle_sequence_dict(seq_dict)\n {'x': tensor([[2, 3],\n [0, 1],\n [4, 5]]),\n 'y': ['b', 'a', 'c']}\n ```\n \"\"\"\n # Determine batch size from the first non-None sequence\n batch_size = len(next(v for v in seq_dict.values() if v is not None))\n permutation = torch.randperm(batch_size)\n\n def permute(v: Sequence | None) -> Sequence | None:\n if v is None:\n return None\n if isinstance(v, torch.Tensor) and v.ndim == 0:\n return v\n if isinstance(v, torch.Tensor) and v.ndim >= 1:\n return v[permutation]\n return [v[i] for i in permutation]\n\n return {key: permute(val) for key, val in seq_dict.items()}\n\n\ndef nanmin(tensor: torch.Tensor) -> torch.Tensor:\n \"\"\"\n Compute the minimum value of a tensor, ignoring NaNs. This function only supports 1D tensors.\n\n Args:\n tensor (`torch.Tensor`): Input tensor of shape `(N,)`.\n\n Returns:\n `torch.Tensor`: Minimum value of the tensor, ignoring NaNs. Returns NaN if all values are NaN.\n \"\"\"\n if torch.isnan(tensor).all():\n return torch.tensor(float(\"nan\"), dtype=tensor.dtype, device=tensor.device)\n return torch.min(tensor[~torch.isnan(tensor)])\n\n\ndef nanmax(tensor: torch.Tensor) -> torch.Tensor:\n \"\"\"\n Compute the maximum value of a tensor, ignoring NaNs. This function only supports 1D tensors.\n\n Args:\n tensor (`torch.Tensor`): Input tensor of shape `(N,)`.\n\n Returns:\n `torch.Tensor`: Maximum value of the tensor, ignoring NaNs. Returns NaN if all values are NaN.\n \"\"\"\n if torch.isnan(tensor).all():\n return torch.tensor(float(\"nan\"), dtype=tensor.dtype, device=tensor.device)\n return torch.max(tensor[~torch.isnan(tensor)])\n\n\ndef identity(x):\n \"\"\"Do we really need docs for this?\"\"\"\n return x\n\n\ndef split_pixel_values_by_grid(batch: dict[str, torch.Tensor]) -> dict[str, torch.Tensor | list[torch.Tensor]]:\n \"\"\"\n Splits `batch[\"pixel_values\"]` into a list of tensors based on the product of each row in `batch[\"image_grid_thw\"]`\n and batch[\"num_images\"] while keeping other entries unchanged.\n \"\"\"\n if \"image_grid_thw\" not in batch or \"pixel_values\" not in batch or \"num_images\" not in batch:\n return batch\n\n lengths = batch[\"image_grid_thw\"].prod(-1).tolist() # [num_images]\n pixel_values = batch[\"pixel_values\"] # [total, feature_dim]\n\n if sum(lengths) != pixel_values.size(0):\n raise ValueError(f\"Mismatch: sum(lengths) = {sum(lengths)} != pixel_values.size(0) = {pixel_values.size(0)}\")\n\n boundaries = [0, *accumulate(batch[\"num_images\"])] # [3, 4, 5] -> [0, 3, 7, 12]\n sections = [sum(lengths[boundaries[i] : boundaries[i + 1]]) for i in range(len(batch[\"num_images\"]))]\n split_values = list(torch.split(batch[\"pixel_values\"], sections, dim=0))\n image_grid_thw = list(torch.split(batch[\"image_grid_thw\"], batch[\"num_images\"], dim=0))\n return {**batch, \"pixel_values\": split_values, \"image_grid_thw\": image_grid_thw}\n\n\ndef unsplit_pixel_values_by_grid(batch: dict[str, torch.Tensor | list[torch.Tensor]]) -> dict[str, torch.Tensor]:\n \"\"\"\n Opposite of `split_pixel_values_by_grid`. Merges a list of tensors in `batch[\"pixel_values\"]` back into a single\n tensor along the first dimension.\n \"\"\"\n pixel_values = batch.get(\"pixel_values\")\n if isinstance(pixel_values, list):\n merged = torch.cat(pixel_values, dim=0)\n batch = {**batch, \"pixel_values\": merged}\n\n image_grid_thw = batch.get(\"image_grid_thw\")\n if isinstance(image_grid_thw, list):\n merged = torch.cat(image_grid_thw, dim=0)\n batch = {**batch, \"image_grid_thw\": merged}\n\n return batch\n\n\nTListOrMapping = TypeVar(\"TListOrMapping\", list, Mapping)\n\n\ndef remove_none_values(example: TListOrMapping) -> TListOrMapping:\n \"\"\"\n Recursively removes entries with `None` values from a nested structure (list or dictionary).\n\n Args:\n example (`list` or `Mapping`):\n Input nested structure (list or dictionary) from which to remove `None`.\n\n Example:\n ```python\n >>> [\n ... {\n ... \"a\": {\"aa\": None, \"ab\": 1},\n ... \"b\": \"my_string\",\n ... }\n ... ]\n >>> remove_none_values(example)\n [{'a': {'ab': 1}, 'b': 'my_string'}]\n ```\n \"\"\"\n if isinstance(example, list):\n return [remove_none_values(value) if isinstance(value, (dict, list)) else value for value in example]\n elif isinstance(example, Mapping):\n return {\n key: remove_none_values(value) if isinstance(value, (dict, list)) else value\n for key, value in example.items()\n if value is not None\n }\n else:\n raise TypeError(\"Input must be a list or a dictionary.\")\n\n\ndef create_model_from_path(\n model_id: str, architecture: _BaseAutoModelClass | None = None, **kwargs\n) -> PreTrainedModel:\n \"\"\"\n Create a model from a given path using the specified initialization arguments.\n\n Args:\n model_id (`str`):\n Path to the model. Can be either a local directory or a model identifier from the Hugging Face Hub.\n architecture (`_BaseAutoModelClass` or `None`, *optional*):\n Model architecture class to instantiate. The model is initialized using the `from_pretrained` method of\n this class. If `None`, the architecture will be inferred from the model's configuration.\n kwargs (`dict`):\n Initialization keyword arguments to pass to the model's `from_pretrained` method. When `'dtype'` is\n specified, it can be either a `torch.dtype` or one of the strings: `'bfloat16'`, `'float16'`, `'float32'`,\n or `'auto'`. If not explicitly set, `dtype` defaults to `'float32'`.\n\n Returns:\n [`~transformers.PreTrainedModel`]:\n The instantiated model.\n \"\"\"\n dtype = kwargs.get(\"dtype\", \"float32\")\n if isinstance(dtype, torch.dtype) or dtype == \"auto\" or dtype is None:\n pass # dtype is already a torch.dtype or \"auto\" or None\n elif isinstance(dtype, str) and dtype in [\"bfloat16\", \"float16\", \"float32\"]:\n kwargs[\"dtype\"] = getattr(torch, dtype)\n else:\n raise ValueError(\n \"Invalid `dtype` passed to the config. Expected either 'auto' or a string representing \"\n f\"a valid `torch.dtype` (e.g., 'float32'), but got {dtype}.\"\n )\n kwargs[\"device_map\"] = kwargs.get(\"device_map\", \"auto\")\n if architecture is None:\n config = AutoConfig.from_pretrained(model_id)\n architecture = getattr(transformers, config.architectures[0])\n model = architecture.from_pretrained(model_id, **kwargs)\n return model\n\n\ndef hash_module(module: torch.nn.Module) -> str:\n h = hashlib.sha256()\n for _, tensor in sorted(module.state_dict().items()):\n tensor = tensor.cpu()\n h.update(str(tensor.dtype).encode())\n if tensor.dtype in [torch.bfloat16, torch.float8_e4m3fn, torch.float8_e5m2]:\n tensor = tensor.to(torch.float32)\n h.update(tensor.numpy().tobytes())\n return h.hexdigest()\n\n\ndef get_config_model_id(config: PretrainedConfig) -> str:\n \"\"\"\n Retrieve the model identifier from a given model configuration.\n\n Args:\n config ([`~transformers.PreTrainedConfig`]):\n Configuration from which to extract the model identifier.\n\n Returns:\n `str`:\n The model identifier associated with the model configuration.\n \"\"\"\n return getattr(config, \"_name_or_path\", \"\")\n\n\n@dataclass\nclass CausalLMOutputWithPastAndFlatLogits(CausalLMOutputWithPast):\n flat_logits: torch.Tensor | None = None\n\n\ndef forward_masked_logits(\n model: PreTrainedModel, logits_mask: torch.LongTensor, **kwargs\n) -> CausalLMOutputWithPastAndFlatLogits:\n \"\"\"\n Run a Causal LM forward pass while computing logits only for masked positions to reduce memory usage.\n\n These are always equal:\n\n ```python\n full_outputs = model(input_ids=input_ids)\n masked_outputs = forward_masked_logits(model, mask, input_ids=input_ids)\n\n assert torch.equal(\n masked_outputs.flat_logits,\n full_outputs.logits[mask.bool()],\n )\n ```\n\n Args:\n model ([`~transformers.PreTrainedModel`]):\n A causal language model.\n logits_mask (`torch.LongTensor`):\n Boolean-like tensor indicating which token positions should have logits computed. Shape should match the\n input sequence shape in `kwargs` (typically `[batch, seq_len]`).\n **kwargs:\n Keyword arguments forwarded to the inner decoder (e.g., `input_ids`, `attention_mask`, `past_key_values`).\n\n Returns:\n `CausalLMOutputWithPastAndFlatLogits`: Output containing logits only for the unmasked positions.\n\n Raises:\n ValueError: If `logits_to_keep` or `labels` are provided in `kwargs`.\n \"\"\"\n if kwargs.get(\"logits_to_keep\") is not None:\n raise ValueError(\"`logits_to_keep` is not supported by this forward helper.\")\n if kwargs.get(\"labels\") is not None:\n raise ValueError(\"`labels` is not yet supported by this forward helper.\")\n\n outputs: BaseModelOutputWithPast = model.get_decoder()(**kwargs)\n hidden_states = outputs.last_hidden_state\n\n # Only compute necessary logits, and do not upcast them to float if we are not computing the loss\n flat_logits = model.lm_head(hidden_states[logits_mask.bool()])\n if hasattr(model, \"logit_scale\"): # CohereForCausalLM has this attribute\n flat_logits = flat_logits * model.logit_scale\n\n return CausalLMOutputWithPastAndFlatLogits(\n flat_logits=flat_logits,\n # We use .get(...) because some models like FalconMambaForCausalLM don't return past_key_values or attentions\n past_key_values=outputs.get(\"past_key_values\"),\n hidden_states=outputs.hidden_states,\n attentions=outputs.get(\"attentions\"),\n )\n\n\n@contextmanager\ndef use_adapter(model: \"PeftModel\", adapter_name: str | None):\n \"\"\"\n Context manager to temporarily set and reset the active adapter in a PEFT model.\n\n Args:\n model ([`~peft.PeftModel`]):\n PEFT model to manage.\n adapter_name (`str` or `None`):\n Name of the adapter to set as active. If `None`, the context manager will disable all adapters.\n\n Example:\n ```python\n >>> from trl.trainer.utils import use_adapter\n >>> from peft import AutoPeftModelForCausalLM\n >>> import torch\n\n >>> model = AutoPeftModelForCausalLM.from_pretrained(\"path/to/model\")\n >>> input_ids = torch.tensor([[1, 2, 3]])\n >>> with use_adapter(model, \"adapter_name\"):\n ... outputs = model(input_ids)\n ```\n \"\"\"\n\n if not is_peft_available():\n raise ImportError(\n \"You're trying to use a PEFT adapter but PEFT is not installed. Please install it with `pip install peft`.\"\n )\n if adapter_name is None:\n with model.disable_adapter():\n yield\n else:\n previous_adapter = model.active_adapter\n model.set_adapter(adapter_name)\n try:\n yield\n finally:\n model.set_adapter(previous_adapter)\n\n\ndef start_event_loop_in_daemon(\n name: str | None = None,\n) -> tuple[threading.Thread, asyncio.AbstractEventLoop, threading.Event]:\n \"\"\"\n This function creates a new daemon thread that runs the provided event loop.\n\n Args:\n name (`str`, *optional*):\n Name of the thread. If `None`, the default thread naming will be used.\n\n Returns:\n `threading.Thread`:\n The thread running the event loop.\n `asyncio.AbstractEventLoop`:\n The event loop being run in the thread.\n `threading.Event`:\n An event that is set when the loop is ready.\n \"\"\"\n loop = asyncio.new_event_loop()\n loop_ready_event = threading.Event()\n\n def run_loop():\n asyncio.set_event_loop(loop)\n loop_ready_event.set()\n loop.run_forever()\n\n thread = threading.Thread(target=run_loop, name=name, daemon=True)\n thread.start()\n return thread, loop, loop_ready_event\n\n\ndef shutdown_event_loop_in_daemon(\n thread: threading.Thread | None,\n loop: asyncio.AbstractEventLoop | None,\n) -> None:\n \"\"\"\n Shutdown an asyncio event loop running in a separate thread.\n\n This function stops the event loop and waits for the associated thread to finish execution.\n\n Args:\n thread (`threading.Thread`):\n The thread running the event loop.\n loop (`asyncio.AbstractEventLoop`):\n The asyncio event loop to shut down.\n \"\"\"\n if loop is None or thread is None:\n return\n loop.call_soon_threadsafe(loop.stop)\n thread.join(timeout=5)\n" } ]